Date: Thu, 23 Jun 2022 10:15:52 -0700
From: Nathan Bossart <nathandbossart@gmail.com>
To: Robert Haas <robertmhaas@gmail.com>
Cc: Simon Riggs <simon.riggs@enterprisedb.com>,
	Andres Freund <andres@anarazel.de>,
	"Bossart, Nathan" <bossartn@amazon.com>,
	Bharath Rupireddy <bharath.rupireddyforpostgres@gmail.com>,
	Maxim Orlov <orlovmg@gmail.com>, Amul Sul <sulamul@gmail.com>,
	Bruce Momjian <bruce@momjian.us>,
	"pgsql-hackers@postgresql.org" <pgsql-hackers@postgresql.org>
Subject: Re: O(n) tasks cause lengthy startups and checkpoints
Message-ID: <20220623171552.GA96638@nathanxps13>
References: <20220217182337.GA3247866@nathanxps13>
 <20220217192709.eyf2cnymhxqsals7@alap3.anarazel.de>
 <20220217210022.GA3248793@nathanxps13>
 <20220217222829.lndzwq72fh4v34c2@alap3.anarazel.de>
 <20220217225838.GA3292179@nathanxps13>
 <20220217231247.llfqjvvuk6rkawel@alap3.anarazel.de>
 <20220218164454.GA3403811@nathanxps13>
 <20220218205111.GA3447220@nathanxps13>
 <CANbhV-H9=DOBdpKE77XB_hiPDKVre-+1vuLvKWTke07Y3K1P5w@mail.gmail.com>
 <CA+TgmoYkDY7r8PwAGhD3tMqVAeKBfY-_7miJKcj-ZzPWN0UYzQ@mail.gmail.com>
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: 
 <CA+TgmoYkDY7r8PwAGhD3tMqVAeKBfY-_7miJKcj-ZzPWN0UYzQ@mail.gmail.com>
Archived-At: 
 <https://www.postgresql.org/message-id/20220623171552.GA96638%40nathanxps13>
Precedence: bulk

On Thu, Jun 23, 2022 at 09:46:28AM -0400, Robert Haas wrote:
> I do agree that a general mechanism for getting cleanup tasks done in
> the background could be a useful thing to have, but I feel like it's
> hard to see exactly how to make it work well. We can't just allow it
> to spin up a million new processes, but at the same time, if it can't
> guarantee that time-critical tasks get performed relatively quickly,
> it's pretty worthless.

My intent with this new auxiliary process is to offload tasks that aren't
particularly time-critical.  They are only time-critical in the sense that
1) you might eventually run out of space and 2) you might encounter
wraparound with the logical replication files.  But AFAICT these same risks
exist today in the checkpointer approach, although maybe not to the same
extent.  In any case, 2 seems solvable to me outside of this patch set.

I'm grateful for the discussion in this thread so far, but I'm not seeing a
clear path forward.  I'm glad to see threads like the one to stop doing
end-of-recovery checkpoints [0], but I don't know if it will be possible to
solve all of these availability concerns in a piecemeal fashion.  I remain
open to exploring other suggested approaches beyond creating a new
auxiliary process.

[0] https://postgr.es/m/CA%2BTgmobrM2jvkiccCS9NgFcdjNSgAvk1qcAPx5S6F%2BoJT3D2mQ%40mail.gmail.com

-- 
Nathan Bossart
Amazon Web Services: https://aws.amazon.com