Date: Fri, 18 Feb 2022 08:44:54 -0800
From: Nathan Bossart <nathandbossart@gmail.com>
To: Andres Freund <andres@anarazel.de>
Cc: "Bossart, Nathan" <bossartn@amazon.com>,
	Bharath Rupireddy <bharath.rupireddyforpostgres@gmail.com>,
	Maxim Orlov <orlovmg@gmail.com>, Amul Sul <sulamul@gmail.com>,
	Bruce Momjian <bruce@momjian.us>,
	Robert Haas <robertmhaas@gmail.com>,
	"pgsql-hackers@postgresql.org" <pgsql-hackers@postgresql.org>
Subject: Re: O(n) tasks cause lengthy startups and checkpoints
Message-ID: <20220218164454.GA3403811@nathanxps13>
References: <20220217005057.GA3201038@nathanxps13>
 <20220217015052.2y3wxibeommk37ey@alap3.anarazel.de>
 <20220217041404.GA3243546@nathanxps13>
 <20220217065938.x2esfdppzypegn5j@alap3.anarazel.de>
 <20220217182337.GA3247866@nathanxps13>
 <20220217192709.eyf2cnymhxqsals7@alap3.anarazel.de>
 <20220217210022.GA3248793@nathanxps13>
 <20220217222829.lndzwq72fh4v34c2@alap3.anarazel.de>
 <20220217225838.GA3292179@nathanxps13>
 <20220217231247.llfqjvvuk6rkawel@alap3.anarazel.de>
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <20220217231247.llfqjvvuk6rkawel@alap3.anarazel.de>
Archived-At: <https://www.postgresql.org/message-id/20220218164454.GA3403811%40nathanxps13>
Precedence: bulk

On Thu, Feb 17, 2022 at 03:12:47PM -0800, Andres Freund wrote:
>> > The improvements around deleting temporary files and serialized snapshots
>> > afaict don't require a dedicated process - they're only relevant during
>> > startup. We could use the approach of renaming the directory out of the way as
>> > done in this patchset but perform the cleanup in the startup process after
>> > we're up.
>> 
>> Perhaps this is a good place to start.  As I mentioned above, IME the
>> temporary file cleanup is the most common problem, so I think even getting
>> that one fixed would be a huge improvement.
> 
> Cool.

Hm.  How should this work for standbys?  I can think of the following
options:
	1. Do temporary file cleanup in the postmaster (as it does today).
	2. Pause after allowing connections to clean up temporary files.
	3. Do small amounts of temporary file cleanup whenever there is an
	   opportunity during recovery.
	4. Wait until recovery completes before cleaning up temporary files.

I'm not too thrilled about any of these options.

-- 
Nathan Bossart
Amazon Web Services: https://aws.amazon.com