Date: Sun, 2 Apr 2023 14:31:52 -0700
From: Nathan Bossart <nathandbossart@gmail.com>
To: Tom Lane <tgl@sss.pgh.pa.us>
Cc: Andres Freund <andres@anarazel.de>,
	Bharath Rupireddy <bharath.rupireddyforpostgres@gmail.com>,
	Robert Haas <robertmhaas@gmail.com>,
	"Bossart, Nathan" <bossartn@amazon.com>,
	Maxim Orlov <orlovmg@gmail.com>, Amul Sul <sulamul@gmail.com>,
	Bruce Momjian <bruce@momjian.us>,
	"pgsql-hackers@postgresql.org" <pgsql-hackers@postgresql.org>
Subject: Re: O(n) tasks cause lengthy startups and checkpoints
Message-ID: <20230402213152.GB27239@nathanxps13>
References: <20221201214026.GA1799688@nathanxps13>
 <CALj2ACWupmFrhv4ZOdiB8rrt2UuC8bpCO_v=Sbu+fFEoSgYoNw@mail.gmail.com>
 <20221202191507.GA2277157@nathanxps13>
 <CALj2ACW1F-T866iaqA_9h9fA+Axdyc86CHz7_k55RGH5HH2rtA@mail.gmail.com>
 <20230203054808.GA83788@nathanxps13>
 <20230217234344.GA3357392@nathanxps13>
 <1031491.1680457205@sss.pgh.pa.us>
 <20230402184226.kkjplqvqu6utvzbt@awork3.anarazel.de>
 <20230402195005.GB25018@nathanxps13>
 <1058306.1680467858@sss.pgh.pa.us>
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <1058306.1680467858@sss.pgh.pa.us>
Archived-At: 
 <https://www.postgresql.org/message-id/20230402213152.GB27239%40nathanxps13>
Precedence: bulk

On Sun, Apr 02, 2023 at 04:37:38PM -0400, Tom Lane wrote:
> Nathan Bossart <nathandbossart@gmail.com> writes:
>> It's been a little while since I dug into this, but I do see your point
>> that the wraparound risk could be higher in some cases.  For example, if
>> you have a billion temp files to clean up, the custodian could be stuck on
>> that task for a long time.  I will give this some further thought.  I'm all
>> ears if anyone has ideas about how to reduce this risk.
> 
> I wonder if a single long-lived custodian task is the right model at all.
> At least for RemovePgTempFiles, it'd make more sense to write it as a
> background worker that spawns, does its work, and then exits,
> independently of anything else.  Of course, then you need some mechanism
> for ensuring that a bgworker slot is available when needed, but that
> doesn't seem horridly difficult --- we could have a few "reserved
> bgworker" slots, perhaps.  An idle bgworker slot doesn't cost much.

This has crossed my mind.  Even if we use the custodian for several
different tasks, perhaps it could shut down while not in use.  For many
servers, the custodian process will be used sparingly, if at all.  And if
we introduce something like custodian_max_workers, perhaps we could dodge
the wraparound issue a bit by setting the default to the number of
supported tasks.  That being said, this approach adds some complexity.

-- 
Nathan Bossart
Amazon Web Services: https://aws.amazon.com