From: Nathan Bossart <nathandbossart@gmail.com>
To: Tom Lane <tgl@sss.pgh.pa.us>
Cc: Andres Freund <andres@anarazel.de>
Cc: Bharath Rupireddy <bharath.rupireddyforpostgres@gmail.com>
Cc: Robert Haas <robertmhaas@gmail.com>
Cc: Bossart, Nathan <bossartn@amazon.com>
Cc: Maxim Orlov <orlovmg@gmail.com>
Cc: Amul Sul <sulamul@gmail.com>
Cc: Bruce Momjian <bruce@momjian.us>
Cc: pgsql-hackers@postgresql.org <pgsql-hackers@postgresql.org>
Subject: Re: O(n) tasks cause lengthy startups and checkpoints
Date: Sun, 2 Apr 2023 14:31:52 -0700
Message-ID: <20230402213152.GB27239@nathanxps13> (raw)
In-Reply-To: <1058306.1680467858@sss.pgh.pa.us>
References: <20221201214026.GA1799688@nathanxps13>
<CALj2ACWupmFrhv4ZOdiB8rrt2UuC8bpCO_v=Sbu+fFEoSgYoNw@mail.gmail.com>
<20221202191507.GA2277157@nathanxps13>
<CALj2ACW1F-T866iaqA_9h9fA+Axdyc86CHz7_k55RGH5HH2rtA@mail.gmail.com>
<20230203054808.GA83788@nathanxps13>
<20230217234344.GA3357392@nathanxps13>
<1031491.1680457205@sss.pgh.pa.us>
<20230402184226.kkjplqvqu6utvzbt@awork3.anarazel.de>
<20230402195005.GB25018@nathanxps13>
<1058306.1680467858@sss.pgh.pa.us>
On Sun, Apr 02, 2023 at 04:37:38PM -0400, Tom Lane wrote:
> Nathan Bossart <nathandbossart@gmail.com> writes:
>> It's been a little while since I dug into this, but I do see your point
>> that the wraparound risk could be higher in some cases. For example, if
>> you have a billion temp files to clean up, the custodian could be stuck on
>> that task for a long time. I will give this some further thought. I'm all
>> ears if anyone has ideas about how to reduce this risk.
>
> I wonder if a single long-lived custodian task is the right model at all.
> At least for RemovePgTempFiles, it'd make more sense to write it as a
> background worker that spawns, does its work, and then exits,
> independently of anything else. Of course, then you need some mechanism
> for ensuring that a bgworker slot is available when needed, but that
> doesn't seem horridly difficult --- we could have a few "reserved
> bgworker" slots, perhaps. An idle bgworker slot doesn't cost much.
This has crossed my mind. Even if we use the custodian for several
different tasks, perhaps it could shut down while not in use. For many
servers, the custodian process will be used sparingly, if at all. And if
we introduce something like custodian_max_workers, perhaps we could dodge
the wraparound issue a bit by setting the default to the number of
supported tasks. That being said, this approach adds some complexity.
--
Nathan Bossart
Amazon Web Services: https://aws.amazon.com
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Reply to all the recipients using the --to and --cc options:
reply via email
To: pgsql-hackers@postgresql.org
Cc: nathandbossart@gmail.com, tgl@sss.pgh.pa.us, andres@anarazel.de, bharath.rupireddyforpostgres@gmail.com, robertmhaas@gmail.com, bossartn@amazon.com, orlovmg@gmail.com, sulamul@gmail.com, bruce@momjian.us
Subject: Re: O(n) tasks cause lengthy startups and checkpoints
In-Reply-To: <20230402213152.GB27239@nathanxps13>
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox