From: Nathan Bossart <nathandbossart@gmail.com>
To: Robert Haas <robertmhaas@gmail.com>
Cc: Simon Riggs <simon.riggs@enterprisedb.com>
Cc: Andres Freund <andres@anarazel.de>
Cc: Bossart, Nathan <bossartn@amazon.com>
Cc: Bharath Rupireddy <bharath.rupireddyforpostgres@gmail.com>
Cc: Maxim Orlov <orlovmg@gmail.com>
Cc: Amul Sul <sulamul@gmail.com>
Cc: Bruce Momjian <bruce@momjian.us>
Cc: pgsql-hackers@postgresql.org <pgsql-hackers@postgresql.org>
Subject: Re: O(n) tasks cause lengthy startups and checkpoints
Date: Thu, 23 Jun 2022 10:15:52 -0700
Message-ID: <20220623171552.GA96638@nathanxps13> (raw)
In-Reply-To: <CA+TgmoYkDY7r8PwAGhD3tMqVAeKBfY-_7miJKcj-ZzPWN0UYzQ@mail.gmail.com>
References: <20220217182337.GA3247866@nathanxps13>
<20220217192709.eyf2cnymhxqsals7@alap3.anarazel.de>
<20220217210022.GA3248793@nathanxps13>
<20220217222829.lndzwq72fh4v34c2@alap3.anarazel.de>
<20220217225838.GA3292179@nathanxps13>
<20220217231247.llfqjvvuk6rkawel@alap3.anarazel.de>
<20220218164454.GA3403811@nathanxps13>
<20220218205111.GA3447220@nathanxps13>
<CANbhV-H9=DOBdpKE77XB_hiPDKVre-+1vuLvKWTke07Y3K1P5w@mail.gmail.com>
<CA+TgmoYkDY7r8PwAGhD3tMqVAeKBfY-_7miJKcj-ZzPWN0UYzQ@mail.gmail.com>
On Thu, Jun 23, 2022 at 09:46:28AM -0400, Robert Haas wrote:
> I do agree that a general mechanism for getting cleanup tasks done in
> the background could be a useful thing to have, but I feel like it's
> hard to see exactly how to make it work well. We can't just allow it
> to spin up a million new processes, but at the same time, if it can't
> guarantee that time-critical tasks get performed relatively quickly,
> it's pretty worthless.
My intent with this new auxiliary process is to offload tasks that aren't
particularly time-critical. They are only time-critical in the sense that
1) you might eventually run out of space and 2) you might encounter
wraparound with the logical replication files. But AFAICT these same risks
exist today in the checkpointer approach, although maybe not to the same
extent. In any case, 2 seems solvable to me outside of this patch set.
I'm grateful for the discussion in this thread so far, but I'm not seeing a
clear path forward. I'm glad to see threads like the one to stop doing
end-of-recovery checkpoints [0], but I don't know if it will be possible to
solve all of these availability concerns in a piecemeal fashion. I remain
open to exploring other suggested approaches beyond creating a new
auxiliary process.
[0] https://postgr.es/m/CA%2BTgmobrM2jvkiccCS9NgFcdjNSgAvk1qcAPx5S6F%2BoJT3D2mQ%40mail.gmail.com
--
Nathan Bossart
Amazon Web Services: https://aws.amazon.com
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Reply to all the recipients using the --to and --cc options:
reply via email
To: pgsql-hackers@postgresql.org
Cc: nathandbossart@gmail.com, robertmhaas@gmail.com, simon.riggs@enterprisedb.com, andres@anarazel.de, bossartn@amazon.com, bharath.rupireddyforpostgres@gmail.com, orlovmg@gmail.com, sulamul@gmail.com, bruce@momjian.us
Subject: Re: O(n) tasks cause lengthy startups and checkpoints
In-Reply-To: <20220623171552.GA96638@nathanxps13>
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox