public inbox for [email protected]  

help / color / mirror / Atom feed

From: Bossart, Nathan <[email protected]>
To: Bharath Rupireddy <[email protected]>
Cc: [email protected] <[email protected]>
Subject: Re: O(n) tasks cause lengthy startups and checkpoints
Date: Fri, 3 Dec 2021 18:20:44 +0000
Message-ID: <[email protected]> (raw)
In-Reply-To: <CALj2ACUPSG4aMqCEO9L+xSyfLYaL88E377rqV_Jsc7P82+U2xA@mail.gmail.com>
References: <[email protected]>
	<CALj2ACXYDPwhR6BkQuKTaJjO-y4i8kQCQxydTcBZy96UjN6FqA@mail.gmail.com>
	<[email protected]>
	<CALj2ACUPSG4aMqCEO9L+xSyfLYaL88E377rqV_Jsc7P82+U2xA@mail.gmail.com>

On 12/3/21, 5:57 AM, "Bharath Rupireddy" <[email protected]> wrote:
> On Fri, Dec 3, 2021 at 3:01 AM Bossart, Nathan <[email protected]> wrote:
>>
>> On 12/1/21, 6:48 PM, "Bharath Rupireddy" <[email protected]> wrote:
>> > +1 for the overall idea of making the checkpoint faster. In fact, we
>> > here at our team have been thinking about this problem for a while. If
>> > there are a lot of files that checkpoint has to loop over and remove,
>> > IMO, that task can be delegated to someone else (maybe a background
>> > worker called background cleaner or bg cleaner, of course, we can have
>> > a GUC to enable or disable it). The checkpoint can just write some
>>
>> Right.  IMO it isn't optimal to have critical things like startup and
>> checkpointing depend on somewhat-unrelated tasks.  I understand the
>> desire to avoid adding additional processes, and maybe it is a bigger
>> hammer than what is necessary to reduce the impact, but it seemed like
>> a natural solution for this problem.  That being said, I'm all for
>> exploring other ways to handle this.
>
> Having a generic background cleaner process (controllable via a few
> GUCs), which can delete a bunch of files (snapshot, mapping, old WAL,
> temp files etc.) or some other task on behalf of the checkpointer,
> seems to be the easiest solution.
>
> I'm too open for other ideas.

I might hack something together for the separate worker approach, if
for no other reason than to make sure I really understand how these
functions work.  If/when a better idea emerges, we can alter course.

Nathan

view thread (59+ messages)  latest in thread

reply

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Reply to all the recipients using the --to and --cc options:
  reply via email

  To: [email protected]
  Cc: [email protected], [email protected]
  Subject: Re: O(n) tasks cause lengthy startups and checkpoints
  In-Reply-To: <[email protected]>

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

This inbox is served by agora; see mirroring instructions
for how to clone and mirror all data and code used for this inbox