Date: Mon, 13 Dec 2021 11:19:35 -0600
From: Justin Pryzby <pryzby@telsasoft.com>
To: Robert Haas <robertmhaas@gmail.com>
Cc: "Bossart, Nathan" <bossartn@amazon.com>,
	Bharath Rupireddy <bharath.rupireddyforpostgres@gmail.com>,
	pgsql-hackers@postgresql.org
Subject: Re: O(n) tasks cause lengthy startups and checkpoints
Message-ID: <20211213171935.GX17618@telsasoft.com>
References: <C1EE64B0-D4DB-40F3-98C8-0CED324D34CB@amazon.com>
 <CALj2ACXYDPwhR6BkQuKTaJjO-y4i8kQCQxydTcBZy96UjN6FqA@mail.gmail.com>
 <5E58EBB2-3DAE-4400-9EBE-60D1FE4C5B10@amazon.com>
 <CALj2ACUPSG4aMqCEO9L+xSyfLYaL88E377rqV_Jsc7P82+U2xA@mail.gmail.com>
 <8ACCEA4C-BA36-494B-92B4-EF14DAF1BC3B@amazon.com>
 <CALj2ACUJ8LGwvc=ZzMrW7X6O5nWA11AqPHFjUdDL66kv3qQgpQ@mail.gmail.com>
 <2B230EF6-7509-479C-8C5A-34B56A1B4063@amazon.com>
 <6DFA78EC-EF4D-4CB0-82C8-AD5DA1B8E0AC@amazon.com>
 <CA+Tgmoag+stJPU9Vxoyaq6gzZGPLQ9C+edZcKb4PgW-yTB8LaA@mail.gmail.com>
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: 
 <CA+Tgmoag+stJPU9Vxoyaq6gzZGPLQ9C+edZcKb4PgW-yTB8LaA@mail.gmail.com>
User-Agent: Mutt/1.9.4 (2018-02-28)
Archived-At: 
 <https://www.postgresql.org/message-id/20211213171935.GX17618%40telsasoft.com>
Precedence: bulk

On Mon, Dec 13, 2021 at 08:53:37AM -0500, Robert Haas wrote:
> On Fri, Dec 10, 2021 at 2:03 PM Bossart, Nathan <bossartn@amazon.com> wrote:
> > Well, I haven't had a chance to look at your patch, and my patch set
> > still only has handling for CheckPointSnapBuild() and
> > RemovePgTempFiles(), but I thought I'd share what I have anyway.  I
> > split it into 5 patches:
> >
> > 0001 - Adds a new "custodian" auxiliary process that does nothing.
...
> 
> I don't know whether this kind of idea is good or not.
...
> 
> Another issue is that we don't want to increase the number of
> processes without bound. Processes use memory and CPU resources and if
> we run too many of them it becomes a burden on the system. Low-end
> systems may not have too many resources in total, and high-end systems
> can struggle to fit demanding workloads within the resources that they
> have. Maybe it would be cheaper to do more things at once if we were
> using threads rather than processes, but that day still seems fairly
> far off.

Maybe that's an argument that this should be a dynamic background worker
instead of an auxilliary process.  Then maybe it would be controlled by
max_parallel_maintenance_workers (or something similar).  The checkpointer
would need to do these tasks itself if parallel workers were disabled or
couldn't be launched.

-- 
Justin