pg.ddx.io pgsql-committers@postgresql.org mailing list archive
help / color / mirror / Atom feedpgsql: aio: Infrastructure for io_method=worker
5+ messages / 3 participants
[nested] [flat]
Commits from this thread:
2025-04-29 810a8b1c80 Give up on running with NetBSD/OpenBSD's default semaphore settings.
* pgsql: aio: Infrastructure for io_method=worker
@ 2025-03-18 16:01 Andres Freund <andres@anarazel.de>
2025-03-18 22:53 ` Re: pgsql: aio: Infrastructure for io_method=worker Tom Lane <tgl@sss.pgh.pa.us>
0 siblings, 1 reply; 5+ messages in thread
From: Andres Freund @ 2025-03-18 16:01 UTC (permalink / raw)
To: pgsql-committers@lists.postgresql.org
aio: Infrastructure for io_method=worker
This commit contains the basic, system-wide, infrastructure for
io_method=worker. It does not yet actually execute IO, this commit just
provides the infrastructure for running IO workers, kept separate for easier
review.
The number of IO workers can be adjusted with a PGC_SIGHUP GUC. Eventually
we'd like to make the number of workers dynamically scale up/down based on the
current "IO load".
To allow the number of IO workers to be increased without a restart, we need
to reserve PGPROC entries for the workers unconditionally. This has been
judged to be worth the cost. If it turns out to be problematic, we can
introduce a PGC_POSTMASTER GUC to control the maximum number.
As io workers might be needed during shutdown, e.g. for AIO during the
shutdown checkpoint, a new PMState phase is added. IO workers are shut down
after the shutdown checkpoint has been performed and walsender/archiver have
shut down, but before the checkpointer itself shuts down. See also
87a6690cc69.
Updates PGSTAT_FILE_FORMAT_ID due to the addition of a new BackendType.
Reviewed-by: Noah Misch <noah@leadboat.com>
Co-authored-by: Thomas Munro <thomas.munro@gmail.com>
Co-authored-by: Andres Freund <andres@anarazel.de>
Discussion: https://postgr.es/m/uvrtrknj4kdytuboidbhwclo4gxhswwcpgadptsjvjqcluzmah%40brqs62irg4dt
Discussion: https://postgr.es/m/20210223100344.llw5an2aklengrmn@alap3.anarazel.de
Discussion: https://postgr.es/m/stj36ea6yyhoxtqkhpieia2z4krnam7qyetc57rfezgk4zgapf@gcnactj4z56m
Branch
------
master
Details
-------
https://git.postgresql.org/pg/commitdiff/55b454d0e14084c841a034073abbf1a0ea937a45
Modified Files
--------------
doc/src/sgml/config.sgml | 19 +++
src/backend/postmaster/launch_backend.c | 2 +
src/backend/postmaster/pmchild.c | 1 +
src/backend/postmaster/postmaster.c | 174 ++++++++++++++++++++++--
src/backend/storage/aio/Makefile | 1 +
src/backend/storage/aio/meson.build | 1 +
src/backend/storage/aio/method_worker.c | 88 ++++++++++++
src/backend/tcop/postgres.c | 7 +
src/backend/utils/activity/pgstat_backend.c | 1 +
src/backend/utils/activity/pgstat_io.c | 1 +
src/backend/utils/activity/wait_event_names.txt | 1 +
src/backend/utils/init/miscinit.c | 3 +
src/backend/utils/misc/guc_tables.c | 13 ++
src/backend/utils/misc/postgresql.conf.sample | 1 +
src/include/miscadmin.h | 2 +
src/include/pgstat.h | 2 +-
src/include/storage/aio_subsys.h | 4 +
src/include/storage/io_worker.h | 22 +++
src/include/storage/proc.h | 4 +-
src/test/regress/expected/stats.out | 10 +-
20 files changed, 342 insertions(+), 15 deletions(-)
^ permalink raw reply [nested|flat] 5+ messages in thread
* Re: pgsql: aio: Infrastructure for io_method=worker
2025-03-18 16:01 pgsql: aio: Infrastructure for io_method=worker Andres Freund <andres@anarazel.de>
@ 2025-03-18 22:53 ` Tom Lane <tgl@sss.pgh.pa.us>
2025-03-18 23:07 ` Re: pgsql: aio: Infrastructure for io_method=worker Andres Freund <andres@anarazel.de>
0 siblings, 1 reply; 5+ messages in thread
From: Tom Lane @ 2025-03-18 22:53 UTC (permalink / raw)
To: Andres Freund <andres@anarazel.de>; +Cc: pgsql-hackers@lists.postgresql.org
Andres Freund <andres@anarazel.de> writes:
> To allow the number of IO workers to be increased without a restart, we need
> to reserve PGPROC entries for the workers unconditionally. This has been
> judged to be worth the cost. If it turns out to be problematic, we can
> introduce a PGC_POSTMASTER GUC to control the maximum number.
So I see this patch added 32 PGPROCs and hence 32 semaphores to the
system's requirements. Unsurprisingly, this broke OpenBSD/NetBSD
again:
https://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=sawshark&dt=2025-03-18%2016%3A20%3A05
It's probably time to just abandon the idea of being able to run with
only 60 semaphores. I wonder though if we ought to revert 38da05346
and/or 6d0154196 in view of that.
regards, tom lane
^ permalink raw reply [nested|flat] 5+ messages in thread
* Re: pgsql: aio: Infrastructure for io_method=worker
2025-03-18 16:01 pgsql: aio: Infrastructure for io_method=worker Andres Freund <andres@anarazel.de>
2025-03-18 22:53 ` Re: pgsql: aio: Infrastructure for io_method=worker Tom Lane <tgl@sss.pgh.pa.us>
@ 2025-03-18 23:07 ` Andres Freund <andres@anarazel.de>
2025-03-19 00:41 ` Re: pgsql: aio: Infrastructure for io_method=worker Nathan Bossart <nathandbossart@gmail.com>
2025-03-19 01:24 ` Re: pgsql: aio: Infrastructure for io_method=worker Tom Lane <tgl@sss.pgh.pa.us>
0 siblings, 2 replies; 5+ messages in thread
From: Andres Freund @ 2025-03-18 23:07 UTC (permalink / raw)
To: Tom Lane <tgl@sss.pgh.pa.us>; +Cc: pgsql-hackers@lists.postgresql.org
Hi,
On 2025-03-18 18:53:48 -0400, Tom Lane wrote:
> Andres Freund <andres@anarazel.de> writes:
> > To allow the number of IO workers to be increased without a restart, we need
> > to reserve PGPROC entries for the workers unconditionally. This has been
> > judged to be worth the cost. If it turns out to be problematic, we can
> > introduce a PGC_POSTMASTER GUC to control the maximum number.
>
> So I see this patch added 32 PGPROCs and hence 32 semaphores to the
> system's requirements.
Yea - I brought that suspicion up a few times in the thread and in the
resulting discussion it wasn't deemed worth introducing a PGC_POSTMASTER guc
to control the max.
> It's probably time to just abandon the idea of being able to run with
> only 60 semaphores.
Agreed. It's an absurdly outdated OS default configuration. Realistically one
can't run postgres even in a toy scenario without changing it. So the value of
being able to test postgres with the default OS settings doesn't seem that
high. If it were on a very commonly used platform I would think differently,
but net/openbsd ain't that.
> I wonder though if we ought to revert 38da05346 and/or 6d0154196 in view of
> that.
38da05346 doesn't seem to have much value if it doesn't help us run the tests
by default - but it also doesn't really hurt. So, shrug, I guess.
6d0154196 - a higher autovacuum_worker_slots increases resource usage more
than IO workers do, because autovac workers are included in computations like
lock space. But 16 isn't that much either way...
Greetings,
Andres Freund
^ permalink raw reply [nested|flat] 5+ messages in thread
* Re: pgsql: aio: Infrastructure for io_method=worker
2025-03-18 16:01 pgsql: aio: Infrastructure for io_method=worker Andres Freund <andres@anarazel.de>
2025-03-18 22:53 ` Re: pgsql: aio: Infrastructure for io_method=worker Tom Lane <tgl@sss.pgh.pa.us>
2025-03-18 23:07 ` Re: pgsql: aio: Infrastructure for io_method=worker Andres Freund <andres@anarazel.de>
@ 2025-03-19 00:41 ` Nathan Bossart <nathandbossart@gmail.com>
1 sibling, 0 replies; 5+ messages in thread
From: Nathan Bossart @ 2025-03-19 00:41 UTC (permalink / raw)
To: Andres Freund <andres@anarazel.de>; +Cc: Tom Lane <tgl@sss.pgh.pa.us>; pgsql-hackers@lists.postgresql.org
On Tue, Mar 18, 2025 at 07:07:53PM -0400, Andres Freund wrote:
> On 2025-03-18 18:53:48 -0400, Tom Lane wrote:
>> It's probably time to just abandon the idea of being able to run with
>> only 60 semaphores.
>
> Agreed. It's an absurdly outdated OS default configuration. Realistically one
> can't run postgres even in a toy scenario without changing it. So the value of
> being able to test postgres with the default OS settings doesn't seem that
> high. If it were on a very commonly used platform I would think differently,
> but net/openbsd ain't that.
+1. The platforms with low defaults do force us to think carefully about
increasing the default requirements for Postgres, but I don't think that's
a strong enough argument for the maintenance effort.
--
nathan
^ permalink raw reply [nested|flat] 5+ messages in thread
* Re: pgsql: aio: Infrastructure for io_method=worker
2025-03-18 16:01 pgsql: aio: Infrastructure for io_method=worker Andres Freund <andres@anarazel.de>
2025-03-18 22:53 ` Re: pgsql: aio: Infrastructure for io_method=worker Tom Lane <tgl@sss.pgh.pa.us>
2025-03-18 23:07 ` Re: pgsql: aio: Infrastructure for io_method=worker Andres Freund <andres@anarazel.de>
@ 2025-03-19 01:24 ` Tom Lane <tgl@sss.pgh.pa.us>
1 sibling, 0 replies; 5+ messages in thread
From: Tom Lane @ 2025-03-19 01:24 UTC (permalink / raw)
To: Andres Freund <andres@anarazel.de>; +Cc: pgsql-hackers@lists.postgresql.org
Andres Freund <andres@anarazel.de> writes:
> On 2025-03-18 18:53:48 -0400, Tom Lane wrote:
>> I wonder though if we ought to revert 38da05346 and/or 6d0154196 in view of
>> that.
> 38da05346 doesn't seem to have much value if it doesn't help us run the tests
> by default - but it also doesn't really hurt. So, shrug, I guess.
> 6d0154196 - a higher autovacuum_worker_slots increases resource usage more
> than IO workers do, because autovac workers are included in computations like
> lock space. But 16 isn't that much either way...
IMV the argument for reverting either would basically be simplicity.
It's not even so much the code, as the comments defending these odd
looking choices. Future hackers will read those and wonder if the
arguments still apply --- and the answer will be "no".
38da05346 was back-patched into 17, and I'd leave it as-is there,
since it was holding the line on "can start with 60 semaphores"
for that branch. But once we've lost that battle, it's hard to
see what it's doing for us.
regards, tom lane
^ permalink raw reply [nested|flat] 5+ messages in thread
end of thread, other threads:[~2025-03-19 01:24 UTC | newest]
Thread overview: 5+ messages (download: mbox mbox.gz follow: Atom feed)
-- links below jump to the message on this page --
2025-03-18 16:01 pgsql: aio: Infrastructure for io_method=worker Andres Freund <andres@anarazel.de>
2025-03-18 22:53 ` Tom Lane <tgl@sss.pgh.pa.us>
2025-03-18 23:07 ` Andres Freund <andres@anarazel.de>
2025-03-19 00:41 ` Nathan Bossart <nathandbossart@gmail.com>
2025-03-19 01:24 ` Tom Lane <tgl@sss.pgh.pa.us>
This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox