pg.ddx.io pgsql-hackers@postgresql.org mailing list archive
help / color / mirror / Atom feedRe: AIO v2.3
13+ messages / 5 participants
[nested] [flat]
* Re: AIO v2.3
@ 2025-01-23 04:29 Andres Freund <andres@anarazel.de>
2025-02-06 10:50 ` Re: AIO v2.3 Jakub Wartak <jakub.wartak@enterprisedb.com>
2025-02-10 22:48 ` Re: AIO v2.3 Thomas Munro <thomas.munro@gmail.com>
0 siblings, 2 replies; 13+ messages in thread
From: Andres Freund @ 2025-01-23 04:29 UTC (permalink / raw)
To: pgsql-hackers; +Cc: Heikki Linnakangas <hlinnaka@iki.fi>; Robert Haas <robertmhaas@gmail.com>; Noah Misch <noah@leadboat.com>; Jakub Wartak <jakub.wartak@enterprisedb.com>
Hi,
Attached is v2.3.
There are a lot of changes - primarily renaming things based on on-list and
off-list feedback. But also some other things
Functional:
- Added pg_aios view
- md.c registering sync requests, that was previously omitted
- This triggered stats issues during shutdown, as it can lead to IO workers
emitting stats in some corner cases. I've written a patch series to
address this [1]. For now I've included them in this patchset, but I would
like to push the reordering patches soon.
- Testing error handling for temp table IO made me realize that the previous
pattern of just tracking the refcount held by the IO subsystem in the
LocalRefCount array leads to spurious buffer leak warnings [2]. I attached
a prototype patch to deal with this by bringing localbuf.c more in line with
bufmgr.c, but it needs some cleanup.
That's in v2.3-0020-WIP-localbuf-Track-pincount-in-BufferDesc-as-we.patch
- Wait for all IOs to finish during shutdown. This is primarily required to
ensure there aren't IOs initiated by a prior "owner" of a ProcNumber when a
new backend starts. But there are also some kernels that don't like when
exiting while IO is in flight.
- Re-armed local completion callbacks, they're required for correctness of
temporary table IO
- Added a bunch of central debug helpers that only lead to output if
PGAIO_VERBOSE is defined. That did make code a good bit more readable.
Polishing:
- Lots of copy editing, a lot of it thanks to feedback by Noah and Heikki
- Renamed the previous concept of a "subject" of an IO (i.e. what the IO is
executed on, an smgr relation, a WAL file, ...) to "target". I'm not in
love with that name, but I went through dozens of variations, and it does
seem better than subject.
Not sure anymore how I ended up with subject, it's grammatically off and not
very descriptive to boot.
- Renamed "PgAioHandleRef" and related functions to
PgAioWaitRef/pgaio_wref_*(), that seems a lot more descriptive.
- Renamed pgaio_io_get() to pgaio_io_acquire()
- Renamed the IO handle states (PREPARED to STAGED, IN_FLIGHT to SUBMITTED,
REAPED to COMPLETED_IO).
Particularly the various COMPLETED state names aren't necessarily final,
I've been debating a bunch of variations with Thomas and Robert
- Renamed aio_ref.h to aio_types.h, moved a few more types into it.
- Renamed completion callbacks to not use "shared" anymore - ->prepare was not
really shared and now local callbacks are back (in a restricted form).
s/PgAioHandleSharedCallback/PgAioHandleCallback/
s/pgaio_io_add_shared_cb/pgaio_io_register_callbacks/
Not entirely sure *register_callbacks is the best, happy to adjust.
- Renamed the ->error IO handle callback to ->report
Also renamed s/pgaio_result_log/pgaio_result_report/g
- Renamed the ->prepare IO handle callback to ->stage
Also renamed s/pgaio_result_log/pgaio_result_report/g
- Partially addressed request to reorder aio/README.md
- Determine shared memory allocation size with PG_IOV_MAX not io_combine_limit
io_combine_limit is USERSET, so it's not correct to use it for shmem
allocations. I chose PG_IOV_MAX instead of MAX_IO_COMBINE_LIMIT because this
is a more generic limit than bufmgr.c IO.
- Prefix PgAio* enums with PGAIO_, global variables with pgaio_*
- Split out callback related cod from aio_subject.c (now aio_target.c) into
aio_callback.c. The target specific code is rather small, so this makes a
lot more sense.
- Distributed functions into more appropriate .c files, documented the choice
in aio.h, reorder them
- Disowned lwlock: More consistent naming, reduce diff size, resume interrupts
Heikki asked to clear ->owner when disowning the lock - but we currently
*never* clear it doesn't seem right to do so when disowning the lock.
- IO data that can be set on a handle (to e.g. transport an array of Buffers
to the completion callbacks) is now done with
pgaio_io_(get|set)_handle_data(). Mainly to distinguish it from data that's
actually the target/source of a read/write.
Heikki suggested to make this per-callback data, but I don't think there's
currently a use case for that, and it'd add a fair bit of memory overhead. I
added a comment documenting this.
- Lots of other cleanups, added comments and the like
Todo:
- Reorder README further
- Make per backend state not indexed by ProcNumber, as that requires reserving
per-backend state for IO workers, which will never need them
- Clean up localbuf.c "preparation" patches
- Add more tests - I had hoped to get to this, but got sidetracked with a
bunch of things I found while testing
- I started looking into having a distinct type for the public pgaio_io_*
related functions that can be used just by the issuer of the IO. It does
make things a bit easier to understand, but also complicates naming. Not
sure if it's worth it yet.
- Need to define (and test) the behavior when an IO worker fails to reopen the
file for an IO
- Heikki doesn't love pgaio_submit_staged(), suggested pgaio_kick_staged() or
such. I don't love that name though.
- There's some duplicated code in aio_callback.c, it'd be nice to deduplicate
the callback invaction of the different callbacks
- Local callbacks are triggered from within pgaio_io_reclaim(), that's not
exactly pretty. But it's currently the most central place to deal with the
case of IOs for which the shared completion callback was called in another
backend.
- As Jakub suggested (below [3]), when io_method=io_uring is used, we can run
of out file descriptors much more easily. At the very least we need a good
error message, perhaps also some rlimit adjusting (probably as a second
step, if so).
- Thomas is working on the read_stream.c <-> bufmgr.c integration piece
- Start to write docs adjustments
[1] https://postgr.es/m/kgng5nrvnlv335evmsuvpnh354rw7qyazl73kdysev2cr2v5zu%40m3cfzxicm5kp
[2] https://postgr.es/m/j6hny5ivrfqw356ugoy3ti5ccadamluekxod4k6amao5snew6c%40t5h3bwhrgfqx
[3] https://postgr.es/m/tp63m6tcbi7mmsjlqgxd55sghhwvjxp3mkgeljffkbaujezvdl%40fvmdr3c6uhat
Attachments:
[text/x-diff] v2.3-0001-checkpointer-Request-checkpoint-via-latch-inste.patch (4.6K, ../../p5flybfcjrzgpgwcnzkxsp3xdw3czfoafkvks6g7en6ewmvymh@wrz7z2vwwc7c/2-v2.3-0001-checkpointer-Request-checkpoint-via-latch-inste.patch)
download | inline diff:
From 369d7d8f81f26bdaf4097c6acd09b58cc8f8d151 Mon Sep 17 00:00:00 2001
From: Andres Freund <andres@anarazel.de>
Date: Fri, 10 Jan 2025 11:11:40 -0500
Subject: [PATCH v2.3 01/30] checkpointer: Request checkpoint via latch instead
of signal
The main reason for this is that a future commit would like to use SIGINT for
another purpose. But it's also a tad nicer and tad more efficient to use
SetLatch(), as it avoids a signal when checkpointer already is busy.
Reviewed-by: Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
Reviewed-by: Nazir Bilal Yavuz <byavuz81@gmail.com>
Discussion: https://postgr.es/m/kgng5nrvnlv335evmsuvpnh354rw7qyazl73kdysev2cr2v5zu@m3cfzxicm5kp
---
src/backend/postmaster/checkpointer.c | 60 +++++++++------------------
1 file changed, 19 insertions(+), 41 deletions(-)
diff --git a/src/backend/postmaster/checkpointer.c b/src/backend/postmaster/checkpointer.c
index 9bfd0fd665c..dd2c8376c6e 100644
--- a/src/backend/postmaster/checkpointer.c
+++ b/src/backend/postmaster/checkpointer.c
@@ -159,9 +159,6 @@ static bool ImmediateCheckpointRequested(void);
static bool CompactCheckpointerRequestQueue(void);
static void UpdateSharedMemoryConfig(void);
-/* Signal handlers */
-static void ReqCheckpointHandler(SIGNAL_ARGS);
-
/*
* Main entry point for checkpointer process
@@ -191,7 +188,7 @@ CheckpointerMain(char *startup_data, size_t startup_data_len)
* tell us it's okay to shut down (via SIGUSR2).
*/
pqsignal(SIGHUP, SignalHandlerForConfigReload);
- pqsignal(SIGINT, ReqCheckpointHandler); /* request checkpoint */
+ pqsignal(SIGINT, SIG_IGN);
pqsignal(SIGTERM, SIG_IGN); /* ignore SIGTERM */
/* SIGQUIT handler was already set up by InitPostmasterChild */
pqsignal(SIGALRM, SIG_IGN);
@@ -860,23 +857,6 @@ IsCheckpointOnSchedule(double progress)
}
-/* --------------------------------
- * signal handler routines
- * --------------------------------
- */
-
-/* SIGINT: set flag to run a normal checkpoint right away */
-static void
-ReqCheckpointHandler(SIGNAL_ARGS)
-{
- /*
- * The signaling process should have set ckpt_flags nonzero, so all we
- * need do is ensure that our main loop gets kicked out of any wait.
- */
- SetLatch(MyLatch);
-}
-
-
/* --------------------------------
* communication with backends
* --------------------------------
@@ -990,38 +970,36 @@ RequestCheckpoint(int flags)
SpinLockRelease(&CheckpointerShmem->ckpt_lck);
/*
- * Send signal to request checkpoint. It's possible that the checkpointer
- * hasn't started yet, or is in process of restarting, so we will retry a
- * few times if needed. (Actually, more than a few times, since on slow
- * or overloaded buildfarm machines, it's been observed that the
- * checkpointer can take several seconds to start.) However, if not told
- * to wait for the checkpoint to occur, we consider failure to send the
- * signal to be nonfatal and merely LOG it. The checkpointer should see
- * the request when it does start, with or without getting a signal.
+ * Set checkpointer's latch to request checkpoint. It's possible that the
+ * checkpointer hasn't started yet, so we will retry a few times if
+ * needed. (Actually, more than a few times, since on slow or overloaded
+ * buildfarm machines, it's been observed that the checkpointer can take
+ * several seconds to start.) However, if not told to wait for the
+ * checkpoint to occur, we consider failure to set the latch to be
+ * nonfatal and merely LOG it. The checkpointer should see the request
+ * when it does start, with or without the SetLatch().
*/
#define MAX_SIGNAL_TRIES 600 /* max wait 60.0 sec */
for (ntries = 0;; ntries++)
{
- if (CheckpointerShmem->checkpointer_pid == 0)
+ volatile PROC_HDR *procglobal = ProcGlobal;
+ ProcNumber checkpointerProc = procglobal->checkpointerProc;
+
+ if (checkpointerProc == INVALID_PROC_NUMBER)
{
if (ntries >= MAX_SIGNAL_TRIES || !(flags & CHECKPOINT_WAIT))
{
elog((flags & CHECKPOINT_WAIT) ? ERROR : LOG,
- "could not signal for checkpoint: checkpointer is not running");
- break;
- }
- }
- else if (kill(CheckpointerShmem->checkpointer_pid, SIGINT) != 0)
- {
- if (ntries >= MAX_SIGNAL_TRIES || !(flags & CHECKPOINT_WAIT))
- {
- elog((flags & CHECKPOINT_WAIT) ? ERROR : LOG,
- "could not signal for checkpoint: %m");
+ "could not notify checkpoint: checkpointer is not running");
break;
}
}
else
- break; /* signal sent successfully */
+ {
+ SetLatch(&GetPGProcByNumber(checkpointerProc)->procLatch);
+ /* notified successfully */
+ break;
+ }
CHECK_FOR_INTERRUPTS();
pg_usleep(100000L); /* wait 0.1 sec, then retry */
--
2.48.1.76.g4e746b1a31.dirty
[text/x-diff] v2.3-0002-postmaster-Don-t-open-code-TerminateChildren-in.patch (2.9K, ../../p5flybfcjrzgpgwcnzkxsp3xdw3czfoafkvks6g7en6ewmvymh@wrz7z2vwwc7c/3-v2.3-0002-postmaster-Don-t-open-code-TerminateChildren-in.patch)
download | inline diff:
From ea4f243a510b7151f0853b8b984fc81070c618c2 Mon Sep 17 00:00:00 2001
From: Andres Freund <andres@anarazel.de>
Date: Mon, 13 Jan 2025 23:20:25 -0500
Subject: [PATCH v2.3 02/30] postmaster: Don't open-code TerminateChildren() in
HandleChildCrash()
After removing the duplication no user of sigquit_child() remains, therefore
remove it.
Author:
Reviewed-by:
Discussion: https://postgr.es/m/
Backpatch:
---
src/backend/postmaster/postmaster.c | 42 +++--------------------------
1 file changed, 4 insertions(+), 38 deletions(-)
diff --git a/src/backend/postmaster/postmaster.c b/src/backend/postmaster/postmaster.c
index 5f615d0f605..8153edc446c 100644
--- a/src/backend/postmaster/postmaster.c
+++ b/src/backend/postmaster/postmaster.c
@@ -424,7 +424,6 @@ static int BackendStartup(ClientSocket *client_sock);
static void report_fork_failure_to_client(ClientSocket *client_sock, int errnum);
static CAC_state canAcceptConnections(BackendType backend_type);
static void signal_child(PMChild *pmchild, int signal);
-static void sigquit_child(PMChild *pmchild);
static bool SignalChildren(int signal, BackendTypeMask targetMask);
static void TerminateChildren(int signal);
static int CountChildren(BackendTypeMask targetMask);
@@ -2699,32 +2698,12 @@ HandleChildCrash(int pid, int exitstatus, const char *procname)
/*
* Signal all other child processes to exit. The crashed process has
* already been removed from ActiveChildList.
+ *
+ * We could exclude dead-end children here, but at least when sending
+ * SIGABRT it seems better to include them.
*/
if (take_action)
- {
- dlist_iter iter;
-
- dlist_foreach(iter, &ActiveChildList)
- {
- PMChild *bp = dlist_container(PMChild, elem, iter.cur);
-
- /* We do NOT restart the syslogger */
- if (bp == SysLoggerPMChild)
- continue;
-
- if (bp == StartupPMChild)
- StartupStatus = STARTUP_SIGNALED;
-
- /*
- * This backend is still alive. Unless we did so already, tell it
- * to commit hara-kiri.
- *
- * We could exclude dead-end children here, but at least when
- * sending SIGABRT it seems better to include them.
- */
- sigquit_child(bp);
- }
- }
+ TerminateChildren(send_abort_for_crash ? SIGABRT : SIGQUIT);
if (Shutdown != ImmediateShutdown)
FatalError = true;
@@ -3347,19 +3326,6 @@ signal_child(PMChild *pmchild, int signal)
#endif
}
-/*
- * Convenience function for killing a child process after a crash of some
- * other child process. We apply send_abort_for_crash to decide which signal
- * to send. Normally it's SIGQUIT -- and most other comments in this file are
- * written on the assumption that it is -- but developers might prefer to use
- * SIGABRT to collect per-child core dumps.
- */
-static void
-sigquit_child(PMChild *pmchild)
-{
- signal_child(pmchild, (send_abort_for_crash ? SIGABRT : SIGQUIT));
-}
-
/*
* Send a signal to the targeted children.
*/
--
2.48.1.76.g4e746b1a31.dirty
[text/x-diff] v2.3-0003-postmaster-Don-t-repeatedly-transition-to-crash.patch (2.4K, ../../p5flybfcjrzgpgwcnzkxsp3xdw3czfoafkvks6g7en6ewmvymh@wrz7z2vwwc7c/4-v2.3-0003-postmaster-Don-t-repeatedly-transition-to-crash.patch)
download | inline diff:
From ae79a4158d88ab0fbe78df9ab6ec15be3152343a Mon Sep 17 00:00:00 2001
From: Andres Freund <andres@anarazel.de>
Date: Mon, 13 Jan 2025 23:30:46 -0500
Subject: [PATCH v2.3 03/30] postmaster: Don't repeatedly transition to
crashing state
Previously HandleChildCrash() skipped logging and signalling child exits if
already in an immediate shutdown or FatalError, but still transitioned server
state in response to a crash. That's redundant.
To make it easier to combine different paths for entering FatalError state,
only do so once.
Author:
Reviewed-by:
Discussion: https://postgr.es/m/
Backpatch:
---
src/backend/postmaster/postmaster.c | 19 +++++++------------
1 file changed, 7 insertions(+), 12 deletions(-)
diff --git a/src/backend/postmaster/postmaster.c b/src/backend/postmaster/postmaster.c
index 8153edc446c..939b1b2ef82 100644
--- a/src/backend/postmaster/postmaster.c
+++ b/src/backend/postmaster/postmaster.c
@@ -2676,8 +2676,6 @@ CleanupBackend(PMChild *bp,
static void
HandleChildCrash(int pid, int exitstatus, const char *procname)
{
- bool take_action;
-
/*
* We only log messages and send signals if this is the first process
* crash and we're not doing an immediate shutdown; otherwise, we're only
@@ -2685,15 +2683,13 @@ HandleChildCrash(int pid, int exitstatus, const char *procname)
* signaled children, nonzero exit status is to be expected, so don't
* clutter log.
*/
- take_action = !FatalError && Shutdown != ImmediateShutdown;
+ if (FatalError || Shutdown == ImmediateShutdown)
+ return;
- if (take_action)
- {
- LogChildExit(LOG, procname, pid, exitstatus);
- ereport(LOG,
- (errmsg("terminating any other active server processes")));
- SetQuitSignalReason(PMQUIT_FOR_CRASH);
- }
+ LogChildExit(LOG, procname, pid, exitstatus);
+ ereport(LOG,
+ (errmsg("terminating any other active server processes")));
+ SetQuitSignalReason(PMQUIT_FOR_CRASH);
/*
* Signal all other child processes to exit. The crashed process has
@@ -2702,8 +2698,7 @@ HandleChildCrash(int pid, int exitstatus, const char *procname)
* We could exclude dead-end children here, but at least when sending
* SIGABRT it seems better to include them.
*/
- if (take_action)
- TerminateChildren(send_abort_for_crash ? SIGABRT : SIGQUIT);
+ TerminateChildren(send_abort_for_crash ? SIGABRT : SIGQUIT);
if (Shutdown != ImmediateShutdown)
FatalError = true;
--
2.48.1.76.g4e746b1a31.dirty
[text/x-diff] v2.3-0004-postmaster-Move-code-to-switch-into-FatalError-.patch (4.2K, ../../p5flybfcjrzgpgwcnzkxsp3xdw3czfoafkvks6g7en6ewmvymh@wrz7z2vwwc7c/5-v2.3-0004-postmaster-Move-code-to-switch-into-FatalError-.patch)
download | inline diff:
From c816b542699fdde710bbf5a909be45ffc9b8488e Mon Sep 17 00:00:00 2001
From: Andres Freund <andres@anarazel.de>
Date: Tue, 14 Jan 2025 00:57:12 -0500
Subject: [PATCH v2.3 04/30] postmaster: Move code to switch into FatalError
state into function
There are two places switching to FatalError mode, behaving somewhat
differently. An upcoming commit will introduce a third. That doesn't seem seem
like a good idea.
This commit just moves the FatalError related code from HandleChildCrash()
into its own function, a subsequent commit will evolve the state machine
change to be suitable for other callers.
Author:
Reviewed-by:
Discussion: https://postgr.es/m/
Backpatch:
---
src/backend/postmaster/postmaster.c | 70 +++++++++++++++++++----------
1 file changed, 46 insertions(+), 24 deletions(-)
diff --git a/src/backend/postmaster/postmaster.c b/src/backend/postmaster/postmaster.c
index 939b1b2ef82..13d49eecd22 100644
--- a/src/backend/postmaster/postmaster.c
+++ b/src/backend/postmaster/postmaster.c
@@ -2665,40 +2665,29 @@ CleanupBackend(PMChild *bp,
}
/*
- * HandleChildCrash -- cleanup after failed backend, bgwriter, checkpointer,
- * walwriter, autovacuum, archiver, slot sync worker, or background worker.
- *
- * The objectives here are to clean up our local state about the child
- * process, and to signal all other remaining children to quickdie.
- *
- * The caller has already released its PMChild slot.
+ * Transition into FatalError state, in response to something bad having
+ * happened. Commonly the caller will have logged the reason for entering
+ * FatalError state.
*/
static void
-HandleChildCrash(int pid, int exitstatus, const char *procname)
+HandleFatalError(QuitSignalReason reason, bool consider_sigabrt)
{
- /*
- * We only log messages and send signals if this is the first process
- * crash and we're not doing an immediate shutdown; otherwise, we're only
- * here to update postmaster's idea of live processes. If we have already
- * signaled children, nonzero exit status is to be expected, so don't
- * clutter log.
- */
- if (FatalError || Shutdown == ImmediateShutdown)
- return;
+ int sigtosend;
+
+ SetQuitSignalReason(reason);
- LogChildExit(LOG, procname, pid, exitstatus);
- ereport(LOG,
- (errmsg("terminating any other active server processes")));
- SetQuitSignalReason(PMQUIT_FOR_CRASH);
+ if (consider_sigabrt && send_abort_for_crash)
+ sigtosend = SIGABRT;
+ else
+ sigtosend = SIGQUIT;
/*
- * Signal all other child processes to exit. The crashed process has
- * already been removed from ActiveChildList.
+ * Signal all other child processes to exit.
*
* We could exclude dead-end children here, but at least when sending
* SIGABRT it seems better to include them.
*/
- TerminateChildren(send_abort_for_crash ? SIGABRT : SIGQUIT);
+ TerminateChildren(sigtosend);
if (Shutdown != ImmediateShutdown)
FatalError = true;
@@ -2719,6 +2708,39 @@ HandleChildCrash(int pid, int exitstatus, const char *procname)
AbortStartTime = time(NULL);
}
+/*
+ * HandleChildCrash -- cleanup after failed backend, bgwriter, checkpointer,
+ * walwriter, autovacuum, archiver, slot sync worker, or background worker.
+ *
+ * The objectives here are to clean up our local state about the child
+ * process, and to signal all other remaining children to quickdie.
+ *
+ * The caller has already released its PMChild slot.
+ */
+static void
+HandleChildCrash(int pid, int exitstatus, const char *procname)
+{
+ /*
+ * We only log messages and send signals if this is the first process
+ * crash and we're not doing an immediate shutdown; otherwise, we're only
+ * here to update postmaster's idea of live processes. If we have already
+ * signaled children, nonzero exit status is to be expected, so don't
+ * clutter log.
+ */
+ if (FatalError || Shutdown == ImmediateShutdown)
+ return;
+
+ LogChildExit(LOG, procname, pid, exitstatus);
+ ereport(LOG,
+ (errmsg("terminating any other active server processes")));
+
+ /*
+ * Switch into error state. The crashed process has already been removed
+ * from ActiveChildList.
+ */
+ HandleFatalError(PMQUIT_FOR_CRASH, true);
+}
+
/*
* Log the death of a child process.
*/
--
2.48.1.76.g4e746b1a31.dirty
[text/x-diff] v2.3-0005-WIP-postmaster-Commonalize-FatalError-paths.patch (3.4K, ../../p5flybfcjrzgpgwcnzkxsp3xdw3czfoafkvks6g7en6ewmvymh@wrz7z2vwwc7c/6-v2.3-0005-WIP-postmaster-Commonalize-FatalError-paths.patch)
download | inline diff:
From 97b4983b1443d03525b0565eb104b359a43044af Mon Sep 17 00:00:00 2001
From: Andres Freund <andres@anarazel.de>
Date: Tue, 14 Jan 2025 00:25:01 -0500
Subject: [PATCH v2.3 05/30] WIP: postmaster: Commonalize FatalError paths
This includes some behavioural changes:
- Previously PM_WAIT_XLOG_ARCHIVAL wasn't handled in HandleFatalError(), that
doesn't seem quite right.
- Failure to fork checkpointer now transitions through PM_WAIT_BACKENDS, like
child crashes. That's not necessarily great, but...
Author:
Reviewed-by:
Discussion: https://postgr.es/m/
Backpatch:
---
src/backend/postmaster/postmaster.c | 61 +++++++++++++++++++++++------
1 file changed, 49 insertions(+), 12 deletions(-)
diff --git a/src/backend/postmaster/postmaster.c b/src/backend/postmaster/postmaster.c
index 13d49eecd22..41f2bbc214c 100644
--- a/src/backend/postmaster/postmaster.c
+++ b/src/backend/postmaster/postmaster.c
@@ -2693,12 +2693,47 @@ HandleFatalError(QuitSignalReason reason, bool consider_sigabrt)
FatalError = true;
/* We now transit into a state of waiting for children to die */
- if (pmState == PM_RECOVERY ||
- pmState == PM_HOT_STANDBY ||
- pmState == PM_RUN ||
- pmState == PM_STOP_BACKENDS ||
- pmState == PM_WAIT_XLOG_SHUTDOWN)
- UpdatePMState(PM_WAIT_BACKENDS);
+ switch (pmState)
+ {
+ case PM_INIT:
+ /* shouldn't have any children */
+ Assert(false);
+ break;
+ case PM_STARTUP:
+ /* should have been handled in process_pm_child_exit */
+ Assert(false);
+ break;
+
+ /* wait for children to die */
+ case PM_RECOVERY:
+ case PM_HOT_STANDBY:
+ case PM_RUN:
+ case PM_STOP_BACKENDS:
+ UpdatePMState(PM_WAIT_BACKENDS);
+ break;
+
+ case PM_WAIT_BACKENDS:
+ /* there might be more backends to wait for */
+ break;
+
+ case PM_WAIT_XLOG_SHUTDOWN:
+ case PM_WAIT_XLOG_ARCHIVAL:
+
+ /*
+ * Note that we switch *back* to PM_WAIT_BACKENDS here. This way
+ * the PM_WAIT_BACKENDS && FatalError code in
+ * PostmasterStateMachine does not have to be duplicated.
+ *
+ * XXX: This seems rather ugly, but it's not obvious if the
+ * alternative is better.
+ */
+ UpdatePMState(PM_WAIT_BACKENDS);
+ break;
+
+ case PM_WAIT_DEAD_END:
+ case PM_NO_CHILDREN:
+ break;
+ }
/*
* .. and if this doesn't happen quickly enough, now the clock is ticking
@@ -2836,6 +2871,9 @@ PostmasterStateMachine(void)
* PM_WAIT_BACKENDS, but we signal the processes first, before waiting for
* them. Treating it as a distinct pmState allows us to share this code
* across multiple shutdown code paths.
+ *
+ * Note that HandleFatalError() switches to PM_WAIT_BACKENDS even if we
+ * were, before the fatal error, in a "more advanced" state.
*/
if (pmState == PM_STOP_BACKENDS || pmState == PM_WAIT_BACKENDS)
{
@@ -2967,13 +3005,12 @@ PostmasterStateMachine(void)
* We don't consult send_abort_for_crash here, as it's
* unlikely that dumping cores would illuminate the reason
* for checkpointer fork failure.
+ *
+ * XXX: Is it worth inventing a different PMQUIT value
+ * that signals that the cluster is in a bad state,
+ * without a process having crashed?
*/
- FatalError = true;
- UpdatePMState(PM_WAIT_DEAD_END);
- ConfigurePostmasterWaitSet(false);
-
- /* Kill the walsenders and archiver too */
- SignalChildren(SIGQUIT, btmask_all_except(B_LOGGER));
+ HandleFatalError(PMQUIT_FOR_CRASH, false);
}
}
}
--
2.48.1.76.g4e746b1a31.dirty
[text/x-diff] v2.3-0006-postmaster-Adjust-which-processes-we-expect-to-.patch (2.8K, ../../p5flybfcjrzgpgwcnzkxsp3xdw3czfoafkvks6g7en6ewmvymh@wrz7z2vwwc7c/7-v2.3-0006-postmaster-Adjust-which-processes-we-expect-to-.patch)
download | inline diff:
From 8f44b56322e97dbf7f5e8e514c8e6d3e603b73bd Mon Sep 17 00:00:00 2001
From: Andres Freund <andres@anarazel.de>
Date: Tue, 14 Jan 2025 00:31:03 -0500
Subject: [PATCH v2.3 06/30] postmaster: Adjust which processes we expect to
have exited
Comments and code stated that we expect checkpointer to have been signalled in
case of immediate shutdown / fatal errors, but didn't treat archiver and
walsenders the same. That doesn't seem right.
I had started digging through the history to see where this oddity was
introduced, but it's not the fault of a single commit.
Instead treat archiver, checkpointer, and walsenders the same.
Author:
Reviewed-by:
Discussion: https://postgr.es/m/
Backpatch:
---
src/backend/postmaster/postmaster.c | 28 ++++++++++++++++++----------
1 file changed, 18 insertions(+), 10 deletions(-)
diff --git a/src/backend/postmaster/postmaster.c b/src/backend/postmaster/postmaster.c
index 41f2bbc214c..54801a32609 100644
--- a/src/backend/postmaster/postmaster.c
+++ b/src/backend/postmaster/postmaster.c
@@ -2906,16 +2906,20 @@ PostmasterStateMachine(void)
/*
* If we are doing crash recovery or an immediate shutdown then we
- * expect the checkpointer to exit as well, otherwise not.
+ * expect archiver, checkpointer and walsender to exit as well,
+ * otherwise not.
*/
if (FatalError || Shutdown >= ImmediateShutdown)
- targetMask = btmask_add(targetMask, B_CHECKPOINTER);
+ targetMask = btmask_add(targetMask,
+ B_CHECKPOINTER,
+ B_ARCHIVER,
+ B_WAL_SENDER);
/*
- * Walsenders and archiver will continue running; they will be
- * terminated later after writing the checkpoint record. We also let
- * dead-end children to keep running for now. The syslogger process
- * exits last.
+ * Normally walsenders and archiver will continue running; they will
+ * be terminated later after writing the checkpoint record. We also
+ * let dead-end children to keep running for now. The syslogger
+ * process exits last.
*
* This assertion checks that we have covered all backend types,
* either by including them in targetMask, or by noting here that they
@@ -2926,13 +2930,17 @@ PostmasterStateMachine(void)
BackendTypeMask remainMask = BTYPE_MASK_NONE;
remainMask = btmask_add(remainMask,
- B_WAL_SENDER,
- B_ARCHIVER,
B_DEAD_END_BACKEND,
B_LOGGER);
- /* checkpointer may or may not be in targetMask already */
- remainMask = btmask_add(remainMask, B_CHECKPOINTER);
+ /*
+ * Archiver, checkpointer and walsender may or may not be in
+ * targetMask already.
+ */
+ remainMask = btmask_add(remainMask,
+ B_ARCHIVER,
+ B_CHECKPOINTER,
+ B_WAL_SENDER);
/* these are not real postmaster children */
remainMask = btmask_add(remainMask,
--
2.48.1.76.g4e746b1a31.dirty
[text/x-diff] v2.3-0007-Change-shutdown-sequence-to-terminate-checkpoin.patch (18.4K, ../../p5flybfcjrzgpgwcnzkxsp3xdw3czfoafkvks6g7en6ewmvymh@wrz7z2vwwc7c/8-v2.3-0007-Change-shutdown-sequence-to-terminate-checkpoin.patch)
download | inline diff:
From ecb9f5995b5f0b38b01c8b86168aa848c9459c83 Mon Sep 17 00:00:00 2001
From: Andres Freund <andres@anarazel.de>
Date: Tue, 14 Jan 2025 01:18:42 -0500
Subject: [PATCH v2.3 07/30] Change shutdown sequence to terminate checkpointer
last
The main motivation for this change is to have a process that can serialize
stats after all other processes have terminated. Serializing stats already
happens in checkpointer, even though walsenders can be active longer.
The only reason the current state does not actively cause problems is that
walsender currently generate any stats. However, there is a patch to change
that.
Another need for this change originates in the AIO patchset, where IO
workers (which, in some edge cases, can emit stats of their own) need to run
while the shutdown checkpoint is being written.
This commit changes the shutdown sequence so checkpointer is signalled (via
SIGINT) to trigger writing the shutdown checkpoint without terminating
it. Once checkpointer wrote the checkpoint it will wait for a termination
signal (SIGUSR2, as before).
Postmaster now triggers the shutdown checkpoint via SIGINT, where we
previously did so by terminating checkpointer. Checkpointer now is terminated
after all children other than dead-end ones have been terminated, tracked
using the new PM_WAIT_CHECKPOINTER PMState.
Reviewed-by: Heikki Linnakangas <hlinnaka@iki.fi>
Reviewed-by: Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
Discussion: https://postgr.es/m/kgng5nrvnlv335evmsuvpnh354rw7qyazl73kdysev2cr2v5zu@m3cfzxicm5kp
---
src/include/storage/pmsignal.h | 3 +-
src/backend/postmaster/checkpointer.c | 125 +++++++++++----
src/backend/postmaster/postmaster.c | 143 +++++++++++++-----
.../utils/activity/wait_event_names.txt | 1 +
4 files changed, 200 insertions(+), 72 deletions(-)
diff --git a/src/include/storage/pmsignal.h b/src/include/storage/pmsignal.h
index 3fbe5bf1136..d84a383047e 100644
--- a/src/include/storage/pmsignal.h
+++ b/src/include/storage/pmsignal.h
@@ -40,9 +40,10 @@ typedef enum
PMSIGNAL_BACKGROUND_WORKER_CHANGE, /* background worker state change */
PMSIGNAL_START_WALRECEIVER, /* start a walreceiver */
PMSIGNAL_ADVANCE_STATE_MACHINE, /* advance postmaster's state machine */
+ PMSIGNAL_XLOG_IS_SHUTDOWN, /* ShutdownXLOG() completed */
} PMSignalReason;
-#define NUM_PMSIGNALS (PMSIGNAL_ADVANCE_STATE_MACHINE+1)
+#define NUM_PMSIGNALS (PMSIGNAL_XLOG_IS_SHUTDOWN+1)
/*
* Reasons why the postmaster would send SIGQUIT to its children.
diff --git a/src/backend/postmaster/checkpointer.c b/src/backend/postmaster/checkpointer.c
index dd2c8376c6e..767bf9f5cf8 100644
--- a/src/backend/postmaster/checkpointer.c
+++ b/src/backend/postmaster/checkpointer.c
@@ -10,10 +10,13 @@
* fill WAL segments; the checkpointer itself doesn't watch for the
* condition.)
*
- * Normal termination is by SIGUSR2, which instructs the checkpointer to
- * execute a shutdown checkpoint and then exit(0). (All backends must be
- * stopped before SIGUSR2 is issued!) Emergency termination is by SIGQUIT;
- * like any backend, the checkpointer will simply abort and exit on SIGQUIT.
+ * The normal termination sequence is that checkpointer is instructed to
+ * execute the shutdown checkpoint by SIGINT. After that checkpointer waits
+ * to be terminated via SIGUSR2, which instructs the checkpointer to exit(0).
+ * All backends must be stopped before SIGINT or SIGUSR2 is issued!
+ *
+ * Emergency termination is by SIGQUIT; like any backend, the checkpointer
+ * will simply abort and exit on SIGQUIT.
*
* If the checkpointer exits unexpectedly, the postmaster treats that the same
* as a backend crash: shared memory may be corrupted, so remaining backends
@@ -51,6 +54,7 @@
#include "storage/fd.h"
#include "storage/ipc.h"
#include "storage/lwlock.h"
+#include "storage/pmsignal.h"
#include "storage/proc.h"
#include "storage/procsignal.h"
#include "storage/shmem.h"
@@ -141,6 +145,7 @@ double CheckPointCompletionTarget = 0.9;
* Private state
*/
static bool ckpt_active = false;
+static volatile sig_atomic_t ShutdownXLOGPending = false;
/* these values are valid when ckpt_active is true: */
static pg_time_t ckpt_start_time;
@@ -159,6 +164,9 @@ static bool ImmediateCheckpointRequested(void);
static bool CompactCheckpointerRequestQueue(void);
static void UpdateSharedMemoryConfig(void);
+/* Signal handlers */
+static void ReqShutdownXLOG(SIGNAL_ARGS);
+
/*
* Main entry point for checkpointer process
@@ -188,7 +196,7 @@ CheckpointerMain(char *startup_data, size_t startup_data_len)
* tell us it's okay to shut down (via SIGUSR2).
*/
pqsignal(SIGHUP, SignalHandlerForConfigReload);
- pqsignal(SIGINT, SIG_IGN);
+ pqsignal(SIGINT, ReqShutdownXLOG);
pqsignal(SIGTERM, SIG_IGN); /* ignore SIGTERM */
/* SIGQUIT handler was already set up by InitPostmasterChild */
pqsignal(SIGALRM, SIG_IGN);
@@ -211,8 +219,11 @@ CheckpointerMain(char *startup_data, size_t startup_data_len)
* process during a normal shutdown, and since checkpointer is shut down
* very late...
*
- * Walsenders are shut down after the checkpointer, but currently don't
- * report stats. If that changes, we need a more complicated solution.
+ * While e.g. walsenders are active after the shutdown checkpoint has been
+ * written (and thus could produce more stats), checkpointer stays around
+ * after the shutdown checkpoint has been written. postmaster will only
+ * signal checkpointer to exit after all processes that could emit stats
+ * have been shut down.
*/
before_shmem_exit(pgstat_before_server_shutdown, 0);
@@ -327,7 +338,7 @@ CheckpointerMain(char *startup_data, size_t startup_data_len)
ProcGlobal->checkpointerProc = MyProcNumber;
/*
- * Loop forever
+ * Loop until we've been asked to write shutdown checkpoint or terminate.
*/
for (;;)
{
@@ -346,7 +357,10 @@ CheckpointerMain(char *startup_data, size_t startup_data_len)
* Process any requests or signals received recently.
*/
AbsorbSyncRequests();
+
HandleCheckpointerInterrupts();
+ if (ShutdownXLOGPending || ShutdownRequestPending)
+ break;
/*
* Detect a pending checkpoint request by checking whether the flags
@@ -517,8 +531,13 @@ CheckpointerMain(char *startup_data, size_t startup_data_len)
ckpt_active = false;
- /* We may have received an interrupt during the checkpoint. */
+ /*
+ * We may have received an interrupt during the checkpoint and the
+ * latch might have been reset (e.g. in CheckpointWriteDelay).
+ */
HandleCheckpointerInterrupts();
+ if (ShutdownXLOGPending || ShutdownRequestPending)
+ break;
}
/* Check for archive_timeout and switch xlog files if necessary. */
@@ -557,6 +576,56 @@ CheckpointerMain(char *startup_data, size_t startup_data_len)
cur_timeout * 1000L /* convert to ms */ ,
WAIT_EVENT_CHECKPOINTER_MAIN);
}
+
+ /*
+ * From here on, elog(ERROR) should end with exit(1), not send control
+ * back to the sigsetjmp block above.
+ */
+ ExitOnAnyError = true;
+
+ if (ShutdownXLOGPending)
+ {
+ /*
+ * Close down the database.
+ *
+ * Since ShutdownXLOG() creates restartpoint or checkpoint, and
+ * updates the statistics, increment the checkpoint request and flush
+ * out pending statistic.
+ */
+ PendingCheckpointerStats.num_requested++;
+ ShutdownXLOG(0, 0);
+ pgstat_report_checkpointer();
+ pgstat_report_wal(true);
+
+ /*
+ * Tell postmaster that we're done.
+ */
+ SendPostmasterSignal(PMSIGNAL_XLOG_IS_SHUTDOWN);
+ }
+
+ /*
+ * Wait until we're asked to shut down. By separating the writing of the
+ * shutdown checkpoint from checkpointer exiting, checkpointer can perform
+ * some should-be-as-late-as-possible work like writing out stats.
+ */
+ for (;;)
+ {
+ /* Clear any already-pending wakeups */
+ ResetLatch(MyLatch);
+
+ HandleCheckpointerInterrupts();
+
+ if (ShutdownRequestPending)
+ break;
+
+ (void) WaitLatch(MyLatch,
+ WL_LATCH_SET | WL_EXIT_ON_PM_DEATH,
+ 0,
+ WAIT_EVENT_CHECKPOINTER_SHUTDOWN);
+ }
+
+ /* Normal exit from the checkpointer is here */
+ proc_exit(0); /* done */
}
/*
@@ -586,29 +655,6 @@ HandleCheckpointerInterrupts(void)
*/
UpdateSharedMemoryConfig();
}
- if (ShutdownRequestPending)
- {
- /*
- * From here on, elog(ERROR) should end with exit(1), not send control
- * back to the sigsetjmp block above
- */
- ExitOnAnyError = true;
-
- /*
- * Close down the database.
- *
- * Since ShutdownXLOG() creates restartpoint or checkpoint, and
- * updates the statistics, increment the checkpoint request and flush
- * out pending statistic.
- */
- PendingCheckpointerStats.num_requested++;
- ShutdownXLOG(0, 0);
- pgstat_report_checkpointer();
- pgstat_report_wal(true);
-
- /* Normal exit from the checkpointer is here */
- proc_exit(0); /* done */
- }
/* Perform logging of memory contexts of this process */
if (LogMemoryContextPending)
@@ -729,6 +775,7 @@ CheckpointWriteDelay(int flags, double progress)
* in which case we just try to catch up as quickly as possible.
*/
if (!(flags & CHECKPOINT_IMMEDIATE) &&
+ !ShutdownXLOGPending &&
!ShutdownRequestPending &&
!ImmediateCheckpointRequested() &&
IsCheckpointOnSchedule(progress))
@@ -857,6 +904,20 @@ IsCheckpointOnSchedule(double progress)
}
+/* --------------------------------
+ * signal handler routines
+ * --------------------------------
+ */
+
+/* SIGINT: set flag to trigger writing of shutdown checkpoint */
+static void
+ReqShutdownXLOG(SIGNAL_ARGS)
+{
+ ShutdownXLOGPending = true;
+ SetLatch(MyLatch);
+}
+
+
/* --------------------------------
* communication with backends
* --------------------------------
diff --git a/src/backend/postmaster/postmaster.c b/src/backend/postmaster/postmaster.c
index 54801a32609..115ad3d31d2 100644
--- a/src/backend/postmaster/postmaster.c
+++ b/src/backend/postmaster/postmaster.c
@@ -334,6 +334,7 @@ typedef enum
* ckpt */
PM_WAIT_XLOG_ARCHIVAL, /* waiting for archiver and walsenders to
* finish */
+ PM_WAIT_CHECKPOINTER, /* waiting for checkpointer to shut down */
PM_WAIT_DEAD_END, /* waiting for dead-end children to exit */
PM_NO_CHILDREN, /* all important children have exited */
} PMState;
@@ -2354,35 +2355,19 @@ process_pm_child_exit(void)
{
ReleasePostmasterChildSlot(CheckpointerPMChild);
CheckpointerPMChild = NULL;
- if (EXIT_STATUS_0(exitstatus) && pmState == PM_WAIT_XLOG_SHUTDOWN)
+ if (EXIT_STATUS_0(exitstatus) && pmState == PM_WAIT_CHECKPOINTER)
{
/*
* OK, we saw normal exit of the checkpointer after it's been
- * told to shut down. We expect that it wrote a shutdown
- * checkpoint. (If for some reason it didn't, recovery will
- * occur on next postmaster start.)
+ * told to shut down. We know checkpointer wrote a shutdown
+ * checkpoint, otherwise we'd still be in
+ * PM_WAIT_XLOG_SHUTDOWN state.
*
- * At this point we should have no normal backend children
- * left (else we'd not be in PM_WAIT_XLOG_SHUTDOWN state) but
- * we might have dead-end children to wait for.
- *
- * If we have an archiver subprocess, tell it to do a last
- * archive cycle and quit. Likewise, if we have walsender
- * processes, tell them to send any remaining WAL and quit.
- */
- Assert(Shutdown > NoShutdown);
-
- /* Waken archiver for the last time */
- if (PgArchPMChild != NULL)
- signal_child(PgArchPMChild, SIGUSR2);
-
- /*
- * Waken walsenders for the last time. No regular backends
- * should be around anymore.
+ * At this point only dead-end children should be left.
*/
- SignalChildren(SIGUSR2, btmask(B_WAL_SENDER));
-
- UpdatePMState(PM_WAIT_XLOG_ARCHIVAL);
+ UpdatePMState(PM_WAIT_DEAD_END);
+ ConfigurePostmasterWaitSet(false);
+ SignalChildren(SIGTERM, btmask_all_except(B_LOGGER));
}
else
{
@@ -2718,6 +2703,7 @@ HandleFatalError(QuitSignalReason reason, bool consider_sigabrt)
case PM_WAIT_XLOG_SHUTDOWN:
case PM_WAIT_XLOG_ARCHIVAL:
+ case PM_WAIT_CHECKPOINTER:
/*
* Note that we switch *back* to PM_WAIT_BACKENDS here. This way
@@ -2980,9 +2966,9 @@ PostmasterStateMachine(void)
SignalChildren(SIGQUIT, btmask(B_DEAD_END_BACKEND));
/*
- * We already SIGQUIT'd walsenders and the archiver, if any,
- * when we started immediate shutdown or entered FatalError
- * state.
+ * We already SIGQUIT'd archiver, checkpointer and walsenders,
+ * if any, when we started immediate shutdown or entered
+ * FatalError state.
*/
}
else
@@ -2996,10 +2982,10 @@ PostmasterStateMachine(void)
/* Start the checkpointer if not running */
if (CheckpointerPMChild == NULL)
CheckpointerPMChild = StartChildProcess(B_CHECKPOINTER);
- /* And tell it to shut down */
+ /* And tell it to write the shutdown checkpoint */
if (CheckpointerPMChild != NULL)
{
- signal_child(CheckpointerPMChild, SIGUSR2);
+ signal_child(CheckpointerPMChild, SIGINT);
UpdatePMState(PM_WAIT_XLOG_SHUTDOWN);
}
else
@@ -3024,22 +3010,39 @@ PostmasterStateMachine(void)
}
}
+ /*
+ * The state transition from PM_WAIT_XLOG_SHUTDOWN to
+ * PM_WAIT_XLOG_ARCHIVAL is in proccess_pm_pmsignal(), in response to
+ * PMSIGNAL_XLOG_IS_SHUTDOWN.
+ */
+
if (pmState == PM_WAIT_XLOG_ARCHIVAL)
{
/*
- * PM_WAIT_XLOG_ARCHIVAL state ends when there's no other children
- * than dead-end children left. There shouldn't be any regular
- * backends left by now anyway; what we're really waiting for is
- * walsenders and archiver.
+ * PM_WAIT_XLOG_ARCHIVAL state ends when there's no children other
+ * than checkpointer and dead-end children left. There shouldn't be
+ * any regular backends left by now anyway; what we're really waiting
+ * for is for walsenders and archiver to exit.
*/
- if (CountChildren(btmask_all_except(B_LOGGER, B_DEAD_END_BACKEND)) == 0)
+ if (CountChildren(btmask_all_except(B_CHECKPOINTER, B_LOGGER, B_DEAD_END_BACKEND)) == 0)
{
- UpdatePMState(PM_WAIT_DEAD_END);
- ConfigurePostmasterWaitSet(false);
- SignalChildren(SIGTERM, btmask_all_except(B_LOGGER));
+ UpdatePMState(PM_WAIT_CHECKPOINTER);
+
+ /*
+ * Now that everyone important is gone, tell checkpointer to shut
+ * down too. That allows checkpointer to perform some last bits of
+ * cleanup without other processes interfering.
+ */
+ if (CheckpointerPMChild != NULL)
+ signal_child(CheckpointerPMChild, SIGUSR2);
}
}
+ /*
+ * The state transition from PM_WAIT_CHECKPOINTER to PM_WAIT_DEAD_END is
+ * in proccess_pm_child_exit().
+ */
+
if (pmState == PM_WAIT_DEAD_END)
{
/*
@@ -3176,6 +3179,7 @@ pmstate_name(PMState state)
PM_TOSTR_CASE(PM_WAIT_XLOG_SHUTDOWN);
PM_TOSTR_CASE(PM_WAIT_XLOG_ARCHIVAL);
PM_TOSTR_CASE(PM_WAIT_DEAD_END);
+ PM_TOSTR_CASE(PM_WAIT_CHECKPOINTER);
PM_TOSTR_CASE(PM_NO_CHILDREN);
}
#undef PM_TOSTR_CASE
@@ -3593,6 +3597,8 @@ ExitPostmaster(int status)
static void
process_pm_pmsignal(void)
{
+ bool request_state_update = false;
+
pending_pm_pmsignal = false;
ereport(DEBUG2,
@@ -3704,9 +3710,67 @@ process_pm_pmsignal(void)
WalReceiverRequested = true;
}
+ if (CheckPostmasterSignal(PMSIGNAL_XLOG_IS_SHUTDOWN))
+ {
+ /* Checkpointer completed the shutdown checkpoint */
+ if (pmState == PM_WAIT_XLOG_SHUTDOWN)
+ {
+ /*
+ * If we have an archiver subprocess, tell it to do a last archive
+ * cycle and quit. Likewise, if we have walsender processes, tell
+ * them to send any remaining WAL and quit.
+ */
+ Assert(Shutdown > NoShutdown);
+
+ /* Waken archiver for the last time */
+ if (PgArchPMChild != NULL)
+ signal_child(PgArchPMChild, SIGUSR2);
+
+ /*
+ * Waken walsenders for the last time. No regular backends should
+ * be around anymore.
+ */
+ SignalChildren(SIGUSR2, btmask(B_WAL_SENDER));
+
+ UpdatePMState(PM_WAIT_XLOG_ARCHIVAL);
+ }
+ else if (!FatalError && Shutdown != ImmediateShutdown)
+ {
+ /*
+ * Checkpointer only ought to perform the shutdown checkpoint
+ * during shutdown. If somehow checkpointer did so in another
+ * situation, we have no choice but to crash-restart.
+ *
+ * It's possible however that we get PMSIGNAL_XLOG_IS_SHUTDOWN
+ * outside of PM_WAIT_XLOG_SHUTDOWN if an orderly shutdown was
+ * "interrupted" by a crash or an immediate shutdown.
+ */
+ ereport(LOG,
+ (errmsg("WAL was shut down unexpectedly")));
+
+ /*
+ * Doesn't seem likely to help to take send_abort_for_crash into
+ * account here.
+ */
+ HandleFatalError(PMQUIT_FOR_CRASH, false);
+ }
+
+ /*
+ * Need to run PostmasterStateMachine() to check if we already can go
+ * to the next state.
+ */
+ request_state_update = true;
+ }
+
/*
* Try to advance postmaster's state machine, if a child requests it.
- *
+ */
+ if (CheckPostmasterSignal(PMSIGNAL_ADVANCE_STATE_MACHINE))
+ {
+ request_state_update = true;
+ }
+
+ /*
* Be careful about the order of this action relative to this function's
* other actions. Generally, this should be after other actions, in case
* they have effects PostmasterStateMachine would need to know about.
@@ -3714,7 +3778,7 @@ process_pm_pmsignal(void)
* cannot have any (immediate) effect on the state machine, but does
* depend on what state we're in now.
*/
- if (CheckPostmasterSignal(PMSIGNAL_ADVANCE_STATE_MACHINE))
+ if (request_state_update)
{
PostmasterStateMachine();
}
@@ -4025,6 +4089,7 @@ bgworker_should_start_now(BgWorkerStartTime start_time)
switch (pmState)
{
case PM_NO_CHILDREN:
+ case PM_WAIT_CHECKPOINTER:
case PM_WAIT_DEAD_END:
case PM_WAIT_XLOG_ARCHIVAL:
case PM_WAIT_XLOG_SHUTDOWN:
diff --git a/src/backend/utils/activity/wait_event_names.txt b/src/backend/utils/activity/wait_event_names.txt
index 0b53cba807d..e199f071628 100644
--- a/src/backend/utils/activity/wait_event_names.txt
+++ b/src/backend/utils/activity/wait_event_names.txt
@@ -56,6 +56,7 @@ AUTOVACUUM_MAIN "Waiting in main loop of autovacuum launcher process."
BGWRITER_HIBERNATE "Waiting in background writer process, hibernating."
BGWRITER_MAIN "Waiting in main loop of background writer process."
CHECKPOINTER_MAIN "Waiting in main loop of checkpointer process."
+CHECKPOINTER_SHUTDOWN "Waiting for checkpointer process to be terminated."
LOGICAL_APPLY_MAIN "Waiting in main loop of logical replication apply process."
LOGICAL_LAUNCHER_MAIN "Waiting in main loop of logical replication launcher process."
LOGICAL_PARALLEL_APPLY_MAIN "Waiting in main loop of logical replication parallel apply process."
--
2.48.1.76.g4e746b1a31.dirty
[text/x-diff] v2.3-0008-Ensure-a-resowner-exists-for-all-paths-that-may.patch (2.6K, ../../p5flybfcjrzgpgwcnzkxsp3xdw3czfoafkvks6g7en6ewmvymh@wrz7z2vwwc7c/9-v2.3-0008-Ensure-a-resowner-exists-for-all-paths-that-may.patch)
download | inline diff:
From 1476ef34b2a2c36e8e1eccbf6d2ac12607b4dab7 Mon Sep 17 00:00:00 2001
From: Andres Freund <andres@anarazel.de>
Date: Tue, 8 Oct 2024 14:34:38 -0400
Subject: [PATCH v2.3 08/30] Ensure a resowner exists for all paths that may
perform AIO
Reviewed-by: Noah Misch <noah@leadboat.com>
Discussion: https://postgr.es/m/1f6b50a7-38ef-4d87-8246-786d39f46ab9@iki.fi
---
src/backend/bootstrap/bootstrap.c | 7 +++++++
src/backend/replication/logical/logical.c | 6 ++++++
src/backend/utils/init/postinit.c | 6 +++++-
3 files changed, 18 insertions(+), 1 deletion(-)
diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c
index 359f58a8f95..5d41cfc6eb0 100644
--- a/src/backend/bootstrap/bootstrap.c
+++ b/src/backend/bootstrap/bootstrap.c
@@ -361,8 +361,15 @@ BootstrapModeMain(int argc, char *argv[], bool check_only)
BaseInit();
bootstrap_signals();
+
+ /* need a resowner for IO during BootStrapXLOG() */
+ CreateAuxProcessResourceOwner();
+
BootStrapXLOG(bootstrap_data_checksum_version);
+ ReleaseAuxProcessResources(true);
+ CurrentResourceOwner = NULL;
+
/*
* To ensure that src/common/link-canary.c is linked into the backend, we
* must call it from somewhere. Here is as good as anywhere.
diff --git a/src/backend/replication/logical/logical.c b/src/backend/replication/logical/logical.c
index 0b25efafe2b..1f8ec3daa6a 100644
--- a/src/backend/replication/logical/logical.c
+++ b/src/backend/replication/logical/logical.c
@@ -386,6 +386,12 @@ CreateInitDecodingContext(const char *plugin,
slot->data.plugin = plugin_name;
SpinLockRelease(&slot->mutex);
+ if (CurrentResourceOwner == NULL)
+ {
+ Assert(am_walsender);
+ CurrentResourceOwner = AuxProcessResourceOwner;
+ }
+
if (XLogRecPtrIsInvalid(restart_lsn))
ReplicationSlotReserveWal();
else
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index 01bb6a410cb..b491d04de58 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -755,8 +755,12 @@ InitPostgres(const char *in_dbname, Oid dboid,
* We don't yet have an aux-process resource owner, but StartupXLOG
* and ShutdownXLOG will need one. Hence, create said resource owner
* (and register a callback to clean it up after ShutdownXLOG runs).
+ *
+ * In bootstrap mode CreateAuxProcessResourceOwner() was already
+ * called in BootstrapModeMain().
*/
- CreateAuxProcessResourceOwner();
+ if (!bootstrap)
+ CreateAuxProcessResourceOwner();
StartupXLOG();
/* Release (and warn about) any buffer pins leaked in StartupXLOG */
--
2.48.1.76.g4e746b1a31.dirty
[text/x-diff] v2.3-0009-Allow-lwlocks-to-be-unowned.patch (4.6K, ../../p5flybfcjrzgpgwcnzkxsp3xdw3czfoafkvks6g7en6ewmvymh@wrz7z2vwwc7c/10-v2.3-0009-Allow-lwlocks-to-be-unowned.patch)
download | inline diff:
From 552b094c4f52b4092d7998cce01908bff5ddcf8b Mon Sep 17 00:00:00 2001
From: Andres Freund <andres@anarazel.de>
Date: Tue, 5 Jan 2021 10:10:36 -0800
Subject: [PATCH v2.3 09/30] Allow lwlocks to be unowned
This is required for AIO so that the lock hold during a write can be released
in another backend. Which in turn is required to avoid the potential for
deadlocks.
---
src/include/storage/lwlock.h | 2 +
src/backend/storage/lmgr/lwlock.c | 108 +++++++++++++++++++++++-------
2 files changed, 85 insertions(+), 25 deletions(-)
diff --git a/src/include/storage/lwlock.h b/src/include/storage/lwlock.h
index 2aa46fd50da..13a7dc89980 100644
--- a/src/include/storage/lwlock.h
+++ b/src/include/storage/lwlock.h
@@ -129,6 +129,8 @@ extern bool LWLockAcquireOrWait(LWLock *lock, LWLockMode mode);
extern void LWLockRelease(LWLock *lock);
extern void LWLockReleaseClearVar(LWLock *lock, pg_atomic_uint64 *valptr, uint64 val);
extern void LWLockReleaseAll(void);
+extern void LWLockDisown(LWLock *l);
+extern void LWLockReleaseDisowned(LWLock *l, LWLockMode mode);
extern bool LWLockHeldByMe(LWLock *lock);
extern bool LWLockAnyHeldByMe(LWLock *lock, int nlocks, size_t stride);
extern bool LWLockHeldByMeInMode(LWLock *lock, LWLockMode mode);
diff --git a/src/backend/storage/lmgr/lwlock.c b/src/backend/storage/lmgr/lwlock.c
index 2f558ffea14..c3d6f886e3c 100644
--- a/src/backend/storage/lmgr/lwlock.c
+++ b/src/backend/storage/lmgr/lwlock.c
@@ -1773,36 +1773,15 @@ LWLockUpdateVar(LWLock *lock, pg_atomic_uint64 *valptr, uint64 val)
}
}
-
/*
- * LWLockRelease - release a previously acquired lock
+ * Helper function to release lock, shared between LWLockRelease() and
+ * LWLockeleaseDisowned().
*/
-void
-LWLockRelease(LWLock *lock)
+static void
+LWLockReleaseInternal(LWLock *lock, LWLockMode mode)
{
- LWLockMode mode;
uint32 oldstate;
bool check_waiters;
- int i;
-
- /*
- * Remove lock from list of locks held. Usually, but not always, it will
- * be the latest-acquired lock; so search array backwards.
- */
- for (i = num_held_lwlocks; --i >= 0;)
- if (lock == held_lwlocks[i].lock)
- break;
-
- if (i < 0)
- elog(ERROR, "lock %s is not held", T_NAME(lock));
-
- mode = held_lwlocks[i].mode;
-
- num_held_lwlocks--;
- for (; i < num_held_lwlocks; i++)
- held_lwlocks[i] = held_lwlocks[i + 1];
-
- PRINT_LWDEBUG("LWLockRelease", lock, mode);
/*
* Release my hold on lock, after that it can immediately be acquired by
@@ -1840,6 +1819,85 @@ LWLockRelease(LWLock *lock)
LOG_LWDEBUG("LWLockRelease", lock, "releasing waiters");
LWLockWakeup(lock);
}
+}
+
+void
+LWLockReleaseDisowned(LWLock *lock, LWLockMode mode)
+{
+ LWLockReleaseInternal(lock, mode);
+}
+
+/*
+ * Stop treating lock as held by current backend.
+ *
+ * This is the code that can be shared between actually releasing a lock
+ * (LWLockRelease()) and just not tracking ownership of the lock anymore
+ * without releasing the lock (LWLockDisown()).
+ *
+ * Returns the mode in which the lock was held by the current backend.
+ *
+ * NB: This does not call RESUME_INTERRUPTS(), but leaves that responsibility
+ * of the caller.
+ *
+ * NB: This will leave lock->owner pointing to the current backend (if
+ * LOCK_DEBUG is set). This is somewhat intentional, as it makes it easier to
+ * debug cases of missing wakeups during lock release.
+ */
+static inline LWLockMode
+LWLockDisownInternal(LWLock *lock)
+{
+ LWLockMode mode;
+ int i;
+
+ /*
+ * Remove lock from list of locks held. Usually, but not always, it will
+ * be the latest-acquired lock; so search array backwards.
+ */
+ for (i = num_held_lwlocks; --i >= 0;)
+ if (lock == held_lwlocks[i].lock)
+ break;
+
+ if (i < 0)
+ elog(ERROR, "lock %s is not held", T_NAME(lock));
+
+ mode = held_lwlocks[i].mode;
+
+ num_held_lwlocks--;
+ for (; i < num_held_lwlocks; i++)
+ held_lwlocks[i] = held_lwlocks[i + 1];
+
+ return mode;
+}
+
+/*
+ * Stop treating lock as held by current backend.
+ *
+ * After calling this function it's the callers responsibility to ensure that
+ * the lock gets released (via LWLockReleaseDisowned()), even in case of an
+ * error. This only is desirable if the lock is going to be released in a
+ * different process than the process that acquired it.
+ */
+void
+LWLockDisown(LWLock *lock)
+{
+ LWLockDisownInternal(lock);
+
+ RESUME_INTERRUPTS();
+}
+
+/*
+ * LWLockRelease - release a previously acquired lock
+ */
+void
+LWLockRelease(LWLock *lock)
+{
+ LWLockMode mode;
+
+ mode = LWLockDisownInternal(lock);
+
+ PRINT_LWDEBUG("LWLockRelease", lock, mode);
+
+ LWLockReleaseInternal(lock, mode);
/*
* Now okay to allow cancel/die interrupts.
--
2.48.1.76.g4e746b1a31.dirty
[text/x-diff] v2.3-0010-aio-Basic-subsystem-initialization.patch (10.5K, ../../p5flybfcjrzgpgwcnzkxsp3xdw3czfoafkvks6g7en6ewmvymh@wrz7z2vwwc7c/11-v2.3-0010-aio-Basic-subsystem-initialization.patch)
download | inline diff:
From a6f1745cefdfb932be393f0374765e60563ab23d Mon Sep 17 00:00:00 2001
From: Andres Freund <andres@anarazel.de>
Date: Mon, 10 Jun 2024 13:42:58 -0700
Subject: [PATCH v2.3 10/30] aio: Basic subsystem initialization
This is just separate to make it easier to review the tendrils into various
places.
---
src/include/storage/aio.h | 37 +++++++++++++++++++
src/include/storage/aio_init.h | 24 ++++++++++++
src/include/utils/guc.h | 1 +
src/backend/storage/aio/Makefile | 2 +
src/backend/storage/aio/aio.c | 36 ++++++++++++++++++
src/backend/storage/aio/aio_init.c | 37 +++++++++++++++++++
src/backend/storage/aio/meson.build | 2 +
src/backend/storage/ipc/ipci.c | 3 ++
src/backend/utils/init/postinit.c | 7 ++++
src/backend/utils/misc/guc_tables.c | 23 ++++++++++++
src/backend/utils/misc/postgresql.conf.sample | 11 ++++++
src/tools/pgindent/typedefs.list | 1 +
12 files changed, 184 insertions(+)
create mode 100644 src/include/storage/aio.h
create mode 100644 src/include/storage/aio_init.h
create mode 100644 src/backend/storage/aio/aio.c
create mode 100644 src/backend/storage/aio/aio_init.c
diff --git a/src/include/storage/aio.h b/src/include/storage/aio.h
new file mode 100644
index 00000000000..0e3fadac543
--- /dev/null
+++ b/src/include/storage/aio.h
@@ -0,0 +1,37 @@
+/*-------------------------------------------------------------------------
+ *
+ * aio.h
+ * Main AIO interface
+ *
+ *
+ * Portions Copyright (c) 1996-2025, PostgreSQL Global Development Group
+ * Portions Copyright (c) 1994, Regents of the University of California
+ *
+ * src/include/storage/aio.h
+ *
+ *-------------------------------------------------------------------------
+ */
+#ifndef AIO_H
+#define AIO_H
+
+
+
+/* Enum for io_method GUC. */
+typedef enum IoMethod
+{
+ IOMETHOD_SYNC = 0,
+} IoMethod;
+
+/* We'll default to synchronous execution. */
+#define DEFAULT_IO_METHOD IOMETHOD_SYNC
+
+
+extern void assign_io_method(int newval, void *extra);
+
+
+/* GUCs */
+extern PGDLLIMPORT int io_method;
+extern PGDLLIMPORT int io_max_concurrency;
+
+
+#endif /* AIO_H */
diff --git a/src/include/storage/aio_init.h b/src/include/storage/aio_init.h
new file mode 100644
index 00000000000..44151ef55bf
--- /dev/null
+++ b/src/include/storage/aio_init.h
@@ -0,0 +1,24 @@
+/*-------------------------------------------------------------------------
+ *
+ * aio_init.h
+ * AIO initialization - kept separate as initialization sites don't need to
+ * know about AIO itself and AIO users don't need to know about initialization.
+ *
+ *
+ * Portions Copyright (c) 1996-2025, PostgreSQL Global Development Group
+ * Portions Copyright (c) 1994, Regents of the University of California
+ *
+ * src/include/storage/aio_init.h
+ *
+ *-------------------------------------------------------------------------
+ */
+#ifndef AIO_INIT_H
+#define AIO_INIT_H
+
+
+extern Size AioShmemSize(void);
+extern void AioShmemInit(void);
+
+extern void pgaio_init_backend(void);
+
+#endif /* AIO_INIT_H */
diff --git a/src/include/utils/guc.h b/src/include/utils/guc.h
index 532d6642bb4..aa859c92085 100644
--- a/src/include/utils/guc.h
+++ b/src/include/utils/guc.h
@@ -314,6 +314,7 @@ extern PGDLLIMPORT bool optimize_bounded_sort;
*/
extern PGDLLIMPORT const struct config_enum_entry archive_mode_options[];
extern PGDLLIMPORT const struct config_enum_entry dynamic_shared_memory_options[];
+extern PGDLLIMPORT const struct config_enum_entry io_method_options[];
extern PGDLLIMPORT const struct config_enum_entry recovery_target_action_options[];
extern PGDLLIMPORT const struct config_enum_entry wal_level_options[];
extern PGDLLIMPORT const struct config_enum_entry wal_sync_method_options[];
diff --git a/src/backend/storage/aio/Makefile b/src/backend/storage/aio/Makefile
index 2f29a9ec4d1..eaeaeeee8e3 100644
--- a/src/backend/storage/aio/Makefile
+++ b/src/backend/storage/aio/Makefile
@@ -9,6 +9,8 @@ top_builddir = ../../../..
include $(top_builddir)/src/Makefile.global
OBJS = \
+ aio.o \
+ aio_init.o \
read_stream.o
include $(top_srcdir)/src/backend/common.mk
diff --git a/src/backend/storage/aio/aio.c b/src/backend/storage/aio/aio.c
new file mode 100644
index 00000000000..f68cbc2b3f4
--- /dev/null
+++ b/src/backend/storage/aio/aio.c
@@ -0,0 +1,36 @@
+/*-------------------------------------------------------------------------
+ *
+ * aio.c
+ * AIO - Core Logic
+ *
+ * Portions Copyright (c) 1996-2025, PostgreSQL Global Development Group
+ * Portions Copyright (c) 1994, Regents of the University of California
+ *
+ * IDENTIFICATION
+ * src/backend/storage/aio/aio.c
+ *
+ *-------------------------------------------------------------------------
+ */
+
+#include "postgres.h"
+
+#include "storage/aio.h"
+#include "utils/guc.h"
+
+
+/* Options for io_method. */
+const struct config_enum_entry io_method_options[] = {
+ {"sync", IOMETHOD_SYNC, false},
+ {NULL, 0, false}
+};
+
+/* GUCs */
+int io_method = DEFAULT_IO_METHOD;
+int io_max_concurrency = -1;
+
+
+
+void
+assign_io_method(int newval, void *extra)
+{
+}
diff --git a/src/backend/storage/aio/aio_init.c b/src/backend/storage/aio/aio_init.c
new file mode 100644
index 00000000000..f7ee8270756
--- /dev/null
+++ b/src/backend/storage/aio/aio_init.c
@@ -0,0 +1,37 @@
+/*-------------------------------------------------------------------------
+ *
+ * aio_init.c
+ * AIO - Subsystem Initialization
+ *
+ * Portions Copyright (c) 1996-2025, PostgreSQL Global Development Group
+ * Portions Copyright (c) 1994, Regents of the University of California
+ *
+ * IDENTIFICATION
+ * src/backend/storage/aio/aio_init.c
+ *
+ *-------------------------------------------------------------------------
+ */
+
+#include "postgres.h"
+
+#include "storage/aio_init.h"
+
+
+
+Size
+AioShmemSize(void)
+{
+ Size sz = 0;
+
+ return sz;
+}
+
+void
+AioShmemInit(void)
+{
+}
+
+void
+pgaio_init_backend(void)
+{
+}
diff --git a/src/backend/storage/aio/meson.build b/src/backend/storage/aio/meson.build
index 8abe0eb4863..c822fd4ddf7 100644
--- a/src/backend/storage/aio/meson.build
+++ b/src/backend/storage/aio/meson.build
@@ -1,5 +1,7 @@
# Copyright (c) 2024-2025, PostgreSQL Global Development Group
backend_sources += files(
+ 'aio.c',
+ 'aio_init.c',
'read_stream.c',
)
diff --git a/src/backend/storage/ipc/ipci.c b/src/backend/storage/ipc/ipci.c
index 174eed70367..e11e82fc897 100644
--- a/src/backend/storage/ipc/ipci.c
+++ b/src/backend/storage/ipc/ipci.c
@@ -37,6 +37,7 @@
#include "replication/slotsync.h"
#include "replication/walreceiver.h"
#include "replication/walsender.h"
+#include "storage/aio_init.h"
#include "storage/bufmgr.h"
#include "storage/dsm.h"
#include "storage/dsm_registry.h"
@@ -148,6 +149,7 @@ CalculateShmemSize(int *num_semaphores)
size = add_size(size, WaitEventCustomShmemSize());
size = add_size(size, InjectionPointShmemSize());
size = add_size(size, SlotSyncShmemSize());
+ size = add_size(size, AioShmemSize());
/* include additional requested shmem from preload libraries */
size = add_size(size, total_addin_request);
@@ -340,6 +342,7 @@ CreateOrAttachShmemStructs(void)
StatsShmemInit();
WaitEventCustomShmemInit();
InjectionPointShmemInit();
+ AioShmemInit();
}
/*
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index b491d04de58..8ea50314a4e 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -43,6 +43,7 @@
#include "replication/slot.h"
#include "replication/slotsync.h"
#include "replication/walsender.h"
+#include "storage/aio_init.h"
#include "storage/bufmgr.h"
#include "storage/fd.h"
#include "storage/ipc.h"
@@ -626,6 +627,12 @@ BaseInit(void)
*/
pgstat_initialize();
+ /*
+ * Initialize AIO before infrastructure that might need to actually
+ * execute AIO.
+ */
+ pgaio_init_backend();
+
/* Do local initialization of storage and buffer managers */
InitSync();
smgrinit();
diff --git a/src/backend/utils/misc/guc_tables.c b/src/backend/utils/misc/guc_tables.c
index 38cb9e970d5..de524eccad5 100644
--- a/src/backend/utils/misc/guc_tables.c
+++ b/src/backend/utils/misc/guc_tables.c
@@ -71,6 +71,7 @@
#include "replication/slot.h"
#include "replication/slotsync.h"
#include "replication/syncrep.h"
+#include "storage/aio.h"
#include "storage/bufmgr.h"
#include "storage/bufpage.h"
#include "storage/large_object.h"
@@ -3220,6 +3221,18 @@ struct config_int ConfigureNamesInt[] =
NULL, NULL, NULL
},
+ {
+ {"io_max_concurrency",
+ PGC_POSTMASTER,
+ RESOURCES_ASYNCHRONOUS,
+ gettext_noop("Number of IOs that may be in flight in one backend."),
+ NULL,
+ },
+ &io_max_concurrency,
+ -1, -1, 1024,
+ NULL, NULL, NULL
+ },
+
{
{"backend_flush_after", PGC_USERSET, RESOURCES_ASYNCHRONOUS,
gettext_noop("Number of pages after which previously performed writes are flushed to disk."),
@@ -5236,6 +5249,16 @@ struct config_enum ConfigureNamesEnum[] =
NULL, NULL, NULL
},
+ {
+ {"io_method", PGC_POSTMASTER, RESOURCES_MEM,
+ gettext_noop("Selects the method of asynchronous I/O to use."),
+ NULL
+ },
+ &io_method,
+ DEFAULT_IO_METHOD, io_method_options,
+ NULL, assign_io_method, NULL
+ },
+
/* End-of-list marker */
{
{NULL, 0, 0, NULL, NULL}, NULL, 0, NULL, NULL, NULL, NULL
diff --git a/src/backend/utils/misc/postgresql.conf.sample b/src/backend/utils/misc/postgresql.conf.sample
index 079efa1baa7..fba0ad4b624 100644
--- a/src/backend/utils/misc/postgresql.conf.sample
+++ b/src/backend/utils/misc/postgresql.conf.sample
@@ -843,6 +843,17 @@ autovacuum_worker_slots = 16 # autovacuum worker slots to allocate
#include = '...' # include file
+#------------------------------------------------------------------------------
+# WIP AIO GUC docs
+#------------------------------------------------------------------------------
+
+#io_method = sync # (change requires restart)
+
+#io_max_concurrency = 32 # Max number of IOs that may be in
+ # flight at the same time in one backend
+ # (change requires restart)
+
+
#------------------------------------------------------------------------------
# CUSTOMIZED OPTIONS
#------------------------------------------------------------------------------
diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list
index d5aa5c295ae..3bec090428d 100644
--- a/src/tools/pgindent/typedefs.list
+++ b/src/tools/pgindent/typedefs.list
@@ -1266,6 +1266,7 @@ IntoClause
InvalMessageArray
InvalidationInfo
InvalidationMsgsGroup
+IoMethod
IpcMemoryId
IpcMemoryKey
IpcMemoryState
--
2.48.1.76.g4e746b1a31.dirty
[text/x-diff] v2.3-0011-aio-Core-AIO-implementation.patch (76.6K, ../../p5flybfcjrzgpgwcnzkxsp3xdw3czfoafkvks6g7en6ewmvymh@wrz7z2vwwc7c/12-v2.3-0011-aio-Core-AIO-implementation.patch)
download | inline diff:
From ac42f990b85ae4034f16acf9929ce28e18ec2088 Mon Sep 17 00:00:00 2001
From: Andres Freund <andres@anarazel.de>
Date: Tue, 7 Jan 2025 14:42:12 -0500
Subject: [PATCH v2.3 11/30] aio: Core AIO implementation
At this point nothing can use AIO - this commit does not include any
implementation of aio subjects / callbacks. That will come in later commits.
Todo:
- lots of cleanup
---
src/include/storage/aio.h | 301 ++++++
src/include/storage/aio_internal.h | 295 ++++++
src/include/storage/aio_types.h | 115 +++
src/include/utils/resowner.h | 5 +
src/backend/access/transam/xact.c | 9 +
src/backend/storage/aio/Makefile | 4 +
src/backend/storage/aio/aio.c | 904 ++++++++++++++++++
src/backend/storage/aio/aio_callback.c | 280 ++++++
src/backend/storage/aio/aio_init.c | 186 ++++
src/backend/storage/aio/aio_io.c | 175 ++++
src/backend/storage/aio/aio_target.c | 108 +++
src/backend/storage/aio/meson.build | 4 +
src/backend/storage/aio/method_sync.c | 47 +
.../utils/activity/wait_event_names.txt | 3 +
src/backend/utils/resowner/resowner.c | 30 +
src/tools/pgindent/typedefs.list | 21 +
16 files changed, 2487 insertions(+)
create mode 100644 src/include/storage/aio_internal.h
create mode 100644 src/include/storage/aio_types.h
create mode 100644 src/backend/storage/aio/aio_callback.c
create mode 100644 src/backend/storage/aio/aio_io.c
create mode 100644 src/backend/storage/aio/aio_target.c
create mode 100644 src/backend/storage/aio/method_sync.c
diff --git a/src/include/storage/aio.h b/src/include/storage/aio.h
index 0e3fadac543..ffd382593d0 100644
--- a/src/include/storage/aio.h
+++ b/src/include/storage/aio.h
@@ -14,6 +14,9 @@
#ifndef AIO_H
#define AIO_H
+#include "storage/aio_types.h"
+#include "storage/procnumber.h"
+
/* Enum for io_method GUC. */
@@ -26,9 +29,307 @@ typedef enum IoMethod
#define DEFAULT_IO_METHOD IOMETHOD_SYNC
+/*
+ * Flags for an IO that can be set with pgaio_io_set_flag().
+ */
+typedef enum PgAioHandleFlags
+{
+ /*
+ * The IO references backend local memory.
+ *
+ * This needs to be set on an IO whenever the IO references process-local
+ * memory. Some IO methods do not support executing IO that references
+ * process local memory and thus need to fall back to executing IO
+ * synchronously for IOs with this flag set.
+ *
+ * Required for correctness.
+ */
+ PGAIO_HF_REFERENCES_LOCAL = 1 << 1,
+
+ /*
+ * Hint that IO will be executed synchronously.
+ *
+ * This can make it a bit cheaper to execute synchronous IO via the AIO
+ * interface, to avoid needing an AIO and non-AIO version of code.
+ *
+ * Advantageous to set, if applicable, but not required for correctness.
+ */
+ PGAIO_HF_SYNCHRONOUS = 1 << 0,
+
+ /*
+ * IO is using buffered IO, used to control heuristic in some IO methods.
+ *
+ * Advantageous to set, if applicable, but not required for correctness.
+ */
+ PGAIO_HF_BUFFERED = 1 << 2,
+} PgAioHandleFlags;
+
+/*
+ * The IO operations supported by the AIO subsystem.
+ *
+ * This could be in aio_internal.h, as it is not publically referenced, but
+ * PgAioOpData currently *does* need to be public, therefore keeping this
+ * public seems to make sense.
+ */
+typedef enum PgAioOp
+{
+ /* intentionally the zero value, to help catch zeroed memory etc */
+ PGAIO_OP_INVALID = 0,
+
+ PGAIO_OP_READV,
+ PGAIO_OP_WRITEV,
+
+ /**
+ * In the near term we'll need at least:
+ * - fsync / fdatasync
+ * - flush_range
+ *
+ * Eventually we'll additionally want at least:
+ * - send
+ * - recv
+ * - accept
+ **/
+} PgAioOp;
+
+#define PGAIO_OP_COUNT (PGAIO_OP_WRITEV + 1)
+
+
+/*
+ * On what is IO being performed.
+ *
+ * PgAioTargetID specific behaviour should be implemented in
+ * aio_target.c.
+ */
+typedef enum PgAioTargetID
+{
+ /* intentionally the zero value, to help catch zeroed memory etc */
+ PGAIO_TID_INVALID = 0,
+} PgAioTargetID;
+
+#define PGAIO_TID_COUNT (PGAIO_TID_INVALID + 1)
+
+
+/*
+ * Data necessary for support IO operations (see PgAioOp).
+ *
+ * NB: Note that the FDs in here may *not* be relied upon for re-issuing
+ * requests (e.g. for partial reads/writes) - the FD might be from another
+ * process, or closed since. That's not a problem for IOs waiting to be issued
+ * only because the queue is flushed when closing an FD.
+ */
+typedef union
+{
+ struct
+ {
+ int fd;
+ uint16 iov_length;
+ uint64 offset;
+ } read;
+
+ struct
+ {
+ int fd;
+ uint16 iov_length;
+ uint64 offset;
+ } write;
+} PgAioOpData;
+
+
+/*
+ * Information the object that IO is executed on. Mostly callbacks that
+ * operate on PgAioTargetData.
+ */
+typedef struct PgAioTargetInfo
+{
+ void (*reopen) (PgAioHandle *ioh);
+
+ char *(*describe_identity) (const PgAioTargetData *sd);
+
+ const char *name;
+} PgAioTargetInfo;
+
+
+/*
+ * IDs for callbacks that can be registered on an IO.
+ *
+ * Callbacks are identified by an ID rather than a function pointer. There are
+ * two main reasons:
+ *
+ * 1) Memory within PgAioHandle is precious, due to the number of PgAioHandle
+ * structs in pre-allocated shared memory.
+ *
+ * 2) Due to EXEC_BACKEND function pointers are not necessarily stable between
+ * different backends, therefore function pointers cannot directly be in
+ * shared memory.
+ *
+ * Without 2), we could fairly easily allow to add new callbacks, by filling a
+ * ID->pointer mapping table on demand. In the presence of 2 that's still
+ * doable, but harder, because every process has to re-register the pointers
+ * so that a local ID->"backend local pointer" mapping can be maintained.
+ */
+typedef enum PgAioHandleCallbackID
+{
+ PGAIO_HCB_INVALID,
+} PgAioHandleCallbackID;
+
+
+typedef void (*PgAioHandleCallbackStage) (PgAioHandle *ioh);
+typedef PgAioResult (*PgAioHandleCallbackComplete) (PgAioHandle *ioh, PgAioResult prior_result);
+typedef void (*PgAioHandleCallbackReport) (PgAioResult result, const PgAioTargetData *target_data, int elevel);
+
+typedef struct PgAioHandleCallbacks
+{
+ /*
+ * Prepare resources affected by the IO for execution. This could e.g.
+ * include moving ownership of buffer pins to the AIO subsystem.
+ */
+ PgAioHandleCallbackStage stage;
+
+ /*
+ * Update the state of resources affected by the IO to reflect completion
+ * of the IO. This could e.g. include updating shared buffer state to
+ * signal the IO has finished.
+ *
+ * The _shared suffix indicates that this is executed by the backend that
+ * completed the IO, which may or may not be the backend that issued the
+ * IO. Obviously the callback thus can only modify resources in shared
+ * memory.
+ *
+ * The latest registered callback is called first. This allows
+ * higher-level code to register callbacks that can rely on callbacks
+ * registered by lower-level code to already have been executed.
+ *
+ * NB: This is called in a critical section. Errors can be signalled by
+ * the callback's return value, it's the responsibility of the IO's issuer
+ * to react appropriately.
+ */
+ PgAioHandleCallbackComplete complete_shared;
+
+ /*
+ * Like complete_shared, except called in the issuing backend.
+ *
+ * This variant of the completion callback is useful when backend-local
+ * state has to be updated to reflect the IO's completion. E.g. a
+ * temporary buffer's BufferDesc isn't accessible in complete_shared.
+ *
+ * Local callbacks are only called after complete_shared for all
+ * registered callbacks has been called.
+ */
+ PgAioHandleCallbackComplete complete_local;
+
+ /*
+ * Report the result of an IO operation. This is e.g. used to raise an
+ * error after an IO failed at the appropriate time (i.e. not when the IO
+ * failed, but under control of the code that issued the IO).
+ */
+ PgAioHandleCallbackReport report;
+} PgAioHandleCallbacks;
+
+
+
+/*
+ * How many callbacks can be registered for one IO handle. Currently we only
+ * need two, but it's not hard to imagine needing a few more.
+ */
+#define PGAIO_HANDLE_MAX_CALLBACKS 4
+
+
+
+/* AIO API */
+
+
+/* --------------------------------------------------------------------------------
+ * IO Handles
+ * --------------------------------------------------------------------------------
+ */
+
+/* functions in aio.c */
+struct ResourceOwnerData;
+extern PgAioHandle *pgaio_io_acquire(struct ResourceOwnerData *resowner, PgAioReturn *ret);
+extern PgAioHandle *pgaio_io_acquire_nb(struct ResourceOwnerData *resowner, PgAioReturn *ret);
+
+extern void pgaio_io_release(PgAioHandle *ioh);
+struct dlist_node;
+extern void pgaio_io_release_resowner(struct dlist_node *ioh_node, bool on_error);
+
+extern void pgaio_io_set_flag(PgAioHandle *ioh, PgAioHandleFlags flag);
+
+extern int pgaio_io_get_id(PgAioHandle *ioh);
+extern ProcNumber pgaio_io_get_owner(PgAioHandle *ioh);
+
+extern void pgaio_io_get_wref(PgAioHandle *ioh, PgAioWaitRef *iow);
+
+/* functions in aio_io.c */
+struct iovec;
+extern int pgaio_io_get_iovec(PgAioHandle *ioh, struct iovec **iov);
+
+extern PgAioOpData *pgaio_io_get_op_data(PgAioHandle *ioh);
+
+extern void pgaio_io_prep_readv(PgAioHandle *ioh,
+ int fd, int iovcnt, uint64 offset);
+extern void pgaio_io_prep_writev(PgAioHandle *ioh,
+ int fd, int iovcnt, uint64 offset);
+
+/* functions in aio_target.c */
+extern void pgaio_io_set_target(PgAioHandle *ioh, PgAioTargetID targetid);
+extern bool pgaio_io_has_target(PgAioHandle *ioh);
+extern PgAioTargetData *pgaio_io_get_target_data(PgAioHandle *ioh);
+extern char *pgaio_io_get_target_description(PgAioHandle *ioh);
+
+/* functions in aio_callback.c */
+extern void pgaio_io_register_callbacks(PgAioHandle *ioh, PgAioHandleCallbackID cbid);
+extern void pgaio_io_set_handle_data_64(PgAioHandle *ioh, uint64 *data, uint8 len);
+extern void pgaio_io_set_handle_data_32(PgAioHandle *ioh, uint32 *data, uint8 len);
+extern uint64 *pgaio_io_get_handle_data(PgAioHandle *ioh, uint8 *len);
+
+
+
+/* --------------------------------------------------------------------------------
+ * IO Wait References
+ * --------------------------------------------------------------------------------
+ */
+
+extern void pgaio_wref_clear(PgAioWaitRef *iow);
+extern bool pgaio_wref_valid(PgAioWaitRef *iow);
+extern int pgaio_wref_get_id(PgAioWaitRef *iow);
+
+extern void pgaio_wref_wait(PgAioWaitRef *iow);
+extern bool pgaio_wref_check_done(PgAioWaitRef *iow);
+
+
+
+/* --------------------------------------------------------------------------------
+ * IO Result
+ * --------------------------------------------------------------------------------
+ */
+
+extern void pgaio_result_report(PgAioResult result, const PgAioTargetData *target_data,
+ int elevel);
+
+
+
+/* --------------------------------------------------------------------------------
+ * Actions on multiple IOs.
+ * --------------------------------------------------------------------------------
+ */
+
+extern void pgaio_submit_staged(void);
+extern bool pgaio_have_staged(void);
+
+
+
+/* --------------------------------------------------------------------------------
+ * Other
+ * --------------------------------------------------------------------------------
+ */
+
+extern void pgaio_closing_fd(int fd);
+extern void pgaio_at_xact_end(bool is_subxact, bool is_commit);
+extern void pgaio_at_error(void);
extern void assign_io_method(int newval, void *extra);
+
/* GUCs */
extern PGDLLIMPORT int io_method;
extern PGDLLIMPORT int io_max_concurrency;
diff --git a/src/include/storage/aio_internal.h b/src/include/storage/aio_internal.h
new file mode 100644
index 00000000000..174d365f9c0
--- /dev/null
+++ b/src/include/storage/aio_internal.h
@@ -0,0 +1,295 @@
+/*-------------------------------------------------------------------------
+ *
+ * aio_internal.h
+ * AIO related declarations that shoul only be used by the AIO subsystem
+ * internally.
+ *
+ *
+ * Portions Copyright (c) 1996-2025, PostgreSQL Global Development Group
+ * Portions Copyright (c) 1994, Regents of the University of California
+ *
+ * src/include/storage/aio_internal.h
+ *
+ *-------------------------------------------------------------------------
+ */
+#ifndef AIO_INTERNAL_H
+#define AIO_INTERNAL_H
+
+
+#include "lib/ilist.h"
+#include "port/pg_iovec.h"
+#include "storage/aio.h"
+#include "storage/condition_variable.h"
+
+
+/* AFIXME */
+#define PGAIO_SUBMIT_BATCH_SIZE 32
+
+
+
+typedef enum PgAioHandleState
+{
+ /* not in use */
+ PGAIO_HS_IDLE = 0,
+
+ /* returned by pgaio_io_acquire() */
+ PGAIO_HS_HANDED_OUT,
+
+ /* pgaio_io_prep_*() has been called, but IO hasn't been submitted yet */
+ PGAIO_HS_DEFINED,
+
+ /* target's stage() callback has been called, ready to be submitted */
+ PGAIO_HS_STAGED,
+
+ /* IO has been submitted and is being executed */
+ PGAIO_HS_SUBMITTED,
+
+ /* IO finished, but result has not yet been processed */
+ PGAIO_HS_COMPLETED_IO,
+
+ /* IO completed, shared completion has been called */
+ PGAIO_HS_COMPLETED_SHARED,
+
+ /* IO completed, local completion has been called */
+ PGAIO_HS_COMPLETED_LOCAL,
+} PgAioHandleState;
+
+
+struct ResourceOwnerData;
+
+/* typedef is in public header */
+struct PgAioHandle
+{
+ /* all state updates should go through pgaio_io_update_state() */
+ PgAioHandleState state:8;
+
+ /* what are we operating on */
+ PgAioTargetID target:8;
+
+ /* which IO operation */
+ PgAioOp op:8;
+
+ /* bitfield of PgAioHandleFlags */
+ uint8 flags;
+
+ uint8 num_shared_callbacks;
+
+ /* using the proper type here would use more space */
+ uint8 shared_callbacks[PGAIO_HANDLE_MAX_CALLBACKS];
+
+ /*
+ * Length of data associated with handle using
+ * pgaio_io_set_handle_data_*().
+ */
+ uint8 handle_data_len;
+
+ /* XXX: could be optimized out with some pointer math */
+ int32 owner_procno;
+
+ /* raw result of the IO operation */
+ int32 result;
+
+ /*
+ * Index into PgAioCtl->iovecs and PgAioCtl->handle_data.
+ *
+ * At the moment there's no need to differentiate between the two, but
+ * that won't necessarily stay that way.
+ */
+ uint32 iovec_off;
+
+ /**
+ * In which list the handle is registered, depends on the state:
+ * - IDLE, in per-backend list
+ * - HANDED_OUT - not in a list
+ * - DEFINED - in per-backend staged list
+ * - STAGED - in per-backend staged list
+ * - SUBMITTED - in issuer's in_flight list
+ * - COMPLETED_IO - in issuer's in_flight list
+ * - COMPLETED_SHARED - in issuer's in_flight list
+ **/
+ dlist_node node;
+
+ struct ResourceOwnerData *resowner;
+ dlist_node resowner_node;
+
+ /* incremented every time the IO handle is reused */
+ uint64 generation;
+
+ ConditionVariable cv;
+
+ /* result of shared callback, passed to issuer callback */
+ PgAioResult distilled_result;
+
+ PgAioReturn *report_return;
+
+ PgAioOpData op_data;
+
+ /*
+ * Data necessary to identify the object undergoing IO to higher-level
+ * code. Needs to be sufficient to allow another backend to reopen the
+ * file.
+ */
+ PgAioTargetData target_data;
+};
+
+
+typedef struct PgAioBackend
+{
+ /* index into PgAioCtl->io_handles */
+ uint32 io_handle_off;
+
+ /* IO Handles that currently are not used */
+ dclist_head idle_ios;
+
+ /*
+ * Only one IO may be returned by pgaio_io_acquire()/pgaio_io_acquire()
+ * without having been either defined (by actually associating it with IO)
+ * or by released (with pgaio_io_release()). This restriction is necessary
+ * to guarantee that we always can acquire an IO. ->handed_out_io is used
+ * to enforce that rule.
+ */
+ PgAioHandle *handed_out_io;
+
+ /*
+ * IOs that are defined, but not yet submitted.
+ */
+ uint16 num_staged_ios;
+ PgAioHandle *staged_ios[PGAIO_SUBMIT_BATCH_SIZE];
+
+ /*
+ * List of in-flight IOs. Also contains IOs that aren't strict speaking
+ * in-flight anymore, but have been waited-for and completed by another
+ * backend. Once this backend sees such an IO it'll be reclaimed.
+ *
+ * The list is ordered by submission time, with more recently submitted
+ * IOs being appended at the end.
+ */
+ dclist_head in_flight_ios;
+} PgAioBackend;
+
+
+typedef struct PgAioCtl
+{
+ int backend_state_count;
+ PgAioBackend *backend_state;
+
+ /*
+ * Array of iovec structs. Each iovec is owned by a specific backend. The
+ * allocation is in PgAioCtl to allow the maximum number of iovecs for
+ * individual IOs to be configurable with PGC_POSTMASTER GUC.
+ */
+ uint64 iovec_count;
+ struct iovec *iovecs;
+
+ /*
+ * For, e.g., an IO covering multiple buffers in shared / temp buffers, we
+ * need to get Buffer IDs during completion to be able to change the
+ * BufferDesc state accordingly. This space can be used to store e.g.
+ * Buffer IDs. Note that the actual iovec might be shorter than this,
+ * because we combine neighboring pages into one larger iovec entry.
+ */
+ uint64 *handle_data;
+
+ uint64 io_handle_count;
+ PgAioHandle *io_handles;
+} PgAioCtl;
+
+
+
+/*
+ * The set of callbacks that each IO method must implement.
+ *
+ * AFIXME: Document these.
+ */
+typedef struct IoMethodOps
+{
+ /* global initialization */
+ size_t (*shmem_size) (void);
+ void (*shmem_init) (bool first_time);
+
+ /* per-backend initialization */
+ void (*init_backend) (void);
+
+ /* handling of IOs */
+ bool (*needs_synchronous_execution) (PgAioHandle *ioh);
+ int (*submit) (uint16 num_staged_ios, PgAioHandle **staged_ios);
+
+ void (*wait_one) (PgAioHandle *ioh,
+ uint64 ref_generation);
+} IoMethodOps;
+
+
+/* aio.c */
+extern bool pgaio_io_was_recycled(PgAioHandle *ioh, uint64 ref_generation, PgAioHandleState *state);
+extern void pgaio_io_stage(PgAioHandle *ioh, PgAioOp op);
+extern void pgaio_io_process_completion(PgAioHandle *ioh, int result);
+extern void pgaio_io_prepare_submit(PgAioHandle *ioh);
+extern bool pgaio_io_needs_synchronous_execution(PgAioHandle *ioh);
+extern const char *pgaio_io_get_state_name(PgAioHandle *ioh);
+extern void pgaio_shutdown(int code, Datum arg);
+
+/* aio_callback.c */
+extern void pgaio_io_call_stage(PgAioHandle *ioh);
+extern void pgaio_io_call_complete_shared(PgAioHandle *ioh);
+extern void pgaio_io_call_complete_local(PgAioHandle *ioh);
+
+/* aio_io.c */
+extern void pgaio_io_perform_synchronously(PgAioHandle *ioh);
+extern const char *pgaio_io_get_op_name(PgAioHandle *ioh);
+
+/* aio_target.c */
+extern bool pgaio_io_can_reopen(PgAioHandle *ioh);
+extern void pgaio_io_reopen(PgAioHandle *ioh);
+extern const char *pgaio_io_get_target_name(PgAioHandle *ioh);
+
+
+/*
+ * The AIO subsystem has fairly verbose debug logging support. This can be
+ * enabled/disabled at buildtime. The reason for this is that
+ * a) the verbosity can make debugging things on higher levels hard
+ * b) even if logging can be skipped due to elevel checks, it still causes a
+ * measurable slowdown
+ */
+#define PGAIO_VERBOSE 1
+
+/*
+ * Simple ereport() wrapper that only logs if PGAIO_VERBOSE is defined.
+ *
+ * This intentionally still compiles the code, guarded by a constant if (0),
+ * if verbose logging is disabled, to make it less likely that debug logging
+ * is silently broken.
+ *
+ * The current definition requires passing at least one argument.
+ */
+#define pgaio_debug(elevel, msg, ...) \
+ do { \
+ if (PGAIO_VERBOSE) \
+ ereport(elevel, \
+ errhidestmt(true), errhidecontext(true), \
+ errmsg_internal(msg, \
+ __VA_ARGS__)); \
+ } while(0)
+
+/*
+ * Simple ereport() wrapper. Note that the definition requires passing at
+ * least one argument.
+ */
+#define pgaio_debug_io(elevel, ioh, msg, ...) \
+ pgaio_debug(elevel, "io %-10d|op %-5s|target %-4s|state %-16s: " msg, \
+ pgaio_io_get_id(ioh), \
+ pgaio_io_get_op_name(ioh), \
+ pgaio_io_get_target_name(ioh), \
+ pgaio_io_get_state_name(ioh), \
+ __VA_ARGS__)
+
+
+/* Declarations for the tables of function pointers exposed by each IO method. */
+extern PGDLLIMPORT const IoMethodOps pgaio_sync_ops;
+
+extern PGDLLIMPORT const IoMethodOps *pgaio_method_ops;
+extern PGDLLIMPORT PgAioCtl *pgaio_ctl;
+extern PGDLLIMPORT PgAioBackend *pgaio_my_backend;
+
+
+
+#endif /* AIO_INTERNAL_H */
diff --git a/src/include/storage/aio_types.h b/src/include/storage/aio_types.h
new file mode 100644
index 00000000000..d2617139a25
--- /dev/null
+++ b/src/include/storage/aio_types.h
@@ -0,0 +1,115 @@
+/*-------------------------------------------------------------------------
+ *
+ * aio_types.h
+ * AIO related types that are useful to include separately, to reduce the
+ * "include burden".
+ *
+ *
+ * Portions Copyright (c) 1996-2025, PostgreSQL Global Development Group
+ * Portions Copyright (c) 1994, Regents of the University of California
+ *
+ * src/include/storage/aio_types.h
+ *
+ *-------------------------------------------------------------------------
+ */
+#ifndef AIO_TYPES_H
+#define AIO_TYPES_H
+
+#include "storage/block.h"
+#include "storage/relfilelocator.h"
+
+
+typedef struct PgAioHandle PgAioHandle;
+
+/*
+ * A reference to an IO that can be used to wait for the IO (using
+ * pgaio_wref_wait()) to complete.
+ *
+ * These can be passed across process boundaries.
+ */
+typedef struct PgAioWaitRef
+{
+ /* internal ID identifying the specific PgAioHandle */
+ uint32 aio_index;
+
+ /*
+ * IO handles are reused. To detect if a handle was reused, and thereby
+ * avoid unnecessarily waiting for a newer IO, each time the handle is
+ * reused a generation number is increased.
+ *
+ * To avoid requiring alignment sufficient for an int64, split the
+ * generation into two.
+ */
+ uint32 generation_upper;
+ uint32 generation_lower;
+} PgAioWaitRef;
+
+
+/*
+ * Information identifying what the IO is being performed on.
+ *
+ * This needs sufficient information to
+ *
+ * a) Reopen the file for the IO if the IO is executed in a context that
+ * cannot use the FD provided initially (e.g. because the IO is executed in
+ * a worker process).
+ *
+ * b) Describe the object the IO is performed on in log / error messages.
+ */
+typedef union PgAioTargetData
+{
+ /* just as an example placeholder for later */
+ struct
+ {
+ uint32 queue_id;
+ } wal;
+} PgAioTargetData;
+
+
+/*
+ * The status of an AIO operation.
+ */
+typedef enum PgAioResultStatus
+{
+ ARS_UNKNOWN, /* not yet completed / uninitialized */
+ ARS_OK,
+ ARS_PARTIAL, /* did not fully succeed, but no error */
+ ARS_ERROR,
+} PgAioResultStatus;
+
+
+/*
+ * Result of IO operation, visible only to the initiator of IO.
+ */
+typedef struct PgAioResult
+{
+ /*
+ * This is of type PgAioHandleCallbackID, but can't use a bitfield of an
+ * enum, because some compilers treat enums as signed.
+ */
+ uint32 id:8;
+
+ /* of type PgAioResultStatus, see above */
+ uint32 status:2;
+
+ /* meaning defined by callback->error */
+ uint32 error_data:22;
+
+ int32 result;
+} PgAioResult;
+
+
+/*
+ * Combination of PgAioResult with minimal metadata about the IO.
+ *
+ * Contains sufficient information to be able, in case the IO [partially]
+ * fails, to log/raise an error under control of the IO issuing code.
+ */
+typedef struct PgAioReturn
+{
+ PgAioResult result;
+ PgAioTargetData target_data;
+} PgAioReturn;
+
+
+#endif /* AIO_TYPES_H */
diff --git a/src/include/utils/resowner.h b/src/include/utils/resowner.h
index e8d452ca7ee..aede4bfc820 100644
--- a/src/include/utils/resowner.h
+++ b/src/include/utils/resowner.h
@@ -164,4 +164,9 @@ struct LOCALLOCK;
extern void ResourceOwnerRememberLock(ResourceOwner owner, struct LOCALLOCK *locallock);
extern void ResourceOwnerForgetLock(ResourceOwner owner, struct LOCALLOCK *locallock);
+/* special support for AIO */
+struct dlist_node;
+extern void ResourceOwnerRememberAioHandle(ResourceOwner owner, struct dlist_node *ioh_node);
+extern void ResourceOwnerForgetAioHandle(ResourceOwner owner, struct dlist_node *ioh_node);
+
#endif /* RESOWNER_H */
diff --git a/src/backend/access/transam/xact.c b/src/backend/access/transam/xact.c
index d331ab90d78..a252c3a81b4 100644
--- a/src/backend/access/transam/xact.c
+++ b/src/backend/access/transam/xact.c
@@ -51,6 +51,7 @@
#include "replication/origin.h"
#include "replication/snapbuild.h"
#include "replication/syncrep.h"
+#include "storage/aio.h"
#include "storage/condition_variable.h"
#include "storage/fd.h"
#include "storage/lmgr.h"
@@ -2475,6 +2476,8 @@ CommitTransaction(void)
AtEOXact_LogicalRepWorkers(true);
pgstat_report_xact_timestamp(0);
+ pgaio_at_xact_end( /* is_subxact = */ false, /* is_commit = */ true);
+
ResourceOwnerDelete(TopTransactionResourceOwner);
s->curTransactionOwner = NULL;
CurTransactionResourceOwner = NULL;
@@ -2988,6 +2991,8 @@ AbortTransaction(void)
pgstat_report_xact_timestamp(0);
}
+ pgaio_at_xact_end( /* is_subxact = */ false, /* is_commit = */ false);
+
/*
* State remains TRANS_ABORT until CleanupTransaction().
*/
@@ -5185,6 +5190,8 @@ CommitSubTransaction(void)
AtEOSubXact_PgStat(true, s->nestingLevel);
AtSubCommit_Snapshot(s->nestingLevel);
+ pgaio_at_xact_end( /* is_subxact = */ true, /* is_commit = */ true);
+
/*
* We need to restore the upper transaction's read-only state, in case the
* upper is read-write while the child is read-only; GUC will incorrectly
@@ -5351,6 +5358,8 @@ AbortSubTransaction(void)
AtSubAbort_Snapshot(s->nestingLevel);
}
+ pgaio_at_xact_end( /* is_subxact = */ true, /* is_commit = */ false);
+
/*
* Restore the upper transaction's read-only state, too. This should be
* redundant with GUC's cleanup but we may as well do it for consistency
diff --git a/src/backend/storage/aio/Makefile b/src/backend/storage/aio/Makefile
index eaeaeeee8e3..89f821ea7e1 100644
--- a/src/backend/storage/aio/Makefile
+++ b/src/backend/storage/aio/Makefile
@@ -10,7 +10,11 @@ include $(top_builddir)/src/Makefile.global
OBJS = \
aio.o \
+ aio_callback.o \
aio_init.o \
+ aio_io.o \
+ aio_target.o \
+ method_sync.o \
read_stream.o
include $(top_srcdir)/src/backend/common.mk
diff --git a/src/backend/storage/aio/aio.c b/src/backend/storage/aio/aio.c
index f68cbc2b3f4..cefa888884c 100644
--- a/src/backend/storage/aio/aio.c
+++ b/src/backend/storage/aio/aio.c
@@ -3,6 +3,28 @@
* aio.c
* AIO - Core Logic
*
+ * For documentation about how AIO works on a higher level, including a
+ * schematic example, see README.md.
+ *
+ *
+ * AIO is a complicated subsystem. To keep things navigable it is split across
+ * a number of files:
+ *
+ * - method_*.c - different ways of executing AIO (e.g. worker process)
+ *
+ * - aio_target.c - IO on different kinds of targets
+ *
+ * - aio_io.c - method-independent code for specific IO ops (e.g. readv)
+ *
+ * - aio_callback.c - callbacks at IO operation lifecycle events
+ *
+ * - aio_init.c - per-server and per-backend initialization
+ *
+ * - aio.c - all other topics
+ *
+ * - read_stream.c - helper for reading buffered relation data
+ *
+ *
* Portions Copyright (c) 1996-2025, PostgreSQL Global Development Group
* Portions Copyright (c) 1994, Regents of the University of California
*
@@ -14,8 +36,22 @@
#include "postgres.h"
+#include "miscadmin.h"
+#include "port/atomics.h"
#include "storage/aio.h"
+#include "storage/aio_internal.h"
#include "utils/guc.h"
+#include "utils/resowner.h"
+#include "utils/wait_event_types.h"
+
+
+static inline void pgaio_io_update_state(PgAioHandle *ioh, PgAioHandleState new_state);
+static void pgaio_io_reclaim(PgAioHandle *ioh);
+static void pgaio_io_resowner_register(PgAioHandle *ioh);
+static void pgaio_io_wait_for_free(void);
+static PgAioHandle *pgaio_io_from_wref(PgAioWaitRef *iow, uint64 *ref_generation);
+static const char *pgaio_io_state_get_name(PgAioHandleState s);
+static void pgaio_io_wait(PgAioHandle *ioh, uint64 ref_generation);
/* Options for io_method. */
@@ -28,9 +64,877 @@ const struct config_enum_entry io_method_options[] = {
int io_method = DEFAULT_IO_METHOD;
int io_max_concurrency = -1;
+/* global control for AIO */
+PgAioCtl *pgaio_ctl;
+/* current backend's per-backend state */
+PgAioBackend *pgaio_my_backend;
+
+
+static const IoMethodOps *const pgaio_method_ops_table[] = {
+ [IOMETHOD_SYNC] = &pgaio_sync_ops,
+};
+
+/* callbacks for the configured io_method, set by assign_io_method */
+const IoMethodOps *pgaio_method_ops;
+
+
+
+/* --------------------------------------------------------------------------------
+ * Public Functions related to PgAioHandle
+ * --------------------------------------------------------------------------------
+ */
+
+/*
+ * Acquire an AioHandle, waiting for IO completion if necessary.
+ *
+ * Each backend can only have one AIO handle that that has been "handed out"
+ * to code, but not yet submitted or released. This restriction is necessary
+ * to ensure that it is possible for code to wait for an unused handle by
+ * waiting for in-flight IO to complete. There is a limited number of handles
+ * in each backend, if multiple handles could be handed out without being
+ * submitted, waiting for all in-flight IO to complete would not guarantee
+ * that handles free up.
+ *
+ * It is cheap to acquire an IO handle, unless all handles are in use. In that
+ * case this function waits for the oldest IO to complete. In case that is not
+ * desirable, see pgaio_io_acquire_nb().
+ *
+ * If a handle was acquired but then does not turn out to be needed,
+ * e.g. because pgaio_io_acquire() is called before starting an IO in a
+ * critical section, the handle needs to be released with pgaio_io_release().
+ *
+ *
+ * To react to the completion of the IO as soon as it is known to have
+ * completed, callbacks can be registered with pgaio_io_register_callbacks().
+ *
+ * To actually execute IO using the returned handle, the pgaio_io_prep_*()
+ * family of functions is used. In many cases the pgaio_io_prep_*() call will
+ * not be done directly by code that acquired the handle, but by lower level
+ * code that gets passed the handle. E.g. if code in bufmgr.c wants to perform
+ * AIO, it typically will pass the handle to smgr., which will pass it on to
+ * md.c, on to fd.c, which then finally calls pgaio_io_prep_*(). This
+ * forwarding allows the various layers to react to the IO's completion by
+ * registering callbacks. These callbacks in turn can translate a lower
+ * layer's result into a result understandable by a higher layer.
+ *
+ * Once pgaio_io_prep_*() is called, the IO may be in the process of being
+ * executed and might even complete before the functions return. That is,
+ * however, not guaranteed, to allow IO submission to be batched. To guarantee
+ * IO submission pgaio_submit_staged() needs to be called.
+ *
+ * After pgaio_io_prep_*() the AioHandle is "consumed" and may not be
+ * referenced by the IO issuing code. To e.g. wait for IO, references to the
+ * IO can be established with pgaio_io_get_wref() *before* pgaio_io_prep_*()
+ * is called. pgaio_wref_wait() can be used to wait for the IO to complete.
+ *
+ *
+ * To know if the IO [partially] succeeded or failed, a PgAioReturn * can be
+ * passed to pgaio_io_acquire(). Once the issuing backend has called
+ * pgaio_wref_wait(), the PgAioReturn contains information about whether the
+ * operation succeeded and details about the first failure, if any. The error
+ * can be raised / logged with pgaio_result_report().
+ *
+ * The lifetime of the memory pointed to be *ret needs to be at least as long
+ * as the passed in resowner. If the resowner releases resources before the IO
+ * completes (typically due to an error), the reference to *ret will be
+ * cleared. In case of resowner cleanup *ret will not be updated with the
+ * results of the IO operation.
+ */
+PgAioHandle *
+pgaio_io_acquire(struct ResourceOwnerData *resowner, PgAioReturn *ret)
+{
+ PgAioHandle *h;
+
+ while (true)
+ {
+ h = pgaio_io_acquire_nb(resowner, ret);
+
+ if (h != NULL)
+ return h;
+
+ /*
+ * Evidently all handles by this backend are in use. Just wait for
+ * some to complete.
+ */
+ pgaio_io_wait_for_free();
+ }
+}
+
+/*
+ * Acquire an AioHandle, returning NULL if no handles are free.
+ *
+ * See pgaio_io_acquire(). The only difference is that this function will return
+ * NULL if there are no idle handles, instead of blocking.
+ */
+PgAioHandle *
+pgaio_io_acquire_nb(struct ResourceOwnerData *resowner, PgAioReturn *ret)
+{
+ if (pgaio_my_backend->num_staged_ios >= PGAIO_SUBMIT_BATCH_SIZE)
+ {
+ Assert(pgaio_my_backend->num_staged_ios == PGAIO_SUBMIT_BATCH_SIZE);
+ pgaio_submit_staged();
+ }
+
+ if (pgaio_my_backend->handed_out_io)
+ {
+ ereport(ERROR,
+ errmsg("API violation: Only one IO can be handed out"));
+ }
+
+ if (!dclist_is_empty(&pgaio_my_backend->idle_ios))
+ {
+ dlist_node *ion = dclist_pop_head_node(&pgaio_my_backend->idle_ios);
+ PgAioHandle *ioh = dclist_container(PgAioHandle, node, ion);
+
+ Assert(ioh->state == PGAIO_HS_IDLE);
+ Assert(ioh->owner_procno == MyProcNumber);
+
+ pgaio_io_update_state(ioh, PGAIO_HS_HANDED_OUT);
+ pgaio_my_backend->handed_out_io = ioh;
+
+ if (resowner)
+ pgaio_io_resowner_register(ioh);
+
+ if (ret)
+ {
+ ioh->report_return = ret;
+ ret->result.status = ARS_UNKNOWN;
+ }
+
+ return ioh;
+ }
+
+ return NULL;
+}
+
+/*
+ * Release IO handle that turned out to not be required.
+ *
+ * See pgaio_io_acquire() for more details.
+ */
+void
+pgaio_io_release(PgAioHandle *ioh)
+{
+ if (ioh == pgaio_my_backend->handed_out_io)
+ {
+ Assert(ioh->state == PGAIO_HS_HANDED_OUT);
+ Assert(ioh->resowner);
+
+ pgaio_my_backend->handed_out_io = NULL;
+ pgaio_io_reclaim(ioh);
+ }
+ else
+ {
+ elog(ERROR, "release in unexpected state");
+ }
+}
+
+/*
+ * Release IO handle during resource owner cleanup.
+ */
+void
+pgaio_io_release_resowner(dlist_node *ioh_node, bool on_error)
+{
+ PgAioHandle *ioh = dlist_container(PgAioHandle, resowner_node, ioh_node);
+
+ Assert(ioh->resowner);
+
+ ResourceOwnerForgetAioHandle(ioh->resowner, &ioh->resowner_node);
+ ioh->resowner = NULL;
+
+ switch (ioh->state)
+ {
+ case PGAIO_HS_IDLE:
+ elog(ERROR, "unexpected");
+ break;
+ case PGAIO_HS_HANDED_OUT:
+ Assert(ioh == pgaio_my_backend->handed_out_io || pgaio_my_backend->handed_out_io == NULL);
+
+ if (ioh == pgaio_my_backend->handed_out_io)
+ {
+ pgaio_my_backend->handed_out_io = NULL;
+ if (!on_error)
+ elog(WARNING, "leaked AIO handle");
+ }
+
+ pgaio_io_reclaim(ioh);
+ break;
+ case PGAIO_HS_DEFINED:
+ case PGAIO_HS_STAGED:
+ /* XXX: Should we warn about this when is_commit? */
+ pgaio_submit_staged();
+ break;
+ case PGAIO_HS_SUBMITTED:
+ case PGAIO_HS_COMPLETED_IO:
+ case PGAIO_HS_COMPLETED_SHARED:
+ case PGAIO_HS_COMPLETED_LOCAL:
+ /* this is expected to happen */
+ break;
+ }
+
+ /*
+ * Need to unregister the reporting of the IO's result, the memory it's
+ * referencing likely has gone away.
+ */
+ if (ioh->report_return)
+ ioh->report_return = NULL;
+}
+
+/*
+ * Add a [set of] flags to the IO.
+ *
+ * Note that this combines flags with already set flags, rather than set flags
+ * to explicitly the passed in parameters. This is to allow multiple callsites
+ * to set flags.
+ */
+void
+pgaio_io_set_flag(PgAioHandle *ioh, PgAioHandleFlags flag)
+{
+ Assert(ioh->state == PGAIO_HS_HANDED_OUT);
+
+ ioh->flags |= flag;
+}
+
+int
+pgaio_io_get_id(PgAioHandle *ioh)
+{
+ Assert(ioh >= pgaio_ctl->io_handles &&
+ ioh < (pgaio_ctl->io_handles + pgaio_ctl->io_handle_count));
+ return ioh - pgaio_ctl->io_handles;
+}
+
+ProcNumber
+pgaio_io_get_owner(PgAioHandle *ioh)
+{
+ return ioh->owner_procno;
+}
+
+void
+pgaio_io_get_wref(PgAioHandle *ioh, PgAioWaitRef *iow)
+{
+ Assert(ioh->state == PGAIO_HS_HANDED_OUT ||
+ ioh->state == PGAIO_HS_DEFINED ||
+ ioh->state == PGAIO_HS_STAGED);
+ Assert(ioh->generation != 0);
+
+ iow->aio_index = ioh - pgaio_ctl->io_handles;
+ iow->generation_upper = (uint32) (ioh->generation >> 32);
+ iow->generation_lower = (uint32) ioh->generation;
+}
+
+
+
+/* --------------------------------------------------------------------------------
+ * Internal Functions related to PgAioHandle
+ * --------------------------------------------------------------------------------
+ */
+
+static inline void
+pgaio_io_update_state(PgAioHandle *ioh, PgAioHandleState new_state)
+{
+ pgaio_debug_io(DEBUG4, ioh,
+ "updating state to %s",
+ pgaio_io_state_get_name(new_state));
+
+ /*
+ * Ensure the changes signified by the new state are visible before the
+ * new state becomes visible.
+ */
+ pg_write_barrier();
+
+ ioh->state = new_state;
+}
+
+static void
+pgaio_io_resowner_register(PgAioHandle *ioh)
+{
+ Assert(!ioh->resowner);
+ Assert(CurrentResourceOwner);
+
+ ResourceOwnerRememberAioHandle(CurrentResourceOwner, &ioh->resowner_node);
+ ioh->resowner = CurrentResourceOwner;
+}
+
+/*
+ * Should only be called from pgaio_io_prep_*().
+ */
+void
+pgaio_io_stage(PgAioHandle *ioh, PgAioOp op)
+{
+ bool needs_synchronous;
+
+ Assert(ioh->state == PGAIO_HS_HANDED_OUT);
+ Assert(pgaio_io_has_target(ioh));
+
+ ioh->op = op;
+ ioh->result = 0;
+
+ pgaio_io_update_state(ioh, PGAIO_HS_DEFINED);
+
+ /* allow a new IO to be staged */
+ pgaio_my_backend->handed_out_io = NULL;
+
+ pgaio_io_call_stage(ioh);
+
+ pgaio_io_update_state(ioh, PGAIO_HS_STAGED);
+
+ needs_synchronous = pgaio_io_needs_synchronous_execution(ioh);
+
+ pgaio_debug_io(DEBUG3, ioh,
+ "prepared, executing synchronously: %d",
+ needs_synchronous);
+
+ if (!needs_synchronous)
+ {
+ pgaio_my_backend->staged_ios[pgaio_my_backend->num_staged_ios++] = ioh;
+ Assert(pgaio_my_backend->num_staged_ios <= PGAIO_SUBMIT_BATCH_SIZE);
+ }
+ else
+ {
+ pgaio_io_prepare_submit(ioh);
+ pgaio_io_perform_synchronously(ioh);
+ }
+}
+
+bool
+pgaio_io_needs_synchronous_execution(PgAioHandle *ioh)
+{
+ if (ioh->flags & PGAIO_HF_SYNCHRONOUS)
+ {
+ /* XXX: should we also check if there are other IOs staged? */
+ return true;
+ }
+
+ if (pgaio_method_ops->needs_synchronous_execution)
+ return pgaio_method_ops->needs_synchronous_execution(ioh);
+ return false;
+}
+
+/*
+ * Handle IO being processed by IO method.
+ *
+ * Should be called by IO methods / synchronous IO execution, just before the
+ * IO is performed.
+ */
+void
+pgaio_io_prepare_submit(PgAioHandle *ioh)
+{
+ pgaio_io_update_state(ioh, PGAIO_HS_SUBMITTED);
+
+ dclist_push_tail(&pgaio_my_backend->in_flight_ios, &ioh->node);
+}
+
+/*
+ * Handle IO getting completed by a method.
+ *
+ * Should be called by IO methods / synchronous IO execution
+ */
+void
+pgaio_io_process_completion(PgAioHandle *ioh, int result)
+{
+ Assert(ioh->state == PGAIO_HS_SUBMITTED);
+
+ ioh->result = result;
+
+ pgaio_io_update_state(ioh, PGAIO_HS_COMPLETED_IO);
+
+ pgaio_io_call_complete_shared(ioh);
+
+ pgaio_io_update_state(ioh, PGAIO_HS_COMPLETED_SHARED);
+
+ /* condition variable broadcast ensures state is visible before wakeup */
+ ConditionVariableBroadcast(&ioh->cv);
+
+ if (ioh->owner_procno == MyProcNumber)
+ pgaio_io_reclaim(ioh);
+}
+
+bool
+pgaio_io_was_recycled(PgAioHandle *ioh, uint64 ref_generation, PgAioHandleState *state)
+{
+ *state = ioh->state;
+ pg_read_barrier();
+
+ return ioh->generation != ref_generation;
+}
+
+/*
+ * Wait for IO to complete. External code should never use this, outside of
+ * the AIO subsystem waits are only allowed via pgaio_wref_wait().
+ */
+static void
+pgaio_io_wait(PgAioHandle *ioh, uint64 ref_generation)
+{
+ PgAioHandleState state;
+ bool am_owner;
+
+ am_owner = ioh->owner_procno == MyProcNumber;
+
+ if (pgaio_io_was_recycled(ioh, ref_generation, &state))
+ return;
+
+ if (am_owner)
+ {
+ if (state == PGAIO_HS_STAGED)
+ {
+ /* XXX: Arguably this should be prevented by callers? */
+ pgaio_submit_staged();
+ }
+ else if (state != PGAIO_HS_SUBMITTED
+ && state != PGAIO_HS_COMPLETED_IO
+ && state != PGAIO_HS_COMPLETED_SHARED
+ && state != PGAIO_HS_COMPLETED_LOCAL)
+ {
+ elog(PANIC, "waiting for own IO in wrong state: %d",
+ state);
+ }
+ }
+
+ while (true)
+ {
+ if (pgaio_io_was_recycled(ioh, ref_generation, &state))
+ return;
+
+ switch (state)
+ {
+ case PGAIO_HS_IDLE:
+ case PGAIO_HS_HANDED_OUT:
+ elog(ERROR, "IO in wrong state: %d", state);
+ break;
+
+ case PGAIO_HS_SUBMITTED:
+
+ /*
+ * If we need to wait via the IO method, do so now. Don't
+ * check via the IO method if the issuing backend is executing
+ * the IO synchronously.
+ */
+ if (pgaio_method_ops->wait_one && !(ioh->flags & PGAIO_HF_SYNCHRONOUS))
+ {
+ pgaio_method_ops->wait_one(ioh, ref_generation);
+ continue;
+ }
+ /* fallthrough */
+
+ /* waiting for owner to submit */
+ case PGAIO_HS_DEFINED:
+ case PGAIO_HS_STAGED:
+ /* waiting for reaper to complete */
+ /* fallthrough */
+ case PGAIO_HS_COMPLETED_IO:
+ /* shouldn't be able to hit this otherwise */
+ Assert(IsUnderPostmaster);
+ /* ensure we're going to get woken up */
+ ConditionVariablePrepareToSleep(&ioh->cv);
+
+ while (!pgaio_io_was_recycled(ioh, ref_generation, &state))
+ {
+ if (state == PGAIO_HS_COMPLETED_SHARED ||
+ state == PGAIO_HS_COMPLETED_LOCAL)
+ break;
+ ConditionVariableSleep(&ioh->cv, WAIT_EVENT_AIO_COMPLETION);
+ }
+
+ ConditionVariableCancelSleep();
+ break;
+
+ case PGAIO_HS_COMPLETED_SHARED:
+ case PGAIO_HS_COMPLETED_LOCAL:
+ /* see above */
+ if (am_owner)
+ pgaio_io_reclaim(ioh);
+ return;
+ }
+ }
+}
+
+static void
+pgaio_io_reclaim(PgAioHandle *ioh)
+{
+ /* This is only ok if it's our IO */
+ Assert(ioh->owner_procno == MyProcNumber);
+
+ pgaio_debug_io(DEBUG4, ioh,
+ "reclaiming, result: %d, distilled_result: AFIXME, report to: %p",
+ ioh->result,
+ ioh->report_return);
+
+ if (ioh->state == PGAIO_HS_COMPLETED_SHARED)
+ {
+ pgaio_io_call_complete_local(ioh);
+ pgaio_io_update_state(ioh, PGAIO_HS_COMPLETED_LOCAL);
+ }
+
+ /* if the IO has been defined, we might need to do more work */
+ if (ioh->state != PGAIO_HS_HANDED_OUT)
+ {
+ dclist_delete_from(&pgaio_my_backend->in_flight_ios, &ioh->node);
+
+ if (ioh->report_return)
+ {
+ ioh->report_return->result = ioh->distilled_result;
+ ioh->report_return->target_data = ioh->target_data;
+ }
+ }
+
+ if (ioh->resowner)
+ {
+ ResourceOwnerForgetAioHandle(ioh->resowner, &ioh->resowner_node);
+ ioh->resowner = NULL;
+ }
+
+ Assert(!ioh->resowner);
+
+ ioh->op = PGAIO_OP_INVALID;
+ ioh->target = PGAIO_TID_INVALID;
+ ioh->flags = 0;
+ ioh->num_shared_callbacks = 0;
+ ioh->handle_data_len = 0;
+ ioh->report_return = NULL;
+ ioh->result = 0;
+ ioh->distilled_result.status = ARS_UNKNOWN;
+
+ /* XXX: the barrier is probably superfluous */
+ pg_write_barrier();
+ ioh->generation++;
+
+ pgaio_io_update_state(ioh, PGAIO_HS_IDLE);
+
+ /*
+ * We push the IO to the head of the idle IO list, that seems more cache
+ * efficient in cases where only a few IOs are used.
+ */
+ dclist_push_head(&pgaio_my_backend->idle_ios, &ioh->node);
+}
+
+static void
+pgaio_io_wait_for_free(void)
+{
+ int reclaimed = 0;
+
+ pgaio_debug(DEBUG2, "waiting for self with %d pending",
+ pgaio_my_backend->num_staged_ios);
+
+ /*
+ * First check if any of our IOs actually have completed - when using
+ * worker, that'll often be the case. We could do so as part of the loop
+ * below, but that'd potentially lead us to wait for some IO submitted
+ * before.
+ */
+ for (int i = 0; i < io_max_concurrency; i++)
+ {
+ PgAioHandle *ioh = &pgaio_ctl->io_handles[pgaio_my_backend->io_handle_off + i];
+
+ if (ioh->state == PGAIO_HS_COMPLETED_SHARED)
+ {
+ pgaio_io_reclaim(ioh);
+ reclaimed++;
+ }
+ }
+
+ if (reclaimed > 0)
+ return;
+
+ /*
+ * If we have any unsubmitted IOs, submit them now. We'll start waiting in
+ * a second, so it's better they're in flight. This also addresses the
+ * edge-case that all IOs are unsubmitted.
+ */
+ if (pgaio_my_backend->num_staged_ios > 0)
+ {
+ pgaio_submit_staged();
+ }
+
+ /*
+ * It's possible that we recognized there were free IOs while submitting.
+ */
+ if (dclist_count(&pgaio_my_backend->in_flight_ios) == 0)
+ {
+ elog(ERROR, "no free IOs despite no in-flight IOs");
+ }
+
+ /*
+ * Wait for the oldest in-flight IO to complete.
+ *
+ * XXX: Reusing the general IO wait is suboptimal, we don't need to wait
+ * for that specific IO to complete, we just need *any* IO to complete.
+ */
+ {
+ PgAioHandle *ioh = dclist_head_element(PgAioHandle, node, &pgaio_my_backend->in_flight_ios);
+
+ switch (ioh->state)
+ {
+ /* should not be in in-flight list */
+ case PGAIO_HS_IDLE:
+ case PGAIO_HS_DEFINED:
+ case PGAIO_HS_HANDED_OUT:
+ case PGAIO_HS_STAGED:
+ case PGAIO_HS_COMPLETED_LOCAL:
+ elog(ERROR, "shouldn't get here with io:%d in state %d",
+ pgaio_io_get_id(ioh), ioh->state);
+ break;
+
+ case PGAIO_HS_COMPLETED_IO:
+ case PGAIO_HS_SUBMITTED:
+ pgaio_debug_io(DEBUG2, ioh,
+ "waiting for free io with %d in flight",
+ dclist_count(&pgaio_my_backend->in_flight_ios));
+
+ /*
+ * In a more general case this would be racy, because the
+ * generation could increase after we read ioh->state above.
+ * But we are only looking at IOs by the current backend and
+ * the IO can only be recycled by this backend.
+ */
+ pgaio_io_wait(ioh, ioh->generation);
+ break;
+
+ case PGAIO_HS_COMPLETED_SHARED:
+ /* it's possible that another backend just finished this IO */
+ pgaio_io_reclaim(ioh);
+ break;
+ }
+
+ if (dclist_count(&pgaio_my_backend->idle_ios) == 0)
+ elog(PANIC, "no idle IOs after waiting");
+ return;
+ }
+}
+
+/*
+ * Internal - code outside of AIO should never need this and it'd be hard for
+ * such code to be safe.
+ */
+static PgAioHandle *
+pgaio_io_from_wref(PgAioWaitRef *iow, uint64 *ref_generation)
+{
+ PgAioHandle *ioh;
+
+ Assert(iow->aio_index < pgaio_ctl->io_handle_count);
+
+ ioh = &pgaio_ctl->io_handles[iow->aio_index];
+
+ *ref_generation = ((uint64) iow->generation_upper) << 32 |
+ iow->generation_lower;
+
+ Assert(*ref_generation != 0);
+
+ return ioh;
+}
+
+static const char *
+pgaio_io_state_get_name(PgAioHandleState s)
+{
+#define PGAIO_HS_TOSTR_CASE(sym) case PGAIO_HS_##sym: return #sym
+ switch (s)
+ {
+ PGAIO_HS_TOSTR_CASE(IDLE);
+ PGAIO_HS_TOSTR_CASE(HANDED_OUT);
+ PGAIO_HS_TOSTR_CASE(DEFINED);
+ PGAIO_HS_TOSTR_CASE(STAGED);
+ PGAIO_HS_TOSTR_CASE(SUBMITTED);
+ PGAIO_HS_TOSTR_CASE(COMPLETED_IO);
+ PGAIO_HS_TOSTR_CASE(COMPLETED_SHARED);
+ PGAIO_HS_TOSTR_CASE(COMPLETED_LOCAL);
+ }
+#undef PGAIO_HS_TOSTR_CASE
+
+ return NULL; /* silence compiler */
+}
+
+const char *
+pgaio_io_get_state_name(PgAioHandle *ioh)
+{
+ return pgaio_io_state_get_name(ioh->state);
+}
+
+
+
+/* --------------------------------------------------------------------------------
+ * Functions primarily related to IO Wait References
+ * --------------------------------------------------------------------------------
+ */
+
+void
+pgaio_wref_clear(PgAioWaitRef *iow)
+{
+ iow->aio_index = PG_UINT32_MAX;
+}
+
+bool
+pgaio_wref_valid(PgAioWaitRef *iow)
+{
+ return iow->aio_index != PG_UINT32_MAX;
+}
+
+int
+pgaio_wref_get_id(PgAioWaitRef *iow)
+{
+ Assert(pgaio_wref_valid(iow));
+ return iow->aio_index;
+}
+
+/*
+ * Wait for the IO to have completed.
+ */
+void
+pgaio_wref_wait(PgAioWaitRef *iow)
+{
+ uint64 ref_generation;
+ PgAioHandle *ioh;
+
+ ioh = pgaio_io_from_wref(iow, &ref_generation);
+
+ pgaio_io_wait(ioh, ref_generation);
+}
+
+/*
+ * Check if the the referenced IO completed, without blocking.
+ */
+bool
+pgaio_wref_check_done(PgAioWaitRef *iow)
+{
+ uint64 ref_generation;
+ PgAioHandleState state;
+ bool am_owner;
+ PgAioHandle *ioh;
+
+ ioh = pgaio_io_from_wref(iow, &ref_generation);
+
+ if (pgaio_io_was_recycled(ioh, ref_generation, &state))
+ return true;
+
+ if (state == PGAIO_HS_IDLE)
+ return true;
+
+ am_owner = ioh->owner_procno == MyProcNumber;
+
+ if (state == PGAIO_HS_COMPLETED_SHARED ||
+ state == PGAIO_HS_COMPLETED_LOCAL)
+ {
+ if (am_owner)
+ pgaio_io_reclaim(ioh);
+ return true;
+ }
+
+ return false;
+}
+
+
+
+/* --------------------------------------------------------------------------------
+ * Actions on multiple IOs.
+ * --------------------------------------------------------------------------------
+ */
+
+void
+pgaio_submit_staged(void)
+{
+ int total_submitted = 0;
+ int did_submit;
+
+ if (pgaio_my_backend->num_staged_ios == 0)
+ return;
+
+
+ START_CRIT_SECTION();
+
+ did_submit = pgaio_method_ops->submit(pgaio_my_backend->num_staged_ios,
+ pgaio_my_backend->staged_ios);
+
+ END_CRIT_SECTION();
+
+ total_submitted += did_submit;
+
+ Assert(total_submitted == did_submit);
+
+ pgaio_my_backend->num_staged_ios = 0;
+
+ pgaio_debug(DEBUG4,
+ "aio: submitted %d IOs",
+ total_submitted);
+}
+
+bool
+pgaio_have_staged(void)
+{
+ return pgaio_my_backend->num_staged_ios > 0;
+}
+
+
+
+/* --------------------------------------------------------------------------------
+ * Other
+ * --------------------------------------------------------------------------------
+ */
+
+/*
+ * Need to submit staged but not yet submitted IOs using the fd, otherwise
+ * the IO would end up targeting something bogus.
+ */
+void
+pgaio_closing_fd(int fd)
+{
+ /*
+ * Might be called before AIO is initialized or in a subprocess that
+ * doesn't use AIO.
+ */
+ if (!pgaio_my_backend)
+ return;
+
+ /*
+ * For now just submit all staged IOs - we could be more selective, but
+ * it's probably not worth it.
+ */
+ pgaio_submit_staged();
+}
+
+void
+pgaio_at_xact_end(bool is_subxact, bool is_commit)
+{
+ Assert(!pgaio_my_backend->handed_out_io);
+}
+
+/*
+ * Similar to pgaio_at_xact_end(..., is_commit = false), but for cases where
+ * errors happen outside of transactions.
+ */
+void
+pgaio_at_error(void)
+{
+ Assert(!pgaio_my_backend->handed_out_io);
+}
+
+void
+pgaio_shutdown(int code, Datum arg)
+{
+ Assert(pgaio_my_backend);
+ Assert(!pgaio_my_backend->handed_out_io);
+
+ /*
+ * Before exiting, make sure that all IOs are finished. That has two main
+ * purposes: - it's somewhat annoying to see partially finished IOs in
+ * stats views etc - it's rumored that some kernel-level AIO mechanisms
+ * don't deal well with the issuer of an AIO exiting
+ */
+
+ while (!dclist_is_empty(&pgaio_my_backend->in_flight_ios))
+ {
+ PgAioHandle *ioh = dclist_head_element(PgAioHandle, node, &pgaio_my_backend->in_flight_ios);
+
+ /* see comment in pgaio_io_wait_for_free() about raciness */
+ pgaio_io_wait(ioh, ioh->generation);
+ }
+
+ pgaio_my_backend = NULL;
+}
void
assign_io_method(int newval, void *extra)
{
+ Assert(pgaio_method_ops_table[newval] != NULL);
+ Assert(newval < lengthof(io_method_options));
+
+ pgaio_method_ops = pgaio_method_ops_table[newval];
}
diff --git a/src/backend/storage/aio/aio_callback.c b/src/backend/storage/aio/aio_callback.c
new file mode 100644
index 00000000000..93f71690169
--- /dev/null
+++ b/src/backend/storage/aio/aio_callback.c
@@ -0,0 +1,280 @@
+/*-------------------------------------------------------------------------
+ *
+ * aio_callback.c
+ * AIO - Functionality related to callbacks that can be registered on IO
+ * Handles
+ *
+ * Portions Copyright (c) 1996-2025, PostgreSQL Global Development Group
+ * Portions Copyright (c) 1994, Regents of the University of California
+ *
+ * IDENTIFICATION
+ * src/backend/storage/aio/aio_callback.c
+ *
+ *-------------------------------------------------------------------------
+ */
+
+#include "postgres.h"
+
+#include "miscadmin.h"
+#include "storage/aio.h"
+#include "storage/aio_internal.h"
+#include "utils/memutils.h"
+
+
+/* just to have something to put into the aio_handle_cbs */
+static const struct PgAioHandleCallbacks aio_invalid_cb = {0};
+
+typedef struct PgAioHandleCallbacksEntry
+{
+ const PgAioHandleCallbacks *const cb;
+ const char *const name;
+} PgAioHandleCallbacksEntry;
+
+/*
+ * Callback definition for the callbacks that can be registered on an IO
+ * handle. See PgAioHandleCallbackID's definition for an explanation for why
+ * callbacks are not identified by a pointer.
+ */
+static const PgAioHandleCallbacksEntry aio_handle_cbs[] = {
+#define CALLBACK_ENTRY(id, callback) [id] = {.cb = &callback, .name = #callback}
+ CALLBACK_ENTRY(PGAIO_HCB_INVALID, aio_invalid_cb),
+#undef CALLBACK_ENTRY
+};
+
+
+
+/*
+ * Register callback for the IO handle.
+ *
+ * Only a limited number (PGAIO_HANDLE_MAX_CALLBACKS) of callbacks can be
+ * registered for each IO.
+ *
+ * Callbacks need to be registered before [indirectly] calling
+ * pgaio_io_prep_*(), as the IO may be executed immediately.
+ *
+ *
+ * Note that callbacks are executed in critical sections. This is necessary
+ * to be able to execute IO in critical sections (consider e.g. WAL
+ * logging). To perform AIO we first need to acquire a handle, which, if there
+ * are no free handles, requires waiting for IOs to complete and to execute
+ * their completion callbacks.
+ *
+ * Callbacks may be executed in the issuing backend but also in another
+ * backend (because that backend is waiting for the IO) or in IO workers (if
+ * io_method=worker is used).
+ *
+ *
+ * See PgAioHandleCallbackID's definition for an explanation for why
+ * callbacks are not identified by a pointer.
+ */
+void
+pgaio_io_register_callbacks(PgAioHandle *ioh, PgAioHandleCallbackID cbid)
+{
+ const PgAioHandleCallbacksEntry *ce = &aio_handle_cbs[cbid];
+
+ if (cbid >= lengthof(aio_handle_cbs))
+ elog(ERROR, "callback %d is out of range", cbid);
+ if (aio_handle_cbs[cbid].cb->complete_shared == NULL &&
+ aio_handle_cbs[cbid].cb->complete_local == NULL)
+ elog(ERROR, "callback %d does not have completion callback", cbid);
+ if (ioh->num_shared_callbacks >= PGAIO_HANDLE_MAX_CALLBACKS)
+ elog(PANIC, "too many callbacks, the max is %d", PGAIO_HANDLE_MAX_CALLBACKS);
+ ioh->shared_callbacks[ioh->num_shared_callbacks] = cbid;
+
+ pgaio_debug_io(DEBUG3, ioh,
+ "adding cb #%d, id %d/%s",
+ ioh->num_shared_callbacks + 1,
+ cbid, ce->name);
+
+ ioh->num_shared_callbacks++;
+}
+
+/*
+ * Associate an array of data with the Handle. This is e.g. useful to the
+ * transport knowledge about which buffers a multi-block IO affects to
+ * completion callbacks.
+ *
+ * Right now this can be done only once for each IO, even though multiple
+ * callbacks can be registered. There aren't any known usecases requiring more
+ * and the required amount of shared memory does add up, so it doesn't seem
+ * worth multiplying memory usage by PGAIO_HANDLE_MAX_CALLBACKS.
+ */
+void
+pgaio_io_set_handle_data_64(PgAioHandle *ioh, uint64 *data, uint8 len)
+{
+ Assert(ioh->state == PGAIO_HS_HANDED_OUT);
+ Assert(ioh->handle_data_len == 0);
+ Assert(len <= PG_IOV_MAX);
+
+ for (int i = 0; i < len; i++)
+ pgaio_ctl->handle_data[ioh->iovec_off + i] = data[i];
+ ioh->handle_data_len = len;
+}
+
+/*
+ * Convenience version of pgaio_io_set_handle_data_64() that converts a 32bit
+ * array to a 64bit array. Without it callers would end up needing to
+ * open-code equivalent code.
+ */
+void
+pgaio_io_set_handle_data_32(PgAioHandle *ioh, uint32 *data, uint8 len)
+{
+ Assert(ioh->state == PGAIO_HS_HANDED_OUT);
+ Assert(ioh->handle_data_len == 0);
+ Assert(len <= PG_IOV_MAX);
+
+ for (int i = 0; i < len; i++)
+ pgaio_ctl->handle_data[ioh->iovec_off + i] = data[i];
+ ioh->handle_data_len = len;
+}
+
+/*
+ * Return data set with pgaio_io_set_handle_data_*().
+ */
+uint64 *
+pgaio_io_get_handle_data(PgAioHandle *ioh, uint8 *len)
+{
+ Assert(ioh->handle_data_len > 0);
+
+ *len = ioh->handle_data_len;
+
+ return &pgaio_ctl->handle_data[ioh->iovec_off];
+}
+
+/*
+ * Internal function which invokes ->stage for all the registered callbacks.
+ */
+void
+pgaio_io_call_stage(PgAioHandle *ioh)
+{
+ Assert(ioh->target > PGAIO_TID_INVALID && ioh->target < PGAIO_TID_COUNT);
+ Assert(ioh->op > PGAIO_OP_INVALID && ioh->op < PGAIO_OP_COUNT);
+
+ for (int i = ioh->num_shared_callbacks; i > 0; i--)
+ {
+ PgAioHandleCallbackID cbid = ioh->shared_callbacks[i - 1];
+ const PgAioHandleCallbacksEntry *ce = &aio_handle_cbs[cbid];
+
+ if (!ce->cb->stage)
+ continue;
+
+ pgaio_debug_io(DEBUG3, ioh,
+ "calling cb #%d %d/%s->stage",
+ i, cbid, ce->name);
+ ce->cb->stage(ioh);
+ }
+}
+
+/*
+ * Internal function which invokes ->complete_shared for all the registered
+ * callbacks.
+ */
+void
+pgaio_io_call_complete_shared(PgAioHandle *ioh)
+{
+ PgAioResult result;
+
+ START_CRIT_SECTION();
+
+ Assert(ioh->target > PGAIO_TID_INVALID && ioh->target < PGAIO_TID_COUNT);
+ Assert(ioh->op > PGAIO_OP_INVALID && ioh->op < PGAIO_OP_COUNT);
+
+ result.status = ARS_OK; /* low level IO is always considered OK */
+ result.result = ioh->result;
+ result.id = PGAIO_HCB_INVALID;
+ result.error_data = 0;
+
+ for (int i = ioh->num_shared_callbacks; i > 0; i--)
+ {
+ PgAioHandleCallbackID cbid = ioh->shared_callbacks[i - 1];
+ const PgAioHandleCallbacksEntry *ce = &aio_handle_cbs[cbid];
+
+ if (!ce->cb->complete_shared)
+ continue;
+
+ pgaio_debug_io(DEBUG3, ioh,
+ "calling cb #%d, id %d/%s->complete_shared with distilled result status %d, id %u, error_data: %d, result: %d",
+ i, cbid, ce->name,
+ result.status, result.id, result.error_data, result.result);
+ result = ce->cb->complete_shared(ioh, result);
+ }
+
+ ioh->distilled_result = result;
+
+ pgaio_debug_io(DEBUG3, ioh,
+ "distilled result status %d, id %u, error_data: %d, result: %d, raw_result %d",
+ result.status, result.id, result.error_data, result.result,
+ ioh->result);
+
+ END_CRIT_SECTION();
+}
+
+
+/*
+ * Internal function which invokes ->complete_local for all the registered
+ * callbacks.
+ *
+ * XXX: It'd be nice to deduplicate with pgaio_io_call_complete_shared().
+ */
+void
+pgaio_io_call_complete_local(PgAioHandle *ioh)
+{
+ PgAioResult result;
+
+ START_CRIT_SECTION();
+
+ Assert(ioh->target > PGAIO_TID_INVALID && ioh->target < PGAIO_TID_COUNT);
+ Assert(ioh->op > PGAIO_OP_INVALID && ioh->op < PGAIO_OP_COUNT);
+
+ /* start with distilled result from shared callback */
+ result = ioh->distilled_result;
+
+ for (int i = ioh->num_shared_callbacks; i > 0; i--)
+ {
+ PgAioHandleCallbackID cbid = ioh->shared_callbacks[i - 1];
+ const PgAioHandleCallbacksEntry *ce = &aio_handle_cbs[cbid];
+
+ if (!ce->cb->complete_local)
+ continue;
+
+ pgaio_debug_io(DEBUG3, ioh,
+ "calling cb #%d, id %d/%s->complete_local with distilled result status %d, id %u, error_data: %d, result: %d",
+ i, cbid, ce->name,
+ result.status, result.id, result.error_data, result.result);
+ result = ce->cb->complete_local(ioh, result);
+ }
+
+ /*
+ * Note that we don't save the result in ioh->distilled_result, the local
+ * callback's result should not ever matter to other waiters.
+ */
+ pgaio_debug_io(DEBUG3, ioh,
+ "distilled result status %d, id %u, error_data: %d, result: %d, raw_result %d",
+ result.status, result.id, result.error_data, result.result,
+ ioh->result);
+
+ END_CRIT_SECTION();
+}
+
+
+
+/* --------------------------------------------------------------------------------
+ * IO Result
+ * --------------------------------------------------------------------------------
+ */
+
+void
+pgaio_result_report(PgAioResult result, const PgAioTargetData *target_data, int elevel)
+{
+ PgAioHandleCallbackID cbid = result.id;
+ const PgAioHandleCallbacksEntry *ce = &aio_handle_cbs[cbid];
+
+ Assert(result.status != ARS_UNKNOWN);
+ Assert(result.status != ARS_OK);
+
+ if (ce->cb->report == NULL)
+ elog(ERROR, "callback %d/%s does not have report callback",
+ result.id, ce->name);
+
+ ce->cb->report(result, target_data, elevel);
+}
diff --git a/src/backend/storage/aio/aio_init.c b/src/backend/storage/aio/aio_init.c
index f7ee8270756..0e98cc0c8fb 100644
--- a/src/backend/storage/aio/aio_init.c
+++ b/src/backend/storage/aio/aio_init.c
@@ -14,24 +14,210 @@
#include "postgres.h"
+#include "miscadmin.h"
+#include "storage/aio.h"
#include "storage/aio_init.h"
+#include "storage/aio_internal.h"
+#include "storage/ipc.h"
+#include "storage/proc.h"
+#include "storage/shmem.h"
+#include "utils/guc.h"
+static Size
+AioCtlShmemSize(void)
+{
+ Size sz;
+
+ /* pgaio_ctl itself */
+ sz = offsetof(PgAioCtl, io_handles);
+
+ return sz;
+}
+
+static uint32
+AioProcs(void)
+{
+ return MaxBackends + NUM_AUXILIARY_PROCS;
+}
+
+static Size
+AioBackendShmemSize(void)
+{
+ return mul_size(AioProcs(), sizeof(PgAioBackend));
+}
+
+static Size
+AioHandleShmemSize(void)
+{
+ Size sz;
+
+ /* ios */
+ sz = mul_size(AioProcs(),
+ mul_size(io_max_concurrency, sizeof(PgAioHandle)));
+
+ return sz;
+}
+
+static Size
+AioHandleIOVShmemSize(void)
+{
+ return mul_size(sizeof(struct iovec),
+ mul_size(mul_size(PG_IOV_MAX, AioProcs()),
+ io_max_concurrency));
+}
+
+static Size
+AioHandleDataShmemSize(void)
+{
+ return mul_size(sizeof(uint64),
+ mul_size(mul_size(PG_IOV_MAX, AioProcs()),
+ io_max_concurrency));
+}
+
+/*
+ * Choose a suitable value for io_max_concurrency.
+ *
+ * It's unlikely that we could have more IOs in flight than buffers that we
+ * would be allowed to pin.
+ *
+ * On the upper end, apply a cap too - just because shared_buffers is large,
+ * it doesn't make sense have millions of buffers undergo IO concurrently.
+ */
+static int
+AioChooseMaxConccurrency(void)
+{
+ uint32 max_backends;
+ int max_proportional_pins;
+
+ /* Similar logic to LimitAdditionalPins() */
+ max_backends = MaxBackends + NUM_AUXILIARY_PROCS;
+ max_proportional_pins = NBuffers / max_backends;
+
+ max_proportional_pins = Max(max_proportional_pins, 1);
+
+ /* apply upper limit */
+ return Min(max_proportional_pins, 64);
+}
+
Size
AioShmemSize(void)
{
Size sz = 0;
+ /*
+ * We prefer to report this value's source as PGC_S_DYNAMIC_DEFAULT.
+ * However, if the DBA explicitly set wal_buffers = -1 in the config file,
+ * then PGC_S_DYNAMIC_DEFAULT will fail to override that and we must force
+ *
+ */
+ if (io_max_concurrency == -1)
+ {
+ char buf[32];
+
+ snprintf(buf, sizeof(buf), "%d", AioChooseMaxConccurrency());
+ SetConfigOption("io_max_concurrency", buf, PGC_POSTMASTER,
+ PGC_S_DYNAMIC_DEFAULT);
+ if (io_max_concurrency == -1) /* failed to apply it? */
+ SetConfigOption("io_max_concurrency", buf, PGC_POSTMASTER,
+ PGC_S_OVERRIDE);
+ }
+
+ sz = add_size(sz, AioCtlShmemSize());
+ sz = add_size(sz, AioBackendShmemSize());
+ sz = add_size(sz, AioHandleShmemSize());
+ sz = add_size(sz, AioHandleIOVShmemSize());
+ sz = add_size(sz, AioHandleDataShmemSize());
+
+ if (pgaio_method_ops->shmem_size)
+ sz = add_size(sz, pgaio_method_ops->shmem_size());
+
return sz;
}
void
AioShmemInit(void)
{
+ bool found;
+ uint32 io_handle_off = 0;
+ uint32 iovec_off = 0;
+ uint32 per_backend_iovecs = io_max_concurrency * PG_IOV_MAX;
+
+ pgaio_ctl = (PgAioCtl *)
+ ShmemInitStruct("AioCtl", AioCtlShmemSize(), &found);
+
+ if (found)
+ goto out;
+
+ memset(pgaio_ctl, 0, AioCtlShmemSize());
+
+ pgaio_ctl->io_handle_count = AioProcs() * io_max_concurrency;
+ pgaio_ctl->iovec_count = AioProcs() * per_backend_iovecs;
+
+ pgaio_ctl->backend_state = (PgAioBackend *)
+ ShmemInitStruct("AioBackend", AioBackendShmemSize(), &found);
+
+ pgaio_ctl->io_handles = (PgAioHandle *)
+ ShmemInitStruct("AioHandle", AioHandleShmemSize(), &found);
+
+ pgaio_ctl->iovecs = (struct iovec *)
+ ShmemInitStruct("AioHandleIOV", AioHandleIOVShmemSize(), &found);
+ pgaio_ctl->handle_data = (uint64 *)
+ ShmemInitStruct("AioHandleData", AioHandleDataShmemSize(), &found);
+
+ for (int procno = 0; procno < AioProcs(); procno++)
+ {
+ PgAioBackend *bs = &pgaio_ctl->backend_state[procno];
+
+ bs->io_handle_off = io_handle_off;
+ io_handle_off += io_max_concurrency;
+
+ dclist_init(&bs->idle_ios);
+ memset(bs->staged_ios, 0, sizeof(PgAioHandle *) * PGAIO_SUBMIT_BATCH_SIZE);
+ dclist_init(&bs->in_flight_ios);
+
+ /* initialize per-backend IOs */
+ for (int i = 0; i < io_max_concurrency; i++)
+ {
+ PgAioHandle *ioh = &pgaio_ctl->io_handles[bs->io_handle_off + i];
+
+ ioh->generation = 1;
+ ioh->owner_procno = procno;
+ ioh->iovec_off = iovec_off;
+ ioh->handle_data_len = 0;
+ ioh->report_return = NULL;
+ ioh->resowner = NULL;
+ ioh->num_shared_callbacks = 0;
+ ioh->distilled_result.status = ARS_UNKNOWN;
+ ioh->flags = 0;
+
+ ConditionVariableInit(&ioh->cv);
+
+ dclist_push_tail(&bs->idle_ios, &ioh->node);
+ iovec_off += PG_IOV_MAX;
+ }
+ }
+
+out:
+ /* Initialize IO method specific resources. */
+ if (pgaio_method_ops->shmem_init)
+ pgaio_method_ops->shmem_init(!found);
}
void
pgaio_init_backend(void)
{
+ /* shouldn't be initialized twice */
+ Assert(!pgaio_my_backend);
+
+ if (MyProc == NULL || MyProcNumber >= AioProcs())
+ elog(ERROR, "aio requires a normal PGPROC");
+
+ pgaio_my_backend = &pgaio_ctl->backend_state[MyProcNumber];
+
+ if (pgaio_method_ops->init_backend)
+ pgaio_method_ops->init_backend();
+
+ before_shmem_exit(pgaio_shutdown, 0);
}
diff --git a/src/backend/storage/aio/aio_io.c b/src/backend/storage/aio/aio_io.c
new file mode 100644
index 00000000000..bb010d6152c
--- /dev/null
+++ b/src/backend/storage/aio/aio_io.c
@@ -0,0 +1,175 @@
+/*-------------------------------------------------------------------------
+ *
+ * aio_io.c
+ * AIO - Low Level IO Handling
+ *
+ * Functions related to associating IO operations to IO Handles and IO-method
+ * independent support functions for actually performing IO.
+ *
+ *
+ * Portions Copyright (c) 1996-2025, PostgreSQL Global Development Group
+ * Portions Copyright (c) 1994, Regents of the University of California
+ *
+ * IDENTIFICATION
+ * src/backend/storage/aio/aio_io.c
+ *
+ *-------------------------------------------------------------------------
+ */
+
+#include "postgres.h"
+
+#include "storage/aio.h"
+#include "storage/aio_internal.h"
+#include "storage/fd.h"
+#include "utils/wait_event.h"
+
+
+static void pgaio_io_before_prep(PgAioHandle *ioh);
+
+
+
+/* --------------------------------------------------------------------------------
+ * Public IO related functions operating on IO Handles
+ * --------------------------------------------------------------------------------
+ */
+
+/*
+ * Scatter/gather IO needs to associate an iovec with the Handle. To support
+ * worker mode this data needs to be in shared memory.
+ *
+ * XXX: Right now the amount of space available for each IO is
+ * PG_IOV_MAX. While it's tempting to use the io_combine_limit GUC, that's
+ * PGC_USERSET, so we can't allocate shared memory based on that.
+ */
+int
+pgaio_io_get_iovec(PgAioHandle *ioh, struct iovec **iov)
+{
+ Assert(ioh->state == PGAIO_HS_HANDED_OUT);
+
+ *iov = &pgaio_ctl->iovecs[ioh->iovec_off];
+
+ return PG_IOV_MAX;
+}
+
+PgAioOpData *
+pgaio_io_get_op_data(PgAioHandle *ioh)
+{
+ return &ioh->op_data;
+}
+
+
+
+/* --------------------------------------------------------------------------------
+ * "Preparation" routines for individual IO operations
+ *
+ * These are called by the code actually initiating an IO, to associate the IO
+ * specific data with an AIO handle.
+ *
+ * Each of the preparation routines first needs to call
+ * pgaio_io_before_prep(), then fill IO specific fields in the handle and then
+ * finally call pgaio_io_stage().
+ * --------------------------------------------------------------------------------
+ */
+
+void
+pgaio_io_prep_readv(PgAioHandle *ioh,
+ int fd, int iovcnt, uint64 offset)
+{
+ pgaio_io_before_prep(ioh);
+
+ ioh->op_data.read.fd = fd;
+ ioh->op_data.read.offset = offset;
+ ioh->op_data.read.iov_length = iovcnt;
+
+ pgaio_io_stage(ioh, PGAIO_OP_READV);
+}
+
+void
+pgaio_io_prep_writev(PgAioHandle *ioh,
+ int fd, int iovcnt, uint64 offset)
+{
+ pgaio_io_before_prep(ioh);
+
+ ioh->op_data.write.fd = fd;
+ ioh->op_data.write.offset = offset;
+ ioh->op_data.write.iov_length = iovcnt;
+
+ pgaio_io_stage(ioh, PGAIO_OP_WRITEV);
+}
+
+
+
+/* --------------------------------------------------------------------------------
+ * Internal IO related functions operating on IO Handles
+ * --------------------------------------------------------------------------------
+ */
+
+/*
+ * Execute IO operation synchronously. This is implemented here, not in
+ * method_sync.c, because other IO methods lso might use it / fall back to it.
+ */
+void
+pgaio_io_perform_synchronously(PgAioHandle *ioh)
+{
+ ssize_t result = 0;
+ struct iovec *iov = &pgaio_ctl->iovecs[ioh->iovec_off];
+
+ /* Perform IO. */
+ switch (ioh->op)
+ {
+ case PGAIO_OP_READV:
+ pgstat_report_wait_start(WAIT_EVENT_DATA_FILE_READ);
+ result = pg_preadv(ioh->op_data.read.fd, iov,
+ ioh->op_data.read.iov_length,
+ ioh->op_data.read.offset);
+ pgstat_report_wait_end();
+ break;
+ case PGAIO_OP_WRITEV:
+ pgstat_report_wait_start(WAIT_EVENT_DATA_FILE_WRITE);
+ result = pg_pwritev(ioh->op_data.write.fd, iov,
+ ioh->op_data.write.iov_length,
+ ioh->op_data.write.offset);
+ pgstat_report_wait_end();
+ break;
+ case PGAIO_OP_INVALID:
+ elog(ERROR, "trying to execute invalid IO operation");
+ }
+
+ ioh->result = result < 0 ? -errno : result;
+
+ pgaio_io_process_completion(ioh, ioh->result);
+}
+
+/*
+ * Helper function to be called by IO operation preparation functions, before
+ * any data in the handle is set. Mostly to centralize assertions.
+ */
+static void
+pgaio_io_before_prep(PgAioHandle *ioh)
+{
+ Assert(ioh->state == PGAIO_HS_HANDED_OUT);
+ Assert(pgaio_io_has_target(ioh));
+ Assert(ioh->op == PGAIO_OP_INVALID);
+}
+
+/*
+ * Could be made part of the public interface, but it's not clear there's
+ * really a use case for that.
+ */
+const char *
+pgaio_io_get_op_name(PgAioHandle *ioh)
+{
+ Assert(ioh->op >= 0 && ioh->op < PGAIO_OP_COUNT);
+
+ switch (ioh->op)
+ {
+ case PGAIO_OP_INVALID:
+ return "invalid";
+ case PGAIO_OP_READV:
+ return "read";
+ case PGAIO_OP_WRITEV:
+ return "write";
+ }
+
+ return NULL; /* silence compiler */
+}
diff --git a/src/backend/storage/aio/aio_target.c b/src/backend/storage/aio/aio_target.c
new file mode 100644
index 00000000000..15428968e58
--- /dev/null
+++ b/src/backend/storage/aio/aio_target.c
@@ -0,0 +1,108 @@
+/*-------------------------------------------------------------------------
+ *
+ * aio_target.c
+ * AIO - Functionality related to executing IO for different targets
+ *
+ * XXX Write me
+ *
+ * Portions Copyright (c) 1996-2025, PostgreSQL Global Development Group
+ * Portions Copyright (c) 1994, Regents of the University of California
+ *
+ * IDENTIFICATION
+ * src/backend/storage/aio/aio_target.c
+ *
+ *-------------------------------------------------------------------------
+ */
+
+#include "postgres.h"
+
+#include "storage/aio.h"
+#include "storage/aio_internal.h"
+
+
+/*
+ * Registry for entities that can be the target of AIO.
+ *
+ * To support executing using worker processes, the file descriptor for an IO
+ * may need to be be reopened in a different process. This is done via the
+ * PgAioTargetInfo.reopen callback.
+ */
+static const PgAioTargetInfo *pgaio_target_info[] = {
+ [PGAIO_TID_INVALID] = &(PgAioTargetInfo) {
+ .name = "invalid",
+ },
+};
+
+
+
+bool
+pgaio_io_has_target(PgAioHandle *ioh)
+{
+ return ioh->target != PGAIO_TID_INVALID;
+}
+
+/*
+ * Return the name for the target associated with the IO. Mostly useful for
+ * debugging/logging.
+ */
+const char *
+pgaio_io_get_target_name(PgAioHandle *ioh)
+{
+ Assert(ioh->target >= 0 && ioh->target < PGAIO_TID_COUNT);
+
+ return pgaio_target_info[ioh->target]->name;
+}
+
+/*
+ * Assign a target to the IO.
+ *
+ * This has to be called exactly once before pgaio_io_prep_*() is called.
+ */
+void
+pgaio_io_set_target(PgAioHandle *ioh, PgAioTargetID targetid)
+{
+ Assert(ioh->state == PGAIO_HS_HANDED_OUT);
+ Assert(ioh->target == PGAIO_TID_INVALID);
+
+ ioh->target = targetid;
+}
+
+PgAioTargetData *
+pgaio_io_get_target_data(PgAioHandle *ioh)
+{
+ return &ioh->target_data;
+}
+
+/*
+ * Return a stringified description of the IO's target.
+ *
+ * The string is localized and allocated in the current memory context.
+ */
+char *
+pgaio_io_get_target_description(PgAioHandle *ioh)
+{
+ return pgaio_target_info[ioh->target]->describe_identity(&ioh->target_data);
+}
+
+/*
+ * Internal: Check if pgaio_io_reopen() is available for the IO.
+ */
+bool
+pgaio_io_can_reopen(PgAioHandle *ioh)
+{
+ return pgaio_target_info[ioh->target]->reopen != NULL;
+}
+
+/*
+ * Internal: Before executing an IO outside of the context of the process the
+ * IO has been prepared in, the file descriptor has to be reopened - any FD
+ * referenced in the IO itself, won't be valid in the separate process.
+ */
+void
+pgaio_io_reopen(PgAioHandle *ioh)
+{
+ Assert(ioh->target >= 0 && ioh->target < PGAIO_TID_COUNT);
+ Assert(ioh->op >= 0 && ioh->op < PGAIO_OP_COUNT);
+
+ pgaio_target_info[ioh->target]->reopen(ioh);
+}
diff --git a/src/backend/storage/aio/meson.build b/src/backend/storage/aio/meson.build
index c822fd4ddf7..2c26089d52e 100644
--- a/src/backend/storage/aio/meson.build
+++ b/src/backend/storage/aio/meson.build
@@ -2,6 +2,10 @@
backend_sources += files(
'aio.c',
+ 'aio_callback.c',
'aio_init.c',
+ 'aio_io.c',
+ 'aio_target.c',
+ 'method_sync.c',
'read_stream.c',
)
diff --git a/src/backend/storage/aio/method_sync.c b/src/backend/storage/aio/method_sync.c
new file mode 100644
index 00000000000..43f9c8bd0b3
--- /dev/null
+++ b/src/backend/storage/aio/method_sync.c
@@ -0,0 +1,47 @@
+/*-------------------------------------------------------------------------
+ *
+ * method_sync.c
+ * AIO - perform "AIO" by executing it synchronously
+ *
+ * This method is mainly to check if AIO use causes regressions. Other IO
+ * methods might also fall back to the synchronous method for functionality
+ * they cannot provide.
+ *
+ * Portions Copyright (c) 1996-2025, PostgreSQL Global Development Group
+ * Portions Copyright (c) 1994, Regents of the University of California
+ *
+ * IDENTIFICATION
+ * src/backend/storage/aio/method_sync.c
+ *
+ *-------------------------------------------------------------------------
+ */
+
+#include "postgres.h"
+
+#include "storage/aio.h"
+#include "storage/aio_internal.h"
+
+static bool pgaio_sync_needs_synchronous_execution(PgAioHandle *ioh);
+static int pgaio_sync_submit(uint16 num_staged_ios, PgAioHandle **staged_ios);
+
+
+const IoMethodOps pgaio_sync_ops = {
+ .needs_synchronous_execution = pgaio_sync_needs_synchronous_execution,
+ .submit = pgaio_sync_submit,
+};
+
+
+
+static bool
+pgaio_sync_needs_synchronous_execution(PgAioHandle *ioh)
+{
+ return true;
+}
+
+static int
+pgaio_sync_submit(uint16 num_staged_ios, PgAioHandle **staged_ios)
+{
+ elog(ERROR, "should be unreachable");
+
+ return 0;
+}
diff --git a/src/backend/utils/activity/wait_event_names.txt b/src/backend/utils/activity/wait_event_names.txt
index e199f071628..b5d3dcbf1e9 100644
--- a/src/backend/utils/activity/wait_event_names.txt
+++ b/src/backend/utils/activity/wait_event_names.txt
@@ -191,6 +191,9 @@ ABI_compatibility:
Section: ClassName - WaitEventIO
+AIO_SUBMIT "Waiting for AIO submission."
+AIO_DRAIN "Waiting for IOs to finish."
+AIO_COMPLETION "Waiting for completion callback."
BASEBACKUP_READ "Waiting for base backup to read from a file."
BASEBACKUP_SYNC "Waiting for data written by a base backup to reach durable storage."
BASEBACKUP_WRITE "Waiting for base backup to write to a file."
diff --git a/src/backend/utils/resowner/resowner.c b/src/backend/utils/resowner/resowner.c
index ac5ca4a765e..e5d852b5ee6 100644
--- a/src/backend/utils/resowner/resowner.c
+++ b/src/backend/utils/resowner/resowner.c
@@ -47,6 +47,8 @@
#include "common/hashfn.h"
#include "common/int.h"
+#include "lib/ilist.h"
+#include "storage/aio.h"
#include "storage/ipc.h"
#include "storage/predicate.h"
#include "storage/proc.h"
@@ -155,6 +157,12 @@ struct ResourceOwnerData
/* The local locks cache. */
LOCALLOCK *locks[MAX_RESOWNER_LOCKS]; /* list of owned locks */
+
+ /*
+ * AIO handles need be registered in critical sections and therefore
+ * cannot use the normal ResoureElem mechanism.
+ */
+ dlist_head aio_handles;
};
@@ -425,6 +433,8 @@ ResourceOwnerCreate(ResourceOwner parent, const char *name)
parent->firstchild = owner;
}
+ dlist_init(&owner->aio_handles);
+
return owner;
}
@@ -725,6 +735,14 @@ ResourceOwnerReleaseInternal(ResourceOwner owner,
* so issue warnings. In the abort case, just clean up quietly.
*/
ResourceOwnerReleaseAll(owner, phase, isCommit);
+
+ /* XXX: Could probably be a later phase? */
+ while (!dlist_is_empty(&owner->aio_handles))
+ {
+ dlist_node *node = dlist_head_node(&owner->aio_handles);
+
+ pgaio_io_release_resowner(node, !isCommit);
+ }
}
else if (phase == RESOURCE_RELEASE_LOCKS)
{
@@ -1082,3 +1100,15 @@ ResourceOwnerForgetLock(ResourceOwner owner, LOCALLOCK *locallock)
elog(ERROR, "lock reference %p is not owned by resource owner %s",
locallock, owner->name);
}
+
+void
+ResourceOwnerRememberAioHandle(ResourceOwner owner, struct dlist_node *ioh_node)
+{
+ dlist_push_tail(&owner->aio_handles, ioh_node);
+}
+
+void
+ResourceOwnerForgetAioHandle(ResourceOwner owner, struct dlist_node *ioh_node)
+{
+ dlist_delete_from(&owner->aio_handles, ioh_node);
+}
diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list
index 3bec090428d..c7f34559b1b 100644
--- a/src/tools/pgindent/typedefs.list
+++ b/src/tools/pgindent/typedefs.list
@@ -1267,6 +1267,7 @@ InvalMessageArray
InvalidationInfo
InvalidationMsgsGroup
IoMethod
+IoMethodOps
IpcMemoryId
IpcMemoryKey
IpcMemoryState
@@ -2105,6 +2106,26 @@ Permutation
PermutationStep
PermutationStepBlocker
PermutationStepBlockerType
+PgAioBackend
+PgAioCtl
+PgAioHandle
+PgAioHandleCallbackID
+PgAioHandleCallbackStage
+PgAioHandleCallbackComplete
+PgAioHandleCallbackReport
+PgAioHandleCallbacks
+PgAioHandleCallbacksEntry
+PgAioHandleFlags
+PgAioHandleState
+PgAioOp
+PgAioOpData
+PgAioResult
+PgAioResultStatus
+PgAioReturn
+PgAioTargetData
+PgAioTargetID
+PgAioTargetInfo
+PgAioWaitRef
PgArchData
PgBackendGSSStatus
PgBackendSSLStatus
--
2.48.1.76.g4e746b1a31.dirty
[text/x-diff] v2.3-0012-aio-Skeleton-IO-worker-infrastructure.patch (22.7K, ../../p5flybfcjrzgpgwcnzkxsp3xdw3czfoafkvks6g7en6ewmvymh@wrz7z2vwwc7c/13-v2.3-0012-aio-Skeleton-IO-worker-infrastructure.patch)
download | inline diff:
From 5e84720afa46fdfd892a8bac36585f0f7a29d3f3 Mon Sep 17 00:00:00 2001
From: Andres Freund <andres@anarazel.de>
Date: Wed, 22 Jan 2025 13:43:40 -0500
Subject: [PATCH v2.3 12/30] aio: Skeleton IO worker infrastructure
This doesn't do anything useful on its own, but the code that needs to be
touched is independent of other changes.
Remarks:
- should completely get rid of ID assignment logic in postmaster.c
- postmaster.c badly needs a refactoring.
- dynamic increase / decrease of workers based on IO load
Author:
Reviewed-by:
Discussion: https://postgr.es/m/
Backpatch:
---
src/include/miscadmin.h | 2 +
src/include/postmaster/postmaster.h | 1 +
src/include/storage/aio_init.h | 2 +
src/include/storage/io_worker.h | 22 +++
src/include/storage/proc.h | 4 +-
src/backend/postmaster/launch_backend.c | 2 +
src/backend/postmaster/pmchild.c | 1 +
src/backend/postmaster/postmaster.c | 169 ++++++++++++++++--
src/backend/storage/aio/Makefile | 1 +
src/backend/storage/aio/aio_init.c | 7 +
src/backend/storage/aio/meson.build | 1 +
src/backend/storage/aio/method_worker.c | 86 +++++++++
src/backend/tcop/postgres.c | 2 +
src/backend/utils/activity/pgstat_backend.c | 1 +
src/backend/utils/activity/pgstat_io.c | 1 +
.../utils/activity/wait_event_names.txt | 1 +
src/backend/utils/init/miscinit.c | 3 +
src/backend/utils/misc/guc_tables.c | 13 ++
src/backend/utils/misc/postgresql.conf.sample | 1 +
19 files changed, 305 insertions(+), 15 deletions(-)
create mode 100644 src/include/storage/io_worker.h
create mode 100644 src/backend/storage/aio/method_worker.c
diff --git a/src/include/miscadmin.h b/src/include/miscadmin.h
index d016a9c9248..c2b3e27c613 100644
--- a/src/include/miscadmin.h
+++ b/src/include/miscadmin.h
@@ -360,6 +360,7 @@ typedef enum BackendType
B_ARCHIVER,
B_BG_WRITER,
B_CHECKPOINTER,
+ B_IO_WORKER,
B_STARTUP,
B_WAL_RECEIVER,
B_WAL_SUMMARIZER,
@@ -389,6 +390,7 @@ extern PGDLLIMPORT BackendType MyBackendType;
#define AmWalReceiverProcess() (MyBackendType == B_WAL_RECEIVER)
#define AmWalSummarizerProcess() (MyBackendType == B_WAL_SUMMARIZER)
#define AmWalWriterProcess() (MyBackendType == B_WAL_WRITER)
+#define AmIoWorkerProcess() (MyBackendType == B_IO_WORKER)
#define AmSpecialWorkerProcess() \
(AmAutoVacuumLauncherProcess() || \
diff --git a/src/include/postmaster/postmaster.h b/src/include/postmaster/postmaster.h
index 188a06e2379..253dc98c50e 100644
--- a/src/include/postmaster/postmaster.h
+++ b/src/include/postmaster/postmaster.h
@@ -98,6 +98,7 @@ extern void InitProcessGlobals(void);
extern int MaxLivePostmasterChildren(void);
extern bool PostmasterMarkPIDForWorkerNotify(int);
+extern void assign_io_workers(int newval, void *extra);
#ifdef WIN32
extern void pgwin32_register_deadchild_callback(HANDLE procHandle, DWORD procId);
diff --git a/src/include/storage/aio_init.h b/src/include/storage/aio_init.h
index 44151ef55bf..bc15b720fca 100644
--- a/src/include/storage/aio_init.h
+++ b/src/include/storage/aio_init.h
@@ -21,4 +21,6 @@ extern void AioShmemInit(void);
extern void pgaio_init_backend(void);
+extern bool pgaio_workers_enabled(void);
+
#endif /* AIO_INIT_H */
diff --git a/src/include/storage/io_worker.h b/src/include/storage/io_worker.h
new file mode 100644
index 00000000000..223d614dc4a
--- /dev/null
+++ b/src/include/storage/io_worker.h
@@ -0,0 +1,22 @@
+/*-------------------------------------------------------------------------
+ *
+ * io_worker.h
+ * IO worker for implementing AIO "ourselves"
+ *
+ *
+ * Portions Copyright (c) 1996-2025, PostgreSQL Global Development Group
+ * Portions Copyright (c) 1994, Regents of the University of California
+ *
+ * src/include/storage/io.h
+ *
+ *-------------------------------------------------------------------------
+ */
+#ifndef IO_WORKER_H
+#define IO_WORKER_H
+
+
+extern void IoWorkerMain(char *startup_data, size_t startup_data_len) pg_attribute_noreturn();
+
+extern int io_workers;
+
+#endif /* IO_WORKER_H */
diff --git a/src/include/storage/proc.h b/src/include/storage/proc.h
index 20777f7d5ae..64e9b8ff8c5 100644
--- a/src/include/storage/proc.h
+++ b/src/include/storage/proc.h
@@ -448,7 +448,9 @@ extern PGDLLIMPORT PGPROC *PreparedXactProcs;
* 2 slots, but WAL writer is launched only after startup has exited, so we
* only need 6 slots.
*/
-#define NUM_AUXILIARY_PROCS 6
+#define MAX_IO_WORKERS 32
+#define NUM_AUXILIARY_PROCS (6 + MAX_IO_WORKERS)
+
/* configurable options */
extern PGDLLIMPORT int DeadlockTimeout;
diff --git a/src/backend/postmaster/launch_backend.c b/src/backend/postmaster/launch_backend.c
index a97a1eda6da..54b4c22bd63 100644
--- a/src/backend/postmaster/launch_backend.c
+++ b/src/backend/postmaster/launch_backend.c
@@ -48,6 +48,7 @@
#include "replication/slotsync.h"
#include "replication/walreceiver.h"
#include "storage/dsm.h"
+#include "storage/io_worker.h"
#include "storage/pg_shmem.h"
#include "tcop/backend_startup.h"
#include "utils/memutils.h"
@@ -197,6 +198,7 @@ static child_process_kind child_process_kinds[] = {
[B_ARCHIVER] = {"archiver", PgArchiverMain, true},
[B_BG_WRITER] = {"bgwriter", BackgroundWriterMain, true},
[B_CHECKPOINTER] = {"checkpointer", CheckpointerMain, true},
+ [B_IO_WORKER] = {"io_worker", IoWorkerMain, true},
[B_STARTUP] = {"startup", StartupProcessMain, true},
[B_WAL_RECEIVER] = {"wal_receiver", WalReceiverMain, true},
[B_WAL_SUMMARIZER] = {"wal_summarizer", WalSummarizerMain, true},
diff --git a/src/backend/postmaster/pmchild.c b/src/backend/postmaster/pmchild.c
index 0d473226c3a..cde1d23a4ca 100644
--- a/src/backend/postmaster/pmchild.c
+++ b/src/backend/postmaster/pmchild.c
@@ -101,6 +101,7 @@ InitPostmasterChildSlots(void)
pmchild_pools[B_AUTOVAC_WORKER].size = autovacuum_worker_slots;
pmchild_pools[B_BG_WORKER].size = max_worker_processes;
+ pmchild_pools[B_IO_WORKER].size = MAX_IO_WORKERS;
/*
* There can be only one of each of these running at a time. They each
diff --git a/src/backend/postmaster/postmaster.c b/src/backend/postmaster/postmaster.c
index 115ad3d31d2..ddd82b94720 100644
--- a/src/backend/postmaster/postmaster.c
+++ b/src/backend/postmaster/postmaster.c
@@ -108,9 +108,12 @@
#include "replication/logicallauncher.h"
#include "replication/slotsync.h"
#include "replication/walsender.h"
+#include "storage/aio_init.h"
#include "storage/fd.h"
+#include "storage/io_worker.h"
#include "storage/ipc.h"
#include "storage/pmsignal.h"
+#include "storage/proc.h"
#include "tcop/backend_startup.h"
#include "tcop/tcopprot.h"
#include "utils/datetime.h"
@@ -334,6 +337,7 @@ typedef enum
* ckpt */
PM_WAIT_XLOG_ARCHIVAL, /* waiting for archiver and walsenders to
* finish */
+ PM_WAIT_IO_WORKERS, /* waiting for io workers to exit */
PM_WAIT_CHECKPOINTER, /* waiting for checkpointer to shut down */
PM_WAIT_DEAD_END, /* waiting for dead-end children to exit */
PM_NO_CHILDREN, /* all important children have exited */
@@ -396,6 +400,10 @@ bool LoadedSSL = false;
static DNSServiceRef bonjour_sdref = NULL;
#endif
+/* State for IO worker management. */
+static int io_worker_count = 0;
+static PMChild *io_worker_children[MAX_IO_WORKERS];
+
/*
* postmaster.c - function prototypes
*/
@@ -430,6 +438,8 @@ static void TerminateChildren(int signal);
static int CountChildren(BackendTypeMask targetMask);
static void LaunchMissingBackgroundProcesses(void);
static void maybe_start_bgworkers(void);
+static bool maybe_reap_io_worker(int pid);
+static void maybe_adjust_io_workers(void);
static bool CreateOptsFile(int argc, char *argv[], char *fullprogname);
static PMChild *StartChildProcess(BackendType type);
static void StartSysLogger(void);
@@ -1357,6 +1367,11 @@ PostmasterMain(int argc, char *argv[])
*/
AddToDataDirLockFile(LOCK_FILE_LINE_PM_STATUS, PM_STATUS_STARTING);
+ UpdatePMState(PM_STARTUP);
+
+ /* Make sure we can perform I/O while starting up. */
+ maybe_adjust_io_workers();
+
/* Start bgwriter and checkpointer so they can help with recovery */
if (CheckpointerPMChild == NULL)
CheckpointerPMChild = StartChildProcess(B_CHECKPOINTER);
@@ -1369,7 +1384,6 @@ PostmasterMain(int argc, char *argv[])
StartupPMChild = StartChildProcess(B_STARTUP);
Assert(StartupPMChild != NULL);
StartupStatus = STARTUP_RUNNING;
- UpdatePMState(PM_STARTUP);
/* Some workers may be scheduled to start now */
maybe_start_bgworkers();
@@ -2493,6 +2507,16 @@ process_pm_child_exit(void)
continue;
}
+ /* Was it an IO worker? */
+ if (maybe_reap_io_worker(pid))
+ {
+ if (!EXIT_STATUS_0(exitstatus) && !EXIT_STATUS_1(exitstatus))
+ HandleChildCrash(pid, exitstatus, _("io worker"));
+
+ maybe_adjust_io_workers();
+ continue;
+ }
+
/*
* Was it a backend or a background worker?
*/
@@ -2704,6 +2728,7 @@ HandleFatalError(QuitSignalReason reason, bool consider_sigabrt)
case PM_WAIT_XLOG_SHUTDOWN:
case PM_WAIT_XLOG_ARCHIVAL:
case PM_WAIT_CHECKPOINTER:
+ case PM_WAIT_IO_WORKERS:
/*
* Note that we switch *back* to PM_WAIT_BACKENDS here. This way
@@ -2892,20 +2917,21 @@ PostmasterStateMachine(void)
/*
* If we are doing crash recovery or an immediate shutdown then we
- * expect archiver, checkpointer and walsender to exit as well,
- * otherwise not.
+ * expect archiver, checkpointer, io workers and walsender to exit as
+ * well, otherwise not.
*/
if (FatalError || Shutdown >= ImmediateShutdown)
targetMask = btmask_add(targetMask,
B_CHECKPOINTER,
B_ARCHIVER,
+ B_IO_WORKER,
B_WAL_SENDER);
/*
- * Normally walsenders and archiver will continue running; they will
- * be terminated later after writing the checkpoint record. We also
- * let dead-end children to keep running for now. The syslogger
- * process exits last.
+ * Normally archiver, checkpointer, IO workers and walsenders will
+ * continue running; they will be terminated later after writing the
+ * checkpoint record. We also let dead-end children to keep running
+ * for now. The syslogger process exits last.
*
* This assertion checks that we have covered all backend types,
* either by including them in targetMask, or by noting here that they
@@ -2920,12 +2946,13 @@ PostmasterStateMachine(void)
B_LOGGER);
/*
- * Archiver, checkpointer and walsender may or may not be in
- * targetMask already.
+ * Archiver, checkpointer, IO workers, and walsender may or may
+ * not be in targetMask already.
*/
remainMask = btmask_add(remainMask,
B_ARCHIVER,
B_CHECKPOINTER,
+ B_IO_WORKER,
B_WAL_SENDER);
/* these are not real postmaster children */
@@ -3020,11 +3047,25 @@ PostmasterStateMachine(void)
{
/*
* PM_WAIT_XLOG_ARCHIVAL state ends when there's no children other
- * than checkpointer and dead-end children left. There shouldn't be
- * any regular backends left by now anyway; what we're really waiting
- * for is for walsenders and archiver to exit.
+ * than checkpointer, io workers and dead-end children left. There
+ * shouldn't be any regular backends left by now anyway; what we're
+ * really waiting for is for walsenders and archiver to exit.
*/
- if (CountChildren(btmask_all_except(B_CHECKPOINTER, B_LOGGER, B_DEAD_END_BACKEND)) == 0)
+ if (CountChildren(btmask_all_except(B_CHECKPOINTER, B_IO_WORKER,
+ B_LOGGER, B_DEAD_END_BACKEND)) == 0)
+ {
+ UpdatePMState(PM_WAIT_IO_WORKERS);
+ SignalChildren(SIGUSR2, btmask(B_IO_WORKER));
+ }
+ }
+
+ if (pmState == PM_WAIT_IO_WORKERS)
+ {
+ /*
+ * PM_WAIT_IO_WORKERS state ends when there's only checkpointer and
+ * dead_end children left.
+ */
+ if (io_worker_count == 0)
{
UpdatePMState(PM_WAIT_CHECKPOINTER);
@@ -3151,10 +3192,14 @@ PostmasterStateMachine(void)
/* re-create shared memory and semaphores */
CreateSharedMemoryAndSemaphores();
+ UpdatePMState(PM_STARTUP);
+
+ /* Make sure we can perform I/O while starting up. */
+ maybe_adjust_io_workers();
+
StartupPMChild = StartChildProcess(B_STARTUP);
Assert(StartupPMChild != NULL);
StartupStatus = STARTUP_RUNNING;
- UpdatePMState(PM_STARTUP);
/* crash recovery started, reset SIGKILL flag */
AbortStartTime = 0;
@@ -3178,6 +3223,7 @@ pmstate_name(PMState state)
PM_TOSTR_CASE(PM_WAIT_BACKENDS);
PM_TOSTR_CASE(PM_WAIT_XLOG_SHUTDOWN);
PM_TOSTR_CASE(PM_WAIT_XLOG_ARCHIVAL);
+ PM_TOSTR_CASE(PM_WAIT_IO_WORKERS);
PM_TOSTR_CASE(PM_WAIT_DEAD_END);
PM_TOSTR_CASE(PM_WAIT_CHECKPOINTER);
PM_TOSTR_CASE(PM_NO_CHILDREN);
@@ -4093,6 +4139,7 @@ bgworker_should_start_now(BgWorkerStartTime start_time)
case PM_WAIT_DEAD_END:
case PM_WAIT_XLOG_ARCHIVAL:
case PM_WAIT_XLOG_SHUTDOWN:
+ case PM_WAIT_IO_WORKERS:
case PM_WAIT_BACKENDS:
case PM_STOP_BACKENDS:
break;
@@ -4243,6 +4290,100 @@ maybe_start_bgworkers(void)
}
}
+static bool
+maybe_reap_io_worker(int pid)
+{
+ for (int id = 0; id < MAX_IO_WORKERS; ++id)
+ {
+ if (io_worker_children[id] &&
+ io_worker_children[id]->pid == pid)
+ {
+ ReleasePostmasterChildSlot(io_worker_children[id]);
+
+ --io_worker_count;
+ io_worker_children[id] = NULL;
+ return true;
+ }
+ }
+ return false;
+}
+
+static void
+maybe_adjust_io_workers(void)
+{
+ if (!pgaio_workers_enabled())
+ return;
+
+ /*
+ * If we're in final shutting down state, then we're just waiting for all
+ * processes to exit.
+ */
+ if (pmState >= PM_WAIT_IO_WORKERS)
+ return;
+
+ /* Don't start new workers during an immediate shutdown either. */
+ if (Shutdown >= ImmediateShutdown)
+ return;
+
+ /*
+ * Don't start new workers if we're in the shutdown phase of a crash
+ * restart. But we *do* need to start if we're already starting up again.
+ */
+ if (FatalError && pmState >= PM_STOP_BACKENDS)
+ return;
+
+ Assert(pmState < PM_WAIT_IO_WORKERS);
+
+ /* Not enough running? */
+ while (io_worker_count < io_workers)
+ {
+ PMChild *child;
+ int id;
+
+ /* find unused entry in io_worker_children array */
+ for (id = 0; id < MAX_IO_WORKERS; ++id)
+ {
+ if (io_worker_children[id] == NULL)
+ break;
+ }
+ if (id == MAX_IO_WORKERS)
+ elog(ERROR, "could not find a free IO worker ID");
+
+ /* Try to launch one. */
+ child = StartChildProcess(B_IO_WORKER);
+ if (child != NULL)
+ {
+ io_worker_children[id] = child;
+ ++io_worker_count;
+ }
+ else
+ break; /* XXX try again soon? */
+ }
+
+ /* Too many running? */
+ if (io_worker_count > io_workers)
+ {
+ /* ask the IO worker in the highest slot to exit */
+ for (int id = MAX_IO_WORKERS - 1; id >= 0; --id)
+ {
+ if (io_worker_children[id] != NULL)
+ {
+ kill(io_worker_children[id]->pid, SIGUSR2);
+ break;
+ }
+ }
+ }
+}
+
+void
+assign_io_workers(int newval, void *extra)
+{
+ io_workers = newval;
+ if (!IsUnderPostmaster && pmState > PM_INIT)
+ maybe_adjust_io_workers();
+}
+
+
/*
* When a backend asks to be notified about worker state changes, we
* set a flag in its backend entry. The background worker machinery needs
diff --git a/src/backend/storage/aio/Makefile b/src/backend/storage/aio/Makefile
index 89f821ea7e1..f51c34a37f8 100644
--- a/src/backend/storage/aio/Makefile
+++ b/src/backend/storage/aio/Makefile
@@ -15,6 +15,7 @@ OBJS = \
aio_io.o \
aio_target.o \
method_sync.o \
+ method_worker.o \
read_stream.o
include $(top_srcdir)/src/backend/common.mk
diff --git a/src/backend/storage/aio/aio_init.c b/src/backend/storage/aio/aio_init.c
index 0e98cc0c8fb..233c144965b 100644
--- a/src/backend/storage/aio/aio_init.c
+++ b/src/backend/storage/aio/aio_init.c
@@ -221,3 +221,10 @@ pgaio_init_backend(void)
before_shmem_exit(pgaio_shutdown, 0);
}
+
+bool
+pgaio_workers_enabled(void)
+{
+ /* placeholder for future commit */
+ return false;
+}
diff --git a/src/backend/storage/aio/meson.build b/src/backend/storage/aio/meson.build
index 2c26089d52e..74f94c6e40b 100644
--- a/src/backend/storage/aio/meson.build
+++ b/src/backend/storage/aio/meson.build
@@ -7,5 +7,6 @@ backend_sources += files(
'aio_io.c',
'aio_target.c',
'method_sync.c',
+ 'method_worker.c',
'read_stream.c',
)
diff --git a/src/backend/storage/aio/method_worker.c b/src/backend/storage/aio/method_worker.c
new file mode 100644
index 00000000000..1d79e7e85ef
--- /dev/null
+++ b/src/backend/storage/aio/method_worker.c
@@ -0,0 +1,86 @@
+/*-------------------------------------------------------------------------
+ *
+ * method_worker.c
+ * AIO implementation using workers
+ *
+ * Portions Copyright (c) 1996-2025, PostgreSQL Global Development Group
+ * Portions Copyright (c) 1994, Regents of the University of California
+ *
+ * IDENTIFICATION
+ * src/backend/storage/aio/method_worker.c
+ *
+ *-------------------------------------------------------------------------
+ */
+
+#include "postgres.h"
+
+#include "libpq/pqsignal.h"
+#include "miscadmin.h"
+#include "postmaster/auxprocess.h"
+#include "postmaster/interrupt.h"
+#include "storage/io_worker.h"
+#include "storage/ipc.h"
+#include "storage/latch.h"
+#include "storage/proc.h"
+#include "tcop/tcopprot.h"
+#include "utils/wait_event.h"
+
+
+int io_workers = 3;
+
+
+void
+IoWorkerMain(char *startup_data, size_t startup_data_len)
+{
+ sigjmp_buf local_sigjmp_buf;
+
+ MyBackendType = B_IO_WORKER;
+ AuxiliaryProcessMainCommon();
+
+ /* TODO review all signals */
+ pqsignal(SIGHUP, SignalHandlerForConfigReload);
+ pqsignal(SIGINT, die); /* to allow manually triggering worker restart */
+
+ /*
+ * Ignore SIGTERM, will get explicit shutdown via SIGUSR2 later in the
+ * shutdown sequence, similar to checkpointer.
+ */
+ pqsignal(SIGTERM, SIG_IGN);
+ /* SIGQUIT handler was already set up by InitPostmasterChild */
+ pqsignal(SIGALRM, SIG_IGN);
+ pqsignal(SIGPIPE, SIG_IGN);
+ pqsignal(SIGUSR1, procsignal_sigusr1_handler);
+ pqsignal(SIGUSR2, SignalHandlerForShutdownRequest);
+ sigprocmask(SIG_SETMASK, &UnBlockSig, NULL);
+
+ /* see PostgresMain() */
+ if (sigsetjmp(local_sigjmp_buf, 1) != 0)
+ {
+ error_context_stack = NULL;
+ HOLD_INTERRUPTS();
+
+ /*
+ * We normally shouldn't get errors here. Need to do just enough error
+ * recovery so that we can mark the IO as failed and then exit.
+ */
+ LWLockReleaseAll();
+
+ /* TODO: recover from IO errors */
+
+ EmitErrorReport();
+ proc_exit(1);
+ }
+
+ /* We can now handle ereport(ERROR) */
+ PG_exception_stack = &local_sigjmp_buf;
+
+ while (!ShutdownRequestPending)
+ {
+ WaitLatch(MyLatch, WL_LATCH_SET | WL_EXIT_ON_PM_DEATH, -1,
+ WAIT_EVENT_IO_WORKER_MAIN);
+ ResetLatch(MyLatch);
+ CHECK_FOR_INTERRUPTS();
+ }
+
+ proc_exit(0);
+}
diff --git a/src/backend/tcop/postgres.c b/src/backend/tcop/postgres.c
index 5655348a2e2..605c8950043 100644
--- a/src/backend/tcop/postgres.c
+++ b/src/backend/tcop/postgres.c
@@ -3313,6 +3313,8 @@ ProcessInterrupts(void)
(errcode(ERRCODE_ADMIN_SHUTDOWN),
errmsg("terminating background worker \"%s\" due to administrator command",
MyBgworkerEntry->bgw_type)));
+ else if (AmIoWorkerProcess())
+ proc_exit(0);
else
ereport(FATAL,
(errcode(ERRCODE_ADMIN_SHUTDOWN),
diff --git a/src/backend/utils/activity/pgstat_backend.c b/src/backend/utils/activity/pgstat_backend.c
index bcf9e4b1487..b2151ab4ca3 100644
--- a/src/backend/utils/activity/pgstat_backend.c
+++ b/src/backend/utils/activity/pgstat_backend.c
@@ -241,6 +241,7 @@ pgstat_tracks_backend_bktype(BackendType bktype)
case B_WAL_SUMMARIZER:
case B_BG_WRITER:
case B_CHECKPOINTER:
+ case B_IO_WORKER:
case B_STARTUP:
return false;
diff --git a/src/backend/utils/activity/pgstat_io.c b/src/backend/utils/activity/pgstat_io.c
index 6ff5d9e96a1..70518749142 100644
--- a/src/backend/utils/activity/pgstat_io.c
+++ b/src/backend/utils/activity/pgstat_io.c
@@ -365,6 +365,7 @@ pgstat_tracks_io_bktype(BackendType bktype)
case B_BG_WORKER:
case B_BG_WRITER:
case B_CHECKPOINTER:
+ case B_IO_WORKER:
case B_SLOTSYNC_WORKER:
case B_STANDALONE_BACKEND:
case B_STARTUP:
diff --git a/src/backend/utils/activity/wait_event_names.txt b/src/backend/utils/activity/wait_event_names.txt
index b5d3dcbf1e9..e702aa7152a 100644
--- a/src/backend/utils/activity/wait_event_names.txt
+++ b/src/backend/utils/activity/wait_event_names.txt
@@ -57,6 +57,7 @@ BGWRITER_HIBERNATE "Waiting in background writer process, hibernating."
BGWRITER_MAIN "Waiting in main loop of background writer process."
CHECKPOINTER_MAIN "Waiting in main loop of checkpointer process."
CHECKPOINTER_SHUTDOWN "Waiting for checkpointer process to be terminated."
+IO_WORKER_MAIN "Waiting in main loop of IO Worker process."
LOGICAL_APPLY_MAIN "Waiting in main loop of logical replication apply process."
LOGICAL_LAUNCHER_MAIN "Waiting in main loop of logical replication launcher process."
LOGICAL_PARALLEL_APPLY_MAIN "Waiting in main loop of logical replication parallel apply process."
diff --git a/src/backend/utils/init/miscinit.c b/src/backend/utils/init/miscinit.c
index 0347fc11092..cbca090d2b0 100644
--- a/src/backend/utils/init/miscinit.c
+++ b/src/backend/utils/init/miscinit.c
@@ -293,6 +293,9 @@ GetBackendTypeDesc(BackendType backendType)
case B_CHECKPOINTER:
backendDesc = gettext_noop("checkpointer");
break;
+ case B_IO_WORKER:
+ backendDesc = "io worker";
+ break;
case B_LOGGER:
backendDesc = gettext_noop("logger");
break;
diff --git a/src/backend/utils/misc/guc_tables.c b/src/backend/utils/misc/guc_tables.c
index de524eccad5..8a83dcc820d 100644
--- a/src/backend/utils/misc/guc_tables.c
+++ b/src/backend/utils/misc/guc_tables.c
@@ -74,6 +74,7 @@
#include "storage/aio.h"
#include "storage/bufmgr.h"
#include "storage/bufpage.h"
+#include "storage/io_worker.h"
#include "storage/large_object.h"
#include "storage/pg_shmem.h"
#include "storage/predicate.h"
@@ -3233,6 +3234,18 @@ struct config_int ConfigureNamesInt[] =
NULL, NULL, NULL
},
+ {
+ {"io_workers",
+ PGC_SIGHUP,
+ RESOURCES_ASYNCHRONOUS,
+ gettext_noop("Number of IO worker processes, for io_method=worker."),
+ NULL,
+ },
+ &io_workers,
+ 3, 1, MAX_IO_WORKERS,
+ NULL, assign_io_workers, NULL
+ },
+
{
{"backend_flush_after", PGC_USERSET, RESOURCES_ASYNCHRONOUS,
gettext_noop("Number of pages after which previously performed writes are flushed to disk."),
diff --git a/src/backend/utils/misc/postgresql.conf.sample b/src/backend/utils/misc/postgresql.conf.sample
index fba0ad4b624..e68e112c72f 100644
--- a/src/backend/utils/misc/postgresql.conf.sample
+++ b/src/backend/utils/misc/postgresql.conf.sample
@@ -848,6 +848,7 @@ autovacuum_worker_slots = 16 # autovacuum worker slots to allocate
#------------------------------------------------------------------------------
#io_method = sync # (change requires restart)
+#io_workers = 3 # 1-32;
#io_max_concurrency = 32 # Max number of IOs that may be in
# flight at the same time in one backend
--
2.48.1.76.g4e746b1a31.dirty
[text/x-diff] v2.3-0013-aio-Add-worker-method.patch (17.3K, ../../p5flybfcjrzgpgwcnzkxsp3xdw3czfoafkvks6g7en6ewmvymh@wrz7z2vwwc7c/14-v2.3-0013-aio-Add-worker-method.patch)
download | inline diff:
From 5bdabe467f82dc7cc7348d8698b0c10f7bbeb7b8 Mon Sep 17 00:00:00 2001
From: Andres Freund <andres@anarazel.de>
Date: Wed, 22 Jan 2025 13:44:34 -0500
Subject: [PATCH v2.3 13/30] aio: Add worker method
---
src/include/storage/aio.h | 5 +-
src/include/storage/aio_internal.h | 1 +
src/include/storage/lwlocklist.h | 1 +
src/backend/storage/aio/aio.c | 2 +
src/backend/storage/aio/aio_init.c | 12 +-
src/backend/storage/aio/method_worker.c | 394 +++++++++++++++++-
.../utils/activity/wait_event_names.txt | 1 +
src/backend/utils/misc/postgresql.conf.sample | 2 +-
src/tools/pgindent/typedefs.list | 3 +
9 files changed, 410 insertions(+), 11 deletions(-)
diff --git a/src/include/storage/aio.h b/src/include/storage/aio.h
index ffd382593d0..39d7e4cff55 100644
--- a/src/include/storage/aio.h
+++ b/src/include/storage/aio.h
@@ -23,10 +23,11 @@
typedef enum IoMethod
{
IOMETHOD_SYNC = 0,
+ IOMETHOD_WORKER,
} IoMethod;
-/* We'll default to synchronous execution. */
-#define DEFAULT_IO_METHOD IOMETHOD_SYNC
+/* We'll default to worker based execution. */
+#define DEFAULT_IO_METHOD IOMETHOD_WORKER
/*
diff --git a/src/include/storage/aio_internal.h b/src/include/storage/aio_internal.h
index 174d365f9c0..86d8d099c91 100644
--- a/src/include/storage/aio_internal.h
+++ b/src/include/storage/aio_internal.h
@@ -285,6 +285,7 @@ extern const char *pgaio_io_get_target_name(PgAioHandle *ioh);
/* Declarations for the tables of function pointers exposed by each IO method. */
extern PGDLLIMPORT const IoMethodOps pgaio_sync_ops;
+extern PGDLLIMPORT const IoMethodOps pgaio_worker_ops;
extern PGDLLIMPORT const IoMethodOps *pgaio_method_ops;
extern PGDLLIMPORT PgAioCtl *pgaio_ctl;
diff --git a/src/include/storage/lwlocklist.h b/src/include/storage/lwlocklist.h
index cf565452382..932024b1b0b 100644
--- a/src/include/storage/lwlocklist.h
+++ b/src/include/storage/lwlocklist.h
@@ -83,3 +83,4 @@ PG_LWLOCK(49, WALSummarizer)
PG_LWLOCK(50, DSMRegistry)
PG_LWLOCK(51, InjectionPoint)
PG_LWLOCK(52, SerialControl)
+PG_LWLOCK(53, AioWorkerSubmissionQueue)
diff --git a/src/backend/storage/aio/aio.c b/src/backend/storage/aio/aio.c
index cefa888884c..6c264b61ca5 100644
--- a/src/backend/storage/aio/aio.c
+++ b/src/backend/storage/aio/aio.c
@@ -57,6 +57,7 @@ static void pgaio_io_wait(PgAioHandle *ioh, uint64 ref_generation);
/* Options for io_method. */
const struct config_enum_entry io_method_options[] = {
{"sync", IOMETHOD_SYNC, false},
+ {"worker", IOMETHOD_WORKER, false},
{NULL, 0, false}
};
@@ -73,6 +74,7 @@ PgAioBackend *pgaio_my_backend;
static const IoMethodOps *const pgaio_method_ops_table[] = {
[IOMETHOD_SYNC] = &pgaio_sync_ops,
+ [IOMETHOD_WORKER] = &pgaio_worker_ops,
};
/* callbacks for the configured io_method, set by assign_io_method */
diff --git a/src/backend/storage/aio/aio_init.c b/src/backend/storage/aio/aio_init.c
index 233c144965b..76fcdf64670 100644
--- a/src/backend/storage/aio/aio_init.c
+++ b/src/backend/storage/aio/aio_init.c
@@ -18,6 +18,7 @@
#include "storage/aio.h"
#include "storage/aio_init.h"
#include "storage/aio_internal.h"
+#include "storage/io_worker.h"
#include "storage/ipc.h"
#include "storage/proc.h"
#include "storage/shmem.h"
@@ -39,6 +40,11 @@ AioCtlShmemSize(void)
static uint32
AioProcs(void)
{
+ /*
+ * While AIO workers don't need their own AIO context, we can't currently
+ * guarantee nothing gets assigned to the a ProcNumber for an IO worker if
+ * we just subtracted MAX_IO_WORKERS.
+ */
return MaxBackends + NUM_AUXILIARY_PROCS;
}
@@ -211,6 +217,9 @@ pgaio_init_backend(void)
/* shouldn't be initialized twice */
Assert(!pgaio_my_backend);
+ if (MyBackendType == B_IO_WORKER)
+ return;
+
if (MyProc == NULL || MyProcNumber >= AioProcs())
elog(ERROR, "aio requires a normal PGPROC");
@@ -225,6 +234,5 @@ pgaio_init_backend(void)
bool
pgaio_workers_enabled(void)
{
- /* placeholder for future commit */
- return false;
+ return io_method == IOMETHOD_WORKER;
}
diff --git a/src/backend/storage/aio/method_worker.c b/src/backend/storage/aio/method_worker.c
index 1d79e7e85ef..92415467c71 100644
--- a/src/backend/storage/aio/method_worker.c
+++ b/src/backend/storage/aio/method_worker.c
@@ -1,7 +1,22 @@
/*-------------------------------------------------------------------------
*
* method_worker.c
- * AIO implementation using workers
+ * AIO - perform AIO using worker processes
+ *
+ * Worker processes consume IOs from a shared memory submission queue, run
+ * traditional synchronous system calls, and perform the shared completion
+ * handling immediately. Client code submits most requests by pushing IOs
+ * into the submission queue, and waits (if necessary) using condition
+ * variables. Some IOs cannot be performed in another process due to lack of
+ * infrastructure for reopening the file, and must processed synchronously by
+ * the client code when submitted.
+ *
+ * So that the submitter can make just one system call when submitting a batch
+ * of IOs, wakeups "fan out"; each woken backend can wake two more. XXX This
+ * could be improved by using futexes instead of latches to wake N waiters.
+ *
+ * This method of AIO is available in all builds on all operating systems, and
+ * is the default.
*
* Portions Copyright (c) 1996-2025, PostgreSQL Global Development Group
* Portions Copyright (c) 1994, Regents of the University of California
@@ -16,23 +31,323 @@
#include "libpq/pqsignal.h"
#include "miscadmin.h"
+#include "port/pg_bitutils.h"
#include "postmaster/auxprocess.h"
#include "postmaster/interrupt.h"
+#include "storage/aio.h"
+#include "storage/aio_internal.h"
#include "storage/io_worker.h"
#include "storage/ipc.h"
#include "storage/latch.h"
#include "storage/proc.h"
#include "tcop/tcopprot.h"
+#include "utils/ps_status.h"
#include "utils/wait_event.h"
+/* How many workers should each worker wake up if needed? */
+#define IO_WORKER_WAKEUP_FANOUT 2
+
+
+typedef struct AioWorkerSubmissionQueue
+{
+ uint32 size;
+ uint32 mask;
+ uint32 head;
+ uint32 tail;
+ uint32 ios[FLEXIBLE_ARRAY_MEMBER];
+} AioWorkerSubmissionQueue;
+
+typedef struct AioWorkerSlot
+{
+ Latch *latch;
+ bool in_use;
+} AioWorkerSlot;
+
+typedef struct AioWorkerControl
+{
+ uint64 idle_worker_mask;
+ AioWorkerSlot workers[FLEXIBLE_ARRAY_MEMBER];
+} AioWorkerControl;
+
+
+static size_t pgaio_worker_shmem_size(void);
+static void pgaio_worker_shmem_init(bool first_time);
+
+static bool pgaio_worker_needs_synchronous_execution(PgAioHandle *ioh);
+static int pgaio_worker_submit(uint16 num_staged_ios, PgAioHandle **staged_ios);
+
+
+const IoMethodOps pgaio_worker_ops = {
+ .shmem_size = pgaio_worker_shmem_size,
+ .shmem_init = pgaio_worker_shmem_init,
+
+ .needs_synchronous_execution = pgaio_worker_needs_synchronous_execution,
+ .submit = pgaio_worker_submit,
+};
+
+
int io_workers = 3;
+static int io_worker_queue_size = 64;
+static int MyIoWorkerId;
+
+
+static AioWorkerSubmissionQueue *io_worker_submission_queue;
+static AioWorkerControl *io_worker_control;
+
+
+static size_t
+pgaio_worker_shmem_size(void)
+{
+ return
+ offsetof(AioWorkerSubmissionQueue, ios) +
+ sizeof(uint32) * io_worker_queue_size +
+ offsetof(AioWorkerControl, workers) +
+ sizeof(AioWorkerSlot) * io_workers;
+}
+
+static void
+pgaio_worker_shmem_init(bool first_time)
+{
+ bool found;
+ int size;
+
+ /* Round size up to next power of two so we can make a mask. */
+ size = pg_nextpower2_32(io_worker_queue_size);
+
+ io_worker_submission_queue =
+ ShmemInitStruct("AioWorkerSubmissionQueue",
+ offsetof(AioWorkerSubmissionQueue, ios) +
+ sizeof(uint32) * size,
+ &found);
+ if (!found)
+ {
+ io_worker_submission_queue->size = size;
+ io_worker_submission_queue->head = 0;
+ io_worker_submission_queue->tail = 0;
+ }
+
+ io_worker_control =
+ ShmemInitStruct("AioWorkerControl",
+ offsetof(AioWorkerControl, workers) +
+ sizeof(AioWorkerSlot) * io_workers,
+ &found);
+ if (!found)
+ {
+ io_worker_control->idle_worker_mask = 0;
+ for (int i = 0; i < io_workers; ++i)
+ {
+ io_worker_control->workers[i].latch = NULL;
+ io_worker_control->workers[i].in_use = false;
+ }
+ }
+}
+
+
+static int
+pgaio_choose_idle_worker(void)
+{
+ int worker;
+
+ if (io_worker_control->idle_worker_mask == 0)
+ return -1;
+
+ /* Find the lowest bit position, and clear it. */
+ worker = pg_rightmost_one_pos64(io_worker_control->idle_worker_mask);
+ io_worker_control->idle_worker_mask &= ~(UINT64_C(1) << worker);
+
+ return worker;
+}
+
+static bool
+pgaio_worker_submission_queue_insert(PgAioHandle *ioh)
+{
+ AioWorkerSubmissionQueue *queue;
+ uint32 new_head;
+
+ queue = io_worker_submission_queue;
+ new_head = (queue->head + 1) & (queue->size - 1);
+ if (new_head == queue->tail)
+ {
+ pgaio_debug(DEBUG1, "io queue is full, at %ud elements",
+ io_worker_submission_queue->size);
+ return false; /* full */
+ }
+
+ queue->ios[queue->head] = pgaio_io_get_id(ioh);
+ queue->head = new_head;
+
+ return true;
+}
+
+static uint32
+pgaio_worker_submission_queue_consume(void)
+{
+ AioWorkerSubmissionQueue *queue;
+ uint32 result;
+
+ queue = io_worker_submission_queue;
+ if (queue->tail == queue->head)
+ return UINT32_MAX; /* empty */
+
+ result = queue->ios[queue->tail];
+ queue->tail = (queue->tail + 1) & (queue->size - 1);
+
+ return result;
+}
+
+static uint32
+pgaio_worker_submission_queue_depth(void)
+{
+ uint32 head;
+ uint32 tail;
+
+ head = io_worker_submission_queue->head;
+ tail = io_worker_submission_queue->tail;
+
+ if (tail > head)
+ head += io_worker_submission_queue->size;
+
+ Assert(head >= tail);
+
+ return head - tail;
+}
+
+static void
+pgaio_worker_submit_internal(int nios, PgAioHandle *ios[])
+{
+ PgAioHandle *synchronous_ios[PGAIO_SUBMIT_BATCH_SIZE];
+ int nsync = 0;
+ Latch *wakeup = NULL;
+ int worker;
+
+ Assert(nios <= PGAIO_SUBMIT_BATCH_SIZE);
+
+ LWLockAcquire(AioWorkerSubmissionQueueLock, LW_EXCLUSIVE);
+ for (int i = 0; i < nios; ++i)
+ {
+ Assert(!pgaio_worker_needs_synchronous_execution(ios[i]));
+ if (!pgaio_worker_submission_queue_insert(ios[i]))
+ {
+ /*
+ * We'll do it synchronously, but only after we've sent as many as
+ * we can to workers, to maximize concurrency.
+ */
+ synchronous_ios[nsync++] = ios[i];
+ continue;
+ }
+
+ if (wakeup == NULL)
+ {
+ /* Choose an idle worker to wake up if we haven't already. */
+ worker = pgaio_choose_idle_worker();
+ if (worker >= 0)
+ wakeup = io_worker_control->workers[worker].latch;
+
+ pgaio_debug_io(DEBUG4, ios[i],
+ "choosing worker %d",
+ worker);
+ }
+ }
+ LWLockRelease(AioWorkerSubmissionQueueLock);
+
+ if (wakeup)
+ SetLatch(wakeup);
+
+ /* Run whatever is left synchronously. */
+ if (nsync > 0)
+ {
+ for (int i = 0; i < nsync; ++i)
+ {
+ pgaio_io_perform_synchronously(synchronous_ios[i]);
+ }
+ }
+}
+
+static bool
+pgaio_worker_needs_synchronous_execution(PgAioHandle *ioh)
+{
+ return
+ !IsUnderPostmaster
+ || ioh->flags & PGAIO_HF_REFERENCES_LOCAL
+ || !pgaio_io_can_reopen(ioh);
+}
+
+static int
+pgaio_worker_submit(uint16 num_staged_ios, PgAioHandle **staged_ios)
+{
+ for (int i = 0; i < num_staged_ios; i++)
+ {
+ PgAioHandle *ioh = staged_ios[i];
+
+ pgaio_io_prepare_submit(ioh);
+ }
+
+ pgaio_worker_submit_internal(num_staged_ios, staged_ios);
+
+ return num_staged_ios;
+}
+
+/*
+ * shmem_exit() callback that releases the worker's slot in io_worker_control.
+ */
+static void
+pgaio_worker_die(int code, Datum arg)
+{
+ LWLockAcquire(AioWorkerSubmissionQueueLock, LW_EXCLUSIVE);
+ Assert(io_worker_control->workers[MyIoWorkerId].in_use);
+ Assert(io_worker_control->workers[MyIoWorkerId].latch == MyLatch);
+
+ io_worker_control->workers[MyIoWorkerId].in_use = false;
+ io_worker_control->workers[MyIoWorkerId].latch = NULL;
+ LWLockRelease(AioWorkerSubmissionQueueLock);
+}
+
+/*
+ * Register the worker in shared memory, assign MyWorkerId and register a
+ * shutdown callback to release registration.
+ */
+static void
+pgaio_worker_register(void)
+{
+ MyIoWorkerId = -1;
+
+ /*
+ * XXX: This could do with more fine-grained locking. But it's also not
+ * very common for the number of workers to change at the moment...
+ */
+ LWLockAcquire(AioWorkerSubmissionQueueLock, LW_EXCLUSIVE);
+
+ for (int i = 0; i < io_workers; ++i)
+ {
+ if (!io_worker_control->workers[i].in_use)
+ {
+ Assert(io_worker_control->workers[i].latch == NULL);
+ io_worker_control->workers[i].in_use = true;
+ MyIoWorkerId = i;
+ break;
+ }
+ else
+ Assert(io_worker_control->workers[i].latch != NULL);
+ }
+
+ if (MyIoWorkerId == -1)
+ elog(ERROR, "couldn't find a free worker slot");
+
+ io_worker_control->idle_worker_mask |= (UINT64_C(1) << MyIoWorkerId);
+ io_worker_control->workers[MyIoWorkerId].latch = MyLatch;
+ LWLockRelease(AioWorkerSubmissionQueueLock);
+
+ on_shmem_exit(pgaio_worker_die, 0);
+}
void
IoWorkerMain(char *startup_data, size_t startup_data_len)
{
sigjmp_buf local_sigjmp_buf;
+ volatile PgAioHandle *ioh = NULL;
+ char cmd[128];
MyBackendType = B_IO_WORKER;
AuxiliaryProcessMainCommon();
@@ -53,6 +368,11 @@ IoWorkerMain(char *startup_data, size_t startup_data_len)
pqsignal(SIGUSR2, SignalHandlerForShutdownRequest);
sigprocmask(SIG_SETMASK, &UnBlockSig, NULL);
+ pgaio_worker_register();
+
+ sprintf(cmd, "io worker: %d", MyIoWorkerId);
+ set_ps_display(cmd);
+
/* see PostgresMain() */
if (sigsetjmp(local_sigjmp_buf, 1) != 0)
{
@@ -65,9 +385,18 @@ IoWorkerMain(char *startup_data, size_t startup_data_len)
*/
LWLockReleaseAll();
- /* TODO: recover from IO errors */
+ /* FIXME: recover from IO errors */
+ if (ioh != NULL)
+ {
+#if 0
+ /* EINTR is treated as a retryable error */
+ pgaio_process_io_completion(unvolatize(PgAioInProgress *, io),
+ EINTR);
+#endif
+ }
EmitErrorReport();
+
proc_exit(1);
}
@@ -76,10 +405,63 @@ IoWorkerMain(char *startup_data, size_t startup_data_len)
while (!ShutdownRequestPending)
{
- WaitLatch(MyLatch, WL_LATCH_SET | WL_EXIT_ON_PM_DEATH, -1,
- WAIT_EVENT_IO_WORKER_MAIN);
- ResetLatch(MyLatch);
- CHECK_FOR_INTERRUPTS();
+ uint32 io_index;
+ Latch *latches[IO_WORKER_WAKEUP_FANOUT];
+ int nlatches = 0;
+ int nwakeups = 0;
+ int worker;
+
+ /* Try to get a job to do. */
+ LWLockAcquire(AioWorkerSubmissionQueueLock, LW_EXCLUSIVE);
+ if ((io_index = pgaio_worker_submission_queue_consume()) == UINT32_MAX)
+ {
+ /* Nothing to do. Mark self idle. */
+ /*
+ * XXX: Invent some kind of back pressure to reduce useless
+ * wakeups?
+ */
+ io_worker_control->idle_worker_mask |= (UINT64_C(1) << MyIoWorkerId);
+ }
+ else
+ {
+ /* Got one. Clear idle flag. */
+ io_worker_control->idle_worker_mask &= ~(UINT64_C(1) << MyIoWorkerId);
+
+ /* See if we can wake up some peers. */
+ nwakeups = Min(pgaio_worker_submission_queue_depth(),
+ IO_WORKER_WAKEUP_FANOUT);
+ for (int i = 0; i < nwakeups; ++i)
+ {
+ if ((worker = pgaio_choose_idle_worker()) < 0)
+ break;
+ latches[nlatches++] = io_worker_control->workers[worker].latch;
+ }
+ }
+ LWLockRelease(AioWorkerSubmissionQueueLock);
+
+ for (int i = 0; i < nlatches; ++i)
+ SetLatch(latches[i]);
+
+ if (io_index != UINT32_MAX)
+ {
+ ioh = &pgaio_ctl->io_handles[io_index];
+
+ pgaio_debug_io(DEBUG4, unvolatize(PgAioHandle *, ioh),
+ "worker %d processing IO",
+ MyIoWorkerId);
+
+ pgaio_io_reopen(unvolatize(PgAioHandle *, ioh));
+ pgaio_io_perform_synchronously(unvolatize(PgAioHandle *, ioh));
+
+ ioh = NULL;
+ }
+ else
+ {
+ WaitLatch(MyLatch, WL_LATCH_SET | WL_EXIT_ON_PM_DEATH, -1,
+ WAIT_EVENT_IO_WORKER_MAIN);
+ ResetLatch(MyLatch);
+ CHECK_FOR_INTERRUPTS();
+ }
}
proc_exit(0);
diff --git a/src/backend/utils/activity/wait_event_names.txt b/src/backend/utils/activity/wait_event_names.txt
index e702aa7152a..05751417482 100644
--- a/src/backend/utils/activity/wait_event_names.txt
+++ b/src/backend/utils/activity/wait_event_names.txt
@@ -350,6 +350,7 @@ WALSummarizer "Waiting to read or update WAL summarization state."
DSMRegistry "Waiting to read or update the dynamic shared memory registry."
InjectionPoint "Waiting to read or update information related to injection points."
SerialControl "Waiting to read or update shared <filename>pg_serial</filename> state."
+AioWorkerSubmissionQueue "Waiting to access AIO worker submission queue."
#
# END OF PREDEFINED LWLOCKS (DO NOT CHANGE THIS LINE)
diff --git a/src/backend/utils/misc/postgresql.conf.sample b/src/backend/utils/misc/postgresql.conf.sample
index e68e112c72f..5005e65cee0 100644
--- a/src/backend/utils/misc/postgresql.conf.sample
+++ b/src/backend/utils/misc/postgresql.conf.sample
@@ -847,7 +847,7 @@ autovacuum_worker_slots = 16 # autovacuum worker slots to allocate
# WIP AIO GUC docs
#------------------------------------------------------------------------------
-#io_method = sync # (change requires restart)
+#io_method = worker # (change requires restart)
#io_workers = 3 # 1-32;
#io_max_concurrency = 32 # Max number of IOs that may be in
diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list
index c7f34559b1b..1e7bbeff1b6 100644
--- a/src/tools/pgindent/typedefs.list
+++ b/src/tools/pgindent/typedefs.list
@@ -55,6 +55,9 @@ AggStrategy
AggTransInfo
Aggref
AggregateInstrumentation
+AioWorkerControl
+AioWorkerSlot
+AioWorkerSubmissionQueue
AlenState
Alias
AllocBlock
--
2.48.1.76.g4e746b1a31.dirty
[text/x-diff] v2.3-0014-aio-Add-liburing-dependency.patch (11.3K, ../../p5flybfcjrzgpgwcnzkxsp3xdw3czfoafkvks6g7en6ewmvymh@wrz7z2vwwc7c/15-v2.3-0014-aio-Add-liburing-dependency.patch)
download | inline diff:
From cbd5bc8e99f0d80fa37f5065c893751f238c26da Mon Sep 17 00:00:00 2001
From: Andres Freund <andres@anarazel.de>
Date: Wed, 5 Jun 2024 19:37:25 -0700
Subject: [PATCH v2.3 14/30] aio: Add liburing dependency
Not yet used.
Author:
Reviewed-by:
Discussion: https://postgr.es/m/
Backpatch:
---
meson.build | 14 ++++
meson_options.txt | 3 +
configure.ac | 11 +++
src/makefiles/meson.build | 3 +
src/include/pg_config.h.in | 3 +
src/backend/Makefile | 7 +-
configure | 138 +++++++++++++++++++++++++++++++++++++
.cirrus.tasks.yml | 1 +
src/Makefile.global.in | 4 ++
9 files changed, 181 insertions(+), 3 deletions(-)
diff --git a/meson.build b/meson.build
index 32fc89f3a4b..2bca586e5f3 100644
--- a/meson.build
+++ b/meson.build
@@ -854,6 +854,18 @@ endif
+###############################################################
+# Library: liburing
+###############################################################
+
+liburingopt = get_option('liburing')
+liburing = dependency('liburing', required: liburingopt)
+if liburing.found()
+ cdata.set('USE_LIBURING', 1)
+endif
+
+
+
###############################################################
# Library: libxml
###############################################################
@@ -3058,6 +3070,7 @@ backend_both_deps += [
icu_i18n,
ldap,
libintl,
+ liburing,
libxml,
lz4,
pam,
@@ -3702,6 +3715,7 @@ if meson.version().version_compare('>=0.57')
'gss': gssapi,
'icu': icu,
'ldap': ldap,
+ 'liburing': liburing,
'libxml': libxml,
'libxslt': libxslt,
'llvm': llvm,
diff --git a/meson_options.txt b/meson_options.txt
index d9c7ddccbc4..abe8600ec35 100644
--- a/meson_options.txt
+++ b/meson_options.txt
@@ -103,6 +103,9 @@ option('ldap', type: 'feature', value: 'auto',
option('libedit_preferred', type: 'boolean', value: false,
description: 'Prefer BSD Libedit over GNU Readline')
+option('liburing', type : 'feature', value: 'auto',
+ description: 'Use liburing for async io')
+
option('libxml', type: 'feature', value: 'auto',
description: 'XML support')
diff --git a/configure.ac b/configure.ac
index d713360f340..00d6c366ecd 100644
--- a/configure.ac
+++ b/configure.ac
@@ -975,6 +975,14 @@ AC_SUBST(with_readline)
PGAC_ARG_BOOL(with, libedit-preferred, no,
[prefer BSD Libedit over GNU Readline])
+#
+# liburing
+#
+AC_MSG_CHECKING([whether to build with liburing support])
+PGAC_ARG_BOOL(with, liburing, no, [use liburing for async io],
+ [AC_DEFINE([USE_LIBURING], 1, [Define to build with io-uring support. (--with-liburing)])])
+AC_MSG_RESULT([$with_liburing])
+AC_SUBST(with_liburing)
#
# UUID library
@@ -1427,6 +1435,9 @@ elif test "$with_uuid" = ossp ; then
fi
AC_SUBST(UUID_LIBS)
+if test "$with_liburing" = yes; then
+ PKG_CHECK_MODULES(LIBURING, liburing)
+fi
##
## Header files
diff --git a/src/makefiles/meson.build b/src/makefiles/meson.build
index d49b2079a44..714b7ccaa4e 100644
--- a/src/makefiles/meson.build
+++ b/src/makefiles/meson.build
@@ -199,6 +199,8 @@ pgxs_empty = [
'PTHREAD_CFLAGS', 'PTHREAD_LIBS',
'ICU_LIBS',
+
+ 'LIBURING_CFLAGS', 'LIBURING_LIBS',
]
if host_system == 'windows' and cc.get_argument_syntax() != 'msvc'
@@ -229,6 +231,7 @@ pgxs_deps = {
'gssapi': gssapi,
'icu': icu,
'ldap': ldap,
+ 'liburing': liburing,
'libxml': libxml,
'libxslt': libxslt,
'llvm': llvm,
diff --git a/src/include/pg_config.h.in b/src/include/pg_config.h.in
index 07b2f798abd..6ab71a3dffe 100644
--- a/src/include/pg_config.h.in
+++ b/src/include/pg_config.h.in
@@ -663,6 +663,9 @@
/* Define to 1 to build with LDAP support. (--with-ldap) */
#undef USE_LDAP
+/* Define to build with io-uring support. (--with-liburing) */
+#undef USE_LIBURING
+
/* Define to 1 to build with XML support. (--with-libxml) */
#undef USE_LIBXML
diff --git a/src/backend/Makefile b/src/backend/Makefile
index 42d4a28e5aa..7344c8c7f5c 100644
--- a/src/backend/Makefile
+++ b/src/backend/Makefile
@@ -43,9 +43,10 @@ OBJS = \
$(top_builddir)/src/common/libpgcommon_srv.a \
$(top_builddir)/src/port/libpgport_srv.a
-# We put libpgport and libpgcommon into OBJS, so remove it from LIBS; also add
-# libldap and ICU
-LIBS := $(filter-out -lpgport -lpgcommon, $(LIBS)) $(LDAP_LIBS_BE) $(ICU_LIBS)
+# We put libpgport and libpgcommon into OBJS, so remove it from LIBS.
+LIBS := $(filter-out -lpgport -lpgcommon, $(LIBS))
+# The backend conditionally needs libraries that most executables don't need.
+LIBS += $(LDAP_LIBS_BE) $(ICU_LIBS) $(LIBURING_LIBS)
# The backend doesn't need everything that's in LIBS, however
LIBS := $(filter-out -lreadline -ledit -ltermcap -lncurses -lcurses, $(LIBS))
diff --git a/configure b/configure
index ceeef9b0915..e477baedfb6 100755
--- a/configure
+++ b/configure
@@ -651,6 +651,8 @@ LIBOBJS
OPENSSL
ZSTD
LZ4
+LIBURING_LIBS
+LIBURING_CFLAGS
UUID_LIBS
LDAP_LIBS_BE
LDAP_LIBS_FE
@@ -709,6 +711,7 @@ XML2_CFLAGS
XML2_CONFIG
with_libxml
with_uuid
+with_liburing
with_readline
with_systemd
with_selinux
@@ -862,6 +865,7 @@ with_selinux
with_systemd
with_readline
with_libedit_preferred
+with_liburing
with_uuid
with_ossp_uuid
with_libxml
@@ -905,6 +909,8 @@ LDFLAGS_EX
LDFLAGS_SL
PERL
PYTHON
+LIBURING_CFLAGS
+LIBURING_LIBS
MSGFMT
TCLSH'
@@ -1572,6 +1578,7 @@ Optional Packages:
--without-readline do not use GNU Readline nor BSD Libedit for editing
--with-libedit-preferred
prefer BSD Libedit over GNU Readline
+ --with-liburing use liburing for async io
--with-uuid=LIB build contrib/uuid-ossp using LIB (bsd,e2fs,ossp)
--with-ossp-uuid obsolete spelling of --with-uuid=ossp
--with-libxml build with XML support
@@ -1618,6 +1625,10 @@ Some influential environment variables:
LDFLAGS_SL extra linker flags for linking shared libraries only
PERL Perl program
PYTHON Python program
+ LIBURING_CFLAGS
+ C compiler flags for LIBURING, overriding pkg-config
+ LIBURING_LIBS
+ linker flags for LIBURING, overriding pkg-config
MSGFMT msgfmt program for NLS
TCLSH Tcl interpreter program (tclsh)
@@ -8681,6 +8692,40 @@ fi
+#
+# liburing
+#
+{ $as_echo "$as_me:${as_lineno-$LINENO}: checking whether to build with liburing support" >&5
+$as_echo_n "checking whether to build with liburing support... " >&6; }
+
+
+
+# Check whether --with-liburing was given.
+if test "${with_liburing+set}" = set; then :
+ withval=$with_liburing;
+ case $withval in
+ yes)
+
+$as_echo "#define USE_LIBURING 1" >>confdefs.h
+
+ ;;
+ no)
+ :
+ ;;
+ *)
+ as_fn_error $? "no argument expected for --with-liburing option" "$LINENO" 5
+ ;;
+ esac
+
+else
+ with_liburing=no
+
+fi
+
+
+{ $as_echo "$as_me:${as_lineno-$LINENO}: result: $with_liburing" >&5
+$as_echo "$with_liburing" >&6; }
+
#
# UUID library
@@ -13231,6 +13276,99 @@ fi
fi
+if test "$with_liburing" = yes; then
+
+pkg_failed=no
+{ $as_echo "$as_me:${as_lineno-$LINENO}: checking for liburing" >&5
+$as_echo_n "checking for liburing... " >&6; }
+
+if test -n "$LIBURING_CFLAGS"; then
+ pkg_cv_LIBURING_CFLAGS="$LIBURING_CFLAGS"
+ elif test -n "$PKG_CONFIG"; then
+ if test -n "$PKG_CONFIG" && \
+ { { $as_echo "$as_me:${as_lineno-$LINENO}: \$PKG_CONFIG --exists --print-errors \"liburing\""; } >&5
+ ($PKG_CONFIG --exists --print-errors "liburing") 2>&5
+ ac_status=$?
+ $as_echo "$as_me:${as_lineno-$LINENO}: \$? = $ac_status" >&5
+ test $ac_status = 0; }; then
+ pkg_cv_LIBURING_CFLAGS=`$PKG_CONFIG --cflags "liburing" 2>/dev/null`
+ test "x$?" != "x0" && pkg_failed=yes
+else
+ pkg_failed=yes
+fi
+ else
+ pkg_failed=untried
+fi
+if test -n "$LIBURING_LIBS"; then
+ pkg_cv_LIBURING_LIBS="$LIBURING_LIBS"
+ elif test -n "$PKG_CONFIG"; then
+ if test -n "$PKG_CONFIG" && \
+ { { $as_echo "$as_me:${as_lineno-$LINENO}: \$PKG_CONFIG --exists --print-errors \"liburing\""; } >&5
+ ($PKG_CONFIG --exists --print-errors "liburing") 2>&5
+ ac_status=$?
+ $as_echo "$as_me:${as_lineno-$LINENO}: \$? = $ac_status" >&5
+ test $ac_status = 0; }; then
+ pkg_cv_LIBURING_LIBS=`$PKG_CONFIG --libs "liburing" 2>/dev/null`
+ test "x$?" != "x0" && pkg_failed=yes
+else
+ pkg_failed=yes
+fi
+ else
+ pkg_failed=untried
+fi
+
+
+
+if test $pkg_failed = yes; then
+ { $as_echo "$as_me:${as_lineno-$LINENO}: result: no" >&5
+$as_echo "no" >&6; }
+
+if $PKG_CONFIG --atleast-pkgconfig-version 0.20; then
+ _pkg_short_errors_supported=yes
+else
+ _pkg_short_errors_supported=no
+fi
+ if test $_pkg_short_errors_supported = yes; then
+ LIBURING_PKG_ERRORS=`$PKG_CONFIG --short-errors --print-errors --cflags --libs "liburing" 2>&1`
+ else
+ LIBURING_PKG_ERRORS=`$PKG_CONFIG --print-errors --cflags --libs "liburing" 2>&1`
+ fi
+ # Put the nasty error message in config.log where it belongs
+ echo "$LIBURING_PKG_ERRORS" >&5
+
+ as_fn_error $? "Package requirements (liburing) were not met:
+
+$LIBURING_PKG_ERRORS
+
+Consider adjusting the PKG_CONFIG_PATH environment variable if you
+installed software in a non-standard prefix.
+
+Alternatively, you may set the environment variables LIBURING_CFLAGS
+and LIBURING_LIBS to avoid the need to call pkg-config.
+See the pkg-config man page for more details." "$LINENO" 5
+elif test $pkg_failed = untried; then
+ { $as_echo "$as_me:${as_lineno-$LINENO}: result: no" >&5
+$as_echo "no" >&6; }
+ { { $as_echo "$as_me:${as_lineno-$LINENO}: error: in \`$ac_pwd':" >&5
+$as_echo "$as_me: error: in \`$ac_pwd':" >&2;}
+as_fn_error $? "The pkg-config script could not be found or is too old. Make sure it
+is in your PATH or set the PKG_CONFIG environment variable to the full
+path to pkg-config.
+
+Alternatively, you may set the environment variables LIBURING_CFLAGS
+and LIBURING_LIBS to avoid the need to call pkg-config.
+See the pkg-config man page for more details.
+
+To get pkg-config, see <http://pkg-config.freedesktop.org/>.
+See \`config.log' for more details" "$LINENO" 5; }
+else
+ LIBURING_CFLAGS=$pkg_cv_LIBURING_CFLAGS
+ LIBURING_LIBS=$pkg_cv_LIBURING_LIBS
+ { $as_echo "$as_me:${as_lineno-$LINENO}: result: yes" >&5
+$as_echo "yes" >&6; }
+
+fi
+fi
##
## Header files
diff --git a/.cirrus.tasks.yml b/.cirrus.tasks.yml
index 18e944ca89d..67d3d77fb10 100644
--- a/.cirrus.tasks.yml
+++ b/.cirrus.tasks.yml
@@ -334,6 +334,7 @@ task:
--enable-cassert --enable-injection-points --enable-debug \
--enable-tap-tests --enable-nls \
--with-segsize-blocks=6 \
+ --with-liburing \
\
${LINUX_CONFIGURE_FEATURES} \
\
diff --git a/src/Makefile.global.in b/src/Makefile.global.in
index 1278b7744f4..8ad259a54cd 100644
--- a/src/Makefile.global.in
+++ b/src/Makefile.global.in
@@ -190,6 +190,7 @@ with_systemd = @with_systemd@
with_gssapi = @with_gssapi@
with_krb_srvnam = @with_krb_srvnam@
with_ldap = @with_ldap@
+with_liburing = @with_liburing@
with_libxml = @with_libxml@
with_libxslt = @with_libxslt@
with_llvm = @with_llvm@
@@ -216,6 +217,9 @@ krb_srvtab = @krb_srvtab@
ICU_CFLAGS = @ICU_CFLAGS@
ICU_LIBS = @ICU_LIBS@
+LIBURING_CFLAGS = @LIBURING_CFLAGS@
+LIBURING_LIBS = @LIBURING_LIBS@
+
TCLSH = @TCLSH@
TCL_LIBS = @TCL_LIBS@
TCL_LIB_SPEC = @TCL_LIB_SPEC@
--
2.48.1.76.g4e746b1a31.dirty
[text/x-diff] v2.3-0015-aio-Add-io_uring-method.patch (14.2K, ../../p5flybfcjrzgpgwcnzkxsp3xdw3czfoafkvks6g7en6ewmvymh@wrz7z2vwwc7c/16-v2.3-0015-aio-Add-io_uring-method.patch)
download | inline diff:
From 8729492fc8eb698851442f0165cb12948c4db4f4 Mon Sep 17 00:00:00 2001
From: Andres Freund <andres@anarazel.de>
Date: Wed, 22 Jan 2025 13:44:36 -0500
Subject: [PATCH v2.3 15/30] aio: Add io_uring method
---
src/include/storage/aio.h | 3 +
src/include/storage/aio_internal.h | 3 +
src/include/storage/lwlock.h | 1 +
src/backend/storage/aio/Makefile | 1 +
src/backend/storage/aio/aio.c | 6 +
src/backend/storage/aio/meson.build | 1 +
src/backend/storage/aio/method_io_uring.c | 382 ++++++++++++++++++++++
src/backend/storage/lmgr/lwlock.c | 1 +
src/tools/pgindent/typedefs.list | 1 +
9 files changed, 399 insertions(+)
create mode 100644 src/backend/storage/aio/method_io_uring.c
diff --git a/src/include/storage/aio.h b/src/include/storage/aio.h
index 39d7e4cff55..8c1b9a1b496 100644
--- a/src/include/storage/aio.h
+++ b/src/include/storage/aio.h
@@ -24,6 +24,9 @@ typedef enum IoMethod
{
IOMETHOD_SYNC = 0,
IOMETHOD_WORKER,
+#ifdef USE_LIBURING
+ IOMETHOD_IO_URING,
+#endif
} IoMethod;
/* We'll default to worker based execution. */
diff --git a/src/include/storage/aio_internal.h b/src/include/storage/aio_internal.h
index 86d8d099c91..eff544ce621 100644
--- a/src/include/storage/aio_internal.h
+++ b/src/include/storage/aio_internal.h
@@ -286,6 +286,9 @@ extern const char *pgaio_io_get_target_name(PgAioHandle *ioh);
/* Declarations for the tables of function pointers exposed by each IO method. */
extern PGDLLIMPORT const IoMethodOps pgaio_sync_ops;
extern PGDLLIMPORT const IoMethodOps pgaio_worker_ops;
+#ifdef USE_LIBURING
+extern PGDLLIMPORT const IoMethodOps pgaio_uring_ops;
+#endif
extern PGDLLIMPORT const IoMethodOps *pgaio_method_ops;
extern PGDLLIMPORT PgAioCtl *pgaio_ctl;
diff --git a/src/include/storage/lwlock.h b/src/include/storage/lwlock.h
index 13a7dc89980..043e8bae7a9 100644
--- a/src/include/storage/lwlock.h
+++ b/src/include/storage/lwlock.h
@@ -217,6 +217,7 @@ typedef enum BuiltinTrancheIds
LWTRANCHE_SUBTRANS_SLRU,
LWTRANCHE_XACT_SLRU,
LWTRANCHE_PARALLEL_VACUUM_DSA,
+ LWTRANCHE_AIO_URING_COMPLETION,
LWTRANCHE_FIRST_USER_DEFINED,
} BuiltinTrancheIds;
diff --git a/src/backend/storage/aio/Makefile b/src/backend/storage/aio/Makefile
index f51c34a37f8..c06c50771e0 100644
--- a/src/backend/storage/aio/Makefile
+++ b/src/backend/storage/aio/Makefile
@@ -14,6 +14,7 @@ OBJS = \
aio_init.o \
aio_io.o \
aio_target.o \
+ method_io_uring.o \
method_sync.o \
method_worker.o \
read_stream.o
diff --git a/src/backend/storage/aio/aio.c b/src/backend/storage/aio/aio.c
index 6c264b61ca5..c1dd073e37f 100644
--- a/src/backend/storage/aio/aio.c
+++ b/src/backend/storage/aio/aio.c
@@ -58,6 +58,9 @@ static void pgaio_io_wait(PgAioHandle *ioh, uint64 ref_generation);
const struct config_enum_entry io_method_options[] = {
{"sync", IOMETHOD_SYNC, false},
{"worker", IOMETHOD_WORKER, false},
+#ifdef USE_LIBURING
+ {"io_uring", IOMETHOD_IO_URING, false},
+#endif
{NULL, 0, false}
};
@@ -75,6 +78,9 @@ PgAioBackend *pgaio_my_backend;
static const IoMethodOps *const pgaio_method_ops_table[] = {
[IOMETHOD_SYNC] = &pgaio_sync_ops,
[IOMETHOD_WORKER] = &pgaio_worker_ops,
+#ifdef USE_LIBURING
+ [IOMETHOD_IO_URING] = &pgaio_uring_ops,
+#endif
};
/* callbacks for the configured io_method, set by assign_io_method */
diff --git a/src/backend/storage/aio/meson.build b/src/backend/storage/aio/meson.build
index 74f94c6e40b..2f0f03d8071 100644
--- a/src/backend/storage/aio/meson.build
+++ b/src/backend/storage/aio/meson.build
@@ -6,6 +6,7 @@ backend_sources += files(
'aio_init.c',
'aio_io.c',
'aio_target.c',
+ 'method_io_uring.c',
'method_sync.c',
'method_worker.c',
'read_stream.c',
diff --git a/src/backend/storage/aio/method_io_uring.c b/src/backend/storage/aio/method_io_uring.c
new file mode 100644
index 00000000000..da92795fce7
--- /dev/null
+++ b/src/backend/storage/aio/method_io_uring.c
@@ -0,0 +1,382 @@
+/*-------------------------------------------------------------------------
+ *
+ * method_io_uring.c
+ * AIO - perform AIO using Linux' io_uring
+ *
+ * XXX Write me
+ *
+ * Portions Copyright (c) 1996-2025, PostgreSQL Global Development Group
+ * Portions Copyright (c) 1994, Regents of the University of California
+ *
+ * IDENTIFICATION
+ * src/backend/storage/aio/method_io_uring.c
+ *
+ *-------------------------------------------------------------------------
+ */
+
+#include "postgres.h"
+
+#ifdef USE_LIBURING
+
+#include <liburing.h>
+
+#include "pgstat.h"
+#include "port/pg_iovec.h"
+#include "storage/aio_internal.h"
+#include "storage/fd.h"
+#include "storage/proc.h"
+#include "storage/shmem.h"
+
+
+/* Entry points for IoMethodOps. */
+static size_t pgaio_uring_shmem_size(void);
+static void pgaio_uring_shmem_init(bool first_time);
+static void pgaio_uring_init_backend(void);
+
+static int pgaio_uring_submit(uint16 num_staged_ios, PgAioHandle **staged_ios);
+static void pgaio_uring_wait_one(PgAioHandle *ioh, uint64 ref_generation);
+
+static void pgaio_uring_sq_from_io(PgAioHandle *ioh, struct io_uring_sqe *sqe);
+
+
+const IoMethodOps pgaio_uring_ops = {
+ .shmem_size = pgaio_uring_shmem_size,
+ .shmem_init = pgaio_uring_shmem_init,
+ .init_backend = pgaio_uring_init_backend,
+
+ .submit = pgaio_uring_submit,
+ .wait_one = pgaio_uring_wait_one,
+};
+
+typedef struct PgAioUringContext
+{
+ LWLock completion_lock;
+
+ struct io_uring io_uring_ring;
+ /* XXX: probably worth padding to a cacheline boundary here */
+} PgAioUringContext;
+
+
+static PgAioUringContext *pgaio_uring_contexts;
+static PgAioUringContext *pgaio_my_uring_context;
+
+/* io_uring local state */
+static struct io_uring local_ring;
+
+
+
+static Size
+pgaio_uring_context_shmem_size(void)
+{
+ uint32 TotalProcs = MaxBackends + NUM_AUXILIARY_PROCS - MAX_IO_WORKERS;
+
+ return mul_size(TotalProcs, sizeof(PgAioUringContext));
+}
+
+static size_t
+pgaio_uring_shmem_size(void)
+{
+ return pgaio_uring_context_shmem_size();
+}
+
+static void
+pgaio_uring_shmem_init(bool first_time)
+{
+ uint32 TotalProcs = MaxBackends + NUM_AUXILIARY_PROCS - MAX_IO_WORKERS;
+ bool found;
+
+ pgaio_uring_contexts = (PgAioUringContext *)
+ ShmemInitStruct("AioUring", pgaio_uring_shmem_size(), &found);
+
+ if (found)
+ return;
+
+ for (int contextno = 0; contextno < TotalProcs; contextno++)
+ {
+ PgAioUringContext *context = &pgaio_uring_contexts[contextno];
+ int ret;
+
+ /*
+ * XXX: Probably worth sharing the WQ between the different rings,
+ * when supported by the kernel. Could also cause additional
+ * contention, I guess?
+ */
+#if 0
+ if (!AcquireExternalFD())
+ elog(ERROR, "No external FD available");
+#endif
+ ret = io_uring_queue_init(io_max_concurrency, &context->io_uring_ring, 0);
+ if (ret < 0)
+ elog(ERROR, "io_uring_queue_init failed: %s", strerror(-ret));
+
+ LWLockInitialize(&context->completion_lock, LWTRANCHE_AIO_URING_COMPLETION);
+ }
+}
+
+static void
+pgaio_uring_init_backend(void)
+{
+ int ret;
+
+ pgaio_my_uring_context = &pgaio_uring_contexts[MyProcNumber];
+
+ ret = io_uring_queue_init(32, &local_ring, 0);
+ if (ret < 0)
+ elog(ERROR, "io_uring_queue_init failed: %s", strerror(-ret));
+}
+
+static int
+pgaio_uring_submit(uint16 num_staged_ios, PgAioHandle **staged_ios)
+{
+ struct io_uring *uring_instance = &pgaio_my_uring_context->io_uring_ring;
+ int in_flight_before = dclist_count(&pgaio_my_backend->in_flight_ios);
+
+ Assert(num_staged_ios <= PGAIO_SUBMIT_BATCH_SIZE);
+
+ for (int i = 0; i < num_staged_ios; i++)
+ {
+ PgAioHandle *ioh = staged_ios[i];
+ struct io_uring_sqe *sqe;
+
+ sqe = io_uring_get_sqe(uring_instance);
+
+ if (!sqe)
+ elog(ERROR, "io_uring submission queue is unexpectedly full");
+
+ pgaio_io_prepare_submit(ioh);
+ pgaio_uring_sq_from_io(ioh, sqe);
+
+ /*
+ * io_uring executes IO in process context if possible. That's
+ * generally good, as it reduces context switching. When performing a
+ * lot of buffered IO that means that copying between page cache and
+ * userspace memory happens in the foreground, as it can't be
+ * offloaded to DMA hardware as is possible when using direct IO. When
+ * executing a lot of buffered IO this causes io_uring to be slower
+ * than worker mode, as worker mode parallelizes the copying. io_uring
+ * can be told to offload work to worker threads instead.
+ *
+ * If an IO is buffered IO and we already have IOs in flight or
+ * multiple IOs are being submitted, we thus tell io_uring to execute
+ * the IO in the background. We don't do so for the first few IOs
+ * being submitted as executing in this process' context has lower
+ * latency.
+ */
+ if (in_flight_before > 4 && (ioh->flags & PGAIO_HF_BUFFERED))
+ io_uring_sqe_set_flags(sqe, IOSQE_ASYNC);
+
+ in_flight_before++;
+ }
+
+ while (true)
+ {
+ int ret;
+
+ pgstat_report_wait_start(WAIT_EVENT_AIO_SUBMIT);
+ ret = io_uring_submit(uring_instance);
+ pgstat_report_wait_end();
+
+ if (ret == -EINTR)
+ {
+ pgaio_debug(DEBUG3,
+ "aio method uring: submit EINTR, nios: %d",
+ num_staged_ios);
+ continue;
+ }
+ if (ret < 0)
+ elog(PANIC, "failed: %d/%s",
+ ret, strerror(-ret));
+ else if (ret != num_staged_ios)
+ {
+ /* likely unreachable, but if it is, we would need to re-submit */
+ elog(PANIC, "submitted only %d of %d",
+ ret, num_staged_ios);
+ }
+ else
+ {
+ pgaio_debug(DEBUG4,
+ "aio method uring: submitted %d IOs",
+ num_staged_ios);
+ }
+ break;
+ }
+
+ return num_staged_ios;
+}
+
+
+#define PGAIO_MAX_LOCAL_COMPLETED_IO 32
+
+static void
+pgaio_uring_drain_locked(PgAioUringContext *context)
+{
+ int ready;
+ int orig_ready;
+
+ /*
+ * Don't drain more events than available right now. Otherwise it's
+ * plausible that one backend could get stuck, for a while, receiving CQEs
+ * without actually processing them.
+ */
+ orig_ready = ready = io_uring_cq_ready(&context->io_uring_ring);
+
+ while (ready > 0)
+ {
+ struct io_uring_cqe *cqes[PGAIO_MAX_LOCAL_COMPLETED_IO];
+ uint32 ncqes;
+
+ START_CRIT_SECTION();
+ ncqes =
+ io_uring_peek_batch_cqe(&context->io_uring_ring,
+ cqes,
+ Min(PGAIO_MAX_LOCAL_COMPLETED_IO, ready));
+ Assert(ncqes <= ready);
+
+ ready -= ncqes;
+
+ for (int i = 0; i < ncqes; i++)
+ {
+ struct io_uring_cqe *cqe = cqes[i];
+ PgAioHandle *ioh;
+
+ ioh = io_uring_cqe_get_data(cqe);
+ io_uring_cqe_seen(&context->io_uring_ring, cqe);
+
+ pgaio_io_process_completion(ioh, cqe->res);
+ }
+
+ END_CRIT_SECTION();
+
+ pgaio_debug(DEBUG3,
+ "drained %d/%d, now expecting %d",
+ ncqes, orig_ready, io_uring_cq_ready(&context->io_uring_ring));
+ }
+}
+
+static void
+pgaio_uring_wait_one(PgAioHandle *ioh, uint64 ref_generation)
+{
+ PgAioHandleState state;
+ ProcNumber owner_procno = ioh->owner_procno;
+ PgAioUringContext *owner_context = &pgaio_uring_contexts[owner_procno];
+ bool expect_cqe;
+ int waited = 0;
+
+ /*
+ * We ought to have a smarter locking scheme, nearly all the time the
+ * backend owning the ring will consume the completions, making the
+ * locking unnecessarily expensive.
+ */
+ LWLockAcquire(&owner_context->completion_lock, LW_EXCLUSIVE);
+
+ while (true)
+ {
+ pgaio_debug_io(DEBUG3, ioh,
+ "wait_one io_gen: %llu, ref_gen: %llu, cycle %d",
+ (long long unsigned) ref_generation,
+ (long long unsigned) ioh->generation,
+ waited);
+
+ if (pgaio_io_was_recycled(ioh, ref_generation, &state) ||
+ state != PGAIO_HS_SUBMITTED)
+ {
+ break;
+ }
+ else if (io_uring_cq_ready(&owner_context->io_uring_ring))
+ {
+ expect_cqe = true;
+ }
+ else
+ {
+ int ret;
+ struct io_uring_cqe *cqes;
+
+ pgstat_report_wait_start(WAIT_EVENT_AIO_DRAIN);
+ ret = io_uring_wait_cqes(&owner_context->io_uring_ring, &cqes, 1, NULL, NULL);
+ pgstat_report_wait_end();
+
+ if (ret == -EINTR)
+ {
+ continue;
+ }
+ else if (ret != 0)
+ {
+ elog(PANIC, "unexpected: %d/%s: %m", ret, strerror(-ret));
+ }
+ else
+ {
+ Assert(cqes != NULL);
+ expect_cqe = true;
+ waited++;
+ }
+ }
+
+ if (expect_cqe)
+ {
+ pgaio_uring_drain_locked(owner_context);
+ }
+ }
+
+ LWLockRelease(&owner_context->completion_lock);
+
+ pgaio_debug(DEBUG3,
+ "wait_one with %d sleeps",
+ waited);
+}
+
+static void
+pgaio_uring_sq_from_io(PgAioHandle *ioh, struct io_uring_sqe *sqe)
+{
+ struct iovec *iov;
+
+ switch (ioh->op)
+ {
+ case PGAIO_OP_READV:
+ iov = &pgaio_ctl->iovecs[ioh->iovec_off];
+ if (ioh->op_data.read.iov_length == 1)
+ {
+ io_uring_prep_read(sqe,
+ ioh->op_data.read.fd,
+ iov->iov_base,
+ iov->iov_len,
+ ioh->op_data.read.offset);
+ }
+ else
+ {
+ io_uring_prep_readv(sqe,
+ ioh->op_data.read.fd,
+ iov,
+ ioh->op_data.read.iov_length,
+ ioh->op_data.read.offset);
+
+ }
+ break;
+
+ case PGAIO_OP_WRITEV:
+ iov = &pgaio_ctl->iovecs[ioh->iovec_off];
+ if (ioh->op_data.write.iov_length == 1)
+ {
+ io_uring_prep_write(sqe,
+ ioh->op_data.write.fd,
+ iov->iov_base,
+ iov->iov_len,
+ ioh->op_data.write.offset);
+ }
+ else
+ {
+ io_uring_prep_writev(sqe,
+ ioh->op_data.write.fd,
+ iov,
+ ioh->op_data.write.iov_length,
+ ioh->op_data.write.offset);
+ }
+ break;
+
+ case PGAIO_OP_INVALID:
+ elog(ERROR, "trying to prepare invalid IO operation for execution");
+ }
+
+ io_uring_sqe_set_data(sqe, ioh);
+}
+
+#endif /* USE_LIBURING */
diff --git a/src/backend/storage/lmgr/lwlock.c b/src/backend/storage/lmgr/lwlock.c
index c3d6f886e3c..dbc169c8541 100644
--- a/src/backend/storage/lmgr/lwlock.c
+++ b/src/backend/storage/lmgr/lwlock.c
@@ -166,6 +166,7 @@ static const char *const BuiltinTrancheNames[] = {
[LWTRANCHE_SUBTRANS_SLRU] = "SubtransSLRU",
[LWTRANCHE_XACT_SLRU] = "XactSLRU",
[LWTRANCHE_PARALLEL_VACUUM_DSA] = "ParallelVacuumDSA",
+ [LWTRANCHE_AIO_URING_COMPLETION] = "AioUringCompletion",
};
StaticAssertDecl(lengthof(BuiltinTrancheNames) ==
diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list
index 1e7bbeff1b6..be2dd22f1d7 100644
--- a/src/tools/pgindent/typedefs.list
+++ b/src/tools/pgindent/typedefs.list
@@ -2128,6 +2128,7 @@ PgAioReturn
PgAioTargetData
PgAioTargetID
PgAioTargetInfo
+PgAioUringContext
PgAioWaitRef
PgArchData
PgBackendGSSStatus
--
2.48.1.76.g4e746b1a31.dirty
[text/x-diff] v2.3-0016-aio-Add-README.md-explaining-higher-level-desig.patch (18.8K, ../../p5flybfcjrzgpgwcnzkxsp3xdw3czfoafkvks6g7en6ewmvymh@wrz7z2vwwc7c/17-v2.3-0016-aio-Add-README.md-explaining-higher-level-desig.patch)
download | inline diff:
From 0a201985c794113e4cf062e8f5037fb7ab03c1ea Mon Sep 17 00:00:00 2001
From: Andres Freund <andres@anarazel.de>
Date: Wed, 22 Jan 2025 13:44:37 -0500
Subject: [PATCH v2.3 16/30] aio: Add README.md explaining higher level design
---
src/backend/storage/aio/README.md | 430 ++++++++++++++++++++++++++++++
src/backend/storage/aio/aio.c | 2 +
2 files changed, 432 insertions(+)
create mode 100644 src/backend/storage/aio/README.md
diff --git a/src/backend/storage/aio/README.md b/src/backend/storage/aio/README.md
new file mode 100644
index 00000000000..1b6f9d2c40b
--- /dev/null
+++ b/src/backend/storage/aio/README.md
@@ -0,0 +1,430 @@
+# Asynchronous & Direct IO
+
+## Motivation
+
+### Why Asynchronous IO
+
+Until the introduction of asynchronous IO Postgres relied on the operating
+system to hide the cost of synchronous IO from Postgres. While this worked
+surprisingly well in a lot of workloads, it does not do as good a job on
+prefetching and controlled writeback as we would like.
+
+There are important expensive operations like `fdatasync()` where the operating
+system cannot hide the storage latency. This is particularly important for WAL
+writes, where the ability to asynchronously issue `fdatasync()` or O_DSYNC
+writes can yield significantly higher throughput.
+
+
+### Why Direct / unbuffered IO
+
+The main reason to want to use Direct IO are:
+
+- Lower CPU usage / higher throughput. Particularly on modern storage buffered
+ writes are bottlenecked by the operating system having to copy data from the
+ kernel's page cache to postgres buffer pool using the CPU. Whereas direct IO
+ can often move the data directly between the storage devices and postgres'
+ buffer cache, using DMA. While that transfer is ongoing, the CPU is free to
+ perform other work.
+- Reduced latency - Direct IO can have substantially lower latency than
+ buffered IO, which can be impactful for OLTP workloads bottlenecked by WAL
+ write latency.
+- Avoiding double buffering between operating system cache and postgres'
+ shared_buffers.
+- Better control over the timing and pace of dirty data writeback.
+
+
+The main reason *not* to use Direct IO are:
+
+- Without AIO, Direct IO is unusably slow for most purposes.
+- Even with AIO, many parts of postgres need to be modified to perform
+ explicit prefetching.
+- In situations where shared_buffers cannot be set appropriately large,
+ e.g. because there are many different postgres instances hosted on shared
+ hardware, performance will often be worse than when using buffered IO.
+
+
+## AIO Usage Example
+
+In many cases code that can benefit from AIO does not directly have to
+interact with the AIO interface, but can use AIO via higher-level
+abstractions. See [Helpers](#helpers).
+
+In this example, a buffer will be read into shared buffers.
+
+```C
+/*
+ * Result of the operation, only to be accessed in this backend.
+ */
+PgAioReturn ioret;
+
+/*
+ * Acquire an AIO Handle, ioret will get result upon completion.
+ *
+ * Note that ioret needs to stay alive until the IO completes or
+ * CurrentResourceOwner is released (i.e. an error is thrown).
+ */
+PgAioHandle *ioh = pgaio_io_acquire(CurrentResourceOwner, &ioret);
+
+/*
+ * Reference that can be used to wait for the IO we initiate below. This
+ * reference can reside in local or shared memory and waited upon by any
+ * process. An arbitrary number of references can be made for each IO.
+ */
+PgAioWaitRef iow;
+
+pgaio_io_get_wref(ioh, &iow);
+
+/*
+ * Arrange for shared buffer completion callbacks to be called upon completion
+ * of the IO. This callback will update the buffer descriptors associated with
+ * the AioHandle, which e.g. allows other backends to access the buffer.
+ *
+ * Multiple completion callbacks can be registered for each handle.
+ */
+pgaio_io_register_callbacks(ioh, PGAIO_HCB_SHARED_BUFFER_READV);
+
+/*
+ * The completion callback needs to know which buffers to update when the IO
+ * completes. As the AIO subsystem does not know about buffers, we have to
+ * associate this information with the AioHandle, for use by the completion
+ * callback registered above.
+ *
+ * In this example we're reading only a single buffer, hence the 1.
+ */
+pgaio_io_set_handle_data_32(ioh, (uint32 *) buffer, 1);
+
+/*
+ * Pass the AIO handle to lower-level function. When operating on the level of
+ * buffers, we don't know how exactly the IO is performed, that is the
+ * responsibility of the storage manager implementation.
+ *
+ * E.g. md.c needs to translate block numbers into offsets in segments.
+ *
+ * Once the IO handle has been handed off to smgstartreadv(), it may not
+ * further be used, as the IO may immediately get executed below
+ * smgrstartreadv() and the handle reused for another IO.
+ */
+smgrstartreadv(ioh, operation->smgr, forknum, blkno,
+ BufferGetBlock(buffer), 1);
+
+/*
+ * As mentioned above, the IO might be initiated within smgrstartreadv(). That
+ * is however not guaranteed, to allow IO submission to be batched.
+ *
+ * Note that one needs to be careful while there may be unsubmitted IOs, as
+ * another backend may need to wait for one of the unsubmitted IOs. If this
+ * backend were to wait for the other backend, we'd have a deadlock. To avoid
+ * that, pending IOs need to be explicitly submitted before this backend
+ * might be blocked by a backend waiting for IO.
+ *
+ * Note that the IO might have immediately been submitted (e.g. due to reaching
+ * a limit on the number of unsubmitted IOs) and even completed during the
+ * smgrstartreadv() above.
+ *
+ * Once submitted, the IO is in-flight and can complete at any time.
+ *
+ * TODO: rename to kick as suggested by Heikki?
+ */
+pgaio_submit_staged();
+
+/*
+ * To benefit from AIO, it is beneficial to perform other work, including
+ * submitting other IOs, before waiting for the IO to complete. Otherwise
+ * we could just have used synchronous, blocking IO.
+ */
+perform_other_work();
+
+/*
+ * We did some other work and now need the IO operation to have completed to
+ * continue.
+ */
+pgaio_wref_wait(&iow);
+
+/*
+ * At this point the IO has completed. We do not yet know whether it succeeded
+ * or failed, however. The buffer's state has been updated, which allows other
+ * backends to use the buffer (if the IO succeeded), or retry the IO (if it
+ * failed).
+ *
+ * Note that in case the IO has failed, a LOG message may have been emitted,
+ * but no ERROR has been raised. This is crucial, as another backend waiting
+ * for this IO should not see an ERROR.
+ *
+ * To check whether the operation succeeded, and to raise an ERROR, or if more
+ * appropriate LOG, the PgAioReturn we passed to pgaio_io_acquire() is used.
+ */
+if (ioret.result.status == ARS_ERROR)
+ pgaio_result_report(aio_ret.result, &aio_ret.target_data, ERROR);
+
+/*
+ * Besides having succeeded completely, the IO could also have partially
+ * completed. If we e.g. tried to read many blocks at once, the read might have
+ * only succeeded for the first few blocks.
+ *
+ * If the IO partially succeeded and this backend needs all blocks to have
+ * completed, this backend needs to reissue the IO for the remaining buffers.
+ * The AIO subsystem cannot handle this retry transparently.
+ *
+ * As this example is already long, and we only read a single block, we'll just
+ * error out if there's a partial read.
+ */
+if (ioret.result.status == ARS_PARTIAL)
+ pgaio_result_report(aio_ret.result, &aio_ret.target_data, ERROR);
+
+/*
+ * The IO succeeded, so we can use the buffer now.
+ */
+```
+
+
+## Design Criteria & Motivation
+
+### Deadlock and Starvation Dangers due to AIO
+
+Using AIO in a naive way can easily lead to deadlocks in an environment where
+the source/target of AIO are shared resources, like pages in postgres'
+shared_buffers.
+
+Consider one backend performing readahead on a table, initiating IO for a
+number of buffers ahead of the current "scan position". If that backend then
+performs some operation that blocks, or even just is slow, the IO completion
+for the asynchronously initiated read may not be processed.
+
+This AIO implementation solves this problem by requiring that AIO methods
+either allow AIO completions to be processed by any backend in the system
+(e.g. io_uring), or to guarantee that AIO processing will happen even when the
+issuing backend is blocked (e.g. worker mode, which offloads completion
+processing to the AIO workers).
+
+
+### IO can be started in critical sections
+
+Using AIO for WAL writes can reduce the overhead of WAL logging substantially:
+
+- AIO allows to start WAL writes eagerly, so they complete before needing to
+ wait
+- AIO allows to have multiple WAL flushes in progress at the same time
+- AIO makes it more realistic to use O\_DIRECT + O\_DSYNC, which can reduce
+ the number of roundtrips to storage on some OSs and storage HW (buffered IO
+ and direct IO without O_DSYNC needs to issue a write and after the writes
+ completion a cache cache flush, whereas O\_DIRECT + O\_DSYNC can use a
+ single FUA write).
+
+The need to be able to execute IO in critical sections has substantial design
+implication on the AIO subsystem. Mainly because completing IOs (see prior
+section) needs to be possible within a critical section, even if the
+to-be-completed IO itself was not issued in a critical section. Consider
+e.g. the case of a backend first starting a number of writes from shared
+buffers and then starting to flush the WAL. Because only a limited amount of
+IO can be in-progress at the same time, initiating IO for flushing the WAL may
+require to first complete IO that was started earlier.
+
+
+### State for AIO needs to live in shared memory
+
+Because postgres uses a process model and because AIOs need to be
+complete-able by any backend much of the state of the AIO subsystem needs to
+live in shared memory.
+
+In an `EXEC_BACKEND` build backends executable code and other process local
+state is not necessarily mapped to the same addresses in each process due to
+ASLR. This means that the shared memory cannot contain pointer to callbacks.
+
+
+## Design of the AIO Subsystem
+
+
+### AIO Methods
+
+To achieve portability and performance, multiple methods of performing AIO are
+implemented and others are likely worth adding in the future.
+
+
+#### Synchronous Mode
+
+`io_method=sync` does not actually perform AIO but allows to use the AIO API
+while performing synchronous IO. This can be useful for debugging. The code
+for the synchronous mode is also used as a fallback by e.g. the [worker
+mode](#worker) uses it to execute IO that cannot be executed by workers.
+
+
+#### Worker
+
+`io_method=worker` is available on every platform postgres runs on, and
+implements asynchronous IO - from the view of the issuing process - by
+dispatching the IO to one of several worker processes performing the IO in a
+synchronous manner.
+
+
+#### io_uring
+
+`io_method=io_uring` is available on Linux 5.1+. In contrast to worker mode it
+dispatches all IO from within the process, lowering context switch rate /
+latency.
+
+
+### AIO Handles
+
+The central API piece for postgres' AIO abstraction are AIO handles. To
+execute an IO one first has to acquire an IO handle (`pgaio_io_acquire()`) and
+then "defined", i.e. associate an IO operation with the handle.
+
+Often AIO handles are acquired on a higher level and then passed to a lower
+level to be fully defined. E.g., for IO to/from shared buffers, bufmgr.c
+routines acquire the handle, which is then passed through smgr.c, md.c to be
+finally fully defined in fd.c.
+
+The functions used at the lowest level to define the operation are
+`pgaio_io_prep_*()`.
+
+Because acquisition of an IO handle
+[must always succeed](#io-can-be-started-in-critical-sections)
+and the number of AIO Handles
+[has to be limited](#state-for-aio-needs-to-live-in-shared-memory)
+AIO handles can be reused as soon as they have completed. Obviously code needs
+to be able to react to IO completion. Shared state can be updated using
+[AIO Completion callbacks](#aio-callbacks)
+and the issuing backend can provide a backend local variable to receive the
+result of the IO, as described in
+[AIO Result](#aio-results)
+. An IO can be waited for, by both the issuing and any other backend, using
+[AIO References](#aio-wait-references).
+
+
+Because an AIO Handle is not executable just after calling `pgaio_io_acquire()`
+and because `pgaio_io_acquire()` needs to be able to succeed, only a single AIO
+Handle may be acquired (i.e. returned by `pgaio_io_acquire()`) without causing
+the IO to have been defined (by, potentially indirectly, causing
+`pgaio_io_prep_*()` to have been called). Otherwise a backend could trivially
+self-deadlock by using up all AIO Handles without the ability to wait for some
+of the IOs to complete.
+
+If it turns out that an AIO Handle is not needed, e.g., because the handle was
+acquired before holding a contended lock, it can be released without being
+defined using `pgaio_io_release()`.
+
+
+### AIO Callbacks
+
+Commonly several layers need to react to completion of an IO. E.g. for a read
+md.c needs to check if the IO outright failed or was shorter than needed,
+bufmgr.c needs to verify the page looks valid and bufmgr.c needs to update the
+BufferDesc to update the buffer's state.
+
+The fact that several layers / subsystems need to react to IO completion poses
+a few challenges:
+
+- Upper layers should not need to know details of lower layers. E.g. bufmgr.c
+ should not assume the IO will pass through md.c. Therefore upper levels
+ cannot know what lower layers would consider an error.
+
+- Lower layers should not need to know about upper layers. E.g. smgr APIs are
+ used going through shared buffers but are also used bypassing shared
+ buffers. This means that e.g. md.c is not in a position to validate
+ checksums.
+
+- Having code in the AIO subsystem for every possible combination of layers
+ would lead to a lot of duplication.
+
+The "solution" to this the ability to associate multiple completion callbacks
+with a handle. E.g. bufmgr.c can have a callback to update the BufferDesc
+state and to verify the page and md.c. another callback to check if the IO
+operation was successful.
+
+As [mentioned](#state-for-aio-needs-to-live-in-shared-memory), shared memory
+currently cannot contain function pointers. Because of that completion
+callbacks are not directly identified by function pointers but by IDs
+(`PgAioHandleCallbackID`). A substantial added benefit is that that
+allows callbacks to be identified by much smaller amount of memory (a single
+byte currently).
+
+In addition to completion, AIO callbacks also are called to "prepare" an
+IO. This is, e.g., used to increase buffer reference counts to account for the
+AIO subsystem referencing the buffer, which is required to handle the case
+where the issuing backend errors out and releases its own pins while the IO is
+still ongoing.
+
+As [explained earlier](#io-can-be-started-in-critical-sections) IO completions
+need to be safe to execute in critical sections. To allow the backend that
+issued the IO to error out in case of failure [AIO Result](#aio-results) can
+be used.
+
+
+### AIO Targets
+
+In addition to the completion callbacks describe above, each AIO Handle has
+exactly one "target". Each target has some space inside an AIO Handle with
+information specific to the target and can provide callbacks to allow to
+reopen the underlying file (required for worker mode) and to describe the IO
+operation (used for debug logging and error messages).
+
+I.e., if two different uses of AIO can describe the identity of the file being
+operated on the same way, it likely makes sense to use the same
+target. E.g. different smgr implementations can describe IO with
+RelFileLocator, ForkNumber and BlockNumber and can thus share a target. In
+contrast, IO for a WAL file would be described with TimeLineID and XLogRecPtr
+and it would not make sense to use the same target for smgr and WAL.
+
+
+### AIO Wait References
+
+As [described above](#aio-handles), AIO Handles can be reused immediately
+after completion and therefore cannot be used to wait for completion of the
+IO. Waiting is enabled using AIO wait references, which do not just identify
+an AIO Handle but also include the handles "generation".
+
+A reference to an AIO Handle can be acquired using `pgaio_io_get_wref()` and
+then waited upon using `pgaio_wref_wait()`.
+
+
+### AIO Results
+
+As AIO completion callbacks
+[are executed in critical sections](#io-can-be-started-in-critical-sections)
+and [may be executed by any backend](#deadlock-and-starvation-dangers-due-to-aio)
+completion callbacks cannot be used to, e.g., make the query that triggered an
+IO ERROR out.
+
+To allow to react to failing IOs the issuing backend can pass a pointer to a
+`PgAioReturn` in backend local memory. Before an AIO Handle is reused the
+`PgAioReturn` is filled with information about the IO. This includes
+information about whether the IO was successful (as a value of
+`PgAioResultStatus`) and enough information to raise an error in case of a
+failure (via `pgaio_result_report()`, with the error details encoded in
+`PgAioResult`).
+
+XXX: "return" vs "result" vs "result status" seems quite confusing. The naming
+should be improved.
+
+
+### AIO Errors
+
+It would be very convenient to have shared completion callbacks encode the
+details of errors as an `ErrorData` that could be raised at a later
+time. Unfortunately doing so would require allocating memory. While elog.c can
+guarantee (well, kinda) that logging a message will not run out of memory,
+that only works because a very limited number of messages are in the process
+of being logged. With AIO a large number of concurrently issued AIOs might
+fail.
+
+To avoid the need for preallocating a potentially large amount of memory (in
+shared memory no less!), completion callbacks instead have to encode errors in
+a more compact format that can be converted into an error message.
+
+
+## Helpers
+
+Using the low-level AIO API introduces too much complexity to do so all over
+the tree. Most uses of AIO should be done via reusable, higher-level,
+helpers.
+
+
+### Read Stream
+
+A common and very beneficial use of AIO are reads where a substantial number
+of to-be-read locations are known ahead of time. E.g., for a sequential scan
+the set of blocks that need to be read can be determined solely by knowing the
+current position and checking the buffer mapping table.
+
+The [Read Stream](../../../include/storage/read_stream.h) interface makes it
+comparatively easy to use AIO for such use cases.
diff --git a/src/backend/storage/aio/aio.c b/src/backend/storage/aio/aio.c
index c1dd073e37f..b3b4e74c3ce 100644
--- a/src/backend/storage/aio/aio.c
+++ b/src/backend/storage/aio/aio.c
@@ -24,6 +24,8 @@
*
* - read_stream.c - helper for reading buffered relation data
*
+ * - README.md - higher-level overview over AIO
+ *
*
* Portions Copyright (c) 1996-2025, PostgreSQL Global Development Group
* Portions Copyright (c) 1994, Regents of the University of California
--
2.48.1.76.g4e746b1a31.dirty
[text/x-diff] v2.3-0017-aio-Implement-smgr-md-fd-aio-methods.patch (28.5K, ../../p5flybfcjrzgpgwcnzkxsp3xdw3czfoafkvks6g7en6ewmvymh@wrz7z2vwwc7c/18-v2.3-0017-aio-Implement-smgr-md-fd-aio-methods.patch)
download | inline diff:
From 6c9493bdbc9164decc460c7ab74aaceea19d67a0 Mon Sep 17 00:00:00 2001
From: Andres Freund <andres@anarazel.de>
Date: Wed, 22 Jan 2025 16:06:51 -0500
Subject: [PATCH v2.3 17/30] aio: Implement smgr/md/fd aio methods
---
src/include/storage/aio.h | 6 +-
src/include/storage/aio_types.h | 12 +-
src/include/storage/fd.h | 6 +
src/include/storage/md.h | 12 +
src/include/storage/smgr.h | 22 ++
src/backend/storage/aio/aio_callback.c | 4 +
src/backend/storage/aio/aio_target.c | 2 +
src/backend/storage/file/fd.c | 68 +++++
src/backend/storage/smgr/md.c | 360 +++++++++++++++++++++++++
src/backend/storage/smgr/smgr.c | 126 +++++++++
10 files changed, 614 insertions(+), 4 deletions(-)
diff --git a/src/include/storage/aio.h b/src/include/storage/aio.h
index 8c1b9a1b496..a948eaeefa7 100644
--- a/src/include/storage/aio.h
+++ b/src/include/storage/aio.h
@@ -108,9 +108,10 @@ typedef enum PgAioTargetID
{
/* intentionally the zero value, to help catch zeroed memory etc */
PGAIO_TID_INVALID = 0,
+ PGAIO_TID_SMGR,
} PgAioTargetID;
-#define PGAIO_TID_COUNT (PGAIO_TID_INVALID + 1)
+#define PGAIO_TID_COUNT (PGAIO_TID_SMGR + 1)
/*
@@ -174,6 +175,9 @@ typedef struct PgAioTargetInfo
typedef enum PgAioHandleCallbackID
{
PGAIO_HCB_INVALID,
+
+ PGAIO_HCB_MD_READV,
+ PGAIO_HCB_MD_WRITEV,
} PgAioHandleCallbackID;
diff --git a/src/include/storage/aio_types.h b/src/include/storage/aio_types.h
index d2617139a25..762fce3f075 100644
--- a/src/include/storage/aio_types.h
+++ b/src/include/storage/aio_types.h
@@ -58,11 +58,17 @@ typedef struct PgAioWaitRef
*/
typedef union PgAioTargetData
{
- /* just as an example placeholder for later */
struct
{
- uint32 queue_id;
- } wal;
+ RelFileLocator rlocator; /* physical relation identifier */
+ BlockNumber blockNum; /* blknum relative to begin of reln */
+ BlockNumber nblocks;
+ ForkNumber forkNum:8; /* don't waste 4 byte for four values */
+ bool is_temp:1; /* proc can be inferred by owning AIO */
+ bool release_lock:1;
+ bool skip_fsync:1;
+ uint8 mode;
+ } smgr;
} PgAioTargetData;
diff --git a/src/include/storage/fd.h b/src/include/storage/fd.h
index e3067ab6597..e2fd896646e 100644
--- a/src/include/storage/fd.h
+++ b/src/include/storage/fd.h
@@ -101,6 +101,8 @@ extern PGDLLIMPORT int max_safe_fds;
* prototypes for functions in fd.c
*/
+struct PgAioHandle;
+
/* Operations on virtual Files --- equivalent to Unix kernel file ops */
extern File PathNameOpenFile(const char *fileName, int fileFlags);
extern File PathNameOpenFilePerm(const char *fileName, int fileFlags, mode_t fileMode);
@@ -109,6 +111,10 @@ extern void FileClose(File file);
extern int FilePrefetch(File file, off_t offset, off_t amount, uint32 wait_event_info);
extern ssize_t FileReadV(File file, const struct iovec *iov, int iovcnt, off_t offset, uint32 wait_event_info);
extern ssize_t FileWriteV(File file, const struct iovec *iov, int iovcnt, off_t offset, uint32 wait_event_info);
+extern ssize_t FileReadV(File file, const struct iovec *iov, int iovcnt, off_t offset, uint32 wait_event_info);
+extern int FileStartReadV(struct PgAioHandle *ioh, File file, int iovcnt, off_t offset, uint32 wait_event_info);
+extern ssize_t FileWriteV(File file, const struct iovec *iov, int iovcnt, off_t offset, uint32 wait_event_info);
+extern int FileStartWriteV(struct PgAioHandle *ioh, File file, int iovcnt, off_t offset, uint32 wait_event_info);
extern int FileSync(File file, uint32 wait_event_info);
extern int FileZero(File file, off_t offset, off_t amount, uint32 wait_event_info);
extern int FileFallocate(File file, off_t offset, off_t amount, uint32 wait_event_info);
diff --git a/src/include/storage/md.h b/src/include/storage/md.h
index 05bf537066e..7b28c3d482c 100644
--- a/src/include/storage/md.h
+++ b/src/include/storage/md.h
@@ -19,6 +19,10 @@
#include "storage/smgr.h"
#include "storage/sync.h"
+struct PgAioHandleCallbacks;
+extern const struct PgAioHandleCallbacks aio_md_readv_cb;
+extern const struct PgAioHandleCallbacks aio_md_writev_cb;
+
/* md storage manager functionality */
extern void mdinit(void);
extern void mdopen(SMgrRelation reln);
@@ -36,9 +40,16 @@ extern uint32 mdmaxcombine(SMgrRelation reln, ForkNumber forknum,
BlockNumber blocknum);
extern void mdreadv(SMgrRelation reln, ForkNumber forknum, BlockNumber blocknum,
void **buffers, BlockNumber nblocks);
+extern void mdstartreadv(struct PgAioHandle *ioh,
+ SMgrRelation reln, ForkNumber forknum, BlockNumber blocknum,
+ void **buffers, BlockNumber nblocks);
extern void mdwritev(SMgrRelation reln, ForkNumber forknum,
BlockNumber blocknum,
const void **buffers, BlockNumber nblocks, bool skipFsync);
+extern void mdstartwritev(struct PgAioHandle *ioh,
+ SMgrRelation reln, ForkNumber forknum,
+ BlockNumber blocknum,
+ const void **buffers, BlockNumber nblocks, bool skipFsync);
extern void mdwriteback(SMgrRelation reln, ForkNumber forknum,
BlockNumber blocknum, BlockNumber nblocks);
extern BlockNumber mdnblocks(SMgrRelation reln, ForkNumber forknum);
@@ -46,6 +57,7 @@ extern void mdtruncate(SMgrRelation reln, ForkNumber forknum,
BlockNumber old_blocks, BlockNumber nblocks);
extern void mdimmedsync(SMgrRelation reln, ForkNumber forknum);
extern void mdregistersync(SMgrRelation reln, ForkNumber forknum);
+extern int mdfd(SMgrRelation reln, ForkNumber forknum, BlockNumber blocknum, uint32 *off);
extern void ForgetDatabaseSyncRequests(Oid dbid);
extern void DropRelationFiles(RelFileLocator *delrels, int ndelrels, bool isRedo);
diff --git a/src/include/storage/smgr.h b/src/include/storage/smgr.h
index 4016b206ad6..86fa07b110f 100644
--- a/src/include/storage/smgr.h
+++ b/src/include/storage/smgr.h
@@ -73,6 +73,11 @@ typedef SMgrRelationData *SMgrRelation;
#define SmgrIsTemp(smgr) \
RelFileLocatorBackendIsTemp((smgr)->smgr_rlocator)
+struct PgAioHandle;
+struct PgAioTargetInfo;
+
+extern const struct PgAioTargetInfo aio_smgr_target_info;
+
extern void smgrinit(void);
extern SMgrRelation smgropen(RelFileLocator rlocator, ProcNumber backend);
extern bool smgrexists(SMgrRelation reln, ForkNumber forknum);
@@ -97,10 +102,19 @@ extern uint32 smgrmaxcombine(SMgrRelation reln, ForkNumber forknum,
extern void smgrreadv(SMgrRelation reln, ForkNumber forknum,
BlockNumber blocknum,
void **buffers, BlockNumber nblocks);
+extern void smgrstartreadv(struct PgAioHandle *ioh,
+ SMgrRelation reln, ForkNumber forknum,
+ BlockNumber blocknum,
+ void **buffers, BlockNumber nblocks);
extern void smgrwritev(SMgrRelation reln, ForkNumber forknum,
BlockNumber blocknum,
const void **buffers, BlockNumber nblocks,
bool skipFsync);
+extern void smgrstartwritev(struct PgAioHandle *ioh,
+ SMgrRelation reln, ForkNumber forknum,
+ BlockNumber blocknum,
+ const void **buffers, BlockNumber nblocks,
+ bool skipFsync);
extern void smgrwriteback(SMgrRelation reln, ForkNumber forknum,
BlockNumber blocknum, BlockNumber nblocks);
extern BlockNumber smgrnblocks(SMgrRelation reln, ForkNumber forknum);
@@ -110,6 +124,7 @@ extern void smgrtruncate(SMgrRelation reln, ForkNumber *forknum, int nforks,
BlockNumber *nblocks);
extern void smgrimmedsync(SMgrRelation reln, ForkNumber forknum);
extern void smgrregistersync(SMgrRelation reln, ForkNumber forknum);
+extern int smgrfd(SMgrRelation reln, ForkNumber forknum, BlockNumber blocknum, uint32 *off);
extern void AtEOXact_SMgr(void);
extern bool ProcessBarrierSmgrRelease(void);
@@ -127,4 +142,11 @@ smgrwrite(SMgrRelation reln, ForkNumber forknum, BlockNumber blocknum,
smgrwritev(reln, forknum, blocknum, &buffer, 1, skipFsync);
}
+extern void pgaio_io_set_target_smgr(struct PgAioHandle *ioh,
+ SMgrRelationData *smgr,
+ ForkNumber forknum,
+ BlockNumber blocknum,
+ int nblocks,
+ bool skip_fsync);
+
#endif /* SMGR_H */
diff --git a/src/backend/storage/aio/aio_callback.c b/src/backend/storage/aio/aio_callback.c
index 93f71690169..7fd42880535 100644
--- a/src/backend/storage/aio/aio_callback.c
+++ b/src/backend/storage/aio/aio_callback.c
@@ -18,6 +18,7 @@
#include "miscadmin.h"
#include "storage/aio.h"
#include "storage/aio_internal.h"
+#include "storage/md.h"
#include "utils/memutils.h"
@@ -38,6 +39,9 @@ typedef struct PgAioHandleCallbacksEntry
static const PgAioHandleCallbacksEntry aio_handle_cbs[] = {
#define CALLBACK_ENTRY(id, callback) [id] = {.cb = &callback, .name = #callback}
CALLBACK_ENTRY(PGAIO_HCB_INVALID, aio_invalid_cb),
+
+ CALLBACK_ENTRY(PGAIO_HCB_MD_READV, aio_md_readv_cb),
+ CALLBACK_ENTRY(PGAIO_HCB_MD_WRITEV, aio_md_writev_cb),
#undef CALLBACK_ENTRY
};
diff --git a/src/backend/storage/aio/aio_target.c b/src/backend/storage/aio/aio_target.c
index 15428968e58..a43edd89890 100644
--- a/src/backend/storage/aio/aio_target.c
+++ b/src/backend/storage/aio/aio_target.c
@@ -18,6 +18,7 @@
#include "storage/aio.h"
#include "storage/aio_internal.h"
+#include "storage/smgr.h"
/*
@@ -31,6 +32,7 @@ static const PgAioTargetInfo *pgaio_target_info[] = {
[PGAIO_TID_INVALID] = &(PgAioTargetInfo) {
.name = "invalid",
},
+ [PGAIO_TID_SMGR] = &aio_smgr_target_info,
};
diff --git a/src/backend/storage/file/fd.c b/src/backend/storage/file/fd.c
index 843d1021cf9..89f2dc29555 100644
--- a/src/backend/storage/file/fd.c
+++ b/src/backend/storage/file/fd.c
@@ -94,6 +94,7 @@
#include "miscadmin.h"
#include "pgstat.h"
#include "postmaster/startup.h"
+#include "storage/aio.h"
#include "storage/fd.h"
#include "storage/ipc.h"
#include "utils/guc.h"
@@ -1294,6 +1295,8 @@ LruDelete(File file)
vfdP = &VfdCache[file];
+ pgaio_closing_fd(vfdP->fd);
+
/*
* Close the file. We aren't expecting this to fail; if it does, better
* to leak the FD than to mess up our internal state.
@@ -1987,6 +1990,8 @@ FileClose(File file)
if (!FileIsNotOpen(file))
{
+ pgaio_closing_fd(vfdP->fd);
+
/* close the file */
if (close(vfdP->fd) != 0)
{
@@ -2210,6 +2215,32 @@ retry:
return returnCode;
}
+int
+FileStartReadV(struct PgAioHandle *ioh, File file,
+ int iovcnt, off_t offset,
+ uint32 wait_event_info)
+{
+ int returnCode;
+ Vfd *vfdP;
+
+ Assert(FileIsValid(file));
+
+ DO_DB(elog(LOG, "FileStartReadV: %d (%s) " INT64_FORMAT " %d",
+ file, VfdCache[file].fileName,
+ (int64) offset,
+ iovcnt));
+
+ returnCode = FileAccess(file);
+ if (returnCode < 0)
+ return returnCode;
+
+ vfdP = &VfdCache[file];
+
+ pgaio_io_prep_readv(ioh, vfdP->fd, iovcnt, offset);
+
+ return 0;
+}
+
ssize_t
FileWriteV(File file, const struct iovec *iov, int iovcnt, off_t offset,
uint32 wait_event_info)
@@ -2315,6 +2346,34 @@ retry:
return returnCode;
}
+int
+FileStartWriteV(struct PgAioHandle *ioh, File file,
+ int iovcnt, off_t offset,
+ uint32 wait_event_info)
+{
+ int returnCode;
+ Vfd *vfdP;
+
+ Assert(FileIsValid(file));
+
+ DO_DB(elog(LOG, "FileStartWriteV: %d (%s) " INT64_FORMAT " %d",
+ file, VfdCache[file].fileName,
+ (int64) offset,
+ iovcnt));
+
+ returnCode = FileAccess(file);
+ if (returnCode < 0)
+ return returnCode;
+
+ vfdP = &VfdCache[file];
+
+ /* FIXME: think about / reimplement temp_file_limit */
+
+ pgaio_io_prep_writev(ioh, vfdP->fd, iovcnt, offset);
+
+ return 0;
+}
+
int
FileSync(File file, uint32 wait_event_info)
{
@@ -2498,6 +2557,12 @@ FilePathName(File file)
int
FileGetRawDesc(File file)
{
+ int returnCode;
+
+ returnCode = FileAccess(file);
+ if (returnCode < 0)
+ return returnCode;
+
Assert(FileIsValid(file));
return VfdCache[file].fd;
}
@@ -2778,6 +2843,7 @@ FreeDesc(AllocateDesc *desc)
result = closedir(desc->desc.dir);
break;
case AllocateDescRawFD:
+ pgaio_closing_fd(desc->desc.fd);
result = close(desc->desc.fd);
break;
default:
@@ -2846,6 +2912,8 @@ CloseTransientFile(int fd)
/* Only get here if someone passes us a file not in allocatedDescs */
elog(WARNING, "fd passed to CloseTransientFile was not obtained from OpenTransientFile");
+ pgaio_closing_fd(fd);
+
return close(fd);
}
diff --git a/src/backend/storage/smgr/md.c b/src/backend/storage/smgr/md.c
index 7bf0b45e2c3..e204b7abba6 100644
--- a/src/backend/storage/smgr/md.c
+++ b/src/backend/storage/smgr/md.c
@@ -31,6 +31,7 @@
#include "miscadmin.h"
#include "pg_trace.h"
#include "pgstat.h"
+#include "storage/aio.h"
#include "storage/bufmgr.h"
#include "storage/fd.h"
#include "storage/md.h"
@@ -132,6 +133,22 @@ static MdfdVec *_mdfd_getseg(SMgrRelation reln, ForkNumber forknum,
static BlockNumber _mdnblocks(SMgrRelation reln, ForkNumber forknum,
MdfdVec *seg);
+static PgAioResult md_readv_complete(PgAioHandle *ioh, PgAioResult prior_result);
+static void md_readv_report(PgAioResult result, const PgAioTargetData *target_data, int elevel);
+static PgAioResult md_writev_complete(PgAioHandle *ioh, PgAioResult prior_result);
+static void md_writev_report(PgAioResult result, const PgAioTargetData *target_data, int elevel);
+
+const struct PgAioHandleCallbacks aio_md_readv_cb = {
+ .complete_shared = md_readv_complete,
+ .report = md_readv_report,
+};
+
+const struct PgAioHandleCallbacks aio_md_writev_cb = {
+ .complete_shared = md_writev_complete,
+ .report = md_writev_report,
+};
+
+
static inline int
_mdfd_open_flags(void)
{
@@ -927,6 +944,53 @@ mdreadv(SMgrRelation reln, ForkNumber forknum, BlockNumber blocknum,
}
}
+void
+mdstartreadv(PgAioHandle *ioh,
+ SMgrRelation reln, ForkNumber forknum, BlockNumber blocknum,
+ void **buffers, BlockNumber nblocks)
+{
+ off_t seekpos;
+ MdfdVec *v;
+ BlockNumber nblocks_this_segment;
+ struct iovec *iov;
+ int iovcnt;
+
+ v = _mdfd_getseg(reln, forknum, blocknum, false,
+ EXTENSION_FAIL | EXTENSION_CREATE_RECOVERY);
+
+ seekpos = (off_t) BLCKSZ * (blocknum % ((BlockNumber) RELSEG_SIZE));
+
+ Assert(seekpos < (off_t) BLCKSZ * RELSEG_SIZE);
+
+ nblocks_this_segment =
+ Min(nblocks,
+ RELSEG_SIZE - (blocknum % ((BlockNumber) RELSEG_SIZE)));
+
+ if (nblocks_this_segment != nblocks)
+ elog(ERROR, "read crossing segment boundary");
+
+ iovcnt = pgaio_io_get_iovec(ioh, &iov);
+
+ Assert(nblocks <= iovcnt);
+
+ iovcnt = buffers_to_iovec(iov, buffers, nblocks_this_segment);
+
+ Assert(iovcnt <= nblocks_this_segment);
+
+ if (!(io_direct_flags & IO_DIRECT_DATA))
+ pgaio_io_set_flag(ioh, PGAIO_HF_BUFFERED);
+
+ pgaio_io_set_target_smgr(ioh,
+ reln,
+ forknum,
+ blocknum,
+ nblocks,
+ false);
+ pgaio_io_register_callbacks(ioh, PGAIO_HCB_MD_READV);
+
+ FileStartReadV(ioh, v->mdfd_vfd, iovcnt, seekpos, WAIT_EVENT_DATA_FILE_READ);
+}
+
/*
* mdwritev() -- Write the supplied blocks at the appropriate location.
*
@@ -1032,6 +1096,53 @@ mdwritev(SMgrRelation reln, ForkNumber forknum, BlockNumber blocknum,
}
}
+void
+mdstartwritev(PgAioHandle *ioh,
+ SMgrRelation reln, ForkNumber forknum, BlockNumber blocknum,
+ const void **buffers, BlockNumber nblocks, bool skipFsync)
+{
+ off_t seekpos;
+ MdfdVec *v;
+ BlockNumber nblocks_this_segment;
+ struct iovec *iov;
+ int iovcnt;
+
+ v = _mdfd_getseg(reln, forknum, blocknum, false,
+ EXTENSION_FAIL | EXTENSION_CREATE_RECOVERY);
+
+ seekpos = (off_t) BLCKSZ * (blocknum % ((BlockNumber) RELSEG_SIZE));
+
+ Assert(seekpos < (off_t) BLCKSZ * RELSEG_SIZE);
+
+ nblocks_this_segment =
+ Min(nblocks,
+ RELSEG_SIZE - (blocknum % ((BlockNumber) RELSEG_SIZE)));
+
+ if (nblocks_this_segment != nblocks)
+ elog(ERROR, "write crossing segment boundary");
+
+ iovcnt = pgaio_io_get_iovec(ioh, &iov);
+
+ Assert(nblocks <= iovcnt);
+
+ iovcnt = buffers_to_iovec(iov, unconstify(void **, buffers), nblocks_this_segment);
+
+ Assert(iovcnt <= nblocks_this_segment);
+
+ if (!(io_direct_flags & IO_DIRECT_DATA))
+ pgaio_io_set_flag(ioh, PGAIO_HF_BUFFERED);
+
+ pgaio_io_set_target_smgr(ioh,
+ reln,
+ forknum,
+ blocknum,
+ nblocks,
+ skipFsync);
+ pgaio_io_register_callbacks(ioh, PGAIO_HCB_MD_WRITEV);
+
+ FileStartWriteV(ioh, v->mdfd_vfd, iovcnt, seekpos, WAIT_EVENT_DATA_FILE_WRITE);
+}
+
/*
* mdwriteback() -- Tell the kernel to write pages back to storage.
@@ -1355,6 +1466,21 @@ mdimmedsync(SMgrRelation reln, ForkNumber forknum)
}
}
+int
+mdfd(SMgrRelation reln, ForkNumber forknum, BlockNumber blocknum, uint32 *off)
+{
+ MdfdVec *v = mdopenfork(reln, forknum, EXTENSION_FAIL);
+
+ v = _mdfd_getseg(reln, forknum, blocknum, false,
+ EXTENSION_FAIL);
+
+ *off = (off_t) BLCKSZ * (blocknum % ((BlockNumber) RELSEG_SIZE));
+
+ Assert(*off < (off_t) BLCKSZ * RELSEG_SIZE);
+
+ return FileGetRawDesc(v->mdfd_vfd);
+}
+
/*
* register_dirty_segment() -- Mark a relation segment as needing fsync
*
@@ -1405,6 +1531,35 @@ register_dirty_segment(SMgrRelation reln, ForkNumber forknum, MdfdVec *seg)
}
}
+/*
+ * Like register_dirty_segment(), except for use by AIO. In the completion
+ * callback we don't have access to the MdfdVec (the completion callback might
+ * be executed in a different backend than the issuing backend), therefore we
+ * have to implement this slightly differently.
+ */
+static void
+register_dirty_segment_aio(RelFileLocator locator, ForkNumber forknum, uint64 segno)
+{
+ FileTag tag;
+
+ INIT_MD_FILETAG(tag, locator, forknum, segno);
+
+ if (!RegisterSyncRequest(&tag, SYNC_REQUEST, false /* retryOnError */ ))
+ {
+ char path[MAXPGPATH];
+
+ ereport(DEBUG1,
+ (errmsg_internal("could not forward fsync request because request queue is full")));
+
+ /* reuse mdsyncfiletag() to avoid duplicating code */
+ if (mdsyncfiletag(&tag, path))
+ ereport(data_sync_elevel(ERROR),
+ (errcode_for_file_access(),
+ errmsg("could not fsync file \"%s\": %m",
+ path)));
+ }
+}
+
/*
* register_unlink_segment() -- Schedule a file to be deleted after next checkpoint
*/
@@ -1838,3 +1993,208 @@ mdfiletagmatches(const FileTag *ftag, const FileTag *candidate)
*/
return ftag->rlocator.dbOid == candidate->rlocator.dbOid;
}
+
+/*
+ * AIO completion callback for mdstartreadv().
+ */
+static PgAioResult
+md_readv_complete(PgAioHandle *ioh, PgAioResult prior_result)
+{
+ PgAioTargetData *sd = pgaio_io_get_target_data(ioh);
+ PgAioResult result = prior_result;
+
+ if (prior_result.result < 0)
+ {
+ result.status = ARS_ERROR;
+ result.id = PGAIO_HCB_MD_READV;
+ /* For "hard" errors, track the error number in error_data */
+ result.error_data = -prior_result.result;
+ result.result = 0;
+
+ md_readv_report(result, sd, LOG);
+
+ return result;
+ }
+
+ result.result /= BLCKSZ;
+
+ if (result.result == 0)
+ {
+ /* consider 0 blocks read a failure */
+ result.status = ARS_ERROR;
+ result.id = PGAIO_HCB_MD_READV;
+ result.error_data = 0;
+
+ md_readv_report(result, sd, LOG);
+ }
+
+ if (result.status != ARS_ERROR &&
+ result.result < sd->smgr.nblocks)
+ {
+ /* partial reads should be retried at upper level */
+ result.status = ARS_PARTIAL;
+ result.id = PGAIO_HCB_MD_READV;
+ }
+
+ /* AFIXME: post-read portion of mdreadv() */
+
+ return result;
+}
+
+/*
+ * AIO error reporting callback for mdstartreadv().
+ */
+static void
+md_readv_report(PgAioResult result, const PgAioTargetData *sd, int elevel)
+{
+ MemoryContext oldContext = CurrentMemoryContext;
+ char *path;
+
+ /* AFIXME: */
+ oldContext = MemoryContextSwitchTo(ErrorContext);
+
+ path = relpathbackend(sd->smgr.rlocator,
+ sd->smgr.is_temp ? MyProcNumber : INVALID_PROC_NUMBER,
+ sd->smgr.forkNum);
+
+ if (result.error_data != 0)
+ {
+ errno = result.error_data; /* for errcode_for_file_access() */
+
+ ereport(elevel,
+ errcode_for_file_access(),
+ errmsg("could not read blocks %u..%u in file \"%s\": %m",
+ sd->smgr.blockNum,
+ sd->smgr.blockNum + sd->smgr.nblocks,
+ path
+ )
+ );
+ }
+ else
+ {
+ /*
+ * NB: This will typically only be output in debug messages, while
+ * retrying a partial IO.
+ */
+ ereport(elevel,
+ errcode(ERRCODE_DATA_CORRUPTED),
+ errmsg("could not read blocks %u..%u in file \"%s\": read only %zu of %zu bytes",
+ sd->smgr.blockNum,
+ sd->smgr.blockNum + sd->smgr.nblocks - 1,
+ path,
+ result.result * (size_t) BLCKSZ,
+ sd->smgr.nblocks * (size_t) BLCKSZ
+ )
+ );
+ }
+
+ pfree(path);
+ MemoryContextSwitchTo(oldContext);
+}
+
+/*
+ * AIO completion callback for mdstartwritev().
+ */
+static PgAioResult
+md_writev_complete(PgAioHandle *ioh, PgAioResult prior_result)
+{
+ PgAioTargetData *sd = pgaio_io_get_target_data(ioh);
+ PgAioResult result = prior_result;
+
+ if (prior_result.result < 0)
+ {
+ result.status = ARS_ERROR;
+ result.id = PGAIO_HCB_MD_WRITEV;
+ /* For "hard" errors, track the error number in error_data */
+ result.error_data = -prior_result.result;
+ result.result = 0;
+
+ md_writev_report(result, sd, LOG);
+
+ return result;
+ }
+
+ result.result /= BLCKSZ;
+
+ if (result.result == 0)
+ {
+ /* consider 0 blocks written a failure */
+ result.status = ARS_ERROR;
+ result.id = PGAIO_HCB_MD_WRITEV;
+ result.error_data = 0;
+
+ md_writev_report(result, sd, LOG);
+ }
+
+ if (result.status != ARS_ERROR &&
+ result.result < sd->smgr.nblocks)
+ {
+ /* partial writes should be retried at upper level */
+ result.status = ARS_PARTIAL;
+ result.id = PGAIO_HCB_MD_WRITEV;
+ }
+
+ if (prior_result.status == ARS_ERROR)
+ {
+ /* AFIXME: complain */
+ return prior_result;
+ }
+
+ prior_result.result /= BLCKSZ;
+
+ if (!sd->smgr.skip_fsync)
+ register_dirty_segment_aio(sd->smgr.rlocator, sd->smgr.forkNum,
+ sd->smgr.blockNum / ((BlockNumber) RELSEG_SIZE));
+
+ return prior_result;
+}
+
+/*
+ * AIO error reporting callback for mdstartwritev().
+ */
+static void
+md_writev_report(PgAioResult result, const PgAioTargetData *sd, int elevel)
+{
+ MemoryContext oldContext = CurrentMemoryContext;
+ char *path;
+
+ /* AFIXME: */
+ oldContext = MemoryContextSwitchTo(ErrorContext);
+
+ path = relpathbackend(sd->smgr.rlocator,
+ sd->smgr.is_temp ? MyProcNumber : INVALID_PROC_NUMBER,
+ sd->smgr.forkNum);
+
+ if (result.error_data != 0)
+ {
+ errno = result.error_data; /* for errcode_for_file_access() */
+
+ ereport(elevel,
+ errcode_for_file_access(),
+ errmsg("could not write blocks %u..%u in file \"%s\": %m",
+ sd->smgr.blockNum,
+ sd->smgr.blockNum + sd->smgr.nblocks,
+ path)
+ );
+ }
+ else
+ {
+ /*
+ * NB: This will typically only be output in debug messages, while
+ * retrying a partial IO.
+ */
+ ereport(elevel,
+ errcode(ERRCODE_DATA_CORRUPTED),
+ errmsg("could not write blocks %u..%u in file \"%s\": wrote only %zu of %zu bytes",
+ sd->smgr.blockNum,
+ sd->smgr.blockNum + sd->smgr.nblocks - 1,
+ path,
+ result.result * (size_t) BLCKSZ,
+ sd->smgr.nblocks * (size_t) BLCKSZ
+ )
+ );
+ }
+
+ pfree(path);
+ MemoryContextSwitchTo(oldContext);
+}
diff --git a/src/backend/storage/smgr/smgr.c b/src/backend/storage/smgr/smgr.c
index ebe35c04de5..fb231e6ad48 100644
--- a/src/backend/storage/smgr/smgr.c
+++ b/src/backend/storage/smgr/smgr.c
@@ -53,6 +53,7 @@
#include "access/xlogutils.h"
#include "lib/ilist.h"
+#include "storage/aio.h"
#include "storage/bufmgr.h"
#include "storage/ipc.h"
#include "storage/md.h"
@@ -93,10 +94,19 @@ typedef struct f_smgr
void (*smgr_readv) (SMgrRelation reln, ForkNumber forknum,
BlockNumber blocknum,
void **buffers, BlockNumber nblocks);
+ void (*smgr_startreadv) (struct PgAioHandle *ioh,
+ SMgrRelation reln, ForkNumber forknum,
+ BlockNumber blocknum,
+ void **buffers, BlockNumber nblocks);
void (*smgr_writev) (SMgrRelation reln, ForkNumber forknum,
BlockNumber blocknum,
const void **buffers, BlockNumber nblocks,
bool skipFsync);
+ void (*smgr_startwritev) (struct PgAioHandle *ioh,
+ SMgrRelation reln, ForkNumber forknum,
+ BlockNumber blocknum,
+ const void **buffers, BlockNumber nblocks,
+ bool skipFsync);
void (*smgr_writeback) (SMgrRelation reln, ForkNumber forknum,
BlockNumber blocknum, BlockNumber nblocks);
BlockNumber (*smgr_nblocks) (SMgrRelation reln, ForkNumber forknum);
@@ -104,6 +114,7 @@ typedef struct f_smgr
BlockNumber old_blocks, BlockNumber nblocks);
void (*smgr_immedsync) (SMgrRelation reln, ForkNumber forknum);
void (*smgr_registersync) (SMgrRelation reln, ForkNumber forknum);
+ int (*smgr_fd) (SMgrRelation reln, ForkNumber forknum, BlockNumber blocknum, uint32 *off);
} f_smgr;
static const f_smgr smgrsw[] = {
@@ -121,12 +132,15 @@ static const f_smgr smgrsw[] = {
.smgr_prefetch = mdprefetch,
.smgr_maxcombine = mdmaxcombine,
.smgr_readv = mdreadv,
+ .smgr_startreadv = mdstartreadv,
.smgr_writev = mdwritev,
+ .smgr_startwritev = mdstartwritev,
.smgr_writeback = mdwriteback,
.smgr_nblocks = mdnblocks,
.smgr_truncate = mdtruncate,
.smgr_immedsync = mdimmedsync,
.smgr_registersync = mdregistersync,
+ .smgr_fd = mdfd,
}
};
@@ -145,6 +159,16 @@ static void smgrshutdown(int code, Datum arg);
static void smgrdestroy(SMgrRelation reln);
+static void smgr_aio_reopen(PgAioHandle *ioh);
+static char *smgr_aio_describe_identity(const PgAioTargetData *sd);
+
+const struct PgAioTargetInfo aio_smgr_target_info = {
+ .name = "smgr",
+ .reopen = smgr_aio_reopen,
+ .describe_identity = smgr_aio_describe_identity,
+};
+
+
/*
* smgrinit(), smgrshutdown() -- Initialize or shut down storage
* managers.
@@ -623,6 +647,19 @@ smgrreadv(SMgrRelation reln, ForkNumber forknum, BlockNumber blocknum,
nblocks);
}
+/*
+ * AFIXME: FILL ME IN
+ */
+void
+smgrstartreadv(struct PgAioHandle *ioh,
+ SMgrRelation reln, ForkNumber forknum, BlockNumber blocknum,
+ void **buffers, BlockNumber nblocks)
+{
+ smgrsw[reln->smgr_which].smgr_startreadv(ioh,
+ reln, forknum, blocknum, buffers,
+ nblocks);
+}
+
/*
* smgrwritev() -- Write the supplied buffers out.
*
@@ -657,6 +694,19 @@ smgrwritev(SMgrRelation reln, ForkNumber forknum, BlockNumber blocknum,
buffers, nblocks, skipFsync);
}
+/*
+ * AFIXME: FILL ME IN
+ */
+void
+smgrstartwritev(struct PgAioHandle *ioh,
+ SMgrRelation reln, ForkNumber forknum, BlockNumber blocknum,
+ const void **buffers, BlockNumber nblocks, bool skipFsync)
+{
+ smgrsw[reln->smgr_which].smgr_startwritev(ioh,
+ reln, forknum, blocknum, buffers,
+ nblocks, skipFsync);
+}
+
/*
* smgrwriteback() -- Trigger kernel writeback for the supplied range of
* blocks.
@@ -819,6 +869,12 @@ smgrimmedsync(SMgrRelation reln, ForkNumber forknum)
smgrsw[reln->smgr_which].smgr_immedsync(reln, forknum);
}
+int
+smgrfd(SMgrRelation reln, ForkNumber forknum, BlockNumber blocknum, uint32 *off)
+{
+ return smgrsw[reln->smgr_which].smgr_fd(reln, forknum, blocknum, off);
+}
+
/*
* AtEOXact_SMgr
*
@@ -847,3 +903,73 @@ ProcessBarrierSmgrRelease(void)
smgrreleaseall();
return true;
}
+
+void
+pgaio_io_set_target_smgr(PgAioHandle *ioh,
+ struct SMgrRelationData *smgr,
+ ForkNumber forknum,
+ BlockNumber blocknum,
+ int nblocks,
+ bool skip_fsync)
+{
+ PgAioTargetData *sd = pgaio_io_get_target_data(ioh);
+
+ pgaio_io_set_target(ioh, PGAIO_TID_SMGR);
+
+ /* backend is implied via IO owner */
+ sd->smgr.rlocator = smgr->smgr_rlocator.locator;
+ sd->smgr.forkNum = forknum;
+ sd->smgr.blockNum = blocknum;
+ sd->smgr.nblocks = nblocks;
+ sd->smgr.is_temp = SmgrIsTemp(smgr);
+ sd->smgr.release_lock = false;
+ /* Temp relations should never be fsync'd */
+ sd->smgr.skip_fsync = skip_fsync && !SmgrIsTemp(smgr);
+ sd->smgr.mode = RBM_NORMAL;
+}
+
+static void
+smgr_aio_reopen(PgAioHandle *ioh)
+{
+ PgAioTargetData *sd = pgaio_io_get_target_data(ioh);
+ PgAioOpData *od = pgaio_io_get_op_data(ioh);
+ SMgrRelation reln;
+ ProcNumber procno;
+ uint32 off;
+
+ if (sd->smgr.is_temp)
+ procno = pgaio_io_get_owner(ioh);
+ else
+ procno = INVALID_PROC_NUMBER;
+
+ reln = smgropen(sd->smgr.rlocator, procno);
+ od->read.fd = smgrfd(reln, sd->smgr.forkNum, sd->smgr.blockNum, &off);
+ Assert(off == od->read.offset);
+}
+
+static char *
+smgr_aio_describe_identity(const PgAioTargetData *sd)
+{
+ char *path;
+ char *desc;
+
+ path = relpathbackend(sd->smgr.rlocator,
+ sd->smgr.is_temp ? MyProcNumber : INVALID_PROC_NUMBER,
+ sd->smgr.forkNum);
+
+ if (sd->smgr.nblocks == 0)
+ desc = psprintf(_("file \"%s\""), path);
+ else if (sd->smgr.nblocks == 1)
+ desc = psprintf(_("block %u in file \"%s\""),
+ sd->smgr.blockNum,
+ path);
+ else
+ desc = psprintf(_("blocks %u..%u in file \"%s\""),
+ sd->smgr.blockNum,
+ sd->smgr.blockNum + sd->smgr.nblocks - 1,
+ path);
+
+ pfree(path);
+
+ return desc;
+}
--
2.48.1.76.g4e746b1a31.dirty
[text/x-diff] v2.3-0018-aio-Add-pg_aios-view.patch (10.1K, ../../p5flybfcjrzgpgwcnzkxsp3xdw3czfoafkvks6g7en6ewmvymh@wrz7z2vwwc7c/19-v2.3-0018-aio-Add-pg_aios-view.patch)
download | inline diff:
From a92ecb8ff29feaa485c50c10914f30678d3694ad Mon Sep 17 00:00:00 2001
From: Andres Freund <andres@anarazel.de>
Date: Wed, 22 Jan 2025 13:44:40 -0500
Subject: [PATCH v2.3 18/30] aio: Add pg_aios view
Author:
Reviewed-by:
Discussion: https://postgr.es/m/
Backpatch:
---
src/include/catalog/pg_proc.dat | 10 ++
src/backend/catalog/system_views.sql | 3 +
src/backend/storage/aio/Makefile | 1 +
src/backend/storage/aio/aio_funcs.c | 240 +++++++++++++++++++++++++++
src/backend/storage/aio/meson.build | 1 +
src/test/regress/expected/rules.out | 17 ++
6 files changed, 272 insertions(+)
create mode 100644 src/backend/storage/aio/aio_funcs.c
diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat
index 18560755d26..df29275d7b1 100644
--- a/src/include/catalog/pg_proc.dat
+++ b/src/include/catalog/pg_proc.dat
@@ -12435,4 +12435,14 @@
proargtypes => 'int4',
prosrc => 'gist_stratnum_common' },
+# AIO related functions
+{ oid => '9200', descr => 'information about in-progress asynchronous IOs',
+ proname => 'pg_get_aios', prorows => '100', proretset => 't',
+ provolatile => 'v', proparallel => 'r', prorettype => 'record', proargtypes => '',
+ proallargtypes => '{int4,int4,int8,text,text,int8,int8,text,int2,int4,text,text,text,bool,bool,bool}',
+ proargmodes => '{o,o,o,o,o,o,o,o,o,o,o,o,o,o,o,o}',
+ proargnames => '{pid,io_id,io_generation,state,operation,offset,length,target,handle_data_len,raw_result,result,error_desc,target_desc,f_sync,f_localmem,f_buffered}',
+ prosrc => 'pg_get_aios' },
+
+
]
diff --git a/src/backend/catalog/system_views.sql b/src/backend/catalog/system_views.sql
index 46868bf7e89..884c73cd2bf 100644
--- a/src/backend/catalog/system_views.sql
+++ b/src/backend/catalog/system_views.sql
@@ -1388,3 +1388,6 @@ CREATE VIEW pg_stat_subscription_stats AS
CREATE VIEW pg_wait_events AS
SELECT * FROM pg_get_wait_events();
+
+CREATE VIEW pg_aios AS
+ SELECT * FROM pg_get_aios();
diff --git a/src/backend/storage/aio/Makefile b/src/backend/storage/aio/Makefile
index c06c50771e0..3f2469cc399 100644
--- a/src/backend/storage/aio/Makefile
+++ b/src/backend/storage/aio/Makefile
@@ -11,6 +11,7 @@ include $(top_builddir)/src/Makefile.global
OBJS = \
aio.o \
aio_callback.o \
+ aio_funcs.o \
aio_init.o \
aio_io.o \
aio_target.o \
diff --git a/src/backend/storage/aio/aio_funcs.c b/src/backend/storage/aio/aio_funcs.c
new file mode 100644
index 00000000000..65ee3cb22a6
--- /dev/null
+++ b/src/backend/storage/aio/aio_funcs.c
@@ -0,0 +1,240 @@
+/*-------------------------------------------------------------------------
+ *
+ * aio_funcs.c
+ * AIO - SQL interface for AIO
+ *
+ *
+ * Portions Copyright (c) 1996-2025, PostgreSQL Global Development Group
+ * Portions Copyright (c) 1994, Regents of the University of California
+ *
+ * IDENTIFICATION
+ * src/backend/storage/aio/aio_funcs.c
+ *
+ *-------------------------------------------------------------------------
+ */
+
+#include "postgres.h"
+
+#include "storage/aio.h"
+#include "storage/aio_internal.h"
+#include "utils/builtins.h"
+#include "funcapi.h"
+#include "storage/proc.h"
+
+
+/*
+ * Byte length of an iovec.
+ */
+static size_t
+iov_byte_length(const struct iovec *iov, int cnt)
+{
+ size_t len = 0;
+
+ for (int i = 0; i < cnt; i++)
+ {
+ len += iov[i].iov_len;
+ }
+
+ return len;
+}
+
+static const char *
+pgaio_result_status_string(PgAioResultStatus rs)
+{
+ switch (rs)
+ {
+ case ARS_UNKNOWN:
+ return "UNKNOWN";
+ case ARS_OK:
+ return "OK";
+ case ARS_PARTIAL:
+ return "PARTIAL";
+ case ARS_ERROR:
+ return "ERROR";
+ }
+
+ return NULL; /* silence compiler */
+}
+
+Datum
+pg_get_aios(PG_FUNCTION_ARGS)
+{
+ ReturnSetInfo *rsinfo = (ReturnSetInfo *) fcinfo->resultinfo;
+
+ InitMaterializedSRF(fcinfo, 0);
+
+#define PG_GET_AIOS_COLS 16
+
+ for (uint64 i = 0; i < pgaio_ctl->io_handle_count; i++)
+ {
+ PgAioHandle *live_ioh = &pgaio_ctl->io_handles[i];
+ uint32 ioh_id = pgaio_io_get_id(live_ioh);
+ Datum values[PG_GET_AIOS_COLS] = {0};
+ bool nulls[PG_GET_AIOS_COLS] = {0};
+ ProcNumber owner;
+ PGPROC *owner_proc;
+ int32 owner_pid;
+ PgAioHandleState start_state;
+ uint64 start_generation;
+ PgAioHandle ioh_copy;
+ struct iovec iov_copy[PG_IOV_MAX];
+
+retry:
+
+ /*
+ * There is no lock that could prevent the state of the IO to advance
+ * concurrently - and we don't want to introduce one, as that would
+ * introduce atomics into a very common path. Instead we
+ *
+ * 1) determine the state + generation of the IO
+ *
+ * 2) copy the IO to local memory
+ *
+ * 3) check if state and generation of the IO changed
+ */
+
+ /* 1) from above */
+ start_generation = live_ioh->generation;
+ pg_read_barrier();
+ start_state = live_ioh->state;
+
+ if (start_state == PGAIO_HS_IDLE)
+ continue;
+
+ /* 2) from above */
+ memcpy(&ioh_copy, live_ioh, sizeof(PgAioHandle));
+
+ /*
+ * Safe to copy even if no iovec is used - we always reserve the
+ * required space.
+ */
+ memcpy(&iov_copy, &pgaio_ctl->iovecs[ioh_copy.iovec_off],
+ PG_IOV_MAX * sizeof(struct iovec));
+
+ /*
+ * Copy information about owner before 3) below, if the process exited
+ * it'd have to wait for the IO to finish first, which we would detect
+ * in 3).
+ */
+ owner = ioh_copy.owner_procno;
+ owner_proc = GetPGProcByNumber(owner);
+ owner_pid = owner_proc->pid;
+
+ /* 3) from above */
+ pg_read_barrier();
+
+ /*
+ * The IO completed and a new one was started with the same ID. Don't
+ * display it - it really started after this function was called.
+ * There be a risk of a livelock if we just retried endlessly, if IOs
+ * complete very quickly.
+ */
+ if (live_ioh->generation != start_generation)
+ continue;
+
+ /*
+ * The IOs state changed while we were "rendering" it. Just start from
+ * scratch. There's no risk of a livelock here, as an IO has a limited
+ * sets of states it can be in, and state changes go only in a single
+ * direction.
+ */
+ if (live_ioh->state != start_state)
+ goto retry;
+
+ /*
+ * Now that we have copied the IO into local memory and checked that
+ * it's still in the same state, we are not allowed to access "live"
+ * memory anymore. To make it slightly easier to catch such cases, set
+ * the "live" pointers to NULL.
+ */
+ live_ioh = NULL;
+ owner_proc = NULL;
+
+
+ /* column: owning pid */
+ if (owner_pid != 0)
+ values[0] = Int32GetDatum(owner_pid);
+ else
+ nulls[0] = false;
+
+ /* column: IO's id */
+ values[1] = ioh_id;
+
+ /* column: IO's generation */
+ values[2] = Int64GetDatum(start_generation);
+
+ /* column: IO's state */
+ values[3] = CStringGetTextDatum(pgaio_io_get_state_name(&ioh_copy));
+
+ /*
+ * If the IO is in PGAIO_HS_HANDED_OUT state, none of it's fields are
+ * valid yet (or are in the process of being set). Therefore we don't
+ * want to display any other columns.
+ */
+ if (start_state == PGAIO_HS_HANDED_OUT)
+ {
+ memset(nulls + 4, 1, (lengthof(nulls) - 4) * sizeof(bool));
+ goto display;
+ }
+
+ /* column: IO's operation */
+ values[4] = CStringGetTextDatum(pgaio_io_get_op_name(&ioh_copy));
+
+ /* columns: details about the IO's operation */
+ switch (ioh_copy.op)
+ {
+ case PGAIO_OP_INVALID:
+ nulls[5] = true;
+ nulls[6] = true;
+ break;
+ case PGAIO_OP_READV:
+ values[5] = Int64GetDatum(ioh_copy.op_data.read.offset);
+ values[6] =
+ Int64GetDatum(iov_byte_length(iov_copy, ioh_copy.op_data.read.iov_length));
+ break;
+ case PGAIO_OP_WRITEV:
+ values[5] = Int64GetDatum(ioh_copy.op_data.write.offset);
+ values[6] =
+ Int64GetDatum(iov_byte_length(iov_copy, ioh_copy.op_data.write.iov_length));
+ break;
+ }
+
+ /* column: IO's target */
+ values[7] = CStringGetTextDatum(pgaio_io_get_target_name(&ioh_copy));
+
+ /* column: length of IO's data array */
+ values[8] = Int16GetDatum(ioh_copy.handle_data_len);
+
+ /* column: raw result (i.e. some form of syscall return value) */
+ if (start_state == PGAIO_HS_COMPLETED_IO
+ || start_state == PGAIO_HS_COMPLETED_SHARED)
+ values[9] = Int32GetDatum(ioh_copy.result);
+ else
+ nulls[9] = true;
+
+ /*
+ * column: result in the higher level representation (unknown if not
+ * finished
+ */
+ values[10] =
+ CStringGetTextDatum(pgaio_result_status_string(ioh_copy.distilled_result.status));
+
+ /* column: error description */
+ /* AFIXME: implement */
+ nulls[11] = true;
+
+ /* column: target description */
+ values[12] = CStringGetTextDatum(pgaio_io_get_target_description(&ioh_copy));
+
+ /* columns: one for each flag */
+ values[13] = BoolGetDatum(ioh_copy.flags & PGAIO_HF_SYNCHRONOUS);
+ values[14] = BoolGetDatum(ioh_copy.flags & PGAIO_HF_REFERENCES_LOCAL);
+ values[15] = BoolGetDatum(ioh_copy.flags & PGAIO_HF_BUFFERED);
+
+display:
+
+ tuplestore_putvalues(rsinfo->setResult, rsinfo->setDesc, values, nulls);
+ }
+
+ return (Datum) 0;
+}
diff --git a/src/backend/storage/aio/meson.build b/src/backend/storage/aio/meson.build
index 2f0f03d8071..da6df2d3654 100644
--- a/src/backend/storage/aio/meson.build
+++ b/src/backend/storage/aio/meson.build
@@ -3,6 +3,7 @@
backend_sources += files(
'aio.c',
'aio_callback.c',
+ 'aio_funcs.c',
'aio_init.c',
'aio_io.c',
'aio_target.c',
diff --git a/src/test/regress/expected/rules.out b/src/test/regress/expected/rules.out
index 856a8349c50..c0e18a350f5 100644
--- a/src/test/regress/expected/rules.out
+++ b/src/test/regress/expected/rules.out
@@ -1286,6 +1286,23 @@ drop table cchild;
SELECT viewname, definition FROM pg_views
WHERE schemaname = 'pg_catalog'
ORDER BY viewname;
+pg_aios| SELECT pid,
+ io_id,
+ io_generation,
+ state,
+ operation,
+ "offset",
+ length,
+ target,
+ handle_data_len,
+ raw_result,
+ result,
+ error_desc,
+ target_desc,
+ f_sync,
+ f_localmem,
+ f_buffered
+ FROM pg_get_aios() pg_get_aios(pid, io_id, io_generation, state, operation, "offset", length, target, handle_data_len, raw_result, result, error_desc, target_desc, f_sync, f_localmem, f_buffered);
pg_available_extension_versions| SELECT e.name,
e.version,
(x.extname IS NOT NULL) AS installed,
--
2.48.1.76.g4e746b1a31.dirty
[text/x-diff] v2.3-0019-bufmgr-Implement-AIO-read-support.patch (20.1K, ../../p5flybfcjrzgpgwcnzkxsp3xdw3czfoafkvks6g7en6ewmvymh@wrz7z2vwwc7c/20-v2.3-0019-bufmgr-Implement-AIO-read-support.patch)
download | inline diff:
From 34bdf7e671846828be4d194cee881218b78a817b Mon Sep 17 00:00:00 2001
From: Andres Freund <andres@anarazel.de>
Date: Wed, 22 Jan 2025 16:08:58 -0500
Subject: [PATCH v2.3 19/30] bufmgr: Implement AIO read support
As of this commit there are no users of these AIO facilities, that'll come in
later commits.
Author:
Reviewed-By:
Discussion: https://postgr.es/m/
Backpatch:
---
src/include/storage/aio.h | 4 +
src/include/storage/buf_internals.h | 6 +
src/include/storage/bufmgr.h | 8 +
src/backend/storage/aio/aio_callback.c | 5 +
src/backend/storage/buffer/buf_init.c | 3 +
src/backend/storage/buffer/bufmgr.c | 389 ++++++++++++++++++++++++-
src/backend/storage/buffer/localbuf.c | 65 +++++
7 files changed, 473 insertions(+), 7 deletions(-)
diff --git a/src/include/storage/aio.h b/src/include/storage/aio.h
index a948eaeefa7..6f36a0b9e4d 100644
--- a/src/include/storage/aio.h
+++ b/src/include/storage/aio.h
@@ -178,6 +178,10 @@ typedef enum PgAioHandleCallbackID
PGAIO_HCB_MD_READV,
PGAIO_HCB_MD_WRITEV,
+
+ PGAIO_HCB_SHARED_BUFFER_READV,
+
+ PGAIO_HCB_LOCAL_BUFFER_READV,
} PgAioHandleCallbackID;
diff --git a/src/include/storage/buf_internals.h b/src/include/storage/buf_internals.h
index 1a65342177d..9f936cd6b84 100644
--- a/src/include/storage/buf_internals.h
+++ b/src/include/storage/buf_internals.h
@@ -17,6 +17,7 @@
#include "pgstat.h"
#include "port/atomics.h"
+#include "storage/aio_types.h"
#include "storage/buf.h"
#include "storage/bufmgr.h"
#include "storage/condition_variable.h"
@@ -251,6 +252,8 @@ typedef struct BufferDesc
int wait_backend_pgprocno; /* backend of pin-count waiter */
int freeNext; /* link in freelist chain */
+
+ PgAioWaitRef io_wref;
LWLock content_lock; /* to lock access to buffer contents */
} BufferDesc;
@@ -464,4 +467,7 @@ extern void DropRelationLocalBuffers(RelFileLocator rlocator,
extern void DropRelationAllLocalBuffers(RelFileLocator rlocator);
extern void AtEOXact_LocalBuffers(bool isCommit);
+
+extern bool ReadBufferCompleteReadLocal(Buffer buffer, int mode, bool failed);
+
#endif /* BUFMGR_INTERNALS_H */
diff --git a/src/include/storage/bufmgr.h b/src/include/storage/bufmgr.h
index 46b4e0d90f3..5cff4e223f9 100644
--- a/src/include/storage/bufmgr.h
+++ b/src/include/storage/bufmgr.h
@@ -177,6 +177,12 @@ extern PGDLLIMPORT int NLocBuffer;
extern PGDLLIMPORT Block *LocalBufferBlockPointers;
extern PGDLLIMPORT int32 *LocalRefCount;
+
+struct PgAioHandleCallbacks;
+extern const struct PgAioHandleCallbacks aio_shared_buffer_readv_cb;
+extern const struct PgAioHandleCallbacks aio_local_buffer_readv_cb;
+
+
/* upper limit for effective_io_concurrency */
#define MAX_IO_CONCURRENCY 1000
@@ -194,6 +200,8 @@ extern PGDLLIMPORT int32 *LocalRefCount;
/*
* prototypes for functions in bufmgr.c
*/
+struct PgAioHandle;
+
extern PrefetchBufferResult PrefetchSharedBuffer(struct SMgrRelationData *smgr_reln,
ForkNumber forkNum,
BlockNumber blockNum);
diff --git a/src/backend/storage/aio/aio_callback.c b/src/backend/storage/aio/aio_callback.c
index 7fd42880535..6054f57eb23 100644
--- a/src/backend/storage/aio/aio_callback.c
+++ b/src/backend/storage/aio/aio_callback.c
@@ -18,6 +18,7 @@
#include "miscadmin.h"
#include "storage/aio.h"
#include "storage/aio_internal.h"
+#include "storage/bufmgr.h"
#include "storage/md.h"
#include "utils/memutils.h"
@@ -42,6 +43,10 @@ static const PgAioHandleCallbacksEntry aio_handle_cbs[] = {
CALLBACK_ENTRY(PGAIO_HCB_MD_READV, aio_md_readv_cb),
CALLBACK_ENTRY(PGAIO_HCB_MD_WRITEV, aio_md_writev_cb),
+
+ CALLBACK_ENTRY(PGAIO_HCB_SHARED_BUFFER_READV, aio_shared_buffer_readv_cb),
+
+ CALLBACK_ENTRY(PGAIO_HCB_LOCAL_BUFFER_READV, aio_local_buffer_readv_cb),
#undef CALLBACK_ENTRY
};
diff --git a/src/backend/storage/buffer/buf_init.c b/src/backend/storage/buffer/buf_init.c
index ed1f8e03190..ed1dc488a42 100644
--- a/src/backend/storage/buffer/buf_init.c
+++ b/src/backend/storage/buffer/buf_init.c
@@ -14,6 +14,7 @@
*/
#include "postgres.h"
+#include "storage/aio.h"
#include "storage/buf_internals.h"
#include "storage/bufmgr.h"
@@ -125,6 +126,8 @@ BufferManagerShmemInit(void)
buf->buf_id = i;
+ pgaio_wref_clear(&buf->io_wref);
+
/*
* Initially link all the buffers together as unused. Subsequent
* management of this list is done by freelist.c.
diff --git a/src/backend/storage/buffer/bufmgr.c b/src/backend/storage/buffer/bufmgr.c
index 0d8849bf894..169829e8031 100644
--- a/src/backend/storage/buffer/bufmgr.c
+++ b/src/backend/storage/buffer/bufmgr.c
@@ -48,6 +48,7 @@
#include "pg_trace.h"
#include "pgstat.h"
#include "postmaster/bgwriter.h"
+#include "storage/aio.h"
#include "storage/buf_internals.h"
#include "storage/bufmgr.h"
#include "storage/fd.h"
@@ -58,6 +59,7 @@
#include "storage/smgr.h"
#include "storage/standby.h"
#include "utils/memdebug.h"
+#include "utils/memutils.h"
#include "utils/ps_status.h"
#include "utils/rel.h"
#include "utils/resowner.h"
@@ -514,7 +516,8 @@ static int SyncOneBuffer(int buf_id, bool skip_recently_used,
static void WaitIO(BufferDesc *buf);
static bool StartBufferIO(BufferDesc *buf, bool forInput, bool nowait);
static void TerminateBufferIO(BufferDesc *buf, bool clear_dirty,
- uint32 set_flag_bits, bool forget_owner);
+ uint32 set_flag_bits, bool forget_owner,
+ bool syncio);
static void AbortBufferIO(Buffer buffer);
static void shared_buffer_write_error_callback(void *arg);
static void local_buffer_write_error_callback(void *arg);
@@ -1081,7 +1084,7 @@ ZeroAndLockBuffer(Buffer buffer, ReadBufferMode mode, bool already_valid)
else
{
/* Set BM_VALID, terminate IO, and wake up any waiters */
- TerminateBufferIO(bufHdr, false, BM_VALID, true);
+ TerminateBufferIO(bufHdr, false, BM_VALID, true, true);
}
}
else if (!isLocalBuf)
@@ -1566,7 +1569,7 @@ WaitReadBuffers(ReadBuffersOperation *operation)
else
{
/* Set BM_VALID, terminate IO, and wake up any waiters */
- TerminateBufferIO(bufHdr, false, BM_VALID, true);
+ TerminateBufferIO(bufHdr, false, BM_VALID, true, true);
}
/* Report I/Os as completing individually. */
@@ -2450,7 +2453,7 @@ ExtendBufferedRelShared(BufferManagerRelation bmr,
if (lock)
LWLockAcquire(BufferDescriptorGetContentLock(buf_hdr), LW_EXCLUSIVE);
- TerminateBufferIO(buf_hdr, false, BM_VALID, true);
+ TerminateBufferIO(buf_hdr, false, BM_VALID, true, true);
}
pgBufferUsage.shared_blks_written += extend_by;
@@ -3899,7 +3902,7 @@ FlushBuffer(BufferDesc *buf, SMgrRelation reln, IOObject io_object,
* Mark the buffer as clean (unless BM_JUST_DIRTIED has become set) and
* end the BM_IO_IN_PROGRESS state.
*/
- TerminateBufferIO(buf, true, 0, true);
+ TerminateBufferIO(buf, true, 0, true, true);
TRACE_POSTGRESQL_BUFFER_FLUSH_DONE(BufTagGetForkNum(&buf->tag),
buf->tag.blockNum,
@@ -5456,6 +5459,7 @@ WaitIO(BufferDesc *buf)
for (;;)
{
uint32 buf_state;
+ PgAioWaitRef iow;
/*
* It may not be necessary to acquire the spinlock to check the flag
@@ -5463,10 +5467,19 @@ WaitIO(BufferDesc *buf)
* play it safe.
*/
buf_state = LockBufHdr(buf);
+ iow = buf->io_wref;
UnlockBufHdr(buf, buf_state);
if (!(buf_state & BM_IO_IN_PROGRESS))
break;
+
+ if (pgaio_wref_valid(&iow))
+ {
+ pgaio_wref_wait(&iow);
+ ConditionVariablePrepareToSleep(cv);
+ continue;
+ }
+
ConditionVariableSleep(cv, WAIT_EVENT_BUFFER_IO);
}
ConditionVariableCancelSleep();
@@ -5555,7 +5568,7 @@ StartBufferIO(BufferDesc *buf, bool forInput, bool nowait)
*/
static void
TerminateBufferIO(BufferDesc *buf, bool clear_dirty, uint32 set_flag_bits,
- bool forget_owner)
+ bool forget_owner, bool syncio)
{
uint32 buf_state;
@@ -5567,6 +5580,13 @@ TerminateBufferIO(BufferDesc *buf, bool clear_dirty, uint32 set_flag_bits,
if (clear_dirty && !(buf_state & BM_JUST_DIRTIED))
buf_state &= ~(BM_DIRTY | BM_CHECKPOINT_NEEDED);
+ if (!syncio)
+ {
+ /* release ownership by the AIO subsystem */
+ buf_state -= BUF_REFCOUNT_ONE;
+ pgaio_wref_clear(&buf->io_wref);
+ }
+
buf_state |= set_flag_bits;
UnlockBufHdr(buf, buf_state);
@@ -5575,6 +5595,40 @@ TerminateBufferIO(BufferDesc *buf, bool clear_dirty, uint32 set_flag_bits,
BufferDescriptorGetBuffer(buf));
ConditionVariableBroadcast(BufferDescriptorGetIOCV(buf));
+
+ /*
+ * If we just released a pin, need to do BM_PIN_COUNT_WAITER handling.
+ * Most of the time the current backend will hold another pin preventing
+ * that from happening, but that's e.g. not the case when completing an IO
+ * another backend started.
+ *
+ * AFIXME: Deduplicate with UnpinBufferNoOwner() or just replace
+ * BM_PIN_COUNT_WAITER with something saner.
+ */
+ /* Support LockBufferForCleanup() */
+ if (buf_state & BM_PIN_COUNT_WAITER)
+ {
+ /*
+ * Acquire the buffer header lock, re-check that there's a waiter.
+ * Another backend could have unpinned this buffer, and already woken
+ * up the waiter. There's no danger of the buffer being replaced
+ * after we unpinned it above, as it's pinned by the waiter.
+ */
+ buf_state = LockBufHdr(buf);
+
+ if ((buf_state & BM_PIN_COUNT_WAITER) &&
+ BUF_STATE_GET_REFCOUNT(buf_state) == 1)
+ {
+ /* we just released the last pin other than the waiter's */
+ int wait_backend_pgprocno = buf->wait_backend_pgprocno;
+
+ buf_state &= ~BM_PIN_COUNT_WAITER;
+ UnlockBufHdr(buf, buf_state);
+ ProcSendSignal(wait_backend_pgprocno);
+ }
+ else
+ UnlockBufHdr(buf, buf_state);
+ }
}
/*
@@ -5626,7 +5680,7 @@ AbortBufferIO(Buffer buffer)
}
}
- TerminateBufferIO(buf_hdr, false, BM_IO_ERROR, false);
+ TerminateBufferIO(buf_hdr, false, BM_IO_ERROR, false, true);
}
/*
@@ -6085,3 +6139,324 @@ EvictUnpinnedBuffer(Buffer buf)
return result;
}
+
+static bool
+ReadBufferCompleteReadShared(Buffer buffer, int mode, bool failed)
+{
+ BufferDesc *bufHdr = NULL;
+ BlockNumber blockno;
+ bool buf_failed = false;
+ char *bufdata = BufferGetBlock(buffer);
+
+ Assert(BufferIsValid(buffer));
+
+ bufHdr = GetBufferDescriptor(buffer - 1);
+ blockno = bufHdr->tag.blockNum;
+
+#ifdef USE_ASSERT_CHECKING
+ {
+ uint32 buf_state = pg_atomic_read_u32(&bufHdr->state);
+
+ Assert(buf_state & BM_TAG_VALID);
+ Assert(!(buf_state & BM_VALID));
+ Assert(buf_state & BM_IO_IN_PROGRESS);
+ Assert(!(buf_state & BM_DIRTY));
+ }
+#endif
+
+ /* check for garbage data */
+ if (!failed &&
+ !PageIsVerifiedExtended((Page) bufdata, blockno,
+ PIV_LOG_WARNING | PIV_REPORT_STAT))
+ {
+ RelFileLocator rlocator = BufTagGetRelFileLocator(&bufHdr->tag);
+ BlockNumber forkNum = bufHdr->tag.forkNum;
+
+ /* AFIXME: relpathperm allocates memory */
+ MemoryContextSwitchTo(ErrorContext);
+ if (mode == READ_BUFFERS_ZERO_ON_ERROR || zero_damaged_pages)
+ {
+ ereport(LOG,
+ (errcode(ERRCODE_DATA_CORRUPTED),
+ errmsg("invalid page in block %u of relation %s; zeroing out page",
+ blockno,
+ relpathperm(rlocator, forkNum))));
+ memset(bufdata, 0, BLCKSZ);
+ }
+ else
+ {
+ ereport(LOG,
+ (errcode(ERRCODE_DATA_CORRUPTED),
+ errmsg("invalid page in block %u of relation %s",
+ blockno,
+ relpathperm(rlocator, forkNum))));
+ failed = true;
+ buf_failed = true;
+ }
+ }
+
+ /* Terminate I/O and set BM_VALID. */
+ TerminateBufferIO(bufHdr, false,
+ failed ? BM_IO_ERROR : BM_VALID,
+ false, false);
+
+ /* Report I/Os as completing individually. */
+
+ /* FIXME: Should we do TRACE_POSTGRESQL_BUFFER_READ_DONE here? */
+ return buf_failed;
+}
+
+/*
+ * Helper to prepare IO on shared buffers for execution, shared between reads
+ * and writes.
+ */
+static void
+shared_buffer_stage_common(PgAioHandle *ioh, bool is_write)
+{
+ uint64 *io_data;
+ uint8 handle_data_len;
+ PgAioWaitRef io_ref;
+ BufferTag first PG_USED_FOR_ASSERTS_ONLY = {0};
+
+ io_data = pgaio_io_get_handle_data(ioh, &handle_data_len);
+
+ pgaio_io_get_wref(ioh, &io_ref);
+
+ for (int i = 0; i < handle_data_len; i++)
+ {
+ Buffer buf = (Buffer) io_data[i];
+ BufferDesc *bufHdr;
+ uint32 buf_state;
+
+ bufHdr = GetBufferDescriptor(buf - 1);
+
+ if (i == 0)
+ first = bufHdr->tag;
+ else
+ {
+ Assert(bufHdr->tag.relNumber == first.relNumber);
+ Assert(bufHdr->tag.blockNum == first.blockNum + i);
+ }
+
+
+ buf_state = LockBufHdr(bufHdr);
+
+ Assert(buf_state & BM_TAG_VALID);
+ if (is_write)
+ {
+ Assert(buf_state & BM_VALID);
+ Assert(buf_state & BM_DIRTY);
+ }
+ else
+ Assert(!(buf_state & BM_VALID));
+
+ Assert(buf_state & BM_IO_IN_PROGRESS);
+ Assert(BUF_STATE_GET_REFCOUNT(buf_state) >= 1);
+
+ buf_state += BUF_REFCOUNT_ONE;
+ bufHdr->io_wref = io_ref;
+
+ UnlockBufHdr(bufHdr, buf_state);
+
+ if (is_write)
+ {
+ LWLock *content_lock;
+
+ content_lock = BufferDescriptorGetContentLock(bufHdr);
+
+ Assert(LWLockHeldByMe(content_lock));
+
+ /*
+ * Lock is now owned by AIO subsystem.
+ */
+ LWLockDisown(content_lock);
+ RESUME_INTERRUPTS();
+ }
+
+ /*
+ * Stop tracking this buffer via the resowner - the AIO system now
+ * keeps track.
+ */
+ ResourceOwnerForgetBufferIO(CurrentResourceOwner, buf);
+ }
+}
+
+static void
+shared_buffer_readv_stage(PgAioHandle *ioh)
+{
+ shared_buffer_stage_common(ioh, false);
+}
+
+static PgAioResult
+shared_buffer_readv_complete(PgAioHandle *ioh, PgAioResult prior_result)
+{
+ PgAioResult result = prior_result;
+ int mode = pgaio_io_get_target_data(ioh)->smgr.mode;
+ uint64 *io_data;
+ uint8 handle_data_len;
+
+ ereport(DEBUG5,
+ errmsg("%s: %d %d", __func__, prior_result.status, prior_result.result),
+ errhidestmt(true), errhidecontext(true));
+
+ io_data = pgaio_io_get_handle_data(ioh, &handle_data_len);
+
+ for (int io_data_off = 0; io_data_off < handle_data_len; io_data_off++)
+ {
+ Buffer buf = io_data[io_data_off];
+ bool buf_failed;
+ bool failed;
+
+ failed =
+ prior_result.status == ARS_ERROR
+ || prior_result.result <= io_data_off;
+
+ ereport(DEBUG5,
+ errmsg("calling rbcrs for buf %d with failed %d, status: %d, result: %d, data_off: %d",
+ buf, failed, prior_result.status, prior_result.result, io_data_off),
+ errhidestmt(true), errhidecontext(true));
+
+ /*
+ * XXX: It might be better to not set BM_IO_ERROR (which is what
+ * failed = true leads to) when it's just a short read...
+ */
+ buf_failed = ReadBufferCompleteReadShared(buf,
+ mode,
+ failed);
+
+ if (result.status != ARS_ERROR && buf_failed)
+ {
+ result.status = ARS_ERROR;
+ result.id = PGAIO_HCB_SHARED_BUFFER_READV;
+ result.error_data = io_data_off + 1;
+ }
+ }
+
+ return result;
+}
+
+static void
+buffer_readv_report(PgAioResult result, const PgAioTargetData *target_data, int elevel)
+{
+ MemoryContext oldContext = CurrentMemoryContext;
+ ProcNumber errProc;
+
+ if (target_data->smgr.is_temp)
+ errProc = MyProcNumber;
+ else
+ errProc = INVALID_PROC_NUMBER;
+
+ /*
+ * AFIXME: need infrastructure to allow memory allocation for error
+ * reporting
+ */
+ oldContext = MemoryContextSwitchTo(ErrorContext);
+
+ ereport(elevel,
+ errcode(ERRCODE_DATA_CORRUPTED),
+ errmsg("invalid page in block %u of relation %s",
+ target_data->smgr.blockNum + result.error_data,
+ relpathbackend(target_data->smgr.rlocator, errProc, target_data->smgr.forkNum)
+ )
+ );
+ MemoryContextSwitchTo(oldContext);
+}
+
+/*
+ * Helper to stage IO on local buffers for execution, shared between reads
+ * and writes.
+ */
+static void
+local_buffer_readv_stage(PgAioHandle *ioh)
+{
+ uint64 *io_data;
+ uint8 handle_data_len;
+ PgAioWaitRef io_wref;
+
+ io_data = pgaio_io_get_handle_data(ioh, &handle_data_len);
+
+ pgaio_io_get_wref(ioh, &io_wref);
+
+ for (int i = 0; i < handle_data_len; i++)
+ {
+ Buffer buf = (Buffer) io_data[i];
+ BufferDesc *bufHdr;
+ uint32 buf_state;
+
+ bufHdr = GetLocalBufferDescriptor(-buf - 1);
+
+ buf_state = pg_atomic_read_u32(&bufHdr->state);
+
+ bufHdr->io_wref = io_wref;
+ LocalRefCount[-buf - 1] += 1;
+
+ UnlockBufHdr(bufHdr, buf_state);
+ }
+}
+
+static PgAioResult
+local_buffer_readv_complete(PgAioHandle *ioh, PgAioResult prior_result)
+{
+ PgAioResult result = prior_result;
+ PgAioTargetData *td = pgaio_io_get_target_data(ioh);
+ int mode = td->smgr.mode;
+ uint64 *io_data;
+ uint8 handle_data_len;
+
+ Assert(td->smgr.is_temp);
+ Assert(pgaio_io_get_owner(ioh) == MyProcNumber);
+
+ ereport(DEBUG5,
+ errmsg("%s: %d %d", __func__, prior_result.status, prior_result.result),
+ errhidestmt(true), errhidecontext(true));
+
+ io_data = pgaio_io_get_handle_data(ioh, &handle_data_len);
+
+ for (int io_data_off = 0; io_data_off < handle_data_len; io_data_off++)
+ {
+ Buffer buf = io_data[io_data_off];
+ bool buf_failed;
+ bool failed;
+
+ failed =
+ prior_result.status == ARS_ERROR
+ || prior_result.result <= io_data_off;
+
+ ereport(DEBUG5,
+ errmsg("calling rbcrl for buf %d with failed %d, status: %d, result: %d, data_off: %d",
+ buf, failed, prior_result.status, prior_result.result, io_data_off),
+ errhidestmt(true), errhidecontext(true));
+
+ buf_failed = ReadBufferCompleteReadLocal(buf,
+ mode,
+ failed);
+
+ if (result.status != ARS_ERROR && buf_failed)
+ {
+ result.status = ARS_ERROR;
+ result.id = PGAIO_HCB_LOCAL_BUFFER_READV;
+ result.error_data = io_data_off + 1;
+ }
+ }
+
+ return result;
+}
+
+
+const struct PgAioHandleCallbacks aio_shared_buffer_readv_cb = {
+ .stage = shared_buffer_readv_stage,
+ .complete_shared = shared_buffer_readv_complete,
+ .report = buffer_readv_report,
+};
+const struct PgAioHandleCallbacks aio_local_buffer_readv_cb = {
+ .stage = local_buffer_readv_stage,
+
+ /*
+ * Note that this, in contrast to the shared_buffers case, uses
+ * complete_local, as only the issuing backend has access to the required
+ * datastructures. This is important in case the IO completion may be
+ * consumed incidentally by another backend.
+ */
+ .complete_local = local_buffer_readv_complete,
+ .report = buffer_readv_report,
+};
diff --git a/src/backend/storage/buffer/localbuf.c b/src/backend/storage/buffer/localbuf.c
index 8f81428970b..b3805c1ff94 100644
--- a/src/backend/storage/buffer/localbuf.c
+++ b/src/backend/storage/buffer/localbuf.c
@@ -18,6 +18,7 @@
#include "access/parallel.h"
#include "executor/instrument.h"
#include "pgstat.h"
+#include "storage/aio.h"
#include "storage/buf_internals.h"
#include "storage/bufmgr.h"
#include "storage/fd.h"
@@ -621,6 +622,8 @@ InitLocalBuffers(void)
*/
buf->buf_id = -i - 2;
+ pgaio_wref_clear(&buf->io_wref);
+
/*
* Intentionally do not initialize the buffer's atomic variable
* (besides zeroing the underlying memory above). That way we get
@@ -837,3 +840,65 @@ AtProcExit_LocalBuffers(void)
*/
CheckForLocalBufferLeaks();
}
+
+bool
+ReadBufferCompleteReadLocal(Buffer buffer, int mode, bool failed)
+{
+ BufferDesc *buf_hdr = NULL;
+ BlockNumber blockno;
+ bool buf_failed = false;
+ char *bufdata = BufferGetBlock(buffer);
+
+ Assert(BufferIsValid(buffer));
+
+ buf_hdr = GetLocalBufferDescriptor(-buffer - 1);
+ blockno = buf_hdr->tag.blockNum;
+
+ /* check for garbage data */
+ if (!failed &&
+ !PageIsVerifiedExtended((Page) bufdata, blockno,
+ PIV_LOG_WARNING | PIV_REPORT_STAT))
+ {
+ RelFileLocator rlocator = BufTagGetRelFileLocator(&buf_hdr->tag);
+ BlockNumber forkNum = buf_hdr->tag.forkNum;
+
+ MemoryContextSwitchTo(ErrorContext);
+
+ if (mode == READ_BUFFERS_ZERO_ON_ERROR || zero_damaged_pages)
+ {
+
+ ereport(LOG,
+ (errcode(ERRCODE_DATA_CORRUPTED),
+ errmsg("invalid page in block %u of relation %s; zeroing out page",
+ blockno,
+ relpathbackend(rlocator, MyProcNumber, forkNum))));
+ memset(bufdata, 0, BLCKSZ);
+ }
+ else
+ {
+ ereport(LOG,
+ (errcode(ERRCODE_DATA_CORRUPTED),
+ errmsg("invalid page in block %u of relation %s",
+ blockno,
+ relpathbackend(rlocator, MyProcNumber, forkNum))));
+ failed = true;
+ buf_failed = true;
+ }
+ }
+
+ /* Terminate I/O and set BM_VALID. */
+ pgaio_wref_clear(&buf_hdr->io_wref);
+
+ {
+ uint32 buf_state;
+
+ buf_state = pg_atomic_read_u32(&buf_hdr->state);
+ buf_state |= BM_VALID;
+ pg_atomic_unlocked_write_u32(&buf_hdr->state, buf_state);
+ }
+
+ /* release pin held by IO subsystem */
+ LocalRefCount[-buffer - 1] -= 1;
+
+ return buf_failed;
+}
--
2.48.1.76.g4e746b1a31.dirty
[text/x-diff] v2.3-0020-WIP-localbuf-Track-pincount-in-BufferDesc-as-we.patch (8.7K, ../../p5flybfcjrzgpgwcnzkxsp3xdw3czfoafkvks6g7en6ewmvymh@wrz7z2vwwc7c/21-v2.3-0020-WIP-localbuf-Track-pincount-in-BufferDesc-as-we.patch)
download | inline diff:
From 9c9745754dc88502e050b5822d90d20b517a052b Mon Sep 17 00:00:00 2001
From: Andres Freund <andres@anarazel.de>
Date: Wed, 22 Jan 2025 13:44:44 -0500
Subject: [PATCH v2.3 20/30] WIP: localbuf: Track pincount in BufferDesc as
well
For AIO on temp tables the AIO subsystem needs to be able to ensure a pin on a
buffer while AIO is going on, even if the IO issuing query errors out. To do
so, track the refcount in BufferDesc.state, not ust LocalRefCount.
Note that we still don't need locking, AIO completion callbacks for local
buffers are executed in the issuing session (nobody else has access to the
BufferDesc).
---
src/backend/storage/buffer/bufmgr.c | 40 ++++++++--
src/backend/storage/buffer/localbuf.c | 108 ++++++++++++++++----------
2 files changed, 101 insertions(+), 47 deletions(-)
diff --git a/src/backend/storage/buffer/bufmgr.c b/src/backend/storage/buffer/bufmgr.c
index 169829e8031..fe871691350 100644
--- a/src/backend/storage/buffer/bufmgr.c
+++ b/src/backend/storage/buffer/bufmgr.c
@@ -5356,8 +5356,20 @@ ConditionalLockBufferForCleanup(Buffer buffer)
Assert(refcount > 0);
if (refcount != 1)
return false;
- /* Nobody else to wait for */
- return true;
+
+ bufHdr = GetLocalBufferDescriptor(-buffer - 1);
+ buf_state = pg_atomic_read_u32(&bufHdr->state);
+
+ /*
+ * Check that the AIO subsystem doesn't have a pin. Likely not
+ * possible today, but better safe than sorry.
+ */
+ refcount = BUF_STATE_GET_REFCOUNT(buf_state);
+ Assert(refcount > 0);
+ if (refcount == 1)
+ return true;
+
+ return false;
}
/* There should be exactly one local pin */
@@ -5409,8 +5421,18 @@ IsBufferCleanupOK(Buffer buffer)
/* There should be exactly one pin */
if (LocalRefCount[-buffer - 1] != 1)
return false;
- /* Nobody else to wait for */
- return true;
+
+ bufHdr = GetLocalBufferDescriptor(-buffer - 1);
+ buf_state = pg_atomic_read_u32(&bufHdr->state);
+
+ /*
+ * Check that the AIO subsystem doesn't have a pin. Likely not
+ * possible today, but better safe than sorry.
+ */
+ if (BUF_STATE_GET_REFCOUNT(buf_state) == 1)
+ return true;
+
+ return false;
}
/* There should be exactly one local pin */
@@ -6388,9 +6410,15 @@ local_buffer_readv_stage(PgAioHandle *ioh)
buf_state = pg_atomic_read_u32(&bufHdr->state);
bufHdr->io_wref = io_wref;
- LocalRefCount[-buf - 1] += 1;
- UnlockBufHdr(bufHdr, buf_state);
+ /*
+ * Track pin by AIO subsystem in BufferDesc, not in LocalRefCount as
+ * one might initially think. This is necessary to handle this backend
+ * erroring out while AIO is still in progress.
+ */
+ buf_state += BUF_REFCOUNT_ONE;
+
+ pg_atomic_unlocked_write_u32(&bufHdr->state, buf_state);
}
}
diff --git a/src/backend/storage/buffer/localbuf.c b/src/backend/storage/buffer/localbuf.c
index b3805c1ff94..72c93ae15a2 100644
--- a/src/backend/storage/buffer/localbuf.c
+++ b/src/backend/storage/buffer/localbuf.c
@@ -208,10 +208,19 @@ GetLocalVictimBuffer(void)
pg_atomic_unlocked_write_u32(&bufHdr->state, buf_state);
trycounter = NLocBuffer;
}
+ else if (BUF_STATE_GET_REFCOUNT(buf_state) > 0)
+ {
+ /*
+ * This can be reached if the backend initiated AIO for this
+ * buffer and then errored out.
+ */
+ }
else
{
/* Found a usable buffer */
PinLocalBuffer(bufHdr, false);
+ /* the buf_state may be modified inside PinLocalBuffer */
+ buf_state = pg_atomic_read_u32(&bufHdr->state);
break;
}
}
@@ -476,6 +485,44 @@ MarkLocalBufferDirty(Buffer buffer)
pg_atomic_unlocked_write_u32(&bufHdr->state, buf_state);
}
+static void
+InvalidateLocalBuffer(BufferDesc *bufHdr)
+{
+ Buffer buffer = BufferDescriptorGetBuffer(bufHdr);
+ int bufid = -buffer - 1;
+ uint32 buf_state;
+ LocalBufferLookupEnt *hresult;
+
+ buf_state = pg_atomic_read_u32(&bufHdr->state);
+
+ /*
+ * We need to test not just LocalRefCount[bufid] but also the BufferDesc
+ * itself, as the latter is used to represent a pin by the AIO subsystem.
+ * This can happen if AIO is initiated and then the query errors out.
+ */
+ if (LocalRefCount[bufid] != 0 ||
+ BUF_STATE_GET_REFCOUNT(buf_state) > 0)
+ elog(ERROR, "block %u of %s is still referenced (local %u)",
+ bufHdr->tag.blockNum,
+ relpathbackend(BufTagGetRelFileLocator(&bufHdr->tag),
+ MyProcNumber,
+ BufTagGetForkNum(&bufHdr->tag)),
+ LocalRefCount[bufid]);
+
+ /* Remove entry from hashtable */
+ hresult = (LocalBufferLookupEnt *)
+ hash_search(LocalBufHash, &bufHdr->tag, HASH_REMOVE, NULL);
+ if (!hresult) /* shouldn't happen */
+ elog(ERROR, "local buffer hash table corrupted");
+ /* Mark buffer invalid */
+ ClearBufferTag(&bufHdr->tag);
+
+ buf_state &= ~BUF_FLAG_MASK;
+ buf_state &= ~BUF_USAGECOUNT_MASK;
+ pg_atomic_unlocked_write_u32(&bufHdr->state, buf_state);
+
+}
+
/*
* DropRelationLocalBuffers
* This function removes from the buffer pool all the pages of the
@@ -496,7 +543,6 @@ DropRelationLocalBuffers(RelFileLocator rlocator, ForkNumber forkNum,
for (i = 0; i < NLocBuffer; i++)
{
BufferDesc *bufHdr = GetLocalBufferDescriptor(i);
- LocalBufferLookupEnt *hresult;
uint32 buf_state;
buf_state = pg_atomic_read_u32(&bufHdr->state);
@@ -506,24 +552,7 @@ DropRelationLocalBuffers(RelFileLocator rlocator, ForkNumber forkNum,
BufTagGetForkNum(&bufHdr->tag) == forkNum &&
bufHdr->tag.blockNum >= firstDelBlock)
{
- if (LocalRefCount[i] != 0)
- elog(ERROR, "block %u of %s is still referenced (local %u)",
- bufHdr->tag.blockNum,
- relpathbackend(BufTagGetRelFileLocator(&bufHdr->tag),
- MyProcNumber,
- BufTagGetForkNum(&bufHdr->tag)),
- LocalRefCount[i]);
-
- /* Remove entry from hashtable */
- hresult = (LocalBufferLookupEnt *)
- hash_search(LocalBufHash, &bufHdr->tag, HASH_REMOVE, NULL);
- if (!hresult) /* shouldn't happen */
- elog(ERROR, "local buffer hash table corrupted");
- /* Mark buffer invalid */
- ClearBufferTag(&bufHdr->tag);
- buf_state &= ~BUF_FLAG_MASK;
- buf_state &= ~BUF_USAGECOUNT_MASK;
- pg_atomic_unlocked_write_u32(&bufHdr->state, buf_state);
+ InvalidateLocalBuffer(bufHdr);
}
}
}
@@ -543,7 +572,6 @@ DropRelationAllLocalBuffers(RelFileLocator rlocator)
for (i = 0; i < NLocBuffer; i++)
{
BufferDesc *bufHdr = GetLocalBufferDescriptor(i);
- LocalBufferLookupEnt *hresult;
uint32 buf_state;
buf_state = pg_atomic_read_u32(&bufHdr->state);
@@ -551,23 +579,7 @@ DropRelationAllLocalBuffers(RelFileLocator rlocator)
if ((buf_state & BM_TAG_VALID) &&
BufTagMatchesRelFileLocator(&bufHdr->tag, &rlocator))
{
- if (LocalRefCount[i] != 0)
- elog(ERROR, "block %u of %s is still referenced (local %u)",
- bufHdr->tag.blockNum,
- relpathbackend(BufTagGetRelFileLocator(&bufHdr->tag),
- MyProcNumber,
- BufTagGetForkNum(&bufHdr->tag)),
- LocalRefCount[i]);
- /* Remove entry from hashtable */
- hresult = (LocalBufferLookupEnt *)
- hash_search(LocalBufHash, &bufHdr->tag, HASH_REMOVE, NULL);
- if (!hresult) /* shouldn't happen */
- elog(ERROR, "local buffer hash table corrupted");
- /* Mark buffer invalid */
- ClearBufferTag(&bufHdr->tag);
- buf_state &= ~BUF_FLAG_MASK;
- buf_state &= ~BUF_USAGECOUNT_MASK;
- pg_atomic_unlocked_write_u32(&bufHdr->state, buf_state);
+ InvalidateLocalBuffer(bufHdr);
}
}
}
@@ -667,12 +679,13 @@ PinLocalBuffer(BufferDesc *buf_hdr, bool adjust_usagecount)
if (LocalRefCount[bufid] == 0)
{
NLocalPinnedBuffers++;
+ buf_state += BUF_REFCOUNT_ONE;
if (adjust_usagecount &&
BUF_STATE_GET_USAGECOUNT(buf_state) < BM_MAX_USAGE_COUNT)
{
buf_state += BUF_USAGECOUNT_ONE;
- pg_atomic_unlocked_write_u32(&buf_hdr->state, buf_state);
}
+ pg_atomic_unlocked_write_u32(&buf_hdr->state, buf_state);
}
LocalRefCount[bufid]++;
ResourceOwnerRememberBuffer(CurrentResourceOwner,
@@ -698,7 +711,17 @@ UnpinLocalBufferNoOwner(Buffer buffer)
Assert(NLocalPinnedBuffers > 0);
if (--LocalRefCount[buffid] == 0)
+ {
+ BufferDesc *buf_hdr = GetLocalBufferDescriptor(buffid);
+ uint32 buf_state;
+
NLocalPinnedBuffers--;
+
+ buf_state = pg_atomic_read_u32(&buf_hdr->state);
+ Assert(BUF_STATE_GET_REFCOUNT(buf_state) > 0);
+ buf_state -= BUF_REFCOUNT_ONE;
+ pg_atomic_unlocked_write_u32(&buf_hdr->state, buf_state);
+ }
}
/*
@@ -894,11 +917,14 @@ ReadBufferCompleteReadLocal(Buffer buffer, int mode, bool failed)
buf_state = pg_atomic_read_u32(&buf_hdr->state);
buf_state |= BM_VALID;
+
+ /*
+ * Release pin held by IO subsystem, see also
+ * local_buffer_readv_prepare().
+ */
+ buf_state -= BUF_REFCOUNT_ONE;
pg_atomic_unlocked_write_u32(&buf_hdr->state, buf_state);
}
- /* release pin held by IO subsystem */
- LocalRefCount[-buffer - 1] -= 1;
-
return buf_failed;
}
--
2.48.1.76.g4e746b1a31.dirty
[text/x-diff] v2.3-0021-bufmgr-Use-aio-for-StartReadBuffers.patch (18.5K, ../../p5flybfcjrzgpgwcnzkxsp3xdw3czfoafkvks6g7en6ewmvymh@wrz7z2vwwc7c/22-v2.3-0021-bufmgr-Use-aio-for-StartReadBuffers.patch)
download | inline diff:
From b153f4c8c7cf10171dd7390920ef38e079be1c87 Mon Sep 17 00:00:00 2001
From: Andres Freund <andres@anarazel.de>
Date: Wed, 22 Jan 2025 13:44:45 -0500
Subject: [PATCH v2.3 21/30] bufmgr: Use aio for StartReadBuffers()
Author:
Reviewed-By:
Discussion: https://postgr.es/m/
Backpatch:
---
src/include/storage/bufmgr.h | 25 +-
src/backend/storage/buffer/bufmgr.c | 377 ++++++++++++++++++++--------
2 files changed, 298 insertions(+), 104 deletions(-)
diff --git a/src/include/storage/bufmgr.h b/src/include/storage/bufmgr.h
index 5cff4e223f9..46ee957e99c 100644
--- a/src/include/storage/bufmgr.h
+++ b/src/include/storage/bufmgr.h
@@ -15,6 +15,7 @@
#define BUFMGR_H
#include "port/pg_iovec.h"
+#include "storage/aio_types.h"
#include "storage/block.h"
#include "storage/buf.h"
#include "storage/bufpage.h"
@@ -107,10 +108,18 @@ typedef struct BufferManagerRelation
#define BMR_REL(p_rel) ((BufferManagerRelation){.rel = p_rel})
#define BMR_SMGR(p_smgr, p_relpersistence) ((BufferManagerRelation){.smgr = p_smgr, .relpersistence = p_relpersistence})
+
+#define MAX_IO_COMBINE_LIMIT PG_IOV_MAX
+#define DEFAULT_IO_COMBINE_LIMIT Min(MAX_IO_COMBINE_LIMIT, (128 * 1024) / BLCKSZ)
+
+
/* Zero out page if reading fails. */
#define READ_BUFFERS_ZERO_ON_ERROR (1 << 0)
/* Call smgrprefetch() if I/O necessary. */
#define READ_BUFFERS_ISSUE_ADVICE (1 << 1)
+/* IO will immediately be waited for */
+#define READ_BUFFERS_SYNCHRONOUSLY (1 << 2)
+
struct ReadBuffersOperation
{
@@ -131,6 +140,20 @@ struct ReadBuffersOperation
int flags;
int16 nblocks;
int16 io_buffers_len;
+
+ /*
+ * In some rare-ish cases one operation causes multiple reads (e.g. if a
+ * buffer was concurrently read by another backend). It'd be much better
+ * if we ensured that each ReadBuffersOperation covered only one IO - but
+ * that's not entirely trivial, due to having pinned victim buffers before
+ * starting IOs.
+ *
+ * TODO: Change the API of StartReadBuffers() to ensure we only ever need
+ * one IO.
+ */
+ int16 nios;
+ PgAioWaitRef wrefs[MAX_IO_COMBINE_LIMIT];
+ PgAioReturn returns[MAX_IO_COMBINE_LIMIT];
};
typedef struct ReadBuffersOperation ReadBuffersOperation;
@@ -161,8 +184,6 @@ extern PGDLLIMPORT bool track_io_timing;
extern PGDLLIMPORT int effective_io_concurrency;
extern PGDLLIMPORT int maintenance_io_concurrency;
-#define MAX_IO_COMBINE_LIMIT PG_IOV_MAX
-#define DEFAULT_IO_COMBINE_LIMIT Min(MAX_IO_COMBINE_LIMIT, (128 * 1024) / BLCKSZ)
extern PGDLLIMPORT int io_combine_limit;
extern PGDLLIMPORT int checkpoint_flush_after;
diff --git a/src/backend/storage/buffer/bufmgr.c b/src/backend/storage/buffer/bufmgr.c
index fe871691350..70f1da84083 100644
--- a/src/backend/storage/buffer/bufmgr.c
+++ b/src/backend/storage/buffer/bufmgr.c
@@ -1235,10 +1235,9 @@ ReadBuffer_common(Relation rel, SMgrRelation smgr, char smgr_persistence,
return buffer;
}
+ flags = READ_BUFFERS_SYNCHRONOUSLY;
if (mode == RBM_ZERO_ON_ERROR)
- flags = READ_BUFFERS_ZERO_ON_ERROR;
- else
- flags = 0;
+ flags |= READ_BUFFERS_ZERO_ON_ERROR;
operation.smgr = smgr;
operation.rel = rel;
operation.persistence = persistence;
@@ -1253,6 +1252,9 @@ ReadBuffer_common(Relation rel, SMgrRelation smgr, char smgr_persistence,
return buffer;
}
+static bool AsyncReadBuffers(ReadBuffersOperation *operation,
+ int nblocks);
+
static pg_attribute_always_inline bool
StartReadBuffersImpl(ReadBuffersOperation *operation,
Buffer *buffers,
@@ -1288,6 +1290,11 @@ StartReadBuffersImpl(ReadBuffersOperation *operation,
* so we stop here.
*/
actual_nblocks = i + 1;
+
+ ereport(DEBUG3,
+ errmsg("found buf at idx %i: %s",
+ i, DebugPrintBufferRefcount(buffers[i])),
+ errhidestmt(true), errhidecontext(true));
break;
}
else
@@ -1324,28 +1331,51 @@ StartReadBuffersImpl(ReadBuffersOperation *operation,
operation->flags = flags;
operation->nblocks = actual_nblocks;
operation->io_buffers_len = io_buffers_len;
+ operation->nios = 0;
- if (flags & READ_BUFFERS_ISSUE_ADVICE)
+ /*
+ * When using AIO, start the IO in the background. If not, issue prefetch
+ * requests if desired by the caller.
+ *
+ * The reason we have a dedicated path for IOMETHOD_SYNC here is to derisk
+ * the introduction of AIO somewhat. It's a large architectural change,
+ * with lots of chances for unanticipated performance effects. Use of
+ * IOMETHOD_SYNC already leads to not actually performing IO
+ * asynchronously, but without the check here we'd execute IO earlier than
+ * we used to.
+ */
+ if (io_method != IOMETHOD_SYNC)
{
- /*
- * In theory we should only do this if PinBufferForBlock() had to
- * allocate new buffers above. That way, if two calls to
- * StartReadBuffers() were made for the same blocks before
- * WaitReadBuffers(), only the first would issue the advice. That'd be
- * a better simulation of true asynchronous I/O, which would only
- * start the I/O once, but isn't done here for simplicity. Note also
- * that the following call might actually issue two advice calls if we
- * cross a segment boundary; in a true asynchronous version we might
- * choose to process only one real I/O at a time in that case.
- */
- smgrprefetch(operation->smgr,
- operation->forknum,
- blockNum,
- operation->io_buffers_len);
+ /* initiate the IO asynchronously */
+ return AsyncReadBuffers(operation, io_buffers_len);
}
+ else
+ {
+ operation->flags |= READ_BUFFERS_SYNCHRONOUSLY;
+
+ if (flags & READ_BUFFERS_ISSUE_ADVICE)
+ {
+ /*
+ * In theory we should only do this if PinBufferForBlock() had to
+ * allocate new buffers above. That way, if two calls to
+ * StartReadBuffers() were made for the same blocks before
+ * WaitReadBuffers(), only the first would issue the advice.
+ * That'd be a better simulation of true asynchronous I/O, which
+ * would only start the I/O once, but isn't done here for
+ * simplicity. Note also that the following call might actually
+ * issue two advice calls if we cross a segment boundary; in a
+ * true asynchronous version we might choose to process only one
+ * real I/O at a time in that case.
+ */
+ smgrprefetch(operation->smgr,
+ operation->forknum,
+ blockNum,
+ operation->io_buffers_len);
+ }
- /* Indicate that WaitReadBuffers() should be called. */
- return true;
+ /* Indicate that WaitReadBuffers() should be called. */
+ return true;
+ }
}
/*
@@ -1397,12 +1427,31 @@ StartReadBuffer(ReadBuffersOperation *operation,
}
static inline bool
-WaitReadBuffersCanStartIO(Buffer buffer, bool nowait)
+ReadBuffersCanStartIO(Buffer buffer, bool nowait)
{
if (BufferIsLocal(buffer))
{
BufferDesc *bufHdr = GetLocalBufferDescriptor(-buffer - 1);
+ /*
+ * The buffer could have IO in progress by another scan. Right now
+ * localbuf.c doesn't use IO_IN_PROGRESS, which is why we need this
+ * hack.
+ *
+ * TODO: localbuf.c should use IO_IN_PROGRESS / have an equivalent of
+ * StartBufferIO().
+ */
+ if (pgaio_wref_valid(&bufHdr->io_wref))
+ {
+ PgAioWaitRef iow = bufHdr->io_wref;
+
+ ereport(DEBUG3,
+ errmsg("waiting for temp buffer IO in CSIO"),
+ errhidestmt(true), errhidecontext(true));
+ pgaio_wref_wait(&iow);
+ return false;
+ }
+
return (pg_atomic_read_u32(&bufHdr->state) & BM_VALID) == 0;
}
else
@@ -1412,13 +1461,38 @@ WaitReadBuffersCanStartIO(Buffer buffer, bool nowait)
void
WaitReadBuffers(ReadBuffersOperation *operation)
{
- Buffer *buffers;
+ IOContext io_context;
+ IOObject io_object;
int nblocks;
- BlockNumber blocknum;
- ForkNumber forknum;
- IOContext io_context;
- IOObject io_object;
- char persistence;
+ bool have_retryable_failure;
+
+ /*
+ * If we get here without any IO operations having been issued, the
+ * io_method == IOMETHOD_SYNC path must have been used. In that case, we
+ * start - as we used to before - the IO now, just before waiting.
+ */
+ if (operation->nios == 0)
+ {
+ Assert(io_method == IOMETHOD_SYNC);
+ if (!AsyncReadBuffers(operation, operation->io_buffers_len))
+ {
+ /* all blocks were already read in concurrently */
+ return;
+ }
+ }
+
+ if (operation->persistence == RELPERSISTENCE_TEMP)
+ {
+ io_context = IOCONTEXT_NORMAL;
+ io_object = IOOBJECT_TEMP_RELATION;
+ }
+ else
+ {
+ io_context = IOContextForStrategy(operation->strategy);
+ io_object = IOOBJECT_RELATION;
+ }
+
+restart:
/*
* Currently operations are only allowed to include a read of some range,
@@ -1433,15 +1507,101 @@ WaitReadBuffers(ReadBuffersOperation *operation)
if (nblocks == 0)
return; /* nothing to do */
- buffers = &operation->buffers[0];
- blocknum = operation->blocknum;
- forknum = operation->forknum;
- persistence = operation->persistence;
+ Assert(operation->nios > 0);
+ /*
+ * For IO timing we just count the time spent waiting for the IO.
+ *
+ * XXX: We probably should track the IO operation, rather than its time,
+ * separately, when initiating the IO. But right now that's not quite
+ * allowed by the interface.
+ */
+ have_retryable_failure = false;
+ for (int i = 0; i < operation->nios; i++)
+ {
+ PgAioReturn *aio_ret = &operation->returns[i];
+
+ /*
+ * Tracking a wait even if we don't actually need to wait a) is not
+ * cheap b) reports some time as waiting, even if we never waited.
+ */
+ if (aio_ret->result.status == ARS_UNKNOWN &&
+ !pgaio_wref_check_done(&operation->wrefs[i]))
+ {
+ instr_time io_start = pgstat_prepare_io_time(track_io_timing);
+
+ pgaio_wref_wait(&operation->wrefs[i]);
+
+ /*
+ * The IO operation itself was already counted earlier, in
+ * AsyncReadBuffers().
+ */
+ pgstat_count_io_op_time(io_object, io_context, IOOP_READ,
+ io_start, 0, 0);
+ }
+ else
+ {
+ Assert(pgaio_wref_check_done(&operation->wrefs[i]));
+ }
+
+ if (aio_ret->result.status == ARS_PARTIAL)
+ {
+ /*
+ * We'll retry below, so we just emit a debug message the server
+ * log (or not even that in prod scenarios).
+ */
+ pgaio_result_report(aio_ret->result, &aio_ret->target_data, DEBUG1);
+ have_retryable_failure = true;
+ }
+ else if (aio_ret->result.status != ARS_OK)
+ pgaio_result_report(aio_ret->result, &aio_ret->target_data, ERROR);
+ }
+
+ /*
+ * If any of the associated IOs failed, try again to issue IOs. Buffers
+ * for which IO has completed successfully will be discovered as such and
+ * not retried.
+ */
+ if (have_retryable_failure)
+ {
+ nblocks = operation->io_buffers_len;
+
+ elog(DEBUG3, "retrying IO after partial failure");
+ CHECK_FOR_INTERRUPTS();
+ AsyncReadBuffers(operation, nblocks);
+ goto restart;
+ }
+
+ if (VacuumCostActive)
+ VacuumCostBalance += VacuumCostPageMiss * nblocks;
+
+ /* FIXME: READ_DONE tracepoint */
+}
+
+static bool
+AsyncReadBuffers(ReadBuffersOperation *operation,
+ int nblocks)
+{
+ int io_buffers_len = 0;
+ Buffer *buffers = &operation->buffers[0];
+ int flags = operation->flags;
+ BlockNumber blocknum = operation->blocknum;
+ ForkNumber forknum = operation->forknum;
+ IOContext io_context;
+ IOObject io_object;
+ char persistence;
+ bool did_start_io_overall = false;
+ PgAioHandle *ioh = NULL;
+ uint32 ioh_flags = 0;
+
+ persistence = operation->rel
+ ? operation->rel->rd_rel->relpersistence
+ : RELPERSISTENCE_PERMANENT;
if (persistence == RELPERSISTENCE_TEMP)
{
io_context = IOCONTEXT_NORMAL;
io_object = IOOBJECT_TEMP_RELATION;
+ ioh_flags |= PGAIO_HF_REFERENCES_LOCAL;
}
else
{
@@ -1449,6 +1609,16 @@ WaitReadBuffers(ReadBuffersOperation *operation)
io_object = IOOBJECT_RELATION;
}
+ /*
+ * When this IO is executed synchronously, either because the caller will
+ * immediately block waiting for the IO or because IOMETHOD_SYNC is used,
+ * the AIO subsystem needs to know.
+ */
+ if (flags & READ_BUFFERS_SYNCHRONOUSLY)
+ ioh_flags |= PGAIO_HF_SYNCHRONOUS;
+
+ operation->nios = 0;
+
/*
* We count all these blocks as read by this backend. This is traditional
* behavior, but might turn out to be not true if we find that someone
@@ -1464,19 +1634,39 @@ WaitReadBuffers(ReadBuffersOperation *operation)
for (int i = 0; i < nblocks; ++i)
{
- int io_buffers_len;
- Buffer io_buffers[MAX_IO_COMBINE_LIMIT];
void *io_pages[MAX_IO_COMBINE_LIMIT];
- instr_time io_start;
+ Buffer io_buffers[MAX_IO_COMBINE_LIMIT];
BlockNumber io_first_block;
+ bool did_start_io_this = false;
/*
- * Skip this block if someone else has already completed it. If an
- * I/O is already in progress in another backend, this will wait for
- * the outcome: either done, or something went wrong and we will
- * retry.
+ * Get IO before ReadBuffersCanStartIO, as pgaio_io_acquire() might
+ * block, which we don't want after setting IO_IN_PROGRESS.
+ *
+ * XXX: Should we attribute the time spent in here to the IO? If there
+ * already are a lot of IO operations in progress, getting an IO
+ * handle will block waiting for some other IO operation to finish.
+ *
+ * In most cases it'll be free to get the IO, so a timer would be
+ * overhead. Perhaps we should use pgaio_io_acquire_nb() and only
+ * account IO time when pgaio_io_acquire_nb() returned false?
*/
- if (!WaitReadBuffersCanStartIO(buffers[i], false))
+ if (likely(!ioh))
+ ioh = pgaio_io_acquire(CurrentResourceOwner,
+ &operation->returns[operation->nios]);
+
+ /*
+ * Skip this block if someone else has already completed it.
+ *
+ * If an I/O is already in progress in another backend, this will wait
+ * for the outcome: either done, or something went wrong and we will
+ * retry. But don't wait if we have staged, but haven't issued,
+ * another IO.
+ *
+ * XXX: If we can't start IO due to unsubmitted IO, it might be worth
+ * to submit and then try to start IO again.
+ */
+ if (!ReadBuffersCanStartIO(buffers[i], did_start_io_overall))
{
/*
* Report this as a 'hit' for this backend, even though it must
@@ -1488,6 +1678,11 @@ WaitReadBuffers(ReadBuffersOperation *operation)
operation->smgr->smgr_rlocator.locator.relNumber,
operation->smgr->smgr_rlocator.backend,
true);
+
+ ereport(DEBUG3,
+ errmsg("can't start io for first buffer %u: %s",
+ buffers[i], DebugPrintBufferRefcount(buffers[i])),
+ errhidestmt(true), errhidecontext(true));
continue;
}
@@ -1497,6 +1692,11 @@ WaitReadBuffers(ReadBuffersOperation *operation)
io_first_block = blocknum + i;
io_buffers_len = 1;
+ ereport(DEBUG5,
+ errmsg("first prepped for io: %s, offset %d",
+ DebugPrintBufferRefcount(io_buffers[0]), i),
+ errhidestmt(true), errhidecontext(true));
+
/*
* How many neighboring-on-disk blocks can we scatter-read into other
* buffers at the same time? In this case we don't wait if we see an
@@ -1505,85 +1705,58 @@ WaitReadBuffers(ReadBuffersOperation *operation)
* We'll come back to this block again, above.
*/
while ((i + 1) < nblocks &&
- WaitReadBuffersCanStartIO(buffers[i + 1], true))
+ ReadBuffersCanStartIO(buffers[i + 1], true))
{
/* Must be consecutive block numbers. */
Assert(BufferGetBlockNumber(buffers[i + 1]) ==
BufferGetBlockNumber(buffers[i]) + 1);
+ ereport(DEBUG5,
+ errmsg("seq prepped for io: %s, offset %d",
+ DebugPrintBufferRefcount(buffers[i + 1]),
+ i + 1),
+ errhidestmt(true), errhidecontext(true));
+
io_buffers[io_buffers_len] = buffers[++i];
io_pages[io_buffers_len++] = BufferGetBlock(buffers[i]);
}
- io_start = pgstat_prepare_io_time(track_io_timing);
- smgrreadv(operation->smgr, forknum, io_first_block, io_pages, io_buffers_len);
- pgstat_count_io_op_time(io_object, io_context, IOOP_READ, io_start,
- 1, io_buffers_len * BLCKSZ);
+ pgaio_io_get_wref(ioh, &operation->wrefs[operation->nios]);
- /* Verify each block we read, and terminate the I/O. */
- for (int j = 0; j < io_buffers_len; ++j)
- {
- BufferDesc *bufHdr;
- Block bufBlock;
+ pgaio_io_set_handle_data_32(ioh, (uint32 *) io_buffers, io_buffers_len);
- if (persistence == RELPERSISTENCE_TEMP)
- {
- bufHdr = GetLocalBufferDescriptor(-io_buffers[j] - 1);
- bufBlock = LocalBufHdrGetBlock(bufHdr);
- }
- else
- {
- bufHdr = GetBufferDescriptor(io_buffers[j] - 1);
- bufBlock = BufHdrGetBlock(bufHdr);
- }
- /* check for garbage data */
- if (!PageIsVerifiedExtended((Page) bufBlock, io_first_block + j,
- PIV_LOG_WARNING | PIV_REPORT_STAT))
- {
- if ((operation->flags & READ_BUFFERS_ZERO_ON_ERROR) || zero_damaged_pages)
- {
- ereport(WARNING,
- (errcode(ERRCODE_DATA_CORRUPTED),
- errmsg("invalid page in block %u of relation %s; zeroing out page",
- io_first_block + j,
- relpath(operation->smgr->smgr_rlocator, forknum))));
- memset(bufBlock, 0, BLCKSZ);
- }
- else
- ereport(ERROR,
- (errcode(ERRCODE_DATA_CORRUPTED),
- errmsg("invalid page in block %u of relation %s",
- io_first_block + j,
- relpath(operation->smgr->smgr_rlocator, forknum))));
- }
+ if (persistence == RELPERSISTENCE_TEMP)
+ pgaio_io_register_callbacks(ioh, PGAIO_HCB_LOCAL_BUFFER_READV);
+ else
+ pgaio_io_register_callbacks(ioh, PGAIO_HCB_SHARED_BUFFER_READV);
- /* Terminate I/O and set BM_VALID. */
- if (persistence == RELPERSISTENCE_TEMP)
- {
- uint32 buf_state = pg_atomic_read_u32(&bufHdr->state);
+ pgaio_io_set_flag(ioh, ioh_flags);
- buf_state |= BM_VALID;
- pg_atomic_unlocked_write_u32(&bufHdr->state, buf_state);
- }
- else
- {
- /* Set BM_VALID, terminate IO, and wake up any waiters */
- TerminateBufferIO(bufHdr, false, BM_VALID, true, true);
- }
+ did_start_io_overall = did_start_io_this = true;
+ smgrstartreadv(ioh, operation->smgr, forknum, io_first_block,
+ io_pages, io_buffers_len);
+ ioh = NULL;
+ operation->nios++;
- /* Report I/Os as completing individually. */
- TRACE_POSTGRESQL_BUFFER_READ_DONE(forknum, io_first_block + j,
- operation->smgr->smgr_rlocator.locator.spcOid,
- operation->smgr->smgr_rlocator.locator.dbOid,
- operation->smgr->smgr_rlocator.locator.relNumber,
- operation->smgr->smgr_rlocator.backend,
- false);
- }
+ /* not obvious what we'd use for time */
+ pgstat_count_io_op(io_object, io_context, IOOP_READ,
+ 1, io_buffers_len * BLCKSZ);
+ }
+
+ if (ioh)
+ {
+ pgaio_io_release(ioh);
+ ioh = NULL;
+ }
- if (VacuumCostActive)
- VacuumCostBalance += VacuumCostPageMiss * io_buffers_len;
+ if (did_start_io_overall)
+ {
+ pgaio_submit_staged();
+ return true;
}
+ else
+ return false;
}
/*
--
2.48.1.76.g4e746b1a31.dirty
[text/x-diff] v2.3-0022-aio-Very-WIP-read_stream.c-adjustments-for-real.patch (5.0K, ../../p5flybfcjrzgpgwcnzkxsp3xdw3czfoafkvks6g7en6ewmvymh@wrz7z2vwwc7c/23-v2.3-0022-aio-Very-WIP-read_stream.c-adjustments-for-real.patch)
download | inline diff:
From b0bb4b478b27c2a38bf819ee927be9167e551d28 Mon Sep 17 00:00:00 2001
From: Andres Freund <andres@anarazel.de>
Date: Sat, 31 Aug 2024 21:39:30 -0400
Subject: [PATCH v2.3 22/30] aio: Very-WIP: read_stream.c adjustments for real
AIO
Things that need to be fixed / are fixed in this:
- max pinned buffers should be limited by io_combine_limit, not * 4
- overflow distance
- pins need to be limited in more places
---
src/include/storage/bufmgr.h | 2 ++
src/backend/storage/aio/read_stream.c | 31 +++++++++++++++++++++------
src/backend/storage/buffer/bufmgr.c | 3 ++-
3 files changed, 28 insertions(+), 8 deletions(-)
diff --git a/src/include/storage/bufmgr.h b/src/include/storage/bufmgr.h
index 46ee957e99c..f205643c4ef 100644
--- a/src/include/storage/bufmgr.h
+++ b/src/include/storage/bufmgr.h
@@ -119,6 +119,8 @@ typedef struct BufferManagerRelation
#define READ_BUFFERS_ISSUE_ADVICE (1 << 1)
/* IO will immediately be waited for */
#define READ_BUFFERS_SYNCHRONOUSLY (1 << 2)
+/* caller will issue more io, don't submit */
+#define READ_BUFFERS_MORE_MORE_MORE (1 << 3)
struct ReadBuffersOperation
diff --git a/src/backend/storage/aio/read_stream.c b/src/backend/storage/aio/read_stream.c
index e4414b2e915..c2211cab02a 100644
--- a/src/backend/storage/aio/read_stream.c
+++ b/src/backend/storage/aio/read_stream.c
@@ -90,6 +90,7 @@
#include "postgres.h"
#include "miscadmin.h"
+#include "storage/aio.h"
#include "storage/fd.h"
#include "storage/smgr.h"
#include "storage/read_stream.h"
@@ -240,14 +241,18 @@ read_stream_start_pending_read(ReadStream *stream, bool suppress_advice)
/*
* If advice hasn't been suppressed, this system supports it, and this
* isn't a strictly sequential pattern, then we'll issue advice.
+ *
+ * XXX: Used to also check stream->pending_read_blocknum !=
+ * stream->seq_blocknum
*/
if (!suppress_advice &&
- stream->advice_enabled &&
- stream->pending_read_blocknum != stream->seq_blocknum)
+ stream->advice_enabled)
flags = READ_BUFFERS_ISSUE_ADVICE;
else
flags = 0;
+ flags |= READ_BUFFERS_MORE_MORE_MORE;
+
/* We say how many blocks we want to read, but may be smaller on return. */
buffer_index = stream->next_buffer_index;
io_index = stream->next_io_index;
@@ -306,6 +311,14 @@ read_stream_start_pending_read(ReadStream *stream, bool suppress_advice)
static void
read_stream_look_ahead(ReadStream *stream, bool suppress_advice)
{
+ if (stream->distance > (io_combine_limit * 8))
+ {
+ if (stream->pinned_buffers + stream->pending_read_nblocks > ((stream->distance * 3) / 4))
+ {
+ return;
+ }
+ }
+
while (stream->ios_in_progress < stream->max_ios &&
stream->pinned_buffers + stream->pending_read_nblocks < stream->distance)
{
@@ -355,6 +368,7 @@ read_stream_look_ahead(ReadStream *stream, bool suppress_advice)
{
/* And we've hit the limit. Rewind, and stop here. */
read_stream_unget_block(stream, blocknum);
+ pgaio_submit_staged();
return;
}
}
@@ -379,6 +393,8 @@ read_stream_look_ahead(ReadStream *stream, bool suppress_advice)
stream->distance == 0) &&
stream->ios_in_progress < stream->max_ios)
read_stream_start_pending_read(stream, suppress_advice);
+
+ pgaio_submit_staged();
}
/*
@@ -442,7 +458,7 @@ read_stream_begin_impl(int flags,
* overflow (even though that's not possible with the current GUC range
* limits), allowing also for the spare entry and the overflow space.
*/
- max_pinned_buffers = Max(max_ios * 4, io_combine_limit);
+ max_pinned_buffers = Max(max_ios * io_combine_limit, io_combine_limit);
max_pinned_buffers = Min(max_pinned_buffers,
PG_INT16_MAX - io_combine_limit - 1);
@@ -493,10 +509,11 @@ read_stream_begin_impl(int flags,
* direct I/O isn't enabled, the caller hasn't promised sequential access
* (overriding our detection heuristics), and max_ios hasn't been set to
* zero.
+ *
+ * FIXME: Used to also check (io_direct_flags & IO_DIRECT_DATA) == 0 &&
+ * (flags & READ_STREAM_SEQUENTIAL) == 0
*/
- if ((io_direct_flags & IO_DIRECT_DATA) == 0 &&
- (flags & READ_STREAM_SEQUENTIAL) == 0 &&
- max_ios > 0)
+ if (max_ios > 0)
stream->advice_enabled = true;
#endif
@@ -727,7 +744,7 @@ read_stream_next_buffer(ReadStream *stream, void **per_buffer_data)
if (++stream->oldest_io_index == stream->max_ios)
stream->oldest_io_index = 0;
- if (stream->ios[io_index].op.flags & READ_BUFFERS_ISSUE_ADVICE)
+ if (stream->ios[io_index].op.flags & (READ_BUFFERS_ISSUE_ADVICE | READ_BUFFERS_MORE_MORE_MORE))
{
/* Distance ramps up fast (behavior C). */
distance = stream->distance * 2;
diff --git a/src/backend/storage/buffer/bufmgr.c b/src/backend/storage/buffer/bufmgr.c
index 70f1da84083..118a6e1ca31 100644
--- a/src/backend/storage/buffer/bufmgr.c
+++ b/src/backend/storage/buffer/bufmgr.c
@@ -1752,7 +1752,8 @@ AsyncReadBuffers(ReadBuffersOperation *operation,
if (did_start_io_overall)
{
- pgaio_submit_staged();
+ if (!(flags & READ_BUFFERS_MORE_MORE_MORE))
+ pgaio_submit_staged();
return true;
}
else
--
2.48.1.76.g4e746b1a31.dirty
[text/x-diff] v2.3-0023-aio-Add-bounce-buffers.patch (20.8K, ../../p5flybfcjrzgpgwcnzkxsp3xdw3czfoafkvks6g7en6ewmvymh@wrz7z2vwwc7c/24-v2.3-0023-aio-Add-bounce-buffers.patch)
download | inline diff:
From 398eabe5a83ff4ed74141b7803b7f6b23d0a3bdd Mon Sep 17 00:00:00 2001
From: Andres Freund <andres@anarazel.de>
Date: Wed, 22 Jan 2025 17:13:46 -0500
Subject: [PATCH v2.3 23/30] aio: Add bounce buffers
---
src/include/storage/aio.h | 19 ++
src/include/storage/aio_internal.h | 33 ++++
src/include/utils/resowner.h | 2 +
src/backend/storage/aio/README.md | 27 +++
src/backend/storage/aio/aio.c | 180 ++++++++++++++++++
src/backend/storage/aio/aio_init.c | 123 ++++++++++++
src/backend/utils/misc/guc_tables.c | 13 ++
src/backend/utils/misc/postgresql.conf.sample | 2 +
src/backend/utils/resowner/resowner.c | 25 ++-
src/tools/pgindent/typedefs.list | 1 +
10 files changed, 423 insertions(+), 2 deletions(-)
diff --git a/src/include/storage/aio.h b/src/include/storage/aio.h
index 6f36a0b9e4d..30b08495f3d 100644
--- a/src/include/storage/aio.h
+++ b/src/include/storage/aio.h
@@ -247,6 +247,10 @@ typedef struct PgAioHandleCallbacks
+typedef struct PgAioBounceBuffer PgAioBounceBuffer;
+
+
+
/* AIO API */
@@ -330,6 +334,20 @@ extern bool pgaio_have_staged(void);
+/* --------------------------------------------------------------------------------
+ * Bounce Buffers
+ * --------------------------------------------------------------------------------
+ */
+
+extern PgAioBounceBuffer *pgaio_bounce_buffer_get(void);
+extern void pgaio_io_assoc_bounce_buffer(PgAioHandle *ioh, PgAioBounceBuffer *bb);
+extern uint32 pgaio_bounce_buffer_id(PgAioBounceBuffer *bb);
+extern void pgaio_bounce_buffer_release(PgAioBounceBuffer *bb);
+extern char *pgaio_bounce_buffer_buffer(PgAioBounceBuffer *bb);
+extern void pgaio_bounce_buffer_release_resowner(struct dlist_node *bb_node, bool on_error);
+
+
+
/* --------------------------------------------------------------------------------
* Other
* --------------------------------------------------------------------------------
@@ -345,6 +363,7 @@ extern void assign_io_method(int newval, void *extra);
/* GUCs */
extern PGDLLIMPORT int io_method;
extern PGDLLIMPORT int io_max_concurrency;
+extern PGDLLIMPORT int io_bounce_buffers;
#endif /* AIO_H */
diff --git a/src/include/storage/aio_internal.h b/src/include/storage/aio_internal.h
index eff544ce621..531532e306a 100644
--- a/src/include/storage/aio_internal.h
+++ b/src/include/storage/aio_internal.h
@@ -97,6 +97,12 @@ struct PgAioHandle
*/
uint32 iovec_off;
+ /*
+ * List of bounce_buffers owned by IO. It would suffice to use an index
+ * based linked list here.
+ */
+ slist_head bounce_buffers;
+
/**
* In which list the handle is registered, depends on the state:
* - IDLE, in per-backend list
@@ -133,11 +139,23 @@ struct PgAioHandle
};
+struct PgAioBounceBuffer
+{
+ slist_node node;
+ struct ResourceOwnerData *resowner;
+ dlist_node resowner_node;
+ char *buffer;
+};
+
+
typedef struct PgAioBackend
{
/* index into PgAioCtl->io_handles */
uint32 io_handle_off;
+ /* index into PgAioCtl->bounce_buffers */
+ uint32 bounce_buffers_off;
+
/* IO Handles that currently are not used */
dclist_head idle_ios;
@@ -165,6 +183,12 @@ typedef struct PgAioBackend
* IOs being appended at the end.
*/
dclist_head in_flight_ios;
+
+ /* Bounce Buffers that currently are not used */
+ slist_head idle_bbs;
+
+ /* see handed_out_io */
+ PgAioBounceBuffer *handed_out_bb;
} PgAioBackend;
@@ -190,6 +214,15 @@ typedef struct PgAioCtl
*/
uint64 *handle_data;
+ /*
+ * To perform AIO on buffers that are not located in shared memory (either
+ * because they are not in shared memory or because we need to operate on
+ * a copy, as e.g. the case for writes when checksums are in use)
+ */
+ uint64 bounce_buffers_count;
+ PgAioBounceBuffer *bounce_buffers;
+ char *bounce_buffers_data;
+
uint64 io_handle_count;
PgAioHandle *io_handles;
} PgAioCtl;
diff --git a/src/include/utils/resowner.h b/src/include/utils/resowner.h
index aede4bfc820..7e2ec224169 100644
--- a/src/include/utils/resowner.h
+++ b/src/include/utils/resowner.h
@@ -168,5 +168,7 @@ extern void ResourceOwnerForgetLock(ResourceOwner owner, struct LOCALLOCK *local
struct dlist_node;
extern void ResourceOwnerRememberAioHandle(ResourceOwner owner, struct dlist_node *ioh_node);
extern void ResourceOwnerForgetAioHandle(ResourceOwner owner, struct dlist_node *ioh_node);
+extern void ResourceOwnerRememberAioBounceBuffer(ResourceOwner owner, struct dlist_node *bb_node);
+extern void ResourceOwnerForgetAioBounceBuffer(ResourceOwner owner, struct dlist_node *bb_node);
#endif /* RESOWNER_H */
diff --git a/src/backend/storage/aio/README.md b/src/backend/storage/aio/README.md
index 1b6f9d2c40b..dacff46ad12 100644
--- a/src/backend/storage/aio/README.md
+++ b/src/backend/storage/aio/README.md
@@ -412,6 +412,33 @@ shared memory no less!), completion callbacks instead have to encode errors in
a more compact format that can be converted into an error message.
+### AIO Bounce Buffers
+
+For some uses of AIO there is no convenient memory location as the source /
+destination of an AIO. E.g. when data checksums are enabled, writes from
+shared buffers currently cannot be done directly from shared buffers, as a
+shared buffer lock still allows some modification, e.g., for hint bits(see
+`FlushBuffer()`). If the write were done in-place, such modifications can
+cause the checksum to fail.
+
+For synchronous IO this is solved by copying the buffer to separate memory
+before computing the checksum and using that copy as the source buffer for the
+AIO.
+
+However, for AIO that is not a workable solution:
+- Instead of a single buffer many buffers are required, as many IOs might be
+ in flight
+- When using the [worker method](#worker), the source/target of IO needs to be
+ in shared memory, otherwise the workers won't be able to access the memory.
+
+The AIO subsystem addresses this by providing a limited number of bounce
+buffers that can be used as the source / target for IO. A bounce buffer be
+acquired with `pgaio_bounce_buffer_get()` and multiple bounce buffers can be
+associated with an AIO Handle with `pgaio_io_assoc_bounce_buffer()`.
+
+Bounce buffers are automatically released when the IO completes.
+
+
## Helpers
Using the low-level AIO API introduces too much complexity to do so all over
diff --git a/src/backend/storage/aio/aio.c b/src/backend/storage/aio/aio.c
index b3b4e74c3ce..431f2c2e5af 100644
--- a/src/backend/storage/aio/aio.c
+++ b/src/backend/storage/aio/aio.c
@@ -55,6 +55,8 @@ static PgAioHandle *pgaio_io_from_wref(PgAioWaitRef *iow, uint64 *ref_generation
static const char *pgaio_io_state_get_name(PgAioHandleState s);
static void pgaio_io_wait(PgAioHandle *ioh, uint64 ref_generation);
+static void pgaio_bounce_buffer_wait_for_free(void);
+
/* Options for io_method. */
const struct config_enum_entry io_method_options[] = {
@@ -69,6 +71,7 @@ const struct config_enum_entry io_method_options[] = {
/* GUCs */
int io_method = DEFAULT_IO_METHOD;
int io_max_concurrency = -1;
+int io_bounce_buffers = -1;
/* global control for AIO */
PgAioCtl *pgaio_ctl;
@@ -588,6 +591,21 @@ pgaio_io_reclaim(PgAioHandle *ioh)
}
}
+ /* reclaim all associated bounce buffers */
+ if (!slist_is_empty(&ioh->bounce_buffers))
+ {
+ slist_mutable_iter it;
+
+ slist_foreach_modify(it, &ioh->bounce_buffers)
+ {
+ PgAioBounceBuffer *bb = slist_container(PgAioBounceBuffer, node, it.cur);
+
+ slist_delete_current(&it);
+
+ slist_push_head(&pgaio_my_backend->idle_bbs, &bb->node);
+ }
+ }
+
if (ioh->resowner)
{
ResourceOwnerForgetAioHandle(ioh->resowner, &ioh->resowner_node);
@@ -874,6 +892,166 @@ pgaio_have_staged(void)
+/* --------------------------------------------------------------------------------
+ * Functions primarily related to PgAioBounceBuffer
+ * --------------------------------------------------------------------------------
+ */
+
+PgAioBounceBuffer *
+pgaio_bounce_buffer_get(void)
+{
+ PgAioBounceBuffer *bb = NULL;
+ slist_node *node;
+
+ if (pgaio_my_backend->handed_out_bb != NULL)
+ elog(ERROR, "can only hand out one BB");
+
+ /*
+ * FIXME It probably is not correct to have bounce buffers be per backend,
+ * they use too much memory.
+ */
+ if (slist_is_empty(&pgaio_my_backend->idle_bbs))
+ {
+ pgaio_bounce_buffer_wait_for_free();
+ }
+
+ node = slist_pop_head_node(&pgaio_my_backend->idle_bbs);
+ bb = slist_container(PgAioBounceBuffer, node, node);
+
+ pgaio_my_backend->handed_out_bb = bb;
+
+ bb->resowner = CurrentResourceOwner;
+ ResourceOwnerRememberAioBounceBuffer(bb->resowner, &bb->resowner_node);
+
+ return bb;
+}
+
+void
+pgaio_io_assoc_bounce_buffer(PgAioHandle *ioh, PgAioBounceBuffer *bb)
+{
+ if (pgaio_my_backend->handed_out_bb != bb)
+ elog(ERROR, "can only assign handed out BB");
+ pgaio_my_backend->handed_out_bb = NULL;
+
+ /*
+ * There can be many bounce buffers assigned in case of vectorized IOs.
+ */
+ slist_push_head(&ioh->bounce_buffers, &bb->node);
+
+ /* once associated with an IO, the IO has ownership */
+ ResourceOwnerForgetAioBounceBuffer(bb->resowner, &bb->resowner_node);
+ bb->resowner = NULL;
+}
+
+uint32
+pgaio_bounce_buffer_id(PgAioBounceBuffer *bb)
+{
+ return bb - pgaio_ctl->bounce_buffers;
+}
+
+void
+pgaio_bounce_buffer_release(PgAioBounceBuffer *bb)
+{
+ if (pgaio_my_backend->handed_out_bb != bb)
+ elog(ERROR, "can only release handed out BB");
+
+ slist_push_head(&pgaio_my_backend->idle_bbs, &bb->node);
+ pgaio_my_backend->handed_out_bb = NULL;
+
+ ResourceOwnerForgetAioBounceBuffer(bb->resowner, &bb->resowner_node);
+ bb->resowner = NULL;
+}
+
+void
+pgaio_bounce_buffer_release_resowner(dlist_node *bb_node, bool on_error)
+{
+ PgAioBounceBuffer *bb = dlist_container(PgAioBounceBuffer, resowner_node, bb_node);
+
+ Assert(bb->resowner);
+
+ if (!on_error)
+ elog(WARNING, "leaked AIO bounce buffer");
+
+ pgaio_bounce_buffer_release(bb);
+}
+
+char *
+pgaio_bounce_buffer_buffer(PgAioBounceBuffer *bb)
+{
+ return bb->buffer;
+}
+
+static void
+pgaio_bounce_buffer_wait_for_free(void)
+{
+ static uint32 lastpos = 0;
+
+ if (pgaio_my_backend->num_staged_ios > 0)
+ {
+ pgaio_debug(DEBUG2, "submitting %d, while acquiring free bb",
+ pgaio_my_backend->num_staged_ios);
+ pgaio_submit_staged();
+ }
+
+ for (uint32 i = lastpos; i < lastpos + io_max_concurrency; i++)
+ {
+ uint32 thisoff = pgaio_my_backend->io_handle_off + (i % io_max_concurrency);
+ PgAioHandle *ioh = &pgaio_ctl->io_handles[thisoff];
+
+ switch (ioh->state)
+ {
+ case PGAIO_HS_IDLE:
+ case PGAIO_HS_HANDED_OUT:
+ continue;
+ case PGAIO_HS_DEFINED: /* should have been submitted above */
+ case PGAIO_HS_STAGED:
+ elog(ERROR, "shouldn't get here with io:%d in state %d",
+ pgaio_io_get_id(ioh), ioh->state);
+ break;
+ case PGAIO_HS_COMPLETED_IO:
+ case PGAIO_HS_SUBMITTED:
+ if (!slist_is_empty(&ioh->bounce_buffers))
+ {
+ pgaio_debug_io(DEBUG2, ioh,
+ "waiting for IO to reclaim BB with %d in flight",
+ dclist_count(&pgaio_my_backend->in_flight_ios));
+
+ /* see comment in pgaio_io_wait_for_free() about raciness */
+ pgaio_io_wait(ioh, ioh->generation);
+
+ if (slist_is_empty(&pgaio_my_backend->idle_bbs))
+ elog(WARNING, "empty after wait");
+
+ if (!slist_is_empty(&pgaio_my_backend->idle_bbs))
+ {
+ lastpos = i;
+ return;
+ }
+ }
+ break;
+ case PGAIO_HS_COMPLETED_SHARED:
+ case PGAIO_HS_COMPLETED_LOCAL:
+ /* reclaim */
+ pgaio_io_reclaim(ioh);
+
+ if (!slist_is_empty(&pgaio_my_backend->idle_bbs))
+ {
+ lastpos = i;
+ return;
+ }
+ break;
+ }
+ }
+
+ /*
+ * The submission above could have caused the IO to complete at any time.
+ */
+ if (slist_is_empty(&pgaio_my_backend->idle_bbs))
+ elog(PANIC, "no more bbs");
+}
+
+
+
/* --------------------------------------------------------------------------------
* Other
* --------------------------------------------------------------------------------
@@ -904,6 +1082,7 @@ void
pgaio_at_xact_end(bool is_subxact, bool is_commit)
{
Assert(!pgaio_my_backend->handed_out_io);
+ Assert(!pgaio_my_backend->handed_out_bb);
}
/*
@@ -914,6 +1093,7 @@ void
pgaio_at_error(void)
{
Assert(!pgaio_my_backend->handed_out_io);
+ Assert(!pgaio_my_backend->handed_out_bb);
}
void
diff --git a/src/backend/storage/aio/aio_init.c b/src/backend/storage/aio/aio_init.c
index 76fcdf64670..a4f4a0b698e 100644
--- a/src/backend/storage/aio/aio_init.c
+++ b/src/backend/storage/aio/aio_init.c
@@ -82,6 +82,32 @@ AioHandleDataShmemSize(void)
io_max_concurrency));
}
+static Size
+AioBounceBufferDescShmemSize(void)
+{
+ Size sz;
+
+ /* PgAioBounceBuffer itself */
+ sz = mul_size(sizeof(PgAioBounceBuffer),
+ mul_size(AioProcs(), io_bounce_buffers));
+
+ return sz;
+}
+
+static Size
+AioBounceBufferDataShmemSize(void)
+{
+ Size sz;
+
+ /* and the associated buffer */
+ sz = mul_size(BLCKSZ,
+ mul_size(io_bounce_buffers, AioProcs()));
+ /* memory for alignment */
+ sz += BLCKSZ;
+
+ return sz;
+}
+
/*
* Choose a suitable value for io_max_concurrency.
*
@@ -107,6 +133,33 @@ AioChooseMaxConccurrency(void)
return Min(max_proportional_pins, 64);
}
+/*
+ * Choose a suitable value for io_bounce_buffers.
+ *
+ * It's very unlikely to be useful to allocate more bounce buffers for each
+ * backend than the backend is allowed to pin. Additionally, bounce buffers
+ * currently are used for writes, it seems very uncommon for more than 10% of
+ * shared_buffers to be written out concurrently.
+ *
+ * XXX: This quickly can take up significant amounts of memory, the logic
+ * should probably fine tuned.
+ */
+static int
+AioChooseBounceBuffers(void)
+{
+ uint32 max_backends;
+ int max_proportional_pins;
+
+ /* Similar logic to LimitAdditionalPins() */
+ max_backends = MaxBackends + NUM_AUXILIARY_PROCS;
+ max_proportional_pins = (NBuffers / 10) / max_backends;
+
+ max_proportional_pins = Max(max_proportional_pins, 1);
+
+ /* apply upper limit */
+ return Min(max_proportional_pins, 256);
+}
+
Size
AioShmemSize(void)
{
@@ -130,11 +183,31 @@ AioShmemSize(void)
PGC_S_OVERRIDE);
}
+
+ /*
+ * If io_bounce_buffers is -1, we automatically choose a suitable value.
+ *
+ * See also comment above.
+ */
+ if (io_bounce_buffers == -1)
+ {
+ char buf[32];
+
+ snprintf(buf, sizeof(buf), "%d", AioChooseBounceBuffers());
+ SetConfigOption("io_bounce_buffers", buf, PGC_POSTMASTER,
+ PGC_S_DYNAMIC_DEFAULT);
+ if (io_bounce_buffers == -1) /* failed to apply it? */
+ SetConfigOption("io_bounce_buffers", buf, PGC_POSTMASTER,
+ PGC_S_OVERRIDE);
+ }
+
sz = add_size(sz, AioCtlShmemSize());
sz = add_size(sz, AioBackendShmemSize());
sz = add_size(sz, AioHandleShmemSize());
sz = add_size(sz, AioHandleIOVShmemSize());
sz = add_size(sz, AioHandleDataShmemSize());
+ sz = add_size(sz, AioBounceBufferDescShmemSize());
+ sz = add_size(sz, AioBounceBufferDataShmemSize());
if (pgaio_method_ops->shmem_size)
sz = add_size(sz, pgaio_method_ops->shmem_size());
@@ -149,6 +222,9 @@ AioShmemInit(void)
uint32 io_handle_off = 0;
uint32 iovec_off = 0;
uint32 per_backend_iovecs = io_max_concurrency * PG_IOV_MAX;
+ uint32 bounce_buffers_off = 0;
+ uint32 per_backend_bb = io_bounce_buffers;
+ char *bounce_buffers_data;
pgaio_ctl = (PgAioCtl *)
ShmemInitStruct("AioCtl", AioCtlShmemSize(), &found);
@@ -160,6 +236,7 @@ AioShmemInit(void)
pgaio_ctl->io_handle_count = AioProcs() * io_max_concurrency;
pgaio_ctl->iovec_count = AioProcs() * per_backend_iovecs;
+ pgaio_ctl->bounce_buffers_count = AioProcs() * per_backend_bb;
pgaio_ctl->backend_state = (PgAioBackend *)
ShmemInitStruct("AioBackend", AioBackendShmemSize(), &found);
@@ -172,6 +249,40 @@ AioShmemInit(void)
pgaio_ctl->handle_data = (uint64 *)
ShmemInitStruct("AioHandleData", AioHandleDataShmemSize(), &found);
+ pgaio_ctl->bounce_buffers = (PgAioBounceBuffer *)
+ ShmemInitStruct("AioBounceBufferDesc", AioBounceBufferDescShmemSize(),
+ &found);
+
+ bounce_buffers_data =
+ ShmemInitStruct("AioBounceBufferData", AioBounceBufferDataShmemSize(),
+ &found);
+ bounce_buffers_data =
+ (char *) TYPEALIGN(BLCKSZ, (uintptr_t) bounce_buffers_data);
+ pgaio_ctl->bounce_buffers_data = bounce_buffers_data;
+
+
+ /* Initialize IO handles. */
+ for (uint64 i = 0; i < pgaio_ctl->io_handle_count; i++)
+ {
+ PgAioHandle *ioh = &pgaio_ctl->io_handles[i];
+
+ ioh->op = PGAIO_OP_INVALID;
+ ioh->target = PGAIO_TID_INVALID;
+ ioh->state = PGAIO_HS_IDLE;
+
+ slist_init(&ioh->bounce_buffers);
+ }
+
+ /* Initialize Bounce Buffers. */
+ for (uint64 i = 0; i < pgaio_ctl->bounce_buffers_count; i++)
+ {
+ PgAioBounceBuffer *bb = &pgaio_ctl->bounce_buffers[i];
+
+ bb->buffer = bounce_buffers_data;
+ bounce_buffers_data += BLCKSZ;
+ }
+
+
for (int procno = 0; procno < AioProcs(); procno++)
{
PgAioBackend *bs = &pgaio_ctl->backend_state[procno];
@@ -179,9 +290,13 @@ AioShmemInit(void)
bs->io_handle_off = io_handle_off;
io_handle_off += io_max_concurrency;
+ bs->bounce_buffers_off = bounce_buffers_off;
+ bounce_buffers_off += per_backend_bb;
+
dclist_init(&bs->idle_ios);
memset(bs->staged_ios, 0, sizeof(PgAioHandle *) * PGAIO_SUBMIT_BATCH_SIZE);
dclist_init(&bs->in_flight_ios);
+ slist_init(&bs->idle_bbs);
/* initialize per-backend IOs */
for (int i = 0; i < io_max_concurrency; i++)
@@ -203,6 +318,14 @@ AioShmemInit(void)
dclist_push_tail(&bs->idle_ios, &ioh->node);
iovec_off += PG_IOV_MAX;
}
+
+ /* initialize per-backend bounce buffers */
+ for (int i = 0; i < per_backend_bb; i++)
+ {
+ PgAioBounceBuffer *bb = &pgaio_ctl->bounce_buffers[bs->bounce_buffers_off + i];
+
+ slist_push_head(&bs->idle_bbs, &bb->node);
+ }
}
out:
diff --git a/src/backend/utils/misc/guc_tables.c b/src/backend/utils/misc/guc_tables.c
index 8a83dcc820d..57865d45124 100644
--- a/src/backend/utils/misc/guc_tables.c
+++ b/src/backend/utils/misc/guc_tables.c
@@ -3234,6 +3234,19 @@ struct config_int ConfigureNamesInt[] =
NULL, NULL, NULL
},
+ {
+ {"io_bounce_buffers",
+ PGC_POSTMASTER,
+ RESOURCES_ASYNCHRONOUS,
+ gettext_noop("Number of IO Bounce Buffers reserved for each backend."),
+ NULL,
+ GUC_UNIT_BLOCKS
+ },
+ &io_bounce_buffers,
+ -1, -1, 4096,
+ NULL, NULL, NULL
+ },
+
{
{"io_workers",
PGC_SIGHUP,
diff --git a/src/backend/utils/misc/postgresql.conf.sample b/src/backend/utils/misc/postgresql.conf.sample
index 5005e65cee0..294d661ebf4 100644
--- a/src/backend/utils/misc/postgresql.conf.sample
+++ b/src/backend/utils/misc/postgresql.conf.sample
@@ -853,6 +853,8 @@ autovacuum_worker_slots = 16 # autovacuum worker slots to allocate
#io_max_concurrency = 32 # Max number of IOs that may be in
# flight at the same time in one backend
# (change requires restart)
+#io_bounce_buffers = -1 # -1 sets based on shared_buffers
+ # (change requires restart)
#------------------------------------------------------------------------------
diff --git a/src/backend/utils/resowner/resowner.c b/src/backend/utils/resowner/resowner.c
index e5d852b5ee6..9db3c07326c 100644
--- a/src/backend/utils/resowner/resowner.c
+++ b/src/backend/utils/resowner/resowner.c
@@ -159,10 +159,11 @@ struct ResourceOwnerData
LOCALLOCK *locks[MAX_RESOWNER_LOCKS]; /* list of owned locks */
/*
- * AIO handles need be registered in critical sections and therefore
- * cannot use the normal ResoureElem mechanism.
+ * AIO handles & bounce buffers need be registered in critical sections
+ * and therefore cannot use the normal ResoureElem mechanism.
*/
dlist_head aio_handles;
+ dlist_head aio_bounce_buffers;
};
@@ -434,6 +435,7 @@ ResourceOwnerCreate(ResourceOwner parent, const char *name)
}
dlist_init(&owner->aio_handles);
+ dlist_init(&owner->aio_bounce_buffers);
return owner;
}
@@ -743,6 +745,13 @@ ResourceOwnerReleaseInternal(ResourceOwner owner,
pgaio_io_release_resowner(node, !isCommit);
}
+
+ while (!dlist_is_empty(&owner->aio_bounce_buffers))
+ {
+ dlist_node *node = dlist_head_node(&owner->aio_bounce_buffers);
+
+ pgaio_bounce_buffer_release_resowner(node, !isCommit);
+ }
}
else if (phase == RESOURCE_RELEASE_LOCKS)
{
@@ -1112,3 +1121,15 @@ ResourceOwnerForgetAioHandle(ResourceOwner owner, struct dlist_node *ioh_node)
{
dlist_delete_from(&owner->aio_handles, ioh_node);
}
+
+void
+ResourceOwnerRememberAioBounceBuffer(ResourceOwner owner, struct dlist_node *ioh_node)
+{
+ dlist_push_tail(&owner->aio_bounce_buffers, ioh_node);
+}
+
+void
+ResourceOwnerForgetAioBounceBuffer(ResourceOwner owner, struct dlist_node *ioh_node)
+{
+ dlist_delete_from(&owner->aio_bounce_buffers, ioh_node);
+}
diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list
index be2dd22f1d7..b3f06711e6a 100644
--- a/src/tools/pgindent/typedefs.list
+++ b/src/tools/pgindent/typedefs.list
@@ -2110,6 +2110,7 @@ PermutationStep
PermutationStepBlocker
PermutationStepBlockerType
PgAioBackend
+PgAioBounceBuffer
PgAioCtl
PgAioHandle
PgAioHandleCallbackID
--
2.48.1.76.g4e746b1a31.dirty
[text/x-diff] v2.3-0024-bufmgr-Implement-AIO-write-support.patch (6.2K, ../../p5flybfcjrzgpgwcnzkxsp3xdw3czfoafkvks6g7en6ewmvymh@wrz7z2vwwc7c/25-v2.3-0024-bufmgr-Implement-AIO-write-support.patch)
download | inline diff:
From 98ba93250f1fd40e4a97387bf08f90b28686705c Mon Sep 17 00:00:00 2001
From: Andres Freund <andres@anarazel.de>
Date: Wed, 22 Jan 2025 16:09:51 -0500
Subject: [PATCH v2.3 24/30] bufmgr: Implement AIO write support
As of this commit there are no users of these AIO facilities, that'll come in
later commits.
Author:
Reviewed-By:
Discussion: https://postgr.es/m/
Backpatch:
---
src/include/storage/aio.h | 2 +
src/include/storage/bufmgr.h | 2 +
src/backend/storage/aio/aio_callback.c | 2 +
src/backend/storage/buffer/bufmgr.c | 90 +++++++++++++++++++++++++-
4 files changed, 95 insertions(+), 1 deletion(-)
diff --git a/src/include/storage/aio.h b/src/include/storage/aio.h
index 30b08495f3d..7bdce41121e 100644
--- a/src/include/storage/aio.h
+++ b/src/include/storage/aio.h
@@ -180,8 +180,10 @@ typedef enum PgAioHandleCallbackID
PGAIO_HCB_MD_WRITEV,
PGAIO_HCB_SHARED_BUFFER_READV,
+ PGAIO_HCB_SHARED_BUFFER_WRITEV,
PGAIO_HCB_LOCAL_BUFFER_READV,
+ PGAIO_HCB_LOCAL_BUFFER_WRITEV,
} PgAioHandleCallbackID;
diff --git a/src/include/storage/bufmgr.h b/src/include/storage/bufmgr.h
index f205643c4ef..cf9d0a63aed 100644
--- a/src/include/storage/bufmgr.h
+++ b/src/include/storage/bufmgr.h
@@ -203,7 +203,9 @@ extern PGDLLIMPORT int32 *LocalRefCount;
struct PgAioHandleCallbacks;
extern const struct PgAioHandleCallbacks aio_shared_buffer_readv_cb;
+extern const struct PgAioHandleCallbacks aio_shared_buffer_writev_cb;
extern const struct PgAioHandleCallbacks aio_local_buffer_readv_cb;
+extern const struct PgAioHandleCallbacks aio_local_buffer_writev_cb;
/* upper limit for effective_io_concurrency */
diff --git a/src/backend/storage/aio/aio_callback.c b/src/backend/storage/aio/aio_callback.c
index 6054f57eb23..acfed50bfeb 100644
--- a/src/backend/storage/aio/aio_callback.c
+++ b/src/backend/storage/aio/aio_callback.c
@@ -45,8 +45,10 @@ static const PgAioHandleCallbacksEntry aio_handle_cbs[] = {
CALLBACK_ENTRY(PGAIO_HCB_MD_WRITEV, aio_md_writev_cb),
CALLBACK_ENTRY(PGAIO_HCB_SHARED_BUFFER_READV, aio_shared_buffer_readv_cb),
+ CALLBACK_ENTRY(PGAIO_HCB_SHARED_BUFFER_WRITEV, aio_shared_buffer_writev_cb),
CALLBACK_ENTRY(PGAIO_HCB_LOCAL_BUFFER_READV, aio_local_buffer_readv_cb),
+ CALLBACK_ENTRY(PGAIO_HCB_LOCAL_BUFFER_WRITEV, aio_local_buffer_writev_cb),
#undef CALLBACK_ENTRY
};
diff --git a/src/backend/storage/buffer/bufmgr.c b/src/backend/storage/buffer/bufmgr.c
index 118a6e1ca31..d5212da4912 100644
--- a/src/backend/storage/buffer/bufmgr.c
+++ b/src/backend/storage/buffer/bufmgr.c
@@ -6402,6 +6402,42 @@ ReadBufferCompleteReadShared(Buffer buffer, int mode, bool failed)
return buf_failed;
}
+static uint64
+ReadBufferCompleteWriteShared(Buffer buffer, bool release_lock, bool failed)
+{
+ BufferDesc *bufHdr;
+ bool result = false;
+
+ Assert(BufferIsValid(buffer));
+
+ bufHdr = GetBufferDescriptor(buffer - 1);
+
+#ifdef USE_ASSERT_CHECKING
+ {
+ uint32 buf_state = pg_atomic_read_u32(&bufHdr->state);
+
+ Assert(buf_state & BM_VALID);
+ Assert(buf_state & BM_TAG_VALID);
+ Assert(buf_state & BM_IO_IN_PROGRESS);
+ Assert(buf_state & BM_DIRTY);
+ }
+#endif
+
+ TerminateBufferIO(bufHdr, /* clear_dirty = */ true,
+ failed ? BM_IO_ERROR : 0,
+ /* forget_owner = */ false,
+ /* syncio = */ false);
+
+ /*
+ * The initiator of IO is not managing the lock (i.e. called
+ * LWLockDisown()), we are.
+ */
+ if (release_lock)
+ LWLockReleaseDisowned(BufferDescriptorGetContentLock(bufHdr), LW_SHARED);
+
+ return result;
+}
+
/*
* Helper to prepare IO on shared buffers for execution, shared between reads
* and writes.
@@ -6466,7 +6502,6 @@ shared_buffer_stage_common(PgAioHandle *ioh, bool is_write)
* Lock is now owned by AIO subsystem.
*/
LWLockDisown(content_lock);
- RESUME_INTERRUPTS();
}
/*
@@ -6483,6 +6518,12 @@ shared_buffer_readv_stage(PgAioHandle *ioh)
shared_buffer_stage_common(ioh, false);
}
+static void
+shared_buffer_writev_stage(PgAioHandle *ioh)
+{
+ shared_buffer_stage_common(ioh, true);
+}
+
static PgAioResult
shared_buffer_readv_complete(PgAioHandle *ioh, PgAioResult prior_result)
{
@@ -6558,6 +6599,36 @@ buffer_readv_report(PgAioResult result, const PgAioTargetData *target_data, int
MemoryContextSwitchTo(oldContext);
}
+static PgAioResult
+shared_buffer_writev_complete(PgAioHandle *ioh, PgAioResult prior_result)
+{
+ PgAioResult result = prior_result;
+ uint64 *io_data;
+ uint8 handle_data_len;
+
+ ereport(DEBUG5,
+ errmsg("%s: %d %d", __func__, prior_result.status, prior_result.result),
+ errhidestmt(true), errhidecontext(true));
+
+ io_data = pgaio_io_get_handle_data(ioh, &handle_data_len);
+
+ /* FIXME: handle outright errors */
+
+ for (int io_data_off = 0; io_data_off < handle_data_len; io_data_off++)
+ {
+ Buffer buf = io_data[io_data_off];
+
+ /* FIXME: handle short writes / failures */
+ /* FIXME: ioh->target_data.shared_buffer.release_lock */
+ ReadBufferCompleteWriteShared(buf,
+ true,
+ false);
+
+ }
+
+ return result;
+}
+
/*
* Helper to stage IO on local buffers for execution, shared between reads
* and writes.
@@ -6644,12 +6715,26 @@ local_buffer_readv_complete(PgAioHandle *ioh, PgAioResult prior_result)
return result;
}
+static void
+local_buffer_writev_stage(PgAioHandle *ioh)
+{
+ /*
+ * Currently this is unreachable as the only write support is for
+ * checkpointer / bgwriter, which don't deal with local buffers.
+ */
+ elog(ERROR, "not yet");
+}
+
const struct PgAioHandleCallbacks aio_shared_buffer_readv_cb = {
.stage = shared_buffer_readv_stage,
.complete_shared = shared_buffer_readv_complete,
.report = buffer_readv_report,
};
+const struct PgAioHandleCallbacks aio_shared_buffer_writev_cb = {
+ .stage = shared_buffer_writev_stage,
+ .complete_shared = shared_buffer_writev_complete,
+};
const struct PgAioHandleCallbacks aio_local_buffer_readv_cb = {
.stage = local_buffer_readv_stage,
@@ -6662,3 +6747,6 @@ const struct PgAioHandleCallbacks aio_local_buffer_readv_cb = {
.complete_local = local_buffer_readv_complete,
.report = buffer_readv_report,
};
+const struct PgAioHandleCallbacks aio_local_buffer_writev_cb = {
+ .stage = local_buffer_writev_stage,
+};
--
2.48.1.76.g4e746b1a31.dirty
[text/x-diff] v2.3-0025-aio-Add-IO-queue-helper.patch (7.2K, ../../p5flybfcjrzgpgwcnzkxsp3xdw3czfoafkvks6g7en6ewmvymh@wrz7z2vwwc7c/26-v2.3-0025-aio-Add-IO-queue-helper.patch)
download | inline diff:
From e7e8e954a1432f531d830242fd170564f268521c Mon Sep 17 00:00:00 2001
From: Andres Freund <andres@anarazel.de>
Date: Wed, 22 Jan 2025 13:44:50 -0500
Subject: [PATCH v2.3 25/30] aio: Add IO queue helper
This is likely never going to anywhere - Thomas Munro is working on something
more complete. But I needed a way to exercise aio for checkpointer / bgwriter.
---
src/include/storage/io_queue.h | 31 +++++
src/backend/storage/aio/Makefile | 1 +
src/backend/storage/aio/io_queue.c | 198 ++++++++++++++++++++++++++++
src/backend/storage/aio/meson.build | 1 +
src/tools/pgindent/typedefs.list | 2 +
5 files changed, 233 insertions(+)
create mode 100644 src/include/storage/io_queue.h
create mode 100644 src/backend/storage/aio/io_queue.c
diff --git a/src/include/storage/io_queue.h b/src/include/storage/io_queue.h
new file mode 100644
index 00000000000..f5e1bc07ff3
--- /dev/null
+++ b/src/include/storage/io_queue.h
@@ -0,0 +1,31 @@
+/*-------------------------------------------------------------------------
+ *
+ * io_queue.h
+ * Mechanism for tracking many IOs
+ *
+ *
+ * Portions Copyright (c) 1996-2025, PostgreSQL Global Development Group
+ * Portions Copyright (c) 1994, Regents of the University of California
+ *
+ * src/include/storage/io_queue.h
+ *
+ *-------------------------------------------------------------------------
+ */
+#ifndef IO_QUEUE_H
+#define IO_QUEUE_H
+
+struct IOQueue;
+typedef struct IOQueue IOQueue;
+
+struct PgAioWaitRef;
+
+extern IOQueue *io_queue_create(int depth, int flags);
+extern void io_queue_track(IOQueue *ioq, const struct PgAioWaitRef *iow);
+extern void io_queue_wait_one(IOQueue *ioq);
+extern void io_queue_wait_all(IOQueue *ioq);
+extern bool io_queue_is_empty(IOQueue *ioq);
+extern void io_queue_reserve(IOQueue *ioq);
+extern struct PgAioHandle *io_queue_acquire_io(IOQueue *ioq);
+extern void io_queue_free(IOQueue *ioq);
+
+#endif /* IO_QUEUE_H */
diff --git a/src/backend/storage/aio/Makefile b/src/backend/storage/aio/Makefile
index 3f2469cc399..86fa4276fda 100644
--- a/src/backend/storage/aio/Makefile
+++ b/src/backend/storage/aio/Makefile
@@ -15,6 +15,7 @@ OBJS = \
aio_init.o \
aio_io.o \
aio_target.o \
+ io_queue.o \
method_io_uring.o \
method_sync.o \
method_worker.o \
diff --git a/src/backend/storage/aio/io_queue.c b/src/backend/storage/aio/io_queue.c
new file mode 100644
index 00000000000..62ad06c8bfe
--- /dev/null
+++ b/src/backend/storage/aio/io_queue.c
@@ -0,0 +1,198 @@
+/*-------------------------------------------------------------------------
+ *
+ * io_queue.c
+ * AIO - Mechanism for tracking many IOs
+ *
+ * Portions Copyright (c) 1996-2025, PostgreSQL Global Development Group
+ * Portions Copyright (c) 1994, Regents of the University of California
+ *
+ * IDENTIFICATION
+ * src/backend/storage/aio/io_queue.c
+ *
+ *-------------------------------------------------------------------------
+ */
+
+#include "postgres.h"
+
+#include "lib/ilist.h"
+#include "storage/aio.h"
+#include "storage/io_queue.h"
+#include "utils/resowner.h"
+
+
+
+typedef struct TrackedIO
+{
+ PgAioWaitRef iow;
+ dlist_node node;
+} TrackedIO;
+
+struct IOQueue
+{
+ int depth;
+ int unsubmitted;
+
+ bool has_reserved;
+
+ dclist_head idle;
+ dclist_head in_progress;
+
+ TrackedIO tracked_ios[FLEXIBLE_ARRAY_MEMBER];
+};
+
+
+IOQueue *
+io_queue_create(int depth, int flags)
+{
+ size_t sz;
+ IOQueue *ioq;
+
+ sz = offsetof(IOQueue, tracked_ios)
+ + sizeof(TrackedIO) * depth;
+
+ ioq = palloc0(sz);
+
+ ioq->depth = 0;
+
+ for (int i = 0; i < depth; i++)
+ {
+ TrackedIO *tio = &ioq->tracked_ios[i];
+
+ pgaio_wref_clear(&tio->iow);
+ dclist_push_tail(&ioq->idle, &tio->node);
+ }
+
+ return ioq;
+}
+
+void
+io_queue_wait_one(IOQueue *ioq)
+{
+ while (!dclist_is_empty(&ioq->in_progress))
+ {
+ /* FIXME: Should we really pop here already? */
+ dlist_node *node = dclist_pop_head_node(&ioq->in_progress);
+ TrackedIO *tio = dclist_container(TrackedIO, node, node);
+
+ pgaio_wref_wait(&tio->iow);
+ dclist_push_head(&ioq->idle, &tio->node);
+ }
+}
+
+void
+io_queue_reserve(IOQueue *ioq)
+{
+ if (ioq->has_reserved)
+ return;
+
+ if (dclist_is_empty(&ioq->idle))
+ io_queue_wait_one(ioq);
+
+ Assert(!dclist_is_empty(&ioq->idle));
+
+ ioq->has_reserved = true;
+}
+
+PgAioHandle *
+io_queue_acquire_io(IOQueue *ioq)
+{
+ PgAioHandle *ioh;
+
+ io_queue_reserve(ioq);
+
+ Assert(!dclist_is_empty(&ioq->idle));
+
+ if (!io_queue_is_empty(ioq))
+ {
+ ioh = pgaio_io_acquire_nb(CurrentResourceOwner, NULL);
+ if (ioh == NULL)
+ {
+ /*
+ * Need to wait for all IOs, blocking might not be legal in the
+ * context.
+ *
+ * XXX: This doesn't make a whole lot of sense, we're also
+ * blocking here. What was I smoking when I wrote the above?
+ */
+ io_queue_wait_all(ioq);
+ ioh = pgaio_io_acquire(CurrentResourceOwner, NULL);
+ }
+ }
+ else
+ {
+ ioh = pgaio_io_acquire(CurrentResourceOwner, NULL);
+ }
+
+ return ioh;
+}
+
+void
+io_queue_track(IOQueue *ioq, const struct PgAioWaitRef *iow)
+{
+ dlist_node *node;
+ TrackedIO *tio;
+
+ Assert(ioq->has_reserved);
+ ioq->has_reserved = false;
+
+ Assert(!dclist_is_empty(&ioq->idle));
+
+ node = dclist_pop_head_node(&ioq->idle);
+ tio = dclist_container(TrackedIO, node, node);
+
+ tio->iow = *iow;
+
+ dclist_push_tail(&ioq->in_progress, &tio->node);
+
+ ioq->unsubmitted++;
+
+ /*
+ * XXX: Should have some smarter logic here. We don't want to wait too
+ * long to submit, that'll mean we're more likely to block. But we also
+ * don't want to have the overhead of submitting every IO individually.
+ */
+ if (ioq->unsubmitted >= 4)
+ {
+ pgaio_submit_staged();
+ ioq->unsubmitted = 0;
+ }
+}
+
+void
+io_queue_wait_all(IOQueue *ioq)
+{
+ while (!dclist_is_empty(&ioq->in_progress))
+ {
+ /* wait for the last IO to minimize unnecessary wakeups */
+ dlist_node *node = dclist_tail_node(&ioq->in_progress);
+ TrackedIO *tio = dclist_container(TrackedIO, node, node);
+
+ if (!pgaio_wref_check_done(&tio->iow))
+ {
+ ereport(DEBUG3,
+ errmsg("io_queue_wait_all for io:%d",
+ pgaio_wref_get_id(&tio->iow)),
+ errhidestmt(true),
+ errhidecontext(true));
+
+ pgaio_wref_wait(&tio->iow);
+ }
+
+ dclist_delete_from(&ioq->in_progress, &tio->node);
+ dclist_push_head(&ioq->idle, &tio->node);
+ }
+}
+
+bool
+io_queue_is_empty(IOQueue *ioq)
+{
+ return dclist_is_empty(&ioq->in_progress);
+}
+
+void
+io_queue_free(IOQueue *ioq)
+{
+ io_queue_wait_all(ioq);
+
+ pfree(ioq);
+}
diff --git a/src/backend/storage/aio/meson.build b/src/backend/storage/aio/meson.build
index da6df2d3654..270c4a64428 100644
--- a/src/backend/storage/aio/meson.build
+++ b/src/backend/storage/aio/meson.build
@@ -7,6 +7,7 @@ backend_sources += files(
'aio_init.c',
'aio_io.c',
'aio_target.c',
+ 'io_queue.c',
'method_io_uring.c',
'method_sync.c',
'method_worker.c',
diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list
index b3f06711e6a..91d8198af9f 100644
--- a/src/tools/pgindent/typedefs.list
+++ b/src/tools/pgindent/typedefs.list
@@ -1179,6 +1179,7 @@ IOContext
IOFuncSelector
IOObject
IOOp
+IOQueue
IO_STATUS_BLOCK
IPCompareMethod
ITEM
@@ -2986,6 +2987,7 @@ TocEntry
TokenAuxData
TokenizedAuthLine
TrackItem
+TrackedIO
TransApplyAction
TransInvalidationInfo
TransState
--
2.48.1.76.g4e746b1a31.dirty
[text/x-diff] v2.3-0026-bufmgr-use-AIO-in-checkpointer-bgwriter.patch (31.6K, ../../p5flybfcjrzgpgwcnzkxsp3xdw3czfoafkvks6g7en6ewmvymh@wrz7z2vwwc7c/27-v2.3-0026-bufmgr-use-AIO-in-checkpointer-bgwriter.patch)
download | inline diff:
From 0e410933546b25b259ee8a02fa27bb7f34b3f736 Mon Sep 17 00:00:00 2001
From: Andres Freund <andres@anarazel.de>
Date: Wed, 22 Jan 2025 13:44:52 -0500
Subject: [PATCH v2.3 26/30] bufmgr: use AIO in checkpointer, bgwriter
This is far from ready - just included to be able to exercise AIO writes and
get some preliminary numbers. In all likelihood this will instead be based
ontop of work by Thomas Munro instead of the preceding commit.
---
src/include/postmaster/bgwriter.h | 3 +-
src/include/storage/buf_internals.h | 2 +
src/include/storage/bufmgr.h | 3 +-
src/include/storage/bufpage.h | 1 +
src/backend/postmaster/bgwriter.c | 20 +-
src/backend/postmaster/checkpointer.c | 12 +-
src/backend/storage/buffer/bufmgr.c | 587 +++++++++++++++++++++++---
src/backend/storage/page/bufpage.c | 10 +
src/tools/pgindent/typedefs.list | 1 +
9 files changed, 581 insertions(+), 58 deletions(-)
diff --git a/src/include/postmaster/bgwriter.h b/src/include/postmaster/bgwriter.h
index 2d5854e6879..517c40cd804 100644
--- a/src/include/postmaster/bgwriter.h
+++ b/src/include/postmaster/bgwriter.h
@@ -31,7 +31,8 @@ extern void BackgroundWriterMain(char *startup_data, size_t startup_data_len) pg
extern void CheckpointerMain(char *startup_data, size_t startup_data_len) pg_attribute_noreturn();
extern void RequestCheckpoint(int flags);
-extern void CheckpointWriteDelay(int flags, double progress);
+struct IOQueue;
+extern void CheckpointWriteDelay(struct IOQueue *ioq, int flags, double progress);
extern bool ForwardSyncRequest(const FileTag *ftag, SyncRequestType type);
diff --git a/src/include/storage/buf_internals.h b/src/include/storage/buf_internals.h
index 9f936cd6b84..aeefb1746ec 100644
--- a/src/include/storage/buf_internals.h
+++ b/src/include/storage/buf_internals.h
@@ -21,6 +21,8 @@
#include "storage/buf.h"
#include "storage/bufmgr.h"
#include "storage/condition_variable.h"
+#include "storage/io_queue.h"
+#include "storage/latch.h"
#include "storage/lwlock.h"
#include "storage/shmem.h"
#include "storage/smgr.h"
diff --git a/src/include/storage/bufmgr.h b/src/include/storage/bufmgr.h
index cf9d0a63aed..bc7ee73246e 100644
--- a/src/include/storage/bufmgr.h
+++ b/src/include/storage/bufmgr.h
@@ -321,7 +321,8 @@ extern bool ConditionalLockBufferForCleanup(Buffer buffer);
extern bool IsBufferCleanupOK(Buffer buffer);
extern bool HoldingBufferPinThatDelaysRecovery(void);
-extern bool BgBufferSync(struct WritebackContext *wb_context);
+struct IOQueue;
+extern bool BgBufferSync(struct IOQueue *ioq, struct WritebackContext *wb_context);
extern void LimitAdditionalPins(uint32 *additional_pins);
extern void LimitAdditionalLocalPins(uint32 *additional_pins);
diff --git a/src/include/storage/bufpage.h b/src/include/storage/bufpage.h
index d06208b7ce6..a2bd1db92d0 100644
--- a/src/include/storage/bufpage.h
+++ b/src/include/storage/bufpage.h
@@ -509,5 +509,6 @@ extern bool PageIndexTupleOverwrite(Page page, OffsetNumber offnum,
Item newtup, Size newsize);
extern char *PageSetChecksumCopy(Page page, BlockNumber blkno);
extern void PageSetChecksumInplace(Page page, BlockNumber blkno);
+extern bool PageNeedsChecksumCopy(Page page);
#endif /* BUFPAGE_H */
diff --git a/src/backend/postmaster/bgwriter.c b/src/backend/postmaster/bgwriter.c
index 3eff5dc6f0e..cf16f8bed5d 100644
--- a/src/backend/postmaster/bgwriter.c
+++ b/src/backend/postmaster/bgwriter.c
@@ -38,10 +38,12 @@
#include "postmaster/auxprocess.h"
#include "postmaster/bgwriter.h"
#include "postmaster/interrupt.h"
+#include "storage/aio.h"
#include "storage/buf_internals.h"
#include "storage/bufmgr.h"
#include "storage/condition_variable.h"
#include "storage/fd.h"
+#include "storage/io_queue.h"
#include "storage/lwlock.h"
#include "storage/proc.h"
#include "storage/procsignal.h"
@@ -89,6 +91,7 @@ BackgroundWriterMain(char *startup_data, size_t startup_data_len)
sigjmp_buf local_sigjmp_buf;
MemoryContext bgwriter_context;
bool prev_hibernate;
+ IOQueue *ioq;
WritebackContext wb_context;
Assert(startup_data_len == 0);
@@ -130,6 +133,7 @@ BackgroundWriterMain(char *startup_data, size_t startup_data_len)
ALLOCSET_DEFAULT_SIZES);
MemoryContextSwitchTo(bgwriter_context);
+ ioq = io_queue_create(128, 0);
WritebackContextInit(&wb_context, &bgwriter_flush_after);
/*
@@ -170,6 +174,7 @@ BackgroundWriterMain(char *startup_data, size_t startup_data_len)
ConditionVariableCancelSleep();
UnlockBuffers();
ReleaseAuxProcessResources(false);
+ pgaio_at_error();
AtEOXact_Buffers(false);
AtEOXact_SMgr();
AtEOXact_Files(false);
@@ -226,12 +231,22 @@ BackgroundWriterMain(char *startup_data, size_t startup_data_len)
/* Clear any already-pending wakeups */
ResetLatch(MyLatch);
+ /*
+ * FIXME: this is theoretically racy, but I didn't want to copy
+ * HandleMainLoopInterrupts() remaining body here.
+ */
+ if (ShutdownRequestPending)
+ {
+ io_queue_wait_all(ioq);
+ io_queue_free(ioq);
+ }
+
HandleMainLoopInterrupts();
/*
* Do one cycle of dirty-buffer writing.
*/
- can_hibernate = BgBufferSync(&wb_context);
+ can_hibernate = BgBufferSync(ioq, &wb_context);
/* Report pending statistics to the cumulative stats system */
pgstat_report_bgwriter();
@@ -248,6 +263,9 @@ BackgroundWriterMain(char *startup_data, size_t startup_data_len)
smgrdestroyall();
}
+ /* finish IO before sleeping, to avoid blocking other backends */
+ io_queue_wait_all(ioq);
+
/*
* Log a new xl_running_xacts every now and then so replication can
* get into a consistent state faster (think of suboverflowed
diff --git a/src/backend/postmaster/checkpointer.c b/src/backend/postmaster/checkpointer.c
index 767bf9f5cf8..0fb7f3b7275 100644
--- a/src/backend/postmaster/checkpointer.c
+++ b/src/backend/postmaster/checkpointer.c
@@ -49,9 +49,11 @@
#include "postmaster/bgwriter.h"
#include "postmaster/interrupt.h"
#include "replication/syncrep.h"
+#include "storage/aio.h"
#include "storage/bufmgr.h"
#include "storage/condition_variable.h"
#include "storage/fd.h"
+#include "storage/io_queue.h"
#include "storage/ipc.h"
#include "storage/lwlock.h"
#include "storage/pmsignal.h"
@@ -278,6 +280,7 @@ CheckpointerMain(char *startup_data, size_t startup_data_len)
pgstat_report_wait_end();
UnlockBuffers();
ReleaseAuxProcessResources(false);
+ pgaio_at_error();
AtEOXact_Buffers(false);
AtEOXact_SMgr();
AtEOXact_Files(false);
@@ -762,7 +765,7 @@ ImmediateCheckpointRequested(void)
* fraction between 0.0 meaning none, and 1.0 meaning all done.
*/
void
-CheckpointWriteDelay(int flags, double progress)
+CheckpointWriteDelay(IOQueue *ioq, int flags, double progress)
{
static int absorb_counter = WRITES_PER_ABSORB;
@@ -796,6 +799,13 @@ CheckpointWriteDelay(int flags, double progress)
/* Report interim statistics to the cumulative stats system */
pgstat_report_checkpointer();
+ /*
+ * Ensure all pending IO is submitted to avoid unnecessary delays for
+ * other processes.
+ */
+ io_queue_wait_all(ioq);
+
+
/*
* This sleep used to be connected to bgwriter_delay, typically 200ms.
* That resulted in more frequent wakeups if not much work to do.
diff --git a/src/backend/storage/buffer/bufmgr.c b/src/backend/storage/buffer/bufmgr.c
index d5212da4912..1e8793d1630 100644
--- a/src/backend/storage/buffer/bufmgr.c
+++ b/src/backend/storage/buffer/bufmgr.c
@@ -52,6 +52,7 @@
#include "storage/buf_internals.h"
#include "storage/bufmgr.h"
#include "storage/fd.h"
+#include "storage/io_queue.h"
#include "storage/ipc.h"
#include "storage/lmgr.h"
#include "storage/proc.h"
@@ -77,6 +78,7 @@
/* Bits in SyncOneBuffer's return value */
#define BUF_WRITTEN 0x01
#define BUF_REUSABLE 0x02
+#define BUF_CANT_MERGE 0x04
#define RELS_BSEARCH_THRESHOLD 20
@@ -511,8 +513,6 @@ static void UnpinBuffer(BufferDesc *buf);
static void UnpinBufferNoOwner(BufferDesc *buf);
static void BufferSync(int flags);
static uint32 WaitBufHdrUnlocked(BufferDesc *buf);
-static int SyncOneBuffer(int buf_id, bool skip_recently_used,
- WritebackContext *wb_context);
static void WaitIO(BufferDesc *buf);
static bool StartBufferIO(BufferDesc *buf, bool forInput, bool nowait);
static void TerminateBufferIO(BufferDesc *buf, bool clear_dirty,
@@ -530,6 +530,7 @@ static inline BufferDesc *BufferAlloc(SMgrRelation smgr,
static Buffer GetVictimBuffer(BufferAccessStrategy strategy, IOContext io_context);
static void FlushBuffer(BufferDesc *buf, SMgrRelation reln,
IOObject io_object, IOContext io_context);
+
static void FindAndDropRelationBuffers(RelFileLocator rlocator,
ForkNumber forkNum,
BlockNumber nForkBlock,
@@ -3068,6 +3069,56 @@ UnpinBufferNoOwner(BufferDesc *buf)
}
}
+typedef struct BuffersToWrite
+{
+ int nbuffers;
+ BufferTag start_at_tag;
+ uint32 max_combine;
+
+ XLogRecPtr max_lsn;
+
+ PgAioHandle *ioh;
+ PgAioWaitRef iow;
+
+ uint64 total_writes;
+
+ Buffer buffers[IOV_MAX];
+ PgAioBounceBuffer *bounce_buffers[IOV_MAX];
+ const void *data_ptrs[IOV_MAX];
+} BuffersToWrite;
+
+static int PrepareToWriteBuffer(BuffersToWrite *to_write, Buffer buf,
+ bool skip_recently_used,
+ IOQueue *ioq, WritebackContext *wb_context);
+
+static void WriteBuffers(BuffersToWrite *to_write,
+ IOQueue *ioq, WritebackContext *wb_context);
+
+static void
+BuffersToWriteInit(BuffersToWrite *to_write,
+ IOQueue *ioq, WritebackContext *wb_context)
+{
+ to_write->total_writes = 0;
+ to_write->nbuffers = 0;
+ to_write->ioh = NULL;
+ pgaio_wref_clear(&to_write->iow);
+ to_write->max_lsn = InvalidXLogRecPtr;
+}
+
+static void
+BuffersToWriteEnd(BuffersToWrite *to_write)
+{
+ if (to_write->ioh != NULL)
+ {
+ pgaio_io_release(to_write->ioh);
+ to_write->ioh = NULL;
+ }
+
+ if (to_write->total_writes > 0)
+ pgaio_submit_staged();
+}
+
+
#define ST_SORT sort_checkpoint_bufferids
#define ST_ELEMENT_TYPE CkptSortItem
#define ST_COMPARE(a, b) ckpt_buforder_comparator(a, b)
@@ -3099,7 +3150,10 @@ BufferSync(int flags)
binaryheap *ts_heap;
int i;
int mask = BM_DIRTY;
+ IOQueue *ioq;
WritebackContext wb_context;
+ BuffersToWrite to_write;
+ int max_combine;
/*
* Unless this is a shutdown checkpoint or we have been explicitly told,
@@ -3161,7 +3215,9 @@ BufferSync(int flags)
if (num_to_scan == 0)
return; /* nothing to do */
+ ioq = io_queue_create(512, 0);
WritebackContextInit(&wb_context, &checkpoint_flush_after);
+ max_combine = Min(io_bounce_buffers, io_combine_limit);
TRACE_POSTGRESQL_BUFFER_SYNC_START(NBuffers, num_to_scan);
@@ -3269,48 +3325,91 @@ BufferSync(int flags)
*/
num_processed = 0;
num_written = 0;
+
+ BuffersToWriteInit(&to_write, ioq, &wb_context);
+
while (!binaryheap_empty(ts_heap))
{
BufferDesc *bufHdr = NULL;
CkptTsStatus *ts_stat = (CkptTsStatus *)
DatumGetPointer(binaryheap_first(ts_heap));
+ bool batch_continue = true;
- buf_id = CkptBufferIds[ts_stat->index].buf_id;
- Assert(buf_id != -1);
-
- bufHdr = GetBufferDescriptor(buf_id);
-
- num_processed++;
+ Assert(ts_stat->num_scanned <= ts_stat->num_to_scan);
/*
- * We don't need to acquire the lock here, because we're only looking
- * at a single bit. It's possible that someone else writes the buffer
- * and clears the flag right after we check, but that doesn't matter
- * since SyncOneBuffer will then do nothing. However, there is a
- * further race condition: it's conceivable that between the time we
- * examine the bit here and the time SyncOneBuffer acquires the lock,
- * someone else not only wrote the buffer but replaced it with another
- * page and dirtied it. In that improbable case, SyncOneBuffer will
- * write the buffer though we didn't need to. It doesn't seem worth
- * guarding against this, though.
+ * Collect a batch of buffers to write out from the current
+ * tablespace. That causes some imbalance between the tablespaces, but
+ * that's more than outweighed by the efficiency gain due to batching.
*/
- if (pg_atomic_read_u32(&bufHdr->state) & BM_CHECKPOINT_NEEDED)
+ while (batch_continue &&
+ to_write.nbuffers < max_combine &&
+ ts_stat->num_scanned < ts_stat->num_to_scan)
{
- if (SyncOneBuffer(buf_id, false, &wb_context) & BUF_WRITTEN)
+ buf_id = CkptBufferIds[ts_stat->index].buf_id;
+ Assert(buf_id != -1);
+
+ bufHdr = GetBufferDescriptor(buf_id);
+
+ num_processed++;
+
+ /*
+ * We don't need to acquire the lock here, because we're only
+ * looking at a single bit. It's possible that someone else writes
+ * the buffer and clears the flag right after we check, but that
+ * doesn't matter since SyncOneBuffer will then do nothing.
+ * However, there is a further race condition: it's conceivable
+ * that between the time we examine the bit here and the time
+ * SyncOneBuffer acquires the lock, someone else not only wrote
+ * the buffer but replaced it with another page and dirtied it. In
+ * that improbable case, SyncOneBuffer will write the buffer
+ * though we didn't need to. It doesn't seem worth guarding
+ * against this, though.
+ */
+ if (pg_atomic_read_u32(&bufHdr->state) & BM_CHECKPOINT_NEEDED)
{
- TRACE_POSTGRESQL_BUFFER_SYNC_WRITTEN(buf_id);
- PendingCheckpointerStats.buffers_written++;
- num_written++;
+ int result = PrepareToWriteBuffer(&to_write, buf_id + 1, false,
+ ioq, &wb_context);
+
+ if (result & BUF_CANT_MERGE)
+ {
+ Assert(to_write.nbuffers > 0);
+ WriteBuffers(&to_write, ioq, &wb_context);
+
+ result = PrepareToWriteBuffer(&to_write, buf_id + 1, false,
+ ioq, &wb_context);
+ Assert(result != BUF_CANT_MERGE);
+ }
+
+ if (result & BUF_WRITTEN)
+ {
+ TRACE_POSTGRESQL_BUFFER_SYNC_WRITTEN(buf_id);
+ PendingCheckpointerStats.buffers_written++;
+ num_written++;
+ }
+ else
+ {
+ batch_continue = false;
+ }
}
+ else
+ {
+ if (to_write.nbuffers > 0)
+ WriteBuffers(&to_write, ioq, &wb_context);
+ }
+
+ /*
+ * Measure progress independent of actually having to flush the
+ * buffer - otherwise writing become unbalanced.
+ */
+ ts_stat->progress += ts_stat->progress_slice;
+ ts_stat->num_scanned++;
+ ts_stat->index++;
}
- /*
- * Measure progress independent of actually having to flush the buffer
- * - otherwise writing become unbalanced.
- */
- ts_stat->progress += ts_stat->progress_slice;
- ts_stat->num_scanned++;
- ts_stat->index++;
+ if (to_write.nbuffers > 0)
+ WriteBuffers(&to_write, ioq, &wb_context);
+
/* Have all the buffers from the tablespace been processed? */
if (ts_stat->num_scanned == ts_stat->num_to_scan)
@@ -3328,15 +3427,23 @@ BufferSync(int flags)
*
* (This will check for barrier events even if it doesn't sleep.)
*/
- CheckpointWriteDelay(flags, (double) num_processed / num_to_scan);
+ CheckpointWriteDelay(ioq, flags, (double) num_processed / num_to_scan);
}
+ Assert(to_write.nbuffers == 0);
+ io_queue_wait_all(ioq);
+
/*
* Issue all pending flushes. Only checkpointer calls BufferSync(), so
* IOContext will always be IOCONTEXT_NORMAL.
*/
IssuePendingWritebacks(&wb_context, IOCONTEXT_NORMAL);
+ io_queue_wait_all(ioq); /* IssuePendingWritebacks might have added
+ * more */
+ io_queue_free(ioq);
+ BuffersToWriteEnd(&to_write);
+
pfree(per_ts_stat);
per_ts_stat = NULL;
binaryheap_free(ts_heap);
@@ -3362,7 +3469,7 @@ BufferSync(int flags)
* bgwriter_lru_maxpages to 0.)
*/
bool
-BgBufferSync(WritebackContext *wb_context)
+BgBufferSync(IOQueue *ioq, WritebackContext *wb_context)
{
/* info obtained from freelist.c */
int strategy_buf_id;
@@ -3405,6 +3512,9 @@ BgBufferSync(WritebackContext *wb_context)
long new_strategy_delta;
uint32 new_recent_alloc;
+ BuffersToWrite to_write;
+ int max_combine;
+
/*
* Find out where the freelist clock sweep currently is, and how many
* buffer allocations have happened since our last call.
@@ -3425,6 +3535,8 @@ BgBufferSync(WritebackContext *wb_context)
return true;
}
+ max_combine = Min(io_bounce_buffers, io_combine_limit);
+
/*
* Compute strategy_delta = how many buffers have been scanned by the
* clock sweep since last time. If first time through, assume none. Then
@@ -3581,11 +3693,25 @@ BgBufferSync(WritebackContext *wb_context)
num_written = 0;
reusable_buffers = reusable_buffers_est;
+ BuffersToWriteInit(&to_write, ioq, wb_context);
+
/* Execute the LRU scan */
while (num_to_scan > 0 && reusable_buffers < upcoming_alloc_est)
{
- int sync_state = SyncOneBuffer(next_to_clean, true,
- wb_context);
+ int sync_state;
+
+ sync_state = PrepareToWriteBuffer(&to_write, next_to_clean + 1,
+ true, ioq, wb_context);
+ if (sync_state & BUF_CANT_MERGE)
+ {
+ Assert(to_write.nbuffers > 0);
+
+ WriteBuffers(&to_write, ioq, wb_context);
+
+ sync_state = PrepareToWriteBuffer(&to_write, next_to_clean + 1,
+ true, ioq, wb_context);
+ Assert(sync_state != BUF_CANT_MERGE);
+ }
if (++next_to_clean >= NBuffers)
{
@@ -3596,6 +3722,13 @@ BgBufferSync(WritebackContext *wb_context)
if (sync_state & BUF_WRITTEN)
{
+ Assert(sync_state & BUF_REUSABLE);
+
+ if (to_write.nbuffers == max_combine)
+ {
+ WriteBuffers(&to_write, ioq, wb_context);
+ }
+
reusable_buffers++;
if (++num_written >= bgwriter_lru_maxpages)
{
@@ -3607,6 +3740,11 @@ BgBufferSync(WritebackContext *wb_context)
reusable_buffers++;
}
+ if (to_write.nbuffers > 0)
+ WriteBuffers(&to_write, ioq, wb_context);
+
+ BuffersToWriteEnd(&to_write);
+
PendingBgWriterStats.buf_written_clean += num_written;
#ifdef BGW_DEBUG
@@ -3645,8 +3783,66 @@ BgBufferSync(WritebackContext *wb_context)
return (bufs_to_lap == 0 && recent_alloc == 0);
}
+static inline bool
+BufferTagsSameRel(const BufferTag *tag1, const BufferTag *tag2)
+{
+ return (tag1->spcOid == tag2->spcOid) &&
+ (tag1->dbOid == tag2->dbOid) &&
+ (tag1->relNumber == tag2->relNumber) &&
+ (tag1->forkNum == tag2->forkNum)
+ ;
+}
+
+static bool
+CanMergeWrite(BuffersToWrite *to_write, BufferDesc *cur_buf_hdr)
+{
+ BlockNumber cur_block = cur_buf_hdr->tag.blockNum;
+
+ Assert(to_write->nbuffers > 0); /* can't merge with nothing */
+ Assert(to_write->start_at_tag.relNumber != InvalidOid);
+ Assert(to_write->start_at_tag.blockNum != InvalidBlockNumber);
+
+ Assert(to_write->ioh != NULL);
+
+ /*
+ * First check if the blocknumber is one that we could actually merge,
+ * that's cheaper than checking the tablespace/db/relnumber/fork match.
+ */
+ if (to_write->start_at_tag.blockNum + to_write->nbuffers != cur_block)
+ return false;
+
+ if (!BufferTagsSameRel(&to_write->start_at_tag, &cur_buf_hdr->tag))
+ return false;
+
+ /*
+ * Need to check with smgr how large a write we're allowed to make. To
+ * reduce the overhead of the smgr check, only inquire once, when
+ * processing the first to-be-merged buffer. That avoids the overhead in
+ * the common case of writing out buffers that definitely not mergeable.
+ */
+ if (to_write->nbuffers == 1)
+ {
+ SMgrRelation smgr;
+
+ smgr = smgropen(BufTagGetRelFileLocator(&to_write->start_at_tag), INVALID_PROC_NUMBER);
+
+ to_write->max_combine = smgrmaxcombine(smgr,
+ to_write->start_at_tag.forkNum,
+ to_write->start_at_tag.blockNum);
+ }
+ else
+ {
+ Assert(to_write->max_combine > 0);
+ }
+
+ if (to_write->start_at_tag.blockNum + to_write->max_combine <= cur_block)
+ return false;
+
+ return true;
+}
+
/*
- * SyncOneBuffer -- process a single buffer during syncing.
+ * PrepareToWriteBuffer -- process a single buffer during syncing.
*
* If skip_recently_used is true, we don't write currently-pinned buffers, nor
* buffers marked recently used, as these are not replacement candidates.
@@ -3655,22 +3851,50 @@ BgBufferSync(WritebackContext *wb_context)
* BUF_WRITTEN: we wrote the buffer.
* BUF_REUSABLE: buffer is available for replacement, ie, it has
* pin count 0 and usage count 0.
+ * BUF_CANT_MERGE: can't combine this write with prior writes, caller needs
+ * to issue those first
*
* (BUF_WRITTEN could be set in error if FlushBuffer finds the buffer clean
* after locking it, but we don't care all that much.)
*/
static int
-SyncOneBuffer(int buf_id, bool skip_recently_used, WritebackContext *wb_context)
+PrepareToWriteBuffer(BuffersToWrite *to_write, Buffer buf,
+ bool skip_recently_used,
+ IOQueue *ioq, WritebackContext *wb_context)
{
- BufferDesc *bufHdr = GetBufferDescriptor(buf_id);
+ BufferDesc *cur_buf_hdr = GetBufferDescriptor(buf - 1);
+ uint32 buf_state;
int result = 0;
- uint32 buf_state;
- BufferTag tag;
+ XLogRecPtr cur_buf_lsn;
+ LWLock *content_lock;
+ bool may_block;
+
+ /*
+ * Check if this buffer can be written out together with already prepared
+ * writes. We check before we have pinned the buffer, so the buffer can be
+ * written out and replaced between this check and us pinning the buffer -
+ * we'll recheck below. The reason for the pre-check is that we don't want
+ * to pin the buffer just to find out that we can't merge the IO.
+ */
+ if (to_write->nbuffers != 0)
+ {
+ if (!CanMergeWrite(to_write, cur_buf_hdr))
+ {
+ result |= BUF_CANT_MERGE;
+ return result;
+ }
+ }
+ else
+ {
+ to_write->start_at_tag = cur_buf_hdr->tag;
+ }
/* Make sure we can handle the pin */
ReservePrivateRefCountEntry();
ResourceOwnerEnlarge(CurrentResourceOwner);
+ /* XXX: Should also check if we are allowed to pin one more buffer */
+
/*
* Check whether buffer needs writing.
*
@@ -3680,7 +3904,7 @@ SyncOneBuffer(int buf_id, bool skip_recently_used, WritebackContext *wb_context)
* don't worry because our checkpoint.redo points before log record for
* upcoming changes and so we are not required to write such dirty buffer.
*/
- buf_state = LockBufHdr(bufHdr);
+ buf_state = LockBufHdr(cur_buf_hdr);
if (BUF_STATE_GET_REFCOUNT(buf_state) == 0 &&
BUF_STATE_GET_USAGECOUNT(buf_state) == 0)
@@ -3689,40 +3913,294 @@ SyncOneBuffer(int buf_id, bool skip_recently_used, WritebackContext *wb_context)
}
else if (skip_recently_used)
{
+#if 0
+ elog(LOG, "at block %d: skip recent with nbuffers %d",
+ cur_buf_hdr->tag.blockNum, to_write->nbuffers);
+#endif
/* Caller told us not to write recently-used buffers */
- UnlockBufHdr(bufHdr, buf_state);
+ UnlockBufHdr(cur_buf_hdr, buf_state);
return result;
}
if (!(buf_state & BM_VALID) || !(buf_state & BM_DIRTY))
{
/* It's clean, so nothing to do */
- UnlockBufHdr(bufHdr, buf_state);
+ UnlockBufHdr(cur_buf_hdr, buf_state);
return result;
}
+ /* pin the buffer, from now on its identity can't change anymore */
+ PinBuffer_Locked(cur_buf_hdr);
+
+ /*
+ * Acquire IO, if needed, now that it's likely that we'll need to write.
+ */
+ if (to_write->ioh == NULL)
+ {
+ /* otherwise we should already have acquired a handle */
+ Assert(to_write->nbuffers == 0);
+
+ to_write->ioh = io_queue_acquire_io(ioq);
+ pgaio_io_get_wref(to_write->ioh, &to_write->iow);
+ }
+
/*
- * Pin it, share-lock it, write it. (FlushBuffer will do nothing if the
- * buffer is clean by the time we've locked it.)
+ * If we are merging, check if the buffer's identity possibly changed
+ * while we hadn't yet pinned it.
+ *
+ * XXX: It might be worth checking if we still want to write the buffer
+ * out, e.g. it could have been replaced with a buffer that doesn't have
+ * BM_CHECKPOINT_NEEDED set.
*/
- PinBuffer_Locked(bufHdr);
- LWLockAcquire(BufferDescriptorGetContentLock(bufHdr), LW_SHARED);
+ if (to_write->nbuffers != 0)
+ {
+ if (!CanMergeWrite(to_write, cur_buf_hdr))
+ {
+ elog(LOG, "changed identity");
+ UnpinBuffer(cur_buf_hdr);
+
+ result |= BUF_CANT_MERGE;
+
+ return result;
+ }
+ }
+
+ may_block = to_write->nbuffers == 0
+ && !pgaio_have_staged()
+ && io_queue_is_empty(ioq)
+ ;
+ content_lock = BufferDescriptorGetContentLock(cur_buf_hdr);
+
+ if (!may_block)
+ {
+ if (LWLockConditionalAcquire(content_lock, LW_SHARED))
+ {
+ /* done */
+ }
+ else if (to_write->nbuffers == 0)
+ {
+ /*
+ * Need to wait for all prior IO to finish before blocking for
+ * lock acquisition, to avoid the risk a deadlock due to us
+ * waiting for another backend that is waiting for our unsubmitted
+ * IO to complete.
+ */
+ pgaio_submit_staged();
+ io_queue_wait_all(ioq);
+
+ elog(DEBUG2, "at block %u: can't block, nbuffers = 0",
+ cur_buf_hdr->tag.blockNum
+ );
+
+ may_block = to_write->nbuffers == 0
+ && !pgaio_have_staged()
+ && io_queue_is_empty(ioq)
+ ;
+ Assert(may_block);
+
+ LWLockAcquire(content_lock, LW_SHARED);
+ }
+ else
+ {
+ elog(DEBUG2, "at block %d: can't block nbuffers = %d",
+ cur_buf_hdr->tag.blockNum,
+ to_write->nbuffers);
- FlushBuffer(bufHdr, NULL, IOOBJECT_RELATION, IOCONTEXT_NORMAL);
+ UnpinBuffer(cur_buf_hdr);
+ result |= BUF_CANT_MERGE;
+ Assert(to_write->nbuffers > 0);
- LWLockRelease(BufferDescriptorGetContentLock(bufHdr));
+ return result;
+ }
+ }
+ else
+ {
+ LWLockAcquire(content_lock, LW_SHARED);
+ }
- tag = bufHdr->tag;
+ if (!may_block)
+ {
+ if (!StartBufferIO(cur_buf_hdr, false, !may_block))
+ {
+ pgaio_submit_staged();
+ io_queue_wait_all(ioq);
- UnpinBuffer(bufHdr);
+ may_block = io_queue_is_empty(ioq) && to_write->nbuffers == 0 && !pgaio_have_staged();
+
+ if (!StartBufferIO(cur_buf_hdr, false, !may_block))
+ {
+ elog(DEBUG2, "at block %d: non-waitable StartBufferIO returns false, %d",
+ cur_buf_hdr->tag.blockNum,
+ may_block);
+
+ /*
+ * FIXME: can't tell whether this is because the buffer has
+ * been cleaned
+ */
+ if (!may_block)
+ {
+ result |= BUF_CANT_MERGE;
+ Assert(to_write->nbuffers > 0);
+ }
+ LWLockRelease(content_lock);
+ UnpinBuffer(cur_buf_hdr);
+
+ return result;
+ }
+ }
+ }
+ else
+ {
+ if (!StartBufferIO(cur_buf_hdr, false, false))
+ {
+ elog(DEBUG2, "waitable StartBufferIO returns false");
+ LWLockRelease(content_lock);
+ UnpinBuffer(cur_buf_hdr);
+
+ /*
+ * FIXME: Historically we returned BUF_WRITTEN in this case, which
+ * seems wrong
+ */
+ return result;
+ }
+ }
/*
- * SyncOneBuffer() is only called by checkpointer and bgwriter, so
- * IOContext will always be IOCONTEXT_NORMAL.
+ * Run PageGetLSN while holding header lock, since we don't have the
+ * buffer locked exclusively in all cases.
*/
- ScheduleBufferTagForWriteback(wb_context, IOCONTEXT_NORMAL, &tag);
+ buf_state = LockBufHdr(cur_buf_hdr);
+
+ cur_buf_lsn = BufferGetLSN(cur_buf_hdr);
+
+ /* To check if block content changes while flushing. - vadim 01/17/97 */
+ buf_state &= ~BM_JUST_DIRTIED;
+
+ UnlockBufHdr(cur_buf_hdr, buf_state);
+
+ to_write->buffers[to_write->nbuffers] = buf;
+ to_write->nbuffers++;
+
+ if (buf_state & BM_PERMANENT &&
+ (to_write->max_lsn == InvalidXLogRecPtr || to_write->max_lsn < cur_buf_lsn))
+ {
+ to_write->max_lsn = cur_buf_lsn;
+ }
+
+ result |= BUF_WRITTEN;
+
+ return result;
+}
+
+static void
+WriteBuffers(BuffersToWrite *to_write,
+ IOQueue *ioq, WritebackContext *wb_context)
+{
+ SMgrRelation smgr;
+ Buffer first_buf;
+ BufferDesc *first_buf_hdr;
+ bool needs_checksum;
+
+ Assert(to_write->nbuffers > 0 && to_write->nbuffers <= io_combine_limit);
+
+ first_buf = to_write->buffers[0];
+ first_buf_hdr = GetBufferDescriptor(first_buf - 1);
+
+ smgr = smgropen(BufTagGetRelFileLocator(&first_buf_hdr->tag), INVALID_PROC_NUMBER);
+
+ /*
+ * Force XLOG flush up to buffer's LSN. This implements the basic WAL
+ * rule that log updates must hit disk before any of the data-file changes
+ * they describe do.
+ *
+ * However, this rule does not apply to unlogged relations, which will be
+ * lost after a crash anyway. Most unlogged relation pages do not bear
+ * LSNs since we never emit WAL records for them, and therefore flushing
+ * up through the buffer LSN would be useless, but harmless. However,
+ * GiST indexes use LSNs internally to track page-splits, and therefore
+ * unlogged GiST pages bear "fake" LSNs generated by
+ * GetFakeLSNForUnloggedRel. It is unlikely but possible that the fake
+ * LSN counter could advance past the WAL insertion point; and if it did
+ * happen, attempting to flush WAL through that location would fail, with
+ * disastrous system-wide consequences. To make sure that can't happen,
+ * skip the flush if the buffer isn't permanent.
+ */
+ if (to_write->max_lsn != InvalidXLogRecPtr)
+ XLogFlush(to_write->max_lsn);
+
+ /*
+ * Now it's safe to write buffer to disk. Note that no one else should
+ * have been able to write it while we were busy with log flushing because
+ * only one process at a time can set the BM_IO_IN_PROGRESS bit.
+ */
+
+ for (int nbuf = 0; nbuf < to_write->nbuffers; nbuf++)
+ {
+ Buffer cur_buf = to_write->buffers[nbuf];
+ BufferDesc *cur_buf_hdr = GetBufferDescriptor(cur_buf - 1);
+ Block bufBlock;
+ char *bufToWrite;
+
+ bufBlock = BufHdrGetBlock(cur_buf_hdr);
+ needs_checksum = PageNeedsChecksumCopy((Page) bufBlock);
+
+ /*
+ * Update page checksum if desired. Since we have only shared lock on
+ * the buffer, other processes might be updating hint bits in it, so
+ * we must copy the page to a bounce buffer if we do checksumming.
+ */
+ if (needs_checksum)
+ {
+ PgAioBounceBuffer *bb = pgaio_bounce_buffer_get();
+
+ pgaio_io_assoc_bounce_buffer(to_write->ioh, bb);
+
+ bufToWrite = pgaio_bounce_buffer_buffer(bb);
+ memcpy(bufToWrite, bufBlock, BLCKSZ);
+ PageSetChecksumInplace((Page) bufToWrite, cur_buf_hdr->tag.blockNum);
+ }
+ else
+ {
+ bufToWrite = bufBlock;
+ }
+
+ to_write->data_ptrs[nbuf] = bufToWrite;
+ }
+
+ pgaio_io_set_handle_data_32(to_write->ioh,
+ (uint32 *) to_write->buffers,
+ to_write->nbuffers);
+ pgaio_io_register_callbacks(to_write->ioh, PGAIO_HCB_SHARED_BUFFER_WRITEV);
+
+ smgrstartwritev(to_write->ioh, smgr,
+ BufTagGetForkNum(&first_buf_hdr->tag),
+ first_buf_hdr->tag.blockNum,
+ to_write->data_ptrs,
+ to_write->nbuffers,
+ false);
+ pgstat_count_io_op(IOOBJECT_RELATION, IOCONTEXT_NORMAL,
+ IOOP_WRITE, 1, BLCKSZ * to_write->nbuffers);
+
+
+ for (int nbuf = 0; nbuf < to_write->nbuffers; nbuf++)
+ {
+ Buffer cur_buf = to_write->buffers[nbuf];
+ BufferDesc *cur_buf_hdr = GetBufferDescriptor(cur_buf - 1);
+
+ UnpinBuffer(cur_buf_hdr);
+ }
+
+ io_queue_track(ioq, &to_write->iow);
+ to_write->total_writes++;
- return result | BUF_WRITTEN;
+ /* clear state for next write */
+ to_write->nbuffers = 0;
+ to_write->start_at_tag.relNumber = InvalidOid;
+ to_write->start_at_tag.blockNum = InvalidBlockNumber;
+ to_write->max_combine = 0;
+ to_write->max_lsn = InvalidXLogRecPtr;
+ to_write->ioh = NULL;
+ pgaio_wref_clear(&to_write->iow);
}
/*
@@ -4088,6 +4566,7 @@ FlushBuffer(BufferDesc *buf, SMgrRelation reln, IOObject io_object,
error_context_stack = errcallback.previous;
}
+
/*
* RelationGetNumberOfBlocksInFork
* Determines the current number of pages in the specified relation fork.
diff --git a/src/backend/storage/page/bufpage.c b/src/backend/storage/page/bufpage.c
index a931cdba151..7fd8e7681ae 100644
--- a/src/backend/storage/page/bufpage.c
+++ b/src/backend/storage/page/bufpage.c
@@ -1480,6 +1480,16 @@ PageIndexTupleOverwrite(Page page, OffsetNumber offnum,
return true;
}
+bool
+PageNeedsChecksumCopy(Page page)
+{
+ if (PageIsNew(page))
+ return false;
+
+ /* If we don't need a checksum, just return the passed-in data */
+ return DataChecksumsEnabled();
+}
+
/*
* Set checksum for a page in shared buffers.
diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list
index 91d8198af9f..bbd08cd6b4d 100644
--- a/src/tools/pgindent/typedefs.list
+++ b/src/tools/pgindent/typedefs.list
@@ -347,6 +347,7 @@ BufferManagerRelation
BufferStrategyControl
BufferTag
BufferUsage
+BuffersToWrite
BuildAccumulator
BuiltinScript
BulkInsertState
--
2.48.1.76.g4e746b1a31.dirty
[text/x-diff] v2.3-0027-very-wip-test_aio-module.patch (42.4K, ../../p5flybfcjrzgpgwcnzkxsp3xdw3czfoafkvks6g7en6ewmvymh@wrz7z2vwwc7c/28-v2.3-0027-very-wip-test_aio-module.patch)
download | inline diff:
From 4507a8ca905a9272bad59198f93e01e40da87451 Mon Sep 17 00:00:00 2001
From: Andres Freund <andres@anarazel.de>
Date: Wed, 22 Jan 2025 13:44:54 -0500
Subject: [PATCH v2.3 27/30] very-wip: test_aio module
Author:
Reviewed-by:
Discussion: https://postgr.es/m/
Backpatch:
---
src/include/storage/aio_internal.h | 8 +
src/include/storage/buf_internals.h | 4 +
src/backend/storage/aio/aio.c | 39 ++
src/backend/storage/buffer/bufmgr.c | 3 +-
src/test/modules/Makefile | 1 +
src/test/modules/meson.build | 1 +
src/test/modules/test_aio/.gitignore | 6 +
src/test/modules/test_aio/Makefile | 34 ++
src/test/modules/test_aio/expected/inject.out | 295 ++++++++++
src/test/modules/test_aio/expected/io.out | 40 ++
.../modules/test_aio/expected/ownership.out | 148 +++++
src/test/modules/test_aio/expected/prep.out | 17 +
src/test/modules/test_aio/io_uring.conf | 5 +
src/test/modules/test_aio/meson.build | 78 +++
src/test/modules/test_aio/sql/inject.sql | 84 +++
src/test/modules/test_aio/sql/io.sql | 16 +
src/test/modules/test_aio/sql/ownership.sql | 65 +++
src/test/modules/test_aio/sql/prep.sql | 9 +
src/test/modules/test_aio/sync.conf | 5 +
src/test/modules/test_aio/test_aio--1.0.sql | 99 ++++
src/test/modules/test_aio/test_aio.c | 504 ++++++++++++++++++
src/test/modules/test_aio/test_aio.control | 3 +
src/test/modules/test_aio/worker.conf | 5 +
23 files changed, 1467 insertions(+), 2 deletions(-)
create mode 100644 src/test/modules/test_aio/.gitignore
create mode 100644 src/test/modules/test_aio/Makefile
create mode 100644 src/test/modules/test_aio/expected/inject.out
create mode 100644 src/test/modules/test_aio/expected/io.out
create mode 100644 src/test/modules/test_aio/expected/ownership.out
create mode 100644 src/test/modules/test_aio/expected/prep.out
create mode 100644 src/test/modules/test_aio/io_uring.conf
create mode 100644 src/test/modules/test_aio/meson.build
create mode 100644 src/test/modules/test_aio/sql/inject.sql
create mode 100644 src/test/modules/test_aio/sql/io.sql
create mode 100644 src/test/modules/test_aio/sql/ownership.sql
create mode 100644 src/test/modules/test_aio/sql/prep.sql
create mode 100644 src/test/modules/test_aio/sync.conf
create mode 100644 src/test/modules/test_aio/test_aio--1.0.sql
create mode 100644 src/test/modules/test_aio/test_aio.c
create mode 100644 src/test/modules/test_aio/test_aio.control
create mode 100644 src/test/modules/test_aio/worker.conf
diff --git a/src/include/storage/aio_internal.h b/src/include/storage/aio_internal.h
index 531532e306a..1855b57f355 100644
--- a/src/include/storage/aio_internal.h
+++ b/src/include/storage/aio_internal.h
@@ -316,6 +316,14 @@ extern const char *pgaio_io_get_target_name(PgAioHandle *ioh);
__VA_ARGS__)
+/* These functions are just for use in tests, from within injection points */
+#ifdef USE_INJECTION_POINTS
+
+extern PgAioHandle *pgaio_inj_io_get(void);
+
+#endif
+
+
/* Declarations for the tables of function pointers exposed by each IO method. */
extern PGDLLIMPORT const IoMethodOps pgaio_sync_ops;
extern PGDLLIMPORT const IoMethodOps pgaio_worker_ops;
diff --git a/src/include/storage/buf_internals.h b/src/include/storage/buf_internals.h
index aeefb1746ec..9939032d5f0 100644
--- a/src/include/storage/buf_internals.h
+++ b/src/include/storage/buf_internals.h
@@ -423,6 +423,10 @@ extern void IssuePendingWritebacks(WritebackContext *wb_context, IOContext io_co
extern void ScheduleBufferTagForWriteback(WritebackContext *wb_context,
IOContext io_context, BufferTag *tag);
+/* solely to make it easier to write tests */
+extern bool StartBufferIO(BufferDesc *buf, bool forInput, bool nowait);
+
+
/* freelist.c */
extern IOContext IOContextForStrategy(BufferAccessStrategy strategy);
extern BufferDesc *StrategyGetBuffer(BufferAccessStrategy strategy,
diff --git a/src/backend/storage/aio/aio.c b/src/backend/storage/aio/aio.c
index 431f2c2e5af..7a873f6ffbb 100644
--- a/src/backend/storage/aio/aio.c
+++ b/src/backend/storage/aio/aio.c
@@ -46,6 +46,10 @@
#include "utils/resowner.h"
#include "utils/wait_event_types.h"
+#ifdef USE_INJECTION_POINTS
+#include "utils/injection_point.h"
+#endif
+
static inline void pgaio_io_update_state(PgAioHandle *ioh, PgAioHandleState new_state);
static void pgaio_io_reclaim(PgAioHandle *ioh);
@@ -92,6 +96,11 @@ static const IoMethodOps *const pgaio_method_ops_table[] = {
const IoMethodOps *pgaio_method_ops;
+#ifdef USE_INJECTION_POINTS
+static PgAioHandle *pgaio_inj_cur_handle;
+#endif
+
+
/* --------------------------------------------------------------------------------
* Public Functions related to PgAioHandle
@@ -452,6 +461,19 @@ pgaio_io_process_completion(PgAioHandle *ioh, int result)
pgaio_io_update_state(ioh, PGAIO_HS_COMPLETED_IO);
+#ifdef USE_INJECTION_POINTS
+ pgaio_inj_cur_handle = ioh;
+
+ /*
+ * FIXME: This could be in a critical section - but it looks like we can't
+ * just InjectionPointLoad() at process start, as the injection point
+ * might not yet be defined.
+ */
+ InjectionPointCached("AIO_PROCESS_COMPLETION_BEFORE_SHARED");
+
+ pgaio_inj_cur_handle = NULL;
+#endif
+
pgaio_io_call_complete_shared(ioh);
pgaio_io_update_state(ioh, PGAIO_HS_COMPLETED_SHARED);
@@ -1128,3 +1150,20 @@ assign_io_method(int newval, void *extra)
pgaio_method_ops = pgaio_method_ops_table[newval];
}
+
+
+
+/* --------------------------------------------------------------------------------
+ * Injection point support
+ * --------------------------------------------------------------------------------
+ */
+
+#ifdef USE_INJECTION_POINTS
+
+PgAioHandle *
+pgaio_inj_io_get(void)
+{
+ return pgaio_inj_cur_handle;
+}
+
+#endif
diff --git a/src/backend/storage/buffer/bufmgr.c b/src/backend/storage/buffer/bufmgr.c
index 1e8793d1630..7f6eabcb92e 100644
--- a/src/backend/storage/buffer/bufmgr.c
+++ b/src/backend/storage/buffer/bufmgr.c
@@ -514,7 +514,6 @@ static void UnpinBufferNoOwner(BufferDesc *buf);
static void BufferSync(int flags);
static uint32 WaitBufHdrUnlocked(BufferDesc *buf);
static void WaitIO(BufferDesc *buf);
-static bool StartBufferIO(BufferDesc *buf, bool forInput, bool nowait);
static void TerminateBufferIO(BufferDesc *buf, bool clear_dirty,
uint32 set_flag_bits, bool forget_owner,
bool syncio);
@@ -6184,7 +6183,7 @@ WaitIO(BufferDesc *buf)
* find out if they can perform the I/O as part of a larger operation, without
* waiting for the answer or distinguishing the reasons why not.
*/
-static bool
+bool
StartBufferIO(BufferDesc *buf, bool forInput, bool nowait)
{
uint32 buf_state;
diff --git a/src/test/modules/Makefile b/src/test/modules/Makefile
index c0d3cf0e14b..73ff9c55687 100644
--- a/src/test/modules/Makefile
+++ b/src/test/modules/Makefile
@@ -13,6 +13,7 @@ SUBDIRS = \
libpq_pipeline \
plsample \
spgist_name_ops \
+ test_aio \
test_bloomfilter \
test_copy_callbacks \
test_custom_rmgrs \
diff --git a/src/test/modules/meson.build b/src/test/modules/meson.build
index 4f544a042d4..b11dd72334c 100644
--- a/src/test/modules/meson.build
+++ b/src/test/modules/meson.build
@@ -1,5 +1,6 @@
# Copyright (c) 2022-2025, PostgreSQL Global Development Group
+subdir('test_aio')
subdir('brin')
subdir('commit_ts')
subdir('delay_execution')
diff --git a/src/test/modules/test_aio/.gitignore b/src/test/modules/test_aio/.gitignore
new file mode 100644
index 00000000000..b4903eba657
--- /dev/null
+++ b/src/test/modules/test_aio/.gitignore
@@ -0,0 +1,6 @@
+# Generated subdirectories
+/log/
+/results/
+/output_iso/
+/tmp_check/
+/tmp_check_iso/
diff --git a/src/test/modules/test_aio/Makefile b/src/test/modules/test_aio/Makefile
new file mode 100644
index 00000000000..ae6d685835b
--- /dev/null
+++ b/src/test/modules/test_aio/Makefile
@@ -0,0 +1,34 @@
+# src/test/modules/delay_execution/Makefile
+
+PGFILEDESC = "test_aio - test code for AIO"
+
+MODULE_big = test_aio
+OBJS = \
+ $(WIN32RES) \
+ test_aio.o
+
+EXTENSION = test_aio
+DATA = test_aio--1.0.sql
+
+REGRESS = prep ownership io
+
+ifeq ($(enable_injection_points),yes)
+REGRESS += inject
+endif
+
+# FIXME: with meson this runs the tests once with worker and once - if
+# supported - with io_uring.
+
+# requires custom config
+NO_INSTALLCHECK = 1
+
+ifdef USE_PGXS
+PG_CONFIG = pg_config
+PGXS := $(shell $(PG_CONFIG) --pgxs)
+include $(PGXS)
+else
+subdir = src/test/modules/test_aio
+top_builddir = ../../../..
+include $(top_builddir)/src/Makefile.global
+include $(top_srcdir)/contrib/contrib-global.mk
+endif
diff --git a/src/test/modules/test_aio/expected/inject.out b/src/test/modules/test_aio/expected/inject.out
new file mode 100644
index 00000000000..e62e3718845
--- /dev/null
+++ b/src/test/modules/test_aio/expected/inject.out
@@ -0,0 +1,295 @@
+SELECT count(*) FROM tbl_b WHERE ctid = '(2, 1)';
+ count
+-------
+ 1
+(1 row)
+
+-- injected what we'd expect
+SELECT inj_io_short_read_attach(8192);
+ inj_io_short_read_attach
+--------------------------
+
+(1 row)
+
+SELECT invalidate_rel_block('tbl_b', 2);
+ invalidate_rel_block
+----------------------
+
+(1 row)
+
+SELECT count(*) FROM tbl_b WHERE ctid = '(2, 1)';
+ count
+-------
+ 1
+(1 row)
+
+SELECT inj_io_short_read_detach();
+ inj_io_short_read_detach
+--------------------------
+
+(1 row)
+
+-- injected a read shorter than a single block, expecting error
+SELECT inj_io_short_read_attach(17);
+ inj_io_short_read_attach
+--------------------------
+
+(1 row)
+
+SELECT invalidate_rel_block('tbl_b', 2);
+ invalidate_rel_block
+----------------------
+
+(1 row)
+
+SELECT redact($$
+ SELECT count(*) FROM tbl_b WHERE ctid = '(2, 1)';
+$$);
+NOTICE: wrapped error: could not read blocks 2..2 in file base/<redacted>: read only 0 of 8192 bytes
+ redact
+--------
+ f
+(1 row)
+
+SELECT inj_io_short_read_detach();
+ inj_io_short_read_detach
+--------------------------
+
+(1 row)
+
+-- shorten multi-block read to a single block, should retry
+SELECT count(*) FROM tbl_b; -- for comparison
+ count
+-------
+ 10000
+(1 row)
+
+SELECT invalidate_rel_block('tbl_b', 0);
+ invalidate_rel_block
+----------------------
+
+(1 row)
+
+SELECT invalidate_rel_block('tbl_b', 1);
+ invalidate_rel_block
+----------------------
+
+(1 row)
+
+SELECT invalidate_rel_block('tbl_b', 2);
+ invalidate_rel_block
+----------------------
+
+(1 row)
+
+SELECT inj_io_short_read_attach(8192);
+ inj_io_short_read_attach
+--------------------------
+
+(1 row)
+
+-- no need to redact, no messages to client
+SELECT count(*) FROM tbl_b;
+ count
+-------
+ 10000
+(1 row)
+
+SELECT inj_io_short_read_detach();
+ inj_io_short_read_detach
+--------------------------
+
+(1 row)
+
+-- shorten multi-block read to 1 1/2 blocks, should retry
+SELECT count(*) FROM tbl_b; -- for comparison
+ count
+-------
+ 10000
+(1 row)
+
+SELECT invalidate_rel_block('tbl_b', 0);
+ invalidate_rel_block
+----------------------
+
+(1 row)
+
+SELECT invalidate_rel_block('tbl_b', 1);
+ invalidate_rel_block
+----------------------
+
+(1 row)
+
+SELECT invalidate_rel_block('tbl_b', 2);
+ invalidate_rel_block
+----------------------
+
+(1 row)
+
+SELECT inj_io_short_read_attach(8192 + 4096);
+ inj_io_short_read_attach
+--------------------------
+
+(1 row)
+
+-- no need to redact, no messages to client
+SELECT count(*) FROM tbl_b;
+ count
+-------
+ 10000
+(1 row)
+
+SELECT inj_io_short_read_detach();
+ inj_io_short_read_detach
+--------------------------
+
+(1 row)
+
+-- shorten single-block read to read that block partially, we'll error out,
+-- because we assume we can read at least one block at a time.
+SELECT count(*) FROM tbl_b WHERE ctid = '(2, 1)'; -- for comparison
+ count
+-------
+ 1
+(1 row)
+
+SELECT invalidate_rel_block('tbl_b', 2);
+ invalidate_rel_block
+----------------------
+
+(1 row)
+
+SELECT inj_io_short_read_attach(4096);
+ inj_io_short_read_attach
+--------------------------
+
+(1 row)
+
+SELECT redact($$
+ SELECT count(*) FROM tbl_b WHERE ctid = '(2, 1)';
+$$);
+NOTICE: wrapped error: could not read blocks 2..2 in file base/<redacted>: read only 0 of 8192 bytes
+ redact
+--------
+ f
+(1 row)
+
+SELECT inj_io_short_read_detach();
+ inj_io_short_read_detach
+--------------------------
+
+(1 row)
+
+-- shorten single-block read to read 0 bytes, expect that to error out
+SELECT count(*) FROM tbl_b WHERE ctid = '(2, 1)'; -- for comparison
+ count
+-------
+ 1
+(1 row)
+
+SELECT invalidate_rel_block('tbl_b', 2);
+ invalidate_rel_block
+----------------------
+
+(1 row)
+
+SELECT inj_io_short_read_attach(0);
+ inj_io_short_read_attach
+--------------------------
+
+(1 row)
+
+SELECT redact($$
+ SELECT count(*) FROM tbl_b WHERE ctid = '(2, 1)';
+$$);
+NOTICE: wrapped error: could not read blocks 2..2 in file base/<redacted>: read only 0 of 8192 bytes
+ redact
+--------
+ f
+(1 row)
+
+SELECT inj_io_short_read_detach();
+ inj_io_short_read_detach
+--------------------------
+
+(1 row)
+
+-- verify that checksum errors are detected even as part of a shortened
+-- multi-block read
+-- (tbl_a, block 1 is corrupted)
+SELECT redact($$
+ SELECT count(*) FROM tbl_a WHERE ctid < '(2, 1)';
+$$);
+NOTICE: wrapped error: invalid page in block 2 of relation base/<redacted>
+ redact
+--------
+ f
+(1 row)
+
+SELECT inj_io_short_read_attach(8192);
+ inj_io_short_read_attach
+--------------------------
+
+(1 row)
+
+SELECT invalidate_rel_block('tbl_a', 0);
+ invalidate_rel_block
+----------------------
+
+(1 row)
+
+SELECT invalidate_rel_block('tbl_a', 1);
+ invalidate_rel_block
+----------------------
+
+(1 row)
+
+SELECT invalidate_rel_block('tbl_a', 2);
+ invalidate_rel_block
+----------------------
+
+(1 row)
+
+SELECT redact($$
+ SELECT count(*) FROM tbl_a WHERE ctid < '(2, 1)';
+$$);
+NOTICE: wrapped error: invalid page in block 2 of relation base/<redacted>
+ redact
+--------
+ f
+(1 row)
+
+SELECT inj_io_short_read_detach();
+ inj_io_short_read_detach
+--------------------------
+
+(1 row)
+
+-- trigger a hard error, should error out
+SELECT inj_io_short_read_attach(-errno_from_string('EIO'));
+ inj_io_short_read_attach
+--------------------------
+
+(1 row)
+
+SELECT invalidate_rel_block('tbl_b', 2);
+ invalidate_rel_block
+----------------------
+
+(1 row)
+
+SELECT redact($$
+ SELECT count(*) FROM tbl_b WHERE ctid = '(2, 1)';
+$$);
+NOTICE: wrapped error: could not read blocks 2..3 in file base/<redacted>: Input/output error
+ redact
+--------
+ f
+(1 row)
+
+SELECT inj_io_short_read_detach();
+ inj_io_short_read_detach
+--------------------------
+
+(1 row)
+
diff --git a/src/test/modules/test_aio/expected/io.out b/src/test/modules/test_aio/expected/io.out
new file mode 100644
index 00000000000..e46b582f290
--- /dev/null
+++ b/src/test/modules/test_aio/expected/io.out
@@ -0,0 +1,40 @@
+SELECT count(*) FROM tbl_a WHERE ctid = '(1, 1)';
+ count
+-------
+ 1
+(1 row)
+
+SELECT corrupt_rel_block('tbl_a', 1);
+ corrupt_rel_block
+-------------------
+
+(1 row)
+
+-- FIXME: Should report the error
+SELECT redact($$
+ SELECT read_corrupt_rel_block('tbl_a', 1);
+$$);
+ redact
+--------
+ t
+(1 row)
+
+-- verify the error is reported
+SELECT redact($$
+ SELECT count(*) FROM tbl_a WHERE ctid = '(1, 1)';
+$$);
+NOTICE: wrapped error: invalid page in block 2 of relation base/<redacted>
+ redact
+--------
+ f
+(1 row)
+
+SELECT redact($$
+ SELECT count(*) FROM tbl_a;
+$$);
+NOTICE: wrapped error: invalid page in block 2 of relation base/<redacted>
+ redact
+--------
+ f
+(1 row)
+
diff --git a/src/test/modules/test_aio/expected/ownership.out b/src/test/modules/test_aio/expected/ownership.out
new file mode 100644
index 00000000000..97fdad6c629
--- /dev/null
+++ b/src/test/modules/test_aio/expected/ownership.out
@@ -0,0 +1,148 @@
+-----
+-- IO handles
+----
+-- leak warning: implicit xact
+SELECT handle_get();
+WARNING: leaked AIO handle
+ handle_get
+------------
+
+(1 row)
+
+-- leak warning: explicit xact
+BEGIN; SELECT handle_get(); COMMIT;
+WARNING: leaked AIO handle
+ handle_get
+------------
+
+(1 row)
+
+-- leak warning + error: released in different command (thus resowner)
+BEGIN; SELECT handle_get(); SELECT handle_release_last(); COMMIT;
+WARNING: leaked AIO handle
+ handle_get
+------------
+
+(1 row)
+
+ERROR: release in unexpected state
+-- no leak, same command
+BEGIN; SELECT handle_get() UNION ALL SELECT handle_release_last(); COMMIT;
+ handle_get
+------------
+
+
+(2 rows)
+
+-- leak warning: subtrans
+BEGIN; SAVEPOINT foo; SELECT handle_get(); COMMIT;
+WARNING: leaked AIO handle
+ handle_get
+------------
+
+(1 row)
+
+-- normal handle use
+SELECT handle_get_release();
+ handle_get_release
+--------------------
+
+(1 row)
+
+-- should error out, API violation
+SELECT handle_get_twice();
+ERROR: API violation: Only one IO can be handed out
+-- recover after error in implicit xact
+SELECT handle_get_and_error(); SELECT handle_get_release();
+ERROR: as you command
+ handle_get_release
+--------------------
+
+(1 row)
+
+-- recover after error in explicit xact
+BEGIN; SELECT handle_get_and_error(); ROLLBACK; SELECT handle_get_release();
+ERROR: as you command
+ handle_get_release
+--------------------
+
+(1 row)
+
+-- recover after error in subtrans
+BEGIN; SAVEPOINT foo; SELECT handle_get_and_error(); ROLLBACK TO SAVEPOINT foo; SELECT handle_get_release(); ROLLBACK;
+ERROR: as you command
+ handle_get_release
+--------------------
+
+(1 row)
+
+-----
+-- Bounce Buffers handles
+----
+-- leak warning: implicit xact
+SELECT bb_get();
+WARNING: leaked AIO bounce buffer
+ bb_get
+--------
+
+(1 row)
+
+-- leak warning: explicit xact
+BEGIN; SELECT bb_get(); COMMIT;
+WARNING: leaked AIO bounce buffer
+ bb_get
+--------
+
+(1 row)
+
+-- missing leak warning: we should warn at command boundaries, not xact boundaries
+BEGIN; SELECT bb_get(); SELECT bb_release_last(); COMMIT;
+WARNING: leaked AIO bounce buffer
+ bb_get
+--------
+
+(1 row)
+
+ERROR: can only release handed out BB
+-- leak warning: subtrans
+BEGIN; SAVEPOINT foo; SELECT bb_get(); COMMIT;
+WARNING: leaked AIO bounce buffer
+ bb_get
+--------
+
+(1 row)
+
+-- normal bb use
+SELECT bb_get_release();
+ bb_get_release
+----------------
+
+(1 row)
+
+-- should error out, API violation
+SELECT bb_get_twice();
+ERROR: can only hand out one BB
+-- recover after error in implicit xact
+SELECT bb_get_and_error(); SELECT bb_get_release();
+ERROR: as you command
+ bb_get_release
+----------------
+
+(1 row)
+
+-- recover after error in explicit xact
+BEGIN; SELECT bb_get_and_error(); ROLLBACK; SELECT bb_get_release();
+ERROR: as you command
+ bb_get_release
+----------------
+
+(1 row)
+
+-- recover after error in subtrans
+BEGIN; SAVEPOINT foo; SELECT bb_get_and_error(); ROLLBACK TO SAVEPOINT foo; SELECT bb_get_release(); ROLLBACK;
+ERROR: as you command
+ bb_get_release
+----------------
+
+(1 row)
+
diff --git a/src/test/modules/test_aio/expected/prep.out b/src/test/modules/test_aio/expected/prep.out
new file mode 100644
index 00000000000..7fad6280db5
--- /dev/null
+++ b/src/test/modules/test_aio/expected/prep.out
@@ -0,0 +1,17 @@
+CREATE EXTENSION test_aio;
+CREATE TABLE tbl_a(data int not null);
+CREATE TABLE tbl_b(data int not null);
+INSERT INTO tbl_a SELECT generate_series(1, 10000);
+INSERT INTO tbl_b SELECT generate_series(1, 10000);
+SELECT grow_rel('tbl_a', 500);
+ grow_rel
+----------
+
+(1 row)
+
+SELECT grow_rel('tbl_b', 550);
+ grow_rel
+----------
+
+(1 row)
+
diff --git a/src/test/modules/test_aio/io_uring.conf b/src/test/modules/test_aio/io_uring.conf
new file mode 100644
index 00000000000..efd7ad143ff
--- /dev/null
+++ b/src/test/modules/test_aio/io_uring.conf
@@ -0,0 +1,5 @@
+shared_preload_libraries=test_aio
+io_method = 'io_uring'
+log_min_messages = 'DEBUG3'
+log_statement=all
+restart_after_crash=false
diff --git a/src/test/modules/test_aio/meson.build b/src/test/modules/test_aio/meson.build
new file mode 100644
index 00000000000..a4bef0ceeb0
--- /dev/null
+++ b/src/test/modules/test_aio/meson.build
@@ -0,0 +1,78 @@
+# Copyright (c) 2022-2024, PostgreSQL Global Development Group
+
+test_aio_sources = files(
+ 'test_aio.c',
+)
+
+if host_system == 'windows'
+ test_aio_sources += rc_lib_gen.process(win32ver_rc, extra_args: [
+ '--NAME', 'test_aio',
+ '--FILEDESC', 'test_aio - test code for AIO',])
+endif
+
+test_aio = shared_module('test_aio',
+ test_aio_sources,
+ kwargs: pg_test_mod_args,
+)
+test_install_libs += test_aio
+
+test_install_data += files(
+ 'test_aio.control',
+ 'test_aio--1.0.sql',
+)
+
+
+testfiles = [
+ 'prep',
+ 'ownership',
+ 'io',
+]
+
+if get_option('injection_points')
+ testfiles += 'inject'
+endif
+
+
+tests += {
+ 'name': 'test_aio_sync',
+ 'sd': meson.current_source_dir(),
+ 'bd': meson.current_build_dir(),
+ 'regress': {
+ 'sql': testfiles,
+ 'regress_args': [
+ '--temp-config', files('sync.conf'),
+ ],
+ # requires custom config
+ 'runningcheck': false,
+ },
+}
+
+tests += {
+ 'name': 'test_aio_worker',
+ 'sd': meson.current_source_dir(),
+ 'bd': meson.current_build_dir(),
+ 'regress': {
+ 'sql': testfiles,
+ 'regress_args': [
+ '--temp-config', files('worker.conf'),
+ ],
+ # requires custom config
+ 'runningcheck': false,
+ },
+}
+
+if liburing.found()
+ tests += {
+ 'name': 'test_aio_uring',
+ 'sd': meson.current_source_dir(),
+ 'bd': meson.current_build_dir(),
+ 'regress': {
+ 'sql': testfiles,
+ 'regress_args': [
+ '--temp-config', files('io_uring.conf'),
+ ],
+ # requires custom config
+ 'runningcheck': false,
+ }
+ }
+endif
diff --git a/src/test/modules/test_aio/sql/inject.sql b/src/test/modules/test_aio/sql/inject.sql
new file mode 100644
index 00000000000..1190531f5ad
--- /dev/null
+++ b/src/test/modules/test_aio/sql/inject.sql
@@ -0,0 +1,84 @@
+SELECT count(*) FROM tbl_b WHERE ctid = '(2, 1)';
+
+-- injected what we'd expect
+SELECT inj_io_short_read_attach(8192);
+SELECT invalidate_rel_block('tbl_b', 2);
+SELECT count(*) FROM tbl_b WHERE ctid = '(2, 1)';
+SELECT inj_io_short_read_detach();
+
+
+-- injected a read shorter than a single block, expecting error
+SELECT inj_io_short_read_attach(17);
+SELECT invalidate_rel_block('tbl_b', 2);
+SELECT redact($$
+ SELECT count(*) FROM tbl_b WHERE ctid = '(2, 1)';
+$$);
+SELECT inj_io_short_read_detach();
+
+
+-- shorten multi-block read to a single block, should retry
+SELECT count(*) FROM tbl_b; -- for comparison
+SELECT invalidate_rel_block('tbl_b', 0);
+SELECT invalidate_rel_block('tbl_b', 1);
+SELECT invalidate_rel_block('tbl_b', 2);
+SELECT inj_io_short_read_attach(8192);
+-- no need to redact, no messages to client
+SELECT count(*) FROM tbl_b;
+SELECT inj_io_short_read_detach();
+
+
+-- shorten multi-block read to 1 1/2 blocks, should retry
+SELECT count(*) FROM tbl_b; -- for comparison
+SELECT invalidate_rel_block('tbl_b', 0);
+SELECT invalidate_rel_block('tbl_b', 1);
+SELECT invalidate_rel_block('tbl_b', 2);
+SELECT inj_io_short_read_attach(8192 + 4096);
+-- no need to redact, no messages to client
+SELECT count(*) FROM tbl_b;
+SELECT inj_io_short_read_detach();
+
+
+-- shorten single-block read to read that block partially, we'll error out,
+-- because we assume we can read at least one block at a time.
+SELECT count(*) FROM tbl_b WHERE ctid = '(2, 1)'; -- for comparison
+SELECT invalidate_rel_block('tbl_b', 2);
+SELECT inj_io_short_read_attach(4096);
+SELECT redact($$
+ SELECT count(*) FROM tbl_b WHERE ctid = '(2, 1)';
+$$);
+SELECT inj_io_short_read_detach();
+
+
+-- shorten single-block read to read 0 bytes, expect that to error out
+SELECT count(*) FROM tbl_b WHERE ctid = '(2, 1)'; -- for comparison
+SELECT invalidate_rel_block('tbl_b', 2);
+SELECT inj_io_short_read_attach(0);
+SELECT redact($$
+ SELECT count(*) FROM tbl_b WHERE ctid = '(2, 1)';
+$$);
+SELECT inj_io_short_read_detach();
+
+
+-- verify that checksum errors are detected even as part of a shortened
+-- multi-block read
+-- (tbl_a, block 1 is corrupted)
+SELECT redact($$
+ SELECT count(*) FROM tbl_a WHERE ctid < '(2, 1)';
+$$);
+SELECT inj_io_short_read_attach(8192);
+SELECT invalidate_rel_block('tbl_a', 0);
+SELECT invalidate_rel_block('tbl_a', 1);
+SELECT invalidate_rel_block('tbl_a', 2);
+SELECT redact($$
+ SELECT count(*) FROM tbl_a WHERE ctid < '(2, 1)';
+$$);
+SELECT inj_io_short_read_detach();
+
+
+-- trigger a hard error, should error out
+SELECT inj_io_short_read_attach(-errno_from_string('EIO'));
+SELECT invalidate_rel_block('tbl_b', 2);
+SELECT redact($$
+ SELECT count(*) FROM tbl_b WHERE ctid = '(2, 1)';
+$$);
+SELECT inj_io_short_read_detach();
diff --git a/src/test/modules/test_aio/sql/io.sql b/src/test/modules/test_aio/sql/io.sql
new file mode 100644
index 00000000000..a29bb4eb15d
--- /dev/null
+++ b/src/test/modules/test_aio/sql/io.sql
@@ -0,0 +1,16 @@
+SELECT count(*) FROM tbl_a WHERE ctid = '(1, 1)';
+
+SELECT corrupt_rel_block('tbl_a', 1);
+
+-- FIXME: Should report the error
+SELECT redact($$
+ SELECT read_corrupt_rel_block('tbl_a', 1);
+$$);
+
+-- verify the error is reported
+SELECT redact($$
+ SELECT count(*) FROM tbl_a WHERE ctid = '(1, 1)';
+$$);
+SELECT redact($$
+ SELECT count(*) FROM tbl_a;
+$$);
diff --git a/src/test/modules/test_aio/sql/ownership.sql b/src/test/modules/test_aio/sql/ownership.sql
new file mode 100644
index 00000000000..63cf40c802a
--- /dev/null
+++ b/src/test/modules/test_aio/sql/ownership.sql
@@ -0,0 +1,65 @@
+-----
+-- IO handles
+----
+
+-- leak warning: implicit xact
+SELECT handle_get();
+
+-- leak warning: explicit xact
+BEGIN; SELECT handle_get(); COMMIT;
+
+-- leak warning + error: released in different command (thus resowner)
+BEGIN; SELECT handle_get(); SELECT handle_release_last(); COMMIT;
+
+-- no leak, same command
+BEGIN; SELECT handle_get() UNION ALL SELECT handle_release_last(); COMMIT;
+
+-- leak warning: subtrans
+BEGIN; SAVEPOINT foo; SELECT handle_get(); COMMIT;
+
+-- normal handle use
+SELECT handle_get_release();
+
+-- should error out, API violation
+SELECT handle_get_twice();
+
+-- recover after error in implicit xact
+SELECT handle_get_and_error(); SELECT handle_get_release();
+
+-- recover after error in explicit xact
+BEGIN; SELECT handle_get_and_error(); ROLLBACK; SELECT handle_get_release();
+
+-- recover after error in subtrans
+BEGIN; SAVEPOINT foo; SELECT handle_get_and_error(); ROLLBACK TO SAVEPOINT foo; SELECT handle_get_release(); ROLLBACK;
+
+
+-----
+-- Bounce Buffers handles
+----
+
+-- leak warning: implicit xact
+SELECT bb_get();
+
+-- leak warning: explicit xact
+BEGIN; SELECT bb_get(); COMMIT;
+
+-- missing leak warning: we should warn at command boundaries, not xact boundaries
+BEGIN; SELECT bb_get(); SELECT bb_release_last(); COMMIT;
+
+-- leak warning: subtrans
+BEGIN; SAVEPOINT foo; SELECT bb_get(); COMMIT;
+
+-- normal bb use
+SELECT bb_get_release();
+
+-- should error out, API violation
+SELECT bb_get_twice();
+
+-- recover after error in implicit xact
+SELECT bb_get_and_error(); SELECT bb_get_release();
+
+-- recover after error in explicit xact
+BEGIN; SELECT bb_get_and_error(); ROLLBACK; SELECT bb_get_release();
+
+-- recover after error in subtrans
+BEGIN; SAVEPOINT foo; SELECT bb_get_and_error(); ROLLBACK TO SAVEPOINT foo; SELECT bb_get_release(); ROLLBACK;
diff --git a/src/test/modules/test_aio/sql/prep.sql b/src/test/modules/test_aio/sql/prep.sql
new file mode 100644
index 00000000000..b8f225cbc98
--- /dev/null
+++ b/src/test/modules/test_aio/sql/prep.sql
@@ -0,0 +1,9 @@
+CREATE EXTENSION test_aio;
+
+CREATE TABLE tbl_a(data int not null);
+CREATE TABLE tbl_b(data int not null);
+
+INSERT INTO tbl_a SELECT generate_series(1, 10000);
+INSERT INTO tbl_b SELECT generate_series(1, 10000);
+SELECT grow_rel('tbl_a', 500);
+SELECT grow_rel('tbl_b', 550);
diff --git a/src/test/modules/test_aio/sync.conf b/src/test/modules/test_aio/sync.conf
new file mode 100644
index 00000000000..c480922d6cf
--- /dev/null
+++ b/src/test/modules/test_aio/sync.conf
@@ -0,0 +1,5 @@
+shared_preload_libraries=test_aio
+io_method = 'sync'
+log_min_messages = 'DEBUG3'
+log_statement=all
+restart_after_crash=false
diff --git a/src/test/modules/test_aio/test_aio--1.0.sql b/src/test/modules/test_aio/test_aio--1.0.sql
new file mode 100644
index 00000000000..e3d5ce29c60
--- /dev/null
+++ b/src/test/modules/test_aio/test_aio--1.0.sql
@@ -0,0 +1,99 @@
+/* src/test/modules/test_aio/test_aio--1.0.sql */
+
+-- complain if script is sourced in psql, rather than via CREATE EXTENSION
+\echo Use "CREATE EXTENSION test_aio" to load this file. \quit
+
+
+CREATE FUNCTION errno_from_string(sym text)
+RETURNS pg_catalog.int4 STRICT
+AS 'MODULE_PATHNAME' LANGUAGE C;
+
+
+CREATE FUNCTION grow_rel(rel regclass, nblocks int)
+RETURNS pg_catalog.void STRICT
+AS 'MODULE_PATHNAME' LANGUAGE C;
+
+
+CREATE FUNCTION corrupt_rel_block(rel regclass, blockno int)
+RETURNS pg_catalog.void STRICT
+AS 'MODULE_PATHNAME' LANGUAGE C;
+
+CREATE FUNCTION read_corrupt_rel_block(rel regclass, blockno int)
+RETURNS pg_catalog.void STRICT
+AS 'MODULE_PATHNAME' LANGUAGE C;
+
+CREATE FUNCTION invalidate_rel_block(rel regclass, blockno int)
+RETURNS pg_catalog.void STRICT
+AS 'MODULE_PATHNAME' LANGUAGE C;
+
+CREATE FUNCTION handle_get_and_error()
+RETURNS pg_catalog.void STRICT
+AS 'MODULE_PATHNAME' LANGUAGE C;
+
+CREATE FUNCTION handle_get_twice()
+RETURNS pg_catalog.void STRICT
+AS 'MODULE_PATHNAME' LANGUAGE C;
+
+CREATE FUNCTION handle_get()
+RETURNS pg_catalog.void STRICT
+AS 'MODULE_PATHNAME' LANGUAGE C;
+
+CREATE FUNCTION handle_get_release()
+RETURNS pg_catalog.void STRICT
+AS 'MODULE_PATHNAME' LANGUAGE C;
+
+CREATE FUNCTION handle_release_last()
+RETURNS pg_catalog.void STRICT
+AS 'MODULE_PATHNAME' LANGUAGE C;
+
+
+CREATE FUNCTION bb_get_and_error()
+RETURNS pg_catalog.void STRICT
+AS 'MODULE_PATHNAME' LANGUAGE C;
+
+CREATE FUNCTION bb_get_twice()
+RETURNS pg_catalog.void STRICT
+AS 'MODULE_PATHNAME' LANGUAGE C;
+
+CREATE FUNCTION bb_get()
+RETURNS pg_catalog.void STRICT
+AS 'MODULE_PATHNAME' LANGUAGE C;
+
+CREATE FUNCTION bb_get_release()
+RETURNS pg_catalog.void STRICT
+AS 'MODULE_PATHNAME' LANGUAGE C;
+
+CREATE FUNCTION bb_release_last()
+RETURNS pg_catalog.void STRICT
+AS 'MODULE_PATHNAME' LANGUAGE C;
+
+
+CREATE OR REPLACE FUNCTION redact(p_sql text)
+RETURNS bool
+LANGUAGE plpgsql
+AS $$
+ DECLARE
+ err_state text;
+ err_msg text;
+ BEGIN
+ EXECUTE p_sql;
+ RETURN true;
+ EXCEPTION WHEN OTHERS THEN
+ GET STACKED DIAGNOSTICS
+ err_state = RETURNED_SQLSTATE,
+ err_msg = MESSAGE_TEXT;
+ err_msg = regexp_replace(err_msg, '(file|relation) "?base/[0-9]+/[0-9]+"?', '\1 base/<redacted>');
+ RAISE NOTICE 'wrapped error: %', err_msg
+ USING ERRCODE = err_state;
+ RETURN false;
+ END;
+$$;
+
+
+CREATE FUNCTION inj_io_short_read_attach(result int)
+RETURNS pg_catalog.void STRICT
+AS 'MODULE_PATHNAME' LANGUAGE C;
+
+CREATE FUNCTION inj_io_short_read_detach()
+RETURNS pg_catalog.void STRICT
+AS 'MODULE_PATHNAME' LANGUAGE C;
diff --git a/src/test/modules/test_aio/test_aio.c b/src/test/modules/test_aio/test_aio.c
new file mode 100644
index 00000000000..20d7e6dc82f
--- /dev/null
+++ b/src/test/modules/test_aio/test_aio.c
@@ -0,0 +1,504 @@
+/*-------------------------------------------------------------------------
+ *
+ * delay_execution.c
+ * Test module to allow delay between parsing and execution of a query.
+ *
+ * The delay is implemented by taking and immediately releasing a specified
+ * advisory lock. If another process has previously taken that lock, the
+ * current process will be blocked until the lock is released; otherwise,
+ * there's no effect. This allows an isolationtester script to reliably
+ * test behaviors where some specified action happens in another backend
+ * between parsing and execution of any desired query.
+ *
+ * Copyright (c) 2020-2024, PostgreSQL Global Development Group
+ *
+ * IDENTIFICATION
+ * src/test/modules/delay_execution/delay_execution.c
+ *
+ *-------------------------------------------------------------------------
+ */
+
+#include "postgres.h"
+
+#include "access/relation.h"
+#include "fmgr.h"
+#include "storage/aio.h"
+#include "storage/aio_internal.h"
+#include "storage/buf_internals.h"
+#include "storage/bufmgr.h"
+#include "storage/ipc.h"
+#include "storage/lwlock.h"
+#include "utils/builtins.h"
+#include "utils/injection_point.h"
+#include "utils/rel.h"
+
+
+PG_MODULE_MAGIC;
+
+
+typedef struct InjIoErrorState
+{
+ bool enabled;
+ bool result_set;
+ int result;
+} InjIoErrorState;
+
+static InjIoErrorState * inj_io_error_state;
+
+/* Shared memory init callbacks */
+static shmem_request_hook_type prev_shmem_request_hook = NULL;
+static shmem_startup_hook_type prev_shmem_startup_hook = NULL;
+
+
+static PgAioHandle *last_handle;
+static PgAioBounceBuffer *last_bb;
+
+
+
+static void
+test_aio_shmem_request(void)
+{
+ if (prev_shmem_request_hook)
+ prev_shmem_request_hook();
+
+ RequestAddinShmemSpace(sizeof(InjIoErrorState));
+}
+
+static void
+test_aio_shmem_startup(void)
+{
+ bool found;
+
+ if (prev_shmem_startup_hook)
+ prev_shmem_startup_hook();
+
+ /* Create or attach to the shared memory state */
+ LWLockAcquire(AddinShmemInitLock, LW_EXCLUSIVE);
+
+ inj_io_error_state = ShmemInitStruct("injection_points",
+ sizeof(InjIoErrorState),
+ &found);
+
+ if (!found)
+ {
+ /*
+ * First time through, so initialize. This is shared with the dynamic
+ * initialization using a DSM.
+ */
+ inj_io_error_state->enabled = false;
+
+#ifdef USE_INJECTION_POINTS
+ InjectionPointAttach("AIO_PROCESS_COMPLETION_BEFORE_SHARED",
+ "test_aio",
+ "inj_io_short_read",
+ NULL,
+ 0);
+ InjectionPointLoad("AIO_PROCESS_COMPLETION_BEFORE_SHARED");
+#endif
+ }
+ else
+ {
+#ifdef USE_INJECTION_POINTS
+ InjectionPointLoad("AIO_PROCESS_COMPLETION_BEFORE_SHARED");
+ elog(LOG, "injection point loaded");
+#endif
+ }
+
+ LWLockRelease(AddinShmemInitLock);
+}
+
+void
+_PG_init(void)
+{
+ if (!process_shared_preload_libraries_in_progress)
+ return;
+
+ /* Shared memory initialization */
+ prev_shmem_request_hook = shmem_request_hook;
+ shmem_request_hook = test_aio_shmem_request;
+ prev_shmem_startup_hook = shmem_startup_hook;
+ shmem_startup_hook = test_aio_shmem_startup;
+}
+
+
+PG_FUNCTION_INFO_V1(errno_from_string);
+Datum
+errno_from_string(PG_FUNCTION_ARGS)
+{
+ const char *sym = text_to_cstring(PG_GETARG_TEXT_PP(0));
+
+ if (strcmp(sym, "EIO") == 0)
+ PG_RETURN_INT32(EIO);
+ else if (strcmp(sym, "EAGAIN") == 0)
+ PG_RETURN_INT32(EAGAIN);
+ else if (strcmp(sym, "EINTR") == 0)
+ PG_RETURN_INT32(EINTR);
+ else if (strcmp(sym, "ENOSPC") == 0)
+ PG_RETURN_INT32(ENOSPC);
+ else if (strcmp(sym, "EROFS") == 0)
+ PG_RETURN_INT32(EROFS);
+
+ ereport(ERROR,
+ errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ errmsg_internal("%s is not a supported errno value", sym));
+ PG_RETURN_INT32(0);
+}
+
+
+PG_FUNCTION_INFO_V1(grow_rel);
+Datum
+grow_rel(PG_FUNCTION_ARGS)
+{
+ Oid relid = PG_GETARG_OID(0);
+ uint32 nblocks = PG_GETARG_UINT32(1);
+ Relation rel;
+#define MAX_BUFFERS_TO_EXTEND_BY 64
+ Buffer victim_buffers[MAX_BUFFERS_TO_EXTEND_BY];
+
+ rel = relation_open(relid, AccessExclusiveLock);
+
+ while (nblocks > 0)
+ {
+ uint32 extend_by_pages;
+
+ extend_by_pages = Min(nblocks, MAX_BUFFERS_TO_EXTEND_BY);
+
+ ExtendBufferedRelBy(BMR_REL(rel),
+ MAIN_FORKNUM,
+ NULL,
+ 0,
+ extend_by_pages,
+ victim_buffers,
+ &extend_by_pages);
+
+ nblocks -= extend_by_pages;
+
+ for (uint32 i = 0; i < extend_by_pages; i++)
+ {
+ ReleaseBuffer(victim_buffers[i]);
+ }
+ }
+
+ relation_close(rel, NoLock);
+
+ PG_RETURN_VOID();
+}
+
+PG_FUNCTION_INFO_V1(corrupt_rel_block);
+Datum
+corrupt_rel_block(PG_FUNCTION_ARGS)
+{
+ Oid relid = PG_GETARG_OID(0);
+ uint32 block = PG_GETARG_UINT32(1);
+ Relation rel;
+ Buffer buf;
+ Page page;
+ PageHeader ph;
+
+ rel = relation_open(relid, AccessExclusiveLock);
+
+ buf = ReadBuffer(rel, block);
+ page = BufferGetPage(buf);
+
+ LockBuffer(buf, BUFFER_LOCK_EXCLUSIVE);
+
+ MarkBufferDirty(buf);
+
+ PageInit(page, BufferGetPageSize(buf), 0);
+
+ ph = (PageHeader) page;
+ ph->pd_special = BLCKSZ + 1;
+
+ FlushOneBuffer(buf);
+
+ LockBuffer(buf, BUFFER_LOCK_UNLOCK);
+
+ ReleaseBuffer(buf);
+
+ EvictUnpinnedBuffer(buf);
+
+ relation_close(rel, NoLock);
+
+ PG_RETURN_VOID();
+}
+
+PG_FUNCTION_INFO_V1(read_corrupt_rel_block);
+Datum
+read_corrupt_rel_block(PG_FUNCTION_ARGS)
+{
+ Oid relid = PG_GETARG_OID(0);
+ uint32 block = PG_GETARG_UINT32(1);
+ Relation rel;
+ Buffer buf;
+ BufferDesc *buf_hdr;
+ Page page;
+ PgAioHandle *ioh;
+ PgAioWaitRef iow;
+ SMgrRelation smgr;
+ uint32 buf_state;
+
+ rel = relation_open(relid, AccessExclusiveLock);
+
+ /* read buffer without erroring out */
+ buf = ReadBufferExtended(rel, MAIN_FORKNUM, block, RBM_ZERO_AND_LOCK, NULL);
+ LockBuffer(buf, BUFFER_LOCK_UNLOCK);
+
+ page = BufferGetBlock(buf);
+
+ ioh = pgaio_io_acquire(CurrentResourceOwner, NULL);
+ pgaio_io_get_wref(ioh, &iow);
+
+ buf_hdr = GetBufferDescriptor(buf - 1);
+ smgr = RelationGetSmgr(rel);
+
+ /* FIXME: even if just a test, we should verify nobody else uses this */
+ buf_state = LockBufHdr(buf_hdr);
+ buf_state &= ~(BM_VALID | BM_DIRTY);
+ UnlockBufHdr(buf_hdr, buf_state);
+
+ StartBufferIO(buf_hdr, true, false);
+
+ pgaio_io_set_handle_data_32(ioh, (uint32 *) &buf, 1);
+ pgaio_io_register_callbacks(ioh, PGAIO_HCB_SHARED_BUFFER_READV);
+
+ smgrstartreadv(ioh, smgr, MAIN_FORKNUM, block,
+ (void *) &page, 1);
+
+ ReleaseBuffer(buf);
+
+ pgaio_wref_wait(&iow);
+
+ relation_close(rel, NoLock);
+
+ PG_RETURN_VOID();
+}
+
+PG_FUNCTION_INFO_V1(invalidate_rel_block);
+Datum
+invalidate_rel_block(PG_FUNCTION_ARGS)
+{
+ Oid relid = PG_GETARG_OID(0);
+ uint32 block = PG_GETARG_UINT32(1);
+ Relation rel;
+ PrefetchBufferResult pr;
+ Buffer buf;
+
+ rel = relation_open(relid, AccessExclusiveLock);
+
+ /* this is a gross hack, but there's no good API exposed */
+ pr = PrefetchBuffer(rel, MAIN_FORKNUM, block);
+ buf = pr.recent_buffer;
+ elog(LOG, "recent: %d", buf);
+ if (BufferIsValid(buf))
+ {
+ /* if the buffer contents aren't valid, this'll return false */
+ if (ReadRecentBuffer(rel->rd_locator, MAIN_FORKNUM, block, buf))
+ {
+ LockBuffer(buf, BUFFER_LOCK_EXCLUSIVE);
+ FlushOneBuffer(buf);
+ LockBuffer(buf, BUFFER_LOCK_UNLOCK);
+ ReleaseBuffer(buf);
+
+ if (!EvictUnpinnedBuffer(buf))
+ elog(ERROR, "couldn't unpin");
+ }
+ }
+
+ relation_close(rel, AccessExclusiveLock);
+
+ PG_RETURN_VOID();
+}
+
+#if 0
+PG_FUNCTION_INFO_V1(test_unsubmitted_vs_close);
+Datum
+test_unsubmitted_vs_close(PG_FUNCTION_ARGS)
+{
+ Oid relid = PG_GETARG_OID(0);
+ uint32 block = PG_GETARG_UINT32(1);
+ Relation rel;
+ Buffer buf;
+ Page page;
+ PageHeader ph;
+
+ rel = relation_open(relid, AccessExclusiveLock);
+
+ buf = ReadBufferExtended(rel, MAIN_FORKNUM, block, RBM_ZERO_AND_LOCK, NULL);
+
+ buf = ReadBuffer(rel, block);
+ page = BufferGetPage(buf);
+
+ EvictUnpinnedBuffer(buf);
+
+ LockBuffer(buf, BUFFER_LOCK_UNLOCK);
+
+
+ MarkBufferDirty(buf);
+ ph->pd_special = BLCKSZ + 1;
+
+ /* last_handle = pgaio_io_acquire(); */
+
+ PG_RETURN_VOID();
+}
+#endif
+
+PG_FUNCTION_INFO_V1(handle_get);
+Datum
+handle_get(PG_FUNCTION_ARGS)
+{
+ last_handle = pgaio_io_acquire(CurrentResourceOwner, NULL);
+
+ PG_RETURN_VOID();
+}
+
+PG_FUNCTION_INFO_V1(handle_release_last);
+Datum
+handle_release_last(PG_FUNCTION_ARGS)
+{
+ if (!last_handle)
+ elog(ERROR, "no handle");
+
+ pgaio_io_release(last_handle);
+
+ PG_RETURN_VOID();
+}
+
+PG_FUNCTION_INFO_V1(handle_get_and_error);
+Datum
+handle_get_and_error(PG_FUNCTION_ARGS)
+{
+ pgaio_io_acquire(CurrentResourceOwner, NULL);
+
+ elog(ERROR, "as you command");
+ PG_RETURN_VOID();
+}
+
+PG_FUNCTION_INFO_V1(handle_get_twice);
+Datum
+handle_get_twice(PG_FUNCTION_ARGS)
+{
+ pgaio_io_acquire(CurrentResourceOwner, NULL);
+ pgaio_io_acquire(CurrentResourceOwner, NULL);
+
+ PG_RETURN_VOID();
+}
+
+
+PG_FUNCTION_INFO_V1(handle_get_release);
+Datum
+handle_get_release(PG_FUNCTION_ARGS)
+{
+ PgAioHandle *handle;
+
+ handle = pgaio_io_acquire(CurrentResourceOwner, NULL);
+ pgaio_io_release(handle);
+
+ PG_RETURN_VOID();
+}
+
+PG_FUNCTION_INFO_V1(bb_get);
+Datum
+bb_get(PG_FUNCTION_ARGS)
+{
+ last_bb = pgaio_bounce_buffer_get();
+
+ PG_RETURN_VOID();
+}
+
+PG_FUNCTION_INFO_V1(bb_release_last);
+Datum
+bb_release_last(PG_FUNCTION_ARGS)
+{
+ if (!last_bb)
+ elog(ERROR, "no bb");
+
+ pgaio_bounce_buffer_release(last_bb);
+
+ PG_RETURN_VOID();
+}
+
+PG_FUNCTION_INFO_V1(bb_get_and_error);
+Datum
+bb_get_and_error(PG_FUNCTION_ARGS)
+{
+ pgaio_bounce_buffer_get();
+
+ elog(ERROR, "as you command");
+ PG_RETURN_VOID();
+}
+
+PG_FUNCTION_INFO_V1(bb_get_twice);
+Datum
+bb_get_twice(PG_FUNCTION_ARGS)
+{
+ pgaio_bounce_buffer_get();
+ pgaio_bounce_buffer_get();
+
+ PG_RETURN_VOID();
+}
+
+
+PG_FUNCTION_INFO_V1(bb_get_release);
+Datum
+bb_get_release(PG_FUNCTION_ARGS)
+{
+ PgAioBounceBuffer *bb;
+
+ bb = pgaio_bounce_buffer_get();
+ pgaio_bounce_buffer_release(bb);
+
+ PG_RETURN_VOID();
+}
+
+#ifdef USE_INJECTION_POINTS
+extern PGDLLEXPORT void inj_io_short_read(const char *name, const void *private_data);
+
+void
+inj_io_short_read(const char *name, const void *private_data)
+{
+ PgAioHandle *ioh;
+
+ elog(LOG, "short read called: %d", inj_io_error_state->enabled);
+
+ if (inj_io_error_state->enabled)
+ {
+ ioh = pgaio_inj_io_get();
+
+ if (inj_io_error_state->result_set)
+ {
+ elog(LOG, "short read, changing result from %d to %d",
+ ioh->result, inj_io_error_state->result);
+
+ ioh->result = inj_io_error_state->result;
+ }
+ }
+}
+#endif
+
+PG_FUNCTION_INFO_V1(inj_io_short_read_attach);
+Datum
+inj_io_short_read_attach(PG_FUNCTION_ARGS)
+{
+#ifdef USE_INJECTION_POINTS
+ inj_io_error_state->enabled = true;
+ inj_io_error_state->result_set = !PG_ARGISNULL(0);
+ if (inj_io_error_state->result_set)
+ inj_io_error_state->result = PG_GETARG_INT32(0);
+#else
+ elog(ERROR, "injection points not supported");
+#endif
+
+ PG_RETURN_VOID();
+}
+
+PG_FUNCTION_INFO_V1(inj_io_short_read_detach);
+Datum
+inj_io_short_read_detach(PG_FUNCTION_ARGS)
+{
+#ifdef USE_INJECTION_POINTS
+ inj_io_error_state->enabled = false;
+#else
+ elog(ERROR, "injection points not supported");
+#endif
+ PG_RETURN_VOID();
+}
diff --git a/src/test/modules/test_aio/test_aio.control b/src/test/modules/test_aio/test_aio.control
new file mode 100644
index 00000000000..cd91c3ed16b
--- /dev/null
+++ b/src/test/modules/test_aio/test_aio.control
@@ -0,0 +1,3 @@
+comment = 'Test code for AIO'
+default_version = '1.0'
+module_pathname = '$libdir/test_aio'
diff --git a/src/test/modules/test_aio/worker.conf b/src/test/modules/test_aio/worker.conf
new file mode 100644
index 00000000000..8104c201924
--- /dev/null
+++ b/src/test/modules/test_aio/worker.conf
@@ -0,0 +1,5 @@
+shared_preload_libraries=test_aio
+io_method = 'worker'
+log_min_messages = 'DEBUG3'
+log_statement=all
+restart_after_crash=false
--
2.48.1.76.g4e746b1a31.dirty
[text/x-diff] v2.3-0028-Temporary-Increase-BAS_BULKREAD-size.patch (1.3K, ../../p5flybfcjrzgpgwcnzkxsp3xdw3czfoafkvks6g7en6ewmvymh@wrz7z2vwwc7c/29-v2.3-0028-Temporary-Increase-BAS_BULKREAD-size.patch)
download | inline diff:
From 2dea8961fd6383afe1e457926131c2213db211f0 Mon Sep 17 00:00:00 2001
From: Andres Freund <andres@anarazel.de>
Date: Sun, 1 Sep 2024 00:42:27 -0400
Subject: [PATCH v2.3 28/30] Temporary: Increase BAS_BULKREAD size
Without this we only can execute very little AIO for sequential scans, as
there's just not enough buffers in the ring. This isn't the right fix, as
just increasing the ring size can have negative performance implications in
workloads where the kernel has all the data cached.
Author:
Reviewed-By:
Discussion: https://postgr.es/m/
Backpatch:
---
src/backend/storage/buffer/freelist.c | 7 ++++++-
1 file changed, 6 insertions(+), 1 deletion(-)
diff --git a/src/backend/storage/buffer/freelist.c b/src/backend/storage/buffer/freelist.c
index 1f757d96f07..ac19fb87433 100644
--- a/src/backend/storage/buffer/freelist.c
+++ b/src/backend/storage/buffer/freelist.c
@@ -555,7 +555,12 @@ GetAccessStrategy(BufferAccessStrategyType btype)
return NULL;
case BAS_BULKREAD:
- ring_size_kb = 256;
+
+ /*
+ * FIXME: Temporary increase to allow large enough streaming reads
+ * to actually benefit from AIO. This needs a better solution.
+ */
+ ring_size_kb = 2 * 1024;
break;
case BAS_BULKWRITE:
ring_size_kb = 16 * 1024;
--
2.48.1.76.g4e746b1a31.dirty
[text/x-diff] v2.3-0029-WIP-Use-MAP_POPULATE.patch (1.1K, ../../p5flybfcjrzgpgwcnzkxsp3xdw3czfoafkvks6g7en6ewmvymh@wrz7z2vwwc7c/30-v2.3-0029-WIP-Use-MAP_POPULATE.patch)
download | inline diff:
From ca1654b4d99e3565b2e14525b3409bc8c164849e Mon Sep 17 00:00:00 2001
From: Andres Freund <andres@anarazel.de>
Date: Tue, 31 Dec 2024 13:25:56 -0500
Subject: [PATCH v2.3 29/30] WIP: Use MAP_POPULATE
For benchmarking it's quite annoying that the first time a memory is touched
has completely different perf characteristics than subsequent accesses. Using
MAP_POPULATE reduces that substantially.
---
src/backend/port/sysv_shmem.c | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/src/backend/port/sysv_shmem.c b/src/backend/port/sysv_shmem.c
index 197926d44f6..a700b02d5a1 100644
--- a/src/backend/port/sysv_shmem.c
+++ b/src/backend/port/sysv_shmem.c
@@ -620,7 +620,7 @@ CreateAnonymousSegment(Size *size)
allocsize += hugepagesize - (allocsize % hugepagesize);
ptr = mmap(NULL, allocsize, PROT_READ | PROT_WRITE,
- PG_MMAP_FLAGS | mmap_flags, -1, 0);
+ PG_MMAP_FLAGS | MAP_POPULATE | mmap_flags, -1, 0);
mmap_errno = errno;
if (huge_pages == HUGE_PAGES_TRY && ptr == MAP_FAILED)
elog(DEBUG1, "mmap(%zu) with MAP_HUGETLB failed, huge pages disabled: %m",
--
2.48.1.76.g4e746b1a31.dirty
^ permalink raw reply [nested|flat] 13+ messages in thread
* Re: AIO v2.3
2025-01-23 04:29 Re: AIO v2.3 Andres Freund <andres@anarazel.de>
@ 2025-02-06 10:50 ` Jakub Wartak <jakub.wartak@enterprisedb.com>
2025-02-10 23:10 ` Re: AIO v2.3 Andres Freund <andres@anarazel.de>
1 sibling, 1 reply; 13+ messages in thread
From: Jakub Wartak @ 2025-02-06 10:50 UTC (permalink / raw)
To: Andres Freund <andres@anarazel.de>; +Cc: pgsql-hackers@postgresql.org, Heikki Linnakangas <hlinnaka@iki.fi>; Robert Haas <robertmhaas@gmail.com>; Noah Misch <noah@leadboat.com>
On Thu, Jan 23, 2025 at 5:29 AM Andres Freund <andres@anarazel.de> wrote:
> Hi,
>
> Attached is v2.3.
>
> There are a lot of changes - primarily renaming things based on on-list and
> off-list feedback. But also some other things
[..snip]
Hi Andres, OK, so I've hastily launched AIO v2.3 (full, 29 patches)
patchset probe run before going for short vacations and here results
are attached*. TLDR; in terms of SELECTs the master vs aioworkers
looks very solid! I was kind of afraid that additional IPC to separate
processes would put workers at a disadvantage a little bit , but
that's amazingly not true. The intention of this effort just to see if
committing AIO with defaults as it stands is good enough to not cause
basic regressions for users and to me it looks like it is nearly
finished :)). So here to save time I have *not* tested aio23 with
io_uring, it's just about aioworkers (the future default).
Random notes and thoughts:
1. not a single crash was observed , but those were pretty short runs
2. my very limited in terms of time data analysis thoughts
- most of the time perf with aioworkers is identical (+/- 3%) as of
the master, in most cases it is much BETTER
- up to like 2.01x boosts can be spotted even on low-end like this but
with fast I/O even without IO_URING (just workers)
- on seqscans "sata" with datasets bigger than VFS-cache ("big") and
without parallel workers, it looks like it's always better
- on parallel seqscans "sata" with datasets bigger than VFS-cache
("big") and high e_io_c with high client counts(sigh!), it looks like
it would user noticeable big regression but to me it's not regression
itself, probably we are issuing way too many posix_fadvise()
readaheads with diminishing returns. Just letting you know. Not sure
it is worth introducing some global (shared aioworkers e_io_c
limiter), I think not. I think it could also be some maintenance noise
on that I/O device, but I have no isolated SATA RAID10 with like 8x
HDDs in home to launch such a test to be absolutely sure.
3. with aioworkers in documentation it would worth pointing out that
`iotop` won't be good enough to show which PID is doing I/O anymore .
I've often get question like this: who is taking the most of I/O right
now because storage is fully saturated on multi-use system. Not sure
it would require new view or not (pg_aios output seems to be not more
like in-memory debug view that would be have to be sampled
aggressively, and pg_statio_all_tables shows well table, but not PID
-- same for pg_stat_io). IMHO if docs would be simple like
"In order to understand which processes (PIDs) are issuing lots of
IOs, please check pg_stat_activty for *IO/AioCompletion* waits events"
it should be good enough for a start.
Bench machine: it was intentionally much smaller hardware. Azure's
Lsv2 L8s_v2 (1st gen EPYC/1s4c8t, with kernel 6.10.11+bpo-cloud-amd64
and booted with mem=12GB that limited real usable RAM memory to just
like ~8GB to stress I/O). liburing 2.9. Normal standard compile
options were used without asserts (such as normal users would use).
Bench had those two I/O storage (with XFS) attached:
- "sata" stands for Azure's "Premium SSD LRS" mounted on /sata
(Size=255GB, Max IOPS=1100 (@ 4kB?), Max throughput=125MB/s)
- "nvme" stands for bulit-in NVME on that VM mounted on /nvme
(Size=1788GB, Max IOPS=8000 (@ 4kB?))
I'll try to see in the coming weeks if dedicating more time is
possible (long run tests, more write tests, maybe some basic I/O
failure injections tests).
-J.
* = 8640 test runs, always with restart and flushing VFS cache, took
probably 2-3 days? I've had to reduce tries to 1 and limit myself to
just reads just to get it running solid, before I left and not to miss
the plane :^)
Attachments:
[image/png] aio23_potential_parallel_seqscan_regression.png (98.8K, ../../CAKZiRmzP5K-SxMXU9Xr_Lsr8_-qKG4N2L6t1r1QegUMdpyhh1A@mail.gmail.com/2-aio23_potential_parallel_seqscan_regression.png)
download | view image
[application/x-compressed] results.out.bz2 (97.9K, ../../CAKZiRmzP5K-SxMXU9Xr_Lsr8_-qKG4N2L6t1r1QegUMdpyhh1A@mail.gmail.com/3-results.out.bz2)
download
[application/vnd.openxmlformats-officedocument.spreadsheetml.sheet] pg_aio23_basic_tests_v001.xlsx (1.0M, ../../CAKZiRmzP5K-SxMXU9Xr_Lsr8_-qKG4N2L6t1r1QegUMdpyhh1A@mail.gmail.com/4-pg_aio23_basic_tests_v001.xlsx)
download
[application/vnd.oasis.opendocument.spreadsheet] pg_aio23_basic_tests_v001.ods (770.4K, ../../CAKZiRmzP5K-SxMXU9Xr_Lsr8_-qKG4N2L6t1r1QegUMdpyhh1A@mail.gmail.com/5-pg_aio23_basic_tests_v001.ods)
download
^ permalink raw reply [nested|flat] 13+ messages in thread
* Re: AIO v2.3
2025-01-23 04:29 Re: AIO v2.3 Andres Freund <andres@anarazel.de>
2025-02-06 10:50 ` Re: AIO v2.3 Jakub Wartak <jakub.wartak@enterprisedb.com>
@ 2025-02-10 23:10 ` Andres Freund <andres@anarazel.de>
2025-02-14 07:40 ` Re: AIO v2.3 Jakub Wartak <jakub.wartak@enterprisedb.com>
0 siblings, 1 reply; 13+ messages in thread
From: Andres Freund @ 2025-02-10 23:10 UTC (permalink / raw)
To: Jakub Wartak <jakub.wartak@enterprisedb.com>; +Cc: pgsql-hackers@postgresql.org, Heikki Linnakangas <hlinnaka@iki.fi>; Robert Haas <robertmhaas@gmail.com>; Noah Misch <noah@leadboat.com>
Hi,
On 2025-02-06 11:50:04 +0100, Jakub Wartak wrote:
> Hi Andres, OK, so I've hastily launched AIO v2.3 (full, 29 patches)
> patchset probe run before going for short vacations and here results
> are attached*.
Thanks for doing that work!
> TLDR; in terms of SELECTs the master vs aioworkers looks very solid!
Phew! Weee! Yay.
> I was kind of afraid that additional IPC to separate processes would put
> workers at a disadvantage a little bit , but that's amazingly not true.
It's a measurable disadvantage, it's just more than counteracted by being able
to do IO asynchronously :).
It's possible to make it more visible, by setting io_combine_limit = 1. If you
have a small shared buffers with everything in the kernel cache, the dispatch
overhead starts to be noticeable above several GB/s. But that's ok, I think.
> The intention of this effort just to see if committing AIO with defaults as
> it stands is good enough to not cause basic regressions for users and to me
> it looks like it is nearly finished :)).
That's really good to hear. I think we can improve things a lot in the
future, but we gotta start somewhere...
> 1. not a single crash was observed , but those were pretty short runs
>
> 2. my very limited in terms of time data analysis thoughts
> - most of the time perf with aioworkers is identical (+/- 3%) as of
> the master, in most cases it is much BETTER
I assume s/most/some/ for the second most?
> - on parallel seqscans "sata" with datasets bigger than VFS-cache
> ("big") and high e_io_c with high client counts(sigh!), it looks like
> it would user noticeable big regression but to me it's not regression
> itself, probably we are issuing way too many posix_fadvise()
> readaheads with diminishing returns. Just letting you know. Not sure
> it is worth introducing some global (shared aioworkers e_io_c
> limiter), I think not. I think it could also be some maintenance noise
> on that I/O device, but I have no isolated SATA RAID10 with like 8x
> HDDs in home to launch such a test to be absolutely sure.
I'm inclined to not introduce a global limit for now - it's pretty hard to
make that scale to fast IO devices, so you need a multi-level design, where
each backend can issue a few IOs without consulting the global limit and only
after that you need to get the right to issue even more IOs from the shared
"pool".
I think this is basically a configuration issue - configuring a high e_io_c
for a device that can't handle that and then load it up with a lot of clients,
well, that'll not work out great.
> 3. with aioworkers in documentation it would worth pointing out that
> `iotop` won't be good enough to show which PID is doing I/O anymore .
> I've often get question like this: who is taking the most of I/O right
> now because storage is fully saturated on multi-use system. Not sure
> it would require new view or not (pg_aios output seems to be not more
> like in-memory debug view that would be have to be sampled
> aggressively, and pg_statio_all_tables shows well table, but not PID
> -- same for pg_stat_io). IMHO if docs would be simple like
> "In order to understand which processes (PIDs) are issuing lots of
> IOs, please check pg_stat_activty for *IO/AioCompletion* waits events"
> it should be good enough for a start.
pg_stat_get_backend_io() should allow to answer that, albeit with the usual
weakness of our stats system, namely that the user has to diff two snapshots
themselves. It probably also has the weakness of not showing results for
queries before they've finished, although I think that's something we should
be able to improve without too much trouble (not in this release though, I
suspect).
I guess we could easily reference pg_stat_get_backend_io(), but a more
complete recipe isn't entirely trivial...
> Bench machine: it was intentionally much smaller hardware. Azure's
> Lsv2 L8s_v2 (1st gen EPYC/1s4c8t, with kernel 6.10.11+bpo-cloud-amd64
> and booted with mem=12GB that limited real usable RAM memory to just
> like ~8GB to stress I/O). liburing 2.9. Normal standard compile
> options were used without asserts (such as normal users would use).
Good - the asserts in the aio patches are a bit more noticeable than the ones
in master.
Thanks again for running these!
Greetings,
Andres Freund
^ permalink raw reply [nested|flat] 13+ messages in thread
* Re: AIO v2.3
2025-01-23 04:29 Re: AIO v2.3 Andres Freund <andres@anarazel.de>
2025-02-06 10:50 ` Re: AIO v2.3 Jakub Wartak <jakub.wartak@enterprisedb.com>
2025-02-10 23:10 ` Re: AIO v2.3 Andres Freund <andres@anarazel.de>
@ 2025-02-14 07:40 ` Jakub Wartak <jakub.wartak@enterprisedb.com>
0 siblings, 0 replies; 13+ messages in thread
From: Jakub Wartak @ 2025-02-14 07:40 UTC (permalink / raw)
To: Andres Freund <andres@anarazel.de>; +Cc: pgsql-hackers@postgresql.org, Heikki Linnakangas <hlinnaka@iki.fi>; Robert Haas <robertmhaas@gmail.com>; Noah Misch <noah@leadboat.com>
On Tue, Feb 11, 2025 at 12:10 AM Andres Freund <andres@anarazel.de> wrote:
>> TLDR; in terms of SELECTs the master vs aioworkers looks very solid!
> Phew! Weee! Yay.
Another good news: I've completed a full 24h pgbench run on the same
machine and it did not fail or report anything suspicious. FYI,
patchset didn't not apply anymore (seems patches 1..6 are already
applied on master due to checkpoint shutdown sequence), but there was
a failed hunk in patch #12 yesterday too:
[..]
patching file src/backend/postmaster/postmaster.c
Hunk #10 succeeded at 2960 (offset 14 lines).
Hunk #11 FAILED at 3047.
[..]
1 out of 15 hunks FAILED -- saving rejects to file
src/backend/postmaster/postmaster.c.rej
anyway, so on master @ a5579a90af05814eb5dc2fd5f68ce803899d2504 (~ Jan
24) to have clean apply I've used the below asserted build:
meson setup build --reconfigure --prefix=/usr/pgsql18.aio --debug
-Dsegsize_blocks=13 -Dcassert=true
/usr/pgsql18.aio/bin/pgbench -i -s 500 --partitions=100 # ~8GB
/usr/pgsql18.aio/bin/pgbench -R 1500 -c 100 -j 4 -P 1 -T 86400
with some add-on functionalities:
effective_io_concurrency = '4'
shared_buffers = '2GB'
max_connections = '1000'
archive_command = 'cp %p /dev/null'
archive_mode = 'on'
summarize_wal = 'on'
wal_summary_keep_time = '1h'
wal_compression = 'on'
wal_log_hints = 'on'
max_wal_size = '1GB'
shared_preload_libraries = 'pg_stat_statements'
huge_pages = 'off'
wal_receiver_status_interval = '1s'
so the above got perfect run:
[..]
duration: 86400 s
number of transactions actually processed: 129615534
number of failed transactions: 0 (0.000%)
latency average = 5.332 ms
latency stddev = 24.107 ms
rate limit schedule lag: avg 0.748 (max 1992.517) ms
initial connection time = 124.472 ms
tps = 1500.179231 (without initial connection time)
> > I was kind of afraid that additional IPC to separate processes would put
> > workers at a disadvantage a little bit , but that's amazingly not true.
>
> It's a measurable disadvantage, it's just more than counteracted by being able
> to do IO asynchronously :).
>
> It's possible to make it more visible, by setting io_combine_limit = 1. If you
> have a small shared buffers with everything in the kernel cache, the dispatch
> overhead starts to be noticeable above several GB/s. But that's ok, I think.
Sure it is.
> > 2. my very limited in terms of time data analysis thoughts
> > - most of the time perf with aioworkers is identical (+/- 3%) as of
> > the master, in most cases it is much BETTER
>
> I assume s/most/some/ for the second most?
Right, pardon for my excited moment ;)
> > - on parallel seqscans "sata" with datasets bigger than VFS-cache
> > ("big") and high e_io_c with high client counts(sigh!), it looks like
> > it would user noticeable big regression but to me it's not regression
> > itself, probably we are issuing way too many posix_fadvise()
> > readaheads with diminishing returns. Just letting you know. Not sure
> > it is worth introducing some global (shared aioworkers e_io_c
> > limiter), I think not. I think it could also be some maintenance noise
> > on that I/O device, but I have no isolated SATA RAID10 with like 8x
> > HDDs in home to launch such a test to be absolutely sure.
>
> I think this is basically a configuration issue - configuring a high e_io_c
> for a device that can't handle that and then load it up with a lot of clients,
> well, that'll not work out great.
Sure, btw i'm going to also an idea about autotuning that e_io_c in
that related thread where everybody is complaining about it
> > 3. with aioworkers in documentation it would worth pointing out that
> > `iotop` won't be good enough to show which PID is doing I/O anymore .
> > I've often get question like this: who is taking the most of I/O right
> > now because storage is fully saturated on multi-use system. Not sure
> > it would require new view or not (pg_aios output seems to be not more
> > like in-memory debug view that would be have to be sampled
> > aggressively, and pg_statio_all_tables shows well table, but not PID
> > -- same for pg_stat_io). IMHO if docs would be simple like
> > "In order to understand which processes (PIDs) are issuing lots of
> > IOs, please check pg_stat_activty for *IO/AioCompletion* waits events"
> > it should be good enough for a start.
>
> pg_stat_get_backend_io() should allow to answer that, albeit with the usual
> weakness of our stats system, namely that the user has to diff two snapshots
> themselves. It probably also has the weakness of not showing results for
> queries before they've finished, although I think that's something we should
> be able to improve without too much trouble (not in this release though, I
> suspect).
>
> I guess we could easily reference pg_stat_get_backend_io(), but a more
> complete recipe isn't entirely trivial...
I was trying to come out with something that could be added to docs,
but the below thing is too ugly and as you stated the primary weakness
is that query needs to finish before it is reflected:
WITH
b AS (SELECT 0 AS step, pid, round(sum(write_bytes)/1024/1024) AS
wMB, NULL::void, NULL::void FROM pg_stat_activity,
pg_stat_get_backend_io(pid) GROUP BY pid),
flush AS (SELECT 0 as step, 0, 0, pg_sleep(1), pg_stat_clear_snapshot()),
e AS (SELECT 1 AS step, pid, round(sum(write_bytes)/1024/1024) AS
wMB, NULL::void, NULL::void FROM pg_stat_activity,
pg_stat_get_backend_io(pid) GROUP BY pid),
picture AS MATERIALIZED (
SELECT * FROM b
UNION ALL
SELECt * FROM flush
UNION ALL
SELECT * FROM e
)
SELECT * FROM (
SELECT pid, wMB - LAG(wMB, 1) OVER (PARTITION BY pid ORDER BY
step) AS "wMB/s" FROM picture
) WHERE "wMB/s" > 0;
\watch 1
-J.
^ permalink raw reply [nested|flat] 13+ messages in thread
* Re: AIO v2.3
2025-01-23 04:29 Re: AIO v2.3 Andres Freund <andres@anarazel.de>
@ 2025-02-10 22:48 ` Thomas Munro <thomas.munro@gmail.com>
2025-02-10 22:54 ` Re: AIO v2.3 Andres Freund <andres@anarazel.de>
2025-02-11 17:10 ` Re: AIO v2.3 James Hunter <james.hunter.pg@gmail.com>
2025-02-11 21:43 ` Re: AIO v2.3 Andres Freund <andres@anarazel.de>
1 sibling, 3 replies; 13+ messages in thread
From: Thomas Munro @ 2025-02-10 22:48 UTC (permalink / raw)
To: Andres Freund <andres@anarazel.de>; +Cc: pgsql-hackers@postgresql.org, Heikki Linnakangas <hlinnaka@iki.fi>; Robert Haas <robertmhaas@gmail.com>; Noah Misch <noah@leadboat.com>; Jakub Wartak <jakub.wartak@enterprisedb.com>
On Thu, Jan 23, 2025 at 5:29 PM Andres Freund <andres@anarazel.de> wrote:
+/* caller will issue more io, don't submit */
+#define READ_BUFFERS_MORE_MORE_MORE (1 << 3)
> - Heikki doesn't love pgaio_submit_staged(), suggested pgaio_kick_staged() or
> such. I don't love that name though.
Problem statement: You want to be able to batch I/O submission, ie
make a single call to ioring_enter() (and other mechanisms) to start
several I/Os, but the code that submits is inside StartReadBuffers()
and the code that knows how many I/Os it wants to start now is at a
higher level, read_stream.c and in future elsewhere. So you invented
this flag to tell StartReadBuffers() not to call
pgaio_submit_staged(), because you promise to do it later, via this
staging list. Additionally, there is a kind of programming rule here
that you *must* submit I/Os that you stage, you aren't allowed to (for
example) stage I/Os and then sleep, so it has to be a fairly tight
piece of code.
Would the API be better like this?: When you want to create a batch
of I/Os submitted together, you wrap the work in pgaio_begin_batch()
and pgaio_submit_batch(), eg the loop in read_stream_lookahead().
Then bufmgr wouldn't need this flag: when it (or anything else) calls
smgrstartreadv(), if there is not currently an explicit batch then it
would be submitted immediately, and otherwise it would only be staged.
This way, batch construction (or whatever word you prefer for batch)
is in a clearly and explicitly demarcated stretch of code in one
lexical scope (though its effect is dynamically scoped just like the
staging list itself because we don't want to pass explicit I/O
contexts through the layers), but code that doesn't call those and
reaches AsyncReadBuffer() or whatever gets an implicit batch of size
one and that's also OK. Not sure what semantics nesting would have
but I doubt it matters much.
> Things that need to be fixed / are fixed in this:
> - max pinned buffers should be limited by io_combine_limit, not * 4
> - overflow distance
> - pins need to be limited in more places
I have patches for these and a few more things and will post in a
separate thread shortly because they can be understood without
reference to this AIO stuff and that'll hopefully be more digestible.
+ /*
+ * In some rare-ish cases one operation causes multiple reads (e.g. if a
+ * buffer was concurrently read by another backend). It'd be much better
+ * if we ensured that each ReadBuffersOperation covered only one IO - but
+ * that's not entirely trivial, due to having pinned victim buffers before
+ * starting IOs.
+ *
+ * TODO: Change the API of StartReadBuffers() to ensure we only ever need
+ * one IO.
Likewise.
+ /* IO finished, but result has not yet been processed */
+ PGAIO_HS_COMPLETED_IO,
+
+ /* IO completed, shared completion has been called */
+ PGAIO_HS_COMPLETED_SHARED,
+
+ /* IO completed, local completion has been called */
+ PGAIO_HS_COMPLETED_LOCAL,
(Repeating something I mentioned in off-list bikeshedding) I wondered
if it might be clearer to use the terminology "terminated" for the
work that PostgreSQL has to do after an I/O completes, instead of
overloading/subdividing the term "completed". We already "terminate"
an I/O when smgr I/O completes in pre-existing bufmgr terminology, and
this feels like a sort of generalisation of that notion. In this AIO
world, some work is done by the backend that receives the completion
notification from the kernel, and some is done by the backend that
submitted the I/O in the first place, a division that doesn't exist
with simple synchronous system calls. I wonder if it would be clearer
to use terms based on those two roles, rather than "shared" and
"local", leading to something like:
PGAIO_HS_COMPLETED,
PGAIO_HS_TERMINATED_BY_COMPLETER,
PGAIO_HS_TERMINATED_BY_SUBMITTER,
^ permalink raw reply [nested|flat] 13+ messages in thread
* Re: AIO v2.3
2025-01-23 04:29 Re: AIO v2.3 Andres Freund <andres@anarazel.de>
2025-02-10 22:48 ` Re: AIO v2.3 Thomas Munro <thomas.munro@gmail.com>
@ 2025-02-10 22:54 ` Andres Freund <andres@anarazel.de>
2 siblings, 0 replies; 13+ messages in thread
From: Andres Freund @ 2025-02-10 22:54 UTC (permalink / raw)
To: Thomas Munro <thomas.munro@gmail.com>; +Cc: pgsql-hackers@postgresql.org, Heikki Linnakangas <hlinnaka@iki.fi>; Robert Haas <robertmhaas@gmail.com>; Noah Misch <noah@leadboat.com>; Jakub Wartak <jakub.wartak@enterprisedb.com>
Hi,
On 2025-02-11 11:48:38 +1300, Thomas Munro wrote:
> On Thu, Jan 23, 2025 at 5:29 PM Andres Freund <andres@anarazel.de> wrote:
> > - Heikki doesn't love pgaio_submit_staged(), suggested pgaio_kick_staged() or
> > such. I don't love that name though.
>
> Problem statement: You want to be able to batch I/O submission, ie
> make a single call to ioring_enter() (and other mechanisms) to start
> several I/Os, but the code that submits is inside StartReadBuffers()
> and the code that knows how many I/Os it wants to start now is at a
> higher level, read_stream.c and in future elsewhere. So you invented
> this flag to tell StartReadBuffers() not to call
> pgaio_submit_staged(), because you promise to do it later, via this
> staging list. Additionally, there is a kind of programming rule here
> that you *must* submit I/Os that you stage, you aren't allowed to (for
> example) stage I/Os and then sleep, so it has to be a fairly tight
> piece of code.
>
> Would the API be better like this?: When you want to create a batch
> of I/Os submitted together, you wrap the work in pgaio_begin_batch()
> and pgaio_submit_batch(), eg the loop in read_stream_lookahead().
> Then bufmgr wouldn't need this flag: when it (or anything else) calls
> smgrstartreadv(), if there is not currently an explicit batch then it
> would be submitted immediately, and otherwise it would only be staged.
> This way, batch construction (or whatever word you prefer for batch)
> is in a clearly and explicitly demarcated stretch of code in one
> lexical scope (though its effect is dynamically scoped just like the
> staging list itself because we don't want to pass explicit I/O
> contexts through the layers), but code that doesn't call those and
> reaches AsyncReadBuffer() or whatever gets an implicit batch of size
> one and that's also OK. Not sure what semantics nesting would have
> but I doubt it matters much.
I'm a bit unexcited about the work to redesign this, but I also admit that you
have a point :)
Linux calls a similar concept "plugging" the queue. I think I like "batch"
better, but only marginally.
> > Things that need to be fixed / are fixed in this:
> > - max pinned buffers should be limited by io_combine_limit, not * 4
> > - overflow distance
> > - pins need to be limited in more places
>
> I have patches for these and a few more things and will post in a
> separate thread shortly because they can be understood without
> reference to this AIO stuff and that'll hopefully be more digestible.
Yay!
> + /* IO finished, but result has not yet been processed */
> + PGAIO_HS_COMPLETED_IO,
> +
> + /* IO completed, shared completion has been called */
> + PGAIO_HS_COMPLETED_SHARED,
> +
> + /* IO completed, local completion has been called */
> + PGAIO_HS_COMPLETED_LOCAL,
>
> (Repeating something I mentioned in off-list bikeshedding) I wondered
> if it might be clearer to use the terminology "terminated" for the
> work that PostgreSQL has to do after an I/O completes, instead of
> overloading/subdividing the term "completed". We already "terminate"
> an I/O when smgr I/O completes in pre-existing bufmgr terminology, and
> this feels like a sort of generalisation of that notion.
I have a mild laziness preference for complete over terminate, but not
more. If others agree with Thomas, I'm ok with changing it.
> In this AIO world, some work is done by the backend that receives the
> completion notification from the kernel, and some is done by the backend
> that submitted the I/O in the first place, a division that doesn't exist
> with simple synchronous system calls. I wonder if it would be clearer to
> use terms based on those two roles, rather than "shared" and "local",
> leading to something like:
>
> PGAIO_HS_COMPLETED,
> PGAIO_HS_TERMINATED_BY_COMPLETER,
> PGAIO_HS_TERMINATED_BY_SUBMITTER,
I don't love those, because the SHARED/LOCAL does imply more clearly what you
have access to. I.e. executing things in a shared completion callback for IO
on a temporary buffer doesn't make sense, you won't have access to the local
buffer table.
Greetings,
Andres Freund
^ permalink raw reply [nested|flat] 13+ messages in thread
* Re: AIO v2.3
2025-01-23 04:29 Re: AIO v2.3 Andres Freund <andres@anarazel.de>
2025-02-10 22:48 ` Re: AIO v2.3 Thomas Munro <thomas.munro@gmail.com>
@ 2025-02-11 17:10 ` James Hunter <james.hunter.pg@gmail.com>
2025-02-11 17:34 ` Re: AIO v2.3 Robert Haas <robertmhaas@gmail.com>
2 siblings, 1 reply; 13+ messages in thread
From: James Hunter @ 2025-02-11 17:10 UTC (permalink / raw)
To: Thomas Munro <thomas.munro@gmail.com>; +Cc: Andres Freund <andres@anarazel.de>; pgsql-hackers@postgresql.org, Heikki Linnakangas <hlinnaka@iki.fi>; Robert Haas <robertmhaas@gmail.com>; Noah Misch <noah@leadboat.com>; Jakub Wartak <jakub.wartak@enterprisedb.com>
On Mon, Feb 10, 2025 at 2:40 PM Thomas Munro <thomas.munro@gmail.com> wrote:
>
...
> Problem statement: You want to be able to batch I/O submission, ie
> make a single call to ioring_enter() (and other mechanisms) to start
> several I/Os, but the code that submits is inside StartReadBuffers()
> and the code that knows how many I/Os it wants to start now is at a
> higher level, read_stream.c and in future elsewhere. So you invented
> this flag to tell StartReadBuffers() not to call
> pgaio_submit_staged(), because you promise to do it later, via this
> staging list. Additionally, there is a kind of programming rule here
> that you *must* submit I/Os that you stage, you aren't allowed to (for
> example) stage I/Os and then sleep, so it has to be a fairly tight
> piece of code.
>
> Would the API be better like this?: When you want to create a batch
> of I/Os submitted together, you wrap the work in pgaio_begin_batch()
> and pgaio_submit_batch(), eg the loop in read_stream_lookahead().
> Then bufmgr wouldn't need this flag: when it (or anything else) calls
> smgrstartreadv(), if there is not currently an explicit batch then it
> would be submitted immediately, and otherwise it would only be staged.
> This way, batch construction (or whatever word you prefer for batch)
> is in a clearly and explicitly demarcated stretch of code in one
> lexical scope (though its effect is dynamically scoped just like the
> staging list itself because we don't want to pass explicit I/O
> contexts through the layers), but code that doesn't call those and
> reaches AsyncReadBuffer() or whatever gets an implicit batch of size
> one and that's also OK. Not sure what semantics nesting would have
> but I doubt it matters much.
I like this idea. If we want to submit a batch, then just submit a batch.
James
^ permalink raw reply [nested|flat] 13+ messages in thread
* Re: AIO v2.3
2025-01-23 04:29 Re: AIO v2.3 Andres Freund <andres@anarazel.de>
2025-02-10 22:48 ` Re: AIO v2.3 Thomas Munro <thomas.munro@gmail.com>
2025-02-11 17:10 ` Re: AIO v2.3 James Hunter <james.hunter.pg@gmail.com>
@ 2025-02-11 17:34 ` Robert Haas <robertmhaas@gmail.com>
0 siblings, 0 replies; 13+ messages in thread
From: Robert Haas @ 2025-02-11 17:34 UTC (permalink / raw)
To: James Hunter <james.hunter.pg@gmail.com>; +Cc: Thomas Munro <thomas.munro@gmail.com>; Andres Freund <andres@anarazel.de>; pgsql-hackers@postgresql.org, Heikki Linnakangas <hlinnaka@iki.fi>; Noah Misch <noah@leadboat.com>; Jakub Wartak <jakub.wartak@enterprisedb.com>
On Tue, Feb 11, 2025 at 12:11 PM James Hunter <james.hunter.pg@gmail.com> wrote:
> I like this idea. If we want to submit a batch, then just submit a batch.
Sounds good to me, too.
--
Robert Haas
EDB: http://www.enterprisedb.com
^ permalink raw reply [nested|flat] 13+ messages in thread
* Re: AIO v2.3
2025-01-23 04:29 Re: AIO v2.3 Andres Freund <andres@anarazel.de>
2025-02-10 22:48 ` Re: AIO v2.3 Thomas Munro <thomas.munro@gmail.com>
@ 2025-02-11 21:43 ` Andres Freund <andres@anarazel.de>
2025-02-12 18:00 ` Re: AIO v2.3 Robert Haas <robertmhaas@gmail.com>
2025-02-12 20:26 ` Re: AIO v2.3 James Hunter <james.hunter.pg@gmail.com>
2 siblings, 2 replies; 13+ messages in thread
From: Andres Freund @ 2025-02-11 21:43 UTC (permalink / raw)
To: Thomas Munro <thomas.munro@gmail.com>; +Cc: pgsql-hackers@postgresql.org, Heikki Linnakangas <hlinnaka@iki.fi>; Robert Haas <robertmhaas@gmail.com>; Noah Misch <noah@leadboat.com>; Jakub Wartak <jakub.wartak@enterprisedb.com>
Hi,
On 2025-02-11 11:48:38 +1300, Thomas Munro wrote:
> Would the API be better like this?: When you want to create a batch
> of I/Os submitted together, you wrap the work in pgaio_begin_batch()
> and pgaio_submit_batch(), eg the loop in read_stream_lookahead().
One annoying detail is that an API like this would afaict need resowner
support or something along those lines (e.g. xact callbacks plus code in each
aux process' sigsetjmp() block). Otherwise I don't know how we would ensure
that the "batch-is-in-progress" flag/counter would get reset.
Alternatively we could make pgaio_batch_begin() basically start a critical
section, but that doesn't seem like a good idea, because too much that needs
to happen around buffered IO isn't compatible with critical sections.
Does anybody see a need for batches to be nested? I'm inclined to think that
that would be indicative of bugs and should therefore error/assert out.
One way we could avoid the need for a mechanism to reset-batch-in-progress
would be to make batch submission controlled by a flag on the IO. Something
like
pgaio_io_set_flag(ioh, PGAIO_HF_BATCH_SUBMIT)
IFF PGAIO_HF_BATCH_SUBMIT is set, the IOs would need to be explicitly
submitted using something like the existing
pgaio_submit_staged();
(although renaming it to something with batch in the name might be
appropriate)
That way there's no explicit "we are in a batch" state that needs to be reset
in case of errors.
Greetings,
Andres Freund
^ permalink raw reply [nested|flat] 13+ messages in thread
* Re: AIO v2.3
2025-01-23 04:29 Re: AIO v2.3 Andres Freund <andres@anarazel.de>
2025-02-10 22:48 ` Re: AIO v2.3 Thomas Munro <thomas.munro@gmail.com>
2025-02-11 21:43 ` Re: AIO v2.3 Andres Freund <andres@anarazel.de>
@ 2025-02-12 18:00 ` Robert Haas <robertmhaas@gmail.com>
2025-02-17 21:01 ` Re: AIO v2.3 Andres Freund <andres@anarazel.de>
1 sibling, 1 reply; 13+ messages in thread
From: Robert Haas @ 2025-02-12 18:00 UTC (permalink / raw)
To: Andres Freund <andres@anarazel.de>; +Cc: Thomas Munro <thomas.munro@gmail.com>; pgsql-hackers@postgresql.org, Heikki Linnakangas <hlinnaka@iki.fi>; Noah Misch <noah@leadboat.com>; Jakub Wartak <jakub.wartak@enterprisedb.com>
On Tue, Feb 11, 2025 at 4:43 PM Andres Freund <andres@anarazel.de> wrote:
> Alternatively we could make pgaio_batch_begin() basically start a critical
> section, but that doesn't seem like a good idea, because too much that needs
> to happen around buffered IO isn't compatible with critical sections.
A critical section sounds like a bad plan.
> Does anybody see a need for batches to be nested? I'm inclined to think that
> that would be indicative of bugs and should therefore error/assert out.
I can imagine somebody wanting to do it, but I think we can just say
no. I mean, it's no different from WAL record construction. There's no
theoretical reason you couldn't want to concurrently construct
multiple WAL records and then submit them one after another, but if
you want to do that, you have to do your own bookkeeping. It seems
fine to apply the same principle here.
> One way we could avoid the need for a mechanism to reset-batch-in-progress
> would be to make batch submission controlled by a flag on the IO. Something
> like
> pgaio_io_set_flag(ioh, PGAIO_HF_BATCH_SUBMIT)
>
> IFF PGAIO_HF_BATCH_SUBMIT is set, the IOs would need to be explicitly
> submitted using something like the existing
> pgaio_submit_staged();
> (although renaming it to something with batch in the name might be
> appropriate)
>
> That way there's no explicit "we are in a batch" state that needs to be reset
> in case of errors.
I'll defer to Thomas or others on whether this is better or worse,
because I don't know. It means that the individual I/Os have to know
that they are in a batch, which isn't necessary with the begin/end
batch interface. But if we're expecting that to happen in a pretty
confined amount of code -- similar to WAL record construction -- then
that might not be a problem anyway.
I think if you don't do this, I'd do (sub)xact callbacks rather than a
resowner integration, unless you decide you want to support multiple
concurrent batches. You don't really need or want to tie it to a
resowner unless there are multiple objects each of which can have its
own resources.
--
Robert Haas
EDB: http://www.enterprisedb.com
^ permalink raw reply [nested|flat] 13+ messages in thread
* Re: AIO v2.3
2025-01-23 04:29 Re: AIO v2.3 Andres Freund <andres@anarazel.de>
2025-02-10 22:48 ` Re: AIO v2.3 Thomas Munro <thomas.munro@gmail.com>
2025-02-11 21:43 ` Re: AIO v2.3 Andres Freund <andres@anarazel.de>
2025-02-12 18:00 ` Re: AIO v2.3 Robert Haas <robertmhaas@gmail.com>
@ 2025-02-17 21:01 ` Andres Freund <andres@anarazel.de>
2025-02-18 19:19 ` Re: AIO v2.3 Robert Haas <robertmhaas@gmail.com>
0 siblings, 1 reply; 13+ messages in thread
From: Andres Freund @ 2025-02-17 21:01 UTC (permalink / raw)
To: Robert Haas <robertmhaas@gmail.com>; +Cc: Thomas Munro <thomas.munro@gmail.com>; pgsql-hackers@postgresql.org, Heikki Linnakangas <hlinnaka@iki.fi>; Noah Misch <noah@leadboat.com>; Jakub Wartak <jakub.wartak@enterprisedb.com>
Hi,
On 2025-02-12 13:00:22 -0500, Robert Haas wrote:
> On Tue, Feb 11, 2025 at 4:43 PM Andres Freund <andres@anarazel.de> wrote:
> > One way we could avoid the need for a mechanism to reset-batch-in-progress
> > would be to make batch submission controlled by a flag on the IO. Something
> > like
> > pgaio_io_set_flag(ioh, PGAIO_HF_BATCH_SUBMIT)
> >
> > IFF PGAIO_HF_BATCH_SUBMIT is set, the IOs would need to be explicitly
> > submitted using something like the existing
> > pgaio_submit_staged();
> > (although renaming it to something with batch in the name might be
> > appropriate)
> >
> > That way there's no explicit "we are in a batch" state that needs to be reset
> > in case of errors.
>
> I'll defer to Thomas or others on whether this is better or worse,
> because I don't know. It means that the individual I/Os have to know
> that they are in a batch, which isn't necessary with the begin/end
> batch interface. But if we're expecting that to happen in a pretty
> confined amount of code -- similar to WAL record construction -- then
> that might not be a problem anyway.
>
> I think if you don't do this, I'd do (sub)xact callbacks rather than a
> resowner integration, unless you decide you want to support multiple
> concurrent batches. You don't really need or want to tie it to a
> resowner unless there are multiple objects each of which can have its
> own resources.
I have working code for that. There unfortunately is an annoying problem:
It afaict is not really possible to trigger a WARNING/ERRROR/Assert in xact
callbacks at the end of commands in an explicit transaction. Consider
something like:
BEGIN;
SELECT start_but_not_end_aio_batch();
we would like to flag that the SELECT query started an AIO batch but didn't
end it. But at the momment there, afaict, is no proper way to do that, because
xact.c won't actually do much at the end of a command in a transaction block:
/*
* This is the case when we have finished executing a command
* someplace within a transaction block. We increment the command
* counter and return.
*/
case TBLOCK_INPROGRESS:
case TBLOCK_IMPLICIT_INPROGRESS:
case TBLOCK_SUBINPROGRESS:
CommandCounterIncrement();
break;
If one instead integrates with resowners, that kind of thing works, because
exec_simple_query() calls PortalDrop(), which in turn calls
ResourceOwnerRelease().
And we don't reliably warn at a later time. While xact.c integration triggers
a warning for:
BEGIN;
SELECT start_but_not_end_aio_batch();
COMMIT;
as we'd still have the batch open at COMMIT, it wouldn't trigger for
BEGIN;
SELECT start_but_not_end_aio_batch();
ROLLBACK;
as AbortTransaction() might be called in an error and therefore can't assume
that it's a problem for a batch to still be open.
I guess I could just put something alongside that CommandCounterIncrement()
call, but that doesn't seem right. I guess putting it alongside the
ResourceOwnerRelease() in PortalDrop() is a bit less bad? But still doesn't
seem great.
Just using resowners doesn't seem right either, it's not really free to
register something with resowners, and for read intensive IO we can start a
*lot* of batches, so doing unnecessary work isn't great.
Greetings,
Andres Freund
^ permalink raw reply [nested|flat] 13+ messages in thread
* Re: AIO v2.3
2025-01-23 04:29 Re: AIO v2.3 Andres Freund <andres@anarazel.de>
2025-02-10 22:48 ` Re: AIO v2.3 Thomas Munro <thomas.munro@gmail.com>
2025-02-11 21:43 ` Re: AIO v2.3 Andres Freund <andres@anarazel.de>
2025-02-12 18:00 ` Re: AIO v2.3 Robert Haas <robertmhaas@gmail.com>
2025-02-17 21:01 ` Re: AIO v2.3 Andres Freund <andres@anarazel.de>
@ 2025-02-18 19:19 ` Robert Haas <robertmhaas@gmail.com>
0 siblings, 0 replies; 13+ messages in thread
From: Robert Haas @ 2025-02-18 19:19 UTC (permalink / raw)
To: Andres Freund <andres@anarazel.de>; +Cc: Thomas Munro <thomas.munro@gmail.com>; pgsql-hackers@postgresql.org, Heikki Linnakangas <hlinnaka@iki.fi>; Noah Misch <noah@leadboat.com>; Jakub Wartak <jakub.wartak@enterprisedb.com>
On Mon, Feb 17, 2025 at 4:01 PM Andres Freund <andres@anarazel.de> wrote:
> If one instead integrates with resowners, that kind of thing works, because
> exec_simple_query() calls PortalDrop(), which in turn calls
> ResourceOwnerRelease().
Hmm, so maybe that's a reason to do it via resowner.c, then. The fact
that it's a singleton object is a bit annoying, but you could make it
not a singleton, and then either pass the relevant one to the
interface functions, or store the current one in a global variable
similar to CurrentMemoryContext or similar.
> I guess I could just put something alongside that CommandCounterIncrement()
> call, but that doesn't seem right. I guess putting it alongside the
> ResourceOwnerRelease() in PortalDrop() is a bit less bad? But still doesn't
> seem great.
The thing that's weird about that is that it isn't really logically
linked to the portal. It feels like it more properly belongs in
StartTransactionCommand() / CommitTransactionCommand().
> Just using resowners doesn't seem right either, it's not really free to
> register something with resowners, and for read intensive IO we can start a
> *lot* of batches, so doing unnecessary work isn't great.
You don't necessarily have to register a new object for every batch,
do you? You could just register one and keep reusing it for the
lifetime of the query.
--
Robert Haas
EDB: http://www.enterprisedb.com
^ permalink raw reply [nested|flat] 13+ messages in thread
* Re: AIO v2.3
2025-01-23 04:29 Re: AIO v2.3 Andres Freund <andres@anarazel.de>
2025-02-10 22:48 ` Re: AIO v2.3 Thomas Munro <thomas.munro@gmail.com>
2025-02-11 21:43 ` Re: AIO v2.3 Andres Freund <andres@anarazel.de>
@ 2025-02-12 20:26 ` James Hunter <james.hunter.pg@gmail.com>
1 sibling, 0 replies; 13+ messages in thread
From: James Hunter @ 2025-02-12 20:26 UTC (permalink / raw)
To: Andres Freund <andres@anarazel.de>; +Cc: Thomas Munro <thomas.munro@gmail.com>; pgsql-hackers@postgresql.org, Heikki Linnakangas <hlinnaka@iki.fi>; Robert Haas <robertmhaas@gmail.com>; Noah Misch <noah@leadboat.com>; Jakub Wartak <jakub.wartak@enterprisedb.com>
On Tue, Feb 11, 2025 at 1:44 PM Andres Freund <andres@anarazel.de> wrote:
>
...
> Alternatively we could make pgaio_batch_begin() basically start a critical
> section, but that doesn't seem like a good idea, because too much that needs
> to happen around buffered IO isn't compatible with critical sections.
>
>
> Does anybody see a need for batches to be nested? I'm inclined to think that
> that would be indicative of bugs and should therefore error/assert out.
Fwiw, in a similar situation in the past, I just blocked, waiting for
the in-flight batch to complete, before sending the next batch. So I
had something like:
void begin_batch(...)
{
if (batch_in_progress())
complete_batch(...);
/* ok start the batch now */
}
In my case, batches were nested because different access methods
(e.g., Index) can call/trigger other access methods (Heap), and both
access methods might want to issue batch reads. However, the "inner"
access method might not be aware of the "outer" access method.
For simplicity, then, I just completed the outer batch. Note that this
is not optimal for performance (because a nested batch ends up
stalling the outer batch), but it does keep the code simple...
James
^ permalink raw reply [nested|flat] 13+ messages in thread
end of thread, other threads:[~2025-02-18 19:19 UTC | newest]
Thread overview: 13+ messages (download: mbox mbox.gz follow: Atom feed)
-- links below jump to the message on this page --
2025-01-23 04:29 Re: AIO v2.3 Andres Freund <andres@anarazel.de>
2025-02-06 10:50 ` Jakub Wartak <jakub.wartak@enterprisedb.com>
2025-02-10 23:10 ` Andres Freund <andres@anarazel.de>
2025-02-14 07:40 ` Jakub Wartak <jakub.wartak@enterprisedb.com>
2025-02-10 22:48 ` Thomas Munro <thomas.munro@gmail.com>
2025-02-10 22:54 ` Andres Freund <andres@anarazel.de>
2025-02-11 17:10 ` James Hunter <james.hunter.pg@gmail.com>
2025-02-11 17:34 ` Robert Haas <robertmhaas@gmail.com>
2025-02-11 21:43 ` Andres Freund <andres@anarazel.de>
2025-02-12 18:00 ` Robert Haas <robertmhaas@gmail.com>
2025-02-17 21:01 ` Andres Freund <andres@anarazel.de>
2025-02-18 19:19 ` Robert Haas <robertmhaas@gmail.com>
2025-02-12 20:26 ` James Hunter <james.hunter.pg@gmail.com>
This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox