pg.ddx.io pgsql-hackers@postgresql.org mailing list archive
help / color / mirror / Atom feedAIO v2.0
29+ messages / 11 participants
[nested] [flat]
Commits from this thread:
2025-04-08 15f0cb26b5 Increase BAS_BULKREAD based on effective_io_concurrency
2025-04-01 fdd146a8ef aio: Add README.md explaining higher level design
2025-04-01 93bc3d75d8 aio: Add test_aio module
2025-04-01 60f566b4f2 aio: Add pg_aios view
2025-03-30 ae3df4b341 read_stream: Introduce and use optional batchmode support
2025-03-30 12ce89fd07 bufmgr: Use AIO in StartReadBuffers()
2025-03-30 047cba7fa0 bufmgr: Implement AIO read support
2025-03-29 d6d8054dc7 localbuf: Track pincount in BufferDesc as well
2025-03-29 50cb7505b3 aio: Implement support for reads in smgr/md/fd
2025-03-26 c325a7633f aio: Add io_method=io_uring
2025-03-26 8eadd5c73c aio: Add liburing dependency
2025-03-18 247ce06b88 aio: Add io_method=worker
2025-03-18 55b454d0e1 aio: Infrastructure for io_method=worker
2025-03-17 da7226993f aio: Add core asynchronous I/O infrastructure
2025-03-17 02844012b3 aio: Basic subsystem initialization
2025-02-22 f8d7f29b3e Allow lwlocks to be disowned
2024-10-08 57f3702471 Use aux process resource owner in walsender
2024-10-08 755a4c10d1 bufmgr/smgr: Don't cross segment boundaries in StartReadBuffers()
2024-10-08 488f826c72 bufmgr: Return early in ScheduleBufferTagForWriteback() if fsync=off
* AIO v2.0
@ 2024-09-01 06:27 Andres Freund <andres@anarazel.de>
0 siblings, 3 replies; 29+ messages in thread
From: Andres Freund @ 2024-09-01 06:27 UTC (permalink / raw)
To: pgsql-hackers
Hi,
It's been quite a while since the last version of the AIO patchset that I have
posted. Of course parts of the larger project have since gone upstream [1].
A lot of time since the last versions was spent understanding the performance
characteristics of using AIO with WAL and understanding some other odd
performance characteristics I didn't understand. I think I mostly understand
that now and what the design implications for an AIO subsystem are.
The prototype I had been working on unfortunately suffered from a few design
issues that weren't trivial to fix.
The biggest was that each backend could essentially have hard references to
unbounded numbers of "AIO handles" and that these references prevented these
handles from being reused. Because "AIO handles" have to live in shared memory
(so other backends can wait on them, that IO workers can perform them, etc)
that's obviously an issue. There was always a way to just run out of AIO
handles. I went through quite a few iterations of a design for how to resolve
that - I think I finally got there.
Another significant issue was that when I wrote the AIO prototype,
bufmgr.c/smgr.c/md.c only issued IOs in BLCKSZ increments, with the AIO
subsystem merging them into larger IOs. Thomas et al's work around streaming
read make bufmgr.c issue larger IOs - which is good for performance. But it
was surprisingly hard to fit into my older design.
It took me much longer than I had hoped to address these issues in
prototype. In the end I made progress by working on a rewriting the patchset
from scratch (well, with a bit of copy & paste).
The main reason I had previously implemented WAL AIO etc was to know the
design implications - but now that they're somewhat understood, I'm planning
to keep the patchset much smaller, with the goal of making it upstreamable.
While making v2 somewhat presentable I unfortunately found a few more design
issues - they're now mostly resolved, I think. But I only resolved the last
one a few hours ago, who knows what a few nights of sleeping on it will
bring. Unfortunately that prevented me from doing some of the polishing that I
had wanted to finish...
Because of the aforementioned move, I currently do not have access to my
workstation. I just have access to my laptop - which has enough thermal issues
to make benchmarks not particularly reliable.
So here are just a few teaser numbers, on an PCIe v4 NVMe SSD, note however
that this is with the BAS_BULKREAD size increased, with the default 256kB, we
can only keep one IO in flight at a time (due to io_combine_limit building
larger IOs) - we'll need to do something better than this, but that's yet
another separate discussion.
Workload: pg_prewarm('pgbench_accounts') of a scale 5k database, which is
bigger than memory:
time
master: 59.097
aio v2.0, worker: 11.211
aio v2.0, uring *: 19.991
aio v2.0, direct, worker: 09.617
aio v2.0, direct, uring *: 09.802
Workload: SELECT sum(abalance) FROM pgbench_accounts;
0 workers 1 worker 2 workers 4 workers
master: 65.753 33.246 21.095 12.918
aio v2.0, worker: 21.519 12.636 10.450 10.004
aio v2.0, uring*: 31.446 17.745 12.889 10.395
aio v2.0, uring** 23.497 13.824 10.881 10.589
aio v2.0, direct, worker: 22.377 11.989 09.915 09.772
aio v2.0, direct, uring*: 24.502 12.603 10.058 09.759
* the reason io_uring is slower is that workers effectively parallelize
*memcpy, at the cost of increased CPU usage
** a simple heuristic to use IOSQE_ASYNC to force some parallelism of memcpys
Workload: checkpointing ~20GB of dirty data, mostly sequential:
time
master: 10.209
aio v2.0, worker: 05.391
aio v2.0, uring: 04.593
aio v2.0, direct, worker: 07.745
aio v2.0, direct, uring: 03.351
To solve the issue with an unbounded number of AIO references there are few
changes compared to the prior approach:
1) Only one AIO handle can be "handed out" to a backend, without being
defined. Previously the process of getting an AIO handle wasn't super
lightweight, which made it appealing to cache AIO handles - which was one
part of the problem for running out of AIO handles.
2) Nothing in a backend can force a "defined" AIO handle (i.e. one that is a
valid operation) to stay around, it's always possible to execute the AIO
operation and then reuse the handle. This provides a forward guarantee, by
ensuring that completing AIOs can free up handles (previously they couldn't
be reused until the backend local reference was released).
3) Callbacks on AIOs are not allowed to error out anymore, unless it's ok to
take the server down.
4) Obviously some code needs to know the result of AIO operation and be able
to error out. To allow for that the issuer of an AIO can provide a pointer
to local memory that'll receive the result of an AIO, including details
about what kind of errors occurred (possible errors are e.g. a read failing
or a buffer's checksum validation failing).
In the next few days I'll add a bunch more documentation and comments as well
as some better perf numbers (assuming my workstation survived...).
Besides that, I am planning to introduce "io_method=sync", which will just
execute IO synchrously. Besides that being a good capability to have, it'll
also make it more sensible to split off worker mode support into its own
commit(s).
Greetings,
Andres Freund
[1] bulk relation extension, streaming read
[2] personal health challenges, family health challenges and now moving from
the US West Coast to the East Coast, ...
Attachments: download all (17 files, .tar.gz)
v2.0-0001-bufmgr-Return-early-in-ScheduleBufferTagForWrit.patch (text/x-diff, 1.0K) download
From e05cf468cab4003baa510053ff921063ca32c19a Mon Sep 17 00:00:00 2001
From: Andres Freund <andres@anarazel.de>
Date: Thu, 27 Jul 2023 18:59:25 -0700
Subject: [PATCH v2.0 01/17] bufmgr: Return early in
ScheduleBufferTagForWriteback() if fsync=off
Author:
Reviewed-by:
Discussion: https://postgr.es/m/
Backpatch:
---
src/backend/storage/buffer/bufmgr.c | 7 ++++++-
1 file changed, 6 insertions(+), 1 deletion(-)
diff --git a/src/backend/storage/buffer/bufmgr.c b/src/backend/storage/buffer/bufmgr.c
index 5cdd2f10fc8..ec957635f2a 100644
--- a/src/backend/storage/buffer/bufmgr.c
+++ b/src/backend/storage/buffer/bufmgr.c
@@ -5926,7 +5926,12 @@ ScheduleBufferTagForWriteback(WritebackContext *wb_context, IOContext io_context
{
PendingWriteback *pending;
- if (io_direct_flags & IO_DIRECT_DATA)
+ /*
+ * As pg_flush_data() doesn't do anything with fsync disabled, there's no
+ * point in tracking in that case.
+ */
+ if (io_direct_flags & IO_DIRECT_DATA ||
+ !enableFsync)
return;
/*
--
2.45.2.827.g557ae147e6
v2.0-0002-Allow-lwlocks-to-be-unowned.patch (text/x-diff, 4.5K) download
From a1f0fd69a34d146294bd4398bd5a5712cdc002ce Mon Sep 17 00:00:00 2001
From: Andres Freund <andres@anarazel.de>
Date: Tue, 5 Jan 2021 10:10:36 -0800
Subject: [PATCH v2.0 02/17] Allow lwlocks to be unowned
This is required for AIO so that the lock hold during a write can be released
in another backend. Which in turn is required to avoid the potential for
deadlocks.
---
src/include/storage/lwlock.h | 2 +
src/backend/storage/lmgr/lwlock.c | 96 +++++++++++++++++++++----------
2 files changed, 68 insertions(+), 30 deletions(-)
diff --git a/src/include/storage/lwlock.h b/src/include/storage/lwlock.h
index d70e6d37e09..00e8022fbad 100644
--- a/src/include/storage/lwlock.h
+++ b/src/include/storage/lwlock.h
@@ -129,6 +129,8 @@ extern bool LWLockAcquireOrWait(LWLock *lock, LWLockMode mode);
extern void LWLockRelease(LWLock *lock);
extern void LWLockReleaseClearVar(LWLock *lock, pg_atomic_uint64 *valptr, uint64 val);
extern void LWLockReleaseAll(void);
+extern LWLockMode LWLockReleaseOwnership(LWLock *l);
+extern void LWLockReleaseUnowned(LWLock *l, LWLockMode mode);
extern bool LWLockHeldByMe(LWLock *lock);
extern bool LWLockAnyHeldByMe(LWLock *lock, int nlocks, size_t stride);
extern bool LWLockHeldByMeInMode(LWLock *lock, LWLockMode mode);
diff --git a/src/backend/storage/lmgr/lwlock.c b/src/backend/storage/lmgr/lwlock.c
index e765754d805..f3d3435b1f5 100644
--- a/src/backend/storage/lmgr/lwlock.c
+++ b/src/backend/storage/lmgr/lwlock.c
@@ -1773,52 +1773,36 @@ LWLockUpdateVar(LWLock *lock, pg_atomic_uint64 *valptr, uint64 val)
}
}
-
-/*
- * LWLockRelease - release a previously acquired lock
- */
-void
-LWLockRelease(LWLock *lock)
+static void
+LWLockReleaseInternal(LWLock *lock, LWLockMode mode)
{
- LWLockMode mode;
uint32 oldstate;
bool check_waiters;
- int i;
-
- /*
- * Remove lock from list of locks held. Usually, but not always, it will
- * be the latest-acquired lock; so search array backwards.
- */
- for (i = num_held_lwlocks; --i >= 0;)
- if (lock == held_lwlocks[i].lock)
- break;
-
- if (i < 0)
- elog(ERROR, "lock %s is not held", T_NAME(lock));
-
- mode = held_lwlocks[i].mode;
-
- num_held_lwlocks--;
- for (; i < num_held_lwlocks; i++)
- held_lwlocks[i] = held_lwlocks[i + 1];
-
- PRINT_LWDEBUG("LWLockRelease", lock, mode);
/*
* Release my hold on lock, after that it can immediately be acquired by
* others, even if we still have to wakeup other waiters.
*/
if (mode == LW_EXCLUSIVE)
- oldstate = pg_atomic_sub_fetch_u32(&lock->state, LW_VAL_EXCLUSIVE);
+ oldstate = pg_atomic_fetch_sub_u32(&lock->state, LW_VAL_EXCLUSIVE);
else
- oldstate = pg_atomic_sub_fetch_u32(&lock->state, LW_VAL_SHARED);
+ oldstate = pg_atomic_fetch_sub_u32(&lock->state, LW_VAL_SHARED);
/* nobody else can have that kind of lock */
- Assert(!(oldstate & LW_VAL_EXCLUSIVE));
+ if (mode == LW_EXCLUSIVE)
+ Assert((oldstate & LW_LOCK_MASK) == LW_VAL_EXCLUSIVE);
+ else
+ Assert((oldstate & LW_LOCK_MASK) < LW_VAL_EXCLUSIVE &&
+ (oldstate & LW_LOCK_MASK) >= LW_VAL_SHARED);
if (TRACE_POSTGRESQL_LWLOCK_RELEASE_ENABLED())
TRACE_POSTGRESQL_LWLOCK_RELEASE(T_NAME(lock));
+ if (mode == LW_EXCLUSIVE)
+ oldstate -= LW_VAL_EXCLUSIVE;
+ else
+ oldstate -= LW_VAL_SHARED;
+
/*
* We're still waiting for backends to get scheduled, don't wake them up
* again.
@@ -1841,6 +1825,58 @@ LWLockRelease(LWLock *lock)
LWLockWakeup(lock);
}
+ TRACE_POSTGRESQL_LWLOCK_RELEASE(T_NAME(lock));
+}
+
+void
+LWLockReleaseUnowned(LWLock *lock, LWLockMode mode)
+{
+ LWLockReleaseInternal(lock, mode);
+}
+
+/*
+ * XXX: this doesn't do a RESUME_INTERRUPTS(), responsibility of the caller.
+ */
+LWLockMode
+LWLockReleaseOwnership(LWLock *lock)
+{
+ LWLockMode mode;
+ int i;
+
+ /*
+ * Remove lock from list of locks held. Usually, but not always, it will
+ * be the latest-acquired lock; so search array backwards.
+ */
+ for (i = num_held_lwlocks; --i >= 0;)
+ if (lock == held_lwlocks[i].lock)
+ break;
+
+ if (i < 0)
+ elog(ERROR, "lock %s is not held", T_NAME(lock));
+
+ mode = held_lwlocks[i].mode;
+
+ num_held_lwlocks--;
+ for (; i < num_held_lwlocks; i++)
+ held_lwlocks[i] = held_lwlocks[i + 1];
+
+ return mode;
+}
+
+/*
+ * LWLockRelease - release a previously acquired lock
+ */
+void
+LWLockRelease(LWLock *lock)
+{
+ LWLockMode mode;
+
+ mode = LWLockReleaseOwnership(lock);
+
+ PRINT_LWDEBUG("LWLockRelease", lock, mode);
+
+ LWLockReleaseInternal(lock, mode);
+
/*
* Now okay to allow cancel/die interrupts.
*/
--
2.45.2.827.g557ae147e6
v2.0-0003-Use-aux-process-resource-owner-in-walsender.patch (text/x-diff, 4.7K) download
From 97e621ddc5fb3b7f60b8dd5517c45fac16e1f6f7 Mon Sep 17 00:00:00 2001
From: Andres Freund <andres@anarazel.de>
Date: Tue, 31 Aug 2021 12:16:28 -0700
Subject: [PATCH v2.0 03/17] Use aux process resource owner in walsender
AIO will need a resource owner to do IO. Right now we create a resowner
on-demand during basebackup, and we could do the same for AIO. But it seems
easier to just always create an aux process resowner.
---
src/include/replication/walsender.h | 1 -
src/backend/backup/basebackup.c | 8 ++++--
src/backend/replication/walsender.c | 44 ++++++-----------------------
3 files changed, 13 insertions(+), 40 deletions(-)
diff --git a/src/include/replication/walsender.h b/src/include/replication/walsender.h
index f2d8297f016..aff0f7a51ca 100644
--- a/src/include/replication/walsender.h
+++ b/src/include/replication/walsender.h
@@ -38,7 +38,6 @@ extern PGDLLIMPORT bool log_replication_commands;
extern void InitWalSender(void);
extern bool exec_replication_command(const char *cmd_string);
extern void WalSndErrorCleanup(void);
-extern void WalSndResourceCleanup(bool isCommit);
extern void PhysicalWakeupLogicalWalSnd(void);
extern XLogRecPtr GetStandbyFlushRecPtr(TimeLineID *tli);
extern void WalSndSignals(void);
diff --git a/src/backend/backup/basebackup.c b/src/backend/backup/basebackup.c
index de16afac749..23bf8bf2db0 100644
--- a/src/backend/backup/basebackup.c
+++ b/src/backend/backup/basebackup.c
@@ -250,8 +250,10 @@ perform_base_backup(basebackup_options *opt, bbsink *sink,
state.bytes_total_is_valid = false;
/* we're going to use a BufFile, so we need a ResourceOwner */
- Assert(CurrentResourceOwner == NULL);
- CurrentResourceOwner = ResourceOwnerCreate(NULL, "base backup");
+ Assert(AuxProcessResourceOwner != NULL);
+ Assert(CurrentResourceOwner == AuxProcessResourceOwner ||
+ CurrentResourceOwner == NULL);
+ CurrentResourceOwner = AuxProcessResourceOwner;
backup_started_in_recovery = RecoveryInProgress();
@@ -672,7 +674,7 @@ perform_base_backup(basebackup_options *opt, bbsink *sink,
FreeBackupManifest(&manifest);
/* clean up the resource owner we created */
- WalSndResourceCleanup(true);
+ ReleaseAuxProcessResources(true);
basebackup_progress_done();
}
diff --git a/src/backend/replication/walsender.c b/src/backend/replication/walsender.c
index c5f1009f370..0e847535a64 100644
--- a/src/backend/replication/walsender.c
+++ b/src/backend/replication/walsender.c
@@ -282,10 +282,8 @@ InitWalSender(void)
/* Create a per-walsender data structure in shared memory */
InitWalSenderSlot();
- /*
- * We don't currently need any ResourceOwner in a walsender process, but
- * if we did, we could call CreateAuxProcessResourceOwner here.
- */
+ /* need resource owner for e.g. basebackups */
+ CreateAuxProcessResourceOwner();
/*
* Let postmaster know that we're a WAL sender. Once we've declared us as
@@ -346,7 +344,7 @@ WalSndErrorCleanup(void)
* without a transaction, we've got to clean that up now.
*/
if (!IsTransactionOrTransactionBlock())
- WalSndResourceCleanup(false);
+ ReleaseAuxProcessResources(false);
if (got_STOPPING || got_SIGUSR2)
proc_exit(0);
@@ -355,34 +353,6 @@ WalSndErrorCleanup(void)
WalSndSetState(WALSNDSTATE_STARTUP);
}
-/*
- * Clean up any ResourceOwner we created.
- */
-void
-WalSndResourceCleanup(bool isCommit)
-{
- ResourceOwner resowner;
-
- if (CurrentResourceOwner == NULL)
- return;
-
- /*
- * Deleting CurrentResourceOwner is not allowed, so we must save a pointer
- * in a local variable and clear it first.
- */
- resowner = CurrentResourceOwner;
- CurrentResourceOwner = NULL;
-
- /* Now we can release resources and delete it. */
- ResourceOwnerRelease(resowner,
- RESOURCE_RELEASE_BEFORE_LOCKS, isCommit, true);
- ResourceOwnerRelease(resowner,
- RESOURCE_RELEASE_LOCKS, isCommit, true);
- ResourceOwnerRelease(resowner,
- RESOURCE_RELEASE_AFTER_LOCKS, isCommit, true);
- ResourceOwnerDelete(resowner);
-}
-
/*
* Handle a client's connection abort in an orderly manner.
*/
@@ -685,8 +655,10 @@ UploadManifest(void)
* parsing the manifest will use the cryptohash stuff, which requires a
* resource owner
*/
- Assert(CurrentResourceOwner == NULL);
- CurrentResourceOwner = ResourceOwnerCreate(NULL, "base backup");
+ Assert(AuxProcessResourceOwner != NULL);
+ Assert(CurrentResourceOwner == AuxProcessResourceOwner ||
+ CurrentResourceOwner == NULL);
+ CurrentResourceOwner = AuxProcessResourceOwner;
/* Prepare to read manifest data into a temporary context. */
mcxt = AllocSetContextCreate(CurrentMemoryContext,
@@ -723,7 +695,7 @@ UploadManifest(void)
uploaded_manifest_mcxt = mcxt;
/* clean up the resource owner we created */
- WalSndResourceCleanup(true);
+ ReleaseAuxProcessResources(true);
}
/*
--
2.45.2.827.g557ae147e6
v2.0-0004-Ensure-a-resowner-exists-for-all-paths-that-may.patch (text/x-diff, 2.3K) download
From 6e9b170059b75642e348e93e4a83b332ef9b3f99 Mon Sep 17 00:00:00 2001
From: Andres Freund <andres@anarazel.de>
Date: Thu, 1 Aug 2024 09:56:36 -0700
Subject: [PATCH v2.0 04/17] Ensure a resowner exists for all paths that may
perform AIO
---
src/backend/bootstrap/bootstrap.c | 7 +++++++
src/backend/replication/logical/logical.c | 6 ++++++
src/backend/utils/init/postinit.c | 3 ++-
3 files changed, 15 insertions(+), 1 deletion(-)
diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c
index 7637581a184..234fdc57ca7 100644
--- a/src/backend/bootstrap/bootstrap.c
+++ b/src/backend/bootstrap/bootstrap.c
@@ -331,8 +331,15 @@ BootstrapModeMain(int argc, char *argv[], bool check_only)
BaseInit();
bootstrap_signals();
+
+ /* need a resowner for IO during BootStrapXLOG() */
+ CreateAuxProcessResourceOwner();
+
BootStrapXLOG(bootstrap_data_checksum_version);
+ ReleaseAuxProcessResources(true);
+ CurrentResourceOwner = NULL;
+
/*
* To ensure that src/common/link-canary.c is linked into the backend, we
* must call it from somewhere. Here is as good as anywhere.
diff --git a/src/backend/replication/logical/logical.c b/src/backend/replication/logical/logical.c
index 3fe1774a1e9..be0c7846d00 100644
--- a/src/backend/replication/logical/logical.c
+++ b/src/backend/replication/logical/logical.c
@@ -386,6 +386,12 @@ CreateInitDecodingContext(const char *plugin,
slot->data.plugin = plugin_name;
SpinLockRelease(&slot->mutex);
+ if (CurrentResourceOwner == NULL)
+ {
+ Assert(am_walsender);
+ CurrentResourceOwner = AuxProcessResourceOwner;
+ }
+
if (XLogRecPtrIsInvalid(restart_lsn))
ReplicationSlotReserveWal();
else
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index 3b50ce19a2c..11128ea461c 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -719,7 +719,8 @@ InitPostgres(const char *in_dbname, Oid dboid,
* and ShutdownXLOG will need one. Hence, create said resource owner
* (and register a callback to clean it up after ShutdownXLOG runs).
*/
- CreateAuxProcessResourceOwner();
+ if (!bootstrap)
+ CreateAuxProcessResourceOwner();
StartupXLOG();
/* Release (and warn about) any buffer pins leaked in StartupXLOG */
--
2.45.2.827.g557ae147e6
v2.0-0005-bufmgr-smgr-Don-t-cross-segment-boundaries-in-S.patch (text/x-diff, 6.2K) download
From 7d58cc85191c96d8dc731b62810b64c5b366743b Mon Sep 17 00:00:00 2001
From: Andres Freund <andres@anarazel.de>
Date: Sat, 31 Aug 2024 22:10:35 -0400
Subject: [PATCH v2.0 05/17] bufmgr/smgr: Don't cross segment boundaries in
StartReadBuffers()
With real AIO it doesn't make sense to cross segment boundaries with one
IO. Add smgrmaxcombine() to allow upper layers to query which buffers can be
merged.
---
src/include/storage/md.h | 2 ++
src/include/storage/smgr.h | 2 ++
src/backend/storage/buffer/bufmgr.c | 18 ++++++++++++++++++
src/backend/storage/smgr/md.c | 17 +++++++++++++++++
src/backend/storage/smgr/smgr.c | 16 ++++++++++++++++
5 files changed, 55 insertions(+)
diff --git a/src/include/storage/md.h b/src/include/storage/md.h
index 620f10abdeb..b72293c79a5 100644
--- a/src/include/storage/md.h
+++ b/src/include/storage/md.h
@@ -32,6 +32,8 @@ extern void mdzeroextend(SMgrRelation reln, ForkNumber forknum,
BlockNumber blocknum, int nblocks, bool skipFsync);
extern bool mdprefetch(SMgrRelation reln, ForkNumber forknum,
BlockNumber blocknum, int nblocks);
+extern uint32 mdmaxcombine(SMgrRelation reln, ForkNumber forknum,
+ BlockNumber blocknum);
extern void mdreadv(SMgrRelation reln, ForkNumber forknum, BlockNumber blocknum,
void **buffers, BlockNumber nblocks);
extern void mdwritev(SMgrRelation reln, ForkNumber forknum,
diff --git a/src/include/storage/smgr.h b/src/include/storage/smgr.h
index e15b20a566a..899d0d681c5 100644
--- a/src/include/storage/smgr.h
+++ b/src/include/storage/smgr.h
@@ -92,6 +92,8 @@ extern void smgrzeroextend(SMgrRelation reln, ForkNumber forknum,
BlockNumber blocknum, int nblocks, bool skipFsync);
extern bool smgrprefetch(SMgrRelation reln, ForkNumber forknum,
BlockNumber blocknum, int nblocks);
+extern uint32 smgrmaxcombine(SMgrRelation reln, ForkNumber forknum,
+ BlockNumber blocknum);
extern void smgrreadv(SMgrRelation reln, ForkNumber forknum,
BlockNumber blocknum,
void **buffers, BlockNumber nblocks);
diff --git a/src/backend/storage/buffer/bufmgr.c b/src/backend/storage/buffer/bufmgr.c
index ec957635f2a..f2e608f597d 100644
--- a/src/backend/storage/buffer/bufmgr.c
+++ b/src/backend/storage/buffer/bufmgr.c
@@ -1286,6 +1286,7 @@ StartReadBuffersImpl(ReadBuffersOperation *operation,
{
int actual_nblocks = *nblocks;
int io_buffers_len = 0;
+ int maxcombine = 0;
Assert(*nblocks > 0);
Assert(*nblocks <= MAX_IO_COMBINE_LIMIT);
@@ -1317,6 +1318,23 @@ StartReadBuffersImpl(ReadBuffersOperation *operation,
{
/* Extend the readable range to cover this block. */
io_buffers_len++;
+
+ /*
+ * Check how many blocks we can cover with the same IO. The smgr
+ * implementation might e.g. be limited due to a segment boundary.
+ */
+ if (i == 0 && actual_nblocks > 1)
+ {
+ maxcombine = smgrmaxcombine(operation->smgr,
+ operation->forknum,
+ blockNum);
+ if (maxcombine < actual_nblocks)
+ {
+ elog(DEBUG2, "limiting nblocks at %u from %u to %u",
+ blockNum, actual_nblocks, maxcombine);
+ actual_nblocks = maxcombine;
+ }
+ }
}
}
*nblocks = actual_nblocks;
diff --git a/src/backend/storage/smgr/md.c b/src/backend/storage/smgr/md.c
index 6796756358f..6cd81a61faa 100644
--- a/src/backend/storage/smgr/md.c
+++ b/src/backend/storage/smgr/md.c
@@ -803,6 +803,17 @@ buffers_to_iovec(struct iovec *iov, void **buffers, int nblocks)
return iovcnt;
}
+uint32
+mdmaxcombine(SMgrRelation reln, ForkNumber forknum,
+ BlockNumber blocknum)
+{
+ BlockNumber segoff;
+
+ segoff = blocknum % ((BlockNumber) RELSEG_SIZE);
+
+ return RELSEG_SIZE - segoff;
+}
+
/*
* mdreadv() -- Read the specified blocks from a relation.
*/
@@ -833,6 +844,9 @@ mdreadv(SMgrRelation reln, ForkNumber forknum, BlockNumber blocknum,
RELSEG_SIZE - (blocknum % ((BlockNumber) RELSEG_SIZE)));
nblocks_this_segment = Min(nblocks_this_segment, lengthof(iov));
+ if (nblocks_this_segment != nblocks)
+ elog(ERROR, "read crossing segment boundary");
+
iovcnt = buffers_to_iovec(iov, buffers, nblocks_this_segment);
size_this_segment = nblocks_this_segment * BLCKSZ;
transferred_this_segment = 0;
@@ -956,6 +970,9 @@ mdwritev(SMgrRelation reln, ForkNumber forknum, BlockNumber blocknum,
RELSEG_SIZE - (blocknum % ((BlockNumber) RELSEG_SIZE)));
nblocks_this_segment = Min(nblocks_this_segment, lengthof(iov));
+ if (nblocks_this_segment != nblocks)
+ elog(ERROR, "write crossing segment boundary");
+
iovcnt = buffers_to_iovec(iov, (void **) buffers, nblocks_this_segment);
size_this_segment = nblocks_this_segment * BLCKSZ;
transferred_this_segment = 0;
diff --git a/src/backend/storage/smgr/smgr.c b/src/backend/storage/smgr/smgr.c
index 7b9fa103eff..ee31db85eec 100644
--- a/src/backend/storage/smgr/smgr.c
+++ b/src/backend/storage/smgr/smgr.c
@@ -88,6 +88,8 @@ typedef struct f_smgr
BlockNumber blocknum, int nblocks, bool skipFsync);
bool (*smgr_prefetch) (SMgrRelation reln, ForkNumber forknum,
BlockNumber blocknum, int nblocks);
+ uint32 (*smgr_maxcombine) (SMgrRelation reln, ForkNumber forknum,
+ BlockNumber blocknum);
void (*smgr_readv) (SMgrRelation reln, ForkNumber forknum,
BlockNumber blocknum,
void **buffers, BlockNumber nblocks);
@@ -117,6 +119,7 @@ static const f_smgr smgrsw[] = {
.smgr_extend = mdextend,
.smgr_zeroextend = mdzeroextend,
.smgr_prefetch = mdprefetch,
+ .smgr_maxcombine = mdmaxcombine,
.smgr_readv = mdreadv,
.smgr_writev = mdwritev,
.smgr_writeback = mdwriteback,
@@ -588,6 +591,19 @@ smgrprefetch(SMgrRelation reln, ForkNumber forknum, BlockNumber blocknum,
return smgrsw[reln->smgr_which].smgr_prefetch(reln, forknum, blocknum, nblocks);
}
+/*
+ * smgrmaxcombine() - Return the maximum number of total blocks that can be
+ * combined with an IO starting at blocknum.
+ *
+ * The returned value includes the io for blocknum itself.
+ */
+uint32
+smgrmaxcombine(SMgrRelation reln, ForkNumber forknum,
+ BlockNumber blocknum)
+{
+ return smgrsw[reln->smgr_which].smgr_maxcombine(reln, forknum, blocknum);
+}
+
/*
* smgrreadv() -- read a particular block range from a relation into the
* supplied buffers.
--
2.45.2.827.g557ae147e6
v2.0-0006-aio-Add-liburing-dependency.patch (text/x-diff, 9.9K) download
From 1e1c9d880f71f4548c9616d57d6071e3f90d8f70 Mon Sep 17 00:00:00 2001
From: Andres Freund <andres@anarazel.de>
Date: Wed, 5 Jun 2024 19:37:25 -0700
Subject: [PATCH v2.0 06/17] aio: Add liburing dependency
Not yet used.
Author:
Reviewed-by:
Discussion: https://postgr.es/m/
Backpatch:
---
src/include/pg_config.h.in | 3 +
src/makefiles/meson.build | 3 +
configure | 138 +++++++++++++++++++++++++++++++++++++
configure.ac | 11 +++
meson.build | 14 ++++
meson_options.txt | 3 +
src/Makefile.global.in | 4 ++
7 files changed, 176 insertions(+)
diff --git a/src/include/pg_config.h.in b/src/include/pg_config.h.in
index 979925cc2e2..397133b51ac 100644
--- a/src/include/pg_config.h.in
+++ b/src/include/pg_config.h.in
@@ -708,6 +708,9 @@
/* Define to 1 to build with LDAP support. (--with-ldap) */
#undef USE_LDAP
+/* Define to build with io-uring support. (--with-liburing) */
+#undef USE_LIBURING
+
/* Define to 1 to build with XML support. (--with-libxml) */
#undef USE_LIBXML
diff --git a/src/makefiles/meson.build b/src/makefiles/meson.build
index 850e9275845..cca689b2028 100644
--- a/src/makefiles/meson.build
+++ b/src/makefiles/meson.build
@@ -200,6 +200,8 @@ pgxs_empty = [
'PTHREAD_CFLAGS', 'PTHREAD_LIBS',
'ICU_LIBS',
+
+ 'LIBURING_CFLAGS', 'LIBURING_LIBS',
]
if host_system == 'windows' and cc.get_argument_syntax() != 'msvc'
@@ -230,6 +232,7 @@ pgxs_deps = {
'gssapi': gssapi,
'icu': icu,
'ldap': ldap,
+ 'liburing': liburing,
'libxml': libxml,
'libxslt': libxslt,
'llvm': llvm,
diff --git a/configure b/configure
index 537366945c0..317a462f610 100755
--- a/configure
+++ b/configure
@@ -654,6 +654,8 @@ LIBOBJS
OPENSSL
ZSTD
LZ4
+LIBURING_LIBS
+LIBURING_CFLAGS
UUID_LIBS
LDAP_LIBS_BE
LDAP_LIBS_FE
@@ -712,6 +714,7 @@ XML2_CFLAGS
XML2_CONFIG
with_libxml
with_uuid
+with_liburing
with_readline
with_systemd
with_selinux
@@ -865,6 +868,7 @@ with_selinux
with_systemd
with_readline
with_libedit_preferred
+with_liburing
with_uuid
with_ossp_uuid
with_libxml
@@ -907,6 +911,8 @@ LDFLAGS_EX
LDFLAGS_SL
PERL
PYTHON
+LIBURING_CFLAGS
+LIBURING_LIBS
MSGFMT
TCLSH'
@@ -1574,6 +1580,7 @@ Optional Packages:
--without-readline do not use GNU Readline nor BSD Libedit for editing
--with-libedit-preferred
prefer BSD Libedit over GNU Readline
+ --with-liburing use liburing for async io
--with-uuid=LIB build contrib/uuid-ossp using LIB (bsd,e2fs,ossp)
--with-ossp-uuid obsolete spelling of --with-uuid=ossp
--with-libxml build with XML support
@@ -1617,6 +1624,10 @@ Some influential environment variables:
LDFLAGS_SL extra linker flags for linking shared libraries only
PERL Perl program
PYTHON Python program
+ LIBURING_CFLAGS
+ C compiler flags for LIBURING, overriding pkg-config
+ LIBURING_LIBS
+ linker flags for LIBURING, overriding pkg-config
MSGFMT msgfmt program for NLS
TCLSH Tcl interpreter program (tclsh)
@@ -8664,6 +8675,40 @@ fi
+#
+# liburing
+#
+{ $as_echo "$as_me:${as_lineno-$LINENO}: checking whether to build with liburing support" >&5
+$as_echo_n "checking whether to build with liburing support... " >&6; }
+
+
+
+# Check whether --with-liburing was given.
+if test "${with_liburing+set}" = set; then :
+ withval=$with_liburing;
+ case $withval in
+ yes)
+
+$as_echo "#define USE_LIBURING 1" >>confdefs.h
+
+ ;;
+ no)
+ :
+ ;;
+ *)
+ as_fn_error $? "no argument expected for --with-liburing option" "$LINENO" 5
+ ;;
+ esac
+
+else
+ with_liburing=no
+
+fi
+
+
+{ $as_echo "$as_me:${as_lineno-$LINENO}: result: $with_liburing" >&5
+$as_echo "$with_liburing" >&6; }
+
#
# UUID library
@@ -13222,6 +13267,99 @@ fi
fi
+if test "$with_liburing" = yes; then
+
+pkg_failed=no
+{ $as_echo "$as_me:${as_lineno-$LINENO}: checking for liburing" >&5
+$as_echo_n "checking for liburing... " >&6; }
+
+if test -n "$LIBURING_CFLAGS"; then
+ pkg_cv_LIBURING_CFLAGS="$LIBURING_CFLAGS"
+ elif test -n "$PKG_CONFIG"; then
+ if test -n "$PKG_CONFIG" && \
+ { { $as_echo "$as_me:${as_lineno-$LINENO}: \$PKG_CONFIG --exists --print-errors \"liburing\""; } >&5
+ ($PKG_CONFIG --exists --print-errors "liburing") 2>&5
+ ac_status=$?
+ $as_echo "$as_me:${as_lineno-$LINENO}: \$? = $ac_status" >&5
+ test $ac_status = 0; }; then
+ pkg_cv_LIBURING_CFLAGS=`$PKG_CONFIG --cflags "liburing" 2>/dev/null`
+ test "x$?" != "x0" && pkg_failed=yes
+else
+ pkg_failed=yes
+fi
+ else
+ pkg_failed=untried
+fi
+if test -n "$LIBURING_LIBS"; then
+ pkg_cv_LIBURING_LIBS="$LIBURING_LIBS"
+ elif test -n "$PKG_CONFIG"; then
+ if test -n "$PKG_CONFIG" && \
+ { { $as_echo "$as_me:${as_lineno-$LINENO}: \$PKG_CONFIG --exists --print-errors \"liburing\""; } >&5
+ ($PKG_CONFIG --exists --print-errors "liburing") 2>&5
+ ac_status=$?
+ $as_echo "$as_me:${as_lineno-$LINENO}: \$? = $ac_status" >&5
+ test $ac_status = 0; }; then
+ pkg_cv_LIBURING_LIBS=`$PKG_CONFIG --libs "liburing" 2>/dev/null`
+ test "x$?" != "x0" && pkg_failed=yes
+else
+ pkg_failed=yes
+fi
+ else
+ pkg_failed=untried
+fi
+
+
+
+if test $pkg_failed = yes; then
+ { $as_echo "$as_me:${as_lineno-$LINENO}: result: no" >&5
+$as_echo "no" >&6; }
+
+if $PKG_CONFIG --atleast-pkgconfig-version 0.20; then
+ _pkg_short_errors_supported=yes
+else
+ _pkg_short_errors_supported=no
+fi
+ if test $_pkg_short_errors_supported = yes; then
+ LIBURING_PKG_ERRORS=`$PKG_CONFIG --short-errors --print-errors --cflags --libs "liburing" 2>&1`
+ else
+ LIBURING_PKG_ERRORS=`$PKG_CONFIG --print-errors --cflags --libs "liburing" 2>&1`
+ fi
+ # Put the nasty error message in config.log where it belongs
+ echo "$LIBURING_PKG_ERRORS" >&5
+
+ as_fn_error $? "Package requirements (liburing) were not met:
+
+$LIBURING_PKG_ERRORS
+
+Consider adjusting the PKG_CONFIG_PATH environment variable if you
+installed software in a non-standard prefix.
+
+Alternatively, you may set the environment variables LIBURING_CFLAGS
+and LIBURING_LIBS to avoid the need to call pkg-config.
+See the pkg-config man page for more details." "$LINENO" 5
+elif test $pkg_failed = untried; then
+ { $as_echo "$as_me:${as_lineno-$LINENO}: result: no" >&5
+$as_echo "no" >&6; }
+ { { $as_echo "$as_me:${as_lineno-$LINENO}: error: in \`$ac_pwd':" >&5
+$as_echo "$as_me: error: in \`$ac_pwd':" >&2;}
+as_fn_error $? "The pkg-config script could not be found or is too old. Make sure it
+is in your PATH or set the PKG_CONFIG environment variable to the full
+path to pkg-config.
+
+Alternatively, you may set the environment variables LIBURING_CFLAGS
+and LIBURING_LIBS to avoid the need to call pkg-config.
+See the pkg-config man page for more details.
+
+To get pkg-config, see <http://pkg-config.freedesktop.org/>.
+See \`config.log' for more details" "$LINENO" 5; }
+else
+ LIBURING_CFLAGS=$pkg_cv_LIBURING_CFLAGS
+ LIBURING_LIBS=$pkg_cv_LIBURING_LIBS
+ { $as_echo "$as_me:${as_lineno-$LINENO}: result: yes" >&5
+$as_echo "yes" >&6; }
+
+fi
+fi
##
## Header files
diff --git a/configure.ac b/configure.ac
index 4e279c4bd66..fa634ecf9e0 100644
--- a/configure.ac
+++ b/configure.ac
@@ -970,6 +970,14 @@ AC_SUBST(with_readline)
PGAC_ARG_BOOL(with, libedit-preferred, no,
[prefer BSD Libedit over GNU Readline])
+#
+# liburing
+#
+AC_MSG_CHECKING([whether to build with liburing support])
+PGAC_ARG_BOOL(with, liburing, no, [use liburing for async io],
+ [AC_DEFINE([USE_LIBURING], 1, [Define to build with io-uring support. (--with-liburing)])])
+AC_MSG_RESULT([$with_liburing])
+AC_SUBST(with_liburing)
#
# UUID library
@@ -1430,6 +1438,9 @@ elif test "$with_uuid" = ossp ; then
fi
AC_SUBST(UUID_LIBS)
+if test "$with_liburing" = yes; then
+ PKG_CHECK_MODULES(LIBURING, liburing)
+fi
##
## Header files
diff --git a/meson.build b/meson.build
index ea07126f78e..71200f4cb8f 100644
--- a/meson.build
+++ b/meson.build
@@ -848,6 +848,18 @@ endif
+###############################################################
+# Library: liburing
+###############################################################
+
+liburingopt = get_option('liburing')
+liburing = dependency('liburing', required: liburingopt)
+if liburing.found()
+ cdata.set('USE_LIBURING', 1)
+endif
+
+
+
###############################################################
# Library: libxml
###############################################################
@@ -3103,6 +3115,7 @@ backend_both_deps += [
icu_i18n,
ldap,
libintl,
+ liburing,
libxml,
lz4,
pam,
@@ -3747,6 +3760,7 @@ if meson.version().version_compare('>=0.57')
'gss': gssapi,
'icu': icu,
'ldap': ldap,
+ 'liburing': liburing,
'libxml': libxml,
'libxslt': libxslt,
'llvm': llvm,
diff --git a/meson_options.txt b/meson_options.txt
index b9421557606..084eebe72d7 100644
--- a/meson_options.txt
+++ b/meson_options.txt
@@ -103,6 +103,9 @@ option('ldap', type: 'feature', value: 'auto',
option('libedit_preferred', type: 'boolean', value: false,
description: 'Prefer BSD Libedit over GNU Readline')
+option('liburing', type : 'feature', value: 'auto',
+ description: 'Use liburing for async io')
+
option('libxml', type: 'feature', value: 'auto',
description: 'XML support')
diff --git a/src/Makefile.global.in b/src/Makefile.global.in
index 42f50b49761..a8ff18faed6 100644
--- a/src/Makefile.global.in
+++ b/src/Makefile.global.in
@@ -190,6 +190,7 @@ with_systemd = @with_systemd@
with_gssapi = @with_gssapi@
with_krb_srvnam = @with_krb_srvnam@
with_ldap = @with_ldap@
+with_liburing = @with_liburing@
with_libxml = @with_libxml@
with_libxslt = @with_libxslt@
with_llvm = @with_llvm@
@@ -216,6 +217,9 @@ krb_srvtab = @krb_srvtab@
ICU_CFLAGS = @ICU_CFLAGS@
ICU_LIBS = @ICU_LIBS@
+LIBURING_CFLAGS = @LIBURING_CFLAGS@
+LIBURING_LIBS = @LIBURING_LIBS@
+
TCLSH = @TCLSH@
TCL_LIBS = @TCL_LIBS@
TCL_LIB_SPEC = @TCL_LIB_SPEC@
--
2.45.2.827.g557ae147e6
v2.0-0007-aio-Basic-subsystem-initialization.patch (text/x-diff, 11.6K) download
From 8b3dabb0ec36a6aea6b5f9d30fadefc8748bfb9c Mon Sep 17 00:00:00 2001
From: Andres Freund <andres@anarazel.de>
Date: Mon, 10 Jun 2024 13:42:58 -0700
Subject: [PATCH v2.0 07/17] aio: Basic subsystem initialization
This is just separate to make it easier to review the tendrils into various
places.
---
src/include/storage/aio.h | 42 +++++++++++++++++
src/include/storage/aio_init.h | 26 +++++++++++
src/backend/postmaster/postmaster.c | 8 ++++
src/backend/storage/aio/Makefile | 2 +
src/backend/storage/aio/aio.c | 35 ++++++++++++++
src/backend/storage/aio/aio_init.c | 46 +++++++++++++++++++
src/backend/storage/aio/meson.build | 2 +
src/backend/storage/ipc/ipci.c | 3 ++
src/backend/tcop/postgres.c | 7 +++
src/backend/utils/init/miscinit.c | 3 ++
src/backend/utils/init/postinit.c | 3 ++
src/backend/utils/misc/guc_tables.c | 11 +++++
src/backend/utils/misc/postgresql.conf.sample | 7 +++
src/tools/pgindent/typedefs.list | 1 +
14 files changed, 196 insertions(+)
create mode 100644 src/include/storage/aio.h
create mode 100644 src/include/storage/aio_init.h
create mode 100644 src/backend/storage/aio/aio.c
create mode 100644 src/backend/storage/aio/aio_init.c
diff --git a/src/include/storage/aio.h b/src/include/storage/aio.h
new file mode 100644
index 00000000000..98fafcf9bc4
--- /dev/null
+++ b/src/include/storage/aio.h
@@ -0,0 +1,42 @@
+/*-------------------------------------------------------------------------
+ *
+ * aio.h
+ * Main AIO interface
+ *
+ *
+ * Portions Copyright (c) 1996-2024, PostgreSQL Global Development Group
+ * Portions Copyright (c) 1994, Regents of the University of California
+ *
+ * src/include/storage/aio.h
+ *
+ *-------------------------------------------------------------------------
+ */
+#ifndef AIO_H
+#define AIO_H
+
+
+#include "utils/guc_tables.h"
+
+
+/* GUC related */
+extern void assign_io_method(int newval, void *extra);
+
+
+/* Enum for io_method GUC. */
+typedef enum IoMethod
+{
+ IOMETHOD_WORKER = 0,
+ IOMETHOD_IO_URING,
+} IoMethod;
+
+
+/* We'll default to bgworker. */
+#define DEFAULT_IO_METHOD IOMETHOD_WORKER
+
+
+/* GUCs */
+extern const struct config_enum_entry io_method_options[];
+extern int io_method;
+
+
+#endif /* AIO_H */
diff --git a/src/include/storage/aio_init.h b/src/include/storage/aio_init.h
new file mode 100644
index 00000000000..5bcfb8a9d58
--- /dev/null
+++ b/src/include/storage/aio_init.h
@@ -0,0 +1,26 @@
+/*-------------------------------------------------------------------------
+ *
+ * aio_init.h
+ * AIO initialization - kept separate as initialization sites don't need to
+ * know about AIO itself and AIO users don't need to know about initialization.
+ *
+ *
+ * Portions Copyright (c) 1996-2024, PostgreSQL Global Development Group
+ * Portions Copyright (c) 1994, Regents of the University of California
+ *
+ * src/include/storage/aio_init.h
+ *
+ *-------------------------------------------------------------------------
+ */
+#ifndef AIO_INIT_H
+#define AIO_INIT_H
+
+
+extern Size AioShmemSize(void);
+extern void AioShmemInit(void);
+
+extern void pgaio_postmaster_init(void);
+extern void pgaio_postmaster_child_init_local(void);
+extern void pgaio_postmaster_child_init(void);
+
+#endif /* AIO_INIT_H */
diff --git a/src/backend/postmaster/postmaster.c b/src/backend/postmaster/postmaster.c
index a6fff93db34..921073a2ca4 100644
--- a/src/backend/postmaster/postmaster.c
+++ b/src/backend/postmaster/postmaster.c
@@ -111,6 +111,7 @@
#include "replication/logicallauncher.h"
#include "replication/slotsync.h"
#include "replication/walsender.h"
+#include "storage/aio_init.h"
#include "storage/fd.h"
#include "storage/ipc.h"
#include "storage/pmsignal.h"
@@ -941,6 +942,13 @@ PostmasterMain(int argc, char *argv[])
ExitPostmaster(0);
}
+ /*
+ * As AIO might create internal FDs, and will trigger shared memory
+ * allocations, need to do this before reset_shared() and
+ * set_max_safe_fds().
+ */
+ pgaio_postmaster_init();
+
/*
* Set up shared memory and semaphores.
*
diff --git a/src/backend/storage/aio/Makefile b/src/backend/storage/aio/Makefile
index 2f29a9ec4d1..eaeaeeee8e3 100644
--- a/src/backend/storage/aio/Makefile
+++ b/src/backend/storage/aio/Makefile
@@ -9,6 +9,8 @@ top_builddir = ../../../..
include $(top_builddir)/src/Makefile.global
OBJS = \
+ aio.o \
+ aio_init.o \
read_stream.o
include $(top_srcdir)/src/backend/common.mk
diff --git a/src/backend/storage/aio/aio.c b/src/backend/storage/aio/aio.c
new file mode 100644
index 00000000000..67f6b52de91
--- /dev/null
+++ b/src/backend/storage/aio/aio.c
@@ -0,0 +1,35 @@
+/*-------------------------------------------------------------------------
+ *
+ * aio.c
+ * Asynchronous I/O subsytem.
+ *
+ * Portions Copyright (c) 1996-2024, PostgreSQL Global Development Group
+ * Portions Copyright (c) 1994, Regents of the University of California
+ *
+ * IDENTIFICATION
+ * src/backend/storage/aio/aio.c
+ *
+ *-------------------------------------------------------------------------
+ */
+
+#include "postgres.h"
+
+#include "storage/aio.h"
+
+
+/* Options for io_method. */
+const struct config_enum_entry io_method_options[] = {
+ {"worker", IOMETHOD_WORKER, false},
+#ifdef USE_LIBURING
+ {"io_uring", IOMETHOD_IO_URING, false},
+#endif
+ {NULL, 0, false}
+};
+
+int io_method = IOMETHOD_WORKER;
+
+
+void
+assign_io_method(int newval, void *extra)
+{
+}
diff --git a/src/backend/storage/aio/aio_init.c b/src/backend/storage/aio/aio_init.c
new file mode 100644
index 00000000000..1c277a7eb3b
--- /dev/null
+++ b/src/backend/storage/aio/aio_init.c
@@ -0,0 +1,46 @@
+/*-------------------------------------------------------------------------
+ *
+ * aio_init.c
+ * Asynchronous I/O subsytem - Initialization
+ *
+ * Portions Copyright (c) 1996-2024, PostgreSQL Global Development Group
+ * Portions Copyright (c) 1994, Regents of the University of California
+ *
+ * IDENTIFICATION
+ * src/backend/storage/aio/aio_init.c
+ *
+ *-------------------------------------------------------------------------
+ */
+
+#include "postgres.h"
+
+#include "storage/aio_init.h"
+
+
+Size
+AioShmemSize(void)
+{
+ Size sz = 0;
+
+ return sz;
+}
+
+void
+AioShmemInit(void)
+{
+}
+
+void
+pgaio_postmaster_init(void)
+{
+}
+
+void
+pgaio_postmaster_child_init(void)
+{
+}
+
+void
+pgaio_postmaster_child_init_local(void)
+{
+}
diff --git a/src/backend/storage/aio/meson.build b/src/backend/storage/aio/meson.build
index 10e1aa3b20b..8d20759ebf8 100644
--- a/src/backend/storage/aio/meson.build
+++ b/src/backend/storage/aio/meson.build
@@ -1,5 +1,7 @@
# Copyright (c) 2024, PostgreSQL Global Development Group
backend_sources += files(
+ 'aio.c',
+ 'aio_init.c',
'read_stream.c',
)
diff --git a/src/backend/storage/ipc/ipci.c b/src/backend/storage/ipc/ipci.c
index 6caeca3a8e6..f0227a12a7d 100644
--- a/src/backend/storage/ipc/ipci.c
+++ b/src/backend/storage/ipc/ipci.c
@@ -39,6 +39,7 @@
#include "replication/slotsync.h"
#include "replication/walreceiver.h"
#include "replication/walsender.h"
+#include "storage/aio_init.h"
#include "storage/bufmgr.h"
#include "storage/dsm.h"
#include "storage/dsm_registry.h"
@@ -152,6 +153,7 @@ CalculateShmemSize(int *num_semaphores)
size = add_size(size, InjectionPointShmemSize());
size = add_size(size, SlotSyncShmemSize());
size = add_size(size, WaitLSNShmemSize());
+ size = add_size(size, AioShmemSize());
/* include additional requested shmem from preload libraries */
size = add_size(size, total_addin_request);
@@ -339,6 +341,7 @@ CreateOrAttachShmemStructs(void)
WaitEventCustomShmemInit();
InjectionPointShmemInit();
WaitLSNShmemInit();
+ AioShmemInit();
}
/*
diff --git a/src/backend/tcop/postgres.c b/src/backend/tcop/postgres.c
index 8bc6bea1135..4dc46b17b41 100644
--- a/src/backend/tcop/postgres.c
+++ b/src/backend/tcop/postgres.c
@@ -61,6 +61,7 @@
#include "replication/slot.h"
#include "replication/walsender.h"
#include "rewrite/rewriteHandler.h"
+#include "storage/aio_init.h"
#include "storage/bufmgr.h"
#include "storage/ipc.h"
#include "storage/pmsignal.h"
@@ -4198,6 +4199,12 @@ PostgresSingleUserMain(int argc, char *argv[],
*/
InitProcess();
+ /* AIO is needed during InitPostgres() */
+ pgaio_postmaster_init();
+ pgaio_postmaster_child_init_local();
+
+ set_max_safe_fds();
+
/*
* Now that sufficient infrastructure has been initialized, PostgresMain()
* can do the rest.
diff --git a/src/backend/utils/init/miscinit.c b/src/backend/utils/init/miscinit.c
index 537d92c0cfd..b8fa2e64ffe 100644
--- a/src/backend/utils/init/miscinit.c
+++ b/src/backend/utils/init/miscinit.c
@@ -40,6 +40,7 @@
#include "postmaster/interrupt.h"
#include "postmaster/postmaster.h"
#include "replication/slotsync.h"
+#include "storage/aio_init.h"
#include "storage/fd.h"
#include "storage/ipc.h"
#include "storage/latch.h"
@@ -137,6 +138,8 @@ InitPostmasterChild(void)
InitProcessLocalLatch();
InitializeLatchWaitSet();
+ pgaio_postmaster_child_init_local();
+
/*
* If possible, make this process a group leader, so that the postmaster
* can signal any child processes too. Not all processes will have
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index 11128ea461c..f1151645242 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -43,6 +43,7 @@
#include "replication/slot.h"
#include "replication/slotsync.h"
#include "replication/walsender.h"
+#include "storage/aio_init.h"
#include "storage/bufmgr.h"
#include "storage/fd.h"
#include "storage/ipc.h"
@@ -589,6 +590,8 @@ BaseInit(void)
*/
pgstat_initialize();
+ pgaio_postmaster_child_init();
+
/* Do local initialization of storage and buffer managers */
InitSync();
smgrinit();
diff --git a/src/backend/utils/misc/guc_tables.c b/src/backend/utils/misc/guc_tables.c
index 521ec5591c8..4961a5f4b16 100644
--- a/src/backend/utils/misc/guc_tables.c
+++ b/src/backend/utils/misc/guc_tables.c
@@ -71,6 +71,7 @@
#include "replication/slot.h"
#include "replication/slotsync.h"
#include "replication/syncrep.h"
+#include "storage/aio.h"
#include "storage/bufmgr.h"
#include "storage/bufpage.h"
#include "storage/large_object.h"
@@ -5196,6 +5197,16 @@ struct config_enum ConfigureNamesEnum[] =
NULL, NULL, NULL
},
+ {
+ {"io_method", PGC_POSTMASTER, RESOURCES_MEM,
+ gettext_noop("Selects the method of asynchronous I/O to use."),
+ NULL
+ },
+ &io_method,
+ DEFAULT_IO_METHOD, io_method_options,
+ NULL, assign_io_method, NULL
+ },
+
/* End-of-list marker */
{
{NULL, 0, 0, NULL, NULL}, NULL, 0, NULL, NULL, NULL, NULL
diff --git a/src/backend/utils/misc/postgresql.conf.sample b/src/backend/utils/misc/postgresql.conf.sample
index 667e0dc40a2..e904c3fea30 100644
--- a/src/backend/utils/misc/postgresql.conf.sample
+++ b/src/backend/utils/misc/postgresql.conf.sample
@@ -835,6 +835,13 @@
#include = '...' # include file
+#------------------------------------------------------------------------------
+# WIP AIO GUC docs
+#------------------------------------------------------------------------------
+
+#io_method = worker
+
+
#------------------------------------------------------------------------------
# CUSTOMIZED OPTIONS
#------------------------------------------------------------------------------
diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list
index 9e951a9e6f3..309686627e7 100644
--- a/src/tools/pgindent/typedefs.list
+++ b/src/tools/pgindent/typedefs.list
@@ -1257,6 +1257,7 @@ IntervalAggState
IntoClause
InvalMessageArray
InvalidationMsgsGroup
+IoMethod
IpcMemoryId
IpcMemoryKey
IpcMemoryState
--
2.45.2.827.g557ae147e6
v2.0-0008-aio-Skeleton-IO-worker-infrastructure.patch (text/x-diff, 21.3K) download
From 4f6f260ff706c769d5e4f40e5fc23c2c3105afa2 Mon Sep 17 00:00:00 2001
From: Andres Freund <andres@anarazel.de>
Date: Wed, 28 Aug 2024 14:28:36 -0400
Subject: [PATCH v2.0 08/17] aio: Skeleton IO worker infrastructure
This doesn't do anything useful on its own, but the code that needs to be
touched is independent of other changes.
Remarks:
- should completely get rid of ID assignment logic in postmaster.c
- postmaster.c badly needs a refactoring.
- dynamic increase / decrease of workers based on IO load
Author:
Reviewed-by:
Discussion: https://postgr.es/m/
Backpatch:
---
src/include/miscadmin.h | 2 +
src/include/postmaster/postmaster.h | 1 +
src/include/storage/aio_init.h | 2 +
src/include/storage/io_worker.h | 22 +++
src/include/storage/proc.h | 4 +-
src/backend/postmaster/launch_backend.c | 2 +
src/backend/postmaster/postmaster.c | 186 ++++++++++++++++--
src/backend/storage/aio/Makefile | 1 +
src/backend/storage/aio/meson.build | 1 +
src/backend/storage/aio/method_worker.c | 84 ++++++++
src/backend/tcop/postgres.c | 2 +
src/backend/utils/activity/pgstat_io.c | 1 +
.../utils/activity/wait_event_names.txt | 1 +
src/backend/utils/init/miscinit.c | 3 +
src/backend/utils/misc/guc_tables.c | 13 ++
src/backend/utils/misc/postgresql.conf.sample | 3 +-
16 files changed, 312 insertions(+), 16 deletions(-)
create mode 100644 src/include/storage/io_worker.h
create mode 100644 src/backend/storage/aio/method_worker.c
diff --git a/src/include/miscadmin.h b/src/include/miscadmin.h
index 25348e71eb9..d043445b544 100644
--- a/src/include/miscadmin.h
+++ b/src/include/miscadmin.h
@@ -352,6 +352,7 @@ typedef enum BackendType
B_ARCHIVER,
B_BG_WRITER,
B_CHECKPOINTER,
+ B_IO_WORKER,
B_STARTUP,
B_WAL_RECEIVER,
B_WAL_SUMMARIZER,
@@ -380,6 +381,7 @@ extern PGDLLIMPORT BackendType MyBackendType;
#define AmWalReceiverProcess() (MyBackendType == B_WAL_RECEIVER)
#define AmWalSummarizerProcess() (MyBackendType == B_WAL_SUMMARIZER)
#define AmWalWriterProcess() (MyBackendType == B_WAL_WRITER)
+#define AmIoWorkerProcess() (MyBackendType == B_IO_WORKER)
extern const char *GetBackendTypeDesc(BackendType backendType);
diff --git a/src/include/postmaster/postmaster.h b/src/include/postmaster/postmaster.h
index 63c12917cfe..4cc000df79e 100644
--- a/src/include/postmaster/postmaster.h
+++ b/src/include/postmaster/postmaster.h
@@ -62,6 +62,7 @@ extern void InitProcessGlobals(void);
extern int MaxLivePostmasterChildren(void);
extern bool PostmasterMarkPIDForWorkerNotify(int);
+extern void assign_io_workers(int newval, void *extra);
#ifdef WIN32
extern void pgwin32_register_deadchild_callback(HANDLE procHandle, DWORD procId);
diff --git a/src/include/storage/aio_init.h b/src/include/storage/aio_init.h
index 5bcfb8a9d58..a38dd982fbe 100644
--- a/src/include/storage/aio_init.h
+++ b/src/include/storage/aio_init.h
@@ -23,4 +23,6 @@ extern void pgaio_postmaster_init(void);
extern void pgaio_postmaster_child_init_local(void);
extern void pgaio_postmaster_child_init(void);
+extern bool pgaio_workers_enabled(void);
+
#endif /* AIO_INIT_H */
diff --git a/src/include/storage/io_worker.h b/src/include/storage/io_worker.h
new file mode 100644
index 00000000000..ba5dcb9e6e4
--- /dev/null
+++ b/src/include/storage/io_worker.h
@@ -0,0 +1,22 @@
+/*-------------------------------------------------------------------------
+ *
+ * io_worker.h
+ * IO worker for implementing AIO "ourselves"
+ *
+ *
+ * Portions Copyright (c) 1996-2024, PostgreSQL Global Development Group
+ * Portions Copyright (c) 1994, Regents of the University of California
+ *
+ * src/include/storage/io.h
+ *
+ *-------------------------------------------------------------------------
+ */
+#ifndef IO_WORKER_H
+#define IO_WORKER_H
+
+
+extern void IoWorkerMain(char *startup_data, size_t startup_data_len) pg_attribute_noreturn();
+
+extern int io_workers;
+
+#endif /* IO_WORKER_H */
diff --git a/src/include/storage/proc.h b/src/include/storage/proc.h
index deeb06c9e01..b466ba843d6 100644
--- a/src/include/storage/proc.h
+++ b/src/include/storage/proc.h
@@ -442,7 +442,9 @@ extern PGDLLIMPORT PGPROC *PreparedXactProcs;
* 2 slots, but WAL writer is launched only after startup has exited, so we
* only need 6 slots.
*/
-#define NUM_AUXILIARY_PROCS 6
+#define MAX_IO_WORKERS 32
+#define NUM_AUXILIARY_PROCS (6 + MAX_IO_WORKERS)
+
/* configurable options */
extern PGDLLIMPORT int DeadlockTimeout;
diff --git a/src/backend/postmaster/launch_backend.c b/src/backend/postmaster/launch_backend.c
index 0ae23fdf55e..78429b2af2f 100644
--- a/src/backend/postmaster/launch_backend.c
+++ b/src/backend/postmaster/launch_backend.c
@@ -55,6 +55,7 @@
#include "replication/walreceiver.h"
#include "storage/dsm.h"
#include "storage/fd.h"
+#include "storage/io_worker.h"
#include "storage/ipc.h"
#include "storage/pg_shmem.h"
#include "storage/pmsignal.h"
@@ -199,6 +200,7 @@ static child_process_kind child_process_kinds[] = {
[B_ARCHIVER] = {"archiver", PgArchiverMain, true},
[B_BG_WRITER] = {"bgwriter", BackgroundWriterMain, true},
[B_CHECKPOINTER] = {"checkpointer", CheckpointerMain, true},
+ [B_IO_WORKER] = {"io_worker", IoWorkerMain, true},
[B_STARTUP] = {"startup", StartupProcessMain, true},
[B_WAL_RECEIVER] = {"wal_receiver", WalReceiverMain, true},
[B_WAL_SUMMARIZER] = {"wal_summarizer", WalSummarizerMain, true},
diff --git a/src/backend/postmaster/postmaster.c b/src/backend/postmaster/postmaster.c
index 921073a2ca4..fc3901d5347 100644
--- a/src/backend/postmaster/postmaster.c
+++ b/src/backend/postmaster/postmaster.c
@@ -113,6 +113,7 @@
#include "replication/walsender.h"
#include "storage/aio_init.h"
#include "storage/fd.h"
+#include "storage/io_worker.h"
#include "storage/ipc.h"
#include "storage/pmsignal.h"
#include "storage/proc.h"
@@ -321,6 +322,7 @@ typedef enum
* ckpt */
PM_SHUTDOWN_2, /* waiting for archiver and walsenders to
* finish */
+ PM_SHUTDOWN_IO, /* waiting for io workers to exit */
PM_WAIT_DEAD_END, /* waiting for dead_end children to exit */
PM_NO_CHILDREN, /* all important children have exited */
} PMState;
@@ -382,6 +384,10 @@ bool LoadedSSL = false;
static DNSServiceRef bonjour_sdref = NULL;
#endif
+/* State for IO worker management. */
+static int io_worker_count = 0;
+static pid_t io_worker_pids[MAX_IO_WORKERS];
+
/*
* postmaster.c - function prototypes
*/
@@ -420,6 +426,9 @@ static int CountChildren(int target);
static Backend *assign_backendlist_entry(void);
static void LaunchMissingBackgroundProcesses(void);
static void maybe_start_bgworkers(void);
+static bool maybe_reap_io_worker(int pid);
+static void maybe_adjust_io_workers(void);
+static void signal_io_workers(int signal);
static bool CreateOptsFile(int argc, char *argv[], char *fullprogname);
static pid_t StartChildProcess(BackendType type);
static void StartAutovacuumWorker(void);
@@ -1334,6 +1343,11 @@ PostmasterMain(int argc, char *argv[])
*/
AddToDataDirLockFile(LOCK_FILE_LINE_PM_STATUS, PM_STATUS_STARTING);
+ pmState = PM_STARTUP;
+
+ /* Make sure we can perform I/O while starting up. */
+ maybe_adjust_io_workers();
+
/* Start bgwriter and checkpointer so they can help with recovery */
if (CheckpointerPID == 0)
CheckpointerPID = StartChildProcess(B_CHECKPOINTER);
@@ -1346,7 +1360,6 @@ PostmasterMain(int argc, char *argv[])
StartupPID = StartChildProcess(B_STARTUP);
Assert(StartupPID != 0);
StartupStatus = STARTUP_RUNNING;
- pmState = PM_STARTUP;
/* Some workers may be scheduled to start now */
maybe_start_bgworkers();
@@ -1995,6 +2008,7 @@ process_pm_reload_request(void)
signal_child(SysLoggerPID, SIGHUP);
if (SlotSyncWorkerPID != 0)
signal_child(SlotSyncWorkerPID, SIGHUP);
+ signal_io_workers(SIGHUP);
/* Reload authentication config files too */
if (!load_hba())
@@ -2527,6 +2541,22 @@ process_pm_child_exit(void)
}
}
+ /* Was it an IO worker? */
+ if (maybe_reap_io_worker(pid))
+ {
+ if (!EXIT_STATUS_0(exitstatus) && !EXIT_STATUS_1(exitstatus))
+ HandleChildCrash(pid, exitstatus, _("io worker"));
+
+ maybe_adjust_io_workers();
+
+ if (io_worker_count == 0 &&
+ pmState >= PM_SHUTDOWN_IO)
+ {
+ pmState = PM_WAIT_DEAD_END;
+ }
+ continue;
+ }
+
/*
* We don't know anything about this child process. That's highly
* unexpected, as we do track all the child processes that we fork.
@@ -2763,6 +2793,9 @@ HandleChildCrash(int pid, int exitstatus, const char *procname)
if (SlotSyncWorkerPID != 0)
sigquit_child(SlotSyncWorkerPID);
+ /* Take care of io workers too */
+ signal_io_workers(SIGQUIT);
+
/* We do NOT restart the syslogger */
}
@@ -2986,10 +3019,11 @@ PostmasterStateMachine(void)
FatalError = true;
pmState = PM_WAIT_DEAD_END;
- /* Kill the walsenders and archiver too */
+ /* Kill walsenders, archiver and aio workers too */
SignalChildren(SIGQUIT);
if (PgArchPID != 0)
signal_child(PgArchPID, SIGQUIT);
+ signal_io_workers(SIGQUIT);
}
}
}
@@ -2999,16 +3033,26 @@ PostmasterStateMachine(void)
{
/*
* PM_SHUTDOWN_2 state ends when there's no other children than
- * dead_end children left. There shouldn't be any regular backends
- * left by now anyway; what we're really waiting for is walsenders and
- * archiver.
+ * dead_end children and aio workers left. There shouldn't be any
+ * regular backends left by now anyway; what we're really waiting for
+ * is walsenders and archiver.
*/
if (PgArchPID == 0 && CountChildren(BACKEND_TYPE_ALL) == 0)
{
- pmState = PM_WAIT_DEAD_END;
+ pmState = PM_SHUTDOWN_IO;
+ signal_io_workers(SIGUSR2);
}
}
+ if (pmState == PM_SHUTDOWN_IO)
+ {
+ /*
+ * PM_SHUTDOWN_IO state ends when there's only dead_end children left.
+ */
+ if (io_worker_count == 0)
+ pmState = PM_WAIT_DEAD_END;
+ }
+
if (pmState == PM_WAIT_DEAD_END)
{
/* Don't allow any new socket connection events. */
@@ -3016,17 +3060,22 @@ PostmasterStateMachine(void)
/*
* PM_WAIT_DEAD_END state ends when the BackendList is entirely empty
- * (ie, no dead_end children remain), and the archiver is gone too.
+ * (ie, no dead_end children remain), and the archiver and aio workers
+ * are all gone too.
*
- * The reason we wait for those two is to protect them against a new
+ * We need to wait for those because we might have transitioned
+ * directly to PM_WAIT_DEAD_END due to immediate shutdown or fatal
+ * error. Note that they have already been sent appropriate shutdown
+ * signals, either during a normal state transition leading up to
+ * PM_WAIT_DEAD_END, or during FatalError processing.
+ *
+ * The reason we wait for those is to protect them against a new
* postmaster starting conflicting subprocesses; this isn't an
* ironclad protection, but it at least helps in the
- * shutdown-and-immediately-restart scenario. Note that they have
- * already been sent appropriate shutdown signals, either during a
- * normal state transition leading up to PM_WAIT_DEAD_END, or during
- * FatalError processing.
+ * shutdown-and-immediately-restart scenario.
*/
- if (dlist_is_empty(&BackendList) && PgArchPID == 0)
+ if (dlist_is_empty(&BackendList) && io_worker_count == 0
+ && PgArchPID == 0)
{
/* These other guys should be dead already */
Assert(StartupPID == 0);
@@ -3119,10 +3168,14 @@ PostmasterStateMachine(void)
/* re-create shared memory and semaphores */
CreateSharedMemoryAndSemaphores();
+ pmState = PM_STARTUP;
+
+ /* Make sure we can perform I/O while starting up. */
+ maybe_adjust_io_workers();
+
StartupPID = StartChildProcess(B_STARTUP);
Assert(StartupPID != 0);
StartupStatus = STARTUP_RUNNING;
- pmState = PM_STARTUP;
/* crash recovery started, reset SIGKILL flag */
AbortStartTime = 0;
@@ -3374,6 +3427,7 @@ TerminateChildren(int signal)
signal_child(PgArchPID, signal);
if (SlotSyncWorkerPID != 0)
signal_child(SlotSyncWorkerPID, signal);
+ signal_io_workers(signal);
}
/*
@@ -3955,6 +4009,7 @@ bgworker_should_start_now(BgWorkerStartTime start_time)
{
case PM_NO_CHILDREN:
case PM_WAIT_DEAD_END:
+ case PM_SHUTDOWN_IO:
case PM_SHUTDOWN_2:
case PM_SHUTDOWN:
case PM_WAIT_BACKENDS:
@@ -4148,6 +4203,109 @@ maybe_start_bgworkers(void)
}
}
+static bool
+maybe_reap_io_worker(int pid)
+{
+ for (int id = 0; id < MAX_IO_WORKERS; ++id)
+ {
+ if (io_worker_pids[id] == pid)
+ {
+ --io_worker_count;
+ io_worker_pids[id] = 0;
+ return true;
+ }
+ }
+ return false;
+}
+
+static void
+maybe_adjust_io_workers(void)
+{
+ /* ATODO: This will need to check if io_method == worker */
+
+ /*
+ * If we're in final shutting down state, then we're just waiting for all
+ * processes to exit.
+ */
+ if (pmState >= PM_SHUTDOWN_IO)
+ return;
+
+ /* Don't start new workers during an immediate shutdown either. */
+ if (Shutdown >= ImmediateShutdown)
+ return;
+
+ /*
+ * Don't start new workers if we're in the shutdown phase of a crash
+ * restart. But we *do* need to start if we're already starting up again.
+ */
+ if (FatalError && pmState >= PM_STOP_BACKENDS)
+ return;
+
+ /* Not enough running? */
+ while (io_worker_count < io_workers)
+ {
+ int pid;
+ int id;
+
+ /* Find the lowest unused IO worker ID. */
+
+ /*
+ * AFIXME: This logic doesn't work right now, the ids aren't
+ * transported to workers anymore.
+ */
+ for (id = 0; id < MAX_IO_WORKERS; ++id)
+ {
+ if (io_worker_pids[id] == 0)
+ break;
+ }
+ if (id == MAX_IO_WORKERS)
+ elog(ERROR, "could not find a free IO worker ID");
+
+ Assert(pmState < PM_SHUTDOWN_IO);
+
+ /* Try to launch one. */
+ pid = StartChildProcess(B_IO_WORKER);
+ if (pid > 0)
+ {
+ io_worker_pids[id] = pid;
+ ++io_worker_count;
+ }
+ else
+ break; /* XXX try again soon? */
+ }
+
+ /* Too many running? */
+ if (io_worker_count > io_workers)
+ {
+ /* Ask the highest used IO worker ID to exit. */
+ for (int id = MAX_IO_WORKERS - 1; id >= 0; --id)
+ {
+ if (io_worker_pids[id] != 0)
+ {
+ kill(io_worker_pids[id], SIGUSR2);
+ break;
+ }
+ }
+ }
+}
+
+static void
+signal_io_workers(int signal)
+{
+ for (int i = 0; i < MAX_IO_WORKERS; ++i)
+ if (io_worker_pids[i] != 0)
+ signal_child(io_worker_pids[i], signal);
+}
+
+void
+assign_io_workers(int newval, void *extra)
+{
+ io_workers = newval;
+ if (!IsUnderPostmaster && pmState > PM_INIT)
+ maybe_adjust_io_workers();
+}
+
+
/*
* When a backend asks to be notified about worker state changes, we
* set a flag in its backend entry. The background worker machinery needs
diff --git a/src/backend/storage/aio/Makefile b/src/backend/storage/aio/Makefile
index eaeaeeee8e3..824682e7354 100644
--- a/src/backend/storage/aio/Makefile
+++ b/src/backend/storage/aio/Makefile
@@ -11,6 +11,7 @@ include $(top_builddir)/src/Makefile.global
OBJS = \
aio.o \
aio_init.o \
+ method_worker.o \
read_stream.o
include $(top_srcdir)/src/backend/common.mk
diff --git a/src/backend/storage/aio/meson.build b/src/backend/storage/aio/meson.build
index 8d20759ebf8..e13728b73da 100644
--- a/src/backend/storage/aio/meson.build
+++ b/src/backend/storage/aio/meson.build
@@ -3,5 +3,6 @@
backend_sources += files(
'aio.c',
'aio_init.c',
+ 'method_worker.c',
'read_stream.c',
)
diff --git a/src/backend/storage/aio/method_worker.c b/src/backend/storage/aio/method_worker.c
new file mode 100644
index 00000000000..5df2eea4a03
--- /dev/null
+++ b/src/backend/storage/aio/method_worker.c
@@ -0,0 +1,84 @@
+/*-------------------------------------------------------------------------
+ *
+ * method_worker.c
+ * AIO implementation using workers
+ *
+ * Portions Copyright (c) 1996-2021, PostgreSQL Global Development Group
+ * Portions Copyright (c) 1994, Regents of the University of California
+ *
+ * IDENTIFICATION
+ * src/backend/storage/aio/method_worker.c
+ *
+ *-------------------------------------------------------------------------
+ */
+
+#include "postgres.h"
+
+#include "libpq/pqsignal.h"
+#include "miscadmin.h"
+#include "postmaster/interrupt.h"
+#include "storage/io_worker.h"
+#include "storage/ipc.h"
+#include "storage/latch.h"
+#include "storage/proc.h"
+#include "tcop/tcopprot.h"
+#include "utils/wait_event.h"
+
+
+int io_workers = 3;
+
+
+void
+IoWorkerMain(char *startup_data, size_t startup_data_len)
+{
+ sigjmp_buf local_sigjmp_buf;
+
+ MyBackendType = B_IO_WORKER;
+
+ /* TODO review all signals */
+ pqsignal(SIGHUP, SignalHandlerForConfigReload);
+ pqsignal(SIGINT, die); /* to allow manually triggering worker restart */
+
+ /*
+ * Ignore SIGTERM, will get explicit shutdown via SIGUSR2 later in the
+ * shutdown sequence, similar to checkpointer.
+ */
+ pqsignal(SIGTERM, SIG_IGN);
+ /* SIGQUIT handler was already set up by InitPostmasterChild */
+ pqsignal(SIGALRM, SIG_IGN);
+ pqsignal(SIGPIPE, SIG_IGN);
+ pqsignal(SIGUSR1, procsignal_sigusr1_handler);
+ pqsignal(SIGUSR2, SignalHandlerForShutdownRequest);
+ sigprocmask(SIG_SETMASK, &UnBlockSig, NULL);
+
+ /* see PostgresMain() */
+ if (sigsetjmp(local_sigjmp_buf, 1) != 0)
+ {
+ error_context_stack = NULL;
+ HOLD_INTERRUPTS();
+
+ /*
+ * We normally shouldn't get errors here. Need to do just enough error
+ * recovery so that we can mark the IO as failed and then exit.
+ */
+ LWLockReleaseAll();
+
+ /* TODO: recover from IO errors */
+
+ EmitErrorReport();
+ proc_exit(1);
+ }
+
+ /* We can now handle ereport(ERROR) */
+ PG_exception_stack = &local_sigjmp_buf;
+
+ while (!ShutdownRequestPending)
+ {
+ WaitLatch(MyLatch, WL_LATCH_SET | WL_EXIT_ON_PM_DEATH, -1,
+ WAIT_EVENT_IO_WORKER_MAIN);
+ ResetLatch(MyLatch);
+ CHECK_FOR_INTERRUPTS();
+ }
+
+ proc_exit(0);
+}
diff --git a/src/backend/tcop/postgres.c b/src/backend/tcop/postgres.c
index 4dc46b17b41..d42546db195 100644
--- a/src/backend/tcop/postgres.c
+++ b/src/backend/tcop/postgres.c
@@ -3294,6 +3294,8 @@ ProcessInterrupts(void)
(errcode(ERRCODE_ADMIN_SHUTDOWN),
errmsg("terminating background worker \"%s\" due to administrator command",
MyBgworkerEntry->bgw_type)));
+ else if (AmIoWorkerProcess())
+ proc_exit(0);
else
ereport(FATAL,
(errcode(ERRCODE_ADMIN_SHUTDOWN),
diff --git a/src/backend/utils/activity/pgstat_io.c b/src/backend/utils/activity/pgstat_io.c
index 8af55989eed..a750caa9b2a 100644
--- a/src/backend/utils/activity/pgstat_io.c
+++ b/src/backend/utils/activity/pgstat_io.c
@@ -335,6 +335,7 @@ pgstat_tracks_io_bktype(BackendType bktype)
{
case B_INVALID:
case B_ARCHIVER:
+ case B_IO_WORKER:
case B_LOGGER:
case B_WAL_RECEIVER:
case B_WAL_WRITER:
diff --git a/src/backend/utils/activity/wait_event_names.txt b/src/backend/utils/activity/wait_event_names.txt
index 8efb4044d6f..47a2c4d126b 100644
--- a/src/backend/utils/activity/wait_event_names.txt
+++ b/src/backend/utils/activity/wait_event_names.txt
@@ -56,6 +56,7 @@ AUTOVACUUM_MAIN "Waiting in main loop of autovacuum launcher process."
BGWRITER_HIBERNATE "Waiting in background writer process, hibernating."
BGWRITER_MAIN "Waiting in main loop of background writer process."
CHECKPOINTER_MAIN "Waiting in main loop of checkpointer process."
+IO_WORKER_MAIN "Waiting in main loop of IO Worker process."
LOGICAL_APPLY_MAIN "Waiting in main loop of logical replication apply process."
LOGICAL_LAUNCHER_MAIN "Waiting in main loop of logical replication launcher process."
LOGICAL_PARALLEL_APPLY_MAIN "Waiting in main loop of logical replication parallel apply process."
diff --git a/src/backend/utils/init/miscinit.c b/src/backend/utils/init/miscinit.c
index b8fa2e64ffe..bedeed588d3 100644
--- a/src/backend/utils/init/miscinit.c
+++ b/src/backend/utils/init/miscinit.c
@@ -293,6 +293,9 @@ GetBackendTypeDesc(BackendType backendType)
case B_CHECKPOINTER:
backendDesc = "checkpointer";
break;
+ case B_IO_WORKER:
+ backendDesc = "io worker";
+ break;
case B_LOGGER:
backendDesc = "logger";
break;
diff --git a/src/backend/utils/misc/guc_tables.c b/src/backend/utils/misc/guc_tables.c
index 4961a5f4b16..5670f40478a 100644
--- a/src/backend/utils/misc/guc_tables.c
+++ b/src/backend/utils/misc/guc_tables.c
@@ -74,6 +74,7 @@
#include "storage/aio.h"
#include "storage/bufmgr.h"
#include "storage/bufpage.h"
+#include "storage/io_worker.h"
#include "storage/large_object.h"
#include "storage/pg_shmem.h"
#include "storage/predicate.h"
@@ -3201,6 +3202,18 @@ struct config_int ConfigureNamesInt[] =
NULL, NULL, NULL
},
+ {
+ {"io_workers",
+ PGC_SIGHUP,
+ RESOURCES_ASYNCHRONOUS,
+ gettext_noop("Number of IO worker processes, for io_method=worker."),
+ NULL,
+ },
+ &io_workers,
+ 3, 1, MAX_IO_WORKERS,
+ NULL, assign_io_workers, NULL
+ },
+
{
{"backend_flush_after", PGC_USERSET, RESOURCES_ASYNCHRONOUS,
gettext_noop("Number of pages after which previously performed writes are flushed to disk."),
diff --git a/src/backend/utils/misc/postgresql.conf.sample b/src/backend/utils/misc/postgresql.conf.sample
index e904c3fea30..90430381efa 100644
--- a/src/backend/utils/misc/postgresql.conf.sample
+++ b/src/backend/utils/misc/postgresql.conf.sample
@@ -839,7 +839,8 @@
# WIP AIO GUC docs
#------------------------------------------------------------------------------
-#io_method = worker
+#io_method = worker # (change requires restart)
+#io_workers = 3 # 1-32;
#------------------------------------------------------------------------------
--
2.45.2.827.g557ae147e6
v2.0-0009-aio-Basic-AIO-implementation.patch (text/x-diff, 89.4K) download
From 0de554082f3ff6468ff352000774245b337d6d64 Mon Sep 17 00:00:00 2001
From: Andres Freund <andres@anarazel.de>
Date: Sat, 31 Aug 2024 21:23:37 -0400
Subject: [PATCH v2.0 09/17] aio: Basic AIO implementation
At this point nothing can use AIO - this commit does not include any
implementation of aio subjects / callbacks. That will come in later commits.
Todo:
- implement "synchronous" AIO method
- split worker, io_uring methods out into separate commits
- lots of cleanup
---
src/include/storage/aio.h | 308 ++++++
src/include/storage/aio_internal.h | 274 +++++
src/include/storage/aio_ref.h | 24 +
src/include/storage/lwlock.h | 1 +
src/include/storage/lwlocklist.h | 1 +
src/include/utils/resowner.h | 7 +
src/backend/access/transam/xact.c | 9 +
src/backend/postmaster/postmaster.c | 3 +-
src/backend/storage/aio/Makefile | 3 +
src/backend/storage/aio/aio.c | 963 +++++++++++++++++-
src/backend/storage/aio/aio_init.c | 318 ++++++
src/backend/storage/aio/aio_io.c | 111 ++
src/backend/storage/aio/aio_subject.c | 170 ++++
src/backend/storage/aio/meson.build | 3 +
src/backend/storage/aio/method_io_uring.c | 393 +++++++
src/backend/storage/aio/method_worker.c | 413 +++++++-
src/backend/storage/lmgr/lwlock.c | 1 +
.../utils/activity/wait_event_names.txt | 4 +
src/backend/utils/misc/guc_tables.c | 25 +
src/backend/utils/misc/postgresql.conf.sample | 6 +
src/backend/utils/resowner/resowner.c | 51 +
src/tools/pgindent/typedefs.list | 23 +
22 files changed, 3104 insertions(+), 7 deletions(-)
create mode 100644 src/include/storage/aio_internal.h
create mode 100644 src/include/storage/aio_ref.h
create mode 100644 src/backend/storage/aio/aio_io.c
create mode 100644 src/backend/storage/aio/aio_subject.c
create mode 100644 src/backend/storage/aio/method_io_uring.c
diff --git a/src/include/storage/aio.h b/src/include/storage/aio.h
index 98fafcf9bc4..65052462b02 100644
--- a/src/include/storage/aio.h
+++ b/src/include/storage/aio.h
@@ -15,9 +15,315 @@
#define AIO_H
+#include "storage/aio_ref.h"
+#include "storage/procnumber.h"
#include "utils/guc_tables.h"
+typedef struct PgAioHandle PgAioHandle;
+
+typedef enum PgAioOp
+{
+ /* intentionally the zero value, to help catch zeroed memory etc */
+ PGAIO_OP_INVALID = 0,
+
+ PGAIO_OP_READ,
+ PGAIO_OP_WRITE,
+
+ PGAIO_OP_FSYNC,
+
+ PGAIO_OP_FLUSH_RANGE,
+
+ PGAIO_OP_NOP,
+
+ /**
+ * Eventually we'll additionally want at least:
+ * - send
+ * - recv
+ * - accept
+ **/
+} PgAioOp;
+
+#define PGAIO_OP_COUNT (PGAIO_OP_NOP + 1)
+
+
+/*
+ * On what is IO being performed.
+ *
+ * PgAioSharedCallback specific behaviour should be implemented in
+ * aio_subject.c.
+ */
+typedef enum PgAioSubjectID
+{
+ /* intentionally the zero value, to help catch zeroed memory etc */
+ ASI_INVALID = 0,
+} PgAioSubjectID;
+
+#define ASI_COUNT (ASI_INVALID + 1)
+
+/*
+ * Flags for an IO that can be set with pgaio_io_set_flag().
+ */
+typedef enum PgAioHandleFlags
+{
+ AHF_REFERENCES_LOCAL = 1 << 0,
+} PgAioHandleFlags;
+
+
+/*
+ * IDs for callbacks that can be registered on an IO.
+ *
+ * Callbacks are identified by an ID rather than a function pointer. There are
+ * two main reasons:
+
+ * 1) Memory within PgAioHandle is precious, due to the number of PgAioHandle
+ * structs in pre-allocated shared memory.
+
+ * 2) Due to EXEC_BACKEND function pointers are not necessarily stable between
+ * different backends, therefore function pointers cannot directly be in
+ * shared memory.
+ *
+ * Without 2), we could fairly easily allow to add new callbacks, by filling a
+ * ID->pointer mapping table on demand. In the presence of 2 that's still
+ * doable, but harder, because every process has to re-register the pointers
+ * so that a local ID->"backend local pointer" mapping can be maintained.
+ */
+typedef enum PgAioHandleSharedCallbackID
+{
+ ASC_PLACEHOLDER /* empty enums are invalid */ ,
+} PgAioHandleSharedCallbackID;
+
+
+/*
+ * Data necessary for basic IO types (PgAioOp).
+ *
+ * NB: Note that the FDs in here may *not* be relied upon for re-issuing
+ * requests (e.g. for partial reads/writes) - the FD might be from another
+ * process, or closed since. That's not a problem for IOs waiting to be issued
+ * only because the queue is flushed when closing an FD.
+ */
+typedef union
+{
+ struct
+ {
+ int fd;
+ uint16 iov_length;
+ uint64 offset;
+ } read;
+
+ struct
+ {
+ int fd;
+ uint16 iov_length;
+ uint64 offset;
+ } write;
+
+ struct
+ {
+ int fd;
+ bool datasync;
+ } fsync;
+
+ struct
+ {
+ int fd;
+ uint32 nbytes;
+ uint64 offset;
+ } flush_range;
+} PgAioOpData;
+
+
+/* XXX: Perhaps it's worth moving this to a dedicated file? */
+#include "storage/block.h"
+#include "storage/relfilelocator.h"
+
+typedef union PgAioSubjectData
+{
+ /* just as an example placeholder for later */
+ struct
+ {
+ uint32 queue_id;
+ } wal;
+} PgAioSubjectData;
+
+
+
+typedef enum PgAioResultStatus
+{
+ ARS_UNKNOWN,
+ ARS_OK,
+ ARS_PARTIAL,
+ ARS_ERROR,
+} PgAioResultStatus;
+
+typedef struct PgAioResult
+{
+ PgAioHandleSharedCallbackID id:8;
+ PgAioResultStatus status:2;
+ uint32 error_data:22;
+ int32 result;
+} PgAioResult;
+
+typedef struct PgAioReturn
+{
+ PgAioResult result;
+ PgAioSubjectData subject_data;
+} PgAioReturn;
+
+
+typedef struct PgAioSubjectInfo
+{
+ void (*reopen) (PgAioHandle *ioh);
+
+#ifdef NOT_YET
+ char *(*describe_identity) (PgAioHandle *ioh);
+#endif
+
+ const char *name;
+} PgAioSubjectInfo;
+
+
+typedef PgAioResult (*PgAioHandleSharedCallbackComplete) (PgAioHandle *ioh, PgAioResult prior_result);
+typedef void (*PgAioHandleSharedCallbackPrepare) (PgAioHandle *ioh);
+typedef void (*PgAioHandleSharedCallbackError) (PgAioResult result, const PgAioSubjectData *subject_data, int elevel);
+
+typedef struct PgAioHandleSharedCallbacks
+{
+ PgAioHandleSharedCallbackPrepare prepare;
+ PgAioHandleSharedCallbackComplete complete;
+ PgAioHandleSharedCallbackError error;
+} PgAioHandleSharedCallbacks;
+
+
+
+typedef struct PgAioBounceBuffer PgAioBounceBuffer;
+
+
+/*
+ * How many callbacks can be registered for one IO handle. Currently we only
+ * need two, but it's not hard to imagine needing a few more.
+ */
+#define AIO_MAX_SHARED_CALLBACKS 4
+
+
+
+/* AIO API */
+
+
+/* --------------------------------------------------------------------------------
+ * IO Handles
+ * --------------------------------------------------------------------------------
+ */
+
+struct ResourceOwnerData;
+extern PgAioHandle *pgaio_io_get(struct ResourceOwnerData *resowner, PgAioReturn *ret);
+extern PgAioHandle *pgaio_io_get_nb(struct ResourceOwnerData *resowner, PgAioReturn *ret);
+
+extern void pgaio_io_release(PgAioHandle *ioh);
+extern void pgaio_io_release_resowner(dlist_node *ioh_node, bool on_error);
+
+extern void pgaio_io_get_ref(PgAioHandle *ioh, PgAioHandleRef *ior);
+
+extern void pgaio_io_set_subject(PgAioHandle *ioh, PgAioSubjectID subjid);
+extern void pgaio_io_set_flag(PgAioHandle *ioh, PgAioHandleFlags flag);
+
+extern void pgaio_io_add_shared_cb(PgAioHandle *ioh, PgAioHandleSharedCallbackID cbid);
+
+extern void pgaio_io_set_io_data_32(PgAioHandle *ioh, uint32 *data, uint8 len);
+extern void pgaio_io_set_io_data_64(PgAioHandle *ioh, uint64 *data, uint8 len);
+extern uint64 *pgaio_io_get_io_data(PgAioHandle *ioh, uint8 *len);
+
+extern void pgaio_io_prepare(PgAioHandle *ioh, PgAioOp op);
+
+extern int pgaio_io_get_id(PgAioHandle *ioh);
+struct iovec;
+extern int pgaio_io_get_iovec(PgAioHandle *ioh, struct iovec **iov);
+extern bool pgaio_io_has_subject(PgAioHandle *ioh);
+
+extern PgAioSubjectData *pgaio_io_get_subject_data(PgAioHandle *ioh);
+extern PgAioOpData *pgaio_io_get_op_data(PgAioHandle *ioh);
+extern ProcNumber pgaio_io_get_owner(PgAioHandle *ioh);
+
+
+
+/* --------------------------------------------------------------------------------
+ * IO References
+ * --------------------------------------------------------------------------------
+ */
+
+extern void pgaio_io_ref_clear(PgAioHandleRef *ior);
+extern bool pgaio_io_ref_valid(PgAioHandleRef *ior);
+extern int pgaio_io_ref_get_id(PgAioHandleRef *ior);
+
+
+extern void pgaio_io_ref_wait(PgAioHandleRef *ior);
+extern bool pgaio_io_ref_check_done(PgAioHandleRef *ior);
+
+
+
+/* --------------------------------------------------------------------------------
+ * IO Result
+ * --------------------------------------------------------------------------------
+ */
+
+extern void pgaio_result_log(PgAioResult result, const PgAioSubjectData *subject_data,
+ int elevel);
+
+
+
+/* --------------------------------------------------------------------------------
+ * Bounce Buffers
+ * --------------------------------------------------------------------------------
+ */
+
+extern PgAioBounceBuffer *pgaio_bounce_buffer_get(void);
+extern void pgaio_io_assoc_bounce_buffer(PgAioHandle *ioh, PgAioBounceBuffer *bb);
+extern uint32 pgaio_bounce_buffer_id(PgAioBounceBuffer *bb);
+extern void pgaio_bounce_buffer_release(PgAioBounceBuffer *bb);
+extern char *pgaio_bounce_buffer_buffer(PgAioBounceBuffer *bb);
+extern void pgaio_bounce_buffer_release_resowner(dlist_node *bb_node, bool on_error);
+
+
+
+/* --------------------------------------------------------------------------------
+ * Actions on multiple IOs.
+ * --------------------------------------------------------------------------------
+ */
+
+extern void pgaio_submit_staged(void);
+extern bool pgaio_have_staged(void);
+
+
+
+/* --------------------------------------------------------------------------------
+ * Low level IO preparation routines
+ *
+ * These will often be called by code lowest level of initiating an
+ * IO. E.g. bufmgr.c may initiate IO for a buffer, but pgaio_io_prep_readv()
+ * will be called from within fd.c.
+ *
+ * Implemented in aio_io.c
+ * --------------------------------------------------------------------------------
+ */
+
+extern void pgaio_io_prep_readv(PgAioHandle *ioh,
+ int fd, int iovcnt, uint64 offset);
+
+extern void pgaio_io_prep_writev(PgAioHandle *ioh,
+ int fd, int iovcnt, uint64 offset);
+
+
+
+/* --------------------------------------------------------------------------------
+ * Other
+ * --------------------------------------------------------------------------------
+ */
+
+extern void pgaio_closing_fd(int fd);
+extern void pgaio_at_xact_end(bool is_subxact, bool is_commit);
+extern void pgaio_at_error(void);
+
+
/* GUC related */
extern void assign_io_method(int newval, void *extra);
@@ -37,6 +343,8 @@ typedef enum IoMethod
/* GUCs */
extern const struct config_enum_entry io_method_options[];
extern int io_method;
+extern int io_max_concurrency;
+extern int io_bounce_buffers;
#endif /* AIO_H */
diff --git a/src/include/storage/aio_internal.h b/src/include/storage/aio_internal.h
new file mode 100644
index 00000000000..67d994cc0b1
--- /dev/null
+++ b/src/include/storage/aio_internal.h
@@ -0,0 +1,274 @@
+/*-------------------------------------------------------------------------
+ *
+ * aio_internal.h
+ * aio_internal
+ *
+ *
+ * Portions Copyright (c) 1996-2024, PostgreSQL Global Development Group
+ * Portions Copyright (c) 1994, Regents of the University of California
+ *
+ * src/include/storage/aio_internal.h
+ *
+ *-------------------------------------------------------------------------
+ */
+#ifndef AIO_INTERNAL_H
+#define AIO_INTERNAL_H
+
+
+#include "lib/ilist.h"
+#include "port/pg_iovec.h"
+#include "storage/aio.h"
+#include "storage/condition_variable.h"
+
+
+#define PGAIO_VERBOSE
+
+
+/* AFIXME */
+#define PGAIO_SUBMIT_BATCH_SIZE 32
+
+
+
+typedef enum PgAioHandleState
+{
+ /* not in use */
+ AHS_IDLE = 0,
+
+ /* returned by pgaio_io_get() */
+ AHS_HANDED_OUT,
+
+ /* pgaio_io_start_*() has been called, but IO hasn't been submitted yet */
+ AHS_DEFINED,
+
+ /* subjects prepare() callback has been called */
+ AHS_PREPARED,
+
+ /* IO is being executed */
+ AHS_IN_FLIGHT,
+
+ /* IO finished, but result has not yet been processed */
+ AHS_REAPED,
+
+ /* IO completed, shared completion has been called */
+ AHS_COMPLETED_SHARED,
+
+ /* IO completed, local completion has been called */
+ AHS_COMPLETED_LOCAL,
+} PgAioHandleState;
+
+
+struct ResourceOwnerData;
+
+/* typedef is in public header */
+struct PgAioHandle
+{
+ PgAioHandleState state:8;
+
+ /* what are we operating on */
+ PgAioSubjectID subject:8;
+
+ /* which operation */
+ PgAioOp op:8;
+
+ /* bitfield of PgAioHandleFlags */
+ uint8 flags;
+
+ uint8 num_shared_callbacks;
+
+ /* using the proper type here would use more space */
+ uint8 shared_callbacks[AIO_MAX_SHARED_CALLBACKS];
+
+ uint8 iovec_data_len;
+
+ /* XXX: could be optimized out with some pointer math */
+ int32 owner_procno;
+
+ /* FIXME: remove in favor of distilled_result */
+ /* raw result of the IO operation */
+ int32 result;
+
+ /* index into PgAioCtl->iovecs */
+ uint32 iovec_off;
+
+ /*
+ * List of bounce_buffers owned by IO. It would suffice to use an index
+ * based linked list here.
+ */
+ slist_head bounce_buffers;
+
+ /**
+ * In which list the handle is registered, depends on the state:
+ * - IDLE, in per-backend list
+ * - HANDED_OUT - not in a list
+ * - DEFINED - in per-backend staged list
+ * - PREPARED - in per-backend staged list
+ * - IN_FLIGHT - not in any list
+ * - REAPED - in per-reap context list
+ * - COMPLETED_SHARED - not in any list
+ * - COMPLETED_LOCAL - not in any list
+ *
+ * XXX: It probably make sense to optimize this out to save on per-io
+ * memory at the cost of per-backend memory.
+ **/
+ dlist_node node;
+
+ struct ResourceOwnerData *resowner;
+ dlist_node resowner_node;
+
+ /* incremented every time the IO handle is reused */
+ uint64 generation;
+
+ ConditionVariable cv;
+
+ /* result of shared callback, passed to issuer callback */
+ PgAioResult distilled_result;
+
+ PgAioReturn *report_return;
+
+ PgAioOpData op_data;
+
+ /*
+ * Data necessary for shared completions. Needs to be sufficient to allow
+ * another backend to retry an IO.
+ */
+ PgAioSubjectData scb_data;
+};
+
+
+struct PgAioBounceBuffer
+{
+ slist_node node;
+ struct ResourceOwnerData *resowner;
+ dlist_node resowner_node;
+ char *buffer;
+};
+
+
+typedef struct PgAioPerBackend
+{
+ /* index into PgAioCtl->io_handles */
+ uint32 io_handle_off;
+
+ /* index into PgAioCtl->bounce_buffers */
+ uint32 bounce_buffers_off;
+
+ /* IO Handles that currently are not used */
+ dclist_head idle_ios;
+
+ /*
+ * Only one IO may be returned by pgaio_io_get()/pgaio_io_get() without
+ * having been either defined (by actually associating it with IO) or by
+ * released (with pgaio_io_release()). This restriction is necessary to
+ * guarantee that we always can acquire an IO. ->handed_out_io is used to
+ * enforce that rule.
+ */
+ PgAioHandle *handed_out_io;
+
+ /*
+ * IOs that are defined, but not yet submitted.
+ */
+ dclist_head staged_ios;
+
+ /* Bounce Buffers that currently are not used */
+ slist_head idle_bbs;
+
+ /* see handed_out_io */
+ PgAioBounceBuffer *handed_out_bb;
+} PgAioPerBackend;
+
+
+typedef struct PgAioCtl
+{
+ int backend_state_count;
+ PgAioPerBackend *backend_state;
+
+ /*
+ * Array of iovec structs. Each iovec is owned by a specific backend. The
+ * allocation is in PgAioCtl to allow the maximum number of iovecs for
+ * individual IOs to be configurable with PGC_POSTMASTER GUC.
+ */
+ uint64 iovec_count;
+ struct iovec *iovecs;
+
+ /*
+ * For, e.g., an IO covering multiple buffers in shared / temp buffers, we
+ * need to get Buffer IDs during completion to be able to change the
+ * BufferDesc state accordingly. This space can be used to store e.g.
+ * Buffer IDs. Note that the actual iovec might be shorter than this,
+ * because we combine neighboring pages into one larger iovec entry.
+ */
+ uint64 *iovecs_data;
+
+ /*
+ * To perform AIO on buffers that are not located in shared memory (either
+ * because they are not in shared memory or because we need to operate on
+ * a copy, as e.g. the case for writes when checksums are in use)
+ */
+ uint64 bounce_buffers_count;
+ PgAioBounceBuffer *bounce_buffers;
+ char *bounce_buffers_data;
+
+ uint64 io_handle_count;
+ PgAioHandle *io_handles;
+} PgAioCtl;
+
+
+
+/*
+ * The set of callbacks that each IO method must implement.
+ */
+typedef struct IoMethodOps
+{
+ /* initialization */
+ size_t (*shmem_size) (void);
+ void (*shmem_init) (bool first_time);
+
+ void (*postmaster_init) (void);
+ void (*postmaster_child_init_local) (void);
+ void (*postmaster_child_init) (void);
+
+ /* teardown */
+ void (*postmaster_before_child_exit) (void);
+
+ /* handling of IOs */
+ int (*submit) (void);
+
+ void (*wait_one) (PgAioHandle *ioh,
+ uint64 ref_generation);
+
+ /* properties */
+ bool can_scatter_gather_direct;
+ bool can_scatter_gather_buffered;
+} IoMethodOps;
+
+
+extern bool pgaio_io_was_recycled(PgAioHandle *ioh, uint64 ref_generation, PgAioHandleState *state);
+
+extern void pgaio_io_prepare_subject(PgAioHandle *ioh);
+extern void pgaio_io_process_completion_subject(PgAioHandle *ioh);
+extern void pgaio_io_process_completion(PgAioHandle *ioh, int result);
+extern void pgaio_io_prepare_submit(PgAioHandle *ioh);
+
+extern bool pgaio_io_needs_synchronously(PgAioHandle *ioh);
+extern void pgaio_io_perform_synchronously(PgAioHandle *ioh);
+
+extern void pgaio_io_reopen(PgAioHandle *ioh);
+
+extern const char *pgaio_io_get_subject_name(PgAioHandle *ioh);
+extern const char *pgaio_io_get_op_name(PgAioHandle *ioh);
+extern const char *pgaio_io_get_state_name(PgAioHandle *ioh);
+
+
+/* Declarations for the tables of function pointers exposed by each IO method. */
+extern const IoMethodOps pgaio_worker_ops;
+#ifdef USE_LIBURING
+extern const IoMethodOps pgaio_uring_ops;
+#endif
+
+extern const IoMethodOps *pgaio_impl;
+extern PgAioCtl *aio_ctl;
+extern PgAioPerBackend *my_aio;
+
+
+
+#endif /* AIO_INTERNAL_H */
diff --git a/src/include/storage/aio_ref.h b/src/include/storage/aio_ref.h
new file mode 100644
index 00000000000..ad7e9ad34f3
--- /dev/null
+++ b/src/include/storage/aio_ref.h
@@ -0,0 +1,24 @@
+/*-------------------------------------------------------------------------
+ *
+ * aio_ref.h Definition of PgAioHandleRef, which sometimes needs to be used in
+ * headers.
+ *
+ *
+ * Portions Copyright (c) 1996-2024, PostgreSQL Global Development Group
+ * Portions Copyright (c) 1994, Regents of the University of California
+ *
+ * src/include/storage/aio_ref.h
+ *
+ *-------------------------------------------------------------------------
+ */
+#ifndef AIO_REF_H
+#define AIO_REF_H
+
+typedef struct PgAioHandleRef
+{
+ uint32 aio_index;
+ uint32 generation_upper;
+ uint32 generation_lower;
+} PgAioHandleRef;
+
+#endif /* AIO_REF_H */
diff --git a/src/include/storage/lwlock.h b/src/include/storage/lwlock.h
index 00e8022fbad..f4e6abce327 100644
--- a/src/include/storage/lwlock.h
+++ b/src/include/storage/lwlock.h
@@ -217,6 +217,7 @@ typedef enum BuiltinTrancheIds
LWTRANCHE_SUBTRANS_SLRU,
LWTRANCHE_XACT_SLRU,
LWTRANCHE_PARALLEL_VACUUM_DSA,
+ LWTRANCHE_AIO_URING_COMPLETION,
LWTRANCHE_FIRST_USER_DEFINED,
} BuiltinTrancheIds;
diff --git a/src/include/storage/lwlocklist.h b/src/include/storage/lwlocklist.h
index 88dc79b2bd6..7aaccf69d1e 100644
--- a/src/include/storage/lwlocklist.h
+++ b/src/include/storage/lwlocklist.h
@@ -84,3 +84,4 @@ PG_LWLOCK(50, DSMRegistry)
PG_LWLOCK(51, InjectionPoint)
PG_LWLOCK(52, SerialControl)
PG_LWLOCK(53, WaitLSN)
+PG_LWLOCK(54, AioWorkerSubmissionQueue)
diff --git a/src/include/utils/resowner.h b/src/include/utils/resowner.h
index 4e534bc3e70..0cdd0c13ffb 100644
--- a/src/include/utils/resowner.h
+++ b/src/include/utils/resowner.h
@@ -164,4 +164,11 @@ struct LOCALLOCK;
extern void ResourceOwnerRememberLock(ResourceOwner owner, struct LOCALLOCK *locallock);
extern void ResourceOwnerForgetLock(ResourceOwner owner, struct LOCALLOCK *locallock);
+/* special support for AIO */
+struct dlist_node;
+extern void ResourceOwnerRememberAioHandle(ResourceOwner owner, struct dlist_node *ioh_node);
+extern void ResourceOwnerForgetAioHandle(ResourceOwner owner, struct dlist_node *ioh_node);
+extern void ResourceOwnerRememberAioBounceBuffer(ResourceOwner owner, struct dlist_node *bb_node);
+extern void ResourceOwnerForgetAioBounceBuffer(ResourceOwner owner, struct dlist_node *bb_node);
+
#endif /* RESOWNER_H */
diff --git a/src/backend/access/transam/xact.c b/src/backend/access/transam/xact.c
index 0fe1630fca8..cb4ee5dfd1f 100644
--- a/src/backend/access/transam/xact.c
+++ b/src/backend/access/transam/xact.c
@@ -52,6 +52,7 @@
#include "replication/origin.h"
#include "replication/snapbuild.h"
#include "replication/syncrep.h"
+#include "storage/aio.h"
#include "storage/condition_variable.h"
#include "storage/fd.h"
#include "storage/lmgr.h"
@@ -2462,6 +2463,8 @@ CommitTransaction(void)
AtEOXact_LogicalRepWorkers(true);
pgstat_report_xact_timestamp(0);
+ pgaio_at_xact_end( /* is_subxact = */ false, /* is_commit = */ true);
+
ResourceOwnerDelete(TopTransactionResourceOwner);
s->curTransactionOwner = NULL;
CurTransactionResourceOwner = NULL;
@@ -2976,6 +2979,8 @@ AbortTransaction(void)
pgstat_report_xact_timestamp(0);
}
+ pgaio_at_xact_end( /* is_subxact = */ false, /* is_commit = */ false);
+
/*
* State remains TRANS_ABORT until CleanupTransaction().
*/
@@ -5185,6 +5190,8 @@ CommitSubTransaction(void)
AtEOSubXact_PgStat(true, s->nestingLevel);
AtSubCommit_Snapshot(s->nestingLevel);
+ pgaio_at_xact_end( /* is_subxact = */ true, /* is_commit = */ true);
+
/*
* We need to restore the upper transaction's read-only state, in case the
* upper is read-write while the child is read-only; GUC will incorrectly
@@ -5350,6 +5357,8 @@ AbortSubTransaction(void)
AtSubAbort_Snapshot(s->nestingLevel);
}
+ pgaio_at_xact_end( /* is_subxact = */ true, /* is_commit = */ false);
+
/*
* Restore the upper transaction's read-only state, too. This should be
* redundant with GUC's cleanup but we may as well do it for consistency
diff --git a/src/backend/postmaster/postmaster.c b/src/backend/postmaster/postmaster.c
index fc3901d5347..71930094309 100644
--- a/src/backend/postmaster/postmaster.c
+++ b/src/backend/postmaster/postmaster.c
@@ -4221,7 +4221,8 @@ maybe_reap_io_worker(int pid)
static void
maybe_adjust_io_workers(void)
{
- /* ATODO: This will need to check if io_method == worker */
+ if (!pgaio_workers_enabled())
+ return;
/*
* If we're in final shutting down state, then we're just waiting for all
diff --git a/src/backend/storage/aio/Makefile b/src/backend/storage/aio/Makefile
index 824682e7354..2a5e72a8024 100644
--- a/src/backend/storage/aio/Makefile
+++ b/src/backend/storage/aio/Makefile
@@ -10,8 +10,11 @@ include $(top_builddir)/src/Makefile.global
OBJS = \
aio.o \
+ aio_io.o \
aio_init.o \
+ aio_subject.o \
method_worker.o \
+ method_io_uring.o \
read_stream.o
include $(top_srcdir)/src/backend/common.mk
diff --git a/src/backend/storage/aio/aio.c b/src/backend/storage/aio/aio.c
index 67f6b52de91..d6f9f658b97 100644
--- a/src/backend/storage/aio/aio.c
+++ b/src/backend/storage/aio/aio.c
@@ -14,7 +14,23 @@
#include "postgres.h"
+#include "miscadmin.h"
+#include "port/atomics.h"
#include "storage/aio.h"
+#include "storage/aio_internal.h"
+#include "storage/bufmgr.h"
+#include "utils/resowner.h"
+#include "utils/wait_event_types.h"
+
+
+
+static void pgaio_io_reclaim(PgAioHandle *ioh);
+static void pgaio_io_resowner_register(PgAioHandle *ioh);
+static void pgaio_io_wait_for_free(void);
+static PgAioHandle *pgaio_io_from_ref(PgAioHandleRef *ior, uint64 *ref_generation);
+
+static void pgaio_bounce_buffer_wait_for_free(void);
+
/* Options for io_method. */
@@ -26,10 +42,955 @@ const struct config_enum_entry io_method_options[] = {
{NULL, 0, false}
};
-int io_method = IOMETHOD_WORKER;
+int io_method = DEFAULT_IO_METHOD;
+int io_max_concurrency = -1;
+int io_bounce_buffers = -1;
+
+
+/* global control for AIO */
+PgAioCtl *aio_ctl;
+
+/* current backend's per-backend state */
+PgAioPerBackend *my_aio;
+
+
+static const IoMethodOps *pgaio_ops_table[] = {
+ [IOMETHOD_WORKER] = &pgaio_worker_ops,
+#ifdef USE_LIBURING
+ [IOMETHOD_IO_URING] = &pgaio_uring_ops,
+#endif
+};
+
+
+const IoMethodOps *pgaio_impl;
+
+
+
+/* --------------------------------------------------------------------------------
+ * "Core" IO Api
+ * --------------------------------------------------------------------------------
+ */
+
+/*
+ * AFIXME: rewrite
+ *
+ * Shared completion callbacks can be executed by any backend (otherwise there
+ * would be deadlocks). Therefore they cannot update state for the issuer of
+ * the IO. That can be done with issuer callbacks.
+ *
+ * Note that issuer callbacks are effectively executed in a critical
+ * section. This is necessary as we need to be able to execute IO in critical
+ * sections (consider e.g. WAL logging) and to be able to execute IOs we need
+ * to acquire an IO, which in turn requires executing issuer callbacks. An
+ * alternative scheme could be to defer local callback execution until a later
+ * point, but that gets complicated quickly.
+ *
+ * Therefore the typical pattern is to use an issuer callback to set some
+ * flags in backend local memory, which can then be used to error out at a
+ * later time.
+ *
+ * NB: The issuer callback is cleared when the resowner owning the IO goes out
+ * of scope.
+ */
+PgAioHandle *
+pgaio_io_get(struct ResourceOwnerData *resowner, PgAioReturn *ret)
+{
+ PgAioHandle *h;
+
+ while (true)
+ {
+ h = pgaio_io_get_nb(resowner, ret);
+
+ if (h != NULL)
+ return h;
+
+ /*
+ * Evidently all handles by this backend are in use. Just wait for
+ * some to complete.
+ */
+ pgaio_io_wait_for_free();
+ }
+}
+
+PgAioHandle *
+pgaio_io_get_nb(struct ResourceOwnerData *resowner, PgAioReturn *ret)
+{
+ if (dclist_count(&my_aio->staged_ios) >= PGAIO_SUBMIT_BATCH_SIZE)
+ {
+ pgaio_submit_staged();
+ }
+
+ if (my_aio->handed_out_io)
+ {
+ ereport(ERROR,
+ errmsg("API violation: Only one IO can be handed out"));
+ }
+
+ if (!dclist_is_empty(&my_aio->idle_ios))
+ {
+ dlist_node *ion = dclist_pop_head_node(&my_aio->idle_ios);
+ PgAioHandle *ioh = dclist_container(PgAioHandle, node, ion);
+
+ Assert(ioh->state == AHS_IDLE);
+ Assert(ioh->owner_procno == MyProcNumber);
+
+ ioh->state = AHS_HANDED_OUT;
+ my_aio->handed_out_io = ioh;
+
+ if (resowner)
+ pgaio_io_resowner_register(ioh);
+
+ if (ret)
+ ioh->report_return = ret;
+
+ return ioh;
+ }
+
+ return NULL;
+}
+
+void
+pgaio_io_release(PgAioHandle *ioh)
+{
+ if (ioh == my_aio->handed_out_io)
+ {
+ Assert(ioh->state == AHS_HANDED_OUT);
+ Assert(ioh->resowner);
+
+ my_aio->handed_out_io = NULL;
+ pgaio_io_reclaim(ioh);
+ }
+ else
+ {
+ elog(ERROR, "release in unexpected state");
+ }
+}
+
+void
+pgaio_io_release_resowner(dlist_node *ioh_node, bool on_error)
+{
+ PgAioHandle *ioh = dlist_container(PgAioHandle, resowner_node, ioh_node);
+
+ Assert(ioh->resowner);
+
+ ResourceOwnerForgetAioHandle(ioh->resowner, &ioh->resowner_node);
+ ioh->resowner = NULL;
+
+ switch (ioh->state)
+ {
+ case AHS_IDLE:
+ elog(ERROR, "unexpected");
+ break;
+ case AHS_HANDED_OUT:
+ Assert(ioh == my_aio->handed_out_io || my_aio->handed_out_io == NULL);
+
+ if (ioh == my_aio->handed_out_io)
+ {
+ my_aio->handed_out_io = NULL;
+ if (!on_error)
+ elog(WARNING, "leaked AIO handle");
+ }
+
+ pgaio_io_reclaim(ioh);
+ break;
+ case AHS_DEFINED:
+ case AHS_PREPARED:
+ /* XXX: Should we warn about this when is_commit? */
+ pgaio_submit_staged();
+ break;
+ case AHS_IN_FLIGHT:
+ case AHS_REAPED:
+ case AHS_COMPLETED_SHARED:
+ /* this is expected to happen */
+ break;
+ case AHS_COMPLETED_LOCAL:
+ /* XXX: unclear if this ought to be possible? */
+ pgaio_io_reclaim(ioh);
+ break;
+ }
+
+ /*
+ * Need to unregister the reporting of the IO's result, the memory it's
+ * referencing likely has gone away.
+ */
+ if (ioh->report_return)
+ ioh->report_return = NULL;
+}
+
+int
+pgaio_io_get_iovec(PgAioHandle *ioh, struct iovec **iov)
+{
+ Assert(ioh->state == AHS_HANDED_OUT);
+
+ *iov = &aio_ctl->iovecs[ioh->iovec_off];
+
+ /* AFIXME: Needs to be the value at startup time */
+ return io_combine_limit;
+}
+
+PgAioSubjectData *
+pgaio_io_get_subject_data(PgAioHandle *ioh)
+{
+ return &ioh->scb_data;
+}
+
+PgAioOpData *
+pgaio_io_get_op_data(PgAioHandle *ioh)
+{
+ return &ioh->op_data;
+}
+
+ProcNumber
+pgaio_io_get_owner(PgAioHandle *ioh)
+{
+ return ioh->owner_procno;
+}
+
+bool
+pgaio_io_has_subject(PgAioHandle *ioh)
+{
+ return ioh->subject != ASI_INVALID;
+}
+
+void
+pgaio_io_set_flag(PgAioHandle *ioh, PgAioHandleFlags flag)
+{
+ Assert(ioh->state == AHS_HANDED_OUT);
+
+ ioh->flags |= flag;
+}
+
+void
+pgaio_io_set_io_data_32(PgAioHandle *ioh, uint32 *data, uint8 len)
+{
+ Assert(ioh->state == AHS_HANDED_OUT);
+
+ for (int i = 0; i < len; i++)
+ aio_ctl->iovecs_data[ioh->iovec_off + i] = data[i];
+ ioh->iovec_data_len = len;
+}
+
+uint64 *
+pgaio_io_get_io_data(PgAioHandle *ioh, uint8 *len)
+{
+ Assert(ioh->iovec_data_len > 0);
+
+ *len = ioh->iovec_data_len;
+
+ return &aio_ctl->iovecs_data[ioh->iovec_off];
+}
+
+void
+pgaio_io_set_subject(PgAioHandle *ioh, PgAioSubjectID subjid)
+{
+ Assert(ioh->state == AHS_HANDED_OUT);
+
+ ioh->subject = subjid;
+
+ elog(DEBUG3, "io:%d, op %s, subject %s, set subject",
+ pgaio_io_get_id(ioh),
+ pgaio_io_get_op_name(ioh),
+ pgaio_io_get_subject_name(ioh));
+}
+
+void
+pgaio_io_get_ref(PgAioHandle *ioh, PgAioHandleRef *ior)
+{
+ Assert(ioh->state == AHS_HANDED_OUT ||
+ ioh->state == AHS_DEFINED ||
+ ioh->state == AHS_PREPARED);
+ Assert(ioh->generation != 0);
+
+ ior->aio_index = ioh - aio_ctl->io_handles;
+ ior->generation_upper = (uint32) (ioh->generation >> 32);
+ ior->generation_lower = (uint32) ioh->generation;
+}
+
+void
+pgaio_io_ref_clear(PgAioHandleRef *ior)
+{
+ ior->aio_index = PG_UINT32_MAX;
+}
+
+bool
+pgaio_io_ref_valid(PgAioHandleRef *ior)
+{
+ return ior->aio_index != PG_UINT32_MAX;
+}
+
+int
+pgaio_io_ref_get_id(PgAioHandleRef *ior)
+{
+ Assert(pgaio_io_ref_valid(ior));
+ return ior->aio_index;
+}
+
+bool
+pgaio_io_was_recycled(PgAioHandle *ioh, uint64 ref_generation, PgAioHandleState *state)
+{
+ *state = ioh->state;
+ pg_read_barrier();
+
+ return ioh->generation != ref_generation;
+}
+
+void
+pgaio_io_ref_wait(PgAioHandleRef *ior)
+{
+ uint64 ref_generation;
+ PgAioHandleState state;
+ bool am_owner;
+ PgAioHandle *ioh;
+
+ ioh = pgaio_io_from_ref(ior, &ref_generation);
+
+ am_owner = ioh->owner_procno == MyProcNumber;
+
+
+ if (pgaio_io_was_recycled(ioh, ref_generation, &state))
+ return;
+
+ if (am_owner)
+ {
+ if (state == AHS_DEFINED || state == AHS_PREPARED)
+ {
+ /* XXX: Arguably this should be prevented by callers? */
+ pgaio_submit_staged();
+ }
+ else if (state != AHS_IN_FLIGHT && state != AHS_REAPED && state != AHS_COMPLETED_SHARED && state != AHS_COMPLETED_LOCAL)
+ {
+ elog(PANIC, "waiting for own IO in wrong state: %d",
+ state);
+ }
+
+ /*
+ * Somebody else completed the IO, need to execute issuer callback, so
+ * reclaim eagerly.
+ */
+ if (state == AHS_COMPLETED_LOCAL)
+ {
+ pgaio_io_reclaim(ioh);
+
+ return;
+ }
+ }
+
+ while (true)
+ {
+ if (pgaio_io_was_recycled(ioh, ref_generation, &state))
+ return;
+
+ switch (state)
+ {
+ case AHS_IDLE:
+ case AHS_HANDED_OUT:
+ elog(ERROR, "IO in wrong state: %d", state);
+ break;
+
+ case AHS_IN_FLIGHT:
+ if (pgaio_impl->wait_one)
+ {
+ pgaio_impl->wait_one(ioh, ref_generation);
+ continue;
+ }
+ /* fallthrough */
+
+ /* waiting for owner to submit */
+ case AHS_PREPARED:
+ case AHS_DEFINED:
+ /* waiting for reaper to complete */
+ /* fallthrough */
+ case AHS_REAPED:
+ /* shouldn't be able to hit this otherwise */
+ Assert(IsUnderPostmaster);
+ /* ensure we're going to get woken up */
+ ConditionVariablePrepareToSleep(&ioh->cv);
+
+ while (!pgaio_io_was_recycled(ioh, ref_generation, &state))
+ {
+ if (state != AHS_REAPED && state != AHS_DEFINED &&
+ state != AHS_IN_FLIGHT)
+ break;
+ ConditionVariableSleep(&ioh->cv, WAIT_EVENT_AIO_COMPLETION);
+ }
+
+ ConditionVariableCancelSleep();
+ break;
+
+ case AHS_COMPLETED_SHARED:
+ /* see above */
+ if (am_owner)
+ pgaio_io_reclaim(ioh);
+ return;
+ case AHS_COMPLETED_LOCAL:
+ return;
+ }
+ }
+}
+
+bool
+pgaio_io_ref_check_done(PgAioHandleRef *ior)
+{
+ uint64 ref_generation;
+ PgAioHandleState state;
+ bool am_owner;
+ PgAioHandle *ioh;
+
+ ioh = pgaio_io_from_ref(ior, &ref_generation);
+
+ if (pgaio_io_was_recycled(ioh, ref_generation, &state))
+ return true;
+
+
+ if (state == AHS_IDLE)
+ return true;
+
+ am_owner = ioh->owner_procno == MyProcNumber;
+
+ if (state == AHS_COMPLETED_SHARED || state == AHS_COMPLETED_LOCAL)
+ {
+ if (am_owner)
+ pgaio_io_reclaim(ioh);
+ return true;
+ }
+
+ return false;
+}
+
+int
+pgaio_io_get_id(PgAioHandle *ioh)
+{
+ Assert(ioh >= aio_ctl->io_handles &&
+ ioh <= (aio_ctl->io_handles + aio_ctl->io_handle_count));
+ return ioh - aio_ctl->io_handles;
+}
+
+const char *
+pgaio_io_get_state_name(PgAioHandle *ioh)
+{
+ switch (ioh->state)
+ {
+ case AHS_IDLE:
+ return "idle";
+ case AHS_HANDED_OUT:
+ return "handed_out";
+ case AHS_DEFINED:
+ return "DEFINED";
+ case AHS_PREPARED:
+ return "PREPARED";
+ case AHS_IN_FLIGHT:
+ return "IN_FLIGHT";
+ case AHS_REAPED:
+ return "REAPED";
+ case AHS_COMPLETED_SHARED:
+ return "COMPLETED_SHARED";
+ case AHS_COMPLETED_LOCAL:
+ return "COMPLETED_LOCAL";
+ }
+ pg_unreachable();
+}
+
+/*
+ * Internal, should only be called from pgaio_io_prep_*().
+ */
+void
+pgaio_io_prepare(PgAioHandle *ioh, PgAioOp op)
+{
+ Assert(ioh->state == AHS_HANDED_OUT);
+ Assert(pgaio_io_has_subject(ioh));
+
+ ioh->op = op;
+ ioh->state = AHS_DEFINED;
+ ioh->result = 0;
+
+ /* allow a new IO to be staged */
+ my_aio->handed_out_io = NULL;
+
+ dclist_push_tail(&my_aio->staged_ios, &ioh->node);
+
+ pgaio_io_prepare_subject(ioh);
+
+ ioh->state = AHS_PREPARED;
+
+ elog(DEBUG3, "io:%d: prepared %s",
+ pgaio_io_get_id(ioh), pgaio_io_get_op_name(ioh));
+}
+
+/*
+ * Handle IO getting completed by a method.
+ */
+void
+pgaio_io_process_completion(PgAioHandle *ioh, int result)
+{
+ ioh->result = result;
+
+ pg_write_barrier();
+
+ /* FIXME: should be done in separate function */
+ ioh->state = AHS_REAPED;
+
+ pgaio_io_process_completion_subject(ioh);
+
+ /* ensure results of completion are visible before the new state */
+ pg_write_barrier();
+
+ ioh->state = AHS_COMPLETED_SHARED;
+
+ /* condition variable broadcast ensures state is visible before wakeup */
+ ConditionVariableBroadcast(&ioh->cv);
+
+ if (ioh->owner_procno == MyProcNumber)
+ pgaio_io_reclaim(ioh);
+}
+
+/*
+ * Handle IO being processed by IO method.
+ */
+void
+pgaio_io_prepare_submit(PgAioHandle *ioh)
+{
+ ioh->state = AHS_IN_FLIGHT;
+ pg_write_barrier();
+
+ dclist_delete_from(&my_aio->staged_ios, &ioh->node);
+}
+
+static PgAioHandle *
+pgaio_io_from_ref(PgAioHandleRef *ior, uint64 *ref_generation)
+{
+ PgAioHandle *ioh;
+
+ Assert(ior->aio_index < aio_ctl->io_handle_count);
+
+ ioh = &aio_ctl->io_handles[ior->aio_index];
+
+ *ref_generation = ((uint64) ior->generation_upper) << 32 |
+ ior->generation_lower;
+
+ Assert(*ref_generation != 0);
+
+ return ioh;
+}
+
+static void
+pgaio_io_resowner_register(PgAioHandle *ioh)
+{
+ Assert(!ioh->resowner);
+ Assert(CurrentResourceOwner);
+
+ ResourceOwnerRememberAioHandle(CurrentResourceOwner, &ioh->resowner_node);
+ ioh->resowner = CurrentResourceOwner;
+}
+
+static void
+pgaio_io_reclaim(PgAioHandle *ioh)
+{
+ /* This is only ok if it's our IO */
+ Assert(ioh->owner_procno == MyProcNumber);
+
+ ereport(DEBUG3,
+ errmsg("reclaiming io:%d, state: %s, op %s, subject %s, result: %d, distilled_result: AFIXME, report to: %p",
+ pgaio_io_get_id(ioh),
+ pgaio_io_get_state_name(ioh),
+ pgaio_io_get_op_name(ioh),
+ pgaio_io_get_subject_name(ioh),
+ ioh->result,
+ ioh->report_return
+ ),
+ errhidestmt(true), errhidecontext(true));
+
+ if (ioh->report_return)
+ {
+ if (ioh->state != AHS_HANDED_OUT)
+ {
+ ioh->report_return->result = ioh->distilled_result;
+ ioh->report_return->subject_data = ioh->scb_data;
+ }
+ }
+
+ /* reclaim all associated bounce buffers */
+ if (!slist_is_empty(&ioh->bounce_buffers))
+ {
+ slist_mutable_iter it;
+
+ slist_foreach_modify(it, &ioh->bounce_buffers)
+ {
+ PgAioBounceBuffer *bb = slist_container(PgAioBounceBuffer, node, it.cur);
+
+ slist_delete_current(&it);
+
+ slist_push_head(&my_aio->idle_bbs, &bb->node);
+ }
+ }
+
+ if (ioh->resowner)
+ {
+ ResourceOwnerForgetAioHandle(ioh->resowner, &ioh->resowner_node);
+ ioh->resowner = NULL;
+ }
+
+ Assert(!ioh->resowner);
+
+ ioh->num_shared_callbacks = 0;
+ ioh->iovec_data_len = 0;
+ ioh->report_return = NULL;
+ ioh->flags = 0;
+
+ pg_write_barrier();
+ ioh->generation++;
+ pg_write_barrier();
+ ioh->state = AHS_IDLE;
+ pg_write_barrier();
+
+ dclist_push_tail(&my_aio->idle_ios, &ioh->node);
+}
+
+static void
+pgaio_io_wait_for_free(void)
+{
+ bool found_handed_out = false;
+ int reclaimed = 0;
+ static uint32 lastpos = 0;
+
+ elog(DEBUG2,
+ "waiting for self: %d pending",
+ dclist_count(&my_aio->staged_ios));
+
+ /*
+ * First check if any of our IOs actually have completed - when using
+ * worker, that'll often be the case. We could do so as part of the loop
+ * below, but that'd potentially lead us to wait for some IO submitted
+ * before.
+ */
+ for (int i = 0; i < io_max_concurrency; i++)
+ {
+ PgAioHandle *ioh = &aio_ctl->io_handles[my_aio->io_handle_off + i];
+
+ if (ioh->state == AHS_COMPLETED_SHARED)
+ {
+ pgaio_io_reclaim(ioh);
+ reclaimed++;
+ }
+ }
+
+ if (reclaimed > 0)
+ return;
+
+ if (!dclist_is_empty(&my_aio->staged_ios))
+ {
+ elog(DEBUG2, "submitting while acquiring free io");
+ pgaio_submit_staged();
+ }
+
+ for (uint32 i = lastpos; i < lastpos + io_max_concurrency; i++)
+ {
+ uint32 thisoff = my_aio->io_handle_off + (i % io_max_concurrency);
+ PgAioHandle *ioh = &aio_ctl->io_handles[thisoff];
+
+ switch (ioh->state)
+ {
+ case AHS_IDLE:
+
+ /*
+ * While one might think that pgaio_io_get_nb() should have
+ * succeeded, this is reachable because the IO could have
+ * completed during the submission above.
+ */
+ return;
+ case AHS_DEFINED: /* should have been submitted above */
+ case AHS_PREPARED:
+ case AHS_COMPLETED_LOCAL:
+ elog(ERROR, "shouldn't get here with io:%d in state %d",
+ pgaio_io_get_id(ioh), ioh->state);
+ break;
+ case AHS_HANDED_OUT:
+ if (found_handed_out)
+ elog(ERROR, "more than one handed out IO");
+ found_handed_out = true;
+ continue;
+ case AHS_REAPED:
+ case AHS_IN_FLIGHT:
+ {
+ PgAioHandleRef ior;
+
+ ior.aio_index = ioh - aio_ctl->io_handles;
+ ior.generation_upper = (uint32) (ioh->generation >> 32);
+ ior.generation_lower = (uint32) ioh->generation;
+
+ pgaio_io_ref_wait(&ior);
+ elog(DEBUG2, "waited for io:%d",
+ pgaio_io_get_id(ioh));
+ lastpos = i;
+ return;
+ }
+ break;
+ case AHS_COMPLETED_SHARED:
+ /* reclaim */
+ pgaio_io_reclaim(ioh);
+ lastpos = i;
+ return;
+ }
+ }
+
+ elog(PANIC, "could not reclaim any handles");
+}
+
+
+
+/* --------------------------------------------------------------------------------
+ * Bounce Buffers
+ * --------------------------------------------------------------------------------
+ */
+
+PgAioBounceBuffer *
+pgaio_bounce_buffer_get(void)
+{
+ PgAioBounceBuffer *bb = NULL;
+ slist_node *node;
+
+ if (my_aio->handed_out_bb != NULL)
+ elog(ERROR, "can only hand out one BB");
+
+ /*
+ * FIXME It probably is not correct to have bounce buffers be per backend,
+ * they use too much memory.
+ */
+ if (slist_is_empty(&my_aio->idle_bbs))
+ {
+ pgaio_bounce_buffer_wait_for_free();
+ }
+
+ node = slist_pop_head_node(&my_aio->idle_bbs);
+ bb = slist_container(PgAioBounceBuffer, node, node);
+
+ my_aio->handed_out_bb = bb;
+
+ bb->resowner = CurrentResourceOwner;
+ ResourceOwnerRememberAioBounceBuffer(bb->resowner, &bb->resowner_node);
+
+ return bb;
+}
+
+void
+pgaio_io_assoc_bounce_buffer(PgAioHandle *ioh, PgAioBounceBuffer *bb)
+{
+ if (my_aio->handed_out_bb != bb)
+ elog(ERROR, "can only assign handed out BB");
+ my_aio->handed_out_bb = NULL;
+
+ /*
+ * There can be many bounce buffers assigned in case of vectorized IOs.
+ */
+ slist_push_head(&ioh->bounce_buffers, &bb->node);
+
+ /* once associated with an IO, the IO has ownership */
+ ResourceOwnerForgetAioBounceBuffer(bb->resowner, &bb->resowner_node);
+ bb->resowner = NULL;
+}
+
+uint32
+pgaio_bounce_buffer_id(PgAioBounceBuffer *bb)
+{
+ return bb - aio_ctl->bounce_buffers;
+}
+
+void
+pgaio_bounce_buffer_release(PgAioBounceBuffer *bb)
+{
+ if (my_aio->handed_out_bb != bb)
+ elog(ERROR, "can only release handed out BB");
+
+ slist_push_head(&my_aio->idle_bbs, &bb->node);
+ my_aio->handed_out_bb = NULL;
+
+ ResourceOwnerForgetAioBounceBuffer(bb->resowner, &bb->resowner_node);
+ bb->resowner = NULL;
+}
+
+void
+pgaio_bounce_buffer_release_resowner(dlist_node *bb_node, bool on_error)
+{
+ PgAioBounceBuffer *bb = dlist_container(PgAioBounceBuffer, resowner_node, bb_node);
+
+ Assert(bb->resowner);
+
+ if (!on_error)
+ elog(WARNING, "leaked AIO bounce buffer");
+
+ pgaio_bounce_buffer_release(bb);
+}
+
+char *
+pgaio_bounce_buffer_buffer(PgAioBounceBuffer *bb)
+{
+ return bb->buffer;
+}
+
+static void
+pgaio_bounce_buffer_wait_for_free(void)
+{
+ static uint32 lastpos = 0;
+
+ if (!dclist_is_empty(&my_aio->staged_ios))
+ {
+ elog(DEBUG2, "submitting while acquiring free bb");
+ pgaio_submit_staged();
+ }
+
+ for (uint32 i = lastpos; i < lastpos + io_max_concurrency; i++)
+ {
+ uint32 thisoff = my_aio->io_handle_off + (i % io_max_concurrency);
+ PgAioHandle *ioh = &aio_ctl->io_handles[thisoff];
+
+ switch (ioh->state)
+ {
+ case AHS_IDLE:
+ case AHS_HANDED_OUT:
+ continue;
+ case AHS_DEFINED: /* should have been submitted above */
+ case AHS_PREPARED:
+ elog(ERROR, "shouldn't get here with io:%d in state %d",
+ pgaio_io_get_id(ioh), ioh->state);
+ break;
+ case AHS_REAPED:
+ case AHS_IN_FLIGHT:
+ if (!slist_is_empty(&ioh->bounce_buffers))
+ {
+ PgAioHandleRef ior;
+
+ ior.aio_index = ioh - aio_ctl->io_handles;
+ ior.generation_upper = (uint32) (ioh->generation >> 32);
+ ior.generation_lower = (uint32) ioh->generation;
+
+ pgaio_io_ref_wait(&ior);
+ elog(DEBUG2, "waited for io:%d to reclaim BB",
+ pgaio_io_get_id(ioh));
+
+ if (slist_is_empty(&my_aio->idle_bbs))
+ elog(WARNING, "empty after wait");
+
+ if (!slist_is_empty(&my_aio->idle_bbs))
+ {
+ lastpos = i;
+ return;
+ }
+ }
+ break;
+ case AHS_COMPLETED_SHARED:
+ case AHS_COMPLETED_LOCAL:
+ /* reclaim */
+ pgaio_io_reclaim(ioh);
+
+ if (!slist_is_empty(&my_aio->idle_bbs))
+ {
+ lastpos = i;
+ return;
+ }
+ break;
+ }
+ }
+
+ /*
+ * The submission above could have caused the IO to complete at any time.
+ */
+ if (slist_is_empty(&my_aio->idle_bbs))
+ elog(PANIC, "no more bbs");
+}
+
+
+
+/* --------------------------------------------------------------------------------
+ * Actions on multiple IOs.
+ * --------------------------------------------------------------------------------
+ */
+
+void
+pgaio_submit_staged(void)
+{
+ int total_submitted = 0;
+
+ if (dclist_is_empty(&my_aio->staged_ios))
+ return;
+
+ while (!dclist_is_empty(&my_aio->staged_ios))
+ {
+ int staged_count PG_USED_FOR_ASSERTS_ONLY = dclist_count(&my_aio->staged_ios);
+ int did_submit;
+
+ Assert(staged_count > 0);
+
+ START_CRIT_SECTION();
+ END_CRIT_SECTION();
+
+ did_submit = pgaio_impl->submit();
+
+ total_submitted += did_submit;
+ }
+
+#ifdef PGAIO_VERBOSE
+ ereport(DEBUG2,
+ errmsg("submitted %d",
+ total_submitted),
+ errhidestmt(true),
+ errhidecontext(true));
+#endif
+}
+
+bool
+pgaio_have_staged(void)
+{
+ return !dclist_is_empty(&my_aio->staged_ios);
+}
+
+
+
+/* --------------------------------------------------------------------------------
+ * Other
+ * --------------------------------------------------------------------------------
+ */
+
+/*
+ * Need to submit staged but not yet submitted IOs using the fd, otherwise
+ * the IO would end up targeting something bogus.
+ */
+void
+pgaio_closing_fd(int fd)
+{
+ /*
+ * Might be called before AIO is initialized or in a subprocess that
+ * doesn't use AIO.
+ */
+ if (!my_aio)
+ return;
+
+ /*
+ * For now just submit all staged IOs - we could be more selective, but
+ * it's probably not worth it.
+ */
+ pgaio_submit_staged();
+}
+
+void
+pgaio_at_xact_end(bool is_subxact, bool is_commit)
+{
+ Assert(!my_aio->handed_out_io);
+ Assert(!my_aio->handed_out_bb);
+}
+
+/*
+ * Similar to pgaio_at_xact_end(..., is_commit = false), but for cases where
+ * errors happen outside of transactions.
+ */
+void
+pgaio_at_error(void)
+{
+ Assert(!my_aio->handed_out_io);
+ Assert(!my_aio->handed_out_bb);
+}
void
assign_io_method(int newval, void *extra)
{
+ pgaio_impl = pgaio_ops_table[newval];
}
diff --git a/src/backend/storage/aio/aio_init.c b/src/backend/storage/aio/aio_init.c
index 1c277a7eb3b..cf3512f79fc 100644
--- a/src/backend/storage/aio/aio_init.c
+++ b/src/backend/storage/aio/aio_init.c
@@ -14,33 +14,351 @@
#include "postgres.h"
+#include "miscadmin.h"
+#include "storage/aio.h"
#include "storage/aio_init.h"
+#include "storage/aio_internal.h"
+#include "storage/bufmgr.h"
+#include "storage/io_worker.h"
+#include "storage/proc.h"
+#include "storage/shmem.h"
+static Size
+AioCtlShmemSize(void)
+{
+ Size sz;
+
+ /* aio_ctl itself */
+ sz = offsetof(PgAioCtl, io_handles);
+
+ return sz;
+}
+
+static uint32
+AioProcs(void)
+{
+ /*
+ * While AIO workers don't need their own AIO context, we can't currently
+ * guarantee nothing gets assigned to the a ProcNumber for an IO worker if
+ * we just subtracted MAX_IO_WORKERS.
+ */
+ return MaxBackends + NUM_AUXILIARY_PROCS;
+}
+
+static Size
+AioBackendShmemSize(void)
+{
+ return mul_size(AioProcs(), sizeof(PgAioPerBackend));
+}
+
+static Size
+AioHandleShmemSize(void)
+{
+ Size sz;
+
+ /* ios */
+ sz = mul_size(AioProcs(),
+ mul_size(io_max_concurrency, sizeof(PgAioHandle)));
+
+ return sz;
+}
+
+static Size
+AioIOVShmemSize(void)
+{
+ /* FIXME: io_combine_limit is USERSET */
+ return mul_size(sizeof(struct iovec),
+ mul_size(mul_size(io_combine_limit, AioProcs()),
+ io_max_concurrency));
+}
+
+static Size
+AioIOVDataShmemSize(void)
+{
+ /* FIXME: io_combine_limit is USERSET */
+ return mul_size(sizeof(uint64),
+ mul_size(mul_size(io_combine_limit, AioProcs()),
+ io_max_concurrency));
+}
+
+static Size
+AioBounceBufferDescShmemSize(void)
+{
+ Size sz;
+
+ /* PgAioBounceBuffer itself */
+ sz = mul_size(sizeof(PgAioBounceBuffer),
+ mul_size(AioProcs(), io_bounce_buffers));
+
+ return sz;
+}
+
+static Size
+AioBounceBufferDataShmemSize(void)
+{
+ Size sz;
+
+ /* and the associated buffer */
+ sz = mul_size(BLCKSZ,
+ mul_size(io_bounce_buffers, AioProcs()));
+ /* memory for alignment */
+ sz += BLCKSZ;
+
+ return sz;
+}
+
+/*
+ * Choose a suitable value for io_max_concurrency.
+ *
+ * It's unlikely that we could have more IOs in flight than buffers that we
+ * would be allowed to pin.
+ *
+ * On the upper end, apply a cap too - just because shared_buffers is large,
+ * it doesn't make sense have millions of buffers undergo IO concurrently.
+ */
+static int
+AioChooseMaxConccurrency(void)
+{
+ uint32 max_backends;
+ int max_proportional_pins;
+
+ /* Similar logic to LimitAdditionalPins() */
+ max_backends = MaxBackends + NUM_AUXILIARY_PROCS;
+ max_proportional_pins = NBuffers / max_backends;
+
+ max_proportional_pins = Max(max_proportional_pins, 1);
+
+ /* apply upper limit */
+ return Min(max_proportional_pins, 64);
+}
+
+/*
+ * Choose a suitable value for io_bounce_buffers.
+ *
+ * It's very unlikely to be useful to allocate more bounce buffers for each
+ * backend than the backend is allowed to pin. Additionally, bounce buffers
+ * currently are used for writes, it seems very uncommon for more than 10% of
+ * shared_buffers to be written out concurrently.
+ *
+ * XXX: This quickly can take up significant amounts of memory, the logic
+ * should probably fine tuned.
+ */
+static int
+AioChooseBounceBuffers(void)
+{
+ uint32 max_backends;
+ int max_proportional_pins;
+
+ /* Similar logic to LimitAdditionalPins() */
+ max_backends = MaxBackends + NUM_AUXILIARY_PROCS;
+ max_proportional_pins = (NBuffers / 10) / max_backends;
+
+ max_proportional_pins = Max(max_proportional_pins, 1);
+
+ /* apply upper limit */
+ return Min(max_proportional_pins, 256);
+}
+
Size
AioShmemSize(void)
{
Size sz = 0;
+ /*
+ * We prefer to report this value's source as PGC_S_DYNAMIC_DEFAULT.
+ * However, if the DBA explicitly set wal_buffers = -1 in the config file,
+ * then PGC_S_DYNAMIC_DEFAULT will fail to override that and we must force
+ *
+ */
+ if (io_max_concurrency == -1)
+ {
+ char buf[32];
+
+ snprintf(buf, sizeof(buf), "%d", AioChooseMaxConccurrency());
+ SetConfigOption("io_max_concurrency", buf, PGC_POSTMASTER,
+ PGC_S_DYNAMIC_DEFAULT);
+ if (io_bounce_buffers == -1) /* failed to apply it? */
+ SetConfigOption("io_max_concurrency", buf, PGC_POSTMASTER,
+ PGC_S_OVERRIDE);
+ }
+
+
+ /*
+ * If io_bounce_buffers is -1, we automatically choose a suitable value.
+ *
+ * See also comment above.
+ */
+ if (io_bounce_buffers == -1)
+ {
+ char buf[32];
+
+ snprintf(buf, sizeof(buf), "%d", AioChooseBounceBuffers());
+ SetConfigOption("io_bounce_buffers", buf, PGC_POSTMASTER,
+ PGC_S_DYNAMIC_DEFAULT);
+ if (io_bounce_buffers == -1) /* failed to apply it? */
+ SetConfigOption("io_bounce_buffers", buf, PGC_POSTMASTER,
+ PGC_S_OVERRIDE);
+ }
+
+ sz = add_size(sz, AioCtlShmemSize());
+ sz = add_size(sz, AioBackendShmemSize());
+ sz = add_size(sz, AioHandleShmemSize());
+ sz = add_size(sz, AioIOVShmemSize());
+ sz = add_size(sz, AioIOVDataShmemSize());
+ sz = add_size(sz, AioBounceBufferDescShmemSize());
+ sz = add_size(sz, AioBounceBufferDataShmemSize());
+
+ if (pgaio_impl->shmem_size)
+ sz = add_size(sz, pgaio_impl->shmem_size());
+
return sz;
}
void
AioShmemInit(void)
{
+ bool found;
+ uint32 io_handle_off = 0;
+ uint32 iovec_off = 0;
+ uint32 bounce_buffers_off = 0;
+ uint32 per_backend_iovecs = io_max_concurrency * io_combine_limit;
+ uint32 per_backend_bb = io_bounce_buffers;
+ char *bounce_buffers_data;
+
+ aio_ctl = (PgAioCtl *)
+ ShmemInitStruct("AioCtl", AioCtlShmemSize(), &found);
+
+ if (found)
+ goto out;
+
+ memset(aio_ctl, 0, AioCtlShmemSize());
+
+ aio_ctl->io_handle_count = AioProcs() * io_max_concurrency;
+ aio_ctl->iovec_count = AioProcs() * per_backend_iovecs;
+ aio_ctl->bounce_buffers_count = AioProcs() * per_backend_bb;
+
+ aio_ctl->backend_state = (PgAioPerBackend *)
+ ShmemInitStruct("AioBackend", AioBackendShmemSize(), &found);
+
+ aio_ctl->io_handles = (PgAioHandle *)
+ ShmemInitStruct("AioHandle", AioHandleShmemSize(), &found);
+
+ aio_ctl->iovecs = ShmemInitStruct("AioIOV", AioIOVShmemSize(), &found);
+ aio_ctl->iovecs_data = ShmemInitStruct("AioIOVData", AioIOVDataShmemSize(), &found);
+
+ aio_ctl->bounce_buffers = ShmemInitStruct("AioBounceBufferDesc", AioBounceBufferDescShmemSize(), &found);
+
+ bounce_buffers_data = ShmemInitStruct("AioBounceBufferData", AioBounceBufferDataShmemSize(), &found);
+ bounce_buffers_data = (char *) TYPEALIGN(BLCKSZ, (uintptr_t) bounce_buffers_data);
+ aio_ctl->bounce_buffers_data = bounce_buffers_data;
+
+
+ /* Initialize IO handles. */
+ for (uint64 i = 0; i < aio_ctl->io_handle_count; i++)
+ {
+ PgAioHandle *ioh = &aio_ctl->io_handles[i];
+
+ ioh->op = PGAIO_OP_INVALID;
+ ioh->subject = ASI_INVALID;
+ ioh->state = AHS_IDLE;
+
+ slist_init(&ioh->bounce_buffers);
+ }
+
+ /* Initialize Bounce Buffers. */
+ for (uint64 i = 0; i < aio_ctl->bounce_buffers_count; i++)
+ {
+ PgAioBounceBuffer *bb = &aio_ctl->bounce_buffers[i];
+
+ bb->buffer = bounce_buffers_data;
+ bounce_buffers_data += BLCKSZ;
+ }
+
+
+ for (int procno = 0; procno < AioProcs(); procno++)
+ {
+ PgAioPerBackend *bs = &aio_ctl->backend_state[procno];
+
+ bs->io_handle_off = io_handle_off;
+ io_handle_off += io_max_concurrency;
+
+ bs->bounce_buffers_off = bounce_buffers_off;
+ bounce_buffers_off += per_backend_bb;
+
+ dclist_init(&bs->idle_ios);
+ dclist_init(&bs->staged_ios);
+ slist_init(&bs->idle_bbs);
+
+ /* initialize per-backend IOs */
+ for (int i = 0; i < io_max_concurrency; i++)
+ {
+ PgAioHandle *ioh = &aio_ctl->io_handles[bs->io_handle_off + i];
+
+ ioh->generation = 1;
+ ioh->owner_procno = procno;
+ ioh->iovec_off = iovec_off;
+ ioh->iovec_data_len = 0;
+ ioh->report_return = NULL;
+ ioh->resowner = NULL;
+ ioh->num_shared_callbacks = 0;
+ ioh->distilled_result.status = ARS_UNKNOWN;
+ ioh->flags = 0;
+
+ ConditionVariableInit(&ioh->cv);
+
+ dclist_push_tail(&bs->idle_ios, &ioh->node);
+ iovec_off += io_combine_limit;
+ }
+
+ /* initialize per-backend bounce buffers */
+ for (int i = 0; i < per_backend_bb; i++)
+ {
+ PgAioBounceBuffer *bb = &aio_ctl->bounce_buffers[bs->bounce_buffers_off + i];
+
+ slist_push_head(&bs->idle_bbs, &bb->node);
+ }
+ }
+
+out:
+ /* Initialize IO method specific resources. */
+ pgaio_impl->shmem_init(!found);
}
void
pgaio_postmaster_init(void)
{
+ if (pgaio_impl->postmaster_init)
+ pgaio_impl->postmaster_init();
}
void
pgaio_postmaster_child_init(void)
{
+ /* shouldn't be initialized twice */
+ Assert(!my_aio);
+
+ if (MyBackendType == B_IO_WORKER)
+ return;
+
+ if (MyProc == NULL || MyProcNumber >= AioProcs())
+ elog(ERROR, "aio requires a normal PGPROC");
+
+ my_aio = &aio_ctl->backend_state[MyProcNumber];
+
+ if (pgaio_impl->postmaster_child_init)
+ pgaio_impl->postmaster_child_init();
}
void
pgaio_postmaster_child_init_local(void)
{
+ if (pgaio_impl->postmaster_child_init_local)
+ pgaio_impl->postmaster_child_init_local();
+}
+
+bool
+pgaio_workers_enabled(void)
+{
+ return io_method == IOMETHOD_WORKER;
}
diff --git a/src/backend/storage/aio/aio_io.c b/src/backend/storage/aio/aio_io.c
new file mode 100644
index 00000000000..5b2f9ee3ba6
--- /dev/null
+++ b/src/backend/storage/aio/aio_io.c
@@ -0,0 +1,111 @@
+/*-------------------------------------------------------------------------
+ *
+ * aio_io.c
+ * Asynchronous I/O subsytem.
+ *
+ * Portions Copyright (c) 1996-2024, PostgreSQL Global Development Group
+ * Portions Copyright (c) 1994, Regents of the University of California
+ *
+ * IDENTIFICATION
+ * src/backend/storage/aio/aio_io.c
+ *
+ *-------------------------------------------------------------------------
+ */
+
+#include "postgres.h"
+
+#include "storage/aio.h"
+#include "storage/aio_internal.h"
+#include "storage/fd.h"
+#include "utils/wait_event.h"
+
+
+static void
+pgaio_io_before_prep(PgAioHandle *ioh)
+{
+ Assert(ioh->state == AHS_HANDED_OUT);
+ Assert(pgaio_io_has_subject(ioh));
+}
+
+const char *
+pgaio_io_get_op_name(PgAioHandle *ioh)
+{
+ Assert(ioh->op >= 0 && ioh->op < PGAIO_OP_COUNT);
+
+ switch (ioh->op)
+ {
+ case PGAIO_OP_INVALID:
+ return "invalid";
+ case PGAIO_OP_READ:
+ return "read";
+ case PGAIO_OP_WRITE:
+ return "write";
+ case PGAIO_OP_FSYNC:
+ return "fsync";
+ case PGAIO_OP_FLUSH_RANGE:
+ return "flush_range";
+ case PGAIO_OP_NOP:
+ return "nop";
+ }
+
+ pg_unreachable();
+}
+
+void
+pgaio_io_prep_readv(PgAioHandle *ioh,
+ int fd, int iovcnt, uint64 offset)
+{
+ pgaio_io_before_prep(ioh);
+
+ ioh->op_data.read.fd = fd;
+ ioh->op_data.read.offset = offset;
+ ioh->op_data.read.iov_length = iovcnt;
+
+ pgaio_io_prepare(ioh, PGAIO_OP_READ);
+}
+
+void
+pgaio_io_prep_writev(PgAioHandle *ioh,
+ int fd, int iovcnt, uint64 offset)
+{
+ pgaio_io_before_prep(ioh);
+
+ ioh->op_data.write.fd = fd;
+ ioh->op_data.write.offset = offset;
+ ioh->op_data.write.iov_length = iovcnt;
+
+ pgaio_io_prepare(ioh, PGAIO_OP_WRITE);
+}
+
+
+extern void
+pgaio_io_perform_synchronously(PgAioHandle *ioh)
+{
+ ssize_t result = 0;
+ struct iovec *iov = &aio_ctl->iovecs[ioh->iovec_off];
+
+ /* Perform IO. */
+ switch (ioh->op)
+ {
+ case PGAIO_OP_READ:
+ pgstat_report_wait_start(WAIT_EVENT_DATA_FILE_READ);
+ result = pg_preadv(ioh->op_data.read.fd, iov,
+ ioh->op_data.read.iov_length,
+ ioh->op_data.read.offset);
+ pgstat_report_wait_end();
+ break;
+ case PGAIO_OP_WRITE:
+ pgstat_report_wait_start(WAIT_EVENT_DATA_FILE_WRITE);
+ result = pg_pwritev(ioh->op_data.write.fd, iov,
+ ioh->op_data.write.iov_length,
+ ioh->op_data.write.offset);
+ pgstat_report_wait_end();
+ break;
+ default:
+ elog(ERROR, "not yet");
+ }
+
+ ioh->result = result < 0 ? -errno : result;
+
+ pgaio_io_process_completion(ioh, ioh->result);
+}
diff --git a/src/backend/storage/aio/aio_subject.c b/src/backend/storage/aio/aio_subject.c
new file mode 100644
index 00000000000..68e9e80074c
--- /dev/null
+++ b/src/backend/storage/aio/aio_subject.c
@@ -0,0 +1,170 @@
+/*-------------------------------------------------------------------------
+ *
+ * aio_subject.c
+ * IO completion handling for IOs on different subjects
+ *
+ * XXX Write me
+ *
+ * Portions Copyright (c) 1996-2024, PostgreSQL Global Development Group
+ * Portions Copyright (c) 1994, Regents of the University of California
+ *
+ * IDENTIFICATION
+ * src/backend/storage/aio/aio_subject.c
+ *
+ *-------------------------------------------------------------------------
+ */
+
+#include "postgres.h"
+
+#include "storage/aio.h"
+#include "storage/aio_internal.h"
+#include "storage/buf_internals.h"
+#include "storage/bufmgr.h"
+#include "storage/smgr.h"
+#include "utils/memutils.h"
+
+
+static const PgAioSubjectInfo *aio_subject_info[] = {
+ [ASI_INVALID] = &(PgAioSubjectInfo) {
+ .name = "invalid",
+ },
+};
+
+static const PgAioHandleSharedCallbacks *aio_shared_cbs[] = {
+};
+
+
+void
+pgaio_io_add_shared_cb(PgAioHandle *ioh, PgAioHandleSharedCallbackID cbid)
+{
+ if (cbid >= lengthof(aio_shared_cbs))
+ elog(ERROR, "callback %d is out of range", cbid);
+ if (aio_shared_cbs[cbid]->complete == NULL)
+ elog(ERROR, "callback %d is undefined", cbid);
+ if (ioh->num_shared_callbacks >= AIO_MAX_SHARED_CALLBACKS)
+ elog(PANIC, "too many callbacks, the max is %d", AIO_MAX_SHARED_CALLBACKS);
+ ioh->shared_callbacks[ioh->num_shared_callbacks] = cbid;
+
+ elog(DEBUG3, "io:%d, op %s, subject %s, adding cbid num %d, id %d",
+ pgaio_io_get_id(ioh),
+ pgaio_io_get_op_name(ioh),
+ pgaio_io_get_subject_name(ioh),
+ ioh->num_shared_callbacks + 1, cbid);
+
+ ioh->num_shared_callbacks++;
+}
+
+const char *
+pgaio_io_get_subject_name(PgAioHandle *ioh)
+{
+ Assert(ioh->subject >= 0 && ioh->subject < ASI_COUNT);
+
+ return aio_subject_info[ioh->subject]->name;
+}
+
+void
+pgaio_io_prepare_subject(PgAioHandle *ioh)
+{
+ Assert(ioh->subject > ASI_INVALID && ioh->subject < ASI_COUNT);
+ Assert(ioh->op >= 0 && ioh->op < PGAIO_OP_COUNT);
+
+ for (int i = ioh->num_shared_callbacks; i > 0; i--)
+ {
+ PgAioHandleSharedCallbackID cbid = ioh->shared_callbacks[i - 1];
+ const PgAioHandleSharedCallbacks *cbs = aio_shared_cbs[cbid];
+
+ if (!cbs->prepare)
+ continue;
+
+ elog(DEBUG3, "io:%d, op %s, subject %s, calling cbid num %d, id %d: prepare",
+ pgaio_io_get_id(ioh),
+ pgaio_io_get_op_name(ioh),
+ pgaio_io_get_subject_name(ioh),
+ i, cbid);
+ cbs->prepare(ioh);
+ }
+}
+
+void
+pgaio_io_process_completion_subject(PgAioHandle *ioh)
+{
+ PgAioResult result;
+
+ Assert(ioh->subject >= 0 && ioh->subject < ASI_COUNT);
+ Assert(ioh->op >= 0 && ioh->op < PGAIO_OP_COUNT);
+
+ result.status = ARS_OK; /* low level IO is always considered OK */
+ result.result = ioh->result;
+ result.id = 0; /* FIXME */
+ result.error_data = 0;
+
+ for (int i = ioh->num_shared_callbacks; i > 0; i--)
+ {
+ PgAioHandleSharedCallbackID cbid;
+
+ cbid = ioh->shared_callbacks[i - 1];
+ elog(DEBUG3, "io:%d, op %s, subject %s, calling cbid num %d, id %d with distilled result status %d, id %u, error_data: %d, result: %d",
+ pgaio_io_get_id(ioh),
+ pgaio_io_get_op_name(ioh),
+ pgaio_io_get_subject_name(ioh),
+ i, cbid,
+ result.status,
+ result.id,
+ result.error_data,
+ result.result);
+ result = aio_shared_cbs[cbid]->complete(ioh, result);
+ }
+
+ ioh->distilled_result = result;
+
+ elog(DEBUG3, "io:%d, op %s, subject %s, distilled result status %d, id %u, error_data: %d, result: %d, raw_result %d",
+ pgaio_io_get_id(ioh),
+ pgaio_io_get_op_name(ioh),
+ pgaio_io_get_subject_name(ioh),
+ result.status,
+ result.id,
+ result.error_data,
+ result.result,
+ ioh->result);
+}
+
+void
+pgaio_io_reopen(PgAioHandle *ioh)
+{
+ Assert(ioh->subject >= 0 && ioh->subject < ASI_COUNT);
+ Assert(ioh->op >= 0 && ioh->op < PGAIO_OP_COUNT);
+
+ aio_subject_info[ioh->subject]->reopen(ioh);
+}
+
+bool
+pgaio_io_needs_synchronously(PgAioHandle *ioh)
+{
+ if (aio_subject_info[ioh->subject]->reopen == NULL)
+ return true;
+
+ return false;
+}
+
+
+
+/* --------------------------------------------------------------------------------
+ * IO Result
+ * --------------------------------------------------------------------------------
+ */
+
+void
+pgaio_result_log(PgAioResult result, const PgAioSubjectData *subject_data, int elevel)
+{
+ const PgAioHandleSharedCallbacks *scb;
+
+ Assert(result.status != ARS_UNKNOWN);
+ Assert(result.status != ARS_OK);
+
+ scb = aio_shared_cbs[result.id];
+
+ if (scb->error == NULL)
+ elog(ERROR, "scb id %d does not have error callback", result.id);
+
+ scb->error(result, subject_data, elevel);
+}
diff --git a/src/backend/storage/aio/meson.build b/src/backend/storage/aio/meson.build
index e13728b73da..8960223194a 100644
--- a/src/backend/storage/aio/meson.build
+++ b/src/backend/storage/aio/meson.build
@@ -2,7 +2,10 @@
backend_sources += files(
'aio.c',
+ 'aio_io.c',
'aio_init.c',
+ 'aio_subject.c',
+ 'method_io_uring.c',
'method_worker.c',
'read_stream.c',
)
diff --git a/src/backend/storage/aio/method_io_uring.c b/src/backend/storage/aio/method_io_uring.c
new file mode 100644
index 00000000000..f76533b4cdc
--- /dev/null
+++ b/src/backend/storage/aio/method_io_uring.c
@@ -0,0 +1,393 @@
+/*-------------------------------------------------------------------------
+ *
+ * method_io_uring.c
+ * AIO implementation using io_uring on Linux
+ *
+ * XXX Write me
+ *
+ * Portions Copyright (c) 1996-2024, PostgreSQL Global Development Group
+ * Portions Copyright (c) 1994, Regents of the University of California
+ *
+ * IDENTIFICATION
+ * src/backend/storage/aio/method_io_uring.c
+ *
+ *-------------------------------------------------------------------------
+ */
+
+#include "postgres.h"
+
+#ifdef USE_LIBURING
+
+#include <liburing.h>
+
+#include "pgstat.h"
+#include "port/pg_iovec.h"
+#include "storage/aio_internal.h"
+#include "storage/fd.h"
+#include "storage/proc.h"
+#include "storage/shmem.h"
+
+
+/* Entry points for IoMethodOps. */
+static size_t pgaio_uring_shmem_size(void);
+static void pgaio_uring_postmaster_init(void);
+static void pgaio_uring_shmem_init(bool first_time);
+static void pgaio_uring_postmaster_child_init(void);
+static void pgaio_uring_postmaster_child_init_local(void);
+
+static int pgaio_uring_submit(void);
+static void pgaio_uring_wait_one(PgAioHandle *ioh, uint64 ref_generation);
+
+static void pgaio_uring_sq_from_io(PgAioHandle *ioh, struct io_uring_sqe *sqe);
+
+
+const IoMethodOps pgaio_uring_ops = {
+ .shmem_size = pgaio_uring_shmem_size,
+ .shmem_init = pgaio_uring_shmem_init,
+ .postmaster_init = pgaio_uring_postmaster_init,
+ .postmaster_child_init = pgaio_uring_postmaster_child_init,
+ .postmaster_child_init_local = pgaio_uring_postmaster_child_init_local,
+
+ .submit = pgaio_uring_submit,
+ .wait_one = pgaio_uring_wait_one,
+#if 0
+ .retry = pgaio_uring_io_retry,
+ .wait_one = pgaio_uring_wait_one,
+ .drain = pgaio_uring_drain,
+#endif
+ .can_scatter_gather_direct = true,
+ .can_scatter_gather_buffered = true
+};
+
+typedef struct PgAioUringContext
+{
+ LWLock completion_lock;
+
+ struct io_uring io_uring_ring;
+ /* XXX: probably worth padding to a cacheline boundary here */
+} PgAioUringContext;
+
+
+static PgAioUringContext *aio_uring_contexts;
+static PgAioUringContext *my_shared_uring_context;
+
+/* io_uring local state */
+static struct io_uring local_ring;
+
+
+
+static Size
+AioContextShmemSize(void)
+{
+ uint32 TotalProcs = MaxBackends + NUM_AUXILIARY_PROCS - MAX_IO_WORKERS;
+
+ return mul_size(TotalProcs, sizeof(PgAioUringContext));
+}
+
+static size_t
+pgaio_uring_shmem_size(void)
+{
+ return AioContextShmemSize();
+}
+
+static void
+pgaio_uring_shmem_init(bool first_time)
+{
+ uint32 TotalProcs = MaxBackends + NUM_AUXILIARY_PROCS - MAX_IO_WORKERS;
+ bool found;
+
+ aio_uring_contexts = (PgAioUringContext *)
+ ShmemInitStruct("AioUring", pgaio_uring_shmem_size(), &found);
+
+ if (found)
+ return;
+
+ for (int contextno = 0; contextno < TotalProcs; contextno++)
+ {
+ PgAioUringContext *context = &aio_uring_contexts[contextno];
+ int ret;
+
+ /*
+ * XXX: Probably worth sharing the WQ between the different rings,
+ * when supported by the kernel. Could also cause additional
+ * contention, I guess?
+ */
+#if 0
+ if (!AcquireExternalFD())
+ elog(ERROR, "No external FD available");
+#endif
+ ret = io_uring_queue_init(io_max_concurrency, &context->io_uring_ring, 0);
+ if (ret < 0)
+ elog(ERROR, "io_uring_queue_init failed: %s", strerror(-ret));
+
+ LWLockInitialize(&context->completion_lock, LWTRANCHE_AIO_URING_COMPLETION);
+ }
+}
+
+static void
+pgaio_uring_postmaster_init(void)
+{
+ uint32 TotalProcs =
+ MaxBackends + NUM_AUXILIARY_PROCS - MAX_IO_WORKERS;
+
+ for (int i = 0; i < TotalProcs; i++)
+ ReserveExternalFD();
+}
+
+static void
+pgaio_uring_postmaster_child_init(void)
+{
+ my_shared_uring_context = &aio_uring_contexts[MyProcNumber];
+}
+
+static void
+pgaio_uring_postmaster_child_init_local(void)
+{
+ int ret;
+
+ ret = io_uring_queue_init(32, &local_ring, 0);
+ if (ret < 0)
+ elog(ERROR, "io_uring_queue_init failed: %s", strerror(-ret));
+}
+
+static int
+pgaio_uring_submit(void)
+{
+ PgAioHandle *ios[PGAIO_SUBMIT_BATCH_SIZE];
+ struct io_uring_sqe *sqe[PGAIO_SUBMIT_BATCH_SIZE];
+ struct io_uring *uring_instance = &my_shared_uring_context->io_uring_ring;
+ int nios = 0;
+
+ while (!dclist_is_empty(&my_aio->staged_ios))
+ {
+ dlist_node *node;
+ PgAioHandle *ioh;
+
+ node = dclist_head_node(&my_aio->staged_ios);
+ ioh = dlist_container(PgAioHandle, node, node);
+
+ sqe[nios] = io_uring_get_sqe(uring_instance);
+ ios[nios] = ioh;
+
+ pgaio_io_prepare_submit(ioh);
+ pgaio_uring_sq_from_io(ios[nios], sqe[nios]);
+
+ nios++;
+
+ if (nios + 1 > PGAIO_SUBMIT_BATCH_SIZE)
+ break;
+ }
+
+ while (true)
+ {
+ int ret;
+
+ pgstat_report_wait_start(WAIT_EVENT_AIO_SUBMIT);
+ ret = io_uring_submit(uring_instance);
+ pgstat_report_wait_end();
+
+ if (ret == -EINTR)
+ {
+ elog(DEBUG3, "submit EINTR, nios: %d", nios);
+ continue;
+ }
+ if (ret < 0)
+ elog(PANIC, "failed: %d/%s",
+ ret, strerror(-ret));
+ else if (ret != nios)
+ {
+ /* likely unreachable, but if it is, we would need to re-submit */
+ elog(PANIC, "submitted only %d of %d",
+ ret, nios);
+ }
+ else
+ {
+ elog(DEBUG3, "submit nios: %d", nios);
+ }
+ break;
+ }
+
+ return nios;
+}
+
+
+#define PGAIO_MAX_LOCAL_REAPED 16
+
+static void
+pgaio_uring_drain_locked(PgAioUringContext *context)
+{
+ int ready;
+ int orig_ready;
+
+ /*
+ * Don't drain more events than available right now. Otherwise it's
+ * plausible that one backend could get stuck, for a while, receiving CQEs
+ * without actually processing them.
+ */
+ orig_ready = ready = io_uring_cq_ready(&context->io_uring_ring);
+
+ while (ready > 0)
+ {
+ struct io_uring_cqe *reaped_cqes[PGAIO_MAX_LOCAL_REAPED];
+ uint32 reaped;
+
+ START_CRIT_SECTION();
+ reaped =
+ io_uring_peek_batch_cqe(&context->io_uring_ring,
+ reaped_cqes,
+ Min(PGAIO_MAX_LOCAL_REAPED, ready));
+ Assert(reaped <= ready);
+
+ ready -= reaped;
+
+ for (int i = 0; i < reaped; i++)
+ {
+ struct io_uring_cqe *cqe = reaped_cqes[i];
+ PgAioHandle *ioh;
+
+ ioh = io_uring_cqe_get_data(cqe);
+ io_uring_cqe_seen(&context->io_uring_ring, cqe);
+
+ pgaio_io_process_completion(ioh, cqe->res);
+ }
+
+ END_CRIT_SECTION();
+
+ ereport(DEBUG3,
+ errmsg("drained %d/%d, now expecting %d",
+ reaped, orig_ready, io_uring_cq_ready(&context->io_uring_ring)),
+ errhidestmt(true),
+ errhidecontext(true));
+
+ }
+}
+
+static void
+pgaio_uring_wait_one(PgAioHandle *ioh, uint64 ref_generation)
+{
+ PgAioHandleState state;
+ ProcNumber owner_procno = ioh->owner_procno;
+ PgAioUringContext *owner_context = &aio_uring_contexts[owner_procno];
+ bool expect_cqe;
+ int waited = 0;
+
+ /*
+ * We ought to have a smarter locking scheme, nearly all the time the
+ * backend owning the ring will reap the completions, making the locking
+ * unnecessarily expensive.
+ */
+ LWLockAcquire(&owner_context->completion_lock, LW_EXCLUSIVE);
+
+ while (true)
+ {
+ ereport(DEBUG3,
+ errmsg("wait_one for io:%d in state %s, cycle %d",
+ pgaio_io_get_id(ioh), pgaio_io_get_state_name(ioh), waited),
+ errhidestmt(true),
+ errhidecontext(true));
+
+ if (pgaio_io_was_recycled(ioh, ref_generation, &state) ||
+ state != AHS_IN_FLIGHT)
+ {
+ break;
+ }
+ else if (io_uring_cq_ready(&owner_context->io_uring_ring))
+ {
+ expect_cqe = true;
+ }
+ else
+ {
+ int ret;
+ struct io_uring_cqe *cqes;
+
+ pgstat_report_wait_start(WAIT_EVENT_AIO_DRAIN);
+ ret = io_uring_wait_cqes(&owner_context->io_uring_ring, &cqes, 1, NULL, NULL);
+ pgstat_report_wait_end();
+
+ if (ret == -EINTR)
+ {
+ continue;
+ }
+ else if (ret != 0)
+ {
+ elog(PANIC, "unexpected: %d/%s: %m", ret, strerror(-ret));
+ }
+ else
+ {
+ Assert(cqes != NULL);
+ expect_cqe = true;
+ waited++;
+ }
+ }
+
+ if (expect_cqe)
+ {
+ pgaio_uring_drain_locked(owner_context);
+ }
+ }
+
+ LWLockRelease(&owner_context->completion_lock);
+
+ ereport(DEBUG3,
+ errmsg("wait_one with %d sleeps",
+ waited),
+ errhidestmt(true),
+ errhidecontext(true));
+}
+
+static void
+pgaio_uring_sq_from_io(PgAioHandle *ioh, struct io_uring_sqe *sqe)
+{
+ struct iovec *iov;
+
+ switch (ioh->op)
+ {
+ case PGAIO_OP_READ:
+ iov = &aio_ctl->iovecs[ioh->iovec_off];
+ if (ioh->op_data.read.iov_length == 1)
+ {
+ io_uring_prep_read(sqe,
+ ioh->op_data.read.fd,
+ iov->iov_base,
+ iov->iov_len,
+ ioh->op_data.read.offset);
+ }
+ else
+ {
+ io_uring_prep_readv(sqe,
+ ioh->op_data.read.fd,
+ iov,
+ ioh->op_data.read.iov_length,
+ ioh->op_data.read.offset);
+
+ }
+ break;
+
+ case PGAIO_OP_WRITE:
+ iov = &aio_ctl->iovecs[ioh->iovec_off];
+ if (ioh->op_data.write.iov_length == 1)
+ {
+ io_uring_prep_write(sqe,
+ ioh->op_data.write.fd,
+ iov->iov_base,
+ iov->iov_len,
+ ioh->op_data.write.offset);
+ }
+ else
+ {
+ io_uring_prep_writev(sqe,
+ ioh->op_data.write.fd,
+ iov,
+ ioh->op_data.write.iov_length,
+ ioh->op_data.write.offset);
+ }
+ break;
+
+ default:
+ elog(ERROR, "not implemented");
+ }
+
+ io_uring_sqe_set_data(sqe, ioh);
+}
+
+#endif /* USE_LIBURING */
diff --git a/src/backend/storage/aio/method_worker.c b/src/backend/storage/aio/method_worker.c
index 5df2eea4a03..cd79bf1fba6 100644
--- a/src/backend/storage/aio/method_worker.c
+++ b/src/backend/storage/aio/method_worker.c
@@ -3,6 +3,21 @@
* method_worker.c
* AIO implementation using workers
*
+ * Worker processes consume IOs from a shared memory submission queue, run
+ * traditional synchronous system calls, and perform the shared completion
+ * handling immediately. Client code submits most requests by pushing IOs
+ * into the submission queue, and waits (if necessary) using condition
+ * variables. Some IOs cannot be performed in another process due to lack of
+ * infrastructure for reopening the file, and must processed synchronously by
+ * the client code when submitted.
+ *
+ * So that the submitter can make just one system call when submitting a batch
+ * of IOs, wakeups "fan out"; each woken backend can wake two more. XXX This
+ * could be improved by using futexes instead of latches to wake N waiters.
+ *
+ * This method of AIO is available in all builds on all operating systems, and
+ * is the default.
+ *
* Portions Copyright (c) 1996-2021, PostgreSQL Global Development Group
* Portions Copyright (c) 1994, Regents of the University of California
*
@@ -16,24 +31,299 @@
#include "libpq/pqsignal.h"
#include "miscadmin.h"
+#include "port/pg_bitutils.h"
+#include "postmaster/auxprocess.h"
#include "postmaster/interrupt.h"
+#include "storage/aio.h"
+#include "storage/aio_internal.h"
#include "storage/io_worker.h"
#include "storage/ipc.h"
#include "storage/latch.h"
#include "storage/proc.h"
#include "tcop/tcopprot.h"
#include "utils/wait_event.h"
+#include "utils/ps_status.h"
+
+
+/* How many workers should each worker wake up if needed? */
+#define IO_WORKER_WAKEUP_FANOUT 2
+
+
+typedef struct AioWorkerSubmissionQueue
+{
+ uint32 size;
+ uint32 mask;
+ uint32 head;
+ uint32 tail;
+ uint32 ios[FLEXIBLE_ARRAY_MEMBER];
+} AioWorkerSubmissionQueue;
+
+typedef struct AioWorkerSlot
+{
+ Latch *latch;
+ bool in_use;
+} AioWorkerSlot;
+
+typedef struct AioWorkerControl
+{
+ uint64 idle_worker_mask;
+ AioWorkerSlot workers[FLEXIBLE_ARRAY_MEMBER];
+} AioWorkerControl;
+
+
+static size_t pgaio_worker_shmem_size(void);
+static void pgaio_worker_shmem_init(bool first_time);
+static void pgaio_worker_postmaster_child_init_local(void);
+
+static int pgaio_worker_submit(void);
+
+
+const IoMethodOps pgaio_worker_ops = {
+ .shmem_size = pgaio_worker_shmem_size,
+ .shmem_init = pgaio_worker_shmem_init,
+ .postmaster_child_init_local = pgaio_worker_postmaster_child_init_local,
+ .submit = pgaio_worker_submit,
+#if 0
+ .wait_one = pgaio_worker_wait_one,
+ .retry = pgaio_worker_io_retry,
+ .drain = pgaio_worker_drain,
+#endif
+
+ .can_scatter_gather_direct = true,
+ .can_scatter_gather_buffered = true
+};
int io_workers = 3;
+static int io_worker_queue_size = 64;
+static int MyIoWorkerId;
+
+
+static AioWorkerSubmissionQueue *io_worker_submission_queue;
+static AioWorkerControl *io_worker_control;
+
+
+static size_t
+pgaio_worker_shmem_size(void)
+{
+ return
+ offsetof(AioWorkerSubmissionQueue, ios) +
+ sizeof(uint32) * io_worker_queue_size +
+ offsetof(AioWorkerControl, workers) +
+ sizeof(AioWorkerSlot) * io_workers;
+}
+
+static void
+pgaio_worker_shmem_init(bool first_time)
+{
+ bool found;
+ int size;
+
+ /* Round size up to next power of two so we can make a mask. */
+ size = pg_nextpower2_32(io_worker_queue_size);
+
+ io_worker_submission_queue =
+ ShmemInitStruct("AioWorkerSubmissionQueue",
+ offsetof(AioWorkerSubmissionQueue, ios) +
+ sizeof(uint32) * size,
+ &found);
+ if (!found)
+ {
+ io_worker_submission_queue->size = size;
+ io_worker_submission_queue->head = 0;
+ io_worker_submission_queue->tail = 0;
+ }
+
+ io_worker_control =
+ ShmemInitStruct("AioWorkerControl",
+ offsetof(AioWorkerControl, workers) +
+ sizeof(AioWorkerSlot) * io_workers,
+ &found);
+ if (!found)
+ {
+ io_worker_control->idle_worker_mask = 0;
+ for (int i = 0; i < io_workers; ++i)
+ {
+ io_worker_control->workers[i].latch = NULL;
+ io_worker_control->workers[i].in_use = false;
+ }
+ }
+}
+
+static void
+pgaio_worker_postmaster_child_init_local(void)
+{
+}
+
+
+static int
+pgaio_choose_idle_worker(void)
+{
+ int worker;
+
+ if (io_worker_control->idle_worker_mask == 0)
+ return -1;
+
+ /* Find the lowest bit position, and clear it. */
+ worker = pg_rightmost_one_pos64(io_worker_control->idle_worker_mask);
+ io_worker_control->idle_worker_mask &= ~(UINT64_C(1) << worker);
+
+ return worker;
+}
+
+static bool
+pgaio_worker_submission_queue_insert(PgAioHandle *ioh)
+{
+ AioWorkerSubmissionQueue *queue;
+ uint32 new_head;
+
+ queue = io_worker_submission_queue;
+ new_head = (queue->head + 1) & (queue->size - 1);
+ if (new_head == queue->tail)
+ {
+ elog(DEBUG1, "full");
+ return false; /* full */
+ }
+
+ queue->ios[queue->head] = pgaio_io_get_id(ioh);
+ queue->head = new_head;
+
+ return true;
+}
+
+static uint32
+pgaio_worker_submission_queue_consume(void)
+{
+ AioWorkerSubmissionQueue *queue;
+ uint32 result;
+
+ queue = io_worker_submission_queue;
+ if (queue->tail == queue->head)
+ return UINT32_MAX; /* empty */
+
+ result = queue->ios[queue->tail];
+ queue->tail = (queue->tail + 1) & (queue->size - 1);
+
+ return result;
+}
+
+static uint32
+pgaio_worker_submission_queue_depth(void)
+{
+ uint32 head;
+ uint32 tail;
+
+ head = io_worker_submission_queue->head;
+ tail = io_worker_submission_queue->tail;
+
+ if (tail > head)
+ head += io_worker_submission_queue->size;
+
+ Assert(head >= tail);
+
+ return head - tail;
+}
+
+static bool
+pgaio_worker_need_synchronous(PgAioHandle *ioh)
+{
+ return
+ !IsUnderPostmaster
+ || ioh->flags & AHF_REFERENCES_LOCAL
+ || pgaio_io_needs_synchronously(ioh);
+}
+
+static void
+pgaio_worker_submit_internal(PgAioHandle *ios[], int nios)
+{
+ PgAioHandle *synchronous_ios[PGAIO_SUBMIT_BATCH_SIZE];
+ int nsync = 0;
+ Latch *wakeup = NULL;
+ int worker;
+
+ Assert(nios <= PGAIO_SUBMIT_BATCH_SIZE);
+
+ LWLockAcquire(AioWorkerSubmissionQueueLock, LW_EXCLUSIVE);
+ for (int i = 0; i < nios; ++i)
+ {
+ if (pgaio_worker_need_synchronous(ios[i]) ||
+ !pgaio_worker_submission_queue_insert(ios[i]))
+ {
+ /*
+ * We'll do it synchronously, but only after we've sent as many as
+ * we can to workers, to maximize concurrency.
+ */
+ synchronous_ios[nsync++] = ios[i];
+ continue;
+ }
+
+ if (wakeup == NULL)
+ {
+ /* Choose an idle worker to wake up if we haven't already. */
+ worker = pgaio_choose_idle_worker();
+ if (worker >= 0)
+ wakeup = io_worker_control->workers[worker].latch;
+
+ ereport(DEBUG3,
+ errmsg("submission for io:%d choosing worker %d, latch %p",
+ pgaio_io_get_id(ios[i]), worker, wakeup),
+ errhidestmt(true), errhidecontext(true));
+ }
+ }
+ LWLockRelease(AioWorkerSubmissionQueueLock);
+
+ if (wakeup)
+ SetLatch(wakeup);
+
+ /* Run whatever is left synchronously. */
+ if (nsync > 0)
+ {
+ for (int i = 0; i < nsync; ++i)
+ {
+ pgaio_io_perform_synchronously(synchronous_ios[i]);
+ }
+ }
+}
+
+static int
+pgaio_worker_submit(void)
+{
+ PgAioHandle *ios[PGAIO_SUBMIT_BATCH_SIZE];
+ int nios = 0;
+
+ while (!dclist_is_empty(&my_aio->staged_ios))
+ {
+ dlist_node *node;
+ PgAioHandle *ioh;
+
+ node = dclist_head_node(&my_aio->staged_ios);
+ ioh = dlist_container(PgAioHandle, node, node);
+
+ pgaio_io_prepare_submit(ioh);
+
+ Assert(nios < PGAIO_SUBMIT_BATCH_SIZE);
+
+ ios[nios++] = ioh;
+
+ if (nios + 1 == PGAIO_SUBMIT_BATCH_SIZE)
+ break;
+ }
+
+ pgaio_worker_submit_internal(ios, nios);
+
+ return nios;
+}
void
IoWorkerMain(char *startup_data, size_t startup_data_len)
{
sigjmp_buf local_sigjmp_buf;
+ volatile PgAioHandle *ioh = NULL;
+ char cmd[128];
MyBackendType = B_IO_WORKER;
+ AuxiliaryProcessMainCommon();
/* TODO review all signals */
pqsignal(SIGHUP, SignalHandlerForConfigReload);
@@ -49,7 +339,34 @@ IoWorkerMain(char *startup_data, size_t startup_data_len)
pqsignal(SIGPIPE, SIG_IGN);
pqsignal(SIGUSR1, procsignal_sigusr1_handler);
pqsignal(SIGUSR2, SignalHandlerForShutdownRequest);
- sigprocmask(SIG_SETMASK, &UnBlockSig, NULL);
+
+ /* FIXME: locking */
+ MyIoWorkerId = -1;
+
+ LWLockAcquire(AioWorkerSubmissionQueueLock, LW_EXCLUSIVE);
+
+ for (int i = 0; i < io_workers; ++i)
+ {
+ if (!io_worker_control->workers[i].in_use)
+ {
+ Assert(io_worker_control->workers[i].latch == NULL);
+ io_worker_control->workers[i].in_use = true;
+ MyIoWorkerId = i;
+ break;
+ }
+ else
+ Assert(io_worker_control->workers[i].latch != NULL);
+ }
+
+ if (MyIoWorkerId == -1)
+ elog(ERROR, "couldn't find a free worker slot");
+
+ io_worker_control->idle_worker_mask |= (UINT64_C(1) << MyIoWorkerId);
+ io_worker_control->workers[MyIoWorkerId].latch = MyLatch;
+ LWLockRelease(AioWorkerSubmissionQueueLock);
+
+ sprintf(cmd, "worker: %d", MyIoWorkerId);
+ set_ps_display(cmd);
/* see PostgresMain() */
if (sigsetjmp(local_sigjmp_buf, 1) != 0)
@@ -64,21 +381,107 @@ IoWorkerMain(char *startup_data, size_t startup_data_len)
LWLockReleaseAll();
/* TODO: recover from IO errors */
+ if (ioh != NULL)
+ {
+#if 0
+ /* EINTR is treated as a retryable error */
+ pgaio_process_io_completion(unvolatize(PgAioInProgress *, io),
+ EINTR);
+#endif
+ }
EmitErrorReport();
+
+ /* FIXME: should probably be a before-shmem-exit instead */
+ LWLockAcquire(AioWorkerSubmissionQueueLock, LW_EXCLUSIVE);
+ Assert(io_worker_control->workers[MyIoWorkerId].in_use);
+ Assert(io_worker_control->workers[MyIoWorkerId].latch == MyLatch);
+
+ io_worker_control->workers[MyIoWorkerId].in_use = false;
+ io_worker_control->workers[MyIoWorkerId].latch = NULL;
+ LWLockRelease(AioWorkerSubmissionQueueLock);
+
proc_exit(1);
}
/* We can now handle ereport(ERROR) */
PG_exception_stack = &local_sigjmp_buf;
+ sigprocmask(SIG_SETMASK, &UnBlockSig, NULL);
+
while (!ShutdownRequestPending)
{
- WaitLatch(MyLatch, WL_LATCH_SET | WL_EXIT_ON_PM_DEATH, -1,
- WAIT_EVENT_IO_WORKER_MAIN);
- ResetLatch(MyLatch);
- CHECK_FOR_INTERRUPTS();
+ uint32 io_index;
+ Latch *latches[IO_WORKER_WAKEUP_FANOUT];
+ int nlatches = 0;
+ int nwakeups = 0;
+ int worker;
+
+ /* Try to get a job to do. */
+ LWLockAcquire(AioWorkerSubmissionQueueLock, LW_EXCLUSIVE);
+ if ((io_index = pgaio_worker_submission_queue_consume()) == UINT32_MAX)
+ {
+ /* Nothing to do. Mark self idle. */
+ /*
+ * XXX: Invent some kind of back pressure to reduce useless
+ * wakeups?
+ */
+ io_worker_control->idle_worker_mask |= (UINT64_C(1) << MyIoWorkerId);
+ }
+ else
+ {
+ /* Got one. Clear idle flag. */
+ io_worker_control->idle_worker_mask &= ~(UINT64_C(1) << MyIoWorkerId);
+
+ /* See if we can wake up some peers. */
+ nwakeups = Min(pgaio_worker_submission_queue_depth(),
+ IO_WORKER_WAKEUP_FANOUT);
+ for (int i = 0; i < nwakeups; ++i)
+ {
+ if ((worker = pgaio_choose_idle_worker()) < 0)
+ break;
+ latches[nlatches++] = io_worker_control->workers[worker].latch;
+ }
+#if 0
+ if (nwakeups > 0)
+ elog(LOG, "wake %d", nwakeups);
+#endif
+ }
+ LWLockRelease(AioWorkerSubmissionQueueLock);
+
+ for (int i = 0; i < nlatches; ++i)
+ SetLatch(latches[i]);
+
+ if (io_index != UINT32_MAX)
+ {
+ ioh = &aio_ctl->io_handles[io_index];
+
+ ereport(DEBUG3,
+ errmsg("worker processing io:%d",
+ pgaio_io_get_id(unvolatize(PgAioHandle *, ioh))),
+ errhidestmt(true), errhidecontext(true));
+
+ pgaio_io_reopen(unvolatize(PgAioHandle *, ioh));
+ pgaio_io_perform_synchronously(unvolatize(PgAioHandle *, ioh));
+
+ ioh = NULL;
+ }
+ else
+ {
+ WaitLatch(MyLatch, WL_LATCH_SET | WL_EXIT_ON_PM_DEATH, -1,
+ WAIT_EVENT_IO_WORKER_MAIN);
+ ResetLatch(MyLatch);
+ CHECK_FOR_INTERRUPTS();
+ }
}
+ LWLockAcquire(AioWorkerSubmissionQueueLock, LW_EXCLUSIVE);
+ Assert(io_worker_control->workers[MyIoWorkerId].in_use);
+ Assert(io_worker_control->workers[MyIoWorkerId].latch == MyLatch);
+
+ io_worker_control->workers[MyIoWorkerId].in_use = false;
+ io_worker_control->workers[MyIoWorkerId].latch = NULL;
+ LWLockRelease(AioWorkerSubmissionQueueLock);
+
proc_exit(0);
}
diff --git a/src/backend/storage/lmgr/lwlock.c b/src/backend/storage/lmgr/lwlock.c
index f3d3435b1f5..63d1f905554 100644
--- a/src/backend/storage/lmgr/lwlock.c
+++ b/src/backend/storage/lmgr/lwlock.c
@@ -166,6 +166,7 @@ static const char *const BuiltinTrancheNames[] = {
[LWTRANCHE_SUBTRANS_SLRU] = "SubtransSLRU",
[LWTRANCHE_XACT_SLRU] = "XactSLRU",
[LWTRANCHE_PARALLEL_VACUUM_DSA] = "ParallelVacuumDSA",
+ [LWTRANCHE_AIO_URING_COMPLETION] = "AioUringCompletion",
};
StaticAssertDecl(lengthof(BuiltinTrancheNames) ==
diff --git a/src/backend/utils/activity/wait_event_names.txt b/src/backend/utils/activity/wait_event_names.txt
index 47a2c4d126b..3678f2b3e43 100644
--- a/src/backend/utils/activity/wait_event_names.txt
+++ b/src/backend/utils/activity/wait_event_names.txt
@@ -192,6 +192,9 @@ ABI_compatibility:
Section: ClassName - WaitEventIO
+AIO_SUBMIT "Waiting for AIO submission."
+AIO_DRAIN "Waiting for IOs to finish."
+AIO_COMPLETION "Waiting for completion callback."
BASEBACKUP_READ "Waiting for base backup to read from a file."
BASEBACKUP_SYNC "Waiting for data written by a base backup to reach durable storage."
BASEBACKUP_WRITE "Waiting for base backup to write to a file."
@@ -348,6 +351,7 @@ DSMRegistry "Waiting to read or update the dynamic shared memory registry."
InjectionPoint "Waiting to read or update information related to injection points."
SerialControl "Waiting to read or update shared <filename>pg_serial</filename> state."
WaitLSN "Waiting to read or update shared Wait-for-LSN state."
+AioWorkerSubmissionQueue "Waiting to access AIO worker submission queue."
#
# END OF PREDEFINED LWLOCKS (DO NOT CHANGE THIS LINE)
diff --git a/src/backend/utils/misc/guc_tables.c b/src/backend/utils/misc/guc_tables.c
index 5670f40478a..5828072a48e 100644
--- a/src/backend/utils/misc/guc_tables.c
+++ b/src/backend/utils/misc/guc_tables.c
@@ -3214,6 +3214,31 @@ struct config_int ConfigureNamesInt[] =
NULL, assign_io_workers, NULL
},
+ {
+ {"io_max_concurrency",
+ PGC_POSTMASTER,
+ RESOURCES_ASYNCHRONOUS,
+ gettext_noop("Number of IOs that may be in flight in one backend."),
+ NULL,
+ },
+ &io_max_concurrency,
+ -1, -1, 1024,
+ NULL, NULL, NULL
+ },
+
+ {
+ {"io_bounce_buffers",
+ PGC_POSTMASTER,
+ RESOURCES_ASYNCHRONOUS,
+ gettext_noop("Number of IO Bounce Buffers reserved for each backend."),
+ NULL,
+ GUC_UNIT_BLOCKS
+ },
+ &io_bounce_buffers,
+ -1, -1, 4096,
+ NULL, NULL, NULL
+ },
+
{
{"backend_flush_after", PGC_USERSET, RESOURCES_ASYNCHRONOUS,
gettext_noop("Number of pages after which previously performed writes are flushed to disk."),
diff --git a/src/backend/utils/misc/postgresql.conf.sample b/src/backend/utils/misc/postgresql.conf.sample
index 90430381efa..1fc8336496c 100644
--- a/src/backend/utils/misc/postgresql.conf.sample
+++ b/src/backend/utils/misc/postgresql.conf.sample
@@ -842,6 +842,12 @@
#io_method = worker # (change requires restart)
#io_workers = 3 # 1-32;
+#io_max_concurrency = 32 # Max number of IOs that may be in
+ # flight at the same time in one backend
+ # (change requires restart)
+#io_bounce_buffers = -1 # -1 sets based on shared_buffers
+ # (change requires restart)
+
#------------------------------------------------------------------------------
# CUSTOMIZED OPTIONS
diff --git a/src/backend/utils/resowner/resowner.c b/src/backend/utils/resowner/resowner.c
index 505534ee8d3..d1932b7393c 100644
--- a/src/backend/utils/resowner/resowner.c
+++ b/src/backend/utils/resowner/resowner.c
@@ -47,6 +47,8 @@
#include "common/hashfn.h"
#include "common/int.h"
+#include "lib/ilist.h"
+#include "storage/aio.h"
#include "storage/ipc.h"
#include "storage/predicate.h"
#include "storage/proc.h"
@@ -155,6 +157,13 @@ struct ResourceOwnerData
/* The local locks cache. */
LOCALLOCK *locks[MAX_RESOWNER_LOCKS]; /* list of owned locks */
+
+ /*
+ * AIO handles & bounce buffers need be registered in critical sections
+ * and therefore cannot use the normal ResoureElem mechanism.
+ */
+ dlist_head aio_handles;
+ dlist_head aio_bounce_buffers;
};
@@ -425,6 +434,9 @@ ResourceOwnerCreate(ResourceOwner parent, const char *name)
parent->firstchild = owner;
}
+ dlist_init(&owner->aio_handles);
+ dlist_init(&owner->aio_bounce_buffers);
+
return owner;
}
@@ -725,6 +737,21 @@ ResourceOwnerReleaseInternal(ResourceOwner owner,
* so issue warnings. In the abort case, just clean up quietly.
*/
ResourceOwnerReleaseAll(owner, phase, isCommit);
+
+ /* XXX: Could probably be a later phase? */
+ while (!dlist_is_empty(&owner->aio_handles))
+ {
+ dlist_node *node = dlist_head_node(&owner->aio_handles);
+
+ pgaio_io_release_resowner(node, !isCommit);
+ }
+
+ while (!dlist_is_empty(&owner->aio_bounce_buffers))
+ {
+ dlist_node *node = dlist_head_node(&owner->aio_bounce_buffers);
+
+ pgaio_bounce_buffer_release_resowner(node, !isCommit);
+ }
}
else if (phase == RESOURCE_RELEASE_LOCKS)
{
@@ -1082,3 +1109,27 @@ ResourceOwnerForgetLock(ResourceOwner owner, LOCALLOCK *locallock)
elog(ERROR, "lock reference %p is not owned by resource owner %s",
locallock, owner->name);
}
+
+void
+ResourceOwnerRememberAioHandle(ResourceOwner owner, struct dlist_node *ioh_node)
+{
+ dlist_push_tail(&owner->aio_handles, ioh_node);
+}
+
+void
+ResourceOwnerForgetAioHandle(ResourceOwner owner, struct dlist_node *ioh_node)
+{
+ dlist_delete_from(&owner->aio_handles, ioh_node);
+}
+
+void
+ResourceOwnerRememberAioBounceBuffer(ResourceOwner owner, struct dlist_node *ioh_node)
+{
+ dlist_push_tail(&owner->aio_bounce_buffers, ioh_node);
+}
+
+void
+ResourceOwnerForgetAioBounceBuffer(ResourceOwner owner, struct dlist_node *ioh_node)
+{
+ dlist_delete_from(&owner->aio_bounce_buffers, ioh_node);
+}
diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list
index 309686627e7..be8be9fbff0 100644
--- a/src/tools/pgindent/typedefs.list
+++ b/src/tools/pgindent/typedefs.list
@@ -54,6 +54,9 @@ AggStrategy
AggTransInfo
Aggref
AggregateInstrumentation
+AioWorkerControl
+AioWorkerSlot
+AioWorkerSubmissionQueue
AlenState
Alias
AllocBlock
@@ -1258,6 +1261,7 @@ IntoClause
InvalMessageArray
InvalidationMsgsGroup
IoMethod
+IoMethodOps
IpcMemoryId
IpcMemoryKey
IpcMemoryState
@@ -2093,6 +2097,25 @@ Permutation
PermutationStep
PermutationStepBlocker
PermutationStepBlockerType
+PgAioBounceBuffer
+PgAioCtl
+PgAioHandle
+PgAioHandleFlags
+PgAioHandleRef
+PgAioHandleState
+PgAioHandleSharedCallbacks
+PgAioHandleSharedCallbackID
+PgAioHandleSharedCallbacks
+PgAioOp
+PgAioOpData
+PgAioPerBackend
+PgAioResultStatus
+PgAioResult
+PgAioReturn
+PgAioSubjectData
+PgAioSubjectID
+PgAioSubjectInfo
+PgAioUringContext
PgArchData
PgBackendGSSStatus
PgBackendSSLStatus
--
2.45.2.827.g557ae147e6
v2.0-0010-aio-Implement-smgr-md.c-aio-methods.patch (text/x-diff, 22.6K) download
From 03723ac0d170aba51febc975921296d814af7765 Mon Sep 17 00:00:00 2001
From: Andres Freund <andres@anarazel.de>
Date: Sat, 31 Aug 2024 22:33:30 -0400
Subject: [PATCH v2.0 10/17] aio: Implement smgr/md.c aio methods
---
src/include/storage/aio.h | 17 +-
src/include/storage/fd.h | 6 +
src/include/storage/md.h | 12 ++
src/include/storage/smgr.h | 21 +++
src/backend/storage/aio/aio_subject.c | 4 +
src/backend/storage/file/fd.c | 68 ++++++++
src/backend/storage/smgr/md.c | 217 ++++++++++++++++++++++++++
src/backend/storage/smgr/smgr.c | 91 +++++++++++
8 files changed, 434 insertions(+), 2 deletions(-)
diff --git a/src/include/storage/aio.h b/src/include/storage/aio.h
index 65052462b02..acfd50c587c 100644
--- a/src/include/storage/aio.h
+++ b/src/include/storage/aio.h
@@ -57,9 +57,10 @@ typedef enum PgAioSubjectID
{
/* intentionally the zero value, to help catch zeroed memory etc */
ASI_INVALID = 0,
+ ASI_SMGR,
} PgAioSubjectID;
-#define ASI_COUNT (ASI_INVALID + 1)
+#define ASI_COUNT (ASI_SMGR + 1)
/*
* Flags for an IO that can be set with pgaio_io_set_flag().
@@ -90,7 +91,8 @@ typedef enum PgAioHandleFlags
*/
typedef enum PgAioHandleSharedCallbackID
{
- ASC_PLACEHOLDER /* empty enums are invalid */ ,
+ ASC_MD_READV,
+ ASC_MD_WRITEV,
} PgAioHandleSharedCallbackID;
@@ -139,6 +141,17 @@ typedef union
typedef union PgAioSubjectData
{
+ struct
+ {
+ RelFileLocator rlocator; /* physical relation identifier */
+ BlockNumber blockNum; /* blknum relative to begin of reln */
+ int nblocks;
+ ForkNumber forkNum:8; /* don't waste 4 byte for four values */
+ bool is_temp; /* proc can be inferred by owning AIO */
+ bool release_lock;
+ int8 mode;
+ } smgr;
+
/* just as an example placeholder for later */
struct
{
diff --git a/src/include/storage/fd.h b/src/include/storage/fd.h
index 1456ab383a4..e993e1b671f 100644
--- a/src/include/storage/fd.h
+++ b/src/include/storage/fd.h
@@ -101,6 +101,8 @@ extern PGDLLIMPORT int max_safe_fds;
* prototypes for functions in fd.c
*/
+struct PgAioHandle;
+
/* Operations on virtual Files --- equivalent to Unix kernel file ops */
extern File PathNameOpenFile(const char *fileName, int fileFlags);
extern File PathNameOpenFilePerm(const char *fileName, int fileFlags, mode_t fileMode);
@@ -109,6 +111,10 @@ extern void FileClose(File file);
extern int FilePrefetch(File file, off_t offset, off_t amount, uint32 wait_event_info);
extern ssize_t FileReadV(File file, const struct iovec *iov, int iovcnt, off_t offset, uint32 wait_event_info);
extern ssize_t FileWriteV(File file, const struct iovec *iov, int iovcnt, off_t offset, uint32 wait_event_info);
+extern ssize_t FileReadV(File file, const struct iovec *iov, int iovcnt, off_t offset, uint32 wait_event_info);
+extern int FileStartReadV(struct PgAioHandle *ioh, File file, int iovcnt, off_t offset, uint32 wait_event_info);
+extern ssize_t FileWriteV(File file, const struct iovec *iov, int iovcnt, off_t offset, uint32 wait_event_info);
+extern int FileStartWriteV(struct PgAioHandle *ioh, File file, int iovcnt, off_t offset, uint32 wait_event_info);
extern int FileSync(File file, uint32 wait_event_info);
extern int FileZero(File file, off_t offset, off_t amount, uint32 wait_event_info);
extern int FileFallocate(File file, off_t offset, off_t amount, uint32 wait_event_info);
diff --git a/src/include/storage/md.h b/src/include/storage/md.h
index b72293c79a5..ede77695853 100644
--- a/src/include/storage/md.h
+++ b/src/include/storage/md.h
@@ -19,6 +19,10 @@
#include "storage/smgr.h"
#include "storage/sync.h"
+struct PgAioHandleSharedCallbacks;
+extern const struct PgAioHandleSharedCallbacks aio_md_readv_cb;
+extern const struct PgAioHandleSharedCallbacks aio_md_writev_cb;
+
/* md storage manager functionality */
extern void mdinit(void);
extern void mdopen(SMgrRelation reln);
@@ -36,9 +40,16 @@ extern uint32 mdmaxcombine(SMgrRelation reln, ForkNumber forknum,
BlockNumber blocknum);
extern void mdreadv(SMgrRelation reln, ForkNumber forknum, BlockNumber blocknum,
void **buffers, BlockNumber nblocks);
+extern void mdstartreadv(struct PgAioHandle *ioh,
+ SMgrRelation reln, ForkNumber forknum, BlockNumber blocknum,
+ void **buffers, BlockNumber nblocks);
extern void mdwritev(SMgrRelation reln, ForkNumber forknum,
BlockNumber blocknum,
const void **buffers, BlockNumber nblocks, bool skipFsync);
+extern void mdstartwritev(struct PgAioHandle *ioh,
+ SMgrRelation reln, ForkNumber forknum,
+ BlockNumber blocknum,
+ const void **buffers, BlockNumber nblocks, bool skipFsync);
extern void mdwriteback(SMgrRelation reln, ForkNumber forknum,
BlockNumber blocknum, BlockNumber nblocks);
extern BlockNumber mdnblocks(SMgrRelation reln, ForkNumber forknum);
@@ -46,6 +57,7 @@ extern void mdtruncate(SMgrRelation reln, ForkNumber forknum,
BlockNumber nblocks);
extern void mdimmedsync(SMgrRelation reln, ForkNumber forknum);
extern void mdregistersync(SMgrRelation reln, ForkNumber forknum);
+extern int mdfd(SMgrRelation reln, ForkNumber forknum, BlockNumber blocknum, uint32 *off);
extern void ForgetDatabaseSyncRequests(Oid dbid);
extern void DropRelationFiles(RelFileLocator *delrels, int ndelrels, bool isRedo);
diff --git a/src/include/storage/smgr.h b/src/include/storage/smgr.h
index 899d0d681c5..66730bc24fa 100644
--- a/src/include/storage/smgr.h
+++ b/src/include/storage/smgr.h
@@ -73,6 +73,11 @@ typedef SMgrRelationData *SMgrRelation;
#define SmgrIsTemp(smgr) \
RelFileLocatorBackendIsTemp((smgr)->smgr_rlocator)
+struct PgAioHandle;
+struct PgAioSubjectInfo;
+
+extern const struct PgAioSubjectInfo aio_smgr_subject_info;
+
extern void smgrinit(void);
extern SMgrRelation smgropen(RelFileLocator rlocator, ProcNumber backend);
extern bool smgrexists(SMgrRelation reln, ForkNumber forknum);
@@ -97,10 +102,19 @@ extern uint32 smgrmaxcombine(SMgrRelation reln, ForkNumber forknum,
extern void smgrreadv(SMgrRelation reln, ForkNumber forknum,
BlockNumber blocknum,
void **buffers, BlockNumber nblocks);
+extern void smgrstartreadv(struct PgAioHandle *ioh,
+ SMgrRelation reln, ForkNumber forknum,
+ BlockNumber blocknum,
+ void **buffers, BlockNumber nblocks);
extern void smgrwritev(SMgrRelation reln, ForkNumber forknum,
BlockNumber blocknum,
const void **buffers, BlockNumber nblocks,
bool skipFsync);
+extern void smgrstartwritev(struct PgAioHandle *ioh,
+ SMgrRelation reln, ForkNumber forknum,
+ BlockNumber blocknum,
+ const void **buffers, BlockNumber nblocks,
+ bool skipFsync);
extern void smgrwriteback(SMgrRelation reln, ForkNumber forknum,
BlockNumber blocknum, BlockNumber nblocks);
extern BlockNumber smgrnblocks(SMgrRelation reln, ForkNumber forknum);
@@ -109,6 +123,7 @@ extern void smgrtruncate(SMgrRelation reln, ForkNumber *forknum,
int nforks, BlockNumber *nblocks);
extern void smgrimmedsync(SMgrRelation reln, ForkNumber forknum);
extern void smgrregistersync(SMgrRelation reln, ForkNumber forknum);
+extern int smgrfd(SMgrRelation reln, ForkNumber forknum, BlockNumber blocknum, uint32 *off);
extern void AtEOXact_SMgr(void);
extern bool ProcessBarrierSmgrRelease(void);
@@ -126,4 +141,10 @@ smgrwrite(SMgrRelation reln, ForkNumber forknum, BlockNumber blocknum,
smgrwritev(reln, forknum, blocknum, &buffer, 1, skipFsync);
}
+extern void pgaio_io_set_subject_smgr(struct PgAioHandle *ioh,
+ SMgrRelationData *smgr,
+ ForkNumber forknum,
+ BlockNumber blocknum,
+ int nblocks);
+
#endif /* SMGR_H */
diff --git a/src/backend/storage/aio/aio_subject.c b/src/backend/storage/aio/aio_subject.c
index 68e9e80074c..12ab1730f49 100644
--- a/src/backend/storage/aio/aio_subject.c
+++ b/src/backend/storage/aio/aio_subject.c
@@ -20,6 +20,7 @@
#include "storage/aio_internal.h"
#include "storage/buf_internals.h"
#include "storage/bufmgr.h"
+#include "storage/md.h"
#include "storage/smgr.h"
#include "utils/memutils.h"
@@ -28,9 +29,12 @@ static const PgAioSubjectInfo *aio_subject_info[] = {
[ASI_INVALID] = &(PgAioSubjectInfo) {
.name = "invalid",
},
+ [ASI_SMGR] = &aio_smgr_subject_info,
};
static const PgAioHandleSharedCallbacks *aio_shared_cbs[] = {
+ [ASC_MD_READV] = &aio_md_readv_cb,
+ [ASC_MD_WRITEV] = &aio_md_writev_cb,
};
diff --git a/src/backend/storage/file/fd.c b/src/backend/storage/file/fd.c
index 368cc9455cf..35bf3c1e7bd 100644
--- a/src/backend/storage/file/fd.c
+++ b/src/backend/storage/file/fd.c
@@ -95,6 +95,7 @@
#include "pgstat.h"
#include "portability/mem.h"
#include "postmaster/startup.h"
+#include "storage/aio.h"
#include "storage/fd.h"
#include "storage/ipc.h"
#include "utils/guc.h"
@@ -1295,6 +1296,8 @@ LruDelete(File file)
vfdP = &VfdCache[file];
+ pgaio_closing_fd(vfdP->fd);
+
/*
* Close the file. We aren't expecting this to fail; if it does, better
* to leak the FD than to mess up our internal state.
@@ -1988,6 +1991,8 @@ FileClose(File file)
if (!FileIsNotOpen(file))
{
+ pgaio_closing_fd(vfdP->fd);
+
/* close the file */
if (close(vfdP->fd) != 0)
{
@@ -2211,6 +2216,32 @@ retry:
return returnCode;
}
+int
+FileStartReadV(struct PgAioHandle *ioh, File file,
+ int iovcnt, off_t offset,
+ uint32 wait_event_info)
+{
+ int returnCode;
+ Vfd *vfdP;
+
+ Assert(FileIsValid(file));
+
+ DO_DB(elog(LOG, "FileStartReadV: %d (%s) " INT64_FORMAT " %d",
+ file, VfdCache[file].fileName,
+ (int64) offset,
+ iovcnt));
+
+ returnCode = FileAccess(file);
+ if (returnCode < 0)
+ return returnCode;
+
+ vfdP = &VfdCache[file];
+
+ pgaio_io_prep_readv(ioh, vfdP->fd, iovcnt, offset);
+
+ return 0;
+}
+
ssize_t
FileWriteV(File file, const struct iovec *iov, int iovcnt, off_t offset,
uint32 wait_event_info)
@@ -2316,6 +2347,34 @@ retry:
return returnCode;
}
+int
+FileStartWriteV(struct PgAioHandle *ioh, File file,
+ int iovcnt, off_t offset,
+ uint32 wait_event_info)
+{
+ int returnCode;
+ Vfd *vfdP;
+
+ Assert(FileIsValid(file));
+
+ DO_DB(elog(LOG, "FileStartWriteV: %d (%s) " INT64_FORMAT " %d",
+ file, VfdCache[file].fileName,
+ (int64) offset,
+ iovcnt));
+
+ returnCode = FileAccess(file);
+ if (returnCode < 0)
+ return returnCode;
+
+ vfdP = &VfdCache[file];
+
+ /* FIXME: think about / reimplement temp_file_limit */
+
+ pgaio_io_prep_writev(ioh, vfdP->fd, iovcnt, offset);
+
+ return 0;
+}
+
int
FileSync(File file, uint32 wait_event_info)
{
@@ -2499,6 +2558,12 @@ FilePathName(File file)
int
FileGetRawDesc(File file)
{
+ int returnCode;
+
+ returnCode = FileAccess(file);
+ if (returnCode < 0)
+ return returnCode;
+
Assert(FileIsValid(file));
return VfdCache[file].fd;
}
@@ -2779,6 +2844,7 @@ FreeDesc(AllocateDesc *desc)
result = closedir(desc->desc.dir);
break;
case AllocateDescRawFD:
+ pgaio_closing_fd(desc->desc.fd);
result = close(desc->desc.fd);
break;
default:
@@ -2847,6 +2913,8 @@ CloseTransientFile(int fd)
/* Only get here if someone passes us a file not in allocatedDescs */
elog(WARNING, "fd passed to CloseTransientFile was not obtained from OpenTransientFile");
+ pgaio_closing_fd(fd);
+
return close(fd);
}
diff --git a/src/backend/storage/smgr/md.c b/src/backend/storage/smgr/md.c
index 6cd81a61faa..f96308490d9 100644
--- a/src/backend/storage/smgr/md.c
+++ b/src/backend/storage/smgr/md.c
@@ -31,6 +31,7 @@
#include "miscadmin.h"
#include "pg_trace.h"
#include "pgstat.h"
+#include "storage/aio.h"
#include "storage/bufmgr.h"
#include "storage/fd.h"
#include "storage/md.h"
@@ -931,6 +932,49 @@ mdreadv(SMgrRelation reln, ForkNumber forknum, BlockNumber blocknum,
}
}
+void
+mdstartreadv(PgAioHandle *ioh,
+ SMgrRelation reln, ForkNumber forknum, BlockNumber blocknum,
+ void **buffers, BlockNumber nblocks)
+{
+ off_t seekpos;
+ MdfdVec *v;
+ BlockNumber nblocks_this_segment;
+ struct iovec *iov;
+ int iovcnt;
+
+ v = _mdfd_getseg(reln, forknum, blocknum, false,
+ EXTENSION_FAIL | EXTENSION_CREATE_RECOVERY);
+
+ seekpos = (off_t) BLCKSZ * (blocknum % ((BlockNumber) RELSEG_SIZE));
+
+ Assert(seekpos < (off_t) BLCKSZ * RELSEG_SIZE);
+
+ nblocks_this_segment =
+ Min(nblocks,
+ RELSEG_SIZE - (blocknum % ((BlockNumber) RELSEG_SIZE)));
+
+ if (nblocks_this_segment != nblocks)
+ elog(ERROR, "read crossing segment boundary");
+
+ iovcnt = pgaio_io_get_iovec(ioh, &iov);
+
+ Assert(nblocks <= iovcnt);
+
+ iovcnt = buffers_to_iovec(iov, buffers, nblocks_this_segment);
+
+ Assert(iovcnt <= nblocks_this_segment);
+
+ pgaio_io_set_subject_smgr(ioh,
+ reln,
+ forknum,
+ blocknum,
+ nblocks);
+ pgaio_io_add_shared_cb(ioh, ASC_MD_READV);
+
+ FileStartReadV(ioh, v->mdfd_vfd, iovcnt, seekpos, WAIT_EVENT_DATA_FILE_READ);
+}
+
/*
* mdwritev() -- Write the supplied blocks at the appropriate location.
*
@@ -1036,6 +1080,49 @@ mdwritev(SMgrRelation reln, ForkNumber forknum, BlockNumber blocknum,
}
}
+void
+mdstartwritev(PgAioHandle *ioh,
+ SMgrRelation reln, ForkNumber forknum, BlockNumber blocknum,
+ const void **buffers, BlockNumber nblocks, bool skipFsync)
+{
+ off_t seekpos;
+ MdfdVec *v;
+ BlockNumber nblocks_this_segment;
+ struct iovec *iov;
+ int iovcnt;
+
+ v = _mdfd_getseg(reln, forknum, blocknum, false,
+ EXTENSION_FAIL | EXTENSION_CREATE_RECOVERY);
+
+ seekpos = (off_t) BLCKSZ * (blocknum % ((BlockNumber) RELSEG_SIZE));
+
+ Assert(seekpos < (off_t) BLCKSZ * RELSEG_SIZE);
+
+ nblocks_this_segment =
+ Min(nblocks,
+ RELSEG_SIZE - (blocknum % ((BlockNumber) RELSEG_SIZE)));
+
+ if (nblocks_this_segment != nblocks)
+ elog(ERROR, "write crossing segment boundary");
+
+ iovcnt = pgaio_io_get_iovec(ioh, &iov);
+
+ Assert(nblocks <= iovcnt);
+
+ iovcnt = buffers_to_iovec(iov, unconstify(void **, buffers), nblocks_this_segment);
+
+ Assert(iovcnt <= nblocks_this_segment);
+
+ pgaio_io_set_subject_smgr(ioh,
+ reln,
+ forknum,
+ blocknum,
+ nblocks);
+ pgaio_io_add_shared_cb(ioh, ASC_MD_WRITEV);
+
+ FileStartWriteV(ioh, v->mdfd_vfd, iovcnt, seekpos, WAIT_EVENT_DATA_FILE_WRITE);
+}
+
/*
* mdwriteback() -- Tell the kernel to write pages back to storage.
@@ -1357,6 +1444,21 @@ mdimmedsync(SMgrRelation reln, ForkNumber forknum)
}
}
+int
+mdfd(SMgrRelation reln, ForkNumber forknum, BlockNumber blocknum, uint32 *off)
+{
+ MdfdVec *v = mdopenfork(reln, forknum, EXTENSION_FAIL);
+
+ v = _mdfd_getseg(reln, forknum, blocknum, false,
+ EXTENSION_FAIL);
+
+ *off = (off_t) BLCKSZ * (blocknum % ((BlockNumber) RELSEG_SIZE));
+
+ Assert(*off < (off_t) BLCKSZ * RELSEG_SIZE);
+
+ return FileGetRawDesc(v->mdfd_vfd);
+}
+
/*
* register_dirty_segment() -- Mark a relation segment as needing fsync
*
@@ -1832,3 +1934,118 @@ mdfiletagmatches(const FileTag *ftag, const FileTag *candidate)
*/
return ftag->rlocator.dbOid == candidate->rlocator.dbOid;
}
+
+
+
+static PgAioResult md_readv_complete(PgAioHandle *ioh, PgAioResult prior_result);
+static PgAioResult md_writev_complete(PgAioHandle *ioh, PgAioResult prior_result);
+static void md_readv_error(PgAioResult result, const PgAioSubjectData *subject_data, int elevel);
+
+const struct PgAioHandleSharedCallbacks aio_md_readv_cb = {
+ .complete = md_readv_complete,
+ .error = md_readv_error,
+};
+
+const struct PgAioHandleSharedCallbacks aio_md_writev_cb = {
+ .complete = md_writev_complete,
+};
+
+static PgAioResult
+md_readv_complete(PgAioHandle *ioh, PgAioResult prior_result)
+{
+ PgAioSubjectData *sd = pgaio_io_get_subject_data(ioh);
+ PgAioResult result = prior_result;
+
+ elog(DEBUG3, "%s: %d %d", __func__, prior_result.status, prior_result.result);
+
+ if (prior_result.result < 0)
+ {
+ result.status = ARS_ERROR;
+ result.id = ASC_MD_READV;
+ result.error_data = -prior_result.result;
+ result.result = 0;
+
+ md_readv_error(result, sd, LOG);
+
+ return result;
+ }
+
+ result.result /= BLCKSZ;
+
+ if (result.result == 0)
+ {
+ /* consider 0 blocks read a failure */
+ result.status = ARS_ERROR;
+ result.id = ASC_MD_READV;
+ result.error_data = 0;
+
+ md_readv_error(result, sd, LOG);
+ }
+
+ if (result.status != ARS_ERROR &&
+ result.result < sd->smgr.nblocks)
+ {
+ /* partial reads should be retried at upper level */
+ result.id = ASC_MD_READV;
+ result.status = ARS_PARTIAL;
+ }
+
+ /* AFIXME: post-read portion of mdreadv() */
+
+ return result;
+}
+
+static void
+md_readv_error(PgAioResult result, const PgAioSubjectData *subject_data, int elevel)
+{
+ MemoryContext oldContext = CurrentMemoryContext;
+
+ /* AFIXME: */
+ oldContext = MemoryContextSwitchTo(ErrorContext);
+
+ if (result.error_data != 0)
+ {
+ errno = result.error_data; /* for errcode_for_file_access() */
+
+ ereport(elevel,
+ errcode_for_file_access(),
+ errmsg("could not read blocks %u..%u in file \"%s\": %m",
+ subject_data->smgr.blockNum,
+ subject_data->smgr.blockNum + subject_data->smgr.nblocks,
+ relpathperm(subject_data->smgr.rlocator, subject_data->smgr.forkNum)
+ )
+ );
+ }
+ else
+ {
+ ereport(elevel,
+ errcode(ERRCODE_DATA_CORRUPTED),
+ errmsg("could not read blocks %u..%u in file \"%s\": read only %zu of %zu bytes",
+ subject_data->smgr.blockNum,
+ subject_data->smgr.blockNum + subject_data->smgr.nblocks - 1,
+ relpathperm(subject_data->smgr.rlocator, subject_data->smgr.forkNum),
+ result.result * (size_t) BLCKSZ,
+ subject_data->smgr.nblocks * (size_t) BLCKSZ
+ )
+ );
+ }
+
+ MemoryContextSwitchTo(oldContext);
+}
+
+
+static PgAioResult
+md_writev_complete(PgAioHandle *ioh, PgAioResult prior_result)
+{
+ elog(DEBUG3, "%s: %d %d", __func__, prior_result.status, prior_result.result);
+
+ if (prior_result.status == ARS_ERROR)
+ {
+ /* AFIXME: complain */
+ return prior_result;
+ }
+
+ prior_result.result /= BLCKSZ;
+
+ return prior_result;
+}
diff --git a/src/backend/storage/smgr/smgr.c b/src/backend/storage/smgr/smgr.c
index ee31db85eec..2dacb361a4f 100644
--- a/src/backend/storage/smgr/smgr.c
+++ b/src/backend/storage/smgr/smgr.c
@@ -53,6 +53,7 @@
#include "access/xlogutils.h"
#include "lib/ilist.h"
+#include "storage/aio.h"
#include "storage/bufmgr.h"
#include "storage/ipc.h"
#include "storage/md.h"
@@ -93,10 +94,19 @@ typedef struct f_smgr
void (*smgr_readv) (SMgrRelation reln, ForkNumber forknum,
BlockNumber blocknum,
void **buffers, BlockNumber nblocks);
+ void (*smgr_startreadv) (struct PgAioHandle *ioh,
+ SMgrRelation reln, ForkNumber forknum,
+ BlockNumber blocknum,
+ void **buffers, BlockNumber nblocks);
void (*smgr_writev) (SMgrRelation reln, ForkNumber forknum,
BlockNumber blocknum,
const void **buffers, BlockNumber nblocks,
bool skipFsync);
+ void (*smgr_startwritev) (struct PgAioHandle *ioh,
+ SMgrRelation reln, ForkNumber forknum,
+ BlockNumber blocknum,
+ const void **buffers, BlockNumber nblocks,
+ bool skipFsync);
void (*smgr_writeback) (SMgrRelation reln, ForkNumber forknum,
BlockNumber blocknum, BlockNumber nblocks);
BlockNumber (*smgr_nblocks) (SMgrRelation reln, ForkNumber forknum);
@@ -104,6 +114,7 @@ typedef struct f_smgr
BlockNumber nblocks);
void (*smgr_immedsync) (SMgrRelation reln, ForkNumber forknum);
void (*smgr_registersync) (SMgrRelation reln, ForkNumber forknum);
+ int (*smgr_fd) (SMgrRelation reln, ForkNumber forknum, BlockNumber blocknum, uint32 *off);
} f_smgr;
static const f_smgr smgrsw[] = {
@@ -121,12 +132,15 @@ static const f_smgr smgrsw[] = {
.smgr_prefetch = mdprefetch,
.smgr_maxcombine = mdmaxcombine,
.smgr_readv = mdreadv,
+ .smgr_startreadv = mdstartreadv,
.smgr_writev = mdwritev,
+ .smgr_startwritev = mdstartwritev,
.smgr_writeback = mdwriteback,
.smgr_nblocks = mdnblocks,
.smgr_truncate = mdtruncate,
.smgr_immedsync = mdimmedsync,
.smgr_registersync = mdregistersync,
+ .smgr_fd = mdfd,
}
};
@@ -145,6 +159,14 @@ static void smgrshutdown(int code, Datum arg);
static void smgrdestroy(SMgrRelation reln);
+static void smgr_aio_reopen(PgAioHandle *ioh);
+
+const struct PgAioSubjectInfo aio_smgr_subject_info = {
+ .name = "smgr",
+ .reopen = smgr_aio_reopen,
+};
+
+
/*
* smgrinit(), smgrshutdown() -- Initialize or shut down storage
* managers.
@@ -620,6 +642,19 @@ smgrreadv(SMgrRelation reln, ForkNumber forknum, BlockNumber blocknum,
nblocks);
}
+/*
+ * FILL ME IN
+ */
+void
+smgrstartreadv(struct PgAioHandle *ioh,
+ SMgrRelation reln, ForkNumber forknum, BlockNumber blocknum,
+ void **buffers, BlockNumber nblocks)
+{
+ smgrsw[reln->smgr_which].smgr_startreadv(ioh,
+ reln, forknum, blocknum, buffers,
+ nblocks);
+}
+
/*
* smgrwritev() -- Write the supplied buffers out.
*
@@ -651,6 +686,16 @@ smgrwritev(SMgrRelation reln, ForkNumber forknum, BlockNumber blocknum,
buffers, nblocks, skipFsync);
}
+void
+smgrstartwritev(struct PgAioHandle *ioh,
+ SMgrRelation reln, ForkNumber forknum, BlockNumber blocknum,
+ const void **buffers, BlockNumber nblocks, bool skipFsync)
+{
+ smgrsw[reln->smgr_which].smgr_startwritev(ioh,
+ reln, forknum, blocknum, buffers,
+ nblocks, skipFsync);
+}
+
/*
* smgrwriteback() -- Trigger kernel writeback for the supplied range of
* blocks.
@@ -807,6 +852,12 @@ smgrimmedsync(SMgrRelation reln, ForkNumber forknum)
smgrsw[reln->smgr_which].smgr_immedsync(reln, forknum);
}
+int
+smgrfd(SMgrRelation reln, ForkNumber forknum, BlockNumber blocknum, uint32 *off)
+{
+ return smgrsw[reln->smgr_which].smgr_fd(reln, forknum, blocknum, off);
+}
+
/*
* AtEOXact_SMgr
*
@@ -835,3 +886,43 @@ ProcessBarrierSmgrRelease(void)
smgrreleaseall();
return true;
}
+
+void
+pgaio_io_set_subject_smgr(PgAioHandle *ioh,
+ struct SMgrRelationData *smgr,
+ ForkNumber forknum,
+ BlockNumber blocknum,
+ int nblocks)
+{
+ PgAioSubjectData *sd = pgaio_io_get_subject_data(ioh);
+
+ pgaio_io_set_subject(ioh, ASI_SMGR);
+
+ /* backend is implied via IO owner */
+ sd->smgr.rlocator = smgr->smgr_rlocator.locator;
+ sd->smgr.forkNum = forknum;
+ sd->smgr.blockNum = blocknum;
+ sd->smgr.nblocks = nblocks;
+ sd->smgr.is_temp = SmgrIsTemp(smgr);
+ sd->smgr.release_lock = false;
+ sd->smgr.mode = RBM_NORMAL;
+}
+
+static void
+smgr_aio_reopen(PgAioHandle *ioh)
+{
+ PgAioSubjectData *sd = pgaio_io_get_subject_data(ioh);
+ PgAioOpData *od = pgaio_io_get_op_data(ioh);
+ SMgrRelation reln;
+ ProcNumber procno;
+ uint32 off;
+
+ if (sd->smgr.is_temp)
+ procno = pgaio_io_get_owner(ioh);
+ else
+ procno = INVALID_PROC_NUMBER;
+
+ reln = smgropen(sd->smgr.rlocator, procno);
+ od->read.fd = smgrfd(reln, sd->smgr.forkNum, sd->smgr.blockNum, &off);
+ Assert(off == od->read.offset);
+}
--
2.45.2.827.g557ae147e6
v2.0-0011-bufmgr-Implement-AIO-support.patch (text/x-diff, 20.9K) download
From 9d8c6210e3a5e39d585d0a8ebebeac8a9e9b62a2 Mon Sep 17 00:00:00 2001
From: Andres Freund <andres@anarazel.de>
Date: Sat, 31 Aug 2024 21:39:01 -0400
Subject: [PATCH v2.0 11/17] bufmgr: Implement AIO support
As of this commit there are no users of these AIO facilities, that'll come in
later commits.
Author:
Reviewed-By:
Discussion: https://postgr.es/m/
Backpatch:
---
src/include/storage/aio.h | 6 +
src/include/storage/buf_internals.h | 6 +
src/include/storage/bufmgr.h | 10 +
src/backend/storage/aio/aio_subject.c | 5 +
src/backend/storage/buffer/buf_init.c | 3 +
src/backend/storage/buffer/bufmgr.c | 431 +++++++++++++++++++++++++-
src/backend/storage/buffer/localbuf.c | 65 ++++
7 files changed, 519 insertions(+), 7 deletions(-)
diff --git a/src/include/storage/aio.h b/src/include/storage/aio.h
index acfd50c587c..40c80a2fed4 100644
--- a/src/include/storage/aio.h
+++ b/src/include/storage/aio.h
@@ -93,6 +93,12 @@ typedef enum PgAioHandleSharedCallbackID
{
ASC_MD_READV,
ASC_MD_WRITEV,
+
+ ASC_SHARED_BUFFER_READ,
+ ASC_SHARED_BUFFER_WRITE,
+
+ ASC_LOCAL_BUFFER_READ,
+ ASC_LOCAL_BUFFER_WRITE,
} PgAioHandleSharedCallbackID;
diff --git a/src/include/storage/buf_internals.h b/src/include/storage/buf_internals.h
index f190e6e5e46..5cfa7dbd1f1 100644
--- a/src/include/storage/buf_internals.h
+++ b/src/include/storage/buf_internals.h
@@ -17,6 +17,7 @@
#include "pgstat.h"
#include "port/atomics.h"
+#include "storage/aio_ref.h"
#include "storage/buf.h"
#include "storage/bufmgr.h"
#include "storage/condition_variable.h"
@@ -252,6 +253,8 @@ typedef struct BufferDesc
int wait_backend_pgprocno; /* backend of pin-count waiter */
int freeNext; /* link in freelist chain */
+
+ PgAioHandleRef io_in_progress;
LWLock content_lock; /* to lock access to buffer contents */
} BufferDesc;
@@ -465,4 +468,7 @@ extern void DropRelationLocalBuffers(RelFileLocator rlocator,
extern void DropRelationAllLocalBuffers(RelFileLocator rlocator);
extern void AtEOXact_LocalBuffers(bool isCommit);
+
+extern bool ReadBufferCompleteReadLocal(Buffer buffer, int mode, bool failed);
+
#endif /* BUFMGR_INTERNALS_H */
diff --git a/src/include/storage/bufmgr.h b/src/include/storage/bufmgr.h
index eb0fba4230b..6cd64b8c2b3 100644
--- a/src/include/storage/bufmgr.h
+++ b/src/include/storage/bufmgr.h
@@ -177,6 +177,14 @@ extern PGDLLIMPORT int NLocBuffer;
extern PGDLLIMPORT Block *LocalBufferBlockPointers;
extern PGDLLIMPORT int32 *LocalRefCount;
+
+struct PgAioHandleSharedCallbacks;
+extern const struct PgAioHandleSharedCallbacks aio_shared_buffer_read_cb;
+extern const struct PgAioHandleSharedCallbacks aio_shared_buffer_write_cb;
+extern const struct PgAioHandleSharedCallbacks aio_local_buffer_read_cb;
+extern const struct PgAioHandleSharedCallbacks aio_local_buffer_write_cb;
+
+
/* upper limit for effective_io_concurrency */
#define MAX_IO_CONCURRENCY 1000
@@ -194,6 +202,8 @@ extern PGDLLIMPORT int32 *LocalRefCount;
/*
* prototypes for functions in bufmgr.c
*/
+struct PgAioHandle;
+
extern PrefetchBufferResult PrefetchSharedBuffer(struct SMgrRelationData *smgr_reln,
ForkNumber forkNum,
BlockNumber blockNum);
diff --git a/src/backend/storage/aio/aio_subject.c b/src/backend/storage/aio/aio_subject.c
index 12ab1730f49..0676f3d3a66 100644
--- a/src/backend/storage/aio/aio_subject.c
+++ b/src/backend/storage/aio/aio_subject.c
@@ -35,6 +35,11 @@ static const PgAioSubjectInfo *aio_subject_info[] = {
static const PgAioHandleSharedCallbacks *aio_shared_cbs[] = {
[ASC_MD_READV] = &aio_md_readv_cb,
[ASC_MD_WRITEV] = &aio_md_writev_cb,
+
+ [ASC_SHARED_BUFFER_READ] = &aio_shared_buffer_read_cb,
+ [ASC_SHARED_BUFFER_WRITE] = &aio_shared_buffer_write_cb,
+ [ASC_LOCAL_BUFFER_READ] = &aio_local_buffer_read_cb,
+ [ASC_LOCAL_BUFFER_WRITE] = &aio_local_buffer_write_cb,
};
diff --git a/src/backend/storage/buffer/buf_init.c b/src/backend/storage/buffer/buf_init.c
index 09bec6449b6..059a80dfb13 100644
--- a/src/backend/storage/buffer/buf_init.c
+++ b/src/backend/storage/buffer/buf_init.c
@@ -14,6 +14,7 @@
*/
#include "postgres.h"
+#include "storage/aio.h"
#include "storage/buf_internals.h"
#include "storage/bufmgr.h"
#include "storage/proc.h"
@@ -126,6 +127,8 @@ BufferManagerShmemInit(void)
buf->buf_id = i;
+ pgaio_io_ref_clear(&buf->io_in_progress);
+
/*
* Initially link all the buffers together as unused. Subsequent
* management of this list is done by freelist.c.
diff --git a/src/backend/storage/buffer/bufmgr.c b/src/backend/storage/buffer/bufmgr.c
index f2e608f597d..8feafd6e53c 100644
--- a/src/backend/storage/buffer/bufmgr.c
+++ b/src/backend/storage/buffer/bufmgr.c
@@ -48,6 +48,7 @@
#include "pg_trace.h"
#include "pgstat.h"
#include "postmaster/bgwriter.h"
+#include "storage/aio.h"
#include "storage/buf_internals.h"
#include "storage/bufmgr.h"
#include "storage/fd.h"
@@ -58,6 +59,7 @@
#include "storage/smgr.h"
#include "storage/standby.h"
#include "utils/memdebug.h"
+#include "utils/memutils.h"
#include "utils/ps_status.h"
#include "utils/rel.h"
#include "utils/resowner.h"
@@ -541,7 +543,8 @@ static int SyncOneBuffer(int buf_id, bool skip_recently_used,
static void WaitIO(BufferDesc *buf);
static bool StartBufferIO(BufferDesc *buf, bool forInput, bool nowait);
static void TerminateBufferIO(BufferDesc *buf, bool clear_dirty,
- uint32 set_flag_bits, bool forget_owner);
+ uint32 set_flag_bits, bool forget_owner,
+ bool syncio);
static void AbortBufferIO(Buffer buffer);
static void shared_buffer_write_error_callback(void *arg);
static void local_buffer_write_error_callback(void *arg);
@@ -1108,7 +1111,7 @@ ZeroAndLockBuffer(Buffer buffer, ReadBufferMode mode, bool already_valid)
else
{
/* Set BM_VALID, terminate IO, and wake up any waiters */
- TerminateBufferIO(bufHdr, false, BM_VALID, true);
+ TerminateBufferIO(bufHdr, false, BM_VALID, true, true);
}
}
else if (!isLocalBuf)
@@ -1593,7 +1596,7 @@ WaitReadBuffers(ReadBuffersOperation *operation)
else
{
/* Set BM_VALID, terminate IO, and wake up any waiters */
- TerminateBufferIO(bufHdr, false, BM_VALID, true);
+ TerminateBufferIO(bufHdr, false, BM_VALID, true, true);
}
/* Report I/Os as completing individually. */
@@ -2477,7 +2480,7 @@ ExtendBufferedRelShared(BufferManagerRelation bmr,
if (lock)
LWLockAcquire(BufferDescriptorGetContentLock(buf_hdr), LW_EXCLUSIVE);
- TerminateBufferIO(buf_hdr, false, BM_VALID, true);
+ TerminateBufferIO(buf_hdr, false, BM_VALID, true, true);
}
pgBufferUsage.shared_blks_written += extend_by;
@@ -3926,7 +3929,7 @@ FlushBuffer(BufferDesc *buf, SMgrRelation reln, IOObject io_object,
* Mark the buffer as clean (unless BM_JUST_DIRTIED has become set) and
* end the BM_IO_IN_PROGRESS state.
*/
- TerminateBufferIO(buf, true, 0, true);
+ TerminateBufferIO(buf, true, 0, true, true);
TRACE_POSTGRESQL_BUFFER_FLUSH_DONE(BufTagGetForkNum(&buf->tag),
buf->tag.blockNum,
@@ -5541,6 +5544,7 @@ WaitIO(BufferDesc *buf)
for (;;)
{
uint32 buf_state;
+ PgAioHandleRef ior;
/*
* It may not be necessary to acquire the spinlock to check the flag
@@ -5548,10 +5552,19 @@ WaitIO(BufferDesc *buf)
* play it safe.
*/
buf_state = LockBufHdr(buf);
+ ior = buf->io_in_progress;
UnlockBufHdr(buf, buf_state);
if (!(buf_state & BM_IO_IN_PROGRESS))
break;
+
+ if (pgaio_io_ref_valid(&ior))
+ {
+ pgaio_io_ref_wait(&ior);
+ ConditionVariablePrepareToSleep(cv);
+ continue;
+ }
+
ConditionVariableSleep(cv, WAIT_EVENT_BUFFER_IO);
}
ConditionVariableCancelSleep();
@@ -5640,7 +5653,7 @@ StartBufferIO(BufferDesc *buf, bool forInput, bool nowait)
*/
static void
TerminateBufferIO(BufferDesc *buf, bool clear_dirty, uint32 set_flag_bits,
- bool forget_owner)
+ bool forget_owner, bool syncio)
{
uint32 buf_state;
@@ -5652,6 +5665,13 @@ TerminateBufferIO(BufferDesc *buf, bool clear_dirty, uint32 set_flag_bits,
if (clear_dirty && !(buf_state & BM_JUST_DIRTIED))
buf_state &= ~(BM_DIRTY | BM_CHECKPOINT_NEEDED);
+ if (!syncio)
+ {
+ /* release ownership by the AIO subsystem */
+ buf_state -= BUF_REFCOUNT_ONE;
+ pgaio_io_ref_clear(&buf->io_in_progress);
+ }
+
buf_state |= set_flag_bits;
UnlockBufHdr(buf, buf_state);
@@ -5660,6 +5680,40 @@ TerminateBufferIO(BufferDesc *buf, bool clear_dirty, uint32 set_flag_bits,
BufferDescriptorGetBuffer(buf));
ConditionVariableBroadcast(BufferDescriptorGetIOCV(buf));
+
+ /*
+ * If we just released a pin, need to do BM_PIN_COUNT_WAITER handling.
+ * Most of the time the current backend will hold another pin preventing
+ * that from happening, but that's e.g. not the case when completing an IO
+ * another backend started.
+ *
+ * AFIXME: Deduplicate with UnpinBufferNoOwner() or just replace
+ * BM_PIN_COUNT_WAITER with something saner.
+ */
+ /* Support LockBufferForCleanup() */
+ if (buf_state & BM_PIN_COUNT_WAITER)
+ {
+ /*
+ * Acquire the buffer header lock, re-check that there's a waiter.
+ * Another backend could have unpinned this buffer, and already woken
+ * up the waiter. There's no danger of the buffer being replaced
+ * after we unpinned it above, as it's pinned by the waiter.
+ */
+ buf_state = LockBufHdr(buf);
+
+ if ((buf_state & BM_PIN_COUNT_WAITER) &&
+ BUF_STATE_GET_REFCOUNT(buf_state) == 1)
+ {
+ /* we just released the last pin other than the waiter's */
+ int wait_backend_pgprocno = buf->wait_backend_pgprocno;
+
+ buf_state &= ~BM_PIN_COUNT_WAITER;
+ UnlockBufHdr(buf, buf_state);
+ ProcSendSignal(wait_backend_pgprocno);
+ }
+ else
+ UnlockBufHdr(buf, buf_state);
+ }
}
/*
@@ -5711,7 +5765,7 @@ AbortBufferIO(Buffer buffer)
}
}
- TerminateBufferIO(buf_hdr, false, BM_IO_ERROR, false);
+ TerminateBufferIO(buf_hdr, false, BM_IO_ERROR, false, true);
}
/*
@@ -6170,3 +6224,366 @@ EvictUnpinnedBuffer(Buffer buf)
return result;
}
+
+static bool
+ReadBufferCompleteReadShared(Buffer buffer, int mode, bool failed)
+{
+ BufferDesc *bufHdr = NULL;
+ BlockNumber blockno;
+ bool buf_failed = false;
+ char *bufdata = BufferGetBlock(buffer);
+
+ Assert(BufferIsValid(buffer));
+
+ bufHdr = GetBufferDescriptor(buffer - 1);
+ blockno = bufHdr->tag.blockNum;
+
+#ifdef USE_ASSERT_CHECKING
+ {
+ uint32 buf_state = pg_atomic_read_u32(&bufHdr->state);
+
+ Assert(buf_state & BM_TAG_VALID);
+ Assert(!(buf_state & BM_VALID));
+ Assert(buf_state & BM_IO_IN_PROGRESS);
+ Assert(!(buf_state & BM_DIRTY));
+ }
+#endif
+
+ /* check for garbage data */
+ if (!failed &&
+ !PageIsVerifiedExtended((Page) bufdata, blockno,
+ PIV_LOG_WARNING | PIV_REPORT_STAT))
+ {
+ RelFileLocator rlocator = BufTagGetRelFileLocator(&bufHdr->tag);
+ BlockNumber forkNum = bufHdr->tag.forkNum;
+
+ /* AFIXME: relpathperm allocates memory */
+ MemoryContextSwitchTo(ErrorContext);
+ if (mode == READ_BUFFERS_ZERO_ON_ERROR || zero_damaged_pages)
+ {
+ ereport(LOG,
+ (errcode(ERRCODE_DATA_CORRUPTED),
+ errmsg("invalid page in block %u of relation %s; zeroing out page",
+ blockno,
+ relpathperm(rlocator, forkNum))));
+ memset(bufdata, 0, BLCKSZ);
+ }
+ else
+ {
+ ereport(LOG,
+ (errcode(ERRCODE_DATA_CORRUPTED),
+ errmsg("invalid page in block %u of relation %s",
+ blockno,
+ relpathperm(rlocator, forkNum))));
+ failed = true;
+ buf_failed = true;
+ }
+ }
+
+ /* Terminate I/O and set BM_VALID. */
+ TerminateBufferIO(bufHdr, false,
+ failed ? BM_IO_ERROR : BM_VALID,
+ false, false);
+
+ /* Report I/Os as completing individually. */
+
+ /* FIXME: Should we do TRACE_POSTGRESQL_BUFFER_READ_DONE here? */
+ return buf_failed;
+}
+
+static uint64
+ReadBufferCompleteWriteShared(Buffer buffer, bool release_lock, bool failed)
+{
+ BufferDesc *bufHdr;
+ bool result = false;
+
+ Assert(BufferIsValid(buffer));
+
+ bufHdr = GetBufferDescriptor(buffer - 1);
+
+#ifdef USE_ASSERT_CHECKING
+ {
+ uint32 buf_state = pg_atomic_read_u32(&bufHdr->state);
+
+ Assert(buf_state & BM_VALID);
+ Assert(buf_state & BM_TAG_VALID);
+ Assert(buf_state & BM_IO_IN_PROGRESS);
+ Assert(buf_state & BM_DIRTY);
+ }
+#endif
+
+ /* AFIXME: implement track_io_timing */
+
+ TerminateBufferIO(bufHdr, /* clear_dirty = */ true,
+ failed ? BM_IO_ERROR : 0,
+ /* forget_owner = */ false,
+ /* syncio = */ false);
+
+ /*
+ * The initiator of IO is not managing the lock (i.e. called
+ * LWLockReleaseOwnership()), we are.
+ */
+ if (release_lock)
+ LWLockReleaseUnowned(BufferDescriptorGetContentLock(bufHdr), LW_SHARED);
+
+ return result;
+}
+
+static void
+shared_buffer_prepare_common(PgAioHandle *ioh, bool is_write)
+{
+ uint64 *io_data;
+ uint8 io_data_len;
+ PgAioHandleRef io_ref;
+ BufferTag first PG_USED_FOR_ASSERTS_ONLY = {0};
+
+ io_data = pgaio_io_get_io_data(ioh, &io_data_len);
+
+ pgaio_io_get_ref(ioh, &io_ref);
+
+ for (int i = 0; i < io_data_len; i++)
+ {
+ Buffer buf = (Buffer) io_data[i];
+ BufferDesc *bufHdr;
+ uint32 buf_state;
+
+ bufHdr = GetBufferDescriptor(buf - 1);
+
+ if (i == 0)
+ first = bufHdr->tag;
+ else
+ {
+ Assert(bufHdr->tag.relNumber == first.relNumber);
+ Assert(bufHdr->tag.blockNum == first.blockNum + i);
+ }
+
+
+ buf_state = LockBufHdr(bufHdr);
+
+ Assert(buf_state & BM_TAG_VALID);
+ if (is_write)
+ {
+ Assert(buf_state & BM_VALID);
+ Assert(buf_state & BM_DIRTY);
+ }
+ else
+ Assert(!(buf_state & BM_VALID));
+
+ Assert(buf_state & BM_IO_IN_PROGRESS);
+ Assert(BUF_STATE_GET_REFCOUNT(buf_state) >= 1);
+
+ buf_state += BUF_REFCOUNT_ONE;
+ bufHdr->io_in_progress = io_ref;
+
+ UnlockBufHdr(bufHdr, buf_state);
+
+ if (is_write)
+ {
+ LWLock *content_lock;
+
+ content_lock = BufferDescriptorGetContentLock(bufHdr);
+
+ Assert(LWLockHeldByMe(content_lock));
+
+ /*
+ * Lock now owned by IO.
+ */
+ LWLockReleaseOwnership(content_lock);
+ }
+
+ /*
+ * Stop tracking this buffer via the resowner - the AIO system now
+ * keeps track.
+ */
+ ResourceOwnerForgetBufferIO(CurrentResourceOwner, buf);
+ }
+}
+
+static void
+shared_buffer_read_prepare(PgAioHandle *ioh)
+{
+ shared_buffer_prepare_common(ioh, false);
+}
+
+static void
+shared_buffer_write_prepare(PgAioHandle *ioh)
+{
+ shared_buffer_prepare_common(ioh, true);
+}
+
+
+static PgAioResult
+shared_buffer_read_complete(PgAioHandle *ioh, PgAioResult prior_result)
+{
+ PgAioResult result = prior_result;
+ int mode = pgaio_io_get_subject_data(ioh)->smgr.mode;
+ uint64 *io_data;
+ uint8 io_data_len;
+
+ elog(DEBUG3, "%s: %d %d", __func__, prior_result.status, prior_result.result);
+
+ io_data = pgaio_io_get_io_data(ioh, &io_data_len);
+
+ for (int io_data_off = 0; io_data_off < io_data_len; io_data_off++)
+ {
+ Buffer buf = io_data[io_data_off];
+ bool buf_failed;
+ bool failed;
+
+ failed =
+ prior_result.status == ARS_ERROR
+ || prior_result.result <= io_data_off;
+
+ elog(DEBUG3, "calling rbcrs for buf %d with failed %d, error: %d, result: %d, data_off: %d",
+ buf, failed, prior_result.status, prior_result.result, io_data_off);
+
+ /*
+ * AFIXME: It'd probably be better to not set BM_IO_ERROR (which is
+ * what failed = true leads to) when it's just a short read...
+ */
+ buf_failed = ReadBufferCompleteReadShared(buf,
+ mode,
+ failed);
+
+ if (result.status != ARS_ERROR && buf_failed)
+ {
+ result.status = ARS_ERROR;
+ result.id = ASC_SHARED_BUFFER_READ;
+ result.error_data = io_data_off + 1;
+ }
+ }
+
+ return result;
+}
+
+static void
+shared_buffer_read_error(PgAioResult result, const PgAioSubjectData *subject_data, int elevel)
+{
+ MemoryContext oldContext = CurrentMemoryContext;
+
+ /* AFIXME: */
+ oldContext = MemoryContextSwitchTo(ErrorContext);
+
+ ereport(elevel,
+ errcode(ERRCODE_DATA_CORRUPTED),
+ errmsg("invalid page in block %u of relation %s",
+ subject_data->smgr.blockNum + result.error_data,
+ relpathperm(subject_data->smgr.rlocator, subject_data->smgr.forkNum)
+ )
+ );
+ MemoryContextSwitchTo(oldContext);
+}
+
+static PgAioResult
+shared_buffer_write_complete(PgAioHandle *ioh, PgAioResult prior_result)
+{
+ PgAioResult result = prior_result;
+ uint64 *io_data;
+ uint8 io_data_len;
+
+ elog(DEBUG3, "%s: %d %d", __func__, prior_result.status, prior_result.result);
+
+ io_data = pgaio_io_get_io_data(ioh, &io_data_len);
+
+ /* FIXME: handle outright errors */
+
+ for (int io_data_off = 0; io_data_off < io_data_len; io_data_off++)
+ {
+ Buffer buf = io_data[io_data_off];
+
+ /* FIXME: handle short writes / failures */
+ /* FIXME: ioh->scb_data.shared_buffer.release_lock */
+ ReadBufferCompleteWriteShared(buf,
+ true,
+ false);
+
+ }
+
+ return result;
+}
+
+static void
+local_buffer_read_prepare(PgAioHandle *ioh)
+{
+ uint64 *io_data;
+ uint8 io_data_len;
+ PgAioHandleRef io_ref;
+
+ io_data = pgaio_io_get_io_data(ioh, &io_data_len);
+
+ pgaio_io_get_ref(ioh, &io_ref);
+
+ for (int i = 0; i < io_data_len; i++)
+ {
+ Buffer buf = (Buffer) io_data[i];
+ BufferDesc *bufHdr;
+ uint32 buf_state;
+
+ bufHdr = GetLocalBufferDescriptor(-buf - 1);
+
+ buf_state = pg_atomic_read_u32(&bufHdr->state);
+
+ bufHdr->io_in_progress = io_ref;
+ LocalRefCount[-buf - 1] += 1;
+
+ UnlockBufHdr(bufHdr, buf_state);
+ }
+}
+
+static PgAioResult
+local_buffer_read_complete(PgAioHandle *ioh, PgAioResult prior_result)
+{
+ PgAioResult result = prior_result;
+ int mode = pgaio_io_get_subject_data(ioh)->smgr.mode;
+ uint64 *io_data;
+ uint8 io_data_len;
+
+ elog(DEBUG3, "%s: %d %d", __func__, prior_result.status, prior_result.result);
+
+ io_data = pgaio_io_get_io_data(ioh, &io_data_len);
+
+ /* FIXME: error handling */
+
+ for (int io_data_off = 0; io_data_off < io_data_len; io_data_off++)
+ {
+ Buffer buf = io_data[io_data_off];
+ bool buf_failed;
+
+ buf_failed = ReadBufferCompleteReadLocal(buf,
+ mode,
+ false);
+
+ if (result.status != ARS_ERROR && buf_failed)
+ {
+ result.status = ARS_ERROR;
+ result.id = ASC_LOCAL_BUFFER_READ;
+ result.error_data = io_data_off + 1;
+ }
+ }
+
+ return result;
+}
+
+static void
+local_buffer_write_prepare(PgAioHandle *ioh)
+{
+ elog(ERROR, "not yet");
+}
+
+
+const struct PgAioHandleSharedCallbacks aio_shared_buffer_read_cb = {
+ .prepare = shared_buffer_read_prepare,
+ .complete = shared_buffer_read_complete,
+ .error = shared_buffer_read_error,
+};
+const struct PgAioHandleSharedCallbacks aio_shared_buffer_write_cb = {
+ .prepare = shared_buffer_write_prepare,
+ .complete = shared_buffer_write_complete,
+};
+const struct PgAioHandleSharedCallbacks aio_local_buffer_read_cb = {
+ .prepare = local_buffer_read_prepare,
+ .complete = local_buffer_read_complete,
+};
+const struct PgAioHandleSharedCallbacks aio_local_buffer_write_cb = {
+ .prepare = local_buffer_write_prepare,
+};
diff --git a/src/backend/storage/buffer/localbuf.c b/src/backend/storage/buffer/localbuf.c
index 8da7dd6c98a..a7eb723f1e9 100644
--- a/src/backend/storage/buffer/localbuf.c
+++ b/src/backend/storage/buffer/localbuf.c
@@ -18,6 +18,7 @@
#include "access/parallel.h"
#include "executor/instrument.h"
#include "pgstat.h"
+#include "storage/aio.h"
#include "storage/buf_internals.h"
#include "storage/bufmgr.h"
#include "storage/fd.h"
@@ -620,6 +621,8 @@ InitLocalBuffers(void)
*/
buf->buf_id = -i - 2;
+ pgaio_io_ref_clear(&buf->io_in_progress);
+
/*
* Intentionally do not initialize the buffer's atomic variable
* (besides zeroing the underlying memory above). That way we get
@@ -836,3 +839,65 @@ AtProcExit_LocalBuffers(void)
*/
CheckForLocalBufferLeaks();
}
+
+bool
+ReadBufferCompleteReadLocal(Buffer buffer, int mode, bool failed)
+{
+ BufferDesc *buf_hdr = NULL;
+ BlockNumber blockno;
+ bool buf_failed = false;
+ char *bufdata = BufferGetBlock(buffer);
+
+ Assert(BufferIsValid(buffer));
+
+ buf_hdr = GetLocalBufferDescriptor(-buffer - 1);
+ blockno = buf_hdr->tag.blockNum;
+
+ /* check for garbage data */
+ if (!failed &&
+ !PageIsVerifiedExtended((Page) bufdata, blockno,
+ PIV_LOG_WARNING | PIV_REPORT_STAT))
+ {
+ RelFileLocator rlocator = BufTagGetRelFileLocator(&buf_hdr->tag);
+ BlockNumber forkNum = buf_hdr->tag.forkNum;
+
+ MemoryContextSwitchTo(ErrorContext);
+
+ if (mode == READ_BUFFERS_ZERO_ON_ERROR || zero_damaged_pages)
+ {
+
+ ereport(WARNING,
+ (errcode(ERRCODE_DATA_CORRUPTED),
+ errmsg("invalid page in block %u of relation %s; zeroing out page",
+ blockno,
+ relpathperm(rlocator, forkNum))));
+ memset(bufdata, 0, BLCKSZ);
+ }
+ else
+ {
+ ereport(LOG,
+ (errcode(ERRCODE_DATA_CORRUPTED),
+ errmsg("invalid page in block %u of relation %s",
+ blockno,
+ relpathperm(rlocator, forkNum))));
+ failed = true;
+ buf_failed = true;
+ }
+ }
+
+ /* Terminate I/O and set BM_VALID. */
+ pgaio_io_ref_clear(&buf_hdr->io_in_progress);
+
+ {
+ uint32 buf_state;
+
+ buf_state = pg_atomic_read_u32(&buf_hdr->state);
+ buf_state |= BM_VALID;
+ pg_atomic_unlocked_write_u32(&buf_hdr->state, buf_state);
+ }
+
+ /* release pin held by IO subsystem */
+ LocalRefCount[-buffer - 1] -= 1;
+
+ return buf_failed;
+}
--
2.45.2.827.g557ae147e6
v2.0-0012-bufmgr-Use-aio-for-StartReadBuffers.patch (text/x-diff, 14.0K) download
From fe6df768de29f124263f6fe250017f04454ca699 Mon Sep 17 00:00:00 2001
From: Andres Freund <andres@anarazel.de>
Date: Sat, 31 Aug 2024 21:55:59 -0400
Subject: [PATCH v2.0 12/17] bufmgr: Use aio for StartReadBuffers()
Author:
Reviewed-By:
Discussion: https://postgr.es/m/
Backpatch:
---
src/include/storage/bufmgr.h | 25 ++-
src/backend/storage/buffer/bufmgr.c | 259 +++++++++++++++++-----------
2 files changed, 182 insertions(+), 102 deletions(-)
diff --git a/src/include/storage/bufmgr.h b/src/include/storage/bufmgr.h
index 6cd64b8c2b3..a075a40b2ed 100644
--- a/src/include/storage/bufmgr.h
+++ b/src/include/storage/bufmgr.h
@@ -15,6 +15,7 @@
#define BUFMGR_H
#include "port/pg_iovec.h"
+#include "storage/aio_ref.h"
#include "storage/block.h"
#include "storage/buf.h"
#include "storage/bufpage.h"
@@ -107,11 +108,22 @@ typedef struct BufferManagerRelation
#define BMR_REL(p_rel) ((BufferManagerRelation){.rel = p_rel})
#define BMR_SMGR(p_smgr, p_relpersistence) ((BufferManagerRelation){.smgr = p_smgr, .relpersistence = p_relpersistence})
+
+#define MAX_IO_COMBINE_LIMIT PG_IOV_MAX
+#define DEFAULT_IO_COMBINE_LIMIT Min(MAX_IO_COMBINE_LIMIT, (128 * 1024) / BLCKSZ)
+
+
/* Zero out page if reading fails. */
#define READ_BUFFERS_ZERO_ON_ERROR (1 << 0)
/* Call smgrprefetch() if I/O necessary. */
#define READ_BUFFERS_ISSUE_ADVICE (1 << 1)
+/*
+ * FIXME: PgAioReturn is defined in aio.h. It'd be much better if we didn't
+ * need to include that here. Perhaps this could live in a separate header?
+ */
+#include "storage/aio.h"
+
struct ReadBuffersOperation
{
/* The following members should be set by the caller. */
@@ -131,6 +143,17 @@ struct ReadBuffersOperation
int flags;
int16 nblocks;
int16 io_buffers_len;
+
+ /*
+ * In some rare-ish cases one operation causes multiple reads (e.g. if a
+ * buffer was concurrently read by another backend). It'd be much better
+ * if we ensured that each ReadBuffersOperation covered only one IO - but
+ * that's not entirely trivial, due to having pinned victim buffers before
+ * starting IOs.
+ */
+ int16 nios;
+ PgAioHandleRef refs[MAX_IO_COMBINE_LIMIT];
+ PgAioReturn returns[MAX_IO_COMBINE_LIMIT];
};
typedef struct ReadBuffersOperation ReadBuffersOperation;
@@ -161,8 +184,6 @@ extern PGDLLIMPORT bool track_io_timing;
extern PGDLLIMPORT int effective_io_concurrency;
extern PGDLLIMPORT int maintenance_io_concurrency;
-#define MAX_IO_COMBINE_LIMIT PG_IOV_MAX
-#define DEFAULT_IO_COMBINE_LIMIT Min(MAX_IO_COMBINE_LIMIT, (128 * 1024) / BLCKSZ)
extern PGDLLIMPORT int io_combine_limit;
extern PGDLLIMPORT int checkpoint_flush_after;
diff --git a/src/backend/storage/buffer/bufmgr.c b/src/backend/storage/buffer/bufmgr.c
index 8feafd6e53c..90e873d278f 100644
--- a/src/backend/storage/buffer/bufmgr.c
+++ b/src/backend/storage/buffer/bufmgr.c
@@ -1280,6 +1280,12 @@ ReadBuffer_common(Relation rel, SMgrRelation smgr, char smgr_persistence,
return buffer;
}
+static bool AsyncReadBuffers(ReadBuffersOperation *operation,
+ Buffer *buffers,
+ BlockNumber blockNum,
+ int *nblocks,
+ int flags);
+
static pg_attribute_always_inline bool
StartReadBuffersImpl(ReadBuffersOperation *operation,
Buffer *buffers,
@@ -1315,6 +1321,12 @@ StartReadBuffersImpl(ReadBuffersOperation *operation,
* so we stop here.
*/
actual_nblocks = i + 1;
+
+ ereport(DEBUG3,
+ errmsg("found buf %d, idx %i: %s, data %p",
+ buffers[i], i, DebugPrintBufferRefcount(buffers[i]),
+ BufferGetBlock(buffers[i])),
+ errhidestmt(true), errhidecontext(true));
break;
}
else
@@ -1352,27 +1364,18 @@ StartReadBuffersImpl(ReadBuffersOperation *operation,
operation->nblocks = actual_nblocks;
operation->io_buffers_len = io_buffers_len;
- if (flags & READ_BUFFERS_ISSUE_ADVICE)
- {
- /*
- * In theory we should only do this if PinBufferForBlock() had to
- * allocate new buffers above. That way, if two calls to
- * StartReadBuffers() were made for the same blocks before
- * WaitReadBuffers(), only the first would issue the advice. That'd be
- * a better simulation of true asynchronous I/O, which would only
- * start the I/O once, but isn't done here for simplicity. Note also
- * that the following call might actually issue two advice calls if we
- * cross a segment boundary; in a true asynchronous version we might
- * choose to process only one real I/O at a time in that case.
- */
- smgrprefetch(operation->smgr,
- operation->forknum,
- blockNum,
- operation->io_buffers_len);
- }
+ operation->nios = 0;
- /* Indicate that WaitReadBuffers() should be called. */
- return true;
+ /*
+ * TODO: When called for synchronous IO execution, we probably should
+ * enter a dedicated fastpath here.
+ */
+
+ /* initiate the IO */
+ return AsyncReadBuffers(operation,
+ buffers,
+ blockNum,
+ nblocks, flags);
}
/*
@@ -1424,12 +1427,31 @@ StartReadBuffer(ReadBuffersOperation *operation,
}
static inline bool
-WaitReadBuffersCanStartIO(Buffer buffer, bool nowait)
+ReadBuffersCanStartIO(Buffer buffer, bool nowait)
{
if (BufferIsLocal(buffer))
{
BufferDesc *bufHdr = GetLocalBufferDescriptor(-buffer - 1);
+ /*
+ * The buffer could have IO in progress by another scan. Right now
+ * localbuf.c doesn't use IO_IN_PROGRESS, which is why we need this
+ * hack.
+ *
+ * AFIXME: localbuf.c should use IO_IN_PROGRESS / have an equivalent
+ * of StartBufferIO().
+ */
+ if (pgaio_io_ref_valid(&bufHdr->io_in_progress))
+ {
+ PgAioHandleRef ior = bufHdr->io_in_progress;
+
+ ereport(DEBUG3,
+ errmsg("waiting for temp buffer IO in CSIO"),
+ errhidestmt(true), errhidecontext(true));
+ pgaio_io_ref_wait(&ior);
+ return false;
+ }
+
return (pg_atomic_read_u32(&bufHdr->state) & BM_VALID) == 0;
}
else
@@ -1439,12 +1461,7 @@ WaitReadBuffersCanStartIO(Buffer buffer, bool nowait)
void
WaitReadBuffers(ReadBuffersOperation *operation)
{
- Buffer *buffers;
int nblocks;
- BlockNumber blocknum;
- ForkNumber forknum;
- IOContext io_context;
- IOObject io_object;
char persistence;
/*
@@ -1460,11 +1477,65 @@ WaitReadBuffers(ReadBuffersOperation *operation)
if (nblocks == 0)
return; /* nothing to do */
+ persistence = operation->persistence;
+
+ Assert(operation->nios > 0);
+
+ for (int i = 0; i < operation->nios; i++)
+ {
+ PgAioReturn *aio_ret;
+
+ pgaio_io_ref_wait(&operation->refs[i]);
+
+ aio_ret = &operation->returns[i];
+
+ if (aio_ret->result.status != ARS_OK)
+ pgaio_result_log(aio_ret->result, &aio_ret->subject_data, ERROR);
+ }
+
+ /*
+ * We count all these blocks as read by this backend. This is traditional
+ * behavior, but might turn out to be not true if we find that someone
+ * else has beaten us and completed the read of some of these blocks. In
+ * that case the system globally double-counts, but we traditionally don't
+ * count this as a "hit", and we don't have a separate counter for "miss,
+ * but another backend completed the read".
+ */
+ if (persistence == RELPERSISTENCE_TEMP)
+ pgBufferUsage.local_blks_read += nblocks;
+ else
+ pgBufferUsage.shared_blks_read += nblocks;
+
+ if (VacuumCostActive)
+ VacuumCostBalance += VacuumCostPageMiss * nblocks;
+
+ /* FIXME: io timing */
+ /* FIXME: READ_DONE tracepoint */
+}
+
+static bool
+AsyncReadBuffers(ReadBuffersOperation *operation,
+ Buffer *buffers,
+ BlockNumber blockNum,
+ int *nblocks,
+ int flags)
+{
+ int io_buffers_len = 0;
+ BlockNumber blocknum;
+ ForkNumber forknum;
+ IOContext io_context;
+ IOObject io_object;
+ char persistence;
+ bool did_start_io_overall = false;
+ PgAioHandle *ioh = NULL;
+
buffers = &operation->buffers[0];
blocknum = operation->blocknum;
forknum = operation->forknum;
- persistence = operation->persistence;
+ persistence = operation->rel
+ ? operation->rel->rd_rel->relpersistence
+ : RELPERSISTENCE_PERMANENT;
if (persistence == RELPERSISTENCE_TEMP)
{
io_context = IOCONTEXT_NORMAL;
@@ -1485,25 +1556,33 @@ WaitReadBuffers(ReadBuffersOperation *operation)
* but another backend completed the read".
*/
if (persistence == RELPERSISTENCE_TEMP)
- pgBufferUsage.local_blks_read += nblocks;
+ pgBufferUsage.local_blks_read += *nblocks;
else
- pgBufferUsage.shared_blks_read += nblocks;
+ pgBufferUsage.shared_blks_read += *nblocks;
- for (int i = 0; i < nblocks; ++i)
+ for (int i = 0; i < *nblocks; ++i)
{
- int io_buffers_len;
- Buffer io_buffers[MAX_IO_COMBINE_LIMIT];
void *io_pages[MAX_IO_COMBINE_LIMIT];
- instr_time io_start;
+ Buffer io_buffers[MAX_IO_COMBINE_LIMIT];
BlockNumber io_first_block;
+ bool did_start_io_this = false;
+
+ /*
+ * Get IO before ReadBuffersCanStartIO, as pgaio_io_get() might block,
+ * which we don't want after setting IO_IN_PROGRESS.
+ */
+ if (likely(!ioh))
+ ioh = pgaio_io_get(CurrentResourceOwner, &operation->returns[operation->nios]);
/*
* Skip this block if someone else has already completed it. If an
* I/O is already in progress in another backend, this will wait for
* the outcome: either done, or something went wrong and we will
* retry.
+ *
+ * ATODO: Should we wait if we already submitted another IO?
*/
- if (!WaitReadBuffersCanStartIO(buffers[i], false))
+ if (!ReadBuffersCanStartIO(buffers[i], did_start_io_overall))
{
/*
* Report this as a 'hit' for this backend, even though it must
@@ -1515,6 +1594,10 @@ WaitReadBuffers(ReadBuffersOperation *operation)
operation->smgr->smgr_rlocator.locator.relNumber,
operation->smgr->smgr_rlocator.backend,
true);
+
+ ereport(DEBUG3,
+ errmsg("can't start io for first buffer %u", buffers[i]),
+ errhidestmt(true), errhidecontext(true));
continue;
}
@@ -1524,6 +1607,11 @@ WaitReadBuffers(ReadBuffersOperation *operation)
io_first_block = blocknum + i;
io_buffers_len = 1;
+ ereport(DEBUG3,
+ errmsg("first prepped for io: %s, offset %d",
+ DebugPrintBufferRefcount(io_buffers[0]), i),
+ errhidestmt(true), errhidecontext(true));
+
/*
* How many neighboring-on-disk blocks can we can scatter-read into
* other buffers at the same time? In this case we don't wait if we
@@ -1531,86 +1619,57 @@ WaitReadBuffers(ReadBuffersOperation *operation)
* for the head block, so we should get on with that I/O as soon as
* possible. We'll come back to this block again, above.
*/
- while ((i + 1) < nblocks &&
- WaitReadBuffersCanStartIO(buffers[i + 1], true))
+ while ((i + 1) < *nblocks &&
+ ReadBuffersCanStartIO(buffers[i + 1], true))
{
/* Must be consecutive block numbers. */
Assert(BufferGetBlockNumber(buffers[i + 1]) ==
BufferGetBlockNumber(buffers[i]) + 1);
+ ereport(DEBUG3,
+ errmsg("seq prepped for io: %s, offset %d",
+ DebugPrintBufferRefcount(buffers[i + 1]),
+ i + 1),
+ errhidestmt(true), errhidecontext(true));
+
io_buffers[io_buffers_len] = buffers[++i];
io_pages[io_buffers_len++] = BufferGetBlock(buffers[i]);
}
- io_start = pgstat_prepare_io_time(track_io_timing);
- smgrreadv(operation->smgr, forknum, io_first_block, io_pages, io_buffers_len);
- pgstat_count_io_op_time(io_object, io_context, IOOP_READ, io_start,
- io_buffers_len);
+ pgaio_io_get_ref(ioh, &operation->refs[operation->nios]);
- /* Verify each block we read, and terminate the I/O. */
- for (int j = 0; j < io_buffers_len; ++j)
+ pgaio_io_set_io_data_32(ioh, (uint32 *) io_buffers, io_buffers_len);
+ if (persistence == RELPERSISTENCE_TEMP)
{
- BufferDesc *bufHdr;
- Block bufBlock;
-
- if (persistence == RELPERSISTENCE_TEMP)
- {
- bufHdr = GetLocalBufferDescriptor(-io_buffers[j] - 1);
- bufBlock = LocalBufHdrGetBlock(bufHdr);
- }
- else
- {
- bufHdr = GetBufferDescriptor(io_buffers[j] - 1);
- bufBlock = BufHdrGetBlock(bufHdr);
- }
-
- /* check for garbage data */
- if (!PageIsVerifiedExtended((Page) bufBlock, io_first_block + j,
- PIV_LOG_WARNING | PIV_REPORT_STAT))
- {
- if ((operation->flags & READ_BUFFERS_ZERO_ON_ERROR) || zero_damaged_pages)
- {
- ereport(WARNING,
- (errcode(ERRCODE_DATA_CORRUPTED),
- errmsg("invalid page in block %u of relation %s; zeroing out page",
- io_first_block + j,
- relpath(operation->smgr->smgr_rlocator, forknum))));
- memset(bufBlock, 0, BLCKSZ);
- }
- else
- ereport(ERROR,
- (errcode(ERRCODE_DATA_CORRUPTED),
- errmsg("invalid page in block %u of relation %s",
- io_first_block + j,
- relpath(operation->smgr->smgr_rlocator, forknum))));
- }
-
- /* Terminate I/O and set BM_VALID. */
- if (persistence == RELPERSISTENCE_TEMP)
- {
- uint32 buf_state = pg_atomic_read_u32(&bufHdr->state);
-
- buf_state |= BM_VALID;
- pg_atomic_unlocked_write_u32(&bufHdr->state, buf_state);
- }
- else
- {
- /* Set BM_VALID, terminate IO, and wake up any waiters */
- TerminateBufferIO(bufHdr, false, BM_VALID, true, true);
- }
-
- /* Report I/Os as completing individually. */
- TRACE_POSTGRESQL_BUFFER_READ_DONE(forknum, io_first_block + j,
- operation->smgr->smgr_rlocator.locator.spcOid,
- operation->smgr->smgr_rlocator.locator.dbOid,
- operation->smgr->smgr_rlocator.locator.relNumber,
- operation->smgr->smgr_rlocator.backend,
- false);
+ pgaio_io_add_shared_cb(ioh, ASC_LOCAL_BUFFER_READ);
+ pgaio_io_set_flag(ioh, AHF_REFERENCES_LOCAL);
}
+ else
+ pgaio_io_add_shared_cb(ioh, ASC_SHARED_BUFFER_READ);
- if (VacuumCostActive)
- VacuumCostBalance += VacuumCostPageMiss * io_buffers_len;
+ did_start_io_overall = did_start_io_this = true;
+ smgrstartreadv(ioh, operation->smgr, forknum, io_first_block,
+ io_pages, io_buffers_len);
+ ioh = NULL;
+ operation->nios++;
+
+ /* not obvious what we'd use for time */
+ pgstat_count_io_op_n(io_object, io_context, IOOP_READ, io_buffers_len);
}
+
+ if (ioh)
+ {
+ pgaio_io_release(ioh);
+ ioh = NULL;
+ }
+
+ if (did_start_io_overall)
+ {
+ pgaio_submit_staged();
+ return true;
+ }
+ else
+ return false;
}
/*
--
2.45.2.827.g557ae147e6
v2.0-0013-aio-Very-WIP-read_stream.c-adjustments-for-real.patch (text/x-diff, 4.5K) download
From 6fcd84b237df81097a4271198e380dd82c76757b Mon Sep 17 00:00:00 2001
From: Andres Freund <andres@anarazel.de>
Date: Sat, 31 Aug 2024 21:39:30 -0400
Subject: [PATCH v2.0 13/17] aio: Very-WIP: read_stream.c adjustments for real
AIO
---
src/include/storage/bufmgr.h | 2 ++
src/backend/storage/aio/read_stream.c | 29 +++++++++++++++++++++------
src/backend/storage/buffer/bufmgr.c | 3 ++-
3 files changed, 27 insertions(+), 7 deletions(-)
diff --git a/src/include/storage/bufmgr.h b/src/include/storage/bufmgr.h
index a075a40b2ed..ac6496bb1eb 100644
--- a/src/include/storage/bufmgr.h
+++ b/src/include/storage/bufmgr.h
@@ -117,6 +117,8 @@ typedef struct BufferManagerRelation
#define READ_BUFFERS_ZERO_ON_ERROR (1 << 0)
/* Call smgrprefetch() if I/O necessary. */
#define READ_BUFFERS_ISSUE_ADVICE (1 << 1)
+/* caller will issue more io, don't submit */
+#define READ_BUFFERS_MORE_MORE_MORE (1 << 2)
/*
* FIXME: PgAioReturn is defined in aio.h. It'd be much better if we didn't
diff --git a/src/backend/storage/aio/read_stream.c b/src/backend/storage/aio/read_stream.c
index 93cdd35fea0..42b2434918b 100644
--- a/src/backend/storage/aio/read_stream.c
+++ b/src/backend/storage/aio/read_stream.c
@@ -90,6 +90,7 @@
#include "catalog/pg_tablespace.h"
#include "miscadmin.h"
+#include "storage/aio.h"
#include "storage/fd.h"
#include "storage/smgr.h"
#include "storage/read_stream.h"
@@ -223,14 +224,18 @@ read_stream_start_pending_read(ReadStream *stream, bool suppress_advice)
/*
* If advice hasn't been suppressed, this system supports it, and this
* isn't a strictly sequential pattern, then we'll issue advice.
+ *
+ * XXX: Used to also check stream->pending_read_blocknum !=
+ * stream->seq_blocknum
*/
if (!suppress_advice &&
- stream->advice_enabled &&
- stream->pending_read_blocknum != stream->seq_blocknum)
+ stream->advice_enabled)
flags = READ_BUFFERS_ISSUE_ADVICE;
else
flags = 0;
+ flags |= READ_BUFFERS_MORE_MORE_MORE;
+
/* We say how many blocks we want to read, but may be smaller on return. */
buffer_index = stream->next_buffer_index;
io_index = stream->next_io_index;
@@ -289,6 +294,14 @@ read_stream_start_pending_read(ReadStream *stream, bool suppress_advice)
static void
read_stream_look_ahead(ReadStream *stream, bool suppress_advice)
{
+ if (stream->distance > (io_combine_limit * 8))
+ {
+ if (stream->pinned_buffers + stream->pending_read_nblocks > ((stream->distance * 3) / 4))
+ {
+ return;
+ }
+ }
+
while (stream->ios_in_progress < stream->max_ios &&
stream->pinned_buffers + stream->pending_read_nblocks < stream->distance)
{
@@ -338,6 +351,7 @@ read_stream_look_ahead(ReadStream *stream, bool suppress_advice)
{
/* And we've hit the limit. Rewind, and stop here. */
read_stream_unget_block(stream, blocknum);
+ pgaio_submit_staged();
return;
}
}
@@ -362,6 +376,8 @@ read_stream_look_ahead(ReadStream *stream, bool suppress_advice)
stream->distance == 0) &&
stream->ios_in_progress < stream->max_ios)
read_stream_start_pending_read(stream, suppress_advice);
+
+ pgaio_submit_staged();
}
/*
@@ -476,10 +492,11 @@ read_stream_begin_impl(int flags,
* direct I/O isn't enabled, the caller hasn't promised sequential access
* (overriding our detection heuristics), and max_ios hasn't been set to
* zero.
+ *
+ * FIXME: Used to also check (io_direct_flags & IO_DIRECT_DATA) == 0 &&
+ * (flags & READ_STREAM_SEQUENTIAL) == 0
*/
- if ((io_direct_flags & IO_DIRECT_DATA) == 0 &&
- (flags & READ_STREAM_SEQUENTIAL) == 0 &&
- max_ios > 0)
+ if (max_ios > 0)
stream->advice_enabled = true;
#endif
@@ -710,7 +727,7 @@ read_stream_next_buffer(ReadStream *stream, void **per_buffer_data)
if (++stream->oldest_io_index == stream->max_ios)
stream->oldest_io_index = 0;
- if (stream->ios[io_index].op.flags & READ_BUFFERS_ISSUE_ADVICE)
+ if (stream->ios[io_index].op.flags & (READ_BUFFERS_ISSUE_ADVICE | READ_BUFFERS_MORE_MORE_MORE))
{
/* Distance ramps up fast (behavior C). */
distance = stream->distance * 2;
diff --git a/src/backend/storage/buffer/bufmgr.c b/src/backend/storage/buffer/bufmgr.c
index 90e873d278f..59f4b22457d 100644
--- a/src/backend/storage/buffer/bufmgr.c
+++ b/src/backend/storage/buffer/bufmgr.c
@@ -1665,7 +1665,8 @@ AsyncReadBuffers(ReadBuffersOperation *operation,
if (did_start_io_overall)
{
- pgaio_submit_staged();
+ if (!(flags & READ_BUFFERS_MORE_MORE_MORE))
+ pgaio_submit_staged();
return true;
}
else
--
2.45.2.827.g557ae147e6
v2.0-0014-aio-Add-IO-queue-helper.patch (text/x-diff, 7.2K) download
From 2df34d8ac4fa381da607358ad3d214aadd05fdc7 Mon Sep 17 00:00:00 2001
From: Andres Freund <andres@anarazel.de>
Date: Tue, 23 Jul 2024 10:00:06 -0700
Subject: [PATCH v2.0 14/17] aio: Add IO queue helper
This is likely never going to anywhere - Thomas Munro is working on something
more complete. But I needed a way to exercise aio for checkpointer / bgwriter.
---
src/include/storage/io_queue.h | 33 +++++
src/backend/storage/aio/Makefile | 1 +
src/backend/storage/aio/io_queue.c | 195 ++++++++++++++++++++++++++++
src/backend/storage/aio/meson.build | 1 +
src/tools/pgindent/typedefs.list | 2 +
5 files changed, 232 insertions(+)
create mode 100644 src/include/storage/io_queue.h
create mode 100644 src/backend/storage/aio/io_queue.c
diff --git a/src/include/storage/io_queue.h b/src/include/storage/io_queue.h
new file mode 100644
index 00000000000..28077158d6d
--- /dev/null
+++ b/src/include/storage/io_queue.h
@@ -0,0 +1,33 @@
+/*-------------------------------------------------------------------------
+ *
+ * io_queue.h
+ * Mechanism for tracking many IOs
+ *
+ *
+ * Portions Copyright (c) 1996-2024, PostgreSQL Global Development Group
+ * Portions Copyright (c) 1994, Regents of the University of California
+ *
+ * src/include/storage/io_queue.h
+ *
+ *-------------------------------------------------------------------------
+ */
+#ifndef IO_QUEUE_H
+#define IO_QUEUE_H
+
+#include "storage/bufmgr.h"
+
+struct IOQueue;
+typedef struct IOQueue IOQueue;
+
+struct PgAioHandleRef;
+
+extern IOQueue *io_queue_create(int depth, int flags);
+extern void io_queue_track(IOQueue *ioq, const struct PgAioHandleRef *ior);
+extern void io_queue_wait_one(IOQueue *ioq);
+extern void io_queue_wait_all(IOQueue *ioq);
+extern bool io_queue_is_empty(IOQueue *ioq);
+extern void io_queue_reserve(IOQueue *ioq);
+extern struct PgAioHandle *io_queue_get_io(IOQueue *ioq);
+extern void io_queue_free(IOQueue *ioq);
+
+#endif /* IO_QUEUE_H */
diff --git a/src/backend/storage/aio/Makefile b/src/backend/storage/aio/Makefile
index 2a5e72a8024..3fb527ed0d1 100644
--- a/src/backend/storage/aio/Makefile
+++ b/src/backend/storage/aio/Makefile
@@ -13,6 +13,7 @@ OBJS = \
aio_io.o \
aio_init.o \
aio_subject.o \
+ io_queue.o \
method_worker.o \
method_io_uring.o \
read_stream.o
diff --git a/src/backend/storage/aio/io_queue.c b/src/backend/storage/aio/io_queue.c
new file mode 100644
index 00000000000..4dda2f4e20e
--- /dev/null
+++ b/src/backend/storage/aio/io_queue.c
@@ -0,0 +1,195 @@
+/*-------------------------------------------------------------------------
+ *
+ * io_queue.c
+ * Mechanism for tracking many IOs
+ *
+ * Portions Copyright (c) 2024, PostgreSQL Global Development Group
+ * Portions Copyright (c) 1994, Regents of the University of California
+ *
+ * IDENTIFICATION
+ * src/backend/storage/aio/io_queue.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#include "storage/io_queue.h"
+
+#include "storage/aio.h"
+
+
+typedef struct TrackedIO
+{
+ PgAioHandleRef ior;
+ dlist_node node;
+} TrackedIO;
+
+struct IOQueue
+{
+ int depth;
+ int unsubmitted;
+
+ bool has_reserved;
+
+ dclist_head idle;
+ dclist_head in_progress;
+
+ TrackedIO tracked_ios[FLEXIBLE_ARRAY_MEMBER];
+};
+
+
+IOQueue *
+io_queue_create(int depth, int flags)
+{
+ size_t sz;
+ IOQueue *ioq;
+
+ sz = offsetof(IOQueue, tracked_ios)
+ + sizeof(TrackedIO) * depth;
+
+ ioq = palloc0(sz);
+
+ ioq->depth = 0;
+
+ for (int i = 0; i < depth; i++)
+ {
+ TrackedIO *tio = &ioq->tracked_ios[i];
+
+ pgaio_io_ref_clear(&tio->ior);
+ dclist_push_tail(&ioq->idle, &tio->node);
+ }
+
+ return ioq;
+}
+
+void
+io_queue_wait_one(IOQueue *ioq)
+{
+ while (!dclist_is_empty(&ioq->in_progress))
+ {
+ /* FIXME: Should we really pop here already? */
+ dlist_node *node = dclist_pop_head_node(&ioq->in_progress);
+ TrackedIO *tio = dclist_container(TrackedIO, node, node);
+
+ pgaio_io_ref_wait(&tio->ior);
+ dclist_push_head(&ioq->idle, &tio->node);
+ }
+}
+
+void
+io_queue_reserve(IOQueue *ioq)
+{
+ if (ioq->has_reserved)
+ return;
+
+ if (dclist_is_empty(&ioq->idle))
+ io_queue_wait_one(ioq);
+
+ Assert(!dclist_is_empty(&ioq->idle));
+
+ ioq->has_reserved = true;
+}
+
+PgAioHandle *
+io_queue_get_io(IOQueue *ioq)
+{
+ PgAioHandle *ioh;
+
+ io_queue_reserve(ioq);
+
+ Assert(!dclist_is_empty(&ioq->idle));
+
+ if (!io_queue_is_empty(ioq))
+ {
+ ioh = pgaio_io_get_nb(CurrentResourceOwner, NULL);
+ if (ioh == NULL)
+ {
+ /*
+ * Need to wait for all IOs, blocking might not be legal in the
+ * context.
+ *
+ * XXX: This doesn't make a whole lot of sense, we're also
+ * blocking here. What was I smoking when I wrote the above?
+ */
+ io_queue_wait_all(ioq);
+ ioh = pgaio_io_get(CurrentResourceOwner, NULL);
+ }
+ }
+ else
+ {
+ ioh = pgaio_io_get(CurrentResourceOwner, NULL);
+ }
+
+ return ioh;
+}
+
+void
+io_queue_track(IOQueue *ioq, const struct PgAioHandleRef *ior)
+{
+ dlist_node *node;
+ TrackedIO *tio;
+
+ Assert(ioq->has_reserved);
+ ioq->has_reserved = false;
+
+ Assert(!dclist_is_empty(&ioq->idle));
+
+ node = dclist_pop_head_node(&ioq->idle);
+ tio = dclist_container(TrackedIO, node, node);
+
+ tio->ior = *ior;
+
+ dclist_push_tail(&ioq->in_progress, &tio->node);
+
+ ioq->unsubmitted++;
+
+ /*
+ * XXX: Should have some smarter logic here. We don't want to wait too
+ * long to submit, that'll mean we're more likely to block. But we also
+ * don't want to have the overhead of submitting every IO individually.
+ */
+ if (ioq->unsubmitted >= 4)
+ {
+ pgaio_submit_staged();
+ ioq->unsubmitted = 0;
+ }
+}
+
+void
+io_queue_wait_all(IOQueue *ioq)
+{
+ while (!dclist_is_empty(&ioq->in_progress))
+ {
+ /* wait for the last IO to minimize unnecessary wakeups */
+ dlist_node *node = dclist_tail_node(&ioq->in_progress);
+ TrackedIO *tio = dclist_container(TrackedIO, node, node);
+
+ if (!pgaio_io_ref_check_done(&tio->ior))
+ {
+ ereport(DEBUG3,
+ errmsg("io_queue_wait_all for io:%d",
+ pgaio_io_ref_get_id(&tio->ior)),
+ errhidestmt(true),
+ errhidecontext(true));
+
+ pgaio_io_ref_wait(&tio->ior);
+ }
+
+ dclist_delete_from(&ioq->in_progress, &tio->node);
+ dclist_push_head(&ioq->idle, &tio->node);
+ }
+}
+
+bool
+io_queue_is_empty(IOQueue *ioq)
+{
+ return dclist_is_empty(&ioq->in_progress);
+}
+
+void
+io_queue_free(IOQueue *ioq)
+{
+ io_queue_wait_all(ioq);
+
+ pfree(ioq);
+}
diff --git a/src/backend/storage/aio/meson.build b/src/backend/storage/aio/meson.build
index 8960223194a..6d64c75a49c 100644
--- a/src/backend/storage/aio/meson.build
+++ b/src/backend/storage/aio/meson.build
@@ -5,6 +5,7 @@ backend_sources += files(
'aio_io.c',
'aio_init.c',
'aio_subject.c',
+ 'io_queue.c',
'method_io_uring.c',
'method_worker.c',
'read_stream.c',
diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list
index be8be9fbff0..6f39abcdf3c 100644
--- a/src/tools/pgindent/typedefs.list
+++ b/src/tools/pgindent/typedefs.list
@@ -1171,6 +1171,7 @@ IOContext
IOFuncSelector
IOObject
IOOp
+IOQueue
IO_STATUS_BLOCK
IPCompareMethod
ITEM
@@ -2959,6 +2960,7 @@ TocEntry
TokenAuxData
TokenizedAuthLine
TrackItem
+TrackedIO
TransApplyAction
TransInvalidationInfo
TransState
--
2.45.2.827.g557ae147e6
v2.0-0015-bufmgr-use-AIO-in-checkpointer-bgwriter.patch (text/x-diff, 31.2K) download
From f7ad1fbd6a37434b67cb50916a5c28255d3a14eb Mon Sep 17 00:00:00 2001
From: Andres Freund <andres@anarazel.de>
Date: Tue, 23 Jul 2024 10:01:23 -0700
Subject: [PATCH v2.0 15/17] bufmgr: use AIO in checkpointer, bgwriter
This is far from ready - just included to be able to exercise AIO writes and
get some preliminary numbers. In all likelihood this will instead be based
ontop of work by Thomas Munro instead of the preceding commit.
---
src/include/postmaster/bgwriter.h | 3 +-
src/include/storage/buf_internals.h | 1 +
src/include/storage/bufmgr.h | 3 +-
src/include/storage/bufpage.h | 1 +
src/backend/postmaster/bgwriter.c | 25 +-
src/backend/postmaster/checkpointer.c | 12 +-
src/backend/storage/buffer/bufmgr.c | 588 +++++++++++++++++++++++---
src/backend/storage/page/bufpage.c | 10 +
src/tools/pgindent/typedefs.list | 1 +
9 files changed, 580 insertions(+), 64 deletions(-)
diff --git a/src/include/postmaster/bgwriter.h b/src/include/postmaster/bgwriter.h
index 407f26e5302..01a936fbc0a 100644
--- a/src/include/postmaster/bgwriter.h
+++ b/src/include/postmaster/bgwriter.h
@@ -31,7 +31,8 @@ extern void BackgroundWriterMain(char *startup_data, size_t startup_data_len) pg
extern void CheckpointerMain(char *startup_data, size_t startup_data_len) pg_attribute_noreturn();
extern void RequestCheckpoint(int flags);
-extern void CheckpointWriteDelay(int flags, double progress);
+struct IOQueue;
+extern void CheckpointWriteDelay(struct IOQueue *ioq, int flags, double progress);
extern bool ForwardSyncRequest(const FileTag *ftag, SyncRequestType type);
diff --git a/src/include/storage/buf_internals.h b/src/include/storage/buf_internals.h
index 5cfa7dbd1f1..9d3123663b3 100644
--- a/src/include/storage/buf_internals.h
+++ b/src/include/storage/buf_internals.h
@@ -21,6 +21,7 @@
#include "storage/buf.h"
#include "storage/bufmgr.h"
#include "storage/condition_variable.h"
+#include "storage/io_queue.h"
#include "storage/latch.h"
#include "storage/lwlock.h"
#include "storage/shmem.h"
diff --git a/src/include/storage/bufmgr.h b/src/include/storage/bufmgr.h
index ac6496bb1eb..a65888c8915 100644
--- a/src/include/storage/bufmgr.h
+++ b/src/include/storage/bufmgr.h
@@ -325,7 +325,8 @@ extern bool ConditionalLockBufferForCleanup(Buffer buffer);
extern bool IsBufferCleanupOK(Buffer buffer);
extern bool HoldingBufferPinThatDelaysRecovery(void);
-extern bool BgBufferSync(struct WritebackContext *wb_context);
+struct IOQueue;
+extern bool BgBufferSync(struct IOQueue *ioq, struct WritebackContext *wb_context);
extern void LimitAdditionalPins(uint32 *additional_pins);
extern void LimitAdditionalLocalPins(uint32 *additional_pins);
diff --git a/src/include/storage/bufpage.h b/src/include/storage/bufpage.h
index 5999e5ca5a5..f5f5adb066d 100644
--- a/src/include/storage/bufpage.h
+++ b/src/include/storage/bufpage.h
@@ -509,5 +509,6 @@ extern bool PageIndexTupleOverwrite(Page page, OffsetNumber offnum,
Item newtup, Size newsize);
extern char *PageSetChecksumCopy(Page page, BlockNumber blkno);
extern void PageSetChecksumInplace(Page page, BlockNumber blkno);
+extern bool PageNeedsChecksumCopy(Page page);
#endif /* BUFPAGE_H */
diff --git a/src/backend/postmaster/bgwriter.c b/src/backend/postmaster/bgwriter.c
index 0f75548759a..71c08da45db 100644
--- a/src/backend/postmaster/bgwriter.c
+++ b/src/backend/postmaster/bgwriter.c
@@ -38,10 +38,12 @@
#include "postmaster/auxprocess.h"
#include "postmaster/bgwriter.h"
#include "postmaster/interrupt.h"
+#include "storage/aio.h"
#include "storage/buf_internals.h"
#include "storage/bufmgr.h"
#include "storage/condition_variable.h"
#include "storage/fd.h"
+#include "storage/io_queue.h"
#include "storage/lwlock.h"
#include "storage/proc.h"
#include "storage/procsignal.h"
@@ -89,6 +91,7 @@ BackgroundWriterMain(char *startup_data, size_t startup_data_len)
sigjmp_buf local_sigjmp_buf;
MemoryContext bgwriter_context;
bool prev_hibernate;
+ IOQueue *ioq;
WritebackContext wb_context;
Assert(startup_data_len == 0);
@@ -130,6 +133,7 @@ BackgroundWriterMain(char *startup_data, size_t startup_data_len)
ALLOCSET_DEFAULT_SIZES);
MemoryContextSwitchTo(bgwriter_context);
+ ioq = io_queue_create(128, 0);
WritebackContextInit(&wb_context, &bgwriter_flush_after);
/*
@@ -167,6 +171,7 @@ BackgroundWriterMain(char *startup_data, size_t startup_data_len)
* about in bgwriter, but we do have LWLocks, buffers, and temp files.
*/
LWLockReleaseAll();
+ pgaio_at_error();
ConditionVariableCancelSleep();
UnlockBuffers();
ReleaseAuxProcessResources(false);
@@ -226,12 +231,27 @@ BackgroundWriterMain(char *startup_data, size_t startup_data_len)
/* Clear any already-pending wakeups */
ResetLatch(MyLatch);
+ /*
+ * XXX: Before exiting, wait for all IO to finish. That's only
+ * important to avoid spurious PrintBufferLeakWarning() /
+ * PrintAioIPLeakWarning() calls, triggered by
+ * ReleaseAuxProcessResources() being called with isCommit=true.
+ *
+ * FIXME: this is theoretically racy, but I didn't want to copy
+ * HandleMainLoopInterrupts() remaining body here.
+ */
+ if (ShutdownRequestPending)
+ {
+ io_queue_wait_all(ioq);
+ io_queue_free(ioq);
+ }
+
HandleMainLoopInterrupts();
/*
* Do one cycle of dirty-buffer writing.
*/
- can_hibernate = BgBufferSync(&wb_context);
+ can_hibernate = BgBufferSync(ioq, &wb_context);
/* Report pending statistics to the cumulative stats system */
pgstat_report_bgwriter();
@@ -248,6 +268,9 @@ BackgroundWriterMain(char *startup_data, size_t startup_data_len)
smgrdestroyall();
}
+ /* finish IO before sleeping, to avoid blocking other backends */
+ io_queue_wait_all(ioq);
+
/*
* Log a new xl_running_xacts every now and then so replication can
* get into a consistent state faster (think of suboverflowed
diff --git a/src/backend/postmaster/checkpointer.c b/src/backend/postmaster/checkpointer.c
index 199f008bcda..0350a71cab4 100644
--- a/src/backend/postmaster/checkpointer.c
+++ b/src/backend/postmaster/checkpointer.c
@@ -46,9 +46,11 @@
#include "postmaster/bgwriter.h"
#include "postmaster/interrupt.h"
#include "replication/syncrep.h"
+#include "storage/aio.h"
#include "storage/bufmgr.h"
#include "storage/condition_variable.h"
#include "storage/fd.h"
+#include "storage/io_queue.h"
#include "storage/ipc.h"
#include "storage/lwlock.h"
#include "storage/proc.h"
@@ -266,6 +268,7 @@ CheckpointerMain(char *startup_data, size_t startup_data_len)
* files.
*/
LWLockReleaseAll();
+ pgaio_at_error();
ConditionVariableCancelSleep();
pgstat_report_wait_end();
UnlockBuffers();
@@ -708,7 +711,7 @@ ImmediateCheckpointRequested(void)
* fraction between 0.0 meaning none, and 1.0 meaning all done.
*/
void
-CheckpointWriteDelay(int flags, double progress)
+CheckpointWriteDelay(IOQueue *ioq, int flags, double progress)
{
static int absorb_counter = WRITES_PER_ABSORB;
@@ -741,6 +744,13 @@ CheckpointWriteDelay(int flags, double progress)
/* Report interim statistics to the cumulative stats system */
pgstat_report_checkpointer();
+ /*
+ * Ensure all pending IO is submitted to avoid unnecessary delays for
+ * other processes.
+ */
+ io_queue_wait_all(ioq);
+
+
/*
* This sleep used to be connected to bgwriter_delay, typically 200ms.
* That resulted in more frequent wakeups if not much work to do.
diff --git a/src/backend/storage/buffer/bufmgr.c b/src/backend/storage/buffer/bufmgr.c
index 59f4b22457d..e62f2de2034 100644
--- a/src/backend/storage/buffer/bufmgr.c
+++ b/src/backend/storage/buffer/bufmgr.c
@@ -52,6 +52,7 @@
#include "storage/buf_internals.h"
#include "storage/bufmgr.h"
#include "storage/fd.h"
+#include "storage/io_queue.h"
#include "storage/ipc.h"
#include "storage/lmgr.h"
#include "storage/proc.h"
@@ -77,6 +78,7 @@
/* Bits in SyncOneBuffer's return value */
#define BUF_WRITTEN 0x01
#define BUF_REUSABLE 0x02
+#define BUF_CANT_MERGE 0x04
#define RELS_BSEARCH_THRESHOLD 20
@@ -538,8 +540,6 @@ static void UnpinBuffer(BufferDesc *buf);
static void UnpinBufferNoOwner(BufferDesc *buf);
static void BufferSync(int flags);
static uint32 WaitBufHdrUnlocked(BufferDesc *buf);
-static int SyncOneBuffer(int buf_id, bool skip_recently_used,
- WritebackContext *wb_context);
static void WaitIO(BufferDesc *buf);
static bool StartBufferIO(BufferDesc *buf, bool forInput, bool nowait);
static void TerminateBufferIO(BufferDesc *buf, bool clear_dirty,
@@ -557,6 +557,7 @@ static inline BufferDesc *BufferAlloc(SMgrRelation smgr,
static Buffer GetVictimBuffer(BufferAccessStrategy strategy, IOContext io_context);
static void FlushBuffer(BufferDesc *buf, SMgrRelation reln,
IOObject io_object, IOContext io_context);
+
static void FindAndDropRelationBuffers(RelFileLocator rlocator,
ForkNumber forkNum,
BlockNumber nForkBlock,
@@ -2981,6 +2982,56 @@ UnpinBufferNoOwner(BufferDesc *buf)
}
}
+typedef struct BuffersToWrite
+{
+ int nbuffers;
+ BufferTag start_at_tag;
+ uint32 max_combine;
+
+ XLogRecPtr max_lsn;
+
+ PgAioHandle *ioh;
+ PgAioHandleRef ior;
+
+ uint64 total_writes;
+
+ Buffer buffers[IOV_MAX];
+ PgAioBounceBuffer *bounce_buffers[IOV_MAX];
+ const void *data_ptrs[IOV_MAX];
+} BuffersToWrite;
+
+static int PrepareToWriteBuffer(BuffersToWrite *to_write, Buffer buf,
+ bool skip_recently_used,
+ IOQueue *ioq, WritebackContext *wb_context);
+
+static void WriteBuffers(BuffersToWrite *to_write,
+ IOQueue *ioq, WritebackContext *wb_context);
+
+static void
+BuffersToWriteInit(BuffersToWrite *to_write,
+ IOQueue *ioq, WritebackContext *wb_context)
+{
+ to_write->total_writes = 0;
+ to_write->nbuffers = 0;
+ to_write->ioh = NULL;
+ pgaio_io_ref_clear(&to_write->ior);
+ to_write->max_lsn = InvalidXLogRecPtr;
+}
+
+static void
+BuffersToWriteEnd(BuffersToWrite *to_write)
+{
+ if (to_write->ioh != NULL)
+ {
+ pgaio_io_release(to_write->ioh);
+ to_write->ioh = NULL;
+ }
+
+ if (to_write->total_writes > 0)
+ pgaio_submit_staged();
+}
+
+
#define ST_SORT sort_checkpoint_bufferids
#define ST_ELEMENT_TYPE CkptSortItem
#define ST_COMPARE(a, b) ckpt_buforder_comparator(a, b)
@@ -3012,7 +3063,10 @@ BufferSync(int flags)
binaryheap *ts_heap;
int i;
int mask = BM_DIRTY;
+ IOQueue *ioq;
WritebackContext wb_context;
+ BuffersToWrite to_write;
+ int max_combine;
/*
* Unless this is a shutdown checkpoint or we have been explicitly told,
@@ -3074,7 +3128,9 @@ BufferSync(int flags)
if (num_to_scan == 0)
return; /* nothing to do */
+ ioq = io_queue_create(512, 0);
WritebackContextInit(&wb_context, &checkpoint_flush_after);
+ max_combine = Min(io_bounce_buffers, io_combine_limit);
TRACE_POSTGRESQL_BUFFER_SYNC_START(NBuffers, num_to_scan);
@@ -3182,48 +3238,89 @@ BufferSync(int flags)
*/
num_processed = 0;
num_written = 0;
+
+ BuffersToWriteInit(&to_write, ioq, &wb_context);
+
while (!binaryheap_empty(ts_heap))
{
BufferDesc *bufHdr = NULL;
CkptTsStatus *ts_stat = (CkptTsStatus *)
DatumGetPointer(binaryheap_first(ts_heap));
- buf_id = CkptBufferIds[ts_stat->index].buf_id;
- Assert(buf_id != -1);
-
- bufHdr = GetBufferDescriptor(buf_id);
-
- num_processed++;
+ Assert(ts_stat->num_scanned <= ts_stat->num_to_scan);
/*
- * We don't need to acquire the lock here, because we're only looking
- * at a single bit. It's possible that someone else writes the buffer
- * and clears the flag right after we check, but that doesn't matter
- * since SyncOneBuffer will then do nothing. However, there is a
- * further race condition: it's conceivable that between the time we
- * examine the bit here and the time SyncOneBuffer acquires the lock,
- * someone else not only wrote the buffer but replaced it with another
- * page and dirtied it. In that improbable case, SyncOneBuffer will
- * write the buffer though we didn't need to. It doesn't seem worth
- * guarding against this, though.
+ * Collect a batch of buffers to write out from the current
+ * tablespace. That causes some imbalance between the tablespaces, but
+ * that's more than outweighed by the efficiency gain due to batching.
*/
- if (pg_atomic_read_u32(&bufHdr->state) & BM_CHECKPOINT_NEEDED)
+ while (to_write.nbuffers < max_combine &&
+ ts_stat->num_scanned < ts_stat->num_to_scan)
{
- if (SyncOneBuffer(buf_id, false, &wb_context) & BUF_WRITTEN)
+ buf_id = CkptBufferIds[ts_stat->index].buf_id;
+ Assert(buf_id != -1);
+
+ bufHdr = GetBufferDescriptor(buf_id);
+
+ num_processed++;
+
+ /*
+ * We don't need to acquire the lock here, because we're only
+ * looking at a single bit. It's possible that someone else writes
+ * the buffer and clears the flag right after we check, but that
+ * doesn't matter since SyncOneBuffer will then do nothing.
+ * However, there is a further race condition: it's conceivable
+ * that between the time we examine the bit here and the time
+ * SyncOneBuffer acquires the lock, someone else not only wrote
+ * the buffer but replaced it with another page and dirtied it. In
+ * that improbable case, SyncOneBuffer will write the buffer
+ * though we didn't need to. It doesn't seem worth guarding
+ * against this, though.
+ */
+ if (pg_atomic_read_u32(&bufHdr->state) & BM_CHECKPOINT_NEEDED)
{
- TRACE_POSTGRESQL_BUFFER_SYNC_WRITTEN(buf_id);
- PendingCheckpointerStats.buffers_written++;
- num_written++;
+ int result = PrepareToWriteBuffer(&to_write, buf_id + 1, false,
+ ioq, &wb_context);
+
+ if (result & BUF_CANT_MERGE)
+ {
+ Assert(to_write.nbuffers > 0);
+ WriteBuffers(&to_write, ioq, &wb_context);
+
+ result = PrepareToWriteBuffer(&to_write, buf_id + 1, false,
+ ioq, &wb_context);
+ Assert(result != BUF_CANT_MERGE);
+ }
+
+ if (result & BUF_WRITTEN)
+ {
+ TRACE_POSTGRESQL_BUFFER_SYNC_WRITTEN(buf_id);
+ PendingCheckpointerStats.buffers_written++;
+ num_written++;
+ }
+ else
+ {
+ break;
+ }
}
+ else
+ {
+ if (to_write.nbuffers > 0)
+ WriteBuffers(&to_write, ioq, &wb_context);
+ }
+
+ /*
+ * Measure progress independent of actually having to flush the
+ * buffer - otherwise writing become unbalanced.
+ */
+ ts_stat->progress += ts_stat->progress_slice;
+ ts_stat->num_scanned++;
+ ts_stat->index++;
}
- /*
- * Measure progress independent of actually having to flush the buffer
- * - otherwise writing become unbalanced.
- */
- ts_stat->progress += ts_stat->progress_slice;
- ts_stat->num_scanned++;
- ts_stat->index++;
+ if (to_write.nbuffers > 0)
+ WriteBuffers(&to_write, ioq, &wb_context);
+
/* Have all the buffers from the tablespace been processed? */
if (ts_stat->num_scanned == ts_stat->num_to_scan)
@@ -3241,15 +3338,23 @@ BufferSync(int flags)
*
* (This will check for barrier events even if it doesn't sleep.)
*/
- CheckpointWriteDelay(flags, (double) num_processed / num_to_scan);
+ CheckpointWriteDelay(ioq, flags, (double) num_processed / num_to_scan);
}
+ Assert(to_write.nbuffers == 0);
+ io_queue_wait_all(ioq);
+
/*
* Issue all pending flushes. Only checkpointer calls BufferSync(), so
* IOContext will always be IOCONTEXT_NORMAL.
*/
IssuePendingWritebacks(&wb_context, IOCONTEXT_NORMAL);
+ io_queue_wait_all(ioq); /* IssuePendingWritebacks might have added
+ * more */
+ io_queue_free(ioq);
+ BuffersToWriteEnd(&to_write);
+
pfree(per_ts_stat);
per_ts_stat = NULL;
binaryheap_free(ts_heap);
@@ -3275,7 +3380,7 @@ BufferSync(int flags)
* bgwriter_lru_maxpages to 0.)
*/
bool
-BgBufferSync(WritebackContext *wb_context)
+BgBufferSync(IOQueue *ioq, WritebackContext *wb_context)
{
/* info obtained from freelist.c */
int strategy_buf_id;
@@ -3318,6 +3423,8 @@ BgBufferSync(WritebackContext *wb_context)
long new_strategy_delta;
uint32 new_recent_alloc;
+ BuffersToWrite to_write;
+
/*
* Find out where the freelist clock sweep currently is, and how many
* buffer allocations have happened since our last call.
@@ -3494,11 +3601,25 @@ BgBufferSync(WritebackContext *wb_context)
num_written = 0;
reusable_buffers = reusable_buffers_est;
+ BuffersToWriteInit(&to_write, ioq, wb_context);
+
/* Execute the LRU scan */
while (num_to_scan > 0 && reusable_buffers < upcoming_alloc_est)
{
- int sync_state = SyncOneBuffer(next_to_clean, true,
- wb_context);
+ int sync_state;
+
+ sync_state = PrepareToWriteBuffer(&to_write, next_to_clean + 1,
+ true, ioq, wb_context);
+ if (sync_state & BUF_CANT_MERGE)
+ {
+ Assert(to_write.nbuffers > 0);
+
+ WriteBuffers(&to_write, ioq, wb_context);
+
+ sync_state = PrepareToWriteBuffer(&to_write, next_to_clean + 1,
+ true, ioq, wb_context);
+ Assert(sync_state != BUF_CANT_MERGE);
+ }
if (++next_to_clean >= NBuffers)
{
@@ -3509,6 +3630,13 @@ BgBufferSync(WritebackContext *wb_context)
if (sync_state & BUF_WRITTEN)
{
+ Assert(sync_state & BUF_REUSABLE);
+
+ if (to_write.nbuffers == io_combine_limit)
+ {
+ WriteBuffers(&to_write, ioq, wb_context);
+ }
+
reusable_buffers++;
if (++num_written >= bgwriter_lru_maxpages)
{
@@ -3520,6 +3648,11 @@ BgBufferSync(WritebackContext *wb_context)
reusable_buffers++;
}
+ if (to_write.nbuffers > 0)
+ WriteBuffers(&to_write, ioq, wb_context);
+
+ BuffersToWriteEnd(&to_write);
+
PendingBgWriterStats.buf_written_clean += num_written;
#ifdef BGW_DEBUG
@@ -3558,8 +3691,66 @@ BgBufferSync(WritebackContext *wb_context)
return (bufs_to_lap == 0 && recent_alloc == 0);
}
+static inline bool
+BufferTagsSameRel(const BufferTag *tag1, const BufferTag *tag2)
+{
+ return (tag1->spcOid == tag2->spcOid) &&
+ (tag1->dbOid == tag2->dbOid) &&
+ (tag1->relNumber == tag2->relNumber) &&
+ (tag1->forkNum == tag2->forkNum)
+ ;
+}
+
+static bool
+CanMergeWrite(BuffersToWrite *to_write, BufferDesc *cur_buf_hdr)
+{
+ BlockNumber cur_block = cur_buf_hdr->tag.blockNum;
+
+ Assert(to_write->nbuffers > 0); /* can't merge with nothing */
+ Assert(to_write->start_at_tag.relNumber != InvalidOid);
+ Assert(to_write->start_at_tag.blockNum != InvalidBlockNumber);
+
+ Assert(to_write->ioh != NULL);
+
+ /*
+ * First check if the blocknumber is one that we could actually merge,
+ * that's cheaper than checking the tablespace/db/relnumber/fork match.
+ */
+ if (to_write->start_at_tag.blockNum + to_write->nbuffers != cur_block)
+ return false;
+
+ if (!BufferTagsSameRel(&to_write->start_at_tag, &cur_buf_hdr->tag))
+ return false;
+
+ /*
+ * Need to check with smgr how large a write we're allowed to make. To
+ * reduce the overhead of the smgr check, only inquire once, when
+ * processing the first to-be-merged buffer. That avoids the overhead in
+ * the common case of writing out buffers that definitely not mergeable.
+ */
+ if (to_write->nbuffers == 1)
+ {
+ SMgrRelation smgr;
+
+ smgr = smgropen(BufTagGetRelFileLocator(&to_write->start_at_tag), INVALID_PROC_NUMBER);
+
+ to_write->max_combine = smgrmaxcombine(smgr,
+ to_write->start_at_tag.forkNum,
+ to_write->start_at_tag.blockNum);
+ }
+ else
+ {
+ Assert(to_write->max_combine > 0);
+ }
+
+ if (to_write->start_at_tag.blockNum + to_write->max_combine <= cur_block)
+ return false;
+
+ return true;
+}
+
/*
- * SyncOneBuffer -- process a single buffer during syncing.
+ * PrepareToWriteBuffer -- process a single buffer during syncing.
*
* If skip_recently_used is true, we don't write currently-pinned buffers, nor
* buffers marked recently used, as these are not replacement candidates.
@@ -3568,22 +3759,56 @@ BgBufferSync(WritebackContext *wb_context)
* BUF_WRITTEN: we wrote the buffer.
* BUF_REUSABLE: buffer is available for replacement, ie, it has
* pin count 0 and usage count 0.
+ * BUF_CANT_MERGE: can't combine this write with prior writes, caller needs
+ * to issue those first
*
* (BUF_WRITTEN could be set in error if FlushBuffer finds the buffer clean
* after locking it, but we don't care all that much.)
*/
static int
-SyncOneBuffer(int buf_id, bool skip_recently_used, WritebackContext *wb_context)
+PrepareToWriteBuffer(BuffersToWrite *to_write, Buffer buf,
+ bool skip_recently_used,
+ IOQueue *ioq, WritebackContext *wb_context)
{
- BufferDesc *bufHdr = GetBufferDescriptor(buf_id);
- int result = 0;
+ BufferDesc *cur_buf_hdr = GetBufferDescriptor(buf - 1);
uint32 buf_state;
- BufferTag tag;
+ int result = 0;
+ XLogRecPtr cur_buf_lsn;
+ LWLock *content_lock;
+ bool may_block;
+
+ /*
+ * Check if this buffer can be written out together with already prepared
+ * writes. We check before we have pinned the buffer, so the buffer can be
+ * written out and replaced between this check and us pinning the buffer -
+ * we'll recheck below. The reason for the pre-check is that we don't want
+ * to pin the buffer just to find out that we can't merge the IO.
+ */
+ if (to_write->nbuffers != 0)
+ {
+ if (!CanMergeWrite(to_write, cur_buf_hdr))
+ {
+ result |= BUF_CANT_MERGE;
+ return result;
+ }
+ }
+ else
+ {
+ if (to_write->ioh == NULL)
+ {
+ to_write->ioh = io_queue_get_io(ioq);
+ pgaio_io_get_ref(to_write->ioh, &to_write->ior);
+ }
+
+ to_write->start_at_tag = cur_buf_hdr->tag;
+ }
/* Make sure we can handle the pin */
ReservePrivateRefCountEntry();
ResourceOwnerEnlarge(CurrentResourceOwner);
+ /* XXX: Should also check if we are allowed to pin one more buffer */
+
/*
* Check whether buffer needs writing.
*
@@ -3593,7 +3818,7 @@ SyncOneBuffer(int buf_id, bool skip_recently_used, WritebackContext *wb_context)
* don't worry because our checkpoint.redo points before log record for
* upcoming changes and so we are not required to write such dirty buffer.
*/
- buf_state = LockBufHdr(bufHdr);
+ buf_state = LockBufHdr(cur_buf_hdr);
if (BUF_STATE_GET_REFCOUNT(buf_state) == 0 &&
BUF_STATE_GET_USAGECOUNT(buf_state) == 0)
@@ -3602,40 +3827,282 @@ SyncOneBuffer(int buf_id, bool skip_recently_used, WritebackContext *wb_context)
}
else if (skip_recently_used)
{
+#if 0
+ elog(LOG, "at block %d: skip recent with nbuffers %d",
+ cur_buf_hdr->tag.blockNum, to_write->nbuffers);
+#endif
/* Caller told us not to write recently-used buffers */
- UnlockBufHdr(bufHdr, buf_state);
+ UnlockBufHdr(cur_buf_hdr, buf_state);
return result;
}
if (!(buf_state & BM_VALID) || !(buf_state & BM_DIRTY))
{
/* It's clean, so nothing to do */
- UnlockBufHdr(bufHdr, buf_state);
+ UnlockBufHdr(cur_buf_hdr, buf_state);
return result;
}
- /*
- * Pin it, share-lock it, write it. (FlushBuffer will do nothing if the
- * buffer is clean by the time we've locked it.)
- */
- PinBuffer_Locked(bufHdr);
- LWLockAcquire(BufferDescriptorGetContentLock(bufHdr), LW_SHARED);
-
- FlushBuffer(bufHdr, NULL, IOOBJECT_RELATION, IOCONTEXT_NORMAL);
-
- LWLockRelease(BufferDescriptorGetContentLock(bufHdr));
-
- tag = bufHdr->tag;
-
- UnpinBuffer(bufHdr);
+ /* pin the buffer, from now on its identity can't change anymore */
+ PinBuffer_Locked(cur_buf_hdr);
/*
- * SyncOneBuffer() is only called by checkpointer and bgwriter, so
- * IOContext will always be IOCONTEXT_NORMAL.
+ * If we are merging, check if the buffer's identity possibly changed
+ * while we hadn't yet pinned it.
+ *
+ * XXX: It might be worth checking if we still want to write the buffer
+ * out, e.g. it could have been replaced with a buffer that doesn't have
+ * BM_CHECKPOINT_NEEDED set.
*/
- ScheduleBufferTagForWriteback(wb_context, IOCONTEXT_NORMAL, &tag);
+ if (to_write->nbuffers != 0)
+ {
+ if (!CanMergeWrite(to_write, cur_buf_hdr))
+ {
+ elog(LOG, "changed identity");
+ UnpinBuffer(cur_buf_hdr);
- return result | BUF_WRITTEN;
+ result |= BUF_CANT_MERGE;
+
+ return result;
+ }
+ }
+
+ may_block = to_write->nbuffers == 0
+ && !pgaio_have_staged()
+ && io_queue_is_empty(ioq)
+ ;
+ content_lock = BufferDescriptorGetContentLock(cur_buf_hdr);
+
+ if (!may_block)
+ {
+ if (LWLockConditionalAcquire(content_lock, LW_SHARED))
+ {
+ /* done */
+ }
+ else if (to_write->nbuffers == 0)
+ {
+ /*
+ * Need to wait for all prior IO to finish before blocking for
+ * lock acquisition, to avoid the risk a deadlock due to us
+ * waiting for another backend that is waiting for our unsubmitted
+ * IO to complete.
+ */
+ pgaio_submit_staged();
+ io_queue_wait_all(ioq);
+
+ elog(DEBUG2, "at block %u: can't block, nbuffers = 0",
+ cur_buf_hdr->tag.blockNum
+ );
+
+ may_block = to_write->nbuffers == 0
+ && !pgaio_have_staged()
+ && io_queue_is_empty(ioq)
+ ;
+ Assert(may_block);
+
+ LWLockAcquire(content_lock, LW_SHARED);
+ }
+ else
+ {
+ elog(DEBUG2, "at block %d: can't block nbuffers = %d",
+ cur_buf_hdr->tag.blockNum,
+ to_write->nbuffers);
+
+ UnpinBuffer(cur_buf_hdr);
+ result |= BUF_CANT_MERGE;
+ Assert(to_write->nbuffers > 0);
+
+ return result;
+ }
+ }
+ else
+ {
+ LWLockAcquire(content_lock, LW_SHARED);
+ }
+
+ if (!may_block)
+ {
+ if (!StartBufferIO(cur_buf_hdr, false, !may_block))
+ {
+ pgaio_submit_staged();
+ io_queue_wait_all(ioq);
+
+ may_block = io_queue_is_empty(ioq) && to_write->nbuffers == 0 && !pgaio_have_staged();
+
+ if (!StartBufferIO(cur_buf_hdr, false, !may_block))
+ {
+ elog(DEBUG2, "at block %d: non-waitable StartBufferIO returns false, %d",
+ cur_buf_hdr->tag.blockNum,
+ may_block);
+
+ /*
+ * FIXME: can't tell whether this is because the buffer has
+ * been cleaned
+ */
+ if (!may_block)
+ {
+ result |= BUF_CANT_MERGE;
+ Assert(to_write->nbuffers > 0);
+ }
+ LWLockRelease(content_lock);
+ UnpinBuffer(cur_buf_hdr);
+
+ return result;
+ }
+ }
+ }
+ else
+ {
+ if (!StartBufferIO(cur_buf_hdr, false, false))
+ {
+ elog(DEBUG2, "waitable StartBufferIO returns false");
+ LWLockRelease(content_lock);
+ UnpinBuffer(cur_buf_hdr);
+
+ /*
+ * FIXME: Historically we returned BUF_WRITTEN in this case, which
+ * seems wrong
+ */
+ return result;
+ }
+ }
+
+ /*
+ * Run PageGetLSN while holding header lock, since we don't have the
+ * buffer locked exclusively in all cases.
+ */
+ buf_state = LockBufHdr(cur_buf_hdr);
+
+ cur_buf_lsn = BufferGetLSN(cur_buf_hdr);
+
+ /* To check if block content changes while flushing. - vadim 01/17/97 */
+ buf_state &= ~BM_JUST_DIRTIED;
+
+ UnlockBufHdr(cur_buf_hdr, buf_state);
+
+ to_write->buffers[to_write->nbuffers] = buf;
+ to_write->nbuffers++;
+
+ if (buf_state & BM_PERMANENT &&
+ (to_write->max_lsn == InvalidXLogRecPtr || to_write->max_lsn < cur_buf_lsn))
+ {
+ to_write->max_lsn = cur_buf_lsn;
+ }
+
+ result |= BUF_WRITTEN;
+
+ return result;
+}
+
+static void
+WriteBuffers(BuffersToWrite *to_write,
+ IOQueue *ioq, WritebackContext *wb_context)
+{
+ SMgrRelation smgr;
+ Buffer first_buf;
+ BufferDesc *first_buf_hdr;
+ bool needs_checksum;
+
+ Assert(to_write->nbuffers > 0 && to_write->nbuffers <= io_combine_limit);
+
+ first_buf = to_write->buffers[0];
+ first_buf_hdr = GetBufferDescriptor(first_buf - 1);
+
+ smgr = smgropen(BufTagGetRelFileLocator(&first_buf_hdr->tag), INVALID_PROC_NUMBER);
+
+ /*
+ * Force XLOG flush up to buffer's LSN. This implements the basic WAL
+ * rule that log updates must hit disk before any of the data-file changes
+ * they describe do.
+ *
+ * However, this rule does not apply to unlogged relations, which will be
+ * lost after a crash anyway. Most unlogged relation pages do not bear
+ * LSNs since we never emit WAL records for them, and therefore flushing
+ * up through the buffer LSN would be useless, but harmless. However,
+ * GiST indexes use LSNs internally to track page-splits, and therefore
+ * unlogged GiST pages bear "fake" LSNs generated by
+ * GetFakeLSNForUnloggedRel. It is unlikely but possible that the fake
+ * LSN counter could advance past the WAL insertion point; and if it did
+ * happen, attempting to flush WAL through that location would fail, with
+ * disastrous system-wide consequences. To make sure that can't happen,
+ * skip the flush if the buffer isn't permanent.
+ */
+ if (to_write->max_lsn != InvalidXLogRecPtr)
+ XLogFlush(to_write->max_lsn);
+
+ /*
+ * Now it's safe to write buffer to disk. Note that no one else should
+ * have been able to write it while we were busy with log flushing because
+ * only one process at a time can set the BM_IO_IN_PROGRESS bit.
+ */
+
+ for (int nbuf = 0; nbuf < to_write->nbuffers; nbuf++)
+ {
+ Buffer cur_buf = to_write->buffers[nbuf];
+ BufferDesc *cur_buf_hdr = GetBufferDescriptor(cur_buf - 1);
+ Block bufBlock;
+ char *bufToWrite;
+
+ bufBlock = BufHdrGetBlock(cur_buf_hdr);
+ needs_checksum = PageNeedsChecksumCopy((Page) bufBlock);
+
+ /*
+ * Update page checksum if desired. Since we have only shared lock on
+ * the buffer, other processes might be updating hint bits in it, so
+ * we must copy the page to a bounce buffer if we do checksumming.
+ */
+ if (needs_checksum)
+ {
+ PgAioBounceBuffer *bb = pgaio_bounce_buffer_get();
+
+ pgaio_io_assoc_bounce_buffer(to_write->ioh, bb);
+
+ bufToWrite = pgaio_bounce_buffer_buffer(bb);
+ memcpy(bufToWrite, bufBlock, BLCKSZ);
+ PageSetChecksumInplace((Page) bufToWrite, cur_buf_hdr->tag.blockNum);
+ }
+ else
+ {
+ bufToWrite = bufBlock;
+ }
+
+ to_write->data_ptrs[nbuf] = bufToWrite;
+ }
+
+ pgaio_io_set_io_data_32(to_write->ioh,
+ (uint32 *) to_write->buffers,
+ to_write->nbuffers);
+ pgaio_io_add_shared_cb(to_write->ioh, ASC_SHARED_BUFFER_WRITE);
+
+ smgrstartwritev(to_write->ioh, smgr,
+ BufTagGetForkNum(&first_buf_hdr->tag),
+ first_buf_hdr->tag.blockNum,
+ to_write->data_ptrs,
+ to_write->nbuffers,
+ false);
+ pgstat_count_io_op_n(IOOBJECT_RELATION, IOCONTEXT_NORMAL,
+ IOOP_WRITE, to_write->nbuffers);
+
+
+ for (int nbuf = 0; nbuf < to_write->nbuffers; nbuf++)
+ {
+ Buffer cur_buf = to_write->buffers[nbuf];
+ BufferDesc *cur_buf_hdr = GetBufferDescriptor(cur_buf - 1);
+
+ UnpinBuffer(cur_buf_hdr);
+ }
+
+ io_queue_track(ioq, &to_write->ior);
+ to_write->total_writes++;
+
+ /* clear state for next write */
+ to_write->nbuffers = 0;
+ to_write->start_at_tag.relNumber = InvalidOid;
+ to_write->start_at_tag.blockNum = InvalidBlockNumber;
+ to_write->max_combine = 0;
+ to_write->max_lsn = InvalidXLogRecPtr;
+ to_write->ioh = NULL;
+ pgaio_io_ref_clear(&to_write->ior);
}
/*
@@ -4001,6 +4468,7 @@ FlushBuffer(BufferDesc *buf, SMgrRelation reln, IOObject io_object,
error_context_stack = errcallback.previous;
}
+
/*
* RelationGetNumberOfBlocksInFork
* Determines the current number of pages in the specified relation fork.
diff --git a/src/backend/storage/page/bufpage.c b/src/backend/storage/page/bufpage.c
index be6f1f62d29..8295e3fb0a0 100644
--- a/src/backend/storage/page/bufpage.c
+++ b/src/backend/storage/page/bufpage.c
@@ -1491,6 +1491,16 @@ PageIndexTupleOverwrite(Page page, OffsetNumber offnum,
return true;
}
+bool
+PageNeedsChecksumCopy(Page page)
+{
+ if (PageIsNew(page))
+ return false;
+
+ /* If we don't need a checksum, just return the passed-in data */
+ return DataChecksumsEnabled();
+}
+
/*
* Set checksum for a page in shared buffers.
diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list
index 6f39abcdf3c..ca6dd0bebf0 100644
--- a/src/tools/pgindent/typedefs.list
+++ b/src/tools/pgindent/typedefs.list
@@ -344,6 +344,7 @@ BufferManagerRelation
BufferStrategyControl
BufferTag
BufferUsage
+BuffersToWrite
BuildAccumulator
BuiltinScript
BulkInsertState
--
2.45.2.827.g557ae147e6
v2.0-0016-very-wip-test_aio-module.patch (text/x-diff, 37.3K) download
From c3a8731578a7fc1b03609e5bdb800e4fc18db80e Mon Sep 17 00:00:00 2001
From: Andres Freund <andres@anarazel.de>
Date: Sat, 31 Aug 2024 21:13:48 -0400
Subject: [PATCH v2.0 16/17] very-wip: test_aio module
Author:
Reviewed-by:
Discussion: https://postgr.es/m/
Backpatch:
---
src/include/storage/aio_internal.h | 10 +
src/include/storage/buf_internals.h | 4 +
src/backend/storage/aio/aio.c | 38 ++
src/backend/storage/buffer/bufmgr.c | 3 +-
src/test/modules/Makefile | 1 +
src/test/modules/meson.build | 1 +
src/test/modules/test_aio/.gitignore | 6 +
src/test/modules/test_aio/Makefile | 34 ++
src/test/modules/test_aio/expected/inject.out | 180 +++++++
src/test/modules/test_aio/expected/io.out | 40 ++
.../modules/test_aio/expected/ownership.out | 148 ++++++
src/test/modules/test_aio/expected/prep.out | 17 +
src/test/modules/test_aio/io_uring.conf | 5 +
src/test/modules/test_aio/meson.build | 65 +++
src/test/modules/test_aio/sql/inject.sql | 51 ++
src/test/modules/test_aio/sql/io.sql | 16 +
src/test/modules/test_aio/sql/ownership.sql | 65 +++
src/test/modules/test_aio/sql/prep.sql | 9 +
src/test/modules/test_aio/test_aio--1.0.sql | 94 ++++
src/test/modules/test_aio/test_aio.c | 479 ++++++++++++++++++
src/test/modules/test_aio/test_aio.control | 3 +
src/test/modules/test_aio/worker.conf | 5 +
22 files changed, 1272 insertions(+), 2 deletions(-)
create mode 100644 src/test/modules/test_aio/.gitignore
create mode 100644 src/test/modules/test_aio/Makefile
create mode 100644 src/test/modules/test_aio/expected/inject.out
create mode 100644 src/test/modules/test_aio/expected/io.out
create mode 100644 src/test/modules/test_aio/expected/ownership.out
create mode 100644 src/test/modules/test_aio/expected/prep.out
create mode 100644 src/test/modules/test_aio/io_uring.conf
create mode 100644 src/test/modules/test_aio/meson.build
create mode 100644 src/test/modules/test_aio/sql/inject.sql
create mode 100644 src/test/modules/test_aio/sql/io.sql
create mode 100644 src/test/modules/test_aio/sql/ownership.sql
create mode 100644 src/test/modules/test_aio/sql/prep.sql
create mode 100644 src/test/modules/test_aio/test_aio--1.0.sql
create mode 100644 src/test/modules/test_aio/test_aio.c
create mode 100644 src/test/modules/test_aio/test_aio.control
create mode 100644 src/test/modules/test_aio/worker.conf
diff --git a/src/include/storage/aio_internal.h b/src/include/storage/aio_internal.h
index 67d994cc0b1..cd3063f6c11 100644
--- a/src/include/storage/aio_internal.h
+++ b/src/include/storage/aio_internal.h
@@ -259,6 +259,16 @@ extern const char *pgaio_io_get_op_name(PgAioHandle *ioh);
extern const char *pgaio_io_get_state_name(PgAioHandle *ioh);
+
+/* These functions are just for use in tests, from within injection points */
+#ifdef USE_INJECTION_POINTS
+
+extern PgAioHandle *pgaio_inj_io_get(void);
+
+#endif
+
+
+
/* Declarations for the tables of function pointers exposed by each IO method. */
extern const IoMethodOps pgaio_worker_ops;
#ifdef USE_LIBURING
diff --git a/src/include/storage/buf_internals.h b/src/include/storage/buf_internals.h
index 9d3123663b3..1b3329a25b4 100644
--- a/src/include/storage/buf_internals.h
+++ b/src/include/storage/buf_internals.h
@@ -423,6 +423,10 @@ extern void IssuePendingWritebacks(WritebackContext *wb_context, IOContext io_co
extern void ScheduleBufferTagForWriteback(WritebackContext *wb_context,
IOContext io_context, BufferTag *tag);
+/* solely to make it easier to write tests */
+extern bool StartBufferIO(BufferDesc *buf, bool forInput, bool nowait);
+
+
/* freelist.c */
extern IOContext IOContextForStrategy(BufferAccessStrategy strategy);
extern BufferDesc *StrategyGetBuffer(BufferAccessStrategy strategy,
diff --git a/src/backend/storage/aio/aio.c b/src/backend/storage/aio/aio.c
index d6f9f658b97..9db661b1cd0 100644
--- a/src/backend/storage/aio/aio.c
+++ b/src/backend/storage/aio/aio.c
@@ -22,6 +22,9 @@
#include "utils/resowner.h"
#include "utils/wait_event_types.h"
+#ifdef USE_INJECTION_POINTS
+#include "utils/injection_point.h"
+#endif
static void pgaio_io_reclaim(PgAioHandle *ioh);
@@ -65,6 +68,11 @@ static const IoMethodOps *pgaio_ops_table[] = {
const IoMethodOps *pgaio_impl;
+#ifdef USE_INJECTION_POINTS
+static PgAioHandle *inj_cur_handle;
+#endif
+
+
/* --------------------------------------------------------------------------------
* "Core" IO Api
@@ -529,6 +537,19 @@ pgaio_io_process_completion(PgAioHandle *ioh, int result)
/* FIXME: should be done in separate function */
ioh->state = AHS_REAPED;
+#ifdef USE_INJECTION_POINTS
+ inj_cur_handle = ioh;
+
+ /*
+ * FIXME: This could be in a critical section - but it looks like we can't
+ * just InjectionPointLoad() at process start, as the injection point
+ * might not yet be defined.
+ */
+ InjectionPointCached("AIO_PROCESS_COMPLETION_BEFORE_SHARED");
+
+ inj_cur_handle = NULL;
+#endif
+
pgaio_io_process_completion_subject(ioh);
/* ensure results of completion are visible before the new state */
@@ -994,3 +1015,20 @@ assign_io_method(int newval, void *extra)
{
pgaio_impl = pgaio_ops_table[newval];
}
+
+
+
+/* --------------------------------------------------------------------------------
+ * Injection point support
+ * --------------------------------------------------------------------------------
+ */
+
+#ifdef USE_INJECTION_POINTS
+
+PgAioHandle *
+pgaio_inj_io_get(void)
+{
+ return inj_cur_handle;
+}
+
+#endif
diff --git a/src/backend/storage/buffer/bufmgr.c b/src/backend/storage/buffer/bufmgr.c
index e62f2de2034..f774b42651a 100644
--- a/src/backend/storage/buffer/bufmgr.c
+++ b/src/backend/storage/buffer/bufmgr.c
@@ -541,7 +541,6 @@ static void UnpinBufferNoOwner(BufferDesc *buf);
static void BufferSync(int flags);
static uint32 WaitBufHdrUnlocked(BufferDesc *buf);
static void WaitIO(BufferDesc *buf);
-static bool StartBufferIO(BufferDesc *buf, bool forInput, bool nowait);
static void TerminateBufferIO(BufferDesc *buf, bool clear_dirty,
uint32 set_flag_bits, bool forget_owner,
bool syncio);
@@ -6122,7 +6121,7 @@ WaitIO(BufferDesc *buf)
* find out if they can perform the I/O as part of a larger operation, without
* waiting for the answer or distinguishing the reasons why not.
*/
-static bool
+bool
StartBufferIO(BufferDesc *buf, bool forInput, bool nowait)
{
uint32 buf_state;
diff --git a/src/test/modules/Makefile b/src/test/modules/Makefile
index 256799f520a..7df90602e90 100644
--- a/src/test/modules/Makefile
+++ b/src/test/modules/Makefile
@@ -13,6 +13,7 @@ SUBDIRS = \
libpq_pipeline \
plsample \
spgist_name_ops \
+ test_aio \
test_bloomfilter \
test_copy_callbacks \
test_custom_rmgrs \
diff --git a/src/test/modules/meson.build b/src/test/modules/meson.build
index d8fe059d236..bc7d19e694f 100644
--- a/src/test/modules/meson.build
+++ b/src/test/modules/meson.build
@@ -1,5 +1,6 @@
# Copyright (c) 2022-2024, PostgreSQL Global Development Group
+subdir('test_aio')
subdir('brin')
subdir('commit_ts')
subdir('delay_execution')
diff --git a/src/test/modules/test_aio/.gitignore b/src/test/modules/test_aio/.gitignore
new file mode 100644
index 00000000000..b4903eba657
--- /dev/null
+++ b/src/test/modules/test_aio/.gitignore
@@ -0,0 +1,6 @@
+# Generated subdirectories
+/log/
+/results/
+/output_iso/
+/tmp_check/
+/tmp_check_iso/
diff --git a/src/test/modules/test_aio/Makefile b/src/test/modules/test_aio/Makefile
new file mode 100644
index 00000000000..ae6d685835b
--- /dev/null
+++ b/src/test/modules/test_aio/Makefile
@@ -0,0 +1,34 @@
+# src/test/modules/delay_execution/Makefile
+
+PGFILEDESC = "test_aio - test code for AIO"
+
+MODULE_big = test_aio
+OBJS = \
+ $(WIN32RES) \
+ test_aio.o
+
+EXTENSION = test_aio
+DATA = test_aio--1.0.sql
+
+REGRESS = prep ownership io
+
+ifeq ($(enable_injection_points),yes)
+REGRESS += inject
+endif
+
+# FIXME: with meson this runs the tests once with worker and once - if
+# supported - with io_uring.
+
+# requires custom config
+NO_INSTALLCHECK = 1
+
+ifdef USE_PGXS
+PG_CONFIG = pg_config
+PGXS := $(shell $(PG_CONFIG) --pgxs)
+include $(PGXS)
+else
+subdir = src/test/modules/test_aio
+top_builddir = ../../../..
+include $(top_builddir)/src/Makefile.global
+include $(top_srcdir)/contrib/contrib-global.mk
+endif
diff --git a/src/test/modules/test_aio/expected/inject.out b/src/test/modules/test_aio/expected/inject.out
new file mode 100644
index 00000000000..e52b0f086dd
--- /dev/null
+++ b/src/test/modules/test_aio/expected/inject.out
@@ -0,0 +1,180 @@
+SELECT count(*) FROM tbl_b WHERE ctid = '(2, 1)';
+ count
+-------
+ 1
+(1 row)
+
+-- injected what we'd expect
+SELECT inj_io_short_read_attach(8192);
+ inj_io_short_read_attach
+--------------------------
+
+(1 row)
+
+SELECT invalidate_rel_block('tbl_b', 2);
+ invalidate_rel_block
+----------------------
+
+(1 row)
+
+SELECT count(*) FROM tbl_b WHERE ctid = '(2, 1)';
+ count
+-------
+ 1
+(1 row)
+
+SELECT inj_io_short_read_detach();
+ inj_io_short_read_detach
+--------------------------
+
+(1 row)
+
+-- injected a read shorter than a single block, expecting error
+SELECT inj_io_short_read_attach(17);
+ inj_io_short_read_attach
+--------------------------
+
+(1 row)
+
+SELECT invalidate_rel_block('tbl_b', 2);
+ invalidate_rel_block
+----------------------
+
+(1 row)
+
+SELECT redact($$
+ SELECT count(*) FROM tbl_b WHERE ctid = '(2, 1)';
+$$);
+NOTICE: wrapped error: could not read blocks 2..2 in file base/<redacted>: read only 0 of 8192 bytes
+ redact
+--------
+ f
+(1 row)
+
+SELECT inj_io_short_read_detach();
+ inj_io_short_read_detach
+--------------------------
+
+(1 row)
+
+-- shorten multi-block read to a single block, should retry, but that's not
+-- implemented yet
+SELECT inj_io_short_read_attach(8192);
+ inj_io_short_read_attach
+--------------------------
+
+(1 row)
+
+SELECT invalidate_rel_block('tbl_b', 0);
+ invalidate_rel_block
+----------------------
+
+(1 row)
+
+SELECT invalidate_rel_block('tbl_b', 1);
+ invalidate_rel_block
+----------------------
+
+(1 row)
+
+SELECT invalidate_rel_block('tbl_b', 2);
+ invalidate_rel_block
+----------------------
+
+(1 row)
+
+SELECT redact($$
+ SELECT count(*) FROM tbl_b;
+$$);
+NOTICE: wrapped error: could not read blocks 1..2 in file base/<redacted>: read only 8192 of 16384 bytes
+ redact
+--------
+ f
+(1 row)
+
+SELECT inj_io_short_read_detach();
+ inj_io_short_read_detach
+--------------------------
+
+(1 row)
+
+-- verify that checksum errors are detected even as part of a shortened
+-- multi-block read
+-- (tbl_a, block 1 is corrupted)
+SELECT redact($$
+ SELECT count(*) FROM tbl_a WHERE ctid < '(2, 1)';
+$$);
+NOTICE: wrapped error: invalid page in block 2 of relation base/<redacted>
+ redact
+--------
+ f
+(1 row)
+
+SELECT inj_io_short_read_attach(8192);
+ inj_io_short_read_attach
+--------------------------
+
+(1 row)
+
+SELECT invalidate_rel_block('tbl_a', 0);
+ invalidate_rel_block
+----------------------
+
+(1 row)
+
+SELECT invalidate_rel_block('tbl_a', 1);
+ invalidate_rel_block
+----------------------
+
+(1 row)
+
+SELECT invalidate_rel_block('tbl_a', 2);
+ invalidate_rel_block
+----------------------
+
+(1 row)
+
+SELECT redact($$
+ SELECT count(*) FROM tbl_a WHERE ctid < '(2, 1)';
+$$);
+NOTICE: wrapped error: invalid page in block 2 of relation base/<redacted>
+ redact
+--------
+ f
+(1 row)
+
+SELECT inj_io_short_read_detach();
+ inj_io_short_read_detach
+--------------------------
+
+(1 row)
+
+-- FIXME: Should error
+-- FIXME: errno encoding?
+SELECT inj_io_short_read_attach(-5);
+ inj_io_short_read_attach
+--------------------------
+
+(1 row)
+
+SELECT invalidate_rel_block('tbl_b', 2);
+ invalidate_rel_block
+----------------------
+
+(1 row)
+
+SELECT redact($$
+ SELECT count(*) FROM tbl_b WHERE ctid = '(2, 1)';
+$$);
+NOTICE: wrapped error: could not read blocks 2..3 in file base/<redacted>: Input/output error
+ redact
+--------
+ f
+(1 row)
+
+SELECT inj_io_short_read_detach();
+ inj_io_short_read_detach
+--------------------------
+
+(1 row)
+
diff --git a/src/test/modules/test_aio/expected/io.out b/src/test/modules/test_aio/expected/io.out
new file mode 100644
index 00000000000..e46b582f290
--- /dev/null
+++ b/src/test/modules/test_aio/expected/io.out
@@ -0,0 +1,40 @@
+SELECT count(*) FROM tbl_a WHERE ctid = '(1, 1)';
+ count
+-------
+ 1
+(1 row)
+
+SELECT corrupt_rel_block('tbl_a', 1);
+ corrupt_rel_block
+-------------------
+
+(1 row)
+
+-- FIXME: Should report the error
+SELECT redact($$
+ SELECT read_corrupt_rel_block('tbl_a', 1);
+$$);
+ redact
+--------
+ t
+(1 row)
+
+-- verify the error is reported
+SELECT redact($$
+ SELECT count(*) FROM tbl_a WHERE ctid = '(1, 1)';
+$$);
+NOTICE: wrapped error: invalid page in block 2 of relation base/<redacted>
+ redact
+--------
+ f
+(1 row)
+
+SELECT redact($$
+ SELECT count(*) FROM tbl_a;
+$$);
+NOTICE: wrapped error: invalid page in block 2 of relation base/<redacted>
+ redact
+--------
+ f
+(1 row)
+
diff --git a/src/test/modules/test_aio/expected/ownership.out b/src/test/modules/test_aio/expected/ownership.out
new file mode 100644
index 00000000000..97fdad6c629
--- /dev/null
+++ b/src/test/modules/test_aio/expected/ownership.out
@@ -0,0 +1,148 @@
+-----
+-- IO handles
+----
+-- leak warning: implicit xact
+SELECT handle_get();
+WARNING: leaked AIO handle
+ handle_get
+------------
+
+(1 row)
+
+-- leak warning: explicit xact
+BEGIN; SELECT handle_get(); COMMIT;
+WARNING: leaked AIO handle
+ handle_get
+------------
+
+(1 row)
+
+-- leak warning + error: released in different command (thus resowner)
+BEGIN; SELECT handle_get(); SELECT handle_release_last(); COMMIT;
+WARNING: leaked AIO handle
+ handle_get
+------------
+
+(1 row)
+
+ERROR: release in unexpected state
+-- no leak, same command
+BEGIN; SELECT handle_get() UNION ALL SELECT handle_release_last(); COMMIT;
+ handle_get
+------------
+
+
+(2 rows)
+
+-- leak warning: subtrans
+BEGIN; SAVEPOINT foo; SELECT handle_get(); COMMIT;
+WARNING: leaked AIO handle
+ handle_get
+------------
+
+(1 row)
+
+-- normal handle use
+SELECT handle_get_release();
+ handle_get_release
+--------------------
+
+(1 row)
+
+-- should error out, API violation
+SELECT handle_get_twice();
+ERROR: API violation: Only one IO can be handed out
+-- recover after error in implicit xact
+SELECT handle_get_and_error(); SELECT handle_get_release();
+ERROR: as you command
+ handle_get_release
+--------------------
+
+(1 row)
+
+-- recover after error in explicit xact
+BEGIN; SELECT handle_get_and_error(); ROLLBACK; SELECT handle_get_release();
+ERROR: as you command
+ handle_get_release
+--------------------
+
+(1 row)
+
+-- recover after error in subtrans
+BEGIN; SAVEPOINT foo; SELECT handle_get_and_error(); ROLLBACK TO SAVEPOINT foo; SELECT handle_get_release(); ROLLBACK;
+ERROR: as you command
+ handle_get_release
+--------------------
+
+(1 row)
+
+-----
+-- Bounce Buffers handles
+----
+-- leak warning: implicit xact
+SELECT bb_get();
+WARNING: leaked AIO bounce buffer
+ bb_get
+--------
+
+(1 row)
+
+-- leak warning: explicit xact
+BEGIN; SELECT bb_get(); COMMIT;
+WARNING: leaked AIO bounce buffer
+ bb_get
+--------
+
+(1 row)
+
+-- missing leak warning: we should warn at command boundaries, not xact boundaries
+BEGIN; SELECT bb_get(); SELECT bb_release_last(); COMMIT;
+WARNING: leaked AIO bounce buffer
+ bb_get
+--------
+
+(1 row)
+
+ERROR: can only release handed out BB
+-- leak warning: subtrans
+BEGIN; SAVEPOINT foo; SELECT bb_get(); COMMIT;
+WARNING: leaked AIO bounce buffer
+ bb_get
+--------
+
+(1 row)
+
+-- normal bb use
+SELECT bb_get_release();
+ bb_get_release
+----------------
+
+(1 row)
+
+-- should error out, API violation
+SELECT bb_get_twice();
+ERROR: can only hand out one BB
+-- recover after error in implicit xact
+SELECT bb_get_and_error(); SELECT bb_get_release();
+ERROR: as you command
+ bb_get_release
+----------------
+
+(1 row)
+
+-- recover after error in explicit xact
+BEGIN; SELECT bb_get_and_error(); ROLLBACK; SELECT bb_get_release();
+ERROR: as you command
+ bb_get_release
+----------------
+
+(1 row)
+
+-- recover after error in subtrans
+BEGIN; SAVEPOINT foo; SELECT bb_get_and_error(); ROLLBACK TO SAVEPOINT foo; SELECT bb_get_release(); ROLLBACK;
+ERROR: as you command
+ bb_get_release
+----------------
+
+(1 row)
+
diff --git a/src/test/modules/test_aio/expected/prep.out b/src/test/modules/test_aio/expected/prep.out
new file mode 100644
index 00000000000..7fad6280db5
--- /dev/null
+++ b/src/test/modules/test_aio/expected/prep.out
@@ -0,0 +1,17 @@
+CREATE EXTENSION test_aio;
+CREATE TABLE tbl_a(data int not null);
+CREATE TABLE tbl_b(data int not null);
+INSERT INTO tbl_a SELECT generate_series(1, 10000);
+INSERT INTO tbl_b SELECT generate_series(1, 10000);
+SELECT grow_rel('tbl_a', 500);
+ grow_rel
+----------
+
+(1 row)
+
+SELECT grow_rel('tbl_b', 550);
+ grow_rel
+----------
+
+(1 row)
+
diff --git a/src/test/modules/test_aio/io_uring.conf b/src/test/modules/test_aio/io_uring.conf
new file mode 100644
index 00000000000..efd7ad143ff
--- /dev/null
+++ b/src/test/modules/test_aio/io_uring.conf
@@ -0,0 +1,5 @@
+shared_preload_libraries=test_aio
+io_method = 'io_uring'
+log_min_messages = 'DEBUG3'
+log_statement=all
+restart_after_crash=false
diff --git a/src/test/modules/test_aio/meson.build b/src/test/modules/test_aio/meson.build
new file mode 100644
index 00000000000..102c2e01537
--- /dev/null
+++ b/src/test/modules/test_aio/meson.build
@@ -0,0 +1,65 @@
+# Copyright (c) 2022-2024, PostgreSQL Global Development Group
+
+test_aio_sources = files(
+ 'test_aio.c',
+)
+
+if host_system == 'windows'
+ test_aio_sources += rc_lib_gen.process(win32ver_rc, extra_args: [
+ '--NAME', 'test_aio',
+ '--FILEDESC', 'test_aio - test code for AIO',])
+endif
+
+test_aio = shared_module('test_aio',
+ test_aio_sources,
+ kwargs: pg_test_mod_args,
+)
+test_install_libs += test_aio
+
+test_install_data += files(
+ 'test_aio.control',
+ 'test_aio--1.0.sql',
+)
+
+
+testfiles = [
+ 'prep',
+ 'ownership',
+ 'io',
+]
+
+if get_option('injection_points')
+ testfiles += 'inject'
+endif
+
+
+
+tests += {
+ 'name': 'test_aio_worker',
+ 'sd': meson.current_source_dir(),
+ 'bd': meson.current_build_dir(),
+ 'regress': {
+ 'sql': testfiles,
+ 'regress_args': [
+ '--temp-config', files('worker.conf'),
+ ],
+ # requires custom config
+ 'runningcheck': false,
+ },
+}
+
+if liburing.found()
+ tests += {
+ 'name': 'test_aio_uring',
+ 'sd': meson.current_source_dir(),
+ 'bd': meson.current_build_dir(),
+ 'regress': {
+ 'sql': testfiles,
+ 'regress_args': [
+ '--temp-config', files('io_uring.conf'),
+ ],
+ # requires custom config
+ 'runningcheck': false,
+ }
+ }
+endif
diff --git a/src/test/modules/test_aio/sql/inject.sql b/src/test/modules/test_aio/sql/inject.sql
new file mode 100644
index 00000000000..b3d34de8977
--- /dev/null
+++ b/src/test/modules/test_aio/sql/inject.sql
@@ -0,0 +1,51 @@
+SELECT count(*) FROM tbl_b WHERE ctid = '(2, 1)';
+
+-- injected what we'd expect
+SELECT inj_io_short_read_attach(8192);
+SELECT invalidate_rel_block('tbl_b', 2);
+SELECT count(*) FROM tbl_b WHERE ctid = '(2, 1)';
+SELECT inj_io_short_read_detach();
+
+-- injected a read shorter than a single block, expecting error
+SELECT inj_io_short_read_attach(17);
+SELECT invalidate_rel_block('tbl_b', 2);
+SELECT redact($$
+ SELECT count(*) FROM tbl_b WHERE ctid = '(2, 1)';
+$$);
+SELECT inj_io_short_read_detach();
+
+-- shorten multi-block read to a single block, should retry, but that's not
+-- implemented yet
+SELECT inj_io_short_read_attach(8192);
+SELECT invalidate_rel_block('tbl_b', 0);
+SELECT invalidate_rel_block('tbl_b', 1);
+SELECT invalidate_rel_block('tbl_b', 2);
+SELECT redact($$
+ SELECT count(*) FROM tbl_b;
+$$);
+SELECT inj_io_short_read_detach();
+
+-- verify that checksum errors are detected even as part of a shortened
+-- multi-block read
+-- (tbl_a, block 1 is corrupted)
+SELECT redact($$
+ SELECT count(*) FROM tbl_a WHERE ctid < '(2, 1)';
+$$);
+SELECT inj_io_short_read_attach(8192);
+SELECT invalidate_rel_block('tbl_a', 0);
+SELECT invalidate_rel_block('tbl_a', 1);
+SELECT invalidate_rel_block('tbl_a', 2);
+SELECT redact($$
+ SELECT count(*) FROM tbl_a WHERE ctid < '(2, 1)';
+$$);
+SELECT inj_io_short_read_detach();
+
+
+-- FIXME: Should error
+-- FIXME: errno encoding?
+SELECT inj_io_short_read_attach(-5);
+SELECT invalidate_rel_block('tbl_b', 2);
+SELECT redact($$
+ SELECT count(*) FROM tbl_b WHERE ctid = '(2, 1)';
+$$);
+SELECT inj_io_short_read_detach();
diff --git a/src/test/modules/test_aio/sql/io.sql b/src/test/modules/test_aio/sql/io.sql
new file mode 100644
index 00000000000..a29bb4eb15d
--- /dev/null
+++ b/src/test/modules/test_aio/sql/io.sql
@@ -0,0 +1,16 @@
+SELECT count(*) FROM tbl_a WHERE ctid = '(1, 1)';
+
+SELECT corrupt_rel_block('tbl_a', 1);
+
+-- FIXME: Should report the error
+SELECT redact($$
+ SELECT read_corrupt_rel_block('tbl_a', 1);
+$$);
+
+-- verify the error is reported
+SELECT redact($$
+ SELECT count(*) FROM tbl_a WHERE ctid = '(1, 1)';
+$$);
+SELECT redact($$
+ SELECT count(*) FROM tbl_a;
+$$);
diff --git a/src/test/modules/test_aio/sql/ownership.sql b/src/test/modules/test_aio/sql/ownership.sql
new file mode 100644
index 00000000000..63cf40c802a
--- /dev/null
+++ b/src/test/modules/test_aio/sql/ownership.sql
@@ -0,0 +1,65 @@
+-----
+-- IO handles
+----
+
+-- leak warning: implicit xact
+SELECT handle_get();
+
+-- leak warning: explicit xact
+BEGIN; SELECT handle_get(); COMMIT;
+
+-- leak warning + error: released in different command (thus resowner)
+BEGIN; SELECT handle_get(); SELECT handle_release_last(); COMMIT;
+
+-- no leak, same command
+BEGIN; SELECT handle_get() UNION ALL SELECT handle_release_last(); COMMIT;
+
+-- leak warning: subtrans
+BEGIN; SAVEPOINT foo; SELECT handle_get(); COMMIT;
+
+-- normal handle use
+SELECT handle_get_release();
+
+-- should error out, API violation
+SELECT handle_get_twice();
+
+-- recover after error in implicit xact
+SELECT handle_get_and_error(); SELECT handle_get_release();
+
+-- recover after error in explicit xact
+BEGIN; SELECT handle_get_and_error(); ROLLBACK; SELECT handle_get_release();
+
+-- recover after error in subtrans
+BEGIN; SAVEPOINT foo; SELECT handle_get_and_error(); ROLLBACK TO SAVEPOINT foo; SELECT handle_get_release(); ROLLBACK;
+
+
+-----
+-- Bounce Buffers handles
+----
+
+-- leak warning: implicit xact
+SELECT bb_get();
+
+-- leak warning: explicit xact
+BEGIN; SELECT bb_get(); COMMIT;
+
+-- missing leak warning: we should warn at command boundaries, not xact boundaries
+BEGIN; SELECT bb_get(); SELECT bb_release_last(); COMMIT;
+
+-- leak warning: subtrans
+BEGIN; SAVEPOINT foo; SELECT bb_get(); COMMIT;
+
+-- normal bb use
+SELECT bb_get_release();
+
+-- should error out, API violation
+SELECT bb_get_twice();
+
+-- recover after error in implicit xact
+SELECT bb_get_and_error(); SELECT bb_get_release();
+
+-- recover after error in explicit xact
+BEGIN; SELECT bb_get_and_error(); ROLLBACK; SELECT bb_get_release();
+
+-- recover after error in subtrans
+BEGIN; SAVEPOINT foo; SELECT bb_get_and_error(); ROLLBACK TO SAVEPOINT foo; SELECT bb_get_release(); ROLLBACK;
diff --git a/src/test/modules/test_aio/sql/prep.sql b/src/test/modules/test_aio/sql/prep.sql
new file mode 100644
index 00000000000..b8f225cbc98
--- /dev/null
+++ b/src/test/modules/test_aio/sql/prep.sql
@@ -0,0 +1,9 @@
+CREATE EXTENSION test_aio;
+
+CREATE TABLE tbl_a(data int not null);
+CREATE TABLE tbl_b(data int not null);
+
+INSERT INTO tbl_a SELECT generate_series(1, 10000);
+INSERT INTO tbl_b SELECT generate_series(1, 10000);
+SELECT grow_rel('tbl_a', 500);
+SELECT grow_rel('tbl_b', 550);
diff --git a/src/test/modules/test_aio/test_aio--1.0.sql b/src/test/modules/test_aio/test_aio--1.0.sql
new file mode 100644
index 00000000000..ea9ad43ed8f
--- /dev/null
+++ b/src/test/modules/test_aio/test_aio--1.0.sql
@@ -0,0 +1,94 @@
+/* src/test/modules/test_aio/test_aio--1.0.sql */
+
+-- complain if script is sourced in psql, rather than via CREATE EXTENSION
+\echo Use "CREATE EXTENSION test_aio" to load this file. \quit
+
+
+CREATE FUNCTION grow_rel(rel regclass, nblocks int)
+RETURNS pg_catalog.void STRICT
+AS 'MODULE_PATHNAME' LANGUAGE C;
+
+
+CREATE FUNCTION corrupt_rel_block(rel regclass, blockno int)
+RETURNS pg_catalog.void STRICT
+AS 'MODULE_PATHNAME' LANGUAGE C;
+
+CREATE FUNCTION read_corrupt_rel_block(rel regclass, blockno int)
+RETURNS pg_catalog.void STRICT
+AS 'MODULE_PATHNAME' LANGUAGE C;
+
+CREATE FUNCTION invalidate_rel_block(rel regclass, blockno int)
+RETURNS pg_catalog.void STRICT
+AS 'MODULE_PATHNAME' LANGUAGE C;
+
+CREATE FUNCTION handle_get_and_error()
+RETURNS pg_catalog.void STRICT
+AS 'MODULE_PATHNAME' LANGUAGE C;
+
+CREATE FUNCTION handle_get_twice()
+RETURNS pg_catalog.void STRICT
+AS 'MODULE_PATHNAME' LANGUAGE C;
+
+CREATE FUNCTION handle_get()
+RETURNS pg_catalog.void STRICT
+AS 'MODULE_PATHNAME' LANGUAGE C;
+
+CREATE FUNCTION handle_get_release()
+RETURNS pg_catalog.void STRICT
+AS 'MODULE_PATHNAME' LANGUAGE C;
+
+CREATE FUNCTION handle_release_last()
+RETURNS pg_catalog.void STRICT
+AS 'MODULE_PATHNAME' LANGUAGE C;
+
+
+CREATE FUNCTION bb_get_and_error()
+RETURNS pg_catalog.void STRICT
+AS 'MODULE_PATHNAME' LANGUAGE C;
+
+CREATE FUNCTION bb_get_twice()
+RETURNS pg_catalog.void STRICT
+AS 'MODULE_PATHNAME' LANGUAGE C;
+
+CREATE FUNCTION bb_get()
+RETURNS pg_catalog.void STRICT
+AS 'MODULE_PATHNAME' LANGUAGE C;
+
+CREATE FUNCTION bb_get_release()
+RETURNS pg_catalog.void STRICT
+AS 'MODULE_PATHNAME' LANGUAGE C;
+
+CREATE FUNCTION bb_release_last()
+RETURNS pg_catalog.void STRICT
+AS 'MODULE_PATHNAME' LANGUAGE C;
+
+
+CREATE OR REPLACE FUNCTION redact(p_sql text)
+RETURNS bool
+LANGUAGE plpgsql
+AS $$
+ DECLARE
+ err_state text;
+ err_msg text;
+ BEGIN
+ EXECUTE p_sql;
+ RETURN true;
+ EXCEPTION WHEN OTHERS THEN
+ GET STACKED DIAGNOSTICS
+ err_state = RETURNED_SQLSTATE,
+ err_msg = MESSAGE_TEXT;
+ err_msg = regexp_replace(err_msg, '(file|relation) "?base/[0-9]+/[0-9]+"?', '\1 base/<redacted>');
+ RAISE NOTICE 'wrapped error: %', err_msg
+ USING ERRCODE = err_state;
+ RETURN false;
+ END;
+$$;
+
+
+CREATE FUNCTION inj_io_short_read_attach(result int)
+RETURNS pg_catalog.void STRICT
+AS 'MODULE_PATHNAME' LANGUAGE C;
+
+CREATE FUNCTION inj_io_short_read_detach()
+RETURNS pg_catalog.void STRICT
+AS 'MODULE_PATHNAME' LANGUAGE C;
diff --git a/src/test/modules/test_aio/test_aio.c b/src/test/modules/test_aio/test_aio.c
new file mode 100644
index 00000000000..9626d495241
--- /dev/null
+++ b/src/test/modules/test_aio/test_aio.c
@@ -0,0 +1,479 @@
+/*-------------------------------------------------------------------------
+ *
+ * delay_execution.c
+ * Test module to allow delay between parsing and execution of a query.
+ *
+ * The delay is implemented by taking and immediately releasing a specified
+ * advisory lock. If another process has previously taken that lock, the
+ * current process will be blocked until the lock is released; otherwise,
+ * there's no effect. This allows an isolationtester script to reliably
+ * test behaviors where some specified action happens in another backend
+ * between parsing and execution of any desired query.
+ *
+ * Copyright (c) 2020-2024, PostgreSQL Global Development Group
+ *
+ * IDENTIFICATION
+ * src/test/modules/delay_execution/delay_execution.c
+ *
+ *-------------------------------------------------------------------------
+ */
+
+#include "postgres.h"
+
+#include "fmgr.h"
+#include "storage/aio.h"
+#include "storage/aio_internal.h"
+#include "storage/bufmgr.h"
+#include "storage/buf_internals.h"
+#include "storage/lwlock.h"
+#include "storage/ipc.h"
+#include "access/relation.h"
+#include "utils/rel.h"
+#include "utils/injection_point.h"
+
+
+PG_MODULE_MAGIC;
+
+
+typedef struct InjIoErrorState
+{
+ bool enabled;
+ bool result_set;
+ int result;
+} InjIoErrorState;
+
+static InjIoErrorState * inj_io_error_state;
+
+/* Shared memory init callbacks */
+static shmem_request_hook_type prev_shmem_request_hook = NULL;
+static shmem_startup_hook_type prev_shmem_startup_hook = NULL;
+
+
+static PgAioHandle *last_handle;
+static PgAioBounceBuffer *last_bb;
+
+
+
+static void
+test_aio_shmem_request(void)
+{
+ if (prev_shmem_request_hook)
+ prev_shmem_request_hook();
+
+ RequestAddinShmemSpace(sizeof(InjIoErrorState));
+}
+
+static void
+test_aio_shmem_startup(void)
+{
+ bool found;
+
+ if (prev_shmem_startup_hook)
+ prev_shmem_startup_hook();
+
+ /* Create or attach to the shared memory state */
+ LWLockAcquire(AddinShmemInitLock, LW_EXCLUSIVE);
+
+ inj_io_error_state = ShmemInitStruct("injection_points",
+ sizeof(InjIoErrorState),
+ &found);
+
+ if (!found)
+ {
+ /*
+ * First time through, so initialize. This is shared with the dynamic
+ * initialization using a DSM.
+ */
+ inj_io_error_state->enabled = false;
+
+#ifdef USE_INJECTION_POINTS
+ InjectionPointAttach("AIO_PROCESS_COMPLETION_BEFORE_SHARED",
+ "test_aio",
+ "inj_io_short_read",
+ NULL,
+ 0);
+ InjectionPointLoad("AIO_PROCESS_COMPLETION_BEFORE_SHARED");
+#endif
+ }
+ else
+ {
+#ifdef USE_INJECTION_POINTS
+ InjectionPointLoad("AIO_PROCESS_COMPLETION_BEFORE_SHARED");
+ elog(LOG, "injection point loaded");
+#endif
+ }
+
+ LWLockRelease(AddinShmemInitLock);
+}
+
+void
+_PG_init(void)
+{
+ if (!process_shared_preload_libraries_in_progress)
+ return;
+
+ /* Shared memory initialization */
+ prev_shmem_request_hook = shmem_request_hook;
+ shmem_request_hook = test_aio_shmem_request;
+ prev_shmem_startup_hook = shmem_startup_hook;
+ shmem_startup_hook = test_aio_shmem_startup;
+}
+
+
+PG_FUNCTION_INFO_V1(grow_rel);
+Datum
+grow_rel(PG_FUNCTION_ARGS)
+{
+ Oid relid = PG_GETARG_OID(0);
+ uint32 nblocks = PG_GETARG_UINT32(1);
+ Relation rel;
+#define MAX_BUFFERS_TO_EXTEND_BY 64
+ Buffer victim_buffers[MAX_BUFFERS_TO_EXTEND_BY];
+
+ rel = relation_open(relid, AccessExclusiveLock);
+
+ while (nblocks > 0)
+ {
+ uint32 extend_by_pages;
+
+ extend_by_pages = Min(nblocks, MAX_BUFFERS_TO_EXTEND_BY);
+
+ ExtendBufferedRelBy(BMR_REL(rel),
+ MAIN_FORKNUM,
+ NULL,
+ 0,
+ extend_by_pages,
+ victim_buffers,
+ &extend_by_pages);
+
+ nblocks -= extend_by_pages;
+
+ for (uint32 i = 0; i < extend_by_pages; i++)
+ {
+ ReleaseBuffer(victim_buffers[i]);
+ }
+ }
+
+ relation_close(rel, NoLock);
+
+ PG_RETURN_VOID();
+}
+
+PG_FUNCTION_INFO_V1(corrupt_rel_block);
+Datum
+corrupt_rel_block(PG_FUNCTION_ARGS)
+{
+ Oid relid = PG_GETARG_OID(0);
+ uint32 block = PG_GETARG_UINT32(1);
+ Relation rel;
+ Buffer buf;
+ Page page;
+ PageHeader ph;
+
+ rel = relation_open(relid, AccessExclusiveLock);
+
+ buf = ReadBuffer(rel, block);
+ page = BufferGetPage(buf);
+
+ LockBuffer(buf, BUFFER_LOCK_EXCLUSIVE);
+
+ MarkBufferDirty(buf);
+
+ PageInit(page, BufferGetPageSize(buf), 0);
+
+ ph = (PageHeader) page;
+ ph->pd_special = BLCKSZ + 1;
+
+ FlushOneBuffer(buf);
+
+ LockBuffer(buf, BUFFER_LOCK_UNLOCK);
+
+ ReleaseBuffer(buf);
+
+ EvictUnpinnedBuffer(buf);
+
+ relation_close(rel, NoLock);
+
+ PG_RETURN_VOID();
+}
+
+PG_FUNCTION_INFO_V1(read_corrupt_rel_block);
+Datum
+read_corrupt_rel_block(PG_FUNCTION_ARGS)
+{
+ Oid relid = PG_GETARG_OID(0);
+ uint32 block = PG_GETARG_UINT32(1);
+ Relation rel;
+ Buffer buf;
+ BufferDesc *buf_hdr;
+ Page page;
+ PgAioHandle *ioh;
+ PgAioHandleRef ior;
+ SMgrRelation smgr;
+ uint32 buf_state;
+
+ rel = relation_open(relid, AccessExclusiveLock);
+
+ /* read buffer without erroring out */
+ buf = ReadBufferExtended(rel, MAIN_FORKNUM, block, RBM_ZERO_AND_LOCK, NULL);
+ LockBuffer(buf, BUFFER_LOCK_UNLOCK);
+
+ page = BufferGetBlock(buf);
+
+ ioh = pgaio_io_get(CurrentResourceOwner, NULL);
+ pgaio_io_get_ref(ioh, &ior);
+
+ buf_hdr = GetBufferDescriptor(buf - 1);
+ smgr = RelationGetSmgr(rel);
+
+ /* FIXME: even if just a test, we should verify nobody else uses this */
+ buf_state = LockBufHdr(buf_hdr);
+ buf_state &= ~(BM_VALID | BM_DIRTY);
+ UnlockBufHdr(buf_hdr, buf_state);
+
+ StartBufferIO(buf_hdr, true, false);
+
+ pgaio_io_set_io_data_32(ioh, (uint32 *) &buf, 1);
+ pgaio_io_add_shared_cb(ioh, ASC_SHARED_BUFFER_READ);
+
+ smgrstartreadv(ioh, smgr, MAIN_FORKNUM, block,
+ (void *) &page, 1);
+
+ ReleaseBuffer(buf);
+
+ pgaio_io_ref_wait(&ior);
+
+ relation_close(rel, NoLock);
+
+ PG_RETURN_VOID();
+}
+
+PG_FUNCTION_INFO_V1(invalidate_rel_block);
+Datum
+invalidate_rel_block(PG_FUNCTION_ARGS)
+{
+ Oid relid = PG_GETARG_OID(0);
+ uint32 block = PG_GETARG_UINT32(1);
+ Relation rel;
+ PrefetchBufferResult pr;
+ Buffer buf;
+
+ rel = relation_open(relid, AccessExclusiveLock);
+
+ /* this is a gross hack, but there's no good API exposed */
+ pr = PrefetchBuffer(rel, MAIN_FORKNUM, block);
+ buf = pr.recent_buffer;
+ elog(LOG, "recent: %d", buf);
+ if (BufferIsValid(buf))
+ {
+ /* if the buffer contents aren't valid, this'll return false */
+ if (ReadRecentBuffer(rel->rd_locator, MAIN_FORKNUM, block, buf))
+ {
+ LockBuffer(buf, BUFFER_LOCK_EXCLUSIVE);
+ FlushOneBuffer(buf);
+ LockBuffer(buf, BUFFER_LOCK_UNLOCK);
+ ReleaseBuffer(buf);
+
+ if (!EvictUnpinnedBuffer(buf))
+ elog(ERROR, "couldn't unpin");
+ }
+ }
+
+ relation_close(rel, AccessExclusiveLock);
+
+ PG_RETURN_VOID();
+}
+
+#if 0
+PG_FUNCTION_INFO_V1(test_unsubmitted_vs_close);
+Datum
+test_unsubmitted_vs_close(PG_FUNCTION_ARGS)
+{
+ Oid relid = PG_GETARG_OID(0);
+ uint32 block = PG_GETARG_UINT32(1);
+ Relation rel;
+ Buffer buf;
+ Page page;
+ PageHeader ph;
+
+ rel = relation_open(relid, AccessExclusiveLock);
+
+ buf = ReadBufferExtended(rel, MAIN_FORKNUM, block, RBM_ZERO_AND_LOCK, NULL);
+
+ buf = ReadBuffer(rel, block);
+ page = BufferGetPage(buf);
+
+ EvictUnpinnedBuffer(buf);
+
+ LockBuffer(buf, BUFFER_LOCK_UNLOCK);
+
+
+ MarkBufferDirty(buf);
+ ph->pd_special = BLCKSZ + 1;
+
+ /* last_handle = pgaio_io_get(); */
+
+ PG_RETURN_VOID();
+}
+#endif
+
+PG_FUNCTION_INFO_V1(handle_get);
+Datum
+handle_get(PG_FUNCTION_ARGS)
+{
+ last_handle = pgaio_io_get(CurrentResourceOwner, NULL);
+
+ PG_RETURN_VOID();
+}
+
+PG_FUNCTION_INFO_V1(handle_release_last);
+Datum
+handle_release_last(PG_FUNCTION_ARGS)
+{
+ if (!last_handle)
+ elog(ERROR, "no handle");
+
+ pgaio_io_release(last_handle);
+
+ PG_RETURN_VOID();
+}
+
+PG_FUNCTION_INFO_V1(handle_get_and_error);
+Datum
+handle_get_and_error(PG_FUNCTION_ARGS)
+{
+ pgaio_io_get(CurrentResourceOwner, NULL);
+
+ elog(ERROR, "as you command");
+ PG_RETURN_VOID();
+}
+
+PG_FUNCTION_INFO_V1(handle_get_twice);
+Datum
+handle_get_twice(PG_FUNCTION_ARGS)
+{
+ pgaio_io_get(CurrentResourceOwner, NULL);
+ pgaio_io_get(CurrentResourceOwner, NULL);
+
+ PG_RETURN_VOID();
+}
+
+
+PG_FUNCTION_INFO_V1(handle_get_release);
+Datum
+handle_get_release(PG_FUNCTION_ARGS)
+{
+ PgAioHandle *handle;
+
+ handle = pgaio_io_get(CurrentResourceOwner, NULL);
+ pgaio_io_release(handle);
+
+ PG_RETURN_VOID();
+}
+
+PG_FUNCTION_INFO_V1(bb_get);
+Datum
+bb_get(PG_FUNCTION_ARGS)
+{
+ last_bb = pgaio_bounce_buffer_get();
+
+ PG_RETURN_VOID();
+}
+
+PG_FUNCTION_INFO_V1(bb_release_last);
+Datum
+bb_release_last(PG_FUNCTION_ARGS)
+{
+ if (!last_bb)
+ elog(ERROR, "no bb");
+
+ pgaio_bounce_buffer_release(last_bb);
+
+ PG_RETURN_VOID();
+}
+
+PG_FUNCTION_INFO_V1(bb_get_and_error);
+Datum
+bb_get_and_error(PG_FUNCTION_ARGS)
+{
+ pgaio_bounce_buffer_get();
+
+ elog(ERROR, "as you command");
+ PG_RETURN_VOID();
+}
+
+PG_FUNCTION_INFO_V1(bb_get_twice);
+Datum
+bb_get_twice(PG_FUNCTION_ARGS)
+{
+ pgaio_bounce_buffer_get();
+ pgaio_bounce_buffer_get();
+
+ PG_RETURN_VOID();
+}
+
+
+PG_FUNCTION_INFO_V1(bb_get_release);
+Datum
+bb_get_release(PG_FUNCTION_ARGS)
+{
+ PgAioBounceBuffer *bb;
+
+ bb = pgaio_bounce_buffer_get();
+ pgaio_bounce_buffer_release(bb);
+
+ PG_RETURN_VOID();
+}
+
+#ifdef USE_INJECTION_POINTS
+extern PGDLLEXPORT void inj_io_short_read(const char *name, const void *private_data);
+
+void
+inj_io_short_read(const char *name, const void *private_data)
+{
+ PgAioHandle *ioh;
+
+ elog(LOG, "short read called: %d", inj_io_error_state->enabled);
+
+ if (inj_io_error_state->enabled)
+ {
+ ioh = pgaio_inj_io_get();
+
+ if (inj_io_error_state->result_set)
+ {
+ elog(LOG, "short read, changing result from %d to %d",
+ ioh->result, inj_io_error_state->result);
+
+ ioh->result = inj_io_error_state->result;
+ }
+ }
+}
+#endif
+
+PG_FUNCTION_INFO_V1(inj_io_short_read_attach);
+Datum
+inj_io_short_read_attach(PG_FUNCTION_ARGS)
+{
+#ifdef USE_INJECTION_POINTS
+ inj_io_error_state->enabled = true;
+ inj_io_error_state->result_set = !PG_ARGISNULL(0);
+ if (inj_io_error_state->result_set)
+ inj_io_error_state->result = PG_GETARG_INT32(0);
+#else
+ elog(ERROR, "injection points not supported");
+#endif
+
+ PG_RETURN_VOID();
+}
+
+PG_FUNCTION_INFO_V1(inj_io_short_read_detach);
+Datum
+inj_io_short_read_detach(PG_FUNCTION_ARGS)
+{
+#ifdef USE_INJECTION_POINTS
+ inj_io_error_state->enabled = false;
+#else
+ elog(ERROR, "injection points not supported");
+#endif
+ PG_RETURN_VOID();
+}
diff --git a/src/test/modules/test_aio/test_aio.control b/src/test/modules/test_aio/test_aio.control
new file mode 100644
index 00000000000..cd91c3ed16b
--- /dev/null
+++ b/src/test/modules/test_aio/test_aio.control
@@ -0,0 +1,3 @@
+comment = 'Test code for AIO'
+default_version = '1.0'
+module_pathname = '$libdir/test_aio'
diff --git a/src/test/modules/test_aio/worker.conf b/src/test/modules/test_aio/worker.conf
new file mode 100644
index 00000000000..8104c201924
--- /dev/null
+++ b/src/test/modules/test_aio/worker.conf
@@ -0,0 +1,5 @@
+shared_preload_libraries=test_aio
+io_method = 'worker'
+log_min_messages = 'DEBUG3'
+log_statement=all
+restart_after_crash=false
--
2.45.2.827.g557ae147e6
v2.0-0017-Temporary-Increase-BAS_BULKREAD-size.patch (text/x-diff, 1.3K) download
From 6ad40d5b074c4af85289e48b574c3461dbab9a4c Mon Sep 17 00:00:00 2001
From: Andres Freund <andres@anarazel.de>
Date: Sun, 1 Sep 2024 00:42:27 -0400
Subject: [PATCH v2.0 17/17] Temporary: Increase BAS_BULKREAD size
Without this we only can execute very little AIO for sequential scans, as
there's just not enough buffers in the ring. This isn't the right fix, as
just increasing the ring size can have negative performance implications in
workloads where the kernel has all the data cached.
Author:
Reviewed-By:
Discussion: https://postgr.es/m/
Backpatch:
---
src/backend/storage/buffer/freelist.c | 6 +++++-
1 file changed, 5 insertions(+), 1 deletion(-)
diff --git a/src/backend/storage/buffer/freelist.c b/src/backend/storage/buffer/freelist.c
index dffdd57e9b5..5be8125ad3a 100644
--- a/src/backend/storage/buffer/freelist.c
+++ b/src/backend/storage/buffer/freelist.c
@@ -555,7 +555,11 @@ GetAccessStrategy(BufferAccessStrategyType btype)
return NULL;
case BAS_BULKREAD:
- ring_size_kb = 256;
+ /*
+ * FIXME: Temporary increase to allow large enough streaming reads
+ * to actually benefit from AIO. This needs a better solution.
+ */
+ ring_size_kb = 2 * 1024;
break;
case BAS_BULKWRITE:
ring_size_kb = 16 * 1024;
--
2.45.2.827.g557ae147e6
* Re: AIO v2.0
@ 2024-09-02 10:03 Heikki Linnakangas <hlinnaka@iki.fi>
parent: Andres Freund <andres@anarazel.de>
2 siblings, 1 reply; 29+ messages in thread
From: Heikki Linnakangas @ 2024-09-02 10:03 UTC (permalink / raw)
To: Andres Freund <andres@anarazel.de>; pgsql-hackers
On 01/09/2024 09:27, Andres Freund wrote:
> The main reason I had previously implemented WAL AIO etc was to know the
> design implications - but now that they're somewhat understood, I'm planning
> to keep the patchset much smaller, with the goal of making it upstreamable.
+1 on that approach.
> To solve the issue with an unbounded number of AIO references there are few
> changes compared to the prior approach:
>
> 1) Only one AIO handle can be "handed out" to a backend, without being
> defined. Previously the process of getting an AIO handle wasn't super
> lightweight, which made it appealing to cache AIO handles - which was one
> part of the problem for running out of AIO handles.
>
> 2) Nothing in a backend can force a "defined" AIO handle (i.e. one that is a
> valid operation) to stay around, it's always possible to execute the AIO
> operation and then reuse the handle. This provides a forward guarantee, by
> ensuring that completing AIOs can free up handles (previously they couldn't
> be reused until the backend local reference was released).
>
> 3) Callbacks on AIOs are not allowed to error out anymore, unless it's ok to
> take the server down.
>
> 4) Obviously some code needs to know the result of AIO operation and be able
> to error out. To allow for that the issuer of an AIO can provide a pointer
> to local memory that'll receive the result of an AIO, including details
> about what kind of errors occurred (possible errors are e.g. a read failing
> or a buffer's checksum validation failing).
>
>
> In the next few days I'll add a bunch more documentation and comments as well
> as some better perf numbers (assuming my workstation survived...).
Yeah, a high-level README would be nice. Without that, it's hard to
follow what "handed out" and "defined" above means for example.
A few quick comments the patches:
v2.0-0001-bufmgr-Return-early-in-ScheduleBufferTagForWrit.patch
+1, this seems ready to be committed right away.
v2.0-0002-Allow-lwlocks-to-be-unowned.patch
With LOCK_DEBUG, LWLock->owner will point to the backend that acquired
the lock, but it doesn't own it anymore. That's reasonable, but maybe
add a boolean to the LWLock to mark whether the lock is currently owned
or not.
The LWLockReleaseOwnership() name is a bit confusing together with
LWLockReleaseUnowned() and LWLockrelease(). From the names, you might
think that they all release the lock, but LWLockReleaseOwnership() just
disassociates it from the current process. Rename it to LWLockDisown()
perhaps.
v2.0-0003-Use-aux-process-resource-owner-in-walsender.patch
+1. The old comment "We don't currently need any ResourceOwner in a
walsender process" was a bit misleading, because the walsender did
create the short-lived "base backup" resource owner, so it's nice to get
that fixed.
v2.0-0008-aio-Skeleton-IO-worker-infrastructure.patch
My refactoring around postmaster.c child process handling will conflict
with this [1]. Not in any fundamental way, but can I ask you to review
those patch, please? After those patches, AIO workers should also have
PMChild slots (formerly known as Backend structs).
[1]
https://www.postgresql.org/message-id/a102f15f-eac4-4ff2-af02-f9ff209ec66f@iki.fi
--
Heikki Linnakangas
Neon (https://neon.tech)
^ permalink raw reply [nested|flat] 29+ messages in thread
* Re: AIO v2.0
@ 2024-09-03 14:29 Andres Freund <andres@anarazel.de>
parent: Heikki Linnakangas <hlinnaka@iki.fi>
0 siblings, 1 reply; 29+ messages in thread
From: Andres Freund @ 2024-09-03 14:29 UTC (permalink / raw)
To: Heikki Linnakangas <hlinnaka@iki.fi>; +Cc: pgsql-hackers
Hi,
On 2024-09-02 13:03:07 +0300, Heikki Linnakangas wrote:
> On 01/09/2024 09:27, Andres Freund wrote:
> > In the next few days I'll add a bunch more documentation and comments as well
> > as some better perf numbers (assuming my workstation survived...).
>
> Yeah, a high-level README would be nice. Without that, it's hard to follow
> what "handed out" and "defined" above means for example.
Yea - I had actually written a bunch of that before, but then redesigns just
obsoleted most of it :(
FWIW, "handed out" is an IO handle acquired by code, which doesn't yet have an
operation associated with it. Once "defined" it actually could be - but isn't
yet - executed.
> A few quick comments the patches:
>
> v2.0-0001-bufmgr-Return-early-in-ScheduleBufferTagForWrit.patch
>
> +1, this seems ready to be committed right away.
Cool
> v2.0-0002-Allow-lwlocks-to-be-unowned.patch
>
> With LOCK_DEBUG, LWLock->owner will point to the backend that acquired the
> lock, but it doesn't own it anymore. That's reasonable, but maybe add a
> boolean to the LWLock to mark whether the lock is currently owned or not.
Hm, not sure it's worth doing that...
> The LWLockReleaseOwnership() name is a bit confusing together with
> LWLockReleaseUnowned() and LWLockrelease(). From the names, you might think
> that they all release the lock, but LWLockReleaseOwnership() just
> disassociates it from the current process. Rename it to LWLockDisown()
> perhaps.
Yea, that makes sense.
> v2.0-0008-aio-Skeleton-IO-worker-infrastructure.patch
>
> My refactoring around postmaster.c child process handling will conflict with
> this [1]. Not in any fundamental way, but can I ask you to review those
> patch, please? After those patches, AIO workers should also have PMChild
> slots (formerly known as Backend structs).
I'll try to do that soonish!
Greetings,
Andres Freund
^ permalink raw reply [nested|flat] 29+ messages in thread
* Re: AIO v2.0
@ 2024-09-04 17:37 陈宗志 <baotiao@gmail.com>
parent: Andres Freund <andres@anarazel.de>
0 siblings, 1 reply; 29+ messages in thread
From: 陈宗志 @ 2024-09-04 17:37 UTC (permalink / raw)
To: Andres Freund <andres@anarazel.de>; +Cc: Heikki Linnakangas <hlinnaka@iki.fi>; pgsql-hackers
I hope there can be a high-level design document that includes a
description, high-level architecture, and low-level design.
This way, others can also participate in reviewing the code.
For example, which paths were modified in the AIO module? Is it the
path for writing WAL logs, or the path for flushing pages, etc.?
Also, I recommend keeping this patch as small as possible.
For example, the first step could be to introduce libaio only, without
considering io_uring, as that would make it too complex.
^ permalink raw reply [nested|flat] 29+ messages in thread
* Re: AIO v2.0
@ 2024-09-06 01:42 David Rowley <dgrowleyml@gmail.com>
parent: Andres Freund <andres@anarazel.de>
2 siblings, 0 replies; 29+ messages in thread
From: David Rowley @ 2024-09-06 01:42 UTC (permalink / raw)
To: Andres Freund <andres@anarazel.de>; +Cc: pgsql-hackers
On Sun, 1 Sept 2024 at 18:28, Andres Freund <andres@anarazel.de> wrote:
> 0 workers 1 worker 2 workers 4 workers
> master: 65.753 33.246 21.095 12.918
> aio v2.0, worker: 21.519 12.636 10.450 10.004
> aio v2.0, uring*: 31.446 17.745 12.889 10.395
> aio v2.0, uring** 23.497 13.824 10.881 10.589
> aio v2.0, direct, worker: 22.377 11.989 09.915 09.772
> aio v2.0, direct, uring*: 24.502 12.603 10.058 09.759
I took this for a test drive on an AMD 3990x machine with a 1TB
Samsung 980 Pro SSD on PCIe 4. I only tried io_method = io_uring, but
I did try with and without direct IO.
This machine has 64GB RAM and I was using ClickBench Q2 [1], which is
"SELECT SUM(AdvEngineID), COUNT(*), AVG(ResolutionWidth) FROM hits;"
(for some reason they use 0-based query IDs). This table is 64GBs
without indexes.
I'm seeing direct IO slower than buffered IO with smaller worker
counts. That's counter to what I would have expected as I'd have
expected the memcpys from the kernel space to be quite an overhead in
the buffered IO case. With larger worker counts the bottleneck is
certainly disk. The part that surprised me was that the bottleneck is
reached more quickly with buffered IO. I was seeing iotop going up to
5.54GB/s at higher worker counts.
times in milliseconds
workers buffered direct cmp
0 58880 102852 57%
1 33622 53538 63%
2 24573 40436 61%
4 18557 27359 68%
8 14844 17330 86%
16 12491 12754 98%
32 11802 11956 99%
64 11895 11941 100%
Is there some other information I can provide to help this make sense?
(Or maybe it does already to you.)
David
[1] https://github.com/ClickHouse/ClickBench/blob/main/postgresql-tuned/queries.sql
^ permalink raw reply [nested|flat] 29+ messages in thread
* Re: AIO v2.0
@ 2024-09-06 19:38 Andres Freund <andres@anarazel.de>
parent: Andres Freund <andres@anarazel.de>
2 siblings, 2 replies; 29+ messages in thread
From: Andres Freund @ 2024-09-06 19:38 UTC (permalink / raw)
To: pgsql-hackers; +Cc: Heikki Linnakangas <hlinnaka@iki.fi>; 陈宗志 <baotiao@gmail.com>
Hi,
Attached is the next version of the patchset. Changes:
- added "sync" io method, the main benefit of that is that the main AIO commit
doesn't need to include worker mode
- split worker and io_uring methods into their own commits
- added src/backend/storage/aio/README.md, explaining design constraints and
the resulting design on a high level
- renamed LWLockReleaseOwnership as suggested by Heikki
- a bunch of small cleanups and improvements
There's plenty more to do, but I thought this would be a useful checkpoint.
Greetings,
Andres Freund
Attachments: download all (20 files, .tar.gz)
v2.1-0001-bufmgr-Return-early-in-ScheduleBufferTagForWrit.patch (text/x-diff, 1.3K) download
From ea3373e8793932e849d2904046f76b14ec971549 Mon Sep 17 00:00:00 2001
From: Andres Freund <andres@anarazel.de>
Date: Thu, 27 Jul 2023 18:59:25 -0700
Subject: [PATCH v2.1 01/20] bufmgr: Return early in
ScheduleBufferTagForWriteback() if fsync=off
As pg_flush_data() doesn't do anything with fsync disabled, there's no point
in tracking the buffer for writeback. Arguably the better fix would be to
change pg_flush_data() to flush data even with fsync off, but that's a
behavioral change, whereas this is just a small optimization.
Reviewed-by: Heikki Linnakangas <hlinnaka@iki.fi>
Discussion: https://postgr.es/m/1f6b50a7-38ef-4d87-8246-786d39f46ab9@iki.fi
---
src/backend/storage/buffer/bufmgr.c | 7 ++++++-
1 file changed, 6 insertions(+), 1 deletion(-)
diff --git a/src/backend/storage/buffer/bufmgr.c b/src/backend/storage/buffer/bufmgr.c
index 48520443001..b8680cc8fd4 100644
--- a/src/backend/storage/buffer/bufmgr.c
+++ b/src/backend/storage/buffer/bufmgr.c
@@ -5899,7 +5899,12 @@ ScheduleBufferTagForWriteback(WritebackContext *wb_context, IOContext io_context
{
PendingWriteback *pending;
- if (io_direct_flags & IO_DIRECT_DATA)
+ /*
+ * As pg_flush_data() doesn't do anything with fsync disabled, there's no
+ * point in tracking in that case.
+ */
+ if (io_direct_flags & IO_DIRECT_DATA ||
+ !enableFsync)
return;
/*
--
2.45.2.827.g557ae147e6
v2.1-0002-Allow-lwlocks-to-be-unowned.patch (text/x-diff, 5.0K) download
From 7daeafca64fd950bf63fb43cdb31fd578f27c85d Mon Sep 17 00:00:00 2001
From: Andres Freund <andres@anarazel.de>
Date: Tue, 5 Jan 2021 10:10:36 -0800
Subject: [PATCH v2.1 02/20] Allow lwlocks to be unowned
This is required for AIO so that the lock hold during a write can be released
in another backend. Which in turn is required to avoid the potential for
deadlocks.
---
src/include/storage/lwlock.h | 2 +
src/backend/storage/lmgr/lwlock.c | 110 ++++++++++++++++++++++--------
2 files changed, 82 insertions(+), 30 deletions(-)
diff --git a/src/include/storage/lwlock.h b/src/include/storage/lwlock.h
index d70e6d37e09..eabf813ce05 100644
--- a/src/include/storage/lwlock.h
+++ b/src/include/storage/lwlock.h
@@ -129,6 +129,8 @@ extern bool LWLockAcquireOrWait(LWLock *lock, LWLockMode mode);
extern void LWLockRelease(LWLock *lock);
extern void LWLockReleaseClearVar(LWLock *lock, pg_atomic_uint64 *valptr, uint64 val);
extern void LWLockReleaseAll(void);
+extern LWLockMode LWLockDisown(LWLock *l);
+extern void LWLockReleaseUnowned(LWLock *l, LWLockMode mode);
extern bool LWLockHeldByMe(LWLock *lock);
extern bool LWLockAnyHeldByMe(LWLock *lock, int nlocks, size_t stride);
extern bool LWLockHeldByMeInMode(LWLock *lock, LWLockMode mode);
diff --git a/src/backend/storage/lmgr/lwlock.c b/src/backend/storage/lmgr/lwlock.c
index db6ed784ab3..a5fa77412ed 100644
--- a/src/backend/storage/lmgr/lwlock.c
+++ b/src/backend/storage/lmgr/lwlock.c
@@ -1773,52 +1773,36 @@ LWLockUpdateVar(LWLock *lock, pg_atomic_uint64 *valptr, uint64 val)
}
}
-
-/*
- * LWLockRelease - release a previously acquired lock
- */
-void
-LWLockRelease(LWLock *lock)
+static void
+LWLockReleaseInternal(LWLock *lock, LWLockMode mode)
{
- LWLockMode mode;
uint32 oldstate;
bool check_waiters;
- int i;
-
- /*
- * Remove lock from list of locks held. Usually, but not always, it will
- * be the latest-acquired lock; so search array backwards.
- */
- for (i = num_held_lwlocks; --i >= 0;)
- if (lock == held_lwlocks[i].lock)
- break;
-
- if (i < 0)
- elog(ERROR, "lock %s is not held", T_NAME(lock));
-
- mode = held_lwlocks[i].mode;
-
- num_held_lwlocks--;
- for (; i < num_held_lwlocks; i++)
- held_lwlocks[i] = held_lwlocks[i + 1];
-
- PRINT_LWDEBUG("LWLockRelease", lock, mode);
/*
* Release my hold on lock, after that it can immediately be acquired by
* others, even if we still have to wakeup other waiters.
*/
if (mode == LW_EXCLUSIVE)
- oldstate = pg_atomic_sub_fetch_u32(&lock->state, LW_VAL_EXCLUSIVE);
+ oldstate = pg_atomic_fetch_sub_u32(&lock->state, LW_VAL_EXCLUSIVE);
else
- oldstate = pg_atomic_sub_fetch_u32(&lock->state, LW_VAL_SHARED);
+ oldstate = pg_atomic_fetch_sub_u32(&lock->state, LW_VAL_SHARED);
/* nobody else can have that kind of lock */
- Assert(!(oldstate & LW_VAL_EXCLUSIVE));
+ if (mode == LW_EXCLUSIVE)
+ Assert((oldstate & LW_LOCK_MASK) == LW_VAL_EXCLUSIVE);
+ else
+ Assert((oldstate & LW_LOCK_MASK) < LW_VAL_EXCLUSIVE &&
+ (oldstate & LW_LOCK_MASK) >= LW_VAL_SHARED);
if (TRACE_POSTGRESQL_LWLOCK_RELEASE_ENABLED())
TRACE_POSTGRESQL_LWLOCK_RELEASE(T_NAME(lock));
+ if (mode == LW_EXCLUSIVE)
+ oldstate -= LW_VAL_EXCLUSIVE;
+ else
+ oldstate -= LW_VAL_SHARED;
+
/*
* We're still waiting for backends to get scheduled, don't wake them up
* again.
@@ -1841,6 +1825,72 @@ LWLockRelease(LWLock *lock)
LWLockWakeup(lock);
}
+ TRACE_POSTGRESQL_LWLOCK_RELEASE(T_NAME(lock));
+}
+
+void
+LWLockReleaseUnowned(LWLock *lock, LWLockMode mode)
+{
+ LWLockReleaseInternal(lock, mode);
+}
+
+/*
+ * Stop treating lock as held by current backend.
+ *
+ * After calling this function it's the callers responsibility to ensure that
+ * the lock gets released, even in case of an error. This only is desirable if
+ * the lock is going to be released in a different process than the process
+ * that acquired it.
+ *
+ * Returns the mode in which the lock was held by the current backend.
+ *
+ * NB: This will leave lock->owner pointing to the current backend (if
+ * LOCK_DEBUG is set). We could add a separate flag indicating that, but it
+ * doesn't really seem worth it.
+ *
+ * NB: This does not call RESUME_INTERRUPTS(), but leaves that responsibility
+ * of the caller.
+ */
+LWLockMode
+LWLockDisown(LWLock *lock)
+{
+ LWLockMode mode;
+ int i;
+
+ /*
+ * Remove lock from list of locks held. Usually, but not always, it will
+ * be the latest-acquired lock; so search array backwards.
+ */
+ for (i = num_held_lwlocks; --i >= 0;)
+ if (lock == held_lwlocks[i].lock)
+ break;
+
+ if (i < 0)
+ elog(ERROR, "lock %s is not held", T_NAME(lock));
+
+ mode = held_lwlocks[i].mode;
+
+ num_held_lwlocks--;
+ for (; i < num_held_lwlocks; i++)
+ held_lwlocks[i] = held_lwlocks[i + 1];
+
+ return mode;
+}
+
+/*
+ * LWLockRelease - release a previously acquired lock
+ */
+void
+LWLockRelease(LWLock *lock)
+{
+ LWLockMode mode;
+
+ mode = LWLockDisown(lock);
+
+ PRINT_LWDEBUG("LWLockRelease", lock, mode);
+
+ LWLockReleaseInternal(lock, mode);
+
/*
* Now okay to allow cancel/die interrupts.
*/
--
2.45.2.827.g557ae147e6
v2.1-0003-Use-aux-process-resource-owner-in-walsender.patch (text/x-diff, 4.7K) download
From 6dacd88481c9c79042a6b5bdc5783ca8f8ce1cce Mon Sep 17 00:00:00 2001
From: Andres Freund <andres@anarazel.de>
Date: Tue, 31 Aug 2021 12:16:28 -0700
Subject: [PATCH v2.1 03/20] Use aux process resource owner in walsender
AIO will need a resource owner to do IO. Right now we create a resowner
on-demand during basebackup, and we could do the same for AIO. But it seems
easier to just always create an aux process resowner.
---
src/include/replication/walsender.h | 1 -
src/backend/backup/basebackup.c | 8 ++++--
src/backend/replication/walsender.c | 44 ++++++-----------------------
3 files changed, 13 insertions(+), 40 deletions(-)
diff --git a/src/include/replication/walsender.h b/src/include/replication/walsender.h
index f2d8297f016..aff0f7a51ca 100644
--- a/src/include/replication/walsender.h
+++ b/src/include/replication/walsender.h
@@ -38,7 +38,6 @@ extern PGDLLIMPORT bool log_replication_commands;
extern void InitWalSender(void);
extern bool exec_replication_command(const char *cmd_string);
extern void WalSndErrorCleanup(void);
-extern void WalSndResourceCleanup(bool isCommit);
extern void PhysicalWakeupLogicalWalSnd(void);
extern XLogRecPtr GetStandbyFlushRecPtr(TimeLineID *tli);
extern void WalSndSignals(void);
diff --git a/src/backend/backup/basebackup.c b/src/backend/backup/basebackup.c
index 14e5ba72e97..0f8cddcbeeb 100644
--- a/src/backend/backup/basebackup.c
+++ b/src/backend/backup/basebackup.c
@@ -250,8 +250,10 @@ perform_base_backup(basebackup_options *opt, bbsink *sink,
state.bytes_total_is_valid = false;
/* we're going to use a BufFile, so we need a ResourceOwner */
- Assert(CurrentResourceOwner == NULL);
- CurrentResourceOwner = ResourceOwnerCreate(NULL, "base backup");
+ Assert(AuxProcessResourceOwner != NULL);
+ Assert(CurrentResourceOwner == AuxProcessResourceOwner ||
+ CurrentResourceOwner == NULL);
+ CurrentResourceOwner = AuxProcessResourceOwner;
backup_started_in_recovery = RecoveryInProgress();
@@ -672,7 +674,7 @@ perform_base_backup(basebackup_options *opt, bbsink *sink,
FreeBackupManifest(&manifest);
/* clean up the resource owner we created */
- WalSndResourceCleanup(true);
+ ReleaseAuxProcessResources(true);
basebackup_progress_done();
}
diff --git a/src/backend/replication/walsender.c b/src/backend/replication/walsender.c
index c5f1009f370..0e847535a64 100644
--- a/src/backend/replication/walsender.c
+++ b/src/backend/replication/walsender.c
@@ -282,10 +282,8 @@ InitWalSender(void)
/* Create a per-walsender data structure in shared memory */
InitWalSenderSlot();
- /*
- * We don't currently need any ResourceOwner in a walsender process, but
- * if we did, we could call CreateAuxProcessResourceOwner here.
- */
+ /* need resource owner for e.g. basebackups */
+ CreateAuxProcessResourceOwner();
/*
* Let postmaster know that we're a WAL sender. Once we've declared us as
@@ -346,7 +344,7 @@ WalSndErrorCleanup(void)
* without a transaction, we've got to clean that up now.
*/
if (!IsTransactionOrTransactionBlock())
- WalSndResourceCleanup(false);
+ ReleaseAuxProcessResources(false);
if (got_STOPPING || got_SIGUSR2)
proc_exit(0);
@@ -355,34 +353,6 @@ WalSndErrorCleanup(void)
WalSndSetState(WALSNDSTATE_STARTUP);
}
-/*
- * Clean up any ResourceOwner we created.
- */
-void
-WalSndResourceCleanup(bool isCommit)
-{
- ResourceOwner resowner;
-
- if (CurrentResourceOwner == NULL)
- return;
-
- /*
- * Deleting CurrentResourceOwner is not allowed, so we must save a pointer
- * in a local variable and clear it first.
- */
- resowner = CurrentResourceOwner;
- CurrentResourceOwner = NULL;
-
- /* Now we can release resources and delete it. */
- ResourceOwnerRelease(resowner,
- RESOURCE_RELEASE_BEFORE_LOCKS, isCommit, true);
- ResourceOwnerRelease(resowner,
- RESOURCE_RELEASE_LOCKS, isCommit, true);
- ResourceOwnerRelease(resowner,
- RESOURCE_RELEASE_AFTER_LOCKS, isCommit, true);
- ResourceOwnerDelete(resowner);
-}
-
/*
* Handle a client's connection abort in an orderly manner.
*/
@@ -685,8 +655,10 @@ UploadManifest(void)
* parsing the manifest will use the cryptohash stuff, which requires a
* resource owner
*/
- Assert(CurrentResourceOwner == NULL);
- CurrentResourceOwner = ResourceOwnerCreate(NULL, "base backup");
+ Assert(AuxProcessResourceOwner != NULL);
+ Assert(CurrentResourceOwner == AuxProcessResourceOwner ||
+ CurrentResourceOwner == NULL);
+ CurrentResourceOwner = AuxProcessResourceOwner;
/* Prepare to read manifest data into a temporary context. */
mcxt = AllocSetContextCreate(CurrentMemoryContext,
@@ -723,7 +695,7 @@ UploadManifest(void)
uploaded_manifest_mcxt = mcxt;
/* clean up the resource owner we created */
- WalSndResourceCleanup(true);
+ ReleaseAuxProcessResources(true);
}
/*
--
2.45.2.827.g557ae147e6
v2.1-0004-Ensure-a-resowner-exists-for-all-paths-that-may.patch (text/x-diff, 2.3K) download
From a70eeb4cc7dd87a693162f0632d5d60bfa17575e Mon Sep 17 00:00:00 2001
From: Andres Freund <andres@anarazel.de>
Date: Thu, 1 Aug 2024 09:56:36 -0700
Subject: [PATCH v2.1 04/20] Ensure a resowner exists for all paths that may
perform AIO
---
src/backend/bootstrap/bootstrap.c | 7 +++++++
src/backend/replication/logical/logical.c | 6 ++++++
src/backend/utils/init/postinit.c | 3 ++-
3 files changed, 15 insertions(+), 1 deletion(-)
diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c
index 7637581a184..234fdc57ca7 100644
--- a/src/backend/bootstrap/bootstrap.c
+++ b/src/backend/bootstrap/bootstrap.c
@@ -331,8 +331,15 @@ BootstrapModeMain(int argc, char *argv[], bool check_only)
BaseInit();
bootstrap_signals();
+
+ /* need a resowner for IO during BootStrapXLOG() */
+ CreateAuxProcessResourceOwner();
+
BootStrapXLOG(bootstrap_data_checksum_version);
+ ReleaseAuxProcessResources(true);
+ CurrentResourceOwner = NULL;
+
/*
* To ensure that src/common/link-canary.c is linked into the backend, we
* must call it from somewhere. Here is as good as anywhere.
diff --git a/src/backend/replication/logical/logical.c b/src/backend/replication/logical/logical.c
index 3fe1774a1e9..be0c7846d00 100644
--- a/src/backend/replication/logical/logical.c
+++ b/src/backend/replication/logical/logical.c
@@ -386,6 +386,12 @@ CreateInitDecodingContext(const char *plugin,
slot->data.plugin = plugin_name;
SpinLockRelease(&slot->mutex);
+ if (CurrentResourceOwner == NULL)
+ {
+ Assert(am_walsender);
+ CurrentResourceOwner = AuxProcessResourceOwner;
+ }
+
if (XLogRecPtrIsInvalid(restart_lsn))
ReplicationSlotReserveWal();
else
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index 3b50ce19a2c..11128ea461c 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -719,7 +719,8 @@ InitPostgres(const char *in_dbname, Oid dboid,
* and ShutdownXLOG will need one. Hence, create said resource owner
* (and register a callback to clean it up after ShutdownXLOG runs).
*/
- CreateAuxProcessResourceOwner();
+ if (!bootstrap)
+ CreateAuxProcessResourceOwner();
StartupXLOG();
/* Release (and warn about) any buffer pins leaked in StartupXLOG */
--
2.45.2.827.g557ae147e6
v2.1-0005-bufmgr-smgr-Don-t-cross-segment-boundaries-in-S.patch (text/x-diff, 6.2K) download
From 5308c29e3fd09601ad2e63669837f1e7eef45921 Mon Sep 17 00:00:00 2001
From: Andres Freund <andres@anarazel.de>
Date: Sat, 31 Aug 2024 22:10:35 -0400
Subject: [PATCH v2.1 05/20] bufmgr/smgr: Don't cross segment boundaries in
StartReadBuffers()
With real AIO it doesn't make sense to cross segment boundaries with one
IO. Add smgrmaxcombine() to allow upper layers to query which buffers can be
merged.
---
src/include/storage/md.h | 2 ++
src/include/storage/smgr.h | 2 ++
src/backend/storage/buffer/bufmgr.c | 18 ++++++++++++++++++
src/backend/storage/smgr/md.c | 17 +++++++++++++++++
src/backend/storage/smgr/smgr.c | 16 ++++++++++++++++
5 files changed, 55 insertions(+)
diff --git a/src/include/storage/md.h b/src/include/storage/md.h
index 620f10abdeb..b72293c79a5 100644
--- a/src/include/storage/md.h
+++ b/src/include/storage/md.h
@@ -32,6 +32,8 @@ extern void mdzeroextend(SMgrRelation reln, ForkNumber forknum,
BlockNumber blocknum, int nblocks, bool skipFsync);
extern bool mdprefetch(SMgrRelation reln, ForkNumber forknum,
BlockNumber blocknum, int nblocks);
+extern uint32 mdmaxcombine(SMgrRelation reln, ForkNumber forknum,
+ BlockNumber blocknum);
extern void mdreadv(SMgrRelation reln, ForkNumber forknum, BlockNumber blocknum,
void **buffers, BlockNumber nblocks);
extern void mdwritev(SMgrRelation reln, ForkNumber forknum,
diff --git a/src/include/storage/smgr.h b/src/include/storage/smgr.h
index e15b20a566a..899d0d681c5 100644
--- a/src/include/storage/smgr.h
+++ b/src/include/storage/smgr.h
@@ -92,6 +92,8 @@ extern void smgrzeroextend(SMgrRelation reln, ForkNumber forknum,
BlockNumber blocknum, int nblocks, bool skipFsync);
extern bool smgrprefetch(SMgrRelation reln, ForkNumber forknum,
BlockNumber blocknum, int nblocks);
+extern uint32 smgrmaxcombine(SMgrRelation reln, ForkNumber forknum,
+ BlockNumber blocknum);
extern void smgrreadv(SMgrRelation reln, ForkNumber forknum,
BlockNumber blocknum,
void **buffers, BlockNumber nblocks);
diff --git a/src/backend/storage/buffer/bufmgr.c b/src/backend/storage/buffer/bufmgr.c
index b8680cc8fd4..7e987836335 100644
--- a/src/backend/storage/buffer/bufmgr.c
+++ b/src/backend/storage/buffer/bufmgr.c
@@ -1259,6 +1259,7 @@ StartReadBuffersImpl(ReadBuffersOperation *operation,
{
int actual_nblocks = *nblocks;
int io_buffers_len = 0;
+ int maxcombine = 0;
Assert(*nblocks > 0);
Assert(*nblocks <= MAX_IO_COMBINE_LIMIT);
@@ -1290,6 +1291,23 @@ StartReadBuffersImpl(ReadBuffersOperation *operation,
{
/* Extend the readable range to cover this block. */
io_buffers_len++;
+
+ /*
+ * Check how many blocks we can cover with the same IO. The smgr
+ * implementation might e.g. be limited due to a segment boundary.
+ */
+ if (i == 0 && actual_nblocks > 1)
+ {
+ maxcombine = smgrmaxcombine(operation->smgr,
+ operation->forknum,
+ blockNum);
+ if (maxcombine < actual_nblocks)
+ {
+ elog(DEBUG2, "limiting nblocks at %u from %u to %u",
+ blockNum, actual_nblocks, maxcombine);
+ actual_nblocks = maxcombine;
+ }
+ }
}
}
*nblocks = actual_nblocks;
diff --git a/src/backend/storage/smgr/md.c b/src/backend/storage/smgr/md.c
index 6796756358f..6cd81a61faa 100644
--- a/src/backend/storage/smgr/md.c
+++ b/src/backend/storage/smgr/md.c
@@ -803,6 +803,17 @@ buffers_to_iovec(struct iovec *iov, void **buffers, int nblocks)
return iovcnt;
}
+uint32
+mdmaxcombine(SMgrRelation reln, ForkNumber forknum,
+ BlockNumber blocknum)
+{
+ BlockNumber segoff;
+
+ segoff = blocknum % ((BlockNumber) RELSEG_SIZE);
+
+ return RELSEG_SIZE - segoff;
+}
+
/*
* mdreadv() -- Read the specified blocks from a relation.
*/
@@ -833,6 +844,9 @@ mdreadv(SMgrRelation reln, ForkNumber forknum, BlockNumber blocknum,
RELSEG_SIZE - (blocknum % ((BlockNumber) RELSEG_SIZE)));
nblocks_this_segment = Min(nblocks_this_segment, lengthof(iov));
+ if (nblocks_this_segment != nblocks)
+ elog(ERROR, "read crossing segment boundary");
+
iovcnt = buffers_to_iovec(iov, buffers, nblocks_this_segment);
size_this_segment = nblocks_this_segment * BLCKSZ;
transferred_this_segment = 0;
@@ -956,6 +970,9 @@ mdwritev(SMgrRelation reln, ForkNumber forknum, BlockNumber blocknum,
RELSEG_SIZE - (blocknum % ((BlockNumber) RELSEG_SIZE)));
nblocks_this_segment = Min(nblocks_this_segment, lengthof(iov));
+ if (nblocks_this_segment != nblocks)
+ elog(ERROR, "write crossing segment boundary");
+
iovcnt = buffers_to_iovec(iov, (void **) buffers, nblocks_this_segment);
size_this_segment = nblocks_this_segment * BLCKSZ;
transferred_this_segment = 0;
diff --git a/src/backend/storage/smgr/smgr.c b/src/backend/storage/smgr/smgr.c
index 7b9fa103eff..ee31db85eec 100644
--- a/src/backend/storage/smgr/smgr.c
+++ b/src/backend/storage/smgr/smgr.c
@@ -88,6 +88,8 @@ typedef struct f_smgr
BlockNumber blocknum, int nblocks, bool skipFsync);
bool (*smgr_prefetch) (SMgrRelation reln, ForkNumber forknum,
BlockNumber blocknum, int nblocks);
+ uint32 (*smgr_maxcombine) (SMgrRelation reln, ForkNumber forknum,
+ BlockNumber blocknum);
void (*smgr_readv) (SMgrRelation reln, ForkNumber forknum,
BlockNumber blocknum,
void **buffers, BlockNumber nblocks);
@@ -117,6 +119,7 @@ static const f_smgr smgrsw[] = {
.smgr_extend = mdextend,
.smgr_zeroextend = mdzeroextend,
.smgr_prefetch = mdprefetch,
+ .smgr_maxcombine = mdmaxcombine,
.smgr_readv = mdreadv,
.smgr_writev = mdwritev,
.smgr_writeback = mdwriteback,
@@ -588,6 +591,19 @@ smgrprefetch(SMgrRelation reln, ForkNumber forknum, BlockNumber blocknum,
return smgrsw[reln->smgr_which].smgr_prefetch(reln, forknum, blocknum, nblocks);
}
+/*
+ * smgrmaxcombine() - Return the maximum number of total blocks that can be
+ * combined with an IO starting at blocknum.
+ *
+ * The returned value includes the io for blocknum itself.
+ */
+uint32
+smgrmaxcombine(SMgrRelation reln, ForkNumber forknum,
+ BlockNumber blocknum)
+{
+ return smgrsw[reln->smgr_which].smgr_maxcombine(reln, forknum, blocknum);
+}
+
/*
* smgrreadv() -- read a particular block range from a relation into the
* supplied buffers.
--
2.45.2.827.g557ae147e6
v2.1-0006-aio-Basic-subsystem-initialization.patch (text/x-diff, 11.6K) download
From 177af4d07a51bac7b785dc02b2abea019d7395e4 Mon Sep 17 00:00:00 2001
From: Andres Freund <andres@anarazel.de>
Date: Mon, 10 Jun 2024 13:42:58 -0700
Subject: [PATCH v2.1 06/20] aio: Basic subsystem initialization
This is just separate to make it easier to review the tendrils into various
places.
---
src/include/storage/aio.h | 41 +++++++++++++++++
src/include/storage/aio_init.h | 26 +++++++++++
src/backend/postmaster/postmaster.c | 8 ++++
src/backend/storage/aio/Makefile | 2 +
src/backend/storage/aio/aio.c | 32 +++++++++++++
src/backend/storage/aio/aio_init.c | 46 +++++++++++++++++++
src/backend/storage/aio/meson.build | 2 +
src/backend/storage/ipc/ipci.c | 3 ++
src/backend/tcop/postgres.c | 7 +++
src/backend/utils/init/miscinit.c | 3 ++
src/backend/utils/init/postinit.c | 3 ++
src/backend/utils/misc/guc_tables.c | 11 +++++
src/backend/utils/misc/postgresql.conf.sample | 7 +++
src/tools/pgindent/typedefs.list | 1 +
14 files changed, 192 insertions(+)
create mode 100644 src/include/storage/aio.h
create mode 100644 src/include/storage/aio_init.h
create mode 100644 src/backend/storage/aio/aio.c
create mode 100644 src/backend/storage/aio/aio_init.c
diff --git a/src/include/storage/aio.h b/src/include/storage/aio.h
new file mode 100644
index 00000000000..1e4dfd07e89
--- /dev/null
+++ b/src/include/storage/aio.h
@@ -0,0 +1,41 @@
+/*-------------------------------------------------------------------------
+ *
+ * aio.h
+ * Main AIO interface
+ *
+ *
+ * Portions Copyright (c) 1996-2024, PostgreSQL Global Development Group
+ * Portions Copyright (c) 1994, Regents of the University of California
+ *
+ * src/include/storage/aio.h
+ *
+ *-------------------------------------------------------------------------
+ */
+#ifndef AIO_H
+#define AIO_H
+
+
+#include "utils/guc_tables.h"
+
+
+/* GUC related */
+extern void assign_io_method(int newval, void *extra);
+
+
+/* Enum for io_method GUC. */
+typedef enum IoMethod
+{
+ IOMETHOD_SYNC = 0,
+} IoMethod;
+
+
+/* We'll default to synchronous execution. */
+#define DEFAULT_IO_METHOD IOMETHOD_SYNC
+
+
+/* GUCs */
+extern const struct config_enum_entry io_method_options[];
+extern int io_method;
+
+
+#endif /* AIO_H */
diff --git a/src/include/storage/aio_init.h b/src/include/storage/aio_init.h
new file mode 100644
index 00000000000..5bcfb8a9d58
--- /dev/null
+++ b/src/include/storage/aio_init.h
@@ -0,0 +1,26 @@
+/*-------------------------------------------------------------------------
+ *
+ * aio_init.h
+ * AIO initialization - kept separate as initialization sites don't need to
+ * know about AIO itself and AIO users don't need to know about initialization.
+ *
+ *
+ * Portions Copyright (c) 1996-2024, PostgreSQL Global Development Group
+ * Portions Copyright (c) 1994, Regents of the University of California
+ *
+ * src/include/storage/aio_init.h
+ *
+ *-------------------------------------------------------------------------
+ */
+#ifndef AIO_INIT_H
+#define AIO_INIT_H
+
+
+extern Size AioShmemSize(void);
+extern void AioShmemInit(void);
+
+extern void pgaio_postmaster_init(void);
+extern void pgaio_postmaster_child_init_local(void);
+extern void pgaio_postmaster_child_init(void);
+
+#endif /* AIO_INIT_H */
diff --git a/src/backend/postmaster/postmaster.c b/src/backend/postmaster/postmaster.c
index 96bc1d1cfed..70c5ce19f6e 100644
--- a/src/backend/postmaster/postmaster.c
+++ b/src/backend/postmaster/postmaster.c
@@ -111,6 +111,7 @@
#include "replication/logicallauncher.h"
#include "replication/slotsync.h"
#include "replication/walsender.h"
+#include "storage/aio_init.h"
#include "storage/fd.h"
#include "storage/ipc.h"
#include "storage/pmsignal.h"
@@ -941,6 +942,13 @@ PostmasterMain(int argc, char *argv[])
ExitPostmaster(0);
}
+ /*
+ * As AIO might create internal FDs, and will trigger shared memory
+ * allocations, need to do this before reset_shared() and
+ * set_max_safe_fds().
+ */
+ pgaio_postmaster_init();
+
/*
* Set up shared memory and semaphores.
*
diff --git a/src/backend/storage/aio/Makefile b/src/backend/storage/aio/Makefile
index 2f29a9ec4d1..eaeaeeee8e3 100644
--- a/src/backend/storage/aio/Makefile
+++ b/src/backend/storage/aio/Makefile
@@ -9,6 +9,8 @@ top_builddir = ../../../..
include $(top_builddir)/src/Makefile.global
OBJS = \
+ aio.o \
+ aio_init.o \
read_stream.o
include $(top_srcdir)/src/backend/common.mk
diff --git a/src/backend/storage/aio/aio.c b/src/backend/storage/aio/aio.c
new file mode 100644
index 00000000000..d831c772960
--- /dev/null
+++ b/src/backend/storage/aio/aio.c
@@ -0,0 +1,32 @@
+/*-------------------------------------------------------------------------
+ *
+ * aio.c
+ * Asynchronous I/O subsytem.
+ *
+ * Portions Copyright (c) 1996-2024, PostgreSQL Global Development Group
+ * Portions Copyright (c) 1994, Regents of the University of California
+ *
+ * IDENTIFICATION
+ * src/backend/storage/aio/aio.c
+ *
+ *-------------------------------------------------------------------------
+ */
+
+#include "postgres.h"
+
+#include "storage/aio.h"
+
+
+/* Options for io_method. */
+const struct config_enum_entry io_method_options[] = {
+ {"sync", IOMETHOD_SYNC, false},
+ {NULL, 0, false}
+};
+
+int io_method = DEFAULT_IO_METHOD;
+
+
+void
+assign_io_method(int newval, void *extra)
+{
+}
diff --git a/src/backend/storage/aio/aio_init.c b/src/backend/storage/aio/aio_init.c
new file mode 100644
index 00000000000..1c277a7eb3b
--- /dev/null
+++ b/src/backend/storage/aio/aio_init.c
@@ -0,0 +1,46 @@
+/*-------------------------------------------------------------------------
+ *
+ * aio_init.c
+ * Asynchronous I/O subsytem - Initialization
+ *
+ * Portions Copyright (c) 1996-2024, PostgreSQL Global Development Group
+ * Portions Copyright (c) 1994, Regents of the University of California
+ *
+ * IDENTIFICATION
+ * src/backend/storage/aio/aio_init.c
+ *
+ *-------------------------------------------------------------------------
+ */
+
+#include "postgres.h"
+
+#include "storage/aio_init.h"
+
+
+Size
+AioShmemSize(void)
+{
+ Size sz = 0;
+
+ return sz;
+}
+
+void
+AioShmemInit(void)
+{
+}
+
+void
+pgaio_postmaster_init(void)
+{
+}
+
+void
+pgaio_postmaster_child_init(void)
+{
+}
+
+void
+pgaio_postmaster_child_init_local(void)
+{
+}
diff --git a/src/backend/storage/aio/meson.build b/src/backend/storage/aio/meson.build
index 10e1aa3b20b..8d20759ebf8 100644
--- a/src/backend/storage/aio/meson.build
+++ b/src/backend/storage/aio/meson.build
@@ -1,5 +1,7 @@
# Copyright (c) 2024, PostgreSQL Global Development Group
backend_sources += files(
+ 'aio.c',
+ 'aio_init.c',
'read_stream.c',
)
diff --git a/src/backend/storage/ipc/ipci.c b/src/backend/storage/ipc/ipci.c
index 6caeca3a8e6..f0227a12a7d 100644
--- a/src/backend/storage/ipc/ipci.c
+++ b/src/backend/storage/ipc/ipci.c
@@ -39,6 +39,7 @@
#include "replication/slotsync.h"
#include "replication/walreceiver.h"
#include "replication/walsender.h"
+#include "storage/aio_init.h"
#include "storage/bufmgr.h"
#include "storage/dsm.h"
#include "storage/dsm_registry.h"
@@ -152,6 +153,7 @@ CalculateShmemSize(int *num_semaphores)
size = add_size(size, InjectionPointShmemSize());
size = add_size(size, SlotSyncShmemSize());
size = add_size(size, WaitLSNShmemSize());
+ size = add_size(size, AioShmemSize());
/* include additional requested shmem from preload libraries */
size = add_size(size, total_addin_request);
@@ -339,6 +341,7 @@ CreateOrAttachShmemStructs(void)
WaitEventCustomShmemInit();
InjectionPointShmemInit();
WaitLSNShmemInit();
+ AioShmemInit();
}
/*
diff --git a/src/backend/tcop/postgres.c b/src/backend/tcop/postgres.c
index 8bc6bea1135..4dc46b17b41 100644
--- a/src/backend/tcop/postgres.c
+++ b/src/backend/tcop/postgres.c
@@ -61,6 +61,7 @@
#include "replication/slot.h"
#include "replication/walsender.h"
#include "rewrite/rewriteHandler.h"
+#include "storage/aio_init.h"
#include "storage/bufmgr.h"
#include "storage/ipc.h"
#include "storage/pmsignal.h"
@@ -4198,6 +4199,12 @@ PostgresSingleUserMain(int argc, char *argv[],
*/
InitProcess();
+ /* AIO is needed during InitPostgres() */
+ pgaio_postmaster_init();
+ pgaio_postmaster_child_init_local();
+
+ set_max_safe_fds();
+
/*
* Now that sufficient infrastructure has been initialized, PostgresMain()
* can do the rest.
diff --git a/src/backend/utils/init/miscinit.c b/src/backend/utils/init/miscinit.c
index 537d92c0cfd..b8fa2e64ffe 100644
--- a/src/backend/utils/init/miscinit.c
+++ b/src/backend/utils/init/miscinit.c
@@ -40,6 +40,7 @@
#include "postmaster/interrupt.h"
#include "postmaster/postmaster.h"
#include "replication/slotsync.h"
+#include "storage/aio_init.h"
#include "storage/fd.h"
#include "storage/ipc.h"
#include "storage/latch.h"
@@ -137,6 +138,8 @@ InitPostmasterChild(void)
InitProcessLocalLatch();
InitializeLatchWaitSet();
+ pgaio_postmaster_child_init_local();
+
/*
* If possible, make this process a group leader, so that the postmaster
* can signal any child processes too. Not all processes will have
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index 11128ea461c..f1151645242 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -43,6 +43,7 @@
#include "replication/slot.h"
#include "replication/slotsync.h"
#include "replication/walsender.h"
+#include "storage/aio_init.h"
#include "storage/bufmgr.h"
#include "storage/fd.h"
#include "storage/ipc.h"
@@ -589,6 +590,8 @@ BaseInit(void)
*/
pgstat_initialize();
+ pgaio_postmaster_child_init();
+
/* Do local initialization of storage and buffer managers */
InitSync();
smgrinit();
diff --git a/src/backend/utils/misc/guc_tables.c b/src/backend/utils/misc/guc_tables.c
index 686309db58b..a4b3c7c62bd 100644
--- a/src/backend/utils/misc/guc_tables.c
+++ b/src/backend/utils/misc/guc_tables.c
@@ -71,6 +71,7 @@
#include "replication/slot.h"
#include "replication/slotsync.h"
#include "replication/syncrep.h"
+#include "storage/aio.h"
#include "storage/bufmgr.h"
#include "storage/bufpage.h"
#include "storage/large_object.h"
@@ -5196,6 +5197,16 @@ struct config_enum ConfigureNamesEnum[] =
NULL, NULL, NULL
},
+ {
+ {"io_method", PGC_POSTMASTER, RESOURCES_MEM,
+ gettext_noop("Selects the method of asynchronous I/O to use."),
+ NULL
+ },
+ &io_method,
+ DEFAULT_IO_METHOD, io_method_options,
+ NULL, assign_io_method, NULL
+ },
+
/* End-of-list marker */
{
{NULL, 0, 0, NULL, NULL}, NULL, 0, NULL, NULL, NULL, NULL
diff --git a/src/backend/utils/misc/postgresql.conf.sample b/src/backend/utils/misc/postgresql.conf.sample
index 667e0dc40a2..3a5e307c9dc 100644
--- a/src/backend/utils/misc/postgresql.conf.sample
+++ b/src/backend/utils/misc/postgresql.conf.sample
@@ -835,6 +835,13 @@
#include = '...' # include file
+#------------------------------------------------------------------------------
+# WIP AIO GUC docs
+#------------------------------------------------------------------------------
+
+#io_method = sync # (change requires restart)
+
+
#------------------------------------------------------------------------------
# CUSTOMIZED OPTIONS
#------------------------------------------------------------------------------
diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list
index df3f336bec0..2681dd51bb7 100644
--- a/src/tools/pgindent/typedefs.list
+++ b/src/tools/pgindent/typedefs.list
@@ -1258,6 +1258,7 @@ IntervalAggState
IntoClause
InvalMessageArray
InvalidationMsgsGroup
+IoMethod
IpcMemoryId
IpcMemoryKey
IpcMemoryState
--
2.45.2.827.g557ae147e6
v2.1-0007-aio-Core-AIO-implementation.patch (text/x-diff, 66.6K) download
From d1c318432d40aee43b46db6187a033872af96b31 Mon Sep 17 00:00:00 2001
From: Andres Freund <andres@anarazel.de>
Date: Wed, 4 Sep 2024 15:23:08 -0400
Subject: [PATCH v2.1 07/20] aio: Core AIO implementation
At this point nothing can use AIO - this commit does not include any
implementation of aio subjects / callbacks. That will come in later commits.
Todo:
- lots of cleanup
---
src/include/storage/aio.h | 308 ++++++
src/include/storage/aio_internal.h | 274 +++++
src/include/storage/aio_ref.h | 24 +
src/include/utils/resowner.h | 7 +
src/backend/access/transam/xact.c | 9 +
src/backend/storage/aio/Makefile | 3 +
src/backend/storage/aio/aio.c | 975 ++++++++++++++++++
src/backend/storage/aio/aio_init.c | 304 ++++++
src/backend/storage/aio/aio_io.c | 111 ++
src/backend/storage/aio/aio_subject.c | 167 +++
src/backend/storage/aio/meson.build | 3 +
src/backend/storage/aio/method_sync.c | 43 +
.../utils/activity/wait_event_names.txt | 3 +
src/backend/utils/misc/guc_tables.c | 25 +
src/backend/utils/misc/postgresql.conf.sample | 6 +
src/backend/utils/resowner/resowner.c | 51 +
src/tools/pgindent/typedefs.list | 19 +
17 files changed, 2332 insertions(+)
create mode 100644 src/include/storage/aio_internal.h
create mode 100644 src/include/storage/aio_ref.h
create mode 100644 src/backend/storage/aio/aio_io.c
create mode 100644 src/backend/storage/aio/aio_subject.c
create mode 100644 src/backend/storage/aio/method_sync.c
diff --git a/src/include/storage/aio.h b/src/include/storage/aio.h
index 1e4dfd07e89..c0a59f47bc0 100644
--- a/src/include/storage/aio.h
+++ b/src/include/storage/aio.h
@@ -15,9 +15,315 @@
#define AIO_H
+#include "storage/aio_ref.h"
+#include "storage/procnumber.h"
#include "utils/guc_tables.h"
+typedef struct PgAioHandle PgAioHandle;
+
+typedef enum PgAioOp
+{
+ /* intentionally the zero value, to help catch zeroed memory etc */
+ PGAIO_OP_INVALID = 0,
+
+ PGAIO_OP_READ,
+ PGAIO_OP_WRITE,
+
+ PGAIO_OP_FSYNC,
+
+ PGAIO_OP_FLUSH_RANGE,
+
+ PGAIO_OP_NOP,
+
+ /**
+ * Eventually we'll additionally want at least:
+ * - send
+ * - recv
+ * - accept
+ **/
+} PgAioOp;
+
+#define PGAIO_OP_COUNT (PGAIO_OP_NOP + 1)
+
+
+/*
+ * On what is IO being performed.
+ *
+ * PgAioSharedCallback specific behaviour should be implemented in
+ * aio_subject.c.
+ */
+typedef enum PgAioSubjectID
+{
+ /* intentionally the zero value, to help catch zeroed memory etc */
+ ASI_INVALID = 0,
+} PgAioSubjectID;
+
+#define ASI_COUNT (ASI_INVALID + 1)
+
+/*
+ * Flags for an IO that can be set with pgaio_io_set_flag().
+ */
+typedef enum PgAioHandleFlags
+{
+ AHF_REFERENCES_LOCAL = 1 << 0,
+} PgAioHandleFlags;
+
+
+/*
+ * IDs for callbacks that can be registered on an IO.
+ *
+ * Callbacks are identified by an ID rather than a function pointer. There are
+ * two main reasons:
+
+ * 1) Memory within PgAioHandle is precious, due to the number of PgAioHandle
+ * structs in pre-allocated shared memory.
+
+ * 2) Due to EXEC_BACKEND function pointers are not necessarily stable between
+ * different backends, therefore function pointers cannot directly be in
+ * shared memory.
+ *
+ * Without 2), we could fairly easily allow to add new callbacks, by filling a
+ * ID->pointer mapping table on demand. In the presence of 2 that's still
+ * doable, but harder, because every process has to re-register the pointers
+ * so that a local ID->"backend local pointer" mapping can be maintained.
+ */
+typedef enum PgAioHandleSharedCallbackID
+{
+ ASC_PLACEHOLDER /* empty enums are invalid */ ,
+} PgAioHandleSharedCallbackID;
+
+
+/*
+ * Data necessary for basic IO types (PgAioOp).
+ *
+ * NB: Note that the FDs in here may *not* be relied upon for re-issuing
+ * requests (e.g. for partial reads/writes) - the FD might be from another
+ * process, or closed since. That's not a problem for IOs waiting to be issued
+ * only because the queue is flushed when closing an FD.
+ */
+typedef union
+{
+ struct
+ {
+ int fd;
+ uint16 iov_length;
+ uint64 offset;
+ } read;
+
+ struct
+ {
+ int fd;
+ uint16 iov_length;
+ uint64 offset;
+ } write;
+
+ struct
+ {
+ int fd;
+ bool datasync;
+ } fsync;
+
+ struct
+ {
+ int fd;
+ uint32 nbytes;
+ uint64 offset;
+ } flush_range;
+} PgAioOpData;
+
+
+/* XXX: Perhaps it's worth moving this to a dedicated file? */
+#include "storage/block.h"
+#include "storage/relfilelocator.h"
+
+typedef union PgAioSubjectData
+{
+ /* just as an example placeholder for later */
+ struct
+ {
+ uint32 queue_id;
+ } wal;
+} PgAioSubjectData;
+
+
+
+typedef enum PgAioResultStatus
+{
+ ARS_UNKNOWN,
+ ARS_OK,
+ ARS_PARTIAL,
+ ARS_ERROR,
+} PgAioResultStatus;
+
+typedef struct PgAioResult
+{
+ PgAioHandleSharedCallbackID id:8;
+ PgAioResultStatus status:2;
+ uint32 error_data:22;
+ int32 result;
+} PgAioResult;
+
+typedef struct PgAioReturn
+{
+ PgAioResult result;
+ PgAioSubjectData subject_data;
+} PgAioReturn;
+
+
+typedef struct PgAioSubjectInfo
+{
+ void (*reopen) (PgAioHandle *ioh);
+
+#ifdef NOT_YET
+ char *(*describe_identity) (PgAioHandle *ioh);
+#endif
+
+ const char *name;
+} PgAioSubjectInfo;
+
+
+typedef PgAioResult (*PgAioHandleSharedCallbackComplete) (PgAioHandle *ioh, PgAioResult prior_result);
+typedef void (*PgAioHandleSharedCallbackPrepare) (PgAioHandle *ioh);
+typedef void (*PgAioHandleSharedCallbackError) (PgAioResult result, const PgAioSubjectData *subject_data, int elevel);
+
+typedef struct PgAioHandleSharedCallbacks
+{
+ PgAioHandleSharedCallbackPrepare prepare;
+ PgAioHandleSharedCallbackComplete complete;
+ PgAioHandleSharedCallbackError error;
+} PgAioHandleSharedCallbacks;
+
+
+
+typedef struct PgAioBounceBuffer PgAioBounceBuffer;
+
+
+/*
+ * How many callbacks can be registered for one IO handle. Currently we only
+ * need two, but it's not hard to imagine needing a few more.
+ */
+#define AIO_MAX_SHARED_CALLBACKS 4
+
+
+
+/* AIO API */
+
+
+/* --------------------------------------------------------------------------------
+ * IO Handles
+ * --------------------------------------------------------------------------------
+ */
+
+struct ResourceOwnerData;
+extern PgAioHandle *pgaio_io_get(struct ResourceOwnerData *resowner, PgAioReturn *ret);
+extern PgAioHandle *pgaio_io_get_nb(struct ResourceOwnerData *resowner, PgAioReturn *ret);
+
+extern void pgaio_io_release(PgAioHandle *ioh);
+extern void pgaio_io_release_resowner(dlist_node *ioh_node, bool on_error);
+
+extern void pgaio_io_get_ref(PgAioHandle *ioh, PgAioHandleRef *ior);
+
+extern void pgaio_io_set_subject(PgAioHandle *ioh, PgAioSubjectID subjid);
+extern void pgaio_io_set_flag(PgAioHandle *ioh, PgAioHandleFlags flag);
+
+extern void pgaio_io_add_shared_cb(PgAioHandle *ioh, PgAioHandleSharedCallbackID cbid);
+
+extern void pgaio_io_set_io_data_32(PgAioHandle *ioh, uint32 *data, uint8 len);
+extern void pgaio_io_set_io_data_64(PgAioHandle *ioh, uint64 *data, uint8 len);
+extern uint64 *pgaio_io_get_io_data(PgAioHandle *ioh, uint8 *len);
+
+extern void pgaio_io_prepare(PgAioHandle *ioh, PgAioOp op);
+
+extern int pgaio_io_get_id(PgAioHandle *ioh);
+struct iovec;
+extern int pgaio_io_get_iovec(PgAioHandle *ioh, struct iovec **iov);
+extern bool pgaio_io_has_subject(PgAioHandle *ioh);
+
+extern PgAioSubjectData *pgaio_io_get_subject_data(PgAioHandle *ioh);
+extern PgAioOpData *pgaio_io_get_op_data(PgAioHandle *ioh);
+extern ProcNumber pgaio_io_get_owner(PgAioHandle *ioh);
+
+
+
+/* --------------------------------------------------------------------------------
+ * IO References
+ * --------------------------------------------------------------------------------
+ */
+
+extern void pgaio_io_ref_clear(PgAioHandleRef *ior);
+extern bool pgaio_io_ref_valid(PgAioHandleRef *ior);
+extern int pgaio_io_ref_get_id(PgAioHandleRef *ior);
+
+
+extern void pgaio_io_ref_wait(PgAioHandleRef *ior);
+extern bool pgaio_io_ref_check_done(PgAioHandleRef *ior);
+
+
+
+/* --------------------------------------------------------------------------------
+ * IO Result
+ * --------------------------------------------------------------------------------
+ */
+
+extern void pgaio_result_log(PgAioResult result, const PgAioSubjectData *subject_data,
+ int elevel);
+
+
+
+/* --------------------------------------------------------------------------------
+ * Bounce Buffers
+ * --------------------------------------------------------------------------------
+ */
+
+extern PgAioBounceBuffer *pgaio_bounce_buffer_get(void);
+extern void pgaio_io_assoc_bounce_buffer(PgAioHandle *ioh, PgAioBounceBuffer *bb);
+extern uint32 pgaio_bounce_buffer_id(PgAioBounceBuffer *bb);
+extern void pgaio_bounce_buffer_release(PgAioBounceBuffer *bb);
+extern char *pgaio_bounce_buffer_buffer(PgAioBounceBuffer *bb);
+extern void pgaio_bounce_buffer_release_resowner(dlist_node *bb_node, bool on_error);
+
+
+
+/* --------------------------------------------------------------------------------
+ * Actions on multiple IOs.
+ * --------------------------------------------------------------------------------
+ */
+
+extern void pgaio_submit_staged(void);
+extern bool pgaio_have_staged(void);
+
+
+
+/* --------------------------------------------------------------------------------
+ * Low level IO preparation routines
+ *
+ * These will often be called by code lowest level of initiating an
+ * IO. E.g. bufmgr.c may initiate IO for a buffer, but pgaio_io_prep_readv()
+ * will be called from within fd.c.
+ *
+ * Implemented in aio_io.c
+ * --------------------------------------------------------------------------------
+ */
+
+extern void pgaio_io_prep_readv(PgAioHandle *ioh,
+ int fd, int iovcnt, uint64 offset);
+
+extern void pgaio_io_prep_writev(PgAioHandle *ioh,
+ int fd, int iovcnt, uint64 offset);
+
+
+
+/* --------------------------------------------------------------------------------
+ * Other
+ * --------------------------------------------------------------------------------
+ */
+
+extern void pgaio_closing_fd(int fd);
+extern void pgaio_at_xact_end(bool is_subxact, bool is_commit);
+extern void pgaio_at_error(void);
+
+
/* GUC related */
extern void assign_io_method(int newval, void *extra);
@@ -36,6 +342,8 @@ typedef enum IoMethod
/* GUCs */
extern const struct config_enum_entry io_method_options[];
extern int io_method;
+extern int io_max_concurrency;
+extern int io_bounce_buffers;
#endif /* AIO_H */
diff --git a/src/include/storage/aio_internal.h b/src/include/storage/aio_internal.h
new file mode 100644
index 00000000000..82bce1cf27c
--- /dev/null
+++ b/src/include/storage/aio_internal.h
@@ -0,0 +1,274 @@
+/*-------------------------------------------------------------------------
+ *
+ * aio_internal.h
+ * aio_internal
+ *
+ *
+ * Portions Copyright (c) 1996-2024, PostgreSQL Global Development Group
+ * Portions Copyright (c) 1994, Regents of the University of California
+ *
+ * src/include/storage/aio_internal.h
+ *
+ *-------------------------------------------------------------------------
+ */
+#ifndef AIO_INTERNAL_H
+#define AIO_INTERNAL_H
+
+
+#include "lib/ilist.h"
+#include "port/pg_iovec.h"
+#include "storage/aio.h"
+#include "storage/condition_variable.h"
+
+
+#define PGAIO_VERBOSE
+
+
+/* AFIXME */
+#define PGAIO_SUBMIT_BATCH_SIZE 32
+
+
+
+typedef enum PgAioHandleState
+{
+ /* not in use */
+ AHS_IDLE = 0,
+
+ /* returned by pgaio_io_get() */
+ AHS_HANDED_OUT,
+
+ /* pgaio_io_start_*() has been called, but IO hasn't been submitted yet */
+ AHS_DEFINED,
+
+ /* subjects prepare() callback has been called */
+ AHS_PREPARED,
+
+ /* IO is being executed */
+ AHS_IN_FLIGHT,
+
+ /* IO finished, but result has not yet been processed */
+ AHS_REAPED,
+
+ /* IO completed, shared completion has been called */
+ AHS_COMPLETED_SHARED,
+
+ /* IO completed, local completion has been called */
+ AHS_COMPLETED_LOCAL,
+} PgAioHandleState;
+
+
+struct ResourceOwnerData;
+
+/* typedef is in public header */
+struct PgAioHandle
+{
+ PgAioHandleState state:8;
+
+ /* what are we operating on */
+ PgAioSubjectID subject:8;
+
+ /* which operation */
+ PgAioOp op:8;
+
+ /* bitfield of PgAioHandleFlags */
+ uint8 flags;
+
+ uint8 num_shared_callbacks;
+
+ /* using the proper type here would use more space */
+ uint8 shared_callbacks[AIO_MAX_SHARED_CALLBACKS];
+
+ uint8 iovec_data_len;
+
+ /* XXX: could be optimized out with some pointer math */
+ int32 owner_procno;
+
+ /* FIXME: remove in favor of distilled_result */
+ /* raw result of the IO operation */
+ int32 result;
+
+ /* index into PgAioCtl->iovecs */
+ uint32 iovec_off;
+
+ /*
+ * List of bounce_buffers owned by IO. It would suffice to use an index
+ * based linked list here.
+ */
+ slist_head bounce_buffers;
+
+ /**
+ * In which list the handle is registered, depends on the state:
+ * - IDLE, in per-backend list
+ * - HANDED_OUT - not in a list
+ * - DEFINED - in per-backend staged list
+ * - PREPARED - in per-backend staged list
+ * - IN_FLIGHT - not in any list
+ * - REAPED - in per-reap context list
+ * - COMPLETED_SHARED - not in any list
+ * - COMPLETED_LOCAL - not in any list
+ *
+ * XXX: It probably make sense to optimize this out to save on per-io
+ * memory at the cost of per-backend memory.
+ **/
+ dlist_node node;
+
+ struct ResourceOwnerData *resowner;
+ dlist_node resowner_node;
+
+ /* incremented every time the IO handle is reused */
+ uint64 generation;
+
+ ConditionVariable cv;
+
+ /* result of shared callback, passed to issuer callback */
+ PgAioResult distilled_result;
+
+ PgAioReturn *report_return;
+
+ PgAioOpData op_data;
+
+ /*
+ * Data necessary for shared completions. Needs to be sufficient to allow
+ * another backend to retry an IO.
+ */
+ PgAioSubjectData scb_data;
+};
+
+
+struct PgAioBounceBuffer
+{
+ slist_node node;
+ struct ResourceOwnerData *resowner;
+ dlist_node resowner_node;
+ char *buffer;
+};
+
+
+typedef struct PgAioPerBackend
+{
+ /* index into PgAioCtl->io_handles */
+ uint32 io_handle_off;
+
+ /* index into PgAioCtl->bounce_buffers */
+ uint32 bounce_buffers_off;
+
+ /* IO Handles that currently are not used */
+ dclist_head idle_ios;
+
+ /*
+ * Only one IO may be returned by pgaio_io_get()/pgaio_io_get() without
+ * having been either defined (by actually associating it with IO) or by
+ * released (with pgaio_io_release()). This restriction is necessary to
+ * guarantee that we always can acquire an IO. ->handed_out_io is used to
+ * enforce that rule.
+ */
+ PgAioHandle *handed_out_io;
+
+ /*
+ * IOs that are defined, but not yet submitted.
+ */
+ uint16 num_staged_ios;
+ PgAioHandle *staged_ios[PGAIO_SUBMIT_BATCH_SIZE];
+
+ /* Bounce Buffers that currently are not used */
+ slist_head idle_bbs;
+
+ /* see handed_out_io */
+ PgAioBounceBuffer *handed_out_bb;
+} PgAioPerBackend;
+
+
+typedef struct PgAioCtl
+{
+ int backend_state_count;
+ PgAioPerBackend *backend_state;
+
+ /*
+ * Array of iovec structs. Each iovec is owned by a specific backend. The
+ * allocation is in PgAioCtl to allow the maximum number of iovecs for
+ * individual IOs to be configurable with PGC_POSTMASTER GUC.
+ */
+ uint64 iovec_count;
+ struct iovec *iovecs;
+
+ /*
+ * For, e.g., an IO covering multiple buffers in shared / temp buffers, we
+ * need to get Buffer IDs during completion to be able to change the
+ * BufferDesc state accordingly. This space can be used to store e.g.
+ * Buffer IDs. Note that the actual iovec might be shorter than this,
+ * because we combine neighboring pages into one larger iovec entry.
+ */
+ uint64 *iovecs_data;
+
+ /*
+ * To perform AIO on buffers that are not located in shared memory (either
+ * because they are not in shared memory or because we need to operate on
+ * a copy, as e.g. the case for writes when checksums are in use)
+ */
+ uint64 bounce_buffers_count;
+ PgAioBounceBuffer *bounce_buffers;
+ char *bounce_buffers_data;
+
+ uint64 io_handle_count;
+ PgAioHandle *io_handles;
+} PgAioCtl;
+
+
+
+/*
+ * The set of callbacks that each IO method must implement.
+ */
+typedef struct IoMethodOps
+{
+ /* initialization */
+ size_t (*shmem_size) (void);
+ void (*shmem_init) (bool first_time);
+
+ void (*postmaster_init) (void);
+ void (*postmaster_child_init_local) (void);
+ void (*postmaster_child_init) (void);
+
+ /* teardown */
+ void (*postmaster_before_child_exit) (void);
+
+ /* handling of IOs */
+ bool (*needs_synchronous_execution)(PgAioHandle *ioh);
+ int (*submit) (uint16 num_staged_ios, PgAioHandle **staged_ios);
+
+ void (*wait_one) (PgAioHandle *ioh,
+ uint64 ref_generation);
+
+ /* properties */
+ bool can_scatter_gather_direct;
+ bool can_scatter_gather_buffered;
+} IoMethodOps;
+
+
+extern bool pgaio_io_was_recycled(PgAioHandle *ioh, uint64 ref_generation, PgAioHandleState *state);
+
+extern void pgaio_io_prepare_subject(PgAioHandle *ioh);
+extern void pgaio_io_process_completion_subject(PgAioHandle *ioh);
+extern void pgaio_io_process_completion(PgAioHandle *ioh, int result);
+extern void pgaio_io_prepare_submit(PgAioHandle *ioh);
+
+extern bool pgaio_io_needs_synchronous_execution(PgAioHandle *ioh);
+extern void pgaio_io_perform_synchronously(PgAioHandle *ioh);
+
+extern bool pgaio_io_can_reopen(PgAioHandle *ioh);
+extern void pgaio_io_reopen(PgAioHandle *ioh);
+
+extern const char *pgaio_io_get_subject_name(PgAioHandle *ioh);
+extern const char *pgaio_io_get_op_name(PgAioHandle *ioh);
+extern const char *pgaio_io_get_state_name(PgAioHandle *ioh);
+
+
+/* Declarations for the tables of function pointers exposed by each IO method. */
+extern const IoMethodOps pgaio_sync_ops;
+
+extern const IoMethodOps *pgaio_impl;
+extern PgAioCtl *aio_ctl;
+extern PgAioPerBackend *my_aio;
+
+
+
+#endif /* AIO_INTERNAL_H */
diff --git a/src/include/storage/aio_ref.h b/src/include/storage/aio_ref.h
new file mode 100644
index 00000000000..ad7e9ad34f3
--- /dev/null
+++ b/src/include/storage/aio_ref.h
@@ -0,0 +1,24 @@
+/*-------------------------------------------------------------------------
+ *
+ * aio_ref.h Definition of PgAioHandleRef, which sometimes needs to be used in
+ * headers.
+ *
+ *
+ * Portions Copyright (c) 1996-2024, PostgreSQL Global Development Group
+ * Portions Copyright (c) 1994, Regents of the University of California
+ *
+ * src/include/storage/aio_ref.h
+ *
+ *-------------------------------------------------------------------------
+ */
+#ifndef AIO_REF_H
+#define AIO_REF_H
+
+typedef struct PgAioHandleRef
+{
+ uint32 aio_index;
+ uint32 generation_upper;
+ uint32 generation_lower;
+} PgAioHandleRef;
+
+#endif /* AIO_REF_H */
diff --git a/src/include/utils/resowner.h b/src/include/utils/resowner.h
index 4e534bc3e70..0cdd0c13ffb 100644
--- a/src/include/utils/resowner.h
+++ b/src/include/utils/resowner.h
@@ -164,4 +164,11 @@ struct LOCALLOCK;
extern void ResourceOwnerRememberLock(ResourceOwner owner, struct LOCALLOCK *locallock);
extern void ResourceOwnerForgetLock(ResourceOwner owner, struct LOCALLOCK *locallock);
+/* special support for AIO */
+struct dlist_node;
+extern void ResourceOwnerRememberAioHandle(ResourceOwner owner, struct dlist_node *ioh_node);
+extern void ResourceOwnerForgetAioHandle(ResourceOwner owner, struct dlist_node *ioh_node);
+extern void ResourceOwnerRememberAioBounceBuffer(ResourceOwner owner, struct dlist_node *bb_node);
+extern void ResourceOwnerForgetAioBounceBuffer(ResourceOwner owner, struct dlist_node *bb_node);
+
#endif /* RESOWNER_H */
diff --git a/src/backend/access/transam/xact.c b/src/backend/access/transam/xact.c
index 87700c7c5c7..1fccaa3eb79 100644
--- a/src/backend/access/transam/xact.c
+++ b/src/backend/access/transam/xact.c
@@ -52,6 +52,7 @@
#include "replication/origin.h"
#include "replication/snapbuild.h"
#include "replication/syncrep.h"
+#include "storage/aio.h"
#include "storage/condition_variable.h"
#include "storage/fd.h"
#include "storage/lmgr.h"
@@ -2462,6 +2463,8 @@ CommitTransaction(void)
AtEOXact_LogicalRepWorkers(true);
pgstat_report_xact_timestamp(0);
+ pgaio_at_xact_end( /* is_subxact = */ false, /* is_commit = */ true);
+
ResourceOwnerDelete(TopTransactionResourceOwner);
s->curTransactionOwner = NULL;
CurTransactionResourceOwner = NULL;
@@ -2976,6 +2979,8 @@ AbortTransaction(void)
pgstat_report_xact_timestamp(0);
}
+ pgaio_at_xact_end( /* is_subxact = */ false, /* is_commit = */ false);
+
/*
* State remains TRANS_ABORT until CleanupTransaction().
*/
@@ -5185,6 +5190,8 @@ CommitSubTransaction(void)
AtEOSubXact_PgStat(true, s->nestingLevel);
AtSubCommit_Snapshot(s->nestingLevel);
+ pgaio_at_xact_end( /* is_subxact = */ true, /* is_commit = */ true);
+
/*
* We need to restore the upper transaction's read-only state, in case the
* upper is read-write while the child is read-only; GUC will incorrectly
@@ -5350,6 +5357,8 @@ AbortSubTransaction(void)
AtSubAbort_Snapshot(s->nestingLevel);
}
+ pgaio_at_xact_end( /* is_subxact = */ true, /* is_commit = */ false);
+
/*
* Restore the upper transaction's read-only state, too. This should be
* redundant with GUC's cleanup but we may as well do it for consistency
diff --git a/src/backend/storage/aio/Makefile b/src/backend/storage/aio/Makefile
index eaeaeeee8e3..b253278f3c1 100644
--- a/src/backend/storage/aio/Makefile
+++ b/src/backend/storage/aio/Makefile
@@ -11,6 +11,9 @@ include $(top_builddir)/src/Makefile.global
OBJS = \
aio.o \
aio_init.o \
+ aio_io.o \
+ aio_subject.o \
+ method_sync.o \
read_stream.o
include $(top_srcdir)/src/backend/common.mk
diff --git a/src/backend/storage/aio/aio.c b/src/backend/storage/aio/aio.c
index d831c772960..b5370330620 100644
--- a/src/backend/storage/aio/aio.c
+++ b/src/backend/storage/aio/aio.c
@@ -14,7 +14,23 @@
#include "postgres.h"
+#include "miscadmin.h"
+#include "port/atomics.h"
#include "storage/aio.h"
+#include "storage/aio_internal.h"
+#include "storage/bufmgr.h"
+#include "utils/resowner.h"
+#include "utils/wait_event_types.h"
+
+
+
+static void pgaio_io_reclaim(PgAioHandle *ioh);
+static void pgaio_io_resowner_register(PgAioHandle *ioh);
+static void pgaio_io_wait_for_free(void);
+static PgAioHandle *pgaio_io_from_ref(PgAioHandleRef *ior, uint64 *ref_generation);
+
+static void pgaio_bounce_buffer_wait_for_free(void);
+
/* Options for io_method. */
@@ -24,9 +40,968 @@ const struct config_enum_entry io_method_options[] = {
};
int io_method = DEFAULT_IO_METHOD;
+int io_max_concurrency = -1;
+int io_bounce_buffers = -1;
+
+
+/* global control for AIO */
+PgAioCtl *aio_ctl;
+
+/* current backend's per-backend state */
+PgAioPerBackend *my_aio;
+
+
+static const IoMethodOps *pgaio_ops_table[] = {
+ [IOMETHOD_SYNC] = &pgaio_sync_ops,
+};
+
+
+const IoMethodOps *pgaio_impl;
+
+
+
+/* --------------------------------------------------------------------------------
+ * "Core" IO Api
+ * --------------------------------------------------------------------------------
+ */
+
+/*
+ * AFIXME: rewrite
+ *
+ * Shared completion callbacks can be executed by any backend (otherwise there
+ * would be deadlocks). Therefore they cannot update state for the issuer of
+ * the IO. That can be done with issuer callbacks.
+ *
+ * Note that issuer callbacks are effectively executed in a critical
+ * section. This is necessary as we need to be able to execute IO in critical
+ * sections (consider e.g. WAL logging) and to be able to execute IOs we need
+ * to acquire an IO, which in turn requires executing issuer callbacks. An
+ * alternative scheme could be to defer local callback execution until a later
+ * point, but that gets complicated quickly.
+ *
+ * Therefore the typical pattern is to use an issuer callback to set some
+ * flags in backend local memory, which can then be used to error out at a
+ * later time.
+ *
+ * NB: The issuer callback is cleared when the resowner owning the IO goes out
+ * of scope.
+ */
+PgAioHandle *
+pgaio_io_get(struct ResourceOwnerData *resowner, PgAioReturn *ret)
+{
+ PgAioHandle *h;
+
+ while (true)
+ {
+ h = pgaio_io_get_nb(resowner, ret);
+
+ if (h != NULL)
+ return h;
+
+ /*
+ * Evidently all handles by this backend are in use. Just wait for
+ * some to complete.
+ */
+ pgaio_io_wait_for_free();
+ }
+}
+
+PgAioHandle *
+pgaio_io_get_nb(struct ResourceOwnerData *resowner, PgAioReturn *ret)
+{
+ if (my_aio->num_staged_ios >= PGAIO_SUBMIT_BATCH_SIZE)
+ {
+ Assert(my_aio->num_staged_ios == PGAIO_SUBMIT_BATCH_SIZE);
+ pgaio_submit_staged();
+ }
+
+ if (my_aio->handed_out_io)
+ {
+ ereport(ERROR,
+ errmsg("API violation: Only one IO can be handed out"));
+ }
+
+ if (!dclist_is_empty(&my_aio->idle_ios))
+ {
+ dlist_node *ion = dclist_pop_head_node(&my_aio->idle_ios);
+ PgAioHandle *ioh = dclist_container(PgAioHandle, node, ion);
+
+ Assert(ioh->state == AHS_IDLE);
+ Assert(ioh->owner_procno == MyProcNumber);
+
+ ioh->state = AHS_HANDED_OUT;
+ my_aio->handed_out_io = ioh;
+
+ if (resowner)
+ pgaio_io_resowner_register(ioh);
+
+ if (ret)
+ ioh->report_return = ret;
+
+ return ioh;
+ }
+
+ return NULL;
+}
+
+void
+pgaio_io_release(PgAioHandle *ioh)
+{
+ if (ioh == my_aio->handed_out_io)
+ {
+ Assert(ioh->state == AHS_HANDED_OUT);
+ Assert(ioh->resowner);
+
+ my_aio->handed_out_io = NULL;
+ pgaio_io_reclaim(ioh);
+ }
+ else
+ {
+ elog(ERROR, "release in unexpected state");
+ }
+}
+
+void
+pgaio_io_release_resowner(dlist_node *ioh_node, bool on_error)
+{
+ PgAioHandle *ioh = dlist_container(PgAioHandle, resowner_node, ioh_node);
+
+ Assert(ioh->resowner);
+
+ ResourceOwnerForgetAioHandle(ioh->resowner, &ioh->resowner_node);
+ ioh->resowner = NULL;
+
+ switch (ioh->state)
+ {
+ case AHS_IDLE:
+ elog(ERROR, "unexpected");
+ break;
+ case AHS_HANDED_OUT:
+ Assert(ioh == my_aio->handed_out_io || my_aio->handed_out_io == NULL);
+
+ if (ioh == my_aio->handed_out_io)
+ {
+ my_aio->handed_out_io = NULL;
+ if (!on_error)
+ elog(WARNING, "leaked AIO handle");
+ }
+
+ pgaio_io_reclaim(ioh);
+ break;
+ case AHS_DEFINED:
+ case AHS_PREPARED:
+ /* XXX: Should we warn about this when is_commit? */
+ pgaio_submit_staged();
+ break;
+ case AHS_IN_FLIGHT:
+ case AHS_REAPED:
+ case AHS_COMPLETED_SHARED:
+ /* this is expected to happen */
+ break;
+ case AHS_COMPLETED_LOCAL:
+ /* XXX: unclear if this ought to be possible? */
+ pgaio_io_reclaim(ioh);
+ break;
+ }
+
+ /*
+ * Need to unregister the reporting of the IO's result, the memory it's
+ * referencing likely has gone away.
+ */
+ if (ioh->report_return)
+ ioh->report_return = NULL;
+}
+
+int
+pgaio_io_get_iovec(PgAioHandle *ioh, struct iovec **iov)
+{
+ Assert(ioh->state == AHS_HANDED_OUT);
+
+ *iov = &aio_ctl->iovecs[ioh->iovec_off];
+
+ /* AFIXME: Needs to be the value at startup time */
+ return io_combine_limit;
+}
+
+PgAioSubjectData *
+pgaio_io_get_subject_data(PgAioHandle *ioh)
+{
+ return &ioh->scb_data;
+}
+
+PgAioOpData *
+pgaio_io_get_op_data(PgAioHandle *ioh)
+{
+ return &ioh->op_data;
+}
+
+ProcNumber
+pgaio_io_get_owner(PgAioHandle *ioh)
+{
+ return ioh->owner_procno;
+}
+
+bool
+pgaio_io_has_subject(PgAioHandle *ioh)
+{
+ return ioh->subject != ASI_INVALID;
+}
+
+void
+pgaio_io_set_flag(PgAioHandle *ioh, PgAioHandleFlags flag)
+{
+ Assert(ioh->state == AHS_HANDED_OUT);
+
+ ioh->flags |= flag;
+}
+
+void
+pgaio_io_set_io_data_32(PgAioHandle *ioh, uint32 *data, uint8 len)
+{
+ Assert(ioh->state == AHS_HANDED_OUT);
+
+ for (int i = 0; i < len; i++)
+ aio_ctl->iovecs_data[ioh->iovec_off + i] = data[i];
+ ioh->iovec_data_len = len;
+}
+
+uint64 *
+pgaio_io_get_io_data(PgAioHandle *ioh, uint8 *len)
+{
+ Assert(ioh->iovec_data_len > 0);
+
+ *len = ioh->iovec_data_len;
+
+ return &aio_ctl->iovecs_data[ioh->iovec_off];
+}
+
+void
+pgaio_io_set_subject(PgAioHandle *ioh, PgAioSubjectID subjid)
+{
+ Assert(ioh->state == AHS_HANDED_OUT);
+
+ ioh->subject = subjid;
+
+ elog(DEBUG3, "io:%d, op %s, subject %s, set subject",
+ pgaio_io_get_id(ioh),
+ pgaio_io_get_op_name(ioh),
+ pgaio_io_get_subject_name(ioh));
+}
+
+void
+pgaio_io_get_ref(PgAioHandle *ioh, PgAioHandleRef *ior)
+{
+ Assert(ioh->state == AHS_HANDED_OUT ||
+ ioh->state == AHS_DEFINED ||
+ ioh->state == AHS_PREPARED);
+ Assert(ioh->generation != 0);
+
+ ior->aio_index = ioh - aio_ctl->io_handles;
+ ior->generation_upper = (uint32) (ioh->generation >> 32);
+ ior->generation_lower = (uint32) ioh->generation;
+}
+
+void
+pgaio_io_ref_clear(PgAioHandleRef *ior)
+{
+ ior->aio_index = PG_UINT32_MAX;
+}
+
+bool
+pgaio_io_ref_valid(PgAioHandleRef *ior)
+{
+ return ior->aio_index != PG_UINT32_MAX;
+}
+
+int
+pgaio_io_ref_get_id(PgAioHandleRef *ior)
+{
+ Assert(pgaio_io_ref_valid(ior));
+ return ior->aio_index;
+}
+
+bool
+pgaio_io_was_recycled(PgAioHandle *ioh, uint64 ref_generation, PgAioHandleState *state)
+{
+ *state = ioh->state;
+ pg_read_barrier();
+
+ return ioh->generation != ref_generation;
+}
+
+void
+pgaio_io_ref_wait(PgAioHandleRef *ior)
+{
+ uint64 ref_generation;
+ PgAioHandleState state;
+ bool am_owner;
+ PgAioHandle *ioh;
+
+ ioh = pgaio_io_from_ref(ior, &ref_generation);
+
+ am_owner = ioh->owner_procno == MyProcNumber;
+
+
+ if (pgaio_io_was_recycled(ioh, ref_generation, &state))
+ return;
+
+ if (am_owner)
+ {
+ if (state == AHS_DEFINED || state == AHS_PREPARED)
+ {
+ /* XXX: Arguably this should be prevented by callers? */
+ pgaio_submit_staged();
+ }
+ else if (state != AHS_IN_FLIGHT && state != AHS_REAPED && state != AHS_COMPLETED_SHARED && state != AHS_COMPLETED_LOCAL)
+ {
+ elog(PANIC, "waiting for own IO in wrong state: %d",
+ state);
+ }
+
+ /*
+ * Somebody else completed the IO, need to execute issuer callback, so
+ * reclaim eagerly.
+ */
+ if (state == AHS_COMPLETED_LOCAL)
+ {
+ pgaio_io_reclaim(ioh);
+
+ return;
+ }
+ }
+
+ while (true)
+ {
+ if (pgaio_io_was_recycled(ioh, ref_generation, &state))
+ return;
+
+ switch (state)
+ {
+ case AHS_IDLE:
+ case AHS_HANDED_OUT:
+ elog(ERROR, "IO in wrong state: %d", state);
+ break;
+
+ case AHS_IN_FLIGHT:
+ if (pgaio_impl->wait_one)
+ {
+ pgaio_impl->wait_one(ioh, ref_generation);
+ continue;
+ }
+ /* fallthrough */
+
+ /* waiting for owner to submit */
+ case AHS_PREPARED:
+ case AHS_DEFINED:
+ /* waiting for reaper to complete */
+ /* fallthrough */
+ case AHS_REAPED:
+ /* shouldn't be able to hit this otherwise */
+ Assert(IsUnderPostmaster);
+ /* ensure we're going to get woken up */
+ ConditionVariablePrepareToSleep(&ioh->cv);
+
+ while (!pgaio_io_was_recycled(ioh, ref_generation, &state))
+ {
+ if (state != AHS_REAPED && state != AHS_DEFINED &&
+ state != AHS_IN_FLIGHT)
+ break;
+ ConditionVariableSleep(&ioh->cv, WAIT_EVENT_AIO_COMPLETION);
+ }
+
+ ConditionVariableCancelSleep();
+ break;
+
+ case AHS_COMPLETED_SHARED:
+ /* see above */
+ if (am_owner)
+ pgaio_io_reclaim(ioh);
+ return;
+ case AHS_COMPLETED_LOCAL:
+ return;
+ }
+ }
+}
+
+bool
+pgaio_io_ref_check_done(PgAioHandleRef *ior)
+{
+ uint64 ref_generation;
+ PgAioHandleState state;
+ bool am_owner;
+ PgAioHandle *ioh;
+
+ ioh = pgaio_io_from_ref(ior, &ref_generation);
+
+ if (pgaio_io_was_recycled(ioh, ref_generation, &state))
+ return true;
+
+
+ if (state == AHS_IDLE)
+ return true;
+
+ am_owner = ioh->owner_procno == MyProcNumber;
+
+ if (state == AHS_COMPLETED_SHARED || state == AHS_COMPLETED_LOCAL)
+ {
+ if (am_owner)
+ pgaio_io_reclaim(ioh);
+ return true;
+ }
+
+ return false;
+}
+
+int
+pgaio_io_get_id(PgAioHandle *ioh)
+{
+ Assert(ioh >= aio_ctl->io_handles &&
+ ioh <= (aio_ctl->io_handles + aio_ctl->io_handle_count));
+ return ioh - aio_ctl->io_handles;
+}
+
+const char *
+pgaio_io_get_state_name(PgAioHandle *ioh)
+{
+ switch (ioh->state)
+ {
+ case AHS_IDLE:
+ return "idle";
+ case AHS_HANDED_OUT:
+ return "handed_out";
+ case AHS_DEFINED:
+ return "DEFINED";
+ case AHS_PREPARED:
+ return "PREPARED";
+ case AHS_IN_FLIGHT:
+ return "IN_FLIGHT";
+ case AHS_REAPED:
+ return "REAPED";
+ case AHS_COMPLETED_SHARED:
+ return "COMPLETED_SHARED";
+ case AHS_COMPLETED_LOCAL:
+ return "COMPLETED_LOCAL";
+ }
+ pg_unreachable();
+}
+
+/*
+ * Internal, should only be called from pgaio_io_prep_*().
+ */
+void
+pgaio_io_prepare(PgAioHandle *ioh, PgAioOp op)
+{
+ Assert(ioh->state == AHS_HANDED_OUT);
+ Assert(pgaio_io_has_subject(ioh));
+
+ ioh->op = op;
+ ioh->state = AHS_DEFINED;
+ ioh->result = 0;
+
+ /* allow a new IO to be staged */
+ my_aio->handed_out_io = NULL;
+
+ pgaio_io_prepare_subject(ioh);
+
+ ioh->state = AHS_PREPARED;
+
+ elog(DEBUG3, "io:%d: prepared %s",
+ pgaio_io_get_id(ioh), pgaio_io_get_op_name(ioh));
+
+ if (!pgaio_io_needs_synchronous_execution(ioh))
+ {
+ my_aio->staged_ios[my_aio->num_staged_ios++] = ioh;
+ Assert(my_aio->num_staged_ios <= PGAIO_SUBMIT_BATCH_SIZE);
+ }
+ else
+ {
+ pgaio_io_prepare_submit(ioh);
+ pgaio_io_perform_synchronously(ioh);
+ }
+}
+
+/*
+ * Handle IO getting completed by a method.
+ */
+void
+pgaio_io_process_completion(PgAioHandle *ioh, int result)
+{
+ Assert(ioh->state == AHS_IN_FLIGHT);
+
+ ioh->result = result;
+
+ pg_write_barrier();
+
+ /* FIXME: should be done in separate function */
+ ioh->state = AHS_REAPED;
+
+ pgaio_io_process_completion_subject(ioh);
+
+ /* ensure results of completion are visible before the new state */
+ pg_write_barrier();
+
+ ioh->state = AHS_COMPLETED_SHARED;
+
+ /* condition variable broadcast ensures state is visible before wakeup */
+ ConditionVariableBroadcast(&ioh->cv);
+
+ if (ioh->owner_procno == MyProcNumber)
+ pgaio_io_reclaim(ioh);
+}
+
+bool
+pgaio_io_needs_synchronous_execution(PgAioHandle *ioh)
+{
+ if (pgaio_impl->needs_synchronous_execution)
+ return pgaio_impl->needs_synchronous_execution(ioh);
+ return false;
+}
+
+/*
+ * Handle IO being processed by IO method.
+ */
+void
+pgaio_io_prepare_submit(PgAioHandle *ioh)
+{
+ ioh->state = AHS_IN_FLIGHT;
+ pg_write_barrier();
+}
+
+static PgAioHandle *
+pgaio_io_from_ref(PgAioHandleRef *ior, uint64 *ref_generation)
+{
+ PgAioHandle *ioh;
+
+ Assert(ior->aio_index < aio_ctl->io_handle_count);
+
+ ioh = &aio_ctl->io_handles[ior->aio_index];
+
+ *ref_generation = ((uint64) ior->generation_upper) << 32 |
+ ior->generation_lower;
+
+ Assert(*ref_generation != 0);
+
+ return ioh;
+}
+
+static void
+pgaio_io_resowner_register(PgAioHandle *ioh)
+{
+ Assert(!ioh->resowner);
+ Assert(CurrentResourceOwner);
+
+ ResourceOwnerRememberAioHandle(CurrentResourceOwner, &ioh->resowner_node);
+ ioh->resowner = CurrentResourceOwner;
+}
+
+static void
+pgaio_io_reclaim(PgAioHandle *ioh)
+{
+ /* This is only ok if it's our IO */
+ Assert(ioh->owner_procno == MyProcNumber);
+
+ ereport(DEBUG3,
+ errmsg("reclaiming io:%d, state: %s, op %s, subject %s, result: %d, distilled_result: AFIXME, report to: %p",
+ pgaio_io_get_id(ioh),
+ pgaio_io_get_state_name(ioh),
+ pgaio_io_get_op_name(ioh),
+ pgaio_io_get_subject_name(ioh),
+ ioh->result,
+ ioh->report_return
+ ),
+ errhidestmt(true), errhidecontext(true));
+
+ if (ioh->report_return)
+ {
+ if (ioh->state != AHS_HANDED_OUT)
+ {
+ ioh->report_return->result = ioh->distilled_result;
+ ioh->report_return->subject_data = ioh->scb_data;
+ }
+ }
+
+ /* reclaim all associated bounce buffers */
+ if (!slist_is_empty(&ioh->bounce_buffers))
+ {
+ slist_mutable_iter it;
+
+ slist_foreach_modify(it, &ioh->bounce_buffers)
+ {
+ PgAioBounceBuffer *bb = slist_container(PgAioBounceBuffer, node, it.cur);
+
+ slist_delete_current(&it);
+
+ slist_push_head(&my_aio->idle_bbs, &bb->node);
+ }
+ }
+
+ if (ioh->resowner)
+ {
+ ResourceOwnerForgetAioHandle(ioh->resowner, &ioh->resowner_node);
+ ioh->resowner = NULL;
+ }
+
+ Assert(!ioh->resowner);
+
+ ioh->num_shared_callbacks = 0;
+ ioh->iovec_data_len = 0;
+ ioh->report_return = NULL;
+ ioh->flags = 0;
+
+ pg_write_barrier();
+ ioh->generation++;
+ pg_write_barrier();
+ ioh->state = AHS_IDLE;
+ pg_write_barrier();
+
+ dclist_push_tail(&my_aio->idle_ios, &ioh->node);
+}
+
+static void
+pgaio_io_wait_for_free(void)
+{
+ bool found_handed_out = false;
+ int reclaimed = 0;
+ static uint32 lastpos = 0;
+
+ elog(DEBUG2,
+ "waiting for self: %d pending",
+ my_aio->num_staged_ios);
+
+ /*
+ * First check if any of our IOs actually have completed - when using
+ * worker, that'll often be the case. We could do so as part of the loop
+ * below, but that'd potentially lead us to wait for some IO submitted
+ * before.
+ */
+ for (int i = 0; i < io_max_concurrency; i++)
+ {
+ PgAioHandle *ioh = &aio_ctl->io_handles[my_aio->io_handle_off + i];
+
+ if (ioh->state == AHS_COMPLETED_SHARED)
+ {
+ pgaio_io_reclaim(ioh);
+ reclaimed++;
+ }
+ }
+
+ if (reclaimed > 0)
+ return;
+
+ if (my_aio->num_staged_ios > 0)
+ {
+ elog(DEBUG2, "submitting while acquiring free io");
+ pgaio_submit_staged();
+ }
+
+ for (uint32 i = lastpos; i < lastpos + io_max_concurrency; i++)
+ {
+ uint32 thisoff = my_aio->io_handle_off + (i % io_max_concurrency);
+ PgAioHandle *ioh = &aio_ctl->io_handles[thisoff];
+
+ switch (ioh->state)
+ {
+ case AHS_IDLE:
+
+ /*
+ * While one might think that pgaio_io_get_nb() should have
+ * succeeded, this is reachable because the IO could have
+ * completed during the submission above.
+ */
+ return;
+ case AHS_DEFINED: /* should have been submitted above */
+ case AHS_PREPARED:
+ case AHS_COMPLETED_LOCAL:
+ elog(ERROR, "shouldn't get here with io:%d in state %d",
+ pgaio_io_get_id(ioh), ioh->state);
+ break;
+ case AHS_HANDED_OUT:
+ if (found_handed_out)
+ elog(ERROR, "more than one handed out IO");
+ found_handed_out = true;
+ continue;
+ case AHS_REAPED:
+ case AHS_IN_FLIGHT:
+ {
+ PgAioHandleRef ior;
+
+ ior.aio_index = ioh - aio_ctl->io_handles;
+ ior.generation_upper = (uint32) (ioh->generation >> 32);
+ ior.generation_lower = (uint32) ioh->generation;
+
+ pgaio_io_ref_wait(&ior);
+ elog(DEBUG2, "waited for io:%d",
+ pgaio_io_get_id(ioh));
+ lastpos = i;
+ return;
+ }
+ break;
+ case AHS_COMPLETED_SHARED:
+ /* reclaim */
+ pgaio_io_reclaim(ioh);
+ lastpos = i;
+ return;
+ }
+ }
+
+ elog(PANIC, "could not reclaim any handles");
+}
+
+
+
+/* --------------------------------------------------------------------------------
+ * Bounce Buffers
+ * --------------------------------------------------------------------------------
+ */
+
+PgAioBounceBuffer *
+pgaio_bounce_buffer_get(void)
+{
+ PgAioBounceBuffer *bb = NULL;
+ slist_node *node;
+
+ if (my_aio->handed_out_bb != NULL)
+ elog(ERROR, "can only hand out one BB");
+
+ /*
+ * FIXME It probably is not correct to have bounce buffers be per backend,
+ * they use too much memory.
+ */
+ if (slist_is_empty(&my_aio->idle_bbs))
+ {
+ pgaio_bounce_buffer_wait_for_free();
+ }
+
+ node = slist_pop_head_node(&my_aio->idle_bbs);
+ bb = slist_container(PgAioBounceBuffer, node, node);
+
+ my_aio->handed_out_bb = bb;
+
+ bb->resowner = CurrentResourceOwner;
+ ResourceOwnerRememberAioBounceBuffer(bb->resowner, &bb->resowner_node);
+
+ return bb;
+}
+
+void
+pgaio_io_assoc_bounce_buffer(PgAioHandle *ioh, PgAioBounceBuffer *bb)
+{
+ if (my_aio->handed_out_bb != bb)
+ elog(ERROR, "can only assign handed out BB");
+ my_aio->handed_out_bb = NULL;
+
+ /*
+ * There can be many bounce buffers assigned in case of vectorized IOs.
+ */
+ slist_push_head(&ioh->bounce_buffers, &bb->node);
+
+ /* once associated with an IO, the IO has ownership */
+ ResourceOwnerForgetAioBounceBuffer(bb->resowner, &bb->resowner_node);
+ bb->resowner = NULL;
+}
+
+uint32
+pgaio_bounce_buffer_id(PgAioBounceBuffer *bb)
+{
+ return bb - aio_ctl->bounce_buffers;
+}
+
+void
+pgaio_bounce_buffer_release(PgAioBounceBuffer *bb)
+{
+ if (my_aio->handed_out_bb != bb)
+ elog(ERROR, "can only release handed out BB");
+
+ slist_push_head(&my_aio->idle_bbs, &bb->node);
+ my_aio->handed_out_bb = NULL;
+
+ ResourceOwnerForgetAioBounceBuffer(bb->resowner, &bb->resowner_node);
+ bb->resowner = NULL;
+}
+
+void
+pgaio_bounce_buffer_release_resowner(dlist_node *bb_node, bool on_error)
+{
+ PgAioBounceBuffer *bb = dlist_container(PgAioBounceBuffer, resowner_node, bb_node);
+
+ Assert(bb->resowner);
+
+ if (!on_error)
+ elog(WARNING, "leaked AIO bounce buffer");
+
+ pgaio_bounce_buffer_release(bb);
+}
+
+char *
+pgaio_bounce_buffer_buffer(PgAioBounceBuffer *bb)
+{
+ return bb->buffer;
+}
+
+static void
+pgaio_bounce_buffer_wait_for_free(void)
+{
+ static uint32 lastpos = 0;
+
+ if (my_aio->num_staged_ios > 0)
+ {
+ elog(DEBUG2, "submitting while acquiring free bb");
+ pgaio_submit_staged();
+ }
+
+ for (uint32 i = lastpos; i < lastpos + io_max_concurrency; i++)
+ {
+ uint32 thisoff = my_aio->io_handle_off + (i % io_max_concurrency);
+ PgAioHandle *ioh = &aio_ctl->io_handles[thisoff];
+
+ switch (ioh->state)
+ {
+ case AHS_IDLE:
+ case AHS_HANDED_OUT:
+ continue;
+ case AHS_DEFINED: /* should have been submitted above */
+ case AHS_PREPARED:
+ elog(ERROR, "shouldn't get here with io:%d in state %d",
+ pgaio_io_get_id(ioh), ioh->state);
+ break;
+ case AHS_REAPED:
+ case AHS_IN_FLIGHT:
+ if (!slist_is_empty(&ioh->bounce_buffers))
+ {
+ PgAioHandleRef ior;
+
+ ior.aio_index = ioh - aio_ctl->io_handles;
+ ior.generation_upper = (uint32) (ioh->generation >> 32);
+ ior.generation_lower = (uint32) ioh->generation;
+
+ pgaio_io_ref_wait(&ior);
+ elog(DEBUG2, "waited for io:%d to reclaim BB",
+ pgaio_io_get_id(ioh));
+
+ if (slist_is_empty(&my_aio->idle_bbs))
+ elog(WARNING, "empty after wait");
+
+ if (!slist_is_empty(&my_aio->idle_bbs))
+ {
+ lastpos = i;
+ return;
+ }
+ }
+ break;
+ case AHS_COMPLETED_SHARED:
+ case AHS_COMPLETED_LOCAL:
+ /* reclaim */
+ pgaio_io_reclaim(ioh);
+
+ if (!slist_is_empty(&my_aio->idle_bbs))
+ {
+ lastpos = i;
+ return;
+ }
+ break;
+ }
+ }
+
+ /*
+ * The submission above could have caused the IO to complete at any time.
+ */
+ if (slist_is_empty(&my_aio->idle_bbs))
+ elog(PANIC, "no more bbs");
+}
+
+
+
+/* --------------------------------------------------------------------------------
+ * Actions on multiple IOs.
+ * --------------------------------------------------------------------------------
+ */
+
+void
+pgaio_submit_staged(void)
+{
+ int total_submitted = 0;
+ int did_submit;
+
+ if (my_aio->num_staged_ios == 0)
+ return;
+
+
+ START_CRIT_SECTION();
+
+ did_submit = pgaio_impl->submit(my_aio->num_staged_ios, my_aio->staged_ios);
+
+ END_CRIT_SECTION();
+
+ total_submitted += did_submit;
+
+ Assert(total_submitted == did_submit);
+
+ my_aio->num_staged_ios = 0;
+
+#ifdef PGAIO_VERBOSE
+ ereport(DEBUG2,
+ errmsg("submitted %d",
+ total_submitted),
+ errhidestmt(true),
+ errhidecontext(true));
+#endif
+}
+
+bool
+pgaio_have_staged(void)
+{
+ return my_aio->num_staged_ios > 0;
+}
+
+
+
+/* --------------------------------------------------------------------------------
+ * Other
+ * --------------------------------------------------------------------------------
+ */
+
+/*
+ * Need to submit staged but not yet submitted IOs using the fd, otherwise
+ * the IO would end up targeting something bogus.
+ */
+void
+pgaio_closing_fd(int fd)
+{
+ /*
+ * Might be called before AIO is initialized or in a subprocess that
+ * doesn't use AIO.
+ */
+ if (!my_aio)
+ return;
+
+ /*
+ * For now just submit all staged IOs - we could be more selective, but
+ * it's probably not worth it.
+ */
+ pgaio_submit_staged();
+}
+
+void
+pgaio_at_xact_end(bool is_subxact, bool is_commit)
+{
+ Assert(!my_aio->handed_out_io);
+ Assert(!my_aio->handed_out_bb);
+}
+
+/*
+ * Similar to pgaio_at_xact_end(..., is_commit = false), but for cases where
+ * errors happen outside of transactions.
+ */
+void
+pgaio_at_error(void)
+{
+ Assert(!my_aio->handed_out_io);
+ Assert(!my_aio->handed_out_bb);
+}
void
assign_io_method(int newval, void *extra)
{
+ pgaio_impl = pgaio_ops_table[newval];
}
diff --git a/src/backend/storage/aio/aio_init.c b/src/backend/storage/aio/aio_init.c
index 1c277a7eb3b..e25bdf1dba0 100644
--- a/src/backend/storage/aio/aio_init.c
+++ b/src/backend/storage/aio/aio_init.c
@@ -14,33 +14,337 @@
#include "postgres.h"
+#include "miscadmin.h"
+#include "storage/aio.h"
#include "storage/aio_init.h"
+#include "storage/aio_internal.h"
+#include "storage/bufmgr.h"
+#include "storage/proc.h"
+#include "storage/shmem.h"
+static Size
+AioCtlShmemSize(void)
+{
+ Size sz;
+
+ /* aio_ctl itself */
+ sz = offsetof(PgAioCtl, io_handles);
+
+ return sz;
+}
+
+static uint32
+AioProcs(void)
+{
+ return MaxBackends + NUM_AUXILIARY_PROCS;
+}
+
+static Size
+AioBackendShmemSize(void)
+{
+ return mul_size(AioProcs(), sizeof(PgAioPerBackend));
+}
+
+static Size
+AioHandleShmemSize(void)
+{
+ Size sz;
+
+ /* ios */
+ sz = mul_size(AioProcs(),
+ mul_size(io_max_concurrency, sizeof(PgAioHandle)));
+
+ return sz;
+}
+
+static Size
+AioIOVShmemSize(void)
+{
+ /* FIXME: io_combine_limit is USERSET */
+ return mul_size(sizeof(struct iovec),
+ mul_size(mul_size(io_combine_limit, AioProcs()),
+ io_max_concurrency));
+}
+
+static Size
+AioIOVDataShmemSize(void)
+{
+ /* FIXME: io_combine_limit is USERSET */
+ return mul_size(sizeof(uint64),
+ mul_size(mul_size(io_combine_limit, AioProcs()),
+ io_max_concurrency));
+}
+
+static Size
+AioBounceBufferDescShmemSize(void)
+{
+ Size sz;
+
+ /* PgAioBounceBuffer itself */
+ sz = mul_size(sizeof(PgAioBounceBuffer),
+ mul_size(AioProcs(), io_bounce_buffers));
+
+ return sz;
+}
+
+static Size
+AioBounceBufferDataShmemSize(void)
+{
+ Size sz;
+
+ /* and the associated buffer */
+ sz = mul_size(BLCKSZ,
+ mul_size(io_bounce_buffers, AioProcs()));
+ /* memory for alignment */
+ sz += BLCKSZ;
+
+ return sz;
+}
+
+/*
+ * Choose a suitable value for io_max_concurrency.
+ *
+ * It's unlikely that we could have more IOs in flight than buffers that we
+ * would be allowed to pin.
+ *
+ * On the upper end, apply a cap too - just because shared_buffers is large,
+ * it doesn't make sense have millions of buffers undergo IO concurrently.
+ */
+static int
+AioChooseMaxConccurrency(void)
+{
+ uint32 max_backends;
+ int max_proportional_pins;
+
+ /* Similar logic to LimitAdditionalPins() */
+ max_backends = MaxBackends + NUM_AUXILIARY_PROCS;
+ max_proportional_pins = NBuffers / max_backends;
+
+ max_proportional_pins = Max(max_proportional_pins, 1);
+
+ /* apply upper limit */
+ return Min(max_proportional_pins, 64);
+}
+
+/*
+ * Choose a suitable value for io_bounce_buffers.
+ *
+ * It's very unlikely to be useful to allocate more bounce buffers for each
+ * backend than the backend is allowed to pin. Additionally, bounce buffers
+ * currently are used for writes, it seems very uncommon for more than 10% of
+ * shared_buffers to be written out concurrently.
+ *
+ * XXX: This quickly can take up significant amounts of memory, the logic
+ * should probably fine tuned.
+ */
+static int
+AioChooseBounceBuffers(void)
+{
+ uint32 max_backends;
+ int max_proportional_pins;
+
+ /* Similar logic to LimitAdditionalPins() */
+ max_backends = MaxBackends + NUM_AUXILIARY_PROCS;
+ max_proportional_pins = (NBuffers / 10) / max_backends;
+
+ max_proportional_pins = Max(max_proportional_pins, 1);
+
+ /* apply upper limit */
+ return Min(max_proportional_pins, 256);
+}
+
Size
AioShmemSize(void)
{
Size sz = 0;
+ /*
+ * We prefer to report this value's source as PGC_S_DYNAMIC_DEFAULT.
+ * However, if the DBA explicitly set wal_buffers = -1 in the config file,
+ * then PGC_S_DYNAMIC_DEFAULT will fail to override that and we must force
+ *
+ */
+ if (io_max_concurrency == -1)
+ {
+ char buf[32];
+
+ snprintf(buf, sizeof(buf), "%d", AioChooseMaxConccurrency());
+ SetConfigOption("io_max_concurrency", buf, PGC_POSTMASTER,
+ PGC_S_DYNAMIC_DEFAULT);
+ if (io_bounce_buffers == -1) /* failed to apply it? */
+ SetConfigOption("io_max_concurrency", buf, PGC_POSTMASTER,
+ PGC_S_OVERRIDE);
+ }
+
+
+ /*
+ * If io_bounce_buffers is -1, we automatically choose a suitable value.
+ *
+ * See also comment above.
+ */
+ if (io_bounce_buffers == -1)
+ {
+ char buf[32];
+
+ snprintf(buf, sizeof(buf), "%d", AioChooseBounceBuffers());
+ SetConfigOption("io_bounce_buffers", buf, PGC_POSTMASTER,
+ PGC_S_DYNAMIC_DEFAULT);
+ if (io_bounce_buffers == -1) /* failed to apply it? */
+ SetConfigOption("io_bounce_buffers", buf, PGC_POSTMASTER,
+ PGC_S_OVERRIDE);
+ }
+
+ sz = add_size(sz, AioCtlShmemSize());
+ sz = add_size(sz, AioBackendShmemSize());
+ sz = add_size(sz, AioHandleShmemSize());
+ sz = add_size(sz, AioIOVShmemSize());
+ sz = add_size(sz, AioIOVDataShmemSize());
+ sz = add_size(sz, AioBounceBufferDescShmemSize());
+ sz = add_size(sz, AioBounceBufferDataShmemSize());
+
+ if (pgaio_impl->shmem_size)
+ sz = add_size(sz, pgaio_impl->shmem_size());
+
return sz;
}
void
AioShmemInit(void)
{
+ bool found;
+ uint32 io_handle_off = 0;
+ uint32 iovec_off = 0;
+ uint32 bounce_buffers_off = 0;
+ uint32 per_backend_iovecs = io_max_concurrency * io_combine_limit;
+ uint32 per_backend_bb = io_bounce_buffers;
+ char *bounce_buffers_data;
+
+ aio_ctl = (PgAioCtl *)
+ ShmemInitStruct("AioCtl", AioCtlShmemSize(), &found);
+
+ if (found)
+ goto out;
+
+ memset(aio_ctl, 0, AioCtlShmemSize());
+
+ aio_ctl->io_handle_count = AioProcs() * io_max_concurrency;
+ aio_ctl->iovec_count = AioProcs() * per_backend_iovecs;
+ aio_ctl->bounce_buffers_count = AioProcs() * per_backend_bb;
+
+ aio_ctl->backend_state = (PgAioPerBackend *)
+ ShmemInitStruct("AioBackend", AioBackendShmemSize(), &found);
+
+ aio_ctl->io_handles = (PgAioHandle *)
+ ShmemInitStruct("AioHandle", AioHandleShmemSize(), &found);
+
+ aio_ctl->iovecs = ShmemInitStruct("AioIOV", AioIOVShmemSize(), &found);
+ aio_ctl->iovecs_data = ShmemInitStruct("AioIOVData", AioIOVDataShmemSize(), &found);
+
+ aio_ctl->bounce_buffers = ShmemInitStruct("AioBounceBufferDesc", AioBounceBufferDescShmemSize(), &found);
+
+ bounce_buffers_data = ShmemInitStruct("AioBounceBufferData", AioBounceBufferDataShmemSize(), &found);
+ bounce_buffers_data = (char *) TYPEALIGN(BLCKSZ, (uintptr_t) bounce_buffers_data);
+ aio_ctl->bounce_buffers_data = bounce_buffers_data;
+
+
+ /* Initialize IO handles. */
+ for (uint64 i = 0; i < aio_ctl->io_handle_count; i++)
+ {
+ PgAioHandle *ioh = &aio_ctl->io_handles[i];
+
+ ioh->op = PGAIO_OP_INVALID;
+ ioh->subject = ASI_INVALID;
+ ioh->state = AHS_IDLE;
+
+ slist_init(&ioh->bounce_buffers);
+ }
+
+ /* Initialize Bounce Buffers. */
+ for (uint64 i = 0; i < aio_ctl->bounce_buffers_count; i++)
+ {
+ PgAioBounceBuffer *bb = &aio_ctl->bounce_buffers[i];
+
+ bb->buffer = bounce_buffers_data;
+ bounce_buffers_data += BLCKSZ;
+ }
+
+
+ for (int procno = 0; procno < AioProcs(); procno++)
+ {
+ PgAioPerBackend *bs = &aio_ctl->backend_state[procno];
+
+ bs->io_handle_off = io_handle_off;
+ io_handle_off += io_max_concurrency;
+
+ bs->bounce_buffers_off = bounce_buffers_off;
+ bounce_buffers_off += per_backend_bb;
+
+ dclist_init(&bs->idle_ios);
+ memset(bs->staged_ios, 0, sizeof(PgAioHandle *) * PGAIO_SUBMIT_BATCH_SIZE);
+ slist_init(&bs->idle_bbs);
+
+ /* initialize per-backend IOs */
+ for (int i = 0; i < io_max_concurrency; i++)
+ {
+ PgAioHandle *ioh = &aio_ctl->io_handles[bs->io_handle_off + i];
+
+ ioh->generation = 1;
+ ioh->owner_procno = procno;
+ ioh->iovec_off = iovec_off;
+ ioh->iovec_data_len = 0;
+ ioh->report_return = NULL;
+ ioh->resowner = NULL;
+ ioh->num_shared_callbacks = 0;
+ ioh->distilled_result.status = ARS_UNKNOWN;
+ ioh->flags = 0;
+
+ ConditionVariableInit(&ioh->cv);
+
+ dclist_push_tail(&bs->idle_ios, &ioh->node);
+ iovec_off += io_combine_limit;
+ }
+
+ /* initialize per-backend bounce buffers */
+ for (int i = 0; i < per_backend_bb; i++)
+ {
+ PgAioBounceBuffer *bb = &aio_ctl->bounce_buffers[bs->bounce_buffers_off + i];
+
+ slist_push_head(&bs->idle_bbs, &bb->node);
+ }
+ }
+
+out:
+ /* Initialize IO method specific resources. */
+ if (pgaio_impl->shmem_init)
+ pgaio_impl->shmem_init(!found);
}
void
pgaio_postmaster_init(void)
{
+ if (pgaio_impl->postmaster_init)
+ pgaio_impl->postmaster_init();
}
void
pgaio_postmaster_child_init(void)
{
+ /* shouldn't be initialized twice */
+ Assert(!my_aio);
+
+ if (MyProc == NULL || MyProcNumber >= AioProcs())
+ elog(ERROR, "aio requires a normal PGPROC");
+
+ my_aio = &aio_ctl->backend_state[MyProcNumber];
+
+ if (pgaio_impl->postmaster_child_init)
+ pgaio_impl->postmaster_child_init();
}
void
pgaio_postmaster_child_init_local(void)
{
+ if (pgaio_impl->postmaster_child_init_local)
+ pgaio_impl->postmaster_child_init_local();
}
diff --git a/src/backend/storage/aio/aio_io.c b/src/backend/storage/aio/aio_io.c
new file mode 100644
index 00000000000..5b2f9ee3ba6
--- /dev/null
+++ b/src/backend/storage/aio/aio_io.c
@@ -0,0 +1,111 @@
+/*-------------------------------------------------------------------------
+ *
+ * aio_io.c
+ * Asynchronous I/O subsytem.
+ *
+ * Portions Copyright (c) 1996-2024, PostgreSQL Global Development Group
+ * Portions Copyright (c) 1994, Regents of the University of California
+ *
+ * IDENTIFICATION
+ * src/backend/storage/aio/aio_io.c
+ *
+ *-------------------------------------------------------------------------
+ */
+
+#include "postgres.h"
+
+#include "storage/aio.h"
+#include "storage/aio_internal.h"
+#include "storage/fd.h"
+#include "utils/wait_event.h"
+
+
+static void
+pgaio_io_before_prep(PgAioHandle *ioh)
+{
+ Assert(ioh->state == AHS_HANDED_OUT);
+ Assert(pgaio_io_has_subject(ioh));
+}
+
+const char *
+pgaio_io_get_op_name(PgAioHandle *ioh)
+{
+ Assert(ioh->op >= 0 && ioh->op < PGAIO_OP_COUNT);
+
+ switch (ioh->op)
+ {
+ case PGAIO_OP_INVALID:
+ return "invalid";
+ case PGAIO_OP_READ:
+ return "read";
+ case PGAIO_OP_WRITE:
+ return "write";
+ case PGAIO_OP_FSYNC:
+ return "fsync";
+ case PGAIO_OP_FLUSH_RANGE:
+ return "flush_range";
+ case PGAIO_OP_NOP:
+ return "nop";
+ }
+
+ pg_unreachable();
+}
+
+void
+pgaio_io_prep_readv(PgAioHandle *ioh,
+ int fd, int iovcnt, uint64 offset)
+{
+ pgaio_io_before_prep(ioh);
+
+ ioh->op_data.read.fd = fd;
+ ioh->op_data.read.offset = offset;
+ ioh->op_data.read.iov_length = iovcnt;
+
+ pgaio_io_prepare(ioh, PGAIO_OP_READ);
+}
+
+void
+pgaio_io_prep_writev(PgAioHandle *ioh,
+ int fd, int iovcnt, uint64 offset)
+{
+ pgaio_io_before_prep(ioh);
+
+ ioh->op_data.write.fd = fd;
+ ioh->op_data.write.offset = offset;
+ ioh->op_data.write.iov_length = iovcnt;
+
+ pgaio_io_prepare(ioh, PGAIO_OP_WRITE);
+}
+
+
+extern void
+pgaio_io_perform_synchronously(PgAioHandle *ioh)
+{
+ ssize_t result = 0;
+ struct iovec *iov = &aio_ctl->iovecs[ioh->iovec_off];
+
+ /* Perform IO. */
+ switch (ioh->op)
+ {
+ case PGAIO_OP_READ:
+ pgstat_report_wait_start(WAIT_EVENT_DATA_FILE_READ);
+ result = pg_preadv(ioh->op_data.read.fd, iov,
+ ioh->op_data.read.iov_length,
+ ioh->op_data.read.offset);
+ pgstat_report_wait_end();
+ break;
+ case PGAIO_OP_WRITE:
+ pgstat_report_wait_start(WAIT_EVENT_DATA_FILE_WRITE);
+ result = pg_pwritev(ioh->op_data.write.fd, iov,
+ ioh->op_data.write.iov_length,
+ ioh->op_data.write.offset);
+ pgstat_report_wait_end();
+ break;
+ default:
+ elog(ERROR, "not yet");
+ }
+
+ ioh->result = result < 0 ? -errno : result;
+
+ pgaio_io_process_completion(ioh, ioh->result);
+}
diff --git a/src/backend/storage/aio/aio_subject.c b/src/backend/storage/aio/aio_subject.c
new file mode 100644
index 00000000000..51ee3b3969d
--- /dev/null
+++ b/src/backend/storage/aio/aio_subject.c
@@ -0,0 +1,167 @@
+/*-------------------------------------------------------------------------
+ *
+ * aio_subject.c
+ * IO completion handling for IOs on different subjects
+ *
+ * XXX Write me
+ *
+ * Portions Copyright (c) 1996-2024, PostgreSQL Global Development Group
+ * Portions Copyright (c) 1994, Regents of the University of California
+ *
+ * IDENTIFICATION
+ * src/backend/storage/aio/aio_subject.c
+ *
+ *-------------------------------------------------------------------------
+ */
+
+#include "postgres.h"
+
+#include "storage/aio.h"
+#include "storage/aio_internal.h"
+#include "storage/buf_internals.h"
+#include "storage/bufmgr.h"
+#include "storage/smgr.h"
+#include "utils/memutils.h"
+
+
+static const PgAioSubjectInfo *aio_subject_info[] = {
+ [ASI_INVALID] = &(PgAioSubjectInfo) {
+ .name = "invalid",
+ },
+};
+
+static const PgAioHandleSharedCallbacks *aio_shared_cbs[] = {
+};
+
+
+void
+pgaio_io_add_shared_cb(PgAioHandle *ioh, PgAioHandleSharedCallbackID cbid)
+{
+ if (cbid >= lengthof(aio_shared_cbs))
+ elog(ERROR, "callback %d is out of range", cbid);
+ if (aio_shared_cbs[cbid]->complete == NULL)
+ elog(ERROR, "callback %d is undefined", cbid);
+ if (ioh->num_shared_callbacks >= AIO_MAX_SHARED_CALLBACKS)
+ elog(PANIC, "too many callbacks, the max is %d", AIO_MAX_SHARED_CALLBACKS);
+ ioh->shared_callbacks[ioh->num_shared_callbacks] = cbid;
+
+ elog(DEBUG3, "io:%d, op %s, subject %s, adding cbid num %d, id %d",
+ pgaio_io_get_id(ioh),
+ pgaio_io_get_op_name(ioh),
+ pgaio_io_get_subject_name(ioh),
+ ioh->num_shared_callbacks + 1, cbid);
+
+ ioh->num_shared_callbacks++;
+}
+
+const char *
+pgaio_io_get_subject_name(PgAioHandle *ioh)
+{
+ Assert(ioh->subject >= 0 && ioh->subject < ASI_COUNT);
+
+ return aio_subject_info[ioh->subject]->name;
+}
+
+void
+pgaio_io_prepare_subject(PgAioHandle *ioh)
+{
+ Assert(ioh->subject > ASI_INVALID && ioh->subject < ASI_COUNT);
+ Assert(ioh->op >= 0 && ioh->op < PGAIO_OP_COUNT);
+
+ for (int i = ioh->num_shared_callbacks; i > 0; i--)
+ {
+ PgAioHandleSharedCallbackID cbid = ioh->shared_callbacks[i - 1];
+ const PgAioHandleSharedCallbacks *cbs = aio_shared_cbs[cbid];
+
+ if (!cbs->prepare)
+ continue;
+
+ elog(DEBUG3, "io:%d, op %s, subject %s, calling cbid num %d, id %d: prepare",
+ pgaio_io_get_id(ioh),
+ pgaio_io_get_op_name(ioh),
+ pgaio_io_get_subject_name(ioh),
+ i, cbid);
+ cbs->prepare(ioh);
+ }
+}
+
+void
+pgaio_io_process_completion_subject(PgAioHandle *ioh)
+{
+ PgAioResult result;
+
+ Assert(ioh->subject >= 0 && ioh->subject < ASI_COUNT);
+ Assert(ioh->op >= 0 && ioh->op < PGAIO_OP_COUNT);
+
+ result.status = ARS_OK; /* low level IO is always considered OK */
+ result.result = ioh->result;
+ result.id = 0; /* FIXME */
+ result.error_data = 0;
+
+ for (int i = ioh->num_shared_callbacks; i > 0; i--)
+ {
+ PgAioHandleSharedCallbackID cbid;
+
+ cbid = ioh->shared_callbacks[i - 1];
+ elog(DEBUG3, "io:%d, op %s, subject %s, calling cbid num %d, id %d with distilled result status %d, id %u, error_data: %d, result: %d",
+ pgaio_io_get_id(ioh),
+ pgaio_io_get_op_name(ioh),
+ pgaio_io_get_subject_name(ioh),
+ i, cbid,
+ result.status,
+ result.id,
+ result.error_data,
+ result.result);
+ result = aio_shared_cbs[cbid]->complete(ioh, result);
+ }
+
+ ioh->distilled_result = result;
+
+ elog(DEBUG3, "io:%d, op %s, subject %s, distilled result status %d, id %u, error_data: %d, result: %d, raw_result %d",
+ pgaio_io_get_id(ioh),
+ pgaio_io_get_op_name(ioh),
+ pgaio_io_get_subject_name(ioh),
+ result.status,
+ result.id,
+ result.error_data,
+ result.result,
+ ioh->result);
+}
+
+bool
+pgaio_io_can_reopen(PgAioHandle *ioh)
+{
+ return aio_subject_info[ioh->subject]->reopen != NULL;
+}
+
+void
+pgaio_io_reopen(PgAioHandle *ioh)
+{
+ Assert(ioh->subject >= 0 && ioh->subject < ASI_COUNT);
+ Assert(ioh->op >= 0 && ioh->op < PGAIO_OP_COUNT);
+
+ aio_subject_info[ioh->subject]->reopen(ioh);
+}
+
+
+
+/* --------------------------------------------------------------------------------
+ * IO Result
+ * --------------------------------------------------------------------------------
+ */
+
+void
+pgaio_result_log(PgAioResult result, const PgAioSubjectData *subject_data, int elevel)
+{
+ const PgAioHandleSharedCallbacks *scb;
+
+ Assert(result.status != ARS_UNKNOWN);
+ Assert(result.status != ARS_OK);
+
+ scb = aio_shared_cbs[result.id];
+
+ if (scb->error == NULL)
+ elog(ERROR, "scb id %d does not have error callback", result.id);
+
+ scb->error(result, subject_data, elevel);
+}
diff --git a/src/backend/storage/aio/meson.build b/src/backend/storage/aio/meson.build
index 8d20759ebf8..8339d473aae 100644
--- a/src/backend/storage/aio/meson.build
+++ b/src/backend/storage/aio/meson.build
@@ -3,5 +3,8 @@
backend_sources += files(
'aio.c',
'aio_init.c',
+ 'aio_io.c',
+ 'aio_subject.c',
+ 'method_sync.c',
'read_stream.c',
)
diff --git a/src/backend/storage/aio/method_sync.c b/src/backend/storage/aio/method_sync.c
new file mode 100644
index 00000000000..9a3e70bde33
--- /dev/null
+++ b/src/backend/storage/aio/method_sync.c
@@ -0,0 +1,43 @@
+/*-------------------------------------------------------------------------
+ *
+ * method_sync.c
+ * "AIO" implementation that just executes IO synchronously
+ *
+ * This method is mainly to check if AIO use causes regressions. Other IO
+ * methods might also fall back to the synchronous method for functionality
+ * they cannot provide.
+ *
+ * Portions Copyright (c) 1996-2021, PostgreSQL Global Development Group
+ * Portions Copyright (c) 1994, Regents of the University of California
+ *
+ * IDENTIFICATION
+ * src/backend/storage/aio/method_sync.c
+ *
+ *-------------------------------------------------------------------------
+ */
+
+#include "postgres.h"
+
+#include "storage/aio.h"
+#include "storage/aio_internal.h"
+
+static bool pgaio_sync_needs_synchronous_execution(PgAioHandle *ioh);
+static int pgaio_sync_submit(uint16 num_staged_ios, PgAioHandle **staged_ios);
+
+
+const IoMethodOps pgaio_sync_ops = {
+ .needs_synchronous_execution = pgaio_sync_needs_synchronous_execution,
+ .submit = pgaio_sync_submit,
+};
+
+static bool
+pgaio_sync_needs_synchronous_execution(PgAioHandle *ioh)
+{
+ return true;
+}
+
+static int
+pgaio_sync_submit(uint16 num_staged_ios, PgAioHandle **staged_ios)
+{
+ elog(ERROR, "should be unreachable");
+}
diff --git a/src/backend/utils/activity/wait_event_names.txt b/src/backend/utils/activity/wait_event_names.txt
index 8efb4044d6f..99ec8321746 100644
--- a/src/backend/utils/activity/wait_event_names.txt
+++ b/src/backend/utils/activity/wait_event_names.txt
@@ -191,6 +191,9 @@ ABI_compatibility:
Section: ClassName - WaitEventIO
+AIO_SUBMIT "Waiting for AIO submission."
+AIO_DRAIN "Waiting for IOs to finish."
+AIO_COMPLETION "Waiting for completion callback."
BASEBACKUP_READ "Waiting for base backup to read from a file."
BASEBACKUP_SYNC "Waiting for data written by a base backup to reach durable storage."
BASEBACKUP_WRITE "Waiting for base backup to write to a file."
diff --git a/src/backend/utils/misc/guc_tables.c b/src/backend/utils/misc/guc_tables.c
index a4b3c7c62bd..e5886f3b0e9 100644
--- a/src/backend/utils/misc/guc_tables.c
+++ b/src/backend/utils/misc/guc_tables.c
@@ -3201,6 +3201,31 @@ struct config_int ConfigureNamesInt[] =
NULL, NULL, NULL
},
+ {
+ {"io_max_concurrency",
+ PGC_POSTMASTER,
+ RESOURCES_ASYNCHRONOUS,
+ gettext_noop("Number of IOs that may be in flight in one backend."),
+ NULL,
+ },
+ &io_max_concurrency,
+ -1, -1, 1024,
+ NULL, NULL, NULL
+ },
+
+ {
+ {"io_bounce_buffers",
+ PGC_POSTMASTER,
+ RESOURCES_ASYNCHRONOUS,
+ gettext_noop("Number of IO Bounce Buffers reserved for each backend."),
+ NULL,
+ GUC_UNIT_BLOCKS
+ },
+ &io_bounce_buffers,
+ -1, -1, 4096,
+ NULL, NULL, NULL
+ },
+
{
{"backend_flush_after", PGC_USERSET, RESOURCES_ASYNCHRONOUS,
gettext_noop("Number of pages after which previously performed writes are flushed to disk."),
diff --git a/src/backend/utils/misc/postgresql.conf.sample b/src/backend/utils/misc/postgresql.conf.sample
index 3a5e307c9dc..ed746b8a533 100644
--- a/src/backend/utils/misc/postgresql.conf.sample
+++ b/src/backend/utils/misc/postgresql.conf.sample
@@ -841,6 +841,12 @@
#io_method = sync # (change requires restart)
+#io_max_concurrency = 32 # Max number of IOs that may be in
+ # flight at the same time in one backend
+ # (change requires restart)
+#io_bounce_buffers = -1 # -1 sets based on shared_buffers
+ # (change requires restart)
+
#------------------------------------------------------------------------------
# CUSTOMIZED OPTIONS
diff --git a/src/backend/utils/resowner/resowner.c b/src/backend/utils/resowner/resowner.c
index 505534ee8d3..d1932b7393c 100644
--- a/src/backend/utils/resowner/resowner.c
+++ b/src/backend/utils/resowner/resowner.c
@@ -47,6 +47,8 @@
#include "common/hashfn.h"
#include "common/int.h"
+#include "lib/ilist.h"
+#include "storage/aio.h"
#include "storage/ipc.h"
#include "storage/predicate.h"
#include "storage/proc.h"
@@ -155,6 +157,13 @@ struct ResourceOwnerData
/* The local locks cache. */
LOCALLOCK *locks[MAX_RESOWNER_LOCKS]; /* list of owned locks */
+
+ /*
+ * AIO handles & bounce buffers need be registered in critical sections
+ * and therefore cannot use the normal ResoureElem mechanism.
+ */
+ dlist_head aio_handles;
+ dlist_head aio_bounce_buffers;
};
@@ -425,6 +434,9 @@ ResourceOwnerCreate(ResourceOwner parent, const char *name)
parent->firstchild = owner;
}
+ dlist_init(&owner->aio_handles);
+ dlist_init(&owner->aio_bounce_buffers);
+
return owner;
}
@@ -725,6 +737,21 @@ ResourceOwnerReleaseInternal(ResourceOwner owner,
* so issue warnings. In the abort case, just clean up quietly.
*/
ResourceOwnerReleaseAll(owner, phase, isCommit);
+
+ /* XXX: Could probably be a later phase? */
+ while (!dlist_is_empty(&owner->aio_handles))
+ {
+ dlist_node *node = dlist_head_node(&owner->aio_handles);
+
+ pgaio_io_release_resowner(node, !isCommit);
+ }
+
+ while (!dlist_is_empty(&owner->aio_bounce_buffers))
+ {
+ dlist_node *node = dlist_head_node(&owner->aio_bounce_buffers);
+
+ pgaio_bounce_buffer_release_resowner(node, !isCommit);
+ }
}
else if (phase == RESOURCE_RELEASE_LOCKS)
{
@@ -1082,3 +1109,27 @@ ResourceOwnerForgetLock(ResourceOwner owner, LOCALLOCK *locallock)
elog(ERROR, "lock reference %p is not owned by resource owner %s",
locallock, owner->name);
}
+
+void
+ResourceOwnerRememberAioHandle(ResourceOwner owner, struct dlist_node *ioh_node)
+{
+ dlist_push_tail(&owner->aio_handles, ioh_node);
+}
+
+void
+ResourceOwnerForgetAioHandle(ResourceOwner owner, struct dlist_node *ioh_node)
+{
+ dlist_delete_from(&owner->aio_handles, ioh_node);
+}
+
+void
+ResourceOwnerRememberAioBounceBuffer(ResourceOwner owner, struct dlist_node *ioh_node)
+{
+ dlist_push_tail(&owner->aio_bounce_buffers, ioh_node);
+}
+
+void
+ResourceOwnerForgetAioBounceBuffer(ResourceOwner owner, struct dlist_node *ioh_node)
+{
+ dlist_delete_from(&owner->aio_bounce_buffers, ioh_node);
+}
diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list
index 2681dd51bb7..2f463d29ca1 100644
--- a/src/tools/pgindent/typedefs.list
+++ b/src/tools/pgindent/typedefs.list
@@ -1259,6 +1259,7 @@ IntoClause
InvalMessageArray
InvalidationMsgsGroup
IoMethod
+IoMethodOps
IpcMemoryId
IpcMemoryKey
IpcMemoryState
@@ -2094,6 +2095,24 @@ Permutation
PermutationStep
PermutationStepBlocker
PermutationStepBlockerType
+PgAioBounceBuffer
+PgAioCtl
+PgAioHandle
+PgAioHandleFlags
+PgAioHandleRef
+PgAioHandleState
+PgAioHandleSharedCallbacks
+PgAioHandleSharedCallbackID
+PgAioHandleSharedCallbacks
+PgAioOp
+PgAioOpData
+PgAioPerBackend
+PgAioResultStatus
+PgAioResult
+PgAioReturn
+PgAioSubjectData
+PgAioSubjectID
+PgAioSubjectInfo
PgArchData
PgBackendGSSStatus
PgBackendSSLStatus
--
2.45.2.827.g557ae147e6
v2.1-0008-aio-Skeleton-IO-worker-infrastructure.patch (text/x-diff, 21.3K) download
From 00976ef4bb067dda2454e0f4c4a74fc421715954 Mon Sep 17 00:00:00 2001
From: Andres Freund <andres@anarazel.de>
Date: Wed, 4 Sep 2024 15:24:51 -0400
Subject: [PATCH v2.1 08/20] aio: Skeleton IO worker infrastructure
This doesn't do anything useful on its own, but the code that needs to be
touched is independent of other changes.
Remarks:
- should completely get rid of ID assignment logic in postmaster.c
- postmaster.c badly needs a refactoring.
- dynamic increase / decrease of workers based on IO load
Author:
Reviewed-by:
Discussion: https://postgr.es/m/
Backpatch:
---
src/include/miscadmin.h | 2 +
src/include/postmaster/postmaster.h | 1 +
src/include/storage/aio_init.h | 2 +
src/include/storage/io_worker.h | 22 +++
src/include/storage/proc.h | 4 +-
src/backend/postmaster/launch_backend.c | 2 +
src/backend/postmaster/postmaster.c | 186 ++++++++++++++++--
src/backend/storage/aio/Makefile | 1 +
src/backend/storage/aio/meson.build | 1 +
src/backend/storage/aio/method_worker.c | 84 ++++++++
src/backend/tcop/postgres.c | 2 +
src/backend/utils/activity/pgstat_io.c | 1 +
.../utils/activity/wait_event_names.txt | 1 +
src/backend/utils/init/miscinit.c | 3 +
src/backend/utils/misc/guc_tables.c | 13 ++
src/backend/utils/misc/postgresql.conf.sample | 1 +
16 files changed, 311 insertions(+), 15 deletions(-)
create mode 100644 src/include/storage/io_worker.h
create mode 100644 src/backend/storage/aio/method_worker.c
diff --git a/src/include/miscadmin.h b/src/include/miscadmin.h
index 25348e71eb9..d043445b544 100644
--- a/src/include/miscadmin.h
+++ b/src/include/miscadmin.h
@@ -352,6 +352,7 @@ typedef enum BackendType
B_ARCHIVER,
B_BG_WRITER,
B_CHECKPOINTER,
+ B_IO_WORKER,
B_STARTUP,
B_WAL_RECEIVER,
B_WAL_SUMMARIZER,
@@ -380,6 +381,7 @@ extern PGDLLIMPORT BackendType MyBackendType;
#define AmWalReceiverProcess() (MyBackendType == B_WAL_RECEIVER)
#define AmWalSummarizerProcess() (MyBackendType == B_WAL_SUMMARIZER)
#define AmWalWriterProcess() (MyBackendType == B_WAL_WRITER)
+#define AmIoWorkerProcess() (MyBackendType == B_IO_WORKER)
extern const char *GetBackendTypeDesc(BackendType backendType);
diff --git a/src/include/postmaster/postmaster.h b/src/include/postmaster/postmaster.h
index 63c12917cfe..4cc000df79e 100644
--- a/src/include/postmaster/postmaster.h
+++ b/src/include/postmaster/postmaster.h
@@ -62,6 +62,7 @@ extern void InitProcessGlobals(void);
extern int MaxLivePostmasterChildren(void);
extern bool PostmasterMarkPIDForWorkerNotify(int);
+extern void assign_io_workers(int newval, void *extra);
#ifdef WIN32
extern void pgwin32_register_deadchild_callback(HANDLE procHandle, DWORD procId);
diff --git a/src/include/storage/aio_init.h b/src/include/storage/aio_init.h
index 5bcfb8a9d58..a38dd982fbe 100644
--- a/src/include/storage/aio_init.h
+++ b/src/include/storage/aio_init.h
@@ -23,4 +23,6 @@ extern void pgaio_postmaster_init(void);
extern void pgaio_postmaster_child_init_local(void);
extern void pgaio_postmaster_child_init(void);
+extern bool pgaio_workers_enabled(void);
+
#endif /* AIO_INIT_H */
diff --git a/src/include/storage/io_worker.h b/src/include/storage/io_worker.h
new file mode 100644
index 00000000000..ba5dcb9e6e4
--- /dev/null
+++ b/src/include/storage/io_worker.h
@@ -0,0 +1,22 @@
+/*-------------------------------------------------------------------------
+ *
+ * io_worker.h
+ * IO worker for implementing AIO "ourselves"
+ *
+ *
+ * Portions Copyright (c) 1996-2024, PostgreSQL Global Development Group
+ * Portions Copyright (c) 1994, Regents of the University of California
+ *
+ * src/include/storage/io.h
+ *
+ *-------------------------------------------------------------------------
+ */
+#ifndef IO_WORKER_H
+#define IO_WORKER_H
+
+
+extern void IoWorkerMain(char *startup_data, size_t startup_data_len) pg_attribute_noreturn();
+
+extern int io_workers;
+
+#endif /* IO_WORKER_H */
diff --git a/src/include/storage/proc.h b/src/include/storage/proc.h
index deeb06c9e01..b466ba843d6 100644
--- a/src/include/storage/proc.h
+++ b/src/include/storage/proc.h
@@ -442,7 +442,9 @@ extern PGDLLIMPORT PGPROC *PreparedXactProcs;
* 2 slots, but WAL writer is launched only after startup has exited, so we
* only need 6 slots.
*/
-#define NUM_AUXILIARY_PROCS 6
+#define MAX_IO_WORKERS 32
+#define NUM_AUXILIARY_PROCS (6 + MAX_IO_WORKERS)
+
/* configurable options */
extern PGDLLIMPORT int DeadlockTimeout;
diff --git a/src/backend/postmaster/launch_backend.c b/src/backend/postmaster/launch_backend.c
index 0ae23fdf55e..78429b2af2f 100644
--- a/src/backend/postmaster/launch_backend.c
+++ b/src/backend/postmaster/launch_backend.c
@@ -55,6 +55,7 @@
#include "replication/walreceiver.h"
#include "storage/dsm.h"
#include "storage/fd.h"
+#include "storage/io_worker.h"
#include "storage/ipc.h"
#include "storage/pg_shmem.h"
#include "storage/pmsignal.h"
@@ -199,6 +200,7 @@ static child_process_kind child_process_kinds[] = {
[B_ARCHIVER] = {"archiver", PgArchiverMain, true},
[B_BG_WRITER] = {"bgwriter", BackgroundWriterMain, true},
[B_CHECKPOINTER] = {"checkpointer", CheckpointerMain, true},
+ [B_IO_WORKER] = {"io_worker", IoWorkerMain, true},
[B_STARTUP] = {"startup", StartupProcessMain, true},
[B_WAL_RECEIVER] = {"wal_receiver", WalReceiverMain, true},
[B_WAL_SUMMARIZER] = {"wal_summarizer", WalSummarizerMain, true},
diff --git a/src/backend/postmaster/postmaster.c b/src/backend/postmaster/postmaster.c
index 70c5ce19f6e..3d970374733 100644
--- a/src/backend/postmaster/postmaster.c
+++ b/src/backend/postmaster/postmaster.c
@@ -113,6 +113,7 @@
#include "replication/walsender.h"
#include "storage/aio_init.h"
#include "storage/fd.h"
+#include "storage/io_worker.h"
#include "storage/ipc.h"
#include "storage/pmsignal.h"
#include "storage/proc.h"
@@ -321,6 +322,7 @@ typedef enum
* ckpt */
PM_SHUTDOWN_2, /* waiting for archiver and walsenders to
* finish */
+ PM_SHUTDOWN_IO, /* waiting for io workers to exit */
PM_WAIT_DEAD_END, /* waiting for dead_end children to exit */
PM_NO_CHILDREN, /* all important children have exited */
} PMState;
@@ -382,6 +384,10 @@ bool LoadedSSL = false;
static DNSServiceRef bonjour_sdref = NULL;
#endif
+/* State for IO worker management. */
+static int io_worker_count = 0;
+static pid_t io_worker_pids[MAX_IO_WORKERS];
+
/*
* postmaster.c - function prototypes
*/
@@ -420,6 +426,9 @@ static int CountChildren(int target);
static Backend *assign_backendlist_entry(void);
static void LaunchMissingBackgroundProcesses(void);
static void maybe_start_bgworkers(void);
+static bool maybe_reap_io_worker(int pid);
+static void maybe_adjust_io_workers(void);
+static void signal_io_workers(int signal);
static bool CreateOptsFile(int argc, char *argv[], char *fullprogname);
static pid_t StartChildProcess(BackendType type);
static void StartAutovacuumWorker(void);
@@ -1334,6 +1343,11 @@ PostmasterMain(int argc, char *argv[])
*/
AddToDataDirLockFile(LOCK_FILE_LINE_PM_STATUS, PM_STATUS_STARTING);
+ pmState = PM_STARTUP;
+
+ /* Make sure we can perform I/O while starting up. */
+ maybe_adjust_io_workers();
+
/* Start bgwriter and checkpointer so they can help with recovery */
if (CheckpointerPID == 0)
CheckpointerPID = StartChildProcess(B_CHECKPOINTER);
@@ -1346,7 +1360,6 @@ PostmasterMain(int argc, char *argv[])
StartupPID = StartChildProcess(B_STARTUP);
Assert(StartupPID != 0);
StartupStatus = STARTUP_RUNNING;
- pmState = PM_STARTUP;
/* Some workers may be scheduled to start now */
maybe_start_bgworkers();
@@ -1995,6 +2008,7 @@ process_pm_reload_request(void)
signal_child(SysLoggerPID, SIGHUP);
if (SlotSyncWorkerPID != 0)
signal_child(SlotSyncWorkerPID, SIGHUP);
+ signal_io_workers(SIGHUP);
/* Reload authentication config files too */
if (!load_hba())
@@ -2527,6 +2541,22 @@ process_pm_child_exit(void)
}
}
+ /* Was it an IO worker? */
+ if (maybe_reap_io_worker(pid))
+ {
+ if (!EXIT_STATUS_0(exitstatus) && !EXIT_STATUS_1(exitstatus))
+ HandleChildCrash(pid, exitstatus, _("io worker"));
+
+ maybe_adjust_io_workers();
+
+ if (io_worker_count == 0 &&
+ pmState >= PM_SHUTDOWN_IO)
+ {
+ pmState = PM_WAIT_DEAD_END;
+ }
+ continue;
+ }
+
/*
* We don't know anything about this child process. That's highly
* unexpected, as we do track all the child processes that we fork.
@@ -2764,6 +2794,9 @@ HandleChildCrash(int pid, int exitstatus, const char *procname)
if (SlotSyncWorkerPID != 0)
sigquit_child(SlotSyncWorkerPID);
+ /* Take care of io workers too */
+ signal_io_workers(SIGQUIT);
+
/* We do NOT restart the syslogger */
}
@@ -2987,10 +3020,11 @@ PostmasterStateMachine(void)
FatalError = true;
pmState = PM_WAIT_DEAD_END;
- /* Kill the walsenders and archiver too */
+ /* Kill walsenders, archiver and aio workers too */
SignalChildren(SIGQUIT);
if (PgArchPID != 0)
signal_child(PgArchPID, SIGQUIT);
+ signal_io_workers(SIGQUIT);
}
}
}
@@ -3000,16 +3034,26 @@ PostmasterStateMachine(void)
{
/*
* PM_SHUTDOWN_2 state ends when there's no other children than
- * dead_end children left. There shouldn't be any regular backends
- * left by now anyway; what we're really waiting for is walsenders and
- * archiver.
+ * dead_end children and aio workers left. There shouldn't be any
+ * regular backends left by now anyway; what we're really waiting for
+ * is walsenders and archiver.
*/
if (PgArchPID == 0 && CountChildren(BACKEND_TYPE_ALL) == 0)
{
- pmState = PM_WAIT_DEAD_END;
+ pmState = PM_SHUTDOWN_IO;
+ signal_io_workers(SIGUSR2);
}
}
+ if (pmState == PM_SHUTDOWN_IO)
+ {
+ /*
+ * PM_SHUTDOWN_IO state ends when there's only dead_end children left.
+ */
+ if (io_worker_count == 0)
+ pmState = PM_WAIT_DEAD_END;
+ }
+
if (pmState == PM_WAIT_DEAD_END)
{
/* Don't allow any new socket connection events. */
@@ -3017,17 +3061,22 @@ PostmasterStateMachine(void)
/*
* PM_WAIT_DEAD_END state ends when the BackendList is entirely empty
- * (ie, no dead_end children remain), and the archiver is gone too.
+ * (ie, no dead_end children remain), and the archiver and aio workers
+ * are all gone too.
*
- * The reason we wait for those two is to protect them against a new
+ * We need to wait for those because we might have transitioned
+ * directly to PM_WAIT_DEAD_END due to immediate shutdown or fatal
+ * error. Note that they have already been sent appropriate shutdown
+ * signals, either during a normal state transition leading up to
+ * PM_WAIT_DEAD_END, or during FatalError processing.
+ *
+ * The reason we wait for those is to protect them against a new
* postmaster starting conflicting subprocesses; this isn't an
* ironclad protection, but it at least helps in the
- * shutdown-and-immediately-restart scenario. Note that they have
- * already been sent appropriate shutdown signals, either during a
- * normal state transition leading up to PM_WAIT_DEAD_END, or during
- * FatalError processing.
+ * shutdown-and-immediately-restart scenario.
*/
- if (dlist_is_empty(&BackendList) && PgArchPID == 0)
+ if (dlist_is_empty(&BackendList) && io_worker_count == 0
+ && PgArchPID == 0)
{
/* These other guys should be dead already */
Assert(StartupPID == 0);
@@ -3120,10 +3169,14 @@ PostmasterStateMachine(void)
/* re-create shared memory and semaphores */
CreateSharedMemoryAndSemaphores();
+ pmState = PM_STARTUP;
+
+ /* Make sure we can perform I/O while starting up. */
+ maybe_adjust_io_workers();
+
StartupPID = StartChildProcess(B_STARTUP);
Assert(StartupPID != 0);
StartupStatus = STARTUP_RUNNING;
- pmState = PM_STARTUP;
/* crash recovery started, reset SIGKILL flag */
AbortStartTime = 0;
@@ -3375,6 +3428,7 @@ TerminateChildren(int signal)
signal_child(PgArchPID, signal);
if (SlotSyncWorkerPID != 0)
signal_child(SlotSyncWorkerPID, signal);
+ signal_io_workers(signal);
}
/*
@@ -3956,6 +4010,7 @@ bgworker_should_start_now(BgWorkerStartTime start_time)
{
case PM_NO_CHILDREN:
case PM_WAIT_DEAD_END:
+ case PM_SHUTDOWN_IO:
case PM_SHUTDOWN_2:
case PM_SHUTDOWN:
case PM_WAIT_BACKENDS:
@@ -4149,6 +4204,109 @@ maybe_start_bgworkers(void)
}
}
+static bool
+maybe_reap_io_worker(int pid)
+{
+ for (int id = 0; id < MAX_IO_WORKERS; ++id)
+ {
+ if (io_worker_pids[id] == pid)
+ {
+ --io_worker_count;
+ io_worker_pids[id] = 0;
+ return true;
+ }
+ }
+ return false;
+}
+
+static void
+maybe_adjust_io_workers(void)
+{
+ /* ATODO: This will need to check if io_method == worker */
+
+ /*
+ * If we're in final shutting down state, then we're just waiting for all
+ * processes to exit.
+ */
+ if (pmState >= PM_SHUTDOWN_IO)
+ return;
+
+ /* Don't start new workers during an immediate shutdown either. */
+ if (Shutdown >= ImmediateShutdown)
+ return;
+
+ /*
+ * Don't start new workers if we're in the shutdown phase of a crash
+ * restart. But we *do* need to start if we're already starting up again.
+ */
+ if (FatalError && pmState >= PM_STOP_BACKENDS)
+ return;
+
+ /* Not enough running? */
+ while (io_worker_count < io_workers)
+ {
+ int pid;
+ int id;
+
+ /* Find the lowest unused IO worker ID. */
+
+ /*
+ * AFIXME: This logic doesn't work right now, the ids aren't
+ * transported to workers anymore.
+ */
+ for (id = 0; id < MAX_IO_WORKERS; ++id)
+ {
+ if (io_worker_pids[id] == 0)
+ break;
+ }
+ if (id == MAX_IO_WORKERS)
+ elog(ERROR, "could not find a free IO worker ID");
+
+ Assert(pmState < PM_SHUTDOWN_IO);
+
+ /* Try to launch one. */
+ pid = StartChildProcess(B_IO_WORKER);
+ if (pid > 0)
+ {
+ io_worker_pids[id] = pid;
+ ++io_worker_count;
+ }
+ else
+ break; /* XXX try again soon? */
+ }
+
+ /* Too many running? */
+ if (io_worker_count > io_workers)
+ {
+ /* Ask the highest used IO worker ID to exit. */
+ for (int id = MAX_IO_WORKERS - 1; id >= 0; --id)
+ {
+ if (io_worker_pids[id] != 0)
+ {
+ kill(io_worker_pids[id], SIGUSR2);
+ break;
+ }
+ }
+ }
+}
+
+static void
+signal_io_workers(int signal)
+{
+ for (int i = 0; i < MAX_IO_WORKERS; ++i)
+ if (io_worker_pids[i] != 0)
+ signal_child(io_worker_pids[i], signal);
+}
+
+void
+assign_io_workers(int newval, void *extra)
+{
+ io_workers = newval;
+ if (!IsUnderPostmaster && pmState > PM_INIT)
+ maybe_adjust_io_workers();
+}
+
+
/*
* When a backend asks to be notified about worker state changes, we
* set a flag in its backend entry. The background worker machinery needs
diff --git a/src/backend/storage/aio/Makefile b/src/backend/storage/aio/Makefile
index b253278f3c1..fa2a7e9e5df 100644
--- a/src/backend/storage/aio/Makefile
+++ b/src/backend/storage/aio/Makefile
@@ -14,6 +14,7 @@ OBJS = \
aio_io.o \
aio_subject.o \
method_sync.o \
+ method_worker.o \
read_stream.o
include $(top_srcdir)/src/backend/common.mk
diff --git a/src/backend/storage/aio/meson.build b/src/backend/storage/aio/meson.build
index 8339d473aae..62738ce1d14 100644
--- a/src/backend/storage/aio/meson.build
+++ b/src/backend/storage/aio/meson.build
@@ -6,5 +6,6 @@ backend_sources += files(
'aio_io.c',
'aio_subject.c',
'method_sync.c',
+ 'method_worker.c',
'read_stream.c',
)
diff --git a/src/backend/storage/aio/method_worker.c b/src/backend/storage/aio/method_worker.c
new file mode 100644
index 00000000000..5df2eea4a03
--- /dev/null
+++ b/src/backend/storage/aio/method_worker.c
@@ -0,0 +1,84 @@
+/*-------------------------------------------------------------------------
+ *
+ * method_worker.c
+ * AIO implementation using workers
+ *
+ * Portions Copyright (c) 1996-2021, PostgreSQL Global Development Group
+ * Portions Copyright (c) 1994, Regents of the University of California
+ *
+ * IDENTIFICATION
+ * src/backend/storage/aio/method_worker.c
+ *
+ *-------------------------------------------------------------------------
+ */
+
+#include "postgres.h"
+
+#include "libpq/pqsignal.h"
+#include "miscadmin.h"
+#include "postmaster/interrupt.h"
+#include "storage/io_worker.h"
+#include "storage/ipc.h"
+#include "storage/latch.h"
+#include "storage/proc.h"
+#include "tcop/tcopprot.h"
+#include "utils/wait_event.h"
+
+
+int io_workers = 3;
+
+
+void
+IoWorkerMain(char *startup_data, size_t startup_data_len)
+{
+ sigjmp_buf local_sigjmp_buf;
+
+ MyBackendType = B_IO_WORKER;
+
+ /* TODO review all signals */
+ pqsignal(SIGHUP, SignalHandlerForConfigReload);
+ pqsignal(SIGINT, die); /* to allow manually triggering worker restart */
+
+ /*
+ * Ignore SIGTERM, will get explicit shutdown via SIGUSR2 later in the
+ * shutdown sequence, similar to checkpointer.
+ */
+ pqsignal(SIGTERM, SIG_IGN);
+ /* SIGQUIT handler was already set up by InitPostmasterChild */
+ pqsignal(SIGALRM, SIG_IGN);
+ pqsignal(SIGPIPE, SIG_IGN);
+ pqsignal(SIGUSR1, procsignal_sigusr1_handler);
+ pqsignal(SIGUSR2, SignalHandlerForShutdownRequest);
+ sigprocmask(SIG_SETMASK, &UnBlockSig, NULL);
+
+ /* see PostgresMain() */
+ if (sigsetjmp(local_sigjmp_buf, 1) != 0)
+ {
+ error_context_stack = NULL;
+ HOLD_INTERRUPTS();
+
+ /*
+ * We normally shouldn't get errors here. Need to do just enough error
+ * recovery so that we can mark the IO as failed and then exit.
+ */
+ LWLockReleaseAll();
+
+ /* TODO: recover from IO errors */
+
+ EmitErrorReport();
+ proc_exit(1);
+ }
+
+ /* We can now handle ereport(ERROR) */
+ PG_exception_stack = &local_sigjmp_buf;
+
+ while (!ShutdownRequestPending)
+ {
+ WaitLatch(MyLatch, WL_LATCH_SET | WL_EXIT_ON_PM_DEATH, -1,
+ WAIT_EVENT_IO_WORKER_MAIN);
+ ResetLatch(MyLatch);
+ CHECK_FOR_INTERRUPTS();
+ }
+
+ proc_exit(0);
+}
diff --git a/src/backend/tcop/postgres.c b/src/backend/tcop/postgres.c
index 4dc46b17b41..d42546db195 100644
--- a/src/backend/tcop/postgres.c
+++ b/src/backend/tcop/postgres.c
@@ -3294,6 +3294,8 @@ ProcessInterrupts(void)
(errcode(ERRCODE_ADMIN_SHUTDOWN),
errmsg("terminating background worker \"%s\" due to administrator command",
MyBgworkerEntry->bgw_type)));
+ else if (AmIoWorkerProcess())
+ proc_exit(0);
else
ereport(FATAL,
(errcode(ERRCODE_ADMIN_SHUTDOWN),
diff --git a/src/backend/utils/activity/pgstat_io.c b/src/backend/utils/activity/pgstat_io.c
index 8af55989eed..a750caa9b2a 100644
--- a/src/backend/utils/activity/pgstat_io.c
+++ b/src/backend/utils/activity/pgstat_io.c
@@ -335,6 +335,7 @@ pgstat_tracks_io_bktype(BackendType bktype)
{
case B_INVALID:
case B_ARCHIVER:
+ case B_IO_WORKER:
case B_LOGGER:
case B_WAL_RECEIVER:
case B_WAL_WRITER:
diff --git a/src/backend/utils/activity/wait_event_names.txt b/src/backend/utils/activity/wait_event_names.txt
index 99ec8321746..ecc513aa7bd 100644
--- a/src/backend/utils/activity/wait_event_names.txt
+++ b/src/backend/utils/activity/wait_event_names.txt
@@ -56,6 +56,7 @@ AUTOVACUUM_MAIN "Waiting in main loop of autovacuum launcher process."
BGWRITER_HIBERNATE "Waiting in background writer process, hibernating."
BGWRITER_MAIN "Waiting in main loop of background writer process."
CHECKPOINTER_MAIN "Waiting in main loop of checkpointer process."
+IO_WORKER_MAIN "Waiting in main loop of IO Worker process."
LOGICAL_APPLY_MAIN "Waiting in main loop of logical replication apply process."
LOGICAL_LAUNCHER_MAIN "Waiting in main loop of logical replication launcher process."
LOGICAL_PARALLEL_APPLY_MAIN "Waiting in main loop of logical replication parallel apply process."
diff --git a/src/backend/utils/init/miscinit.c b/src/backend/utils/init/miscinit.c
index b8fa2e64ffe..bedeed588d3 100644
--- a/src/backend/utils/init/miscinit.c
+++ b/src/backend/utils/init/miscinit.c
@@ -293,6 +293,9 @@ GetBackendTypeDesc(BackendType backendType)
case B_CHECKPOINTER:
backendDesc = "checkpointer";
break;
+ case B_IO_WORKER:
+ backendDesc = "io worker";
+ break;
case B_LOGGER:
backendDesc = "logger";
break;
diff --git a/src/backend/utils/misc/guc_tables.c b/src/backend/utils/misc/guc_tables.c
index e5886f3b0e9..40737882fb4 100644
--- a/src/backend/utils/misc/guc_tables.c
+++ b/src/backend/utils/misc/guc_tables.c
@@ -74,6 +74,7 @@
#include "storage/aio.h"
#include "storage/bufmgr.h"
#include "storage/bufpage.h"
+#include "storage/io_worker.h"
#include "storage/large_object.h"
#include "storage/pg_shmem.h"
#include "storage/predicate.h"
@@ -3226,6 +3227,18 @@ struct config_int ConfigureNamesInt[] =
NULL, NULL, NULL
},
+ {
+ {"io_workers",
+ PGC_SIGHUP,
+ RESOURCES_ASYNCHRONOUS,
+ gettext_noop("Number of IO worker processes, for io_method=worker."),
+ NULL,
+ },
+ &io_workers,
+ 3, 1, MAX_IO_WORKERS,
+ NULL, assign_io_workers, NULL
+ },
+
{
{"backend_flush_after", PGC_USERSET, RESOURCES_ASYNCHRONOUS,
gettext_noop("Number of pages after which previously performed writes are flushed to disk."),
diff --git a/src/backend/utils/misc/postgresql.conf.sample b/src/backend/utils/misc/postgresql.conf.sample
index ed746b8a533..8c062240373 100644
--- a/src/backend/utils/misc/postgresql.conf.sample
+++ b/src/backend/utils/misc/postgresql.conf.sample
@@ -840,6 +840,7 @@
#------------------------------------------------------------------------------
#io_method = sync # (change requires restart)
+#io_workers = 3 # 1-32;
#io_max_concurrency = 32 # Max number of IOs that may be in
# flight at the same time in one backend
--
2.45.2.827.g557ae147e6
v2.1-0009-aio-Add-worker-method.patch (text/x-diff, 18.2K) download
From bc2016ad468094ccc09507d3ddd755f5c7692d4b Mon Sep 17 00:00:00 2001
From: Andres Freund <andres@anarazel.de>
Date: Wed, 4 Sep 2024 15:27:00 -0400
Subject: [PATCH v2.1 09/20] aio: Add worker method
---
src/include/storage/aio.h | 5 +-
src/include/storage/aio_internal.h | 1 +
src/include/storage/lwlocklist.h | 1 +
src/backend/postmaster/postmaster.c | 3 +-
src/backend/storage/aio/aio.c | 2 +
src/backend/storage/aio/aio_init.c | 15 +
src/backend/storage/aio/method_worker.c | 404 +++++++++++++++++-
.../utils/activity/wait_event_names.txt | 1 +
src/backend/utils/misc/postgresql.conf.sample | 2 +-
src/tools/pgindent/typedefs.list | 3 +
10 files changed, 428 insertions(+), 9 deletions(-)
diff --git a/src/include/storage/aio.h b/src/include/storage/aio.h
index c0a59f47bc0..1e4c8807c71 100644
--- a/src/include/storage/aio.h
+++ b/src/include/storage/aio.h
@@ -332,11 +332,12 @@ extern void assign_io_method(int newval, void *extra);
typedef enum IoMethod
{
IOMETHOD_SYNC = 0,
+ IOMETHOD_WORKER,
} IoMethod;
-/* We'll default to synchronous execution. */
-#define DEFAULT_IO_METHOD IOMETHOD_SYNC
+/* We'll default to bgworker. */
+#define DEFAULT_IO_METHOD IOMETHOD_WORKER
/* GUCs */
diff --git a/src/include/storage/aio_internal.h b/src/include/storage/aio_internal.h
index 82bce1cf27c..b6f44a875dd 100644
--- a/src/include/storage/aio_internal.h
+++ b/src/include/storage/aio_internal.h
@@ -264,6 +264,7 @@ extern const char *pgaio_io_get_state_name(PgAioHandle *ioh);
/* Declarations for the tables of function pointers exposed by each IO method. */
extern const IoMethodOps pgaio_sync_ops;
+extern const IoMethodOps pgaio_worker_ops;
extern const IoMethodOps *pgaio_impl;
extern PgAioCtl *aio_ctl;
diff --git a/src/include/storage/lwlocklist.h b/src/include/storage/lwlocklist.h
index 88dc79b2bd6..7aaccf69d1e 100644
--- a/src/include/storage/lwlocklist.h
+++ b/src/include/storage/lwlocklist.h
@@ -84,3 +84,4 @@ PG_LWLOCK(50, DSMRegistry)
PG_LWLOCK(51, InjectionPoint)
PG_LWLOCK(52, SerialControl)
PG_LWLOCK(53, WaitLSN)
+PG_LWLOCK(54, AioWorkerSubmissionQueue)
diff --git a/src/backend/postmaster/postmaster.c b/src/backend/postmaster/postmaster.c
index 3d970374733..76440321d18 100644
--- a/src/backend/postmaster/postmaster.c
+++ b/src/backend/postmaster/postmaster.c
@@ -4222,7 +4222,8 @@ maybe_reap_io_worker(int pid)
static void
maybe_adjust_io_workers(void)
{
- /* ATODO: This will need to check if io_method == worker */
+ if (!pgaio_workers_enabled())
+ return;
/*
* If we're in final shutting down state, then we're just waiting for all
diff --git a/src/backend/storage/aio/aio.c b/src/backend/storage/aio/aio.c
index b5370330620..0ca641d9322 100644
--- a/src/backend/storage/aio/aio.c
+++ b/src/backend/storage/aio/aio.c
@@ -36,6 +36,7 @@ static void pgaio_bounce_buffer_wait_for_free(void);
/* Options for io_method. */
const struct config_enum_entry io_method_options[] = {
{"sync", IOMETHOD_SYNC, false},
+ {"worker", IOMETHOD_WORKER, false},
{NULL, 0, false}
};
@@ -53,6 +54,7 @@ PgAioPerBackend *my_aio;
static const IoMethodOps *pgaio_ops_table[] = {
[IOMETHOD_SYNC] = &pgaio_sync_ops,
+ [IOMETHOD_WORKER] = &pgaio_worker_ops,
};
diff --git a/src/backend/storage/aio/aio_init.c b/src/backend/storage/aio/aio_init.c
index e25bdf1dba0..ca3513019a6 100644
--- a/src/backend/storage/aio/aio_init.c
+++ b/src/backend/storage/aio/aio_init.c
@@ -19,6 +19,7 @@
#include "storage/aio_init.h"
#include "storage/aio_internal.h"
#include "storage/bufmgr.h"
+#include "storage/io_worker.h"
#include "storage/proc.h"
#include "storage/shmem.h"
@@ -37,6 +38,11 @@ AioCtlShmemSize(void)
static uint32
AioProcs(void)
{
+ /*
+ * While AIO workers don't need their own AIO context, we can't currently
+ * guarantee nothing gets assigned to the a ProcNumber for an IO worker if
+ * we just subtracted MAX_IO_WORKERS.
+ */
return MaxBackends + NUM_AUXILIARY_PROCS;
}
@@ -333,6 +339,9 @@ pgaio_postmaster_child_init(void)
/* shouldn't be initialized twice */
Assert(!my_aio);
+ if (MyBackendType == B_IO_WORKER)
+ return;
+
if (MyProc == NULL || MyProcNumber >= AioProcs())
elog(ERROR, "aio requires a normal PGPROC");
@@ -348,3 +357,9 @@ pgaio_postmaster_child_init_local(void)
if (pgaio_impl->postmaster_child_init_local)
pgaio_impl->postmaster_child_init_local();
}
+
+bool
+pgaio_workers_enabled(void)
+{
+ return io_method == IOMETHOD_WORKER;
+}
diff --git a/src/backend/storage/aio/method_worker.c b/src/backend/storage/aio/method_worker.c
index 5df2eea4a03..a6c21df2ea5 100644
--- a/src/backend/storage/aio/method_worker.c
+++ b/src/backend/storage/aio/method_worker.c
@@ -3,6 +3,21 @@
* method_worker.c
* AIO implementation using workers
*
+ * Worker processes consume IOs from a shared memory submission queue, run
+ * traditional synchronous system calls, and perform the shared completion
+ * handling immediately. Client code submits most requests by pushing IOs
+ * into the submission queue, and waits (if necessary) using condition
+ * variables. Some IOs cannot be performed in another process due to lack of
+ * infrastructure for reopening the file, and must processed synchronously by
+ * the client code when submitted.
+ *
+ * So that the submitter can make just one system call when submitting a batch
+ * of IOs, wakeups "fan out"; each woken backend can wake two more. XXX This
+ * could be improved by using futexes instead of latches to wake N waiters.
+ *
+ * This method of AIO is available in all builds on all operating systems, and
+ * is the default.
+ *
* Portions Copyright (c) 1996-2021, PostgreSQL Global Development Group
* Portions Copyright (c) 1994, Regents of the University of California
*
@@ -16,24 +31,290 @@
#include "libpq/pqsignal.h"
#include "miscadmin.h"
+#include "port/pg_bitutils.h"
+#include "postmaster/auxprocess.h"
#include "postmaster/interrupt.h"
+#include "storage/aio.h"
+#include "storage/aio_internal.h"
#include "storage/io_worker.h"
#include "storage/ipc.h"
#include "storage/latch.h"
#include "storage/proc.h"
#include "tcop/tcopprot.h"
#include "utils/wait_event.h"
+#include "utils/ps_status.h"
+
+
+/* How many workers should each worker wake up if needed? */
+#define IO_WORKER_WAKEUP_FANOUT 2
+
+
+typedef struct AioWorkerSubmissionQueue
+{
+ uint32 size;
+ uint32 mask;
+ uint32 head;
+ uint32 tail;
+ uint32 ios[FLEXIBLE_ARRAY_MEMBER];
+} AioWorkerSubmissionQueue;
+
+typedef struct AioWorkerSlot
+{
+ Latch *latch;
+ bool in_use;
+} AioWorkerSlot;
+
+typedef struct AioWorkerControl
+{
+ uint64 idle_worker_mask;
+ AioWorkerSlot workers[FLEXIBLE_ARRAY_MEMBER];
+} AioWorkerControl;
+
+
+static size_t pgaio_worker_shmem_size(void);
+static void pgaio_worker_shmem_init(bool first_time);
+static void pgaio_worker_postmaster_child_init_local(void);
+
+static bool pgaio_worker_needs_synchronous_execution(PgAioHandle *ioh);
+static int pgaio_worker_submit(uint16 num_staged_ios, PgAioHandle **staged_ios);
+
+
+const IoMethodOps pgaio_worker_ops = {
+ .shmem_size = pgaio_worker_shmem_size,
+ .shmem_init = pgaio_worker_shmem_init,
+ .postmaster_child_init_local = pgaio_worker_postmaster_child_init_local,
+
+ .needs_synchronous_execution = pgaio_worker_needs_synchronous_execution,
+ .submit = pgaio_worker_submit,
+#if 0
+ .wait_one = pgaio_worker_wait_one,
+ .retry = pgaio_worker_io_retry,
+ .drain = pgaio_worker_drain,
+#endif
+
+ .can_scatter_gather_direct = true,
+ .can_scatter_gather_buffered = true
+};
int io_workers = 3;
+static int io_worker_queue_size = 64;
+static int MyIoWorkerId;
+
+
+static AioWorkerSubmissionQueue *io_worker_submission_queue;
+static AioWorkerControl *io_worker_control;
+
+
+static size_t
+pgaio_worker_shmem_size(void)
+{
+ return
+ offsetof(AioWorkerSubmissionQueue, ios) +
+ sizeof(uint32) * io_worker_queue_size +
+ offsetof(AioWorkerControl, workers) +
+ sizeof(AioWorkerSlot) * io_workers;
+}
+
+static void
+pgaio_worker_shmem_init(bool first_time)
+{
+ bool found;
+ int size;
+
+ /* Round size up to next power of two so we can make a mask. */
+ size = pg_nextpower2_32(io_worker_queue_size);
+
+ io_worker_submission_queue =
+ ShmemInitStruct("AioWorkerSubmissionQueue",
+ offsetof(AioWorkerSubmissionQueue, ios) +
+ sizeof(uint32) * size,
+ &found);
+ if (!found)
+ {
+ io_worker_submission_queue->size = size;
+ io_worker_submission_queue->head = 0;
+ io_worker_submission_queue->tail = 0;
+ }
+
+ io_worker_control =
+ ShmemInitStruct("AioWorkerControl",
+ offsetof(AioWorkerControl, workers) +
+ sizeof(AioWorkerSlot) * io_workers,
+ &found);
+ if (!found)
+ {
+ io_worker_control->idle_worker_mask = 0;
+ for (int i = 0; i < io_workers; ++i)
+ {
+ io_worker_control->workers[i].latch = NULL;
+ io_worker_control->workers[i].in_use = false;
+ }
+ }
+}
+
+static void
+pgaio_worker_postmaster_child_init_local(void)
+{
+}
+
+
+static int
+pgaio_choose_idle_worker(void)
+{
+ int worker;
+
+ if (io_worker_control->idle_worker_mask == 0)
+ return -1;
+
+ /* Find the lowest bit position, and clear it. */
+ worker = pg_rightmost_one_pos64(io_worker_control->idle_worker_mask);
+ io_worker_control->idle_worker_mask &= ~(UINT64_C(1) << worker);
+
+ return worker;
+}
+
+static bool
+pgaio_worker_submission_queue_insert(PgAioHandle *ioh)
+{
+ AioWorkerSubmissionQueue *queue;
+ uint32 new_head;
+
+ queue = io_worker_submission_queue;
+ new_head = (queue->head + 1) & (queue->size - 1);
+ if (new_head == queue->tail)
+ {
+ elog(DEBUG1, "full");
+ return false; /* full */
+ }
+
+ queue->ios[queue->head] = pgaio_io_get_id(ioh);
+ queue->head = new_head;
+
+ return true;
+}
+
+static uint32
+pgaio_worker_submission_queue_consume(void)
+{
+ AioWorkerSubmissionQueue *queue;
+ uint32 result;
+
+ queue = io_worker_submission_queue;
+ if (queue->tail == queue->head)
+ return UINT32_MAX; /* empty */
+
+ result = queue->ios[queue->tail];
+ queue->tail = (queue->tail + 1) & (queue->size - 1);
+
+ return result;
+}
+
+static uint32
+pgaio_worker_submission_queue_depth(void)
+{
+ uint32 head;
+ uint32 tail;
+
+ head = io_worker_submission_queue->head;
+ tail = io_worker_submission_queue->tail;
+
+ if (tail > head)
+ head += io_worker_submission_queue->size;
+
+ Assert(head >= tail);
+
+ return head - tail;
+}
+
+static void
+pgaio_worker_submit_internal(int nios, PgAioHandle *ios[])
+{
+ PgAioHandle *synchronous_ios[PGAIO_SUBMIT_BATCH_SIZE];
+ int nsync = 0;
+ Latch *wakeup = NULL;
+ int worker;
+
+ Assert(nios <= PGAIO_SUBMIT_BATCH_SIZE);
+
+ LWLockAcquire(AioWorkerSubmissionQueueLock, LW_EXCLUSIVE);
+ for (int i = 0; i < nios; ++i)
+ {
+ Assert(!pgaio_worker_needs_synchronous_execution(ios[i]));
+ if (!pgaio_worker_submission_queue_insert(ios[i]))
+ {
+ /*
+ * We'll do it synchronously, but only after we've sent as many as
+ * we can to workers, to maximize concurrency.
+ */
+ synchronous_ios[nsync++] = ios[i];
+ continue;
+ }
+
+ if (wakeup == NULL)
+ {
+ /* Choose an idle worker to wake up if we haven't already. */
+ worker = pgaio_choose_idle_worker();
+ if (worker >= 0)
+ wakeup = io_worker_control->workers[worker].latch;
+
+ ereport(DEBUG3,
+ errmsg("submission for io:%d choosing worker %d, latch %p",
+ pgaio_io_get_id(ios[i]), worker, wakeup),
+ errhidestmt(true), errhidecontext(true));
+ }
+ }
+ LWLockRelease(AioWorkerSubmissionQueueLock);
+
+ if (wakeup)
+ SetLatch(wakeup);
+
+ /* Run whatever is left synchronously. */
+ if (nsync > 0)
+ {
+ for (int i = 0; i < nsync; ++i)
+ {
+ pgaio_io_perform_synchronously(synchronous_ios[i]);
+ }
+ }
+}
+
+static bool
+pgaio_worker_needs_synchronous_execution(PgAioHandle *ioh)
+{
+ return
+ !IsUnderPostmaster
+ || ioh->flags & AHF_REFERENCES_LOCAL
+ || pgaio_io_can_reopen(ioh);
+}
+
+static int
+pgaio_worker_submit(uint16 num_staged_ios, PgAioHandle **staged_ios)
+{
+ int nios = 0;
+
+ for (int i = 0; i < num_staged_ios; i++)
+ {
+ PgAioHandle *ioh = staged_ios[i];
+
+ pgaio_io_prepare_submit(ioh);
+ }
+
+ pgaio_worker_submit_internal(num_staged_ios, staged_ios);
+
+ return nios;
+}
void
IoWorkerMain(char *startup_data, size_t startup_data_len)
{
sigjmp_buf local_sigjmp_buf;
+ volatile PgAioHandle *ioh = NULL;
+ char cmd[128];
MyBackendType = B_IO_WORKER;
+ AuxiliaryProcessMainCommon();
/* TODO review all signals */
pqsignal(SIGHUP, SignalHandlerForConfigReload);
@@ -49,7 +330,34 @@ IoWorkerMain(char *startup_data, size_t startup_data_len)
pqsignal(SIGPIPE, SIG_IGN);
pqsignal(SIGUSR1, procsignal_sigusr1_handler);
pqsignal(SIGUSR2, SignalHandlerForShutdownRequest);
- sigprocmask(SIG_SETMASK, &UnBlockSig, NULL);
+
+ /* FIXME: locking */
+ MyIoWorkerId = -1;
+
+ LWLockAcquire(AioWorkerSubmissionQueueLock, LW_EXCLUSIVE);
+
+ for (int i = 0; i < io_workers; ++i)
+ {
+ if (!io_worker_control->workers[i].in_use)
+ {
+ Assert(io_worker_control->workers[i].latch == NULL);
+ io_worker_control->workers[i].in_use = true;
+ MyIoWorkerId = i;
+ break;
+ }
+ else
+ Assert(io_worker_control->workers[i].latch != NULL);
+ }
+
+ if (MyIoWorkerId == -1)
+ elog(ERROR, "couldn't find a free worker slot");
+
+ io_worker_control->idle_worker_mask |= (UINT64_C(1) << MyIoWorkerId);
+ io_worker_control->workers[MyIoWorkerId].latch = MyLatch;
+ LWLockRelease(AioWorkerSubmissionQueueLock);
+
+ sprintf(cmd, "worker: %d", MyIoWorkerId);
+ set_ps_display(cmd);
/* see PostgresMain() */
if (sigsetjmp(local_sigjmp_buf, 1) != 0)
@@ -64,21 +372,107 @@ IoWorkerMain(char *startup_data, size_t startup_data_len)
LWLockReleaseAll();
/* TODO: recover from IO errors */
+ if (ioh != NULL)
+ {
+#if 0
+ /* EINTR is treated as a retryable error */
+ pgaio_process_io_completion(unvolatize(PgAioInProgress *, io),
+ EINTR);
+#endif
+ }
EmitErrorReport();
+
+ /* FIXME: should probably be a before-shmem-exit instead */
+ LWLockAcquire(AioWorkerSubmissionQueueLock, LW_EXCLUSIVE);
+ Assert(io_worker_control->workers[MyIoWorkerId].in_use);
+ Assert(io_worker_control->workers[MyIoWorkerId].latch == MyLatch);
+
+ io_worker_control->workers[MyIoWorkerId].in_use = false;
+ io_worker_control->workers[MyIoWorkerId].latch = NULL;
+ LWLockRelease(AioWorkerSubmissionQueueLock);
+
proc_exit(1);
}
/* We can now handle ereport(ERROR) */
PG_exception_stack = &local_sigjmp_buf;
+ sigprocmask(SIG_SETMASK, &UnBlockSig, NULL);
+
while (!ShutdownRequestPending)
{
- WaitLatch(MyLatch, WL_LATCH_SET | WL_EXIT_ON_PM_DEATH, -1,
- WAIT_EVENT_IO_WORKER_MAIN);
- ResetLatch(MyLatch);
- CHECK_FOR_INTERRUPTS();
+ uint32 io_index;
+ Latch *latches[IO_WORKER_WAKEUP_FANOUT];
+ int nlatches = 0;
+ int nwakeups = 0;
+ int worker;
+
+ /* Try to get a job to do. */
+ LWLockAcquire(AioWorkerSubmissionQueueLock, LW_EXCLUSIVE);
+ if ((io_index = pgaio_worker_submission_queue_consume()) == UINT32_MAX)
+ {
+ /* Nothing to do. Mark self idle. */
+ /*
+ * XXX: Invent some kind of back pressure to reduce useless
+ * wakeups?
+ */
+ io_worker_control->idle_worker_mask |= (UINT64_C(1) << MyIoWorkerId);
+ }
+ else
+ {
+ /* Got one. Clear idle flag. */
+ io_worker_control->idle_worker_mask &= ~(UINT64_C(1) << MyIoWorkerId);
+
+ /* See if we can wake up some peers. */
+ nwakeups = Min(pgaio_worker_submission_queue_depth(),
+ IO_WORKER_WAKEUP_FANOUT);
+ for (int i = 0; i < nwakeups; ++i)
+ {
+ if ((worker = pgaio_choose_idle_worker()) < 0)
+ break;
+ latches[nlatches++] = io_worker_control->workers[worker].latch;
+ }
+#if 0
+ if (nwakeups > 0)
+ elog(LOG, "wake %d", nwakeups);
+#endif
+ }
+ LWLockRelease(AioWorkerSubmissionQueueLock);
+
+ for (int i = 0; i < nlatches; ++i)
+ SetLatch(latches[i]);
+
+ if (io_index != UINT32_MAX)
+ {
+ ioh = &aio_ctl->io_handles[io_index];
+
+ ereport(DEBUG3,
+ errmsg("worker processing io:%d",
+ pgaio_io_get_id(unvolatize(PgAioHandle *, ioh))),
+ errhidestmt(true), errhidecontext(true));
+
+ pgaio_io_reopen(unvolatize(PgAioHandle *, ioh));
+ pgaio_io_perform_synchronously(unvolatize(PgAioHandle *, ioh));
+
+ ioh = NULL;
+ }
+ else
+ {
+ WaitLatch(MyLatch, WL_LATCH_SET | WL_EXIT_ON_PM_DEATH, -1,
+ WAIT_EVENT_IO_WORKER_MAIN);
+ ResetLatch(MyLatch);
+ CHECK_FOR_INTERRUPTS();
+ }
}
+ LWLockAcquire(AioWorkerSubmissionQueueLock, LW_EXCLUSIVE);
+ Assert(io_worker_control->workers[MyIoWorkerId].in_use);
+ Assert(io_worker_control->workers[MyIoWorkerId].latch == MyLatch);
+
+ io_worker_control->workers[MyIoWorkerId].in_use = false;
+ io_worker_control->workers[MyIoWorkerId].latch = NULL;
+ LWLockRelease(AioWorkerSubmissionQueueLock);
+
proc_exit(0);
}
diff --git a/src/backend/utils/activity/wait_event_names.txt b/src/backend/utils/activity/wait_event_names.txt
index ecc513aa7bd..3678f2b3e43 100644
--- a/src/backend/utils/activity/wait_event_names.txt
+++ b/src/backend/utils/activity/wait_event_names.txt
@@ -351,6 +351,7 @@ DSMRegistry "Waiting to read or update the dynamic shared memory registry."
InjectionPoint "Waiting to read or update information related to injection points."
SerialControl "Waiting to read or update shared <filename>pg_serial</filename> state."
WaitLSN "Waiting to read or update shared Wait-for-LSN state."
+AioWorkerSubmissionQueue "Waiting to access AIO worker submission queue."
#
# END OF PREDEFINED LWLOCKS (DO NOT CHANGE THIS LINE)
diff --git a/src/backend/utils/misc/postgresql.conf.sample b/src/backend/utils/misc/postgresql.conf.sample
index 8c062240373..1fc8336496c 100644
--- a/src/backend/utils/misc/postgresql.conf.sample
+++ b/src/backend/utils/misc/postgresql.conf.sample
@@ -839,7 +839,7 @@
# WIP AIO GUC docs
#------------------------------------------------------------------------------
-#io_method = sync # (change requires restart)
+#io_method = worker # (change requires restart)
#io_workers = 3 # 1-32;
#io_max_concurrency = 32 # Max number of IOs that may be in
diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list
index 2f463d29ca1..f1cac7aa5bf 100644
--- a/src/tools/pgindent/typedefs.list
+++ b/src/tools/pgindent/typedefs.list
@@ -54,6 +54,9 @@ AggStrategy
AggTransInfo
Aggref
AggregateInstrumentation
+AioWorkerControl
+AioWorkerSlot
+AioWorkerSubmissionQueue
AlenState
Alias
AllocBlock
--
2.45.2.827.g557ae147e6
v2.1-0010-aio-Add-liburing-dependency.patch (text/x-diff, 9.9K) download
From 8cacec347f18d4d6390648928769cd084f57b77f Mon Sep 17 00:00:00 2001
From: Andres Freund <andres@anarazel.de>
Date: Wed, 5 Jun 2024 19:37:25 -0700
Subject: [PATCH v2.1 10/20] aio: Add liburing dependency
Not yet used.
Author:
Reviewed-by:
Discussion: https://postgr.es/m/
Backpatch:
---
src/include/pg_config.h.in | 3 +
src/makefiles/meson.build | 3 +
configure | 138 +++++++++++++++++++++++++++++++++++++
configure.ac | 11 +++
meson.build | 14 ++++
meson_options.txt | 3 +
src/Makefile.global.in | 4 ++
7 files changed, 176 insertions(+)
diff --git a/src/include/pg_config.h.in b/src/include/pg_config.h.in
index 38006367a40..7d2fcb9d0f5 100644
--- a/src/include/pg_config.h.in
+++ b/src/include/pg_config.h.in
@@ -693,6 +693,9 @@
/* Define to 1 to build with LDAP support. (--with-ldap) */
#undef USE_LDAP
+/* Define to build with io-uring support. (--with-liburing) */
+#undef USE_LIBURING
+
/* Define to 1 to build with XML support. (--with-libxml) */
#undef USE_LIBXML
diff --git a/src/makefiles/meson.build b/src/makefiles/meson.build
index 850e9275845..cca689b2028 100644
--- a/src/makefiles/meson.build
+++ b/src/makefiles/meson.build
@@ -200,6 +200,8 @@ pgxs_empty = [
'PTHREAD_CFLAGS', 'PTHREAD_LIBS',
'ICU_LIBS',
+
+ 'LIBURING_CFLAGS', 'LIBURING_LIBS',
]
if host_system == 'windows' and cc.get_argument_syntax() != 'msvc'
@@ -230,6 +232,7 @@ pgxs_deps = {
'gssapi': gssapi,
'icu': icu,
'ldap': ldap,
+ 'liburing': liburing,
'libxml': libxml,
'libxslt': libxslt,
'llvm': llvm,
diff --git a/configure b/configure
index 53c8a1f2bad..aa82bafe783 100755
--- a/configure
+++ b/configure
@@ -654,6 +654,8 @@ LIBOBJS
OPENSSL
ZSTD
LZ4
+LIBURING_LIBS
+LIBURING_CFLAGS
UUID_LIBS
LDAP_LIBS_BE
LDAP_LIBS_FE
@@ -712,6 +714,7 @@ XML2_CFLAGS
XML2_CONFIG
with_libxml
with_uuid
+with_liburing
with_readline
with_systemd
with_selinux
@@ -865,6 +868,7 @@ with_selinux
with_systemd
with_readline
with_libedit_preferred
+with_liburing
with_uuid
with_ossp_uuid
with_libxml
@@ -907,6 +911,8 @@ LDFLAGS_EX
LDFLAGS_SL
PERL
PYTHON
+LIBURING_CFLAGS
+LIBURING_LIBS
MSGFMT
TCLSH'
@@ -1574,6 +1580,7 @@ Optional Packages:
--without-readline do not use GNU Readline nor BSD Libedit for editing
--with-libedit-preferred
prefer BSD Libedit over GNU Readline
+ --with-liburing use liburing for async io
--with-uuid=LIB build contrib/uuid-ossp using LIB (bsd,e2fs,ossp)
--with-ossp-uuid obsolete spelling of --with-uuid=ossp
--with-libxml build with XML support
@@ -1617,6 +1624,10 @@ Some influential environment variables:
LDFLAGS_SL extra linker flags for linking shared libraries only
PERL Perl program
PYTHON Python program
+ LIBURING_CFLAGS
+ C compiler flags for LIBURING, overriding pkg-config
+ LIBURING_LIBS
+ linker flags for LIBURING, overriding pkg-config
MSGFMT msgfmt program for NLS
TCLSH Tcl interpreter program (tclsh)
@@ -8664,6 +8675,40 @@ fi
+#
+# liburing
+#
+{ $as_echo "$as_me:${as_lineno-$LINENO}: checking whether to build with liburing support" >&5
+$as_echo_n "checking whether to build with liburing support... " >&6; }
+
+
+
+# Check whether --with-liburing was given.
+if test "${with_liburing+set}" = set; then :
+ withval=$with_liburing;
+ case $withval in
+ yes)
+
+$as_echo "#define USE_LIBURING 1" >>confdefs.h
+
+ ;;
+ no)
+ :
+ ;;
+ *)
+ as_fn_error $? "no argument expected for --with-liburing option" "$LINENO" 5
+ ;;
+ esac
+
+else
+ with_liburing=no
+
+fi
+
+
+{ $as_echo "$as_me:${as_lineno-$LINENO}: result: $with_liburing" >&5
+$as_echo "$with_liburing" >&6; }
+
#
# UUID library
@@ -13209,6 +13254,99 @@ fi
fi
+if test "$with_liburing" = yes; then
+
+pkg_failed=no
+{ $as_echo "$as_me:${as_lineno-$LINENO}: checking for liburing" >&5
+$as_echo_n "checking for liburing... " >&6; }
+
+if test -n "$LIBURING_CFLAGS"; then
+ pkg_cv_LIBURING_CFLAGS="$LIBURING_CFLAGS"
+ elif test -n "$PKG_CONFIG"; then
+ if test -n "$PKG_CONFIG" && \
+ { { $as_echo "$as_me:${as_lineno-$LINENO}: \$PKG_CONFIG --exists --print-errors \"liburing\""; } >&5
+ ($PKG_CONFIG --exists --print-errors "liburing") 2>&5
+ ac_status=$?
+ $as_echo "$as_me:${as_lineno-$LINENO}: \$? = $ac_status" >&5
+ test $ac_status = 0; }; then
+ pkg_cv_LIBURING_CFLAGS=`$PKG_CONFIG --cflags "liburing" 2>/dev/null`
+ test "x$?" != "x0" && pkg_failed=yes
+else
+ pkg_failed=yes
+fi
+ else
+ pkg_failed=untried
+fi
+if test -n "$LIBURING_LIBS"; then
+ pkg_cv_LIBURING_LIBS="$LIBURING_LIBS"
+ elif test -n "$PKG_CONFIG"; then
+ if test -n "$PKG_CONFIG" && \
+ { { $as_echo "$as_me:${as_lineno-$LINENO}: \$PKG_CONFIG --exists --print-errors \"liburing\""; } >&5
+ ($PKG_CONFIG --exists --print-errors "liburing") 2>&5
+ ac_status=$?
+ $as_echo "$as_me:${as_lineno-$LINENO}: \$? = $ac_status" >&5
+ test $ac_status = 0; }; then
+ pkg_cv_LIBURING_LIBS=`$PKG_CONFIG --libs "liburing" 2>/dev/null`
+ test "x$?" != "x0" && pkg_failed=yes
+else
+ pkg_failed=yes
+fi
+ else
+ pkg_failed=untried
+fi
+
+
+
+if test $pkg_failed = yes; then
+ { $as_echo "$as_me:${as_lineno-$LINENO}: result: no" >&5
+$as_echo "no" >&6; }
+
+if $PKG_CONFIG --atleast-pkgconfig-version 0.20; then
+ _pkg_short_errors_supported=yes
+else
+ _pkg_short_errors_supported=no
+fi
+ if test $_pkg_short_errors_supported = yes; then
+ LIBURING_PKG_ERRORS=`$PKG_CONFIG --short-errors --print-errors --cflags --libs "liburing" 2>&1`
+ else
+ LIBURING_PKG_ERRORS=`$PKG_CONFIG --print-errors --cflags --libs "liburing" 2>&1`
+ fi
+ # Put the nasty error message in config.log where it belongs
+ echo "$LIBURING_PKG_ERRORS" >&5
+
+ as_fn_error $? "Package requirements (liburing) were not met:
+
+$LIBURING_PKG_ERRORS
+
+Consider adjusting the PKG_CONFIG_PATH environment variable if you
+installed software in a non-standard prefix.
+
+Alternatively, you may set the environment variables LIBURING_CFLAGS
+and LIBURING_LIBS to avoid the need to call pkg-config.
+See the pkg-config man page for more details." "$LINENO" 5
+elif test $pkg_failed = untried; then
+ { $as_echo "$as_me:${as_lineno-$LINENO}: result: no" >&5
+$as_echo "no" >&6; }
+ { { $as_echo "$as_me:${as_lineno-$LINENO}: error: in \`$ac_pwd':" >&5
+$as_echo "$as_me: error: in \`$ac_pwd':" >&2;}
+as_fn_error $? "The pkg-config script could not be found or is too old. Make sure it
+is in your PATH or set the PKG_CONFIG environment variable to the full
+path to pkg-config.
+
+Alternatively, you may set the environment variables LIBURING_CFLAGS
+and LIBURING_LIBS to avoid the need to call pkg-config.
+See the pkg-config man page for more details.
+
+To get pkg-config, see <http://pkg-config.freedesktop.org/>.
+See \`config.log' for more details" "$LINENO" 5; }
+else
+ LIBURING_CFLAGS=$pkg_cv_LIBURING_CFLAGS
+ LIBURING_LIBS=$pkg_cv_LIBURING_LIBS
+ { $as_echo "$as_me:${as_lineno-$LINENO}: result: yes" >&5
+$as_echo "yes" >&6; }
+
+fi
+fi
##
## Header files
diff --git a/configure.ac b/configure.ac
index 6a35b2880bf..04480cdea0a 100644
--- a/configure.ac
+++ b/configure.ac
@@ -970,6 +970,14 @@ AC_SUBST(with_readline)
PGAC_ARG_BOOL(with, libedit-preferred, no,
[prefer BSD Libedit over GNU Readline])
+#
+# liburing
+#
+AC_MSG_CHECKING([whether to build with liburing support])
+PGAC_ARG_BOOL(with, liburing, no, [use liburing for async io],
+ [AC_DEFINE([USE_LIBURING], 1, [Define to build with io-uring support. (--with-liburing)])])
+AC_MSG_RESULT([$with_liburing])
+AC_SUBST(with_liburing)
#
# UUID library
@@ -1426,6 +1434,9 @@ elif test "$with_uuid" = ossp ; then
fi
AC_SUBST(UUID_LIBS)
+if test "$with_liburing" = yes; then
+ PKG_CHECK_MODULES(LIBURING, liburing)
+fi
##
## Header files
diff --git a/meson.build b/meson.build
index 4764b09266e..53266e04005 100644
--- a/meson.build
+++ b/meson.build
@@ -848,6 +848,18 @@ endif
+###############################################################
+# Library: liburing
+###############################################################
+
+liburingopt = get_option('liburing')
+liburing = dependency('liburing', required: liburingopt)
+if liburing.found()
+ cdata.set('USE_LIBURING', 1)
+endif
+
+
+
###############################################################
# Library: libxml
###############################################################
@@ -3094,6 +3106,7 @@ backend_both_deps += [
icu_i18n,
ldap,
libintl,
+ liburing,
libxml,
lz4,
pam,
@@ -3738,6 +3751,7 @@ if meson.version().version_compare('>=0.57')
'gss': gssapi,
'icu': icu,
'ldap': ldap,
+ 'liburing': liburing,
'libxml': libxml,
'libxslt': libxslt,
'llvm': llvm,
diff --git a/meson_options.txt b/meson_options.txt
index b9421557606..084eebe72d7 100644
--- a/meson_options.txt
+++ b/meson_options.txt
@@ -103,6 +103,9 @@ option('ldap', type: 'feature', value: 'auto',
option('libedit_preferred', type: 'boolean', value: false,
description: 'Prefer BSD Libedit over GNU Readline')
+option('liburing', type : 'feature', value: 'auto',
+ description: 'Use liburing for async io')
+
option('libxml', type: 'feature', value: 'auto',
description: 'XML support')
diff --git a/src/Makefile.global.in b/src/Makefile.global.in
index 42f50b49761..a8ff18faed6 100644
--- a/src/Makefile.global.in
+++ b/src/Makefile.global.in
@@ -190,6 +190,7 @@ with_systemd = @with_systemd@
with_gssapi = @with_gssapi@
with_krb_srvnam = @with_krb_srvnam@
with_ldap = @with_ldap@
+with_liburing = @with_liburing@
with_libxml = @with_libxml@
with_libxslt = @with_libxslt@
with_llvm = @with_llvm@
@@ -216,6 +217,9 @@ krb_srvtab = @krb_srvtab@
ICU_CFLAGS = @ICU_CFLAGS@
ICU_LIBS = @ICU_LIBS@
+LIBURING_CFLAGS = @LIBURING_CFLAGS@
+LIBURING_LIBS = @LIBURING_LIBS@
+
TCLSH = @TCLSH@
TCL_LIBS = @TCL_LIBS@
TCL_LIB_SPEC = @TCL_LIB_SPEC@
--
2.45.2.827.g557ae147e6
v2.1-0011-aio-Add-io_uring-method.patch (text/x-diff, 13.5K) download
From b50761d00455ef1fd0a0c9625624866c60a7333f Mon Sep 17 00:00:00 2001
From: Andres Freund <andres@anarazel.de>
Date: Wed, 4 Sep 2024 16:15:17 -0400
Subject: [PATCH v2.1 11/20] aio: Add io_uring method
---
src/include/storage/aio.h | 1 +
src/include/storage/aio_internal.h | 3 +
src/include/storage/lwlock.h | 1 +
src/backend/storage/aio/Makefile | 1 +
src/backend/storage/aio/aio.c | 6 +
src/backend/storage/aio/meson.build | 1 +
src/backend/storage/aio/method_io_uring.c | 383 ++++++++++++++++++++++
src/backend/storage/lmgr/lwlock.c | 1 +
src/tools/pgindent/typedefs.list | 1 +
9 files changed, 398 insertions(+)
create mode 100644 src/backend/storage/aio/method_io_uring.c
diff --git a/src/include/storage/aio.h b/src/include/storage/aio.h
index 1e4c8807c71..b8c743548c9 100644
--- a/src/include/storage/aio.h
+++ b/src/include/storage/aio.h
@@ -333,6 +333,7 @@ typedef enum IoMethod
{
IOMETHOD_SYNC = 0,
IOMETHOD_WORKER,
+ IOMETHOD_IO_URING,
} IoMethod;
diff --git a/src/include/storage/aio_internal.h b/src/include/storage/aio_internal.h
index b6f44a875dd..5d18d112e2d 100644
--- a/src/include/storage/aio_internal.h
+++ b/src/include/storage/aio_internal.h
@@ -265,6 +265,9 @@ extern const char *pgaio_io_get_state_name(PgAioHandle *ioh);
/* Declarations for the tables of function pointers exposed by each IO method. */
extern const IoMethodOps pgaio_sync_ops;
extern const IoMethodOps pgaio_worker_ops;
+#ifdef USE_LIBURING
+extern const IoMethodOps pgaio_uring_ops;
+#endif
extern const IoMethodOps *pgaio_impl;
extern PgAioCtl *aio_ctl;
diff --git a/src/include/storage/lwlock.h b/src/include/storage/lwlock.h
index eabf813ce05..72f928b7602 100644
--- a/src/include/storage/lwlock.h
+++ b/src/include/storage/lwlock.h
@@ -217,6 +217,7 @@ typedef enum BuiltinTrancheIds
LWTRANCHE_SUBTRANS_SLRU,
LWTRANCHE_XACT_SLRU,
LWTRANCHE_PARALLEL_VACUUM_DSA,
+ LWTRANCHE_AIO_URING_COMPLETION,
LWTRANCHE_FIRST_USER_DEFINED,
} BuiltinTrancheIds;
diff --git a/src/backend/storage/aio/Makefile b/src/backend/storage/aio/Makefile
index fa2a7e9e5df..3bcb8a0b2ed 100644
--- a/src/backend/storage/aio/Makefile
+++ b/src/backend/storage/aio/Makefile
@@ -13,6 +13,7 @@ OBJS = \
aio_init.o \
aio_io.o \
aio_subject.o \
+ method_io_uring.o \
method_sync.o \
method_worker.o \
read_stream.o
diff --git a/src/backend/storage/aio/aio.c b/src/backend/storage/aio/aio.c
index 0ca641d9322..8877a33b9f2 100644
--- a/src/backend/storage/aio/aio.c
+++ b/src/backend/storage/aio/aio.c
@@ -37,6 +37,9 @@ static void pgaio_bounce_buffer_wait_for_free(void);
const struct config_enum_entry io_method_options[] = {
{"sync", IOMETHOD_SYNC, false},
{"worker", IOMETHOD_WORKER, false},
+#ifdef USE_LIBURING
+ {"io_uring", IOMETHOD_IO_URING, false},
+#endif
{NULL, 0, false}
};
@@ -55,6 +58,9 @@ PgAioPerBackend *my_aio;
static const IoMethodOps *pgaio_ops_table[] = {
[IOMETHOD_SYNC] = &pgaio_sync_ops,
[IOMETHOD_WORKER] = &pgaio_worker_ops,
+#ifdef USE_LIBURING
+ [IOMETHOD_IO_URING] = &pgaio_uring_ops,
+#endif
};
diff --git a/src/backend/storage/aio/meson.build b/src/backend/storage/aio/meson.build
index 62738ce1d14..537f23d446d 100644
--- a/src/backend/storage/aio/meson.build
+++ b/src/backend/storage/aio/meson.build
@@ -5,6 +5,7 @@ backend_sources += files(
'aio_init.c',
'aio_io.c',
'aio_subject.c',
+ 'method_io_uring.c',
'method_sync.c',
'method_worker.c',
'read_stream.c',
diff --git a/src/backend/storage/aio/method_io_uring.c b/src/backend/storage/aio/method_io_uring.c
new file mode 100644
index 00000000000..0f0eda0ce9b
--- /dev/null
+++ b/src/backend/storage/aio/method_io_uring.c
@@ -0,0 +1,383 @@
+/*-------------------------------------------------------------------------
+ *
+ * method_io_uring.c
+ * AIO implementation using io_uring on Linux
+ *
+ * XXX Write me
+ *
+ * Portions Copyright (c) 1996-2024, PostgreSQL Global Development Group
+ * Portions Copyright (c) 1994, Regents of the University of California
+ *
+ * IDENTIFICATION
+ * src/backend/storage/aio/method_io_uring.c
+ *
+ *-------------------------------------------------------------------------
+ */
+
+#include "postgres.h"
+
+#ifdef USE_LIBURING
+
+#include <liburing.h>
+
+#include "pgstat.h"
+#include "port/pg_iovec.h"
+#include "storage/aio_internal.h"
+#include "storage/fd.h"
+#include "storage/proc.h"
+#include "storage/shmem.h"
+
+
+/* Entry points for IoMethodOps. */
+static size_t pgaio_uring_shmem_size(void);
+static void pgaio_uring_postmaster_init(void);
+static void pgaio_uring_shmem_init(bool first_time);
+static void pgaio_uring_postmaster_child_init(void);
+static void pgaio_uring_postmaster_child_init_local(void);
+
+static int pgaio_uring_submit(uint16 num_staged_ios, PgAioHandle **staged_ios);
+static void pgaio_uring_wait_one(PgAioHandle *ioh, uint64 ref_generation);
+
+static void pgaio_uring_sq_from_io(PgAioHandle *ioh, struct io_uring_sqe *sqe);
+
+
+const IoMethodOps pgaio_uring_ops = {
+ .shmem_size = pgaio_uring_shmem_size,
+ .shmem_init = pgaio_uring_shmem_init,
+ .postmaster_init = pgaio_uring_postmaster_init,
+ .postmaster_child_init = pgaio_uring_postmaster_child_init,
+ .postmaster_child_init_local = pgaio_uring_postmaster_child_init_local,
+
+ .submit = pgaio_uring_submit,
+ .wait_one = pgaio_uring_wait_one,
+#if 0
+ .retry = pgaio_uring_io_retry,
+ .wait_one = pgaio_uring_wait_one,
+ .drain = pgaio_uring_drain,
+#endif
+ .can_scatter_gather_direct = true,
+ .can_scatter_gather_buffered = true
+};
+
+typedef struct PgAioUringContext
+{
+ LWLock completion_lock;
+
+ struct io_uring io_uring_ring;
+ /* XXX: probably worth padding to a cacheline boundary here */
+} PgAioUringContext;
+
+
+static PgAioUringContext *aio_uring_contexts;
+static PgAioUringContext *my_shared_uring_context;
+
+/* io_uring local state */
+static struct io_uring local_ring;
+
+
+
+static Size
+AioContextShmemSize(void)
+{
+ uint32 TotalProcs = MaxBackends + NUM_AUXILIARY_PROCS - MAX_IO_WORKERS;
+
+ return mul_size(TotalProcs, sizeof(PgAioUringContext));
+}
+
+static size_t
+pgaio_uring_shmem_size(void)
+{
+ return AioContextShmemSize();
+}
+
+static void
+pgaio_uring_shmem_init(bool first_time)
+{
+ uint32 TotalProcs = MaxBackends + NUM_AUXILIARY_PROCS - MAX_IO_WORKERS;
+ bool found;
+
+ aio_uring_contexts = (PgAioUringContext *)
+ ShmemInitStruct("AioUring", pgaio_uring_shmem_size(), &found);
+
+ if (found)
+ return;
+
+ for (int contextno = 0; contextno < TotalProcs; contextno++)
+ {
+ PgAioUringContext *context = &aio_uring_contexts[contextno];
+ int ret;
+
+ /*
+ * XXX: Probably worth sharing the WQ between the different rings,
+ * when supported by the kernel. Could also cause additional
+ * contention, I guess?
+ */
+#if 0
+ if (!AcquireExternalFD())
+ elog(ERROR, "No external FD available");
+#endif
+ ret = io_uring_queue_init(io_max_concurrency, &context->io_uring_ring, 0);
+ if (ret < 0)
+ elog(ERROR, "io_uring_queue_init failed: %s", strerror(-ret));
+
+ LWLockInitialize(&context->completion_lock, LWTRANCHE_AIO_URING_COMPLETION);
+ }
+}
+
+static void
+pgaio_uring_postmaster_init(void)
+{
+ uint32 TotalProcs =
+ MaxBackends + NUM_AUXILIARY_PROCS - MAX_IO_WORKERS;
+
+ for (int i = 0; i < TotalProcs; i++)
+ ReserveExternalFD();
+}
+
+static void
+pgaio_uring_postmaster_child_init(void)
+{
+ my_shared_uring_context = &aio_uring_contexts[MyProcNumber];
+}
+
+static void
+pgaio_uring_postmaster_child_init_local(void)
+{
+ int ret;
+
+ ret = io_uring_queue_init(32, &local_ring, 0);
+ if (ret < 0)
+ elog(ERROR, "io_uring_queue_init failed: %s", strerror(-ret));
+}
+
+static int
+pgaio_uring_submit(uint16 num_staged_ios, PgAioHandle **staged_ios)
+{
+ struct io_uring *uring_instance = &my_shared_uring_context->io_uring_ring;
+
+ Assert(num_staged_ios <= PGAIO_SUBMIT_BATCH_SIZE);
+
+ for (int i = 0; i < num_staged_ios; i++)
+ {
+ PgAioHandle *ioh = staged_ios[i];
+ struct io_uring_sqe *sqe;
+
+ sqe = io_uring_get_sqe(uring_instance);
+
+ pgaio_io_prepare_submit(ioh);
+ pgaio_uring_sq_from_io(ioh, sqe);
+ }
+
+ while (true)
+ {
+ int ret;
+
+ pgstat_report_wait_start(WAIT_EVENT_AIO_SUBMIT);
+ ret = io_uring_submit(uring_instance);
+ pgstat_report_wait_end();
+
+ if (ret == -EINTR)
+ {
+ elog(DEBUG3, "submit EINTR, nios: %d", num_staged_ios);
+ continue;
+ }
+ if (ret < 0)
+ elog(PANIC, "failed: %d/%s",
+ ret, strerror(-ret));
+ else if (ret != num_staged_ios)
+ {
+ /* likely unreachable, but if it is, we would need to re-submit */
+ elog(PANIC, "submitted only %d of %d",
+ ret, num_staged_ios);
+ }
+ else
+ {
+ elog(DEBUG3, "submit nios: %d", num_staged_ios);
+ }
+ break;
+ }
+
+ return num_staged_ios;
+}
+
+
+#define PGAIO_MAX_LOCAL_REAPED 16
+
+static void
+pgaio_uring_drain_locked(PgAioUringContext *context)
+{
+ int ready;
+ int orig_ready;
+
+ /*
+ * Don't drain more events than available right now. Otherwise it's
+ * plausible that one backend could get stuck, for a while, receiving CQEs
+ * without actually processing them.
+ */
+ orig_ready = ready = io_uring_cq_ready(&context->io_uring_ring);
+
+ while (ready > 0)
+ {
+ struct io_uring_cqe *reaped_cqes[PGAIO_MAX_LOCAL_REAPED];
+ uint32 reaped;
+
+ START_CRIT_SECTION();
+ reaped =
+ io_uring_peek_batch_cqe(&context->io_uring_ring,
+ reaped_cqes,
+ Min(PGAIO_MAX_LOCAL_REAPED, ready));
+ Assert(reaped <= ready);
+
+ ready -= reaped;
+
+ for (int i = 0; i < reaped; i++)
+ {
+ struct io_uring_cqe *cqe = reaped_cqes[i];
+ PgAioHandle *ioh;
+
+ ioh = io_uring_cqe_get_data(cqe);
+ io_uring_cqe_seen(&context->io_uring_ring, cqe);
+
+ pgaio_io_process_completion(ioh, cqe->res);
+ }
+
+ END_CRIT_SECTION();
+
+ ereport(DEBUG3,
+ errmsg("drained %d/%d, now expecting %d",
+ reaped, orig_ready, io_uring_cq_ready(&context->io_uring_ring)),
+ errhidestmt(true),
+ errhidecontext(true));
+
+ }
+}
+
+static void
+pgaio_uring_wait_one(PgAioHandle *ioh, uint64 ref_generation)
+{
+ PgAioHandleState state;
+ ProcNumber owner_procno = ioh->owner_procno;
+ PgAioUringContext *owner_context = &aio_uring_contexts[owner_procno];
+ bool expect_cqe;
+ int waited = 0;
+
+ /*
+ * We ought to have a smarter locking scheme, nearly all the time the
+ * backend owning the ring will reap the completions, making the locking
+ * unnecessarily expensive.
+ */
+ LWLockAcquire(&owner_context->completion_lock, LW_EXCLUSIVE);
+
+ while (true)
+ {
+ ereport(DEBUG3,
+ errmsg("wait_one for io:%d in state %s, cycle %d",
+ pgaio_io_get_id(ioh), pgaio_io_get_state_name(ioh), waited),
+ errhidestmt(true),
+ errhidecontext(true));
+
+ if (pgaio_io_was_recycled(ioh, ref_generation, &state) ||
+ state != AHS_IN_FLIGHT)
+ {
+ break;
+ }
+ else if (io_uring_cq_ready(&owner_context->io_uring_ring))
+ {
+ expect_cqe = true;
+ }
+ else
+ {
+ int ret;
+ struct io_uring_cqe *cqes;
+
+ pgstat_report_wait_start(WAIT_EVENT_AIO_DRAIN);
+ ret = io_uring_wait_cqes(&owner_context->io_uring_ring, &cqes, 1, NULL, NULL);
+ pgstat_report_wait_end();
+
+ if (ret == -EINTR)
+ {
+ continue;
+ }
+ else if (ret != 0)
+ {
+ elog(PANIC, "unexpected: %d/%s: %m", ret, strerror(-ret));
+ }
+ else
+ {
+ Assert(cqes != NULL);
+ expect_cqe = true;
+ waited++;
+ }
+ }
+
+ if (expect_cqe)
+ {
+ pgaio_uring_drain_locked(owner_context);
+ }
+ }
+
+ LWLockRelease(&owner_context->completion_lock);
+
+ ereport(DEBUG3,
+ errmsg("wait_one with %d sleeps",
+ waited),
+ errhidestmt(true),
+ errhidecontext(true));
+}
+
+static void
+pgaio_uring_sq_from_io(PgAioHandle *ioh, struct io_uring_sqe *sqe)
+{
+ struct iovec *iov;
+
+ switch (ioh->op)
+ {
+ case PGAIO_OP_READ:
+ iov = &aio_ctl->iovecs[ioh->iovec_off];
+ if (ioh->op_data.read.iov_length == 1)
+ {
+ io_uring_prep_read(sqe,
+ ioh->op_data.read.fd,
+ iov->iov_base,
+ iov->iov_len,
+ ioh->op_data.read.offset);
+ }
+ else
+ {
+ io_uring_prep_readv(sqe,
+ ioh->op_data.read.fd,
+ iov,
+ ioh->op_data.read.iov_length,
+ ioh->op_data.read.offset);
+
+ }
+ break;
+
+ case PGAIO_OP_WRITE:
+ iov = &aio_ctl->iovecs[ioh->iovec_off];
+ if (ioh->op_data.write.iov_length == 1)
+ {
+ io_uring_prep_write(sqe,
+ ioh->op_data.write.fd,
+ iov->iov_base,
+ iov->iov_len,
+ ioh->op_data.write.offset);
+ }
+ else
+ {
+ io_uring_prep_writev(sqe,
+ ioh->op_data.write.fd,
+ iov,
+ ioh->op_data.write.iov_length,
+ ioh->op_data.write.offset);
+ }
+ break;
+
+ default:
+ elog(ERROR, "not implemented");
+ }
+
+ io_uring_sqe_set_data(sqe, ioh);
+}
+
+#endif /* USE_LIBURING */
diff --git a/src/backend/storage/lmgr/lwlock.c b/src/backend/storage/lmgr/lwlock.c
index a5fa77412ed..b138a36c461 100644
--- a/src/backend/storage/lmgr/lwlock.c
+++ b/src/backend/storage/lmgr/lwlock.c
@@ -166,6 +166,7 @@ static const char *const BuiltinTrancheNames[] = {
[LWTRANCHE_SUBTRANS_SLRU] = "SubtransSLRU",
[LWTRANCHE_XACT_SLRU] = "XactSLRU",
[LWTRANCHE_PARALLEL_VACUUM_DSA] = "ParallelVacuumDSA",
+ [LWTRANCHE_AIO_URING_COMPLETION] = "AioUringCompletion",
};
StaticAssertDecl(lengthof(BuiltinTrancheNames) ==
diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list
index f1cac7aa5bf..46d31cf2b9f 100644
--- a/src/tools/pgindent/typedefs.list
+++ b/src/tools/pgindent/typedefs.list
@@ -2116,6 +2116,7 @@ PgAioReturn
PgAioSubjectData
PgAioSubjectID
PgAioSubjectInfo
+PgAioUringContext
PgArchData
PgBackendGSSStatus
PgBackendSSLStatus
--
2.45.2.827.g557ae147e6
v2.1-0012-aio-Add-README.md-explaining-higher-level-desig.patch (text/x-diff, 14.1K) download
From 55448fdaa5e54983fdfd147ff1f28cf3867d58e5 Mon Sep 17 00:00:00 2001
From: Andres Freund <andres@anarazel.de>
Date: Fri, 6 Sep 2024 15:27:57 -0400
Subject: [PATCH v2.1 12/20] aio: Add README.md explaining higher level design
---
src/backend/storage/aio/README.md | 311 ++++++++++++++++++++++++++++++
1 file changed, 311 insertions(+)
create mode 100644 src/backend/storage/aio/README.md
diff --git a/src/backend/storage/aio/README.md b/src/backend/storage/aio/README.md
new file mode 100644
index 00000000000..9c3a11f2063
--- /dev/null
+++ b/src/backend/storage/aio/README.md
@@ -0,0 +1,311 @@
+# Asynchronous & Direct IO
+
+## Design Criteria & Motivation
+
+### Why Asynchronous IO
+
+Until the introduction of asynchronous IO Postgres relied on the operating
+system to hide the cost of synchronous IO from Postgres. While this worked
+surprisingly well in a lot of workloads, it does not do as good a job on
+prefetching and controlled writeback as we would like.
+
+There are important expensive operations like `fdatasync()` where the operating
+system cannot hide the storage latency. This is particularly important for WAL
+writes, where the ability to asynchronously issue `fdatasync()` or O_DSYNC
+writes can yield significantly higher throughput.
+
+
+### Why Direct / unbuffered IO
+
+The main reason to want to use Direct IO are:
+
+- Lower CPU usage / higher throughput. Particularly on modern storage buffered
+ writes are bottlenecked by the operating system having to copy data from the
+ kernel's page cache to postgres buffer pool using the CPU. Whereas direct IO
+ can often move the data directly between the storage devices and postgres'
+ buffer cache, using DMA. While that transfer is ongoing, the CPU is free to
+ perform other work.
+- Reduced latency - Direct IO can have substantially lower latency than
+ buffered IO, which can be impactful for OLTP workloads bottlenecked by WAL
+ write latency.
+- Avoiding double buffering between operating system cache and postgres'
+ shared_buffers.
+- Better control over the timing and pace of dirty data writeback.
+
+
+The main reason *not* to use Direct IO are:
+
+- Without AIO, Direct IO is unusably slow for most purposes.
+- Even with AIO, many parts of postgres need to be modified to perform
+ explicit prefetching.
+- In situations where shared_buffers cannot be set appropriately large,
+ e.g. because there are many different postgres instances hosted on shared
+ hardware, performance will often be worse then when using buffered IO.
+
+
+### Deadlock and Starvation Dangers due to AIO
+
+Using AIO in a naive way can easily lead to deadlocks in an environment where
+the source/target of AIO are shared resources, like pages in postgres'
+shared_buffers.
+
+Consider one backend performing readahead on a table, initiating IO for a
+number of buffers ahead of the current "scan position". If that backend then
+performs some operation that blocks, or even just is slow, the IO completion
+for the asynchronously initiated read may not be processed.
+
+This AIO implementation solves this problem by requiring that AIO methods
+either allow AIO completions to be processed by any backend in the system
+(e.g. io_uring), or to guarantee that AIO processing will happen even when the
+issuing backend is blocked (e.g. worker mode, which offloads completion
+processing to the AIO workers).
+
+
+### IO can be started in critical sections
+
+Using AIO for WAL writes can reduce the overhead of WAL logging substantially:
+
+- AIO allows to start WAL writes eagerly, so they complete before needing to
+ wait
+- AIO allows to have multiple WAL flushes in progress at the same time
+- AIO makes it more realistic to use O\_DIRECT + O\_DSYNC, which can reduce
+ the number of roundtrips to storage on some OSs and storage HW (buffered IO
+ and direct IO without O_DSYNC needs to issue a write and after the writes
+ completion a cache cache flush, whereas O\_DIRECT + O\_DSYNC can use a
+ single FUA write).
+
+The need to be able to execute IO in critical sections has substantial design
+implication on the AIO subsystem. Mainly because completing IOs (see prior
+section) needs to be possible within a critical section, even if the
+to-be-completed IO itself was not issued in a critical section. Consider
+e.g. the case of a backend first starting a number of writes from shared
+buffers and then starting to flush the WAL. Because only a limited amount of
+IO can be in-progress at the same time, initiating the IO for flushing the WAL
+may require to first finish executing IO executed earlier.
+
+
+### State for AIO needs to live in shared memory
+
+Because postgres uses a process model and because AIOs need to be
+complete-able by any backend much of the state of the AIO subsystem needs to
+live in shared memory.
+
+In an `EXEC_BACKEND` build backends executable code and other process local
+state is not necessarily mapped to the same addresses in each process due to
+ASLR. This means that the shared memory cannot contain pointer to callbacks.
+
+
+## Design of the AIO Subsystem
+
+
+### AIO Methods
+
+To achieve portability and performance, multiple methods of performing AIO are
+implemented and others are likely worth adding in the future.
+
+
+#### Synchronous Mode
+
+`io_method=sync` does not actually perform AIO but allows to use the AIO API
+while performing synchronous IO. This can be useful for debugging. The code
+for the synchronous mode is also used as a fallback by e.g. the [worker
+mode](#Worker) uses it to execute IO that cannot be executed by workers.
+
+
+#### Worker
+
+`io_method=worker` is available on every platform postgres runs on, and
+implements asynchronous IO - from the view of the issuing process - by
+dispatching the IO to one of several worker processes performing the IO in a
+synchronous manner.
+
+
+#### io_uring
+
+`io_method=io_uring` is available on Linux 5.1+. In contrast to worker mode it
+dispatches all IO from within the process, lowering context switch rate /
+latency.
+
+
+### AIO Handles
+
+The central API piece for postgres' AIO abstraction are AIO handles. To
+execute an IO one first has to acquire an IO handle (`pgaio_io_get()`) and
+then "defined", i.e. associate an IO operation with the handle.
+
+Often AIO handles are acquired on a higher level and then passed to a lower
+level to be fully defined. E.g., for IO to/from shared buffers, bufmgr.c
+routines acquire the handle, which is then passed through smgr.c, md.c to be
+finally fully defined in fd.c.
+
+The functions used at the lowest level to define the operation are
+`pgaio_io_prep_*()`.
+
+Because acquisition of an IO handle
+[must always succeed](#IO-can-be-started-in-critical-sections)
+and the number of AIO Handles
+[has to be limited](#State-for-AIO-needs-to-live-in-shared-memory)
+AIO handles can be reused as soon as they have completed. Obviously code needs
+to be able to react to IO completion. Shared state can be updated using
+[AIO Completion callbacks](#AIO-Callbacks)
+and the issuing backend can provide a backend local variable to receive the
+result of the IO, as described in
+[AIO Result](#AIO-Result)
+. An IO can be waited for, by both the issuing and any other backend, using
+[AIO Referencs](#AIO-References).
+
+
+Because an AIO Handle is not executable just after calling `pgaio_io_get()`
+and because `pgaio_io_get()` needs to be able to succeed, only a single AIO
+Handle may be acquired (i.e. returned by `pgaio_io_get()`) without causing the
+IO to have been defined (by, potentially indirectly, causing
+`pgaio_io_prep_*()` to have been called). Otherwise a backend could trivially
+self-deadlock by using up all AIO Handles without the ability to wait for some
+of the IOs to complete.
+
+If it turns out that an AIO Handle is not needed, e.g., because the handle was
+acquired before holding a contended lock, it can be released without being
+defined using `pgaio_io_release()`.
+
+
+### AIO Callbacks
+
+Commonly several layers need to react to completion of an IO. E.g. for a read
+md.c needs to check if the IO outright failed or was shorter than needed,
+bufmgr.c needs to verify the page looks valid and bufmgr.c needs to update the
+BufferDesc to update the buffer's state.
+
+The fact that several layers / subsystems need to react to IO completion poses
+a few challenges:
+
+- Upper layers should not need to know details of lower layers. E.g. bufmgr.c
+ should not assume the IO will pass through md.c. Therefore upper levels
+ cannot know what lower layers would consider an error.
+
+- Lower layers should not need to know about upper layers. E.g. smgr APIs are
+ used going through shared buffers but are also used bypassing shared
+ buffers. This means that e.g. md.c is not in a position to validate
+ checksums.
+
+- Having code in the AIO subsystem for every possible combination of layers
+ would lead to a lot of duplication.
+
+The "solution" to this the ability to associate multiple completion callbacks
+with a handle. E.g. bufmgr.c can have a callback to update the BufferDesc
+state and to verify the page and md.c. another callback to check if the IO
+operation was successful.
+
+As [mentioned](#State-for-AIO-needs-to-live-in-shared-memory), shared memory
+currently cannot contain function pointers. Because of that completion
+callbacks are not directly identified by function pointers but by IDs
+(`PgAioHandleSharedCallbackID`). A substantial added benefit is that that
+allows callbacks to be identified by much smaller amount of memory (a single
+byte currently).
+
+In addition to completion, AIO callbacks also are called to "prepare" an
+IO. This is, e.g., used to acquire buffer pins owned by the AIO subsystem for
+IO to/from shared buffers, which is required to handle the case where the
+issuing backend errors out and releases its own pins.
+
+As [explained earlier](#IO-can-be-started-in-critical-sections) IO completions
+need to be safe to execute in critical sections. To allow the backend that
+issued the IO to error out in case of failure [AIO Result](#AIO-Results) can
+be used.
+
+
+### AIO Subjects
+
+In addition to the completion callbacks describe above, each AIO Handle has
+exactly one "subject". Each subject has some space inside an AIO Handle with
+information specific to the subject and can provide callbacks to allow to
+reopen the underlying file (required for worker mode) and to describe the IO
+operation (used for debug logging and error messages).
+
+
+### AIO References
+
+As [described above](#AIO-Handles) can be reused immediately after completion
+and therefore cannot be used to wait for completion of the IO. Waiting is
+enabled using AIO references, which do not just identify an AIO Handle but
+also include the handles "generation".
+
+A reference to an AIO Handle can be acquired using `pgaio_io_get_ref()` and
+then waited upon using `pgaio_io_ref_wait()`.
+
+
+### AIO Results
+
+As AIO completion callbacks
+[are executed in critical sections](#IO-can-be-started-in-critical-sections)
+and [may be executed by any backend](#Deadlock-and-Starvation-Dangers-due-to-AIO)
+completion callbacks cannot be used to, e.g., make the query that triggered an
+IO ERROR out.
+
+To allow to react to failing IOs the issuing backend can pass a pointer to a
+`PgAioReturn` in backend local memory. Before an AIO Handle is reused the
+`PgAioReturn` is filled with information about the IO. This includes
+information about whether the IO was successful (as a value of
+`PgAioResultStatus`) and enough information to raise an error in case of a
+failure (via `pgaio_result_log()`, with the error details encoded in
+`PgAioResult`).
+
+XXX: "return" vs "result" vs "result status" seems quite confusing. The naming
+should be improved.
+
+
+### AIO Errors
+
+It would be very convenient to have shared completion callbacks encode the
+details of errors as an `ErrorData` that could be raised at a later
+time. Unfortunately doing so would require allocating memory. While elog.c can
+guarantee (well, kinda) that logging a message will not run out of memory,
+that only works because a very limited number of messages are in the process
+of being logged. With AIO a large number of concurrently issued AIOs might
+fail.
+
+To avoid the need for preallocating a potentially large amount of memory (in
+shared memory no less!), completion callbacks instead have to encode errors in
+a more compact format that can be converted into an error message.
+
+
+### AIO Bounce Buffers
+
+For some uses of AIO there is no convenient memory location as the source /
+destination of an AIO. E.g. when data checksums are enabled, writes from
+shared buffers currently cannot be done directly from shared buffers, as a
+shared buffer lock still allows some modification, e.g., for hint bits(see
+`FlushBuffer()`). If the write were done in-place, such modifications can
+cause the checksum to fail.
+
+For synchronous IO this is solved by copying the buffer to separate memory
+before computing the checksum and using that copy as the source buffer for the
+AIO.
+
+However, for AIO that is not a workable solution:
+- Instead of a single buffer many buffers are required, as many IOs might be
+ in flight
+- When using the [worker method](#worker), the source/target of IO needs to be
+ in shared memory, otherwise the workers won't be able to access the memory.
+
+The AIO subsystem addresses this by providing a limited number of bounce
+buffers that can be used as the source / target for IO. A bounce buffer be
+acquired with `pgaio_bounce_buffer_get()` and multiple bounce buffers can be
+associated with an AIO Handle with `pgaio_io_assoc_bounce_buffer()`.
+
+Bounce buffers are automatically released when the IO completes.
+
+
+## Helpers
+
+Using the low-level AIO API introduces too much complexity to do so all over
+the tree. Most uses of AIO should be done via reusable, higher-level, helpers.
+
+
+### Read Stream
+
+A common and very beneficial use of AIO are reads where a substantial number
+of to-be-read locations are known ahead of time. E.g., for a sequential scan
+the set of blocks that need to be read can be determined solely by knowing the
+current position and checking the buffer mapping table.
+[Read stream](../../include/storage/read_stream.h)
+makes it comparatively easy to use AIO for such use cases.
--
2.45.2.827.g557ae147e6
v2.1-0013-aio-Implement-smgr-md.c-aio-methods.patch (text/x-diff, 22.6K) download
From f138cbab018b104e416d23175a38141d8827232d Mon Sep 17 00:00:00 2001
From: Andres Freund <andres@anarazel.de>
Date: Sat, 31 Aug 2024 22:33:30 -0400
Subject: [PATCH v2.1 13/20] aio: Implement smgr/md.c aio methods
---
src/include/storage/aio.h | 17 +-
src/include/storage/fd.h | 6 +
src/include/storage/md.h | 12 ++
src/include/storage/smgr.h | 21 +++
src/backend/storage/aio/aio_subject.c | 4 +
src/backend/storage/file/fd.c | 68 ++++++++
src/backend/storage/smgr/md.c | 217 ++++++++++++++++++++++++++
src/backend/storage/smgr/smgr.c | 91 +++++++++++
8 files changed, 434 insertions(+), 2 deletions(-)
diff --git a/src/include/storage/aio.h b/src/include/storage/aio.h
index b8c743548c9..07bf92a6b7a 100644
--- a/src/include/storage/aio.h
+++ b/src/include/storage/aio.h
@@ -57,9 +57,10 @@ typedef enum PgAioSubjectID
{
/* intentionally the zero value, to help catch zeroed memory etc */
ASI_INVALID = 0,
+ ASI_SMGR,
} PgAioSubjectID;
-#define ASI_COUNT (ASI_INVALID + 1)
+#define ASI_COUNT (ASI_SMGR + 1)
/*
* Flags for an IO that can be set with pgaio_io_set_flag().
@@ -90,7 +91,8 @@ typedef enum PgAioHandleFlags
*/
typedef enum PgAioHandleSharedCallbackID
{
- ASC_PLACEHOLDER /* empty enums are invalid */ ,
+ ASC_MD_READV,
+ ASC_MD_WRITEV,
} PgAioHandleSharedCallbackID;
@@ -139,6 +141,17 @@ typedef union
typedef union PgAioSubjectData
{
+ struct
+ {
+ RelFileLocator rlocator; /* physical relation identifier */
+ BlockNumber blockNum; /* blknum relative to begin of reln */
+ int nblocks;
+ ForkNumber forkNum:8; /* don't waste 4 byte for four values */
+ bool is_temp; /* proc can be inferred by owning AIO */
+ bool release_lock;
+ int8 mode;
+ } smgr;
+
/* just as an example placeholder for later */
struct
{
diff --git a/src/include/storage/fd.h b/src/include/storage/fd.h
index 1456ab383a4..e993e1b671f 100644
--- a/src/include/storage/fd.h
+++ b/src/include/storage/fd.h
@@ -101,6 +101,8 @@ extern PGDLLIMPORT int max_safe_fds;
* prototypes for functions in fd.c
*/
+struct PgAioHandle;
+
/* Operations on virtual Files --- equivalent to Unix kernel file ops */
extern File PathNameOpenFile(const char *fileName, int fileFlags);
extern File PathNameOpenFilePerm(const char *fileName, int fileFlags, mode_t fileMode);
@@ -109,6 +111,10 @@ extern void FileClose(File file);
extern int FilePrefetch(File file, off_t offset, off_t amount, uint32 wait_event_info);
extern ssize_t FileReadV(File file, const struct iovec *iov, int iovcnt, off_t offset, uint32 wait_event_info);
extern ssize_t FileWriteV(File file, const struct iovec *iov, int iovcnt, off_t offset, uint32 wait_event_info);
+extern ssize_t FileReadV(File file, const struct iovec *iov, int iovcnt, off_t offset, uint32 wait_event_info);
+extern int FileStartReadV(struct PgAioHandle *ioh, File file, int iovcnt, off_t offset, uint32 wait_event_info);
+extern ssize_t FileWriteV(File file, const struct iovec *iov, int iovcnt, off_t offset, uint32 wait_event_info);
+extern int FileStartWriteV(struct PgAioHandle *ioh, File file, int iovcnt, off_t offset, uint32 wait_event_info);
extern int FileSync(File file, uint32 wait_event_info);
extern int FileZero(File file, off_t offset, off_t amount, uint32 wait_event_info);
extern int FileFallocate(File file, off_t offset, off_t amount, uint32 wait_event_info);
diff --git a/src/include/storage/md.h b/src/include/storage/md.h
index b72293c79a5..ede77695853 100644
--- a/src/include/storage/md.h
+++ b/src/include/storage/md.h
@@ -19,6 +19,10 @@
#include "storage/smgr.h"
#include "storage/sync.h"
+struct PgAioHandleSharedCallbacks;
+extern const struct PgAioHandleSharedCallbacks aio_md_readv_cb;
+extern const struct PgAioHandleSharedCallbacks aio_md_writev_cb;
+
/* md storage manager functionality */
extern void mdinit(void);
extern void mdopen(SMgrRelation reln);
@@ -36,9 +40,16 @@ extern uint32 mdmaxcombine(SMgrRelation reln, ForkNumber forknum,
BlockNumber blocknum);
extern void mdreadv(SMgrRelation reln, ForkNumber forknum, BlockNumber blocknum,
void **buffers, BlockNumber nblocks);
+extern void mdstartreadv(struct PgAioHandle *ioh,
+ SMgrRelation reln, ForkNumber forknum, BlockNumber blocknum,
+ void **buffers, BlockNumber nblocks);
extern void mdwritev(SMgrRelation reln, ForkNumber forknum,
BlockNumber blocknum,
const void **buffers, BlockNumber nblocks, bool skipFsync);
+extern void mdstartwritev(struct PgAioHandle *ioh,
+ SMgrRelation reln, ForkNumber forknum,
+ BlockNumber blocknum,
+ const void **buffers, BlockNumber nblocks, bool skipFsync);
extern void mdwriteback(SMgrRelation reln, ForkNumber forknum,
BlockNumber blocknum, BlockNumber nblocks);
extern BlockNumber mdnblocks(SMgrRelation reln, ForkNumber forknum);
@@ -46,6 +57,7 @@ extern void mdtruncate(SMgrRelation reln, ForkNumber forknum,
BlockNumber nblocks);
extern void mdimmedsync(SMgrRelation reln, ForkNumber forknum);
extern void mdregistersync(SMgrRelation reln, ForkNumber forknum);
+extern int mdfd(SMgrRelation reln, ForkNumber forknum, BlockNumber blocknum, uint32 *off);
extern void ForgetDatabaseSyncRequests(Oid dbid);
extern void DropRelationFiles(RelFileLocator *delrels, int ndelrels, bool isRedo);
diff --git a/src/include/storage/smgr.h b/src/include/storage/smgr.h
index 899d0d681c5..66730bc24fa 100644
--- a/src/include/storage/smgr.h
+++ b/src/include/storage/smgr.h
@@ -73,6 +73,11 @@ typedef SMgrRelationData *SMgrRelation;
#define SmgrIsTemp(smgr) \
RelFileLocatorBackendIsTemp((smgr)->smgr_rlocator)
+struct PgAioHandle;
+struct PgAioSubjectInfo;
+
+extern const struct PgAioSubjectInfo aio_smgr_subject_info;
+
extern void smgrinit(void);
extern SMgrRelation smgropen(RelFileLocator rlocator, ProcNumber backend);
extern bool smgrexists(SMgrRelation reln, ForkNumber forknum);
@@ -97,10 +102,19 @@ extern uint32 smgrmaxcombine(SMgrRelation reln, ForkNumber forknum,
extern void smgrreadv(SMgrRelation reln, ForkNumber forknum,
BlockNumber blocknum,
void **buffers, BlockNumber nblocks);
+extern void smgrstartreadv(struct PgAioHandle *ioh,
+ SMgrRelation reln, ForkNumber forknum,
+ BlockNumber blocknum,
+ void **buffers, BlockNumber nblocks);
extern void smgrwritev(SMgrRelation reln, ForkNumber forknum,
BlockNumber blocknum,
const void **buffers, BlockNumber nblocks,
bool skipFsync);
+extern void smgrstartwritev(struct PgAioHandle *ioh,
+ SMgrRelation reln, ForkNumber forknum,
+ BlockNumber blocknum,
+ const void **buffers, BlockNumber nblocks,
+ bool skipFsync);
extern void smgrwriteback(SMgrRelation reln, ForkNumber forknum,
BlockNumber blocknum, BlockNumber nblocks);
extern BlockNumber smgrnblocks(SMgrRelation reln, ForkNumber forknum);
@@ -109,6 +123,7 @@ extern void smgrtruncate(SMgrRelation reln, ForkNumber *forknum,
int nforks, BlockNumber *nblocks);
extern void smgrimmedsync(SMgrRelation reln, ForkNumber forknum);
extern void smgrregistersync(SMgrRelation reln, ForkNumber forknum);
+extern int smgrfd(SMgrRelation reln, ForkNumber forknum, BlockNumber blocknum, uint32 *off);
extern void AtEOXact_SMgr(void);
extern bool ProcessBarrierSmgrRelease(void);
@@ -126,4 +141,10 @@ smgrwrite(SMgrRelation reln, ForkNumber forknum, BlockNumber blocknum,
smgrwritev(reln, forknum, blocknum, &buffer, 1, skipFsync);
}
+extern void pgaio_io_set_subject_smgr(struct PgAioHandle *ioh,
+ SMgrRelationData *smgr,
+ ForkNumber forknum,
+ BlockNumber blocknum,
+ int nblocks);
+
#endif /* SMGR_H */
diff --git a/src/backend/storage/aio/aio_subject.c b/src/backend/storage/aio/aio_subject.c
index 51ee3b3969d..14be8432f5a 100644
--- a/src/backend/storage/aio/aio_subject.c
+++ b/src/backend/storage/aio/aio_subject.c
@@ -20,6 +20,7 @@
#include "storage/aio_internal.h"
#include "storage/buf_internals.h"
#include "storage/bufmgr.h"
+#include "storage/md.h"
#include "storage/smgr.h"
#include "utils/memutils.h"
@@ -28,9 +29,12 @@ static const PgAioSubjectInfo *aio_subject_info[] = {
[ASI_INVALID] = &(PgAioSubjectInfo) {
.name = "invalid",
},
+ [ASI_SMGR] = &aio_smgr_subject_info,
};
static const PgAioHandleSharedCallbacks *aio_shared_cbs[] = {
+ [ASC_MD_READV] = &aio_md_readv_cb,
+ [ASC_MD_WRITEV] = &aio_md_writev_cb,
};
diff --git a/src/backend/storage/file/fd.c b/src/backend/storage/file/fd.c
index ec1505802b9..f5ff554f946 100644
--- a/src/backend/storage/file/fd.c
+++ b/src/backend/storage/file/fd.c
@@ -95,6 +95,7 @@
#include "pgstat.h"
#include "portability/mem.h"
#include "postmaster/startup.h"
+#include "storage/aio.h"
#include "storage/fd.h"
#include "storage/ipc.h"
#include "utils/guc.h"
@@ -1295,6 +1296,8 @@ LruDelete(File file)
vfdP = &VfdCache[file];
+ pgaio_closing_fd(vfdP->fd);
+
/*
* Close the file. We aren't expecting this to fail; if it does, better
* to leak the FD than to mess up our internal state.
@@ -1988,6 +1991,8 @@ FileClose(File file)
if (!FileIsNotOpen(file))
{
+ pgaio_closing_fd(vfdP->fd);
+
/* close the file */
if (close(vfdP->fd) != 0)
{
@@ -2211,6 +2216,32 @@ retry:
return returnCode;
}
+int
+FileStartReadV(struct PgAioHandle *ioh, File file,
+ int iovcnt, off_t offset,
+ uint32 wait_event_info)
+{
+ int returnCode;
+ Vfd *vfdP;
+
+ Assert(FileIsValid(file));
+
+ DO_DB(elog(LOG, "FileStartReadV: %d (%s) " INT64_FORMAT " %d",
+ file, VfdCache[file].fileName,
+ (int64) offset,
+ iovcnt));
+
+ returnCode = FileAccess(file);
+ if (returnCode < 0)
+ return returnCode;
+
+ vfdP = &VfdCache[file];
+
+ pgaio_io_prep_readv(ioh, vfdP->fd, iovcnt, offset);
+
+ return 0;
+}
+
ssize_t
FileWriteV(File file, const struct iovec *iov, int iovcnt, off_t offset,
uint32 wait_event_info)
@@ -2316,6 +2347,34 @@ retry:
return returnCode;
}
+int
+FileStartWriteV(struct PgAioHandle *ioh, File file,
+ int iovcnt, off_t offset,
+ uint32 wait_event_info)
+{
+ int returnCode;
+ Vfd *vfdP;
+
+ Assert(FileIsValid(file));
+
+ DO_DB(elog(LOG, "FileStartWriteV: %d (%s) " INT64_FORMAT " %d",
+ file, VfdCache[file].fileName,
+ (int64) offset,
+ iovcnt));
+
+ returnCode = FileAccess(file);
+ if (returnCode < 0)
+ return returnCode;
+
+ vfdP = &VfdCache[file];
+
+ /* FIXME: think about / reimplement temp_file_limit */
+
+ pgaio_io_prep_writev(ioh, vfdP->fd, iovcnt, offset);
+
+ return 0;
+}
+
int
FileSync(File file, uint32 wait_event_info)
{
@@ -2499,6 +2558,12 @@ FilePathName(File file)
int
FileGetRawDesc(File file)
{
+ int returnCode;
+
+ returnCode = FileAccess(file);
+ if (returnCode < 0)
+ return returnCode;
+
Assert(FileIsValid(file));
return VfdCache[file].fd;
}
@@ -2779,6 +2844,7 @@ FreeDesc(AllocateDesc *desc)
result = closedir(desc->desc.dir);
break;
case AllocateDescRawFD:
+ pgaio_closing_fd(desc->desc.fd);
result = close(desc->desc.fd);
break;
default:
@@ -2847,6 +2913,8 @@ CloseTransientFile(int fd)
/* Only get here if someone passes us a file not in allocatedDescs */
elog(WARNING, "fd passed to CloseTransientFile was not obtained from OpenTransientFile");
+ pgaio_closing_fd(fd);
+
return close(fd);
}
diff --git a/src/backend/storage/smgr/md.c b/src/backend/storage/smgr/md.c
index 6cd81a61faa..f96308490d9 100644
--- a/src/backend/storage/smgr/md.c
+++ b/src/backend/storage/smgr/md.c
@@ -31,6 +31,7 @@
#include "miscadmin.h"
#include "pg_trace.h"
#include "pgstat.h"
+#include "storage/aio.h"
#include "storage/bufmgr.h"
#include "storage/fd.h"
#include "storage/md.h"
@@ -931,6 +932,49 @@ mdreadv(SMgrRelation reln, ForkNumber forknum, BlockNumber blocknum,
}
}
+void
+mdstartreadv(PgAioHandle *ioh,
+ SMgrRelation reln, ForkNumber forknum, BlockNumber blocknum,
+ void **buffers, BlockNumber nblocks)
+{
+ off_t seekpos;
+ MdfdVec *v;
+ BlockNumber nblocks_this_segment;
+ struct iovec *iov;
+ int iovcnt;
+
+ v = _mdfd_getseg(reln, forknum, blocknum, false,
+ EXTENSION_FAIL | EXTENSION_CREATE_RECOVERY);
+
+ seekpos = (off_t) BLCKSZ * (blocknum % ((BlockNumber) RELSEG_SIZE));
+
+ Assert(seekpos < (off_t) BLCKSZ * RELSEG_SIZE);
+
+ nblocks_this_segment =
+ Min(nblocks,
+ RELSEG_SIZE - (blocknum % ((BlockNumber) RELSEG_SIZE)));
+
+ if (nblocks_this_segment != nblocks)
+ elog(ERROR, "read crossing segment boundary");
+
+ iovcnt = pgaio_io_get_iovec(ioh, &iov);
+
+ Assert(nblocks <= iovcnt);
+
+ iovcnt = buffers_to_iovec(iov, buffers, nblocks_this_segment);
+
+ Assert(iovcnt <= nblocks_this_segment);
+
+ pgaio_io_set_subject_smgr(ioh,
+ reln,
+ forknum,
+ blocknum,
+ nblocks);
+ pgaio_io_add_shared_cb(ioh, ASC_MD_READV);
+
+ FileStartReadV(ioh, v->mdfd_vfd, iovcnt, seekpos, WAIT_EVENT_DATA_FILE_READ);
+}
+
/*
* mdwritev() -- Write the supplied blocks at the appropriate location.
*
@@ -1036,6 +1080,49 @@ mdwritev(SMgrRelation reln, ForkNumber forknum, BlockNumber blocknum,
}
}
+void
+mdstartwritev(PgAioHandle *ioh,
+ SMgrRelation reln, ForkNumber forknum, BlockNumber blocknum,
+ const void **buffers, BlockNumber nblocks, bool skipFsync)
+{
+ off_t seekpos;
+ MdfdVec *v;
+ BlockNumber nblocks_this_segment;
+ struct iovec *iov;
+ int iovcnt;
+
+ v = _mdfd_getseg(reln, forknum, blocknum, false,
+ EXTENSION_FAIL | EXTENSION_CREATE_RECOVERY);
+
+ seekpos = (off_t) BLCKSZ * (blocknum % ((BlockNumber) RELSEG_SIZE));
+
+ Assert(seekpos < (off_t) BLCKSZ * RELSEG_SIZE);
+
+ nblocks_this_segment =
+ Min(nblocks,
+ RELSEG_SIZE - (blocknum % ((BlockNumber) RELSEG_SIZE)));
+
+ if (nblocks_this_segment != nblocks)
+ elog(ERROR, "write crossing segment boundary");
+
+ iovcnt = pgaio_io_get_iovec(ioh, &iov);
+
+ Assert(nblocks <= iovcnt);
+
+ iovcnt = buffers_to_iovec(iov, unconstify(void **, buffers), nblocks_this_segment);
+
+ Assert(iovcnt <= nblocks_this_segment);
+
+ pgaio_io_set_subject_smgr(ioh,
+ reln,
+ forknum,
+ blocknum,
+ nblocks);
+ pgaio_io_add_shared_cb(ioh, ASC_MD_WRITEV);
+
+ FileStartWriteV(ioh, v->mdfd_vfd, iovcnt, seekpos, WAIT_EVENT_DATA_FILE_WRITE);
+}
+
/*
* mdwriteback() -- Tell the kernel to write pages back to storage.
@@ -1357,6 +1444,21 @@ mdimmedsync(SMgrRelation reln, ForkNumber forknum)
}
}
+int
+mdfd(SMgrRelation reln, ForkNumber forknum, BlockNumber blocknum, uint32 *off)
+{
+ MdfdVec *v = mdopenfork(reln, forknum, EXTENSION_FAIL);
+
+ v = _mdfd_getseg(reln, forknum, blocknum, false,
+ EXTENSION_FAIL);
+
+ *off = (off_t) BLCKSZ * (blocknum % ((BlockNumber) RELSEG_SIZE));
+
+ Assert(*off < (off_t) BLCKSZ * RELSEG_SIZE);
+
+ return FileGetRawDesc(v->mdfd_vfd);
+}
+
/*
* register_dirty_segment() -- Mark a relation segment as needing fsync
*
@@ -1832,3 +1934,118 @@ mdfiletagmatches(const FileTag *ftag, const FileTag *candidate)
*/
return ftag->rlocator.dbOid == candidate->rlocator.dbOid;
}
+
+
+
+static PgAioResult md_readv_complete(PgAioHandle *ioh, PgAioResult prior_result);
+static PgAioResult md_writev_complete(PgAioHandle *ioh, PgAioResult prior_result);
+static void md_readv_error(PgAioResult result, const PgAioSubjectData *subject_data, int elevel);
+
+const struct PgAioHandleSharedCallbacks aio_md_readv_cb = {
+ .complete = md_readv_complete,
+ .error = md_readv_error,
+};
+
+const struct PgAioHandleSharedCallbacks aio_md_writev_cb = {
+ .complete = md_writev_complete,
+};
+
+static PgAioResult
+md_readv_complete(PgAioHandle *ioh, PgAioResult prior_result)
+{
+ PgAioSubjectData *sd = pgaio_io_get_subject_data(ioh);
+ PgAioResult result = prior_result;
+
+ elog(DEBUG3, "%s: %d %d", __func__, prior_result.status, prior_result.result);
+
+ if (prior_result.result < 0)
+ {
+ result.status = ARS_ERROR;
+ result.id = ASC_MD_READV;
+ result.error_data = -prior_result.result;
+ result.result = 0;
+
+ md_readv_error(result, sd, LOG);
+
+ return result;
+ }
+
+ result.result /= BLCKSZ;
+
+ if (result.result == 0)
+ {
+ /* consider 0 blocks read a failure */
+ result.status = ARS_ERROR;
+ result.id = ASC_MD_READV;
+ result.error_data = 0;
+
+ md_readv_error(result, sd, LOG);
+ }
+
+ if (result.status != ARS_ERROR &&
+ result.result < sd->smgr.nblocks)
+ {
+ /* partial reads should be retried at upper level */
+ result.id = ASC_MD_READV;
+ result.status = ARS_PARTIAL;
+ }
+
+ /* AFIXME: post-read portion of mdreadv() */
+
+ return result;
+}
+
+static void
+md_readv_error(PgAioResult result, const PgAioSubjectData *subject_data, int elevel)
+{
+ MemoryContext oldContext = CurrentMemoryContext;
+
+ /* AFIXME: */
+ oldContext = MemoryContextSwitchTo(ErrorContext);
+
+ if (result.error_data != 0)
+ {
+ errno = result.error_data; /* for errcode_for_file_access() */
+
+ ereport(elevel,
+ errcode_for_file_access(),
+ errmsg("could not read blocks %u..%u in file \"%s\": %m",
+ subject_data->smgr.blockNum,
+ subject_data->smgr.blockNum + subject_data->smgr.nblocks,
+ relpathperm(subject_data->smgr.rlocator, subject_data->smgr.forkNum)
+ )
+ );
+ }
+ else
+ {
+ ereport(elevel,
+ errcode(ERRCODE_DATA_CORRUPTED),
+ errmsg("could not read blocks %u..%u in file \"%s\": read only %zu of %zu bytes",
+ subject_data->smgr.blockNum,
+ subject_data->smgr.blockNum + subject_data->smgr.nblocks - 1,
+ relpathperm(subject_data->smgr.rlocator, subject_data->smgr.forkNum),
+ result.result * (size_t) BLCKSZ,
+ subject_data->smgr.nblocks * (size_t) BLCKSZ
+ )
+ );
+ }
+
+ MemoryContextSwitchTo(oldContext);
+}
+
+
+static PgAioResult
+md_writev_complete(PgAioHandle *ioh, PgAioResult prior_result)
+{
+ elog(DEBUG3, "%s: %d %d", __func__, prior_result.status, prior_result.result);
+
+ if (prior_result.status == ARS_ERROR)
+ {
+ /* AFIXME: complain */
+ return prior_result;
+ }
+
+ prior_result.result /= BLCKSZ;
+
+ return prior_result;
+}
diff --git a/src/backend/storage/smgr/smgr.c b/src/backend/storage/smgr/smgr.c
index ee31db85eec..2dacb361a4f 100644
--- a/src/backend/storage/smgr/smgr.c
+++ b/src/backend/storage/smgr/smgr.c
@@ -53,6 +53,7 @@
#include "access/xlogutils.h"
#include "lib/ilist.h"
+#include "storage/aio.h"
#include "storage/bufmgr.h"
#include "storage/ipc.h"
#include "storage/md.h"
@@ -93,10 +94,19 @@ typedef struct f_smgr
void (*smgr_readv) (SMgrRelation reln, ForkNumber forknum,
BlockNumber blocknum,
void **buffers, BlockNumber nblocks);
+ void (*smgr_startreadv) (struct PgAioHandle *ioh,
+ SMgrRelation reln, ForkNumber forknum,
+ BlockNumber blocknum,
+ void **buffers, BlockNumber nblocks);
void (*smgr_writev) (SMgrRelation reln, ForkNumber forknum,
BlockNumber blocknum,
const void **buffers, BlockNumber nblocks,
bool skipFsync);
+ void (*smgr_startwritev) (struct PgAioHandle *ioh,
+ SMgrRelation reln, ForkNumber forknum,
+ BlockNumber blocknum,
+ const void **buffers, BlockNumber nblocks,
+ bool skipFsync);
void (*smgr_writeback) (SMgrRelation reln, ForkNumber forknum,
BlockNumber blocknum, BlockNumber nblocks);
BlockNumber (*smgr_nblocks) (SMgrRelation reln, ForkNumber forknum);
@@ -104,6 +114,7 @@ typedef struct f_smgr
BlockNumber nblocks);
void (*smgr_immedsync) (SMgrRelation reln, ForkNumber forknum);
void (*smgr_registersync) (SMgrRelation reln, ForkNumber forknum);
+ int (*smgr_fd) (SMgrRelation reln, ForkNumber forknum, BlockNumber blocknum, uint32 *off);
} f_smgr;
static const f_smgr smgrsw[] = {
@@ -121,12 +132,15 @@ static const f_smgr smgrsw[] = {
.smgr_prefetch = mdprefetch,
.smgr_maxcombine = mdmaxcombine,
.smgr_readv = mdreadv,
+ .smgr_startreadv = mdstartreadv,
.smgr_writev = mdwritev,
+ .smgr_startwritev = mdstartwritev,
.smgr_writeback = mdwriteback,
.smgr_nblocks = mdnblocks,
.smgr_truncate = mdtruncate,
.smgr_immedsync = mdimmedsync,
.smgr_registersync = mdregistersync,
+ .smgr_fd = mdfd,
}
};
@@ -145,6 +159,14 @@ static void smgrshutdown(int code, Datum arg);
static void smgrdestroy(SMgrRelation reln);
+static void smgr_aio_reopen(PgAioHandle *ioh);
+
+const struct PgAioSubjectInfo aio_smgr_subject_info = {
+ .name = "smgr",
+ .reopen = smgr_aio_reopen,
+};
+
+
/*
* smgrinit(), smgrshutdown() -- Initialize or shut down storage
* managers.
@@ -620,6 +642,19 @@ smgrreadv(SMgrRelation reln, ForkNumber forknum, BlockNumber blocknum,
nblocks);
}
+/*
+ * FILL ME IN
+ */
+void
+smgrstartreadv(struct PgAioHandle *ioh,
+ SMgrRelation reln, ForkNumber forknum, BlockNumber blocknum,
+ void **buffers, BlockNumber nblocks)
+{
+ smgrsw[reln->smgr_which].smgr_startreadv(ioh,
+ reln, forknum, blocknum, buffers,
+ nblocks);
+}
+
/*
* smgrwritev() -- Write the supplied buffers out.
*
@@ -651,6 +686,16 @@ smgrwritev(SMgrRelation reln, ForkNumber forknum, BlockNumber blocknum,
buffers, nblocks, skipFsync);
}
+void
+smgrstartwritev(struct PgAioHandle *ioh,
+ SMgrRelation reln, ForkNumber forknum, BlockNumber blocknum,
+ const void **buffers, BlockNumber nblocks, bool skipFsync)
+{
+ smgrsw[reln->smgr_which].smgr_startwritev(ioh,
+ reln, forknum, blocknum, buffers,
+ nblocks, skipFsync);
+}
+
/*
* smgrwriteback() -- Trigger kernel writeback for the supplied range of
* blocks.
@@ -807,6 +852,12 @@ smgrimmedsync(SMgrRelation reln, ForkNumber forknum)
smgrsw[reln->smgr_which].smgr_immedsync(reln, forknum);
}
+int
+smgrfd(SMgrRelation reln, ForkNumber forknum, BlockNumber blocknum, uint32 *off)
+{
+ return smgrsw[reln->smgr_which].smgr_fd(reln, forknum, blocknum, off);
+}
+
/*
* AtEOXact_SMgr
*
@@ -835,3 +886,43 @@ ProcessBarrierSmgrRelease(void)
smgrreleaseall();
return true;
}
+
+void
+pgaio_io_set_subject_smgr(PgAioHandle *ioh,
+ struct SMgrRelationData *smgr,
+ ForkNumber forknum,
+ BlockNumber blocknum,
+ int nblocks)
+{
+ PgAioSubjectData *sd = pgaio_io_get_subject_data(ioh);
+
+ pgaio_io_set_subject(ioh, ASI_SMGR);
+
+ /* backend is implied via IO owner */
+ sd->smgr.rlocator = smgr->smgr_rlocator.locator;
+ sd->smgr.forkNum = forknum;
+ sd->smgr.blockNum = blocknum;
+ sd->smgr.nblocks = nblocks;
+ sd->smgr.is_temp = SmgrIsTemp(smgr);
+ sd->smgr.release_lock = false;
+ sd->smgr.mode = RBM_NORMAL;
+}
+
+static void
+smgr_aio_reopen(PgAioHandle *ioh)
+{
+ PgAioSubjectData *sd = pgaio_io_get_subject_data(ioh);
+ PgAioOpData *od = pgaio_io_get_op_data(ioh);
+ SMgrRelation reln;
+ ProcNumber procno;
+ uint32 off;
+
+ if (sd->smgr.is_temp)
+ procno = pgaio_io_get_owner(ioh);
+ else
+ procno = INVALID_PROC_NUMBER;
+
+ reln = smgropen(sd->smgr.rlocator, procno);
+ od->read.fd = smgrfd(reln, sd->smgr.forkNum, sd->smgr.blockNum, &off);
+ Assert(off == od->read.offset);
+}
--
2.45.2.827.g557ae147e6
v2.1-0014-bufmgr-Implement-AIO-support.patch (text/x-diff, 20.9K) download
From 34a11207d325b445d15a12e2c63aff4b90a935d8 Mon Sep 17 00:00:00 2001
From: Andres Freund <andres@anarazel.de>
Date: Sat, 31 Aug 2024 21:39:01 -0400
Subject: [PATCH v2.1 14/20] bufmgr: Implement AIO support
As of this commit there are no users of these AIO facilities, that'll come in
later commits.
Author:
Reviewed-By:
Discussion: https://postgr.es/m/
Backpatch:
---
src/include/storage/aio.h | 6 +
src/include/storage/buf_internals.h | 6 +
src/include/storage/bufmgr.h | 10 +
src/backend/storage/aio/aio_subject.c | 5 +
src/backend/storage/buffer/buf_init.c | 3 +
src/backend/storage/buffer/bufmgr.c | 432 +++++++++++++++++++++++++-
src/backend/storage/buffer/localbuf.c | 65 ++++
7 files changed, 520 insertions(+), 7 deletions(-)
diff --git a/src/include/storage/aio.h b/src/include/storage/aio.h
index 07bf92a6b7a..260c3701247 100644
--- a/src/include/storage/aio.h
+++ b/src/include/storage/aio.h
@@ -93,6 +93,12 @@ typedef enum PgAioHandleSharedCallbackID
{
ASC_MD_READV,
ASC_MD_WRITEV,
+
+ ASC_SHARED_BUFFER_READ,
+ ASC_SHARED_BUFFER_WRITE,
+
+ ASC_LOCAL_BUFFER_READ,
+ ASC_LOCAL_BUFFER_WRITE,
} PgAioHandleSharedCallbackID;
diff --git a/src/include/storage/buf_internals.h b/src/include/storage/buf_internals.h
index f190e6e5e46..5cfa7dbd1f1 100644
--- a/src/include/storage/buf_internals.h
+++ b/src/include/storage/buf_internals.h
@@ -17,6 +17,7 @@
#include "pgstat.h"
#include "port/atomics.h"
+#include "storage/aio_ref.h"
#include "storage/buf.h"
#include "storage/bufmgr.h"
#include "storage/condition_variable.h"
@@ -252,6 +253,8 @@ typedef struct BufferDesc
int wait_backend_pgprocno; /* backend of pin-count waiter */
int freeNext; /* link in freelist chain */
+
+ PgAioHandleRef io_in_progress;
LWLock content_lock; /* to lock access to buffer contents */
} BufferDesc;
@@ -465,4 +468,7 @@ extern void DropRelationLocalBuffers(RelFileLocator rlocator,
extern void DropRelationAllLocalBuffers(RelFileLocator rlocator);
extern void AtEOXact_LocalBuffers(bool isCommit);
+
+extern bool ReadBufferCompleteReadLocal(Buffer buffer, int mode, bool failed);
+
#endif /* BUFMGR_INTERNALS_H */
diff --git a/src/include/storage/bufmgr.h b/src/include/storage/bufmgr.h
index eb0fba4230b..6cd64b8c2b3 100644
--- a/src/include/storage/bufmgr.h
+++ b/src/include/storage/bufmgr.h
@@ -177,6 +177,14 @@ extern PGDLLIMPORT int NLocBuffer;
extern PGDLLIMPORT Block *LocalBufferBlockPointers;
extern PGDLLIMPORT int32 *LocalRefCount;
+
+struct PgAioHandleSharedCallbacks;
+extern const struct PgAioHandleSharedCallbacks aio_shared_buffer_read_cb;
+extern const struct PgAioHandleSharedCallbacks aio_shared_buffer_write_cb;
+extern const struct PgAioHandleSharedCallbacks aio_local_buffer_read_cb;
+extern const struct PgAioHandleSharedCallbacks aio_local_buffer_write_cb;
+
+
/* upper limit for effective_io_concurrency */
#define MAX_IO_CONCURRENCY 1000
@@ -194,6 +202,8 @@ extern PGDLLIMPORT int32 *LocalRefCount;
/*
* prototypes for functions in bufmgr.c
*/
+struct PgAioHandle;
+
extern PrefetchBufferResult PrefetchSharedBuffer(struct SMgrRelationData *smgr_reln,
ForkNumber forkNum,
BlockNumber blockNum);
diff --git a/src/backend/storage/aio/aio_subject.c b/src/backend/storage/aio/aio_subject.c
index 14be8432f5a..07c7989b273 100644
--- a/src/backend/storage/aio/aio_subject.c
+++ b/src/backend/storage/aio/aio_subject.c
@@ -35,6 +35,11 @@ static const PgAioSubjectInfo *aio_subject_info[] = {
static const PgAioHandleSharedCallbacks *aio_shared_cbs[] = {
[ASC_MD_READV] = &aio_md_readv_cb,
[ASC_MD_WRITEV] = &aio_md_writev_cb,
+
+ [ASC_SHARED_BUFFER_READ] = &aio_shared_buffer_read_cb,
+ [ASC_SHARED_BUFFER_WRITE] = &aio_shared_buffer_write_cb,
+ [ASC_LOCAL_BUFFER_READ] = &aio_local_buffer_read_cb,
+ [ASC_LOCAL_BUFFER_WRITE] = &aio_local_buffer_write_cb,
};
diff --git a/src/backend/storage/buffer/buf_init.c b/src/backend/storage/buffer/buf_init.c
index 09bec6449b6..059a80dfb13 100644
--- a/src/backend/storage/buffer/buf_init.c
+++ b/src/backend/storage/buffer/buf_init.c
@@ -14,6 +14,7 @@
*/
#include "postgres.h"
+#include "storage/aio.h"
#include "storage/buf_internals.h"
#include "storage/bufmgr.h"
#include "storage/proc.h"
@@ -126,6 +127,8 @@ BufferManagerShmemInit(void)
buf->buf_id = i;
+ pgaio_io_ref_clear(&buf->io_in_progress);
+
/*
* Initially link all the buffers together as unused. Subsequent
* management of this list is done by freelist.c.
diff --git a/src/backend/storage/buffer/bufmgr.c b/src/backend/storage/buffer/bufmgr.c
index 7e987836335..976ced82b6a 100644
--- a/src/backend/storage/buffer/bufmgr.c
+++ b/src/backend/storage/buffer/bufmgr.c
@@ -48,6 +48,7 @@
#include "pg_trace.h"
#include "pgstat.h"
#include "postmaster/bgwriter.h"
+#include "storage/aio.h"
#include "storage/buf_internals.h"
#include "storage/bufmgr.h"
#include "storage/fd.h"
@@ -58,6 +59,7 @@
#include "storage/smgr.h"
#include "storage/standby.h"
#include "utils/memdebug.h"
+#include "utils/memutils.h"
#include "utils/ps_status.h"
#include "utils/rel.h"
#include "utils/resowner.h"
@@ -514,7 +516,8 @@ static int SyncOneBuffer(int buf_id, bool skip_recently_used,
static void WaitIO(BufferDesc *buf);
static bool StartBufferIO(BufferDesc *buf, bool forInput, bool nowait);
static void TerminateBufferIO(BufferDesc *buf, bool clear_dirty,
- uint32 set_flag_bits, bool forget_owner);
+ uint32 set_flag_bits, bool forget_owner,
+ bool syncio);
static void AbortBufferIO(Buffer buffer);
static void shared_buffer_write_error_callback(void *arg);
static void local_buffer_write_error_callback(void *arg);
@@ -1081,7 +1084,7 @@ ZeroAndLockBuffer(Buffer buffer, ReadBufferMode mode, bool already_valid)
else
{
/* Set BM_VALID, terminate IO, and wake up any waiters */
- TerminateBufferIO(bufHdr, false, BM_VALID, true);
+ TerminateBufferIO(bufHdr, false, BM_VALID, true, true);
}
}
else if (!isLocalBuf)
@@ -1566,7 +1569,7 @@ WaitReadBuffers(ReadBuffersOperation *operation)
else
{
/* Set BM_VALID, terminate IO, and wake up any waiters */
- TerminateBufferIO(bufHdr, false, BM_VALID, true);
+ TerminateBufferIO(bufHdr, false, BM_VALID, true, true);
}
/* Report I/Os as completing individually. */
@@ -2450,7 +2453,7 @@ ExtendBufferedRelShared(BufferManagerRelation bmr,
if (lock)
LWLockAcquire(BufferDescriptorGetContentLock(buf_hdr), LW_EXCLUSIVE);
- TerminateBufferIO(buf_hdr, false, BM_VALID, true);
+ TerminateBufferIO(buf_hdr, false, BM_VALID, true, true);
}
pgBufferUsage.shared_blks_written += extend_by;
@@ -3899,7 +3902,7 @@ FlushBuffer(BufferDesc *buf, SMgrRelation reln, IOObject io_object,
* Mark the buffer as clean (unless BM_JUST_DIRTIED has become set) and
* end the BM_IO_IN_PROGRESS state.
*/
- TerminateBufferIO(buf, true, 0, true);
+ TerminateBufferIO(buf, true, 0, true, true);
TRACE_POSTGRESQL_BUFFER_FLUSH_DONE(BufTagGetForkNum(&buf->tag),
buf->tag.blockNum,
@@ -5514,6 +5517,7 @@ WaitIO(BufferDesc *buf)
for (;;)
{
uint32 buf_state;
+ PgAioHandleRef ior;
/*
* It may not be necessary to acquire the spinlock to check the flag
@@ -5521,10 +5525,19 @@ WaitIO(BufferDesc *buf)
* play it safe.
*/
buf_state = LockBufHdr(buf);
+ ior = buf->io_in_progress;
UnlockBufHdr(buf, buf_state);
if (!(buf_state & BM_IO_IN_PROGRESS))
break;
+
+ if (pgaio_io_ref_valid(&ior))
+ {
+ pgaio_io_ref_wait(&ior);
+ ConditionVariablePrepareToSleep(cv);
+ continue;
+ }
+
ConditionVariableSleep(cv, WAIT_EVENT_BUFFER_IO);
}
ConditionVariableCancelSleep();
@@ -5613,7 +5626,7 @@ StartBufferIO(BufferDesc *buf, bool forInput, bool nowait)
*/
static void
TerminateBufferIO(BufferDesc *buf, bool clear_dirty, uint32 set_flag_bits,
- bool forget_owner)
+ bool forget_owner, bool syncio)
{
uint32 buf_state;
@@ -5625,6 +5638,13 @@ TerminateBufferIO(BufferDesc *buf, bool clear_dirty, uint32 set_flag_bits,
if (clear_dirty && !(buf_state & BM_JUST_DIRTIED))
buf_state &= ~(BM_DIRTY | BM_CHECKPOINT_NEEDED);
+ if (!syncio)
+ {
+ /* release ownership by the AIO subsystem */
+ buf_state -= BUF_REFCOUNT_ONE;
+ pgaio_io_ref_clear(&buf->io_in_progress);
+ }
+
buf_state |= set_flag_bits;
UnlockBufHdr(buf, buf_state);
@@ -5633,6 +5653,40 @@ TerminateBufferIO(BufferDesc *buf, bool clear_dirty, uint32 set_flag_bits,
BufferDescriptorGetBuffer(buf));
ConditionVariableBroadcast(BufferDescriptorGetIOCV(buf));
+
+ /*
+ * If we just released a pin, need to do BM_PIN_COUNT_WAITER handling.
+ * Most of the time the current backend will hold another pin preventing
+ * that from happening, but that's e.g. not the case when completing an IO
+ * another backend started.
+ *
+ * AFIXME: Deduplicate with UnpinBufferNoOwner() or just replace
+ * BM_PIN_COUNT_WAITER with something saner.
+ */
+ /* Support LockBufferForCleanup() */
+ if (buf_state & BM_PIN_COUNT_WAITER)
+ {
+ /*
+ * Acquire the buffer header lock, re-check that there's a waiter.
+ * Another backend could have unpinned this buffer, and already woken
+ * up the waiter. There's no danger of the buffer being replaced
+ * after we unpinned it above, as it's pinned by the waiter.
+ */
+ buf_state = LockBufHdr(buf);
+
+ if ((buf_state & BM_PIN_COUNT_WAITER) &&
+ BUF_STATE_GET_REFCOUNT(buf_state) == 1)
+ {
+ /* we just released the last pin other than the waiter's */
+ int wait_backend_pgprocno = buf->wait_backend_pgprocno;
+
+ buf_state &= ~BM_PIN_COUNT_WAITER;
+ UnlockBufHdr(buf, buf_state);
+ ProcSendSignal(wait_backend_pgprocno);
+ }
+ else
+ UnlockBufHdr(buf, buf_state);
+ }
}
/*
@@ -5684,7 +5738,7 @@ AbortBufferIO(Buffer buffer)
}
}
- TerminateBufferIO(buf_hdr, false, BM_IO_ERROR, false);
+ TerminateBufferIO(buf_hdr, false, BM_IO_ERROR, false, true);
}
/*
@@ -6143,3 +6197,367 @@ EvictUnpinnedBuffer(Buffer buf)
return result;
}
+
+static bool
+ReadBufferCompleteReadShared(Buffer buffer, int mode, bool failed)
+{
+ BufferDesc *bufHdr = NULL;
+ BlockNumber blockno;
+ bool buf_failed = false;
+ char *bufdata = BufferGetBlock(buffer);
+
+ Assert(BufferIsValid(buffer));
+
+ bufHdr = GetBufferDescriptor(buffer - 1);
+ blockno = bufHdr->tag.blockNum;
+
+#ifdef USE_ASSERT_CHECKING
+ {
+ uint32 buf_state = pg_atomic_read_u32(&bufHdr->state);
+
+ Assert(buf_state & BM_TAG_VALID);
+ Assert(!(buf_state & BM_VALID));
+ Assert(buf_state & BM_IO_IN_PROGRESS);
+ Assert(!(buf_state & BM_DIRTY));
+ }
+#endif
+
+ /* check for garbage data */
+ if (!failed &&
+ !PageIsVerifiedExtended((Page) bufdata, blockno,
+ PIV_LOG_WARNING | PIV_REPORT_STAT))
+ {
+ RelFileLocator rlocator = BufTagGetRelFileLocator(&bufHdr->tag);
+ BlockNumber forkNum = bufHdr->tag.forkNum;
+
+ /* AFIXME: relpathperm allocates memory */
+ MemoryContextSwitchTo(ErrorContext);
+ if (mode == READ_BUFFERS_ZERO_ON_ERROR || zero_damaged_pages)
+ {
+ ereport(LOG,
+ (errcode(ERRCODE_DATA_CORRUPTED),
+ errmsg("invalid page in block %u of relation %s; zeroing out page",
+ blockno,
+ relpathperm(rlocator, forkNum))));
+ memset(bufdata, 0, BLCKSZ);
+ }
+ else
+ {
+ ereport(LOG,
+ (errcode(ERRCODE_DATA_CORRUPTED),
+ errmsg("invalid page in block %u of relation %s",
+ blockno,
+ relpathperm(rlocator, forkNum))));
+ failed = true;
+ buf_failed = true;
+ }
+ }
+
+ /* Terminate I/O and set BM_VALID. */
+ TerminateBufferIO(bufHdr, false,
+ failed ? BM_IO_ERROR : BM_VALID,
+ false, false);
+
+ /* Report I/Os as completing individually. */
+
+ /* FIXME: Should we do TRACE_POSTGRESQL_BUFFER_READ_DONE here? */
+ return buf_failed;
+}
+
+static uint64
+ReadBufferCompleteWriteShared(Buffer buffer, bool release_lock, bool failed)
+{
+ BufferDesc *bufHdr;
+ bool result = false;
+
+ Assert(BufferIsValid(buffer));
+
+ bufHdr = GetBufferDescriptor(buffer - 1);
+
+#ifdef USE_ASSERT_CHECKING
+ {
+ uint32 buf_state = pg_atomic_read_u32(&bufHdr->state);
+
+ Assert(buf_state & BM_VALID);
+ Assert(buf_state & BM_TAG_VALID);
+ Assert(buf_state & BM_IO_IN_PROGRESS);
+ Assert(buf_state & BM_DIRTY);
+ }
+#endif
+
+ /* AFIXME: implement track_io_timing */
+
+ TerminateBufferIO(bufHdr, /* clear_dirty = */ true,
+ failed ? BM_IO_ERROR : 0,
+ /* forget_owner = */ false,
+ /* syncio = */ false);
+
+ /*
+ * The initiator of IO is not managing the lock (i.e. called
+ * LWLockDisown()), we are.
+ */
+ if (release_lock)
+ LWLockReleaseUnowned(BufferDescriptorGetContentLock(bufHdr), LW_SHARED);
+
+ return result;
+}
+
+static void
+shared_buffer_prepare_common(PgAioHandle *ioh, bool is_write)
+{
+ uint64 *io_data;
+ uint8 io_data_len;
+ PgAioHandleRef io_ref;
+ BufferTag first PG_USED_FOR_ASSERTS_ONLY = {0};
+
+ io_data = pgaio_io_get_io_data(ioh, &io_data_len);
+
+ pgaio_io_get_ref(ioh, &io_ref);
+
+ for (int i = 0; i < io_data_len; i++)
+ {
+ Buffer buf = (Buffer) io_data[i];
+ BufferDesc *bufHdr;
+ uint32 buf_state;
+
+ bufHdr = GetBufferDescriptor(buf - 1);
+
+ if (i == 0)
+ first = bufHdr->tag;
+ else
+ {
+ Assert(bufHdr->tag.relNumber == first.relNumber);
+ Assert(bufHdr->tag.blockNum == first.blockNum + i);
+ }
+
+
+ buf_state = LockBufHdr(bufHdr);
+
+ Assert(buf_state & BM_TAG_VALID);
+ if (is_write)
+ {
+ Assert(buf_state & BM_VALID);
+ Assert(buf_state & BM_DIRTY);
+ }
+ else
+ Assert(!(buf_state & BM_VALID));
+
+ Assert(buf_state & BM_IO_IN_PROGRESS);
+ Assert(BUF_STATE_GET_REFCOUNT(buf_state) >= 1);
+
+ buf_state += BUF_REFCOUNT_ONE;
+ bufHdr->io_in_progress = io_ref;
+
+ UnlockBufHdr(bufHdr, buf_state);
+
+ if (is_write)
+ {
+ LWLock *content_lock;
+
+ content_lock = BufferDescriptorGetContentLock(bufHdr);
+
+ Assert(LWLockHeldByMe(content_lock));
+
+ /*
+ * Lock is now owned by IO.
+ */
+ LWLockDisown(content_lock);
+ RESUME_INTERRUPTS();
+ }
+
+ /*
+ * Stop tracking this buffer via the resowner - the AIO system now
+ * keeps track.
+ */
+ ResourceOwnerForgetBufferIO(CurrentResourceOwner, buf);
+ }
+}
+
+static void
+shared_buffer_read_prepare(PgAioHandle *ioh)
+{
+ shared_buffer_prepare_common(ioh, false);
+}
+
+static void
+shared_buffer_write_prepare(PgAioHandle *ioh)
+{
+ shared_buffer_prepare_common(ioh, true);
+}
+
+
+static PgAioResult
+shared_buffer_read_complete(PgAioHandle *ioh, PgAioResult prior_result)
+{
+ PgAioResult result = prior_result;
+ int mode = pgaio_io_get_subject_data(ioh)->smgr.mode;
+ uint64 *io_data;
+ uint8 io_data_len;
+
+ elog(DEBUG3, "%s: %d %d", __func__, prior_result.status, prior_result.result);
+
+ io_data = pgaio_io_get_io_data(ioh, &io_data_len);
+
+ for (int io_data_off = 0; io_data_off < io_data_len; io_data_off++)
+ {
+ Buffer buf = io_data[io_data_off];
+ bool buf_failed;
+ bool failed;
+
+ failed =
+ prior_result.status == ARS_ERROR
+ || prior_result.result <= io_data_off;
+
+ elog(DEBUG3, "calling rbcrs for buf %d with failed %d, error: %d, result: %d, data_off: %d",
+ buf, failed, prior_result.status, prior_result.result, io_data_off);
+
+ /*
+ * AFIXME: It'd probably be better to not set BM_IO_ERROR (which is
+ * what failed = true leads to) when it's just a short read...
+ */
+ buf_failed = ReadBufferCompleteReadShared(buf,
+ mode,
+ failed);
+
+ if (result.status != ARS_ERROR && buf_failed)
+ {
+ result.status = ARS_ERROR;
+ result.id = ASC_SHARED_BUFFER_READ;
+ result.error_data = io_data_off + 1;
+ }
+ }
+
+ return result;
+}
+
+static void
+shared_buffer_read_error(PgAioResult result, const PgAioSubjectData *subject_data, int elevel)
+{
+ MemoryContext oldContext = CurrentMemoryContext;
+
+ /* AFIXME: */
+ oldContext = MemoryContextSwitchTo(ErrorContext);
+
+ ereport(elevel,
+ errcode(ERRCODE_DATA_CORRUPTED),
+ errmsg("invalid page in block %u of relation %s",
+ subject_data->smgr.blockNum + result.error_data,
+ relpathperm(subject_data->smgr.rlocator, subject_data->smgr.forkNum)
+ )
+ );
+ MemoryContextSwitchTo(oldContext);
+}
+
+static PgAioResult
+shared_buffer_write_complete(PgAioHandle *ioh, PgAioResult prior_result)
+{
+ PgAioResult result = prior_result;
+ uint64 *io_data;
+ uint8 io_data_len;
+
+ elog(DEBUG3, "%s: %d %d", __func__, prior_result.status, prior_result.result);
+
+ io_data = pgaio_io_get_io_data(ioh, &io_data_len);
+
+ /* FIXME: handle outright errors */
+
+ for (int io_data_off = 0; io_data_off < io_data_len; io_data_off++)
+ {
+ Buffer buf = io_data[io_data_off];
+
+ /* FIXME: handle short writes / failures */
+ /* FIXME: ioh->scb_data.shared_buffer.release_lock */
+ ReadBufferCompleteWriteShared(buf,
+ true,
+ false);
+
+ }
+
+ return result;
+}
+
+static void
+local_buffer_read_prepare(PgAioHandle *ioh)
+{
+ uint64 *io_data;
+ uint8 io_data_len;
+ PgAioHandleRef io_ref;
+
+ io_data = pgaio_io_get_io_data(ioh, &io_data_len);
+
+ pgaio_io_get_ref(ioh, &io_ref);
+
+ for (int i = 0; i < io_data_len; i++)
+ {
+ Buffer buf = (Buffer) io_data[i];
+ BufferDesc *bufHdr;
+ uint32 buf_state;
+
+ bufHdr = GetLocalBufferDescriptor(-buf - 1);
+
+ buf_state = pg_atomic_read_u32(&bufHdr->state);
+
+ bufHdr->io_in_progress = io_ref;
+ LocalRefCount[-buf - 1] += 1;
+
+ UnlockBufHdr(bufHdr, buf_state);
+ }
+}
+
+static PgAioResult
+local_buffer_read_complete(PgAioHandle *ioh, PgAioResult prior_result)
+{
+ PgAioResult result = prior_result;
+ int mode = pgaio_io_get_subject_data(ioh)->smgr.mode;
+ uint64 *io_data;
+ uint8 io_data_len;
+
+ elog(DEBUG3, "%s: %d %d", __func__, prior_result.status, prior_result.result);
+
+ io_data = pgaio_io_get_io_data(ioh, &io_data_len);
+
+ /* FIXME: error handling */
+
+ for (int io_data_off = 0; io_data_off < io_data_len; io_data_off++)
+ {
+ Buffer buf = io_data[io_data_off];
+ bool buf_failed;
+
+ buf_failed = ReadBufferCompleteReadLocal(buf,
+ mode,
+ false);
+
+ if (result.status != ARS_ERROR && buf_failed)
+ {
+ result.status = ARS_ERROR;
+ result.id = ASC_LOCAL_BUFFER_READ;
+ result.error_data = io_data_off + 1;
+ }
+ }
+
+ return result;
+}
+
+static void
+local_buffer_write_prepare(PgAioHandle *ioh)
+{
+ elog(ERROR, "not yet");
+}
+
+
+const struct PgAioHandleSharedCallbacks aio_shared_buffer_read_cb = {
+ .prepare = shared_buffer_read_prepare,
+ .complete = shared_buffer_read_complete,
+ .error = shared_buffer_read_error,
+};
+const struct PgAioHandleSharedCallbacks aio_shared_buffer_write_cb = {
+ .prepare = shared_buffer_write_prepare,
+ .complete = shared_buffer_write_complete,
+};
+const struct PgAioHandleSharedCallbacks aio_local_buffer_read_cb = {
+ .prepare = local_buffer_read_prepare,
+ .complete = local_buffer_read_complete,
+};
+const struct PgAioHandleSharedCallbacks aio_local_buffer_write_cb = {
+ .prepare = local_buffer_write_prepare,
+};
diff --git a/src/backend/storage/buffer/localbuf.c b/src/backend/storage/buffer/localbuf.c
index 8da7dd6c98a..a7eb723f1e9 100644
--- a/src/backend/storage/buffer/localbuf.c
+++ b/src/backend/storage/buffer/localbuf.c
@@ -18,6 +18,7 @@
#include "access/parallel.h"
#include "executor/instrument.h"
#include "pgstat.h"
+#include "storage/aio.h"
#include "storage/buf_internals.h"
#include "storage/bufmgr.h"
#include "storage/fd.h"
@@ -620,6 +621,8 @@ InitLocalBuffers(void)
*/
buf->buf_id = -i - 2;
+ pgaio_io_ref_clear(&buf->io_in_progress);
+
/*
* Intentionally do not initialize the buffer's atomic variable
* (besides zeroing the underlying memory above). That way we get
@@ -836,3 +839,65 @@ AtProcExit_LocalBuffers(void)
*/
CheckForLocalBufferLeaks();
}
+
+bool
+ReadBufferCompleteReadLocal(Buffer buffer, int mode, bool failed)
+{
+ BufferDesc *buf_hdr = NULL;
+ BlockNumber blockno;
+ bool buf_failed = false;
+ char *bufdata = BufferGetBlock(buffer);
+
+ Assert(BufferIsValid(buffer));
+
+ buf_hdr = GetLocalBufferDescriptor(-buffer - 1);
+ blockno = buf_hdr->tag.blockNum;
+
+ /* check for garbage data */
+ if (!failed &&
+ !PageIsVerifiedExtended((Page) bufdata, blockno,
+ PIV_LOG_WARNING | PIV_REPORT_STAT))
+ {
+ RelFileLocator rlocator = BufTagGetRelFileLocator(&buf_hdr->tag);
+ BlockNumber forkNum = buf_hdr->tag.forkNum;
+
+ MemoryContextSwitchTo(ErrorContext);
+
+ if (mode == READ_BUFFERS_ZERO_ON_ERROR || zero_damaged_pages)
+ {
+
+ ereport(WARNING,
+ (errcode(ERRCODE_DATA_CORRUPTED),
+ errmsg("invalid page in block %u of relation %s; zeroing out page",
+ blockno,
+ relpathperm(rlocator, forkNum))));
+ memset(bufdata, 0, BLCKSZ);
+ }
+ else
+ {
+ ereport(LOG,
+ (errcode(ERRCODE_DATA_CORRUPTED),
+ errmsg("invalid page in block %u of relation %s",
+ blockno,
+ relpathperm(rlocator, forkNum))));
+ failed = true;
+ buf_failed = true;
+ }
+ }
+
+ /* Terminate I/O and set BM_VALID. */
+ pgaio_io_ref_clear(&buf_hdr->io_in_progress);
+
+ {
+ uint32 buf_state;
+
+ buf_state = pg_atomic_read_u32(&buf_hdr->state);
+ buf_state |= BM_VALID;
+ pg_atomic_unlocked_write_u32(&buf_hdr->state, buf_state);
+ }
+
+ /* release pin held by IO subsystem */
+ LocalRefCount[-buffer - 1] -= 1;
+
+ return buf_failed;
+}
--
2.45.2.827.g557ae147e6
v2.1-0015-bufmgr-Use-aio-for-StartReadBuffers.patch (text/x-diff, 14.0K) download
From bfd939b88a8dcdbc424c1e7452d70195a46910ae Mon Sep 17 00:00:00 2001
From: Andres Freund <andres@anarazel.de>
Date: Sat, 31 Aug 2024 21:55:59 -0400
Subject: [PATCH v2.1 15/20] bufmgr: Use aio for StartReadBuffers()
Author:
Reviewed-By:
Discussion: https://postgr.es/m/
Backpatch:
---
src/include/storage/bufmgr.h | 25 ++-
src/backend/storage/buffer/bufmgr.c | 259 +++++++++++++++++-----------
2 files changed, 182 insertions(+), 102 deletions(-)
diff --git a/src/include/storage/bufmgr.h b/src/include/storage/bufmgr.h
index 6cd64b8c2b3..a075a40b2ed 100644
--- a/src/include/storage/bufmgr.h
+++ b/src/include/storage/bufmgr.h
@@ -15,6 +15,7 @@
#define BUFMGR_H
#include "port/pg_iovec.h"
+#include "storage/aio_ref.h"
#include "storage/block.h"
#include "storage/buf.h"
#include "storage/bufpage.h"
@@ -107,11 +108,22 @@ typedef struct BufferManagerRelation
#define BMR_REL(p_rel) ((BufferManagerRelation){.rel = p_rel})
#define BMR_SMGR(p_smgr, p_relpersistence) ((BufferManagerRelation){.smgr = p_smgr, .relpersistence = p_relpersistence})
+
+#define MAX_IO_COMBINE_LIMIT PG_IOV_MAX
+#define DEFAULT_IO_COMBINE_LIMIT Min(MAX_IO_COMBINE_LIMIT, (128 * 1024) / BLCKSZ)
+
+
/* Zero out page if reading fails. */
#define READ_BUFFERS_ZERO_ON_ERROR (1 << 0)
/* Call smgrprefetch() if I/O necessary. */
#define READ_BUFFERS_ISSUE_ADVICE (1 << 1)
+/*
+ * FIXME: PgAioReturn is defined in aio.h. It'd be much better if we didn't
+ * need to include that here. Perhaps this could live in a separate header?
+ */
+#include "storage/aio.h"
+
struct ReadBuffersOperation
{
/* The following members should be set by the caller. */
@@ -131,6 +143,17 @@ struct ReadBuffersOperation
int flags;
int16 nblocks;
int16 io_buffers_len;
+
+ /*
+ * In some rare-ish cases one operation causes multiple reads (e.g. if a
+ * buffer was concurrently read by another backend). It'd be much better
+ * if we ensured that each ReadBuffersOperation covered only one IO - but
+ * that's not entirely trivial, due to having pinned victim buffers before
+ * starting IOs.
+ */
+ int16 nios;
+ PgAioHandleRef refs[MAX_IO_COMBINE_LIMIT];
+ PgAioReturn returns[MAX_IO_COMBINE_LIMIT];
};
typedef struct ReadBuffersOperation ReadBuffersOperation;
@@ -161,8 +184,6 @@ extern PGDLLIMPORT bool track_io_timing;
extern PGDLLIMPORT int effective_io_concurrency;
extern PGDLLIMPORT int maintenance_io_concurrency;
-#define MAX_IO_COMBINE_LIMIT PG_IOV_MAX
-#define DEFAULT_IO_COMBINE_LIMIT Min(MAX_IO_COMBINE_LIMIT, (128 * 1024) / BLCKSZ)
extern PGDLLIMPORT int io_combine_limit;
extern PGDLLIMPORT int checkpoint_flush_after;
diff --git a/src/backend/storage/buffer/bufmgr.c b/src/backend/storage/buffer/bufmgr.c
index 976ced82b6a..4914c71d41e 100644
--- a/src/backend/storage/buffer/bufmgr.c
+++ b/src/backend/storage/buffer/bufmgr.c
@@ -1253,6 +1253,12 @@ ReadBuffer_common(Relation rel, SMgrRelation smgr, char smgr_persistence,
return buffer;
}
+static bool AsyncReadBuffers(ReadBuffersOperation *operation,
+ Buffer *buffers,
+ BlockNumber blockNum,
+ int *nblocks,
+ int flags);
+
static pg_attribute_always_inline bool
StartReadBuffersImpl(ReadBuffersOperation *operation,
Buffer *buffers,
@@ -1288,6 +1294,12 @@ StartReadBuffersImpl(ReadBuffersOperation *operation,
* so we stop here.
*/
actual_nblocks = i + 1;
+
+ ereport(DEBUG3,
+ errmsg("found buf %d, idx %i: %s, data %p",
+ buffers[i], i, DebugPrintBufferRefcount(buffers[i]),
+ BufferGetBlock(buffers[i])),
+ errhidestmt(true), errhidecontext(true));
break;
}
else
@@ -1325,27 +1337,18 @@ StartReadBuffersImpl(ReadBuffersOperation *operation,
operation->nblocks = actual_nblocks;
operation->io_buffers_len = io_buffers_len;
- if (flags & READ_BUFFERS_ISSUE_ADVICE)
- {
- /*
- * In theory we should only do this if PinBufferForBlock() had to
- * allocate new buffers above. That way, if two calls to
- * StartReadBuffers() were made for the same blocks before
- * WaitReadBuffers(), only the first would issue the advice. That'd be
- * a better simulation of true asynchronous I/O, which would only
- * start the I/O once, but isn't done here for simplicity. Note also
- * that the following call might actually issue two advice calls if we
- * cross a segment boundary; in a true asynchronous version we might
- * choose to process only one real I/O at a time in that case.
- */
- smgrprefetch(operation->smgr,
- operation->forknum,
- blockNum,
- operation->io_buffers_len);
- }
+ operation->nios = 0;
- /* Indicate that WaitReadBuffers() should be called. */
- return true;
+ /*
+ * TODO: When called for synchronous IO execution, we probably should
+ * enter a dedicated fastpath here.
+ */
+
+ /* initiate the IO */
+ return AsyncReadBuffers(operation,
+ buffers,
+ blockNum,
+ nblocks, flags);
}
/*
@@ -1397,12 +1400,31 @@ StartReadBuffer(ReadBuffersOperation *operation,
}
static inline bool
-WaitReadBuffersCanStartIO(Buffer buffer, bool nowait)
+ReadBuffersCanStartIO(Buffer buffer, bool nowait)
{
if (BufferIsLocal(buffer))
{
BufferDesc *bufHdr = GetLocalBufferDescriptor(-buffer - 1);
+ /*
+ * The buffer could have IO in progress by another scan. Right now
+ * localbuf.c doesn't use IO_IN_PROGRESS, which is why we need this
+ * hack.
+ *
+ * AFIXME: localbuf.c should use IO_IN_PROGRESS / have an equivalent
+ * of StartBufferIO().
+ */
+ if (pgaio_io_ref_valid(&bufHdr->io_in_progress))
+ {
+ PgAioHandleRef ior = bufHdr->io_in_progress;
+
+ ereport(DEBUG3,
+ errmsg("waiting for temp buffer IO in CSIO"),
+ errhidestmt(true), errhidecontext(true));
+ pgaio_io_ref_wait(&ior);
+ return false;
+ }
+
return (pg_atomic_read_u32(&bufHdr->state) & BM_VALID) == 0;
}
else
@@ -1412,12 +1434,7 @@ WaitReadBuffersCanStartIO(Buffer buffer, bool nowait)
void
WaitReadBuffers(ReadBuffersOperation *operation)
{
- Buffer *buffers;
int nblocks;
- BlockNumber blocknum;
- ForkNumber forknum;
- IOContext io_context;
- IOObject io_object;
char persistence;
/*
@@ -1433,11 +1450,65 @@ WaitReadBuffers(ReadBuffersOperation *operation)
if (nblocks == 0)
return; /* nothing to do */
+ persistence = operation->persistence;
+
+ Assert(operation->nios > 0);
+
+ for (int i = 0; i < operation->nios; i++)
+ {
+ PgAioReturn *aio_ret;
+
+ pgaio_io_ref_wait(&operation->refs[i]);
+
+ aio_ret = &operation->returns[i];
+
+ if (aio_ret->result.status != ARS_OK)
+ pgaio_result_log(aio_ret->result, &aio_ret->subject_data, ERROR);
+ }
+
+ /*
+ * We count all these blocks as read by this backend. This is traditional
+ * behavior, but might turn out to be not true if we find that someone
+ * else has beaten us and completed the read of some of these blocks. In
+ * that case the system globally double-counts, but we traditionally don't
+ * count this as a "hit", and we don't have a separate counter for "miss,
+ * but another backend completed the read".
+ */
+ if (persistence == RELPERSISTENCE_TEMP)
+ pgBufferUsage.local_blks_read += nblocks;
+ else
+ pgBufferUsage.shared_blks_read += nblocks;
+
+ if (VacuumCostActive)
+ VacuumCostBalance += VacuumCostPageMiss * nblocks;
+
+ /* FIXME: io timing */
+ /* FIXME: READ_DONE tracepoint */
+}
+
+static bool
+AsyncReadBuffers(ReadBuffersOperation *operation,
+ Buffer *buffers,
+ BlockNumber blockNum,
+ int *nblocks,
+ int flags)
+{
+ int io_buffers_len = 0;
+ BlockNumber blocknum;
+ ForkNumber forknum;
+ IOContext io_context;
+ IOObject io_object;
+ char persistence;
+ bool did_start_io_overall = false;
+ PgAioHandle *ioh = NULL;
+
buffers = &operation->buffers[0];
blocknum = operation->blocknum;
forknum = operation->forknum;
- persistence = operation->persistence;
+ persistence = operation->rel
+ ? operation->rel->rd_rel->relpersistence
+ : RELPERSISTENCE_PERMANENT;
if (persistence == RELPERSISTENCE_TEMP)
{
io_context = IOCONTEXT_NORMAL;
@@ -1458,25 +1529,33 @@ WaitReadBuffers(ReadBuffersOperation *operation)
* but another backend completed the read".
*/
if (persistence == RELPERSISTENCE_TEMP)
- pgBufferUsage.local_blks_read += nblocks;
+ pgBufferUsage.local_blks_read += *nblocks;
else
- pgBufferUsage.shared_blks_read += nblocks;
+ pgBufferUsage.shared_blks_read += *nblocks;
- for (int i = 0; i < nblocks; ++i)
+ for (int i = 0; i < *nblocks; ++i)
{
- int io_buffers_len;
- Buffer io_buffers[MAX_IO_COMBINE_LIMIT];
void *io_pages[MAX_IO_COMBINE_LIMIT];
- instr_time io_start;
+ Buffer io_buffers[MAX_IO_COMBINE_LIMIT];
BlockNumber io_first_block;
+ bool did_start_io_this = false;
+
+ /*
+ * Get IO before ReadBuffersCanStartIO, as pgaio_io_get() might block,
+ * which we don't want after setting IO_IN_PROGRESS.
+ */
+ if (likely(!ioh))
+ ioh = pgaio_io_get(CurrentResourceOwner, &operation->returns[operation->nios]);
/*
* Skip this block if someone else has already completed it. If an
* I/O is already in progress in another backend, this will wait for
* the outcome: either done, or something went wrong and we will
* retry.
+ *
+ * ATODO: Should we wait if we already submitted another IO?
*/
- if (!WaitReadBuffersCanStartIO(buffers[i], false))
+ if (!ReadBuffersCanStartIO(buffers[i], did_start_io_overall))
{
/*
* Report this as a 'hit' for this backend, even though it must
@@ -1488,6 +1567,10 @@ WaitReadBuffers(ReadBuffersOperation *operation)
operation->smgr->smgr_rlocator.locator.relNumber,
operation->smgr->smgr_rlocator.backend,
true);
+
+ ereport(DEBUG3,
+ errmsg("can't start io for first buffer %u", buffers[i]),
+ errhidestmt(true), errhidecontext(true));
continue;
}
@@ -1497,6 +1580,11 @@ WaitReadBuffers(ReadBuffersOperation *operation)
io_first_block = blocknum + i;
io_buffers_len = 1;
+ ereport(DEBUG3,
+ errmsg("first prepped for io: %s, offset %d",
+ DebugPrintBufferRefcount(io_buffers[0]), i),
+ errhidestmt(true), errhidecontext(true));
+
/*
* How many neighboring-on-disk blocks can we can scatter-read into
* other buffers at the same time? In this case we don't wait if we
@@ -1504,86 +1592,57 @@ WaitReadBuffers(ReadBuffersOperation *operation)
* for the head block, so we should get on with that I/O as soon as
* possible. We'll come back to this block again, above.
*/
- while ((i + 1) < nblocks &&
- WaitReadBuffersCanStartIO(buffers[i + 1], true))
+ while ((i + 1) < *nblocks &&
+ ReadBuffersCanStartIO(buffers[i + 1], true))
{
/* Must be consecutive block numbers. */
Assert(BufferGetBlockNumber(buffers[i + 1]) ==
BufferGetBlockNumber(buffers[i]) + 1);
+ ereport(DEBUG3,
+ errmsg("seq prepped for io: %s, offset %d",
+ DebugPrintBufferRefcount(buffers[i + 1]),
+ i + 1),
+ errhidestmt(true), errhidecontext(true));
+
io_buffers[io_buffers_len] = buffers[++i];
io_pages[io_buffers_len++] = BufferGetBlock(buffers[i]);
}
- io_start = pgstat_prepare_io_time(track_io_timing);
- smgrreadv(operation->smgr, forknum, io_first_block, io_pages, io_buffers_len);
- pgstat_count_io_op_time(io_object, io_context, IOOP_READ, io_start,
- io_buffers_len);
+ pgaio_io_get_ref(ioh, &operation->refs[operation->nios]);
- /* Verify each block we read, and terminate the I/O. */
- for (int j = 0; j < io_buffers_len; ++j)
+ pgaio_io_set_io_data_32(ioh, (uint32 *) io_buffers, io_buffers_len);
+ if (persistence == RELPERSISTENCE_TEMP)
{
- BufferDesc *bufHdr;
- Block bufBlock;
-
- if (persistence == RELPERSISTENCE_TEMP)
- {
- bufHdr = GetLocalBufferDescriptor(-io_buffers[j] - 1);
- bufBlock = LocalBufHdrGetBlock(bufHdr);
- }
- else
- {
- bufHdr = GetBufferDescriptor(io_buffers[j] - 1);
- bufBlock = BufHdrGetBlock(bufHdr);
- }
-
- /* check for garbage data */
- if (!PageIsVerifiedExtended((Page) bufBlock, io_first_block + j,
- PIV_LOG_WARNING | PIV_REPORT_STAT))
- {
- if ((operation->flags & READ_BUFFERS_ZERO_ON_ERROR) || zero_damaged_pages)
- {
- ereport(WARNING,
- (errcode(ERRCODE_DATA_CORRUPTED),
- errmsg("invalid page in block %u of relation %s; zeroing out page",
- io_first_block + j,
- relpath(operation->smgr->smgr_rlocator, forknum))));
- memset(bufBlock, 0, BLCKSZ);
- }
- else
- ereport(ERROR,
- (errcode(ERRCODE_DATA_CORRUPTED),
- errmsg("invalid page in block %u of relation %s",
- io_first_block + j,
- relpath(operation->smgr->smgr_rlocator, forknum))));
- }
-
- /* Terminate I/O and set BM_VALID. */
- if (persistence == RELPERSISTENCE_TEMP)
- {
- uint32 buf_state = pg_atomic_read_u32(&bufHdr->state);
-
- buf_state |= BM_VALID;
- pg_atomic_unlocked_write_u32(&bufHdr->state, buf_state);
- }
- else
- {
- /* Set BM_VALID, terminate IO, and wake up any waiters */
- TerminateBufferIO(bufHdr, false, BM_VALID, true, true);
- }
-
- /* Report I/Os as completing individually. */
- TRACE_POSTGRESQL_BUFFER_READ_DONE(forknum, io_first_block + j,
- operation->smgr->smgr_rlocator.locator.spcOid,
- operation->smgr->smgr_rlocator.locator.dbOid,
- operation->smgr->smgr_rlocator.locator.relNumber,
- operation->smgr->smgr_rlocator.backend,
- false);
+ pgaio_io_add_shared_cb(ioh, ASC_LOCAL_BUFFER_READ);
+ pgaio_io_set_flag(ioh, AHF_REFERENCES_LOCAL);
}
+ else
+ pgaio_io_add_shared_cb(ioh, ASC_SHARED_BUFFER_READ);
- if (VacuumCostActive)
- VacuumCostBalance += VacuumCostPageMiss * io_buffers_len;
+ did_start_io_overall = did_start_io_this = true;
+ smgrstartreadv(ioh, operation->smgr, forknum, io_first_block,
+ io_pages, io_buffers_len);
+ ioh = NULL;
+ operation->nios++;
+
+ /* not obvious what we'd use for time */
+ pgstat_count_io_op_n(io_object, io_context, IOOP_READ, io_buffers_len);
}
+
+ if (ioh)
+ {
+ pgaio_io_release(ioh);
+ ioh = NULL;
+ }
+
+ if (did_start_io_overall)
+ {
+ pgaio_submit_staged();
+ return true;
+ }
+ else
+ return false;
}
/*
--
2.45.2.827.g557ae147e6
v2.1-0016-aio-Very-WIP-read_stream.c-adjustments-for-real.patch (text/x-diff, 4.5K) download
From 3b51bfa51eac42157c8177437fb6993ed349c0f3 Mon Sep 17 00:00:00 2001
From: Andres Freund <andres@anarazel.de>
Date: Sat, 31 Aug 2024 21:39:30 -0400
Subject: [PATCH v2.1 16/20] aio: Very-WIP: read_stream.c adjustments for real
AIO
---
src/include/storage/bufmgr.h | 2 ++
src/backend/storage/aio/read_stream.c | 29 +++++++++++++++++++++------
src/backend/storage/buffer/bufmgr.c | 3 ++-
3 files changed, 27 insertions(+), 7 deletions(-)
diff --git a/src/include/storage/bufmgr.h b/src/include/storage/bufmgr.h
index a075a40b2ed..ac6496bb1eb 100644
--- a/src/include/storage/bufmgr.h
+++ b/src/include/storage/bufmgr.h
@@ -117,6 +117,8 @@ typedef struct BufferManagerRelation
#define READ_BUFFERS_ZERO_ON_ERROR (1 << 0)
/* Call smgrprefetch() if I/O necessary. */
#define READ_BUFFERS_ISSUE_ADVICE (1 << 1)
+/* caller will issue more io, don't submit */
+#define READ_BUFFERS_MORE_MORE_MORE (1 << 2)
/*
* FIXME: PgAioReturn is defined in aio.h. It'd be much better if we didn't
diff --git a/src/backend/storage/aio/read_stream.c b/src/backend/storage/aio/read_stream.c
index 7f0e07d9586..7ff2d6a2071 100644
--- a/src/backend/storage/aio/read_stream.c
+++ b/src/backend/storage/aio/read_stream.c
@@ -91,6 +91,7 @@
#include "catalog/pg_tablespace.h"
#include "miscadmin.h"
+#include "storage/aio.h"
#include "storage/fd.h"
#include "storage/smgr.h"
#include "storage/read_stream.h"
@@ -241,14 +242,18 @@ read_stream_start_pending_read(ReadStream *stream, bool suppress_advice)
/*
* If advice hasn't been suppressed, this system supports it, and this
* isn't a strictly sequential pattern, then we'll issue advice.
+ *
+ * XXX: Used to also check stream->pending_read_blocknum !=
+ * stream->seq_blocknum
*/
if (!suppress_advice &&
- stream->advice_enabled &&
- stream->pending_read_blocknum != stream->seq_blocknum)
+ stream->advice_enabled)
flags = READ_BUFFERS_ISSUE_ADVICE;
else
flags = 0;
+ flags |= READ_BUFFERS_MORE_MORE_MORE;
+
/* We say how many blocks we want to read, but may be smaller on return. */
buffer_index = stream->next_buffer_index;
io_index = stream->next_io_index;
@@ -307,6 +312,14 @@ read_stream_start_pending_read(ReadStream *stream, bool suppress_advice)
static void
read_stream_look_ahead(ReadStream *stream, bool suppress_advice)
{
+ if (stream->distance > (io_combine_limit * 8))
+ {
+ if (stream->pinned_buffers + stream->pending_read_nblocks > ((stream->distance * 3) / 4))
+ {
+ return;
+ }
+ }
+
while (stream->ios_in_progress < stream->max_ios &&
stream->pinned_buffers + stream->pending_read_nblocks < stream->distance)
{
@@ -356,6 +369,7 @@ read_stream_look_ahead(ReadStream *stream, bool suppress_advice)
{
/* And we've hit the limit. Rewind, and stop here. */
read_stream_unget_block(stream, blocknum);
+ pgaio_submit_staged();
return;
}
}
@@ -380,6 +394,8 @@ read_stream_look_ahead(ReadStream *stream, bool suppress_advice)
stream->distance == 0) &&
stream->ios_in_progress < stream->max_ios)
read_stream_start_pending_read(stream, suppress_advice);
+
+ pgaio_submit_staged();
}
/*
@@ -494,10 +510,11 @@ read_stream_begin_impl(int flags,
* direct I/O isn't enabled, the caller hasn't promised sequential access
* (overriding our detection heuristics), and max_ios hasn't been set to
* zero.
+ *
+ * FIXME: Used to also check (io_direct_flags & IO_DIRECT_DATA) == 0 &&
+ * (flags & READ_STREAM_SEQUENTIAL) == 0
*/
- if ((io_direct_flags & IO_DIRECT_DATA) == 0 &&
- (flags & READ_STREAM_SEQUENTIAL) == 0 &&
- max_ios > 0)
+ if (max_ios > 0)
stream->advice_enabled = true;
#endif
@@ -728,7 +745,7 @@ read_stream_next_buffer(ReadStream *stream, void **per_buffer_data)
if (++stream->oldest_io_index == stream->max_ios)
stream->oldest_io_index = 0;
- if (stream->ios[io_index].op.flags & READ_BUFFERS_ISSUE_ADVICE)
+ if (stream->ios[io_index].op.flags & (READ_BUFFERS_ISSUE_ADVICE | READ_BUFFERS_MORE_MORE_MORE))
{
/* Distance ramps up fast (behavior C). */
distance = stream->distance * 2;
diff --git a/src/backend/storage/buffer/bufmgr.c b/src/backend/storage/buffer/bufmgr.c
index 4914c71d41e..ed384fa1a44 100644
--- a/src/backend/storage/buffer/bufmgr.c
+++ b/src/backend/storage/buffer/bufmgr.c
@@ -1638,7 +1638,8 @@ AsyncReadBuffers(ReadBuffersOperation *operation,
if (did_start_io_overall)
{
- pgaio_submit_staged();
+ if (!(flags & READ_BUFFERS_MORE_MORE_MORE))
+ pgaio_submit_staged();
return true;
}
else
--
2.45.2.827.g557ae147e6
v2.1-0017-aio-Add-IO-queue-helper.patch (text/x-diff, 7.2K) download
From 2d390f78e46219c8bace6d37ff35d20f6ff0fd30 Mon Sep 17 00:00:00 2001
From: Andres Freund <andres@anarazel.de>
Date: Wed, 4 Sep 2024 16:15:42 -0400
Subject: [PATCH v2.1 17/20] aio: Add IO queue helper
This is likely never going to anywhere - Thomas Munro is working on something
more complete. But I needed a way to exercise aio for checkpointer / bgwriter.
---
src/include/storage/io_queue.h | 33 +++++
src/backend/storage/aio/Makefile | 1 +
src/backend/storage/aio/io_queue.c | 195 ++++++++++++++++++++++++++++
src/backend/storage/aio/meson.build | 1 +
src/tools/pgindent/typedefs.list | 2 +
5 files changed, 232 insertions(+)
create mode 100644 src/include/storage/io_queue.h
create mode 100644 src/backend/storage/aio/io_queue.c
diff --git a/src/include/storage/io_queue.h b/src/include/storage/io_queue.h
new file mode 100644
index 00000000000..28077158d6d
--- /dev/null
+++ b/src/include/storage/io_queue.h
@@ -0,0 +1,33 @@
+/*-------------------------------------------------------------------------
+ *
+ * io_queue.h
+ * Mechanism for tracking many IOs
+ *
+ *
+ * Portions Copyright (c) 1996-2024, PostgreSQL Global Development Group
+ * Portions Copyright (c) 1994, Regents of the University of California
+ *
+ * src/include/storage/io_queue.h
+ *
+ *-------------------------------------------------------------------------
+ */
+#ifndef IO_QUEUE_H
+#define IO_QUEUE_H
+
+#include "storage/bufmgr.h"
+
+struct IOQueue;
+typedef struct IOQueue IOQueue;
+
+struct PgAioHandleRef;
+
+extern IOQueue *io_queue_create(int depth, int flags);
+extern void io_queue_track(IOQueue *ioq, const struct PgAioHandleRef *ior);
+extern void io_queue_wait_one(IOQueue *ioq);
+extern void io_queue_wait_all(IOQueue *ioq);
+extern bool io_queue_is_empty(IOQueue *ioq);
+extern void io_queue_reserve(IOQueue *ioq);
+extern struct PgAioHandle *io_queue_get_io(IOQueue *ioq);
+extern void io_queue_free(IOQueue *ioq);
+
+#endif /* IO_QUEUE_H */
diff --git a/src/backend/storage/aio/Makefile b/src/backend/storage/aio/Makefile
index 3bcb8a0b2ed..f3a7f9e63d6 100644
--- a/src/backend/storage/aio/Makefile
+++ b/src/backend/storage/aio/Makefile
@@ -13,6 +13,7 @@ OBJS = \
aio_init.o \
aio_io.o \
aio_subject.o \
+ io_queue.o \
method_io_uring.o \
method_sync.o \
method_worker.o \
diff --git a/src/backend/storage/aio/io_queue.c b/src/backend/storage/aio/io_queue.c
new file mode 100644
index 00000000000..4dda2f4e20e
--- /dev/null
+++ b/src/backend/storage/aio/io_queue.c
@@ -0,0 +1,195 @@
+/*-------------------------------------------------------------------------
+ *
+ * io_queue.c
+ * Mechanism for tracking many IOs
+ *
+ * Portions Copyright (c) 2024, PostgreSQL Global Development Group
+ * Portions Copyright (c) 1994, Regents of the University of California
+ *
+ * IDENTIFICATION
+ * src/backend/storage/aio/io_queue.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#include "storage/io_queue.h"
+
+#include "storage/aio.h"
+
+
+typedef struct TrackedIO
+{
+ PgAioHandleRef ior;
+ dlist_node node;
+} TrackedIO;
+
+struct IOQueue
+{
+ int depth;
+ int unsubmitted;
+
+ bool has_reserved;
+
+ dclist_head idle;
+ dclist_head in_progress;
+
+ TrackedIO tracked_ios[FLEXIBLE_ARRAY_MEMBER];
+};
+
+
+IOQueue *
+io_queue_create(int depth, int flags)
+{
+ size_t sz;
+ IOQueue *ioq;
+
+ sz = offsetof(IOQueue, tracked_ios)
+ + sizeof(TrackedIO) * depth;
+
+ ioq = palloc0(sz);
+
+ ioq->depth = 0;
+
+ for (int i = 0; i < depth; i++)
+ {
+ TrackedIO *tio = &ioq->tracked_ios[i];
+
+ pgaio_io_ref_clear(&tio->ior);
+ dclist_push_tail(&ioq->idle, &tio->node);
+ }
+
+ return ioq;
+}
+
+void
+io_queue_wait_one(IOQueue *ioq)
+{
+ while (!dclist_is_empty(&ioq->in_progress))
+ {
+ /* FIXME: Should we really pop here already? */
+ dlist_node *node = dclist_pop_head_node(&ioq->in_progress);
+ TrackedIO *tio = dclist_container(TrackedIO, node, node);
+
+ pgaio_io_ref_wait(&tio->ior);
+ dclist_push_head(&ioq->idle, &tio->node);
+ }
+}
+
+void
+io_queue_reserve(IOQueue *ioq)
+{
+ if (ioq->has_reserved)
+ return;
+
+ if (dclist_is_empty(&ioq->idle))
+ io_queue_wait_one(ioq);
+
+ Assert(!dclist_is_empty(&ioq->idle));
+
+ ioq->has_reserved = true;
+}
+
+PgAioHandle *
+io_queue_get_io(IOQueue *ioq)
+{
+ PgAioHandle *ioh;
+
+ io_queue_reserve(ioq);
+
+ Assert(!dclist_is_empty(&ioq->idle));
+
+ if (!io_queue_is_empty(ioq))
+ {
+ ioh = pgaio_io_get_nb(CurrentResourceOwner, NULL);
+ if (ioh == NULL)
+ {
+ /*
+ * Need to wait for all IOs, blocking might not be legal in the
+ * context.
+ *
+ * XXX: This doesn't make a whole lot of sense, we're also
+ * blocking here. What was I smoking when I wrote the above?
+ */
+ io_queue_wait_all(ioq);
+ ioh = pgaio_io_get(CurrentResourceOwner, NULL);
+ }
+ }
+ else
+ {
+ ioh = pgaio_io_get(CurrentResourceOwner, NULL);
+ }
+
+ return ioh;
+}
+
+void
+io_queue_track(IOQueue *ioq, const struct PgAioHandleRef *ior)
+{
+ dlist_node *node;
+ TrackedIO *tio;
+
+ Assert(ioq->has_reserved);
+ ioq->has_reserved = false;
+
+ Assert(!dclist_is_empty(&ioq->idle));
+
+ node = dclist_pop_head_node(&ioq->idle);
+ tio = dclist_container(TrackedIO, node, node);
+
+ tio->ior = *ior;
+
+ dclist_push_tail(&ioq->in_progress, &tio->node);
+
+ ioq->unsubmitted++;
+
+ /*
+ * XXX: Should have some smarter logic here. We don't want to wait too
+ * long to submit, that'll mean we're more likely to block. But we also
+ * don't want to have the overhead of submitting every IO individually.
+ */
+ if (ioq->unsubmitted >= 4)
+ {
+ pgaio_submit_staged();
+ ioq->unsubmitted = 0;
+ }
+}
+
+void
+io_queue_wait_all(IOQueue *ioq)
+{
+ while (!dclist_is_empty(&ioq->in_progress))
+ {
+ /* wait for the last IO to minimize unnecessary wakeups */
+ dlist_node *node = dclist_tail_node(&ioq->in_progress);
+ TrackedIO *tio = dclist_container(TrackedIO, node, node);
+
+ if (!pgaio_io_ref_check_done(&tio->ior))
+ {
+ ereport(DEBUG3,
+ errmsg("io_queue_wait_all for io:%d",
+ pgaio_io_ref_get_id(&tio->ior)),
+ errhidestmt(true),
+ errhidecontext(true));
+
+ pgaio_io_ref_wait(&tio->ior);
+ }
+
+ dclist_delete_from(&ioq->in_progress, &tio->node);
+ dclist_push_head(&ioq->idle, &tio->node);
+ }
+}
+
+bool
+io_queue_is_empty(IOQueue *ioq)
+{
+ return dclist_is_empty(&ioq->in_progress);
+}
+
+void
+io_queue_free(IOQueue *ioq)
+{
+ io_queue_wait_all(ioq);
+
+ pfree(ioq);
+}
diff --git a/src/backend/storage/aio/meson.build b/src/backend/storage/aio/meson.build
index 537f23d446d..e8a88e615c0 100644
--- a/src/backend/storage/aio/meson.build
+++ b/src/backend/storage/aio/meson.build
@@ -5,6 +5,7 @@ backend_sources += files(
'aio_init.c',
'aio_io.c',
'aio_subject.c',
+ 'io_queue.c',
'method_io_uring.c',
'method_sync.c',
'method_worker.c',
diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list
index 46d31cf2b9f..a38141b4e50 100644
--- a/src/tools/pgindent/typedefs.list
+++ b/src/tools/pgindent/typedefs.list
@@ -1172,6 +1172,7 @@ IOContext
IOFuncSelector
IOObject
IOOp
+IOQueue
IO_STATUS_BLOCK
IPCompareMethod
ITEM
@@ -2960,6 +2961,7 @@ TocEntry
TokenAuxData
TokenizedAuthLine
TrackItem
+TrackedIO
TransApplyAction
TransInvalidationInfo
TransState
--
2.45.2.827.g557ae147e6
v2.1-0018-bufmgr-use-AIO-in-checkpointer-bgwriter.patch (text/x-diff, 31.2K) download
From 52aab8396a446a90e23178fd0c593fddfa433a7a Mon Sep 17 00:00:00 2001
From: Andres Freund <andres@anarazel.de>
Date: Tue, 23 Jul 2024 10:01:23 -0700
Subject: [PATCH v2.1 18/20] bufmgr: use AIO in checkpointer, bgwriter
This is far from ready - just included to be able to exercise AIO writes and
get some preliminary numbers. In all likelihood this will instead be based
ontop of work by Thomas Munro instead of the preceding commit.
---
src/include/postmaster/bgwriter.h | 3 +-
src/include/storage/buf_internals.h | 1 +
src/include/storage/bufmgr.h | 3 +-
src/include/storage/bufpage.h | 1 +
src/backend/postmaster/bgwriter.c | 25 +-
src/backend/postmaster/checkpointer.c | 12 +-
src/backend/storage/buffer/bufmgr.c | 588 +++++++++++++++++++++++---
src/backend/storage/page/bufpage.c | 10 +
src/tools/pgindent/typedefs.list | 1 +
9 files changed, 580 insertions(+), 64 deletions(-)
diff --git a/src/include/postmaster/bgwriter.h b/src/include/postmaster/bgwriter.h
index 407f26e5302..01a936fbc0a 100644
--- a/src/include/postmaster/bgwriter.h
+++ b/src/include/postmaster/bgwriter.h
@@ -31,7 +31,8 @@ extern void BackgroundWriterMain(char *startup_data, size_t startup_data_len) pg
extern void CheckpointerMain(char *startup_data, size_t startup_data_len) pg_attribute_noreturn();
extern void RequestCheckpoint(int flags);
-extern void CheckpointWriteDelay(int flags, double progress);
+struct IOQueue;
+extern void CheckpointWriteDelay(struct IOQueue *ioq, int flags, double progress);
extern bool ForwardSyncRequest(const FileTag *ftag, SyncRequestType type);
diff --git a/src/include/storage/buf_internals.h b/src/include/storage/buf_internals.h
index 5cfa7dbd1f1..9d3123663b3 100644
--- a/src/include/storage/buf_internals.h
+++ b/src/include/storage/buf_internals.h
@@ -21,6 +21,7 @@
#include "storage/buf.h"
#include "storage/bufmgr.h"
#include "storage/condition_variable.h"
+#include "storage/io_queue.h"
#include "storage/latch.h"
#include "storage/lwlock.h"
#include "storage/shmem.h"
diff --git a/src/include/storage/bufmgr.h b/src/include/storage/bufmgr.h
index ac6496bb1eb..a65888c8915 100644
--- a/src/include/storage/bufmgr.h
+++ b/src/include/storage/bufmgr.h
@@ -325,7 +325,8 @@ extern bool ConditionalLockBufferForCleanup(Buffer buffer);
extern bool IsBufferCleanupOK(Buffer buffer);
extern bool HoldingBufferPinThatDelaysRecovery(void);
-extern bool BgBufferSync(struct WritebackContext *wb_context);
+struct IOQueue;
+extern bool BgBufferSync(struct IOQueue *ioq, struct WritebackContext *wb_context);
extern void LimitAdditionalPins(uint32 *additional_pins);
extern void LimitAdditionalLocalPins(uint32 *additional_pins);
diff --git a/src/include/storage/bufpage.h b/src/include/storage/bufpage.h
index 6222d46e535..6f8fe796da3 100644
--- a/src/include/storage/bufpage.h
+++ b/src/include/storage/bufpage.h
@@ -509,5 +509,6 @@ extern bool PageIndexTupleOverwrite(Page page, OffsetNumber offnum,
Item newtup, Size newsize);
extern char *PageSetChecksumCopy(Page page, BlockNumber blkno);
extern void PageSetChecksumInplace(Page page, BlockNumber blkno);
+extern bool PageNeedsChecksumCopy(Page page);
#endif /* BUFPAGE_H */
diff --git a/src/backend/postmaster/bgwriter.c b/src/backend/postmaster/bgwriter.c
index 0f75548759a..71c08da45db 100644
--- a/src/backend/postmaster/bgwriter.c
+++ b/src/backend/postmaster/bgwriter.c
@@ -38,10 +38,12 @@
#include "postmaster/auxprocess.h"
#include "postmaster/bgwriter.h"
#include "postmaster/interrupt.h"
+#include "storage/aio.h"
#include "storage/buf_internals.h"
#include "storage/bufmgr.h"
#include "storage/condition_variable.h"
#include "storage/fd.h"
+#include "storage/io_queue.h"
#include "storage/lwlock.h"
#include "storage/proc.h"
#include "storage/procsignal.h"
@@ -89,6 +91,7 @@ BackgroundWriterMain(char *startup_data, size_t startup_data_len)
sigjmp_buf local_sigjmp_buf;
MemoryContext bgwriter_context;
bool prev_hibernate;
+ IOQueue *ioq;
WritebackContext wb_context;
Assert(startup_data_len == 0);
@@ -130,6 +133,7 @@ BackgroundWriterMain(char *startup_data, size_t startup_data_len)
ALLOCSET_DEFAULT_SIZES);
MemoryContextSwitchTo(bgwriter_context);
+ ioq = io_queue_create(128, 0);
WritebackContextInit(&wb_context, &bgwriter_flush_after);
/*
@@ -167,6 +171,7 @@ BackgroundWriterMain(char *startup_data, size_t startup_data_len)
* about in bgwriter, but we do have LWLocks, buffers, and temp files.
*/
LWLockReleaseAll();
+ pgaio_at_error();
ConditionVariableCancelSleep();
UnlockBuffers();
ReleaseAuxProcessResources(false);
@@ -226,12 +231,27 @@ BackgroundWriterMain(char *startup_data, size_t startup_data_len)
/* Clear any already-pending wakeups */
ResetLatch(MyLatch);
+ /*
+ * XXX: Before exiting, wait for all IO to finish. That's only
+ * important to avoid spurious PrintBufferLeakWarning() /
+ * PrintAioIPLeakWarning() calls, triggered by
+ * ReleaseAuxProcessResources() being called with isCommit=true.
+ *
+ * FIXME: this is theoretically racy, but I didn't want to copy
+ * HandleMainLoopInterrupts() remaining body here.
+ */
+ if (ShutdownRequestPending)
+ {
+ io_queue_wait_all(ioq);
+ io_queue_free(ioq);
+ }
+
HandleMainLoopInterrupts();
/*
* Do one cycle of dirty-buffer writing.
*/
- can_hibernate = BgBufferSync(&wb_context);
+ can_hibernate = BgBufferSync(ioq, &wb_context);
/* Report pending statistics to the cumulative stats system */
pgstat_report_bgwriter();
@@ -248,6 +268,9 @@ BackgroundWriterMain(char *startup_data, size_t startup_data_len)
smgrdestroyall();
}
+ /* finish IO before sleeping, to avoid blocking other backends */
+ io_queue_wait_all(ioq);
+
/*
* Log a new xl_running_xacts every now and then so replication can
* get into a consistent state faster (think of suboverflowed
diff --git a/src/backend/postmaster/checkpointer.c b/src/backend/postmaster/checkpointer.c
index eeb73c85726..17aa980aa80 100644
--- a/src/backend/postmaster/checkpointer.c
+++ b/src/backend/postmaster/checkpointer.c
@@ -46,9 +46,11 @@
#include "postmaster/bgwriter.h"
#include "postmaster/interrupt.h"
#include "replication/syncrep.h"
+#include "storage/aio.h"
#include "storage/bufmgr.h"
#include "storage/condition_variable.h"
#include "storage/fd.h"
+#include "storage/io_queue.h"
#include "storage/ipc.h"
#include "storage/lwlock.h"
#include "storage/proc.h"
@@ -266,6 +268,7 @@ CheckpointerMain(char *startup_data, size_t startup_data_len)
* files.
*/
LWLockReleaseAll();
+ pgaio_at_error();
ConditionVariableCancelSleep();
pgstat_report_wait_end();
UnlockBuffers();
@@ -708,7 +711,7 @@ ImmediateCheckpointRequested(void)
* fraction between 0.0 meaning none, and 1.0 meaning all done.
*/
void
-CheckpointWriteDelay(int flags, double progress)
+CheckpointWriteDelay(IOQueue *ioq, int flags, double progress)
{
static int absorb_counter = WRITES_PER_ABSORB;
@@ -741,6 +744,13 @@ CheckpointWriteDelay(int flags, double progress)
/* Report interim statistics to the cumulative stats system */
pgstat_report_checkpointer();
+ /*
+ * Ensure all pending IO is submitted to avoid unnecessary delays for
+ * other processes.
+ */
+ io_queue_wait_all(ioq);
+
+
/*
* This sleep used to be connected to bgwriter_delay, typically 200ms.
* That resulted in more frequent wakeups if not much work to do.
diff --git a/src/backend/storage/buffer/bufmgr.c b/src/backend/storage/buffer/bufmgr.c
index ed384fa1a44..6ec700e5ef2 100644
--- a/src/backend/storage/buffer/bufmgr.c
+++ b/src/backend/storage/buffer/bufmgr.c
@@ -52,6 +52,7 @@
#include "storage/buf_internals.h"
#include "storage/bufmgr.h"
#include "storage/fd.h"
+#include "storage/io_queue.h"
#include "storage/ipc.h"
#include "storage/lmgr.h"
#include "storage/proc.h"
@@ -77,6 +78,7 @@
/* Bits in SyncOneBuffer's return value */
#define BUF_WRITTEN 0x01
#define BUF_REUSABLE 0x02
+#define BUF_CANT_MERGE 0x04
#define RELS_BSEARCH_THRESHOLD 20
@@ -511,8 +513,6 @@ static void UnpinBuffer(BufferDesc *buf);
static void UnpinBufferNoOwner(BufferDesc *buf);
static void BufferSync(int flags);
static uint32 WaitBufHdrUnlocked(BufferDesc *buf);
-static int SyncOneBuffer(int buf_id, bool skip_recently_used,
- WritebackContext *wb_context);
static void WaitIO(BufferDesc *buf);
static bool StartBufferIO(BufferDesc *buf, bool forInput, bool nowait);
static void TerminateBufferIO(BufferDesc *buf, bool clear_dirty,
@@ -530,6 +530,7 @@ static inline BufferDesc *BufferAlloc(SMgrRelation smgr,
static Buffer GetVictimBuffer(BufferAccessStrategy strategy, IOContext io_context);
static void FlushBuffer(BufferDesc *buf, SMgrRelation reln,
IOObject io_object, IOContext io_context);
+
static void FindAndDropRelationBuffers(RelFileLocator rlocator,
ForkNumber forkNum,
BlockNumber nForkBlock,
@@ -2954,6 +2955,56 @@ UnpinBufferNoOwner(BufferDesc *buf)
}
}
+typedef struct BuffersToWrite
+{
+ int nbuffers;
+ BufferTag start_at_tag;
+ uint32 max_combine;
+
+ XLogRecPtr max_lsn;
+
+ PgAioHandle *ioh;
+ PgAioHandleRef ior;
+
+ uint64 total_writes;
+
+ Buffer buffers[IOV_MAX];
+ PgAioBounceBuffer *bounce_buffers[IOV_MAX];
+ const void *data_ptrs[IOV_MAX];
+} BuffersToWrite;
+
+static int PrepareToWriteBuffer(BuffersToWrite *to_write, Buffer buf,
+ bool skip_recently_used,
+ IOQueue *ioq, WritebackContext *wb_context);
+
+static void WriteBuffers(BuffersToWrite *to_write,
+ IOQueue *ioq, WritebackContext *wb_context);
+
+static void
+BuffersToWriteInit(BuffersToWrite *to_write,
+ IOQueue *ioq, WritebackContext *wb_context)
+{
+ to_write->total_writes = 0;
+ to_write->nbuffers = 0;
+ to_write->ioh = NULL;
+ pgaio_io_ref_clear(&to_write->ior);
+ to_write->max_lsn = InvalidXLogRecPtr;
+}
+
+static void
+BuffersToWriteEnd(BuffersToWrite *to_write)
+{
+ if (to_write->ioh != NULL)
+ {
+ pgaio_io_release(to_write->ioh);
+ to_write->ioh = NULL;
+ }
+
+ if (to_write->total_writes > 0)
+ pgaio_submit_staged();
+}
+
+
#define ST_SORT sort_checkpoint_bufferids
#define ST_ELEMENT_TYPE CkptSortItem
#define ST_COMPARE(a, b) ckpt_buforder_comparator(a, b)
@@ -2985,7 +3036,10 @@ BufferSync(int flags)
binaryheap *ts_heap;
int i;
int mask = BM_DIRTY;
+ IOQueue *ioq;
WritebackContext wb_context;
+ BuffersToWrite to_write;
+ int max_combine;
/*
* Unless this is a shutdown checkpoint or we have been explicitly told,
@@ -3047,7 +3101,9 @@ BufferSync(int flags)
if (num_to_scan == 0)
return; /* nothing to do */
+ ioq = io_queue_create(512, 0);
WritebackContextInit(&wb_context, &checkpoint_flush_after);
+ max_combine = Min(io_bounce_buffers, io_combine_limit);
TRACE_POSTGRESQL_BUFFER_SYNC_START(NBuffers, num_to_scan);
@@ -3155,48 +3211,89 @@ BufferSync(int flags)
*/
num_processed = 0;
num_written = 0;
+
+ BuffersToWriteInit(&to_write, ioq, &wb_context);
+
while (!binaryheap_empty(ts_heap))
{
BufferDesc *bufHdr = NULL;
CkptTsStatus *ts_stat = (CkptTsStatus *)
DatumGetPointer(binaryheap_first(ts_heap));
- buf_id = CkptBufferIds[ts_stat->index].buf_id;
- Assert(buf_id != -1);
-
- bufHdr = GetBufferDescriptor(buf_id);
-
- num_processed++;
+ Assert(ts_stat->num_scanned <= ts_stat->num_to_scan);
/*
- * We don't need to acquire the lock here, because we're only looking
- * at a single bit. It's possible that someone else writes the buffer
- * and clears the flag right after we check, but that doesn't matter
- * since SyncOneBuffer will then do nothing. However, there is a
- * further race condition: it's conceivable that between the time we
- * examine the bit here and the time SyncOneBuffer acquires the lock,
- * someone else not only wrote the buffer but replaced it with another
- * page and dirtied it. In that improbable case, SyncOneBuffer will
- * write the buffer though we didn't need to. It doesn't seem worth
- * guarding against this, though.
+ * Collect a batch of buffers to write out from the current
+ * tablespace. That causes some imbalance between the tablespaces, but
+ * that's more than outweighed by the efficiency gain due to batching.
*/
- if (pg_atomic_read_u32(&bufHdr->state) & BM_CHECKPOINT_NEEDED)
+ while (to_write.nbuffers < max_combine &&
+ ts_stat->num_scanned < ts_stat->num_to_scan)
{
- if (SyncOneBuffer(buf_id, false, &wb_context) & BUF_WRITTEN)
+ buf_id = CkptBufferIds[ts_stat->index].buf_id;
+ Assert(buf_id != -1);
+
+ bufHdr = GetBufferDescriptor(buf_id);
+
+ num_processed++;
+
+ /*
+ * We don't need to acquire the lock here, because we're only
+ * looking at a single bit. It's possible that someone else writes
+ * the buffer and clears the flag right after we check, but that
+ * doesn't matter since SyncOneBuffer will then do nothing.
+ * However, there is a further race condition: it's conceivable
+ * that between the time we examine the bit here and the time
+ * SyncOneBuffer acquires the lock, someone else not only wrote
+ * the buffer but replaced it with another page and dirtied it. In
+ * that improbable case, SyncOneBuffer will write the buffer
+ * though we didn't need to. It doesn't seem worth guarding
+ * against this, though.
+ */
+ if (pg_atomic_read_u32(&bufHdr->state) & BM_CHECKPOINT_NEEDED)
{
- TRACE_POSTGRESQL_BUFFER_SYNC_WRITTEN(buf_id);
- PendingCheckpointerStats.buffers_written++;
- num_written++;
+ int result = PrepareToWriteBuffer(&to_write, buf_id + 1, false,
+ ioq, &wb_context);
+
+ if (result & BUF_CANT_MERGE)
+ {
+ Assert(to_write.nbuffers > 0);
+ WriteBuffers(&to_write, ioq, &wb_context);
+
+ result = PrepareToWriteBuffer(&to_write, buf_id + 1, false,
+ ioq, &wb_context);
+ Assert(result != BUF_CANT_MERGE);
+ }
+
+ if (result & BUF_WRITTEN)
+ {
+ TRACE_POSTGRESQL_BUFFER_SYNC_WRITTEN(buf_id);
+ PendingCheckpointerStats.buffers_written++;
+ num_written++;
+ }
+ else
+ {
+ break;
+ }
}
+ else
+ {
+ if (to_write.nbuffers > 0)
+ WriteBuffers(&to_write, ioq, &wb_context);
+ }
+
+ /*
+ * Measure progress independent of actually having to flush the
+ * buffer - otherwise writing become unbalanced.
+ */
+ ts_stat->progress += ts_stat->progress_slice;
+ ts_stat->num_scanned++;
+ ts_stat->index++;
}
- /*
- * Measure progress independent of actually having to flush the buffer
- * - otherwise writing become unbalanced.
- */
- ts_stat->progress += ts_stat->progress_slice;
- ts_stat->num_scanned++;
- ts_stat->index++;
+ if (to_write.nbuffers > 0)
+ WriteBuffers(&to_write, ioq, &wb_context);
+
/* Have all the buffers from the tablespace been processed? */
if (ts_stat->num_scanned == ts_stat->num_to_scan)
@@ -3214,15 +3311,23 @@ BufferSync(int flags)
*
* (This will check for barrier events even if it doesn't sleep.)
*/
- CheckpointWriteDelay(flags, (double) num_processed / num_to_scan);
+ CheckpointWriteDelay(ioq, flags, (double) num_processed / num_to_scan);
}
+ Assert(to_write.nbuffers == 0);
+ io_queue_wait_all(ioq);
+
/*
* Issue all pending flushes. Only checkpointer calls BufferSync(), so
* IOContext will always be IOCONTEXT_NORMAL.
*/
IssuePendingWritebacks(&wb_context, IOCONTEXT_NORMAL);
+ io_queue_wait_all(ioq); /* IssuePendingWritebacks might have added
+ * more */
+ io_queue_free(ioq);
+ BuffersToWriteEnd(&to_write);
+
pfree(per_ts_stat);
per_ts_stat = NULL;
binaryheap_free(ts_heap);
@@ -3248,7 +3353,7 @@ BufferSync(int flags)
* bgwriter_lru_maxpages to 0.)
*/
bool
-BgBufferSync(WritebackContext *wb_context)
+BgBufferSync(IOQueue *ioq, WritebackContext *wb_context)
{
/* info obtained from freelist.c */
int strategy_buf_id;
@@ -3291,6 +3396,8 @@ BgBufferSync(WritebackContext *wb_context)
long new_strategy_delta;
uint32 new_recent_alloc;
+ BuffersToWrite to_write;
+
/*
* Find out where the freelist clock sweep currently is, and how many
* buffer allocations have happened since our last call.
@@ -3467,11 +3574,25 @@ BgBufferSync(WritebackContext *wb_context)
num_written = 0;
reusable_buffers = reusable_buffers_est;
+ BuffersToWriteInit(&to_write, ioq, wb_context);
+
/* Execute the LRU scan */
while (num_to_scan > 0 && reusable_buffers < upcoming_alloc_est)
{
- int sync_state = SyncOneBuffer(next_to_clean, true,
- wb_context);
+ int sync_state;
+
+ sync_state = PrepareToWriteBuffer(&to_write, next_to_clean + 1,
+ true, ioq, wb_context);
+ if (sync_state & BUF_CANT_MERGE)
+ {
+ Assert(to_write.nbuffers > 0);
+
+ WriteBuffers(&to_write, ioq, wb_context);
+
+ sync_state = PrepareToWriteBuffer(&to_write, next_to_clean + 1,
+ true, ioq, wb_context);
+ Assert(sync_state != BUF_CANT_MERGE);
+ }
if (++next_to_clean >= NBuffers)
{
@@ -3482,6 +3603,13 @@ BgBufferSync(WritebackContext *wb_context)
if (sync_state & BUF_WRITTEN)
{
+ Assert(sync_state & BUF_REUSABLE);
+
+ if (to_write.nbuffers == io_combine_limit)
+ {
+ WriteBuffers(&to_write, ioq, wb_context);
+ }
+
reusable_buffers++;
if (++num_written >= bgwriter_lru_maxpages)
{
@@ -3493,6 +3621,11 @@ BgBufferSync(WritebackContext *wb_context)
reusable_buffers++;
}
+ if (to_write.nbuffers > 0)
+ WriteBuffers(&to_write, ioq, wb_context);
+
+ BuffersToWriteEnd(&to_write);
+
PendingBgWriterStats.buf_written_clean += num_written;
#ifdef BGW_DEBUG
@@ -3531,8 +3664,66 @@ BgBufferSync(WritebackContext *wb_context)
return (bufs_to_lap == 0 && recent_alloc == 0);
}
+static inline bool
+BufferTagsSameRel(const BufferTag *tag1, const BufferTag *tag2)
+{
+ return (tag1->spcOid == tag2->spcOid) &&
+ (tag1->dbOid == tag2->dbOid) &&
+ (tag1->relNumber == tag2->relNumber) &&
+ (tag1->forkNum == tag2->forkNum)
+ ;
+}
+
+static bool
+CanMergeWrite(BuffersToWrite *to_write, BufferDesc *cur_buf_hdr)
+{
+ BlockNumber cur_block = cur_buf_hdr->tag.blockNum;
+
+ Assert(to_write->nbuffers > 0); /* can't merge with nothing */
+ Assert(to_write->start_at_tag.relNumber != InvalidOid);
+ Assert(to_write->start_at_tag.blockNum != InvalidBlockNumber);
+
+ Assert(to_write->ioh != NULL);
+
+ /*
+ * First check if the blocknumber is one that we could actually merge,
+ * that's cheaper than checking the tablespace/db/relnumber/fork match.
+ */
+ if (to_write->start_at_tag.blockNum + to_write->nbuffers != cur_block)
+ return false;
+
+ if (!BufferTagsSameRel(&to_write->start_at_tag, &cur_buf_hdr->tag))
+ return false;
+
+ /*
+ * Need to check with smgr how large a write we're allowed to make. To
+ * reduce the overhead of the smgr check, only inquire once, when
+ * processing the first to-be-merged buffer. That avoids the overhead in
+ * the common case of writing out buffers that definitely not mergeable.
+ */
+ if (to_write->nbuffers == 1)
+ {
+ SMgrRelation smgr;
+
+ smgr = smgropen(BufTagGetRelFileLocator(&to_write->start_at_tag), INVALID_PROC_NUMBER);
+
+ to_write->max_combine = smgrmaxcombine(smgr,
+ to_write->start_at_tag.forkNum,
+ to_write->start_at_tag.blockNum);
+ }
+ else
+ {
+ Assert(to_write->max_combine > 0);
+ }
+
+ if (to_write->start_at_tag.blockNum + to_write->max_combine <= cur_block)
+ return false;
+
+ return true;
+}
+
/*
- * SyncOneBuffer -- process a single buffer during syncing.
+ * PrepareToWriteBuffer -- process a single buffer during syncing.
*
* If skip_recently_used is true, we don't write currently-pinned buffers, nor
* buffers marked recently used, as these are not replacement candidates.
@@ -3541,22 +3732,56 @@ BgBufferSync(WritebackContext *wb_context)
* BUF_WRITTEN: we wrote the buffer.
* BUF_REUSABLE: buffer is available for replacement, ie, it has
* pin count 0 and usage count 0.
+ * BUF_CANT_MERGE: can't combine this write with prior writes, caller needs
+ * to issue those first
*
* (BUF_WRITTEN could be set in error if FlushBuffer finds the buffer clean
* after locking it, but we don't care all that much.)
*/
static int
-SyncOneBuffer(int buf_id, bool skip_recently_used, WritebackContext *wb_context)
+PrepareToWriteBuffer(BuffersToWrite *to_write, Buffer buf,
+ bool skip_recently_used,
+ IOQueue *ioq, WritebackContext *wb_context)
{
- BufferDesc *bufHdr = GetBufferDescriptor(buf_id);
- int result = 0;
+ BufferDesc *cur_buf_hdr = GetBufferDescriptor(buf - 1);
uint32 buf_state;
- BufferTag tag;
+ int result = 0;
+ XLogRecPtr cur_buf_lsn;
+ LWLock *content_lock;
+ bool may_block;
+
+ /*
+ * Check if this buffer can be written out together with already prepared
+ * writes. We check before we have pinned the buffer, so the buffer can be
+ * written out and replaced between this check and us pinning the buffer -
+ * we'll recheck below. The reason for the pre-check is that we don't want
+ * to pin the buffer just to find out that we can't merge the IO.
+ */
+ if (to_write->nbuffers != 0)
+ {
+ if (!CanMergeWrite(to_write, cur_buf_hdr))
+ {
+ result |= BUF_CANT_MERGE;
+ return result;
+ }
+ }
+ else
+ {
+ if (to_write->ioh == NULL)
+ {
+ to_write->ioh = io_queue_get_io(ioq);
+ pgaio_io_get_ref(to_write->ioh, &to_write->ior);
+ }
+
+ to_write->start_at_tag = cur_buf_hdr->tag;
+ }
/* Make sure we can handle the pin */
ReservePrivateRefCountEntry();
ResourceOwnerEnlarge(CurrentResourceOwner);
+ /* XXX: Should also check if we are allowed to pin one more buffer */
+
/*
* Check whether buffer needs writing.
*
@@ -3566,7 +3791,7 @@ SyncOneBuffer(int buf_id, bool skip_recently_used, WritebackContext *wb_context)
* don't worry because our checkpoint.redo points before log record for
* upcoming changes and so we are not required to write such dirty buffer.
*/
- buf_state = LockBufHdr(bufHdr);
+ buf_state = LockBufHdr(cur_buf_hdr);
if (BUF_STATE_GET_REFCOUNT(buf_state) == 0 &&
BUF_STATE_GET_USAGECOUNT(buf_state) == 0)
@@ -3575,40 +3800,282 @@ SyncOneBuffer(int buf_id, bool skip_recently_used, WritebackContext *wb_context)
}
else if (skip_recently_used)
{
+#if 0
+ elog(LOG, "at block %d: skip recent with nbuffers %d",
+ cur_buf_hdr->tag.blockNum, to_write->nbuffers);
+#endif
/* Caller told us not to write recently-used buffers */
- UnlockBufHdr(bufHdr, buf_state);
+ UnlockBufHdr(cur_buf_hdr, buf_state);
return result;
}
if (!(buf_state & BM_VALID) || !(buf_state & BM_DIRTY))
{
/* It's clean, so nothing to do */
- UnlockBufHdr(bufHdr, buf_state);
+ UnlockBufHdr(cur_buf_hdr, buf_state);
return result;
}
- /*
- * Pin it, share-lock it, write it. (FlushBuffer will do nothing if the
- * buffer is clean by the time we've locked it.)
- */
- PinBuffer_Locked(bufHdr);
- LWLockAcquire(BufferDescriptorGetContentLock(bufHdr), LW_SHARED);
-
- FlushBuffer(bufHdr, NULL, IOOBJECT_RELATION, IOCONTEXT_NORMAL);
-
- LWLockRelease(BufferDescriptorGetContentLock(bufHdr));
-
- tag = bufHdr->tag;
-
- UnpinBuffer(bufHdr);
+ /* pin the buffer, from now on its identity can't change anymore */
+ PinBuffer_Locked(cur_buf_hdr);
/*
- * SyncOneBuffer() is only called by checkpointer and bgwriter, so
- * IOContext will always be IOCONTEXT_NORMAL.
+ * If we are merging, check if the buffer's identity possibly changed
+ * while we hadn't yet pinned it.
+ *
+ * XXX: It might be worth checking if we still want to write the buffer
+ * out, e.g. it could have been replaced with a buffer that doesn't have
+ * BM_CHECKPOINT_NEEDED set.
*/
- ScheduleBufferTagForWriteback(wb_context, IOCONTEXT_NORMAL, &tag);
+ if (to_write->nbuffers != 0)
+ {
+ if (!CanMergeWrite(to_write, cur_buf_hdr))
+ {
+ elog(LOG, "changed identity");
+ UnpinBuffer(cur_buf_hdr);
- return result | BUF_WRITTEN;
+ result |= BUF_CANT_MERGE;
+
+ return result;
+ }
+ }
+
+ may_block = to_write->nbuffers == 0
+ && !pgaio_have_staged()
+ && io_queue_is_empty(ioq)
+ ;
+ content_lock = BufferDescriptorGetContentLock(cur_buf_hdr);
+
+ if (!may_block)
+ {
+ if (LWLockConditionalAcquire(content_lock, LW_SHARED))
+ {
+ /* done */
+ }
+ else if (to_write->nbuffers == 0)
+ {
+ /*
+ * Need to wait for all prior IO to finish before blocking for
+ * lock acquisition, to avoid the risk a deadlock due to us
+ * waiting for another backend that is waiting for our unsubmitted
+ * IO to complete.
+ */
+ pgaio_submit_staged();
+ io_queue_wait_all(ioq);
+
+ elog(DEBUG2, "at block %u: can't block, nbuffers = 0",
+ cur_buf_hdr->tag.blockNum
+ );
+
+ may_block = to_write->nbuffers == 0
+ && !pgaio_have_staged()
+ && io_queue_is_empty(ioq)
+ ;
+ Assert(may_block);
+
+ LWLockAcquire(content_lock, LW_SHARED);
+ }
+ else
+ {
+ elog(DEBUG2, "at block %d: can't block nbuffers = %d",
+ cur_buf_hdr->tag.blockNum,
+ to_write->nbuffers);
+
+ UnpinBuffer(cur_buf_hdr);
+ result |= BUF_CANT_MERGE;
+ Assert(to_write->nbuffers > 0);
+
+ return result;
+ }
+ }
+ else
+ {
+ LWLockAcquire(content_lock, LW_SHARED);
+ }
+
+ if (!may_block)
+ {
+ if (!StartBufferIO(cur_buf_hdr, false, !may_block))
+ {
+ pgaio_submit_staged();
+ io_queue_wait_all(ioq);
+
+ may_block = io_queue_is_empty(ioq) && to_write->nbuffers == 0 && !pgaio_have_staged();
+
+ if (!StartBufferIO(cur_buf_hdr, false, !may_block))
+ {
+ elog(DEBUG2, "at block %d: non-waitable StartBufferIO returns false, %d",
+ cur_buf_hdr->tag.blockNum,
+ may_block);
+
+ /*
+ * FIXME: can't tell whether this is because the buffer has
+ * been cleaned
+ */
+ if (!may_block)
+ {
+ result |= BUF_CANT_MERGE;
+ Assert(to_write->nbuffers > 0);
+ }
+ LWLockRelease(content_lock);
+ UnpinBuffer(cur_buf_hdr);
+
+ return result;
+ }
+ }
+ }
+ else
+ {
+ if (!StartBufferIO(cur_buf_hdr, false, false))
+ {
+ elog(DEBUG2, "waitable StartBufferIO returns false");
+ LWLockRelease(content_lock);
+ UnpinBuffer(cur_buf_hdr);
+
+ /*
+ * FIXME: Historically we returned BUF_WRITTEN in this case, which
+ * seems wrong
+ */
+ return result;
+ }
+ }
+
+ /*
+ * Run PageGetLSN while holding header lock, since we don't have the
+ * buffer locked exclusively in all cases.
+ */
+ buf_state = LockBufHdr(cur_buf_hdr);
+
+ cur_buf_lsn = BufferGetLSN(cur_buf_hdr);
+
+ /* To check if block content changes while flushing. - vadim 01/17/97 */
+ buf_state &= ~BM_JUST_DIRTIED;
+
+ UnlockBufHdr(cur_buf_hdr, buf_state);
+
+ to_write->buffers[to_write->nbuffers] = buf;
+ to_write->nbuffers++;
+
+ if (buf_state & BM_PERMANENT &&
+ (to_write->max_lsn == InvalidXLogRecPtr || to_write->max_lsn < cur_buf_lsn))
+ {
+ to_write->max_lsn = cur_buf_lsn;
+ }
+
+ result |= BUF_WRITTEN;
+
+ return result;
+}
+
+static void
+WriteBuffers(BuffersToWrite *to_write,
+ IOQueue *ioq, WritebackContext *wb_context)
+{
+ SMgrRelation smgr;
+ Buffer first_buf;
+ BufferDesc *first_buf_hdr;
+ bool needs_checksum;
+
+ Assert(to_write->nbuffers > 0 && to_write->nbuffers <= io_combine_limit);
+
+ first_buf = to_write->buffers[0];
+ first_buf_hdr = GetBufferDescriptor(first_buf - 1);
+
+ smgr = smgropen(BufTagGetRelFileLocator(&first_buf_hdr->tag), INVALID_PROC_NUMBER);
+
+ /*
+ * Force XLOG flush up to buffer's LSN. This implements the basic WAL
+ * rule that log updates must hit disk before any of the data-file changes
+ * they describe do.
+ *
+ * However, this rule does not apply to unlogged relations, which will be
+ * lost after a crash anyway. Most unlogged relation pages do not bear
+ * LSNs since we never emit WAL records for them, and therefore flushing
+ * up through the buffer LSN would be useless, but harmless. However,
+ * GiST indexes use LSNs internally to track page-splits, and therefore
+ * unlogged GiST pages bear "fake" LSNs generated by
+ * GetFakeLSNForUnloggedRel. It is unlikely but possible that the fake
+ * LSN counter could advance past the WAL insertion point; and if it did
+ * happen, attempting to flush WAL through that location would fail, with
+ * disastrous system-wide consequences. To make sure that can't happen,
+ * skip the flush if the buffer isn't permanent.
+ */
+ if (to_write->max_lsn != InvalidXLogRecPtr)
+ XLogFlush(to_write->max_lsn);
+
+ /*
+ * Now it's safe to write buffer to disk. Note that no one else should
+ * have been able to write it while we were busy with log flushing because
+ * only one process at a time can set the BM_IO_IN_PROGRESS bit.
+ */
+
+ for (int nbuf = 0; nbuf < to_write->nbuffers; nbuf++)
+ {
+ Buffer cur_buf = to_write->buffers[nbuf];
+ BufferDesc *cur_buf_hdr = GetBufferDescriptor(cur_buf - 1);
+ Block bufBlock;
+ char *bufToWrite;
+
+ bufBlock = BufHdrGetBlock(cur_buf_hdr);
+ needs_checksum = PageNeedsChecksumCopy((Page) bufBlock);
+
+ /*
+ * Update page checksum if desired. Since we have only shared lock on
+ * the buffer, other processes might be updating hint bits in it, so
+ * we must copy the page to a bounce buffer if we do checksumming.
+ */
+ if (needs_checksum)
+ {
+ PgAioBounceBuffer *bb = pgaio_bounce_buffer_get();
+
+ pgaio_io_assoc_bounce_buffer(to_write->ioh, bb);
+
+ bufToWrite = pgaio_bounce_buffer_buffer(bb);
+ memcpy(bufToWrite, bufBlock, BLCKSZ);
+ PageSetChecksumInplace((Page) bufToWrite, cur_buf_hdr->tag.blockNum);
+ }
+ else
+ {
+ bufToWrite = bufBlock;
+ }
+
+ to_write->data_ptrs[nbuf] = bufToWrite;
+ }
+
+ pgaio_io_set_io_data_32(to_write->ioh,
+ (uint32 *) to_write->buffers,
+ to_write->nbuffers);
+ pgaio_io_add_shared_cb(to_write->ioh, ASC_SHARED_BUFFER_WRITE);
+
+ smgrstartwritev(to_write->ioh, smgr,
+ BufTagGetForkNum(&first_buf_hdr->tag),
+ first_buf_hdr->tag.blockNum,
+ to_write->data_ptrs,
+ to_write->nbuffers,
+ false);
+ pgstat_count_io_op_n(IOOBJECT_RELATION, IOCONTEXT_NORMAL,
+ IOOP_WRITE, to_write->nbuffers);
+
+
+ for (int nbuf = 0; nbuf < to_write->nbuffers; nbuf++)
+ {
+ Buffer cur_buf = to_write->buffers[nbuf];
+ BufferDesc *cur_buf_hdr = GetBufferDescriptor(cur_buf - 1);
+
+ UnpinBuffer(cur_buf_hdr);
+ }
+
+ io_queue_track(ioq, &to_write->ior);
+ to_write->total_writes++;
+
+ /* clear state for next write */
+ to_write->nbuffers = 0;
+ to_write->start_at_tag.relNumber = InvalidOid;
+ to_write->start_at_tag.blockNum = InvalidBlockNumber;
+ to_write->max_combine = 0;
+ to_write->max_lsn = InvalidXLogRecPtr;
+ to_write->ioh = NULL;
+ pgaio_io_ref_clear(&to_write->ior);
}
/*
@@ -3974,6 +4441,7 @@ FlushBuffer(BufferDesc *buf, SMgrRelation reln, IOObject io_object,
error_context_stack = errcallback.previous;
}
+
/*
* RelationGetNumberOfBlocksInFork
* Determines the current number of pages in the specified relation fork.
diff --git a/src/backend/storage/page/bufpage.c b/src/backend/storage/page/bufpage.c
index be6f1f62d29..8295e3fb0a0 100644
--- a/src/backend/storage/page/bufpage.c
+++ b/src/backend/storage/page/bufpage.c
@@ -1491,6 +1491,16 @@ PageIndexTupleOverwrite(Page page, OffsetNumber offnum,
return true;
}
+bool
+PageNeedsChecksumCopy(Page page)
+{
+ if (PageIsNew(page))
+ return false;
+
+ /* If we don't need a checksum, just return the passed-in data */
+ return DataChecksumsEnabled();
+}
+
/*
* Set checksum for a page in shared buffers.
diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list
index a38141b4e50..9973162dc86 100644
--- a/src/tools/pgindent/typedefs.list
+++ b/src/tools/pgindent/typedefs.list
@@ -345,6 +345,7 @@ BufferManagerRelation
BufferStrategyControl
BufferTag
BufferUsage
+BuffersToWrite
BuildAccumulator
BuiltinScript
BulkInsertState
--
2.45.2.827.g557ae147e6
v2.1-0019-very-wip-test_aio-module.patch (text/x-diff, 38.1K) download
From b3d46d7af01fb746ab8a366a771420b4608a337e Mon Sep 17 00:00:00 2001
From: Andres Freund <andres@anarazel.de>
Date: Sat, 31 Aug 2024 21:13:48 -0400
Subject: [PATCH v2.1 19/20] very-wip: test_aio module
Author:
Reviewed-by:
Discussion: https://postgr.es/m/
Backpatch:
---
src/include/storage/aio_internal.h | 10 +
src/include/storage/buf_internals.h | 4 +
src/backend/storage/aio/aio.c | 38 ++
src/backend/storage/buffer/bufmgr.c | 3 +-
src/test/modules/Makefile | 1 +
src/test/modules/meson.build | 1 +
src/test/modules/test_aio/.gitignore | 6 +
src/test/modules/test_aio/Makefile | 34 ++
src/test/modules/test_aio/expected/inject.out | 180 +++++++
src/test/modules/test_aio/expected/io.out | 40 ++
.../modules/test_aio/expected/ownership.out | 148 ++++++
src/test/modules/test_aio/expected/prep.out | 17 +
src/test/modules/test_aio/io_uring.conf | 5 +
src/test/modules/test_aio/meson.build | 78 +++
src/test/modules/test_aio/sql/inject.sql | 51 ++
src/test/modules/test_aio/sql/io.sql | 16 +
src/test/modules/test_aio/sql/ownership.sql | 65 +++
src/test/modules/test_aio/sql/prep.sql | 9 +
src/test/modules/test_aio/sync.conf | 5 +
src/test/modules/test_aio/test_aio--1.0.sql | 94 ++++
src/test/modules/test_aio/test_aio.c | 479 ++++++++++++++++++
src/test/modules/test_aio/test_aio.control | 3 +
src/test/modules/test_aio/worker.conf | 5 +
23 files changed, 1290 insertions(+), 2 deletions(-)
create mode 100644 src/test/modules/test_aio/.gitignore
create mode 100644 src/test/modules/test_aio/Makefile
create mode 100644 src/test/modules/test_aio/expected/inject.out
create mode 100644 src/test/modules/test_aio/expected/io.out
create mode 100644 src/test/modules/test_aio/expected/ownership.out
create mode 100644 src/test/modules/test_aio/expected/prep.out
create mode 100644 src/test/modules/test_aio/io_uring.conf
create mode 100644 src/test/modules/test_aio/meson.build
create mode 100644 src/test/modules/test_aio/sql/inject.sql
create mode 100644 src/test/modules/test_aio/sql/io.sql
create mode 100644 src/test/modules/test_aio/sql/ownership.sql
create mode 100644 src/test/modules/test_aio/sql/prep.sql
create mode 100644 src/test/modules/test_aio/sync.conf
create mode 100644 src/test/modules/test_aio/test_aio--1.0.sql
create mode 100644 src/test/modules/test_aio/test_aio.c
create mode 100644 src/test/modules/test_aio/test_aio.control
create mode 100644 src/test/modules/test_aio/worker.conf
diff --git a/src/include/storage/aio_internal.h b/src/include/storage/aio_internal.h
index 5d18d112e2d..a44cdb457ee 100644
--- a/src/include/storage/aio_internal.h
+++ b/src/include/storage/aio_internal.h
@@ -262,6 +262,16 @@ extern const char *pgaio_io_get_op_name(PgAioHandle *ioh);
extern const char *pgaio_io_get_state_name(PgAioHandle *ioh);
+
+/* These functions are just for use in tests, from within injection points */
+#ifdef USE_INJECTION_POINTS
+
+extern PgAioHandle *pgaio_inj_io_get(void);
+
+#endif
+
+
+
/* Declarations for the tables of function pointers exposed by each IO method. */
extern const IoMethodOps pgaio_sync_ops;
extern const IoMethodOps pgaio_worker_ops;
diff --git a/src/include/storage/buf_internals.h b/src/include/storage/buf_internals.h
index 9d3123663b3..1b3329a25b4 100644
--- a/src/include/storage/buf_internals.h
+++ b/src/include/storage/buf_internals.h
@@ -423,6 +423,10 @@ extern void IssuePendingWritebacks(WritebackContext *wb_context, IOContext io_co
extern void ScheduleBufferTagForWriteback(WritebackContext *wb_context,
IOContext io_context, BufferTag *tag);
+/* solely to make it easier to write tests */
+extern bool StartBufferIO(BufferDesc *buf, bool forInput, bool nowait);
+
+
/* freelist.c */
extern IOContext IOContextForStrategy(BufferAccessStrategy strategy);
extern BufferDesc *StrategyGetBuffer(BufferAccessStrategy strategy,
diff --git a/src/backend/storage/aio/aio.c b/src/backend/storage/aio/aio.c
index 8877a33b9f2..7efc9631f5f 100644
--- a/src/backend/storage/aio/aio.c
+++ b/src/backend/storage/aio/aio.c
@@ -22,6 +22,9 @@
#include "utils/resowner.h"
#include "utils/wait_event_types.h"
+#ifdef USE_INJECTION_POINTS
+#include "utils/injection_point.h"
+#endif
static void pgaio_io_reclaim(PgAioHandle *ioh);
@@ -67,6 +70,11 @@ static const IoMethodOps *pgaio_ops_table[] = {
const IoMethodOps *pgaio_impl;
+#ifdef USE_INJECTION_POINTS
+static PgAioHandle *inj_cur_handle;
+#endif
+
+
/* --------------------------------------------------------------------------------
* "Core" IO Api
@@ -543,6 +551,19 @@ pgaio_io_process_completion(PgAioHandle *ioh, int result)
/* FIXME: should be done in separate function */
ioh->state = AHS_REAPED;
+#ifdef USE_INJECTION_POINTS
+ inj_cur_handle = ioh;
+
+ /*
+ * FIXME: This could be in a critical section - but it looks like we can't
+ * just InjectionPointLoad() at process start, as the injection point
+ * might not yet be defined.
+ */
+ InjectionPointCached("AIO_PROCESS_COMPLETION_BEFORE_SHARED");
+
+ inj_cur_handle = NULL;
+#endif
+
pgaio_io_process_completion_subject(ioh);
/* ensure results of completion are visible before the new state */
@@ -1013,3 +1034,20 @@ assign_io_method(int newval, void *extra)
{
pgaio_impl = pgaio_ops_table[newval];
}
+
+
+
+/* --------------------------------------------------------------------------------
+ * Injection point support
+ * --------------------------------------------------------------------------------
+ */
+
+#ifdef USE_INJECTION_POINTS
+
+PgAioHandle *
+pgaio_inj_io_get(void)
+{
+ return inj_cur_handle;
+}
+
+#endif
diff --git a/src/backend/storage/buffer/bufmgr.c b/src/backend/storage/buffer/bufmgr.c
index 6ec700e5ef2..44b1b6fb316 100644
--- a/src/backend/storage/buffer/bufmgr.c
+++ b/src/backend/storage/buffer/bufmgr.c
@@ -514,7 +514,6 @@ static void UnpinBufferNoOwner(BufferDesc *buf);
static void BufferSync(int flags);
static uint32 WaitBufHdrUnlocked(BufferDesc *buf);
static void WaitIO(BufferDesc *buf);
-static bool StartBufferIO(BufferDesc *buf, bool forInput, bool nowait);
static void TerminateBufferIO(BufferDesc *buf, bool clear_dirty,
uint32 set_flag_bits, bool forget_owner,
bool syncio);
@@ -6095,7 +6094,7 @@ WaitIO(BufferDesc *buf)
* find out if they can perform the I/O as part of a larger operation, without
* waiting for the answer or distinguishing the reasons why not.
*/
-static bool
+bool
StartBufferIO(BufferDesc *buf, bool forInput, bool nowait)
{
uint32 buf_state;
diff --git a/src/test/modules/Makefile b/src/test/modules/Makefile
index 256799f520a..7df90602e90 100644
--- a/src/test/modules/Makefile
+++ b/src/test/modules/Makefile
@@ -13,6 +13,7 @@ SUBDIRS = \
libpq_pipeline \
plsample \
spgist_name_ops \
+ test_aio \
test_bloomfilter \
test_copy_callbacks \
test_custom_rmgrs \
diff --git a/src/test/modules/meson.build b/src/test/modules/meson.build
index d8fe059d236..bc7d19e694f 100644
--- a/src/test/modules/meson.build
+++ b/src/test/modules/meson.build
@@ -1,5 +1,6 @@
# Copyright (c) 2022-2024, PostgreSQL Global Development Group
+subdir('test_aio')
subdir('brin')
subdir('commit_ts')
subdir('delay_execution')
diff --git a/src/test/modules/test_aio/.gitignore b/src/test/modules/test_aio/.gitignore
new file mode 100644
index 00000000000..b4903eba657
--- /dev/null
+++ b/src/test/modules/test_aio/.gitignore
@@ -0,0 +1,6 @@
+# Generated subdirectories
+/log/
+/results/
+/output_iso/
+/tmp_check/
+/tmp_check_iso/
diff --git a/src/test/modules/test_aio/Makefile b/src/test/modules/test_aio/Makefile
new file mode 100644
index 00000000000..ae6d685835b
--- /dev/null
+++ b/src/test/modules/test_aio/Makefile
@@ -0,0 +1,34 @@
+# src/test/modules/delay_execution/Makefile
+
+PGFILEDESC = "test_aio - test code for AIO"
+
+MODULE_big = test_aio
+OBJS = \
+ $(WIN32RES) \
+ test_aio.o
+
+EXTENSION = test_aio
+DATA = test_aio--1.0.sql
+
+REGRESS = prep ownership io
+
+ifeq ($(enable_injection_points),yes)
+REGRESS += inject
+endif
+
+# FIXME: with meson this runs the tests once with worker and once - if
+# supported - with io_uring.
+
+# requires custom config
+NO_INSTALLCHECK = 1
+
+ifdef USE_PGXS
+PG_CONFIG = pg_config
+PGXS := $(shell $(PG_CONFIG) --pgxs)
+include $(PGXS)
+else
+subdir = src/test/modules/test_aio
+top_builddir = ../../../..
+include $(top_builddir)/src/Makefile.global
+include $(top_srcdir)/contrib/contrib-global.mk
+endif
diff --git a/src/test/modules/test_aio/expected/inject.out b/src/test/modules/test_aio/expected/inject.out
new file mode 100644
index 00000000000..e52b0f086dd
--- /dev/null
+++ b/src/test/modules/test_aio/expected/inject.out
@@ -0,0 +1,180 @@
+SELECT count(*) FROM tbl_b WHERE ctid = '(2, 1)';
+ count
+-------
+ 1
+(1 row)
+
+-- injected what we'd expect
+SELECT inj_io_short_read_attach(8192);
+ inj_io_short_read_attach
+--------------------------
+
+(1 row)
+
+SELECT invalidate_rel_block('tbl_b', 2);
+ invalidate_rel_block
+----------------------
+
+(1 row)
+
+SELECT count(*) FROM tbl_b WHERE ctid = '(2, 1)';
+ count
+-------
+ 1
+(1 row)
+
+SELECT inj_io_short_read_detach();
+ inj_io_short_read_detach
+--------------------------
+
+(1 row)
+
+-- injected a read shorter than a single block, expecting error
+SELECT inj_io_short_read_attach(17);
+ inj_io_short_read_attach
+--------------------------
+
+(1 row)
+
+SELECT invalidate_rel_block('tbl_b', 2);
+ invalidate_rel_block
+----------------------
+
+(1 row)
+
+SELECT redact($$
+ SELECT count(*) FROM tbl_b WHERE ctid = '(2, 1)';
+$$);
+NOTICE: wrapped error: could not read blocks 2..2 in file base/<redacted>: read only 0 of 8192 bytes
+ redact
+--------
+ f
+(1 row)
+
+SELECT inj_io_short_read_detach();
+ inj_io_short_read_detach
+--------------------------
+
+(1 row)
+
+-- shorten multi-block read to a single block, should retry, but that's not
+-- implemented yet
+SELECT inj_io_short_read_attach(8192);
+ inj_io_short_read_attach
+--------------------------
+
+(1 row)
+
+SELECT invalidate_rel_block('tbl_b', 0);
+ invalidate_rel_block
+----------------------
+
+(1 row)
+
+SELECT invalidate_rel_block('tbl_b', 1);
+ invalidate_rel_block
+----------------------
+
+(1 row)
+
+SELECT invalidate_rel_block('tbl_b', 2);
+ invalidate_rel_block
+----------------------
+
+(1 row)
+
+SELECT redact($$
+ SELECT count(*) FROM tbl_b;
+$$);
+NOTICE: wrapped error: could not read blocks 1..2 in file base/<redacted>: read only 8192 of 16384 bytes
+ redact
+--------
+ f
+(1 row)
+
+SELECT inj_io_short_read_detach();
+ inj_io_short_read_detach
+--------------------------
+
+(1 row)
+
+-- verify that checksum errors are detected even as part of a shortened
+-- multi-block read
+-- (tbl_a, block 1 is corrupted)
+SELECT redact($$
+ SELECT count(*) FROM tbl_a WHERE ctid < '(2, 1)';
+$$);
+NOTICE: wrapped error: invalid page in block 2 of relation base/<redacted>
+ redact
+--------
+ f
+(1 row)
+
+SELECT inj_io_short_read_attach(8192);
+ inj_io_short_read_attach
+--------------------------
+
+(1 row)
+
+SELECT invalidate_rel_block('tbl_a', 0);
+ invalidate_rel_block
+----------------------
+
+(1 row)
+
+SELECT invalidate_rel_block('tbl_a', 1);
+ invalidate_rel_block
+----------------------
+
+(1 row)
+
+SELECT invalidate_rel_block('tbl_a', 2);
+ invalidate_rel_block
+----------------------
+
+(1 row)
+
+SELECT redact($$
+ SELECT count(*) FROM tbl_a WHERE ctid < '(2, 1)';
+$$);
+NOTICE: wrapped error: invalid page in block 2 of relation base/<redacted>
+ redact
+--------
+ f
+(1 row)
+
+SELECT inj_io_short_read_detach();
+ inj_io_short_read_detach
+--------------------------
+
+(1 row)
+
+-- FIXME: Should error
+-- FIXME: errno encoding?
+SELECT inj_io_short_read_attach(-5);
+ inj_io_short_read_attach
+--------------------------
+
+(1 row)
+
+SELECT invalidate_rel_block('tbl_b', 2);
+ invalidate_rel_block
+----------------------
+
+(1 row)
+
+SELECT redact($$
+ SELECT count(*) FROM tbl_b WHERE ctid = '(2, 1)';
+$$);
+NOTICE: wrapped error: could not read blocks 2..3 in file base/<redacted>: Input/output error
+ redact
+--------
+ f
+(1 row)
+
+SELECT inj_io_short_read_detach();
+ inj_io_short_read_detach
+--------------------------
+
+(1 row)
+
diff --git a/src/test/modules/test_aio/expected/io.out b/src/test/modules/test_aio/expected/io.out
new file mode 100644
index 00000000000..e46b582f290
--- /dev/null
+++ b/src/test/modules/test_aio/expected/io.out
@@ -0,0 +1,40 @@
+SELECT count(*) FROM tbl_a WHERE ctid = '(1, 1)';
+ count
+-------
+ 1
+(1 row)
+
+SELECT corrupt_rel_block('tbl_a', 1);
+ corrupt_rel_block
+-------------------
+
+(1 row)
+
+-- FIXME: Should report the error
+SELECT redact($$
+ SELECT read_corrupt_rel_block('tbl_a', 1);
+$$);
+ redact
+--------
+ t
+(1 row)
+
+-- verify the error is reported
+SELECT redact($$
+ SELECT count(*) FROM tbl_a WHERE ctid = '(1, 1)';
+$$);
+NOTICE: wrapped error: invalid page in block 2 of relation base/<redacted>
+ redact
+--------
+ f
+(1 row)
+
+SELECT redact($$
+ SELECT count(*) FROM tbl_a;
+$$);
+NOTICE: wrapped error: invalid page in block 2 of relation base/<redacted>
+ redact
+--------
+ f
+(1 row)
+
diff --git a/src/test/modules/test_aio/expected/ownership.out b/src/test/modules/test_aio/expected/ownership.out
new file mode 100644
index 00000000000..97fdad6c629
--- /dev/null
+++ b/src/test/modules/test_aio/expected/ownership.out
@@ -0,0 +1,148 @@
+-----
+-- IO handles
+----
+-- leak warning: implicit xact
+SELECT handle_get();
+WARNING: leaked AIO handle
+ handle_get
+------------
+
+(1 row)
+
+-- leak warning: explicit xact
+BEGIN; SELECT handle_get(); COMMIT;
+WARNING: leaked AIO handle
+ handle_get
+------------
+
+(1 row)
+
+-- leak warning + error: released in different command (thus resowner)
+BEGIN; SELECT handle_get(); SELECT handle_release_last(); COMMIT;
+WARNING: leaked AIO handle
+ handle_get
+------------
+
+(1 row)
+
+ERROR: release in unexpected state
+-- no leak, same command
+BEGIN; SELECT handle_get() UNION ALL SELECT handle_release_last(); COMMIT;
+ handle_get
+------------
+
+
+(2 rows)
+
+-- leak warning: subtrans
+BEGIN; SAVEPOINT foo; SELECT handle_get(); COMMIT;
+WARNING: leaked AIO handle
+ handle_get
+------------
+
+(1 row)
+
+-- normal handle use
+SELECT handle_get_release();
+ handle_get_release
+--------------------
+
+(1 row)
+
+-- should error out, API violation
+SELECT handle_get_twice();
+ERROR: API violation: Only one IO can be handed out
+-- recover after error in implicit xact
+SELECT handle_get_and_error(); SELECT handle_get_release();
+ERROR: as you command
+ handle_get_release
+--------------------
+
+(1 row)
+
+-- recover after error in explicit xact
+BEGIN; SELECT handle_get_and_error(); ROLLBACK; SELECT handle_get_release();
+ERROR: as you command
+ handle_get_release
+--------------------
+
+(1 row)
+
+-- recover after error in subtrans
+BEGIN; SAVEPOINT foo; SELECT handle_get_and_error(); ROLLBACK TO SAVEPOINT foo; SELECT handle_get_release(); ROLLBACK;
+ERROR: as you command
+ handle_get_release
+--------------------
+
+(1 row)
+
+-----
+-- Bounce Buffers handles
+----
+-- leak warning: implicit xact
+SELECT bb_get();
+WARNING: leaked AIO bounce buffer
+ bb_get
+--------
+
+(1 row)
+
+-- leak warning: explicit xact
+BEGIN; SELECT bb_get(); COMMIT;
+WARNING: leaked AIO bounce buffer
+ bb_get
+--------
+
+(1 row)
+
+-- missing leak warning: we should warn at command boundaries, not xact boundaries
+BEGIN; SELECT bb_get(); SELECT bb_release_last(); COMMIT;
+WARNING: leaked AIO bounce buffer
+ bb_get
+--------
+
+(1 row)
+
+ERROR: can only release handed out BB
+-- leak warning: subtrans
+BEGIN; SAVEPOINT foo; SELECT bb_get(); COMMIT;
+WARNING: leaked AIO bounce buffer
+ bb_get
+--------
+
+(1 row)
+
+-- normal bb use
+SELECT bb_get_release();
+ bb_get_release
+----------------
+
+(1 row)
+
+-- should error out, API violation
+SELECT bb_get_twice();
+ERROR: can only hand out one BB
+-- recover after error in implicit xact
+SELECT bb_get_and_error(); SELECT bb_get_release();
+ERROR: as you command
+ bb_get_release
+----------------
+
+(1 row)
+
+-- recover after error in explicit xact
+BEGIN; SELECT bb_get_and_error(); ROLLBACK; SELECT bb_get_release();
+ERROR: as you command
+ bb_get_release
+----------------
+
+(1 row)
+
+-- recover after error in subtrans
+BEGIN; SAVEPOINT foo; SELECT bb_get_and_error(); ROLLBACK TO SAVEPOINT foo; SELECT bb_get_release(); ROLLBACK;
+ERROR: as you command
+ bb_get_release
+----------------
+
+(1 row)
+
diff --git a/src/test/modules/test_aio/expected/prep.out b/src/test/modules/test_aio/expected/prep.out
new file mode 100644
index 00000000000..7fad6280db5
--- /dev/null
+++ b/src/test/modules/test_aio/expected/prep.out
@@ -0,0 +1,17 @@
+CREATE EXTENSION test_aio;
+CREATE TABLE tbl_a(data int not null);
+CREATE TABLE tbl_b(data int not null);
+INSERT INTO tbl_a SELECT generate_series(1, 10000);
+INSERT INTO tbl_b SELECT generate_series(1, 10000);
+SELECT grow_rel('tbl_a', 500);
+ grow_rel
+----------
+
+(1 row)
+
+SELECT grow_rel('tbl_b', 550);
+ grow_rel
+----------
+
+(1 row)
+
diff --git a/src/test/modules/test_aio/io_uring.conf b/src/test/modules/test_aio/io_uring.conf
new file mode 100644
index 00000000000..efd7ad143ff
--- /dev/null
+++ b/src/test/modules/test_aio/io_uring.conf
@@ -0,0 +1,5 @@
+shared_preload_libraries=test_aio
+io_method = 'io_uring'
+log_min_messages = 'DEBUG3'
+log_statement=all
+restart_after_crash=false
diff --git a/src/test/modules/test_aio/meson.build b/src/test/modules/test_aio/meson.build
new file mode 100644
index 00000000000..a4bef0ceeb0
--- /dev/null
+++ b/src/test/modules/test_aio/meson.build
@@ -0,0 +1,78 @@
+# Copyright (c) 2022-2024, PostgreSQL Global Development Group
+
+test_aio_sources = files(
+ 'test_aio.c',
+)
+
+if host_system == 'windows'
+ test_aio_sources += rc_lib_gen.process(win32ver_rc, extra_args: [
+ '--NAME', 'test_aio',
+ '--FILEDESC', 'test_aio - test code for AIO',])
+endif
+
+test_aio = shared_module('test_aio',
+ test_aio_sources,
+ kwargs: pg_test_mod_args,
+)
+test_install_libs += test_aio
+
+test_install_data += files(
+ 'test_aio.control',
+ 'test_aio--1.0.sql',
+)
+
+
+testfiles = [
+ 'prep',
+ 'ownership',
+ 'io',
+]
+
+if get_option('injection_points')
+ testfiles += 'inject'
+endif
+
+
+tests += {
+ 'name': 'test_aio_sync',
+ 'sd': meson.current_source_dir(),
+ 'bd': meson.current_build_dir(),
+ 'regress': {
+ 'sql': testfiles,
+ 'regress_args': [
+ '--temp-config', files('sync.conf'),
+ ],
+ # requires custom config
+ 'runningcheck': false,
+ },
+}
+
+tests += {
+ 'name': 'test_aio_worker',
+ 'sd': meson.current_source_dir(),
+ 'bd': meson.current_build_dir(),
+ 'regress': {
+ 'sql': testfiles,
+ 'regress_args': [
+ '--temp-config', files('worker.conf'),
+ ],
+ # requires custom config
+ 'runningcheck': false,
+ },
+}
+
+if liburing.found()
+ tests += {
+ 'name': 'test_aio_uring',
+ 'sd': meson.current_source_dir(),
+ 'bd': meson.current_build_dir(),
+ 'regress': {
+ 'sql': testfiles,
+ 'regress_args': [
+ '--temp-config', files('io_uring.conf'),
+ ],
+ # requires custom config
+ 'runningcheck': false,
+ }
+ }
+endif
diff --git a/src/test/modules/test_aio/sql/inject.sql b/src/test/modules/test_aio/sql/inject.sql
new file mode 100644
index 00000000000..b3d34de8977
--- /dev/null
+++ b/src/test/modules/test_aio/sql/inject.sql
@@ -0,0 +1,51 @@
+SELECT count(*) FROM tbl_b WHERE ctid = '(2, 1)';
+
+-- injected what we'd expect
+SELECT inj_io_short_read_attach(8192);
+SELECT invalidate_rel_block('tbl_b', 2);
+SELECT count(*) FROM tbl_b WHERE ctid = '(2, 1)';
+SELECT inj_io_short_read_detach();
+
+-- injected a read shorter than a single block, expecting error
+SELECT inj_io_short_read_attach(17);
+SELECT invalidate_rel_block('tbl_b', 2);
+SELECT redact($$
+ SELECT count(*) FROM tbl_b WHERE ctid = '(2, 1)';
+$$);
+SELECT inj_io_short_read_detach();
+
+-- shorten multi-block read to a single block, should retry, but that's not
+-- implemented yet
+SELECT inj_io_short_read_attach(8192);
+SELECT invalidate_rel_block('tbl_b', 0);
+SELECT invalidate_rel_block('tbl_b', 1);
+SELECT invalidate_rel_block('tbl_b', 2);
+SELECT redact($$
+ SELECT count(*) FROM tbl_b;
+$$);
+SELECT inj_io_short_read_detach();
+
+-- verify that checksum errors are detected even as part of a shortened
+-- multi-block read
+-- (tbl_a, block 1 is corrupted)
+SELECT redact($$
+ SELECT count(*) FROM tbl_a WHERE ctid < '(2, 1)';
+$$);
+SELECT inj_io_short_read_attach(8192);
+SELECT invalidate_rel_block('tbl_a', 0);
+SELECT invalidate_rel_block('tbl_a', 1);
+SELECT invalidate_rel_block('tbl_a', 2);
+SELECT redact($$
+ SELECT count(*) FROM tbl_a WHERE ctid < '(2, 1)';
+$$);
+SELECT inj_io_short_read_detach();
+
+
+-- FIXME: Should error
+-- FIXME: errno encoding?
+SELECT inj_io_short_read_attach(-5);
+SELECT invalidate_rel_block('tbl_b', 2);
+SELECT redact($$
+ SELECT count(*) FROM tbl_b WHERE ctid = '(2, 1)';
+$$);
+SELECT inj_io_short_read_detach();
diff --git a/src/test/modules/test_aio/sql/io.sql b/src/test/modules/test_aio/sql/io.sql
new file mode 100644
index 00000000000..a29bb4eb15d
--- /dev/null
+++ b/src/test/modules/test_aio/sql/io.sql
@@ -0,0 +1,16 @@
+SELECT count(*) FROM tbl_a WHERE ctid = '(1, 1)';
+
+SELECT corrupt_rel_block('tbl_a', 1);
+
+-- FIXME: Should report the error
+SELECT redact($$
+ SELECT read_corrupt_rel_block('tbl_a', 1);
+$$);
+
+-- verify the error is reported
+SELECT redact($$
+ SELECT count(*) FROM tbl_a WHERE ctid = '(1, 1)';
+$$);
+SELECT redact($$
+ SELECT count(*) FROM tbl_a;
+$$);
diff --git a/src/test/modules/test_aio/sql/ownership.sql b/src/test/modules/test_aio/sql/ownership.sql
new file mode 100644
index 00000000000..63cf40c802a
--- /dev/null
+++ b/src/test/modules/test_aio/sql/ownership.sql
@@ -0,0 +1,65 @@
+-----
+-- IO handles
+----
+
+-- leak warning: implicit xact
+SELECT handle_get();
+
+-- leak warning: explicit xact
+BEGIN; SELECT handle_get(); COMMIT;
+
+-- leak warning + error: released in different command (thus resowner)
+BEGIN; SELECT handle_get(); SELECT handle_release_last(); COMMIT;
+
+-- no leak, same command
+BEGIN; SELECT handle_get() UNION ALL SELECT handle_release_last(); COMMIT;
+
+-- leak warning: subtrans
+BEGIN; SAVEPOINT foo; SELECT handle_get(); COMMIT;
+
+-- normal handle use
+SELECT handle_get_release();
+
+-- should error out, API violation
+SELECT handle_get_twice();
+
+-- recover after error in implicit xact
+SELECT handle_get_and_error(); SELECT handle_get_release();
+
+-- recover after error in explicit xact
+BEGIN; SELECT handle_get_and_error(); ROLLBACK; SELECT handle_get_release();
+
+-- recover after error in subtrans
+BEGIN; SAVEPOINT foo; SELECT handle_get_and_error(); ROLLBACK TO SAVEPOINT foo; SELECT handle_get_release(); ROLLBACK;
+
+
+-----
+-- Bounce Buffers handles
+----
+
+-- leak warning: implicit xact
+SELECT bb_get();
+
+-- leak warning: explicit xact
+BEGIN; SELECT bb_get(); COMMIT;
+
+-- missing leak warning: we should warn at command boundaries, not xact boundaries
+BEGIN; SELECT bb_get(); SELECT bb_release_last(); COMMIT;
+
+-- leak warning: subtrans
+BEGIN; SAVEPOINT foo; SELECT bb_get(); COMMIT;
+
+-- normal bb use
+SELECT bb_get_release();
+
+-- should error out, API violation
+SELECT bb_get_twice();
+
+-- recover after error in implicit xact
+SELECT bb_get_and_error(); SELECT bb_get_release();
+
+-- recover after error in explicit xact
+BEGIN; SELECT bb_get_and_error(); ROLLBACK; SELECT bb_get_release();
+
+-- recover after error in subtrans
+BEGIN; SAVEPOINT foo; SELECT bb_get_and_error(); ROLLBACK TO SAVEPOINT foo; SELECT bb_get_release(); ROLLBACK;
diff --git a/src/test/modules/test_aio/sql/prep.sql b/src/test/modules/test_aio/sql/prep.sql
new file mode 100644
index 00000000000..b8f225cbc98
--- /dev/null
+++ b/src/test/modules/test_aio/sql/prep.sql
@@ -0,0 +1,9 @@
+CREATE EXTENSION test_aio;
+
+CREATE TABLE tbl_a(data int not null);
+CREATE TABLE tbl_b(data int not null);
+
+INSERT INTO tbl_a SELECT generate_series(1, 10000);
+INSERT INTO tbl_b SELECT generate_series(1, 10000);
+SELECT grow_rel('tbl_a', 500);
+SELECT grow_rel('tbl_b', 550);
diff --git a/src/test/modules/test_aio/sync.conf b/src/test/modules/test_aio/sync.conf
new file mode 100644
index 00000000000..c480922d6cf
--- /dev/null
+++ b/src/test/modules/test_aio/sync.conf
@@ -0,0 +1,5 @@
+shared_preload_libraries=test_aio
+io_method = 'sync'
+log_min_messages = 'DEBUG3'
+log_statement=all
+restart_after_crash=false
diff --git a/src/test/modules/test_aio/test_aio--1.0.sql b/src/test/modules/test_aio/test_aio--1.0.sql
new file mode 100644
index 00000000000..ea9ad43ed8f
--- /dev/null
+++ b/src/test/modules/test_aio/test_aio--1.0.sql
@@ -0,0 +1,94 @@
+/* src/test/modules/test_aio/test_aio--1.0.sql */
+
+-- complain if script is sourced in psql, rather than via CREATE EXTENSION
+\echo Use "CREATE EXTENSION test_aio" to load this file. \quit
+
+
+CREATE FUNCTION grow_rel(rel regclass, nblocks int)
+RETURNS pg_catalog.void STRICT
+AS 'MODULE_PATHNAME' LANGUAGE C;
+
+
+CREATE FUNCTION corrupt_rel_block(rel regclass, blockno int)
+RETURNS pg_catalog.void STRICT
+AS 'MODULE_PATHNAME' LANGUAGE C;
+
+CREATE FUNCTION read_corrupt_rel_block(rel regclass, blockno int)
+RETURNS pg_catalog.void STRICT
+AS 'MODULE_PATHNAME' LANGUAGE C;
+
+CREATE FUNCTION invalidate_rel_block(rel regclass, blockno int)
+RETURNS pg_catalog.void STRICT
+AS 'MODULE_PATHNAME' LANGUAGE C;
+
+CREATE FUNCTION handle_get_and_error()
+RETURNS pg_catalog.void STRICT
+AS 'MODULE_PATHNAME' LANGUAGE C;
+
+CREATE FUNCTION handle_get_twice()
+RETURNS pg_catalog.void STRICT
+AS 'MODULE_PATHNAME' LANGUAGE C;
+
+CREATE FUNCTION handle_get()
+RETURNS pg_catalog.void STRICT
+AS 'MODULE_PATHNAME' LANGUAGE C;
+
+CREATE FUNCTION handle_get_release()
+RETURNS pg_catalog.void STRICT
+AS 'MODULE_PATHNAME' LANGUAGE C;
+
+CREATE FUNCTION handle_release_last()
+RETURNS pg_catalog.void STRICT
+AS 'MODULE_PATHNAME' LANGUAGE C;
+
+
+CREATE FUNCTION bb_get_and_error()
+RETURNS pg_catalog.void STRICT
+AS 'MODULE_PATHNAME' LANGUAGE C;
+
+CREATE FUNCTION bb_get_twice()
+RETURNS pg_catalog.void STRICT
+AS 'MODULE_PATHNAME' LANGUAGE C;
+
+CREATE FUNCTION bb_get()
+RETURNS pg_catalog.void STRICT
+AS 'MODULE_PATHNAME' LANGUAGE C;
+
+CREATE FUNCTION bb_get_release()
+RETURNS pg_catalog.void STRICT
+AS 'MODULE_PATHNAME' LANGUAGE C;
+
+CREATE FUNCTION bb_release_last()
+RETURNS pg_catalog.void STRICT
+AS 'MODULE_PATHNAME' LANGUAGE C;
+
+
+CREATE OR REPLACE FUNCTION redact(p_sql text)
+RETURNS bool
+LANGUAGE plpgsql
+AS $$
+ DECLARE
+ err_state text;
+ err_msg text;
+ BEGIN
+ EXECUTE p_sql;
+ RETURN true;
+ EXCEPTION WHEN OTHERS THEN
+ GET STACKED DIAGNOSTICS
+ err_state = RETURNED_SQLSTATE,
+ err_msg = MESSAGE_TEXT;
+ err_msg = regexp_replace(err_msg, '(file|relation) "?base/[0-9]+/[0-9]+"?', '\1 base/<redacted>');
+ RAISE NOTICE 'wrapped error: %', err_msg
+ USING ERRCODE = err_state;
+ RETURN false;
+ END;
+$$;
+
+
+CREATE FUNCTION inj_io_short_read_attach(result int)
+RETURNS pg_catalog.void STRICT
+AS 'MODULE_PATHNAME' LANGUAGE C;
+
+CREATE FUNCTION inj_io_short_read_detach()
+RETURNS pg_catalog.void STRICT
+AS 'MODULE_PATHNAME' LANGUAGE C;
diff --git a/src/test/modules/test_aio/test_aio.c b/src/test/modules/test_aio/test_aio.c
new file mode 100644
index 00000000000..9626d495241
--- /dev/null
+++ b/src/test/modules/test_aio/test_aio.c
@@ -0,0 +1,479 @@
+/*-------------------------------------------------------------------------
+ *
+ * delay_execution.c
+ * Test module to allow delay between parsing and execution of a query.
+ *
+ * The delay is implemented by taking and immediately releasing a specified
+ * advisory lock. If another process has previously taken that lock, the
+ * current process will be blocked until the lock is released; otherwise,
+ * there's no effect. This allows an isolationtester script to reliably
+ * test behaviors where some specified action happens in another backend
+ * between parsing and execution of any desired query.
+ *
+ * Copyright (c) 2020-2024, PostgreSQL Global Development Group
+ *
+ * IDENTIFICATION
+ * src/test/modules/delay_execution/delay_execution.c
+ *
+ *-------------------------------------------------------------------------
+ */
+
+#include "postgres.h"
+
+#include "fmgr.h"
+#include "storage/aio.h"
+#include "storage/aio_internal.h"
+#include "storage/bufmgr.h"
+#include "storage/buf_internals.h"
+#include "storage/lwlock.h"
+#include "storage/ipc.h"
+#include "access/relation.h"
+#include "utils/rel.h"
+#include "utils/injection_point.h"
+
+
+PG_MODULE_MAGIC;
+
+
+typedef struct InjIoErrorState
+{
+ bool enabled;
+ bool result_set;
+ int result;
+} InjIoErrorState;
+
+static InjIoErrorState * inj_io_error_state;
+
+/* Shared memory init callbacks */
+static shmem_request_hook_type prev_shmem_request_hook = NULL;
+static shmem_startup_hook_type prev_shmem_startup_hook = NULL;
+
+
+static PgAioHandle *last_handle;
+static PgAioBounceBuffer *last_bb;
+
+
+
+static void
+test_aio_shmem_request(void)
+{
+ if (prev_shmem_request_hook)
+ prev_shmem_request_hook();
+
+ RequestAddinShmemSpace(sizeof(InjIoErrorState));
+}
+
+static void
+test_aio_shmem_startup(void)
+{
+ bool found;
+
+ if (prev_shmem_startup_hook)
+ prev_shmem_startup_hook();
+
+ /* Create or attach to the shared memory state */
+ LWLockAcquire(AddinShmemInitLock, LW_EXCLUSIVE);
+
+ inj_io_error_state = ShmemInitStruct("injection_points",
+ sizeof(InjIoErrorState),
+ &found);
+
+ if (!found)
+ {
+ /*
+ * First time through, so initialize. This is shared with the dynamic
+ * initialization using a DSM.
+ */
+ inj_io_error_state->enabled = false;
+
+#ifdef USE_INJECTION_POINTS
+ InjectionPointAttach("AIO_PROCESS_COMPLETION_BEFORE_SHARED",
+ "test_aio",
+ "inj_io_short_read",
+ NULL,
+ 0);
+ InjectionPointLoad("AIO_PROCESS_COMPLETION_BEFORE_SHARED");
+#endif
+ }
+ else
+ {
+#ifdef USE_INJECTION_POINTS
+ InjectionPointLoad("AIO_PROCESS_COMPLETION_BEFORE_SHARED");
+ elog(LOG, "injection point loaded");
+#endif
+ }
+
+ LWLockRelease(AddinShmemInitLock);
+}
+
+void
+_PG_init(void)
+{
+ if (!process_shared_preload_libraries_in_progress)
+ return;
+
+ /* Shared memory initialization */
+ prev_shmem_request_hook = shmem_request_hook;
+ shmem_request_hook = test_aio_shmem_request;
+ prev_shmem_startup_hook = shmem_startup_hook;
+ shmem_startup_hook = test_aio_shmem_startup;
+}
+
+
+PG_FUNCTION_INFO_V1(grow_rel);
+Datum
+grow_rel(PG_FUNCTION_ARGS)
+{
+ Oid relid = PG_GETARG_OID(0);
+ uint32 nblocks = PG_GETARG_UINT32(1);
+ Relation rel;
+#define MAX_BUFFERS_TO_EXTEND_BY 64
+ Buffer victim_buffers[MAX_BUFFERS_TO_EXTEND_BY];
+
+ rel = relation_open(relid, AccessExclusiveLock);
+
+ while (nblocks > 0)
+ {
+ uint32 extend_by_pages;
+
+ extend_by_pages = Min(nblocks, MAX_BUFFERS_TO_EXTEND_BY);
+
+ ExtendBufferedRelBy(BMR_REL(rel),
+ MAIN_FORKNUM,
+ NULL,
+ 0,
+ extend_by_pages,
+ victim_buffers,
+ &extend_by_pages);
+
+ nblocks -= extend_by_pages;
+
+ for (uint32 i = 0; i < extend_by_pages; i++)
+ {
+ ReleaseBuffer(victim_buffers[i]);
+ }
+ }
+
+ relation_close(rel, NoLock);
+
+ PG_RETURN_VOID();
+}
+
+PG_FUNCTION_INFO_V1(corrupt_rel_block);
+Datum
+corrupt_rel_block(PG_FUNCTION_ARGS)
+{
+ Oid relid = PG_GETARG_OID(0);
+ uint32 block = PG_GETARG_UINT32(1);
+ Relation rel;
+ Buffer buf;
+ Page page;
+ PageHeader ph;
+
+ rel = relation_open(relid, AccessExclusiveLock);
+
+ buf = ReadBuffer(rel, block);
+ page = BufferGetPage(buf);
+
+ LockBuffer(buf, BUFFER_LOCK_EXCLUSIVE);
+
+ MarkBufferDirty(buf);
+
+ PageInit(page, BufferGetPageSize(buf), 0);
+
+ ph = (PageHeader) page;
+ ph->pd_special = BLCKSZ + 1;
+
+ FlushOneBuffer(buf);
+
+ LockBuffer(buf, BUFFER_LOCK_UNLOCK);
+
+ ReleaseBuffer(buf);
+
+ EvictUnpinnedBuffer(buf);
+
+ relation_close(rel, NoLock);
+
+ PG_RETURN_VOID();
+}
+
+PG_FUNCTION_INFO_V1(read_corrupt_rel_block);
+Datum
+read_corrupt_rel_block(PG_FUNCTION_ARGS)
+{
+ Oid relid = PG_GETARG_OID(0);
+ uint32 block = PG_GETARG_UINT32(1);
+ Relation rel;
+ Buffer buf;
+ BufferDesc *buf_hdr;
+ Page page;
+ PgAioHandle *ioh;
+ PgAioHandleRef ior;
+ SMgrRelation smgr;
+ uint32 buf_state;
+
+ rel = relation_open(relid, AccessExclusiveLock);
+
+ /* read buffer without erroring out */
+ buf = ReadBufferExtended(rel, MAIN_FORKNUM, block, RBM_ZERO_AND_LOCK, NULL);
+ LockBuffer(buf, BUFFER_LOCK_UNLOCK);
+
+ page = BufferGetBlock(buf);
+
+ ioh = pgaio_io_get(CurrentResourceOwner, NULL);
+ pgaio_io_get_ref(ioh, &ior);
+
+ buf_hdr = GetBufferDescriptor(buf - 1);
+ smgr = RelationGetSmgr(rel);
+
+ /* FIXME: even if just a test, we should verify nobody else uses this */
+ buf_state = LockBufHdr(buf_hdr);
+ buf_state &= ~(BM_VALID | BM_DIRTY);
+ UnlockBufHdr(buf_hdr, buf_state);
+
+ StartBufferIO(buf_hdr, true, false);
+
+ pgaio_io_set_io_data_32(ioh, (uint32 *) &buf, 1);
+ pgaio_io_add_shared_cb(ioh, ASC_SHARED_BUFFER_READ);
+
+ smgrstartreadv(ioh, smgr, MAIN_FORKNUM, block,
+ (void *) &page, 1);
+
+ ReleaseBuffer(buf);
+
+ pgaio_io_ref_wait(&ior);
+
+ relation_close(rel, NoLock);
+
+ PG_RETURN_VOID();
+}
+
+PG_FUNCTION_INFO_V1(invalidate_rel_block);
+Datum
+invalidate_rel_block(PG_FUNCTION_ARGS)
+{
+ Oid relid = PG_GETARG_OID(0);
+ uint32 block = PG_GETARG_UINT32(1);
+ Relation rel;
+ PrefetchBufferResult pr;
+ Buffer buf;
+
+ rel = relation_open(relid, AccessExclusiveLock);
+
+ /* this is a gross hack, but there's no good API exposed */
+ pr = PrefetchBuffer(rel, MAIN_FORKNUM, block);
+ buf = pr.recent_buffer;
+ elog(LOG, "recent: %d", buf);
+ if (BufferIsValid(buf))
+ {
+ /* if the buffer contents aren't valid, this'll return false */
+ if (ReadRecentBuffer(rel->rd_locator, MAIN_FORKNUM, block, buf))
+ {
+ LockBuffer(buf, BUFFER_LOCK_EXCLUSIVE);
+ FlushOneBuffer(buf);
+ LockBuffer(buf, BUFFER_LOCK_UNLOCK);
+ ReleaseBuffer(buf);
+
+ if (!EvictUnpinnedBuffer(buf))
+ elog(ERROR, "couldn't unpin");
+ }
+ }
+
+ relation_close(rel, AccessExclusiveLock);
+
+ PG_RETURN_VOID();
+}
+
+#if 0
+PG_FUNCTION_INFO_V1(test_unsubmitted_vs_close);
+Datum
+test_unsubmitted_vs_close(PG_FUNCTION_ARGS)
+{
+ Oid relid = PG_GETARG_OID(0);
+ uint32 block = PG_GETARG_UINT32(1);
+ Relation rel;
+ Buffer buf;
+ Page page;
+ PageHeader ph;
+
+ rel = relation_open(relid, AccessExclusiveLock);
+
+ buf = ReadBufferExtended(rel, MAIN_FORKNUM, block, RBM_ZERO_AND_LOCK, NULL);
+
+ buf = ReadBuffer(rel, block);
+ page = BufferGetPage(buf);
+
+ EvictUnpinnedBuffer(buf);
+
+ LockBuffer(buf, BUFFER_LOCK_UNLOCK);
+
+
+ MarkBufferDirty(buf);
+ ph->pd_special = BLCKSZ + 1;
+
+ /* last_handle = pgaio_io_get(); */
+
+ PG_RETURN_VOID();
+}
+#endif
+
+PG_FUNCTION_INFO_V1(handle_get);
+Datum
+handle_get(PG_FUNCTION_ARGS)
+{
+ last_handle = pgaio_io_get(CurrentResourceOwner, NULL);
+
+ PG_RETURN_VOID();
+}
+
+PG_FUNCTION_INFO_V1(handle_release_last);
+Datum
+handle_release_last(PG_FUNCTION_ARGS)
+{
+ if (!last_handle)
+ elog(ERROR, "no handle");
+
+ pgaio_io_release(last_handle);
+
+ PG_RETURN_VOID();
+}
+
+PG_FUNCTION_INFO_V1(handle_get_and_error);
+Datum
+handle_get_and_error(PG_FUNCTION_ARGS)
+{
+ pgaio_io_get(CurrentResourceOwner, NULL);
+
+ elog(ERROR, "as you command");
+ PG_RETURN_VOID();
+}
+
+PG_FUNCTION_INFO_V1(handle_get_twice);
+Datum
+handle_get_twice(PG_FUNCTION_ARGS)
+{
+ pgaio_io_get(CurrentResourceOwner, NULL);
+ pgaio_io_get(CurrentResourceOwner, NULL);
+
+ PG_RETURN_VOID();
+}
+
+
+PG_FUNCTION_INFO_V1(handle_get_release);
+Datum
+handle_get_release(PG_FUNCTION_ARGS)
+{
+ PgAioHandle *handle;
+
+ handle = pgaio_io_get(CurrentResourceOwner, NULL);
+ pgaio_io_release(handle);
+
+ PG_RETURN_VOID();
+}
+
+PG_FUNCTION_INFO_V1(bb_get);
+Datum
+bb_get(PG_FUNCTION_ARGS)
+{
+ last_bb = pgaio_bounce_buffer_get();
+
+ PG_RETURN_VOID();
+}
+
+PG_FUNCTION_INFO_V1(bb_release_last);
+Datum
+bb_release_last(PG_FUNCTION_ARGS)
+{
+ if (!last_bb)
+ elog(ERROR, "no bb");
+
+ pgaio_bounce_buffer_release(last_bb);
+
+ PG_RETURN_VOID();
+}
+
+PG_FUNCTION_INFO_V1(bb_get_and_error);
+Datum
+bb_get_and_error(PG_FUNCTION_ARGS)
+{
+ pgaio_bounce_buffer_get();
+
+ elog(ERROR, "as you command");
+ PG_RETURN_VOID();
+}
+
+PG_FUNCTION_INFO_V1(bb_get_twice);
+Datum
+bb_get_twice(PG_FUNCTION_ARGS)
+{
+ pgaio_bounce_buffer_get();
+ pgaio_bounce_buffer_get();
+
+ PG_RETURN_VOID();
+}
+
+
+PG_FUNCTION_INFO_V1(bb_get_release);
+Datum
+bb_get_release(PG_FUNCTION_ARGS)
+{
+ PgAioBounceBuffer *bb;
+
+ bb = pgaio_bounce_buffer_get();
+ pgaio_bounce_buffer_release(bb);
+
+ PG_RETURN_VOID();
+}
+
+#ifdef USE_INJECTION_POINTS
+extern PGDLLEXPORT void inj_io_short_read(const char *name, const void *private_data);
+
+void
+inj_io_short_read(const char *name, const void *private_data)
+{
+ PgAioHandle *ioh;
+
+ elog(LOG, "short read called: %d", inj_io_error_state->enabled);
+
+ if (inj_io_error_state->enabled)
+ {
+ ioh = pgaio_inj_io_get();
+
+ if (inj_io_error_state->result_set)
+ {
+ elog(LOG, "short read, changing result from %d to %d",
+ ioh->result, inj_io_error_state->result);
+
+ ioh->result = inj_io_error_state->result;
+ }
+ }
+}
+#endif
+
+PG_FUNCTION_INFO_V1(inj_io_short_read_attach);
+Datum
+inj_io_short_read_attach(PG_FUNCTION_ARGS)
+{
+#ifdef USE_INJECTION_POINTS
+ inj_io_error_state->enabled = true;
+ inj_io_error_state->result_set = !PG_ARGISNULL(0);
+ if (inj_io_error_state->result_set)
+ inj_io_error_state->result = PG_GETARG_INT32(0);
+#else
+ elog(ERROR, "injection points not supported");
+#endif
+
+ PG_RETURN_VOID();
+}
+
+PG_FUNCTION_INFO_V1(inj_io_short_read_detach);
+Datum
+inj_io_short_read_detach(PG_FUNCTION_ARGS)
+{
+#ifdef USE_INJECTION_POINTS
+ inj_io_error_state->enabled = false;
+#else
+ elog(ERROR, "injection points not supported");
+#endif
+ PG_RETURN_VOID();
+}
diff --git a/src/test/modules/test_aio/test_aio.control b/src/test/modules/test_aio/test_aio.control
new file mode 100644
index 00000000000..cd91c3ed16b
--- /dev/null
+++ b/src/test/modules/test_aio/test_aio.control
@@ -0,0 +1,3 @@
+comment = 'Test code for AIO'
+default_version = '1.0'
+module_pathname = '$libdir/test_aio'
diff --git a/src/test/modules/test_aio/worker.conf b/src/test/modules/test_aio/worker.conf
new file mode 100644
index 00000000000..8104c201924
--- /dev/null
+++ b/src/test/modules/test_aio/worker.conf
@@ -0,0 +1,5 @@
+shared_preload_libraries=test_aio
+io_method = 'worker'
+log_min_messages = 'DEBUG3'
+log_statement=all
+restart_after_crash=false
--
2.45.2.827.g557ae147e6
v2.1-0020-Temporary-Increase-BAS_BULKREAD-size.patch (text/x-diff, 1.3K) download
From 32ae60ce61cefe5c2e30341049e0c08b15e36de6 Mon Sep 17 00:00:00 2001
From: Andres Freund <andres@anarazel.de>
Date: Sun, 1 Sep 2024 00:42:27 -0400
Subject: [PATCH v2.1 20/20] Temporary: Increase BAS_BULKREAD size
Without this we only can execute very little AIO for sequential scans, as
there's just not enough buffers in the ring. This isn't the right fix, as
just increasing the ring size can have negative performance implications in
workloads where the kernel has all the data cached.
Author:
Reviewed-By:
Discussion: https://postgr.es/m/
Backpatch:
---
src/backend/storage/buffer/freelist.c | 6 +++++-
1 file changed, 5 insertions(+), 1 deletion(-)
diff --git a/src/backend/storage/buffer/freelist.c b/src/backend/storage/buffer/freelist.c
index dffdd57e9b5..5be8125ad3a 100644
--- a/src/backend/storage/buffer/freelist.c
+++ b/src/backend/storage/buffer/freelist.c
@@ -555,7 +555,11 @@ GetAccessStrategy(BufferAccessStrategyType btype)
return NULL;
case BAS_BULKREAD:
- ring_size_kb = 256;
+ /*
+ * FIXME: Temporary increase to allow large enough streaming reads
+ * to actually benefit from AIO. This needs a better solution.
+ */
+ ring_size_kb = 2 * 1024;
break;
case BAS_BULKWRITE:
ring_size_kb = 16 * 1024;
--
2.45.2.827.g557ae147e6
* Re: AIO v2.0
@ 2024-09-06 19:47 Andres Freund <andres@anarazel.de>
parent: 陈宗志 <baotiao@gmail.com>
0 siblings, 1 reply; 29+ messages in thread
From: Andres Freund @ 2024-09-06 19:47 UTC (permalink / raw)
To: 陈宗志 <baotiao@gmail.com>; +Cc: Heikki Linnakangas <hlinnaka@iki.fi>; pgsql-hackers
Hi,
On 2024-09-05 01:37:34 +0800, 陈宗志 wrote:
> I hope there can be a high-level design document that includes a
> description, high-level architecture, and low-level design.
> This way, others can also participate in reviewing the code.
Yep, that was already on my todo list. The version I just posted includes
that.
> For example, which paths were modified in the AIO module?
> Is it the path for writing WAL logs, or the path for flushing pages, etc.?
I don't think it's good to document this in a design document - that's just
bound to get out of date.
For now the patchset causes AIO to be used for
1) all users of read_stream.h, e.g. sequential scans
2) bgwriter / checkpointer, mainly to have way to exercise the write path. As
mentioned in my email upthread, the code for that is in a somewhat rough
shape as Thomas Munro is working on a more general abstraction for some of
this.
The earlier patchset added a lot more AIO uses because I needed to know all
the design constraints. It e.g. added AIO use in WAL. While that allowed me to
learn a lot, it's not something that makes sense to continue working on for
now, as it requires a lot of work that's independent of AIO. Thus I am
focusing on the above users for now.
> Also, I recommend keeping this patch as small as possible.
Yep. That's my goal (as mentioned upthread).
> For example, the first step could be to introduce libaio only, without
> considering io_uring, as that would make it too complex.
Currently the patchset doesn't contain libaio support and I am not planning to
work on using libaio. Nor do I think it makes sense for anybody else to do so
- libaio doesn't work for buffered IO, making it imo not particularly useful
for us.
The io_uring specific code isn't particularly complex / large compared to the
main AIO infrastructure.
Greetings,
Andres Freund
^ permalink raw reply [nested|flat] 29+ messages in thread
* Re: AIO v2.0
@ 2024-09-12 21:55 Robert Pang <robertpang@google.com>
parent: Andres Freund <andres@anarazel.de>
0 siblings, 1 reply; 29+ messages in thread
From: Robert Pang @ 2024-09-12 21:55 UTC (permalink / raw)
To: Andres Freund <andres@anarazel.de>; +Cc: 陈宗志 <baotiao@gmail.com>; Heikki Linnakangas <hlinnaka@iki.fi>; pgsql-hackers
Hi Andres
Thanks for the AIO patch update. I gave it a try and ran into a FATAL
in bgwriter when executing a benchmark.
2024-09-12 01:38:00.851 PDT [2780939] PANIC: no more bbs
2024-09-12 01:38:00.854 PDT [2780473] LOG: background writer process
(PID 2780939) was terminated by signal 6: Aborted
2024-09-12 01:38:00.854 PDT [2780473] LOG: terminating any other
active server processes
I debugged a bit and found that BgBufferSync() is not capping the
batch size under io_bounce_buffers like BufferSync() for checkpoint.
Here is a small patch to fix it.
Best regards
Robert
On Fri, Sep 6, 2024 at 12:47 PM Andres Freund <andres@anarazel.de> wrote:
>
> Hi,
>
> On 2024-09-05 01:37:34 +0800, 陈宗志 wrote:
> > I hope there can be a high-level design document that includes a
> > description, high-level architecture, and low-level design.
> > This way, others can also participate in reviewing the code.
>
> Yep, that was already on my todo list. The version I just posted includes
> that.
>
>
> > For example, which paths were modified in the AIO module?
> > Is it the path for writing WAL logs, or the path for flushing pages, etc.?
>
> I don't think it's good to document this in a design document - that's just
> bound to get out of date.
>
> For now the patchset causes AIO to be used for
>
> 1) all users of read_stream.h, e.g. sequential scans
>
> 2) bgwriter / checkpointer, mainly to have way to exercise the write path. As
> mentioned in my email upthread, the code for that is in a somewhat rough
> shape as Thomas Munro is working on a more general abstraction for some of
> this.
>
> The earlier patchset added a lot more AIO uses because I needed to know all
> the design constraints. It e.g. added AIO use in WAL. While that allowed me to
> learn a lot, it's not something that makes sense to continue working on for
> now, as it requires a lot of work that's independent of AIO. Thus I am
> focusing on the above users for now.
>
>
> > Also, I recommend keeping this patch as small as possible.
>
> Yep. That's my goal (as mentioned upthread).
>
>
> > For example, the first step could be to introduce libaio only, without
> > considering io_uring, as that would make it too complex.
>
> Currently the patchset doesn't contain libaio support and I am not planning to
> work on using libaio. Nor do I think it makes sense for anybody else to do so
> - libaio doesn't work for buffered IO, making it imo not particularly useful
> for us.
>
> The io_uring specific code isn't particularly complex / large compared to the
> main AIO infrastructure.
>
> Greetings,
>
> Andres Freund
>
>
Attachments:
0001-Fix-BgBufferSync-to-limit-batch-size-under-io_bounce.patch (text/x-patch, 1.4K) download
From bd04bd18ce62cf3f88568d3578503d4efeeb6603 Mon Sep 17 00:00:00 2001
From: Robert Pang <robertpang@google.com>
Date: Thu, 12 Sep 2024 14:36:16 -0700
Subject: [PATCH] Fix BgBufferSync to limit batch size under io_bounce_buffers
for bgwriter.
---
src/backend/storage/buffer/bufmgr.c | 5 ++++-
1 file changed, 4 insertions(+), 1 deletion(-)
diff --git a/src/backend/storage/buffer/bufmgr.c b/src/backend/storage/buffer/bufmgr.c
index 44b1b6fb31..4cd959b295 100644
--- a/src/backend/storage/buffer/bufmgr.c
+++ b/src/backend/storage/buffer/bufmgr.c
@@ -3396,6 +3396,7 @@ BgBufferSync(IOQueue *ioq, WritebackContext *wb_context)
uint32 new_recent_alloc;
BuffersToWrite to_write;
+ int max_combine;
/*
* Find out where the freelist clock sweep currently is, and how many
@@ -3417,6 +3418,8 @@ BgBufferSync(IOQueue *ioq, WritebackContext *wb_context)
return true;
}
+ max_combine = Min(io_bounce_buffers, io_combine_limit);
+
/*
* Compute strategy_delta = how many buffers have been scanned by the
* clock sweep since last time. If first time through, assume none. Then
@@ -3604,7 +3607,7 @@ BgBufferSync(IOQueue *ioq, WritebackContext *wb_context)
{
Assert(sync_state & BUF_REUSABLE);
- if (to_write.nbuffers == io_combine_limit)
+ if (to_write.nbuffers == max_combine)
{
WriteBuffers(&to_write, ioq, wb_context);
}
--
2.46.0.662.g92d0881bb0-goog
* Re: AIO v2.0
@ 2024-09-16 14:43 Noah Misch <noah@leadboat.com>
parent: Andres Freund <andres@anarazel.de>
1 sibling, 1 reply; 29+ messages in thread
From: Noah Misch @ 2024-09-16 14:43 UTC (permalink / raw)
To: Andres Freund <andres@anarazel.de>; +Cc: pgsql-hackers@postgresql.org, Heikki Linnakangas <hlinnaka@iki.fi>; 陈宗志 <baotiao@gmail.com>
On Fri, Sep 06, 2024 at 03:38:16PM -0400, Andres Freund wrote:
> There's plenty more to do, but I thought this would be a useful checkpoint.
I find patches 1-5 are Ready for Committer.
> +typedef enum PgAioHandleState
This enum clarified a lot for me, so I wish I had read it before anything
else. I recommend referring to it in README.md. Would you also cover the
valid state transitions and which of them any backend can do vs. which are
specific to the defining backend?
> +{
> + /* not in use */
> + AHS_IDLE = 0,
> +
> + /* returned by pgaio_io_get() */
> + AHS_HANDED_OUT,
> +
> + /* pgaio_io_start_*() has been called, but IO hasn't been submitted yet */
> + AHS_DEFINED,
> +
> + /* subjects prepare() callback has been called */
> + AHS_PREPARED,
> +
> + /* IO is being executed */
> + AHS_IN_FLIGHT,
Let's align terms between functions and states those functions reach. For
example, I recommend calling this state AHS_SUBMITTED, because
pgaio_io_prepare_submit() is the function reaching this state.
(Alternatively, use in_flight in the function name.)
> +
> + /* IO finished, but result has not yet been processed */
> + AHS_REAPED,
> +
> + /* IO completed, shared completion has been called */
> + AHS_COMPLETED_SHARED,
> +
> + /* IO completed, local completion has been called */
> + AHS_COMPLETED_LOCAL,
> +} PgAioHandleState;
> +void
> +pgaio_io_release_resowner(dlist_node *ioh_node, bool on_error)
> +{
> + PgAioHandle *ioh = dlist_container(PgAioHandle, resowner_node, ioh_node);
> +
> + Assert(ioh->resowner);
> +
> + ResourceOwnerForgetAioHandle(ioh->resowner, &ioh->resowner_node);
> + ioh->resowner = NULL;
> +
> + switch (ioh->state)
> + {
> + case AHS_IDLE:
> + elog(ERROR, "unexpected");
> + break;
> + case AHS_HANDED_OUT:
> + Assert(ioh == my_aio->handed_out_io || my_aio->handed_out_io == NULL);
> +
> + if (ioh == my_aio->handed_out_io)
> + {
> + my_aio->handed_out_io = NULL;
> + if (!on_error)
> + elog(WARNING, "leaked AIO handle");
> + }
> +
> + pgaio_io_reclaim(ioh);
> + break;
> + case AHS_DEFINED:
> + case AHS_PREPARED:
> + /* XXX: Should we warn about this when is_commit? */
Yes.
> + pgaio_submit_staged();
> + break;
> + case AHS_IN_FLIGHT:
> + case AHS_REAPED:
> + case AHS_COMPLETED_SHARED:
> + /* this is expected to happen */
> + break;
> + case AHS_COMPLETED_LOCAL:
> + /* XXX: unclear if this ought to be possible? */
> + pgaio_io_reclaim(ioh);
> + break;
> + }
> +void
> +pgaio_io_ref_wait(PgAioHandleRef *ior)
> +{
> + uint64 ref_generation;
> + PgAioHandleState state;
> + bool am_owner;
> + PgAioHandle *ioh;
> +
> + ioh = pgaio_io_from_ref(ior, &ref_generation);
> +
> + am_owner = ioh->owner_procno == MyProcNumber;
> +
> +
> + if (pgaio_io_was_recycled(ioh, ref_generation, &state))
> + return;
> +
> + if (am_owner)
> + {
> + if (state == AHS_DEFINED || state == AHS_PREPARED)
> + {
> + /* XXX: Arguably this should be prevented by callers? */
> + pgaio_submit_staged();
Agreed for AHS_DEFINED, if not both. AHS_DEFINED here would suggest a past
longjmp out of pgaio_io_prepare() w/o a subxact rollback to cleanup. Even so,
the next point might remove the need here:
> +void
> +pgaio_io_prepare(PgAioHandle *ioh, PgAioOp op)
> +{
> + Assert(ioh->state == AHS_HANDED_OUT);
> + Assert(pgaio_io_has_subject(ioh));
> +
> + ioh->op = op;
> + ioh->state = AHS_DEFINED;
> + ioh->result = 0;
> +
> + /* allow a new IO to be staged */
> + my_aio->handed_out_io = NULL;
> +
> + pgaio_io_prepare_subject(ioh);
> +
> + ioh->state = AHS_PREPARED;
As defense in depth, let's add a critical section from before assigning
AHS_DEFINED to here. This code already needs to be safe for that (per
README.md). When running outside a critical section, an ERROR in a subject
callback could leak the lwlock disowned in shared_buffer_prepare_common(). I
doubt there's a plausible way to reach that leak today, but future subject
callbacks could add risk over time.
> +if test "$with_liburing" = yes; then
> + PKG_CHECK_MODULES(LIBURING, liburing)
> +fi
I used the attached makefile patch to build w/ liburing.
> +pgaio_uring_shmem_init(bool first_time)
> +{
> + uint32 TotalProcs = MaxBackends + NUM_AUXILIARY_PROCS - MAX_IO_WORKERS;
> + bool found;
> +
> + aio_uring_contexts = (PgAioUringContext *)
> + ShmemInitStruct("AioUring", pgaio_uring_shmem_size(), &found);
> +
> + if (found)
> + return;
> +
> + for (int contextno = 0; contextno < TotalProcs; contextno++)
> + {
> + PgAioUringContext *context = &aio_uring_contexts[contextno];
> + int ret;
> +
> + /*
> + * XXX: Probably worth sharing the WQ between the different rings,
> + * when supported by the kernel. Could also cause additional
> + * contention, I guess?
> + */
> +#if 0
> + if (!AcquireExternalFD())
> + elog(ERROR, "No external FD available");
> +#endif
> + ret = io_uring_queue_init(io_max_concurrency, &context->io_uring_ring, 0);
With EXEC_BACKEND, "make check PG_TEST_INITDB_EXTRA_OPTS=-cio_method=io_uring"
fails early:
2024-09-15 12:46:08.168 PDT postmaster[2069397] LOG: starting PostgreSQL 18devel on x86_64-pc-linux-gnu, compiled by gcc (Debian 13.2.0-13) 13.2.0, 64-bit
2024-09-15 12:46:08.168 PDT postmaster[2069397] LOG: listening on Unix socket "/tmp/pg_regress-xgQOPH/.s.PGSQL.65312"
2024-09-15 12:46:08.203 PDT startup[2069423] LOG: database system was shut down at 2024-09-15 12:46:07 PDT
2024-09-15 12:46:08.209 PDT client backend[2069425] [unknown] FATAL: the database system is starting up
2024-09-15 12:46:08.222 PDT postmaster[2069397] LOG: database system is ready to accept connections
2024-09-15 12:46:08.254 PDT autovacuum launcher[2069435] PANIC: failed: -9/Bad file descriptor
2024-09-15 12:46:08.286 PDT client backend[2069444] [unknown] PANIC: failed: -95/Operation not supported
2024-09-15 12:46:08.355 PDT client backend[2069455] [unknown] PANIC: unexpected: -95/Operation not supported: No such file or directory
2024-09-15 12:46:08.370 PDT postmaster[2069397] LOG: received fast shutdown request
I expect that's from io_uring_queue_init() stashing in shared memory a file
descriptor and mmap address, which aren't valid in EXEC_BACKEND children.
Reattaching descriptors and memory in each child may work, or one could just
block io_method=io_uring under EXEC_BACKEND.
> +pgaio_uring_submit(uint16 num_staged_ios, PgAioHandle **staged_ios)
> +{
> + struct io_uring *uring_instance = &my_shared_uring_context->io_uring_ring;
> +
> + Assert(num_staged_ios <= PGAIO_SUBMIT_BATCH_SIZE);
> +
> + for (int i = 0; i < num_staged_ios; i++)
> + {
> + PgAioHandle *ioh = staged_ios[i];
> + struct io_uring_sqe *sqe;
> +
> + sqe = io_uring_get_sqe(uring_instance);
> +
> + pgaio_io_prepare_submit(ioh);
> + pgaio_uring_sq_from_io(ioh, sqe);
> + }
> +
> + while (true)
> + {
> + int ret;
> +
> + pgstat_report_wait_start(WAIT_EVENT_AIO_SUBMIT);
> + ret = io_uring_submit(uring_instance);
> + pgstat_report_wait_end();
> +
> + if (ret == -EINTR)
> + {
> + elog(DEBUG3, "submit EINTR, nios: %d", num_staged_ios);
> + continue;
> + }
Since io_uring_submit() is a wrapper around io_uring_enter(), this should also
retry on EAGAIN. "man io_uring_enter" has:
EAGAIN The kernel was unable to allocate memory for the request, or
otherwise ran out of resources to handle it. The application should wait
for some completions and try again.
> +FileStartWriteV(struct PgAioHandle *ioh, File file,
> + int iovcnt, off_t offset,
> + uint32 wait_event_info)
> +{
> + int returnCode;
> + Vfd *vfdP;
> +
> + Assert(FileIsValid(file));
> +
> + DO_DB(elog(LOG, "FileStartWriteV: %d (%s) " INT64_FORMAT " %d",
> + file, VfdCache[file].fileName,
> + (int64) offset,
> + iovcnt));
> +
> + returnCode = FileAccess(file);
> + if (returnCode < 0)
> + return returnCode;
> +
> + vfdP = &VfdCache[file];
> +
> + /* FIXME: think about / reimplement temp_file_limit */
> +
> + pgaio_io_prep_writev(ioh, vfdP->fd, iovcnt, offset);
> +
> + return 0;
> +}
FileStartWriteV() gets to state AHS_PREPARED, so let's align with the state
name by calling it FilePrepareWriteV (or FileWriteVPrepare or whatever).
For non-sync IO methods, I gather it's essential that a process other than the
IO definer be scanning for incomplete IOs and completing them. Otherwise,
deadlocks like this would happen:
backend1 locks blk1 for non-IO reasons
backend2 locks blk2, starts AIO write
backend1 waits for lock on blk2 for non-IO reasons
backend2 waits for lock on blk1 for non-IO reasons
If that's right, in worker mode, the IO worker resolves that deadlock. What
resolves it under io_uring? Another process that happens to do
pgaio_io_ref_wait() would dislodge things, but I didn't locate the code to
make that happen systematically. Could you add a mention of "deadlock" in the
comment at whichever code achieves that?
I could share more-tactical observations about patches 6-20, but they're
probably things you'd change without those observations. Is there any
specific decision you'd like to settle before patch 6 exits WIP?
Thanks,
nm
Attachments:
uring-makefile-v1.patch (text/plain, 841B) download
diff --git a/src/backend/Makefile b/src/backend/Makefile
index 84302cc..b123fdc 100644
--- a/src/backend/Makefile
+++ b/src/backend/Makefile
@@ -43,9 +43,10 @@ OBJS = \
$(top_builddir)/src/common/libpgcommon_srv.a \
$(top_builddir)/src/port/libpgport_srv.a
-# We put libpgport and libpgcommon into OBJS, so remove it from LIBS; also add
-# libldap and ICU
-LIBS := $(filter-out -lpgport -lpgcommon, $(LIBS)) $(LDAP_LIBS_BE) $(ICU_LIBS)
+# We put libpgport and libpgcommon into OBJS, so remove it from LIBS.
+LIBS := $(filter-out -lpgport -lpgcommon, $(LIBS))
+# The backend conditionally needs libraries that most executables don't need.
+LIBS += $(LDAP_LIBS_BE) $(ICU_LIBS) $(LIBURING_LIBS)
# The backend doesn't need everything that's in LIBS, however
LIBS := $(filter-out -lreadline -ledit -ltermcap -lncurses -lcurses, $(LIBS))
* Re: AIO v2.0
@ 2024-09-16 17:51 Andres Freund <andres@anarazel.de>
parent: Noah Misch <noah@leadboat.com>
0 siblings, 1 reply; 29+ messages in thread
From: Andres Freund @ 2024-09-16 17:51 UTC (permalink / raw)
To: Noah Misch <noah@leadboat.com>; +Cc: pgsql-hackers@postgresql.org, Heikki Linnakangas <hlinnaka@iki.fi>; 陈宗志 <baotiao@gmail.com>
Hi,
Thanks for the review!
On 2024-09-16 07:43:49 -0700, Noah Misch wrote:
> On Fri, Sep 06, 2024 at 03:38:16PM -0400, Andres Freund wrote:
> > There's plenty more to do, but I thought this would be a useful checkpoint.
>
> I find patches 1-5 are Ready for Committer.
Cool!
> > +typedef enum PgAioHandleState
>
> This enum clarified a lot for me, so I wish I had read it before anything
> else. I recommend referring to it in README.md.
Makes sense.
> Would you also cover the valid state transitions and which of them any
> backend can do vs. which are specific to the defining backend?
Yea, we should. I earlier had something, but because details were still
changing it was hard to keep up2date.
> > +{
> > + /* not in use */
> > + AHS_IDLE = 0,
> > +
> > + /* returned by pgaio_io_get() */
> > + AHS_HANDED_OUT,
> > +
> > + /* pgaio_io_start_*() has been called, but IO hasn't been submitted yet */
> > + AHS_DEFINED,
> > +
> > + /* subjects prepare() callback has been called */
> > + AHS_PREPARED,
> > +
> > + /* IO is being executed */
> > + AHS_IN_FLIGHT,
>
> Let's align terms between functions and states those functions reach. For
> example, I recommend calling this state AHS_SUBMITTED, because
> pgaio_io_prepare_submit() is the function reaching this state.
> (Alternatively, use in_flight in the function name.)
There used to be a separate SUBMITTED, but I removed it at some point as not
necessary anymore. Arguably it might be useful to re-introduce it so that
e.g. with worker mode one can tell the difference between the IO being queued
and the IO actually being processed.
> > +void
> > +pgaio_io_ref_wait(PgAioHandleRef *ior)
> > +{
> > + uint64 ref_generation;
> > + PgAioHandleState state;
> > + bool am_owner;
> > + PgAioHandle *ioh;
> > +
> > + ioh = pgaio_io_from_ref(ior, &ref_generation);
> > +
> > + am_owner = ioh->owner_procno == MyProcNumber;
> > +
> > +
> > + if (pgaio_io_was_recycled(ioh, ref_generation, &state))
> > + return;
> > +
> > + if (am_owner)
> > + {
> > + if (state == AHS_DEFINED || state == AHS_PREPARED)
> > + {
> > + /* XXX: Arguably this should be prevented by callers? */
> > + pgaio_submit_staged();
>
> Agreed for AHS_DEFINED, if not both. AHS_DEFINED here would suggest a past
> longjmp out of pgaio_io_prepare() w/o a subxact rollback to cleanup.
That, or not having submitted the IO. One thing I've been thinking about as
being potentially helpful infrastructure is to have something similar to a
critical section, except that it asserts that one is not allowed to block or
forget submitting staged IOs.
> > +void
> > +pgaio_io_prepare(PgAioHandle *ioh, PgAioOp op)
> > +{
> > + Assert(ioh->state == AHS_HANDED_OUT);
> > + Assert(pgaio_io_has_subject(ioh));
> > +
> > + ioh->op = op;
> > + ioh->state = AHS_DEFINED;
> > + ioh->result = 0;
> > +
> > + /* allow a new IO to be staged */
> > + my_aio->handed_out_io = NULL;
> > +
> > + pgaio_io_prepare_subject(ioh);
> > +
> > + ioh->state = AHS_PREPARED;
>
> As defense in depth, let's add a critical section from before assigning
> AHS_DEFINED to here. This code already needs to be safe for that (per
> README.md). When running outside a critical section, an ERROR in a subject
> callback could leak the lwlock disowned in shared_buffer_prepare_common(). I
> doubt there's a plausible way to reach that leak today, but future subject
> callbacks could add risk over time.
Makes sense.
> > +if test "$with_liburing" = yes; then
> > + PKG_CHECK_MODULES(LIBURING, liburing)
> > +fi
>
> I used the attached makefile patch to build w/ liburing.
Thanks, will incorporate.
> With EXEC_BACKEND, "make check PG_TEST_INITDB_EXTRA_OPTS=-cio_method=io_uring"
> fails early:
Right - that's to be expected.
> 2024-09-15 12:46:08.168 PDT postmaster[2069397] LOG: starting PostgreSQL 18devel on x86_64-pc-linux-gnu, compiled by gcc (Debian 13.2.0-13) 13.2.0, 64-bit
> 2024-09-15 12:46:08.168 PDT postmaster[2069397] LOG: listening on Unix socket "/tmp/pg_regress-xgQOPH/.s.PGSQL.65312"
> 2024-09-15 12:46:08.203 PDT startup[2069423] LOG: database system was shut down at 2024-09-15 12:46:07 PDT
> 2024-09-15 12:46:08.209 PDT client backend[2069425] [unknown] FATAL: the database system is starting up
> 2024-09-15 12:46:08.222 PDT postmaster[2069397] LOG: database system is ready to accept connections
> 2024-09-15 12:46:08.254 PDT autovacuum launcher[2069435] PANIC: failed: -9/Bad file descriptor
> 2024-09-15 12:46:08.286 PDT client backend[2069444] [unknown] PANIC: failed: -95/Operation not supported
> 2024-09-15 12:46:08.355 PDT client backend[2069455] [unknown] PANIC: unexpected: -95/Operation not supported: No such file or directory
> 2024-09-15 12:46:08.370 PDT postmaster[2069397] LOG: received fast shutdown request
>
> I expect that's from io_uring_queue_init() stashing in shared memory a file
> descriptor and mmap address, which aren't valid in EXEC_BACKEND children.
> Reattaching descriptors and memory in each child may work, or one could just
> block io_method=io_uring under EXEC_BACKEND.
I think the latter option is saner - I don't think there's anything to be
gained by supporting io_uring in this situation. It's not like anybody will
use it for real-world workloads where performance matters. Nor would it be
useful fo portability testing.
> > +pgaio_uring_submit(uint16 num_staged_ios, PgAioHandle **staged_ios)
> > +{
> > + if (ret == -EINTR)
> > + {
> > + elog(DEBUG3, "submit EINTR, nios: %d", num_staged_ios);
> > + continue;
> > + }
>
> Since io_uring_submit() is a wrapper around io_uring_enter(), this should also
> retry on EAGAIN. "man io_uring_enter" has:
>
> EAGAIN The kernel was unable to allocate memory for the request, or
> otherwise ran out of resources to handle it. The application should wait
> for some completions and try again.
Hm. I'm not sure that makes sense. We only allow a limited number of IOs to be
in flight for each uring instance. That's different to a use of uring to
e.g. wait for incoming network data on thousands of sockets, where you could
have essentially unbounded amount of requests outstanding.
What would we wait for? What if we were holding a critical lock in that
moment? Would it be safe to just block for some completions? What if there's
actually no IO in progress?
> > +FileStartWriteV(struct PgAioHandle *ioh, File file,
> > + int iovcnt, off_t offset,
> > + uint32 wait_event_info)
> > +{
> > ...
>
> FileStartWriteV() gets to state AHS_PREPARED, so let's align with the state
> name by calling it FilePrepareWriteV (or FileWriteVPrepare or whatever).
Hm - that doesn't necessarily seem right to me. I don't think the caller
should assume that the IO will just be prepared and not already completed by
the time FileStartWriteV() returns - we might actually do the IO
synchronously.
> For non-sync IO methods, I gather it's essential that a process other than the
> IO definer be scanning for incomplete IOs and completing them.
Yep - it's something I've been fighting with / redesigning a *lot*. Earlier
the AIO subsystem could transparently retry IOs, but that ends up being a
nightmare - or at least I couldn't find a way to not make it a
nightmare. There are two main complexities:
1) What if the IO is being completed in a critical section? We can't reopen
the file in that situation. My initial fix for this was to defer retries,
but that's problematic too:
2) Acquiring an IO needs to be able to guarantee forward progress. Because
there's a limited number of IOs that means we need to be able to complete
IOs while acquiring an IO. So we can't just keep the IO handle around -
which in turn means that we'd need to save the state for retrying
somewhere. Which would require some pre-allocated memory to save that
state.
Thus I think it's actually better if we delegate retries to the callsites. I
was thinking that for partial reads of shared buffers we ought to not set
BM_IO_ERROR though...
> Otherwise, deadlocks like this would happen:
> backend1 locks blk1 for non-IO reasons
> backend2 locks blk2, starts AIO write
> backend1 waits for lock on blk2 for non-IO reasons
> backend2 waits for lock on blk1 for non-IO reasons
>
> If that's right, in worker mode, the IO worker resolves that deadlock. What
> resolves it under io_uring? Another process that happens to do
> pgaio_io_ref_wait() would dislodge things, but I didn't locate the code to
> make that happen systematically.
Yea, it's code that I haven't forward ported yet. I think basically
LockBuffer[ForCleanup] ought to call pgaio_io_ref_wait() when it can't
immediately acquire the lock and if the buffer has IO going on.
> I could share more-tactical observations about patches 6-20, but they're
> probably things you'd change without those observations.
Agreed.
> Is there any specific decision you'd like to settle before patch 6 exits
> WIP?
Patch 6 specifically? That I really mainly kept separate for review - it
doesn't seem particularly interesting to commit it earlier than 7, or do you
think differently?
In case you mean 6+7 or 6 to ~11, I can think of the following:
- I am worried about the need for bounce buffers for writes of checksummed
buffers. That quickly ends up being a significant chunk of memory,
particularly when using a small shared_buffers with a higher than default
number of connection. I'm currently hacking up a prototype that'd prevent us
from setting hint bits with just a share lock. I'm planning to start a
separate thread about that.
- The header split doesn't yet quite seem right yet
- I'd like to implement retries in the later patches, to make sure that it
doesn't have design implications
- Worker mode needs to be able to automatically adjust the number of running
workers, I think - otherwise it's going to be too hard to tune.
- I think the PgAioHandles need to be slimmed down a bit - there's some design
evolution visible that should not end up in the tree.
- I'm not sure that I like name "subject" for the different things AIO is
performed for
- I am wondering if the need for pgaio_io_set_io_data_32() (to store the set
of buffer ids that are affected by one IO) could be replaced by repurposing
BufferDesc->freeNext or something along those lines. I don't like the amount
of memory required for storing those arrays, even if it's not that much
compared to needing space to store struct iovec[PG_IOV_MAX] for each AIO
handle.
- I'd like to extend the test module to actually test more cases, it's too
hard to reach some paths, particularly without [a lot] of users yet. That's
not strictly a dependency of the earlier patches - since the initial patches
can't actually do much in the way of IO.
- We shouldn't reserve AioHandles etc for io workers - but because different
tpes of aux processes don't use a predetermined ProcNumber, that's not
entirely trivial without adding more complexity. I've actually wondered
whether IO workes should be their own "top-level" kind of process, rather
than an aux process. But that seems quite costly.
- Right now the io_uring mode has each backend's io_uring instance visible to
each other process. That ends up using a fair number of FDs. That's OK from
an efficiency perspective, but I think we'd need to add code to adjust the
soft RLIMIT_NOFILE (it's set to 1024 on most distros because there are
various programs that iterate over all possible FDs, causing significant
slowdowns when the soft limit defaults to something high). I earlier had a
limited number of io_uring instances, but that added a fair amount of
overhead because then submitting IO would require a lock.
That again doesn't have to be solved as part of the earlier patches but
might have some minor design impact.
Thanks again,
Andres Freund
^ permalink raw reply [nested|flat] 29+ messages in thread
* Re: AIO v2.0
@ 2024-09-16 17:56 Andres Freund <andres@anarazel.de>
parent: Robert Pang <robertpang@google.com>
0 siblings, 0 replies; 29+ messages in thread
From: Andres Freund @ 2024-09-16 17:56 UTC (permalink / raw)
To: Robert Pang <robertpang@google.com>; +Cc: 陈宗志 <baotiao@gmail.com>; Heikki Linnakangas <hlinnaka@iki.fi>; pgsql-hackers
Hi,
On 2024-09-12 14:55:49 -0700, Robert Pang wrote:
> Hi Andres
>
> Thanks for the AIO patch update. I gave it a try and ran into a FATAL
> in bgwriter when executing a benchmark.
>
> 2024-09-12 01:38:00.851 PDT [2780939] PANIC: no more bbs
> 2024-09-12 01:38:00.854 PDT [2780473] LOG: background writer process
> (PID 2780939) was terminated by signal 6: Aborted
> 2024-09-12 01:38:00.854 PDT [2780473] LOG: terminating any other
> active server processes
>
> I debugged a bit and found that BgBufferSync() is not capping the
> batch size under io_bounce_buffers like BufferSync() for checkpoint.
> Here is a small patch to fix it.
Good catch, thanks!
I am hoping (as described in my email to Noah a few minutes ago) that we can
get away from needing bounce buffers. They are a quite expensive solution to a
problem we made for ourselves...
Greetings,
Andres Freund
^ permalink raw reply [nested|flat] 29+ messages in thread
* Re: AIO v2.0
@ 2024-09-17 18:08 Noah Misch <noah@leadboat.com>
parent: Andres Freund <andres@anarazel.de>
0 siblings, 1 reply; 29+ messages in thread
From: Noah Misch @ 2024-09-17 18:08 UTC (permalink / raw)
To: Andres Freund <andres@anarazel.de>; +Cc: pgsql-hackers@postgresql.org, Heikki Linnakangas <hlinnaka@iki.fi>; 陈宗志 <baotiao@gmail.com>
On Mon, Sep 16, 2024 at 01:51:42PM -0400, Andres Freund wrote:
> On 2024-09-16 07:43:49 -0700, Noah Misch wrote:
> > On Fri, Sep 06, 2024 at 03:38:16PM -0400, Andres Freund wrote:
> > Reattaching descriptors and memory in each child may work, or one could just
> > block io_method=io_uring under EXEC_BACKEND.
>
> I think the latter option is saner
Works for me.
> > > +pgaio_uring_submit(uint16 num_staged_ios, PgAioHandle **staged_ios)
> > > +{
>
> > > + if (ret == -EINTR)
> > > + {
> > > + elog(DEBUG3, "submit EINTR, nios: %d", num_staged_ios);
> > > + continue;
> > > + }
> >
> > Since io_uring_submit() is a wrapper around io_uring_enter(), this should also
> > retry on EAGAIN. "man io_uring_enter" has:
> >
> > EAGAIN The kernel was unable to allocate memory for the request, or
> > otherwise ran out of resources to handle it. The application should wait
> > for some completions and try again.
>
> Hm. I'm not sure that makes sense. We only allow a limited number of IOs to be
> in flight for each uring instance. That's different to a use of uring to
> e.g. wait for incoming network data on thousands of sockets, where you could
> have essentially unbounded amount of requests outstanding.
>
> What would we wait for? What if we were holding a critical lock in that
> moment? Would it be safe to just block for some completions? What if there's
> actually no IO in progress?
I'd try the following. First, scan for all IOs of all processes at
AHS_DEFINED and later, advancing them to AHS_COMPLETED_SHARED. This might be
unsafe today, but discovering why it's unsafe likely will inform design beyond
EAGAIN returns. I don't specifically know of a way it's unsafe. Do just one
pass of that; there may be newer IOs in progress afterward. If submit still
gets EAGAIN, sleep a bit and retry. Like we do in pgwin32_open_handle(), fail
after a fixed number of iterations. This isn't great if we hold a critical
lock, but it beats the alternative of PANIC on the first EAGAIN.
> > > +FileStartWriteV(struct PgAioHandle *ioh, File file,
> > > + int iovcnt, off_t offset,
> > > + uint32 wait_event_info)
> > > +{
> > > ...
> >
> > FileStartWriteV() gets to state AHS_PREPARED, so let's align with the state
> > name by calling it FilePrepareWriteV (or FileWriteVPrepare or whatever).
>
> Hm - that doesn't necessarily seem right to me. I don't think the caller
> should assume that the IO will just be prepared and not already completed by
> the time FileStartWriteV() returns - we might actually do the IO
> synchronously.
Yes. Even if it doesn't become synchronous IO, some other process may advance
the IO to AHS_COMPLETED_SHARED by the next wake-up of the process that defined
the IO. Still, I think this shouldn't use the term "Start" while no state
name uses that term. What else could remove that mismatch?
> > Is there any specific decision you'd like to settle before patch 6 exits
> > WIP?
>
> Patch 6 specifically? That I really mainly kept separate for review - it
No. I'll rephrase as "Is there any specific decision you'd like to settle
before the next cohort of patches exits WIP?"
> doesn't seem particularly interesting to commit it earlier than 7, or do you
> think differently?
No, I agree a lone commit of 6 isn't a win. Roughly, the eight patches
6-9,12-15 could be a minimal attractive unit. I've not thought through that
grouping much.
> In case you mean 6+7 or 6 to ~11, I can think of the following:
>
> - I am worried about the need for bounce buffers for writes of checksummed
> buffers. That quickly ends up being a significant chunk of memory,
> particularly when using a small shared_buffers with a higher than default
> number of connection. I'm currently hacking up a prototype that'd prevent us
> from setting hint bits with just a share lock. I'm planning to start a
> separate thread about that.
AioChooseBounceBuffers() limits usage to 256 blocks (2MB) per MaxBackends.
Doing better is nice, but I don't consider this a blocker. I recommend
dealing with the worry by reducing the limit initially (128 blocks?). Can
always raise it later.
> - The header split doesn't yet quite seem right yet
I won't have a strong opinion on that one. The aio.c/aio_io.c split did catch
my attention. I made a note to check it again once those files get header
comments.
> - I'd like to implement retries in the later patches, to make sure that it
> doesn't have design implications
Yes, that's a blocker to me.
> - Worker mode needs to be able to automatically adjust the number of running
> workers, I think - otherwise it's going to be too hard to tune.
Changing that later wouldn't affect much else, so I'd not consider it a
blocker. (The worst case is that we think the initial AIO release will be a
loss for most users, so we wrap it in debug_ terminology like we did for
debug_io_direct. I'm not saying worker scaling will push AIO from one side of
that line to another, but that's why I'm fine with commits that omit
self-contained optimizations.)
> - I think the PgAioHandles need to be slimmed down a bit - there's some design
> evolution visible that should not end up in the tree.
Okay.
> - I'm not sure that I like name "subject" for the different things AIO is
> performed for
How about one of these six terms:
- listener, observer [if you view smgr as an observer of IOs in the sense of https://en.wikipedia.org/wiki/Observer_pattern]
- class, subclass, type, tag [if you view an SmgrIO as a subclass of an IO, in the object-oriented sense]
> - I am wondering if the need for pgaio_io_set_io_data_32() (to store the set
> of buffer ids that are affected by one IO) could be replaced by repurposing
> BufferDesc->freeNext or something along those lines. I don't like the amount
> of memory required for storing those arrays, even if it's not that much
> compared to needing space to store struct iovec[PG_IOV_MAX] for each AIO
> handle.
Here too, changing that later wouldn't affect much else, so I'd not consider
it a blocker.
> - I'd like to extend the test module to actually test more cases, it's too
> hard to reach some paths, particularly without [a lot] of users yet. That's
> not strictly a dependency of the earlier patches - since the initial patches
> can't actually do much in the way of IO.
Agreed. Among the post-patch check-world coverage, which uncovered parts have
the most risk?
> - We shouldn't reserve AioHandles etc for io workers - but because different
> tpes of aux processes don't use a predetermined ProcNumber, that's not
> entirely trivial without adding more complexity. I've actually wondered
> whether IO workes should be their own "top-level" kind of process, rather
> than an aux process. But that seems quite costly.
Here too, changing that later wouldn't affect much else, so I'd not consider
it a blocker. Of these ones I'm calling non-blockers, which would you most
regret deferring?
> - Right now the io_uring mode has each backend's io_uring instance visible to
> each other process. That ends up using a fair number of FDs. That's OK from
> an efficiency perspective, but I think we'd need to add code to adjust the
> soft RLIMIT_NOFILE (it's set to 1024 on most distros because there are
> various programs that iterate over all possible FDs, causing significant
Agreed on raising the soft limit. Docs and/or errhint() likely will need to
mention system configuration nonetheless, since some users will encounter
RLIMIT_MEMLOCK or /proc/sys/kernel/io_uring_disabled.
> slowdowns when the soft limit defaults to something high). I earlier had a
> limited number of io_uring instances, but that added a fair amount of
> overhead because then submitting IO would require a lock.
>
> That again doesn't have to be solved as part of the earlier patches but
> might have some minor design impact.
How far do you see the design impact spreading on that one?
Thanks,
nm
^ permalink raw reply [nested|flat] 29+ messages in thread
* Re: AIO v2.0
@ 2024-09-30 14:49 Andres Freund <andres@anarazel.de>
parent: Noah Misch <noah@leadboat.com>
0 siblings, 2 replies; 29+ messages in thread
From: Andres Freund @ 2024-09-30 14:49 UTC (permalink / raw)
To: Noah Misch <noah@leadboat.com>; +Cc: pgsql-hackers@postgresql.org, Heikki Linnakangas <hlinnaka@iki.fi>; 陈宗志 <baotiao@gmail.com>
Hi,
On 2024-09-17 11:08:19 -0700, Noah Misch wrote:
> > - I am worried about the need for bounce buffers for writes of checksummed
> > buffers. That quickly ends up being a significant chunk of memory,
> > particularly when using a small shared_buffers with a higher than default
> > number of connection. I'm currently hacking up a prototype that'd prevent us
> > from setting hint bits with just a share lock. I'm planning to start a
> > separate thread about that.
>
> AioChooseBounceBuffers() limits usage to 256 blocks (2MB) per MaxBackends.
> Doing better is nice, but I don't consider this a blocker. I recommend
> dealing with the worry by reducing the limit initially (128 blocks?). Can
> always raise it later.
On storage that has nontrivial latency, like just about all cloud storage,
even 256 will be too low. Particularly for checkpointer.
Assuming 1ms latency - which isn't the high end of cloud storage latency - 256
blocks in flight limits you to <= 256MByte/s, even on storage that can have a
lot more throughput. With 3ms, which isn't uncommon, it's 85MB/s.
Of course this could be addressed by tuning, but it seems like something that
shouldn't need to be tuned by the majority of folks running postgres.
We also discussed the topic at https://postgr.es/m/20240925020022.c5.nmisch%40google.com
> ... neither BM_SETTING_HINTS nor keeping bounce buffers looks like a bad
> decision. From what I've heard so far of the performance effects, if it were
> me, I would keep the bounce buffers. I'd pursue BM_SETTING_HINTS and bounce
> buffer removal as a distinct project after the main AIO capability. Bounce
> buffers have an implementation. They aren't harming other design decisions.
> The AIO project is big, so I'd want to err on the side of not designating
> other projects as its prerequisites.
Given the issues that modifying pages while in flight causes, not just with PG
level checksums, but also filesystem level checksum, I don't feel like it's a
particularly promising approach.
However, I think this doesn't have to mean that the BM_SETTING_HINTS stuff has
to be completed before we can move forward with AIO. If I split out the write
portion from the read portion a bit further, the main AIO changes and the
shared-buffer read user can be merged before there's a dependency on the hint
bit stuff being done.
Does that seem reasonable?
Greetings,
Andres Freund
^ permalink raw reply [nested|flat] 29+ messages in thread
* Re: AIO v2.0
@ 2024-09-30 15:55 Matthias van de Meent <boekewurm+postgres@gmail.com>
parent: Andres Freund <andres@anarazel.de>
1 sibling, 0 replies; 29+ messages in thread
From: Matthias van de Meent @ 2024-09-30 15:55 UTC (permalink / raw)
To: Andres Freund <andres@anarazel.de>; +Cc: pgsql-hackers@postgresql.org, Noah Misch <noah@leadboat.com>; Heikki Linnakangas <hlinnaka@iki.fi>; 陈宗志 <baotiao@gmail.com>
On Mon, 30 Sept 2024 at 16:49, Andres Freund <andres@anarazel.de> wrote:
> On 2024-09-17 11:08:19 -0700, Noah Misch wrote:
> > > - I am worried about the need for bounce buffers for writes of checksummed
> > > buffers. That quickly ends up being a significant chunk of memory,
> > > particularly when using a small shared_buffers with a higher than default
> > > number of connection. I'm currently hacking up a prototype that'd prevent us
> > > from setting hint bits with just a share lock. I'm planning to start a
> > > separate thread about that.
> >
> > AioChooseBounceBuffers() limits usage to 256 blocks (2MB) per MaxBackends.
> > Doing better is nice, but I don't consider this a blocker. I recommend
> > dealing with the worry by reducing the limit initially (128 blocks?). Can
> > always raise it later.
>
> On storage that has nontrivial latency, like just about all cloud storage,
> even 256 will be too low. Particularly for checkpointer.
>
> Assuming 1ms latency - which isn't the high end of cloud storage latency - 256
> blocks in flight limits you to <= 256MByte/s, even on storage that can have a
> lot more throughput. With 3ms, which isn't uncommon, it's 85MB/s.
FYI, I think you're off by a factor 8, i.e. that would be 2GB/sec and
666MB/sec respectively, given a normal page size of 8kB and exactly
1ms/3ms full round trip latency:
1 page/1 ms * 8kB/page * 256 concurrency = 256 pages/ms * 8kB/page =
2MiB/ms ~= 2GiB/sec.
for 3ms divide by 3 -> ~666MiB/sec.
Kind regards,
Matthias van de Meent
Neon (https://neon.tech)
^ permalink raw reply [nested|flat] 29+ messages in thread
* Re: AIO v2.0
@ 2024-09-30 19:29 Noah Misch <noah@leadboat.com>
parent: Andres Freund <andres@anarazel.de>
1 sibling, 0 replies; 29+ messages in thread
From: Noah Misch @ 2024-09-30 19:29 UTC (permalink / raw)
To: Andres Freund <andres@anarazel.de>; +Cc: pgsql-hackers@postgresql.org, Heikki Linnakangas <hlinnaka@iki.fi>; 陈宗志 <baotiao@gmail.com>
On Mon, Sep 30, 2024 at 10:49:17AM -0400, Andres Freund wrote:
> We also discussed the topic at https://postgr.es/m/20240925020022.c5.nmisch%40google.com
> > ... neither BM_SETTING_HINTS nor keeping bounce buffers looks like a bad
> > decision. From what I've heard so far of the performance effects, if it were
> > me, I would keep the bounce buffers. I'd pursue BM_SETTING_HINTS and bounce
> > buffer removal as a distinct project after the main AIO capability. Bounce
> > buffers have an implementation. They aren't harming other design decisions.
> > The AIO project is big, so I'd want to err on the side of not designating
> > other projects as its prerequisites.
>
> Given the issues that modifying pages while in flight causes, not just with PG
> level checksums, but also filesystem level checksum, I don't feel like it's a
> particularly promising approach.
>
> However, I think this doesn't have to mean that the BM_SETTING_HINTS stuff has
> to be completed before we can move forward with AIO. If I split out the write
> portion from the read portion a bit further, the main AIO changes and the
> shared-buffer read user can be merged before there's a dependency on the hint
> bit stuff being done.
>
> Does that seem reasonable?
Yes.
^ permalink raw reply [nested|flat] 29+ messages in thread
* Re: AIO v2.0
@ 2024-11-18 12:19 Jakub Wartak <jakub.wartak@enterprisedb.com>
parent: Andres Freund <andres@anarazel.de>
1 sibling, 1 reply; 29+ messages in thread
From: Jakub Wartak @ 2024-11-18 12:19 UTC (permalink / raw)
To: Andres Freund <andres@anarazel.de>; +Cc: pgsql-hackers@postgresql.org, Heikki Linnakangas <hlinnaka@iki.fi>; 陈宗志 <baotiao@gmail.com>
On Fri, Sep 6, 2024 at 9:38 PM Andres Freund <andres@anarazel.de> wrote:
> Hi,
>
> Attached is the next version of the patchset. (..)
Hi Andres,
Thank You for worth admiring persistence on this. Please do not take it as
criticism, just more like set of questions regarding the patchset v2.1 that
I finally got little time to play with:
0. Doesn't the v2.1-0011-aio-Add-io_uring-method.patch -> in
pgaio_uring_submit() -> io_uring_get_sqe() need a return value check ?
Otherwise we'll never know that SQ is full in theory, perhaps at least such
a check should be made with Assert() ? (I understand right now that we
allow just up to io_uring_queue_init(io_max_concurrency), but what happens
if:
a. previous io_uring_submit() failed for some reason and we do not have
free space for SQ?
b. (hypothetical) someday someone will try to make PG multithreaded and the
code starts using just one big queue, still without checking for
io_uring_get_sqe()?
1. In [0] you wrote that there's this high amount of FDs consumed for
io_uring (dangerously close to RLIMIT_NOFILE). I can attest that there are
many customers who are using extremely high max_connections (4k-5k, but
there outliers with 10k in the wild too) - so they won't even start - and I
have one doubt on the user-friendliness impact of this. I'm quite certain
it's going to be the same as with pgbouncer where one is forced to tweak
OS(systemd/pam/limits.conf/etc), but in PG we are better because PG tries
to preallocate and then close() a lot of FDs, so that's safer in runtime.
IMVHO even if we just consume e.g. say > 30% of FDs just for io_uring, the
max_files_per_process looses it's spirit a little bit and PG is going to
start loose efficiency too due to frequent open()/close() calls as fd cache
is too small. Tomas also complained about it some time ago in [1])
So maybe it would be good to introduce couple of sanity checks too (even
after setting higher limit):
- issue FATAL in case of using io_method = io_ring && max_connections would
be close to getrusage(RLIMIT_NOFILE)
- issue warning in case of using io_method = io_ring && we wouldnt have
even real 1k FDs free for handling relation FDs (detect something bad like:
getrusage(RLIMIT_NOFILE) <= max_connections + max_files_per_process)
2. In pgaio_uring_postmaster_child_init_local() there
"io_uring_queue_init(32,...)" - why 32? :) And also there's separate
io_uring_queue_init(io_max_concurrency) which seems to be derived from
AioChooseMaxConccurrency() which can go up to 64?
3. I find having two GUCs named literally the same
(effective_io_concurrency, io_max_concurrency). It is clear from IO_URING
perspective what is io_max_concurrency all about, but I bet having also
effective_io_concurrency in the mix is going to be a little confusing for
users (well, it is to me). Maybe that README.md could elaborate a little
bit on the relation between those two? Or maybe do you plan to remove
io_max_concurrency and bind it to effective_io_concurrency in future? To
add more fun , there's MAX_IO_CONCURRENCY nearby in v2.1-0014 too while the
earlier mentioned AioChooseMaxConccurrency() goes up to just 64
4. While we are at this, shouldn't the patch rename debug_io_direct to
simply io_direct so that GUCs are consistent in terms of naming?
5. It appears that pg_stat_io.reads seems to be not refreshed until they
query seems to be finished. While running a query for minutes with this
patchset, I've got:
now | reads | read_time
-------------------------------+----------+-----------
2024-11-15 12:09:09.151631+00 | 15004271 | 0
[..]
2024-11-15 12:10:25.241175+00 | 15004271 | 0
2024-11-15 12:10:26.241179+00 | 15004271 | 0
2024-11-15 12:10:27.241139+00 | 18250913 | 0
Or is that how it is supposed to work? Also pg_stat_io.read_time would be
something vague with io_uring/worker, so maybe zero is good here (?).
Otherwise we would have to measure time spent on waiting alone, but that
would force more instructions for calculating io times...
6. After playing with some basic measurements - which went fine, I wanted
to go test simple PostGIS even with sequential scans to see any
compatibility issues (AFAIR Thomas Munro on PGConfEU indicated as good
testing point), but before that I've tried to see what's the TOAST
performance alone with AIO+DIO (debug_io_direct=data). One issue I have
found is that DIO seems to be unusable until somebody will teach TOAST to
use readstreams, is that correct? Maybe I'm doing something wrong, but I
haven't seen any TOAST <-> readstreams topic:
-- 12MB table , 25GB toast
create table t (id bigint, t text storage external);
insert into t select i::bigint as id, repeat(md5(i::text),4000)::text as r
from generate_series(1,200000) s(i);
set max_parallel_workers_per_gather=0;
\timing
-- with cold caches: empty s_b, echo 3 > drop_caches
select sum(length(t)) from t;
master 101897.823 ms (01:41.898)
AIO 99758.399 ms (01:39.758)
AIO+DIO 191479.079 ms (03:11.479)
hotpath was detoast_attr() -> toast_fetch_datum() ->
heap_fetch_toast_slice() -> systable_getnext_ordered() ->
index_getnext_slot() -> index_fetch_heap() -> heapam_index_fetch_tuple() ->
ReadBufferExtended -> AIO code.
The difference is that on cold caches with DIO gets 2x slowdown; with clean
s_b and so on:
* getting normal heap data seqscan: up to 285MB/s
* but TOASTs maxes out at 132MB/s when using io_uring+DIO
Not about patch itself, but questions about related stack functionality:
----------------------------------------------------------------------------------------------------
7. Is pg_stat_aios still on the table or not ? (AIO 2021 had it). Any hints
on how to inspect real I/O calls requested to review if the code is issuing
sensible calls: there's no strace for uring, or do you stick to DEBUG3 or
perhaps using some bpftrace / xfsslower is the best way to go ?
8. Not sure if that helps, but I've managed the somehow to hit the
impossible situation You describe in pgaio_uring_submit() "(ret !=
num_staged_ios)", but I had to push urings really hard into using futexes
and probably I've could made some error in coding too for that too occur
[3]. As it stands in that patch from my thread, it was not covered: /*
FIXME: fix ret != submitted ?! seems like bug?! */ (but i had that hit that
code-path pretty often with 6.10.x kernel)
9. Please let me know, what's the current up to date line of thinking about
this patchset: is it intended to be committed as v18 ? As a debug feature
or as non-debug feature? (that is which of the IO methods should be
scrutinized the most as it is going to be the new default - sync or worker?)
10. At this point, does it even make sense to give a try experimenty try to
pwritev2() with RWF_ATOMIC? (that thing is already in the open, but XFS is
going to cover it with 6.12.x apparently, but I could try with some -rcX)
-J.
p.s. I hope I did not ask stupid questions nor missed anything.
[0] -
https://www.postgresql.org/message-id/237y5rabqim2c2v37js53li6i34v2525y2baf32isyexecn4ic%40bqmlx5mrn...
- "Right now the io_uring mode has each backend's io_uring instance visible
to
each other process.(..)"
[1] -
https://www.postgresql.org/message-id/flat/510b887e-c0ce-4a0c-a17a-2c6abb8d9a5c%40enterprisedb.com
- sentence after: "FWIW there's another bottleneck people may not realize
(..)"
[2] -
https://www.postgresql.org/message-id/flat/x3f32prdpgalmiieyialqtn53j5uvb2e4c47nvnjetkipq3zyk%40xk7j...
[3] -
https://www.postgresql.org/message-id/CAKZiRmwrBjCbCJ433wV5zjvwt_OuY7BsVX12MBKiBu%2BeNZDm6g%40mail.g...
^ permalink raw reply [nested|flat] 29+ messages in thread
* Re: AIO v2.0
@ 2024-12-19 22:29 Andres Freund <andres@anarazel.de>
parent: Jakub Wartak <jakub.wartak@enterprisedb.com>
0 siblings, 2 replies; 29+ messages in thread
From: Andres Freund @ 2024-12-19 22:29 UTC (permalink / raw)
To: Jakub Wartak <jakub.wartak@enterprisedb.com>; +Cc: pgsql-hackers@postgresql.org, Heikki Linnakangas <hlinnaka@iki.fi>; 陈宗志 <baotiao@gmail.com>
Hi,
Sorry for loosing track of your message for this long, I saw it just now
because I was working on posting a new version.
On 2024-11-18 13:19:58 +0100, Jakub Wartak wrote:
> On Fri, Sep 6, 2024 at 9:38 PM Andres Freund <andres@anarazel.de> wrote:
> Thank You for worth admiring persistence on this. Please do not take it as
> criticism, just more like set of questions regarding the patchset v2.1 that
> I finally got little time to play with:
>
> 0. Doesn't the v2.1-0011-aio-Add-io_uring-method.patch -> in
> pgaio_uring_submit() -> io_uring_get_sqe() need a return value check ?
Yea, it shouldn't ever happen, but it's worth adding a check.
> Otherwise we'll never know that SQ is full in theory, perhaps at least such
> a check should be made with Assert() ? (I understand right now that we
> allow just up to io_uring_queue_init(io_max_concurrency), but what happens
> if:
> a. previous io_uring_submit() failed for some reason and we do not have
> free space for SQ?
We'd have PANICed at that failure :)
> b. (hypothetical) someday someone will try to make PG multithreaded and the
> code starts using just one big queue, still without checking for
> io_uring_get_sqe()?
That'd not make sense - you'd still want to use separate rings, to avoid
contention.
> 1. In [0] you wrote that there's this high amount of FDs consumed for
> io_uring (dangerously close to RLIMIT_NOFILE). I can attest that there are
> many customers who are using extremely high max_connections (4k-5k, but
> there outliers with 10k in the wild too) - so they won't even start - and I
> have one doubt on the user-friendliness impact of this. I'm quite certain
> it's going to be the same as with pgbouncer where one is forced to tweak
> OS(systemd/pam/limits.conf/etc), but in PG we are better because PG tries
> to preallocate and then close() a lot of FDs, so that's safer in runtime.
> IMVHO even if we just consume e.g. say > 30% of FDs just for io_uring, the
> max_files_per_process looses it's spirit a little bit and PG is going to
> start loose efficiency too due to frequent open()/close() calls as fd cache
> is too small. Tomas also complained about it some time ago in [1])
My current thoughts around this are that we should generally, independent of
io_uring, increase the FD limit ourselves.
In most distros the soft ulimit is set to something like 1024, but the hard
limit is much higher. The reason for that is that some applications try to
close all fds between 0 and RLIMIT_NOFILE - which takes a long time if
RLIMIT_NOFILE is high. By setting only the soft limit to a low value any
application needing higher limits can just opt into using more FDs.
On several of my machines the hard limit is 1073741816.
> So maybe it would be good to introduce couple of sanity checks too (even
> after setting higher limit):
> - issue FATAL in case of using io_method = io_ring && max_connections would
> be close to getrusage(RLIMIT_NOFILE)
> - issue warning in case of using io_method = io_ring && we wouldnt have
> even real 1k FDs free for handling relation FDs (detect something bad like:
> getrusage(RLIMIT_NOFILE) <= max_connections + max_files_per_process)
Probably still worth adding something like this, even if we were to do what I
am suggesting above.
> 2. In pgaio_uring_postmaster_child_init_local() there
> "io_uring_queue_init(32,...)" - why 32? :) And also there's separate
> io_uring_queue_init(io_max_concurrency) which seems to be derived from
> AioChooseMaxConccurrency() which can go up to 64?
Yea, that's probably not right.
> 3. I find having two GUCs named literally the same
> (effective_io_concurrency, io_max_concurrency). It is clear from IO_URING
> perspective what is io_max_concurrency all about, but I bet having also
> effective_io_concurrency in the mix is going to be a little confusing for
> users (well, it is to me). Maybe that README.md could elaborate a little
> bit on the relation between those two? Or maybe do you plan to remove
> io_max_concurrency and bind it to effective_io_concurrency in future?
io_max_concurrency is a hard maximum that needs to be set at server start,
because it requires allocating shared memory. Whereas effective_io_concurrency
can be changed on a per-session and per-tablespace
basis. I.e. io_max_concurrency is a hard upper limit for an entire backend,
whereas effective_io_concurrency controls how much one scan (or whatever does
prefetching) can issue.
> To add more fun , there's MAX_IO_CONCURRENCY nearby in v2.1-0014 too while
> the earlier mentioned AioChooseMaxConccurrency() goes up to just 64
Yea, that should probably be disambiguated.
> 4. While we are at this, shouldn't the patch rename debug_io_direct to
> simply io_direct so that GUCs are consistent in terms of naming?
I used to have a patch like that in the series and it was a pain to
rebase...
I also suspect sure this is quite enough to make debug_io_direct quite
production ready, even if just considering io_direct=data. Without streaming
read use in heap + index VACUUM, RelationCopyStorage() and a few other places
the performance consequences of using direct IO can be, um, surprising.
> 5. It appears that pg_stat_io.reads seems to be not refreshed until they
> query seems to be finished. While running a query for minutes with this
> patchset, I've got:
> now | reads | read_time
> -------------------------------+----------+-----------
> 2024-11-15 12:09:09.151631+00 | 15004271 | 0
> [..]
> 2024-11-15 12:10:25.241175+00 | 15004271 | 0
> 2024-11-15 12:10:26.241179+00 | 15004271 | 0
> 2024-11-15 12:10:27.241139+00 | 18250913 | 0
>
> Or is that how it is supposed to work?
Currently the patch has a FIXME to add some IO statistics (I think I raised
that somewhere in this thread, too). It's not clear to me what IO time ought
to mean. I suspect the least bad answer is what you suggest:
> Also pg_stat_io.read_time would be something vague with io_uring/worker, so
> maybe zero is good here (?). Otherwise we would have to measure time spent
> on waiting alone, but that would force more instructions for calculating io
> times...
I.e. we should track the amount of time spent waiting for IOs.
I don't think tracking time in worker or such would make much sense, that'd
often end up with reporting more IO time than a query took.
> 6. After playing with some basic measurements - which went fine, I wanted
> to go test simple PostGIS even with sequential scans to see any
> compatibility issues (AFAIR Thomas Munro on PGConfEU indicated as good
> testing point), but before that I've tried to see what's the TOAST
> performance alone with AIO+DIO (debug_io_direct=data).
It's worth noting that with the last posted version you needed to increase
effective_io_concurrency to something very high to see sensible
performance.
That's due to the way read_stream_begin_impl() limited the number of buffers
pinned to effective_io_concurrency * 4 - which, due to io_combine_limit, ends
up allowing only a single IO in flight in case of sequential blocks until
effective_io_concurrency is set to 8 or such. I've adjusted that to some
degree now, but I think that might need a bit more sophistication.
> One issue I have found is that DIO seems to be unusable until somebody will
> teach TOAST to use readstreams, is that correct? Maybe I'm doing something
> wrong, but I haven't seen any TOAST <-> readstreams topic:
Hm, I suspect that aq read stream won't help a whole lot in manyq toast
cases. Unless you have particularly long toast datums, the time is going to be
dominated by the random accesses, as each toast datum is looked up in a
non-predictable way.
Generally, using DIO requires tuning shared buffers much more aggressively
than not using DIO, no amount of stream use will change that. Of course we
shoul try to reduce that "downside"...
I'm not sure if the best way to do prefetching toast chunks would be to rely
on more generalized index->table prefetching support, or to have dedicated code.
> -- 12MB table , 25GB toast
> create table t (id bigint, t text storage external);
> insert into t select i::bigint as id, repeat(md5(i::text),4000)::text as r
> from generate_series(1,200000) s(i);
> set max_parallel_workers_per_gather=0;
> \timing
> -- with cold caches: empty s_b, echo 3 > drop_caches
> select sum(length(t)) from t;
> master 101897.823 ms (01:41.898)
> AIO 99758.399 ms (01:39.758)
> AIO+DIO 191479.079 ms (03:11.479)
>
> hotpath was detoast_attr() -> toast_fetch_datum() ->
> heap_fetch_toast_slice() -> systable_getnext_ordered() ->
> index_getnext_slot() -> index_fetch_heap() -> heapam_index_fetch_tuple() ->
> ReadBufferExtended -> AIO code.
>
> The difference is that on cold caches with DIO gets 2x slowdown; with clean
> s_b and so on:
> * getting normal heap data seqscan: up to 285MB/s
> * but TOASTs maxes out at 132MB/s when using io_uring+DIO
I started loading the data to try this out myself :).
> Not about patch itself, but questions about related stack functionality:
> ----------------------------------------------------------------------------------------------------
>
>
> 7. Is pg_stat_aios still on the table or not ? (AIO 2021 had it). Any hints
> on how to inspect real I/O calls requested to review if the code is issuing
> sensible calls: there's no strace for uring, or do you stick to DEBUG3 or
> perhaps using some bpftrace / xfsslower is the best way to go ?
I think we still want something like it, but I don't think it needs to be in
the initial commits.
There are kernel events that you can track using e.g. perf. Particularly
useful are
io_uring:io_uring_submit_req
io_uring:io_uring_complete
> 8. Not sure if that helps, but I've managed the somehow to hit the
> impossible situation You describe in pgaio_uring_submit() "(ret !=
> num_staged_ios)", but I had to push urings really hard into using futexes
> and probably I've could made some error in coding too for that too occur
> [3]. As it stands in that patch from my thread, it was not covered: /*
> FIXME: fix ret != submitted ?! seems like bug?! */ (but i had that hit that
> code-path pretty often with 6.10.x kernel)
I think you can hit that if you don't take care to limit the number of IOs
being submitted at once or if you're not consuming completions. If the
completion queue is full enough the kernel at some point won't allow more IOs
to be submitted.
> 9. Please let me know, what's the current up to date line of thinking about
> this patchset: is it intended to be committed as v18 ?
I'd love to get some of it into 18. I don't quite know whether we can make it
happen and to what extent.
> As a debug feature or as non-debug feature? (that is which of the IO methods
> should be scrutinized the most as it is going to be the new default - sync
> or worker?)
I'd say initially worker, with a beta 1 or 2 checklist item to revise it.
> 10. At this point, does it even make sense to give a try experimenty try to
> pwritev2() with RWF_ATOMIC? (that thing is already in the open, but XFS is
> going to cover it with 6.12.x apparently, but I could try with some -rcX)
I don't think that's worth doing right now. There's too many dependencies and
it's going to be a while till the kernel support for that is widespread enough
to matter.
There's also the issue that, to my knowledge, outside of cloud environments
there's pretty much no hardware that actually reports power-fail atomicity
sizes bigger than a sector.
> p.s. I hope I did not ask stupid questions nor missed anything.
You did not!
Greetings,
Andres Freund
^ permalink raw reply [nested|flat] 29+ messages in thread
* Re: AIO v2.0
@ 2024-12-19 22:34 Tom Lane <tgl@sss.pgh.pa.us>
parent: Andres Freund <andres@anarazel.de>
1 sibling, 1 reply; 29+ messages in thread
From: Tom Lane @ 2024-12-19 22:34 UTC (permalink / raw)
To: Andres Freund <andres@anarazel.de>; +Cc: Jakub Wartak <jakub.wartak@enterprisedb.com>; pgsql-hackers@postgresql.org, Heikki Linnakangas <hlinnaka@iki.fi>; 陈宗志 <baotiao@gmail.com>
Andres Freund <andres@anarazel.de> writes:
> My current thoughts around this are that we should generally, independent of
> io_uring, increase the FD limit ourselves.
I'm seriously down on that, because it amounts to an assumption that
we own the machine and can appropriate all its resources. If ENFILE
weren't a thing, it'd be all right, but that is a thing. We have no
business trying to consume resources the DBA didn't tell us we could.
regards, tom lane
^ permalink raw reply [nested|flat] 29+ messages in thread
* Re: AIO v2.0
@ 2024-12-20 00:54 Andres Freund <andres@anarazel.de>
parent: Tom Lane <tgl@sss.pgh.pa.us>
0 siblings, 1 reply; 29+ messages in thread
From: Andres Freund @ 2024-12-20 00:54 UTC (permalink / raw)
To: Tom Lane <tgl@sss.pgh.pa.us>; +Cc: Jakub Wartak <jakub.wartak@enterprisedb.com>; pgsql-hackers@postgresql.org, Heikki Linnakangas <hlinnaka@iki.fi>; 陈宗志 <baotiao@gmail.com>
Hi,
On 2024-12-19 17:34:29 -0500, Tom Lane wrote:
> Andres Freund <andres@anarazel.de> writes:
> > My current thoughts around this are that we should generally, independent of
> > io_uring, increase the FD limit ourselves.
>
> I'm seriously down on that, because it amounts to an assumption that
> we own the machine and can appropriate all its resources. If ENFILE
> weren't a thing, it'd be all right, but that is a thing. We have no
> business trying to consume resources the DBA didn't tell us we could.
Arguably the configuration *did* tell us, by having a higher hard limit...
I'm not saying that we should increase the limit without a bound or without a
configuration option, btw.
As I had mentioned, the problem with relying on increasing the soft limit that
is that it's not generally sensible to do so, because it causes a bunch of
binaries to do be weirdly slow.
Another reason to not increase the soft rlimit is that doing so can break
programs relying on select().
But opting into a higher rlimit, while obviously adhering to the hard limit
and perhaps some other config knob, seems fine?
Greetings,
Andres Freund
^ permalink raw reply [nested|flat] 29+ messages in thread
* Re: AIO v2.0
@ 2024-12-20 17:27 Jelte Fennema-Nio <postgres@jeltef.nl>
parent: Andres Freund <andres@anarazel.de>
0 siblings, 1 reply; 29+ messages in thread
From: Jelte Fennema-Nio @ 2024-12-20 17:27 UTC (permalink / raw)
To: Andres Freund <andres@anarazel.de>; +Cc: Tom Lane <tgl@sss.pgh.pa.us>; Jakub Wartak <jakub.wartak@enterprisedb.com>; pgsql-hackers@postgresql.org, Heikki Linnakangas <hlinnaka@iki.fi>; 陈宗志 <baotiao@gmail.com>
On Fri, 20 Dec 2024 at 01:54, Andres Freund <andres@anarazel.de> wrote:
> Arguably the configuration *did* tell us, by having a higher hard limit...
> <snip>
> But opting into a higher rlimit, while obviously adhering to the hard limit
> and perhaps some other config knob, seems fine?
Yes, totally fine. That's exactly the reasoning why the hard limit is
so much larger than the soft limit by default on systems with systemd:
https://0pointer.net/blog/file-descriptor-limits.html
^ permalink raw reply [nested|flat] 29+ messages in thread
* Re: AIO v2.0
@ 2024-12-20 17:39 Andres Freund <andres@anarazel.de>
parent: Jelte Fennema-Nio <postgres@jeltef.nl>
0 siblings, 0 replies; 29+ messages in thread
From: Andres Freund @ 2024-12-20 17:39 UTC (permalink / raw)
To: Jelte Fennema-Nio <postgres@jeltef.nl>; +Cc: Tom Lane <tgl@sss.pgh.pa.us>; Jakub Wartak <jakub.wartak@enterprisedb.com>; pgsql-hackers@postgresql.org, Heikki Linnakangas <hlinnaka@iki.fi>; 陈宗志 <baotiao@gmail.com>
Hi,
On 2024-12-20 18:27:13 +0100, Jelte Fennema-Nio wrote:
> On Fri, 20 Dec 2024 at 01:54, Andres Freund <andres@anarazel.de> wrote:
> > Arguably the configuration *did* tell us, by having a higher hard limit...
> > <snip>
> > But opting into a higher rlimit, while obviously adhering to the hard limit
> > and perhaps some other config knob, seems fine?
>
> Yes, totally fine. That's exactly the reasoning why the hard limit is
> so much larger than the soft limit by default on systems with systemd:
>
> https://0pointer.net/blog/file-descriptor-limits.html
Good link.
This isn't just relevant for using io_uring:
There obviously are several people working on threaded postgres. Even if we
didn't duplicate fd.c file descriptors between threads (we probably will, at
least initially), the client connection FDs alone will mean that we have a lot
more FDs open. Due to the select() issue the soft limit won't be increased
beyond 1024, requiring everyone to add a 'ulimit -n $somehighnumber' before
starting postgres on linux doesn't help anyone.
Greetings,
Andres Freund
^ permalink raw reply [nested|flat] 29+ messages in thread
* Re: AIO v2.0
@ 2025-01-06 16:28 Andres Freund <andres@anarazel.de>
parent: Andres Freund <andres@anarazel.de>
1 sibling, 1 reply; 29+ messages in thread
From: Andres Freund @ 2025-01-06 16:28 UTC (permalink / raw)
To: Jakub Wartak <jakub.wartak@enterprisedb.com>; Thomas Munro <thomas.munro@gmail.com>; +Cc: pgsql-hackers@postgresql.org, Heikki Linnakangas <hlinnaka@iki.fi>; 陈宗志 <baotiao@gmail.com>
Hi,
On 2024-12-19 17:29:12 -0500, Andres Freund wrote:
> > Not about patch itself, but questions about related stack functionality:
> > ----------------------------------------------------------------------------------------------------
> >
> >
> > 7. Is pg_stat_aios still on the table or not ? (AIO 2021 had it). Any hints
> > on how to inspect real I/O calls requested to review if the code is issuing
> > sensible calls: there's no strace for uring, or do you stick to DEBUG3 or
> > perhaps using some bpftrace / xfsslower is the best way to go ?
>
> I think we still want something like it, but I don't think it needs to be in
> the initial commits.
After I got this question from Thomas as well, I started hacking one up.
What information would you like to see?
Here's what I currently have:
┌─[ RECORD 1 ]───┬────────────────────────────────────────────────┐
│ pid │ 358212 │
│ io_id │ 2050 │
│ io_generation │ 4209 │
│ state │ COMPLETED_SHARED │
│ operation │ read │
│ offset │ 509083648 │
│ length │ 262144 │
│ subject │ smgr │
│ iovec_data_len │ 32 │
│ raw_result │ 262144 │
│ result │ OK │
│ error_desc │ (null) │
│ subject_desc │ blocks 1372864..1372895 in file "base/5/16388" │
│ flag_sync │ f │
│ flag_localmem │ f │
│ flag_buffered │ t │
├─[ RECORD 2 ]───┼────────────────────────────────────────────────┤
│ pid │ 358212 │
│ io_id │ 2051 │
│ io_generation │ 4199 │
│ state │ IN_FLIGHT │
│ operation │ read │
│ offset │ 511967232 │
│ length │ 262144 │
│ subject │ smgr │
│ iovec_data_len │ 32 │
│ raw_result │ (null) │
│ result │ UNKNOWN │
│ error_desc │ (null) │
│ subject_desc │ blocks 1373216..1373247 in file "base/5/16388" │
│ flag_sync │ f │
│ flag_localmem │ f │
│ flag_buffered │ t │
I didn't think that pg_stat_* was quite the right namespace, given that it
shows not stats, but the currently ongoing IOs. I am going with pg_aios for
now, but I don't particularly like that.
I think we'll want a pg_stat_aio as well, tracking things like:
- how often the queue to IO workes was full
- how many times we submitted IO to the kernel (<= #ios with io_uring)
- how many times we asked the kernel for events (<= #ios with io_uring)
- how many times we had to wait for in-flight IOs before issuing more IOs
Greetings,
Andres Freund
^ permalink raw reply [nested|flat] 29+ messages in thread
* Re: AIO v2.0
@ 2025-01-08 14:04 Jakub Wartak <jakub.wartak@enterprisedb.com>
parent: Andres Freund <andres@anarazel.de>
0 siblings, 1 reply; 29+ messages in thread
From: Jakub Wartak @ 2025-01-08 14:04 UTC (permalink / raw)
To: Andres Freund <andres@anarazel.de>; +Cc: Thomas Munro <thomas.munro@gmail.com>; pgsql-hackers@postgresql.org, Heikki Linnakangas <hlinnaka@iki.fi>; 陈宗志 <baotiao@gmail.com>
On Mon, Jan 6, 2025 at 5:28 PM Andres Freund <andres@anarazel.de> wrote:
>
> Hi,
>
> On 2024-12-19 17:29:12 -0500, Andres Freund wrote:
> > > Not about patch itself, but questions about related stack functionality:
> > > ----------------------------------------------------------------------------------------------------
> > >
> > >
> > > 7. Is pg_stat_aios still on the table or not ? (AIO 2021 had it). Any hints
> > > on how to inspect real I/O calls requested to review if the code is issuing
> > > sensible calls: there's no strace for uring, or do you stick to DEBUG3 or
> > > perhaps using some bpftrace / xfsslower is the best way to go ?
> >
> > I think we still want something like it, but I don't think it needs to be in
> > the initial commits.
>
> After I got this question from Thomas as well, I started hacking one up.
>
> What information would you like to see?
>
> Here's what I currently have:
..
> ├─[ RECORD 2 ]───┼────────────────────────────────────────────────┤
> │ pid │ 358212 │
> │ io_id │ 2051 │
> │ io_generation │ 4199 │
> │ state │ IN_FLIGHT │
> │ operation │ read │
> │ offset │ 511967232 │
> │ length │ 262144 │
> │ subject │ smgr │
> │ iovec_data_len │ 32 │
> │ raw_result │ (null) │
> │ result │ UNKNOWN │
> │ error_desc │ (null) │
> │ subject_desc │ blocks 1373216..1373247 in file "base/5/16388" │
> │ flag_sync │ f │
> │ flag_localmem │ f │
> │ flag_buffered │ t │
Cool! It's more than enough for me in future, thanks!
> I didn't think that pg_stat_* was quite the right namespace, given that it
> shows not stats, but the currently ongoing IOs. I am going with pg_aios for
> now, but I don't particularly like that.
If you are looking for other proposals:
* pg_aios_progress ? (to follow pattern of pg_stat_copy|vaccuum_progress?)
* pg_debug_aios ?
* pg_debug_io ?
> I think we'll want a pg_stat_aio as well, tracking things like:
>
> - how often the queue to IO workes was full
> - how many times we submitted IO to the kernel (<= #ios with io_uring)
> - how many times we asked the kernel for events (<= #ios with io_uring)
> - how many times we had to wait for in-flight IOs before issuing more IOs
If I could dream of one thing that would be 99.9% percentile of IO
response times in milliseconds for different classes of I/O traffic
(read/write/flush). But it sounds like it would be very similiar to
pg_stat_io and potentially would have to be
per-tablespace/IO-traffic(subject)-type too. AFAIU pg_stat_io has
improper structure to have that there.
BTW: before trying to even start to compile that AIO v2.2* and
responding to the previous review, what are You looking interested to
hear the most about it so that it adds some value ? Any workload
specific measurements? just general feedback, functionality gaps?
Integrity/data testing with stuff like dm-dust, dm-flakey, dm-delay to
try the error handling routines? Some kind of AIO <-> standby/recovery
interactions?
* - btw, Date: 2025-01-01 04:03:33 - I saw what you did there! so
let's officially recognize the 2025 as the year of AIO in PG, as it
was 1st message :D
-J.
^ permalink raw reply [nested|flat] 29+ messages in thread
* Re: AIO v2.0
@ 2025-01-08 20:58 Andres Freund <andres@anarazel.de>
parent: Jakub Wartak <jakub.wartak@enterprisedb.com>
0 siblings, 1 reply; 29+ messages in thread
From: Andres Freund @ 2025-01-08 20:58 UTC (permalink / raw)
To: Jakub Wartak <jakub.wartak@enterprisedb.com>; +Cc: Thomas Munro <thomas.munro@gmail.com>; pgsql-hackers@postgresql.org, Heikki Linnakangas <hlinnaka@iki.fi>; 陈宗志 <baotiao@gmail.com>
Hi,
On 2025-01-08 15:04:39 +0100, Jakub Wartak wrote:
> On Mon, Jan 6, 2025 at 5:28 PM Andres Freund <andres@anarazel.de> wrote:
> > I didn't think that pg_stat_* was quite the right namespace, given that it
> > shows not stats, but the currently ongoing IOs. I am going with pg_aios for
> > now, but I don't particularly like that.
>
> If you are looking for other proposals:
> * pg_aios_progress ? (to follow pattern of pg_stat_copy|vaccuum_progress?)
> * pg_debug_aios ?
> * pg_debug_io ?
I think pg_aios is better than those, if not by much. Seems others are ok
with that name too. And we easily can evolve it later.
> > I think we'll want a pg_stat_aio as well, tracking things like:
> >
> > - how often the queue to IO workes was full
> > - how many times we submitted IO to the kernel (<= #ios with io_uring)
> > - how many times we asked the kernel for events (<= #ios with io_uring)
> > - how many times we had to wait for in-flight IOs before issuing more IOs
>
> If I could dream of one thing that would be 99.9% percentile of IO
> response times in milliseconds for different classes of I/O traffic
> (read/write/flush). But it sounds like it would be very similiar to
> pg_stat_io and potentially would have to be
> per-tablespace/IO-traffic(subject)-type too.
Yea, that's a significant project on its own. It's not that cheap to compute
reasonably accurate percentiles and we have no infrastructure for doing so
right now.
> AFAIU pg_stat_io has improper structure to have that there.
Hm, not obvious to me why? It might make the view a bit wide to add it as an
additional column, but otherwise I don't see a problem?
> BTW: before trying to even start to compile that AIO v2.2* and
> responding to the previous review, what are You looking interested to
> hear the most about it so that it adds some value?
Due to the rather limited "users" of AIO in the patchset, I think most
benchmarks aren't expected to show any meaningful gains. However, they
shouldn't show any significant regressions either (when not using direct
IO). I think trying to find regressions would be a rather valuable thing.
I'm tempted to collect a few of the reasonbly-ready read stream conversions
into the patchset, to make the potential gains more visible. But I am not sure
it's a good investment of time right now.
One small regression I do know about, namely scans of large relations that are
bigger than shared buffers but do fit in the kernel page cache. The increase
of BAS_BULKREAD does cause a small slowdown - but without it we never can do
sufficient asynchronous IO. I think the slowdown is small enough to just
accept that, but it's worth qualifying that on a few machines.
> Any workload specific measurements? just general feedback, functionality
> gaps?
To see the benefits it'd be interesting to compare:
1) sequential scan performance with data not in shared buffers, using buffered IO
2) same, but using direct IO when testing the patch
3) checkpoint performance
In my experiments 1) gains a decent amount of performance in many cases, but
nothing overwhelming - sequential scans are easy for the kernel to read ahead.
I do see very significant gains for 2) - On a system with 10 striped NVMe SSDs
that each can do ~3.5 GB/s I measured very parallel sequential scans (I had
to use ALTER TABLE to get sufficient numbers of workers):
master: ~18 GB/s
patch, buffered: ~20 GB/s
patch, direct, worker: ~28 GB/s
patch, direct, uring: ~35 GB/s
This was with io_workers=32, io_max_concurrency=128,
effective_io_concurrency=1000 (doesn't need to be that high, but it's what I
still have the numbers for).
This was without data checksums enabled as otherwise the checksum code becomes
a *huge* bottleneck.
I also see significant gains with 3). Bigger when using direct IO. One
complicating factor measuring 3) is that the first write to a block will often
be slower than subsequent writes because the filesystem will need to update
some journaled metadata, presenting a bottleneck.
Checkpoint performance is also severely limited by data checksum computation
if enabled - independent of this patchset.
One annoying thing when testing DIO is that right now VACUUM will be rather
slow if the data isn't already in s_b, as it isn't yet read-stream-ified.
> Integrity/data testing with stuff like dm-dust, dm-flakey, dm-delay
> to try the error handling routines?
Hm. I don't think that's going to work very well even on master. If the
filesystem fails there's not much that PG can do...
> Some kind of AIO <-> standby/recovery interactions?
I wouldn't expect anything there. I think Thomas somewhere has a patch that
read-stream-ifies recovery prefetching, once that's done it would be more
interesting.
> * - btw, Date: 2025-01-01 04:03:33 - I saw what you did there! so
> let's officially recognize the 2025 as the year of AIO in PG, as it
> was 1st message :D
Hah, that was actually the opposite of what I intended :). I'd hoped to post
earlier, but jetlag had caught up with me...
Greetings,
Andres Freund
^ permalink raw reply [nested|flat] 29+ messages in thread
* Re: AIO v2.0
@ 2025-01-09 08:59 Ants Aasma <ants.aasma@cybertec.at>
parent: Andres Freund <andres@anarazel.de>
0 siblings, 1 reply; 29+ messages in thread
From: Ants Aasma @ 2025-01-09 08:59 UTC (permalink / raw)
To: Andres Freund <andres@anarazel.de>; +Cc: Jakub Wartak <jakub.wartak@enterprisedb.com>; Thomas Munro <thomas.munro@gmail.com>; pgsql-hackers@postgresql.org, Heikki Linnakangas <hlinnaka@iki.fi>; 陈宗志 <baotiao@gmail.com>
On Wed, 8 Jan 2025 at 22:58, Andres Freund <andres@anarazel.de> wrote:
> master: ~18 GB/s
> patch, buffered: ~20 GB/s
> patch, direct, worker: ~28 GB/s
> patch, direct, uring: ~35 GB/s
>
>
> This was with io_workers=32, io_max_concurrency=128,
> effective_io_concurrency=1000 (doesn't need to be that high, but it's what I
> still have the numbers for).
>
>
> This was without data checksums enabled as otherwise the checksum code becomes
> a *huge* bottleneck.
I'm curious about this because the checksum code should be fast enough
to easily handle that throughput. I remember checksum overhead being
negligible even when pulling in pages from page cache. Is it just that
the calculation is slow, or is it the fact that checksumming needs to
bring the page into the CPU cache. Did you notice any hints which
might be the case? I don't really have a machine at hand that can do
anywhere close to this amount of I/O.
I'm asking because if it's the calculation that is slow then it seems
like it's time to compile different ISA extension variants of the
checksum code and select the best one at runtime.
--
Ants Aasma
^ permalink raw reply [nested|flat] 29+ messages in thread
* Re: AIO v2.0
@ 2025-01-09 16:25 Andres Freund <andres@anarazel.de>
parent: Ants Aasma <ants.aasma@cybertec.at>
0 siblings, 1 reply; 29+ messages in thread
From: Andres Freund @ 2025-01-09 16:25 UTC (permalink / raw)
To: Ants Aasma <ants.aasma@cybertec.at>; +Cc: Jakub Wartak <jakub.wartak@enterprisedb.com>; Thomas Munro <thomas.munro@gmail.com>; pgsql-hackers@postgresql.org, Heikki Linnakangas <hlinnaka@iki.fi>; 陈宗志 <baotiao@gmail.com>
Hi,
On 2025-01-09 10:59:22 +0200, Ants Aasma wrote:
> On Wed, 8 Jan 2025 at 22:58, Andres Freund <andres@anarazel.de> wrote:
> > master: ~18 GB/s
> > patch, buffered: ~20 GB/s
> > patch, direct, worker: ~28 GB/s
> > patch, direct, uring: ~35 GB/s
> >
> >
> > This was with io_workers=32, io_max_concurrency=128,
> > effective_io_concurrency=1000 (doesn't need to be that high, but it's what I
> > still have the numbers for).
> >
> >
> > This was without data checksums enabled as otherwise the checksum code becomes
> > a *huge* bottleneck.
>
> I'm curious about this because the checksum code should be fast enough
> to easily handle that throughput.
It seems to top out at about ~5-6 GB/s on my 2x Xeon Gold 6442Y
workstation. But we don't have a good ready-made way of testing that without
also doing IO, so it's kinda hard to say.
> I remember checksum overhead being negligible even when pulling in pages
> from page cache.
It's indeed much less of an issue when pulling pages from the page cache, as
the copy from the page cache is fairly slow. With direct-IO, where the copy
from the page cache isn't the main driver of CPU use anymore, it becomes much
clearer.
Even with buffered IO it became a bigger issue with 17, due to
io_combine_limit. It turns out that lots of tiny syscalls are slow, so the
peak throughput that could reach the checksumming code was lower.
I created a 21554MB relation and measured the time to do a pg_prewarm() of
that relation after evicting all of shared buffers (the first time buffers are
touched has a bit different perf characteristics). I am using direct IO and
io_uring here, as buffered IO would include the page cache copy cost and
worker mode could parallelize the checksum computation on reads. The checksum
cost is a bigger issue for writes than reads, but it's harder to quickly
generate enough dirty data for a repeatable benchmark.
This system can do about 12.5GB/s of read IO.
Just to show the effect of the read size on page cache copy performance:
config checksums time in ms
buffered io_engine=sync io_combine_limit=1 0 6712.153
buffered io_engine=sync io_combine_limit=2 0 5919.215
buffered io_engine=sync io_combine_limit=4 0 5738.496
buffered io_engine=sync io_combine_limit=8 0 5396.415
buffered io_engine=sync io_combine_limit=16 0 5312.803
buffered io_engine=sync io_combine_limit=32 0 5275.389
To see the effect of page cache copy overhead:
config checksums time in ms
buffered io_engine=io_uring 0 3901.625
direct io_engine=io_uring 0 2075.330
Now to show the effect of checksums (enabled/disabled with pg_checksums):
config checksums time in ms
buffered io_engine=io_uring 0 3883.127
buffered io_engine=io_uring 1 5880.892
direct io_engine=io_uring 0 2067.142
direct io_engine=io_uring 1 3835.968
So with direct + uring w/o checksums, we can reach 10427 MB/s (close-ish to
disk speed), but with checksums we only reach 5620 MB/s.
> Is it just that the calculation is slow, or is it the fact that checksumming
> needs to bring the page into the CPU cache. Did you notice any hints which
> might be the case?
I don't think the issue is that checksumming pulls the data into CPU caches
1) This is visible with SELECT that actually uses the data
2) I added prefetching to avoid any meaningful amount of cache misses and it
doesn't change the overall timing much
3) It's visible with buffered IO, which has pulled the data into CPU caches
already
> I don't really have a machine at hand that can do anywhere close to this
> amount of I/O.
It's visible even when pulling from the page cache, if to a somewhat lesser
degree.
I wonder if it's worth adding a test function that computes checksums of all
shared buffers in memory already. That'd allow exercising the checksum code in
a realistic context (i.e. buffer locking etc preventing some out-of-order
effects, using 8kB chunks etc) without also needing to involve the IO path.
> I'm asking because if it's the calculation that is slow then it seems
> like it's time to compile different ISA extension variants of the
> checksum code and select the best one at runtime.
You think it's ISA specific? I don't see a significant effect of compiling
with -march=native or not - and that should suffice to make the checksum code
built with sufficiently high ISA support, right?
FWIW CPU profiles show all the time being spent in the "main checksum
calculation" loop:
Percent | Source code & Disassembly of postgres for cycles:P (5866 samples, percent: local period)
--------------------------------------------------------------------------------------------------------
:
:
:
: 3 Disassembly of section .text:
:
: 5 00000000009e3670 <pg_checksum_page>:
: 6 * calculation isn't affected by the old checksum stored on the page.
: 7 * Restore it after, because actually updating the checksum is NOT part of
: 8 * the API of this function.
: 9 */
: 10 save_checksum = cpage->phdr.pd_checksum;
: 11 cpage->phdr.pd_checksum = 0;
0.00 : 9e3670: xor %eax,%eax
: 13 CHECKSUM_COMP(sums[j], page->data[i][j]);
0.00 : 9e3672: mov $0x1000193,%r8d
: 15 cpage->phdr.pd_checksum = 0;
0.00 : 9e3678: vmovdqa -0x693fa0(%rip),%ymm3 # 34f6e0 <.LC0>
0.05 : 9e3680: vmovdqa -0x6935c8(%rip),%ymm4 # 3500c0 <.LC1>
0.00 : 9e3688: vmovdqa -0x693c10(%rip),%ymm0 # 34fa80 <.LC2>
0.00 : 9e3690: vmovdqa -0x693598(%rip),%ymm1 # 350100 <.LC3>
: 20 {
0.00 : 9e3698: mov %esi,%ecx
0.02 : 9e369a: lea 0x2000(%rdi),%rdx
: 23 save_checksum = cpage->phdr.pd_checksum;
0.00 : 9e36a1: movzwl 0x8(%rdi),%esi
: 25 CHECKSUM_COMP(sums[j], page->data[i][j]);
0.00 : 9e36a5: vpbroadcastd %r8d,%ymm5
: 27 cpage->phdr.pd_checksum = 0;
0.00 : 9e36ab: mov %ax,0x8(%rdi)
: 29 for (i = 0; i < (uint32) (BLCKSZ / (sizeof(uint32) * N_SUMS)); i++)
0.14 : 9e36af: mov %rdi,%rax
0.00 : 9e36b2: nopw 0x0(%rax,%rax,1)
: 32 CHECKSUM_COMP(sums[j], page->data[i][j]);
15.36 : 9e36b8: vpxord (%rax),%ymm1,%ymm1
4.79 : 9e36be: vmovdqu 0x80(%rax),%ymm2
: 35 for (i = 0; i < (uint32) (BLCKSZ / (sizeof(uint32) * N_SUMS)); i++)
0.07 : 9e36c6: add $0x100,%rax
: 37 CHECKSUM_COMP(sums[j], page->data[i][j]);
2.45 : 9e36cc: vpxord -0xe0(%rax),%ymm0,%ymm0
2.85 : 9e36d3: vpmulld %ymm5,%ymm1,%ymm6
0.02 : 9e36d8: vpsrld $0x11,%ymm1,%ymm1
3.17 : 9e36dd: vpternlogd $0x96,%ymm6,%ymm1,%ymm2
2.01 : 9e36e4: vpmulld %ymm5,%ymm0,%ymm6
13.16 : 9e36e9: vpmulld %ymm5,%ymm2,%ymm1
0.03 : 9e36ee: vpsrld $0x11,%ymm2,%ymm2
0.02 : 9e36f3: vpsrld $0x11,%ymm0,%ymm0
2.57 : 9e36f8: vpxord %ymm2,%ymm1,%ymm1
0.89 : 9e36fe: vmovdqu -0x60(%rax),%ymm2
0.12 : 9e3703: vpternlogd $0x96,%ymm6,%ymm0,%ymm2
4.48 : 9e370a: vpmulld %ymm5,%ymm2,%ymm0
0.49 : 9e370f: vpsrld $0x11,%ymm2,%ymm2
0.99 : 9e3714: vpxord %ymm2,%ymm0,%ymm0
11.88 : 9e371a: vpxord -0xc0(%rax),%ymm4,%ymm2
2.80 : 9e3721: vpmulld %ymm5,%ymm2,%ymm6
0.68 : 9e3726: vpsrld $0x11,%ymm2,%ymm4
4.94 : 9e372b: vmovdqu -0x40(%rax),%ymm2
1.45 : 9e3730: vpternlogd $0x96,%ymm6,%ymm4,%ymm2
8.63 : 9e3737: vpmulld %ymm5,%ymm2,%ymm4
0.17 : 9e373c: vpsrld $0x11,%ymm2,%ymm2
1.81 : 9e3741: vpxord %ymm2,%ymm4,%ymm4
0.10 : 9e3747: vpxord -0xa0(%rax),%ymm3,%ymm2
0.70 : 9e374e: vpmulld %ymm5,%ymm2,%ymm6
1.65 : 9e3753: vpsrld $0x11,%ymm2,%ymm3
0.03 : 9e3758: vmovdqu -0x20(%rax),%ymm2
0.85 : 9e375d: vpternlogd $0x96,%ymm6,%ymm3,%ymm2
3.73 : 9e3764: vpmulld %ymm5,%ymm2,%ymm3
0.07 : 9e3769: vpsrld $0x11,%ymm2,%ymm2
1.48 : 9e376e: vpxord %ymm2,%ymm3,%ymm3
: 68 for (i = 0; i < (uint32) (BLCKSZ / (sizeof(uint32) * N_SUMS)); i++)
0.02 : 9e3774: cmp %rax,%rdx
2.32 : 9e3777: jne 9e36b8 <pg_checksum_page+0x48>
: 71 CHECKSUM_COMP(sums[j], 0);
0.17 : 9e377d: vpmulld %ymm5,%ymm0,%ymm7
0.07 : 9e3782: vpmulld %ymm5,%ymm1,%ymm6
: 74 checksum = pg_checksum_block(cpage);
: 75 cpage->phdr.pd_checksum = save_checksum;
0.00 : 9e3787: mov %si,0x8(%rdi)
: 77 CHECKSUM_COMP(sums[j], 0);
0.02 : 9e378b: vpsrld $0x11,%ymm0,%ymm0
0.02 : 9e3790: vpsrld $0x11,%ymm1,%ymm1
0.02 : 9e3795: vpsrld $0x11,%ymm4,%ymm2
0.00 : 9e379a: vpxord %ymm0,%ymm7,%ymm7
0.10 : 9e37a0: vpmulld %ymm5,%ymm4,%ymm0
0.00 : 9e37a5: vpxord %ymm1,%ymm6,%ymm6
0.17 : 9e37ab: vpmulld %ymm5,%ymm3,%ymm1
0.19 : 9e37b0: vpmulld %ymm5,%ymm6,%ymm4
0.00 : 9e37b5: vpsrld $0x11,%ymm6,%ymm6
0.02 : 9e37ba: vpxord %ymm2,%ymm0,%ymm0
0.00 : 9e37c0: vpsrld $0x11,%ymm3,%ymm2
0.22 : 9e37c5: vpmulld %ymm5,%ymm7,%ymm3
0.02 : 9e37ca: vpsrld $0x11,%ymm7,%ymm7
0.00 : 9e37cf: vpxord %ymm2,%ymm1,%ymm1
0.03 : 9e37d5: vpsrld $0x11,%ymm0,%ymm2
0.15 : 9e37da: vpmulld %ymm5,%ymm0,%ymm0
: 94 result ^= sums[i];
0.00 : 9e37df: vpternlogd $0x96,%ymm3,%ymm7,%ymm2
: 96 CHECKSUM_COMP(sums[j], 0);
0.05 : 9e37e6: vpsrld $0x11,%ymm1,%ymm3
0.19 : 9e37eb: vpmulld %ymm5,%ymm1,%ymm1
: 99 result ^= sums[i];
0.02 : 9e37f0: vpternlogd $0x96,%ymm4,%ymm6,%ymm0
0.10 : 9e37f7: vpxord %ymm1,%ymm0,%ymm0
0.07 : 9e37fd: vpternlogd $0x96,%ymm2,%ymm3,%ymm0
0.15 : 9e3804: vextracti32x4 $0x1,%ymm0,%xmm1
0.03 : 9e380b: vpxord %xmm0,%xmm1,%xmm0
0.14 : 9e3811: vpsrldq $0x8,%xmm0,%xmm1
0.12 : 9e3816: vpxord %xmm1,%xmm0,%xmm0
0.09 : 9e381c: vpsrldq $0x4,%xmm0,%xmm1
0.12 : 9e3821: vpxord %xmm1,%xmm0,%xmm0
0.05 : 9e3827: vmovd %xmm0,%eax
:
: 111 /* Mix in the block number to detect transposed pages */
: 112 checksum ^= blkno;
0.07 : 9e382b: xor %ecx,%eax
:
: 115 /*
: 116 * Reduce to a uint16 (to fit in the pd_checksum field) with an offset of
: 117 * one. That avoids checksums of zero, which seems like a good idea.
: 118 */
: 119 return (uint16) ((checksum % 65535) + 1);
0.00 : 9e382d: mov $0x80008001,%ecx
0.03 : 9e3832: mov %eax,%edx
0.27 : 9e3834: imul %rcx,%rdx
0.09 : 9e3838: shr $0x2f,%rdx
0.07 : 9e383c: lea 0x1(%rax,%rdx,1),%eax
0.00 : 9e3840: vzeroupper
: 126 }
0.15 : 9e3843: ret
I did briefly experiment with changing N_SUMS. 16 is substantially worse, 64
seems to be about the same as 32.
Greetings,
Andres Freund
^ permalink raw reply [nested|flat] 29+ messages in thread
* Re: AIO v2.0
@ 2025-01-09 18:10 Ants Aasma <ants.aasma@cybertec.at>
parent: Andres Freund <andres@anarazel.de>
0 siblings, 1 reply; 29+ messages in thread
From: Ants Aasma @ 2025-01-09 18:10 UTC (permalink / raw)
To: Andres Freund <andres@anarazel.de>; +Cc: Jakub Wartak <jakub.wartak@enterprisedb.com>; Thomas Munro <thomas.munro@gmail.com>; pgsql-hackers@postgresql.org, Heikki Linnakangas <hlinnaka@iki.fi>; 陈宗志 <baotiao@gmail.com>
On Thu, 9 Jan 2025 at 18:25, Andres Freund <andres@anarazel.de> wrote:
> > I'm curious about this because the checksum code should be fast enough
> > to easily handle that throughput.
>
> It seems to top out at about ~5-6 GB/s on my 2x Xeon Gold 6442Y
> workstation. But we don't have a good ready-made way of testing that without
> also doing IO, so it's kinda hard to say.
Interesting, I wonder if it's related to Intel increasing vpmulld
latency to 10 already back in Haswell. The Zen 3 I'm testing on has
latency 3 and has twice the throughput.
Attached is a naive and crude benchmark that I used for testing here.
Compiled with:
gcc -O2 -funroll-loops -ftree-vectorize -march=native \
-I$(pg_config --includedir-server) \
bench-checksums.c -o bench-checksums-native
Just fills up an array of pages and checksums them, first argument is
number of checksums, second is array size. I used 1M checksums and 100
pages for in cache behavior and 100000 pages for in memory
performance.
869.85927ms @ 9.418 GB/s - generic from memory
772.12252ms @ 10.610 GB/s - generic in cache
442.61869ms @ 18.508 GB/s - native from memory
137.07573ms @ 59.763 GB/s - native in cache
> > Is it just that the calculation is slow, or is it the fact that checksumming
> > needs to bring the page into the CPU cache. Did you notice any hints which
> > might be the case?
>
> I don't think the issue is that checksumming pulls the data into CPU caches
>
> 1) This is visible with SELECT that actually uses the data
>
> 2) I added prefetching to avoid any meaningful amount of cache misses and it
> doesn't change the overall timing much
>
> 3) It's visible with buffered IO, which has pulled the data into CPU caches
> already
I didn't yet check the code, when doing aio completions checksumming
be running on the same core as is going to be using the page?
It could also be that for some reason the checksumming is creating
extra bandwidth on memory bus or CPU internal rings, which due to the
already high amount of data already flying around causes contention.
> > I don't really have a machine at hand that can do anywhere close to this
> > amount of I/O.
>
> It's visible even when pulling from the page cache, if to a somewhat lesser
> degree.
Good point, I'll see if I can reproduce.
> I wonder if it's worth adding a test function that computes checksums of all
> shared buffers in memory already. That'd allow exercising the checksum code in
> a realistic context (i.e. buffer locking etc preventing some out-of-order
> effects, using 8kB chunks etc) without also needing to involve the IO path.
OoO shouldn't matter that much, over here even in the best case it's
still taking 500+ cycles per iteration.
> > I'm asking because if it's the calculation that is slow then it seems
> > like it's time to compile different ISA extension variants of the
> > checksum code and select the best one at runtime.
>
> You think it's ISA specific? I don't see a significant effect of compiling
> with -march=native or not - and that should suffice to make the checksum code
> built with sufficiently high ISA support, right?
Right, the disassembly below looked very good.
> FWIW CPU profiles show all the time being spent in the "main checksum
> calculation" loop:
.. disassembly omitted for brevity
Not sure if it's applicable here or not due to microarch differences.
But in my case when bounded by memory bandwidth the main loop events
were clustered around a few instructions like it was in here, whereas
when running from cache all instructions were about equally
represented.
> I did briefly experiment with changing N_SUMS. 16 is substantially worse, 64
> seems to be about the same as 32.
This suggests that mulld latency is not the culprit.
Regards,
Ants
Attachments:
bench-checksums.c (text/x-csrc, 954B) download
#include "postgres.h"
#include "storage/checksum_impl.h"
#include <time.h>
#undef printf
int __attribute__ ((noinline)) checksum_block(char *page, uint32 blockno)
{
return pg_checksum_page(page, blockno);
}
int main(int argc, char *argv[]) {
char *page;
uint64 i;
uint64 sum = 0;
struct timespec start;
struct timespec end;
double delta;
if (argc<3) {
printf("Usage: %s niterations nblocks\n", argv[0]);
return 1;
}
uint64 n = strtoull(argv[1], 0, 10);
uint64 b = strtoull(argv[2], 0, 10);
page = malloc(BLCKSZ*b);
for (i = 0; i < BLCKSZ*b; i++)
page[i] = (i*997) & 0xFF;
clock_gettime(CLOCK_MONOTONIC_RAW, &start);
for (i = 0; i < n; i++)
sum += checksum_block(page + BLCKSZ*(i % b), (uint32) i);
clock_gettime(CLOCK_MONOTONIC_RAW, &end);
delta = (double)(end.tv_sec - start.tv_sec) + (1e-9*(double) (end.tv_nsec - start.tv_nsec));
printf("%0.5fms @ %0.3f GB/s\n", delta*1000, (8192.0 * n)/delta/1e9);
return 0;
}
* Re: AIO v2.0
@ 2025-01-09 20:53 Andres Freund <andres@anarazel.de>
parent: Ants Aasma <ants.aasma@cybertec.at>
0 siblings, 1 reply; 29+ messages in thread
From: Andres Freund @ 2025-01-09 20:53 UTC (permalink / raw)
To: Ants Aasma <ants.aasma@cybertec.at>; +Cc: Jakub Wartak <jakub.wartak@enterprisedb.com>; Thomas Munro <thomas.munro@gmail.com>; pgsql-hackers@postgresql.org, Heikki Linnakangas <hlinnaka@iki.fi>; 陈宗志 <baotiao@gmail.com>
Hi,
On 2025-01-09 20:10:24 +0200, Ants Aasma wrote:
> On Thu, 9 Jan 2025 at 18:25, Andres Freund <andres@anarazel.de> wrote:
> > > I'm curious about this because the checksum code should be fast enough
> > > to easily handle that throughput.
> >
> > It seems to top out at about ~5-6 GB/s on my 2x Xeon Gold 6442Y
> > workstation. But we don't have a good ready-made way of testing that without
> > also doing IO, so it's kinda hard to say.
>
> Interesting, I wonder if it's related to Intel increasing vpmulld
> latency to 10 already back in Haswell. The Zen 3 I'm testing on has
> latency 3 and has twice the throughput.
> Attached is a naive and crude benchmark that I used for testing here.
> Compiled with:
>
> gcc -O2 -funroll-loops -ftree-vectorize -march=native \
> -I$(pg_config --includedir-server) \
> bench-checksums.c -o bench-checksums-native
>
> Just fills up an array of pages and checksums them, first argument is
> number of checksums, second is array size. I used 1M checksums and 100
> pages for in cache behavior and 100000 pages for in memory
> performance.
>
> 869.85927ms @ 9.418 GB/s - generic from memory
> 772.12252ms @ 10.610 GB/s - generic in cache
> 442.61869ms @ 18.508 GB/s - native from memory
> 137.07573ms @ 59.763 GB/s - native in cache
printf '%16s\t%16s\t%s\n' march mem result; for mem in 100 100000 1000000; do for march in x86-64 x86-64-v2 x86-64-v3 x86-64-v4 native; do printf "%16s\t%16s\t" $march $mem; gcc -g -g3 -O2 -funroll-loops -ftree-vectorize -march=$march -I ~/src/postgresql/src/include/ -I src/include/ /tmp/bench-checksums.c -o bench-checksums-native && numactl --physcpubind 1 --membind 0 ./bench-checksums-native 1000000 $mem;done; done
Workstation w/ 2x Xeon Gold 6442Y:
march mem result
x86-64 100 731.87779ms @ 11.193 GB/s
x86-64-v2 100 327.18580ms @ 25.038 GB/s
x86-64-v3 100 264.03547ms @ 31.026 GB/s
x86-64-v4 100 282.08065ms @ 29.041 GB/s
native 100 246.13766ms @ 33.282 GB/s
x86-64 100000 842.66827ms @ 9.722 GB/s
x86-64-v2 100000 604.52959ms @ 13.551 GB/s
x86-64-v3 100000 477.16239ms @ 17.168 GB/s
x86-64-v4 100000 476.07039ms @ 17.208 GB/s
native 100000 456.08080ms @ 17.962 GB/s
x86-64 1000000 845.51132ms @ 9.689 GB/s
x86-64-v2 1000000 612.07973ms @ 13.384 GB/s
x86-64-v3 1000000 485.23738ms @ 16.882 GB/s
x86-64-v4 1000000 483.86411ms @ 16.930 GB/s
native 1000000 462.88461ms @ 17.698 GB/s
Zen 4 laptop (AMD Ryzen 7 PRO 7840U):
march mem result
x86-64 100 417.19762ms @ 19.636 GB/s
x86-64-v2 100 130.67596ms @ 62.689 GB/s
x86-64-v3 100 97.07758ms @ 84.386 GB/s
x86-64-v4 100 95.67704ms @ 85.621 GB/s
native 100 95.15734ms @ 86.089 GB/s
x86-64 100000 431.38370ms @ 18.990 GB/s
x86-64-v2 100000 215.74856ms @ 37.970 GB/s
x86-64-v3 100000 199.74492ms @ 41.012 GB/s
x86-64-v4 100000 186.98300ms @ 43.811 GB/s
native 100000 187.68125ms @ 43.648 GB/s
x86-64 1000000 433.87893ms @ 18.881 GB/s
x86-64-v2 1000000 217.46561ms @ 37.670 GB/s
x86-64-v3 1000000 200.40667ms @ 40.877 GB/s
x86-64-v4 1000000 187.51978ms @ 43.686 GB/s
native 1000000 190.29273ms @ 43.049 GB/s
Workstation w/ 2x Xeon Gold 5215:
march mem result
x86-64 100 780.38881ms @ 10.497 GB/s
x86-64-v2 100 389.62005ms @ 21.026 GB/s
x86-64-v3 100 323.97294ms @ 25.286 GB/s
x86-64-v4 100 274.19493ms @ 29.877 GB/s
native 100 283.48674ms @ 28.897 GB/s
x86-64 100000 1112.63898ms @ 7.363 GB/s
x86-64-v2 100000 831.45641ms @ 9.853 GB/s
x86-64-v3 100000 696.20789ms @ 11.767 GB/s
x86-64-v4 100000 685.61636ms @ 11.948 GB/s
native 100000 689.78023ms @ 11.876 GB/s
x86-64 1000000 1128.65580ms @ 7.258 GB/s
x86-64-v2 1000000 843.92594ms @ 9.707 GB/s
x86-64-v3 1000000 718.78848ms @ 11.397 GB/s
x86-64-v4 1000000 687.68258ms @ 11.912 GB/s
native 1000000 705.34731ms @ 11.614 GB/s
That's quite the drastic difference between amd and intel. Of course it's also
comparing a multi-core server uarch (lower per-core bandwidth, much higher
aggregate bandwidth) with a client uarch.
The difference between the baseline CPU target and a more modern profile is
also rather impressive. Looks like some cpu-capability based dispatch would
likely be worth it, even if it didn't matter in my case due to -march=native.
I just realized that
a) The meson build doesn't use the relevant flags for bufpage.c - it didn't
matter in my numbers though because I was building with -O3 and
march=native.
This clearly ought to be fixed.
b) Neither build uses the optimized flags for pg_checksum and pg_upgrade, both
of which include checksum_imp.h directly.
This probably should be fixed too - perhaps by building the relevant code
once as part of fe_utils or such?
It probably matters less than it used to - these days -O2 turns on
-ftree-loop-vectorize -ftree-slp-vectorize. But loop unrolling isn't
enabled.
I do see a perf difference at -O2 between using/not using
-funroll-loops. Interestingly not at -O3, despite -funroll-loops not actually
being enabled by -O3. I think the relevant option that *is* turned on by O3 is
-fpeel-loops.
Here's a comparison of different flags run the 6442Y
printf '%16s\t%32s\t%16s\t%s\n' march flags mem result; for mem in 100 100000; do for march in x86-64 x86-64-v2 x86-64-v3 x86-64-v4 native; do for flags in "-O2" "-O2 -funroll-loops" "-O3" "-O3 -funroll-loops"; do printf "%16s\t%32s\t%16s\t" "$march" "$flags" "$mem"; gcc $flags -march=$march -I ~/src/postgresql/src/include/ -I src/include/ /tmp/bench-checksums.c -o bench-checksums-native && numactl --physcpubind 3 --membind 0 ./bench-checksums-native 3000000 $mem;done; done;done
march flags mem result
x86-64 -O2 100 2280.86253ms @ 10.775 GB/s
x86-64 -O2 -funroll-loops 100 2195.66942ms @ 11.193 GB/s
x86-64 -O3 100 2422.57588ms @ 10.145 GB/s
x86-64 -O3 -funroll-loops 100 2243.75826ms @ 10.953 GB/s
x86-64-v2 -O2 100 1243.68063ms @ 19.761 GB/s
x86-64-v2 -O2 -funroll-loops 100 979.67783ms @ 25.086 GB/s
x86-64-v2 -O3 100 988.80296ms @ 24.854 GB/s
x86-64-v2 -O3 -funroll-loops 100 991.31632ms @ 24.791 GB/s
x86-64-v3 -O2 100 1146.90165ms @ 21.428 GB/s
x86-64-v3 -O2 -funroll-loops 100 785.81395ms @ 31.275 GB/s
x86-64-v3 -O3 100 800.53627ms @ 30.699 GB/s
x86-64-v3 -O3 -funroll-loops 100 790.21230ms @ 31.101 GB/s
x86-64-v4 -O2 100 883.82916ms @ 27.806 GB/s
x86-64-v4 -O2 -funroll-loops 100 831.55372ms @ 29.554 GB/s
x86-64-v4 -O3 100 843.23141ms @ 29.145 GB/s
x86-64-v4 -O3 -funroll-loops 100 821.19969ms @ 29.927 GB/s
native -O2 100 1197.41357ms @ 20.524 GB/s
native -O2 -funroll-loops 100 718.05253ms @ 34.226 GB/s
native -O3 100 747.94090ms @ 32.858 GB/s
native -O3 -funroll-loops 100 751.52379ms @ 32.702 GB/s
x86-64 -O2 100000 2911.47087ms @ 8.441 GB/s
x86-64 -O2 -funroll-loops 100000 2525.45504ms @ 9.731 GB/s
x86-64 -O3 100000 2497.42016ms @ 9.841 GB/s
x86-64 -O3 -funroll-loops 100000 2346.33551ms @ 10.474 GB/s
x86-64-v2 -O2 100000 2124.10102ms @ 11.570 GB/s
x86-64-v2 -O2 -funroll-loops 100000 1819.09659ms @ 13.510 GB/s
x86-64-v2 -O3 100000 1613.45823ms @ 15.232 GB/s
x86-64-v2 -O3 -funroll-loops 100000 1607.09245ms @ 15.292 GB/s
x86-64-v3 -O2 100000 1972.89390ms @ 12.457 GB/s
x86-64-v3 -O2 -funroll-loops 100000 1432.58229ms @ 17.155 GB/s
x86-64-v3 -O3 100000 1533.18003ms @ 16.029 GB/s
x86-64-v3 -O3 -funroll-loops 100000 1539.39779ms @ 15.965 GB/s
x86-64-v4 -O2 100000 1591.96881ms @ 15.437 GB/s
x86-64-v4 -O2 -funroll-loops 100000 1434.91828ms @ 17.127 GB/s
x86-64-v4 -O3 100000 1454.30133ms @ 16.899 GB/s
x86-64-v4 -O3 -funroll-loops 100000 1429.13733ms @ 17.196 GB/s
native -O2 100000 1980.53734ms @ 12.409 GB/s
native -O2 -funroll-loops 100000 1373.95337ms @ 17.887 GB/s
native -O3 100000 1517.90164ms @ 16.191 GB/s
native -O3 -funroll-loops 100000 1508.37021ms @ 16.293 GB/s
> > > Is it just that the calculation is slow, or is it the fact that checksumming
> > > needs to bring the page into the CPU cache. Did you notice any hints which
> > > might be the case?
> >
> > I don't think the issue is that checksumming pulls the data into CPU caches
> >
> > 1) This is visible with SELECT that actually uses the data
> >
> > 2) I added prefetching to avoid any meaningful amount of cache misses and it
> > doesn't change the overall timing much
> >
> > 3) It's visible with buffered IO, which has pulled the data into CPU caches
> > already
>
> I didn't yet check the code, when doing aio completions checksumming
> be running on the same core as is going to be using the page?
With io_uring normally yes, the exception being that another backend that
needs the same page could end up running the completion.
With worker mode normally no.
Greetings,
Andres Freund
^ permalink raw reply [nested|flat] 29+ messages in thread
* Re: AIO v2.0
@ 2025-01-10 10:33 Ants Aasma <ants.aasma@cybertec.at>
parent: Andres Freund <andres@anarazel.de>
0 siblings, 0 replies; 29+ messages in thread
From: Ants Aasma @ 2025-01-10 10:33 UTC (permalink / raw)
To: Andres Freund <andres@anarazel.de>; +Cc: Jakub Wartak <jakub.wartak@enterprisedb.com>; Thomas Munro <thomas.munro@gmail.com>; pgsql-hackers@postgresql.org, Heikki Linnakangas <hlinnaka@iki.fi>; 陈宗志 <baotiao@gmail.com>
On Thu, 9 Jan 2025 at 22:53, Andres Freund <andres@anarazel.de> wrote:
<Edited to highlight interesting numbers>
> Workstation w/ 2x Xeon Gold 6442Y:
>
> march mem result
> native 100 246.13766ms @ 33.282 GB/s
> native 100000 456.08080ms @ 17.962 GB/s
>
> Zen 4 laptop (AMD Ryzen 7 PRO 7840U):
> march mem result
> native 100 95.15734ms @ 86.089 GB/s
> native 100000 187.68125ms @ 43.648 GB/s
>
> Workstation w/ 2x Xeon Gold 5215:
> march mem result
> native 100 283.48674ms @ 28.897 GB/s
> native 100000 689.78023ms @ 11.876 GB/s
>
> That's quite the drastic difference between amd and intel. Of course it's also
> comparing a multi-core server uarch (lower per-core bandwidth, much higher
> aggregate bandwidth) with a client uarch.
In hindsight building the hash around mulld primitive was a bad decision
because Intel for whatever reason decided to kill the performance of it:
vpmulld latency throughput
(values/cycle)
Sandy Bridge 5 4
Alder Lake 10 8
Zen 4 3 16
Zen 5 3 32
Most top performing hashes these days seem to be built around AES
instructions.
But I was curious why there is such a difference in streaming results.
Turns out that for whatever reason one core gets access to much less
bandwidth on Intel than on AMD. [1]
This made me take another look at your previous prewarm numbers. It looks
like performance is suspiciously proportional to the number of copies of
data the CPU has to make:
config checksums time in ms number of copies
buffered io_engine=io_uring 0 3883.127 2
buffered io_engine=io_uring 1 5880.892 3
direct io_engine=io_uring 0 2067.142 1
direct io_engine=io_uring 1 3835.968 2
To me that feels like there is a bandwidth bottleneck in this workload and
checksumming the page when the contents is not looked at just adds to
consumed bandwidth, bringing down the performance correspondingly.
This doesn't explain why you observed slowdown in the select case, but I
think that might be due to the per-core bandwidth limitation. Both cases
might pull in the same amount of data into the cache, but without checksums
it is spread out over a longer time allowing other work to happen
concurrently.
[1] https://chipsandcheese.com/p/a-peek-at-sapphire-rapids#%C2%A7bandwidth
> The difference between the baseline CPU target and a more modern profile is
> also rather impressive. Looks like some cpu-capability based dispatch would
> likely be worth it, even if it didn't matter in my case due to -march=native.
Yes, along with using function attributes for the optimization flags to avoid
the build system hacks.
--
Ants
^ permalink raw reply [nested|flat] 29+ messages in thread
end of thread, other threads:[~2025-01-10 10:33 UTC | newest]
Thread overview: 29+ messages (download: mbox mbox.gz follow: Atom feed)
-- links below jump to the message on this page --
2024-09-01 06:27 AIO v2.0 Andres Freund <andres@anarazel.de>
2024-09-02 10:03 ` Heikki Linnakangas <hlinnaka@iki.fi>
2024-09-03 14:29 ` Andres Freund <andres@anarazel.de>
2024-09-04 17:37 ` 陈宗志 <baotiao@gmail.com>
2024-09-06 19:47 ` Andres Freund <andres@anarazel.de>
2024-09-12 21:55 ` Robert Pang <robertpang@google.com>
2024-09-16 17:56 ` Andres Freund <andres@anarazel.de>
2024-09-06 01:42 ` David Rowley <dgrowleyml@gmail.com>
2024-09-06 19:38 ` Andres Freund <andres@anarazel.de>
2024-09-16 14:43 ` Noah Misch <noah@leadboat.com>
2024-09-16 17:51 ` Andres Freund <andres@anarazel.de>
2024-09-17 18:08 ` Noah Misch <noah@leadboat.com>
2024-09-30 14:49 ` Andres Freund <andres@anarazel.de>
2024-09-30 15:55 ` Matthias van de Meent <boekewurm+postgres@gmail.com>
2024-09-30 19:29 ` Noah Misch <noah@leadboat.com>
2024-11-18 12:19 ` Jakub Wartak <jakub.wartak@enterprisedb.com>
2024-12-19 22:29 ` Andres Freund <andres@anarazel.de>
2024-12-19 22:34 ` Tom Lane <tgl@sss.pgh.pa.us>
2024-12-20 00:54 ` Andres Freund <andres@anarazel.de>
2024-12-20 17:27 ` Jelte Fennema-Nio <postgres@jeltef.nl>
2024-12-20 17:39 ` Andres Freund <andres@anarazel.de>
2025-01-06 16:28 ` Andres Freund <andres@anarazel.de>
2025-01-08 14:04 ` Jakub Wartak <jakub.wartak@enterprisedb.com>
2025-01-08 20:58 ` Andres Freund <andres@anarazel.de>
2025-01-09 08:59 ` Ants Aasma <ants.aasma@cybertec.at>
2025-01-09 16:25 ` Andres Freund <andres@anarazel.de>
2025-01-09 18:10 ` Ants Aasma <ants.aasma@cybertec.at>
2025-01-09 20:53 ` Andres Freund <andres@anarazel.de>
2025-01-10 10:33 ` Ants Aasma <ants.aasma@cybertec.at>
This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox