pg.ddx.io pgsql-hackers@postgresql.org mailing list archive
help / color / mirror / Atom feedAIO v2.5
153+ messages / 15 participants
[nested] [flat]
* AIO v2.5
@ 2025-03-04 19:00 Andres Freund <andres@anarazel.de>
2025-03-06 11:36 ` Re: AIO v2.5 Jakub Wartak <jakub.wartak@enterprisedb.com>
2025-03-06 15:33 ` Re: AIO v2.5 Robert Haas <robertmhaas@gmail.com>
2025-03-10 18:23 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
0 siblings, 3 replies; 153+ messages in thread
From: Andres Freund @ 2025-03-04 19:00 UTC (permalink / raw)
To: pgsql-hackers; +Cc: Thomas Munro <thomas.munro@gmail.com>; Heikki Linnakangas <hlinnaka@iki.fi>; Noah Misch <noah@leadboat.com>; Robert Haas <robertmhaas@gmail.com>; Jakub Wartak <jakub.wartak@enterprisedb.com>
Hi,
Attached is v2.5 of the AIO patchset.
Relative to 2.4 I:
- Committed some earlier commits. I ended up *not* committing the patch to
create resowners in more backends (e.g. walsender), as that's not really a
dependency for now.
One of the more important things to get committed was in a separate thread:
https://postgr.es/m/b6vveqz6r3wno66rho5lqi6z5kyhfgtvi3jcodyq5rlpp3cu44%40c6dsgf3z7yhs
Now relpath() can be used for logging while in a critical section. That
alone allowed to remove most of the remaining FIXMEs.
- Split md.c read/write patches, the write side is more complicated and isn't
needed before write support arrives (much later in the queue and very likely
not for 18).
The complicated bit about write support is needing to
register_dirty_segment() after completion of the write. If
RegisterSyncRequest() fails, the IO completer needs to open the file and
sync itself, unfortunately PathNameOpenFile() allocates memory, which isn't
ok while in a critical section (even though it'd not be detected, as it's
using malloc()).
- Reordered patches so that Thomas' read_stream work is after the basic AIO
infrastructure patches, there's no dependency to the earlier patches
I think Thomas might have a newer version of some of these, but since
they're not intended to be committed as part of this, I didn't spend the
time to rebase to the last version.
- Added a small bit of data that can be provided to callbacks, that makes it a
lot cleaner to transport information like ZERO_ON_ERROR.
I also did s/shared_callbacks/callbacks/, as the prior name was outdated.
- Substantially expanded tests, most importantly generic temp file tests and
AIO specific cross-backend tests
As part of the expanded tests I also needed to export TerminateBufferIO(),
like, as previously mentioned, already done in an earlier version for
StartBufferIO(). Nobody commented on that, so I think that's ok.
I also renamed the tests away from the very inventively named tbl_a, tbl_b...
- Moved the commit to create resownern in more places to much later in the
queue, it's not actually needed for bufmgr.c IO, and nothing needing it will
land in 18
- Added a proper commit message fo the main commit. I'd appreciate folks
reading through it. I'm sure I forgot a lot of folks and a lot of things.
- Did a fair bit of of comment polishing
- Addressed an XXX in the "aio infrastructure" commit suggesting that we might
want to error out if a backend is waiting on is own unsubmitted IO. Noah
argued for erroring out. I now made it so.
- Temporarily added a commit to increase open-file limit on openbsd. I saw
related errors without this patch too, but it fails more often with. I
already sent a separate email about this.
At this point I am not aware of anything significant left to do in the main
AIO commit, safe some of the questions below. There's a lot more potential
optimizations etc, but this is already a very complicated piece of work, so I
think they just has to wait for later.
There are a few things to clean up in the bufmgr.c commits, I don't yet quite
like the function naming and there could be a bit less duplication. But I
don't think that needs to be resolved before the main commit.
Questions:
- My current thinking is that we'd set io_method = worker initially - so we
actually get some coverage - and then decide whether to switch to
io_method=sync by default for 18 sometime around beta1/2. Does that sound
reasonable?
- We could reduce memory usage a tiny bit if we made the mapping between
pgproc and per-backend-aio-state more complicated, i.e. not just indexed by
ProcNumber. Right now IO workers have the per-backend AIO state, but don't
actually need it. I'm mildly inclined to think that the complexity isn't
worth it, but on the fence.
- Three of the commits in the series really are just precursor commits to
their subsequent commits, which I found helpful for development and review,
namely:
- aio: Basic subsystem initialization
- aio: Skeleton IO worker infrastructure
- aio: Add liburing dependency
Not sure if it's worth keeping these separate or whether they should just be
merged with their "real commit".
- Thomas suggested renaming
COMPLETED_IO->COMPLETED,
COMPLETED_SHARED->TERMINATED_BY_COMPLETER,
COMPLETED_SHARED->TERMINATED_BY_SUBMITTER
in
https://www.postgresql.org/message-id/CA%2BhUKGLxH1tsUgzZfng4BU6GqnS6bKF2ThvxH1_w5c7-sLRKQw%40mail.g...
While the other things in the email were commented upon by others and
addressed in v2.4, the naming aspect wasn't further remarked upon by others.
I'm not personally in love with the suggested names, but I could live with
them.
- Right now this series defines PGAIO_VERBOSE to 1. That's good for debugging,
but all the ereport()s add a noticeable amount of overhead at high IO
throughput (at multiple gigabytes/second), so that's probably not right
forever. I'd leave this on initially and then change it to default to off
later. I think that's ok?
- To allow io_workers to be PGC_SIGHUP, and to eventually allow to
automatically in/decrease active workers, the max number of workers (32) is
always allocated. That means we use more semaphores than before. I think
that's ok, it's not 1995 anymore. Alternatively we can add a
"io_workers_max" GUC and probe for it in initdb.
- pg_stat_aios currently has the IO Handle flags as dedicated columns. Not
sure that's great?
They could be an enum array or such too? That'd perhaps be a bit more
extensible? OTOH, we don't currently use enums in the catalogs and arrays
are somewhat annoying to conjure up from C.
Todo:
- A few more passes over the main commit, I'm sure there's a few more inartful
comments, odd formatting and such.
- Check if there's a decent way to deduplicate pgaio_io_call_complete_shared() and
pgaio_io_call_complete_local()
- Figure out how to deduplicate support for LockBufferForCleanup() in
TerminateBufferIO().
- Documentation for pg_stat_aios.
- Check if documentation for track_io_timing needs to be adjusted, after the
bufmgr.c changes we only track waiting for an IO.
- Some of the test_aio code is specific to non-temp tables, it probably is
worth generalizing to deal with temp tables and invoke them for both.
Greetings,
Andres
Attachments:
[text/x-diff] v2.5-0001-aio-Basic-subsystem-initialization.patch (16.7K, ../../vz5i2x2wkjjpp6l2z4g4l3umpxktrxsejgdwefi7n6kzw666h3@rnhlvlpadig7/2-v2.5-0001-aio-Basic-subsystem-initialization.patch)
download | inline diff:
From 4557d9f70bf051beb8622de086756505bce684b8 Mon Sep 17 00:00:00 2001
From: Andres Freund <andres@anarazel.de>
Date: Tue, 4 Mar 2025 10:44:56 -0500
Subject: [PATCH v2.5 01/30] aio: Basic subsystem initialization
This is just separate to make it easier to review the tendrils into various
places. It will likely be merged with the subsequent commit before being
merged.
---
src/include/storage/aio.h | 38 +++++++++++
src/include/storage/aio_subsys.h | 28 ++++++++
src/include/utils/guc.h | 1 +
src/include/utils/guc_hooks.h | 2 +
src/include/utils/resowner.h | 5 ++
src/backend/storage/aio/Makefile | 2 +
src/backend/storage/aio/aio.c | 67 +++++++++++++++++++
src/backend/storage/aio/aio_init.c | 37 ++++++++++
src/backend/storage/aio/meson.build | 2 +
src/backend/storage/ipc/ipci.c | 3 +
src/backend/utils/init/postinit.c | 7 ++
src/backend/utils/misc/guc_tables.c | 23 +++++++
src/backend/utils/misc/postgresql.conf.sample | 6 ++
src/backend/utils/resowner/resowner.c | 29 ++++++++
doc/src/sgml/config.sgml | 51 ++++++++++++++
src/tools/pgindent/typedefs.list | 1 +
16 files changed, 302 insertions(+)
create mode 100644 src/include/storage/aio.h
create mode 100644 src/include/storage/aio_subsys.h
create mode 100644 src/backend/storage/aio/aio.c
create mode 100644 src/backend/storage/aio/aio_init.c
diff --git a/src/include/storage/aio.h b/src/include/storage/aio.h
new file mode 100644
index 00000000000..e79d5343038
--- /dev/null
+++ b/src/include/storage/aio.h
@@ -0,0 +1,38 @@
+/*-------------------------------------------------------------------------
+ *
+ * aio.h
+ * Main AIO interface
+ *
+ *
+ * Portions Copyright (c) 1996-2025, PostgreSQL Global Development Group
+ * Portions Copyright (c) 1994, Regents of the University of California
+ *
+ * src/include/storage/aio.h
+ *
+ *-------------------------------------------------------------------------
+ */
+#ifndef AIO_H
+#define AIO_H
+
+
+
+/* Enum for io_method GUC. */
+typedef enum IoMethod
+{
+ IOMETHOD_SYNC = 0,
+} IoMethod;
+
+/* We'll default to synchronous execution. */
+#define DEFAULT_IO_METHOD IOMETHOD_SYNC
+
+
+struct dlist_node;
+extern void pgaio_io_release_resowner(struct dlist_node *ioh_node, bool on_error);
+
+
+/* GUCs */
+extern PGDLLIMPORT int io_method;
+extern PGDLLIMPORT int io_max_concurrency;
+
+
+#endif /* AIO_H */
diff --git a/src/include/storage/aio_subsys.h b/src/include/storage/aio_subsys.h
new file mode 100644
index 00000000000..0b8aaaf00d3
--- /dev/null
+++ b/src/include/storage/aio_subsys.h
@@ -0,0 +1,28 @@
+/*-------------------------------------------------------------------------
+ *
+ * aio_subsys.h
+ * Interaction with AIO as a subsystem, rather than actually issuing AIO
+ *
+ * This header is for AIO related functionality that's being called by files
+ * that don't perform AIO but interact with it in some form. E.g. postmaster.c
+ * and shared memory initialization need to initialize AIO but don't perform
+ * AIO.
+ *
+ * Portions Copyright (c) 1996-2025, PostgreSQL Global Development Group
+ * Portions Copyright (c) 1994, Regents of the University of California
+ *
+ * src/include/storage/aio_subsys.h
+ *
+ *-------------------------------------------------------------------------
+ */
+#ifndef AIO_SUBSYS_H
+#define AIO_SUBSYS_H
+
+
+/* aio_init.c */
+extern Size AioShmemSize(void);
+extern void AioShmemInit(void);
+
+extern void pgaio_init_backend(void);
+
+#endif /* AIO_SUBSYS_H */
diff --git a/src/include/utils/guc.h b/src/include/utils/guc.h
index 1233e07d7da..de3bc37264f 100644
--- a/src/include/utils/guc.h
+++ b/src/include/utils/guc.h
@@ -318,6 +318,7 @@ extern PGDLLIMPORT bool optimize_bounded_sort;
*/
extern PGDLLIMPORT const struct config_enum_entry archive_mode_options[];
extern PGDLLIMPORT const struct config_enum_entry dynamic_shared_memory_options[];
+extern PGDLLIMPORT const struct config_enum_entry io_method_options[];
extern PGDLLIMPORT const struct config_enum_entry recovery_target_action_options[];
extern PGDLLIMPORT const struct config_enum_entry wal_level_options[];
extern PGDLLIMPORT const struct config_enum_entry wal_sync_method_options[];
diff --git a/src/include/utils/guc_hooks.h b/src/include/utils/guc_hooks.h
index 951451a9765..d1b49adc547 100644
--- a/src/include/utils/guc_hooks.h
+++ b/src/include/utils/guc_hooks.h
@@ -62,6 +62,8 @@ extern bool check_default_with_oids(bool *newval, void **extra,
extern bool check_effective_io_concurrency(int *newval, void **extra,
GucSource source);
extern bool check_huge_page_size(int *newval, void **extra, GucSource source);
+extern void assign_io_method(int newval, void *extra);
+extern bool check_io_max_concurrency(int *newval, void **extra, GucSource source);
extern const char *show_in_hot_standby(void);
extern bool check_locale_messages(char **newval, void **extra, GucSource source);
extern void assign_locale_messages(const char *newval, void *extra);
diff --git a/src/include/utils/resowner.h b/src/include/utils/resowner.h
index e8d452ca7ee..aede4bfc820 100644
--- a/src/include/utils/resowner.h
+++ b/src/include/utils/resowner.h
@@ -164,4 +164,9 @@ struct LOCALLOCK;
extern void ResourceOwnerRememberLock(ResourceOwner owner, struct LOCALLOCK *locallock);
extern void ResourceOwnerForgetLock(ResourceOwner owner, struct LOCALLOCK *locallock);
+/* special support for AIO */
+struct dlist_node;
+extern void ResourceOwnerRememberAioHandle(ResourceOwner owner, struct dlist_node *ioh_node);
+extern void ResourceOwnerForgetAioHandle(ResourceOwner owner, struct dlist_node *ioh_node);
+
#endif /* RESOWNER_H */
diff --git a/src/backend/storage/aio/Makefile b/src/backend/storage/aio/Makefile
index 2f29a9ec4d1..eaeaeeee8e3 100644
--- a/src/backend/storage/aio/Makefile
+++ b/src/backend/storage/aio/Makefile
@@ -9,6 +9,8 @@ top_builddir = ../../../..
include $(top_builddir)/src/Makefile.global
OBJS = \
+ aio.o \
+ aio_init.o \
read_stream.o
include $(top_srcdir)/src/backend/common.mk
diff --git a/src/backend/storage/aio/aio.c b/src/backend/storage/aio/aio.c
new file mode 100644
index 00000000000..8eb6a5f1292
--- /dev/null
+++ b/src/backend/storage/aio/aio.c
@@ -0,0 +1,67 @@
+/*-------------------------------------------------------------------------
+ *
+ * aio.c
+ * AIO - Core Logic
+ *
+ * Portions Copyright (c) 1996-2025, PostgreSQL Global Development Group
+ * Portions Copyright (c) 1994, Regents of the University of California
+ *
+ * IDENTIFICATION
+ * src/backend/storage/aio/aio.c
+ *
+ *-------------------------------------------------------------------------
+ */
+
+#include "postgres.h"
+
+#include "lib/ilist.h"
+#include "storage/aio.h"
+#include "utils/guc.h"
+#include "utils/guc_hooks.h"
+
+
+/* Options for io_method. */
+const struct config_enum_entry io_method_options[] = {
+ {"sync", IOMETHOD_SYNC, false},
+ {NULL, 0, false}
+};
+
+/* GUCs */
+int io_method = DEFAULT_IO_METHOD;
+int io_max_concurrency = -1;
+
+
+
+void
+assign_io_method(int newval, void *extra)
+{
+}
+
+bool
+check_io_max_concurrency(int *newval, void **extra, GucSource source)
+{
+ if (*newval == -1)
+ {
+ /*
+ * Auto-tuning will be applied later during startup, as auto-tuning
+ * depends on the value of various GUCs.
+ */
+ return true;
+ }
+ else if (*newval == 0)
+ {
+ GUC_check_errdetail("Only -1 or values bigger than 0 are valid.");
+ return false;
+ }
+
+ return true;
+}
+
+/*
+ * Release IO handle during resource owner cleanup.
+ */
+void
+pgaio_io_release_resowner(dlist_node *ioh_node, bool on_error)
+{
+ /* placeholder for later */
+}
diff --git a/src/backend/storage/aio/aio_init.c b/src/backend/storage/aio/aio_init.c
new file mode 100644
index 00000000000..aeacc144149
--- /dev/null
+++ b/src/backend/storage/aio/aio_init.c
@@ -0,0 +1,37 @@
+/*-------------------------------------------------------------------------
+ *
+ * aio_init.c
+ * AIO - Subsystem Initialization
+ *
+ * Portions Copyright (c) 1996-2025, PostgreSQL Global Development Group
+ * Portions Copyright (c) 1994, Regents of the University of California
+ *
+ * IDENTIFICATION
+ * src/backend/storage/aio/aio_init.c
+ *
+ *-------------------------------------------------------------------------
+ */
+
+#include "postgres.h"
+
+#include "storage/aio_subsys.h"
+
+
+
+Size
+AioShmemSize(void)
+{
+ Size sz = 0;
+
+ return sz;
+}
+
+void
+AioShmemInit(void)
+{
+}
+
+void
+pgaio_init_backend(void)
+{
+}
diff --git a/src/backend/storage/aio/meson.build b/src/backend/storage/aio/meson.build
index 8abe0eb4863..c822fd4ddf7 100644
--- a/src/backend/storage/aio/meson.build
+++ b/src/backend/storage/aio/meson.build
@@ -1,5 +1,7 @@
# Copyright (c) 2024-2025, PostgreSQL Global Development Group
backend_sources += files(
+ 'aio.c',
+ 'aio_init.c',
'read_stream.c',
)
diff --git a/src/backend/storage/ipc/ipci.c b/src/backend/storage/ipc/ipci.c
index 174eed70367..2fa045e6b0f 100644
--- a/src/backend/storage/ipc/ipci.c
+++ b/src/backend/storage/ipc/ipci.c
@@ -37,6 +37,7 @@
#include "replication/slotsync.h"
#include "replication/walreceiver.h"
#include "replication/walsender.h"
+#include "storage/aio_subsys.h"
#include "storage/bufmgr.h"
#include "storage/dsm.h"
#include "storage/dsm_registry.h"
@@ -148,6 +149,7 @@ CalculateShmemSize(int *num_semaphores)
size = add_size(size, WaitEventCustomShmemSize());
size = add_size(size, InjectionPointShmemSize());
size = add_size(size, SlotSyncShmemSize());
+ size = add_size(size, AioShmemSize());
/* include additional requested shmem from preload libraries */
size = add_size(size, total_addin_request);
@@ -340,6 +342,7 @@ CreateOrAttachShmemStructs(void)
StatsShmemInit();
WaitEventCustomShmemInit();
InjectionPointShmemInit();
+ AioShmemInit();
}
/*
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index b428a59bdd2..ad1fb624e3c 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -43,6 +43,7 @@
#include "replication/slot.h"
#include "replication/slotsync.h"
#include "replication/walsender.h"
+#include "storage/aio_subsys.h"
#include "storage/bufmgr.h"
#include "storage/fd.h"
#include "storage/ipc.h"
@@ -628,6 +629,12 @@ BaseInit(void)
*/
pgstat_initialize();
+ /*
+ * Initialize AIO before infrastructure that might need to actually
+ * execute AIO.
+ */
+ pgaio_init_backend();
+
/* Do local initialization of storage and buffer managers */
InitSync();
smgrinit();
diff --git a/src/backend/utils/misc/guc_tables.c b/src/backend/utils/misc/guc_tables.c
index ad25cbb39c5..28dcc465762 100644
--- a/src/backend/utils/misc/guc_tables.c
+++ b/src/backend/utils/misc/guc_tables.c
@@ -72,6 +72,7 @@
#include "replication/slot.h"
#include "replication/slotsync.h"
#include "replication/syncrep.h"
+#include "storage/aio.h"
#include "storage/bufmgr.h"
#include "storage/bufpage.h"
#include "storage/large_object.h"
@@ -3254,6 +3255,18 @@ struct config_int ConfigureNamesInt[] =
NULL, NULL, NULL
},
+ {
+ {"io_max_concurrency",
+ PGC_POSTMASTER,
+ RESOURCES_IO,
+ gettext_noop("Max number of IOs that one process can execute simultaneously."),
+ NULL,
+ },
+ &io_max_concurrency,
+ -1, -1, 1024,
+ check_io_max_concurrency, NULL, NULL
+ },
+
{
{"backend_flush_after", PGC_USERSET, RESOURCES_IO,
gettext_noop("Number of pages after which previously performed writes are flushed to disk."),
@@ -5299,6 +5312,16 @@ struct config_enum ConfigureNamesEnum[] =
NULL, NULL, NULL
},
+ {
+ {"io_method", PGC_POSTMASTER, RESOURCES_IO,
+ gettext_noop("Selects the method for executing asynchronous I/O."),
+ NULL
+ },
+ &io_method,
+ DEFAULT_IO_METHOD, io_method_options,
+ NULL, assign_io_method, NULL
+ },
+
/* End-of-list marker */
{
{NULL, 0, 0, NULL, NULL}, NULL, 0, NULL, NULL, NULL, NULL
diff --git a/src/backend/utils/misc/postgresql.conf.sample b/src/backend/utils/misc/postgresql.conf.sample
index 2d1de9c37bd..a31c9963ce4 100644
--- a/src/backend/utils/misc/postgresql.conf.sample
+++ b/src/backend/utils/misc/postgresql.conf.sample
@@ -202,6 +202,12 @@
#maintenance_io_concurrency = 10 # 1-1000; 0 disables prefetching
#io_combine_limit = 128kB # usually 1-32 blocks (depends on OS)
+#io_method = sync # sync (change requires restart)
+#io_max_concurrency = -1 # Max number of IOs that one process
+ # can execute simultaneously
+ # -1 sets based on shared_buffers
+ # (change requires restart)
+
# - Worker Processes -
#max_worker_processes = 8 # (change requires restart)
diff --git a/src/backend/utils/resowner/resowner.c b/src/backend/utils/resowner/resowner.c
index ac5ca4a765e..76b9cec1e26 100644
--- a/src/backend/utils/resowner/resowner.c
+++ b/src/backend/utils/resowner/resowner.c
@@ -47,6 +47,8 @@
#include "common/hashfn.h"
#include "common/int.h"
+#include "lib/ilist.h"
+#include "storage/aio.h"
#include "storage/ipc.h"
#include "storage/predicate.h"
#include "storage/proc.h"
@@ -155,6 +157,12 @@ struct ResourceOwnerData
/* The local locks cache. */
LOCALLOCK *locks[MAX_RESOWNER_LOCKS]; /* list of owned locks */
+
+ /*
+ * AIO handles need be registered in critical sections and therefore
+ * cannot use the normal ResoureElem mechanism.
+ */
+ dlist_head aio_handles;
};
@@ -425,6 +433,8 @@ ResourceOwnerCreate(ResourceOwner parent, const char *name)
parent->firstchild = owner;
}
+ dlist_init(&owner->aio_handles);
+
return owner;
}
@@ -725,6 +735,13 @@ ResourceOwnerReleaseInternal(ResourceOwner owner,
* so issue warnings. In the abort case, just clean up quietly.
*/
ResourceOwnerReleaseAll(owner, phase, isCommit);
+
+ while (!dlist_is_empty(&owner->aio_handles))
+ {
+ dlist_node *node = dlist_head_node(&owner->aio_handles);
+
+ pgaio_io_release_resowner(node, !isCommit);
+ }
}
else if (phase == RESOURCE_RELEASE_LOCKS)
{
@@ -1082,3 +1099,15 @@ ResourceOwnerForgetLock(ResourceOwner owner, LOCALLOCK *locallock)
elog(ERROR, "lock reference %p is not owned by resource owner %s",
locallock, owner->name);
}
+
+void
+ResourceOwnerRememberAioHandle(ResourceOwner owner, struct dlist_node *ioh_node)
+{
+ dlist_push_tail(&owner->aio_handles, ioh_node);
+}
+
+void
+ResourceOwnerForgetAioHandle(ResourceOwner owner, struct dlist_node *ioh_node)
+{
+ dlist_delete_from(&owner->aio_handles, ioh_node);
+}
diff --git a/doc/src/sgml/config.sgml b/doc/src/sgml/config.sgml
index d2fa5f7d1a9..7d5482f31cd 100644
--- a/doc/src/sgml/config.sgml
+++ b/doc/src/sgml/config.sgml
@@ -2644,6 +2644,57 @@ include_dir 'conf.d'
</para>
</listitem>
</varlistentry>
+
+ <varlistentry id="guc-io-max-concurrency" xreflabel="io_max_concurrency">
+ <term><varname>io_max_concurrency</varname> (<type>integer</type>)
+ <indexterm>
+ <primary><varname>io_max_concurrency</varname> configuration parameter</primary>
+ </indexterm>
+ </term>
+ <listitem>
+ <para>
+ Controls the maximum number of I/O operations that one process can
+ execute simultaneously.
+ </para>
+ <para>
+ The default setting of <literal>-1</literal> selects a number based
+ on <xref linkend="guc-shared-buffers"/> and the maximum number of
+ processes (<xref linkend="guc-max-connections"/>, <xref
+ linkend="guc-autovacuum-worker-slots"/>, <xref
+ linkend="guc-max-worker-processes"/> and <xref
+ linkend="guc-max-wal-senders"/>), but not more than
+ <literal>64</literal>.
+ </para>
+ <para>
+ This parameter can only be set at server start.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry id="guc-io-method" xreflabel="io_method">
+ <term><varname>io_method</varname> (<type>enum</type>)
+ <indexterm>
+ <primary><varname>io_method</varname> configuration parameter</primary>
+ </indexterm>
+ </term>
+ <listitem>
+ <para>
+ Selects the method for executing asynchronous I/O.
+ Possible values are:
+ <itemizedlist>
+ <listitem>
+ <para>
+ <literal>sync</literal> (execute asynchronous I/O synchronously)
+ </para>
+ </listitem>
+ </itemizedlist>
+ </para>
+ <para>
+ This parameter can only be set at server start.
+ </para>
+ </listitem>
+ </varlistentry>
+
</variablelist>
</sect2>
diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list
index 19ff271ba50..c80365974e4 100644
--- a/src/tools/pgindent/typedefs.list
+++ b/src/tools/pgindent/typedefs.list
@@ -1276,6 +1276,7 @@ IntoClause
InvalMessageArray
InvalidationInfo
InvalidationMsgsGroup
+IoMethod
IpcMemoryId
IpcMemoryKey
IpcMemoryState
--
2.48.1.76.g4e746b1a31.dirty
[text/x-diff] v2.5-0002-aio-Add-asynchronous-I-O-infrastructure.patch (93.4K, ../../vz5i2x2wkjjpp6l2z4g4l3umpxktrxsejgdwefi7n6kzw666h3@rnhlvlpadig7/3-v2.5-0002-aio-Add-asynchronous-I-O-infrastructure.patch)
download | inline diff:
From a7db2aee940dd5c7595fc99f36892e47d988da92 Mon Sep 17 00:00:00 2001
From: Andres Freund <andres@anarazel.de>
Date: Tue, 11 Feb 2025 14:57:59 -0500
Subject: [PATCH v2.5 02/30] aio: Add asynchronous I/O infrastructure
The main motivations to use AIO in PostgreSQL are:
a) Reduce the time spent waiting for IO by issuing IO sufficiently early.
In a few places we have approximated this using posix_fadvise() based
prefetching, but that is fairly limited (no completion feedback, double the
syscalls, only works with buffered IO, only works on some OSs).
b) Allow to use Direct-I/O (DIO).
DIO can offload most of the work for IO to hardware and thus increase
throughput / decrease CPU utilization, as well as reduce latency. While we
have gained the ability to configure DIO in d4e71df6, it is not yet usable
for real world workloads, as every IO is executed synchronously.
For portability, the new AIO infrastructure allows to implement AIO using
different methods. The choice of the AIO method is controlled by a new
io_method GUC. As of this commit, the only implemented method is "sync",
i.e. AIO is not actually executed asynchronously. The "sync" method exists to
allow to bypass most of the new code initially.
Subsequent commits will introduce additional IO methods, including a
cross-platform method implemented using worker processes and a linux specific
method using io_uring.
To allow different parts of postgres to use AIO, the core AIO infrastructure
does not need to know what kind of files it is operating on. The necessary
behavioral differences for different files are abstracted as "AIO
Targets". One example target would be smgr. For boring portability reasons all
targets currently need to be added to an array in aio_target.c. This commit
does not implement any AIO targets, just the infrastructure for them. The smgr
target will be added in a later commit.
Completion (and other events) of IOs for one type of file (i.e. one AIO
target) need to be reacted to differently based on the IO operation and the
callsite. This is made possible by callbacks that can be registered on
IOs. E.g. an smgr read into a local buffer does not need to update the
corresponding BufferDesc (as there is none), but a read into shared buffers
does. This commit does not contain any callbacks, they will be added in
subsequent commits.
For now the AIO infrastructure only understands READV and WRITEV operations,
but it is expected that more operations will be added. E.g. fsync/fdatasync,
flush_range and network operations like send/recv.
As of this commit nothing uses the AIO infrastructure. Other commits will add
an smgr target, md.c and bufmgr.c callbacks and then finally use AIO for
read_stream.c IO, which, in one fell swoop, will convert all read stream users
to AIO.
The goal is to use AIO in many more places. There are patches to use AIO for
checkpointer and bgwriter that are reasonably close to being ready. There also
are prototypes to use it for WAL, relation extension, backend writes and many
more. Those prototypes were important to ensure the design of the AIO
subsystem is not too limiting (e.g. WAL writes need to happen in critical
sections, which influenced a lot of the design).
A future commit will add an AIO README explaining the AIO architecture and how
to use the AIO subsystem. The README is added later, as it references details
only added in later commits.
Many many more people than the folks named below have contributed with
feedback, work on semi-independent patches etc. E.g. various folks have
contributed patches to use the read stream infrastructure (added by Thomas in
b5a9b18cd0b) in more places. Similarly, a *lot* of folks have contributed to
the CI infrastructure, that I started to work on to make adding AIO feasible.
Some of the work by contributors has gone into the "v1" prototype of AIO,
which heavily influenced the current design of the AIO subsystem. None of the
code from that directly survives, but without the prototype, the current
version of the AIO infrastructure would not exist.
Similarly, the reviewers below have not necessarily looked at the current
design or the whole infrastructure, but have provided very valuable input. I
am to blame for problems, not they.
Author: Andres Freund <andres@anarazel.de>
Co-authored-by: Thomas Munro <thomas.munro@gmail.com>
Co-authored-by: Nazir Bilal Yavuz <byavuz81@gmail.com>
Co-authored-by: Melanie Plageman <melanieplageman@gmail.com>
Reviewed-by: Heikki Linnakangas <hlinnaka@iki.fi>
Reviewed-by: Noah Misch <noah@leadboat.com>
Reviewed-by: Jakub Wartak <jakub.wartak@enterprisedb.com>
Reviewed-by: Melanie Plageman <melanieplageman@gmail.com>
Reviewed-by: Robert Haas <robertmhaas@gmail.com>
Reviewed-by: Dmitry Dolgov <9erthalion6@gmail.com>
Discussion: https://postgr.es/m/uvrtrknj4kdytuboidbhwclo4gxhswwcpgadptsjvjqcluzmah%40brqs62irg4dt
Discussion: https://postgr.es/m/20210223100344.llw5an2aklengrmn@alap3.anarazel.de
Discussion: https://postgr.es/m/stj36ea6yyhoxtqkhpieia2z4krnam7qyetc57rfezgk4zgapf@gcnactj4z56m
---
src/include/storage/aio.h | 300 +++++
src/include/storage/aio_internal.h | 351 ++++++
src/include/storage/aio_subsys.h | 5 +
src/include/storage/aio_types.h | 115 ++
src/backend/access/transam/xact.c | 12 +
src/backend/postmaster/autovacuum.c | 2 +
src/backend/postmaster/bgwriter.c | 2 +
src/backend/postmaster/checkpointer.c | 2 +
src/backend/postmaster/pgarch.c | 2 +
src/backend/postmaster/walsummarizer.c | 2 +
src/backend/postmaster/walwriter.c | 2 +
src/backend/replication/walsender.c | 2 +
src/backend/storage/aio/Makefile | 4 +
src/backend/storage/aio/aio.c | 1117 ++++++++++++++++-
src/backend/storage/aio/aio_callback.c | 295 +++++
src/backend/storage/aio/aio_init.c | 186 +++
src/backend/storage/aio/aio_io.c | 180 +++
src/backend/storage/aio/aio_target.c | 106 ++
src/backend/storage/aio/meson.build | 4 +
src/backend/storage/aio/method_sync.c | 47 +
.../utils/activity/wait_event_names.txt | 1 +
src/tools/pgindent/typedefs.list | 21 +
22 files changed, 2754 insertions(+), 4 deletions(-)
create mode 100644 src/include/storage/aio_internal.h
create mode 100644 src/include/storage/aio_types.h
create mode 100644 src/backend/storage/aio/aio_callback.c
create mode 100644 src/backend/storage/aio/aio_io.c
create mode 100644 src/backend/storage/aio/aio_target.c
create mode 100644 src/backend/storage/aio/method_sync.c
diff --git a/src/include/storage/aio.h b/src/include/storage/aio.h
index e79d5343038..747ad83d037 100644
--- a/src/include/storage/aio.h
+++ b/src/include/storage/aio.h
@@ -14,6 +14,9 @@
#ifndef AIO_H
#define AIO_H
+#include "storage/aio_types.h"
+#include "storage/procnumber.h"
+
/* Enum for io_method GUC. */
@@ -26,9 +29,306 @@ typedef enum IoMethod
#define DEFAULT_IO_METHOD IOMETHOD_SYNC
+/*
+ * Flags for an IO that can be set with pgaio_io_set_flag().
+ */
+typedef enum PgAioHandleFlags
+{
+ /*
+ * The IO references backend local memory.
+ *
+ * This needs to be set on an IO whenever the IO references process-local
+ * memory. Some IO methods do not support executing IO that references
+ * process local memory and thus need to fall back to executing IO
+ * synchronously for IOs with this flag set.
+ *
+ * Required for correctness.
+ */
+ PGAIO_HF_REFERENCES_LOCAL = 1 << 1,
+
+ /*
+ * Hint that IO will be executed synchronously.
+ *
+ * This can make it a bit cheaper to execute synchronous IO via the AIO
+ * interface, to avoid needing an AIO and non-AIO version of code.
+ *
+ * Advantageous to set, if applicable, but not required for correctness.
+ */
+ PGAIO_HF_SYNCHRONOUS = 1 << 0,
+
+ /*
+ * IO is using buffered IO, used to control heuristic in some IO methods.
+ *
+ * Advantageous to set, if applicable, but not required for correctness.
+ */
+ PGAIO_HF_BUFFERED = 1 << 2,
+} PgAioHandleFlags;
+
+/*
+ * The IO operations supported by the AIO subsystem.
+ *
+ * This could be in aio_internal.h, as it is not publically referenced, but
+ * PgAioOpData currently *does* need to be public, therefore keeping this
+ * public seems to make sense.
+ */
+typedef enum PgAioOp
+{
+ /* intentionally the zero value, to help catch zeroed memory etc */
+ PGAIO_OP_INVALID = 0,
+
+ PGAIO_OP_READV,
+ PGAIO_OP_WRITEV,
+
+ /**
+ * In the near term we'll need at least:
+ * - fsync / fdatasync
+ * - flush_range
+ *
+ * Eventually we'll additionally want at least:
+ * - send
+ * - recv
+ * - accept
+ **/
+} PgAioOp;
+
+#define PGAIO_OP_COUNT (PGAIO_OP_WRITEV + 1)
+
+
+/*
+ * On what is IO being performed.
+ *
+ * PgAioTargetID specific behaviour should be implemented in
+ * aio_target.c.
+ */
+typedef enum PgAioTargetID
+{
+ /* intentionally the zero value, to help catch zeroed memory etc */
+ PGAIO_TID_INVALID = 0,
+} PgAioTargetID;
+
+#define PGAIO_TID_COUNT (PGAIO_TID_INVALID + 1)
+
+
+/*
+ * Data necessary for support IO operations (see PgAioOp).
+ *
+ * NB: Note that the FDs in here may *not* be relied upon for re-issuing
+ * requests (e.g. for partial reads/writes) - the FD might be from another
+ * process, or closed since. That's not a problem for IOs waiting to be issued
+ * only because the queue is flushed when closing an FD.
+ */
+typedef union
+{
+ struct
+ {
+ int fd;
+ uint16 iov_length;
+ uint64 offset;
+ } read;
+
+ struct
+ {
+ int fd;
+ uint16 iov_length;
+ uint64 offset;
+ } write;
+} PgAioOpData;
+
+
+/*
+ * Information the object that IO is executed on. Mostly callbacks that
+ * operate on PgAioTargetData.
+ */
+typedef struct PgAioTargetInfo
+{
+ void (*reopen) (PgAioHandle *ioh);
+
+ char *(*describe_identity) (const PgAioTargetData *sd);
+
+ const char *name;
+} PgAioTargetInfo;
+
+
+/*
+ * IDs for callbacks that can be registered on an IO.
+ *
+ * Callbacks are identified by an ID rather than a function pointer. There are
+ * two main reasons:
+ *
+ * 1) Memory within PgAioHandle is precious, due to the number of PgAioHandle
+ * structs in pre-allocated shared memory.
+ *
+ * 2) Due to EXEC_BACKEND function pointers are not necessarily stable between
+ * different backends, therefore function pointers cannot directly be in
+ * shared memory.
+ *
+ * Without 2), we could fairly easily allow to add new callbacks, by filling a
+ * ID->pointer mapping table on demand. In the presence of 2 that's still
+ * doable, but harder, because every process has to re-register the pointers
+ * so that a local ID->"backend local pointer" mapping can be maintained.
+ */
+typedef enum PgAioHandleCallbackID
+{
+ PGAIO_HCB_INVALID,
+} PgAioHandleCallbackID;
+
+
+typedef void (*PgAioHandleCallbackStage) (PgAioHandle *ioh, uint8 cb_flags);
+typedef PgAioResult (*PgAioHandleCallbackComplete) (PgAioHandle *ioh, PgAioResult prior_result, uint8 cb_flags);
+typedef void (*PgAioHandleCallbackReport) (PgAioResult result, const PgAioTargetData *target_data, int elevel);
+
+typedef struct PgAioHandleCallbacks
+{
+ /*
+ * Prepare resources affected by the IO for execution. This could e.g.
+ * include moving ownership of buffer pins to the AIO subsystem.
+ */
+ PgAioHandleCallbackStage stage;
+
+ /*
+ * Update the state of resources affected by the IO to reflect completion
+ * of the IO. This could e.g. include updating shared buffer state to
+ * signal the IO has finished.
+ *
+ * The _shared suffix indicates that this is executed by the backend that
+ * completed the IO, which may or may not be the backend that issued the
+ * IO. Obviously the callback thus can only modify resources in shared
+ * memory.
+ *
+ * The latest registered callback is called first. This allows
+ * higher-level code to register callbacks that can rely on callbacks
+ * registered by lower-level code to already have been executed.
+ *
+ * NB: This is called in a critical section. Errors can be signalled by
+ * the callback's return value, it's the responsibility of the IO's issuer
+ * to react appropriately.
+ */
+ PgAioHandleCallbackComplete complete_shared;
+
+ /*
+ * Like complete_shared, except called in the issuing backend.
+ *
+ * This variant of the completion callback is useful when backend-local
+ * state has to be updated to reflect the IO's completion. E.g. a
+ * temporary buffer's BufferDesc isn't accessible in complete_shared.
+ *
+ * Local callbacks are only called after complete_shared for all
+ * registered callbacks has been called.
+ */
+ PgAioHandleCallbackComplete complete_local;
+
+ /*
+ * Report the result of an IO operation. This is e.g. used to raise an
+ * error after an IO failed at the appropriate time (i.e. not when the IO
+ * failed, but under control of the code that issued the IO).
+ */
+ PgAioHandleCallbackReport report;
+} PgAioHandleCallbacks;
+
+
+
+/*
+ * How many callbacks can be registered for one IO handle. Currently we only
+ * need two, but it's not hard to imagine needing a few more.
+ */
+#define PGAIO_HANDLE_MAX_CALLBACKS 4
+
+
+
+/* AIO API */
+
+
+/* --------------------------------------------------------------------------------
+ * IO Handles
+ * --------------------------------------------------------------------------------
+ */
+
+/* functions in aio.c */
+struct ResourceOwnerData;
+extern PgAioHandle *pgaio_io_acquire(struct ResourceOwnerData *resowner, PgAioReturn *ret);
+extern PgAioHandle *pgaio_io_acquire_nb(struct ResourceOwnerData *resowner, PgAioReturn *ret);
+
+extern void pgaio_io_release(PgAioHandle *ioh);
struct dlist_node;
extern void pgaio_io_release_resowner(struct dlist_node *ioh_node, bool on_error);
+extern void pgaio_io_set_flag(PgAioHandle *ioh, PgAioHandleFlags flag);
+
+extern int pgaio_io_get_id(PgAioHandle *ioh);
+extern ProcNumber pgaio_io_get_owner(PgAioHandle *ioh);
+
+extern void pgaio_io_get_wref(PgAioHandle *ioh, PgAioWaitRef *iow);
+
+/* functions in aio_io.c */
+struct iovec;
+extern int pgaio_io_get_iovec(PgAioHandle *ioh, struct iovec **iov);
+
+extern PgAioOpData *pgaio_io_get_op_data(PgAioHandle *ioh);
+
+extern void pgaio_io_prep_readv(PgAioHandle *ioh,
+ int fd, int iovcnt, uint64 offset);
+extern void pgaio_io_prep_writev(PgAioHandle *ioh,
+ int fd, int iovcnt, uint64 offset);
+
+/* functions in aio_target.c */
+extern void pgaio_io_set_target(PgAioHandle *ioh, PgAioTargetID targetid);
+extern bool pgaio_io_has_target(PgAioHandle *ioh);
+extern PgAioTargetData *pgaio_io_get_target_data(PgAioHandle *ioh);
+extern char *pgaio_io_get_target_description(PgAioHandle *ioh);
+
+/* functions in aio_callback.c */
+extern void pgaio_io_register_callbacks(PgAioHandle *ioh, PgAioHandleCallbackID cb_id,
+ uint8 cb_data);
+extern void pgaio_io_set_handle_data_64(PgAioHandle *ioh, uint64 *data, uint8 len);
+extern void pgaio_io_set_handle_data_32(PgAioHandle *ioh, uint32 *data, uint8 len);
+extern uint64 *pgaio_io_get_handle_data(PgAioHandle *ioh, uint8 *len);
+
+
+
+/* --------------------------------------------------------------------------------
+ * IO Wait References
+ * --------------------------------------------------------------------------------
+ */
+
+extern void pgaio_wref_clear(PgAioWaitRef *iow);
+extern bool pgaio_wref_valid(PgAioWaitRef *iow);
+extern int pgaio_wref_get_id(PgAioWaitRef *iow);
+
+extern void pgaio_wref_wait(PgAioWaitRef *iow);
+extern bool pgaio_wref_check_done(PgAioWaitRef *iow);
+
+
+
+/* --------------------------------------------------------------------------------
+ * IO Result
+ * --------------------------------------------------------------------------------
+ */
+
+extern void pgaio_result_report(PgAioResult result, const PgAioTargetData *target_data,
+ int elevel);
+
+
+
+/* --------------------------------------------------------------------------------
+ * Actions on multiple IOs.
+ * --------------------------------------------------------------------------------
+ */
+
+extern void pgaio_enter_batchmode(void);
+extern void pgaio_exit_batchmode(void);
+extern void pgaio_submit_staged(void);
+extern bool pgaio_have_staged(void);
+
+
+
+/* --------------------------------------------------------------------------------
+ * Other
+ * --------------------------------------------------------------------------------
+ */
+
+extern void pgaio_closing_fd(int fd);
+
+
/* GUCs */
extern PGDLLIMPORT int io_method;
diff --git a/src/include/storage/aio_internal.h b/src/include/storage/aio_internal.h
new file mode 100644
index 00000000000..9d9cda80ccb
--- /dev/null
+++ b/src/include/storage/aio_internal.h
@@ -0,0 +1,351 @@
+/*-------------------------------------------------------------------------
+ *
+ * aio_internal.h
+ * AIO related declarations that shoul only be used by the AIO subsystem
+ * internally.
+ *
+ *
+ * Portions Copyright (c) 1996-2025, PostgreSQL Global Development Group
+ * Portions Copyright (c) 1994, Regents of the University of California
+ *
+ * src/include/storage/aio_internal.h
+ *
+ *-------------------------------------------------------------------------
+ */
+#ifndef AIO_INTERNAL_H
+#define AIO_INTERNAL_H
+
+
+#include "lib/ilist.h"
+#include "port/pg_iovec.h"
+#include "storage/aio.h"
+#include "storage/condition_variable.h"
+
+
+/*
+ * The maximum number of IOs that can be batch submitted at once.
+ */
+#define PGAIO_SUBMIT_BATCH_SIZE 32
+
+
+
+typedef enum PgAioHandleState
+{
+ /* not in use */
+ PGAIO_HS_IDLE = 0,
+
+ /* returned by pgaio_io_acquire() */
+ PGAIO_HS_HANDED_OUT,
+
+ /* pgaio_io_prep_*() has been called, but IO hasn't been submitted yet */
+ PGAIO_HS_DEFINED,
+
+ /* target's stage() callback has been called, ready to be submitted */
+ PGAIO_HS_STAGED,
+
+ /* IO has been submitted and is being executed */
+ PGAIO_HS_SUBMITTED,
+
+ /* IO finished, but result has not yet been processed */
+ PGAIO_HS_COMPLETED_IO,
+
+ /* IO completed, shared completion has been called */
+ PGAIO_HS_COMPLETED_SHARED,
+
+ /* IO completed, local completion has been called */
+ PGAIO_HS_COMPLETED_LOCAL,
+} PgAioHandleState;
+
+
+struct ResourceOwnerData;
+
+/* typedef is in public header */
+struct PgAioHandle
+{
+ /* all state updates should go through pgaio_io_update_state() */
+ PgAioHandleState state:8;
+
+ /* what are we operating on */
+ PgAioTargetID target:8;
+
+ /* which IO operation */
+ PgAioOp op:8;
+
+ /* bitfield of PgAioHandleFlags */
+ uint8 flags;
+
+ uint8 num_callbacks;
+
+ /* using the proper type here would use more space */
+ uint8 callbacks[PGAIO_HANDLE_MAX_CALLBACKS];
+
+ /* data forwarded to each callback */
+ uint8 callbacks_data[PGAIO_HANDLE_MAX_CALLBACKS];
+
+ /*
+ * Length of data associated with handle using
+ * pgaio_io_set_handle_data_*().
+ */
+ uint8 handle_data_len;
+
+ /* XXX: could be optimized out with some pointer math */
+ int32 owner_procno;
+
+ /* raw result of the IO operation */
+ int32 result;
+
+ /*
+ * Index into PgAioCtl->iovecs and PgAioCtl->handle_data.
+ *
+ * At the moment there's no need to differentiate between the two, but
+ * that won't necessarily stay that way.
+ */
+ uint32 iovec_off;
+
+ /**
+ * In which list the handle is registered, depends on the state:
+ * - IDLE, in per-backend list
+ * - HANDED_OUT - not in a list
+ * - DEFINED - in per-backend staged list
+ * - STAGED - in per-backend staged list
+ * - SUBMITTED - in issuer's in_flight list
+ * - COMPLETED_IO - in issuer's in_flight list
+ * - COMPLETED_SHARED - in issuer's in_flight list
+ **/
+ dlist_node node;
+
+ struct ResourceOwnerData *resowner;
+ dlist_node resowner_node;
+
+ /* incremented every time the IO handle is reused */
+ uint64 generation;
+
+ ConditionVariable cv;
+
+ /* result of shared callback, passed to issuer callback */
+ PgAioResult distilled_result;
+
+ PgAioReturn *report_return;
+
+ PgAioOpData op_data;
+
+ /*
+ * Data necessary to identify the object undergoing IO to higher-level
+ * code. Needs to be sufficient to allow another backend to reopen the
+ * file.
+ */
+ PgAioTargetData target_data;
+};
+
+
+typedef struct PgAioBackend
+{
+ /* index into PgAioCtl->io_handles */
+ uint32 io_handle_off;
+
+ /* IO Handles that currently are not used */
+ dclist_head idle_ios;
+
+ /*
+ * Only one IO may be returned by pgaio_io_acquire()/pgaio_io_acquire()
+ * without having been either defined (by actually associating it with IO)
+ * or by released (with pgaio_io_release()). This restriction is necessary
+ * to guarantee that we always can acquire an IO. ->handed_out_io is used
+ * to enforce that rule.
+ */
+ PgAioHandle *handed_out_io;
+
+ /* Are we currently in batchmode? See pgaio_enter_batchmode(). */
+ bool in_batchmode;
+
+ /*
+ * IOs that are defined, but not yet submitted.
+ */
+ uint16 num_staged_ios;
+ PgAioHandle *staged_ios[PGAIO_SUBMIT_BATCH_SIZE];
+
+ /*
+ * List of in-flight IOs. Also contains IOs that aren't strict speaking
+ * in-flight anymore, but have been waited-for and completed by another
+ * backend. Once this backend sees such an IO it'll be reclaimed.
+ *
+ * The list is ordered by submission time, with more recently submitted
+ * IOs being appended at the end.
+ */
+ dclist_head in_flight_ios;
+} PgAioBackend;
+
+
+typedef struct PgAioCtl
+{
+ int backend_state_count;
+ PgAioBackend *backend_state;
+
+ /*
+ * Array of iovec structs. Each iovec is owned by a specific backend. The
+ * allocation is in PgAioCtl to allow the maximum number of iovecs for
+ * individual IOs to be configurable with PGC_POSTMASTER GUC.
+ */
+ uint64 iovec_count;
+ struct iovec *iovecs;
+
+ /*
+ * For, e.g., an IO covering multiple buffers in shared / temp buffers, we
+ * need to get Buffer IDs during completion to be able to change the
+ * BufferDesc state accordingly. This space can be used to store e.g.
+ * Buffer IDs. Note that the actual iovec might be shorter than this,
+ * because we combine neighboring pages into one larger iovec entry.
+ */
+ uint64 *handle_data;
+
+ uint64 io_handle_count;
+ PgAioHandle *io_handles;
+} PgAioCtl;
+
+
+
+/*
+ * Callbacks used to implement an IO method.
+ */
+typedef struct IoMethodOps
+{
+ /* global initialization */
+
+ /*
+ * Amount of additional shared memory to reserve for the io_method. Called
+ * just like a normal ipci.c style *Size() function. Optional.
+ */
+ size_t (*shmem_size) (void);
+
+ /*
+ * Initialize shared memory. First time is true if AIO's shared memory was
+ * just initialized, false otherwise. Optional.
+ */
+ void (*shmem_init) (bool first_time);
+
+ /*
+ * Per-backend initialization. Optional.
+ */
+ void (*init_backend) (void);
+
+
+ /* handling of IOs */
+
+ /* optional */
+ bool (*needs_synchronous_execution) (PgAioHandle *ioh);
+
+ /*
+ * Start executing passed in IOs.
+ *
+ * Will not be called if ->needs_synchronous_execution() returned true.
+ *
+ * num_staged_ios is <= PGAIO_SUBMIT_BATCH_SIZE.
+ *
+ */
+ int (*submit) (uint16 num_staged_ios, PgAioHandle **staged_ios);
+
+ /*
+ * Wait for the IO to complete. Optional.
+ *
+ * If not provided, it needs to be guaranteed that the IO method calls
+ * pgaio_io_process_completion() without further interaction by the
+ * issuing backend.
+ */
+ void (*wait_one) (PgAioHandle *ioh,
+ uint64 ref_generation);
+} IoMethodOps;
+
+
+/* aio.c */
+extern bool pgaio_io_was_recycled(PgAioHandle *ioh, uint64 ref_generation, PgAioHandleState *state);
+extern void pgaio_io_stage(PgAioHandle *ioh, PgAioOp op);
+extern void pgaio_io_process_completion(PgAioHandle *ioh, int result);
+extern void pgaio_io_prepare_submit(PgAioHandle *ioh);
+extern bool pgaio_io_needs_synchronous_execution(PgAioHandle *ioh);
+extern const char *pgaio_io_get_state_name(PgAioHandle *ioh);
+const char *pgaio_result_status_string(PgAioResultStatus rs);
+extern void pgaio_shutdown(int code, Datum arg);
+
+/* aio_callback.c */
+extern void pgaio_io_call_stage(PgAioHandle *ioh);
+extern void pgaio_io_call_complete_shared(PgAioHandle *ioh);
+extern void pgaio_io_call_complete_local(PgAioHandle *ioh);
+
+/* aio_io.c */
+extern void pgaio_io_perform_synchronously(PgAioHandle *ioh);
+extern const char *pgaio_io_get_op_name(PgAioHandle *ioh);
+
+/* aio_target.c */
+extern bool pgaio_io_can_reopen(PgAioHandle *ioh);
+extern void pgaio_io_reopen(PgAioHandle *ioh);
+extern const char *pgaio_io_get_target_name(PgAioHandle *ioh);
+
+
+/*
+ * The AIO subsystem has fairly verbose debug logging support. This can be
+ * enabled/disabled at buildtime. The reason for this is that
+ * a) the verbosity can make debugging things on higher levels hard
+ * b) even if logging can be skipped due to elevel checks, it still causes a
+ * measurable slowdown
+ */
+#define PGAIO_VERBOSE 1
+
+/*
+ * Simple ereport() wrapper that only logs if PGAIO_VERBOSE is defined.
+ *
+ * This intentionally still compiles the code, guarded by a constant if (0),
+ * if verbose logging is disabled, to make it less likely that debug logging
+ * is silently broken.
+ *
+ * The current definition requires passing at least one argument.
+ */
+#define pgaio_debug(elevel, msg, ...) \
+ do { \
+ if (PGAIO_VERBOSE) \
+ ereport(elevel, \
+ errhidestmt(true), errhidecontext(true), \
+ errmsg_internal(msg, \
+ __VA_ARGS__)); \
+ } while(0)
+
+/*
+ * Simple ereport() wrapper. Note that the definition requires passing at
+ * least one argument.
+ */
+#define pgaio_debug_io(elevel, ioh, msg, ...) \
+ pgaio_debug(elevel, "io %-10d|op %-5s|target %-4s|state %-16s: " msg, \
+ pgaio_io_get_id(ioh), \
+ pgaio_io_get_op_name(ioh), \
+ pgaio_io_get_target_name(ioh), \
+ pgaio_io_get_state_name(ioh), \
+ __VA_ARGS__)
+
+
+#ifdef USE_INJECTION_POINTS
+
+extern void pgaio_io_call_inj(PgAioHandle *ioh, const char *injection_point);
+
+/* just for use in tests, from within injection points */
+extern PgAioHandle *pgaio_inj_io_get(void);
+
+#else
+
+#define pgaio_io_call_inj(ioh, injection_point) (void) 0
+/*
+ * no fallback for pgaio_inj_io_get, all code using injection points better be
+ * guarded by USE_INJECTION_POINTS.
+ */
+
+#endif
+
+
+/* Declarations for the tables of function pointers exposed by each IO method. */
+extern PGDLLIMPORT const IoMethodOps pgaio_sync_ops;
+
+extern PGDLLIMPORT const IoMethodOps *pgaio_method_ops;
+extern PGDLLIMPORT PgAioCtl *pgaio_ctl;
+extern PGDLLIMPORT PgAioBackend *pgaio_my_backend;
+
+
+
+#endif /* AIO_INTERNAL_H */
diff --git a/src/include/storage/aio_subsys.h b/src/include/storage/aio_subsys.h
index 0b8aaaf00d3..e4faf692a38 100644
--- a/src/include/storage/aio_subsys.h
+++ b/src/include/storage/aio_subsys.h
@@ -25,4 +25,9 @@ extern void AioShmemInit(void);
extern void pgaio_init_backend(void);
+
+/* aio.c */
+extern void pgaio_error_cleanup(void);
+extern void AtEOXact_Aio(bool is_commit);
+
#endif /* AIO_SUBSYS_H */
diff --git a/src/include/storage/aio_types.h b/src/include/storage/aio_types.h
new file mode 100644
index 00000000000..d2617139a25
--- /dev/null
+++ b/src/include/storage/aio_types.h
@@ -0,0 +1,115 @@
+/*-------------------------------------------------------------------------
+ *
+ * aio_types.h
+ * AIO related types that are useful to include separately, to reduce the
+ * "include burden".
+ *
+ *
+ * Portions Copyright (c) 1996-2025, PostgreSQL Global Development Group
+ * Portions Copyright (c) 1994, Regents of the University of California
+ *
+ * src/include/storage/aio_types.h
+ *
+ *-------------------------------------------------------------------------
+ */
+#ifndef AIO_TYPES_H
+#define AIO_TYPES_H
+
+#include "storage/block.h"
+#include "storage/relfilelocator.h"
+
+
+typedef struct PgAioHandle PgAioHandle;
+
+/*
+ * A reference to an IO that can be used to wait for the IO (using
+ * pgaio_wref_wait()) to complete.
+ *
+ * These can be passed across process boundaries.
+ */
+typedef struct PgAioWaitRef
+{
+ /* internal ID identifying the specific PgAioHandle */
+ uint32 aio_index;
+
+ /*
+ * IO handles are reused. To detect if a handle was reused, and thereby
+ * avoid unnecessarily waiting for a newer IO, each time the handle is
+ * reused a generation number is increased.
+ *
+ * To avoid requiring alignment sufficient for an int64, split the
+ * generation into two.
+ */
+ uint32 generation_upper;
+ uint32 generation_lower;
+} PgAioWaitRef;
+
+
+/*
+ * Information identifying what the IO is being performed on.
+ *
+ * This needs sufficient information to
+ *
+ * a) Reopen the file for the IO if the IO is executed in a context that
+ * cannot use the FD provided initially (e.g. because the IO is executed in
+ * a worker process).
+ *
+ * b) Describe the object the IO is performed on in log / error messages.
+ */
+typedef union PgAioTargetData
+{
+ /* just as an example placeholder for later */
+ struct
+ {
+ uint32 queue_id;
+ } wal;
+} PgAioTargetData;
+
+
+/*
+ * The status of an AIO operation.
+ */
+typedef enum PgAioResultStatus
+{
+ ARS_UNKNOWN, /* not yet completed / uninitialized */
+ ARS_OK,
+ ARS_PARTIAL, /* did not fully succeed, but no error */
+ ARS_ERROR,
+} PgAioResultStatus;
+
+
+/*
+ * Result of IO operation, visible only to the initiator of IO.
+ */
+typedef struct PgAioResult
+{
+ /*
+ * This is of type PgAioHandleCallbackID, but can't use a bitfield of an
+ * enum, because some compilers treat enums as signed.
+ */
+ uint32 id:8;
+
+ /* of type PgAioResultStatus, see above */
+ uint32 status:2;
+
+ /* meaning defined by callback->error */
+ uint32 error_data:22;
+
+ int32 result;
+} PgAioResult;
+
+
+/*
+ * Combination of PgAioResult with minimal metadata about the IO.
+ *
+ * Contains sufficient information to be able, in case the IO [partially]
+ * fails, to log/raise an error under control of the IO issuing code.
+ */
+typedef struct PgAioReturn
+{
+ PgAioResult result;
+ PgAioTargetData target_data;
+} PgAioReturn;
+
+
+#endif /* AIO_TYPES_H */
diff --git a/src/backend/access/transam/xact.c b/src/backend/access/transam/xact.c
index 1b4f21a88d3..b885513f765 100644
--- a/src/backend/access/transam/xact.c
+++ b/src/backend/access/transam/xact.c
@@ -51,6 +51,7 @@
#include "replication/origin.h"
#include "replication/snapbuild.h"
#include "replication/syncrep.h"
+#include "storage/aio_subsys.h"
#include "storage/condition_variable.h"
#include "storage/fd.h"
#include "storage/lmgr.h"
@@ -2411,6 +2412,8 @@ CommitTransaction(void)
RESOURCE_RELEASE_BEFORE_LOCKS,
true, true);
+ AtEOXact_Aio(true);
+
/* Check we've released all buffer pins */
AtEOXact_Buffers(true);
@@ -2716,6 +2719,8 @@ PrepareTransaction(void)
RESOURCE_RELEASE_BEFORE_LOCKS,
true, true);
+ AtEOXact_Aio(true);
+
/* Check we've released all buffer pins */
AtEOXact_Buffers(true);
@@ -2830,6 +2835,8 @@ AbortTransaction(void)
pgstat_report_wait_end();
pgstat_progress_end_command();
+ pgaio_error_cleanup();
+
/* Clean up buffer content locks, too */
UnlockBuffers();
@@ -2960,6 +2967,7 @@ AbortTransaction(void)
ResourceOwnerRelease(TopTransactionResourceOwner,
RESOURCE_RELEASE_BEFORE_LOCKS,
false, true);
+ AtEOXact_Aio(false);
AtEOXact_Buffers(false);
AtEOXact_RelationCache(false);
AtEOXact_TypeCache();
@@ -5232,6 +5240,9 @@ AbortSubTransaction(void)
pgstat_report_wait_end();
pgstat_progress_end_command();
+
+ pgaio_error_cleanup();
+
UnlockBuffers();
/* Reset WAL record construction state */
@@ -5326,6 +5337,7 @@ AbortSubTransaction(void)
RESOURCE_RELEASE_BEFORE_LOCKS,
false, false);
+ AtEOXact_Aio(false);
AtEOSubXact_RelationCache(false, s->subTransactionId,
s->parent->subTransactionId);
AtEOSubXact_TypeCache();
diff --git a/src/backend/postmaster/autovacuum.c b/src/backend/postmaster/autovacuum.c
index dfb8d068ecf..c8e7022ef0a 100644
--- a/src/backend/postmaster/autovacuum.c
+++ b/src/backend/postmaster/autovacuum.c
@@ -88,6 +88,7 @@
#include "postmaster/autovacuum.h"
#include "postmaster/interrupt.h"
#include "postmaster/postmaster.h"
+#include "storage/aio_subsys.h"
#include "storage/bufmgr.h"
#include "storage/ipc.h"
#include "storage/latch.h"
@@ -465,6 +466,7 @@ AutoVacLauncherMain(const void *startup_data, size_t startup_data_len)
*/
LWLockReleaseAll();
pgstat_report_wait_end();
+ pgaio_error_cleanup();
UnlockBuffers();
/* this is probably dead code, but let's be safe: */
if (AuxProcessResourceOwner)
diff --git a/src/backend/postmaster/bgwriter.c b/src/backend/postmaster/bgwriter.c
index 64524d1831b..3d350c611a5 100644
--- a/src/backend/postmaster/bgwriter.c
+++ b/src/backend/postmaster/bgwriter.c
@@ -38,6 +38,7 @@
#include "postmaster/auxprocess.h"
#include "postmaster/bgwriter.h"
#include "postmaster/interrupt.h"
+#include "storage/aio_subsys.h"
#include "storage/buf_internals.h"
#include "storage/bufmgr.h"
#include "storage/condition_variable.h"
@@ -168,6 +169,7 @@ BackgroundWriterMain(const void *startup_data, size_t startup_data_len)
*/
LWLockReleaseAll();
ConditionVariableCancelSleep();
+ pgaio_error_cleanup();
UnlockBuffers();
ReleaseAuxProcessResources(false);
AtEOXact_Buffers(false);
diff --git a/src/backend/postmaster/checkpointer.c b/src/backend/postmaster/checkpointer.c
index 7acbbd3e267..4dc24739fbd 100644
--- a/src/backend/postmaster/checkpointer.c
+++ b/src/backend/postmaster/checkpointer.c
@@ -49,6 +49,7 @@
#include "postmaster/bgwriter.h"
#include "postmaster/interrupt.h"
#include "replication/syncrep.h"
+#include "storage/aio_subsys.h"
#include "storage/bufmgr.h"
#include "storage/condition_variable.h"
#include "storage/fd.h"
@@ -276,6 +277,7 @@ CheckpointerMain(const void *startup_data, size_t startup_data_len)
LWLockReleaseAll();
ConditionVariableCancelSleep();
pgstat_report_wait_end();
+ pgaio_error_cleanup();
UnlockBuffers();
ReleaseAuxProcessResources(false);
AtEOXact_Buffers(false);
diff --git a/src/backend/postmaster/pgarch.c b/src/backend/postmaster/pgarch.c
index e6cd78679ce..c42479452ed 100644
--- a/src/backend/postmaster/pgarch.c
+++ b/src/backend/postmaster/pgarch.c
@@ -40,6 +40,7 @@
#include "postmaster/interrupt.h"
#include "postmaster/pgarch.h"
#include "storage/condition_variable.h"
+#include "storage/aio_subsys.h"
#include "storage/fd.h"
#include "storage/ipc.h"
#include "storage/latch.h"
@@ -568,6 +569,7 @@ pgarch_archiveXlog(char *xlog)
LWLockReleaseAll();
ConditionVariableCancelSleep();
pgstat_report_wait_end();
+ pgaio_error_cleanup();
ReleaseAuxProcessResources(false);
AtEOXact_Files(false);
AtEOXact_HashTables(false);
diff --git a/src/backend/postmaster/walsummarizer.c b/src/backend/postmaster/walsummarizer.c
index f4d61c1f3bb..bd36b956191 100644
--- a/src/backend/postmaster/walsummarizer.c
+++ b/src/backend/postmaster/walsummarizer.c
@@ -37,6 +37,7 @@
#include "postmaster/interrupt.h"
#include "postmaster/walsummarizer.h"
#include "replication/walreceiver.h"
+#include "storage/aio_subsys.h"
#include "storage/fd.h"
#include "storage/ipc.h"
#include "storage/latch.h"
@@ -288,6 +289,7 @@ WalSummarizerMain(const void *startup_data, size_t startup_data_len)
LWLockReleaseAll();
ConditionVariableCancelSleep();
pgstat_report_wait_end();
+ pgaio_error_cleanup();
ReleaseAuxProcessResources(false);
AtEOXact_Files(false);
AtEOXact_HashTables(false);
diff --git a/src/backend/postmaster/walwriter.c b/src/backend/postmaster/walwriter.c
index 15a71ad684d..a55db844419 100644
--- a/src/backend/postmaster/walwriter.c
+++ b/src/backend/postmaster/walwriter.c
@@ -51,6 +51,7 @@
#include "postmaster/auxprocess.h"
#include "postmaster/interrupt.h"
#include "postmaster/walwriter.h"
+#include "storage/aio_subsys.h"
#include "storage/bufmgr.h"
#include "storage/condition_variable.h"
#include "storage/fd.h"
@@ -164,6 +165,7 @@ WalWriterMain(const void *startup_data, size_t startup_data_len)
LWLockReleaseAll();
ConditionVariableCancelSleep();
pgstat_report_wait_end();
+ pgaio_error_cleanup();
UnlockBuffers();
ReleaseAuxProcessResources(false);
AtEOXact_Buffers(false);
diff --git a/src/backend/replication/walsender.c b/src/backend/replication/walsender.c
index 446d10c1a7d..69097125606 100644
--- a/src/backend/replication/walsender.c
+++ b/src/backend/replication/walsender.c
@@ -79,6 +79,7 @@
#include "replication/walsender.h"
#include "replication/walsender_private.h"
#include "storage/condition_variable.h"
+#include "storage/aio_subsys.h"
#include "storage/fd.h"
#include "storage/ipc.h"
#include "storage/pmsignal.h"
@@ -327,6 +328,7 @@ WalSndErrorCleanup(void)
LWLockReleaseAll();
ConditionVariableCancelSleep();
pgstat_report_wait_end();
+ pgaio_error_cleanup();
if (xlogreader != NULL && xlogreader->seg.ws_file >= 0)
wal_segment_close(xlogreader);
diff --git a/src/backend/storage/aio/Makefile b/src/backend/storage/aio/Makefile
index eaeaeeee8e3..89f821ea7e1 100644
--- a/src/backend/storage/aio/Makefile
+++ b/src/backend/storage/aio/Makefile
@@ -10,7 +10,11 @@ include $(top_builddir)/src/Makefile.global
OBJS = \
aio.o \
+ aio_callback.o \
aio_init.o \
+ aio_io.o \
+ aio_target.o \
+ method_sync.o \
read_stream.o
include $(top_srcdir)/src/backend/common.mk
diff --git a/src/backend/storage/aio/aio.c b/src/backend/storage/aio/aio.c
index 8eb6a5f1292..b8cfe012c4d 100644
--- a/src/backend/storage/aio/aio.c
+++ b/src/backend/storage/aio/aio.c
@@ -3,6 +3,28 @@
* aio.c
* AIO - Core Logic
*
+ * For documentation about how AIO works on a higher level, including a
+ * schematic example, see README.md.
+ *
+ *
+ * AIO is a complicated subsystem. To keep things navigable it is split across
+ * a number of files:
+ *
+ * - method_*.c - different ways of executing AIO (e.g. worker process)
+ *
+ * - aio_target.c - IO on different kinds of targets
+ *
+ * - aio_io.c - method-independent code for specific IO ops (e.g. readv)
+ *
+ * - aio_callback.c - callbacks at IO operation lifecycle events
+ *
+ * - aio_init.c - per-server and per-backend initialization
+ *
+ * - aio.c - all other topics
+ *
+ * - read_stream.c - helper for reading buffered relation data
+ *
+ *
* Portions Copyright (c) 1996-2025, PostgreSQL Global Development Group
* Portions Copyright (c) 1994, Regents of the University of California
*
@@ -14,10 +36,28 @@
#include "postgres.h"
-#include "lib/ilist.h"
+#include "miscadmin.h"
+#include "port/atomics.h"
#include "storage/aio.h"
+#include "storage/aio_internal.h"
+#include "storage/aio_subsys.h"
#include "utils/guc.h"
#include "utils/guc_hooks.h"
+#include "utils/resowner.h"
+#include "utils/wait_event_types.h"
+
+#ifdef USE_INJECTION_POINTS
+#include "utils/injection_point.h"
+#endif
+
+
+static inline void pgaio_io_update_state(PgAioHandle *ioh, PgAioHandleState new_state);
+static void pgaio_io_reclaim(PgAioHandle *ioh);
+static void pgaio_io_resowner_register(PgAioHandle *ioh);
+static void pgaio_io_wait_for_free(void);
+static PgAioHandle *pgaio_io_from_wref(PgAioWaitRef *iow, uint64 *ref_generation);
+static const char *pgaio_io_state_get_name(PgAioHandleState s);
+static void pgaio_io_wait(PgAioHandle *ioh, uint64 ref_generation);
/* Options for io_method. */
@@ -30,11 +70,1050 @@ const struct config_enum_entry io_method_options[] = {
int io_method = DEFAULT_IO_METHOD;
int io_max_concurrency = -1;
+/* global control for AIO */
+PgAioCtl *pgaio_ctl;
+/* current backend's per-backend state */
+PgAioBackend *pgaio_my_backend;
+
+
+static const IoMethodOps *const pgaio_method_ops_table[] = {
+ [IOMETHOD_SYNC] = &pgaio_sync_ops,
+};
+
+/* callbacks for the configured io_method, set by assign_io_method */
+const IoMethodOps *pgaio_method_ops;
+
+
+/*
+ * Currently there's no infrastructure to pass arguments to injection points,
+ * so we instead set this up for the duration of the injection point
+ * invocation.
+ */
+#ifdef USE_INJECTION_POINTS
+static PgAioHandle *pgaio_inj_cur_handle;
+#endif
+
+
+
+/* --------------------------------------------------------------------------------
+ * Public Functions related to PgAioHandle
+ * --------------------------------------------------------------------------------
+ */
+
+/*
+ * Acquire an AioHandle, waiting for IO completion if necessary.
+ *
+ * Each backend can only have one AIO handle that that has been "handed out"
+ * to code, but not yet submitted or released. This restriction is necessary
+ * to ensure that it is possible for code to wait for an unused handle by
+ * waiting for in-flight IO to complete. There is a limited number of handles
+ * in each backend, if multiple handles could be handed out without being
+ * submitted, waiting for all in-flight IO to complete would not guarantee
+ * that handles free up.
+ *
+ * It is cheap to acquire an IO handle, unless all handles are in use. In that
+ * case this function waits for the oldest IO to complete. In case that is not
+ * desirable, see pgaio_io_acquire_nb().
+ *
+ * If a handle was acquired but then does not turn out to be needed,
+ * e.g. because pgaio_io_acquire() is called before starting an IO in a
+ * critical section, the handle needs to be released with pgaio_io_release().
+ *
+ *
+ * To react to the completion of the IO as soon as it is known to have
+ * completed, callbacks can be registered with pgaio_io_register_callbacks().
+ *
+ * To actually execute IO using the returned handle, the pgaio_io_prep_*()
+ * family of functions is used. In many cases the pgaio_io_prep_*() call will
+ * not be done directly by code that acquired the handle, but by lower level
+ * code that gets passed the handle. E.g. if code in bufmgr.c wants to perform
+ * AIO, it typically will pass the handle to smgr., which will pass it on to
+ * md.c, on to fd.c, which then finally calls pgaio_io_prep_*(). This
+ * forwarding allows the various layers to react to the IO's completion by
+ * registering callbacks. These callbacks in turn can translate a lower
+ * layer's result into a result understandable by a higher layer.
+ *
+ * During pgaio_io_prep_*() the IO is staged (i.e. prepared for execution but
+ * not submitted to the kernel). Unless in batchmode
+ * (c.f. pgaio_enter_batchmode()) the IO will also have been submitted for
+ * execution. Note that, whether in batchmode or not, the IO might even
+ * complete before the functions return.
+ *
+ * After pgaio_io_prep_*() the AioHandle is "consumed" and may not be
+ * referenced by the IO issuing code. To e.g. wait for IO, references to the
+ * IO can be established with pgaio_io_get_wref() *before* pgaio_io_prep_*()
+ * is called. pgaio_wref_wait() can be used to wait for the IO to complete.
+ *
+ *
+ * To know if the IO [partially] succeeded or failed, a PgAioReturn * can be
+ * passed to pgaio_io_acquire(). Once the issuing backend has called
+ * pgaio_wref_wait(), the PgAioReturn contains information about whether the
+ * operation succeeded and details about the first failure, if any. The error
+ * can be raised / logged with pgaio_result_report().
+ *
+ * The lifetime of the memory pointed to be *ret needs to be at least as long
+ * as the passed in resowner. If the resowner releases resources before the IO
+ * completes (typically due to an error), the reference to *ret will be
+ * cleared. In case of resowner cleanup *ret will not be updated with the
+ * results of the IO operation.
+ */
+PgAioHandle *
+pgaio_io_acquire(struct ResourceOwnerData *resowner, PgAioReturn *ret)
+{
+ PgAioHandle *h;
+
+ while (true)
+ {
+ h = pgaio_io_acquire_nb(resowner, ret);
+
+ if (h != NULL)
+ return h;
+
+ /*
+ * Evidently all handles by this backend are in use. Just wait for
+ * some to complete.
+ */
+ pgaio_io_wait_for_free();
+ }
+}
+
+/*
+ * Acquire an AioHandle, returning NULL if no handles are free.
+ *
+ * See pgaio_io_acquire(). The only difference is that this function will return
+ * NULL if there are no idle handles, instead of blocking.
+ */
+PgAioHandle *
+pgaio_io_acquire_nb(struct ResourceOwnerData *resowner, PgAioReturn *ret)
+{
+ if (pgaio_my_backend->num_staged_ios >= PGAIO_SUBMIT_BATCH_SIZE)
+ {
+ Assert(pgaio_my_backend->num_staged_ios == PGAIO_SUBMIT_BATCH_SIZE);
+ pgaio_submit_staged();
+ }
+
+ if (pgaio_my_backend->handed_out_io)
+ {
+ ereport(ERROR,
+ errmsg("API violation: Only one IO can be handed out"));
+ }
+
+ if (!dclist_is_empty(&pgaio_my_backend->idle_ios))
+ {
+ dlist_node *ion = dclist_pop_head_node(&pgaio_my_backend->idle_ios);
+ PgAioHandle *ioh = dclist_container(PgAioHandle, node, ion);
+
+ Assert(ioh->state == PGAIO_HS_IDLE);
+ Assert(ioh->owner_procno == MyProcNumber);
+
+ pgaio_io_update_state(ioh, PGAIO_HS_HANDED_OUT);
+ pgaio_my_backend->handed_out_io = ioh;
+
+ if (resowner)
+ pgaio_io_resowner_register(ioh);
+
+ if (ret)
+ {
+ ioh->report_return = ret;
+ ret->result.status = ARS_UNKNOWN;
+ }
+
+ return ioh;
+ }
+
+ return NULL;
+}
+
+/*
+ * Release IO handle that turned out to not be required.
+ *
+ * See pgaio_io_acquire() for more details.
+ */
+void
+pgaio_io_release(PgAioHandle *ioh)
+{
+ if (ioh == pgaio_my_backend->handed_out_io)
+ {
+ Assert(ioh->state == PGAIO_HS_HANDED_OUT);
+ Assert(ioh->resowner);
+
+ pgaio_my_backend->handed_out_io = NULL;
+ pgaio_io_reclaim(ioh);
+ }
+ else
+ {
+ elog(ERROR, "release in unexpected state");
+ }
+}
+
+/*
+ * Release IO handle during resource owner cleanup.
+ */
+void
+pgaio_io_release_resowner(dlist_node *ioh_node, bool on_error)
+{
+ PgAioHandle *ioh = dlist_container(PgAioHandle, resowner_node, ioh_node);
+
+ Assert(ioh->resowner);
+
+ ResourceOwnerForgetAioHandle(ioh->resowner, &ioh->resowner_node);
+ ioh->resowner = NULL;
+
+ switch (ioh->state)
+ {
+ case PGAIO_HS_IDLE:
+ elog(ERROR, "unexpected");
+ break;
+ case PGAIO_HS_HANDED_OUT:
+ Assert(ioh == pgaio_my_backend->handed_out_io || pgaio_my_backend->handed_out_io == NULL);
+
+ if (ioh == pgaio_my_backend->handed_out_io)
+ {
+ pgaio_my_backend->handed_out_io = NULL;
+ if (!on_error)
+ elog(WARNING, "leaked AIO handle");
+ }
+
+ pgaio_io_reclaim(ioh);
+ break;
+ case PGAIO_HS_DEFINED:
+ case PGAIO_HS_STAGED:
+ if (!on_error)
+ elog(WARNING, "AIO handle was not submitted");
+ pgaio_submit_staged();
+ break;
+ case PGAIO_HS_SUBMITTED:
+ case PGAIO_HS_COMPLETED_IO:
+ case PGAIO_HS_COMPLETED_SHARED:
+ case PGAIO_HS_COMPLETED_LOCAL:
+ /* this is expected to happen */
+ break;
+ }
+
+ /*
+ * Need to unregister the reporting of the IO's result, the memory it's
+ * referencing likely has gone away.
+ */
+ if (ioh->report_return)
+ ioh->report_return = NULL;
+}
+
+/*
+ * Add a [set of] flags to the IO.
+ *
+ * Note that this combines flags with already set flags, rather than set flags
+ * to explicitly the passed in parameters. This is to allow multiple callsites
+ * to set flags.
+ */
+void
+pgaio_io_set_flag(PgAioHandle *ioh, PgAioHandleFlags flag)
+{
+ Assert(ioh->state == PGAIO_HS_HANDED_OUT);
+
+ ioh->flags |= flag;
+}
+
+int
+pgaio_io_get_id(PgAioHandle *ioh)
+{
+ Assert(ioh >= pgaio_ctl->io_handles &&
+ ioh < (pgaio_ctl->io_handles + pgaio_ctl->io_handle_count));
+ return ioh - pgaio_ctl->io_handles;
+}
+
+ProcNumber
+pgaio_io_get_owner(PgAioHandle *ioh)
+{
+ return ioh->owner_procno;
+}
+
+void
+pgaio_io_get_wref(PgAioHandle *ioh, PgAioWaitRef *iow)
+{
+ Assert(ioh->state == PGAIO_HS_HANDED_OUT ||
+ ioh->state == PGAIO_HS_DEFINED ||
+ ioh->state == PGAIO_HS_STAGED);
+ Assert(ioh->generation != 0);
+
+ iow->aio_index = ioh - pgaio_ctl->io_handles;
+ iow->generation_upper = (uint32) (ioh->generation >> 32);
+ iow->generation_lower = (uint32) ioh->generation;
+}
+
+
+
+/* --------------------------------------------------------------------------------
+ * Internal Functions related to PgAioHandle
+ * --------------------------------------------------------------------------------
+ */
+
+static inline void
+pgaio_io_update_state(PgAioHandle *ioh, PgAioHandleState new_state)
+{
+ pgaio_debug_io(DEBUG5, ioh,
+ "updating state to %s",
+ pgaio_io_state_get_name(new_state));
+
+ /*
+ * Ensure the changes signified by the new state are visible before the
+ * new state becomes visible.
+ */
+ pg_write_barrier();
+
+ ioh->state = new_state;
+}
+
+static void
+pgaio_io_resowner_register(PgAioHandle *ioh)
+{
+ Assert(!ioh->resowner);
+ Assert(CurrentResourceOwner);
+
+ ResourceOwnerRememberAioHandle(CurrentResourceOwner, &ioh->resowner_node);
+ ioh->resowner = CurrentResourceOwner;
+}
+
+/*
+ * Stage IO for execution and, if necessary, submit it immediately.
+ *
+ * Should only be called from pgaio_io_prep_*().
+ */
+void
+pgaio_io_stage(PgAioHandle *ioh, PgAioOp op)
+{
+ bool needs_synchronous;
+
+ Assert(ioh->state == PGAIO_HS_HANDED_OUT);
+ Assert(pgaio_io_has_target(ioh));
+
+ ioh->op = op;
+ ioh->result = 0;
+
+ pgaio_io_update_state(ioh, PGAIO_HS_DEFINED);
+
+ /* allow a new IO to be staged */
+ pgaio_my_backend->handed_out_io = NULL;
+
+ pgaio_io_call_stage(ioh);
+
+ pgaio_io_update_state(ioh, PGAIO_HS_STAGED);
+
+ /*
+ * Synchronous execution has to be executed, well, synchronously, so check
+ * that first.
+ */
+ needs_synchronous = pgaio_io_needs_synchronous_execution(ioh);
+
+ pgaio_debug_io(DEBUG3, ioh,
+ "prepared (synchronous: %d, in_batch: %d)",
+ needs_synchronous, pgaio_my_backend->in_batchmode);
+
+ if (!needs_synchronous)
+ {
+ pgaio_my_backend->staged_ios[pgaio_my_backend->num_staged_ios++] = ioh;
+ Assert(pgaio_my_backend->num_staged_ios <= PGAIO_SUBMIT_BATCH_SIZE);
+
+ /*
+ * Unless code explicitly opted into batching IOs, submit the IO
+ * immediately.
+ */
+ if (!pgaio_my_backend->in_batchmode)
+ pgaio_submit_staged();
+ }
+ else
+ {
+ pgaio_io_prepare_submit(ioh);
+ pgaio_io_perform_synchronously(ioh);
+ }
+}
+
+bool
+pgaio_io_needs_synchronous_execution(PgAioHandle *ioh)
+{
+ /*
+ * If the caller said to execute the IO synchronously, do so.
+ *
+ * XXX: We could optimize the logic when to execute synchronously by first
+ * checking if there are other IOs in flight and only synchronously
+ * executing if not. Unclear whether that'll be sufficiently common to be
+ * worth worrying about.
+ */
+ if (ioh->flags & PGAIO_HF_SYNCHRONOUS)
+ return true;
+
+ /* Check if the IO method requires synchronous execution of IO */
+ if (pgaio_method_ops->needs_synchronous_execution)
+ return pgaio_method_ops->needs_synchronous_execution(ioh);
+
+ return false;
+}
+
+/*
+ * Handle IO being processed by IO method.
+ *
+ * Should be called by IO methods / synchronous IO execution, just before the
+ * IO is performed.
+ */
+void
+pgaio_io_prepare_submit(PgAioHandle *ioh)
+{
+ pgaio_io_update_state(ioh, PGAIO_HS_SUBMITTED);
+
+ dclist_push_tail(&pgaio_my_backend->in_flight_ios, &ioh->node);
+}
+
+/*
+ * Handle IO getting completed by a method.
+ *
+ * Should be called by IO methods / synchronous IO execution, just after the
+ * IO has been performed.
+ *
+ * Expects to be called in a critical section. We expect IOs to be usable for
+ * WAL etc, which requires being able to execute completion callbacks in a
+ * critical section.
+ */
+void
+pgaio_io_process_completion(PgAioHandle *ioh, int result)
+{
+ Assert(ioh->state == PGAIO_HS_SUBMITTED);
+
+ Assert(CritSectionCount > 0);
+
+ ioh->result = result;
+
+ pgaio_io_update_state(ioh, PGAIO_HS_COMPLETED_IO);
+
+ pgaio_io_call_inj(ioh, "AIO_PROCESS_COMPLETION_BEFORE_SHARED");
+
+ pgaio_io_call_complete_shared(ioh);
+
+ pgaio_io_update_state(ioh, PGAIO_HS_COMPLETED_SHARED);
+
+ /* condition variable broadcast ensures state is visible before wakeup */
+ ConditionVariableBroadcast(&ioh->cv);
+
+ /* contains call to pgaio_io_call_complete_local() */
+ if (ioh->owner_procno == MyProcNumber)
+ pgaio_io_reclaim(ioh);
+}
+
+bool
+pgaio_io_was_recycled(PgAioHandle *ioh, uint64 ref_generation, PgAioHandleState *state)
+{
+ *state = ioh->state;
+ pg_read_barrier();
+
+ return ioh->generation != ref_generation;
+}
+
+/*
+ * Wait for IO to complete. External code should never use this, outside of
+ * the AIO subsystem waits are only allowed via pgaio_wref_wait().
+ */
+static void
+pgaio_io_wait(PgAioHandle *ioh, uint64 ref_generation)
+{
+ PgAioHandleState state;
+ bool am_owner;
+
+ am_owner = ioh->owner_procno == MyProcNumber;
+
+ if (pgaio_io_was_recycled(ioh, ref_generation, &state))
+ return;
+
+ if (am_owner)
+ {
+ if (state != PGAIO_HS_SUBMITTED
+ && state != PGAIO_HS_COMPLETED_IO
+ && state != PGAIO_HS_COMPLETED_SHARED
+ && state != PGAIO_HS_COMPLETED_LOCAL)
+ {
+ elog(PANIC, "waiting for own IO in wrong state: %d",
+ state);
+ }
+ }
+
+ while (true)
+ {
+ if (pgaio_io_was_recycled(ioh, ref_generation, &state))
+ return;
+
+ switch (state)
+ {
+ case PGAIO_HS_IDLE:
+ case PGAIO_HS_HANDED_OUT:
+ elog(ERROR, "IO in wrong state: %d", state);
+ break;
+
+ case PGAIO_HS_SUBMITTED:
+
+ /*
+ * If we need to wait via the IO method, do so now. Don't
+ * check via the IO method if the issuing backend is executing
+ * the IO synchronously.
+ */
+ if (pgaio_method_ops->wait_one && !(ioh->flags & PGAIO_HF_SYNCHRONOUS))
+ {
+ pgaio_method_ops->wait_one(ioh, ref_generation);
+ continue;
+ }
+ /* fallthrough */
+
+ /* waiting for owner to submit */
+ case PGAIO_HS_DEFINED:
+ case PGAIO_HS_STAGED:
+ /* waiting for reaper to complete */
+ /* fallthrough */
+ case PGAIO_HS_COMPLETED_IO:
+ /* shouldn't be able to hit this otherwise */
+ Assert(IsUnderPostmaster);
+ /* ensure we're going to get woken up */
+ ConditionVariablePrepareToSleep(&ioh->cv);
+
+ while (!pgaio_io_was_recycled(ioh, ref_generation, &state))
+ {
+ if (state == PGAIO_HS_COMPLETED_SHARED ||
+ state == PGAIO_HS_COMPLETED_LOCAL)
+ break;
+ ConditionVariableSleep(&ioh->cv, WAIT_EVENT_AIO_IO_COMPLETION);
+ }
+
+ ConditionVariableCancelSleep();
+ break;
+
+ case PGAIO_HS_COMPLETED_SHARED:
+ case PGAIO_HS_COMPLETED_LOCAL:
+ /* see above */
+ if (am_owner)
+ pgaio_io_reclaim(ioh);
+ return;
+ }
+ }
+}
+
+static void
+pgaio_io_reclaim(PgAioHandle *ioh)
+{
+ /* This is only ok if it's our IO */
+ Assert(ioh->owner_procno == MyProcNumber);
+
+ /*
+ * It's a bit ugly, but right now the easiest place to put the execution
+ * of shared completion callbacks is this function, as we need to execute
+ * locallbacks just before reclaiming at multiple callsites.
+ */
+ if (ioh->state == PGAIO_HS_COMPLETED_SHARED)
+ {
+ pgaio_io_call_complete_local(ioh);
+ pgaio_io_update_state(ioh, PGAIO_HS_COMPLETED_LOCAL);
+ }
+
+ pgaio_debug_io(DEBUG4, ioh,
+ "reclaiming: distilled_result: (status %s, id %u, error_data %d), raw_result: %d",
+ pgaio_result_status_string(ioh->distilled_result.status),
+ ioh->distilled_result.id,
+ ioh->distilled_result.error_data,
+ ioh->result);
+
+ /* if the IO has been defined, we might need to do more work */
+ if (ioh->state != PGAIO_HS_HANDED_OUT)
+ {
+ dclist_delete_from(&pgaio_my_backend->in_flight_ios, &ioh->node);
+
+ if (ioh->report_return)
+ {
+ ioh->report_return->result = ioh->distilled_result;
+ ioh->report_return->target_data = ioh->target_data;
+ }
+ }
+
+ if (ioh->resowner)
+ {
+ ResourceOwnerForgetAioHandle(ioh->resowner, &ioh->resowner_node);
+ ioh->resowner = NULL;
+ }
+
+ Assert(!ioh->resowner);
+
+ ioh->op = PGAIO_OP_INVALID;
+ ioh->target = PGAIO_TID_INVALID;
+ ioh->flags = 0;
+ ioh->num_callbacks = 0;
+ ioh->handle_data_len = 0;
+ ioh->report_return = NULL;
+ ioh->result = 0;
+ ioh->distilled_result.status = ARS_UNKNOWN;
+
+ /* XXX: the barrier is probably superfluous */
+ pg_write_barrier();
+ ioh->generation++;
+
+ pgaio_io_update_state(ioh, PGAIO_HS_IDLE);
+
+ /*
+ * We push the IO to the head of the idle IO list, that seems more cache
+ * efficient in cases where only a few IOs are used.
+ */
+ dclist_push_head(&pgaio_my_backend->idle_ios, &ioh->node);
+}
+
+static void
+pgaio_io_wait_for_free(void)
+{
+ int reclaimed = 0;
+
+ pgaio_debug(DEBUG2, "waiting for self with %d pending",
+ pgaio_my_backend->num_staged_ios);
+
+ /*
+ * First check if any of our IOs actually have completed - when using
+ * worker, that'll often be the case. We could do so as part of the loop
+ * below, but that'd potentially lead us to wait for some IO submitted
+ * before.
+ */
+ for (int i = 0; i < io_max_concurrency; i++)
+ {
+ PgAioHandle *ioh = &pgaio_ctl->io_handles[pgaio_my_backend->io_handle_off + i];
+
+ if (ioh->state == PGAIO_HS_COMPLETED_SHARED)
+ {
+ pgaio_io_reclaim(ioh);
+ reclaimed++;
+ }
+ }
+
+ if (reclaimed > 0)
+ return;
+
+ /*
+ * If we have any unsubmitted IOs, submit them now. We'll start waiting in
+ * a second, so it's better they're in flight. This also addresses the
+ * edge-case that all IOs are unsubmitted.
+ */
+ if (pgaio_my_backend->num_staged_ios > 0)
+ {
+ pgaio_submit_staged();
+ }
+
+ /*
+ * It's possible that we recognized there were free IOs while submitting.
+ */
+ if (dclist_count(&pgaio_my_backend->in_flight_ios) == 0)
+ {
+ elog(ERROR, "no free IOs despite no in-flight IOs");
+ }
+
+ /*
+ * Wait for the oldest in-flight IO to complete.
+ *
+ * XXX: Reusing the general IO wait is suboptimal, we don't need to wait
+ * for that specific IO to complete, we just need *any* IO to complete.
+ */
+ {
+ PgAioHandle *ioh = dclist_head_element(PgAioHandle, node, &pgaio_my_backend->in_flight_ios);
+
+ switch (ioh->state)
+ {
+ /* should not be in in-flight list */
+ case PGAIO_HS_IDLE:
+ case PGAIO_HS_DEFINED:
+ case PGAIO_HS_HANDED_OUT:
+ case PGAIO_HS_STAGED:
+ case PGAIO_HS_COMPLETED_LOCAL:
+ elog(ERROR, "shouldn't get here with io:%d in state %d",
+ pgaio_io_get_id(ioh), ioh->state);
+ break;
+
+ case PGAIO_HS_COMPLETED_IO:
+ case PGAIO_HS_SUBMITTED:
+ pgaio_debug_io(DEBUG2, ioh,
+ "waiting for free io with %d in flight",
+ dclist_count(&pgaio_my_backend->in_flight_ios));
+
+ /*
+ * In a more general case this would be racy, because the
+ * generation could increase after we read ioh->state above.
+ * But we are only looking at IOs by the current backend and
+ * the IO can only be recycled by this backend.
+ */
+ pgaio_io_wait(ioh, ioh->generation);
+ break;
+
+ case PGAIO_HS_COMPLETED_SHARED:
+ /* it's possible that another backend just finished this IO */
+ pgaio_io_reclaim(ioh);
+ break;
+ }
+
+ if (dclist_count(&pgaio_my_backend->idle_ios) == 0)
+ elog(PANIC, "no idle IOs after waiting");
+ return;
+ }
+}
+
+/*
+ * Internal - code outside of AIO should never need this and it'd be hard for
+ * such code to be safe.
+ */
+static PgAioHandle *
+pgaio_io_from_wref(PgAioWaitRef *iow, uint64 *ref_generation)
+{
+ PgAioHandle *ioh;
+
+ Assert(iow->aio_index < pgaio_ctl->io_handle_count);
+
+ ioh = &pgaio_ctl->io_handles[iow->aio_index];
+
+ *ref_generation = ((uint64) iow->generation_upper) << 32 |
+ iow->generation_lower;
+
+ Assert(*ref_generation != 0);
+
+ return ioh;
+}
+
+static const char *
+pgaio_io_state_get_name(PgAioHandleState s)
+{
+#define PGAIO_HS_TOSTR_CASE(sym) case PGAIO_HS_##sym: return #sym
+ switch (s)
+ {
+ PGAIO_HS_TOSTR_CASE(IDLE);
+ PGAIO_HS_TOSTR_CASE(HANDED_OUT);
+ PGAIO_HS_TOSTR_CASE(DEFINED);
+ PGAIO_HS_TOSTR_CASE(STAGED);
+ PGAIO_HS_TOSTR_CASE(SUBMITTED);
+ PGAIO_HS_TOSTR_CASE(COMPLETED_IO);
+ PGAIO_HS_TOSTR_CASE(COMPLETED_SHARED);
+ PGAIO_HS_TOSTR_CASE(COMPLETED_LOCAL);
+ }
+#undef PGAIO_HS_TOSTR_CASE
+
+ return NULL; /* silence compiler */
+}
+
+const char *
+pgaio_io_get_state_name(PgAioHandle *ioh)
+{
+ return pgaio_io_state_get_name(ioh->state);
+}
+
+const char *
+pgaio_result_status_string(PgAioResultStatus rs)
+{
+ switch (rs)
+ {
+ case ARS_UNKNOWN:
+ return "UNKNOWN";
+ case ARS_OK:
+ return "OK";
+ case ARS_PARTIAL:
+ return "PARTIAL";
+ case ARS_ERROR:
+ return "ERROR";
+ }
+
+ return NULL; /* silence compiler */
+}
+
+
+
+/* --------------------------------------------------------------------------------
+ * Functions primarily related to IO Wait References
+ * --------------------------------------------------------------------------------
+ */
+
+void
+pgaio_wref_clear(PgAioWaitRef *iow)
+{
+ iow->aio_index = PG_UINT32_MAX;
+}
+
+bool
+pgaio_wref_valid(PgAioWaitRef *iow)
+{
+ return iow->aio_index != PG_UINT32_MAX;
+}
+
+int
+pgaio_wref_get_id(PgAioWaitRef *iow)
+{
+ Assert(pgaio_wref_valid(iow));
+ return iow->aio_index;
+}
+
+/*
+ * Wait for the IO to have completed.
+ */
+void
+pgaio_wref_wait(PgAioWaitRef *iow)
+{
+ uint64 ref_generation;
+ PgAioHandle *ioh;
+
+ ioh = pgaio_io_from_wref(iow, &ref_generation);
+
+ pgaio_io_wait(ioh, ref_generation);
+}
+
+/*
+ * Check if the the referenced IO completed, without blocking.
+ */
+bool
+pgaio_wref_check_done(PgAioWaitRef *iow)
+{
+ uint64 ref_generation;
+ PgAioHandleState state;
+ bool am_owner;
+ PgAioHandle *ioh;
+
+ ioh = pgaio_io_from_wref(iow, &ref_generation);
+
+ if (pgaio_io_was_recycled(ioh, ref_generation, &state))
+ return true;
+
+ if (state == PGAIO_HS_IDLE)
+ return true;
+
+ am_owner = ioh->owner_procno == MyProcNumber;
+
+ if (state == PGAIO_HS_COMPLETED_SHARED ||
+ state == PGAIO_HS_COMPLETED_LOCAL)
+ {
+ if (am_owner)
+ pgaio_io_reclaim(ioh);
+ return true;
+ }
+
+ return false;
+}
+
+
+
+/* --------------------------------------------------------------------------------
+ * Actions on multiple IOs.
+ * --------------------------------------------------------------------------------
+ */
+
+/*
+ * Submit IOs in batches going forward.
+ *
+ * Submitting multiple IOs at once can be substantially faster than doing so
+ * one-by-one. At the same time submitting multiple IOs at once requires more
+ * care to avoid deadlocks.
+ *
+ * Consider backend A staging an IO for buffer 1 and then trying to start IO
+ * on buffer 2, while backend B does the inverse. If A submitted the IO before
+ * moving on to buffer 2, this works just fine, B will wait for the IO to
+ * complete. But if batching were used, each backend will wait for IO that has
+ * not yet been submitted to complete, i.e. forever.
+ *
+ * Batch submission mode needs to explicitly ended with
+ * pgaio_exit_batchmode(), but it is allowed to throw errors, in which case
+ * error recovery will end the batch.
+ *
+ * To avoid deadlocks, code needs to ensure that it will not wait for another
+ * backend while there is unsubmitted IO. E.g. by using conditional lock
+ * acquisition when acquiring buffer locks. To check if there currently are
+ * staged IOs, call pgaio_have_staged() and to submit all staged IOs call
+ * pgaio_submit_staged().
+ *
+ * It is not allowed to enter batchmode while already in batchmode, it's
+ * unlikely to ever be needed, as code needs to be explicitly aware of being
+ * called in batchmode, to avoid the deadlock risks explained above.
+ *
+ * Note that IOs may get submitted before pgaio_exit_batchmode() is called,
+ * e.g. because too many IOs have been staged or because pgaio_submit_staged()
+ * was called.
+ */
+void
+pgaio_enter_batchmode(void)
+{
+ if (pgaio_my_backend->in_batchmode)
+ elog(ERROR, "starting batch while batch already in progress");
+ pgaio_my_backend->in_batchmode = true;
+}
+
+/*
+ * Stop submitting IOs in batches.
+ */
+void
+pgaio_exit_batchmode(void)
+{
+ Assert(pgaio_my_backend->in_batchmode);
+
+ pgaio_submit_staged();
+ pgaio_my_backend->in_batchmode = false;
+}
+
+/*
+ * Are there staged but unsubmitted IOs?
+ *
+ * See comment above pgaio_enter_batchmode() for why code may need to check if
+ * there is IO in that state.
+ */
+bool
+pgaio_have_staged(void)
+{
+ Assert(pgaio_my_backend->in_batchmode ||
+ pgaio_my_backend->num_staged_ios == 0);
+ return pgaio_my_backend->num_staged_ios > 0;
+}
+
+/*
+ * Submit all staged but not yet submitted IOs.
+ *
+ * Unless in batch mode, this never needs to be called, as IOs get submitted
+ * as soon as possible. While in batchmode pgaio_submit_staged() can be called
+ * before waiting on another backend, to avoid the risk of deadlocks. See
+ * pgaio_enter_batchmode().
+ */
+void
+pgaio_submit_staged(void)
+{
+ int total_submitted = 0;
+ int did_submit;
+
+ if (pgaio_my_backend->num_staged_ios == 0)
+ return;
+
+
+ START_CRIT_SECTION();
+
+ did_submit = pgaio_method_ops->submit(pgaio_my_backend->num_staged_ios,
+ pgaio_my_backend->staged_ios);
+
+ END_CRIT_SECTION();
+
+ total_submitted += did_submit;
+
+ Assert(total_submitted == did_submit);
+
+ pgaio_my_backend->num_staged_ios = 0;
+
+ pgaio_debug(DEBUG4,
+ "aio: submitted %d IOs",
+ total_submitted);
+}
+
+
+
+/* --------------------------------------------------------------------------------
+ * Other
+ * --------------------------------------------------------------------------------
+ */
+
+/*
+ * Need to submit staged but not yet submitted IOs using the fd, otherwise
+ * the IO would end up targeting something bogus.
+ */
+void
+pgaio_closing_fd(int fd)
+{
+ /*
+ * Might be called before AIO is initialized or in a subprocess that
+ * doesn't use AIO.
+ */
+ if (!pgaio_my_backend)
+ return;
+
+ /*
+ * For now just submit all staged IOs - we could be more selective, but
+ * it's probably not worth it.
+ */
+ pgaio_submit_staged();
+}
+
+/*
+ * Perform AIO related cleanup after an error.
+ *
+ * This should be called early in the error recovery paths, as later steps may
+ * need to issue AIO (e.g. to record a transaction abort WAL record).
+ */
+void
+pgaio_error_cleanup(void)
+{
+ /*
+ * It is possible that code errored out after pgaio_enter_batchmode() but
+ * before pgaio_exit_batchmode() was called. In that case we need to
+ * submit the IO now.
+ */
+ if (pgaio_my_backend->in_batchmode)
+ {
+ pgaio_my_backend->in_batchmode = false;
+
+ pgaio_submit_staged();
+ }
+
+ /*
+ * As we aren't in batchmode, there shouldn't be any unsubmitted IOs.
+ */
+ Assert(pgaio_my_backend->num_staged_ios == 0);
+}
+
+/*
+ * Perform AIO related checks at (sub-)transactional boundaries.
+ *
+ * This should be called late during (sub-)transactional commit/abort, after
+ * all steps that might need to perform AIO, so that we can verify that the
+ * AIO subsystem is in a valid state at the end of a transaction.
+ */
+void
+AtEOXact_Aio(bool is_commit)
+{
+ /*
+ * We should never be in batch mode at transactional boundaries. In case
+ * an error was thrown while in batch mode, pgaio_error_cleanup() should
+ * have exited batchmode.
+ *
+ * In case we are in batchmode somehow, make sure to submit all staged
+ * IOs, other backends may need them to complete to continue.
+ */
+ if (pgaio_my_backend->in_batchmode)
+ {
+ pgaio_error_cleanup();
+ elog(WARNING, "open AIO batch at end of (sub-)transaction");
+ }
+
+ /*
+ * As we aren't in batchmode, there shouldn't be any unsubmitted IOs.
+ */
+ Assert(pgaio_my_backend->num_staged_ios == 0);
+}
+
+void
+pgaio_shutdown(int code, Datum arg)
+{
+ AtEOXact_Aio(code == 0);
+ Assert(pgaio_my_backend);
+ Assert(!pgaio_my_backend->handed_out_io);
+
+ /*
+ * Before exiting, make sure that all IOs are finished. That has two main
+ * purposes: - it's somewhat annoying to see partially finished IOs in
+ * stats views etc - it's rumored that some kernel-level AIO mechanisms
+ * don't deal well with the issuer of an AIO exiting
+ */
+
+ while (!dclist_is_empty(&pgaio_my_backend->in_flight_ios))
+ {
+ PgAioHandle *ioh = dclist_head_element(PgAioHandle, node, &pgaio_my_backend->in_flight_ios);
+
+ /* see comment in pgaio_io_wait_for_free() about raciness */
+ pgaio_io_wait(ioh, ioh->generation);
+ }
+
+ pgaio_my_backend = NULL;
+}
void
assign_io_method(int newval, void *extra)
{
+ Assert(pgaio_method_ops_table[newval] != NULL);
+ Assert(newval < lengthof(io_method_options));
+
+ pgaio_method_ops = pgaio_method_ops_table[newval];
}
bool
@@ -57,11 +1136,41 @@ check_io_max_concurrency(int *newval, void **extra, GucSource source)
return true;
}
+
+/* --------------------------------------------------------------------------------
+ * Injection point support
+ * --------------------------------------------------------------------------------
+ */
+
+#ifdef USE_INJECTION_POINTS
+
/*
- * Release IO handle during resource owner cleanup.
+ * Call injection point with support for pgaio_inj_io_get().
*/
void
-pgaio_io_release_resowner(dlist_node *ioh_node, bool on_error)
+pgaio_io_call_inj(PgAioHandle *ioh, const char *injection_point)
{
- /* placeholder for later */
+ pgaio_inj_cur_handle = ioh;
+
+ PG_TRY();
+ {
+ InjectionPointCached(injection_point);
+ }
+ PG_FINALLY();
+ {
+ pgaio_inj_cur_handle = NULL;
+ }
+ PG_END_TRY();
}
+
+/*
+ * Return IO associated with injection point invocation. This is only needed
+ * as injection points currently don't support arguments.
+ */
+PgAioHandle *
+pgaio_inj_io_get(void)
+{
+ return pgaio_inj_cur_handle;
+}
+
+#endif
diff --git a/src/backend/storage/aio/aio_callback.c b/src/backend/storage/aio/aio_callback.c
new file mode 100644
index 00000000000..3071bf19f23
--- /dev/null
+++ b/src/backend/storage/aio/aio_callback.c
@@ -0,0 +1,295 @@
+/*-------------------------------------------------------------------------
+ *
+ * aio_callback.c
+ * AIO - Functionality related to callbacks that can be registered on IO
+ * Handles
+ *
+ * Portions Copyright (c) 1996-2025, PostgreSQL Global Development Group
+ * Portions Copyright (c) 1994, Regents of the University of California
+ *
+ * IDENTIFICATION
+ * src/backend/storage/aio/aio_callback.c
+ *
+ *-------------------------------------------------------------------------
+ */
+
+#include "postgres.h"
+
+#include "miscadmin.h"
+#include "storage/aio.h"
+#include "storage/aio_internal.h"
+
+
+/* just to have something to put into the aio_handle_cbs */
+static const struct PgAioHandleCallbacks aio_invalid_cb = {0};
+
+typedef struct PgAioHandleCallbacksEntry
+{
+ const PgAioHandleCallbacks *const cb;
+ const char *const name;
+} PgAioHandleCallbacksEntry;
+
+/*
+ * Callback definition for the callbacks that can be registered on an IO
+ * handle. See PgAioHandleCallbackID's definition for an explanation for why
+ * callbacks are not identified by a pointer.
+ */
+static const PgAioHandleCallbacksEntry aio_handle_cbs[] = {
+#define CALLBACK_ENTRY(id, callback) [id] = {.cb = &callback, .name = #callback}
+ CALLBACK_ENTRY(PGAIO_HCB_INVALID, aio_invalid_cb),
+#undef CALLBACK_ENTRY
+};
+
+
+
+/*
+ * Register callback for the IO handle.
+ *
+ * Only a limited number (PGAIO_HANDLE_MAX_CALLBACKS) of callbacks can be
+ * registered for each IO.
+ *
+ * Callbacks need to be registered before [indirectly] calling
+ * pgaio_io_prep_*(), as the IO may be executed immediately.
+ *
+ * A callback can be passed a small bit of data, e.g. to indicate whether to
+ * zero a buffer if it is invalid.
+ *
+ *
+ * Note that callbacks are executed in critical sections. This is necessary
+ * to be able to execute IO in critical sections (consider e.g. WAL
+ * logging). To perform AIO we first need to acquire a handle, which, if there
+ * are no free handles, requires waiting for IOs to complete and to execute
+ * their completion callbacks.
+ *
+ * Callbacks may be executed in the issuing backend but also in another
+ * backend (because that backend is waiting for the IO) or in IO workers (if
+ * io_method=worker is used).
+ *
+ *
+ * See PgAioHandleCallbackID's definition for an explanation for why
+ * callbacks are not identified by a pointer.
+ */
+void
+pgaio_io_register_callbacks(PgAioHandle *ioh, PgAioHandleCallbackID cb_id,
+ uint8 cb_data)
+{
+ const PgAioHandleCallbacksEntry *ce = &aio_handle_cbs[cb_id];
+
+ if (cb_id >= lengthof(aio_handle_cbs))
+ elog(ERROR, "callback %d is out of range", cb_id);
+ if (aio_handle_cbs[cb_id].cb->complete_shared == NULL &&
+ aio_handle_cbs[cb_id].cb->complete_local == NULL)
+ elog(ERROR, "callback %d does not have completion callback", cb_id);
+ if (ioh->num_callbacks >= PGAIO_HANDLE_MAX_CALLBACKS)
+ elog(PANIC, "too many callbacks, the max is %d", PGAIO_HANDLE_MAX_CALLBACKS);
+ ioh->callbacks[ioh->num_callbacks] = cb_id;
+ ioh->callbacks_data[ioh->num_callbacks] = cb_data;
+
+ pgaio_debug_io(DEBUG3, ioh,
+ "adding cb #%d, id %d/%s",
+ ioh->num_callbacks + 1,
+ cb_id, ce->name);
+
+ ioh->num_callbacks++;
+}
+
+/*
+ * Associate an array of data with the Handle. This is e.g. useful to the
+ * transport knowledge about which buffers a multi-block IO affects to
+ * completion callbacks.
+ *
+ * Right now this can be done only once for each IO, even though multiple
+ * callbacks can be registered. There aren't any known usecases requiring more
+ * and the required amount of shared memory does add up, so it doesn't seem
+ * worth multiplying memory usage by PGAIO_HANDLE_MAX_CALLBACKS.
+ */
+void
+pgaio_io_set_handle_data_64(PgAioHandle *ioh, uint64 *data, uint8 len)
+{
+ Assert(ioh->state == PGAIO_HS_HANDED_OUT);
+ Assert(ioh->handle_data_len == 0);
+ Assert(len <= PG_IOV_MAX);
+
+ for (int i = 0; i < len; i++)
+ pgaio_ctl->handle_data[ioh->iovec_off + i] = data[i];
+ ioh->handle_data_len = len;
+}
+
+/*
+ * Convenience version of pgaio_io_set_handle_data_64() that converts a 32bit
+ * array to a 64bit array. Without it callers would end up needing to
+ * open-code equivalent code.
+ */
+void
+pgaio_io_set_handle_data_32(PgAioHandle *ioh, uint32 *data, uint8 len)
+{
+ Assert(ioh->state == PGAIO_HS_HANDED_OUT);
+ Assert(ioh->handle_data_len == 0);
+ Assert(len <= PG_IOV_MAX);
+
+ for (int i = 0; i < len; i++)
+ pgaio_ctl->handle_data[ioh->iovec_off + i] = data[i];
+ ioh->handle_data_len = len;
+}
+
+/*
+ * Return data set with pgaio_io_set_handle_data_*().
+ */
+uint64 *
+pgaio_io_get_handle_data(PgAioHandle *ioh, uint8 *len)
+{
+ Assert(ioh->handle_data_len > 0);
+
+ *len = ioh->handle_data_len;
+
+ return &pgaio_ctl->handle_data[ioh->iovec_off];
+}
+
+/*
+ * Internal function which invokes ->stage for all the registered callbacks.
+ */
+void
+pgaio_io_call_stage(PgAioHandle *ioh)
+{
+ Assert(ioh->target > PGAIO_TID_INVALID && ioh->target < PGAIO_TID_COUNT);
+ Assert(ioh->op > PGAIO_OP_INVALID && ioh->op < PGAIO_OP_COUNT);
+
+ for (int i = ioh->num_callbacks; i > 0; i--)
+ {
+ PgAioHandleCallbackID cb_id = ioh->callbacks[i - 1];
+ uint8 cb_data = ioh->callbacks_data[i - 1];
+ const PgAioHandleCallbacksEntry *ce = &aio_handle_cbs[cb_id];
+
+ if (!ce->cb->stage)
+ continue;
+
+ pgaio_debug_io(DEBUG3, ioh,
+ "calling cb #%d %d/%s->stage(%u)",
+ i, cb_id, ce->name, cb_data);
+ ce->cb->stage(ioh, cb_data);
+ }
+}
+
+/*
+ * Internal function which invokes ->complete_shared for all the registered
+ * callbacks.
+ */
+void
+pgaio_io_call_complete_shared(PgAioHandle *ioh)
+{
+ PgAioResult result;
+
+ START_CRIT_SECTION();
+
+ Assert(ioh->target > PGAIO_TID_INVALID && ioh->target < PGAIO_TID_COUNT);
+ Assert(ioh->op > PGAIO_OP_INVALID && ioh->op < PGAIO_OP_COUNT);
+
+ result.status = ARS_OK; /* low level IO is always considered OK */
+ result.result = ioh->result;
+ result.id = PGAIO_HCB_INVALID;
+ result.error_data = 0;
+
+ /*
+ * Call callbacks with the last registered (innermost) callback first.
+ * Each callback can modify the result forwarded to the next callback.
+ */
+ for (int i = ioh->num_callbacks; i > 0; i--)
+ {
+ PgAioHandleCallbackID cb_id = ioh->callbacks[i - 1];
+ uint8 cb_data = ioh->callbacks_data[i - 1];
+ const PgAioHandleCallbacksEntry *ce = &aio_handle_cbs[cb_id];
+
+ if (!ce->cb->complete_shared)
+ continue;
+
+ pgaio_debug_io(DEBUG4, ioh,
+ "calling cb #%d, id %d/%s->complete_shared(%u) with distilled result: (status %s, id %u, error_data %d, result %d)",
+ i, cb_id, ce->name,
+ cb_data,
+ pgaio_result_status_string(result.status),
+ result.id, result.error_data, result.result);
+ result = ce->cb->complete_shared(ioh, result, cb_data);
+ }
+
+ ioh->distilled_result = result;
+
+ pgaio_debug_io(DEBUG3, ioh,
+ "after shared completion: distilled result: (status %s, id %u, error_data: %d, result %d), raw_result: %d",
+ pgaio_result_status_string(result.status),
+ result.id, result.error_data, result.result,
+ ioh->result);
+
+ END_CRIT_SECTION();
+}
+
+/*
+ * Internal function which invokes ->complete_local for all the registered
+ * callbacks.
+ *
+ * XXX: It'd be nice to deduplicate with pgaio_io_call_complete_shared().
+ */
+void
+pgaio_io_call_complete_local(PgAioHandle *ioh)
+{
+ PgAioResult result;
+
+ START_CRIT_SECTION();
+
+ Assert(ioh->target > PGAIO_TID_INVALID && ioh->target < PGAIO_TID_COUNT);
+ Assert(ioh->op > PGAIO_OP_INVALID && ioh->op < PGAIO_OP_COUNT);
+
+ /* start with distilled result from shared callback */
+ result = ioh->distilled_result;
+
+ for (int i = ioh->num_callbacks; i > 0; i--)
+ {
+ PgAioHandleCallbackID cb_id = ioh->callbacks[i - 1];
+ uint8 cb_data = ioh->callbacks_data[i - 1];
+ const PgAioHandleCallbacksEntry *ce = &aio_handle_cbs[cb_id];
+
+ if (!ce->cb->complete_local)
+ continue;
+
+ pgaio_debug_io(DEBUG4, ioh,
+ "calling cb #%d, id %d/%s->complete_local(%u) with distilled result: status %s, id %u, error_data %d, result %d",
+ i, cb_id, ce->name, cb_data,
+ pgaio_result_status_string(result.status),
+ result.id, result.error_data, result.result);
+ result = ce->cb->complete_local(ioh, result, cb_data);
+ }
+
+ /*
+ * Note that we don't save the result in ioh->distilled_result, the local
+ * callback's result should not ever matter to other waiters.
+ */
+ pgaio_debug_io(DEBUG3, ioh,
+ "after local completion: distilled result: (status %s, id %u, error_data %d, result %d), raw_result: %d",
+ pgaio_result_status_string(result.status),
+ result.id, result.error_data, result.result,
+ ioh->result);
+
+ END_CRIT_SECTION();
+}
+
+
+
+/* --------------------------------------------------------------------------------
+ * IO Result
+ * --------------------------------------------------------------------------------
+ */
+
+void
+pgaio_result_report(PgAioResult result, const PgAioTargetData *target_data, int elevel)
+{
+ PgAioHandleCallbackID cb_id = result.id;
+ const PgAioHandleCallbacksEntry *ce = &aio_handle_cbs[cb_id];
+
+ Assert(result.status != ARS_UNKNOWN);
+ Assert(result.status != ARS_OK);
+
+ if (ce->cb->report == NULL)
+ elog(ERROR, "callback %d/%s does not have report callback",
+ result.id, ce->name);
+
+ ce->cb->report(result, target_data, elevel);
+}
diff --git a/src/backend/storage/aio/aio_init.c b/src/backend/storage/aio/aio_init.c
index aeacc144149..8e1162b09dc 100644
--- a/src/backend/storage/aio/aio_init.c
+++ b/src/backend/storage/aio/aio_init.c
@@ -14,24 +14,210 @@
#include "postgres.h"
+#include "miscadmin.h"
+#include "storage/aio.h"
+#include "storage/aio_internal.h"
#include "storage/aio_subsys.h"
+#include "storage/ipc.h"
+#include "storage/proc.h"
+#include "storage/shmem.h"
+#include "utils/guc.h"
+static Size
+AioCtlShmemSize(void)
+{
+ Size sz;
+
+ /* pgaio_ctl itself */
+ sz = offsetof(PgAioCtl, io_handles);
+
+ return sz;
+}
+
+static uint32
+AioProcs(void)
+{
+ return MaxBackends + NUM_AUXILIARY_PROCS;
+}
+
+static Size
+AioBackendShmemSize(void)
+{
+ return mul_size(AioProcs(), sizeof(PgAioBackend));
+}
+
+static Size
+AioHandleShmemSize(void)
+{
+ Size sz;
+
+ /* ios */
+ sz = mul_size(AioProcs(),
+ mul_size(io_max_concurrency, sizeof(PgAioHandle)));
+
+ return sz;
+}
+
+static Size
+AioHandleIOVShmemSize(void)
+{
+ return mul_size(sizeof(struct iovec),
+ mul_size(mul_size(PG_IOV_MAX, AioProcs()),
+ io_max_concurrency));
+}
+
+static Size
+AioHandleDataShmemSize(void)
+{
+ return mul_size(sizeof(uint64),
+ mul_size(mul_size(PG_IOV_MAX, AioProcs()),
+ io_max_concurrency));
+}
+
+/*
+ * Choose a suitable value for io_max_concurrency.
+ *
+ * It's unlikely that we could have more IOs in flight than buffers that we
+ * would be allowed to pin.
+ *
+ * On the upper end, apply a cap too - just because shared_buffers is large,
+ * it doesn't make sense have millions of buffers undergo IO concurrently.
+ */
+static int
+AioChooseMaxConcurrency(void)
+{
+ uint32 max_backends;
+ int max_proportional_pins;
+
+ /* Similar logic to LimitAdditionalPins() */
+ max_backends = MaxBackends + NUM_AUXILIARY_PROCS;
+ max_proportional_pins = NBuffers / max_backends;
+
+ max_proportional_pins = Max(max_proportional_pins, 1);
+
+ /* apply upper limit */
+ return Min(max_proportional_pins, 64);
+}
+
Size
AioShmemSize(void)
{
Size sz = 0;
+ /*
+ * We prefer to report this value's source as PGC_S_DYNAMIC_DEFAULT.
+ * However, if the DBA explicitly set wal_buffers = -1 in the config file,
+ * then PGC_S_DYNAMIC_DEFAULT will fail to override that and we must force
+ *
+ */
+ if (io_max_concurrency == -1)
+ {
+ char buf[32];
+
+ snprintf(buf, sizeof(buf), "%d", AioChooseMaxConcurrency());
+ SetConfigOption("io_max_concurrency", buf, PGC_POSTMASTER,
+ PGC_S_DYNAMIC_DEFAULT);
+ if (io_max_concurrency == -1) /* failed to apply it? */
+ SetConfigOption("io_max_concurrency", buf, PGC_POSTMASTER,
+ PGC_S_OVERRIDE);
+ }
+
+ sz = add_size(sz, AioCtlShmemSize());
+ sz = add_size(sz, AioBackendShmemSize());
+ sz = add_size(sz, AioHandleShmemSize());
+ sz = add_size(sz, AioHandleIOVShmemSize());
+ sz = add_size(sz, AioHandleDataShmemSize());
+
+ if (pgaio_method_ops->shmem_size)
+ sz = add_size(sz, pgaio_method_ops->shmem_size());
+
return sz;
}
void
AioShmemInit(void)
{
+ bool found;
+ uint32 io_handle_off = 0;
+ uint32 iovec_off = 0;
+ uint32 per_backend_iovecs = io_max_concurrency * PG_IOV_MAX;
+
+ pgaio_ctl = (PgAioCtl *)
+ ShmemInitStruct("AioCtl", AioCtlShmemSize(), &found);
+
+ if (found)
+ goto out;
+
+ memset(pgaio_ctl, 0, AioCtlShmemSize());
+
+ pgaio_ctl->io_handle_count = AioProcs() * io_max_concurrency;
+ pgaio_ctl->iovec_count = AioProcs() * per_backend_iovecs;
+
+ pgaio_ctl->backend_state = (PgAioBackend *)
+ ShmemInitStruct("AioBackend", AioBackendShmemSize(), &found);
+
+ pgaio_ctl->io_handles = (PgAioHandle *)
+ ShmemInitStruct("AioHandle", AioHandleShmemSize(), &found);
+
+ pgaio_ctl->iovecs = (struct iovec *)
+ ShmemInitStruct("AioHandleIOV", AioHandleIOVShmemSize(), &found);
+ pgaio_ctl->handle_data = (uint64 *)
+ ShmemInitStruct("AioHandleData", AioHandleDataShmemSize(), &found);
+
+ for (int procno = 0; procno < AioProcs(); procno++)
+ {
+ PgAioBackend *bs = &pgaio_ctl->backend_state[procno];
+
+ bs->io_handle_off = io_handle_off;
+ io_handle_off += io_max_concurrency;
+
+ dclist_init(&bs->idle_ios);
+ memset(bs->staged_ios, 0, sizeof(PgAioHandle *) * PGAIO_SUBMIT_BATCH_SIZE);
+ dclist_init(&bs->in_flight_ios);
+
+ /* initialize per-backend IOs */
+ for (int i = 0; i < io_max_concurrency; i++)
+ {
+ PgAioHandle *ioh = &pgaio_ctl->io_handles[bs->io_handle_off + i];
+
+ ioh->generation = 1;
+ ioh->owner_procno = procno;
+ ioh->iovec_off = iovec_off;
+ ioh->handle_data_len = 0;
+ ioh->report_return = NULL;
+ ioh->resowner = NULL;
+ ioh->num_callbacks = 0;
+ ioh->distilled_result.status = ARS_UNKNOWN;
+ ioh->flags = 0;
+
+ ConditionVariableInit(&ioh->cv);
+
+ dclist_push_tail(&bs->idle_ios, &ioh->node);
+ iovec_off += PG_IOV_MAX;
+ }
+ }
+
+out:
+ /* Initialize IO method specific resources. */
+ if (pgaio_method_ops->shmem_init)
+ pgaio_method_ops->shmem_init(!found);
}
void
pgaio_init_backend(void)
{
+ /* shouldn't be initialized twice */
+ Assert(!pgaio_my_backend);
+
+ if (MyProc == NULL || MyProcNumber >= AioProcs())
+ elog(ERROR, "aio requires a normal PGPROC");
+
+ pgaio_my_backend = &pgaio_ctl->backend_state[MyProcNumber];
+
+ if (pgaio_method_ops->init_backend)
+ pgaio_method_ops->init_backend();
+
+ before_shmem_exit(pgaio_shutdown, 0);
}
diff --git a/src/backend/storage/aio/aio_io.c b/src/backend/storage/aio/aio_io.c
new file mode 100644
index 00000000000..89376ff4040
--- /dev/null
+++ b/src/backend/storage/aio/aio_io.c
@@ -0,0 +1,180 @@
+/*-------------------------------------------------------------------------
+ *
+ * aio_io.c
+ * AIO - Low Level IO Handling
+ *
+ * Functions related to associating IO operations to IO Handles and IO-method
+ * independent support functions for actually performing IO.
+ *
+ *
+ * Portions Copyright (c) 1996-2025, PostgreSQL Global Development Group
+ * Portions Copyright (c) 1994, Regents of the University of California
+ *
+ * IDENTIFICATION
+ * src/backend/storage/aio/aio_io.c
+ *
+ *-------------------------------------------------------------------------
+ */
+
+#include "postgres.h"
+
+#include "miscadmin.h"
+#include "storage/aio.h"
+#include "storage/aio_internal.h"
+#include "storage/fd.h"
+#include "utils/wait_event.h"
+
+
+static void pgaio_io_before_prep(PgAioHandle *ioh);
+
+
+
+/* --------------------------------------------------------------------------------
+ * Public IO related functions operating on IO Handles
+ * --------------------------------------------------------------------------------
+ */
+
+/*
+ * Scatter/gather IO needs to associate an iovec with the Handle. To support
+ * worker mode this data needs to be in shared memory.
+ *
+ * XXX: Right now the amount of space available for each IO is
+ * PG_IOV_MAX. While it's tempting to use the io_combine_limit GUC, that's
+ * PGC_USERSET, so we can't allocate shared memory based on that.
+ */
+int
+pgaio_io_get_iovec(PgAioHandle *ioh, struct iovec **iov)
+{
+ Assert(ioh->state == PGAIO_HS_HANDED_OUT);
+
+ *iov = &pgaio_ctl->iovecs[ioh->iovec_off];
+
+ return PG_IOV_MAX;
+}
+
+PgAioOpData *
+pgaio_io_get_op_data(PgAioHandle *ioh)
+{
+ return &ioh->op_data;
+}
+
+
+
+/* --------------------------------------------------------------------------------
+ * "Preparation" routines for individual IO operations
+ *
+ * These are called by the code actually initiating an IO, to associate the IO
+ * specific data with an AIO handle.
+ *
+ * Each of the preparation routines first needs to call
+ * pgaio_io_before_prep(), then fill IO specific fields in the handle and then
+ * finally call pgaio_io_stage().
+ * --------------------------------------------------------------------------------
+ */
+
+void
+pgaio_io_prep_readv(PgAioHandle *ioh,
+ int fd, int iovcnt, uint64 offset)
+{
+ pgaio_io_before_prep(ioh);
+
+ ioh->op_data.read.fd = fd;
+ ioh->op_data.read.offset = offset;
+ ioh->op_data.read.iov_length = iovcnt;
+
+ pgaio_io_stage(ioh, PGAIO_OP_READV);
+}
+
+void
+pgaio_io_prep_writev(PgAioHandle *ioh,
+ int fd, int iovcnt, uint64 offset)
+{
+ pgaio_io_before_prep(ioh);
+
+ ioh->op_data.write.fd = fd;
+ ioh->op_data.write.offset = offset;
+ ioh->op_data.write.iov_length = iovcnt;
+
+ pgaio_io_stage(ioh, PGAIO_OP_WRITEV);
+}
+
+
+
+/* --------------------------------------------------------------------------------
+ * Internal IO related functions operating on IO Handles
+ * --------------------------------------------------------------------------------
+ */
+
+/*
+ * Execute IO operation synchronously. This is implemented here, not in
+ * method_sync.c, because other IO methods lso might use it / fall back to it.
+ */
+void
+pgaio_io_perform_synchronously(PgAioHandle *ioh)
+{
+ ssize_t result = 0;
+ struct iovec *iov = &pgaio_ctl->iovecs[ioh->iovec_off];
+
+ START_CRIT_SECTION();
+
+ /* Perform IO. */
+ switch (ioh->op)
+ {
+ case PGAIO_OP_READV:
+ pgstat_report_wait_start(WAIT_EVENT_DATA_FILE_READ);
+ result = pg_preadv(ioh->op_data.read.fd, iov,
+ ioh->op_data.read.iov_length,
+ ioh->op_data.read.offset);
+ pgstat_report_wait_end();
+ break;
+ case PGAIO_OP_WRITEV:
+ pgstat_report_wait_start(WAIT_EVENT_DATA_FILE_WRITE);
+ result = pg_pwritev(ioh->op_data.write.fd, iov,
+ ioh->op_data.write.iov_length,
+ ioh->op_data.write.offset);
+ pgstat_report_wait_end();
+ break;
+ case PGAIO_OP_INVALID:
+ elog(ERROR, "trying to execute invalid IO operation");
+ }
+
+ ioh->result = result < 0 ? -errno : result;
+
+ pgaio_io_process_completion(ioh, ioh->result);
+
+ END_CRIT_SECTION();
+}
+
+/*
+ * Helper function to be called by IO operation preparation functions, before
+ * any data in the handle is set. Mostly to centralize assertions.
+ */
+static void
+pgaio_io_before_prep(PgAioHandle *ioh)
+{
+ Assert(ioh->state == PGAIO_HS_HANDED_OUT);
+ Assert(pgaio_io_has_target(ioh));
+ Assert(ioh->op == PGAIO_OP_INVALID);
+}
+
+/*
+ * Could be made part of the public interface, but it's not clear there's
+ * really a use case for that.
+ */
+const char *
+pgaio_io_get_op_name(PgAioHandle *ioh)
+{
+ Assert(ioh->op >= 0 && ioh->op < PGAIO_OP_COUNT);
+
+ switch (ioh->op)
+ {
+ case PGAIO_OP_INVALID:
+ return "invalid";
+ case PGAIO_OP_READV:
+ return "read";
+ case PGAIO_OP_WRITEV:
+ return "write";
+ }
+
+ return NULL; /* silence compiler */
+}
diff --git a/src/backend/storage/aio/aio_target.c b/src/backend/storage/aio/aio_target.c
new file mode 100644
index 00000000000..637e3aa0928
--- /dev/null
+++ b/src/backend/storage/aio/aio_target.c
@@ -0,0 +1,106 @@
+/*-------------------------------------------------------------------------
+ *
+ * aio_target.c
+ * AIO - Functionality related to executing IO for different targets
+ *
+ * Portions Copyright (c) 1996-2025, PostgreSQL Global Development Group
+ * Portions Copyright (c) 1994, Regents of the University of California
+ *
+ * IDENTIFICATION
+ * src/backend/storage/aio/aio_target.c
+ *
+ *-------------------------------------------------------------------------
+ */
+
+#include "postgres.h"
+
+#include "storage/aio.h"
+#include "storage/aio_internal.h"
+
+
+/*
+ * Registry for entities that can be the target of AIO.
+ *
+ * To support executing using worker processes, the file descriptor for an IO
+ * may need to be be reopened in a different process. This is done via the
+ * PgAioTargetInfo.reopen callback.
+ */
+static const PgAioTargetInfo *pgaio_target_info[] = {
+ [PGAIO_TID_INVALID] = &(PgAioTargetInfo) {
+ .name = "invalid",
+ },
+};
+
+
+
+bool
+pgaio_io_has_target(PgAioHandle *ioh)
+{
+ return ioh->target != PGAIO_TID_INVALID;
+}
+
+/*
+ * Return the name for the target associated with the IO. Mostly useful for
+ * debugging/logging.
+ */
+const char *
+pgaio_io_get_target_name(PgAioHandle *ioh)
+{
+ Assert(ioh->target >= 0 && ioh->target < PGAIO_TID_COUNT);
+
+ return pgaio_target_info[ioh->target]->name;
+}
+
+/*
+ * Assign a target to the IO.
+ *
+ * This has to be called exactly once before pgaio_io_prep_*() is called.
+ */
+void
+pgaio_io_set_target(PgAioHandle *ioh, PgAioTargetID targetid)
+{
+ Assert(ioh->state == PGAIO_HS_HANDED_OUT);
+ Assert(ioh->target == PGAIO_TID_INVALID);
+
+ ioh->target = targetid;
+}
+
+PgAioTargetData *
+pgaio_io_get_target_data(PgAioHandle *ioh)
+{
+ return &ioh->target_data;
+}
+
+/*
+ * Return a stringified description of the IO's target.
+ *
+ * The string is localized and allocated in the current memory context.
+ */
+char *
+pgaio_io_get_target_description(PgAioHandle *ioh)
+{
+ return pgaio_target_info[ioh->target]->describe_identity(&ioh->target_data);
+}
+
+/*
+ * Internal: Check if pgaio_io_reopen() is available for the IO.
+ */
+bool
+pgaio_io_can_reopen(PgAioHandle *ioh)
+{
+ return pgaio_target_info[ioh->target]->reopen != NULL;
+}
+
+/*
+ * Internal: Before executing an IO outside of the context of the process the
+ * IO has been prepared in, the file descriptor has to be reopened - any FD
+ * referenced in the IO itself, won't be valid in the separate process.
+ */
+void
+pgaio_io_reopen(PgAioHandle *ioh)
+{
+ Assert(ioh->target >= 0 && ioh->target < PGAIO_TID_COUNT);
+ Assert(ioh->op >= 0 && ioh->op < PGAIO_OP_COUNT);
+
+ pgaio_target_info[ioh->target]->reopen(ioh);
+}
diff --git a/src/backend/storage/aio/meson.build b/src/backend/storage/aio/meson.build
index c822fd4ddf7..2c26089d52e 100644
--- a/src/backend/storage/aio/meson.build
+++ b/src/backend/storage/aio/meson.build
@@ -2,6 +2,10 @@
backend_sources += files(
'aio.c',
+ 'aio_callback.c',
'aio_init.c',
+ 'aio_io.c',
+ 'aio_target.c',
+ 'method_sync.c',
'read_stream.c',
)
diff --git a/src/backend/storage/aio/method_sync.c b/src/backend/storage/aio/method_sync.c
new file mode 100644
index 00000000000..43f9c8bd0b3
--- /dev/null
+++ b/src/backend/storage/aio/method_sync.c
@@ -0,0 +1,47 @@
+/*-------------------------------------------------------------------------
+ *
+ * method_sync.c
+ * AIO - perform "AIO" by executing it synchronously
+ *
+ * This method is mainly to check if AIO use causes regressions. Other IO
+ * methods might also fall back to the synchronous method for functionality
+ * they cannot provide.
+ *
+ * Portions Copyright (c) 1996-2025, PostgreSQL Global Development Group
+ * Portions Copyright (c) 1994, Regents of the University of California
+ *
+ * IDENTIFICATION
+ * src/backend/storage/aio/method_sync.c
+ *
+ *-------------------------------------------------------------------------
+ */
+
+#include "postgres.h"
+
+#include "storage/aio.h"
+#include "storage/aio_internal.h"
+
+static bool pgaio_sync_needs_synchronous_execution(PgAioHandle *ioh);
+static int pgaio_sync_submit(uint16 num_staged_ios, PgAioHandle **staged_ios);
+
+
+const IoMethodOps pgaio_sync_ops = {
+ .needs_synchronous_execution = pgaio_sync_needs_synchronous_execution,
+ .submit = pgaio_sync_submit,
+};
+
+
+
+static bool
+pgaio_sync_needs_synchronous_execution(PgAioHandle *ioh)
+{
+ return true;
+}
+
+static int
+pgaio_sync_submit(uint16 num_staged_ios, PgAioHandle **staged_ios)
+{
+ elog(ERROR, "should be unreachable");
+
+ return 0;
+}
diff --git a/src/backend/utils/activity/wait_event_names.txt b/src/backend/utils/activity/wait_event_names.txt
index e199f071628..6f3ca878bd1 100644
--- a/src/backend/utils/activity/wait_event_names.txt
+++ b/src/backend/utils/activity/wait_event_names.txt
@@ -191,6 +191,7 @@ ABI_compatibility:
Section: ClassName - WaitEventIO
+AIO_IO_COMPLETION "Waiting for IO completion."
BASEBACKUP_READ "Waiting for base backup to read from a file."
BASEBACKUP_SYNC "Waiting for data written by a base backup to reach durable storage."
BASEBACKUP_WRITE "Waiting for base backup to write to a file."
diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list
index c80365974e4..4405c2409d1 100644
--- a/src/tools/pgindent/typedefs.list
+++ b/src/tools/pgindent/typedefs.list
@@ -1277,6 +1277,7 @@ InvalMessageArray
InvalidationInfo
InvalidationMsgsGroup
IoMethod
+IoMethodOps
IpcMemoryId
IpcMemoryKey
IpcMemoryState
@@ -2123,6 +2124,26 @@ Permutation
PermutationStep
PermutationStepBlocker
PermutationStepBlockerType
+PgAioBackend
+PgAioCtl
+PgAioHandle
+PgAioHandleCallbackID
+PgAioHandleCallbackStage
+PgAioHandleCallbackComplete
+PgAioHandleCallbackReport
+PgAioHandleCallbacks
+PgAioHandleCallbacksEntry
+PgAioHandleFlags
+PgAioHandleState
+PgAioOp
+PgAioOpData
+PgAioResult
+PgAioResultStatus
+PgAioReturn
+PgAioTargetData
+PgAioTargetID
+PgAioTargetInfo
+PgAioWaitRef
PgArchData
PgBackendGSSStatus
PgBackendSSLStatus
--
2.48.1.76.g4e746b1a31.dirty
[text/x-diff] v2.5-0003-aio-Skeleton-IO-worker-infrastructure.patch (20.5K, ../../vz5i2x2wkjjpp6l2z4g4l3umpxktrxsejgdwefi7n6kzw666h3@rnhlvlpadig7/4-v2.5-0003-aio-Skeleton-IO-worker-infrastructure.patch)
download | inline diff:
From ddf5c176f211907cc3f338744ff36c6bd98b674e Mon Sep 17 00:00:00 2001
From: Andres Freund <andres@anarazel.de>
Date: Mon, 17 Feb 2025 17:57:00 -0500
Subject: [PATCH v2.5 03/30] aio: Skeleton IO worker infrastructure
This doesn't do anything useful on its own, but the code that needs to be
touched is independent of other changes.
Remarks:
- dynamic increase / decrease of workers based on IO load
Author:
Reviewed-by:
Discussion: https://postgr.es/m/
Backpatch:
---
src/include/miscadmin.h | 2 +
src/include/postmaster/postmaster.h | 1 +
src/include/storage/aio_subsys.h | 4 +
src/include/storage/io_worker.h | 22 +++
src/include/storage/proc.h | 4 +-
src/backend/postmaster/launch_backend.c | 2 +
src/backend/postmaster/pmchild.c | 1 +
src/backend/postmaster/postmaster.c | 165 ++++++++++++++++--
src/backend/storage/aio/Makefile | 1 +
src/backend/storage/aio/meson.build | 1 +
src/backend/storage/aio/method_worker.c | 88 ++++++++++
src/backend/tcop/postgres.c | 2 +
src/backend/utils/activity/pgstat_backend.c | 1 +
src/backend/utils/activity/pgstat_io.c | 1 +
.../utils/activity/wait_event_names.txt | 1 +
src/backend/utils/init/miscinit.c | 3 +
16 files changed, 286 insertions(+), 13 deletions(-)
create mode 100644 src/include/storage/io_worker.h
create mode 100644 src/backend/storage/aio/method_worker.c
diff --git a/src/include/miscadmin.h b/src/include/miscadmin.h
index a2b63495eec..54429e046a9 100644
--- a/src/include/miscadmin.h
+++ b/src/include/miscadmin.h
@@ -360,6 +360,7 @@ typedef enum BackendType
B_ARCHIVER,
B_BG_WRITER,
B_CHECKPOINTER,
+ B_IO_WORKER,
B_STARTUP,
B_WAL_RECEIVER,
B_WAL_SUMMARIZER,
@@ -389,6 +390,7 @@ extern PGDLLIMPORT BackendType MyBackendType;
#define AmWalReceiverProcess() (MyBackendType == B_WAL_RECEIVER)
#define AmWalSummarizerProcess() (MyBackendType == B_WAL_SUMMARIZER)
#define AmWalWriterProcess() (MyBackendType == B_WAL_WRITER)
+#define AmIoWorkerProcess() (MyBackendType == B_IO_WORKER)
#define AmSpecialWorkerProcess() \
(AmAutoVacuumLauncherProcess() || \
diff --git a/src/include/postmaster/postmaster.h b/src/include/postmaster/postmaster.h
index b6a3f275a1b..afefa35e2cc 100644
--- a/src/include/postmaster/postmaster.h
+++ b/src/include/postmaster/postmaster.h
@@ -98,6 +98,7 @@ extern void InitProcessGlobals(void);
extern int MaxLivePostmasterChildren(void);
extern bool PostmasterMarkPIDForWorkerNotify(int);
+extern void assign_io_workers(int newval, void *extra);
#ifdef WIN32
extern void pgwin32_register_deadchild_callback(HANDLE procHandle, DWORD procId);
diff --git a/src/include/storage/aio_subsys.h b/src/include/storage/aio_subsys.h
index e4faf692a38..ed00d5c47cd 100644
--- a/src/include/storage/aio_subsys.h
+++ b/src/include/storage/aio_subsys.h
@@ -30,4 +30,8 @@ extern void pgaio_init_backend(void);
extern void pgaio_error_cleanup(void);
extern void AtEOXact_Aio(bool is_commit);
+
+/* aio_worker.c */
+extern bool pgaio_workers_enabled(void);
+
#endif /* AIO_SUBSYS_H */
diff --git a/src/include/storage/io_worker.h b/src/include/storage/io_worker.h
new file mode 100644
index 00000000000..872241ae299
--- /dev/null
+++ b/src/include/storage/io_worker.h
@@ -0,0 +1,22 @@
+/*-------------------------------------------------------------------------
+ *
+ * io_worker.h
+ * IO worker for implementing AIO "ourselves"
+ *
+ *
+ * Portions Copyright (c) 1996-2025, PostgreSQL Global Development Group
+ * Portions Copyright (c) 1994, Regents of the University of California
+ *
+ * src/include/storage/io.h
+ *
+ *-------------------------------------------------------------------------
+ */
+#ifndef IO_WORKER_H
+#define IO_WORKER_H
+
+
+extern void IoWorkerMain(const void *startup_data, size_t startup_data_len) pg_attribute_noreturn();
+
+extern int io_workers;
+
+#endif /* IO_WORKER_H */
diff --git a/src/include/storage/proc.h b/src/include/storage/proc.h
index 20777f7d5ae..64e9b8ff8c5 100644
--- a/src/include/storage/proc.h
+++ b/src/include/storage/proc.h
@@ -448,7 +448,9 @@ extern PGDLLIMPORT PGPROC *PreparedXactProcs;
* 2 slots, but WAL writer is launched only after startup has exited, so we
* only need 6 slots.
*/
-#define NUM_AUXILIARY_PROCS 6
+#define MAX_IO_WORKERS 32
+#define NUM_AUXILIARY_PROCS (6 + MAX_IO_WORKERS)
+
/* configurable options */
extern PGDLLIMPORT int DeadlockTimeout;
diff --git a/src/backend/postmaster/launch_backend.c b/src/backend/postmaster/launch_backend.c
index 47375e5bfaa..516f2b5d212 100644
--- a/src/backend/postmaster/launch_backend.c
+++ b/src/backend/postmaster/launch_backend.c
@@ -48,6 +48,7 @@
#include "replication/slotsync.h"
#include "replication/walreceiver.h"
#include "storage/dsm.h"
+#include "storage/io_worker.h"
#include "storage/pg_shmem.h"
#include "tcop/backend_startup.h"
#include "utils/memutils.h"
@@ -197,6 +198,7 @@ static child_process_kind child_process_kinds[] = {
[B_ARCHIVER] = {"archiver", PgArchiverMain, true},
[B_BG_WRITER] = {"bgwriter", BackgroundWriterMain, true},
[B_CHECKPOINTER] = {"checkpointer", CheckpointerMain, true},
+ [B_IO_WORKER] = {"io_worker", IoWorkerMain, true},
[B_STARTUP] = {"startup", StartupProcessMain, true},
[B_WAL_RECEIVER] = {"wal_receiver", WalReceiverMain, true},
[B_WAL_SUMMARIZER] = {"wal_summarizer", WalSummarizerMain, true},
diff --git a/src/backend/postmaster/pmchild.c b/src/backend/postmaster/pmchild.c
index 0d473226c3a..cde1d23a4ca 100644
--- a/src/backend/postmaster/pmchild.c
+++ b/src/backend/postmaster/pmchild.c
@@ -101,6 +101,7 @@ InitPostmasterChildSlots(void)
pmchild_pools[B_AUTOVAC_WORKER].size = autovacuum_worker_slots;
pmchild_pools[B_BG_WORKER].size = max_worker_processes;
+ pmchild_pools[B_IO_WORKER].size = MAX_IO_WORKERS;
/*
* There can be only one of each of these running at a time. They each
diff --git a/src/backend/postmaster/postmaster.c b/src/backend/postmaster/postmaster.c
index 5dd3b6a4fd4..3cf4fffd3b4 100644
--- a/src/backend/postmaster/postmaster.c
+++ b/src/backend/postmaster/postmaster.c
@@ -108,9 +108,12 @@
#include "replication/logicallauncher.h"
#include "replication/slotsync.h"
#include "replication/walsender.h"
+#include "storage/aio_subsys.h"
#include "storage/fd.h"
+#include "storage/io_worker.h"
#include "storage/ipc.h"
#include "storage/pmsignal.h"
+#include "storage/proc.h"
#include "tcop/backend_startup.h"
#include "tcop/tcopprot.h"
#include "utils/datetime.h"
@@ -341,6 +344,7 @@ typedef enum
* ckpt */
PM_WAIT_XLOG_ARCHIVAL, /* waiting for archiver and walsenders to
* finish */
+ PM_WAIT_IO_WORKERS, /* waiting for io workers to exit */
PM_WAIT_CHECKPOINTER, /* waiting for checkpointer to shut down */
PM_WAIT_DEAD_END, /* waiting for dead-end children to exit */
PM_NO_CHILDREN, /* all important children have exited */
@@ -403,6 +407,10 @@ bool LoadedSSL = false;
static DNSServiceRef bonjour_sdref = NULL;
#endif
+/* State for IO worker management. */
+static int io_worker_count = 0;
+static PMChild *io_worker_children[MAX_IO_WORKERS];
+
/*
* postmaster.c - function prototypes
*/
@@ -437,6 +445,8 @@ static void TerminateChildren(int signal);
static int CountChildren(BackendTypeMask targetMask);
static void LaunchMissingBackgroundProcesses(void);
static void maybe_start_bgworkers(void);
+static bool maybe_reap_io_worker(int pid);
+static void maybe_adjust_io_workers(void);
static bool CreateOptsFile(int argc, char *argv[], char *fullprogname);
static PMChild *StartChildProcess(BackendType type);
static void StartSysLogger(void);
@@ -1366,6 +1376,11 @@ PostmasterMain(int argc, char *argv[])
*/
AddToDataDirLockFile(LOCK_FILE_LINE_PM_STATUS, PM_STATUS_STARTING);
+ UpdatePMState(PM_STARTUP);
+
+ /* Make sure we can perform I/O while starting up. */
+ maybe_adjust_io_workers();
+
/* Start bgwriter and checkpointer so they can help with recovery */
if (CheckpointerPMChild == NULL)
CheckpointerPMChild = StartChildProcess(B_CHECKPOINTER);
@@ -1378,7 +1393,6 @@ PostmasterMain(int argc, char *argv[])
StartupPMChild = StartChildProcess(B_STARTUP);
Assert(StartupPMChild != NULL);
StartupStatus = STARTUP_RUNNING;
- UpdatePMState(PM_STARTUP);
/* Some workers may be scheduled to start now */
maybe_start_bgworkers();
@@ -2503,6 +2517,16 @@ process_pm_child_exit(void)
continue;
}
+ /* Was it an IO worker? */
+ if (maybe_reap_io_worker(pid))
+ {
+ if (!EXIT_STATUS_0(exitstatus) && !EXIT_STATUS_1(exitstatus))
+ HandleChildCrash(pid, exitstatus, _("io worker"));
+
+ maybe_adjust_io_workers();
+ continue;
+ }
+
/*
* Was it a backend or a background worker?
*/
@@ -2724,6 +2748,7 @@ HandleFatalError(QuitSignalReason reason, bool consider_sigabrt)
case PM_WAIT_XLOG_SHUTDOWN:
case PM_WAIT_XLOG_ARCHIVAL:
case PM_WAIT_CHECKPOINTER:
+ case PM_WAIT_IO_WORKERS:
/*
* NB: Similar code exists in PostmasterStateMachine()'s handling
@@ -2906,20 +2931,21 @@ PostmasterStateMachine(void)
/*
* If we are doing crash recovery or an immediate shutdown then we
- * expect archiver, checkpointer and walsender to exit as well,
- * otherwise not.
+ * expect archiver, checkpointer, io workers and walsender to exit as
+ * well, otherwise not.
*/
if (FatalError || Shutdown >= ImmediateShutdown)
targetMask = btmask_add(targetMask,
B_CHECKPOINTER,
B_ARCHIVER,
+ B_IO_WORKER,
B_WAL_SENDER);
/*
- * Normally walsenders and archiver will continue running; they will
- * be terminated later after writing the checkpoint record. We also
- * let dead-end children to keep running for now. The syslogger
- * process exits last.
+ * Normally archiver, checkpointer, IO workers and walsenders will
+ * continue running; they will be terminated later after writing the
+ * checkpoint record. We also let dead-end children to keep running
+ * for now. The syslogger process exits last.
*
* This assertion checks that we have covered all backend types,
* either by including them in targetMask, or by noting here that they
@@ -2934,12 +2960,13 @@ PostmasterStateMachine(void)
B_LOGGER);
/*
- * Archiver, checkpointer and walsender may or may not be in
- * targetMask already.
+ * Archiver, checkpointer, IO workers, and walsender may or may
+ * not be in targetMask already.
*/
remainMask = btmask_add(remainMask,
B_ARCHIVER,
B_CHECKPOINTER,
+ B_IO_WORKER,
B_WAL_SENDER);
/* these are not real postmaster children */
@@ -3040,11 +3067,25 @@ PostmasterStateMachine(void)
{
/*
* PM_WAIT_XLOG_ARCHIVAL state ends when there are no children other
- * than checkpointer, dead-end children and logger left. There
+ * than checkpointer, io workers and dead-end children left. There
* shouldn't be any regular backends left by now anyway; what we're
* really waiting for is for walsenders and archiver to exit.
*/
- if (CountChildren(btmask_all_except(B_CHECKPOINTER, B_LOGGER, B_DEAD_END_BACKEND)) == 0)
+ if (CountChildren(btmask_all_except(B_CHECKPOINTER, B_IO_WORKER,
+ B_LOGGER, B_DEAD_END_BACKEND)) == 0)
+ {
+ UpdatePMState(PM_WAIT_IO_WORKERS);
+ SignalChildren(SIGUSR2, btmask(B_IO_WORKER));
+ }
+ }
+
+ if (pmState == PM_WAIT_IO_WORKERS)
+ {
+ /*
+ * PM_WAIT_IO_WORKERS state ends when there's only checkpointer and
+ * dead_end children left.
+ */
+ if (io_worker_count == 0)
{
UpdatePMState(PM_WAIT_CHECKPOINTER);
@@ -3172,10 +3213,14 @@ PostmasterStateMachine(void)
/* re-create shared memory and semaphores */
CreateSharedMemoryAndSemaphores();
+ UpdatePMState(PM_STARTUP);
+
+ /* Make sure we can perform I/O while starting up. */
+ maybe_adjust_io_workers();
+
StartupPMChild = StartChildProcess(B_STARTUP);
Assert(StartupPMChild != NULL);
StartupStatus = STARTUP_RUNNING;
- UpdatePMState(PM_STARTUP);
/* crash recovery started, reset SIGKILL flag */
AbortStartTime = 0;
@@ -3199,6 +3244,7 @@ pmstate_name(PMState state)
PM_TOSTR_CASE(PM_WAIT_BACKENDS);
PM_TOSTR_CASE(PM_WAIT_XLOG_SHUTDOWN);
PM_TOSTR_CASE(PM_WAIT_XLOG_ARCHIVAL);
+ PM_TOSTR_CASE(PM_WAIT_IO_WORKERS);
PM_TOSTR_CASE(PM_WAIT_DEAD_END);
PM_TOSTR_CASE(PM_WAIT_CHECKPOINTER);
PM_TOSTR_CASE(PM_NO_CHILDREN);
@@ -4115,6 +4161,7 @@ bgworker_should_start_now(BgWorkerStartTime start_time)
case PM_WAIT_DEAD_END:
case PM_WAIT_XLOG_ARCHIVAL:
case PM_WAIT_XLOG_SHUTDOWN:
+ case PM_WAIT_IO_WORKERS:
case PM_WAIT_BACKENDS:
case PM_STOP_BACKENDS:
break;
@@ -4265,6 +4312,100 @@ maybe_start_bgworkers(void)
}
}
+static bool
+maybe_reap_io_worker(int pid)
+{
+ for (int id = 0; id < MAX_IO_WORKERS; ++id)
+ {
+ if (io_worker_children[id] &&
+ io_worker_children[id]->pid == pid)
+ {
+ ReleasePostmasterChildSlot(io_worker_children[id]);
+
+ --io_worker_count;
+ io_worker_children[id] = NULL;
+ return true;
+ }
+ }
+ return false;
+}
+
+static void
+maybe_adjust_io_workers(void)
+{
+ if (!pgaio_workers_enabled())
+ return;
+
+ /*
+ * If we're in final shutting down state, then we're just waiting for all
+ * processes to exit.
+ */
+ if (pmState >= PM_WAIT_IO_WORKERS)
+ return;
+
+ /* Don't start new workers during an immediate shutdown either. */
+ if (Shutdown >= ImmediateShutdown)
+ return;
+
+ /*
+ * Don't start new workers if we're in the shutdown phase of a crash
+ * restart. But we *do* need to start if we're already starting up again.
+ */
+ if (FatalError && pmState >= PM_STOP_BACKENDS)
+ return;
+
+ Assert(pmState < PM_WAIT_IO_WORKERS);
+
+ /* Not enough running? */
+ while (io_worker_count < io_workers)
+ {
+ PMChild *child;
+ int id;
+
+ /* find unused entry in io_worker_children array */
+ for (id = 0; id < MAX_IO_WORKERS; ++id)
+ {
+ if (io_worker_children[id] == NULL)
+ break;
+ }
+ if (id == MAX_IO_WORKERS)
+ elog(ERROR, "could not find a free IO worker ID");
+
+ /* Try to launch one. */
+ child = StartChildProcess(B_IO_WORKER);
+ if (child != NULL)
+ {
+ io_worker_children[id] = child;
+ ++io_worker_count;
+ }
+ else
+ break; /* XXX try again soon? */
+ }
+
+ /* Too many running? */
+ if (io_worker_count > io_workers)
+ {
+ /* ask the IO worker in the highest slot to exit */
+ for (int id = MAX_IO_WORKERS - 1; id >= 0; --id)
+ {
+ if (io_worker_children[id] != NULL)
+ {
+ kill(io_worker_children[id]->pid, SIGUSR2);
+ break;
+ }
+ }
+ }
+}
+
+void
+assign_io_workers(int newval, void *extra)
+{
+ io_workers = newval;
+ if (!IsUnderPostmaster && pmState > PM_INIT)
+ maybe_adjust_io_workers();
+}
+
+
/*
* When a backend asks to be notified about worker state changes, we
* set a flag in its backend entry. The background worker machinery needs
diff --git a/src/backend/storage/aio/Makefile b/src/backend/storage/aio/Makefile
index 89f821ea7e1..f51c34a37f8 100644
--- a/src/backend/storage/aio/Makefile
+++ b/src/backend/storage/aio/Makefile
@@ -15,6 +15,7 @@ OBJS = \
aio_io.o \
aio_target.o \
method_sync.o \
+ method_worker.o \
read_stream.o
include $(top_srcdir)/src/backend/common.mk
diff --git a/src/backend/storage/aio/meson.build b/src/backend/storage/aio/meson.build
index 2c26089d52e..74f94c6e40b 100644
--- a/src/backend/storage/aio/meson.build
+++ b/src/backend/storage/aio/meson.build
@@ -7,5 +7,6 @@ backend_sources += files(
'aio_io.c',
'aio_target.c',
'method_sync.c',
+ 'method_worker.c',
'read_stream.c',
)
diff --git a/src/backend/storage/aio/method_worker.c b/src/backend/storage/aio/method_worker.c
new file mode 100644
index 00000000000..8ca66895e97
--- /dev/null
+++ b/src/backend/storage/aio/method_worker.c
@@ -0,0 +1,88 @@
+/*-------------------------------------------------------------------------
+ *
+ * method_worker.c
+ * AIO - perform AIO using worker processes
+ *
+ * Portions Copyright (c) 1996-2025, PostgreSQL Global Development Group
+ * Portions Copyright (c) 1994, Regents of the University of California
+ *
+ * IDENTIFICATION
+ * src/backend/storage/aio/method_worker.c
+ *
+ *-------------------------------------------------------------------------
+ */
+
+#include "postgres.h"
+
+#include "libpq/pqsignal.h"
+#include "miscadmin.h"
+#include "postmaster/auxprocess.h"
+#include "postmaster/interrupt.h"
+#include "storage/aio_subsys.h"
+#include "storage/io_worker.h"
+#include "storage/ipc.h"
+#include "storage/latch.h"
+#include "storage/proc.h"
+#include "tcop/tcopprot.h"
+#include "utils/wait_event.h"
+
+
+/* GUCs */
+int io_workers = 3;
+
+
+void
+IoWorkerMain(const void *startup_data, size_t startup_data_len)
+{
+ sigjmp_buf local_sigjmp_buf;
+
+ MyBackendType = B_IO_WORKER;
+ AuxiliaryProcessMainCommon();
+
+ /* TODO review all signals */
+ pqsignal(SIGHUP, SignalHandlerForConfigReload);
+ pqsignal(SIGINT, die); /* to allow manually triggering worker restart */
+
+ /*
+ * Ignore SIGTERM, will get explicit shutdown via SIGUSR2 later in the
+ * shutdown sequence, similar to checkpointer.
+ */
+ pqsignal(SIGTERM, SIG_IGN);
+ /* SIGQUIT handler was already set up by InitPostmasterChild */
+ pqsignal(SIGALRM, SIG_IGN);
+ pqsignal(SIGPIPE, SIG_IGN);
+ pqsignal(SIGUSR1, procsignal_sigusr1_handler);
+ pqsignal(SIGUSR2, SignalHandlerForShutdownRequest);
+ sigprocmask(SIG_SETMASK, &UnBlockSig, NULL);
+
+ /* see PostgresMain() */
+ if (sigsetjmp(local_sigjmp_buf, 1) != 0)
+ {
+ error_context_stack = NULL;
+ HOLD_INTERRUPTS();
+
+ EmitErrorReport();
+
+ proc_exit(1);
+ }
+
+ /* We can now handle ereport(ERROR) */
+ PG_exception_stack = &local_sigjmp_buf;
+
+ while (!ShutdownRequestPending)
+ {
+ WaitLatch(MyLatch, WL_LATCH_SET | WL_EXIT_ON_PM_DEATH, -1,
+ WAIT_EVENT_IO_WORKER_MAIN);
+ ResetLatch(MyLatch);
+ CHECK_FOR_INTERRUPTS();
+ }
+
+ proc_exit(0);
+}
+
+bool
+pgaio_workers_enabled(void)
+{
+ /* placeholder for future commit */
+ return false;
+}
diff --git a/src/backend/tcop/postgres.c b/src/backend/tcop/postgres.c
index f2f75aa0f88..583a0623676 100644
--- a/src/backend/tcop/postgres.c
+++ b/src/backend/tcop/postgres.c
@@ -3315,6 +3315,8 @@ ProcessInterrupts(void)
(errcode(ERRCODE_ADMIN_SHUTDOWN),
errmsg("terminating background worker \"%s\" due to administrator command",
MyBgworkerEntry->bgw_type)));
+ else if (AmIoWorkerProcess())
+ proc_exit(0);
else
ereport(FATAL,
(errcode(ERRCODE_ADMIN_SHUTDOWN),
diff --git a/src/backend/utils/activity/pgstat_backend.c b/src/backend/utils/activity/pgstat_backend.c
index a9343b7b59e..5d2d11ab9e3 100644
--- a/src/backend/utils/activity/pgstat_backend.c
+++ b/src/backend/utils/activity/pgstat_backend.c
@@ -291,6 +291,7 @@ pgstat_tracks_backend_bktype(BackendType bktype)
case B_LOGGER:
case B_BG_WRITER:
case B_CHECKPOINTER:
+ case B_IO_WORKER:
case B_STARTUP:
return false;
diff --git a/src/backend/utils/activity/pgstat_io.c b/src/backend/utils/activity/pgstat_io.c
index ba11545a17f..73c6d2d342e 100644
--- a/src/backend/utils/activity/pgstat_io.c
+++ b/src/backend/utils/activity/pgstat_io.c
@@ -376,6 +376,7 @@ pgstat_tracks_io_bktype(BackendType bktype)
case B_BG_WORKER:
case B_BG_WRITER:
case B_CHECKPOINTER:
+ case B_IO_WORKER:
case B_SLOTSYNC_WORKER:
case B_STANDALONE_BACKEND:
case B_STARTUP:
diff --git a/src/backend/utils/activity/wait_event_names.txt b/src/backend/utils/activity/wait_event_names.txt
index 6f3ca878bd1..d18b788b8a1 100644
--- a/src/backend/utils/activity/wait_event_names.txt
+++ b/src/backend/utils/activity/wait_event_names.txt
@@ -57,6 +57,7 @@ BGWRITER_HIBERNATE "Waiting in background writer process, hibernating."
BGWRITER_MAIN "Waiting in main loop of background writer process."
CHECKPOINTER_MAIN "Waiting in main loop of checkpointer process."
CHECKPOINTER_SHUTDOWN "Waiting for checkpointer process to be terminated."
+IO_WORKER_MAIN "Waiting in main loop of IO Worker process."
LOGICAL_APPLY_MAIN "Waiting in main loop of logical replication apply process."
LOGICAL_LAUNCHER_MAIN "Waiting in main loop of logical replication launcher process."
LOGICAL_PARALLEL_APPLY_MAIN "Waiting in main loop of logical replication parallel apply process."
diff --git a/src/backend/utils/init/miscinit.c b/src/backend/utils/init/miscinit.c
index 0347fc11092..cbca090d2b0 100644
--- a/src/backend/utils/init/miscinit.c
+++ b/src/backend/utils/init/miscinit.c
@@ -293,6 +293,9 @@ GetBackendTypeDesc(BackendType backendType)
case B_CHECKPOINTER:
backendDesc = gettext_noop("checkpointer");
break;
+ case B_IO_WORKER:
+ backendDesc = "io worker";
+ break;
case B_LOGGER:
backendDesc = gettext_noop("logger");
break;
--
2.48.1.76.g4e746b1a31.dirty
[text/x-diff] v2.5-0004-aio-Add-worker-method.patch (21.2K, ../../vz5i2x2wkjjpp6l2z4g4l3umpxktrxsejgdwefi7n6kzw666h3@rnhlvlpadig7/5-v2.5-0004-aio-Add-worker-method.patch)
download | inline diff:
From da4a66c46ea377109697b0fb45ebff27b23951f5 Mon Sep 17 00:00:00 2001
From: Andres Freund <andres@anarazel.de>
Date: Fri, 31 Jan 2025 13:46:35 -0500
Subject: [PATCH v2.5 04/30] aio: Add worker method
---
src/include/storage/aio.h | 5 +-
src/include/storage/aio_internal.h | 1 +
src/include/storage/lwlocklist.h | 1 +
src/backend/storage/aio/aio.c | 2 +
src/backend/storage/aio/aio_init.c | 9 +
src/backend/storage/aio/method_worker.c | 435 +++++++++++++++++-
.../utils/activity/wait_event_names.txt | 1 +
src/backend/utils/misc/guc_tables.c | 13 +
src/backend/utils/misc/postgresql.conf.sample | 3 +-
doc/src/sgml/config.sgml | 21 +
src/tools/pgindent/typedefs.list | 3 +
11 files changed, 483 insertions(+), 11 deletions(-)
diff --git a/src/include/storage/aio.h b/src/include/storage/aio.h
index 747ad83d037..c5ffd9b861a 100644
--- a/src/include/storage/aio.h
+++ b/src/include/storage/aio.h
@@ -23,10 +23,11 @@
typedef enum IoMethod
{
IOMETHOD_SYNC = 0,
+ IOMETHOD_WORKER,
} IoMethod;
-/* We'll default to synchronous execution. */
-#define DEFAULT_IO_METHOD IOMETHOD_SYNC
+/* We'll default to worker based execution. */
+#define DEFAULT_IO_METHOD IOMETHOD_WORKER
/*
diff --git a/src/include/storage/aio_internal.h b/src/include/storage/aio_internal.h
index 9d9cda80ccb..ba086b91be3 100644
--- a/src/include/storage/aio_internal.h
+++ b/src/include/storage/aio_internal.h
@@ -341,6 +341,7 @@ extern PgAioHandle *pgaio_inj_io_get(void);
/* Declarations for the tables of function pointers exposed by each IO method. */
extern PGDLLIMPORT const IoMethodOps pgaio_sync_ops;
+extern PGDLLIMPORT const IoMethodOps pgaio_worker_ops;
extern PGDLLIMPORT const IoMethodOps *pgaio_method_ops;
extern PGDLLIMPORT PgAioCtl *pgaio_ctl;
diff --git a/src/include/storage/lwlocklist.h b/src/include/storage/lwlocklist.h
index cf565452382..932024b1b0b 100644
--- a/src/include/storage/lwlocklist.h
+++ b/src/include/storage/lwlocklist.h
@@ -83,3 +83,4 @@ PG_LWLOCK(49, WALSummarizer)
PG_LWLOCK(50, DSMRegistry)
PG_LWLOCK(51, InjectionPoint)
PG_LWLOCK(52, SerialControl)
+PG_LWLOCK(53, AioWorkerSubmissionQueue)
diff --git a/src/backend/storage/aio/aio.c b/src/backend/storage/aio/aio.c
index b8cfe012c4d..d6d21c44199 100644
--- a/src/backend/storage/aio/aio.c
+++ b/src/backend/storage/aio/aio.c
@@ -63,6 +63,7 @@ static void pgaio_io_wait(PgAioHandle *ioh, uint64 ref_generation);
/* Options for io_method. */
const struct config_enum_entry io_method_options[] = {
{"sync", IOMETHOD_SYNC, false},
+ {"worker", IOMETHOD_WORKER, false},
{NULL, 0, false}
};
@@ -79,6 +80,7 @@ PgAioBackend *pgaio_my_backend;
static const IoMethodOps *const pgaio_method_ops_table[] = {
[IOMETHOD_SYNC] = &pgaio_sync_ops,
+ [IOMETHOD_WORKER] = &pgaio_worker_ops,
};
/* callbacks for the configured io_method, set by assign_io_method */
diff --git a/src/backend/storage/aio/aio_init.c b/src/backend/storage/aio/aio_init.c
index 8e1162b09dc..2205658cd9a 100644
--- a/src/backend/storage/aio/aio_init.c
+++ b/src/backend/storage/aio/aio_init.c
@@ -18,6 +18,7 @@
#include "storage/aio.h"
#include "storage/aio_internal.h"
#include "storage/aio_subsys.h"
+#include "storage/io_worker.h"
#include "storage/ipc.h"
#include "storage/proc.h"
#include "storage/shmem.h"
@@ -39,6 +40,11 @@ AioCtlShmemSize(void)
static uint32
AioProcs(void)
{
+ /*
+ * While AIO workers don't need their own AIO context, we can't currently
+ * guarantee nothing gets assigned to the a ProcNumber for an IO worker if
+ * we just subtracted MAX_IO_WORKERS.
+ */
return MaxBackends + NUM_AUXILIARY_PROCS;
}
@@ -211,6 +217,9 @@ pgaio_init_backend(void)
/* shouldn't be initialized twice */
Assert(!pgaio_my_backend);
+ if (MyBackendType == B_IO_WORKER)
+ return;
+
if (MyProc == NULL || MyProcNumber >= AioProcs())
elog(ERROR, "aio requires a normal PGPROC");
diff --git a/src/backend/storage/aio/method_worker.c b/src/backend/storage/aio/method_worker.c
index 8ca66895e97..5034ec8923f 100644
--- a/src/backend/storage/aio/method_worker.c
+++ b/src/backend/storage/aio/method_worker.c
@@ -3,6 +3,21 @@
* method_worker.c
* AIO - perform AIO using worker processes
*
+ * Worker processes consume IOs from a shared memory submission queue, run
+ * traditional synchronous system calls, and perform the shared completion
+ * handling immediately. Client code submits most requests by pushing IOs
+ * into the submission queue, and waits (if necessary) using condition
+ * variables. Some IOs cannot be performed in another process due to lack of
+ * infrastructure for reopening the file, and must processed synchronously by
+ * the client code when submitted.
+ *
+ * So that the submitter can make just one system call when submitting a batch
+ * of IOs, wakeups "fan out"; each woken backend can wake two more. XXX This
+ * could be improved by using futexes instead of latches to wake N waiters.
+ *
+ * This method of AIO is available in all builds on all operating systems, and
+ * is the default.
+ *
* Portions Copyright (c) 1996-2025, PostgreSQL Global Development Group
* Portions Copyright (c) 1994, Regents of the University of California
*
@@ -16,30 +31,329 @@
#include "libpq/pqsignal.h"
#include "miscadmin.h"
+#include "port/pg_bitutils.h"
#include "postmaster/auxprocess.h"
#include "postmaster/interrupt.h"
+#include "storage/aio.h"
+#include "storage/aio_internal.h"
#include "storage/aio_subsys.h"
#include "storage/io_worker.h"
#include "storage/ipc.h"
#include "storage/latch.h"
#include "storage/proc.h"
#include "tcop/tcopprot.h"
+#include "utils/ps_status.h"
#include "utils/wait_event.h"
+/* How many workers should each worker wake up if needed? */
+#define IO_WORKER_WAKEUP_FANOUT 2
+
+
+typedef struct AioWorkerSubmissionQueue
+{
+ uint32 size;
+ uint32 mask;
+ uint32 head;
+ uint32 tail;
+ uint32 ios[FLEXIBLE_ARRAY_MEMBER];
+} AioWorkerSubmissionQueue;
+
+typedef struct AioWorkerSlot
+{
+ Latch *latch;
+ bool in_use;
+} AioWorkerSlot;
+
+typedef struct AioWorkerControl
+{
+ uint64 idle_worker_mask;
+ AioWorkerSlot workers[FLEXIBLE_ARRAY_MEMBER];
+} AioWorkerControl;
+
+
+static size_t pgaio_worker_shmem_size(void);
+static void pgaio_worker_shmem_init(bool first_time);
+
+static bool pgaio_worker_needs_synchronous_execution(PgAioHandle *ioh);
+static int pgaio_worker_submit(uint16 num_staged_ios, PgAioHandle **staged_ios);
+
+
+const IoMethodOps pgaio_worker_ops = {
+ .shmem_size = pgaio_worker_shmem_size,
+ .shmem_init = pgaio_worker_shmem_init,
+
+ .needs_synchronous_execution = pgaio_worker_needs_synchronous_execution,
+ .submit = pgaio_worker_submit,
+};
+
+
/* GUCs */
int io_workers = 3;
+static int io_worker_queue_size = 64;
+static int MyIoWorkerId;
+static AioWorkerSubmissionQueue *io_worker_submission_queue;
+static AioWorkerControl *io_worker_control;
+
+
+static size_t
+pgaio_worker_shmem_size(void)
+{
+ return
+ offsetof(AioWorkerSubmissionQueue, ios) +
+ sizeof(uint32) * MAX_IO_WORKERS * io_max_concurrency +
+ offsetof(AioWorkerControl, workers) +
+ sizeof(AioWorkerSlot) * MAX_IO_WORKERS;
+}
+
+static void
+pgaio_worker_shmem_init(bool first_time)
+{
+ bool found;
+ int size;
+
+ /* Round size up to next power of two so we can make a mask. */
+ size = pg_nextpower2_32(io_worker_queue_size);
+
+ io_worker_submission_queue =
+ ShmemInitStruct("AioWorkerSubmissionQueue",
+ offsetof(AioWorkerSubmissionQueue, ios) +
+ sizeof(uint32) * size,
+ &found);
+ if (!found)
+ {
+ io_worker_submission_queue->size = size;
+ io_worker_submission_queue->head = 0;
+ io_worker_submission_queue->tail = 0;
+ }
+
+ io_worker_control =
+ ShmemInitStruct("AioWorkerControl",
+ offsetof(AioWorkerControl, workers) +
+ sizeof(AioWorkerSlot) * io_workers,
+ &found);
+ if (!found)
+ {
+ io_worker_control->idle_worker_mask = 0;
+ for (int i = 0; i < io_workers; ++i)
+ {
+ io_worker_control->workers[i].latch = NULL;
+ io_worker_control->workers[i].in_use = false;
+ }
+ }
+}
+
+static int
+pgaio_choose_idle_worker(void)
+{
+ int worker;
+
+ if (io_worker_control->idle_worker_mask == 0)
+ return -1;
+
+ /* Find the lowest bit position, and clear it. */
+ worker = pg_rightmost_one_pos64(io_worker_control->idle_worker_mask);
+ io_worker_control->idle_worker_mask &= ~(UINT64_C(1) << worker);
+
+ return worker;
+}
+
+static bool
+pgaio_worker_submission_queue_insert(PgAioHandle *ioh)
+{
+ AioWorkerSubmissionQueue *queue;
+ uint32 new_head;
+
+ queue = io_worker_submission_queue;
+ new_head = (queue->head + 1) & (queue->size - 1);
+ if (new_head == queue->tail)
+ {
+ pgaio_debug(DEBUG1, "io queue is full, at %u elements",
+ io_worker_submission_queue->size);
+ return false; /* full */
+ }
+
+ queue->ios[queue->head] = pgaio_io_get_id(ioh);
+ queue->head = new_head;
+
+ return true;
+}
+
+static uint32
+pgaio_worker_submission_queue_consume(void)
+{
+ AioWorkerSubmissionQueue *queue;
+ uint32 result;
+
+ queue = io_worker_submission_queue;
+ if (queue->tail == queue->head)
+ return UINT32_MAX; /* empty */
+
+ result = queue->ios[queue->tail];
+ queue->tail = (queue->tail + 1) & (queue->size - 1);
+
+ return result;
+}
+
+static uint32
+pgaio_worker_submission_queue_depth(void)
+{
+ uint32 head;
+ uint32 tail;
+
+ head = io_worker_submission_queue->head;
+ tail = io_worker_submission_queue->tail;
+
+ if (tail > head)
+ head += io_worker_submission_queue->size;
+
+ Assert(head >= tail);
+
+ return head - tail;
+}
+
+static bool
+pgaio_worker_needs_synchronous_execution(PgAioHandle *ioh)
+{
+ return
+ !IsUnderPostmaster
+ || ioh->flags & PGAIO_HF_REFERENCES_LOCAL
+ || !pgaio_io_can_reopen(ioh);
+}
+
+static void
+pgaio_worker_submit_internal(int nios, PgAioHandle *ios[])
+{
+ PgAioHandle *synchronous_ios[PGAIO_SUBMIT_BATCH_SIZE];
+ int nsync = 0;
+ Latch *wakeup = NULL;
+ int worker;
+
+ Assert(nios <= PGAIO_SUBMIT_BATCH_SIZE);
+
+ LWLockAcquire(AioWorkerSubmissionQueueLock, LW_EXCLUSIVE);
+ for (int i = 0; i < nios; ++i)
+ {
+ Assert(!pgaio_worker_needs_synchronous_execution(ios[i]));
+ if (!pgaio_worker_submission_queue_insert(ios[i]))
+ {
+ /*
+ * We'll do it synchronously, but only after we've sent as many as
+ * we can to workers, to maximize concurrency.
+ */
+ synchronous_ios[nsync++] = ios[i];
+ continue;
+ }
+
+ if (wakeup == NULL)
+ {
+ /* Choose an idle worker to wake up if we haven't already. */
+ worker = pgaio_choose_idle_worker();
+ if (worker >= 0)
+ wakeup = io_worker_control->workers[worker].latch;
+
+ pgaio_debug_io(DEBUG4, ios[i],
+ "choosing worker %d",
+ worker);
+ }
+ }
+ LWLockRelease(AioWorkerSubmissionQueueLock);
+
+ if (wakeup)
+ SetLatch(wakeup);
+
+ /* Run whatever is left synchronously. */
+ if (nsync > 0)
+ {
+ for (int i = 0; i < nsync; ++i)
+ {
+ pgaio_io_perform_synchronously(synchronous_ios[i]);
+ }
+ }
+}
+
+static int
+pgaio_worker_submit(uint16 num_staged_ios, PgAioHandle **staged_ios)
+{
+ for (int i = 0; i < num_staged_ios; i++)
+ {
+ PgAioHandle *ioh = staged_ios[i];
+
+ pgaio_io_prepare_submit(ioh);
+ }
+
+ pgaio_worker_submit_internal(num_staged_ios, staged_ios);
+
+ return num_staged_ios;
+}
+
+/*
+ * on_shmem_exit() callback that releases the worker's slot in
+ * io_worker_control.
+ */
+static void
+pgaio_worker_die(int code, Datum arg)
+{
+ LWLockAcquire(AioWorkerSubmissionQueueLock, LW_EXCLUSIVE);
+ Assert(io_worker_control->workers[MyIoWorkerId].in_use);
+ Assert(io_worker_control->workers[MyIoWorkerId].latch == MyLatch);
+
+ io_worker_control->workers[MyIoWorkerId].in_use = false;
+ io_worker_control->workers[MyIoWorkerId].latch = NULL;
+ LWLockRelease(AioWorkerSubmissionQueueLock);
+}
+
+/*
+ * Register the worker in shared memory, assign MyWorkerId and register a
+ * shutdown callback to release registration.
+ */
+static void
+pgaio_worker_register(void)
+{
+ MyIoWorkerId = -1;
+
+ /*
+ * XXX: This could do with more fine-grained locking. But it's also not
+ * very common for the number of workers to change at the moment...
+ */
+ LWLockAcquire(AioWorkerSubmissionQueueLock, LW_EXCLUSIVE);
+
+ for (int i = 0; i < io_workers; ++i)
+ {
+ if (!io_worker_control->workers[i].in_use)
+ {
+ Assert(io_worker_control->workers[i].latch == NULL);
+ io_worker_control->workers[i].in_use = true;
+ MyIoWorkerId = i;
+ break;
+ }
+ else
+ Assert(io_worker_control->workers[i].latch != NULL);
+ }
+
+ if (MyIoWorkerId == -1)
+ elog(ERROR, "couldn't find a free worker slot");
+
+ io_worker_control->idle_worker_mask |= (UINT64_C(1) << MyIoWorkerId);
+ io_worker_control->workers[MyIoWorkerId].latch = MyLatch;
+ LWLockRelease(AioWorkerSubmissionQueueLock);
+
+ on_shmem_exit(pgaio_worker_die, 0);
+}
+
void
IoWorkerMain(const void *startup_data, size_t startup_data_len)
{
sigjmp_buf local_sigjmp_buf;
+ PgAioHandle *volatile error_ioh = NULL;
+ volatile int error_errno = 0;
+ char cmd[128];
MyBackendType = B_IO_WORKER;
AuxiliaryProcessMainCommon();
- /* TODO review all signals */
pqsignal(SIGHUP, SignalHandlerForConfigReload);
pqsignal(SIGINT, die); /* to allow manually triggering worker restart */
@@ -53,7 +367,12 @@ IoWorkerMain(const void *startup_data, size_t startup_data_len)
pqsignal(SIGPIPE, SIG_IGN);
pqsignal(SIGUSR1, procsignal_sigusr1_handler);
pqsignal(SIGUSR2, SignalHandlerForShutdownRequest);
- sigprocmask(SIG_SETMASK, &UnBlockSig, NULL);
+
+ /* also registers a shutdown callback to unregister */
+ pgaio_worker_register();
+
+ sprintf(cmd, "io worker: %d", MyIoWorkerId);
+ set_ps_display(cmd);
/* see PostgresMain() */
if (sigsetjmp(local_sigjmp_buf, 1) != 0)
@@ -61,6 +380,27 @@ IoWorkerMain(const void *startup_data, size_t startup_data_len)
error_context_stack = NULL;
HOLD_INTERRUPTS();
+ /*
+ * In the - very unlikely - case that the IO failed in a way that
+ * raises an error we need to mark the IO as failed.
+ *
+ * Need to do just enough error recovery so that we can mark the IO as
+ * failed and then exit (postmaster will start a new worker).
+ */
+ LWLockReleaseAll();
+
+ if (error_ioh != NULL)
+ {
+ /* should never fail without setting error_errno */
+ Assert(error_errno != 0);
+
+ errno = error_errno;
+
+ START_CRIT_SECTION();
+ pgaio_io_process_completion(error_ioh, -error_errno);
+ END_CRIT_SECTION();
+ }
+
EmitErrorReport();
proc_exit(1);
@@ -69,12 +409,92 @@ IoWorkerMain(const void *startup_data, size_t startup_data_len)
/* We can now handle ereport(ERROR) */
PG_exception_stack = &local_sigjmp_buf;
+ sigprocmask(SIG_SETMASK, &UnBlockSig, NULL);
+
while (!ShutdownRequestPending)
{
- WaitLatch(MyLatch, WL_LATCH_SET | WL_EXIT_ON_PM_DEATH, -1,
- WAIT_EVENT_IO_WORKER_MAIN);
- ResetLatch(MyLatch);
- CHECK_FOR_INTERRUPTS();
+ uint32 io_index;
+ Latch *latches[IO_WORKER_WAKEUP_FANOUT];
+ int nlatches = 0;
+ int nwakeups = 0;
+ int worker;
+
+ /* Try to get a job to do. */
+ LWLockAcquire(AioWorkerSubmissionQueueLock, LW_EXCLUSIVE);
+ if ((io_index = pgaio_worker_submission_queue_consume()) == UINT32_MAX)
+ {
+ /*
+ * Nothing to do. Mark self idle.
+ *
+ * XXX: Invent some kind of back pressure to reduce useless
+ * wakeups?
+ */
+ io_worker_control->idle_worker_mask |= (UINT64_C(1) << MyIoWorkerId);
+ }
+ else
+ {
+ /* Got one. Clear idle flag. */
+ io_worker_control->idle_worker_mask &= ~(UINT64_C(1) << MyIoWorkerId);
+
+ /* See if we can wake up some peers. */
+ nwakeups = Min(pgaio_worker_submission_queue_depth(),
+ IO_WORKER_WAKEUP_FANOUT);
+ for (int i = 0; i < nwakeups; ++i)
+ {
+ if ((worker = pgaio_choose_idle_worker()) < 0)
+ break;
+ latches[nlatches++] = io_worker_control->workers[worker].latch;
+ }
+ }
+ LWLockRelease(AioWorkerSubmissionQueueLock);
+
+ for (int i = 0; i < nlatches; ++i)
+ SetLatch(latches[i]);
+
+ if (io_index != UINT32_MAX)
+ {
+ PgAioHandle *ioh = NULL;
+
+ ioh = &pgaio_ctl->io_handles[io_index];
+ error_ioh = ioh;
+
+ pgaio_debug_io(DEBUG4, ioh,
+ "worker %d processing IO",
+ MyIoWorkerId);
+
+ /*
+ * It's very unlikely, but possible, that reopen fails. E.g. due
+ * to memory allocations failing or file permissions changing or
+ * such. In that case we need to fail the IO.
+ *
+ * There's not really a good errno we can report here.
+ */
+ error_errno = ENOENT;
+ pgaio_io_reopen(ioh);
+
+ /*
+ * To be able to exercise the reopen-fails path, allow injection
+ * points to trigger a failure at this point.
+ */
+ pgaio_io_call_inj(ioh, "AIO_WORKER_AFTER_REOPEN");
+
+ error_errno = 0;
+ error_ioh = NULL;
+
+ /*
+ * We don't expect this to ever fail, no need to keep error_ioh
+ * around. pgaio_io_perform_synchronously() contains a critical
+ * section.
+ */
+ pgaio_io_perform_synchronously(ioh);
+ }
+ else
+ {
+ WaitLatch(MyLatch, WL_LATCH_SET | WL_EXIT_ON_PM_DEATH, -1,
+ WAIT_EVENT_IO_WORKER_MAIN);
+ ResetLatch(MyLatch);
+ CHECK_FOR_INTERRUPTS();
+ }
}
proc_exit(0);
@@ -83,6 +503,5 @@ IoWorkerMain(const void *startup_data, size_t startup_data_len)
bool
pgaio_workers_enabled(void)
{
- /* placeholder for future commit */
- return false;
+ return io_method == IOMETHOD_WORKER;
}
diff --git a/src/backend/utils/activity/wait_event_names.txt b/src/backend/utils/activity/wait_event_names.txt
index d18b788b8a1..cb977b049d8 100644
--- a/src/backend/utils/activity/wait_event_names.txt
+++ b/src/backend/utils/activity/wait_event_names.txt
@@ -348,6 +348,7 @@ WALSummarizer "Waiting to read or update WAL summarization state."
DSMRegistry "Waiting to read or update the dynamic shared memory registry."
InjectionPoint "Waiting to read or update information related to injection points."
SerialControl "Waiting to read or update shared <filename>pg_serial</filename> state."
+AioWorkerSubmissionQueue "Waiting to access AIO worker submission queue."
#
# END OF PREDEFINED LWLOCKS (DO NOT CHANGE THIS LINE)
diff --git a/src/backend/utils/misc/guc_tables.c b/src/backend/utils/misc/guc_tables.c
index 28dcc465762..4317cfc9d2f 100644
--- a/src/backend/utils/misc/guc_tables.c
+++ b/src/backend/utils/misc/guc_tables.c
@@ -75,6 +75,7 @@
#include "storage/aio.h"
#include "storage/bufmgr.h"
#include "storage/bufpage.h"
+#include "storage/io_worker.h"
#include "storage/large_object.h"
#include "storage/pg_shmem.h"
#include "storage/predicate.h"
@@ -3267,6 +3268,18 @@ struct config_int ConfigureNamesInt[] =
check_io_max_concurrency, NULL, NULL
},
+ {
+ {"io_workers",
+ PGC_SIGHUP,
+ RESOURCES_IO,
+ gettext_noop("Number of IO worker processes, for io_method=worker."),
+ NULL,
+ },
+ &io_workers,
+ 3, 1, MAX_IO_WORKERS,
+ NULL, assign_io_workers, NULL
+ },
+
{
{"backend_flush_after", PGC_USERSET, RESOURCES_IO,
gettext_noop("Number of pages after which previously performed writes are flushed to disk."),
diff --git a/src/backend/utils/misc/postgresql.conf.sample b/src/backend/utils/misc/postgresql.conf.sample
index a31c9963ce4..1058726285f 100644
--- a/src/backend/utils/misc/postgresql.conf.sample
+++ b/src/backend/utils/misc/postgresql.conf.sample
@@ -202,11 +202,12 @@
#maintenance_io_concurrency = 10 # 1-1000; 0 disables prefetching
#io_combine_limit = 128kB # usually 1-32 blocks (depends on OS)
-#io_method = sync # sync (change requires restart)
+#io_method = worker # worker, sync (change requires restart)
#io_max_concurrency = -1 # Max number of IOs that one process
# can execute simultaneously
# -1 sets based on shared_buffers
# (change requires restart)
+#io_workers = 3 # 1-32;
# - Worker Processes -
diff --git a/doc/src/sgml/config.sgml b/doc/src/sgml/config.sgml
index 7d5482f31cd..8b078f476d1 100644
--- a/doc/src/sgml/config.sgml
+++ b/doc/src/sgml/config.sgml
@@ -2682,6 +2682,11 @@ include_dir 'conf.d'
Selects the method for executing asynchronous I/O.
Possible values are:
<itemizedlist>
+ <listitem>
+ <para>
+ <literal>worker</literal> (execute asynchronous I/O using worker processes)
+ </para>
+ </listitem>
<listitem>
<para>
<literal>sync</literal> (execute asynchronous I/O synchronously)
@@ -2695,6 +2700,22 @@ include_dir 'conf.d'
</listitem>
</varlistentry>
+ <varlistentry id="guc-io-workers" xreflabel="io_workers">
+ <term><varname>io_workers</varname> (<type>int</type>)
+ <indexterm>
+ <primary><varname>io_workers</varname> configuration parameter</primary>
+ </indexterm>
+ </term>
+ <listitem>
+ <para>
+ Selects the number of I/O worker processes to use. The default is 3.
+ </para>
+ <para>
+ Only has an effect if <xref linkend="guc-max-wal-senders"/> is set to
+ <literal>worker</literal>.
+ </para>
+ </listitem>
+ </varlistentry>
</variablelist>
</sect2>
diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list
index 4405c2409d1..2db01bc4617 100644
--- a/src/tools/pgindent/typedefs.list
+++ b/src/tools/pgindent/typedefs.list
@@ -55,6 +55,9 @@ AggStrategy
AggTransInfo
Aggref
AggregateInstrumentation
+AioWorkerControl
+AioWorkerSlot
+AioWorkerSubmissionQueue
AlenState
Alias
AllocBlock
--
2.48.1.76.g4e746b1a31.dirty
[text/x-diff] v2.5-0005-aio-Add-liburing-dependency.patch (13.2K, ../../vz5i2x2wkjjpp6l2z4g4l3umpxktrxsejgdwefi7n6kzw666h3@rnhlvlpadig7/6-v2.5-0005-aio-Add-liburing-dependency.patch)
download | inline diff:
From 3236bbbe9f6ce5e9c00ca7e40ab46326fd3841bd Mon Sep 17 00:00:00 2001
From: Andres Freund <andres@anarazel.de>
Date: Thu, 20 Feb 2025 17:56:05 -0500
Subject: [PATCH v2.5 05/30] aio: Add liburing dependency
Not yet used.
Author:
Reviewed-by:
Discussion: https://postgr.es/m/
Backpatch:
---
meson.build | 14 ++++
meson_options.txt | 3 +
configure.ac | 11 +++
src/makefiles/meson.build | 3 +
src/include/pg_config.h.in | 3 +
src/backend/Makefile | 7 +-
doc/src/sgml/installation.sgml | 34 ++++++++
configure | 138 +++++++++++++++++++++++++++++++++
.cirrus.tasks.yml | 1 +
src/Makefile.global.in | 4 +
10 files changed, 215 insertions(+), 3 deletions(-)
diff --git a/meson.build b/meson.build
index 13c13748e5d..bed36153c7b 100644
--- a/meson.build
+++ b/meson.build
@@ -950,6 +950,18 @@ endif
+###############################################################
+# Library: liburing
+###############################################################
+
+liburingopt = get_option('liburing')
+liburing = dependency('liburing', required: liburingopt)
+if liburing.found()
+ cdata.set('USE_LIBURING', 1)
+endif
+
+
+
###############################################################
# Library: libxml
###############################################################
@@ -3168,6 +3180,7 @@ backend_both_deps += [
icu_i18n,
ldap,
libintl,
+ liburing,
libxml,
lz4,
pam,
@@ -3823,6 +3836,7 @@ if meson.version().version_compare('>=0.57')
'icu': icu,
'ldap': ldap,
'libcurl': libcurl,
+ 'liburing': liburing,
'libxml': libxml,
'libxslt': libxslt,
'llvm': llvm,
diff --git a/meson_options.txt b/meson_options.txt
index 702c4517145..dd7126da3a7 100644
--- a/meson_options.txt
+++ b/meson_options.txt
@@ -106,6 +106,9 @@ option('libcurl', type : 'feature', value: 'auto',
option('libedit_preferred', type: 'boolean', value: false,
description: 'Prefer BSD Libedit over GNU Readline')
+option('liburing', type : 'feature', value: 'auto',
+ description: 'io_uring support, for asynchronous I/O')
+
option('libxml', type: 'feature', value: 'auto',
description: 'XML support')
diff --git a/configure.ac b/configure.ac
index b6d02f5ecc7..3e2d0dc59c0 100644
--- a/configure.ac
+++ b/configure.ac
@@ -975,6 +975,14 @@ AC_SUBST(with_readline)
PGAC_ARG_BOOL(with, libedit-preferred, no,
[prefer BSD Libedit over GNU Readline])
+#
+# liburing
+#
+AC_MSG_CHECKING([whether to build with liburing support])
+PGAC_ARG_BOOL(with, liburing, no, [io_uring support, for asynchronous I/O],
+ [AC_DEFINE([USE_LIBURING], 1, [Define to build with io_uring support. (--with-liburing)])])
+AC_MSG_RESULT([$with_liburing])
+AC_SUBST(with_liburing)
#
# UUID library
@@ -1463,6 +1471,9 @@ elif test "$with_uuid" = ossp ; then
fi
AC_SUBST(UUID_LIBS)
+if test "$with_liburing" = yes; then
+ PKG_CHECK_MODULES(LIBURING, liburing)
+fi
##
## Header files
diff --git a/src/makefiles/meson.build b/src/makefiles/meson.build
index 60e13d50235..46d8da070e8 100644
--- a/src/makefiles/meson.build
+++ b/src/makefiles/meson.build
@@ -199,6 +199,8 @@ pgxs_empty = [
'PTHREAD_CFLAGS', 'PTHREAD_LIBS',
'ICU_LIBS',
+
+ 'LIBURING_CFLAGS', 'LIBURING_LIBS',
]
if host_system == 'windows' and cc.get_argument_syntax() != 'msvc'
@@ -230,6 +232,7 @@ pgxs_deps = {
'icu': icu,
'ldap': ldap,
'libcurl': libcurl,
+ 'liburing': liburing,
'libxml': libxml,
'libxslt': libxslt,
'llvm': llvm,
diff --git a/src/include/pg_config.h.in b/src/include/pg_config.h.in
index db6454090d2..f2422241133 100644
--- a/src/include/pg_config.h.in
+++ b/src/include/pg_config.h.in
@@ -672,6 +672,9 @@
/* Define to 1 to build with libcurl support. (--with-libcurl) */
#undef USE_LIBCURL
+/* Define to build with io_uring support. (--with-liburing) */
+#undef USE_LIBURING
+
/* Define to 1 to build with XML support. (--with-libxml) */
#undef USE_LIBXML
diff --git a/src/backend/Makefile b/src/backend/Makefile
index 42d4a28e5aa..7344c8c7f5c 100644
--- a/src/backend/Makefile
+++ b/src/backend/Makefile
@@ -43,9 +43,10 @@ OBJS = \
$(top_builddir)/src/common/libpgcommon_srv.a \
$(top_builddir)/src/port/libpgport_srv.a
-# We put libpgport and libpgcommon into OBJS, so remove it from LIBS; also add
-# libldap and ICU
-LIBS := $(filter-out -lpgport -lpgcommon, $(LIBS)) $(LDAP_LIBS_BE) $(ICU_LIBS)
+# We put libpgport and libpgcommon into OBJS, so remove it from LIBS.
+LIBS := $(filter-out -lpgport -lpgcommon, $(LIBS))
+# The backend conditionally needs libraries that most executables don't need.
+LIBS += $(LDAP_LIBS_BE) $(ICU_LIBS) $(LIBURING_LIBS)
# The backend doesn't need everything that's in LIBS, however
LIBS := $(filter-out -lreadline -ledit -ltermcap -lncurses -lcurses, $(LIBS))
diff --git a/doc/src/sgml/installation.sgml b/doc/src/sgml/installation.sgml
index e076cefa3b9..cc28f041330 100644
--- a/doc/src/sgml/installation.sgml
+++ b/doc/src/sgml/installation.sgml
@@ -1156,6 +1156,24 @@ build-postgresql:
</listitem>
</varlistentry>
+ <varlistentry id="configure-option-with-liburing">
+ <term><option>--with-liburing</option></term>
+ <listitem>
+ <para>
+ Build with liburing, enabling io_uring support for asynchronous I/O.
+ </para>
+ <para>
+ To detect the required compiler and linker options, PostgreSQL will
+ query <command>pkg-config</command>.
+ </para>
+ <para>
+ To use a liburing installation that is in an unusual location, you
+ can set <command>pkg-config</command>-related environment
+ variables (see its documentation).
+ </para>
+ </listitem>
+ </varlistentry>
+
<varlistentry id="configure-option-with-libxml">
<term><option>--with-libxml</option></term>
<listitem>
@@ -2611,6 +2629,22 @@ ninja install
</listitem>
</varlistentry>
+ <varlistentry id="configure-with-liburing-meson">
+ <term><option>-Dliburing={ auto | enabled | disabled }</option></term>
+ <listitem>
+ <para>
+ Build with liburing, enabling io_uring support for asynchronous I/O.
+ Defaults to auto.
+ </para>
+
+ <para>
+ To use a liburing installation that is in an unusual location, you
+ can set <command>pkg-config</command>-related environment
+ variables (see its documentation).
+ </para>
+ </listitem>
+ </varlistentry>
+
<varlistentry id="configure-with-libxml-meson">
<term><option>-Dlibxml={ auto | enabled | disabled }</option></term>
<listitem>
diff --git a/configure b/configure
index 93fddd69981..8ffcdfc2e1c 100755
--- a/configure
+++ b/configure
@@ -651,6 +651,8 @@ LIBOBJS
OPENSSL
ZSTD
LZ4
+LIBURING_LIBS
+LIBURING_CFLAGS
UUID_LIBS
LDAP_LIBS_BE
LDAP_LIBS_FE
@@ -712,6 +714,7 @@ LIBCURL_LIBS
LIBCURL_CFLAGS
with_libcurl
with_uuid
+with_liburing
with_readline
with_systemd
with_selinux
@@ -865,6 +868,7 @@ with_selinux
with_systemd
with_readline
with_libedit_preferred
+with_liburing
with_uuid
with_ossp_uuid
with_libcurl
@@ -911,6 +915,8 @@ LDFLAGS_EX
LDFLAGS_SL
PERL
PYTHON
+LIBURING_CFLAGS
+LIBURING_LIBS
MSGFMT
TCLSH'
@@ -1578,6 +1584,7 @@ Optional Packages:
--without-readline do not use GNU Readline nor BSD Libedit for editing
--with-libedit-preferred
prefer BSD Libedit over GNU Readline
+ --with-liburing io_uring support, for asynchronous I/O
--with-uuid=LIB build contrib/uuid-ossp using LIB (bsd,e2fs,ossp)
--with-ossp-uuid obsolete spelling of --with-uuid=ossp
--with-libcurl build with libcurl support
@@ -1629,6 +1636,10 @@ Some influential environment variables:
LDFLAGS_SL extra linker flags for linking shared libraries only
PERL Perl program
PYTHON Python program
+ LIBURING_CFLAGS
+ C compiler flags for LIBURING, overriding pkg-config
+ LIBURING_LIBS
+ linker flags for LIBURING, overriding pkg-config
MSGFMT msgfmt program for NLS
TCLSH Tcl interpreter program (tclsh)
@@ -8692,6 +8703,40 @@ fi
+#
+# liburing
+#
+{ $as_echo "$as_me:${as_lineno-$LINENO}: checking whether to build with liburing support" >&5
+$as_echo_n "checking whether to build with liburing support... " >&6; }
+
+
+
+# Check whether --with-liburing was given.
+if test "${with_liburing+set}" = set; then :
+ withval=$with_liburing;
+ case $withval in
+ yes)
+
+$as_echo "#define USE_LIBURING 1" >>confdefs.h
+
+ ;;
+ no)
+ :
+ ;;
+ *)
+ as_fn_error $? "no argument expected for --with-liburing option" "$LINENO" 5
+ ;;
+ esac
+
+else
+ with_liburing=no
+
+fi
+
+
+{ $as_echo "$as_me:${as_lineno-$LINENO}: result: $with_liburing" >&5
+$as_echo "$with_liburing" >&6; }
+
#
# UUID library
@@ -13444,6 +13489,99 @@ fi
fi
+if test "$with_liburing" = yes; then
+
+pkg_failed=no
+{ $as_echo "$as_me:${as_lineno-$LINENO}: checking for liburing" >&5
+$as_echo_n "checking for liburing... " >&6; }
+
+if test -n "$LIBURING_CFLAGS"; then
+ pkg_cv_LIBURING_CFLAGS="$LIBURING_CFLAGS"
+ elif test -n "$PKG_CONFIG"; then
+ if test -n "$PKG_CONFIG" && \
+ { { $as_echo "$as_me:${as_lineno-$LINENO}: \$PKG_CONFIG --exists --print-errors \"liburing\""; } >&5
+ ($PKG_CONFIG --exists --print-errors "liburing") 2>&5
+ ac_status=$?
+ $as_echo "$as_me:${as_lineno-$LINENO}: \$? = $ac_status" >&5
+ test $ac_status = 0; }; then
+ pkg_cv_LIBURING_CFLAGS=`$PKG_CONFIG --cflags "liburing" 2>/dev/null`
+ test "x$?" != "x0" && pkg_failed=yes
+else
+ pkg_failed=yes
+fi
+ else
+ pkg_failed=untried
+fi
+if test -n "$LIBURING_LIBS"; then
+ pkg_cv_LIBURING_LIBS="$LIBURING_LIBS"
+ elif test -n "$PKG_CONFIG"; then
+ if test -n "$PKG_CONFIG" && \
+ { { $as_echo "$as_me:${as_lineno-$LINENO}: \$PKG_CONFIG --exists --print-errors \"liburing\""; } >&5
+ ($PKG_CONFIG --exists --print-errors "liburing") 2>&5
+ ac_status=$?
+ $as_echo "$as_me:${as_lineno-$LINENO}: \$? = $ac_status" >&5
+ test $ac_status = 0; }; then
+ pkg_cv_LIBURING_LIBS=`$PKG_CONFIG --libs "liburing" 2>/dev/null`
+ test "x$?" != "x0" && pkg_failed=yes
+else
+ pkg_failed=yes
+fi
+ else
+ pkg_failed=untried
+fi
+
+
+
+if test $pkg_failed = yes; then
+ { $as_echo "$as_me:${as_lineno-$LINENO}: result: no" >&5
+$as_echo "no" >&6; }
+
+if $PKG_CONFIG --atleast-pkgconfig-version 0.20; then
+ _pkg_short_errors_supported=yes
+else
+ _pkg_short_errors_supported=no
+fi
+ if test $_pkg_short_errors_supported = yes; then
+ LIBURING_PKG_ERRORS=`$PKG_CONFIG --short-errors --print-errors --cflags --libs "liburing" 2>&1`
+ else
+ LIBURING_PKG_ERRORS=`$PKG_CONFIG --print-errors --cflags --libs "liburing" 2>&1`
+ fi
+ # Put the nasty error message in config.log where it belongs
+ echo "$LIBURING_PKG_ERRORS" >&5
+
+ as_fn_error $? "Package requirements (liburing) were not met:
+
+$LIBURING_PKG_ERRORS
+
+Consider adjusting the PKG_CONFIG_PATH environment variable if you
+installed software in a non-standard prefix.
+
+Alternatively, you may set the environment variables LIBURING_CFLAGS
+and LIBURING_LIBS to avoid the need to call pkg-config.
+See the pkg-config man page for more details." "$LINENO" 5
+elif test $pkg_failed = untried; then
+ { $as_echo "$as_me:${as_lineno-$LINENO}: result: no" >&5
+$as_echo "no" >&6; }
+ { { $as_echo "$as_me:${as_lineno-$LINENO}: error: in \`$ac_pwd':" >&5
+$as_echo "$as_me: error: in \`$ac_pwd':" >&2;}
+as_fn_error $? "The pkg-config script could not be found or is too old. Make sure it
+is in your PATH or set the PKG_CONFIG environment variable to the full
+path to pkg-config.
+
+Alternatively, you may set the environment variables LIBURING_CFLAGS
+and LIBURING_LIBS to avoid the need to call pkg-config.
+See the pkg-config man page for more details.
+
+To get pkg-config, see <http://pkg-config.freedesktop.org/>.
+See \`config.log' for more details" "$LINENO" 5; }
+else
+ LIBURING_CFLAGS=$pkg_cv_LIBURING_CFLAGS
+ LIBURING_LIBS=$pkg_cv_LIBURING_LIBS
+ { $as_echo "$as_me:${as_lineno-$LINENO}: result: yes" >&5
+$as_echo "yes" >&6; }
+
+fi
+fi
##
## Header files
diff --git a/.cirrus.tasks.yml b/.cirrus.tasks.yml
index c5e7b743bfb..ba4bfa43a50 100644
--- a/.cirrus.tasks.yml
+++ b/.cirrus.tasks.yml
@@ -447,6 +447,7 @@ task:
--enable-cassert --enable-injection-points --enable-debug \
--enable-tap-tests --enable-nls \
--with-segsize-blocks=6 \
+ --with-liburing \
\
${LINUX_CONFIGURE_FEATURES} \
\
diff --git a/src/Makefile.global.in b/src/Makefile.global.in
index 3b620bac5ac..d45f5fe4ed7 100644
--- a/src/Makefile.global.in
+++ b/src/Makefile.global.in
@@ -191,6 +191,7 @@ with_gssapi = @with_gssapi@
with_krb_srvnam = @with_krb_srvnam@
with_ldap = @with_ldap@
with_libcurl = @with_libcurl@
+with_liburing = @with_liburing@
with_libxml = @with_libxml@
with_libxslt = @with_libxslt@
with_llvm = @with_llvm@
@@ -217,6 +218,9 @@ krb_srvtab = @krb_srvtab@
ICU_CFLAGS = @ICU_CFLAGS@
ICU_LIBS = @ICU_LIBS@
+LIBURING_CFLAGS = @LIBURING_CFLAGS@
+LIBURING_LIBS = @LIBURING_LIBS@
+
TCLSH = @TCLSH@
TCL_LIBS = @TCL_LIBS@
TCL_LIB_SPEC = @TCL_LIB_SPEC@
--
2.48.1.76.g4e746b1a31.dirty
[text/x-diff] v2.5-0006-aio-Add-io_uring-method.patch (16.4K, ../../vz5i2x2wkjjpp6l2z4g4l3umpxktrxsejgdwefi7n6kzw666h3@rnhlvlpadig7/7-v2.5-0006-aio-Add-io_uring-method.patch)
download | inline diff:
From 7655cc62092c9f479464f3f57276c1dda8c79941 Mon Sep 17 00:00:00 2001
From: Andres Freund <andres@anarazel.de>
Date: Tue, 11 Feb 2025 14:29:37 -0500
Subject: [PATCH v2.5 06/30] aio: Add io_uring method
---
src/include/storage/aio.h | 3 +
src/include/storage/aio_internal.h | 3 +
src/include/storage/lwlock.h | 1 +
src/backend/storage/aio/Makefile | 1 +
src/backend/storage/aio/aio.c | 6 +
src/backend/storage/aio/meson.build | 1 +
src/backend/storage/aio/method_io_uring.c | 382 ++++++++++++++++++
src/backend/storage/lmgr/lwlock.c | 1 +
.../utils/activity/wait_event_names.txt | 2 +
src/backend/utils/misc/postgresql.conf.sample | 3 +-
doc/src/sgml/config.sgml | 6 +
src/tools/pgindent/typedefs.list | 1 +
12 files changed, 409 insertions(+), 1 deletion(-)
create mode 100644 src/backend/storage/aio/method_io_uring.c
diff --git a/src/include/storage/aio.h b/src/include/storage/aio.h
index c5ffd9b861a..d45d70d12dc 100644
--- a/src/include/storage/aio.h
+++ b/src/include/storage/aio.h
@@ -24,6 +24,9 @@ typedef enum IoMethod
{
IOMETHOD_SYNC = 0,
IOMETHOD_WORKER,
+#ifdef USE_LIBURING
+ IOMETHOD_IO_URING,
+#endif
} IoMethod;
/* We'll default to worker based execution. */
diff --git a/src/include/storage/aio_internal.h b/src/include/storage/aio_internal.h
index ba086b91be3..f905b13e85f 100644
--- a/src/include/storage/aio_internal.h
+++ b/src/include/storage/aio_internal.h
@@ -342,6 +342,9 @@ extern PgAioHandle *pgaio_inj_io_get(void);
/* Declarations for the tables of function pointers exposed by each IO method. */
extern PGDLLIMPORT const IoMethodOps pgaio_sync_ops;
extern PGDLLIMPORT const IoMethodOps pgaio_worker_ops;
+#ifdef USE_LIBURING
+extern PGDLLIMPORT const IoMethodOps pgaio_uring_ops;
+#endif
extern PGDLLIMPORT const IoMethodOps *pgaio_method_ops;
extern PGDLLIMPORT PgAioCtl *pgaio_ctl;
diff --git a/src/include/storage/lwlock.h b/src/include/storage/lwlock.h
index 13a7dc89980..043e8bae7a9 100644
--- a/src/include/storage/lwlock.h
+++ b/src/include/storage/lwlock.h
@@ -217,6 +217,7 @@ typedef enum BuiltinTrancheIds
LWTRANCHE_SUBTRANS_SLRU,
LWTRANCHE_XACT_SLRU,
LWTRANCHE_PARALLEL_VACUUM_DSA,
+ LWTRANCHE_AIO_URING_COMPLETION,
LWTRANCHE_FIRST_USER_DEFINED,
} BuiltinTrancheIds;
diff --git a/src/backend/storage/aio/Makefile b/src/backend/storage/aio/Makefile
index f51c34a37f8..c06c50771e0 100644
--- a/src/backend/storage/aio/Makefile
+++ b/src/backend/storage/aio/Makefile
@@ -14,6 +14,7 @@ OBJS = \
aio_init.o \
aio_io.o \
aio_target.o \
+ method_io_uring.o \
method_sync.o \
method_worker.o \
read_stream.o
diff --git a/src/backend/storage/aio/aio.c b/src/backend/storage/aio/aio.c
index d6d21c44199..24c7e48cf9c 100644
--- a/src/backend/storage/aio/aio.c
+++ b/src/backend/storage/aio/aio.c
@@ -64,6 +64,9 @@ static void pgaio_io_wait(PgAioHandle *ioh, uint64 ref_generation);
const struct config_enum_entry io_method_options[] = {
{"sync", IOMETHOD_SYNC, false},
{"worker", IOMETHOD_WORKER, false},
+#ifdef USE_LIBURING
+ {"io_uring", IOMETHOD_IO_URING, false},
+#endif
{NULL, 0, false}
};
@@ -81,6 +84,9 @@ PgAioBackend *pgaio_my_backend;
static const IoMethodOps *const pgaio_method_ops_table[] = {
[IOMETHOD_SYNC] = &pgaio_sync_ops,
[IOMETHOD_WORKER] = &pgaio_worker_ops,
+#ifdef USE_LIBURING
+ [IOMETHOD_IO_URING] = &pgaio_uring_ops,
+#endif
};
/* callbacks for the configured io_method, set by assign_io_method */
diff --git a/src/backend/storage/aio/meson.build b/src/backend/storage/aio/meson.build
index 74f94c6e40b..2f0f03d8071 100644
--- a/src/backend/storage/aio/meson.build
+++ b/src/backend/storage/aio/meson.build
@@ -6,6 +6,7 @@ backend_sources += files(
'aio_init.c',
'aio_io.c',
'aio_target.c',
+ 'method_io_uring.c',
'method_sync.c',
'method_worker.c',
'read_stream.c',
diff --git a/src/backend/storage/aio/method_io_uring.c b/src/backend/storage/aio/method_io_uring.c
new file mode 100644
index 00000000000..43f7576498c
--- /dev/null
+++ b/src/backend/storage/aio/method_io_uring.c
@@ -0,0 +1,382 @@
+/*-------------------------------------------------------------------------
+ *
+ * method_io_uring.c
+ * AIO - perform AIO using Linux' io_uring
+ *
+ * XXX Write me
+ *
+ * Portions Copyright (c) 1996-2025, PostgreSQL Global Development Group
+ * Portions Copyright (c) 1994, Regents of the University of California
+ *
+ * IDENTIFICATION
+ * src/backend/storage/aio/method_io_uring.c
+ *
+ *-------------------------------------------------------------------------
+ */
+
+#include "postgres.h"
+
+#ifdef USE_LIBURING
+
+#include <liburing.h>
+
+#include "pgstat.h"
+#include "port/pg_iovec.h"
+#include "storage/aio_internal.h"
+#include "storage/fd.h"
+#include "storage/proc.h"
+#include "storage/shmem.h"
+
+
+/* Entry points for IoMethodOps. */
+static size_t pgaio_uring_shmem_size(void);
+static void pgaio_uring_shmem_init(bool first_time);
+static void pgaio_uring_init_backend(void);
+
+static int pgaio_uring_submit(uint16 num_staged_ios, PgAioHandle **staged_ios);
+static void pgaio_uring_wait_one(PgAioHandle *ioh, uint64 ref_generation);
+
+static void pgaio_uring_sq_from_io(PgAioHandle *ioh, struct io_uring_sqe *sqe);
+
+
+const IoMethodOps pgaio_uring_ops = {
+ .shmem_size = pgaio_uring_shmem_size,
+ .shmem_init = pgaio_uring_shmem_init,
+ .init_backend = pgaio_uring_init_backend,
+
+ .submit = pgaio_uring_submit,
+ .wait_one = pgaio_uring_wait_one,
+};
+
+typedef struct PgAioUringContext
+{
+ LWLock completion_lock;
+
+ struct io_uring io_uring_ring;
+ /* XXX: probably worth padding to a cacheline boundary here */
+} PgAioUringContext;
+
+
+static PgAioUringContext *pgaio_uring_contexts;
+static PgAioUringContext *pgaio_my_uring_context;
+
+/* io_uring local state */
+static struct io_uring local_ring;
+
+
+
+static Size
+pgaio_uring_context_shmem_size(void)
+{
+ uint32 TotalProcs = MaxBackends + NUM_AUXILIARY_PROCS - MAX_IO_WORKERS;
+
+ return mul_size(TotalProcs, sizeof(PgAioUringContext));
+}
+
+static size_t
+pgaio_uring_shmem_size(void)
+{
+ return pgaio_uring_context_shmem_size();
+}
+
+static void
+pgaio_uring_shmem_init(bool first_time)
+{
+ uint32 TotalProcs = MaxBackends + NUM_AUXILIARY_PROCS - MAX_IO_WORKERS;
+ bool found;
+
+ pgaio_uring_contexts = (PgAioUringContext *)
+ ShmemInitStruct("AioUring", pgaio_uring_shmem_size(), &found);
+
+ if (found)
+ return;
+
+ for (int contextno = 0; contextno < TotalProcs; contextno++)
+ {
+ PgAioUringContext *context = &pgaio_uring_contexts[contextno];
+ int ret;
+
+ /*
+ * XXX: Probably worth sharing the WQ between the different rings,
+ * when supported by the kernel. Could also cause additional
+ * contention, I guess?
+ */
+#if 0
+ if (!AcquireExternalFD())
+ elog(ERROR, "No external FD available");
+#endif
+ ret = io_uring_queue_init(io_max_concurrency, &context->io_uring_ring, 0);
+ if (ret < 0)
+ elog(ERROR, "io_uring_queue_init failed: %s", strerror(-ret));
+
+ LWLockInitialize(&context->completion_lock, LWTRANCHE_AIO_URING_COMPLETION);
+ }
+}
+
+static void
+pgaio_uring_init_backend(void)
+{
+ int ret;
+
+ pgaio_my_uring_context = &pgaio_uring_contexts[MyProcNumber];
+
+ ret = io_uring_queue_init(32, &local_ring, 0);
+ if (ret < 0)
+ elog(ERROR, "io_uring_queue_init failed: %s", strerror(-ret));
+}
+
+static int
+pgaio_uring_submit(uint16 num_staged_ios, PgAioHandle **staged_ios)
+{
+ struct io_uring *uring_instance = &pgaio_my_uring_context->io_uring_ring;
+ int in_flight_before = dclist_count(&pgaio_my_backend->in_flight_ios);
+
+ Assert(num_staged_ios <= PGAIO_SUBMIT_BATCH_SIZE);
+
+ for (int i = 0; i < num_staged_ios; i++)
+ {
+ PgAioHandle *ioh = staged_ios[i];
+ struct io_uring_sqe *sqe;
+
+ sqe = io_uring_get_sqe(uring_instance);
+
+ if (!sqe)
+ elog(ERROR, "io_uring submission queue is unexpectedly full");
+
+ pgaio_io_prepare_submit(ioh);
+ pgaio_uring_sq_from_io(ioh, sqe);
+
+ /*
+ * io_uring executes IO in process context if possible. That's
+ * generally good, as it reduces context switching. When performing a
+ * lot of buffered IO that means that copying between page cache and
+ * userspace memory happens in the foreground, as it can't be
+ * offloaded to DMA hardware as is possible when using direct IO. When
+ * executing a lot of buffered IO this causes io_uring to be slower
+ * than worker mode, as worker mode parallelizes the copying. io_uring
+ * can be told to offload work to worker threads instead.
+ *
+ * If an IO is buffered IO and we already have IOs in flight or
+ * multiple IOs are being submitted, we thus tell io_uring to execute
+ * the IO in the background. We don't do so for the first few IOs
+ * being submitted as executing in this process' context has lower
+ * latency.
+ */
+ if (in_flight_before > 4 && (ioh->flags & PGAIO_HF_BUFFERED))
+ io_uring_sqe_set_flags(sqe, IOSQE_ASYNC);
+
+ in_flight_before++;
+ }
+
+ while (true)
+ {
+ int ret;
+
+ pgstat_report_wait_start(WAIT_EVENT_AIO_IO_URING_SUBMIT);
+ ret = io_uring_submit(uring_instance);
+ pgstat_report_wait_end();
+
+ if (ret == -EINTR)
+ {
+ pgaio_debug(DEBUG3,
+ "aio method uring: submit EINTR, nios: %d",
+ num_staged_ios);
+ continue;
+ }
+ if (ret < 0)
+ elog(PANIC, "failed: %d/%s",
+ ret, strerror(-ret));
+ else if (ret != num_staged_ios)
+ {
+ /* likely unreachable, but if it is, we would need to re-submit */
+ elog(PANIC, "submitted only %d of %d",
+ ret, num_staged_ios);
+ }
+ else
+ {
+ pgaio_debug(DEBUG4,
+ "aio method uring: submitted %d IOs",
+ num_staged_ios);
+ }
+ break;
+ }
+
+ return num_staged_ios;
+}
+
+
+#define PGAIO_MAX_LOCAL_COMPLETED_IO 32
+
+static void
+pgaio_uring_drain_locked(PgAioUringContext *context)
+{
+ int ready;
+ int orig_ready;
+
+ /*
+ * Don't drain more events than available right now. Otherwise it's
+ * plausible that one backend could get stuck, for a while, receiving CQEs
+ * without actually processing them.
+ */
+ orig_ready = ready = io_uring_cq_ready(&context->io_uring_ring);
+
+ while (ready > 0)
+ {
+ struct io_uring_cqe *cqes[PGAIO_MAX_LOCAL_COMPLETED_IO];
+ uint32 ncqes;
+
+ START_CRIT_SECTION();
+ ncqes =
+ io_uring_peek_batch_cqe(&context->io_uring_ring,
+ cqes,
+ Min(PGAIO_MAX_LOCAL_COMPLETED_IO, ready));
+ Assert(ncqes <= ready);
+
+ ready -= ncqes;
+
+ for (int i = 0; i < ncqes; i++)
+ {
+ struct io_uring_cqe *cqe = cqes[i];
+ PgAioHandle *ioh;
+
+ ioh = io_uring_cqe_get_data(cqe);
+ io_uring_cqe_seen(&context->io_uring_ring, cqe);
+
+ pgaio_io_process_completion(ioh, cqe->res);
+ }
+
+ END_CRIT_SECTION();
+
+ pgaio_debug(DEBUG3,
+ "drained %d/%d, now expecting %d",
+ ncqes, orig_ready, io_uring_cq_ready(&context->io_uring_ring));
+ }
+}
+
+static void
+pgaio_uring_wait_one(PgAioHandle *ioh, uint64 ref_generation)
+{
+ PgAioHandleState state;
+ ProcNumber owner_procno = ioh->owner_procno;
+ PgAioUringContext *owner_context = &pgaio_uring_contexts[owner_procno];
+ bool expect_cqe;
+ int waited = 0;
+
+ /*
+ * We ought to have a smarter locking scheme, nearly all the time the
+ * backend owning the ring will consume the completions, making the
+ * locking unnecessarily expensive.
+ */
+ LWLockAcquire(&owner_context->completion_lock, LW_EXCLUSIVE);
+
+ while (true)
+ {
+ pgaio_debug_io(DEBUG3, ioh,
+ "wait_one io_gen: %llu, ref_gen: %llu, cycle %d",
+ (long long unsigned) ref_generation,
+ (long long unsigned) ioh->generation,
+ waited);
+
+ if (pgaio_io_was_recycled(ioh, ref_generation, &state) ||
+ state != PGAIO_HS_SUBMITTED)
+ {
+ break;
+ }
+ else if (io_uring_cq_ready(&owner_context->io_uring_ring))
+ {
+ expect_cqe = true;
+ }
+ else
+ {
+ int ret;
+ struct io_uring_cqe *cqes;
+
+ pgstat_report_wait_start(WAIT_EVENT_AIO_IO_URING_COMPLETION);
+ ret = io_uring_wait_cqes(&owner_context->io_uring_ring, &cqes, 1, NULL, NULL);
+ pgstat_report_wait_end();
+
+ if (ret == -EINTR)
+ {
+ continue;
+ }
+ else if (ret != 0)
+ {
+ elog(PANIC, "unexpected: %d/%s: %m", ret, strerror(-ret));
+ }
+ else
+ {
+ Assert(cqes != NULL);
+ expect_cqe = true;
+ waited++;
+ }
+ }
+
+ if (expect_cqe)
+ {
+ pgaio_uring_drain_locked(owner_context);
+ }
+ }
+
+ LWLockRelease(&owner_context->completion_lock);
+
+ pgaio_debug(DEBUG3,
+ "wait_one with %d sleeps",
+ waited);
+}
+
+static void
+pgaio_uring_sq_from_io(PgAioHandle *ioh, struct io_uring_sqe *sqe)
+{
+ struct iovec *iov;
+
+ switch (ioh->op)
+ {
+ case PGAIO_OP_READV:
+ iov = &pgaio_ctl->iovecs[ioh->iovec_off];
+ if (ioh->op_data.read.iov_length == 1)
+ {
+ io_uring_prep_read(sqe,
+ ioh->op_data.read.fd,
+ iov->iov_base,
+ iov->iov_len,
+ ioh->op_data.read.offset);
+ }
+ else
+ {
+ io_uring_prep_readv(sqe,
+ ioh->op_data.read.fd,
+ iov,
+ ioh->op_data.read.iov_length,
+ ioh->op_data.read.offset);
+
+ }
+ break;
+
+ case PGAIO_OP_WRITEV:
+ iov = &pgaio_ctl->iovecs[ioh->iovec_off];
+ if (ioh->op_data.write.iov_length == 1)
+ {
+ io_uring_prep_write(sqe,
+ ioh->op_data.write.fd,
+ iov->iov_base,
+ iov->iov_len,
+ ioh->op_data.write.offset);
+ }
+ else
+ {
+ io_uring_prep_writev(sqe,
+ ioh->op_data.write.fd,
+ iov,
+ ioh->op_data.write.iov_length,
+ ioh->op_data.write.offset);
+ }
+ break;
+
+ case PGAIO_OP_INVALID:
+ elog(ERROR, "trying to prepare invalid IO operation for execution");
+ }
+
+ io_uring_sqe_set_data(sqe, ioh);
+}
+
+#endif /* USE_LIBURING */
diff --git a/src/backend/storage/lmgr/lwlock.c b/src/backend/storage/lmgr/lwlock.c
index 8adf2730277..4a52bb23a7a 100644
--- a/src/backend/storage/lmgr/lwlock.c
+++ b/src/backend/storage/lmgr/lwlock.c
@@ -176,6 +176,7 @@ static const char *const BuiltinTrancheNames[] = {
[LWTRANCHE_SUBTRANS_SLRU] = "SubtransSLRU",
[LWTRANCHE_XACT_SLRU] = "XactSLRU",
[LWTRANCHE_PARALLEL_VACUUM_DSA] = "ParallelVacuumDSA",
+ [LWTRANCHE_AIO_URING_COMPLETION] = "AioUringCompletion",
};
StaticAssertDecl(lengthof(BuiltinTrancheNames) ==
diff --git a/src/backend/utils/activity/wait_event_names.txt b/src/backend/utils/activity/wait_event_names.txt
index cb977b049d8..a2d1a9fa4ec 100644
--- a/src/backend/utils/activity/wait_event_names.txt
+++ b/src/backend/utils/activity/wait_event_names.txt
@@ -192,6 +192,8 @@ ABI_compatibility:
Section: ClassName - WaitEventIO
+AIO_IO_URING_SUBMIT "Waiting for IO submission via io_uring."
+AIO_IO_URING_COMPLETION "Waiting for IO completion via io_uring."
AIO_IO_COMPLETION "Waiting for IO completion."
BASEBACKUP_READ "Waiting for base backup to read from a file."
BASEBACKUP_SYNC "Waiting for data written by a base backup to reach durable storage."
diff --git a/src/backend/utils/misc/postgresql.conf.sample b/src/backend/utils/misc/postgresql.conf.sample
index 1058726285f..8738ad51bf1 100644
--- a/src/backend/utils/misc/postgresql.conf.sample
+++ b/src/backend/utils/misc/postgresql.conf.sample
@@ -202,7 +202,8 @@
#maintenance_io_concurrency = 10 # 1-1000; 0 disables prefetching
#io_combine_limit = 128kB # usually 1-32 blocks (depends on OS)
-#io_method = worker # worker, sync (change requires restart)
+#io_method = worker # worker, io_uring, sync
+ # (change requires restart)
#io_max_concurrency = -1 # Max number of IOs that one process
# can execute simultaneously
# -1 sets based on shared_buffers
diff --git a/doc/src/sgml/config.sgml b/doc/src/sgml/config.sgml
index 8b078f476d1..fe6216df800 100644
--- a/doc/src/sgml/config.sgml
+++ b/doc/src/sgml/config.sgml
@@ -2687,6 +2687,12 @@ include_dir 'conf.d'
<literal>worker</literal> (execute asynchronous I/O using worker processes)
</para>
</listitem>
+ <listitem>
+ <para>
+ <literal>io_uring</literal> (execute asynchronous I/O using
+ io_uring, if available)
+ </para>
+ </listitem>
<listitem>
<para>
<literal>sync</literal> (execute asynchronous I/O synchronously)
diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list
index 2db01bc4617..b35cf61cd36 100644
--- a/src/tools/pgindent/typedefs.list
+++ b/src/tools/pgindent/typedefs.list
@@ -2146,6 +2146,7 @@ PgAioReturn
PgAioTargetData
PgAioTargetID
PgAioTargetInfo
+PgAioUringContext
PgAioWaitRef
PgArchData
PgBackendGSSStatus
--
2.48.1.76.g4e746b1a31.dirty
[text/x-diff] v2.5-0007-aio-Add-README.md-explaining-higher-level-desig.patch (18.6K, ../../vz5i2x2wkjjpp6l2z4g4l3umpxktrxsejgdwefi7n6kzw666h3@rnhlvlpadig7/8-v2.5-0007-aio-Add-README.md-explaining-higher-level-desig.patch)
download | inline diff:
From 4fc6a398b1cab4fd1d9a03535ea2e1ea4f56e378 Mon Sep 17 00:00:00 2001
From: Andres Freund <andres@anarazel.de>
Date: Wed, 22 Jan 2025 13:44:37 -0500
Subject: [PATCH v2.5 07/30] aio: Add README.md explaining higher level design
---
src/backend/storage/aio/README.md | 422 ++++++++++++++++++++++++++++++
src/backend/storage/aio/aio.c | 2 +
2 files changed, 424 insertions(+)
create mode 100644 src/backend/storage/aio/README.md
diff --git a/src/backend/storage/aio/README.md b/src/backend/storage/aio/README.md
new file mode 100644
index 00000000000..de6c7db894b
--- /dev/null
+++ b/src/backend/storage/aio/README.md
@@ -0,0 +1,422 @@
+# Asynchronous & Direct IO
+
+## Motivation
+
+### Why Asynchronous IO
+
+Until the introduction of asynchronous IO Postgres relied on the operating
+system to hide the cost of synchronous IO from Postgres. While this worked
+surprisingly well in a lot of workloads, it does not do as good a job on
+prefetching and controlled writeback as we would like.
+
+There are important expensive operations like `fdatasync()` where the operating
+system cannot hide the storage latency. This is particularly important for WAL
+writes, where the ability to asynchronously issue `fdatasync()` or O_DSYNC
+writes can yield significantly higher throughput.
+
+
+### Why Direct / unbuffered IO
+
+The main reason to want to use Direct IO are:
+
+- Lower CPU usage / higher throughput. Particularly on modern storage buffered
+ writes are bottlenecked by the operating system having to copy data from the
+ kernel's page cache to postgres buffer pool using the CPU. Whereas direct IO
+ can often move the data directly between the storage devices and postgres'
+ buffer cache, using DMA. While that transfer is ongoing, the CPU is free to
+ perform other work.
+- Reduced latency - Direct IO can have substantially lower latency than
+ buffered IO, which can be impactful for OLTP workloads bottlenecked by WAL
+ write latency.
+- Avoiding double buffering between operating system cache and postgres'
+ shared_buffers.
+- Better control over the timing and pace of dirty data writeback.
+
+
+The main reason *not* to use Direct IO are:
+
+- Without AIO, Direct IO is unusably slow for most purposes.
+- Even with AIO, many parts of postgres need to be modified to perform
+ explicit prefetching.
+- In situations where shared_buffers cannot be set appropriately large,
+ e.g. because there are many different postgres instances hosted on shared
+ hardware, performance will often be worse than when using buffered IO.
+
+
+## AIO Usage Example
+
+In many cases code that can benefit from AIO does not directly have to
+interact with the AIO interface, but can use AIO via higher-level
+abstractions. See [Helpers](#helpers).
+
+In this example, a buffer will be read into shared buffers.
+
+```C
+/*
+ * Result of the operation, only to be accessed in this backend.
+ */
+PgAioReturn ioret;
+
+/*
+ * Acquire an AIO Handle, ioret will get result upon completion.
+ *
+ * Note that ioret needs to stay alive until the IO completes or
+ * CurrentResourceOwner is released (i.e. an error is thrown).
+ */
+PgAioHandle *ioh = pgaio_io_acquire(CurrentResourceOwner, &ioret);
+
+/*
+ * Reference that can be used to wait for the IO we initiate below. This
+ * reference can reside in local or shared memory and waited upon by any
+ * process. An arbitrary number of references can be made for each IO.
+ */
+PgAioWaitRef iow;
+
+pgaio_io_get_wref(ioh, &iow);
+
+/*
+ * Arrange for shared buffer completion callbacks to be called upon completion
+ * of the IO. This callback will update the buffer descriptors associated with
+ * the AioHandle, which e.g. allows other backends to access the buffer.
+ *
+ * A callback can be passed a small bit of data, e.g. to indicate whether to
+ * zero a buffer if it is invalid.
+ *
+ * Multiple completion callbacks can be registered for each handle.
+ */
+pgaio_io_register_callbacks(ioh, PGAIO_HCB_SHARED_BUFFER_READV, 0);
+
+/*
+ * The completion callback needs to know which buffers to update when the IO
+ * completes. As the AIO subsystem does not know about buffers, we have to
+ * associate this information with the AioHandle, for use by the completion
+ * callback registered above.
+ *
+ * In this example we're reading only a single buffer, hence the 1.
+ */
+pgaio_io_set_handle_data_32(ioh, (uint32 *) buffer, 1);
+
+/*
+ * Pass the AIO handle to lower-level function. When operating on the level of
+ * buffers, we don't know how exactly the IO is performed, that is the
+ * responsibility of the storage manager implementation.
+ *
+ * E.g. md.c needs to translate block numbers into offsets in segments.
+ *
+ * Once the IO handle has been handed off to smgstartreadv(), it may not
+ * further be used, as the IO may immediately get executed below
+ * smgrstartreadv() and the handle reused for another IO.
+ *
+ * To issue multiple IOs in an efficient way, a caller can call
+ * pgaio_enter_batchmode() before starting multiple IOs, and end that batch
+ * with pgaio_exit_batchmode(). Note that one needs to be careful while there
+ * may be unsubmitted IOs, as another backend may need to wait for one of the
+ * unsubmitted IOs. If this backend then had to to wait for the other backend,
+ * it'd end in an undetected deadlock. See pgaio_enter_batchmode() for more
+ * details.
+ *
+ * Note that even while in batchmode an IO might get submitted immediately,
+ * e.g. due to reaching a limit on the number of unsubmitted IOs, and even
+ * complete before smgrstartreadv() returns.
+ */
+smgrstartreadv(ioh, operation->smgr, forknum, blkno,
+ BufferGetBlock(buffer), 1);
+
+/*
+ * To benefit from AIO, it is beneficial to perform other work, including
+ * submitting other IOs, before waiting for the IO to complete. Otherwise
+ * we could just have used synchronous, blocking IO.
+ */
+perform_other_work();
+
+/*
+ * We did some other work and now need the IO operation to have completed to
+ * continue.
+ */
+pgaio_wref_wait(&iow);
+
+/*
+ * At this point the IO has completed. We do not yet know whether it succeeded
+ * or failed, however. The buffer's state has been updated, which allows other
+ * backends to use the buffer (if the IO succeeded), or retry the IO (if it
+ * failed).
+ *
+ * Note that in case the IO has failed, a LOG message may have been emitted,
+ * but no ERROR has been raised. This is crucial, as another backend waiting
+ * for this IO should not see an ERROR.
+ *
+ * To check whether the operation succeeded, and to raise an ERROR, or if more
+ * appropriate LOG, the PgAioReturn we passed to pgaio_io_acquire() is used.
+ */
+if (ioret.result.status == ARS_ERROR)
+ pgaio_result_report(aio_ret.result, &aio_ret.target_data, ERROR);
+
+/*
+ * Besides having succeeded completely, the IO could also have partially
+ * completed. If we e.g. tried to read many blocks at once, the read might have
+ * only succeeded for the first few blocks.
+ *
+ * If the IO partially succeeded and this backend needs all blocks to have
+ * completed, this backend needs to reissue the IO for the remaining buffers.
+ * The AIO subsystem cannot handle this retry transparently.
+ *
+ * As this example is already long, and we only read a single block, we'll just
+ * error out if there's a partial read.
+ */
+if (ioret.result.status == ARS_PARTIAL)
+ pgaio_result_report(aio_ret.result, &aio_ret.target_data, ERROR);
+
+/*
+ * The IO succeeded, so we can use the buffer now.
+ */
+```
+
+
+## Design Criteria & Motivation
+
+### Deadlock and Starvation Dangers due to AIO
+
+Using AIO in a naive way can easily lead to deadlocks in an environment where
+the source/target of AIO are shared resources, like pages in postgres'
+shared_buffers.
+
+Consider one backend performing readahead on a table, initiating IO for a
+number of buffers ahead of the current "scan position". If that backend then
+performs some operation that blocks, or even just is slow, the IO completion
+for the asynchronously initiated read may not be processed.
+
+This AIO implementation solves this problem by requiring that AIO methods
+either allow AIO completions to be processed by any backend in the system
+(e.g. io_uring), or to guarantee that AIO processing will happen even when the
+issuing backend is blocked (e.g. worker mode, which offloads completion
+processing to the AIO workers).
+
+
+### IO can be started in critical sections
+
+Using AIO for WAL writes can reduce the overhead of WAL logging substantially:
+
+- AIO allows to start WAL writes eagerly, so they complete before needing to
+ wait
+- AIO allows to have multiple WAL flushes in progress at the same time
+- AIO makes it more realistic to use O\_DIRECT + O\_DSYNC, which can reduce
+ the number of roundtrips to storage on some OSs and storage HW (buffered IO
+ and direct IO without O_DSYNC needs to issue a write and after the writes
+ completion a cache cache flush, whereas O\_DIRECT + O\_DSYNC can use a
+ single FUA write).
+
+The need to be able to execute IO in critical sections has substantial design
+implication on the AIO subsystem. Mainly because completing IOs (see prior
+section) needs to be possible within a critical section, even if the
+to-be-completed IO itself was not issued in a critical section. Consider
+e.g. the case of a backend first starting a number of writes from shared
+buffers and then starting to flush the WAL. Because only a limited amount of
+IO can be in-progress at the same time, initiating IO for flushing the WAL may
+require to first complete IO that was started earlier.
+
+
+### State for AIO needs to live in shared memory
+
+Because postgres uses a process model and because AIOs need to be
+complete-able by any backend much of the state of the AIO subsystem needs to
+live in shared memory.
+
+In an `EXEC_BACKEND` build backends executable code and other process local
+state is not necessarily mapped to the same addresses in each process due to
+ASLR. This means that the shared memory cannot contain pointer to callbacks.
+
+
+## Design of the AIO Subsystem
+
+
+### AIO Methods
+
+To achieve portability and performance, multiple methods of performing AIO are
+implemented and others are likely worth adding in the future.
+
+
+#### Synchronous Mode
+
+`io_method=sync` does not actually perform AIO but allows to use the AIO API
+while performing synchronous IO. This can be useful for debugging. The code
+for the synchronous mode is also used as a fallback by e.g. the [worker
+mode](#worker) uses it to execute IO that cannot be executed by workers.
+
+
+#### Worker
+
+`io_method=worker` is available on every platform postgres runs on, and
+implements asynchronous IO - from the view of the issuing process - by
+dispatching the IO to one of several worker processes performing the IO in a
+synchronous manner.
+
+
+#### io_uring
+
+`io_method=io_uring` is available on Linux 5.1+. In contrast to worker mode it
+dispatches all IO from within the process, lowering context switch rate /
+latency.
+
+
+### AIO Handles
+
+The central API piece for postgres' AIO abstraction are AIO handles. To
+execute an IO one first has to acquire an IO handle (`pgaio_io_acquire()`) and
+then "defined", i.e. associate an IO operation with the handle.
+
+Often AIO handles are acquired on a higher level and then passed to a lower
+level to be fully defined. E.g., for IO to/from shared buffers, bufmgr.c
+routines acquire the handle, which is then passed through smgr.c, md.c to be
+finally fully defined in fd.c.
+
+The functions used at the lowest level to define the operation are
+`pgaio_io_prep_*()`.
+
+Because acquisition of an IO handle
+[must always succeed](#io-can-be-started-in-critical-sections)
+and the number of AIO Handles
+[has to be limited](#state-for-aio-needs-to-live-in-shared-memory)
+AIO handles can be reused as soon as they have completed. Obviously code needs
+to be able to react to IO completion. Shared state can be updated using
+[AIO Completion callbacks](#aio-callbacks)
+and the issuing backend can provide a backend local variable to receive the
+result of the IO, as described in
+[AIO Result](#aio-results)
+. An IO can be waited for, by both the issuing and any other backend, using
+[AIO References](#aio-wait-references).
+
+
+Because an AIO Handle is not executable just after calling `pgaio_io_acquire()`
+and because `pgaio_io_acquire()` needs to be able to succeed, only a single AIO
+Handle may be acquired (i.e. returned by `pgaio_io_acquire()`) without causing
+the IO to have been defined (by, potentially indirectly, causing
+`pgaio_io_prep_*()` to have been called). Otherwise a backend could trivially
+self-deadlock by using up all AIO Handles without the ability to wait for some
+of the IOs to complete.
+
+If it turns out that an AIO Handle is not needed, e.g., because the handle was
+acquired before holding a contended lock, it can be released without being
+defined using `pgaio_io_release()`.
+
+
+### AIO Callbacks
+
+Commonly several layers need to react to completion of an IO. E.g. for a read
+md.c needs to check if the IO outright failed or was shorter than needed,
+bufmgr.c needs to verify the page looks valid and bufmgr.c needs to update the
+BufferDesc to update the buffer's state.
+
+The fact that several layers / subsystems need to react to IO completion poses
+a few challenges:
+
+- Upper layers should not need to know details of lower layers. E.g. bufmgr.c
+ should not assume the IO will pass through md.c. Therefore upper levels
+ cannot know what lower layers would consider an error.
+
+- Lower layers should not need to know about upper layers. E.g. smgr APIs are
+ used going through shared buffers but are also used bypassing shared
+ buffers. This means that e.g. md.c is not in a position to validate
+ checksums.
+
+- Having code in the AIO subsystem for every possible combination of layers
+ would lead to a lot of duplication.
+
+The "solution" to this the ability to associate multiple completion callbacks
+with a handle. E.g. bufmgr.c can have a callback to update the BufferDesc
+state and to verify the page and md.c. another callback to check if the IO
+operation was successful.
+
+As [mentioned](#state-for-aio-needs-to-live-in-shared-memory), shared memory
+currently cannot contain function pointers. Because of that completion
+callbacks are not directly identified by function pointers but by IDs
+(`PgAioHandleCallbackID`). A substantial added benefit is that that
+allows callbacks to be identified by much smaller amount of memory (a single
+byte currently).
+
+In addition to completion, AIO callbacks also are called to "prepare" an
+IO. This is, e.g., used to increase buffer reference counts to account for the
+AIO subsystem referencing the buffer, which is required to handle the case
+where the issuing backend errors out and releases its own pins while the IO is
+still ongoing.
+
+As [explained earlier](#io-can-be-started-in-critical-sections) IO completions
+need to be safe to execute in critical sections. To allow the backend that
+issued the IO to error out in case of failure [AIO Result](#aio-results) can
+be used.
+
+
+### AIO Targets
+
+In addition to the completion callbacks describe above, each AIO Handle has
+exactly one "target". Each target has some space inside an AIO Handle with
+information specific to the target and can provide callbacks to allow to
+reopen the underlying file (required for worker mode) and to describe the IO
+operation (used for debug logging and error messages).
+
+I.e., if two different uses of AIO can describe the identity of the file being
+operated on the same way, it likely makes sense to use the same
+target. E.g. different smgr implementations can describe IO with
+RelFileLocator, ForkNumber and BlockNumber and can thus share a target. In
+contrast, IO for a WAL file would be described with TimeLineID and XLogRecPtr
+and it would not make sense to use the same target for smgr and WAL.
+
+
+### AIO Wait References
+
+As [described above](#aio-handles), AIO Handles can be reused immediately
+after completion and therefore cannot be used to wait for completion of the
+IO. Waiting is enabled using AIO wait references, which do not just identify
+an AIO Handle but also include the handles "generation".
+
+A reference to an AIO Handle can be acquired using `pgaio_io_get_wref()` and
+then waited upon using `pgaio_wref_wait()`.
+
+
+### AIO Results
+
+As AIO completion callbacks
+[are executed in critical sections](#io-can-be-started-in-critical-sections)
+and [may be executed by any backend](#deadlock-and-starvation-dangers-due-to-aio)
+completion callbacks cannot be used to, e.g., make the query that triggered an
+IO ERROR out.
+
+To allow to react to failing IOs the issuing backend can pass a pointer to a
+`PgAioReturn` in backend local memory. Before an AIO Handle is reused the
+`PgAioReturn` is filled with information about the IO. This includes
+information about whether the IO was successful (as a value of
+`PgAioResultStatus`) and enough information to raise an error in case of a
+failure (via `pgaio_result_report()`, with the error details encoded in
+`PgAioResult`).
+
+
+### AIO Errors
+
+It would be very convenient to have shared completion callbacks encode the
+details of errors as an `ErrorData` that could be raised at a later
+time. Unfortunately doing so would require allocating memory. While elog.c can
+guarantee (well, kinda) that logging a message will not run out of memory,
+that only works because a very limited number of messages are in the process
+of being logged. With AIO a large number of concurrently issued AIOs might
+fail.
+
+To avoid the need for preallocating a potentially large amount of memory (in
+shared memory no less!), completion callbacks instead have to encode errors in
+a more compact format that can be converted into an error message.
+
+
+## Helpers
+
+Using the low-level AIO API introduces too much complexity to do so all over
+the tree. Most uses of AIO should be done via reusable, higher-level,
+helpers.
+
+
+### Read Stream
+
+A common and very beneficial use of AIO are reads where a substantial number
+of to-be-read locations are known ahead of time. E.g., for a sequential scan
+the set of blocks that need to be read can be determined solely by knowing the
+current position and checking the buffer mapping table.
+
+The [Read Stream](../../../include/storage/read_stream.h) interface makes it
+comparatively easy to use AIO for such use cases.
diff --git a/src/backend/storage/aio/aio.c b/src/backend/storage/aio/aio.c
index 24c7e48cf9c..3af61227ea8 100644
--- a/src/backend/storage/aio/aio.c
+++ b/src/backend/storage/aio/aio.c
@@ -24,6 +24,8 @@
*
* - read_stream.c - helper for reading buffered relation data
*
+ * - README.md - higher-level overview over AIO
+ *
*
* Portions Copyright (c) 1996-2025, PostgreSQL Global Development Group
* Portions Copyright (c) 1994, Regents of the University of California
--
2.48.1.76.g4e746b1a31.dirty
[text/x-diff] v2.5-0008-aio-Implement-smgr-md-fd-read-support.patch (20.5K, ../../vz5i2x2wkjjpp6l2z4g4l3umpxktrxsejgdwefi7n6kzw666h3@rnhlvlpadig7/9-v2.5-0008-aio-Implement-smgr-md-fd-read-support.patch)
download | inline diff:
From ea0abea26bcd6ddd05c8c4ff952557ea4861c0cd Mon Sep 17 00:00:00 2001
From: Andres Freund <andres@anarazel.de>
Date: Wed, 22 Jan 2025 16:06:51 -0500
Subject: [PATCH v2.5 08/30] aio: Implement smgr/md/fd read support
---
src/include/storage/aio.h | 6 +-
src/include/storage/aio_types.h | 10 +-
src/include/storage/fd.h | 4 +
src/include/storage/md.h | 7 +
src/include/storage/smgr.h | 17 +++
src/backend/storage/aio/aio_callback.c | 3 +
src/backend/storage/aio/aio_target.c | 2 +
src/backend/storage/file/fd.c | 40 ++++++
src/backend/storage/smgr/md.c | 175 +++++++++++++++++++++++++
src/backend/storage/smgr/smgr.c | 106 +++++++++++++++
10 files changed, 366 insertions(+), 4 deletions(-)
diff --git a/src/include/storage/aio.h b/src/include/storage/aio.h
index d45d70d12dc..8fbd206c343 100644
--- a/src/include/storage/aio.h
+++ b/src/include/storage/aio.h
@@ -108,9 +108,10 @@ typedef enum PgAioTargetID
{
/* intentionally the zero value, to help catch zeroed memory etc */
PGAIO_TID_INVALID = 0,
+ PGAIO_TID_SMGR,
} PgAioTargetID;
-#define PGAIO_TID_COUNT (PGAIO_TID_INVALID + 1)
+#define PGAIO_TID_COUNT (PGAIO_TID_SMGR + 1)
/*
@@ -174,6 +175,9 @@ typedef struct PgAioTargetInfo
typedef enum PgAioHandleCallbackID
{
PGAIO_HCB_INVALID,
+
+ PGAIO_HCB_MD_READV,
+ PGAIO_HCB_MD_WRITEV,
} PgAioHandleCallbackID;
diff --git a/src/include/storage/aio_types.h b/src/include/storage/aio_types.h
index d2617139a25..3ff9282bb1b 100644
--- a/src/include/storage/aio_types.h
+++ b/src/include/storage/aio_types.h
@@ -58,11 +58,15 @@ typedef struct PgAioWaitRef
*/
typedef union PgAioTargetData
{
- /* just as an example placeholder for later */
struct
{
- uint32 queue_id;
- } wal;
+ RelFileLocator rlocator; /* physical relation identifier */
+ BlockNumber blockNum; /* blknum relative to begin of reln */
+ BlockNumber nblocks;
+ ForkNumber forkNum:8; /* don't waste 4 byte for four values */
+ bool is_temp:1; /* proc can be inferred by owning AIO */
+ bool skip_fsync:1;
+ } smgr;
} PgAioTargetData;
diff --git a/src/include/storage/fd.h b/src/include/storage/fd.h
index e3067ab6597..f44f3b939d6 100644
--- a/src/include/storage/fd.h
+++ b/src/include/storage/fd.h
@@ -101,6 +101,8 @@ extern PGDLLIMPORT int max_safe_fds;
* prototypes for functions in fd.c
*/
+struct PgAioHandle;
+
/* Operations on virtual Files --- equivalent to Unix kernel file ops */
extern File PathNameOpenFile(const char *fileName, int fileFlags);
extern File PathNameOpenFilePerm(const char *fileName, int fileFlags, mode_t fileMode);
@@ -109,6 +111,8 @@ extern void FileClose(File file);
extern int FilePrefetch(File file, off_t offset, off_t amount, uint32 wait_event_info);
extern ssize_t FileReadV(File file, const struct iovec *iov, int iovcnt, off_t offset, uint32 wait_event_info);
extern ssize_t FileWriteV(File file, const struct iovec *iov, int iovcnt, off_t offset, uint32 wait_event_info);
+extern ssize_t FileReadV(File file, const struct iovec *iov, int iovcnt, off_t offset, uint32 wait_event_info);
+extern int FileStartReadV(struct PgAioHandle *ioh, File file, int iovcnt, off_t offset, uint32 wait_event_info);
extern int FileSync(File file, uint32 wait_event_info);
extern int FileZero(File file, off_t offset, off_t amount, uint32 wait_event_info);
extern int FileFallocate(File file, off_t offset, off_t amount, uint32 wait_event_info);
diff --git a/src/include/storage/md.h b/src/include/storage/md.h
index 05bf537066e..bf714a8896d 100644
--- a/src/include/storage/md.h
+++ b/src/include/storage/md.h
@@ -19,6 +19,9 @@
#include "storage/smgr.h"
#include "storage/sync.h"
+struct PgAioHandleCallbacks;
+extern const struct PgAioHandleCallbacks aio_md_readv_cb;
+
/* md storage manager functionality */
extern void mdinit(void);
extern void mdopen(SMgrRelation reln);
@@ -36,6 +39,9 @@ extern uint32 mdmaxcombine(SMgrRelation reln, ForkNumber forknum,
BlockNumber blocknum);
extern void mdreadv(SMgrRelation reln, ForkNumber forknum, BlockNumber blocknum,
void **buffers, BlockNumber nblocks);
+extern void mdstartreadv(struct PgAioHandle *ioh,
+ SMgrRelation reln, ForkNumber forknum, BlockNumber blocknum,
+ void **buffers, BlockNumber nblocks);
extern void mdwritev(SMgrRelation reln, ForkNumber forknum,
BlockNumber blocknum,
const void **buffers, BlockNumber nblocks, bool skipFsync);
@@ -46,6 +52,7 @@ extern void mdtruncate(SMgrRelation reln, ForkNumber forknum,
BlockNumber old_blocks, BlockNumber nblocks);
extern void mdimmedsync(SMgrRelation reln, ForkNumber forknum);
extern void mdregistersync(SMgrRelation reln, ForkNumber forknum);
+extern int mdfd(SMgrRelation reln, ForkNumber forknum, BlockNumber blocknum, uint32 *off);
extern void ForgetDatabaseSyncRequests(Oid dbid);
extern void DropRelationFiles(RelFileLocator *delrels, int ndelrels, bool isRedo);
diff --git a/src/include/storage/smgr.h b/src/include/storage/smgr.h
index 4016b206ad6..85562fa0cc0 100644
--- a/src/include/storage/smgr.h
+++ b/src/include/storage/smgr.h
@@ -73,6 +73,11 @@ typedef SMgrRelationData *SMgrRelation;
#define SmgrIsTemp(smgr) \
RelFileLocatorBackendIsTemp((smgr)->smgr_rlocator)
+struct PgAioHandle;
+struct PgAioTargetInfo;
+
+extern const struct PgAioTargetInfo aio_smgr_target_info;
+
extern void smgrinit(void);
extern SMgrRelation smgropen(RelFileLocator rlocator, ProcNumber backend);
extern bool smgrexists(SMgrRelation reln, ForkNumber forknum);
@@ -97,6 +102,10 @@ extern uint32 smgrmaxcombine(SMgrRelation reln, ForkNumber forknum,
extern void smgrreadv(SMgrRelation reln, ForkNumber forknum,
BlockNumber blocknum,
void **buffers, BlockNumber nblocks);
+extern void smgrstartreadv(struct PgAioHandle *ioh,
+ SMgrRelation reln, ForkNumber forknum,
+ BlockNumber blocknum,
+ void **buffers, BlockNumber nblocks);
extern void smgrwritev(SMgrRelation reln, ForkNumber forknum,
BlockNumber blocknum,
const void **buffers, BlockNumber nblocks,
@@ -110,6 +119,7 @@ extern void smgrtruncate(SMgrRelation reln, ForkNumber *forknum, int nforks,
BlockNumber *nblocks);
extern void smgrimmedsync(SMgrRelation reln, ForkNumber forknum);
extern void smgrregistersync(SMgrRelation reln, ForkNumber forknum);
+extern int smgrfd(SMgrRelation reln, ForkNumber forknum, BlockNumber blocknum, uint32 *off);
extern void AtEOXact_SMgr(void);
extern bool ProcessBarrierSmgrRelease(void);
@@ -127,4 +137,11 @@ smgrwrite(SMgrRelation reln, ForkNumber forknum, BlockNumber blocknum,
smgrwritev(reln, forknum, blocknum, &buffer, 1, skipFsync);
}
+extern void pgaio_io_set_target_smgr(struct PgAioHandle *ioh,
+ SMgrRelationData *smgr,
+ ForkNumber forknum,
+ BlockNumber blocknum,
+ int nblocks,
+ bool skip_fsync);
+
#endif /* SMGR_H */
diff --git a/src/backend/storage/aio/aio_callback.c b/src/backend/storage/aio/aio_callback.c
index 3071bf19f23..3112b935676 100644
--- a/src/backend/storage/aio/aio_callback.c
+++ b/src/backend/storage/aio/aio_callback.c
@@ -18,6 +18,7 @@
#include "miscadmin.h"
#include "storage/aio.h"
#include "storage/aio_internal.h"
+#include "storage/md.h"
/* just to have something to put into the aio_handle_cbs */
@@ -37,6 +38,8 @@ typedef struct PgAioHandleCallbacksEntry
static const PgAioHandleCallbacksEntry aio_handle_cbs[] = {
#define CALLBACK_ENTRY(id, callback) [id] = {.cb = &callback, .name = #callback}
CALLBACK_ENTRY(PGAIO_HCB_INVALID, aio_invalid_cb),
+
+ CALLBACK_ENTRY(PGAIO_HCB_MD_READV, aio_md_readv_cb),
#undef CALLBACK_ENTRY
};
diff --git a/src/backend/storage/aio/aio_target.c b/src/backend/storage/aio/aio_target.c
index 637e3aa0928..45a4af68f24 100644
--- a/src/backend/storage/aio/aio_target.c
+++ b/src/backend/storage/aio/aio_target.c
@@ -16,6 +16,7 @@
#include "storage/aio.h"
#include "storage/aio_internal.h"
+#include "storage/smgr.h"
/*
@@ -29,6 +30,7 @@ static const PgAioTargetInfo *pgaio_target_info[] = {
[PGAIO_TID_INVALID] = &(PgAioTargetInfo) {
.name = "invalid",
},
+ [PGAIO_TID_SMGR] = &aio_smgr_target_info,
};
diff --git a/src/backend/storage/file/fd.c b/src/backend/storage/file/fd.c
index e454db4c020..996458fac71 100644
--- a/src/backend/storage/file/fd.c
+++ b/src/backend/storage/file/fd.c
@@ -94,6 +94,7 @@
#include "miscadmin.h"
#include "pgstat.h"
#include "postmaster/startup.h"
+#include "storage/aio.h"
#include "storage/fd.h"
#include "storage/ipc.h"
#include "utils/guc.h"
@@ -1294,6 +1295,8 @@ LruDelete(File file)
vfdP = &VfdCache[file];
+ pgaio_closing_fd(vfdP->fd);
+
/*
* Close the file. We aren't expecting this to fail; if it does, better
* to leak the FD than to mess up our internal state.
@@ -1987,6 +1990,8 @@ FileClose(File file)
if (!FileIsNotOpen(file))
{
+ pgaio_closing_fd(vfdP->fd);
+
/* close the file */
if (close(vfdP->fd) != 0)
{
@@ -2210,6 +2215,32 @@ retry:
return returnCode;
}
+int
+FileStartReadV(struct PgAioHandle *ioh, File file,
+ int iovcnt, off_t offset,
+ uint32 wait_event_info)
+{
+ int returnCode;
+ Vfd *vfdP;
+
+ Assert(FileIsValid(file));
+
+ DO_DB(elog(LOG, "FileStartReadV: %d (%s) " INT64_FORMAT " %d",
+ file, VfdCache[file].fileName,
+ (int64) offset,
+ iovcnt));
+
+ returnCode = FileAccess(file);
+ if (returnCode < 0)
+ return returnCode;
+
+ vfdP = &VfdCache[file];
+
+ pgaio_io_prep_readv(ioh, vfdP->fd, iovcnt, offset);
+
+ return 0;
+}
+
ssize_t
FileWriteV(File file, const struct iovec *iov, int iovcnt, off_t offset,
uint32 wait_event_info)
@@ -2498,6 +2529,12 @@ FilePathName(File file)
int
FileGetRawDesc(File file)
{
+ int returnCode;
+
+ returnCode = FileAccess(file);
+ if (returnCode < 0)
+ return returnCode;
+
Assert(FileIsValid(file));
return VfdCache[file].fd;
}
@@ -2778,6 +2815,7 @@ FreeDesc(AllocateDesc *desc)
result = closedir(desc->desc.dir);
break;
case AllocateDescRawFD:
+ pgaio_closing_fd(desc->desc.fd);
result = close(desc->desc.fd);
break;
default:
@@ -2846,6 +2884,8 @@ CloseTransientFile(int fd)
/* Only get here if someone passes us a file not in allocatedDescs */
elog(WARNING, "fd passed to CloseTransientFile was not obtained from OpenTransientFile");
+ pgaio_closing_fd(fd);
+
return close(fd);
}
diff --git a/src/backend/storage/smgr/md.c b/src/backend/storage/smgr/md.c
index f3220f98dc4..d01ae5f6a09 100644
--- a/src/backend/storage/smgr/md.c
+++ b/src/backend/storage/smgr/md.c
@@ -31,6 +31,7 @@
#include "miscadmin.h"
#include "pg_trace.h"
#include "pgstat.h"
+#include "storage/aio.h"
#include "storage/bufmgr.h"
#include "storage/fd.h"
#include "storage/md.h"
@@ -152,6 +153,15 @@ static MdfdVec *_mdfd_getseg(SMgrRelation reln, ForkNumber forknum,
static BlockNumber _mdnblocks(SMgrRelation reln, ForkNumber forknum,
MdfdVec *seg);
+static PgAioResult md_readv_complete(PgAioHandle *ioh, PgAioResult prior_result, uint8 cb_data);
+static void md_readv_report(PgAioResult result, const PgAioTargetData *target_data, int elevel);
+
+const struct PgAioHandleCallbacks aio_md_readv_cb = {
+ .complete_shared = md_readv_complete,
+ .report = md_readv_report,
+};
+
+
static inline int
_mdfd_open_flags(void)
{
@@ -937,6 +947,61 @@ mdreadv(SMgrRelation reln, ForkNumber forknum, BlockNumber blocknum,
}
}
+/*
+ * mdstartreadv() -- Asynchronous version of mdreadv().
+ */
+void
+mdstartreadv(PgAioHandle *ioh,
+ SMgrRelation reln, ForkNumber forknum, BlockNumber blocknum,
+ void **buffers, BlockNumber nblocks)
+{
+ off_t seekpos;
+ MdfdVec *v;
+ BlockNumber nblocks_this_segment;
+ struct iovec *iov;
+ int iovcnt;
+
+ v = _mdfd_getseg(reln, forknum, blocknum, false,
+ EXTENSION_FAIL | EXTENSION_CREATE_RECOVERY);
+
+ seekpos = (off_t) BLCKSZ * (blocknum % ((BlockNumber) RELSEG_SIZE));
+
+ Assert(seekpos < (off_t) BLCKSZ * RELSEG_SIZE);
+
+ nblocks_this_segment =
+ Min(nblocks,
+ RELSEG_SIZE - (blocknum % ((BlockNumber) RELSEG_SIZE)));
+
+ if (nblocks_this_segment != nblocks)
+ elog(ERROR, "read crossing segment boundary");
+
+ iovcnt = pgaio_io_get_iovec(ioh, &iov);
+
+ Assert(nblocks <= iovcnt);
+
+ iovcnt = buffers_to_iovec(iov, buffers, nblocks_this_segment);
+
+ Assert(iovcnt <= nblocks_this_segment);
+
+ if (!(io_direct_flags & IO_DIRECT_DATA))
+ pgaio_io_set_flag(ioh, PGAIO_HF_BUFFERED);
+
+ pgaio_io_set_target_smgr(ioh,
+ reln,
+ forknum,
+ blocknum,
+ nblocks,
+ false);
+ pgaio_io_register_callbacks(ioh, PGAIO_HCB_MD_READV, 0);
+
+ FileStartReadV(ioh, v->mdfd_vfd, iovcnt, seekpos, WAIT_EVENT_DATA_FILE_READ);
+
+ /*
+ * The error checks corresponding to the post-read checks in mdread() are
+ * in md_readv_complete().
+ */
+}
+
/*
* mdwritev() -- Write the supplied blocks at the appropriate location.
*
@@ -1365,6 +1430,21 @@ mdimmedsync(SMgrRelation reln, ForkNumber forknum)
}
}
+int
+mdfd(SMgrRelation reln, ForkNumber forknum, BlockNumber blocknum, uint32 *off)
+{
+ MdfdVec *v = mdopenfork(reln, forknum, EXTENSION_FAIL);
+
+ v = _mdfd_getseg(reln, forknum, blocknum, false,
+ EXTENSION_FAIL);
+
+ *off = (off_t) BLCKSZ * (blocknum % ((BlockNumber) RELSEG_SIZE));
+
+ Assert(*off < (off_t) BLCKSZ * RELSEG_SIZE);
+
+ return FileGetRawDesc(v->mdfd_vfd);
+}
+
/*
* register_dirty_segment() -- Mark a relation segment as needing fsync
*
@@ -1841,3 +1921,98 @@ mdfiletagmatches(const FileTag *ftag, const FileTag *candidate)
*/
return ftag->rlocator.dbOid == candidate->rlocator.dbOid;
}
+
+/*
+ * AIO completion callback for mdstartreadv().
+ */
+static PgAioResult
+md_readv_complete(PgAioHandle *ioh, PgAioResult prior_result, uint8 cb_data)
+{
+ PgAioTargetData *td = pgaio_io_get_target_data(ioh);
+ PgAioResult result = prior_result;
+
+ if (prior_result.result < 0)
+ {
+ result.status = ARS_ERROR;
+ result.id = PGAIO_HCB_MD_READV;
+ /* For "hard" errors, track the error number in error_data */
+ result.error_data = -prior_result.result;
+ result.result = 0;
+
+ md_readv_report(result, td, LOG);
+
+ return result;
+ }
+
+ /*
+ * The smgr API operates in blocks, therefore convert the result from
+ * bytes to blocks.
+ */
+ result.result /= BLCKSZ;
+
+ if (result.result == 0)
+ {
+ /* consider 0 blocks read a failure */
+ result.status = ARS_ERROR;
+ result.id = PGAIO_HCB_MD_READV;
+ result.error_data = 0;
+
+ md_readv_report(result, td, LOG);
+
+ return result;
+ }
+
+ if (result.status != ARS_ERROR &&
+ result.result < td->smgr.nblocks)
+ {
+ /* partial reads should be retried at upper level */
+ result.status = ARS_PARTIAL;
+ result.id = PGAIO_HCB_MD_READV;
+ }
+
+ return result;
+}
+
+/*
+ * AIO error reporting callback for mdstartreadv().
+ */
+static void
+md_readv_report(PgAioResult result, const PgAioTargetData *td, int elevel)
+{
+ RelPathStr path;
+
+ path = relpathbackend(td->smgr.rlocator,
+ td->smgr.is_temp ? MyProcNumber : INVALID_PROC_NUMBER,
+ td->smgr.forkNum);
+
+ if (result.error_data != 0)
+ {
+ errno = result.error_data; /* for errcode_for_file_access() */
+
+ ereport(elevel,
+ errcode_for_file_access(),
+ errmsg("could not read blocks %u..%u in file \"%s\": %m",
+ td->smgr.blockNum,
+ td->smgr.blockNum + td->smgr.nblocks,
+ path.str
+ )
+ );
+ }
+ else
+ {
+ /*
+ * NB: This will typically only be output in debug messages, while
+ * retrying a partial IO.
+ */
+ ereport(elevel,
+ errcode(ERRCODE_DATA_CORRUPTED),
+ errmsg("could not read blocks %u..%u in file \"%s\": read only %zu of %zu bytes",
+ td->smgr.blockNum,
+ td->smgr.blockNum + td->smgr.nblocks - 1,
+ path.str,
+ result.result * (size_t) BLCKSZ,
+ td->smgr.nblocks * (size_t) BLCKSZ
+ )
+ );
+ }
+}
diff --git a/src/backend/storage/smgr/smgr.c b/src/backend/storage/smgr/smgr.c
index ebe35c04de5..ea8b42ee4e8 100644
--- a/src/backend/storage/smgr/smgr.c
+++ b/src/backend/storage/smgr/smgr.c
@@ -53,6 +53,7 @@
#include "access/xlogutils.h"
#include "lib/ilist.h"
+#include "storage/aio.h"
#include "storage/bufmgr.h"
#include "storage/ipc.h"
#include "storage/md.h"
@@ -93,6 +94,10 @@ typedef struct f_smgr
void (*smgr_readv) (SMgrRelation reln, ForkNumber forknum,
BlockNumber blocknum,
void **buffers, BlockNumber nblocks);
+ void (*smgr_startreadv) (struct PgAioHandle *ioh,
+ SMgrRelation reln, ForkNumber forknum,
+ BlockNumber blocknum,
+ void **buffers, BlockNumber nblocks);
void (*smgr_writev) (SMgrRelation reln, ForkNumber forknum,
BlockNumber blocknum,
const void **buffers, BlockNumber nblocks,
@@ -104,6 +109,7 @@ typedef struct f_smgr
BlockNumber old_blocks, BlockNumber nblocks);
void (*smgr_immedsync) (SMgrRelation reln, ForkNumber forknum);
void (*smgr_registersync) (SMgrRelation reln, ForkNumber forknum);
+ int (*smgr_fd) (SMgrRelation reln, ForkNumber forknum, BlockNumber blocknum, uint32 *off);
} f_smgr;
static const f_smgr smgrsw[] = {
@@ -121,12 +127,14 @@ static const f_smgr smgrsw[] = {
.smgr_prefetch = mdprefetch,
.smgr_maxcombine = mdmaxcombine,
.smgr_readv = mdreadv,
+ .smgr_startreadv = mdstartreadv,
.smgr_writev = mdwritev,
.smgr_writeback = mdwriteback,
.smgr_nblocks = mdnblocks,
.smgr_truncate = mdtruncate,
.smgr_immedsync = mdimmedsync,
.smgr_registersync = mdregistersync,
+ .smgr_fd = mdfd,
}
};
@@ -145,6 +153,16 @@ static void smgrshutdown(int code, Datum arg);
static void smgrdestroy(SMgrRelation reln);
+static void smgr_aio_reopen(PgAioHandle *ioh);
+static char *smgr_aio_describe_identity(const PgAioTargetData *sd);
+
+const struct PgAioTargetInfo aio_smgr_target_info = {
+ .name = "smgr",
+ .reopen = smgr_aio_reopen,
+ .describe_identity = smgr_aio_describe_identity,
+};
+
+
/*
* smgrinit(), smgrshutdown() -- Initialize or shut down storage
* managers.
@@ -623,6 +641,22 @@ smgrreadv(SMgrRelation reln, ForkNumber forknum, BlockNumber blocknum,
nblocks);
}
+/*
+ * smgrstartreadv() -- asynchronous version of smgrreadv()
+ *
+ * This starts an asynchronous readv IO with the IO handle `ioh`. Other than
+ * `ioh` all parameters are the same as smgrreadv().
+ */
+void
+smgrstartreadv(struct PgAioHandle *ioh,
+ SMgrRelation reln, ForkNumber forknum, BlockNumber blocknum,
+ void **buffers, BlockNumber nblocks)
+{
+ smgrsw[reln->smgr_which].smgr_startreadv(ioh,
+ reln, forknum, blocknum, buffers,
+ nblocks);
+}
+
/*
* smgrwritev() -- Write the supplied buffers out.
*
@@ -819,6 +853,12 @@ smgrimmedsync(SMgrRelation reln, ForkNumber forknum)
smgrsw[reln->smgr_which].smgr_immedsync(reln, forknum);
}
+int
+smgrfd(SMgrRelation reln, ForkNumber forknum, BlockNumber blocknum, uint32 *off)
+{
+ return smgrsw[reln->smgr_which].smgr_fd(reln, forknum, blocknum, off);
+}
+
/*
* AtEOXact_SMgr
*
@@ -847,3 +887,69 @@ ProcessBarrierSmgrRelease(void)
smgrreleaseall();
return true;
}
+
+void
+pgaio_io_set_target_smgr(PgAioHandle *ioh,
+ struct SMgrRelationData *smgr,
+ ForkNumber forknum,
+ BlockNumber blocknum,
+ int nblocks,
+ bool skip_fsync)
+{
+ PgAioTargetData *sd = pgaio_io_get_target_data(ioh);
+
+ pgaio_io_set_target(ioh, PGAIO_TID_SMGR);
+
+ /* backend is implied via IO owner */
+ sd->smgr.rlocator = smgr->smgr_rlocator.locator;
+ sd->smgr.forkNum = forknum;
+ sd->smgr.blockNum = blocknum;
+ sd->smgr.nblocks = nblocks;
+ sd->smgr.is_temp = SmgrIsTemp(smgr);
+ /* Temp relations should never be fsync'd */
+ sd->smgr.skip_fsync = skip_fsync && !SmgrIsTemp(smgr);
+}
+
+static void
+smgr_aio_reopen(PgAioHandle *ioh)
+{
+ PgAioTargetData *sd = pgaio_io_get_target_data(ioh);
+ PgAioOpData *od = pgaio_io_get_op_data(ioh);
+ SMgrRelation reln;
+ ProcNumber procno;
+ uint32 off;
+
+ if (sd->smgr.is_temp)
+ procno = pgaio_io_get_owner(ioh);
+ else
+ procno = INVALID_PROC_NUMBER;
+
+ reln = smgropen(sd->smgr.rlocator, procno);
+ od->read.fd = smgrfd(reln, sd->smgr.forkNum, sd->smgr.blockNum, &off);
+ Assert(off == od->read.offset);
+}
+
+static char *
+smgr_aio_describe_identity(const PgAioTargetData *sd)
+{
+ RelPathStr path;
+ char *desc;
+
+ path = relpathbackend(sd->smgr.rlocator,
+ sd->smgr.is_temp ? MyProcNumber : INVALID_PROC_NUMBER,
+ sd->smgr.forkNum);
+
+ if (sd->smgr.nblocks == 0)
+ desc = psprintf(_("file \"%s\""), path.str);
+ else if (sd->smgr.nblocks == 1)
+ desc = psprintf(_("block %u in file \"%s\""),
+ sd->smgr.blockNum,
+ path.str);
+ else
+ desc = psprintf(_("blocks %u..%u in file \"%s\""),
+ sd->smgr.blockNum,
+ sd->smgr.blockNum + sd->smgr.nblocks - 1,
+ path.str);
+
+ return desc;
+}
--
2.48.1.76.g4e746b1a31.dirty
[text/x-diff] v2.5-0009-aio-Add-pg_aios-view.patch (9.8K, ../../vz5i2x2wkjjpp6l2z4g4l3umpxktrxsejgdwefi7n6kzw666h3@rnhlvlpadig7/10-v2.5-0009-aio-Add-pg_aios-view.patch)
download | inline diff:
From cdb3e408eb390222bf68c07f07a44b90762f4f82 Mon Sep 17 00:00:00 2001
From: Andres Freund <andres@anarazel.de>
Date: Wed, 22 Jan 2025 13:44:40 -0500
Subject: [PATCH v2.5 09/30] aio: Add pg_aios view
Author:
Reviewed-by:
Discussion: https://postgr.es/m/
Backpatch:
---
src/include/catalog/pg_proc.dat | 10 ++
src/backend/catalog/system_views.sql | 3 +
src/backend/storage/aio/Makefile | 1 +
src/backend/storage/aio/aio_funcs.c | 222 +++++++++++++++++++++++++++
src/backend/storage/aio/meson.build | 1 +
src/test/regress/expected/rules.out | 17 ++
6 files changed, 254 insertions(+)
create mode 100644 src/backend/storage/aio/aio_funcs.c
diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat
index cd9422d0bac..da72f1da78e 100644
--- a/src/include/catalog/pg_proc.dat
+++ b/src/include/catalog/pg_proc.dat
@@ -12450,4 +12450,14 @@
proargtypes => 'int4',
prosrc => 'gist_stratnum_common' },
+# AIO related functions
+{ oid => '9200', descr => 'information about in-progress asynchronous IOs',
+ proname => 'pg_get_aios', prorows => '100', proretset => 't',
+ provolatile => 'v', proparallel => 'r', prorettype => 'record', proargtypes => '',
+ proallargtypes => '{int4,int4,int8,text,text,int8,int8,text,int2,int4,text,text,text,bool,bool,bool}',
+ proargmodes => '{o,o,o,o,o,o,o,o,o,o,o,o,o,o,o,o}',
+ proargnames => '{pid,io_id,io_generation,state,operation,offset,length,target,handle_data_len,raw_result,result,error_desc,target_desc,f_sync,f_localmem,f_buffered}',
+ prosrc => 'pg_get_aios' },
+
+
]
diff --git a/src/backend/catalog/system_views.sql b/src/backend/catalog/system_views.sql
index a4d2cfdcaf5..73902a763d1 100644
--- a/src/backend/catalog/system_views.sql
+++ b/src/backend/catalog/system_views.sql
@@ -1390,3 +1390,6 @@ CREATE VIEW pg_stat_subscription_stats AS
CREATE VIEW pg_wait_events AS
SELECT * FROM pg_get_wait_events();
+
+CREATE VIEW pg_aios AS
+ SELECT * FROM pg_get_aios();
diff --git a/src/backend/storage/aio/Makefile b/src/backend/storage/aio/Makefile
index c06c50771e0..3f2469cc399 100644
--- a/src/backend/storage/aio/Makefile
+++ b/src/backend/storage/aio/Makefile
@@ -11,6 +11,7 @@ include $(top_builddir)/src/Makefile.global
OBJS = \
aio.o \
aio_callback.o \
+ aio_funcs.o \
aio_init.o \
aio_io.o \
aio_target.o \
diff --git a/src/backend/storage/aio/aio_funcs.c b/src/backend/storage/aio/aio_funcs.c
new file mode 100644
index 00000000000..cb4f3d29201
--- /dev/null
+++ b/src/backend/storage/aio/aio_funcs.c
@@ -0,0 +1,222 @@
+/*-------------------------------------------------------------------------
+ *
+ * aio_funcs.c
+ * AIO - SQL interface for AIO
+ *
+ *
+ * Portions Copyright (c) 1996-2025, PostgreSQL Global Development Group
+ * Portions Copyright (c) 1994, Regents of the University of California
+ *
+ * IDENTIFICATION
+ * src/backend/storage/aio/aio_funcs.c
+ *
+ *-------------------------------------------------------------------------
+ */
+
+#include "postgres.h"
+
+#include "storage/aio.h"
+#include "storage/aio_internal.h"
+#include "utils/builtins.h"
+#include "funcapi.h"
+#include "storage/proc.h"
+
+
+/*
+ * Byte length of an iovec.
+ */
+static size_t
+iov_byte_length(const struct iovec *iov, int cnt)
+{
+ size_t len = 0;
+
+ for (int i = 0; i < cnt; i++)
+ {
+ len += iov[i].iov_len;
+ }
+
+ return len;
+}
+
+Datum
+pg_get_aios(PG_FUNCTION_ARGS)
+{
+ ReturnSetInfo *rsinfo = (ReturnSetInfo *) fcinfo->resultinfo;
+
+ InitMaterializedSRF(fcinfo, 0);
+
+#define PG_GET_AIOS_COLS 16
+
+ for (uint64 i = 0; i < pgaio_ctl->io_handle_count; i++)
+ {
+ PgAioHandle *live_ioh = &pgaio_ctl->io_handles[i];
+ uint32 ioh_id = pgaio_io_get_id(live_ioh);
+ Datum values[PG_GET_AIOS_COLS] = {0};
+ bool nulls[PG_GET_AIOS_COLS] = {0};
+ ProcNumber owner;
+ PGPROC *owner_proc;
+ int32 owner_pid;
+ PgAioHandleState start_state;
+ uint64 start_generation;
+ PgAioHandle ioh_copy;
+ struct iovec iov_copy[PG_IOV_MAX];
+
+retry:
+
+ /*
+ * There is no lock that could prevent the state of the IO to advance
+ * concurrently - and we don't want to introduce one, as that would
+ * introduce atomics into a very common path. Instead we
+ *
+ * 1) determine the state + generation of the IO
+ *
+ * 2) copy the IO to local memory
+ *
+ * 3) check if state and generation of the IO changed
+ */
+
+ /* 1) from above */
+ start_generation = live_ioh->generation;
+ pg_read_barrier();
+ start_state = live_ioh->state;
+
+ if (start_state == PGAIO_HS_IDLE)
+ continue;
+
+ /* 2) from above */
+ memcpy(&ioh_copy, live_ioh, sizeof(PgAioHandle));
+
+ /*
+ * Safe to copy even if no iovec is used - we always reserve the
+ * required space.
+ */
+ memcpy(&iov_copy, &pgaio_ctl->iovecs[ioh_copy.iovec_off],
+ PG_IOV_MAX * sizeof(struct iovec));
+
+ /*
+ * Copy information about owner before 3) below, if the process exited
+ * it'd have to wait for the IO to finish first, which we would detect
+ * in 3).
+ */
+ owner = ioh_copy.owner_procno;
+ owner_proc = GetPGProcByNumber(owner);
+ owner_pid = owner_proc->pid;
+
+ /* 3) from above */
+ pg_read_barrier();
+
+ /*
+ * The IO completed and a new one was started with the same ID. Don't
+ * display it - it really started after this function was called.
+ * There be a risk of a livelock if we just retried endlessly, if IOs
+ * complete very quickly.
+ */
+ if (live_ioh->generation != start_generation)
+ continue;
+
+ /*
+ * The IOs state changed while we were "rendering" it. Just start from
+ * scratch. There's no risk of a livelock here, as an IO has a limited
+ * sets of states it can be in, and state changes go only in a single
+ * direction.
+ */
+ if (live_ioh->state != start_state)
+ goto retry;
+
+ /*
+ * Now that we have copied the IO into local memory and checked that
+ * it's still in the same state, we are not allowed to access "live"
+ * memory anymore. To make it slightly easier to catch such cases, set
+ * the "live" pointers to NULL.
+ */
+ live_ioh = NULL;
+ owner_proc = NULL;
+
+
+ /* column: owning pid */
+ if (owner_pid != 0)
+ values[0] = Int32GetDatum(owner_pid);
+ else
+ nulls[0] = false;
+
+ /* column: IO's id */
+ values[1] = ioh_id;
+
+ /* column: IO's generation */
+ values[2] = Int64GetDatum(start_generation);
+
+ /* column: IO's state */
+ values[3] = CStringGetTextDatum(pgaio_io_get_state_name(&ioh_copy));
+
+ /*
+ * If the IO is in PGAIO_HS_HANDED_OUT state, none of it's fields are
+ * valid yet (or are in the process of being set). Therefore we don't
+ * want to display any other columns.
+ */
+ if (start_state == PGAIO_HS_HANDED_OUT)
+ {
+ memset(nulls + 4, 1, (lengthof(nulls) - 4) * sizeof(bool));
+ goto display;
+ }
+
+ /* column: IO's operation */
+ values[4] = CStringGetTextDatum(pgaio_io_get_op_name(&ioh_copy));
+
+ /* columns: details about the IO's operation */
+ switch (ioh_copy.op)
+ {
+ case PGAIO_OP_INVALID:
+ nulls[5] = true;
+ nulls[6] = true;
+ break;
+ case PGAIO_OP_READV:
+ values[5] = Int64GetDatum(ioh_copy.op_data.read.offset);
+ values[6] =
+ Int64GetDatum(iov_byte_length(iov_copy, ioh_copy.op_data.read.iov_length));
+ break;
+ case PGAIO_OP_WRITEV:
+ values[5] = Int64GetDatum(ioh_copy.op_data.write.offset);
+ values[6] =
+ Int64GetDatum(iov_byte_length(iov_copy, ioh_copy.op_data.write.iov_length));
+ break;
+ }
+
+ /* column: IO's target */
+ values[7] = CStringGetTextDatum(pgaio_io_get_target_name(&ioh_copy));
+
+ /* column: length of IO's data array */
+ values[8] = Int16GetDatum(ioh_copy.handle_data_len);
+
+ /* column: raw result (i.e. some form of syscall return value) */
+ if (start_state == PGAIO_HS_COMPLETED_IO
+ || start_state == PGAIO_HS_COMPLETED_SHARED)
+ values[9] = Int32GetDatum(ioh_copy.result);
+ else
+ nulls[9] = true;
+
+ /*
+ * column: result in the higher level representation (unknown if not
+ * finished
+ */
+ values[10] =
+ CStringGetTextDatum(pgaio_result_status_string(ioh_copy.distilled_result.status));
+
+ /* column: error description */
+ /* AFIXME: implement */
+ nulls[11] = true;
+
+ /* column: target description */
+ values[12] = CStringGetTextDatum(pgaio_io_get_target_description(&ioh_copy));
+
+ /* columns: one for each flag */
+ values[13] = BoolGetDatum(ioh_copy.flags & PGAIO_HF_SYNCHRONOUS);
+ values[14] = BoolGetDatum(ioh_copy.flags & PGAIO_HF_REFERENCES_LOCAL);
+ values[15] = BoolGetDatum(ioh_copy.flags & PGAIO_HF_BUFFERED);
+
+display:
+
+ tuplestore_putvalues(rsinfo->setResult, rsinfo->setDesc, values, nulls);
+ }
+
+ return (Datum) 0;
+}
diff --git a/src/backend/storage/aio/meson.build b/src/backend/storage/aio/meson.build
index 2f0f03d8071..da6df2d3654 100644
--- a/src/backend/storage/aio/meson.build
+++ b/src/backend/storage/aio/meson.build
@@ -3,6 +3,7 @@
backend_sources += files(
'aio.c',
'aio_callback.c',
+ 'aio_funcs.c',
'aio_init.c',
'aio_io.c',
'aio_target.c',
diff --git a/src/test/regress/expected/rules.out b/src/test/regress/expected/rules.out
index 62f69ac20b2..e8ab411cd92 100644
--- a/src/test/regress/expected/rules.out
+++ b/src/test/regress/expected/rules.out
@@ -1286,6 +1286,23 @@ drop table cchild;
SELECT viewname, definition FROM pg_views
WHERE schemaname = 'pg_catalog'
ORDER BY viewname;
+pg_aios| SELECT pid,
+ io_id,
+ io_generation,
+ state,
+ operation,
+ "offset",
+ length,
+ target,
+ handle_data_len,
+ raw_result,
+ result,
+ error_desc,
+ target_desc,
+ f_sync,
+ f_localmem,
+ f_buffered
+ FROM pg_get_aios() pg_get_aios(pid, io_id, io_generation, state, operation, "offset", length, target, handle_data_len, raw_result, result, error_desc, target_desc, f_sync, f_localmem, f_buffered);
pg_available_extension_versions| SELECT e.name,
e.version,
(x.extname IS NOT NULL) AS installed,
--
2.48.1.76.g4e746b1a31.dirty
[text/x-diff] v2.5-0010-Refactor-read_stream.c-s-circular-arithmetic.patch (4.6K, ../../vz5i2x2wkjjpp6l2z4g4l3umpxktrxsejgdwefi7n6kzw666h3@rnhlvlpadig7/11-v2.5-0010-Refactor-read_stream.c-s-circular-arithmetic.patch)
download | inline diff:
From ec605c2c1b276560e457c40e4274af653d869773 Mon Sep 17 00:00:00 2001
From: Thomas Munro <thomas.munro@gmail.com>
Date: Sat, 15 Feb 2025 14:47:25 +1300
Subject: [PATCH v2.5 10/30] Refactor read_stream.c's circular arithmetic.
Several places have open-coded circular index arithmetic. Make some
common functions for better readability and consistent assertion
checking.
This avoids adding yet more open-coded duplication in later patches, and
standardizes on the vocabulary "advance" and "retreat" as used elsewhere
in PostgreSQL.
---
src/backend/storage/aio/read_stream.c | 78 +++++++++++++++++++++------
1 file changed, 61 insertions(+), 17 deletions(-)
diff --git a/src/backend/storage/aio/read_stream.c b/src/backend/storage/aio/read_stream.c
index 04bdb5e6d4b..dee51fa85a9 100644
--- a/src/backend/storage/aio/read_stream.c
+++ b/src/backend/storage/aio/read_stream.c
@@ -224,6 +224,55 @@ read_stream_unget_block(ReadStream *stream, BlockNumber blocknum)
stream->buffered_blocknum = blocknum;
}
+/*
+ * Increment index, wrapping around at queue size.
+ */
+static inline void
+read_stream_index_advance(ReadStream *stream, int16 *index)
+{
+ Assert(*index >= 0);
+ Assert(*index < stream->queue_size);
+
+ *index += 1;
+ if (*index == stream->queue_size)
+ *index = 0;
+}
+
+/*
+ * Increment index by n, wrapping around at queue size.
+ */
+static inline void
+read_stream_index_advance_n(ReadStream *stream, int16 *index, int16 n)
+{
+ Assert(*index >= 0);
+ Assert(*index < stream->queue_size);
+ Assert(n <= MAX_IO_COMBINE_LIMIT);
+
+ *index += n;
+ if (*index >= stream->queue_size)
+ *index -= stream->queue_size;
+
+ Assert(*index >= 0);
+ Assert(*index < stream->queue_size);
+}
+
+#if defined(CLOBBER_FREED_MEMORY) || defined(USE_VALGRIND)
+/*
+ * Decrement index, wrapping around at queue size.
+ */
+static inline void
+read_stream_index_retreat(ReadStream *stream, int16 *index)
+{
+ Assert(*index >= 0);
+ Assert(*index < stream->queue_size);
+
+ if (*index == 0)
+ *index = stream->queue_size - 1;
+ else
+ *index -= 1;
+}
+#endif
+
static void
read_stream_start_pending_read(ReadStream *stream, bool suppress_advice)
{
@@ -302,11 +351,8 @@ read_stream_start_pending_read(ReadStream *stream, bool suppress_advice)
&stream->buffers[stream->queue_size],
sizeof(stream->buffers[0]) * overflow);
- /* Compute location of start of next read, without using % operator. */
- buffer_index += nblocks;
- if (buffer_index >= stream->queue_size)
- buffer_index -= stream->queue_size;
- Assert(buffer_index >= 0 && buffer_index < stream->queue_size);
+ /* Move to the location of start of next read. */
+ read_stream_index_advance_n(stream, &buffer_index, nblocks);
stream->next_buffer_index = buffer_index;
/* Adjust the pending read to cover the remaining portion, if any. */
@@ -334,12 +380,12 @@ read_stream_look_ahead(ReadStream *stream, bool suppress_advice)
/*
* See which block the callback wants next in the stream. We need to
* compute the index of the Nth block of the pending read including
- * wrap-around, but we don't want to use the expensive % operator.
+ * wrap-around.
*/
- buffer_index = stream->next_buffer_index + stream->pending_read_nblocks;
- if (buffer_index >= stream->queue_size)
- buffer_index -= stream->queue_size;
- Assert(buffer_index >= 0 && buffer_index < stream->queue_size);
+ buffer_index = stream->next_buffer_index;
+ read_stream_index_advance_n(stream,
+ &buffer_index,
+ stream->pending_read_nblocks);
per_buffer_data = get_per_buffer_data(stream, buffer_index);
blocknum = read_stream_get_block(stream, per_buffer_data);
if (blocknum == InvalidBlockNumber)
@@ -781,12 +827,12 @@ read_stream_next_buffer(ReadStream *stream, void **per_buffer_data)
*/
if (stream->per_buffer_data)
{
+ int16 index;
void *per_buffer_data;
- per_buffer_data = get_per_buffer_data(stream,
- oldest_buffer_index == 0 ?
- stream->queue_size - 1 :
- oldest_buffer_index - 1);
+ index = oldest_buffer_index;
+ read_stream_index_retreat(stream, &index);
+ per_buffer_data = get_per_buffer_data(stream, index);
#if defined(CLOBBER_FREED_MEMORY)
/* This also tells Valgrind the memory is "noaccess". */
@@ -804,9 +850,7 @@ read_stream_next_buffer(ReadStream *stream, void **per_buffer_data)
stream->pinned_buffers--;
/* Advance oldest buffer, with wrap-around. */
- stream->oldest_buffer_index++;
- if (stream->oldest_buffer_index == stream->queue_size)
- stream->oldest_buffer_index = 0;
+ read_stream_index_advance(stream, &stream->oldest_buffer_index);
/* Prepare for the next call. */
read_stream_look_ahead(stream, false);
--
2.48.1.76.g4e746b1a31.dirty
[text/x-diff] v2.5-0011-Improve-buffer-pool-API-for-per-backend-pin-lim.patch (6.4K, ../../vz5i2x2wkjjpp6l2z4g4l3umpxktrxsejgdwefi7n6kzw666h3@rnhlvlpadig7/12-v2.5-0011-Improve-buffer-pool-API-for-per-backend-pin-lim.patch)
download | inline diff:
From d372fbde22ac9011659bd99c7ac6e72dd54eeec7 Mon Sep 17 00:00:00 2001
From: Thomas Munro <thomas.munro@gmail.com>
Date: Fri, 24 Jan 2025 10:59:39 +1300
Subject: [PATCH v2.5 11/30] Improve buffer pool API for per-backend pin
limits.
Previously the support functions assumed that you needed one additional
pin to make progress, and could optionally use some more. Add a couple
more functions for callers that want to know:
* what the maximum possible number could be, for space planning
purposes, called the "soft pin limit"
* how many additional pins they could acquire right now, without the
special case allowing one pin (ie for users that already hold pins and
can already make progress even if zero extra pins are available now)
These APIs are better suited to read_stream.c, which will be adjusted in
a follow-up patch. Also move the computation of the each backend's fair
share of the buffer pool to backend initialization time, since the
answer doesn't change and we don't want to perform a division operation
every time we compute availability.
---
src/include/storage/bufmgr.h | 4 ++
src/backend/storage/buffer/bufmgr.c | 75 ++++++++++++++++++++-------
src/backend/storage/buffer/localbuf.c | 16 ++++++
3 files changed, 77 insertions(+), 18 deletions(-)
diff --git a/src/include/storage/bufmgr.h b/src/include/storage/bufmgr.h
index 7c1e4316dde..597ecb97897 100644
--- a/src/include/storage/bufmgr.h
+++ b/src/include/storage/bufmgr.h
@@ -290,6 +290,10 @@ extern bool HoldingBufferPinThatDelaysRecovery(void);
extern bool BgBufferSync(struct WritebackContext *wb_context);
+extern uint32 GetSoftPinLimit(void);
+extern uint32 GetSoftLocalPinLimit(void);
+extern uint32 GetAdditionalPinLimit(void);
+extern uint32 GetAdditionalLocalPinLimit(void);
extern void LimitAdditionalPins(uint32 *additional_pins);
extern void LimitAdditionalLocalPins(uint32 *additional_pins);
diff --git a/src/backend/storage/buffer/bufmgr.c b/src/backend/storage/buffer/bufmgr.c
index 7915ed624c1..11c146763db 100644
--- a/src/backend/storage/buffer/bufmgr.c
+++ b/src/backend/storage/buffer/bufmgr.c
@@ -211,6 +211,8 @@ static int32 PrivateRefCountOverflowed = 0;
static uint32 PrivateRefCountClock = 0;
static PrivateRefCountEntry *ReservedRefCountEntry = NULL;
+static uint32 MaxProportionalPins;
+
static void ReservePrivateRefCountEntry(void);
static PrivateRefCountEntry *NewPrivateRefCountEntry(Buffer buffer);
static PrivateRefCountEntry *GetPrivateRefCountEntry(Buffer buffer, bool do_move);
@@ -2097,6 +2099,46 @@ again:
return buf;
}
+/*
+ * Return the maximum number of buffer than this backend should try to pin at
+ * once, to avoid pinning more than its fair share. This is the highest value
+ * that GetAdditionalPinLimit() and LimitAdditionalPins() could ever return.
+ *
+ * It's called a soft limit because nothing stops a backend from trying to
+ * acquire more pins than this this with ReadBuffer(), but code that wants more
+ * for I/O optimizations should respect this per-backend limit when it can
+ * still make progress without them.
+ */
+uint32
+GetSoftPinLimit(void)
+{
+ return MaxProportionalPins;
+}
+
+/*
+ * Return the maximum number of additional buffers that this backend should
+ * pin if it wants to stay under the per-backend soft limit, considering the
+ * number of buffers it has already pinned.
+ */
+uint32
+GetAdditionalPinLimit(void)
+{
+ uint32 estimated_pins_held;
+
+ /*
+ * We get the number of "overflowed" pins for free, but don't know the
+ * number of pins in PrivateRefCountArray. The cost of calculating that
+ * exactly doesn't seem worth it, so just assume the max.
+ */
+ estimated_pins_held = PrivateRefCountOverflowed + REFCOUNT_ARRAY_ENTRIES;
+
+ /* Is this backend already holding more than its fair share? */
+ if (estimated_pins_held > MaxProportionalPins)
+ return 0;
+
+ return MaxProportionalPins - estimated_pins_held;
+}
+
/*
* Limit the number of pins a batch operation may additionally acquire, to
* avoid running out of pinnable buffers.
@@ -2112,28 +2154,15 @@ again:
void
LimitAdditionalPins(uint32 *additional_pins)
{
- uint32 max_backends;
- int max_proportional_pins;
+ uint32 limit;
if (*additional_pins <= 1)
return;
- max_backends = MaxBackends + NUM_AUXILIARY_PROCS;
- max_proportional_pins = NBuffers / max_backends;
-
- /*
- * Subtract the approximate number of buffers already pinned by this
- * backend. We get the number of "overflowed" pins for free, but don't
- * know the number of pins in PrivateRefCountArray. The cost of
- * calculating that exactly doesn't seem worth it, so just assume the max.
- */
- max_proportional_pins -= PrivateRefCountOverflowed + REFCOUNT_ARRAY_ENTRIES;
-
- if (max_proportional_pins <= 0)
- max_proportional_pins = 1;
-
- if (*additional_pins > max_proportional_pins)
- *additional_pins = max_proportional_pins;
+ limit = GetAdditionalPinLimit();
+ limit = Max(limit, 1);
+ if (limit < *additional_pins)
+ *additional_pins = limit;
}
/*
@@ -3575,6 +3604,16 @@ InitBufferManagerAccess(void)
{
HASHCTL hash_ctl;
+ /*
+ * The soft limit on the number of pins each backend should respect, bast
+ * on shared_buffers and the maximum number of connections possible.
+ * That's very pessimistic, but outside toy-sized shared_buffers it should
+ * allow plenty of pins. Higher level code that pins non-trivial numbers
+ * of buffers should use LimitAdditionalPins() or GetAdditionalPinLimit()
+ * to stay under this limit.
+ */
+ MaxProportionalPins = NBuffers / (MaxBackends + NUM_AUXILIARY_PROCS);
+
memset(&PrivateRefCountArray, 0, sizeof(PrivateRefCountArray));
hash_ctl.keysize = sizeof(int32);
diff --git a/src/backend/storage/buffer/localbuf.c b/src/backend/storage/buffer/localbuf.c
index 80b83444eb2..5378ba84316 100644
--- a/src/backend/storage/buffer/localbuf.c
+++ b/src/backend/storage/buffer/localbuf.c
@@ -286,6 +286,22 @@ GetLocalVictimBuffer(void)
return BufferDescriptorGetBuffer(bufHdr);
}
+/* see GetSoftPinLimit() */
+uint32
+GetSoftLocalPinLimit(void)
+{
+ /* Every backend has its own temporary buffers, and can pin them all. */
+ return num_temp_buffers;
+}
+
+/* see GetAdditionalPinLimit() */
+uint32
+GetAdditionalLocalPinLimit(void)
+{
+ Assert(NLocalPinnedBuffers <= num_temp_buffers);
+ return num_temp_buffers - NLocalPinnedBuffers;
+}
+
/* see LimitAdditionalPins() */
void
LimitAdditionalLocalPins(uint32 *additional_pins)
--
2.48.1.76.g4e746b1a31.dirty
[text/x-diff] v2.5-0012-Respect-pin-limits-accurately-in-read_stream.c.patch (9.9K, ../../vz5i2x2wkjjpp6l2z4g4l3umpxktrxsejgdwefi7n6kzw666h3@rnhlvlpadig7/13-v2.5-0012-Respect-pin-limits-accurately-in-read_stream.c.patch)
download | inline diff:
From fecdca253a1231debaea900df43137508e35c8e1 Mon Sep 17 00:00:00 2001
From: Thomas Munro <thomas.munro@gmail.com>
Date: Fri, 24 Jan 2025 23:52:53 +1300
Subject: [PATCH v2.5 12/30] Respect pin limits accurately in read_stream.c.
Read streams pin multiple buffers at once as required to combine I/O.
This also avoids having to unpin and repin later when issuing read-ahead
advice, and will be needed for proposed work that starts "real"
asynchronous I/O.
To avoid pinning too much of the buffer pool at once, we previously used
LimitAdditionalBuffers() to avoid pinning more than this backend's fair
share of the pool as a cap. The coding was a naive and only checked the
cap once at stream initialization.
This commit moves the check to the time of use with new bufmgr APIs from
an earlier commit, since the result might change later due to pins
acquired later outside this stream. No extra CPU cycles are added to
the all-buffered fast-path code (it only pins one buffer at a time), but
the I/O-starting path now re-checks the limit every time using simple
arithmetic.
In practice it was difficult to exceed the limit, but you could contrive
a workload to do it using multiple CURSORs and FETCHing from sequential
scans in round-robin fashion, so that each underlying stream computes
its limit before all the others have ramped up to their full look-ahead
distance. Therefore, no back-patch for now.
Per code review from Andres, in the course of his AIO work.
Reported-by: Andres Freund <andres@anarazel.de>
---
src/backend/storage/aio/read_stream.c | 111 ++++++++++++++++++++++----
1 file changed, 95 insertions(+), 16 deletions(-)
diff --git a/src/backend/storage/aio/read_stream.c b/src/backend/storage/aio/read_stream.c
index dee51fa85a9..6a39bd1d92d 100644
--- a/src/backend/storage/aio/read_stream.c
+++ b/src/backend/storage/aio/read_stream.c
@@ -115,6 +115,7 @@ struct ReadStream
int16 pinned_buffers;
int16 distance;
bool advice_enabled;
+ bool temporary;
/*
* One-block buffer to support 'ungetting' a block number, to resolve flow
@@ -274,7 +275,9 @@ read_stream_index_retreat(ReadStream *stream, int16 *index)
#endif
static void
-read_stream_start_pending_read(ReadStream *stream, bool suppress_advice)
+read_stream_start_pending_read(ReadStream *stream,
+ int16 buffer_limit,
+ bool suppress_advice)
{
bool need_wait;
int nblocks;
@@ -308,10 +311,14 @@ read_stream_start_pending_read(ReadStream *stream, bool suppress_advice)
else
flags = 0;
- /* We say how many blocks we want to read, but may be smaller on return. */
+ /*
+ * We say how many blocks we want to read, but may be smaller on return.
+ * On memory-constrained systems we may be also have to ask for a smaller
+ * read ourselves.
+ */
buffer_index = stream->next_buffer_index;
io_index = stream->next_io_index;
- nblocks = stream->pending_read_nblocks;
+ nblocks = Min(buffer_limit, stream->pending_read_nblocks);
need_wait = StartReadBuffers(&stream->ios[io_index].op,
&stream->buffers[buffer_index],
stream->pending_read_blocknum,
@@ -360,11 +367,60 @@ read_stream_start_pending_read(ReadStream *stream, bool suppress_advice)
stream->pending_read_nblocks -= nblocks;
}
+/*
+ * How many more buffers could we use, while respecting the soft limit?
+ */
+static int16
+read_stream_get_buffer_limit(ReadStream *stream)
+{
+ uint32 buffers;
+
+ /* Check how many local or shared pins we could acquire. */
+ if (stream->temporary)
+ buffers = GetAdditionalLocalPinLimit();
+ else
+ buffers = GetAdditionalPinLimit();
+
+ /*
+ * Each stream is always allowed to try to acquire one pin if it doesn't
+ * hold one already. This is needed to guarantee progress, and just like
+ * the simple ReadBuffer() operation in code that is not using this stream
+ * API, if a buffer can't be pinned we'll raise an error when trying to
+ * pin, ie the buffer pool is simply too small for the workload.
+ */
+ if (buffers == 0 && stream->pinned_buffers == 0)
+ return 1;
+
+ /*
+ * Otherwise, see how many additional pins the backend can currently pin,
+ * which may be zero. As above, this only guarantees that this backend
+ * won't use more than its fair share if all backends can respect the soft
+ * limit, not that a pin can actually be acquired without error.
+ */
+ return Min(buffers, INT16_MAX);
+}
+
static void
read_stream_look_ahead(ReadStream *stream, bool suppress_advice)
{
+ int16 buffer_limit;
+
+ /*
+ * Check how many pins we could acquire now. We do this here rather than
+ * pushing it down into read_stream_start_pending_read(), because it
+ * allows more flexibility in behavior when we run out of allowed pins.
+ * Currently the policy is to start an I/O when we've run out of allowed
+ * pins only if we have to to make progress, and otherwise to stop looking
+ * ahead until more pins become available, so that we don't start issuing
+ * a lot of smaller I/Os, prefering to build the largest ones we can. This
+ * choice is debatable, but it should only really come up with the buffer
+ * pool/connection ratio is very constrained.
+ */
+ buffer_limit = read_stream_get_buffer_limit(stream);
+
while (stream->ios_in_progress < stream->max_ios &&
- stream->pinned_buffers + stream->pending_read_nblocks < stream->distance)
+ stream->pinned_buffers + stream->pending_read_nblocks <
+ Min(stream->distance, buffer_limit))
{
BlockNumber blocknum;
int16 buffer_index;
@@ -372,7 +428,9 @@ read_stream_look_ahead(ReadStream *stream, bool suppress_advice)
if (stream->pending_read_nblocks == io_combine_limit)
{
- read_stream_start_pending_read(stream, suppress_advice);
+ read_stream_start_pending_read(stream, buffer_limit,
+ suppress_advice);
+ buffer_limit = read_stream_get_buffer_limit(stream);
suppress_advice = false;
continue;
}
@@ -406,11 +464,12 @@ read_stream_look_ahead(ReadStream *stream, bool suppress_advice)
/* We have to start the pending read before we can build another. */
while (stream->pending_read_nblocks > 0)
{
- read_stream_start_pending_read(stream, suppress_advice);
+ read_stream_start_pending_read(stream, buffer_limit, suppress_advice);
+ buffer_limit = read_stream_get_buffer_limit(stream);
suppress_advice = false;
- if (stream->ios_in_progress == stream->max_ios)
+ if (stream->ios_in_progress == stream->max_ios || buffer_limit == 0)
{
- /* And we've hit the limit. Rewind, and stop here. */
+ /* And we've hit a limit. Rewind, and stop here. */
read_stream_unget_block(stream, blocknum);
return;
}
@@ -426,16 +485,17 @@ read_stream_look_ahead(ReadStream *stream, bool suppress_advice)
* limit, preferring to give it another chance to grow to full
* io_combine_limit size once more buffers have been consumed. However,
* if we've already reached io_combine_limit, or we've reached the
- * distance limit and there isn't anything pinned yet, or the callback has
- * signaled end-of-stream, we start the read immediately.
+ * distance limit or buffer limit and there isn't anything pinned yet, or
+ * the callback has signaled end-of-stream, we start the read immediately.
*/
if (stream->pending_read_nblocks > 0 &&
(stream->pending_read_nblocks == io_combine_limit ||
- (stream->pending_read_nblocks == stream->distance &&
+ ((stream->pending_read_nblocks == stream->distance ||
+ stream->pending_read_nblocks == buffer_limit) &&
stream->pinned_buffers == 0) ||
stream->distance == 0) &&
stream->ios_in_progress < stream->max_ios)
- read_stream_start_pending_read(stream, suppress_advice);
+ read_stream_start_pending_read(stream, buffer_limit, suppress_advice);
}
/*
@@ -464,6 +524,7 @@ read_stream_begin_impl(int flags,
int max_ios;
int strategy_pin_limit;
uint32 max_pinned_buffers;
+ uint32 max_possible_buffer_limit;
Oid tablespace_id;
/*
@@ -511,12 +572,23 @@ read_stream_begin_impl(int flags,
strategy_pin_limit = GetAccessStrategyPinLimit(strategy);
max_pinned_buffers = Min(strategy_pin_limit, max_pinned_buffers);
- /* Don't allow this backend to pin more than its share of buffers. */
+ /*
+ * Also limit by the maximum possible number of pins we could be allowed
+ * to acquire according to bufmgr. We may not be able to use them all due
+ * to other pins held by this backend, but we'll enforce the dynamic limit
+ * later when starting I/O.
+ */
if (SmgrIsTemp(smgr))
- LimitAdditionalLocalPins(&max_pinned_buffers);
+ max_possible_buffer_limit = GetSoftLocalPinLimit();
else
- LimitAdditionalPins(&max_pinned_buffers);
- Assert(max_pinned_buffers > 0);
+ max_possible_buffer_limit = GetSoftPinLimit();
+ max_pinned_buffers = Min(max_pinned_buffers, max_possible_buffer_limit);
+
+ /*
+ * The soft limit might be zero on a system configured with more
+ * connections than buffers. We need at least one.
+ */
+ max_pinned_buffers = Max(1, max_pinned_buffers);
/*
* We need one extra entry for buffers and per-buffer data, because users
@@ -576,6 +648,7 @@ read_stream_begin_impl(int flags,
stream->callback = callback;
stream->callback_private_data = callback_private_data;
stream->buffered_blocknum = InvalidBlockNumber;
+ stream->temporary = SmgrIsTemp(smgr);
/*
* Skip the initial ramp-up phase if the caller says we're going to be
@@ -704,6 +777,12 @@ read_stream_next_buffer(ReadStream *stream, void **per_buffer_data)
* arbitrary I/O entry (they're all free). We don't have to
* adjust pinned_buffers because we're transferring one to caller
* but pinning one more.
+ *
+ * In the fast path we don't need to check the pin limit. We're
+ * always allowed at least one pin so that progress can be made,
+ * and that's all we need here. Although two pins are momentarily
+ * held at the same time, the model used here is that the stream
+ * holds only one, and the other now belongs to the caller.
*/
if (likely(!StartReadBuffer(&stream->ios[0].op,
&stream->buffers[oldest_buffer_index],
--
2.48.1.76.g4e746b1a31.dirty
[text/x-diff] v2.5-0013-Support-buffer-forwarding-in-read_stream.c.patch (8.9K, ../../vz5i2x2wkjjpp6l2z4g4l3umpxktrxsejgdwefi7n6kzw666h3@rnhlvlpadig7/14-v2.5-0013-Support-buffer-forwarding-in-read_stream.c.patch)
download | inline diff:
From fffe68285db092c14a15cd6f1c0bd4e932c7527b Mon Sep 17 00:00:00 2001
From: Thomas Munro <thomas.munro@gmail.com>
Date: Thu, 30 Jan 2025 11:42:03 +1300
Subject: [PATCH v2.5 13/30] Support buffer forwarding in read_stream.c.
In preparation for a following change to the buffer manager, teach read
stream to keep track of buffers that were "forwarded" from one call to
StartReadBuffers() to the next.
Since StartReadBuffers() buffers argument will become an in/out
argument, we need to initialize the buffer queue entries with
InvalidBuffer. We don't want to do that up front, because we try to
keep stream initialization cheap and code that uses the fast path stays
in one single buffer queue element. Satisfy both goals by initializing
the queue incrementally on the first cycle.
---
src/backend/storage/aio/read_stream.c | 108 ++++++++++++++++++++++----
1 file changed, 94 insertions(+), 14 deletions(-)
diff --git a/src/backend/storage/aio/read_stream.c b/src/backend/storage/aio/read_stream.c
index 6a39bd1d92d..e58e0edf221 100644
--- a/src/backend/storage/aio/read_stream.c
+++ b/src/backend/storage/aio/read_stream.c
@@ -112,8 +112,10 @@ struct ReadStream
int16 ios_in_progress;
int16 queue_size;
int16 max_pinned_buffers;
+ int16 forwarded_buffers;
int16 pinned_buffers;
int16 distance;
+ int16 initialized_buffers;
bool advice_enabled;
bool temporary;
@@ -280,7 +282,9 @@ read_stream_start_pending_read(ReadStream *stream,
bool suppress_advice)
{
bool need_wait;
+ int requested_nblocks;
int nblocks;
+ int forwarded;
int flags;
int16 io_index;
int16 overflow;
@@ -312,13 +316,34 @@ read_stream_start_pending_read(ReadStream *stream,
flags = 0;
/*
- * We say how many blocks we want to read, but may be smaller on return.
- * On memory-constrained systems we may be also have to ask for a smaller
- * read ourselves.
+ * On buffer-constrained systems we may need to limit the I/O size by the
+ * available pin count.
*/
+ requested_nblocks = Min(buffer_limit, stream->pending_read_nblocks);
+ nblocks = requested_nblocks;
buffer_index = stream->next_buffer_index;
io_index = stream->next_io_index;
- nblocks = Min(buffer_limit, stream->pending_read_nblocks);
+
+ /*
+ * The first time around the queue we initialize it as we go, including
+ * the overflow zone, because otherwise the entries would appear as
+ * forwarded buffers. This avoids initializing the whole queue up front
+ * in cases where it is large but we don't ever use it due to the
+ * all-cached fast path or small scans.
+ */
+ while (stream->initialized_buffers < buffer_index + nblocks)
+ stream->buffers[stream->initialized_buffers++] = InvalidBuffer;
+
+ /*
+ * Start the I/O. Any buffers that are not InvalidBuffer will be
+ * interpreted as already pinned, forwarded by an earlier call to
+ * StartReadBuffers(), and must map to the expected blocks. The nblocks
+ * value may be smaller on return indicating the size of the I/O that
+ * could be started. Buffers beyond the output nblocks number may also
+ * have been pinned without starting I/O due to various edge cases. In
+ * that case we'll just leave them in the queue ahead of us, "forwarded"
+ * to the next call, avoiding the need to unpin/repin.
+ */
need_wait = StartReadBuffers(&stream->ios[io_index].op,
&stream->buffers[buffer_index],
stream->pending_read_blocknum,
@@ -347,16 +372,35 @@ read_stream_start_pending_read(ReadStream *stream,
stream->seq_blocknum = stream->pending_read_blocknum + nblocks;
}
+ /*
+ * How many pins were acquired but forwarded to the next call? These need
+ * to be passed to the next StartReadBuffers() call, or released if the
+ * stream ends early. We need the number for accounting purposes, since
+ * they are not counted in stream->pinned_buffers but we already hold
+ * them.
+ */
+ forwarded = 0;
+ while (nblocks + forwarded < requested_nblocks &&
+ stream->buffers[buffer_index + nblocks + forwarded] != InvalidBuffer)
+ forwarded++;
+ stream->forwarded_buffers = forwarded;
+
/*
* We gave a contiguous range of buffer space to StartReadBuffers(), but
- * we want it to wrap around at queue_size. Slide overflowing buffers to
- * the front of the array.
+ * we want it to wrap around at queue_size. Copy overflowing buffers to
+ * the front of the array where they'll be consumed, but also leave a copy
+ * in the overflow zone which the I/O operation has a pointer to (it needs
+ * a contiguous array). Both copies will be cleared when the buffers are
+ * handed to the consumer.
*/
- overflow = (buffer_index + nblocks) - stream->queue_size;
+ overflow = (buffer_index + nblocks + forwarded) - stream->queue_size;
if (overflow > 0)
- memmove(&stream->buffers[0],
- &stream->buffers[stream->queue_size],
- sizeof(stream->buffers[0]) * overflow);
+ {
+ Assert(overflow < stream->queue_size); /* can't overlap */
+ memcpy(&stream->buffers[0],
+ &stream->buffers[stream->queue_size],
+ sizeof(stream->buffers[0]) * overflow);
+ }
/* Move to the location of start of next read. */
read_stream_index_advance_n(stream, &buffer_index, nblocks);
@@ -381,6 +425,15 @@ read_stream_get_buffer_limit(ReadStream *stream)
else
buffers = GetAdditionalPinLimit();
+ /*
+ * If we already have some forwarded buffers, we can certainly use those.
+ * They are already pinned, and are mapped to the starting blocks of the
+ * pending read, they just don't have any I/O started yet and are not
+ * counted in stream->pinned_buffers.
+ */
+ Assert(stream->forwarded_buffers <= stream->pending_read_nblocks);
+ buffers += stream->forwarded_buffers;
+
/*
* Each stream is always allowed to try to acquire one pin if it doesn't
* hold one already. This is needed to guarantee progress, and just like
@@ -389,7 +442,7 @@ read_stream_get_buffer_limit(ReadStream *stream)
* pin, ie the buffer pool is simply too small for the workload.
*/
if (buffers == 0 && stream->pinned_buffers == 0)
- return 1;
+ buffers = 1;
/*
* Otherwise, see how many additional pins the backend can currently pin,
@@ -755,10 +808,12 @@ read_stream_next_buffer(ReadStream *stream, void **per_buffer_data)
/* Fast path assumptions. */
Assert(stream->ios_in_progress == 0);
+ Assert(stream->forwarded_buffers == 0);
Assert(stream->pinned_buffers == 1);
Assert(stream->distance == 1);
Assert(stream->pending_read_nblocks == 0);
Assert(stream->per_buffer_data_size == 0);
+ Assert(stream->initialized_buffers > stream->oldest_buffer_index);
/* We're going to return the buffer we pinned last time. */
oldest_buffer_index = stream->oldest_buffer_index;
@@ -807,6 +862,7 @@ read_stream_next_buffer(ReadStream *stream, void **per_buffer_data)
stream->distance = 0;
stream->oldest_buffer_index = stream->next_buffer_index;
stream->pinned_buffers = 0;
+ stream->buffers[oldest_buffer_index] = InvalidBuffer;
}
stream->fast_path = false;
@@ -891,10 +947,15 @@ read_stream_next_buffer(ReadStream *stream, void **per_buffer_data)
}
}
-#ifdef CLOBBER_FREED_MEMORY
- /* Clobber old buffer for debugging purposes. */
+ /*
+ * We must zap this queue entry, or else it would appear as a forwarded
+ * buffer. If it's potentially in the overflow zone (ie it wrapped around
+ * the queue), also zap that copy.
+ */
stream->buffers[oldest_buffer_index] = InvalidBuffer;
-#endif
+ if (oldest_buffer_index < io_combine_limit - 1)
+ stream->buffers[stream->queue_size + oldest_buffer_index] =
+ InvalidBuffer;
#if defined(CLOBBER_FREED_MEMORY) || defined(USE_VALGRIND)
@@ -937,6 +998,7 @@ read_stream_next_buffer(ReadStream *stream, void **per_buffer_data)
#ifndef READ_STREAM_DISABLE_FAST_PATH
/* See if we can take the fast path for all-cached scans next time. */
if (stream->ios_in_progress == 0 &&
+ stream->forwarded_buffers == 0 &&
stream->pinned_buffers == 1 &&
stream->distance == 1 &&
stream->pending_read_nblocks == 0 &&
@@ -972,6 +1034,7 @@ read_stream_next_block(ReadStream *stream, BufferAccessStrategy *strategy)
void
read_stream_reset(ReadStream *stream)
{
+ int16 index;
Buffer buffer;
/* Stop looking ahead. */
@@ -985,6 +1048,23 @@ read_stream_reset(ReadStream *stream)
while ((buffer = read_stream_next_buffer(stream, NULL)) != InvalidBuffer)
ReleaseBuffer(buffer);
+ /* Unpin any unused forwarded buffers. */
+ index = stream->next_buffer_index;
+ while (index < stream->initialized_buffers &&
+ (buffer = stream->buffers[index]) != InvalidBuffer)
+ {
+ Assert(stream->forwarded_buffers > 0);
+ stream->forwarded_buffers--;
+ ReleaseBuffer(buffer);
+
+ stream->buffers[index] = InvalidBuffer;
+ if (index < io_combine_limit - 1)
+ stream->buffers[stream->queue_size + index] = InvalidBuffer;
+
+ read_stream_index_advance(stream, &index);
+ }
+
+ Assert(stream->forwarded_buffers == 0);
Assert(stream->pinned_buffers == 0);
Assert(stream->ios_in_progress == 0);
--
2.48.1.76.g4e746b1a31.dirty
[text/x-diff] v2.5-0014-Support-buffer-forwarding-in-StartReadBuffers.patch (10.5K, ../../vz5i2x2wkjjpp6l2z4g4l3umpxktrxsejgdwefi7n6kzw666h3@rnhlvlpadig7/15-v2.5-0014-Support-buffer-forwarding-in-StartReadBuffers.patch)
download | inline diff:
From 809aa501cee7737bafca06960ae73786c4472a2f Mon Sep 17 00:00:00 2001
From: Thomas Munro <thomas.munro@gmail.com>
Date: Mon, 10 Feb 2025 21:55:40 +1300
Subject: [PATCH v2.5 14/30] Support buffer forwarding in StartReadBuffers().
Sometimes we have to perform a short read because we hit a cached block
that ends a contiguous run of blocks requiring I/O. We don't want
StartReadBuffers() to have to start more than one I/O, so we stop there.
We also don't want to have to unpin the cached block (and repin it
later), so previously we'd silently pretend the hit was part of the I/O,
and just leave it out of the read from disk. Now, we'll "forward" it to
the next call. We still write it to the buffers[] array for the caller
to pass back to us later, but it's not included in *nblocks.
This policy means that we no longer mix hits and misses in a single
operation's results, so we avoid the requirement to call
WaitReadBuffers(), which might stall, before the caller can make use of
the hits. The caller will get the hit in the next call instead, and
know that it doesn't have to wait. That's important for later work on
out-of-order read streams that minimize I/O stalls.
This also makes life easier for proposed work on true AIO, which
occasionally needs to split a large I/O after pinning all the buffers,
while the current coding only ever forwards a single bookending hit.
This API is natural for read_stream.c: it just leaves forwarded buffers
where they are in its circular queue, where the next call will pick them
up and continue, minimizing pin churn.
If we ever think of a good reason to disable this feature, i.e. for
other users of StartReadBuffers() that don't want to deal with forwarded
buffers, then we could add a flag for that. For now read_steam.c is the
only user.
---
src/include/storage/bufmgr.h | 1 -
src/backend/storage/buffer/bufmgr.c | 128 ++++++++++++++++++++--------
2 files changed, 91 insertions(+), 38 deletions(-)
diff --git a/src/include/storage/bufmgr.h b/src/include/storage/bufmgr.h
index 597ecb97897..4a035f59a7d 100644
--- a/src/include/storage/bufmgr.h
+++ b/src/include/storage/bufmgr.h
@@ -130,7 +130,6 @@ struct ReadBuffersOperation
BlockNumber blocknum;
int flags;
int16 nblocks;
- int16 io_buffers_len;
};
typedef struct ReadBuffersOperation ReadBuffersOperation;
diff --git a/src/backend/storage/buffer/bufmgr.c b/src/backend/storage/buffer/bufmgr.c
index 11c146763db..75a928e802a 100644
--- a/src/backend/storage/buffer/bufmgr.c
+++ b/src/backend/storage/buffer/bufmgr.c
@@ -1257,10 +1257,10 @@ StartReadBuffersImpl(ReadBuffersOperation *operation,
Buffer *buffers,
BlockNumber blockNum,
int *nblocks,
- int flags)
+ int flags,
+ bool allow_forwarding)
{
int actual_nblocks = *nblocks;
- int io_buffers_len = 0;
int maxcombine = 0;
Assert(*nblocks > 0);
@@ -1270,30 +1270,80 @@ StartReadBuffersImpl(ReadBuffersOperation *operation,
{
bool found;
- buffers[i] = PinBufferForBlock(operation->rel,
- operation->smgr,
- operation->persistence,
- operation->forknum,
- blockNum + i,
- operation->strategy,
- &found);
+ if (allow_forwarding && buffers[i] != InvalidBuffer)
+ {
+ BufferDesc *bufHdr;
+
+ /*
+ * This is a buffer that was pinned by an earlier call to
+ * StartReadBuffers(), but couldn't be handled in one operation at
+ * that time. The operation was split, and the caller has passed
+ * an already pinned buffer back to us to handle the rest of the
+ * operation. It must continue at the expected block number.
+ */
+ Assert(BufferGetBlockNumber(buffers[i]) == blockNum + i);
+
+ /*
+ * It might be an already valid buffer (a hit) that followed the
+ * final contiguous block of an earlier I/O (a miss) marking the
+ * end of it, or a buffer that some other backend has since made
+ * valid by performing the I/O for us, in which case we can handle
+ * it as a hit now. It is safe to check for a BM_VALID flag with
+ * a relaxed load, because we got a fresh view of it while pinning
+ * it in the previous call.
+ *
+ * On the other hand if we don't see BM_VALID yet, it must be an
+ * I/O that was split by the previous call and we need to try to
+ * start a new I/O from this block. We're also racing against any
+ * other backend that might start the I/O or even manage to mark
+ * it BM_VALID after this check, BM_VALID after this check, but
+ * StartBufferIO() will handle those cases.
+ */
+ if (BufferIsLocal(buffers[i]))
+ bufHdr = GetLocalBufferDescriptor(-buffers[i] - 1);
+ else
+ bufHdr = GetBufferDescriptor(buffers[i] - 1);
+ found = pg_atomic_read_u32(&bufHdr->state) & BM_VALID;
+ }
+ else
+ {
+ buffers[i] = PinBufferForBlock(operation->rel,
+ operation->smgr,
+ operation->persistence,
+ operation->forknum,
+ blockNum + i,
+ operation->strategy,
+ &found);
+ }
if (found)
{
/*
- * Terminate the read as soon as we get a hit. It could be a
- * single buffer hit, or it could be a hit that follows a readable
- * range. We don't want to create more than one readable range,
- * so we stop here.
+ * We have a hit. If it's the first block in the requested range,
+ * we can return it immediately and report that WaitReadBuffers()
+ * does not need to be called. If the initial value of *nblocks
+ * was larger, the caller will have to call again for the rest.
*/
- actual_nblocks = i + 1;
+ if (i == 0)
+ {
+ *nblocks = 1;
+ return false;
+ }
+
+ /*
+ * Otherwise we already have an I/O to perform, but this block
+ * can't be included as it is already valid. Split the I/O here.
+ * There may or may not be more blocks requiring I/O after this
+ * one, we haven't checked, but it can't be contiguous with this
+ * hit in the way. We'll leave this buffer pinned, forwarding it
+ * to the next call, avoiding the need to unpin it here and re-pin
+ * it in the next call.
+ */
+ actual_nblocks = i;
break;
}
else
{
- /* Extend the readable range to cover this block. */
- io_buffers_len++;
-
/*
* Check how many blocks we can cover with the same IO. The smgr
* implementation might e.g. be limited due to a segment boundary.
@@ -1314,15 +1364,11 @@ StartReadBuffersImpl(ReadBuffersOperation *operation,
}
*nblocks = actual_nblocks;
- if (likely(io_buffers_len == 0))
- return false;
-
/* Populate information needed for I/O. */
operation->buffers = buffers;
operation->blocknum = blockNum;
operation->flags = flags;
operation->nblocks = actual_nblocks;
- operation->io_buffers_len = io_buffers_len;
if (flags & READ_BUFFERS_ISSUE_ADVICE)
{
@@ -1337,7 +1383,7 @@ StartReadBuffersImpl(ReadBuffersOperation *operation,
smgrprefetch(operation->smgr,
operation->forknum,
blockNum,
- operation->io_buffers_len);
+ actual_nblocks);
}
/* Indicate that WaitReadBuffers() should be called. */
@@ -1351,11 +1397,21 @@ StartReadBuffersImpl(ReadBuffersOperation *operation,
* actual number, which may be fewer than requested. Caller sets some of the
* members of operation; see struct definition.
*
+ * The initial contents of the elements of buffers up to *nblocks should
+ * either be InvalidBuffer or an already-pinned buffer that was left by an
+ * preceding call to StartReadBuffers() that had to be split. On return, some
+ * elements of buffers may hold pinned buffers beyond the number indicated by
+ * the updated value of *nblocks. Operations are split on boundaries known to
+ * smgr (eg md.c segment boundaries that require crossing into a different
+ * underlying file), or when already cached blocks are found in the buffer
+ * that prevent the formation of a contiguous read.
+ *
* If false is returned, no I/O is necessary. If true is returned, one I/O
* has been started, and WaitReadBuffers() must be called with the same
* operation object before the buffers are accessed. Along with the operation
* object, the caller-supplied array of buffers must remain valid until
- * WaitReadBuffers() is called.
+ * WaitReadBuffers() is called, and any forwarded buffers must also be
+ * preserved for a future call unless explicitly released.
*
* Currently the I/O is only started with optional operating system advice if
* requested by the caller with READ_BUFFERS_ISSUE_ADVICE, and the real I/O
@@ -1369,13 +1425,18 @@ StartReadBuffers(ReadBuffersOperation *operation,
int *nblocks,
int flags)
{
- return StartReadBuffersImpl(operation, buffers, blockNum, nblocks, flags);
+ return StartReadBuffersImpl(operation, buffers, blockNum, nblocks, flags,
+ true /* expect forwarded buffers */ );
}
/*
* Single block version of the StartReadBuffers(). This might save a few
* instructions when called from another translation unit, because it is
* specialized for nblocks == 1.
+ *
+ * This version does not support "forwarded" buffers: they cannot be created
+ * by reading only one block, and the current contents of *buffer is ignored
+ * on entry.
*/
bool
StartReadBuffer(ReadBuffersOperation *operation,
@@ -1386,7 +1447,8 @@ StartReadBuffer(ReadBuffersOperation *operation,
int nblocks = 1;
bool result;
- result = StartReadBuffersImpl(operation, buffer, blocknum, &nblocks, flags);
+ result = StartReadBuffersImpl(operation, buffer, blocknum, &nblocks, flags,
+ false /* single block, no forwarding */ );
Assert(nblocks == 1); /* single block can't be short */
return result;
@@ -1416,24 +1478,16 @@ WaitReadBuffers(ReadBuffersOperation *operation)
IOObject io_object;
char persistence;
- /*
- * Currently operations are only allowed to include a read of some range,
- * with an optional extra buffer that is already pinned at the end. So
- * nblocks can be at most one more than io_buffers_len.
- */
- Assert((operation->nblocks == operation->io_buffers_len) ||
- (operation->nblocks == operation->io_buffers_len + 1));
-
/* Find the range of the physical read we need to perform. */
- nblocks = operation->io_buffers_len;
- if (nblocks == 0)
- return; /* nothing to do */
-
+ nblocks = operation->nblocks;
buffers = &operation->buffers[0];
blocknum = operation->blocknum;
forknum = operation->forknum;
persistence = operation->persistence;
+ Assert(nblocks > 0);
+ Assert(nblocks <= MAX_IO_COMBINE_LIMIT);
+
if (persistence == RELPERSISTENCE_TEMP)
{
io_context = IOCONTEXT_NORMAL;
--
2.48.1.76.g4e746b1a31.dirty
[text/x-diff] v2.5-0015-WIP-tests-Expand-temp-table-tests-to-some-pin-r.patch (9.6K, ../../vz5i2x2wkjjpp6l2z4g4l3umpxktrxsejgdwefi7n6kzw666h3@rnhlvlpadig7/16-v2.5-0015-WIP-tests-Expand-temp-table-tests-to-some-pin-r.patch)
download | inline diff:
From 7c01d8dd84dff8ba1b31ad4641933ae394428fa8 Mon Sep 17 00:00:00 2001
From: Andres Freund <andres@anarazel.de>
Date: Tue, 4 Mar 2025 11:07:47 -0500
Subject: [PATCH v2.5 15/30] WIP: tests: Expand temp table tests to some pin
related matters
Author:
Reviewed-By:
Discussion: https://postgr.es/m/
Backpatch:
---
src/test/regress/expected/temp.out | 156 +++++++++++++++++++++++++++++
src/test/regress/parallel_schedule | 2 +-
src/test/regress/sql/temp.sql | 107 ++++++++++++++++++++
3 files changed, 264 insertions(+), 1 deletion(-)
diff --git a/src/test/regress/expected/temp.out b/src/test/regress/expected/temp.out
index 2a246a7e123..91fe519b1cc 100644
--- a/src/test/regress/expected/temp.out
+++ b/src/test/regress/expected/temp.out
@@ -410,3 +410,159 @@ SELECT current_schema() ~ 'pg_temp' AS is_temp_schema;
PREPARE TRANSACTION 'twophase_search';
ERROR: cannot PREPARE a transaction that has operated on temporary objects
+-- Tests to verify we recover correctly from exhausting buffer pins and
+-- related matters.
+-- use lower possible buffer limit to make the test cheaper
+\c
+SET temp_buffers = 100;
+CREATE TEMPORARY TABLE test_temp(a int not null unique, b TEXT not null, cnt int not null);
+INSERT INTO test_temp SELECT generate_series(1, 10000) as id, repeat('a', 200), 0;
+-- should be at least 2x as large than temp_buffers
+SELECT pg_relation_size('test_temp') / current_setting('block_size')::int8 > 200;
+ ?column?
+----------
+ t
+(1 row)
+
+-- Don't want cursor names and plpgsql function lines in the error messages
+\set VERBOSITY terse
+/* helper function to create cursors for each page in [p_start, p_end] */
+CREATE FUNCTION test_temp_pin(p_start int, p_end int)
+RETURNS void
+LANGUAGE plpgsql
+AS $f$
+ DECLARE
+ cursorname text;
+ query text;
+ BEGIN
+ FOR i IN p_start..p_end LOOP
+ cursorname = 'c_'||i;
+ query = format($q$DECLARE %I CURSOR FOR SELECT ctid FROM test_temp WHERE ctid >= '( %s, 1)'::tid $q$, cursorname, i);
+ EXECUTE query;
+ EXECUTE 'FETCH NEXT FROM '||cursorname;
+ -- for test development
+ -- RAISE NOTICE '%: %', cursorname, query;
+ END LOOP;
+ END;
+$f$;
+-- Test overflow of temp table buffers is handled correctly
+BEGIN;
+-- should work, below max
+SELECT test_temp_pin(0, 9);
+ test_temp_pin
+---------------
+
+(1 row)
+
+-- should fail, too many buffers pinned
+SELECT test_temp_pin(10, 105);
+ERROR: no empty local buffer available
+ROLLBACK;
+BEGIN;
+-- have some working cursors to test after errors
+SELECT test_temp_pin(0, 9);
+ test_temp_pin
+---------------
+
+(1 row)
+
+FETCH NEXT FROM c_3;
+ ctid
+-------
+ (3,2)
+(1 row)
+
+-- exhaust buffer pins in subtrans, check things work after
+SAVEPOINT rescue_me;
+SELECT test_temp_pin(10, 105);
+ERROR: no empty local buffer available
+ROLLBACK TO SAVEPOINT rescue_me;
+-- pre-subtrans cursors continue to work
+FETCH NEXT FROM c_3;
+ ctid
+-------
+ (3,3)
+(1 row)
+
+-- new cursors with pins can be created after subtrans rollback
+SELECT test_temp_pin(10, 95);
+ test_temp_pin
+---------------
+
+(1 row)
+
+-- Check that read streams deal with lower number of pins available
+SELECT count(*), max(a) max_a, min(a) min_a, max(cnt) max_cnt FROM test_temp;
+ count | max_a | min_a | max_cnt
+-------+-------+-------+---------
+ 10000 | 10000 | 1 | 0
+(1 row)
+
+ROLLBACK;
+-- Check that temp tables with existing cursors can't be dropped.
+BEGIN;
+SELECT test_temp_pin(0, 1);
+ test_temp_pin
+---------------
+
+(1 row)
+
+DROP TABLE test_temp;
+ERROR: cannot DROP TABLE "test_temp" because it is being used by active queries in this session
+COMMIT;
+-- Check that temp tables with existing cursors can't be dropped.
+BEGIN;
+SELECT test_temp_pin(0, 1);
+ test_temp_pin
+---------------
+
+(1 row)
+
+TRUNCATE test_temp;
+ERROR: cannot TRUNCATE "test_temp" because it is being used by active queries in this session
+COMMIT;
+-- Check that temp tables that are dropped in transaction that's rolled back
+-- preserve buffer contents
+SELECT count(*), max(a) max_a, min(a) min_a, max(cnt) max_cnt FROM test_temp;
+ count | max_a | min_a | max_cnt
+-------+-------+-------+---------
+ 10000 | 10000 | 1 | 0
+(1 row)
+
+INSERT INTO test_temp(a, b, cnt) VALUES (-1, '', 0);
+BEGIN;
+INSERT INTO test_temp(a, b, cnt) VALUES (-2, '', 0);
+DROP TABLE test_temp;
+ROLLBACK;
+SELECT count(*), max(a) max_a, min(a) min_a, max(cnt) max_cnt FROM test_temp;
+ count | max_a | min_a | max_cnt
+-------+-------+-------+---------
+ 10001 | 10000 | -1 | 0
+(1 row)
+
+-- Check that temp table drop is transactional and preserves dirty
+-- buffer contents
+UPDATE test_temp SET cnt = cnt + 1 WHERE a = -1;
+BEGIN;
+DROP TABLE test_temp;
+ROLLBACK;
+SELECT count(*), max(a) max_a, min(a) min_a, max(cnt) max_cnt FROM test_temp;
+ count | max_a | min_a | max_cnt
+-------+-------+-------+---------
+ 10001 | 10000 | -1 | 1
+(1 row)
+
+-- Check that temp table truncation is transactional and preserves dirty
+-- buffer contents
+UPDATE test_temp SET cnt = cnt + 1 WHERE a = -1;
+BEGIN;
+TRUNCATE test_temp;
+ROLLBACK;
+SELECT count(*), max(a) max_a, min(a) min_a, max(cnt) max_cnt FROM test_temp;
+ count | max_a | min_a | max_cnt
+-------+-------+-------+---------
+ 10001 | 10000 | -1 | 2
+(1 row)
+
+-- cleanup
+DROP FUNCTION test_temp_pin(int, int);
diff --git a/src/test/regress/parallel_schedule b/src/test/regress/parallel_schedule
index 37b6d21e1f9..0a35f2f8f6a 100644
--- a/src/test/regress/parallel_schedule
+++ b/src/test/regress/parallel_schedule
@@ -108,7 +108,7 @@ test: json jsonb json_encoding jsonpath jsonpath_encoding jsonb_jsonpath sqljson
# ----------
# Another group of parallel tests
# with depends on create_misc
-# NB: temp.sql does a reconnect which transiently uses 2 connections,
+# NB: temp.sql does reconnects which transiently use 2 connections,
# so keep this parallel group to at most 19 tests
# ----------
test: plancache limit plpgsql copy2 temp domain rangefuncs prepare conversion truncate alter_table sequence polymorphism rowtypes returning largeobject with xml
diff --git a/src/test/regress/sql/temp.sql b/src/test/regress/sql/temp.sql
index 2a487a1ef7f..12091f968de 100644
--- a/src/test/regress/sql/temp.sql
+++ b/src/test/regress/sql/temp.sql
@@ -311,3 +311,110 @@ SET search_path TO 'pg_temp';
BEGIN;
SELECT current_schema() ~ 'pg_temp' AS is_temp_schema;
PREPARE TRANSACTION 'twophase_search';
+
+
+-- Tests to verify we recover correctly from exhausting buffer pins and
+-- related matters.
+
+-- use lower possible buffer limit to make the test cheaper
+\c
+SET temp_buffers = 100;
+
+CREATE TEMPORARY TABLE test_temp(a int not null unique, b TEXT not null, cnt int not null);
+INSERT INTO test_temp SELECT generate_series(1, 10000) as id, repeat('a', 200), 0;
+-- should be at least 2x as large than temp_buffers
+SELECT pg_relation_size('test_temp') / current_setting('block_size')::int8 > 200;
+
+-- Don't want cursor names and plpgsql function lines in the error messages
+\set VERBOSITY terse
+
+/* helper function to create cursors for each page in [p_start, p_end] */
+CREATE FUNCTION test_temp_pin(p_start int, p_end int)
+RETURNS void
+LANGUAGE plpgsql
+AS $f$
+ DECLARE
+ cursorname text;
+ query text;
+ BEGIN
+ FOR i IN p_start..p_end LOOP
+ cursorname = 'c_'||i;
+ query = format($q$DECLARE %I CURSOR FOR SELECT ctid FROM test_temp WHERE ctid >= '( %s, 1)'::tid $q$, cursorname, i);
+ EXECUTE query;
+ EXECUTE 'FETCH NEXT FROM '||cursorname;
+ -- for test development
+ -- RAISE NOTICE '%: %', cursorname, query;
+ END LOOP;
+ END;
+$f$;
+
+
+-- Test overflow of temp table buffers is handled correctly
+BEGIN;
+-- should work, below max
+SELECT test_temp_pin(0, 9);
+-- should fail, too many buffers pinned
+SELECT test_temp_pin(10, 105);
+ROLLBACK;
+
+BEGIN;
+-- have some working cursors to test after errors
+SELECT test_temp_pin(0, 9);
+FETCH NEXT FROM c_3;
+-- exhaust buffer pins in subtrans, check things work after
+SAVEPOINT rescue_me;
+SELECT test_temp_pin(10, 105);
+ROLLBACK TO SAVEPOINT rescue_me;
+-- pre-subtrans cursors continue to work
+FETCH NEXT FROM c_3;
+
+-- new cursors with pins can be created after subtrans rollback
+SELECT test_temp_pin(10, 95);
+
+-- Check that read streams deal with lower number of pins available
+SELECT count(*), max(a) max_a, min(a) min_a, max(cnt) max_cnt FROM test_temp;
+
+ROLLBACK;
+
+
+-- Check that temp tables with existing cursors can't be dropped.
+BEGIN;
+SELECT test_temp_pin(0, 1);
+DROP TABLE test_temp;
+COMMIT;
+
+-- Check that temp tables with existing cursors can't be dropped.
+BEGIN;
+SELECT test_temp_pin(0, 1);
+TRUNCATE test_temp;
+COMMIT;
+
+-- Check that temp tables that are dropped in transaction that's rolled back
+-- preserve buffer contents
+SELECT count(*), max(a) max_a, min(a) min_a, max(cnt) max_cnt FROM test_temp;
+INSERT INTO test_temp(a, b, cnt) VALUES (-1, '', 0);
+BEGIN;
+INSERT INTO test_temp(a, b, cnt) VALUES (-2, '', 0);
+DROP TABLE test_temp;
+ROLLBACK;
+SELECT count(*), max(a) max_a, min(a) min_a, max(cnt) max_cnt FROM test_temp;
+
+-- Check that temp table drop is transactional and preserves dirty
+-- buffer contents
+UPDATE test_temp SET cnt = cnt + 1 WHERE a = -1;
+BEGIN;
+DROP TABLE test_temp;
+ROLLBACK;
+SELECT count(*), max(a) max_a, min(a) min_a, max(cnt) max_cnt FROM test_temp;
+
+-- Check that temp table truncation is transactional and preserves dirty
+-- buffer contents
+UPDATE test_temp SET cnt = cnt + 1 WHERE a = -1;
+BEGIN;
+TRUNCATE test_temp;
+ROLLBACK;
+SELECT count(*), max(a) max_a, min(a) min_a, max(cnt) max_cnt FROM test_temp;
+
+
+-- cleanup
+DROP FUNCTION test_temp_pin(int, int);
--
2.48.1.76.g4e746b1a31.dirty
[text/x-diff] v2.5-0016-WIP-localbuf-Track-pincount-in-BufferDesc-as-we.patch (7.7K, ../../vz5i2x2wkjjpp6l2z4g4l3umpxktrxsejgdwefi7n6kzw666h3@rnhlvlpadig7/17-v2.5-0016-WIP-localbuf-Track-pincount-in-BufferDesc-as-we.patch)
download | inline diff:
From d68313d45f1fb5dc1b3dca8f18b55d6cfd64d77d Mon Sep 17 00:00:00 2001
From: Andres Freund <andres@anarazel.de>
Date: Thu, 20 Feb 2025 18:04:48 -0500
Subject: [PATCH v2.5 16/30] WIP: localbuf: Track pincount in BufferDesc as
well
For AIO on temp tables the AIO subsystem needs to be able to ensure a pin on a
buffer while AIO is going on, even if the IO issuing query errors out. To do
so, track the refcount in BufferDesc.state, not ust LocalRefCount.
Note that we still don't need locking, AIO completion callbacks for local
buffers are executed in the issuing session (nobody else has access to the
BufferDesc).
---
src/backend/storage/buffer/bufmgr.c | 30 ++++++--
src/backend/storage/buffer/localbuf.c | 99 +++++++++++++++++----------
2 files changed, 87 insertions(+), 42 deletions(-)
diff --git a/src/backend/storage/buffer/bufmgr.c b/src/backend/storage/buffer/bufmgr.c
index 75a928e802a..e14888ace85 100644
--- a/src/backend/storage/buffer/bufmgr.c
+++ b/src/backend/storage/buffer/bufmgr.c
@@ -5443,8 +5443,20 @@ ConditionalLockBufferForCleanup(Buffer buffer)
Assert(refcount > 0);
if (refcount != 1)
return false;
- /* Nobody else to wait for */
- return true;
+
+ bufHdr = GetLocalBufferDescriptor(-buffer - 1);
+ buf_state = pg_atomic_read_u32(&bufHdr->state);
+
+ /*
+ * Check that the AIO subsystem doesn't have a pin. Likely not
+ * possible today, but better safe than sorry.
+ */
+ refcount = BUF_STATE_GET_REFCOUNT(buf_state);
+ Assert(refcount > 0);
+ if (refcount == 1)
+ return true;
+
+ return false;
}
/* There should be exactly one local pin */
@@ -5496,8 +5508,18 @@ IsBufferCleanupOK(Buffer buffer)
/* There should be exactly one pin */
if (LocalRefCount[-buffer - 1] != 1)
return false;
- /* Nobody else to wait for */
- return true;
+
+ bufHdr = GetLocalBufferDescriptor(-buffer - 1);
+ buf_state = pg_atomic_read_u32(&bufHdr->state);
+
+ /*
+ * Check that the AIO subsystem doesn't have a pin. Likely not
+ * possible today, but better safe than sorry.
+ */
+ if (BUF_STATE_GET_REFCOUNT(buf_state) == 1)
+ return true;
+
+ return false;
}
/* There should be exactly one local pin */
diff --git a/src/backend/storage/buffer/localbuf.c b/src/backend/storage/buffer/localbuf.c
index 5378ba84316..913a91f061e 100644
--- a/src/backend/storage/buffer/localbuf.c
+++ b/src/backend/storage/buffer/localbuf.c
@@ -207,10 +207,19 @@ GetLocalVictimBuffer(void)
pg_atomic_unlocked_write_u32(&bufHdr->state, buf_state);
trycounter = NLocBuffer;
}
+ else if (BUF_STATE_GET_REFCOUNT(buf_state) > 0)
+ {
+ /*
+ * This can be reached if the backend initiated AIO for this
+ * buffer and then errored out.
+ */
+ }
else
{
/* Found a usable buffer */
PinLocalBuffer(bufHdr, false);
+ /* the buf_state may be modified inside PinLocalBuffer */
+ buf_state = pg_atomic_read_u32(&bufHdr->state);
break;
}
}
@@ -491,6 +500,44 @@ MarkLocalBufferDirty(Buffer buffer)
pg_atomic_unlocked_write_u32(&bufHdr->state, buf_state);
}
+static void
+InvalidateLocalBuffer(BufferDesc *bufHdr)
+{
+ Buffer buffer = BufferDescriptorGetBuffer(bufHdr);
+ int bufid = -buffer - 1;
+ uint32 buf_state;
+ LocalBufferLookupEnt *hresult;
+
+ buf_state = pg_atomic_read_u32(&bufHdr->state);
+
+ /*
+ * We need to test not just LocalRefCount[bufid] but also the BufferDesc
+ * itself, as the latter is used to represent a pin by the AIO subsystem.
+ * This can happen if AIO is initiated and then the query errors out.
+ */
+ if (LocalRefCount[bufid] != 0 ||
+ BUF_STATE_GET_REFCOUNT(buf_state) > 0)
+ elog(ERROR, "block %u of %s is still referenced (local %u)",
+ bufHdr->tag.blockNum,
+ relpathbackend(BufTagGetRelFileLocator(&bufHdr->tag),
+ MyProcNumber,
+ BufTagGetForkNum(&bufHdr->tag)).str,
+ LocalRefCount[bufid]);
+
+ /* Remove entry from hashtable */
+ hresult = (LocalBufferLookupEnt *)
+ hash_search(LocalBufHash, &bufHdr->tag, HASH_REMOVE, NULL);
+ if (!hresult) /* shouldn't happen */
+ elog(ERROR, "local buffer hash table corrupted");
+ /* Mark buffer invalid */
+ ClearBufferTag(&bufHdr->tag);
+
+ buf_state &= ~BUF_FLAG_MASK;
+ buf_state &= ~BUF_USAGECOUNT_MASK;
+ pg_atomic_unlocked_write_u32(&bufHdr->state, buf_state);
+
+}
+
/*
* DropRelationLocalBuffers
* This function removes from the buffer pool all the pages of the
@@ -511,7 +558,6 @@ DropRelationLocalBuffers(RelFileLocator rlocator, ForkNumber forkNum,
for (i = 0; i < NLocBuffer; i++)
{
BufferDesc *bufHdr = GetLocalBufferDescriptor(i);
- LocalBufferLookupEnt *hresult;
uint32 buf_state;
buf_state = pg_atomic_read_u32(&bufHdr->state);
@@ -521,24 +567,7 @@ DropRelationLocalBuffers(RelFileLocator rlocator, ForkNumber forkNum,
BufTagGetForkNum(&bufHdr->tag) == forkNum &&
bufHdr->tag.blockNum >= firstDelBlock)
{
- if (LocalRefCount[i] != 0)
- elog(ERROR, "block %u of %s is still referenced (local %u)",
- bufHdr->tag.blockNum,
- relpathbackend(BufTagGetRelFileLocator(&bufHdr->tag),
- MyProcNumber,
- BufTagGetForkNum(&bufHdr->tag)).str,
- LocalRefCount[i]);
-
- /* Remove entry from hashtable */
- hresult = (LocalBufferLookupEnt *)
- hash_search(LocalBufHash, &bufHdr->tag, HASH_REMOVE, NULL);
- if (!hresult) /* shouldn't happen */
- elog(ERROR, "local buffer hash table corrupted");
- /* Mark buffer invalid */
- ClearBufferTag(&bufHdr->tag);
- buf_state &= ~BUF_FLAG_MASK;
- buf_state &= ~BUF_USAGECOUNT_MASK;
- pg_atomic_unlocked_write_u32(&bufHdr->state, buf_state);
+ InvalidateLocalBuffer(bufHdr);
}
}
}
@@ -558,7 +587,6 @@ DropRelationAllLocalBuffers(RelFileLocator rlocator)
for (i = 0; i < NLocBuffer; i++)
{
BufferDesc *bufHdr = GetLocalBufferDescriptor(i);
- LocalBufferLookupEnt *hresult;
uint32 buf_state;
buf_state = pg_atomic_read_u32(&bufHdr->state);
@@ -566,23 +594,7 @@ DropRelationAllLocalBuffers(RelFileLocator rlocator)
if ((buf_state & BM_TAG_VALID) &&
BufTagMatchesRelFileLocator(&bufHdr->tag, &rlocator))
{
- if (LocalRefCount[i] != 0)
- elog(ERROR, "block %u of %s is still referenced (local %u)",
- bufHdr->tag.blockNum,
- relpathbackend(BufTagGetRelFileLocator(&bufHdr->tag),
- MyProcNumber,
- BufTagGetForkNum(&bufHdr->tag)).str,
- LocalRefCount[i]);
- /* Remove entry from hashtable */
- hresult = (LocalBufferLookupEnt *)
- hash_search(LocalBufHash, &bufHdr->tag, HASH_REMOVE, NULL);
- if (!hresult) /* shouldn't happen */
- elog(ERROR, "local buffer hash table corrupted");
- /* Mark buffer invalid */
- ClearBufferTag(&bufHdr->tag);
- buf_state &= ~BUF_FLAG_MASK;
- buf_state &= ~BUF_USAGECOUNT_MASK;
- pg_atomic_unlocked_write_u32(&bufHdr->state, buf_state);
+ InvalidateLocalBuffer(bufHdr);
}
}
}
@@ -680,12 +692,13 @@ PinLocalBuffer(BufferDesc *buf_hdr, bool adjust_usagecount)
if (LocalRefCount[bufid] == 0)
{
NLocalPinnedBuffers++;
+ buf_state += BUF_REFCOUNT_ONE;
if (adjust_usagecount &&
BUF_STATE_GET_USAGECOUNT(buf_state) < BM_MAX_USAGE_COUNT)
{
buf_state += BUF_USAGECOUNT_ONE;
- pg_atomic_unlocked_write_u32(&buf_hdr->state, buf_state);
}
+ pg_atomic_unlocked_write_u32(&buf_hdr->state, buf_state);
}
LocalRefCount[bufid]++;
ResourceOwnerRememberBuffer(CurrentResourceOwner,
@@ -711,7 +724,17 @@ UnpinLocalBufferNoOwner(Buffer buffer)
Assert(NLocalPinnedBuffers > 0);
if (--LocalRefCount[buffid] == 0)
+ {
+ BufferDesc *buf_hdr = GetLocalBufferDescriptor(buffid);
+ uint32 buf_state;
+
NLocalPinnedBuffers--;
+
+ buf_state = pg_atomic_read_u32(&buf_hdr->state);
+ Assert(BUF_STATE_GET_REFCOUNT(buf_state) > 0);
+ buf_state -= BUF_REFCOUNT_ONE;
+ pg_atomic_unlocked_write_u32(&buf_hdr->state, buf_state);
+ }
}
/*
--
2.48.1.76.g4e746b1a31.dirty
[text/x-diff] v2.5-0017-bufmgr-Implement-AIO-read-support.patch (19.4K, ../../vz5i2x2wkjjpp6l2z4g4l3umpxktrxsejgdwefi7n6kzw666h3@rnhlvlpadig7/18-v2.5-0017-bufmgr-Implement-AIO-read-support.patch)
download | inline diff:
From e11a2f867e50d63e77175f3860e8f03bd422922d Mon Sep 17 00:00:00 2001
From: Andres Freund <andres@anarazel.de>
Date: Wed, 26 Feb 2025 11:31:40 -0500
Subject: [PATCH v2.5 17/30] bufmgr: Implement AIO read support
As of this commit there are no users of these AIO facilities, that'll come in
later commits.
Author:
Reviewed-By:
Discussion: https://postgr.es/m/
Backpatch:
---
src/include/storage/aio.h | 4 +
src/include/storage/buf_internals.h | 6 +
src/include/storage/bufmgr.h | 6 +
src/backend/storage/aio/aio_callback.c | 5 +
src/backend/storage/buffer/buf_init.c | 3 +
src/backend/storage/buffer/bufmgr.c | 367 ++++++++++++++++++++++++-
src/backend/storage/buffer/localbuf.c | 77 ++++++
7 files changed, 461 insertions(+), 7 deletions(-)
diff --git a/src/include/storage/aio.h b/src/include/storage/aio.h
index 8fbd206c343..4913ff723a9 100644
--- a/src/include/storage/aio.h
+++ b/src/include/storage/aio.h
@@ -178,6 +178,10 @@ typedef enum PgAioHandleCallbackID
PGAIO_HCB_MD_READV,
PGAIO_HCB_MD_WRITEV,
+
+ PGAIO_HCB_SHARED_BUFFER_READV,
+
+ PGAIO_HCB_LOCAL_BUFFER_READV,
} PgAioHandleCallbackID;
diff --git a/src/include/storage/buf_internals.h b/src/include/storage/buf_internals.h
index 8b32fb108b0..cdeb2f0b528 100644
--- a/src/include/storage/buf_internals.h
+++ b/src/include/storage/buf_internals.h
@@ -17,6 +17,7 @@
#include "pgstat.h"
#include "port/atomics.h"
+#include "storage/aio_types.h"
#include "storage/buf.h"
#include "storage/bufmgr.h"
#include "storage/condition_variable.h"
@@ -264,6 +265,8 @@ typedef struct BufferDesc
int wait_backend_pgprocno; /* backend of pin-count waiter */
int freeNext; /* link in freelist chain */
+
+ PgAioWaitRef io_wref;
LWLock content_lock; /* to lock access to buffer contents */
} BufferDesc;
@@ -477,4 +480,7 @@ extern void DropRelationLocalBuffers(RelFileLocator rlocator,
extern void DropRelationAllLocalBuffers(RelFileLocator rlocator);
extern void AtEOXact_LocalBuffers(bool isCommit);
+
+extern PgAioResult LocalBufferCompleteRead(int buf_off, Buffer buffer, uint8 flags, bool failed);
+
#endif /* BUFMGR_INTERNALS_H */
diff --git a/src/include/storage/bufmgr.h b/src/include/storage/bufmgr.h
index 4a035f59a7d..8bd8d75e940 100644
--- a/src/include/storage/bufmgr.h
+++ b/src/include/storage/bufmgr.h
@@ -176,6 +176,12 @@ extern PGDLLIMPORT int NLocBuffer;
extern PGDLLIMPORT Block *LocalBufferBlockPointers;
extern PGDLLIMPORT int32 *LocalRefCount;
+
+struct PgAioHandleCallbacks;
+extern const struct PgAioHandleCallbacks aio_shared_buffer_readv_cb;
+extern const struct PgAioHandleCallbacks aio_local_buffer_readv_cb;
+
+
/* upper limit for effective_io_concurrency */
#define MAX_IO_CONCURRENCY 1000
diff --git a/src/backend/storage/aio/aio_callback.c b/src/backend/storage/aio/aio_callback.c
index 3112b935676..ff7b2f25c99 100644
--- a/src/backend/storage/aio/aio_callback.c
+++ b/src/backend/storage/aio/aio_callback.c
@@ -18,6 +18,7 @@
#include "miscadmin.h"
#include "storage/aio.h"
#include "storage/aio_internal.h"
+#include "storage/bufmgr.h"
#include "storage/md.h"
@@ -40,6 +41,10 @@ static const PgAioHandleCallbacksEntry aio_handle_cbs[] = {
CALLBACK_ENTRY(PGAIO_HCB_INVALID, aio_invalid_cb),
CALLBACK_ENTRY(PGAIO_HCB_MD_READV, aio_md_readv_cb),
+
+ CALLBACK_ENTRY(PGAIO_HCB_SHARED_BUFFER_READV, aio_shared_buffer_readv_cb),
+
+ CALLBACK_ENTRY(PGAIO_HCB_LOCAL_BUFFER_READV, aio_local_buffer_readv_cb),
#undef CALLBACK_ENTRY
};
diff --git a/src/backend/storage/buffer/buf_init.c b/src/backend/storage/buffer/buf_init.c
index ed1f8e03190..ed1dc488a42 100644
--- a/src/backend/storage/buffer/buf_init.c
+++ b/src/backend/storage/buffer/buf_init.c
@@ -14,6 +14,7 @@
*/
#include "postgres.h"
+#include "storage/aio.h"
#include "storage/buf_internals.h"
#include "storage/bufmgr.h"
@@ -125,6 +126,8 @@ BufferManagerShmemInit(void)
buf->buf_id = i;
+ pgaio_wref_clear(&buf->io_wref);
+
/*
* Initially link all the buffers together as unused. Subsequent
* management of this list is done by freelist.c.
diff --git a/src/backend/storage/buffer/bufmgr.c b/src/backend/storage/buffer/bufmgr.c
index e14888ace85..7526af9ccfc 100644
--- a/src/backend/storage/buffer/bufmgr.c
+++ b/src/backend/storage/buffer/bufmgr.c
@@ -48,6 +48,7 @@
#include "pg_trace.h"
#include "pgstat.h"
#include "postmaster/bgwriter.h"
+#include "storage/aio.h"
#include "storage/buf_internals.h"
#include "storage/bufmgr.h"
#include "storage/fd.h"
@@ -516,7 +517,8 @@ static int SyncOneBuffer(int buf_id, bool skip_recently_used,
static void WaitIO(BufferDesc *buf);
static bool StartBufferIO(BufferDesc *buf, bool forInput, bool nowait);
static void TerminateBufferIO(BufferDesc *buf, bool clear_dirty,
- uint32 set_flag_bits, bool forget_owner);
+ uint32 set_flag_bits, bool forget_owner,
+ bool syncio);
static void AbortBufferIO(Buffer buffer);
static void shared_buffer_write_error_callback(void *arg);
static void local_buffer_write_error_callback(void *arg);
@@ -1083,7 +1085,7 @@ ZeroAndLockBuffer(Buffer buffer, ReadBufferMode mode, bool already_valid)
else
{
/* Set BM_VALID, terminate IO, and wake up any waiters */
- TerminateBufferIO(bufHdr, false, BM_VALID, true);
+ TerminateBufferIO(bufHdr, false, BM_VALID, true, true);
}
}
else if (!isLocalBuf)
@@ -1619,7 +1621,7 @@ WaitReadBuffers(ReadBuffersOperation *operation)
else
{
/* Set BM_VALID, terminate IO, and wake up any waiters */
- TerminateBufferIO(bufHdr, false, BM_VALID, true);
+ TerminateBufferIO(bufHdr, false, BM_VALID, true, true);
}
/* Report I/Os as completing individually. */
@@ -2531,7 +2533,7 @@ ExtendBufferedRelShared(BufferManagerRelation bmr,
if (lock)
LWLockAcquire(BufferDescriptorGetContentLock(buf_hdr), LW_EXCLUSIVE);
- TerminateBufferIO(buf_hdr, false, BM_VALID, true);
+ TerminateBufferIO(buf_hdr, false, BM_VALID, true, true);
}
pgBufferUsage.shared_blks_written += extend_by;
@@ -3988,7 +3990,7 @@ FlushBuffer(BufferDesc *buf, SMgrRelation reln, IOObject io_object,
* Mark the buffer as clean (unless BM_JUST_DIRTIED has become set) and
* end the BM_IO_IN_PROGRESS state.
*/
- TerminateBufferIO(buf, true, 0, true);
+ TerminateBufferIO(buf, true, 0, true, true);
TRACE_POSTGRESQL_BUFFER_FLUSH_DONE(BufTagGetForkNum(&buf->tag),
buf->tag.blockNum,
@@ -5568,6 +5570,7 @@ WaitIO(BufferDesc *buf)
for (;;)
{
uint32 buf_state;
+ PgAioWaitRef iow;
/*
* It may not be necessary to acquire the spinlock to check the flag
@@ -5575,10 +5578,19 @@ WaitIO(BufferDesc *buf)
* play it safe.
*/
buf_state = LockBufHdr(buf);
+ iow = buf->io_wref;
UnlockBufHdr(buf, buf_state);
if (!(buf_state & BM_IO_IN_PROGRESS))
break;
+
+ if (pgaio_wref_valid(&iow))
+ {
+ pgaio_wref_wait(&iow);
+ ConditionVariablePrepareToSleep(cv);
+ continue;
+ }
+
ConditionVariableSleep(cv, WAIT_EVENT_BUFFER_IO);
}
ConditionVariableCancelSleep();
@@ -5667,7 +5679,7 @@ StartBufferIO(BufferDesc *buf, bool forInput, bool nowait)
*/
static void
TerminateBufferIO(BufferDesc *buf, bool clear_dirty, uint32 set_flag_bits,
- bool forget_owner)
+ bool forget_owner, bool syncio)
{
uint32 buf_state;
@@ -5679,6 +5691,14 @@ TerminateBufferIO(BufferDesc *buf, bool clear_dirty, uint32 set_flag_bits,
if (clear_dirty && !(buf_state & BM_JUST_DIRTIED))
buf_state &= ~(BM_DIRTY | BM_CHECKPOINT_NEEDED);
+ if (!syncio)
+ {
+ /* release ownership by the AIO subsystem */
+ Assert(BUF_STATE_GET_REFCOUNT(buf_state) > 0);
+ buf_state -= BUF_REFCOUNT_ONE;
+ pgaio_wref_clear(&buf->io_wref);
+ }
+
buf_state |= set_flag_bits;
UnlockBufHdr(buf, buf_state);
@@ -5687,6 +5707,40 @@ TerminateBufferIO(BufferDesc *buf, bool clear_dirty, uint32 set_flag_bits,
BufferDescriptorGetBuffer(buf));
ConditionVariableBroadcast(BufferDescriptorGetIOCV(buf));
+
+ /*
+ * If we just released a pin, need to do BM_PIN_COUNT_WAITER handling.
+ * Most of the time the current backend will hold another pin preventing
+ * that from happening, but that's e.g. not the case when completing an IO
+ * another backend started.
+ *
+ * AFIXME: Deduplicate with UnpinBufferNoOwner() or just replace
+ * BM_PIN_COUNT_WAITER with something saner.
+ */
+ /* Support LockBufferForCleanup() */
+ if (buf_state & BM_PIN_COUNT_WAITER)
+ {
+ /*
+ * Acquire the buffer header lock, re-check that there's a waiter.
+ * Another backend could have unpinned this buffer, and already woken
+ * up the waiter. There's no danger of the buffer being replaced
+ * after we unpinned it above, as it's pinned by the waiter.
+ */
+ buf_state = LockBufHdr(buf);
+
+ if ((buf_state & BM_PIN_COUNT_WAITER) &&
+ BUF_STATE_GET_REFCOUNT(buf_state) == 1)
+ {
+ /* we just released the last pin other than the waiter's */
+ int wait_backend_pgprocno = buf->wait_backend_pgprocno;
+
+ buf_state &= ~BM_PIN_COUNT_WAITER;
+ UnlockBufHdr(buf, buf_state);
+ ProcSendSignal(wait_backend_pgprocno);
+ }
+ else
+ UnlockBufHdr(buf, buf_state);
+ }
}
/*
@@ -5735,7 +5789,7 @@ AbortBufferIO(Buffer buffer)
}
}
- TerminateBufferIO(buf_hdr, false, BM_IO_ERROR, false);
+ TerminateBufferIO(buf_hdr, false, BM_IO_ERROR, false, true);
}
/*
@@ -6186,3 +6240,302 @@ EvictUnpinnedBuffer(Buffer buf)
return result;
}
+
+static PgAioResult
+SharedBufferCompleteRead(int buf_off, Buffer buffer, uint8 flags, bool failed)
+{
+ BufferDesc *bufHdr = GetBufferDescriptor(buffer - 1);
+ BufferTag tag = bufHdr->tag;
+ char *bufdata = BufferGetBlock(buffer);
+ PgAioResult result;
+
+ Assert(BufferIsValid(buffer));
+
+#ifdef USE_ASSERT_CHECKING
+ {
+ uint32 buf_state = pg_atomic_read_u32(&bufHdr->state);
+
+ Assert(buf_state & BM_TAG_VALID);
+ Assert(!(buf_state & BM_VALID));
+ Assert(buf_state & BM_IO_IN_PROGRESS);
+ Assert(!(buf_state & BM_DIRTY));
+ }
+#endif
+
+ result.status = ARS_OK;
+
+ /* check for garbage data */
+ if (!failed &&
+ !PageIsVerifiedExtended((Page) bufdata, tag.blockNum,
+ PIV_LOG_WARNING | PIV_REPORT_STAT))
+ {
+ RelFileLocator rlocator = BufTagGetRelFileLocator(&tag);
+
+ if ((flags & READ_BUFFERS_ZERO_ON_ERROR) || zero_damaged_pages)
+ {
+ ereport(LOG,
+ (errcode(ERRCODE_DATA_CORRUPTED),
+ errmsg("invalid page in block %u of relation %s; zeroing out page",
+ tag.blockNum,
+ relpathperm(rlocator, tag.forkNum).str)));
+ memset(bufdata, 0, BLCKSZ);
+ }
+ else
+ {
+ /* mark buffer as having failed */
+ failed = true;
+
+ /* encode error for buffer_readv_report */
+ result.status = ARS_ERROR;
+ result.id = PGAIO_HCB_SHARED_BUFFER_READV;
+ result.error_data = buf_off;
+ }
+ }
+
+ /* Terminate I/O and set BM_VALID. */
+ TerminateBufferIO(bufHdr, false,
+ failed ? BM_IO_ERROR : BM_VALID,
+ false, false);
+
+ TRACE_POSTGRESQL_BUFFER_READ_DONE(tag.forkNum,
+ tag.blockNum,
+ tag.spcOid,
+ tag.dbOid,
+ tag.relNumber,
+ INVALID_PROC_NUMBER,
+ false);
+
+ return result;
+}
+
+/*
+ * Helper to prepare IO on shared buffers for execution, shared between reads
+ * and writes.
+ */
+static void
+shared_buffer_stage_common(PgAioHandle *ioh, bool is_write)
+{
+ uint64 *io_data;
+ uint8 handle_data_len;
+ PgAioWaitRef io_ref;
+ BufferTag first PG_USED_FOR_ASSERTS_ONLY = {0};
+
+ io_data = pgaio_io_get_handle_data(ioh, &handle_data_len);
+
+ pgaio_io_get_wref(ioh, &io_ref);
+
+ for (int i = 0; i < handle_data_len; i++)
+ {
+ Buffer buf = (Buffer) io_data[i];
+ BufferDesc *bufHdr;
+ uint32 buf_state;
+
+ bufHdr = GetBufferDescriptor(buf - 1);
+
+ if (i == 0)
+ first = bufHdr->tag;
+ else
+ {
+ Assert(bufHdr->tag.relNumber == first.relNumber);
+ Assert(bufHdr->tag.blockNum == first.blockNum + i);
+ }
+
+
+ buf_state = LockBufHdr(bufHdr);
+
+ Assert(buf_state & BM_TAG_VALID);
+ if (is_write)
+ {
+ Assert(buf_state & BM_VALID);
+ Assert(buf_state & BM_DIRTY);
+ }
+ else
+ Assert(!(buf_state & BM_VALID));
+
+ Assert(buf_state & BM_IO_IN_PROGRESS);
+ Assert(BUF_STATE_GET_REFCOUNT(buf_state) >= 1);
+
+ buf_state += BUF_REFCOUNT_ONE;
+ bufHdr->io_wref = io_ref;
+
+ UnlockBufHdr(bufHdr, buf_state);
+
+ if (is_write)
+ {
+ LWLock *content_lock;
+
+ content_lock = BufferDescriptorGetContentLock(bufHdr);
+
+ Assert(LWLockHeldByMe(content_lock));
+
+ /*
+ * Lock is now owned by AIO subsystem.
+ */
+ LWLockDisown(content_lock);
+ }
+
+ /*
+ * Stop tracking this buffer via the resowner - the AIO system now
+ * keeps track.
+ */
+ ResourceOwnerForgetBufferIO(CurrentResourceOwner, buf);
+ }
+}
+
+static void
+shared_buffer_readv_stage(PgAioHandle *ioh, uint8 cb_data)
+{
+ shared_buffer_stage_common(ioh, false);
+}
+
+static void
+buffer_readv_report(PgAioResult result, const PgAioTargetData *target_data, int elevel)
+{
+ ProcNumber errProc;
+
+ if (target_data->smgr.is_temp)
+ errProc = MyProcNumber;
+ else
+ errProc = INVALID_PROC_NUMBER;
+
+ ereport(elevel,
+ errcode(ERRCODE_DATA_CORRUPTED),
+ errmsg("invalid page in block %u of relation %s",
+ target_data->smgr.blockNum + result.error_data,
+ relpathbackend(target_data->smgr.rlocator, errProc, target_data->smgr.forkNum).str
+ )
+ );
+}
+
+/*
+ * Perform completion handling of a single AIO read. This read may cover
+ * multiple blocks / buffers.
+ *
+ * Shared between shared and local buffers, to reduce code duplication.
+ */
+static PgAioResult
+buffer_readv_complete_common(PgAioHandle *ioh, PgAioResult prior_result, bool is_temp, uint8 cb_data)
+{
+ PgAioResult result = prior_result;
+ PgAioTargetData *td = pgaio_io_get_target_data(ioh);
+ uint64 *io_data;
+ uint8 handle_data_len;
+
+ if (is_temp)
+ {
+ Assert(td->smgr.is_temp);
+ Assert(pgaio_io_get_owner(ioh) == MyProcNumber);
+ }
+ else
+ Assert(!td->smgr.is_temp);
+
+ /*
+ * Iterate over all the buffers affected by this IO and call appropriate
+ * per-buffer completion function for each buffer.
+ */
+ io_data = pgaio_io_get_handle_data(ioh, &handle_data_len);
+ for (int buf_off = 0; buf_off < handle_data_len; buf_off++)
+ {
+ Buffer buf = io_data[buf_off];
+ PgAioResult buf_result;
+ bool failed;
+
+ /*
+ * If the entire failed on a lower-level, each buffer needs to be
+ * marked as failed. In case of a partial read, some buffers may be
+ * ok.
+ */
+ failed =
+ prior_result.status == ARS_ERROR
+ || prior_result.result <= buf_off;
+
+ if (is_temp)
+ buf_result = LocalBufferCompleteRead(buf_off, buf, cb_data, failed);
+ else
+ buf_result = SharedBufferCompleteRead(buf_off, buf, cb_data, failed);
+
+ /*
+ * If there wasn't any prior error and the IO for this page failed in
+ * some form, set the whole IO's to the page's result.
+ */
+ if (result.status != ARS_ERROR && buf_result.status != ARS_OK)
+ {
+ buffer_readv_report(result, td, LOG);
+ result = buf_result;
+ }
+ }
+
+ return result;
+}
+
+static PgAioResult
+shared_buffer_readv_complete(PgAioHandle *ioh, PgAioResult prior_result, uint8 cb_data)
+{
+ return buffer_readv_complete_common(ioh, prior_result, false, cb_data);
+}
+
+/*
+ * Helper to stage IO on local buffers for execution, shared between reads
+ * and writes.
+ */
+static void
+local_buffer_readv_stage(PgAioHandle *ioh, uint8 cb_data)
+{
+ uint64 *io_data;
+ uint8 handle_data_len;
+ PgAioWaitRef io_wref;
+
+ io_data = pgaio_io_get_handle_data(ioh, &handle_data_len);
+
+ pgaio_io_get_wref(ioh, &io_wref);
+
+ for (int i = 0; i < handle_data_len; i++)
+ {
+ Buffer buf = (Buffer) io_data[i];
+ BufferDesc *bufHdr;
+ uint32 buf_state;
+
+ bufHdr = GetLocalBufferDescriptor(-buf - 1);
+
+ buf_state = pg_atomic_read_u32(&bufHdr->state);
+
+ bufHdr->io_wref = io_wref;
+
+ /*
+ * Track pin by AIO subsystem in BufferDesc, not in LocalRefCount as
+ * one might initially think. This is necessary to handle this backend
+ * erroring out while AIO is still in progress.
+ */
+ buf_state += BUF_REFCOUNT_ONE;
+
+ pg_atomic_unlocked_write_u32(&bufHdr->state, buf_state);
+ }
+}
+
+static PgAioResult
+local_buffer_readv_complete(PgAioHandle *ioh, PgAioResult prior_result, uint8 cb_data)
+{
+ return buffer_readv_complete_common(ioh, prior_result, true, cb_data);
+}
+
+
+/* readv callback is is passed READ_BUFFERS_* flags as callback data */
+const struct PgAioHandleCallbacks aio_shared_buffer_readv_cb = {
+ .stage = shared_buffer_readv_stage,
+ .complete_shared = shared_buffer_readv_complete,
+ .report = buffer_readv_report,
+};
+
+/* readv callback is is passed READ_BUFFERS_* flags as callback data */
+const struct PgAioHandleCallbacks aio_local_buffer_readv_cb = {
+ .stage = local_buffer_readv_stage,
+
+ /*
+ * Note that this, in contrast to the shared_buffers case, uses
+ * complete_local, as only the issuing backend has access to the required
+ * datastructures. This is important in case the IO completion may be
+ * consumed incidentally by another backend.
+ */
+ .complete_local = local_buffer_readv_complete,
+ .report = buffer_readv_report,
+};
diff --git a/src/backend/storage/buffer/localbuf.c b/src/backend/storage/buffer/localbuf.c
index 913a91f061e..2b7bb5d3dc8 100644
--- a/src/backend/storage/buffer/localbuf.c
+++ b/src/backend/storage/buffer/localbuf.c
@@ -17,7 +17,9 @@
#include "access/parallel.h"
#include "executor/instrument.h"
+#include "pg_trace.h"
#include "pgstat.h"
+#include "storage/aio.h"
#include "storage/buf_internals.h"
#include "storage/bufmgr.h"
#include "storage/fd.h"
@@ -649,6 +651,8 @@ InitLocalBuffers(void)
*/
buf->buf_id = -i - 2;
+ pgaio_wref_clear(&buf->io_wref);
+
/*
* Intentionally do not initialize the buffer's atomic variable
* (besides zeroing the underlying memory above). That way we get
@@ -876,3 +880,76 @@ AtProcExit_LocalBuffers(void)
*/
CheckForLocalBufferLeaks();
}
+
+PgAioResult
+LocalBufferCompleteRead(int buf_off, Buffer buffer, uint8 flags, bool failed)
+{
+ BufferDesc *bufHdr = GetLocalBufferDescriptor(-buffer - 1);
+ BufferTag tag = bufHdr->tag;
+ char *bufdata = BufferGetBlock(buffer);
+ PgAioResult result;
+
+ Assert(BufferIsValid(buffer));
+
+ result.status = ARS_OK;
+
+ /* check for garbage data */
+ if (!failed &&
+ !PageIsVerifiedExtended((Page) bufdata, tag.blockNum,
+ PIV_LOG_WARNING | PIV_REPORT_STAT))
+ {
+ RelFileLocator rlocator = BufTagGetRelFileLocator(&tag);
+ BlockNumber forkNum = tag.forkNum;
+
+ MemoryContextSwitchTo(ErrorContext);
+
+ if ((flags & READ_BUFFERS_ZERO_ON_ERROR) || zero_damaged_pages)
+ {
+
+ ereport(LOG,
+ (errcode(ERRCODE_DATA_CORRUPTED),
+ errmsg("invalid page in block %u of relation %s; zeroing out page",
+ tag.blockNum,
+ relpathbackend(rlocator, MyProcNumber, forkNum).str)));
+ memset(bufdata, 0, BLCKSZ);
+ }
+ else
+ {
+ /* mark buffer as having failed */
+ failed = true;
+
+ /* encode error for buffer_readv_report */
+ result.status = ARS_ERROR;
+ result.id = PGAIO_HCB_LOCAL_BUFFER_READV;
+ result.error_data = buf_off;
+ }
+ }
+
+ /* Terminate I/O and set BM_VALID. */
+ pgaio_wref_clear(&bufHdr->io_wref);
+
+ {
+ uint32 buf_state;
+
+ buf_state = pg_atomic_read_u32(&bufHdr->state);
+ buf_state |= BM_VALID;
+
+ /*
+ * Release pin held by IO subsystem, see also
+ * local_buffer_readv_prepare().
+ */
+ buf_state -= BUF_REFCOUNT_ONE;
+ pg_atomic_unlocked_write_u32(&bufHdr->state, buf_state);
+ }
+
+
+ TRACE_POSTGRESQL_BUFFER_READ_DONE(tag.forkNum,
+ tag.blockNum,
+ tag.spcOid,
+ tag.dbOid,
+ tag.relNumber,
+ INVALID_PROC_NUMBER,
+ false);
+
+ return result;
+}
--
2.48.1.76.g4e746b1a31.dirty
[text/x-diff] v2.5-0018-bufmgr-Use-aio-for-StartReadBuffers.patch (18.8K, ../../vz5i2x2wkjjpp6l2z4g4l3umpxktrxsejgdwefi7n6kzw666h3@rnhlvlpadig7/19-v2.5-0018-bufmgr-Use-aio-for-StartReadBuffers.patch)
download | inline diff:
From dc958619858a2ef83c21ae9a7ce0f692473190e8 Mon Sep 17 00:00:00 2001
From: Andres Freund <andres@anarazel.de>
Date: Thu, 20 Feb 2025 18:08:10 -0500
Subject: [PATCH v2.5 18/30] bufmgr: Use aio for StartReadBuffers()
Author:
Reviewed-By:
Discussion: https://postgr.es/m/
Backpatch:
---
src/include/storage/bufmgr.h | 7 +
src/backend/storage/buffer/bufmgr.c | 412 +++++++++++++++++++++-------
2 files changed, 318 insertions(+), 101 deletions(-)
diff --git a/src/include/storage/bufmgr.h b/src/include/storage/bufmgr.h
index 8bd8d75e940..beeb4c47c1c 100644
--- a/src/include/storage/bufmgr.h
+++ b/src/include/storage/bufmgr.h
@@ -15,6 +15,7 @@
#define BUFMGR_H
#include "port/pg_iovec.h"
+#include "storage/aio_types.h"
#include "storage/block.h"
#include "storage/buf.h"
#include "storage/bufpage.h"
@@ -111,6 +112,9 @@ typedef struct BufferManagerRelation
#define READ_BUFFERS_ZERO_ON_ERROR (1 << 0)
/* Call smgrprefetch() if I/O necessary. */
#define READ_BUFFERS_ISSUE_ADVICE (1 << 1)
+/* IO will immediately be waited for */
+#define READ_BUFFERS_SYNCHRONOUSLY (1 << 2)
+
struct ReadBuffersOperation
{
@@ -130,6 +134,9 @@ struct ReadBuffersOperation
BlockNumber blocknum;
int flags;
int16 nblocks;
+
+ PgAioWaitRef io_wref;
+ PgAioReturn io_return;
};
typedef struct ReadBuffersOperation ReadBuffersOperation;
diff --git a/src/backend/storage/buffer/bufmgr.c b/src/backend/storage/buffer/bufmgr.c
index 7526af9ccfc..6aa5d51d693 100644
--- a/src/backend/storage/buffer/bufmgr.c
+++ b/src/backend/storage/buffer/bufmgr.c
@@ -528,6 +528,8 @@ static inline BufferDesc *BufferAlloc(SMgrRelation smgr,
BlockNumber blockNum,
BufferAccessStrategy strategy,
bool *foundPtr, IOContext io_context);
+static bool AsyncReadBuffers(ReadBuffersOperation *operation,
+ int *nblocks);
static Buffer GetVictimBuffer(BufferAccessStrategy strategy, IOContext io_context);
static void FlushBuffer(BufferDesc *buf, SMgrRelation reln,
IOObject io_object, IOContext io_context);
@@ -1236,10 +1238,9 @@ ReadBuffer_common(Relation rel, SMgrRelation smgr, char smgr_persistence,
return buffer;
}
+ flags = READ_BUFFERS_SYNCHRONOUSLY;
if (mode == RBM_ZERO_ON_ERROR)
- flags = READ_BUFFERS_ZERO_ON_ERROR;
- else
- flags = 0;
+ flags |= READ_BUFFERS_ZERO_ON_ERROR;
operation.smgr = smgr;
operation.rel = rel;
operation.persistence = persistence;
@@ -1267,6 +1268,7 @@ StartReadBuffersImpl(ReadBuffersOperation *operation,
Assert(*nblocks > 0);
Assert(*nblocks <= MAX_IO_COMBINE_LIMIT);
+ Assert(*nblocks == 1 || allow_forwarding);
for (int i = 0; i < actual_nblocks; ++i)
{
@@ -1306,6 +1308,11 @@ StartReadBuffersImpl(ReadBuffersOperation *operation,
else
bufHdr = GetBufferDescriptor(buffers[i] - 1);
found = pg_atomic_read_u32(&bufHdr->state) & BM_VALID;
+
+ ereport(DEBUG3,
+ errmsg("found forwarded buffer %d",
+ buffers[i]),
+ errhidestmt(true), errhidecontext(true));
}
else
{
@@ -1371,25 +1378,59 @@ StartReadBuffersImpl(ReadBuffersOperation *operation,
operation->blocknum = blockNum;
operation->flags = flags;
operation->nblocks = actual_nblocks;
+ pgaio_wref_clear(&operation->io_wref);
- if (flags & READ_BUFFERS_ISSUE_ADVICE)
+ /*
+ * When using AIO, start the IO in the background. If not, issue prefetch
+ * requests if desired by the caller.
+ *
+ * The reason we have a dedicated path for IOMETHOD_SYNC here is to
+ * de-risk the introduction of AIO somewhat. It's a large architectural
+ * change, with lots of chances for unanticipated performance effects.
+ *
+ * Use of IOMETHOD_SYNC already leads to not actually performing IO
+ * asynchronously, but without the check here we'd execute IO earlier than
+ * we used to. Eventually this IOMETHOD_SYNC specific path should go away.
+ */
+ if (io_method != IOMETHOD_SYNC)
{
/*
- * In theory we should only do this if PinBufferForBlock() had to
- * allocate new buffers above. That way, if two calls to
- * StartReadBuffers() were made for the same blocks before
- * WaitReadBuffers(), only the first would issue the advice. That'd be
- * a better simulation of true asynchronous I/O, which would only
- * start the I/O once, but isn't done here for simplicity.
+ * Try to start IO asynchronously. It's possible that no IO needs to
+ * be started, if another backend already performed the IO.
+ *
+ * Note that if an IO is started, it might not cover the entire
+ * requested range, e.g. because an intermediary block has been read
+ * in by another backend. In that case any "trailing" buffers we
+ * already pinned above will be "forwarded" by read_stream.c to the
+ * next call to StartReadBuffers(). This is signalled to the caller by
+ * decrementing *nblocks.
*/
- smgrprefetch(operation->smgr,
- operation->forknum,
- blockNum,
- actual_nblocks);
+ return AsyncReadBuffers(operation, nblocks);
}
+ else
+ {
+ operation->flags |= READ_BUFFERS_SYNCHRONOUSLY;
- /* Indicate that WaitReadBuffers() should be called. */
- return true;
+ if (flags & READ_BUFFERS_ISSUE_ADVICE)
+ {
+ /*
+ * In theory we should only do this if PinBufferForBlock() had to
+ * allocate new buffers above. That way, if two calls to
+ * StartReadBuffers() were made for the same blocks before
+ * WaitReadBuffers(), only the first would issue the advice.
+ * That'd be a better simulation of true asynchronous I/O, which
+ * would only start the I/O once, but isn't done here for
+ * simplicity.
+ */
+ smgrprefetch(operation->smgr,
+ operation->forknum,
+ blockNum,
+ actual_nblocks);
+ }
+
+ /* Indicate that WaitReadBuffers() should be called. */
+ return true;
+ }
}
/*
@@ -1457,12 +1498,31 @@ StartReadBuffer(ReadBuffersOperation *operation,
}
static inline bool
-WaitReadBuffersCanStartIO(Buffer buffer, bool nowait)
+ReadBuffersCanStartIO(Buffer buffer, bool nowait)
{
if (BufferIsLocal(buffer))
{
BufferDesc *bufHdr = GetLocalBufferDescriptor(-buffer - 1);
+ /*
+ * The buffer could have IO in progress by another scan. Right now
+ * localbuf.c doesn't use IO_IN_PROGRESS, which is why we need this
+ * hack.
+ *
+ * TODO: localbuf.c should use IO_IN_PROGRESS / have an equivalent of
+ * StartBufferIO().
+ */
+ if (pgaio_wref_valid(&bufHdr->io_wref))
+ {
+ PgAioWaitRef iow = bufHdr->io_wref;
+
+ ereport(DEBUG3,
+ errmsg("waiting for temp buffer IO in CSIO"),
+ errhidestmt(true), errhidecontext(true));
+ pgaio_wref_wait(&iow);
+ return false;
+ }
+
return (pg_atomic_read_u32(&bufHdr->state) & BM_VALID) == 0;
}
else
@@ -1472,28 +1532,163 @@ WaitReadBuffersCanStartIO(Buffer buffer, bool nowait)
void
WaitReadBuffers(ReadBuffersOperation *operation)
{
- Buffer *buffers;
+ IOContext io_context;
+ IOObject io_object;
int nblocks;
- BlockNumber blocknum;
- ForkNumber forknum;
- IOContext io_context;
- IOObject io_object;
- char persistence;
+ PgAioReturn *aio_ret;
+
+ /*
+ * If we get here without an IO operation having been issued, io_method ==
+ * IOMETHOD_SYNC path must have been used. In that case, we start - as we
+ * used to before - the IO now, just before waiting.
+ *
+ * This path is expected to eventually go away.
+ */
+ if (!pgaio_wref_valid(&operation->io_wref))
+ {
+ Assert(io_method == IOMETHOD_SYNC);
+
+ while (true)
+ {
+ nblocks = operation->nblocks;
+
+ if (!AsyncReadBuffers(operation, &nblocks))
+ {
+ /* all blocks were already read in concurrently */
+ Assert(nblocks == operation->nblocks);
+ return;
+ }
+
+ Assert(nblocks > 0 && nblocks <= operation->nblocks);
+
+ if (nblocks == operation->nblocks)
+ {
+ /* will wait below as if this had been normal AIO */
+ break;
+ }
+
+ /*
+ * It's unlikely, but possible, that AsyncReadBuffers() wasn't
+ * able to initiate IO for all the relevant buffers. In that case
+ * we need to wait for the prior IO before issuing more IO.
+ */
+ WaitReadBuffers(operation);
+ }
+ }
+
+ if (operation->persistence == RELPERSISTENCE_TEMP)
+ {
+ io_context = IOCONTEXT_NORMAL;
+ io_object = IOOBJECT_TEMP_RELATION;
+ }
+ else
+ {
+ io_context = IOContextForStrategy(operation->strategy);
+ io_object = IOOBJECT_RELATION;
+ }
+
+restart:
/* Find the range of the physical read we need to perform. */
nblocks = operation->nblocks;
- buffers = &operation->buffers[0];
- blocknum = operation->blocknum;
- forknum = operation->forknum;
- persistence = operation->persistence;
-
Assert(nblocks > 0);
Assert(nblocks <= MAX_IO_COMBINE_LIMIT);
+ aio_ret = &operation->io_return;
+
+ /*
+ * For IO timing we just count the time spent waiting for the IO.
+ *
+ * XXX: We probably should track the IO operation, rather than its time,
+ * separately, when initiating the IO. But right now that's not quite
+ * allowed by the interface.
+ */
+
+ /*
+ * Tracking a wait even if we don't actually need to wait
+ *
+ * a) is not cheap
+ *
+ * b) reports some time as waiting, even if we never waited.
+ */
+ if (aio_ret->result.status == ARS_UNKNOWN &&
+ !pgaio_wref_check_done(&operation->io_wref))
+ {
+ instr_time io_start = pgstat_prepare_io_time(track_io_timing);
+
+ pgaio_wref_wait(&operation->io_wref);
+
+ /*
+ * The IO operation itself was already counted earlier, in
+ * AsyncReadBuffers().
+ */
+ pgstat_count_io_op_time(io_object, io_context, IOOP_READ,
+ io_start, 0, 0);
+ }
+ else
+ {
+ Assert(pgaio_wref_check_done(&operation->io_wref));
+ }
+
+ if (aio_ret->result.status == ARS_PARTIAL)
+ {
+ /*
+ * We'll retry below, so we just emit a debug message the server log
+ * (or not even that in prod scenarios).
+ */
+ pgaio_result_report(aio_ret->result, &aio_ret->target_data, DEBUG1);
+
+ /*
+ * Try to perform the rest of the IO. Buffers for which IO has
+ * completed successfully will be discovered as such and not retried.
+ */
+ nblocks = operation->nblocks;
+
+ elog(DEBUG3, "retrying IO after partial failure");
+ CHECK_FOR_INTERRUPTS();
+ AsyncReadBuffers(operation, &nblocks);
+ goto restart;
+ }
+ else if (aio_ret->result.status != ARS_OK)
+ pgaio_result_report(aio_ret->result, &aio_ret->target_data, ERROR);
+
+ if (VacuumCostActive)
+ VacuumCostBalance += VacuumCostPageMiss * nblocks;
+
+ /* NB: READ_DONE tracepoint is executed in IO completion callback */
+}
+
+/*
+ * Initiate IO for the ReadBuffersOperation. If IO is only initiated for a
+ * subset of the blocks, *nblocks is updated to reflect that.
+ *
+ * Returns true if IO was initiated, false if no IO was necessary.
+ */
+static bool
+AsyncReadBuffers(ReadBuffersOperation *operation,
+ int *nblocks)
+{
+ int io_buffers_len = 0;
+ Buffer *buffers = &operation->buffers[0];
+ int flags = operation->flags;
+ BlockNumber blocknum = operation->blocknum;
+ ForkNumber forknum = operation->forknum;
+ bool did_start_io = false;
+ PgAioHandle *ioh = NULL;
+ uint32 ioh_flags = 0;
+ IOContext io_context;
+ IOObject io_object;
+ char persistence;
+
+ persistence = operation->rel
+ ? operation->rel->rd_rel->relpersistence
+ : RELPERSISTENCE_PERMANENT;
+
if (persistence == RELPERSISTENCE_TEMP)
{
io_context = IOCONTEXT_NORMAL;
io_object = IOOBJECT_TEMP_RELATION;
+ ioh_flags |= PGAIO_HF_REFERENCES_LOCAL;
}
else
{
@@ -1501,6 +1696,14 @@ WaitReadBuffers(ReadBuffersOperation *operation)
io_object = IOOBJECT_RELATION;
}
+ /*
+ * When this IO is executed synchronously, either because the caller will
+ * immediately block waiting for the IO or because IOMETHOD_SYNC is used,
+ * the AIO subsystem needs to know.
+ */
+ if (flags & READ_BUFFERS_SYNCHRONOUSLY)
+ ioh_flags |= PGAIO_HF_SYNCHRONOUS;
+
/*
* We count all these blocks as read by this backend. This is traditional
* behavior, but might turn out to be not true if we find that someone
@@ -1510,25 +1713,53 @@ WaitReadBuffers(ReadBuffersOperation *operation)
* but another backend completed the read".
*/
if (persistence == RELPERSISTENCE_TEMP)
- pgBufferUsage.local_blks_read += nblocks;
+ pgBufferUsage.local_blks_read += *nblocks;
else
- pgBufferUsage.shared_blks_read += nblocks;
+ pgBufferUsage.shared_blks_read += *nblocks;
- for (int i = 0; i < nblocks; ++i)
+ pgaio_wref_clear(&operation->io_wref);
+
+ /*
+ * Loop until we have started one IO or we discover that all buffers are
+ * already valid.
+ */
+ for (int i = 0; i < *nblocks; ++i)
{
- int io_buffers_len;
Buffer io_buffers[MAX_IO_COMBINE_LIMIT];
void *io_pages[MAX_IO_COMBINE_LIMIT];
- instr_time io_start;
BlockNumber io_first_block;
/*
- * Skip this block if someone else has already completed it. If an
- * I/O is already in progress in another backend, this will wait for
- * the outcome: either done, or something went wrong and we will
- * retry.
+ * Get IO before ReadBuffersCanStartIO, as pgaio_io_acquire() might
+ * block, which we don't want after setting IO_IN_PROGRESS.
+ *
+ * XXX: Should we attribute the time spent in here to the IO? If there
+ * already are a lot of IO operations in progress, getting an IO
+ * handle will block waiting for some other IO operation to finish.
+ *
+ * In most cases it'll be free to get the IO, so a timer would be
+ * overhead. Perhaps we should use pgaio_io_acquire_nb() and only
+ * account IO time when pgaio_io_acquire_nb() returned false?
*/
- if (!WaitReadBuffersCanStartIO(buffers[i], false))
+ if (likely(!ioh))
+ ioh = pgaio_io_acquire(CurrentResourceOwner,
+ &operation->io_return);
+
+ /*
+ * Skip this block if someone else has already completed it.
+ *
+ * If an I/O is already in progress in another backend, this will wait
+ * for the outcome: either done, or something went wrong and we will
+ * retry. But don't wait if we have staged, but haven't issued,
+ * another IO.
+ *
+ * It's safe to start IO while we have unsubmitted IO, but it'd be
+ * better to first submit it. But right now the boolean return value
+ * from ReadBuffersCanStartIO()/StartBufferIO() doesn't allow to
+ * distinguish between nowait=true trigger failure and the buffer
+ * already being valid.
+ */
+ if (!ReadBuffersCanStartIO(buffers[i], false))
{
/*
* Report this as a 'hit' for this backend, even though it must
@@ -1540,6 +1771,11 @@ WaitReadBuffers(ReadBuffersOperation *operation)
operation->smgr->smgr_rlocator.locator.relNumber,
operation->smgr->smgr_rlocator.backend,
true);
+
+ ereport(DEBUG3,
+ errmsg("can't start io for first buffer %u: %s",
+ buffers[i], DebugPrintBufferRefcount(buffers[i])),
+ errhidestmt(true), errhidecontext(true));
continue;
}
@@ -1549,6 +1785,11 @@ WaitReadBuffers(ReadBuffersOperation *operation)
io_first_block = blocknum + i;
io_buffers_len = 1;
+ ereport(DEBUG5,
+ errmsg("first prepped for io: %s, offset %d",
+ DebugPrintBufferRefcount(io_buffers[0]), i),
+ errhidestmt(true), errhidecontext(true));
+
/*
* How many neighboring-on-disk blocks can we scatter-read into other
* buffers at the same time? In this case we don't wait if we see an
@@ -1556,86 +1797,55 @@ WaitReadBuffers(ReadBuffersOperation *operation)
* head block, so we should get on with that I/O as soon as possible.
* We'll come back to this block again, above.
*/
- while ((i + 1) < nblocks &&
- WaitReadBuffersCanStartIO(buffers[i + 1], true))
+ while ((i + 1) < *nblocks &&
+ ReadBuffersCanStartIO(buffers[i + 1], true))
{
/* Must be consecutive block numbers. */
Assert(BufferGetBlockNumber(buffers[i + 1]) ==
BufferGetBlockNumber(buffers[i]) + 1);
+ ereport(DEBUG5,
+ errmsg("seq prepped for io: %s, offset %d",
+ DebugPrintBufferRefcount(buffers[i + 1]),
+ i + 1),
+ errhidestmt(true), errhidecontext(true));
+
io_buffers[io_buffers_len] = buffers[++i];
io_pages[io_buffers_len++] = BufferGetBlock(buffers[i]);
}
- io_start = pgstat_prepare_io_time(track_io_timing);
- smgrreadv(operation->smgr, forknum, io_first_block, io_pages, io_buffers_len);
- pgstat_count_io_op_time(io_object, io_context, IOOP_READ, io_start,
- 1, io_buffers_len * BLCKSZ);
+ pgaio_io_get_wref(ioh, &operation->io_wref);
- /* Verify each block we read, and terminate the I/O. */
- for (int j = 0; j < io_buffers_len; ++j)
- {
- BufferDesc *bufHdr;
- Block bufBlock;
+ pgaio_io_set_handle_data_32(ioh, (uint32 *) io_buffers, io_buffers_len);
- if (persistence == RELPERSISTENCE_TEMP)
- {
- bufHdr = GetLocalBufferDescriptor(-io_buffers[j] - 1);
- bufBlock = LocalBufHdrGetBlock(bufHdr);
- }
- else
- {
- bufHdr = GetBufferDescriptor(io_buffers[j] - 1);
- bufBlock = BufHdrGetBlock(bufHdr);
- }
+ pgaio_io_register_callbacks(ioh,
+ persistence == RELPERSISTENCE_TEMP ?
+ PGAIO_HCB_LOCAL_BUFFER_READV :
+ PGAIO_HCB_SHARED_BUFFER_READV,
+ flags);
- /* check for garbage data */
- if (!PageIsVerifiedExtended((Page) bufBlock, io_first_block + j,
- PIV_LOG_WARNING | PIV_REPORT_STAT))
- {
- if ((operation->flags & READ_BUFFERS_ZERO_ON_ERROR) || zero_damaged_pages)
- {
- ereport(WARNING,
- (errcode(ERRCODE_DATA_CORRUPTED),
- errmsg("invalid page in block %u of relation %s; zeroing out page",
- io_first_block + j,
- relpath(operation->smgr->smgr_rlocator, forknum).str)));
- memset(bufBlock, 0, BLCKSZ);
- }
- else
- ereport(ERROR,
- (errcode(ERRCODE_DATA_CORRUPTED),
- errmsg("invalid page in block %u of relation %s",
- io_first_block + j,
- relpath(operation->smgr->smgr_rlocator, forknum).str)));
- }
+ pgaio_io_set_flag(ioh, ioh_flags);
- /* Terminate I/O and set BM_VALID. */
- if (persistence == RELPERSISTENCE_TEMP)
- {
- uint32 buf_state = pg_atomic_read_u32(&bufHdr->state);
+ did_start_io = true;
+ smgrstartreadv(ioh, operation->smgr, forknum, io_first_block,
+ io_pages, io_buffers_len);
+ ioh = NULL;
- buf_state |= BM_VALID;
- pg_atomic_unlocked_write_u32(&bufHdr->state, buf_state);
- }
- else
- {
- /* Set BM_VALID, terminate IO, and wake up any waiters */
- TerminateBufferIO(bufHdr, false, BM_VALID, true, true);
- }
+ /* not obvious what we'd use for time */
+ pgstat_count_io_op(io_object, io_context, IOOP_READ,
+ 1, io_buffers_len * BLCKSZ);
- /* Report I/Os as completing individually. */
- TRACE_POSTGRESQL_BUFFER_READ_DONE(forknum, io_first_block + j,
- operation->smgr->smgr_rlocator.locator.spcOid,
- operation->smgr->smgr_rlocator.locator.dbOid,
- operation->smgr->smgr_rlocator.locator.relNumber,
- operation->smgr->smgr_rlocator.backend,
- false);
- }
+ *nblocks = io_buffers_len;
+ break;
+ }
- if (VacuumCostActive)
- VacuumCostBalance += VacuumCostPageMiss * io_buffers_len;
+ if (ioh)
+ {
+ pgaio_io_release(ioh);
+ ioh = NULL;
}
+
+ return did_start_io;
}
/*
--
2.48.1.76.g4e746b1a31.dirty
[text/x-diff] v2.5-0019-WIP-aio-read_stream.c-adjustments-for-real-AIO.patch (3.7K, ../../vz5i2x2wkjjpp6l2z4g4l3umpxktrxsejgdwefi7n6kzw666h3@rnhlvlpadig7/20-v2.5-0019-WIP-aio-read_stream.c-adjustments-for-real-AIO.patch)
download | inline diff:
From 957228ccf17af3c3911ec7b64fd1e0bad965a261 Mon Sep 17 00:00:00 2001
From: Andres Freund <andres@anarazel.de>
Date: Tue, 11 Feb 2025 18:21:47 -0500
Subject: [PATCH v2.5 19/30] WIP: aio: read_stream.c adjustments for real AIO
Comments need to be fixed.
The batching logic probably needs to be adjusted.
---
src/backend/storage/aio/read_stream.c | 31 +++++++++++++++++++++++++--
1 file changed, 29 insertions(+), 2 deletions(-)
diff --git a/src/backend/storage/aio/read_stream.c b/src/backend/storage/aio/read_stream.c
index e58e0edf221..508e145efcc 100644
--- a/src/backend/storage/aio/read_stream.c
+++ b/src/backend/storage/aio/read_stream.c
@@ -32,10 +32,15 @@
* calls. Looking further ahead would pin many buffers and perform
* speculative work for no benefit.
*
+ * FIXME: This only applies to io_method == sync, otherwise this path is not
+ * used.
+ *
* C) I/O is necessary, it appears to be random, and this system supports
* read-ahead advice. We'll look further ahead in order to reach the
* configured level of I/O concurrency.
*
+ * FIXME: restriction to random only applies to io_method == sync
+ *
* The distance increases rapidly and decays slowly, so that it moves towards
* those levels as different I/O patterns are discovered. For example, a
* sequential scan of fully cached data doesn't bother looking ahead, but a
@@ -90,6 +95,7 @@
#include "postgres.h"
#include "miscadmin.h"
+#include "storage/aio.h"
#include "storage/fd.h"
#include "storage/smgr.h"
#include "storage/read_stream.h"
@@ -116,6 +122,7 @@ struct ReadStream
int16 pinned_buffers;
int16 distance;
int16 initialized_buffers;
+ bool sync_mode;
bool advice_enabled;
bool temporary;
@@ -458,6 +465,19 @@ read_stream_look_ahead(ReadStream *stream, bool suppress_advice)
{
int16 buffer_limit;
+ if (stream->distance > (io_combine_limit * 8))
+ {
+ if (stream->pinned_buffers + stream->pending_read_nblocks > ((stream->distance * 3) / 4))
+ {
+ return;
+ }
+ }
+
+ /*
+ * Try to amortize cost of submitting IOs over multiple IOs.
+ */
+ pgaio_enter_batchmode();
+
/*
* Check how many pins we could acquire now. We do this here rather than
* pushing it down into read_stream_start_pending_read(), because it
@@ -524,6 +544,7 @@ read_stream_look_ahead(ReadStream *stream, bool suppress_advice)
{
/* And we've hit a limit. Rewind, and stop here. */
read_stream_unget_block(stream, blocknum);
+ pgaio_exit_batchmode();
return;
}
}
@@ -549,6 +570,8 @@ read_stream_look_ahead(ReadStream *stream, bool suppress_advice)
stream->distance == 0) &&
stream->ios_in_progress < stream->max_ios)
read_stream_start_pending_read(stream, buffer_limit, suppress_advice);
+
+ pgaio_exit_batchmode();
}
/*
@@ -672,6 +695,8 @@ read_stream_begin_impl(int flags,
stream->per_buffer_data = (void *)
MAXALIGN(&stream->ios[Max(1, max_ios)]);
+ stream->sync_mode = io_method == IOMETHOD_SYNC;
+
#ifdef USE_PREFETCH
/*
@@ -680,7 +705,8 @@ read_stream_begin_impl(int flags,
* (overriding our detection heuristics), and max_ios hasn't been set to
* zero.
*/
- if ((io_direct_flags & IO_DIRECT_DATA) == 0 &&
+ if (stream->sync_mode &&
+ (io_direct_flags & IO_DIRECT_DATA) == 0 &&
(flags & READ_STREAM_SEQUENTIAL) == 0 &&
max_ios > 0)
stream->advice_enabled = true;
@@ -923,7 +949,8 @@ read_stream_next_buffer(ReadStream *stream, void **per_buffer_data)
if (++stream->oldest_io_index == stream->max_ios)
stream->oldest_io_index = 0;
- if (stream->ios[io_index].op.flags & READ_BUFFERS_ISSUE_ADVICE)
+ if (!stream->sync_mode ||
+ stream->ios[io_index].op.flags & READ_BUFFERS_ISSUE_ADVICE)
{
/* Distance ramps up fast (behavior C). */
distance = stream->distance * 2;
--
2.48.1.76.g4e746b1a31.dirty
[text/x-diff] v2.5-0020-aio-Add-test_aio-module.patch (41.7K, ../../vz5i2x2wkjjpp6l2z4g4l3umpxktrxsejgdwefi7n6kzw666h3@rnhlvlpadig7/21-v2.5-0020-aio-Add-test_aio-module.patch)
download | inline diff:
From 7768536dac083a1e951e05bd9a80d3704241a59b Mon Sep 17 00:00:00 2001
From: Andres Freund <andres@anarazel.de>
Date: Wed, 22 Jan 2025 13:44:54 -0500
Subject: [PATCH v2.5 20/30] aio: Add test_aio module
---
src/include/storage/buf_internals.h | 6 +
src/backend/storage/buffer/bufmgr.c | 8 +-
src/test/modules/Makefile | 1 +
src/test/modules/meson.build | 1 +
src/test/modules/test_aio/.gitignore | 2 +
src/test/modules/test_aio/Makefile | 27 +
src/test/modules/test_aio/meson.build | 37 ++
src/test/modules/test_aio/t/001_aio.pl | 652 ++++++++++++++++++++
src/test/modules/test_aio/test_aio--1.0.sql | 97 +++
src/test/modules/test_aio/test_aio.c | 616 ++++++++++++++++++
src/test/modules/test_aio/test_aio.control | 3 +
11 files changed, 1444 insertions(+), 6 deletions(-)
create mode 100644 src/test/modules/test_aio/.gitignore
create mode 100644 src/test/modules/test_aio/Makefile
create mode 100644 src/test/modules/test_aio/meson.build
create mode 100644 src/test/modules/test_aio/t/001_aio.pl
create mode 100644 src/test/modules/test_aio/test_aio--1.0.sql
create mode 100644 src/test/modules/test_aio/test_aio.c
create mode 100644 src/test/modules/test_aio/test_aio.control
diff --git a/src/include/storage/buf_internals.h b/src/include/storage/buf_internals.h
index cdeb2f0b528..817c85ec9ed 100644
--- a/src/include/storage/buf_internals.h
+++ b/src/include/storage/buf_internals.h
@@ -434,6 +434,12 @@ extern void IssuePendingWritebacks(WritebackContext *wb_context, IOContext io_co
extern void ScheduleBufferTagForWriteback(WritebackContext *wb_context,
IOContext io_context, BufferTag *tag);
+/* solely to make it easier to write tests */
+extern bool StartBufferIO(BufferDesc *buf, bool forInput, bool nowait);
+extern void TerminateBufferIO(BufferDesc *buf, bool clear_dirty, uint32 set_flag_bits,
+ bool forget_owner, bool syncio);
+
+
/* freelist.c */
extern IOContext IOContextForStrategy(BufferAccessStrategy strategy);
extern BufferDesc *StrategyGetBuffer(BufferAccessStrategy strategy,
diff --git a/src/backend/storage/buffer/bufmgr.c b/src/backend/storage/buffer/bufmgr.c
index 6aa5d51d693..efd5b6601ad 100644
--- a/src/backend/storage/buffer/bufmgr.c
+++ b/src/backend/storage/buffer/bufmgr.c
@@ -515,10 +515,6 @@ static uint32 WaitBufHdrUnlocked(BufferDesc *buf);
static int SyncOneBuffer(int buf_id, bool skip_recently_used,
WritebackContext *wb_context);
static void WaitIO(BufferDesc *buf);
-static bool StartBufferIO(BufferDesc *buf, bool forInput, bool nowait);
-static void TerminateBufferIO(BufferDesc *buf, bool clear_dirty,
- uint32 set_flag_bits, bool forget_owner,
- bool syncio);
static void AbortBufferIO(Buffer buffer);
static void shared_buffer_write_error_callback(void *arg);
static void local_buffer_write_error_callback(void *arg);
@@ -5830,7 +5826,7 @@ WaitIO(BufferDesc *buf)
* find out if they can perform the I/O as part of a larger operation, without
* waiting for the answer or distinguishing the reasons why not.
*/
-static bool
+bool
StartBufferIO(BufferDesc *buf, bool forInput, bool nowait)
{
uint32 buf_state;
@@ -5887,7 +5883,7 @@ StartBufferIO(BufferDesc *buf, bool forInput, bool nowait)
* resource owner. (forget_owner=false is used when the resource owner itself
* is being released)
*/
-static void
+void
TerminateBufferIO(BufferDesc *buf, bool clear_dirty, uint32 set_flag_bits,
bool forget_owner, bool syncio)
{
diff --git a/src/test/modules/Makefile b/src/test/modules/Makefile
index 4e4be3fa511..aa1d27bbed3 100644
--- a/src/test/modules/Makefile
+++ b/src/test/modules/Makefile
@@ -14,6 +14,7 @@ SUBDIRS = \
oauth_validator \
plsample \
spgist_name_ops \
+ test_aio \
test_bloomfilter \
test_copy_callbacks \
test_custom_rmgrs \
diff --git a/src/test/modules/meson.build b/src/test/modules/meson.build
index 2b057451473..4fda1acfb32 100644
--- a/src/test/modules/meson.build
+++ b/src/test/modules/meson.build
@@ -1,5 +1,6 @@
# Copyright (c) 2022-2025, PostgreSQL Global Development Group
+subdir('test_aio')
subdir('brin')
subdir('commit_ts')
subdir('delay_execution')
diff --git a/src/test/modules/test_aio/.gitignore b/src/test/modules/test_aio/.gitignore
new file mode 100644
index 00000000000..716e17f5a2a
--- /dev/null
+++ b/src/test/modules/test_aio/.gitignore
@@ -0,0 +1,2 @@
+# Generated subdirectories
+/tmp_check/
diff --git a/src/test/modules/test_aio/Makefile b/src/test/modules/test_aio/Makefile
new file mode 100644
index 00000000000..87d5315ba00
--- /dev/null
+++ b/src/test/modules/test_aio/Makefile
@@ -0,0 +1,27 @@
+# src/test/modules/delay_execution/Makefile
+
+PGFILEDESC = "test_aio - test code for AIO"
+
+MODULE_big = test_aio
+OBJS = \
+ $(WIN32RES) \
+ test_aio.o
+
+EXTENSION = test_aio
+DATA = test_aio--1.0.sql
+
+TAP_TESTS = 1
+
+export enable_injection_points
+export with_liburing
+
+ifdef USE_PGXS
+PG_CONFIG = pg_config
+PGXS := $(shell $(PG_CONFIG) --pgxs)
+include $(PGXS)
+else
+subdir = src/test/modules/test_aio
+top_builddir = ../../../..
+include $(top_builddir)/src/Makefile.global
+include $(top_srcdir)/contrib/contrib-global.mk
+endif
diff --git a/src/test/modules/test_aio/meson.build b/src/test/modules/test_aio/meson.build
new file mode 100644
index 00000000000..ac846b2b6f3
--- /dev/null
+++ b/src/test/modules/test_aio/meson.build
@@ -0,0 +1,37 @@
+# Copyright (c) 2022-2024, PostgreSQL Global Development Group
+
+test_aio_sources = files(
+ 'test_aio.c',
+)
+
+if host_system == 'windows'
+ test_aio_sources += rc_lib_gen.process(win32ver_rc, extra_args: [
+ '--NAME', 'test_aio',
+ '--FILEDESC', 'test_aio - test code for AIO',])
+endif
+
+test_aio = shared_module('test_aio',
+ test_aio_sources,
+ kwargs: pg_test_mod_args,
+)
+test_install_libs += test_aio
+
+test_install_data += files(
+ 'test_aio.control',
+ 'test_aio--1.0.sql',
+)
+
+tests += {
+ 'name': 'test_aio',
+ 'sd': meson.current_source_dir(),
+ 'bd': meson.current_build_dir(),
+ 'tap': {
+ 'env': {
+ 'enable_injection_points': get_option('injection_points') ? 'yes' : 'no',
+ 'with_liburing': liburing.found() ? 'yes' : 'no',
+ },
+ 'tests': [
+ 't/001_aio.pl',
+ ],
+ },
+}
diff --git a/src/test/modules/test_aio/t/001_aio.pl b/src/test/modules/test_aio/t/001_aio.pl
new file mode 100644
index 00000000000..2743c9a72d5
--- /dev/null
+++ b/src/test/modules/test_aio/t/001_aio.pl
@@ -0,0 +1,652 @@
+# Copyright (c) 2025, PostgreSQL Global Development Group
+
+use strict;
+use warnings FATAL => 'all';
+
+use PostgreSQL::Test::Cluster;
+use PostgreSQL::Test::Utils;
+use Test::More;
+
+
+###
+# Test io_method=worker
+###
+my $node_worker = create_node('worker');
+$node_worker->start();
+
+run_generic_test('worker', $node_worker);
+SKIP:
+{
+ skip 'Injection points not supported by this build', 1
+ unless $ENV{enable_injection_points} eq 'yes';
+ test_inject_worker('worker', $node_worker);
+}
+
+$node_worker->stop();
+
+
+###
+# Test io_method=io_uring
+###
+
+if ($ENV{with_liburing} eq 'yes')
+{
+ my $node_uring = create_node('io_uring');
+ $node_uring->start();
+ run_generic_test('io_uring', $node_uring);
+ $node_uring->stop();
+}
+
+
+###
+# Test io_method=sync
+###
+
+my $node_sync = create_node('sync');
+
+# just to have one test not use the default auto-tuning
+
+$node_sync->append_conf(
+ 'postgresql.conf', qq(
+io_max_concurrency=4
+));
+
+$node_sync->start();
+run_generic_test('sync', $node_sync);
+$node_sync->stop();
+
+done_testing();
+
+
+###
+# Test Helpers
+###
+
+
+sub create_node
+{
+ my $io_method = shift;
+
+ my $node = PostgreSQL::Test::Cluster->new($io_method);
+
+ # Want to test initdb for each IO method, otherwise we could just reuse
+ # the cluster.
+ $node->init(extra => [ '-c', "io_method=$io_method" ]);
+
+ $node->append_conf(
+ 'postgresql.conf', qq(
+shared_preload_libraries=test_aio
+log_min_messages = 'DEBUG3'
+log_statement=all
+restart_after_crash=false
+));
+
+ ok(1, "$io_method: initdb");
+
+ return $node;
+}
+
+
+sub psql_like
+{
+ my $io_method = shift;
+ my $psql = shift;
+ my $name = shift;
+ my $sql = shift;
+ my $expected_stdout = shift;
+ my $expected_stderr = shift;
+ my ($cmdret, $output);
+
+ ($output, $cmdret) = $psql->query($sql);
+
+ like($output, $expected_stdout, "$io_method: $name: expected stdout");
+ like($psql->{stderr}, $expected_stderr,
+ "$io_method: $name: expected stderr");
+ $psql->{stderr} = '';
+}
+
+sub query_wait_block
+{
+ my $io_method = shift;
+ my $node = shift;
+ my $psql = shift;
+ my $name = shift;
+ my $sql = shift;
+ my $waitfor = shift;
+
+ my $pid = $psql->query_safe('SELECT pg_backend_pid()');
+
+ $psql->{stdin} .= qq($sql;\n);
+ $psql->{run}->pump_nb();
+ ok(1, "$io_method: $name: issued sql");
+
+ $node->poll_query_until('postgres',
+ qq(SELECT wait_event FROM pg_stat_activity WHERE pid = $pid),
+ $waitfor,);
+ ok(1, "$io_method: $name: observed $waitfor wait event");
+}
+
+sub test_handle
+{
+ my $io_method = shift;
+ my $node = shift;
+
+ my $psql = $node->background_psql('postgres', on_error_stop => 0);
+
+ # leak warning: implicit xact
+ psql_like(
+ $io_method,
+ $psql,
+ "handle_get() leak in implicit xact",
+ qq(SELECT handle_get()),
+ qr/^$/,
+ qr/leaked AIO handle/,
+ "$io_method: leaky handle_get() warns");
+
+ # leak warning: explicit xact
+ psql_like(
+ $io_method, $psql,
+ "handle_get() leak in explicit xact",
+ qq(BEGIN; SELECT handle_get(); COMMIT),
+ qr/^$/, qr/leaked AIO handle/);
+
+
+ # leak warning: explicit xact, rollback
+ psql_like(
+ $io_method,
+ $psql,
+ "handle_get() leak in explicit xact, rollback",
+ qq(BEGIN; SELECT handle_get(); ROLLBACK;),
+ qr/^$/,
+ qr/leaked AIO handle/);
+
+ # leak warning: subtrans
+ psql_like(
+ $io_method,
+ $psql,
+ "handle_get() leak in subxact",
+ qq(BEGIN; SAVEPOINT foo; SELECT handle_get(); COMMIT;),
+ qr/^$/,
+ qr/leaked AIO handle/);
+
+ # leak warning + error: released in different command (thus resowner)
+ psql_like(
+ $io_method,
+ $psql,
+ "handle_release() in different command",
+ qq(BEGIN; SELECT handle_get(); SELECT handle_release_last(); COMMIT;),
+ qr/^$/,
+ qr/leaked AIO handle.*release in unexpected state/ms);
+
+ # no leak, release in same command
+ psql_like(
+ $io_method,
+ $psql,
+ "handle_release() in same command",
+ qq(BEGIN; SELECT handle_get() UNION ALL SELECT handle_release_last(); COMMIT;),
+ qr/^$/,
+ qr/^$/);
+
+ # normal handle use
+ psql_like($io_method, $psql, "handle_get_release()",
+ qq(SELECT handle_get_release()),
+ qr/^$/, qr/^$/);
+
+ # should error out, API violation
+ psql_like($io_method, $psql, "handle_get_twice()",
+ qq(SELECT handle_get_release()),
+ qr/^$/, qr/^$/);
+
+ # recover after error in implicit xact
+ psql_like(
+ $io_method,
+ $psql,
+ "handle error recovery in implicit xact",
+ qq(SELECT handle_get_and_error(); SELECT 'ok', handle_get_release()),
+ qr/^|ok$/,
+ qr/ERROR.*as you command/);
+
+ # recover after error in implicit xact
+ psql_like(
+ $io_method,
+ $psql,
+ "handle error recovery in explicit xact",
+ qq(BEGIN; SELECT handle_get_and_error(); SELECT handle_get_release(), 'ok'; COMMIT;),
+ qr/^|ok$/,
+ qr/ERROR.*as you command/);
+
+ # recover after error in subtrans
+ psql_like(
+ $io_method,
+ $psql,
+ "handle error recovery in explicit subxact",
+ qq(BEGIN; SAVEPOINT foo; SELECT handle_get_and_error(); ROLLBACK TO SAVEPOINT foo; SELECT handle_get_release(); ROLLBACK;),
+ qr/^|ok$/,
+ qr/ERROR.*as you command/);
+
+ $psql->quit();
+}
+
+
+sub test_batch
+{
+ my $io_method = shift;
+ my $node = shift;
+
+ my $psql = $node->background_psql('postgres', on_error_stop => 0);
+
+ # leak warning & recovery: implicit xact
+ psql_like(
+ $io_method,
+ $psql,
+ "batch_start() leak & cleanup in implicit xact",
+ qq(SELECT batch_start()),
+ qr/^$/,
+ qr/open AIO batch at end/,
+ "$io_method: leaky batch_start() warns");
+
+ # leak warning & recovery: explicit xact
+ psql_like(
+ $io_method,
+ $psql,
+ "batch_start() leak & cleanup in explicit xact",
+ qq(BEGIN; SELECT batch_start(); COMMIT;),
+ qr/^$/,
+ qr/open AIO batch at end/,
+ "$io_method: leaky batch_start() warns");
+
+
+ # leak warning & recovery: explicit xact, rollback
+ #
+ # XXX: This doesn't fail right now, due to not getting a chance to do
+ # something at transaction command commit. That's not a correctness issue,
+ # it just means it's a bit harder to find buggy code.
+ #psql_like($io_method, $psql,
+ # "batch_start() leak & cleanup after abort",
+ # qq(BEGIN; SELECT batch_start(); ROLLBACK;),
+ # qr/^$/,
+ # qr/open AIO batch at end/, "$io_method: leaky batch_start() warns");
+
+ # no warning, batch closed in same command
+ psql_like(
+ $io_method,
+ $psql,
+ "batch_start(), batch_end() works",
+ qq(SELECT batch_start() UNION ALL SELECT batch_end()),
+ qr/^$/,
+ qr/^$/,
+ "$io_method: batch_start(), batch_end()");
+
+ $psql->quit();
+}
+
+sub test_io_error
+{
+ my $io_method = shift;
+ my $node = shift;
+ my ($ret, $output);
+
+ my $psql = $node->background_psql('postgres', on_error_stop => 0);
+
+ # verify the error is reported in custom C code
+ ($output, $ret) = $psql->query(
+ qq(SELECT read_rel_block_ll('tbl_corr', 1, wait_complete=>true);));
+ is($ret, 1, "$io_method: read_rel_block_ll() of tbl_corr page fails");
+ like(
+ $psql->{stderr},
+ qr/invalid page in block 1 of relation base\/.*/,
+ "$io_method: read_rel_block_ll() of tbl_corr page reports error");
+ $psql->{stderr} = '';
+
+ # verify the error is reported for bufmgr reads
+ ($output, $ret) =
+ $psql->query(qq(SELECT count(*) FROM tbl_corr WHERE ctid = '(1, 1)'));
+ is($ret, 1, "$io_method: tid scan reading tbl_corr block fails");
+ like(
+ $psql->{stderr},
+ qr/invalid page in block 1 of relation base\/.*/,
+ "$io_method: tid scan reading tbl_corr block reports error");
+ $psql->{stderr} = '';
+
+ # verify the error is reported for bufmgr reads
+ ($output, $ret) =
+ $psql->query(qq(SELECT count(*) FROM tbl_corr WHERE ctid = '(1, 1)'));
+ is($ret, 1, "$io_method: sequential scan reading tbl_corr block fails");
+ like(
+ $psql->{stderr},
+ qr/invalid page in block 1 of relation base\/.*/,
+ "$io_method: sequential scan reading tbl_corr block reports error");
+ $psql->{stderr} = '';
+
+ $psql->quit();
+}
+
+# Test interplay between StartBufferIO and TerminateBufferIO
+sub test_startwait_io
+{
+ my $io_method = shift;
+ my $node = shift;
+ my ($ret, $output);
+
+ my $psql_a = $node->background_psql('postgres', on_error_stop => 0);
+ my $psql_b = $node->background_psql('postgres', on_error_stop => 0);
+
+ # create a buffer we can play around with
+ ($output, $ret) =
+ $psql_a->query(qq(SELECT buffer_create_toy('tbl_ok', 1);));
+ is($ret, 0, "$io_method: create toy succeeds");
+ like($output, qr/^\d+$/, "$io_method: create toy returns numeric");
+ my $buf_id = $output;
+
+ # check that one backend can perform StartBufferIO
+ psql_like(
+ $io_method,
+ $psql_a,
+ "first StartBufferIO",
+ qq(SELECT buffer_call_start_io($buf_id, for_input=>true, nowait=>false);),
+ qr/^t$/,
+ qr/^$/);
+
+ # but not twice on the same buffer (non-waiting)
+ psql_like(
+ $io_method,
+ $psql_a,
+ "second StartBufferIO fails, same session",
+ qq(SELECT buffer_call_start_io($buf_id, for_input=>true, nowait=>true);),
+ qr/^f$/,
+ qr/^$/);
+ psql_like(
+ $io_method,
+ $psql_b,
+ "second StartBufferIO fails, other session",
+ qq(SELECT buffer_call_start_io($buf_id, for_input=>true, nowait=>true);),
+ qr/^f$/,
+ qr/^$/);
+
+ # start io in a different session, will block
+ query_wait_block(
+ $io_method,
+ $node,
+ $psql_b,
+ "blocking start buffer io",
+ qq(SELECT buffer_call_start_io($buf_id, for_input=>true, nowait=>false);),
+ "BufferIo");
+
+ # Terminate the IO, without marking it as success, this should trigger the
+ # waiting session to be able to start the io
+ ($output, $ret) = $psql_a->query(
+ qq(SELECT buffer_call_terminate_io($buf_id, for_input=>true, succeed=>false, io_error=>false, syncio=>true);)
+ );
+ is($ret, 0,
+ "$io_method: blocking start buffer io, terminating io, not valid");
+
+ # Because the IO was terminated, but not marked as valid, second session should get the right to start io
+ pump_until($psql_b->{run}, $psql_b->{timeout}, \$psql_b->{stdout}, qr/t/);
+ ok(1, "$io_method: blocking start buffer io, can start io");
+
+ # terminate the IO again
+ $psql_b->query_safe(
+ qq(SELECT buffer_call_terminate_io($buf_id, for_input=>true, succeed=>false, io_error=>false, syncio=>true);)
+ );
+
+
+ # same as the above scenario, but mark IO as having succeeded
+ ($output, $ret) =
+ $psql_a->query(qq(SELECT buffer_call_start_io($buf_id, true, false);));
+ is($ret, 0,
+ "$io_method: blocking buffer io w/ success: first start buffer io succeeds"
+ );
+ is($output, "t",
+ "$io_method: blocking buffer io w/ success: first start buffer io returns true"
+ );
+
+
+ # start io in a different session, will block
+ query_wait_block(
+ $io_method,
+ $node,
+ $psql_b,
+ "blocking start buffer io",
+ qq(SELECT buffer_call_start_io($buf_id, for_input=>true, nowait=>false);),
+ "BufferIo");
+
+ ($output, $ret) = $psql_a->query(
+ qq(SELECT buffer_call_terminate_io($buf_id, for_input=>true, succeed=>true, io_error=>false, syncio=>true);)
+ );
+ is($ret, 0,
+ "$io_method: blocking start buffer io, terminating IO, valid");
+
+ # Because the IO was terminated, and marked as valid, second session should complete but not need io
+ pump_until($psql_b->{run}, $psql_b->{timeout}, \$psql_b->{stdout}, qr/f/);
+ ok(1, "$io_method: blocking start buffer io, no need to start io");
+
+
+ # buffer is valid now, make it invalid again
+ $buf_id = $psql_a->query_safe(qq(SELECT buffer_create_toy('tbl_ok', 1);));
+
+ $psql_a->quit();
+ $psql_b->quit();
+}
+
+# Test that if the backend issuing a read doesn't wait for the IO's
+# completion, another backend can complete the IO
+sub test_complete_foreign
+{
+ my $io_method = shift;
+ my $node = shift;
+ my ($ret, $output);
+
+ my $psql_a = $node->background_psql('postgres', on_error_stop => 0);
+ my $psql_b = $node->background_psql('postgres', on_error_stop => 0);
+
+ # Issue IO without waiting for completion, then sleep
+ $psql_a->query_safe(
+ qq(SELECT read_rel_block_ll('tbl_ok', 1, wait_complete=>false);));
+
+ # Check that another backend can read the relevant block
+ psql_like(
+ $io_method,
+ $psql_b,
+ "completing read started by sleeping backend",
+ qq(SELECT count(*) FROM tbl_ok WHERE ctid = '(1,1)' LIMIT 1),
+ qr/^1$/,
+ qr/^$/);
+
+ # Issue IO without waiting for completion, then exit
+ $psql_a->query_safe(
+ qq(SELECT read_rel_block_ll('tbl_ok', 1, wait_complete=>false);));
+ $psql_a->reconnect_and_clear();
+
+ # Check that another backend can read the relevant block
+ psql_like(
+ $io_method,
+ $psql_b,
+ "completing read started by exited backend",
+ qq(SELECT count(*) FROM tbl_ok WHERE ctid = '(1,1)' LIMIT 1),
+ qr/^1$/,
+ qr/^$/);
+
+ # Read a tbl_corr block, then sleep. The other session will retry the IO
+ # and also fail. The easiest thing to verify that seems to be to check
+ # that both are in the log.
+ my $log_location = -s $node->logfile;
+ $psql_a->query_safe(
+ qq(SELECT read_rel_block_ll('tbl_corr', 1, wait_complete=>false);));
+
+ psql_like(
+ $io_method,
+ $psql_b,
+ "completing read of tbl_corr block started by other backend",
+ qq(SELECT count(*) FROM tbl_corr WHERE ctid = '(1,1)' LIMIT 1),
+ qr/^$/,
+ qr/invalid page in block/);
+
+ # The log message issued for the read_rel_block_ll() should be logged as a LOG
+ $node->wait_for_log(qr/LOG[^\n]+invalid page in/, $log_location);
+ ok(1,
+ "$io_method: completing read of tbl_corr block started by other backend: LOG message for background read"
+ );
+
+ # But for the SELECT, it should be an ERROR
+ $log_location =
+ $node->wait_for_log(qr/ERROR[^\n]+invalid page in/, $log_location);
+ ok(1,
+ "$io_method: completing read of tbl_corr block started by other backend: ERROR message for foreground read"
+ );
+
+ $psql_a->quit();
+ $psql_b->quit();
+}
+
+sub test_inject
+{
+ my $io_method = shift;
+ my $node = shift;
+ my ($ret, $output);
+
+ my $psql = $node->background_psql('postgres', on_error_stop => 0);
+
+ # injected what we'd expect
+ $psql->query_safe(qq(SELECT inj_io_short_read_attach(8192);));
+ $psql->query_safe(qq(SELECT invalidate_rel_block('tbl_ok', 2);));
+ ($output, $ret) =
+ $psql->query(qq(SELECT count(*) FROM tbl_ok WHERE ctid = '(2, 1)';));
+ is($ret, 0,
+ "$io_method: injection point not triggering failure succeeds");
+
+ # injected a read shorter than a single block, expecting error
+ $psql->query_safe(qq(SELECT inj_io_short_read_attach(17);));
+ $psql->query_safe(qq(SELECT invalidate_rel_block('tbl_ok', 2);));
+ ($output, $ret) =
+ $psql->query(qq(SELECT count(*) FROM tbl_ok WHERE ctid = '(2, 1)';));
+ is($ret, 1, "$io_method: single block short read fails");
+ like(
+ $psql->{stderr},
+ qr/ERROR:.*could not read blocks 2\.\.2 in file "base\/.*": read only 0 of 8192 bytes/,
+ "$io_method: single block short read reports error");
+ $psql->{stderr} = '';
+
+ # shorten multi-block read to a single block, should retry
+ $psql->query_safe(
+ qq(
+SELECT invalidate_rel_block('tbl_ok', 0);
+SELECT invalidate_rel_block('tbl_ok', 1);
+SELECT invalidate_rel_block('tbl_ok', 2);
+SELECT inj_io_short_read_attach(8192);
+ ));
+ ($output, $ret) = $psql->query(qq(SELECT count(*) FROM tbl_ok;));
+ is($ret, 0, "$io_method: multi block short read is retried");
+
+ # verify that page verification errors are detected even as part of a
+ # shortened multi-block read (tbl_corr, block 1 is tbl_corred)
+ $psql->query_safe(
+ qq(
+SELECT invalidate_rel_block('tbl_corr', 0);
+SELECT invalidate_rel_block('tbl_corr', 1);
+SELECT invalidate_rel_block('tbl_corr', 2);
+SELECT inj_io_short_read_attach(8192);
+ ));
+ ($output, $ret) =
+ $psql->query(qq(SELECT count(*) FROM tbl_corr WHERE ctid < '(2, 1)'));
+ is($ret, 1,
+ "$io_method: shortened multi-block read detects invalid page");
+ like(
+ $psql->{stderr},
+ qr/ERROR:.*invalid page in block 1 of relation base\/.*/,
+ "$io_method: shortened multi-block reads reports invalid page");
+ $psql->{stderr} = '';
+
+ # trigger a hard error, should error out
+ $psql->query_safe(
+ qq(
+SELECT inj_io_short_read_attach(-errno_from_string('EIO'));
+SELECT invalidate_rel_block('tbl_ok', 2);
+ ));
+ ($output, $ret) =
+ $psql->query(qq(SELECT count(*) FROM tbl_ok; SELECT 1;));
+ is($ret, 1, "$io_method: hard IO error is detected");
+ like(
+ $psql->{stderr},
+ qr/ERROR:.*could not read blocks 2..3 in file \"base\/.*\": Input\/output error/,
+ "$io_method: hard IO error is reported");
+ $psql->{stderr} = '';
+
+ $psql->query_safe(
+ qq(
+SELECT inj_io_short_read_detach();
+ ));
+
+ $psql->quit();
+}
+
+sub test_inject_worker
+{
+ my $io_method = shift;
+ my $node = shift;
+ my ($ret, $output);
+
+ my $psql = $node->background_psql('postgres', on_error_stop => 0);
+
+ # trigger a failure to reopen, should error out, but should recover
+ $psql->query_safe(
+ qq(
+SELECT inj_io_reopen_attach();
+SELECT invalidate_rel_block('tbl_ok', 1);
+ ));
+ ($output, $ret) = $psql->query(qq(SELECT count(*) FROM tbl_ok;));
+ is($ret, 1, "$io_method: failure to open is detected");
+ like(
+ $psql->{stderr},
+ qr/ERROR:.*could not read blocks 1..2 in file "base\/.*": No such file or directory/,
+ "$io_method: failure to open is reported");
+ $psql->{stderr} = '';
+
+ $psql->query_safe(
+ qq(
+SELECT inj_io_reopen_detach();
+ ));
+
+ # check that we indeed recover
+ ($output, $ret) = $psql->query(qq(SELECT count(*) FROM tbl_ok;));
+ is($ret, 0, "$io_method: recovers from failure to open ");
+
+
+ $psql->quit();
+}
+
+sub run_generic_test
+{
+ my $io_method = shift;
+ my $node = shift;
+
+ is($node->safe_psql('postgres', 'SHOW io_method'),
+ $io_method, "$io_method: io_method set correctly");
+
+ $node->safe_psql(
+ 'postgres', qq(
+CREATE EXTENSION test_aio;
+CREATE TABLE tbl_corr(data int not null) WITH (AUTOVACUUM_ENABLED = false);
+CREATE TABLE tbl_ok(data int not null) WITH (AUTOVACUUM_ENABLED = false);
+
+INSERT INTO tbl_corr SELECT generate_series(1, 10000);
+INSERT INTO tbl_ok SELECT generate_series(1, 10000);
+SELECT grow_rel('tbl_corr', 16);
+SELECT grow_rel('tbl_ok', 16);
+
+SELECT corrupt_rel_block('tbl_corr', 1);
+CHECKPOINT;
+));
+
+ test_handle($io_method, $node);
+ test_io_error($io_method, $node);
+ test_batch($io_method, $node);
+ test_startwait_io($io_method, $node);
+ test_complete_foreign($io_method, $node);
+
+ SKIP:
+ {
+ skip 'Injection points not supported by this build', 1
+ unless $ENV{enable_injection_points} eq 'yes';
+ test_inject($io_method, $node);
+ }
+}
diff --git a/src/test/modules/test_aio/test_aio--1.0.sql b/src/test/modules/test_aio/test_aio--1.0.sql
new file mode 100644
index 00000000000..3d4e9bb070f
--- /dev/null
+++ b/src/test/modules/test_aio/test_aio--1.0.sql
@@ -0,0 +1,97 @@
+/* src/test/modules/test_aio/test_aio--1.0.sql */
+
+-- complain if script is sourced in psql, rather than via CREATE EXTENSION
+\echo Use "CREATE EXTENSION test_aio" to load this file. \quit
+
+
+CREATE FUNCTION errno_from_string(sym text)
+RETURNS pg_catalog.int4 STRICT
+AS 'MODULE_PATHNAME' LANGUAGE C;
+
+
+CREATE FUNCTION grow_rel(rel regclass, nblocks int)
+RETURNS pg_catalog.void STRICT
+AS 'MODULE_PATHNAME' LANGUAGE C;
+
+
+CREATE FUNCTION corrupt_rel_block(rel regclass, blockno int)
+RETURNS pg_catalog.void STRICT
+AS 'MODULE_PATHNAME' LANGUAGE C;
+
+CREATE FUNCTION read_rel_block_ll(rel regclass, blockno int, wait_complete bool)
+RETURNS pg_catalog.void STRICT
+AS 'MODULE_PATHNAME' LANGUAGE C;
+
+CREATE FUNCTION invalidate_rel_block(rel regclass, blockno int)
+RETURNS pg_catalog.void STRICT
+AS 'MODULE_PATHNAME' LANGUAGE C;
+
+CREATE FUNCTION buffer_create_toy(rel regclass, blockno int4)
+RETURNS pg_catalog.int4 STRICT
+AS 'MODULE_PATHNAME' LANGUAGE C;
+
+CREATE FUNCTION buffer_call_start_io(buffer int, for_input bool, nowait bool)
+RETURNS pg_catalog.bool STRICT
+AS 'MODULE_PATHNAME' LANGUAGE C;
+
+CREATE FUNCTION buffer_call_terminate_io(buffer int, for_input bool, succeed bool, io_error bool, syncio bool)
+RETURNS pg_catalog.void STRICT
+AS 'MODULE_PATHNAME' LANGUAGE C;
+
+
+
+/*
+ * Handle related functions
+ */
+CREATE FUNCTION handle_get_and_error()
+RETURNS pg_catalog.void STRICT
+AS 'MODULE_PATHNAME' LANGUAGE C;
+
+CREATE FUNCTION handle_get_twice()
+RETURNS pg_catalog.void STRICT
+AS 'MODULE_PATHNAME' LANGUAGE C;
+
+CREATE FUNCTION handle_get()
+RETURNS pg_catalog.void STRICT
+AS 'MODULE_PATHNAME' LANGUAGE C;
+
+CREATE FUNCTION handle_get_release()
+RETURNS pg_catalog.void STRICT
+AS 'MODULE_PATHNAME' LANGUAGE C;
+
+CREATE FUNCTION handle_release_last()
+RETURNS pg_catalog.void STRICT
+AS 'MODULE_PATHNAME' LANGUAGE C;
+
+
+/*
+ * Batchmode related functions
+ */
+CREATE FUNCTION batch_start()
+RETURNS pg_catalog.void STRICT
+AS 'MODULE_PATHNAME' LANGUAGE C;
+
+CREATE FUNCTION batch_end()
+RETURNS pg_catalog.void STRICT
+AS 'MODULE_PATHNAME' LANGUAGE C;
+
+
+
+/*
+ * Injection point related functions
+ */
+CREATE FUNCTION inj_io_short_read_attach(result int)
+RETURNS pg_catalog.void STRICT
+AS 'MODULE_PATHNAME' LANGUAGE C;
+
+CREATE FUNCTION inj_io_short_read_detach()
+RETURNS pg_catalog.void STRICT
+AS 'MODULE_PATHNAME' LANGUAGE C;
+
+CREATE FUNCTION inj_io_reopen_attach()
+RETURNS pg_catalog.void STRICT
+AS 'MODULE_PATHNAME' LANGUAGE C;
+
+CREATE FUNCTION inj_io_reopen_detach()
+RETURNS pg_catalog.void STRICT
+AS 'MODULE_PATHNAME' LANGUAGE C;
diff --git a/src/test/modules/test_aio/test_aio.c b/src/test/modules/test_aio/test_aio.c
new file mode 100644
index 00000000000..81b4d732206
--- /dev/null
+++ b/src/test/modules/test_aio/test_aio.c
@@ -0,0 +1,616 @@
+/*-------------------------------------------------------------------------
+ *
+ * delay_execution.c
+ * Test module to allow delay between parsing and execution of a query.
+ *
+ * The delay is implemented by taking and immediately releasing a specified
+ * advisory lock. If another process has previously taken that lock, the
+ * current process will be blocked until the lock is released; otherwise,
+ * there's no effect. This allows an isolationtester script to reliably
+ * test behaviors where some specified action happens in another backend
+ * between parsing and execution of any desired query.
+ *
+ * Copyright (c) 2020-2025, PostgreSQL Global Development Group
+ *
+ * IDENTIFICATION
+ * src/test/modules/delay_execution/delay_execution.c
+ *
+ *-------------------------------------------------------------------------
+ */
+
+#include "postgres.h"
+
+#include "access/relation.h"
+#include "fmgr.h"
+#include "storage/aio.h"
+#include "storage/aio_internal.h"
+#include "storage/buf_internals.h"
+#include "storage/bufmgr.h"
+#include "storage/ipc.h"
+#include "storage/lwlock.h"
+#include "utils/builtins.h"
+#include "utils/injection_point.h"
+#include "utils/rel.h"
+
+
+PG_MODULE_MAGIC;
+
+
+typedef struct InjIoErrorState
+{
+ bool enabled_short_read;
+ bool enabled_reopen;
+
+ bool short_read_result_set;
+ int short_read_result;
+} InjIoErrorState;
+
+static InjIoErrorState * inj_io_error_state;
+
+/* Shared memory init callbacks */
+static shmem_request_hook_type prev_shmem_request_hook = NULL;
+static shmem_startup_hook_type prev_shmem_startup_hook = NULL;
+
+
+static PgAioHandle *last_handle;
+
+
+
+static void
+test_aio_shmem_request(void)
+{
+ if (prev_shmem_request_hook)
+ prev_shmem_request_hook();
+
+ RequestAddinShmemSpace(sizeof(InjIoErrorState));
+}
+
+static void
+test_aio_shmem_startup(void)
+{
+ bool found;
+
+ if (prev_shmem_startup_hook)
+ prev_shmem_startup_hook();
+
+ /* Create or attach to the shared memory state */
+ LWLockAcquire(AddinShmemInitLock, LW_EXCLUSIVE);
+
+ inj_io_error_state = ShmemInitStruct("injection_points",
+ sizeof(InjIoErrorState),
+ &found);
+
+ if (!found)
+ {
+ /*
+ * First time through, so initialize. This is shared with the dynamic
+ * initialization using a DSM.
+ */
+ inj_io_error_state->enabled_short_read = false;
+ inj_io_error_state->enabled_reopen = false;
+
+#ifdef USE_INJECTION_POINTS
+ InjectionPointAttach("AIO_PROCESS_COMPLETION_BEFORE_SHARED",
+ "test_aio",
+ "inj_io_short_read",
+ NULL,
+ 0);
+ InjectionPointLoad("AIO_PROCESS_COMPLETION_BEFORE_SHARED");
+
+ InjectionPointAttach("AIO_WORKER_AFTER_REOPEN",
+ "test_aio",
+ "inj_io_reopen",
+ NULL,
+ 0);
+ InjectionPointLoad("AIO_WORKER_AFTER_REOPEN");
+
+#endif
+ }
+ else
+ {
+#ifdef USE_INJECTION_POINTS
+ InjectionPointLoad("AIO_PROCESS_COMPLETION_BEFORE_SHARED");
+ InjectionPointLoad("AIO_WORKER_AFTER_REOPEN");
+ elog(LOG, "injection point loaded");
+#endif
+ }
+
+ LWLockRelease(AddinShmemInitLock);
+}
+
+void
+_PG_init(void)
+{
+ if (!process_shared_preload_libraries_in_progress)
+ return;
+
+ /* Shared memory initialization */
+ prev_shmem_request_hook = shmem_request_hook;
+ shmem_request_hook = test_aio_shmem_request;
+ prev_shmem_startup_hook = shmem_startup_hook;
+ shmem_startup_hook = test_aio_shmem_startup;
+}
+
+
+PG_FUNCTION_INFO_V1(errno_from_string);
+Datum
+errno_from_string(PG_FUNCTION_ARGS)
+{
+ const char *sym = text_to_cstring(PG_GETARG_TEXT_PP(0));
+
+ if (strcmp(sym, "EIO") == 0)
+ PG_RETURN_INT32(EIO);
+ else if (strcmp(sym, "EAGAIN") == 0)
+ PG_RETURN_INT32(EAGAIN);
+ else if (strcmp(sym, "EINTR") == 0)
+ PG_RETURN_INT32(EINTR);
+ else if (strcmp(sym, "ENOSPC") == 0)
+ PG_RETURN_INT32(ENOSPC);
+ else if (strcmp(sym, "EROFS") == 0)
+ PG_RETURN_INT32(EROFS);
+
+ ereport(ERROR,
+ errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ errmsg_internal("%s is not a supported errno value", sym));
+ PG_RETURN_INT32(0);
+}
+
+
+PG_FUNCTION_INFO_V1(grow_rel);
+Datum
+grow_rel(PG_FUNCTION_ARGS)
+{
+ Oid relid = PG_GETARG_OID(0);
+ uint32 nblocks = PG_GETARG_UINT32(1);
+ Relation rel;
+#define MAX_BUFFERS_TO_EXTEND_BY 64
+ Buffer victim_buffers[MAX_BUFFERS_TO_EXTEND_BY];
+
+ rel = relation_open(relid, AccessExclusiveLock);
+
+ while (nblocks > 0)
+ {
+ uint32 extend_by_pages;
+
+ extend_by_pages = Min(nblocks, MAX_BUFFERS_TO_EXTEND_BY);
+
+ ExtendBufferedRelBy(BMR_REL(rel),
+ MAIN_FORKNUM,
+ NULL,
+ 0,
+ extend_by_pages,
+ victim_buffers,
+ &extend_by_pages);
+
+ nblocks -= extend_by_pages;
+
+ for (uint32 i = 0; i < extend_by_pages; i++)
+ {
+ ReleaseBuffer(victim_buffers[i]);
+ }
+ }
+
+ relation_close(rel, NoLock);
+
+ PG_RETURN_VOID();
+}
+
+PG_FUNCTION_INFO_V1(corrupt_rel_block);
+Datum
+corrupt_rel_block(PG_FUNCTION_ARGS)
+{
+ Oid relid = PG_GETARG_OID(0);
+ BlockNumber blkno = PG_GETARG_UINT32(1);
+ Relation rel;
+ Buffer buf;
+ Page page;
+ PageHeader ph;
+
+ rel = relation_open(relid, AccessExclusiveLock);
+
+ buf = ReadBuffer(rel, blkno);
+ page = BufferGetPage(buf);
+
+ LockBuffer(buf, BUFFER_LOCK_EXCLUSIVE);
+
+ MarkBufferDirty(buf);
+
+ PageInit(page, BufferGetPageSize(buf), 0);
+
+ ph = (PageHeader) page;
+ ph->pd_special = BLCKSZ + 1;
+
+ FlushOneBuffer(buf);
+
+ LockBuffer(buf, BUFFER_LOCK_UNLOCK);
+
+ ReleaseBuffer(buf);
+
+ EvictUnpinnedBuffer(buf);
+
+ relation_close(rel, NoLock);
+
+ PG_RETURN_VOID();
+}
+
+/*
+ * Ensures a buffer for rel & blkno is in shared buffers, without actually
+ * caring about the buffer contents. Used to set up test scenarios.
+ */
+static Buffer
+create_toy_buffer(Relation rel, BlockNumber blkno)
+{
+ Buffer buf;
+ BufferDesc *buf_hdr;
+ uint32 buf_state;
+ bool was_pinned = false;
+
+ /* read buffer without erroring out */
+ buf = ReadBufferExtended(rel, MAIN_FORKNUM, blkno, RBM_ZERO_ON_ERROR, NULL);
+
+ buf_hdr = GetBufferDescriptor(buf - 1);
+
+ buf_state = LockBufHdr(buf_hdr);
+
+ /*
+ * We should be the only backend accessing this buffer. This is just a
+ * small bit of belt-and-suspenders defense, none of this code should ever
+ * run in a cluster with real data.
+ */
+ if (BUF_STATE_GET_REFCOUNT(buf_state) > 1)
+ was_pinned = true;
+ else
+ buf_state &= ~(BM_VALID | BM_DIRTY);
+
+ UnlockBufHdr(buf_hdr, buf_state);
+
+ if (was_pinned)
+ elog(ERROR, "toy buffer %d was already pinned",
+ buf);
+
+ return buf;
+}
+
+PG_FUNCTION_INFO_V1(read_rel_block_ll);
+Datum
+read_rel_block_ll(PG_FUNCTION_ARGS)
+{
+ Oid relid = PG_GETARG_OID(0);
+ BlockNumber blkno = PG_GETARG_UINT32(1);
+ bool wait_complete = PG_GETARG_BOOL(2);
+ Relation rel;
+ Buffer buf;
+ Page pages[1];
+ PgAioReturn ior;
+ PgAioHandle *ioh;
+ PgAioWaitRef iow;
+ SMgrRelation smgr;
+
+ rel = relation_open(relid, AccessExclusiveLock);
+
+ buf = create_toy_buffer(rel, blkno);
+
+ pages[0] = BufferGetBlock(buf);
+
+ ioh = pgaio_io_acquire(CurrentResourceOwner, &ior);
+ pgaio_io_get_wref(ioh, &iow);
+
+ smgr = RelationGetSmgr(rel);
+
+ StartBufferIO(GetBufferDescriptor(buf - 1), true, false);
+
+ pgaio_io_set_handle_data_32(ioh, (uint32 *) &buf, 1);
+ pgaio_io_register_callbacks(ioh, PGAIO_HCB_SHARED_BUFFER_READV, 0);
+
+ elog(LOG, "about to smgrstartreadv");
+ smgrstartreadv(ioh, smgr, MAIN_FORKNUM, blkno,
+ (void *) pages, 1);
+
+ ReleaseBuffer(buf);
+
+ if (wait_complete)
+ {
+ pgaio_wref_wait(&iow);
+
+ if (ior.result.status != ARS_OK)
+ pgaio_result_report(ior.result, &ior.target_data,
+ ior.result.status == ARS_PARTIAL ? WARNING : ERROR);
+ }
+
+ relation_close(rel, NoLock);
+
+ PG_RETURN_VOID();
+}
+
+PG_FUNCTION_INFO_V1(invalidate_rel_block);
+Datum
+invalidate_rel_block(PG_FUNCTION_ARGS)
+{
+ Oid relid = PG_GETARG_OID(0);
+ BlockNumber blkno = PG_GETARG_UINT32(1);
+ Relation rel;
+ PrefetchBufferResult pr;
+ Buffer buf;
+
+ rel = relation_open(relid, AccessExclusiveLock);
+
+ /*
+ * This is a gross hack, but there's no other API exposed that allows to
+ * get a buffer ID without actually reading the block in.
+ */
+ pr = PrefetchBuffer(rel, MAIN_FORKNUM, blkno);
+ buf = pr.recent_buffer;
+
+ if (BufferIsValid(buf))
+ {
+ /* if the buffer contents aren't valid, this'll return false */
+ if (ReadRecentBuffer(rel->rd_locator, MAIN_FORKNUM, blkno, buf))
+ {
+ LockBuffer(buf, BUFFER_LOCK_EXCLUSIVE);
+ FlushOneBuffer(buf);
+ LockBuffer(buf, BUFFER_LOCK_UNLOCK);
+ ReleaseBuffer(buf);
+
+ if (!EvictUnpinnedBuffer(buf))
+ elog(ERROR, "couldn't evict");
+ }
+ }
+
+ relation_close(rel, AccessExclusiveLock);
+
+ PG_RETURN_VOID();
+}
+
+PG_FUNCTION_INFO_V1(buffer_create_toy);
+Datum
+buffer_create_toy(PG_FUNCTION_ARGS)
+{
+ Oid relid = PG_GETARG_OID(0);
+ BlockNumber blkno = PG_GETARG_UINT32(1);
+ Relation rel;
+ Buffer buf;
+
+ rel = relation_open(relid, AccessExclusiveLock);
+
+ buf = create_toy_buffer(rel, blkno);
+ ReleaseBuffer(buf);
+
+ relation_close(rel, NoLock);
+
+ PG_RETURN_INT32(buf);
+}
+
+PG_FUNCTION_INFO_V1(buffer_call_start_io);
+Datum
+buffer_call_start_io(PG_FUNCTION_ARGS)
+{
+ Buffer buf = PG_GETARG_INT32(0);
+ bool for_input = PG_GETARG_BOOL(1);
+ bool nowait = PG_GETARG_BOOL(2);
+ bool can_start;
+
+ can_start = StartBufferIO(GetBufferDescriptor(buf - 1), for_input, nowait);
+
+ /*
+ * Tor tests we don't want the resowner release preventing us from
+ * orchestrating odd scenarios.
+ */
+ if (can_start)
+ ResourceOwnerForgetBufferIO(CurrentResourceOwner,
+ buf);
+
+ ereport(LOG,
+ errmsg("buffer %d after StartBufferIO: %s",
+ buf, DebugPrintBufferRefcount(buf)),
+ errhidestmt(true), errhidecontext(true));
+
+ PG_RETURN_BOOL(can_start);
+}
+
+/*
+CREATE FUNCTION buffer_call_terminate_io(buffer int, for_input bool, failed bool, syncio bool)
+RETURNS pg_catalog.void STRICT
+AS 'MODULE_PATHNAME' LANGUAGE C;
+*/
+PG_FUNCTION_INFO_V1(buffer_call_terminate_io);
+Datum
+buffer_call_terminate_io(PG_FUNCTION_ARGS)
+{
+ Buffer buf = PG_GETARG_INT32(0);
+ bool for_input = PG_GETARG_BOOL(1);
+ bool succeed = PG_GETARG_BOOL(2);
+ bool io_error = PG_GETARG_BOOL(3);
+ bool syncio = PG_GETARG_BOOL(4);
+ bool clear_dirty = false;
+ uint32 set_flag_bits = 0;
+
+ if (io_error)
+ set_flag_bits |= BM_IO_ERROR;
+
+ if (for_input)
+ {
+ clear_dirty = false;
+
+ if (succeed)
+ set_flag_bits |= BM_VALID;
+ }
+ else
+ {
+ if (succeed)
+ clear_dirty = true;
+ }
+
+ ereport(LOG,
+ errmsg("buffer %d before TerminateBufferIO: %s",
+ buf, DebugPrintBufferRefcount(buf)),
+ errhidestmt(true), errhidecontext(true));
+
+ TerminateBufferIO(GetBufferDescriptor(buf - 1), clear_dirty, set_flag_bits,
+ false, syncio);
+
+ ereport(LOG,
+ errmsg("buffer %d after TerminateBufferIO: %s",
+ buf, DebugPrintBufferRefcount(buf)),
+ errhidestmt(true), errhidecontext(true));
+
+ PG_RETURN_VOID();
+}
+
+PG_FUNCTION_INFO_V1(handle_get);
+Datum
+handle_get(PG_FUNCTION_ARGS)
+{
+ last_handle = pgaio_io_acquire(CurrentResourceOwner, NULL);
+
+ PG_RETURN_VOID();
+}
+
+PG_FUNCTION_INFO_V1(handle_release_last);
+Datum
+handle_release_last(PG_FUNCTION_ARGS)
+{
+ if (!last_handle)
+ elog(ERROR, "no handle");
+
+ pgaio_io_release(last_handle);
+
+ PG_RETURN_VOID();
+}
+
+PG_FUNCTION_INFO_V1(handle_get_and_error);
+Datum
+handle_get_and_error(PG_FUNCTION_ARGS)
+{
+ pgaio_io_acquire(CurrentResourceOwner, NULL);
+
+ elog(ERROR, "as you command");
+ PG_RETURN_VOID();
+}
+
+PG_FUNCTION_INFO_V1(handle_get_twice);
+Datum
+handle_get_twice(PG_FUNCTION_ARGS)
+{
+ pgaio_io_acquire(CurrentResourceOwner, NULL);
+ pgaio_io_acquire(CurrentResourceOwner, NULL);
+
+ PG_RETURN_VOID();
+}
+
+PG_FUNCTION_INFO_V1(handle_get_release);
+Datum
+handle_get_release(PG_FUNCTION_ARGS)
+{
+ PgAioHandle *handle;
+
+ handle = pgaio_io_acquire(CurrentResourceOwner, NULL);
+ pgaio_io_release(handle);
+
+ PG_RETURN_VOID();
+}
+
+PG_FUNCTION_INFO_V1(batch_start);
+Datum
+batch_start(PG_FUNCTION_ARGS)
+{
+ pgaio_enter_batchmode();
+ PG_RETURN_VOID();
+}
+
+PG_FUNCTION_INFO_V1(batch_end);
+Datum
+batch_end(PG_FUNCTION_ARGS)
+{
+ pgaio_exit_batchmode();
+ PG_RETURN_VOID();
+}
+
+#ifdef USE_INJECTION_POINTS
+extern PGDLLEXPORT void inj_io_short_read(const char *name, const void *private_data);
+extern PGDLLEXPORT void inj_io_reopen(const char *name, const void *private_data);
+
+void
+inj_io_short_read(const char *name, const void *private_data)
+{
+ PgAioHandle *ioh;
+
+ elog(LOG, "short read called: %d", inj_io_error_state->enabled_short_read);
+
+ if (inj_io_error_state->enabled_short_read)
+ {
+ ioh = pgaio_inj_io_get();
+
+ if (inj_io_error_state->short_read_result_set)
+ {
+ elog(LOG, "short read, changing result from %d to %d",
+ ioh->result, inj_io_error_state->short_read_result);
+
+ ioh->result = inj_io_error_state->short_read_result;
+ }
+ }
+}
+
+void
+inj_io_reopen(const char *name, const void *private_data)
+{
+ elog(LOG, "reopen called: %d", inj_io_error_state->enabled_reopen);
+
+ if (inj_io_error_state->enabled_reopen)
+ {
+ elog(ERROR, "injection point triggering failure to reopen ");
+ }
+}
+#endif
+
+PG_FUNCTION_INFO_V1(inj_io_short_read_attach);
+Datum
+inj_io_short_read_attach(PG_FUNCTION_ARGS)
+{
+#ifdef USE_INJECTION_POINTS
+ inj_io_error_state->enabled_short_read = true;
+ inj_io_error_state->short_read_result_set = !PG_ARGISNULL(0);
+ if (inj_io_error_state->short_read_result_set)
+ inj_io_error_state->short_read_result = PG_GETARG_INT32(0);
+#else
+ elog(ERROR, "injection points not supported");
+#endif
+
+ PG_RETURN_VOID();
+}
+
+PG_FUNCTION_INFO_V1(inj_io_short_read_detach);
+Datum
+inj_io_short_read_detach(PG_FUNCTION_ARGS)
+{
+#ifdef USE_INJECTION_POINTS
+ inj_io_error_state->enabled_short_read = false;
+#else
+ elog(ERROR, "injection points not supported");
+#endif
+ PG_RETURN_VOID();
+}
+
+PG_FUNCTION_INFO_V1(inj_io_reopen_attach);
+Datum
+inj_io_reopen_attach(PG_FUNCTION_ARGS)
+{
+#ifdef USE_INJECTION_POINTS
+ inj_io_error_state->enabled_reopen = true;
+#else
+ elog(ERROR, "injection points not supported");
+#endif
+
+ PG_RETURN_VOID();
+}
+
+PG_FUNCTION_INFO_V1(inj_io_reopen_detach);
+Datum
+inj_io_reopen_detach(PG_FUNCTION_ARGS)
+{
+#ifdef USE_INJECTION_POINTS
+ inj_io_error_state->enabled_reopen = false;
+#else
+ elog(ERROR, "injection points not supported");
+#endif
+ PG_RETURN_VOID();
+}
diff --git a/src/test/modules/test_aio/test_aio.control b/src/test/modules/test_aio/test_aio.control
new file mode 100644
index 00000000000..cd91c3ed16b
--- /dev/null
+++ b/src/test/modules/test_aio/test_aio.control
@@ -0,0 +1,3 @@
+comment = 'Test code for AIO'
+default_version = '1.0'
+module_pathname = '$libdir/test_aio'
--
2.48.1.76.g4e746b1a31.dirty
[text/x-diff] v2.5-0021-wip-ci-Increase-openbsd-kern.maxfiles-to-fix-co.patch (964B, ../../vz5i2x2wkjjpp6l2z4g4l3umpxktrxsejgdwefi7n6kzw666h3@rnhlvlpadig7/22-v2.5-0021-wip-ci-Increase-openbsd-kern.maxfiles-to-fix-co.patch)
download | inline diff:
From 1aaeed590e5b972072f24ac1de9f22936d7b02af Mon Sep 17 00:00:00 2001
From: Andres Freund <andres@anarazel.de>
Date: Tue, 4 Mar 2025 11:10:21 -0500
Subject: [PATCH v2.5 21/30] wip: ci: Increase openbsd kern.maxfiles to fix
concurrent test issues
Author:
Reviewed-by:
Discussion: https://postgr.es/m/
Backpatch:
---
.cirrus.tasks.yml | 4 ++++
1 file changed, 4 insertions(+)
diff --git a/.cirrus.tasks.yml b/.cirrus.tasks.yml
index ba4bfa43a50..96d2b18cee0 100644
--- a/.cirrus.tasks.yml
+++ b/.cirrus.tasks.yml
@@ -262,6 +262,10 @@ task:
TCL: -Dtcl_version=tcl86
setup_additional_packages_script: |
#pkg_add -I ...
+ sysctl_script: |
+ sysctl -a|grep ^kern.max
+ # otherwise parallel tests run into limits
+ sysctl kern.maxfiles=10000
# Always core dump to ${CORE_DUMP_DIR}
set_core_dump_script: sysctl -w kern.nosuidcoredump=2
<<: *openbsd_task_template
--
2.48.1.76.g4e746b1a31.dirty
[text/x-diff] v2.5-0022-aio-Implement-smgr-md-fd-write-support.patch (12.8K, ../../vz5i2x2wkjjpp6l2z4g4l3umpxktrxsejgdwefi7n6kzw666h3@rnhlvlpadig7/23-v2.5-0022-aio-Implement-smgr-md-fd-write-support.patch)
download | inline diff:
From d08717aa72457821a842271fd63579b702056531 Mon Sep 17 00:00:00 2001
From: Andres Freund <andres@anarazel.de>
Date: Wed, 26 Feb 2025 11:26:41 -0500
Subject: [PATCH v2.5 22/30] aio: Implement smgr/md/fd write support
TODO:
- register_dirty_segment_aio() can error out in edge cases
---
src/include/storage/fd.h | 2 +
src/include/storage/md.h | 5 +
src/include/storage/smgr.h | 5 +
src/backend/storage/aio/aio_callback.c | 1 +
src/backend/storage/file/fd.c | 28 ++++
src/backend/storage/smgr/md.c | 189 +++++++++++++++++++++++++
src/backend/storage/smgr/smgr.c | 22 +++
7 files changed, 252 insertions(+)
diff --git a/src/include/storage/fd.h b/src/include/storage/fd.h
index f44f3b939d6..e2fd896646e 100644
--- a/src/include/storage/fd.h
+++ b/src/include/storage/fd.h
@@ -113,6 +113,8 @@ extern ssize_t FileReadV(File file, const struct iovec *iov, int iovcnt, off_t o
extern ssize_t FileWriteV(File file, const struct iovec *iov, int iovcnt, off_t offset, uint32 wait_event_info);
extern ssize_t FileReadV(File file, const struct iovec *iov, int iovcnt, off_t offset, uint32 wait_event_info);
extern int FileStartReadV(struct PgAioHandle *ioh, File file, int iovcnt, off_t offset, uint32 wait_event_info);
+extern ssize_t FileWriteV(File file, const struct iovec *iov, int iovcnt, off_t offset, uint32 wait_event_info);
+extern int FileStartWriteV(struct PgAioHandle *ioh, File file, int iovcnt, off_t offset, uint32 wait_event_info);
extern int FileSync(File file, uint32 wait_event_info);
extern int FileZero(File file, off_t offset, off_t amount, uint32 wait_event_info);
extern int FileFallocate(File file, off_t offset, off_t amount, uint32 wait_event_info);
diff --git a/src/include/storage/md.h b/src/include/storage/md.h
index bf714a8896d..7b28c3d482c 100644
--- a/src/include/storage/md.h
+++ b/src/include/storage/md.h
@@ -21,6 +21,7 @@
struct PgAioHandleCallbacks;
extern const struct PgAioHandleCallbacks aio_md_readv_cb;
+extern const struct PgAioHandleCallbacks aio_md_writev_cb;
/* md storage manager functionality */
extern void mdinit(void);
@@ -45,6 +46,10 @@ extern void mdstartreadv(struct PgAioHandle *ioh,
extern void mdwritev(SMgrRelation reln, ForkNumber forknum,
BlockNumber blocknum,
const void **buffers, BlockNumber nblocks, bool skipFsync);
+extern void mdstartwritev(struct PgAioHandle *ioh,
+ SMgrRelation reln, ForkNumber forknum,
+ BlockNumber blocknum,
+ const void **buffers, BlockNumber nblocks, bool skipFsync);
extern void mdwriteback(SMgrRelation reln, ForkNumber forknum,
BlockNumber blocknum, BlockNumber nblocks);
extern BlockNumber mdnblocks(SMgrRelation reln, ForkNumber forknum);
diff --git a/src/include/storage/smgr.h b/src/include/storage/smgr.h
index 85562fa0cc0..86fa07b110f 100644
--- a/src/include/storage/smgr.h
+++ b/src/include/storage/smgr.h
@@ -110,6 +110,11 @@ extern void smgrwritev(SMgrRelation reln, ForkNumber forknum,
BlockNumber blocknum,
const void **buffers, BlockNumber nblocks,
bool skipFsync);
+extern void smgrstartwritev(struct PgAioHandle *ioh,
+ SMgrRelation reln, ForkNumber forknum,
+ BlockNumber blocknum,
+ const void **buffers, BlockNumber nblocks,
+ bool skipFsync);
extern void smgrwriteback(SMgrRelation reln, ForkNumber forknum,
BlockNumber blocknum, BlockNumber nblocks);
extern BlockNumber smgrnblocks(SMgrRelation reln, ForkNumber forknum);
diff --git a/src/backend/storage/aio/aio_callback.c b/src/backend/storage/aio/aio_callback.c
index ff7b2f25c99..f0414085eba 100644
--- a/src/backend/storage/aio/aio_callback.c
+++ b/src/backend/storage/aio/aio_callback.c
@@ -41,6 +41,7 @@ static const PgAioHandleCallbacksEntry aio_handle_cbs[] = {
CALLBACK_ENTRY(PGAIO_HCB_INVALID, aio_invalid_cb),
CALLBACK_ENTRY(PGAIO_HCB_MD_READV, aio_md_readv_cb),
+ CALLBACK_ENTRY(PGAIO_HCB_MD_WRITEV, aio_md_writev_cb),
CALLBACK_ENTRY(PGAIO_HCB_SHARED_BUFFER_READV, aio_shared_buffer_readv_cb),
diff --git a/src/backend/storage/file/fd.c b/src/backend/storage/file/fd.c
index 996458fac71..a9c90bc4e59 100644
--- a/src/backend/storage/file/fd.c
+++ b/src/backend/storage/file/fd.c
@@ -2346,6 +2346,34 @@ retry:
return returnCode;
}
+int
+FileStartWriteV(struct PgAioHandle *ioh, File file,
+ int iovcnt, off_t offset,
+ uint32 wait_event_info)
+{
+ int returnCode;
+ Vfd *vfdP;
+
+ Assert(FileIsValid(file));
+
+ DO_DB(elog(LOG, "FileStartWriteV: %d (%s) " INT64_FORMAT " %d",
+ file, VfdCache[file].fileName,
+ (int64) offset,
+ iovcnt));
+
+ returnCode = FileAccess(file);
+ if (returnCode < 0)
+ return returnCode;
+
+ vfdP = &VfdCache[file];
+
+ /* FIXME: think about / reimplement temp_file_limit */
+
+ pgaio_io_prep_writev(ioh, vfdP->fd, iovcnt, offset);
+
+ return 0;
+}
+
int
FileSync(File file, uint32 wait_event_info)
{
diff --git a/src/backend/storage/smgr/md.c b/src/backend/storage/smgr/md.c
index d01ae5f6a09..4c0e5371875 100644
--- a/src/backend/storage/smgr/md.c
+++ b/src/backend/storage/smgr/md.c
@@ -155,12 +155,19 @@ static BlockNumber _mdnblocks(SMgrRelation reln, ForkNumber forknum,
static PgAioResult md_readv_complete(PgAioHandle *ioh, PgAioResult prior_result, uint8 cb_data);
static void md_readv_report(PgAioResult result, const PgAioTargetData *target_data, int elevel);
+static PgAioResult md_writev_complete(PgAioHandle *ioh, PgAioResult prior_result, uint8 cb_data);
+static void md_writev_report(PgAioResult result, const PgAioTargetData *target_data, int elevel);
const struct PgAioHandleCallbacks aio_md_readv_cb = {
.complete_shared = md_readv_complete,
.report = md_readv_report,
};
+const struct PgAioHandleCallbacks aio_md_writev_cb = {
+ .complete_shared = md_writev_complete,
+ .report = md_writev_report,
+};
+
static inline int
_mdfd_open_flags(void)
@@ -1107,6 +1114,56 @@ mdwritev(SMgrRelation reln, ForkNumber forknum, BlockNumber blocknum,
}
}
+/*
+ * mdstartwritev() -- Asynchronous version of mdrwritev().
+ */
+void
+mdstartwritev(PgAioHandle *ioh,
+ SMgrRelation reln, ForkNumber forknum, BlockNumber blocknum,
+ const void **buffers, BlockNumber nblocks, bool skipFsync)
+{
+ off_t seekpos;
+ MdfdVec *v;
+ BlockNumber nblocks_this_segment;
+ struct iovec *iov;
+ int iovcnt;
+
+ v = _mdfd_getseg(reln, forknum, blocknum, false,
+ EXTENSION_FAIL | EXTENSION_CREATE_RECOVERY);
+
+ seekpos = (off_t) BLCKSZ * (blocknum % ((BlockNumber) RELSEG_SIZE));
+
+ Assert(seekpos < (off_t) BLCKSZ * RELSEG_SIZE);
+
+ nblocks_this_segment =
+ Min(nblocks,
+ RELSEG_SIZE - (blocknum % ((BlockNumber) RELSEG_SIZE)));
+
+ if (nblocks_this_segment != nblocks)
+ elog(ERROR, "write crossing segment boundary");
+
+ iovcnt = pgaio_io_get_iovec(ioh, &iov);
+
+ Assert(nblocks <= iovcnt);
+
+ iovcnt = buffers_to_iovec(iov, unconstify(void **, buffers), nblocks_this_segment);
+
+ Assert(iovcnt <= nblocks_this_segment);
+
+ if (!(io_direct_flags & IO_DIRECT_DATA))
+ pgaio_io_set_flag(ioh, PGAIO_HF_BUFFERED);
+
+ pgaio_io_set_target_smgr(ioh,
+ reln,
+ forknum,
+ blocknum,
+ nblocks,
+ skipFsync);
+ pgaio_io_register_callbacks(ioh, PGAIO_HCB_MD_WRITEV, 0);
+
+ FileStartWriteV(ioh, v->mdfd_vfd, iovcnt, seekpos, WAIT_EVENT_DATA_FILE_WRITE);
+}
+
/*
* mdwriteback() -- Tell the kernel to write pages back to storage.
@@ -1495,6 +1552,40 @@ register_dirty_segment(SMgrRelation reln, ForkNumber forknum, MdfdVec *seg)
}
}
+/*
+ * Like register_dirty_segment(), except for use by AIO. In the completion
+ * callback we don't have access to the MdfdVec (the completion callback might
+ * be executed in a different backend than the issuing backend), therefore we
+ * have to implement this slightly differently.
+ */
+static void
+register_dirty_segment_aio(RelFileLocator locator, ForkNumber forknum, uint64 segno)
+{
+ FileTag tag;
+
+ INIT_MD_FILETAG(tag, locator, forknum, segno);
+
+ /*
+ * Can't block here waiting for checkpointer to accept our sync request,
+ * as checkpointer might be waiting for this AIO to finish if offloaded to
+ * a worker.
+ */
+ if (!RegisterSyncRequest(&tag, SYNC_REQUEST, false /* retryOnError */ ))
+ {
+ char path[MAXPGPATH];
+
+ ereport(DEBUG1,
+ (errmsg_internal("could not forward fsync request because request queue is full")));
+
+ /* reuse mdsyncfiletag() to avoid duplicating code */
+ if (mdsyncfiletag(&tag, path))
+ ereport(data_sync_elevel(ERROR),
+ (errcode_for_file_access(),
+ errmsg("could not fsync file \"%s\": %m",
+ path)));
+ }
+}
+
/*
* register_unlink_segment() -- Schedule a file to be deleted after next checkpoint
*/
@@ -2016,3 +2107,101 @@ md_readv_report(PgAioResult result, const PgAioTargetData *td, int elevel)
);
}
}
+
+/*
+ * AIO completion callback for mdstartwritev().
+ */
+static PgAioResult
+md_writev_complete(PgAioHandle *ioh, PgAioResult prior_result, uint8 cb_data)
+{
+ PgAioTargetData *td = pgaio_io_get_target_data(ioh);
+ PgAioResult result = prior_result;
+
+ if (prior_result.result < 0)
+ {
+ result.status = ARS_ERROR;
+ result.id = PGAIO_HCB_MD_WRITEV;
+ /* For "hard" errors, track the error number in error_data */
+ result.error_data = -prior_result.result;
+ result.result = 0;
+
+ md_writev_report(result, td, LOG);
+
+ return result;
+ }
+
+ /*
+ * The smgr API operates in blocks, therefore convert the result from
+ * bytes to blocks.
+ */
+ result.result /= BLCKSZ;
+
+ if (result.result == 0)
+ {
+ /* consider 0 blocks written a failure */
+ result.status = ARS_ERROR;
+ result.id = PGAIO_HCB_MD_WRITEV;
+ result.error_data = 0;
+
+ md_writev_report(result, td, LOG);
+
+ return result;
+ }
+
+ if (result.status != ARS_ERROR &&
+ result.result < td->smgr.nblocks)
+ {
+ /* partial writes should be retried at upper level */
+ result.status = ARS_PARTIAL;
+ result.id = PGAIO_HCB_MD_WRITEV;
+ }
+
+ if (!td->smgr.skip_fsync)
+ register_dirty_segment_aio(td->smgr.rlocator, td->smgr.forkNum,
+ td->smgr.blockNum / ((BlockNumber) RELSEG_SIZE));
+
+ return result;
+}
+
+/*
+ * AIO error reporting callback for mdstartwritev().
+ */
+static void
+md_writev_report(PgAioResult result, const PgAioTargetData *td, int elevel)
+{
+ RelPathStr path;
+
+ path = relpathbackend(td->smgr.rlocator,
+ td->smgr.is_temp ? MyProcNumber : INVALID_PROC_NUMBER,
+ td->smgr.forkNum);
+
+ if (result.error_data != 0)
+ {
+ errno = result.error_data; /* for errcode_for_file_access() */
+
+ ereport(elevel,
+ errcode_for_file_access(),
+ errmsg("could not write blocks %u..%u in file \"%s\": %m",
+ td->smgr.blockNum,
+ td->smgr.blockNum + td->smgr.nblocks,
+ path.str)
+ );
+ }
+ else
+ {
+ /*
+ * NB: This will typically only be output in debug messages, while
+ * retrying a partial IO.
+ */
+ ereport(elevel,
+ errcode(ERRCODE_DATA_CORRUPTED),
+ errmsg("could not write blocks %u..%u in file \"%s\": wrote only %zu of %zu bytes",
+ td->smgr.blockNum,
+ td->smgr.blockNum + td->smgr.nblocks - 1,
+ path.str,
+ result.result * (size_t) BLCKSZ,
+ td->smgr.nblocks * (size_t) BLCKSZ
+ )
+ );
+ }
+}
diff --git a/src/backend/storage/smgr/smgr.c b/src/backend/storage/smgr/smgr.c
index ea8b42ee4e8..6cbf8939fdc 100644
--- a/src/backend/storage/smgr/smgr.c
+++ b/src/backend/storage/smgr/smgr.c
@@ -102,6 +102,11 @@ typedef struct f_smgr
BlockNumber blocknum,
const void **buffers, BlockNumber nblocks,
bool skipFsync);
+ void (*smgr_startwritev) (struct PgAioHandle *ioh,
+ SMgrRelation reln, ForkNumber forknum,
+ BlockNumber blocknum,
+ const void **buffers, BlockNumber nblocks,
+ bool skipFsync);
void (*smgr_writeback) (SMgrRelation reln, ForkNumber forknum,
BlockNumber blocknum, BlockNumber nblocks);
BlockNumber (*smgr_nblocks) (SMgrRelation reln, ForkNumber forknum);
@@ -129,6 +134,7 @@ static const f_smgr smgrsw[] = {
.smgr_readv = mdreadv,
.smgr_startreadv = mdstartreadv,
.smgr_writev = mdwritev,
+ .smgr_startwritev = mdstartwritev,
.smgr_writeback = mdwriteback,
.smgr_nblocks = mdnblocks,
.smgr_truncate = mdtruncate,
@@ -691,6 +697,22 @@ smgrwritev(SMgrRelation reln, ForkNumber forknum, BlockNumber blocknum,
buffers, nblocks, skipFsync);
}
+/*
+ * smgrstartwritev() -- asynchronous version of smgrwritev()
+ *
+ * This starts an asynchronous writev IO with the IO handle `ioh`. Other than
+ * `ioh` all parameters are the same as smgrwritev().
+ */
+void
+smgrstartwritev(struct PgAioHandle *ioh,
+ SMgrRelation reln, ForkNumber forknum, BlockNumber blocknum,
+ const void **buffers, BlockNumber nblocks, bool skipFsync)
+{
+ smgrsw[reln->smgr_which].smgr_startwritev(ioh,
+ reln, forknum, blocknum, buffers,
+ nblocks, skipFsync);
+}
+
/*
* smgrwriteback() -- Trigger kernel writeback for the supplied range of
* blocks.
--
2.48.1.76.g4e746b1a31.dirty
[text/x-diff] v2.5-0023-aio-Add-bounce-buffers.patch (22.8K, ../../vz5i2x2wkjjpp6l2z4g4l3umpxktrxsejgdwefi7n6kzw666h3@rnhlvlpadig7/24-v2.5-0023-aio-Add-bounce-buffers.patch)
download | inline diff:
From f4cfdf93a1e5409302c99f4ef0c652840cf73f3f Mon Sep 17 00:00:00 2001
From: Andres Freund <andres@anarazel.de>
Date: Mon, 17 Feb 2025 20:03:15 -0500
Subject: [PATCH v2.5 23/30] aio: Add bounce buffers
---
src/include/storage/aio.h | 19 ++
src/include/storage/aio_internal.h | 33 ++++
src/include/utils/resowner.h | 2 +
src/backend/storage/aio/README.md | 27 +++
src/backend/storage/aio/aio.c | 178 ++++++++++++++++++
src/backend/storage/aio/aio_init.c | 123 ++++++++++++
src/backend/utils/misc/guc_tables.c | 13 ++
src/backend/utils/misc/postgresql.conf.sample | 2 +
src/backend/utils/resowner/resowner.c | 25 ++-
src/test/modules/test_aio/test_aio--1.0.sql | 21 +++
src/test/modules/test_aio/test_aio.c | 55 ++++++
src/tools/pgindent/typedefs.list | 1 +
12 files changed, 497 insertions(+), 2 deletions(-)
diff --git a/src/include/storage/aio.h b/src/include/storage/aio.h
index 4913ff723a9..ce5c18424bd 100644
--- a/src/include/storage/aio.h
+++ b/src/include/storage/aio.h
@@ -247,6 +247,10 @@ typedef struct PgAioHandleCallbacks
+typedef struct PgAioBounceBuffer PgAioBounceBuffer;
+
+
+
/* AIO API */
@@ -333,6 +337,20 @@ extern bool pgaio_have_staged(void);
+/* --------------------------------------------------------------------------------
+ * Bounce Buffers
+ * --------------------------------------------------------------------------------
+ */
+
+extern PgAioBounceBuffer *pgaio_bounce_buffer_get(void);
+extern void pgaio_io_assoc_bounce_buffer(PgAioHandle *ioh, PgAioBounceBuffer *bb);
+extern uint32 pgaio_bounce_buffer_id(PgAioBounceBuffer *bb);
+extern void pgaio_bounce_buffer_release(PgAioBounceBuffer *bb);
+extern char *pgaio_bounce_buffer_buffer(PgAioBounceBuffer *bb);
+extern void pgaio_bounce_buffer_release_resowner(struct dlist_node *bb_node, bool on_error);
+
+
+
/* --------------------------------------------------------------------------------
* Other
* --------------------------------------------------------------------------------
@@ -345,6 +363,7 @@ extern void pgaio_closing_fd(int fd);
/* GUCs */
extern PGDLLIMPORT int io_method;
extern PGDLLIMPORT int io_max_concurrency;
+extern PGDLLIMPORT int io_bounce_buffers;
#endif /* AIO_H */
diff --git a/src/include/storage/aio_internal.h b/src/include/storage/aio_internal.h
index f905b13e85f..835dafb5439 100644
--- a/src/include/storage/aio_internal.h
+++ b/src/include/storage/aio_internal.h
@@ -102,6 +102,12 @@ struct PgAioHandle
*/
uint32 iovec_off;
+ /*
+ * List of bounce_buffers owned by IO. It would suffice to use an index
+ * based linked list here.
+ */
+ slist_head bounce_buffers;
+
/**
* In which list the handle is registered, depends on the state:
* - IDLE, in per-backend list
@@ -138,11 +144,23 @@ struct PgAioHandle
};
+struct PgAioBounceBuffer
+{
+ slist_node node;
+ struct ResourceOwnerData *resowner;
+ dlist_node resowner_node;
+ char *buffer;
+};
+
+
typedef struct PgAioBackend
{
/* index into PgAioCtl->io_handles */
uint32 io_handle_off;
+ /* index into PgAioCtl->bounce_buffers */
+ uint32 bounce_buffers_off;
+
/* IO Handles that currently are not used */
dclist_head idle_ios;
@@ -173,6 +191,12 @@ typedef struct PgAioBackend
* IOs being appended at the end.
*/
dclist_head in_flight_ios;
+
+ /* Bounce Buffers that currently are not used */
+ slist_head idle_bbs;
+
+ /* see handed_out_io */
+ PgAioBounceBuffer *handed_out_bb;
} PgAioBackend;
@@ -198,6 +222,15 @@ typedef struct PgAioCtl
*/
uint64 *handle_data;
+ /*
+ * To perform AIO on buffers that are not located in shared memory (either
+ * because they are not in shared memory or because we need to operate on
+ * a copy, as e.g. the case for writes when checksums are in use)
+ */
+ uint64 bounce_buffers_count;
+ PgAioBounceBuffer *bounce_buffers;
+ char *bounce_buffers_data;
+
uint64 io_handle_count;
PgAioHandle *io_handles;
} PgAioCtl;
diff --git a/src/include/utils/resowner.h b/src/include/utils/resowner.h
index aede4bfc820..7e2ec224169 100644
--- a/src/include/utils/resowner.h
+++ b/src/include/utils/resowner.h
@@ -168,5 +168,7 @@ extern void ResourceOwnerForgetLock(ResourceOwner owner, struct LOCALLOCK *local
struct dlist_node;
extern void ResourceOwnerRememberAioHandle(ResourceOwner owner, struct dlist_node *ioh_node);
extern void ResourceOwnerForgetAioHandle(ResourceOwner owner, struct dlist_node *ioh_node);
+extern void ResourceOwnerRememberAioBounceBuffer(ResourceOwner owner, struct dlist_node *bb_node);
+extern void ResourceOwnerForgetAioBounceBuffer(ResourceOwner owner, struct dlist_node *bb_node);
#endif /* RESOWNER_H */
diff --git a/src/backend/storage/aio/README.md b/src/backend/storage/aio/README.md
index de6c7db894b..bd9436d665d 100644
--- a/src/backend/storage/aio/README.md
+++ b/src/backend/storage/aio/README.md
@@ -404,6 +404,33 @@ shared memory no less!), completion callbacks instead have to encode errors in
a more compact format that can be converted into an error message.
+### AIO Bounce Buffers
+
+For some uses of AIO there is no convenient memory location as the source /
+destination of an AIO. E.g. when data checksums are enabled, writes from
+shared buffers currently cannot be done directly from shared buffers, as a
+shared buffer lock still allows some modification, e.g., for hint bits(see
+`FlushBuffer()`). If the write were done in-place, such modifications can
+cause the checksum to fail.
+
+For synchronous IO this is solved by copying the buffer to separate memory
+before computing the checksum and using that copy as the source buffer for the
+AIO.
+
+However, for AIO that is not a workable solution:
+- Instead of a single buffer many buffers are required, as many IOs might be
+ in flight
+- When using the [worker method](#worker), the source/target of IO needs to be
+ in shared memory, otherwise the workers won't be able to access the memory.
+
+The AIO subsystem addresses this by providing a limited number of bounce
+buffers that can be used as the source / target for IO. A bounce buffer be
+acquired with `pgaio_bounce_buffer_get()` and multiple bounce buffers can be
+associated with an AIO Handle with `pgaio_io_assoc_bounce_buffer()`.
+
+Bounce buffers are automatically released when the IO completes.
+
+
## Helpers
Using the low-level AIO API introduces too much complexity to do so all over
diff --git a/src/backend/storage/aio/aio.c b/src/backend/storage/aio/aio.c
index 3af61227ea8..ad0e033390c 100644
--- a/src/backend/storage/aio/aio.c
+++ b/src/backend/storage/aio/aio.c
@@ -61,6 +61,8 @@ static PgAioHandle *pgaio_io_from_wref(PgAioWaitRef *iow, uint64 *ref_generation
static const char *pgaio_io_state_get_name(PgAioHandleState s);
static void pgaio_io_wait(PgAioHandle *ioh, uint64 ref_generation);
+static void pgaio_bounce_buffer_wait_for_free(void);
+
/* Options for io_method. */
const struct config_enum_entry io_method_options[] = {
@@ -75,6 +77,7 @@ const struct config_enum_entry io_method_options[] = {
/* GUCs */
int io_method = DEFAULT_IO_METHOD;
int io_max_concurrency = -1;
+int io_bounce_buffers = -1;
/* global control for AIO */
PgAioCtl *pgaio_ctl;
@@ -638,6 +641,21 @@ pgaio_io_reclaim(PgAioHandle *ioh)
}
}
+ /* reclaim all associated bounce buffers */
+ if (!slist_is_empty(&ioh->bounce_buffers))
+ {
+ slist_mutable_iter it;
+
+ slist_foreach_modify(it, &ioh->bounce_buffers)
+ {
+ PgAioBounceBuffer *bb = slist_container(PgAioBounceBuffer, node, it.cur);
+
+ slist_delete_current(&it);
+
+ slist_push_head(&pgaio_my_backend->idle_bbs, &bb->node);
+ }
+ }
+
if (ioh->resowner)
{
ResourceOwnerForgetAioHandle(ioh->resowner, &ioh->resowner_node);
@@ -1009,6 +1027,166 @@ pgaio_submit_staged(void)
+/* --------------------------------------------------------------------------------
+ * Functions primarily related to PgAioBounceBuffer
+ * --------------------------------------------------------------------------------
+ */
+
+PgAioBounceBuffer *
+pgaio_bounce_buffer_get(void)
+{
+ PgAioBounceBuffer *bb = NULL;
+ slist_node *node;
+
+ if (pgaio_my_backend->handed_out_bb != NULL)
+ elog(ERROR, "can only hand out one BB");
+
+ /*
+ * XXX: It probably is not a good idea to have bounce buffers be per
+ * backend, that's a fair bit of memory.
+ */
+ if (slist_is_empty(&pgaio_my_backend->idle_bbs))
+ {
+ pgaio_bounce_buffer_wait_for_free();
+ }
+
+ node = slist_pop_head_node(&pgaio_my_backend->idle_bbs);
+ bb = slist_container(PgAioBounceBuffer, node, node);
+
+ pgaio_my_backend->handed_out_bb = bb;
+
+ bb->resowner = CurrentResourceOwner;
+ ResourceOwnerRememberAioBounceBuffer(bb->resowner, &bb->resowner_node);
+
+ return bb;
+}
+
+void
+pgaio_io_assoc_bounce_buffer(PgAioHandle *ioh, PgAioBounceBuffer *bb)
+{
+ if (pgaio_my_backend->handed_out_bb != bb)
+ elog(ERROR, "can only assign handed out BB");
+ pgaio_my_backend->handed_out_bb = NULL;
+
+ /*
+ * There can be many bounce buffers assigned in case of vectorized IOs.
+ */
+ slist_push_head(&ioh->bounce_buffers, &bb->node);
+
+ /* once associated with an IO, the IO has ownership */
+ ResourceOwnerForgetAioBounceBuffer(bb->resowner, &bb->resowner_node);
+ bb->resowner = NULL;
+}
+
+uint32
+pgaio_bounce_buffer_id(PgAioBounceBuffer *bb)
+{
+ return bb - pgaio_ctl->bounce_buffers;
+}
+
+void
+pgaio_bounce_buffer_release(PgAioBounceBuffer *bb)
+{
+ if (pgaio_my_backend->handed_out_bb != bb)
+ elog(ERROR, "can only release handed out BB");
+
+ slist_push_head(&pgaio_my_backend->idle_bbs, &bb->node);
+ pgaio_my_backend->handed_out_bb = NULL;
+
+ ResourceOwnerForgetAioBounceBuffer(bb->resowner, &bb->resowner_node);
+ bb->resowner = NULL;
+}
+
+void
+pgaio_bounce_buffer_release_resowner(dlist_node *bb_node, bool on_error)
+{
+ PgAioBounceBuffer *bb = dlist_container(PgAioBounceBuffer, resowner_node, bb_node);
+
+ Assert(bb->resowner);
+
+ if (!on_error)
+ elog(WARNING, "leaked AIO bounce buffer");
+
+ pgaio_bounce_buffer_release(bb);
+}
+
+char *
+pgaio_bounce_buffer_buffer(PgAioBounceBuffer *bb)
+{
+ return bb->buffer;
+}
+
+static void
+pgaio_bounce_buffer_wait_for_free(void)
+{
+ static uint32 lastpos = 0;
+
+ if (pgaio_my_backend->num_staged_ios > 0)
+ {
+ pgaio_debug(DEBUG2, "submitting %d, while acquiring free bb",
+ pgaio_my_backend->num_staged_ios);
+ pgaio_submit_staged();
+ }
+
+ for (uint32 i = lastpos; i < lastpos + io_max_concurrency; i++)
+ {
+ uint32 thisoff = pgaio_my_backend->io_handle_off + (i % io_max_concurrency);
+ PgAioHandle *ioh = &pgaio_ctl->io_handles[thisoff];
+
+ switch (ioh->state)
+ {
+ case PGAIO_HS_IDLE:
+ case PGAIO_HS_HANDED_OUT:
+ continue;
+ case PGAIO_HS_DEFINED: /* should have been submitted above */
+ case PGAIO_HS_STAGED:
+ elog(ERROR, "shouldn't get here with io:%d in state %d",
+ pgaio_io_get_id(ioh), ioh->state);
+ break;
+ case PGAIO_HS_COMPLETED_IO:
+ case PGAIO_HS_SUBMITTED:
+ if (!slist_is_empty(&ioh->bounce_buffers))
+ {
+ pgaio_debug_io(DEBUG2, ioh,
+ "waiting for IO to reclaim BB with %d in flight",
+ dclist_count(&pgaio_my_backend->in_flight_ios));
+
+ /* see comment in pgaio_io_wait_for_free() about raciness */
+ pgaio_io_wait(ioh, ioh->generation);
+
+ if (slist_is_empty(&pgaio_my_backend->idle_bbs))
+ elog(WARNING, "empty after wait");
+
+ if (!slist_is_empty(&pgaio_my_backend->idle_bbs))
+ {
+ lastpos = i;
+ return;
+ }
+ }
+ break;
+ case PGAIO_HS_COMPLETED_SHARED:
+ case PGAIO_HS_COMPLETED_LOCAL:
+ /* reclaim */
+ pgaio_io_reclaim(ioh);
+
+ if (!slist_is_empty(&pgaio_my_backend->idle_bbs))
+ {
+ lastpos = i;
+ return;
+ }
+ break;
+ }
+ }
+
+ /*
+ * The submission above could have caused the IO to complete at any time.
+ */
+ if (slist_is_empty(&pgaio_my_backend->idle_bbs))
+ elog(PANIC, "no more bbs");
+}
+
+
+
/* --------------------------------------------------------------------------------
* Other
* --------------------------------------------------------------------------------
diff --git a/src/backend/storage/aio/aio_init.c b/src/backend/storage/aio/aio_init.c
index 2205658cd9a..a56126a772a 100644
--- a/src/backend/storage/aio/aio_init.c
+++ b/src/backend/storage/aio/aio_init.c
@@ -82,6 +82,32 @@ AioHandleDataShmemSize(void)
io_max_concurrency));
}
+static Size
+AioBounceBufferDescShmemSize(void)
+{
+ Size sz;
+
+ /* PgAioBounceBuffer itself */
+ sz = mul_size(sizeof(PgAioBounceBuffer),
+ mul_size(AioProcs(), io_bounce_buffers));
+
+ return sz;
+}
+
+static Size
+AioBounceBufferDataShmemSize(void)
+{
+ Size sz;
+
+ /* and the associated buffer */
+ sz = mul_size(BLCKSZ,
+ mul_size(io_bounce_buffers, AioProcs()));
+ /* memory for alignment */
+ sz += BLCKSZ;
+
+ return sz;
+}
+
/*
* Choose a suitable value for io_max_concurrency.
*
@@ -107,6 +133,33 @@ AioChooseMaxConcurrency(void)
return Min(max_proportional_pins, 64);
}
+/*
+ * Choose a suitable value for io_bounce_buffers.
+ *
+ * It's very unlikely to be useful to allocate more bounce buffers for each
+ * backend than the backend is allowed to pin. Additionally, bounce buffers
+ * currently are used for writes, it seems very uncommon for more than 10% of
+ * shared_buffers to be written out concurrently.
+ *
+ * XXX: This quickly can take up significant amounts of memory, the logic
+ * should probably fine tuned.
+ */
+static int
+AioChooseBounceBuffers(void)
+{
+ uint32 max_backends;
+ int max_proportional_pins;
+
+ /* Similar logic to LimitAdditionalPins() */
+ max_backends = MaxBackends + NUM_AUXILIARY_PROCS;
+ max_proportional_pins = (NBuffers / 10) / max_backends;
+
+ max_proportional_pins = Max(max_proportional_pins, 1);
+
+ /* apply upper limit */
+ return Min(max_proportional_pins, 256);
+}
+
Size
AioShmemSize(void)
{
@@ -130,11 +183,31 @@ AioShmemSize(void)
PGC_S_OVERRIDE);
}
+
+ /*
+ * If io_bounce_buffers is -1, we automatically choose a suitable value.
+ *
+ * See also comment above.
+ */
+ if (io_bounce_buffers == -1)
+ {
+ char buf[32];
+
+ snprintf(buf, sizeof(buf), "%d", AioChooseBounceBuffers());
+ SetConfigOption("io_bounce_buffers", buf, PGC_POSTMASTER,
+ PGC_S_DYNAMIC_DEFAULT);
+ if (io_bounce_buffers == -1) /* failed to apply it? */
+ SetConfigOption("io_bounce_buffers", buf, PGC_POSTMASTER,
+ PGC_S_OVERRIDE);
+ }
+
sz = add_size(sz, AioCtlShmemSize());
sz = add_size(sz, AioBackendShmemSize());
sz = add_size(sz, AioHandleShmemSize());
sz = add_size(sz, AioHandleIOVShmemSize());
sz = add_size(sz, AioHandleDataShmemSize());
+ sz = add_size(sz, AioBounceBufferDescShmemSize());
+ sz = add_size(sz, AioBounceBufferDataShmemSize());
if (pgaio_method_ops->shmem_size)
sz = add_size(sz, pgaio_method_ops->shmem_size());
@@ -149,6 +222,9 @@ AioShmemInit(void)
uint32 io_handle_off = 0;
uint32 iovec_off = 0;
uint32 per_backend_iovecs = io_max_concurrency * PG_IOV_MAX;
+ uint32 bounce_buffers_off = 0;
+ uint32 per_backend_bb = io_bounce_buffers;
+ char *bounce_buffers_data;
pgaio_ctl = (PgAioCtl *)
ShmemInitStruct("AioCtl", AioCtlShmemSize(), &found);
@@ -160,6 +236,7 @@ AioShmemInit(void)
pgaio_ctl->io_handle_count = AioProcs() * io_max_concurrency;
pgaio_ctl->iovec_count = AioProcs() * per_backend_iovecs;
+ pgaio_ctl->bounce_buffers_count = AioProcs() * per_backend_bb;
pgaio_ctl->backend_state = (PgAioBackend *)
ShmemInitStruct("AioBackend", AioBackendShmemSize(), &found);
@@ -172,6 +249,40 @@ AioShmemInit(void)
pgaio_ctl->handle_data = (uint64 *)
ShmemInitStruct("AioHandleData", AioHandleDataShmemSize(), &found);
+ pgaio_ctl->bounce_buffers = (PgAioBounceBuffer *)
+ ShmemInitStruct("AioBounceBufferDesc", AioBounceBufferDescShmemSize(),
+ &found);
+
+ bounce_buffers_data =
+ ShmemInitStruct("AioBounceBufferData", AioBounceBufferDataShmemSize(),
+ &found);
+ bounce_buffers_data =
+ (char *) TYPEALIGN(BLCKSZ, (uintptr_t) bounce_buffers_data);
+ pgaio_ctl->bounce_buffers_data = bounce_buffers_data;
+
+
+ /* Initialize IO handles. */
+ for (uint64 i = 0; i < pgaio_ctl->io_handle_count; i++)
+ {
+ PgAioHandle *ioh = &pgaio_ctl->io_handles[i];
+
+ ioh->op = PGAIO_OP_INVALID;
+ ioh->target = PGAIO_TID_INVALID;
+ ioh->state = PGAIO_HS_IDLE;
+
+ slist_init(&ioh->bounce_buffers);
+ }
+
+ /* Initialize Bounce Buffers. */
+ for (uint64 i = 0; i < pgaio_ctl->bounce_buffers_count; i++)
+ {
+ PgAioBounceBuffer *bb = &pgaio_ctl->bounce_buffers[i];
+
+ bb->buffer = bounce_buffers_data;
+ bounce_buffers_data += BLCKSZ;
+ }
+
+
for (int procno = 0; procno < AioProcs(); procno++)
{
PgAioBackend *bs = &pgaio_ctl->backend_state[procno];
@@ -179,9 +290,13 @@ AioShmemInit(void)
bs->io_handle_off = io_handle_off;
io_handle_off += io_max_concurrency;
+ bs->bounce_buffers_off = bounce_buffers_off;
+ bounce_buffers_off += per_backend_bb;
+
dclist_init(&bs->idle_ios);
memset(bs->staged_ios, 0, sizeof(PgAioHandle *) * PGAIO_SUBMIT_BATCH_SIZE);
dclist_init(&bs->in_flight_ios);
+ slist_init(&bs->idle_bbs);
/* initialize per-backend IOs */
for (int i = 0; i < io_max_concurrency; i++)
@@ -203,6 +318,14 @@ AioShmemInit(void)
dclist_push_tail(&bs->idle_ios, &ioh->node);
iovec_off += PG_IOV_MAX;
}
+
+ /* initialize per-backend bounce buffers */
+ for (int i = 0; i < per_backend_bb; i++)
+ {
+ PgAioBounceBuffer *bb = &pgaio_ctl->bounce_buffers[bs->bounce_buffers_off + i];
+
+ slist_push_head(&bs->idle_bbs, &bb->node);
+ }
}
out:
diff --git a/src/backend/utils/misc/guc_tables.c b/src/backend/utils/misc/guc_tables.c
index 4317cfc9d2f..41a4025c66d 100644
--- a/src/backend/utils/misc/guc_tables.c
+++ b/src/backend/utils/misc/guc_tables.c
@@ -3268,6 +3268,19 @@ struct config_int ConfigureNamesInt[] =
check_io_max_concurrency, NULL, NULL
},
+ {
+ {"io_bounce_buffers",
+ PGC_POSTMASTER,
+ RESOURCES_IO,
+ gettext_noop("Number of IO Bounce Buffers reserved for each backend."),
+ NULL,
+ GUC_UNIT_BLOCKS
+ },
+ &io_bounce_buffers,
+ -1, -1, 4096,
+ NULL, NULL, NULL
+ },
+
{
{"io_workers",
PGC_SIGHUP,
diff --git a/src/backend/utils/misc/postgresql.conf.sample b/src/backend/utils/misc/postgresql.conf.sample
index 8738ad51bf1..291ca8ab38d 100644
--- a/src/backend/utils/misc/postgresql.conf.sample
+++ b/src/backend/utils/misc/postgresql.conf.sample
@@ -209,6 +209,8 @@
# -1 sets based on shared_buffers
# (change requires restart)
#io_workers = 3 # 1-32;
+#io_bounce_buffers = -1 # -1 sets based on shared_buffers
+ # (change requires restart)
# - Worker Processes -
diff --git a/src/backend/utils/resowner/resowner.c b/src/backend/utils/resowner/resowner.c
index 76b9cec1e26..de00346b549 100644
--- a/src/backend/utils/resowner/resowner.c
+++ b/src/backend/utils/resowner/resowner.c
@@ -159,10 +159,11 @@ struct ResourceOwnerData
LOCALLOCK *locks[MAX_RESOWNER_LOCKS]; /* list of owned locks */
/*
- * AIO handles need be registered in critical sections and therefore
- * cannot use the normal ResoureElem mechanism.
+ * AIO handles & bounce buffers need be registered in critical sections
+ * and therefore cannot use the normal ResoureElem mechanism.
*/
dlist_head aio_handles;
+ dlist_head aio_bounce_buffers;
};
@@ -434,6 +435,7 @@ ResourceOwnerCreate(ResourceOwner parent, const char *name)
}
dlist_init(&owner->aio_handles);
+ dlist_init(&owner->aio_bounce_buffers);
return owner;
}
@@ -742,6 +744,13 @@ ResourceOwnerReleaseInternal(ResourceOwner owner,
pgaio_io_release_resowner(node, !isCommit);
}
+
+ while (!dlist_is_empty(&owner->aio_bounce_buffers))
+ {
+ dlist_node *node = dlist_head_node(&owner->aio_bounce_buffers);
+
+ pgaio_bounce_buffer_release_resowner(node, !isCommit);
+ }
}
else if (phase == RESOURCE_RELEASE_LOCKS)
{
@@ -1111,3 +1120,15 @@ ResourceOwnerForgetAioHandle(ResourceOwner owner, struct dlist_node *ioh_node)
{
dlist_delete_from(&owner->aio_handles, ioh_node);
}
+
+void
+ResourceOwnerRememberAioBounceBuffer(ResourceOwner owner, struct dlist_node *ioh_node)
+{
+ dlist_push_tail(&owner->aio_bounce_buffers, ioh_node);
+}
+
+void
+ResourceOwnerForgetAioBounceBuffer(ResourceOwner owner, struct dlist_node *ioh_node)
+{
+ dlist_delete_from(&owner->aio_bounce_buffers, ioh_node);
+}
diff --git a/src/test/modules/test_aio/test_aio--1.0.sql b/src/test/modules/test_aio/test_aio--1.0.sql
index 3d4e9bb070f..16f248619fc 100644
--- a/src/test/modules/test_aio/test_aio--1.0.sql
+++ b/src/test/modules/test_aio/test_aio--1.0.sql
@@ -76,6 +76,27 @@ RETURNS pg_catalog.void STRICT
AS 'MODULE_PATHNAME' LANGUAGE C;
+CREATE FUNCTION bb_get_and_error()
+RETURNS pg_catalog.void STRICT
+AS 'MODULE_PATHNAME' LANGUAGE C;
+
+CREATE FUNCTION bb_get_twice()
+RETURNS pg_catalog.void STRICT
+AS 'MODULE_PATHNAME' LANGUAGE C;
+
+CREATE FUNCTION bb_get()
+RETURNS pg_catalog.void STRICT
+AS 'MODULE_PATHNAME' LANGUAGE C;
+
+CREATE FUNCTION bb_get_release()
+RETURNS pg_catalog.void STRICT
+AS 'MODULE_PATHNAME' LANGUAGE C;
+
+CREATE FUNCTION bb_release_last()
+RETURNS pg_catalog.void STRICT
+AS 'MODULE_PATHNAME' LANGUAGE C;
+
+
/*
* Injection point related functions
diff --git a/src/test/modules/test_aio/test_aio.c b/src/test/modules/test_aio/test_aio.c
index 81b4d732206..60c01b251fe 100644
--- a/src/test/modules/test_aio/test_aio.c
+++ b/src/test/modules/test_aio/test_aio.c
@@ -53,6 +53,7 @@ static shmem_startup_hook_type prev_shmem_startup_hook = NULL;
static PgAioHandle *last_handle;
+static PgAioBounceBuffer *last_bb;
@@ -525,6 +526,60 @@ batch_end(PG_FUNCTION_ARGS)
PG_RETURN_VOID();
}
+PG_FUNCTION_INFO_V1(bb_get);
+Datum
+bb_get(PG_FUNCTION_ARGS)
+{
+ last_bb = pgaio_bounce_buffer_get();
+
+ PG_RETURN_VOID();
+}
+
+PG_FUNCTION_INFO_V1(bb_release_last);
+Datum
+bb_release_last(PG_FUNCTION_ARGS)
+{
+ if (!last_bb)
+ elog(ERROR, "no bb");
+
+ pgaio_bounce_buffer_release(last_bb);
+
+ PG_RETURN_VOID();
+}
+
+PG_FUNCTION_INFO_V1(bb_get_and_error);
+Datum
+bb_get_and_error(PG_FUNCTION_ARGS)
+{
+ pgaio_bounce_buffer_get();
+
+ elog(ERROR, "as you command");
+ PG_RETURN_VOID();
+}
+
+PG_FUNCTION_INFO_V1(bb_get_twice);
+Datum
+bb_get_twice(PG_FUNCTION_ARGS)
+{
+ pgaio_bounce_buffer_get();
+ pgaio_bounce_buffer_get();
+
+ PG_RETURN_VOID();
+}
+
+
+PG_FUNCTION_INFO_V1(bb_get_release);
+Datum
+bb_get_release(PG_FUNCTION_ARGS)
+{
+ PgAioBounceBuffer *bb;
+
+ bb = pgaio_bounce_buffer_get();
+ pgaio_bounce_buffer_release(bb);
+
+ PG_RETURN_VOID();
+}
+
#ifdef USE_INJECTION_POINTS
extern PGDLLEXPORT void inj_io_short_read(const char *name, const void *private_data);
extern PGDLLEXPORT void inj_io_reopen(const char *name, const void *private_data);
diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list
index b35cf61cd36..86b46e93536 100644
--- a/src/tools/pgindent/typedefs.list
+++ b/src/tools/pgindent/typedefs.list
@@ -2128,6 +2128,7 @@ PermutationStep
PermutationStepBlocker
PermutationStepBlockerType
PgAioBackend
+PgAioBounceBuffer
PgAioCtl
PgAioHandle
PgAioHandleCallbackID
--
2.48.1.76.g4e746b1a31.dirty
[text/x-diff] v2.5-0024-bufmgr-Implement-AIO-write-support.patch (6.2K, ../../vz5i2x2wkjjpp6l2z4g4l3umpxktrxsejgdwefi7n6kzw666h3@rnhlvlpadig7/25-v2.5-0024-bufmgr-Implement-AIO-write-support.patch)
download | inline diff:
From 40ec3844785b610c2c9aa66af644a9f9d4fbb5da Mon Sep 17 00:00:00 2001
From: Andres Freund <andres@anarazel.de>
Date: Wed, 22 Jan 2025 16:09:51 -0500
Subject: [PATCH v2.5 24/30] bufmgr: Implement AIO write support
As of this commit there are no users of these AIO facilities, that'll come in
later commits.
Author:
Reviewed-By:
Discussion: https://postgr.es/m/
Backpatch:
---
src/include/storage/aio.h | 2 +
src/include/storage/bufmgr.h | 2 +
src/backend/storage/aio/aio_callback.c | 2 +
src/backend/storage/buffer/bufmgr.c | 88 ++++++++++++++++++++++++++
4 files changed, 94 insertions(+)
diff --git a/src/include/storage/aio.h b/src/include/storage/aio.h
index ce5c18424bd..912bce87197 100644
--- a/src/include/storage/aio.h
+++ b/src/include/storage/aio.h
@@ -180,8 +180,10 @@ typedef enum PgAioHandleCallbackID
PGAIO_HCB_MD_WRITEV,
PGAIO_HCB_SHARED_BUFFER_READV,
+ PGAIO_HCB_SHARED_BUFFER_WRITEV,
PGAIO_HCB_LOCAL_BUFFER_READV,
+ PGAIO_HCB_LOCAL_BUFFER_WRITEV,
} PgAioHandleCallbackID;
diff --git a/src/include/storage/bufmgr.h b/src/include/storage/bufmgr.h
index beeb4c47c1c..7d8618b0b85 100644
--- a/src/include/storage/bufmgr.h
+++ b/src/include/storage/bufmgr.h
@@ -186,7 +186,9 @@ extern PGDLLIMPORT int32 *LocalRefCount;
struct PgAioHandleCallbacks;
extern const struct PgAioHandleCallbacks aio_shared_buffer_readv_cb;
+extern const struct PgAioHandleCallbacks aio_shared_buffer_writev_cb;
extern const struct PgAioHandleCallbacks aio_local_buffer_readv_cb;
+extern const struct PgAioHandleCallbacks aio_local_buffer_writev_cb;
/* upper limit for effective_io_concurrency */
diff --git a/src/backend/storage/aio/aio_callback.c b/src/backend/storage/aio/aio_callback.c
index f0414085eba..00098504efb 100644
--- a/src/backend/storage/aio/aio_callback.c
+++ b/src/backend/storage/aio/aio_callback.c
@@ -44,8 +44,10 @@ static const PgAioHandleCallbacksEntry aio_handle_cbs[] = {
CALLBACK_ENTRY(PGAIO_HCB_MD_WRITEV, aio_md_writev_cb),
CALLBACK_ENTRY(PGAIO_HCB_SHARED_BUFFER_READV, aio_shared_buffer_readv_cb),
+ CALLBACK_ENTRY(PGAIO_HCB_SHARED_BUFFER_WRITEV, aio_shared_buffer_writev_cb),
CALLBACK_ENTRY(PGAIO_HCB_LOCAL_BUFFER_READV, aio_local_buffer_readv_cb),
+ CALLBACK_ENTRY(PGAIO_HCB_LOCAL_BUFFER_WRITEV, aio_local_buffer_writev_cb),
#undef CALLBACK_ENTRY
};
diff --git a/src/backend/storage/buffer/bufmgr.c b/src/backend/storage/buffer/bufmgr.c
index efd5b6601ad..0944914bb80 100644
--- a/src/backend/storage/buffer/bufmgr.c
+++ b/src/backend/storage/buffer/bufmgr.c
@@ -6514,6 +6514,42 @@ SharedBufferCompleteRead(int buf_off, Buffer buffer, uint8 flags, bool failed)
return result;
}
+static uint64
+BufferCompleteWriteShared(Buffer buffer, bool release_lock, bool failed)
+{
+ BufferDesc *bufHdr;
+ bool result = false;
+
+ Assert(BufferIsValid(buffer));
+
+ bufHdr = GetBufferDescriptor(buffer - 1);
+
+#ifdef USE_ASSERT_CHECKING
+ {
+ uint32 buf_state = pg_atomic_read_u32(&bufHdr->state);
+
+ Assert(buf_state & BM_VALID);
+ Assert(buf_state & BM_TAG_VALID);
+ Assert(buf_state & BM_IO_IN_PROGRESS);
+ Assert(buf_state & BM_DIRTY);
+ }
+#endif
+
+ TerminateBufferIO(bufHdr, /* clear_dirty = */ true,
+ failed ? BM_IO_ERROR : 0,
+ /* forget_owner = */ false,
+ /* syncio = */ false);
+
+ /*
+ * The initiator of IO is not managing the lock (i.e. called
+ * LWLockDisown()), we are.
+ */
+ if (release_lock)
+ LWLockReleaseDisowned(BufferDescriptorGetContentLock(bufHdr), LW_SHARED);
+
+ return result;
+}
+
/*
* Helper to prepare IO on shared buffers for execution, shared between reads
* and writes.
@@ -6594,6 +6630,12 @@ shared_buffer_readv_stage(PgAioHandle *ioh, uint8 cb_data)
shared_buffer_stage_common(ioh, false);
}
+static void
+shared_buffer_writev_stage(PgAioHandle *ioh, uint8 cb_data)
+{
+ shared_buffer_stage_common(ioh, true);
+}
+
static void
buffer_readv_report(PgAioResult result, const PgAioTargetData *target_data, int elevel)
{
@@ -6680,6 +6722,33 @@ shared_buffer_readv_complete(PgAioHandle *ioh, PgAioResult prior_result, uint8 c
return buffer_readv_complete_common(ioh, prior_result, false, cb_data);
}
+static PgAioResult
+shared_buffer_writev_complete(PgAioHandle *ioh, PgAioResult prior_result, uint8 cb_data)
+{
+ PgAioResult result = prior_result;
+ uint64 *io_data;
+ uint8 handle_data_len;
+
+ ereport(DEBUG5,
+ errmsg("%s: %d %d", __func__, prior_result.status, prior_result.result),
+ errhidestmt(true), errhidecontext(true));
+
+ io_data = pgaio_io_get_handle_data(ioh, &handle_data_len);
+
+ /* FIXME: handle outright errors */
+
+ for (int io_data_off = 0; io_data_off < handle_data_len; io_data_off++)
+ {
+ Buffer buf = io_data[io_data_off];
+
+ /* FIXME: handle short writes / failures */
+ /* FIXME: ioh->target_data.shared_buffer.release_lock */
+ BufferCompleteWriteShared(buf, true, false);
+ }
+
+ return result;
+}
+
/*
* Helper to stage IO on local buffers for execution, shared between reads
* and writes.
@@ -6724,6 +6793,16 @@ local_buffer_readv_complete(PgAioHandle *ioh, PgAioResult prior_result, uint8 cb
return buffer_readv_complete_common(ioh, prior_result, true, cb_data);
}
+static void
+local_buffer_writev_stage(PgAioHandle *ioh, uint8 cb_data)
+{
+ /*
+ * Currently this is unreachable as the only write support is for
+ * checkpointer / bgwriter, which don't deal with local buffers.
+ */
+ elog(ERROR, "not yet");
+}
+
/* readv callback is is passed READ_BUFFERS_* flags as callback data */
const struct PgAioHandleCallbacks aio_shared_buffer_readv_cb = {
@@ -6732,6 +6811,11 @@ const struct PgAioHandleCallbacks aio_shared_buffer_readv_cb = {
.report = buffer_readv_report,
};
+const struct PgAioHandleCallbacks aio_shared_buffer_writev_cb = {
+ .stage = shared_buffer_writev_stage,
+ .complete_shared = shared_buffer_writev_complete,
+};
+
/* readv callback is is passed READ_BUFFERS_* flags as callback data */
const struct PgAioHandleCallbacks aio_local_buffer_readv_cb = {
.stage = local_buffer_readv_stage,
@@ -6745,3 +6829,7 @@ const struct PgAioHandleCallbacks aio_local_buffer_readv_cb = {
.complete_local = local_buffer_readv_complete,
.report = buffer_readv_report,
};
+
+const struct PgAioHandleCallbacks aio_local_buffer_writev_cb = {
+ .stage = local_buffer_writev_stage,
+};
--
2.48.1.76.g4e746b1a31.dirty
[text/x-diff] v2.5-0025-aio-Add-IO-queue-helper.patch (7.4K, ../../vz5i2x2wkjjpp6l2z4g4l3umpxktrxsejgdwefi7n6kzw666h3@rnhlvlpadig7/26-v2.5-0025-aio-Add-IO-queue-helper.patch)
download | inline diff:
From c67abebdc3bd5bff9aee5cf83a31792d29c9a8ee Mon Sep 17 00:00:00 2001
From: Andres Freund <andres@anarazel.de>
Date: Wed, 22 Jan 2025 13:44:50 -0500
Subject: [PATCH v2.5 25/30] aio: Add IO queue helper
This is likely never going to anywhere - Thomas Munro is working on something
more complete. But I needed a way to exercise aio for checkpointer / bgwriter.
---
src/include/storage/io_queue.h | 31 +++++
src/backend/storage/aio/Makefile | 1 +
src/backend/storage/aio/io_queue.c | 204 ++++++++++++++++++++++++++++
src/backend/storage/aio/meson.build | 1 +
src/tools/pgindent/typedefs.list | 2 +
5 files changed, 239 insertions(+)
create mode 100644 src/include/storage/io_queue.h
create mode 100644 src/backend/storage/aio/io_queue.c
diff --git a/src/include/storage/io_queue.h b/src/include/storage/io_queue.h
new file mode 100644
index 00000000000..f5e1bc07ff3
--- /dev/null
+++ b/src/include/storage/io_queue.h
@@ -0,0 +1,31 @@
+/*-------------------------------------------------------------------------
+ *
+ * io_queue.h
+ * Mechanism for tracking many IOs
+ *
+ *
+ * Portions Copyright (c) 1996-2025, PostgreSQL Global Development Group
+ * Portions Copyright (c) 1994, Regents of the University of California
+ *
+ * src/include/storage/io_queue.h
+ *
+ *-------------------------------------------------------------------------
+ */
+#ifndef IO_QUEUE_H
+#define IO_QUEUE_H
+
+struct IOQueue;
+typedef struct IOQueue IOQueue;
+
+struct PgAioWaitRef;
+
+extern IOQueue *io_queue_create(int depth, int flags);
+extern void io_queue_track(IOQueue *ioq, const struct PgAioWaitRef *iow);
+extern void io_queue_wait_one(IOQueue *ioq);
+extern void io_queue_wait_all(IOQueue *ioq);
+extern bool io_queue_is_empty(IOQueue *ioq);
+extern void io_queue_reserve(IOQueue *ioq);
+extern struct PgAioHandle *io_queue_acquire_io(IOQueue *ioq);
+extern void io_queue_free(IOQueue *ioq);
+
+#endif /* IO_QUEUE_H */
diff --git a/src/backend/storage/aio/Makefile b/src/backend/storage/aio/Makefile
index 3f2469cc399..86fa4276fda 100644
--- a/src/backend/storage/aio/Makefile
+++ b/src/backend/storage/aio/Makefile
@@ -15,6 +15,7 @@ OBJS = \
aio_init.o \
aio_io.o \
aio_target.o \
+ io_queue.o \
method_io_uring.o \
method_sync.o \
method_worker.o \
diff --git a/src/backend/storage/aio/io_queue.c b/src/backend/storage/aio/io_queue.c
new file mode 100644
index 00000000000..526aa1d5e06
--- /dev/null
+++ b/src/backend/storage/aio/io_queue.c
@@ -0,0 +1,204 @@
+/*-------------------------------------------------------------------------
+ *
+ * io_queue.c
+ * AIO - Mechanism for tracking many IOs
+ *
+ * Portions Copyright (c) 1996-2025, PostgreSQL Global Development Group
+ * Portions Copyright (c) 1994, Regents of the University of California
+ *
+ * IDENTIFICATION
+ * src/backend/storage/aio/io_queue.c
+ *
+ *-------------------------------------------------------------------------
+ */
+
+#include "postgres.h"
+
+#include "lib/ilist.h"
+#include "storage/aio.h"
+#include "storage/io_queue.h"
+#include "utils/resowner.h"
+
+
+
+typedef struct TrackedIO
+{
+ PgAioWaitRef iow;
+ dlist_node node;
+} TrackedIO;
+
+struct IOQueue
+{
+ int depth;
+ int unsubmitted;
+
+ bool has_reserved;
+
+ dclist_head idle;
+ dclist_head in_progress;
+
+ TrackedIO tracked_ios[FLEXIBLE_ARRAY_MEMBER];
+};
+
+
+IOQueue *
+io_queue_create(int depth, int flags)
+{
+ size_t sz;
+ IOQueue *ioq;
+
+ sz = offsetof(IOQueue, tracked_ios)
+ + sizeof(TrackedIO) * depth;
+
+ ioq = palloc0(sz);
+
+ ioq->depth = 0;
+
+ for (int i = 0; i < depth; i++)
+ {
+ TrackedIO *tio = &ioq->tracked_ios[i];
+
+ pgaio_wref_clear(&tio->iow);
+ dclist_push_tail(&ioq->idle, &tio->node);
+ }
+
+ return ioq;
+}
+
+void
+io_queue_wait_one(IOQueue *ioq)
+{
+ /* submit all pending IO before waiting */
+ pgaio_submit_staged();
+
+ while (!dclist_is_empty(&ioq->in_progress))
+ {
+ /* FIXME: Should we really pop here already? */
+ dlist_node *node = dclist_pop_head_node(&ioq->in_progress);
+ TrackedIO *tio = dclist_container(TrackedIO, node, node);
+
+ pgaio_wref_wait(&tio->iow);
+ dclist_push_head(&ioq->idle, &tio->node);
+ }
+}
+
+void
+io_queue_reserve(IOQueue *ioq)
+{
+ if (ioq->has_reserved)
+ return;
+
+ if (dclist_is_empty(&ioq->idle))
+ io_queue_wait_one(ioq);
+
+ Assert(!dclist_is_empty(&ioq->idle));
+
+ ioq->has_reserved = true;
+}
+
+PgAioHandle *
+io_queue_acquire_io(IOQueue *ioq)
+{
+ PgAioHandle *ioh;
+
+ io_queue_reserve(ioq);
+
+ Assert(!dclist_is_empty(&ioq->idle));
+
+ if (!io_queue_is_empty(ioq))
+ {
+ ioh = pgaio_io_acquire_nb(CurrentResourceOwner, NULL);
+ if (ioh == NULL)
+ {
+ /*
+ * Need to wait for all IOs, blocking might not be legal in the
+ * context.
+ *
+ * XXX: This doesn't make a whole lot of sense, we're also
+ * blocking here. What was I smoking when I wrote the above?
+ */
+ io_queue_wait_all(ioq);
+ ioh = pgaio_io_acquire(CurrentResourceOwner, NULL);
+ }
+ }
+ else
+ {
+ ioh = pgaio_io_acquire(CurrentResourceOwner, NULL);
+ }
+
+ return ioh;
+}
+
+void
+io_queue_track(IOQueue *ioq, const struct PgAioWaitRef *iow)
+{
+ dlist_node *node;
+ TrackedIO *tio;
+
+ Assert(ioq->has_reserved);
+ ioq->has_reserved = false;
+
+ Assert(!dclist_is_empty(&ioq->idle));
+
+ node = dclist_pop_head_node(&ioq->idle);
+ tio = dclist_container(TrackedIO, node, node);
+
+ tio->iow = *iow;
+
+ dclist_push_tail(&ioq->in_progress, &tio->node);
+
+ ioq->unsubmitted++;
+
+ /*
+ * XXX: Should have some smarter logic here. We don't want to wait too
+ * long to submit, that'll mean we're more likely to block. But we also
+ * don't want to have the overhead of submitting every IO individually.
+ */
+ if (ioq->unsubmitted >= 4)
+ {
+ pgaio_submit_staged();
+ ioq->unsubmitted = 0;
+ }
+}
+
+void
+io_queue_wait_all(IOQueue *ioq)
+{
+ /* submit all pending IO before waiting */
+ pgaio_submit_staged();
+
+ while (!dclist_is_empty(&ioq->in_progress))
+ {
+ /* wait for the last IO to minimize unnecessary wakeups */
+ dlist_node *node = dclist_tail_node(&ioq->in_progress);
+ TrackedIO *tio = dclist_container(TrackedIO, node, node);
+
+ if (!pgaio_wref_check_done(&tio->iow))
+ {
+ ereport(DEBUG3,
+ errmsg("io_queue_wait_all for io:%d",
+ pgaio_wref_get_id(&tio->iow)),
+ errhidestmt(true),
+ errhidecontext(true));
+
+ pgaio_wref_wait(&tio->iow);
+ }
+
+ dclist_delete_from(&ioq->in_progress, &tio->node);
+ dclist_push_head(&ioq->idle, &tio->node);
+ }
+}
+
+bool
+io_queue_is_empty(IOQueue *ioq)
+{
+ return dclist_is_empty(&ioq->in_progress);
+}
+
+void
+io_queue_free(IOQueue *ioq)
+{
+ io_queue_wait_all(ioq);
+
+ pfree(ioq);
+}
diff --git a/src/backend/storage/aio/meson.build b/src/backend/storage/aio/meson.build
index da6df2d3654..270c4a64428 100644
--- a/src/backend/storage/aio/meson.build
+++ b/src/backend/storage/aio/meson.build
@@ -7,6 +7,7 @@ backend_sources += files(
'aio_init.c',
'aio_io.c',
'aio_target.c',
+ 'io_queue.c',
'method_io_uring.c',
'method_sync.c',
'method_worker.c',
diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list
index 86b46e93536..8f22fba3479 100644
--- a/src/tools/pgindent/typedefs.list
+++ b/src/tools/pgindent/typedefs.list
@@ -1189,6 +1189,7 @@ IOContext
IOFuncSelector
IOObject
IOOp
+IOQueue
IO_STATUS_BLOCK
IPCompareMethod
ITEM
@@ -3010,6 +3011,7 @@ TocEntry
TokenAuxData
TokenizedAuthLine
TrackItem
+TrackedIO
TransApplyAction
TransInvalidationInfo
TransState
--
2.48.1.76.g4e746b1a31.dirty
[text/x-diff] v2.5-0026-bufmgr-use-AIO-in-checkpointer-bgwriter.patch (31.1K, ../../vz5i2x2wkjjpp6l2z4g4l3umpxktrxsejgdwefi7n6kzw666h3@rnhlvlpadig7/27-v2.5-0026-bufmgr-use-AIO-in-checkpointer-bgwriter.patch)
download | inline diff:
From a1c9b7ac4fd4772bcbaa9cdd9fd9a1a503a4a9e1 Mon Sep 17 00:00:00 2001
From: Andres Freund <andres@anarazel.de>
Date: Wed, 22 Jan 2025 13:44:52 -0500
Subject: [PATCH v2.5 26/30] bufmgr: use AIO in checkpointer, bgwriter
This is far from ready - just included to be able to exercise AIO writes and
get some preliminary numbers. In all likelihood this will instead be based
ontop of work by Thomas Munro instead of the preceding commit.
---
src/include/postmaster/bgwriter.h | 3 +-
src/include/storage/buf_internals.h | 2 +
src/include/storage/bufmgr.h | 3 +-
src/include/storage/bufpage.h | 1 +
src/backend/postmaster/bgwriter.c | 19 +-
src/backend/postmaster/checkpointer.c | 11 +-
src/backend/storage/buffer/bufmgr.c | 588 +++++++++++++++++++++++---
src/backend/storage/page/bufpage.c | 10 +
src/tools/pgindent/typedefs.list | 1 +
9 files changed, 580 insertions(+), 58 deletions(-)
diff --git a/src/include/postmaster/bgwriter.h b/src/include/postmaster/bgwriter.h
index 4fd717169f0..4208d5f2c97 100644
--- a/src/include/postmaster/bgwriter.h
+++ b/src/include/postmaster/bgwriter.h
@@ -31,7 +31,8 @@ extern void BackgroundWriterMain(const void *startup_data, size_t startup_data_l
extern void CheckpointerMain(const void *startup_data, size_t startup_data_len) pg_attribute_noreturn();
extern void RequestCheckpoint(int flags);
-extern void CheckpointWriteDelay(int flags, double progress);
+struct IOQueue;
+extern void CheckpointWriteDelay(struct IOQueue *ioq, int flags, double progress);
extern bool ForwardSyncRequest(const FileTag *ftag, SyncRequestType type);
diff --git a/src/include/storage/buf_internals.h b/src/include/storage/buf_internals.h
index 817c85ec9ed..13d7926e3a3 100644
--- a/src/include/storage/buf_internals.h
+++ b/src/include/storage/buf_internals.h
@@ -21,6 +21,8 @@
#include "storage/buf.h"
#include "storage/bufmgr.h"
#include "storage/condition_variable.h"
+#include "storage/io_queue.h"
+#include "storage/latch.h"
#include "storage/lwlock.h"
#include "storage/procnumber.h"
#include "storage/shmem.h"
diff --git a/src/include/storage/bufmgr.h b/src/include/storage/bufmgr.h
index 7d8618b0b85..1449f540a28 100644
--- a/src/include/storage/bufmgr.h
+++ b/src/include/storage/bufmgr.h
@@ -302,7 +302,8 @@ extern bool ConditionalLockBufferForCleanup(Buffer buffer);
extern bool IsBufferCleanupOK(Buffer buffer);
extern bool HoldingBufferPinThatDelaysRecovery(void);
-extern bool BgBufferSync(struct WritebackContext *wb_context);
+struct IOQueue;
+extern bool BgBufferSync(struct IOQueue *ioq, struct WritebackContext *wb_context);
extern uint32 GetSoftPinLimit(void);
extern uint32 GetSoftLocalPinLimit(void);
diff --git a/src/include/storage/bufpage.h b/src/include/storage/bufpage.h
index 6646b6f6371..9c045e81857 100644
--- a/src/include/storage/bufpage.h
+++ b/src/include/storage/bufpage.h
@@ -509,5 +509,6 @@ extern bool PageIndexTupleOverwrite(Page page, OffsetNumber offnum,
Item newtup, Size newsize);
extern char *PageSetChecksumCopy(Page page, BlockNumber blkno);
extern void PageSetChecksumInplace(Page page, BlockNumber blkno);
+extern bool PageNeedsChecksumCopy(Page page);
#endif /* BUFPAGE_H */
diff --git a/src/backend/postmaster/bgwriter.c b/src/backend/postmaster/bgwriter.c
index 3d350c611a5..4190fa0a591 100644
--- a/src/backend/postmaster/bgwriter.c
+++ b/src/backend/postmaster/bgwriter.c
@@ -38,11 +38,13 @@
#include "postmaster/auxprocess.h"
#include "postmaster/bgwriter.h"
#include "postmaster/interrupt.h"
+#include "storage/aio.h"
#include "storage/aio_subsys.h"
#include "storage/buf_internals.h"
#include "storage/bufmgr.h"
#include "storage/condition_variable.h"
#include "storage/fd.h"
+#include "storage/io_queue.h"
#include "storage/lwlock.h"
#include "storage/proc.h"
#include "storage/procsignal.h"
@@ -90,6 +92,7 @@ BackgroundWriterMain(const void *startup_data, size_t startup_data_len)
sigjmp_buf local_sigjmp_buf;
MemoryContext bgwriter_context;
bool prev_hibernate;
+ IOQueue *ioq;
WritebackContext wb_context;
Assert(startup_data_len == 0);
@@ -131,6 +134,7 @@ BackgroundWriterMain(const void *startup_data, size_t startup_data_len)
ALLOCSET_DEFAULT_SIZES);
MemoryContextSwitchTo(bgwriter_context);
+ ioq = io_queue_create(128, 0);
WritebackContextInit(&wb_context, &bgwriter_flush_after);
/*
@@ -228,12 +232,22 @@ BackgroundWriterMain(const void *startup_data, size_t startup_data_len)
/* Clear any already-pending wakeups */
ResetLatch(MyLatch);
+ /*
+ * FIXME: this is theoretically racy, but I didn't want to copy
+ * HandleMainLoopInterrupts() remaining body here.
+ */
+ if (ShutdownRequestPending)
+ {
+ io_queue_wait_all(ioq);
+ io_queue_free(ioq);
+ }
+
HandleMainLoopInterrupts();
/*
* Do one cycle of dirty-buffer writing.
*/
- can_hibernate = BgBufferSync(&wb_context);
+ can_hibernate = BgBufferSync(ioq, &wb_context);
/* Report pending statistics to the cumulative stats system */
pgstat_report_bgwriter();
@@ -250,6 +264,9 @@ BackgroundWriterMain(const void *startup_data, size_t startup_data_len)
smgrdestroyall();
}
+ /* finish IO before sleeping, to avoid blocking other backends */
+ io_queue_wait_all(ioq);
+
/*
* Log a new xl_running_xacts every now and then so replication can
* get into a consistent state faster (think of suboverflowed
diff --git a/src/backend/postmaster/checkpointer.c b/src/backend/postmaster/checkpointer.c
index 4dc24739fbd..7fb9545e477 100644
--- a/src/backend/postmaster/checkpointer.c
+++ b/src/backend/postmaster/checkpointer.c
@@ -49,10 +49,12 @@
#include "postmaster/bgwriter.h"
#include "postmaster/interrupt.h"
#include "replication/syncrep.h"
+#include "storage/aio.h"
#include "storage/aio_subsys.h"
#include "storage/bufmgr.h"
#include "storage/condition_variable.h"
#include "storage/fd.h"
+#include "storage/io_queue.h"
#include "storage/ipc.h"
#include "storage/lwlock.h"
#include "storage/pmsignal.h"
@@ -766,7 +768,7 @@ ImmediateCheckpointRequested(void)
* fraction between 0.0 meaning none, and 1.0 meaning all done.
*/
void
-CheckpointWriteDelay(int flags, double progress)
+CheckpointWriteDelay(IOQueue *ioq, int flags, double progress)
{
static int absorb_counter = WRITES_PER_ABSORB;
@@ -800,6 +802,13 @@ CheckpointWriteDelay(int flags, double progress)
/* Report interim statistics to the cumulative stats system */
pgstat_report_checkpointer();
+ /*
+ * Ensure all pending IO is submitted to avoid unnecessary delays for
+ * other processes.
+ */
+ io_queue_wait_all(ioq);
+
+
/*
* This sleep used to be connected to bgwriter_delay, typically 200ms.
* That resulted in more frequent wakeups if not much work to do.
diff --git a/src/backend/storage/buffer/bufmgr.c b/src/backend/storage/buffer/bufmgr.c
index 0944914bb80..4ba69c8b341 100644
--- a/src/backend/storage/buffer/bufmgr.c
+++ b/src/backend/storage/buffer/bufmgr.c
@@ -52,6 +52,7 @@
#include "storage/buf_internals.h"
#include "storage/bufmgr.h"
#include "storage/fd.h"
+#include "storage/io_queue.h"
#include "storage/ipc.h"
#include "storage/lmgr.h"
#include "storage/proc.h"
@@ -76,6 +77,7 @@
/* Bits in SyncOneBuffer's return value */
#define BUF_WRITTEN 0x01
#define BUF_REUSABLE 0x02
+#define BUF_CANT_MERGE 0x04
#define RELS_BSEARCH_THRESHOLD 20
@@ -512,8 +514,6 @@ static void UnpinBuffer(BufferDesc *buf);
static void UnpinBufferNoOwner(BufferDesc *buf);
static void BufferSync(int flags);
static uint32 WaitBufHdrUnlocked(BufferDesc *buf);
-static int SyncOneBuffer(int buf_id, bool skip_recently_used,
- WritebackContext *wb_context);
static void WaitIO(BufferDesc *buf);
static void AbortBufferIO(Buffer buffer);
static void shared_buffer_write_error_callback(void *arg);
@@ -529,6 +529,7 @@ static bool AsyncReadBuffers(ReadBuffersOperation *operation,
static Buffer GetVictimBuffer(BufferAccessStrategy strategy, IOContext io_context);
static void FlushBuffer(BufferDesc *buf, SMgrRelation reln,
IOObject io_object, IOContext io_context);
+
static void FindAndDropRelationBuffers(RelFileLocator rlocator,
ForkNumber forkNum,
BlockNumber nForkBlock,
@@ -3180,6 +3181,57 @@ UnpinBufferNoOwner(BufferDesc *buf)
}
}
+typedef struct BuffersToWrite
+{
+ int nbuffers;
+ BufferTag start_at_tag;
+ uint32 max_combine;
+
+ XLogRecPtr max_lsn;
+
+ PgAioHandle *ioh;
+ PgAioWaitRef iow;
+
+ uint64 total_writes;
+
+ Buffer buffers[IOV_MAX];
+ PgAioBounceBuffer *bounce_buffers[IOV_MAX];
+ const void *data_ptrs[IOV_MAX];
+} BuffersToWrite;
+
+static int PrepareToWriteBuffer(BuffersToWrite *to_write, Buffer buf,
+ bool skip_recently_used,
+ IOQueue *ioq, WritebackContext *wb_context);
+
+static void WriteBuffers(BuffersToWrite *to_write,
+ IOQueue *ioq, WritebackContext *wb_context);
+
+static void
+BuffersToWriteInit(BuffersToWrite *to_write,
+ IOQueue *ioq, WritebackContext *wb_context)
+{
+ to_write->total_writes = 0;
+ to_write->nbuffers = 0;
+ to_write->ioh = NULL;
+ pgaio_wref_clear(&to_write->iow);
+ to_write->max_lsn = InvalidXLogRecPtr;
+
+ pgaio_enter_batchmode();
+}
+
+static void
+BuffersToWriteEnd(BuffersToWrite *to_write)
+{
+ if (to_write->ioh != NULL)
+ {
+ pgaio_io_release(to_write->ioh);
+ to_write->ioh = NULL;
+ }
+
+ pgaio_exit_batchmode();
+}
+
+
#define ST_SORT sort_checkpoint_bufferids
#define ST_ELEMENT_TYPE CkptSortItem
#define ST_COMPARE(a, b) ckpt_buforder_comparator(a, b)
@@ -3211,7 +3263,10 @@ BufferSync(int flags)
binaryheap *ts_heap;
int i;
int mask = BM_DIRTY;
+ IOQueue *ioq;
WritebackContext wb_context;
+ BuffersToWrite to_write;
+ int max_combine;
/*
* Unless this is a shutdown checkpoint or we have been explicitly told,
@@ -3273,7 +3328,9 @@ BufferSync(int flags)
if (num_to_scan == 0)
return; /* nothing to do */
+ ioq = io_queue_create(512, 0);
WritebackContextInit(&wb_context, &checkpoint_flush_after);
+ max_combine = Min(io_bounce_buffers, io_combine_limit);
TRACE_POSTGRESQL_BUFFER_SYNC_START(NBuffers, num_to_scan);
@@ -3381,48 +3438,91 @@ BufferSync(int flags)
*/
num_processed = 0;
num_written = 0;
+
+ BuffersToWriteInit(&to_write, ioq, &wb_context);
+
while (!binaryheap_empty(ts_heap))
{
BufferDesc *bufHdr = NULL;
CkptTsStatus *ts_stat = (CkptTsStatus *)
DatumGetPointer(binaryheap_first(ts_heap));
+ bool batch_continue = true;
- buf_id = CkptBufferIds[ts_stat->index].buf_id;
- Assert(buf_id != -1);
-
- bufHdr = GetBufferDescriptor(buf_id);
-
- num_processed++;
+ Assert(ts_stat->num_scanned <= ts_stat->num_to_scan);
/*
- * We don't need to acquire the lock here, because we're only looking
- * at a single bit. It's possible that someone else writes the buffer
- * and clears the flag right after we check, but that doesn't matter
- * since SyncOneBuffer will then do nothing. However, there is a
- * further race condition: it's conceivable that between the time we
- * examine the bit here and the time SyncOneBuffer acquires the lock,
- * someone else not only wrote the buffer but replaced it with another
- * page and dirtied it. In that improbable case, SyncOneBuffer will
- * write the buffer though we didn't need to. It doesn't seem worth
- * guarding against this, though.
+ * Collect a batch of buffers to write out from the current
+ * tablespace. That causes some imbalance between the tablespaces, but
+ * that's more than outweighed by the efficiency gain due to batching.
*/
- if (pg_atomic_read_u32(&bufHdr->state) & BM_CHECKPOINT_NEEDED)
+ while (batch_continue &&
+ to_write.nbuffers < max_combine &&
+ ts_stat->num_scanned < ts_stat->num_to_scan)
{
- if (SyncOneBuffer(buf_id, false, &wb_context) & BUF_WRITTEN)
+ buf_id = CkptBufferIds[ts_stat->index].buf_id;
+ Assert(buf_id != -1);
+
+ bufHdr = GetBufferDescriptor(buf_id);
+
+ num_processed++;
+
+ /*
+ * We don't need to acquire the lock here, because we're only
+ * looking at a single bit. It's possible that someone else writes
+ * the buffer and clears the flag right after we check, but that
+ * doesn't matter since SyncOneBuffer will then do nothing.
+ * However, there is a further race condition: it's conceivable
+ * that between the time we examine the bit here and the time
+ * SyncOneBuffer acquires the lock, someone else not only wrote
+ * the buffer but replaced it with another page and dirtied it. In
+ * that improbable case, SyncOneBuffer will write the buffer
+ * though we didn't need to. It doesn't seem worth guarding
+ * against this, though.
+ */
+ if (pg_atomic_read_u32(&bufHdr->state) & BM_CHECKPOINT_NEEDED)
{
- TRACE_POSTGRESQL_BUFFER_SYNC_WRITTEN(buf_id);
- PendingCheckpointerStats.buffers_written++;
- num_written++;
+ int result = PrepareToWriteBuffer(&to_write, buf_id + 1, false,
+ ioq, &wb_context);
+
+ if (result & BUF_CANT_MERGE)
+ {
+ Assert(to_write.nbuffers > 0);
+ WriteBuffers(&to_write, ioq, &wb_context);
+
+ result = PrepareToWriteBuffer(&to_write, buf_id + 1, false,
+ ioq, &wb_context);
+ Assert(result != BUF_CANT_MERGE);
+ }
+
+ if (result & BUF_WRITTEN)
+ {
+ TRACE_POSTGRESQL_BUFFER_SYNC_WRITTEN(buf_id);
+ PendingCheckpointerStats.buffers_written++;
+ num_written++;
+ }
+ else
+ {
+ batch_continue = false;
+ }
}
+ else
+ {
+ if (to_write.nbuffers > 0)
+ WriteBuffers(&to_write, ioq, &wb_context);
+ }
+
+ /*
+ * Measure progress independent of actually having to flush the
+ * buffer - otherwise writing become unbalanced.
+ */
+ ts_stat->progress += ts_stat->progress_slice;
+ ts_stat->num_scanned++;
+ ts_stat->index++;
}
- /*
- * Measure progress independent of actually having to flush the buffer
- * - otherwise writing become unbalanced.
- */
- ts_stat->progress += ts_stat->progress_slice;
- ts_stat->num_scanned++;
- ts_stat->index++;
+ if (to_write.nbuffers > 0)
+ WriteBuffers(&to_write, ioq, &wb_context);
+
/* Have all the buffers from the tablespace been processed? */
if (ts_stat->num_scanned == ts_stat->num_to_scan)
@@ -3440,15 +3540,23 @@ BufferSync(int flags)
*
* (This will check for barrier events even if it doesn't sleep.)
*/
- CheckpointWriteDelay(flags, (double) num_processed / num_to_scan);
+ CheckpointWriteDelay(ioq, flags, (double) num_processed / num_to_scan);
}
+ Assert(to_write.nbuffers == 0);
+ io_queue_wait_all(ioq);
+
/*
* Issue all pending flushes. Only checkpointer calls BufferSync(), so
* IOContext will always be IOCONTEXT_NORMAL.
*/
IssuePendingWritebacks(&wb_context, IOCONTEXT_NORMAL);
+ io_queue_wait_all(ioq); /* IssuePendingWritebacks might have added
+ * more */
+ io_queue_free(ioq);
+ BuffersToWriteEnd(&to_write);
+
pfree(per_ts_stat);
per_ts_stat = NULL;
binaryheap_free(ts_heap);
@@ -3474,7 +3582,7 @@ BufferSync(int flags)
* bgwriter_lru_maxpages to 0.)
*/
bool
-BgBufferSync(WritebackContext *wb_context)
+BgBufferSync(IOQueue *ioq, WritebackContext *wb_context)
{
/* info obtained from freelist.c */
int strategy_buf_id;
@@ -3517,6 +3625,9 @@ BgBufferSync(WritebackContext *wb_context)
long new_strategy_delta;
uint32 new_recent_alloc;
+ BuffersToWrite to_write;
+ int max_combine;
+
/*
* Find out where the freelist clock sweep currently is, and how many
* buffer allocations have happened since our last call.
@@ -3537,6 +3648,8 @@ BgBufferSync(WritebackContext *wb_context)
return true;
}
+ max_combine = Min(io_bounce_buffers, io_combine_limit);
+
/*
* Compute strategy_delta = how many buffers have been scanned by the
* clock sweep since last time. If first time through, assume none. Then
@@ -3693,11 +3806,25 @@ BgBufferSync(WritebackContext *wb_context)
num_written = 0;
reusable_buffers = reusable_buffers_est;
+ BuffersToWriteInit(&to_write, ioq, wb_context);
+
/* Execute the LRU scan */
while (num_to_scan > 0 && reusable_buffers < upcoming_alloc_est)
{
- int sync_state = SyncOneBuffer(next_to_clean, true,
- wb_context);
+ int sync_state;
+
+ sync_state = PrepareToWriteBuffer(&to_write, next_to_clean + 1,
+ true, ioq, wb_context);
+ if (sync_state & BUF_CANT_MERGE)
+ {
+ Assert(to_write.nbuffers > 0);
+
+ WriteBuffers(&to_write, ioq, wb_context);
+
+ sync_state = PrepareToWriteBuffer(&to_write, next_to_clean + 1,
+ true, ioq, wb_context);
+ Assert(sync_state != BUF_CANT_MERGE);
+ }
if (++next_to_clean >= NBuffers)
{
@@ -3708,6 +3835,13 @@ BgBufferSync(WritebackContext *wb_context)
if (sync_state & BUF_WRITTEN)
{
+ Assert(sync_state & BUF_REUSABLE);
+
+ if (to_write.nbuffers == max_combine)
+ {
+ WriteBuffers(&to_write, ioq, wb_context);
+ }
+
reusable_buffers++;
if (++num_written >= bgwriter_lru_maxpages)
{
@@ -3719,6 +3853,11 @@ BgBufferSync(WritebackContext *wb_context)
reusable_buffers++;
}
+ if (to_write.nbuffers > 0)
+ WriteBuffers(&to_write, ioq, wb_context);
+
+ BuffersToWriteEnd(&to_write);
+
PendingBgWriterStats.buf_written_clean += num_written;
#ifdef BGW_DEBUG
@@ -3757,8 +3896,66 @@ BgBufferSync(WritebackContext *wb_context)
return (bufs_to_lap == 0 && recent_alloc == 0);
}
+static inline bool
+BufferTagsSameRel(const BufferTag *tag1, const BufferTag *tag2)
+{
+ return (tag1->spcOid == tag2->spcOid) &&
+ (tag1->dbOid == tag2->dbOid) &&
+ (tag1->relNumber == tag2->relNumber) &&
+ (tag1->forkNum == tag2->forkNum)
+ ;
+}
+
+static bool
+CanMergeWrite(BuffersToWrite *to_write, BufferDesc *cur_buf_hdr)
+{
+ BlockNumber cur_block = cur_buf_hdr->tag.blockNum;
+
+ Assert(to_write->nbuffers > 0); /* can't merge with nothing */
+ Assert(to_write->start_at_tag.relNumber != InvalidOid);
+ Assert(to_write->start_at_tag.blockNum != InvalidBlockNumber);
+
+ Assert(to_write->ioh != NULL);
+
+ /*
+ * First check if the blocknumber is one that we could actually merge,
+ * that's cheaper than checking the tablespace/db/relnumber/fork match.
+ */
+ if (to_write->start_at_tag.blockNum + to_write->nbuffers != cur_block)
+ return false;
+
+ if (!BufferTagsSameRel(&to_write->start_at_tag, &cur_buf_hdr->tag))
+ return false;
+
+ /*
+ * Need to check with smgr how large a write we're allowed to make. To
+ * reduce the overhead of the smgr check, only inquire once, when
+ * processing the first to-be-merged buffer. That avoids the overhead in
+ * the common case of writing out buffers that definitely not mergeable.
+ */
+ if (to_write->nbuffers == 1)
+ {
+ SMgrRelation smgr;
+
+ smgr = smgropen(BufTagGetRelFileLocator(&to_write->start_at_tag), INVALID_PROC_NUMBER);
+
+ to_write->max_combine = smgrmaxcombine(smgr,
+ to_write->start_at_tag.forkNum,
+ to_write->start_at_tag.blockNum);
+ }
+ else
+ {
+ Assert(to_write->max_combine > 0);
+ }
+
+ if (to_write->start_at_tag.blockNum + to_write->max_combine <= cur_block)
+ return false;
+
+ return true;
+}
+
/*
- * SyncOneBuffer -- process a single buffer during syncing.
+ * PrepareToWriteBuffer -- process a single buffer during syncing.
*
* If skip_recently_used is true, we don't write currently-pinned buffers, nor
* buffers marked recently used, as these are not replacement candidates.
@@ -3767,22 +3964,50 @@ BgBufferSync(WritebackContext *wb_context)
* BUF_WRITTEN: we wrote the buffer.
* BUF_REUSABLE: buffer is available for replacement, ie, it has
* pin count 0 and usage count 0.
+ * BUF_CANT_MERGE: can't combine this write with prior writes, caller needs
+ * to issue those first
*
* (BUF_WRITTEN could be set in error if FlushBuffer finds the buffer clean
* after locking it, but we don't care all that much.)
*/
static int
-SyncOneBuffer(int buf_id, bool skip_recently_used, WritebackContext *wb_context)
+PrepareToWriteBuffer(BuffersToWrite *to_write, Buffer buf,
+ bool skip_recently_used,
+ IOQueue *ioq, WritebackContext *wb_context)
{
- BufferDesc *bufHdr = GetBufferDescriptor(buf_id);
+ BufferDesc *cur_buf_hdr = GetBufferDescriptor(buf - 1);
+ uint32 buf_state;
int result = 0;
- uint32 buf_state;
- BufferTag tag;
+ XLogRecPtr cur_buf_lsn;
+ LWLock *content_lock;
+ bool may_block;
+
+ /*
+ * Check if this buffer can be written out together with already prepared
+ * writes. We check before we have pinned the buffer, so the buffer can be
+ * written out and replaced between this check and us pinning the buffer -
+ * we'll recheck below. The reason for the pre-check is that we don't want
+ * to pin the buffer just to find out that we can't merge the IO.
+ */
+ if (to_write->nbuffers != 0)
+ {
+ if (!CanMergeWrite(to_write, cur_buf_hdr))
+ {
+ result |= BUF_CANT_MERGE;
+ return result;
+ }
+ }
+ else
+ {
+ to_write->start_at_tag = cur_buf_hdr->tag;
+ }
/* Make sure we can handle the pin */
ReservePrivateRefCountEntry();
ResourceOwnerEnlarge(CurrentResourceOwner);
+ /* XXX: Should also check if we are allowed to pin one more buffer */
+
/*
* Check whether buffer needs writing.
*
@@ -3792,7 +4017,7 @@ SyncOneBuffer(int buf_id, bool skip_recently_used, WritebackContext *wb_context)
* don't worry because our checkpoint.redo points before log record for
* upcoming changes and so we are not required to write such dirty buffer.
*/
- buf_state = LockBufHdr(bufHdr);
+ buf_state = LockBufHdr(cur_buf_hdr);
if (BUF_STATE_GET_REFCOUNT(buf_state) == 0 &&
BUF_STATE_GET_USAGECOUNT(buf_state) == 0)
@@ -3801,40 +4026,294 @@ SyncOneBuffer(int buf_id, bool skip_recently_used, WritebackContext *wb_context)
}
else if (skip_recently_used)
{
+#if 0
+ elog(LOG, "at block %d: skip recent with nbuffers %d",
+ cur_buf_hdr->tag.blockNum, to_write->nbuffers);
+#endif
/* Caller told us not to write recently-used buffers */
- UnlockBufHdr(bufHdr, buf_state);
+ UnlockBufHdr(cur_buf_hdr, buf_state);
return result;
}
if (!(buf_state & BM_VALID) || !(buf_state & BM_DIRTY))
{
/* It's clean, so nothing to do */
- UnlockBufHdr(bufHdr, buf_state);
+ UnlockBufHdr(cur_buf_hdr, buf_state);
return result;
}
+ /* pin the buffer, from now on its identity can't change anymore */
+ PinBuffer_Locked(cur_buf_hdr);
+
+ /*
+ * Acquire IO, if needed, now that it's likely that we'll need to write.
+ */
+ if (to_write->ioh == NULL)
+ {
+ /* otherwise we should already have acquired a handle */
+ Assert(to_write->nbuffers == 0);
+
+ to_write->ioh = io_queue_acquire_io(ioq);
+ pgaio_io_get_wref(to_write->ioh, &to_write->iow);
+ }
+
/*
- * Pin it, share-lock it, write it. (FlushBuffer will do nothing if the
- * buffer is clean by the time we've locked it.)
+ * If we are merging, check if the buffer's identity possibly changed
+ * while we hadn't yet pinned it.
+ *
+ * XXX: It might be worth checking if we still want to write the buffer
+ * out, e.g. it could have been replaced with a buffer that doesn't have
+ * BM_CHECKPOINT_NEEDED set.
*/
- PinBuffer_Locked(bufHdr);
- LWLockAcquire(BufferDescriptorGetContentLock(bufHdr), LW_SHARED);
+ if (to_write->nbuffers != 0)
+ {
+ if (!CanMergeWrite(to_write, cur_buf_hdr))
+ {
+ elog(LOG, "changed identity");
+ UnpinBuffer(cur_buf_hdr);
+
+ result |= BUF_CANT_MERGE;
+
+ return result;
+ }
+ }
+
+ may_block = to_write->nbuffers == 0
+ && !pgaio_have_staged()
+ && io_queue_is_empty(ioq)
+ ;
+ content_lock = BufferDescriptorGetContentLock(cur_buf_hdr);
+
+ if (!may_block)
+ {
+ if (LWLockConditionalAcquire(content_lock, LW_SHARED))
+ {
+ /* done */
+ }
+ else if (to_write->nbuffers == 0)
+ {
+ /*
+ * Need to wait for all prior IO to finish before blocking for
+ * lock acquisition, to avoid the risk a deadlock due to us
+ * waiting for another backend that is waiting for our unsubmitted
+ * IO to complete.
+ */
+ pgaio_submit_staged();
+ io_queue_wait_all(ioq);
+
+ elog(DEBUG2, "at block %u: can't block, nbuffers = 0",
+ cur_buf_hdr->tag.blockNum
+ );
+
+ may_block = to_write->nbuffers == 0
+ && !pgaio_have_staged()
+ && io_queue_is_empty(ioq)
+ ;
+ Assert(may_block);
+
+ LWLockAcquire(content_lock, LW_SHARED);
+ }
+ else
+ {
+ elog(DEBUG2, "at block %d: can't block nbuffers = %d",
+ cur_buf_hdr->tag.blockNum,
+ to_write->nbuffers);
- FlushBuffer(bufHdr, NULL, IOOBJECT_RELATION, IOCONTEXT_NORMAL);
+ UnpinBuffer(cur_buf_hdr);
+ result |= BUF_CANT_MERGE;
+ Assert(to_write->nbuffers > 0);
- LWLockRelease(BufferDescriptorGetContentLock(bufHdr));
+ return result;
+ }
+ }
+ else
+ {
+ LWLockAcquire(content_lock, LW_SHARED);
+ }
- tag = bufHdr->tag;
+ if (!may_block)
+ {
+ if (!StartBufferIO(cur_buf_hdr, false, !may_block))
+ {
+ pgaio_submit_staged();
+ io_queue_wait_all(ioq);
- UnpinBuffer(bufHdr);
+ may_block = io_queue_is_empty(ioq) && to_write->nbuffers == 0 && !pgaio_have_staged();
+
+ if (!StartBufferIO(cur_buf_hdr, false, !may_block))
+ {
+ elog(DEBUG2, "at block %d: non-waitable StartBufferIO returns false, %d",
+ cur_buf_hdr->tag.blockNum,
+ may_block);
+
+ /*
+ * FIXME: can't tell whether this is because the buffer has
+ * been cleaned
+ */
+ if (!may_block)
+ {
+ result |= BUF_CANT_MERGE;
+ Assert(to_write->nbuffers > 0);
+ }
+ LWLockRelease(content_lock);
+ UnpinBuffer(cur_buf_hdr);
+
+ return result;
+ }
+ }
+ }
+ else
+ {
+ if (!StartBufferIO(cur_buf_hdr, false, false))
+ {
+ elog(DEBUG2, "waitable StartBufferIO returns false");
+ LWLockRelease(content_lock);
+ UnpinBuffer(cur_buf_hdr);
+
+ /*
+ * FIXME: Historically we returned BUF_WRITTEN in this case, which
+ * seems wrong
+ */
+ return result;
+ }
+ }
/*
- * SyncOneBuffer() is only called by checkpointer and bgwriter, so
- * IOContext will always be IOCONTEXT_NORMAL.
+ * Run PageGetLSN while holding header lock, since we don't have the
+ * buffer locked exclusively in all cases.
*/
- ScheduleBufferTagForWriteback(wb_context, IOCONTEXT_NORMAL, &tag);
+ buf_state = LockBufHdr(cur_buf_hdr);
+
+ cur_buf_lsn = BufferGetLSN(cur_buf_hdr);
+
+ /* To check if block content changes while flushing. - vadim 01/17/97 */
+ buf_state &= ~BM_JUST_DIRTIED;
+
+ UnlockBufHdr(cur_buf_hdr, buf_state);
+
+ to_write->buffers[to_write->nbuffers] = buf;
+ to_write->nbuffers++;
+
+ if (buf_state & BM_PERMANENT &&
+ (to_write->max_lsn == InvalidXLogRecPtr || to_write->max_lsn < cur_buf_lsn))
+ {
+ to_write->max_lsn = cur_buf_lsn;
+ }
+
+ result |= BUF_WRITTEN;
+
+ return result;
+}
+
+static void
+WriteBuffers(BuffersToWrite *to_write,
+ IOQueue *ioq, WritebackContext *wb_context)
+{
+ SMgrRelation smgr;
+ Buffer first_buf;
+ BufferDesc *first_buf_hdr;
+ bool needs_checksum;
+
+ Assert(to_write->nbuffers > 0 && to_write->nbuffers <= io_combine_limit);
+
+ first_buf = to_write->buffers[0];
+ first_buf_hdr = GetBufferDescriptor(first_buf - 1);
+
+ smgr = smgropen(BufTagGetRelFileLocator(&first_buf_hdr->tag), INVALID_PROC_NUMBER);
+
+ /*
+ * Force XLOG flush up to buffer's LSN. This implements the basic WAL
+ * rule that log updates must hit disk before any of the data-file changes
+ * they describe do.
+ *
+ * However, this rule does not apply to unlogged relations, which will be
+ * lost after a crash anyway. Most unlogged relation pages do not bear
+ * LSNs since we never emit WAL records for them, and therefore flushing
+ * up through the buffer LSN would be useless, but harmless. However,
+ * GiST indexes use LSNs internally to track page-splits, and therefore
+ * unlogged GiST pages bear "fake" LSNs generated by
+ * GetFakeLSNForUnloggedRel. It is unlikely but possible that the fake
+ * LSN counter could advance past the WAL insertion point; and if it did
+ * happen, attempting to flush WAL through that location would fail, with
+ * disastrous system-wide consequences. To make sure that can't happen,
+ * skip the flush if the buffer isn't permanent.
+ */
+ if (to_write->max_lsn != InvalidXLogRecPtr)
+ XLogFlush(to_write->max_lsn);
+
+ /*
+ * Now it's safe to write buffer to disk. Note that no one else should
+ * have been able to write it while we were busy with log flushing because
+ * only one process at a time can set the BM_IO_IN_PROGRESS bit.
+ */
+
+ for (int nbuf = 0; nbuf < to_write->nbuffers; nbuf++)
+ {
+ Buffer cur_buf = to_write->buffers[nbuf];
+ BufferDesc *cur_buf_hdr = GetBufferDescriptor(cur_buf - 1);
+ Block bufBlock;
+ char *bufToWrite;
+
+ bufBlock = BufHdrGetBlock(cur_buf_hdr);
+ needs_checksum = PageNeedsChecksumCopy((Page) bufBlock);
+
+ /*
+ * Update page checksum if desired. Since we have only shared lock on
+ * the buffer, other processes might be updating hint bits in it, so
+ * we must copy the page to a bounce buffer if we do checksumming.
+ */
+ if (needs_checksum)
+ {
+ PgAioBounceBuffer *bb = pgaio_bounce_buffer_get();
+
+ pgaio_io_assoc_bounce_buffer(to_write->ioh, bb);
+
+ bufToWrite = pgaio_bounce_buffer_buffer(bb);
+ memcpy(bufToWrite, bufBlock, BLCKSZ);
+ PageSetChecksumInplace((Page) bufToWrite, cur_buf_hdr->tag.blockNum);
+ }
+ else
+ {
+ bufToWrite = bufBlock;
+ }
+
+ to_write->data_ptrs[nbuf] = bufToWrite;
+ }
+
+ pgaio_io_set_handle_data_32(to_write->ioh,
+ (uint32 *) to_write->buffers,
+ to_write->nbuffers);
+ pgaio_io_register_callbacks(to_write->ioh, PGAIO_HCB_SHARED_BUFFER_WRITEV, 0);
+
+ smgrstartwritev(to_write->ioh, smgr,
+ BufTagGetForkNum(&first_buf_hdr->tag),
+ first_buf_hdr->tag.blockNum,
+ to_write->data_ptrs,
+ to_write->nbuffers,
+ false);
+ pgstat_count_io_op(IOOBJECT_RELATION, IOCONTEXT_NORMAL,
+ IOOP_WRITE, 1, BLCKSZ * to_write->nbuffers);
+
+
+ for (int nbuf = 0; nbuf < to_write->nbuffers; nbuf++)
+ {
+ Buffer cur_buf = to_write->buffers[nbuf];
+ BufferDesc *cur_buf_hdr = GetBufferDescriptor(cur_buf - 1);
+
+ UnpinBuffer(cur_buf_hdr);
+ }
+
+ io_queue_track(ioq, &to_write->iow);
+ to_write->total_writes++;
- return result | BUF_WRITTEN;
+ /* clear state for next write */
+ to_write->nbuffers = 0;
+ to_write->start_at_tag.relNumber = InvalidOid;
+ to_write->start_at_tag.blockNum = InvalidBlockNumber;
+ to_write->max_combine = 0;
+ to_write->max_lsn = InvalidXLogRecPtr;
+ to_write->ioh = NULL;
+ pgaio_wref_clear(&to_write->iow);
}
/*
@@ -4208,6 +4687,7 @@ FlushBuffer(BufferDesc *buf, SMgrRelation reln, IOObject io_object,
error_context_stack = errcallback.previous;
}
+
/*
* RelationGetNumberOfBlocksInFork
* Determines the current number of pages in the specified relation fork.
diff --git a/src/backend/storage/page/bufpage.c b/src/backend/storage/page/bufpage.c
index ecc81aacfc3..71dc6f559dd 100644
--- a/src/backend/storage/page/bufpage.c
+++ b/src/backend/storage/page/bufpage.c
@@ -1480,6 +1480,16 @@ PageIndexTupleOverwrite(Page page, OffsetNumber offnum,
return true;
}
+bool
+PageNeedsChecksumCopy(Page page)
+{
+ if (PageIsNew(page))
+ return false;
+
+ /* If we don't need a checksum, just return the passed-in data */
+ return DataChecksumsEnabled();
+}
+
/*
* Set checksum for a page in shared buffers.
diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list
index 8f22fba3479..ed938a96d29 100644
--- a/src/tools/pgindent/typedefs.list
+++ b/src/tools/pgindent/typedefs.list
@@ -348,6 +348,7 @@ BufferManagerRelation
BufferStrategyControl
BufferTag
BufferUsage
+BuffersToWrite
BuildAccumulator
BuiltinScript
BulkInsertState
--
2.48.1.76.g4e746b1a31.dirty
[text/x-diff] v2.5-0027-Ensure-a-resowner-exists-for-all-paths-that-may.patch (2.6K, ../../vz5i2x2wkjjpp6l2z4g4l3umpxktrxsejgdwefi7n6kzw666h3@rnhlvlpadig7/28-v2.5-0027-Ensure-a-resowner-exists-for-all-paths-that-may.patch)
download | inline diff:
From bdc9e5f5a1471710e7dd49ef539fbbe4c45a7d5a Mon Sep 17 00:00:00 2001
From: Andres Freund <andres@anarazel.de>
Date: Tue, 8 Oct 2024 14:34:38 -0400
Subject: [PATCH v2.5 27/30] Ensure a resowner exists for all paths that may
perform AIO
Reviewed-by: Noah Misch <noah@leadboat.com>
Discussion: https://postgr.es/m/1f6b50a7-38ef-4d87-8246-786d39f46ab9@iki.fi
---
src/backend/bootstrap/bootstrap.c | 7 +++++++
src/backend/replication/logical/logical.c | 6 ++++++
src/backend/utils/init/postinit.c | 6 +++++-
3 files changed, 18 insertions(+), 1 deletion(-)
diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c
index 6db864892d0..e554504e1f0 100644
--- a/src/backend/bootstrap/bootstrap.c
+++ b/src/backend/bootstrap/bootstrap.c
@@ -361,8 +361,15 @@ BootstrapModeMain(int argc, char *argv[], bool check_only)
BaseInit();
bootstrap_signals();
+
+ /* need a resowner for IO during BootStrapXLOG() */
+ CreateAuxProcessResourceOwner();
+
BootStrapXLOG(bootstrap_data_checksum_version);
+ ReleaseAuxProcessResources(true);
+ CurrentResourceOwner = NULL;
+
/*
* To ensure that src/common/link-canary.c is linked into the backend, we
* must call it from somewhere. Here is as good as anywhere.
diff --git a/src/backend/replication/logical/logical.c b/src/backend/replication/logical/logical.c
index 8ea846bfc3b..2b936714040 100644
--- a/src/backend/replication/logical/logical.c
+++ b/src/backend/replication/logical/logical.c
@@ -386,6 +386,12 @@ CreateInitDecodingContext(const char *plugin,
slot->data.plugin = plugin_name;
SpinLockRelease(&slot->mutex);
+ if (CurrentResourceOwner == NULL)
+ {
+ Assert(am_walsender);
+ CurrentResourceOwner = AuxProcessResourceOwner;
+ }
+
if (XLogRecPtrIsInvalid(restart_lsn))
ReplicationSlotReserveWal();
else
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index ad1fb624e3c..8515a55dbf1 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -778,8 +778,12 @@ InitPostgres(const char *in_dbname, Oid dboid,
* We don't yet have an aux-process resource owner, but StartupXLOG
* and ShutdownXLOG will need one. Hence, create said resource owner
* (and register a callback to clean it up after ShutdownXLOG runs).
+ *
+ * In bootstrap mode CreateAuxProcessResourceOwner() was already
+ * called in BootstrapModeMain().
*/
- CreateAuxProcessResourceOwner();
+ if (!bootstrap)
+ CreateAuxProcessResourceOwner();
StartupXLOG();
/* Release (and warn about) any buffer pins leaked in StartupXLOG */
--
2.48.1.76.g4e746b1a31.dirty
[text/x-diff] v2.5-0028-Temporary-Increase-BAS_BULKREAD-size.patch (1.3K, ../../vz5i2x2wkjjpp6l2z4g4l3umpxktrxsejgdwefi7n6kzw666h3@rnhlvlpadig7/29-v2.5-0028-Temporary-Increase-BAS_BULKREAD-size.patch)
download | inline diff:
From 422c0093aff9ee1d17a70af6bd06d7e83d316629 Mon Sep 17 00:00:00 2001
From: Andres Freund <andres@anarazel.de>
Date: Sun, 1 Sep 2024 00:42:27 -0400
Subject: [PATCH v2.5 28/30] Temporary: Increase BAS_BULKREAD size
Without this we only can execute very little AIO for sequential scans, as
there's just not enough buffers in the ring. This isn't the right fix, as
just increasing the ring size can have negative performance implications in
workloads where the kernel has all the data cached.
Author:
Reviewed-By:
Discussion: https://postgr.es/m/
Backpatch:
---
src/backend/storage/buffer/freelist.c | 7 ++++++-
1 file changed, 6 insertions(+), 1 deletion(-)
diff --git a/src/backend/storage/buffer/freelist.c b/src/backend/storage/buffer/freelist.c
index 336715b6c63..b72a5957a20 100644
--- a/src/backend/storage/buffer/freelist.c
+++ b/src/backend/storage/buffer/freelist.c
@@ -555,7 +555,12 @@ GetAccessStrategy(BufferAccessStrategyType btype)
return NULL;
case BAS_BULKREAD:
- ring_size_kb = 256;
+
+ /*
+ * FIXME: Temporary increase to allow large enough streaming reads
+ * to actually benefit from AIO. This needs a better solution.
+ */
+ ring_size_kb = 2 * 1024;
break;
case BAS_BULKWRITE:
ring_size_kb = 16 * 1024;
--
2.48.1.76.g4e746b1a31.dirty
[text/x-diff] v2.5-0029-WIP-Use-MAP_POPULATE.patch (1.1K, ../../vz5i2x2wkjjpp6l2z4g4l3umpxktrxsejgdwefi7n6kzw666h3@rnhlvlpadig7/30-v2.5-0029-WIP-Use-MAP_POPULATE.patch)
download | inline diff:
From 9674f686e4dc39f2412437c0b7bc97c728bd4e1e Mon Sep 17 00:00:00 2001
From: Andres Freund <andres@anarazel.de>
Date: Tue, 31 Dec 2024 13:25:56 -0500
Subject: [PATCH v2.5 29/30] WIP: Use MAP_POPULATE
For benchmarking it's quite annoying that the first time a memory is touched
has completely different perf characteristics than subsequent accesses. Using
MAP_POPULATE reduces that substantially.
---
src/backend/port/sysv_shmem.c | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/src/backend/port/sysv_shmem.c b/src/backend/port/sysv_shmem.c
index 197926d44f6..a700b02d5a1 100644
--- a/src/backend/port/sysv_shmem.c
+++ b/src/backend/port/sysv_shmem.c
@@ -620,7 +620,7 @@ CreateAnonymousSegment(Size *size)
allocsize += hugepagesize - (allocsize % hugepagesize);
ptr = mmap(NULL, allocsize, PROT_READ | PROT_WRITE,
- PG_MMAP_FLAGS | mmap_flags, -1, 0);
+ PG_MMAP_FLAGS | MAP_POPULATE | mmap_flags, -1, 0);
mmap_errno = errno;
if (huge_pages == HUGE_PAGES_TRY && ptr == MAP_FAILED)
elog(DEBUG1, "mmap(%zu) with MAP_HUGETLB failed, huge pages disabled: %m",
--
2.48.1.76.g4e746b1a31.dirty
^ permalink raw reply [nested|flat] 153+ messages in thread
* Re: AIO v2.5
2025-03-04 19:00 AIO v2.5 Andres Freund <andres@anarazel.de>
@ 2025-03-06 11:36 ` Jakub Wartak <jakub.wartak@enterprisedb.com>
2025-03-06 13:13 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2 siblings, 1 reply; 153+ messages in thread
From: Jakub Wartak @ 2025-03-06 11:36 UTC (permalink / raw)
To: Andres Freund <andres@anarazel.de>; +Cc: pgsql-hackers@postgresql.org, Thomas Munro <thomas.munro@gmail.com>; Heikki Linnakangas <hlinnaka@iki.fi>; Noah Misch <noah@leadboat.com>; Robert Haas <robertmhaas@gmail.com>
On Tue, Mar 4, 2025 at 8:00 PM Andres Freund <andres@anarazel.de> wrote:
> Attached is v2.5 of the AIO patchset.
[..]
Hi, Thanks for working on this!
> Questions:
>
> - My current thinking is that we'd set io_method = worker initially - so we
> actually get some coverage - and then decide whether to switch to
> io_method=sync by default for 18 sometime around beta1/2. Does that sound
> reasonable?
IMHO, yes, good idea. Anyway final outcomes partially will depend on
how many other stream-consumers be committed, right?
> - Three of the commits in the series really are just precursor commits to
> their subsequent commits, which I found helpful for development and review,
> namely:
>
> - aio: Basic subsystem initialization
> - aio: Skeleton IO worker infrastructure
> - aio: Add liburing dependency
>
> Not sure if it's worth keeping these separate or whether they should just be
> merged with their "real commit".
For me it was easier to read those when they are separate.
> - Right now this series defines PGAIO_VERBOSE to 1. That's good for debugging,
> but all the ereport()s add a noticeable amount of overhead at high IO
> throughput (at multiple gigabytes/second), so that's probably not right
> forever. I'd leave this on initially and then change it to default to off
> later. I think that's ok?
+1, hopefully nothing is recording/logging/running with
log_min_messages>=debug3 because only then it starts to be visible.
> - To allow io_workers to be PGC_SIGHUP, and to eventually allow to
> automatically in/decrease active workers, the max number of workers (32) is
> always allocated. That means we use more semaphores than before. I think
> that's ok, it's not 1995 anymore. Alternatively we can add a
> "io_workers_max" GUC and probe for it in initdb.
Wouldn't that matter only on *BSDs?
BTW I somehow cannot imagine someone saturating >= 32 workers (if one
does, better to switch to uring anyway?), but I have a related
question about closing fd by those workers.
> - pg_stat_aios currently has the IO Handle flags as dedicated columns. Not
> sure that's great?
>
> They could be an enum array or such too? That'd perhaps be a bit more
> extensible? OTOH, we don't currently use enums in the catalogs and arrays
> are somewhat annoying to conjure up from C.
s/pg_stat_aios/pg_aios/ ? :^) It looks good to me as it is. Anyway it
is a debugging view - perhaps mark it as such in the docs - so there
is no stable API for that and shouldn't be queried by any software
anyway.
> - Documentation for pg_stat_aios.
pg_aios! :)
So, I've taken aio-2 branch from Your's github repo for a small ride
on legacy RHEL 8.7 with dm-flakey to inject I/O errors. This is more a
question: perhaps IO workers should auto-close fd on errors or should
we use SIGUSR2 for it? The scenario is like this:
#dm-dust is not that available even on modern distros(not always
compiled), but flakey seemed to work on 4.18.x:
losetup /dev/loop0 /dd.img
mkfs.ext4 -j /dev/loop0
mkdir /flakey
mount /dev/loop0 /flakey # for now it will work
mkdir /flakey/tblspace
chown postgres /flakey/tblspace
chmod 0700 /flakey/tblspace
CREATE TABLESPACE test1 LOCATION '/flakey/tblspace'
CREATE TABLE on t1fail on that test1 tablespace + INSERT SOME DATA
pg_ctl stop
umount /flakey
echo "0 `blockdev --getsz /dev/loop0` flakey /dev/loop0 0 1 1" |
dmsetup create flakey # after 1s start throwing IO errors
mount /dev/mapper/flakey /flakey
#might even say: mount: /flakey: can't read superblock on /dev/mapper/flakey.
mount /dev/mapper/flakey /flakey
pg_ctl start
and then this will happen:
postgres=# insert into t1fail select generate_series(1000001, 2000001);
ERROR: could not read blocks 0..1 in file
"pg_tblspc/24579/PG_18_202503031/5/24586_fsm": Input/output error
postgres=# insert into t1fail select generate_series(1000001, 2000001);
ERROR: could not read blocks 0..1 in file
"pg_tblspc/24579/PG_18_202503031/5/24586_fsm": Input/output error
postgres=# insert into t1fail select generate_series(1000001, 2000001);
ERROR: could not read blocks 0..1 in file
"pg_tblspc/24579/PG_18_202503031/5/24586_fsm": Input/output error
postgres=# insert into t1fail select generate_series(1000001, 2000001);
ERROR: could not open file
"pg_tblspc/24579/PG_18_202503031/5/24586_vm": Read-only file system
so usual stuff with kernel remounting it RO, but here's the dragon
with io_method=worker:
# mount -o remount,rw /flakey/
mount: /flakey: cannot remount /dev/mapper/flakey read-write, is
write-protected.
# umount /flakey # to fsck or just mount rw again
umount: /flakey: target is busy.
# lsof /flakey/
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
postgres 103483 postgres 14u REG 253,2 36249600 17
/flakey/tblspace/PG_18_202503031/5/24586
postgres 103484 postgres 6u REG 253,2 36249600 17
/flakey/tblspace/PG_18_202503031/5/24586
postgres 103485 postgres 6u REG 253,2 36249600 17
/flakey/tblspace/PG_18_202503031/5/24586
Those 10348[345] are IO workers, they have still open fds and there's
no way to close those without restart -- well without close()
injection probably via gdb. pg_terminate_backend() on those won't
work. The only thing that works seems to be sending SIGUSR2, but is
that safe [there could be some errors after pwrite() ] ? With
io_worker=sync just quitting the backend of course works. Not sure
what your thoughts are because any other bgworker could be having open
fds there. It's a very minor thing. Otherwise that outage of separate
tablespace (rarely used) would potentially cause inability to fsck
there and lower the availability of the DB (due to potential restart
required). I'm thinking especially of scenarios where lots of schemas
are used with lots of tablespaces OR where temp_tablespace is employed
for some dedicated (fast/furious/faulty) device. So I'm hoping SIGUSR2
is enough right (4231f4059e5e54d78c56b904f30a5873da88e163 seems to be
doing it anyway) ?
BTW: While at this, I've tried amcheck/pg_surgery for 1 min and they
both seem to work.
-J.
^ permalink raw reply [nested|flat] 153+ messages in thread
* Re: AIO v2.5
2025-03-04 19:00 AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-06 11:36 ` Re: AIO v2.5 Jakub Wartak <jakub.wartak@enterprisedb.com>
@ 2025-03-06 13:13 ` Andres Freund <andres@anarazel.de>
2025-03-07 10:21 ` Re: AIO v2.5 Jakub Wartak <jakub.wartak@enterprisedb.com>
0 siblings, 1 reply; 153+ messages in thread
From: Andres Freund @ 2025-03-06 13:13 UTC (permalink / raw)
To: Jakub Wartak <jakub.wartak@enterprisedb.com>; +Cc: pgsql-hackers@postgresql.org, Thomas Munro <thomas.munro@gmail.com>; Heikki Linnakangas <hlinnaka@iki.fi>; Noah Misch <noah@leadboat.com>; Robert Haas <robertmhaas@gmail.com>
Hi,
On 2025-03-06 12:36:43 +0100, Jakub Wartak wrote:
> On Tue, Mar 4, 2025 at 8:00 PM Andres Freund <andres@anarazel.de> wrote:
> > Questions:
> >
> > - My current thinking is that we'd set io_method = worker initially - so we
> > actually get some coverage - and then decide whether to switch to
> > io_method=sync by default for 18 sometime around beta1/2. Does that sound
> > reasonable?
>
> IMHO, yes, good idea. Anyway final outcomes partially will depend on
> how many other stream-consumers be committed, right?
I think it's more whether we find cases where it performs substantially worse
with the read stream users that exists. The behaviour for non-read-stream IO
shouldn't change.
> > - To allow io_workers to be PGC_SIGHUP, and to eventually allow to
> > automatically in/decrease active workers, the max number of workers (32) is
> > always allocated. That means we use more semaphores than before. I think
> > that's ok, it's not 1995 anymore. Alternatively we can add a
> > "io_workers_max" GUC and probe for it in initdb.
>
> Wouldn't that matter only on *BSDs?
Yea, NetBSD and OpenBSD only, I think.
> > - pg_stat_aios currently has the IO Handle flags as dedicated columns. Not
> > sure that's great?
> >
> > They could be an enum array or such too? That'd perhaps be a bit more
> > extensible? OTOH, we don't currently use enums in the catalogs and arrays
> > are somewhat annoying to conjure up from C.
>
> s/pg_stat_aios/pg_aios/ ? :^)
Ooops, yes.
> It looks good to me as it is.
> Anyway it
> is a debugging view - perhaps mark it as such in the docs - so there
> is no stable API for that and shouldn't be queried by any software
> anyway.
Cool
> > - Documentation for pg_stat_aios.
>
> pg_aios! :)
>
> So, I've taken aio-2 branch from Your's github repo for a small ride
> on legacy RHEL 8.7 with dm-flakey to inject I/O errors. This is more a
> question: perhaps IO workers should auto-close fd on errors or should
> we use SIGUSR2 for it? The scenario is like this:
When you say "auto-close", you mean that one IO error should trigger *all*
workers to close their FDs?
> so usual stuff with kernel remounting it RO, but here's the dragon
> with io_method=worker:
>
> # mount -o remount,rw /flakey/
> mount: /flakey: cannot remount /dev/mapper/flakey read-write, is
> write-protected.
> # umount /flakey # to fsck or just mount rw again
> umount: /flakey: target is busy.
> # lsof /flakey/
> COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
> postgres 103483 postgres 14u REG 253,2 36249600 17
> /flakey/tblspace/PG_18_202503031/5/24586
> postgres 103484 postgres 6u REG 253,2 36249600 17
> /flakey/tblspace/PG_18_202503031/5/24586
> postgres 103485 postgres 6u REG 253,2 36249600 17
> /flakey/tblspace/PG_18_202503031/5/24586
>
> Those 10348[345] are IO workers, they have still open fds and there's
> no way to close those without restart -- well without close()
> injection probably via gdb.
The same is already true with bgwriter, checkpointer etc?
> pg_terminate_backend() on those won't work. The only thing that works seems
> to be sending SIGUSR2
Sending SIGINT works.
> , but is that safe [there could be some errors after pwrite() ]?
Could you expand on that?
> With
> io_worker=sync just quitting the backend of course works. Not sure
> what your thoughts are because any other bgworker could be having open
> fds there. It's a very minor thing. Otherwise that outage of separate
> tablespace (rarely used) would potentially cause inability to fsck
> there and lower the availability of the DB (due to potential restart
> required).
I think a crash-restart is the only valid thing to get out of a scenario like
that, independent of AIO:
- If there had been any writes we need to perform crash recovery anyway, to
recreate those writes
- If there just were reads, it's good to restart as well, as otherwise there
might be pages in the buffer pool that don't exist on disk anymore, due to
the errors.
Greetings,
Andres Freund
^ permalink raw reply [nested|flat] 153+ messages in thread
* Re: AIO v2.5
2025-03-04 19:00 AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-06 11:36 ` Re: AIO v2.5 Jakub Wartak <jakub.wartak@enterprisedb.com>
2025-03-06 13:13 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
@ 2025-03-07 10:21 ` Jakub Wartak <jakub.wartak@enterprisedb.com>
2025-03-07 13:54 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
0 siblings, 1 reply; 153+ messages in thread
From: Jakub Wartak @ 2025-03-07 10:21 UTC (permalink / raw)
To: Andres Freund <andres@anarazel.de>; +Cc: pgsql-hackers@postgresql.org, Thomas Munro <thomas.munro@gmail.com>; Heikki Linnakangas <hlinnaka@iki.fi>; Noah Misch <noah@leadboat.com>; Robert Haas <robertmhaas@gmail.com>
On Thu, Mar 6, 2025 at 2:13 PM Andres Freund <andres@anarazel.de> wrote:
> On 2025-03-06 12:36:43 +0100, Jakub Wartak wrote:
> > On Tue, Mar 4, 2025 at 8:00 PM Andres Freund <andres@anarazel.de> wrote:
> > > Questions:
> > >
> > > - My current thinking is that we'd set io_method = worker initially - so we
> > > actually get some coverage - and then decide whether to switch to
> > > io_method=sync by default for 18 sometime around beta1/2. Does that sound
> > > reasonable?
> >
> > IMHO, yes, good idea. Anyway final outcomes partially will depend on
> > how many other stream-consumers be committed, right?
>
> I think it's more whether we find cases where it performs substantially worse
> with the read stream users that exist. The behaviour for non-read-stream IO
> shouldn't change.
OK, so in order to to get full picture for v18beta this would mean
$thread + following ones?:
- Use read streams in autoprewarm
- BitmapHeapScan table AM violation removal (and use streaming read API)
- Index Prefetching (it seems it has stalled?)
or is there something more planned? (I'm asking what to apply on top
of AIO to minimize number of potential test runs which seem to take
lots of time, so to do it all in one go)
> > So, I've taken aio-2 branch from Your's github repo for a small ride
> > on legacy RHEL 8.7 with dm-flakey to inject I/O errors. This is more a
> > question: perhaps IO workers should auto-close fd on errors or should
> > we use SIGUSR2 for it? The scenario is like this:
>
> When you say "auto-close", you mean that one IO error should trigger *all*
> workers to close their FDs?
Yeah I somehow was thinking about such a thing, but after You have
bolded that "*all*", my question sounds much more stupid than it was
yesterday. Sorry for asking stupid question :)
> The same is already true with bgwriter, checkpointer etc?
Yeah.. I was kind of looking for a way of getting "higher
availability" in the presence of partial IO (tablespace) errors.
> > pg_terminate_backend() on those won't work. The only thing that works seems
> > to be sending SIGUSR2
>
> Sending SIGINT works.
Ugh, ok, it looks like I've been overthinking that, cool.
> > , but is that safe [there could be some errors after pwrite() ]?
>
> Could you expand on that?
It is pure speculation on my side: well I'm always concerned about
leaving something out there without cleanup after errors and then
re-using it for something else much later, especially on edge-cases
like NFS or FUSE. In the backend we could maintain some state, but
io_workes are shared across backends. E.g. some pwrite() failing on
NFS, we are not closing that fd, and then reusing it for something
else much latter for different backend (although AFAIK close() does
not guarantee anything, but e.g. it could be that some inode/path or
something was simply marked dangling - the fresh pair of
close()/open() could could could return error, but here we would just
keep on pwriting() there?).
OK the only question remains: does it make sense to try something like
pgbench on NFS UDP mountopt=hard,nointr + intermittent iptables DROP
from time to time , or is it not worth trying?
> > With
> > io_worker=sync just quitting the backend of course works. Not sure
> > what your thoughts are because any other bgworker could be having open
> > fds there. It's a very minor thing. Otherwise that outage of separate
> > tablespace (rarely used) would potentially cause inability to fsck
> > there and lower the availability of the DB (due to potential restart
> > required).
>
> I think a crash-restart is the only valid thing to get out of a scenario like
> that, independent of AIO:
>
> - If there had been any writes we need to perform crash recovery anyway, to
> recreate those writes
> - If there just were reads, it's good to restart as well, as otherwise there
> might be pages in the buffer pool that don't exist on disk anymore, due to
> the errors.
OK, cool, thanks!
-J.
^ permalink raw reply [nested|flat] 153+ messages in thread
* Re: AIO v2.5
2025-03-04 19:00 AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-06 11:36 ` Re: AIO v2.5 Jakub Wartak <jakub.wartak@enterprisedb.com>
2025-03-06 13:13 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-07 10:21 ` Re: AIO v2.5 Jakub Wartak <jakub.wartak@enterprisedb.com>
@ 2025-03-07 13:54 ` Andres Freund <andres@anarazel.de>
0 siblings, 0 replies; 153+ messages in thread
From: Andres Freund @ 2025-03-07 13:54 UTC (permalink / raw)
To: Jakub Wartak <jakub.wartak@enterprisedb.com>; +Cc: pgsql-hackers@postgresql.org, Thomas Munro <thomas.munro@gmail.com>; Heikki Linnakangas <hlinnaka@iki.fi>; Noah Misch <noah@leadboat.com>; Robert Haas <robertmhaas@gmail.com>
Hi,
On 2025-03-07 11:21:09 +0100, Jakub Wartak wrote:
> On Thu, Mar 6, 2025 at 2:13 PM Andres Freund <andres@anarazel.de> wrote:
>
> > On 2025-03-06 12:36:43 +0100, Jakub Wartak wrote:
> > > On Tue, Mar 4, 2025 at 8:00 PM Andres Freund <andres@anarazel.de> wrote:
> > > > Questions:
> > > >
> > > > - My current thinking is that we'd set io_method = worker initially - so we
> > > > actually get some coverage - and then decide whether to switch to
> > > > io_method=sync by default for 18 sometime around beta1/2. Does that sound
> > > > reasonable?
> > >
> > > IMHO, yes, good idea. Anyway final outcomes partially will depend on
> > > how many other stream-consumers be committed, right?
> >
> > I think it's more whether we find cases where it performs substantially worse
> > with the read stream users that exist. The behaviour for non-read-stream IO
> > shouldn't change.
>
> OK, so in order to to get full picture for v18beta this would mean
> $thread + following ones?:
> - Use read streams in autoprewarm
> - BitmapHeapScan table AM violation removal (and use streaming read API)
Yep.
> - Index Prefetching (it seems it has stalled?)
I don't think there's any chance it'll be in 18. There's a good bit more work
needed before it can go in...
> or is there something more planned? (I'm asking what to apply on top
> of AIO to minimize number of potential test runs which seem to take
> lots of time, so to do it all in one go)
I think there may be some more (e.g. btree index vacuuming), but I don't think
they'll have *that* big an impact.
> > > So, I've taken aio-2 branch from Your's github repo for a small ride
> > > on legacy RHEL 8.7 with dm-flakey to inject I/O errors. This is more a
> > > question: perhaps IO workers should auto-close fd on errors or should
> > > we use SIGUSR2 for it? The scenario is like this:
> >
> > When you say "auto-close", you mean that one IO error should trigger *all*
> > workers to close their FDs?
>
> Yeah I somehow was thinking about such a thing, but after You have
> bolded that "*all*", my question sounds much more stupid than it was
> yesterday. Sorry for asking stupid question :)
Don't worry about that :)
> > The same is already true with bgwriter, checkpointer etc?
>
> Yeah.. I was kind of looking for a way of getting "higher
> availability" in the presence of partial IO (tablespace) errors.
I'm really doubtful that's that worthwhile to pursue. IME the system is pretty
much hosed once this starts to happening and it's often made *worse* by trying
to limp along.
> OK the only question remains: does it make sense to try something like
> pgbench on NFS UDP mountopt=hard,nointr + intermittent iptables DROP
> from time to time , or is it not worth trying?
I don't think it's particularly interesting. But then I'd *never* trust any
meaningful data to a PG running on NFS.
Greetings,
Andres Freund
^ permalink raw reply [nested|flat] 153+ messages in thread
* Re: AIO v2.5
2025-03-04 19:00 AIO v2.5 Andres Freund <andres@anarazel.de>
@ 2025-03-06 15:33 ` Robert Haas <robertmhaas@gmail.com>
2025-03-06 16:53 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2 siblings, 1 reply; 153+ messages in thread
From: Robert Haas @ 2025-03-06 15:33 UTC (permalink / raw)
To: Andres Freund <andres@anarazel.de>; +Cc: pgsql-hackers@postgresql.org, Thomas Munro <thomas.munro@gmail.com>; Heikki Linnakangas <hlinnaka@iki.fi>; Noah Misch <noah@leadboat.com>; Jakub Wartak <jakub.wartak@enterprisedb.com>
On Tue, Mar 4, 2025 at 2:00 PM Andres Freund <andres@anarazel.de> wrote:
> - pg_stat_aios currently has the IO Handle flags as dedicated columns. Not
> sure that's great?
I don't like the name. Pluralization abbreviations is weird, and it's
even weirder when the abbreviation is not one that is universally
known. Maybe just drop the "s".
--
Robert Haas
EDB: http://www.enterprisedb.com
^ permalink raw reply [nested|flat] 153+ messages in thread
* Re: AIO v2.5
2025-03-04 19:00 AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-06 15:33 ` Re: AIO v2.5 Robert Haas <robertmhaas@gmail.com>
@ 2025-03-06 16:53 ` Andres Freund <andres@anarazel.de>
2025-03-07 21:23 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
0 siblings, 1 reply; 153+ messages in thread
From: Andres Freund @ 2025-03-06 16:53 UTC (permalink / raw)
To: Robert Haas <robertmhaas@gmail.com>; +Cc: pgsql-hackers@postgresql.org, Thomas Munro <thomas.munro@gmail.com>; Heikki Linnakangas <hlinnaka@iki.fi>; Noah Misch <noah@leadboat.com>; Jakub Wartak <jakub.wartak@enterprisedb.com>
Hi,
On 2025-03-06 10:33:33 -0500, Robert Haas wrote:
> On Tue, Mar 4, 2025 at 2:00 PM Andres Freund <andres@anarazel.de> wrote:
> > - pg_stat_aios currently has the IO Handle flags as dedicated columns. Not
> > sure that's great?
>
> I don't like the name.
I don't think it changes anything, but as Jakub pointed out, I thinko'd the
name in the email you're responding to, it's pg_aios, not pg_stat_aios.
It shows the currently in-flight IOs, not accumulated statistics about them,
hence no _stat_.
I don't like the name either, I IIRC asked for suggestions elsewhere in the
thread, not a lot was forthcoming, so I left it at pg_aios.
> Pluralization abbreviations is weird, and it's even weirder when the
> abbreviation is not one that is universally known. Maybe just drop the "s".
I went with plural because that's what we have in other views showing the
"current" state:
- pg_cursors
- pg_file_settings
- pg_prepared_statements
- pg_prepared_xacts
- pg_replication_slots
- pg_locks
- ...
But you're right that those aren't abbreviations.
Greetings,
Andres Freund
^ permalink raw reply [nested|flat] 153+ messages in thread
* Re: AIO v2.5
2025-03-04 19:00 AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-06 15:33 ` Re: AIO v2.5 Robert Haas <robertmhaas@gmail.com>
2025-03-06 16:53 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
@ 2025-03-07 21:23 ` Andres Freund <andres@anarazel.de>
2025-03-08 02:11 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
0 siblings, 1 reply; 153+ messages in thread
From: Andres Freund @ 2025-03-07 21:23 UTC (permalink / raw)
To: Robert Haas <robertmhaas@gmail.com>; +Cc: pgsql-hackers@postgresql.org, Thomas Munro <thomas.munro@gmail.com>; Heikki Linnakangas <hlinnaka@iki.fi>; Noah Misch <noah@leadboat.com>; Jakub Wartak <jakub.wartak@enterprisedb.com>
Hi,
On 2025-03-06 11:53:41 -0500, Andres Freund wrote:
> On 2025-03-06 10:33:33 -0500, Robert Haas wrote:
> > On Tue, Mar 4, 2025 at 2:00 PM Andres Freund <andres@anarazel.de> wrote:
> > > - pg_stat_aios currently has the IO Handle flags as dedicated columns. Not
> > > sure that's great?
> >
> > I don't like the name.
>
> I don't think it changes anything, but as Jakub pointed out, I thinko'd the
> name in the email you're responding to, it's pg_aios, not pg_stat_aios.
>
> It shows the currently in-flight IOs, not accumulated statistics about them,
> hence no _stat_.
>
> I don't like the name either, I IIRC asked for suggestions elsewhere in the
> thread, not a lot was forthcoming, so I left it at pg_aios.
What about pg_io_handles?
Greetings,
Andres Freund
^ permalink raw reply [nested|flat] 153+ messages in thread
* Re: AIO v2.5
2025-03-04 19:00 AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-06 15:33 ` Re: AIO v2.5 Robert Haas <robertmhaas@gmail.com>
2025-03-06 16:53 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-07 21:23 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
@ 2025-03-08 02:11 ` Andres Freund <andres@anarazel.de>
2025-03-09 17:51 ` Re: AIO v2.5 Tom Lane <tgl@sss.pgh.pa.us>
0 siblings, 1 reply; 153+ messages in thread
From: Andres Freund @ 2025-03-08 02:11 UTC (permalink / raw)
To: Robert Haas <robertmhaas@gmail.com>; Tom Lane <tgl@sss.pgh.pa.us>; +Cc: pgsql-hackers@postgresql.org, Thomas Munro <thomas.munro@gmail.com>; Heikki Linnakangas <hlinnaka@iki.fi>; Noah Misch <noah@leadboat.com>; Jakub Wartak <jakub.wartak@enterprisedb.com>
Hi,
Tom, CCed you since you have worked most on elog.c
On 2025-03-07 16:23:51 -0500, Andres Freund wrote:
> What about pg_io_handles?
While looking at the view I felt motivated to tackle the one FIXME in the
implementation of the view. Namely that the "error_desc" column wasn't
populated (the view did show that there was an error, but not what the error
was).
Which lead me down a sad sad rabbit hole, largely independent of AIO.
A bit of background:
For AIO completion callbacks can signal errors (e.g. a page header failing
validation). That error can be logged in the callback and/or raised later,
e.g. by the query that issued the IO.
AIO callbacks happen in critical sections, which is required to be able to use
AIO for WAL (see README.md for more details).
Currently errors are logged/raised by ereport()s in functions that gets passed
in an elevel, pretty standard.
A few of the ereports() use errcode_for_file_access() to translate an errno to
an sqlerrcode.
Now on to the problem:
The result of an ereport() can't be put into a view, obviously. I didn't think
it'd be good if the each kind of error needed to be implemented twice, once
with ereport() and once to just return a string to put in the view.
I tried a few things:
1) Use errsave() to allow delayed reporting of the error
I encountered a few problems:
- errsave() doesn't allow the log level to be specified, which means it can't
directly be used to LOG if no context is specified.
This could be worked around by always specifying the context, with
ErrorSaveContext.details_wanted = true and having generic code that changes
the elevel to whatever is appropriate and then using ThrowErrorData() to log the
message.
- ersave_start() sets assoc_context to CurrentMemoryContext and
errsave_finish() allocates an ErrorData copy in CurrentMemoryContext
This makes naive use of this approach when logging in a critical section
impossible. If ErrorSaveContext is not passed in an ERROR will be raised,
even if we just want to log. If ErrorSaveContext is used, we allocate
memory in the caller context, which isn't allowed in a critical section.
The only way I saw to work around that was to switch to ErrorContext before
calling errsave(). That's doable, the logging is called from one function
(pgaio_result_report()). That kinda works, but as a consequence we more than
double the memory usage in ErrorContext as errsave_finish() will palloc a
new ErrorData and ThrowErrorData() copies that ErrorData and all its string
back to ErrorContext.
2) Have the error callback format the error using a helper function instead of
using ereport()
Problems:
- errcode_for_file_access() would need to be reimplemented / split into a
function translating an errnode into an sqlerrcode without getting it from
the error data stack
- emitting the log message in a critical section would require either doing
the error formatting in ErrorContext or creating another context with
reserved memory to do so.
- allowing to specify DETAIL, HINT etc basically requires a small elog.c
interface reimplementation
3) Use pre_format_elog_string(), format_elog_string() similar to what guc.c
does for check hooks, via GUC_check_errmsg(), GUC_check_errhint() ...
Problems:
- Requires to duplicate errcode_for_file_access() for similar reason as in 2)
- Not exactly pretty
- Somewhat gnarly, but doable, to make use of %m safe, the way it's done in
guc.h afaict isn't safe:
pre_format_elog_string() is called for each of
GUC_check_{errmsg,errdetail,errhint}. As the global errno might get set
during the format_elog_string(), it'll not be the right one during
the next GUC_check_*.
4) Don't use ereport() directly, but instead put the errstart() in
pgaio_result_report(), before calling the error description callback.
When emitting a log message, call errfinish() after the callback. For the
view, get the message out via CopyErrorData() and free the memory again
using FlushErrorState
Problems:
- Seems extremely hacky
I implemented all, but don't really like any of them.
Unless somebody has a better idea or we agree that one of the above is
actually a acceptable approach, I'm inclined to simply remove the column
containing the description of the error. The window in which one could see an
IO with an error is rather short most of the time anyway and the error will
also be logged.
It's a bit annoying that adding the column later would require revising the
signature of the error reporting callback at that time, but I think that
degree of churn is acceptable.
The main reason I wanted to write this up is that it seems that we're just
lacking some infrastructure here.
Greetings,
Andres Freund
^ permalink raw reply [nested|flat] 153+ messages in thread
* Re: AIO v2.5
2025-03-04 19:00 AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-06 15:33 ` Re: AIO v2.5 Robert Haas <robertmhaas@gmail.com>
2025-03-06 16:53 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-07 21:23 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-08 02:11 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
@ 2025-03-09 17:51 ` Tom Lane <tgl@sss.pgh.pa.us>
0 siblings, 0 replies; 153+ messages in thread
From: Tom Lane @ 2025-03-09 17:51 UTC (permalink / raw)
To: Andres Freund <andres@anarazel.de>; +Cc: Robert Haas <robertmhaas@gmail.com>; pgsql-hackers@postgresql.org, Thomas Munro <thomas.munro@gmail.com>; Heikki Linnakangas <hlinnaka@iki.fi>; Noah Misch <noah@leadboat.com>; Jakub Wartak <jakub.wartak@enterprisedb.com>
Andres Freund <andres@anarazel.de> writes:
> While looking at the view I felt motivated to tackle the one FIXME in the
> implementation of the view. Namely that the "error_desc" column wasn't
> populated (the view did show that there was an error, but not what the error
> was).
> Which lead me down a sad sad rabbit hole, largely independent of AIO.
> ...
> The main reason I wanted to write this up is that it seems that we're just
> lacking some infrastructure here.
Maybe. The mention of elog.c in the same breath with critical
sections is already enough to scare me; we surely daren't invoke
gettext() in a critical section, for instance. I feel the most
we could hope for here is to report a constant string that would
not get translated till later, outside the critical section.
That seems less about infrastructure and more about how the AIO
error handling/reporting code is laid out. In the meantime,
if leaving the error out of this view is enough to make the problem
go away, let's do that.
regards, tom lane
^ permalink raw reply [nested|flat] 153+ messages in thread
* Re: AIO v2.5
2025-03-04 19:00 AIO v2.5 Andres Freund <andres@anarazel.de>
@ 2025-03-10 18:23 ` Andres Freund <andres@anarazel.de>
2025-03-11 15:31 ` Re: AIO v2.5 Melanie Plageman <melanieplageman@gmail.com>
2025-03-11 19:41 ` Re: AIO v2.5 Noah Misch <noah@leadboat.com>
2 siblings, 2 replies; 153+ messages in thread
From: Andres Freund @ 2025-03-10 18:23 UTC (permalink / raw)
To: pgsql-hackers; +Cc: Thomas Munro <thomas.munro@gmail.com>; Heikki Linnakangas <hlinnaka@iki.fi>; Noah Misch <noah@leadboat.com>; Robert Haas <robertmhaas@gmail.com>; Jakub Wartak <jakub.wartak@enterprisedb.com>
Hi,
Attached is v2.6 of the AIO patchset.
Relative to 2.5 I:
- Improved the split between subsystem initialization and main AIO commit, as
well as the one between worker infrastructure and io_method=worker
Seemed worth as the only one voicing an opinion about squashing those
commits was opposed.
- Added a lot more comments to aio.h/aio_internal.h. I think just about
anything that should conceivably have a comment has one.
- Reordered fields in PgAioHandle to waste less due to padding
- Narrowed a few *count fields, they were 64bit without ever being al to reach
that
- Used aio_types.h more widely, instead of "manual" forward declarations. This
required moving a few typedefs to aio_types.h
- Substantial commit message improvements.
- Removed the pg_aios.error_desc column, due to:
https://postgr.es/m/qzxq6mqqozctlfcg2kg5744gmyubicvuehnp4a7up472thlvz2%40y5xqgd5wcwhw
- Reordered the commits slightly, to put the README just after the
smgr.c/md.c/... support, as the readme references those in the examples
- Stopped creating backend-local io_uring instances, that is vestigial for
now. We likely will want to reintroduce them at some point (e.g. for network
IO), but we can do that at that time.
- There were a lot of duplicated codepaths in bufmgr.c support for AIO due to
temp tables. I added a few commits refactoring the temp buffers state
management to look a lot more like the shared buffer code.
I'm not sure that that's the best path, but they all seemed substantial
improvements on their own.
- putting io_method in PG_TEST_INITDB_EXTRA_OPTS previously broke a test,
because Cluster::init() puts PG_TEST_INITDB_EXTRA_OPTS after the # options
specified by ->extra. I now worked around that by appending the io method to
a local PG_TEST_INITDB_EXTRA_OPTS, but brrr.
- The tracepoint for read completion omitted the fact that it was a temp
table, if so.
- Fixed some duplicated function decls, due to a misresolved merge-conflict
Current state:
- 0001, 0002 - core AIO - IMO pretty much ready
- 0003, 0004 - IO worker - same
- 0005, 0006 - io_uring support - close, but we need to do something about
set_max_fds(), which errors out spuriously in some cases
- 0007 - smgr/md/fd.c readv support - seems quite close, but might benefit from
another pass through
- 0008 - README - I think it's good, but I'm probably not seeing the trees for
the forest anymore
- 0009 - pg_aios view - naming not resolved, docs missing
- 0010 to 0014 - from another thread, just included here due to a dependency
- 0016 to 0020 - cleanups for temp buffers code - I just wrote these to clean
up the code before making larger changes, needs review
- 0021 - keep BufferDesc refcount up2date for temp buffers - I think that's
pretty much ready, but depends on earlier patches
- 0022 - bufmgr readv AIO suppot - some naming, some code duplication needs to
be resolved, but otherwise quite close
- 0023 - use AIO in StartReadBuffers() - perhaps a bit of polishing needed
- 0024 - adjust read_stream.c for AIO - I think Thomas has a better patch for
this in the works
- 0025 - tests for AIO - I think it's reasonable, unless somebody objects to
exporting a few bufmgr.c functions to the test
- the rest: Not for 18
Greetings,
Andres Freund
Attachments:
[text/x-diff] v2.6-0001-aio-Basic-subsystem-initialization.patch (25.1K, ../../ge6nsuddurhpmll3xj22vucvqwp4agqz6ndtcf2mhyeydzarst@l75dman5x53p/2-v2.6-0001-aio-Basic-subsystem-initialization.patch)
download | inline diff:
From 49f5873485e478672b70362b698b6d44c22f4c16 Mon Sep 17 00:00:00 2001
From: Andres Freund <andres@anarazel.de>
Date: Tue, 4 Mar 2025 10:44:56 -0500
Subject: [PATCH v2.6 01/34] aio: Basic subsystem initialization
This commit just does the minimal wiring up of the AIO subsystem, to be fully
added in the next commit, to the rest of the system. The next commit contains
more details about motivation and architecture.
This commit is kept separate to make it easier to review, reviewing how the
AIO subsystem works is a big enough task on its own an rather separate from
reviewing the changes across the tree.
We discussed squashing this commit with the main commit before merging AIO,
but there has been a mild preference for keeping it separate.
Discussion: https://postgr.es/m/uvrtrknj4kdytuboidbhwclo4gxhswwcpgadptsjvjqcluzmah%40brqs62irg4dt
---
src/include/storage/aio.h | 38 ++++++++
src/include/storage/aio_subsys.h | 33 +++++++
src/include/utils/guc.h | 1 +
src/include/utils/guc_hooks.h | 2 +
src/include/utils/resowner.h | 5 ++
src/backend/access/transam/xact.c | 12 +++
src/backend/postmaster/autovacuum.c | 2 +
src/backend/postmaster/bgwriter.c | 2 +
src/backend/postmaster/checkpointer.c | 2 +
src/backend/postmaster/pgarch.c | 2 +
src/backend/postmaster/walsummarizer.c | 2 +
src/backend/postmaster/walwriter.c | 2 +
src/backend/replication/walsender.c | 2 +
src/backend/storage/aio/Makefile | 2 +
src/backend/storage/aio/aio.c | 90 +++++++++++++++++++
src/backend/storage/aio/aio_init.c | 37 ++++++++
src/backend/storage/aio/meson.build | 2 +
src/backend/storage/ipc/ipci.c | 3 +
src/backend/utils/init/postinit.c | 7 ++
src/backend/utils/misc/guc_tables.c | 23 +++++
src/backend/utils/misc/postgresql.conf.sample | 6 ++
src/backend/utils/resowner/resowner.c | 29 ++++++
doc/src/sgml/config.sgml | 51 +++++++++++
src/tools/pgindent/typedefs.list | 1 +
24 files changed, 356 insertions(+)
create mode 100644 src/include/storage/aio.h
create mode 100644 src/include/storage/aio_subsys.h
create mode 100644 src/backend/storage/aio/aio.c
create mode 100644 src/backend/storage/aio/aio_init.c
diff --git a/src/include/storage/aio.h b/src/include/storage/aio.h
new file mode 100644
index 00000000000..e79d5343038
--- /dev/null
+++ b/src/include/storage/aio.h
@@ -0,0 +1,38 @@
+/*-------------------------------------------------------------------------
+ *
+ * aio.h
+ * Main AIO interface
+ *
+ *
+ * Portions Copyright (c) 1996-2025, PostgreSQL Global Development Group
+ * Portions Copyright (c) 1994, Regents of the University of California
+ *
+ * src/include/storage/aio.h
+ *
+ *-------------------------------------------------------------------------
+ */
+#ifndef AIO_H
+#define AIO_H
+
+
+
+/* Enum for io_method GUC. */
+typedef enum IoMethod
+{
+ IOMETHOD_SYNC = 0,
+} IoMethod;
+
+/* We'll default to synchronous execution. */
+#define DEFAULT_IO_METHOD IOMETHOD_SYNC
+
+
+struct dlist_node;
+extern void pgaio_io_release_resowner(struct dlist_node *ioh_node, bool on_error);
+
+
+/* GUCs */
+extern PGDLLIMPORT int io_method;
+extern PGDLLIMPORT int io_max_concurrency;
+
+
+#endif /* AIO_H */
diff --git a/src/include/storage/aio_subsys.h b/src/include/storage/aio_subsys.h
new file mode 100644
index 00000000000..e4faf692a38
--- /dev/null
+++ b/src/include/storage/aio_subsys.h
@@ -0,0 +1,33 @@
+/*-------------------------------------------------------------------------
+ *
+ * aio_subsys.h
+ * Interaction with AIO as a subsystem, rather than actually issuing AIO
+ *
+ * This header is for AIO related functionality that's being called by files
+ * that don't perform AIO but interact with it in some form. E.g. postmaster.c
+ * and shared memory initialization need to initialize AIO but don't perform
+ * AIO.
+ *
+ * Portions Copyright (c) 1996-2025, PostgreSQL Global Development Group
+ * Portions Copyright (c) 1994, Regents of the University of California
+ *
+ * src/include/storage/aio_subsys.h
+ *
+ *-------------------------------------------------------------------------
+ */
+#ifndef AIO_SUBSYS_H
+#define AIO_SUBSYS_H
+
+
+/* aio_init.c */
+extern Size AioShmemSize(void);
+extern void AioShmemInit(void);
+
+extern void pgaio_init_backend(void);
+
+
+/* aio.c */
+extern void pgaio_error_cleanup(void);
+extern void AtEOXact_Aio(bool is_commit);
+
+#endif /* AIO_SUBSYS_H */
diff --git a/src/include/utils/guc.h b/src/include/utils/guc.h
index 1233e07d7da..de3bc37264f 100644
--- a/src/include/utils/guc.h
+++ b/src/include/utils/guc.h
@@ -318,6 +318,7 @@ extern PGDLLIMPORT bool optimize_bounded_sort;
*/
extern PGDLLIMPORT const struct config_enum_entry archive_mode_options[];
extern PGDLLIMPORT const struct config_enum_entry dynamic_shared_memory_options[];
+extern PGDLLIMPORT const struct config_enum_entry io_method_options[];
extern PGDLLIMPORT const struct config_enum_entry recovery_target_action_options[];
extern PGDLLIMPORT const struct config_enum_entry wal_level_options[];
extern PGDLLIMPORT const struct config_enum_entry wal_sync_method_options[];
diff --git a/src/include/utils/guc_hooks.h b/src/include/utils/guc_hooks.h
index 951451a9765..d1b49adc547 100644
--- a/src/include/utils/guc_hooks.h
+++ b/src/include/utils/guc_hooks.h
@@ -62,6 +62,8 @@ extern bool check_default_with_oids(bool *newval, void **extra,
extern bool check_effective_io_concurrency(int *newval, void **extra,
GucSource source);
extern bool check_huge_page_size(int *newval, void **extra, GucSource source);
+extern void assign_io_method(int newval, void *extra);
+extern bool check_io_max_concurrency(int *newval, void **extra, GucSource source);
extern const char *show_in_hot_standby(void);
extern bool check_locale_messages(char **newval, void **extra, GucSource source);
extern void assign_locale_messages(const char *newval, void *extra);
diff --git a/src/include/utils/resowner.h b/src/include/utils/resowner.h
index e8d452ca7ee..aede4bfc820 100644
--- a/src/include/utils/resowner.h
+++ b/src/include/utils/resowner.h
@@ -164,4 +164,9 @@ struct LOCALLOCK;
extern void ResourceOwnerRememberLock(ResourceOwner owner, struct LOCALLOCK *locallock);
extern void ResourceOwnerForgetLock(ResourceOwner owner, struct LOCALLOCK *locallock);
+/* special support for AIO */
+struct dlist_node;
+extern void ResourceOwnerRememberAioHandle(ResourceOwner owner, struct dlist_node *ioh_node);
+extern void ResourceOwnerForgetAioHandle(ResourceOwner owner, struct dlist_node *ioh_node);
+
#endif /* RESOWNER_H */
diff --git a/src/backend/access/transam/xact.c b/src/backend/access/transam/xact.c
index 1b4f21a88d3..b885513f765 100644
--- a/src/backend/access/transam/xact.c
+++ b/src/backend/access/transam/xact.c
@@ -51,6 +51,7 @@
#include "replication/origin.h"
#include "replication/snapbuild.h"
#include "replication/syncrep.h"
+#include "storage/aio_subsys.h"
#include "storage/condition_variable.h"
#include "storage/fd.h"
#include "storage/lmgr.h"
@@ -2411,6 +2412,8 @@ CommitTransaction(void)
RESOURCE_RELEASE_BEFORE_LOCKS,
true, true);
+ AtEOXact_Aio(true);
+
/* Check we've released all buffer pins */
AtEOXact_Buffers(true);
@@ -2716,6 +2719,8 @@ PrepareTransaction(void)
RESOURCE_RELEASE_BEFORE_LOCKS,
true, true);
+ AtEOXact_Aio(true);
+
/* Check we've released all buffer pins */
AtEOXact_Buffers(true);
@@ -2830,6 +2835,8 @@ AbortTransaction(void)
pgstat_report_wait_end();
pgstat_progress_end_command();
+ pgaio_error_cleanup();
+
/* Clean up buffer content locks, too */
UnlockBuffers();
@@ -2960,6 +2967,7 @@ AbortTransaction(void)
ResourceOwnerRelease(TopTransactionResourceOwner,
RESOURCE_RELEASE_BEFORE_LOCKS,
false, true);
+ AtEOXact_Aio(false);
AtEOXact_Buffers(false);
AtEOXact_RelationCache(false);
AtEOXact_TypeCache();
@@ -5232,6 +5240,9 @@ AbortSubTransaction(void)
pgstat_report_wait_end();
pgstat_progress_end_command();
+
+ pgaio_error_cleanup();
+
UnlockBuffers();
/* Reset WAL record construction state */
@@ -5326,6 +5337,7 @@ AbortSubTransaction(void)
RESOURCE_RELEASE_BEFORE_LOCKS,
false, false);
+ AtEOXact_Aio(false);
AtEOSubXact_RelationCache(false, s->subTransactionId,
s->parent->subTransactionId);
AtEOSubXact_TypeCache();
diff --git a/src/backend/postmaster/autovacuum.c b/src/backend/postmaster/autovacuum.c
index 800815dfbcc..5aa0fa665c2 100644
--- a/src/backend/postmaster/autovacuum.c
+++ b/src/backend/postmaster/autovacuum.c
@@ -88,6 +88,7 @@
#include "postmaster/autovacuum.h"
#include "postmaster/interrupt.h"
#include "postmaster/postmaster.h"
+#include "storage/aio_subsys.h"
#include "storage/bufmgr.h"
#include "storage/ipc.h"
#include "storage/latch.h"
@@ -465,6 +466,7 @@ AutoVacLauncherMain(const void *startup_data, size_t startup_data_len)
*/
LWLockReleaseAll();
pgstat_report_wait_end();
+ pgaio_error_cleanup();
UnlockBuffers();
/* this is probably dead code, but let's be safe: */
if (AuxProcessResourceOwner)
diff --git a/src/backend/postmaster/bgwriter.c b/src/backend/postmaster/bgwriter.c
index a688cc5d2a1..72f5acceec7 100644
--- a/src/backend/postmaster/bgwriter.c
+++ b/src/backend/postmaster/bgwriter.c
@@ -38,6 +38,7 @@
#include "postmaster/auxprocess.h"
#include "postmaster/bgwriter.h"
#include "postmaster/interrupt.h"
+#include "storage/aio_subsys.h"
#include "storage/buf_internals.h"
#include "storage/bufmgr.h"
#include "storage/condition_variable.h"
@@ -168,6 +169,7 @@ BackgroundWriterMain(const void *startup_data, size_t startup_data_len)
*/
LWLockReleaseAll();
ConditionVariableCancelSleep();
+ pgaio_error_cleanup();
UnlockBuffers();
ReleaseAuxProcessResources(false);
AtEOXact_Buffers(false);
diff --git a/src/backend/postmaster/checkpointer.c b/src/backend/postmaster/checkpointer.c
index 0e228d143a0..fda91ffd1ce 100644
--- a/src/backend/postmaster/checkpointer.c
+++ b/src/backend/postmaster/checkpointer.c
@@ -49,6 +49,7 @@
#include "postmaster/bgwriter.h"
#include "postmaster/interrupt.h"
#include "replication/syncrep.h"
+#include "storage/aio_subsys.h"
#include "storage/bufmgr.h"
#include "storage/condition_variable.h"
#include "storage/fd.h"
@@ -276,6 +277,7 @@ CheckpointerMain(const void *startup_data, size_t startup_data_len)
LWLockReleaseAll();
ConditionVariableCancelSleep();
pgstat_report_wait_end();
+ pgaio_error_cleanup();
UnlockBuffers();
ReleaseAuxProcessResources(false);
AtEOXact_Buffers(false);
diff --git a/src/backend/postmaster/pgarch.c b/src/backend/postmaster/pgarch.c
index dbe4e1d426b..7e622ae4bd2 100644
--- a/src/backend/postmaster/pgarch.c
+++ b/src/backend/postmaster/pgarch.c
@@ -40,6 +40,7 @@
#include "postmaster/interrupt.h"
#include "postmaster/pgarch.h"
#include "storage/condition_variable.h"
+#include "storage/aio_subsys.h"
#include "storage/fd.h"
#include "storage/ipc.h"
#include "storage/latch.h"
@@ -568,6 +569,7 @@ pgarch_archiveXlog(char *xlog)
LWLockReleaseAll();
ConditionVariableCancelSleep();
pgstat_report_wait_end();
+ pgaio_error_cleanup();
ReleaseAuxProcessResources(false);
AtEOXact_Files(false);
AtEOXact_HashTables(false);
diff --git a/src/backend/postmaster/walsummarizer.c b/src/backend/postmaster/walsummarizer.c
index ccba0f84e6e..0fec4f1f871 100644
--- a/src/backend/postmaster/walsummarizer.c
+++ b/src/backend/postmaster/walsummarizer.c
@@ -38,6 +38,7 @@
#include "postmaster/interrupt.h"
#include "postmaster/walsummarizer.h"
#include "replication/walreceiver.h"
+#include "storage/aio_subsys.h"
#include "storage/fd.h"
#include "storage/ipc.h"
#include "storage/latch.h"
@@ -289,6 +290,7 @@ WalSummarizerMain(const void *startup_data, size_t startup_data_len)
LWLockReleaseAll();
ConditionVariableCancelSleep();
pgstat_report_wait_end();
+ pgaio_error_cleanup();
ReleaseAuxProcessResources(false);
AtEOXact_Files(false);
AtEOXact_HashTables(false);
diff --git a/src/backend/postmaster/walwriter.c b/src/backend/postmaster/walwriter.c
index 0380601bcbb..fd92c8b7a33 100644
--- a/src/backend/postmaster/walwriter.c
+++ b/src/backend/postmaster/walwriter.c
@@ -51,6 +51,7 @@
#include "postmaster/auxprocess.h"
#include "postmaster/interrupt.h"
#include "postmaster/walwriter.h"
+#include "storage/aio_subsys.h"
#include "storage/bufmgr.h"
#include "storage/condition_variable.h"
#include "storage/fd.h"
@@ -164,6 +165,7 @@ WalWriterMain(const void *startup_data, size_t startup_data_len)
LWLockReleaseAll();
ConditionVariableCancelSleep();
pgstat_report_wait_end();
+ pgaio_error_cleanup();
UnlockBuffers();
ReleaseAuxProcessResources(false);
AtEOXact_Buffers(false);
diff --git a/src/backend/replication/walsender.c b/src/backend/replication/walsender.c
index 446d10c1a7d..69097125606 100644
--- a/src/backend/replication/walsender.c
+++ b/src/backend/replication/walsender.c
@@ -79,6 +79,7 @@
#include "replication/walsender.h"
#include "replication/walsender_private.h"
#include "storage/condition_variable.h"
+#include "storage/aio_subsys.h"
#include "storage/fd.h"
#include "storage/ipc.h"
#include "storage/pmsignal.h"
@@ -327,6 +328,7 @@ WalSndErrorCleanup(void)
LWLockReleaseAll();
ConditionVariableCancelSleep();
pgstat_report_wait_end();
+ pgaio_error_cleanup();
if (xlogreader != NULL && xlogreader->seg.ws_file >= 0)
wal_segment_close(xlogreader);
diff --git a/src/backend/storage/aio/Makefile b/src/backend/storage/aio/Makefile
index 2f29a9ec4d1..eaeaeeee8e3 100644
--- a/src/backend/storage/aio/Makefile
+++ b/src/backend/storage/aio/Makefile
@@ -9,6 +9,8 @@ top_builddir = ../../../..
include $(top_builddir)/src/Makefile.global
OBJS = \
+ aio.o \
+ aio_init.o \
read_stream.o
include $(top_srcdir)/src/backend/common.mk
diff --git a/src/backend/storage/aio/aio.c b/src/backend/storage/aio/aio.c
new file mode 100644
index 00000000000..828a94efdc3
--- /dev/null
+++ b/src/backend/storage/aio/aio.c
@@ -0,0 +1,90 @@
+/*-------------------------------------------------------------------------
+ *
+ * aio.c
+ * AIO - Core Logic
+ *
+ * Portions Copyright (c) 1996-2025, PostgreSQL Global Development Group
+ * Portions Copyright (c) 1994, Regents of the University of California
+ *
+ * IDENTIFICATION
+ * src/backend/storage/aio/aio.c
+ *
+ *-------------------------------------------------------------------------
+ */
+
+#include "postgres.h"
+
+#include "lib/ilist.h"
+#include "storage/aio.h"
+#include "storage/aio_subsys.h"
+#include "utils/guc.h"
+#include "utils/guc_hooks.h"
+
+
+/* Options for io_method. */
+const struct config_enum_entry io_method_options[] = {
+ {"sync", IOMETHOD_SYNC, false},
+ {NULL, 0, false}
+};
+
+/* GUCs */
+int io_method = DEFAULT_IO_METHOD;
+int io_max_concurrency = -1;
+
+
+
+/*
+ * Release IO handle during resource owner cleanup.
+ */
+void
+pgaio_io_release_resowner(dlist_node *ioh_node, bool on_error)
+{
+}
+
+/*
+ * Perform AIO related cleanup after an error.
+ *
+ * This should be called early in the error recovery paths, as later steps may
+ * need to issue AIO (e.g. to record a transaction abort WAL record).
+ */
+void
+pgaio_error_cleanup(void)
+{
+}
+
+/*
+ * Perform AIO related checks at (sub-)transactional boundaries.
+ *
+ * This should be called late during (sub-)transactional commit/abort, after
+ * all steps that might need to perform AIO, so that we can verify that the
+ * AIO subsystem is in a valid state at the end of a transaction.
+ */
+void
+AtEOXact_Aio(bool is_commit)
+{
+}
+
+void
+assign_io_method(int newval, void *extra)
+{
+}
+
+bool
+check_io_max_concurrency(int *newval, void **extra, GucSource source)
+{
+ if (*newval == -1)
+ {
+ /*
+ * Auto-tuning will be applied later during startup, as auto-tuning
+ * depends on the value of various GUCs.
+ */
+ return true;
+ }
+ else if (*newval == 0)
+ {
+ GUC_check_errdetail("Only -1 or values bigger than 0 are valid.");
+ return false;
+ }
+
+ return true;
+}
diff --git a/src/backend/storage/aio/aio_init.c b/src/backend/storage/aio/aio_init.c
new file mode 100644
index 00000000000..aeacc144149
--- /dev/null
+++ b/src/backend/storage/aio/aio_init.c
@@ -0,0 +1,37 @@
+/*-------------------------------------------------------------------------
+ *
+ * aio_init.c
+ * AIO - Subsystem Initialization
+ *
+ * Portions Copyright (c) 1996-2025, PostgreSQL Global Development Group
+ * Portions Copyright (c) 1994, Regents of the University of California
+ *
+ * IDENTIFICATION
+ * src/backend/storage/aio/aio_init.c
+ *
+ *-------------------------------------------------------------------------
+ */
+
+#include "postgres.h"
+
+#include "storage/aio_subsys.h"
+
+
+
+Size
+AioShmemSize(void)
+{
+ Size sz = 0;
+
+ return sz;
+}
+
+void
+AioShmemInit(void)
+{
+}
+
+void
+pgaio_init_backend(void)
+{
+}
diff --git a/src/backend/storage/aio/meson.build b/src/backend/storage/aio/meson.build
index 8abe0eb4863..c822fd4ddf7 100644
--- a/src/backend/storage/aio/meson.build
+++ b/src/backend/storage/aio/meson.build
@@ -1,5 +1,7 @@
# Copyright (c) 2024-2025, PostgreSQL Global Development Group
backend_sources += files(
+ 'aio.c',
+ 'aio_init.c',
'read_stream.c',
)
diff --git a/src/backend/storage/ipc/ipci.c b/src/backend/storage/ipc/ipci.c
index 174eed70367..2fa045e6b0f 100644
--- a/src/backend/storage/ipc/ipci.c
+++ b/src/backend/storage/ipc/ipci.c
@@ -37,6 +37,7 @@
#include "replication/slotsync.h"
#include "replication/walreceiver.h"
#include "replication/walsender.h"
+#include "storage/aio_subsys.h"
#include "storage/bufmgr.h"
#include "storage/dsm.h"
#include "storage/dsm_registry.h"
@@ -148,6 +149,7 @@ CalculateShmemSize(int *num_semaphores)
size = add_size(size, WaitEventCustomShmemSize());
size = add_size(size, InjectionPointShmemSize());
size = add_size(size, SlotSyncShmemSize());
+ size = add_size(size, AioShmemSize());
/* include additional requested shmem from preload libraries */
size = add_size(size, total_addin_request);
@@ -340,6 +342,7 @@ CreateOrAttachShmemStructs(void)
StatsShmemInit();
WaitEventCustomShmemInit();
InjectionPointShmemInit();
+ AioShmemInit();
}
/*
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index ee1a9d5d98b..9b134c03456 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -43,6 +43,7 @@
#include "replication/slot.h"
#include "replication/slotsync.h"
#include "replication/walsender.h"
+#include "storage/aio_subsys.h"
#include "storage/bufmgr.h"
#include "storage/fd.h"
#include "storage/ipc.h"
@@ -628,6 +629,12 @@ BaseInit(void)
*/
pgstat_initialize();
+ /*
+ * Initialize AIO before infrastructure that might need to actually
+ * execute AIO.
+ */
+ pgaio_init_backend();
+
/* Do local initialization of storage and buffer managers */
InitSync();
smgrinit();
diff --git a/src/backend/utils/misc/guc_tables.c b/src/backend/utils/misc/guc_tables.c
index ad25cbb39c5..28dcc465762 100644
--- a/src/backend/utils/misc/guc_tables.c
+++ b/src/backend/utils/misc/guc_tables.c
@@ -72,6 +72,7 @@
#include "replication/slot.h"
#include "replication/slotsync.h"
#include "replication/syncrep.h"
+#include "storage/aio.h"
#include "storage/bufmgr.h"
#include "storage/bufpage.h"
#include "storage/large_object.h"
@@ -3254,6 +3255,18 @@ struct config_int ConfigureNamesInt[] =
NULL, NULL, NULL
},
+ {
+ {"io_max_concurrency",
+ PGC_POSTMASTER,
+ RESOURCES_IO,
+ gettext_noop("Max number of IOs that one process can execute simultaneously."),
+ NULL,
+ },
+ &io_max_concurrency,
+ -1, -1, 1024,
+ check_io_max_concurrency, NULL, NULL
+ },
+
{
{"backend_flush_after", PGC_USERSET, RESOURCES_IO,
gettext_noop("Number of pages after which previously performed writes are flushed to disk."),
@@ -5299,6 +5312,16 @@ struct config_enum ConfigureNamesEnum[] =
NULL, NULL, NULL
},
+ {
+ {"io_method", PGC_POSTMASTER, RESOURCES_IO,
+ gettext_noop("Selects the method for executing asynchronous I/O."),
+ NULL
+ },
+ &io_method,
+ DEFAULT_IO_METHOD, io_method_options,
+ NULL, assign_io_method, NULL
+ },
+
/* End-of-list marker */
{
{NULL, 0, 0, NULL, NULL}, NULL, 0, NULL, NULL, NULL, NULL
diff --git a/src/backend/utils/misc/postgresql.conf.sample b/src/backend/utils/misc/postgresql.conf.sample
index 2d1de9c37bd..a31c9963ce4 100644
--- a/src/backend/utils/misc/postgresql.conf.sample
+++ b/src/backend/utils/misc/postgresql.conf.sample
@@ -202,6 +202,12 @@
#maintenance_io_concurrency = 10 # 1-1000; 0 disables prefetching
#io_combine_limit = 128kB # usually 1-32 blocks (depends on OS)
+#io_method = sync # sync (change requires restart)
+#io_max_concurrency = -1 # Max number of IOs that one process
+ # can execute simultaneously
+ # -1 sets based on shared_buffers
+ # (change requires restart)
+
# - Worker Processes -
#max_worker_processes = 8 # (change requires restart)
diff --git a/src/backend/utils/resowner/resowner.c b/src/backend/utils/resowner/resowner.c
index ac5ca4a765e..76b9cec1e26 100644
--- a/src/backend/utils/resowner/resowner.c
+++ b/src/backend/utils/resowner/resowner.c
@@ -47,6 +47,8 @@
#include "common/hashfn.h"
#include "common/int.h"
+#include "lib/ilist.h"
+#include "storage/aio.h"
#include "storage/ipc.h"
#include "storage/predicate.h"
#include "storage/proc.h"
@@ -155,6 +157,12 @@ struct ResourceOwnerData
/* The local locks cache. */
LOCALLOCK *locks[MAX_RESOWNER_LOCKS]; /* list of owned locks */
+
+ /*
+ * AIO handles need be registered in critical sections and therefore
+ * cannot use the normal ResoureElem mechanism.
+ */
+ dlist_head aio_handles;
};
@@ -425,6 +433,8 @@ ResourceOwnerCreate(ResourceOwner parent, const char *name)
parent->firstchild = owner;
}
+ dlist_init(&owner->aio_handles);
+
return owner;
}
@@ -725,6 +735,13 @@ ResourceOwnerReleaseInternal(ResourceOwner owner,
* so issue warnings. In the abort case, just clean up quietly.
*/
ResourceOwnerReleaseAll(owner, phase, isCommit);
+
+ while (!dlist_is_empty(&owner->aio_handles))
+ {
+ dlist_node *node = dlist_head_node(&owner->aio_handles);
+
+ pgaio_io_release_resowner(node, !isCommit);
+ }
}
else if (phase == RESOURCE_RELEASE_LOCKS)
{
@@ -1082,3 +1099,15 @@ ResourceOwnerForgetLock(ResourceOwner owner, LOCALLOCK *locallock)
elog(ERROR, "lock reference %p is not owned by resource owner %s",
locallock, owner->name);
}
+
+void
+ResourceOwnerRememberAioHandle(ResourceOwner owner, struct dlist_node *ioh_node)
+{
+ dlist_push_tail(&owner->aio_handles, ioh_node);
+}
+
+void
+ResourceOwnerForgetAioHandle(ResourceOwner owner, struct dlist_node *ioh_node)
+{
+ dlist_delete_from(&owner->aio_handles, ioh_node);
+}
diff --git a/doc/src/sgml/config.sgml b/doc/src/sgml/config.sgml
index d2fa5f7d1a9..7d5482f31cd 100644
--- a/doc/src/sgml/config.sgml
+++ b/doc/src/sgml/config.sgml
@@ -2644,6 +2644,57 @@ include_dir 'conf.d'
</para>
</listitem>
</varlistentry>
+
+ <varlistentry id="guc-io-max-concurrency" xreflabel="io_max_concurrency">
+ <term><varname>io_max_concurrency</varname> (<type>integer</type>)
+ <indexterm>
+ <primary><varname>io_max_concurrency</varname> configuration parameter</primary>
+ </indexterm>
+ </term>
+ <listitem>
+ <para>
+ Controls the maximum number of I/O operations that one process can
+ execute simultaneously.
+ </para>
+ <para>
+ The default setting of <literal>-1</literal> selects a number based
+ on <xref linkend="guc-shared-buffers"/> and the maximum number of
+ processes (<xref linkend="guc-max-connections"/>, <xref
+ linkend="guc-autovacuum-worker-slots"/>, <xref
+ linkend="guc-max-worker-processes"/> and <xref
+ linkend="guc-max-wal-senders"/>), but not more than
+ <literal>64</literal>.
+ </para>
+ <para>
+ This parameter can only be set at server start.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry id="guc-io-method" xreflabel="io_method">
+ <term><varname>io_method</varname> (<type>enum</type>)
+ <indexterm>
+ <primary><varname>io_method</varname> configuration parameter</primary>
+ </indexterm>
+ </term>
+ <listitem>
+ <para>
+ Selects the method for executing asynchronous I/O.
+ Possible values are:
+ <itemizedlist>
+ <listitem>
+ <para>
+ <literal>sync</literal> (execute asynchronous I/O synchronously)
+ </para>
+ </listitem>
+ </itemizedlist>
+ </para>
+ <para>
+ This parameter can only be set at server start.
+ </para>
+ </listitem>
+ </varlistentry>
+
</variablelist>
</sect2>
diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list
index 9840060997f..7e6bbc1a8c1 100644
--- a/src/tools/pgindent/typedefs.list
+++ b/src/tools/pgindent/typedefs.list
@@ -1277,6 +1277,7 @@ IntoClause
InvalMessageArray
InvalidationInfo
InvalidationMsgsGroup
+IoMethod
IpcMemoryId
IpcMemoryKey
IpcMemoryState
--
2.48.1.76.g4e746b1a31.dirty
[text/x-diff] v2.6-0002-aio-Add-asynchronous-I-O-infrastructure.patch (88.1K, ../../ge6nsuddurhpmll3xj22vucvqwp4agqz6ndtcf2mhyeydzarst@l75dman5x53p/3-v2.6-0002-aio-Add-asynchronous-I-O-infrastructure.patch)
download | inline diff:
From 99e985aaef9ff99368c17de6e24359375e996401 Mon Sep 17 00:00:00 2001
From: Andres Freund <andres@anarazel.de>
Date: Fri, 7 Mar 2025 10:14:31 -0500
Subject: [PATCH v2.6 02/34] aio: Add asynchronous I/O infrastructure
The main motivations to use AIO in PostgreSQL are:
a) Reduce the time spent waiting for IO by issuing IO sufficiently early.
In a few places we have approximated this using posix_fadvise() based
prefetching, but that is fairly limited (no completion feedback, double the
syscalls, only works with buffered IO, only works on some OSs).
b) Allow to use Direct-I/O (DIO).
DIO can offload most of the work for IO to hardware and thus increase
throughput / decrease CPU utilization, as well as reduce latency. While we
have gained the ability to configure DIO in d4e71df6, it is not yet usable
for real world workloads, as every IO is executed synchronously.
For portability, the new AIO infrastructure allows to implement AIO using
different methods. The choice of the AIO method is controlled by a new
io_method GUC. As of this commit, the only implemented method is "sync",
i.e. AIO is not actually executed asynchronously. The "sync" method exists to
allow to bypass most of the new code initially.
Subsequent commits will introduce additional IO methods, including a
cross-platform method implemented using worker processes and a linux specific
method using io_uring.
To allow different parts of postgres to use AIO, the core AIO infrastructure
does not need to know what kind of files it is operating on. The necessary
behavioral differences for different files are abstracted as "AIO
Targets". One example target would be smgr. For boring portability reasons all
targets currently need to be added to an array in aio_target.c. This commit
does not implement any AIO targets, just the infrastructure for them. The smgr
target will be added in a later commit.
Completion (and other events) of IOs for one type of file (i.e. one AIO
target) need to be reacted to differently based on the IO operation and the
callsite. This is made possible by callbacks that can be registered on
IOs. E.g. an smgr read into a local buffer does not need to update the
corresponding BufferDesc (as there is none), but a read into shared buffers
does. This commit does not contain any callbacks, they will be added in
subsequent commits.
For now the AIO infrastructure only understands READV and WRITEV operations,
but it is expected that more operations will be added. E.g. fsync/fdatasync,
flush_range and network operations like send/recv.
As of this commit nothing uses the AIO infrastructure. Other commits will add
an smgr target, md.c and bufmgr.c callbacks and then finally use AIO for
read_stream.c IO, which, in one fell swoop, will convert all read stream users
to AIO.
The goal is to use AIO in many more places. There are patches to use AIO for
checkpointer and bgwriter that are reasonably close to being ready. There also
are prototypes to use it for WAL, relation extension, backend writes and many
more. Those prototypes were important to ensure the design of the AIO
subsystem is not too limiting (e.g. WAL writes need to happen in critical
sections, which influenced a lot of the design).
A future commit will add an AIO README explaining the AIO architecture and how
to use the AIO subsystem. The README is added later, as it references details
only added in later commits.
Many many more people than the folks named below have contributed with
feedback, work on semi-independent patches etc. E.g. various folks have
contributed patches to use the read stream infrastructure (added by Thomas in
b5a9b18cd0b) in more places. Similarly, a *lot* of folks have contributed to
the CI infrastructure, that I started to work on to make adding AIO feasible.
Some of the work by contributors has gone into the "v1" prototype of AIO,
which heavily influenced the current design of the AIO subsystem. None of the
code from that directly survives, but without the prototype, the current
version of the AIO infrastructure would not exist.
Similarly, the reviewers below have not necessarily looked at the current
design or the whole infrastructure, but have provided very valuable input. I
am to blame for problems, not they.
Author: Andres Freund <andres@anarazel.de>
Co-authored-by: Thomas Munro <thomas.munro@gmail.com>
Co-authored-by: Nazir Bilal Yavuz <byavuz81@gmail.com>
Co-authored-by: Melanie Plageman <melanieplageman@gmail.com>
Reviewed-by: Heikki Linnakangas <hlinnaka@iki.fi>
Reviewed-by: Noah Misch <noah@leadboat.com>
Reviewed-by: Jakub Wartak <jakub.wartak@enterprisedb.com>
Reviewed-by: Melanie Plageman <melanieplageman@gmail.com>
Reviewed-by: Robert Haas <robertmhaas@gmail.com>
Reviewed-by: Dmitry Dolgov <9erthalion6@gmail.com>
Discussion: https://postgr.es/m/uvrtrknj4kdytuboidbhwclo4gxhswwcpgadptsjvjqcluzmah%40brqs62irg4dt
Discussion: https://postgr.es/m/20210223100344.llw5an2aklengrmn@alap3.anarazel.de
Discussion: https://postgr.es/m/stj36ea6yyhoxtqkhpieia2z4krnam7qyetc57rfezgk4zgapf@gcnactj4z56m
---
src/include/storage/aio.h | 300 +++++
src/include/storage/aio_internal.h | 363 ++++++
src/include/storage/aio_types.h | 117 ++
src/backend/storage/aio/Makefile | 4 +
src/backend/storage/aio/aio.c | 1127 +++++++++++++++++
src/backend/storage/aio/aio_callback.c | 307 +++++
src/backend/storage/aio/aio_init.c | 186 +++
src/backend/storage/aio/aio_io.c | 180 +++
src/backend/storage/aio/aio_target.c | 118 ++
src/backend/storage/aio/meson.build | 4 +
src/backend/storage/aio/method_sync.c | 47 +
.../utils/activity/wait_event_names.txt | 1 +
src/tools/pgindent/typedefs.list | 21 +
13 files changed, 2775 insertions(+)
create mode 100644 src/include/storage/aio_internal.h
create mode 100644 src/include/storage/aio_types.h
create mode 100644 src/backend/storage/aio/aio_callback.c
create mode 100644 src/backend/storage/aio/aio_io.c
create mode 100644 src/backend/storage/aio/aio_target.c
create mode 100644 src/backend/storage/aio/method_sync.c
diff --git a/src/include/storage/aio.h b/src/include/storage/aio.h
index e79d5343038..0124dc4d24d 100644
--- a/src/include/storage/aio.h
+++ b/src/include/storage/aio.h
@@ -14,6 +14,9 @@
#ifndef AIO_H
#define AIO_H
+#include "storage/aio_types.h"
+#include "storage/procnumber.h"
+
/* Enum for io_method GUC. */
@@ -26,9 +29,306 @@ typedef enum IoMethod
#define DEFAULT_IO_METHOD IOMETHOD_SYNC
+/*
+ * Flags for an IO that can be set with pgaio_io_set_flag().
+ */
+typedef enum PgAioHandleFlags
+{
+ /*
+ * The IO references backend local memory.
+ *
+ * This needs to be set on an IO whenever the IO references process-local
+ * memory. Some IO methods do not support executing IO that references
+ * process local memory and thus need to fall back to executing IO
+ * synchronously for IOs with this flag set.
+ *
+ * Required for correctness.
+ */
+ PGAIO_HF_REFERENCES_LOCAL = 1 << 1,
+
+ /*
+ * Hint that IO will be executed synchronously.
+ *
+ * This can make it a bit cheaper to execute synchronous IO via the AIO
+ * interface, to avoid needing an AIO and non-AIO version of code.
+ *
+ * Advantageous to set, if applicable, but not required for correctness.
+ */
+ PGAIO_HF_SYNCHRONOUS = 1 << 0,
+
+ /*
+ * IO is using buffered IO, used to control heuristic in some IO methods.
+ *
+ * Advantageous to set, if applicable, but not required for correctness.
+ */
+ PGAIO_HF_BUFFERED = 1 << 2,
+} PgAioHandleFlags;
+
+/*
+ * The IO operations supported by the AIO subsystem.
+ *
+ * This could be in aio_internal.h, as it is not pubicly referenced, but
+ * PgAioOpData currently *does* need to be public, therefore keeping this
+ * public seems to make sense.
+ */
+typedef enum PgAioOp
+{
+ /* intentionally the zero value, to help catch zeroed memory etc */
+ PGAIO_OP_INVALID = 0,
+
+ PGAIO_OP_READV,
+ PGAIO_OP_WRITEV,
+
+ /**
+ * In the near term we'll need at least:
+ * - fsync / fdatasync
+ * - flush_range
+ *
+ * Eventually we'll additionally want at least:
+ * - send
+ * - recv
+ * - accept
+ **/
+} PgAioOp;
+
+#define PGAIO_OP_COUNT (PGAIO_OP_WRITEV + 1)
+
+
+/*
+ * On what is IO being performed.
+ *
+ * PgAioTargetID specific behaviour should be implemented in
+ * aio_target.c.
+ */
+typedef enum PgAioTargetID
+{
+ /* intentionally the zero value, to help catch zeroed memory etc */
+ PGAIO_TID_INVALID = 0,
+} PgAioTargetID;
+
+#define PGAIO_TID_COUNT (PGAIO_TID_INVALID + 1)
+
+
+/*
+ * Data necessary for support IO operations (see PgAioOp).
+ *
+ * NB: Note that the FDs in here may *not* be relied upon for re-issuing
+ * requests (e.g. for partial reads/writes) - the FD might be from another
+ * process, or closed since. That's not a problem for IOs waiting to be issued
+ * only because the queue is flushed when closing an FD.
+ */
+typedef union
+{
+ struct
+ {
+ int fd;
+ uint16 iov_length;
+ uint64 offset;
+ } read;
+
+ struct
+ {
+ int fd;
+ uint16 iov_length;
+ uint64 offset;
+ } write;
+} PgAioOpData;
+
+
+/*
+ * Information the object that IO is executed on. Mostly callbacks that
+ * operate on PgAioTargetData.
+ *
+ * typedef is in aio_types.h
+ */
+struct PgAioTargetInfo
+{
+ void (*reopen) (PgAioHandle *ioh);
+
+ char *(*describe_identity) (const PgAioTargetData *sd);
+
+ const char *name;
+};
+
+
+/*
+ * IDs for callbacks that can be registered on an IO.
+ *
+ * Callbacks are identified by an ID rather than a function pointer. There are
+ * two main reasons:
+ *
+ * 1) Memory within PgAioHandle is precious, due to the number of PgAioHandle
+ * structs in pre-allocated shared memory.
+ *
+ * 2) Due to EXEC_BACKEND function pointers are not necessarily stable between
+ * different backends, therefore function pointers cannot directly be in
+ * shared memory.
+ *
+ * Without 2), we could fairly easily allow to add new callbacks, by filling a
+ * ID->pointer mapping table on demand. In the presence of 2 that's still
+ * doable, but harder, because every process has to re-register the pointers
+ * so that a local ID->"backend local pointer" mapping can be maintained.
+ */
+typedef enum PgAioHandleCallbackID
+{
+ PGAIO_HCB_INVALID,
+} PgAioHandleCallbackID;
+
+
+typedef void (*PgAioHandleCallbackStage) (PgAioHandle *ioh, uint8 cb_flags);
+typedef PgAioResult (*PgAioHandleCallbackComplete) (PgAioHandle *ioh, PgAioResult prior_result, uint8 cb_flags);
+typedef void (*PgAioHandleCallbackReport) (PgAioResult result, const PgAioTargetData *target_data, int elevel);
+
+/* typedef is in aio_types.h */
+struct PgAioHandleCallbacks
+{
+ /*
+ * Prepare resources affected by the IO for execution. This could e.g.
+ * include moving ownership of buffer pins to the AIO subsystem.
+ */
+ PgAioHandleCallbackStage stage;
+
+ /*
+ * Update the state of resources affected by the IO to reflect completion
+ * of the IO. This could e.g. include updating shared buffer state to
+ * signal the IO has finished.
+ *
+ * The _shared suffix indicates that this is executed by the backend that
+ * completed the IO, which may or may not be the backend that issued the
+ * IO. Obviously the callback thus can only modify resources in shared
+ * memory.
+ *
+ * The latest registered callback is called first. This allows
+ * higher-level code to register callbacks that can rely on callbacks
+ * registered by lower-level code to already have been executed.
+ *
+ * NB: This is called in a critical section. Errors can be signalled by
+ * the callback's return value, it's the responsibility of the IO's issuer
+ * to react appropriately.
+ */
+ PgAioHandleCallbackComplete complete_shared;
+
+ /*
+ * Like complete_shared, except called in the issuing backend.
+ *
+ * This variant of the completion callback is useful when backend-local
+ * state has to be updated to reflect the IO's completion. E.g. a
+ * temporary buffer's BufferDesc isn't accessible in complete_shared.
+ *
+ * Local callbacks are only called after complete_shared for all
+ * registered callbacks has been called.
+ */
+ PgAioHandleCallbackComplete complete_local;
+
+ /*
+ * Report the result of an IO operation. This is e.g. used to raise an
+ * error after an IO failed at the appropriate time (i.e. not when the IO
+ * failed, but under control of the code that issued the IO).
+ */
+ PgAioHandleCallbackReport report;
+};
+
+
+
+/*
+ * How many callbacks can be registered for one IO handle. Currently we only
+ * need two, but it's not hard to imagine needing a few more.
+ */
+#define PGAIO_HANDLE_MAX_CALLBACKS 4
+
+
+
+/* --------------------------------------------------------------------------------
+ * IO Handles
+ * --------------------------------------------------------------------------------
+ */
+
+/* functions in aio.c */
+struct ResourceOwnerData;
+extern PgAioHandle *pgaio_io_acquire(struct ResourceOwnerData *resowner, PgAioReturn *ret);
+extern PgAioHandle *pgaio_io_acquire_nb(struct ResourceOwnerData *resowner, PgAioReturn *ret);
+
+extern void pgaio_io_release(PgAioHandle *ioh);
struct dlist_node;
extern void pgaio_io_release_resowner(struct dlist_node *ioh_node, bool on_error);
+extern void pgaio_io_set_flag(PgAioHandle *ioh, PgAioHandleFlags flag);
+
+extern int pgaio_io_get_id(PgAioHandle *ioh);
+extern ProcNumber pgaio_io_get_owner(PgAioHandle *ioh);
+
+extern void pgaio_io_get_wref(PgAioHandle *ioh, PgAioWaitRef *iow);
+
+/* functions in aio_io.c */
+struct iovec;
+extern int pgaio_io_get_iovec(PgAioHandle *ioh, struct iovec **iov);
+
+extern PgAioOpData *pgaio_io_get_op_data(PgAioHandle *ioh);
+
+extern void pgaio_io_prep_readv(PgAioHandle *ioh,
+ int fd, int iovcnt, uint64 offset);
+extern void pgaio_io_prep_writev(PgAioHandle *ioh,
+ int fd, int iovcnt, uint64 offset);
+
+/* functions in aio_target.c */
+extern void pgaio_io_set_target(PgAioHandle *ioh, PgAioTargetID targetid);
+extern bool pgaio_io_has_target(PgAioHandle *ioh);
+extern PgAioTargetData *pgaio_io_get_target_data(PgAioHandle *ioh);
+extern char *pgaio_io_get_target_description(PgAioHandle *ioh);
+
+/* functions in aio_callback.c */
+extern void pgaio_io_register_callbacks(PgAioHandle *ioh, PgAioHandleCallbackID cb_id,
+ uint8 cb_data);
+extern void pgaio_io_set_handle_data_64(PgAioHandle *ioh, uint64 *data, uint8 len);
+extern void pgaio_io_set_handle_data_32(PgAioHandle *ioh, uint32 *data, uint8 len);
+extern uint64 *pgaio_io_get_handle_data(PgAioHandle *ioh, uint8 *len);
+
+
+
+/* --------------------------------------------------------------------------------
+ * IO Wait References
+ * --------------------------------------------------------------------------------
+ */
+
+extern void pgaio_wref_clear(PgAioWaitRef *iow);
+extern bool pgaio_wref_valid(PgAioWaitRef *iow);
+extern int pgaio_wref_get_id(PgAioWaitRef *iow);
+
+extern void pgaio_wref_wait(PgAioWaitRef *iow);
+extern bool pgaio_wref_check_done(PgAioWaitRef *iow);
+
+
+
+/* --------------------------------------------------------------------------------
+ * IO Result
+ * --------------------------------------------------------------------------------
+ */
+
+extern void pgaio_result_report(PgAioResult result, const PgAioTargetData *target_data,
+ int elevel);
+
+
+
+/* --------------------------------------------------------------------------------
+ * Actions on multiple IOs.
+ * --------------------------------------------------------------------------------
+ */
+
+extern void pgaio_enter_batchmode(void);
+extern void pgaio_exit_batchmode(void);
+extern void pgaio_submit_staged(void);
+extern bool pgaio_have_staged(void);
+
+
+
+/* --------------------------------------------------------------------------------
+ * Other
+ * --------------------------------------------------------------------------------
+ */
+
+extern void pgaio_closing_fd(int fd);
+
+
/* GUCs */
extern PGDLLIMPORT int io_method;
diff --git a/src/include/storage/aio_internal.h b/src/include/storage/aio_internal.h
new file mode 100644
index 00000000000..d213b0842a3
--- /dev/null
+++ b/src/include/storage/aio_internal.h
@@ -0,0 +1,363 @@
+/*-------------------------------------------------------------------------
+ *
+ * aio_internal.h
+ * AIO related declarations that should only be used by the AIO subsystem
+ * internally.
+ *
+ *
+ * Portions Copyright (c) 1996-2025, PostgreSQL Global Development Group
+ * Portions Copyright (c) 1994, Regents of the University of California
+ *
+ * src/include/storage/aio_internal.h
+ *
+ *-------------------------------------------------------------------------
+ */
+#ifndef AIO_INTERNAL_H
+#define AIO_INTERNAL_H
+
+
+#include "lib/ilist.h"
+#include "port/pg_iovec.h"
+#include "storage/aio.h"
+#include "storage/condition_variable.h"
+
+
+/*
+ * The maximum number of IOs that can be batch submitted at once.
+ */
+#define PGAIO_SUBMIT_BATCH_SIZE 32
+
+
+
+typedef enum PgAioHandleState
+{
+ /* not in use */
+ PGAIO_HS_IDLE = 0,
+
+ /* returned by pgaio_io_acquire() */
+ PGAIO_HS_HANDED_OUT,
+
+ /* pgaio_io_prep_*() has been called, but IO hasn't been submitted yet */
+ PGAIO_HS_DEFINED,
+
+ /* target's stage() callback has been called, ready to be submitted */
+ PGAIO_HS_STAGED,
+
+ /* IO has been submitted and is being executed */
+ PGAIO_HS_SUBMITTED,
+
+ /* IO finished, but result has not yet been processed */
+ PGAIO_HS_COMPLETED_IO,
+
+ /* IO completed, shared completion has been called */
+ PGAIO_HS_COMPLETED_SHARED,
+
+ /* IO completed, local completion has been called */
+ PGAIO_HS_COMPLETED_LOCAL,
+} PgAioHandleState;
+
+
+struct ResourceOwnerData;
+
+/* typedef is in aio_types.h */
+struct PgAioHandle
+{
+ /* all state updates should go through pgaio_io_update_state() */
+ PgAioHandleState state:8;
+
+ /* what are we operating on */
+ PgAioTargetID target:8;
+
+ /* which IO operation */
+ PgAioOp op:8;
+
+ /* bitfield of PgAioHandleFlags */
+ uint8 flags;
+
+ uint8 num_callbacks;
+
+ /* using the proper type here would use more space */
+ uint8 callbacks[PGAIO_HANDLE_MAX_CALLBACKS];
+
+ /* data forwarded to each callback */
+ uint8 callbacks_data[PGAIO_HANDLE_MAX_CALLBACKS];
+
+ /*
+ * Length of data associated with handle using
+ * pgaio_io_set_handle_data_*().
+ */
+ uint8 handle_data_len;
+
+ /* XXX: could be optimized out with some pointer math */
+ int32 owner_procno;
+
+ /* raw result of the IO operation */
+ int32 result;
+
+ /**
+ * In which list the handle is registered, depends on the state:
+ * - IDLE, in per-backend list
+ * - HANDED_OUT - not in a list
+ * - DEFINED - in per-backend staged list
+ * - STAGED - in per-backend staged list
+ * - SUBMITTED - in issuer's in_flight list
+ * - COMPLETED_IO - in issuer's in_flight list
+ * - COMPLETED_SHARED - in issuer's in_flight list
+ **/
+ dlist_node node;
+
+ struct ResourceOwnerData *resowner;
+ dlist_node resowner_node;
+
+ /* incremented every time the IO handle is reused */
+ uint64 generation;
+
+ /*
+ * To wait for the IO to complete other backends can wait on this CV. Note
+ * that, if in SUBMITTED state, a waiter first needs to check if it needs
+ * to do work via IoMethodOps->wait_one().
+ */
+ ConditionVariable cv;
+
+ /* result of shared callback, passed to issuer callback */
+ PgAioResult distilled_result;
+
+ /*
+ * Index into PgAioCtl->iovecs and PgAioCtl->handle_data.
+ *
+ * At the moment there's no need to differentiate between the two, but
+ * that won't necessarily stay that way.
+ */
+ uint32 iovec_off;
+
+ /*
+ * If not NULL, this memory location will be updated with information
+ * about the IOs completion iff the issuing backend learns about the IOs
+ * completion.
+ */
+ PgAioReturn *report_return;
+
+ /* Data necessary for the IO to be performed */
+ PgAioOpData op_data;
+
+ /*
+ * Data necessary to identify the object undergoing IO to higher-level
+ * code. Needs to be sufficient to allow another backend to reopen the
+ * file.
+ */
+ PgAioTargetData target_data;
+};
+
+
+typedef struct PgAioBackend
+{
+ /* index into PgAioCtl->io_handles */
+ uint32 io_handle_off;
+
+ /* IO Handles that currently are not used */
+ dclist_head idle_ios;
+
+ /*
+ * Only one IO may be returned by pgaio_io_acquire()/pgaio_io_acquire()
+ * without having been either defined (by actually associating it with IO)
+ * or by released (with pgaio_io_release()). This restriction is necessary
+ * to guarantee that we always can acquire an IO. ->handed_out_io is used
+ * to enforce that rule.
+ */
+ PgAioHandle *handed_out_io;
+
+ /* Are we currently in batchmode? See pgaio_enter_batchmode(). */
+ bool in_batchmode;
+
+ /*
+ * IOs that are defined, but not yet submitted.
+ */
+ uint16 num_staged_ios;
+ PgAioHandle *staged_ios[PGAIO_SUBMIT_BATCH_SIZE];
+
+ /*
+ * List of in-flight IOs. Also contains IOs that aren't strict speaking
+ * in-flight anymore, but have been waited-for and completed by another
+ * backend. Once this backend sees such an IO it'll be reclaimed.
+ *
+ * The list is ordered by submission time, with more recently submitted
+ * IOs being appended at the end.
+ */
+ dclist_head in_flight_ios;
+} PgAioBackend;
+
+
+typedef struct PgAioCtl
+{
+ int backend_state_count;
+ PgAioBackend *backend_state;
+
+ /*
+ * Array of iovec structs. Each iovec is owned by a specific backend. The
+ * allocation is in PgAioCtl to allow the maximum number of iovecs for
+ * individual IOs to be configurable with PGC_POSTMASTER GUC.
+ */
+ uint32 iovec_count;
+ struct iovec *iovecs;
+
+ /*
+ * For, e.g., an IO covering multiple buffers in shared / temp buffers, we
+ * need to get Buffer IDs during completion to be able to change the
+ * BufferDesc state accordingly. This space can be used to store e.g.
+ * Buffer IDs. Note that the actual iovec might be shorter than this,
+ * because we combine neighboring pages into one larger iovec entry.
+ */
+ uint64 *handle_data;
+
+ uint32 io_handle_count;
+ PgAioHandle *io_handles;
+} PgAioCtl;
+
+
+
+/*
+ * Callbacks used to implement an IO method.
+ */
+typedef struct IoMethodOps
+{
+ /* global initialization */
+
+ /*
+ * Amount of additional shared memory to reserve for the io_method. Called
+ * just like a normal ipci.c style *Size() function. Optional.
+ */
+ size_t (*shmem_size) (void);
+
+ /*
+ * Initialize shared memory. First time is true if AIO's shared memory was
+ * just initialized, false otherwise. Optional.
+ */
+ void (*shmem_init) (bool first_time);
+
+ /*
+ * Per-backend initialization. Optional.
+ */
+ void (*init_backend) (void);
+
+
+ /* handling of IOs */
+
+ /* optional */
+ bool (*needs_synchronous_execution) (PgAioHandle *ioh);
+
+ /*
+ * Start executing passed in IOs.
+ *
+ * Will not be called if ->needs_synchronous_execution() returned true.
+ *
+ * num_staged_ios is <= PGAIO_SUBMIT_BATCH_SIZE.
+ *
+ */
+ int (*submit) (uint16 num_staged_ios, PgAioHandle **staged_ios);
+
+ /*
+ * Wait for the IO to complete. Optional.
+ *
+ * If not provided, it needs to be guaranteed that the IO method calls
+ * pgaio_io_process_completion() without further interaction by the
+ * issuing backend.
+ */
+ void (*wait_one) (PgAioHandle *ioh,
+ uint64 ref_generation);
+} IoMethodOps;
+
+
+/* aio.c */
+extern bool pgaio_io_was_recycled(PgAioHandle *ioh, uint64 ref_generation, PgAioHandleState *state);
+extern void pgaio_io_stage(PgAioHandle *ioh, PgAioOp op);
+extern void pgaio_io_process_completion(PgAioHandle *ioh, int result);
+extern void pgaio_io_prepare_submit(PgAioHandle *ioh);
+extern bool pgaio_io_needs_synchronous_execution(PgAioHandle *ioh);
+extern const char *pgaio_io_get_state_name(PgAioHandle *ioh);
+const char *pgaio_result_status_string(PgAioResultStatus rs);
+extern void pgaio_shutdown(int code, Datum arg);
+
+/* aio_callback.c */
+extern void pgaio_io_call_stage(PgAioHandle *ioh);
+extern void pgaio_io_call_complete_shared(PgAioHandle *ioh);
+extern void pgaio_io_call_complete_local(PgAioHandle *ioh);
+
+/* aio_io.c */
+extern void pgaio_io_perform_synchronously(PgAioHandle *ioh);
+extern const char *pgaio_io_get_op_name(PgAioHandle *ioh);
+
+/* aio_target.c */
+extern bool pgaio_io_can_reopen(PgAioHandle *ioh);
+extern void pgaio_io_reopen(PgAioHandle *ioh);
+extern const char *pgaio_io_get_target_name(PgAioHandle *ioh);
+
+
+/*
+ * The AIO subsystem has fairly verbose debug logging support. This can be
+ * enabled/disabled at buildtime. The reason for this is that
+ * a) the verbosity can make debugging things on higher levels hard
+ * b) even if logging can be skipped due to elevel checks, it still causes a
+ * measurable slowdown
+ */
+#define PGAIO_VERBOSE 1
+
+/*
+ * Simple ereport() wrapper that only logs if PGAIO_VERBOSE is defined.
+ *
+ * This intentionally still compiles the code, guarded by a constant if (0),
+ * if verbose logging is disabled, to make it less likely that debug logging
+ * is silently broken.
+ *
+ * The current definition requires passing at least one argument.
+ */
+#define pgaio_debug(elevel, msg, ...) \
+ do { \
+ if (PGAIO_VERBOSE) \
+ ereport(elevel, \
+ errhidestmt(true), errhidecontext(true), \
+ errmsg_internal(msg, \
+ __VA_ARGS__)); \
+ } while(0)
+
+/*
+ * Simple ereport() wrapper. Note that the definition requires passing at
+ * least one argument.
+ */
+#define pgaio_debug_io(elevel, ioh, msg, ...) \
+ pgaio_debug(elevel, "io %-10d|op %-5s|target %-4s|state %-16s: " msg, \
+ pgaio_io_get_id(ioh), \
+ pgaio_io_get_op_name(ioh), \
+ pgaio_io_get_target_name(ioh), \
+ pgaio_io_get_state_name(ioh), \
+ __VA_ARGS__)
+
+
+#ifdef USE_INJECTION_POINTS
+
+extern void pgaio_io_call_inj(PgAioHandle *ioh, const char *injection_point);
+
+/* just for use in tests, from within injection points */
+extern PgAioHandle *pgaio_inj_io_get(void);
+
+#else
+
+#define pgaio_io_call_inj(ioh, injection_point) (void) 0
+
+/*
+ * no fallback for pgaio_inj_io_get, all code using injection points better be
+ * guarded by USE_INJECTION_POINTS.
+ */
+
+#endif
+
+
+/* Declarations for the tables of function pointers exposed by each IO method. */
+extern PGDLLIMPORT const IoMethodOps pgaio_sync_ops;
+
+extern PGDLLIMPORT const IoMethodOps *pgaio_method_ops;
+extern PGDLLIMPORT PgAioCtl *pgaio_ctl;
+extern PGDLLIMPORT PgAioBackend *pgaio_my_backend;
+
+
+
+#endif /* AIO_INTERNAL_H */
diff --git a/src/include/storage/aio_types.h b/src/include/storage/aio_types.h
new file mode 100644
index 00000000000..a5cc658efbd
--- /dev/null
+++ b/src/include/storage/aio_types.h
@@ -0,0 +1,117 @@
+/*-------------------------------------------------------------------------
+ *
+ * aio_types.h
+ * AIO related types that are useful to include separately, to reduce the
+ * "include burden".
+ *
+ *
+ * Portions Copyright (c) 1996-2025, PostgreSQL Global Development Group
+ * Portions Copyright (c) 1994, Regents of the University of California
+ *
+ * src/include/storage/aio_types.h
+ *
+ *-------------------------------------------------------------------------
+ */
+#ifndef AIO_TYPES_H
+#define AIO_TYPES_H
+
+#include "storage/block.h"
+#include "storage/relfilelocator.h"
+
+
+typedef struct PgAioHandle PgAioHandle;
+typedef struct PgAioHandleCallbacks PgAioHandleCallbacks;
+typedef struct PgAioTargetInfo PgAioTargetInfo;
+
+/*
+ * A reference to an IO that can be used to wait for the IO (using
+ * pgaio_wref_wait()) to complete.
+ *
+ * These can be passed across process boundaries.
+ */
+typedef struct PgAioWaitRef
+{
+ /* internal ID identifying the specific PgAioHandle */
+ uint32 aio_index;
+
+ /*
+ * IO handles are reused. To detect if a handle was reused, and thereby
+ * avoid unnecessarily waiting for a newer IO, each time the handle is
+ * reused a generation number is increased.
+ *
+ * To avoid requiring alignment sufficient for an int64, split the
+ * generation into two.
+ */
+ uint32 generation_upper;
+ uint32 generation_lower;
+} PgAioWaitRef;
+
+
+/*
+ * Information identifying what the IO is being performed on.
+ *
+ * This needs sufficient information to
+ *
+ * a) Reopen the file for the IO if the IO is executed in a context that
+ * cannot use the FD provided initially (e.g. because the IO is executed in
+ * a worker process).
+ *
+ * b) Describe the object the IO is performed on in log / error messages.
+ */
+typedef union PgAioTargetData
+{
+ /* just as an example placeholder for later */
+ struct
+ {
+ uint32 queue_id;
+ } wal;
+} PgAioTargetData;
+
+
+/*
+ * The status of an AIO operation.
+ */
+typedef enum PgAioResultStatus
+{
+ ARS_UNKNOWN, /* not yet completed / uninitialized */
+ ARS_OK,
+ ARS_PARTIAL, /* did not fully succeed, but no error */
+ ARS_ERROR,
+} PgAioResultStatus;
+
+
+/*
+ * Result of IO operation, visible only to the initiator of IO.
+ */
+typedef struct PgAioResult
+{
+ /*
+ * This is of type PgAioHandleCallbackID, but can't use a bitfield of an
+ * enum, because some compilers treat enums as signed.
+ */
+ uint32 id:8;
+
+ /* of type PgAioResultStatus, see above */
+ uint32 status:2;
+
+ /* meaning defined by callback->error */
+ uint32 error_data:22;
+
+ int32 result;
+} PgAioResult;
+
+
+/*
+ * Combination of PgAioResult with minimal metadata about the IO.
+ *
+ * Contains sufficient information to be able, in case the IO [partially]
+ * fails, to log/raise an error under control of the IO issuing code.
+ */
+typedef struct PgAioReturn
+{
+ PgAioResult result;
+ PgAioTargetData target_data;
+} PgAioReturn;
+
+
+#endif /* AIO_TYPES_H */
diff --git a/src/backend/storage/aio/Makefile b/src/backend/storage/aio/Makefile
index eaeaeeee8e3..89f821ea7e1 100644
--- a/src/backend/storage/aio/Makefile
+++ b/src/backend/storage/aio/Makefile
@@ -10,7 +10,11 @@ include $(top_builddir)/src/Makefile.global
OBJS = \
aio.o \
+ aio_callback.o \
aio_init.o \
+ aio_io.o \
+ aio_target.o \
+ method_sync.o \
read_stream.o
include $(top_srcdir)/src/backend/common.mk
diff --git a/src/backend/storage/aio/aio.c b/src/backend/storage/aio/aio.c
index 828a94efdc3..26dd23cfab5 100644
--- a/src/backend/storage/aio/aio.c
+++ b/src/backend/storage/aio/aio.c
@@ -3,6 +3,28 @@
* aio.c
* AIO - Core Logic
*
+ * For documentation about how AIO works on a higher level, including a
+ * schematic example, see README.md.
+ *
+ *
+ * AIO is a complicated subsystem. To keep things navigable it is split across
+ * a number of files:
+ *
+ * - method_*.c - different ways of executing AIO (e.g. worker process)
+ *
+ * - aio_target.c - IO on different kinds of targets
+ *
+ * - aio_io.c - method-independent code for specific IO ops (e.g. readv)
+ *
+ * - aio_callback.c - callbacks at IO operation lifecycle events
+ *
+ * - aio_init.c - per-server and per-backend initialization
+ *
+ * - aio.c - all other topics
+ *
+ * - read_stream.c - helper for reading buffered relation data
+ *
+ *
* Portions Copyright (c) 1996-2025, PostgreSQL Global Development Group
* Portions Copyright (c) 1994, Regents of the University of California
*
@@ -15,10 +37,28 @@
#include "postgres.h"
#include "lib/ilist.h"
+#include "miscadmin.h"
+#include "port/atomics.h"
#include "storage/aio.h"
+#include "storage/aio_internal.h"
#include "storage/aio_subsys.h"
#include "utils/guc.h"
#include "utils/guc_hooks.h"
+#include "utils/resowner.h"
+#include "utils/wait_event_types.h"
+
+#ifdef USE_INJECTION_POINTS
+#include "utils/injection_point.h"
+#endif
+
+
+static inline void pgaio_io_update_state(PgAioHandle *ioh, PgAioHandleState new_state);
+static void pgaio_io_reclaim(PgAioHandle *ioh);
+static void pgaio_io_resowner_register(PgAioHandle *ioh);
+static void pgaio_io_wait_for_free(void);
+static PgAioHandle *pgaio_io_from_wref(PgAioWaitRef *iow, uint64 *ref_generation);
+static const char *pgaio_io_state_get_name(PgAioHandleState s);
+static void pgaio_io_wait(PgAioHandle *ioh, uint64 ref_generation);
/* Options for io_method. */
@@ -31,7 +71,182 @@ const struct config_enum_entry io_method_options[] = {
int io_method = DEFAULT_IO_METHOD;
int io_max_concurrency = -1;
+/* global control for AIO */
+PgAioCtl *pgaio_ctl;
+/* current backend's per-backend state */
+PgAioBackend *pgaio_my_backend;
+
+
+static const IoMethodOps *const pgaio_method_ops_table[] = {
+ [IOMETHOD_SYNC] = &pgaio_sync_ops,
+};
+
+/* callbacks for the configured io_method, set by assign_io_method */
+const IoMethodOps *pgaio_method_ops;
+
+
+/*
+ * Currently there's no infrastructure to pass arguments to injection points,
+ * so we instead set this up for the duration of the injection point
+ * invocation. See pgaio_io_call_inj().
+ */
+#ifdef USE_INJECTION_POINTS
+static PgAioHandle *pgaio_inj_cur_handle;
+#endif
+
+
+
+/* --------------------------------------------------------------------------------
+ * Public Functions related to PgAioHandle
+ * --------------------------------------------------------------------------------
+ */
+
+/*
+ * Acquire an AioHandle, waiting for IO completion if necessary.
+ *
+ * Each backend can only have one AIO handle that that has been "handed out"
+ * to code, but not yet submitted or released. This restriction is necessary
+ * to ensure that it is possible for code to wait for an unused handle by
+ * waiting for in-flight IO to complete. There is a limited number of handles
+ * in each backend, if multiple handles could be handed out without being
+ * submitted, waiting for all in-flight IO to complete would not guarantee
+ * that handles free up.
+ *
+ * It is cheap to acquire an IO handle, unless all handles are in use. In that
+ * case this function waits for the oldest IO to complete. In case that is not
+ * desirable, see pgaio_io_acquire_nb().
+ *
+ * If a handle was acquired but then does not turn out to be needed,
+ * e.g. because pgaio_io_acquire() is called before starting an IO in a
+ * critical section, the handle needs to be released with pgaio_io_release().
+ *
+ *
+ * To react to the completion of the IO as soon as it is known to have
+ * completed, callbacks can be registered with pgaio_io_register_callbacks().
+ *
+ * To actually execute IO using the returned handle, the pgaio_io_prep_*()
+ * family of functions is used. In many cases the pgaio_io_prep_*() call will
+ * not be done directly by code that acquired the handle, but by lower level
+ * code that gets passed the handle. E.g. if code in bufmgr.c wants to perform
+ * AIO, it typically will pass the handle to smgr., which will pass it on to
+ * md.c, on to fd.c, which then finally calls pgaio_io_prep_*(). This
+ * forwarding allows the various layers to react to the IO's completion by
+ * registering callbacks. These callbacks in turn can translate a lower
+ * layer's result into a result understandable by a higher layer.
+ *
+ * During pgaio_io_prep_*() the IO is staged (i.e. prepared for execution but
+ * not submitted to the kernel). Unless in batchmode
+ * (c.f. pgaio_enter_batchmode()) the IO will also have been submitted for
+ * execution. Note that, whether in batchmode or not, the IO might even
+ * complete before the functions return.
+ *
+ * After pgaio_io_prep_*() the AioHandle is "consumed" and may not be
+ * referenced by the IO issuing code. To e.g. wait for IO, references to the
+ * IO can be established with pgaio_io_get_wref() *before* pgaio_io_prep_*()
+ * is called. pgaio_wref_wait() can be used to wait for the IO to complete.
+ *
+ *
+ * To know if the IO [partially] succeeded or failed, a PgAioReturn * can be
+ * passed to pgaio_io_acquire(). Once the issuing backend has called
+ * pgaio_wref_wait(), the PgAioReturn contains information about whether the
+ * operation succeeded and details about the first failure, if any. The error
+ * can be raised / logged with pgaio_result_report().
+ *
+ * The lifetime of the memory pointed to be *ret needs to be at least as long
+ * as the passed in resowner. If the resowner releases resources before the IO
+ * completes (typically due to an error), the reference to *ret will be
+ * cleared. In case of resowner cleanup *ret will not be updated with the
+ * results of the IO operation.
+ */
+PgAioHandle *
+pgaio_io_acquire(struct ResourceOwnerData *resowner, PgAioReturn *ret)
+{
+ PgAioHandle *h;
+
+ while (true)
+ {
+ h = pgaio_io_acquire_nb(resowner, ret);
+
+ if (h != NULL)
+ return h;
+
+ /*
+ * Evidently all handles by this backend are in use. Just wait for
+ * some to complete.
+ */
+ pgaio_io_wait_for_free();
+ }
+}
+
+/*
+ * Acquire an AioHandle, returning NULL if no handles are free.
+ *
+ * See pgaio_io_acquire(). The only difference is that this function will return
+ * NULL if there are no idle handles, instead of blocking.
+ */
+PgAioHandle *
+pgaio_io_acquire_nb(struct ResourceOwnerData *resowner, PgAioReturn *ret)
+{
+ if (pgaio_my_backend->num_staged_ios >= PGAIO_SUBMIT_BATCH_SIZE)
+ {
+ Assert(pgaio_my_backend->num_staged_ios == PGAIO_SUBMIT_BATCH_SIZE);
+ pgaio_submit_staged();
+ }
+
+ if (pgaio_my_backend->handed_out_io)
+ {
+ ereport(ERROR,
+ errmsg("API violation: Only one IO can be handed out"));
+ }
+
+ if (!dclist_is_empty(&pgaio_my_backend->idle_ios))
+ {
+ dlist_node *ion = dclist_pop_head_node(&pgaio_my_backend->idle_ios);
+ PgAioHandle *ioh = dclist_container(PgAioHandle, node, ion);
+
+ Assert(ioh->state == PGAIO_HS_IDLE);
+ Assert(ioh->owner_procno == MyProcNumber);
+
+ pgaio_io_update_state(ioh, PGAIO_HS_HANDED_OUT);
+ pgaio_my_backend->handed_out_io = ioh;
+
+ if (resowner)
+ pgaio_io_resowner_register(ioh);
+
+ if (ret)
+ {
+ ioh->report_return = ret;
+ ret->result.status = ARS_UNKNOWN;
+ }
+
+ return ioh;
+ }
+
+ return NULL;
+}
+
+/*
+ * Release IO handle that turned out to not be required.
+ *
+ * See pgaio_io_acquire() for more details.
+ */
+void
+pgaio_io_release(PgAioHandle *ioh)
+{
+ if (ioh == pgaio_my_backend->handed_out_io)
+ {
+ Assert(ioh->state == PGAIO_HS_HANDED_OUT);
+ Assert(ioh->resowner);
+
+ pgaio_my_backend->handed_out_io = NULL;
+ pgaio_io_reclaim(ioh);
+ }
+ else
+ {
+ elog(ERROR, "release in unexpected state");
+ }
+}
/*
* Release IO handle during resource owner cleanup.
@@ -39,8 +254,793 @@ int io_max_concurrency = -1;
void
pgaio_io_release_resowner(dlist_node *ioh_node, bool on_error)
{
+ PgAioHandle *ioh = dlist_container(PgAioHandle, resowner_node, ioh_node);
+
+ Assert(ioh->resowner);
+
+ ResourceOwnerForgetAioHandle(ioh->resowner, &ioh->resowner_node);
+ ioh->resowner = NULL;
+
+ switch (ioh->state)
+ {
+ case PGAIO_HS_IDLE:
+ elog(ERROR, "unexpected");
+ break;
+ case PGAIO_HS_HANDED_OUT:
+ Assert(ioh == pgaio_my_backend->handed_out_io || pgaio_my_backend->handed_out_io == NULL);
+
+ if (ioh == pgaio_my_backend->handed_out_io)
+ {
+ pgaio_my_backend->handed_out_io = NULL;
+ if (!on_error)
+ elog(WARNING, "leaked AIO handle");
+ }
+
+ pgaio_io_reclaim(ioh);
+ break;
+ case PGAIO_HS_DEFINED:
+ case PGAIO_HS_STAGED:
+ if (!on_error)
+ elog(WARNING, "AIO handle was not submitted");
+ pgaio_submit_staged();
+ break;
+ case PGAIO_HS_SUBMITTED:
+ case PGAIO_HS_COMPLETED_IO:
+ case PGAIO_HS_COMPLETED_SHARED:
+ case PGAIO_HS_COMPLETED_LOCAL:
+ /* this is expected to happen */
+ break;
+ }
+
+ /*
+ * Need to unregister the reporting of the IO's result, the memory it's
+ * referencing likely has gone away.
+ */
+ if (ioh->report_return)
+ ioh->report_return = NULL;
}
+/*
+ * Add a [set of] flags to the IO.
+ *
+ * Note that this combines flags with already set flags, rather than set flags
+ * to explicitly the passed in parameters. This is to allow multiple callsites
+ * to set flags.
+ */
+void
+pgaio_io_set_flag(PgAioHandle *ioh, PgAioHandleFlags flag)
+{
+ Assert(ioh->state == PGAIO_HS_HANDED_OUT);
+
+ ioh->flags |= flag;
+}
+
+/*
+ * Returns an ID uniquely identifying the IO handle. This is only really
+ * useful for logging, as handles are reused across multiple IOs.
+ */
+int
+pgaio_io_get_id(PgAioHandle *ioh)
+{
+ Assert(ioh >= pgaio_ctl->io_handles &&
+ ioh < (pgaio_ctl->io_handles + pgaio_ctl->io_handle_count));
+ return ioh - pgaio_ctl->io_handles;
+}
+
+/*
+ * Return the ProcNumber for the process that can use an IO handle. The
+ * mapping from IO handles to PGPROCs is static, therefore this even works
+ * when the corresponding PGPROC is not in use.
+ */
+ProcNumber
+pgaio_io_get_owner(PgAioHandle *ioh)
+{
+ return ioh->owner_procno;
+}
+
+/*
+ * Return a wait reference for the IO. Only wait references can be used to
+ * wait for an IOs completion, as handles themselves can be reused after
+ * completion. See also the comment above pgaio_io_acquire().
+ */
+void
+pgaio_io_get_wref(PgAioHandle *ioh, PgAioWaitRef *iow)
+{
+ Assert(ioh->state == PGAIO_HS_HANDED_OUT ||
+ ioh->state == PGAIO_HS_DEFINED ||
+ ioh->state == PGAIO_HS_STAGED);
+ Assert(ioh->generation != 0);
+
+ iow->aio_index = ioh - pgaio_ctl->io_handles;
+ iow->generation_upper = (uint32) (ioh->generation >> 32);
+ iow->generation_lower = (uint32) ioh->generation;
+}
+
+
+
+/* --------------------------------------------------------------------------------
+ * Internal Functions related to PgAioHandle
+ * --------------------------------------------------------------------------------
+ */
+
+static inline void
+pgaio_io_update_state(PgAioHandle *ioh, PgAioHandleState new_state)
+{
+ pgaio_debug_io(DEBUG5, ioh,
+ "updating state to %s",
+ pgaio_io_state_get_name(new_state));
+
+ /*
+ * Ensure the changes signified by the new state are visible before the
+ * new state becomes visible.
+ */
+ pg_write_barrier();
+
+ ioh->state = new_state;
+}
+
+static void
+pgaio_io_resowner_register(PgAioHandle *ioh)
+{
+ Assert(!ioh->resowner);
+ Assert(CurrentResourceOwner);
+
+ ResourceOwnerRememberAioHandle(CurrentResourceOwner, &ioh->resowner_node);
+ ioh->resowner = CurrentResourceOwner;
+}
+
+/*
+ * Stage IO for execution and, if necessary, submit it immediately.
+ *
+ * Should only be called from pgaio_io_prep_*().
+ */
+void
+pgaio_io_stage(PgAioHandle *ioh, PgAioOp op)
+{
+ bool needs_synchronous;
+
+ Assert(ioh->state == PGAIO_HS_HANDED_OUT);
+ Assert(pgaio_io_has_target(ioh));
+
+ ioh->op = op;
+ ioh->result = 0;
+
+ pgaio_io_update_state(ioh, PGAIO_HS_DEFINED);
+
+ /* allow a new IO to be staged */
+ pgaio_my_backend->handed_out_io = NULL;
+
+ pgaio_io_call_stage(ioh);
+
+ pgaio_io_update_state(ioh, PGAIO_HS_STAGED);
+
+ /*
+ * Synchronous execution has to be executed, well, synchronously, so check
+ * that first.
+ */
+ needs_synchronous = pgaio_io_needs_synchronous_execution(ioh);
+
+ pgaio_debug_io(DEBUG3, ioh,
+ "prepared (synchronous: %d, in_batch: %d)",
+ needs_synchronous, pgaio_my_backend->in_batchmode);
+
+ if (!needs_synchronous)
+ {
+ pgaio_my_backend->staged_ios[pgaio_my_backend->num_staged_ios++] = ioh;
+ Assert(pgaio_my_backend->num_staged_ios <= PGAIO_SUBMIT_BATCH_SIZE);
+
+ /*
+ * Unless code explicitly opted into batching IOs, submit the IO
+ * immediately.
+ */
+ if (!pgaio_my_backend->in_batchmode)
+ pgaio_submit_staged();
+ }
+ else
+ {
+ pgaio_io_prepare_submit(ioh);
+ pgaio_io_perform_synchronously(ioh);
+ }
+}
+
+bool
+pgaio_io_needs_synchronous_execution(PgAioHandle *ioh)
+{
+ /*
+ * If the caller said to execute the IO synchronously, do so.
+ *
+ * XXX: We could optimize the logic when to execute synchronously by first
+ * checking if there are other IOs in flight and only synchronously
+ * executing if not. Unclear whether that'll be sufficiently common to be
+ * worth worrying about.
+ */
+ if (ioh->flags & PGAIO_HF_SYNCHRONOUS)
+ return true;
+
+ /* Check if the IO method requires synchronous execution of IO */
+ if (pgaio_method_ops->needs_synchronous_execution)
+ return pgaio_method_ops->needs_synchronous_execution(ioh);
+
+ return false;
+}
+
+/*
+ * Handle IO being processed by IO method.
+ *
+ * Should be called by IO methods / synchronous IO execution, just before the
+ * IO is performed.
+ */
+void
+pgaio_io_prepare_submit(PgAioHandle *ioh)
+{
+ pgaio_io_update_state(ioh, PGAIO_HS_SUBMITTED);
+
+ dclist_push_tail(&pgaio_my_backend->in_flight_ios, &ioh->node);
+}
+
+/*
+ * Handle IO getting completed by a method.
+ *
+ * Should be called by IO methods / synchronous IO execution, just after the
+ * IO has been performed.
+ *
+ * Expects to be called in a critical section. We expect IOs to be usable for
+ * WAL etc, which requires being able to execute completion callbacks in a
+ * critical section.
+ */
+void
+pgaio_io_process_completion(PgAioHandle *ioh, int result)
+{
+ Assert(ioh->state == PGAIO_HS_SUBMITTED);
+
+ Assert(CritSectionCount > 0);
+
+ ioh->result = result;
+
+ pgaio_io_update_state(ioh, PGAIO_HS_COMPLETED_IO);
+
+ pgaio_io_call_inj(ioh, "AIO_PROCESS_COMPLETION_BEFORE_SHARED");
+
+ pgaio_io_call_complete_shared(ioh);
+
+ pgaio_io_update_state(ioh, PGAIO_HS_COMPLETED_SHARED);
+
+ /* condition variable broadcast ensures state is visible before wakeup */
+ ConditionVariableBroadcast(&ioh->cv);
+
+ /* contains call to pgaio_io_call_complete_local() */
+ if (ioh->owner_procno == MyProcNumber)
+ pgaio_io_reclaim(ioh);
+}
+
+/*
+ * Has the IO completed and thus the IO handle been reused?
+ *
+ * This is useful when waiting for IO completion at a low level (e.g. in an IO
+ * method's ->wait_one() callback).
+ */
+bool
+pgaio_io_was_recycled(PgAioHandle *ioh, uint64 ref_generation, PgAioHandleState *state)
+{
+ *state = ioh->state;
+ pg_read_barrier();
+
+ return ioh->generation != ref_generation;
+}
+
+/*
+ * Wait for IO to complete. External code should never use this, outside of
+ * the AIO subsystem waits are only allowed via pgaio_wref_wait().
+ */
+static void
+pgaio_io_wait(PgAioHandle *ioh, uint64 ref_generation)
+{
+ PgAioHandleState state;
+ bool am_owner;
+
+ am_owner = ioh->owner_procno == MyProcNumber;
+
+ if (pgaio_io_was_recycled(ioh, ref_generation, &state))
+ return;
+
+ if (am_owner)
+ {
+ if (state != PGAIO_HS_SUBMITTED
+ && state != PGAIO_HS_COMPLETED_IO
+ && state != PGAIO_HS_COMPLETED_SHARED
+ && state != PGAIO_HS_COMPLETED_LOCAL)
+ {
+ elog(PANIC, "waiting for own IO in wrong state: %d",
+ state);
+ }
+ }
+
+ while (true)
+ {
+ if (pgaio_io_was_recycled(ioh, ref_generation, &state))
+ return;
+
+ switch (state)
+ {
+ case PGAIO_HS_IDLE:
+ case PGAIO_HS_HANDED_OUT:
+ elog(ERROR, "IO in wrong state: %d", state);
+ break;
+
+ case PGAIO_HS_SUBMITTED:
+
+ /*
+ * If we need to wait via the IO method, do so now. Don't
+ * check via the IO method if the issuing backend is executing
+ * the IO synchronously.
+ */
+ if (pgaio_method_ops->wait_one && !(ioh->flags & PGAIO_HF_SYNCHRONOUS))
+ {
+ pgaio_method_ops->wait_one(ioh, ref_generation);
+ continue;
+ }
+ /* fallthrough */
+
+ /* waiting for owner to submit */
+ case PGAIO_HS_DEFINED:
+ case PGAIO_HS_STAGED:
+ /* waiting for reaper to complete */
+ /* fallthrough */
+ case PGAIO_HS_COMPLETED_IO:
+ /* shouldn't be able to hit this otherwise */
+ Assert(IsUnderPostmaster);
+ /* ensure we're going to get woken up */
+ ConditionVariablePrepareToSleep(&ioh->cv);
+
+ while (!pgaio_io_was_recycled(ioh, ref_generation, &state))
+ {
+ if (state == PGAIO_HS_COMPLETED_SHARED ||
+ state == PGAIO_HS_COMPLETED_LOCAL)
+ break;
+ ConditionVariableSleep(&ioh->cv, WAIT_EVENT_AIO_IO_COMPLETION);
+ }
+
+ ConditionVariableCancelSleep();
+ break;
+
+ case PGAIO_HS_COMPLETED_SHARED:
+ case PGAIO_HS_COMPLETED_LOCAL:
+ /* see above */
+ if (am_owner)
+ pgaio_io_reclaim(ioh);
+ return;
+ }
+ }
+}
+
+/*
+ * Make IO handle ready to be reused after IO has completed or after the
+ * handle has been released without being used.
+ */
+static void
+pgaio_io_reclaim(PgAioHandle *ioh)
+{
+ /* This is only ok if it's our IO */
+ Assert(ioh->owner_procno == MyProcNumber);
+
+ /*
+ * It's a bit ugly, but right now the easiest place to put the execution
+ * of shared completion callbacks is this function, as we need to execute
+ * locallbacks just before reclaiming at multiple callsites.
+ */
+ if (ioh->state == PGAIO_HS_COMPLETED_SHARED)
+ {
+ pgaio_io_call_complete_local(ioh);
+ pgaio_io_update_state(ioh, PGAIO_HS_COMPLETED_LOCAL);
+ }
+
+ pgaio_debug_io(DEBUG4, ioh,
+ "reclaiming: distilled_result: (status %s, id %u, error_data %d), raw_result: %d",
+ pgaio_result_status_string(ioh->distilled_result.status),
+ ioh->distilled_result.id,
+ ioh->distilled_result.error_data,
+ ioh->result);
+
+ /* if the IO has been defined, we might need to do more work */
+ if (ioh->state != PGAIO_HS_HANDED_OUT)
+ {
+ dclist_delete_from(&pgaio_my_backend->in_flight_ios, &ioh->node);
+
+ if (ioh->report_return)
+ {
+ ioh->report_return->result = ioh->distilled_result;
+ ioh->report_return->target_data = ioh->target_data;
+ }
+ }
+
+ if (ioh->resowner)
+ {
+ ResourceOwnerForgetAioHandle(ioh->resowner, &ioh->resowner_node);
+ ioh->resowner = NULL;
+ }
+
+ Assert(!ioh->resowner);
+
+ ioh->op = PGAIO_OP_INVALID;
+ ioh->target = PGAIO_TID_INVALID;
+ ioh->flags = 0;
+ ioh->num_callbacks = 0;
+ ioh->handle_data_len = 0;
+ ioh->report_return = NULL;
+ ioh->result = 0;
+ ioh->distilled_result.status = ARS_UNKNOWN;
+
+ /* XXX: the barrier is probably superfluous */
+ pg_write_barrier();
+ ioh->generation++;
+
+ pgaio_io_update_state(ioh, PGAIO_HS_IDLE);
+
+ /*
+ * We push the IO to the head of the idle IO list, that seems more cache
+ * efficient in cases where only a few IOs are used.
+ */
+ dclist_push_head(&pgaio_my_backend->idle_ios, &ioh->node);
+}
+
+/*
+ * Wait for an IO handle to become usable.
+ *
+ * This only really is useful for pgaio_io_acquire().
+ */
+static void
+pgaio_io_wait_for_free(void)
+{
+ int reclaimed = 0;
+
+ pgaio_debug(DEBUG2, "waiting for self with %d pending",
+ pgaio_my_backend->num_staged_ios);
+
+ /*
+ * First check if any of our IOs actually have completed - when using
+ * worker, that'll often be the case. We could do so as part of the loop
+ * below, but that'd potentially lead us to wait for some IO submitted
+ * before.
+ */
+ for (int i = 0; i < io_max_concurrency; i++)
+ {
+ PgAioHandle *ioh = &pgaio_ctl->io_handles[pgaio_my_backend->io_handle_off + i];
+
+ if (ioh->state == PGAIO_HS_COMPLETED_SHARED)
+ {
+ pgaio_io_reclaim(ioh);
+ reclaimed++;
+ }
+ }
+
+ if (reclaimed > 0)
+ return;
+
+ /*
+ * If we have any unsubmitted IOs, submit them now. We'll start waiting in
+ * a second, so it's better they're in flight. This also addresses the
+ * edge-case that all IOs are unsubmitted.
+ */
+ if (pgaio_my_backend->num_staged_ios > 0)
+ pgaio_submit_staged();
+
+ if (dclist_count(&pgaio_my_backend->in_flight_ios) == 0)
+ elog(ERROR, "no free IOs despite no in-flight IOs");
+
+ /*
+ * Wait for the oldest in-flight IO to complete.
+ *
+ * XXX: Reusing the general IO wait is suboptimal, we don't need to wait
+ * for that specific IO to complete, we just need *any* IO to complete.
+ */
+ {
+ PgAioHandle *ioh = dclist_head_element(PgAioHandle, node, &pgaio_my_backend->in_flight_ios);
+
+ switch (ioh->state)
+ {
+ /* should not be in in-flight list */
+ case PGAIO_HS_IDLE:
+ case PGAIO_HS_DEFINED:
+ case PGAIO_HS_HANDED_OUT:
+ case PGAIO_HS_STAGED:
+ case PGAIO_HS_COMPLETED_LOCAL:
+ elog(ERROR, "shouldn't get here with io:%d in state %d",
+ pgaio_io_get_id(ioh), ioh->state);
+ break;
+
+ case PGAIO_HS_COMPLETED_IO:
+ case PGAIO_HS_SUBMITTED:
+ pgaio_debug_io(DEBUG2, ioh,
+ "waiting for free io with %d in flight",
+ dclist_count(&pgaio_my_backend->in_flight_ios));
+
+ /*
+ * In a more general case this would be racy, because the
+ * generation could increase after we read ioh->state above.
+ * But we are only looking at IOs by the current backend and
+ * the IO can only be recycled by this backend.
+ */
+ pgaio_io_wait(ioh, ioh->generation);
+ break;
+
+ case PGAIO_HS_COMPLETED_SHARED:
+ /* it's possible that another backend just finished this IO */
+ pgaio_io_reclaim(ioh);
+ break;
+ }
+
+ if (dclist_count(&pgaio_my_backend->idle_ios) == 0)
+ elog(PANIC, "no idle IOs after waiting");
+ return;
+ }
+}
+
+/*
+ * Internal - code outside of AIO should never need this and it'd be hard for
+ * such code to be safe.
+ */
+static PgAioHandle *
+pgaio_io_from_wref(PgAioWaitRef *iow, uint64 *ref_generation)
+{
+ PgAioHandle *ioh;
+
+ Assert(iow->aio_index < pgaio_ctl->io_handle_count);
+
+ ioh = &pgaio_ctl->io_handles[iow->aio_index];
+
+ *ref_generation = ((uint64) iow->generation_upper) << 32 |
+ iow->generation_lower;
+
+ Assert(*ref_generation != 0);
+
+ return ioh;
+}
+
+static const char *
+pgaio_io_state_get_name(PgAioHandleState s)
+{
+#define PGAIO_HS_TOSTR_CASE(sym) case PGAIO_HS_##sym: return #sym
+ switch (s)
+ {
+ PGAIO_HS_TOSTR_CASE(IDLE);
+ PGAIO_HS_TOSTR_CASE(HANDED_OUT);
+ PGAIO_HS_TOSTR_CASE(DEFINED);
+ PGAIO_HS_TOSTR_CASE(STAGED);
+ PGAIO_HS_TOSTR_CASE(SUBMITTED);
+ PGAIO_HS_TOSTR_CASE(COMPLETED_IO);
+ PGAIO_HS_TOSTR_CASE(COMPLETED_SHARED);
+ PGAIO_HS_TOSTR_CASE(COMPLETED_LOCAL);
+ }
+#undef PGAIO_HS_TOSTR_CASE
+
+ return NULL; /* silence compiler */
+}
+
+const char *
+pgaio_io_get_state_name(PgAioHandle *ioh)
+{
+ return pgaio_io_state_get_name(ioh->state);
+}
+
+const char *
+pgaio_result_status_string(PgAioResultStatus rs)
+{
+ switch (rs)
+ {
+ case ARS_UNKNOWN:
+ return "UNKNOWN";
+ case ARS_OK:
+ return "OK";
+ case ARS_PARTIAL:
+ return "PARTIAL";
+ case ARS_ERROR:
+ return "ERROR";
+ }
+
+ return NULL; /* silence compiler */
+}
+
+
+
+/* --------------------------------------------------------------------------------
+ * Functions primarily related to IO Wait References
+ * --------------------------------------------------------------------------------
+ */
+
+/*
+ * Mark a wait reference as invalid
+ */
+void
+pgaio_wref_clear(PgAioWaitRef *iow)
+{
+ iow->aio_index = PG_UINT32_MAX;
+}
+
+/* Is the wait reference valid? */
+bool
+pgaio_wref_valid(PgAioWaitRef *iow)
+{
+ return iow->aio_index != PG_UINT32_MAX;
+}
+
+/*
+ * Similar to pgaio_io_get_id(), just for wait references.
+ */
+int
+pgaio_wref_get_id(PgAioWaitRef *iow)
+{
+ Assert(pgaio_wref_valid(iow));
+ return iow->aio_index;
+}
+
+/*
+ * Wait for the IO to have completed. Can be called in any process, not just
+ * in the issuing backend.
+ */
+void
+pgaio_wref_wait(PgAioWaitRef *iow)
+{
+ uint64 ref_generation;
+ PgAioHandle *ioh;
+
+ ioh = pgaio_io_from_wref(iow, &ref_generation);
+
+ pgaio_io_wait(ioh, ref_generation);
+}
+
+/*
+ * Check if the the referenced IO completed, without blocking.
+ */
+bool
+pgaio_wref_check_done(PgAioWaitRef *iow)
+{
+ uint64 ref_generation;
+ PgAioHandleState state;
+ bool am_owner;
+ PgAioHandle *ioh;
+
+ ioh = pgaio_io_from_wref(iow, &ref_generation);
+
+ if (pgaio_io_was_recycled(ioh, ref_generation, &state))
+ return true;
+
+ if (state == PGAIO_HS_IDLE)
+ return true;
+
+ am_owner = ioh->owner_procno == MyProcNumber;
+
+ if (state == PGAIO_HS_COMPLETED_SHARED ||
+ state == PGAIO_HS_COMPLETED_LOCAL)
+ {
+ if (am_owner)
+ pgaio_io_reclaim(ioh);
+ return true;
+ }
+
+ /*
+ * XXX: It likely would be worth checking in with the io method, to give
+ * the IO method a chance to check if there are completion events queued.
+ */
+
+ return false;
+}
+
+
+
+/* --------------------------------------------------------------------------------
+ * Actions on multiple IOs.
+ * --------------------------------------------------------------------------------
+ */
+
+/*
+ * Submit IOs in batches going forward.
+ *
+ * Submitting multiple IOs at once can be substantially faster than doing so
+ * one-by-one. At the same time submitting multiple IOs at once requires more
+ * care to avoid deadlocks.
+ *
+ * Consider backend A staging an IO for buffer 1 and then trying to start IO
+ * on buffer 2, while backend B does the inverse. If A submitted the IO before
+ * moving on to buffer 2, this works just fine, B will wait for the IO to
+ * complete. But if batching were used, each backend will wait for IO that has
+ * not yet been submitted to complete, i.e. forever.
+ *
+ * Batch submission mode needs to explicitly ended with
+ * pgaio_exit_batchmode(), but it is allowed to throw errors, in which case
+ * error recovery will end the batch.
+ *
+ * To avoid deadlocks, code needs to ensure that it will not wait for another
+ * backend while there is unsubmitted IO. E.g. by using conditional lock
+ * acquisition when acquiring buffer locks. To check if there currently are
+ * staged IOs, call pgaio_have_staged() and to submit all staged IOs call
+ * pgaio_submit_staged().
+ *
+ * It is not allowed to enter batchmode while already in batchmode, it's
+ * unlikely to ever be needed, as code needs to be explicitly aware of being
+ * called in batchmode, to avoid the deadlock risks explained above.
+ *
+ * Note that IOs may get submitted before pgaio_exit_batchmode() is called,
+ * e.g. because too many IOs have been staged or because pgaio_submit_staged()
+ * was called.
+ */
+void
+pgaio_enter_batchmode(void)
+{
+ if (pgaio_my_backend->in_batchmode)
+ elog(ERROR, "starting batch while batch already in progress");
+ pgaio_my_backend->in_batchmode = true;
+}
+
+/*
+ * Stop submitting IOs in batches.
+ */
+void
+pgaio_exit_batchmode(void)
+{
+ Assert(pgaio_my_backend->in_batchmode);
+
+ pgaio_submit_staged();
+ pgaio_my_backend->in_batchmode = false;
+}
+
+/*
+ * Are there staged but unsubmitted IOs?
+ *
+ * See comment above pgaio_enter_batchmode() for why code may need to check if
+ * there is IO in that state.
+ */
+bool
+pgaio_have_staged(void)
+{
+ Assert(pgaio_my_backend->in_batchmode ||
+ pgaio_my_backend->num_staged_ios == 0);
+ return pgaio_my_backend->num_staged_ios > 0;
+}
+
+/*
+ * Submit all staged but not yet submitted IOs.
+ *
+ * Unless in batch mode, this never needs to be called, as IOs get submitted
+ * as soon as possible. While in batchmode pgaio_submit_staged() can be called
+ * before waiting on another backend, to avoid the risk of deadlocks. See
+ * pgaio_enter_batchmode().
+ */
+void
+pgaio_submit_staged(void)
+{
+ int total_submitted = 0;
+ int did_submit;
+
+ if (pgaio_my_backend->num_staged_ios == 0)
+ return;
+
+
+ START_CRIT_SECTION();
+
+ did_submit = pgaio_method_ops->submit(pgaio_my_backend->num_staged_ios,
+ pgaio_my_backend->staged_ios);
+
+ END_CRIT_SECTION();
+
+ total_submitted += did_submit;
+
+ Assert(total_submitted == did_submit);
+
+ pgaio_my_backend->num_staged_ios = 0;
+
+ pgaio_debug(DEBUG4,
+ "aio: submitted %d IOs",
+ total_submitted);
+}
+
+
+
+/* --------------------------------------------------------------------------------
+ * Other
+ * --------------------------------------------------------------------------------
+ */
+
+
/*
* Perform AIO related cleanup after an error.
*
@@ -50,6 +1050,22 @@ pgaio_io_release_resowner(dlist_node *ioh_node, bool on_error)
void
pgaio_error_cleanup(void)
{
+ /*
+ * It is possible that code errored out after pgaio_enter_batchmode() but
+ * before pgaio_exit_batchmode() was called. In that case we need to
+ * submit the IO now.
+ */
+ if (pgaio_my_backend->in_batchmode)
+ {
+ pgaio_my_backend->in_batchmode = false;
+
+ pgaio_submit_staged();
+ }
+
+ /*
+ * As we aren't in batchmode, there shouldn't be any unsubmitted IOs.
+ */
+ Assert(pgaio_my_backend->num_staged_ios == 0);
}
/*
@@ -62,11 +1078,82 @@ pgaio_error_cleanup(void)
void
AtEOXact_Aio(bool is_commit)
{
+ /*
+ * We should never be in batch mode at transactional boundaries. In case
+ * an error was thrown while in batch mode, pgaio_error_cleanup() should
+ * have exited batchmode.
+ *
+ * In case we are in batchmode somehow, make sure to submit all staged
+ * IOs, other backends may need them to complete to continue.
+ */
+ if (pgaio_my_backend->in_batchmode)
+ {
+ pgaio_error_cleanup();
+ elog(WARNING, "open AIO batch at end of (sub-)transaction");
+ }
+
+ /*
+ * As we aren't in batchmode, there shouldn't be any unsubmitted IOs.
+ */
+ Assert(pgaio_my_backend->num_staged_ios == 0);
+}
+
+/*
+ * Need to submit staged but not yet submitted IOs using the fd, otherwise
+ * the IO would end up targeting something bogus.
+ */
+void
+pgaio_closing_fd(int fd)
+{
+ /*
+ * Might be called before AIO is initialized or in a subprocess that
+ * doesn't use AIO.
+ */
+ if (!pgaio_my_backend)
+ return;
+
+ /*
+ * For now just submit all staged IOs - we could be more selective, but
+ * it's probably not worth it.
+ */
+ pgaio_submit_staged();
+}
+
+/*
+ * Registered as before_shmem_exit() callback in pgaio_init_backend()
+ */
+void
+pgaio_shutdown(int code, Datum arg)
+{
+ AtEOXact_Aio(code == 0);
+ Assert(pgaio_my_backend);
+ Assert(!pgaio_my_backend->handed_out_io);
+
+ /*
+ * Before exiting, make sure that all IOs are finished. That has two main
+ * purposes: - it's somewhat annoying to see partially finished IOs in
+ * stats views etc - it's rumored that some kernel-level AIO mechanisms
+ * don't deal well with the issuer of an AIO exiting
+ */
+
+ while (!dclist_is_empty(&pgaio_my_backend->in_flight_ios))
+ {
+ PgAioHandle *ioh = dclist_head_element(PgAioHandle, node, &pgaio_my_backend->in_flight_ios);
+
+ /* see comment in pgaio_io_wait_for_free() about raciness */
+ pgaio_io_wait(ioh, ioh->generation);
+ }
+
+ pgaio_my_backend = NULL;
}
void
assign_io_method(int newval, void *extra)
{
+ Assert(pgaio_method_ops_table[newval] != NULL);
+ Assert(newval < lengthof(io_method_options));
+
+ pgaio_method_ops = pgaio_method_ops_table[newval];
}
bool
@@ -88,3 +1175,43 @@ check_io_max_concurrency(int *newval, void **extra, GucSource source)
return true;
}
+
+
+
+/* --------------------------------------------------------------------------------
+ * Injection point support
+ * --------------------------------------------------------------------------------
+ */
+
+#ifdef USE_INJECTION_POINTS
+
+/*
+ * Call injection point with support for pgaio_inj_io_get().
+ */
+void
+pgaio_io_call_inj(PgAioHandle *ioh, const char *injection_point)
+{
+ pgaio_inj_cur_handle = ioh;
+
+ PG_TRY();
+ {
+ InjectionPointCached(injection_point);
+ }
+ PG_FINALLY();
+ {
+ pgaio_inj_cur_handle = NULL;
+ }
+ PG_END_TRY();
+}
+
+/*
+ * Return IO associated with injection point invocation. This is only needed
+ * as injection points currently don't support arguments.
+ */
+PgAioHandle *
+pgaio_inj_io_get(void)
+{
+ return pgaio_inj_cur_handle;
+}
+
+#endif
diff --git a/src/backend/storage/aio/aio_callback.c b/src/backend/storage/aio/aio_callback.c
new file mode 100644
index 00000000000..7392b55322c
--- /dev/null
+++ b/src/backend/storage/aio/aio_callback.c
@@ -0,0 +1,307 @@
+/*-------------------------------------------------------------------------
+ *
+ * aio_callback.c
+ * AIO - Functionality related to callbacks that can be registered on IO
+ * Handles
+ *
+ * Portions Copyright (c) 1996-2025, PostgreSQL Global Development Group
+ * Portions Copyright (c) 1994, Regents of the University of California
+ *
+ * IDENTIFICATION
+ * src/backend/storage/aio/aio_callback.c
+ *
+ *-------------------------------------------------------------------------
+ */
+
+#include "postgres.h"
+
+#include "miscadmin.h"
+#include "storage/aio.h"
+#include "storage/aio_internal.h"
+
+
+/* just to have something to put into the aio_handle_cbs */
+static const struct PgAioHandleCallbacks aio_invalid_cb = {0};
+
+typedef struct PgAioHandleCallbacksEntry
+{
+ const PgAioHandleCallbacks *const cb;
+ const char *const name;
+} PgAioHandleCallbacksEntry;
+
+/*
+ * Callback definition for the callbacks that can be registered on an IO
+ * handle. See PgAioHandleCallbackID's definition for an explanation for why
+ * callbacks are not identified by a pointer.
+ */
+static const PgAioHandleCallbacksEntry aio_handle_cbs[] = {
+#define CALLBACK_ENTRY(id, callback) [id] = {.cb = &callback, .name = #callback}
+ CALLBACK_ENTRY(PGAIO_HCB_INVALID, aio_invalid_cb),
+#undef CALLBACK_ENTRY
+};
+
+
+
+/* --------------------------------------------------------------------------------
+ * Public callback related functions operating on IO Handles
+ * --------------------------------------------------------------------------------
+ */
+
+/*
+ * Register callback for the IO handle.
+ *
+ * Only a limited number (PGAIO_HANDLE_MAX_CALLBACKS) of callbacks can be
+ * registered for each IO.
+ *
+ * Callbacks need to be registered before [indirectly] calling
+ * pgaio_io_prep_*(), as the IO may be executed immediately.
+ *
+ * A callback can be passed a small bit of data, e.g. to indicate whether to
+ * zero a buffer if it is invalid.
+ *
+ *
+ * Note that callbacks are executed in critical sections. This is necessary
+ * to be able to execute IO in critical sections (consider e.g. WAL
+ * logging). To perform AIO we first need to acquire a handle, which, if there
+ * are no free handles, requires waiting for IOs to complete and to execute
+ * their completion callbacks.
+ *
+ * Callbacks may be executed in the issuing backend but also in another
+ * backend (because that backend is waiting for the IO) or in IO workers (if
+ * io_method=worker is used).
+ *
+ *
+ * See PgAioHandleCallbackID's definition for an explanation for why
+ * callbacks are not identified by a pointer.
+ */
+void
+pgaio_io_register_callbacks(PgAioHandle *ioh, PgAioHandleCallbackID cb_id,
+ uint8 cb_data)
+{
+ const PgAioHandleCallbacksEntry *ce = &aio_handle_cbs[cb_id];
+
+ if (cb_id >= lengthof(aio_handle_cbs))
+ elog(ERROR, "callback %d is out of range", cb_id);
+ if (aio_handle_cbs[cb_id].cb->complete_shared == NULL &&
+ aio_handle_cbs[cb_id].cb->complete_local == NULL)
+ elog(ERROR, "callback %d does not have completion callback", cb_id);
+ if (ioh->num_callbacks >= PGAIO_HANDLE_MAX_CALLBACKS)
+ elog(PANIC, "too many callbacks, the max is %d", PGAIO_HANDLE_MAX_CALLBACKS);
+ ioh->callbacks[ioh->num_callbacks] = cb_id;
+ ioh->callbacks_data[ioh->num_callbacks] = cb_data;
+
+ pgaio_debug_io(DEBUG3, ioh,
+ "adding cb #%d, id %d/%s",
+ ioh->num_callbacks + 1,
+ cb_id, ce->name);
+
+ ioh->num_callbacks++;
+}
+
+/*
+ * Associate an array of data with the Handle. This is e.g. useful to the
+ * transport knowledge about which buffers a multi-block IO affects to
+ * completion callbacks.
+ *
+ * Right now this can be done only once for each IO, even though multiple
+ * callbacks can be registered. There aren't any known usecases requiring more
+ * and the required amount of shared memory does add up, so it doesn't seem
+ * worth multiplying memory usage by PGAIO_HANDLE_MAX_CALLBACKS.
+ */
+void
+pgaio_io_set_handle_data_64(PgAioHandle *ioh, uint64 *data, uint8 len)
+{
+ Assert(ioh->state == PGAIO_HS_HANDED_OUT);
+ Assert(ioh->handle_data_len == 0);
+ Assert(len <= PG_IOV_MAX);
+
+ for (int i = 0; i < len; i++)
+ pgaio_ctl->handle_data[ioh->iovec_off + i] = data[i];
+ ioh->handle_data_len = len;
+}
+
+/*
+ * Convenience version of pgaio_io_set_handle_data_64() that converts a 32bit
+ * array to a 64bit array. Without it callers would end up needing to
+ * open-code equivalent code.
+ */
+void
+pgaio_io_set_handle_data_32(PgAioHandle *ioh, uint32 *data, uint8 len)
+{
+ Assert(ioh->state == PGAIO_HS_HANDED_OUT);
+ Assert(ioh->handle_data_len == 0);
+ Assert(len <= PG_IOV_MAX);
+
+ for (int i = 0; i < len; i++)
+ pgaio_ctl->handle_data[ioh->iovec_off + i] = data[i];
+ ioh->handle_data_len = len;
+}
+
+/*
+ * Return data set with pgaio_io_set_handle_data_*().
+ */
+uint64 *
+pgaio_io_get_handle_data(PgAioHandle *ioh, uint8 *len)
+{
+ Assert(ioh->handle_data_len > 0);
+
+ *len = ioh->handle_data_len;
+
+ return &pgaio_ctl->handle_data[ioh->iovec_off];
+}
+
+
+
+/* --------------------------------------------------------------------------------
+ * Public IO Result related functions
+ * --------------------------------------------------------------------------------
+ */
+
+void
+pgaio_result_report(PgAioResult result, const PgAioTargetData *target_data, int elevel)
+{
+ PgAioHandleCallbackID cb_id = result.id;
+ const PgAioHandleCallbacksEntry *ce = &aio_handle_cbs[cb_id];
+
+ Assert(result.status != ARS_UNKNOWN);
+ Assert(result.status != ARS_OK);
+
+ if (ce->cb->report == NULL)
+ elog(ERROR, "callback %d/%s does not have report callback",
+ result.id, ce->name);
+
+ ce->cb->report(result, target_data, elevel);
+}
+
+
+
+/* --------------------------------------------------------------------------------
+ * Internal callback related functions operating on IO Handles
+ * --------------------------------------------------------------------------------
+ */
+
+/*
+ * Internal function which invokes ->stage for all the registered callbacks.
+ */
+void
+pgaio_io_call_stage(PgAioHandle *ioh)
+{
+ Assert(ioh->target > PGAIO_TID_INVALID && ioh->target < PGAIO_TID_COUNT);
+ Assert(ioh->op > PGAIO_OP_INVALID && ioh->op < PGAIO_OP_COUNT);
+
+ for (int i = ioh->num_callbacks; i > 0; i--)
+ {
+ PgAioHandleCallbackID cb_id = ioh->callbacks[i - 1];
+ uint8 cb_data = ioh->callbacks_data[i - 1];
+ const PgAioHandleCallbacksEntry *ce = &aio_handle_cbs[cb_id];
+
+ if (!ce->cb->stage)
+ continue;
+
+ pgaio_debug_io(DEBUG3, ioh,
+ "calling cb #%d %d/%s->stage(%u)",
+ i, cb_id, ce->name, cb_data);
+ ce->cb->stage(ioh, cb_data);
+ }
+}
+
+/*
+ * Internal function which invokes ->complete_shared for all the registered
+ * callbacks.
+ */
+void
+pgaio_io_call_complete_shared(PgAioHandle *ioh)
+{
+ PgAioResult result;
+
+ START_CRIT_SECTION();
+
+ Assert(ioh->target > PGAIO_TID_INVALID && ioh->target < PGAIO_TID_COUNT);
+ Assert(ioh->op > PGAIO_OP_INVALID && ioh->op < PGAIO_OP_COUNT);
+
+ result.status = ARS_OK; /* low level IO is always considered OK */
+ result.result = ioh->result;
+ result.id = PGAIO_HCB_INVALID;
+ result.error_data = 0;
+
+ /*
+ * Call callbacks with the last registered (innermost) callback first.
+ * Each callback can modify the result forwarded to the next callback.
+ */
+ for (int i = ioh->num_callbacks; i > 0; i--)
+ {
+ PgAioHandleCallbackID cb_id = ioh->callbacks[i - 1];
+ uint8 cb_data = ioh->callbacks_data[i - 1];
+ const PgAioHandleCallbacksEntry *ce = &aio_handle_cbs[cb_id];
+
+ if (!ce->cb->complete_shared)
+ continue;
+
+ pgaio_debug_io(DEBUG4, ioh,
+ "calling cb #%d, id %d/%s->complete_shared(%u) with distilled result: (status %s, id %u, error_data %d, result %d)",
+ i, cb_id, ce->name,
+ cb_data,
+ pgaio_result_status_string(result.status),
+ result.id, result.error_data, result.result);
+ result = ce->cb->complete_shared(ioh, result, cb_data);
+ }
+
+ ioh->distilled_result = result;
+
+ pgaio_debug_io(DEBUG3, ioh,
+ "after shared completion: distilled result: (status %s, id %u, error_data: %d, result %d), raw_result: %d",
+ pgaio_result_status_string(result.status),
+ result.id, result.error_data, result.result,
+ ioh->result);
+
+ END_CRIT_SECTION();
+}
+
+/*
+ * Internal function which invokes ->complete_local for all the registered
+ * callbacks.
+ *
+ * XXX: It'd be nice to deduplicate with pgaio_io_call_complete_shared().
+ */
+void
+pgaio_io_call_complete_local(PgAioHandle *ioh)
+{
+ PgAioResult result;
+
+ START_CRIT_SECTION();
+
+ Assert(ioh->target > PGAIO_TID_INVALID && ioh->target < PGAIO_TID_COUNT);
+ Assert(ioh->op > PGAIO_OP_INVALID && ioh->op < PGAIO_OP_COUNT);
+
+ /* start with distilled result from shared callback */
+ result = ioh->distilled_result;
+
+ for (int i = ioh->num_callbacks; i > 0; i--)
+ {
+ PgAioHandleCallbackID cb_id = ioh->callbacks[i - 1];
+ uint8 cb_data = ioh->callbacks_data[i - 1];
+ const PgAioHandleCallbacksEntry *ce = &aio_handle_cbs[cb_id];
+
+ if (!ce->cb->complete_local)
+ continue;
+
+ pgaio_debug_io(DEBUG4, ioh,
+ "calling cb #%d, id %d/%s->complete_local(%u) with distilled result: status %s, id %u, error_data %d, result %d",
+ i, cb_id, ce->name, cb_data,
+ pgaio_result_status_string(result.status),
+ result.id, result.error_data, result.result);
+ result = ce->cb->complete_local(ioh, result, cb_data);
+ }
+
+ /*
+ * Note that we don't save the result in ioh->distilled_result, the local
+ * callback's result should not ever matter to other waiters.
+ */
+ pgaio_debug_io(DEBUG3, ioh,
+ "after local completion: distilled result: (status %s, id %u, error_data %d, result %d), raw_result: %d",
+ pgaio_result_status_string(result.status),
+ result.id, result.error_data, result.result,
+ ioh->result);
+
+ END_CRIT_SECTION();
+}
diff --git a/src/backend/storage/aio/aio_init.c b/src/backend/storage/aio/aio_init.c
index aeacc144149..8e1162b09dc 100644
--- a/src/backend/storage/aio/aio_init.c
+++ b/src/backend/storage/aio/aio_init.c
@@ -14,24 +14,210 @@
#include "postgres.h"
+#include "miscadmin.h"
+#include "storage/aio.h"
+#include "storage/aio_internal.h"
#include "storage/aio_subsys.h"
+#include "storage/ipc.h"
+#include "storage/proc.h"
+#include "storage/shmem.h"
+#include "utils/guc.h"
+static Size
+AioCtlShmemSize(void)
+{
+ Size sz;
+
+ /* pgaio_ctl itself */
+ sz = offsetof(PgAioCtl, io_handles);
+
+ return sz;
+}
+
+static uint32
+AioProcs(void)
+{
+ return MaxBackends + NUM_AUXILIARY_PROCS;
+}
+
+static Size
+AioBackendShmemSize(void)
+{
+ return mul_size(AioProcs(), sizeof(PgAioBackend));
+}
+
+static Size
+AioHandleShmemSize(void)
+{
+ Size sz;
+
+ /* ios */
+ sz = mul_size(AioProcs(),
+ mul_size(io_max_concurrency, sizeof(PgAioHandle)));
+
+ return sz;
+}
+
+static Size
+AioHandleIOVShmemSize(void)
+{
+ return mul_size(sizeof(struct iovec),
+ mul_size(mul_size(PG_IOV_MAX, AioProcs()),
+ io_max_concurrency));
+}
+
+static Size
+AioHandleDataShmemSize(void)
+{
+ return mul_size(sizeof(uint64),
+ mul_size(mul_size(PG_IOV_MAX, AioProcs()),
+ io_max_concurrency));
+}
+
+/*
+ * Choose a suitable value for io_max_concurrency.
+ *
+ * It's unlikely that we could have more IOs in flight than buffers that we
+ * would be allowed to pin.
+ *
+ * On the upper end, apply a cap too - just because shared_buffers is large,
+ * it doesn't make sense have millions of buffers undergo IO concurrently.
+ */
+static int
+AioChooseMaxConcurrency(void)
+{
+ uint32 max_backends;
+ int max_proportional_pins;
+
+ /* Similar logic to LimitAdditionalPins() */
+ max_backends = MaxBackends + NUM_AUXILIARY_PROCS;
+ max_proportional_pins = NBuffers / max_backends;
+
+ max_proportional_pins = Max(max_proportional_pins, 1);
+
+ /* apply upper limit */
+ return Min(max_proportional_pins, 64);
+}
+
Size
AioShmemSize(void)
{
Size sz = 0;
+ /*
+ * We prefer to report this value's source as PGC_S_DYNAMIC_DEFAULT.
+ * However, if the DBA explicitly set wal_buffers = -1 in the config file,
+ * then PGC_S_DYNAMIC_DEFAULT will fail to override that and we must force
+ *
+ */
+ if (io_max_concurrency == -1)
+ {
+ char buf[32];
+
+ snprintf(buf, sizeof(buf), "%d", AioChooseMaxConcurrency());
+ SetConfigOption("io_max_concurrency", buf, PGC_POSTMASTER,
+ PGC_S_DYNAMIC_DEFAULT);
+ if (io_max_concurrency == -1) /* failed to apply it? */
+ SetConfigOption("io_max_concurrency", buf, PGC_POSTMASTER,
+ PGC_S_OVERRIDE);
+ }
+
+ sz = add_size(sz, AioCtlShmemSize());
+ sz = add_size(sz, AioBackendShmemSize());
+ sz = add_size(sz, AioHandleShmemSize());
+ sz = add_size(sz, AioHandleIOVShmemSize());
+ sz = add_size(sz, AioHandleDataShmemSize());
+
+ if (pgaio_method_ops->shmem_size)
+ sz = add_size(sz, pgaio_method_ops->shmem_size());
+
return sz;
}
void
AioShmemInit(void)
{
+ bool found;
+ uint32 io_handle_off = 0;
+ uint32 iovec_off = 0;
+ uint32 per_backend_iovecs = io_max_concurrency * PG_IOV_MAX;
+
+ pgaio_ctl = (PgAioCtl *)
+ ShmemInitStruct("AioCtl", AioCtlShmemSize(), &found);
+
+ if (found)
+ goto out;
+
+ memset(pgaio_ctl, 0, AioCtlShmemSize());
+
+ pgaio_ctl->io_handle_count = AioProcs() * io_max_concurrency;
+ pgaio_ctl->iovec_count = AioProcs() * per_backend_iovecs;
+
+ pgaio_ctl->backend_state = (PgAioBackend *)
+ ShmemInitStruct("AioBackend", AioBackendShmemSize(), &found);
+
+ pgaio_ctl->io_handles = (PgAioHandle *)
+ ShmemInitStruct("AioHandle", AioHandleShmemSize(), &found);
+
+ pgaio_ctl->iovecs = (struct iovec *)
+ ShmemInitStruct("AioHandleIOV", AioHandleIOVShmemSize(), &found);
+ pgaio_ctl->handle_data = (uint64 *)
+ ShmemInitStruct("AioHandleData", AioHandleDataShmemSize(), &found);
+
+ for (int procno = 0; procno < AioProcs(); procno++)
+ {
+ PgAioBackend *bs = &pgaio_ctl->backend_state[procno];
+
+ bs->io_handle_off = io_handle_off;
+ io_handle_off += io_max_concurrency;
+
+ dclist_init(&bs->idle_ios);
+ memset(bs->staged_ios, 0, sizeof(PgAioHandle *) * PGAIO_SUBMIT_BATCH_SIZE);
+ dclist_init(&bs->in_flight_ios);
+
+ /* initialize per-backend IOs */
+ for (int i = 0; i < io_max_concurrency; i++)
+ {
+ PgAioHandle *ioh = &pgaio_ctl->io_handles[bs->io_handle_off + i];
+
+ ioh->generation = 1;
+ ioh->owner_procno = procno;
+ ioh->iovec_off = iovec_off;
+ ioh->handle_data_len = 0;
+ ioh->report_return = NULL;
+ ioh->resowner = NULL;
+ ioh->num_callbacks = 0;
+ ioh->distilled_result.status = ARS_UNKNOWN;
+ ioh->flags = 0;
+
+ ConditionVariableInit(&ioh->cv);
+
+ dclist_push_tail(&bs->idle_ios, &ioh->node);
+ iovec_off += PG_IOV_MAX;
+ }
+ }
+
+out:
+ /* Initialize IO method specific resources. */
+ if (pgaio_method_ops->shmem_init)
+ pgaio_method_ops->shmem_init(!found);
}
void
pgaio_init_backend(void)
{
+ /* shouldn't be initialized twice */
+ Assert(!pgaio_my_backend);
+
+ if (MyProc == NULL || MyProcNumber >= AioProcs())
+ elog(ERROR, "aio requires a normal PGPROC");
+
+ pgaio_my_backend = &pgaio_ctl->backend_state[MyProcNumber];
+
+ if (pgaio_method_ops->init_backend)
+ pgaio_method_ops->init_backend();
+
+ before_shmem_exit(pgaio_shutdown, 0);
}
diff --git a/src/backend/storage/aio/aio_io.c b/src/backend/storage/aio/aio_io.c
new file mode 100644
index 00000000000..89376ff4040
--- /dev/null
+++ b/src/backend/storage/aio/aio_io.c
@@ -0,0 +1,180 @@
+/*-------------------------------------------------------------------------
+ *
+ * aio_io.c
+ * AIO - Low Level IO Handling
+ *
+ * Functions related to associating IO operations to IO Handles and IO-method
+ * independent support functions for actually performing IO.
+ *
+ *
+ * Portions Copyright (c) 1996-2025, PostgreSQL Global Development Group
+ * Portions Copyright (c) 1994, Regents of the University of California
+ *
+ * IDENTIFICATION
+ * src/backend/storage/aio/aio_io.c
+ *
+ *-------------------------------------------------------------------------
+ */
+
+#include "postgres.h"
+
+#include "miscadmin.h"
+#include "storage/aio.h"
+#include "storage/aio_internal.h"
+#include "storage/fd.h"
+#include "utils/wait_event.h"
+
+
+static void pgaio_io_before_prep(PgAioHandle *ioh);
+
+
+
+/* --------------------------------------------------------------------------------
+ * Public IO related functions operating on IO Handles
+ * --------------------------------------------------------------------------------
+ */
+
+/*
+ * Scatter/gather IO needs to associate an iovec with the Handle. To support
+ * worker mode this data needs to be in shared memory.
+ *
+ * XXX: Right now the amount of space available for each IO is
+ * PG_IOV_MAX. While it's tempting to use the io_combine_limit GUC, that's
+ * PGC_USERSET, so we can't allocate shared memory based on that.
+ */
+int
+pgaio_io_get_iovec(PgAioHandle *ioh, struct iovec **iov)
+{
+ Assert(ioh->state == PGAIO_HS_HANDED_OUT);
+
+ *iov = &pgaio_ctl->iovecs[ioh->iovec_off];
+
+ return PG_IOV_MAX;
+}
+
+PgAioOpData *
+pgaio_io_get_op_data(PgAioHandle *ioh)
+{
+ return &ioh->op_data;
+}
+
+
+
+/* --------------------------------------------------------------------------------
+ * "Preparation" routines for individual IO operations
+ *
+ * These are called by the code actually initiating an IO, to associate the IO
+ * specific data with an AIO handle.
+ *
+ * Each of the preparation routines first needs to call
+ * pgaio_io_before_prep(), then fill IO specific fields in the handle and then
+ * finally call pgaio_io_stage().
+ * --------------------------------------------------------------------------------
+ */
+
+void
+pgaio_io_prep_readv(PgAioHandle *ioh,
+ int fd, int iovcnt, uint64 offset)
+{
+ pgaio_io_before_prep(ioh);
+
+ ioh->op_data.read.fd = fd;
+ ioh->op_data.read.offset = offset;
+ ioh->op_data.read.iov_length = iovcnt;
+
+ pgaio_io_stage(ioh, PGAIO_OP_READV);
+}
+
+void
+pgaio_io_prep_writev(PgAioHandle *ioh,
+ int fd, int iovcnt, uint64 offset)
+{
+ pgaio_io_before_prep(ioh);
+
+ ioh->op_data.write.fd = fd;
+ ioh->op_data.write.offset = offset;
+ ioh->op_data.write.iov_length = iovcnt;
+
+ pgaio_io_stage(ioh, PGAIO_OP_WRITEV);
+}
+
+
+
+/* --------------------------------------------------------------------------------
+ * Internal IO related functions operating on IO Handles
+ * --------------------------------------------------------------------------------
+ */
+
+/*
+ * Execute IO operation synchronously. This is implemented here, not in
+ * method_sync.c, because other IO methods lso might use it / fall back to it.
+ */
+void
+pgaio_io_perform_synchronously(PgAioHandle *ioh)
+{
+ ssize_t result = 0;
+ struct iovec *iov = &pgaio_ctl->iovecs[ioh->iovec_off];
+
+ START_CRIT_SECTION();
+
+ /* Perform IO. */
+ switch (ioh->op)
+ {
+ case PGAIO_OP_READV:
+ pgstat_report_wait_start(WAIT_EVENT_DATA_FILE_READ);
+ result = pg_preadv(ioh->op_data.read.fd, iov,
+ ioh->op_data.read.iov_length,
+ ioh->op_data.read.offset);
+ pgstat_report_wait_end();
+ break;
+ case PGAIO_OP_WRITEV:
+ pgstat_report_wait_start(WAIT_EVENT_DATA_FILE_WRITE);
+ result = pg_pwritev(ioh->op_data.write.fd, iov,
+ ioh->op_data.write.iov_length,
+ ioh->op_data.write.offset);
+ pgstat_report_wait_end();
+ break;
+ case PGAIO_OP_INVALID:
+ elog(ERROR, "trying to execute invalid IO operation");
+ }
+
+ ioh->result = result < 0 ? -errno : result;
+
+ pgaio_io_process_completion(ioh, ioh->result);
+
+ END_CRIT_SECTION();
+}
+
+/*
+ * Helper function to be called by IO operation preparation functions, before
+ * any data in the handle is set. Mostly to centralize assertions.
+ */
+static void
+pgaio_io_before_prep(PgAioHandle *ioh)
+{
+ Assert(ioh->state == PGAIO_HS_HANDED_OUT);
+ Assert(pgaio_io_has_target(ioh));
+ Assert(ioh->op == PGAIO_OP_INVALID);
+}
+
+/*
+ * Could be made part of the public interface, but it's not clear there's
+ * really a use case for that.
+ */
+const char *
+pgaio_io_get_op_name(PgAioHandle *ioh)
+{
+ Assert(ioh->op >= 0 && ioh->op < PGAIO_OP_COUNT);
+
+ switch (ioh->op)
+ {
+ case PGAIO_OP_INVALID:
+ return "invalid";
+ case PGAIO_OP_READV:
+ return "read";
+ case PGAIO_OP_WRITEV:
+ return "write";
+ }
+
+ return NULL; /* silence compiler */
+}
diff --git a/src/backend/storage/aio/aio_target.c b/src/backend/storage/aio/aio_target.c
new file mode 100644
index 00000000000..3fa813fd592
--- /dev/null
+++ b/src/backend/storage/aio/aio_target.c
@@ -0,0 +1,118 @@
+/*-------------------------------------------------------------------------
+ *
+ * aio_target.c
+ * AIO - Functionality related to executing IO for different targets
+ *
+ * Portions Copyright (c) 1996-2025, PostgreSQL Global Development Group
+ * Portions Copyright (c) 1994, Regents of the University of California
+ *
+ * IDENTIFICATION
+ * src/backend/storage/aio/aio_target.c
+ *
+ *-------------------------------------------------------------------------
+ */
+
+#include "postgres.h"
+
+#include "storage/aio.h"
+#include "storage/aio_internal.h"
+
+
+/*
+ * Registry for entities that can be the target of AIO.
+ *
+ * To support executing using worker processes, the file descriptor for an IO
+ * may need to be be reopened in a different process. This is done via the
+ * PgAioTargetInfo.reopen callback.
+ */
+static const PgAioTargetInfo *pgaio_target_info[] = {
+ [PGAIO_TID_INVALID] = &(PgAioTargetInfo) {
+ .name = "invalid",
+ },
+};
+
+
+
+/* --------------------------------------------------------------------------------
+ * Public target related functions operating on IO Handles
+ * --------------------------------------------------------------------------------
+ */
+
+bool
+pgaio_io_has_target(PgAioHandle *ioh)
+{
+ return ioh->target != PGAIO_TID_INVALID;
+}
+
+/*
+ * Return the name for the target associated with the IO. Mostly useful for
+ * debugging/logging.
+ */
+const char *
+pgaio_io_get_target_name(PgAioHandle *ioh)
+{
+ Assert(ioh->target >= 0 && ioh->target < PGAIO_TID_COUNT);
+
+ return pgaio_target_info[ioh->target]->name;
+}
+
+/*
+ * Assign a target to the IO.
+ *
+ * This has to be called exactly once before pgaio_io_prep_*() is called.
+ */
+void
+pgaio_io_set_target(PgAioHandle *ioh, PgAioTargetID targetid)
+{
+ Assert(ioh->state == PGAIO_HS_HANDED_OUT);
+ Assert(ioh->target == PGAIO_TID_INVALID);
+
+ ioh->target = targetid;
+}
+
+PgAioTargetData *
+pgaio_io_get_target_data(PgAioHandle *ioh)
+{
+ return &ioh->target_data;
+}
+
+/*
+ * Return a stringified description of the IO's target.
+ *
+ * The string is localized and allocated in the current memory context.
+ */
+char *
+pgaio_io_get_target_description(PgAioHandle *ioh)
+{
+ return pgaio_target_info[ioh->target]->describe_identity(&ioh->target_data);
+}
+
+
+
+/* --------------------------------------------------------------------------------
+ * Internal target related functions operating on IO Handles
+ * --------------------------------------------------------------------------------
+ */
+
+/*
+ * Internal: Check if pgaio_io_reopen() is available for the IO.
+ */
+bool
+pgaio_io_can_reopen(PgAioHandle *ioh)
+{
+ return pgaio_target_info[ioh->target]->reopen != NULL;
+}
+
+/*
+ * Internal: Before executing an IO outside of the context of the process the
+ * IO has been prepared in, the file descriptor has to be reopened - any FD
+ * referenced in the IO itself, won't be valid in the separate process.
+ */
+void
+pgaio_io_reopen(PgAioHandle *ioh)
+{
+ Assert(ioh->target >= 0 && ioh->target < PGAIO_TID_COUNT);
+ Assert(ioh->op >= 0 && ioh->op < PGAIO_OP_COUNT);
+
+ pgaio_target_info[ioh->target]->reopen(ioh);
+}
diff --git a/src/backend/storage/aio/meson.build b/src/backend/storage/aio/meson.build
index c822fd4ddf7..2c26089d52e 100644
--- a/src/backend/storage/aio/meson.build
+++ b/src/backend/storage/aio/meson.build
@@ -2,6 +2,10 @@
backend_sources += files(
'aio.c',
+ 'aio_callback.c',
'aio_init.c',
+ 'aio_io.c',
+ 'aio_target.c',
+ 'method_sync.c',
'read_stream.c',
)
diff --git a/src/backend/storage/aio/method_sync.c b/src/backend/storage/aio/method_sync.c
new file mode 100644
index 00000000000..902c2428d41
--- /dev/null
+++ b/src/backend/storage/aio/method_sync.c
@@ -0,0 +1,47 @@
+/*-------------------------------------------------------------------------
+ *
+ * method_sync.c
+ * AIO - perform "AIO" by executing it synchronously
+ *
+ * This method is mainly to check if AIO use causes regressions. Other IO
+ * methods might also fall back to the synchronous method for functionality
+ * they cannot provide.
+ *
+ * Portions Copyright (c) 1996-2025, PostgreSQL Global Development Group
+ * Portions Copyright (c) 1994, Regents of the University of California
+ *
+ * IDENTIFICATION
+ * src/backend/storage/aio/method_sync.c
+ *
+ *-------------------------------------------------------------------------
+ */
+
+#include "postgres.h"
+
+#include "storage/aio.h"
+#include "storage/aio_internal.h"
+
+static bool pgaio_sync_needs_synchronous_execution(PgAioHandle *ioh);
+static int pgaio_sync_submit(uint16 num_staged_ios, PgAioHandle **staged_ios);
+
+
+const IoMethodOps pgaio_sync_ops = {
+ .needs_synchronous_execution = pgaio_sync_needs_synchronous_execution,
+ .submit = pgaio_sync_submit,
+};
+
+
+
+static bool
+pgaio_sync_needs_synchronous_execution(PgAioHandle *ioh)
+{
+ return true;
+}
+
+static int
+pgaio_sync_submit(uint16 num_staged_ios, PgAioHandle **staged_ios)
+{
+ elog(ERROR, "IO should have been executed synchronously");
+
+ return 0;
+}
diff --git a/src/backend/utils/activity/wait_event_names.txt b/src/backend/utils/activity/wait_event_names.txt
index 3c594415bfd..b44e4908b25 100644
--- a/src/backend/utils/activity/wait_event_names.txt
+++ b/src/backend/utils/activity/wait_event_names.txt
@@ -191,6 +191,7 @@ ABI_compatibility:
Section: ClassName - WaitEventIO
+AIO_IO_COMPLETION "Waiting for IO completion."
BASEBACKUP_READ "Waiting for base backup to read from a file."
BASEBACKUP_SYNC "Waiting for data written by a base backup to reach durable storage."
BASEBACKUP_WRITE "Waiting for base backup to write to a file."
diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list
index 7e6bbc1a8c1..ce7f877adca 100644
--- a/src/tools/pgindent/typedefs.list
+++ b/src/tools/pgindent/typedefs.list
@@ -1278,6 +1278,7 @@ InvalMessageArray
InvalidationInfo
InvalidationMsgsGroup
IoMethod
+IoMethodOps
IpcMemoryId
IpcMemoryKey
IpcMemoryState
@@ -2124,6 +2125,26 @@ Permutation
PermutationStep
PermutationStepBlocker
PermutationStepBlockerType
+PgAioBackend
+PgAioCtl
+PgAioHandle
+PgAioHandleCallbackID
+PgAioHandleCallbackStage
+PgAioHandleCallbackComplete
+PgAioHandleCallbackReport
+PgAioHandleCallbacks
+PgAioHandleCallbacksEntry
+PgAioHandleFlags
+PgAioHandleState
+PgAioOp
+PgAioOpData
+PgAioResult
+PgAioResultStatus
+PgAioReturn
+PgAioTargetData
+PgAioTargetID
+PgAioTargetInfo
+PgAioWaitRef
PgArchData
PgBackendGSSStatus
PgBackendSSLStatus
--
2.48.1.76.g4e746b1a31.dirty
[text/x-diff] v2.6-0003-aio-Infrastructure-for-io_method-worker.patch (24.7K, ../../ge6nsuddurhpmll3xj22vucvqwp4agqz6ndtcf2mhyeydzarst@l75dman5x53p/4-v2.6-0003-aio-Infrastructure-for-io_method-worker.patch)
download | inline diff:
From 15a1e1ef5ca66da64d4ce97ba9bc7c6a196dc725 Mon Sep 17 00:00:00 2001
From: Andres Freund <andres@anarazel.de>
Date: Mon, 17 Feb 2025 17:57:00 -0500
Subject: [PATCH v2.6 03/34] aio: Infrastructure for io_method=worker
This commit contains the basic, system-wide, infrastructure for
io_method=worker. It does not yet actually execute IO, this commit just
provides the infrastructure for running IO workers, kept separate for easier
review.
The number of IO workers can be adjusted with a PGC_SIGHUP GUC. Eventually
we'd like to make the number of workers dynamically scale up/down based on the
current "IO load".
To allow the number of IO workers to be increased without a restart, we need
to reserve PGPROC entries for the workers unconditionally. This has been
judged to be worth the cost. If it turns out to be problematic, we can
introduce a PGC_POSTMASTER GUC to control the maximum number.
Co-authored-by: Thomas Munro <thomas.munro@gmail.com>
Co-authored-by: Andres Freund <andres@anarazel.de>
Discussion: https://postgr.es/m/uvrtrknj4kdytuboidbhwclo4gxhswwcpgadptsjvjqcluzmah%40brqs62irg4dt
Discussion: https://postgr.es/m/20210223100344.llw5an2aklengrmn@alap3.anarazel.de
Discussion: https://postgr.es/m/stj36ea6yyhoxtqkhpieia2z4krnam7qyetc57rfezgk4zgapf@gcnactj4z56m
---
src/include/miscadmin.h | 2 +
src/include/postmaster/postmaster.h | 1 +
src/include/storage/aio_subsys.h | 4 +
src/include/storage/io_worker.h | 22 +++
src/include/storage/proc.h | 4 +-
src/backend/postmaster/launch_backend.c | 2 +
src/backend/postmaster/pmchild.c | 1 +
src/backend/postmaster/postmaster.c | 165 ++++++++++++++++--
src/backend/storage/aio/Makefile | 1 +
src/backend/storage/aio/meson.build | 1 +
src/backend/storage/aio/method_worker.c | 88 ++++++++++
src/backend/tcop/postgres.c | 2 +
src/backend/utils/activity/pgstat_backend.c | 1 +
src/backend/utils/activity/pgstat_io.c | 1 +
.../utils/activity/wait_event_names.txt | 1 +
src/backend/utils/init/miscinit.c | 3 +
src/backend/utils/misc/guc_tables.c | 13 ++
src/backend/utils/misc/postgresql.conf.sample | 1 +
doc/src/sgml/config.sgml | 19 ++
src/test/regress/expected/stats.out | 10 +-
20 files changed, 328 insertions(+), 14 deletions(-)
create mode 100644 src/include/storage/io_worker.h
create mode 100644 src/backend/storage/aio/method_worker.c
diff --git a/src/include/miscadmin.h b/src/include/miscadmin.h
index a2b63495eec..54429e046a9 100644
--- a/src/include/miscadmin.h
+++ b/src/include/miscadmin.h
@@ -360,6 +360,7 @@ typedef enum BackendType
B_ARCHIVER,
B_BG_WRITER,
B_CHECKPOINTER,
+ B_IO_WORKER,
B_STARTUP,
B_WAL_RECEIVER,
B_WAL_SUMMARIZER,
@@ -389,6 +390,7 @@ extern PGDLLIMPORT BackendType MyBackendType;
#define AmWalReceiverProcess() (MyBackendType == B_WAL_RECEIVER)
#define AmWalSummarizerProcess() (MyBackendType == B_WAL_SUMMARIZER)
#define AmWalWriterProcess() (MyBackendType == B_WAL_WRITER)
+#define AmIoWorkerProcess() (MyBackendType == B_IO_WORKER)
#define AmSpecialWorkerProcess() \
(AmAutoVacuumLauncherProcess() || \
diff --git a/src/include/postmaster/postmaster.h b/src/include/postmaster/postmaster.h
index b6a3f275a1b..afefa35e2cc 100644
--- a/src/include/postmaster/postmaster.h
+++ b/src/include/postmaster/postmaster.h
@@ -98,6 +98,7 @@ extern void InitProcessGlobals(void);
extern int MaxLivePostmasterChildren(void);
extern bool PostmasterMarkPIDForWorkerNotify(int);
+extern void assign_io_workers(int newval, void *extra);
#ifdef WIN32
extern void pgwin32_register_deadchild_callback(HANDLE procHandle, DWORD procId);
diff --git a/src/include/storage/aio_subsys.h b/src/include/storage/aio_subsys.h
index e4faf692a38..ed00d5c47cd 100644
--- a/src/include/storage/aio_subsys.h
+++ b/src/include/storage/aio_subsys.h
@@ -30,4 +30,8 @@ extern void pgaio_init_backend(void);
extern void pgaio_error_cleanup(void);
extern void AtEOXact_Aio(bool is_commit);
+
+/* aio_worker.c */
+extern bool pgaio_workers_enabled(void);
+
#endif /* AIO_SUBSYS_H */
diff --git a/src/include/storage/io_worker.h b/src/include/storage/io_worker.h
new file mode 100644
index 00000000000..872241ae299
--- /dev/null
+++ b/src/include/storage/io_worker.h
@@ -0,0 +1,22 @@
+/*-------------------------------------------------------------------------
+ *
+ * io_worker.h
+ * IO worker for implementing AIO "ourselves"
+ *
+ *
+ * Portions Copyright (c) 1996-2025, PostgreSQL Global Development Group
+ * Portions Copyright (c) 1994, Regents of the University of California
+ *
+ * src/include/storage/io.h
+ *
+ *-------------------------------------------------------------------------
+ */
+#ifndef IO_WORKER_H
+#define IO_WORKER_H
+
+
+extern void IoWorkerMain(const void *startup_data, size_t startup_data_len) pg_attribute_noreturn();
+
+extern int io_workers;
+
+#endif /* IO_WORKER_H */
diff --git a/src/include/storage/proc.h b/src/include/storage/proc.h
index 114eb1f8f76..4e815f9fde0 100644
--- a/src/include/storage/proc.h
+++ b/src/include/storage/proc.h
@@ -449,7 +449,9 @@ extern PGDLLIMPORT PGPROC *PreparedXactProcs;
* 2 slots, but WAL writer is launched only after startup has exited, so we
* only need 6 slots.
*/
-#define NUM_AUXILIARY_PROCS 6
+#define MAX_IO_WORKERS 32
+#define NUM_AUXILIARY_PROCS (6 + MAX_IO_WORKERS)
+
/* configurable options */
extern PGDLLIMPORT int DeadlockTimeout;
diff --git a/src/backend/postmaster/launch_backend.c b/src/backend/postmaster/launch_backend.c
index 47375e5bfaa..516f2b5d212 100644
--- a/src/backend/postmaster/launch_backend.c
+++ b/src/backend/postmaster/launch_backend.c
@@ -48,6 +48,7 @@
#include "replication/slotsync.h"
#include "replication/walreceiver.h"
#include "storage/dsm.h"
+#include "storage/io_worker.h"
#include "storage/pg_shmem.h"
#include "tcop/backend_startup.h"
#include "utils/memutils.h"
@@ -197,6 +198,7 @@ static child_process_kind child_process_kinds[] = {
[B_ARCHIVER] = {"archiver", PgArchiverMain, true},
[B_BG_WRITER] = {"bgwriter", BackgroundWriterMain, true},
[B_CHECKPOINTER] = {"checkpointer", CheckpointerMain, true},
+ [B_IO_WORKER] = {"io_worker", IoWorkerMain, true},
[B_STARTUP] = {"startup", StartupProcessMain, true},
[B_WAL_RECEIVER] = {"wal_receiver", WalReceiverMain, true},
[B_WAL_SUMMARIZER] = {"wal_summarizer", WalSummarizerMain, true},
diff --git a/src/backend/postmaster/pmchild.c b/src/backend/postmaster/pmchild.c
index 0d473226c3a..cde1d23a4ca 100644
--- a/src/backend/postmaster/pmchild.c
+++ b/src/backend/postmaster/pmchild.c
@@ -101,6 +101,7 @@ InitPostmasterChildSlots(void)
pmchild_pools[B_AUTOVAC_WORKER].size = autovacuum_worker_slots;
pmchild_pools[B_BG_WORKER].size = max_worker_processes;
+ pmchild_pools[B_IO_WORKER].size = MAX_IO_WORKERS;
/*
* There can be only one of each of these running at a time. They each
diff --git a/src/backend/postmaster/postmaster.c b/src/backend/postmaster/postmaster.c
index d2a7a7add6f..1c8ae3b27dd 100644
--- a/src/backend/postmaster/postmaster.c
+++ b/src/backend/postmaster/postmaster.c
@@ -108,9 +108,12 @@
#include "replication/logicallauncher.h"
#include "replication/slotsync.h"
#include "replication/walsender.h"
+#include "storage/aio_subsys.h"
#include "storage/fd.h"
+#include "storage/io_worker.h"
#include "storage/ipc.h"
#include "storage/pmsignal.h"
+#include "storage/proc.h"
#include "tcop/backend_startup.h"
#include "tcop/tcopprot.h"
#include "utils/datetime.h"
@@ -341,6 +344,7 @@ typedef enum
* ckpt */
PM_WAIT_XLOG_ARCHIVAL, /* waiting for archiver and walsenders to
* finish */
+ PM_WAIT_IO_WORKERS, /* waiting for io workers to exit */
PM_WAIT_CHECKPOINTER, /* waiting for checkpointer to shut down */
PM_WAIT_DEAD_END, /* waiting for dead-end children to exit */
PM_NO_CHILDREN, /* all important children have exited */
@@ -403,6 +407,10 @@ bool LoadedSSL = false;
static DNSServiceRef bonjour_sdref = NULL;
#endif
+/* State for IO worker management. */
+static int io_worker_count = 0;
+static PMChild *io_worker_children[MAX_IO_WORKERS];
+
/*
* postmaster.c - function prototypes
*/
@@ -437,6 +445,8 @@ static void TerminateChildren(int signal);
static int CountChildren(BackendTypeMask targetMask);
static void LaunchMissingBackgroundProcesses(void);
static void maybe_start_bgworkers(void);
+static bool maybe_reap_io_worker(int pid);
+static void maybe_adjust_io_workers(void);
static bool CreateOptsFile(int argc, char *argv[], char *fullprogname);
static PMChild *StartChildProcess(BackendType type);
static void StartSysLogger(void);
@@ -1366,6 +1376,11 @@ PostmasterMain(int argc, char *argv[])
*/
AddToDataDirLockFile(LOCK_FILE_LINE_PM_STATUS, PM_STATUS_STARTING);
+ UpdatePMState(PM_STARTUP);
+
+ /* Make sure we can perform I/O while starting up. */
+ maybe_adjust_io_workers();
+
/* Start bgwriter and checkpointer so they can help with recovery */
if (CheckpointerPMChild == NULL)
CheckpointerPMChild = StartChildProcess(B_CHECKPOINTER);
@@ -1378,7 +1393,6 @@ PostmasterMain(int argc, char *argv[])
StartupPMChild = StartChildProcess(B_STARTUP);
Assert(StartupPMChild != NULL);
StartupStatus = STARTUP_RUNNING;
- UpdatePMState(PM_STARTUP);
/* Some workers may be scheduled to start now */
maybe_start_bgworkers();
@@ -2503,6 +2517,16 @@ process_pm_child_exit(void)
continue;
}
+ /* Was it an IO worker? */
+ if (maybe_reap_io_worker(pid))
+ {
+ if (!EXIT_STATUS_0(exitstatus) && !EXIT_STATUS_1(exitstatus))
+ HandleChildCrash(pid, exitstatus, _("io worker"));
+
+ maybe_adjust_io_workers();
+ continue;
+ }
+
/*
* Was it a backend or a background worker?
*/
@@ -2724,6 +2748,7 @@ HandleFatalError(QuitSignalReason reason, bool consider_sigabrt)
case PM_WAIT_XLOG_SHUTDOWN:
case PM_WAIT_XLOG_ARCHIVAL:
case PM_WAIT_CHECKPOINTER:
+ case PM_WAIT_IO_WORKERS:
/*
* NB: Similar code exists in PostmasterStateMachine()'s handling
@@ -2906,20 +2931,21 @@ PostmasterStateMachine(void)
/*
* If we are doing crash recovery or an immediate shutdown then we
- * expect archiver, checkpointer and walsender to exit as well,
- * otherwise not.
+ * expect archiver, checkpointer, io workers and walsender to exit as
+ * well, otherwise not.
*/
if (FatalError || Shutdown >= ImmediateShutdown)
targetMask = btmask_add(targetMask,
B_CHECKPOINTER,
B_ARCHIVER,
+ B_IO_WORKER,
B_WAL_SENDER);
/*
- * Normally walsenders and archiver will continue running; they will
- * be terminated later after writing the checkpoint record. We also
- * let dead-end children to keep running for now. The syslogger
- * process exits last.
+ * Normally archiver, checkpointer, IO workers and walsenders will
+ * continue running; they will be terminated later after writing the
+ * checkpoint record. We also let dead-end children to keep running
+ * for now. The syslogger process exits last.
*
* This assertion checks that we have covered all backend types,
* either by including them in targetMask, or by noting here that they
@@ -2934,12 +2960,13 @@ PostmasterStateMachine(void)
B_LOGGER);
/*
- * Archiver, checkpointer and walsender may or may not be in
- * targetMask already.
+ * Archiver, checkpointer, IO workers, and walsender may or may
+ * not be in targetMask already.
*/
remainMask = btmask_add(remainMask,
B_ARCHIVER,
B_CHECKPOINTER,
+ B_IO_WORKER,
B_WAL_SENDER);
/* these are not real postmaster children */
@@ -3040,11 +3067,25 @@ PostmasterStateMachine(void)
{
/*
* PM_WAIT_XLOG_ARCHIVAL state ends when there are no children other
- * than checkpointer, dead-end children and logger left. There
+ * than checkpointer, io workers and dead-end children left. There
* shouldn't be any regular backends left by now anyway; what we're
* really waiting for is for walsenders and archiver to exit.
*/
- if (CountChildren(btmask_all_except(B_CHECKPOINTER, B_LOGGER, B_DEAD_END_BACKEND)) == 0)
+ if (CountChildren(btmask_all_except(B_CHECKPOINTER, B_IO_WORKER,
+ B_LOGGER, B_DEAD_END_BACKEND)) == 0)
+ {
+ UpdatePMState(PM_WAIT_IO_WORKERS);
+ SignalChildren(SIGUSR2, btmask(B_IO_WORKER));
+ }
+ }
+
+ if (pmState == PM_WAIT_IO_WORKERS)
+ {
+ /*
+ * PM_WAIT_IO_WORKERS state ends when there's only checkpointer and
+ * dead_end children left.
+ */
+ if (io_worker_count == 0)
{
UpdatePMState(PM_WAIT_CHECKPOINTER);
@@ -3172,10 +3213,14 @@ PostmasterStateMachine(void)
/* re-create shared memory and semaphores */
CreateSharedMemoryAndSemaphores();
+ UpdatePMState(PM_STARTUP);
+
+ /* Make sure we can perform I/O while starting up. */
+ maybe_adjust_io_workers();
+
StartupPMChild = StartChildProcess(B_STARTUP);
Assert(StartupPMChild != NULL);
StartupStatus = STARTUP_RUNNING;
- UpdatePMState(PM_STARTUP);
/* crash recovery started, reset SIGKILL flag */
AbortStartTime = 0;
@@ -3199,6 +3244,7 @@ pmstate_name(PMState state)
PM_TOSTR_CASE(PM_WAIT_BACKENDS);
PM_TOSTR_CASE(PM_WAIT_XLOG_SHUTDOWN);
PM_TOSTR_CASE(PM_WAIT_XLOG_ARCHIVAL);
+ PM_TOSTR_CASE(PM_WAIT_IO_WORKERS);
PM_TOSTR_CASE(PM_WAIT_DEAD_END);
PM_TOSTR_CASE(PM_WAIT_CHECKPOINTER);
PM_TOSTR_CASE(PM_NO_CHILDREN);
@@ -4115,6 +4161,7 @@ bgworker_should_start_now(BgWorkerStartTime start_time)
case PM_WAIT_DEAD_END:
case PM_WAIT_XLOG_ARCHIVAL:
case PM_WAIT_XLOG_SHUTDOWN:
+ case PM_WAIT_IO_WORKERS:
case PM_WAIT_BACKENDS:
case PM_STOP_BACKENDS:
break;
@@ -4265,6 +4312,100 @@ maybe_start_bgworkers(void)
}
}
+static bool
+maybe_reap_io_worker(int pid)
+{
+ for (int id = 0; id < MAX_IO_WORKERS; ++id)
+ {
+ if (io_worker_children[id] &&
+ io_worker_children[id]->pid == pid)
+ {
+ ReleasePostmasterChildSlot(io_worker_children[id]);
+
+ --io_worker_count;
+ io_worker_children[id] = NULL;
+ return true;
+ }
+ }
+ return false;
+}
+
+static void
+maybe_adjust_io_workers(void)
+{
+ if (!pgaio_workers_enabled())
+ return;
+
+ /*
+ * If we're in final shutting down state, then we're just waiting for all
+ * processes to exit.
+ */
+ if (pmState >= PM_WAIT_IO_WORKERS)
+ return;
+
+ /* Don't start new workers during an immediate shutdown either. */
+ if (Shutdown >= ImmediateShutdown)
+ return;
+
+ /*
+ * Don't start new workers if we're in the shutdown phase of a crash
+ * restart. But we *do* need to start if we're already starting up again.
+ */
+ if (FatalError && pmState >= PM_STOP_BACKENDS)
+ return;
+
+ Assert(pmState < PM_WAIT_IO_WORKERS);
+
+ /* Not enough running? */
+ while (io_worker_count < io_workers)
+ {
+ PMChild *child;
+ int id;
+
+ /* find unused entry in io_worker_children array */
+ for (id = 0; id < MAX_IO_WORKERS; ++id)
+ {
+ if (io_worker_children[id] == NULL)
+ break;
+ }
+ if (id == MAX_IO_WORKERS)
+ elog(ERROR, "could not find a free IO worker ID");
+
+ /* Try to launch one. */
+ child = StartChildProcess(B_IO_WORKER);
+ if (child != NULL)
+ {
+ io_worker_children[id] = child;
+ ++io_worker_count;
+ }
+ else
+ break; /* XXX try again soon? */
+ }
+
+ /* Too many running? */
+ if (io_worker_count > io_workers)
+ {
+ /* ask the IO worker in the highest slot to exit */
+ for (int id = MAX_IO_WORKERS - 1; id >= 0; --id)
+ {
+ if (io_worker_children[id] != NULL)
+ {
+ kill(io_worker_children[id]->pid, SIGUSR2);
+ break;
+ }
+ }
+ }
+}
+
+void
+assign_io_workers(int newval, void *extra)
+{
+ io_workers = newval;
+ if (!IsUnderPostmaster && pmState > PM_INIT)
+ maybe_adjust_io_workers();
+}
+
+
/*
* When a backend asks to be notified about worker state changes, we
* set a flag in its backend entry. The background worker machinery needs
diff --git a/src/backend/storage/aio/Makefile b/src/backend/storage/aio/Makefile
index 89f821ea7e1..f51c34a37f8 100644
--- a/src/backend/storage/aio/Makefile
+++ b/src/backend/storage/aio/Makefile
@@ -15,6 +15,7 @@ OBJS = \
aio_io.o \
aio_target.o \
method_sync.o \
+ method_worker.o \
read_stream.o
include $(top_srcdir)/src/backend/common.mk
diff --git a/src/backend/storage/aio/meson.build b/src/backend/storage/aio/meson.build
index 2c26089d52e..74f94c6e40b 100644
--- a/src/backend/storage/aio/meson.build
+++ b/src/backend/storage/aio/meson.build
@@ -7,5 +7,6 @@ backend_sources += files(
'aio_io.c',
'aio_target.c',
'method_sync.c',
+ 'method_worker.c',
'read_stream.c',
)
diff --git a/src/backend/storage/aio/method_worker.c b/src/backend/storage/aio/method_worker.c
new file mode 100644
index 00000000000..0ef9ef93e2b
--- /dev/null
+++ b/src/backend/storage/aio/method_worker.c
@@ -0,0 +1,88 @@
+/*-------------------------------------------------------------------------
+ *
+ * method_worker.c
+ * AIO - perform AIO using worker processes
+ *
+ * Portions Copyright (c) 1996-2025, PostgreSQL Global Development Group
+ * Portions Copyright (c) 1994, Regents of the University of California
+ *
+ * IDENTIFICATION
+ * src/backend/storage/aio/method_worker.c
+ *
+ *-------------------------------------------------------------------------
+ */
+
+#include "postgres.h"
+
+#include "libpq/pqsignal.h"
+#include "miscadmin.h"
+#include "postmaster/auxprocess.h"
+#include "postmaster/interrupt.h"
+#include "storage/aio_subsys.h"
+#include "storage/io_worker.h"
+#include "storage/ipc.h"
+#include "storage/latch.h"
+#include "storage/proc.h"
+#include "tcop/tcopprot.h"
+#include "utils/wait_event.h"
+
+
+/* GUCs */
+int io_workers = 3;
+
+
+void
+IoWorkerMain(const void *startup_data, size_t startup_data_len)
+{
+ sigjmp_buf local_sigjmp_buf;
+
+ MyBackendType = B_IO_WORKER;
+ AuxiliaryProcessMainCommon();
+
+ pqsignal(SIGHUP, SignalHandlerForConfigReload);
+ pqsignal(SIGINT, die); /* to allow manually triggering worker restart */
+
+ /*
+ * Ignore SIGTERM, will get explicit shutdown via SIGUSR2 later in the
+ * shutdown sequence, similar to checkpointer.
+ */
+ pqsignal(SIGTERM, SIG_IGN);
+ /* SIGQUIT handler was already set up by InitPostmasterChild */
+ pqsignal(SIGALRM, SIG_IGN);
+ pqsignal(SIGPIPE, SIG_IGN);
+ pqsignal(SIGUSR1, procsignal_sigusr1_handler);
+ pqsignal(SIGUSR2, SignalHandlerForShutdownRequest);
+
+ /* see PostgresMain() */
+ if (sigsetjmp(local_sigjmp_buf, 1) != 0)
+ {
+ error_context_stack = NULL;
+ HOLD_INTERRUPTS();
+
+ EmitErrorReport();
+
+ proc_exit(1);
+ }
+
+ /* We can now handle ereport(ERROR) */
+ PG_exception_stack = &local_sigjmp_buf;
+
+ sigprocmask(SIG_SETMASK, &UnBlockSig, NULL);
+
+ while (!ShutdownRequestPending)
+ {
+ WaitLatch(MyLatch, WL_LATCH_SET | WL_EXIT_ON_PM_DEATH, -1,
+ WAIT_EVENT_IO_WORKER_MAIN);
+ ResetLatch(MyLatch);
+ CHECK_FOR_INTERRUPTS();
+ }
+
+ proc_exit(0);
+}
+
+bool
+pgaio_workers_enabled(void)
+{
+ /* placeholder for future commit */
+ return false;
+}
diff --git a/src/backend/tcop/postgres.c b/src/backend/tcop/postgres.c
index 947ffb40421..812e224cf14 100644
--- a/src/backend/tcop/postgres.c
+++ b/src/backend/tcop/postgres.c
@@ -3315,6 +3315,8 @@ ProcessInterrupts(void)
(errcode(ERRCODE_ADMIN_SHUTDOWN),
errmsg("terminating background worker \"%s\" due to administrator command",
MyBgworkerEntry->bgw_type)));
+ else if (AmIoWorkerProcess())
+ proc_exit(0);
else
ereport(FATAL,
(errcode(ERRCODE_ADMIN_SHUTDOWN),
diff --git a/src/backend/utils/activity/pgstat_backend.c b/src/backend/utils/activity/pgstat_backend.c
index 6efbb650aa8..51f3c550f6d 100644
--- a/src/backend/utils/activity/pgstat_backend.c
+++ b/src/backend/utils/activity/pgstat_backend.c
@@ -294,6 +294,7 @@ pgstat_tracks_backend_bktype(BackendType bktype)
case B_LOGGER:
case B_BG_WRITER:
case B_CHECKPOINTER:
+ case B_IO_WORKER:
case B_STARTUP:
return false;
diff --git a/src/backend/utils/activity/pgstat_io.c b/src/backend/utils/activity/pgstat_io.c
index eb575025596..c8de9c9e2d3 100644
--- a/src/backend/utils/activity/pgstat_io.c
+++ b/src/backend/utils/activity/pgstat_io.c
@@ -376,6 +376,7 @@ pgstat_tracks_io_bktype(BackendType bktype)
case B_BG_WORKER:
case B_BG_WRITER:
case B_CHECKPOINTER:
+ case B_IO_WORKER:
case B_SLOTSYNC_WORKER:
case B_STANDALONE_BACKEND:
case B_STARTUP:
diff --git a/src/backend/utils/activity/wait_event_names.txt b/src/backend/utils/activity/wait_event_names.txt
index b44e4908b25..3f6dc3876b4 100644
--- a/src/backend/utils/activity/wait_event_names.txt
+++ b/src/backend/utils/activity/wait_event_names.txt
@@ -57,6 +57,7 @@ BGWRITER_HIBERNATE "Waiting in background writer process, hibernating."
BGWRITER_MAIN "Waiting in main loop of background writer process."
CHECKPOINTER_MAIN "Waiting in main loop of checkpointer process."
CHECKPOINTER_SHUTDOWN "Waiting for checkpointer process to be terminated."
+IO_WORKER_MAIN "Waiting in main loop of IO Worker process."
LOGICAL_APPLY_MAIN "Waiting in main loop of logical replication apply process."
LOGICAL_LAUNCHER_MAIN "Waiting in main loop of logical replication launcher process."
LOGICAL_PARALLEL_APPLY_MAIN "Waiting in main loop of logical replication parallel apply process."
diff --git a/src/backend/utils/init/miscinit.c b/src/backend/utils/init/miscinit.c
index dc3521457c7..2acc505383e 100644
--- a/src/backend/utils/init/miscinit.c
+++ b/src/backend/utils/init/miscinit.c
@@ -293,6 +293,9 @@ GetBackendTypeDesc(BackendType backendType)
case B_CHECKPOINTER:
backendDesc = gettext_noop("checkpointer");
break;
+ case B_IO_WORKER:
+ backendDesc = "io worker";
+ break;
case B_LOGGER:
backendDesc = gettext_noop("logger");
break;
diff --git a/src/backend/utils/misc/guc_tables.c b/src/backend/utils/misc/guc_tables.c
index 28dcc465762..4317cfc9d2f 100644
--- a/src/backend/utils/misc/guc_tables.c
+++ b/src/backend/utils/misc/guc_tables.c
@@ -75,6 +75,7 @@
#include "storage/aio.h"
#include "storage/bufmgr.h"
#include "storage/bufpage.h"
+#include "storage/io_worker.h"
#include "storage/large_object.h"
#include "storage/pg_shmem.h"
#include "storage/predicate.h"
@@ -3267,6 +3268,18 @@ struct config_int ConfigureNamesInt[] =
check_io_max_concurrency, NULL, NULL
},
+ {
+ {"io_workers",
+ PGC_SIGHUP,
+ RESOURCES_IO,
+ gettext_noop("Number of IO worker processes, for io_method=worker."),
+ NULL,
+ },
+ &io_workers,
+ 3, 1, MAX_IO_WORKERS,
+ NULL, assign_io_workers, NULL
+ },
+
{
{"backend_flush_after", PGC_USERSET, RESOURCES_IO,
gettext_noop("Number of pages after which previously performed writes are flushed to disk."),
diff --git a/src/backend/utils/misc/postgresql.conf.sample b/src/backend/utils/misc/postgresql.conf.sample
index a31c9963ce4..f901352c868 100644
--- a/src/backend/utils/misc/postgresql.conf.sample
+++ b/src/backend/utils/misc/postgresql.conf.sample
@@ -207,6 +207,7 @@
# can execute simultaneously
# -1 sets based on shared_buffers
# (change requires restart)
+#io_workers = 3 # 1-32;
# - Worker Processes -
diff --git a/doc/src/sgml/config.sgml b/doc/src/sgml/config.sgml
index 7d5482f31cd..c6f8171e39a 100644
--- a/doc/src/sgml/config.sgml
+++ b/doc/src/sgml/config.sgml
@@ -2695,6 +2695,25 @@ include_dir 'conf.d'
</listitem>
</varlistentry>
+ <varlistentry id="guc-io-workers" xreflabel="io_workers">
+ <term><varname>io_workers</varname> (<type>int</type>)
+ <indexterm>
+ <primary><varname>io_workers</varname> configuration parameter</primary>
+ </indexterm>
+ </term>
+ <listitem>
+ <para>
+ Selects the number of I/O worker processes to use. The default is
+ 3. This parameter can only be set in the
+ <filename>postgresql.conf</filename> file or on the server command
+ line.
+ </para>
+ <para>
+ Only has an effect if <xref linkend="guc-max-wal-senders"/> is set to
+ <literal>worker</literal>.
+ </para>
+ </listitem>
+ </varlistentry>
</variablelist>
</sect2>
diff --git a/src/test/regress/expected/stats.out b/src/test/regress/expected/stats.out
index 30d763c4aee..0e0b9b8e6b4 100644
--- a/src/test/regress/expected/stats.out
+++ b/src/test/regress/expected/stats.out
@@ -51,6 +51,14 @@ client backend|relation|vacuum
client backend|temp relation|normal
client backend|wal|init
client backend|wal|normal
+io worker|relation|bulkread
+io worker|relation|bulkwrite
+io worker|relation|init
+io worker|relation|normal
+io worker|relation|vacuum
+io worker|temp relation|normal
+io worker|wal|init
+io worker|wal|normal
slotsync worker|relation|bulkread
slotsync worker|relation|bulkwrite
slotsync worker|relation|init
@@ -87,7 +95,7 @@ walsummarizer|wal|init
walsummarizer|wal|normal
walwriter|wal|init
walwriter|wal|normal
-(71 rows)
+(79 rows)
\a
-- ensure that both seqscan and indexscan plans are allowed
SET enable_seqscan TO on;
--
2.48.1.76.g4e746b1a31.dirty
[text/x-diff] v2.6-0004-aio-Add-io_method-worker.patch (20.1K, ../../ge6nsuddurhpmll3xj22vucvqwp4agqz6ndtcf2mhyeydzarst@l75dman5x53p/5-v2.6-0004-aio-Add-io_method-worker.patch)
download | inline diff:
From 40a03bed243e3a31a73bf4912964e8b5254fed16 Mon Sep 17 00:00:00 2001
From: Andres Freund <andres@anarazel.de>
Date: Fri, 31 Jan 2025 13:46:35 -0500
Subject: [PATCH v2.6 04/34] aio: Add io_method=worker
The previous commit introduced the infrastructure to start io_workers. This
commit actually makes the workers execute IOs.
IO workers consume IOs from a shared memory submission queue, run traditional
synchronous system calls, and perform the shared completion handling
immediately. Client code submits most requests by pushing IOs into the
submission queue, and waits (if necessary) using condition variables. Some
IOs cannot be performed in another process due to lack of infrastructure for
reopening the file, and must processed synchronously by the client code when
submitted.
For now the default io_method is changed to "worker". We should re-evaluate
that around beta1, we might want to be careful and set the default to "sync"
for 18.
Co-authored-by: Thomas Munro <thomas.munro@gmail.com>
Co-authored-by: Andres Freund <andres@anarazel.de>
Discussion: https://postgr.es/m/uvrtrknj4kdytuboidbhwclo4gxhswwcpgadptsjvjqcluzmah%40brqs62irg4dt
Discussion: https://postgr.es/m/20210223100344.llw5an2aklengrmn@alap3.anarazel.de
Discussion: https://postgr.es/m/stj36ea6yyhoxtqkhpieia2z4krnam7qyetc57rfezgk4zgapf@gcnactj4z56m
---
src/include/storage/aio.h | 5 +-
src/include/storage/aio_internal.h | 1 +
src/include/storage/lwlocklist.h | 1 +
src/backend/storage/aio/aio.c | 2 +
src/backend/storage/aio/aio_init.c | 9 +
src/backend/storage/aio/method_worker.c | 432 +++++++++++++++++-
.../utils/activity/wait_event_names.txt | 1 +
src/backend/utils/misc/postgresql.conf.sample | 2 +-
doc/src/sgml/config.sgml | 5 +
src/tools/pgindent/typedefs.list | 3 +
10 files changed, 453 insertions(+), 8 deletions(-)
diff --git a/src/include/storage/aio.h b/src/include/storage/aio.h
index 0124dc4d24d..7a422a4d131 100644
--- a/src/include/storage/aio.h
+++ b/src/include/storage/aio.h
@@ -23,10 +23,11 @@
typedef enum IoMethod
{
IOMETHOD_SYNC = 0,
+ IOMETHOD_WORKER,
} IoMethod;
-/* We'll default to synchronous execution. */
-#define DEFAULT_IO_METHOD IOMETHOD_SYNC
+/* We'll default to worker based execution. */
+#define DEFAULT_IO_METHOD IOMETHOD_WORKER
/*
diff --git a/src/include/storage/aio_internal.h b/src/include/storage/aio_internal.h
index d213b0842a3..a626581dd04 100644
--- a/src/include/storage/aio_internal.h
+++ b/src/include/storage/aio_internal.h
@@ -353,6 +353,7 @@ extern PgAioHandle *pgaio_inj_io_get(void);
/* Declarations for the tables of function pointers exposed by each IO method. */
extern PGDLLIMPORT const IoMethodOps pgaio_sync_ops;
+extern PGDLLIMPORT const IoMethodOps pgaio_worker_ops;
extern PGDLLIMPORT const IoMethodOps *pgaio_method_ops;
extern PGDLLIMPORT PgAioCtl *pgaio_ctl;
diff --git a/src/include/storage/lwlocklist.h b/src/include/storage/lwlocklist.h
index cf565452382..932024b1b0b 100644
--- a/src/include/storage/lwlocklist.h
+++ b/src/include/storage/lwlocklist.h
@@ -83,3 +83,4 @@ PG_LWLOCK(49, WALSummarizer)
PG_LWLOCK(50, DSMRegistry)
PG_LWLOCK(51, InjectionPoint)
PG_LWLOCK(52, SerialControl)
+PG_LWLOCK(53, AioWorkerSubmissionQueue)
diff --git a/src/backend/storage/aio/aio.c b/src/backend/storage/aio/aio.c
index 26dd23cfab5..1e756d37b93 100644
--- a/src/backend/storage/aio/aio.c
+++ b/src/backend/storage/aio/aio.c
@@ -64,6 +64,7 @@ static void pgaio_io_wait(PgAioHandle *ioh, uint64 ref_generation);
/* Options for io_method. */
const struct config_enum_entry io_method_options[] = {
{"sync", IOMETHOD_SYNC, false},
+ {"worker", IOMETHOD_WORKER, false},
{NULL, 0, false}
};
@@ -80,6 +81,7 @@ PgAioBackend *pgaio_my_backend;
static const IoMethodOps *const pgaio_method_ops_table[] = {
[IOMETHOD_SYNC] = &pgaio_sync_ops,
+ [IOMETHOD_WORKER] = &pgaio_worker_ops,
};
/* callbacks for the configured io_method, set by assign_io_method */
diff --git a/src/backend/storage/aio/aio_init.c b/src/backend/storage/aio/aio_init.c
index 8e1162b09dc..2205658cd9a 100644
--- a/src/backend/storage/aio/aio_init.c
+++ b/src/backend/storage/aio/aio_init.c
@@ -18,6 +18,7 @@
#include "storage/aio.h"
#include "storage/aio_internal.h"
#include "storage/aio_subsys.h"
+#include "storage/io_worker.h"
#include "storage/ipc.h"
#include "storage/proc.h"
#include "storage/shmem.h"
@@ -39,6 +40,11 @@ AioCtlShmemSize(void)
static uint32
AioProcs(void)
{
+ /*
+ * While AIO workers don't need their own AIO context, we can't currently
+ * guarantee nothing gets assigned to the a ProcNumber for an IO worker if
+ * we just subtracted MAX_IO_WORKERS.
+ */
return MaxBackends + NUM_AUXILIARY_PROCS;
}
@@ -211,6 +217,9 @@ pgaio_init_backend(void)
/* shouldn't be initialized twice */
Assert(!pgaio_my_backend);
+ if (MyBackendType == B_IO_WORKER)
+ return;
+
if (MyProc == NULL || MyProcNumber >= AioProcs())
elog(ERROR, "aio requires a normal PGPROC");
diff --git a/src/backend/storage/aio/method_worker.c b/src/backend/storage/aio/method_worker.c
index 0ef9ef93e2b..29de257565d 100644
--- a/src/backend/storage/aio/method_worker.c
+++ b/src/backend/storage/aio/method_worker.c
@@ -3,6 +3,22 @@
* method_worker.c
* AIO - perform AIO using worker processes
*
+ * IO workers consume IOs from a shared memory submission queue, run
+ * traditional synchronous system calls, and perform the shared completion
+ * handling immediately. Client code submits most requests by pushing IOs
+ * into the submission queue, and waits (if necessary) using condition
+ * variables. Some IOs cannot be performed in another process due to lack of
+ * infrastructure for reopening the file, and must processed synchronously by
+ * the client code when submitted.
+ *
+ * So that the submitter can make just one system call when submitting a batch
+ * of IOs, wakeups "fan out"; each woken IO worker can wake two more. qXXX
+ * This could be improved by using futexes instead of latches to wake N
+ * waiters.
+ *
+ * This method of AIO is available in all builds on all operating systems, and
+ * is the default.
+ *
* Portions Copyright (c) 1996-2025, PostgreSQL Global Development Group
* Portions Copyright (c) 1994, Regents of the University of California
*
@@ -16,25 +32,325 @@
#include "libpq/pqsignal.h"
#include "miscadmin.h"
+#include "port/pg_bitutils.h"
#include "postmaster/auxprocess.h"
#include "postmaster/interrupt.h"
+#include "storage/aio.h"
+#include "storage/aio_internal.h"
#include "storage/aio_subsys.h"
#include "storage/io_worker.h"
#include "storage/ipc.h"
#include "storage/latch.h"
#include "storage/proc.h"
#include "tcop/tcopprot.h"
+#include "utils/ps_status.h"
#include "utils/wait_event.h"
+/* How many workers should each worker wake up if needed? */
+#define IO_WORKER_WAKEUP_FANOUT 2
+
+
+typedef struct AioWorkerSubmissionQueue
+{
+ uint32 size;
+ uint32 mask;
+ uint32 head;
+ uint32 tail;
+ uint32 ios[FLEXIBLE_ARRAY_MEMBER];
+} AioWorkerSubmissionQueue;
+
+typedef struct AioWorkerSlot
+{
+ Latch *latch;
+ bool in_use;
+} AioWorkerSlot;
+
+typedef struct AioWorkerControl
+{
+ uint64 idle_worker_mask;
+ AioWorkerSlot workers[FLEXIBLE_ARRAY_MEMBER];
+} AioWorkerControl;
+
+
+static size_t pgaio_worker_shmem_size(void);
+static void pgaio_worker_shmem_init(bool first_time);
+
+static bool pgaio_worker_needs_synchronous_execution(PgAioHandle *ioh);
+static int pgaio_worker_submit(uint16 num_staged_ios, PgAioHandle **staged_ios);
+
+
+const IoMethodOps pgaio_worker_ops = {
+ .shmem_size = pgaio_worker_shmem_size,
+ .shmem_init = pgaio_worker_shmem_init,
+
+ .needs_synchronous_execution = pgaio_worker_needs_synchronous_execution,
+ .submit = pgaio_worker_submit,
+};
+
+
/* GUCs */
int io_workers = 3;
+static int io_worker_queue_size = 64;
+static int MyIoWorkerId;
+static AioWorkerSubmissionQueue *io_worker_submission_queue;
+static AioWorkerControl *io_worker_control;
+
+
+static size_t
+pgaio_worker_shmem_size(void)
+{
+ return
+ offsetof(AioWorkerSubmissionQueue, ios) +
+ sizeof(uint32) * MAX_IO_WORKERS * io_max_concurrency +
+ offsetof(AioWorkerControl, workers) +
+ sizeof(AioWorkerSlot) * MAX_IO_WORKERS;
+}
+
+static void
+pgaio_worker_shmem_init(bool first_time)
+{
+ bool found;
+ int size;
+
+ /* Round size up to next power of two so we can make a mask. */
+ size = pg_nextpower2_32(io_worker_queue_size);
+
+ io_worker_submission_queue =
+ ShmemInitStruct("AioWorkerSubmissionQueue",
+ offsetof(AioWorkerSubmissionQueue, ios) +
+ sizeof(uint32) * size,
+ &found);
+ if (!found)
+ {
+ io_worker_submission_queue->size = size;
+ io_worker_submission_queue->head = 0;
+ io_worker_submission_queue->tail = 0;
+ }
+
+ io_worker_control =
+ ShmemInitStruct("AioWorkerControl",
+ offsetof(AioWorkerControl, workers) +
+ sizeof(AioWorkerSlot) * io_workers,
+ &found);
+ if (!found)
+ {
+ io_worker_control->idle_worker_mask = 0;
+ for (int i = 0; i < io_workers; ++i)
+ {
+ io_worker_control->workers[i].latch = NULL;
+ io_worker_control->workers[i].in_use = false;
+ }
+ }
+}
+
+static int
+pgaio_choose_idle_worker(void)
+{
+ int worker;
+
+ if (io_worker_control->idle_worker_mask == 0)
+ return -1;
+
+ /* Find the lowest bit position, and clear it. */
+ worker = pg_rightmost_one_pos64(io_worker_control->idle_worker_mask);
+ io_worker_control->idle_worker_mask &= ~(UINT64_C(1) << worker);
+
+ return worker;
+}
+
+static bool
+pgaio_worker_submission_queue_insert(PgAioHandle *ioh)
+{
+ AioWorkerSubmissionQueue *queue;
+ uint32 new_head;
+
+ queue = io_worker_submission_queue;
+ new_head = (queue->head + 1) & (queue->size - 1);
+ if (new_head == queue->tail)
+ {
+ pgaio_debug(DEBUG3, "io queue is full, at %u elements",
+ io_worker_submission_queue->size);
+ return false; /* full */
+ }
+
+ queue->ios[queue->head] = pgaio_io_get_id(ioh);
+ queue->head = new_head;
+
+ return true;
+}
+
+static uint32
+pgaio_worker_submission_queue_consume(void)
+{
+ AioWorkerSubmissionQueue *queue;
+ uint32 result;
+
+ queue = io_worker_submission_queue;
+ if (queue->tail == queue->head)
+ return UINT32_MAX; /* empty */
+
+ result = queue->ios[queue->tail];
+ queue->tail = (queue->tail + 1) & (queue->size - 1);
+
+ return result;
+}
+
+static uint32
+pgaio_worker_submission_queue_depth(void)
+{
+ uint32 head;
+ uint32 tail;
+
+ head = io_worker_submission_queue->head;
+ tail = io_worker_submission_queue->tail;
+
+ if (tail > head)
+ head += io_worker_submission_queue->size;
+
+ Assert(head >= tail);
+
+ return head - tail;
+}
+
+static bool
+pgaio_worker_needs_synchronous_execution(PgAioHandle *ioh)
+{
+ return
+ !IsUnderPostmaster
+ || ioh->flags & PGAIO_HF_REFERENCES_LOCAL
+ || !pgaio_io_can_reopen(ioh);
+}
+
+static void
+pgaio_worker_submit_internal(int nios, PgAioHandle *ios[])
+{
+ PgAioHandle *synchronous_ios[PGAIO_SUBMIT_BATCH_SIZE];
+ int nsync = 0;
+ Latch *wakeup = NULL;
+ int worker;
+
+ Assert(nios <= PGAIO_SUBMIT_BATCH_SIZE);
+
+ LWLockAcquire(AioWorkerSubmissionQueueLock, LW_EXCLUSIVE);
+ for (int i = 0; i < nios; ++i)
+ {
+ Assert(!pgaio_worker_needs_synchronous_execution(ios[i]));
+ if (!pgaio_worker_submission_queue_insert(ios[i]))
+ {
+ /*
+ * We'll do it synchronously, but only after we've sent as many as
+ * we can to workers, to maximize concurrency.
+ */
+ synchronous_ios[nsync++] = ios[i];
+ continue;
+ }
+
+ if (wakeup == NULL)
+ {
+ /* Choose an idle worker to wake up if we haven't already. */
+ worker = pgaio_choose_idle_worker();
+ if (worker >= 0)
+ wakeup = io_worker_control->workers[worker].latch;
+
+ pgaio_debug_io(DEBUG4, ios[i],
+ "choosing worker %d",
+ worker);
+ }
+ }
+ LWLockRelease(AioWorkerSubmissionQueueLock);
+
+ if (wakeup)
+ SetLatch(wakeup);
+
+ /* Run whatever is left synchronously. */
+ if (nsync > 0)
+ {
+ for (int i = 0; i < nsync; ++i)
+ {
+ pgaio_io_perform_synchronously(synchronous_ios[i]);
+ }
+ }
+}
+
+static int
+pgaio_worker_submit(uint16 num_staged_ios, PgAioHandle **staged_ios)
+{
+ for (int i = 0; i < num_staged_ios; i++)
+ {
+ PgAioHandle *ioh = staged_ios[i];
+
+ pgaio_io_prepare_submit(ioh);
+ }
+
+ pgaio_worker_submit_internal(num_staged_ios, staged_ios);
+
+ return num_staged_ios;
+}
+
+/*
+ * on_shmem_exit() callback that releases the worker's slot in
+ * io_worker_control.
+ */
+static void
+pgaio_worker_die(int code, Datum arg)
+{
+ LWLockAcquire(AioWorkerSubmissionQueueLock, LW_EXCLUSIVE);
+ Assert(io_worker_control->workers[MyIoWorkerId].in_use);
+ Assert(io_worker_control->workers[MyIoWorkerId].latch == MyLatch);
+
+ io_worker_control->workers[MyIoWorkerId].in_use = false;
+ io_worker_control->workers[MyIoWorkerId].latch = NULL;
+ LWLockRelease(AioWorkerSubmissionQueueLock);
+}
+
+/*
+ * Register the worker in shared memory, assign MyWorkerId and register a
+ * shutdown callback to release registration.
+ */
+static void
+pgaio_worker_register(void)
+{
+ MyIoWorkerId = -1;
+
+ /*
+ * XXX: This could do with more fine-grained locking. But it's also not
+ * very common for the number of workers to change at the moment...
+ */
+ LWLockAcquire(AioWorkerSubmissionQueueLock, LW_EXCLUSIVE);
+
+ for (int i = 0; i < io_workers; ++i)
+ {
+ if (!io_worker_control->workers[i].in_use)
+ {
+ Assert(io_worker_control->workers[i].latch == NULL);
+ io_worker_control->workers[i].in_use = true;
+ MyIoWorkerId = i;
+ break;
+ }
+ else
+ Assert(io_worker_control->workers[i].latch != NULL);
+ }
+
+ if (MyIoWorkerId == -1)
+ elog(ERROR, "couldn't find a free worker slot");
+
+ io_worker_control->idle_worker_mask |= (UINT64_C(1) << MyIoWorkerId);
+ io_worker_control->workers[MyIoWorkerId].latch = MyLatch;
+ LWLockRelease(AioWorkerSubmissionQueueLock);
+
+ on_shmem_exit(pgaio_worker_die, 0);
+}
+
void
IoWorkerMain(const void *startup_data, size_t startup_data_len)
{
sigjmp_buf local_sigjmp_buf;
+ PgAioHandle *volatile error_ioh = NULL;
+ volatile int error_errno = 0;
+ char cmd[128];
MyBackendType = B_IO_WORKER;
AuxiliaryProcessMainCommon();
@@ -53,6 +369,12 @@ IoWorkerMain(const void *startup_data, size_t startup_data_len)
pqsignal(SIGUSR1, procsignal_sigusr1_handler);
pqsignal(SIGUSR2, SignalHandlerForShutdownRequest);
+ /* also registers a shutdown callback to unregister */
+ pgaio_worker_register();
+
+ sprintf(cmd, "io worker: %d", MyIoWorkerId);
+ set_ps_display(cmd);
+
/* see PostgresMain() */
if (sigsetjmp(local_sigjmp_buf, 1) != 0)
{
@@ -61,6 +383,27 @@ IoWorkerMain(const void *startup_data, size_t startup_data_len)
EmitErrorReport();
+ /*
+ * In the - very unlikely - case that the IO failed in a way that
+ * raises an error we need to mark the IO as failed.
+ *
+ * Need to do just enough error recovery so that we can mark the IO as
+ * failed and then exit (postmaster will start a new worker).
+ */
+ LWLockReleaseAll();
+
+ if (error_ioh != NULL)
+ {
+ /* should never fail without setting error_errno */
+ Assert(error_errno != 0);
+
+ errno = error_errno;
+
+ START_CRIT_SECTION();
+ pgaio_io_process_completion(error_ioh, -error_errno);
+ END_CRIT_SECTION();
+ }
+
proc_exit(1);
}
@@ -71,9 +414,89 @@ IoWorkerMain(const void *startup_data, size_t startup_data_len)
while (!ShutdownRequestPending)
{
- WaitLatch(MyLatch, WL_LATCH_SET | WL_EXIT_ON_PM_DEATH, -1,
- WAIT_EVENT_IO_WORKER_MAIN);
- ResetLatch(MyLatch);
+ uint32 io_index;
+ Latch *latches[IO_WORKER_WAKEUP_FANOUT];
+ int nlatches = 0;
+ int nwakeups = 0;
+ int worker;
+
+ /* Try to get a job to do. */
+ LWLockAcquire(AioWorkerSubmissionQueueLock, LW_EXCLUSIVE);
+ if ((io_index = pgaio_worker_submission_queue_consume()) == UINT32_MAX)
+ {
+ /*
+ * Nothing to do. Mark self idle.
+ *
+ * XXX: Invent some kind of back pressure to reduce useless
+ * wakeups?
+ */
+ io_worker_control->idle_worker_mask |= (UINT64_C(1) << MyIoWorkerId);
+ }
+ else
+ {
+ /* Got one. Clear idle flag. */
+ io_worker_control->idle_worker_mask &= ~(UINT64_C(1) << MyIoWorkerId);
+
+ /* See if we can wake up some peers. */
+ nwakeups = Min(pgaio_worker_submission_queue_depth(),
+ IO_WORKER_WAKEUP_FANOUT);
+ for (int i = 0; i < nwakeups; ++i)
+ {
+ if ((worker = pgaio_choose_idle_worker()) < 0)
+ break;
+ latches[nlatches++] = io_worker_control->workers[worker].latch;
+ }
+ }
+ LWLockRelease(AioWorkerSubmissionQueueLock);
+
+ for (int i = 0; i < nlatches; ++i)
+ SetLatch(latches[i]);
+
+ if (io_index != UINT32_MAX)
+ {
+ PgAioHandle *ioh = NULL;
+
+ ioh = &pgaio_ctl->io_handles[io_index];
+ error_ioh = ioh;
+
+ pgaio_debug_io(DEBUG4, ioh,
+ "worker %d processing IO",
+ MyIoWorkerId);
+
+ /*
+ * It's very unlikely, but possible, that reopen fails. E.g. due
+ * to memory allocations failing or file permissions changing or
+ * such. In that case we need to fail the IO.
+ *
+ * There's not really a good errno we can report here.
+ */
+ error_errno = ENOENT;
+ pgaio_io_reopen(ioh);
+
+ /*
+ * To be able to exercise the reopen-fails path, allow injection
+ * points to trigger a failure at this point.
+ */
+ pgaio_io_call_inj(ioh, "AIO_WORKER_AFTER_REOPEN");
+
+ error_errno = 0;
+ error_ioh = NULL;
+
+ /*
+ * We don't expect this to ever fail with ERROR or FATAL, no need
+ * to keep error_ioh set to the IO.
+ * pgaio_io_perform_synchronously() contains a critical section to
+ * ensure we don't accidentally fail.
+ */
+ pgaio_io_perform_synchronously(ioh);
+ }
+ else
+ {
+ WaitLatch(MyLatch, WL_LATCH_SET | WL_EXIT_ON_PM_DEATH, -1,
+ WAIT_EVENT_IO_WORKER_MAIN);
+ ResetLatch(MyLatch);
+ }
+
CHECK_FOR_INTERRUPTS();
}
@@ -83,6 +506,5 @@ IoWorkerMain(const void *startup_data, size_t startup_data_len)
bool
pgaio_workers_enabled(void)
{
- /* placeholder for future commit */
- return false;
+ return io_method == IOMETHOD_WORKER;
}
diff --git a/src/backend/utils/activity/wait_event_names.txt b/src/backend/utils/activity/wait_event_names.txt
index 3f6dc3876b4..9fa12a555e8 100644
--- a/src/backend/utils/activity/wait_event_names.txt
+++ b/src/backend/utils/activity/wait_event_names.txt
@@ -348,6 +348,7 @@ WALSummarizer "Waiting to read or update WAL summarization state."
DSMRegistry "Waiting to read or update the dynamic shared memory registry."
InjectionPoint "Waiting to read or update information related to injection points."
SerialControl "Waiting to read or update shared <filename>pg_serial</filename> state."
+AioWorkerSubmissionQueue "Waiting to access AIO worker submission queue."
#
# END OF PREDEFINED LWLOCKS (DO NOT CHANGE THIS LINE)
diff --git a/src/backend/utils/misc/postgresql.conf.sample b/src/backend/utils/misc/postgresql.conf.sample
index f901352c868..1058726285f 100644
--- a/src/backend/utils/misc/postgresql.conf.sample
+++ b/src/backend/utils/misc/postgresql.conf.sample
@@ -202,7 +202,7 @@
#maintenance_io_concurrency = 10 # 1-1000; 0 disables prefetching
#io_combine_limit = 128kB # usually 1-32 blocks (depends on OS)
-#io_method = sync # sync (change requires restart)
+#io_method = worker # worker, sync (change requires restart)
#io_max_concurrency = -1 # Max number of IOs that one process
# can execute simultaneously
# -1 sets based on shared_buffers
diff --git a/doc/src/sgml/config.sgml b/doc/src/sgml/config.sgml
index c6f8171e39a..de1b918b06a 100644
--- a/doc/src/sgml/config.sgml
+++ b/doc/src/sgml/config.sgml
@@ -2682,6 +2682,11 @@ include_dir 'conf.d'
Selects the method for executing asynchronous I/O.
Possible values are:
<itemizedlist>
+ <listitem>
+ <para>
+ <literal>worker</literal> (execute asynchronous I/O using worker processes)
+ </para>
+ </listitem>
<listitem>
<para>
<literal>sync</literal> (execute asynchronous I/O synchronously)
diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list
index ce7f877adca..098ef0cd7a5 100644
--- a/src/tools/pgindent/typedefs.list
+++ b/src/tools/pgindent/typedefs.list
@@ -55,6 +55,9 @@ AggStrategy
AggTransInfo
Aggref
AggregateInstrumentation
+AioWorkerControl
+AioWorkerSlot
+AioWorkerSubmissionQueue
AlenState
Alias
AllocBlock
--
2.48.1.76.g4e746b1a31.dirty
[text/x-diff] v2.6-0005-aio-Add-liburing-dependency.patch (13.4K, ../../ge6nsuddurhpmll3xj22vucvqwp4agqz6ndtcf2mhyeydzarst@l75dman5x53p/6-v2.6-0005-aio-Add-liburing-dependency.patch)
download | inline diff:
From 73324630e79ce3788b449c095ee3851f78b23bf3 Mon Sep 17 00:00:00 2001
From: Andres Freund <andres@anarazel.de>
Date: Thu, 20 Feb 2025 17:56:05 -0500
Subject: [PATCH v2.6 05/34] aio: Add liburing dependency
Will be used in a subsequent commit, to implement io_method=io_uring. Kept
separate for easier review.
Reviewed-by: Noah Misch <noah@leadboat.com>
Discussion: https://postgr.es/m/uvrtrknj4kdytuboidbhwclo4gxhswwcpgadptsjvjqcluzmah%40brqs62irg4dt
---
meson.build | 14 ++++
meson_options.txt | 3 +
configure.ac | 11 +++
src/makefiles/meson.build | 3 +
src/include/pg_config.h.in | 3 +
src/backend/Makefile | 7 +-
doc/src/sgml/installation.sgml | 34 ++++++++
configure | 138 +++++++++++++++++++++++++++++++++
.cirrus.tasks.yml | 1 +
src/Makefile.global.in | 4 +
10 files changed, 215 insertions(+), 3 deletions(-)
diff --git a/meson.build b/meson.build
index 13c13748e5d..bed36153c7b 100644
--- a/meson.build
+++ b/meson.build
@@ -950,6 +950,18 @@ endif
+###############################################################
+# Library: liburing
+###############################################################
+
+liburingopt = get_option('liburing')
+liburing = dependency('liburing', required: liburingopt)
+if liburing.found()
+ cdata.set('USE_LIBURING', 1)
+endif
+
+
+
###############################################################
# Library: libxml
###############################################################
@@ -3168,6 +3180,7 @@ backend_both_deps += [
icu_i18n,
ldap,
libintl,
+ liburing,
libxml,
lz4,
pam,
@@ -3823,6 +3836,7 @@ if meson.version().version_compare('>=0.57')
'icu': icu,
'ldap': ldap,
'libcurl': libcurl,
+ 'liburing': liburing,
'libxml': libxml,
'libxslt': libxslt,
'llvm': llvm,
diff --git a/meson_options.txt b/meson_options.txt
index 702c4517145..dd7126da3a7 100644
--- a/meson_options.txt
+++ b/meson_options.txt
@@ -106,6 +106,9 @@ option('libcurl', type : 'feature', value: 'auto',
option('libedit_preferred', type: 'boolean', value: false,
description: 'Prefer BSD Libedit over GNU Readline')
+option('liburing', type : 'feature', value: 'auto',
+ description: 'io_uring support, for asynchronous I/O')
+
option('libxml', type: 'feature', value: 'auto',
description: 'XML support')
diff --git a/configure.ac b/configure.ac
index b6d02f5ecc7..3e2d0dc59c0 100644
--- a/configure.ac
+++ b/configure.ac
@@ -975,6 +975,14 @@ AC_SUBST(with_readline)
PGAC_ARG_BOOL(with, libedit-preferred, no,
[prefer BSD Libedit over GNU Readline])
+#
+# liburing
+#
+AC_MSG_CHECKING([whether to build with liburing support])
+PGAC_ARG_BOOL(with, liburing, no, [io_uring support, for asynchronous I/O],
+ [AC_DEFINE([USE_LIBURING], 1, [Define to build with io_uring support. (--with-liburing)])])
+AC_MSG_RESULT([$with_liburing])
+AC_SUBST(with_liburing)
#
# UUID library
@@ -1463,6 +1471,9 @@ elif test "$with_uuid" = ossp ; then
fi
AC_SUBST(UUID_LIBS)
+if test "$with_liburing" = yes; then
+ PKG_CHECK_MODULES(LIBURING, liburing)
+fi
##
## Header files
diff --git a/src/makefiles/meson.build b/src/makefiles/meson.build
index 60e13d50235..46d8da070e8 100644
--- a/src/makefiles/meson.build
+++ b/src/makefiles/meson.build
@@ -199,6 +199,8 @@ pgxs_empty = [
'PTHREAD_CFLAGS', 'PTHREAD_LIBS',
'ICU_LIBS',
+
+ 'LIBURING_CFLAGS', 'LIBURING_LIBS',
]
if host_system == 'windows' and cc.get_argument_syntax() != 'msvc'
@@ -230,6 +232,7 @@ pgxs_deps = {
'icu': icu,
'ldap': ldap,
'libcurl': libcurl,
+ 'liburing': liburing,
'libxml': libxml,
'libxslt': libxslt,
'llvm': llvm,
diff --git a/src/include/pg_config.h.in b/src/include/pg_config.h.in
index db6454090d2..f2422241133 100644
--- a/src/include/pg_config.h.in
+++ b/src/include/pg_config.h.in
@@ -672,6 +672,9 @@
/* Define to 1 to build with libcurl support. (--with-libcurl) */
#undef USE_LIBCURL
+/* Define to build with io_uring support. (--with-liburing) */
+#undef USE_LIBURING
+
/* Define to 1 to build with XML support. (--with-libxml) */
#undef USE_LIBXML
diff --git a/src/backend/Makefile b/src/backend/Makefile
index 42d4a28e5aa..7344c8c7f5c 100644
--- a/src/backend/Makefile
+++ b/src/backend/Makefile
@@ -43,9 +43,10 @@ OBJS = \
$(top_builddir)/src/common/libpgcommon_srv.a \
$(top_builddir)/src/port/libpgport_srv.a
-# We put libpgport and libpgcommon into OBJS, so remove it from LIBS; also add
-# libldap and ICU
-LIBS := $(filter-out -lpgport -lpgcommon, $(LIBS)) $(LDAP_LIBS_BE) $(ICU_LIBS)
+# We put libpgport and libpgcommon into OBJS, so remove it from LIBS.
+LIBS := $(filter-out -lpgport -lpgcommon, $(LIBS))
+# The backend conditionally needs libraries that most executables don't need.
+LIBS += $(LDAP_LIBS_BE) $(ICU_LIBS) $(LIBURING_LIBS)
# The backend doesn't need everything that's in LIBS, however
LIBS := $(filter-out -lreadline -ledit -ltermcap -lncurses -lcurses, $(LIBS))
diff --git a/doc/src/sgml/installation.sgml b/doc/src/sgml/installation.sgml
index e076cefa3b9..cc28f041330 100644
--- a/doc/src/sgml/installation.sgml
+++ b/doc/src/sgml/installation.sgml
@@ -1156,6 +1156,24 @@ build-postgresql:
</listitem>
</varlistentry>
+ <varlistentry id="configure-option-with-liburing">
+ <term><option>--with-liburing</option></term>
+ <listitem>
+ <para>
+ Build with liburing, enabling io_uring support for asynchronous I/O.
+ </para>
+ <para>
+ To detect the required compiler and linker options, PostgreSQL will
+ query <command>pkg-config</command>.
+ </para>
+ <para>
+ To use a liburing installation that is in an unusual location, you
+ can set <command>pkg-config</command>-related environment
+ variables (see its documentation).
+ </para>
+ </listitem>
+ </varlistentry>
+
<varlistentry id="configure-option-with-libxml">
<term><option>--with-libxml</option></term>
<listitem>
@@ -2611,6 +2629,22 @@ ninja install
</listitem>
</varlistentry>
+ <varlistentry id="configure-with-liburing-meson">
+ <term><option>-Dliburing={ auto | enabled | disabled }</option></term>
+ <listitem>
+ <para>
+ Build with liburing, enabling io_uring support for asynchronous I/O.
+ Defaults to auto.
+ </para>
+
+ <para>
+ To use a liburing installation that is in an unusual location, you
+ can set <command>pkg-config</command>-related environment
+ variables (see its documentation).
+ </para>
+ </listitem>
+ </varlistentry>
+
<varlistentry id="configure-with-libxml-meson">
<term><option>-Dlibxml={ auto | enabled | disabled }</option></term>
<listitem>
diff --git a/configure b/configure
index 93fddd69981..8ffcdfc2e1c 100755
--- a/configure
+++ b/configure
@@ -651,6 +651,8 @@ LIBOBJS
OPENSSL
ZSTD
LZ4
+LIBURING_LIBS
+LIBURING_CFLAGS
UUID_LIBS
LDAP_LIBS_BE
LDAP_LIBS_FE
@@ -712,6 +714,7 @@ LIBCURL_LIBS
LIBCURL_CFLAGS
with_libcurl
with_uuid
+with_liburing
with_readline
with_systemd
with_selinux
@@ -865,6 +868,7 @@ with_selinux
with_systemd
with_readline
with_libedit_preferred
+with_liburing
with_uuid
with_ossp_uuid
with_libcurl
@@ -911,6 +915,8 @@ LDFLAGS_EX
LDFLAGS_SL
PERL
PYTHON
+LIBURING_CFLAGS
+LIBURING_LIBS
MSGFMT
TCLSH'
@@ -1578,6 +1584,7 @@ Optional Packages:
--without-readline do not use GNU Readline nor BSD Libedit for editing
--with-libedit-preferred
prefer BSD Libedit over GNU Readline
+ --with-liburing io_uring support, for asynchronous I/O
--with-uuid=LIB build contrib/uuid-ossp using LIB (bsd,e2fs,ossp)
--with-ossp-uuid obsolete spelling of --with-uuid=ossp
--with-libcurl build with libcurl support
@@ -1629,6 +1636,10 @@ Some influential environment variables:
LDFLAGS_SL extra linker flags for linking shared libraries only
PERL Perl program
PYTHON Python program
+ LIBURING_CFLAGS
+ C compiler flags for LIBURING, overriding pkg-config
+ LIBURING_LIBS
+ linker flags for LIBURING, overriding pkg-config
MSGFMT msgfmt program for NLS
TCLSH Tcl interpreter program (tclsh)
@@ -8692,6 +8703,40 @@ fi
+#
+# liburing
+#
+{ $as_echo "$as_me:${as_lineno-$LINENO}: checking whether to build with liburing support" >&5
+$as_echo_n "checking whether to build with liburing support... " >&6; }
+
+
+
+# Check whether --with-liburing was given.
+if test "${with_liburing+set}" = set; then :
+ withval=$with_liburing;
+ case $withval in
+ yes)
+
+$as_echo "#define USE_LIBURING 1" >>confdefs.h
+
+ ;;
+ no)
+ :
+ ;;
+ *)
+ as_fn_error $? "no argument expected for --with-liburing option" "$LINENO" 5
+ ;;
+ esac
+
+else
+ with_liburing=no
+
+fi
+
+
+{ $as_echo "$as_me:${as_lineno-$LINENO}: result: $with_liburing" >&5
+$as_echo "$with_liburing" >&6; }
+
#
# UUID library
@@ -13444,6 +13489,99 @@ fi
fi
+if test "$with_liburing" = yes; then
+
+pkg_failed=no
+{ $as_echo "$as_me:${as_lineno-$LINENO}: checking for liburing" >&5
+$as_echo_n "checking for liburing... " >&6; }
+
+if test -n "$LIBURING_CFLAGS"; then
+ pkg_cv_LIBURING_CFLAGS="$LIBURING_CFLAGS"
+ elif test -n "$PKG_CONFIG"; then
+ if test -n "$PKG_CONFIG" && \
+ { { $as_echo "$as_me:${as_lineno-$LINENO}: \$PKG_CONFIG --exists --print-errors \"liburing\""; } >&5
+ ($PKG_CONFIG --exists --print-errors "liburing") 2>&5
+ ac_status=$?
+ $as_echo "$as_me:${as_lineno-$LINENO}: \$? = $ac_status" >&5
+ test $ac_status = 0; }; then
+ pkg_cv_LIBURING_CFLAGS=`$PKG_CONFIG --cflags "liburing" 2>/dev/null`
+ test "x$?" != "x0" && pkg_failed=yes
+else
+ pkg_failed=yes
+fi
+ else
+ pkg_failed=untried
+fi
+if test -n "$LIBURING_LIBS"; then
+ pkg_cv_LIBURING_LIBS="$LIBURING_LIBS"
+ elif test -n "$PKG_CONFIG"; then
+ if test -n "$PKG_CONFIG" && \
+ { { $as_echo "$as_me:${as_lineno-$LINENO}: \$PKG_CONFIG --exists --print-errors \"liburing\""; } >&5
+ ($PKG_CONFIG --exists --print-errors "liburing") 2>&5
+ ac_status=$?
+ $as_echo "$as_me:${as_lineno-$LINENO}: \$? = $ac_status" >&5
+ test $ac_status = 0; }; then
+ pkg_cv_LIBURING_LIBS=`$PKG_CONFIG --libs "liburing" 2>/dev/null`
+ test "x$?" != "x0" && pkg_failed=yes
+else
+ pkg_failed=yes
+fi
+ else
+ pkg_failed=untried
+fi
+
+
+
+if test $pkg_failed = yes; then
+ { $as_echo "$as_me:${as_lineno-$LINENO}: result: no" >&5
+$as_echo "no" >&6; }
+
+if $PKG_CONFIG --atleast-pkgconfig-version 0.20; then
+ _pkg_short_errors_supported=yes
+else
+ _pkg_short_errors_supported=no
+fi
+ if test $_pkg_short_errors_supported = yes; then
+ LIBURING_PKG_ERRORS=`$PKG_CONFIG --short-errors --print-errors --cflags --libs "liburing" 2>&1`
+ else
+ LIBURING_PKG_ERRORS=`$PKG_CONFIG --print-errors --cflags --libs "liburing" 2>&1`
+ fi
+ # Put the nasty error message in config.log where it belongs
+ echo "$LIBURING_PKG_ERRORS" >&5
+
+ as_fn_error $? "Package requirements (liburing) were not met:
+
+$LIBURING_PKG_ERRORS
+
+Consider adjusting the PKG_CONFIG_PATH environment variable if you
+installed software in a non-standard prefix.
+
+Alternatively, you may set the environment variables LIBURING_CFLAGS
+and LIBURING_LIBS to avoid the need to call pkg-config.
+See the pkg-config man page for more details." "$LINENO" 5
+elif test $pkg_failed = untried; then
+ { $as_echo "$as_me:${as_lineno-$LINENO}: result: no" >&5
+$as_echo "no" >&6; }
+ { { $as_echo "$as_me:${as_lineno-$LINENO}: error: in \`$ac_pwd':" >&5
+$as_echo "$as_me: error: in \`$ac_pwd':" >&2;}
+as_fn_error $? "The pkg-config script could not be found or is too old. Make sure it
+is in your PATH or set the PKG_CONFIG environment variable to the full
+path to pkg-config.
+
+Alternatively, you may set the environment variables LIBURING_CFLAGS
+and LIBURING_LIBS to avoid the need to call pkg-config.
+See the pkg-config man page for more details.
+
+To get pkg-config, see <http://pkg-config.freedesktop.org/>.
+See \`config.log' for more details" "$LINENO" 5; }
+else
+ LIBURING_CFLAGS=$pkg_cv_LIBURING_CFLAGS
+ LIBURING_LIBS=$pkg_cv_LIBURING_LIBS
+ { $as_echo "$as_me:${as_lineno-$LINENO}: result: yes" >&5
+$as_echo "yes" >&6; }
+
+fi
+fi
##
## Header files
diff --git a/.cirrus.tasks.yml b/.cirrus.tasks.yml
index 5849cbb839a..10d277f5659 100644
--- a/.cirrus.tasks.yml
+++ b/.cirrus.tasks.yml
@@ -471,6 +471,7 @@ task:
--enable-cassert --enable-injection-points --enable-debug \
--enable-tap-tests --enable-nls \
--with-segsize-blocks=6 \
+ --with-liburing \
\
${LINUX_CONFIGURE_FEATURES} \
\
diff --git a/src/Makefile.global.in b/src/Makefile.global.in
index 3b620bac5ac..d45f5fe4ed7 100644
--- a/src/Makefile.global.in
+++ b/src/Makefile.global.in
@@ -191,6 +191,7 @@ with_gssapi = @with_gssapi@
with_krb_srvnam = @with_krb_srvnam@
with_ldap = @with_ldap@
with_libcurl = @with_libcurl@
+with_liburing = @with_liburing@
with_libxml = @with_libxml@
with_libxslt = @with_libxslt@
with_llvm = @with_llvm@
@@ -217,6 +218,9 @@ krb_srvtab = @krb_srvtab@
ICU_CFLAGS = @ICU_CFLAGS@
ICU_LIBS = @ICU_LIBS@
+LIBURING_CFLAGS = @LIBURING_CFLAGS@
+LIBURING_LIBS = @LIBURING_LIBS@
+
TCLSH = @TCLSH@
TCL_LIBS = @TCL_LIBS@
TCL_LIB_SPEC = @TCL_LIB_SPEC@
--
2.48.1.76.g4e746b1a31.dirty
[text/x-diff] v2.6-0006-aio-Add-io_method-io_uring.patch (19.0K, ../../ge6nsuddurhpmll3xj22vucvqwp4agqz6ndtcf2mhyeydzarst@l75dman5x53p/7-v2.6-0006-aio-Add-io_method-io_uring.patch)
download | inline diff:
From 039163efaea51533e573b3ff26f88e501693135d Mon Sep 17 00:00:00 2001
From: Andres Freund <andres@anarazel.de>
Date: Tue, 11 Feb 2025 14:29:37 -0500
Subject: [PATCH v2.6 06/34] aio: Add io_method=io_uring
io_uring, can be considerably faster than io_method=worker. It is, however,
linux specific and requires an additional compile-time dependency (liburing).
Reviewed-by: Jakub Wartak <jakub.wartak@enterprisedb.com>
Discussion: https://postgr.es/m/uvrtrknj4kdytuboidbhwclo4gxhswwcpgadptsjvjqcluzmah%40brqs62irg4dt
Discussion: https://postgr.es/m/20210223100344.llw5an2aklengrmn@alap3.anarazel.de
Discussion: https://postgr.es/m/stj36ea6yyhoxtqkhpieia2z4krnam7qyetc57rfezgk4zgapf@gcnactj4z56m
---
src/include/storage/aio.h | 3 +
src/include/storage/aio_internal.h | 3 +
src/include/storage/lwlock.h | 1 +
src/backend/storage/aio/Makefile | 1 +
src/backend/storage/aio/aio.c | 6 +
src/backend/storage/aio/meson.build | 1 +
src/backend/storage/aio/method_io_uring.c | 410 ++++++++++++++++++
src/backend/storage/lmgr/lwlock.c | 1 +
.../utils/activity/wait_event_names.txt | 2 +
src/backend/utils/misc/postgresql.conf.sample | 3 +-
doc/src/sgml/config.sgml | 6 +
.cirrus.tasks.yml | 3 +
src/tools/pgindent/typedefs.list | 1 +
13 files changed, 440 insertions(+), 1 deletion(-)
create mode 100644 src/backend/storage/aio/method_io_uring.c
diff --git a/src/include/storage/aio.h b/src/include/storage/aio.h
index 7a422a4d131..023616f6805 100644
--- a/src/include/storage/aio.h
+++ b/src/include/storage/aio.h
@@ -24,6 +24,9 @@ typedef enum IoMethod
{
IOMETHOD_SYNC = 0,
IOMETHOD_WORKER,
+#ifdef USE_LIBURING
+ IOMETHOD_IO_URING,
+#endif
} IoMethod;
/* We'll default to worker based execution. */
diff --git a/src/include/storage/aio_internal.h b/src/include/storage/aio_internal.h
index a626581dd04..2412c5e7ecb 100644
--- a/src/include/storage/aio_internal.h
+++ b/src/include/storage/aio_internal.h
@@ -354,6 +354,9 @@ extern PgAioHandle *pgaio_inj_io_get(void);
/* Declarations for the tables of function pointers exposed by each IO method. */
extern PGDLLIMPORT const IoMethodOps pgaio_sync_ops;
extern PGDLLIMPORT const IoMethodOps pgaio_worker_ops;
+#ifdef USE_LIBURING
+extern PGDLLIMPORT const IoMethodOps pgaio_uring_ops;
+#endif
extern PGDLLIMPORT const IoMethodOps *pgaio_method_ops;
extern PGDLLIMPORT PgAioCtl *pgaio_ctl;
diff --git a/src/include/storage/lwlock.h b/src/include/storage/lwlock.h
index ffa03189e2d..4df1d25c045 100644
--- a/src/include/storage/lwlock.h
+++ b/src/include/storage/lwlock.h
@@ -218,6 +218,7 @@ typedef enum BuiltinTrancheIds
LWTRANCHE_SUBTRANS_SLRU,
LWTRANCHE_XACT_SLRU,
LWTRANCHE_PARALLEL_VACUUM_DSA,
+ LWTRANCHE_AIO_URING_COMPLETION,
LWTRANCHE_FIRST_USER_DEFINED,
} BuiltinTrancheIds;
diff --git a/src/backend/storage/aio/Makefile b/src/backend/storage/aio/Makefile
index f51c34a37f8..c06c50771e0 100644
--- a/src/backend/storage/aio/Makefile
+++ b/src/backend/storage/aio/Makefile
@@ -14,6 +14,7 @@ OBJS = \
aio_init.o \
aio_io.o \
aio_target.o \
+ method_io_uring.o \
method_sync.o \
method_worker.o \
read_stream.o
diff --git a/src/backend/storage/aio/aio.c b/src/backend/storage/aio/aio.c
index 1e756d37b93..a8227efa0b5 100644
--- a/src/backend/storage/aio/aio.c
+++ b/src/backend/storage/aio/aio.c
@@ -65,6 +65,9 @@ static void pgaio_io_wait(PgAioHandle *ioh, uint64 ref_generation);
const struct config_enum_entry io_method_options[] = {
{"sync", IOMETHOD_SYNC, false},
{"worker", IOMETHOD_WORKER, false},
+#ifdef USE_LIBURING
+ {"io_uring", IOMETHOD_IO_URING, false},
+#endif
{NULL, 0, false}
};
@@ -82,6 +85,9 @@ PgAioBackend *pgaio_my_backend;
static const IoMethodOps *const pgaio_method_ops_table[] = {
[IOMETHOD_SYNC] = &pgaio_sync_ops,
[IOMETHOD_WORKER] = &pgaio_worker_ops,
+#ifdef USE_LIBURING
+ [IOMETHOD_IO_URING] = &pgaio_uring_ops,
+#endif
};
/* callbacks for the configured io_method, set by assign_io_method */
diff --git a/src/backend/storage/aio/meson.build b/src/backend/storage/aio/meson.build
index 74f94c6e40b..2f0f03d8071 100644
--- a/src/backend/storage/aio/meson.build
+++ b/src/backend/storage/aio/meson.build
@@ -6,6 +6,7 @@ backend_sources += files(
'aio_init.c',
'aio_io.c',
'aio_target.c',
+ 'method_io_uring.c',
'method_sync.c',
'method_worker.c',
'read_stream.c',
diff --git a/src/backend/storage/aio/method_io_uring.c b/src/backend/storage/aio/method_io_uring.c
new file mode 100644
index 00000000000..0d7ad9124dd
--- /dev/null
+++ b/src/backend/storage/aio/method_io_uring.c
@@ -0,0 +1,410 @@
+/*-------------------------------------------------------------------------
+ *
+ * method_io_uring.c
+ * AIO - perform AIO using Linux' io_uring
+ *
+ * For now we create one io_uring instance for each backend. These io_uring
+ * instances have to be created in postmaster, during startup, to allow other
+ * backends to process IO completions, if the issuing backend is currently
+ * busy doing other things. Other backends may not use another backend's
+ * io_uring instance to submit IO, that'd require additional locking that
+ * would likely be harmful for performance.
+ *
+ * We likely will want to introduce a backend-local io_uring instance in the
+ * future, e.g. for FE/BE network IO.
+ *
+ * Portions Copyright (c) 1996-2025, PostgreSQL Global Development Group
+ * Portions Copyright (c) 1994, Regents of the University of California
+ *
+ * IDENTIFICATION
+ * src/backend/storage/aio/method_io_uring.c
+ *
+ *-------------------------------------------------------------------------
+ */
+
+#include "postgres.h"
+
+#ifdef USE_LIBURING
+
+#include <liburing.h>
+
+#include "pgstat.h"
+#include "port/pg_iovec.h"
+#include "storage/aio.h"
+#include "storage/aio_internal.h"
+#include "storage/fd.h"
+#include "storage/proc.h"
+#include "storage/shmem.h"
+
+
+/* number of completions processed at once */
+#define PGAIO_MAX_LOCAL_COMPLETED_IO 32
+
+
+/* Entry points for IoMethodOps. */
+static size_t pgaio_uring_shmem_size(void);
+static void pgaio_uring_shmem_init(bool first_time);
+static void pgaio_uring_init_backend(void);
+static int pgaio_uring_submit(uint16 num_staged_ios, PgAioHandle **staged_ios);
+static void pgaio_uring_wait_one(PgAioHandle *ioh, uint64 ref_generation);
+
+/* helper functions */
+static void pgaio_uring_sq_from_io(PgAioHandle *ioh, struct io_uring_sqe *sqe);
+
+
+const IoMethodOps pgaio_uring_ops = {
+ .shmem_size = pgaio_uring_shmem_size,
+ .shmem_init = pgaio_uring_shmem_init,
+ .init_backend = pgaio_uring_init_backend,
+
+ .submit = pgaio_uring_submit,
+ .wait_one = pgaio_uring_wait_one,
+};
+
+/*
+ * Per-backend state when using io_method=io_uring
+ *
+ * Align the whole struct to a cacheline boundary, to prevent false sharing
+ * between completion_lock and prior backend's io_uring_ring.
+ */
+typedef struct pg_attribute_aligned (PG_CACHE_LINE_SIZE)
+PgAioUringContext
+{
+ /*
+ * Multiple backends can process completions for this backend's io_uring
+ * instance (e.g. when the backend issuing IO is busy doing something
+ * else). To make that safe we have to ensure that only a single backend
+ * gets io completions from the io_uring instance at a time.
+ */
+ LWLock completion_lock;
+
+ struct io_uring io_uring_ring;
+} PgAioUringContext;
+
+/* PgAioUringContexts for all backends */
+static PgAioUringContext *pgaio_uring_contexts;
+
+/* the current backend's context */
+static PgAioUringContext *pgaio_my_uring_context;
+
+
+static uint32
+pgaio_uring_procs(void)
+{
+ /*
+ * We can subtract MAX_IO_WORKERS here as io workers are never used at the
+ * same time as io_method=io_uring.
+ */
+ return MaxBackends + NUM_AUXILIARY_PROCS - MAX_IO_WORKERS;
+}
+
+static Size
+pgaio_uring_context_shmem_size(void)
+{
+ return mul_size(pgaio_uring_procs(), sizeof(PgAioUringContext));
+}
+
+static size_t
+pgaio_uring_shmem_size(void)
+{
+ return pgaio_uring_context_shmem_size();
+}
+
+static void
+pgaio_uring_shmem_init(bool first_time)
+{
+ uint32 TotalProcs = MaxBackends + NUM_AUXILIARY_PROCS - MAX_IO_WORKERS;
+ bool found;
+
+ pgaio_uring_contexts = (PgAioUringContext *)
+ ShmemInitStruct("AioUring", pgaio_uring_shmem_size(), &found);
+
+ if (found)
+ return;
+
+ for (int contextno = 0; contextno < TotalProcs; contextno++)
+ {
+ PgAioUringContext *context = &pgaio_uring_contexts[contextno];
+ int ret;
+
+ /*
+ * XXX: Probably worth sharing the WQ between the different rings,
+ * when supported by the kernel. Could also cause additional
+ * contention, I guess?
+ */
+#if 0
+ if (!AcquireExternalFD())
+ elog(ERROR, "No external FD available");
+#endif
+ ret = io_uring_queue_init(io_max_concurrency, &context->io_uring_ring, 0);
+ if (ret < 0)
+ elog(ERROR, "io_uring_queue_init failed: %s", strerror(-ret));
+
+ LWLockInitialize(&context->completion_lock, LWTRANCHE_AIO_URING_COMPLETION);
+ }
+}
+
+static void
+pgaio_uring_init_backend(void)
+{
+ Assert(MyProcNumber < pgaio_uring_procs());
+
+ pgaio_my_uring_context = &pgaio_uring_contexts[MyProcNumber];
+}
+
+static int
+pgaio_uring_submit(uint16 num_staged_ios, PgAioHandle **staged_ios)
+{
+ struct io_uring *uring_instance = &pgaio_my_uring_context->io_uring_ring;
+ int in_flight_before = dclist_count(&pgaio_my_backend->in_flight_ios);
+
+ Assert(num_staged_ios <= PGAIO_SUBMIT_BATCH_SIZE);
+
+ for (int i = 0; i < num_staged_ios; i++)
+ {
+ PgAioHandle *ioh = staged_ios[i];
+ struct io_uring_sqe *sqe;
+
+ sqe = io_uring_get_sqe(uring_instance);
+
+ if (!sqe)
+ elog(ERROR, "io_uring submission queue is unexpectedly full");
+
+ pgaio_io_prepare_submit(ioh);
+ pgaio_uring_sq_from_io(ioh, sqe);
+
+ /*
+ * io_uring executes IO in process context if possible. That's
+ * generally good, as it reduces context switching. When performing a
+ * lot of buffered IO that means that copying between page cache and
+ * userspace memory happens in the foreground, as it can't be
+ * offloaded to DMA hardware as is possible when using direct IO. When
+ * executing a lot of buffered IO this causes io_uring to be slower
+ * than worker mode, as worker mode parallelizes the copying. io_uring
+ * can be told to offload work to worker threads instead.
+ *
+ * If an IO is buffered IO and we already have IOs in flight or
+ * multiple IOs are being submitted, we thus tell io_uring to execute
+ * the IO in the background. We don't do so for the first few IOs
+ * being submitted as executing in this process' context has lower
+ * latency.
+ */
+ if (in_flight_before > 4 && (ioh->flags & PGAIO_HF_BUFFERED))
+ io_uring_sqe_set_flags(sqe, IOSQE_ASYNC);
+
+ in_flight_before++;
+ }
+
+ while (true)
+ {
+ int ret;
+
+ pgstat_report_wait_start(WAIT_EVENT_AIO_IO_URING_SUBMIT);
+ ret = io_uring_submit(uring_instance);
+ pgstat_report_wait_end();
+
+ if (ret == -EINTR)
+ {
+ pgaio_debug(DEBUG3,
+ "aio method uring: submit EINTR, nios: %d",
+ num_staged_ios);
+ }
+ else if (ret < 0)
+ elog(PANIC, "failed: %d/%s",
+ ret, strerror(-ret));
+ else if (ret != num_staged_ios)
+ {
+ /* likely unreachable, but if it is, we would need to re-submit */
+ elog(PANIC, "submitted only %d of %d",
+ ret, num_staged_ios);
+ }
+ else
+ {
+ pgaio_debug(DEBUG4,
+ "aio method uring: submitted %d IOs",
+ num_staged_ios);
+ break;
+ }
+ }
+
+ return num_staged_ios;
+}
+
+static void
+pgaio_uring_drain_locked(PgAioUringContext *context)
+{
+ int ready;
+ int orig_ready;
+
+ Assert(LWLockHeldByMeInMode(&context->completion_lock, LW_EXCLUSIVE));
+
+ /*
+ * Don't drain more events than available right now. Otherwise it's
+ * plausible that one backend could get stuck, for a while, receiving CQEs
+ * without actually processing them.
+ */
+ orig_ready = ready = io_uring_cq_ready(&context->io_uring_ring);
+
+ while (ready > 0)
+ {
+ struct io_uring_cqe *cqes[PGAIO_MAX_LOCAL_COMPLETED_IO];
+ uint32 ncqes;
+
+ START_CRIT_SECTION();
+ ncqes =
+ io_uring_peek_batch_cqe(&context->io_uring_ring,
+ cqes,
+ Min(PGAIO_MAX_LOCAL_COMPLETED_IO, ready));
+ Assert(ncqes <= ready);
+
+ ready -= ncqes;
+
+ for (int i = 0; i < ncqes; i++)
+ {
+ struct io_uring_cqe *cqe = cqes[i];
+ PgAioHandle *ioh;
+
+ ioh = io_uring_cqe_get_data(cqe);
+ io_uring_cqe_seen(&context->io_uring_ring, cqe);
+
+ pgaio_io_process_completion(ioh, cqe->res);
+ }
+
+ END_CRIT_SECTION();
+
+ pgaio_debug(DEBUG3,
+ "drained %d/%d, now expecting %d",
+ ncqes, orig_ready, io_uring_cq_ready(&context->io_uring_ring));
+ }
+}
+
+static void
+pgaio_uring_wait_one(PgAioHandle *ioh, uint64 ref_generation)
+{
+ PgAioHandleState state;
+ ProcNumber owner_procno = ioh->owner_procno;
+ PgAioUringContext *owner_context = &pgaio_uring_contexts[owner_procno];
+ bool expect_cqe;
+ int waited = 0;
+
+ /*
+ * XXX: It would be nice to have a smarter locking scheme, nearly all the
+ * time the backend owning the ring will consume the completions, making
+ * the locking unnecessarily expensive.
+ */
+ LWLockAcquire(&owner_context->completion_lock, LW_EXCLUSIVE);
+
+ while (true)
+ {
+ pgaio_debug_io(DEBUG3, ioh,
+ "wait_one io_gen: %llu, ref_gen: %llu, cycle %d",
+ (long long unsigned) ref_generation,
+ (long long unsigned) ioh->generation,
+ waited);
+
+ if (pgaio_io_was_recycled(ioh, ref_generation, &state) ||
+ state != PGAIO_HS_SUBMITTED)
+ {
+ /* the IO was completed by another backend */
+ break;
+ }
+ else if (io_uring_cq_ready(&owner_context->io_uring_ring))
+ {
+ /* no need to wait in the kernel, io_uring has a completion */
+ expect_cqe = true;
+ }
+ else
+ {
+ int ret;
+ struct io_uring_cqe *cqes;
+
+ /* need to wait in the kernel */
+ pgstat_report_wait_start(WAIT_EVENT_AIO_IO_URING_COMPLETION);
+ ret = io_uring_wait_cqes(&owner_context->io_uring_ring, &cqes, 1, NULL, NULL);
+ pgstat_report_wait_end();
+
+ if (ret == -EINTR)
+ {
+ continue;
+ }
+ else if (ret != 0)
+ {
+ elog(PANIC, "unexpected: %d/%s: %m", ret, strerror(-ret));
+ }
+ else
+ {
+ Assert(cqes != NULL);
+ expect_cqe = true;
+ waited++;
+ }
+ }
+
+ if (expect_cqe)
+ {
+ pgaio_uring_drain_locked(owner_context);
+ }
+ }
+
+ LWLockRelease(&owner_context->completion_lock);
+
+ pgaio_debug(DEBUG3,
+ "wait_one with %d sleeps",
+ waited);
+}
+
+static void
+pgaio_uring_sq_from_io(PgAioHandle *ioh, struct io_uring_sqe *sqe)
+{
+ struct iovec *iov;
+
+ switch (ioh->op)
+ {
+ case PGAIO_OP_READV:
+ iov = &pgaio_ctl->iovecs[ioh->iovec_off];
+ if (ioh->op_data.read.iov_length == 1)
+ {
+ io_uring_prep_read(sqe,
+ ioh->op_data.read.fd,
+ iov->iov_base,
+ iov->iov_len,
+ ioh->op_data.read.offset);
+ }
+ else
+ {
+ io_uring_prep_readv(sqe,
+ ioh->op_data.read.fd,
+ iov,
+ ioh->op_data.read.iov_length,
+ ioh->op_data.read.offset);
+
+ }
+ break;
+
+ case PGAIO_OP_WRITEV:
+ iov = &pgaio_ctl->iovecs[ioh->iovec_off];
+ if (ioh->op_data.write.iov_length == 1)
+ {
+ io_uring_prep_write(sqe,
+ ioh->op_data.write.fd,
+ iov->iov_base,
+ iov->iov_len,
+ ioh->op_data.write.offset);
+ }
+ else
+ {
+ io_uring_prep_writev(sqe,
+ ioh->op_data.write.fd,
+ iov,
+ ioh->op_data.write.iov_length,
+ ioh->op_data.write.offset);
+ }
+ break;
+
+ case PGAIO_OP_INVALID:
+ elog(ERROR, "trying to prepare invalid IO operation for execution");
+ }
+
+ io_uring_sqe_set_data(sqe, ioh);
+}
+
+#endif /* USE_LIBURING */
diff --git a/src/backend/storage/lmgr/lwlock.c b/src/backend/storage/lmgr/lwlock.c
index 5702c35bb91..3df29658f18 100644
--- a/src/backend/storage/lmgr/lwlock.c
+++ b/src/backend/storage/lmgr/lwlock.c
@@ -177,6 +177,7 @@ static const char *const BuiltinTrancheNames[] = {
[LWTRANCHE_SUBTRANS_SLRU] = "SubtransSLRU",
[LWTRANCHE_XACT_SLRU] = "XactSLRU",
[LWTRANCHE_PARALLEL_VACUUM_DSA] = "ParallelVacuumDSA",
+ [LWTRANCHE_AIO_URING_COMPLETION] = "AioUringCompletion",
};
StaticAssertDecl(lengthof(BuiltinTrancheNames) ==
diff --git a/src/backend/utils/activity/wait_event_names.txt b/src/backend/utils/activity/wait_event_names.txt
index 9fa12a555e8..44eef41f841 100644
--- a/src/backend/utils/activity/wait_event_names.txt
+++ b/src/backend/utils/activity/wait_event_names.txt
@@ -192,6 +192,8 @@ ABI_compatibility:
Section: ClassName - WaitEventIO
+AIO_IO_URING_SUBMIT "Waiting for IO submission via io_uring."
+AIO_IO_URING_COMPLETION "Waiting for IO completion via io_uring."
AIO_IO_COMPLETION "Waiting for IO completion."
BASEBACKUP_READ "Waiting for base backup to read from a file."
BASEBACKUP_SYNC "Waiting for data written by a base backup to reach durable storage."
diff --git a/src/backend/utils/misc/postgresql.conf.sample b/src/backend/utils/misc/postgresql.conf.sample
index 1058726285f..8738ad51bf1 100644
--- a/src/backend/utils/misc/postgresql.conf.sample
+++ b/src/backend/utils/misc/postgresql.conf.sample
@@ -202,7 +202,8 @@
#maintenance_io_concurrency = 10 # 1-1000; 0 disables prefetching
#io_combine_limit = 128kB # usually 1-32 blocks (depends on OS)
-#io_method = worker # worker, sync (change requires restart)
+#io_method = worker # worker, io_uring, sync
+ # (change requires restart)
#io_max_concurrency = -1 # Max number of IOs that one process
# can execute simultaneously
# -1 sets based on shared_buffers
diff --git a/doc/src/sgml/config.sgml b/doc/src/sgml/config.sgml
index de1b918b06a..47dd93e4227 100644
--- a/doc/src/sgml/config.sgml
+++ b/doc/src/sgml/config.sgml
@@ -2687,6 +2687,12 @@ include_dir 'conf.d'
<literal>worker</literal> (execute asynchronous I/O using worker processes)
</para>
</listitem>
+ <listitem>
+ <para>
+ <literal>io_uring</literal> (execute asynchronous I/O using
+ io_uring, if available)
+ </para>
+ </listitem>
<listitem>
<para>
<literal>sync</literal> (execute asynchronous I/O synchronously)
diff --git a/.cirrus.tasks.yml b/.cirrus.tasks.yml
index 10d277f5659..86a1fa9bbdb 100644
--- a/.cirrus.tasks.yml
+++ b/.cirrus.tasks.yml
@@ -493,11 +493,14 @@ task:
# - Uses undefined behaviour and alignment sanitizers, sanitizer failures
# are typically printed in the server log
# - Test both 64bit and 32 bit builds
+ # - uses io_method=io_uring
- name: Linux - Debian Bookworm - Meson
env:
CCACHE_MAXSIZE: "400M" # tests two different builds
SANITIZER_FLAGS: -fsanitize=alignment,undefined
+ PG_TEST_INITDB_EXTRA_OPTS: >-
+ -c io_method=io_uring
configure_script: |
su postgres <<-EOF
diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list
index 098ef0cd7a5..e5a13e6e6dd 100644
--- a/src/tools/pgindent/typedefs.list
+++ b/src/tools/pgindent/typedefs.list
@@ -2147,6 +2147,7 @@ PgAioReturn
PgAioTargetData
PgAioTargetID
PgAioTargetInfo
+PgAioUringContext
PgAioWaitRef
PgArchData
PgBackendGSSStatus
--
2.48.1.76.g4e746b1a31.dirty
[text/x-diff] v2.6-0007-aio-Implement-support-for-reads-in-smgr-md-fd.patch (21.6K, ../../ge6nsuddurhpmll3xj22vucvqwp4agqz6ndtcf2mhyeydzarst@l75dman5x53p/8-v2.6-0007-aio-Implement-support-for-reads-in-smgr-md-fd.patch)
download | inline diff:
From f329d7ef281b59be2a18a4093dc0192bb11bef0d Mon Sep 17 00:00:00 2001
From: Andres Freund <andres@anarazel.de>
Date: Wed, 22 Jan 2025 16:06:51 -0500
Subject: [PATCH v2.6 07/34] aio: Implement support for reads in smgr/md/fd
This implements the following:
1) An smgr AIO target, for AIO on smgr files. This should be usable not just
for md.c but also other SMGR implementation if we ever get them.
2) readv support in fd.c, which requires a small bit of infrastructure work in
fd.c
3) smgr.c and md.c support for readv
There still is nothing performing AIO, but as of this commit it would be
possible.
Discussion: https://postgr.es/m/uvrtrknj4kdytuboidbhwclo4gxhswwcpgadptsjvjqcluzmah%40brqs62irg4dt
Discussion: https://postgr.es/m/20210223100344.llw5an2aklengrmn@alap3.anarazel.de
Discussion: https://postgr.es/m/stj36ea6yyhoxtqkhpieia2z4krnam7qyetc57rfezgk4zgapf@gcnactj4z56m
---
src/include/storage/aio.h | 6 +-
src/include/storage/aio_types.h | 10 +-
src/include/storage/fd.h | 3 +
src/include/storage/md.h | 7 +
src/include/storage/smgr.h | 14 ++
src/backend/storage/aio/aio_callback.c | 5 +-
src/backend/storage/aio/aio_target.c | 2 +
src/backend/storage/file/fd.c | 40 ++++++
src/backend/storage/smgr/md.c | 176 +++++++++++++++++++++++++
src/backend/storage/smgr/smgr.c | 124 +++++++++++++++++
10 files changed, 382 insertions(+), 5 deletions(-)
diff --git a/src/include/storage/aio.h b/src/include/storage/aio.h
index 023616f6805..16476680f12 100644
--- a/src/include/storage/aio.h
+++ b/src/include/storage/aio.h
@@ -108,9 +108,10 @@ typedef enum PgAioTargetID
{
/* intentionally the zero value, to help catch zeroed memory etc */
PGAIO_TID_INVALID = 0,
+ PGAIO_TID_SMGR,
} PgAioTargetID;
-#define PGAIO_TID_COUNT (PGAIO_TID_INVALID + 1)
+#define PGAIO_TID_COUNT (PGAIO_TID_SMGR + 1)
/*
@@ -176,6 +177,9 @@ struct PgAioTargetInfo
typedef enum PgAioHandleCallbackID
{
PGAIO_HCB_INVALID,
+
+ PGAIO_HCB_MD_READV,
+ PGAIO_HCB_MD_WRITEV,
} PgAioHandleCallbackID;
diff --git a/src/include/storage/aio_types.h b/src/include/storage/aio_types.h
index a5cc658efbd..b0b9a2a5c97 100644
--- a/src/include/storage/aio_types.h
+++ b/src/include/storage/aio_types.h
@@ -60,11 +60,15 @@ typedef struct PgAioWaitRef
*/
typedef union PgAioTargetData
{
- /* just as an example placeholder for later */
struct
{
- uint32 queue_id;
- } wal;
+ RelFileLocator rlocator; /* physical relation identifier */
+ BlockNumber blockNum; /* blknum relative to begin of reln */
+ BlockNumber nblocks;
+ ForkNumber forkNum:8; /* don't waste 4 byte for four values */
+ bool is_temp:1; /* proc can be inferred by owning AIO */
+ bool skip_fsync:1;
+ } smgr;
} PgAioTargetData;
diff --git a/src/include/storage/fd.h b/src/include/storage/fd.h
index e3067ab6597..b77d8e5e30e 100644
--- a/src/include/storage/fd.h
+++ b/src/include/storage/fd.h
@@ -101,6 +101,8 @@ extern PGDLLIMPORT int max_safe_fds;
* prototypes for functions in fd.c
*/
+struct PgAioHandle;
+
/* Operations on virtual Files --- equivalent to Unix kernel file ops */
extern File PathNameOpenFile(const char *fileName, int fileFlags);
extern File PathNameOpenFilePerm(const char *fileName, int fileFlags, mode_t fileMode);
@@ -109,6 +111,7 @@ extern void FileClose(File file);
extern int FilePrefetch(File file, off_t offset, off_t amount, uint32 wait_event_info);
extern ssize_t FileReadV(File file, const struct iovec *iov, int iovcnt, off_t offset, uint32 wait_event_info);
extern ssize_t FileWriteV(File file, const struct iovec *iov, int iovcnt, off_t offset, uint32 wait_event_info);
+extern int FileStartReadV(struct PgAioHandle *ioh, File file, int iovcnt, off_t offset, uint32 wait_event_info);
extern int FileSync(File file, uint32 wait_event_info);
extern int FileZero(File file, off_t offset, off_t amount, uint32 wait_event_info);
extern int FileFallocate(File file, off_t offset, off_t amount, uint32 wait_event_info);
diff --git a/src/include/storage/md.h b/src/include/storage/md.h
index 05bf537066e..9d7131eff43 100644
--- a/src/include/storage/md.h
+++ b/src/include/storage/md.h
@@ -14,11 +14,14 @@
#ifndef MD_H
#define MD_H
+#include "storage/aio_types.h"
#include "storage/block.h"
#include "storage/relfilelocator.h"
#include "storage/smgr.h"
#include "storage/sync.h"
+extern const PgAioHandleCallbacks aio_md_readv_cb;
+
/* md storage manager functionality */
extern void mdinit(void);
extern void mdopen(SMgrRelation reln);
@@ -36,6 +39,9 @@ extern uint32 mdmaxcombine(SMgrRelation reln, ForkNumber forknum,
BlockNumber blocknum);
extern void mdreadv(SMgrRelation reln, ForkNumber forknum, BlockNumber blocknum,
void **buffers, BlockNumber nblocks);
+extern void mdstartreadv(PgAioHandle *ioh,
+ SMgrRelation reln, ForkNumber forknum, BlockNumber blocknum,
+ void **buffers, BlockNumber nblocks);
extern void mdwritev(SMgrRelation reln, ForkNumber forknum,
BlockNumber blocknum,
const void **buffers, BlockNumber nblocks, bool skipFsync);
@@ -46,6 +52,7 @@ extern void mdtruncate(SMgrRelation reln, ForkNumber forknum,
BlockNumber old_blocks, BlockNumber nblocks);
extern void mdimmedsync(SMgrRelation reln, ForkNumber forknum);
extern void mdregistersync(SMgrRelation reln, ForkNumber forknum);
+extern int mdfd(SMgrRelation reln, ForkNumber forknum, BlockNumber blocknum, uint32 *off);
extern void ForgetDatabaseSyncRequests(Oid dbid);
extern void DropRelationFiles(RelFileLocator *delrels, int ndelrels, bool isRedo);
diff --git a/src/include/storage/smgr.h b/src/include/storage/smgr.h
index 4016b206ad6..856ebcda350 100644
--- a/src/include/storage/smgr.h
+++ b/src/include/storage/smgr.h
@@ -15,6 +15,7 @@
#define SMGR_H
#include "lib/ilist.h"
+#include "storage/aio_types.h"
#include "storage/block.h"
#include "storage/relfilelocator.h"
@@ -73,6 +74,8 @@ typedef SMgrRelationData *SMgrRelation;
#define SmgrIsTemp(smgr) \
RelFileLocatorBackendIsTemp((smgr)->smgr_rlocator)
+extern const PgAioTargetInfo aio_smgr_target_info;
+
extern void smgrinit(void);
extern SMgrRelation smgropen(RelFileLocator rlocator, ProcNumber backend);
extern bool smgrexists(SMgrRelation reln, ForkNumber forknum);
@@ -97,6 +100,10 @@ extern uint32 smgrmaxcombine(SMgrRelation reln, ForkNumber forknum,
extern void smgrreadv(SMgrRelation reln, ForkNumber forknum,
BlockNumber blocknum,
void **buffers, BlockNumber nblocks);
+extern void smgrstartreadv(PgAioHandle *ioh,
+ SMgrRelation reln, ForkNumber forknum,
+ BlockNumber blocknum,
+ void **buffers, BlockNumber nblocks);
extern void smgrwritev(SMgrRelation reln, ForkNumber forknum,
BlockNumber blocknum,
const void **buffers, BlockNumber nblocks,
@@ -127,4 +134,11 @@ smgrwrite(SMgrRelation reln, ForkNumber forknum, BlockNumber blocknum,
smgrwritev(reln, forknum, blocknum, &buffer, 1, skipFsync);
}
+extern void pgaio_io_set_target_smgr(PgAioHandle *ioh,
+ SMgrRelationData *smgr,
+ ForkNumber forknum,
+ BlockNumber blocknum,
+ int nblocks,
+ bool skip_fsync);
+
#endif /* SMGR_H */
diff --git a/src/backend/storage/aio/aio_callback.c b/src/backend/storage/aio/aio_callback.c
index 7392b55322c..6a21c82396a 100644
--- a/src/backend/storage/aio/aio_callback.c
+++ b/src/backend/storage/aio/aio_callback.c
@@ -18,10 +18,11 @@
#include "miscadmin.h"
#include "storage/aio.h"
#include "storage/aio_internal.h"
+#include "storage/md.h"
/* just to have something to put into the aio_handle_cbs */
-static const struct PgAioHandleCallbacks aio_invalid_cb = {0};
+static const PgAioHandleCallbacks aio_invalid_cb = {0};
typedef struct PgAioHandleCallbacksEntry
{
@@ -37,6 +38,8 @@ typedef struct PgAioHandleCallbacksEntry
static const PgAioHandleCallbacksEntry aio_handle_cbs[] = {
#define CALLBACK_ENTRY(id, callback) [id] = {.cb = &callback, .name = #callback}
CALLBACK_ENTRY(PGAIO_HCB_INVALID, aio_invalid_cb),
+
+ CALLBACK_ENTRY(PGAIO_HCB_MD_READV, aio_md_readv_cb),
#undef CALLBACK_ENTRY
};
diff --git a/src/backend/storage/aio/aio_target.c b/src/backend/storage/aio/aio_target.c
index 3fa813fd592..aab86a358da 100644
--- a/src/backend/storage/aio/aio_target.c
+++ b/src/backend/storage/aio/aio_target.c
@@ -16,6 +16,7 @@
#include "storage/aio.h"
#include "storage/aio_internal.h"
+#include "storage/smgr.h"
/*
@@ -29,6 +30,7 @@ static const PgAioTargetInfo *pgaio_target_info[] = {
[PGAIO_TID_INVALID] = &(PgAioTargetInfo) {
.name = "invalid",
},
+ [PGAIO_TID_SMGR] = &aio_smgr_target_info,
};
diff --git a/src/backend/storage/file/fd.c b/src/backend/storage/file/fd.c
index e454db4c020..be8c4c2d60d 100644
--- a/src/backend/storage/file/fd.c
+++ b/src/backend/storage/file/fd.c
@@ -94,6 +94,7 @@
#include "miscadmin.h"
#include "pgstat.h"
#include "postmaster/startup.h"
+#include "storage/aio.h"
#include "storage/fd.h"
#include "storage/ipc.h"
#include "utils/guc.h"
@@ -1294,6 +1295,8 @@ LruDelete(File file)
vfdP = &VfdCache[file];
+ pgaio_closing_fd(vfdP->fd);
+
/*
* Close the file. We aren't expecting this to fail; if it does, better
* to leak the FD than to mess up our internal state.
@@ -1987,6 +1990,8 @@ FileClose(File file)
if (!FileIsNotOpen(file))
{
+ pgaio_closing_fd(vfdP->fd);
+
/* close the file */
if (close(vfdP->fd) != 0)
{
@@ -2210,6 +2215,32 @@ retry:
return returnCode;
}
+int
+FileStartReadV(PgAioHandle *ioh, File file,
+ int iovcnt, off_t offset,
+ uint32 wait_event_info)
+{
+ int returnCode;
+ Vfd *vfdP;
+
+ Assert(FileIsValid(file));
+
+ DO_DB(elog(LOG, "FileStartReadV: %d (%s) " INT64_FORMAT " %d",
+ file, VfdCache[file].fileName,
+ (int64) offset,
+ iovcnt));
+
+ returnCode = FileAccess(file);
+ if (returnCode < 0)
+ return returnCode;
+
+ vfdP = &VfdCache[file];
+
+ pgaio_io_prep_readv(ioh, vfdP->fd, iovcnt, offset);
+
+ return 0;
+}
+
ssize_t
FileWriteV(File file, const struct iovec *iov, int iovcnt, off_t offset,
uint32 wait_event_info)
@@ -2498,6 +2529,12 @@ FilePathName(File file)
int
FileGetRawDesc(File file)
{
+ int returnCode;
+
+ returnCode = FileAccess(file);
+ if (returnCode < 0)
+ return returnCode;
+
Assert(FileIsValid(file));
return VfdCache[file].fd;
}
@@ -2778,6 +2815,7 @@ FreeDesc(AllocateDesc *desc)
result = closedir(desc->desc.dir);
break;
case AllocateDescRawFD:
+ pgaio_closing_fd(desc->desc.fd);
result = close(desc->desc.fd);
break;
default:
@@ -2846,6 +2884,8 @@ CloseTransientFile(int fd)
/* Only get here if someone passes us a file not in allocatedDescs */
elog(WARNING, "fd passed to CloseTransientFile was not obtained from OpenTransientFile");
+ pgaio_closing_fd(fd);
+
return close(fd);
}
diff --git a/src/backend/storage/smgr/md.c b/src/backend/storage/smgr/md.c
index f3220f98dc4..11d4d5a7aea 100644
--- a/src/backend/storage/smgr/md.c
+++ b/src/backend/storage/smgr/md.c
@@ -31,6 +31,7 @@
#include "miscadmin.h"
#include "pg_trace.h"
#include "pgstat.h"
+#include "storage/aio.h"
#include "storage/bufmgr.h"
#include "storage/fd.h"
#include "storage/md.h"
@@ -152,6 +153,15 @@ static MdfdVec *_mdfd_getseg(SMgrRelation reln, ForkNumber forknum,
static BlockNumber _mdnblocks(SMgrRelation reln, ForkNumber forknum,
MdfdVec *seg);
+static PgAioResult md_readv_complete(PgAioHandle *ioh, PgAioResult prior_result, uint8 cb_data);
+static void md_readv_report(PgAioResult result, const PgAioTargetData *target_data, int elevel);
+
+const PgAioHandleCallbacks aio_md_readv_cb = {
+ .complete_shared = md_readv_complete,
+ .report = md_readv_report,
+};
+
+
static inline int
_mdfd_open_flags(void)
{
@@ -937,6 +947,61 @@ mdreadv(SMgrRelation reln, ForkNumber forknum, BlockNumber blocknum,
}
}
+/*
+ * mdstartreadv() -- Asynchronous version of mdreadv().
+ */
+void
+mdstartreadv(PgAioHandle *ioh,
+ SMgrRelation reln, ForkNumber forknum, BlockNumber blocknum,
+ void **buffers, BlockNumber nblocks)
+{
+ off_t seekpos;
+ MdfdVec *v;
+ BlockNumber nblocks_this_segment;
+ struct iovec *iov;
+ int iovcnt;
+
+ v = _mdfd_getseg(reln, forknum, blocknum, false,
+ EXTENSION_FAIL | EXTENSION_CREATE_RECOVERY);
+
+ seekpos = (off_t) BLCKSZ * (blocknum % ((BlockNumber) RELSEG_SIZE));
+
+ Assert(seekpos < (off_t) BLCKSZ * RELSEG_SIZE);
+
+ nblocks_this_segment =
+ Min(nblocks,
+ RELSEG_SIZE - (blocknum % ((BlockNumber) RELSEG_SIZE)));
+
+ if (nblocks_this_segment != nblocks)
+ elog(ERROR, "read crossing segment boundary");
+
+ iovcnt = pgaio_io_get_iovec(ioh, &iov);
+
+ Assert(nblocks <= iovcnt);
+
+ iovcnt = buffers_to_iovec(iov, buffers, nblocks_this_segment);
+
+ Assert(iovcnt <= nblocks_this_segment);
+
+ if (!(io_direct_flags & IO_DIRECT_DATA))
+ pgaio_io_set_flag(ioh, PGAIO_HF_BUFFERED);
+
+ pgaio_io_set_target_smgr(ioh,
+ reln,
+ forknum,
+ blocknum,
+ nblocks,
+ false);
+ pgaio_io_register_callbacks(ioh, PGAIO_HCB_MD_READV, 0);
+
+ FileStartReadV(ioh, v->mdfd_vfd, iovcnt, seekpos, WAIT_EVENT_DATA_FILE_READ);
+
+ /*
+ * The error checks corresponding to the post-read checks in mdread() are
+ * in md_readv_complete().
+ */
+}
+
/*
* mdwritev() -- Write the supplied blocks at the appropriate location.
*
@@ -1365,6 +1430,21 @@ mdimmedsync(SMgrRelation reln, ForkNumber forknum)
}
}
+int
+mdfd(SMgrRelation reln, ForkNumber forknum, BlockNumber blocknum, uint32 *off)
+{
+ MdfdVec *v = mdopenfork(reln, forknum, EXTENSION_FAIL);
+
+ v = _mdfd_getseg(reln, forknum, blocknum, false,
+ EXTENSION_FAIL);
+
+ *off = (off_t) BLCKSZ * (blocknum % ((BlockNumber) RELSEG_SIZE));
+
+ Assert(*off < (off_t) BLCKSZ * RELSEG_SIZE);
+
+ return FileGetRawDesc(v->mdfd_vfd);
+}
+
/*
* register_dirty_segment() -- Mark a relation segment as needing fsync
*
@@ -1841,3 +1921,99 @@ mdfiletagmatches(const FileTag *ftag, const FileTag *candidate)
*/
return ftag->rlocator.dbOid == candidate->rlocator.dbOid;
}
+
+/*
+ * AIO completion callback for mdstartreadv().
+ */
+static PgAioResult
+md_readv_complete(PgAioHandle *ioh, PgAioResult prior_result, uint8 cb_data)
+{
+ PgAioTargetData *td = pgaio_io_get_target_data(ioh);
+ PgAioResult result = prior_result;
+
+ if (prior_result.result < 0)
+ {
+ result.status = ARS_ERROR;
+ result.id = PGAIO_HCB_MD_READV;
+ /* For "hard" errors, track the error number in error_data */
+ result.error_data = -prior_result.result;
+ result.result = 0;
+
+ pgaio_result_report(result, td, LOG);
+
+ return result;
+ }
+
+ /*
+ * The smgr API operates in blocks, therefore convert the result from
+ * bytes to blocks.
+ */
+ result.result /= BLCKSZ;
+
+ if (result.result == 0)
+ {
+ /* consider 0 blocks read a failure */
+ result.status = ARS_ERROR;
+ result.id = PGAIO_HCB_MD_READV;
+ result.error_data = 0;
+
+ pgaio_result_report(result, td, LOG);
+
+ return result;
+ }
+
+ if (result.status != ARS_ERROR &&
+ result.result < td->smgr.nblocks)
+ {
+ /* partial reads should be retried at upper level */
+ result.status = ARS_PARTIAL;
+ result.id = PGAIO_HCB_MD_READV;
+ }
+
+ return result;
+}
+
+/*
+ * AIO error reporting callback for mdstartreadv().
+ *
+ * Errors are encoded as follows:
+ * - PgAioResult.error_data != 0 encodes IO that failed with errno != 0
+ * - PgAioResult.error_data == 0 encodes IO that didn't read all data
+ */
+static void
+md_readv_report(PgAioResult result, const PgAioTargetData *td, int elevel)
+{
+ RelPathStr path;
+
+ path = relpathbackend(td->smgr.rlocator,
+ td->smgr.is_temp ? MyProcNumber : INVALID_PROC_NUMBER,
+ td->smgr.forkNum);
+
+ if (result.error_data != 0)
+ {
+ /* for errcode_for_file_access() and %m */
+ errno = result.error_data;
+
+ ereport(elevel,
+ errcode_for_file_access(),
+ errmsg("could not read blocks %u..%u in file \"%s\": %m",
+ td->smgr.blockNum,
+ td->smgr.blockNum + td->smgr.nblocks - 1,
+ path.str));
+ }
+ else
+ {
+ /*
+ * NB: This will typically only be output in debug messages, while
+ * retrying a partial IO.
+ */
+ ereport(elevel,
+ errcode(ERRCODE_DATA_CORRUPTED),
+ errmsg("could not read blocks %u..%u in file \"%s\": read only %zu of %zu bytes",
+ td->smgr.blockNum,
+ td->smgr.blockNum + td->smgr.nblocks - 1,
+ path.str,
+ result.result * (size_t) BLCKSZ,
+ td->smgr.nblocks * (size_t) BLCKSZ));
+ }
+}
diff --git a/src/backend/storage/smgr/smgr.c b/src/backend/storage/smgr/smgr.c
index ebe35c04de5..607c14ee173 100644
--- a/src/backend/storage/smgr/smgr.c
+++ b/src/backend/storage/smgr/smgr.c
@@ -53,6 +53,7 @@
#include "access/xlogutils.h"
#include "lib/ilist.h"
+#include "storage/aio.h"
#include "storage/bufmgr.h"
#include "storage/ipc.h"
#include "storage/md.h"
@@ -93,6 +94,10 @@ typedef struct f_smgr
void (*smgr_readv) (SMgrRelation reln, ForkNumber forknum,
BlockNumber blocknum,
void **buffers, BlockNumber nblocks);
+ void (*smgr_startreadv) (PgAioHandle *ioh,
+ SMgrRelation reln, ForkNumber forknum,
+ BlockNumber blocknum,
+ void **buffers, BlockNumber nblocks);
void (*smgr_writev) (SMgrRelation reln, ForkNumber forknum,
BlockNumber blocknum,
const void **buffers, BlockNumber nblocks,
@@ -104,6 +109,7 @@ typedef struct f_smgr
BlockNumber old_blocks, BlockNumber nblocks);
void (*smgr_immedsync) (SMgrRelation reln, ForkNumber forknum);
void (*smgr_registersync) (SMgrRelation reln, ForkNumber forknum);
+ int (*smgr_fd) (SMgrRelation reln, ForkNumber forknum, BlockNumber blocknum, uint32 *off);
} f_smgr;
static const f_smgr smgrsw[] = {
@@ -121,12 +127,14 @@ static const f_smgr smgrsw[] = {
.smgr_prefetch = mdprefetch,
.smgr_maxcombine = mdmaxcombine,
.smgr_readv = mdreadv,
+ .smgr_startreadv = mdstartreadv,
.smgr_writev = mdwritev,
.smgr_writeback = mdwriteback,
.smgr_nblocks = mdnblocks,
.smgr_truncate = mdtruncate,
.smgr_immedsync = mdimmedsync,
.smgr_registersync = mdregistersync,
+ .smgr_fd = mdfd,
}
};
@@ -144,6 +152,16 @@ static dlist_head unpinned_relns;
static void smgrshutdown(int code, Datum arg);
static void smgrdestroy(SMgrRelation reln);
+static void smgr_aio_reopen(PgAioHandle *ioh);
+static char *smgr_aio_describe_identity(const PgAioTargetData *sd);
+
+
+const PgAioTargetInfo aio_smgr_target_info = {
+ .name = "smgr",
+ .reopen = smgr_aio_reopen,
+ .describe_identity = smgr_aio_describe_identity,
+};
+
/*
* smgrinit(), smgrshutdown() -- Initialize or shut down storage
@@ -623,6 +641,22 @@ smgrreadv(SMgrRelation reln, ForkNumber forknum, BlockNumber blocknum,
nblocks);
}
+/*
+ * smgrstartreadv() -- asynchronous version of smgrreadv()
+ *
+ * This starts an asynchronous readv IO with the IO handle `ioh`. Other than
+ * `ioh` all parameters are the same as smgrreadv().
+ */
+void
+smgrstartreadv(PgAioHandle *ioh,
+ SMgrRelation reln, ForkNumber forknum, BlockNumber blocknum,
+ void **buffers, BlockNumber nblocks)
+{
+ smgrsw[reln->smgr_which].smgr_startreadv(ioh,
+ reln, forknum, blocknum, buffers,
+ nblocks);
+}
+
/*
* smgrwritev() -- Write the supplied buffers out.
*
@@ -819,6 +853,19 @@ smgrimmedsync(SMgrRelation reln, ForkNumber forknum)
smgrsw[reln->smgr_which].smgr_immedsync(reln, forknum);
}
+/*
+ * Return fd for the specified block number and update *off to the appropriate
+ * position.
+ *
+ * This is only to be used for when AIO needs to perform the IO in a different
+ * process than where it was issued (e.g. in an IO worker).
+ */
+static int
+smgrfd(SMgrRelation reln, ForkNumber forknum, BlockNumber blocknum, uint32 *off)
+{
+ return smgrsw[reln->smgr_which].smgr_fd(reln, forknum, blocknum, off);
+}
+
/*
* AtEOXact_SMgr
*
@@ -847,3 +894,80 @@ ProcessBarrierSmgrRelease(void)
smgrreleaseall();
return true;
}
+
+/*
+ * Set target of the IO handle to be smgr and initialize all the relevant
+ * pieces of data.
+ */
+void
+pgaio_io_set_target_smgr(PgAioHandle *ioh,
+ SMgrRelationData *smgr,
+ ForkNumber forknum,
+ BlockNumber blocknum,
+ int nblocks,
+ bool skip_fsync)
+{
+ PgAioTargetData *sd = pgaio_io_get_target_data(ioh);
+
+ pgaio_io_set_target(ioh, PGAIO_TID_SMGR);
+
+ /* backend is implied via IO owner */
+ sd->smgr.rlocator = smgr->smgr_rlocator.locator;
+ sd->smgr.forkNum = forknum;
+ sd->smgr.blockNum = blocknum;
+ sd->smgr.nblocks = nblocks;
+ sd->smgr.is_temp = SmgrIsTemp(smgr);
+ /* Temp relations should never be fsync'd */
+ sd->smgr.skip_fsync = skip_fsync && !SmgrIsTemp(smgr);
+}
+
+/*
+ * Callback for smgr AIO targets, to reopen the file (e.g. because the IO is
+ * executed in a worker).
+ */
+static void
+smgr_aio_reopen(PgAioHandle *ioh)
+{
+ PgAioTargetData *sd = pgaio_io_get_target_data(ioh);
+ PgAioOpData *od = pgaio_io_get_op_data(ioh);
+ SMgrRelation reln;
+ ProcNumber procno;
+ uint32 off;
+
+ if (sd->smgr.is_temp)
+ procno = pgaio_io_get_owner(ioh);
+ else
+ procno = INVALID_PROC_NUMBER;
+
+ reln = smgropen(sd->smgr.rlocator, procno);
+ od->read.fd = smgrfd(reln, sd->smgr.forkNum, sd->smgr.blockNum, &off);
+ Assert(off == od->read.offset);
+}
+
+/*
+ * Callback for smgr AIO targets, to describe the target of the IO.
+ */
+static char *
+smgr_aio_describe_identity(const PgAioTargetData *sd)
+{
+ RelPathStr path;
+ char *desc;
+
+ path = relpathbackend(sd->smgr.rlocator,
+ sd->smgr.is_temp ? MyProcNumber : INVALID_PROC_NUMBER,
+ sd->smgr.forkNum);
+
+ if (sd->smgr.nblocks == 0)
+ desc = psprintf(_("file \"%s\""), path.str);
+ else if (sd->smgr.nblocks == 1)
+ desc = psprintf(_("block %u in file \"%s\""),
+ sd->smgr.blockNum,
+ path.str);
+ else
+ desc = psprintf(_("blocks %u..%u in file \"%s\""),
+ sd->smgr.blockNum,
+ sd->smgr.blockNum + sd->smgr.nblocks - 1,
+ path.str);
+
+ return desc;
+}
--
2.48.1.76.g4e746b1a31.dirty
[text/x-diff] v2.6-0008-aio-Add-README.md-explaining-higher-level-desig.patch (18.6K, ../../ge6nsuddurhpmll3xj22vucvqwp4agqz6ndtcf2mhyeydzarst@l75dman5x53p/9-v2.6-0008-aio-Add-README.md-explaining-higher-level-desig.patch)
download | inline diff:
From bd05827716214a60d07ed34a47abd36dd0fac6c0 Mon Sep 17 00:00:00 2001
From: Andres Freund <andres@anarazel.de>
Date: Wed, 22 Jan 2025 13:44:37 -0500
Subject: [PATCH v2.6 08/34] aio: Add README.md explaining higher level design
---
src/backend/storage/aio/README.md | 422 ++++++++++++++++++++++++++++++
src/backend/storage/aio/aio.c | 2 +
2 files changed, 424 insertions(+)
create mode 100644 src/backend/storage/aio/README.md
diff --git a/src/backend/storage/aio/README.md b/src/backend/storage/aio/README.md
new file mode 100644
index 00000000000..de6c7db894b
--- /dev/null
+++ b/src/backend/storage/aio/README.md
@@ -0,0 +1,422 @@
+# Asynchronous & Direct IO
+
+## Motivation
+
+### Why Asynchronous IO
+
+Until the introduction of asynchronous IO Postgres relied on the operating
+system to hide the cost of synchronous IO from Postgres. While this worked
+surprisingly well in a lot of workloads, it does not do as good a job on
+prefetching and controlled writeback as we would like.
+
+There are important expensive operations like `fdatasync()` where the operating
+system cannot hide the storage latency. This is particularly important for WAL
+writes, where the ability to asynchronously issue `fdatasync()` or O_DSYNC
+writes can yield significantly higher throughput.
+
+
+### Why Direct / unbuffered IO
+
+The main reason to want to use Direct IO are:
+
+- Lower CPU usage / higher throughput. Particularly on modern storage buffered
+ writes are bottlenecked by the operating system having to copy data from the
+ kernel's page cache to postgres buffer pool using the CPU. Whereas direct IO
+ can often move the data directly between the storage devices and postgres'
+ buffer cache, using DMA. While that transfer is ongoing, the CPU is free to
+ perform other work.
+- Reduced latency - Direct IO can have substantially lower latency than
+ buffered IO, which can be impactful for OLTP workloads bottlenecked by WAL
+ write latency.
+- Avoiding double buffering between operating system cache and postgres'
+ shared_buffers.
+- Better control over the timing and pace of dirty data writeback.
+
+
+The main reason *not* to use Direct IO are:
+
+- Without AIO, Direct IO is unusably slow for most purposes.
+- Even with AIO, many parts of postgres need to be modified to perform
+ explicit prefetching.
+- In situations where shared_buffers cannot be set appropriately large,
+ e.g. because there are many different postgres instances hosted on shared
+ hardware, performance will often be worse than when using buffered IO.
+
+
+## AIO Usage Example
+
+In many cases code that can benefit from AIO does not directly have to
+interact with the AIO interface, but can use AIO via higher-level
+abstractions. See [Helpers](#helpers).
+
+In this example, a buffer will be read into shared buffers.
+
+```C
+/*
+ * Result of the operation, only to be accessed in this backend.
+ */
+PgAioReturn ioret;
+
+/*
+ * Acquire an AIO Handle, ioret will get result upon completion.
+ *
+ * Note that ioret needs to stay alive until the IO completes or
+ * CurrentResourceOwner is released (i.e. an error is thrown).
+ */
+PgAioHandle *ioh = pgaio_io_acquire(CurrentResourceOwner, &ioret);
+
+/*
+ * Reference that can be used to wait for the IO we initiate below. This
+ * reference can reside in local or shared memory and waited upon by any
+ * process. An arbitrary number of references can be made for each IO.
+ */
+PgAioWaitRef iow;
+
+pgaio_io_get_wref(ioh, &iow);
+
+/*
+ * Arrange for shared buffer completion callbacks to be called upon completion
+ * of the IO. This callback will update the buffer descriptors associated with
+ * the AioHandle, which e.g. allows other backends to access the buffer.
+ *
+ * A callback can be passed a small bit of data, e.g. to indicate whether to
+ * zero a buffer if it is invalid.
+ *
+ * Multiple completion callbacks can be registered for each handle.
+ */
+pgaio_io_register_callbacks(ioh, PGAIO_HCB_SHARED_BUFFER_READV, 0);
+
+/*
+ * The completion callback needs to know which buffers to update when the IO
+ * completes. As the AIO subsystem does not know about buffers, we have to
+ * associate this information with the AioHandle, for use by the completion
+ * callback registered above.
+ *
+ * In this example we're reading only a single buffer, hence the 1.
+ */
+pgaio_io_set_handle_data_32(ioh, (uint32 *) buffer, 1);
+
+/*
+ * Pass the AIO handle to lower-level function. When operating on the level of
+ * buffers, we don't know how exactly the IO is performed, that is the
+ * responsibility of the storage manager implementation.
+ *
+ * E.g. md.c needs to translate block numbers into offsets in segments.
+ *
+ * Once the IO handle has been handed off to smgstartreadv(), it may not
+ * further be used, as the IO may immediately get executed below
+ * smgrstartreadv() and the handle reused for another IO.
+ *
+ * To issue multiple IOs in an efficient way, a caller can call
+ * pgaio_enter_batchmode() before starting multiple IOs, and end that batch
+ * with pgaio_exit_batchmode(). Note that one needs to be careful while there
+ * may be unsubmitted IOs, as another backend may need to wait for one of the
+ * unsubmitted IOs. If this backend then had to to wait for the other backend,
+ * it'd end in an undetected deadlock. See pgaio_enter_batchmode() for more
+ * details.
+ *
+ * Note that even while in batchmode an IO might get submitted immediately,
+ * e.g. due to reaching a limit on the number of unsubmitted IOs, and even
+ * complete before smgrstartreadv() returns.
+ */
+smgrstartreadv(ioh, operation->smgr, forknum, blkno,
+ BufferGetBlock(buffer), 1);
+
+/*
+ * To benefit from AIO, it is beneficial to perform other work, including
+ * submitting other IOs, before waiting for the IO to complete. Otherwise
+ * we could just have used synchronous, blocking IO.
+ */
+perform_other_work();
+
+/*
+ * We did some other work and now need the IO operation to have completed to
+ * continue.
+ */
+pgaio_wref_wait(&iow);
+
+/*
+ * At this point the IO has completed. We do not yet know whether it succeeded
+ * or failed, however. The buffer's state has been updated, which allows other
+ * backends to use the buffer (if the IO succeeded), or retry the IO (if it
+ * failed).
+ *
+ * Note that in case the IO has failed, a LOG message may have been emitted,
+ * but no ERROR has been raised. This is crucial, as another backend waiting
+ * for this IO should not see an ERROR.
+ *
+ * To check whether the operation succeeded, and to raise an ERROR, or if more
+ * appropriate LOG, the PgAioReturn we passed to pgaio_io_acquire() is used.
+ */
+if (ioret.result.status == ARS_ERROR)
+ pgaio_result_report(aio_ret.result, &aio_ret.target_data, ERROR);
+
+/*
+ * Besides having succeeded completely, the IO could also have partially
+ * completed. If we e.g. tried to read many blocks at once, the read might have
+ * only succeeded for the first few blocks.
+ *
+ * If the IO partially succeeded and this backend needs all blocks to have
+ * completed, this backend needs to reissue the IO for the remaining buffers.
+ * The AIO subsystem cannot handle this retry transparently.
+ *
+ * As this example is already long, and we only read a single block, we'll just
+ * error out if there's a partial read.
+ */
+if (ioret.result.status == ARS_PARTIAL)
+ pgaio_result_report(aio_ret.result, &aio_ret.target_data, ERROR);
+
+/*
+ * The IO succeeded, so we can use the buffer now.
+ */
+```
+
+
+## Design Criteria & Motivation
+
+### Deadlock and Starvation Dangers due to AIO
+
+Using AIO in a naive way can easily lead to deadlocks in an environment where
+the source/target of AIO are shared resources, like pages in postgres'
+shared_buffers.
+
+Consider one backend performing readahead on a table, initiating IO for a
+number of buffers ahead of the current "scan position". If that backend then
+performs some operation that blocks, or even just is slow, the IO completion
+for the asynchronously initiated read may not be processed.
+
+This AIO implementation solves this problem by requiring that AIO methods
+either allow AIO completions to be processed by any backend in the system
+(e.g. io_uring), or to guarantee that AIO processing will happen even when the
+issuing backend is blocked (e.g. worker mode, which offloads completion
+processing to the AIO workers).
+
+
+### IO can be started in critical sections
+
+Using AIO for WAL writes can reduce the overhead of WAL logging substantially:
+
+- AIO allows to start WAL writes eagerly, so they complete before needing to
+ wait
+- AIO allows to have multiple WAL flushes in progress at the same time
+- AIO makes it more realistic to use O\_DIRECT + O\_DSYNC, which can reduce
+ the number of roundtrips to storage on some OSs and storage HW (buffered IO
+ and direct IO without O_DSYNC needs to issue a write and after the writes
+ completion a cache cache flush, whereas O\_DIRECT + O\_DSYNC can use a
+ single FUA write).
+
+The need to be able to execute IO in critical sections has substantial design
+implication on the AIO subsystem. Mainly because completing IOs (see prior
+section) needs to be possible within a critical section, even if the
+to-be-completed IO itself was not issued in a critical section. Consider
+e.g. the case of a backend first starting a number of writes from shared
+buffers and then starting to flush the WAL. Because only a limited amount of
+IO can be in-progress at the same time, initiating IO for flushing the WAL may
+require to first complete IO that was started earlier.
+
+
+### State for AIO needs to live in shared memory
+
+Because postgres uses a process model and because AIOs need to be
+complete-able by any backend much of the state of the AIO subsystem needs to
+live in shared memory.
+
+In an `EXEC_BACKEND` build backends executable code and other process local
+state is not necessarily mapped to the same addresses in each process due to
+ASLR. This means that the shared memory cannot contain pointer to callbacks.
+
+
+## Design of the AIO Subsystem
+
+
+### AIO Methods
+
+To achieve portability and performance, multiple methods of performing AIO are
+implemented and others are likely worth adding in the future.
+
+
+#### Synchronous Mode
+
+`io_method=sync` does not actually perform AIO but allows to use the AIO API
+while performing synchronous IO. This can be useful for debugging. The code
+for the synchronous mode is also used as a fallback by e.g. the [worker
+mode](#worker) uses it to execute IO that cannot be executed by workers.
+
+
+#### Worker
+
+`io_method=worker` is available on every platform postgres runs on, and
+implements asynchronous IO - from the view of the issuing process - by
+dispatching the IO to one of several worker processes performing the IO in a
+synchronous manner.
+
+
+#### io_uring
+
+`io_method=io_uring` is available on Linux 5.1+. In contrast to worker mode it
+dispatches all IO from within the process, lowering context switch rate /
+latency.
+
+
+### AIO Handles
+
+The central API piece for postgres' AIO abstraction are AIO handles. To
+execute an IO one first has to acquire an IO handle (`pgaio_io_acquire()`) and
+then "defined", i.e. associate an IO operation with the handle.
+
+Often AIO handles are acquired on a higher level and then passed to a lower
+level to be fully defined. E.g., for IO to/from shared buffers, bufmgr.c
+routines acquire the handle, which is then passed through smgr.c, md.c to be
+finally fully defined in fd.c.
+
+The functions used at the lowest level to define the operation are
+`pgaio_io_prep_*()`.
+
+Because acquisition of an IO handle
+[must always succeed](#io-can-be-started-in-critical-sections)
+and the number of AIO Handles
+[has to be limited](#state-for-aio-needs-to-live-in-shared-memory)
+AIO handles can be reused as soon as they have completed. Obviously code needs
+to be able to react to IO completion. Shared state can be updated using
+[AIO Completion callbacks](#aio-callbacks)
+and the issuing backend can provide a backend local variable to receive the
+result of the IO, as described in
+[AIO Result](#aio-results)
+. An IO can be waited for, by both the issuing and any other backend, using
+[AIO References](#aio-wait-references).
+
+
+Because an AIO Handle is not executable just after calling `pgaio_io_acquire()`
+and because `pgaio_io_acquire()` needs to be able to succeed, only a single AIO
+Handle may be acquired (i.e. returned by `pgaio_io_acquire()`) without causing
+the IO to have been defined (by, potentially indirectly, causing
+`pgaio_io_prep_*()` to have been called). Otherwise a backend could trivially
+self-deadlock by using up all AIO Handles without the ability to wait for some
+of the IOs to complete.
+
+If it turns out that an AIO Handle is not needed, e.g., because the handle was
+acquired before holding a contended lock, it can be released without being
+defined using `pgaio_io_release()`.
+
+
+### AIO Callbacks
+
+Commonly several layers need to react to completion of an IO. E.g. for a read
+md.c needs to check if the IO outright failed or was shorter than needed,
+bufmgr.c needs to verify the page looks valid and bufmgr.c needs to update the
+BufferDesc to update the buffer's state.
+
+The fact that several layers / subsystems need to react to IO completion poses
+a few challenges:
+
+- Upper layers should not need to know details of lower layers. E.g. bufmgr.c
+ should not assume the IO will pass through md.c. Therefore upper levels
+ cannot know what lower layers would consider an error.
+
+- Lower layers should not need to know about upper layers. E.g. smgr APIs are
+ used going through shared buffers but are also used bypassing shared
+ buffers. This means that e.g. md.c is not in a position to validate
+ checksums.
+
+- Having code in the AIO subsystem for every possible combination of layers
+ would lead to a lot of duplication.
+
+The "solution" to this the ability to associate multiple completion callbacks
+with a handle. E.g. bufmgr.c can have a callback to update the BufferDesc
+state and to verify the page and md.c. another callback to check if the IO
+operation was successful.
+
+As [mentioned](#state-for-aio-needs-to-live-in-shared-memory), shared memory
+currently cannot contain function pointers. Because of that completion
+callbacks are not directly identified by function pointers but by IDs
+(`PgAioHandleCallbackID`). A substantial added benefit is that that
+allows callbacks to be identified by much smaller amount of memory (a single
+byte currently).
+
+In addition to completion, AIO callbacks also are called to "prepare" an
+IO. This is, e.g., used to increase buffer reference counts to account for the
+AIO subsystem referencing the buffer, which is required to handle the case
+where the issuing backend errors out and releases its own pins while the IO is
+still ongoing.
+
+As [explained earlier](#io-can-be-started-in-critical-sections) IO completions
+need to be safe to execute in critical sections. To allow the backend that
+issued the IO to error out in case of failure [AIO Result](#aio-results) can
+be used.
+
+
+### AIO Targets
+
+In addition to the completion callbacks describe above, each AIO Handle has
+exactly one "target". Each target has some space inside an AIO Handle with
+information specific to the target and can provide callbacks to allow to
+reopen the underlying file (required for worker mode) and to describe the IO
+operation (used for debug logging and error messages).
+
+I.e., if two different uses of AIO can describe the identity of the file being
+operated on the same way, it likely makes sense to use the same
+target. E.g. different smgr implementations can describe IO with
+RelFileLocator, ForkNumber and BlockNumber and can thus share a target. In
+contrast, IO for a WAL file would be described with TimeLineID and XLogRecPtr
+and it would not make sense to use the same target for smgr and WAL.
+
+
+### AIO Wait References
+
+As [described above](#aio-handles), AIO Handles can be reused immediately
+after completion and therefore cannot be used to wait for completion of the
+IO. Waiting is enabled using AIO wait references, which do not just identify
+an AIO Handle but also include the handles "generation".
+
+A reference to an AIO Handle can be acquired using `pgaio_io_get_wref()` and
+then waited upon using `pgaio_wref_wait()`.
+
+
+### AIO Results
+
+As AIO completion callbacks
+[are executed in critical sections](#io-can-be-started-in-critical-sections)
+and [may be executed by any backend](#deadlock-and-starvation-dangers-due-to-aio)
+completion callbacks cannot be used to, e.g., make the query that triggered an
+IO ERROR out.
+
+To allow to react to failing IOs the issuing backend can pass a pointer to a
+`PgAioReturn` in backend local memory. Before an AIO Handle is reused the
+`PgAioReturn` is filled with information about the IO. This includes
+information about whether the IO was successful (as a value of
+`PgAioResultStatus`) and enough information to raise an error in case of a
+failure (via `pgaio_result_report()`, with the error details encoded in
+`PgAioResult`).
+
+
+### AIO Errors
+
+It would be very convenient to have shared completion callbacks encode the
+details of errors as an `ErrorData` that could be raised at a later
+time. Unfortunately doing so would require allocating memory. While elog.c can
+guarantee (well, kinda) that logging a message will not run out of memory,
+that only works because a very limited number of messages are in the process
+of being logged. With AIO a large number of concurrently issued AIOs might
+fail.
+
+To avoid the need for preallocating a potentially large amount of memory (in
+shared memory no less!), completion callbacks instead have to encode errors in
+a more compact format that can be converted into an error message.
+
+
+## Helpers
+
+Using the low-level AIO API introduces too much complexity to do so all over
+the tree. Most uses of AIO should be done via reusable, higher-level,
+helpers.
+
+
+### Read Stream
+
+A common and very beneficial use of AIO are reads where a substantial number
+of to-be-read locations are known ahead of time. E.g., for a sequential scan
+the set of blocks that need to be read can be determined solely by knowing the
+current position and checking the buffer mapping table.
+
+The [Read Stream](../../../include/storage/read_stream.h) interface makes it
+comparatively easy to use AIO for such use cases.
diff --git a/src/backend/storage/aio/aio.c b/src/backend/storage/aio/aio.c
index a8227efa0b5..c2b34a95e1c 100644
--- a/src/backend/storage/aio/aio.c
+++ b/src/backend/storage/aio/aio.c
@@ -24,6 +24,8 @@
*
* - read_stream.c - helper for reading buffered relation data
*
+ * - README.md - higher-level overview over AIO
+ *
*
* Portions Copyright (c) 1996-2025, PostgreSQL Global Development Group
* Portions Copyright (c) 1994, Regents of the University of California
--
2.48.1.76.g4e746b1a31.dirty
[text/x-diff] v2.6-0009-aio-Add-pg_aios-view.patch (9.7K, ../../ge6nsuddurhpmll3xj22vucvqwp4agqz6ndtcf2mhyeydzarst@l75dman5x53p/10-v2.6-0009-aio-Add-pg_aios-view.patch)
download | inline diff:
From 460e366fe8797f2e44b9e4c1ed7500120af7da30 Mon Sep 17 00:00:00 2001
From: Andres Freund <andres@anarazel.de>
Date: Wed, 22 Jan 2025 13:44:40 -0500
Subject: [PATCH v2.6 09/34] aio: Add pg_aios view
Author:
Reviewed-by:
Discussion: https://postgr.es/m/
Backpatch:
---
src/include/catalog/pg_proc.dat | 10 ++
src/backend/catalog/system_views.sql | 3 +
src/backend/storage/aio/Makefile | 1 +
src/backend/storage/aio/aio_funcs.c | 219 +++++++++++++++++++++++++++
src/backend/storage/aio/meson.build | 1 +
src/test/regress/expected/rules.out | 16 ++
6 files changed, 250 insertions(+)
create mode 100644 src/backend/storage/aio/aio_funcs.c
diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat
index cede992b6e2..f416ed387d0 100644
--- a/src/include/catalog/pg_proc.dat
+++ b/src/include/catalog/pg_proc.dat
@@ -12469,4 +12469,14 @@
proargtypes => 'int4',
prosrc => 'gist_stratnum_common' },
+# AIO related functions
+{ oid => '9200', descr => 'information about in-progress asynchronous IOs',
+ proname => 'pg_get_aios', prorows => '100', proretset => 't',
+ provolatile => 'v', proparallel => 'r', prorettype => 'record', proargtypes => '',
+ proallargtypes => '{int4,int4,int8,text,text,int8,int8,text,int2,int4,text,text,bool,bool,bool}',
+ proargmodes => '{o,o,o,o,o,o,o,o,o,o,o,o,o,o,o}',
+ proargnames => '{pid,io_id,io_generation,state,operation,offset,length,target,handle_data_len,raw_result,result,target_desc,f_sync,f_localmem,f_buffered}',
+ prosrc => 'pg_get_aios' },
+
+
]
diff --git a/src/backend/catalog/system_views.sql b/src/backend/catalog/system_views.sql
index a4d2cfdcaf5..73902a763d1 100644
--- a/src/backend/catalog/system_views.sql
+++ b/src/backend/catalog/system_views.sql
@@ -1390,3 +1390,6 @@ CREATE VIEW pg_stat_subscription_stats AS
CREATE VIEW pg_wait_events AS
SELECT * FROM pg_get_wait_events();
+
+CREATE VIEW pg_aios AS
+ SELECT * FROM pg_get_aios();
diff --git a/src/backend/storage/aio/Makefile b/src/backend/storage/aio/Makefile
index c06c50771e0..3f2469cc399 100644
--- a/src/backend/storage/aio/Makefile
+++ b/src/backend/storage/aio/Makefile
@@ -11,6 +11,7 @@ include $(top_builddir)/src/Makefile.global
OBJS = \
aio.o \
aio_callback.o \
+ aio_funcs.o \
aio_init.o \
aio_io.o \
aio_target.o \
diff --git a/src/backend/storage/aio/aio_funcs.c b/src/backend/storage/aio/aio_funcs.c
new file mode 100644
index 00000000000..58883fd61d8
--- /dev/null
+++ b/src/backend/storage/aio/aio_funcs.c
@@ -0,0 +1,219 @@
+/*-------------------------------------------------------------------------
+ *
+ * aio_funcs.c
+ * AIO - SQL interface for AIO
+ *
+ *
+ * Portions Copyright (c) 1996-2025, PostgreSQL Global Development Group
+ * Portions Copyright (c) 1994, Regents of the University of California
+ *
+ * IDENTIFICATION
+ * src/backend/storage/aio/aio_funcs.c
+ *
+ *-------------------------------------------------------------------------
+ */
+
+#include "postgres.h"
+
+#include "storage/aio.h"
+#include "storage/aio_internal.h"
+#include "utils/builtins.h"
+#include "funcapi.h"
+#include "storage/proc.h"
+
+
+/*
+ * Byte length of an iovec.
+ */
+static size_t
+iov_byte_length(const struct iovec *iov, int cnt)
+{
+ size_t len = 0;
+
+ for (int i = 0; i < cnt; i++)
+ {
+ len += iov[i].iov_len;
+ }
+
+ return len;
+}
+
+Datum
+pg_get_aios(PG_FUNCTION_ARGS)
+{
+ ReturnSetInfo *rsinfo = (ReturnSetInfo *) fcinfo->resultinfo;
+
+ InitMaterializedSRF(fcinfo, 0);
+
+#define PG_GET_AIOS_COLS 15
+
+ for (uint64 i = 0; i < pgaio_ctl->io_handle_count; i++)
+ {
+ PgAioHandle *live_ioh = &pgaio_ctl->io_handles[i];
+ uint32 ioh_id = pgaio_io_get_id(live_ioh);
+ Datum values[PG_GET_AIOS_COLS] = {0};
+ bool nulls[PG_GET_AIOS_COLS] = {0};
+ ProcNumber owner;
+ PGPROC *owner_proc;
+ int32 owner_pid;
+ PgAioHandleState start_state;
+ uint64 start_generation;
+ PgAioHandle ioh_copy;
+ struct iovec iov_copy[PG_IOV_MAX];
+
+retry:
+
+ /*
+ * There is no lock that could prevent the state of the IO to advance
+ * concurrently - and we don't want to introduce one, as that would
+ * introduce atomics into a very common path. Instead we
+ *
+ * 1) Determine the state + generation of the IO.
+ *
+ * 2) Copy the IO to local memory.
+ *
+ * 3) Check if state or generation of the IO changed. If the state
+ * changed, retry, if the generation changed don't display the IO.
+ */
+
+ /* 1) from above */
+ start_generation = live_ioh->generation;
+ pg_read_barrier();
+ start_state = live_ioh->state;
+
+ if (start_state == PGAIO_HS_IDLE)
+ continue;
+
+ /* 2) from above */
+ memcpy(&ioh_copy, live_ioh, sizeof(PgAioHandle));
+
+ /*
+ * Safe to copy even if no iovec is used - we always reserve the
+ * required space.
+ */
+ memcpy(&iov_copy, &pgaio_ctl->iovecs[ioh_copy.iovec_off],
+ PG_IOV_MAX * sizeof(struct iovec));
+
+ /*
+ * Copy information about owner before 3) below, if the process exited
+ * it'd have to wait for the IO to finish first, which we would detect
+ * in 3).
+ */
+ owner = ioh_copy.owner_procno;
+ owner_proc = GetPGProcByNumber(owner);
+ owner_pid = owner_proc->pid;
+
+ /* 3) from above */
+ pg_read_barrier();
+
+ /*
+ * The IO completed and a new one was started with the same ID. Don't
+ * display it - it really started after this function was called.
+ * There be a risk of a livelock if we just retried endlessly, if IOs
+ * complete very quickly.
+ */
+ if (live_ioh->generation != start_generation)
+ continue;
+
+ /*
+ * The IOs state changed while we were "rendering" it. Just start from
+ * scratch. There's no risk of a livelock here, as an IO has a limited
+ * sets of states it can be in, and state changes go only in a single
+ * direction.
+ */
+ if (live_ioh->state != start_state)
+ goto retry;
+
+ /*
+ * Now that we have copied the IO into local memory and checked that
+ * it's still in the same state, we are not allowed to access "live"
+ * memory anymore. To make it slightly easier to catch such cases, set
+ * the "live" pointers to NULL.
+ */
+ live_ioh = NULL;
+ owner_proc = NULL;
+
+
+ /* column: owning pid */
+ if (owner_pid != 0)
+ values[0] = Int32GetDatum(owner_pid);
+ else
+ nulls[0] = false;
+
+ /* column: IO's id */
+ values[1] = ioh_id;
+
+ /* column: IO's generation */
+ values[2] = Int64GetDatum(start_generation);
+
+ /* column: IO's state */
+ values[3] = CStringGetTextDatum(pgaio_io_get_state_name(&ioh_copy));
+
+ /*
+ * If the IO is in PGAIO_HS_HANDED_OUT state, none of it's fields are
+ * valid yet (or are in the process of being set). Therefore we don't
+ * want to display any other columns.
+ */
+ if (start_state == PGAIO_HS_HANDED_OUT)
+ {
+ memset(nulls + 4, 1, (lengthof(nulls) - 4) * sizeof(bool));
+ goto display;
+ }
+
+ /* column: IO's operation */
+ values[4] = CStringGetTextDatum(pgaio_io_get_op_name(&ioh_copy));
+
+ /* columns: details about the IO's operation */
+ switch (ioh_copy.op)
+ {
+ case PGAIO_OP_INVALID:
+ nulls[5] = true;
+ nulls[6] = true;
+ break;
+ case PGAIO_OP_READV:
+ values[5] = Int64GetDatum(ioh_copy.op_data.read.offset);
+ values[6] =
+ Int64GetDatum(iov_byte_length(iov_copy, ioh_copy.op_data.read.iov_length));
+ break;
+ case PGAIO_OP_WRITEV:
+ values[5] = Int64GetDatum(ioh_copy.op_data.write.offset);
+ values[6] =
+ Int64GetDatum(iov_byte_length(iov_copy, ioh_copy.op_data.write.iov_length));
+ break;
+ }
+
+ /* column: IO's target */
+ values[7] = CStringGetTextDatum(pgaio_io_get_target_name(&ioh_copy));
+
+ /* column: length of IO's data array */
+ values[8] = Int16GetDatum(ioh_copy.handle_data_len);
+
+ /* column: raw result (i.e. some form of syscall return value) */
+ if (start_state == PGAIO_HS_COMPLETED_IO
+ || start_state == PGAIO_HS_COMPLETED_SHARED)
+ values[9] = Int32GetDatum(ioh_copy.result);
+ else
+ nulls[9] = true;
+
+ /*
+ * column: result in the higher level representation (unknown if not
+ * finished)
+ */
+ values[10] =
+ CStringGetTextDatum(pgaio_result_status_string(ioh_copy.distilled_result.status));
+
+ /* column: target description */
+ values[11] = CStringGetTextDatum(pgaio_io_get_target_description(&ioh_copy));
+
+ /* columns: one for each flag */
+ values[12] = BoolGetDatum(ioh_copy.flags & PGAIO_HF_SYNCHRONOUS);
+ values[13] = BoolGetDatum(ioh_copy.flags & PGAIO_HF_REFERENCES_LOCAL);
+ values[14] = BoolGetDatum(ioh_copy.flags & PGAIO_HF_BUFFERED);
+
+display:
+
+ tuplestore_putvalues(rsinfo->setResult, rsinfo->setDesc, values, nulls);
+ }
+
+ return (Datum) 0;
+}
diff --git a/src/backend/storage/aio/meson.build b/src/backend/storage/aio/meson.build
index 2f0f03d8071..da6df2d3654 100644
--- a/src/backend/storage/aio/meson.build
+++ b/src/backend/storage/aio/meson.build
@@ -3,6 +3,7 @@
backend_sources += files(
'aio.c',
'aio_callback.c',
+ 'aio_funcs.c',
'aio_init.c',
'aio_io.c',
'aio_target.c',
diff --git a/src/test/regress/expected/rules.out b/src/test/regress/expected/rules.out
index 62f69ac20b2..6570d0cddce 100644
--- a/src/test/regress/expected/rules.out
+++ b/src/test/regress/expected/rules.out
@@ -1286,6 +1286,22 @@ drop table cchild;
SELECT viewname, definition FROM pg_views
WHERE schemaname = 'pg_catalog'
ORDER BY viewname;
+pg_aios| SELECT pid,
+ io_id,
+ io_generation,
+ state,
+ operation,
+ "offset",
+ length,
+ target,
+ handle_data_len,
+ raw_result,
+ result,
+ target_desc,
+ f_sync,
+ f_localmem,
+ f_buffered
+ FROM pg_get_aios() pg_get_aios(pid, io_id, io_generation, state, operation, "offset", length, target, handle_data_len, raw_result, result, target_desc, f_sync, f_localmem, f_buffered);
pg_available_extension_versions| SELECT e.name,
e.version,
(x.extname IS NOT NULL) AS installed,
--
2.48.1.76.g4e746b1a31.dirty
[text/x-diff] v2.6-0010-Refactor-read_stream.c-s-circular-arithmetic.patch (4.6K, ../../ge6nsuddurhpmll3xj22vucvqwp4agqz6ndtcf2mhyeydzarst@l75dman5x53p/11-v2.6-0010-Refactor-read_stream.c-s-circular-arithmetic.patch)
download | inline diff:
From 36a84f449e69e7dbbd739e24fdd17e94fedf875f Mon Sep 17 00:00:00 2001
From: Thomas Munro <thomas.munro@gmail.com>
Date: Sat, 15 Feb 2025 14:47:25 +1300
Subject: [PATCH v2.6 10/34] Refactor read_stream.c's circular arithmetic.
Several places have open-coded circular index arithmetic. Make some
common functions for better readability and consistent assertion
checking.
This avoids adding yet more open-coded duplication in later patches, and
standardizes on the vocabulary "advance" and "retreat" as used elsewhere
in PostgreSQL.
---
src/backend/storage/aio/read_stream.c | 78 +++++++++++++++++++++------
1 file changed, 61 insertions(+), 17 deletions(-)
diff --git a/src/backend/storage/aio/read_stream.c b/src/backend/storage/aio/read_stream.c
index 04bdb5e6d4b..dee51fa85a9 100644
--- a/src/backend/storage/aio/read_stream.c
+++ b/src/backend/storage/aio/read_stream.c
@@ -224,6 +224,55 @@ read_stream_unget_block(ReadStream *stream, BlockNumber blocknum)
stream->buffered_blocknum = blocknum;
}
+/*
+ * Increment index, wrapping around at queue size.
+ */
+static inline void
+read_stream_index_advance(ReadStream *stream, int16 *index)
+{
+ Assert(*index >= 0);
+ Assert(*index < stream->queue_size);
+
+ *index += 1;
+ if (*index == stream->queue_size)
+ *index = 0;
+}
+
+/*
+ * Increment index by n, wrapping around at queue size.
+ */
+static inline void
+read_stream_index_advance_n(ReadStream *stream, int16 *index, int16 n)
+{
+ Assert(*index >= 0);
+ Assert(*index < stream->queue_size);
+ Assert(n <= MAX_IO_COMBINE_LIMIT);
+
+ *index += n;
+ if (*index >= stream->queue_size)
+ *index -= stream->queue_size;
+
+ Assert(*index >= 0);
+ Assert(*index < stream->queue_size);
+}
+
+#if defined(CLOBBER_FREED_MEMORY) || defined(USE_VALGRIND)
+/*
+ * Decrement index, wrapping around at queue size.
+ */
+static inline void
+read_stream_index_retreat(ReadStream *stream, int16 *index)
+{
+ Assert(*index >= 0);
+ Assert(*index < stream->queue_size);
+
+ if (*index == 0)
+ *index = stream->queue_size - 1;
+ else
+ *index -= 1;
+}
+#endif
+
static void
read_stream_start_pending_read(ReadStream *stream, bool suppress_advice)
{
@@ -302,11 +351,8 @@ read_stream_start_pending_read(ReadStream *stream, bool suppress_advice)
&stream->buffers[stream->queue_size],
sizeof(stream->buffers[0]) * overflow);
- /* Compute location of start of next read, without using % operator. */
- buffer_index += nblocks;
- if (buffer_index >= stream->queue_size)
- buffer_index -= stream->queue_size;
- Assert(buffer_index >= 0 && buffer_index < stream->queue_size);
+ /* Move to the location of start of next read. */
+ read_stream_index_advance_n(stream, &buffer_index, nblocks);
stream->next_buffer_index = buffer_index;
/* Adjust the pending read to cover the remaining portion, if any. */
@@ -334,12 +380,12 @@ read_stream_look_ahead(ReadStream *stream, bool suppress_advice)
/*
* See which block the callback wants next in the stream. We need to
* compute the index of the Nth block of the pending read including
- * wrap-around, but we don't want to use the expensive % operator.
+ * wrap-around.
*/
- buffer_index = stream->next_buffer_index + stream->pending_read_nblocks;
- if (buffer_index >= stream->queue_size)
- buffer_index -= stream->queue_size;
- Assert(buffer_index >= 0 && buffer_index < stream->queue_size);
+ buffer_index = stream->next_buffer_index;
+ read_stream_index_advance_n(stream,
+ &buffer_index,
+ stream->pending_read_nblocks);
per_buffer_data = get_per_buffer_data(stream, buffer_index);
blocknum = read_stream_get_block(stream, per_buffer_data);
if (blocknum == InvalidBlockNumber)
@@ -781,12 +827,12 @@ read_stream_next_buffer(ReadStream *stream, void **per_buffer_data)
*/
if (stream->per_buffer_data)
{
+ int16 index;
void *per_buffer_data;
- per_buffer_data = get_per_buffer_data(stream,
- oldest_buffer_index == 0 ?
- stream->queue_size - 1 :
- oldest_buffer_index - 1);
+ index = oldest_buffer_index;
+ read_stream_index_retreat(stream, &index);
+ per_buffer_data = get_per_buffer_data(stream, index);
#if defined(CLOBBER_FREED_MEMORY)
/* This also tells Valgrind the memory is "noaccess". */
@@ -804,9 +850,7 @@ read_stream_next_buffer(ReadStream *stream, void **per_buffer_data)
stream->pinned_buffers--;
/* Advance oldest buffer, with wrap-around. */
- stream->oldest_buffer_index++;
- if (stream->oldest_buffer_index == stream->queue_size)
- stream->oldest_buffer_index = 0;
+ read_stream_index_advance(stream, &stream->oldest_buffer_index);
/* Prepare for the next call. */
read_stream_look_ahead(stream, false);
--
2.48.1.76.g4e746b1a31.dirty
[text/x-diff] v2.6-0011-Improve-buffer-pool-API-for-per-backend-pin-lim.patch (6.4K, ../../ge6nsuddurhpmll3xj22vucvqwp4agqz6ndtcf2mhyeydzarst@l75dman5x53p/12-v2.6-0011-Improve-buffer-pool-API-for-per-backend-pin-lim.patch)
download | inline diff:
From 0b09f52651d5e48ef5e25931209ce231aab58bc6 Mon Sep 17 00:00:00 2001
From: Thomas Munro <thomas.munro@gmail.com>
Date: Fri, 24 Jan 2025 10:59:39 +1300
Subject: [PATCH v2.6 11/34] Improve buffer pool API for per-backend pin
limits.
Previously the support functions assumed that you needed one additional
pin to make progress, and could optionally use some more. Add a couple
more functions for callers that want to know:
* what the maximum possible number could be, for space planning
purposes, called the "soft pin limit"
* how many additional pins they could acquire right now, without the
special case allowing one pin (ie for users that already hold pins and
can already make progress even if zero extra pins are available now)
These APIs are better suited to read_stream.c, which will be adjusted in
a follow-up patch. Also move the computation of the each backend's fair
share of the buffer pool to backend initialization time, since the
answer doesn't change and we don't want to perform a division operation
every time we compute availability.
---
src/include/storage/bufmgr.h | 4 ++
src/backend/storage/buffer/bufmgr.c | 75 ++++++++++++++++++++-------
src/backend/storage/buffer/localbuf.c | 16 ++++++
3 files changed, 77 insertions(+), 18 deletions(-)
diff --git a/src/include/storage/bufmgr.h b/src/include/storage/bufmgr.h
index 7c1e4316dde..597ecb97897 100644
--- a/src/include/storage/bufmgr.h
+++ b/src/include/storage/bufmgr.h
@@ -290,6 +290,10 @@ extern bool HoldingBufferPinThatDelaysRecovery(void);
extern bool BgBufferSync(struct WritebackContext *wb_context);
+extern uint32 GetSoftPinLimit(void);
+extern uint32 GetSoftLocalPinLimit(void);
+extern uint32 GetAdditionalPinLimit(void);
+extern uint32 GetAdditionalLocalPinLimit(void);
extern void LimitAdditionalPins(uint32 *additional_pins);
extern void LimitAdditionalLocalPins(uint32 *additional_pins);
diff --git a/src/backend/storage/buffer/bufmgr.c b/src/backend/storage/buffer/bufmgr.c
index 7915ed624c1..11c146763db 100644
--- a/src/backend/storage/buffer/bufmgr.c
+++ b/src/backend/storage/buffer/bufmgr.c
@@ -211,6 +211,8 @@ static int32 PrivateRefCountOverflowed = 0;
static uint32 PrivateRefCountClock = 0;
static PrivateRefCountEntry *ReservedRefCountEntry = NULL;
+static uint32 MaxProportionalPins;
+
static void ReservePrivateRefCountEntry(void);
static PrivateRefCountEntry *NewPrivateRefCountEntry(Buffer buffer);
static PrivateRefCountEntry *GetPrivateRefCountEntry(Buffer buffer, bool do_move);
@@ -2097,6 +2099,46 @@ again:
return buf;
}
+/*
+ * Return the maximum number of buffer than this backend should try to pin at
+ * once, to avoid pinning more than its fair share. This is the highest value
+ * that GetAdditionalPinLimit() and LimitAdditionalPins() could ever return.
+ *
+ * It's called a soft limit because nothing stops a backend from trying to
+ * acquire more pins than this this with ReadBuffer(), but code that wants more
+ * for I/O optimizations should respect this per-backend limit when it can
+ * still make progress without them.
+ */
+uint32
+GetSoftPinLimit(void)
+{
+ return MaxProportionalPins;
+}
+
+/*
+ * Return the maximum number of additional buffers that this backend should
+ * pin if it wants to stay under the per-backend soft limit, considering the
+ * number of buffers it has already pinned.
+ */
+uint32
+GetAdditionalPinLimit(void)
+{
+ uint32 estimated_pins_held;
+
+ /*
+ * We get the number of "overflowed" pins for free, but don't know the
+ * number of pins in PrivateRefCountArray. The cost of calculating that
+ * exactly doesn't seem worth it, so just assume the max.
+ */
+ estimated_pins_held = PrivateRefCountOverflowed + REFCOUNT_ARRAY_ENTRIES;
+
+ /* Is this backend already holding more than its fair share? */
+ if (estimated_pins_held > MaxProportionalPins)
+ return 0;
+
+ return MaxProportionalPins - estimated_pins_held;
+}
+
/*
* Limit the number of pins a batch operation may additionally acquire, to
* avoid running out of pinnable buffers.
@@ -2112,28 +2154,15 @@ again:
void
LimitAdditionalPins(uint32 *additional_pins)
{
- uint32 max_backends;
- int max_proportional_pins;
+ uint32 limit;
if (*additional_pins <= 1)
return;
- max_backends = MaxBackends + NUM_AUXILIARY_PROCS;
- max_proportional_pins = NBuffers / max_backends;
-
- /*
- * Subtract the approximate number of buffers already pinned by this
- * backend. We get the number of "overflowed" pins for free, but don't
- * know the number of pins in PrivateRefCountArray. The cost of
- * calculating that exactly doesn't seem worth it, so just assume the max.
- */
- max_proportional_pins -= PrivateRefCountOverflowed + REFCOUNT_ARRAY_ENTRIES;
-
- if (max_proportional_pins <= 0)
- max_proportional_pins = 1;
-
- if (*additional_pins > max_proportional_pins)
- *additional_pins = max_proportional_pins;
+ limit = GetAdditionalPinLimit();
+ limit = Max(limit, 1);
+ if (limit < *additional_pins)
+ *additional_pins = limit;
}
/*
@@ -3575,6 +3604,16 @@ InitBufferManagerAccess(void)
{
HASHCTL hash_ctl;
+ /*
+ * The soft limit on the number of pins each backend should respect, bast
+ * on shared_buffers and the maximum number of connections possible.
+ * That's very pessimistic, but outside toy-sized shared_buffers it should
+ * allow plenty of pins. Higher level code that pins non-trivial numbers
+ * of buffers should use LimitAdditionalPins() or GetAdditionalPinLimit()
+ * to stay under this limit.
+ */
+ MaxProportionalPins = NBuffers / (MaxBackends + NUM_AUXILIARY_PROCS);
+
memset(&PrivateRefCountArray, 0, sizeof(PrivateRefCountArray));
hash_ctl.keysize = sizeof(int32);
diff --git a/src/backend/storage/buffer/localbuf.c b/src/backend/storage/buffer/localbuf.c
index 80b83444eb2..5378ba84316 100644
--- a/src/backend/storage/buffer/localbuf.c
+++ b/src/backend/storage/buffer/localbuf.c
@@ -286,6 +286,22 @@ GetLocalVictimBuffer(void)
return BufferDescriptorGetBuffer(bufHdr);
}
+/* see GetSoftPinLimit() */
+uint32
+GetSoftLocalPinLimit(void)
+{
+ /* Every backend has its own temporary buffers, and can pin them all. */
+ return num_temp_buffers;
+}
+
+/* see GetAdditionalPinLimit() */
+uint32
+GetAdditionalLocalPinLimit(void)
+{
+ Assert(NLocalPinnedBuffers <= num_temp_buffers);
+ return num_temp_buffers - NLocalPinnedBuffers;
+}
+
/* see LimitAdditionalPins() */
void
LimitAdditionalLocalPins(uint32 *additional_pins)
--
2.48.1.76.g4e746b1a31.dirty
[text/x-diff] v2.6-0012-Respect-pin-limits-accurately-in-read_stream.c.patch (9.9K, ../../ge6nsuddurhpmll3xj22vucvqwp4agqz6ndtcf2mhyeydzarst@l75dman5x53p/13-v2.6-0012-Respect-pin-limits-accurately-in-read_stream.c.patch)
download | inline diff:
From 38e75e9ee69cac5ad03ed87fb815c1f5b8dce11b Mon Sep 17 00:00:00 2001
From: Thomas Munro <thomas.munro@gmail.com>
Date: Fri, 24 Jan 2025 23:52:53 +1300
Subject: [PATCH v2.6 12/34] Respect pin limits accurately in read_stream.c.
Read streams pin multiple buffers at once as required to combine I/O.
This also avoids having to unpin and repin later when issuing read-ahead
advice, and will be needed for proposed work that starts "real"
asynchronous I/O.
To avoid pinning too much of the buffer pool at once, we previously used
LimitAdditionalBuffers() to avoid pinning more than this backend's fair
share of the pool as a cap. The coding was a naive and only checked the
cap once at stream initialization.
This commit moves the check to the time of use with new bufmgr APIs from
an earlier commit, since the result might change later due to pins
acquired later outside this stream. No extra CPU cycles are added to
the all-buffered fast-path code (it only pins one buffer at a time), but
the I/O-starting path now re-checks the limit every time using simple
arithmetic.
In practice it was difficult to exceed the limit, but you could contrive
a workload to do it using multiple CURSORs and FETCHing from sequential
scans in round-robin fashion, so that each underlying stream computes
its limit before all the others have ramped up to their full look-ahead
distance. Therefore, no back-patch for now.
Per code review from Andres, in the course of his AIO work.
Reported-by: Andres Freund <andres@anarazel.de>
---
src/backend/storage/aio/read_stream.c | 111 ++++++++++++++++++++++----
1 file changed, 95 insertions(+), 16 deletions(-)
diff --git a/src/backend/storage/aio/read_stream.c b/src/backend/storage/aio/read_stream.c
index dee51fa85a9..6a39bd1d92d 100644
--- a/src/backend/storage/aio/read_stream.c
+++ b/src/backend/storage/aio/read_stream.c
@@ -115,6 +115,7 @@ struct ReadStream
int16 pinned_buffers;
int16 distance;
bool advice_enabled;
+ bool temporary;
/*
* One-block buffer to support 'ungetting' a block number, to resolve flow
@@ -274,7 +275,9 @@ read_stream_index_retreat(ReadStream *stream, int16 *index)
#endif
static void
-read_stream_start_pending_read(ReadStream *stream, bool suppress_advice)
+read_stream_start_pending_read(ReadStream *stream,
+ int16 buffer_limit,
+ bool suppress_advice)
{
bool need_wait;
int nblocks;
@@ -308,10 +311,14 @@ read_stream_start_pending_read(ReadStream *stream, bool suppress_advice)
else
flags = 0;
- /* We say how many blocks we want to read, but may be smaller on return. */
+ /*
+ * We say how many blocks we want to read, but may be smaller on return.
+ * On memory-constrained systems we may be also have to ask for a smaller
+ * read ourselves.
+ */
buffer_index = stream->next_buffer_index;
io_index = stream->next_io_index;
- nblocks = stream->pending_read_nblocks;
+ nblocks = Min(buffer_limit, stream->pending_read_nblocks);
need_wait = StartReadBuffers(&stream->ios[io_index].op,
&stream->buffers[buffer_index],
stream->pending_read_blocknum,
@@ -360,11 +367,60 @@ read_stream_start_pending_read(ReadStream *stream, bool suppress_advice)
stream->pending_read_nblocks -= nblocks;
}
+/*
+ * How many more buffers could we use, while respecting the soft limit?
+ */
+static int16
+read_stream_get_buffer_limit(ReadStream *stream)
+{
+ uint32 buffers;
+
+ /* Check how many local or shared pins we could acquire. */
+ if (stream->temporary)
+ buffers = GetAdditionalLocalPinLimit();
+ else
+ buffers = GetAdditionalPinLimit();
+
+ /*
+ * Each stream is always allowed to try to acquire one pin if it doesn't
+ * hold one already. This is needed to guarantee progress, and just like
+ * the simple ReadBuffer() operation in code that is not using this stream
+ * API, if a buffer can't be pinned we'll raise an error when trying to
+ * pin, ie the buffer pool is simply too small for the workload.
+ */
+ if (buffers == 0 && stream->pinned_buffers == 0)
+ return 1;
+
+ /*
+ * Otherwise, see how many additional pins the backend can currently pin,
+ * which may be zero. As above, this only guarantees that this backend
+ * won't use more than its fair share if all backends can respect the soft
+ * limit, not that a pin can actually be acquired without error.
+ */
+ return Min(buffers, INT16_MAX);
+}
+
static void
read_stream_look_ahead(ReadStream *stream, bool suppress_advice)
{
+ int16 buffer_limit;
+
+ /*
+ * Check how many pins we could acquire now. We do this here rather than
+ * pushing it down into read_stream_start_pending_read(), because it
+ * allows more flexibility in behavior when we run out of allowed pins.
+ * Currently the policy is to start an I/O when we've run out of allowed
+ * pins only if we have to to make progress, and otherwise to stop looking
+ * ahead until more pins become available, so that we don't start issuing
+ * a lot of smaller I/Os, prefering to build the largest ones we can. This
+ * choice is debatable, but it should only really come up with the buffer
+ * pool/connection ratio is very constrained.
+ */
+ buffer_limit = read_stream_get_buffer_limit(stream);
+
while (stream->ios_in_progress < stream->max_ios &&
- stream->pinned_buffers + stream->pending_read_nblocks < stream->distance)
+ stream->pinned_buffers + stream->pending_read_nblocks <
+ Min(stream->distance, buffer_limit))
{
BlockNumber blocknum;
int16 buffer_index;
@@ -372,7 +428,9 @@ read_stream_look_ahead(ReadStream *stream, bool suppress_advice)
if (stream->pending_read_nblocks == io_combine_limit)
{
- read_stream_start_pending_read(stream, suppress_advice);
+ read_stream_start_pending_read(stream, buffer_limit,
+ suppress_advice);
+ buffer_limit = read_stream_get_buffer_limit(stream);
suppress_advice = false;
continue;
}
@@ -406,11 +464,12 @@ read_stream_look_ahead(ReadStream *stream, bool suppress_advice)
/* We have to start the pending read before we can build another. */
while (stream->pending_read_nblocks > 0)
{
- read_stream_start_pending_read(stream, suppress_advice);
+ read_stream_start_pending_read(stream, buffer_limit, suppress_advice);
+ buffer_limit = read_stream_get_buffer_limit(stream);
suppress_advice = false;
- if (stream->ios_in_progress == stream->max_ios)
+ if (stream->ios_in_progress == stream->max_ios || buffer_limit == 0)
{
- /* And we've hit the limit. Rewind, and stop here. */
+ /* And we've hit a limit. Rewind, and stop here. */
read_stream_unget_block(stream, blocknum);
return;
}
@@ -426,16 +485,17 @@ read_stream_look_ahead(ReadStream *stream, bool suppress_advice)
* limit, preferring to give it another chance to grow to full
* io_combine_limit size once more buffers have been consumed. However,
* if we've already reached io_combine_limit, or we've reached the
- * distance limit and there isn't anything pinned yet, or the callback has
- * signaled end-of-stream, we start the read immediately.
+ * distance limit or buffer limit and there isn't anything pinned yet, or
+ * the callback has signaled end-of-stream, we start the read immediately.
*/
if (stream->pending_read_nblocks > 0 &&
(stream->pending_read_nblocks == io_combine_limit ||
- (stream->pending_read_nblocks == stream->distance &&
+ ((stream->pending_read_nblocks == stream->distance ||
+ stream->pending_read_nblocks == buffer_limit) &&
stream->pinned_buffers == 0) ||
stream->distance == 0) &&
stream->ios_in_progress < stream->max_ios)
- read_stream_start_pending_read(stream, suppress_advice);
+ read_stream_start_pending_read(stream, buffer_limit, suppress_advice);
}
/*
@@ -464,6 +524,7 @@ read_stream_begin_impl(int flags,
int max_ios;
int strategy_pin_limit;
uint32 max_pinned_buffers;
+ uint32 max_possible_buffer_limit;
Oid tablespace_id;
/*
@@ -511,12 +572,23 @@ read_stream_begin_impl(int flags,
strategy_pin_limit = GetAccessStrategyPinLimit(strategy);
max_pinned_buffers = Min(strategy_pin_limit, max_pinned_buffers);
- /* Don't allow this backend to pin more than its share of buffers. */
+ /*
+ * Also limit by the maximum possible number of pins we could be allowed
+ * to acquire according to bufmgr. We may not be able to use them all due
+ * to other pins held by this backend, but we'll enforce the dynamic limit
+ * later when starting I/O.
+ */
if (SmgrIsTemp(smgr))
- LimitAdditionalLocalPins(&max_pinned_buffers);
+ max_possible_buffer_limit = GetSoftLocalPinLimit();
else
- LimitAdditionalPins(&max_pinned_buffers);
- Assert(max_pinned_buffers > 0);
+ max_possible_buffer_limit = GetSoftPinLimit();
+ max_pinned_buffers = Min(max_pinned_buffers, max_possible_buffer_limit);
+
+ /*
+ * The soft limit might be zero on a system configured with more
+ * connections than buffers. We need at least one.
+ */
+ max_pinned_buffers = Max(1, max_pinned_buffers);
/*
* We need one extra entry for buffers and per-buffer data, because users
@@ -576,6 +648,7 @@ read_stream_begin_impl(int flags,
stream->callback = callback;
stream->callback_private_data = callback_private_data;
stream->buffered_blocknum = InvalidBlockNumber;
+ stream->temporary = SmgrIsTemp(smgr);
/*
* Skip the initial ramp-up phase if the caller says we're going to be
@@ -704,6 +777,12 @@ read_stream_next_buffer(ReadStream *stream, void **per_buffer_data)
* arbitrary I/O entry (they're all free). We don't have to
* adjust pinned_buffers because we're transferring one to caller
* but pinning one more.
+ *
+ * In the fast path we don't need to check the pin limit. We're
+ * always allowed at least one pin so that progress can be made,
+ * and that's all we need here. Although two pins are momentarily
+ * held at the same time, the model used here is that the stream
+ * holds only one, and the other now belongs to the caller.
*/
if (likely(!StartReadBuffer(&stream->ios[0].op,
&stream->buffers[oldest_buffer_index],
--
2.48.1.76.g4e746b1a31.dirty
[text/x-diff] v2.6-0013-Support-buffer-forwarding-in-read_stream.c.patch (8.9K, ../../ge6nsuddurhpmll3xj22vucvqwp4agqz6ndtcf2mhyeydzarst@l75dman5x53p/14-v2.6-0013-Support-buffer-forwarding-in-read_stream.c.patch)
download | inline diff:
From 533585f2fab0c29ac33d0629e5aca5d86fd013f7 Mon Sep 17 00:00:00 2001
From: Thomas Munro <thomas.munro@gmail.com>
Date: Thu, 30 Jan 2025 11:42:03 +1300
Subject: [PATCH v2.6 13/34] Support buffer forwarding in read_stream.c.
In preparation for a following change to the buffer manager, teach read
stream to keep track of buffers that were "forwarded" from one call to
StartReadBuffers() to the next.
Since StartReadBuffers() buffers argument will become an in/out
argument, we need to initialize the buffer queue entries with
InvalidBuffer. We don't want to do that up front, because we try to
keep stream initialization cheap and code that uses the fast path stays
in one single buffer queue element. Satisfy both goals by initializing
the queue incrementally on the first cycle.
---
src/backend/storage/aio/read_stream.c | 108 ++++++++++++++++++++++----
1 file changed, 94 insertions(+), 14 deletions(-)
diff --git a/src/backend/storage/aio/read_stream.c b/src/backend/storage/aio/read_stream.c
index 6a39bd1d92d..e58e0edf221 100644
--- a/src/backend/storage/aio/read_stream.c
+++ b/src/backend/storage/aio/read_stream.c
@@ -112,8 +112,10 @@ struct ReadStream
int16 ios_in_progress;
int16 queue_size;
int16 max_pinned_buffers;
+ int16 forwarded_buffers;
int16 pinned_buffers;
int16 distance;
+ int16 initialized_buffers;
bool advice_enabled;
bool temporary;
@@ -280,7 +282,9 @@ read_stream_start_pending_read(ReadStream *stream,
bool suppress_advice)
{
bool need_wait;
+ int requested_nblocks;
int nblocks;
+ int forwarded;
int flags;
int16 io_index;
int16 overflow;
@@ -312,13 +316,34 @@ read_stream_start_pending_read(ReadStream *stream,
flags = 0;
/*
- * We say how many blocks we want to read, but may be smaller on return.
- * On memory-constrained systems we may be also have to ask for a smaller
- * read ourselves.
+ * On buffer-constrained systems we may need to limit the I/O size by the
+ * available pin count.
*/
+ requested_nblocks = Min(buffer_limit, stream->pending_read_nblocks);
+ nblocks = requested_nblocks;
buffer_index = stream->next_buffer_index;
io_index = stream->next_io_index;
- nblocks = Min(buffer_limit, stream->pending_read_nblocks);
+
+ /*
+ * The first time around the queue we initialize it as we go, including
+ * the overflow zone, because otherwise the entries would appear as
+ * forwarded buffers. This avoids initializing the whole queue up front
+ * in cases where it is large but we don't ever use it due to the
+ * all-cached fast path or small scans.
+ */
+ while (stream->initialized_buffers < buffer_index + nblocks)
+ stream->buffers[stream->initialized_buffers++] = InvalidBuffer;
+
+ /*
+ * Start the I/O. Any buffers that are not InvalidBuffer will be
+ * interpreted as already pinned, forwarded by an earlier call to
+ * StartReadBuffers(), and must map to the expected blocks. The nblocks
+ * value may be smaller on return indicating the size of the I/O that
+ * could be started. Buffers beyond the output nblocks number may also
+ * have been pinned without starting I/O due to various edge cases. In
+ * that case we'll just leave them in the queue ahead of us, "forwarded"
+ * to the next call, avoiding the need to unpin/repin.
+ */
need_wait = StartReadBuffers(&stream->ios[io_index].op,
&stream->buffers[buffer_index],
stream->pending_read_blocknum,
@@ -347,16 +372,35 @@ read_stream_start_pending_read(ReadStream *stream,
stream->seq_blocknum = stream->pending_read_blocknum + nblocks;
}
+ /*
+ * How many pins were acquired but forwarded to the next call? These need
+ * to be passed to the next StartReadBuffers() call, or released if the
+ * stream ends early. We need the number for accounting purposes, since
+ * they are not counted in stream->pinned_buffers but we already hold
+ * them.
+ */
+ forwarded = 0;
+ while (nblocks + forwarded < requested_nblocks &&
+ stream->buffers[buffer_index + nblocks + forwarded] != InvalidBuffer)
+ forwarded++;
+ stream->forwarded_buffers = forwarded;
+
/*
* We gave a contiguous range of buffer space to StartReadBuffers(), but
- * we want it to wrap around at queue_size. Slide overflowing buffers to
- * the front of the array.
+ * we want it to wrap around at queue_size. Copy overflowing buffers to
+ * the front of the array where they'll be consumed, but also leave a copy
+ * in the overflow zone which the I/O operation has a pointer to (it needs
+ * a contiguous array). Both copies will be cleared when the buffers are
+ * handed to the consumer.
*/
- overflow = (buffer_index + nblocks) - stream->queue_size;
+ overflow = (buffer_index + nblocks + forwarded) - stream->queue_size;
if (overflow > 0)
- memmove(&stream->buffers[0],
- &stream->buffers[stream->queue_size],
- sizeof(stream->buffers[0]) * overflow);
+ {
+ Assert(overflow < stream->queue_size); /* can't overlap */
+ memcpy(&stream->buffers[0],
+ &stream->buffers[stream->queue_size],
+ sizeof(stream->buffers[0]) * overflow);
+ }
/* Move to the location of start of next read. */
read_stream_index_advance_n(stream, &buffer_index, nblocks);
@@ -381,6 +425,15 @@ read_stream_get_buffer_limit(ReadStream *stream)
else
buffers = GetAdditionalPinLimit();
+ /*
+ * If we already have some forwarded buffers, we can certainly use those.
+ * They are already pinned, and are mapped to the starting blocks of the
+ * pending read, they just don't have any I/O started yet and are not
+ * counted in stream->pinned_buffers.
+ */
+ Assert(stream->forwarded_buffers <= stream->pending_read_nblocks);
+ buffers += stream->forwarded_buffers;
+
/*
* Each stream is always allowed to try to acquire one pin if it doesn't
* hold one already. This is needed to guarantee progress, and just like
@@ -389,7 +442,7 @@ read_stream_get_buffer_limit(ReadStream *stream)
* pin, ie the buffer pool is simply too small for the workload.
*/
if (buffers == 0 && stream->pinned_buffers == 0)
- return 1;
+ buffers = 1;
/*
* Otherwise, see how many additional pins the backend can currently pin,
@@ -755,10 +808,12 @@ read_stream_next_buffer(ReadStream *stream, void **per_buffer_data)
/* Fast path assumptions. */
Assert(stream->ios_in_progress == 0);
+ Assert(stream->forwarded_buffers == 0);
Assert(stream->pinned_buffers == 1);
Assert(stream->distance == 1);
Assert(stream->pending_read_nblocks == 0);
Assert(stream->per_buffer_data_size == 0);
+ Assert(stream->initialized_buffers > stream->oldest_buffer_index);
/* We're going to return the buffer we pinned last time. */
oldest_buffer_index = stream->oldest_buffer_index;
@@ -807,6 +862,7 @@ read_stream_next_buffer(ReadStream *stream, void **per_buffer_data)
stream->distance = 0;
stream->oldest_buffer_index = stream->next_buffer_index;
stream->pinned_buffers = 0;
+ stream->buffers[oldest_buffer_index] = InvalidBuffer;
}
stream->fast_path = false;
@@ -891,10 +947,15 @@ read_stream_next_buffer(ReadStream *stream, void **per_buffer_data)
}
}
-#ifdef CLOBBER_FREED_MEMORY
- /* Clobber old buffer for debugging purposes. */
+ /*
+ * We must zap this queue entry, or else it would appear as a forwarded
+ * buffer. If it's potentially in the overflow zone (ie it wrapped around
+ * the queue), also zap that copy.
+ */
stream->buffers[oldest_buffer_index] = InvalidBuffer;
-#endif
+ if (oldest_buffer_index < io_combine_limit - 1)
+ stream->buffers[stream->queue_size + oldest_buffer_index] =
+ InvalidBuffer;
#if defined(CLOBBER_FREED_MEMORY) || defined(USE_VALGRIND)
@@ -937,6 +998,7 @@ read_stream_next_buffer(ReadStream *stream, void **per_buffer_data)
#ifndef READ_STREAM_DISABLE_FAST_PATH
/* See if we can take the fast path for all-cached scans next time. */
if (stream->ios_in_progress == 0 &&
+ stream->forwarded_buffers == 0 &&
stream->pinned_buffers == 1 &&
stream->distance == 1 &&
stream->pending_read_nblocks == 0 &&
@@ -972,6 +1034,7 @@ read_stream_next_block(ReadStream *stream, BufferAccessStrategy *strategy)
void
read_stream_reset(ReadStream *stream)
{
+ int16 index;
Buffer buffer;
/* Stop looking ahead. */
@@ -985,6 +1048,23 @@ read_stream_reset(ReadStream *stream)
while ((buffer = read_stream_next_buffer(stream, NULL)) != InvalidBuffer)
ReleaseBuffer(buffer);
+ /* Unpin any unused forwarded buffers. */
+ index = stream->next_buffer_index;
+ while (index < stream->initialized_buffers &&
+ (buffer = stream->buffers[index]) != InvalidBuffer)
+ {
+ Assert(stream->forwarded_buffers > 0);
+ stream->forwarded_buffers--;
+ ReleaseBuffer(buffer);
+
+ stream->buffers[index] = InvalidBuffer;
+ if (index < io_combine_limit - 1)
+ stream->buffers[stream->queue_size + index] = InvalidBuffer;
+
+ read_stream_index_advance(stream, &index);
+ }
+
+ Assert(stream->forwarded_buffers == 0);
Assert(stream->pinned_buffers == 0);
Assert(stream->ios_in_progress == 0);
--
2.48.1.76.g4e746b1a31.dirty
[text/x-diff] v2.6-0014-Support-buffer-forwarding-in-StartReadBuffers.patch (10.5K, ../../ge6nsuddurhpmll3xj22vucvqwp4agqz6ndtcf2mhyeydzarst@l75dman5x53p/15-v2.6-0014-Support-buffer-forwarding-in-StartReadBuffers.patch)
download | inline diff:
From 457213b9b3b2b4476dee5ba10cc046dcc22633fc Mon Sep 17 00:00:00 2001
From: Thomas Munro <thomas.munro@gmail.com>
Date: Mon, 10 Feb 2025 21:55:40 +1300
Subject: [PATCH v2.6 14/34] Support buffer forwarding in StartReadBuffers().
Sometimes we have to perform a short read because we hit a cached block
that ends a contiguous run of blocks requiring I/O. We don't want
StartReadBuffers() to have to start more than one I/O, so we stop there.
We also don't want to have to unpin the cached block (and repin it
later), so previously we'd silently pretend the hit was part of the I/O,
and just leave it out of the read from disk. Now, we'll "forward" it to
the next call. We still write it to the buffers[] array for the caller
to pass back to us later, but it's not included in *nblocks.
This policy means that we no longer mix hits and misses in a single
operation's results, so we avoid the requirement to call
WaitReadBuffers(), which might stall, before the caller can make use of
the hits. The caller will get the hit in the next call instead, and
know that it doesn't have to wait. That's important for later work on
out-of-order read streams that minimize I/O stalls.
This also makes life easier for proposed work on true AIO, which
occasionally needs to split a large I/O after pinning all the buffers,
while the current coding only ever forwards a single bookending hit.
This API is natural for read_stream.c: it just leaves forwarded buffers
where they are in its circular queue, where the next call will pick them
up and continue, minimizing pin churn.
If we ever think of a good reason to disable this feature, i.e. for
other users of StartReadBuffers() that don't want to deal with forwarded
buffers, then we could add a flag for that. For now read_steam.c is the
only user.
---
src/include/storage/bufmgr.h | 1 -
src/backend/storage/buffer/bufmgr.c | 128 ++++++++++++++++++++--------
2 files changed, 91 insertions(+), 38 deletions(-)
diff --git a/src/include/storage/bufmgr.h b/src/include/storage/bufmgr.h
index 597ecb97897..4a035f59a7d 100644
--- a/src/include/storage/bufmgr.h
+++ b/src/include/storage/bufmgr.h
@@ -130,7 +130,6 @@ struct ReadBuffersOperation
BlockNumber blocknum;
int flags;
int16 nblocks;
- int16 io_buffers_len;
};
typedef struct ReadBuffersOperation ReadBuffersOperation;
diff --git a/src/backend/storage/buffer/bufmgr.c b/src/backend/storage/buffer/bufmgr.c
index 11c146763db..75a928e802a 100644
--- a/src/backend/storage/buffer/bufmgr.c
+++ b/src/backend/storage/buffer/bufmgr.c
@@ -1257,10 +1257,10 @@ StartReadBuffersImpl(ReadBuffersOperation *operation,
Buffer *buffers,
BlockNumber blockNum,
int *nblocks,
- int flags)
+ int flags,
+ bool allow_forwarding)
{
int actual_nblocks = *nblocks;
- int io_buffers_len = 0;
int maxcombine = 0;
Assert(*nblocks > 0);
@@ -1270,30 +1270,80 @@ StartReadBuffersImpl(ReadBuffersOperation *operation,
{
bool found;
- buffers[i] = PinBufferForBlock(operation->rel,
- operation->smgr,
- operation->persistence,
- operation->forknum,
- blockNum + i,
- operation->strategy,
- &found);
+ if (allow_forwarding && buffers[i] != InvalidBuffer)
+ {
+ BufferDesc *bufHdr;
+
+ /*
+ * This is a buffer that was pinned by an earlier call to
+ * StartReadBuffers(), but couldn't be handled in one operation at
+ * that time. The operation was split, and the caller has passed
+ * an already pinned buffer back to us to handle the rest of the
+ * operation. It must continue at the expected block number.
+ */
+ Assert(BufferGetBlockNumber(buffers[i]) == blockNum + i);
+
+ /*
+ * It might be an already valid buffer (a hit) that followed the
+ * final contiguous block of an earlier I/O (a miss) marking the
+ * end of it, or a buffer that some other backend has since made
+ * valid by performing the I/O for us, in which case we can handle
+ * it as a hit now. It is safe to check for a BM_VALID flag with
+ * a relaxed load, because we got a fresh view of it while pinning
+ * it in the previous call.
+ *
+ * On the other hand if we don't see BM_VALID yet, it must be an
+ * I/O that was split by the previous call and we need to try to
+ * start a new I/O from this block. We're also racing against any
+ * other backend that might start the I/O or even manage to mark
+ * it BM_VALID after this check, BM_VALID after this check, but
+ * StartBufferIO() will handle those cases.
+ */
+ if (BufferIsLocal(buffers[i]))
+ bufHdr = GetLocalBufferDescriptor(-buffers[i] - 1);
+ else
+ bufHdr = GetBufferDescriptor(buffers[i] - 1);
+ found = pg_atomic_read_u32(&bufHdr->state) & BM_VALID;
+ }
+ else
+ {
+ buffers[i] = PinBufferForBlock(operation->rel,
+ operation->smgr,
+ operation->persistence,
+ operation->forknum,
+ blockNum + i,
+ operation->strategy,
+ &found);
+ }
if (found)
{
/*
- * Terminate the read as soon as we get a hit. It could be a
- * single buffer hit, or it could be a hit that follows a readable
- * range. We don't want to create more than one readable range,
- * so we stop here.
+ * We have a hit. If it's the first block in the requested range,
+ * we can return it immediately and report that WaitReadBuffers()
+ * does not need to be called. If the initial value of *nblocks
+ * was larger, the caller will have to call again for the rest.
*/
- actual_nblocks = i + 1;
+ if (i == 0)
+ {
+ *nblocks = 1;
+ return false;
+ }
+
+ /*
+ * Otherwise we already have an I/O to perform, but this block
+ * can't be included as it is already valid. Split the I/O here.
+ * There may or may not be more blocks requiring I/O after this
+ * one, we haven't checked, but it can't be contiguous with this
+ * hit in the way. We'll leave this buffer pinned, forwarding it
+ * to the next call, avoiding the need to unpin it here and re-pin
+ * it in the next call.
+ */
+ actual_nblocks = i;
break;
}
else
{
- /* Extend the readable range to cover this block. */
- io_buffers_len++;
-
/*
* Check how many blocks we can cover with the same IO. The smgr
* implementation might e.g. be limited due to a segment boundary.
@@ -1314,15 +1364,11 @@ StartReadBuffersImpl(ReadBuffersOperation *operation,
}
*nblocks = actual_nblocks;
- if (likely(io_buffers_len == 0))
- return false;
-
/* Populate information needed for I/O. */
operation->buffers = buffers;
operation->blocknum = blockNum;
operation->flags = flags;
operation->nblocks = actual_nblocks;
- operation->io_buffers_len = io_buffers_len;
if (flags & READ_BUFFERS_ISSUE_ADVICE)
{
@@ -1337,7 +1383,7 @@ StartReadBuffersImpl(ReadBuffersOperation *operation,
smgrprefetch(operation->smgr,
operation->forknum,
blockNum,
- operation->io_buffers_len);
+ actual_nblocks);
}
/* Indicate that WaitReadBuffers() should be called. */
@@ -1351,11 +1397,21 @@ StartReadBuffersImpl(ReadBuffersOperation *operation,
* actual number, which may be fewer than requested. Caller sets some of the
* members of operation; see struct definition.
*
+ * The initial contents of the elements of buffers up to *nblocks should
+ * either be InvalidBuffer or an already-pinned buffer that was left by an
+ * preceding call to StartReadBuffers() that had to be split. On return, some
+ * elements of buffers may hold pinned buffers beyond the number indicated by
+ * the updated value of *nblocks. Operations are split on boundaries known to
+ * smgr (eg md.c segment boundaries that require crossing into a different
+ * underlying file), or when already cached blocks are found in the buffer
+ * that prevent the formation of a contiguous read.
+ *
* If false is returned, no I/O is necessary. If true is returned, one I/O
* has been started, and WaitReadBuffers() must be called with the same
* operation object before the buffers are accessed. Along with the operation
* object, the caller-supplied array of buffers must remain valid until
- * WaitReadBuffers() is called.
+ * WaitReadBuffers() is called, and any forwarded buffers must also be
+ * preserved for a future call unless explicitly released.
*
* Currently the I/O is only started with optional operating system advice if
* requested by the caller with READ_BUFFERS_ISSUE_ADVICE, and the real I/O
@@ -1369,13 +1425,18 @@ StartReadBuffers(ReadBuffersOperation *operation,
int *nblocks,
int flags)
{
- return StartReadBuffersImpl(operation, buffers, blockNum, nblocks, flags);
+ return StartReadBuffersImpl(operation, buffers, blockNum, nblocks, flags,
+ true /* expect forwarded buffers */ );
}
/*
* Single block version of the StartReadBuffers(). This might save a few
* instructions when called from another translation unit, because it is
* specialized for nblocks == 1.
+ *
+ * This version does not support "forwarded" buffers: they cannot be created
+ * by reading only one block, and the current contents of *buffer is ignored
+ * on entry.
*/
bool
StartReadBuffer(ReadBuffersOperation *operation,
@@ -1386,7 +1447,8 @@ StartReadBuffer(ReadBuffersOperation *operation,
int nblocks = 1;
bool result;
- result = StartReadBuffersImpl(operation, buffer, blocknum, &nblocks, flags);
+ result = StartReadBuffersImpl(operation, buffer, blocknum, &nblocks, flags,
+ false /* single block, no forwarding */ );
Assert(nblocks == 1); /* single block can't be short */
return result;
@@ -1416,24 +1478,16 @@ WaitReadBuffers(ReadBuffersOperation *operation)
IOObject io_object;
char persistence;
- /*
- * Currently operations are only allowed to include a read of some range,
- * with an optional extra buffer that is already pinned at the end. So
- * nblocks can be at most one more than io_buffers_len.
- */
- Assert((operation->nblocks == operation->io_buffers_len) ||
- (operation->nblocks == operation->io_buffers_len + 1));
-
/* Find the range of the physical read we need to perform. */
- nblocks = operation->io_buffers_len;
- if (nblocks == 0)
- return; /* nothing to do */
-
+ nblocks = operation->nblocks;
buffers = &operation->buffers[0];
blocknum = operation->blocknum;
forknum = operation->forknum;
persistence = operation->persistence;
+ Assert(nblocks > 0);
+ Assert(nblocks <= MAX_IO_COMBINE_LIMIT);
+
if (persistence == RELPERSISTENCE_TEMP)
{
io_context = IOCONTEXT_NORMAL;
--
2.48.1.76.g4e746b1a31.dirty
[text/x-diff] v2.6-0015-tests-Expand-temp-table-tests-to-some-pin-relat.patch (9.6K, ../../ge6nsuddurhpmll3xj22vucvqwp4agqz6ndtcf2mhyeydzarst@l75dman5x53p/16-v2.6-0015-tests-Expand-temp-table-tests-to-some-pin-relat.patch)
download | inline diff:
From b5f9a66b263506e4981d1bbaaf955336cba16729 Mon Sep 17 00:00:00 2001
From: Andres Freund <andres@anarazel.de>
Date: Tue, 4 Mar 2025 11:07:47 -0500
Subject: [PATCH v2.6 15/34] tests: Expand temp table tests to some pin related
matters
Author:
Reviewed-By:
Discussion: https://postgr.es/m/
Backpatch:
---
src/test/regress/expected/temp.out | 156 +++++++++++++++++++++++++++++
src/test/regress/parallel_schedule | 2 +-
src/test/regress/sql/temp.sql | 107 ++++++++++++++++++++
3 files changed, 264 insertions(+), 1 deletion(-)
diff --git a/src/test/regress/expected/temp.out b/src/test/regress/expected/temp.out
index 2a246a7e123..91fe519b1cc 100644
--- a/src/test/regress/expected/temp.out
+++ b/src/test/regress/expected/temp.out
@@ -410,3 +410,159 @@ SELECT current_schema() ~ 'pg_temp' AS is_temp_schema;
PREPARE TRANSACTION 'twophase_search';
ERROR: cannot PREPARE a transaction that has operated on temporary objects
+-- Tests to verify we recover correctly from exhausting buffer pins and
+-- related matters.
+-- use lower possible buffer limit to make the test cheaper
+\c
+SET temp_buffers = 100;
+CREATE TEMPORARY TABLE test_temp(a int not null unique, b TEXT not null, cnt int not null);
+INSERT INTO test_temp SELECT generate_series(1, 10000) as id, repeat('a', 200), 0;
+-- should be at least 2x as large than temp_buffers
+SELECT pg_relation_size('test_temp') / current_setting('block_size')::int8 > 200;
+ ?column?
+----------
+ t
+(1 row)
+
+-- Don't want cursor names and plpgsql function lines in the error messages
+\set VERBOSITY terse
+/* helper function to create cursors for each page in [p_start, p_end] */
+CREATE FUNCTION test_temp_pin(p_start int, p_end int)
+RETURNS void
+LANGUAGE plpgsql
+AS $f$
+ DECLARE
+ cursorname text;
+ query text;
+ BEGIN
+ FOR i IN p_start..p_end LOOP
+ cursorname = 'c_'||i;
+ query = format($q$DECLARE %I CURSOR FOR SELECT ctid FROM test_temp WHERE ctid >= '( %s, 1)'::tid $q$, cursorname, i);
+ EXECUTE query;
+ EXECUTE 'FETCH NEXT FROM '||cursorname;
+ -- for test development
+ -- RAISE NOTICE '%: %', cursorname, query;
+ END LOOP;
+ END;
+$f$;
+-- Test overflow of temp table buffers is handled correctly
+BEGIN;
+-- should work, below max
+SELECT test_temp_pin(0, 9);
+ test_temp_pin
+---------------
+
+(1 row)
+
+-- should fail, too many buffers pinned
+SELECT test_temp_pin(10, 105);
+ERROR: no empty local buffer available
+ROLLBACK;
+BEGIN;
+-- have some working cursors to test after errors
+SELECT test_temp_pin(0, 9);
+ test_temp_pin
+---------------
+
+(1 row)
+
+FETCH NEXT FROM c_3;
+ ctid
+-------
+ (3,2)
+(1 row)
+
+-- exhaust buffer pins in subtrans, check things work after
+SAVEPOINT rescue_me;
+SELECT test_temp_pin(10, 105);
+ERROR: no empty local buffer available
+ROLLBACK TO SAVEPOINT rescue_me;
+-- pre-subtrans cursors continue to work
+FETCH NEXT FROM c_3;
+ ctid
+-------
+ (3,3)
+(1 row)
+
+-- new cursors with pins can be created after subtrans rollback
+SELECT test_temp_pin(10, 95);
+ test_temp_pin
+---------------
+
+(1 row)
+
+-- Check that read streams deal with lower number of pins available
+SELECT count(*), max(a) max_a, min(a) min_a, max(cnt) max_cnt FROM test_temp;
+ count | max_a | min_a | max_cnt
+-------+-------+-------+---------
+ 10000 | 10000 | 1 | 0
+(1 row)
+
+ROLLBACK;
+-- Check that temp tables with existing cursors can't be dropped.
+BEGIN;
+SELECT test_temp_pin(0, 1);
+ test_temp_pin
+---------------
+
+(1 row)
+
+DROP TABLE test_temp;
+ERROR: cannot DROP TABLE "test_temp" because it is being used by active queries in this session
+COMMIT;
+-- Check that temp tables with existing cursors can't be dropped.
+BEGIN;
+SELECT test_temp_pin(0, 1);
+ test_temp_pin
+---------------
+
+(1 row)
+
+TRUNCATE test_temp;
+ERROR: cannot TRUNCATE "test_temp" because it is being used by active queries in this session
+COMMIT;
+-- Check that temp tables that are dropped in transaction that's rolled back
+-- preserve buffer contents
+SELECT count(*), max(a) max_a, min(a) min_a, max(cnt) max_cnt FROM test_temp;
+ count | max_a | min_a | max_cnt
+-------+-------+-------+---------
+ 10000 | 10000 | 1 | 0
+(1 row)
+
+INSERT INTO test_temp(a, b, cnt) VALUES (-1, '', 0);
+BEGIN;
+INSERT INTO test_temp(a, b, cnt) VALUES (-2, '', 0);
+DROP TABLE test_temp;
+ROLLBACK;
+SELECT count(*), max(a) max_a, min(a) min_a, max(cnt) max_cnt FROM test_temp;
+ count | max_a | min_a | max_cnt
+-------+-------+-------+---------
+ 10001 | 10000 | -1 | 0
+(1 row)
+
+-- Check that temp table drop is transactional and preserves dirty
+-- buffer contents
+UPDATE test_temp SET cnt = cnt + 1 WHERE a = -1;
+BEGIN;
+DROP TABLE test_temp;
+ROLLBACK;
+SELECT count(*), max(a) max_a, min(a) min_a, max(cnt) max_cnt FROM test_temp;
+ count | max_a | min_a | max_cnt
+-------+-------+-------+---------
+ 10001 | 10000 | -1 | 1
+(1 row)
+
+-- Check that temp table truncation is transactional and preserves dirty
+-- buffer contents
+UPDATE test_temp SET cnt = cnt + 1 WHERE a = -1;
+BEGIN;
+TRUNCATE test_temp;
+ROLLBACK;
+SELECT count(*), max(a) max_a, min(a) min_a, max(cnt) max_cnt FROM test_temp;
+ count | max_a | min_a | max_cnt
+-------+-------+-------+---------
+ 10001 | 10000 | -1 | 2
+(1 row)
+
+-- cleanup
+DROP FUNCTION test_temp_pin(int, int);
diff --git a/src/test/regress/parallel_schedule b/src/test/regress/parallel_schedule
index 37b6d21e1f9..0a35f2f8f6a 100644
--- a/src/test/regress/parallel_schedule
+++ b/src/test/regress/parallel_schedule
@@ -108,7 +108,7 @@ test: json jsonb json_encoding jsonpath jsonpath_encoding jsonb_jsonpath sqljson
# ----------
# Another group of parallel tests
# with depends on create_misc
-# NB: temp.sql does a reconnect which transiently uses 2 connections,
+# NB: temp.sql does reconnects which transiently use 2 connections,
# so keep this parallel group to at most 19 tests
# ----------
test: plancache limit plpgsql copy2 temp domain rangefuncs prepare conversion truncate alter_table sequence polymorphism rowtypes returning largeobject with xml
diff --git a/src/test/regress/sql/temp.sql b/src/test/regress/sql/temp.sql
index 2a487a1ef7f..12091f968de 100644
--- a/src/test/regress/sql/temp.sql
+++ b/src/test/regress/sql/temp.sql
@@ -311,3 +311,110 @@ SET search_path TO 'pg_temp';
BEGIN;
SELECT current_schema() ~ 'pg_temp' AS is_temp_schema;
PREPARE TRANSACTION 'twophase_search';
+
+
+-- Tests to verify we recover correctly from exhausting buffer pins and
+-- related matters.
+
+-- use lower possible buffer limit to make the test cheaper
+\c
+SET temp_buffers = 100;
+
+CREATE TEMPORARY TABLE test_temp(a int not null unique, b TEXT not null, cnt int not null);
+INSERT INTO test_temp SELECT generate_series(1, 10000) as id, repeat('a', 200), 0;
+-- should be at least 2x as large than temp_buffers
+SELECT pg_relation_size('test_temp') / current_setting('block_size')::int8 > 200;
+
+-- Don't want cursor names and plpgsql function lines in the error messages
+\set VERBOSITY terse
+
+/* helper function to create cursors for each page in [p_start, p_end] */
+CREATE FUNCTION test_temp_pin(p_start int, p_end int)
+RETURNS void
+LANGUAGE plpgsql
+AS $f$
+ DECLARE
+ cursorname text;
+ query text;
+ BEGIN
+ FOR i IN p_start..p_end LOOP
+ cursorname = 'c_'||i;
+ query = format($q$DECLARE %I CURSOR FOR SELECT ctid FROM test_temp WHERE ctid >= '( %s, 1)'::tid $q$, cursorname, i);
+ EXECUTE query;
+ EXECUTE 'FETCH NEXT FROM '||cursorname;
+ -- for test development
+ -- RAISE NOTICE '%: %', cursorname, query;
+ END LOOP;
+ END;
+$f$;
+
+
+-- Test overflow of temp table buffers is handled correctly
+BEGIN;
+-- should work, below max
+SELECT test_temp_pin(0, 9);
+-- should fail, too many buffers pinned
+SELECT test_temp_pin(10, 105);
+ROLLBACK;
+
+BEGIN;
+-- have some working cursors to test after errors
+SELECT test_temp_pin(0, 9);
+FETCH NEXT FROM c_3;
+-- exhaust buffer pins in subtrans, check things work after
+SAVEPOINT rescue_me;
+SELECT test_temp_pin(10, 105);
+ROLLBACK TO SAVEPOINT rescue_me;
+-- pre-subtrans cursors continue to work
+FETCH NEXT FROM c_3;
+
+-- new cursors with pins can be created after subtrans rollback
+SELECT test_temp_pin(10, 95);
+
+-- Check that read streams deal with lower number of pins available
+SELECT count(*), max(a) max_a, min(a) min_a, max(cnt) max_cnt FROM test_temp;
+
+ROLLBACK;
+
+
+-- Check that temp tables with existing cursors can't be dropped.
+BEGIN;
+SELECT test_temp_pin(0, 1);
+DROP TABLE test_temp;
+COMMIT;
+
+-- Check that temp tables with existing cursors can't be dropped.
+BEGIN;
+SELECT test_temp_pin(0, 1);
+TRUNCATE test_temp;
+COMMIT;
+
+-- Check that temp tables that are dropped in transaction that's rolled back
+-- preserve buffer contents
+SELECT count(*), max(a) max_a, min(a) min_a, max(cnt) max_cnt FROM test_temp;
+INSERT INTO test_temp(a, b, cnt) VALUES (-1, '', 0);
+BEGIN;
+INSERT INTO test_temp(a, b, cnt) VALUES (-2, '', 0);
+DROP TABLE test_temp;
+ROLLBACK;
+SELECT count(*), max(a) max_a, min(a) min_a, max(cnt) max_cnt FROM test_temp;
+
+-- Check that temp table drop is transactional and preserves dirty
+-- buffer contents
+UPDATE test_temp SET cnt = cnt + 1 WHERE a = -1;
+BEGIN;
+DROP TABLE test_temp;
+ROLLBACK;
+SELECT count(*), max(a) max_a, min(a) min_a, max(cnt) max_cnt FROM test_temp;
+
+-- Check that temp table truncation is transactional and preserves dirty
+-- buffer contents
+UPDATE test_temp SET cnt = cnt + 1 WHERE a = -1;
+BEGIN;
+TRUNCATE test_temp;
+ROLLBACK;
+SELECT count(*), max(a) max_a, min(a) min_a, max(cnt) max_cnt FROM test_temp;
+
+
+-- cleanup
+DROP FUNCTION test_temp_pin(int, int);
--
2.48.1.76.g4e746b1a31.dirty
[text/x-diff] v2.6-0016-localbuf-Fix-dangerous-coding-pattern-in-GetLoc.patch (2.5K, ../../ge6nsuddurhpmll3xj22vucvqwp4agqz6ndtcf2mhyeydzarst@l75dman5x53p/17-v2.6-0016-localbuf-Fix-dangerous-coding-pattern-in-GetLoc.patch)
download | inline diff:
From 37a8eee547cfaeee83c68f771a7faf05f83e4422 Mon Sep 17 00:00:00 2001
From: Andres Freund <andres@anarazel.de>
Date: Sun, 9 Mar 2025 20:11:51 -0400
Subject: [PATCH v2.6 16/34] localbuf: Fix dangerous coding pattern in
GetLocalVictimBuffer()
If PinLocalBuffer() were to modify the buf_state, the buf_state in
GetLocalVictimBuffer() would be out of date. Currently that does not happen,
as PinLocalBuffer() only modifies the buf_state if adjust_usagecount=true and
GetLocalVictimBuffer() passes false.
However, it's easy to make this not the case anymore - it cost me a few hours
to debug the consequences.
The minimal fix would be to just refetch the buf_state after after calling
PinLocalBuffer(), but the same danger exists in later parts of the
function. So instead refetch declare buf_state in narrower scopes and always
refetch.
I apparently broke this in 794f2594479.
Author:
Reviewed-by:
Discussion: https://postgr.es/m/
Backpatch:
---
src/backend/storage/buffer/localbuf.c | 9 +++++----
1 file changed, 5 insertions(+), 4 deletions(-)
diff --git a/src/backend/storage/buffer/localbuf.c b/src/backend/storage/buffer/localbuf.c
index 5378ba84316..d3c869f53f9 100644
--- a/src/backend/storage/buffer/localbuf.c
+++ b/src/backend/storage/buffer/localbuf.c
@@ -178,7 +178,6 @@ GetLocalVictimBuffer(void)
{
int victim_bufid;
int trycounter;
- uint32 buf_state;
BufferDesc *bufHdr;
ResourceOwnerEnlarge(CurrentResourceOwner);
@@ -199,7 +198,7 @@ GetLocalVictimBuffer(void)
if (LocalRefCount[victim_bufid] == 0)
{
- buf_state = pg_atomic_read_u32(&bufHdr->state);
+ uint32 buf_state = pg_atomic_read_u32(&bufHdr->state);
if (BUF_STATE_GET_USAGECOUNT(buf_state) > 0)
{
@@ -233,8 +232,9 @@ GetLocalVictimBuffer(void)
* this buffer is not referenced but it might still be dirty. if that's
* the case, write it out before reusing it!
*/
- if (buf_state & BM_DIRTY)
+ if (pg_atomic_read_u32(&bufHdr->state) & BM_DIRTY)
{
+ uint32 buf_state = pg_atomic_read_u32(&bufHdr->state);
instr_time io_start;
SMgrRelation oreln;
Page localpage = (char *) LocalBufHdrGetBlock(bufHdr);
@@ -267,8 +267,9 @@ GetLocalVictimBuffer(void)
/*
* Remove the victim buffer from the hashtable and mark as invalid.
*/
- if (buf_state & BM_TAG_VALID)
+ if (pg_atomic_read_u32(&bufHdr->state) & BM_TAG_VALID)
{
+ uint32 buf_state = pg_atomic_read_u32(&bufHdr->state);
LocalBufferLookupEnt *hresult;
hresult = (LocalBufferLookupEnt *)
--
2.48.1.76.g4e746b1a31.dirty
[text/x-diff] v2.6-0017-localbuf-Introduce-InvalidateLocalBuffer.patch (5.6K, ../../ge6nsuddurhpmll3xj22vucvqwp4agqz6ndtcf2mhyeydzarst@l75dman5x53p/18-v2.6-0017-localbuf-Introduce-InvalidateLocalBuffer.patch)
download | inline diff:
From 4ec576bd7e51fb0c0404e86e68c68473e597235b Mon Sep 17 00:00:00 2001
From: Andres Freund <andres@anarazel.de>
Date: Sun, 9 Mar 2025 20:25:22 -0400
Subject: [PATCH v2.6 17/34] localbuf: Introduce InvalidateLocalBuffer()
There were three copies of this code previously, two of them
identical. There's no good reason for that.
Author:
Reviewed-by:
Discussion: https://postgr.es/m/
Backpatch:
---
src/backend/storage/buffer/localbuf.c | 91 +++++++++++++--------------
1 file changed, 43 insertions(+), 48 deletions(-)
diff --git a/src/backend/storage/buffer/localbuf.c b/src/backend/storage/buffer/localbuf.c
index d3c869f53f9..4444228d645 100644
--- a/src/backend/storage/buffer/localbuf.c
+++ b/src/backend/storage/buffer/localbuf.c
@@ -56,6 +56,7 @@ static int NLocalPinnedBuffers = 0;
static void InitLocalBuffers(void);
static Block GetLocalBufferStorage(void);
static Buffer GetLocalVictimBuffer(void);
+static void InvalidateLocalBuffer(BufferDesc *bufHdr, bool check_unreferenced);
/*
@@ -269,17 +270,7 @@ GetLocalVictimBuffer(void)
*/
if (pg_atomic_read_u32(&bufHdr->state) & BM_TAG_VALID)
{
- uint32 buf_state = pg_atomic_read_u32(&bufHdr->state);
- LocalBufferLookupEnt *hresult;
-
- hresult = (LocalBufferLookupEnt *)
- hash_search(LocalBufHash, &bufHdr->tag, HASH_REMOVE, NULL);
- if (!hresult) /* shouldn't happen */
- elog(ERROR, "local buffer hash table corrupted");
- /* mark buffer invalid just in case hash insert fails */
- ClearBufferTag(&bufHdr->tag);
- buf_state &= ~(BUF_FLAG_MASK | BUF_USAGECOUNT_MASK);
- pg_atomic_unlocked_write_u32(&bufHdr->state, buf_state);
+ InvalidateLocalBuffer(bufHdr, false);
pgstat_count_io_op(IOOBJECT_TEMP_RELATION, IOCONTEXT_NORMAL, IOOP_EVICT, 1, 0);
}
@@ -492,6 +483,45 @@ MarkLocalBufferDirty(Buffer buffer)
pg_atomic_unlocked_write_u32(&bufHdr->state, buf_state);
}
+/*
+ * InvalidateLocalBuffer -- mark a local buffer invalid.
+ *
+ * If check_unreferenced is true, error out if the buffer is still
+ * used. Passing false is appropriate when redesignating the buffer instead
+ * dropping it.
+ *
+ * See also InvalidateBuffer().
+ */
+static void
+InvalidateLocalBuffer(BufferDesc *bufHdr, bool check_unreferenced)
+{
+ Buffer buffer = BufferDescriptorGetBuffer(bufHdr);
+ int bufid = -buffer - 1;
+ uint32 buf_state;
+ LocalBufferLookupEnt *hresult;
+
+ buf_state = pg_atomic_read_u32(&bufHdr->state);
+
+ if (check_unreferenced && LocalRefCount[bufid] != 0)
+ elog(ERROR, "block %u of %s is still referenced (local %u)",
+ bufHdr->tag.blockNum,
+ relpathbackend(BufTagGetRelFileLocator(&bufHdr->tag),
+ MyProcNumber,
+ BufTagGetForkNum(&bufHdr->tag)).str,
+ LocalRefCount[bufid]);
+
+ /* Remove entry from hashtable */
+ hresult = (LocalBufferLookupEnt *)
+ hash_search(LocalBufHash, &bufHdr->tag, HASH_REMOVE, NULL);
+ if (!hresult) /* shouldn't happen */
+ elog(ERROR, "local buffer hash table corrupted");
+ /* Mark buffer invalid */
+ ClearBufferTag(&bufHdr->tag);
+ buf_state &= ~BUF_FLAG_MASK;
+ buf_state &= ~BUF_USAGECOUNT_MASK;
+ pg_atomic_unlocked_write_u32(&bufHdr->state, buf_state);
+}
+
/*
* DropRelationLocalBuffers
* This function removes from the buffer pool all the pages of the
@@ -512,7 +542,6 @@ DropRelationLocalBuffers(RelFileLocator rlocator, ForkNumber forkNum,
for (i = 0; i < NLocBuffer; i++)
{
BufferDesc *bufHdr = GetLocalBufferDescriptor(i);
- LocalBufferLookupEnt *hresult;
uint32 buf_state;
buf_state = pg_atomic_read_u32(&bufHdr->state);
@@ -522,24 +551,7 @@ DropRelationLocalBuffers(RelFileLocator rlocator, ForkNumber forkNum,
BufTagGetForkNum(&bufHdr->tag) == forkNum &&
bufHdr->tag.blockNum >= firstDelBlock)
{
- if (LocalRefCount[i] != 0)
- elog(ERROR, "block %u of %s is still referenced (local %u)",
- bufHdr->tag.blockNum,
- relpathbackend(BufTagGetRelFileLocator(&bufHdr->tag),
- MyProcNumber,
- BufTagGetForkNum(&bufHdr->tag)).str,
- LocalRefCount[i]);
-
- /* Remove entry from hashtable */
- hresult = (LocalBufferLookupEnt *)
- hash_search(LocalBufHash, &bufHdr->tag, HASH_REMOVE, NULL);
- if (!hresult) /* shouldn't happen */
- elog(ERROR, "local buffer hash table corrupted");
- /* Mark buffer invalid */
- ClearBufferTag(&bufHdr->tag);
- buf_state &= ~BUF_FLAG_MASK;
- buf_state &= ~BUF_USAGECOUNT_MASK;
- pg_atomic_unlocked_write_u32(&bufHdr->state, buf_state);
+ InvalidateLocalBuffer(bufHdr, true);
}
}
}
@@ -559,7 +571,6 @@ DropRelationAllLocalBuffers(RelFileLocator rlocator)
for (i = 0; i < NLocBuffer; i++)
{
BufferDesc *bufHdr = GetLocalBufferDescriptor(i);
- LocalBufferLookupEnt *hresult;
uint32 buf_state;
buf_state = pg_atomic_read_u32(&bufHdr->state);
@@ -567,23 +578,7 @@ DropRelationAllLocalBuffers(RelFileLocator rlocator)
if ((buf_state & BM_TAG_VALID) &&
BufTagMatchesRelFileLocator(&bufHdr->tag, &rlocator))
{
- if (LocalRefCount[i] != 0)
- elog(ERROR, "block %u of %s is still referenced (local %u)",
- bufHdr->tag.blockNum,
- relpathbackend(BufTagGetRelFileLocator(&bufHdr->tag),
- MyProcNumber,
- BufTagGetForkNum(&bufHdr->tag)).str,
- LocalRefCount[i]);
- /* Remove entry from hashtable */
- hresult = (LocalBufferLookupEnt *)
- hash_search(LocalBufHash, &bufHdr->tag, HASH_REMOVE, NULL);
- if (!hresult) /* shouldn't happen */
- elog(ERROR, "local buffer hash table corrupted");
- /* Mark buffer invalid */
- ClearBufferTag(&bufHdr->tag);
- buf_state &= ~BUF_FLAG_MASK;
- buf_state &= ~BUF_USAGECOUNT_MASK;
- pg_atomic_unlocked_write_u32(&bufHdr->state, buf_state);
+ InvalidateLocalBuffer(bufHdr, true);
}
}
}
--
2.48.1.76.g4e746b1a31.dirty
[text/x-diff] v2.6-0018-localbuf-Introduce-TerminateLocalBufferIO.patch (5.1K, ../../ge6nsuddurhpmll3xj22vucvqwp4agqz6ndtcf2mhyeydzarst@l75dman5x53p/19-v2.6-0018-localbuf-Introduce-TerminateLocalBufferIO.patch)
download | inline diff:
From 491f65d6b356734065a190da3311391ffc74d08e Mon Sep 17 00:00:00 2001
From: Andres Freund <andres@anarazel.de>
Date: Sun, 9 Mar 2025 19:55:05 -0400
Subject: [PATCH v2.6 18/34] localbuf: Introduce TerminateLocalBufferIO()
Previously TerminateLocalBufferIO() was open-coded in multiple places, which
doesn't seem like a great idea.
Author:
Reviewed-by:
Discussion: https://postgr.es/m/
Backpatch:
---
src/include/storage/buf_internals.h | 2 ++
src/backend/storage/buffer/bufmgr.c | 32 ++++++++-------------------
src/backend/storage/buffer/localbuf.c | 28 +++++++++++++++++++++--
3 files changed, 37 insertions(+), 25 deletions(-)
diff --git a/src/include/storage/buf_internals.h b/src/include/storage/buf_internals.h
index 8b32fb108b0..4611a60d3e0 100644
--- a/src/include/storage/buf_internals.h
+++ b/src/include/storage/buf_internals.h
@@ -471,6 +471,8 @@ extern BlockNumber ExtendBufferedRelLocal(BufferManagerRelation bmr,
Buffer *buffers,
uint32 *extended_by);
extern void MarkLocalBufferDirty(Buffer buffer);
+extern void TerminateLocalBufferIO(BufferDesc *bufHdr, bool clear_dirty,
+ uint32 set_flag_bits);
extern void DropRelationLocalBuffers(RelFileLocator rlocator,
ForkNumber forkNum,
BlockNumber firstDelBlock);
diff --git a/src/backend/storage/buffer/bufmgr.c b/src/backend/storage/buffer/bufmgr.c
index 75a928e802a..ead562249d3 100644
--- a/src/backend/storage/buffer/bufmgr.c
+++ b/src/backend/storage/buffer/bufmgr.c
@@ -1072,19 +1072,11 @@ ZeroAndLockBuffer(Buffer buffer, ReadBufferMode mode, bool already_valid)
if (!isLocalBuf)
LWLockAcquire(BufferDescriptorGetContentLock(bufHdr), LW_EXCLUSIVE);
+ /* Set BM_VALID, terminate IO, and wake up any waiters */
if (isLocalBuf)
- {
- /* Only need to adjust flags */
- uint32 buf_state = pg_atomic_read_u32(&bufHdr->state);
-
- buf_state |= BM_VALID;
- pg_atomic_unlocked_write_u32(&bufHdr->state, buf_state);
- }
+ TerminateLocalBufferIO(bufHdr, false, BM_VALID);
else
- {
- /* Set BM_VALID, terminate IO, and wake up any waiters */
TerminateBufferIO(bufHdr, false, BM_VALID, true);
- }
}
else if (!isLocalBuf)
{
@@ -1608,19 +1600,11 @@ WaitReadBuffers(ReadBuffersOperation *operation)
relpath(operation->smgr->smgr_rlocator, forknum).str)));
}
- /* Terminate I/O and set BM_VALID. */
+ /* Set BM_VALID, terminate IO, and wake up any waiters */
if (persistence == RELPERSISTENCE_TEMP)
- {
- uint32 buf_state = pg_atomic_read_u32(&bufHdr->state);
-
- buf_state |= BM_VALID;
- pg_atomic_unlocked_write_u32(&bufHdr->state, buf_state);
- }
+ TerminateLocalBufferIO(bufHdr, false, BM_VALID);
else
- {
- /* Set BM_VALID, terminate IO, and wake up any waiters */
TerminateBufferIO(bufHdr, false, BM_VALID, true);
- }
/* Report I/Os as completing individually. */
TRACE_POSTGRESQL_BUFFER_READ_DONE(forknum, io_first_block + j,
@@ -4564,8 +4548,7 @@ FlushRelationBuffers(Relation rel)
IOCONTEXT_NORMAL, IOOP_WRITE,
io_start, 1, BLCKSZ);
- buf_state &= ~(BM_DIRTY | BM_JUST_DIRTIED);
- pg_atomic_unlocked_write_u32(&bufHdr->state, buf_state);
+ TerminateLocalBufferIO(bufHdr, true, 0);
pgBufferUsage.local_blks_written++;
@@ -5652,8 +5635,11 @@ TerminateBufferIO(BufferDesc *buf, bool clear_dirty, uint32 set_flag_bits,
buf_state = LockBufHdr(buf);
Assert(buf_state & BM_IO_IN_PROGRESS);
+ buf_state &= ~BM_IO_IN_PROGRESS;
+
+ /* Clear earlier errors, if this IO failed, it'll be marked again */
+ buf_state &= ~BM_IO_ERROR;
- buf_state &= ~(BM_IO_IN_PROGRESS | BM_IO_ERROR);
if (clear_dirty && !(buf_state & BM_JUST_DIRTIED))
buf_state &= ~(BM_DIRTY | BM_CHECKPOINT_NEEDED);
diff --git a/src/backend/storage/buffer/localbuf.c b/src/backend/storage/buffer/localbuf.c
index 4444228d645..ce3f8c78c18 100644
--- a/src/backend/storage/buffer/localbuf.c
+++ b/src/backend/storage/buffer/localbuf.c
@@ -259,8 +259,7 @@ GetLocalVictimBuffer(void)
IOOP_WRITE, io_start, 1, BLCKSZ);
/* Mark not-dirty now in case we error out below */
- buf_state &= ~BM_DIRTY;
- pg_atomic_unlocked_write_u32(&bufHdr->state, buf_state);
+ TerminateLocalBufferIO(bufHdr, true, 0);
pgBufferUsage.local_blks_written++;
}
@@ -483,6 +482,31 @@ MarkLocalBufferDirty(Buffer buffer)
pg_atomic_unlocked_write_u32(&bufHdr->state, buf_state);
}
+/*
+ * Like TerminateBufferIO, but for local buffers
+ */
+void
+TerminateLocalBufferIO(BufferDesc *bufHdr, bool clear_dirty, uint32 set_flag_bits)
+{
+ /* Only need to adjust flags */
+ uint32 buf_state = pg_atomic_read_u32(&bufHdr->state);
+
+ /* BM_IO_IN_PROGRESS isn't currently used for local buffers */
+
+ /* Clear earlier errors, if this IO failed, it'll be marked again */
+ buf_state &= ~BM_IO_ERROR;
+
+ if (clear_dirty)
+ buf_state &= ~BM_DIRTY;
+
+ buf_state |= set_flag_bits;
+ pg_atomic_unlocked_write_u32(&bufHdr->state, buf_state);
+
+ /* local buffers don't track IO using resowners */
+
+ /* local buffers don't use the IO CV, as no other process can see buffer */
+}
+
/*
* InvalidateLocalBuffer -- mark a local buffer invalid.
*
--
2.48.1.76.g4e746b1a31.dirty
[text/x-diff] v2.6-0019-localbuf-Introduce-FlushLocalBuffer.patch (5.1K, ../../ge6nsuddurhpmll3xj22vucvqwp4agqz6ndtcf2mhyeydzarst@l75dman5x53p/20-v2.6-0019-localbuf-Introduce-FlushLocalBuffer.patch)
download | inline diff:
From e227cea815d47fa12314ede9476fdf4e53d6eaf0 Mon Sep 17 00:00:00 2001
From: Andres Freund <andres@anarazel.de>
Date: Sun, 9 Mar 2025 18:33:35 -0400
Subject: [PATCH v2.6 19/34] localbuf: Introduce FlushLocalBuffer()
We unnecessarily had multiple paths containing the code to issue writes for
local buffers.
---
src/include/storage/buf_internals.h | 1 +
src/backend/storage/buffer/bufmgr.c | 22 +--------
src/backend/storage/buffer/localbuf.c | 65 +++++++++++++++------------
3 files changed, 38 insertions(+), 50 deletions(-)
diff --git a/src/include/storage/buf_internals.h b/src/include/storage/buf_internals.h
index 4611a60d3e0..90bc7e0db7b 100644
--- a/src/include/storage/buf_internals.h
+++ b/src/include/storage/buf_internals.h
@@ -473,6 +473,7 @@ extern BlockNumber ExtendBufferedRelLocal(BufferManagerRelation bmr,
extern void MarkLocalBufferDirty(Buffer buffer);
extern void TerminateLocalBufferIO(BufferDesc *bufHdr, bool clear_dirty,
uint32 set_flag_bits);
+extern void FlushLocalBuffer(BufferDesc *bufHdr, SMgrRelation reln);
extern void DropRelationLocalBuffers(RelFileLocator rlocator,
ForkNumber forkNum,
BlockNumber firstDelBlock);
diff --git a/src/backend/storage/buffer/bufmgr.c b/src/backend/storage/buffer/bufmgr.c
index ead562249d3..5ead69b5e16 100644
--- a/src/backend/storage/buffer/bufmgr.c
+++ b/src/backend/storage/buffer/bufmgr.c
@@ -4516,7 +4516,6 @@ FlushRelationBuffers(Relation rel)
for (i = 0; i < NLocBuffer; i++)
{
uint32 buf_state;
- instr_time io_start;
bufHdr = GetLocalBufferDescriptor(i);
if (BufTagMatchesRelFileLocator(&bufHdr->tag, &rel->rd_locator) &&
@@ -4524,9 +4523,6 @@ FlushRelationBuffers(Relation rel)
(BM_VALID | BM_DIRTY)) == (BM_VALID | BM_DIRTY))
{
ErrorContextCallback errcallback;
- Page localpage;
-
- localpage = (char *) LocalBufHdrGetBlock(bufHdr);
/* Setup error traceback support for ereport() */
errcallback.callback = local_buffer_write_error_callback;
@@ -4534,23 +4530,7 @@ FlushRelationBuffers(Relation rel)
errcallback.previous = error_context_stack;
error_context_stack = &errcallback;
- PageSetChecksumInplace(localpage, bufHdr->tag.blockNum);
-
- io_start = pgstat_prepare_io_time(track_io_timing);
-
- smgrwrite(srel,
- BufTagGetForkNum(&bufHdr->tag),
- bufHdr->tag.blockNum,
- localpage,
- false);
-
- pgstat_count_io_op_time(IOOBJECT_TEMP_RELATION,
- IOCONTEXT_NORMAL, IOOP_WRITE,
- io_start, 1, BLCKSZ);
-
- TerminateLocalBufferIO(bufHdr, true, 0);
-
- pgBufferUsage.local_blks_written++;
+ FlushLocalBuffer(bufHdr, srel);
/* Pop the error context stack */
error_context_stack = errcallback.previous;
diff --git a/src/backend/storage/buffer/localbuf.c b/src/backend/storage/buffer/localbuf.c
index ce3f8c78c18..8ea5088f32b 100644
--- a/src/backend/storage/buffer/localbuf.c
+++ b/src/backend/storage/buffer/localbuf.c
@@ -174,6 +174,41 @@ LocalBufferAlloc(SMgrRelation smgr, ForkNumber forkNum, BlockNumber blockNum,
return bufHdr;
}
+/*
+ * Like FlushBuffer(), just for local buffers.
+ */
+void
+FlushLocalBuffer(BufferDesc *bufHdr, SMgrRelation reln)
+{
+ instr_time io_start;
+ Page localpage = (char *) LocalBufHdrGetBlock(bufHdr);
+
+ /* Find smgr relation for buffer */
+ if (reln == NULL)
+ reln = smgropen(BufTagGetRelFileLocator(&bufHdr->tag),
+ MyProcNumber);
+
+ PageSetChecksumInplace(localpage, bufHdr->tag.blockNum);
+
+ io_start = pgstat_prepare_io_time(track_io_timing);
+
+ /* And write... */
+ smgrwrite(reln,
+ BufTagGetForkNum(&bufHdr->tag),
+ bufHdr->tag.blockNum,
+ localpage,
+ false);
+
+ /* Temporary table I/O does not use Buffer Access Strategies */
+ pgstat_count_io_op_time(IOOBJECT_TEMP_RELATION, IOCONTEXT_NORMAL,
+ IOOP_WRITE, io_start, 1, BLCKSZ);
+
+ /* Mark not-dirty */
+ TerminateLocalBufferIO(bufHdr, true, 0);
+
+ pgBufferUsage.local_blks_written++;
+}
+
static Buffer
GetLocalVictimBuffer(void)
{
@@ -234,35 +269,7 @@ GetLocalVictimBuffer(void)
* the case, write it out before reusing it!
*/
if (pg_atomic_read_u32(&bufHdr->state) & BM_DIRTY)
- {
- uint32 buf_state = pg_atomic_read_u32(&bufHdr->state);
- instr_time io_start;
- SMgrRelation oreln;
- Page localpage = (char *) LocalBufHdrGetBlock(bufHdr);
-
- /* Find smgr relation for buffer */
- oreln = smgropen(BufTagGetRelFileLocator(&bufHdr->tag), MyProcNumber);
-
- PageSetChecksumInplace(localpage, bufHdr->tag.blockNum);
-
- io_start = pgstat_prepare_io_time(track_io_timing);
-
- /* And write... */
- smgrwrite(oreln,
- BufTagGetForkNum(&bufHdr->tag),
- bufHdr->tag.blockNum,
- localpage,
- false);
-
- /* Temporary table I/O does not use Buffer Access Strategies */
- pgstat_count_io_op_time(IOOBJECT_TEMP_RELATION, IOCONTEXT_NORMAL,
- IOOP_WRITE, io_start, 1, BLCKSZ);
-
- /* Mark not-dirty now in case we error out below */
- TerminateLocalBufferIO(bufHdr, true, 0);
-
- pgBufferUsage.local_blks_written++;
- }
+ FlushLocalBuffer(bufHdr, NULL);
/*
* Remove the victim buffer from the hashtable and mark as invalid.
--
2.48.1.76.g4e746b1a31.dirty
[text/x-diff] v2.6-0020-localbuf-Introduce-StartLocalBufferIO.patch (4.6K, ../../ge6nsuddurhpmll3xj22vucvqwp4agqz6ndtcf2mhyeydzarst@l75dman5x53p/21-v2.6-0020-localbuf-Introduce-StartLocalBufferIO.patch)
download | inline diff:
From 2d76581fd451dc96e27a68d0a90b2b0ad837dbed Mon Sep 17 00:00:00 2001
From: Andres Freund <andres@anarazel.de>
Date: Sun, 9 Mar 2025 18:44:36 -0400
Subject: [PATCH v2.6 20/34] localbuf: Introduce StartLocalBufferIO()
Doesn't yet do a lot, but will do more in a subsequent commit.
Author:
Reviewed-by:
Discussion: https://postgr.es/m/
Backpatch:
---
src/include/storage/buf_internals.h | 1 +
src/backend/storage/buffer/bufmgr.c | 12 ++++-----
src/backend/storage/buffer/localbuf.c | 37 +++++++++++++++++++++++++++
3 files changed, 43 insertions(+), 7 deletions(-)
diff --git a/src/include/storage/buf_internals.h b/src/include/storage/buf_internals.h
index 90bc7e0db7b..87565ae9040 100644
--- a/src/include/storage/buf_internals.h
+++ b/src/include/storage/buf_internals.h
@@ -473,6 +473,7 @@ extern BlockNumber ExtendBufferedRelLocal(BufferManagerRelation bmr,
extern void MarkLocalBufferDirty(Buffer buffer);
extern void TerminateLocalBufferIO(BufferDesc *bufHdr, bool clear_dirty,
uint32 set_flag_bits);
+extern bool StartLocalBufferIO(BufferDesc *bufHdr, bool forInput, bool nowait);
extern void FlushLocalBuffer(BufferDesc *bufHdr, SMgrRelation reln);
extern void DropRelationLocalBuffers(RelFileLocator rlocator,
ForkNumber forkNum,
diff --git a/src/backend/storage/buffer/bufmgr.c b/src/backend/storage/buffer/bufmgr.c
index 5ead69b5e16..9c8f8ee9a20 100644
--- a/src/backend/storage/buffer/bufmgr.c
+++ b/src/backend/storage/buffer/bufmgr.c
@@ -1038,7 +1038,7 @@ ZeroAndLockBuffer(Buffer buffer, ReadBufferMode mode, bool already_valid)
{
/* Simple case for non-shared buffers. */
bufHdr = GetLocalBufferDescriptor(-buffer - 1);
- need_to_zero = (pg_atomic_read_u32(&bufHdr->state) & BM_VALID) == 0;
+ need_to_zero = StartLocalBufferIO(bufHdr, true, false);
}
else
{
@@ -1450,13 +1450,11 @@ static inline bool
WaitReadBuffersCanStartIO(Buffer buffer, bool nowait)
{
if (BufferIsLocal(buffer))
- {
- BufferDesc *bufHdr = GetLocalBufferDescriptor(-buffer - 1);
-
- return (pg_atomic_read_u32(&bufHdr->state) & BM_VALID) == 0;
- }
+ return StartLocalBufferIO(GetLocalBufferDescriptor(-buffer - 1),
+ true, nowait);
else
- return StartBufferIO(GetBufferDescriptor(buffer - 1), true, nowait);
+ return StartBufferIO(GetBufferDescriptor(buffer - 1),
+ true, nowait);
}
void
diff --git a/src/backend/storage/buffer/localbuf.c b/src/backend/storage/buffer/localbuf.c
index 8ea5088f32b..70d0b91034d 100644
--- a/src/backend/storage/buffer/localbuf.c
+++ b/src/backend/storage/buffer/localbuf.c
@@ -183,6 +183,13 @@ FlushLocalBuffer(BufferDesc *bufHdr, SMgrRelation reln)
instr_time io_start;
Page localpage = (char *) LocalBufHdrGetBlock(bufHdr);
+ /*
+ * Try to start an I/O operation. There currently are no reasons for
+ * StartLocalBufferIO to return false, so we raise an error in that case.
+ */
+ if (!StartLocalBufferIO(bufHdr, false, false))
+ elog(ERROR, "failed to start write IO on local buffer");
+
/* Find smgr relation for buffer */
if (reln == NULL)
reln = smgropen(BufTagGetRelFileLocator(&bufHdr->tag),
@@ -406,11 +413,17 @@ ExtendBufferedRelLocal(BufferManagerRelation bmr,
PinLocalBuffer(existing_hdr, false);
buffers[i] = BufferDescriptorGetBuffer(existing_hdr);
+ /*
+ * Clear the BM_VALID bit, do StartLocalBufferIO() and proceed.
+ */
buf_state = pg_atomic_read_u32(&existing_hdr->state);
Assert(buf_state & BM_TAG_VALID);
Assert(!(buf_state & BM_DIRTY));
buf_state &= ~BM_VALID;
pg_atomic_unlocked_write_u32(&existing_hdr->state, buf_state);
+
+ /* no need to loop for local buffers */
+ StartLocalBufferIO(existing_hdr, true, false);
}
else
{
@@ -425,6 +438,8 @@ ExtendBufferedRelLocal(BufferManagerRelation bmr,
pg_atomic_unlocked_write_u32(&victim_buf_hdr->state, buf_state);
hresult->id = victim_buf_id;
+
+ StartLocalBufferIO(victim_buf_hdr, true, false);
}
}
@@ -489,6 +504,28 @@ MarkLocalBufferDirty(Buffer buffer)
pg_atomic_unlocked_write_u32(&bufHdr->state, buf_state);
}
+/*
+ * Like StartBufferIO, but for local buffers
+ */
+bool
+StartLocalBufferIO(BufferDesc *bufHdr, bool forInput, bool nowait)
+{
+ uint32 buf_state = pg_atomic_read_u32(&bufHdr->state);
+
+ if (forInput ? (buf_state & BM_VALID) : !(buf_state & BM_DIRTY))
+ {
+ /* someone else already did the I/O */
+ UnlockBufHdr(bufHdr, buf_state);
+ return false;
+ }
+
+ /* BM_IO_IN_PROGRESS isn't currently used for local buffers */
+
+ /* local buffers don't track IO using resowners */
+
+ return true;
+}
+
/*
* Like TerminateBufferIO, but for local buffers
*/
--
2.48.1.76.g4e746b1a31.dirty
[text/x-diff] v2.6-0021-localbuf-Track-pincount-in-BufferDesc-as-well.patch (4.3K, ../../ge6nsuddurhpmll3xj22vucvqwp4agqz6ndtcf2mhyeydzarst@l75dman5x53p/22-v2.6-0021-localbuf-Track-pincount-in-BufferDesc-as-well.patch)
download | inline diff:
From fa20461dc62fd32e4d9a72d0012a76819ae8bbb8 Mon Sep 17 00:00:00 2001
From: Andres Freund <andres@anarazel.de>
Date: Thu, 20 Feb 2025 18:04:48 -0500
Subject: [PATCH v2.6 21/34] localbuf: Track pincount in BufferDesc as well
For AIO on temp tables the AIO subsystem needs to be able to ensure a pin on a
buffer while AIO is going on, even if the IO issuing query errors out. To do
so, track the refcount in BufferDesc.state, not just LocalRefCount.
Note that we still don't need locking, AIO completion callbacks for local
buffers are executed in the issuing session (i.e. nobody else has access to
the BufferDesc).
---
src/backend/storage/buffer/bufmgr.c | 22 +++++++++++++++++++++
src/backend/storage/buffer/localbuf.c | 28 +++++++++++++++++++++++++--
2 files changed, 48 insertions(+), 2 deletions(-)
diff --git a/src/backend/storage/buffer/bufmgr.c b/src/backend/storage/buffer/bufmgr.c
index 9c8f8ee9a20..b2ccd087b51 100644
--- a/src/backend/storage/buffer/bufmgr.c
+++ b/src/backend/storage/buffer/bufmgr.c
@@ -5404,6 +5404,18 @@ ConditionalLockBufferForCleanup(Buffer buffer)
Assert(refcount > 0);
if (refcount != 1)
return false;
+
+ /*
+ * Check that the AIO subsystem doesn't have a pin. Likely not
+ * possible today, but better safe than sorry.
+ */
+ bufHdr = GetLocalBufferDescriptor(-buffer - 1);
+ buf_state = pg_atomic_read_u32(&bufHdr->state);
+ refcount = BUF_STATE_GET_REFCOUNT(buf_state);
+ Assert(refcount > 0);
+ if (refcount != 1)
+ return false;
+
/* Nobody else to wait for */
return true;
}
@@ -5457,6 +5469,16 @@ IsBufferCleanupOK(Buffer buffer)
/* There should be exactly one pin */
if (LocalRefCount[-buffer - 1] != 1)
return false;
+
+ /*
+ * Check that the AIO subsystem doesn't have a pin. Likely not
+ * possible today, but better safe than sorry.
+ */
+ bufHdr = GetLocalBufferDescriptor(-buffer - 1);
+ buf_state = pg_atomic_read_u32(&bufHdr->state);
+ if (BUF_STATE_GET_REFCOUNT(buf_state) != 1)
+ return false;
+
/* Nobody else to wait for */
return true;
}
diff --git a/src/backend/storage/buffer/localbuf.c b/src/backend/storage/buffer/localbuf.c
index 70d0b91034d..a446a8620e2 100644
--- a/src/backend/storage/buffer/localbuf.c
+++ b/src/backend/storage/buffer/localbuf.c
@@ -249,6 +249,13 @@ GetLocalVictimBuffer(void)
pg_atomic_unlocked_write_u32(&bufHdr->state, buf_state);
trycounter = NLocBuffer;
}
+ else if (BUF_STATE_GET_REFCOUNT(buf_state) > 0)
+ {
+ /*
+ * This can be reached if the backend initiated AIO for this
+ * buffer and then errored out.
+ */
+ }
else
{
/* Found a usable buffer */
@@ -570,7 +577,13 @@ InvalidateLocalBuffer(BufferDesc *bufHdr, bool check_unreferenced)
buf_state = pg_atomic_read_u32(&bufHdr->state);
- if (check_unreferenced && LocalRefCount[bufid] != 0)
+ /*
+ * We need to test not just LocalRefCount[bufid] but also the BufferDesc
+ * itself, as the latter is used to represent a pin by the AIO subsystem.
+ * This can happen if AIO is initiated and then the query errors out.
+ */
+ if (check_unreferenced &&
+ (LocalRefCount[bufid] != 0 || BUF_STATE_GET_REFCOUNT(buf_state) != 0))
elog(ERROR, "block %u of %s is still referenced (local %u)",
bufHdr->tag.blockNum,
relpathbackend(BufTagGetRelFileLocator(&bufHdr->tag),
@@ -744,12 +757,13 @@ PinLocalBuffer(BufferDesc *buf_hdr, bool adjust_usagecount)
if (LocalRefCount[bufid] == 0)
{
NLocalPinnedBuffers++;
+ buf_state += BUF_REFCOUNT_ONE;
if (adjust_usagecount &&
BUF_STATE_GET_USAGECOUNT(buf_state) < BM_MAX_USAGE_COUNT)
{
buf_state += BUF_USAGECOUNT_ONE;
- pg_atomic_unlocked_write_u32(&buf_hdr->state, buf_state);
}
+ pg_atomic_unlocked_write_u32(&buf_hdr->state, buf_state);
}
LocalRefCount[bufid]++;
ResourceOwnerRememberBuffer(CurrentResourceOwner,
@@ -775,7 +789,17 @@ UnpinLocalBufferNoOwner(Buffer buffer)
Assert(NLocalPinnedBuffers > 0);
if (--LocalRefCount[buffid] == 0)
+ {
+ BufferDesc *buf_hdr = GetLocalBufferDescriptor(buffid);
+ uint32 buf_state;
+
NLocalPinnedBuffers--;
+
+ buf_state = pg_atomic_read_u32(&buf_hdr->state);
+ Assert(BUF_STATE_GET_REFCOUNT(buf_state) > 0);
+ buf_state -= BUF_REFCOUNT_ONE;
+ pg_atomic_unlocked_write_u32(&buf_hdr->state, buf_state);
+ }
}
/*
--
2.48.1.76.g4e746b1a31.dirty
[text/x-diff] v2.6-0022-bufmgr-Implement-AIO-read-support.patch (23.1K, ../../ge6nsuddurhpmll3xj22vucvqwp4agqz6ndtcf2mhyeydzarst@l75dman5x53p/23-v2.6-0022-bufmgr-Implement-AIO-read-support.patch)
download | inline diff:
From ac4b03e3569d522c926553316fba68fea976f637 Mon Sep 17 00:00:00 2001
From: Andres Freund <andres@anarazel.de>
Date: Sun, 9 Mar 2025 18:49:37 -0400
Subject: [PATCH v2.6 22/34] bufmgr: Implement AIO read support
This implements the following:
- Addition of callbacks to maintain buffer state on completion of a readv
- Addition of a wait reference to BufferDesc, to allow backends to wait for
IOs
- StartBufferIO(), WaitIO(), TerminateBufferIO() support for waiting AIO
As of this commit nothing issues AIO on shared buffers. A future commit will
update StartReadBuffers() to do so.
Todo:
- deduplicate shared/local buffer completion callbacks
- deduplicate LockBufferForCleanup() support
- function naming pattern (buffer_readv_complete_common() calls
LocalBufferCompleteRead/SharedBufferCompleteRead, which looks ugly)
Discussion: https://postgr.es/m/uvrtrknj4kdytuboidbhwclo4gxhswwcpgadptsjvjqcluzmah%40brqs62irg4dt
Discussion: https://postgr.es/m/20210223100344.llw5an2aklengrmn@alap3.anarazel.de
Discussion: https://postgr.es/m/stj36ea6yyhoxtqkhpieia2z4krnam7qyetc57rfezgk4zgapf@gcnactj4z56m
---
src/include/storage/aio.h | 4 +
src/include/storage/buf_internals.h | 8 +-
src/include/storage/bufmgr.h | 4 +
src/backend/storage/aio/aio_callback.c | 5 +
src/backend/storage/buffer/buf_init.c | 3 +
src/backend/storage/buffer/bufmgr.c | 379 ++++++++++++++++++++++++-
src/backend/storage/buffer/localbuf.c | 103 ++++++-
7 files changed, 493 insertions(+), 13 deletions(-)
diff --git a/src/include/storage/aio.h b/src/include/storage/aio.h
index 16476680f12..3ce8763ebe8 100644
--- a/src/include/storage/aio.h
+++ b/src/include/storage/aio.h
@@ -180,6 +180,10 @@ typedef enum PgAioHandleCallbackID
PGAIO_HCB_MD_READV,
PGAIO_HCB_MD_WRITEV,
+
+ PGAIO_HCB_SHARED_BUFFER_READV,
+
+ PGAIO_HCB_LOCAL_BUFFER_READV,
} PgAioHandleCallbackID;
diff --git a/src/include/storage/buf_internals.h b/src/include/storage/buf_internals.h
index 87565ae9040..257f8beeeec 100644
--- a/src/include/storage/buf_internals.h
+++ b/src/include/storage/buf_internals.h
@@ -17,6 +17,7 @@
#include "pgstat.h"
#include "port/atomics.h"
+#include "storage/aio_types.h"
#include "storage/buf.h"
#include "storage/bufmgr.h"
#include "storage/condition_variable.h"
@@ -264,6 +265,8 @@ typedef struct BufferDesc
int wait_backend_pgprocno; /* backend of pin-count waiter */
int freeNext; /* link in freelist chain */
+
+ PgAioWaitRef io_wref; /* set iff AIO is in progress */
LWLock content_lock; /* to lock access to buffer contents */
} BufferDesc;
@@ -472,7 +475,7 @@ extern BlockNumber ExtendBufferedRelLocal(BufferManagerRelation bmr,
uint32 *extended_by);
extern void MarkLocalBufferDirty(Buffer buffer);
extern void TerminateLocalBufferIO(BufferDesc *bufHdr, bool clear_dirty,
- uint32 set_flag_bits);
+ uint32 set_flag_bits, bool syncio);
extern bool StartLocalBufferIO(BufferDesc *bufHdr, bool forInput, bool nowait);
extern void FlushLocalBuffer(BufferDesc *bufHdr, SMgrRelation reln);
extern void DropRelationLocalBuffers(RelFileLocator rlocator,
@@ -481,4 +484,7 @@ extern void DropRelationLocalBuffers(RelFileLocator rlocator,
extern void DropRelationAllLocalBuffers(RelFileLocator rlocator);
extern void AtEOXact_LocalBuffers(bool isCommit);
+
+extern PgAioResult LocalBufferCompleteRead(int buf_off, Buffer buffer, uint8 flags, bool failed);
+
#endif /* BUFMGR_INTERNALS_H */
diff --git a/src/include/storage/bufmgr.h b/src/include/storage/bufmgr.h
index 4a035f59a7d..12687fde45e 100644
--- a/src/include/storage/bufmgr.h
+++ b/src/include/storage/bufmgr.h
@@ -15,6 +15,7 @@
#define BUFMGR_H
#include "port/pg_iovec.h"
+#include "storage/aio_types.h"
#include "storage/block.h"
#include "storage/buf.h"
#include "storage/bufpage.h"
@@ -168,6 +169,9 @@ extern PGDLLIMPORT int checkpoint_flush_after;
extern PGDLLIMPORT int backend_flush_after;
extern PGDLLIMPORT int bgwriter_flush_after;
+extern const PgAioHandleCallbacks aio_shared_buffer_readv_cb;
+extern const PgAioHandleCallbacks aio_local_buffer_readv_cb;
+
/* in buf_init.c */
extern PGDLLIMPORT char *BufferBlocks;
diff --git a/src/backend/storage/aio/aio_callback.c b/src/backend/storage/aio/aio_callback.c
index 6a21c82396a..f76f74ba166 100644
--- a/src/backend/storage/aio/aio_callback.c
+++ b/src/backend/storage/aio/aio_callback.c
@@ -18,6 +18,7 @@
#include "miscadmin.h"
#include "storage/aio.h"
#include "storage/aio_internal.h"
+#include "storage/bufmgr.h"
#include "storage/md.h"
@@ -40,6 +41,10 @@ static const PgAioHandleCallbacksEntry aio_handle_cbs[] = {
CALLBACK_ENTRY(PGAIO_HCB_INVALID, aio_invalid_cb),
CALLBACK_ENTRY(PGAIO_HCB_MD_READV, aio_md_readv_cb),
+
+ CALLBACK_ENTRY(PGAIO_HCB_SHARED_BUFFER_READV, aio_shared_buffer_readv_cb),
+
+ CALLBACK_ENTRY(PGAIO_HCB_LOCAL_BUFFER_READV, aio_local_buffer_readv_cb),
#undef CALLBACK_ENTRY
};
diff --git a/src/backend/storage/buffer/buf_init.c b/src/backend/storage/buffer/buf_init.c
index ed1f8e03190..ed1dc488a42 100644
--- a/src/backend/storage/buffer/buf_init.c
+++ b/src/backend/storage/buffer/buf_init.c
@@ -14,6 +14,7 @@
*/
#include "postgres.h"
+#include "storage/aio.h"
#include "storage/buf_internals.h"
#include "storage/bufmgr.h"
@@ -125,6 +126,8 @@ BufferManagerShmemInit(void)
buf->buf_id = i;
+ pgaio_wref_clear(&buf->io_wref);
+
/*
* Initially link all the buffers together as unused. Subsequent
* management of this list is done by freelist.c.
diff --git a/src/backend/storage/buffer/bufmgr.c b/src/backend/storage/buffer/bufmgr.c
index b2ccd087b51..149840f81ea 100644
--- a/src/backend/storage/buffer/bufmgr.c
+++ b/src/backend/storage/buffer/bufmgr.c
@@ -48,6 +48,7 @@
#include "pg_trace.h"
#include "pgstat.h"
#include "postmaster/bgwriter.h"
+#include "storage/aio.h"
#include "storage/buf_internals.h"
#include "storage/bufmgr.h"
#include "storage/fd.h"
@@ -516,7 +517,8 @@ static int SyncOneBuffer(int buf_id, bool skip_recently_used,
static void WaitIO(BufferDesc *buf);
static bool StartBufferIO(BufferDesc *buf, bool forInput, bool nowait);
static void TerminateBufferIO(BufferDesc *buf, bool clear_dirty,
- uint32 set_flag_bits, bool forget_owner);
+ uint32 set_flag_bits, bool forget_owner,
+ bool syncio);
static void AbortBufferIO(Buffer buffer);
static void shared_buffer_write_error_callback(void *arg);
static void local_buffer_write_error_callback(void *arg);
@@ -1074,9 +1076,9 @@ ZeroAndLockBuffer(Buffer buffer, ReadBufferMode mode, bool already_valid)
/* Set BM_VALID, terminate IO, and wake up any waiters */
if (isLocalBuf)
- TerminateLocalBufferIO(bufHdr, false, BM_VALID);
+ TerminateLocalBufferIO(bufHdr, false, BM_VALID, true);
else
- TerminateBufferIO(bufHdr, false, BM_VALID, true);
+ TerminateBufferIO(bufHdr, false, BM_VALID, true, true);
}
else if (!isLocalBuf)
{
@@ -1600,9 +1602,9 @@ WaitReadBuffers(ReadBuffersOperation *operation)
/* Set BM_VALID, terminate IO, and wake up any waiters */
if (persistence == RELPERSISTENCE_TEMP)
- TerminateLocalBufferIO(bufHdr, false, BM_VALID);
+ TerminateLocalBufferIO(bufHdr, false, BM_VALID, true);
else
- TerminateBufferIO(bufHdr, false, BM_VALID, true);
+ TerminateBufferIO(bufHdr, false, BM_VALID, true, true);
/* Report I/Os as completing individually. */
TRACE_POSTGRESQL_BUFFER_READ_DONE(forknum, io_first_block + j,
@@ -2513,7 +2515,7 @@ ExtendBufferedRelShared(BufferManagerRelation bmr,
if (lock)
LWLockAcquire(BufferDescriptorGetContentLock(buf_hdr), LW_EXCLUSIVE);
- TerminateBufferIO(buf_hdr, false, BM_VALID, true);
+ TerminateBufferIO(buf_hdr, false, BM_VALID, true, true);
}
pgBufferUsage.shared_blks_written += extend_by;
@@ -3970,7 +3972,7 @@ FlushBuffer(BufferDesc *buf, SMgrRelation reln, IOObject io_object,
* Mark the buffer as clean (unless BM_JUST_DIRTIED has become set) and
* end the BM_IO_IN_PROGRESS state.
*/
- TerminateBufferIO(buf, true, 0, true);
+ TerminateBufferIO(buf, true, 0, true, true);
TRACE_POSTGRESQL_BUFFER_FLUSH_DONE(BufTagGetForkNum(&buf->tag),
buf->tag.blockNum,
@@ -5529,6 +5531,7 @@ WaitIO(BufferDesc *buf)
for (;;)
{
uint32 buf_state;
+ PgAioWaitRef iow;
/*
* It may not be necessary to acquire the spinlock to check the flag
@@ -5536,10 +5539,19 @@ WaitIO(BufferDesc *buf)
* play it safe.
*/
buf_state = LockBufHdr(buf);
+ iow = buf->io_wref;
UnlockBufHdr(buf, buf_state);
if (!(buf_state & BM_IO_IN_PROGRESS))
break;
+
+ if (pgaio_wref_valid(&iow))
+ {
+ pgaio_wref_wait(&iow);
+ ConditionVariablePrepareToSleep(cv);
+ continue;
+ }
+
ConditionVariableSleep(cv, WAIT_EVENT_BUFFER_IO);
}
ConditionVariableCancelSleep();
@@ -5628,7 +5640,7 @@ StartBufferIO(BufferDesc *buf, bool forInput, bool nowait)
*/
static void
TerminateBufferIO(BufferDesc *buf, bool clear_dirty, uint32 set_flag_bits,
- bool forget_owner)
+ bool forget_owner, bool syncio)
{
uint32 buf_state;
@@ -5643,6 +5655,14 @@ TerminateBufferIO(BufferDesc *buf, bool clear_dirty, uint32 set_flag_bits,
if (clear_dirty && !(buf_state & BM_JUST_DIRTIED))
buf_state &= ~(BM_DIRTY | BM_CHECKPOINT_NEEDED);
+ if (!syncio)
+ {
+ /* release ownership by the AIO subsystem */
+ Assert(BUF_STATE_GET_REFCOUNT(buf_state) > 0);
+ buf_state -= BUF_REFCOUNT_ONE;
+ pgaio_wref_clear(&buf->io_wref);
+ }
+
buf_state |= set_flag_bits;
UnlockBufHdr(buf, buf_state);
@@ -5651,6 +5671,40 @@ TerminateBufferIO(BufferDesc *buf, bool clear_dirty, uint32 set_flag_bits,
BufferDescriptorGetBuffer(buf));
ConditionVariableBroadcast(BufferDescriptorGetIOCV(buf));
+
+ /*
+ * If we just released a pin, need to do BM_PIN_COUNT_WAITER handling.
+ * Most of the time the current backend will hold another pin preventing
+ * that from happening, but that's e.g. not the case when completing an IO
+ * another backend started.
+ *
+ * AFIXME: Deduplicate with UnpinBufferNoOwner() or just replace
+ * BM_PIN_COUNT_WAITER with something saner.
+ */
+ /* Support LockBufferForCleanup() */
+ if (buf_state & BM_PIN_COUNT_WAITER)
+ {
+ /*
+ * Acquire the buffer header lock, re-check that there's a waiter.
+ * Another backend could have unpinned this buffer, and already woken
+ * up the waiter. There's no danger of the buffer being replaced
+ * after we unpinned it above, as it's pinned by the waiter.
+ */
+ buf_state = LockBufHdr(buf);
+
+ if ((buf_state & BM_PIN_COUNT_WAITER) &&
+ BUF_STATE_GET_REFCOUNT(buf_state) == 1)
+ {
+ /* we just released the last pin other than the waiter's */
+ int wait_backend_pgprocno = buf->wait_backend_pgprocno;
+
+ buf_state &= ~BM_PIN_COUNT_WAITER;
+ UnlockBufHdr(buf, buf_state);
+ ProcSendSignal(wait_backend_pgprocno);
+ }
+ else
+ UnlockBufHdr(buf, buf_state);
+ }
}
/*
@@ -5699,7 +5753,7 @@ AbortBufferIO(Buffer buffer)
}
}
- TerminateBufferIO(buf_hdr, false, BM_IO_ERROR, false);
+ TerminateBufferIO(buf_hdr, false, BM_IO_ERROR, false, true);
}
/*
@@ -6150,3 +6204,310 @@ EvictUnpinnedBuffer(Buffer buf)
return result;
}
+
+static PgAioResult
+SharedBufferCompleteRead(int buf_off, Buffer buffer, uint8 flags, bool failed)
+{
+ BufferDesc *bufHdr = GetBufferDescriptor(buffer - 1);
+ BufferTag tag = bufHdr->tag;
+ char *bufdata = BufferGetBlock(buffer);
+ PgAioResult result;
+
+ Assert(BufferIsValid(buffer));
+
+#ifdef USE_ASSERT_CHECKING
+ {
+ uint32 buf_state = pg_atomic_read_u32(&bufHdr->state);
+
+ Assert(buf_state & BM_TAG_VALID);
+ Assert(!(buf_state & BM_VALID));
+ Assert(buf_state & BM_IO_IN_PROGRESS);
+ Assert(!(buf_state & BM_DIRTY));
+ }
+#endif
+
+ result.status = ARS_OK;
+
+ /* check for garbage data */
+ if (!failed &&
+ !PageIsVerifiedExtended((Page) bufdata, tag.blockNum,
+ PIV_LOG_WARNING | PIV_REPORT_STAT))
+ {
+ RelFileLocator rlocator = BufTagGetRelFileLocator(&tag);
+
+ if ((flags & READ_BUFFERS_ZERO_ON_ERROR) || zero_damaged_pages)
+ {
+ ereport(LOG,
+ (errcode(ERRCODE_DATA_CORRUPTED),
+ errmsg("invalid page in block %u of relation %s; zeroing out page",
+ tag.blockNum,
+ relpathperm(rlocator, tag.forkNum).str)));
+ memset(bufdata, 0, BLCKSZ);
+ }
+ else
+ {
+ /* mark buffer as having failed */
+ failed = true;
+
+ /* encode error for buffer_readv_report */
+ result.status = ARS_ERROR;
+ result.id = PGAIO_HCB_SHARED_BUFFER_READV;
+ result.error_data = buf_off;
+ }
+ }
+
+ /* Terminate I/O and set BM_VALID. */
+ TerminateBufferIO(bufHdr, false,
+ failed ? BM_IO_ERROR : BM_VALID,
+ false, false);
+
+ TRACE_POSTGRESQL_BUFFER_READ_DONE(tag.forkNum,
+ tag.blockNum,
+ tag.spcOid,
+ tag.dbOid,
+ tag.relNumber,
+ INVALID_PROC_NUMBER,
+ false);
+
+ return result;
+}
+
+/*
+ * Helper to prepare IO on shared buffers for execution, shared between reads
+ * and writes.
+ */
+static void
+shared_buffer_stage_common(PgAioHandle *ioh, bool is_write)
+{
+ uint64 *io_data;
+ uint8 handle_data_len;
+ PgAioWaitRef io_ref;
+ BufferTag first PG_USED_FOR_ASSERTS_ONLY = {0};
+
+ io_data = pgaio_io_get_handle_data(ioh, &handle_data_len);
+
+ pgaio_io_get_wref(ioh, &io_ref);
+
+ for (int i = 0; i < handle_data_len; i++)
+ {
+ Buffer buf = (Buffer) io_data[i];
+ BufferDesc *bufHdr;
+ uint32 buf_state;
+
+ bufHdr = GetBufferDescriptor(buf - 1);
+
+ if (i == 0)
+ first = bufHdr->tag;
+ else
+ {
+ Assert(bufHdr->tag.relNumber == first.relNumber);
+ Assert(bufHdr->tag.blockNum == first.blockNum + i);
+ }
+
+
+ buf_state = LockBufHdr(bufHdr);
+
+ Assert(buf_state & BM_TAG_VALID);
+ if (is_write)
+ {
+ Assert(buf_state & BM_VALID);
+ Assert(buf_state & BM_DIRTY);
+ }
+ else
+ Assert(!(buf_state & BM_VALID));
+
+ Assert(buf_state & BM_IO_IN_PROGRESS);
+ Assert(BUF_STATE_GET_REFCOUNT(buf_state) >= 1);
+
+ buf_state += BUF_REFCOUNT_ONE;
+ bufHdr->io_wref = io_ref;
+
+ UnlockBufHdr(bufHdr, buf_state);
+
+ if (is_write)
+ {
+ LWLock *content_lock;
+
+ content_lock = BufferDescriptorGetContentLock(bufHdr);
+
+ Assert(LWLockHeldByMe(content_lock));
+
+ /*
+ * Lock is now owned by AIO subsystem.
+ */
+ LWLockDisown(content_lock);
+ }
+
+ /*
+ * Stop tracking this buffer via the resowner - the AIO system now
+ * keeps track.
+ */
+ ResourceOwnerForgetBufferIO(CurrentResourceOwner, buf);
+ }
+}
+
+static void
+shared_buffer_readv_stage(PgAioHandle *ioh, uint8 cb_data)
+{
+ shared_buffer_stage_common(ioh, false);
+}
+
+/*
+ * AIO error reporting callback for aio_shared_buffer_readv_cb and aio_local_buffer_readv_cb.
+ *
+ * Errors are encoded as follows:
+ * - the only error is page verification failing
+ * - result.error_data is the offset of the first page that failed
+ * verification in a larger IO
+ */
+static void
+buffer_readv_report(PgAioResult result, const PgAioTargetData *target_data, int elevel)
+{
+ ProcNumber errProc;
+
+ if (target_data->smgr.is_temp)
+ errProc = MyProcNumber;
+ else
+ errProc = INVALID_PROC_NUMBER;
+
+ ereport(elevel,
+ errcode(ERRCODE_DATA_CORRUPTED),
+ errmsg("invalid page in block %u of relation %s",
+ target_data->smgr.blockNum + result.error_data,
+ relpathbackend(target_data->smgr.rlocator, errProc, target_data->smgr.forkNum).str
+ )
+ );
+}
+
+/*
+ * Perform completion handling of a single AIO read. This read may cover
+ * multiple blocks / buffers.
+ *
+ * Shared between shared and local buffers, to reduce code duplication.
+ */
+static PgAioResult
+buffer_readv_complete_common(PgAioHandle *ioh, PgAioResult prior_result, bool is_temp, uint8 cb_data)
+{
+ PgAioResult result = prior_result;
+ PgAioTargetData *td = pgaio_io_get_target_data(ioh);
+ uint64 *io_data;
+ uint8 handle_data_len;
+
+ if (is_temp)
+ {
+ Assert(td->smgr.is_temp);
+ Assert(pgaio_io_get_owner(ioh) == MyProcNumber);
+ }
+ else
+ Assert(!td->smgr.is_temp);
+
+ /*
+ * Iterate over all the buffers affected by this IO and call appropriate
+ * per-buffer completion function for each buffer.
+ */
+ io_data = pgaio_io_get_handle_data(ioh, &handle_data_len);
+ for (int buf_off = 0; buf_off < handle_data_len; buf_off++)
+ {
+ Buffer buf = io_data[buf_off];
+ PgAioResult buf_result;
+ bool failed;
+
+ /*
+ * If the entire failed on a lower-level, each buffer needs to be
+ * marked as failed. In case of a partial read, some buffers may be
+ * ok.
+ */
+ failed =
+ prior_result.status == ARS_ERROR
+ || prior_result.result <= buf_off;
+
+ if (is_temp)
+ buf_result = LocalBufferCompleteRead(buf_off, buf, cb_data, failed);
+ else
+ buf_result = SharedBufferCompleteRead(buf_off, buf, cb_data, failed);
+
+ /*
+ * If there wasn't any prior error and the IO for this page failed in
+ * some form, set the whole IO's to the page's result.
+ */
+ if (result.status != ARS_ERROR && buf_result.status != ARS_OK)
+ {
+ result = buf_result;
+ pgaio_result_report(result, td, LOG);
+ }
+ }
+
+ return result;
+}
+
+static PgAioResult
+shared_buffer_readv_complete(PgAioHandle *ioh, PgAioResult prior_result, uint8 cb_data)
+{
+ return buffer_readv_complete_common(ioh, prior_result, false, cb_data);
+}
+
+/*
+ * Helper to stage IO on local buffers for execution, shared between reads
+ * and writes.
+ */
+static void
+local_buffer_readv_stage(PgAioHandle *ioh, uint8 cb_data)
+{
+ uint64 *io_data;
+ uint8 handle_data_len;
+ PgAioWaitRef io_wref;
+
+ io_data = pgaio_io_get_handle_data(ioh, &handle_data_len);
+
+ pgaio_io_get_wref(ioh, &io_wref);
+
+ for (int i = 0; i < handle_data_len; i++)
+ {
+ Buffer buf = (Buffer) io_data[i];
+ BufferDesc *bufHdr;
+ uint32 buf_state;
+
+ bufHdr = GetLocalBufferDescriptor(-buf - 1);
+
+ buf_state = pg_atomic_read_u32(&bufHdr->state);
+
+ bufHdr->io_wref = io_wref;
+
+ /*
+ * Track pin by AIO subsystem in BufferDesc, not in LocalRefCount as
+ * one might initially think. This is necessary to handle this backend
+ * erroring out while AIO is still in progress.
+ */
+ buf_state += BUF_REFCOUNT_ONE;
+
+ pg_atomic_unlocked_write_u32(&bufHdr->state, buf_state);
+ }
+}
+
+static PgAioResult
+local_buffer_readv_complete(PgAioHandle *ioh, PgAioResult prior_result, uint8 cb_data)
+{
+ return buffer_readv_complete_common(ioh, prior_result, true, cb_data);
+}
+
+
+/* readv callback is is passed READ_BUFFERS_* flags as callback data */
+const PgAioHandleCallbacks aio_shared_buffer_readv_cb = {
+ .stage = shared_buffer_readv_stage,
+ .complete_shared = shared_buffer_readv_complete,
+ .report = buffer_readv_report,
+};
+
+/* readv callback is is passed READ_BUFFERS_* flags as callback data */
+const PgAioHandleCallbacks aio_local_buffer_readv_cb = {
+ .stage = local_buffer_readv_stage,
+
+ /*
+ * Note that this, in contrast to the shared_buffers case, uses
+ * complete_local, as only the issuing backend has access to the required
+ * datastructures. This is important in case the IO completion may be
+ * consumed incidentally by another backend.
+ */
+ .complete_local = local_buffer_readv_complete,
+ .report = buffer_readv_report,
+};
diff --git a/src/backend/storage/buffer/localbuf.c b/src/backend/storage/buffer/localbuf.c
index a446a8620e2..9a5c4f69635 100644
--- a/src/backend/storage/buffer/localbuf.c
+++ b/src/backend/storage/buffer/localbuf.c
@@ -17,7 +17,9 @@
#include "access/parallel.h"
#include "executor/instrument.h"
+#include "pg_trace.h"
#include "pgstat.h"
+#include "storage/aio.h"
#include "storage/buf_internals.h"
#include "storage/bufmgr.h"
#include "storage/fd.h"
@@ -211,7 +213,7 @@ FlushLocalBuffer(BufferDesc *bufHdr, SMgrRelation reln)
IOOP_WRITE, io_start, 1, BLCKSZ);
/* Mark not-dirty */
- TerminateLocalBufferIO(bufHdr, true, 0);
+ TerminateLocalBufferIO(bufHdr, true, 0, true);
pgBufferUsage.local_blks_written++;
}
@@ -517,8 +519,23 @@ MarkLocalBufferDirty(Buffer buffer)
bool
StartLocalBufferIO(BufferDesc *bufHdr, bool forInput, bool nowait)
{
- uint32 buf_state = pg_atomic_read_u32(&bufHdr->state);
+ uint32 buf_state;
+ /*
+ * The buffer could have IO in progress, e.g. when there are two scans of
+ * the same relation. Either wait for the other IO or return false.
+ */
+ if (pgaio_wref_valid(&bufHdr->io_wref))
+ {
+ PgAioWaitRef iow = bufHdr->io_wref;
+
+ if (nowait)
+ return false;
+
+ pgaio_wref_wait(&iow);
+ }
+
+ buf_state = pg_atomic_read_u32(&bufHdr->state);
if (forInput ? (buf_state & BM_VALID) : !(buf_state & BM_DIRTY))
{
/* someone else already did the I/O */
@@ -537,7 +554,8 @@ StartLocalBufferIO(BufferDesc *bufHdr, bool forInput, bool nowait)
* Like TerminateBufferIO, but for local buffers
*/
void
-TerminateLocalBufferIO(BufferDesc *bufHdr, bool clear_dirty, uint32 set_flag_bits)
+TerminateLocalBufferIO(BufferDesc *bufHdr, bool clear_dirty, uint32 set_flag_bits,
+ bool syncio)
{
/* Only need to adjust flags */
uint32 buf_state = pg_atomic_read_u32(&bufHdr->state);
@@ -550,6 +568,17 @@ TerminateLocalBufferIO(BufferDesc *bufHdr, bool clear_dirty, uint32 set_flag_bit
if (clear_dirty)
buf_state &= ~BM_DIRTY;
+ if (!syncio)
+ {
+ /*
+ * Release pin held by IO subsystem, see also
+ * local_buffer_readv_prepare().
+ */
+ Assert(BUF_STATE_GET_REFCOUNT(buf_state) > 0);
+ buf_state -= BUF_REFCOUNT_ONE;
+ pgaio_wref_clear(&bufHdr->io_wref);
+ }
+
buf_state |= set_flag_bits;
pg_atomic_unlocked_write_u32(&bufHdr->state, buf_state);
@@ -714,6 +743,8 @@ InitLocalBuffers(void)
*/
buf->buf_id = -i - 2;
+ pgaio_wref_clear(&buf->io_wref);
+
/*
* Intentionally do not initialize the buffer's atomic variable
* (besides zeroing the underlying memory above). That way we get
@@ -941,3 +972,69 @@ AtProcExit_LocalBuffers(void)
*/
CheckForLocalBufferLeaks();
}
+
+PgAioResult
+LocalBufferCompleteRead(int buf_off, Buffer buffer, uint8 flags, bool failed)
+{
+ BufferDesc *bufHdr = GetLocalBufferDescriptor(-buffer - 1);
+ BufferTag tag = bufHdr->tag;
+ char *bufdata = BufferGetBlock(buffer);
+ PgAioResult result;
+
+ Assert(BufferIsValid(buffer));
+
+#ifdef USE_ASSERT_CHECKING
+ {
+ uint32 buf_state = pg_atomic_read_u32(&bufHdr->state);
+
+ Assert(buf_state & BM_TAG_VALID);
+ Assert(!(buf_state & BM_VALID));
+ Assert(!(buf_state & BM_DIRTY));
+ }
+#endif
+
+ result.status = ARS_OK;
+
+ /* check for garbage data */
+ if (!failed &&
+ !PageIsVerifiedExtended((Page) bufdata, tag.blockNum,
+ PIV_LOG_WARNING | PIV_REPORT_STAT))
+ {
+ RelFileLocator rlocator = BufTagGetRelFileLocator(&tag);
+
+ if ((flags & READ_BUFFERS_ZERO_ON_ERROR) || zero_damaged_pages)
+ {
+ ereport(LOG,
+ (errcode(ERRCODE_DATA_CORRUPTED),
+ errmsg("invalid page in block %u of relation %s; zeroing out page",
+ tag.blockNum,
+ relpathbackend(rlocator, MyProcNumber, tag.forkNum).str)));
+ memset(bufdata, 0, BLCKSZ);
+ }
+ else
+ {
+ /* mark buffer as having failed */
+ failed = true;
+
+ /* encode error for buffer_readv_report */
+ result.status = ARS_ERROR;
+ result.id = PGAIO_HCB_LOCAL_BUFFER_READV;
+ result.error_data = buf_off;
+ }
+ }
+
+ /* Terminate I/O and set BM_VALID. */
+ TerminateLocalBufferIO(bufHdr, false,
+ failed ? BM_IO_ERROR : BM_VALID,
+ false);
+
+ TRACE_POSTGRESQL_BUFFER_READ_DONE(tag.forkNum,
+ tag.blockNum,
+ tag.spcOid,
+ tag.dbOid,
+ tag.relNumber,
+ MyProcNumber,
+ false);
+
+ return result;
+}
--
2.48.1.76.g4e746b1a31.dirty
[text/x-diff] v2.6-0023-bufmgr-Use-AIO-in-StartReadBuffers.patch (19.6K, ../../ge6nsuddurhpmll3xj22vucvqwp4agqz6ndtcf2mhyeydzarst@l75dman5x53p/24-v2.6-0023-bufmgr-Use-AIO-in-StartReadBuffers.patch)
download | inline diff:
From 7d30d8b7eb9888800a2456d5f9b9a58458590f4d Mon Sep 17 00:00:00 2001
From: Andres Freund <andres@anarazel.de>
Date: Sun, 9 Mar 2025 18:52:26 -0400
Subject: [PATCH v2.6 23/34] bufmgr: Use AIO in StartReadBuffers()
This finally introduces the first actual AIO user. StartReadBuffers() now uses
the AIO routines to issue IO. This converts a lot of callers to use the AIO
infrastructure.
As the implementation of StartReadBuffers() is also used by the functions for
reading individual blocks (StartReadBuffer() and through that
ReadBufferExtended()) this means all buffered read IO passes through the AIO
paths. However, as those are synchronous reads, actually performing the IO
asynchronously would be rarely beneficial. Instead such IOs are flagged to
always be executed synchronously. This way we don't have to duplicate a fair
bit of code.
When io_method=sync is used, the IO patterns generated after this change are
the same as before, i.e. actual reads are only issued in WaitReadBuffers() and
StartReadBuffers() may issue prefetch requests. This allows to bypass most of
the actual asynchronicity, which is important to make a change as big as this
less risky.
One thing worth calling out is that, if IO is actually executed
asynchronously, the precise meaning of what track_io_timing is measuring has
changed. Previously it tracked the time for each IO, but that does not make
sense when multiple IOs are executed concurrently. Now it only measures the
time actually spent waiting for IO.
While AIO is now actually used, the logic in read_stream.c will often prevent
using sufficiently many concurrent IOs. That will be addressed in the next
commits.
Co-authored-by: Andres Freund <andres@anarazel.de>
Co-authored-by: Thomas Munro <thomas.munro@gmail.com>
Discussion: https://postgr.es/m/uvrtrknj4kdytuboidbhwclo4gxhswwcpgadptsjvjqcluzmah%40brqs62irg4dt
Discussion: https://postgr.es/m/20210223100344.llw5an2aklengrmn@alap3.anarazel.de
Discussion: https://postgr.es/m/stj36ea6yyhoxtqkhpieia2z4krnam7qyetc57rfezgk4zgapf@gcnactj4z56m
---
src/include/storage/bufmgr.h | 6 +
src/backend/storage/buffer/bufmgr.c | 399 +++++++++++++++++++++-------
2 files changed, 305 insertions(+), 100 deletions(-)
diff --git a/src/include/storage/bufmgr.h b/src/include/storage/bufmgr.h
index 12687fde45e..db9a4673097 100644
--- a/src/include/storage/bufmgr.h
+++ b/src/include/storage/bufmgr.h
@@ -112,6 +112,9 @@ typedef struct BufferManagerRelation
#define READ_BUFFERS_ZERO_ON_ERROR (1 << 0)
/* Call smgrprefetch() if I/O necessary. */
#define READ_BUFFERS_ISSUE_ADVICE (1 << 1)
+/* IO will immediately be waited for */
+#define READ_BUFFERS_SYNCHRONOUSLY (1 << 2)
+
struct ReadBuffersOperation
{
@@ -131,6 +134,9 @@ struct ReadBuffersOperation
BlockNumber blocknum;
int flags;
int16 nblocks;
+
+ PgAioWaitRef io_wref;
+ PgAioReturn io_return;
};
typedef struct ReadBuffersOperation ReadBuffersOperation;
diff --git a/src/backend/storage/buffer/bufmgr.c b/src/backend/storage/buffer/bufmgr.c
index 149840f81ea..60df9eb8cba 100644
--- a/src/backend/storage/buffer/bufmgr.c
+++ b/src/backend/storage/buffer/bufmgr.c
@@ -528,6 +528,8 @@ static inline BufferDesc *BufferAlloc(SMgrRelation smgr,
BlockNumber blockNum,
BufferAccessStrategy strategy,
bool *foundPtr, IOContext io_context);
+static bool AsyncReadBuffers(ReadBuffersOperation *operation,
+ int *nblocks);
static Buffer GetVictimBuffer(BufferAccessStrategy strategy, IOContext io_context);
static void FlushBuffer(BufferDesc *buf, SMgrRelation reln,
IOObject io_object, IOContext io_context);
@@ -1228,10 +1230,9 @@ ReadBuffer_common(Relation rel, SMgrRelation smgr, char smgr_persistence,
return buffer;
}
+ flags = READ_BUFFERS_SYNCHRONOUSLY;
if (mode == RBM_ZERO_ON_ERROR)
- flags = READ_BUFFERS_ZERO_ON_ERROR;
- else
- flags = 0;
+ flags |= READ_BUFFERS_ZERO_ON_ERROR;
operation.smgr = smgr;
operation.rel = rel;
operation.persistence = persistence;
@@ -1259,6 +1260,7 @@ StartReadBuffersImpl(ReadBuffersOperation *operation,
Assert(*nblocks > 0);
Assert(*nblocks <= MAX_IO_COMBINE_LIMIT);
+ Assert(*nblocks == 1 || allow_forwarding);
for (int i = 0; i < actual_nblocks; ++i)
{
@@ -1298,6 +1300,11 @@ StartReadBuffersImpl(ReadBuffersOperation *operation,
else
bufHdr = GetBufferDescriptor(buffers[i] - 1);
found = pg_atomic_read_u32(&bufHdr->state) & BM_VALID;
+
+ ereport(DEBUG3,
+ errmsg("found forwarded buffer %d",
+ buffers[i]),
+ errhidestmt(true), errhidecontext(true));
}
else
{
@@ -1363,25 +1370,59 @@ StartReadBuffersImpl(ReadBuffersOperation *operation,
operation->blocknum = blockNum;
operation->flags = flags;
operation->nblocks = actual_nblocks;
+ pgaio_wref_clear(&operation->io_wref);
- if (flags & READ_BUFFERS_ISSUE_ADVICE)
+ /*
+ * When using AIO, start the IO in the background. If not, issue prefetch
+ * requests if desired by the caller.
+ *
+ * The reason we have a dedicated path for IOMETHOD_SYNC here is to
+ * de-risk the introduction of AIO somewhat. It's a large architectural
+ * change, with lots of chances for unanticipated performance effects.
+ *
+ * Use of IOMETHOD_SYNC already leads to not actually performing IO
+ * asynchronously, but without the check here we'd execute IO earlier than
+ * we used to. Eventually this IOMETHOD_SYNC specific path should go away.
+ */
+ if (io_method != IOMETHOD_SYNC)
{
/*
- * In theory we should only do this if PinBufferForBlock() had to
- * allocate new buffers above. That way, if two calls to
- * StartReadBuffers() were made for the same blocks before
- * WaitReadBuffers(), only the first would issue the advice. That'd be
- * a better simulation of true asynchronous I/O, which would only
- * start the I/O once, but isn't done here for simplicity.
+ * Try to start IO asynchronously. It's possible that no IO needs to
+ * be started, if another backend already performed the IO.
+ *
+ * Note that if an IO is started, it might not cover the entire
+ * requested range, e.g. because an intermediary block has been read
+ * in by another backend. In that case any "trailing" buffers we
+ * already pinned above will be "forwarded" by read_stream.c to the
+ * next call to StartReadBuffers(). This is signalled to the caller by
+ * decrementing *nblocks.
*/
- smgrprefetch(operation->smgr,
- operation->forknum,
- blockNum,
- actual_nblocks);
+ return AsyncReadBuffers(operation, nblocks);
}
+ else
+ {
+ operation->flags |= READ_BUFFERS_SYNCHRONOUSLY;
- /* Indicate that WaitReadBuffers() should be called. */
- return true;
+ if (flags & READ_BUFFERS_ISSUE_ADVICE)
+ {
+ /*
+ * In theory we should only do this if PinBufferForBlock() had to
+ * allocate new buffers above. That way, if two calls to
+ * StartReadBuffers() were made for the same blocks before
+ * WaitReadBuffers(), only the first would issue the advice.
+ * That'd be a better simulation of true asynchronous I/O, which
+ * would only start the I/O once, but isn't done here for
+ * simplicity.
+ */
+ smgrprefetch(operation->smgr,
+ operation->forknum,
+ blockNum,
+ actual_nblocks);
+ }
+
+ /* Indicate that WaitReadBuffers() should be called. */
+ return true;
+ }
}
/*
@@ -1449,7 +1490,7 @@ StartReadBuffer(ReadBuffersOperation *operation,
}
static inline bool
-WaitReadBuffersCanStartIO(Buffer buffer, bool nowait)
+ReadBuffersCanStartIO(Buffer buffer, bool nowait)
{
if (BufferIsLocal(buffer))
return StartLocalBufferIO(GetLocalBufferDescriptor(-buffer - 1),
@@ -1462,28 +1503,163 @@ WaitReadBuffersCanStartIO(Buffer buffer, bool nowait)
void
WaitReadBuffers(ReadBuffersOperation *operation)
{
- Buffer *buffers;
+ IOContext io_context;
+ IOObject io_object;
int nblocks;
- BlockNumber blocknum;
- ForkNumber forknum;
- IOContext io_context;
- IOObject io_object;
- char persistence;
+ PgAioReturn *aio_ret;
+
+ /*
+ * If we get here without an IO operation having been issued, io_method ==
+ * IOMETHOD_SYNC path must have been used. In that case, we start - as we
+ * used to before - the IO now, just before waiting.
+ *
+ * This path is expected to eventually go away.
+ */
+ if (!pgaio_wref_valid(&operation->io_wref))
+ {
+ Assert(io_method == IOMETHOD_SYNC);
+
+ while (true)
+ {
+ nblocks = operation->nblocks;
+
+ if (!AsyncReadBuffers(operation, &nblocks))
+ {
+ /* all blocks were already read in concurrently */
+ Assert(nblocks == operation->nblocks);
+ return;
+ }
+
+ Assert(nblocks > 0 && nblocks <= operation->nblocks);
+
+ if (nblocks == operation->nblocks)
+ {
+ /* will wait below as if this had been normal AIO */
+ break;
+ }
+
+ /*
+ * It's unlikely, but possible, that AsyncReadBuffers() wasn't
+ * able to initiate IO for all the relevant buffers. In that case
+ * we need to wait for the prior IO before issuing more IO.
+ */
+ WaitReadBuffers(operation);
+ }
+ }
+
+ if (operation->persistence == RELPERSISTENCE_TEMP)
+ {
+ io_context = IOCONTEXT_NORMAL;
+ io_object = IOOBJECT_TEMP_RELATION;
+ }
+ else
+ {
+ io_context = IOContextForStrategy(operation->strategy);
+ io_object = IOOBJECT_RELATION;
+ }
+
+restart:
/* Find the range of the physical read we need to perform. */
nblocks = operation->nblocks;
- buffers = &operation->buffers[0];
- blocknum = operation->blocknum;
- forknum = operation->forknum;
- persistence = operation->persistence;
-
Assert(nblocks > 0);
Assert(nblocks <= MAX_IO_COMBINE_LIMIT);
+ aio_ret = &operation->io_return;
+
+ /*
+ * For IO timing we just count the time spent waiting for the IO.
+ *
+ * XXX: We probably should track the IO operation, rather than its time,
+ * separately, when initiating the IO. But right now that's not quite
+ * allowed by the interface.
+ */
+
+ /*
+ * Tracking a wait even if we don't actually need to wait
+ *
+ * a) is not cheap
+ *
+ * b) reports some time as waiting, even if we never waited.
+ */
+ if (aio_ret->result.status == ARS_UNKNOWN &&
+ !pgaio_wref_check_done(&operation->io_wref))
+ {
+ instr_time io_start = pgstat_prepare_io_time(track_io_timing);
+
+ pgaio_wref_wait(&operation->io_wref);
+
+ /*
+ * The IO operation itself was already counted earlier, in
+ * AsyncReadBuffers().
+ */
+ pgstat_count_io_op_time(io_object, io_context, IOOP_READ,
+ io_start, 0, 0);
+ }
+ else
+ {
+ Assert(pgaio_wref_check_done(&operation->io_wref));
+ }
+
+ if (aio_ret->result.status == ARS_PARTIAL)
+ {
+ /*
+ * We'll retry below, so we just emit a debug message the server log
+ * (or not even that in prod scenarios).
+ */
+ pgaio_result_report(aio_ret->result, &aio_ret->target_data, DEBUG1);
+
+ /*
+ * Try to perform the rest of the IO. Buffers for which IO has
+ * completed successfully will be discovered as such and not retried.
+ */
+ nblocks = operation->nblocks;
+
+ elog(DEBUG3, "retrying IO after partial failure");
+ CHECK_FOR_INTERRUPTS();
+ AsyncReadBuffers(operation, &nblocks);
+ goto restart;
+ }
+ else if (aio_ret->result.status != ARS_OK)
+ pgaio_result_report(aio_ret->result, &aio_ret->target_data, ERROR);
+
+ if (VacuumCostActive)
+ VacuumCostBalance += VacuumCostPageMiss * nblocks;
+
+ /* NB: READ_DONE tracepoint is executed in IO completion callback */
+}
+
+/*
+ * Initiate IO for the ReadBuffersOperation. If IO is only initiated for a
+ * subset of the blocks, *nblocks is updated to reflect that.
+ *
+ * Returns true if IO was initiated, false if no IO was necessary.
+ */
+static bool
+AsyncReadBuffers(ReadBuffersOperation *operation,
+ int *nblocks)
+{
+ int io_buffers_len = 0;
+ Buffer *buffers = &operation->buffers[0];
+ int flags = operation->flags;
+ BlockNumber blocknum = operation->blocknum;
+ ForkNumber forknum = operation->forknum;
+ bool did_start_io = false;
+ PgAioHandle *ioh = NULL;
+ uint32 ioh_flags = 0;
+ IOContext io_context;
+ IOObject io_object;
+ char persistence;
+
+ persistence = operation->rel
+ ? operation->rel->rd_rel->relpersistence
+ : RELPERSISTENCE_PERMANENT;
+
if (persistence == RELPERSISTENCE_TEMP)
{
io_context = IOCONTEXT_NORMAL;
io_object = IOOBJECT_TEMP_RELATION;
+ ioh_flags |= PGAIO_HF_REFERENCES_LOCAL;
}
else
{
@@ -1491,6 +1667,14 @@ WaitReadBuffers(ReadBuffersOperation *operation)
io_object = IOOBJECT_RELATION;
}
+ /*
+ * When this IO is executed synchronously, either because the caller will
+ * immediately block waiting for the IO or because IOMETHOD_SYNC is used,
+ * the AIO subsystem needs to know.
+ */
+ if (flags & READ_BUFFERS_SYNCHRONOUSLY)
+ ioh_flags |= PGAIO_HF_SYNCHRONOUS;
+
/*
* We count all these blocks as read by this backend. This is traditional
* behavior, but might turn out to be not true if we find that someone
@@ -1500,25 +1684,53 @@ WaitReadBuffers(ReadBuffersOperation *operation)
* but another backend completed the read".
*/
if (persistence == RELPERSISTENCE_TEMP)
- pgBufferUsage.local_blks_read += nblocks;
+ pgBufferUsage.local_blks_read += *nblocks;
else
- pgBufferUsage.shared_blks_read += nblocks;
+ pgBufferUsage.shared_blks_read += *nblocks;
- for (int i = 0; i < nblocks; ++i)
+ pgaio_wref_clear(&operation->io_wref);
+
+ /*
+ * Loop until we have started one IO or we discover that all buffers are
+ * already valid.
+ */
+ for (int i = 0; i < *nblocks; ++i)
{
- int io_buffers_len;
Buffer io_buffers[MAX_IO_COMBINE_LIMIT];
void *io_pages[MAX_IO_COMBINE_LIMIT];
- instr_time io_start;
BlockNumber io_first_block;
/*
- * Skip this block if someone else has already completed it. If an
- * I/O is already in progress in another backend, this will wait for
- * the outcome: either done, or something went wrong and we will
- * retry.
+ * Get IO before ReadBuffersCanStartIO, as pgaio_io_acquire() might
+ * block, which we don't want after setting IO_IN_PROGRESS.
+ *
+ * XXX: Should we attribute the time spent in here to the IO? If there
+ * already are a lot of IO operations in progress, getting an IO
+ * handle will block waiting for some other IO operation to finish.
+ *
+ * In most cases it'll be free to get the IO, so a timer would be
+ * overhead. Perhaps we should use pgaio_io_acquire_nb() and only
+ * account IO time when pgaio_io_acquire_nb() returned false?
*/
- if (!WaitReadBuffersCanStartIO(buffers[i], false))
+ if (likely(!ioh))
+ ioh = pgaio_io_acquire(CurrentResourceOwner,
+ &operation->io_return);
+
+ /*
+ * Skip this block if someone else has already completed it.
+ *
+ * If an I/O is already in progress in another backend, this will wait
+ * for the outcome: either done, or something went wrong and we will
+ * retry. But don't wait if we have staged, but haven't issued,
+ * another IO.
+ *
+ * It's safe to start IO while we have unsubmitted IO, but it'd be
+ * better to first submit it. But right now the boolean return value
+ * from ReadBuffersCanStartIO()/StartBufferIO() doesn't allow to
+ * distinguish between nowait=true trigger failure and the buffer
+ * already being valid.
+ */
+ if (!ReadBuffersCanStartIO(buffers[i], false))
{
/*
* Report this as a 'hit' for this backend, even though it must
@@ -1530,6 +1742,11 @@ WaitReadBuffers(ReadBuffersOperation *operation)
operation->smgr->smgr_rlocator.locator.relNumber,
operation->smgr->smgr_rlocator.backend,
true);
+
+ ereport(DEBUG3,
+ errmsg("can't start io for first buffer %u: %s",
+ buffers[i], DebugPrintBufferRefcount(buffers[i])),
+ errhidestmt(true), errhidecontext(true));
continue;
}
@@ -1539,6 +1756,11 @@ WaitReadBuffers(ReadBuffersOperation *operation)
io_first_block = blocknum + i;
io_buffers_len = 1;
+ ereport(DEBUG5,
+ errmsg("first prepped for io: %s, offset %d",
+ DebugPrintBufferRefcount(io_buffers[0]), i),
+ errhidestmt(true), errhidecontext(true));
+
/*
* How many neighboring-on-disk blocks can we scatter-read into other
* buffers at the same time? In this case we don't wait if we see an
@@ -1546,78 +1768,55 @@ WaitReadBuffers(ReadBuffersOperation *operation)
* head block, so we should get on with that I/O as soon as possible.
* We'll come back to this block again, above.
*/
- while ((i + 1) < nblocks &&
- WaitReadBuffersCanStartIO(buffers[i + 1], true))
+ while ((i + 1) < *nblocks &&
+ ReadBuffersCanStartIO(buffers[i + 1], true))
{
/* Must be consecutive block numbers. */
Assert(BufferGetBlockNumber(buffers[i + 1]) ==
BufferGetBlockNumber(buffers[i]) + 1);
+ ereport(DEBUG5,
+ errmsg("seq prepped for io: %s, offset %d",
+ DebugPrintBufferRefcount(buffers[i + 1]),
+ i + 1),
+ errhidestmt(true), errhidecontext(true));
+
io_buffers[io_buffers_len] = buffers[++i];
io_pages[io_buffers_len++] = BufferGetBlock(buffers[i]);
}
- io_start = pgstat_prepare_io_time(track_io_timing);
- smgrreadv(operation->smgr, forknum, io_first_block, io_pages, io_buffers_len);
- pgstat_count_io_op_time(io_object, io_context, IOOP_READ, io_start,
- 1, io_buffers_len * BLCKSZ);
-
- /* Verify each block we read, and terminate the I/O. */
- for (int j = 0; j < io_buffers_len; ++j)
- {
- BufferDesc *bufHdr;
- Block bufBlock;
-
- if (persistence == RELPERSISTENCE_TEMP)
- {
- bufHdr = GetLocalBufferDescriptor(-io_buffers[j] - 1);
- bufBlock = LocalBufHdrGetBlock(bufHdr);
- }
- else
- {
- bufHdr = GetBufferDescriptor(io_buffers[j] - 1);
- bufBlock = BufHdrGetBlock(bufHdr);
- }
-
- /* check for garbage data */
- if (!PageIsVerifiedExtended((Page) bufBlock, io_first_block + j,
- PIV_LOG_WARNING | PIV_REPORT_STAT))
- {
- if ((operation->flags & READ_BUFFERS_ZERO_ON_ERROR) || zero_damaged_pages)
- {
- ereport(WARNING,
- (errcode(ERRCODE_DATA_CORRUPTED),
- errmsg("invalid page in block %u of relation %s; zeroing out page",
- io_first_block + j,
- relpath(operation->smgr->smgr_rlocator, forknum).str)));
- memset(bufBlock, 0, BLCKSZ);
- }
- else
- ereport(ERROR,
- (errcode(ERRCODE_DATA_CORRUPTED),
- errmsg("invalid page in block %u of relation %s",
- io_first_block + j,
- relpath(operation->smgr->smgr_rlocator, forknum).str)));
- }
-
- /* Set BM_VALID, terminate IO, and wake up any waiters */
- if (persistence == RELPERSISTENCE_TEMP)
- TerminateLocalBufferIO(bufHdr, false, BM_VALID, true);
- else
- TerminateBufferIO(bufHdr, false, BM_VALID, true, true);
-
- /* Report I/Os as completing individually. */
- TRACE_POSTGRESQL_BUFFER_READ_DONE(forknum, io_first_block + j,
- operation->smgr->smgr_rlocator.locator.spcOid,
- operation->smgr->smgr_rlocator.locator.dbOid,
- operation->smgr->smgr_rlocator.locator.relNumber,
- operation->smgr->smgr_rlocator.backend,
- false);
- }
-
- if (VacuumCostActive)
- VacuumCostBalance += VacuumCostPageMiss * io_buffers_len;
+ pgaio_io_get_wref(ioh, &operation->io_wref);
+
+ pgaio_io_set_handle_data_32(ioh, (uint32 *) io_buffers, io_buffers_len);
+
+ pgaio_io_register_callbacks(ioh,
+ persistence == RELPERSISTENCE_TEMP ?
+ PGAIO_HCB_LOCAL_BUFFER_READV :
+ PGAIO_HCB_SHARED_BUFFER_READV,
+ flags);
+
+ pgaio_io_set_flag(ioh, ioh_flags);
+
+ did_start_io = true;
+ smgrstartreadv(ioh, operation->smgr, forknum, io_first_block,
+ io_pages, io_buffers_len);
+ ioh = NULL;
+
+ /* not obvious what we'd use for time */
+ pgstat_count_io_op(io_object, io_context, IOOP_READ,
+ 1, io_buffers_len * BLCKSZ);
+
+ *nblocks = io_buffers_len;
+ break;
+ }
+
+ if (ioh)
+ {
+ pgaio_io_release(ioh);
+ ioh = NULL;
}
+
+ return did_start_io;
}
/*
--
2.48.1.76.g4e746b1a31.dirty
[text/x-diff] v2.6-0024-WIP-aio-read_stream.c-adjustments-for-real-AIO.patch (3.7K, ../../ge6nsuddurhpmll3xj22vucvqwp4agqz6ndtcf2mhyeydzarst@l75dman5x53p/25-v2.6-0024-WIP-aio-read_stream.c-adjustments-for-real-AIO.patch)
download | inline diff:
From dacf255f86738d5398995c291668734e1f670e60 Mon Sep 17 00:00:00 2001
From: Andres Freund <andres@anarazel.de>
Date: Tue, 11 Feb 2025 18:21:47 -0500
Subject: [PATCH v2.6 24/34] WIP: aio: read_stream.c adjustments for real AIO
Comments need to be fixed.
The batching logic probably needs to be adjusted.
---
src/backend/storage/aio/read_stream.c | 31 +++++++++++++++++++++++++--
1 file changed, 29 insertions(+), 2 deletions(-)
diff --git a/src/backend/storage/aio/read_stream.c b/src/backend/storage/aio/read_stream.c
index e58e0edf221..508e145efcc 100644
--- a/src/backend/storage/aio/read_stream.c
+++ b/src/backend/storage/aio/read_stream.c
@@ -32,10 +32,15 @@
* calls. Looking further ahead would pin many buffers and perform
* speculative work for no benefit.
*
+ * FIXME: This only applies to io_method == sync, otherwise this path is not
+ * used.
+ *
* C) I/O is necessary, it appears to be random, and this system supports
* read-ahead advice. We'll look further ahead in order to reach the
* configured level of I/O concurrency.
*
+ * FIXME: restriction to random only applies to io_method == sync
+ *
* The distance increases rapidly and decays slowly, so that it moves towards
* those levels as different I/O patterns are discovered. For example, a
* sequential scan of fully cached data doesn't bother looking ahead, but a
@@ -90,6 +95,7 @@
#include "postgres.h"
#include "miscadmin.h"
+#include "storage/aio.h"
#include "storage/fd.h"
#include "storage/smgr.h"
#include "storage/read_stream.h"
@@ -116,6 +122,7 @@ struct ReadStream
int16 pinned_buffers;
int16 distance;
int16 initialized_buffers;
+ bool sync_mode;
bool advice_enabled;
bool temporary;
@@ -458,6 +465,19 @@ read_stream_look_ahead(ReadStream *stream, bool suppress_advice)
{
int16 buffer_limit;
+ if (stream->distance > (io_combine_limit * 8))
+ {
+ if (stream->pinned_buffers + stream->pending_read_nblocks > ((stream->distance * 3) / 4))
+ {
+ return;
+ }
+ }
+
+ /*
+ * Try to amortize cost of submitting IOs over multiple IOs.
+ */
+ pgaio_enter_batchmode();
+
/*
* Check how many pins we could acquire now. We do this here rather than
* pushing it down into read_stream_start_pending_read(), because it
@@ -524,6 +544,7 @@ read_stream_look_ahead(ReadStream *stream, bool suppress_advice)
{
/* And we've hit a limit. Rewind, and stop here. */
read_stream_unget_block(stream, blocknum);
+ pgaio_exit_batchmode();
return;
}
}
@@ -549,6 +570,8 @@ read_stream_look_ahead(ReadStream *stream, bool suppress_advice)
stream->distance == 0) &&
stream->ios_in_progress < stream->max_ios)
read_stream_start_pending_read(stream, buffer_limit, suppress_advice);
+
+ pgaio_exit_batchmode();
}
/*
@@ -672,6 +695,8 @@ read_stream_begin_impl(int flags,
stream->per_buffer_data = (void *)
MAXALIGN(&stream->ios[Max(1, max_ios)]);
+ stream->sync_mode = io_method == IOMETHOD_SYNC;
+
#ifdef USE_PREFETCH
/*
@@ -680,7 +705,8 @@ read_stream_begin_impl(int flags,
* (overriding our detection heuristics), and max_ios hasn't been set to
* zero.
*/
- if ((io_direct_flags & IO_DIRECT_DATA) == 0 &&
+ if (stream->sync_mode &&
+ (io_direct_flags & IO_DIRECT_DATA) == 0 &&
(flags & READ_STREAM_SEQUENTIAL) == 0 &&
max_ios > 0)
stream->advice_enabled = true;
@@ -923,7 +949,8 @@ read_stream_next_buffer(ReadStream *stream, void **per_buffer_data)
if (++stream->oldest_io_index == stream->max_ios)
stream->oldest_io_index = 0;
- if (stream->ios[io_index].op.flags & READ_BUFFERS_ISSUE_ADVICE)
+ if (!stream->sync_mode ||
+ stream->ios[io_index].op.flags & READ_BUFFERS_ISSUE_ADVICE)
{
/* Distance ramps up fast (behavior C). */
distance = stream->distance * 2;
--
2.48.1.76.g4e746b1a31.dirty
[text/x-diff] v2.6-0025-aio-Add-test_aio-module.patch (42.2K, ../../ge6nsuddurhpmll3xj22vucvqwp4agqz6ndtcf2mhyeydzarst@l75dman5x53p/26-v2.6-0025-aio-Add-test_aio-module.patch)
download | inline diff:
From ed3fffb92575e43444e91d96a156c2bfc3512821 Mon Sep 17 00:00:00 2001
From: Andres Freund <andres@anarazel.de>
Date: Wed, 22 Jan 2025 13:44:54 -0500
Subject: [PATCH v2.6 25/34] aio: Add test_aio module
---
src/include/storage/buf_internals.h | 6 +
src/backend/storage/buffer/bufmgr.c | 8 +-
src/test/modules/Makefile | 1 +
src/test/modules/meson.build | 1 +
src/test/modules/test_aio/.gitignore | 2 +
src/test/modules/test_aio/Makefile | 27 +
src/test/modules/test_aio/meson.build | 37 ++
src/test/modules/test_aio/t/001_aio.pl | 667 ++++++++++++++++++++
src/test/modules/test_aio/test_aio--1.0.sql | 97 +++
src/test/modules/test_aio/test_aio.c | 616 ++++++++++++++++++
src/test/modules/test_aio/test_aio.control | 3 +
11 files changed, 1459 insertions(+), 6 deletions(-)
create mode 100644 src/test/modules/test_aio/.gitignore
create mode 100644 src/test/modules/test_aio/Makefile
create mode 100644 src/test/modules/test_aio/meson.build
create mode 100644 src/test/modules/test_aio/t/001_aio.pl
create mode 100644 src/test/modules/test_aio/test_aio--1.0.sql
create mode 100644 src/test/modules/test_aio/test_aio.c
create mode 100644 src/test/modules/test_aio/test_aio.control
diff --git a/src/include/storage/buf_internals.h b/src/include/storage/buf_internals.h
index 257f8beeeec..3d7ebf96a7e 100644
--- a/src/include/storage/buf_internals.h
+++ b/src/include/storage/buf_internals.h
@@ -434,6 +434,12 @@ extern void IssuePendingWritebacks(WritebackContext *wb_context, IOContext io_co
extern void ScheduleBufferTagForWriteback(WritebackContext *wb_context,
IOContext io_context, BufferTag *tag);
+/* solely to make it easier to write tests */
+extern bool StartBufferIO(BufferDesc *buf, bool forInput, bool nowait);
+extern void TerminateBufferIO(BufferDesc *buf, bool clear_dirty, uint32 set_flag_bits,
+ bool forget_owner, bool syncio);
+
+
/* freelist.c */
extern IOContext IOContextForStrategy(BufferAccessStrategy strategy);
extern BufferDesc *StrategyGetBuffer(BufferAccessStrategy strategy,
diff --git a/src/backend/storage/buffer/bufmgr.c b/src/backend/storage/buffer/bufmgr.c
index 60df9eb8cba..20544b39ef9 100644
--- a/src/backend/storage/buffer/bufmgr.c
+++ b/src/backend/storage/buffer/bufmgr.c
@@ -515,10 +515,6 @@ static uint32 WaitBufHdrUnlocked(BufferDesc *buf);
static int SyncOneBuffer(int buf_id, bool skip_recently_used,
WritebackContext *wb_context);
static void WaitIO(BufferDesc *buf);
-static bool StartBufferIO(BufferDesc *buf, bool forInput, bool nowait);
-static void TerminateBufferIO(BufferDesc *buf, bool clear_dirty,
- uint32 set_flag_bits, bool forget_owner,
- bool syncio);
static void AbortBufferIO(Buffer buffer);
static void shared_buffer_write_error_callback(void *arg);
static void local_buffer_write_error_callback(void *arg);
@@ -5780,7 +5776,7 @@ WaitIO(BufferDesc *buf)
* find out if they can perform the I/O as part of a larger operation, without
* waiting for the answer or distinguishing the reasons why not.
*/
-static bool
+bool
StartBufferIO(BufferDesc *buf, bool forInput, bool nowait)
{
uint32 buf_state;
@@ -5837,7 +5833,7 @@ StartBufferIO(BufferDesc *buf, bool forInput, bool nowait)
* resource owner. (forget_owner=false is used when the resource owner itself
* is being released)
*/
-static void
+void
TerminateBufferIO(BufferDesc *buf, bool clear_dirty, uint32 set_flag_bits,
bool forget_owner, bool syncio)
{
diff --git a/src/test/modules/Makefile b/src/test/modules/Makefile
index 4e4be3fa511..aa1d27bbed3 100644
--- a/src/test/modules/Makefile
+++ b/src/test/modules/Makefile
@@ -14,6 +14,7 @@ SUBDIRS = \
oauth_validator \
plsample \
spgist_name_ops \
+ test_aio \
test_bloomfilter \
test_copy_callbacks \
test_custom_rmgrs \
diff --git a/src/test/modules/meson.build b/src/test/modules/meson.build
index 2b057451473..4fda1acfb32 100644
--- a/src/test/modules/meson.build
+++ b/src/test/modules/meson.build
@@ -1,5 +1,6 @@
# Copyright (c) 2022-2025, PostgreSQL Global Development Group
+subdir('test_aio')
subdir('brin')
subdir('commit_ts')
subdir('delay_execution')
diff --git a/src/test/modules/test_aio/.gitignore b/src/test/modules/test_aio/.gitignore
new file mode 100644
index 00000000000..716e17f5a2a
--- /dev/null
+++ b/src/test/modules/test_aio/.gitignore
@@ -0,0 +1,2 @@
+# Generated subdirectories
+/tmp_check/
diff --git a/src/test/modules/test_aio/Makefile b/src/test/modules/test_aio/Makefile
new file mode 100644
index 00000000000..87d5315ba00
--- /dev/null
+++ b/src/test/modules/test_aio/Makefile
@@ -0,0 +1,27 @@
+# src/test/modules/delay_execution/Makefile
+
+PGFILEDESC = "test_aio - test code for AIO"
+
+MODULE_big = test_aio
+OBJS = \
+ $(WIN32RES) \
+ test_aio.o
+
+EXTENSION = test_aio
+DATA = test_aio--1.0.sql
+
+TAP_TESTS = 1
+
+export enable_injection_points
+export with_liburing
+
+ifdef USE_PGXS
+PG_CONFIG = pg_config
+PGXS := $(shell $(PG_CONFIG) --pgxs)
+include $(PGXS)
+else
+subdir = src/test/modules/test_aio
+top_builddir = ../../../..
+include $(top_builddir)/src/Makefile.global
+include $(top_srcdir)/contrib/contrib-global.mk
+endif
diff --git a/src/test/modules/test_aio/meson.build b/src/test/modules/test_aio/meson.build
new file mode 100644
index 00000000000..ac846b2b6f3
--- /dev/null
+++ b/src/test/modules/test_aio/meson.build
@@ -0,0 +1,37 @@
+# Copyright (c) 2022-2024, PostgreSQL Global Development Group
+
+test_aio_sources = files(
+ 'test_aio.c',
+)
+
+if host_system == 'windows'
+ test_aio_sources += rc_lib_gen.process(win32ver_rc, extra_args: [
+ '--NAME', 'test_aio',
+ '--FILEDESC', 'test_aio - test code for AIO',])
+endif
+
+test_aio = shared_module('test_aio',
+ test_aio_sources,
+ kwargs: pg_test_mod_args,
+)
+test_install_libs += test_aio
+
+test_install_data += files(
+ 'test_aio.control',
+ 'test_aio--1.0.sql',
+)
+
+tests += {
+ 'name': 'test_aio',
+ 'sd': meson.current_source_dir(),
+ 'bd': meson.current_build_dir(),
+ 'tap': {
+ 'env': {
+ 'enable_injection_points': get_option('injection_points') ? 'yes' : 'no',
+ 'with_liburing': liburing.found() ? 'yes' : 'no',
+ },
+ 'tests': [
+ 't/001_aio.pl',
+ ],
+ },
+}
diff --git a/src/test/modules/test_aio/t/001_aio.pl b/src/test/modules/test_aio/t/001_aio.pl
new file mode 100644
index 00000000000..2e18c8de338
--- /dev/null
+++ b/src/test/modules/test_aio/t/001_aio.pl
@@ -0,0 +1,667 @@
+# Copyright (c) 2025, PostgreSQL Global Development Group
+
+use strict;
+use warnings FATAL => 'all';
+
+use PostgreSQL::Test::Cluster;
+use PostgreSQL::Test::Utils;
+use Test::More;
+
+
+###
+# Test io_method=worker
+###
+my $node_worker = create_node('worker');
+$node_worker->start();
+
+test_generic('worker', $node_worker);
+SKIP:
+{
+ skip 'Injection points not supported by this build', 1
+ unless $ENV{enable_injection_points} eq 'yes';
+ test_inject_worker('worker', $node_worker);
+}
+
+$node_worker->stop();
+
+
+###
+# Test io_method=io_uring
+###
+
+if ($ENV{with_liburing} eq 'yes')
+{
+ my $node_uring = create_node('io_uring');
+ $node_uring->start();
+ test_generic('io_uring', $node_uring);
+ $node_uring->stop();
+}
+
+
+###
+# Test io_method=sync
+###
+
+my $node_sync = create_node('sync');
+
+# just to have one test not use the default auto-tuning
+
+$node_sync->append_conf(
+ 'postgresql.conf', qq(
+io_max_concurrency=4
+));
+
+$node_sync->start();
+test_generic('sync', $node_sync);
+$node_sync->stop();
+
+done_testing();
+
+
+###
+# Test Helpers
+###
+
+sub create_node
+{
+ my $io_method = shift;
+
+ my $node = PostgreSQL::Test::Cluster->new($io_method);
+
+ # Want to test initdb for each IO method, otherwise we could just reuse
+ # the cluster.
+ #
+ # Unfortunately Cluster::init() puts PG_TEST_INITDB_EXTRA_OPTS after the
+ # options specified by ->extra, if somebody puts -c io_method=xyz in
+ # PG_TEST_INITDB_EXTRA_OPTS it would break this test. Fix that up if we
+ # detect it.
+ local $ENV{PG_TEST_INITDB_EXTRA_OPTS} = $ENV{PG_TEST_INITDB_EXTRA_OPTS};
+ if (defined $ENV{PG_TEST_INITDB_EXTRA_OPTS} &&
+ $ENV{PG_TEST_INITDB_EXTRA_OPTS} =~ m/io_method=/)
+ {
+ $ENV{PG_TEST_INITDB_EXTRA_OPTS} .= " -c io_method=$io_method";
+ }
+
+ $node->init(extra => [ '-c', "io_method=$io_method" ]);
+
+ $node->append_conf(
+ 'postgresql.conf', qq(
+shared_preload_libraries=test_aio
+log_min_messages = 'DEBUG3'
+log_statement=all
+restart_after_crash=false
+));
+
+ ok(1, "$io_method: initdb");
+
+ return $node;
+}
+
+sub psql_like
+{
+ my $io_method = shift;
+ my $psql = shift;
+ my $name = shift;
+ my $sql = shift;
+ my $expected_stdout = shift;
+ my $expected_stderr = shift;
+ my ($cmdret, $output);
+
+ ($output, $cmdret) = $psql->query($sql);
+
+ like($output, $expected_stdout, "$io_method: $name: expected stdout");
+ like($psql->{stderr}, $expected_stderr,
+ "$io_method: $name: expected stderr");
+ $psql->{stderr} = '';
+}
+
+sub query_wait_block
+{
+ my $io_method = shift;
+ my $node = shift;
+ my $psql = shift;
+ my $name = shift;
+ my $sql = shift;
+ my $waitfor = shift;
+
+ my $pid = $psql->query_safe('SELECT pg_backend_pid()');
+
+ $psql->{stdin} .= qq($sql;\n);
+ $psql->{run}->pump_nb();
+ ok(1, "$io_method: $name: issued sql");
+
+ $node->poll_query_until('postgres',
+ qq(SELECT wait_event FROM pg_stat_activity WHERE pid = $pid),
+ $waitfor,);
+ ok(1, "$io_method: $name: observed $waitfor wait event");
+}
+
+
+###
+# Sub-tests
+###
+
+sub test_handle
+{
+ my $io_method = shift;
+ my $node = shift;
+
+ my $psql = $node->background_psql('postgres', on_error_stop => 0);
+
+ # leak warning: implicit xact
+ psql_like(
+ $io_method,
+ $psql,
+ "handle_get() leak in implicit xact",
+ qq(SELECT handle_get()),
+ qr/^$/,
+ qr/leaked AIO handle/,
+ "$io_method: leaky handle_get() warns");
+
+ # leak warning: explicit xact
+ psql_like(
+ $io_method, $psql,
+ "handle_get() leak in explicit xact",
+ qq(BEGIN; SELECT handle_get(); COMMIT),
+ qr/^$/, qr/leaked AIO handle/);
+
+
+ # leak warning: explicit xact, rollback
+ psql_like(
+ $io_method,
+ $psql,
+ "handle_get() leak in explicit xact, rollback",
+ qq(BEGIN; SELECT handle_get(); ROLLBACK;),
+ qr/^$/,
+ qr/leaked AIO handle/);
+
+ # leak warning: subtrans
+ psql_like(
+ $io_method,
+ $psql,
+ "handle_get() leak in subxact",
+ qq(BEGIN; SAVEPOINT foo; SELECT handle_get(); COMMIT;),
+ qr/^$/,
+ qr/leaked AIO handle/);
+
+ # leak warning + error: released in different command (thus resowner)
+ psql_like(
+ $io_method,
+ $psql,
+ "handle_release() in different command",
+ qq(BEGIN; SELECT handle_get(); SELECT handle_release_last(); COMMIT;),
+ qr/^$/,
+ qr/leaked AIO handle.*release in unexpected state/ms);
+
+ # no leak, release in same command
+ psql_like(
+ $io_method,
+ $psql,
+ "handle_release() in same command",
+ qq(BEGIN; SELECT handle_get() UNION ALL SELECT handle_release_last(); COMMIT;),
+ qr/^$/,
+ qr/^$/);
+
+ # normal handle use
+ psql_like($io_method, $psql, "handle_get_release()",
+ qq(SELECT handle_get_release()),
+ qr/^$/, qr/^$/);
+
+ # should error out, API violation
+ psql_like($io_method, $psql, "handle_get_twice()",
+ qq(SELECT handle_get_release()),
+ qr/^$/, qr/^$/);
+
+ # recover after error in implicit xact
+ psql_like(
+ $io_method,
+ $psql,
+ "handle error recovery in implicit xact",
+ qq(SELECT handle_get_and_error(); SELECT 'ok', handle_get_release()),
+ qr/^|ok$/,
+ qr/ERROR.*as you command/);
+
+ # recover after error in implicit xact
+ psql_like(
+ $io_method,
+ $psql,
+ "handle error recovery in explicit xact",
+ qq(BEGIN; SELECT handle_get_and_error(); SELECT handle_get_release(), 'ok'; COMMIT;),
+ qr/^|ok$/,
+ qr/ERROR.*as you command/);
+
+ # recover after error in subtrans
+ psql_like(
+ $io_method,
+ $psql,
+ "handle error recovery in explicit subxact",
+ qq(BEGIN; SAVEPOINT foo; SELECT handle_get_and_error(); ROLLBACK TO SAVEPOINT foo; SELECT handle_get_release(); ROLLBACK;),
+ qr/^|ok$/,
+ qr/ERROR.*as you command/);
+
+ $psql->quit();
+}
+
+
+sub test_batch
+{
+ my $io_method = shift;
+ my $node = shift;
+
+ my $psql = $node->background_psql('postgres', on_error_stop => 0);
+
+ # leak warning & recovery: implicit xact
+ psql_like(
+ $io_method,
+ $psql,
+ "batch_start() leak & cleanup in implicit xact",
+ qq(SELECT batch_start()),
+ qr/^$/,
+ qr/open AIO batch at end/,
+ "$io_method: leaky batch_start() warns");
+
+ # leak warning & recovery: explicit xact
+ psql_like(
+ $io_method,
+ $psql,
+ "batch_start() leak & cleanup in explicit xact",
+ qq(BEGIN; SELECT batch_start(); COMMIT;),
+ qr/^$/,
+ qr/open AIO batch at end/,
+ "$io_method: leaky batch_start() warns");
+
+
+ # leak warning & recovery: explicit xact, rollback
+ #
+ # XXX: This doesn't fail right now, due to not getting a chance to do
+ # something at transaction command commit. That's not a correctness issue,
+ # it just means it's a bit harder to find buggy code.
+ #psql_like($io_method, $psql,
+ # "batch_start() leak & cleanup after abort",
+ # qq(BEGIN; SELECT batch_start(); ROLLBACK;),
+ # qr/^$/,
+ # qr/open AIO batch at end/, "$io_method: leaky batch_start() warns");
+
+ # no warning, batch closed in same command
+ psql_like(
+ $io_method,
+ $psql,
+ "batch_start(), batch_end() works",
+ qq(SELECT batch_start() UNION ALL SELECT batch_end()),
+ qr/^$/,
+ qr/^$/,
+ "$io_method: batch_start(), batch_end()");
+
+ $psql->quit();
+}
+
+sub test_io_error
+{
+ my $io_method = shift;
+ my $node = shift;
+ my ($ret, $output);
+
+ my $psql = $node->background_psql('postgres', on_error_stop => 0);
+
+ # verify the error is reported in custom C code
+ ($output, $ret) = $psql->query(
+ qq(SELECT read_rel_block_ll('tbl_corr', 1, wait_complete=>true);));
+ is($ret, 1, "$io_method: read_rel_block_ll() of tbl_corr page fails");
+ like(
+ $psql->{stderr},
+ qr/invalid page in block 1 of relation base\/.*/,
+ "$io_method: read_rel_block_ll() of tbl_corr page reports error");
+ $psql->{stderr} = '';
+
+ # verify the error is reported for bufmgr reads
+ ($output, $ret) =
+ $psql->query(qq(SELECT count(*) FROM tbl_corr WHERE ctid = '(1, 1)'));
+ is($ret, 1, "$io_method: tid scan reading tbl_corr block fails");
+ like(
+ $psql->{stderr},
+ qr/invalid page in block 1 of relation base\/.*/,
+ "$io_method: tid scan reading tbl_corr block reports error");
+ $psql->{stderr} = '';
+
+ # verify the error is reported for bufmgr reads
+ ($output, $ret) =
+ $psql->query(qq(SELECT count(*) FROM tbl_corr WHERE ctid = '(1, 1)'));
+ is($ret, 1, "$io_method: sequential scan reading tbl_corr block fails");
+ like(
+ $psql->{stderr},
+ qr/invalid page in block 1 of relation base\/.*/,
+ "$io_method: sequential scan reading tbl_corr block reports error");
+ $psql->{stderr} = '';
+
+ $psql->quit();
+}
+
+# Test interplay between StartBufferIO and TerminateBufferIO
+sub test_startwait_io
+{
+ my $io_method = shift;
+ my $node = shift;
+ my ($ret, $output);
+
+ my $psql_a = $node->background_psql('postgres', on_error_stop => 0);
+ my $psql_b = $node->background_psql('postgres', on_error_stop => 0);
+
+ # create a buffer we can play around with
+ ($output, $ret) =
+ $psql_a->query(qq(SELECT buffer_create_toy('tbl_ok', 1);));
+ is($ret, 0, "$io_method: create toy succeeds");
+ like($output, qr/^\d+$/, "$io_method: create toy returns numeric");
+ my $buf_id = $output;
+
+ # check that one backend can perform StartBufferIO
+ psql_like(
+ $io_method,
+ $psql_a,
+ "first StartBufferIO",
+ qq(SELECT buffer_call_start_io($buf_id, for_input=>true, nowait=>false);),
+ qr/^t$/,
+ qr/^$/);
+
+ # but not twice on the same buffer (non-waiting)
+ psql_like(
+ $io_method,
+ $psql_a,
+ "second StartBufferIO fails, same session",
+ qq(SELECT buffer_call_start_io($buf_id, for_input=>true, nowait=>true);),
+ qr/^f$/,
+ qr/^$/);
+ psql_like(
+ $io_method,
+ $psql_b,
+ "second StartBufferIO fails, other session",
+ qq(SELECT buffer_call_start_io($buf_id, for_input=>true, nowait=>true);),
+ qr/^f$/,
+ qr/^$/);
+
+ # start io in a different session, will block
+ query_wait_block(
+ $io_method,
+ $node,
+ $psql_b,
+ "blocking start buffer io",
+ qq(SELECT buffer_call_start_io($buf_id, for_input=>true, nowait=>false);),
+ "BufferIo");
+
+ # Terminate the IO, without marking it as success, this should trigger the
+ # waiting session to be able to start the io
+ ($output, $ret) = $psql_a->query(
+ qq(SELECT buffer_call_terminate_io($buf_id, for_input=>true, succeed=>false, io_error=>false, syncio=>true);)
+ );
+ is($ret, 0,
+ "$io_method: blocking start buffer io, terminating io, not valid");
+
+ # Because the IO was terminated, but not marked as valid, second session should get the right to start io
+ pump_until($psql_b->{run}, $psql_b->{timeout}, \$psql_b->{stdout}, qr/t/);
+ ok(1, "$io_method: blocking start buffer io, can start io");
+
+ # terminate the IO again
+ $psql_b->query_safe(
+ qq(SELECT buffer_call_terminate_io($buf_id, for_input=>true, succeed=>false, io_error=>false, syncio=>true);)
+ );
+
+
+ # same as the above scenario, but mark IO as having succeeded
+ ($output, $ret) =
+ $psql_a->query(qq(SELECT buffer_call_start_io($buf_id, true, false);));
+ is($ret, 0,
+ "$io_method: blocking buffer io w/ success: first start buffer io succeeds"
+ );
+ is($output, "t",
+ "$io_method: blocking buffer io w/ success: first start buffer io returns true"
+ );
+
+
+ # start io in a different session, will block
+ query_wait_block(
+ $io_method,
+ $node,
+ $psql_b,
+ "blocking start buffer io",
+ qq(SELECT buffer_call_start_io($buf_id, for_input=>true, nowait=>false);),
+ "BufferIo");
+
+ ($output, $ret) = $psql_a->query(
+ qq(SELECT buffer_call_terminate_io($buf_id, for_input=>true, succeed=>true, io_error=>false, syncio=>true);)
+ );
+ is($ret, 0,
+ "$io_method: blocking start buffer io, terminating IO, valid");
+
+ # Because the IO was terminated, and marked as valid, second session should complete but not need io
+ pump_until($psql_b->{run}, $psql_b->{timeout}, \$psql_b->{stdout}, qr/f/);
+ ok(1, "$io_method: blocking start buffer io, no need to start io");
+
+
+ # buffer is valid now, make it invalid again
+ $buf_id = $psql_a->query_safe(qq(SELECT buffer_create_toy('tbl_ok', 1);));
+
+ $psql_a->quit();
+ $psql_b->quit();
+}
+
+# Test that if the backend issuing a read doesn't wait for the IO's
+# completion, another backend can complete the IO
+sub test_complete_foreign
+{
+ my $io_method = shift;
+ my $node = shift;
+ my ($ret, $output);
+
+ my $psql_a = $node->background_psql('postgres', on_error_stop => 0);
+ my $psql_b = $node->background_psql('postgres', on_error_stop => 0);
+
+ # Issue IO without waiting for completion, then sleep
+ $psql_a->query_safe(
+ qq(SELECT read_rel_block_ll('tbl_ok', 1, wait_complete=>false);));
+
+ # Check that another backend can read the relevant block
+ psql_like(
+ $io_method,
+ $psql_b,
+ "completing read started by sleeping backend",
+ qq(SELECT count(*) FROM tbl_ok WHERE ctid = '(1,1)' LIMIT 1),
+ qr/^1$/,
+ qr/^$/);
+
+ # Issue IO without waiting for completion, then exit
+ $psql_a->query_safe(
+ qq(SELECT read_rel_block_ll('tbl_ok', 1, wait_complete=>false);));
+ $psql_a->reconnect_and_clear();
+
+ # Check that another backend can read the relevant block
+ psql_like(
+ $io_method,
+ $psql_b,
+ "completing read started by exited backend",
+ qq(SELECT count(*) FROM tbl_ok WHERE ctid = '(1,1)' LIMIT 1),
+ qr/^1$/,
+ qr/^$/);
+
+ # Read a tbl_corr block, then sleep. The other session will retry the IO
+ # and also fail. The easiest thing to verify that seems to be to check
+ # that both are in the log.
+ my $log_location = -s $node->logfile;
+ $psql_a->query_safe(
+ qq(SELECT read_rel_block_ll('tbl_corr', 1, wait_complete=>false);));
+
+ psql_like(
+ $io_method,
+ $psql_b,
+ "completing read of tbl_corr block started by other backend",
+ qq(SELECT count(*) FROM tbl_corr WHERE ctid = '(1,1)' LIMIT 1),
+ qr/^$/,
+ qr/invalid page in block/);
+
+ # The log message issued for the read_rel_block_ll() should be logged as a LOG
+ $node->wait_for_log(qr/LOG[^\n]+invalid page in/, $log_location);
+ ok(1,
+ "$io_method: completing read of tbl_corr block started by other backend: LOG message for background read"
+ );
+
+ # But for the SELECT, it should be an ERROR
+ $log_location =
+ $node->wait_for_log(qr/ERROR[^\n]+invalid page in/, $log_location);
+ ok(1,
+ "$io_method: completing read of tbl_corr block started by other backend: ERROR message for foreground read"
+ );
+
+ $psql_a->quit();
+ $psql_b->quit();
+}
+
+sub test_inject
+{
+ my $io_method = shift;
+ my $node = shift;
+ my ($ret, $output);
+
+ my $psql = $node->background_psql('postgres', on_error_stop => 0);
+
+ # injected what we'd expect
+ $psql->query_safe(qq(SELECT inj_io_short_read_attach(8192);));
+ $psql->query_safe(qq(SELECT invalidate_rel_block('tbl_ok', 2);));
+ ($output, $ret) =
+ $psql->query(qq(SELECT count(*) FROM tbl_ok WHERE ctid = '(2, 1)';));
+ is($ret, 0,
+ "$io_method: injection point not triggering failure succeeds");
+
+ # injected a read shorter than a single block, expecting error
+ $psql->query_safe(qq(SELECT inj_io_short_read_attach(17);));
+ $psql->query_safe(qq(SELECT invalidate_rel_block('tbl_ok', 2);));
+ ($output, $ret) =
+ $psql->query(qq(SELECT count(*) FROM tbl_ok WHERE ctid = '(2, 1)';));
+ is($ret, 1, "$io_method: single block short read fails");
+ like(
+ $psql->{stderr},
+ qr/ERROR:.*could not read blocks 2\.\.2 in file "base\/.*": read only 0 of 8192 bytes/,
+ "$io_method: single block short read reports error");
+ $psql->{stderr} = '';
+
+ # shorten multi-block read to a single block, should retry
+ $psql->query_safe(
+ qq(
+SELECT invalidate_rel_block('tbl_ok', 0);
+SELECT invalidate_rel_block('tbl_ok', 1);
+SELECT invalidate_rel_block('tbl_ok', 2);
+SELECT inj_io_short_read_attach(8192);
+ ));
+ ($output, $ret) = $psql->query(qq(SELECT count(*) FROM tbl_ok;));
+ is($ret, 0, "$io_method: multi block short read is retried");
+
+ # verify that page verification errors are detected even as part of a
+ # shortened multi-block read (tbl_corr, block 1 is tbl_corred)
+ $psql->query_safe(
+ qq(
+SELECT invalidate_rel_block('tbl_corr', 0);
+SELECT invalidate_rel_block('tbl_corr', 1);
+SELECT invalidate_rel_block('tbl_corr', 2);
+SELECT inj_io_short_read_attach(8192);
+ ));
+ ($output, $ret) =
+ $psql->query(qq(SELECT count(*) FROM tbl_corr WHERE ctid < '(2, 1)'));
+ is($ret, 1,
+ "$io_method: shortened multi-block read detects invalid page");
+ like(
+ $psql->{stderr},
+ qr/ERROR:.*invalid page in block 1 of relation base\/.*/,
+ "$io_method: shortened multi-block reads reports invalid page");
+ $psql->{stderr} = '';
+
+ # trigger a hard error, should error out
+ $psql->query_safe(
+ qq(
+SELECT inj_io_short_read_attach(-errno_from_string('EIO'));
+SELECT invalidate_rel_block('tbl_ok', 2);
+ ));
+ ($output, $ret) =
+ $psql->query(qq(SELECT count(*) FROM tbl_ok; SELECT 1;));
+ is($ret, 1, "$io_method: hard IO error is detected");
+ like(
+ $psql->{stderr},
+ qr/ERROR:.*could not read blocks 2\.\.2 in file \"base\/.*\": Input\/output error/,
+ "$io_method: hard IO error is reported");
+ $psql->{stderr} = '';
+
+ $psql->query_safe(
+ qq(
+SELECT inj_io_short_read_detach();
+ ));
+
+ $psql->quit();
+}
+
+sub test_inject_worker
+{
+ my $io_method = shift;
+ my $node = shift;
+ my ($ret, $output);
+
+ my $psql = $node->background_psql('postgres', on_error_stop => 0);
+
+ # trigger a failure to reopen, should error out, but should recover
+ $psql->query_safe(
+ qq(
+SELECT inj_io_reopen_attach();
+SELECT invalidate_rel_block('tbl_ok', 1);
+ ));
+ ($output, $ret) = $psql->query(qq(SELECT count(*) FROM tbl_ok;));
+ is($ret, 1, "$io_method: failure to open is detected");
+ like(
+ $psql->{stderr},
+ qr/ERROR:.*could not read blocks 1\.\.1 in file "base\/.*": No such file or directory/,
+ "$io_method: failure to open is reported");
+ $psql->{stderr} = '';
+
+ $psql->query_safe(
+ qq(
+SELECT inj_io_reopen_detach();
+ ));
+
+ # check that we indeed recover
+ ($output, $ret) = $psql->query(qq(SELECT count(*) FROM tbl_ok;));
+ is($ret, 0, "$io_method: recovers from failure to open ");
+
+
+ $psql->quit();
+}
+
+sub test_generic
+{
+ my $io_method = shift;
+ my $node = shift;
+
+ is($node->safe_psql('postgres', 'SHOW io_method'),
+ $io_method, "$io_method: io_method set correctly");
+
+ $node->safe_psql(
+ 'postgres', qq(
+CREATE EXTENSION test_aio;
+CREATE TABLE tbl_corr(data int not null) WITH (AUTOVACUUM_ENABLED = false);
+CREATE TABLE tbl_ok(data int not null) WITH (AUTOVACUUM_ENABLED = false);
+
+INSERT INTO tbl_corr SELECT generate_series(1, 10000);
+INSERT INTO tbl_ok SELECT generate_series(1, 10000);
+SELECT grow_rel('tbl_corr', 16);
+SELECT grow_rel('tbl_ok', 16);
+
+SELECT corrupt_rel_block('tbl_corr', 1);
+CHECKPOINT;
+));
+
+ test_handle($io_method, $node);
+ test_io_error($io_method, $node);
+ test_batch($io_method, $node);
+ test_startwait_io($io_method, $node);
+ test_complete_foreign($io_method, $node);
+
+ SKIP:
+ {
+ skip 'Injection points not supported by this build', 1
+ unless $ENV{enable_injection_points} eq 'yes';
+ test_inject($io_method, $node);
+ }
+}
diff --git a/src/test/modules/test_aio/test_aio--1.0.sql b/src/test/modules/test_aio/test_aio--1.0.sql
new file mode 100644
index 00000000000..3d4e9bb070f
--- /dev/null
+++ b/src/test/modules/test_aio/test_aio--1.0.sql
@@ -0,0 +1,97 @@
+/* src/test/modules/test_aio/test_aio--1.0.sql */
+
+-- complain if script is sourced in psql, rather than via CREATE EXTENSION
+\echo Use "CREATE EXTENSION test_aio" to load this file. \quit
+
+
+CREATE FUNCTION errno_from_string(sym text)
+RETURNS pg_catalog.int4 STRICT
+AS 'MODULE_PATHNAME' LANGUAGE C;
+
+
+CREATE FUNCTION grow_rel(rel regclass, nblocks int)
+RETURNS pg_catalog.void STRICT
+AS 'MODULE_PATHNAME' LANGUAGE C;
+
+
+CREATE FUNCTION corrupt_rel_block(rel regclass, blockno int)
+RETURNS pg_catalog.void STRICT
+AS 'MODULE_PATHNAME' LANGUAGE C;
+
+CREATE FUNCTION read_rel_block_ll(rel regclass, blockno int, wait_complete bool)
+RETURNS pg_catalog.void STRICT
+AS 'MODULE_PATHNAME' LANGUAGE C;
+
+CREATE FUNCTION invalidate_rel_block(rel regclass, blockno int)
+RETURNS pg_catalog.void STRICT
+AS 'MODULE_PATHNAME' LANGUAGE C;
+
+CREATE FUNCTION buffer_create_toy(rel regclass, blockno int4)
+RETURNS pg_catalog.int4 STRICT
+AS 'MODULE_PATHNAME' LANGUAGE C;
+
+CREATE FUNCTION buffer_call_start_io(buffer int, for_input bool, nowait bool)
+RETURNS pg_catalog.bool STRICT
+AS 'MODULE_PATHNAME' LANGUAGE C;
+
+CREATE FUNCTION buffer_call_terminate_io(buffer int, for_input bool, succeed bool, io_error bool, syncio bool)
+RETURNS pg_catalog.void STRICT
+AS 'MODULE_PATHNAME' LANGUAGE C;
+
+
+
+/*
+ * Handle related functions
+ */
+CREATE FUNCTION handle_get_and_error()
+RETURNS pg_catalog.void STRICT
+AS 'MODULE_PATHNAME' LANGUAGE C;
+
+CREATE FUNCTION handle_get_twice()
+RETURNS pg_catalog.void STRICT
+AS 'MODULE_PATHNAME' LANGUAGE C;
+
+CREATE FUNCTION handle_get()
+RETURNS pg_catalog.void STRICT
+AS 'MODULE_PATHNAME' LANGUAGE C;
+
+CREATE FUNCTION handle_get_release()
+RETURNS pg_catalog.void STRICT
+AS 'MODULE_PATHNAME' LANGUAGE C;
+
+CREATE FUNCTION handle_release_last()
+RETURNS pg_catalog.void STRICT
+AS 'MODULE_PATHNAME' LANGUAGE C;
+
+
+/*
+ * Batchmode related functions
+ */
+CREATE FUNCTION batch_start()
+RETURNS pg_catalog.void STRICT
+AS 'MODULE_PATHNAME' LANGUAGE C;
+
+CREATE FUNCTION batch_end()
+RETURNS pg_catalog.void STRICT
+AS 'MODULE_PATHNAME' LANGUAGE C;
+
+
+
+/*
+ * Injection point related functions
+ */
+CREATE FUNCTION inj_io_short_read_attach(result int)
+RETURNS pg_catalog.void STRICT
+AS 'MODULE_PATHNAME' LANGUAGE C;
+
+CREATE FUNCTION inj_io_short_read_detach()
+RETURNS pg_catalog.void STRICT
+AS 'MODULE_PATHNAME' LANGUAGE C;
+
+CREATE FUNCTION inj_io_reopen_attach()
+RETURNS pg_catalog.void STRICT
+AS 'MODULE_PATHNAME' LANGUAGE C;
+
+CREATE FUNCTION inj_io_reopen_detach()
+RETURNS pg_catalog.void STRICT
+AS 'MODULE_PATHNAME' LANGUAGE C;
diff --git a/src/test/modules/test_aio/test_aio.c b/src/test/modules/test_aio/test_aio.c
new file mode 100644
index 00000000000..81b4d732206
--- /dev/null
+++ b/src/test/modules/test_aio/test_aio.c
@@ -0,0 +1,616 @@
+/*-------------------------------------------------------------------------
+ *
+ * delay_execution.c
+ * Test module to allow delay between parsing and execution of a query.
+ *
+ * The delay is implemented by taking and immediately releasing a specified
+ * advisory lock. If another process has previously taken that lock, the
+ * current process will be blocked until the lock is released; otherwise,
+ * there's no effect. This allows an isolationtester script to reliably
+ * test behaviors where some specified action happens in another backend
+ * between parsing and execution of any desired query.
+ *
+ * Copyright (c) 2020-2025, PostgreSQL Global Development Group
+ *
+ * IDENTIFICATION
+ * src/test/modules/delay_execution/delay_execution.c
+ *
+ *-------------------------------------------------------------------------
+ */
+
+#include "postgres.h"
+
+#include "access/relation.h"
+#include "fmgr.h"
+#include "storage/aio.h"
+#include "storage/aio_internal.h"
+#include "storage/buf_internals.h"
+#include "storage/bufmgr.h"
+#include "storage/ipc.h"
+#include "storage/lwlock.h"
+#include "utils/builtins.h"
+#include "utils/injection_point.h"
+#include "utils/rel.h"
+
+
+PG_MODULE_MAGIC;
+
+
+typedef struct InjIoErrorState
+{
+ bool enabled_short_read;
+ bool enabled_reopen;
+
+ bool short_read_result_set;
+ int short_read_result;
+} InjIoErrorState;
+
+static InjIoErrorState * inj_io_error_state;
+
+/* Shared memory init callbacks */
+static shmem_request_hook_type prev_shmem_request_hook = NULL;
+static shmem_startup_hook_type prev_shmem_startup_hook = NULL;
+
+
+static PgAioHandle *last_handle;
+
+
+
+static void
+test_aio_shmem_request(void)
+{
+ if (prev_shmem_request_hook)
+ prev_shmem_request_hook();
+
+ RequestAddinShmemSpace(sizeof(InjIoErrorState));
+}
+
+static void
+test_aio_shmem_startup(void)
+{
+ bool found;
+
+ if (prev_shmem_startup_hook)
+ prev_shmem_startup_hook();
+
+ /* Create or attach to the shared memory state */
+ LWLockAcquire(AddinShmemInitLock, LW_EXCLUSIVE);
+
+ inj_io_error_state = ShmemInitStruct("injection_points",
+ sizeof(InjIoErrorState),
+ &found);
+
+ if (!found)
+ {
+ /*
+ * First time through, so initialize. This is shared with the dynamic
+ * initialization using a DSM.
+ */
+ inj_io_error_state->enabled_short_read = false;
+ inj_io_error_state->enabled_reopen = false;
+
+#ifdef USE_INJECTION_POINTS
+ InjectionPointAttach("AIO_PROCESS_COMPLETION_BEFORE_SHARED",
+ "test_aio",
+ "inj_io_short_read",
+ NULL,
+ 0);
+ InjectionPointLoad("AIO_PROCESS_COMPLETION_BEFORE_SHARED");
+
+ InjectionPointAttach("AIO_WORKER_AFTER_REOPEN",
+ "test_aio",
+ "inj_io_reopen",
+ NULL,
+ 0);
+ InjectionPointLoad("AIO_WORKER_AFTER_REOPEN");
+
+#endif
+ }
+ else
+ {
+#ifdef USE_INJECTION_POINTS
+ InjectionPointLoad("AIO_PROCESS_COMPLETION_BEFORE_SHARED");
+ InjectionPointLoad("AIO_WORKER_AFTER_REOPEN");
+ elog(LOG, "injection point loaded");
+#endif
+ }
+
+ LWLockRelease(AddinShmemInitLock);
+}
+
+void
+_PG_init(void)
+{
+ if (!process_shared_preload_libraries_in_progress)
+ return;
+
+ /* Shared memory initialization */
+ prev_shmem_request_hook = shmem_request_hook;
+ shmem_request_hook = test_aio_shmem_request;
+ prev_shmem_startup_hook = shmem_startup_hook;
+ shmem_startup_hook = test_aio_shmem_startup;
+}
+
+
+PG_FUNCTION_INFO_V1(errno_from_string);
+Datum
+errno_from_string(PG_FUNCTION_ARGS)
+{
+ const char *sym = text_to_cstring(PG_GETARG_TEXT_PP(0));
+
+ if (strcmp(sym, "EIO") == 0)
+ PG_RETURN_INT32(EIO);
+ else if (strcmp(sym, "EAGAIN") == 0)
+ PG_RETURN_INT32(EAGAIN);
+ else if (strcmp(sym, "EINTR") == 0)
+ PG_RETURN_INT32(EINTR);
+ else if (strcmp(sym, "ENOSPC") == 0)
+ PG_RETURN_INT32(ENOSPC);
+ else if (strcmp(sym, "EROFS") == 0)
+ PG_RETURN_INT32(EROFS);
+
+ ereport(ERROR,
+ errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ errmsg_internal("%s is not a supported errno value", sym));
+ PG_RETURN_INT32(0);
+}
+
+
+PG_FUNCTION_INFO_V1(grow_rel);
+Datum
+grow_rel(PG_FUNCTION_ARGS)
+{
+ Oid relid = PG_GETARG_OID(0);
+ uint32 nblocks = PG_GETARG_UINT32(1);
+ Relation rel;
+#define MAX_BUFFERS_TO_EXTEND_BY 64
+ Buffer victim_buffers[MAX_BUFFERS_TO_EXTEND_BY];
+
+ rel = relation_open(relid, AccessExclusiveLock);
+
+ while (nblocks > 0)
+ {
+ uint32 extend_by_pages;
+
+ extend_by_pages = Min(nblocks, MAX_BUFFERS_TO_EXTEND_BY);
+
+ ExtendBufferedRelBy(BMR_REL(rel),
+ MAIN_FORKNUM,
+ NULL,
+ 0,
+ extend_by_pages,
+ victim_buffers,
+ &extend_by_pages);
+
+ nblocks -= extend_by_pages;
+
+ for (uint32 i = 0; i < extend_by_pages; i++)
+ {
+ ReleaseBuffer(victim_buffers[i]);
+ }
+ }
+
+ relation_close(rel, NoLock);
+
+ PG_RETURN_VOID();
+}
+
+PG_FUNCTION_INFO_V1(corrupt_rel_block);
+Datum
+corrupt_rel_block(PG_FUNCTION_ARGS)
+{
+ Oid relid = PG_GETARG_OID(0);
+ BlockNumber blkno = PG_GETARG_UINT32(1);
+ Relation rel;
+ Buffer buf;
+ Page page;
+ PageHeader ph;
+
+ rel = relation_open(relid, AccessExclusiveLock);
+
+ buf = ReadBuffer(rel, blkno);
+ page = BufferGetPage(buf);
+
+ LockBuffer(buf, BUFFER_LOCK_EXCLUSIVE);
+
+ MarkBufferDirty(buf);
+
+ PageInit(page, BufferGetPageSize(buf), 0);
+
+ ph = (PageHeader) page;
+ ph->pd_special = BLCKSZ + 1;
+
+ FlushOneBuffer(buf);
+
+ LockBuffer(buf, BUFFER_LOCK_UNLOCK);
+
+ ReleaseBuffer(buf);
+
+ EvictUnpinnedBuffer(buf);
+
+ relation_close(rel, NoLock);
+
+ PG_RETURN_VOID();
+}
+
+/*
+ * Ensures a buffer for rel & blkno is in shared buffers, without actually
+ * caring about the buffer contents. Used to set up test scenarios.
+ */
+static Buffer
+create_toy_buffer(Relation rel, BlockNumber blkno)
+{
+ Buffer buf;
+ BufferDesc *buf_hdr;
+ uint32 buf_state;
+ bool was_pinned = false;
+
+ /* read buffer without erroring out */
+ buf = ReadBufferExtended(rel, MAIN_FORKNUM, blkno, RBM_ZERO_ON_ERROR, NULL);
+
+ buf_hdr = GetBufferDescriptor(buf - 1);
+
+ buf_state = LockBufHdr(buf_hdr);
+
+ /*
+ * We should be the only backend accessing this buffer. This is just a
+ * small bit of belt-and-suspenders defense, none of this code should ever
+ * run in a cluster with real data.
+ */
+ if (BUF_STATE_GET_REFCOUNT(buf_state) > 1)
+ was_pinned = true;
+ else
+ buf_state &= ~(BM_VALID | BM_DIRTY);
+
+ UnlockBufHdr(buf_hdr, buf_state);
+
+ if (was_pinned)
+ elog(ERROR, "toy buffer %d was already pinned",
+ buf);
+
+ return buf;
+}
+
+PG_FUNCTION_INFO_V1(read_rel_block_ll);
+Datum
+read_rel_block_ll(PG_FUNCTION_ARGS)
+{
+ Oid relid = PG_GETARG_OID(0);
+ BlockNumber blkno = PG_GETARG_UINT32(1);
+ bool wait_complete = PG_GETARG_BOOL(2);
+ Relation rel;
+ Buffer buf;
+ Page pages[1];
+ PgAioReturn ior;
+ PgAioHandle *ioh;
+ PgAioWaitRef iow;
+ SMgrRelation smgr;
+
+ rel = relation_open(relid, AccessExclusiveLock);
+
+ buf = create_toy_buffer(rel, blkno);
+
+ pages[0] = BufferGetBlock(buf);
+
+ ioh = pgaio_io_acquire(CurrentResourceOwner, &ior);
+ pgaio_io_get_wref(ioh, &iow);
+
+ smgr = RelationGetSmgr(rel);
+
+ StartBufferIO(GetBufferDescriptor(buf - 1), true, false);
+
+ pgaio_io_set_handle_data_32(ioh, (uint32 *) &buf, 1);
+ pgaio_io_register_callbacks(ioh, PGAIO_HCB_SHARED_BUFFER_READV, 0);
+
+ elog(LOG, "about to smgrstartreadv");
+ smgrstartreadv(ioh, smgr, MAIN_FORKNUM, blkno,
+ (void *) pages, 1);
+
+ ReleaseBuffer(buf);
+
+ if (wait_complete)
+ {
+ pgaio_wref_wait(&iow);
+
+ if (ior.result.status != ARS_OK)
+ pgaio_result_report(ior.result, &ior.target_data,
+ ior.result.status == ARS_PARTIAL ? WARNING : ERROR);
+ }
+
+ relation_close(rel, NoLock);
+
+ PG_RETURN_VOID();
+}
+
+PG_FUNCTION_INFO_V1(invalidate_rel_block);
+Datum
+invalidate_rel_block(PG_FUNCTION_ARGS)
+{
+ Oid relid = PG_GETARG_OID(0);
+ BlockNumber blkno = PG_GETARG_UINT32(1);
+ Relation rel;
+ PrefetchBufferResult pr;
+ Buffer buf;
+
+ rel = relation_open(relid, AccessExclusiveLock);
+
+ /*
+ * This is a gross hack, but there's no other API exposed that allows to
+ * get a buffer ID without actually reading the block in.
+ */
+ pr = PrefetchBuffer(rel, MAIN_FORKNUM, blkno);
+ buf = pr.recent_buffer;
+
+ if (BufferIsValid(buf))
+ {
+ /* if the buffer contents aren't valid, this'll return false */
+ if (ReadRecentBuffer(rel->rd_locator, MAIN_FORKNUM, blkno, buf))
+ {
+ LockBuffer(buf, BUFFER_LOCK_EXCLUSIVE);
+ FlushOneBuffer(buf);
+ LockBuffer(buf, BUFFER_LOCK_UNLOCK);
+ ReleaseBuffer(buf);
+
+ if (!EvictUnpinnedBuffer(buf))
+ elog(ERROR, "couldn't evict");
+ }
+ }
+
+ relation_close(rel, AccessExclusiveLock);
+
+ PG_RETURN_VOID();
+}
+
+PG_FUNCTION_INFO_V1(buffer_create_toy);
+Datum
+buffer_create_toy(PG_FUNCTION_ARGS)
+{
+ Oid relid = PG_GETARG_OID(0);
+ BlockNumber blkno = PG_GETARG_UINT32(1);
+ Relation rel;
+ Buffer buf;
+
+ rel = relation_open(relid, AccessExclusiveLock);
+
+ buf = create_toy_buffer(rel, blkno);
+ ReleaseBuffer(buf);
+
+ relation_close(rel, NoLock);
+
+ PG_RETURN_INT32(buf);
+}
+
+PG_FUNCTION_INFO_V1(buffer_call_start_io);
+Datum
+buffer_call_start_io(PG_FUNCTION_ARGS)
+{
+ Buffer buf = PG_GETARG_INT32(0);
+ bool for_input = PG_GETARG_BOOL(1);
+ bool nowait = PG_GETARG_BOOL(2);
+ bool can_start;
+
+ can_start = StartBufferIO(GetBufferDescriptor(buf - 1), for_input, nowait);
+
+ /*
+ * Tor tests we don't want the resowner release preventing us from
+ * orchestrating odd scenarios.
+ */
+ if (can_start)
+ ResourceOwnerForgetBufferIO(CurrentResourceOwner,
+ buf);
+
+ ereport(LOG,
+ errmsg("buffer %d after StartBufferIO: %s",
+ buf, DebugPrintBufferRefcount(buf)),
+ errhidestmt(true), errhidecontext(true));
+
+ PG_RETURN_BOOL(can_start);
+}
+
+/*
+CREATE FUNCTION buffer_call_terminate_io(buffer int, for_input bool, failed bool, syncio bool)
+RETURNS pg_catalog.void STRICT
+AS 'MODULE_PATHNAME' LANGUAGE C;
+*/
+PG_FUNCTION_INFO_V1(buffer_call_terminate_io);
+Datum
+buffer_call_terminate_io(PG_FUNCTION_ARGS)
+{
+ Buffer buf = PG_GETARG_INT32(0);
+ bool for_input = PG_GETARG_BOOL(1);
+ bool succeed = PG_GETARG_BOOL(2);
+ bool io_error = PG_GETARG_BOOL(3);
+ bool syncio = PG_GETARG_BOOL(4);
+ bool clear_dirty = false;
+ uint32 set_flag_bits = 0;
+
+ if (io_error)
+ set_flag_bits |= BM_IO_ERROR;
+
+ if (for_input)
+ {
+ clear_dirty = false;
+
+ if (succeed)
+ set_flag_bits |= BM_VALID;
+ }
+ else
+ {
+ if (succeed)
+ clear_dirty = true;
+ }
+
+ ereport(LOG,
+ errmsg("buffer %d before TerminateBufferIO: %s",
+ buf, DebugPrintBufferRefcount(buf)),
+ errhidestmt(true), errhidecontext(true));
+
+ TerminateBufferIO(GetBufferDescriptor(buf - 1), clear_dirty, set_flag_bits,
+ false, syncio);
+
+ ereport(LOG,
+ errmsg("buffer %d after TerminateBufferIO: %s",
+ buf, DebugPrintBufferRefcount(buf)),
+ errhidestmt(true), errhidecontext(true));
+
+ PG_RETURN_VOID();
+}
+
+PG_FUNCTION_INFO_V1(handle_get);
+Datum
+handle_get(PG_FUNCTION_ARGS)
+{
+ last_handle = pgaio_io_acquire(CurrentResourceOwner, NULL);
+
+ PG_RETURN_VOID();
+}
+
+PG_FUNCTION_INFO_V1(handle_release_last);
+Datum
+handle_release_last(PG_FUNCTION_ARGS)
+{
+ if (!last_handle)
+ elog(ERROR, "no handle");
+
+ pgaio_io_release(last_handle);
+
+ PG_RETURN_VOID();
+}
+
+PG_FUNCTION_INFO_V1(handle_get_and_error);
+Datum
+handle_get_and_error(PG_FUNCTION_ARGS)
+{
+ pgaio_io_acquire(CurrentResourceOwner, NULL);
+
+ elog(ERROR, "as you command");
+ PG_RETURN_VOID();
+}
+
+PG_FUNCTION_INFO_V1(handle_get_twice);
+Datum
+handle_get_twice(PG_FUNCTION_ARGS)
+{
+ pgaio_io_acquire(CurrentResourceOwner, NULL);
+ pgaio_io_acquire(CurrentResourceOwner, NULL);
+
+ PG_RETURN_VOID();
+}
+
+PG_FUNCTION_INFO_V1(handle_get_release);
+Datum
+handle_get_release(PG_FUNCTION_ARGS)
+{
+ PgAioHandle *handle;
+
+ handle = pgaio_io_acquire(CurrentResourceOwner, NULL);
+ pgaio_io_release(handle);
+
+ PG_RETURN_VOID();
+}
+
+PG_FUNCTION_INFO_V1(batch_start);
+Datum
+batch_start(PG_FUNCTION_ARGS)
+{
+ pgaio_enter_batchmode();
+ PG_RETURN_VOID();
+}
+
+PG_FUNCTION_INFO_V1(batch_end);
+Datum
+batch_end(PG_FUNCTION_ARGS)
+{
+ pgaio_exit_batchmode();
+ PG_RETURN_VOID();
+}
+
+#ifdef USE_INJECTION_POINTS
+extern PGDLLEXPORT void inj_io_short_read(const char *name, const void *private_data);
+extern PGDLLEXPORT void inj_io_reopen(const char *name, const void *private_data);
+
+void
+inj_io_short_read(const char *name, const void *private_data)
+{
+ PgAioHandle *ioh;
+
+ elog(LOG, "short read called: %d", inj_io_error_state->enabled_short_read);
+
+ if (inj_io_error_state->enabled_short_read)
+ {
+ ioh = pgaio_inj_io_get();
+
+ if (inj_io_error_state->short_read_result_set)
+ {
+ elog(LOG, "short read, changing result from %d to %d",
+ ioh->result, inj_io_error_state->short_read_result);
+
+ ioh->result = inj_io_error_state->short_read_result;
+ }
+ }
+}
+
+void
+inj_io_reopen(const char *name, const void *private_data)
+{
+ elog(LOG, "reopen called: %d", inj_io_error_state->enabled_reopen);
+
+ if (inj_io_error_state->enabled_reopen)
+ {
+ elog(ERROR, "injection point triggering failure to reopen ");
+ }
+}
+#endif
+
+PG_FUNCTION_INFO_V1(inj_io_short_read_attach);
+Datum
+inj_io_short_read_attach(PG_FUNCTION_ARGS)
+{
+#ifdef USE_INJECTION_POINTS
+ inj_io_error_state->enabled_short_read = true;
+ inj_io_error_state->short_read_result_set = !PG_ARGISNULL(0);
+ if (inj_io_error_state->short_read_result_set)
+ inj_io_error_state->short_read_result = PG_GETARG_INT32(0);
+#else
+ elog(ERROR, "injection points not supported");
+#endif
+
+ PG_RETURN_VOID();
+}
+
+PG_FUNCTION_INFO_V1(inj_io_short_read_detach);
+Datum
+inj_io_short_read_detach(PG_FUNCTION_ARGS)
+{
+#ifdef USE_INJECTION_POINTS
+ inj_io_error_state->enabled_short_read = false;
+#else
+ elog(ERROR, "injection points not supported");
+#endif
+ PG_RETURN_VOID();
+}
+
+PG_FUNCTION_INFO_V1(inj_io_reopen_attach);
+Datum
+inj_io_reopen_attach(PG_FUNCTION_ARGS)
+{
+#ifdef USE_INJECTION_POINTS
+ inj_io_error_state->enabled_reopen = true;
+#else
+ elog(ERROR, "injection points not supported");
+#endif
+
+ PG_RETURN_VOID();
+}
+
+PG_FUNCTION_INFO_V1(inj_io_reopen_detach);
+Datum
+inj_io_reopen_detach(PG_FUNCTION_ARGS)
+{
+#ifdef USE_INJECTION_POINTS
+ inj_io_error_state->enabled_reopen = false;
+#else
+ elog(ERROR, "injection points not supported");
+#endif
+ PG_RETURN_VOID();
+}
diff --git a/src/test/modules/test_aio/test_aio.control b/src/test/modules/test_aio/test_aio.control
new file mode 100644
index 00000000000..cd91c3ed16b
--- /dev/null
+++ b/src/test/modules/test_aio/test_aio.control
@@ -0,0 +1,3 @@
+comment = 'Test code for AIO'
+default_version = '1.0'
+module_pathname = '$libdir/test_aio'
--
2.48.1.76.g4e746b1a31.dirty
[text/x-diff] v2.6-0026-aio-Implement-smgr-md-fd-write-support.patch (12.7K, ../../ge6nsuddurhpmll3xj22vucvqwp4agqz6ndtcf2mhyeydzarst@l75dman5x53p/27-v2.6-0026-aio-Implement-smgr-md-fd-write-support.patch)
download | inline diff:
From 8905fc32399617b7a655a6e8f3f05aecb62a83bc Mon Sep 17 00:00:00 2001
From: Andres Freund <andres@anarazel.de>
Date: Sun, 9 Mar 2025 19:35:04 -0400
Subject: [PATCH v2.6 26/34] aio: Implement smgr/md/fd write support
TODO:
- register_dirty_segment_aio() can error out in edge cases
---
src/include/storage/fd.h | 1 +
src/include/storage/md.h | 5 +
src/include/storage/smgr.h | 5 +
src/backend/storage/aio/aio_callback.c | 1 +
src/backend/storage/file/fd.c | 28 ++++
src/backend/storage/smgr/md.c | 189 +++++++++++++++++++++++++
src/backend/storage/smgr/smgr.c | 22 +++
7 files changed, 251 insertions(+)
diff --git a/src/include/storage/fd.h b/src/include/storage/fd.h
index b77d8e5e30e..2cc7c5a4761 100644
--- a/src/include/storage/fd.h
+++ b/src/include/storage/fd.h
@@ -112,6 +112,7 @@ extern int FilePrefetch(File file, off_t offset, off_t amount, uint32 wait_event
extern ssize_t FileReadV(File file, const struct iovec *iov, int iovcnt, off_t offset, uint32 wait_event_info);
extern ssize_t FileWriteV(File file, const struct iovec *iov, int iovcnt, off_t offset, uint32 wait_event_info);
extern int FileStartReadV(struct PgAioHandle *ioh, File file, int iovcnt, off_t offset, uint32 wait_event_info);
+extern int FileStartWriteV(struct PgAioHandle *ioh, File file, int iovcnt, off_t offset, uint32 wait_event_info);
extern int FileSync(File file, uint32 wait_event_info);
extern int FileZero(File file, off_t offset, off_t amount, uint32 wait_event_info);
extern int FileFallocate(File file, off_t offset, off_t amount, uint32 wait_event_info);
diff --git a/src/include/storage/md.h b/src/include/storage/md.h
index 9d7131eff43..47ae6c36c94 100644
--- a/src/include/storage/md.h
+++ b/src/include/storage/md.h
@@ -21,6 +21,7 @@
#include "storage/sync.h"
extern const PgAioHandleCallbacks aio_md_readv_cb;
+extern const PgAioHandleCallbacks aio_md_writev_cb;
/* md storage manager functionality */
extern void mdinit(void);
@@ -45,6 +46,10 @@ extern void mdstartreadv(PgAioHandle *ioh,
extern void mdwritev(SMgrRelation reln, ForkNumber forknum,
BlockNumber blocknum,
const void **buffers, BlockNumber nblocks, bool skipFsync);
+extern void mdstartwritev(PgAioHandle *ioh,
+ SMgrRelation reln, ForkNumber forknum,
+ BlockNumber blocknum,
+ const void **buffers, BlockNumber nblocks, bool skipFsync);
extern void mdwriteback(SMgrRelation reln, ForkNumber forknum,
BlockNumber blocknum, BlockNumber nblocks);
extern BlockNumber mdnblocks(SMgrRelation reln, ForkNumber forknum);
diff --git a/src/include/storage/smgr.h b/src/include/storage/smgr.h
index 856ebcda350..f00b3763ac9 100644
--- a/src/include/storage/smgr.h
+++ b/src/include/storage/smgr.h
@@ -108,6 +108,11 @@ extern void smgrwritev(SMgrRelation reln, ForkNumber forknum,
BlockNumber blocknum,
const void **buffers, BlockNumber nblocks,
bool skipFsync);
+extern void smgrstartwritev(PgAioHandle *ioh,
+ SMgrRelation reln, ForkNumber forknum,
+ BlockNumber blocknum,
+ const void **buffers, BlockNumber nblocks,
+ bool skipFsync);
extern void smgrwriteback(SMgrRelation reln, ForkNumber forknum,
BlockNumber blocknum, BlockNumber nblocks);
extern BlockNumber smgrnblocks(SMgrRelation reln, ForkNumber forknum);
diff --git a/src/backend/storage/aio/aio_callback.c b/src/backend/storage/aio/aio_callback.c
index f76f74ba166..fb6ac058a09 100644
--- a/src/backend/storage/aio/aio_callback.c
+++ b/src/backend/storage/aio/aio_callback.c
@@ -41,6 +41,7 @@ static const PgAioHandleCallbacksEntry aio_handle_cbs[] = {
CALLBACK_ENTRY(PGAIO_HCB_INVALID, aio_invalid_cb),
CALLBACK_ENTRY(PGAIO_HCB_MD_READV, aio_md_readv_cb),
+ CALLBACK_ENTRY(PGAIO_HCB_MD_WRITEV, aio_md_writev_cb),
CALLBACK_ENTRY(PGAIO_HCB_SHARED_BUFFER_READV, aio_shared_buffer_readv_cb),
diff --git a/src/backend/storage/file/fd.c b/src/backend/storage/file/fd.c
index be8c4c2d60d..65abebefbfb 100644
--- a/src/backend/storage/file/fd.c
+++ b/src/backend/storage/file/fd.c
@@ -2346,6 +2346,34 @@ retry:
return returnCode;
}
+int
+FileStartWriteV(PgAioHandle *ioh, File file,
+ int iovcnt, off_t offset,
+ uint32 wait_event_info)
+{
+ int returnCode;
+ Vfd *vfdP;
+
+ Assert(FileIsValid(file));
+
+ DO_DB(elog(LOG, "FileStartWriteV: %d (%s) " INT64_FORMAT " %d",
+ file, VfdCache[file].fileName,
+ (int64) offset,
+ iovcnt));
+
+ returnCode = FileAccess(file);
+ if (returnCode < 0)
+ return returnCode;
+
+ vfdP = &VfdCache[file];
+
+ /* FIXME: think about / reimplement temp_file_limit */
+
+ pgaio_io_prep_writev(ioh, vfdP->fd, iovcnt, offset);
+
+ return 0;
+}
+
int
FileSync(File file, uint32 wait_event_info)
{
diff --git a/src/backend/storage/smgr/md.c b/src/backend/storage/smgr/md.c
index 11d4d5a7aea..a61fc14805e 100644
--- a/src/backend/storage/smgr/md.c
+++ b/src/backend/storage/smgr/md.c
@@ -155,12 +155,19 @@ static BlockNumber _mdnblocks(SMgrRelation reln, ForkNumber forknum,
static PgAioResult md_readv_complete(PgAioHandle *ioh, PgAioResult prior_result, uint8 cb_data);
static void md_readv_report(PgAioResult result, const PgAioTargetData *target_data, int elevel);
+static PgAioResult md_writev_complete(PgAioHandle *ioh, PgAioResult prior_result, uint8 cb_data);
+static void md_writev_report(PgAioResult result, const PgAioTargetData *target_data, int elevel);
const PgAioHandleCallbacks aio_md_readv_cb = {
.complete_shared = md_readv_complete,
.report = md_readv_report,
};
+const PgAioHandleCallbacks aio_md_writev_cb = {
+ .complete_shared = md_writev_complete,
+ .report = md_writev_report,
+};
+
static inline int
_mdfd_open_flags(void)
@@ -1107,6 +1114,56 @@ mdwritev(SMgrRelation reln, ForkNumber forknum, BlockNumber blocknum,
}
}
+/*
+ * mdstartwritev() -- Asynchronous version of mdrwritev().
+ */
+void
+mdstartwritev(PgAioHandle *ioh,
+ SMgrRelation reln, ForkNumber forknum, BlockNumber blocknum,
+ const void **buffers, BlockNumber nblocks, bool skipFsync)
+{
+ off_t seekpos;
+ MdfdVec *v;
+ BlockNumber nblocks_this_segment;
+ struct iovec *iov;
+ int iovcnt;
+
+ v = _mdfd_getseg(reln, forknum, blocknum, false,
+ EXTENSION_FAIL | EXTENSION_CREATE_RECOVERY);
+
+ seekpos = (off_t) BLCKSZ * (blocknum % ((BlockNumber) RELSEG_SIZE));
+
+ Assert(seekpos < (off_t) BLCKSZ * RELSEG_SIZE);
+
+ nblocks_this_segment =
+ Min(nblocks,
+ RELSEG_SIZE - (blocknum % ((BlockNumber) RELSEG_SIZE)));
+
+ if (nblocks_this_segment != nblocks)
+ elog(ERROR, "write crossing segment boundary");
+
+ iovcnt = pgaio_io_get_iovec(ioh, &iov);
+
+ Assert(nblocks <= iovcnt);
+
+ iovcnt = buffers_to_iovec(iov, unconstify(void **, buffers), nblocks_this_segment);
+
+ Assert(iovcnt <= nblocks_this_segment);
+
+ if (!(io_direct_flags & IO_DIRECT_DATA))
+ pgaio_io_set_flag(ioh, PGAIO_HF_BUFFERED);
+
+ pgaio_io_set_target_smgr(ioh,
+ reln,
+ forknum,
+ blocknum,
+ nblocks,
+ skipFsync);
+ pgaio_io_register_callbacks(ioh, PGAIO_HCB_MD_WRITEV, 0);
+
+ FileStartWriteV(ioh, v->mdfd_vfd, iovcnt, seekpos, WAIT_EVENT_DATA_FILE_WRITE);
+}
+
/*
* mdwriteback() -- Tell the kernel to write pages back to storage.
@@ -1495,6 +1552,40 @@ register_dirty_segment(SMgrRelation reln, ForkNumber forknum, MdfdVec *seg)
}
}
+/*
+ * Like register_dirty_segment(), except for use by AIO. In the completion
+ * callback we don't have access to the MdfdVec (the completion callback might
+ * be executed in a different backend than the issuing backend), therefore we
+ * have to implement this slightly differently.
+ */
+static void
+register_dirty_segment_aio(RelFileLocator locator, ForkNumber forknum, uint64 segno)
+{
+ FileTag tag;
+
+ INIT_MD_FILETAG(tag, locator, forknum, segno);
+
+ /*
+ * Can't block here waiting for checkpointer to accept our sync request,
+ * as checkpointer might be waiting for this AIO to finish if offloaded to
+ * a worker.
+ */
+ if (!RegisterSyncRequest(&tag, SYNC_REQUEST, false /* retryOnError */ ))
+ {
+ char path[MAXPGPATH];
+
+ ereport(DEBUG1,
+ (errmsg_internal("could not forward fsync request because request queue is full")));
+
+ /* reuse mdsyncfiletag() to avoid duplicating code */
+ if (mdsyncfiletag(&tag, path))
+ ereport(data_sync_elevel(ERROR),
+ (errcode_for_file_access(),
+ errmsg("could not fsync file \"%s\": %m",
+ path)));
+ }
+}
+
/*
* register_unlink_segment() -- Schedule a file to be deleted after next checkpoint
*/
@@ -2017,3 +2108,101 @@ md_readv_report(PgAioResult result, const PgAioTargetData *td, int elevel)
td->smgr.nblocks * (size_t) BLCKSZ));
}
}
+
+/*
+ * AIO completion callback for mdstartwritev().
+ */
+static PgAioResult
+md_writev_complete(PgAioHandle *ioh, PgAioResult prior_result, uint8 cb_data)
+{
+ PgAioTargetData *td = pgaio_io_get_target_data(ioh);
+ PgAioResult result = prior_result;
+
+ if (prior_result.result < 0)
+ {
+ result.status = ARS_ERROR;
+ result.id = PGAIO_HCB_MD_WRITEV;
+ /* For "hard" errors, track the error number in error_data */
+ result.error_data = -prior_result.result;
+ result.result = 0;
+
+ pgaio_result_report(result, td, LOG);
+
+ return result;
+ }
+
+ /*
+ * The smgr API operates in blocks, therefore convert the result from
+ * bytes to blocks.
+ */
+ result.result /= BLCKSZ;
+
+ if (result.result == 0)
+ {
+ /* consider 0 blocks written a failure */
+ result.status = ARS_ERROR;
+ result.id = PGAIO_HCB_MD_WRITEV;
+ result.error_data = 0;
+
+ pgaio_result_report(result, td, LOG);
+
+ return result;
+ }
+
+ if (result.status != ARS_ERROR &&
+ result.result < td->smgr.nblocks)
+ {
+ /* partial writes should be retried at upper level */
+ result.status = ARS_PARTIAL;
+ result.id = PGAIO_HCB_MD_WRITEV;
+ }
+
+ if (!td->smgr.skip_fsync)
+ register_dirty_segment_aio(td->smgr.rlocator, td->smgr.forkNum,
+ td->smgr.blockNum / ((BlockNumber) RELSEG_SIZE));
+
+ return result;
+}
+
+/*
+ * AIO error reporting callback for mdstartwritev().
+ */
+static void
+md_writev_report(PgAioResult result, const PgAioTargetData *td, int elevel)
+{
+ RelPathStr path;
+
+ path = relpathbackend(td->smgr.rlocator,
+ td->smgr.is_temp ? MyProcNumber : INVALID_PROC_NUMBER,
+ td->smgr.forkNum);
+
+ if (result.error_data != 0)
+ {
+ errno = result.error_data; /* for errcode_for_file_access() */
+
+ ereport(elevel,
+ errcode_for_file_access(),
+ errmsg("could not write blocks %u..%u in file \"%s\": %m",
+ td->smgr.blockNum,
+ td->smgr.blockNum + td->smgr.nblocks,
+ path.str)
+ );
+ }
+ else
+ {
+ /*
+ * NB: This will typically only be output in debug messages, while
+ * retrying a partial IO.
+ */
+ ereport(elevel,
+ errcode(ERRCODE_DATA_CORRUPTED),
+ errmsg("could not write blocks %u..%u in file \"%s\": wrote only %zu of %zu bytes",
+ td->smgr.blockNum,
+ td->smgr.blockNum + td->smgr.nblocks - 1,
+ path.str,
+ result.result * (size_t) BLCKSZ,
+ td->smgr.nblocks * (size_t) BLCKSZ
+ )
+ );
+ }
+}
diff --git a/src/backend/storage/smgr/smgr.c b/src/backend/storage/smgr/smgr.c
index 607c14ee173..088b189543b 100644
--- a/src/backend/storage/smgr/smgr.c
+++ b/src/backend/storage/smgr/smgr.c
@@ -102,6 +102,11 @@ typedef struct f_smgr
BlockNumber blocknum,
const void **buffers, BlockNumber nblocks,
bool skipFsync);
+ void (*smgr_startwritev) (PgAioHandle *ioh,
+ SMgrRelation reln, ForkNumber forknum,
+ BlockNumber blocknum,
+ const void **buffers, BlockNumber nblocks,
+ bool skipFsync);
void (*smgr_writeback) (SMgrRelation reln, ForkNumber forknum,
BlockNumber blocknum, BlockNumber nblocks);
BlockNumber (*smgr_nblocks) (SMgrRelation reln, ForkNumber forknum);
@@ -129,6 +134,7 @@ static const f_smgr smgrsw[] = {
.smgr_readv = mdreadv,
.smgr_startreadv = mdstartreadv,
.smgr_writev = mdwritev,
+ .smgr_startwritev = mdstartwritev,
.smgr_writeback = mdwriteback,
.smgr_nblocks = mdnblocks,
.smgr_truncate = mdtruncate,
@@ -691,6 +697,22 @@ smgrwritev(SMgrRelation reln, ForkNumber forknum, BlockNumber blocknum,
buffers, nblocks, skipFsync);
}
+/*
+ * smgrstartwritev() -- asynchronous version of smgrwritev()
+ *
+ * This starts an asynchronous writev IO with the IO handle `ioh`. Other than
+ * `ioh` all parameters are the same as smgrwritev().
+ */
+void
+smgrstartwritev(PgAioHandle *ioh,
+ SMgrRelation reln, ForkNumber forknum, BlockNumber blocknum,
+ const void **buffers, BlockNumber nblocks, bool skipFsync)
+{
+ smgrsw[reln->smgr_which].smgr_startwritev(ioh,
+ reln, forknum, blocknum, buffers,
+ nblocks, skipFsync);
+}
+
/*
* smgrwriteback() -- Trigger kernel writeback for the supplied range of
* blocks.
--
2.48.1.76.g4e746b1a31.dirty
[text/x-diff] v2.6-0027-aio-Add-bounce-buffers.patch (22.9K, ../../ge6nsuddurhpmll3xj22vucvqwp4agqz6ndtcf2mhyeydzarst@l75dman5x53p/28-v2.6-0027-aio-Add-bounce-buffers.patch)
download | inline diff:
From 2a0bbc1f858b98eff116a410277920fcb2b7ebae Mon Sep 17 00:00:00 2001
From: Andres Freund <andres@anarazel.de>
Date: Fri, 7 Mar 2025 15:47:38 -0500
Subject: [PATCH v2.6 27/34] aio: Add bounce buffers
---
src/include/storage/aio.h | 18 ++
src/include/storage/aio_internal.h | 33 ++++
src/include/utils/resowner.h | 2 +
src/backend/storage/aio/README.md | 27 +++
src/backend/storage/aio/aio.c | 178 ++++++++++++++++++
src/backend/storage/aio/aio_init.c | 123 ++++++++++++
src/backend/utils/misc/guc_tables.c | 13 ++
src/backend/utils/misc/postgresql.conf.sample | 2 +
src/backend/utils/resowner/resowner.c | 25 ++-
src/test/modules/test_aio/test_aio--1.0.sql | 21 +++
src/test/modules/test_aio/test_aio.c | 55 ++++++
src/tools/pgindent/typedefs.list | 1 +
12 files changed, 496 insertions(+), 2 deletions(-)
diff --git a/src/include/storage/aio.h b/src/include/storage/aio.h
index 3ce8763ebe8..cdf54b90b15 100644
--- a/src/include/storage/aio.h
+++ b/src/include/storage/aio.h
@@ -249,6 +249,9 @@ struct PgAioHandleCallbacks
#define PGAIO_HANDLE_MAX_CALLBACKS 4
+typedef struct PgAioBounceBuffer PgAioBounceBuffer;
+
+
/* --------------------------------------------------------------------------------
* IO Handles
@@ -333,6 +336,20 @@ extern bool pgaio_have_staged(void);
+/* --------------------------------------------------------------------------------
+ * Bounce Buffers
+ * --------------------------------------------------------------------------------
+ */
+
+extern PgAioBounceBuffer *pgaio_bounce_buffer_get(void);
+extern void pgaio_io_assoc_bounce_buffer(PgAioHandle *ioh, PgAioBounceBuffer *bb);
+extern uint32 pgaio_bounce_buffer_id(PgAioBounceBuffer *bb);
+extern void pgaio_bounce_buffer_release(PgAioBounceBuffer *bb);
+extern char *pgaio_bounce_buffer_buffer(PgAioBounceBuffer *bb);
+extern void pgaio_bounce_buffer_release_resowner(struct dlist_node *bb_node, bool on_error);
+
+
+
/* --------------------------------------------------------------------------------
* Other
* --------------------------------------------------------------------------------
@@ -345,6 +362,7 @@ extern void pgaio_closing_fd(int fd);
/* GUCs */
extern PGDLLIMPORT int io_method;
extern PGDLLIMPORT int io_max_concurrency;
+extern PGDLLIMPORT int io_bounce_buffers;
#endif /* AIO_H */
diff --git a/src/include/storage/aio_internal.h b/src/include/storage/aio_internal.h
index 2412c5e7ecb..9471f2a92be 100644
--- a/src/include/storage/aio_internal.h
+++ b/src/include/storage/aio_internal.h
@@ -94,6 +94,12 @@ struct PgAioHandle
/* raw result of the IO operation */
int32 result;
+ /*
+ * List of bounce_buffers owned by IO. It would suffice to use an index
+ * based linked list here.
+ */
+ slist_head bounce_buffers;
+
/**
* In which list the handle is registered, depends on the state:
* - IDLE, in per-backend list
@@ -149,11 +155,23 @@ struct PgAioHandle
};
+struct PgAioBounceBuffer
+{
+ slist_node node;
+ struct ResourceOwnerData *resowner;
+ dlist_node resowner_node;
+ char *buffer;
+};
+
+
typedef struct PgAioBackend
{
/* index into PgAioCtl->io_handles */
uint32 io_handle_off;
+ /* index into PgAioCtl->bounce_buffers */
+ uint32 bounce_buffers_off;
+
/* IO Handles that currently are not used */
dclist_head idle_ios;
@@ -184,6 +202,12 @@ typedef struct PgAioBackend
* IOs being appended at the end.
*/
dclist_head in_flight_ios;
+
+ /* Bounce Buffers that currently are not used */
+ slist_head idle_bbs;
+
+ /* see handed_out_io */
+ PgAioBounceBuffer *handed_out_bb;
} PgAioBackend;
@@ -211,6 +235,15 @@ typedef struct PgAioCtl
uint32 io_handle_count;
PgAioHandle *io_handles;
+
+ /*
+ * To perform AIO on buffers that are not located in shared memory (either
+ * because they are not in shared memory or because we need to operate on
+ * a copy, as e.g. the case for writes when checksums are in use)
+ */
+ uint32 bounce_buffers_count;
+ PgAioBounceBuffer *bounce_buffers;
+ char *bounce_buffers_data;
} PgAioCtl;
diff --git a/src/include/utils/resowner.h b/src/include/utils/resowner.h
index aede4bfc820..7e2ec224169 100644
--- a/src/include/utils/resowner.h
+++ b/src/include/utils/resowner.h
@@ -168,5 +168,7 @@ extern void ResourceOwnerForgetLock(ResourceOwner owner, struct LOCALLOCK *local
struct dlist_node;
extern void ResourceOwnerRememberAioHandle(ResourceOwner owner, struct dlist_node *ioh_node);
extern void ResourceOwnerForgetAioHandle(ResourceOwner owner, struct dlist_node *ioh_node);
+extern void ResourceOwnerRememberAioBounceBuffer(ResourceOwner owner, struct dlist_node *bb_node);
+extern void ResourceOwnerForgetAioBounceBuffer(ResourceOwner owner, struct dlist_node *bb_node);
#endif /* RESOWNER_H */
diff --git a/src/backend/storage/aio/README.md b/src/backend/storage/aio/README.md
index de6c7db894b..bd9436d665d 100644
--- a/src/backend/storage/aio/README.md
+++ b/src/backend/storage/aio/README.md
@@ -404,6 +404,33 @@ shared memory no less!), completion callbacks instead have to encode errors in
a more compact format that can be converted into an error message.
+### AIO Bounce Buffers
+
+For some uses of AIO there is no convenient memory location as the source /
+destination of an AIO. E.g. when data checksums are enabled, writes from
+shared buffers currently cannot be done directly from shared buffers, as a
+shared buffer lock still allows some modification, e.g., for hint bits(see
+`FlushBuffer()`). If the write were done in-place, such modifications can
+cause the checksum to fail.
+
+For synchronous IO this is solved by copying the buffer to separate memory
+before computing the checksum and using that copy as the source buffer for the
+AIO.
+
+However, for AIO that is not a workable solution:
+- Instead of a single buffer many buffers are required, as many IOs might be
+ in flight
+- When using the [worker method](#worker), the source/target of IO needs to be
+ in shared memory, otherwise the workers won't be able to access the memory.
+
+The AIO subsystem addresses this by providing a limited number of bounce
+buffers that can be used as the source / target for IO. A bounce buffer be
+acquired with `pgaio_bounce_buffer_get()` and multiple bounce buffers can be
+associated with an AIO Handle with `pgaio_io_assoc_bounce_buffer()`.
+
+Bounce buffers are automatically released when the IO completes.
+
+
## Helpers
Using the low-level AIO API introduces too much complexity to do so all over
diff --git a/src/backend/storage/aio/aio.c b/src/backend/storage/aio/aio.c
index c2b34a95e1c..e1b39b5d44c 100644
--- a/src/backend/storage/aio/aio.c
+++ b/src/backend/storage/aio/aio.c
@@ -62,6 +62,8 @@ static PgAioHandle *pgaio_io_from_wref(PgAioWaitRef *iow, uint64 *ref_generation
static const char *pgaio_io_state_get_name(PgAioHandleState s);
static void pgaio_io_wait(PgAioHandle *ioh, uint64 ref_generation);
+static void pgaio_bounce_buffer_wait_for_free(void);
+
/* Options for io_method. */
const struct config_enum_entry io_method_options[] = {
@@ -76,6 +78,7 @@ const struct config_enum_entry io_method_options[] = {
/* GUCs */
int io_method = DEFAULT_IO_METHOD;
int io_max_concurrency = -1;
+int io_bounce_buffers = -1;
/* global control for AIO */
PgAioCtl *pgaio_ctl;
@@ -663,6 +666,21 @@ pgaio_io_reclaim(PgAioHandle *ioh)
}
}
+ /* reclaim all associated bounce buffers */
+ if (!slist_is_empty(&ioh->bounce_buffers))
+ {
+ slist_mutable_iter it;
+
+ slist_foreach_modify(it, &ioh->bounce_buffers)
+ {
+ PgAioBounceBuffer *bb = slist_container(PgAioBounceBuffer, node, it.cur);
+
+ slist_delete_current(&it);
+
+ slist_push_head(&pgaio_my_backend->idle_bbs, &bb->node);
+ }
+ }
+
if (ioh->resowner)
{
ResourceOwnerForgetAioHandle(ioh->resowner, &ioh->resowner_node);
@@ -1045,6 +1063,166 @@ pgaio_submit_staged(void)
+/* --------------------------------------------------------------------------------
+ * Functions primarily related to PgAioBounceBuffer
+ * --------------------------------------------------------------------------------
+ */
+
+PgAioBounceBuffer *
+pgaio_bounce_buffer_get(void)
+{
+ PgAioBounceBuffer *bb = NULL;
+ slist_node *node;
+
+ if (pgaio_my_backend->handed_out_bb != NULL)
+ elog(ERROR, "can only hand out one BB");
+
+ /*
+ * XXX: It probably is not a good idea to have bounce buffers be per
+ * backend, that's a fair bit of memory.
+ */
+ if (slist_is_empty(&pgaio_my_backend->idle_bbs))
+ {
+ pgaio_bounce_buffer_wait_for_free();
+ }
+
+ node = slist_pop_head_node(&pgaio_my_backend->idle_bbs);
+ bb = slist_container(PgAioBounceBuffer, node, node);
+
+ pgaio_my_backend->handed_out_bb = bb;
+
+ bb->resowner = CurrentResourceOwner;
+ ResourceOwnerRememberAioBounceBuffer(bb->resowner, &bb->resowner_node);
+
+ return bb;
+}
+
+void
+pgaio_io_assoc_bounce_buffer(PgAioHandle *ioh, PgAioBounceBuffer *bb)
+{
+ if (pgaio_my_backend->handed_out_bb != bb)
+ elog(ERROR, "can only assign handed out BB");
+ pgaio_my_backend->handed_out_bb = NULL;
+
+ /*
+ * There can be many bounce buffers assigned in case of vectorized IOs.
+ */
+ slist_push_head(&ioh->bounce_buffers, &bb->node);
+
+ /* once associated with an IO, the IO has ownership */
+ ResourceOwnerForgetAioBounceBuffer(bb->resowner, &bb->resowner_node);
+ bb->resowner = NULL;
+}
+
+uint32
+pgaio_bounce_buffer_id(PgAioBounceBuffer *bb)
+{
+ return bb - pgaio_ctl->bounce_buffers;
+}
+
+void
+pgaio_bounce_buffer_release(PgAioBounceBuffer *bb)
+{
+ if (pgaio_my_backend->handed_out_bb != bb)
+ elog(ERROR, "can only release handed out BB");
+
+ slist_push_head(&pgaio_my_backend->idle_bbs, &bb->node);
+ pgaio_my_backend->handed_out_bb = NULL;
+
+ ResourceOwnerForgetAioBounceBuffer(bb->resowner, &bb->resowner_node);
+ bb->resowner = NULL;
+}
+
+void
+pgaio_bounce_buffer_release_resowner(dlist_node *bb_node, bool on_error)
+{
+ PgAioBounceBuffer *bb = dlist_container(PgAioBounceBuffer, resowner_node, bb_node);
+
+ Assert(bb->resowner);
+
+ if (!on_error)
+ elog(WARNING, "leaked AIO bounce buffer");
+
+ pgaio_bounce_buffer_release(bb);
+}
+
+char *
+pgaio_bounce_buffer_buffer(PgAioBounceBuffer *bb)
+{
+ return bb->buffer;
+}
+
+static void
+pgaio_bounce_buffer_wait_for_free(void)
+{
+ static uint32 lastpos = 0;
+
+ if (pgaio_my_backend->num_staged_ios > 0)
+ {
+ pgaio_debug(DEBUG2, "submitting %d, while acquiring free bb",
+ pgaio_my_backend->num_staged_ios);
+ pgaio_submit_staged();
+ }
+
+ for (uint32 i = lastpos; i < lastpos + io_max_concurrency; i++)
+ {
+ uint32 thisoff = pgaio_my_backend->io_handle_off + (i % io_max_concurrency);
+ PgAioHandle *ioh = &pgaio_ctl->io_handles[thisoff];
+
+ switch (ioh->state)
+ {
+ case PGAIO_HS_IDLE:
+ case PGAIO_HS_HANDED_OUT:
+ continue;
+ case PGAIO_HS_DEFINED: /* should have been submitted above */
+ case PGAIO_HS_STAGED:
+ elog(ERROR, "shouldn't get here with io:%d in state %d",
+ pgaio_io_get_id(ioh), ioh->state);
+ break;
+ case PGAIO_HS_COMPLETED_IO:
+ case PGAIO_HS_SUBMITTED:
+ if (!slist_is_empty(&ioh->bounce_buffers))
+ {
+ pgaio_debug_io(DEBUG2, ioh,
+ "waiting for IO to reclaim BB with %d in flight",
+ dclist_count(&pgaio_my_backend->in_flight_ios));
+
+ /* see comment in pgaio_io_wait_for_free() about raciness */
+ pgaio_io_wait(ioh, ioh->generation);
+
+ if (slist_is_empty(&pgaio_my_backend->idle_bbs))
+ elog(WARNING, "empty after wait");
+
+ if (!slist_is_empty(&pgaio_my_backend->idle_bbs))
+ {
+ lastpos = i;
+ return;
+ }
+ }
+ break;
+ case PGAIO_HS_COMPLETED_SHARED:
+ case PGAIO_HS_COMPLETED_LOCAL:
+ /* reclaim */
+ pgaio_io_reclaim(ioh);
+
+ if (!slist_is_empty(&pgaio_my_backend->idle_bbs))
+ {
+ lastpos = i;
+ return;
+ }
+ break;
+ }
+ }
+
+ /*
+ * The submission above could have caused the IO to complete at any time.
+ */
+ if (slist_is_empty(&pgaio_my_backend->idle_bbs))
+ elog(PANIC, "no more bbs");
+}
+
+
+
/* --------------------------------------------------------------------------------
* Other
* --------------------------------------------------------------------------------
diff --git a/src/backend/storage/aio/aio_init.c b/src/backend/storage/aio/aio_init.c
index 2205658cd9a..a56126a772a 100644
--- a/src/backend/storage/aio/aio_init.c
+++ b/src/backend/storage/aio/aio_init.c
@@ -82,6 +82,32 @@ AioHandleDataShmemSize(void)
io_max_concurrency));
}
+static Size
+AioBounceBufferDescShmemSize(void)
+{
+ Size sz;
+
+ /* PgAioBounceBuffer itself */
+ sz = mul_size(sizeof(PgAioBounceBuffer),
+ mul_size(AioProcs(), io_bounce_buffers));
+
+ return sz;
+}
+
+static Size
+AioBounceBufferDataShmemSize(void)
+{
+ Size sz;
+
+ /* and the associated buffer */
+ sz = mul_size(BLCKSZ,
+ mul_size(io_bounce_buffers, AioProcs()));
+ /* memory for alignment */
+ sz += BLCKSZ;
+
+ return sz;
+}
+
/*
* Choose a suitable value for io_max_concurrency.
*
@@ -107,6 +133,33 @@ AioChooseMaxConcurrency(void)
return Min(max_proportional_pins, 64);
}
+/*
+ * Choose a suitable value for io_bounce_buffers.
+ *
+ * It's very unlikely to be useful to allocate more bounce buffers for each
+ * backend than the backend is allowed to pin. Additionally, bounce buffers
+ * currently are used for writes, it seems very uncommon for more than 10% of
+ * shared_buffers to be written out concurrently.
+ *
+ * XXX: This quickly can take up significant amounts of memory, the logic
+ * should probably fine tuned.
+ */
+static int
+AioChooseBounceBuffers(void)
+{
+ uint32 max_backends;
+ int max_proportional_pins;
+
+ /* Similar logic to LimitAdditionalPins() */
+ max_backends = MaxBackends + NUM_AUXILIARY_PROCS;
+ max_proportional_pins = (NBuffers / 10) / max_backends;
+
+ max_proportional_pins = Max(max_proportional_pins, 1);
+
+ /* apply upper limit */
+ return Min(max_proportional_pins, 256);
+}
+
Size
AioShmemSize(void)
{
@@ -130,11 +183,31 @@ AioShmemSize(void)
PGC_S_OVERRIDE);
}
+
+ /*
+ * If io_bounce_buffers is -1, we automatically choose a suitable value.
+ *
+ * See also comment above.
+ */
+ if (io_bounce_buffers == -1)
+ {
+ char buf[32];
+
+ snprintf(buf, sizeof(buf), "%d", AioChooseBounceBuffers());
+ SetConfigOption("io_bounce_buffers", buf, PGC_POSTMASTER,
+ PGC_S_DYNAMIC_DEFAULT);
+ if (io_bounce_buffers == -1) /* failed to apply it? */
+ SetConfigOption("io_bounce_buffers", buf, PGC_POSTMASTER,
+ PGC_S_OVERRIDE);
+ }
+
sz = add_size(sz, AioCtlShmemSize());
sz = add_size(sz, AioBackendShmemSize());
sz = add_size(sz, AioHandleShmemSize());
sz = add_size(sz, AioHandleIOVShmemSize());
sz = add_size(sz, AioHandleDataShmemSize());
+ sz = add_size(sz, AioBounceBufferDescShmemSize());
+ sz = add_size(sz, AioBounceBufferDataShmemSize());
if (pgaio_method_ops->shmem_size)
sz = add_size(sz, pgaio_method_ops->shmem_size());
@@ -149,6 +222,9 @@ AioShmemInit(void)
uint32 io_handle_off = 0;
uint32 iovec_off = 0;
uint32 per_backend_iovecs = io_max_concurrency * PG_IOV_MAX;
+ uint32 bounce_buffers_off = 0;
+ uint32 per_backend_bb = io_bounce_buffers;
+ char *bounce_buffers_data;
pgaio_ctl = (PgAioCtl *)
ShmemInitStruct("AioCtl", AioCtlShmemSize(), &found);
@@ -160,6 +236,7 @@ AioShmemInit(void)
pgaio_ctl->io_handle_count = AioProcs() * io_max_concurrency;
pgaio_ctl->iovec_count = AioProcs() * per_backend_iovecs;
+ pgaio_ctl->bounce_buffers_count = AioProcs() * per_backend_bb;
pgaio_ctl->backend_state = (PgAioBackend *)
ShmemInitStruct("AioBackend", AioBackendShmemSize(), &found);
@@ -172,6 +249,40 @@ AioShmemInit(void)
pgaio_ctl->handle_data = (uint64 *)
ShmemInitStruct("AioHandleData", AioHandleDataShmemSize(), &found);
+ pgaio_ctl->bounce_buffers = (PgAioBounceBuffer *)
+ ShmemInitStruct("AioBounceBufferDesc", AioBounceBufferDescShmemSize(),
+ &found);
+
+ bounce_buffers_data =
+ ShmemInitStruct("AioBounceBufferData", AioBounceBufferDataShmemSize(),
+ &found);
+ bounce_buffers_data =
+ (char *) TYPEALIGN(BLCKSZ, (uintptr_t) bounce_buffers_data);
+ pgaio_ctl->bounce_buffers_data = bounce_buffers_data;
+
+
+ /* Initialize IO handles. */
+ for (uint64 i = 0; i < pgaio_ctl->io_handle_count; i++)
+ {
+ PgAioHandle *ioh = &pgaio_ctl->io_handles[i];
+
+ ioh->op = PGAIO_OP_INVALID;
+ ioh->target = PGAIO_TID_INVALID;
+ ioh->state = PGAIO_HS_IDLE;
+
+ slist_init(&ioh->bounce_buffers);
+ }
+
+ /* Initialize Bounce Buffers. */
+ for (uint64 i = 0; i < pgaio_ctl->bounce_buffers_count; i++)
+ {
+ PgAioBounceBuffer *bb = &pgaio_ctl->bounce_buffers[i];
+
+ bb->buffer = bounce_buffers_data;
+ bounce_buffers_data += BLCKSZ;
+ }
+
+
for (int procno = 0; procno < AioProcs(); procno++)
{
PgAioBackend *bs = &pgaio_ctl->backend_state[procno];
@@ -179,9 +290,13 @@ AioShmemInit(void)
bs->io_handle_off = io_handle_off;
io_handle_off += io_max_concurrency;
+ bs->bounce_buffers_off = bounce_buffers_off;
+ bounce_buffers_off += per_backend_bb;
+
dclist_init(&bs->idle_ios);
memset(bs->staged_ios, 0, sizeof(PgAioHandle *) * PGAIO_SUBMIT_BATCH_SIZE);
dclist_init(&bs->in_flight_ios);
+ slist_init(&bs->idle_bbs);
/* initialize per-backend IOs */
for (int i = 0; i < io_max_concurrency; i++)
@@ -203,6 +318,14 @@ AioShmemInit(void)
dclist_push_tail(&bs->idle_ios, &ioh->node);
iovec_off += PG_IOV_MAX;
}
+
+ /* initialize per-backend bounce buffers */
+ for (int i = 0; i < per_backend_bb; i++)
+ {
+ PgAioBounceBuffer *bb = &pgaio_ctl->bounce_buffers[bs->bounce_buffers_off + i];
+
+ slist_push_head(&bs->idle_bbs, &bb->node);
+ }
}
out:
diff --git a/src/backend/utils/misc/guc_tables.c b/src/backend/utils/misc/guc_tables.c
index 4317cfc9d2f..41a4025c66d 100644
--- a/src/backend/utils/misc/guc_tables.c
+++ b/src/backend/utils/misc/guc_tables.c
@@ -3268,6 +3268,19 @@ struct config_int ConfigureNamesInt[] =
check_io_max_concurrency, NULL, NULL
},
+ {
+ {"io_bounce_buffers",
+ PGC_POSTMASTER,
+ RESOURCES_IO,
+ gettext_noop("Number of IO Bounce Buffers reserved for each backend."),
+ NULL,
+ GUC_UNIT_BLOCKS
+ },
+ &io_bounce_buffers,
+ -1, -1, 4096,
+ NULL, NULL, NULL
+ },
+
{
{"io_workers",
PGC_SIGHUP,
diff --git a/src/backend/utils/misc/postgresql.conf.sample b/src/backend/utils/misc/postgresql.conf.sample
index 8738ad51bf1..291ca8ab38d 100644
--- a/src/backend/utils/misc/postgresql.conf.sample
+++ b/src/backend/utils/misc/postgresql.conf.sample
@@ -209,6 +209,8 @@
# -1 sets based on shared_buffers
# (change requires restart)
#io_workers = 3 # 1-32;
+#io_bounce_buffers = -1 # -1 sets based on shared_buffers
+ # (change requires restart)
# - Worker Processes -
diff --git a/src/backend/utils/resowner/resowner.c b/src/backend/utils/resowner/resowner.c
index 76b9cec1e26..de00346b549 100644
--- a/src/backend/utils/resowner/resowner.c
+++ b/src/backend/utils/resowner/resowner.c
@@ -159,10 +159,11 @@ struct ResourceOwnerData
LOCALLOCK *locks[MAX_RESOWNER_LOCKS]; /* list of owned locks */
/*
- * AIO handles need be registered in critical sections and therefore
- * cannot use the normal ResoureElem mechanism.
+ * AIO handles & bounce buffers need be registered in critical sections
+ * and therefore cannot use the normal ResoureElem mechanism.
*/
dlist_head aio_handles;
+ dlist_head aio_bounce_buffers;
};
@@ -434,6 +435,7 @@ ResourceOwnerCreate(ResourceOwner parent, const char *name)
}
dlist_init(&owner->aio_handles);
+ dlist_init(&owner->aio_bounce_buffers);
return owner;
}
@@ -742,6 +744,13 @@ ResourceOwnerReleaseInternal(ResourceOwner owner,
pgaio_io_release_resowner(node, !isCommit);
}
+
+ while (!dlist_is_empty(&owner->aio_bounce_buffers))
+ {
+ dlist_node *node = dlist_head_node(&owner->aio_bounce_buffers);
+
+ pgaio_bounce_buffer_release_resowner(node, !isCommit);
+ }
}
else if (phase == RESOURCE_RELEASE_LOCKS)
{
@@ -1111,3 +1120,15 @@ ResourceOwnerForgetAioHandle(ResourceOwner owner, struct dlist_node *ioh_node)
{
dlist_delete_from(&owner->aio_handles, ioh_node);
}
+
+void
+ResourceOwnerRememberAioBounceBuffer(ResourceOwner owner, struct dlist_node *ioh_node)
+{
+ dlist_push_tail(&owner->aio_bounce_buffers, ioh_node);
+}
+
+void
+ResourceOwnerForgetAioBounceBuffer(ResourceOwner owner, struct dlist_node *ioh_node)
+{
+ dlist_delete_from(&owner->aio_bounce_buffers, ioh_node);
+}
diff --git a/src/test/modules/test_aio/test_aio--1.0.sql b/src/test/modules/test_aio/test_aio--1.0.sql
index 3d4e9bb070f..16f248619fc 100644
--- a/src/test/modules/test_aio/test_aio--1.0.sql
+++ b/src/test/modules/test_aio/test_aio--1.0.sql
@@ -76,6 +76,27 @@ RETURNS pg_catalog.void STRICT
AS 'MODULE_PATHNAME' LANGUAGE C;
+CREATE FUNCTION bb_get_and_error()
+RETURNS pg_catalog.void STRICT
+AS 'MODULE_PATHNAME' LANGUAGE C;
+
+CREATE FUNCTION bb_get_twice()
+RETURNS pg_catalog.void STRICT
+AS 'MODULE_PATHNAME' LANGUAGE C;
+
+CREATE FUNCTION bb_get()
+RETURNS pg_catalog.void STRICT
+AS 'MODULE_PATHNAME' LANGUAGE C;
+
+CREATE FUNCTION bb_get_release()
+RETURNS pg_catalog.void STRICT
+AS 'MODULE_PATHNAME' LANGUAGE C;
+
+CREATE FUNCTION bb_release_last()
+RETURNS pg_catalog.void STRICT
+AS 'MODULE_PATHNAME' LANGUAGE C;
+
+
/*
* Injection point related functions
diff --git a/src/test/modules/test_aio/test_aio.c b/src/test/modules/test_aio/test_aio.c
index 81b4d732206..60c01b251fe 100644
--- a/src/test/modules/test_aio/test_aio.c
+++ b/src/test/modules/test_aio/test_aio.c
@@ -53,6 +53,7 @@ static shmem_startup_hook_type prev_shmem_startup_hook = NULL;
static PgAioHandle *last_handle;
+static PgAioBounceBuffer *last_bb;
@@ -525,6 +526,60 @@ batch_end(PG_FUNCTION_ARGS)
PG_RETURN_VOID();
}
+PG_FUNCTION_INFO_V1(bb_get);
+Datum
+bb_get(PG_FUNCTION_ARGS)
+{
+ last_bb = pgaio_bounce_buffer_get();
+
+ PG_RETURN_VOID();
+}
+
+PG_FUNCTION_INFO_V1(bb_release_last);
+Datum
+bb_release_last(PG_FUNCTION_ARGS)
+{
+ if (!last_bb)
+ elog(ERROR, "no bb");
+
+ pgaio_bounce_buffer_release(last_bb);
+
+ PG_RETURN_VOID();
+}
+
+PG_FUNCTION_INFO_V1(bb_get_and_error);
+Datum
+bb_get_and_error(PG_FUNCTION_ARGS)
+{
+ pgaio_bounce_buffer_get();
+
+ elog(ERROR, "as you command");
+ PG_RETURN_VOID();
+}
+
+PG_FUNCTION_INFO_V1(bb_get_twice);
+Datum
+bb_get_twice(PG_FUNCTION_ARGS)
+{
+ pgaio_bounce_buffer_get();
+ pgaio_bounce_buffer_get();
+
+ PG_RETURN_VOID();
+}
+
+
+PG_FUNCTION_INFO_V1(bb_get_release);
+Datum
+bb_get_release(PG_FUNCTION_ARGS)
+{
+ PgAioBounceBuffer *bb;
+
+ bb = pgaio_bounce_buffer_get();
+ pgaio_bounce_buffer_release(bb);
+
+ PG_RETURN_VOID();
+}
+
#ifdef USE_INJECTION_POINTS
extern PGDLLEXPORT void inj_io_short_read(const char *name, const void *private_data);
extern PGDLLEXPORT void inj_io_reopen(const char *name, const void *private_data);
diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list
index e5a13e6e6dd..fc379ff27eb 100644
--- a/src/tools/pgindent/typedefs.list
+++ b/src/tools/pgindent/typedefs.list
@@ -2129,6 +2129,7 @@ PermutationStep
PermutationStepBlocker
PermutationStepBlockerType
PgAioBackend
+PgAioBounceBuffer
PgAioCtl
PgAioHandle
PgAioHandleCallbackID
--
2.48.1.76.g4e746b1a31.dirty
[text/x-diff] v2.6-0028-bufmgr-Implement-AIO-write-support.patch (6.2K, ../../ge6nsuddurhpmll3xj22vucvqwp4agqz6ndtcf2mhyeydzarst@l75dman5x53p/29-v2.6-0028-bufmgr-Implement-AIO-write-support.patch)
download | inline diff:
From ef08eb591f0d539ca4603686302a2f454c960662 Mon Sep 17 00:00:00 2001
From: Andres Freund <andres@anarazel.de>
Date: Wed, 22 Jan 2025 16:09:51 -0500
Subject: [PATCH v2.6 28/34] bufmgr: Implement AIO write support
As of this commit there are no users of these AIO facilities, that'll come in
later commits.
Author:
Reviewed-By:
Discussion: https://postgr.es/m/
Backpatch:
---
src/include/storage/aio.h | 2 +
src/include/storage/bufmgr.h | 2 +
src/backend/storage/aio/aio_callback.c | 2 +
src/backend/storage/buffer/bufmgr.c | 88 ++++++++++++++++++++++++++
4 files changed, 94 insertions(+)
diff --git a/src/include/storage/aio.h b/src/include/storage/aio.h
index cdf54b90b15..b53aa9748c3 100644
--- a/src/include/storage/aio.h
+++ b/src/include/storage/aio.h
@@ -182,8 +182,10 @@ typedef enum PgAioHandleCallbackID
PGAIO_HCB_MD_WRITEV,
PGAIO_HCB_SHARED_BUFFER_READV,
+ PGAIO_HCB_SHARED_BUFFER_WRITEV,
PGAIO_HCB_LOCAL_BUFFER_READV,
+ PGAIO_HCB_LOCAL_BUFFER_WRITEV,
} PgAioHandleCallbackID;
diff --git a/src/include/storage/bufmgr.h b/src/include/storage/bufmgr.h
index db9a4673097..a2bff99fb55 100644
--- a/src/include/storage/bufmgr.h
+++ b/src/include/storage/bufmgr.h
@@ -176,7 +176,9 @@ extern PGDLLIMPORT int backend_flush_after;
extern PGDLLIMPORT int bgwriter_flush_after;
extern const PgAioHandleCallbacks aio_shared_buffer_readv_cb;
+extern const PgAioHandleCallbacks aio_shared_buffer_writev_cb;
extern const PgAioHandleCallbacks aio_local_buffer_readv_cb;
+extern const PgAioHandleCallbacks aio_local_buffer_writev_cb;
/* in buf_init.c */
extern PGDLLIMPORT char *BufferBlocks;
diff --git a/src/backend/storage/aio/aio_callback.c b/src/backend/storage/aio/aio_callback.c
index fb6ac058a09..7162f722e3c 100644
--- a/src/backend/storage/aio/aio_callback.c
+++ b/src/backend/storage/aio/aio_callback.c
@@ -44,8 +44,10 @@ static const PgAioHandleCallbacksEntry aio_handle_cbs[] = {
CALLBACK_ENTRY(PGAIO_HCB_MD_WRITEV, aio_md_writev_cb),
CALLBACK_ENTRY(PGAIO_HCB_SHARED_BUFFER_READV, aio_shared_buffer_readv_cb),
+ CALLBACK_ENTRY(PGAIO_HCB_SHARED_BUFFER_WRITEV, aio_shared_buffer_writev_cb),
CALLBACK_ENTRY(PGAIO_HCB_LOCAL_BUFFER_READV, aio_local_buffer_readv_cb),
+ CALLBACK_ENTRY(PGAIO_HCB_LOCAL_BUFFER_WRITEV, aio_local_buffer_writev_cb),
#undef CALLBACK_ENTRY
};
diff --git a/src/backend/storage/buffer/bufmgr.c b/src/backend/storage/buffer/bufmgr.c
index 20544b39ef9..1f47edaa7b9 100644
--- a/src/backend/storage/buffer/bufmgr.c
+++ b/src/backend/storage/buffer/bufmgr.c
@@ -6467,6 +6467,42 @@ SharedBufferCompleteRead(int buf_off, Buffer buffer, uint8 flags, bool failed)
return result;
}
+static uint64
+BufferCompleteWriteShared(Buffer buffer, bool release_lock, bool failed)
+{
+ BufferDesc *bufHdr;
+ bool result = false;
+
+ Assert(BufferIsValid(buffer));
+
+ bufHdr = GetBufferDescriptor(buffer - 1);
+
+#ifdef USE_ASSERT_CHECKING
+ {
+ uint32 buf_state = pg_atomic_read_u32(&bufHdr->state);
+
+ Assert(buf_state & BM_VALID);
+ Assert(buf_state & BM_TAG_VALID);
+ Assert(buf_state & BM_IO_IN_PROGRESS);
+ Assert(buf_state & BM_DIRTY);
+ }
+#endif
+
+ TerminateBufferIO(bufHdr, /* clear_dirty = */ true,
+ failed ? BM_IO_ERROR : 0,
+ /* forget_owner = */ false,
+ /* syncio = */ false);
+
+ /*
+ * The initiator of IO is not managing the lock (i.e. called
+ * LWLockDisown()), we are.
+ */
+ if (release_lock)
+ LWLockReleaseDisowned(BufferDescriptorGetContentLock(bufHdr), LW_SHARED);
+
+ return result;
+}
+
/*
* Helper to prepare IO on shared buffers for execution, shared between reads
* and writes.
@@ -6555,6 +6591,12 @@ shared_buffer_readv_stage(PgAioHandle *ioh, uint8 cb_data)
* - result.error_data is the offset of the first page that failed
* verification in a larger IO
*/
+static void
+shared_buffer_writev_stage(PgAioHandle *ioh, uint8 cb_data)
+{
+ shared_buffer_stage_common(ioh, true);
+}
+
static void
buffer_readv_report(PgAioResult result, const PgAioTargetData *target_data, int elevel)
{
@@ -6641,6 +6683,33 @@ shared_buffer_readv_complete(PgAioHandle *ioh, PgAioResult prior_result, uint8 c
return buffer_readv_complete_common(ioh, prior_result, false, cb_data);
}
+static PgAioResult
+shared_buffer_writev_complete(PgAioHandle *ioh, PgAioResult prior_result, uint8 cb_data)
+{
+ PgAioResult result = prior_result;
+ uint64 *io_data;
+ uint8 handle_data_len;
+
+ ereport(DEBUG5,
+ errmsg("%s: %d %d", __func__, prior_result.status, prior_result.result),
+ errhidestmt(true), errhidecontext(true));
+
+ io_data = pgaio_io_get_handle_data(ioh, &handle_data_len);
+
+ /* FIXME: handle outright errors */
+
+ for (int io_data_off = 0; io_data_off < handle_data_len; io_data_off++)
+ {
+ Buffer buf = io_data[io_data_off];
+
+ /* FIXME: handle short writes / failures */
+ /* FIXME: ioh->target_data.shared_buffer.release_lock */
+ BufferCompleteWriteShared(buf, true, false);
+ }
+
+ return result;
+}
+
/*
* Helper to stage IO on local buffers for execution, shared between reads
* and writes.
@@ -6685,6 +6754,16 @@ local_buffer_readv_complete(PgAioHandle *ioh, PgAioResult prior_result, uint8 cb
return buffer_readv_complete_common(ioh, prior_result, true, cb_data);
}
+static void
+local_buffer_writev_stage(PgAioHandle *ioh, uint8 cb_data)
+{
+ /*
+ * Currently this is unreachable as the only write support is for
+ * checkpointer / bgwriter, which don't deal with local buffers.
+ */
+ elog(ERROR, "not yet");
+}
+
/* readv callback is is passed READ_BUFFERS_* flags as callback data */
const PgAioHandleCallbacks aio_shared_buffer_readv_cb = {
@@ -6693,6 +6772,11 @@ const PgAioHandleCallbacks aio_shared_buffer_readv_cb = {
.report = buffer_readv_report,
};
+const PgAioHandleCallbacks aio_shared_buffer_writev_cb = {
+ .stage = shared_buffer_writev_stage,
+ .complete_shared = shared_buffer_writev_complete,
+};
+
/* readv callback is is passed READ_BUFFERS_* flags as callback data */
const PgAioHandleCallbacks aio_local_buffer_readv_cb = {
.stage = local_buffer_readv_stage,
@@ -6706,3 +6790,7 @@ const PgAioHandleCallbacks aio_local_buffer_readv_cb = {
.complete_local = local_buffer_readv_complete,
.report = buffer_readv_report,
};
+
+const PgAioHandleCallbacks aio_local_buffer_writev_cb = {
+ .stage = local_buffer_writev_stage,
+};
--
2.48.1.76.g4e746b1a31.dirty
[text/x-diff] v2.6-0029-aio-Add-IO-queue-helper.patch (7.4K, ../../ge6nsuddurhpmll3xj22vucvqwp4agqz6ndtcf2mhyeydzarst@l75dman5x53p/30-v2.6-0029-aio-Add-IO-queue-helper.patch)
download | inline diff:
From 43d16b992593ff7ad54452a9531eb29c9a31811d Mon Sep 17 00:00:00 2001
From: Andres Freund <andres@anarazel.de>
Date: Wed, 22 Jan 2025 13:44:50 -0500
Subject: [PATCH v2.6 29/34] aio: Add IO queue helper
This is likely never going to anywhere - Thomas Munro is working on something
more complete. But I needed a way to exercise aio for checkpointer / bgwriter.
---
src/include/storage/io_queue.h | 33 +++++
src/backend/storage/aio/Makefile | 1 +
src/backend/storage/aio/io_queue.c | 204 ++++++++++++++++++++++++++++
src/backend/storage/aio/meson.build | 1 +
src/tools/pgindent/typedefs.list | 2 +
5 files changed, 241 insertions(+)
create mode 100644 src/include/storage/io_queue.h
create mode 100644 src/backend/storage/aio/io_queue.c
diff --git a/src/include/storage/io_queue.h b/src/include/storage/io_queue.h
new file mode 100644
index 00000000000..92b1e9afe6f
--- /dev/null
+++ b/src/include/storage/io_queue.h
@@ -0,0 +1,33 @@
+/*-------------------------------------------------------------------------
+ *
+ * io_queue.h
+ * Mechanism for tracking many IOs
+ *
+ *
+ * Portions Copyright (c) 1996-2025, PostgreSQL Global Development Group
+ * Portions Copyright (c) 1994, Regents of the University of California
+ *
+ * src/include/storage/io_queue.h
+ *
+ *-------------------------------------------------------------------------
+ */
+#ifndef IO_QUEUE_H
+#define IO_QUEUE_H
+
+#include "storage/aio_types.h"
+
+struct IOQueue;
+typedef struct IOQueue IOQueue;
+
+struct PgAioWaitRef;
+
+extern IOQueue *io_queue_create(int depth, int flags);
+extern void io_queue_track(IOQueue *ioq, const PgAioWaitRef *iow);
+extern void io_queue_wait_one(IOQueue *ioq);
+extern void io_queue_wait_all(IOQueue *ioq);
+extern bool io_queue_is_empty(IOQueue *ioq);
+extern void io_queue_reserve(IOQueue *ioq);
+extern PgAioHandle *io_queue_acquire_io(IOQueue *ioq);
+extern void io_queue_free(IOQueue *ioq);
+
+#endif /* IO_QUEUE_H */
diff --git a/src/backend/storage/aio/Makefile b/src/backend/storage/aio/Makefile
index 3f2469cc399..86fa4276fda 100644
--- a/src/backend/storage/aio/Makefile
+++ b/src/backend/storage/aio/Makefile
@@ -15,6 +15,7 @@ OBJS = \
aio_init.o \
aio_io.o \
aio_target.o \
+ io_queue.o \
method_io_uring.o \
method_sync.o \
method_worker.o \
diff --git a/src/backend/storage/aio/io_queue.c b/src/backend/storage/aio/io_queue.c
new file mode 100644
index 00000000000..526aa1d5e06
--- /dev/null
+++ b/src/backend/storage/aio/io_queue.c
@@ -0,0 +1,204 @@
+/*-------------------------------------------------------------------------
+ *
+ * io_queue.c
+ * AIO - Mechanism for tracking many IOs
+ *
+ * Portions Copyright (c) 1996-2025, PostgreSQL Global Development Group
+ * Portions Copyright (c) 1994, Regents of the University of California
+ *
+ * IDENTIFICATION
+ * src/backend/storage/aio/io_queue.c
+ *
+ *-------------------------------------------------------------------------
+ */
+
+#include "postgres.h"
+
+#include "lib/ilist.h"
+#include "storage/aio.h"
+#include "storage/io_queue.h"
+#include "utils/resowner.h"
+
+
+
+typedef struct TrackedIO
+{
+ PgAioWaitRef iow;
+ dlist_node node;
+} TrackedIO;
+
+struct IOQueue
+{
+ int depth;
+ int unsubmitted;
+
+ bool has_reserved;
+
+ dclist_head idle;
+ dclist_head in_progress;
+
+ TrackedIO tracked_ios[FLEXIBLE_ARRAY_MEMBER];
+};
+
+
+IOQueue *
+io_queue_create(int depth, int flags)
+{
+ size_t sz;
+ IOQueue *ioq;
+
+ sz = offsetof(IOQueue, tracked_ios)
+ + sizeof(TrackedIO) * depth;
+
+ ioq = palloc0(sz);
+
+ ioq->depth = 0;
+
+ for (int i = 0; i < depth; i++)
+ {
+ TrackedIO *tio = &ioq->tracked_ios[i];
+
+ pgaio_wref_clear(&tio->iow);
+ dclist_push_tail(&ioq->idle, &tio->node);
+ }
+
+ return ioq;
+}
+
+void
+io_queue_wait_one(IOQueue *ioq)
+{
+ /* submit all pending IO before waiting */
+ pgaio_submit_staged();
+
+ while (!dclist_is_empty(&ioq->in_progress))
+ {
+ /* FIXME: Should we really pop here already? */
+ dlist_node *node = dclist_pop_head_node(&ioq->in_progress);
+ TrackedIO *tio = dclist_container(TrackedIO, node, node);
+
+ pgaio_wref_wait(&tio->iow);
+ dclist_push_head(&ioq->idle, &tio->node);
+ }
+}
+
+void
+io_queue_reserve(IOQueue *ioq)
+{
+ if (ioq->has_reserved)
+ return;
+
+ if (dclist_is_empty(&ioq->idle))
+ io_queue_wait_one(ioq);
+
+ Assert(!dclist_is_empty(&ioq->idle));
+
+ ioq->has_reserved = true;
+}
+
+PgAioHandle *
+io_queue_acquire_io(IOQueue *ioq)
+{
+ PgAioHandle *ioh;
+
+ io_queue_reserve(ioq);
+
+ Assert(!dclist_is_empty(&ioq->idle));
+
+ if (!io_queue_is_empty(ioq))
+ {
+ ioh = pgaio_io_acquire_nb(CurrentResourceOwner, NULL);
+ if (ioh == NULL)
+ {
+ /*
+ * Need to wait for all IOs, blocking might not be legal in the
+ * context.
+ *
+ * XXX: This doesn't make a whole lot of sense, we're also
+ * blocking here. What was I smoking when I wrote the above?
+ */
+ io_queue_wait_all(ioq);
+ ioh = pgaio_io_acquire(CurrentResourceOwner, NULL);
+ }
+ }
+ else
+ {
+ ioh = pgaio_io_acquire(CurrentResourceOwner, NULL);
+ }
+
+ return ioh;
+}
+
+void
+io_queue_track(IOQueue *ioq, const struct PgAioWaitRef *iow)
+{
+ dlist_node *node;
+ TrackedIO *tio;
+
+ Assert(ioq->has_reserved);
+ ioq->has_reserved = false;
+
+ Assert(!dclist_is_empty(&ioq->idle));
+
+ node = dclist_pop_head_node(&ioq->idle);
+ tio = dclist_container(TrackedIO, node, node);
+
+ tio->iow = *iow;
+
+ dclist_push_tail(&ioq->in_progress, &tio->node);
+
+ ioq->unsubmitted++;
+
+ /*
+ * XXX: Should have some smarter logic here. We don't want to wait too
+ * long to submit, that'll mean we're more likely to block. But we also
+ * don't want to have the overhead of submitting every IO individually.
+ */
+ if (ioq->unsubmitted >= 4)
+ {
+ pgaio_submit_staged();
+ ioq->unsubmitted = 0;
+ }
+}
+
+void
+io_queue_wait_all(IOQueue *ioq)
+{
+ /* submit all pending IO before waiting */
+ pgaio_submit_staged();
+
+ while (!dclist_is_empty(&ioq->in_progress))
+ {
+ /* wait for the last IO to minimize unnecessary wakeups */
+ dlist_node *node = dclist_tail_node(&ioq->in_progress);
+ TrackedIO *tio = dclist_container(TrackedIO, node, node);
+
+ if (!pgaio_wref_check_done(&tio->iow))
+ {
+ ereport(DEBUG3,
+ errmsg("io_queue_wait_all for io:%d",
+ pgaio_wref_get_id(&tio->iow)),
+ errhidestmt(true),
+ errhidecontext(true));
+
+ pgaio_wref_wait(&tio->iow);
+ }
+
+ dclist_delete_from(&ioq->in_progress, &tio->node);
+ dclist_push_head(&ioq->idle, &tio->node);
+ }
+}
+
+bool
+io_queue_is_empty(IOQueue *ioq)
+{
+ return dclist_is_empty(&ioq->in_progress);
+}
+
+void
+io_queue_free(IOQueue *ioq)
+{
+ io_queue_wait_all(ioq);
+
+ pfree(ioq);
+}
diff --git a/src/backend/storage/aio/meson.build b/src/backend/storage/aio/meson.build
index da6df2d3654..270c4a64428 100644
--- a/src/backend/storage/aio/meson.build
+++ b/src/backend/storage/aio/meson.build
@@ -7,6 +7,7 @@ backend_sources += files(
'aio_init.c',
'aio_io.c',
'aio_target.c',
+ 'io_queue.c',
'method_io_uring.c',
'method_sync.c',
'method_worker.c',
diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list
index fc379ff27eb..92ccd2e0514 100644
--- a/src/tools/pgindent/typedefs.list
+++ b/src/tools/pgindent/typedefs.list
@@ -1190,6 +1190,7 @@ IOContext
IOFuncSelector
IOObject
IOOp
+IOQueue
IO_STATUS_BLOCK
IPCompareMethod
ITEM
@@ -3011,6 +3012,7 @@ TocEntry
TokenAuxData
TokenizedAuthLine
TrackItem
+TrackedIO
TransApplyAction
TransInvalidationInfo
TransState
--
2.48.1.76.g4e746b1a31.dirty
[text/x-diff] v2.6-0030-bufmgr-use-AIO-in-checkpointer-bgwriter.patch (31.1K, ../../ge6nsuddurhpmll3xj22vucvqwp4agqz6ndtcf2mhyeydzarst@l75dman5x53p/31-v2.6-0030-bufmgr-use-AIO-in-checkpointer-bgwriter.patch)
download | inline diff:
From 4a65eb21e12504385b1c334758e5ce53d6100896 Mon Sep 17 00:00:00 2001
From: Andres Freund <andres@anarazel.de>
Date: Wed, 22 Jan 2025 13:44:52 -0500
Subject: [PATCH v2.6 30/34] bufmgr: use AIO in checkpointer, bgwriter
This is far from ready - just included to be able to exercise AIO writes and
get some preliminary numbers. In all likelihood this will instead be based
on-top of work by Thomas Munro instead of the preceding commit.
---
src/include/postmaster/bgwriter.h | 3 +-
src/include/storage/buf_internals.h | 2 +
src/include/storage/bufmgr.h | 3 +-
src/include/storage/bufpage.h | 1 +
src/backend/postmaster/bgwriter.c | 19 +-
src/backend/postmaster/checkpointer.c | 11 +-
src/backend/storage/buffer/bufmgr.c | 588 +++++++++++++++++++++++---
src/backend/storage/page/bufpage.c | 10 +
src/tools/pgindent/typedefs.list | 1 +
9 files changed, 580 insertions(+), 58 deletions(-)
diff --git a/src/include/postmaster/bgwriter.h b/src/include/postmaster/bgwriter.h
index 4fd717169f0..4208d5f2c97 100644
--- a/src/include/postmaster/bgwriter.h
+++ b/src/include/postmaster/bgwriter.h
@@ -31,7 +31,8 @@ extern void BackgroundWriterMain(const void *startup_data, size_t startup_data_l
extern void CheckpointerMain(const void *startup_data, size_t startup_data_len) pg_attribute_noreturn();
extern void RequestCheckpoint(int flags);
-extern void CheckpointWriteDelay(int flags, double progress);
+struct IOQueue;
+extern void CheckpointWriteDelay(struct IOQueue *ioq, int flags, double progress);
extern bool ForwardSyncRequest(const FileTag *ftag, SyncRequestType type);
diff --git a/src/include/storage/buf_internals.h b/src/include/storage/buf_internals.h
index 3d7ebf96a7e..3db2f786177 100644
--- a/src/include/storage/buf_internals.h
+++ b/src/include/storage/buf_internals.h
@@ -21,6 +21,8 @@
#include "storage/buf.h"
#include "storage/bufmgr.h"
#include "storage/condition_variable.h"
+#include "storage/io_queue.h"
+#include "storage/latch.h"
#include "storage/lwlock.h"
#include "storage/procnumber.h"
#include "storage/shmem.h"
diff --git a/src/include/storage/bufmgr.h b/src/include/storage/bufmgr.h
index a2bff99fb55..9ed4468d8e1 100644
--- a/src/include/storage/bufmgr.h
+++ b/src/include/storage/bufmgr.h
@@ -299,7 +299,8 @@ extern bool ConditionalLockBufferForCleanup(Buffer buffer);
extern bool IsBufferCleanupOK(Buffer buffer);
extern bool HoldingBufferPinThatDelaysRecovery(void);
-extern bool BgBufferSync(struct WritebackContext *wb_context);
+struct IOQueue;
+extern bool BgBufferSync(struct IOQueue *ioq, struct WritebackContext *wb_context);
extern uint32 GetSoftPinLimit(void);
extern uint32 GetSoftLocalPinLimit(void);
diff --git a/src/include/storage/bufpage.h b/src/include/storage/bufpage.h
index 6646b6f6371..9c045e81857 100644
--- a/src/include/storage/bufpage.h
+++ b/src/include/storage/bufpage.h
@@ -509,5 +509,6 @@ extern bool PageIndexTupleOverwrite(Page page, OffsetNumber offnum,
Item newtup, Size newsize);
extern char *PageSetChecksumCopy(Page page, BlockNumber blkno);
extern void PageSetChecksumInplace(Page page, BlockNumber blkno);
+extern bool PageNeedsChecksumCopy(Page page);
#endif /* BUFPAGE_H */
diff --git a/src/backend/postmaster/bgwriter.c b/src/backend/postmaster/bgwriter.c
index 72f5acceec7..6e8801a39e3 100644
--- a/src/backend/postmaster/bgwriter.c
+++ b/src/backend/postmaster/bgwriter.c
@@ -38,11 +38,13 @@
#include "postmaster/auxprocess.h"
#include "postmaster/bgwriter.h"
#include "postmaster/interrupt.h"
+#include "storage/aio.h"
#include "storage/aio_subsys.h"
#include "storage/buf_internals.h"
#include "storage/bufmgr.h"
#include "storage/condition_variable.h"
#include "storage/fd.h"
+#include "storage/io_queue.h"
#include "storage/lwlock.h"
#include "storage/proc.h"
#include "storage/procsignal.h"
@@ -90,6 +92,7 @@ BackgroundWriterMain(const void *startup_data, size_t startup_data_len)
sigjmp_buf local_sigjmp_buf;
MemoryContext bgwriter_context;
bool prev_hibernate;
+ IOQueue *ioq;
WritebackContext wb_context;
Assert(startup_data_len == 0);
@@ -131,6 +134,7 @@ BackgroundWriterMain(const void *startup_data, size_t startup_data_len)
ALLOCSET_DEFAULT_SIZES);
MemoryContextSwitchTo(bgwriter_context);
+ ioq = io_queue_create(128, 0);
WritebackContextInit(&wb_context, &bgwriter_flush_after);
/*
@@ -228,12 +232,22 @@ BackgroundWriterMain(const void *startup_data, size_t startup_data_len)
/* Clear any already-pending wakeups */
ResetLatch(MyLatch);
+ /*
+ * FIXME: this is theoretically racy, but I didn't want to copy
+ * ProcessMainLoopInterrupts() remaining body here.
+ */
+ if (ShutdownRequestPending)
+ {
+ io_queue_wait_all(ioq);
+ io_queue_free(ioq);
+ }
+
ProcessMainLoopInterrupts();
/*
* Do one cycle of dirty-buffer writing.
*/
- can_hibernate = BgBufferSync(&wb_context);
+ can_hibernate = BgBufferSync(ioq, &wb_context);
/* Report pending statistics to the cumulative stats system */
pgstat_report_bgwriter();
@@ -250,6 +264,9 @@ BackgroundWriterMain(const void *startup_data, size_t startup_data_len)
smgrdestroyall();
}
+ /* finish IO before sleeping, to avoid blocking other backends */
+ io_queue_wait_all(ioq);
+
/*
* Log a new xl_running_xacts every now and then so replication can
* get into a consistent state faster (think of suboverflowed
diff --git a/src/backend/postmaster/checkpointer.c b/src/backend/postmaster/checkpointer.c
index fda91ffd1ce..904fe167eb4 100644
--- a/src/backend/postmaster/checkpointer.c
+++ b/src/backend/postmaster/checkpointer.c
@@ -49,10 +49,12 @@
#include "postmaster/bgwriter.h"
#include "postmaster/interrupt.h"
#include "replication/syncrep.h"
+#include "storage/aio.h"
#include "storage/aio_subsys.h"
#include "storage/bufmgr.h"
#include "storage/condition_variable.h"
#include "storage/fd.h"
+#include "storage/io_queue.h"
#include "storage/ipc.h"
#include "storage/lwlock.h"
#include "storage/pmsignal.h"
@@ -766,7 +768,7 @@ ImmediateCheckpointRequested(void)
* fraction between 0.0 meaning none, and 1.0 meaning all done.
*/
void
-CheckpointWriteDelay(int flags, double progress)
+CheckpointWriteDelay(IOQueue *ioq, int flags, double progress)
{
static int absorb_counter = WRITES_PER_ABSORB;
@@ -800,6 +802,13 @@ CheckpointWriteDelay(int flags, double progress)
/* Report interim statistics to the cumulative stats system */
pgstat_report_checkpointer();
+ /*
+ * Ensure all pending IO is submitted to avoid unnecessary delays for
+ * other processes.
+ */
+ io_queue_wait_all(ioq);
+
+
/*
* This sleep used to be connected to bgwriter_delay, typically 200ms.
* That resulted in more frequent wakeups if not much work to do.
diff --git a/src/backend/storage/buffer/bufmgr.c b/src/backend/storage/buffer/bufmgr.c
index 1f47edaa7b9..088a6137b60 100644
--- a/src/backend/storage/buffer/bufmgr.c
+++ b/src/backend/storage/buffer/bufmgr.c
@@ -52,6 +52,7 @@
#include "storage/buf_internals.h"
#include "storage/bufmgr.h"
#include "storage/fd.h"
+#include "storage/io_queue.h"
#include "storage/ipc.h"
#include "storage/lmgr.h"
#include "storage/proc.h"
@@ -76,6 +77,7 @@
/* Bits in SyncOneBuffer's return value */
#define BUF_WRITTEN 0x01
#define BUF_REUSABLE 0x02
+#define BUF_CANT_MERGE 0x04
#define RELS_BSEARCH_THRESHOLD 20
@@ -512,8 +514,6 @@ static void UnpinBuffer(BufferDesc *buf);
static void UnpinBufferNoOwner(BufferDesc *buf);
static void BufferSync(int flags);
static uint32 WaitBufHdrUnlocked(BufferDesc *buf);
-static int SyncOneBuffer(int buf_id, bool skip_recently_used,
- WritebackContext *wb_context);
static void WaitIO(BufferDesc *buf);
static void AbortBufferIO(Buffer buffer);
static void shared_buffer_write_error_callback(void *arg);
@@ -529,6 +529,7 @@ static bool AsyncReadBuffers(ReadBuffersOperation *operation,
static Buffer GetVictimBuffer(BufferAccessStrategy strategy, IOContext io_context);
static void FlushBuffer(BufferDesc *buf, SMgrRelation reln,
IOObject io_object, IOContext io_context);
+
static void FindAndDropRelationBuffers(RelFileLocator rlocator,
ForkNumber forkNum,
BlockNumber nForkBlock,
@@ -3151,6 +3152,57 @@ UnpinBufferNoOwner(BufferDesc *buf)
}
}
+typedef struct BuffersToWrite
+{
+ int nbuffers;
+ BufferTag start_at_tag;
+ uint32 max_combine;
+
+ XLogRecPtr max_lsn;
+
+ PgAioHandle *ioh;
+ PgAioWaitRef iow;
+
+ uint64 total_writes;
+
+ Buffer buffers[IOV_MAX];
+ PgAioBounceBuffer *bounce_buffers[IOV_MAX];
+ const void *data_ptrs[IOV_MAX];
+} BuffersToWrite;
+
+static int PrepareToWriteBuffer(BuffersToWrite *to_write, Buffer buf,
+ bool skip_recently_used,
+ IOQueue *ioq, WritebackContext *wb_context);
+
+static void WriteBuffers(BuffersToWrite *to_write,
+ IOQueue *ioq, WritebackContext *wb_context);
+
+static void
+BuffersToWriteInit(BuffersToWrite *to_write,
+ IOQueue *ioq, WritebackContext *wb_context)
+{
+ to_write->total_writes = 0;
+ to_write->nbuffers = 0;
+ to_write->ioh = NULL;
+ pgaio_wref_clear(&to_write->iow);
+ to_write->max_lsn = InvalidXLogRecPtr;
+
+ pgaio_enter_batchmode();
+}
+
+static void
+BuffersToWriteEnd(BuffersToWrite *to_write)
+{
+ if (to_write->ioh != NULL)
+ {
+ pgaio_io_release(to_write->ioh);
+ to_write->ioh = NULL;
+ }
+
+ pgaio_exit_batchmode();
+}
+
+
#define ST_SORT sort_checkpoint_bufferids
#define ST_ELEMENT_TYPE CkptSortItem
#define ST_COMPARE(a, b) ckpt_buforder_comparator(a, b)
@@ -3182,7 +3234,10 @@ BufferSync(int flags)
binaryheap *ts_heap;
int i;
int mask = BM_DIRTY;
+ IOQueue *ioq;
WritebackContext wb_context;
+ BuffersToWrite to_write;
+ int max_combine;
/*
* Unless this is a shutdown checkpoint or we have been explicitly told,
@@ -3244,7 +3299,9 @@ BufferSync(int flags)
if (num_to_scan == 0)
return; /* nothing to do */
+ ioq = io_queue_create(512, 0);
WritebackContextInit(&wb_context, &checkpoint_flush_after);
+ max_combine = Min(io_bounce_buffers, io_combine_limit);
TRACE_POSTGRESQL_BUFFER_SYNC_START(NBuffers, num_to_scan);
@@ -3352,48 +3409,91 @@ BufferSync(int flags)
*/
num_processed = 0;
num_written = 0;
+
+ BuffersToWriteInit(&to_write, ioq, &wb_context);
+
while (!binaryheap_empty(ts_heap))
{
BufferDesc *bufHdr = NULL;
CkptTsStatus *ts_stat = (CkptTsStatus *)
DatumGetPointer(binaryheap_first(ts_heap));
+ bool batch_continue = true;
- buf_id = CkptBufferIds[ts_stat->index].buf_id;
- Assert(buf_id != -1);
-
- bufHdr = GetBufferDescriptor(buf_id);
-
- num_processed++;
+ Assert(ts_stat->num_scanned <= ts_stat->num_to_scan);
/*
- * We don't need to acquire the lock here, because we're only looking
- * at a single bit. It's possible that someone else writes the buffer
- * and clears the flag right after we check, but that doesn't matter
- * since SyncOneBuffer will then do nothing. However, there is a
- * further race condition: it's conceivable that between the time we
- * examine the bit here and the time SyncOneBuffer acquires the lock,
- * someone else not only wrote the buffer but replaced it with another
- * page and dirtied it. In that improbable case, SyncOneBuffer will
- * write the buffer though we didn't need to. It doesn't seem worth
- * guarding against this, though.
+ * Collect a batch of buffers to write out from the current
+ * tablespace. That causes some imbalance between the tablespaces, but
+ * that's more than outweighed by the efficiency gain due to batching.
*/
- if (pg_atomic_read_u32(&bufHdr->state) & BM_CHECKPOINT_NEEDED)
+ while (batch_continue &&
+ to_write.nbuffers < max_combine &&
+ ts_stat->num_scanned < ts_stat->num_to_scan)
{
- if (SyncOneBuffer(buf_id, false, &wb_context) & BUF_WRITTEN)
+ buf_id = CkptBufferIds[ts_stat->index].buf_id;
+ Assert(buf_id != -1);
+
+ bufHdr = GetBufferDescriptor(buf_id);
+
+ num_processed++;
+
+ /*
+ * We don't need to acquire the lock here, because we're only
+ * looking at a single bit. It's possible that someone else writes
+ * the buffer and clears the flag right after we check, but that
+ * doesn't matter since SyncOneBuffer will then do nothing.
+ * However, there is a further race condition: it's conceivable
+ * that between the time we examine the bit here and the time
+ * SyncOneBuffer acquires the lock, someone else not only wrote
+ * the buffer but replaced it with another page and dirtied it. In
+ * that improbable case, SyncOneBuffer will write the buffer
+ * though we didn't need to. It doesn't seem worth guarding
+ * against this, though.
+ */
+ if (pg_atomic_read_u32(&bufHdr->state) & BM_CHECKPOINT_NEEDED)
{
- TRACE_POSTGRESQL_BUFFER_SYNC_WRITTEN(buf_id);
- PendingCheckpointerStats.buffers_written++;
- num_written++;
+ int result = PrepareToWriteBuffer(&to_write, buf_id + 1, false,
+ ioq, &wb_context);
+
+ if (result & BUF_CANT_MERGE)
+ {
+ Assert(to_write.nbuffers > 0);
+ WriteBuffers(&to_write, ioq, &wb_context);
+
+ result = PrepareToWriteBuffer(&to_write, buf_id + 1, false,
+ ioq, &wb_context);
+ Assert(result != BUF_CANT_MERGE);
+ }
+
+ if (result & BUF_WRITTEN)
+ {
+ TRACE_POSTGRESQL_BUFFER_SYNC_WRITTEN(buf_id);
+ PendingCheckpointerStats.buffers_written++;
+ num_written++;
+ }
+ else
+ {
+ batch_continue = false;
+ }
}
+ else
+ {
+ if (to_write.nbuffers > 0)
+ WriteBuffers(&to_write, ioq, &wb_context);
+ }
+
+ /*
+ * Measure progress independent of actually having to flush the
+ * buffer - otherwise writing become unbalanced.
+ */
+ ts_stat->progress += ts_stat->progress_slice;
+ ts_stat->num_scanned++;
+ ts_stat->index++;
}
- /*
- * Measure progress independent of actually having to flush the buffer
- * - otherwise writing become unbalanced.
- */
- ts_stat->progress += ts_stat->progress_slice;
- ts_stat->num_scanned++;
- ts_stat->index++;
+ if (to_write.nbuffers > 0)
+ WriteBuffers(&to_write, ioq, &wb_context);
+
/* Have all the buffers from the tablespace been processed? */
if (ts_stat->num_scanned == ts_stat->num_to_scan)
@@ -3411,15 +3511,23 @@ BufferSync(int flags)
*
* (This will check for barrier events even if it doesn't sleep.)
*/
- CheckpointWriteDelay(flags, (double) num_processed / num_to_scan);
+ CheckpointWriteDelay(ioq, flags, (double) num_processed / num_to_scan);
}
+ Assert(to_write.nbuffers == 0);
+ io_queue_wait_all(ioq);
+
/*
* Issue all pending flushes. Only checkpointer calls BufferSync(), so
* IOContext will always be IOCONTEXT_NORMAL.
*/
IssuePendingWritebacks(&wb_context, IOCONTEXT_NORMAL);
+ io_queue_wait_all(ioq); /* IssuePendingWritebacks might have added
+ * more */
+ io_queue_free(ioq);
+ BuffersToWriteEnd(&to_write);
+
pfree(per_ts_stat);
per_ts_stat = NULL;
binaryheap_free(ts_heap);
@@ -3445,7 +3553,7 @@ BufferSync(int flags)
* bgwriter_lru_maxpages to 0.)
*/
bool
-BgBufferSync(WritebackContext *wb_context)
+BgBufferSync(IOQueue *ioq, WritebackContext *wb_context)
{
/* info obtained from freelist.c */
int strategy_buf_id;
@@ -3488,6 +3596,9 @@ BgBufferSync(WritebackContext *wb_context)
long new_strategy_delta;
uint32 new_recent_alloc;
+ BuffersToWrite to_write;
+ int max_combine;
+
/*
* Find out where the freelist clock sweep currently is, and how many
* buffer allocations have happened since our last call.
@@ -3508,6 +3619,8 @@ BgBufferSync(WritebackContext *wb_context)
return true;
}
+ max_combine = Min(io_bounce_buffers, io_combine_limit);
+
/*
* Compute strategy_delta = how many buffers have been scanned by the
* clock sweep since last time. If first time through, assume none. Then
@@ -3664,11 +3777,25 @@ BgBufferSync(WritebackContext *wb_context)
num_written = 0;
reusable_buffers = reusable_buffers_est;
+ BuffersToWriteInit(&to_write, ioq, wb_context);
+
/* Execute the LRU scan */
while (num_to_scan > 0 && reusable_buffers < upcoming_alloc_est)
{
- int sync_state = SyncOneBuffer(next_to_clean, true,
- wb_context);
+ int sync_state;
+
+ sync_state = PrepareToWriteBuffer(&to_write, next_to_clean + 1,
+ true, ioq, wb_context);
+ if (sync_state & BUF_CANT_MERGE)
+ {
+ Assert(to_write.nbuffers > 0);
+
+ WriteBuffers(&to_write, ioq, wb_context);
+
+ sync_state = PrepareToWriteBuffer(&to_write, next_to_clean + 1,
+ true, ioq, wb_context);
+ Assert(sync_state != BUF_CANT_MERGE);
+ }
if (++next_to_clean >= NBuffers)
{
@@ -3679,6 +3806,13 @@ BgBufferSync(WritebackContext *wb_context)
if (sync_state & BUF_WRITTEN)
{
+ Assert(sync_state & BUF_REUSABLE);
+
+ if (to_write.nbuffers == max_combine)
+ {
+ WriteBuffers(&to_write, ioq, wb_context);
+ }
+
reusable_buffers++;
if (++num_written >= bgwriter_lru_maxpages)
{
@@ -3690,6 +3824,11 @@ BgBufferSync(WritebackContext *wb_context)
reusable_buffers++;
}
+ if (to_write.nbuffers > 0)
+ WriteBuffers(&to_write, ioq, wb_context);
+
+ BuffersToWriteEnd(&to_write);
+
PendingBgWriterStats.buf_written_clean += num_written;
#ifdef BGW_DEBUG
@@ -3728,8 +3867,66 @@ BgBufferSync(WritebackContext *wb_context)
return (bufs_to_lap == 0 && recent_alloc == 0);
}
+static inline bool
+BufferTagsSameRel(const BufferTag *tag1, const BufferTag *tag2)
+{
+ return (tag1->spcOid == tag2->spcOid) &&
+ (tag1->dbOid == tag2->dbOid) &&
+ (tag1->relNumber == tag2->relNumber) &&
+ (tag1->forkNum == tag2->forkNum)
+ ;
+}
+
+static bool
+CanMergeWrite(BuffersToWrite *to_write, BufferDesc *cur_buf_hdr)
+{
+ BlockNumber cur_block = cur_buf_hdr->tag.blockNum;
+
+ Assert(to_write->nbuffers > 0); /* can't merge with nothing */
+ Assert(to_write->start_at_tag.relNumber != InvalidOid);
+ Assert(to_write->start_at_tag.blockNum != InvalidBlockNumber);
+
+ Assert(to_write->ioh != NULL);
+
+ /*
+ * First check if the blocknumber is one that we could actually merge,
+ * that's cheaper than checking the tablespace/db/relnumber/fork match.
+ */
+ if (to_write->start_at_tag.blockNum + to_write->nbuffers != cur_block)
+ return false;
+
+ if (!BufferTagsSameRel(&to_write->start_at_tag, &cur_buf_hdr->tag))
+ return false;
+
+ /*
+ * Need to check with smgr how large a write we're allowed to make. To
+ * reduce the overhead of the smgr check, only inquire once, when
+ * processing the first to-be-merged buffer. That avoids the overhead in
+ * the common case of writing out buffers that definitely not mergeable.
+ */
+ if (to_write->nbuffers == 1)
+ {
+ SMgrRelation smgr;
+
+ smgr = smgropen(BufTagGetRelFileLocator(&to_write->start_at_tag), INVALID_PROC_NUMBER);
+
+ to_write->max_combine = smgrmaxcombine(smgr,
+ to_write->start_at_tag.forkNum,
+ to_write->start_at_tag.blockNum);
+ }
+ else
+ {
+ Assert(to_write->max_combine > 0);
+ }
+
+ if (to_write->start_at_tag.blockNum + to_write->max_combine <= cur_block)
+ return false;
+
+ return true;
+}
+
/*
- * SyncOneBuffer -- process a single buffer during syncing.
+ * PrepareToWriteBuffer -- process a single buffer during syncing.
*
* If skip_recently_used is true, we don't write currently-pinned buffers, nor
* buffers marked recently used, as these are not replacement candidates.
@@ -3738,22 +3935,50 @@ BgBufferSync(WritebackContext *wb_context)
* BUF_WRITTEN: we wrote the buffer.
* BUF_REUSABLE: buffer is available for replacement, ie, it has
* pin count 0 and usage count 0.
+ * BUF_CANT_MERGE: can't combine this write with prior writes, caller needs
+ * to issue those first
*
* (BUF_WRITTEN could be set in error if FlushBuffer finds the buffer clean
* after locking it, but we don't care all that much.)
*/
static int
-SyncOneBuffer(int buf_id, bool skip_recently_used, WritebackContext *wb_context)
+PrepareToWriteBuffer(BuffersToWrite *to_write, Buffer buf,
+ bool skip_recently_used,
+ IOQueue *ioq, WritebackContext *wb_context)
{
- BufferDesc *bufHdr = GetBufferDescriptor(buf_id);
+ BufferDesc *cur_buf_hdr = GetBufferDescriptor(buf - 1);
+ uint32 buf_state;
int result = 0;
- uint32 buf_state;
- BufferTag tag;
+ XLogRecPtr cur_buf_lsn;
+ LWLock *content_lock;
+ bool may_block;
+
+ /*
+ * Check if this buffer can be written out together with already prepared
+ * writes. We check before we have pinned the buffer, so the buffer can be
+ * written out and replaced between this check and us pinning the buffer -
+ * we'll recheck below. The reason for the pre-check is that we don't want
+ * to pin the buffer just to find out that we can't merge the IO.
+ */
+ if (to_write->nbuffers != 0)
+ {
+ if (!CanMergeWrite(to_write, cur_buf_hdr))
+ {
+ result |= BUF_CANT_MERGE;
+ return result;
+ }
+ }
+ else
+ {
+ to_write->start_at_tag = cur_buf_hdr->tag;
+ }
/* Make sure we can handle the pin */
ReservePrivateRefCountEntry();
ResourceOwnerEnlarge(CurrentResourceOwner);
+ /* XXX: Should also check if we are allowed to pin one more buffer */
+
/*
* Check whether buffer needs writing.
*
@@ -3763,7 +3988,7 @@ SyncOneBuffer(int buf_id, bool skip_recently_used, WritebackContext *wb_context)
* don't worry because our checkpoint.redo points before log record for
* upcoming changes and so we are not required to write such dirty buffer.
*/
- buf_state = LockBufHdr(bufHdr);
+ buf_state = LockBufHdr(cur_buf_hdr);
if (BUF_STATE_GET_REFCOUNT(buf_state) == 0 &&
BUF_STATE_GET_USAGECOUNT(buf_state) == 0)
@@ -3772,40 +3997,294 @@ SyncOneBuffer(int buf_id, bool skip_recently_used, WritebackContext *wb_context)
}
else if (skip_recently_used)
{
+#if 0
+ elog(LOG, "at block %d: skip recent with nbuffers %d",
+ cur_buf_hdr->tag.blockNum, to_write->nbuffers);
+#endif
/* Caller told us not to write recently-used buffers */
- UnlockBufHdr(bufHdr, buf_state);
+ UnlockBufHdr(cur_buf_hdr, buf_state);
return result;
}
if (!(buf_state & BM_VALID) || !(buf_state & BM_DIRTY))
{
/* It's clean, so nothing to do */
- UnlockBufHdr(bufHdr, buf_state);
+ UnlockBufHdr(cur_buf_hdr, buf_state);
return result;
}
+ /* pin the buffer, from now on its identity can't change anymore */
+ PinBuffer_Locked(cur_buf_hdr);
+
+ /*
+ * Acquire IO, if needed, now that it's likely that we'll need to write.
+ */
+ if (to_write->ioh == NULL)
+ {
+ /* otherwise we should already have acquired a handle */
+ Assert(to_write->nbuffers == 0);
+
+ to_write->ioh = io_queue_acquire_io(ioq);
+ pgaio_io_get_wref(to_write->ioh, &to_write->iow);
+ }
+
/*
- * Pin it, share-lock it, write it. (FlushBuffer will do nothing if the
- * buffer is clean by the time we've locked it.)
+ * If we are merging, check if the buffer's identity possibly changed
+ * while we hadn't yet pinned it.
+ *
+ * XXX: It might be worth checking if we still want to write the buffer
+ * out, e.g. it could have been replaced with a buffer that doesn't have
+ * BM_CHECKPOINT_NEEDED set.
*/
- PinBuffer_Locked(bufHdr);
- LWLockAcquire(BufferDescriptorGetContentLock(bufHdr), LW_SHARED);
+ if (to_write->nbuffers != 0)
+ {
+ if (!CanMergeWrite(to_write, cur_buf_hdr))
+ {
+ elog(LOG, "changed identity");
+ UnpinBuffer(cur_buf_hdr);
+
+ result |= BUF_CANT_MERGE;
+
+ return result;
+ }
+ }
+
+ may_block = to_write->nbuffers == 0
+ && !pgaio_have_staged()
+ && io_queue_is_empty(ioq)
+ ;
+ content_lock = BufferDescriptorGetContentLock(cur_buf_hdr);
+
+ if (!may_block)
+ {
+ if (LWLockConditionalAcquire(content_lock, LW_SHARED))
+ {
+ /* done */
+ }
+ else if (to_write->nbuffers == 0)
+ {
+ /*
+ * Need to wait for all prior IO to finish before blocking for
+ * lock acquisition, to avoid the risk a deadlock due to us
+ * waiting for another backend that is waiting for our unsubmitted
+ * IO to complete.
+ */
+ pgaio_submit_staged();
+ io_queue_wait_all(ioq);
+
+ elog(DEBUG2, "at block %u: can't block, nbuffers = 0",
+ cur_buf_hdr->tag.blockNum
+ );
+
+ may_block = to_write->nbuffers == 0
+ && !pgaio_have_staged()
+ && io_queue_is_empty(ioq)
+ ;
+ Assert(may_block);
+
+ LWLockAcquire(content_lock, LW_SHARED);
+ }
+ else
+ {
+ elog(DEBUG2, "at block %d: can't block nbuffers = %d",
+ cur_buf_hdr->tag.blockNum,
+ to_write->nbuffers);
- FlushBuffer(bufHdr, NULL, IOOBJECT_RELATION, IOCONTEXT_NORMAL);
+ UnpinBuffer(cur_buf_hdr);
+ result |= BUF_CANT_MERGE;
+ Assert(to_write->nbuffers > 0);
- LWLockRelease(BufferDescriptorGetContentLock(bufHdr));
+ return result;
+ }
+ }
+ else
+ {
+ LWLockAcquire(content_lock, LW_SHARED);
+ }
- tag = bufHdr->tag;
+ if (!may_block)
+ {
+ if (!StartBufferIO(cur_buf_hdr, false, !may_block))
+ {
+ pgaio_submit_staged();
+ io_queue_wait_all(ioq);
- UnpinBuffer(bufHdr);
+ may_block = io_queue_is_empty(ioq) && to_write->nbuffers == 0 && !pgaio_have_staged();
+
+ if (!StartBufferIO(cur_buf_hdr, false, !may_block))
+ {
+ elog(DEBUG2, "at block %d: non-waitable StartBufferIO returns false, %d",
+ cur_buf_hdr->tag.blockNum,
+ may_block);
+
+ /*
+ * FIXME: can't tell whether this is because the buffer has
+ * been cleaned
+ */
+ if (!may_block)
+ {
+ result |= BUF_CANT_MERGE;
+ Assert(to_write->nbuffers > 0);
+ }
+ LWLockRelease(content_lock);
+ UnpinBuffer(cur_buf_hdr);
+
+ return result;
+ }
+ }
+ }
+ else
+ {
+ if (!StartBufferIO(cur_buf_hdr, false, false))
+ {
+ elog(DEBUG2, "waitable StartBufferIO returns false");
+ LWLockRelease(content_lock);
+ UnpinBuffer(cur_buf_hdr);
+
+ /*
+ * FIXME: Historically we returned BUF_WRITTEN in this case, which
+ * seems wrong
+ */
+ return result;
+ }
+ }
/*
- * SyncOneBuffer() is only called by checkpointer and bgwriter, so
- * IOContext will always be IOCONTEXT_NORMAL.
+ * Run PageGetLSN while holding header lock, since we don't have the
+ * buffer locked exclusively in all cases.
*/
- ScheduleBufferTagForWriteback(wb_context, IOCONTEXT_NORMAL, &tag);
+ buf_state = LockBufHdr(cur_buf_hdr);
+
+ cur_buf_lsn = BufferGetLSN(cur_buf_hdr);
+
+ /* To check if block content changes while flushing. - vadim 01/17/97 */
+ buf_state &= ~BM_JUST_DIRTIED;
+
+ UnlockBufHdr(cur_buf_hdr, buf_state);
+
+ to_write->buffers[to_write->nbuffers] = buf;
+ to_write->nbuffers++;
+
+ if (buf_state & BM_PERMANENT &&
+ (to_write->max_lsn == InvalidXLogRecPtr || to_write->max_lsn < cur_buf_lsn))
+ {
+ to_write->max_lsn = cur_buf_lsn;
+ }
+
+ result |= BUF_WRITTEN;
+
+ return result;
+}
+
+static void
+WriteBuffers(BuffersToWrite *to_write,
+ IOQueue *ioq, WritebackContext *wb_context)
+{
+ SMgrRelation smgr;
+ Buffer first_buf;
+ BufferDesc *first_buf_hdr;
+ bool needs_checksum;
+
+ Assert(to_write->nbuffers > 0 && to_write->nbuffers <= io_combine_limit);
+
+ first_buf = to_write->buffers[0];
+ first_buf_hdr = GetBufferDescriptor(first_buf - 1);
+
+ smgr = smgropen(BufTagGetRelFileLocator(&first_buf_hdr->tag), INVALID_PROC_NUMBER);
+
+ /*
+ * Force XLOG flush up to buffer's LSN. This implements the basic WAL
+ * rule that log updates must hit disk before any of the data-file changes
+ * they describe do.
+ *
+ * However, this rule does not apply to unlogged relations, which will be
+ * lost after a crash anyway. Most unlogged relation pages do not bear
+ * LSNs since we never emit WAL records for them, and therefore flushing
+ * up through the buffer LSN would be useless, but harmless. However,
+ * GiST indexes use LSNs internally to track page-splits, and therefore
+ * unlogged GiST pages bear "fake" LSNs generated by
+ * GetFakeLSNForUnloggedRel. It is unlikely but possible that the fake
+ * LSN counter could advance past the WAL insertion point; and if it did
+ * happen, attempting to flush WAL through that location would fail, with
+ * disastrous system-wide consequences. To make sure that can't happen,
+ * skip the flush if the buffer isn't permanent.
+ */
+ if (to_write->max_lsn != InvalidXLogRecPtr)
+ XLogFlush(to_write->max_lsn);
+
+ /*
+ * Now it's safe to write buffer to disk. Note that no one else should
+ * have been able to write it while we were busy with log flushing because
+ * only one process at a time can set the BM_IO_IN_PROGRESS bit.
+ */
+
+ for (int nbuf = 0; nbuf < to_write->nbuffers; nbuf++)
+ {
+ Buffer cur_buf = to_write->buffers[nbuf];
+ BufferDesc *cur_buf_hdr = GetBufferDescriptor(cur_buf - 1);
+ Block bufBlock;
+ char *bufToWrite;
+
+ bufBlock = BufHdrGetBlock(cur_buf_hdr);
+ needs_checksum = PageNeedsChecksumCopy((Page) bufBlock);
+
+ /*
+ * Update page checksum if desired. Since we have only shared lock on
+ * the buffer, other processes might be updating hint bits in it, so
+ * we must copy the page to a bounce buffer if we do checksumming.
+ */
+ if (needs_checksum)
+ {
+ PgAioBounceBuffer *bb = pgaio_bounce_buffer_get();
+
+ pgaio_io_assoc_bounce_buffer(to_write->ioh, bb);
+
+ bufToWrite = pgaio_bounce_buffer_buffer(bb);
+ memcpy(bufToWrite, bufBlock, BLCKSZ);
+ PageSetChecksumInplace((Page) bufToWrite, cur_buf_hdr->tag.blockNum);
+ }
+ else
+ {
+ bufToWrite = bufBlock;
+ }
+
+ to_write->data_ptrs[nbuf] = bufToWrite;
+ }
+
+ pgaio_io_set_handle_data_32(to_write->ioh,
+ (uint32 *) to_write->buffers,
+ to_write->nbuffers);
+ pgaio_io_register_callbacks(to_write->ioh, PGAIO_HCB_SHARED_BUFFER_WRITEV, 0);
+
+ smgrstartwritev(to_write->ioh, smgr,
+ BufTagGetForkNum(&first_buf_hdr->tag),
+ first_buf_hdr->tag.blockNum,
+ to_write->data_ptrs,
+ to_write->nbuffers,
+ false);
+ pgstat_count_io_op(IOOBJECT_RELATION, IOCONTEXT_NORMAL,
+ IOOP_WRITE, 1, BLCKSZ * to_write->nbuffers);
+
+
+ for (int nbuf = 0; nbuf < to_write->nbuffers; nbuf++)
+ {
+ Buffer cur_buf = to_write->buffers[nbuf];
+ BufferDesc *cur_buf_hdr = GetBufferDescriptor(cur_buf - 1);
+
+ UnpinBuffer(cur_buf_hdr);
+ }
+
+ io_queue_track(ioq, &to_write->iow);
+ to_write->total_writes++;
- return result | BUF_WRITTEN;
+ /* clear state for next write */
+ to_write->nbuffers = 0;
+ to_write->start_at_tag.relNumber = InvalidOid;
+ to_write->start_at_tag.blockNum = InvalidBlockNumber;
+ to_write->max_combine = 0;
+ to_write->max_lsn = InvalidXLogRecPtr;
+ to_write->ioh = NULL;
+ pgaio_wref_clear(&to_write->iow);
}
/*
@@ -4179,6 +4658,7 @@ FlushBuffer(BufferDesc *buf, SMgrRelation reln, IOObject io_object,
error_context_stack = errcallback.previous;
}
+
/*
* RelationGetNumberOfBlocksInFork
* Determines the current number of pages in the specified relation fork.
diff --git a/src/backend/storage/page/bufpage.c b/src/backend/storage/page/bufpage.c
index ecc81aacfc3..71dc6f559dd 100644
--- a/src/backend/storage/page/bufpage.c
+++ b/src/backend/storage/page/bufpage.c
@@ -1480,6 +1480,16 @@ PageIndexTupleOverwrite(Page page, OffsetNumber offnum,
return true;
}
+bool
+PageNeedsChecksumCopy(Page page)
+{
+ if (PageIsNew(page))
+ return false;
+
+ /* If we don't need a checksum, just return the passed-in data */
+ return DataChecksumsEnabled();
+}
+
/*
* Set checksum for a page in shared buffers.
diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list
index 92ccd2e0514..c29a17a269f 100644
--- a/src/tools/pgindent/typedefs.list
+++ b/src/tools/pgindent/typedefs.list
@@ -348,6 +348,7 @@ BufferManagerRelation
BufferStrategyControl
BufferTag
BufferUsage
+BuffersToWrite
BuildAccumulator
BuiltinScript
BulkInsertState
--
2.48.1.76.g4e746b1a31.dirty
[text/x-diff] v2.6-0031-Ensure-a-resowner-exists-for-all-paths-that-may.patch (2.6K, ../../ge6nsuddurhpmll3xj22vucvqwp4agqz6ndtcf2mhyeydzarst@l75dman5x53p/32-v2.6-0031-Ensure-a-resowner-exists-for-all-paths-that-may.patch)
download | inline diff:
From 42d43019acba43a03ef19f9e762ab42b21718fc6 Mon Sep 17 00:00:00 2001
From: Andres Freund <andres@anarazel.de>
Date: Tue, 8 Oct 2024 14:34:38 -0400
Subject: [PATCH v2.6 31/34] Ensure a resowner exists for all paths that may
perform AIO
Reviewed-by: Noah Misch <noah@leadboat.com>
Discussion: https://postgr.es/m/1f6b50a7-38ef-4d87-8246-786d39f46ab9@iki.fi
---
src/backend/bootstrap/bootstrap.c | 7 +++++++
src/backend/replication/logical/logical.c | 6 ++++++
src/backend/utils/init/postinit.c | 6 +++++-
3 files changed, 18 insertions(+), 1 deletion(-)
diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c
index 6db864892d0..e554504e1f0 100644
--- a/src/backend/bootstrap/bootstrap.c
+++ b/src/backend/bootstrap/bootstrap.c
@@ -361,8 +361,15 @@ BootstrapModeMain(int argc, char *argv[], bool check_only)
BaseInit();
bootstrap_signals();
+
+ /* need a resowner for IO during BootStrapXLOG() */
+ CreateAuxProcessResourceOwner();
+
BootStrapXLOG(bootstrap_data_checksum_version);
+ ReleaseAuxProcessResources(true);
+ CurrentResourceOwner = NULL;
+
/*
* To ensure that src/common/link-canary.c is linked into the backend, we
* must call it from somewhere. Here is as good as anywhere.
diff --git a/src/backend/replication/logical/logical.c b/src/backend/replication/logical/logical.c
index 8ea846bfc3b..2b936714040 100644
--- a/src/backend/replication/logical/logical.c
+++ b/src/backend/replication/logical/logical.c
@@ -386,6 +386,12 @@ CreateInitDecodingContext(const char *plugin,
slot->data.plugin = plugin_name;
SpinLockRelease(&slot->mutex);
+ if (CurrentResourceOwner == NULL)
+ {
+ Assert(am_walsender);
+ CurrentResourceOwner = AuxProcessResourceOwner;
+ }
+
if (XLogRecPtrIsInvalid(restart_lsn))
ReplicationSlotReserveWal();
else
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index 9b134c03456..c890bf847e3 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -778,8 +778,12 @@ InitPostgres(const char *in_dbname, Oid dboid,
* We don't yet have an aux-process resource owner, but StartupXLOG
* and ShutdownXLOG will need one. Hence, create said resource owner
* (and register a callback to clean it up after ShutdownXLOG runs).
+ *
+ * In bootstrap mode CreateAuxProcessResourceOwner() was already
+ * called in BootstrapModeMain().
*/
- CreateAuxProcessResourceOwner();
+ if (!bootstrap)
+ CreateAuxProcessResourceOwner();
StartupXLOG();
/* Release (and warn about) any buffer pins leaked in StartupXLOG */
--
2.48.1.76.g4e746b1a31.dirty
[text/x-diff] v2.6-0032-Temporary-Increase-BAS_BULKREAD-size.patch (1.3K, ../../ge6nsuddurhpmll3xj22vucvqwp4agqz6ndtcf2mhyeydzarst@l75dman5x53p/33-v2.6-0032-Temporary-Increase-BAS_BULKREAD-size.patch)
download | inline diff:
From 1f9d6dbf3ec211a591652575be22fffcab096d46 Mon Sep 17 00:00:00 2001
From: Andres Freund <andres@anarazel.de>
Date: Sun, 1 Sep 2024 00:42:27 -0400
Subject: [PATCH v2.6 32/34] Temporary: Increase BAS_BULKREAD size
Without this we only can execute very little AIO for sequential scans, as
there's just not enough buffers in the ring. This isn't the right fix, as
just increasing the ring size can have negative performance implications in
workloads where the kernel has all the data cached.
Author:
Reviewed-By:
Discussion: https://postgr.es/m/
Backpatch:
---
src/backend/storage/buffer/freelist.c | 7 ++++++-
1 file changed, 6 insertions(+), 1 deletion(-)
diff --git a/src/backend/storage/buffer/freelist.c b/src/backend/storage/buffer/freelist.c
index 336715b6c63..b72a5957a20 100644
--- a/src/backend/storage/buffer/freelist.c
+++ b/src/backend/storage/buffer/freelist.c
@@ -555,7 +555,12 @@ GetAccessStrategy(BufferAccessStrategyType btype)
return NULL;
case BAS_BULKREAD:
- ring_size_kb = 256;
+
+ /*
+ * FIXME: Temporary increase to allow large enough streaming reads
+ * to actually benefit from AIO. This needs a better solution.
+ */
+ ring_size_kb = 2 * 1024;
break;
case BAS_BULKWRITE:
ring_size_kb = 16 * 1024;
--
2.48.1.76.g4e746b1a31.dirty
[text/x-diff] v2.6-0033-WIP-Use-MAP_POPULATE.patch (1.1K, ../../ge6nsuddurhpmll3xj22vucvqwp4agqz6ndtcf2mhyeydzarst@l75dman5x53p/34-v2.6-0033-WIP-Use-MAP_POPULATE.patch)
download | inline diff:
From 10e695cfb4c8655d939a783ed5ce28f45a460636 Mon Sep 17 00:00:00 2001
From: Andres Freund <andres@anarazel.de>
Date: Tue, 31 Dec 2024 13:25:56 -0500
Subject: [PATCH v2.6 33/34] WIP: Use MAP_POPULATE
For benchmarking it's quite annoying that the first time a memory is touched
has completely different perf characteristics than subsequent accesses. Using
MAP_POPULATE reduces that substantially.
---
src/backend/port/sysv_shmem.c | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/src/backend/port/sysv_shmem.c b/src/backend/port/sysv_shmem.c
index 197926d44f6..a700b02d5a1 100644
--- a/src/backend/port/sysv_shmem.c
+++ b/src/backend/port/sysv_shmem.c
@@ -620,7 +620,7 @@ CreateAnonymousSegment(Size *size)
allocsize += hugepagesize - (allocsize % hugepagesize);
ptr = mmap(NULL, allocsize, PROT_READ | PROT_WRITE,
- PG_MMAP_FLAGS | mmap_flags, -1, 0);
+ PG_MMAP_FLAGS | MAP_POPULATE | mmap_flags, -1, 0);
mmap_errno = errno;
if (huge_pages == HUGE_PAGES_TRY && ptr == MAP_FAILED)
elog(DEBUG1, "mmap(%zu) with MAP_HUGETLB failed, huge pages disabled: %m",
--
2.48.1.76.g4e746b1a31.dirty
^ permalink raw reply [nested|flat] 153+ messages in thread
* Re: AIO v2.5
2025-03-04 19:00 AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-10 18:23 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
@ 2025-03-11 15:31 ` Melanie Plageman <melanieplageman@gmail.com>
2025-03-11 17:56 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
1 sibling, 1 reply; 153+ messages in thread
From: Melanie Plageman @ 2025-03-11 15:31 UTC (permalink / raw)
To: Andres Freund <andres@anarazel.de>; +Cc: pgsql-hackers@postgresql.org, Thomas Munro <thomas.munro@gmail.com>; Heikki Linnakangas <hlinnaka@iki.fi>; Noah Misch <noah@leadboat.com>; Robert Haas <robertmhaas@gmail.com>; Jakub Wartak <jakub.wartak@enterprisedb.com>
On Mon, Mar 10, 2025 at 2:23 PM Andres Freund <andres@anarazel.de> wrote:
>
> - 0016 to 0020 - cleanups for temp buffers code - I just wrote these to clean
> up the code before making larger changes, needs review
This is a review of 0016-0020
Commit messages for 0017-0020 are thin. I assume you will beef them up
a bit before committing. Really, though, those matter much less than
0016 which is an actual bug (or pre-bug) fix. I called out the ones
where I think you should really consider adding more detail to the
commit message.
0016:
* the case, write it out before reusing it!
*/
- if (buf_state & BM_DIRTY)
+ if (pg_atomic_read_u32(&bufHdr->state) & BM_DIRTY)
{
+ uint32 buf_state = pg_atomic_read_u32(&bufHdr->state);
I don't love that you fetch in the if statement and inside the if
statement. You wouldn't normally do this, so it sticks out. I get that
you want to avoid having the problem this commit fixes again, but
maybe it is worth just fetching the buf_state above the if statement
and adding a comment that it could have changed so you must do that.
Anyway, I think your future patches make the local buf_state variable
in this function obsolete, so perhaps it doesn't matter.
Not related to this patch, but while reading this code, I noticed that
this line of code is really weird:
LocalBufHdrGetBlock(bufHdr) = GetLocalBufferStorage();
I actually don't understand what it is doing ... setting the result of
the macro to the result of GetLocalBufferStorage()? I haven't seen
anything like that before.
Otherwise, this patch LGTM.
0017:
+++ b/src/backend/storage/buffer/localbuf.c
@@ -56,6 +56,7 @@ static int NLocalPinnedBuffers = 0;
static Buffer GetLocalVictimBuffer(void);
+static void InvalidateLocalBuffer(BufferDesc *bufHdr, bool check_unreferenced);
Technically this line is too long
+ * InvalidateLocalBuffer -- mark a local buffer invalid.
+ *
+ * If check_unreferenced is true, error out if the buffer is still
+ * used. Passing false is appropriate when redesignating the buffer instead
+ * dropping it.
+ *
+ * See also InvalidateBuffer().
+ */
+static void
+InvalidateLocalBuffer(BufferDesc *bufHdr, bool check_unreferenced)
+{
I was on the fence about the language "buffer is still used", since
this is about the ref count and not the usage count. If this is the
language used elsewhere perhaps it is fine.
I also was not sure what redesignate means here. If you mean to use
this function in the future in other contexts than eviction and
dropping buffers, fine. But otherwise, maybe just use a more obvious
word (like eviction).
0018:
Compiler now warns that buf_state is unused in GetLocalVictimBuffer().
@@ -4564,8 +4548,7 @@ FlushRelationBuffers(Relation rel)
IOCONTEXT_NORMAL, IOOP_WRITE,
io_start, 1, BLCKSZ);
- buf_state &= ~(BM_DIRTY | BM_JUST_DIRTIED);
- pg_atomic_unlocked_write_u32(&bufHdr->state, buf_state);
+ TerminateLocalBufferIO(bufHdr, true, 0);
FlushRelationBuffers() used to clear BM_JUST_DIRTIED, which it seems
like wouldn't have been applicable to local buffers before, but,
actually with async IO could perhaps happen in the future? Anyway,
TerminateLocalBuffers() doesn't clear that flag, so you should call
that out if it was intentional.
@@ -5652,8 +5635,11 @@ TerminateBufferIO(BufferDesc *buf, bool
clear_dirty, uint32 set_flag_bits,
+ buf_state &= ~BM_IO_IN_PROGRESS;
+ buf_state &= ~BM_IO_ERROR;
- buf_state &= ~(BM_IO_IN_PROGRESS | BM_IO_ERROR);
Is it worth mentioning in the commit message that you made a cosmetic
change to TerminateBufferIO()?
0019:
LGTM
0020:
This commit message is probably tooo thin. I think you need to at
least say something about this being used by AIO in the future. Out of
context of this patch set, it will be confusing.
+/*
+ * Like StartBufferIO, but for local buffers
+ */
+bool
+StartLocalBufferIO(BufferDesc *bufHdr, bool forInput, bool nowait)
+{
I think you could use a comment about why nowait might be useful for
local buffers in the future. It wouldn't make sense with synchronous
I/O, so it feels a bit weird without any comment.
+ if (forInput ? (buf_state & BM_VALID) : !(buf_state & BM_DIRTY))
+ {
+ /* someone else already did the I/O */
+ UnlockBufHdr(bufHdr, buf_state);
+ return false;
+ }
UnlockBufHdr() explicitly says it should not be called for local
buffers. I know that code is unreachable right now, but it doesn't
feel quite right. I'm not sure what the architecture of AIO local
buffers will be like, but if other processes can't access these
buffers, I don't know why you would need BM_LOCKED. And if you will, I
think you need to edit the UnlockBufHdr() comment.
@@ -1450,13 +1450,11 @@ static inline bool
WaitReadBuffersCanStartIO(Buffer buffer, bool nowait)
{
if (BufferIsLocal(buffer))
else
- return StartBufferIO(GetBufferDescriptor(buffer - 1), true, nowait);
+ return StartBufferIO(GetBufferDescriptor(buffer - 1),
+ true, nowait);
I'm not sure it is worth the diff in non-local buffer case to reflow
this. It is already confusing enough in this patch that you are adding
some code that is mostly unneeded.
- Melanie
^ permalink raw reply [nested|flat] 153+ messages in thread
* Re: AIO v2.5
2025-03-04 19:00 AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-10 18:23 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-11 15:31 ` Re: AIO v2.5 Melanie Plageman <melanieplageman@gmail.com>
@ 2025-03-11 17:56 ` Andres Freund <andres@anarazel.de>
2025-03-11 18:28 ` Re: AIO v2.5 Melanie Plageman <melanieplageman@gmail.com>
0 siblings, 1 reply; 153+ messages in thread
From: Andres Freund @ 2025-03-11 17:56 UTC (permalink / raw)
To: Melanie Plageman <melanieplageman@gmail.com>; +Cc: pgsql-hackers@postgresql.org, Thomas Munro <thomas.munro@gmail.com>; Heikki Linnakangas <hlinnaka@iki.fi>; Noah Misch <noah@leadboat.com>; Robert Haas <robertmhaas@gmail.com>; Jakub Wartak <jakub.wartak@enterprisedb.com>
Hi,
On 2025-03-11 11:31:18 -0400, Melanie Plageman wrote:
> On Mon, Mar 10, 2025 at 2:23 PM Andres Freund <andres@anarazel.de> wrote:
> >
> > - 0016 to 0020 - cleanups for temp buffers code - I just wrote these to clean
> > up the code before making larger changes, needs review
>
> This is a review of 0016-0020
>
> Commit messages for 0017-0020 are thin. I assume you will beef them up
> a bit before committing.
Yea. I wanted to get some feedback on whether these refactorings are a good
idea or not...
> Really, though, those matter much less than 0016 which is an actual bug (or
> pre-bug) fix. I called out the ones where I think you should really consider
> adding more detail to the commit message.
>
> 0016:
Do you think we should backpatch that change? It's not really an active bug in
16+, but it's also not quite right. The other changes surely shouldn't be
backpatched...
> * the case, write it out before reusing it!
> */
> - if (buf_state & BM_DIRTY)
> + if (pg_atomic_read_u32(&bufHdr->state) & BM_DIRTY)
> {
> + uint32 buf_state = pg_atomic_read_u32(&bufHdr->state);
>
> I don't love that you fetch in the if statement and inside the if
> statement. You wouldn't normally do this, so it sticks out. I get that
> you want to avoid having the problem this commit fixes again, but
> maybe it is worth just fetching the buf_state above the if statement
> and adding a comment that it could have changed so you must do that.
It's seems way too easy to introduce new similar breakages if the scope of
buf_state is that wide - I yesterday did waste 90min because I did in another
similar place. The narrower scopes make that much less likely to be a problem.
> Anyway, I think your future patches make the local buf_state variable
> in this function obsolete, so perhaps it doesn't matter.
Leaving the defensive-programming aspect aside, it does seem like a better
intermediary state to me to have the local vars than to have to change more
lines when introducing FlushLocalBuffer() etc.
> Not related to this patch, but while reading this code, I noticed that
> this line of code is really weird:
> LocalBufHdrGetBlock(bufHdr) = GetLocalBufferStorage();
> I actually don't understand what it is doing ... setting the result of
> the macro to the result of GetLocalBufferStorage()? I haven't seen
> anything like that before.
Yes, that's what it's doing. LocalBufferBlockPointers() evaluates to a value
that can be used as an lvalue in an assignment.
Not exactly pretty...
> Otherwise, this patch LGTM.
>
> 0017:
>
> +++ b/src/backend/storage/buffer/localbuf.c
> @@ -56,6 +56,7 @@ static int NLocalPinnedBuffers = 0;
> static Buffer GetLocalVictimBuffer(void);
> +static void InvalidateLocalBuffer(BufferDesc *bufHdr, bool check_unreferenced);
>
> Technically this line is too long
Oh, do I love our line length limits. But, um, is it actually too long? It's
78 chars, which is exactly our limit, I think?
> + * InvalidateLocalBuffer -- mark a local buffer invalid.
> + *
> + * If check_unreferenced is true, error out if the buffer is still
> + * used. Passing false is appropriate when redesignating the buffer instead
> + * dropping it.
> + *
> + * See also InvalidateBuffer().
> + */
> +static void
> +InvalidateLocalBuffer(BufferDesc *bufHdr, bool check_unreferenced)
> +{
>
> I was on the fence about the language "buffer is still used", since
> this is about the ref count and not the usage count. If this is the
> language used elsewhere perhaps it is fine.
I'll change it to "still pinned"
I guess I can make it "still pinned".
> I also was not sure what redesignate means here. If you mean to use
> this function in the future in other contexts than eviction and
> dropping buffers, fine. But otherwise, maybe just use a more obvious
> word (like eviction).
I was trying to reference changing the identity of the buffer as part of
buffer replacement, where we keep a pin to the buffer. Compared to the use of
InvalidateLocalBuffer() in DropRelationAllLocalBuffers() /
DropRelationLocalBuffers().
/*
* InvalidateLocalBuffer -- mark a local buffer invalid.
*
* If check_unreferenced is true, error out if the buffer is still
* pinned. Passing false is appropriate when calling InvalidateLocalBuffer()
* as part of changing the identity of a buffer, instead of just dropping the
* buffer.
*
* See also InvalidateBuffer().
*/
> 0018:
>
> Compiler now warns that buf_state is unused in GetLocalVictimBuffer().
Oops. Missed that because it was then removed in a later commit...
> @@ -4564,8 +4548,7 @@ FlushRelationBuffers(Relation rel)
> IOCONTEXT_NORMAL, IOOP_WRITE,
> io_start, 1, BLCKSZ);
>
> - buf_state &= ~(BM_DIRTY | BM_JUST_DIRTIED);
> - pg_atomic_unlocked_write_u32(&bufHdr->state, buf_state);
> + TerminateLocalBufferIO(bufHdr, true, 0);
>
> FlushRelationBuffers() used to clear BM_JUST_DIRTIED, which it seems
> like wouldn't have been applicable to local buffers before, but,
> actually with async IO could perhaps happen in the future? Anyway,
> TerminateLocalBuffers() doesn't clear that flag, so you should call
> that out if it was intentional.
I think it'd be good to start using BM_JUST_DIRTIED, even if just to make the
code between local and shared buffers more similar. But that that's better
done separately.
I don't know why FlushRelationBuffers cleared it, it's neer set at the moment.
I'll add a note to the commit message.
> @@ -5652,8 +5635,11 @@ TerminateBufferIO(BufferDesc *buf, bool
> clear_dirty, uint32 set_flag_bits,
> + buf_state &= ~BM_IO_IN_PROGRESS;
> + buf_state &= ~BM_IO_ERROR;
> - buf_state &= ~(BM_IO_IN_PROGRESS | BM_IO_ERROR);
>
> Is it worth mentioning in the commit message that you made a cosmetic
> change to TerminateBufferIO()?
Doesn't really seem worth calling out, but if you think it should, I will.
> 0020:
> This commit message is probably tooo thin. I think you need to at
> least say something about this being used by AIO in the future. Out of
> context of this patch set, it will be confusing.
Yep.
> +/*
> + * Like StartBufferIO, but for local buffers
> + */
> +bool
> +StartLocalBufferIO(BufferDesc *bufHdr, bool forInput, bool nowait)
> +{
>
> I think you could use a comment about why nowait might be useful for
> local buffers in the future. It wouldn't make sense with synchronous
> I/O, so it feels a bit weird without any comment.
Hm, fair point. Another approach would be to defer adding the argument to a
later patch, it doesn't need to be added here.
> + if (forInput ? (buf_state & BM_VALID) : !(buf_state & BM_DIRTY))
> + {
> + /* someone else already did the I/O */
> + UnlockBufHdr(bufHdr, buf_state);
> + return false;
> + }
>
> UnlockBufHdr() explicitly says it should not be called for local
> buffers. I know that code is unreachable right now, but it doesn't
> feel quite right. I'm not sure what the architecture of AIO local
> buffers will be like, but if other processes can't access these
> buffers, I don't know why you would need BM_LOCKED. And if you will, I
> think you need to edit the UnlockBufHdr() comment.
You are right, this is a bug in my change. I started with a copy of
StartBufferIO() and whittled it down insufficiently. Thanks for catching that!
Wonder if we should add an assert against this to UnlockBufHdr()...
> @@ -1450,13 +1450,11 @@ static inline bool
> WaitReadBuffersCanStartIO(Buffer buffer, bool nowait)
> {
> if (BufferIsLocal(buffer))
> else
> - return StartBufferIO(GetBufferDescriptor(buffer - 1), true, nowait);
> + return StartBufferIO(GetBufferDescriptor(buffer - 1),
> + true, nowait);
>
> I'm not sure it is worth the diff in non-local buffer case to reflow
> this. It is already confusing enough in this patch that you are adding
> some code that is mostly unneeded.
Heh, you're right. I had to add a line break in the StartLocalBufferIO() and
it looked wrong to have the two lines formatted differently :)
Thanks for the review!
Greetings,
Andres Freund
^ permalink raw reply [nested|flat] 153+ messages in thread
* Re: AIO v2.5
2025-03-04 19:00 AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-10 18:23 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-11 15:31 ` Re: AIO v2.5 Melanie Plageman <melanieplageman@gmail.com>
2025-03-11 17:56 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
@ 2025-03-11 18:28 ` Melanie Plageman <melanieplageman@gmail.com>
0 siblings, 0 replies; 153+ messages in thread
From: Melanie Plageman @ 2025-03-11 18:28 UTC (permalink / raw)
To: Andres Freund <andres@anarazel.de>; +Cc: pgsql-hackers@postgresql.org, Thomas Munro <thomas.munro@gmail.com>; Heikki Linnakangas <hlinnaka@iki.fi>; Noah Misch <noah@leadboat.com>; Robert Haas <robertmhaas@gmail.com>; Jakub Wartak <jakub.wartak@enterprisedb.com>
On Tue, Mar 11, 2025 at 1:56 PM Andres Freund <andres@anarazel.de> wrote:
>
> Hi,
>
> On 2025-03-11 11:31:18 -0400, Melanie Plageman wrote:
> > Commit messages for 0017-0020 are thin. I assume you will beef them up
> > a bit before committing.
>
> Yea. I wanted to get some feedback on whether these refactorings are a good
> idea or not...
I'd say yes, they seem like a good idea.
> > Really, though, those matter much less than 0016 which is an actual bug (or
> > pre-bug) fix. I called out the ones where I think you should really consider
> > adding more detail to the commit message.
> >
> > 0016:
>
> Do you think we should backpatch that change? It's not really an active bug in
> 16+, but it's also not quite right. The other changes surely shouldn't be
> backpatched...
I don't feel strongly about it. PinLocalBuffer() is passed with
adjust_usagecount false and we have loads of other places where things
would just not work if we changed the boolean flag passed in to a
function called by it (bgwriter and SyncOneBuffer() with
skip_recently_used comes to mind).
On the other hand it's a straightforward fix that only needs to be
backpatched a couple versions, so it definitely doesn't hurt.
> > +++ b/src/backend/storage/buffer/localbuf.c
> > @@ -56,6 +56,7 @@ static int NLocalPinnedBuffers = 0;
> > static Buffer GetLocalVictimBuffer(void);
> > +static void InvalidateLocalBuffer(BufferDesc *bufHdr, bool check_unreferenced);
> >
> > Technically this line is too long
>
> Oh, do I love our line length limits. But, um, is it actually too long? It's
> 78 chars, which is exactly our limit, I think?
Teccchnically it's 79, which is why it showed up for me with this
handy line from the committing wiki page
git diff origin/master -- src/backend/storage/buffer/localbuf.c | grep
-E '^(\+|diff)' | sed 's/^+//' | expand -t4 | awk "length > 78 ||
/^diff/"
But anyway, it doesn't really matter. I only mentioned it because I
noticed it visually looked long.
> > + if (forInput ? (buf_state & BM_VALID) : !(buf_state & BM_DIRTY))
> > + {
> > + /* someone else already did the I/O */
> > + UnlockBufHdr(bufHdr, buf_state);
> > + return false;
> > + }
> >
> > UnlockBufHdr() explicitly says it should not be called for local
> > buffers. I know that code is unreachable right now, but it doesn't
> > feel quite right. I'm not sure what the architecture of AIO local
> > buffers will be like, but if other processes can't access these
> > buffers, I don't know why you would need BM_LOCKED. And if you will, I
> > think you need to edit the UnlockBufHdr() comment.
>
> You are right, this is a bug in my change. I started with a copy of
> StartBufferIO() and whittled it down insufficiently. Thanks for catching that!
>
> Wonder if we should add an assert against this to UnlockBufHdr()...
Yea, I think that makes sense.
- Melanie
^ permalink raw reply [nested|flat] 153+ messages in thread
* Re: AIO v2.5
2025-03-04 19:00 AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-10 18:23 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
@ 2025-03-11 19:41 ` Noah Misch <noah@leadboat.com>
2025-03-11 23:55 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
1 sibling, 1 reply; 153+ messages in thread
From: Noah Misch @ 2025-03-11 19:41 UTC (permalink / raw)
To: Andres Freund <andres@anarazel.de>; +Cc: pgsql-hackers@postgresql.org, Thomas Munro <thomas.munro@gmail.com>; Heikki Linnakangas <hlinnaka@iki.fi>; Robert Haas <robertmhaas@gmail.com>; Jakub Wartak <jakub.wartak@enterprisedb.com>
On Mon, Sep 16, 2024 at 01:51:42PM -0400, Andres Freund wrote:
> On 2024-09-16 07:43:49 -0700, Noah Misch wrote:
> > For non-sync IO methods, I gather it's essential that a process other than the
> > IO definer be scanning for incomplete IOs and completing them.
> > Otherwise, deadlocks like this would happen:
>
> > backend1 locks blk1 for non-IO reasons
> > backend2 locks blk2, starts AIO write
> > backend1 waits for lock on blk2 for non-IO reasons
> > backend2 waits for lock on blk1 for non-IO reasons
> >
> > If that's right, in worker mode, the IO worker resolves that deadlock. What
> > resolves it under io_uring? Another process that happens to do
> > pgaio_io_ref_wait() would dislodge things, but I didn't locate the code to
> > make that happen systematically.
>
> Yea, it's code that I haven't forward ported yet. I think basically
> LockBuffer[ForCleanup] ought to call pgaio_io_ref_wait() when it can't
> immediately acquire the lock and if the buffer has IO going on.
I'm not finding that code in v2.6. What function has it?
[I wrote a bunch of the subsequent comments against v2.5. I may have missed
instances of v2.6 obsoleting them.]
On Tue, Mar 04, 2025 at 02:00:14PM -0500, Andres Freund wrote:
> Attached is v2.5 of the AIO patchset.
> - Added a proper commit message fo the main commit. I'd appreciate folks
> reading through it. I'm sure I forgot a lot of folks and a lot of things.
Commit message looks fine.
> At this point I am not aware of anything significant left to do in the main
> AIO commit, safe some of the questions below.
That is a big milestone.
> Questions:
>
> - My current thinking is that we'd set io_method = worker initially - so we
> actually get some coverage - and then decide whether to switch to
> io_method=sync by default for 18 sometime around beta1/2. Does that sound
> reasonable?
Yes.
> - We could reduce memory usage a tiny bit if we made the mapping between
> pgproc and per-backend-aio-state more complicated, i.e. not just indexed by
> ProcNumber. Right now IO workers have the per-backend AIO state, but don't
> actually need it. I'm mildly inclined to think that the complexity isn't
> worth it, but on the fence.
The max memory savings, for 32 IO workers, is like the difference between
max_connections=500 and max_connections=532, right? If that's right, I
wouldn't bother in the foreseeable future.
> - Three of the commits in the series really are just precursor commits to
> their subsequent commits, which I found helpful for development and review,
> namely:
>
> - aio: Basic subsystem initialization
> - aio: Skeleton IO worker infrastructure
> - aio: Add liburing dependency
>
> Not sure if it's worth keeping these separate or whether they should just be
> merged with their "real commit".
The split aided my review. It's trivial to turn an unmerged stack of commits
into the merged equivalent, but unmerging is hard.
> - Thomas suggested renaming
> COMPLETED_IO->COMPLETED,
> COMPLETED_SHARED->TERMINATED_BY_COMPLETER,
> COMPLETED_SHARED->TERMINATED_BY_SUBMITTER
> in
> https://www.postgresql.org/message-id/CA%2BhUKGLxH1tsUgzZfng4BU6GqnS6bKF2ThvxH1_w5c7-sLRKQw%40mail.g...
>
> While the other things in the email were commented upon by others and
> addressed in v2.4, the naming aspect wasn't further remarked upon by others.
> I'm not personally in love with the suggested names, but I could live with
> them.
I, too, could live with those. None of these naming proposals bother me, and
I would not have raised the topic myself. If I were changing it further, I'd
use these principles:
- use COMPLETED or TERMINATED, not both
- I like COMPLETED, because _complete_ works well in a function name.
_terminate_ sounds more like an abnormal interruption.
- If one state name lacks a suffix, it should be the final state.
So probably one of:
{COMPLETED,TERMINATED,FINISHED,REAPED,DONE}_{KERN,RETURN,RETVAL,ERRNO}
{COMPLETED,TERMINATED,FINISHED,REAPED,DONE}_{SHMEM,SHARED}
{COMPLETED,TERMINATED,FINISHED,REAPED,DONE}{_SUBMITTER,}
If it were me picking today, I'd pick:
COMPLETED_RETURN
COMPLETED_SHMEM
COMPLETED
> - Right now this series defines PGAIO_VERBOSE to 1. That's good for debugging,
> but all the ereport()s add a noticeable amount of overhead at high IO
> throughput (at multiple gigabytes/second), so that's probably not right
> forever. I'd leave this on initially and then change it to default to off
> later. I think that's ok?
Sure. Perhaps make it depend on USE_ASSERT_CHECKING later?
> - To allow io_workers to be PGC_SIGHUP, and to eventually allow to
> automatically in/decrease active workers, the max number of workers (32) is
> always allocated. That means we use more semaphores than before. I think
> that's ok, it's not 1995 anymore. Alternatively we can add a
> "io_workers_max" GUC and probe for it in initdb.
Let's start as you have it. If someone wants to make things perfect for
non-root BSD users, they can add the GUC later. io_method=sync is a
sufficient backup plan indefinitely.
> - pg_stat_aios currently has the IO Handle flags as dedicated columns. Not
> sure that's great?
>
> They could be an enum array or such too? That'd perhaps be a bit more
> extensible? OTOH, we don't currently use enums in the catalogs and arrays
> are somewhat annoying to conjure up from C.
An enum array does seem elegant and extensible, but it has the problems you
say. (I would expect to lose time setting up pg_enum.oid values to not change
between releases.) A possible compromise would be a text array like
heap_tuple_infomask_flags() does. Overall, I'm not seeing a clear need to
change away from the bool columns.
> Todo:
> - Figure out how to deduplicate support for LockBufferForCleanup() in
> TerminateBufferIO().
Yes, I agree there's an opportunity for a WakePinCountWaiter() or similar
subroutine.
> - Check if documentation for track_io_timing needs to be adjusted, after the
> bufmgr.c changes we only track waiting for an IO.
Yes.
On Mon, Mar 10, 2025 at 02:23:12PM -0400, Andres Freund wrote:
> Attached is v2.6 of the AIO patchset.
> - 0005, 0006 - io_uring support - close, but we need to do something about
> set_max_fds(), which errors out spuriously in some cases
What do we know about those cases? I don't see a set_max_fds(); is that
set_max_safe_fds(), or something else?
> - 0025 - tests for AIO - I think it's reasonable, unless somebody objects to
> exporting a few bufmgr.c functions to the test
I'll essentially never object to that.
> + * AIO handles need be registered in critical sections and therefore
> + * cannot use the normal ResoureElem mechanism.
s/ResoureElem/ResourceElem/
> + <varlistentry id="guc-io-method" xreflabel="io_method">
> + <term><varname>io_method</varname> (<type>enum</type>)
> + <indexterm>
> + <primary><varname>io_method</varname> configuration parameter</primary>
> + </indexterm>
> + </term>
> + <listitem>
> + <para>
> + Selects the method for executing asynchronous I/O.
> + Possible values are:
> + <itemizedlist>
> + <listitem>
> + <para>
> + <literal>sync</literal> (execute asynchronous I/O synchronously)
The part in parentheses reads like a contradiction to me. How about phrasing
it like one of these:
(execute I/O synchronously, even I/O eligible for asynchronous execution)
(execute asynchronous-eligible I/O synchronously)
(execute I/O synchronously, even when asynchronous execution was feasible)
> + * This could be in aio_internal.h, as it is not pubicly referenced, but
typo -> publicly
> + * On what is IO being performed.
End sentence with question mark, probably.
> + * List of in-flight IOs. Also contains IOs that aren't strict speaking
s/strict/strictly/
> + /*
> + * Start executing passed in IOs.
> + *
> + * Will not be called if ->needs_synchronous_execution() returned true.
> + *
> + * num_staged_ios is <= PGAIO_SUBMIT_BATCH_SIZE.
> + *
I recommend adding "Always called in a critical section." since at least
pgaio_worker_submit() subtly needs it.
> + */
> + int (*submit) (uint16 num_staged_ios, PgAioHandle **staged_ios);
> + * Each backend can only have one AIO handle that that has been "handed out"
s/that that/that/
> + * AIO, it typically will pass the handle to smgr., which will pass it on to
s/smgr.,/smgr.c,/ or just "smgr"
> +PgAioHandle *
> +pgaio_io_acquire_nb(struct ResourceOwnerData *resowner, PgAioReturn *ret)
> +{
> + if (pgaio_my_backend->num_staged_ios >= PGAIO_SUBMIT_BATCH_SIZE)
> + {
> + Assert(pgaio_my_backend->num_staged_ios == PGAIO_SUBMIT_BATCH_SIZE);
> + pgaio_submit_staged();
I'm seeing the "num_staged_ios >= PGAIO_SUBMIT_BATCH_SIZE" case uncovered in a
check-world coverage report. I tried PGAIO_SUBMIT_BATCH_SIZE=2,
io_max_concurrency=1, and io_max_concurrency=64. Do you already have a recipe
for reaching this case?
> +/*
> + * Stage IO for execution and, if necessary, submit it immediately.
> + *
> + * Should only be called from pgaio_io_prep_*().
> + */
> +void
> +pgaio_io_stage(PgAioHandle *ioh, PgAioOp op)
> +{
We've got closely-associated verbs "prepare", "prep", and "stage". README.md
doesn't mention "stage". Can one of the following two changes happen?
- README.md starts mentioning "stage" and how it differs from the others
- Code stops using "stage"
> + * locallbacks just before reclaiming at multiple callsites.
s/locallbacks/local callbacks/
> + * Check if the the referenced IO completed, without blocking.
s/the the/the/
> + * Batch submission mode needs to explicitly ended with
> + * pgaio_exit_batchmode(), but it is allowed to throw errors, in which case
> + * error recovery will end the batch.
This sentence needs some grammar help, I think. Maybe use:
* End batch submission mode with pgaio_exit_batchmode(). (Throwing errors is
* allowed; error recovery will end the batch.)
> Size
> AioShmemSize(void)
> {
> Size sz = 0;
>
> + /*
> + * We prefer to report this value's source as PGC_S_DYNAMIC_DEFAULT.
> + * However, if the DBA explicitly set wal_buffers = -1 in the config file,
s/wal_buffers/io_max_concurrency/
> +extern int io_workers;
By the rule that GUC vars are PGDLLIMPORT, this should be PGDLLIMPORT.
> +static void
> +maybe_adjust_io_workers(void)
This also restarts workers that exit, so perhaps name it
start_io_workers_if_missing().
> +{
...
> + /* Try to launch one. */
> + child = StartChildProcess(B_IO_WORKER);
> + if (child != NULL)
> + {
> + io_worker_children[id] = child;
> + ++io_worker_count;
> + }
> + else
> + break; /* XXX try again soon? */
Can LaunchMissingBackgroundProcesses() become the sole caller of this
function, replacing the current mix of callers? That would be more conducive
to promptly doing the right thing after launch failure.
> --- a/src/backend/utils/init/miscinit.c
> +++ b/src/backend/utils/init/miscinit.c
> @@ -293,6 +293,9 @@ GetBackendTypeDesc(BackendType backendType)
> case B_CHECKPOINTER:
> backendDesc = gettext_noop("checkpointer");
> break;
> + case B_IO_WORKER:
> + backendDesc = "io worker";
Wrap in gettext_noop() like B_CHECKPOINTER does.
> + Only has an effect if <xref linkend="guc-max-wal-senders"/> is set to
> + <literal>worker</literal>.
s/guc-max-wal-senders/guc-io-method/
> + * of IOs, wakeups "fan out"; each woken IO worker can wake two more. qXXX
s/qXXX/XXX/
> + /*
> + * It's very unlikely, but possible, that reopen fails. E.g. due
> + * to memory allocations failing or file permissions changing or
> + * such. In that case we need to fail the IO.
> + *
> + * There's not really a good errno we can report here.
> + */
> + error_errno = ENOENT;
Agreed there's not a good errno, but let's use a fake errno that we're mighty
unlikely to confuse with an actual case of libc returning that errno. Like
one of EBADF or EOWNERDEAD.
> + for (int contextno = 0; contextno < TotalProcs; contextno++)
> + {
> + PgAioUringContext *context = &pgaio_uring_contexts[contextno];
> + int ret;
> +
> + /*
> + * XXX: Probably worth sharing the WQ between the different rings,
> + * when supported by the kernel. Could also cause additional
> + * contention, I guess?
> + */
> +#if 0
> + if (!AcquireExternalFD())
> + elog(ERROR, "No external FD available");
> +#endif
Probably remove the "#if 0" or add a comment on why it's here.
> + ret = io_uring_submit(uring_instance);
> + pgstat_report_wait_end();
> +
> + if (ret == -EINTR)
> + {
> + pgaio_debug(DEBUG3,
> + "aio method uring: submit EINTR, nios: %d",
> + num_staged_ios);
> + }
> + else if (ret < 0)
> + elog(PANIC, "failed: %d/%s",
> + ret, strerror(-ret));
I still think (see 2024-09-16 review) EAGAIN should do the documented
recommendation instead of PANIC:
EAGAIN The kernel was unable to allocate memory for the request, or
otherwise ran out of resources to handle it. The application should wait for
some completions and try again.
At a minimum, it deserves a comment like "We accept PANIC on memory exhaustion
here."
> + pgstat_report_wait_end();
> +
> + if (ret == -EINTR)
> + {
> + continue;
> + }
> + else if (ret != 0)
> + {
> + elog(PANIC, "unexpected: %d/%s: %m", ret, strerror(-ret));
I think errno isn't meaningful here, so %m doesn't belong.
> --- a/doc/src/sgml/config.sgml
> +++ b/doc/src/sgml/config.sgml
> @@ -2687,6 +2687,12 @@ include_dir 'conf.d'
> <literal>worker</literal> (execute asynchronous I/O using worker processes)
> </para>
> </listitem>
> + <listitem>
> + <para>
> + <literal>io_uring</literal> (execute asynchronous I/O using
> + io_uring, if available)
> + </para>
> + </listitem>
Docs should eventually cover RLIMIT_MEMLOCK per
https://github.com/axboe/liburing "ulimit settings". Maybe RLIMIT_NOFILE,
too.
> @@ -2498,6 +2529,12 @@ FilePathName(File file)
> int
> FileGetRawDesc(File file)
> {
> + int returnCode;
> +
> + returnCode = FileAccess(file);
> + if (returnCode < 0)
> + return returnCode;
> +
> Assert(FileIsValid(file));
> return VfdCache[file].fd;
> }
What's the rationale for this function's change?
> +The main reason to want to use Direct IO are:
> +The main reason *not* to use Direct IO are:
x2 s/main reason/main reasons/
> + and direct IO without O_DSYNC needs to issue a write and after the writes
> + completion a cache cache flush, whereas O\_DIRECT + O\_DSYNC can use a
s/writes/write's/
> + single FUA write).
I recommend including the acronym expansion: s/FUA/Force Unit Access (FUA)/
> +In an `EXEC_BACKEND` build backends executable code and other process local
s/backends/backends'/
> +state is not necessarily mapped to the same addresses in each process due to
> +ASLR. This means that the shared memory cannot contain pointer to callbacks.
s/pointer/pointers/
> +The "solution" to this the ability to associate multiple completion callbacks
> +with a handle. E.g. bufmgr.c can have a callback to update the BufferDesc
> +state and to verify the page and md.c. another callback to check if the IO
> +operation was successful.
One of these or similar:
s/md.c. another/md.c can have another/
s/md.c. /md.c /
I've got one high-level question that I felt could take too long to answer for
myself by code reading. What's the cleanup story if process A does
elog(FATAL) with unfinished I/O? Specifically:
- Suppose some other process B reuses the shared memory AIO data structures
that pertained to process A. After that, some process C completes the I/O
in shmem. Do we avoid confusing B by storing local callback data meant for
A in shared memory now pertaining to B?
- Thinking more about this README paragraph:
+In addition to completion, AIO callbacks also are called to "prepare" an
+IO. This is, e.g., used to increase buffer reference counts to account for the
+AIO subsystem referencing the buffer, which is required to handle the case
+where the issuing backend errors out and releases its own pins while the IO is
+still ongoing.
Which function performs that reference count increase? I'm not finding it
today. I wanted to look at how it ensures the issuing backend still exists
as the function increases the reference count.
One later-patch item:
> +static PgAioResult
> +SharedBufferCompleteRead(int buf_off, Buffer buffer, uint8 flags, bool failed)
> +{
...
> + TRACE_POSTGRESQL_BUFFER_READ_DONE(tag.forkNum,
> + tag.blockNum,
> + tag.spcOid,
> + tag.dbOid,
> + tag.relNumber,
> + INVALID_PROC_NUMBER,
> + false);
I wondered about whether the buffer-read-done probe should happen in the
process that calls the complete_shared callback or in the process that did the
buffer-read-start probe. When I see dtrace examples, they usually involve
explicitly naming each PID to trace. Assuming that's indeed the norm, I think
the local callback would be the better place, so a given trace contains both
probes. If it were reasonable to dtrace all current and future postmaster
kids, that would argue for putting the probe in the complete_shared callback.
Alternatively, would could argue for separate probes buffer-read-done-shmem
and buffer-read-done.
Thanks,
nm
^ permalink raw reply [nested|flat] 153+ messages in thread
* Re: AIO v2.5
2025-03-04 19:00 AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-10 18:23 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-11 19:41 ` Re: AIO v2.5 Noah Misch <noah@leadboat.com>
@ 2025-03-11 23:55 ` Andres Freund <andres@anarazel.de>
2025-03-12 03:57 ` Re: AIO v2.5 Noah Misch <noah@leadboat.com>
2025-03-12 16:10 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
0 siblings, 2 replies; 153+ messages in thread
From: Andres Freund @ 2025-03-11 23:55 UTC (permalink / raw)
To: Noah Misch <noah@leadboat.com>; +Cc: pgsql-hackers@postgresql.org, Thomas Munro <thomas.munro@gmail.com>; Heikki Linnakangas <hlinnaka@iki.fi>; Robert Haas <robertmhaas@gmail.com>; Jakub Wartak <jakub.wartak@enterprisedb.com>
Hi,
On 2025-03-11 12:41:08 -0700, Noah Misch wrote:
> On Mon, Sep 16, 2024 at 01:51:42PM -0400, Andres Freund wrote:
> > On 2024-09-16 07:43:49 -0700, Noah Misch wrote:
> > > For non-sync IO methods, I gather it's essential that a process other than the
> > > IO definer be scanning for incomplete IOs and completing them.
>
> > > Otherwise, deadlocks like this would happen:
> >
> > > backend1 locks blk1 for non-IO reasons
> > > backend2 locks blk2, starts AIO write
> > > backend1 waits for lock on blk2 for non-IO reasons
> > > backend2 waits for lock on blk1 for non-IO reasons
> > >
> > > If that's right, in worker mode, the IO worker resolves that deadlock. What
> > > resolves it under io_uring? Another process that happens to do
> > > pgaio_io_ref_wait() would dislodge things, but I didn't locate the code to
> > > make that happen systematically.
> >
> > Yea, it's code that I haven't forward ported yet. I think basically
> > LockBuffer[ForCleanup] ought to call pgaio_io_ref_wait() when it can't
> > immediately acquire the lock and if the buffer has IO going on.
>
> I'm not finding that code in v2.6. What function has it?
My local version now has it... Sorry, I was focusing on the earlier patches
until now.
What do we want to do for ConditionalLockBufferForCleanup() (I don't think
IsBufferCleanupOK() can matter)? I suspect we should also make it wait for
the IO. See below:
Not for 18, but for full write support, we'll also need logic to wait for IO
in LockBuffer(BUFFER_LOCK_EXCLUSIVE) and answer the same question as for
ConditionalLockBufferForCleanup() for ConditionalLockBuffer().
It's not an issue with the current level of write support in the stack of
patches. But with v1 AIO, which had support for a lot more ways of doing
asynchronous writes, it turned out that not handling it in
ConditionalLockBuffer() triggers an endless loop. This can be
kind-of-reproduced today by just making ConditionalLockBuffer() always return
false - triggers a hang in the regression tests:
spginsert() loops around spgdoinsert() until it succeeds. spgdoinsert() locks
the child page with ConditionalLockBuffer() and gives up if it can't.
That seems like rather bad code in spgist, because, even without AIO, it'll
busy-loop until the buffer is unlocked. Which could take a while, given that
it'll conflict even with a share locker and thus synchronous writes.
Even if we fixed spgist, it seems rather likely that there's other code that
wouldn't tolerate "spurious" failures. Which leads me to think that causing
the IO to complete is probably the safest bet. Triggering IO completion never
requires acquiring new locks that could participate in a deadlock, so it'd be
safe.
> > At this point I am not aware of anything significant left to do in the main
> > AIO commit, safe some of the questions below.
>
> That is a big milestone.
Indeed!
> > - We could reduce memory usage a tiny bit if we made the mapping between
> > pgproc and per-backend-aio-state more complicated, i.e. not just indexed by
> > ProcNumber. Right now IO workers have the per-backend AIO state, but don't
> > actually need it. I'm mildly inclined to think that the complexity isn't
> > worth it, but on the fence.
>
> The max memory savings, for 32 IO workers, is like the difference between
> max_connections=500 and max_connections=532, right?
Even less than that: Aux processes aren't always used as a multiplier in
places where max_connections etc are. E.g. max_locks_per_transaction is just
multiplied by MaxBackends, not MaxBackends+NUM_AUXILIARY_PROCS.
> If that's right, I wouldn't bother in the foreseeable future.
Cool.
> > - Three of the commits in the series really are just precursor commits to
> > their subsequent commits, which I found helpful for development and review,
> > namely:
> >
> > - aio: Basic subsystem initialization
> > - aio: Skeleton IO worker infrastructure
> > - aio: Add liburing dependency
> >
> > Not sure if it's worth keeping these separate or whether they should just be
> > merged with their "real commit".
>
> The split aided my review. It's trivial to turn an unmerged stack of commits
> into the merged equivalent, but unmerging is hard.
That's been the feedback so far, so I'll leave it split.
> > - Right now this series defines PGAIO_VERBOSE to 1. That's good for debugging,
> > but all the ereport()s add a noticeable amount of overhead at high IO
> > throughput (at multiple gigabytes/second), so that's probably not right
> > forever. I'd leave this on initially and then change it to default to off
> > later. I think that's ok?
>
> Sure. Perhaps make it depend on USE_ASSERT_CHECKING later?
Yea, that makes sense.
> > - To allow io_workers to be PGC_SIGHUP, and to eventually allow to
> > automatically in/decrease active workers, the max number of workers (32) is
> > always allocated. That means we use more semaphores than before. I think
> > that's ok, it's not 1995 anymore. Alternatively we can add a
> > "io_workers_max" GUC and probe for it in initdb.
>
> Let's start as you have it. If someone wants to make things perfect for
> non-root BSD users, they can add the GUC later. io_method=sync is a
> sufficient backup plan indefinitely.
Cool.
I think we'll really need to do something about this for BSD users regardless
of AIO. Or maybe those OSs should fix something, but somehow I am not having
high hopes for an OS that claims to have POSIX confirming unnamed semaphores
due to having a syscall that always returns EPERM... [1].
> > - pg_stat_aios currently has the IO Handle flags as dedicated columns. Not
> > sure that's great?
> >
> > They could be an enum array or such too? That'd perhaps be a bit more
> > extensible? OTOH, we don't currently use enums in the catalogs and arrays
> > are somewhat annoying to conjure up from C.
>
> An enum array does seem elegant and extensible, but it has the problems you
> say. (I would expect to lose time setting up pg_enum.oid values to not change
> between releases.) A possible compromise would be a text array like
> heap_tuple_infomask_flags() does. Overall, I'm not seeing a clear need to
> change away from the bool columns.
Yea, I think that's where I ended up too. If we get a dozen flags we can
reconsider.
> > Todo:
>
> > - Figure out how to deduplicate support for LockBufferForCleanup() in
> > TerminateBufferIO().
>
> Yes, I agree there's an opportunity for a WakePinCountWaiter() or similar
> subroutine.
Done.
> > - Check if documentation for track_io_timing needs to be adjusted, after the
> > bufmgr.c changes we only track waiting for an IO.
>
> Yes.
The relevant sentences seem to be:
- "Enables timing of database I/O calls."
s/calls/waits/
- "Time spent in {read,write,writeback,extend,fsync} operations"
s/in/waiting for/
Even though not all of these will use AIO, the "waiting for" formulation
seems just as accurate.
- "Columns tracking I/O time will only be non-zero when <xref
linkend="guc-track-io-timing"/> is enabled."
s/time/wait time/
> On Mon, Mar 10, 2025 at 02:23:12PM -0400, Andres Freund wrote:
> > Attached is v2.6 of the AIO patchset.
>
> > - 0005, 0006 - io_uring support - close, but we need to do something about
> > set_max_fds(), which errors out spuriously in some cases
>
> What do we know about those cases? I don't see a set_max_fds(); is that
> set_max_safe_fds(), or something else?
Sorry, yes, set_max_safe_fds(). The problem basically is that with io_uring we
will have a large number of FDs already allocated by the time
set_max_safe_fds() is called. set_max_safe_fds() subtracts already_open from
max_files_per_process allowing few, and even negative, IOs.
I think we should redefine max_files_per_process to be about the number of
files each *backend* will additionally open. Jelte was working on related
patches, see [2]
> > + * AIO handles need be registered in critical sections and therefore
> > + * cannot use the normal ResoureElem mechanism.
>
> s/ResoureElem/ResourceElem/
Oops, fixed.
> > + <varlistentry id="guc-io-method" xreflabel="io_method">
> > + <term><varname>io_method</varname> (<type>enum</type>)
> > + <indexterm>
> > + <primary><varname>io_method</varname> configuration parameter</primary>
> > + </indexterm>
> > + </term>
> > + <listitem>
> > + <para>
> > + Selects the method for executing asynchronous I/O.
> > + Possible values are:
> > + <itemizedlist>
> > + <listitem>
> > + <para>
> > + <literal>sync</literal> (execute asynchronous I/O synchronously)
>
> The part in parentheses reads like a contradiction to me.
There's something to that...
> How about phrasing it like one of these:
>
> (execute I/O synchronously, even I/O eligible for asynchronous execution)
> (execute asynchronous-eligible I/O synchronously)
> (execute I/O synchronously, even when asynchronous execution was feasible)
I like the second one best, adopted.
> [..]
> End sentence with question mark, probably.
> [..]
> s/strict/strictly/
> [..]
> I recommend adding "Always called in a critical section." since at least
> pgaio_worker_submit() subtly needs it.
> [..]
> s/that that/that/
> [..]
> s/smgr.,/smgr.c,/ or just "smgr"
> [..]
> s/locallbacks/local callbacks/
> [..]
> s/the the/the/
All adopted.
> > +PgAioHandle *
> > +pgaio_io_acquire_nb(struct ResourceOwnerData *resowner, PgAioReturn *ret)
> > +{
> > + if (pgaio_my_backend->num_staged_ios >= PGAIO_SUBMIT_BATCH_SIZE)
> > + {
> > + Assert(pgaio_my_backend->num_staged_ios == PGAIO_SUBMIT_BATCH_SIZE);
> > + pgaio_submit_staged();
>
> I'm seeing the "num_staged_ios >= PGAIO_SUBMIT_BATCH_SIZE" case uncovered in a
> check-world coverage report. I tried PGAIO_SUBMIT_BATCH_SIZE=2,
> io_max_concurrency=1, and io_max_concurrency=64. Do you already have a recipe
> for reaching this case?
With the default server settings it's hard to hit due to read_stream.c
limiting how much IO it issues:
1) The default io_combine_limit=16 makes reads larger, reducing the queue
depth, at least for sequential scans
2) The default shared_buffers/max_connections settings limit the number of
buffers that can be pinned to 86, which will only allow a small number of
IOs due to 86/io_combine_limit = ~5
3) The default effective_io_concurrency only allows one IO in flight
Melanie has a patch to adjust effective_io_concurrency:
https://www.postgresql.org/message-id/CAAKRu_Z4ekRbfTacYYVrvu9xRqS6G4DMbZSbN_1usaVtj%2Bbv2w%40mail.g...
If I increase shared_buffers and decrease io_combine_limit and put an
elog(PANIC) in that branch, it's rather quickly hit.
> > +/*
> > + * Stage IO for execution and, if necessary, submit it immediately.
> > + *
> > + * Should only be called from pgaio_io_prep_*().
> > + */
> > +void
> > +pgaio_io_stage(PgAioHandle *ioh, PgAioOp op)
> > +{
>
> We've got closely-associated verbs "prepare", "prep", and "stage". README.md
> doesn't mention "stage". Can one of the following two changes happen?
>
> - README.md starts mentioning "stage" and how it differs from the others
> - Code stops using "stage"
I'll try to add something to README.md. To me the sequence is prepare->stage.
> > + * Batch submission mode needs to explicitly ended with
> > + * pgaio_exit_batchmode(), but it is allowed to throw errors, in which case
> > + * error recovery will end the batch.
>
> This sentence needs some grammar help, I think.
Indeed.
> Maybe use:
>
> * End batch submission mode with pgaio_exit_batchmode(). (Throwing errors is
> * allowed; error recovery will end the batch.)
I like it.
> > Size
> > AioShmemSize(void)
> > {
> > Size sz = 0;
> >
> > + /*
> > + * We prefer to report this value's source as PGC_S_DYNAMIC_DEFAULT.
> > + * However, if the DBA explicitly set wal_buffers = -1 in the config file,
>
> s/wal_buffers/io_max_concurrency/
Ooops.
> > +extern int io_workers;
>
> By the rule that GUC vars are PGDLLIMPORT, this should be PGDLLIMPORT.
Indeed. I wish we had something finding violations of this automatically...
> > +static void
> > +maybe_adjust_io_workers(void)
>
> This also restarts workers that exit, so perhaps name it
> start_io_workers_if_missing().
But it also stops IO workers if necessary?
> > +{
> ...
> > + /* Try to launch one. */
> > + child = StartChildProcess(B_IO_WORKER);
> > + if (child != NULL)
> > + {
> > + io_worker_children[id] = child;
> > + ++io_worker_count;
> > + }
> > + else
> > + break; /* XXX try again soon? */
>
> Can LaunchMissingBackgroundProcesses() become the sole caller of this
> function, replacing the current mix of callers? That would be more conducive
> to promptly doing the right thing after launch failure.
I'm not sure that'd be a good idea - right now IO workers are started before
the startup process, as the startup process might need to perform IO. If we
started it only later in ServerLoop() we'd potentially do a fair bit of work,
including starting checkpointer, bgwriter, bgworkers before we started IO
workers. That shouldn't actively break anything, but it would likely make
things slower.
I rather dislike the code around when we start what. Leaving AIO aside, during
a normal startup we start checkpointer, bgwriter before the startup
process. But during a crash restart we don't explicitly start them. Why make
things uniform when it coul also be exciting :)
> > --- a/src/backend/utils/init/miscinit.c
> > +++ b/src/backend/utils/init/miscinit.c
> > @@ -293,6 +293,9 @@ GetBackendTypeDesc(BackendType backendType)
> > case B_CHECKPOINTER:
> > backendDesc = gettext_noop("checkpointer");
> > break;
> > + case B_IO_WORKER:
> > + backendDesc = "io worker";
>
> Wrap in gettext_noop() like B_CHECKPOINTER does.
>
> > + Only has an effect if <xref linkend="guc-max-wal-senders"/> is set to
> > + <literal>worker</literal>.
>
> s/guc-max-wal-senders/guc-io-method/
>
> > + * of IOs, wakeups "fan out"; each woken IO worker can wake two more. qXXX
>
> s/qXXX/XXX/
All fixed.
> > + /*
> > + * It's very unlikely, but possible, that reopen fails. E.g. due
> > + * to memory allocations failing or file permissions changing or
> > + * such. In that case we need to fail the IO.
> > + *
> > + * There's not really a good errno we can report here.
> > + */
> > + error_errno = ENOENT;
>
> Agreed there's not a good errno, but let's use a fake errno that we're mighty
> unlikely to confuse with an actual case of libc returning that errno. Like
> one of EBADF or EOWNERDEAD.
Can we rely on that to be present on all platforms, including windows?
> > + for (int contextno = 0; contextno < TotalProcs; contextno++)
> > + {
> > + PgAioUringContext *context = &pgaio_uring_contexts[contextno];
> > + int ret;
> > +
> > + /*
> > + * XXX: Probably worth sharing the WQ between the different rings,
> > + * when supported by the kernel. Could also cause additional
> > + * contention, I guess?
> > + */
> > +#if 0
> > + if (!AcquireExternalFD())
> > + elog(ERROR, "No external FD available");
> > +#endif
>
> Probably remove the "#if 0" or add a comment on why it's here.
Will do. It was an attempt at dealing with the set_max_safe_fds() issue above,
but it turned out to not work at all, given how fd.c currently works.
> > + ret = io_uring_submit(uring_instance);
> > + pgstat_report_wait_end();
> > +
> > + if (ret == -EINTR)
> > + {
> > + pgaio_debug(DEBUG3,
> > + "aio method uring: submit EINTR, nios: %d",
> > + num_staged_ios);
> > + }
> > + else if (ret < 0)
> > + elog(PANIC, "failed: %d/%s",
> > + ret, strerror(-ret));
>
> I still think (see 2024-09-16 review) EAGAIN should do the documented
> recommendation instead of PANIC:
>
> EAGAIN The kernel was unable to allocate memory for the request, or
> otherwise ran out of resources to handle it. The application should wait for
> some completions and try again.
I don't think this can be hit in a recoverable way. We'd likely just end up
with an untested path that quite possibly would be wrong.
What wait time would be appropriate? What problems would it cause if we just
slept while holding critical lwlocks? I think it'd typically just delay the
crash-restart if we did, making it harder to recover from the problem.
Because we are careful to limit how many outstanding IO requests there are on
an io_uring instance, the kernel has to have run *severely* out of memory to
hit this.
I suspect it might currently be *impossible* to hit this due to ENOMEM,
because io_uring will fall back to allocating individual request, if the batch
allocation it normally does, fails. My understanding is that for small
allocations the kernel will try to reclaim memory forever, only large ones can
fail.
Even if it were possible to hit, the likelihood that postgres can continue to
work ok if the kernel can't allocate ~250 bytes seems very low.
How about adding a dedicated error message for EAGAIN? IMO io_uring_enter()'s
meaning of EAGAIN is, uhm, unconvential, so a better error message than
strerror() might be good?
Proposed comment:
/*
* The io_uring_enter() manpage suggests that the appropriate
* reaction to EAGAIN is:
*
* "The application should wait for some completions and try
* again"
*
* However, it seems unlikely that that would help in our case, as
* we apply a low limit to the number of outstanding IOs and thus
* also outstanding completions, making it unlikely that we'd get
* EAGAIN while the OS is in good working order.
*
* Additionally, it would be problematic to just wait here, our
* caller might hold critical locks. It'd possibly lead to
* delaying the crash-restart that seems likely to occur when the
* kernel is under such heavy memory pressure.
*/
> > + pgstat_report_wait_end();
> > +
> > + if (ret == -EINTR)
> > + {
> > + continue;
> > + }
> > + else if (ret != 0)
> > + {
> > + elog(PANIC, "unexpected: %d/%s: %m", ret, strerror(-ret));
>
> I think errno isn't meaningful here, so %m doesn't belong.
You're right. I wonder if we should make errno meaningful though (by setting
it), the elog.c machinery captures it and I know that there are logging hooks
that utilize that fact. That'd also avoid the need to use strerror() here.
> > --- a/doc/src/sgml/config.sgml
> > +++ b/doc/src/sgml/config.sgml
> > @@ -2687,6 +2687,12 @@ include_dir 'conf.d'
> > <literal>worker</literal> (execute asynchronous I/O using worker processes)
> > </para>
> > </listitem>
> > + <listitem>
> > + <para>
> > + <literal>io_uring</literal> (execute asynchronous I/O using
> > + io_uring, if available)
> > + </para>
> > + </listitem>
>
> Docs should eventually cover RLIMIT_MEMLOCK per
> https://github.com/axboe/liburing "ulimit settings".
The way we currently use io_uring (i.e. no registered buffers), the
RLIMIT_MEMLOCK advice only applies to linux <= 5.11. I'm not sure that's
worth documenting?
> Maybe RLIMIT_NOFILE, too.
Yea, we probably need to. Depends a bit on where we go with [2] though.
>
> > @@ -2498,6 +2529,12 @@ FilePathName(File file)
> > int
> > FileGetRawDesc(File file)
> > {
> > + int returnCode;
> > +
> > + returnCode = FileAccess(file);
> > + if (returnCode < 0)
> > + return returnCode;
> > +
> > Assert(FileIsValid(file));
> > return VfdCache[file].fd;
> > }
>
> What's the rationale for this function's change?
It flatly didn't work before. I guess I can make that a separate commit.
> > +The main reason to want to use Direct IO are:
>
> > +The main reason *not* to use Direct IO are:
>
> x2 s/main reason/main reasons/
>
> > + and direct IO without O_DSYNC needs to issue a write and after the writes
> > + completion a cache cache flush, whereas O\_DIRECT + O\_DSYNC can use a
>
> s/writes/write's/
>
> > + single FUA write).
>
> I recommend including the acronym expansion: s/FUA/Force Unit Access (FUA)/
>
> > +In an `EXEC_BACKEND` build backends executable code and other process local
>
> s/backends/backends'/
>
> > +state is not necessarily mapped to the same addresses in each process due to
> > +ASLR. This means that the shared memory cannot contain pointer to callbacks.
>
> s/pointer/pointers/
>
> > +The "solution" to this the ability to associate multiple completion callbacks
> > +with a handle. E.g. bufmgr.c can have a callback to update the BufferDesc
> > +state and to verify the page and md.c. another callback to check if the IO
> > +operation was successful.
>
> One of these or similar:
> s/md.c. another/md.c can have another/
> s/md.c. /md.c /
All applied.
> I've got one high-level question that I felt could take too long to answer for
> myself by code reading. What's the cleanup story if process A does
> elog(FATAL) with unfinished I/O? Specifically:
It's a good question. Luckily there's a relatively easy answer:
pgaio_shutdown() is registered via before_shmem_exit() in pgaio_init_backend()
and pgaio_shutdown() waits for all IOs to finish.
The main reason this exists is that the AIO mechanism in various OSs, at least
in some OS versions, don't like it if the issuing process exits while the IO
is in flight. IIRC that was the case with in v1 with posix_aio (which we
don't support in v2, and probably should never use) and I think also with
io_uring in some kernel versions.
Another reason is that those requests would show up in pg_aios (or whatever we
end up naming it) until they're reused, which doesn't seem great.
> - Suppose some other process B reuses the shared memory AIO data structures
> that pertained to process A. After that, some process C completes the I/O
> in shmem. Do we avoid confusing B by storing local callback data meant for
> A in shared memory now pertaining to B?
This will, before pgaio_shutdown() gets involved, also be prevented by local
callbacks being cleared by resowner cleanup. We take care that that resowner
cleanup happens before process exit. That's important, because the backend
local pointer could be invalidated by an ERROR
> - Thinking more about this README paragraph:
>
> +In addition to completion, AIO callbacks also are called to "prepare" an
> +IO. This is, e.g., used to increase buffer reference counts to account for the
> +AIO subsystem referencing the buffer, which is required to handle the case
> +where the issuing backend errors out and releases its own pins while the IO is
> +still ongoing.
>
> Which function performs that reference count increase? I'm not finding it
> today.
Ugh, I just renamed the relevant functions in my local branch, while trying to
reduce the code duplication between shared and local buffers ;).
In <= v2.6 it's shared_buffer_stage_common() and local_buffer_readv_stage().
In v2.7-to-be it is buffer_stage_common(), which now supports both shared and
local buffers.
> I wanted to look at how it ensures the issuing backend still exists as the
> function increases the reference count.
The reference count is increased solely in the BufferDesc, *not* in the
backend-local pin tracking. Earlier I had tracked the pin in BufferDesc for
shared buffers (as the pin needs to be released upon completion, which might
be in another backend), but in LocalRefCount[] for temp buffers. But that
turned out to not work when the backend errors out, as it would make
CheckForLocalBufferLeaks() complain.
>
> One later-patch item:
>
> > +static PgAioResult
> > +SharedBufferCompleteRead(int buf_off, Buffer buffer, uint8 flags, bool failed)
> > +{
> ...
> > + TRACE_POSTGRESQL_BUFFER_READ_DONE(tag.forkNum,
> > + tag.blockNum,
> > + tag.spcOid,
> > + tag.dbOid,
> > + tag.relNumber,
> > + INVALID_PROC_NUMBER,
> > + false);
>
> I wondered about whether the buffer-read-done probe should happen in the
> process that calls the complete_shared callback or in the process that did the
> buffer-read-start probe.
Yea, that's a good point. I should at least have added a comment pointing out
that it's a choice with pros and cons.
The reason I went for doing it in the completion callback is that it seemed
better to get the READ_DONE event as soon as possible, even if the issuer of
the IO is currently busy doing other things. The shared completion callback is
after all where the buffer state is updated for shared buffers.
But I think you have a point too.
> When I see dtrace examples, they usually involve explicitly naming each PID
> to trace
TBH, i've only ever used our tracepoints via perf and bpftrace, not dtrace
itself. For those it's easy to trace more than just a single pid and to
monitor system-wide. I don't really know enough about using dtrace itself.
> Assuming that's indeed the norm, I think the local callback would
> be the better place, so a given trace contains both probes.
Seems like a shame to add an extra indirect function call for a tracing
feature that afaict approximately nobody ever uses (IIRC we several times have
passed wrong things to tracepoints without that being noticed).
TBH, the tracepoints are so poorly documented and maintained that I was
tempted to suggest removing them a couple times.
This was an awesome review, thanks!
Andres Freund
[1] https://man.openbsd.org/sem_init.3#STANDARDS
[2] https://postgr.es/m/D80MHNSG4EET.6MSV5G9P130F%40jeltef.nl
^ permalink raw reply [nested|flat] 153+ messages in thread
* Re: AIO v2.5
2025-03-04 19:00 AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-10 18:23 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-11 19:41 ` Re: AIO v2.5 Noah Misch <noah@leadboat.com>
2025-03-11 23:55 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
@ 2025-03-12 03:57 ` Noah Misch <noah@leadboat.com>
2025-03-12 17:06 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
1 sibling, 1 reply; 153+ messages in thread
From: Noah Misch @ 2025-03-12 03:57 UTC (permalink / raw)
To: Andres Freund <andres@anarazel.de>; +Cc: pgsql-hackers@postgresql.org, Thomas Munro <thomas.munro@gmail.com>; Heikki Linnakangas <hlinnaka@iki.fi>; Robert Haas <robertmhaas@gmail.com>; Jakub Wartak <jakub.wartak@enterprisedb.com>
On Tue, Mar 11, 2025 at 07:55:35PM -0400, Andres Freund wrote:
> On 2025-03-11 12:41:08 -0700, Noah Misch wrote:
> > On Mon, Sep 16, 2024 at 01:51:42PM -0400, Andres Freund wrote:
> > > On 2024-09-16 07:43:49 -0700, Noah Misch wrote:
> What do we want to do for ConditionalLockBufferForCleanup() (I don't think
> IsBufferCleanupOK() can matter)? I suspect we should also make it wait for
> the IO. See below:
I agree IsBufferCleanupOK() can't matter. It asserts that the caller already
holds the exclusive buffer lock, and it doesn't loop or wait.
> [...] leads me to think that causing
> the IO to complete is probably the safest bet. Triggering IO completion never
> requires acquiring new locks that could participate in a deadlock, so it'd be
> safe.
Yes, that decision looks right to me. I scanned the callers, and none of them
clearly prefers a different choice. If we someday find one caller prefers a
false return over blocking on I/O completion, we can always introduce a new
ConditionalLock* variant for that.
> > > - To allow io_workers to be PGC_SIGHUP, and to eventually allow to
> > > automatically in/decrease active workers, the max number of workers (32) is
> > > always allocated. That means we use more semaphores than before. I think
> > > that's ok, it's not 1995 anymore. Alternatively we can add a
> > > "io_workers_max" GUC and probe for it in initdb.
> >
> > Let's start as you have it. If someone wants to make things perfect for
> > non-root BSD users, they can add the GUC later. io_method=sync is a
> > sufficient backup plan indefinitely.
>
> Cool.
>
> I think we'll really need to do something about this for BSD users regardless
> of AIO. Or maybe those OSs should fix something, but somehow I am not having
> high hopes for an OS that claims to have POSIX confirming unnamed semaphores
> due to having a syscall that always returns EPERM... [1].
I won't mind a project making things better for non-root BSD users. I do
think such a project should not block other projects making things better for
everything else (like $SUBJECT).
> > > - Check if documentation for track_io_timing needs to be adjusted, after the
> > > bufmgr.c changes we only track waiting for an IO.
> >
> > Yes.
>
> The relevant sentences seem to be:
>
> - "Enables timing of database I/O calls."
>
> s/calls/waits/
>
> - "Time spent in {read,write,writeback,extend,fsync} operations"
>
> s/in/waiting for/
>
> Even though not all of these will use AIO, the "waiting for" formulation
> seems just as accurate.
>
> - "Columns tracking I/O time will only be non-zero when <xref
> linkend="guc-track-io-timing"/> is enabled."
>
> s/time/wait time/
Sounds good.
> > On Mon, Mar 10, 2025 at 02:23:12PM -0400, Andres Freund wrote:
> > > Attached is v2.6 of the AIO patchset.
> >
> > > - 0005, 0006 - io_uring support - close, but we need to do something about
> > > set_max_fds(), which errors out spuriously in some cases
> >
> > What do we know about those cases? I don't see a set_max_fds(); is that
> > set_max_safe_fds(), or something else?
>
> Sorry, yes, set_max_safe_fds(). The problem basically is that with io_uring we
> will have a large number of FDs already allocated by the time
> set_max_safe_fds() is called. set_max_safe_fds() subtracts already_open from
> max_files_per_process allowing few, and even negative, IOs.
>
> I think we should redefine max_files_per_process to be about the number of
> files each *backend* will additionally open. Jelte was working on related
> patches, see [2]
Got it. max_files_per_process is a quaint setting, documented as follows (I
needed the reminder):
If the kernel is enforcing
a safe per-process limit, you don't need to worry about this setting.
But on some platforms (notably, most BSD systems), the kernel will
allow individual processes to open many more files than the system
can actually support if many processes all try to open
that many files. If you find yourself seeing <quote>Too many open
files</quote> failures, try reducing this setting.
I could live with
v6-0003-Reflect-the-value-of-max_safe_fds-in-max_files_pe.patch but would lean
against it since it feels unduly novel to have a setting where we use the
postgresql.conf value to calculate a value that becomes the new SHOW-value of
the same setting. Options I'd consider before that:
- Like you say, "redefine max_files_per_process to be about the number of
files each *backend* will additionally open". It will become normal that
each backend's actual FD list length is max_files_per_process + MaxBackends
if io_method=io_uring. Outcome is not unlike
v6-0002-Bump-postmaster-soft-open-file-limit-RLIMIT_NOFIL.patch +
v6-0003-Reflect-the-value-of-max_safe_fds-in-max_files_pe.patch but we don't
mutate max_files_per_process. Benchmark results should not change beyond
the inter-major-version noise level unless one sets io_method=io_uring. I'm
feeling best about this one, but I've not been thinking about it long.
- When building with io_uring, make the max_files_per_process default
something like 10000 instead. Disadvantages: FD usage grows even if you
don't use io_uring. Merely rebuilding with io_uring (not enabling it at
runtime) will change benchmark results. High MaxBackends still needs to
override the value.
> > > +static void
> > > +maybe_adjust_io_workers(void)
> >
> > This also restarts workers that exit, so perhaps name it
> > start_io_workers_if_missing().
>
> But it also stops IO workers if necessary?
Good point. Maybe just add a comment like "start or stop IO workers to close
the gap between the running count and the configured count intent".
> > > +{
> > ...
> > > + /* Try to launch one. */
> > > + child = StartChildProcess(B_IO_WORKER);
> > > + if (child != NULL)
> > > + {
> > > + io_worker_children[id] = child;
> > > + ++io_worker_count;
> > > + }
> > > + else
> > > + break; /* XXX try again soon? */
> >
> > Can LaunchMissingBackgroundProcesses() become the sole caller of this
> > function, replacing the current mix of callers? That would be more conducive
> > to promptly doing the right thing after launch failure.
>
> I'm not sure that'd be a good idea - right now IO workers are started before
> the startup process, as the startup process might need to perform IO. If we
> started it only later in ServerLoop() we'd potentially do a fair bit of work,
> including starting checkpointer, bgwriter, bgworkers before we started IO
> workers. That shouldn't actively break anything, but it would likely make
> things slower.
I missed that. How about keeping the two calls associated with PM_STARTUP but
replacing the assign_io_workers() and process_pm_child_exit() calls with one
in LaunchMissingBackgroundProcesses()? In the event of a launch failure, I
think that would retry the launch quickly, as opposed to maybe-never.
> I rather dislike the code around when we start what. Leaving AIO aside, during
> a normal startup we start checkpointer, bgwriter before the startup
> process. But during a crash restart we don't explicitly start them. Why make
> things uniform when it coul also be exciting :)
It's become some artisanal code! :)
> > > + /*
> > > + * It's very unlikely, but possible, that reopen fails. E.g. due
> > > + * to memory allocations failing or file permissions changing or
> > > + * such. In that case we need to fail the IO.
> > > + *
> > > + * There's not really a good errno we can report here.
> > > + */
> > > + error_errno = ENOENT;
> >
> > Agreed there's not a good errno, but let's use a fake errno that we're mighty
> > unlikely to confuse with an actual case of libc returning that errno. Like
> > one of EBADF or EOWNERDEAD.
>
> Can we rely on that to be present on all platforms, including windows?
I expect EBADF is universal. EBADF would be fine.
EOWNERDEAD is from 2006.
https://learn.microsoft.com/en-us/cpp/c-runtime-library/errno-constants?view=msvc-140
says VS2015 had EOWNERDEAD (the page doesn't have links for older Visual
Studio versions, so I consider them unknown).
https://github.com/coreutils/gnulib/blob/master/doc/posix-headers/errno.texi
lists some OSs not having it, the newest of which looks like NetBSD 9.3
(2022). We could use it and add a #define for platforms lacking it.
> > > + ret = io_uring_submit(uring_instance);
> > > + pgstat_report_wait_end();
> > > +
> > > + if (ret == -EINTR)
> > > + {
> > > + pgaio_debug(DEBUG3,
> > > + "aio method uring: submit EINTR, nios: %d",
> > > + num_staged_ios);
> > > + }
> > > + else if (ret < 0)
> > > + elog(PANIC, "failed: %d/%s",
> > > + ret, strerror(-ret));
> >
> > I still think (see 2024-09-16 review) EAGAIN should do the documented
> > recommendation instead of PANIC:
> >
> > EAGAIN The kernel was unable to allocate memory for the request, or
> > otherwise ran out of resources to handle it. The application should wait for
> > some completions and try again.
>
> I don't think this can be hit in a recoverable way. We'd likely just end up
> with an untested path that quite possibly would be wrong.
>
> What wait time would be appropriate? What problems would it cause if we just
> slept while holding critical lwlocks? I think it'd typically just delay the
> crash-restart if we did, making it harder to recover from the problem.
I might use 30s like pgwin32_open_handle(), but 30s wouldn't be principled.
> Because we are careful to limit how many outstanding IO requests there are on
> an io_uring instance, the kernel has to have run *severely* out of memory to
> hit this.
>
> I suspect it might currently be *impossible* to hit this due to ENOMEM,
> because io_uring will fall back to allocating individual request, if the batch
> allocation it normally does, fails. My understanding is that for small
> allocations the kernel will try to reclaim memory forever, only large ones can
> fail.
>
> Even if it were possible to hit, the likelihood that postgres can continue to
> work ok if the kernel can't allocate ~250 bytes seems very low.
>
> How about adding a dedicated error message for EAGAIN? IMO io_uring_enter()'s
> meaning of EAGAIN is, uhm, unconvential, so a better error message than
> strerror() might be good?
I'm fine with the present error message.
> Proposed comment:
> /*
> * The io_uring_enter() manpage suggests that the appropriate
> * reaction to EAGAIN is:
> *
> * "The application should wait for some completions and try
> * again"
> *
> * However, it seems unlikely that that would help in our case, as
> * we apply a low limit to the number of outstanding IOs and thus
> * also outstanding completions, making it unlikely that we'd get
> * EAGAIN while the OS is in good working order.
> *
> * Additionally, it would be problematic to just wait here, our
> * caller might hold critical locks. It'd possibly lead to
> * delaying the crash-restart that seems likely to occur when the
> * kernel is under such heavy memory pressure.
> */
That works for me. No retry needed, then.
> > > + pgstat_report_wait_end();
> > > +
> > > + if (ret == -EINTR)
> > > + {
> > > + continue;
> > > + }
> > > + else if (ret != 0)
> > > + {
> > > + elog(PANIC, "unexpected: %d/%s: %m", ret, strerror(-ret));
> >
> > I think errno isn't meaningful here, so %m doesn't belong.
>
> You're right. I wonder if we should make errno meaningful though (by setting
> it), the elog.c machinery captures it and I know that there are logging hooks
> that utilize that fact. That'd also avoid the need to use strerror() here.
That's better still.
> > > --- a/doc/src/sgml/config.sgml
> > > +++ b/doc/src/sgml/config.sgml
> > > @@ -2687,6 +2687,12 @@ include_dir 'conf.d'
> > > <literal>worker</literal> (execute asynchronous I/O using worker processes)
> > > </para>
> > > </listitem>
> > > + <listitem>
> > > + <para>
> > > + <literal>io_uring</literal> (execute asynchronous I/O using
> > > + io_uring, if available)
> > > + </para>
> > > + </listitem>
> >
> > Docs should eventually cover RLIMIT_MEMLOCK per
> > https://github.com/axboe/liburing "ulimit settings".
>
> The way we currently use io_uring (i.e. no registered buffers), the
> RLIMIT_MEMLOCK advice only applies to linux <= 5.11. I'm not sure that's
> worth documenting?
Kernel 5.11 will be 5.5 years old by the time v18 is out. Yeah, no need for
doc coverage of that.
> > One later-patch item:
> >
> > > +static PgAioResult
> > > +SharedBufferCompleteRead(int buf_off, Buffer buffer, uint8 flags, bool failed)
> > > +{
> > ...
> > > + TRACE_POSTGRESQL_BUFFER_READ_DONE(tag.forkNum,
> > > + tag.blockNum,
> > > + tag.spcOid,
> > > + tag.dbOid,
> > > + tag.relNumber,
> > > + INVALID_PROC_NUMBER,
> > > + false);
> >
> > I wondered about whether the buffer-read-done probe should happen in the
> > process that calls the complete_shared callback or in the process that did the
> > buffer-read-start probe.
>
> Yea, that's a good point. I should at least have added a comment pointing out
> that it's a choice with pros and cons.
>
> The reason I went for doing it in the completion callback is that it seemed
> better to get the READ_DONE event as soon as possible, even if the issuer of
> the IO is currently busy doing other things. The shared completion callback is
> after all where the buffer state is updated for shared buffers.
>
> But I think you have a point too.
>
>
> > When I see dtrace examples, they usually involve explicitly naming each PID
> > to trace
>
> TBH, i've only ever used our tracepoints via perf and bpftrace, not dtrace
> itself. For those it's easy to trace more than just a single pid and to
> monitor system-wide. I don't really know enough about using dtrace itself.
Perhaps just a comment, then.
> > Assuming that's indeed the norm, I think the local callback would
> > be the better place, so a given trace contains both probes.
>
> Seems like a shame to add an extra indirect function call
Yep.
> This was an awesome review, thanks!
Glad it helped.
> [1] https://man.openbsd.org/sem_init.3#STANDARDS
> [2] https://postgr.es/m/D80MHNSG4EET.6MSV5G9P130F%40jeltef.nl
^ permalink raw reply [nested|flat] 153+ messages in thread
* Re: AIO v2.5
2025-03-04 19:00 AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-10 18:23 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-11 19:41 ` Re: AIO v2.5 Noah Misch <noah@leadboat.com>
2025-03-11 23:55 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-12 03:57 ` Re: AIO v2.5 Noah Misch <noah@leadboat.com>
@ 2025-03-12 17:06 ` Andres Freund <andres@anarazel.de>
2025-03-12 23:23 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
0 siblings, 1 reply; 153+ messages in thread
From: Andres Freund @ 2025-03-12 17:06 UTC (permalink / raw)
To: Noah Misch <noah@leadboat.com>; +Cc: pgsql-hackers@postgresql.org, Thomas Munro <thomas.munro@gmail.com>; Heikki Linnakangas <hlinnaka@iki.fi>; Robert Haas <robertmhaas@gmail.com>; Jakub Wartak <jakub.wartak@enterprisedb.com>; Jelte Fennema-Nio <postgres@jeltef.nl>
Hi,
On 2025-03-11 20:57:43 -0700, Noah Misch wrote:
> > I think we'll really need to do something about this for BSD users regardless
> > of AIO. Or maybe those OSs should fix something, but somehow I am not having
> > high hopes for an OS that claims to have POSIX confirming unnamed semaphores
> > due to having a syscall that always returns EPERM... [1].
>
> I won't mind a project making things better for non-root BSD users. I do
> think such a project should not block other projects making things better for
> everything else (like $SUBJECT).
Oh, I strongly agree. The main reason I would like it to be addressed that
I'm pretty tired of having to think about open/netbsd whenever we update some
default setting.
> > > On Mon, Mar 10, 2025 at 02:23:12PM -0400, Andres Freund wrote:
> > > > Attached is v2.6 of the AIO patchset.
> > >
> > > > - 0005, 0006 - io_uring support - close, but we need to do something about
> > > > set_max_fds(), which errors out spuriously in some cases
> > >
> > > What do we know about those cases? I don't see a set_max_fds(); is that
> > > set_max_safe_fds(), or something else?
> >
> > Sorry, yes, set_max_safe_fds(). The problem basically is that with io_uring we
> > will have a large number of FDs already allocated by the time
> > set_max_safe_fds() is called. set_max_safe_fds() subtracts already_open from
> > max_files_per_process allowing few, and even negative, IOs.
> >
> > I think we should redefine max_files_per_process to be about the number of
> > files each *backend* will additionally open. Jelte was working on related
> > patches, see [2]
>
> Got it. max_files_per_process is a quaint setting, documented as follows (I
> needed the reminder):
>
> If the kernel is enforcing
> a safe per-process limit, you don't need to worry about this setting.
> But on some platforms (notably, most BSD systems), the kernel will
> allow individual processes to open many more files than the system
> can actually support if many processes all try to open
> that many files. If you find yourself seeing <quote>Too many open
> files</quote> failures, try reducing this setting.
>
> I could live with
> v6-0003-Reflect-the-value-of-max_safe_fds-in-max_files_pe.patch but would lean
> against it since it feels unduly novel to have a setting where we use the
> postgresql.conf value to calculate a value that becomes the new SHOW-value of
> the same setting.
I think we may update some other GUCs, but not sure.
> Options I'd consider before that:
> - Like you say, "redefine max_files_per_process to be about the number of
> files each *backend* will additionally open". It will become normal that
> each backend's actual FD list length is max_files_per_process + MaxBackends
> if io_method=io_uring. Outcome is not unlike
> v6-0002-Bump-postmaster-soft-open-file-limit-RLIMIT_NOFIL.patch +
> v6-0003-Reflect-the-value-of-max_safe_fds-in-max_files_pe.patch but we don't
> mutate max_files_per_process. Benchmark results should not change beyond
> the inter-major-version noise level unless one sets io_method=io_uring. I'm
> feeling best about this one, but I've not been thinking about it long.
Yea, I think that's something probably worth doing separately from Jelte's
patch. I do think that it'd be rather helpful to have jelte's patch to
increase NOFILE in addition though.
> > > > +static void
> > > > +maybe_adjust_io_workers(void)
> > >
> > > This also restarts workers that exit, so perhaps name it
> > > start_io_workers_if_missing().
> >
> > But it also stops IO workers if necessary?
>
> Good point. Maybe just add a comment like "start or stop IO workers to close
> the gap between the running count and the configured count intent".
It's now
/*
* Start or stop IO workers, to close the gap between the number of running
* workers and the number of configured workers. Used to respond to change of
* the io_workers GUC (by increasing and decreasing the number of workers), as
* well as workers terminating in response to errors (by starting
* "replacement" workers).
*/
> > > > +{
> > > ...
> > > > + /* Try to launch one. */
> > > > + child = StartChildProcess(B_IO_WORKER);
> > > > + if (child != NULL)
> > > > + {
> > > > + io_worker_children[id] = child;
> > > > + ++io_worker_count;
> > > > + }
> > > > + else
> > > > + break; /* XXX try again soon? */
> > >
> > > Can LaunchMissingBackgroundProcesses() become the sole caller of this
> > > function, replacing the current mix of callers? That would be more conducive
> > > to promptly doing the right thing after launch failure.
> >
> > I'm not sure that'd be a good idea - right now IO workers are started before
> > the startup process, as the startup process might need to perform IO. If we
> > started it only later in ServerLoop() we'd potentially do a fair bit of work,
> > including starting checkpointer, bgwriter, bgworkers before we started IO
> > workers. That shouldn't actively break anything, but it would likely make
> > things slower.
>
> I missed that. How about keeping the two calls associated with PM_STARTUP but
> replacing the assign_io_workers() and process_pm_child_exit() calls with one
> in LaunchMissingBackgroundProcesses()?
I think replacing the call in assign_io_workers() is a good idea, that way we
don't need assign_io_workers().
Less convinced it's a good idea to do the same for process_pm_child_exit() -
if IO workers errored out we'll launch backends etc before we get to
LaunchMissingBackgroundProcesses(). That's not a fundamental problem, but
seems a bit odd.
I think LaunchMissingBackgroundProcesses() should be split into one that
starts aux processes and one that starts bgworkers. The one maintaining aux
processes should be called before we start backends, the latter not.
> In the event of a launch failure, I think that would retry the launch
> quickly, as opposed to maybe-never.
That's a fair point.
> > > > + /*
> > > > + * It's very unlikely, but possible, that reopen fails. E.g. due
> > > > + * to memory allocations failing or file permissions changing or
> > > > + * such. In that case we need to fail the IO.
> > > > + *
> > > > + * There's not really a good errno we can report here.
> > > > + */
> > > > + error_errno = ENOENT;
> > >
> > > Agreed there's not a good errno, but let's use a fake errno that we're mighty
> > > unlikely to confuse with an actual case of libc returning that errno. Like
> > > one of EBADF or EOWNERDEAD.
> >
> > Can we rely on that to be present on all platforms, including windows?
>
> I expect EBADF is universal. EBADF would be fine.
Hm, that's actually an error that could happen for other reasons, and IMO
would be more confusing than ENOENT. The latter describes the issue to a
reasonable extent, EBADFD seems like it would be more confusing.
I'm not sure it's worth investing time in this - it really shouldn't happen,
and we probably have bigger problems than the error code if it does. But if we
do want to do something, I think I can see a way to report a dedicated error
message for this.
> EOWNERDEAD is from 2006.
> https://learn.microsoft.com/en-us/cpp/c-runtime-library/errno-constants?view=msvc-140
> says VS2015 had EOWNERDEAD (the page doesn't have links for older Visual
> Studio versions, so I consider them unknown).
Oh, that's a larger list than I'd have though.
> https://github.com/coreutils/gnulib/blob/master/doc/posix-headers/errno.texi
> lists some OSs not having it, the newest of which looks like NetBSD 9.3
> (2022). We could use it and add a #define for platforms lacking it.
What would we define it as? I guess we could just pick a high value, but...
Greetings,
Andres Freund
^ permalink raw reply [nested|flat] 153+ messages in thread
* Re: AIO v2.5
2025-03-04 19:00 AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-10 18:23 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-11 19:41 ` Re: AIO v2.5 Noah Misch <noah@leadboat.com>
2025-03-11 23:55 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-12 03:57 ` Re: AIO v2.5 Noah Misch <noah@leadboat.com>
2025-03-12 17:06 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
@ 2025-03-12 23:23 ` Andres Freund <andres@anarazel.de>
2025-03-13 10:53 ` Re: AIO v2.5 Antonin Houska <ah@cybertec.at>
0 siblings, 1 reply; 153+ messages in thread
From: Andres Freund @ 2025-03-12 23:23 UTC (permalink / raw)
To: Noah Misch <noah@leadboat.com>; +Cc: pgsql-hackers@postgresql.org, Thomas Munro <thomas.munro@gmail.com>; Heikki Linnakangas <hlinnaka@iki.fi>; Robert Haas <robertmhaas@gmail.com>; Jakub Wartak <jakub.wartak@enterprisedb.com>; Jelte Fennema-Nio <postgres@jeltef.nl>
Hi,
Attached is v2.7, with the following changes:
- Significantly deduplicated AIO related code bufmgr.c
Previously the code for temp and shared buffers was duplicated to an
uncomfortable degree. Now there is a common helper to implements the
behaviour for both cases.
The BM_PIN_COUNT_WAITER supporting code was also deduplicated, by
introducing a helper function.
- Fixed typos / improved phrasing, per Noah's review
- Add comment explaining why retries for EAGAIN for io_uring_enter syscall
failures don't seem to make sense, improve related error messages slightly
- Added a comment to aio.h explaining that aio_types.h might suffice for
function declarations and aio_init.h for initialization related code.
- Added and expanded comments for PgAioHandleState, explaining the state
machine in more detail.
- Updated README to mention the stage callback (instead of the outdated
"prepare"), plus some other minor cleanups.
- Added a commit rephrasing track_io_timing related docs to talk about waits
- Added FIXME to method_uring.c about the set_max_safe_fds() issue. Depending
on when/how that is resolved, the relevant commits can be reordered relative
to the rest.
- Improved localbuf: patches and commit messages, as per Melanie's review
- Added FIXMEs to the bufmgr.c write support (only in later commit, unlikely
to be realistic for 18) denoting that deadlock risk needs to be
addressed. We probably need some lwlock.c improvements to make that
race-free, otherwise I'd just have fixed this.
- Added a comment discussing the placement of the
TRACE_POSTGRESQL_BUFFER_READ_DONE callback
- removed a few debug ereports() from the StartReadBuffers patch
Unresolved:
- Whether to continue starting new workers in process_pm_child_exit()
- What to name the view (currently pg_aios). I'm inclined to go for
pg_io_handles right now.
- set_max_safe_fds() related issues for the io_uring backend
Greetings,
Andres Freund
Attachments:
[text/x-diff] v2.7-0001-aio-Basic-subsystem-initialization.patch (25.1K, ../../4b3f32ug3cayekysqlgspz2qjmeb7lca3gvazayglxr2m3d4dv@il33accgsji7/2-v2.7-0001-aio-Basic-subsystem-initialization.patch)
download | inline diff:
From 073775698bd3b3ef76b2e4fe938952abda3af4f3 Mon Sep 17 00:00:00 2001
From: Andres Freund <andres@anarazel.de>
Date: Tue, 4 Mar 2025 10:44:56 -0500
Subject: [PATCH v2.7 01/35] aio: Basic subsystem initialization
This commit just does the minimal wiring up of the AIO subsystem, to be fully
added in the next commit, to the rest of the system. The next commit contains
more details about motivation and architecture.
This commit is kept separate to make it easier to review, reviewing how the
AIO subsystem works is a big enough task on its own an rather separate from
reviewing the changes across the tree.
We discussed squashing this commit with the main commit before merging AIO,
but there has been a mild preference for keeping it separate.
Discussion: https://postgr.es/m/uvrtrknj4kdytuboidbhwclo4gxhswwcpgadptsjvjqcluzmah%40brqs62irg4dt
---
src/include/storage/aio.h | 38 ++++++++
src/include/storage/aio_subsys.h | 33 +++++++
src/include/utils/guc.h | 1 +
src/include/utils/guc_hooks.h | 2 +
src/include/utils/resowner.h | 5 ++
src/backend/access/transam/xact.c | 12 +++
src/backend/postmaster/autovacuum.c | 2 +
src/backend/postmaster/bgwriter.c | 2 +
src/backend/postmaster/checkpointer.c | 2 +
src/backend/postmaster/pgarch.c | 2 +
src/backend/postmaster/walsummarizer.c | 2 +
src/backend/postmaster/walwriter.c | 2 +
src/backend/replication/walsender.c | 2 +
src/backend/storage/aio/Makefile | 2 +
src/backend/storage/aio/aio.c | 90 +++++++++++++++++++
src/backend/storage/aio/aio_init.c | 37 ++++++++
src/backend/storage/aio/meson.build | 2 +
src/backend/storage/ipc/ipci.c | 3 +
src/backend/utils/init/postinit.c | 7 ++
src/backend/utils/misc/guc_tables.c | 23 +++++
src/backend/utils/misc/postgresql.conf.sample | 6 ++
src/backend/utils/resowner/resowner.c | 29 ++++++
doc/src/sgml/config.sgml | 51 +++++++++++
src/tools/pgindent/typedefs.list | 1 +
24 files changed, 356 insertions(+)
create mode 100644 src/include/storage/aio.h
create mode 100644 src/include/storage/aio_subsys.h
create mode 100644 src/backend/storage/aio/aio.c
create mode 100644 src/backend/storage/aio/aio_init.c
diff --git a/src/include/storage/aio.h b/src/include/storage/aio.h
new file mode 100644
index 00000000000..e79d5343038
--- /dev/null
+++ b/src/include/storage/aio.h
@@ -0,0 +1,38 @@
+/*-------------------------------------------------------------------------
+ *
+ * aio.h
+ * Main AIO interface
+ *
+ *
+ * Portions Copyright (c) 1996-2025, PostgreSQL Global Development Group
+ * Portions Copyright (c) 1994, Regents of the University of California
+ *
+ * src/include/storage/aio.h
+ *
+ *-------------------------------------------------------------------------
+ */
+#ifndef AIO_H
+#define AIO_H
+
+
+
+/* Enum for io_method GUC. */
+typedef enum IoMethod
+{
+ IOMETHOD_SYNC = 0,
+} IoMethod;
+
+/* We'll default to synchronous execution. */
+#define DEFAULT_IO_METHOD IOMETHOD_SYNC
+
+
+struct dlist_node;
+extern void pgaio_io_release_resowner(struct dlist_node *ioh_node, bool on_error);
+
+
+/* GUCs */
+extern PGDLLIMPORT int io_method;
+extern PGDLLIMPORT int io_max_concurrency;
+
+
+#endif /* AIO_H */
diff --git a/src/include/storage/aio_subsys.h b/src/include/storage/aio_subsys.h
new file mode 100644
index 00000000000..e4faf692a38
--- /dev/null
+++ b/src/include/storage/aio_subsys.h
@@ -0,0 +1,33 @@
+/*-------------------------------------------------------------------------
+ *
+ * aio_subsys.h
+ * Interaction with AIO as a subsystem, rather than actually issuing AIO
+ *
+ * This header is for AIO related functionality that's being called by files
+ * that don't perform AIO but interact with it in some form. E.g. postmaster.c
+ * and shared memory initialization need to initialize AIO but don't perform
+ * AIO.
+ *
+ * Portions Copyright (c) 1996-2025, PostgreSQL Global Development Group
+ * Portions Copyright (c) 1994, Regents of the University of California
+ *
+ * src/include/storage/aio_subsys.h
+ *
+ *-------------------------------------------------------------------------
+ */
+#ifndef AIO_SUBSYS_H
+#define AIO_SUBSYS_H
+
+
+/* aio_init.c */
+extern Size AioShmemSize(void);
+extern void AioShmemInit(void);
+
+extern void pgaio_init_backend(void);
+
+
+/* aio.c */
+extern void pgaio_error_cleanup(void);
+extern void AtEOXact_Aio(bool is_commit);
+
+#endif /* AIO_SUBSYS_H */
diff --git a/src/include/utils/guc.h b/src/include/utils/guc.h
index 1233e07d7da..de3bc37264f 100644
--- a/src/include/utils/guc.h
+++ b/src/include/utils/guc.h
@@ -318,6 +318,7 @@ extern PGDLLIMPORT bool optimize_bounded_sort;
*/
extern PGDLLIMPORT const struct config_enum_entry archive_mode_options[];
extern PGDLLIMPORT const struct config_enum_entry dynamic_shared_memory_options[];
+extern PGDLLIMPORT const struct config_enum_entry io_method_options[];
extern PGDLLIMPORT const struct config_enum_entry recovery_target_action_options[];
extern PGDLLIMPORT const struct config_enum_entry wal_level_options[];
extern PGDLLIMPORT const struct config_enum_entry wal_sync_method_options[];
diff --git a/src/include/utils/guc_hooks.h b/src/include/utils/guc_hooks.h
index 9a0d8ec85c7..a3eba8fbe21 100644
--- a/src/include/utils/guc_hooks.h
+++ b/src/include/utils/guc_hooks.h
@@ -64,6 +64,8 @@ extern bool check_default_with_oids(bool *newval, void **extra,
extern bool check_effective_io_concurrency(int *newval, void **extra,
GucSource source);
extern bool check_huge_page_size(int *newval, void **extra, GucSource source);
+extern void assign_io_method(int newval, void *extra);
+extern bool check_io_max_concurrency(int *newval, void **extra, GucSource source);
extern const char *show_in_hot_standby(void);
extern bool check_locale_messages(char **newval, void **extra, GucSource source);
extern void assign_locale_messages(const char *newval, void *extra);
diff --git a/src/include/utils/resowner.h b/src/include/utils/resowner.h
index e8d452ca7ee..aede4bfc820 100644
--- a/src/include/utils/resowner.h
+++ b/src/include/utils/resowner.h
@@ -164,4 +164,9 @@ struct LOCALLOCK;
extern void ResourceOwnerRememberLock(ResourceOwner owner, struct LOCALLOCK *locallock);
extern void ResourceOwnerForgetLock(ResourceOwner owner, struct LOCALLOCK *locallock);
+/* special support for AIO */
+struct dlist_node;
+extern void ResourceOwnerRememberAioHandle(ResourceOwner owner, struct dlist_node *ioh_node);
+extern void ResourceOwnerForgetAioHandle(ResourceOwner owner, struct dlist_node *ioh_node);
+
#endif /* RESOWNER_H */
diff --git a/src/backend/access/transam/xact.c b/src/backend/access/transam/xact.c
index 1b4f21a88d3..b885513f765 100644
--- a/src/backend/access/transam/xact.c
+++ b/src/backend/access/transam/xact.c
@@ -51,6 +51,7 @@
#include "replication/origin.h"
#include "replication/snapbuild.h"
#include "replication/syncrep.h"
+#include "storage/aio_subsys.h"
#include "storage/condition_variable.h"
#include "storage/fd.h"
#include "storage/lmgr.h"
@@ -2411,6 +2412,8 @@ CommitTransaction(void)
RESOURCE_RELEASE_BEFORE_LOCKS,
true, true);
+ AtEOXact_Aio(true);
+
/* Check we've released all buffer pins */
AtEOXact_Buffers(true);
@@ -2716,6 +2719,8 @@ PrepareTransaction(void)
RESOURCE_RELEASE_BEFORE_LOCKS,
true, true);
+ AtEOXact_Aio(true);
+
/* Check we've released all buffer pins */
AtEOXact_Buffers(true);
@@ -2830,6 +2835,8 @@ AbortTransaction(void)
pgstat_report_wait_end();
pgstat_progress_end_command();
+ pgaio_error_cleanup();
+
/* Clean up buffer content locks, too */
UnlockBuffers();
@@ -2960,6 +2967,7 @@ AbortTransaction(void)
ResourceOwnerRelease(TopTransactionResourceOwner,
RESOURCE_RELEASE_BEFORE_LOCKS,
false, true);
+ AtEOXact_Aio(false);
AtEOXact_Buffers(false);
AtEOXact_RelationCache(false);
AtEOXact_TypeCache();
@@ -5232,6 +5240,9 @@ AbortSubTransaction(void)
pgstat_report_wait_end();
pgstat_progress_end_command();
+
+ pgaio_error_cleanup();
+
UnlockBuffers();
/* Reset WAL record construction state */
@@ -5326,6 +5337,7 @@ AbortSubTransaction(void)
RESOURCE_RELEASE_BEFORE_LOCKS,
false, false);
+ AtEOXact_Aio(false);
AtEOSubXact_RelationCache(false, s->subTransactionId,
s->parent->subTransactionId);
AtEOSubXact_TypeCache();
diff --git a/src/backend/postmaster/autovacuum.c b/src/backend/postmaster/autovacuum.c
index 800815dfbcc..5aa0fa665c2 100644
--- a/src/backend/postmaster/autovacuum.c
+++ b/src/backend/postmaster/autovacuum.c
@@ -88,6 +88,7 @@
#include "postmaster/autovacuum.h"
#include "postmaster/interrupt.h"
#include "postmaster/postmaster.h"
+#include "storage/aio_subsys.h"
#include "storage/bufmgr.h"
#include "storage/ipc.h"
#include "storage/latch.h"
@@ -465,6 +466,7 @@ AutoVacLauncherMain(const void *startup_data, size_t startup_data_len)
*/
LWLockReleaseAll();
pgstat_report_wait_end();
+ pgaio_error_cleanup();
UnlockBuffers();
/* this is probably dead code, but let's be safe: */
if (AuxProcessResourceOwner)
diff --git a/src/backend/postmaster/bgwriter.c b/src/backend/postmaster/bgwriter.c
index a688cc5d2a1..72f5acceec7 100644
--- a/src/backend/postmaster/bgwriter.c
+++ b/src/backend/postmaster/bgwriter.c
@@ -38,6 +38,7 @@
#include "postmaster/auxprocess.h"
#include "postmaster/bgwriter.h"
#include "postmaster/interrupt.h"
+#include "storage/aio_subsys.h"
#include "storage/buf_internals.h"
#include "storage/bufmgr.h"
#include "storage/condition_variable.h"
@@ -168,6 +169,7 @@ BackgroundWriterMain(const void *startup_data, size_t startup_data_len)
*/
LWLockReleaseAll();
ConditionVariableCancelSleep();
+ pgaio_error_cleanup();
UnlockBuffers();
ReleaseAuxProcessResources(false);
AtEOXact_Buffers(false);
diff --git a/src/backend/postmaster/checkpointer.c b/src/backend/postmaster/checkpointer.c
index 0e228d143a0..fda91ffd1ce 100644
--- a/src/backend/postmaster/checkpointer.c
+++ b/src/backend/postmaster/checkpointer.c
@@ -49,6 +49,7 @@
#include "postmaster/bgwriter.h"
#include "postmaster/interrupt.h"
#include "replication/syncrep.h"
+#include "storage/aio_subsys.h"
#include "storage/bufmgr.h"
#include "storage/condition_variable.h"
#include "storage/fd.h"
@@ -276,6 +277,7 @@ CheckpointerMain(const void *startup_data, size_t startup_data_len)
LWLockReleaseAll();
ConditionVariableCancelSleep();
pgstat_report_wait_end();
+ pgaio_error_cleanup();
UnlockBuffers();
ReleaseAuxProcessResources(false);
AtEOXact_Buffers(false);
diff --git a/src/backend/postmaster/pgarch.c b/src/backend/postmaster/pgarch.c
index dbe4e1d426b..7e622ae4bd2 100644
--- a/src/backend/postmaster/pgarch.c
+++ b/src/backend/postmaster/pgarch.c
@@ -40,6 +40,7 @@
#include "postmaster/interrupt.h"
#include "postmaster/pgarch.h"
#include "storage/condition_variable.h"
+#include "storage/aio_subsys.h"
#include "storage/fd.h"
#include "storage/ipc.h"
#include "storage/latch.h"
@@ -568,6 +569,7 @@ pgarch_archiveXlog(char *xlog)
LWLockReleaseAll();
ConditionVariableCancelSleep();
pgstat_report_wait_end();
+ pgaio_error_cleanup();
ReleaseAuxProcessResources(false);
AtEOXact_Files(false);
AtEOXact_HashTables(false);
diff --git a/src/backend/postmaster/walsummarizer.c b/src/backend/postmaster/walsummarizer.c
index ccba0f84e6e..0fec4f1f871 100644
--- a/src/backend/postmaster/walsummarizer.c
+++ b/src/backend/postmaster/walsummarizer.c
@@ -38,6 +38,7 @@
#include "postmaster/interrupt.h"
#include "postmaster/walsummarizer.h"
#include "replication/walreceiver.h"
+#include "storage/aio_subsys.h"
#include "storage/fd.h"
#include "storage/ipc.h"
#include "storage/latch.h"
@@ -289,6 +290,7 @@ WalSummarizerMain(const void *startup_data, size_t startup_data_len)
LWLockReleaseAll();
ConditionVariableCancelSleep();
pgstat_report_wait_end();
+ pgaio_error_cleanup();
ReleaseAuxProcessResources(false);
AtEOXact_Files(false);
AtEOXact_HashTables(false);
diff --git a/src/backend/postmaster/walwriter.c b/src/backend/postmaster/walwriter.c
index 0380601bcbb..fd92c8b7a33 100644
--- a/src/backend/postmaster/walwriter.c
+++ b/src/backend/postmaster/walwriter.c
@@ -51,6 +51,7 @@
#include "postmaster/auxprocess.h"
#include "postmaster/interrupt.h"
#include "postmaster/walwriter.h"
+#include "storage/aio_subsys.h"
#include "storage/bufmgr.h"
#include "storage/condition_variable.h"
#include "storage/fd.h"
@@ -164,6 +165,7 @@ WalWriterMain(const void *startup_data, size_t startup_data_len)
LWLockReleaseAll();
ConditionVariableCancelSleep();
pgstat_report_wait_end();
+ pgaio_error_cleanup();
UnlockBuffers();
ReleaseAuxProcessResources(false);
AtEOXact_Buffers(false);
diff --git a/src/backend/replication/walsender.c b/src/backend/replication/walsender.c
index 446d10c1a7d..69097125606 100644
--- a/src/backend/replication/walsender.c
+++ b/src/backend/replication/walsender.c
@@ -79,6 +79,7 @@
#include "replication/walsender.h"
#include "replication/walsender_private.h"
#include "storage/condition_variable.h"
+#include "storage/aio_subsys.h"
#include "storage/fd.h"
#include "storage/ipc.h"
#include "storage/pmsignal.h"
@@ -327,6 +328,7 @@ WalSndErrorCleanup(void)
LWLockReleaseAll();
ConditionVariableCancelSleep();
pgstat_report_wait_end();
+ pgaio_error_cleanup();
if (xlogreader != NULL && xlogreader->seg.ws_file >= 0)
wal_segment_close(xlogreader);
diff --git a/src/backend/storage/aio/Makefile b/src/backend/storage/aio/Makefile
index 2f29a9ec4d1..eaeaeeee8e3 100644
--- a/src/backend/storage/aio/Makefile
+++ b/src/backend/storage/aio/Makefile
@@ -9,6 +9,8 @@ top_builddir = ../../../..
include $(top_builddir)/src/Makefile.global
OBJS = \
+ aio.o \
+ aio_init.o \
read_stream.o
include $(top_srcdir)/src/backend/common.mk
diff --git a/src/backend/storage/aio/aio.c b/src/backend/storage/aio/aio.c
new file mode 100644
index 00000000000..828a94efdc3
--- /dev/null
+++ b/src/backend/storage/aio/aio.c
@@ -0,0 +1,90 @@
+/*-------------------------------------------------------------------------
+ *
+ * aio.c
+ * AIO - Core Logic
+ *
+ * Portions Copyright (c) 1996-2025, PostgreSQL Global Development Group
+ * Portions Copyright (c) 1994, Regents of the University of California
+ *
+ * IDENTIFICATION
+ * src/backend/storage/aio/aio.c
+ *
+ *-------------------------------------------------------------------------
+ */
+
+#include "postgres.h"
+
+#include "lib/ilist.h"
+#include "storage/aio.h"
+#include "storage/aio_subsys.h"
+#include "utils/guc.h"
+#include "utils/guc_hooks.h"
+
+
+/* Options for io_method. */
+const struct config_enum_entry io_method_options[] = {
+ {"sync", IOMETHOD_SYNC, false},
+ {NULL, 0, false}
+};
+
+/* GUCs */
+int io_method = DEFAULT_IO_METHOD;
+int io_max_concurrency = -1;
+
+
+
+/*
+ * Release IO handle during resource owner cleanup.
+ */
+void
+pgaio_io_release_resowner(dlist_node *ioh_node, bool on_error)
+{
+}
+
+/*
+ * Perform AIO related cleanup after an error.
+ *
+ * This should be called early in the error recovery paths, as later steps may
+ * need to issue AIO (e.g. to record a transaction abort WAL record).
+ */
+void
+pgaio_error_cleanup(void)
+{
+}
+
+/*
+ * Perform AIO related checks at (sub-)transactional boundaries.
+ *
+ * This should be called late during (sub-)transactional commit/abort, after
+ * all steps that might need to perform AIO, so that we can verify that the
+ * AIO subsystem is in a valid state at the end of a transaction.
+ */
+void
+AtEOXact_Aio(bool is_commit)
+{
+}
+
+void
+assign_io_method(int newval, void *extra)
+{
+}
+
+bool
+check_io_max_concurrency(int *newval, void **extra, GucSource source)
+{
+ if (*newval == -1)
+ {
+ /*
+ * Auto-tuning will be applied later during startup, as auto-tuning
+ * depends on the value of various GUCs.
+ */
+ return true;
+ }
+ else if (*newval == 0)
+ {
+ GUC_check_errdetail("Only -1 or values bigger than 0 are valid.");
+ return false;
+ }
+
+ return true;
+}
diff --git a/src/backend/storage/aio/aio_init.c b/src/backend/storage/aio/aio_init.c
new file mode 100644
index 00000000000..aeacc144149
--- /dev/null
+++ b/src/backend/storage/aio/aio_init.c
@@ -0,0 +1,37 @@
+/*-------------------------------------------------------------------------
+ *
+ * aio_init.c
+ * AIO - Subsystem Initialization
+ *
+ * Portions Copyright (c) 1996-2025, PostgreSQL Global Development Group
+ * Portions Copyright (c) 1994, Regents of the University of California
+ *
+ * IDENTIFICATION
+ * src/backend/storage/aio/aio_init.c
+ *
+ *-------------------------------------------------------------------------
+ */
+
+#include "postgres.h"
+
+#include "storage/aio_subsys.h"
+
+
+
+Size
+AioShmemSize(void)
+{
+ Size sz = 0;
+
+ return sz;
+}
+
+void
+AioShmemInit(void)
+{
+}
+
+void
+pgaio_init_backend(void)
+{
+}
diff --git a/src/backend/storage/aio/meson.build b/src/backend/storage/aio/meson.build
index 8abe0eb4863..c822fd4ddf7 100644
--- a/src/backend/storage/aio/meson.build
+++ b/src/backend/storage/aio/meson.build
@@ -1,5 +1,7 @@
# Copyright (c) 2024-2025, PostgreSQL Global Development Group
backend_sources += files(
+ 'aio.c',
+ 'aio_init.c',
'read_stream.c',
)
diff --git a/src/backend/storage/ipc/ipci.c b/src/backend/storage/ipc/ipci.c
index 174eed70367..2fa045e6b0f 100644
--- a/src/backend/storage/ipc/ipci.c
+++ b/src/backend/storage/ipc/ipci.c
@@ -37,6 +37,7 @@
#include "replication/slotsync.h"
#include "replication/walreceiver.h"
#include "replication/walsender.h"
+#include "storage/aio_subsys.h"
#include "storage/bufmgr.h"
#include "storage/dsm.h"
#include "storage/dsm_registry.h"
@@ -148,6 +149,7 @@ CalculateShmemSize(int *num_semaphores)
size = add_size(size, WaitEventCustomShmemSize());
size = add_size(size, InjectionPointShmemSize());
size = add_size(size, SlotSyncShmemSize());
+ size = add_size(size, AioShmemSize());
/* include additional requested shmem from preload libraries */
size = add_size(size, total_addin_request);
@@ -340,6 +342,7 @@ CreateOrAttachShmemStructs(void)
StatsShmemInit();
WaitEventCustomShmemInit();
InjectionPointShmemInit();
+ AioShmemInit();
}
/*
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index 4b2faf1ba9d..7958ea11b73 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -43,6 +43,7 @@
#include "replication/slot.h"
#include "replication/slotsync.h"
#include "replication/walsender.h"
+#include "storage/aio_subsys.h"
#include "storage/bufmgr.h"
#include "storage/fd.h"
#include "storage/ipc.h"
@@ -635,6 +636,12 @@ BaseInit(void)
*/
pgstat_initialize();
+ /*
+ * Initialize AIO before infrastructure that might need to actually
+ * execute AIO.
+ */
+ pgaio_init_backend();
+
/* Do local initialization of storage and buffer managers */
InitSync();
smgrinit();
diff --git a/src/backend/utils/misc/guc_tables.c b/src/backend/utils/misc/guc_tables.c
index 508970680d1..a456f27ba1e 100644
--- a/src/backend/utils/misc/guc_tables.c
+++ b/src/backend/utils/misc/guc_tables.c
@@ -72,6 +72,7 @@
#include "replication/slot.h"
#include "replication/slotsync.h"
#include "replication/syncrep.h"
+#include "storage/aio.h"
#include "storage/bufmgr.h"
#include "storage/bufpage.h"
#include "storage/large_object.h"
@@ -3245,6 +3246,18 @@ struct config_int ConfigureNamesInt[] =
NULL, NULL, NULL
},
+ {
+ {"io_max_concurrency",
+ PGC_POSTMASTER,
+ RESOURCES_IO,
+ gettext_noop("Max number of IOs that one process can execute simultaneously."),
+ NULL,
+ },
+ &io_max_concurrency,
+ -1, -1, 1024,
+ check_io_max_concurrency, NULL, NULL
+ },
+
{
{"backend_flush_after", PGC_USERSET, RESOURCES_IO,
gettext_noop("Number of pages after which previously performed writes are flushed to disk."),
@@ -5302,6 +5315,16 @@ struct config_enum ConfigureNamesEnum[] =
NULL, NULL, NULL
},
+ {
+ {"io_method", PGC_POSTMASTER, RESOURCES_IO,
+ gettext_noop("Selects the method for executing asynchronous I/O."),
+ NULL
+ },
+ &io_method,
+ DEFAULT_IO_METHOD, io_method_options,
+ NULL, assign_io_method, NULL
+ },
+
/* End-of-list marker */
{
{NULL, 0, 0, NULL, NULL}, NULL, 0, NULL, NULL, NULL, NULL
diff --git a/src/backend/utils/misc/postgresql.conf.sample b/src/backend/utils/misc/postgresql.conf.sample
index 36cb64d7ebc..58b256e2221 100644
--- a/src/backend/utils/misc/postgresql.conf.sample
+++ b/src/backend/utils/misc/postgresql.conf.sample
@@ -202,6 +202,12 @@
#maintenance_io_concurrency = 10 # 1-1000; 0 disables prefetching
#io_combine_limit = 128kB # usually 1-32 blocks (depends on OS)
+#io_method = sync # sync (change requires restart)
+#io_max_concurrency = -1 # Max number of IOs that one process
+ # can execute simultaneously
+ # -1 sets based on shared_buffers
+ # (change requires restart)
+
# - Worker Processes -
#max_worker_processes = 8 # (change requires restart)
diff --git a/src/backend/utils/resowner/resowner.c b/src/backend/utils/resowner/resowner.c
index ac5ca4a765e..d39f3e1b655 100644
--- a/src/backend/utils/resowner/resowner.c
+++ b/src/backend/utils/resowner/resowner.c
@@ -47,6 +47,8 @@
#include "common/hashfn.h"
#include "common/int.h"
+#include "lib/ilist.h"
+#include "storage/aio.h"
#include "storage/ipc.h"
#include "storage/predicate.h"
#include "storage/proc.h"
@@ -155,6 +157,12 @@ struct ResourceOwnerData
/* The local locks cache. */
LOCALLOCK *locks[MAX_RESOWNER_LOCKS]; /* list of owned locks */
+
+ /*
+ * AIO handles need be registered in critical sections and therefore
+ * cannot use the normal ResourceElem mechanism.
+ */
+ dlist_head aio_handles;
};
@@ -425,6 +433,8 @@ ResourceOwnerCreate(ResourceOwner parent, const char *name)
parent->firstchild = owner;
}
+ dlist_init(&owner->aio_handles);
+
return owner;
}
@@ -725,6 +735,13 @@ ResourceOwnerReleaseInternal(ResourceOwner owner,
* so issue warnings. In the abort case, just clean up quietly.
*/
ResourceOwnerReleaseAll(owner, phase, isCommit);
+
+ while (!dlist_is_empty(&owner->aio_handles))
+ {
+ dlist_node *node = dlist_head_node(&owner->aio_handles);
+
+ pgaio_io_release_resowner(node, !isCommit);
+ }
}
else if (phase == RESOURCE_RELEASE_LOCKS)
{
@@ -1082,3 +1099,15 @@ ResourceOwnerForgetLock(ResourceOwner owner, LOCALLOCK *locallock)
elog(ERROR, "lock reference %p is not owned by resource owner %s",
locallock, owner->name);
}
+
+void
+ResourceOwnerRememberAioHandle(ResourceOwner owner, struct dlist_node *ioh_node)
+{
+ dlist_push_tail(&owner->aio_handles, ioh_node);
+}
+
+void
+ResourceOwnerForgetAioHandle(ResourceOwner owner, struct dlist_node *ioh_node)
+{
+ dlist_delete_from(&owner->aio_handles, ioh_node);
+}
diff --git a/doc/src/sgml/config.sgml b/doc/src/sgml/config.sgml
index 8c82b39a89d..7ca8e7867cb 100644
--- a/doc/src/sgml/config.sgml
+++ b/doc/src/sgml/config.sgml
@@ -2638,6 +2638,57 @@ include_dir 'conf.d'
</para>
</listitem>
</varlistentry>
+
+ <varlistentry id="guc-io-max-concurrency" xreflabel="io_max_concurrency">
+ <term><varname>io_max_concurrency</varname> (<type>integer</type>)
+ <indexterm>
+ <primary><varname>io_max_concurrency</varname> configuration parameter</primary>
+ </indexterm>
+ </term>
+ <listitem>
+ <para>
+ Controls the maximum number of I/O operations that one process can
+ execute simultaneously.
+ </para>
+ <para>
+ The default setting of <literal>-1</literal> selects a number based
+ on <xref linkend="guc-shared-buffers"/> and the maximum number of
+ processes (<xref linkend="guc-max-connections"/>, <xref
+ linkend="guc-autovacuum-worker-slots"/>, <xref
+ linkend="guc-max-worker-processes"/> and <xref
+ linkend="guc-max-wal-senders"/>), but not more than
+ <literal>64</literal>.
+ </para>
+ <para>
+ This parameter can only be set at server start.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry id="guc-io-method" xreflabel="io_method">
+ <term><varname>io_method</varname> (<type>enum</type>)
+ <indexterm>
+ <primary><varname>io_method</varname> configuration parameter</primary>
+ </indexterm>
+ </term>
+ <listitem>
+ <para>
+ Selects the method for executing asynchronous I/O.
+ Possible values are:
+ <itemizedlist>
+ <listitem>
+ <para>
+ <literal>sync</literal> (execute asynchronous-eligible I/O synchronously)
+ </para>
+ </listitem>
+ </itemizedlist>
+ </para>
+ <para>
+ This parameter can only be set at server start.
+ </para>
+ </listitem>
+ </varlistentry>
+
</variablelist>
</sect2>
diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list
index a2e592dbbbb..d3eef76bc2e 100644
--- a/src/tools/pgindent/typedefs.list
+++ b/src/tools/pgindent/typedefs.list
@@ -1279,6 +1279,7 @@ IntoClause
InvalMessageArray
InvalidationInfo
InvalidationMsgsGroup
+IoMethod
IpcMemoryId
IpcMemoryKey
IpcMemoryState
--
2.48.1.76.g4e746b1a31.dirty
[text/x-diff] v2.7-0002-aio-Add-core-asynchronous-I-O-infrastructure.patch (89.6K, ../../4b3f32ug3cayekysqlgspz2qjmeb7lca3gvazayglxr2m3d4dv@il33accgsji7/3-v2.7-0002-aio-Add-core-asynchronous-I-O-infrastructure.patch)
download | inline diff:
From 894e1024ae78353a242d8189115f30c0240a059b Mon Sep 17 00:00:00 2001
From: Andres Freund <andres@anarazel.de>
Date: Fri, 7 Mar 2025 10:14:31 -0500
Subject: [PATCH v2.7 02/35] aio: Add core asynchronous I/O infrastructure
The main motivations to use AIO in PostgreSQL are:
a) Reduce the time spent waiting for IO by issuing IO sufficiently early.
In a few places we have approximated this using posix_fadvise() based
prefetching, but that is fairly limited (no completion feedback, double the
syscalls, only works with buffered IO, only works on some OSs).
b) Allow to use Direct-I/O (DIO).
DIO can offload most of the work for IO to hardware and thus increase
throughput / decrease CPU utilization, as well as reduce latency. While we
have gained the ability to configure DIO in d4e71df6, it is not yet usable
for real world workloads, as every IO is executed synchronously.
For portability, the new AIO infrastructure allows to implement AIO using
different methods. The choice of the AIO method is controlled by a new
io_method GUC. As of this commit, the only implemented method is "sync",
i.e. AIO is not actually executed asynchronously. The "sync" method exists to
allow to bypass most of the new code initially.
Subsequent commits will introduce additional IO methods, including a
cross-platform method implemented using worker processes and a linux specific
method using io_uring.
To allow different parts of postgres to use AIO, the core AIO infrastructure
does not need to know what kind of files it is operating on. The necessary
behavioral differences for different files are abstracted as "AIO
Targets". One example target would be smgr. For boring portability reasons all
targets currently need to be added to an array in aio_target.c. This commit
does not implement any AIO targets, just the infrastructure for them. The smgr
target will be added in a later commit.
Completion (and other events) of IOs for one type of file (i.e. one AIO
target) need to be reacted to differently based on the IO operation and the
callsite. This is made possible by callbacks that can be registered on
IOs. E.g. an smgr read into a local buffer does not need to update the
corresponding BufferDesc (as there is none), but a read into shared buffers
does. This commit does not contain any callbacks, they will be added in
subsequent commits.
For now the AIO infrastructure only understands READV and WRITEV operations,
but it is expected that more operations will be added. E.g. fsync/fdatasync,
flush_range and network operations like send/recv.
As of this commit nothing uses the AIO infrastructure. Other commits will add
an smgr target, md.c and bufmgr.c callbacks and then finally use AIO for
read_stream.c IO, which, in one fell swoop, will convert all read stream users
to AIO.
The goal is to use AIO in many more places. There are patches to use AIO for
checkpointer and bgwriter that are reasonably close to being ready. There also
are prototypes to use it for WAL, relation extension, backend writes and many
more. Those prototypes were important to ensure the design of the AIO
subsystem is not too limiting (e.g. WAL writes need to happen in critical
sections, which influenced a lot of the design).
A future commit will add an AIO README explaining the AIO architecture and how
to use the AIO subsystem. The README is added later, as it references details
only added in later commits.
Many many more people than the folks named below have contributed with
feedback, work on semi-independent patches etc. E.g. various folks have
contributed patches to use the read stream infrastructure (added by Thomas in
b5a9b18cd0b) in more places. Similarly, a *lot* of folks have contributed to
the CI infrastructure, that I started to work on to make adding AIO feasible.
Some of the work by contributors has gone into the "v1" prototype of AIO,
which heavily influenced the current design of the AIO subsystem. None of the
code from that directly survives, but without the prototype, the current
version of the AIO infrastructure would not exist.
Similarly, the reviewers below have not necessarily looked at the current
design or the whole infrastructure, but have provided very valuable input. I
am to blame for problems, not they.
Author: Andres Freund <andres@anarazel.de>
Co-authored-by: Thomas Munro <thomas.munro@gmail.com>
Co-authored-by: Nazir Bilal Yavuz <byavuz81@gmail.com>
Co-authored-by: Melanie Plageman <melanieplageman@gmail.com>
Reviewed-by: Heikki Linnakangas <hlinnaka@iki.fi>
Reviewed-by: Noah Misch <noah@leadboat.com>
Reviewed-by: Jakub Wartak <jakub.wartak@enterprisedb.com>
Reviewed-by: Melanie Plageman <melanieplageman@gmail.com>
Reviewed-by: Robert Haas <robertmhaas@gmail.com>
Reviewed-by: Dmitry Dolgov <9erthalion6@gmail.com>
Discussion: https://postgr.es/m/uvrtrknj4kdytuboidbhwclo4gxhswwcpgadptsjvjqcluzmah%40brqs62irg4dt
Discussion: https://postgr.es/m/20210223100344.llw5an2aklengrmn@alap3.anarazel.de
Discussion: https://postgr.es/m/stj36ea6yyhoxtqkhpieia2z4krnam7qyetc57rfezgk4zgapf@gcnactj4z56m
---
src/include/storage/aio.h | 304 +++++
src/include/storage/aio_internal.h | 395 ++++++
src/include/storage/aio_types.h | 117 ++
src/backend/storage/aio/Makefile | 4 +
src/backend/storage/aio/aio.c | 1130 +++++++++++++++++
src/backend/storage/aio/aio_callback.c | 307 +++++
src/backend/storage/aio/aio_init.c | 187 +++
src/backend/storage/aio/aio_io.c | 180 +++
src/backend/storage/aio/aio_target.c | 118 ++
src/backend/storage/aio/meson.build | 4 +
src/backend/storage/aio/method_sync.c | 47 +
.../utils/activity/wait_event_names.txt | 1 +
src/tools/pgindent/typedefs.list | 21 +
13 files changed, 2815 insertions(+)
create mode 100644 src/include/storage/aio_internal.h
create mode 100644 src/include/storage/aio_types.h
create mode 100644 src/backend/storage/aio/aio_callback.c
create mode 100644 src/backend/storage/aio/aio_io.c
create mode 100644 src/backend/storage/aio/aio_target.c
create mode 100644 src/backend/storage/aio/method_sync.c
diff --git a/src/include/storage/aio.h b/src/include/storage/aio.h
index e79d5343038..a0a0f23f3a5 100644
--- a/src/include/storage/aio.h
+++ b/src/include/storage/aio.h
@@ -3,6 +3,10 @@
* aio.h
* Main AIO interface
*
+ * This is the header to include when actually issuing AIO. When just
+ * declaring functions involving an AIO related type, it might suffice to
+ * include aio_types.h. Initialization related functions are in the dedicated
+ * aio_init.h.
*
* Portions Copyright (c) 1996-2025, PostgreSQL Global Development Group
* Portions Copyright (c) 1994, Regents of the University of California
@@ -14,6 +18,9 @@
#ifndef AIO_H
#define AIO_H
+#include "storage/aio_types.h"
+#include "storage/procnumber.h"
+
/* Enum for io_method GUC. */
@@ -26,9 +33,306 @@ typedef enum IoMethod
#define DEFAULT_IO_METHOD IOMETHOD_SYNC
+/*
+ * Flags for an IO that can be set with pgaio_io_set_flag().
+ */
+typedef enum PgAioHandleFlags
+{
+ /*
+ * The IO references backend local memory.
+ *
+ * This needs to be set on an IO whenever the IO references process-local
+ * memory. Some IO methods do not support executing IO that references
+ * process local memory and thus need to fall back to executing IO
+ * synchronously for IOs with this flag set.
+ *
+ * Required for correctness.
+ */
+ PGAIO_HF_REFERENCES_LOCAL = 1 << 1,
+
+ /*
+ * Hint that IO will be executed synchronously.
+ *
+ * This can make it a bit cheaper to execute synchronous IO via the AIO
+ * interface, to avoid needing an AIO and non-AIO version of code.
+ *
+ * Advantageous to set, if applicable, but not required for correctness.
+ */
+ PGAIO_HF_SYNCHRONOUS = 1 << 0,
+
+ /*
+ * IO is using buffered IO, used to control heuristic in some IO methods.
+ *
+ * Advantageous to set, if applicable, but not required for correctness.
+ */
+ PGAIO_HF_BUFFERED = 1 << 2,
+} PgAioHandleFlags;
+
+/*
+ * The IO operations supported by the AIO subsystem.
+ *
+ * This could be in aio_internal.h, as it is not pubicly referenced, but
+ * PgAioOpData currently *does* need to be public, therefore keeping this
+ * public seems to make sense.
+ */
+typedef enum PgAioOp
+{
+ /* intentionally the zero value, to help catch zeroed memory etc */
+ PGAIO_OP_INVALID = 0,
+
+ PGAIO_OP_READV,
+ PGAIO_OP_WRITEV,
+
+ /**
+ * In the near term we'll need at least:
+ * - fsync / fdatasync
+ * - flush_range
+ *
+ * Eventually we'll additionally want at least:
+ * - send
+ * - recv
+ * - accept
+ **/
+} PgAioOp;
+
+#define PGAIO_OP_COUNT (PGAIO_OP_WRITEV + 1)
+
+
+/*
+ * On what is IO being performed?
+ *
+ * PgAioTargetID specific behaviour should be implemented in
+ * aio_target.c.
+ */
+typedef enum PgAioTargetID
+{
+ /* intentionally the zero value, to help catch zeroed memory etc */
+ PGAIO_TID_INVALID = 0,
+} PgAioTargetID;
+
+#define PGAIO_TID_COUNT (PGAIO_TID_INVALID + 1)
+
+
+/*
+ * Data necessary for support IO operations (see PgAioOp).
+ *
+ * NB: Note that the FDs in here may *not* be relied upon for re-issuing
+ * requests (e.g. for partial reads/writes) - the FD might be from another
+ * process, or closed since. That's not a problem for IOs waiting to be issued
+ * only because the queue is flushed when closing an FD.
+ */
+typedef union
+{
+ struct
+ {
+ int fd;
+ uint16 iov_length;
+ uint64 offset;
+ } read;
+
+ struct
+ {
+ int fd;
+ uint16 iov_length;
+ uint64 offset;
+ } write;
+} PgAioOpData;
+
+
+/*
+ * Information the object that IO is executed on. Mostly callbacks that
+ * operate on PgAioTargetData.
+ *
+ * typedef is in aio_types.h
+ */
+struct PgAioTargetInfo
+{
+ void (*reopen) (PgAioHandle *ioh);
+
+ char *(*describe_identity) (const PgAioTargetData *sd);
+
+ const char *name;
+};
+
+
+/*
+ * IDs for callbacks that can be registered on an IO.
+ *
+ * Callbacks are identified by an ID rather than a function pointer. There are
+ * two main reasons:
+ *
+ * 1) Memory within PgAioHandle is precious, due to the number of PgAioHandle
+ * structs in pre-allocated shared memory.
+ *
+ * 2) Due to EXEC_BACKEND function pointers are not necessarily stable between
+ * different backends, therefore function pointers cannot directly be in
+ * shared memory.
+ *
+ * Without 2), we could fairly easily allow to add new callbacks, by filling a
+ * ID->pointer mapping table on demand. In the presence of 2 that's still
+ * doable, but harder, because every process has to re-register the pointers
+ * so that a local ID->"backend local pointer" mapping can be maintained.
+ */
+typedef enum PgAioHandleCallbackID
+{
+ PGAIO_HCB_INVALID,
+} PgAioHandleCallbackID;
+
+
+typedef void (*PgAioHandleCallbackStage) (PgAioHandle *ioh, uint8 cb_flags);
+typedef PgAioResult (*PgAioHandleCallbackComplete) (PgAioHandle *ioh, PgAioResult prior_result, uint8 cb_flags);
+typedef void (*PgAioHandleCallbackReport) (PgAioResult result, const PgAioTargetData *target_data, int elevel);
+
+/* typedef is in aio_types.h */
+struct PgAioHandleCallbacks
+{
+ /*
+ * Prepare resources affected by the IO for execution. This could e.g.
+ * include moving ownership of buffer pins to the AIO subsystem.
+ */
+ PgAioHandleCallbackStage stage;
+
+ /*
+ * Update the state of resources affected by the IO to reflect completion
+ * of the IO. This could e.g. include updating shared buffer state to
+ * signal the IO has finished.
+ *
+ * The _shared suffix indicates that this is executed by the backend that
+ * completed the IO, which may or may not be the backend that issued the
+ * IO. Obviously the callback thus can only modify resources in shared
+ * memory.
+ *
+ * The latest registered callback is called first. This allows
+ * higher-level code to register callbacks that can rely on callbacks
+ * registered by lower-level code to already have been executed.
+ *
+ * NB: This is called in a critical section. Errors can be signalled by
+ * the callback's return value, it's the responsibility of the IO's issuer
+ * to react appropriately.
+ */
+ PgAioHandleCallbackComplete complete_shared;
+
+ /*
+ * Like complete_shared, except called in the issuing backend.
+ *
+ * This variant of the completion callback is useful when backend-local
+ * state has to be updated to reflect the IO's completion. E.g. a
+ * temporary buffer's BufferDesc isn't accessible in complete_shared.
+ *
+ * Local callbacks are only called after complete_shared for all
+ * registered callbacks has been called.
+ */
+ PgAioHandleCallbackComplete complete_local;
+
+ /*
+ * Report the result of an IO operation. This is e.g. used to raise an
+ * error after an IO failed at the appropriate time (i.e. not when the IO
+ * failed, but under control of the code that issued the IO).
+ */
+ PgAioHandleCallbackReport report;
+};
+
+
+
+/*
+ * How many callbacks can be registered for one IO handle. Currently we only
+ * need two, but it's not hard to imagine needing a few more.
+ */
+#define PGAIO_HANDLE_MAX_CALLBACKS 4
+
+
+
+/* --------------------------------------------------------------------------------
+ * IO Handles
+ * --------------------------------------------------------------------------------
+ */
+
+/* functions in aio.c */
+struct ResourceOwnerData;
+extern PgAioHandle *pgaio_io_acquire(struct ResourceOwnerData *resowner, PgAioReturn *ret);
+extern PgAioHandle *pgaio_io_acquire_nb(struct ResourceOwnerData *resowner, PgAioReturn *ret);
+
+extern void pgaio_io_release(PgAioHandle *ioh);
struct dlist_node;
extern void pgaio_io_release_resowner(struct dlist_node *ioh_node, bool on_error);
+extern void pgaio_io_set_flag(PgAioHandle *ioh, PgAioHandleFlags flag);
+
+extern int pgaio_io_get_id(PgAioHandle *ioh);
+extern ProcNumber pgaio_io_get_owner(PgAioHandle *ioh);
+
+extern void pgaio_io_get_wref(PgAioHandle *ioh, PgAioWaitRef *iow);
+
+/* functions in aio_io.c */
+struct iovec;
+extern int pgaio_io_get_iovec(PgAioHandle *ioh, struct iovec **iov);
+
+extern PgAioOpData *pgaio_io_get_op_data(PgAioHandle *ioh);
+
+extern void pgaio_io_prep_readv(PgAioHandle *ioh,
+ int fd, int iovcnt, uint64 offset);
+extern void pgaio_io_prep_writev(PgAioHandle *ioh,
+ int fd, int iovcnt, uint64 offset);
+
+/* functions in aio_target.c */
+extern void pgaio_io_set_target(PgAioHandle *ioh, PgAioTargetID targetid);
+extern bool pgaio_io_has_target(PgAioHandle *ioh);
+extern PgAioTargetData *pgaio_io_get_target_data(PgAioHandle *ioh);
+extern char *pgaio_io_get_target_description(PgAioHandle *ioh);
+
+/* functions in aio_callback.c */
+extern void pgaio_io_register_callbacks(PgAioHandle *ioh, PgAioHandleCallbackID cb_id,
+ uint8 cb_data);
+extern void pgaio_io_set_handle_data_64(PgAioHandle *ioh, uint64 *data, uint8 len);
+extern void pgaio_io_set_handle_data_32(PgAioHandle *ioh, uint32 *data, uint8 len);
+extern uint64 *pgaio_io_get_handle_data(PgAioHandle *ioh, uint8 *len);
+
+
+
+/* --------------------------------------------------------------------------------
+ * IO Wait References
+ * --------------------------------------------------------------------------------
+ */
+
+extern void pgaio_wref_clear(PgAioWaitRef *iow);
+extern bool pgaio_wref_valid(PgAioWaitRef *iow);
+extern int pgaio_wref_get_id(PgAioWaitRef *iow);
+
+extern void pgaio_wref_wait(PgAioWaitRef *iow);
+extern bool pgaio_wref_check_done(PgAioWaitRef *iow);
+
+
+
+/* --------------------------------------------------------------------------------
+ * IO Result
+ * --------------------------------------------------------------------------------
+ */
+
+extern void pgaio_result_report(PgAioResult result, const PgAioTargetData *target_data,
+ int elevel);
+
+
+
+/* --------------------------------------------------------------------------------
+ * Actions on multiple IOs.
+ * --------------------------------------------------------------------------------
+ */
+
+extern void pgaio_enter_batchmode(void);
+extern void pgaio_exit_batchmode(void);
+extern void pgaio_submit_staged(void);
+extern bool pgaio_have_staged(void);
+
+
+
+/* --------------------------------------------------------------------------------
+ * Other
+ * --------------------------------------------------------------------------------
+ */
+
+extern void pgaio_closing_fd(int fd);
+
+
/* GUCs */
extern PGDLLIMPORT int io_method;
diff --git a/src/include/storage/aio_internal.h b/src/include/storage/aio_internal.h
new file mode 100644
index 00000000000..e42c3dcb06c
--- /dev/null
+++ b/src/include/storage/aio_internal.h
@@ -0,0 +1,395 @@
+/*-------------------------------------------------------------------------
+ *
+ * aio_internal.h
+ * AIO related declarations that should only be used by the AIO subsystem
+ * internally.
+ *
+ *
+ * Portions Copyright (c) 1996-2025, PostgreSQL Global Development Group
+ * Portions Copyright (c) 1994, Regents of the University of California
+ *
+ * src/include/storage/aio_internal.h
+ *
+ *-------------------------------------------------------------------------
+ */
+#ifndef AIO_INTERNAL_H
+#define AIO_INTERNAL_H
+
+
+#include "lib/ilist.h"
+#include "port/pg_iovec.h"
+#include "storage/aio.h"
+#include "storage/condition_variable.h"
+
+
+/*
+ * The maximum number of IOs that can be batch submitted at once.
+ */
+#define PGAIO_SUBMIT_BATCH_SIZE 32
+
+
+
+/*
+ * State machine for handles. With some exceptions, noted below, handles move
+ * linearly through all states.
+ *
+ * State changes should all go through pgaio_io_update_state().
+ */
+typedef enum PgAioHandleState
+{
+ /* not in use */
+ PGAIO_HS_IDLE = 0,
+
+ /*
+ * Returned by pgaio_io_acquire(). The next state is either DEFINED (if
+ * pgaio_io_prep_*() is called), or IDLE (if pgaio_io_release() is
+ * called).
+ */
+ PGAIO_HS_HANDED_OUT,
+
+ /*
+ * pgaio_io_prep_*() has been called, but IO is not yet staged. At this
+ * point the handle has all the information for the IO to be executed.
+ */
+ PGAIO_HS_DEFINED,
+
+ /*
+ * stage() callbacks have been called, handle ready to be submitted for
+ * execution. Unless in batchmode (see c.f. pgaio_enter_batchmode()), the
+ * IO will be submitted immediately after.
+ */
+ PGAIO_HS_STAGED,
+
+ /* IO has been submitted to the IO method for execution */
+ PGAIO_HS_SUBMITTED,
+
+ /* IO finished, but result has not yet been processed */
+ PGAIO_HS_COMPLETED_IO,
+
+ /*
+ * IO completed, shared completion has been called.
+ *
+ * If the IO completion occurs in the issuing backend, local callbacks
+ * will immediately be called. Otherwise the handle stays in
+ * COMPLETED_SHARED until the issuing backend waits for the completion of
+ * the IO.
+ */
+ PGAIO_HS_COMPLETED_SHARED,
+
+ /*
+ * IO completed, local completion has been called.
+ *
+ * After this the handle will be made reusable and go into IDLE state.
+ */
+ PGAIO_HS_COMPLETED_LOCAL,
+} PgAioHandleState;
+
+
+struct ResourceOwnerData;
+
+/* typedef is in aio_types.h */
+struct PgAioHandle
+{
+ /* all state updates should go through pgaio_io_update_state() */
+ PgAioHandleState state:8;
+
+ /* what are we operating on */
+ PgAioTargetID target:8;
+
+ /* which IO operation */
+ PgAioOp op:8;
+
+ /* bitfield of PgAioHandleFlags */
+ uint8 flags;
+
+ uint8 num_callbacks;
+
+ /* using the proper type here would use more space */
+ uint8 callbacks[PGAIO_HANDLE_MAX_CALLBACKS];
+
+ /* data forwarded to each callback */
+ uint8 callbacks_data[PGAIO_HANDLE_MAX_CALLBACKS];
+
+ /*
+ * Length of data associated with handle using
+ * pgaio_io_set_handle_data_*().
+ */
+ uint8 handle_data_len;
+
+ /* XXX: could be optimized out with some pointer math */
+ int32 owner_procno;
+
+ /* raw result of the IO operation */
+ int32 result;
+
+ /**
+ * In which list the handle is registered, depends on the state:
+ * - IDLE, in per-backend list
+ * - HANDED_OUT - not in a list
+ * - DEFINED - in per-backend staged list
+ * - STAGED - in per-backend staged list
+ * - SUBMITTED - in issuer's in_flight list
+ * - COMPLETED_IO - in issuer's in_flight list
+ * - COMPLETED_SHARED - in issuer's in_flight list
+ **/
+ dlist_node node;
+
+ struct ResourceOwnerData *resowner;
+ dlist_node resowner_node;
+
+ /* incremented every time the IO handle is reused */
+ uint64 generation;
+
+ /*
+ * To wait for the IO to complete other backends can wait on this CV. Note
+ * that, if in SUBMITTED state, a waiter first needs to check if it needs
+ * to do work via IoMethodOps->wait_one().
+ */
+ ConditionVariable cv;
+
+ /* result of shared callback, passed to issuer callback */
+ PgAioResult distilled_result;
+
+ /*
+ * Index into PgAioCtl->iovecs and PgAioCtl->handle_data.
+ *
+ * At the moment there's no need to differentiate between the two, but
+ * that won't necessarily stay that way.
+ */
+ uint32 iovec_off;
+
+ /*
+ * If not NULL, this memory location will be updated with information
+ * about the IOs completion iff the issuing backend learns about the IOs
+ * completion.
+ */
+ PgAioReturn *report_return;
+
+ /* Data necessary for the IO to be performed */
+ PgAioOpData op_data;
+
+ /*
+ * Data necessary to identify the object undergoing IO to higher-level
+ * code. Needs to be sufficient to allow another backend to reopen the
+ * file.
+ */
+ PgAioTargetData target_data;
+};
+
+
+typedef struct PgAioBackend
+{
+ /* index into PgAioCtl->io_handles */
+ uint32 io_handle_off;
+
+ /* IO Handles that currently are not used */
+ dclist_head idle_ios;
+
+ /*
+ * Only one IO may be returned by pgaio_io_acquire()/pgaio_io_acquire()
+ * without having been either defined (by actually associating it with IO)
+ * or by released (with pgaio_io_release()). This restriction is necessary
+ * to guarantee that we always can acquire an IO. ->handed_out_io is used
+ * to enforce that rule.
+ */
+ PgAioHandle *handed_out_io;
+
+ /* Are we currently in batchmode? See pgaio_enter_batchmode(). */
+ bool in_batchmode;
+
+ /*
+ * IOs that are defined, but not yet submitted.
+ */
+ uint16 num_staged_ios;
+ PgAioHandle *staged_ios[PGAIO_SUBMIT_BATCH_SIZE];
+
+ /*
+ * List of in-flight IOs. Also contains IOs that aren't strictly speaking
+ * in-flight anymore, but have been waited-for and completed by another
+ * backend. Once this backend sees such an IO it'll be reclaimed.
+ *
+ * The list is ordered by submission time, with more recently submitted
+ * IOs being appended at the end.
+ */
+ dclist_head in_flight_ios;
+} PgAioBackend;
+
+
+typedef struct PgAioCtl
+{
+ int backend_state_count;
+ PgAioBackend *backend_state;
+
+ /*
+ * Array of iovec structs. Each iovec is owned by a specific backend. The
+ * allocation is in PgAioCtl to allow the maximum number of iovecs for
+ * individual IOs to be configurable with PGC_POSTMASTER GUC.
+ */
+ uint32 iovec_count;
+ struct iovec *iovecs;
+
+ /*
+ * For, e.g., an IO covering multiple buffers in shared / temp buffers, we
+ * need to get Buffer IDs during completion to be able to change the
+ * BufferDesc state accordingly. This space can be used to store e.g.
+ * Buffer IDs. Note that the actual iovec might be shorter than this,
+ * because we combine neighboring pages into one larger iovec entry.
+ */
+ uint64 *handle_data;
+
+ uint32 io_handle_count;
+ PgAioHandle *io_handles;
+} PgAioCtl;
+
+
+
+/*
+ * Callbacks used to implement an IO method.
+ */
+typedef struct IoMethodOps
+{
+ /* global initialization */
+
+ /*
+ * Amount of additional shared memory to reserve for the io_method. Called
+ * just like a normal ipci.c style *Size() function. Optional.
+ */
+ size_t (*shmem_size) (void);
+
+ /*
+ * Initialize shared memory. First time is true if AIO's shared memory was
+ * just initialized, false otherwise. Optional.
+ */
+ void (*shmem_init) (bool first_time);
+
+ /*
+ * Per-backend initialization. Optional.
+ */
+ void (*init_backend) (void);
+
+
+ /* handling of IOs */
+
+ /* optional */
+ bool (*needs_synchronous_execution) (PgAioHandle *ioh);
+
+ /*
+ * Start executing passed in IOs.
+ *
+ * Will not be called if ->needs_synchronous_execution() returned true.
+ *
+ * num_staged_ios is <= PGAIO_SUBMIT_BATCH_SIZE.
+ *
+ * Always called in a critical section.
+ */
+ int (*submit) (uint16 num_staged_ios, PgAioHandle **staged_ios);
+
+ /*
+ * Wait for the IO to complete. Optional.
+ *
+ * If not provided, it needs to be guaranteed that the IO method calls
+ * pgaio_io_process_completion() without further interaction by the
+ * issuing backend.
+ */
+ void (*wait_one) (PgAioHandle *ioh,
+ uint64 ref_generation);
+} IoMethodOps;
+
+
+/* aio.c */
+extern bool pgaio_io_was_recycled(PgAioHandle *ioh, uint64 ref_generation, PgAioHandleState *state);
+extern void pgaio_io_stage(PgAioHandle *ioh, PgAioOp op);
+extern void pgaio_io_process_completion(PgAioHandle *ioh, int result);
+extern void pgaio_io_prepare_submit(PgAioHandle *ioh);
+extern bool pgaio_io_needs_synchronous_execution(PgAioHandle *ioh);
+extern const char *pgaio_io_get_state_name(PgAioHandle *ioh);
+const char *pgaio_result_status_string(PgAioResultStatus rs);
+extern void pgaio_shutdown(int code, Datum arg);
+
+/* aio_callback.c */
+extern void pgaio_io_call_stage(PgAioHandle *ioh);
+extern void pgaio_io_call_complete_shared(PgAioHandle *ioh);
+extern void pgaio_io_call_complete_local(PgAioHandle *ioh);
+
+/* aio_io.c */
+extern void pgaio_io_perform_synchronously(PgAioHandle *ioh);
+extern const char *pgaio_io_get_op_name(PgAioHandle *ioh);
+
+/* aio_target.c */
+extern bool pgaio_io_can_reopen(PgAioHandle *ioh);
+extern void pgaio_io_reopen(PgAioHandle *ioh);
+extern const char *pgaio_io_get_target_name(PgAioHandle *ioh);
+
+
+/*
+ * The AIO subsystem has fairly verbose debug logging support. This can be
+ * enabled/disabled at build time. The reason for this is that
+ * a) the verbosity can make debugging things on higher levels hard
+ * b) even if logging can be skipped due to elevel checks, it still causes a
+ * measurable slowdown
+ *
+ * XXX: This likely should be eventually be disabled by default, at least in
+ * non-assert builds.
+ */
+#define PGAIO_VERBOSE 1
+
+/*
+ * Simple ereport() wrapper that only logs if PGAIO_VERBOSE is defined.
+ *
+ * This intentionally still compiles the code, guarded by a constant if (0),
+ * if verbose logging is disabled, to make it less likely that debug logging
+ * is silently broken.
+ *
+ * The current definition requires passing at least one argument.
+ */
+#define pgaio_debug(elevel, msg, ...) \
+ do { \
+ if (PGAIO_VERBOSE) \
+ ereport(elevel, \
+ errhidestmt(true), errhidecontext(true), \
+ errmsg_internal(msg, \
+ __VA_ARGS__)); \
+ } while(0)
+
+/*
+ * Simple ereport() wrapper. Note that the definition requires passing at
+ * least one argument.
+ */
+#define pgaio_debug_io(elevel, ioh, msg, ...) \
+ pgaio_debug(elevel, "io %-10d|op %-5s|target %-4s|state %-16s: " msg, \
+ pgaio_io_get_id(ioh), \
+ pgaio_io_get_op_name(ioh), \
+ pgaio_io_get_target_name(ioh), \
+ pgaio_io_get_state_name(ioh), \
+ __VA_ARGS__)
+
+
+#ifdef USE_INJECTION_POINTS
+
+extern void pgaio_io_call_inj(PgAioHandle *ioh, const char *injection_point);
+
+/* just for use in tests, from within injection points */
+extern PgAioHandle *pgaio_inj_io_get(void);
+
+#else
+
+#define pgaio_io_call_inj(ioh, injection_point) (void) 0
+
+/*
+ * no fallback for pgaio_inj_io_get, all code using injection points better be
+ * guarded by USE_INJECTION_POINTS.
+ */
+
+#endif
+
+
+/* Declarations for the tables of function pointers exposed by each IO method. */
+extern PGDLLIMPORT const IoMethodOps pgaio_sync_ops;
+
+extern PGDLLIMPORT const IoMethodOps *pgaio_method_ops;
+extern PGDLLIMPORT PgAioCtl *pgaio_ctl;
+extern PGDLLIMPORT PgAioBackend *pgaio_my_backend;
+
+
+
+#endif /* AIO_INTERNAL_H */
diff --git a/src/include/storage/aio_types.h b/src/include/storage/aio_types.h
new file mode 100644
index 00000000000..a5cc658efbd
--- /dev/null
+++ b/src/include/storage/aio_types.h
@@ -0,0 +1,117 @@
+/*-------------------------------------------------------------------------
+ *
+ * aio_types.h
+ * AIO related types that are useful to include separately, to reduce the
+ * "include burden".
+ *
+ *
+ * Portions Copyright (c) 1996-2025, PostgreSQL Global Development Group
+ * Portions Copyright (c) 1994, Regents of the University of California
+ *
+ * src/include/storage/aio_types.h
+ *
+ *-------------------------------------------------------------------------
+ */
+#ifndef AIO_TYPES_H
+#define AIO_TYPES_H
+
+#include "storage/block.h"
+#include "storage/relfilelocator.h"
+
+
+typedef struct PgAioHandle PgAioHandle;
+typedef struct PgAioHandleCallbacks PgAioHandleCallbacks;
+typedef struct PgAioTargetInfo PgAioTargetInfo;
+
+/*
+ * A reference to an IO that can be used to wait for the IO (using
+ * pgaio_wref_wait()) to complete.
+ *
+ * These can be passed across process boundaries.
+ */
+typedef struct PgAioWaitRef
+{
+ /* internal ID identifying the specific PgAioHandle */
+ uint32 aio_index;
+
+ /*
+ * IO handles are reused. To detect if a handle was reused, and thereby
+ * avoid unnecessarily waiting for a newer IO, each time the handle is
+ * reused a generation number is increased.
+ *
+ * To avoid requiring alignment sufficient for an int64, split the
+ * generation into two.
+ */
+ uint32 generation_upper;
+ uint32 generation_lower;
+} PgAioWaitRef;
+
+
+/*
+ * Information identifying what the IO is being performed on.
+ *
+ * This needs sufficient information to
+ *
+ * a) Reopen the file for the IO if the IO is executed in a context that
+ * cannot use the FD provided initially (e.g. because the IO is executed in
+ * a worker process).
+ *
+ * b) Describe the object the IO is performed on in log / error messages.
+ */
+typedef union PgAioTargetData
+{
+ /* just as an example placeholder for later */
+ struct
+ {
+ uint32 queue_id;
+ } wal;
+} PgAioTargetData;
+
+
+/*
+ * The status of an AIO operation.
+ */
+typedef enum PgAioResultStatus
+{
+ ARS_UNKNOWN, /* not yet completed / uninitialized */
+ ARS_OK,
+ ARS_PARTIAL, /* did not fully succeed, but no error */
+ ARS_ERROR,
+} PgAioResultStatus;
+
+
+/*
+ * Result of IO operation, visible only to the initiator of IO.
+ */
+typedef struct PgAioResult
+{
+ /*
+ * This is of type PgAioHandleCallbackID, but can't use a bitfield of an
+ * enum, because some compilers treat enums as signed.
+ */
+ uint32 id:8;
+
+ /* of type PgAioResultStatus, see above */
+ uint32 status:2;
+
+ /* meaning defined by callback->error */
+ uint32 error_data:22;
+
+ int32 result;
+} PgAioResult;
+
+
+/*
+ * Combination of PgAioResult with minimal metadata about the IO.
+ *
+ * Contains sufficient information to be able, in case the IO [partially]
+ * fails, to log/raise an error under control of the IO issuing code.
+ */
+typedef struct PgAioReturn
+{
+ PgAioResult result;
+ PgAioTargetData target_data;
+} PgAioReturn;
+
+
+#endif /* AIO_TYPES_H */
diff --git a/src/backend/storage/aio/Makefile b/src/backend/storage/aio/Makefile
index eaeaeeee8e3..89f821ea7e1 100644
--- a/src/backend/storage/aio/Makefile
+++ b/src/backend/storage/aio/Makefile
@@ -10,7 +10,11 @@ include $(top_builddir)/src/Makefile.global
OBJS = \
aio.o \
+ aio_callback.o \
aio_init.o \
+ aio_io.o \
+ aio_target.o \
+ method_sync.o \
read_stream.o
include $(top_srcdir)/src/backend/common.mk
diff --git a/src/backend/storage/aio/aio.c b/src/backend/storage/aio/aio.c
index 828a94efdc3..7eb0bd6f12d 100644
--- a/src/backend/storage/aio/aio.c
+++ b/src/backend/storage/aio/aio.c
@@ -3,6 +3,28 @@
* aio.c
* AIO - Core Logic
*
+ * For documentation about how AIO works on a higher level, including a
+ * schematic example, see README.md.
+ *
+ *
+ * AIO is a complicated subsystem. To keep things navigable it is split across
+ * a number of files:
+ *
+ * - method_*.c - different ways of executing AIO (e.g. worker process)
+ *
+ * - aio_target.c - IO on different kinds of targets
+ *
+ * - aio_io.c - method-independent code for specific IO ops (e.g. readv)
+ *
+ * - aio_callback.c - callbacks at IO operation lifecycle events
+ *
+ * - aio_init.c - per-server and per-backend initialization
+ *
+ * - aio.c - all other topics
+ *
+ * - read_stream.c - helper for reading buffered relation data
+ *
+ *
* Portions Copyright (c) 1996-2025, PostgreSQL Global Development Group
* Portions Copyright (c) 1994, Regents of the University of California
*
@@ -15,10 +37,28 @@
#include "postgres.h"
#include "lib/ilist.h"
+#include "miscadmin.h"
+#include "port/atomics.h"
#include "storage/aio.h"
+#include "storage/aio_internal.h"
#include "storage/aio_subsys.h"
#include "utils/guc.h"
#include "utils/guc_hooks.h"
+#include "utils/resowner.h"
+#include "utils/wait_event_types.h"
+
+#ifdef USE_INJECTION_POINTS
+#include "utils/injection_point.h"
+#endif
+
+
+static inline void pgaio_io_update_state(PgAioHandle *ioh, PgAioHandleState new_state);
+static void pgaio_io_reclaim(PgAioHandle *ioh);
+static void pgaio_io_resowner_register(PgAioHandle *ioh);
+static void pgaio_io_wait_for_free(void);
+static PgAioHandle *pgaio_io_from_wref(PgAioWaitRef *iow, uint64 *ref_generation);
+static const char *pgaio_io_state_get_name(PgAioHandleState s);
+static void pgaio_io_wait(PgAioHandle *ioh, uint64 ref_generation);
/* Options for io_method. */
@@ -31,7 +71,182 @@ const struct config_enum_entry io_method_options[] = {
int io_method = DEFAULT_IO_METHOD;
int io_max_concurrency = -1;
+/* global control for AIO */
+PgAioCtl *pgaio_ctl;
+/* current backend's per-backend state */
+PgAioBackend *pgaio_my_backend;
+
+
+static const IoMethodOps *const pgaio_method_ops_table[] = {
+ [IOMETHOD_SYNC] = &pgaio_sync_ops,
+};
+
+/* callbacks for the configured io_method, set by assign_io_method */
+const IoMethodOps *pgaio_method_ops;
+
+
+/*
+ * Currently there's no infrastructure to pass arguments to injection points,
+ * so we instead set this up for the duration of the injection point
+ * invocation. See pgaio_io_call_inj().
+ */
+#ifdef USE_INJECTION_POINTS
+static PgAioHandle *pgaio_inj_cur_handle;
+#endif
+
+
+
+/* --------------------------------------------------------------------------------
+ * Public Functions related to PgAioHandle
+ * --------------------------------------------------------------------------------
+ */
+
+/*
+ * Acquire an AioHandle, waiting for IO completion if necessary.
+ *
+ * Each backend can only have one AIO handle that has been "handed out" to
+ * code, but not yet submitted or released. This restriction is necessary to
+ * ensure that it is possible for code to wait for an unused handle by waiting
+ * for in-flight IO to complete. There is a limited number of handles in each
+ * backend, if multiple handles could be handed out without being submitted,
+ * waiting for all in-flight IO to complete would not guarantee that handles
+ * free up.
+ *
+ * It is cheap to acquire an IO handle, unless all handles are in use. In that
+ * case this function waits for the oldest IO to complete. In case that is not
+ * desirable, see pgaio_io_acquire_nb().
+ *
+ * If a handle was acquired but then does not turn out to be needed,
+ * e.g. because pgaio_io_acquire() is called before starting an IO in a
+ * critical section, the handle needs to be released with pgaio_io_release().
+ *
+ *
+ * To react to the completion of the IO as soon as it is known to have
+ * completed, callbacks can be registered with pgaio_io_register_callbacks().
+ *
+ * To actually execute IO using the returned handle, the pgaio_io_prep_*()
+ * family of functions is used. In many cases the pgaio_io_prep_*() call will
+ * not be done directly by code that acquired the handle, but by lower level
+ * code that gets passed the handle. E.g. if code in bufmgr.c wants to perform
+ * AIO, it typically will pass the handle to smgr.c, which will pass it on to
+ * md.c, on to fd.c, which then finally calls pgaio_io_prep_*(). This
+ * forwarding allows the various layers to react to the IO's completion by
+ * registering callbacks. These callbacks in turn can translate a lower
+ * layer's result into a result understandable by a higher layer.
+ *
+ * During pgaio_io_prep_*() the IO is staged (i.e. prepared for execution but
+ * not submitted to the kernel). Unless in batchmode
+ * (c.f. pgaio_enter_batchmode()) the IO will also have been submitted for
+ * execution. Note that, whether in batchmode or not, the IO might even
+ * complete before the functions return.
+ *
+ * After pgaio_io_prep_*() the AioHandle is "consumed" and may not be
+ * referenced by the IO issuing code. To e.g. wait for IO, references to the
+ * IO can be established with pgaio_io_get_wref() *before* pgaio_io_prep_*()
+ * is called. pgaio_wref_wait() can be used to wait for the IO to complete.
+ *
+ *
+ * To know if the IO [partially] succeeded or failed, a PgAioReturn * can be
+ * passed to pgaio_io_acquire(). Once the issuing backend has called
+ * pgaio_wref_wait(), the PgAioReturn contains information about whether the
+ * operation succeeded and details about the first failure, if any. The error
+ * can be raised / logged with pgaio_result_report().
+ *
+ * The lifetime of the memory pointed to be *ret needs to be at least as long
+ * as the passed in resowner. If the resowner releases resources before the IO
+ * completes (typically due to an error), the reference to *ret will be
+ * cleared. In case of resowner cleanup *ret will not be updated with the
+ * results of the IO operation.
+ */
+PgAioHandle *
+pgaio_io_acquire(struct ResourceOwnerData *resowner, PgAioReturn *ret)
+{
+ PgAioHandle *h;
+
+ while (true)
+ {
+ h = pgaio_io_acquire_nb(resowner, ret);
+
+ if (h != NULL)
+ return h;
+
+ /*
+ * Evidently all handles by this backend are in use. Just wait for
+ * some to complete.
+ */
+ pgaio_io_wait_for_free();
+ }
+}
+
+/*
+ * Acquire an AioHandle, returning NULL if no handles are free.
+ *
+ * See pgaio_io_acquire(). The only difference is that this function will return
+ * NULL if there are no idle handles, instead of blocking.
+ */
+PgAioHandle *
+pgaio_io_acquire_nb(struct ResourceOwnerData *resowner, PgAioReturn *ret)
+{
+ if (pgaio_my_backend->num_staged_ios >= PGAIO_SUBMIT_BATCH_SIZE)
+ {
+ Assert(pgaio_my_backend->num_staged_ios == PGAIO_SUBMIT_BATCH_SIZE);
+ pgaio_submit_staged();
+ }
+
+ if (pgaio_my_backend->handed_out_io)
+ {
+ ereport(ERROR,
+ errmsg("API violation: Only one IO can be handed out"));
+ }
+
+ if (!dclist_is_empty(&pgaio_my_backend->idle_ios))
+ {
+ dlist_node *ion = dclist_pop_head_node(&pgaio_my_backend->idle_ios);
+ PgAioHandle *ioh = dclist_container(PgAioHandle, node, ion);
+
+ Assert(ioh->state == PGAIO_HS_IDLE);
+ Assert(ioh->owner_procno == MyProcNumber);
+
+ pgaio_io_update_state(ioh, PGAIO_HS_HANDED_OUT);
+ pgaio_my_backend->handed_out_io = ioh;
+
+ if (resowner)
+ pgaio_io_resowner_register(ioh);
+
+ if (ret)
+ {
+ ioh->report_return = ret;
+ ret->result.status = ARS_UNKNOWN;
+ }
+
+ return ioh;
+ }
+
+ return NULL;
+}
+
+/*
+ * Release IO handle that turned out to not be required.
+ *
+ * See pgaio_io_acquire() for more details.
+ */
+void
+pgaio_io_release(PgAioHandle *ioh)
+{
+ if (ioh == pgaio_my_backend->handed_out_io)
+ {
+ Assert(ioh->state == PGAIO_HS_HANDED_OUT);
+ Assert(ioh->resowner);
+
+ pgaio_my_backend->handed_out_io = NULL;
+ pgaio_io_reclaim(ioh);
+ }
+ else
+ {
+ elog(ERROR, "release in unexpected state");
+ }
+}
/*
* Release IO handle during resource owner cleanup.
@@ -39,8 +254,792 @@ int io_max_concurrency = -1;
void
pgaio_io_release_resowner(dlist_node *ioh_node, bool on_error)
{
+ PgAioHandle *ioh = dlist_container(PgAioHandle, resowner_node, ioh_node);
+
+ Assert(ioh->resowner);
+
+ ResourceOwnerForgetAioHandle(ioh->resowner, &ioh->resowner_node);
+ ioh->resowner = NULL;
+
+ switch (ioh->state)
+ {
+ case PGAIO_HS_IDLE:
+ elog(ERROR, "unexpected");
+ break;
+ case PGAIO_HS_HANDED_OUT:
+ Assert(ioh == pgaio_my_backend->handed_out_io || pgaio_my_backend->handed_out_io == NULL);
+
+ if (ioh == pgaio_my_backend->handed_out_io)
+ {
+ pgaio_my_backend->handed_out_io = NULL;
+ if (!on_error)
+ elog(WARNING, "leaked AIO handle");
+ }
+
+ pgaio_io_reclaim(ioh);
+ break;
+ case PGAIO_HS_DEFINED:
+ case PGAIO_HS_STAGED:
+ if (!on_error)
+ elog(WARNING, "AIO handle was not submitted");
+ pgaio_submit_staged();
+ break;
+ case PGAIO_HS_SUBMITTED:
+ case PGAIO_HS_COMPLETED_IO:
+ case PGAIO_HS_COMPLETED_SHARED:
+ case PGAIO_HS_COMPLETED_LOCAL:
+ /* this is expected to happen */
+ break;
+ }
+
+ /*
+ * Need to unregister the reporting of the IO's result, the memory it's
+ * referencing likely has gone away.
+ */
+ if (ioh->report_return)
+ ioh->report_return = NULL;
}
+/*
+ * Add a [set of] flags to the IO.
+ *
+ * Note that this combines flags with already set flags, rather than set flags
+ * to explicitly the passed in parameters. This is to allow multiple callsites
+ * to set flags.
+ */
+void
+pgaio_io_set_flag(PgAioHandle *ioh, PgAioHandleFlags flag)
+{
+ Assert(ioh->state == PGAIO_HS_HANDED_OUT);
+
+ ioh->flags |= flag;
+}
+
+/*
+ * Returns an ID uniquely identifying the IO handle. This is only really
+ * useful for logging, as handles are reused across multiple IOs.
+ */
+int
+pgaio_io_get_id(PgAioHandle *ioh)
+{
+ Assert(ioh >= pgaio_ctl->io_handles &&
+ ioh < (pgaio_ctl->io_handles + pgaio_ctl->io_handle_count));
+ return ioh - pgaio_ctl->io_handles;
+}
+
+/*
+ * Return the ProcNumber for the process that can use an IO handle. The
+ * mapping from IO handles to PGPROCs is static, therefore this even works
+ * when the corresponding PGPROC is not in use.
+ */
+ProcNumber
+pgaio_io_get_owner(PgAioHandle *ioh)
+{
+ return ioh->owner_procno;
+}
+
+/*
+ * Return a wait reference for the IO. Only wait references can be used to
+ * wait for an IOs completion, as handles themselves can be reused after
+ * completion. See also the comment above pgaio_io_acquire().
+ */
+void
+pgaio_io_get_wref(PgAioHandle *ioh, PgAioWaitRef *iow)
+{
+ Assert(ioh->state == PGAIO_HS_HANDED_OUT ||
+ ioh->state == PGAIO_HS_DEFINED ||
+ ioh->state == PGAIO_HS_STAGED);
+ Assert(ioh->generation != 0);
+
+ iow->aio_index = ioh - pgaio_ctl->io_handles;
+ iow->generation_upper = (uint32) (ioh->generation >> 32);
+ iow->generation_lower = (uint32) ioh->generation;
+}
+
+
+
+/* --------------------------------------------------------------------------------
+ * Internal Functions related to PgAioHandle
+ * --------------------------------------------------------------------------------
+ */
+
+static inline void
+pgaio_io_update_state(PgAioHandle *ioh, PgAioHandleState new_state)
+{
+ pgaio_debug_io(DEBUG5, ioh,
+ "updating state to %s",
+ pgaio_io_state_get_name(new_state));
+
+ /*
+ * Ensure the changes signified by the new state are visible before the
+ * new state becomes visible.
+ */
+ pg_write_barrier();
+
+ ioh->state = new_state;
+}
+
+static void
+pgaio_io_resowner_register(PgAioHandle *ioh)
+{
+ Assert(!ioh->resowner);
+ Assert(CurrentResourceOwner);
+
+ ResourceOwnerRememberAioHandle(CurrentResourceOwner, &ioh->resowner_node);
+ ioh->resowner = CurrentResourceOwner;
+}
+
+/*
+ * Stage IO for execution and, if necessary, submit it immediately.
+ *
+ * Should only be called from pgaio_io_prep_*().
+ */
+void
+pgaio_io_stage(PgAioHandle *ioh, PgAioOp op)
+{
+ bool needs_synchronous;
+
+ Assert(ioh->state == PGAIO_HS_HANDED_OUT);
+ Assert(pgaio_io_has_target(ioh));
+
+ ioh->op = op;
+ ioh->result = 0;
+
+ pgaio_io_update_state(ioh, PGAIO_HS_DEFINED);
+
+ /* allow a new IO to be staged */
+ pgaio_my_backend->handed_out_io = NULL;
+
+ pgaio_io_call_stage(ioh);
+
+ pgaio_io_update_state(ioh, PGAIO_HS_STAGED);
+
+ /*
+ * Synchronous execution has to be executed, well, synchronously, so check
+ * that first.
+ */
+ needs_synchronous = pgaio_io_needs_synchronous_execution(ioh);
+
+ pgaio_debug_io(DEBUG3, ioh,
+ "prepared (synchronous: %d, in_batch: %d)",
+ needs_synchronous, pgaio_my_backend->in_batchmode);
+
+ if (!needs_synchronous)
+ {
+ pgaio_my_backend->staged_ios[pgaio_my_backend->num_staged_ios++] = ioh;
+ Assert(pgaio_my_backend->num_staged_ios <= PGAIO_SUBMIT_BATCH_SIZE);
+
+ /*
+ * Unless code explicitly opted into batching IOs, submit the IO
+ * immediately.
+ */
+ if (!pgaio_my_backend->in_batchmode)
+ pgaio_submit_staged();
+ }
+ else
+ {
+ pgaio_io_prepare_submit(ioh);
+ pgaio_io_perform_synchronously(ioh);
+ }
+}
+
+bool
+pgaio_io_needs_synchronous_execution(PgAioHandle *ioh)
+{
+ /*
+ * If the caller said to execute the IO synchronously, do so.
+ *
+ * XXX: We could optimize the logic when to execute synchronously by first
+ * checking if there are other IOs in flight and only synchronously
+ * executing if not. Unclear whether that'll be sufficiently common to be
+ * worth worrying about.
+ */
+ if (ioh->flags & PGAIO_HF_SYNCHRONOUS)
+ return true;
+
+ /* Check if the IO method requires synchronous execution of IO */
+ if (pgaio_method_ops->needs_synchronous_execution)
+ return pgaio_method_ops->needs_synchronous_execution(ioh);
+
+ return false;
+}
+
+/*
+ * Handle IO being processed by IO method.
+ *
+ * Should be called by IO methods / synchronous IO execution, just before the
+ * IO is performed.
+ */
+void
+pgaio_io_prepare_submit(PgAioHandle *ioh)
+{
+ pgaio_io_update_state(ioh, PGAIO_HS_SUBMITTED);
+
+ dclist_push_tail(&pgaio_my_backend->in_flight_ios, &ioh->node);
+}
+
+/*
+ * Handle IO getting completed by a method.
+ *
+ * Should be called by IO methods / synchronous IO execution, just after the
+ * IO has been performed.
+ *
+ * Expects to be called in a critical section. We expect IOs to be usable for
+ * WAL etc, which requires being able to execute completion callbacks in a
+ * critical section.
+ */
+void
+pgaio_io_process_completion(PgAioHandle *ioh, int result)
+{
+ Assert(ioh->state == PGAIO_HS_SUBMITTED);
+
+ Assert(CritSectionCount > 0);
+
+ ioh->result = result;
+
+ pgaio_io_update_state(ioh, PGAIO_HS_COMPLETED_IO);
+
+ pgaio_io_call_inj(ioh, "AIO_PROCESS_COMPLETION_BEFORE_SHARED");
+
+ pgaio_io_call_complete_shared(ioh);
+
+ pgaio_io_update_state(ioh, PGAIO_HS_COMPLETED_SHARED);
+
+ /* condition variable broadcast ensures state is visible before wakeup */
+ ConditionVariableBroadcast(&ioh->cv);
+
+ /* contains call to pgaio_io_call_complete_local() */
+ if (ioh->owner_procno == MyProcNumber)
+ pgaio_io_reclaim(ioh);
+}
+
+/*
+ * Has the IO completed and thus the IO handle been reused?
+ *
+ * This is useful when waiting for IO completion at a low level (e.g. in an IO
+ * method's ->wait_one() callback).
+ */
+bool
+pgaio_io_was_recycled(PgAioHandle *ioh, uint64 ref_generation, PgAioHandleState *state)
+{
+ *state = ioh->state;
+ pg_read_barrier();
+
+ return ioh->generation != ref_generation;
+}
+
+/*
+ * Wait for IO to complete. External code should never use this, outside of
+ * the AIO subsystem waits are only allowed via pgaio_wref_wait().
+ */
+static void
+pgaio_io_wait(PgAioHandle *ioh, uint64 ref_generation)
+{
+ PgAioHandleState state;
+ bool am_owner;
+
+ am_owner = ioh->owner_procno == MyProcNumber;
+
+ if (pgaio_io_was_recycled(ioh, ref_generation, &state))
+ return;
+
+ if (am_owner)
+ {
+ if (state != PGAIO_HS_SUBMITTED
+ && state != PGAIO_HS_COMPLETED_IO
+ && state != PGAIO_HS_COMPLETED_SHARED
+ && state != PGAIO_HS_COMPLETED_LOCAL)
+ {
+ elog(PANIC, "waiting for own IO in wrong state: %d",
+ state);
+ }
+ }
+
+ while (true)
+ {
+ if (pgaio_io_was_recycled(ioh, ref_generation, &state))
+ return;
+
+ switch (state)
+ {
+ case PGAIO_HS_IDLE:
+ case PGAIO_HS_HANDED_OUT:
+ elog(ERROR, "IO in wrong state: %d", state);
+ break;
+
+ case PGAIO_HS_SUBMITTED:
+
+ /*
+ * If we need to wait via the IO method, do so now. Don't
+ * check via the IO method if the issuing backend is executing
+ * the IO synchronously.
+ */
+ if (pgaio_method_ops->wait_one && !(ioh->flags & PGAIO_HF_SYNCHRONOUS))
+ {
+ pgaio_method_ops->wait_one(ioh, ref_generation);
+ continue;
+ }
+ /* fallthrough */
+
+ /* waiting for owner to submit */
+ case PGAIO_HS_DEFINED:
+ case PGAIO_HS_STAGED:
+ /* waiting for reaper to complete */
+ /* fallthrough */
+ case PGAIO_HS_COMPLETED_IO:
+ /* shouldn't be able to hit this otherwise */
+ Assert(IsUnderPostmaster);
+ /* ensure we're going to get woken up */
+ ConditionVariablePrepareToSleep(&ioh->cv);
+
+ while (!pgaio_io_was_recycled(ioh, ref_generation, &state))
+ {
+ if (state == PGAIO_HS_COMPLETED_SHARED ||
+ state == PGAIO_HS_COMPLETED_LOCAL)
+ break;
+ ConditionVariableSleep(&ioh->cv, WAIT_EVENT_AIO_IO_COMPLETION);
+ }
+
+ ConditionVariableCancelSleep();
+ break;
+
+ case PGAIO_HS_COMPLETED_SHARED:
+ case PGAIO_HS_COMPLETED_LOCAL:
+ /* see above */
+ if (am_owner)
+ pgaio_io_reclaim(ioh);
+ return;
+ }
+ }
+}
+
+/*
+ * Make IO handle ready to be reused after IO has completed or after the
+ * handle has been released without being used.
+ */
+static void
+pgaio_io_reclaim(PgAioHandle *ioh)
+{
+ /* This is only ok if it's our IO */
+ Assert(ioh->owner_procno == MyProcNumber);
+
+ /*
+ * It's a bit ugly, but right now the easiest place to put the execution
+ * of shared completion callbacks is this function, as we need to execute
+ * local callbacks just before reclaiming at multiple callsites.
+ */
+ if (ioh->state == PGAIO_HS_COMPLETED_SHARED)
+ {
+ pgaio_io_call_complete_local(ioh);
+ pgaio_io_update_state(ioh, PGAIO_HS_COMPLETED_LOCAL);
+ }
+
+ pgaio_debug_io(DEBUG4, ioh,
+ "reclaiming: distilled_result: (status %s, id %u, error_data %d), raw_result: %d",
+ pgaio_result_status_string(ioh->distilled_result.status),
+ ioh->distilled_result.id,
+ ioh->distilled_result.error_data,
+ ioh->result);
+
+ /* if the IO has been defined, we might need to do more work */
+ if (ioh->state != PGAIO_HS_HANDED_OUT)
+ {
+ dclist_delete_from(&pgaio_my_backend->in_flight_ios, &ioh->node);
+
+ if (ioh->report_return)
+ {
+ ioh->report_return->result = ioh->distilled_result;
+ ioh->report_return->target_data = ioh->target_data;
+ }
+ }
+
+ if (ioh->resowner)
+ {
+ ResourceOwnerForgetAioHandle(ioh->resowner, &ioh->resowner_node);
+ ioh->resowner = NULL;
+ }
+
+ Assert(!ioh->resowner);
+
+ ioh->op = PGAIO_OP_INVALID;
+ ioh->target = PGAIO_TID_INVALID;
+ ioh->flags = 0;
+ ioh->num_callbacks = 0;
+ ioh->handle_data_len = 0;
+ ioh->report_return = NULL;
+ ioh->result = 0;
+ ioh->distilled_result.status = ARS_UNKNOWN;
+
+ /* XXX: the barrier is probably superfluous */
+ pg_write_barrier();
+ ioh->generation++;
+
+ pgaio_io_update_state(ioh, PGAIO_HS_IDLE);
+
+ /*
+ * We push the IO to the head of the idle IO list, that seems more cache
+ * efficient in cases where only a few IOs are used.
+ */
+ dclist_push_head(&pgaio_my_backend->idle_ios, &ioh->node);
+}
+
+/*
+ * Wait for an IO handle to become usable.
+ *
+ * This only really is useful for pgaio_io_acquire().
+ */
+static void
+pgaio_io_wait_for_free(void)
+{
+ int reclaimed = 0;
+
+ pgaio_debug(DEBUG2, "waiting for self with %d pending",
+ pgaio_my_backend->num_staged_ios);
+
+ /*
+ * First check if any of our IOs actually have completed - when using
+ * worker, that'll often be the case. We could do so as part of the loop
+ * below, but that'd potentially lead us to wait for some IO submitted
+ * before.
+ */
+ for (int i = 0; i < io_max_concurrency; i++)
+ {
+ PgAioHandle *ioh = &pgaio_ctl->io_handles[pgaio_my_backend->io_handle_off + i];
+
+ if (ioh->state == PGAIO_HS_COMPLETED_SHARED)
+ {
+ pgaio_io_reclaim(ioh);
+ reclaimed++;
+ }
+ }
+
+ if (reclaimed > 0)
+ return;
+
+ /*
+ * If we have any unsubmitted IOs, submit them now. We'll start waiting in
+ * a second, so it's better they're in flight. This also addresses the
+ * edge-case that all IOs are unsubmitted.
+ */
+ if (pgaio_my_backend->num_staged_ios > 0)
+ pgaio_submit_staged();
+
+ if (dclist_count(&pgaio_my_backend->in_flight_ios) == 0)
+ elog(ERROR, "no free IOs despite no in-flight IOs");
+
+ /*
+ * Wait for the oldest in-flight IO to complete.
+ *
+ * XXX: Reusing the general IO wait is suboptimal, we don't need to wait
+ * for that specific IO to complete, we just need *any* IO to complete.
+ */
+ {
+ PgAioHandle *ioh = dclist_head_element(PgAioHandle, node, &pgaio_my_backend->in_flight_ios);
+
+ switch (ioh->state)
+ {
+ /* should not be in in-flight list */
+ case PGAIO_HS_IDLE:
+ case PGAIO_HS_DEFINED:
+ case PGAIO_HS_HANDED_OUT:
+ case PGAIO_HS_STAGED:
+ case PGAIO_HS_COMPLETED_LOCAL:
+ elog(ERROR, "shouldn't get here with io:%d in state %d",
+ pgaio_io_get_id(ioh), ioh->state);
+ break;
+
+ case PGAIO_HS_COMPLETED_IO:
+ case PGAIO_HS_SUBMITTED:
+ pgaio_debug_io(DEBUG2, ioh,
+ "waiting for free io with %d in flight",
+ dclist_count(&pgaio_my_backend->in_flight_ios));
+
+ /*
+ * In a more general case this would be racy, because the
+ * generation could increase after we read ioh->state above.
+ * But we are only looking at IOs by the current backend and
+ * the IO can only be recycled by this backend.
+ */
+ pgaio_io_wait(ioh, ioh->generation);
+ break;
+
+ case PGAIO_HS_COMPLETED_SHARED:
+ /* it's possible that another backend just finished this IO */
+ pgaio_io_reclaim(ioh);
+ break;
+ }
+
+ if (dclist_count(&pgaio_my_backend->idle_ios) == 0)
+ elog(PANIC, "no idle IOs after waiting");
+ return;
+ }
+}
+
+/*
+ * Internal - code outside of AIO should never need this and it'd be hard for
+ * such code to be safe.
+ */
+static PgAioHandle *
+pgaio_io_from_wref(PgAioWaitRef *iow, uint64 *ref_generation)
+{
+ PgAioHandle *ioh;
+
+ Assert(iow->aio_index < pgaio_ctl->io_handle_count);
+
+ ioh = &pgaio_ctl->io_handles[iow->aio_index];
+
+ *ref_generation = ((uint64) iow->generation_upper) << 32 |
+ iow->generation_lower;
+
+ Assert(*ref_generation != 0);
+
+ return ioh;
+}
+
+static const char *
+pgaio_io_state_get_name(PgAioHandleState s)
+{
+#define PGAIO_HS_TOSTR_CASE(sym) case PGAIO_HS_##sym: return #sym
+ switch (s)
+ {
+ PGAIO_HS_TOSTR_CASE(IDLE);
+ PGAIO_HS_TOSTR_CASE(HANDED_OUT);
+ PGAIO_HS_TOSTR_CASE(DEFINED);
+ PGAIO_HS_TOSTR_CASE(STAGED);
+ PGAIO_HS_TOSTR_CASE(SUBMITTED);
+ PGAIO_HS_TOSTR_CASE(COMPLETED_IO);
+ PGAIO_HS_TOSTR_CASE(COMPLETED_SHARED);
+ PGAIO_HS_TOSTR_CASE(COMPLETED_LOCAL);
+ }
+#undef PGAIO_HS_TOSTR_CASE
+
+ return NULL; /* silence compiler */
+}
+
+const char *
+pgaio_io_get_state_name(PgAioHandle *ioh)
+{
+ return pgaio_io_state_get_name(ioh->state);
+}
+
+const char *
+pgaio_result_status_string(PgAioResultStatus rs)
+{
+ switch (rs)
+ {
+ case ARS_UNKNOWN:
+ return "UNKNOWN";
+ case ARS_OK:
+ return "OK";
+ case ARS_PARTIAL:
+ return "PARTIAL";
+ case ARS_ERROR:
+ return "ERROR";
+ }
+
+ return NULL; /* silence compiler */
+}
+
+
+
+/* --------------------------------------------------------------------------------
+ * Functions primarily related to IO Wait References
+ * --------------------------------------------------------------------------------
+ */
+
+/*
+ * Mark a wait reference as invalid
+ */
+void
+pgaio_wref_clear(PgAioWaitRef *iow)
+{
+ iow->aio_index = PG_UINT32_MAX;
+}
+
+/* Is the wait reference valid? */
+bool
+pgaio_wref_valid(PgAioWaitRef *iow)
+{
+ return iow->aio_index != PG_UINT32_MAX;
+}
+
+/*
+ * Similar to pgaio_io_get_id(), just for wait references.
+ */
+int
+pgaio_wref_get_id(PgAioWaitRef *iow)
+{
+ Assert(pgaio_wref_valid(iow));
+ return iow->aio_index;
+}
+
+/*
+ * Wait for the IO to have completed. Can be called in any process, not just
+ * in the issuing backend.
+ */
+void
+pgaio_wref_wait(PgAioWaitRef *iow)
+{
+ uint64 ref_generation;
+ PgAioHandle *ioh;
+
+ ioh = pgaio_io_from_wref(iow, &ref_generation);
+
+ pgaio_io_wait(ioh, ref_generation);
+}
+
+/*
+ * Check if the referenced IO completed, without blocking.
+ */
+bool
+pgaio_wref_check_done(PgAioWaitRef *iow)
+{
+ uint64 ref_generation;
+ PgAioHandleState state;
+ bool am_owner;
+ PgAioHandle *ioh;
+
+ ioh = pgaio_io_from_wref(iow, &ref_generation);
+
+ if (pgaio_io_was_recycled(ioh, ref_generation, &state))
+ return true;
+
+ if (state == PGAIO_HS_IDLE)
+ return true;
+
+ am_owner = ioh->owner_procno == MyProcNumber;
+
+ if (state == PGAIO_HS_COMPLETED_SHARED ||
+ state == PGAIO_HS_COMPLETED_LOCAL)
+ {
+ if (am_owner)
+ pgaio_io_reclaim(ioh);
+ return true;
+ }
+
+ /*
+ * XXX: It likely would be worth checking in with the io method, to give
+ * the IO method a chance to check if there are completion events queued.
+ */
+
+ return false;
+}
+
+
+
+/* --------------------------------------------------------------------------------
+ * Actions on multiple IOs.
+ * --------------------------------------------------------------------------------
+ */
+
+/*
+ * Submit IOs in batches going forward.
+ *
+ * Submitting multiple IOs at once can be substantially faster than doing so
+ * one-by-one. At the same time submitting multiple IOs at once requires more
+ * care to avoid deadlocks.
+ *
+ * Consider backend A staging an IO for buffer 1 and then trying to start IO
+ * on buffer 2, while backend B does the inverse. If A submitted the IO before
+ * moving on to buffer 2, this works just fine, B will wait for the IO to
+ * complete. But if batching were used, each backend will wait for IO that has
+ * not yet been submitted to complete, i.e. forever.
+ *
+ * End batch submission mode with pgaio_exit_batchmode(). (Throwing errors is
+ * allowed; error recovery will end the batch.)
+ *
+ * To avoid deadlocks, code needs to ensure that it will not wait for another
+ * backend while there is unsubmitted IO. E.g. by using conditional lock
+ * acquisition when acquiring buffer locks. To check if there currently are
+ * staged IOs, call pgaio_have_staged() and to submit all staged IOs call
+ * pgaio_submit_staged().
+ *
+ * It is not allowed to enter batchmode while already in batchmode, it's
+ * unlikely to ever be needed, as code needs to be explicitly aware of being
+ * called in batchmode, to avoid the deadlock risks explained above.
+ *
+ * Note that IOs may get submitted before pgaio_exit_batchmode() is called,
+ * e.g. because too many IOs have been staged or because pgaio_submit_staged()
+ * was called.
+ */
+void
+pgaio_enter_batchmode(void)
+{
+ if (pgaio_my_backend->in_batchmode)
+ elog(ERROR, "starting batch while batch already in progress");
+ pgaio_my_backend->in_batchmode = true;
+}
+
+/*
+ * Stop submitting IOs in batches.
+ */
+void
+pgaio_exit_batchmode(void)
+{
+ Assert(pgaio_my_backend->in_batchmode);
+
+ pgaio_submit_staged();
+ pgaio_my_backend->in_batchmode = false;
+}
+
+/*
+ * Are there staged but unsubmitted IOs?
+ *
+ * See comment above pgaio_enter_batchmode() for why code may need to check if
+ * there is IO in that state.
+ */
+bool
+pgaio_have_staged(void)
+{
+ Assert(pgaio_my_backend->in_batchmode ||
+ pgaio_my_backend->num_staged_ios == 0);
+ return pgaio_my_backend->num_staged_ios > 0;
+}
+
+/*
+ * Submit all staged but not yet submitted IOs.
+ *
+ * Unless in batch mode, this never needs to be called, as IOs get submitted
+ * as soon as possible. While in batchmode pgaio_submit_staged() can be called
+ * before waiting on another backend, to avoid the risk of deadlocks. See
+ * pgaio_enter_batchmode().
+ */
+void
+pgaio_submit_staged(void)
+{
+ int total_submitted = 0;
+ int did_submit;
+
+ if (pgaio_my_backend->num_staged_ios == 0)
+ return;
+
+
+ START_CRIT_SECTION();
+
+ did_submit = pgaio_method_ops->submit(pgaio_my_backend->num_staged_ios,
+ pgaio_my_backend->staged_ios);
+
+ END_CRIT_SECTION();
+
+ total_submitted += did_submit;
+
+ Assert(total_submitted == did_submit);
+
+ pgaio_my_backend->num_staged_ios = 0;
+
+ pgaio_debug(DEBUG4,
+ "aio: submitted %d IOs",
+ total_submitted);
+}
+
+
+
+/* --------------------------------------------------------------------------------
+ * Other
+ * --------------------------------------------------------------------------------
+ */
+
+
/*
* Perform AIO related cleanup after an error.
*
@@ -50,6 +1049,22 @@ pgaio_io_release_resowner(dlist_node *ioh_node, bool on_error)
void
pgaio_error_cleanup(void)
{
+ /*
+ * It is possible that code errored out after pgaio_enter_batchmode() but
+ * before pgaio_exit_batchmode() was called. In that case we need to
+ * submit the IO now.
+ */
+ if (pgaio_my_backend->in_batchmode)
+ {
+ pgaio_my_backend->in_batchmode = false;
+
+ pgaio_submit_staged();
+ }
+
+ /*
+ * As we aren't in batchmode, there shouldn't be any unsubmitted IOs.
+ */
+ Assert(pgaio_my_backend->num_staged_ios == 0);
}
/*
@@ -62,11 +1077,86 @@ pgaio_error_cleanup(void)
void
AtEOXact_Aio(bool is_commit)
{
+ /*
+ * We should never be in batch mode at transactional boundaries. In case
+ * an error was thrown while in batch mode, pgaio_error_cleanup() should
+ * have exited batchmode.
+ *
+ * In case we are in batchmode somehow, make sure to submit all staged
+ * IOs, other backends may need them to complete to continue.
+ */
+ if (pgaio_my_backend->in_batchmode)
+ {
+ pgaio_error_cleanup();
+ elog(WARNING, "open AIO batch at end of (sub-)transaction");
+ }
+
+ /*
+ * As we aren't in batchmode, there shouldn't be any unsubmitted IOs.
+ */
+ Assert(pgaio_my_backend->num_staged_ios == 0);
+}
+
+/*
+ * Need to submit staged but not yet submitted IOs using the fd, otherwise
+ * the IO would end up targeting something bogus.
+ */
+void
+pgaio_closing_fd(int fd)
+{
+ /*
+ * Might be called before AIO is initialized or in a subprocess that
+ * doesn't use AIO.
+ */
+ if (!pgaio_my_backend)
+ return;
+
+ /*
+ * For now just submit all staged IOs - we could be more selective, but
+ * it's probably not worth it.
+ */
+ pgaio_submit_staged();
+}
+
+/*
+ * Registered as before_shmem_exit() callback in pgaio_init_backend()
+ */
+void
+pgaio_shutdown(int code, Datum arg)
+{
+ Assert(pgaio_my_backend);
+ Assert(!pgaio_my_backend->handed_out_io);
+
+ /* first clean up resources as we would at a transaction boundary */
+ AtEOXact_Aio(code == 0);
+
+ /*
+ * Before exiting, make sure that all IOs are finished. That has two main
+ * purposes:
+ *
+ * - Some kernel-level AIO mechanisms don't deal well with the issuer of
+ * an AIO exiting before IO completed
+ *
+ * - It'd be confusing to see partially finished IOs in stats views etc
+ */
+ while (!dclist_is_empty(&pgaio_my_backend->in_flight_ios))
+ {
+ PgAioHandle *ioh = dclist_head_element(PgAioHandle, node, &pgaio_my_backend->in_flight_ios);
+
+ /* see comment in pgaio_io_wait_for_free() about raciness */
+ pgaio_io_wait(ioh, ioh->generation);
+ }
+
+ pgaio_my_backend = NULL;
}
void
assign_io_method(int newval, void *extra)
{
+ Assert(pgaio_method_ops_table[newval] != NULL);
+ Assert(newval < lengthof(io_method_options));
+
+ pgaio_method_ops = pgaio_method_ops_table[newval];
}
bool
@@ -88,3 +1178,43 @@ check_io_max_concurrency(int *newval, void **extra, GucSource source)
return true;
}
+
+
+
+/* --------------------------------------------------------------------------------
+ * Injection point support
+ * --------------------------------------------------------------------------------
+ */
+
+#ifdef USE_INJECTION_POINTS
+
+/*
+ * Call injection point with support for pgaio_inj_io_get().
+ */
+void
+pgaio_io_call_inj(PgAioHandle *ioh, const char *injection_point)
+{
+ pgaio_inj_cur_handle = ioh;
+
+ PG_TRY();
+ {
+ InjectionPointCached(injection_point);
+ }
+ PG_FINALLY();
+ {
+ pgaio_inj_cur_handle = NULL;
+ }
+ PG_END_TRY();
+}
+
+/*
+ * Return IO associated with injection point invocation. This is only needed
+ * as injection points currently don't support arguments.
+ */
+PgAioHandle *
+pgaio_inj_io_get(void)
+{
+ return pgaio_inj_cur_handle;
+}
+
+#endif
diff --git a/src/backend/storage/aio/aio_callback.c b/src/backend/storage/aio/aio_callback.c
new file mode 100644
index 00000000000..7392b55322c
--- /dev/null
+++ b/src/backend/storage/aio/aio_callback.c
@@ -0,0 +1,307 @@
+/*-------------------------------------------------------------------------
+ *
+ * aio_callback.c
+ * AIO - Functionality related to callbacks that can be registered on IO
+ * Handles
+ *
+ * Portions Copyright (c) 1996-2025, PostgreSQL Global Development Group
+ * Portions Copyright (c) 1994, Regents of the University of California
+ *
+ * IDENTIFICATION
+ * src/backend/storage/aio/aio_callback.c
+ *
+ *-------------------------------------------------------------------------
+ */
+
+#include "postgres.h"
+
+#include "miscadmin.h"
+#include "storage/aio.h"
+#include "storage/aio_internal.h"
+
+
+/* just to have something to put into the aio_handle_cbs */
+static const struct PgAioHandleCallbacks aio_invalid_cb = {0};
+
+typedef struct PgAioHandleCallbacksEntry
+{
+ const PgAioHandleCallbacks *const cb;
+ const char *const name;
+} PgAioHandleCallbacksEntry;
+
+/*
+ * Callback definition for the callbacks that can be registered on an IO
+ * handle. See PgAioHandleCallbackID's definition for an explanation for why
+ * callbacks are not identified by a pointer.
+ */
+static const PgAioHandleCallbacksEntry aio_handle_cbs[] = {
+#define CALLBACK_ENTRY(id, callback) [id] = {.cb = &callback, .name = #callback}
+ CALLBACK_ENTRY(PGAIO_HCB_INVALID, aio_invalid_cb),
+#undef CALLBACK_ENTRY
+};
+
+
+
+/* --------------------------------------------------------------------------------
+ * Public callback related functions operating on IO Handles
+ * --------------------------------------------------------------------------------
+ */
+
+/*
+ * Register callback for the IO handle.
+ *
+ * Only a limited number (PGAIO_HANDLE_MAX_CALLBACKS) of callbacks can be
+ * registered for each IO.
+ *
+ * Callbacks need to be registered before [indirectly] calling
+ * pgaio_io_prep_*(), as the IO may be executed immediately.
+ *
+ * A callback can be passed a small bit of data, e.g. to indicate whether to
+ * zero a buffer if it is invalid.
+ *
+ *
+ * Note that callbacks are executed in critical sections. This is necessary
+ * to be able to execute IO in critical sections (consider e.g. WAL
+ * logging). To perform AIO we first need to acquire a handle, which, if there
+ * are no free handles, requires waiting for IOs to complete and to execute
+ * their completion callbacks.
+ *
+ * Callbacks may be executed in the issuing backend but also in another
+ * backend (because that backend is waiting for the IO) or in IO workers (if
+ * io_method=worker is used).
+ *
+ *
+ * See PgAioHandleCallbackID's definition for an explanation for why
+ * callbacks are not identified by a pointer.
+ */
+void
+pgaio_io_register_callbacks(PgAioHandle *ioh, PgAioHandleCallbackID cb_id,
+ uint8 cb_data)
+{
+ const PgAioHandleCallbacksEntry *ce = &aio_handle_cbs[cb_id];
+
+ if (cb_id >= lengthof(aio_handle_cbs))
+ elog(ERROR, "callback %d is out of range", cb_id);
+ if (aio_handle_cbs[cb_id].cb->complete_shared == NULL &&
+ aio_handle_cbs[cb_id].cb->complete_local == NULL)
+ elog(ERROR, "callback %d does not have completion callback", cb_id);
+ if (ioh->num_callbacks >= PGAIO_HANDLE_MAX_CALLBACKS)
+ elog(PANIC, "too many callbacks, the max is %d", PGAIO_HANDLE_MAX_CALLBACKS);
+ ioh->callbacks[ioh->num_callbacks] = cb_id;
+ ioh->callbacks_data[ioh->num_callbacks] = cb_data;
+
+ pgaio_debug_io(DEBUG3, ioh,
+ "adding cb #%d, id %d/%s",
+ ioh->num_callbacks + 1,
+ cb_id, ce->name);
+
+ ioh->num_callbacks++;
+}
+
+/*
+ * Associate an array of data with the Handle. This is e.g. useful to the
+ * transport knowledge about which buffers a multi-block IO affects to
+ * completion callbacks.
+ *
+ * Right now this can be done only once for each IO, even though multiple
+ * callbacks can be registered. There aren't any known usecases requiring more
+ * and the required amount of shared memory does add up, so it doesn't seem
+ * worth multiplying memory usage by PGAIO_HANDLE_MAX_CALLBACKS.
+ */
+void
+pgaio_io_set_handle_data_64(PgAioHandle *ioh, uint64 *data, uint8 len)
+{
+ Assert(ioh->state == PGAIO_HS_HANDED_OUT);
+ Assert(ioh->handle_data_len == 0);
+ Assert(len <= PG_IOV_MAX);
+
+ for (int i = 0; i < len; i++)
+ pgaio_ctl->handle_data[ioh->iovec_off + i] = data[i];
+ ioh->handle_data_len = len;
+}
+
+/*
+ * Convenience version of pgaio_io_set_handle_data_64() that converts a 32bit
+ * array to a 64bit array. Without it callers would end up needing to
+ * open-code equivalent code.
+ */
+void
+pgaio_io_set_handle_data_32(PgAioHandle *ioh, uint32 *data, uint8 len)
+{
+ Assert(ioh->state == PGAIO_HS_HANDED_OUT);
+ Assert(ioh->handle_data_len == 0);
+ Assert(len <= PG_IOV_MAX);
+
+ for (int i = 0; i < len; i++)
+ pgaio_ctl->handle_data[ioh->iovec_off + i] = data[i];
+ ioh->handle_data_len = len;
+}
+
+/*
+ * Return data set with pgaio_io_set_handle_data_*().
+ */
+uint64 *
+pgaio_io_get_handle_data(PgAioHandle *ioh, uint8 *len)
+{
+ Assert(ioh->handle_data_len > 0);
+
+ *len = ioh->handle_data_len;
+
+ return &pgaio_ctl->handle_data[ioh->iovec_off];
+}
+
+
+
+/* --------------------------------------------------------------------------------
+ * Public IO Result related functions
+ * --------------------------------------------------------------------------------
+ */
+
+void
+pgaio_result_report(PgAioResult result, const PgAioTargetData *target_data, int elevel)
+{
+ PgAioHandleCallbackID cb_id = result.id;
+ const PgAioHandleCallbacksEntry *ce = &aio_handle_cbs[cb_id];
+
+ Assert(result.status != ARS_UNKNOWN);
+ Assert(result.status != ARS_OK);
+
+ if (ce->cb->report == NULL)
+ elog(ERROR, "callback %d/%s does not have report callback",
+ result.id, ce->name);
+
+ ce->cb->report(result, target_data, elevel);
+}
+
+
+
+/* --------------------------------------------------------------------------------
+ * Internal callback related functions operating on IO Handles
+ * --------------------------------------------------------------------------------
+ */
+
+/*
+ * Internal function which invokes ->stage for all the registered callbacks.
+ */
+void
+pgaio_io_call_stage(PgAioHandle *ioh)
+{
+ Assert(ioh->target > PGAIO_TID_INVALID && ioh->target < PGAIO_TID_COUNT);
+ Assert(ioh->op > PGAIO_OP_INVALID && ioh->op < PGAIO_OP_COUNT);
+
+ for (int i = ioh->num_callbacks; i > 0; i--)
+ {
+ PgAioHandleCallbackID cb_id = ioh->callbacks[i - 1];
+ uint8 cb_data = ioh->callbacks_data[i - 1];
+ const PgAioHandleCallbacksEntry *ce = &aio_handle_cbs[cb_id];
+
+ if (!ce->cb->stage)
+ continue;
+
+ pgaio_debug_io(DEBUG3, ioh,
+ "calling cb #%d %d/%s->stage(%u)",
+ i, cb_id, ce->name, cb_data);
+ ce->cb->stage(ioh, cb_data);
+ }
+}
+
+/*
+ * Internal function which invokes ->complete_shared for all the registered
+ * callbacks.
+ */
+void
+pgaio_io_call_complete_shared(PgAioHandle *ioh)
+{
+ PgAioResult result;
+
+ START_CRIT_SECTION();
+
+ Assert(ioh->target > PGAIO_TID_INVALID && ioh->target < PGAIO_TID_COUNT);
+ Assert(ioh->op > PGAIO_OP_INVALID && ioh->op < PGAIO_OP_COUNT);
+
+ result.status = ARS_OK; /* low level IO is always considered OK */
+ result.result = ioh->result;
+ result.id = PGAIO_HCB_INVALID;
+ result.error_data = 0;
+
+ /*
+ * Call callbacks with the last registered (innermost) callback first.
+ * Each callback can modify the result forwarded to the next callback.
+ */
+ for (int i = ioh->num_callbacks; i > 0; i--)
+ {
+ PgAioHandleCallbackID cb_id = ioh->callbacks[i - 1];
+ uint8 cb_data = ioh->callbacks_data[i - 1];
+ const PgAioHandleCallbacksEntry *ce = &aio_handle_cbs[cb_id];
+
+ if (!ce->cb->complete_shared)
+ continue;
+
+ pgaio_debug_io(DEBUG4, ioh,
+ "calling cb #%d, id %d/%s->complete_shared(%u) with distilled result: (status %s, id %u, error_data %d, result %d)",
+ i, cb_id, ce->name,
+ cb_data,
+ pgaio_result_status_string(result.status),
+ result.id, result.error_data, result.result);
+ result = ce->cb->complete_shared(ioh, result, cb_data);
+ }
+
+ ioh->distilled_result = result;
+
+ pgaio_debug_io(DEBUG3, ioh,
+ "after shared completion: distilled result: (status %s, id %u, error_data: %d, result %d), raw_result: %d",
+ pgaio_result_status_string(result.status),
+ result.id, result.error_data, result.result,
+ ioh->result);
+
+ END_CRIT_SECTION();
+}
+
+/*
+ * Internal function which invokes ->complete_local for all the registered
+ * callbacks.
+ *
+ * XXX: It'd be nice to deduplicate with pgaio_io_call_complete_shared().
+ */
+void
+pgaio_io_call_complete_local(PgAioHandle *ioh)
+{
+ PgAioResult result;
+
+ START_CRIT_SECTION();
+
+ Assert(ioh->target > PGAIO_TID_INVALID && ioh->target < PGAIO_TID_COUNT);
+ Assert(ioh->op > PGAIO_OP_INVALID && ioh->op < PGAIO_OP_COUNT);
+
+ /* start with distilled result from shared callback */
+ result = ioh->distilled_result;
+
+ for (int i = ioh->num_callbacks; i > 0; i--)
+ {
+ PgAioHandleCallbackID cb_id = ioh->callbacks[i - 1];
+ uint8 cb_data = ioh->callbacks_data[i - 1];
+ const PgAioHandleCallbacksEntry *ce = &aio_handle_cbs[cb_id];
+
+ if (!ce->cb->complete_local)
+ continue;
+
+ pgaio_debug_io(DEBUG4, ioh,
+ "calling cb #%d, id %d/%s->complete_local(%u) with distilled result: status %s, id %u, error_data %d, result %d",
+ i, cb_id, ce->name, cb_data,
+ pgaio_result_status_string(result.status),
+ result.id, result.error_data, result.result);
+ result = ce->cb->complete_local(ioh, result, cb_data);
+ }
+
+ /*
+ * Note that we don't save the result in ioh->distilled_result, the local
+ * callback's result should not ever matter to other waiters.
+ */
+ pgaio_debug_io(DEBUG3, ioh,
+ "after local completion: distilled result: (status %s, id %u, error_data %d, result %d), raw_result: %d",
+ pgaio_result_status_string(result.status),
+ result.id, result.error_data, result.result,
+ ioh->result);
+
+ END_CRIT_SECTION();
+}
diff --git a/src/backend/storage/aio/aio_init.c b/src/backend/storage/aio/aio_init.c
index aeacc144149..e6c3e40f7d4 100644
--- a/src/backend/storage/aio/aio_init.c
+++ b/src/backend/storage/aio/aio_init.c
@@ -14,24 +14,211 @@
#include "postgres.h"
+#include "miscadmin.h"
+#include "storage/aio.h"
+#include "storage/aio_internal.h"
#include "storage/aio_subsys.h"
+#include "storage/ipc.h"
+#include "storage/proc.h"
+#include "storage/shmem.h"
+#include "utils/guc.h"
+static Size
+AioCtlShmemSize(void)
+{
+ Size sz;
+
+ /* pgaio_ctl itself */
+ sz = offsetof(PgAioCtl, io_handles);
+
+ return sz;
+}
+
+static uint32
+AioProcs(void)
+{
+ return MaxBackends + NUM_AUXILIARY_PROCS;
+}
+
+static Size
+AioBackendShmemSize(void)
+{
+ return mul_size(AioProcs(), sizeof(PgAioBackend));
+}
+
+static Size
+AioHandleShmemSize(void)
+{
+ Size sz;
+
+ /* ios */
+ sz = mul_size(AioProcs(),
+ mul_size(io_max_concurrency, sizeof(PgAioHandle)));
+
+ return sz;
+}
+
+static Size
+AioHandleIOVShmemSize(void)
+{
+ return mul_size(sizeof(struct iovec),
+ mul_size(mul_size(PG_IOV_MAX, AioProcs()),
+ io_max_concurrency));
+}
+
+static Size
+AioHandleDataShmemSize(void)
+{
+ return mul_size(sizeof(uint64),
+ mul_size(mul_size(PG_IOV_MAX, AioProcs()),
+ io_max_concurrency));
+}
+
+/*
+ * Choose a suitable value for io_max_concurrency.
+ *
+ * It's unlikely that we could have more IOs in flight than buffers that we
+ * would be allowed to pin.
+ *
+ * On the upper end, apply a cap too - just because shared_buffers is large,
+ * it doesn't make sense have millions of buffers undergo IO concurrently.
+ */
+static int
+AioChooseMaxConcurrency(void)
+{
+ uint32 max_backends;
+ int max_proportional_pins;
+
+ /* Similar logic to LimitAdditionalPins() */
+ max_backends = MaxBackends + NUM_AUXILIARY_PROCS;
+ max_proportional_pins = NBuffers / max_backends;
+
+ max_proportional_pins = Max(max_proportional_pins, 1);
+
+ /* apply upper limit */
+ return Min(max_proportional_pins, 64);
+}
+
Size
AioShmemSize(void)
{
Size sz = 0;
+ /*
+ * We prefer to report this value's source as PGC_S_DYNAMIC_DEFAULT.
+ * However, if the DBA explicitly set io_max_concurrency = -1 in the
+ * config file, then PGC_S_DYNAMIC_DEFAULT will fail to override that and
+ * we must force
+ *
+ */
+ if (io_max_concurrency == -1)
+ {
+ char buf[32];
+
+ snprintf(buf, sizeof(buf), "%d", AioChooseMaxConcurrency());
+ SetConfigOption("io_max_concurrency", buf, PGC_POSTMASTER,
+ PGC_S_DYNAMIC_DEFAULT);
+ if (io_max_concurrency == -1) /* failed to apply it? */
+ SetConfigOption("io_max_concurrency", buf, PGC_POSTMASTER,
+ PGC_S_OVERRIDE);
+ }
+
+ sz = add_size(sz, AioCtlShmemSize());
+ sz = add_size(sz, AioBackendShmemSize());
+ sz = add_size(sz, AioHandleShmemSize());
+ sz = add_size(sz, AioHandleIOVShmemSize());
+ sz = add_size(sz, AioHandleDataShmemSize());
+
+ if (pgaio_method_ops->shmem_size)
+ sz = add_size(sz, pgaio_method_ops->shmem_size());
+
return sz;
}
void
AioShmemInit(void)
{
+ bool found;
+ uint32 io_handle_off = 0;
+ uint32 iovec_off = 0;
+ uint32 per_backend_iovecs = io_max_concurrency * PG_IOV_MAX;
+
+ pgaio_ctl = (PgAioCtl *)
+ ShmemInitStruct("AioCtl", AioCtlShmemSize(), &found);
+
+ if (found)
+ goto out;
+
+ memset(pgaio_ctl, 0, AioCtlShmemSize());
+
+ pgaio_ctl->io_handle_count = AioProcs() * io_max_concurrency;
+ pgaio_ctl->iovec_count = AioProcs() * per_backend_iovecs;
+
+ pgaio_ctl->backend_state = (PgAioBackend *)
+ ShmemInitStruct("AioBackend", AioBackendShmemSize(), &found);
+
+ pgaio_ctl->io_handles = (PgAioHandle *)
+ ShmemInitStruct("AioHandle", AioHandleShmemSize(), &found);
+
+ pgaio_ctl->iovecs = (struct iovec *)
+ ShmemInitStruct("AioHandleIOV", AioHandleIOVShmemSize(), &found);
+ pgaio_ctl->handle_data = (uint64 *)
+ ShmemInitStruct("AioHandleData", AioHandleDataShmemSize(), &found);
+
+ for (int procno = 0; procno < AioProcs(); procno++)
+ {
+ PgAioBackend *bs = &pgaio_ctl->backend_state[procno];
+
+ bs->io_handle_off = io_handle_off;
+ io_handle_off += io_max_concurrency;
+
+ dclist_init(&bs->idle_ios);
+ memset(bs->staged_ios, 0, sizeof(PgAioHandle *) * PGAIO_SUBMIT_BATCH_SIZE);
+ dclist_init(&bs->in_flight_ios);
+
+ /* initialize per-backend IOs */
+ for (int i = 0; i < io_max_concurrency; i++)
+ {
+ PgAioHandle *ioh = &pgaio_ctl->io_handles[bs->io_handle_off + i];
+
+ ioh->generation = 1;
+ ioh->owner_procno = procno;
+ ioh->iovec_off = iovec_off;
+ ioh->handle_data_len = 0;
+ ioh->report_return = NULL;
+ ioh->resowner = NULL;
+ ioh->num_callbacks = 0;
+ ioh->distilled_result.status = ARS_UNKNOWN;
+ ioh->flags = 0;
+
+ ConditionVariableInit(&ioh->cv);
+
+ dclist_push_tail(&bs->idle_ios, &ioh->node);
+ iovec_off += PG_IOV_MAX;
+ }
+ }
+
+out:
+ /* Initialize IO method specific resources. */
+ if (pgaio_method_ops->shmem_init)
+ pgaio_method_ops->shmem_init(!found);
}
void
pgaio_init_backend(void)
{
+ /* shouldn't be initialized twice */
+ Assert(!pgaio_my_backend);
+
+ if (MyProc == NULL || MyProcNumber >= AioProcs())
+ elog(ERROR, "aio requires a normal PGPROC");
+
+ pgaio_my_backend = &pgaio_ctl->backend_state[MyProcNumber];
+
+ if (pgaio_method_ops->init_backend)
+ pgaio_method_ops->init_backend();
+
+ before_shmem_exit(pgaio_shutdown, 0);
}
diff --git a/src/backend/storage/aio/aio_io.c b/src/backend/storage/aio/aio_io.c
new file mode 100644
index 00000000000..89376ff4040
--- /dev/null
+++ b/src/backend/storage/aio/aio_io.c
@@ -0,0 +1,180 @@
+/*-------------------------------------------------------------------------
+ *
+ * aio_io.c
+ * AIO - Low Level IO Handling
+ *
+ * Functions related to associating IO operations to IO Handles and IO-method
+ * independent support functions for actually performing IO.
+ *
+ *
+ * Portions Copyright (c) 1996-2025, PostgreSQL Global Development Group
+ * Portions Copyright (c) 1994, Regents of the University of California
+ *
+ * IDENTIFICATION
+ * src/backend/storage/aio/aio_io.c
+ *
+ *-------------------------------------------------------------------------
+ */
+
+#include "postgres.h"
+
+#include "miscadmin.h"
+#include "storage/aio.h"
+#include "storage/aio_internal.h"
+#include "storage/fd.h"
+#include "utils/wait_event.h"
+
+
+static void pgaio_io_before_prep(PgAioHandle *ioh);
+
+
+
+/* --------------------------------------------------------------------------------
+ * Public IO related functions operating on IO Handles
+ * --------------------------------------------------------------------------------
+ */
+
+/*
+ * Scatter/gather IO needs to associate an iovec with the Handle. To support
+ * worker mode this data needs to be in shared memory.
+ *
+ * XXX: Right now the amount of space available for each IO is
+ * PG_IOV_MAX. While it's tempting to use the io_combine_limit GUC, that's
+ * PGC_USERSET, so we can't allocate shared memory based on that.
+ */
+int
+pgaio_io_get_iovec(PgAioHandle *ioh, struct iovec **iov)
+{
+ Assert(ioh->state == PGAIO_HS_HANDED_OUT);
+
+ *iov = &pgaio_ctl->iovecs[ioh->iovec_off];
+
+ return PG_IOV_MAX;
+}
+
+PgAioOpData *
+pgaio_io_get_op_data(PgAioHandle *ioh)
+{
+ return &ioh->op_data;
+}
+
+
+
+/* --------------------------------------------------------------------------------
+ * "Preparation" routines for individual IO operations
+ *
+ * These are called by the code actually initiating an IO, to associate the IO
+ * specific data with an AIO handle.
+ *
+ * Each of the preparation routines first needs to call
+ * pgaio_io_before_prep(), then fill IO specific fields in the handle and then
+ * finally call pgaio_io_stage().
+ * --------------------------------------------------------------------------------
+ */
+
+void
+pgaio_io_prep_readv(PgAioHandle *ioh,
+ int fd, int iovcnt, uint64 offset)
+{
+ pgaio_io_before_prep(ioh);
+
+ ioh->op_data.read.fd = fd;
+ ioh->op_data.read.offset = offset;
+ ioh->op_data.read.iov_length = iovcnt;
+
+ pgaio_io_stage(ioh, PGAIO_OP_READV);
+}
+
+void
+pgaio_io_prep_writev(PgAioHandle *ioh,
+ int fd, int iovcnt, uint64 offset)
+{
+ pgaio_io_before_prep(ioh);
+
+ ioh->op_data.write.fd = fd;
+ ioh->op_data.write.offset = offset;
+ ioh->op_data.write.iov_length = iovcnt;
+
+ pgaio_io_stage(ioh, PGAIO_OP_WRITEV);
+}
+
+
+
+/* --------------------------------------------------------------------------------
+ * Internal IO related functions operating on IO Handles
+ * --------------------------------------------------------------------------------
+ */
+
+/*
+ * Execute IO operation synchronously. This is implemented here, not in
+ * method_sync.c, because other IO methods lso might use it / fall back to it.
+ */
+void
+pgaio_io_perform_synchronously(PgAioHandle *ioh)
+{
+ ssize_t result = 0;
+ struct iovec *iov = &pgaio_ctl->iovecs[ioh->iovec_off];
+
+ START_CRIT_SECTION();
+
+ /* Perform IO. */
+ switch (ioh->op)
+ {
+ case PGAIO_OP_READV:
+ pgstat_report_wait_start(WAIT_EVENT_DATA_FILE_READ);
+ result = pg_preadv(ioh->op_data.read.fd, iov,
+ ioh->op_data.read.iov_length,
+ ioh->op_data.read.offset);
+ pgstat_report_wait_end();
+ break;
+ case PGAIO_OP_WRITEV:
+ pgstat_report_wait_start(WAIT_EVENT_DATA_FILE_WRITE);
+ result = pg_pwritev(ioh->op_data.write.fd, iov,
+ ioh->op_data.write.iov_length,
+ ioh->op_data.write.offset);
+ pgstat_report_wait_end();
+ break;
+ case PGAIO_OP_INVALID:
+ elog(ERROR, "trying to execute invalid IO operation");
+ }
+
+ ioh->result = result < 0 ? -errno : result;
+
+ pgaio_io_process_completion(ioh, ioh->result);
+
+ END_CRIT_SECTION();
+}
+
+/*
+ * Helper function to be called by IO operation preparation functions, before
+ * any data in the handle is set. Mostly to centralize assertions.
+ */
+static void
+pgaio_io_before_prep(PgAioHandle *ioh)
+{
+ Assert(ioh->state == PGAIO_HS_HANDED_OUT);
+ Assert(pgaio_io_has_target(ioh));
+ Assert(ioh->op == PGAIO_OP_INVALID);
+}
+
+/*
+ * Could be made part of the public interface, but it's not clear there's
+ * really a use case for that.
+ */
+const char *
+pgaio_io_get_op_name(PgAioHandle *ioh)
+{
+ Assert(ioh->op >= 0 && ioh->op < PGAIO_OP_COUNT);
+
+ switch (ioh->op)
+ {
+ case PGAIO_OP_INVALID:
+ return "invalid";
+ case PGAIO_OP_READV:
+ return "read";
+ case PGAIO_OP_WRITEV:
+ return "write";
+ }
+
+ return NULL; /* silence compiler */
+}
diff --git a/src/backend/storage/aio/aio_target.c b/src/backend/storage/aio/aio_target.c
new file mode 100644
index 00000000000..3fa813fd592
--- /dev/null
+++ b/src/backend/storage/aio/aio_target.c
@@ -0,0 +1,118 @@
+/*-------------------------------------------------------------------------
+ *
+ * aio_target.c
+ * AIO - Functionality related to executing IO for different targets
+ *
+ * Portions Copyright (c) 1996-2025, PostgreSQL Global Development Group
+ * Portions Copyright (c) 1994, Regents of the University of California
+ *
+ * IDENTIFICATION
+ * src/backend/storage/aio/aio_target.c
+ *
+ *-------------------------------------------------------------------------
+ */
+
+#include "postgres.h"
+
+#include "storage/aio.h"
+#include "storage/aio_internal.h"
+
+
+/*
+ * Registry for entities that can be the target of AIO.
+ *
+ * To support executing using worker processes, the file descriptor for an IO
+ * may need to be be reopened in a different process. This is done via the
+ * PgAioTargetInfo.reopen callback.
+ */
+static const PgAioTargetInfo *pgaio_target_info[] = {
+ [PGAIO_TID_INVALID] = &(PgAioTargetInfo) {
+ .name = "invalid",
+ },
+};
+
+
+
+/* --------------------------------------------------------------------------------
+ * Public target related functions operating on IO Handles
+ * --------------------------------------------------------------------------------
+ */
+
+bool
+pgaio_io_has_target(PgAioHandle *ioh)
+{
+ return ioh->target != PGAIO_TID_INVALID;
+}
+
+/*
+ * Return the name for the target associated with the IO. Mostly useful for
+ * debugging/logging.
+ */
+const char *
+pgaio_io_get_target_name(PgAioHandle *ioh)
+{
+ Assert(ioh->target >= 0 && ioh->target < PGAIO_TID_COUNT);
+
+ return pgaio_target_info[ioh->target]->name;
+}
+
+/*
+ * Assign a target to the IO.
+ *
+ * This has to be called exactly once before pgaio_io_prep_*() is called.
+ */
+void
+pgaio_io_set_target(PgAioHandle *ioh, PgAioTargetID targetid)
+{
+ Assert(ioh->state == PGAIO_HS_HANDED_OUT);
+ Assert(ioh->target == PGAIO_TID_INVALID);
+
+ ioh->target = targetid;
+}
+
+PgAioTargetData *
+pgaio_io_get_target_data(PgAioHandle *ioh)
+{
+ return &ioh->target_data;
+}
+
+/*
+ * Return a stringified description of the IO's target.
+ *
+ * The string is localized and allocated in the current memory context.
+ */
+char *
+pgaio_io_get_target_description(PgAioHandle *ioh)
+{
+ return pgaio_target_info[ioh->target]->describe_identity(&ioh->target_data);
+}
+
+
+
+/* --------------------------------------------------------------------------------
+ * Internal target related functions operating on IO Handles
+ * --------------------------------------------------------------------------------
+ */
+
+/*
+ * Internal: Check if pgaio_io_reopen() is available for the IO.
+ */
+bool
+pgaio_io_can_reopen(PgAioHandle *ioh)
+{
+ return pgaio_target_info[ioh->target]->reopen != NULL;
+}
+
+/*
+ * Internal: Before executing an IO outside of the context of the process the
+ * IO has been prepared in, the file descriptor has to be reopened - any FD
+ * referenced in the IO itself, won't be valid in the separate process.
+ */
+void
+pgaio_io_reopen(PgAioHandle *ioh)
+{
+ Assert(ioh->target >= 0 && ioh->target < PGAIO_TID_COUNT);
+ Assert(ioh->op >= 0 && ioh->op < PGAIO_OP_COUNT);
+
+ pgaio_target_info[ioh->target]->reopen(ioh);
+}
diff --git a/src/backend/storage/aio/meson.build b/src/backend/storage/aio/meson.build
index c822fd4ddf7..2c26089d52e 100644
--- a/src/backend/storage/aio/meson.build
+++ b/src/backend/storage/aio/meson.build
@@ -2,6 +2,10 @@
backend_sources += files(
'aio.c',
+ 'aio_callback.c',
'aio_init.c',
+ 'aio_io.c',
+ 'aio_target.c',
+ 'method_sync.c',
'read_stream.c',
)
diff --git a/src/backend/storage/aio/method_sync.c b/src/backend/storage/aio/method_sync.c
new file mode 100644
index 00000000000..902c2428d41
--- /dev/null
+++ b/src/backend/storage/aio/method_sync.c
@@ -0,0 +1,47 @@
+/*-------------------------------------------------------------------------
+ *
+ * method_sync.c
+ * AIO - perform "AIO" by executing it synchronously
+ *
+ * This method is mainly to check if AIO use causes regressions. Other IO
+ * methods might also fall back to the synchronous method for functionality
+ * they cannot provide.
+ *
+ * Portions Copyright (c) 1996-2025, PostgreSQL Global Development Group
+ * Portions Copyright (c) 1994, Regents of the University of California
+ *
+ * IDENTIFICATION
+ * src/backend/storage/aio/method_sync.c
+ *
+ *-------------------------------------------------------------------------
+ */
+
+#include "postgres.h"
+
+#include "storage/aio.h"
+#include "storage/aio_internal.h"
+
+static bool pgaio_sync_needs_synchronous_execution(PgAioHandle *ioh);
+static int pgaio_sync_submit(uint16 num_staged_ios, PgAioHandle **staged_ios);
+
+
+const IoMethodOps pgaio_sync_ops = {
+ .needs_synchronous_execution = pgaio_sync_needs_synchronous_execution,
+ .submit = pgaio_sync_submit,
+};
+
+
+
+static bool
+pgaio_sync_needs_synchronous_execution(PgAioHandle *ioh)
+{
+ return true;
+}
+
+static int
+pgaio_sync_submit(uint16 num_staged_ios, PgAioHandle **staged_ios)
+{
+ elog(ERROR, "IO should have been executed synchronously");
+
+ return 0;
+}
diff --git a/src/backend/utils/activity/wait_event_names.txt b/src/backend/utils/activity/wait_event_names.txt
index 3c594415bfd..b44e4908b25 100644
--- a/src/backend/utils/activity/wait_event_names.txt
+++ b/src/backend/utils/activity/wait_event_names.txt
@@ -191,6 +191,7 @@ ABI_compatibility:
Section: ClassName - WaitEventIO
+AIO_IO_COMPLETION "Waiting for IO completion."
BASEBACKUP_READ "Waiting for base backup to read from a file."
BASEBACKUP_SYNC "Waiting for data written by a base backup to reach durable storage."
BASEBACKUP_WRITE "Waiting for base backup to write to a file."
diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list
index d3eef76bc2e..cca85bd8806 100644
--- a/src/tools/pgindent/typedefs.list
+++ b/src/tools/pgindent/typedefs.list
@@ -1280,6 +1280,7 @@ InvalMessageArray
InvalidationInfo
InvalidationMsgsGroup
IoMethod
+IoMethodOps
IpcMemoryId
IpcMemoryKey
IpcMemoryState
@@ -2127,6 +2128,26 @@ Permutation
PermutationStep
PermutationStepBlocker
PermutationStepBlockerType
+PgAioBackend
+PgAioCtl
+PgAioHandle
+PgAioHandleCallbackID
+PgAioHandleCallbackStage
+PgAioHandleCallbackComplete
+PgAioHandleCallbackReport
+PgAioHandleCallbacks
+PgAioHandleCallbacksEntry
+PgAioHandleFlags
+PgAioHandleState
+PgAioOp
+PgAioOpData
+PgAioResult
+PgAioResultStatus
+PgAioReturn
+PgAioTargetData
+PgAioTargetID
+PgAioTargetInfo
+PgAioWaitRef
PgArchData
PgBackendGSSStatus
PgBackendSSLStatus
--
2.48.1.76.g4e746b1a31.dirty
[text/x-diff] v2.7-0003-aio-Infrastructure-for-io_method-worker.patch (24.9K, ../../4b3f32ug3cayekysqlgspz2qjmeb7lca3gvazayglxr2m3d4dv@il33accgsji7/4-v2.7-0003-aio-Infrastructure-for-io_method-worker.patch)
download | inline diff:
From 1cc6b28ffecdef03a55594dcbe92c9b2019c6556 Mon Sep 17 00:00:00 2001
From: Andres Freund <andres@anarazel.de>
Date: Mon, 17 Feb 2025 17:57:00 -0500
Subject: [PATCH v2.7 03/35] aio: Infrastructure for io_method=worker
This commit contains the basic, system-wide, infrastructure for
io_method=worker. It does not yet actually execute IO, this commit just
provides the infrastructure for running IO workers, kept separate for easier
review.
The number of IO workers can be adjusted with a PGC_SIGHUP GUC. Eventually
we'd like to make the number of workers dynamically scale up/down based on the
current "IO load".
To allow the number of IO workers to be increased without a restart, we need
to reserve PGPROC entries for the workers unconditionally. This has been
judged to be worth the cost. If it turns out to be problematic, we can
introduce a PGC_POSTMASTER GUC to control the maximum number.
Reviewed-by: Noah Misch <noah@leadboat.com>
Co-authored-by: Thomas Munro <thomas.munro@gmail.com>
Co-authored-by: Andres Freund <andres@anarazel.de>
Discussion: https://postgr.es/m/uvrtrknj4kdytuboidbhwclo4gxhswwcpgadptsjvjqcluzmah%40brqs62irg4dt
Discussion: https://postgr.es/m/20210223100344.llw5an2aklengrmn@alap3.anarazel.de
Discussion: https://postgr.es/m/stj36ea6yyhoxtqkhpieia2z4krnam7qyetc57rfezgk4zgapf@gcnactj4z56m
---
src/include/miscadmin.h | 2 +
src/include/storage/aio_subsys.h | 4 +
src/include/storage/io_worker.h | 22 +++
src/include/storage/proc.h | 4 +-
src/backend/postmaster/launch_backend.c | 2 +
src/backend/postmaster/pmchild.c | 1 +
src/backend/postmaster/postmaster.c | 174 ++++++++++++++++--
src/backend/storage/aio/Makefile | 1 +
src/backend/storage/aio/meson.build | 1 +
src/backend/storage/aio/method_worker.c | 88 +++++++++
src/backend/tcop/postgres.c | 2 +
src/backend/utils/activity/pgstat_backend.c | 1 +
src/backend/utils/activity/pgstat_io.c | 1 +
.../utils/activity/wait_event_names.txt | 1 +
src/backend/utils/init/miscinit.c | 3 +
src/backend/utils/misc/guc_tables.c | 13 ++
src/backend/utils/misc/postgresql.conf.sample | 1 +
doc/src/sgml/config.sgml | 19 ++
src/test/regress/expected/stats.out | 10 +-
19 files changed, 336 insertions(+), 14 deletions(-)
create mode 100644 src/include/storage/io_worker.h
create mode 100644 src/backend/storage/aio/method_worker.c
diff --git a/src/include/miscadmin.h b/src/include/miscadmin.h
index 6f16794eb63..603d0424354 100644
--- a/src/include/miscadmin.h
+++ b/src/include/miscadmin.h
@@ -360,6 +360,7 @@ typedef enum BackendType
B_ARCHIVER,
B_BG_WRITER,
B_CHECKPOINTER,
+ B_IO_WORKER,
B_STARTUP,
B_WAL_RECEIVER,
B_WAL_SUMMARIZER,
@@ -389,6 +390,7 @@ extern PGDLLIMPORT BackendType MyBackendType;
#define AmWalReceiverProcess() (MyBackendType == B_WAL_RECEIVER)
#define AmWalSummarizerProcess() (MyBackendType == B_WAL_SUMMARIZER)
#define AmWalWriterProcess() (MyBackendType == B_WAL_WRITER)
+#define AmIoWorkerProcess() (MyBackendType == B_IO_WORKER)
#define AmSpecialWorkerProcess() \
(AmAutoVacuumLauncherProcess() || \
diff --git a/src/include/storage/aio_subsys.h b/src/include/storage/aio_subsys.h
index e4faf692a38..ed00d5c47cd 100644
--- a/src/include/storage/aio_subsys.h
+++ b/src/include/storage/aio_subsys.h
@@ -30,4 +30,8 @@ extern void pgaio_init_backend(void);
extern void pgaio_error_cleanup(void);
extern void AtEOXact_Aio(bool is_commit);
+
+/* aio_worker.c */
+extern bool pgaio_workers_enabled(void);
+
#endif /* AIO_SUBSYS_H */
diff --git a/src/include/storage/io_worker.h b/src/include/storage/io_worker.h
new file mode 100644
index 00000000000..14ae6127dd0
--- /dev/null
+++ b/src/include/storage/io_worker.h
@@ -0,0 +1,22 @@
+/*-------------------------------------------------------------------------
+ *
+ * io_worker.h
+ * IO worker for implementing AIO "ourselves"
+ *
+ *
+ * Portions Copyright (c) 1996-2025, PostgreSQL Global Development Group
+ * Portions Copyright (c) 1994, Regents of the University of California
+ *
+ * src/include/storage/io.h
+ *
+ *-------------------------------------------------------------------------
+ */
+#ifndef IO_WORKER_H
+#define IO_WORKER_H
+
+
+extern void IoWorkerMain(const void *startup_data, size_t startup_data_len) pg_attribute_noreturn();
+
+extern PGDLLIMPORT int io_workers;
+
+#endif /* IO_WORKER_H */
diff --git a/src/include/storage/proc.h b/src/include/storage/proc.h
index 114eb1f8f76..4e815f9fde0 100644
--- a/src/include/storage/proc.h
+++ b/src/include/storage/proc.h
@@ -449,7 +449,9 @@ extern PGDLLIMPORT PGPROC *PreparedXactProcs;
* 2 slots, but WAL writer is launched only after startup has exited, so we
* only need 6 slots.
*/
-#define NUM_AUXILIARY_PROCS 6
+#define MAX_IO_WORKERS 32
+#define NUM_AUXILIARY_PROCS (6 + MAX_IO_WORKERS)
+
/* configurable options */
extern PGDLLIMPORT int DeadlockTimeout;
diff --git a/src/backend/postmaster/launch_backend.c b/src/backend/postmaster/launch_backend.c
index ecd04655c2a..6b117b128cb 100644
--- a/src/backend/postmaster/launch_backend.c
+++ b/src/backend/postmaster/launch_backend.c
@@ -48,6 +48,7 @@
#include "replication/slotsync.h"
#include "replication/walreceiver.h"
#include "storage/dsm.h"
+#include "storage/io_worker.h"
#include "storage/pg_shmem.h"
#include "tcop/backend_startup.h"
#include "utils/memutils.h"
@@ -197,6 +198,7 @@ static child_process_kind child_process_kinds[] = {
[B_ARCHIVER] = {"archiver", PgArchiverMain, true},
[B_BG_WRITER] = {"bgwriter", BackgroundWriterMain, true},
[B_CHECKPOINTER] = {"checkpointer", CheckpointerMain, true},
+ [B_IO_WORKER] = {"io_worker", IoWorkerMain, true},
[B_STARTUP] = {"startup", StartupProcessMain, true},
[B_WAL_RECEIVER] = {"wal_receiver", WalReceiverMain, true},
[B_WAL_SUMMARIZER] = {"wal_summarizer", WalSummarizerMain, true},
diff --git a/src/backend/postmaster/pmchild.c b/src/backend/postmaster/pmchild.c
index 0d473226c3a..cde1d23a4ca 100644
--- a/src/backend/postmaster/pmchild.c
+++ b/src/backend/postmaster/pmchild.c
@@ -101,6 +101,7 @@ InitPostmasterChildSlots(void)
pmchild_pools[B_AUTOVAC_WORKER].size = autovacuum_worker_slots;
pmchild_pools[B_BG_WORKER].size = max_worker_processes;
+ pmchild_pools[B_IO_WORKER].size = MAX_IO_WORKERS;
/*
* There can be only one of each of these running at a time. They each
diff --git a/src/backend/postmaster/postmaster.c b/src/backend/postmaster/postmaster.c
index 57155c00e01..7c0330a9780 100644
--- a/src/backend/postmaster/postmaster.c
+++ b/src/backend/postmaster/postmaster.c
@@ -108,9 +108,12 @@
#include "replication/logicallauncher.h"
#include "replication/slotsync.h"
#include "replication/walsender.h"
+#include "storage/aio_subsys.h"
#include "storage/fd.h"
+#include "storage/io_worker.h"
#include "storage/ipc.h"
#include "storage/pmsignal.h"
+#include "storage/proc.h"
#include "tcop/backend_startup.h"
#include "tcop/tcopprot.h"
#include "utils/datetime.h"
@@ -340,6 +343,7 @@ typedef enum
* ckpt */
PM_WAIT_XLOG_ARCHIVAL, /* waiting for archiver and walsenders to
* finish */
+ PM_WAIT_IO_WORKERS, /* waiting for io workers to exit */
PM_WAIT_CHECKPOINTER, /* waiting for checkpointer to shut down */
PM_WAIT_DEAD_END, /* waiting for dead-end children to exit */
PM_NO_CHILDREN, /* all important children have exited */
@@ -402,6 +406,10 @@ bool LoadedSSL = false;
static DNSServiceRef bonjour_sdref = NULL;
#endif
+/* State for IO worker management. */
+static int io_worker_count = 0;
+static PMChild *io_worker_children[MAX_IO_WORKERS];
+
/*
* postmaster.c - function prototypes
*/
@@ -436,6 +444,8 @@ static void TerminateChildren(int signal);
static int CountChildren(BackendTypeMask targetMask);
static void LaunchMissingBackgroundProcesses(void);
static void maybe_start_bgworkers(void);
+static bool maybe_reap_io_worker(int pid);
+static void maybe_adjust_io_workers(void);
static bool CreateOptsFile(int argc, char *argv[], char *fullprogname);
static PMChild *StartChildProcess(BackendType type);
static void StartSysLogger(void);
@@ -1365,6 +1375,11 @@ PostmasterMain(int argc, char *argv[])
*/
AddToDataDirLockFile(LOCK_FILE_LINE_PM_STATUS, PM_STATUS_STARTING);
+ UpdatePMState(PM_STARTUP);
+
+ /* Make sure we can perform I/O while starting up. */
+ maybe_adjust_io_workers();
+
/* Start bgwriter and checkpointer so they can help with recovery */
if (CheckpointerPMChild == NULL)
CheckpointerPMChild = StartChildProcess(B_CHECKPOINTER);
@@ -1377,7 +1392,6 @@ PostmasterMain(int argc, char *argv[])
StartupPMChild = StartChildProcess(B_STARTUP);
Assert(StartupPMChild != NULL);
StartupStatus = STARTUP_RUNNING;
- UpdatePMState(PM_STARTUP);
/* Some workers may be scheduled to start now */
maybe_start_bgworkers();
@@ -2502,6 +2516,16 @@ process_pm_child_exit(void)
continue;
}
+ /* Was it an IO worker? */
+ if (maybe_reap_io_worker(pid))
+ {
+ if (!EXIT_STATUS_0(exitstatus) && !EXIT_STATUS_1(exitstatus))
+ HandleChildCrash(pid, exitstatus, _("io worker"));
+
+ maybe_adjust_io_workers();
+ continue;
+ }
+
/*
* Was it a backend or a background worker?
*/
@@ -2723,6 +2747,7 @@ HandleFatalError(QuitSignalReason reason, bool consider_sigabrt)
case PM_WAIT_XLOG_SHUTDOWN:
case PM_WAIT_XLOG_ARCHIVAL:
case PM_WAIT_CHECKPOINTER:
+ case PM_WAIT_IO_WORKERS:
/*
* NB: Similar code exists in PostmasterStateMachine()'s handling
@@ -2905,20 +2930,21 @@ PostmasterStateMachine(void)
/*
* If we are doing crash recovery or an immediate shutdown then we
- * expect archiver, checkpointer and walsender to exit as well,
- * otherwise not.
+ * expect archiver, checkpointer, io workers and walsender to exit as
+ * well, otherwise not.
*/
if (FatalError || Shutdown >= ImmediateShutdown)
targetMask = btmask_add(targetMask,
B_CHECKPOINTER,
B_ARCHIVER,
+ B_IO_WORKER,
B_WAL_SENDER);
/*
- * Normally walsenders and archiver will continue running; they will
- * be terminated later after writing the checkpoint record. We also
- * let dead-end children to keep running for now. The syslogger
- * process exits last.
+ * Normally archiver, checkpointer, IO workers and walsenders will
+ * continue running; they will be terminated later after writing the
+ * checkpoint record. We also let dead-end children to keep running
+ * for now. The syslogger process exits last.
*
* This assertion checks that we have covered all backend types,
* either by including them in targetMask, or by noting here that they
@@ -2933,12 +2959,13 @@ PostmasterStateMachine(void)
B_LOGGER);
/*
- * Archiver, checkpointer and walsender may or may not be in
- * targetMask already.
+ * Archiver, checkpointer, IO workers, and walsender may or may
+ * not be in targetMask already.
*/
remainMask = btmask_add(remainMask,
B_ARCHIVER,
B_CHECKPOINTER,
+ B_IO_WORKER,
B_WAL_SENDER);
/* these are not real postmaster children */
@@ -3039,11 +3066,25 @@ PostmasterStateMachine(void)
{
/*
* PM_WAIT_XLOG_ARCHIVAL state ends when there are no children other
- * than checkpointer, dead-end children and logger left. There
+ * than checkpointer, io workers and dead-end children left. There
* shouldn't be any regular backends left by now anyway; what we're
* really waiting for is for walsenders and archiver to exit.
*/
- if (CountChildren(btmask_all_except(B_CHECKPOINTER, B_LOGGER, B_DEAD_END_BACKEND)) == 0)
+ if (CountChildren(btmask_all_except(B_CHECKPOINTER, B_IO_WORKER,
+ B_LOGGER, B_DEAD_END_BACKEND)) == 0)
+ {
+ UpdatePMState(PM_WAIT_IO_WORKERS);
+ SignalChildren(SIGUSR2, btmask(B_IO_WORKER));
+ }
+ }
+
+ if (pmState == PM_WAIT_IO_WORKERS)
+ {
+ /*
+ * PM_WAIT_IO_WORKERS state ends when there's only checkpointer and
+ * dead_end children left.
+ */
+ if (io_worker_count == 0)
{
UpdatePMState(PM_WAIT_CHECKPOINTER);
@@ -3171,10 +3212,14 @@ PostmasterStateMachine(void)
/* re-create shared memory and semaphores */
CreateSharedMemoryAndSemaphores();
+ UpdatePMState(PM_STARTUP);
+
+ /* Make sure we can perform I/O while starting up. */
+ maybe_adjust_io_workers();
+
StartupPMChild = StartChildProcess(B_STARTUP);
Assert(StartupPMChild != NULL);
StartupStatus = STARTUP_RUNNING;
- UpdatePMState(PM_STARTUP);
/* crash recovery started, reset SIGKILL flag */
AbortStartTime = 0;
@@ -3198,6 +3243,7 @@ pmstate_name(PMState state)
PM_TOSTR_CASE(PM_WAIT_BACKENDS);
PM_TOSTR_CASE(PM_WAIT_XLOG_SHUTDOWN);
PM_TOSTR_CASE(PM_WAIT_XLOG_ARCHIVAL);
+ PM_TOSTR_CASE(PM_WAIT_IO_WORKERS);
PM_TOSTR_CASE(PM_WAIT_DEAD_END);
PM_TOSTR_CASE(PM_WAIT_CHECKPOINTER);
PM_TOSTR_CASE(PM_NO_CHILDREN);
@@ -3235,6 +3281,16 @@ LaunchMissingBackgroundProcesses(void)
if (SysLoggerPMChild == NULL && Logging_collector)
StartSysLogger();
+ /*
+ * The number of configured configured workers might have changed, or a
+ * prior start of a worker might have failed. Check if we need to
+ * start/stop any workers.
+ *
+ * A config file change will always lead to this function being called, so
+ * we always will process the config change in a timely manner.
+ */
+ maybe_adjust_io_workers();
+
/*
* The checkpointer and the background writer are active from the start,
* until shutdown is initiated.
@@ -4120,6 +4176,7 @@ bgworker_should_start_now(BgWorkerStartTime start_time)
case PM_WAIT_DEAD_END:
case PM_WAIT_XLOG_ARCHIVAL:
case PM_WAIT_XLOG_SHUTDOWN:
+ case PM_WAIT_IO_WORKERS:
case PM_WAIT_BACKENDS:
case PM_STOP_BACKENDS:
break;
@@ -4270,6 +4327,99 @@ maybe_start_bgworkers(void)
}
}
+static bool
+maybe_reap_io_worker(int pid)
+{
+ for (int id = 0; id < MAX_IO_WORKERS; ++id)
+ {
+ if (io_worker_children[id] &&
+ io_worker_children[id]->pid == pid)
+ {
+ ReleasePostmasterChildSlot(io_worker_children[id]);
+
+ --io_worker_count;
+ io_worker_children[id] = NULL;
+ return true;
+ }
+ }
+ return false;
+}
+
+/*
+ * Start or stop IO workers, to close the gap between the number of running
+ * workers and the number of configured workers. Used to respond to change of
+ * the io_workers GUC (by increasing and decreasing the number of workers), as
+ * well as workers terminating in response to errors (by starting
+ * "replacement" workers).
+ */
+static void
+maybe_adjust_io_workers(void)
+{
+ if (!pgaio_workers_enabled())
+ return;
+
+ /*
+ * If we're in final shutting down state, then we're just waiting for all
+ * processes to exit.
+ */
+ if (pmState >= PM_WAIT_IO_WORKERS)
+ return;
+
+ /* Don't start new workers during an immediate shutdown either. */
+ if (Shutdown >= ImmediateShutdown)
+ return;
+
+ /*
+ * Don't start new workers if we're in the shutdown phase of a crash
+ * restart. But we *do* need to start if we're already starting up again.
+ */
+ if (FatalError && pmState >= PM_STOP_BACKENDS)
+ return;
+
+ Assert(pmState < PM_WAIT_IO_WORKERS);
+
+ /* Not enough running? */
+ while (io_worker_count < io_workers)
+ {
+ PMChild *child;
+ int id;
+
+ /* find unused entry in io_worker_children array */
+ for (id = 0; id < MAX_IO_WORKERS; ++id)
+ {
+ if (io_worker_children[id] == NULL)
+ break;
+ }
+ if (id == MAX_IO_WORKERS)
+ elog(ERROR, "could not find a free IO worker ID");
+
+ /* Try to launch one. */
+ child = StartChildProcess(B_IO_WORKER);
+ if (child != NULL)
+ {
+ io_worker_children[id] = child;
+ ++io_worker_count;
+ }
+ else
+ break; /* XXX try again soon? */
+ }
+
+ /* Too many running? */
+ if (io_worker_count > io_workers)
+ {
+ /* ask the IO worker in the highest slot to exit */
+ for (int id = MAX_IO_WORKERS - 1; id >= 0; --id)
+ {
+ if (io_worker_children[id] != NULL)
+ {
+ kill(io_worker_children[id]->pid, SIGUSR2);
+ break;
+ }
+ }
+ }
+}
+
+
/*
* When a backend asks to be notified about worker state changes, we
* set a flag in its backend entry. The background worker machinery needs
diff --git a/src/backend/storage/aio/Makefile b/src/backend/storage/aio/Makefile
index 89f821ea7e1..f51c34a37f8 100644
--- a/src/backend/storage/aio/Makefile
+++ b/src/backend/storage/aio/Makefile
@@ -15,6 +15,7 @@ OBJS = \
aio_io.o \
aio_target.o \
method_sync.o \
+ method_worker.o \
read_stream.o
include $(top_srcdir)/src/backend/common.mk
diff --git a/src/backend/storage/aio/meson.build b/src/backend/storage/aio/meson.build
index 2c26089d52e..74f94c6e40b 100644
--- a/src/backend/storage/aio/meson.build
+++ b/src/backend/storage/aio/meson.build
@@ -7,5 +7,6 @@ backend_sources += files(
'aio_io.c',
'aio_target.c',
'method_sync.c',
+ 'method_worker.c',
'read_stream.c',
)
diff --git a/src/backend/storage/aio/method_worker.c b/src/backend/storage/aio/method_worker.c
new file mode 100644
index 00000000000..0ef9ef93e2b
--- /dev/null
+++ b/src/backend/storage/aio/method_worker.c
@@ -0,0 +1,88 @@
+/*-------------------------------------------------------------------------
+ *
+ * method_worker.c
+ * AIO - perform AIO using worker processes
+ *
+ * Portions Copyright (c) 1996-2025, PostgreSQL Global Development Group
+ * Portions Copyright (c) 1994, Regents of the University of California
+ *
+ * IDENTIFICATION
+ * src/backend/storage/aio/method_worker.c
+ *
+ *-------------------------------------------------------------------------
+ */
+
+#include "postgres.h"
+
+#include "libpq/pqsignal.h"
+#include "miscadmin.h"
+#include "postmaster/auxprocess.h"
+#include "postmaster/interrupt.h"
+#include "storage/aio_subsys.h"
+#include "storage/io_worker.h"
+#include "storage/ipc.h"
+#include "storage/latch.h"
+#include "storage/proc.h"
+#include "tcop/tcopprot.h"
+#include "utils/wait_event.h"
+
+
+/* GUCs */
+int io_workers = 3;
+
+
+void
+IoWorkerMain(const void *startup_data, size_t startup_data_len)
+{
+ sigjmp_buf local_sigjmp_buf;
+
+ MyBackendType = B_IO_WORKER;
+ AuxiliaryProcessMainCommon();
+
+ pqsignal(SIGHUP, SignalHandlerForConfigReload);
+ pqsignal(SIGINT, die); /* to allow manually triggering worker restart */
+
+ /*
+ * Ignore SIGTERM, will get explicit shutdown via SIGUSR2 later in the
+ * shutdown sequence, similar to checkpointer.
+ */
+ pqsignal(SIGTERM, SIG_IGN);
+ /* SIGQUIT handler was already set up by InitPostmasterChild */
+ pqsignal(SIGALRM, SIG_IGN);
+ pqsignal(SIGPIPE, SIG_IGN);
+ pqsignal(SIGUSR1, procsignal_sigusr1_handler);
+ pqsignal(SIGUSR2, SignalHandlerForShutdownRequest);
+
+ /* see PostgresMain() */
+ if (sigsetjmp(local_sigjmp_buf, 1) != 0)
+ {
+ error_context_stack = NULL;
+ HOLD_INTERRUPTS();
+
+ EmitErrorReport();
+
+ proc_exit(1);
+ }
+
+ /* We can now handle ereport(ERROR) */
+ PG_exception_stack = &local_sigjmp_buf;
+
+ sigprocmask(SIG_SETMASK, &UnBlockSig, NULL);
+
+ while (!ShutdownRequestPending)
+ {
+ WaitLatch(MyLatch, WL_LATCH_SET | WL_EXIT_ON_PM_DEATH, -1,
+ WAIT_EVENT_IO_WORKER_MAIN);
+ ResetLatch(MyLatch);
+ CHECK_FOR_INTERRUPTS();
+ }
+
+ proc_exit(0);
+}
+
+bool
+pgaio_workers_enabled(void)
+{
+ /* placeholder for future commit */
+ return false;
+}
diff --git a/src/backend/tcop/postgres.c b/src/backend/tcop/postgres.c
index 55ab2da299b..fcb23239d07 100644
--- a/src/backend/tcop/postgres.c
+++ b/src/backend/tcop/postgres.c
@@ -3316,6 +3316,8 @@ ProcessInterrupts(void)
(errcode(ERRCODE_ADMIN_SHUTDOWN),
errmsg("terminating background worker \"%s\" due to administrator command",
MyBgworkerEntry->bgw_type)));
+ else if (AmIoWorkerProcess())
+ proc_exit(0);
else
ereport(FATAL,
(errcode(ERRCODE_ADMIN_SHUTDOWN),
diff --git a/src/backend/utils/activity/pgstat_backend.c b/src/backend/utils/activity/pgstat_backend.c
index a8cb54a7732..5518a18e060 100644
--- a/src/backend/utils/activity/pgstat_backend.c
+++ b/src/backend/utils/activity/pgstat_backend.c
@@ -375,6 +375,7 @@ pgstat_tracks_backend_bktype(BackendType bktype)
case B_LOGGER:
case B_BG_WRITER:
case B_CHECKPOINTER:
+ case B_IO_WORKER:
case B_STARTUP:
return false;
diff --git a/src/backend/utils/activity/pgstat_io.c b/src/backend/utils/activity/pgstat_io.c
index eb575025596..c8de9c9e2d3 100644
--- a/src/backend/utils/activity/pgstat_io.c
+++ b/src/backend/utils/activity/pgstat_io.c
@@ -376,6 +376,7 @@ pgstat_tracks_io_bktype(BackendType bktype)
case B_BG_WORKER:
case B_BG_WRITER:
case B_CHECKPOINTER:
+ case B_IO_WORKER:
case B_SLOTSYNC_WORKER:
case B_STANDALONE_BACKEND:
case B_STARTUP:
diff --git a/src/backend/utils/activity/wait_event_names.txt b/src/backend/utils/activity/wait_event_names.txt
index b44e4908b25..3f6dc3876b4 100644
--- a/src/backend/utils/activity/wait_event_names.txt
+++ b/src/backend/utils/activity/wait_event_names.txt
@@ -57,6 +57,7 @@ BGWRITER_HIBERNATE "Waiting in background writer process, hibernating."
BGWRITER_MAIN "Waiting in main loop of background writer process."
CHECKPOINTER_MAIN "Waiting in main loop of checkpointer process."
CHECKPOINTER_SHUTDOWN "Waiting for checkpointer process to be terminated."
+IO_WORKER_MAIN "Waiting in main loop of IO Worker process."
LOGICAL_APPLY_MAIN "Waiting in main loop of logical replication apply process."
LOGICAL_LAUNCHER_MAIN "Waiting in main loop of logical replication launcher process."
LOGICAL_PARALLEL_APPLY_MAIN "Waiting in main loop of logical replication parallel apply process."
diff --git a/src/backend/utils/init/miscinit.c b/src/backend/utils/init/miscinit.c
index dc3521457c7..43b4dbccc3d 100644
--- a/src/backend/utils/init/miscinit.c
+++ b/src/backend/utils/init/miscinit.c
@@ -293,6 +293,9 @@ GetBackendTypeDesc(BackendType backendType)
case B_CHECKPOINTER:
backendDesc = gettext_noop("checkpointer");
break;
+ case B_IO_WORKER:
+ backendDesc = gettext_noop("io worker");
+ break;
case B_LOGGER:
backendDesc = gettext_noop("logger");
break;
diff --git a/src/backend/utils/misc/guc_tables.c b/src/backend/utils/misc/guc_tables.c
index a456f27ba1e..d4e63c75cb6 100644
--- a/src/backend/utils/misc/guc_tables.c
+++ b/src/backend/utils/misc/guc_tables.c
@@ -75,6 +75,7 @@
#include "storage/aio.h"
#include "storage/bufmgr.h"
#include "storage/bufpage.h"
+#include "storage/io_worker.h"
#include "storage/large_object.h"
#include "storage/pg_shmem.h"
#include "storage/predicate.h"
@@ -3258,6 +3259,18 @@ struct config_int ConfigureNamesInt[] =
check_io_max_concurrency, NULL, NULL
},
+ {
+ {"io_workers",
+ PGC_SIGHUP,
+ RESOURCES_IO,
+ gettext_noop("Number of IO worker processes, for io_method=worker."),
+ NULL,
+ },
+ &io_workers,
+ 3, 1, MAX_IO_WORKERS,
+ NULL, NULL, NULL
+ },
+
{
{"backend_flush_after", PGC_USERSET, RESOURCES_IO,
gettext_noop("Number of pages after which previously performed writes are flushed to disk."),
diff --git a/src/backend/utils/misc/postgresql.conf.sample b/src/backend/utils/misc/postgresql.conf.sample
index 58b256e2221..1fd56ee5362 100644
--- a/src/backend/utils/misc/postgresql.conf.sample
+++ b/src/backend/utils/misc/postgresql.conf.sample
@@ -207,6 +207,7 @@
# can execute simultaneously
# -1 sets based on shared_buffers
# (change requires restart)
+#io_workers = 3 # 1-32;
# - Worker Processes -
diff --git a/doc/src/sgml/config.sgml b/doc/src/sgml/config.sgml
index 7ca8e7867cb..bd359e6480d 100644
--- a/doc/src/sgml/config.sgml
+++ b/doc/src/sgml/config.sgml
@@ -2689,6 +2689,25 @@ include_dir 'conf.d'
</listitem>
</varlistentry>
+ <varlistentry id="guc-io-workers" xreflabel="io_workers">
+ <term><varname>io_workers</varname> (<type>int</type>)
+ <indexterm>
+ <primary><varname>io_workers</varname> configuration parameter</primary>
+ </indexterm>
+ </term>
+ <listitem>
+ <para>
+ Selects the number of I/O worker processes to use. The default is
+ 3. This parameter can only be set in the
+ <filename>postgresql.conf</filename> file or on the server command
+ line.
+ </para>
+ <para>
+ Only has an effect if <xref linkend="guc-io-method"/> is set to
+ <literal>worker</literal>.
+ </para>
+ </listitem>
+ </varlistentry>
</variablelist>
</sect2>
diff --git a/src/test/regress/expected/stats.out b/src/test/regress/expected/stats.out
index f77caacc17d..cd08a2ca0af 100644
--- a/src/test/regress/expected/stats.out
+++ b/src/test/regress/expected/stats.out
@@ -51,6 +51,14 @@ client backend|relation|vacuum
client backend|temp relation|normal
client backend|wal|init
client backend|wal|normal
+io worker|relation|bulkread
+io worker|relation|bulkwrite
+io worker|relation|init
+io worker|relation|normal
+io worker|relation|vacuum
+io worker|temp relation|normal
+io worker|wal|init
+io worker|wal|normal
slotsync worker|relation|bulkread
slotsync worker|relation|bulkwrite
slotsync worker|relation|init
@@ -87,7 +95,7 @@ walsummarizer|wal|init
walsummarizer|wal|normal
walwriter|wal|init
walwriter|wal|normal
-(71 rows)
+(79 rows)
\a
-- ensure that both seqscan and indexscan plans are allowed
SET enable_seqscan TO on;
--
2.48.1.76.g4e746b1a31.dirty
[text/x-diff] v2.7-0004-aio-Add-io_method-worker.patch (20.2K, ../../4b3f32ug3cayekysqlgspz2qjmeb7lca3gvazayglxr2m3d4dv@il33accgsji7/5-v2.7-0004-aio-Add-io_method-worker.patch)
download | inline diff:
From b54005c7e95e7f2aa448677e38e6f84bd31dc9ee Mon Sep 17 00:00:00 2001
From: Andres Freund <andres@anarazel.de>
Date: Fri, 31 Jan 2025 13:46:35 -0500
Subject: [PATCH v2.7 04/35] aio: Add io_method=worker
The previous commit introduced the infrastructure to start io_workers. This
commit actually makes the workers execute IOs.
IO workers consume IOs from a shared memory submission queue, run traditional
synchronous system calls, and perform the shared completion handling
immediately. Client code submits most requests by pushing IOs into the
submission queue, and waits (if necessary) using condition variables. Some
IOs cannot be performed in another process due to lack of infrastructure for
reopening the file, and must processed synchronously by the client code when
submitted.
For now the default io_method is changed to "worker". We should re-evaluate
that around beta1, we might want to be careful and set the default to "sync"
for 18.
Reviewed-by: Noah Misch <noah@leadboat.com>
Co-authored-by: Thomas Munro <thomas.munro@gmail.com>
Co-authored-by: Andres Freund <andres@anarazel.de>
Discussion: https://postgr.es/m/uvrtrknj4kdytuboidbhwclo4gxhswwcpgadptsjvjqcluzmah%40brqs62irg4dt
Discussion: https://postgr.es/m/20210223100344.llw5an2aklengrmn@alap3.anarazel.de
Discussion: https://postgr.es/m/stj36ea6yyhoxtqkhpieia2z4krnam7qyetc57rfezgk4zgapf@gcnactj4z56m
---
src/include/storage/aio.h | 5 +-
src/include/storage/aio_internal.h | 1 +
src/include/storage/lwlocklist.h | 1 +
src/backend/storage/aio/aio.c | 2 +
src/backend/storage/aio/aio_init.c | 9 +
src/backend/storage/aio/method_worker.c | 431 +++++++++++++++++-
.../utils/activity/wait_event_names.txt | 1 +
src/backend/utils/misc/postgresql.conf.sample | 2 +-
doc/src/sgml/config.sgml | 5 +
src/tools/pgindent/typedefs.list | 3 +
10 files changed, 452 insertions(+), 8 deletions(-)
diff --git a/src/include/storage/aio.h b/src/include/storage/aio.h
index a0a0f23f3a5..769ea33ea96 100644
--- a/src/include/storage/aio.h
+++ b/src/include/storage/aio.h
@@ -27,10 +27,11 @@
typedef enum IoMethod
{
IOMETHOD_SYNC = 0,
+ IOMETHOD_WORKER,
} IoMethod;
-/* We'll default to synchronous execution. */
-#define DEFAULT_IO_METHOD IOMETHOD_SYNC
+/* We'll default to worker based execution. */
+#define DEFAULT_IO_METHOD IOMETHOD_WORKER
/*
diff --git a/src/include/storage/aio_internal.h b/src/include/storage/aio_internal.h
index e42c3dcb06c..b560f69076b 100644
--- a/src/include/storage/aio_internal.h
+++ b/src/include/storage/aio_internal.h
@@ -385,6 +385,7 @@ extern PgAioHandle *pgaio_inj_io_get(void);
/* Declarations for the tables of function pointers exposed by each IO method. */
extern PGDLLIMPORT const IoMethodOps pgaio_sync_ops;
+extern PGDLLIMPORT const IoMethodOps pgaio_worker_ops;
extern PGDLLIMPORT const IoMethodOps *pgaio_method_ops;
extern PGDLLIMPORT PgAioCtl *pgaio_ctl;
diff --git a/src/include/storage/lwlocklist.h b/src/include/storage/lwlocklist.h
index cf565452382..932024b1b0b 100644
--- a/src/include/storage/lwlocklist.h
+++ b/src/include/storage/lwlocklist.h
@@ -83,3 +83,4 @@ PG_LWLOCK(49, WALSummarizer)
PG_LWLOCK(50, DSMRegistry)
PG_LWLOCK(51, InjectionPoint)
PG_LWLOCK(52, SerialControl)
+PG_LWLOCK(53, AioWorkerSubmissionQueue)
diff --git a/src/backend/storage/aio/aio.c b/src/backend/storage/aio/aio.c
index 7eb0bd6f12d..94fbe0aea80 100644
--- a/src/backend/storage/aio/aio.c
+++ b/src/backend/storage/aio/aio.c
@@ -64,6 +64,7 @@ static void pgaio_io_wait(PgAioHandle *ioh, uint64 ref_generation);
/* Options for io_method. */
const struct config_enum_entry io_method_options[] = {
{"sync", IOMETHOD_SYNC, false},
+ {"worker", IOMETHOD_WORKER, false},
{NULL, 0, false}
};
@@ -80,6 +81,7 @@ PgAioBackend *pgaio_my_backend;
static const IoMethodOps *const pgaio_method_ops_table[] = {
[IOMETHOD_SYNC] = &pgaio_sync_ops,
+ [IOMETHOD_WORKER] = &pgaio_worker_ops,
};
/* callbacks for the configured io_method, set by assign_io_method */
diff --git a/src/backend/storage/aio/aio_init.c b/src/backend/storage/aio/aio_init.c
index e6c3e40f7d4..d0d916ea5c9 100644
--- a/src/backend/storage/aio/aio_init.c
+++ b/src/backend/storage/aio/aio_init.c
@@ -18,6 +18,7 @@
#include "storage/aio.h"
#include "storage/aio_internal.h"
#include "storage/aio_subsys.h"
+#include "storage/io_worker.h"
#include "storage/ipc.h"
#include "storage/proc.h"
#include "storage/shmem.h"
@@ -39,6 +40,11 @@ AioCtlShmemSize(void)
static uint32
AioProcs(void)
{
+ /*
+ * While AIO workers don't need their own AIO context, we can't currently
+ * guarantee nothing gets assigned to the a ProcNumber for an IO worker if
+ * we just subtracted MAX_IO_WORKERS.
+ */
return MaxBackends + NUM_AUXILIARY_PROCS;
}
@@ -212,6 +218,9 @@ pgaio_init_backend(void)
/* shouldn't be initialized twice */
Assert(!pgaio_my_backend);
+ if (MyBackendType == B_IO_WORKER)
+ return;
+
if (MyProc == NULL || MyProcNumber >= AioProcs())
elog(ERROR, "aio requires a normal PGPROC");
diff --git a/src/backend/storage/aio/method_worker.c b/src/backend/storage/aio/method_worker.c
index 0ef9ef93e2b..117c9a87db7 100644
--- a/src/backend/storage/aio/method_worker.c
+++ b/src/backend/storage/aio/method_worker.c
@@ -3,6 +3,21 @@
* method_worker.c
* AIO - perform AIO using worker processes
*
+ * IO workers consume IOs from a shared memory submission queue, run
+ * traditional synchronous system calls, and perform the shared completion
+ * handling immediately. Client code submits most requests by pushing IOs
+ * into the submission queue, and waits (if necessary) using condition
+ * variables. Some IOs cannot be performed in another process due to lack of
+ * infrastructure for reopening the file, and must processed synchronously by
+ * the client code when submitted.
+ *
+ * So that the submitter can make just one system call when submitting a batch
+ * of IOs, wakeups "fan out"; each woken IO worker can wake two more. XXX This
+ * could be improved by using futexes instead of latches to wake N waiters.
+ *
+ * This method of AIO is available in all builds on all operating systems, and
+ * is the default.
+ *
* Portions Copyright (c) 1996-2025, PostgreSQL Global Development Group
* Portions Copyright (c) 1994, Regents of the University of California
*
@@ -16,25 +31,325 @@
#include "libpq/pqsignal.h"
#include "miscadmin.h"
+#include "port/pg_bitutils.h"
#include "postmaster/auxprocess.h"
#include "postmaster/interrupt.h"
+#include "storage/aio.h"
+#include "storage/aio_internal.h"
#include "storage/aio_subsys.h"
#include "storage/io_worker.h"
#include "storage/ipc.h"
#include "storage/latch.h"
#include "storage/proc.h"
#include "tcop/tcopprot.h"
+#include "utils/ps_status.h"
#include "utils/wait_event.h"
+/* How many workers should each worker wake up if needed? */
+#define IO_WORKER_WAKEUP_FANOUT 2
+
+
+typedef struct AioWorkerSubmissionQueue
+{
+ uint32 size;
+ uint32 mask;
+ uint32 head;
+ uint32 tail;
+ uint32 ios[FLEXIBLE_ARRAY_MEMBER];
+} AioWorkerSubmissionQueue;
+
+typedef struct AioWorkerSlot
+{
+ Latch *latch;
+ bool in_use;
+} AioWorkerSlot;
+
+typedef struct AioWorkerControl
+{
+ uint64 idle_worker_mask;
+ AioWorkerSlot workers[FLEXIBLE_ARRAY_MEMBER];
+} AioWorkerControl;
+
+
+static size_t pgaio_worker_shmem_size(void);
+static void pgaio_worker_shmem_init(bool first_time);
+
+static bool pgaio_worker_needs_synchronous_execution(PgAioHandle *ioh);
+static int pgaio_worker_submit(uint16 num_staged_ios, PgAioHandle **staged_ios);
+
+
+const IoMethodOps pgaio_worker_ops = {
+ .shmem_size = pgaio_worker_shmem_size,
+ .shmem_init = pgaio_worker_shmem_init,
+
+ .needs_synchronous_execution = pgaio_worker_needs_synchronous_execution,
+ .submit = pgaio_worker_submit,
+};
+
+
/* GUCs */
int io_workers = 3;
+static int io_worker_queue_size = 64;
+static int MyIoWorkerId;
+static AioWorkerSubmissionQueue *io_worker_submission_queue;
+static AioWorkerControl *io_worker_control;
+
+
+static size_t
+pgaio_worker_shmem_size(void)
+{
+ return
+ offsetof(AioWorkerSubmissionQueue, ios) +
+ sizeof(uint32) * MAX_IO_WORKERS * io_max_concurrency +
+ offsetof(AioWorkerControl, workers) +
+ sizeof(AioWorkerSlot) * MAX_IO_WORKERS;
+}
+
+static void
+pgaio_worker_shmem_init(bool first_time)
+{
+ bool found;
+ int size;
+
+ /* Round size up to next power of two so we can make a mask. */
+ size = pg_nextpower2_32(io_worker_queue_size);
+
+ io_worker_submission_queue =
+ ShmemInitStruct("AioWorkerSubmissionQueue",
+ offsetof(AioWorkerSubmissionQueue, ios) +
+ sizeof(uint32) * size,
+ &found);
+ if (!found)
+ {
+ io_worker_submission_queue->size = size;
+ io_worker_submission_queue->head = 0;
+ io_worker_submission_queue->tail = 0;
+ }
+
+ io_worker_control =
+ ShmemInitStruct("AioWorkerControl",
+ offsetof(AioWorkerControl, workers) +
+ sizeof(AioWorkerSlot) * io_workers,
+ &found);
+ if (!found)
+ {
+ io_worker_control->idle_worker_mask = 0;
+ for (int i = 0; i < io_workers; ++i)
+ {
+ io_worker_control->workers[i].latch = NULL;
+ io_worker_control->workers[i].in_use = false;
+ }
+ }
+}
+
+static int
+pgaio_choose_idle_worker(void)
+{
+ int worker;
+
+ if (io_worker_control->idle_worker_mask == 0)
+ return -1;
+
+ /* Find the lowest bit position, and clear it. */
+ worker = pg_rightmost_one_pos64(io_worker_control->idle_worker_mask);
+ io_worker_control->idle_worker_mask &= ~(UINT64_C(1) << worker);
+
+ return worker;
+}
+
+static bool
+pgaio_worker_submission_queue_insert(PgAioHandle *ioh)
+{
+ AioWorkerSubmissionQueue *queue;
+ uint32 new_head;
+
+ queue = io_worker_submission_queue;
+ new_head = (queue->head + 1) & (queue->size - 1);
+ if (new_head == queue->tail)
+ {
+ pgaio_debug(DEBUG3, "io queue is full, at %u elements",
+ io_worker_submission_queue->size);
+ return false; /* full */
+ }
+
+ queue->ios[queue->head] = pgaio_io_get_id(ioh);
+ queue->head = new_head;
+
+ return true;
+}
+
+static uint32
+pgaio_worker_submission_queue_consume(void)
+{
+ AioWorkerSubmissionQueue *queue;
+ uint32 result;
+
+ queue = io_worker_submission_queue;
+ if (queue->tail == queue->head)
+ return UINT32_MAX; /* empty */
+
+ result = queue->ios[queue->tail];
+ queue->tail = (queue->tail + 1) & (queue->size - 1);
+
+ return result;
+}
+
+static uint32
+pgaio_worker_submission_queue_depth(void)
+{
+ uint32 head;
+ uint32 tail;
+
+ head = io_worker_submission_queue->head;
+ tail = io_worker_submission_queue->tail;
+
+ if (tail > head)
+ head += io_worker_submission_queue->size;
+
+ Assert(head >= tail);
+
+ return head - tail;
+}
+
+static bool
+pgaio_worker_needs_synchronous_execution(PgAioHandle *ioh)
+{
+ return
+ !IsUnderPostmaster
+ || ioh->flags & PGAIO_HF_REFERENCES_LOCAL
+ || !pgaio_io_can_reopen(ioh);
+}
+
+static void
+pgaio_worker_submit_internal(int nios, PgAioHandle *ios[])
+{
+ PgAioHandle *synchronous_ios[PGAIO_SUBMIT_BATCH_SIZE];
+ int nsync = 0;
+ Latch *wakeup = NULL;
+ int worker;
+
+ Assert(nios <= PGAIO_SUBMIT_BATCH_SIZE);
+
+ LWLockAcquire(AioWorkerSubmissionQueueLock, LW_EXCLUSIVE);
+ for (int i = 0; i < nios; ++i)
+ {
+ Assert(!pgaio_worker_needs_synchronous_execution(ios[i]));
+ if (!pgaio_worker_submission_queue_insert(ios[i]))
+ {
+ /*
+ * We'll do it synchronously, but only after we've sent as many as
+ * we can to workers, to maximize concurrency.
+ */
+ synchronous_ios[nsync++] = ios[i];
+ continue;
+ }
+
+ if (wakeup == NULL)
+ {
+ /* Choose an idle worker to wake up if we haven't already. */
+ worker = pgaio_choose_idle_worker();
+ if (worker >= 0)
+ wakeup = io_worker_control->workers[worker].latch;
+
+ pgaio_debug_io(DEBUG4, ios[i],
+ "choosing worker %d",
+ worker);
+ }
+ }
+ LWLockRelease(AioWorkerSubmissionQueueLock);
+
+ if (wakeup)
+ SetLatch(wakeup);
+
+ /* Run whatever is left synchronously. */
+ if (nsync > 0)
+ {
+ for (int i = 0; i < nsync; ++i)
+ {
+ pgaio_io_perform_synchronously(synchronous_ios[i]);
+ }
+ }
+}
+
+static int
+pgaio_worker_submit(uint16 num_staged_ios, PgAioHandle **staged_ios)
+{
+ for (int i = 0; i < num_staged_ios; i++)
+ {
+ PgAioHandle *ioh = staged_ios[i];
+
+ pgaio_io_prepare_submit(ioh);
+ }
+
+ pgaio_worker_submit_internal(num_staged_ios, staged_ios);
+
+ return num_staged_ios;
+}
+
+/*
+ * on_shmem_exit() callback that releases the worker's slot in
+ * io_worker_control.
+ */
+static void
+pgaio_worker_die(int code, Datum arg)
+{
+ LWLockAcquire(AioWorkerSubmissionQueueLock, LW_EXCLUSIVE);
+ Assert(io_worker_control->workers[MyIoWorkerId].in_use);
+ Assert(io_worker_control->workers[MyIoWorkerId].latch == MyLatch);
+
+ io_worker_control->workers[MyIoWorkerId].in_use = false;
+ io_worker_control->workers[MyIoWorkerId].latch = NULL;
+ LWLockRelease(AioWorkerSubmissionQueueLock);
+}
+
+/*
+ * Register the worker in shared memory, assign MyWorkerId and register a
+ * shutdown callback to release registration.
+ */
+static void
+pgaio_worker_register(void)
+{
+ MyIoWorkerId = -1;
+
+ /*
+ * XXX: This could do with more fine-grained locking. But it's also not
+ * very common for the number of workers to change at the moment...
+ */
+ LWLockAcquire(AioWorkerSubmissionQueueLock, LW_EXCLUSIVE);
+
+ for (int i = 0; i < io_workers; ++i)
+ {
+ if (!io_worker_control->workers[i].in_use)
+ {
+ Assert(io_worker_control->workers[i].latch == NULL);
+ io_worker_control->workers[i].in_use = true;
+ MyIoWorkerId = i;
+ break;
+ }
+ else
+ Assert(io_worker_control->workers[i].latch != NULL);
+ }
+
+ if (MyIoWorkerId == -1)
+ elog(ERROR, "couldn't find a free worker slot");
+
+ io_worker_control->idle_worker_mask |= (UINT64_C(1) << MyIoWorkerId);
+ io_worker_control->workers[MyIoWorkerId].latch = MyLatch;
+ LWLockRelease(AioWorkerSubmissionQueueLock);
+
+ on_shmem_exit(pgaio_worker_die, 0);
+}
+
void
IoWorkerMain(const void *startup_data, size_t startup_data_len)
{
sigjmp_buf local_sigjmp_buf;
+ PgAioHandle *volatile error_ioh = NULL;
+ volatile int error_errno = 0;
+ char cmd[128];
MyBackendType = B_IO_WORKER;
AuxiliaryProcessMainCommon();
@@ -53,6 +368,12 @@ IoWorkerMain(const void *startup_data, size_t startup_data_len)
pqsignal(SIGUSR1, procsignal_sigusr1_handler);
pqsignal(SIGUSR2, SignalHandlerForShutdownRequest);
+ /* also registers a shutdown callback to unregister */
+ pgaio_worker_register();
+
+ sprintf(cmd, "io worker: %d", MyIoWorkerId);
+ set_ps_display(cmd);
+
/* see PostgresMain() */
if (sigsetjmp(local_sigjmp_buf, 1) != 0)
{
@@ -61,6 +382,27 @@ IoWorkerMain(const void *startup_data, size_t startup_data_len)
EmitErrorReport();
+ /*
+ * In the - very unlikely - case that the IO failed in a way that
+ * raises an error we need to mark the IO as failed.
+ *
+ * Need to do just enough error recovery so that we can mark the IO as
+ * failed and then exit (postmaster will start a new worker).
+ */
+ LWLockReleaseAll();
+
+ if (error_ioh != NULL)
+ {
+ /* should never fail without setting error_errno */
+ Assert(error_errno != 0);
+
+ errno = error_errno;
+
+ START_CRIT_SECTION();
+ pgaio_io_process_completion(error_ioh, -error_errno);
+ END_CRIT_SECTION();
+ }
+
proc_exit(1);
}
@@ -71,9 +413,89 @@ IoWorkerMain(const void *startup_data, size_t startup_data_len)
while (!ShutdownRequestPending)
{
- WaitLatch(MyLatch, WL_LATCH_SET | WL_EXIT_ON_PM_DEATH, -1,
- WAIT_EVENT_IO_WORKER_MAIN);
- ResetLatch(MyLatch);
+ uint32 io_index;
+ Latch *latches[IO_WORKER_WAKEUP_FANOUT];
+ int nlatches = 0;
+ int nwakeups = 0;
+ int worker;
+
+ /* Try to get a job to do. */
+ LWLockAcquire(AioWorkerSubmissionQueueLock, LW_EXCLUSIVE);
+ if ((io_index = pgaio_worker_submission_queue_consume()) == UINT32_MAX)
+ {
+ /*
+ * Nothing to do. Mark self idle.
+ *
+ * XXX: Invent some kind of back pressure to reduce useless
+ * wakeups?
+ */
+ io_worker_control->idle_worker_mask |= (UINT64_C(1) << MyIoWorkerId);
+ }
+ else
+ {
+ /* Got one. Clear idle flag. */
+ io_worker_control->idle_worker_mask &= ~(UINT64_C(1) << MyIoWorkerId);
+
+ /* See if we can wake up some peers. */
+ nwakeups = Min(pgaio_worker_submission_queue_depth(),
+ IO_WORKER_WAKEUP_FANOUT);
+ for (int i = 0; i < nwakeups; ++i)
+ {
+ if ((worker = pgaio_choose_idle_worker()) < 0)
+ break;
+ latches[nlatches++] = io_worker_control->workers[worker].latch;
+ }
+ }
+ LWLockRelease(AioWorkerSubmissionQueueLock);
+
+ for (int i = 0; i < nlatches; ++i)
+ SetLatch(latches[i]);
+
+ if (io_index != UINT32_MAX)
+ {
+ PgAioHandle *ioh = NULL;
+
+ ioh = &pgaio_ctl->io_handles[io_index];
+ error_ioh = ioh;
+
+ pgaio_debug_io(DEBUG4, ioh,
+ "worker %d processing IO",
+ MyIoWorkerId);
+
+ /*
+ * It's very unlikely, but possible, that reopen fails. E.g. due
+ * to memory allocations failing or file permissions changing or
+ * such. In that case we need to fail the IO.
+ *
+ * There's not really a good errno we can report here.
+ */
+ error_errno = ENOENT;
+ pgaio_io_reopen(ioh);
+
+ /*
+ * To be able to exercise the reopen-fails path, allow injection
+ * points to trigger a failure at this point.
+ */
+ pgaio_io_call_inj(ioh, "AIO_WORKER_AFTER_REOPEN");
+
+ error_errno = 0;
+ error_ioh = NULL;
+
+ /*
+ * We don't expect this to ever fail with ERROR or FATAL, no need
+ * to keep error_ioh set to the IO.
+ * pgaio_io_perform_synchronously() contains a critical section to
+ * ensure we don't accidentally fail.
+ */
+ pgaio_io_perform_synchronously(ioh);
+ }
+ else
+ {
+ WaitLatch(MyLatch, WL_LATCH_SET | WL_EXIT_ON_PM_DEATH, -1,
+ WAIT_EVENT_IO_WORKER_MAIN);
+ ResetLatch(MyLatch);
+ }
+
CHECK_FOR_INTERRUPTS();
}
@@ -83,6 +505,5 @@ IoWorkerMain(const void *startup_data, size_t startup_data_len)
bool
pgaio_workers_enabled(void)
{
- /* placeholder for future commit */
- return false;
+ return io_method == IOMETHOD_WORKER;
}
diff --git a/src/backend/utils/activity/wait_event_names.txt b/src/backend/utils/activity/wait_event_names.txt
index 3f6dc3876b4..9fa12a555e8 100644
--- a/src/backend/utils/activity/wait_event_names.txt
+++ b/src/backend/utils/activity/wait_event_names.txt
@@ -348,6 +348,7 @@ WALSummarizer "Waiting to read or update WAL summarization state."
DSMRegistry "Waiting to read or update the dynamic shared memory registry."
InjectionPoint "Waiting to read or update information related to injection points."
SerialControl "Waiting to read or update shared <filename>pg_serial</filename> state."
+AioWorkerSubmissionQueue "Waiting to access AIO worker submission queue."
#
# END OF PREDEFINED LWLOCKS (DO NOT CHANGE THIS LINE)
diff --git a/src/backend/utils/misc/postgresql.conf.sample b/src/backend/utils/misc/postgresql.conf.sample
index 1fd56ee5362..7df01d3a589 100644
--- a/src/backend/utils/misc/postgresql.conf.sample
+++ b/src/backend/utils/misc/postgresql.conf.sample
@@ -202,7 +202,7 @@
#maintenance_io_concurrency = 10 # 1-1000; 0 disables prefetching
#io_combine_limit = 128kB # usually 1-32 blocks (depends on OS)
-#io_method = sync # sync (change requires restart)
+#io_method = worker # worker, sync (change requires restart)
#io_max_concurrency = -1 # Max number of IOs that one process
# can execute simultaneously
# -1 sets based on shared_buffers
diff --git a/doc/src/sgml/config.sgml b/doc/src/sgml/config.sgml
index bd359e6480d..c71a602a04d 100644
--- a/doc/src/sgml/config.sgml
+++ b/doc/src/sgml/config.sgml
@@ -2676,6 +2676,11 @@ include_dir 'conf.d'
Selects the method for executing asynchronous I/O.
Possible values are:
<itemizedlist>
+ <listitem>
+ <para>
+ <literal>worker</literal> (execute asynchronous I/O using worker processes)
+ </para>
+ </listitem>
<listitem>
<para>
<literal>sync</literal> (execute asynchronous-eligible I/O synchronously)
diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list
index cca85bd8806..08490717c98 100644
--- a/src/tools/pgindent/typedefs.list
+++ b/src/tools/pgindent/typedefs.list
@@ -55,6 +55,9 @@ AggStrategy
AggTransInfo
Aggref
AggregateInstrumentation
+AioWorkerControl
+AioWorkerSlot
+AioWorkerSubmissionQueue
AlenState
Alias
AllocBlock
--
2.48.1.76.g4e746b1a31.dirty
[text/x-diff] v2.7-0005-aio-Add-liburing-dependency.patch (13.4K, ../../4b3f32ug3cayekysqlgspz2qjmeb7lca3gvazayglxr2m3d4dv@il33accgsji7/6-v2.7-0005-aio-Add-liburing-dependency.patch)
download | inline diff:
From 2b58fd8ab0969c452d3984c225e0e0708a78fd0f Mon Sep 17 00:00:00 2001
From: Andres Freund <andres@anarazel.de>
Date: Thu, 20 Feb 2025 17:56:05 -0500
Subject: [PATCH v2.7 05/35] aio: Add liburing dependency
Will be used in a subsequent commit, to implement io_method=io_uring. Kept
separate for easier review.
Reviewed-by: Noah Misch <noah@leadboat.com>
Discussion: https://postgr.es/m/uvrtrknj4kdytuboidbhwclo4gxhswwcpgadptsjvjqcluzmah%40brqs62irg4dt
---
meson.build | 14 ++++
meson_options.txt | 3 +
configure.ac | 11 +++
src/makefiles/meson.build | 3 +
src/include/pg_config.h.in | 3 +
src/backend/Makefile | 7 +-
doc/src/sgml/installation.sgml | 34 ++++++++
configure | 138 +++++++++++++++++++++++++++++++++
.cirrus.tasks.yml | 1 +
src/Makefile.global.in | 4 +
10 files changed, 215 insertions(+), 3 deletions(-)
diff --git a/meson.build b/meson.build
index 13c13748e5d..bed36153c7b 100644
--- a/meson.build
+++ b/meson.build
@@ -950,6 +950,18 @@ endif
+###############################################################
+# Library: liburing
+###############################################################
+
+liburingopt = get_option('liburing')
+liburing = dependency('liburing', required: liburingopt)
+if liburing.found()
+ cdata.set('USE_LIBURING', 1)
+endif
+
+
+
###############################################################
# Library: libxml
###############################################################
@@ -3168,6 +3180,7 @@ backend_both_deps += [
icu_i18n,
ldap,
libintl,
+ liburing,
libxml,
lz4,
pam,
@@ -3823,6 +3836,7 @@ if meson.version().version_compare('>=0.57')
'icu': icu,
'ldap': ldap,
'libcurl': libcurl,
+ 'liburing': liburing,
'libxml': libxml,
'libxslt': libxslt,
'llvm': llvm,
diff --git a/meson_options.txt b/meson_options.txt
index 702c4517145..dd7126da3a7 100644
--- a/meson_options.txt
+++ b/meson_options.txt
@@ -106,6 +106,9 @@ option('libcurl', type : 'feature', value: 'auto',
option('libedit_preferred', type: 'boolean', value: false,
description: 'Prefer BSD Libedit over GNU Readline')
+option('liburing', type : 'feature', value: 'auto',
+ description: 'io_uring support, for asynchronous I/O')
+
option('libxml', type: 'feature', value: 'auto',
description: 'XML support')
diff --git a/configure.ac b/configure.ac
index b6d02f5ecc7..3e2d0dc59c0 100644
--- a/configure.ac
+++ b/configure.ac
@@ -975,6 +975,14 @@ AC_SUBST(with_readline)
PGAC_ARG_BOOL(with, libedit-preferred, no,
[prefer BSD Libedit over GNU Readline])
+#
+# liburing
+#
+AC_MSG_CHECKING([whether to build with liburing support])
+PGAC_ARG_BOOL(with, liburing, no, [io_uring support, for asynchronous I/O],
+ [AC_DEFINE([USE_LIBURING], 1, [Define to build with io_uring support. (--with-liburing)])])
+AC_MSG_RESULT([$with_liburing])
+AC_SUBST(with_liburing)
#
# UUID library
@@ -1463,6 +1471,9 @@ elif test "$with_uuid" = ossp ; then
fi
AC_SUBST(UUID_LIBS)
+if test "$with_liburing" = yes; then
+ PKG_CHECK_MODULES(LIBURING, liburing)
+fi
##
## Header files
diff --git a/src/makefiles/meson.build b/src/makefiles/meson.build
index 60e13d50235..46d8da070e8 100644
--- a/src/makefiles/meson.build
+++ b/src/makefiles/meson.build
@@ -199,6 +199,8 @@ pgxs_empty = [
'PTHREAD_CFLAGS', 'PTHREAD_LIBS',
'ICU_LIBS',
+
+ 'LIBURING_CFLAGS', 'LIBURING_LIBS',
]
if host_system == 'windows' and cc.get_argument_syntax() != 'msvc'
@@ -230,6 +232,7 @@ pgxs_deps = {
'icu': icu,
'ldap': ldap,
'libcurl': libcurl,
+ 'liburing': liburing,
'libxml': libxml,
'libxslt': libxslt,
'llvm': llvm,
diff --git a/src/include/pg_config.h.in b/src/include/pg_config.h.in
index db6454090d2..f2422241133 100644
--- a/src/include/pg_config.h.in
+++ b/src/include/pg_config.h.in
@@ -672,6 +672,9 @@
/* Define to 1 to build with libcurl support. (--with-libcurl) */
#undef USE_LIBCURL
+/* Define to build with io_uring support. (--with-liburing) */
+#undef USE_LIBURING
+
/* Define to 1 to build with XML support. (--with-libxml) */
#undef USE_LIBXML
diff --git a/src/backend/Makefile b/src/backend/Makefile
index 42d4a28e5aa..7344c8c7f5c 100644
--- a/src/backend/Makefile
+++ b/src/backend/Makefile
@@ -43,9 +43,10 @@ OBJS = \
$(top_builddir)/src/common/libpgcommon_srv.a \
$(top_builddir)/src/port/libpgport_srv.a
-# We put libpgport and libpgcommon into OBJS, so remove it from LIBS; also add
-# libldap and ICU
-LIBS := $(filter-out -lpgport -lpgcommon, $(LIBS)) $(LDAP_LIBS_BE) $(ICU_LIBS)
+# We put libpgport and libpgcommon into OBJS, so remove it from LIBS.
+LIBS := $(filter-out -lpgport -lpgcommon, $(LIBS))
+# The backend conditionally needs libraries that most executables don't need.
+LIBS += $(LDAP_LIBS_BE) $(ICU_LIBS) $(LIBURING_LIBS)
# The backend doesn't need everything that's in LIBS, however
LIBS := $(filter-out -lreadline -ledit -ltermcap -lncurses -lcurses, $(LIBS))
diff --git a/doc/src/sgml/installation.sgml b/doc/src/sgml/installation.sgml
index e076cefa3b9..cc28f041330 100644
--- a/doc/src/sgml/installation.sgml
+++ b/doc/src/sgml/installation.sgml
@@ -1156,6 +1156,24 @@ build-postgresql:
</listitem>
</varlistentry>
+ <varlistentry id="configure-option-with-liburing">
+ <term><option>--with-liburing</option></term>
+ <listitem>
+ <para>
+ Build with liburing, enabling io_uring support for asynchronous I/O.
+ </para>
+ <para>
+ To detect the required compiler and linker options, PostgreSQL will
+ query <command>pkg-config</command>.
+ </para>
+ <para>
+ To use a liburing installation that is in an unusual location, you
+ can set <command>pkg-config</command>-related environment
+ variables (see its documentation).
+ </para>
+ </listitem>
+ </varlistentry>
+
<varlistentry id="configure-option-with-libxml">
<term><option>--with-libxml</option></term>
<listitem>
@@ -2611,6 +2629,22 @@ ninja install
</listitem>
</varlistentry>
+ <varlistentry id="configure-with-liburing-meson">
+ <term><option>-Dliburing={ auto | enabled | disabled }</option></term>
+ <listitem>
+ <para>
+ Build with liburing, enabling io_uring support for asynchronous I/O.
+ Defaults to auto.
+ </para>
+
+ <para>
+ To use a liburing installation that is in an unusual location, you
+ can set <command>pkg-config</command>-related environment
+ variables (see its documentation).
+ </para>
+ </listitem>
+ </varlistentry>
+
<varlistentry id="configure-with-libxml-meson">
<term><option>-Dlibxml={ auto | enabled | disabled }</option></term>
<listitem>
diff --git a/configure b/configure
index 93fddd69981..8ffcdfc2e1c 100755
--- a/configure
+++ b/configure
@@ -651,6 +651,8 @@ LIBOBJS
OPENSSL
ZSTD
LZ4
+LIBURING_LIBS
+LIBURING_CFLAGS
UUID_LIBS
LDAP_LIBS_BE
LDAP_LIBS_FE
@@ -712,6 +714,7 @@ LIBCURL_LIBS
LIBCURL_CFLAGS
with_libcurl
with_uuid
+with_liburing
with_readline
with_systemd
with_selinux
@@ -865,6 +868,7 @@ with_selinux
with_systemd
with_readline
with_libedit_preferred
+with_liburing
with_uuid
with_ossp_uuid
with_libcurl
@@ -911,6 +915,8 @@ LDFLAGS_EX
LDFLAGS_SL
PERL
PYTHON
+LIBURING_CFLAGS
+LIBURING_LIBS
MSGFMT
TCLSH'
@@ -1578,6 +1584,7 @@ Optional Packages:
--without-readline do not use GNU Readline nor BSD Libedit for editing
--with-libedit-preferred
prefer BSD Libedit over GNU Readline
+ --with-liburing io_uring support, for asynchronous I/O
--with-uuid=LIB build contrib/uuid-ossp using LIB (bsd,e2fs,ossp)
--with-ossp-uuid obsolete spelling of --with-uuid=ossp
--with-libcurl build with libcurl support
@@ -1629,6 +1636,10 @@ Some influential environment variables:
LDFLAGS_SL extra linker flags for linking shared libraries only
PERL Perl program
PYTHON Python program
+ LIBURING_CFLAGS
+ C compiler flags for LIBURING, overriding pkg-config
+ LIBURING_LIBS
+ linker flags for LIBURING, overriding pkg-config
MSGFMT msgfmt program for NLS
TCLSH Tcl interpreter program (tclsh)
@@ -8692,6 +8703,40 @@ fi
+#
+# liburing
+#
+{ $as_echo "$as_me:${as_lineno-$LINENO}: checking whether to build with liburing support" >&5
+$as_echo_n "checking whether to build with liburing support... " >&6; }
+
+
+
+# Check whether --with-liburing was given.
+if test "${with_liburing+set}" = set; then :
+ withval=$with_liburing;
+ case $withval in
+ yes)
+
+$as_echo "#define USE_LIBURING 1" >>confdefs.h
+
+ ;;
+ no)
+ :
+ ;;
+ *)
+ as_fn_error $? "no argument expected for --with-liburing option" "$LINENO" 5
+ ;;
+ esac
+
+else
+ with_liburing=no
+
+fi
+
+
+{ $as_echo "$as_me:${as_lineno-$LINENO}: result: $with_liburing" >&5
+$as_echo "$with_liburing" >&6; }
+
#
# UUID library
@@ -13444,6 +13489,99 @@ fi
fi
+if test "$with_liburing" = yes; then
+
+pkg_failed=no
+{ $as_echo "$as_me:${as_lineno-$LINENO}: checking for liburing" >&5
+$as_echo_n "checking for liburing... " >&6; }
+
+if test -n "$LIBURING_CFLAGS"; then
+ pkg_cv_LIBURING_CFLAGS="$LIBURING_CFLAGS"
+ elif test -n "$PKG_CONFIG"; then
+ if test -n "$PKG_CONFIG" && \
+ { { $as_echo "$as_me:${as_lineno-$LINENO}: \$PKG_CONFIG --exists --print-errors \"liburing\""; } >&5
+ ($PKG_CONFIG --exists --print-errors "liburing") 2>&5
+ ac_status=$?
+ $as_echo "$as_me:${as_lineno-$LINENO}: \$? = $ac_status" >&5
+ test $ac_status = 0; }; then
+ pkg_cv_LIBURING_CFLAGS=`$PKG_CONFIG --cflags "liburing" 2>/dev/null`
+ test "x$?" != "x0" && pkg_failed=yes
+else
+ pkg_failed=yes
+fi
+ else
+ pkg_failed=untried
+fi
+if test -n "$LIBURING_LIBS"; then
+ pkg_cv_LIBURING_LIBS="$LIBURING_LIBS"
+ elif test -n "$PKG_CONFIG"; then
+ if test -n "$PKG_CONFIG" && \
+ { { $as_echo "$as_me:${as_lineno-$LINENO}: \$PKG_CONFIG --exists --print-errors \"liburing\""; } >&5
+ ($PKG_CONFIG --exists --print-errors "liburing") 2>&5
+ ac_status=$?
+ $as_echo "$as_me:${as_lineno-$LINENO}: \$? = $ac_status" >&5
+ test $ac_status = 0; }; then
+ pkg_cv_LIBURING_LIBS=`$PKG_CONFIG --libs "liburing" 2>/dev/null`
+ test "x$?" != "x0" && pkg_failed=yes
+else
+ pkg_failed=yes
+fi
+ else
+ pkg_failed=untried
+fi
+
+
+
+if test $pkg_failed = yes; then
+ { $as_echo "$as_me:${as_lineno-$LINENO}: result: no" >&5
+$as_echo "no" >&6; }
+
+if $PKG_CONFIG --atleast-pkgconfig-version 0.20; then
+ _pkg_short_errors_supported=yes
+else
+ _pkg_short_errors_supported=no
+fi
+ if test $_pkg_short_errors_supported = yes; then
+ LIBURING_PKG_ERRORS=`$PKG_CONFIG --short-errors --print-errors --cflags --libs "liburing" 2>&1`
+ else
+ LIBURING_PKG_ERRORS=`$PKG_CONFIG --print-errors --cflags --libs "liburing" 2>&1`
+ fi
+ # Put the nasty error message in config.log where it belongs
+ echo "$LIBURING_PKG_ERRORS" >&5
+
+ as_fn_error $? "Package requirements (liburing) were not met:
+
+$LIBURING_PKG_ERRORS
+
+Consider adjusting the PKG_CONFIG_PATH environment variable if you
+installed software in a non-standard prefix.
+
+Alternatively, you may set the environment variables LIBURING_CFLAGS
+and LIBURING_LIBS to avoid the need to call pkg-config.
+See the pkg-config man page for more details." "$LINENO" 5
+elif test $pkg_failed = untried; then
+ { $as_echo "$as_me:${as_lineno-$LINENO}: result: no" >&5
+$as_echo "no" >&6; }
+ { { $as_echo "$as_me:${as_lineno-$LINENO}: error: in \`$ac_pwd':" >&5
+$as_echo "$as_me: error: in \`$ac_pwd':" >&2;}
+as_fn_error $? "The pkg-config script could not be found or is too old. Make sure it
+is in your PATH or set the PKG_CONFIG environment variable to the full
+path to pkg-config.
+
+Alternatively, you may set the environment variables LIBURING_CFLAGS
+and LIBURING_LIBS to avoid the need to call pkg-config.
+See the pkg-config man page for more details.
+
+To get pkg-config, see <http://pkg-config.freedesktop.org/>.
+See \`config.log' for more details" "$LINENO" 5; }
+else
+ LIBURING_CFLAGS=$pkg_cv_LIBURING_CFLAGS
+ LIBURING_LIBS=$pkg_cv_LIBURING_LIBS
+ { $as_echo "$as_me:${as_lineno-$LINENO}: result: yes" >&5
+$as_echo "yes" >&6; }
+
+fi
+fi
##
## Header files
diff --git a/.cirrus.tasks.yml b/.cirrus.tasks.yml
index 5849cbb839a..10d277f5659 100644
--- a/.cirrus.tasks.yml
+++ b/.cirrus.tasks.yml
@@ -471,6 +471,7 @@ task:
--enable-cassert --enable-injection-points --enable-debug \
--enable-tap-tests --enable-nls \
--with-segsize-blocks=6 \
+ --with-liburing \
\
${LINUX_CONFIGURE_FEATURES} \
\
diff --git a/src/Makefile.global.in b/src/Makefile.global.in
index 3b620bac5ac..d45f5fe4ed7 100644
--- a/src/Makefile.global.in
+++ b/src/Makefile.global.in
@@ -191,6 +191,7 @@ with_gssapi = @with_gssapi@
with_krb_srvnam = @with_krb_srvnam@
with_ldap = @with_ldap@
with_libcurl = @with_libcurl@
+with_liburing = @with_liburing@
with_libxml = @with_libxml@
with_libxslt = @with_libxslt@
with_llvm = @with_llvm@
@@ -217,6 +218,9 @@ krb_srvtab = @krb_srvtab@
ICU_CFLAGS = @ICU_CFLAGS@
ICU_LIBS = @ICU_LIBS@
+LIBURING_CFLAGS = @LIBURING_CFLAGS@
+LIBURING_LIBS = @LIBURING_LIBS@
+
TCLSH = @TCLSH@
TCL_LIBS = @TCL_LIBS@
TCL_LIB_SPEC = @TCL_LIB_SPEC@
--
2.48.1.76.g4e746b1a31.dirty
[text/x-diff] v2.7-0006-aio-Add-io_method-io_uring.patch (20.2K, ../../4b3f32ug3cayekysqlgspz2qjmeb7lca3gvazayglxr2m3d4dv@il33accgsji7/7-v2.7-0006-aio-Add-io_method-io_uring.patch)
download | inline diff:
From f5004f8efb285f3a711a5a26d9d124cf438d372b Mon Sep 17 00:00:00 2001
From: Andres Freund <andres@anarazel.de>
Date: Tue, 11 Feb 2025 14:29:37 -0500
Subject: [PATCH v2.7 06/35] aio: Add io_method=io_uring
io_uring, can be considerably faster than io_method=worker. It is, however,
linux specific and requires an additional compile-time dependency (liburing).
Reviewed-by: Jakub Wartak <jakub.wartak@enterprisedb.com>
Discussion: https://postgr.es/m/uvrtrknj4kdytuboidbhwclo4gxhswwcpgadptsjvjqcluzmah%40brqs62irg4dt
Discussion: https://postgr.es/m/20210223100344.llw5an2aklengrmn@alap3.anarazel.de
Discussion: https://postgr.es/m/stj36ea6yyhoxtqkhpieia2z4krnam7qyetc57rfezgk4zgapf@gcnactj4z56m
---
src/include/storage/aio.h | 3 +
src/include/storage/aio_internal.h | 3 +
src/include/storage/lwlock.h | 1 +
src/backend/storage/aio/Makefile | 1 +
src/backend/storage/aio/aio.c | 6 +
src/backend/storage/aio/meson.build | 1 +
src/backend/storage/aio/method_io_uring.c | 440 ++++++++++++++++++
src/backend/storage/lmgr/lwlock.c | 1 +
.../utils/activity/wait_event_names.txt | 2 +
src/backend/utils/misc/postgresql.conf.sample | 3 +-
doc/src/sgml/config.sgml | 6 +
.cirrus.tasks.yml | 3 +
src/tools/pgindent/typedefs.list | 1 +
13 files changed, 470 insertions(+), 1 deletion(-)
create mode 100644 src/backend/storage/aio/method_io_uring.c
diff --git a/src/include/storage/aio.h b/src/include/storage/aio.h
index 769ea33ea96..8bc43c33505 100644
--- a/src/include/storage/aio.h
+++ b/src/include/storage/aio.h
@@ -28,6 +28,9 @@ typedef enum IoMethod
{
IOMETHOD_SYNC = 0,
IOMETHOD_WORKER,
+#ifdef USE_LIBURING
+ IOMETHOD_IO_URING,
+#endif
} IoMethod;
/* We'll default to worker based execution. */
diff --git a/src/include/storage/aio_internal.h b/src/include/storage/aio_internal.h
index b560f69076b..2580e34e43b 100644
--- a/src/include/storage/aio_internal.h
+++ b/src/include/storage/aio_internal.h
@@ -386,6 +386,9 @@ extern PgAioHandle *pgaio_inj_io_get(void);
/* Declarations for the tables of function pointers exposed by each IO method. */
extern PGDLLIMPORT const IoMethodOps pgaio_sync_ops;
extern PGDLLIMPORT const IoMethodOps pgaio_worker_ops;
+#ifdef USE_LIBURING
+extern PGDLLIMPORT const IoMethodOps pgaio_uring_ops;
+#endif
extern PGDLLIMPORT const IoMethodOps *pgaio_method_ops;
extern PGDLLIMPORT PgAioCtl *pgaio_ctl;
diff --git a/src/include/storage/lwlock.h b/src/include/storage/lwlock.h
index ffa03189e2d..4df1d25c045 100644
--- a/src/include/storage/lwlock.h
+++ b/src/include/storage/lwlock.h
@@ -218,6 +218,7 @@ typedef enum BuiltinTrancheIds
LWTRANCHE_SUBTRANS_SLRU,
LWTRANCHE_XACT_SLRU,
LWTRANCHE_PARALLEL_VACUUM_DSA,
+ LWTRANCHE_AIO_URING_COMPLETION,
LWTRANCHE_FIRST_USER_DEFINED,
} BuiltinTrancheIds;
diff --git a/src/backend/storage/aio/Makefile b/src/backend/storage/aio/Makefile
index f51c34a37f8..c06c50771e0 100644
--- a/src/backend/storage/aio/Makefile
+++ b/src/backend/storage/aio/Makefile
@@ -14,6 +14,7 @@ OBJS = \
aio_init.o \
aio_io.o \
aio_target.o \
+ method_io_uring.o \
method_sync.o \
method_worker.o \
read_stream.o
diff --git a/src/backend/storage/aio/aio.c b/src/backend/storage/aio/aio.c
index 94fbe0aea80..2fbc380b758 100644
--- a/src/backend/storage/aio/aio.c
+++ b/src/backend/storage/aio/aio.c
@@ -65,6 +65,9 @@ static void pgaio_io_wait(PgAioHandle *ioh, uint64 ref_generation);
const struct config_enum_entry io_method_options[] = {
{"sync", IOMETHOD_SYNC, false},
{"worker", IOMETHOD_WORKER, false},
+#ifdef USE_LIBURING
+ {"io_uring", IOMETHOD_IO_URING, false},
+#endif
{NULL, 0, false}
};
@@ -82,6 +85,9 @@ PgAioBackend *pgaio_my_backend;
static const IoMethodOps *const pgaio_method_ops_table[] = {
[IOMETHOD_SYNC] = &pgaio_sync_ops,
[IOMETHOD_WORKER] = &pgaio_worker_ops,
+#ifdef USE_LIBURING
+ [IOMETHOD_IO_URING] = &pgaio_uring_ops,
+#endif
};
/* callbacks for the configured io_method, set by assign_io_method */
diff --git a/src/backend/storage/aio/meson.build b/src/backend/storage/aio/meson.build
index 74f94c6e40b..2f0f03d8071 100644
--- a/src/backend/storage/aio/meson.build
+++ b/src/backend/storage/aio/meson.build
@@ -6,6 +6,7 @@ backend_sources += files(
'aio_init.c',
'aio_io.c',
'aio_target.c',
+ 'method_io_uring.c',
'method_sync.c',
'method_worker.c',
'read_stream.c',
diff --git a/src/backend/storage/aio/method_io_uring.c b/src/backend/storage/aio/method_io_uring.c
new file mode 100644
index 00000000000..588177eaa4f
--- /dev/null
+++ b/src/backend/storage/aio/method_io_uring.c
@@ -0,0 +1,440 @@
+/*-------------------------------------------------------------------------
+ *
+ * method_io_uring.c
+ * AIO - perform AIO using Linux' io_uring
+ *
+ * For now we create one io_uring instance for each backend. These io_uring
+ * instances have to be created in postmaster, during startup, to allow other
+ * backends to process IO completions, if the issuing backend is currently
+ * busy doing other things. Other backends may not use another backend's
+ * io_uring instance to submit IO, that'd require additional locking that
+ * would likely be harmful for performance.
+ *
+ * We likely will want to introduce a backend-local io_uring instance in the
+ * future, e.g. for FE/BE network IO.
+ *
+ * Portions Copyright (c) 1996-2025, PostgreSQL Global Development Group
+ * Portions Copyright (c) 1994, Regents of the University of California
+ *
+ * IDENTIFICATION
+ * src/backend/storage/aio/method_io_uring.c
+ *
+ *-------------------------------------------------------------------------
+ */
+
+#include "postgres.h"
+
+#ifdef USE_LIBURING
+
+#include <liburing.h>
+
+#include "pgstat.h"
+#include "port/pg_iovec.h"
+#include "storage/aio.h"
+#include "storage/aio_internal.h"
+#include "storage/fd.h"
+#include "storage/proc.h"
+#include "storage/shmem.h"
+
+
+/* number of completions processed at once */
+#define PGAIO_MAX_LOCAL_COMPLETED_IO 32
+
+
+/* Entry points for IoMethodOps. */
+static size_t pgaio_uring_shmem_size(void);
+static void pgaio_uring_shmem_init(bool first_time);
+static void pgaio_uring_init_backend(void);
+static int pgaio_uring_submit(uint16 num_staged_ios, PgAioHandle **staged_ios);
+static void pgaio_uring_wait_one(PgAioHandle *ioh, uint64 ref_generation);
+
+/* helper functions */
+static void pgaio_uring_sq_from_io(PgAioHandle *ioh, struct io_uring_sqe *sqe);
+
+
+const IoMethodOps pgaio_uring_ops = {
+ .shmem_size = pgaio_uring_shmem_size,
+ .shmem_init = pgaio_uring_shmem_init,
+ .init_backend = pgaio_uring_init_backend,
+
+ .submit = pgaio_uring_submit,
+ .wait_one = pgaio_uring_wait_one,
+};
+
+/*
+ * Per-backend state when using io_method=io_uring
+ *
+ * Align the whole struct to a cacheline boundary, to prevent false sharing
+ * between completion_lock and prior backend's io_uring_ring.
+ */
+typedef struct pg_attribute_aligned (PG_CACHE_LINE_SIZE)
+PgAioUringContext
+{
+ /*
+ * Multiple backends can process completions for this backend's io_uring
+ * instance (e.g. when the backend issuing IO is busy doing something
+ * else). To make that safe we have to ensure that only a single backend
+ * gets io completions from the io_uring instance at a time.
+ */
+ LWLock completion_lock;
+
+ struct io_uring io_uring_ring;
+} PgAioUringContext;
+
+/* PgAioUringContexts for all backends */
+static PgAioUringContext *pgaio_uring_contexts;
+
+/* the current backend's context */
+static PgAioUringContext *pgaio_my_uring_context;
+
+
+static uint32
+pgaio_uring_procs(void)
+{
+ /*
+ * We can subtract MAX_IO_WORKERS here as io workers are never used at the
+ * same time as io_method=io_uring.
+ */
+ return MaxBackends + NUM_AUXILIARY_PROCS - MAX_IO_WORKERS;
+}
+
+static Size
+pgaio_uring_context_shmem_size(void)
+{
+ return mul_size(pgaio_uring_procs(), sizeof(PgAioUringContext));
+}
+
+static size_t
+pgaio_uring_shmem_size(void)
+{
+ return pgaio_uring_context_shmem_size();
+}
+
+static void
+pgaio_uring_shmem_init(bool first_time)
+{
+ uint32 TotalProcs = MaxBackends + NUM_AUXILIARY_PROCS - MAX_IO_WORKERS;
+ bool found;
+
+ pgaio_uring_contexts = (PgAioUringContext *)
+ ShmemInitStruct("AioUring", pgaio_uring_shmem_size(), &found);
+
+ if (found)
+ return;
+
+ for (int contextno = 0; contextno < TotalProcs; contextno++)
+ {
+ PgAioUringContext *context = &pgaio_uring_contexts[contextno];
+ int ret;
+
+ /*
+ * FIXME: Right now a high TotalProcs will cause problems in two ways:
+ *
+ * - RLIMIT_NOFILE needs to be big enough to allow MaxBackends fds
+ * created in postmaster, with spare space for max_files_per_process
+ * additional FDs
+ *
+ * - set_max_safe_fds() subtracts the number of already used FDs from
+ * max_files_per_process, ending up with a low limit or even erroring
+ * out.
+ *
+ * XXX: Newer versions of io_uring support sharing the workers that
+ * execute some asynchronous IOs between io_uring instances. It might
+ * be worth using that - also need to evaluate if that causes
+ * noticeable additional contention?
+ */
+ ret = io_uring_queue_init(io_max_concurrency, &context->io_uring_ring, 0);
+ if (ret < 0)
+ elog(ERROR, "io_uring_queue_init failed: %s", strerror(-ret));
+
+ LWLockInitialize(&context->completion_lock, LWTRANCHE_AIO_URING_COMPLETION);
+ }
+}
+
+static void
+pgaio_uring_init_backend(void)
+{
+ Assert(MyProcNumber < pgaio_uring_procs());
+
+ pgaio_my_uring_context = &pgaio_uring_contexts[MyProcNumber];
+}
+
+static int
+pgaio_uring_submit(uint16 num_staged_ios, PgAioHandle **staged_ios)
+{
+ struct io_uring *uring_instance = &pgaio_my_uring_context->io_uring_ring;
+ int in_flight_before = dclist_count(&pgaio_my_backend->in_flight_ios);
+
+ Assert(num_staged_ios <= PGAIO_SUBMIT_BATCH_SIZE);
+
+ for (int i = 0; i < num_staged_ios; i++)
+ {
+ PgAioHandle *ioh = staged_ios[i];
+ struct io_uring_sqe *sqe;
+
+ sqe = io_uring_get_sqe(uring_instance);
+
+ if (!sqe)
+ elog(ERROR, "io_uring submission queue is unexpectedly full");
+
+ pgaio_io_prepare_submit(ioh);
+ pgaio_uring_sq_from_io(ioh, sqe);
+
+ /*
+ * io_uring executes IO in process context if possible. That's
+ * generally good, as it reduces context switching. When performing a
+ * lot of buffered IO that means that copying between page cache and
+ * userspace memory happens in the foreground, as it can't be
+ * offloaded to DMA hardware as is possible when using direct IO. When
+ * executing a lot of buffered IO this causes io_uring to be slower
+ * than worker mode, as worker mode parallelizes the copying. io_uring
+ * can be told to offload work to worker threads instead.
+ *
+ * If an IO is buffered IO and we already have IOs in flight or
+ * multiple IOs are being submitted, we thus tell io_uring to execute
+ * the IO in the background. We don't do so for the first few IOs
+ * being submitted as executing in this process' context has lower
+ * latency.
+ */
+ if (in_flight_before > 4 && (ioh->flags & PGAIO_HF_BUFFERED))
+ io_uring_sqe_set_flags(sqe, IOSQE_ASYNC);
+
+ in_flight_before++;
+ }
+
+ while (true)
+ {
+ int ret;
+
+ pgstat_report_wait_start(WAIT_EVENT_AIO_IO_URING_SUBMIT);
+ ret = io_uring_submit(uring_instance);
+ pgstat_report_wait_end();
+
+ if (ret == -EINTR)
+ {
+ pgaio_debug(DEBUG3,
+ "aio method uring: submit EINTR, nios: %d",
+ num_staged_ios);
+ }
+ else if (ret < 0)
+ {
+ /*
+ * The io_uring_enter() manpage suggests that the appropriate
+ * reaction to EAGAIN is:
+ *
+ * "The application should wait for some completions and try
+ * again"
+ *
+ * However, it seems unlikely that that would help in our case, as
+ * we apply a low limit to the number of outstanding IOs and thus
+ * also outstanding completions, making it unlikely that we'd get
+ * EAGAIN while the OS is in good working order.
+ *
+ * Additionally, it would be problematic to just wait here, our
+ * caller might hold critical locks. It'd possibly lead to
+ * delaying the crash-restart that seems likely to occur when the
+ * kernel is under such heavy memory pressure.
+ *
+ * Update errno to to allow %m to work.
+ */
+ errno = -ret;
+ elog(PANIC, "io_uring submit failed: %m");
+ }
+ else if (ret != num_staged_ios)
+ {
+ /* likely unreachable, but if it is, we would need to re-submit */
+ elog(PANIC, "io_uring submit submitted only %d of %d",
+ ret, num_staged_ios);
+ }
+ else
+ {
+ pgaio_debug(DEBUG4,
+ "aio method uring: submitted %d IOs",
+ num_staged_ios);
+ break;
+ }
+ }
+
+ return num_staged_ios;
+}
+
+static void
+pgaio_uring_drain_locked(PgAioUringContext *context)
+{
+ int ready;
+ int orig_ready;
+
+ Assert(LWLockHeldByMeInMode(&context->completion_lock, LW_EXCLUSIVE));
+
+ /*
+ * Don't drain more events than available right now. Otherwise it's
+ * plausible that one backend could get stuck, for a while, receiving CQEs
+ * without actually processing them.
+ */
+ orig_ready = ready = io_uring_cq_ready(&context->io_uring_ring);
+
+ while (ready > 0)
+ {
+ struct io_uring_cqe *cqes[PGAIO_MAX_LOCAL_COMPLETED_IO];
+ uint32 ncqes;
+
+ START_CRIT_SECTION();
+ ncqes =
+ io_uring_peek_batch_cqe(&context->io_uring_ring,
+ cqes,
+ Min(PGAIO_MAX_LOCAL_COMPLETED_IO, ready));
+ Assert(ncqes <= ready);
+
+ ready -= ncqes;
+
+ for (int i = 0; i < ncqes; i++)
+ {
+ struct io_uring_cqe *cqe = cqes[i];
+ PgAioHandle *ioh;
+
+ ioh = io_uring_cqe_get_data(cqe);
+ io_uring_cqe_seen(&context->io_uring_ring, cqe);
+
+ pgaio_io_process_completion(ioh, cqe->res);
+ }
+
+ END_CRIT_SECTION();
+
+ pgaio_debug(DEBUG3,
+ "drained %d/%d, now expecting %d",
+ ncqes, orig_ready, io_uring_cq_ready(&context->io_uring_ring));
+ }
+}
+
+static void
+pgaio_uring_wait_one(PgAioHandle *ioh, uint64 ref_generation)
+{
+ PgAioHandleState state;
+ ProcNumber owner_procno = ioh->owner_procno;
+ PgAioUringContext *owner_context = &pgaio_uring_contexts[owner_procno];
+ bool expect_cqe;
+ int waited = 0;
+
+ /*
+ * XXX: It would be nice to have a smarter locking scheme, nearly all the
+ * time the backend owning the ring will consume the completions, making
+ * the locking unnecessarily expensive.
+ */
+ LWLockAcquire(&owner_context->completion_lock, LW_EXCLUSIVE);
+
+ while (true)
+ {
+ pgaio_debug_io(DEBUG3, ioh,
+ "wait_one io_gen: %llu, ref_gen: %llu, cycle %d",
+ (long long unsigned) ref_generation,
+ (long long unsigned) ioh->generation,
+ waited);
+
+ if (pgaio_io_was_recycled(ioh, ref_generation, &state) ||
+ state != PGAIO_HS_SUBMITTED)
+ {
+ /* the IO was completed by another backend */
+ break;
+ }
+ else if (io_uring_cq_ready(&owner_context->io_uring_ring))
+ {
+ /* no need to wait in the kernel, io_uring has a completion */
+ expect_cqe = true;
+ }
+ else
+ {
+ int ret;
+ struct io_uring_cqe *cqes;
+
+ /* need to wait in the kernel */
+ pgstat_report_wait_start(WAIT_EVENT_AIO_IO_URING_COMPLETION);
+ ret = io_uring_wait_cqes(&owner_context->io_uring_ring, &cqes, 1, NULL, NULL);
+ pgstat_report_wait_end();
+
+ if (ret == -EINTR)
+ {
+ continue;
+ }
+ else if (ret != 0)
+ {
+ /* see comment after io_uring_submit() */
+ errno = -ret;
+ elog(PANIC, "io_uring wait failed: %m");
+ }
+ else
+ {
+ Assert(cqes != NULL);
+ expect_cqe = true;
+ waited++;
+ }
+ }
+
+ if (expect_cqe)
+ {
+ pgaio_uring_drain_locked(owner_context);
+ }
+ }
+
+ LWLockRelease(&owner_context->completion_lock);
+
+ pgaio_debug(DEBUG3,
+ "wait_one with %d sleeps",
+ waited);
+}
+
+static void
+pgaio_uring_sq_from_io(PgAioHandle *ioh, struct io_uring_sqe *sqe)
+{
+ struct iovec *iov;
+
+ switch (ioh->op)
+ {
+ case PGAIO_OP_READV:
+ iov = &pgaio_ctl->iovecs[ioh->iovec_off];
+ if (ioh->op_data.read.iov_length == 1)
+ {
+ io_uring_prep_read(sqe,
+ ioh->op_data.read.fd,
+ iov->iov_base,
+ iov->iov_len,
+ ioh->op_data.read.offset);
+ }
+ else
+ {
+ io_uring_prep_readv(sqe,
+ ioh->op_data.read.fd,
+ iov,
+ ioh->op_data.read.iov_length,
+ ioh->op_data.read.offset);
+
+ }
+ break;
+
+ case PGAIO_OP_WRITEV:
+ iov = &pgaio_ctl->iovecs[ioh->iovec_off];
+ if (ioh->op_data.write.iov_length == 1)
+ {
+ io_uring_prep_write(sqe,
+ ioh->op_data.write.fd,
+ iov->iov_base,
+ iov->iov_len,
+ ioh->op_data.write.offset);
+ }
+ else
+ {
+ io_uring_prep_writev(sqe,
+ ioh->op_data.write.fd,
+ iov,
+ ioh->op_data.write.iov_length,
+ ioh->op_data.write.offset);
+ }
+ break;
+
+ case PGAIO_OP_INVALID:
+ elog(ERROR, "trying to prepare invalid IO operation for execution");
+ }
+
+ io_uring_sqe_set_data(sqe, ioh);
+}
+
+#endif /* USE_LIBURING */
diff --git a/src/backend/storage/lmgr/lwlock.c b/src/backend/storage/lmgr/lwlock.c
index 5702c35bb91..3df29658f18 100644
--- a/src/backend/storage/lmgr/lwlock.c
+++ b/src/backend/storage/lmgr/lwlock.c
@@ -177,6 +177,7 @@ static const char *const BuiltinTrancheNames[] = {
[LWTRANCHE_SUBTRANS_SLRU] = "SubtransSLRU",
[LWTRANCHE_XACT_SLRU] = "XactSLRU",
[LWTRANCHE_PARALLEL_VACUUM_DSA] = "ParallelVacuumDSA",
+ [LWTRANCHE_AIO_URING_COMPLETION] = "AioUringCompletion",
};
StaticAssertDecl(lengthof(BuiltinTrancheNames) ==
diff --git a/src/backend/utils/activity/wait_event_names.txt b/src/backend/utils/activity/wait_event_names.txt
index 9fa12a555e8..44eef41f841 100644
--- a/src/backend/utils/activity/wait_event_names.txt
+++ b/src/backend/utils/activity/wait_event_names.txt
@@ -192,6 +192,8 @@ ABI_compatibility:
Section: ClassName - WaitEventIO
+AIO_IO_URING_SUBMIT "Waiting for IO submission via io_uring."
+AIO_IO_URING_COMPLETION "Waiting for IO completion via io_uring."
AIO_IO_COMPLETION "Waiting for IO completion."
BASEBACKUP_READ "Waiting for base backup to read from a file."
BASEBACKUP_SYNC "Waiting for data written by a base backup to reach durable storage."
diff --git a/src/backend/utils/misc/postgresql.conf.sample b/src/backend/utils/misc/postgresql.conf.sample
index 7df01d3a589..c9e9e850a99 100644
--- a/src/backend/utils/misc/postgresql.conf.sample
+++ b/src/backend/utils/misc/postgresql.conf.sample
@@ -202,7 +202,8 @@
#maintenance_io_concurrency = 10 # 1-1000; 0 disables prefetching
#io_combine_limit = 128kB # usually 1-32 blocks (depends on OS)
-#io_method = worker # worker, sync (change requires restart)
+#io_method = worker # worker, io_uring, sync
+ # (change requires restart)
#io_max_concurrency = -1 # Max number of IOs that one process
# can execute simultaneously
# -1 sets based on shared_buffers
diff --git a/doc/src/sgml/config.sgml b/doc/src/sgml/config.sgml
index c71a602a04d..fdc8f5c1fb0 100644
--- a/doc/src/sgml/config.sgml
+++ b/doc/src/sgml/config.sgml
@@ -2681,6 +2681,12 @@ include_dir 'conf.d'
<literal>worker</literal> (execute asynchronous I/O using worker processes)
</para>
</listitem>
+ <listitem>
+ <para>
+ <literal>io_uring</literal> (execute asynchronous I/O using
+ io_uring, if available)
+ </para>
+ </listitem>
<listitem>
<para>
<literal>sync</literal> (execute asynchronous-eligible I/O synchronously)
diff --git a/.cirrus.tasks.yml b/.cirrus.tasks.yml
index 10d277f5659..86a1fa9bbdb 100644
--- a/.cirrus.tasks.yml
+++ b/.cirrus.tasks.yml
@@ -493,11 +493,14 @@ task:
# - Uses undefined behaviour and alignment sanitizers, sanitizer failures
# are typically printed in the server log
# - Test both 64bit and 32 bit builds
+ # - uses io_method=io_uring
- name: Linux - Debian Bookworm - Meson
env:
CCACHE_MAXSIZE: "400M" # tests two different builds
SANITIZER_FLAGS: -fsanitize=alignment,undefined
+ PG_TEST_INITDB_EXTRA_OPTS: >-
+ -c io_method=io_uring
configure_script: |
su postgres <<-EOF
diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list
index 08490717c98..7b9affbeb80 100644
--- a/src/tools/pgindent/typedefs.list
+++ b/src/tools/pgindent/typedefs.list
@@ -2150,6 +2150,7 @@ PgAioReturn
PgAioTargetData
PgAioTargetID
PgAioTargetInfo
+PgAioUringContext
PgAioWaitRef
PgArchData
PgBackendGSSStatus
--
2.48.1.76.g4e746b1a31.dirty
[text/x-diff] v2.7-0007-aio-Implement-support-for-reads-in-smgr-md-fd.patch (21.8K, ../../4b3f32ug3cayekysqlgspz2qjmeb7lca3gvazayglxr2m3d4dv@il33accgsji7/8-v2.7-0007-aio-Implement-support-for-reads-in-smgr-md-fd.patch)
download | inline diff:
From efc4ee7b3773beca3b9b169a8f4d01ad391a9bdb Mon Sep 17 00:00:00 2001
From: Andres Freund <andres@anarazel.de>
Date: Wed, 22 Jan 2025 16:06:51 -0500
Subject: [PATCH v2.7 07/35] aio: Implement support for reads in smgr/md/fd
This implements the following:
1) An smgr AIO target, for AIO on smgr files. This should be usable not just
for md.c but also other SMGR implementation if we ever get them.
2) readv support in fd.c, which requires a small bit of infrastructure work in
fd.c
3) smgr.c and md.c support for readv
There still is nothing performing AIO, but as of this commit it would be
possible.
As part of this change FileGetRawDesc() actually ensures that the file is
opened - previously it was basically not usable. It's used to reopen a file in
IO workers.
Discussion: https://postgr.es/m/uvrtrknj4kdytuboidbhwclo4gxhswwcpgadptsjvjqcluzmah%40brqs62irg4dt
Discussion: https://postgr.es/m/20210223100344.llw5an2aklengrmn@alap3.anarazel.de
Discussion: https://postgr.es/m/stj36ea6yyhoxtqkhpieia2z4krnam7qyetc57rfezgk4zgapf@gcnactj4z56m
---
src/include/storage/aio.h | 6 +-
src/include/storage/aio_types.h | 10 +-
src/include/storage/fd.h | 3 +
src/include/storage/md.h | 7 +
src/include/storage/smgr.h | 14 ++
src/backend/storage/aio/aio_callback.c | 5 +-
src/backend/storage/aio/aio_target.c | 2 +
src/backend/storage/file/fd.c | 40 ++++++
src/backend/storage/smgr/md.c | 176 +++++++++++++++++++++++++
src/backend/storage/smgr/smgr.c | 124 +++++++++++++++++
10 files changed, 382 insertions(+), 5 deletions(-)
diff --git a/src/include/storage/aio.h b/src/include/storage/aio.h
index 8bc43c33505..d02c3d27fae 100644
--- a/src/include/storage/aio.h
+++ b/src/include/storage/aio.h
@@ -112,9 +112,10 @@ typedef enum PgAioTargetID
{
/* intentionally the zero value, to help catch zeroed memory etc */
PGAIO_TID_INVALID = 0,
+ PGAIO_TID_SMGR,
} PgAioTargetID;
-#define PGAIO_TID_COUNT (PGAIO_TID_INVALID + 1)
+#define PGAIO_TID_COUNT (PGAIO_TID_SMGR + 1)
/*
@@ -180,6 +181,9 @@ struct PgAioTargetInfo
typedef enum PgAioHandleCallbackID
{
PGAIO_HCB_INVALID,
+
+ PGAIO_HCB_MD_READV,
+ PGAIO_HCB_MD_WRITEV,
} PgAioHandleCallbackID;
diff --git a/src/include/storage/aio_types.h b/src/include/storage/aio_types.h
index a5cc658efbd..b0b9a2a5c97 100644
--- a/src/include/storage/aio_types.h
+++ b/src/include/storage/aio_types.h
@@ -60,11 +60,15 @@ typedef struct PgAioWaitRef
*/
typedef union PgAioTargetData
{
- /* just as an example placeholder for later */
struct
{
- uint32 queue_id;
- } wal;
+ RelFileLocator rlocator; /* physical relation identifier */
+ BlockNumber blockNum; /* blknum relative to begin of reln */
+ BlockNumber nblocks;
+ ForkNumber forkNum:8; /* don't waste 4 byte for four values */
+ bool is_temp:1; /* proc can be inferred by owning AIO */
+ bool skip_fsync:1;
+ } smgr;
} PgAioTargetData;
diff --git a/src/include/storage/fd.h b/src/include/storage/fd.h
index e3067ab6597..b77d8e5e30e 100644
--- a/src/include/storage/fd.h
+++ b/src/include/storage/fd.h
@@ -101,6 +101,8 @@ extern PGDLLIMPORT int max_safe_fds;
* prototypes for functions in fd.c
*/
+struct PgAioHandle;
+
/* Operations on virtual Files --- equivalent to Unix kernel file ops */
extern File PathNameOpenFile(const char *fileName, int fileFlags);
extern File PathNameOpenFilePerm(const char *fileName, int fileFlags, mode_t fileMode);
@@ -109,6 +111,7 @@ extern void FileClose(File file);
extern int FilePrefetch(File file, off_t offset, off_t amount, uint32 wait_event_info);
extern ssize_t FileReadV(File file, const struct iovec *iov, int iovcnt, off_t offset, uint32 wait_event_info);
extern ssize_t FileWriteV(File file, const struct iovec *iov, int iovcnt, off_t offset, uint32 wait_event_info);
+extern int FileStartReadV(struct PgAioHandle *ioh, File file, int iovcnt, off_t offset, uint32 wait_event_info);
extern int FileSync(File file, uint32 wait_event_info);
extern int FileZero(File file, off_t offset, off_t amount, uint32 wait_event_info);
extern int FileFallocate(File file, off_t offset, off_t amount, uint32 wait_event_info);
diff --git a/src/include/storage/md.h b/src/include/storage/md.h
index 05bf537066e..9d7131eff43 100644
--- a/src/include/storage/md.h
+++ b/src/include/storage/md.h
@@ -14,11 +14,14 @@
#ifndef MD_H
#define MD_H
+#include "storage/aio_types.h"
#include "storage/block.h"
#include "storage/relfilelocator.h"
#include "storage/smgr.h"
#include "storage/sync.h"
+extern const PgAioHandleCallbacks aio_md_readv_cb;
+
/* md storage manager functionality */
extern void mdinit(void);
extern void mdopen(SMgrRelation reln);
@@ -36,6 +39,9 @@ extern uint32 mdmaxcombine(SMgrRelation reln, ForkNumber forknum,
BlockNumber blocknum);
extern void mdreadv(SMgrRelation reln, ForkNumber forknum, BlockNumber blocknum,
void **buffers, BlockNumber nblocks);
+extern void mdstartreadv(PgAioHandle *ioh,
+ SMgrRelation reln, ForkNumber forknum, BlockNumber blocknum,
+ void **buffers, BlockNumber nblocks);
extern void mdwritev(SMgrRelation reln, ForkNumber forknum,
BlockNumber blocknum,
const void **buffers, BlockNumber nblocks, bool skipFsync);
@@ -46,6 +52,7 @@ extern void mdtruncate(SMgrRelation reln, ForkNumber forknum,
BlockNumber old_blocks, BlockNumber nblocks);
extern void mdimmedsync(SMgrRelation reln, ForkNumber forknum);
extern void mdregistersync(SMgrRelation reln, ForkNumber forknum);
+extern int mdfd(SMgrRelation reln, ForkNumber forknum, BlockNumber blocknum, uint32 *off);
extern void ForgetDatabaseSyncRequests(Oid dbid);
extern void DropRelationFiles(RelFileLocator *delrels, int ndelrels, bool isRedo);
diff --git a/src/include/storage/smgr.h b/src/include/storage/smgr.h
index 4016b206ad6..856ebcda350 100644
--- a/src/include/storage/smgr.h
+++ b/src/include/storage/smgr.h
@@ -15,6 +15,7 @@
#define SMGR_H
#include "lib/ilist.h"
+#include "storage/aio_types.h"
#include "storage/block.h"
#include "storage/relfilelocator.h"
@@ -73,6 +74,8 @@ typedef SMgrRelationData *SMgrRelation;
#define SmgrIsTemp(smgr) \
RelFileLocatorBackendIsTemp((smgr)->smgr_rlocator)
+extern const PgAioTargetInfo aio_smgr_target_info;
+
extern void smgrinit(void);
extern SMgrRelation smgropen(RelFileLocator rlocator, ProcNumber backend);
extern bool smgrexists(SMgrRelation reln, ForkNumber forknum);
@@ -97,6 +100,10 @@ extern uint32 smgrmaxcombine(SMgrRelation reln, ForkNumber forknum,
extern void smgrreadv(SMgrRelation reln, ForkNumber forknum,
BlockNumber blocknum,
void **buffers, BlockNumber nblocks);
+extern void smgrstartreadv(PgAioHandle *ioh,
+ SMgrRelation reln, ForkNumber forknum,
+ BlockNumber blocknum,
+ void **buffers, BlockNumber nblocks);
extern void smgrwritev(SMgrRelation reln, ForkNumber forknum,
BlockNumber blocknum,
const void **buffers, BlockNumber nblocks,
@@ -127,4 +134,11 @@ smgrwrite(SMgrRelation reln, ForkNumber forknum, BlockNumber blocknum,
smgrwritev(reln, forknum, blocknum, &buffer, 1, skipFsync);
}
+extern void pgaio_io_set_target_smgr(PgAioHandle *ioh,
+ SMgrRelationData *smgr,
+ ForkNumber forknum,
+ BlockNumber blocknum,
+ int nblocks,
+ bool skip_fsync);
+
#endif /* SMGR_H */
diff --git a/src/backend/storage/aio/aio_callback.c b/src/backend/storage/aio/aio_callback.c
index 7392b55322c..6a21c82396a 100644
--- a/src/backend/storage/aio/aio_callback.c
+++ b/src/backend/storage/aio/aio_callback.c
@@ -18,10 +18,11 @@
#include "miscadmin.h"
#include "storage/aio.h"
#include "storage/aio_internal.h"
+#include "storage/md.h"
/* just to have something to put into the aio_handle_cbs */
-static const struct PgAioHandleCallbacks aio_invalid_cb = {0};
+static const PgAioHandleCallbacks aio_invalid_cb = {0};
typedef struct PgAioHandleCallbacksEntry
{
@@ -37,6 +38,8 @@ typedef struct PgAioHandleCallbacksEntry
static const PgAioHandleCallbacksEntry aio_handle_cbs[] = {
#define CALLBACK_ENTRY(id, callback) [id] = {.cb = &callback, .name = #callback}
CALLBACK_ENTRY(PGAIO_HCB_INVALID, aio_invalid_cb),
+
+ CALLBACK_ENTRY(PGAIO_HCB_MD_READV, aio_md_readv_cb),
#undef CALLBACK_ENTRY
};
diff --git a/src/backend/storage/aio/aio_target.c b/src/backend/storage/aio/aio_target.c
index 3fa813fd592..aab86a358da 100644
--- a/src/backend/storage/aio/aio_target.c
+++ b/src/backend/storage/aio/aio_target.c
@@ -16,6 +16,7 @@
#include "storage/aio.h"
#include "storage/aio_internal.h"
+#include "storage/smgr.h"
/*
@@ -29,6 +30,7 @@ static const PgAioTargetInfo *pgaio_target_info[] = {
[PGAIO_TID_INVALID] = &(PgAioTargetInfo) {
.name = "invalid",
},
+ [PGAIO_TID_SMGR] = &aio_smgr_target_info,
};
diff --git a/src/backend/storage/file/fd.c b/src/backend/storage/file/fd.c
index e454db4c020..be8c4c2d60d 100644
--- a/src/backend/storage/file/fd.c
+++ b/src/backend/storage/file/fd.c
@@ -94,6 +94,7 @@
#include "miscadmin.h"
#include "pgstat.h"
#include "postmaster/startup.h"
+#include "storage/aio.h"
#include "storage/fd.h"
#include "storage/ipc.h"
#include "utils/guc.h"
@@ -1294,6 +1295,8 @@ LruDelete(File file)
vfdP = &VfdCache[file];
+ pgaio_closing_fd(vfdP->fd);
+
/*
* Close the file. We aren't expecting this to fail; if it does, better
* to leak the FD than to mess up our internal state.
@@ -1987,6 +1990,8 @@ FileClose(File file)
if (!FileIsNotOpen(file))
{
+ pgaio_closing_fd(vfdP->fd);
+
/* close the file */
if (close(vfdP->fd) != 0)
{
@@ -2210,6 +2215,32 @@ retry:
return returnCode;
}
+int
+FileStartReadV(PgAioHandle *ioh, File file,
+ int iovcnt, off_t offset,
+ uint32 wait_event_info)
+{
+ int returnCode;
+ Vfd *vfdP;
+
+ Assert(FileIsValid(file));
+
+ DO_DB(elog(LOG, "FileStartReadV: %d (%s) " INT64_FORMAT " %d",
+ file, VfdCache[file].fileName,
+ (int64) offset,
+ iovcnt));
+
+ returnCode = FileAccess(file);
+ if (returnCode < 0)
+ return returnCode;
+
+ vfdP = &VfdCache[file];
+
+ pgaio_io_prep_readv(ioh, vfdP->fd, iovcnt, offset);
+
+ return 0;
+}
+
ssize_t
FileWriteV(File file, const struct iovec *iov, int iovcnt, off_t offset,
uint32 wait_event_info)
@@ -2498,6 +2529,12 @@ FilePathName(File file)
int
FileGetRawDesc(File file)
{
+ int returnCode;
+
+ returnCode = FileAccess(file);
+ if (returnCode < 0)
+ return returnCode;
+
Assert(FileIsValid(file));
return VfdCache[file].fd;
}
@@ -2778,6 +2815,7 @@ FreeDesc(AllocateDesc *desc)
result = closedir(desc->desc.dir);
break;
case AllocateDescRawFD:
+ pgaio_closing_fd(desc->desc.fd);
result = close(desc->desc.fd);
break;
default:
@@ -2846,6 +2884,8 @@ CloseTransientFile(int fd)
/* Only get here if someone passes us a file not in allocatedDescs */
elog(WARNING, "fd passed to CloseTransientFile was not obtained from OpenTransientFile");
+ pgaio_closing_fd(fd);
+
return close(fd);
}
diff --git a/src/backend/storage/smgr/md.c b/src/backend/storage/smgr/md.c
index f3220f98dc4..11d4d5a7aea 100644
--- a/src/backend/storage/smgr/md.c
+++ b/src/backend/storage/smgr/md.c
@@ -31,6 +31,7 @@
#include "miscadmin.h"
#include "pg_trace.h"
#include "pgstat.h"
+#include "storage/aio.h"
#include "storage/bufmgr.h"
#include "storage/fd.h"
#include "storage/md.h"
@@ -152,6 +153,15 @@ static MdfdVec *_mdfd_getseg(SMgrRelation reln, ForkNumber forknum,
static BlockNumber _mdnblocks(SMgrRelation reln, ForkNumber forknum,
MdfdVec *seg);
+static PgAioResult md_readv_complete(PgAioHandle *ioh, PgAioResult prior_result, uint8 cb_data);
+static void md_readv_report(PgAioResult result, const PgAioTargetData *target_data, int elevel);
+
+const PgAioHandleCallbacks aio_md_readv_cb = {
+ .complete_shared = md_readv_complete,
+ .report = md_readv_report,
+};
+
+
static inline int
_mdfd_open_flags(void)
{
@@ -937,6 +947,61 @@ mdreadv(SMgrRelation reln, ForkNumber forknum, BlockNumber blocknum,
}
}
+/*
+ * mdstartreadv() -- Asynchronous version of mdreadv().
+ */
+void
+mdstartreadv(PgAioHandle *ioh,
+ SMgrRelation reln, ForkNumber forknum, BlockNumber blocknum,
+ void **buffers, BlockNumber nblocks)
+{
+ off_t seekpos;
+ MdfdVec *v;
+ BlockNumber nblocks_this_segment;
+ struct iovec *iov;
+ int iovcnt;
+
+ v = _mdfd_getseg(reln, forknum, blocknum, false,
+ EXTENSION_FAIL | EXTENSION_CREATE_RECOVERY);
+
+ seekpos = (off_t) BLCKSZ * (blocknum % ((BlockNumber) RELSEG_SIZE));
+
+ Assert(seekpos < (off_t) BLCKSZ * RELSEG_SIZE);
+
+ nblocks_this_segment =
+ Min(nblocks,
+ RELSEG_SIZE - (blocknum % ((BlockNumber) RELSEG_SIZE)));
+
+ if (nblocks_this_segment != nblocks)
+ elog(ERROR, "read crossing segment boundary");
+
+ iovcnt = pgaio_io_get_iovec(ioh, &iov);
+
+ Assert(nblocks <= iovcnt);
+
+ iovcnt = buffers_to_iovec(iov, buffers, nblocks_this_segment);
+
+ Assert(iovcnt <= nblocks_this_segment);
+
+ if (!(io_direct_flags & IO_DIRECT_DATA))
+ pgaio_io_set_flag(ioh, PGAIO_HF_BUFFERED);
+
+ pgaio_io_set_target_smgr(ioh,
+ reln,
+ forknum,
+ blocknum,
+ nblocks,
+ false);
+ pgaio_io_register_callbacks(ioh, PGAIO_HCB_MD_READV, 0);
+
+ FileStartReadV(ioh, v->mdfd_vfd, iovcnt, seekpos, WAIT_EVENT_DATA_FILE_READ);
+
+ /*
+ * The error checks corresponding to the post-read checks in mdread() are
+ * in md_readv_complete().
+ */
+}
+
/*
* mdwritev() -- Write the supplied blocks at the appropriate location.
*
@@ -1365,6 +1430,21 @@ mdimmedsync(SMgrRelation reln, ForkNumber forknum)
}
}
+int
+mdfd(SMgrRelation reln, ForkNumber forknum, BlockNumber blocknum, uint32 *off)
+{
+ MdfdVec *v = mdopenfork(reln, forknum, EXTENSION_FAIL);
+
+ v = _mdfd_getseg(reln, forknum, blocknum, false,
+ EXTENSION_FAIL);
+
+ *off = (off_t) BLCKSZ * (blocknum % ((BlockNumber) RELSEG_SIZE));
+
+ Assert(*off < (off_t) BLCKSZ * RELSEG_SIZE);
+
+ return FileGetRawDesc(v->mdfd_vfd);
+}
+
/*
* register_dirty_segment() -- Mark a relation segment as needing fsync
*
@@ -1841,3 +1921,99 @@ mdfiletagmatches(const FileTag *ftag, const FileTag *candidate)
*/
return ftag->rlocator.dbOid == candidate->rlocator.dbOid;
}
+
+/*
+ * AIO completion callback for mdstartreadv().
+ */
+static PgAioResult
+md_readv_complete(PgAioHandle *ioh, PgAioResult prior_result, uint8 cb_data)
+{
+ PgAioTargetData *td = pgaio_io_get_target_data(ioh);
+ PgAioResult result = prior_result;
+
+ if (prior_result.result < 0)
+ {
+ result.status = ARS_ERROR;
+ result.id = PGAIO_HCB_MD_READV;
+ /* For "hard" errors, track the error number in error_data */
+ result.error_data = -prior_result.result;
+ result.result = 0;
+
+ pgaio_result_report(result, td, LOG);
+
+ return result;
+ }
+
+ /*
+ * The smgr API operates in blocks, therefore convert the result from
+ * bytes to blocks.
+ */
+ result.result /= BLCKSZ;
+
+ if (result.result == 0)
+ {
+ /* consider 0 blocks read a failure */
+ result.status = ARS_ERROR;
+ result.id = PGAIO_HCB_MD_READV;
+ result.error_data = 0;
+
+ pgaio_result_report(result, td, LOG);
+
+ return result;
+ }
+
+ if (result.status != ARS_ERROR &&
+ result.result < td->smgr.nblocks)
+ {
+ /* partial reads should be retried at upper level */
+ result.status = ARS_PARTIAL;
+ result.id = PGAIO_HCB_MD_READV;
+ }
+
+ return result;
+}
+
+/*
+ * AIO error reporting callback for mdstartreadv().
+ *
+ * Errors are encoded as follows:
+ * - PgAioResult.error_data != 0 encodes IO that failed with errno != 0
+ * - PgAioResult.error_data == 0 encodes IO that didn't read all data
+ */
+static void
+md_readv_report(PgAioResult result, const PgAioTargetData *td, int elevel)
+{
+ RelPathStr path;
+
+ path = relpathbackend(td->smgr.rlocator,
+ td->smgr.is_temp ? MyProcNumber : INVALID_PROC_NUMBER,
+ td->smgr.forkNum);
+
+ if (result.error_data != 0)
+ {
+ /* for errcode_for_file_access() and %m */
+ errno = result.error_data;
+
+ ereport(elevel,
+ errcode_for_file_access(),
+ errmsg("could not read blocks %u..%u in file \"%s\": %m",
+ td->smgr.blockNum,
+ td->smgr.blockNum + td->smgr.nblocks - 1,
+ path.str));
+ }
+ else
+ {
+ /*
+ * NB: This will typically only be output in debug messages, while
+ * retrying a partial IO.
+ */
+ ereport(elevel,
+ errcode(ERRCODE_DATA_CORRUPTED),
+ errmsg("could not read blocks %u..%u in file \"%s\": read only %zu of %zu bytes",
+ td->smgr.blockNum,
+ td->smgr.blockNum + td->smgr.nblocks - 1,
+ path.str,
+ result.result * (size_t) BLCKSZ,
+ td->smgr.nblocks * (size_t) BLCKSZ));
+ }
+}
diff --git a/src/backend/storage/smgr/smgr.c b/src/backend/storage/smgr/smgr.c
index ebe35c04de5..607c14ee173 100644
--- a/src/backend/storage/smgr/smgr.c
+++ b/src/backend/storage/smgr/smgr.c
@@ -53,6 +53,7 @@
#include "access/xlogutils.h"
#include "lib/ilist.h"
+#include "storage/aio.h"
#include "storage/bufmgr.h"
#include "storage/ipc.h"
#include "storage/md.h"
@@ -93,6 +94,10 @@ typedef struct f_smgr
void (*smgr_readv) (SMgrRelation reln, ForkNumber forknum,
BlockNumber blocknum,
void **buffers, BlockNumber nblocks);
+ void (*smgr_startreadv) (PgAioHandle *ioh,
+ SMgrRelation reln, ForkNumber forknum,
+ BlockNumber blocknum,
+ void **buffers, BlockNumber nblocks);
void (*smgr_writev) (SMgrRelation reln, ForkNumber forknum,
BlockNumber blocknum,
const void **buffers, BlockNumber nblocks,
@@ -104,6 +109,7 @@ typedef struct f_smgr
BlockNumber old_blocks, BlockNumber nblocks);
void (*smgr_immedsync) (SMgrRelation reln, ForkNumber forknum);
void (*smgr_registersync) (SMgrRelation reln, ForkNumber forknum);
+ int (*smgr_fd) (SMgrRelation reln, ForkNumber forknum, BlockNumber blocknum, uint32 *off);
} f_smgr;
static const f_smgr smgrsw[] = {
@@ -121,12 +127,14 @@ static const f_smgr smgrsw[] = {
.smgr_prefetch = mdprefetch,
.smgr_maxcombine = mdmaxcombine,
.smgr_readv = mdreadv,
+ .smgr_startreadv = mdstartreadv,
.smgr_writev = mdwritev,
.smgr_writeback = mdwriteback,
.smgr_nblocks = mdnblocks,
.smgr_truncate = mdtruncate,
.smgr_immedsync = mdimmedsync,
.smgr_registersync = mdregistersync,
+ .smgr_fd = mdfd,
}
};
@@ -144,6 +152,16 @@ static dlist_head unpinned_relns;
static void smgrshutdown(int code, Datum arg);
static void smgrdestroy(SMgrRelation reln);
+static void smgr_aio_reopen(PgAioHandle *ioh);
+static char *smgr_aio_describe_identity(const PgAioTargetData *sd);
+
+
+const PgAioTargetInfo aio_smgr_target_info = {
+ .name = "smgr",
+ .reopen = smgr_aio_reopen,
+ .describe_identity = smgr_aio_describe_identity,
+};
+
/*
* smgrinit(), smgrshutdown() -- Initialize or shut down storage
@@ -623,6 +641,22 @@ smgrreadv(SMgrRelation reln, ForkNumber forknum, BlockNumber blocknum,
nblocks);
}
+/*
+ * smgrstartreadv() -- asynchronous version of smgrreadv()
+ *
+ * This starts an asynchronous readv IO with the IO handle `ioh`. Other than
+ * `ioh` all parameters are the same as smgrreadv().
+ */
+void
+smgrstartreadv(PgAioHandle *ioh,
+ SMgrRelation reln, ForkNumber forknum, BlockNumber blocknum,
+ void **buffers, BlockNumber nblocks)
+{
+ smgrsw[reln->smgr_which].smgr_startreadv(ioh,
+ reln, forknum, blocknum, buffers,
+ nblocks);
+}
+
/*
* smgrwritev() -- Write the supplied buffers out.
*
@@ -819,6 +853,19 @@ smgrimmedsync(SMgrRelation reln, ForkNumber forknum)
smgrsw[reln->smgr_which].smgr_immedsync(reln, forknum);
}
+/*
+ * Return fd for the specified block number and update *off to the appropriate
+ * position.
+ *
+ * This is only to be used for when AIO needs to perform the IO in a different
+ * process than where it was issued (e.g. in an IO worker).
+ */
+static int
+smgrfd(SMgrRelation reln, ForkNumber forknum, BlockNumber blocknum, uint32 *off)
+{
+ return smgrsw[reln->smgr_which].smgr_fd(reln, forknum, blocknum, off);
+}
+
/*
* AtEOXact_SMgr
*
@@ -847,3 +894,80 @@ ProcessBarrierSmgrRelease(void)
smgrreleaseall();
return true;
}
+
+/*
+ * Set target of the IO handle to be smgr and initialize all the relevant
+ * pieces of data.
+ */
+void
+pgaio_io_set_target_smgr(PgAioHandle *ioh,
+ SMgrRelationData *smgr,
+ ForkNumber forknum,
+ BlockNumber blocknum,
+ int nblocks,
+ bool skip_fsync)
+{
+ PgAioTargetData *sd = pgaio_io_get_target_data(ioh);
+
+ pgaio_io_set_target(ioh, PGAIO_TID_SMGR);
+
+ /* backend is implied via IO owner */
+ sd->smgr.rlocator = smgr->smgr_rlocator.locator;
+ sd->smgr.forkNum = forknum;
+ sd->smgr.blockNum = blocknum;
+ sd->smgr.nblocks = nblocks;
+ sd->smgr.is_temp = SmgrIsTemp(smgr);
+ /* Temp relations should never be fsync'd */
+ sd->smgr.skip_fsync = skip_fsync && !SmgrIsTemp(smgr);
+}
+
+/*
+ * Callback for smgr AIO targets, to reopen the file (e.g. because the IO is
+ * executed in a worker).
+ */
+static void
+smgr_aio_reopen(PgAioHandle *ioh)
+{
+ PgAioTargetData *sd = pgaio_io_get_target_data(ioh);
+ PgAioOpData *od = pgaio_io_get_op_data(ioh);
+ SMgrRelation reln;
+ ProcNumber procno;
+ uint32 off;
+
+ if (sd->smgr.is_temp)
+ procno = pgaio_io_get_owner(ioh);
+ else
+ procno = INVALID_PROC_NUMBER;
+
+ reln = smgropen(sd->smgr.rlocator, procno);
+ od->read.fd = smgrfd(reln, sd->smgr.forkNum, sd->smgr.blockNum, &off);
+ Assert(off == od->read.offset);
+}
+
+/*
+ * Callback for smgr AIO targets, to describe the target of the IO.
+ */
+static char *
+smgr_aio_describe_identity(const PgAioTargetData *sd)
+{
+ RelPathStr path;
+ char *desc;
+
+ path = relpathbackend(sd->smgr.rlocator,
+ sd->smgr.is_temp ? MyProcNumber : INVALID_PROC_NUMBER,
+ sd->smgr.forkNum);
+
+ if (sd->smgr.nblocks == 0)
+ desc = psprintf(_("file \"%s\""), path.str);
+ else if (sd->smgr.nblocks == 1)
+ desc = psprintf(_("block %u in file \"%s\""),
+ sd->smgr.blockNum,
+ path.str);
+ else
+ desc = psprintf(_("blocks %u..%u in file \"%s\""),
+ sd->smgr.blockNum,
+ sd->smgr.blockNum + sd->smgr.nblocks - 1,
+ path.str);
+
+ return desc;
+}
--
2.48.1.76.g4e746b1a31.dirty
[text/x-diff] v2.7-0008-aio-Add-README.md-explaining-higher-level-desig.patch (19.0K, ../../4b3f32ug3cayekysqlgspz2qjmeb7lca3gvazayglxr2m3d4dv@il33accgsji7/9-v2.7-0008-aio-Add-README.md-explaining-higher-level-desig.patch)
download | inline diff:
From 25f6f424df486472ccccfedd7448b67ea4f84cb0 Mon Sep 17 00:00:00 2001
From: Andres Freund <andres@anarazel.de>
Date: Wed, 22 Jan 2025 13:44:37 -0500
Subject: [PATCH v2.7 08/35] aio: Add README.md explaining higher level design
Reviewed-by: Noah Misch <noah@leadboat.com>
Reviewed-by: Heikki Linnakangas <hlinnaka@iki.fi>
Discussion: https://postgr.es/m/uvrtrknj4kdytuboidbhwclo4gxhswwcpgadptsjvjqcluzmah%40brqs62irg4dt
Discussion: https://postgr.es/m/20210223100344.llw5an2aklengrmn@alap3.anarazel.de
Discussion: https://postgr.es/m/stj36ea6yyhoxtqkhpieia2z4krnam7qyetc57rfezgk4zgapf@gcnactj4z56m
---
src/backend/storage/aio/README.md | 423 ++++++++++++++++++++++++++++++
src/backend/storage/aio/aio.c | 2 +
2 files changed, 425 insertions(+)
create mode 100644 src/backend/storage/aio/README.md
diff --git a/src/backend/storage/aio/README.md b/src/backend/storage/aio/README.md
new file mode 100644
index 00000000000..b7dfb80b4b2
--- /dev/null
+++ b/src/backend/storage/aio/README.md
@@ -0,0 +1,423 @@
+# Asynchronous & Direct IO
+
+## Motivation
+
+### Why Asynchronous IO
+
+Until the introduction of asynchronous IO Postgres relied on the operating
+system to hide the cost of synchronous IO from Postgres. While this worked
+surprisingly well in a lot of workloads, it does not do as good a job on
+prefetching and controlled writeback as we would like.
+
+There are important expensive operations like `fdatasync()` where the operating
+system cannot hide the storage latency. This is particularly important for WAL
+writes, where the ability to asynchronously issue `fdatasync()` or O_DSYNC
+writes can yield significantly higher throughput.
+
+
+### Why Direct / unbuffered IO
+
+The main reasons to want to use Direct IO are:
+
+- Lower CPU usage / higher throughput. Particularly on modern storage buffered
+ writes are bottlenecked by the operating system having to copy data from the
+ kernel's page cache to postgres buffer pool using the CPU. Whereas direct IO
+ can often move the data directly between the storage devices and postgres'
+ buffer cache, using DMA. While that transfer is ongoing, the CPU is free to
+ perform other work.
+- Reduced latency - Direct IO can have substantially lower latency than
+ buffered IO, which can be impactful for OLTP workloads bottlenecked by WAL
+ write latency.
+- Avoiding double buffering between operating system cache and postgres'
+ shared_buffers.
+- Better control over the timing and pace of dirty data writeback.
+
+
+The main reasons *not* to use Direct IO are:
+
+- Without AIO, Direct IO is unusably slow for most purposes.
+- Even with AIO, many parts of postgres need to be modified to perform
+ explicit prefetching.
+- In situations where shared_buffers cannot be set appropriately large,
+ e.g. because there are many different postgres instances hosted on shared
+ hardware, performance will often be worse than when using buffered IO.
+
+
+## AIO Usage Example
+
+In many cases code that can benefit from AIO does not directly have to
+interact with the AIO interface, but can use AIO via higher-level
+abstractions. See [Helpers](#helpers).
+
+In this example, a buffer will be read into shared buffers.
+
+```C
+/*
+ * Result of the operation, only to be accessed in this backend.
+ */
+PgAioReturn ioret;
+
+/*
+ * Acquire an AIO Handle, ioret will get result upon completion.
+ *
+ * Note that ioret needs to stay alive until the IO completes or
+ * CurrentResourceOwner is released (i.e. an error is thrown).
+ */
+PgAioHandle *ioh = pgaio_io_acquire(CurrentResourceOwner, &ioret);
+
+/*
+ * Reference that can be used to wait for the IO we initiate below. This
+ * reference can reside in local or shared memory and waited upon by any
+ * process. An arbitrary number of references can be made for each IO.
+ */
+PgAioWaitRef iow;
+
+pgaio_io_get_wref(ioh, &iow);
+
+/*
+ * Arrange for shared buffer completion callbacks to be called upon completion
+ * of the IO. This callback will update the buffer descriptors associated with
+ * the AioHandle, which e.g. allows other backends to access the buffer.
+ *
+ * A callback can be passed a small bit of data, e.g. to indicate whether to
+ * zero a buffer if it is invalid.
+ *
+ * Multiple completion callbacks can be registered for each handle.
+ */
+pgaio_io_register_callbacks(ioh, PGAIO_HCB_SHARED_BUFFER_READV, 0);
+
+/*
+ * The completion callback needs to know which buffers to update when the IO
+ * completes. As the AIO subsystem does not know about buffers, we have to
+ * associate this information with the AioHandle, for use by the completion
+ * callback registered above.
+ *
+ * In this example we're reading only a single buffer, hence the 1.
+ */
+pgaio_io_set_handle_data_32(ioh, (uint32 *) buffer, 1);
+
+/*
+ * Pass the AIO handle to lower-level function. When operating on the level of
+ * buffers, we don't know how exactly the IO is performed, that is the
+ * responsibility of the storage manager implementation.
+ *
+ * E.g. md.c needs to translate block numbers into offsets in segments.
+ *
+ * Once the IO handle has been handed off to smgstartreadv(), it may not
+ * further be used, as the IO may immediately get executed below
+ * smgrstartreadv() and the handle reused for another IO.
+ *
+ * To issue multiple IOs in an efficient way, a caller can call
+ * pgaio_enter_batchmode() before starting multiple IOs, and end that batch
+ * with pgaio_exit_batchmode(). Note that one needs to be careful while there
+ * may be unsubmitted IOs, as another backend may need to wait for one of the
+ * unsubmitted IOs. If this backend then had to to wait for the other backend,
+ * it'd end in an undetected deadlock. See pgaio_enter_batchmode() for more
+ * details.
+ *
+ * Note that even while in batchmode an IO might get submitted immediately,
+ * e.g. due to reaching a limit on the number of unsubmitted IOs, and even
+ * complete before smgrstartreadv() returns.
+ */
+smgrstartreadv(ioh, operation->smgr, forknum, blkno,
+ BufferGetBlock(buffer), 1);
+
+/*
+ * To benefit from AIO, it is beneficial to perform other work, including
+ * submitting other IOs, before waiting for the IO to complete. Otherwise
+ * we could just have used synchronous, blocking IO.
+ */
+perform_other_work();
+
+/*
+ * We did some other work and now need the IO operation to have completed to
+ * continue.
+ */
+pgaio_wref_wait(&iow);
+
+/*
+ * At this point the IO has completed. We do not yet know whether it succeeded
+ * or failed, however. The buffer's state has been updated, which allows other
+ * backends to use the buffer (if the IO succeeded), or retry the IO (if it
+ * failed).
+ *
+ * Note that in case the IO has failed, a LOG message may have been emitted,
+ * but no ERROR has been raised. This is crucial, as another backend waiting
+ * for this IO should not see an ERROR.
+ *
+ * To check whether the operation succeeded, and to raise an ERROR, or if more
+ * appropriate LOG, the PgAioReturn we passed to pgaio_io_acquire() is used.
+ */
+if (ioret.result.status == ARS_ERROR)
+ pgaio_result_report(aio_ret.result, &aio_ret.target_data, ERROR);
+
+/*
+ * Besides having succeeded completely, the IO could also have partially
+ * completed. If we e.g. tried to read many blocks at once, the read might have
+ * only succeeded for the first few blocks.
+ *
+ * If the IO partially succeeded and this backend needs all blocks to have
+ * completed, this backend needs to reissue the IO for the remaining buffers.
+ * The AIO subsystem cannot handle this retry transparently.
+ *
+ * As this example is already long, and we only read a single block, we'll just
+ * error out if there's a partial read.
+ */
+if (ioret.result.status == ARS_PARTIAL)
+ pgaio_result_report(aio_ret.result, &aio_ret.target_data, ERROR);
+
+/*
+ * The IO succeeded, so we can use the buffer now.
+ */
+```
+
+
+## Design Criteria & Motivation
+
+### Deadlock and Starvation Dangers due to AIO
+
+Using AIO in a naive way can easily lead to deadlocks in an environment where
+the source/target of AIO are shared resources, like pages in postgres'
+shared_buffers.
+
+Consider one backend performing readahead on a table, initiating IO for a
+number of buffers ahead of the current "scan position". If that backend then
+performs some operation that blocks, or even just is slow, the IO completion
+for the asynchronously initiated read may not be processed.
+
+This AIO implementation solves this problem by requiring that AIO methods
+either allow AIO completions to be processed by any backend in the system
+(e.g. io_uring), or to guarantee that AIO processing will happen even when the
+issuing backend is blocked (e.g. worker mode, which offloads completion
+processing to the AIO workers).
+
+
+### IO can be started in critical sections
+
+Using AIO for WAL writes can reduce the overhead of WAL logging substantially:
+
+- AIO allows to start WAL writes eagerly, so they complete before needing to
+ wait
+- AIO allows to have multiple WAL flushes in progress at the same time
+- AIO makes it more realistic to use O\_DIRECT + O\_DSYNC, which can reduce
+ the number of roundtrips to storage on some OSs and storage HW (buffered IO
+ and direct IO without O_DSYNC needs to issue a write and after the write's
+ completion a cache cache flush, whereas O\_DIRECT + O\_DSYNC can use a
+ single Force Unit Access (FUA) write).
+
+The need to be able to execute IO in critical sections has substantial design
+implication on the AIO subsystem. Mainly because completing IOs (see prior
+section) needs to be possible within a critical section, even if the
+to-be-completed IO itself was not issued in a critical section. Consider
+e.g. the case of a backend first starting a number of writes from shared
+buffers and then starting to flush the WAL. Because only a limited amount of
+IO can be in-progress at the same time, initiating IO for flushing the WAL may
+require to first complete IO that was started earlier.
+
+
+### State for AIO needs to live in shared memory
+
+Because postgres uses a process model and because AIOs need to be
+complete-able by any backend much of the state of the AIO subsystem needs to
+live in shared memory.
+
+In an `EXEC_BACKEND` build, a backend's executable code and other process
+local state is not necessarily mapped to the same addresses in each process
+due to ASLR. This means that the shared memory cannot contain pointers to
+callbacks.
+
+
+## Design of the AIO Subsystem
+
+
+### AIO Methods
+
+To achieve portability and performance, multiple methods of performing AIO are
+implemented and others are likely worth adding in the future.
+
+
+#### Synchronous Mode
+
+`io_method=sync` does not actually perform AIO but allows to use the AIO API
+while performing synchronous IO. This can be useful for debugging. The code
+for the synchronous mode is also used as a fallback by e.g. the [worker
+mode](#worker) uses it to execute IO that cannot be executed by workers.
+
+
+#### Worker
+
+`io_method=worker` is available on every platform postgres runs on, and
+implements asynchronous IO - from the view of the issuing process - by
+dispatching the IO to one of several worker processes performing the IO in a
+synchronous manner.
+
+
+#### io_uring
+
+`io_method=io_uring` is available on Linux 5.1+. In contrast to worker mode it
+dispatches all IO from within the process, lowering context switch rate /
+latency.
+
+
+### AIO Handles
+
+The central API piece for postgres' AIO abstraction are AIO handles. To
+execute an IO one first has to acquire an IO handle (`pgaio_io_acquire()`) and
+then "defined", i.e. associate an IO operation with the handle.
+
+Often AIO handles are acquired on a higher level and then passed to a lower
+level to be fully defined. E.g., for IO to/from shared buffers, bufmgr.c
+routines acquire the handle, which is then passed through smgr.c, md.c to be
+finally fully defined in fd.c.
+
+The functions used at the lowest level to define the operation are
+`pgaio_io_prep_*()`.
+
+Because acquisition of an IO handle
+[must always succeed](#io-can-be-started-in-critical-sections)
+and the number of AIO Handles
+[has to be limited](#state-for-aio-needs-to-live-in-shared-memory)
+AIO handles can be reused as soon as they have completed. Obviously code needs
+to be able to react to IO completion. State can be updated using
+[AIO Completion callbacks](#aio-callbacks)
+and the issuing backend can provide a backend local variable to receive the
+result of the IO, as described in
+[AIO Result](#aio-results).
+An IO can be waited for, by both the issuing and any other backend, using
+[AIO References](#aio-wait-references).
+
+
+Because an AIO Handle is not executable just after calling
+`pgaio_io_acquire()` and because `pgaio_io_acquire()` needs to always succeed
+(absent a PANIC), only a single AIO Handle may be acquired (i.e. returned by
+`pgaio_io_acquire()`) without causing the IO to have been defined (by,
+potentially indirectly, causing `pgaio_io_prep_*()` to have been
+called). Otherwise a backend could trivially self-deadlock by using up all AIO
+Handles without the ability to wait for some of the IOs to complete.
+
+If it turns out that an AIO Handle is not needed, e.g., because the handle was
+acquired before holding a contended lock, it can be released without being
+defined using `pgaio_io_release()`.
+
+
+### AIO Callbacks
+
+Commonly several layers need to react to completion of an IO. E.g. for a read
+md.c needs to check if the IO outright failed or was shorter than needed,
+bufmgr.c needs to verify the page looks valid and bufmgr.c needs to update the
+BufferDesc to update the buffer's state.
+
+The fact that several layers / subsystems need to react to IO completion poses
+a few challenges:
+
+- Upper layers should not need to know details of lower layers. E.g. bufmgr.c
+ should not assume the IO will pass through md.c. Therefore upper levels
+ cannot know what lower layers would consider an error.
+
+- Lower layers should not need to know about upper layers. E.g. smgr APIs are
+ used going through shared buffers but are also used bypassing shared
+ buffers. This means that e.g. md.c is not in a position to validate
+ checksums.
+
+- Having code in the AIO subsystem for every possible combination of layers
+ would lead to a lot of duplication.
+
+The "solution" to this the ability to associate multiple completion callbacks
+with a handle. E.g. bufmgr.c can have a callback to update the BufferDesc
+state and to verify the page and md.c can have another callback to check if
+the IO operation was successful.
+
+As [mentioned](#state-for-aio-needs-to-live-in-shared-memory), shared memory
+currently cannot contain function pointers. Because of that completion
+callbacks are not directly identified by function pointers but by IDs
+(`PgAioHandleCallbackID`). A substantial added benefit is that that
+allows callbacks to be identified by much smaller amount of memory (a single
+byte currently).
+
+In addition to completion, AIO callbacks also are called to "stage" an
+IO. This is, e.g., used to increase buffer reference counts to account for the
+AIO subsystem referencing the buffer, which is required to handle the case
+where the issuing backend errors out and releases its own pins while the IO is
+still ongoing.
+
+As [explained earlier](#io-can-be-started-in-critical-sections) IO completions
+need to be safe to execute in critical sections. To allow the backend that
+issued the IO to error out in case of failure [AIO Result](#aio-results) can
+be used.
+
+
+### AIO Targets
+
+In addition to the completion callbacks describe above, each AIO Handle has
+exactly one "target". Each target has some space inside an AIO Handle with
+information specific to the target and can provide callbacks to allow to
+reopen the underlying file (required for worker mode) and to describe the IO
+operation (used for debug logging and error messages).
+
+I.e., if two different uses of AIO can describe the identity of the file being
+operated on the same way, it likely makes sense to use the same
+target. E.g. different smgr implementations can describe IO with
+RelFileLocator, ForkNumber and BlockNumber and can thus share a target. In
+contrast, IO for a WAL file would be described with TimeLineID and XLogRecPtr
+and it would not make sense to use the same target for smgr and WAL.
+
+
+### AIO Wait References
+
+As [described above](#aio-handles), AIO Handles can be reused immediately
+after completion and therefore cannot be used to wait for completion of the
+IO. Waiting is enabled using AIO wait references, which do not just identify
+an AIO Handle but also include the handles "generation".
+
+A reference to an AIO Handle can be acquired using `pgaio_io_get_wref()` and
+then waited upon using `pgaio_wref_wait()`.
+
+
+### AIO Results
+
+As AIO completion callbacks
+[are executed in critical sections](#io-can-be-started-in-critical-sections)
+and [may be executed by any backend](#deadlock-and-starvation-dangers-due-to-aio)
+completion callbacks cannot be used to, e.g., make the query that triggered an
+IO ERROR out.
+
+To allow to react to failing IOs the issuing backend can pass a pointer to a
+`PgAioReturn` in backend local memory. Before an AIO Handle is reused the
+`PgAioReturn` is filled with information about the IO. This includes
+information about whether the IO was successful (as a value of
+`PgAioResultStatus`) and enough information to raise an error in case of a
+failure (via `pgaio_result_report()`, with the error details encoded in
+`PgAioResult`).
+
+
+### AIO Errors
+
+It would be very convenient to have shared completion callbacks encode the
+details of errors as an `ErrorData` that could be raised at a later
+time. Unfortunately doing so would require allocating memory. While elog.c can
+guarantee (well, kinda) that logging a message will not run out of memory,
+that only works because a very limited number of messages are in the process
+of being logged. With AIO a large number of concurrently issued AIOs might
+fail.
+
+To avoid the need for preallocating a potentially large amount of memory (in
+shared memory no less!), completion callbacks instead have to encode errors in
+a more compact format that can be converted into an error message.
+
+
+## Helpers
+
+Using the low-level AIO API introduces too much complexity to do so all over
+the tree. Most uses of AIO should be done via reusable, higher-level,
+helpers.
+
+
+### Read Stream
+
+A common and very beneficial use of AIO are reads where a substantial number
+of to-be-read locations are known ahead of time. E.g., for a sequential scan
+the set of blocks that need to be read can be determined solely by knowing the
+current position and checking the buffer mapping table.
+
+The [Read Stream](../../../include/storage/read_stream.h) interface makes it
+comparatively easy to use AIO for such use cases.
diff --git a/src/backend/storage/aio/aio.c b/src/backend/storage/aio/aio.c
index 2fbc380b758..f31e52f7fe0 100644
--- a/src/backend/storage/aio/aio.c
+++ b/src/backend/storage/aio/aio.c
@@ -24,6 +24,8 @@
*
* - read_stream.c - helper for reading buffered relation data
*
+ * - README.md - higher-level overview over AIO
+ *
*
* Portions Copyright (c) 1996-2025, PostgreSQL Global Development Group
* Portions Copyright (c) 1994, Regents of the University of California
--
2.48.1.76.g4e746b1a31.dirty
[text/x-diff] v2.7-0009-aio-Add-pg_aios-view.patch (9.7K, ../../4b3f32ug3cayekysqlgspz2qjmeb7lca3gvazayglxr2m3d4dv@il33accgsji7/10-v2.7-0009-aio-Add-pg_aios-view.patch)
download | inline diff:
From a096c8f7b25acdd1c8448d69fad7827dbb01e08d Mon Sep 17 00:00:00 2001
From: Andres Freund <andres@anarazel.de>
Date: Wed, 22 Jan 2025 13:44:40 -0500
Subject: [PATCH v2.7 09/35] aio: Add pg_aios view
Author:
Reviewed-by:
Discussion: https://postgr.es/m/
Backpatch:
---
src/include/catalog/pg_proc.dat | 10 ++
src/backend/catalog/system_views.sql | 3 +
src/backend/storage/aio/Makefile | 1 +
src/backend/storage/aio/aio_funcs.c | 219 +++++++++++++++++++++++++++
src/backend/storage/aio/meson.build | 1 +
src/test/regress/expected/rules.out | 16 ++
6 files changed, 250 insertions(+)
create mode 100644 src/backend/storage/aio/aio_funcs.c
diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat
index 42e427f8fe8..1b6ba3182da 100644
--- a/src/include/catalog/pg_proc.dat
+++ b/src/include/catalog/pg_proc.dat
@@ -12476,4 +12476,14 @@
proargtypes => 'int4',
prosrc => 'gist_stratnum_common' },
+# AIO related functions
+{ oid => '9200', descr => 'information about in-progress asynchronous IOs',
+ proname => 'pg_get_aios', prorows => '100', proretset => 't',
+ provolatile => 'v', proparallel => 'r', prorettype => 'record', proargtypes => '',
+ proallargtypes => '{int4,int4,int8,text,text,int8,int8,text,int2,int4,text,text,bool,bool,bool}',
+ proargmodes => '{o,o,o,o,o,o,o,o,o,o,o,o,o,o,o}',
+ proargnames => '{pid,io_id,io_generation,state,operation,offset,length,target,handle_data_len,raw_result,result,target_desc,f_sync,f_localmem,f_buffered}',
+ prosrc => 'pg_get_aios' },
+
+
]
diff --git a/src/backend/catalog/system_views.sql b/src/backend/catalog/system_views.sql
index a4d2cfdcaf5..73902a763d1 100644
--- a/src/backend/catalog/system_views.sql
+++ b/src/backend/catalog/system_views.sql
@@ -1390,3 +1390,6 @@ CREATE VIEW pg_stat_subscription_stats AS
CREATE VIEW pg_wait_events AS
SELECT * FROM pg_get_wait_events();
+
+CREATE VIEW pg_aios AS
+ SELECT * FROM pg_get_aios();
diff --git a/src/backend/storage/aio/Makefile b/src/backend/storage/aio/Makefile
index c06c50771e0..3f2469cc399 100644
--- a/src/backend/storage/aio/Makefile
+++ b/src/backend/storage/aio/Makefile
@@ -11,6 +11,7 @@ include $(top_builddir)/src/Makefile.global
OBJS = \
aio.o \
aio_callback.o \
+ aio_funcs.o \
aio_init.o \
aio_io.o \
aio_target.o \
diff --git a/src/backend/storage/aio/aio_funcs.c b/src/backend/storage/aio/aio_funcs.c
new file mode 100644
index 00000000000..58883fd61d8
--- /dev/null
+++ b/src/backend/storage/aio/aio_funcs.c
@@ -0,0 +1,219 @@
+/*-------------------------------------------------------------------------
+ *
+ * aio_funcs.c
+ * AIO - SQL interface for AIO
+ *
+ *
+ * Portions Copyright (c) 1996-2025, PostgreSQL Global Development Group
+ * Portions Copyright (c) 1994, Regents of the University of California
+ *
+ * IDENTIFICATION
+ * src/backend/storage/aio/aio_funcs.c
+ *
+ *-------------------------------------------------------------------------
+ */
+
+#include "postgres.h"
+
+#include "storage/aio.h"
+#include "storage/aio_internal.h"
+#include "utils/builtins.h"
+#include "funcapi.h"
+#include "storage/proc.h"
+
+
+/*
+ * Byte length of an iovec.
+ */
+static size_t
+iov_byte_length(const struct iovec *iov, int cnt)
+{
+ size_t len = 0;
+
+ for (int i = 0; i < cnt; i++)
+ {
+ len += iov[i].iov_len;
+ }
+
+ return len;
+}
+
+Datum
+pg_get_aios(PG_FUNCTION_ARGS)
+{
+ ReturnSetInfo *rsinfo = (ReturnSetInfo *) fcinfo->resultinfo;
+
+ InitMaterializedSRF(fcinfo, 0);
+
+#define PG_GET_AIOS_COLS 15
+
+ for (uint64 i = 0; i < pgaio_ctl->io_handle_count; i++)
+ {
+ PgAioHandle *live_ioh = &pgaio_ctl->io_handles[i];
+ uint32 ioh_id = pgaio_io_get_id(live_ioh);
+ Datum values[PG_GET_AIOS_COLS] = {0};
+ bool nulls[PG_GET_AIOS_COLS] = {0};
+ ProcNumber owner;
+ PGPROC *owner_proc;
+ int32 owner_pid;
+ PgAioHandleState start_state;
+ uint64 start_generation;
+ PgAioHandle ioh_copy;
+ struct iovec iov_copy[PG_IOV_MAX];
+
+retry:
+
+ /*
+ * There is no lock that could prevent the state of the IO to advance
+ * concurrently - and we don't want to introduce one, as that would
+ * introduce atomics into a very common path. Instead we
+ *
+ * 1) Determine the state + generation of the IO.
+ *
+ * 2) Copy the IO to local memory.
+ *
+ * 3) Check if state or generation of the IO changed. If the state
+ * changed, retry, if the generation changed don't display the IO.
+ */
+
+ /* 1) from above */
+ start_generation = live_ioh->generation;
+ pg_read_barrier();
+ start_state = live_ioh->state;
+
+ if (start_state == PGAIO_HS_IDLE)
+ continue;
+
+ /* 2) from above */
+ memcpy(&ioh_copy, live_ioh, sizeof(PgAioHandle));
+
+ /*
+ * Safe to copy even if no iovec is used - we always reserve the
+ * required space.
+ */
+ memcpy(&iov_copy, &pgaio_ctl->iovecs[ioh_copy.iovec_off],
+ PG_IOV_MAX * sizeof(struct iovec));
+
+ /*
+ * Copy information about owner before 3) below, if the process exited
+ * it'd have to wait for the IO to finish first, which we would detect
+ * in 3).
+ */
+ owner = ioh_copy.owner_procno;
+ owner_proc = GetPGProcByNumber(owner);
+ owner_pid = owner_proc->pid;
+
+ /* 3) from above */
+ pg_read_barrier();
+
+ /*
+ * The IO completed and a new one was started with the same ID. Don't
+ * display it - it really started after this function was called.
+ * There be a risk of a livelock if we just retried endlessly, if IOs
+ * complete very quickly.
+ */
+ if (live_ioh->generation != start_generation)
+ continue;
+
+ /*
+ * The IOs state changed while we were "rendering" it. Just start from
+ * scratch. There's no risk of a livelock here, as an IO has a limited
+ * sets of states it can be in, and state changes go only in a single
+ * direction.
+ */
+ if (live_ioh->state != start_state)
+ goto retry;
+
+ /*
+ * Now that we have copied the IO into local memory and checked that
+ * it's still in the same state, we are not allowed to access "live"
+ * memory anymore. To make it slightly easier to catch such cases, set
+ * the "live" pointers to NULL.
+ */
+ live_ioh = NULL;
+ owner_proc = NULL;
+
+
+ /* column: owning pid */
+ if (owner_pid != 0)
+ values[0] = Int32GetDatum(owner_pid);
+ else
+ nulls[0] = false;
+
+ /* column: IO's id */
+ values[1] = ioh_id;
+
+ /* column: IO's generation */
+ values[2] = Int64GetDatum(start_generation);
+
+ /* column: IO's state */
+ values[3] = CStringGetTextDatum(pgaio_io_get_state_name(&ioh_copy));
+
+ /*
+ * If the IO is in PGAIO_HS_HANDED_OUT state, none of it's fields are
+ * valid yet (or are in the process of being set). Therefore we don't
+ * want to display any other columns.
+ */
+ if (start_state == PGAIO_HS_HANDED_OUT)
+ {
+ memset(nulls + 4, 1, (lengthof(nulls) - 4) * sizeof(bool));
+ goto display;
+ }
+
+ /* column: IO's operation */
+ values[4] = CStringGetTextDatum(pgaio_io_get_op_name(&ioh_copy));
+
+ /* columns: details about the IO's operation */
+ switch (ioh_copy.op)
+ {
+ case PGAIO_OP_INVALID:
+ nulls[5] = true;
+ nulls[6] = true;
+ break;
+ case PGAIO_OP_READV:
+ values[5] = Int64GetDatum(ioh_copy.op_data.read.offset);
+ values[6] =
+ Int64GetDatum(iov_byte_length(iov_copy, ioh_copy.op_data.read.iov_length));
+ break;
+ case PGAIO_OP_WRITEV:
+ values[5] = Int64GetDatum(ioh_copy.op_data.write.offset);
+ values[6] =
+ Int64GetDatum(iov_byte_length(iov_copy, ioh_copy.op_data.write.iov_length));
+ break;
+ }
+
+ /* column: IO's target */
+ values[7] = CStringGetTextDatum(pgaio_io_get_target_name(&ioh_copy));
+
+ /* column: length of IO's data array */
+ values[8] = Int16GetDatum(ioh_copy.handle_data_len);
+
+ /* column: raw result (i.e. some form of syscall return value) */
+ if (start_state == PGAIO_HS_COMPLETED_IO
+ || start_state == PGAIO_HS_COMPLETED_SHARED)
+ values[9] = Int32GetDatum(ioh_copy.result);
+ else
+ nulls[9] = true;
+
+ /*
+ * column: result in the higher level representation (unknown if not
+ * finished)
+ */
+ values[10] =
+ CStringGetTextDatum(pgaio_result_status_string(ioh_copy.distilled_result.status));
+
+ /* column: target description */
+ values[11] = CStringGetTextDatum(pgaio_io_get_target_description(&ioh_copy));
+
+ /* columns: one for each flag */
+ values[12] = BoolGetDatum(ioh_copy.flags & PGAIO_HF_SYNCHRONOUS);
+ values[13] = BoolGetDatum(ioh_copy.flags & PGAIO_HF_REFERENCES_LOCAL);
+ values[14] = BoolGetDatum(ioh_copy.flags & PGAIO_HF_BUFFERED);
+
+display:
+
+ tuplestore_putvalues(rsinfo->setResult, rsinfo->setDesc, values, nulls);
+ }
+
+ return (Datum) 0;
+}
diff --git a/src/backend/storage/aio/meson.build b/src/backend/storage/aio/meson.build
index 2f0f03d8071..da6df2d3654 100644
--- a/src/backend/storage/aio/meson.build
+++ b/src/backend/storage/aio/meson.build
@@ -3,6 +3,7 @@
backend_sources += files(
'aio.c',
'aio_callback.c',
+ 'aio_funcs.c',
'aio_init.c',
'aio_io.c',
'aio_target.c',
diff --git a/src/test/regress/expected/rules.out b/src/test/regress/expected/rules.out
index 62f69ac20b2..6570d0cddce 100644
--- a/src/test/regress/expected/rules.out
+++ b/src/test/regress/expected/rules.out
@@ -1286,6 +1286,22 @@ drop table cchild;
SELECT viewname, definition FROM pg_views
WHERE schemaname = 'pg_catalog'
ORDER BY viewname;
+pg_aios| SELECT pid,
+ io_id,
+ io_generation,
+ state,
+ operation,
+ "offset",
+ length,
+ target,
+ handle_data_len,
+ raw_result,
+ result,
+ target_desc,
+ f_sync,
+ f_localmem,
+ f_buffered
+ FROM pg_get_aios() pg_get_aios(pid, io_id, io_generation, state, operation, "offset", length, target, handle_data_len, raw_result, result, target_desc, f_sync, f_localmem, f_buffered);
pg_available_extension_versions| SELECT e.name,
e.version,
(x.extname IS NOT NULL) AS installed,
--
2.48.1.76.g4e746b1a31.dirty
[text/x-diff] v2.7-0010-Refactor-read_stream.c-s-circular-arithmetic.patch (4.6K, ../../4b3f32ug3cayekysqlgspz2qjmeb7lca3gvazayglxr2m3d4dv@il33accgsji7/11-v2.7-0010-Refactor-read_stream.c-s-circular-arithmetic.patch)
download | inline diff:
From b74207351a5b861760dd1fa7ad2d4f902e606922 Mon Sep 17 00:00:00 2001
From: Thomas Munro <thomas.munro@gmail.com>
Date: Sat, 15 Feb 2025 14:47:25 +1300
Subject: [PATCH v2.7 10/35] Refactor read_stream.c's circular arithmetic.
Several places have open-coded circular index arithmetic. Make some
common functions for better readability and consistent assertion
checking.
This avoids adding yet more open-coded duplication in later patches, and
standardizes on the vocabulary "advance" and "retreat" as used elsewhere
in PostgreSQL.
---
src/backend/storage/aio/read_stream.c | 78 +++++++++++++++++++++------
1 file changed, 61 insertions(+), 17 deletions(-)
diff --git a/src/backend/storage/aio/read_stream.c b/src/backend/storage/aio/read_stream.c
index 04bdb5e6d4b..dee51fa85a9 100644
--- a/src/backend/storage/aio/read_stream.c
+++ b/src/backend/storage/aio/read_stream.c
@@ -224,6 +224,55 @@ read_stream_unget_block(ReadStream *stream, BlockNumber blocknum)
stream->buffered_blocknum = blocknum;
}
+/*
+ * Increment index, wrapping around at queue size.
+ */
+static inline void
+read_stream_index_advance(ReadStream *stream, int16 *index)
+{
+ Assert(*index >= 0);
+ Assert(*index < stream->queue_size);
+
+ *index += 1;
+ if (*index == stream->queue_size)
+ *index = 0;
+}
+
+/*
+ * Increment index by n, wrapping around at queue size.
+ */
+static inline void
+read_stream_index_advance_n(ReadStream *stream, int16 *index, int16 n)
+{
+ Assert(*index >= 0);
+ Assert(*index < stream->queue_size);
+ Assert(n <= MAX_IO_COMBINE_LIMIT);
+
+ *index += n;
+ if (*index >= stream->queue_size)
+ *index -= stream->queue_size;
+
+ Assert(*index >= 0);
+ Assert(*index < stream->queue_size);
+}
+
+#if defined(CLOBBER_FREED_MEMORY) || defined(USE_VALGRIND)
+/*
+ * Decrement index, wrapping around at queue size.
+ */
+static inline void
+read_stream_index_retreat(ReadStream *stream, int16 *index)
+{
+ Assert(*index >= 0);
+ Assert(*index < stream->queue_size);
+
+ if (*index == 0)
+ *index = stream->queue_size - 1;
+ else
+ *index -= 1;
+}
+#endif
+
static void
read_stream_start_pending_read(ReadStream *stream, bool suppress_advice)
{
@@ -302,11 +351,8 @@ read_stream_start_pending_read(ReadStream *stream, bool suppress_advice)
&stream->buffers[stream->queue_size],
sizeof(stream->buffers[0]) * overflow);
- /* Compute location of start of next read, without using % operator. */
- buffer_index += nblocks;
- if (buffer_index >= stream->queue_size)
- buffer_index -= stream->queue_size;
- Assert(buffer_index >= 0 && buffer_index < stream->queue_size);
+ /* Move to the location of start of next read. */
+ read_stream_index_advance_n(stream, &buffer_index, nblocks);
stream->next_buffer_index = buffer_index;
/* Adjust the pending read to cover the remaining portion, if any. */
@@ -334,12 +380,12 @@ read_stream_look_ahead(ReadStream *stream, bool suppress_advice)
/*
* See which block the callback wants next in the stream. We need to
* compute the index of the Nth block of the pending read including
- * wrap-around, but we don't want to use the expensive % operator.
+ * wrap-around.
*/
- buffer_index = stream->next_buffer_index + stream->pending_read_nblocks;
- if (buffer_index >= stream->queue_size)
- buffer_index -= stream->queue_size;
- Assert(buffer_index >= 0 && buffer_index < stream->queue_size);
+ buffer_index = stream->next_buffer_index;
+ read_stream_index_advance_n(stream,
+ &buffer_index,
+ stream->pending_read_nblocks);
per_buffer_data = get_per_buffer_data(stream, buffer_index);
blocknum = read_stream_get_block(stream, per_buffer_data);
if (blocknum == InvalidBlockNumber)
@@ -781,12 +827,12 @@ read_stream_next_buffer(ReadStream *stream, void **per_buffer_data)
*/
if (stream->per_buffer_data)
{
+ int16 index;
void *per_buffer_data;
- per_buffer_data = get_per_buffer_data(stream,
- oldest_buffer_index == 0 ?
- stream->queue_size - 1 :
- oldest_buffer_index - 1);
+ index = oldest_buffer_index;
+ read_stream_index_retreat(stream, &index);
+ per_buffer_data = get_per_buffer_data(stream, index);
#if defined(CLOBBER_FREED_MEMORY)
/* This also tells Valgrind the memory is "noaccess". */
@@ -804,9 +850,7 @@ read_stream_next_buffer(ReadStream *stream, void **per_buffer_data)
stream->pinned_buffers--;
/* Advance oldest buffer, with wrap-around. */
- stream->oldest_buffer_index++;
- if (stream->oldest_buffer_index == stream->queue_size)
- stream->oldest_buffer_index = 0;
+ read_stream_index_advance(stream, &stream->oldest_buffer_index);
/* Prepare for the next call. */
read_stream_look_ahead(stream, false);
--
2.48.1.76.g4e746b1a31.dirty
[text/x-diff] v2.7-0011-Improve-buffer-pool-API-for-per-backend-pin-lim.patch (6.4K, ../../4b3f32ug3cayekysqlgspz2qjmeb7lca3gvazayglxr2m3d4dv@il33accgsji7/12-v2.7-0011-Improve-buffer-pool-API-for-per-backend-pin-lim.patch)
download | inline diff:
From 4457195bddb0f584eebac1567b417cf4cc6ed579 Mon Sep 17 00:00:00 2001
From: Thomas Munro <thomas.munro@gmail.com>
Date: Fri, 24 Jan 2025 10:59:39 +1300
Subject: [PATCH v2.7 11/35] Improve buffer pool API for per-backend pin
limits.
Previously the support functions assumed that you needed one additional
pin to make progress, and could optionally use some more. Add a couple
more functions for callers that want to know:
* what the maximum possible number could be, for space planning
purposes, called the "soft pin limit"
* how many additional pins they could acquire right now, without the
special case allowing one pin (ie for users that already hold pins and
can already make progress even if zero extra pins are available now)
These APIs are better suited to read_stream.c, which will be adjusted in
a follow-up patch. Also move the computation of the each backend's fair
share of the buffer pool to backend initialization time, since the
answer doesn't change and we don't want to perform a division operation
every time we compute availability.
---
src/include/storage/bufmgr.h | 4 ++
src/backend/storage/buffer/bufmgr.c | 75 ++++++++++++++++++++-------
src/backend/storage/buffer/localbuf.c | 16 ++++++
3 files changed, 77 insertions(+), 18 deletions(-)
diff --git a/src/include/storage/bufmgr.h b/src/include/storage/bufmgr.h
index b204e4731c1..74b5afe8a1a 100644
--- a/src/include/storage/bufmgr.h
+++ b/src/include/storage/bufmgr.h
@@ -290,6 +290,10 @@ extern bool HoldingBufferPinThatDelaysRecovery(void);
extern bool BgBufferSync(struct WritebackContext *wb_context);
+extern uint32 GetSoftPinLimit(void);
+extern uint32 GetSoftLocalPinLimit(void);
+extern uint32 GetAdditionalPinLimit(void);
+extern uint32 GetAdditionalLocalPinLimit(void);
extern void LimitAdditionalPins(uint32 *additional_pins);
extern void LimitAdditionalLocalPins(uint32 *additional_pins);
diff --git a/src/backend/storage/buffer/bufmgr.c b/src/backend/storage/buffer/bufmgr.c
index 7915ed624c1..11c146763db 100644
--- a/src/backend/storage/buffer/bufmgr.c
+++ b/src/backend/storage/buffer/bufmgr.c
@@ -211,6 +211,8 @@ static int32 PrivateRefCountOverflowed = 0;
static uint32 PrivateRefCountClock = 0;
static PrivateRefCountEntry *ReservedRefCountEntry = NULL;
+static uint32 MaxProportionalPins;
+
static void ReservePrivateRefCountEntry(void);
static PrivateRefCountEntry *NewPrivateRefCountEntry(Buffer buffer);
static PrivateRefCountEntry *GetPrivateRefCountEntry(Buffer buffer, bool do_move);
@@ -2097,6 +2099,46 @@ again:
return buf;
}
+/*
+ * Return the maximum number of buffer than this backend should try to pin at
+ * once, to avoid pinning more than its fair share. This is the highest value
+ * that GetAdditionalPinLimit() and LimitAdditionalPins() could ever return.
+ *
+ * It's called a soft limit because nothing stops a backend from trying to
+ * acquire more pins than this this with ReadBuffer(), but code that wants more
+ * for I/O optimizations should respect this per-backend limit when it can
+ * still make progress without them.
+ */
+uint32
+GetSoftPinLimit(void)
+{
+ return MaxProportionalPins;
+}
+
+/*
+ * Return the maximum number of additional buffers that this backend should
+ * pin if it wants to stay under the per-backend soft limit, considering the
+ * number of buffers it has already pinned.
+ */
+uint32
+GetAdditionalPinLimit(void)
+{
+ uint32 estimated_pins_held;
+
+ /*
+ * We get the number of "overflowed" pins for free, but don't know the
+ * number of pins in PrivateRefCountArray. The cost of calculating that
+ * exactly doesn't seem worth it, so just assume the max.
+ */
+ estimated_pins_held = PrivateRefCountOverflowed + REFCOUNT_ARRAY_ENTRIES;
+
+ /* Is this backend already holding more than its fair share? */
+ if (estimated_pins_held > MaxProportionalPins)
+ return 0;
+
+ return MaxProportionalPins - estimated_pins_held;
+}
+
/*
* Limit the number of pins a batch operation may additionally acquire, to
* avoid running out of pinnable buffers.
@@ -2112,28 +2154,15 @@ again:
void
LimitAdditionalPins(uint32 *additional_pins)
{
- uint32 max_backends;
- int max_proportional_pins;
+ uint32 limit;
if (*additional_pins <= 1)
return;
- max_backends = MaxBackends + NUM_AUXILIARY_PROCS;
- max_proportional_pins = NBuffers / max_backends;
-
- /*
- * Subtract the approximate number of buffers already pinned by this
- * backend. We get the number of "overflowed" pins for free, but don't
- * know the number of pins in PrivateRefCountArray. The cost of
- * calculating that exactly doesn't seem worth it, so just assume the max.
- */
- max_proportional_pins -= PrivateRefCountOverflowed + REFCOUNT_ARRAY_ENTRIES;
-
- if (max_proportional_pins <= 0)
- max_proportional_pins = 1;
-
- if (*additional_pins > max_proportional_pins)
- *additional_pins = max_proportional_pins;
+ limit = GetAdditionalPinLimit();
+ limit = Max(limit, 1);
+ if (limit < *additional_pins)
+ *additional_pins = limit;
}
/*
@@ -3575,6 +3604,16 @@ InitBufferManagerAccess(void)
{
HASHCTL hash_ctl;
+ /*
+ * The soft limit on the number of pins each backend should respect, bast
+ * on shared_buffers and the maximum number of connections possible.
+ * That's very pessimistic, but outside toy-sized shared_buffers it should
+ * allow plenty of pins. Higher level code that pins non-trivial numbers
+ * of buffers should use LimitAdditionalPins() or GetAdditionalPinLimit()
+ * to stay under this limit.
+ */
+ MaxProportionalPins = NBuffers / (MaxBackends + NUM_AUXILIARY_PROCS);
+
memset(&PrivateRefCountArray, 0, sizeof(PrivateRefCountArray));
hash_ctl.keysize = sizeof(int32);
diff --git a/src/backend/storage/buffer/localbuf.c b/src/backend/storage/buffer/localbuf.c
index 80b83444eb2..5378ba84316 100644
--- a/src/backend/storage/buffer/localbuf.c
+++ b/src/backend/storage/buffer/localbuf.c
@@ -286,6 +286,22 @@ GetLocalVictimBuffer(void)
return BufferDescriptorGetBuffer(bufHdr);
}
+/* see GetSoftPinLimit() */
+uint32
+GetSoftLocalPinLimit(void)
+{
+ /* Every backend has its own temporary buffers, and can pin them all. */
+ return num_temp_buffers;
+}
+
+/* see GetAdditionalPinLimit() */
+uint32
+GetAdditionalLocalPinLimit(void)
+{
+ Assert(NLocalPinnedBuffers <= num_temp_buffers);
+ return num_temp_buffers - NLocalPinnedBuffers;
+}
+
/* see LimitAdditionalPins() */
void
LimitAdditionalLocalPins(uint32 *additional_pins)
--
2.48.1.76.g4e746b1a31.dirty
[text/x-diff] v2.7-0012-Respect-pin-limits-accurately-in-read_stream.c.patch (9.9K, ../../4b3f32ug3cayekysqlgspz2qjmeb7lca3gvazayglxr2m3d4dv@il33accgsji7/13-v2.7-0012-Respect-pin-limits-accurately-in-read_stream.c.patch)
download | inline diff:
From 7c0802c756fb9547e2a9240ee00d776011b381ad Mon Sep 17 00:00:00 2001
From: Thomas Munro <thomas.munro@gmail.com>
Date: Fri, 24 Jan 2025 23:52:53 +1300
Subject: [PATCH v2.7 12/35] Respect pin limits accurately in read_stream.c.
Read streams pin multiple buffers at once as required to combine I/O.
This also avoids having to unpin and repin later when issuing read-ahead
advice, and will be needed for proposed work that starts "real"
asynchronous I/O.
To avoid pinning too much of the buffer pool at once, we previously used
LimitAdditionalBuffers() to avoid pinning more than this backend's fair
share of the pool as a cap. The coding was a naive and only checked the
cap once at stream initialization.
This commit moves the check to the time of use with new bufmgr APIs from
an earlier commit, since the result might change later due to pins
acquired later outside this stream. No extra CPU cycles are added to
the all-buffered fast-path code (it only pins one buffer at a time), but
the I/O-starting path now re-checks the limit every time using simple
arithmetic.
In practice it was difficult to exceed the limit, but you could contrive
a workload to do it using multiple CURSORs and FETCHing from sequential
scans in round-robin fashion, so that each underlying stream computes
its limit before all the others have ramped up to their full look-ahead
distance. Therefore, no back-patch for now.
Per code review from Andres, in the course of his AIO work.
Reported-by: Andres Freund <andres@anarazel.de>
---
src/backend/storage/aio/read_stream.c | 111 ++++++++++++++++++++++----
1 file changed, 95 insertions(+), 16 deletions(-)
diff --git a/src/backend/storage/aio/read_stream.c b/src/backend/storage/aio/read_stream.c
index dee51fa85a9..6a39bd1d92d 100644
--- a/src/backend/storage/aio/read_stream.c
+++ b/src/backend/storage/aio/read_stream.c
@@ -115,6 +115,7 @@ struct ReadStream
int16 pinned_buffers;
int16 distance;
bool advice_enabled;
+ bool temporary;
/*
* One-block buffer to support 'ungetting' a block number, to resolve flow
@@ -274,7 +275,9 @@ read_stream_index_retreat(ReadStream *stream, int16 *index)
#endif
static void
-read_stream_start_pending_read(ReadStream *stream, bool suppress_advice)
+read_stream_start_pending_read(ReadStream *stream,
+ int16 buffer_limit,
+ bool suppress_advice)
{
bool need_wait;
int nblocks;
@@ -308,10 +311,14 @@ read_stream_start_pending_read(ReadStream *stream, bool suppress_advice)
else
flags = 0;
- /* We say how many blocks we want to read, but may be smaller on return. */
+ /*
+ * We say how many blocks we want to read, but may be smaller on return.
+ * On memory-constrained systems we may be also have to ask for a smaller
+ * read ourselves.
+ */
buffer_index = stream->next_buffer_index;
io_index = stream->next_io_index;
- nblocks = stream->pending_read_nblocks;
+ nblocks = Min(buffer_limit, stream->pending_read_nblocks);
need_wait = StartReadBuffers(&stream->ios[io_index].op,
&stream->buffers[buffer_index],
stream->pending_read_blocknum,
@@ -360,11 +367,60 @@ read_stream_start_pending_read(ReadStream *stream, bool suppress_advice)
stream->pending_read_nblocks -= nblocks;
}
+/*
+ * How many more buffers could we use, while respecting the soft limit?
+ */
+static int16
+read_stream_get_buffer_limit(ReadStream *stream)
+{
+ uint32 buffers;
+
+ /* Check how many local or shared pins we could acquire. */
+ if (stream->temporary)
+ buffers = GetAdditionalLocalPinLimit();
+ else
+ buffers = GetAdditionalPinLimit();
+
+ /*
+ * Each stream is always allowed to try to acquire one pin if it doesn't
+ * hold one already. This is needed to guarantee progress, and just like
+ * the simple ReadBuffer() operation in code that is not using this stream
+ * API, if a buffer can't be pinned we'll raise an error when trying to
+ * pin, ie the buffer pool is simply too small for the workload.
+ */
+ if (buffers == 0 && stream->pinned_buffers == 0)
+ return 1;
+
+ /*
+ * Otherwise, see how many additional pins the backend can currently pin,
+ * which may be zero. As above, this only guarantees that this backend
+ * won't use more than its fair share if all backends can respect the soft
+ * limit, not that a pin can actually be acquired without error.
+ */
+ return Min(buffers, INT16_MAX);
+}
+
static void
read_stream_look_ahead(ReadStream *stream, bool suppress_advice)
{
+ int16 buffer_limit;
+
+ /*
+ * Check how many pins we could acquire now. We do this here rather than
+ * pushing it down into read_stream_start_pending_read(), because it
+ * allows more flexibility in behavior when we run out of allowed pins.
+ * Currently the policy is to start an I/O when we've run out of allowed
+ * pins only if we have to to make progress, and otherwise to stop looking
+ * ahead until more pins become available, so that we don't start issuing
+ * a lot of smaller I/Os, prefering to build the largest ones we can. This
+ * choice is debatable, but it should only really come up with the buffer
+ * pool/connection ratio is very constrained.
+ */
+ buffer_limit = read_stream_get_buffer_limit(stream);
+
while (stream->ios_in_progress < stream->max_ios &&
- stream->pinned_buffers + stream->pending_read_nblocks < stream->distance)
+ stream->pinned_buffers + stream->pending_read_nblocks <
+ Min(stream->distance, buffer_limit))
{
BlockNumber blocknum;
int16 buffer_index;
@@ -372,7 +428,9 @@ read_stream_look_ahead(ReadStream *stream, bool suppress_advice)
if (stream->pending_read_nblocks == io_combine_limit)
{
- read_stream_start_pending_read(stream, suppress_advice);
+ read_stream_start_pending_read(stream, buffer_limit,
+ suppress_advice);
+ buffer_limit = read_stream_get_buffer_limit(stream);
suppress_advice = false;
continue;
}
@@ -406,11 +464,12 @@ read_stream_look_ahead(ReadStream *stream, bool suppress_advice)
/* We have to start the pending read before we can build another. */
while (stream->pending_read_nblocks > 0)
{
- read_stream_start_pending_read(stream, suppress_advice);
+ read_stream_start_pending_read(stream, buffer_limit, suppress_advice);
+ buffer_limit = read_stream_get_buffer_limit(stream);
suppress_advice = false;
- if (stream->ios_in_progress == stream->max_ios)
+ if (stream->ios_in_progress == stream->max_ios || buffer_limit == 0)
{
- /* And we've hit the limit. Rewind, and stop here. */
+ /* And we've hit a limit. Rewind, and stop here. */
read_stream_unget_block(stream, blocknum);
return;
}
@@ -426,16 +485,17 @@ read_stream_look_ahead(ReadStream *stream, bool suppress_advice)
* limit, preferring to give it another chance to grow to full
* io_combine_limit size once more buffers have been consumed. However,
* if we've already reached io_combine_limit, or we've reached the
- * distance limit and there isn't anything pinned yet, or the callback has
- * signaled end-of-stream, we start the read immediately.
+ * distance limit or buffer limit and there isn't anything pinned yet, or
+ * the callback has signaled end-of-stream, we start the read immediately.
*/
if (stream->pending_read_nblocks > 0 &&
(stream->pending_read_nblocks == io_combine_limit ||
- (stream->pending_read_nblocks == stream->distance &&
+ ((stream->pending_read_nblocks == stream->distance ||
+ stream->pending_read_nblocks == buffer_limit) &&
stream->pinned_buffers == 0) ||
stream->distance == 0) &&
stream->ios_in_progress < stream->max_ios)
- read_stream_start_pending_read(stream, suppress_advice);
+ read_stream_start_pending_read(stream, buffer_limit, suppress_advice);
}
/*
@@ -464,6 +524,7 @@ read_stream_begin_impl(int flags,
int max_ios;
int strategy_pin_limit;
uint32 max_pinned_buffers;
+ uint32 max_possible_buffer_limit;
Oid tablespace_id;
/*
@@ -511,12 +572,23 @@ read_stream_begin_impl(int flags,
strategy_pin_limit = GetAccessStrategyPinLimit(strategy);
max_pinned_buffers = Min(strategy_pin_limit, max_pinned_buffers);
- /* Don't allow this backend to pin more than its share of buffers. */
+ /*
+ * Also limit by the maximum possible number of pins we could be allowed
+ * to acquire according to bufmgr. We may not be able to use them all due
+ * to other pins held by this backend, but we'll enforce the dynamic limit
+ * later when starting I/O.
+ */
if (SmgrIsTemp(smgr))
- LimitAdditionalLocalPins(&max_pinned_buffers);
+ max_possible_buffer_limit = GetSoftLocalPinLimit();
else
- LimitAdditionalPins(&max_pinned_buffers);
- Assert(max_pinned_buffers > 0);
+ max_possible_buffer_limit = GetSoftPinLimit();
+ max_pinned_buffers = Min(max_pinned_buffers, max_possible_buffer_limit);
+
+ /*
+ * The soft limit might be zero on a system configured with more
+ * connections than buffers. We need at least one.
+ */
+ max_pinned_buffers = Max(1, max_pinned_buffers);
/*
* We need one extra entry for buffers and per-buffer data, because users
@@ -576,6 +648,7 @@ read_stream_begin_impl(int flags,
stream->callback = callback;
stream->callback_private_data = callback_private_data;
stream->buffered_blocknum = InvalidBlockNumber;
+ stream->temporary = SmgrIsTemp(smgr);
/*
* Skip the initial ramp-up phase if the caller says we're going to be
@@ -704,6 +777,12 @@ read_stream_next_buffer(ReadStream *stream, void **per_buffer_data)
* arbitrary I/O entry (they're all free). We don't have to
* adjust pinned_buffers because we're transferring one to caller
* but pinning one more.
+ *
+ * In the fast path we don't need to check the pin limit. We're
+ * always allowed at least one pin so that progress can be made,
+ * and that's all we need here. Although two pins are momentarily
+ * held at the same time, the model used here is that the stream
+ * holds only one, and the other now belongs to the caller.
*/
if (likely(!StartReadBuffer(&stream->ios[0].op,
&stream->buffers[oldest_buffer_index],
--
2.48.1.76.g4e746b1a31.dirty
[text/x-diff] v2.7-0013-Support-buffer-forwarding-in-read_stream.c.patch (8.9K, ../../4b3f32ug3cayekysqlgspz2qjmeb7lca3gvazayglxr2m3d4dv@il33accgsji7/14-v2.7-0013-Support-buffer-forwarding-in-read_stream.c.patch)
download | inline diff:
From 7d39560f8da9b08297b6ba528101316eb54c0970 Mon Sep 17 00:00:00 2001
From: Thomas Munro <thomas.munro@gmail.com>
Date: Thu, 30 Jan 2025 11:42:03 +1300
Subject: [PATCH v2.7 13/35] Support buffer forwarding in read_stream.c.
In preparation for a following change to the buffer manager, teach read
stream to keep track of buffers that were "forwarded" from one call to
StartReadBuffers() to the next.
Since StartReadBuffers() buffers argument will become an in/out
argument, we need to initialize the buffer queue entries with
InvalidBuffer. We don't want to do that up front, because we try to
keep stream initialization cheap and code that uses the fast path stays
in one single buffer queue element. Satisfy both goals by initializing
the queue incrementally on the first cycle.
---
src/backend/storage/aio/read_stream.c | 108 ++++++++++++++++++++++----
1 file changed, 94 insertions(+), 14 deletions(-)
diff --git a/src/backend/storage/aio/read_stream.c b/src/backend/storage/aio/read_stream.c
index 6a39bd1d92d..e58e0edf221 100644
--- a/src/backend/storage/aio/read_stream.c
+++ b/src/backend/storage/aio/read_stream.c
@@ -112,8 +112,10 @@ struct ReadStream
int16 ios_in_progress;
int16 queue_size;
int16 max_pinned_buffers;
+ int16 forwarded_buffers;
int16 pinned_buffers;
int16 distance;
+ int16 initialized_buffers;
bool advice_enabled;
bool temporary;
@@ -280,7 +282,9 @@ read_stream_start_pending_read(ReadStream *stream,
bool suppress_advice)
{
bool need_wait;
+ int requested_nblocks;
int nblocks;
+ int forwarded;
int flags;
int16 io_index;
int16 overflow;
@@ -312,13 +316,34 @@ read_stream_start_pending_read(ReadStream *stream,
flags = 0;
/*
- * We say how many blocks we want to read, but may be smaller on return.
- * On memory-constrained systems we may be also have to ask for a smaller
- * read ourselves.
+ * On buffer-constrained systems we may need to limit the I/O size by the
+ * available pin count.
*/
+ requested_nblocks = Min(buffer_limit, stream->pending_read_nblocks);
+ nblocks = requested_nblocks;
buffer_index = stream->next_buffer_index;
io_index = stream->next_io_index;
- nblocks = Min(buffer_limit, stream->pending_read_nblocks);
+
+ /*
+ * The first time around the queue we initialize it as we go, including
+ * the overflow zone, because otherwise the entries would appear as
+ * forwarded buffers. This avoids initializing the whole queue up front
+ * in cases where it is large but we don't ever use it due to the
+ * all-cached fast path or small scans.
+ */
+ while (stream->initialized_buffers < buffer_index + nblocks)
+ stream->buffers[stream->initialized_buffers++] = InvalidBuffer;
+
+ /*
+ * Start the I/O. Any buffers that are not InvalidBuffer will be
+ * interpreted as already pinned, forwarded by an earlier call to
+ * StartReadBuffers(), and must map to the expected blocks. The nblocks
+ * value may be smaller on return indicating the size of the I/O that
+ * could be started. Buffers beyond the output nblocks number may also
+ * have been pinned without starting I/O due to various edge cases. In
+ * that case we'll just leave them in the queue ahead of us, "forwarded"
+ * to the next call, avoiding the need to unpin/repin.
+ */
need_wait = StartReadBuffers(&stream->ios[io_index].op,
&stream->buffers[buffer_index],
stream->pending_read_blocknum,
@@ -347,16 +372,35 @@ read_stream_start_pending_read(ReadStream *stream,
stream->seq_blocknum = stream->pending_read_blocknum + nblocks;
}
+ /*
+ * How many pins were acquired but forwarded to the next call? These need
+ * to be passed to the next StartReadBuffers() call, or released if the
+ * stream ends early. We need the number for accounting purposes, since
+ * they are not counted in stream->pinned_buffers but we already hold
+ * them.
+ */
+ forwarded = 0;
+ while (nblocks + forwarded < requested_nblocks &&
+ stream->buffers[buffer_index + nblocks + forwarded] != InvalidBuffer)
+ forwarded++;
+ stream->forwarded_buffers = forwarded;
+
/*
* We gave a contiguous range of buffer space to StartReadBuffers(), but
- * we want it to wrap around at queue_size. Slide overflowing buffers to
- * the front of the array.
+ * we want it to wrap around at queue_size. Copy overflowing buffers to
+ * the front of the array where they'll be consumed, but also leave a copy
+ * in the overflow zone which the I/O operation has a pointer to (it needs
+ * a contiguous array). Both copies will be cleared when the buffers are
+ * handed to the consumer.
*/
- overflow = (buffer_index + nblocks) - stream->queue_size;
+ overflow = (buffer_index + nblocks + forwarded) - stream->queue_size;
if (overflow > 0)
- memmove(&stream->buffers[0],
- &stream->buffers[stream->queue_size],
- sizeof(stream->buffers[0]) * overflow);
+ {
+ Assert(overflow < stream->queue_size); /* can't overlap */
+ memcpy(&stream->buffers[0],
+ &stream->buffers[stream->queue_size],
+ sizeof(stream->buffers[0]) * overflow);
+ }
/* Move to the location of start of next read. */
read_stream_index_advance_n(stream, &buffer_index, nblocks);
@@ -381,6 +425,15 @@ read_stream_get_buffer_limit(ReadStream *stream)
else
buffers = GetAdditionalPinLimit();
+ /*
+ * If we already have some forwarded buffers, we can certainly use those.
+ * They are already pinned, and are mapped to the starting blocks of the
+ * pending read, they just don't have any I/O started yet and are not
+ * counted in stream->pinned_buffers.
+ */
+ Assert(stream->forwarded_buffers <= stream->pending_read_nblocks);
+ buffers += stream->forwarded_buffers;
+
/*
* Each stream is always allowed to try to acquire one pin if it doesn't
* hold one already. This is needed to guarantee progress, and just like
@@ -389,7 +442,7 @@ read_stream_get_buffer_limit(ReadStream *stream)
* pin, ie the buffer pool is simply too small for the workload.
*/
if (buffers == 0 && stream->pinned_buffers == 0)
- return 1;
+ buffers = 1;
/*
* Otherwise, see how many additional pins the backend can currently pin,
@@ -755,10 +808,12 @@ read_stream_next_buffer(ReadStream *stream, void **per_buffer_data)
/* Fast path assumptions. */
Assert(stream->ios_in_progress == 0);
+ Assert(stream->forwarded_buffers == 0);
Assert(stream->pinned_buffers == 1);
Assert(stream->distance == 1);
Assert(stream->pending_read_nblocks == 0);
Assert(stream->per_buffer_data_size == 0);
+ Assert(stream->initialized_buffers > stream->oldest_buffer_index);
/* We're going to return the buffer we pinned last time. */
oldest_buffer_index = stream->oldest_buffer_index;
@@ -807,6 +862,7 @@ read_stream_next_buffer(ReadStream *stream, void **per_buffer_data)
stream->distance = 0;
stream->oldest_buffer_index = stream->next_buffer_index;
stream->pinned_buffers = 0;
+ stream->buffers[oldest_buffer_index] = InvalidBuffer;
}
stream->fast_path = false;
@@ -891,10 +947,15 @@ read_stream_next_buffer(ReadStream *stream, void **per_buffer_data)
}
}
-#ifdef CLOBBER_FREED_MEMORY
- /* Clobber old buffer for debugging purposes. */
+ /*
+ * We must zap this queue entry, or else it would appear as a forwarded
+ * buffer. If it's potentially in the overflow zone (ie it wrapped around
+ * the queue), also zap that copy.
+ */
stream->buffers[oldest_buffer_index] = InvalidBuffer;
-#endif
+ if (oldest_buffer_index < io_combine_limit - 1)
+ stream->buffers[stream->queue_size + oldest_buffer_index] =
+ InvalidBuffer;
#if defined(CLOBBER_FREED_MEMORY) || defined(USE_VALGRIND)
@@ -937,6 +998,7 @@ read_stream_next_buffer(ReadStream *stream, void **per_buffer_data)
#ifndef READ_STREAM_DISABLE_FAST_PATH
/* See if we can take the fast path for all-cached scans next time. */
if (stream->ios_in_progress == 0 &&
+ stream->forwarded_buffers == 0 &&
stream->pinned_buffers == 1 &&
stream->distance == 1 &&
stream->pending_read_nblocks == 0 &&
@@ -972,6 +1034,7 @@ read_stream_next_block(ReadStream *stream, BufferAccessStrategy *strategy)
void
read_stream_reset(ReadStream *stream)
{
+ int16 index;
Buffer buffer;
/* Stop looking ahead. */
@@ -985,6 +1048,23 @@ read_stream_reset(ReadStream *stream)
while ((buffer = read_stream_next_buffer(stream, NULL)) != InvalidBuffer)
ReleaseBuffer(buffer);
+ /* Unpin any unused forwarded buffers. */
+ index = stream->next_buffer_index;
+ while (index < stream->initialized_buffers &&
+ (buffer = stream->buffers[index]) != InvalidBuffer)
+ {
+ Assert(stream->forwarded_buffers > 0);
+ stream->forwarded_buffers--;
+ ReleaseBuffer(buffer);
+
+ stream->buffers[index] = InvalidBuffer;
+ if (index < io_combine_limit - 1)
+ stream->buffers[stream->queue_size + index] = InvalidBuffer;
+
+ read_stream_index_advance(stream, &index);
+ }
+
+ Assert(stream->forwarded_buffers == 0);
Assert(stream->pinned_buffers == 0);
Assert(stream->ios_in_progress == 0);
--
2.48.1.76.g4e746b1a31.dirty
[text/x-diff] v2.7-0014-Support-buffer-forwarding-in-StartReadBuffers.patch (10.5K, ../../4b3f32ug3cayekysqlgspz2qjmeb7lca3gvazayglxr2m3d4dv@il33accgsji7/15-v2.7-0014-Support-buffer-forwarding-in-StartReadBuffers.patch)
download | inline diff:
From ddfe28078981ed11416fc9bc9eb69c9b0f3e4714 Mon Sep 17 00:00:00 2001
From: Thomas Munro <thomas.munro@gmail.com>
Date: Mon, 10 Feb 2025 21:55:40 +1300
Subject: [PATCH v2.7 14/35] Support buffer forwarding in StartReadBuffers().
Sometimes we have to perform a short read because we hit a cached block
that ends a contiguous run of blocks requiring I/O. We don't want
StartReadBuffers() to have to start more than one I/O, so we stop there.
We also don't want to have to unpin the cached block (and repin it
later), so previously we'd silently pretend the hit was part of the I/O,
and just leave it out of the read from disk. Now, we'll "forward" it to
the next call. We still write it to the buffers[] array for the caller
to pass back to us later, but it's not included in *nblocks.
This policy means that we no longer mix hits and misses in a single
operation's results, so we avoid the requirement to call
WaitReadBuffers(), which might stall, before the caller can make use of
the hits. The caller will get the hit in the next call instead, and
know that it doesn't have to wait. That's important for later work on
out-of-order read streams that minimize I/O stalls.
This also makes life easier for proposed work on true AIO, which
occasionally needs to split a large I/O after pinning all the buffers,
while the current coding only ever forwards a single bookending hit.
This API is natural for read_stream.c: it just leaves forwarded buffers
where they are in its circular queue, where the next call will pick them
up and continue, minimizing pin churn.
If we ever think of a good reason to disable this feature, i.e. for
other users of StartReadBuffers() that don't want to deal with forwarded
buffers, then we could add a flag for that. For now read_steam.c is the
only user.
---
src/include/storage/bufmgr.h | 1 -
src/backend/storage/buffer/bufmgr.c | 128 ++++++++++++++++++++--------
2 files changed, 91 insertions(+), 38 deletions(-)
diff --git a/src/include/storage/bufmgr.h b/src/include/storage/bufmgr.h
index 74b5afe8a1a..307f36af384 100644
--- a/src/include/storage/bufmgr.h
+++ b/src/include/storage/bufmgr.h
@@ -130,7 +130,6 @@ struct ReadBuffersOperation
BlockNumber blocknum;
int flags;
int16 nblocks;
- int16 io_buffers_len;
};
typedef struct ReadBuffersOperation ReadBuffersOperation;
diff --git a/src/backend/storage/buffer/bufmgr.c b/src/backend/storage/buffer/bufmgr.c
index 11c146763db..75a928e802a 100644
--- a/src/backend/storage/buffer/bufmgr.c
+++ b/src/backend/storage/buffer/bufmgr.c
@@ -1257,10 +1257,10 @@ StartReadBuffersImpl(ReadBuffersOperation *operation,
Buffer *buffers,
BlockNumber blockNum,
int *nblocks,
- int flags)
+ int flags,
+ bool allow_forwarding)
{
int actual_nblocks = *nblocks;
- int io_buffers_len = 0;
int maxcombine = 0;
Assert(*nblocks > 0);
@@ -1270,30 +1270,80 @@ StartReadBuffersImpl(ReadBuffersOperation *operation,
{
bool found;
- buffers[i] = PinBufferForBlock(operation->rel,
- operation->smgr,
- operation->persistence,
- operation->forknum,
- blockNum + i,
- operation->strategy,
- &found);
+ if (allow_forwarding && buffers[i] != InvalidBuffer)
+ {
+ BufferDesc *bufHdr;
+
+ /*
+ * This is a buffer that was pinned by an earlier call to
+ * StartReadBuffers(), but couldn't be handled in one operation at
+ * that time. The operation was split, and the caller has passed
+ * an already pinned buffer back to us to handle the rest of the
+ * operation. It must continue at the expected block number.
+ */
+ Assert(BufferGetBlockNumber(buffers[i]) == blockNum + i);
+
+ /*
+ * It might be an already valid buffer (a hit) that followed the
+ * final contiguous block of an earlier I/O (a miss) marking the
+ * end of it, or a buffer that some other backend has since made
+ * valid by performing the I/O for us, in which case we can handle
+ * it as a hit now. It is safe to check for a BM_VALID flag with
+ * a relaxed load, because we got a fresh view of it while pinning
+ * it in the previous call.
+ *
+ * On the other hand if we don't see BM_VALID yet, it must be an
+ * I/O that was split by the previous call and we need to try to
+ * start a new I/O from this block. We're also racing against any
+ * other backend that might start the I/O or even manage to mark
+ * it BM_VALID after this check, BM_VALID after this check, but
+ * StartBufferIO() will handle those cases.
+ */
+ if (BufferIsLocal(buffers[i]))
+ bufHdr = GetLocalBufferDescriptor(-buffers[i] - 1);
+ else
+ bufHdr = GetBufferDescriptor(buffers[i] - 1);
+ found = pg_atomic_read_u32(&bufHdr->state) & BM_VALID;
+ }
+ else
+ {
+ buffers[i] = PinBufferForBlock(operation->rel,
+ operation->smgr,
+ operation->persistence,
+ operation->forknum,
+ blockNum + i,
+ operation->strategy,
+ &found);
+ }
if (found)
{
/*
- * Terminate the read as soon as we get a hit. It could be a
- * single buffer hit, or it could be a hit that follows a readable
- * range. We don't want to create more than one readable range,
- * so we stop here.
+ * We have a hit. If it's the first block in the requested range,
+ * we can return it immediately and report that WaitReadBuffers()
+ * does not need to be called. If the initial value of *nblocks
+ * was larger, the caller will have to call again for the rest.
*/
- actual_nblocks = i + 1;
+ if (i == 0)
+ {
+ *nblocks = 1;
+ return false;
+ }
+
+ /*
+ * Otherwise we already have an I/O to perform, but this block
+ * can't be included as it is already valid. Split the I/O here.
+ * There may or may not be more blocks requiring I/O after this
+ * one, we haven't checked, but it can't be contiguous with this
+ * hit in the way. We'll leave this buffer pinned, forwarding it
+ * to the next call, avoiding the need to unpin it here and re-pin
+ * it in the next call.
+ */
+ actual_nblocks = i;
break;
}
else
{
- /* Extend the readable range to cover this block. */
- io_buffers_len++;
-
/*
* Check how many blocks we can cover with the same IO. The smgr
* implementation might e.g. be limited due to a segment boundary.
@@ -1314,15 +1364,11 @@ StartReadBuffersImpl(ReadBuffersOperation *operation,
}
*nblocks = actual_nblocks;
- if (likely(io_buffers_len == 0))
- return false;
-
/* Populate information needed for I/O. */
operation->buffers = buffers;
operation->blocknum = blockNum;
operation->flags = flags;
operation->nblocks = actual_nblocks;
- operation->io_buffers_len = io_buffers_len;
if (flags & READ_BUFFERS_ISSUE_ADVICE)
{
@@ -1337,7 +1383,7 @@ StartReadBuffersImpl(ReadBuffersOperation *operation,
smgrprefetch(operation->smgr,
operation->forknum,
blockNum,
- operation->io_buffers_len);
+ actual_nblocks);
}
/* Indicate that WaitReadBuffers() should be called. */
@@ -1351,11 +1397,21 @@ StartReadBuffersImpl(ReadBuffersOperation *operation,
* actual number, which may be fewer than requested. Caller sets some of the
* members of operation; see struct definition.
*
+ * The initial contents of the elements of buffers up to *nblocks should
+ * either be InvalidBuffer or an already-pinned buffer that was left by an
+ * preceding call to StartReadBuffers() that had to be split. On return, some
+ * elements of buffers may hold pinned buffers beyond the number indicated by
+ * the updated value of *nblocks. Operations are split on boundaries known to
+ * smgr (eg md.c segment boundaries that require crossing into a different
+ * underlying file), or when already cached blocks are found in the buffer
+ * that prevent the formation of a contiguous read.
+ *
* If false is returned, no I/O is necessary. If true is returned, one I/O
* has been started, and WaitReadBuffers() must be called with the same
* operation object before the buffers are accessed. Along with the operation
* object, the caller-supplied array of buffers must remain valid until
- * WaitReadBuffers() is called.
+ * WaitReadBuffers() is called, and any forwarded buffers must also be
+ * preserved for a future call unless explicitly released.
*
* Currently the I/O is only started with optional operating system advice if
* requested by the caller with READ_BUFFERS_ISSUE_ADVICE, and the real I/O
@@ -1369,13 +1425,18 @@ StartReadBuffers(ReadBuffersOperation *operation,
int *nblocks,
int flags)
{
- return StartReadBuffersImpl(operation, buffers, blockNum, nblocks, flags);
+ return StartReadBuffersImpl(operation, buffers, blockNum, nblocks, flags,
+ true /* expect forwarded buffers */ );
}
/*
* Single block version of the StartReadBuffers(). This might save a few
* instructions when called from another translation unit, because it is
* specialized for nblocks == 1.
+ *
+ * This version does not support "forwarded" buffers: they cannot be created
+ * by reading only one block, and the current contents of *buffer is ignored
+ * on entry.
*/
bool
StartReadBuffer(ReadBuffersOperation *operation,
@@ -1386,7 +1447,8 @@ StartReadBuffer(ReadBuffersOperation *operation,
int nblocks = 1;
bool result;
- result = StartReadBuffersImpl(operation, buffer, blocknum, &nblocks, flags);
+ result = StartReadBuffersImpl(operation, buffer, blocknum, &nblocks, flags,
+ false /* single block, no forwarding */ );
Assert(nblocks == 1); /* single block can't be short */
return result;
@@ -1416,24 +1478,16 @@ WaitReadBuffers(ReadBuffersOperation *operation)
IOObject io_object;
char persistence;
- /*
- * Currently operations are only allowed to include a read of some range,
- * with an optional extra buffer that is already pinned at the end. So
- * nblocks can be at most one more than io_buffers_len.
- */
- Assert((operation->nblocks == operation->io_buffers_len) ||
- (operation->nblocks == operation->io_buffers_len + 1));
-
/* Find the range of the physical read we need to perform. */
- nblocks = operation->io_buffers_len;
- if (nblocks == 0)
- return; /* nothing to do */
-
+ nblocks = operation->nblocks;
buffers = &operation->buffers[0];
blocknum = operation->blocknum;
forknum = operation->forknum;
persistence = operation->persistence;
+ Assert(nblocks > 0);
+ Assert(nblocks <= MAX_IO_COMBINE_LIMIT);
+
if (persistence == RELPERSISTENCE_TEMP)
{
io_context = IOCONTEXT_NORMAL;
--
2.48.1.76.g4e746b1a31.dirty
[text/x-diff] v2.7-0015-tests-Expand-temp-table-tests-to-some-pin-relat.patch (9.6K, ../../4b3f32ug3cayekysqlgspz2qjmeb7lca3gvazayglxr2m3d4dv@il33accgsji7/16-v2.7-0015-tests-Expand-temp-table-tests-to-some-pin-relat.patch)
download | inline diff:
From c157888c49ce7357b0d39f26d51f9d6144f9f56a Mon Sep 17 00:00:00 2001
From: Andres Freund <andres@anarazel.de>
Date: Tue, 4 Mar 2025 11:07:47 -0500
Subject: [PATCH v2.7 15/35] tests: Expand temp table tests to some pin related
matters
Author:
Reviewed-By:
Discussion: https://postgr.es/m/
Backpatch:
---
src/test/regress/expected/temp.out | 156 +++++++++++++++++++++++++++++
src/test/regress/parallel_schedule | 2 +-
src/test/regress/sql/temp.sql | 107 ++++++++++++++++++++
3 files changed, 264 insertions(+), 1 deletion(-)
diff --git a/src/test/regress/expected/temp.out b/src/test/regress/expected/temp.out
index 2a246a7e123..91fe519b1cc 100644
--- a/src/test/regress/expected/temp.out
+++ b/src/test/regress/expected/temp.out
@@ -410,3 +410,159 @@ SELECT current_schema() ~ 'pg_temp' AS is_temp_schema;
PREPARE TRANSACTION 'twophase_search';
ERROR: cannot PREPARE a transaction that has operated on temporary objects
+-- Tests to verify we recover correctly from exhausting buffer pins and
+-- related matters.
+-- use lower possible buffer limit to make the test cheaper
+\c
+SET temp_buffers = 100;
+CREATE TEMPORARY TABLE test_temp(a int not null unique, b TEXT not null, cnt int not null);
+INSERT INTO test_temp SELECT generate_series(1, 10000) as id, repeat('a', 200), 0;
+-- should be at least 2x as large than temp_buffers
+SELECT pg_relation_size('test_temp') / current_setting('block_size')::int8 > 200;
+ ?column?
+----------
+ t
+(1 row)
+
+-- Don't want cursor names and plpgsql function lines in the error messages
+\set VERBOSITY terse
+/* helper function to create cursors for each page in [p_start, p_end] */
+CREATE FUNCTION test_temp_pin(p_start int, p_end int)
+RETURNS void
+LANGUAGE plpgsql
+AS $f$
+ DECLARE
+ cursorname text;
+ query text;
+ BEGIN
+ FOR i IN p_start..p_end LOOP
+ cursorname = 'c_'||i;
+ query = format($q$DECLARE %I CURSOR FOR SELECT ctid FROM test_temp WHERE ctid >= '( %s, 1)'::tid $q$, cursorname, i);
+ EXECUTE query;
+ EXECUTE 'FETCH NEXT FROM '||cursorname;
+ -- for test development
+ -- RAISE NOTICE '%: %', cursorname, query;
+ END LOOP;
+ END;
+$f$;
+-- Test overflow of temp table buffers is handled correctly
+BEGIN;
+-- should work, below max
+SELECT test_temp_pin(0, 9);
+ test_temp_pin
+---------------
+
+(1 row)
+
+-- should fail, too many buffers pinned
+SELECT test_temp_pin(10, 105);
+ERROR: no empty local buffer available
+ROLLBACK;
+BEGIN;
+-- have some working cursors to test after errors
+SELECT test_temp_pin(0, 9);
+ test_temp_pin
+---------------
+
+(1 row)
+
+FETCH NEXT FROM c_3;
+ ctid
+-------
+ (3,2)
+(1 row)
+
+-- exhaust buffer pins in subtrans, check things work after
+SAVEPOINT rescue_me;
+SELECT test_temp_pin(10, 105);
+ERROR: no empty local buffer available
+ROLLBACK TO SAVEPOINT rescue_me;
+-- pre-subtrans cursors continue to work
+FETCH NEXT FROM c_3;
+ ctid
+-------
+ (3,3)
+(1 row)
+
+-- new cursors with pins can be created after subtrans rollback
+SELECT test_temp_pin(10, 95);
+ test_temp_pin
+---------------
+
+(1 row)
+
+-- Check that read streams deal with lower number of pins available
+SELECT count(*), max(a) max_a, min(a) min_a, max(cnt) max_cnt FROM test_temp;
+ count | max_a | min_a | max_cnt
+-------+-------+-------+---------
+ 10000 | 10000 | 1 | 0
+(1 row)
+
+ROLLBACK;
+-- Check that temp tables with existing cursors can't be dropped.
+BEGIN;
+SELECT test_temp_pin(0, 1);
+ test_temp_pin
+---------------
+
+(1 row)
+
+DROP TABLE test_temp;
+ERROR: cannot DROP TABLE "test_temp" because it is being used by active queries in this session
+COMMIT;
+-- Check that temp tables with existing cursors can't be dropped.
+BEGIN;
+SELECT test_temp_pin(0, 1);
+ test_temp_pin
+---------------
+
+(1 row)
+
+TRUNCATE test_temp;
+ERROR: cannot TRUNCATE "test_temp" because it is being used by active queries in this session
+COMMIT;
+-- Check that temp tables that are dropped in transaction that's rolled back
+-- preserve buffer contents
+SELECT count(*), max(a) max_a, min(a) min_a, max(cnt) max_cnt FROM test_temp;
+ count | max_a | min_a | max_cnt
+-------+-------+-------+---------
+ 10000 | 10000 | 1 | 0
+(1 row)
+
+INSERT INTO test_temp(a, b, cnt) VALUES (-1, '', 0);
+BEGIN;
+INSERT INTO test_temp(a, b, cnt) VALUES (-2, '', 0);
+DROP TABLE test_temp;
+ROLLBACK;
+SELECT count(*), max(a) max_a, min(a) min_a, max(cnt) max_cnt FROM test_temp;
+ count | max_a | min_a | max_cnt
+-------+-------+-------+---------
+ 10001 | 10000 | -1 | 0
+(1 row)
+
+-- Check that temp table drop is transactional and preserves dirty
+-- buffer contents
+UPDATE test_temp SET cnt = cnt + 1 WHERE a = -1;
+BEGIN;
+DROP TABLE test_temp;
+ROLLBACK;
+SELECT count(*), max(a) max_a, min(a) min_a, max(cnt) max_cnt FROM test_temp;
+ count | max_a | min_a | max_cnt
+-------+-------+-------+---------
+ 10001 | 10000 | -1 | 1
+(1 row)
+
+-- Check that temp table truncation is transactional and preserves dirty
+-- buffer contents
+UPDATE test_temp SET cnt = cnt + 1 WHERE a = -1;
+BEGIN;
+TRUNCATE test_temp;
+ROLLBACK;
+SELECT count(*), max(a) max_a, min(a) min_a, max(cnt) max_cnt FROM test_temp;
+ count | max_a | min_a | max_cnt
+-------+-------+-------+---------
+ 10001 | 10000 | -1 | 2
+(1 row)
+
+-- cleanup
+DROP FUNCTION test_temp_pin(int, int);
diff --git a/src/test/regress/parallel_schedule b/src/test/regress/parallel_schedule
index 37b6d21e1f9..0a35f2f8f6a 100644
--- a/src/test/regress/parallel_schedule
+++ b/src/test/regress/parallel_schedule
@@ -108,7 +108,7 @@ test: json jsonb json_encoding jsonpath jsonpath_encoding jsonb_jsonpath sqljson
# ----------
# Another group of parallel tests
# with depends on create_misc
-# NB: temp.sql does a reconnect which transiently uses 2 connections,
+# NB: temp.sql does reconnects which transiently use 2 connections,
# so keep this parallel group to at most 19 tests
# ----------
test: plancache limit plpgsql copy2 temp domain rangefuncs prepare conversion truncate alter_table sequence polymorphism rowtypes returning largeobject with xml
diff --git a/src/test/regress/sql/temp.sql b/src/test/regress/sql/temp.sql
index 2a487a1ef7f..12091f968de 100644
--- a/src/test/regress/sql/temp.sql
+++ b/src/test/regress/sql/temp.sql
@@ -311,3 +311,110 @@ SET search_path TO 'pg_temp';
BEGIN;
SELECT current_schema() ~ 'pg_temp' AS is_temp_schema;
PREPARE TRANSACTION 'twophase_search';
+
+
+-- Tests to verify we recover correctly from exhausting buffer pins and
+-- related matters.
+
+-- use lower possible buffer limit to make the test cheaper
+\c
+SET temp_buffers = 100;
+
+CREATE TEMPORARY TABLE test_temp(a int not null unique, b TEXT not null, cnt int not null);
+INSERT INTO test_temp SELECT generate_series(1, 10000) as id, repeat('a', 200), 0;
+-- should be at least 2x as large than temp_buffers
+SELECT pg_relation_size('test_temp') / current_setting('block_size')::int8 > 200;
+
+-- Don't want cursor names and plpgsql function lines in the error messages
+\set VERBOSITY terse
+
+/* helper function to create cursors for each page in [p_start, p_end] */
+CREATE FUNCTION test_temp_pin(p_start int, p_end int)
+RETURNS void
+LANGUAGE plpgsql
+AS $f$
+ DECLARE
+ cursorname text;
+ query text;
+ BEGIN
+ FOR i IN p_start..p_end LOOP
+ cursorname = 'c_'||i;
+ query = format($q$DECLARE %I CURSOR FOR SELECT ctid FROM test_temp WHERE ctid >= '( %s, 1)'::tid $q$, cursorname, i);
+ EXECUTE query;
+ EXECUTE 'FETCH NEXT FROM '||cursorname;
+ -- for test development
+ -- RAISE NOTICE '%: %', cursorname, query;
+ END LOOP;
+ END;
+$f$;
+
+
+-- Test overflow of temp table buffers is handled correctly
+BEGIN;
+-- should work, below max
+SELECT test_temp_pin(0, 9);
+-- should fail, too many buffers pinned
+SELECT test_temp_pin(10, 105);
+ROLLBACK;
+
+BEGIN;
+-- have some working cursors to test after errors
+SELECT test_temp_pin(0, 9);
+FETCH NEXT FROM c_3;
+-- exhaust buffer pins in subtrans, check things work after
+SAVEPOINT rescue_me;
+SELECT test_temp_pin(10, 105);
+ROLLBACK TO SAVEPOINT rescue_me;
+-- pre-subtrans cursors continue to work
+FETCH NEXT FROM c_3;
+
+-- new cursors with pins can be created after subtrans rollback
+SELECT test_temp_pin(10, 95);
+
+-- Check that read streams deal with lower number of pins available
+SELECT count(*), max(a) max_a, min(a) min_a, max(cnt) max_cnt FROM test_temp;
+
+ROLLBACK;
+
+
+-- Check that temp tables with existing cursors can't be dropped.
+BEGIN;
+SELECT test_temp_pin(0, 1);
+DROP TABLE test_temp;
+COMMIT;
+
+-- Check that temp tables with existing cursors can't be dropped.
+BEGIN;
+SELECT test_temp_pin(0, 1);
+TRUNCATE test_temp;
+COMMIT;
+
+-- Check that temp tables that are dropped in transaction that's rolled back
+-- preserve buffer contents
+SELECT count(*), max(a) max_a, min(a) min_a, max(cnt) max_cnt FROM test_temp;
+INSERT INTO test_temp(a, b, cnt) VALUES (-1, '', 0);
+BEGIN;
+INSERT INTO test_temp(a, b, cnt) VALUES (-2, '', 0);
+DROP TABLE test_temp;
+ROLLBACK;
+SELECT count(*), max(a) max_a, min(a) min_a, max(cnt) max_cnt FROM test_temp;
+
+-- Check that temp table drop is transactional and preserves dirty
+-- buffer contents
+UPDATE test_temp SET cnt = cnt + 1 WHERE a = -1;
+BEGIN;
+DROP TABLE test_temp;
+ROLLBACK;
+SELECT count(*), max(a) max_a, min(a) min_a, max(cnt) max_cnt FROM test_temp;
+
+-- Check that temp table truncation is transactional and preserves dirty
+-- buffer contents
+UPDATE test_temp SET cnt = cnt + 1 WHERE a = -1;
+BEGIN;
+TRUNCATE test_temp;
+ROLLBACK;
+SELECT count(*), max(a) max_a, min(a) min_a, max(cnt) max_cnt FROM test_temp;
+
+
+-- cleanup
+DROP FUNCTION test_temp_pin(int, int);
--
2.48.1.76.g4e746b1a31.dirty
[text/x-diff] v2.7-0016-localbuf-Fix-dangerous-coding-pattern-in-GetLoc.patch (2.9K, ../../4b3f32ug3cayekysqlgspz2qjmeb7lca3gvazayglxr2m3d4dv@il33accgsji7/17-v2.7-0016-localbuf-Fix-dangerous-coding-pattern-in-GetLoc.patch)
download | inline diff:
From 9dd0824cb06e20ccabb04c0e8fb185e63a2bc566 Mon Sep 17 00:00:00 2001
From: Andres Freund <andres@anarazel.de>
Date: Sun, 9 Mar 2025 20:11:51 -0400
Subject: [PATCH v2.7 16/35] localbuf: Fix dangerous coding pattern in
GetLocalVictimBuffer()
If PinLocalBuffer() were to modify the buf_state, the buf_state in
GetLocalVictimBuffer() would be out of date. Currently that does not happen,
as PinLocalBuffer() only modifies the buf_state if adjust_usagecount=true and
GetLocalVictimBuffer() passes false.
However, it's easy to make this not the case anymore - it cost me a few hours
to debug the consequences.
The minimal fix would be to just refetch the buf_state after after calling
PinLocalBuffer(), but the same danger exists in later parts of the
function. Instead, declare buf_state in the narrower scopes and re-read the
state in conditional branches. Besides being safer, it also fits well with
an upcoming set of cleanup patches that move the contents of the conditional
branches in GetLocalVictimBuffer() into helper functions.
I "broke" this in 794f2594479.
Arguably this should be backpatched, but as the relevant functions are not
exported and there is no actual misbehaviour, I chose to not backpatch, at
least for now.
Reviewed-by: Melanie Plageman <melanieplageman@gmail.com>
Discussion: https://postgr.es/m/CAAKRu_b9anbWzEs5AAF9WCvcEVmgz-1AkHSQ-CLLy-p7WHzvFw@mail.gmail.com
---
src/backend/storage/buffer/localbuf.c | 9 +++++----
1 file changed, 5 insertions(+), 4 deletions(-)
diff --git a/src/backend/storage/buffer/localbuf.c b/src/backend/storage/buffer/localbuf.c
index 5378ba84316..d3c869f53f9 100644
--- a/src/backend/storage/buffer/localbuf.c
+++ b/src/backend/storage/buffer/localbuf.c
@@ -178,7 +178,6 @@ GetLocalVictimBuffer(void)
{
int victim_bufid;
int trycounter;
- uint32 buf_state;
BufferDesc *bufHdr;
ResourceOwnerEnlarge(CurrentResourceOwner);
@@ -199,7 +198,7 @@ GetLocalVictimBuffer(void)
if (LocalRefCount[victim_bufid] == 0)
{
- buf_state = pg_atomic_read_u32(&bufHdr->state);
+ uint32 buf_state = pg_atomic_read_u32(&bufHdr->state);
if (BUF_STATE_GET_USAGECOUNT(buf_state) > 0)
{
@@ -233,8 +232,9 @@ GetLocalVictimBuffer(void)
* this buffer is not referenced but it might still be dirty. if that's
* the case, write it out before reusing it!
*/
- if (buf_state & BM_DIRTY)
+ if (pg_atomic_read_u32(&bufHdr->state) & BM_DIRTY)
{
+ uint32 buf_state = pg_atomic_read_u32(&bufHdr->state);
instr_time io_start;
SMgrRelation oreln;
Page localpage = (char *) LocalBufHdrGetBlock(bufHdr);
@@ -267,8 +267,9 @@ GetLocalVictimBuffer(void)
/*
* Remove the victim buffer from the hashtable and mark as invalid.
*/
- if (buf_state & BM_TAG_VALID)
+ if (pg_atomic_read_u32(&bufHdr->state) & BM_TAG_VALID)
{
+ uint32 buf_state = pg_atomic_read_u32(&bufHdr->state);
LocalBufferLookupEnt *hresult;
hresult = (LocalBufferLookupEnt *)
--
2.48.1.76.g4e746b1a31.dirty
[text/x-diff] v2.7-0017-localbuf-Introduce-InvalidateLocalBuffer.patch (6.0K, ../../4b3f32ug3cayekysqlgspz2qjmeb7lca3gvazayglxr2m3d4dv@il33accgsji7/18-v2.7-0017-localbuf-Introduce-InvalidateLocalBuffer.patch)
download | inline diff:
From 9b7cb5b1fe6812e358870eb62c90de936a10c716 Mon Sep 17 00:00:00 2001
From: Andres Freund <andres@anarazel.de>
Date: Sun, 9 Mar 2025 20:25:22 -0400
Subject: [PATCH v2.7 17/35] localbuf: Introduce InvalidateLocalBuffer()
Previously, there were three copies of this code, two of them
identical. There's no good reason for that.
This change is nice on its own, but the main motivation is the AIO patchset,
which needs to add extra checks the deduplicated code, which of course is
easier if there is only one version.
Reviewed-by: Melanie Plageman <melanieplageman@gmail.com>
Discussion: https://postgr.es/m/CAAKRu_b9anbWzEs5AAF9WCvcEVmgz-1AkHSQ-CLLy-p7WHzvFw@mail.gmail.com
---
src/backend/storage/buffer/localbuf.c | 92 +++++++++++++--------------
1 file changed, 44 insertions(+), 48 deletions(-)
diff --git a/src/backend/storage/buffer/localbuf.c b/src/backend/storage/buffer/localbuf.c
index d3c869f53f9..f46ba0c0558 100644
--- a/src/backend/storage/buffer/localbuf.c
+++ b/src/backend/storage/buffer/localbuf.c
@@ -56,6 +56,7 @@ static int NLocalPinnedBuffers = 0;
static void InitLocalBuffers(void);
static Block GetLocalBufferStorage(void);
static Buffer GetLocalVictimBuffer(void);
+static void InvalidateLocalBuffer(BufferDesc *bufHdr, bool check_unreferenced);
/*
@@ -269,17 +270,7 @@ GetLocalVictimBuffer(void)
*/
if (pg_atomic_read_u32(&bufHdr->state) & BM_TAG_VALID)
{
- uint32 buf_state = pg_atomic_read_u32(&bufHdr->state);
- LocalBufferLookupEnt *hresult;
-
- hresult = (LocalBufferLookupEnt *)
- hash_search(LocalBufHash, &bufHdr->tag, HASH_REMOVE, NULL);
- if (!hresult) /* shouldn't happen */
- elog(ERROR, "local buffer hash table corrupted");
- /* mark buffer invalid just in case hash insert fails */
- ClearBufferTag(&bufHdr->tag);
- buf_state &= ~(BUF_FLAG_MASK | BUF_USAGECOUNT_MASK);
- pg_atomic_unlocked_write_u32(&bufHdr->state, buf_state);
+ InvalidateLocalBuffer(bufHdr, false);
pgstat_count_io_op(IOOBJECT_TEMP_RELATION, IOCONTEXT_NORMAL, IOOP_EVICT, 1, 0);
}
@@ -492,6 +483,46 @@ MarkLocalBufferDirty(Buffer buffer)
pg_atomic_unlocked_write_u32(&bufHdr->state, buf_state);
}
+/*
+ * InvalidateLocalBuffer -- mark a local buffer invalid.
+ *
+ * If check_unreferenced is true, error out if the buffer is still
+ * pinned. Passing false is appropriate when calling InvalidateLocalBuffer()
+ * as part of changing the identity of a buffer, instead of just dropping the
+ * buffer.
+ *
+ * See also InvalidateBuffer().
+ */
+static void
+InvalidateLocalBuffer(BufferDesc *bufHdr, bool check_unreferenced)
+{
+ Buffer buffer = BufferDescriptorGetBuffer(bufHdr);
+ int bufid = -buffer - 1;
+ uint32 buf_state;
+ LocalBufferLookupEnt *hresult;
+
+ buf_state = pg_atomic_read_u32(&bufHdr->state);
+
+ if (check_unreferenced && LocalRefCount[bufid] != 0)
+ elog(ERROR, "block %u of %s is still referenced (local %u)",
+ bufHdr->tag.blockNum,
+ relpathbackend(BufTagGetRelFileLocator(&bufHdr->tag),
+ MyProcNumber,
+ BufTagGetForkNum(&bufHdr->tag)).str,
+ LocalRefCount[bufid]);
+
+ /* Remove entry from hashtable */
+ hresult = (LocalBufferLookupEnt *)
+ hash_search(LocalBufHash, &bufHdr->tag, HASH_REMOVE, NULL);
+ if (!hresult) /* shouldn't happen */
+ elog(ERROR, "local buffer hash table corrupted");
+ /* Mark buffer invalid */
+ ClearBufferTag(&bufHdr->tag);
+ buf_state &= ~BUF_FLAG_MASK;
+ buf_state &= ~BUF_USAGECOUNT_MASK;
+ pg_atomic_unlocked_write_u32(&bufHdr->state, buf_state);
+}
+
/*
* DropRelationLocalBuffers
* This function removes from the buffer pool all the pages of the
@@ -512,7 +543,6 @@ DropRelationLocalBuffers(RelFileLocator rlocator, ForkNumber forkNum,
for (i = 0; i < NLocBuffer; i++)
{
BufferDesc *bufHdr = GetLocalBufferDescriptor(i);
- LocalBufferLookupEnt *hresult;
uint32 buf_state;
buf_state = pg_atomic_read_u32(&bufHdr->state);
@@ -522,24 +552,7 @@ DropRelationLocalBuffers(RelFileLocator rlocator, ForkNumber forkNum,
BufTagGetForkNum(&bufHdr->tag) == forkNum &&
bufHdr->tag.blockNum >= firstDelBlock)
{
- if (LocalRefCount[i] != 0)
- elog(ERROR, "block %u of %s is still referenced (local %u)",
- bufHdr->tag.blockNum,
- relpathbackend(BufTagGetRelFileLocator(&bufHdr->tag),
- MyProcNumber,
- BufTagGetForkNum(&bufHdr->tag)).str,
- LocalRefCount[i]);
-
- /* Remove entry from hashtable */
- hresult = (LocalBufferLookupEnt *)
- hash_search(LocalBufHash, &bufHdr->tag, HASH_REMOVE, NULL);
- if (!hresult) /* shouldn't happen */
- elog(ERROR, "local buffer hash table corrupted");
- /* Mark buffer invalid */
- ClearBufferTag(&bufHdr->tag);
- buf_state &= ~BUF_FLAG_MASK;
- buf_state &= ~BUF_USAGECOUNT_MASK;
- pg_atomic_unlocked_write_u32(&bufHdr->state, buf_state);
+ InvalidateLocalBuffer(bufHdr, true);
}
}
}
@@ -559,7 +572,6 @@ DropRelationAllLocalBuffers(RelFileLocator rlocator)
for (i = 0; i < NLocBuffer; i++)
{
BufferDesc *bufHdr = GetLocalBufferDescriptor(i);
- LocalBufferLookupEnt *hresult;
uint32 buf_state;
buf_state = pg_atomic_read_u32(&bufHdr->state);
@@ -567,23 +579,7 @@ DropRelationAllLocalBuffers(RelFileLocator rlocator)
if ((buf_state & BM_TAG_VALID) &&
BufTagMatchesRelFileLocator(&bufHdr->tag, &rlocator))
{
- if (LocalRefCount[i] != 0)
- elog(ERROR, "block %u of %s is still referenced (local %u)",
- bufHdr->tag.blockNum,
- relpathbackend(BufTagGetRelFileLocator(&bufHdr->tag),
- MyProcNumber,
- BufTagGetForkNum(&bufHdr->tag)).str,
- LocalRefCount[i]);
- /* Remove entry from hashtable */
- hresult = (LocalBufferLookupEnt *)
- hash_search(LocalBufHash, &bufHdr->tag, HASH_REMOVE, NULL);
- if (!hresult) /* shouldn't happen */
- elog(ERROR, "local buffer hash table corrupted");
- /* Mark buffer invalid */
- ClearBufferTag(&bufHdr->tag);
- buf_state &= ~BUF_FLAG_MASK;
- buf_state &= ~BUF_USAGECOUNT_MASK;
- pg_atomic_unlocked_write_u32(&bufHdr->state, buf_state);
+ InvalidateLocalBuffer(bufHdr, true);
}
}
}
--
2.48.1.76.g4e746b1a31.dirty
[text/x-diff] v2.7-0018-localbuf-Introduce-TerminateLocalBufferIO.patch (5.8K, ../../4b3f32ug3cayekysqlgspz2qjmeb7lca3gvazayglxr2m3d4dv@il33accgsji7/19-v2.7-0018-localbuf-Introduce-TerminateLocalBufferIO.patch)
download | inline diff:
From e546b1554b346c0c90d28c69ebdfabbc57373755 Mon Sep 17 00:00:00 2001
From: Andres Freund <andres@anarazel.de>
Date: Sun, 9 Mar 2025 19:55:05 -0400
Subject: [PATCH v2.7 18/35] localbuf: Introduce TerminateLocalBufferIO()
Previously TerminateLocalBufferIO() was open-coded in multiple places, which
doesn't seem like a great idea. While TerminateLocalBufferIO() currently is
rather simple, an upcoming patch requires additional code to be added to
TerminateLocalBufferIO(), making this modification particularly worthwhile.
For some reason FlushRelationBuffers() previously cleared BM_JUST_DIRTIED,
even though that's never set for temporary buffers. This is not carried over
as part of this change.
Reviewed-by: Melanie Plageman <melanieplageman@gmail.com>
Discussion: https://postgr.es/m/CAAKRu_b9anbWzEs5AAF9WCvcEVmgz-1AkHSQ-CLLy-p7WHzvFw@mail.gmail.com
---
src/include/storage/buf_internals.h | 2 ++
src/backend/storage/buffer/bufmgr.c | 32 ++++++++-------------------
src/backend/storage/buffer/localbuf.c | 29 +++++++++++++++++++++---
3 files changed, 37 insertions(+), 26 deletions(-)
diff --git a/src/include/storage/buf_internals.h b/src/include/storage/buf_internals.h
index 8b32fb108b0..4611a60d3e0 100644
--- a/src/include/storage/buf_internals.h
+++ b/src/include/storage/buf_internals.h
@@ -471,6 +471,8 @@ extern BlockNumber ExtendBufferedRelLocal(BufferManagerRelation bmr,
Buffer *buffers,
uint32 *extended_by);
extern void MarkLocalBufferDirty(Buffer buffer);
+extern void TerminateLocalBufferIO(BufferDesc *bufHdr, bool clear_dirty,
+ uint32 set_flag_bits);
extern void DropRelationLocalBuffers(RelFileLocator rlocator,
ForkNumber forkNum,
BlockNumber firstDelBlock);
diff --git a/src/backend/storage/buffer/bufmgr.c b/src/backend/storage/buffer/bufmgr.c
index 75a928e802a..ead562249d3 100644
--- a/src/backend/storage/buffer/bufmgr.c
+++ b/src/backend/storage/buffer/bufmgr.c
@@ -1072,19 +1072,11 @@ ZeroAndLockBuffer(Buffer buffer, ReadBufferMode mode, bool already_valid)
if (!isLocalBuf)
LWLockAcquire(BufferDescriptorGetContentLock(bufHdr), LW_EXCLUSIVE);
+ /* Set BM_VALID, terminate IO, and wake up any waiters */
if (isLocalBuf)
- {
- /* Only need to adjust flags */
- uint32 buf_state = pg_atomic_read_u32(&bufHdr->state);
-
- buf_state |= BM_VALID;
- pg_atomic_unlocked_write_u32(&bufHdr->state, buf_state);
- }
+ TerminateLocalBufferIO(bufHdr, false, BM_VALID);
else
- {
- /* Set BM_VALID, terminate IO, and wake up any waiters */
TerminateBufferIO(bufHdr, false, BM_VALID, true);
- }
}
else if (!isLocalBuf)
{
@@ -1608,19 +1600,11 @@ WaitReadBuffers(ReadBuffersOperation *operation)
relpath(operation->smgr->smgr_rlocator, forknum).str)));
}
- /* Terminate I/O and set BM_VALID. */
+ /* Set BM_VALID, terminate IO, and wake up any waiters */
if (persistence == RELPERSISTENCE_TEMP)
- {
- uint32 buf_state = pg_atomic_read_u32(&bufHdr->state);
-
- buf_state |= BM_VALID;
- pg_atomic_unlocked_write_u32(&bufHdr->state, buf_state);
- }
+ TerminateLocalBufferIO(bufHdr, false, BM_VALID);
else
- {
- /* Set BM_VALID, terminate IO, and wake up any waiters */
TerminateBufferIO(bufHdr, false, BM_VALID, true);
- }
/* Report I/Os as completing individually. */
TRACE_POSTGRESQL_BUFFER_READ_DONE(forknum, io_first_block + j,
@@ -4564,8 +4548,7 @@ FlushRelationBuffers(Relation rel)
IOCONTEXT_NORMAL, IOOP_WRITE,
io_start, 1, BLCKSZ);
- buf_state &= ~(BM_DIRTY | BM_JUST_DIRTIED);
- pg_atomic_unlocked_write_u32(&bufHdr->state, buf_state);
+ TerminateLocalBufferIO(bufHdr, true, 0);
pgBufferUsage.local_blks_written++;
@@ -5652,8 +5635,11 @@ TerminateBufferIO(BufferDesc *buf, bool clear_dirty, uint32 set_flag_bits,
buf_state = LockBufHdr(buf);
Assert(buf_state & BM_IO_IN_PROGRESS);
+ buf_state &= ~BM_IO_IN_PROGRESS;
+
+ /* Clear earlier errors, if this IO failed, it'll be marked again */
+ buf_state &= ~BM_IO_ERROR;
- buf_state &= ~(BM_IO_IN_PROGRESS | BM_IO_ERROR);
if (clear_dirty && !(buf_state & BM_JUST_DIRTIED))
buf_state &= ~(BM_DIRTY | BM_CHECKPOINT_NEEDED);
diff --git a/src/backend/storage/buffer/localbuf.c b/src/backend/storage/buffer/localbuf.c
index f46ba0c0558..6df51a3a754 100644
--- a/src/backend/storage/buffer/localbuf.c
+++ b/src/backend/storage/buffer/localbuf.c
@@ -235,7 +235,6 @@ GetLocalVictimBuffer(void)
*/
if (pg_atomic_read_u32(&bufHdr->state) & BM_DIRTY)
{
- uint32 buf_state = pg_atomic_read_u32(&bufHdr->state);
instr_time io_start;
SMgrRelation oreln;
Page localpage = (char *) LocalBufHdrGetBlock(bufHdr);
@@ -259,8 +258,7 @@ GetLocalVictimBuffer(void)
IOOP_WRITE, io_start, 1, BLCKSZ);
/* Mark not-dirty now in case we error out below */
- buf_state &= ~BM_DIRTY;
- pg_atomic_unlocked_write_u32(&bufHdr->state, buf_state);
+ TerminateLocalBufferIO(bufHdr, true, 0);
pgBufferUsage.local_blks_written++;
}
@@ -483,6 +481,31 @@ MarkLocalBufferDirty(Buffer buffer)
pg_atomic_unlocked_write_u32(&bufHdr->state, buf_state);
}
+/*
+ * Like TerminateBufferIO, but for local buffers
+ */
+void
+TerminateLocalBufferIO(BufferDesc *bufHdr, bool clear_dirty, uint32 set_flag_bits)
+{
+ /* Only need to adjust flags */
+ uint32 buf_state = pg_atomic_read_u32(&bufHdr->state);
+
+ /* BM_IO_IN_PROGRESS isn't currently used for local buffers */
+
+ /* Clear earlier errors, if this IO failed, it'll be marked again */
+ buf_state &= ~BM_IO_ERROR;
+
+ if (clear_dirty)
+ buf_state &= ~BM_DIRTY;
+
+ buf_state |= set_flag_bits;
+ pg_atomic_unlocked_write_u32(&bufHdr->state, buf_state);
+
+ /* local buffers don't track IO using resowners */
+
+ /* local buffers don't use the IO CV, as no other process can see buffer */
+}
+
/*
* InvalidateLocalBuffer -- mark a local buffer invalid.
*
--
2.48.1.76.g4e746b1a31.dirty
[text/x-diff] v2.7-0019-localbuf-Introduce-FlushLocalBuffer.patch (5.4K, ../../4b3f32ug3cayekysqlgspz2qjmeb7lca3gvazayglxr2m3d4dv@il33accgsji7/20-v2.7-0019-localbuf-Introduce-FlushLocalBuffer.patch)
download | inline diff:
From 5eafd098098b347144cee7454db4c44c3ca814ff Mon Sep 17 00:00:00 2001
From: Andres Freund <andres@anarazel.de>
Date: Sun, 9 Mar 2025 18:33:35 -0400
Subject: [PATCH v2.7 19/35] localbuf: Introduce FlushLocalBuffer()
Previously we had two paths implementing writing out temporary table
buffers. For shared buffers, the logic for that is centralized in
FlushBuffer(). Introduce FlushLocalBuffer() to do the same for local buffers.
Besides being a nice cleanup on its own, it also makes an upcoming change
slightly easier.
Reviewed-by: Melanie Plageman <melanieplageman@gmail.com>
Discussion: https://postgr.es/m/CAAKRu_b9anbWzEs5AAF9WCvcEVmgz-1AkHSQ-CLLy-p7WHzvFw@mail.gmail.com
---
src/include/storage/buf_internals.h | 1 +
src/backend/storage/buffer/bufmgr.c | 22 +--------
src/backend/storage/buffer/localbuf.c | 64 +++++++++++++++------------
3 files changed, 38 insertions(+), 49 deletions(-)
diff --git a/src/include/storage/buf_internals.h b/src/include/storage/buf_internals.h
index 4611a60d3e0..90bc7e0db7b 100644
--- a/src/include/storage/buf_internals.h
+++ b/src/include/storage/buf_internals.h
@@ -473,6 +473,7 @@ extern BlockNumber ExtendBufferedRelLocal(BufferManagerRelation bmr,
extern void MarkLocalBufferDirty(Buffer buffer);
extern void TerminateLocalBufferIO(BufferDesc *bufHdr, bool clear_dirty,
uint32 set_flag_bits);
+extern void FlushLocalBuffer(BufferDesc *bufHdr, SMgrRelation reln);
extern void DropRelationLocalBuffers(RelFileLocator rlocator,
ForkNumber forkNum,
BlockNumber firstDelBlock);
diff --git a/src/backend/storage/buffer/bufmgr.c b/src/backend/storage/buffer/bufmgr.c
index ead562249d3..5ead69b5e16 100644
--- a/src/backend/storage/buffer/bufmgr.c
+++ b/src/backend/storage/buffer/bufmgr.c
@@ -4516,7 +4516,6 @@ FlushRelationBuffers(Relation rel)
for (i = 0; i < NLocBuffer; i++)
{
uint32 buf_state;
- instr_time io_start;
bufHdr = GetLocalBufferDescriptor(i);
if (BufTagMatchesRelFileLocator(&bufHdr->tag, &rel->rd_locator) &&
@@ -4524,9 +4523,6 @@ FlushRelationBuffers(Relation rel)
(BM_VALID | BM_DIRTY)) == (BM_VALID | BM_DIRTY))
{
ErrorContextCallback errcallback;
- Page localpage;
-
- localpage = (char *) LocalBufHdrGetBlock(bufHdr);
/* Setup error traceback support for ereport() */
errcallback.callback = local_buffer_write_error_callback;
@@ -4534,23 +4530,7 @@ FlushRelationBuffers(Relation rel)
errcallback.previous = error_context_stack;
error_context_stack = &errcallback;
- PageSetChecksumInplace(localpage, bufHdr->tag.blockNum);
-
- io_start = pgstat_prepare_io_time(track_io_timing);
-
- smgrwrite(srel,
- BufTagGetForkNum(&bufHdr->tag),
- bufHdr->tag.blockNum,
- localpage,
- false);
-
- pgstat_count_io_op_time(IOOBJECT_TEMP_RELATION,
- IOCONTEXT_NORMAL, IOOP_WRITE,
- io_start, 1, BLCKSZ);
-
- TerminateLocalBufferIO(bufHdr, true, 0);
-
- pgBufferUsage.local_blks_written++;
+ FlushLocalBuffer(bufHdr, srel);
/* Pop the error context stack */
error_context_stack = errcallback.previous;
diff --git a/src/backend/storage/buffer/localbuf.c b/src/backend/storage/buffer/localbuf.c
index 6df51a3a754..251289c95fd 100644
--- a/src/backend/storage/buffer/localbuf.c
+++ b/src/backend/storage/buffer/localbuf.c
@@ -174,6 +174,41 @@ LocalBufferAlloc(SMgrRelation smgr, ForkNumber forkNum, BlockNumber blockNum,
return bufHdr;
}
+/*
+ * Like FlushBuffer(), just for local buffers.
+ */
+void
+FlushLocalBuffer(BufferDesc *bufHdr, SMgrRelation reln)
+{
+ instr_time io_start;
+ Page localpage = (char *) LocalBufHdrGetBlock(bufHdr);
+
+ /* Find smgr relation for buffer */
+ if (reln == NULL)
+ reln = smgropen(BufTagGetRelFileLocator(&bufHdr->tag),
+ MyProcNumber);
+
+ PageSetChecksumInplace(localpage, bufHdr->tag.blockNum);
+
+ io_start = pgstat_prepare_io_time(track_io_timing);
+
+ /* And write... */
+ smgrwrite(reln,
+ BufTagGetForkNum(&bufHdr->tag),
+ bufHdr->tag.blockNum,
+ localpage,
+ false);
+
+ /* Temporary table I/O does not use Buffer Access Strategies */
+ pgstat_count_io_op_time(IOOBJECT_TEMP_RELATION, IOCONTEXT_NORMAL,
+ IOOP_WRITE, io_start, 1, BLCKSZ);
+
+ /* Mark not-dirty */
+ TerminateLocalBufferIO(bufHdr, true, 0);
+
+ pgBufferUsage.local_blks_written++;
+}
+
static Buffer
GetLocalVictimBuffer(void)
{
@@ -234,34 +269,7 @@ GetLocalVictimBuffer(void)
* the case, write it out before reusing it!
*/
if (pg_atomic_read_u32(&bufHdr->state) & BM_DIRTY)
- {
- instr_time io_start;
- SMgrRelation oreln;
- Page localpage = (char *) LocalBufHdrGetBlock(bufHdr);
-
- /* Find smgr relation for buffer */
- oreln = smgropen(BufTagGetRelFileLocator(&bufHdr->tag), MyProcNumber);
-
- PageSetChecksumInplace(localpage, bufHdr->tag.blockNum);
-
- io_start = pgstat_prepare_io_time(track_io_timing);
-
- /* And write... */
- smgrwrite(oreln,
- BufTagGetForkNum(&bufHdr->tag),
- bufHdr->tag.blockNum,
- localpage,
- false);
-
- /* Temporary table I/O does not use Buffer Access Strategies */
- pgstat_count_io_op_time(IOOBJECT_TEMP_RELATION, IOCONTEXT_NORMAL,
- IOOP_WRITE, io_start, 1, BLCKSZ);
-
- /* Mark not-dirty now in case we error out below */
- TerminateLocalBufferIO(bufHdr, true, 0);
-
- pgBufferUsage.local_blks_written++;
- }
+ FlushLocalBuffer(bufHdr, NULL);
/*
* Remove the victim buffer from the hashtable and mark as invalid.
--
2.48.1.76.g4e746b1a31.dirty
[text/x-diff] v2.7-0020-localbuf-Introduce-StartLocalBufferIO.patch (4.9K, ../../4b3f32ug3cayekysqlgspz2qjmeb7lca3gvazayglxr2m3d4dv@il33accgsji7/21-v2.7-0020-localbuf-Introduce-StartLocalBufferIO.patch)
download | inline diff:
From 11489de41c73c8089d2f1ba4f08da04697feabcd Mon Sep 17 00:00:00 2001
From: Andres Freund <andres@anarazel.de>
Date: Sun, 9 Mar 2025 18:44:36 -0400
Subject: [PATCH v2.7 20/35] localbuf: Introduce StartLocalBufferIO()
To initiate IO on a shared buffer we have StartBufferIO(). For temporary table
buffers no similar function exists - likely because the code for that
currently is very simple due to the lack of concurrency.
However, the upcoming AIO support will make it possible to re-encounter a
local buffer, while the buffer already is the target of IO. In that case we
need to wait for already in-progress IO to complete. This commit makes it
easier to add the necessary code, by introducing StartLocalBufferIO().
Reviewed-by: Melanie Plageman <melanieplageman@gmail.com>
Discussion: https://postgr.es/m/CAAKRu_b9anbWzEs5AAF9WCvcEVmgz-1AkHSQ-CLLy-p7WHzvFw@mail.gmail.com
---
src/include/storage/buf_internals.h | 1 +
src/backend/storage/buffer/bufmgr.c | 8 ++----
src/backend/storage/buffer/localbuf.c | 36 +++++++++++++++++++++++++++
3 files changed, 39 insertions(+), 6 deletions(-)
diff --git a/src/include/storage/buf_internals.h b/src/include/storage/buf_internals.h
index 90bc7e0db7b..9327f60c44c 100644
--- a/src/include/storage/buf_internals.h
+++ b/src/include/storage/buf_internals.h
@@ -473,6 +473,7 @@ extern BlockNumber ExtendBufferedRelLocal(BufferManagerRelation bmr,
extern void MarkLocalBufferDirty(Buffer buffer);
extern void TerminateLocalBufferIO(BufferDesc *bufHdr, bool clear_dirty,
uint32 set_flag_bits);
+extern bool StartLocalBufferIO(BufferDesc *bufHdr, bool forInput);
extern void FlushLocalBuffer(BufferDesc *bufHdr, SMgrRelation reln);
extern void DropRelationLocalBuffers(RelFileLocator rlocator,
ForkNumber forkNum,
diff --git a/src/backend/storage/buffer/bufmgr.c b/src/backend/storage/buffer/bufmgr.c
index 5ead69b5e16..07254d7015a 100644
--- a/src/backend/storage/buffer/bufmgr.c
+++ b/src/backend/storage/buffer/bufmgr.c
@@ -1038,7 +1038,7 @@ ZeroAndLockBuffer(Buffer buffer, ReadBufferMode mode, bool already_valid)
{
/* Simple case for non-shared buffers. */
bufHdr = GetLocalBufferDescriptor(-buffer - 1);
- need_to_zero = (pg_atomic_read_u32(&bufHdr->state) & BM_VALID) == 0;
+ need_to_zero = StartLocalBufferIO(bufHdr, true);
}
else
{
@@ -1450,11 +1450,7 @@ static inline bool
WaitReadBuffersCanStartIO(Buffer buffer, bool nowait)
{
if (BufferIsLocal(buffer))
- {
- BufferDesc *bufHdr = GetLocalBufferDescriptor(-buffer - 1);
-
- return (pg_atomic_read_u32(&bufHdr->state) & BM_VALID) == 0;
- }
+ return StartLocalBufferIO(GetLocalBufferDescriptor(-buffer - 1), true);
else
return StartBufferIO(GetBufferDescriptor(buffer - 1), true, nowait);
}
diff --git a/src/backend/storage/buffer/localbuf.c b/src/backend/storage/buffer/localbuf.c
index 251289c95fd..31d29434245 100644
--- a/src/backend/storage/buffer/localbuf.c
+++ b/src/backend/storage/buffer/localbuf.c
@@ -183,6 +183,13 @@ FlushLocalBuffer(BufferDesc *bufHdr, SMgrRelation reln)
instr_time io_start;
Page localpage = (char *) LocalBufHdrGetBlock(bufHdr);
+ /*
+ * Try to start an I/O operation. There currently are no reasons for
+ * StartLocalBufferIO to return false, so we raise an error in that case.
+ */
+ if (!StartLocalBufferIO(bufHdr, false))
+ elog(ERROR, "failed to start write IO on local buffer");
+
/* Find smgr relation for buffer */
if (reln == NULL)
reln = smgropen(BufTagGetRelFileLocator(&bufHdr->tag),
@@ -406,11 +413,17 @@ ExtendBufferedRelLocal(BufferManagerRelation bmr,
PinLocalBuffer(existing_hdr, false);
buffers[i] = BufferDescriptorGetBuffer(existing_hdr);
+ /*
+ * Clear the BM_VALID bit, do StartLocalBufferIO() and proceed.
+ */
buf_state = pg_atomic_read_u32(&existing_hdr->state);
Assert(buf_state & BM_TAG_VALID);
Assert(!(buf_state & BM_DIRTY));
buf_state &= ~BM_VALID;
pg_atomic_unlocked_write_u32(&existing_hdr->state, buf_state);
+
+ /* no need to loop for local buffers */
+ StartLocalBufferIO(existing_hdr, true);
}
else
{
@@ -425,6 +438,8 @@ ExtendBufferedRelLocal(BufferManagerRelation bmr,
pg_atomic_unlocked_write_u32(&victim_buf_hdr->state, buf_state);
hresult->id = victim_buf_id;
+
+ StartLocalBufferIO(victim_buf_hdr, true);
}
}
@@ -489,6 +504,27 @@ MarkLocalBufferDirty(Buffer buffer)
pg_atomic_unlocked_write_u32(&bufHdr->state, buf_state);
}
+/*
+ * Like StartBufferIO, but for local buffers
+ */
+bool
+StartLocalBufferIO(BufferDesc *bufHdr, bool forInput)
+{
+ uint32 buf_state = pg_atomic_read_u32(&bufHdr->state);
+
+ if (forInput ? (buf_state & BM_VALID) : !(buf_state & BM_DIRTY))
+ {
+ /* someone else already did the I/O */
+ return false;
+ }
+
+ /* BM_IO_IN_PROGRESS isn't currently used for local buffers */
+
+ /* local buffers don't track IO using resowners */
+
+ return true;
+}
+
/*
* Like TerminateBufferIO, but for local buffers
*/
--
2.48.1.76.g4e746b1a31.dirty
[text/x-diff] v2.7-0021-localbuf-Track-pincount-in-BufferDesc-as-well.patch (4.5K, ../../4b3f32ug3cayekysqlgspz2qjmeb7lca3gvazayglxr2m3d4dv@il33accgsji7/22-v2.7-0021-localbuf-Track-pincount-in-BufferDesc-as-well.patch)
download | inline diff:
From e61984a7c2a018adaf543d64181499715d92b175 Mon Sep 17 00:00:00 2001
From: Andres Freund <andres@anarazel.de>
Date: Thu, 20 Feb 2025 18:04:48 -0500
Subject: [PATCH v2.7 21/35] localbuf: Track pincount in BufferDesc as well
For AIO on temporary table buffers the AIO subsystem needs to be able to
ensure a pin on a buffer while AIO is going on, even if the IO issuing query
errors out. Tracking the buffer in LocalRefCount does not work, as it would
cause CheckForLocalBufferLeaks() to assert out.
Instead, also track the refcount in BufferDesc.state, not just
LocalRefCount. This also makes local buffers behave a bit more akin to shared
buffers.
Note that we still don't need locking, AIO completion callbacks for local
buffers are executed in the issuing session (i.e. nobody else has access to
the BufferDesc).
Discussion: https://postgr.es/m/uvrtrknj4kdytuboidbhwclo4gxhswwcpgadptsjvjqcluzmah%40brqs62irg4dt
---
src/backend/storage/buffer/bufmgr.c | 22 +++++++++++++++++++++
src/backend/storage/buffer/localbuf.c | 28 +++++++++++++++++++++++++--
2 files changed, 48 insertions(+), 2 deletions(-)
diff --git a/src/backend/storage/buffer/bufmgr.c b/src/backend/storage/buffer/bufmgr.c
index 07254d7015a..dadc1607742 100644
--- a/src/backend/storage/buffer/bufmgr.c
+++ b/src/backend/storage/buffer/bufmgr.c
@@ -5402,6 +5402,18 @@ ConditionalLockBufferForCleanup(Buffer buffer)
Assert(refcount > 0);
if (refcount != 1)
return false;
+
+ /*
+ * Check that the AIO subsystem doesn't have a pin. Likely not
+ * possible today, but better safe than sorry.
+ */
+ bufHdr = GetLocalBufferDescriptor(-buffer - 1);
+ buf_state = pg_atomic_read_u32(&bufHdr->state);
+ refcount = BUF_STATE_GET_REFCOUNT(buf_state);
+ Assert(refcount > 0);
+ if (refcount != 1)
+ return false;
+
/* Nobody else to wait for */
return true;
}
@@ -5455,6 +5467,16 @@ IsBufferCleanupOK(Buffer buffer)
/* There should be exactly one pin */
if (LocalRefCount[-buffer - 1] != 1)
return false;
+
+ /*
+ * Check that the AIO subsystem doesn't have a pin. Likely not
+ * possible today, but better safe than sorry.
+ */
+ bufHdr = GetLocalBufferDescriptor(-buffer - 1);
+ buf_state = pg_atomic_read_u32(&bufHdr->state);
+ if (BUF_STATE_GET_REFCOUNT(buf_state) != 1)
+ return false;
+
/* Nobody else to wait for */
return true;
}
diff --git a/src/backend/storage/buffer/localbuf.c b/src/backend/storage/buffer/localbuf.c
index 31d29434245..02852d92b3e 100644
--- a/src/backend/storage/buffer/localbuf.c
+++ b/src/backend/storage/buffer/localbuf.c
@@ -249,6 +249,13 @@ GetLocalVictimBuffer(void)
pg_atomic_unlocked_write_u32(&bufHdr->state, buf_state);
trycounter = NLocBuffer;
}
+ else if (BUF_STATE_GET_REFCOUNT(buf_state) > 0)
+ {
+ /*
+ * This can be reached if the backend initiated AIO for this
+ * buffer and then errored out.
+ */
+ }
else
{
/* Found a usable buffer */
@@ -570,7 +577,13 @@ InvalidateLocalBuffer(BufferDesc *bufHdr, bool check_unreferenced)
buf_state = pg_atomic_read_u32(&bufHdr->state);
- if (check_unreferenced && LocalRefCount[bufid] != 0)
+ /*
+ * We need to test not just LocalRefCount[bufid] but also the BufferDesc
+ * itself, as the latter is used to represent a pin by the AIO subsystem.
+ * This can happen if AIO is initiated and then the query errors out.
+ */
+ if (check_unreferenced &&
+ (LocalRefCount[bufid] != 0 || BUF_STATE_GET_REFCOUNT(buf_state) != 0))
elog(ERROR, "block %u of %s is still referenced (local %u)",
bufHdr->tag.blockNum,
relpathbackend(BufTagGetRelFileLocator(&bufHdr->tag),
@@ -744,12 +757,13 @@ PinLocalBuffer(BufferDesc *buf_hdr, bool adjust_usagecount)
if (LocalRefCount[bufid] == 0)
{
NLocalPinnedBuffers++;
+ buf_state += BUF_REFCOUNT_ONE;
if (adjust_usagecount &&
BUF_STATE_GET_USAGECOUNT(buf_state) < BM_MAX_USAGE_COUNT)
{
buf_state += BUF_USAGECOUNT_ONE;
- pg_atomic_unlocked_write_u32(&buf_hdr->state, buf_state);
}
+ pg_atomic_unlocked_write_u32(&buf_hdr->state, buf_state);
}
LocalRefCount[bufid]++;
ResourceOwnerRememberBuffer(CurrentResourceOwner,
@@ -775,7 +789,17 @@ UnpinLocalBufferNoOwner(Buffer buffer)
Assert(NLocalPinnedBuffers > 0);
if (--LocalRefCount[buffid] == 0)
+ {
+ BufferDesc *buf_hdr = GetLocalBufferDescriptor(buffid);
+ uint32 buf_state;
+
NLocalPinnedBuffers--;
+
+ buf_state = pg_atomic_read_u32(&buf_hdr->state);
+ Assert(BUF_STATE_GET_REFCOUNT(buf_state) > 0);
+ buf_state -= BUF_REFCOUNT_ONE;
+ pg_atomic_unlocked_write_u32(&buf_hdr->state, buf_state);
+ }
}
/*
--
2.48.1.76.g4e746b1a31.dirty
[text/x-diff] v2.7-0022-bufmgr-Implement-AIO-read-support.patch (25.4K, ../../4b3f32ug3cayekysqlgspz2qjmeb7lca3gvazayglxr2m3d4dv@il33accgsji7/23-v2.7-0022-bufmgr-Implement-AIO-read-support.patch)
download | inline diff:
From 256db69d5f403796bdd8b95609a7ab9a187ad739 Mon Sep 17 00:00:00 2001
From: Andres Freund <andres@anarazel.de>
Date: Tue, 11 Mar 2025 13:51:06 -0400
Subject: [PATCH v2.7 22/35] bufmgr: Implement AIO read support
This implements the following:
- Addition of callbacks to maintain buffer state on completion of a readv
- Addition of a wait reference to BufferDesc, to allow backends to wait for
IOs
- StartBufferIO(), WaitIO(), TerminateBufferIO() support for waiting AIO
As of this commit nothing issues AIO on shared buffers. A future commit will
update StartReadBuffers() to do so.
Todo:
- deduplicate shared/local buffer completion callbacks
- deduplicate LockBufferForCleanup() support
- function naming pattern (buffer_readv_complete_common() calls
LocalBufferCompleteRead/SharedBufferCompleteRead, which looks ugly)
Discussion: https://postgr.es/m/uvrtrknj4kdytuboidbhwclo4gxhswwcpgadptsjvjqcluzmah%40brqs62irg4dt
Discussion: https://postgr.es/m/20210223100344.llw5an2aklengrmn@alap3.anarazel.de
Discussion: https://postgr.es/m/stj36ea6yyhoxtqkhpieia2z4krnam7qyetc57rfezgk4zgapf@gcnactj4z56m
---
src/include/storage/aio.h | 4 +
src/include/storage/buf_internals.h | 7 +-
src/include/storage/bufmgr.h | 4 +
src/backend/storage/aio/aio_callback.c | 5 +
src/backend/storage/buffer/buf_init.c | 3 +
src/backend/storage/buffer/bufmgr.c | 440 +++++++++++++++++++++++--
src/backend/storage/buffer/localbuf.c | 45 ++-
7 files changed, 466 insertions(+), 42 deletions(-)
diff --git a/src/include/storage/aio.h b/src/include/storage/aio.h
index d02c3d27fae..a4d9b51acfe 100644
--- a/src/include/storage/aio.h
+++ b/src/include/storage/aio.h
@@ -184,6 +184,10 @@ typedef enum PgAioHandleCallbackID
PGAIO_HCB_MD_READV,
PGAIO_HCB_MD_WRITEV,
+
+ PGAIO_HCB_SHARED_BUFFER_READV,
+
+ PGAIO_HCB_LOCAL_BUFFER_READV,
} PgAioHandleCallbackID;
diff --git a/src/include/storage/buf_internals.h b/src/include/storage/buf_internals.h
index 9327f60c44c..8a52ace37cd 100644
--- a/src/include/storage/buf_internals.h
+++ b/src/include/storage/buf_internals.h
@@ -17,6 +17,7 @@
#include "pgstat.h"
#include "port/atomics.h"
+#include "storage/aio_types.h"
#include "storage/buf.h"
#include "storage/bufmgr.h"
#include "storage/condition_variable.h"
@@ -264,6 +265,8 @@ typedef struct BufferDesc
int wait_backend_pgprocno; /* backend of pin-count waiter */
int freeNext; /* link in freelist chain */
+
+ PgAioWaitRef io_wref; /* set iff AIO is in progress */
LWLock content_lock; /* to lock access to buffer contents */
} BufferDesc;
@@ -472,8 +475,8 @@ extern BlockNumber ExtendBufferedRelLocal(BufferManagerRelation bmr,
uint32 *extended_by);
extern void MarkLocalBufferDirty(Buffer buffer);
extern void TerminateLocalBufferIO(BufferDesc *bufHdr, bool clear_dirty,
- uint32 set_flag_bits);
-extern bool StartLocalBufferIO(BufferDesc *bufHdr, bool forInput);
+ uint32 set_flag_bits, bool syncio);
+extern bool StartLocalBufferIO(BufferDesc *bufHdr, bool forInput, bool nowait);
extern void FlushLocalBuffer(BufferDesc *bufHdr, SMgrRelation reln);
extern void DropRelationLocalBuffers(RelFileLocator rlocator,
ForkNumber forkNum,
diff --git a/src/include/storage/bufmgr.h b/src/include/storage/bufmgr.h
index 307f36af384..ffb55cfa08a 100644
--- a/src/include/storage/bufmgr.h
+++ b/src/include/storage/bufmgr.h
@@ -15,6 +15,7 @@
#define BUFMGR_H
#include "port/pg_iovec.h"
+#include "storage/aio_types.h"
#include "storage/block.h"
#include "storage/buf.h"
#include "storage/bufpage.h"
@@ -168,6 +169,9 @@ extern PGDLLIMPORT int checkpoint_flush_after;
extern PGDLLIMPORT int backend_flush_after;
extern PGDLLIMPORT int bgwriter_flush_after;
+extern const PgAioHandleCallbacks aio_shared_buffer_readv_cb;
+extern const PgAioHandleCallbacks aio_local_buffer_readv_cb;
+
/* in buf_init.c */
extern PGDLLIMPORT char *BufferBlocks;
diff --git a/src/backend/storage/aio/aio_callback.c b/src/backend/storage/aio/aio_callback.c
index 6a21c82396a..f76f74ba166 100644
--- a/src/backend/storage/aio/aio_callback.c
+++ b/src/backend/storage/aio/aio_callback.c
@@ -18,6 +18,7 @@
#include "miscadmin.h"
#include "storage/aio.h"
#include "storage/aio_internal.h"
+#include "storage/bufmgr.h"
#include "storage/md.h"
@@ -40,6 +41,10 @@ static const PgAioHandleCallbacksEntry aio_handle_cbs[] = {
CALLBACK_ENTRY(PGAIO_HCB_INVALID, aio_invalid_cb),
CALLBACK_ENTRY(PGAIO_HCB_MD_READV, aio_md_readv_cb),
+
+ CALLBACK_ENTRY(PGAIO_HCB_SHARED_BUFFER_READV, aio_shared_buffer_readv_cb),
+
+ CALLBACK_ENTRY(PGAIO_HCB_LOCAL_BUFFER_READV, aio_local_buffer_readv_cb),
#undef CALLBACK_ENTRY
};
diff --git a/src/backend/storage/buffer/buf_init.c b/src/backend/storage/buffer/buf_init.c
index ed1f8e03190..ed1dc488a42 100644
--- a/src/backend/storage/buffer/buf_init.c
+++ b/src/backend/storage/buffer/buf_init.c
@@ -14,6 +14,7 @@
*/
#include "postgres.h"
+#include "storage/aio.h"
#include "storage/buf_internals.h"
#include "storage/bufmgr.h"
@@ -125,6 +126,8 @@ BufferManagerShmemInit(void)
buf->buf_id = i;
+ pgaio_wref_clear(&buf->io_wref);
+
/*
* Initially link all the buffers together as unused. Subsequent
* management of this list is done by freelist.c.
diff --git a/src/backend/storage/buffer/bufmgr.c b/src/backend/storage/buffer/bufmgr.c
index dadc1607742..2bd1c4f371e 100644
--- a/src/backend/storage/buffer/bufmgr.c
+++ b/src/backend/storage/buffer/bufmgr.c
@@ -48,6 +48,7 @@
#include "pg_trace.h"
#include "pgstat.h"
#include "postmaster/bgwriter.h"
+#include "storage/aio.h"
#include "storage/buf_internals.h"
#include "storage/bufmgr.h"
#include "storage/fd.h"
@@ -516,7 +517,8 @@ static int SyncOneBuffer(int buf_id, bool skip_recently_used,
static void WaitIO(BufferDesc *buf);
static bool StartBufferIO(BufferDesc *buf, bool forInput, bool nowait);
static void TerminateBufferIO(BufferDesc *buf, bool clear_dirty,
- uint32 set_flag_bits, bool forget_owner);
+ uint32 set_flag_bits, bool forget_owner,
+ bool syncio);
static void AbortBufferIO(Buffer buffer);
static void shared_buffer_write_error_callback(void *arg);
static void local_buffer_write_error_callback(void *arg);
@@ -1038,7 +1040,7 @@ ZeroAndLockBuffer(Buffer buffer, ReadBufferMode mode, bool already_valid)
{
/* Simple case for non-shared buffers. */
bufHdr = GetLocalBufferDescriptor(-buffer - 1);
- need_to_zero = StartLocalBufferIO(bufHdr, true);
+ need_to_zero = StartLocalBufferIO(bufHdr, true, false);
}
else
{
@@ -1074,9 +1076,9 @@ ZeroAndLockBuffer(Buffer buffer, ReadBufferMode mode, bool already_valid)
/* Set BM_VALID, terminate IO, and wake up any waiters */
if (isLocalBuf)
- TerminateLocalBufferIO(bufHdr, false, BM_VALID);
+ TerminateLocalBufferIO(bufHdr, false, BM_VALID, true);
else
- TerminateBufferIO(bufHdr, false, BM_VALID, true);
+ TerminateBufferIO(bufHdr, false, BM_VALID, true, true);
}
else if (!isLocalBuf)
{
@@ -1450,7 +1452,8 @@ static inline bool
WaitReadBuffersCanStartIO(Buffer buffer, bool nowait)
{
if (BufferIsLocal(buffer))
- return StartLocalBufferIO(GetLocalBufferDescriptor(-buffer - 1), true);
+ return StartLocalBufferIO(GetLocalBufferDescriptor(-buffer - 1),
+ true, nowait);
else
return StartBufferIO(GetBufferDescriptor(buffer - 1), true, nowait);
}
@@ -1598,9 +1601,9 @@ WaitReadBuffers(ReadBuffersOperation *operation)
/* Set BM_VALID, terminate IO, and wake up any waiters */
if (persistence == RELPERSISTENCE_TEMP)
- TerminateLocalBufferIO(bufHdr, false, BM_VALID);
+ TerminateLocalBufferIO(bufHdr, false, BM_VALID, true);
else
- TerminateBufferIO(bufHdr, false, BM_VALID, true);
+ TerminateBufferIO(bufHdr, false, BM_VALID, true, true);
/* Report I/Os as completing individually. */
TRACE_POSTGRESQL_BUFFER_READ_DONE(forknum, io_first_block + j,
@@ -2511,7 +2514,7 @@ ExtendBufferedRelShared(BufferManagerRelation bmr,
if (lock)
LWLockAcquire(BufferDescriptorGetContentLock(buf_hdr), LW_EXCLUSIVE);
- TerminateBufferIO(buf_hdr, false, BM_VALID, true);
+ TerminateBufferIO(buf_hdr, false, BM_VALID, true, true);
}
pgBufferUsage.shared_blks_written += extend_by;
@@ -2857,6 +2860,44 @@ PinBuffer_Locked(BufferDesc *buf)
ResourceOwnerRememberBuffer(CurrentResourceOwner, b);
}
+/*
+ * Support for waking up another backend that is waiting for the cleanup lock
+ * to be released using BM_PIN_COUNT_WAITER.
+ *
+ * See LockBufferForCleanup().
+ *
+ * Expected to be called just after releasing a buffer pin (in a BufferDesc,
+ * not just reducing the backend-local pincount for the buffer).
+ */
+static void
+WakePinCountWaiter(BufferDesc *buf)
+{
+ /*
+ * Acquire the buffer header lock, re-check that there's a waiter. Another
+ * backend could have unpinned this buffer, and already woken up the
+ * waiter.
+ *
+ * There's no danger of the buffer being replaced after we unpinned it
+ * above, as it's pinned by the waiter. The waiter removes
+ * BM_PIN_COUNT_WAITER if it stops waiting for a reason other than this
+ * backend waking it up.
+ */
+ uint32 buf_state = LockBufHdr(buf);
+
+ if ((buf_state & BM_PIN_COUNT_WAITER) &&
+ BUF_STATE_GET_REFCOUNT(buf_state) == 1)
+ {
+ /* we just released the last pin other than the waiter's */
+ int wait_backend_pgprocno = buf->wait_backend_pgprocno;
+
+ buf_state &= ~BM_PIN_COUNT_WAITER;
+ UnlockBufHdr(buf, buf_state);
+ ProcSendSignal(wait_backend_pgprocno);
+ }
+ else
+ UnlockBufHdr(buf, buf_state);
+}
+
/*
* UnpinBuffer -- make buffer available for replacement.
*
@@ -2925,29 +2966,8 @@ UnpinBufferNoOwner(BufferDesc *buf)
/* Support LockBufferForCleanup() */
if (buf_state & BM_PIN_COUNT_WAITER)
- {
- /*
- * Acquire the buffer header lock, re-check that there's a waiter.
- * Another backend could have unpinned this buffer, and already
- * woken up the waiter. There's no danger of the buffer being
- * replaced after we unpinned it above, as it's pinned by the
- * waiter.
- */
- buf_state = LockBufHdr(buf);
+ WakePinCountWaiter(buf);
- if ((buf_state & BM_PIN_COUNT_WAITER) &&
- BUF_STATE_GET_REFCOUNT(buf_state) == 1)
- {
- /* we just released the last pin other than the waiter's */
- int wait_backend_pgprocno = buf->wait_backend_pgprocno;
-
- buf_state &= ~BM_PIN_COUNT_WAITER;
- UnlockBufHdr(buf, buf_state);
- ProcSendSignal(wait_backend_pgprocno);
- }
- else
- UnlockBufHdr(buf, buf_state);
- }
ForgetPrivateRefCountEntry(ref);
}
}
@@ -3968,7 +3988,7 @@ FlushBuffer(BufferDesc *buf, SMgrRelation reln, IOObject io_object,
* Mark the buffer as clean (unless BM_JUST_DIRTIED has become set) and
* end the BM_IO_IN_PROGRESS state.
*/
- TerminateBufferIO(buf, true, 0, true);
+ TerminateBufferIO(buf, true, 0, true, true);
TRACE_POSTGRESQL_BUFFER_FLUSH_DONE(BufTagGetForkNum(&buf->tag),
buf->tag.blockNum,
@@ -5527,6 +5547,7 @@ WaitIO(BufferDesc *buf)
for (;;)
{
uint32 buf_state;
+ PgAioWaitRef iow;
/*
* It may not be necessary to acquire the spinlock to check the flag
@@ -5534,10 +5555,19 @@ WaitIO(BufferDesc *buf)
* play it safe.
*/
buf_state = LockBufHdr(buf);
+ iow = buf->io_wref;
UnlockBufHdr(buf, buf_state);
if (!(buf_state & BM_IO_IN_PROGRESS))
break;
+
+ if (pgaio_wref_valid(&iow))
+ {
+ pgaio_wref_wait(&iow);
+ ConditionVariablePrepareToSleep(cv);
+ continue;
+ }
+
ConditionVariableSleep(cv, WAIT_EVENT_BUFFER_IO);
}
ConditionVariableCancelSleep();
@@ -5626,7 +5656,7 @@ StartBufferIO(BufferDesc *buf, bool forInput, bool nowait)
*/
static void
TerminateBufferIO(BufferDesc *buf, bool clear_dirty, uint32 set_flag_bits,
- bool forget_owner)
+ bool forget_owner, bool syncio)
{
uint32 buf_state;
@@ -5641,6 +5671,14 @@ TerminateBufferIO(BufferDesc *buf, bool clear_dirty, uint32 set_flag_bits,
if (clear_dirty && !(buf_state & BM_JUST_DIRTIED))
buf_state &= ~(BM_DIRTY | BM_CHECKPOINT_NEEDED);
+ if (!syncio)
+ {
+ /* release ownership by the AIO subsystem */
+ Assert(BUF_STATE_GET_REFCOUNT(buf_state) > 0);
+ buf_state -= BUF_REFCOUNT_ONE;
+ pgaio_wref_clear(&buf->io_wref);
+ }
+
buf_state |= set_flag_bits;
UnlockBufHdr(buf, buf_state);
@@ -5649,6 +5687,17 @@ TerminateBufferIO(BufferDesc *buf, bool clear_dirty, uint32 set_flag_bits,
BufferDescriptorGetBuffer(buf));
ConditionVariableBroadcast(BufferDescriptorGetIOCV(buf));
+
+ /*
+ * Support LockBufferForCleanup()
+ *
+ * If we just released a pin, need to do BM_PIN_COUNT_WAITER handling.
+ * Most of the time the current backend will hold another pin preventing
+ * that from happening, but that's e.g. not the case when completing an IO
+ * another backend started.
+ */
+ if (buf_state & BM_PIN_COUNT_WAITER)
+ WakePinCountWaiter(buf);
}
/*
@@ -5697,7 +5746,7 @@ AbortBufferIO(Buffer buffer)
}
}
- TerminateBufferIO(buf_hdr, false, BM_IO_ERROR, false);
+ TerminateBufferIO(buf_hdr, false, BM_IO_ERROR, false, true);
}
/*
@@ -6148,3 +6197,328 @@ EvictUnpinnedBuffer(Buffer buf)
return result;
}
+
+/*
+ * Helper for AIO staging callback for both reads and writes as well as temp
+ * and shared buffers.
+ */
+static pg_attribute_always_inline void
+buffer_stage_common(PgAioHandle *ioh, bool is_write, bool is_temp)
+{
+ uint64 *io_data;
+ uint8 handle_data_len;
+ PgAioWaitRef io_ref;
+ BufferTag first PG_USED_FOR_ASSERTS_ONLY = {0};
+
+ io_data = pgaio_io_get_handle_data(ioh, &handle_data_len);
+
+ pgaio_io_get_wref(ioh, &io_ref);
+
+ for (int i = 0; i < handle_data_len; i++)
+ {
+ Buffer buffer = (Buffer) io_data[i];
+ BufferDesc *buf_hdr = is_temp ?
+ GetLocalBufferDescriptor(-buffer - 1)
+ : GetBufferDescriptor(buffer - 1);
+ uint32 buf_state;
+
+ /*
+ * Check that all the buffers are actually ones that could conceivably
+ * be done in one IO, i.e. are sequential.
+ */
+ if (i == 0)
+ first = buf_hdr->tag;
+ else
+ {
+ Assert(buf_hdr->tag.relNumber == first.relNumber);
+ Assert(buf_hdr->tag.blockNum == first.blockNum + i);
+ }
+
+ if (is_temp)
+ buf_state = pg_atomic_read_u32(&buf_hdr->state);
+ else
+ buf_state = LockBufHdr(buf_hdr);
+
+ Assert(buf_state & BM_TAG_VALID);
+ if (is_write)
+ {
+ Assert(buf_state & BM_VALID);
+ Assert(buf_state & BM_DIRTY);
+ }
+ else
+ {
+ Assert(!(buf_state & BM_VALID));
+ Assert(!(buf_state & BM_DIRTY));
+ }
+
+ /* temp buffers don't use BM_IO_IN_PROGRESS */
+ if (!is_temp)
+ Assert(buf_state & BM_IO_IN_PROGRESS);
+
+ /* we better have ensured the buffer is present until now */
+ Assert(BUF_STATE_GET_REFCOUNT(buf_state) >= 1);
+
+ /*
+ * Reflect that the buffer is now owned by the subsystem.
+ *
+ * For local buffers: This can't be done just in LocalRefCount as one
+ * might initially think, as this backend could error out while AIO is
+ * still in progress, releasing all the pins by the backend itself.
+ */
+ buf_state += BUF_REFCOUNT_ONE;
+ buf_hdr->io_wref = io_ref;
+
+ if (is_temp)
+ pg_atomic_unlocked_write_u32(&buf_hdr->state, buf_state);
+ else
+ UnlockBufHdr(buf_hdr, buf_state);
+
+ if (is_write && !is_temp)
+ {
+ LWLock *content_lock;
+
+ content_lock = BufferDescriptorGetContentLock(buf_hdr);
+
+ Assert(LWLockHeldByMe(content_lock));
+
+ /*
+ * Lock is now owned by AIO subsystem.
+ */
+ LWLockDisown(content_lock);
+ }
+
+ /*
+ * Stop tracking this buffer via the resowner - the AIO system now
+ * keeps track.
+ */
+ if (!is_temp)
+ ResourceOwnerForgetBufferIO(CurrentResourceOwner, buffer);
+ }
+}
+
+/*
+ * Helper for AIO readv completion callbacks, supporting both shared and temp
+ * buffers. Gets called once for each buffer in a multi-page read.
+ */
+static pg_attribute_always_inline PgAioResult
+buffer_readv_complete_one(uint8 buf_off, Buffer buffer, uint8 flags,
+ bool failed, bool is_temp)
+{
+ BufferDesc *buf_hdr = is_temp ?
+ GetLocalBufferDescriptor(-buffer - 1)
+ : GetBufferDescriptor(buffer - 1);
+ BufferTag tag = buf_hdr->tag;
+ char *bufdata = BufferGetBlock(buffer);
+ PgAioResult result;
+ uint32 set_flag_bits;
+
+#ifdef USE_ASSERT_CHECKING
+ {
+ uint32 buf_state = pg_atomic_read_u32(&buf_hdr->state);
+
+ Assert(buf_state & BM_TAG_VALID);
+ Assert(!(buf_state & BM_VALID));
+ /* temp buffers don't use BM_IO_IN_PROGRESS */
+ if (!is_temp)
+ Assert(buf_state & BM_IO_IN_PROGRESS);
+ Assert(!(buf_state & BM_DIRTY));
+ }
+#endif
+
+ result.status = ARS_OK;
+
+ /* check for garbage data */
+ if (!failed &&
+ !PageIsVerifiedExtended((Page) bufdata, tag.blockNum,
+ PIV_LOG_WARNING | PIV_REPORT_STAT))
+ {
+ RelFileLocator rlocator = BufTagGetRelFileLocator(&tag);
+
+ if ((flags & READ_BUFFERS_ZERO_ON_ERROR) || zero_damaged_pages)
+ {
+ ereport(LOG,
+ (errcode(ERRCODE_DATA_CORRUPTED),
+ errmsg("invalid page in block %u of relation %s; zeroing out page",
+ tag.blockNum,
+ relpathbackend(rlocator,
+ is_temp ? MyProcNumber : INVALID_PROC_NUMBER,
+ tag.forkNum).str)));
+ memset(bufdata, 0, BLCKSZ);
+ }
+ else
+ {
+ /* mark buffer as having failed */
+ failed = true;
+
+ /* encode error for buffer_readv_report */
+ result.status = ARS_ERROR;
+ if (is_temp)
+ result.id = PGAIO_HCB_LOCAL_BUFFER_READV;
+ else
+ result.id = PGAIO_HCB_SHARED_BUFFER_READV;
+ result.error_data = buf_off;
+ }
+ }
+
+ /* Terminate I/O and set BM_VALID. */
+ set_flag_bits = failed ? BM_IO_ERROR : BM_VALID;
+ if (is_temp)
+ TerminateLocalBufferIO(buf_hdr, false, set_flag_bits, false);
+ else
+ TerminateBufferIO(buf_hdr, false, set_flag_bits, false, false);
+
+ /*
+ * Call the BUFFER_READ_DONE tracepoint in the callback, even though the
+ * callback may not be executed in the same backend that called
+ * BUFFER_READ_START. The alternative would be to defer calling the
+ * tracepoint to a later point (e.g. the local completion callback for
+ * shared buffer reads), which seems even less helpful and would also
+ * imply some overhead.
+ */
+ TRACE_POSTGRESQL_BUFFER_READ_DONE(tag.forkNum,
+ tag.blockNum,
+ tag.spcOid,
+ tag.dbOid,
+ tag.relNumber,
+ is_temp ? MyProcNumber : INVALID_PROC_NUMBER,
+ false);
+
+ return result;
+}
+
+/*
+ * Perform completion handling of a single AIO read. This read may cover
+ * multiple blocks / buffers.
+ *
+ * Shared between shared and local buffers, to reduce code duplication.
+ */
+static pg_attribute_always_inline PgAioResult
+buffer_readv_complete(PgAioHandle *ioh, PgAioResult prior_result,
+ uint8 cb_data, bool is_temp)
+{
+ PgAioResult result = prior_result;
+ PgAioTargetData *td = pgaio_io_get_target_data(ioh);
+ uint64 *io_data;
+ uint8 handle_data_len;
+
+ if (is_temp)
+ {
+ Assert(td->smgr.is_temp);
+ Assert(pgaio_io_get_owner(ioh) == MyProcNumber);
+ }
+ else
+ Assert(!td->smgr.is_temp);
+
+ /*
+ * Iterate over all the buffers affected by this IO and call appropriate
+ * per-buffer completion function for each buffer.
+ */
+ io_data = pgaio_io_get_handle_data(ioh, &handle_data_len);
+ for (uint8 buf_off = 0; buf_off < handle_data_len; buf_off++)
+ {
+ Buffer buf = io_data[buf_off];
+ PgAioResult buf_result;
+ bool failed;
+
+ Assert(BufferIsValid(buf));
+
+ /*
+ * If the entire failed on a lower-level, each buffer needs to be
+ * marked as failed. In case of a partial read, some buffers may be
+ * ok.
+ */
+ failed =
+ prior_result.status == ARS_ERROR
+ || prior_result.result <= buf_off;
+
+ buf_result = buffer_readv_complete_one(buf_off, buf, cb_data, failed,
+ is_temp);
+
+ /*
+ * If there wasn't any prior error and the IO for this page failed in
+ * some form, set the whole IO's to the page's result.
+ */
+ if (result.status != ARS_ERROR && buf_result.status != ARS_OK)
+ {
+ result = buf_result;
+ pgaio_result_report(result, td, LOG);
+ }
+ }
+
+ return result;
+}
+
+/*
+ * AIO error reporting callback for aio_shared_buffer_readv_cb and
+ * aio_local_buffer_readv_cb.
+ *
+ * Errors are encoded as follows:
+ * - the only error is page verification failing
+ * - result.error_data is the offset of the first page that failed
+ * verification in a larger IO
+ */
+static void
+buffer_readv_report(PgAioResult result, const PgAioTargetData *target_data, int elevel)
+{
+ ProcNumber errProc;
+
+ if (target_data->smgr.is_temp)
+ errProc = MyProcNumber;
+ else
+ errProc = INVALID_PROC_NUMBER;
+
+ ereport(elevel,
+ errcode(ERRCODE_DATA_CORRUPTED),
+ errmsg("invalid page in block %u of relation %s",
+ target_data->smgr.blockNum + result.error_data,
+ relpathbackend(target_data->smgr.rlocator, errProc, target_data->smgr.forkNum).str
+ )
+ );
+}
+
+static void
+shared_buffer_readv_stage(PgAioHandle *ioh, uint8 cb_data)
+{
+ buffer_stage_common(ioh, false, false);
+}
+
+static PgAioResult
+shared_buffer_readv_complete(PgAioHandle *ioh, PgAioResult prior_result,
+ uint8 cb_data)
+{
+ return buffer_readv_complete(ioh, prior_result, cb_data, false);
+}
+
+static void
+local_buffer_readv_stage(PgAioHandle *ioh, uint8 cb_data)
+{
+ buffer_stage_common(ioh, false, true);
+}
+
+static PgAioResult
+local_buffer_readv_complete(PgAioHandle *ioh, PgAioResult prior_result,
+ uint8 cb_data)
+{
+ return buffer_readv_complete(ioh, prior_result, cb_data, true);
+}
+
+/* readv callback is is passed READ_BUFFERS_* flags as callback data */
+const PgAioHandleCallbacks aio_shared_buffer_readv_cb = {
+ .stage = shared_buffer_readv_stage,
+ .complete_shared = shared_buffer_readv_complete,
+ .report = buffer_readv_report,
+};
+
+/* readv callback is is passed READ_BUFFERS_* flags as callback data */
+const PgAioHandleCallbacks aio_local_buffer_readv_cb = {
+ .stage = local_buffer_readv_stage,
+
+ /*
+ * Note that this, in contrast to the shared_buffers case, uses
+ * complete_local, as only the issuing backend has access to the required
+ * datastructures. This is important in case the IO completion may be
+ * consumed incidentally by another backend.
+ */
+ .complete_local = local_buffer_readv_complete,
+ .report = buffer_readv_report,
+};
diff --git a/src/backend/storage/buffer/localbuf.c b/src/backend/storage/buffer/localbuf.c
index 02852d92b3e..86da17ae697 100644
--- a/src/backend/storage/buffer/localbuf.c
+++ b/src/backend/storage/buffer/localbuf.c
@@ -17,7 +17,9 @@
#include "access/parallel.h"
#include "executor/instrument.h"
+#include "pg_trace.h"
#include "pgstat.h"
+#include "storage/aio.h"
#include "storage/buf_internals.h"
#include "storage/bufmgr.h"
#include "storage/fd.h"
@@ -187,7 +189,7 @@ FlushLocalBuffer(BufferDesc *bufHdr, SMgrRelation reln)
* Try to start an I/O operation. There currently are no reasons for
* StartLocalBufferIO to return false, so we raise an error in that case.
*/
- if (!StartLocalBufferIO(bufHdr, false))
+ if (!StartLocalBufferIO(bufHdr, false, false))
elog(ERROR, "failed to start write IO on local buffer");
/* Find smgr relation for buffer */
@@ -211,7 +213,7 @@ FlushLocalBuffer(BufferDesc *bufHdr, SMgrRelation reln)
IOOP_WRITE, io_start, 1, BLCKSZ);
/* Mark not-dirty */
- TerminateLocalBufferIO(bufHdr, true, 0);
+ TerminateLocalBufferIO(bufHdr, true, 0, true);
pgBufferUsage.local_blks_written++;
}
@@ -430,7 +432,7 @@ ExtendBufferedRelLocal(BufferManagerRelation bmr,
pg_atomic_unlocked_write_u32(&existing_hdr->state, buf_state);
/* no need to loop for local buffers */
- StartLocalBufferIO(existing_hdr, true);
+ StartLocalBufferIO(existing_hdr, true, false);
}
else
{
@@ -446,7 +448,7 @@ ExtendBufferedRelLocal(BufferManagerRelation bmr,
hresult->id = victim_buf_id;
- StartLocalBufferIO(victim_buf_hdr, true);
+ StartLocalBufferIO(victim_buf_hdr, true, false);
}
}
@@ -515,10 +517,25 @@ MarkLocalBufferDirty(Buffer buffer)
* Like StartBufferIO, but for local buffers
*/
bool
-StartLocalBufferIO(BufferDesc *bufHdr, bool forInput)
+StartLocalBufferIO(BufferDesc *bufHdr, bool forInput, bool nowait)
{
- uint32 buf_state = pg_atomic_read_u32(&bufHdr->state);
+ uint32 buf_state;
+ /*
+ * The buffer could have IO in progress, e.g. when there are two scans of
+ * the same relation. Either wait for the other IO or return false.
+ */
+ if (pgaio_wref_valid(&bufHdr->io_wref))
+ {
+ PgAioWaitRef iow = bufHdr->io_wref;
+
+ if (nowait)
+ return false;
+
+ pgaio_wref_wait(&iow);
+ }
+
+ buf_state = pg_atomic_read_u32(&bufHdr->state);
if (forInput ? (buf_state & BM_VALID) : !(buf_state & BM_DIRTY))
{
/* someone else already did the I/O */
@@ -536,7 +553,8 @@ StartLocalBufferIO(BufferDesc *bufHdr, bool forInput)
* Like TerminateBufferIO, but for local buffers
*/
void
-TerminateLocalBufferIO(BufferDesc *bufHdr, bool clear_dirty, uint32 set_flag_bits)
+TerminateLocalBufferIO(BufferDesc *bufHdr, bool clear_dirty, uint32 set_flag_bits,
+ bool syncio)
{
/* Only need to adjust flags */
uint32 buf_state = pg_atomic_read_u32(&bufHdr->state);
@@ -549,6 +567,17 @@ TerminateLocalBufferIO(BufferDesc *bufHdr, bool clear_dirty, uint32 set_flag_bit
if (clear_dirty)
buf_state &= ~BM_DIRTY;
+ if (!syncio)
+ {
+ /*
+ * Release pin held by IO subsystem, see also
+ * local_buffer_readv_prepare().
+ */
+ Assert(BUF_STATE_GET_REFCOUNT(buf_state) > 0);
+ buf_state -= BUF_REFCOUNT_ONE;
+ pgaio_wref_clear(&bufHdr->io_wref);
+ }
+
buf_state |= set_flag_bits;
pg_atomic_unlocked_write_u32(&bufHdr->state, buf_state);
@@ -714,6 +743,8 @@ InitLocalBuffers(void)
*/
buf->buf_id = -i - 2;
+ pgaio_wref_clear(&buf->io_wref);
+
/*
* Intentionally do not initialize the buffer's atomic variable
* (besides zeroing the underlying memory above). That way we get
--
2.48.1.76.g4e746b1a31.dirty
[text/x-diff] v2.7-0023-bufmgr-Use-AIO-in-StartReadBuffers.patch (18.4K, ../../4b3f32ug3cayekysqlgspz2qjmeb7lca3gvazayglxr2m3d4dv@il33accgsji7/24-v2.7-0023-bufmgr-Use-AIO-in-StartReadBuffers.patch)
download | inline diff:
From 959ee2bf7b54785ef68aca8af8b7bc8c246fe792 Mon Sep 17 00:00:00 2001
From: Andres Freund <andres@anarazel.de>
Date: Sun, 9 Mar 2025 18:52:26 -0400
Subject: [PATCH v2.7 23/35] bufmgr: Use AIO in StartReadBuffers()
This finally introduces the first actual AIO user. StartReadBuffers() now uses
the AIO routines to issue IO. This converts a lot of callers to use the AIO
infrastructure.
As the implementation of StartReadBuffers() is also used by the functions for
reading individual blocks (StartReadBuffer() and through that
ReadBufferExtended()) this means all buffered read IO passes through the AIO
paths. However, as those are synchronous reads, actually performing the IO
asynchronously would be rarely beneficial. Instead such IOs are flagged to
always be executed synchronously. This way we don't have to duplicate a fair
bit of code.
When io_method=sync is used, the IO patterns generated after this change are
the same as before, i.e. actual reads are only issued in WaitReadBuffers() and
StartReadBuffers() may issue prefetch requests. This allows to bypass most of
the actual asynchronicity, which is important to make a change as big as this
less risky.
One thing worth calling out is that, if IO is actually executed
asynchronously, the precise meaning of what track_io_timing is measuring has
changed. Previously it tracked the time for each IO, but that does not make
sense when multiple IOs are executed concurrently. Now it only measures the
time actually spent waiting for IO.
While AIO is now actually used, the logic in read_stream.c will often prevent
using sufficiently many concurrent IOs. That will be addressed in the next
commits.
Co-authored-by: Andres Freund <andres@anarazel.de>
Co-authored-by: Thomas Munro <thomas.munro@gmail.com>
Discussion: https://postgr.es/m/uvrtrknj4kdytuboidbhwclo4gxhswwcpgadptsjvjqcluzmah%40brqs62irg4dt
Discussion: https://postgr.es/m/20210223100344.llw5an2aklengrmn@alap3.anarazel.de
Discussion: https://postgr.es/m/stj36ea6yyhoxtqkhpieia2z4krnam7qyetc57rfezgk4zgapf@gcnactj4z56m
---
src/include/storage/bufmgr.h | 5 +
src/backend/storage/buffer/bufmgr.c | 380 ++++++++++++++++++++--------
2 files changed, 285 insertions(+), 100 deletions(-)
diff --git a/src/include/storage/bufmgr.h b/src/include/storage/bufmgr.h
index ffb55cfa08a..cd921bdad49 100644
--- a/src/include/storage/bufmgr.h
+++ b/src/include/storage/bufmgr.h
@@ -112,6 +112,9 @@ typedef struct BufferManagerRelation
#define READ_BUFFERS_ZERO_ON_ERROR (1 << 0)
/* Call smgrprefetch() if I/O necessary. */
#define READ_BUFFERS_ISSUE_ADVICE (1 << 1)
+/* IO will immediately be waited for */
+#define READ_BUFFERS_SYNCHRONOUSLY (1 << 2)
+
struct ReadBuffersOperation
{
@@ -131,6 +134,8 @@ struct ReadBuffersOperation
BlockNumber blocknum;
int flags;
int16 nblocks;
+ PgAioWaitRef io_wref;
+ PgAioReturn io_return;
};
typedef struct ReadBuffersOperation ReadBuffersOperation;
diff --git a/src/backend/storage/buffer/bufmgr.c b/src/backend/storage/buffer/bufmgr.c
index 2bd1c4f371e..65fe5741f3f 100644
--- a/src/backend/storage/buffer/bufmgr.c
+++ b/src/backend/storage/buffer/bufmgr.c
@@ -528,6 +528,8 @@ static inline BufferDesc *BufferAlloc(SMgrRelation smgr,
BlockNumber blockNum,
BufferAccessStrategy strategy,
bool *foundPtr, IOContext io_context);
+static bool AsyncReadBuffers(ReadBuffersOperation *operation,
+ int *nblocks);
static Buffer GetVictimBuffer(BufferAccessStrategy strategy, IOContext io_context);
static void FlushBuffer(BufferDesc *buf, SMgrRelation reln,
IOObject io_object, IOContext io_context);
@@ -1228,10 +1230,14 @@ ReadBuffer_common(Relation rel, SMgrRelation smgr, char smgr_persistence,
return buffer;
}
+ /*
+ * Signal that we are going to immediately wait. If we're immediately
+ * waiting, there is no benefit in actually executing the IO
+ * asynchronously, it would just add dispatch overhead.
+ */
+ flags = READ_BUFFERS_SYNCHRONOUSLY;
if (mode == RBM_ZERO_ON_ERROR)
- flags = READ_BUFFERS_ZERO_ON_ERROR;
- else
- flags = 0;
+ flags |= READ_BUFFERS_ZERO_ON_ERROR;
operation.smgr = smgr;
operation.rel = rel;
operation.persistence = persistence;
@@ -1259,6 +1265,7 @@ StartReadBuffersImpl(ReadBuffersOperation *operation,
Assert(*nblocks > 0);
Assert(*nblocks <= MAX_IO_COMBINE_LIMIT);
+ Assert(*nblocks == 1 || allow_forwarding);
for (int i = 0; i < actual_nblocks; ++i)
{
@@ -1363,25 +1370,59 @@ StartReadBuffersImpl(ReadBuffersOperation *operation,
operation->blocknum = blockNum;
operation->flags = flags;
operation->nblocks = actual_nblocks;
+ pgaio_wref_clear(&operation->io_wref);
- if (flags & READ_BUFFERS_ISSUE_ADVICE)
+ /*
+ * When using AIO, start the IO in the background. If not, issue prefetch
+ * requests if desired by the caller.
+ *
+ * The reason we have a dedicated path for IOMETHOD_SYNC here is to
+ * de-risk the introduction of AIO somewhat. It's a large architectural
+ * change, with lots of chances for unanticipated performance effects.
+ *
+ * Use of IOMETHOD_SYNC already leads to not actually performing IO
+ * asynchronously, but without the check here we'd execute IO earlier than
+ * we used to. Eventually this IOMETHOD_SYNC specific path should go away.
+ */
+ if (io_method != IOMETHOD_SYNC)
{
/*
- * In theory we should only do this if PinBufferForBlock() had to
- * allocate new buffers above. That way, if two calls to
- * StartReadBuffers() were made for the same blocks before
- * WaitReadBuffers(), only the first would issue the advice. That'd be
- * a better simulation of true asynchronous I/O, which would only
- * start the I/O once, but isn't done here for simplicity.
+ * Try to start IO asynchronously. It's possible that no IO needs to
+ * be started, if another backend already performed the IO.
+ *
+ * Note that if an IO is started, it might not cover the entire
+ * requested range, e.g. because an intermediary block has been read
+ * in by another backend. In that case any "trailing" buffers we
+ * already pinned above will be "forwarded" by read_stream.c to the
+ * next call to StartReadBuffers(). This is signalled to the caller by
+ * decrementing *nblocks.
*/
- smgrprefetch(operation->smgr,
- operation->forknum,
- blockNum,
- actual_nblocks);
+ return AsyncReadBuffers(operation, nblocks);
}
+ else
+ {
+ operation->flags |= READ_BUFFERS_SYNCHRONOUSLY;
- /* Indicate that WaitReadBuffers() should be called. */
- return true;
+ if (flags & READ_BUFFERS_ISSUE_ADVICE)
+ {
+ /*
+ * In theory we should only do this if PinBufferForBlock() had to
+ * allocate new buffers above. That way, if two calls to
+ * StartReadBuffers() were made for the same blocks before
+ * WaitReadBuffers(), only the first would issue the advice.
+ * That'd be a better simulation of true asynchronous I/O, which
+ * would only start the I/O once, but isn't done here for
+ * simplicity.
+ */
+ smgrprefetch(operation->smgr,
+ operation->forknum,
+ blockNum,
+ actual_nblocks);
+ }
+
+ /* Indicate that WaitReadBuffers() should be called. */
+ return true;
+ }
}
/*
@@ -1449,7 +1490,7 @@ StartReadBuffer(ReadBuffersOperation *operation,
}
static inline bool
-WaitReadBuffersCanStartIO(Buffer buffer, bool nowait)
+ReadBuffersCanStartIO(Buffer buffer, bool nowait)
{
if (BufferIsLocal(buffer))
return StartLocalBufferIO(GetLocalBufferDescriptor(-buffer - 1),
@@ -1461,28 +1502,157 @@ WaitReadBuffersCanStartIO(Buffer buffer, bool nowait)
void
WaitReadBuffers(ReadBuffersOperation *operation)
{
- Buffer *buffers;
+ IOContext io_context;
+ IOObject io_object;
int nblocks;
- BlockNumber blocknum;
- ForkNumber forknum;
- IOContext io_context;
- IOObject io_object;
- char persistence;
+ PgAioReturn *aio_ret;
+
+ /*
+ * If we get here without an IO operation having been issued, io_method ==
+ * IOMETHOD_SYNC path must have been used. In that case, we start - as we
+ * used to before the introducing of AIO - the IO in WaitReadBuffers().
+ *
+ * This path is expected to eventually go away.
+ */
+ if (!pgaio_wref_valid(&operation->io_wref))
+ {
+ Assert(io_method == IOMETHOD_SYNC);
+
+ while (true)
+ {
+ nblocks = operation->nblocks;
+
+ if (!AsyncReadBuffers(operation, &nblocks))
+ {
+ /* all blocks were already read in concurrently */
+ Assert(nblocks == operation->nblocks);
+ return;
+ }
+
+ Assert(nblocks > 0 && nblocks <= operation->nblocks);
+
+ if (nblocks == operation->nblocks)
+ {
+ /* will wait below as if this had been normal AIO */
+ break;
+ }
+
+ /*
+ * It's unlikely, but possible, that AsyncReadBuffers() wasn't
+ * able to initiate IO for all the relevant buffers. In that case
+ * we need to wait for the prior IO before issuing more IO.
+ */
+ WaitReadBuffers(operation);
+ }
+ }
+
+ if (operation->persistence == RELPERSISTENCE_TEMP)
+ {
+ io_context = IOCONTEXT_NORMAL;
+ io_object = IOOBJECT_TEMP_RELATION;
+ }
+ else
+ {
+ io_context = IOContextForStrategy(operation->strategy);
+ io_object = IOOBJECT_RELATION;
+ }
+
+restart:
/* Find the range of the physical read we need to perform. */
nblocks = operation->nblocks;
- buffers = &operation->buffers[0];
- blocknum = operation->blocknum;
- forknum = operation->forknum;
- persistence = operation->persistence;
-
Assert(nblocks > 0);
Assert(nblocks <= MAX_IO_COMBINE_LIMIT);
+ aio_ret = &operation->io_return;
+
+ /*
+ * For IO timing we just count the time spent waiting for the IO.
+ *
+ * Tracking a wait even if we don't actually need to wait
+ *
+ * a) is not cheap, due to the timestamping overhead
+ *
+ * b) reports some time as waiting, even if we never waited
+ */
+ if (aio_ret->result.status == ARS_UNKNOWN &&
+ !pgaio_wref_check_done(&operation->io_wref))
+ {
+ instr_time io_start = pgstat_prepare_io_time(track_io_timing);
+
+ pgaio_wref_wait(&operation->io_wref);
+
+ /*
+ * The IO operation itself was already counted earlier, in
+ * AsyncReadBuffers().
+ */
+ pgstat_count_io_op_time(io_object, io_context, IOOP_READ,
+ io_start, 0, 0);
+ }
+ else
+ {
+ Assert(pgaio_wref_check_done(&operation->io_wref));
+ }
+
+ if (aio_ret->result.status == ARS_PARTIAL)
+ {
+ /*
+ * We'll retry below, so we just emit a debug message the server log
+ * (or not even that in prod scenarios).
+ */
+ pgaio_result_report(aio_ret->result, &aio_ret->target_data, DEBUG1);
+
+ /*
+ * Try to perform the rest of the IO. Buffers for which IO has
+ * completed successfully will be discovered as such and not retried.
+ */
+ nblocks = operation->nblocks;
+
+ elog(DEBUG3, "retrying IO after partial failure");
+ CHECK_FOR_INTERRUPTS();
+ AsyncReadBuffers(operation, &nblocks);
+ goto restart;
+ }
+ else if (aio_ret->result.status != ARS_OK)
+ pgaio_result_report(aio_ret->result, &aio_ret->target_data, ERROR);
+
+ if (VacuumCostActive)
+ VacuumCostBalance += VacuumCostPageMiss * nblocks;
+
+ /* NB: READ_DONE tracepoint is executed in IO completion callback */
+}
+
+/*
+ * Initiate IO for the ReadBuffersOperation. If IO is only initiated for a
+ * subset of the blocks, *nblocks is updated to reflect that.
+ *
+ * Returns true if IO was initiated, false if no IO was necessary.
+ */
+static bool
+AsyncReadBuffers(ReadBuffersOperation *operation,
+ int *nblocks)
+{
+ int io_buffers_len = 0;
+ Buffer *buffers = &operation->buffers[0];
+ int flags = operation->flags;
+ BlockNumber blocknum = operation->blocknum;
+ ForkNumber forknum = operation->forknum;
+ bool did_start_io = false;
+ PgAioHandle *ioh = NULL;
+ uint32 ioh_flags = 0;
+ IOContext io_context;
+ IOObject io_object;
+ char persistence;
+
+ persistence = operation->rel
+ ? operation->rel->rd_rel->relpersistence
+ : RELPERSISTENCE_PERMANENT;
+
if (persistence == RELPERSISTENCE_TEMP)
{
io_context = IOCONTEXT_NORMAL;
io_object = IOOBJECT_TEMP_RELATION;
+ ioh_flags |= PGAIO_HF_REFERENCES_LOCAL;
}
else
{
@@ -1490,6 +1660,14 @@ WaitReadBuffers(ReadBuffersOperation *operation)
io_object = IOOBJECT_RELATION;
}
+ /*
+ * When this IO is executed synchronously, either because the caller will
+ * immediately block waiting for the IO or because IOMETHOD_SYNC is used,
+ * the AIO subsystem needs to know.
+ */
+ if (flags & READ_BUFFERS_SYNCHRONOUSLY)
+ ioh_flags |= PGAIO_HF_SYNCHRONOUS;
+
/*
* We count all these blocks as read by this backend. This is traditional
* behavior, but might turn out to be not true if we find that someone
@@ -1499,25 +1677,54 @@ WaitReadBuffers(ReadBuffersOperation *operation)
* but another backend completed the read".
*/
if (persistence == RELPERSISTENCE_TEMP)
- pgBufferUsage.local_blks_read += nblocks;
+ pgBufferUsage.local_blks_read += *nblocks;
else
- pgBufferUsage.shared_blks_read += nblocks;
+ pgBufferUsage.shared_blks_read += *nblocks;
- for (int i = 0; i < nblocks; ++i)
+ pgaio_wref_clear(&operation->io_wref);
+
+ /*
+ * Loop until we have started one IO or we discover that all buffers are
+ * already valid.
+ */
+ for (int i = 0; i < *nblocks; ++i)
{
- int io_buffers_len;
Buffer io_buffers[MAX_IO_COMBINE_LIMIT];
void *io_pages[MAX_IO_COMBINE_LIMIT];
- instr_time io_start;
BlockNumber io_first_block;
/*
- * Skip this block if someone else has already completed it. If an
- * I/O is already in progress in another backend, this will wait for
- * the outcome: either done, or something went wrong and we will
- * retry.
+ * Get IO before ReadBuffersCanStartIO, as pgaio_io_acquire() might
+ * block, which we don't want after setting IO_IN_PROGRESS.
+ *
+ * XXX: Should we attribute the time spent in pgaio_io_acquire() to
+ * the IO? If there already are a lot of IO operations in progress,
+ * getting an IO handle will block waiting for some other IO operation
+ * to finish.
+ *
+ * In most cases it'll be free to get the IO, so a timer would be
+ * overhead. Perhaps we should use pgaio_io_acquire_nb() and only
+ * account IO time when pgaio_io_acquire_nb() returned false?
*/
- if (!WaitReadBuffersCanStartIO(buffers[i], false))
+ if (likely(!ioh))
+ ioh = pgaio_io_acquire(CurrentResourceOwner,
+ &operation->io_return);
+
+ /*
+ * Skip this block if someone else has already completed it.
+ *
+ * If an I/O is already in progress in another backend, this will wait
+ * for the outcome: either done, or something went wrong and we will
+ * retry. But don't wait if we have staged, but haven't issued,
+ * another IO.
+ *
+ * It's safe to start IO while we have unsubmitted IO, but it'd be
+ * better to first submit it. But right now the boolean return value
+ * from ReadBuffersCanStartIO()/StartBufferIO() doesn't allow to
+ * distinguish between nowait=true trigger failure and the buffer
+ * already being valid.
+ */
+ if (!ReadBuffersCanStartIO(buffers[i], false))
{
/*
* Report this as a 'hit' for this backend, even though it must
@@ -1545,8 +1752,8 @@ WaitReadBuffers(ReadBuffersOperation *operation)
* head block, so we should get on with that I/O as soon as possible.
* We'll come back to this block again, above.
*/
- while ((i + 1) < nblocks &&
- WaitReadBuffersCanStartIO(buffers[i + 1], true))
+ while ((i + 1) < *nblocks &&
+ ReadBuffersCanStartIO(buffers[i + 1], true))
{
/* Must be consecutive block numbers. */
Assert(BufferGetBlockNumber(buffers[i + 1]) ==
@@ -1556,67 +1763,40 @@ WaitReadBuffers(ReadBuffersOperation *operation)
io_pages[io_buffers_len++] = BufferGetBlock(buffers[i]);
}
- io_start = pgstat_prepare_io_time(track_io_timing);
- smgrreadv(operation->smgr, forknum, io_first_block, io_pages, io_buffers_len);
- pgstat_count_io_op_time(io_object, io_context, IOOP_READ, io_start,
- 1, io_buffers_len * BLCKSZ);
-
- /* Verify each block we read, and terminate the I/O. */
- for (int j = 0; j < io_buffers_len; ++j)
- {
- BufferDesc *bufHdr;
- Block bufBlock;
-
- if (persistence == RELPERSISTENCE_TEMP)
- {
- bufHdr = GetLocalBufferDescriptor(-io_buffers[j] - 1);
- bufBlock = LocalBufHdrGetBlock(bufHdr);
- }
- else
- {
- bufHdr = GetBufferDescriptor(io_buffers[j] - 1);
- bufBlock = BufHdrGetBlock(bufHdr);
- }
-
- /* check for garbage data */
- if (!PageIsVerifiedExtended((Page) bufBlock, io_first_block + j,
- PIV_LOG_WARNING | PIV_REPORT_STAT))
- {
- if ((operation->flags & READ_BUFFERS_ZERO_ON_ERROR) || zero_damaged_pages)
- {
- ereport(WARNING,
- (errcode(ERRCODE_DATA_CORRUPTED),
- errmsg("invalid page in block %u of relation %s; zeroing out page",
- io_first_block + j,
- relpath(operation->smgr->smgr_rlocator, forknum).str)));
- memset(bufBlock, 0, BLCKSZ);
- }
- else
- ereport(ERROR,
- (errcode(ERRCODE_DATA_CORRUPTED),
- errmsg("invalid page in block %u of relation %s",
- io_first_block + j,
- relpath(operation->smgr->smgr_rlocator, forknum).str)));
- }
-
- /* Set BM_VALID, terminate IO, and wake up any waiters */
- if (persistence == RELPERSISTENCE_TEMP)
- TerminateLocalBufferIO(bufHdr, false, BM_VALID, true);
- else
- TerminateBufferIO(bufHdr, false, BM_VALID, true, true);
-
- /* Report I/Os as completing individually. */
- TRACE_POSTGRESQL_BUFFER_READ_DONE(forknum, io_first_block + j,
- operation->smgr->smgr_rlocator.locator.spcOid,
- operation->smgr->smgr_rlocator.locator.dbOid,
- operation->smgr->smgr_rlocator.locator.relNumber,
- operation->smgr->smgr_rlocator.backend,
- false);
- }
-
- if (VacuumCostActive)
- VacuumCostBalance += VacuumCostPageMiss * io_buffers_len;
+ /* get a reference to wait for in WaitReadBuffers() */
+ pgaio_io_get_wref(ioh, &operation->io_wref);
+
+ /* provide the list of buffers to the completion callbacks */
+ pgaio_io_set_handle_data_32(ioh, (uint32 *) io_buffers, io_buffers_len);
+
+ pgaio_io_register_callbacks(ioh,
+ persistence == RELPERSISTENCE_TEMP ?
+ PGAIO_HCB_LOCAL_BUFFER_READV :
+ PGAIO_HCB_SHARED_BUFFER_READV,
+ flags);
+
+ pgaio_io_set_flag(ioh, ioh_flags);
+
+ did_start_io = true;
+ smgrstartreadv(ioh, operation->smgr, forknum, io_first_block,
+ io_pages, io_buffers_len);
+ ioh = NULL;
+
+ /* not obvious what we'd use for time */
+ pgstat_count_io_op(io_object, io_context, IOOP_READ,
+ 1, io_buffers_len * BLCKSZ);
+
+ *nblocks = io_buffers_len;
+ break;
+ }
+
+ if (ioh)
+ {
+ pgaio_io_release(ioh);
+ ioh = NULL;
}
+
+ return did_start_io;
}
/*
--
2.48.1.76.g4e746b1a31.dirty
[text/x-diff] v2.7-0024-docs-Reframe-track_io_timing-related-docs-as-wa.patch (5.3K, ../../4b3f32ug3cayekysqlgspz2qjmeb7lca3gvazayglxr2m3d4dv@il33accgsji7/25-v2.7-0024-docs-Reframe-track_io_timing-related-docs-as-wa.patch)
download | inline diff:
From c5d5bfdfa5a99b07b035aba5c37f15ed1bced508 Mon Sep 17 00:00:00 2001
From: Andres Freund <andres@anarazel.de>
Date: Wed, 12 Mar 2025 13:25:16 -0400
Subject: [PATCH v2.7 24/35] docs: Reframe track_io_timing related docs as wait
time
With AIO it does not make sense anymore to track the time for each individual
IO, as multiple IOs can be in-flight at the same time. Instead we now track
the time spent *waiting* for IOs.
This should be reflected in the docs. While, so far, we only do a subset of
reads, and no other operations, via AIO, describing the GUC and view columns
as measuring IO waits is accurate for synchronous and asynchronous IO.
Discussion: https://postgr.es/m/5dzyoduxlvfg55oqtjyjehez5uoq6hnwgzor4kkybkfdgkj7ag@rbi4gsmzaczk
---
doc/src/sgml/config.sgml | 4 ++--
doc/src/sgml/monitoring.sgml | 12 ++++++------
2 files changed, 8 insertions(+), 8 deletions(-)
diff --git a/doc/src/sgml/config.sgml b/doc/src/sgml/config.sgml
index fdc8f5c1fb0..c9b0a93cdc0 100644
--- a/doc/src/sgml/config.sgml
+++ b/doc/src/sgml/config.sgml
@@ -8529,7 +8529,7 @@ COPY postgres_log FROM '/full/path/to/logfile.csv' WITH csv;
</term>
<listitem>
<para>
- Enables timing of database I/O calls. This parameter is off by
+ Enables timing of database I/O waits. This parameter is off by
default, as it will repeatedly query the operating system for
the current time, which may cause significant overhead on some
platforms. You can use the <xref linkend="pgtesttiming"/> tool to
@@ -8563,7 +8563,7 @@ COPY postgres_log FROM '/full/path/to/logfile.csv' WITH csv;
</term>
<listitem>
<para>
- Enables timing of WAL I/O calls. This parameter is off by default,
+ Enables timing of WAL I/O waits. This parameter is off by default,
as it will repeatedly query the operating system for the current time,
which may cause significant overhead on some platforms.
You can use the <application>pg_test_timing</application> tool to
diff --git a/doc/src/sgml/monitoring.sgml b/doc/src/sgml/monitoring.sgml
index aaa6586d3a4..7598340072f 100644
--- a/doc/src/sgml/monitoring.sgml
+++ b/doc/src/sgml/monitoring.sgml
@@ -2733,7 +2733,7 @@ description | Waiting for a newly initialized WAL file to reach durable storage
<structfield>read_time</structfield> <type>double precision</type>
</para>
<para>
- Time spent in read operations in milliseconds (if
+ Time spent waiting for read operations in milliseconds (if
<xref linkend="guc-track-io-timing"/> is enabled and
<varname>object</varname> is not <literal>wal</literal>,
or if <xref linkend="guc-track-wal-io-timing"/> is enabled
@@ -2771,7 +2771,7 @@ description | Waiting for a newly initialized WAL file to reach durable storage
<structfield>write_time</structfield> <type>double precision</type>
</para>
<para>
- Time spent in write operations in milliseconds (if
+ Time spent waiting for write operations in milliseconds (if
<xref linkend="guc-track-io-timing"/> is enabled and
<varname>object</varname> is not <literal>wal</literal>,
or if <xref linkend="guc-track-wal-io-timing"/> is enabled
@@ -2799,7 +2799,7 @@ description | Waiting for a newly initialized WAL file to reach durable storage
<structfield>writeback_time</structfield> <type>double precision</type>
</para>
<para>
- Time spent in writeback operations in milliseconds (if
+ Time spent waiting for writeback operations in milliseconds (if
<xref linkend="guc-track-io-timing"/> is enabled, otherwise zero). This
includes the time spent queueing write-out requests and, potentially,
the time spent to write out the dirty data.
@@ -2835,7 +2835,7 @@ description | Waiting for a newly initialized WAL file to reach durable storage
<structfield>extend_time</structfield> <type>double precision</type>
</para>
<para>
- Time spent in extend operations in milliseconds. (if
+ Time spent waiting for extend operations in milliseconds. (if
<xref linkend="guc-track-io-timing"/> is enabled and
<varname>object</varname> is not <literal>wal</literal>,
or if <xref linkend="guc-track-wal-io-timing"/> is enabled
@@ -2909,7 +2909,7 @@ description | Waiting for a newly initialized WAL file to reach durable storage
<structfield>fsync_time</structfield> <type>double precision</type>
</para>
<para>
- Time spent in fsync operations in milliseconds (if
+ Time spent waiting for fsync operations in milliseconds (if
<xref linkend="guc-track-io-timing"/> is enabled and
<varname>object</varname> is not <literal>wal</literal>,
or if <xref linkend="guc-track-wal-io-timing"/> is enabled
@@ -2996,7 +2996,7 @@ description | Waiting for a newly initialized WAL file to reach durable storage
<note>
<para>
- Columns tracking I/O time will only be non-zero when
+ Columns tracking I/O wait time will only be non-zero when
<xref linkend="guc-track-io-timing"/> is enabled. The user should be
careful when referencing these columns in combination with their
corresponding I/O operations in case <varname>track_io_timing</varname>
--
2.48.1.76.g4e746b1a31.dirty
[text/x-diff] v2.7-0025-WIP-aio-read_stream.c-adjustments-for-real-AIO.patch (3.7K, ../../4b3f32ug3cayekysqlgspz2qjmeb7lca3gvazayglxr2m3d4dv@il33accgsji7/26-v2.7-0025-WIP-aio-read_stream.c-adjustments-for-real-AIO.patch)
download | inline diff:
From f03ea92c345ad8945747029bf193542fbe1ab47f Mon Sep 17 00:00:00 2001
From: Andres Freund <andres@anarazel.de>
Date: Tue, 11 Feb 2025 18:21:47 -0500
Subject: [PATCH v2.7 25/35] WIP: aio: read_stream.c adjustments for real AIO
Comments need to be fixed.
The batching logic probably needs to be adjusted.
---
src/backend/storage/aio/read_stream.c | 31 +++++++++++++++++++++++++--
1 file changed, 29 insertions(+), 2 deletions(-)
diff --git a/src/backend/storage/aio/read_stream.c b/src/backend/storage/aio/read_stream.c
index e58e0edf221..508e145efcc 100644
--- a/src/backend/storage/aio/read_stream.c
+++ b/src/backend/storage/aio/read_stream.c
@@ -32,10 +32,15 @@
* calls. Looking further ahead would pin many buffers and perform
* speculative work for no benefit.
*
+ * FIXME: This only applies to io_method == sync, otherwise this path is not
+ * used.
+ *
* C) I/O is necessary, it appears to be random, and this system supports
* read-ahead advice. We'll look further ahead in order to reach the
* configured level of I/O concurrency.
*
+ * FIXME: restriction to random only applies to io_method == sync
+ *
* The distance increases rapidly and decays slowly, so that it moves towards
* those levels as different I/O patterns are discovered. For example, a
* sequential scan of fully cached data doesn't bother looking ahead, but a
@@ -90,6 +95,7 @@
#include "postgres.h"
#include "miscadmin.h"
+#include "storage/aio.h"
#include "storage/fd.h"
#include "storage/smgr.h"
#include "storage/read_stream.h"
@@ -116,6 +122,7 @@ struct ReadStream
int16 pinned_buffers;
int16 distance;
int16 initialized_buffers;
+ bool sync_mode;
bool advice_enabled;
bool temporary;
@@ -458,6 +465,19 @@ read_stream_look_ahead(ReadStream *stream, bool suppress_advice)
{
int16 buffer_limit;
+ if (stream->distance > (io_combine_limit * 8))
+ {
+ if (stream->pinned_buffers + stream->pending_read_nblocks > ((stream->distance * 3) / 4))
+ {
+ return;
+ }
+ }
+
+ /*
+ * Try to amortize cost of submitting IOs over multiple IOs.
+ */
+ pgaio_enter_batchmode();
+
/*
* Check how many pins we could acquire now. We do this here rather than
* pushing it down into read_stream_start_pending_read(), because it
@@ -524,6 +544,7 @@ read_stream_look_ahead(ReadStream *stream, bool suppress_advice)
{
/* And we've hit a limit. Rewind, and stop here. */
read_stream_unget_block(stream, blocknum);
+ pgaio_exit_batchmode();
return;
}
}
@@ -549,6 +570,8 @@ read_stream_look_ahead(ReadStream *stream, bool suppress_advice)
stream->distance == 0) &&
stream->ios_in_progress < stream->max_ios)
read_stream_start_pending_read(stream, buffer_limit, suppress_advice);
+
+ pgaio_exit_batchmode();
}
/*
@@ -672,6 +695,8 @@ read_stream_begin_impl(int flags,
stream->per_buffer_data = (void *)
MAXALIGN(&stream->ios[Max(1, max_ios)]);
+ stream->sync_mode = io_method == IOMETHOD_SYNC;
+
#ifdef USE_PREFETCH
/*
@@ -680,7 +705,8 @@ read_stream_begin_impl(int flags,
* (overriding our detection heuristics), and max_ios hasn't been set to
* zero.
*/
- if ((io_direct_flags & IO_DIRECT_DATA) == 0 &&
+ if (stream->sync_mode &&
+ (io_direct_flags & IO_DIRECT_DATA) == 0 &&
(flags & READ_STREAM_SEQUENTIAL) == 0 &&
max_ios > 0)
stream->advice_enabled = true;
@@ -923,7 +949,8 @@ read_stream_next_buffer(ReadStream *stream, void **per_buffer_data)
if (++stream->oldest_io_index == stream->max_ios)
stream->oldest_io_index = 0;
- if (stream->ios[io_index].op.flags & READ_BUFFERS_ISSUE_ADVICE)
+ if (!stream->sync_mode ||
+ stream->ios[io_index].op.flags & READ_BUFFERS_ISSUE_ADVICE)
{
/* Distance ramps up fast (behavior C). */
distance = stream->distance * 2;
--
2.48.1.76.g4e746b1a31.dirty
[text/x-diff] v2.7-0026-aio-Add-test_aio-module.patch (42.2K, ../../4b3f32ug3cayekysqlgspz2qjmeb7lca3gvazayglxr2m3d4dv@il33accgsji7/27-v2.7-0026-aio-Add-test_aio-module.patch)
download | inline diff:
From 6622e1573444b0c6a44d5764d37cd194d1d76e74 Mon Sep 17 00:00:00 2001
From: Andres Freund <andres@anarazel.de>
Date: Wed, 22 Jan 2025 13:44:54 -0500
Subject: [PATCH v2.7 26/35] aio: Add test_aio module
---
src/include/storage/buf_internals.h | 6 +
src/backend/storage/buffer/bufmgr.c | 8 +-
src/test/modules/Makefile | 1 +
src/test/modules/meson.build | 1 +
src/test/modules/test_aio/.gitignore | 2 +
src/test/modules/test_aio/Makefile | 27 +
src/test/modules/test_aio/meson.build | 37 ++
src/test/modules/test_aio/t/001_aio.pl | 667 ++++++++++++++++++++
src/test/modules/test_aio/test_aio--1.0.sql | 97 +++
src/test/modules/test_aio/test_aio.c | 616 ++++++++++++++++++
src/test/modules/test_aio/test_aio.control | 3 +
11 files changed, 1459 insertions(+), 6 deletions(-)
create mode 100644 src/test/modules/test_aio/.gitignore
create mode 100644 src/test/modules/test_aio/Makefile
create mode 100644 src/test/modules/test_aio/meson.build
create mode 100644 src/test/modules/test_aio/t/001_aio.pl
create mode 100644 src/test/modules/test_aio/test_aio--1.0.sql
create mode 100644 src/test/modules/test_aio/test_aio.c
create mode 100644 src/test/modules/test_aio/test_aio.control
diff --git a/src/include/storage/buf_internals.h b/src/include/storage/buf_internals.h
index 8a52ace37cd..4b1f52a9fc8 100644
--- a/src/include/storage/buf_internals.h
+++ b/src/include/storage/buf_internals.h
@@ -434,6 +434,12 @@ extern void IssuePendingWritebacks(WritebackContext *wb_context, IOContext io_co
extern void ScheduleBufferTagForWriteback(WritebackContext *wb_context,
IOContext io_context, BufferTag *tag);
+/* solely to make it easier to write tests */
+extern bool StartBufferIO(BufferDesc *buf, bool forInput, bool nowait);
+extern void TerminateBufferIO(BufferDesc *buf, bool clear_dirty, uint32 set_flag_bits,
+ bool forget_owner, bool syncio);
+
+
/* freelist.c */
extern IOContext IOContextForStrategy(BufferAccessStrategy strategy);
extern BufferDesc *StrategyGetBuffer(BufferAccessStrategy strategy,
diff --git a/src/backend/storage/buffer/bufmgr.c b/src/backend/storage/buffer/bufmgr.c
index 65fe5741f3f..55187eefa5d 100644
--- a/src/backend/storage/buffer/bufmgr.c
+++ b/src/backend/storage/buffer/bufmgr.c
@@ -515,10 +515,6 @@ static uint32 WaitBufHdrUnlocked(BufferDesc *buf);
static int SyncOneBuffer(int buf_id, bool skip_recently_used,
WritebackContext *wb_context);
static void WaitIO(BufferDesc *buf);
-static bool StartBufferIO(BufferDesc *buf, bool forInput, bool nowait);
-static void TerminateBufferIO(BufferDesc *buf, bool clear_dirty,
- uint32 set_flag_bits, bool forget_owner,
- bool syncio);
static void AbortBufferIO(Buffer buffer);
static void shared_buffer_write_error_callback(void *arg);
static void local_buffer_write_error_callback(void *arg);
@@ -5777,7 +5773,7 @@ WaitIO(BufferDesc *buf)
* find out if they can perform the I/O as part of a larger operation, without
* waiting for the answer or distinguishing the reasons why not.
*/
-static bool
+bool
StartBufferIO(BufferDesc *buf, bool forInput, bool nowait)
{
uint32 buf_state;
@@ -5834,7 +5830,7 @@ StartBufferIO(BufferDesc *buf, bool forInput, bool nowait)
* resource owner. (forget_owner=false is used when the resource owner itself
* is being released)
*/
-static void
+void
TerminateBufferIO(BufferDesc *buf, bool clear_dirty, uint32 set_flag_bits,
bool forget_owner, bool syncio)
{
diff --git a/src/test/modules/Makefile b/src/test/modules/Makefile
index 4e4be3fa511..aa1d27bbed3 100644
--- a/src/test/modules/Makefile
+++ b/src/test/modules/Makefile
@@ -14,6 +14,7 @@ SUBDIRS = \
oauth_validator \
plsample \
spgist_name_ops \
+ test_aio \
test_bloomfilter \
test_copy_callbacks \
test_custom_rmgrs \
diff --git a/src/test/modules/meson.build b/src/test/modules/meson.build
index 2b057451473..4fda1acfb32 100644
--- a/src/test/modules/meson.build
+++ b/src/test/modules/meson.build
@@ -1,5 +1,6 @@
# Copyright (c) 2022-2025, PostgreSQL Global Development Group
+subdir('test_aio')
subdir('brin')
subdir('commit_ts')
subdir('delay_execution')
diff --git a/src/test/modules/test_aio/.gitignore b/src/test/modules/test_aio/.gitignore
new file mode 100644
index 00000000000..716e17f5a2a
--- /dev/null
+++ b/src/test/modules/test_aio/.gitignore
@@ -0,0 +1,2 @@
+# Generated subdirectories
+/tmp_check/
diff --git a/src/test/modules/test_aio/Makefile b/src/test/modules/test_aio/Makefile
new file mode 100644
index 00000000000..87d5315ba00
--- /dev/null
+++ b/src/test/modules/test_aio/Makefile
@@ -0,0 +1,27 @@
+# src/test/modules/delay_execution/Makefile
+
+PGFILEDESC = "test_aio - test code for AIO"
+
+MODULE_big = test_aio
+OBJS = \
+ $(WIN32RES) \
+ test_aio.o
+
+EXTENSION = test_aio
+DATA = test_aio--1.0.sql
+
+TAP_TESTS = 1
+
+export enable_injection_points
+export with_liburing
+
+ifdef USE_PGXS
+PG_CONFIG = pg_config
+PGXS := $(shell $(PG_CONFIG) --pgxs)
+include $(PGXS)
+else
+subdir = src/test/modules/test_aio
+top_builddir = ../../../..
+include $(top_builddir)/src/Makefile.global
+include $(top_srcdir)/contrib/contrib-global.mk
+endif
diff --git a/src/test/modules/test_aio/meson.build b/src/test/modules/test_aio/meson.build
new file mode 100644
index 00000000000..ac846b2b6f3
--- /dev/null
+++ b/src/test/modules/test_aio/meson.build
@@ -0,0 +1,37 @@
+# Copyright (c) 2022-2024, PostgreSQL Global Development Group
+
+test_aio_sources = files(
+ 'test_aio.c',
+)
+
+if host_system == 'windows'
+ test_aio_sources += rc_lib_gen.process(win32ver_rc, extra_args: [
+ '--NAME', 'test_aio',
+ '--FILEDESC', 'test_aio - test code for AIO',])
+endif
+
+test_aio = shared_module('test_aio',
+ test_aio_sources,
+ kwargs: pg_test_mod_args,
+)
+test_install_libs += test_aio
+
+test_install_data += files(
+ 'test_aio.control',
+ 'test_aio--1.0.sql',
+)
+
+tests += {
+ 'name': 'test_aio',
+ 'sd': meson.current_source_dir(),
+ 'bd': meson.current_build_dir(),
+ 'tap': {
+ 'env': {
+ 'enable_injection_points': get_option('injection_points') ? 'yes' : 'no',
+ 'with_liburing': liburing.found() ? 'yes' : 'no',
+ },
+ 'tests': [
+ 't/001_aio.pl',
+ ],
+ },
+}
diff --git a/src/test/modules/test_aio/t/001_aio.pl b/src/test/modules/test_aio/t/001_aio.pl
new file mode 100644
index 00000000000..2e18c8de338
--- /dev/null
+++ b/src/test/modules/test_aio/t/001_aio.pl
@@ -0,0 +1,667 @@
+# Copyright (c) 2025, PostgreSQL Global Development Group
+
+use strict;
+use warnings FATAL => 'all';
+
+use PostgreSQL::Test::Cluster;
+use PostgreSQL::Test::Utils;
+use Test::More;
+
+
+###
+# Test io_method=worker
+###
+my $node_worker = create_node('worker');
+$node_worker->start();
+
+test_generic('worker', $node_worker);
+SKIP:
+{
+ skip 'Injection points not supported by this build', 1
+ unless $ENV{enable_injection_points} eq 'yes';
+ test_inject_worker('worker', $node_worker);
+}
+
+$node_worker->stop();
+
+
+###
+# Test io_method=io_uring
+###
+
+if ($ENV{with_liburing} eq 'yes')
+{
+ my $node_uring = create_node('io_uring');
+ $node_uring->start();
+ test_generic('io_uring', $node_uring);
+ $node_uring->stop();
+}
+
+
+###
+# Test io_method=sync
+###
+
+my $node_sync = create_node('sync');
+
+# just to have one test not use the default auto-tuning
+
+$node_sync->append_conf(
+ 'postgresql.conf', qq(
+io_max_concurrency=4
+));
+
+$node_sync->start();
+test_generic('sync', $node_sync);
+$node_sync->stop();
+
+done_testing();
+
+
+###
+# Test Helpers
+###
+
+sub create_node
+{
+ my $io_method = shift;
+
+ my $node = PostgreSQL::Test::Cluster->new($io_method);
+
+ # Want to test initdb for each IO method, otherwise we could just reuse
+ # the cluster.
+ #
+ # Unfortunately Cluster::init() puts PG_TEST_INITDB_EXTRA_OPTS after the
+ # options specified by ->extra, if somebody puts -c io_method=xyz in
+ # PG_TEST_INITDB_EXTRA_OPTS it would break this test. Fix that up if we
+ # detect it.
+ local $ENV{PG_TEST_INITDB_EXTRA_OPTS} = $ENV{PG_TEST_INITDB_EXTRA_OPTS};
+ if (defined $ENV{PG_TEST_INITDB_EXTRA_OPTS} &&
+ $ENV{PG_TEST_INITDB_EXTRA_OPTS} =~ m/io_method=/)
+ {
+ $ENV{PG_TEST_INITDB_EXTRA_OPTS} .= " -c io_method=$io_method";
+ }
+
+ $node->init(extra => [ '-c', "io_method=$io_method" ]);
+
+ $node->append_conf(
+ 'postgresql.conf', qq(
+shared_preload_libraries=test_aio
+log_min_messages = 'DEBUG3'
+log_statement=all
+restart_after_crash=false
+));
+
+ ok(1, "$io_method: initdb");
+
+ return $node;
+}
+
+sub psql_like
+{
+ my $io_method = shift;
+ my $psql = shift;
+ my $name = shift;
+ my $sql = shift;
+ my $expected_stdout = shift;
+ my $expected_stderr = shift;
+ my ($cmdret, $output);
+
+ ($output, $cmdret) = $psql->query($sql);
+
+ like($output, $expected_stdout, "$io_method: $name: expected stdout");
+ like($psql->{stderr}, $expected_stderr,
+ "$io_method: $name: expected stderr");
+ $psql->{stderr} = '';
+}
+
+sub query_wait_block
+{
+ my $io_method = shift;
+ my $node = shift;
+ my $psql = shift;
+ my $name = shift;
+ my $sql = shift;
+ my $waitfor = shift;
+
+ my $pid = $psql->query_safe('SELECT pg_backend_pid()');
+
+ $psql->{stdin} .= qq($sql;\n);
+ $psql->{run}->pump_nb();
+ ok(1, "$io_method: $name: issued sql");
+
+ $node->poll_query_until('postgres',
+ qq(SELECT wait_event FROM pg_stat_activity WHERE pid = $pid),
+ $waitfor,);
+ ok(1, "$io_method: $name: observed $waitfor wait event");
+}
+
+
+###
+# Sub-tests
+###
+
+sub test_handle
+{
+ my $io_method = shift;
+ my $node = shift;
+
+ my $psql = $node->background_psql('postgres', on_error_stop => 0);
+
+ # leak warning: implicit xact
+ psql_like(
+ $io_method,
+ $psql,
+ "handle_get() leak in implicit xact",
+ qq(SELECT handle_get()),
+ qr/^$/,
+ qr/leaked AIO handle/,
+ "$io_method: leaky handle_get() warns");
+
+ # leak warning: explicit xact
+ psql_like(
+ $io_method, $psql,
+ "handle_get() leak in explicit xact",
+ qq(BEGIN; SELECT handle_get(); COMMIT),
+ qr/^$/, qr/leaked AIO handle/);
+
+
+ # leak warning: explicit xact, rollback
+ psql_like(
+ $io_method,
+ $psql,
+ "handle_get() leak in explicit xact, rollback",
+ qq(BEGIN; SELECT handle_get(); ROLLBACK;),
+ qr/^$/,
+ qr/leaked AIO handle/);
+
+ # leak warning: subtrans
+ psql_like(
+ $io_method,
+ $psql,
+ "handle_get() leak in subxact",
+ qq(BEGIN; SAVEPOINT foo; SELECT handle_get(); COMMIT;),
+ qr/^$/,
+ qr/leaked AIO handle/);
+
+ # leak warning + error: released in different command (thus resowner)
+ psql_like(
+ $io_method,
+ $psql,
+ "handle_release() in different command",
+ qq(BEGIN; SELECT handle_get(); SELECT handle_release_last(); COMMIT;),
+ qr/^$/,
+ qr/leaked AIO handle.*release in unexpected state/ms);
+
+ # no leak, release in same command
+ psql_like(
+ $io_method,
+ $psql,
+ "handle_release() in same command",
+ qq(BEGIN; SELECT handle_get() UNION ALL SELECT handle_release_last(); COMMIT;),
+ qr/^$/,
+ qr/^$/);
+
+ # normal handle use
+ psql_like($io_method, $psql, "handle_get_release()",
+ qq(SELECT handle_get_release()),
+ qr/^$/, qr/^$/);
+
+ # should error out, API violation
+ psql_like($io_method, $psql, "handle_get_twice()",
+ qq(SELECT handle_get_release()),
+ qr/^$/, qr/^$/);
+
+ # recover after error in implicit xact
+ psql_like(
+ $io_method,
+ $psql,
+ "handle error recovery in implicit xact",
+ qq(SELECT handle_get_and_error(); SELECT 'ok', handle_get_release()),
+ qr/^|ok$/,
+ qr/ERROR.*as you command/);
+
+ # recover after error in implicit xact
+ psql_like(
+ $io_method,
+ $psql,
+ "handle error recovery in explicit xact",
+ qq(BEGIN; SELECT handle_get_and_error(); SELECT handle_get_release(), 'ok'; COMMIT;),
+ qr/^|ok$/,
+ qr/ERROR.*as you command/);
+
+ # recover after error in subtrans
+ psql_like(
+ $io_method,
+ $psql,
+ "handle error recovery in explicit subxact",
+ qq(BEGIN; SAVEPOINT foo; SELECT handle_get_and_error(); ROLLBACK TO SAVEPOINT foo; SELECT handle_get_release(); ROLLBACK;),
+ qr/^|ok$/,
+ qr/ERROR.*as you command/);
+
+ $psql->quit();
+}
+
+
+sub test_batch
+{
+ my $io_method = shift;
+ my $node = shift;
+
+ my $psql = $node->background_psql('postgres', on_error_stop => 0);
+
+ # leak warning & recovery: implicit xact
+ psql_like(
+ $io_method,
+ $psql,
+ "batch_start() leak & cleanup in implicit xact",
+ qq(SELECT batch_start()),
+ qr/^$/,
+ qr/open AIO batch at end/,
+ "$io_method: leaky batch_start() warns");
+
+ # leak warning & recovery: explicit xact
+ psql_like(
+ $io_method,
+ $psql,
+ "batch_start() leak & cleanup in explicit xact",
+ qq(BEGIN; SELECT batch_start(); COMMIT;),
+ qr/^$/,
+ qr/open AIO batch at end/,
+ "$io_method: leaky batch_start() warns");
+
+
+ # leak warning & recovery: explicit xact, rollback
+ #
+ # XXX: This doesn't fail right now, due to not getting a chance to do
+ # something at transaction command commit. That's not a correctness issue,
+ # it just means it's a bit harder to find buggy code.
+ #psql_like($io_method, $psql,
+ # "batch_start() leak & cleanup after abort",
+ # qq(BEGIN; SELECT batch_start(); ROLLBACK;),
+ # qr/^$/,
+ # qr/open AIO batch at end/, "$io_method: leaky batch_start() warns");
+
+ # no warning, batch closed in same command
+ psql_like(
+ $io_method,
+ $psql,
+ "batch_start(), batch_end() works",
+ qq(SELECT batch_start() UNION ALL SELECT batch_end()),
+ qr/^$/,
+ qr/^$/,
+ "$io_method: batch_start(), batch_end()");
+
+ $psql->quit();
+}
+
+sub test_io_error
+{
+ my $io_method = shift;
+ my $node = shift;
+ my ($ret, $output);
+
+ my $psql = $node->background_psql('postgres', on_error_stop => 0);
+
+ # verify the error is reported in custom C code
+ ($output, $ret) = $psql->query(
+ qq(SELECT read_rel_block_ll('tbl_corr', 1, wait_complete=>true);));
+ is($ret, 1, "$io_method: read_rel_block_ll() of tbl_corr page fails");
+ like(
+ $psql->{stderr},
+ qr/invalid page in block 1 of relation base\/.*/,
+ "$io_method: read_rel_block_ll() of tbl_corr page reports error");
+ $psql->{stderr} = '';
+
+ # verify the error is reported for bufmgr reads
+ ($output, $ret) =
+ $psql->query(qq(SELECT count(*) FROM tbl_corr WHERE ctid = '(1, 1)'));
+ is($ret, 1, "$io_method: tid scan reading tbl_corr block fails");
+ like(
+ $psql->{stderr},
+ qr/invalid page in block 1 of relation base\/.*/,
+ "$io_method: tid scan reading tbl_corr block reports error");
+ $psql->{stderr} = '';
+
+ # verify the error is reported for bufmgr reads
+ ($output, $ret) =
+ $psql->query(qq(SELECT count(*) FROM tbl_corr WHERE ctid = '(1, 1)'));
+ is($ret, 1, "$io_method: sequential scan reading tbl_corr block fails");
+ like(
+ $psql->{stderr},
+ qr/invalid page in block 1 of relation base\/.*/,
+ "$io_method: sequential scan reading tbl_corr block reports error");
+ $psql->{stderr} = '';
+
+ $psql->quit();
+}
+
+# Test interplay between StartBufferIO and TerminateBufferIO
+sub test_startwait_io
+{
+ my $io_method = shift;
+ my $node = shift;
+ my ($ret, $output);
+
+ my $psql_a = $node->background_psql('postgres', on_error_stop => 0);
+ my $psql_b = $node->background_psql('postgres', on_error_stop => 0);
+
+ # create a buffer we can play around with
+ ($output, $ret) =
+ $psql_a->query(qq(SELECT buffer_create_toy('tbl_ok', 1);));
+ is($ret, 0, "$io_method: create toy succeeds");
+ like($output, qr/^\d+$/, "$io_method: create toy returns numeric");
+ my $buf_id = $output;
+
+ # check that one backend can perform StartBufferIO
+ psql_like(
+ $io_method,
+ $psql_a,
+ "first StartBufferIO",
+ qq(SELECT buffer_call_start_io($buf_id, for_input=>true, nowait=>false);),
+ qr/^t$/,
+ qr/^$/);
+
+ # but not twice on the same buffer (non-waiting)
+ psql_like(
+ $io_method,
+ $psql_a,
+ "second StartBufferIO fails, same session",
+ qq(SELECT buffer_call_start_io($buf_id, for_input=>true, nowait=>true);),
+ qr/^f$/,
+ qr/^$/);
+ psql_like(
+ $io_method,
+ $psql_b,
+ "second StartBufferIO fails, other session",
+ qq(SELECT buffer_call_start_io($buf_id, for_input=>true, nowait=>true);),
+ qr/^f$/,
+ qr/^$/);
+
+ # start io in a different session, will block
+ query_wait_block(
+ $io_method,
+ $node,
+ $psql_b,
+ "blocking start buffer io",
+ qq(SELECT buffer_call_start_io($buf_id, for_input=>true, nowait=>false);),
+ "BufferIo");
+
+ # Terminate the IO, without marking it as success, this should trigger the
+ # waiting session to be able to start the io
+ ($output, $ret) = $psql_a->query(
+ qq(SELECT buffer_call_terminate_io($buf_id, for_input=>true, succeed=>false, io_error=>false, syncio=>true);)
+ );
+ is($ret, 0,
+ "$io_method: blocking start buffer io, terminating io, not valid");
+
+ # Because the IO was terminated, but not marked as valid, second session should get the right to start io
+ pump_until($psql_b->{run}, $psql_b->{timeout}, \$psql_b->{stdout}, qr/t/);
+ ok(1, "$io_method: blocking start buffer io, can start io");
+
+ # terminate the IO again
+ $psql_b->query_safe(
+ qq(SELECT buffer_call_terminate_io($buf_id, for_input=>true, succeed=>false, io_error=>false, syncio=>true);)
+ );
+
+
+ # same as the above scenario, but mark IO as having succeeded
+ ($output, $ret) =
+ $psql_a->query(qq(SELECT buffer_call_start_io($buf_id, true, false);));
+ is($ret, 0,
+ "$io_method: blocking buffer io w/ success: first start buffer io succeeds"
+ );
+ is($output, "t",
+ "$io_method: blocking buffer io w/ success: first start buffer io returns true"
+ );
+
+
+ # start io in a different session, will block
+ query_wait_block(
+ $io_method,
+ $node,
+ $psql_b,
+ "blocking start buffer io",
+ qq(SELECT buffer_call_start_io($buf_id, for_input=>true, nowait=>false);),
+ "BufferIo");
+
+ ($output, $ret) = $psql_a->query(
+ qq(SELECT buffer_call_terminate_io($buf_id, for_input=>true, succeed=>true, io_error=>false, syncio=>true);)
+ );
+ is($ret, 0,
+ "$io_method: blocking start buffer io, terminating IO, valid");
+
+ # Because the IO was terminated, and marked as valid, second session should complete but not need io
+ pump_until($psql_b->{run}, $psql_b->{timeout}, \$psql_b->{stdout}, qr/f/);
+ ok(1, "$io_method: blocking start buffer io, no need to start io");
+
+
+ # buffer is valid now, make it invalid again
+ $buf_id = $psql_a->query_safe(qq(SELECT buffer_create_toy('tbl_ok', 1);));
+
+ $psql_a->quit();
+ $psql_b->quit();
+}
+
+# Test that if the backend issuing a read doesn't wait for the IO's
+# completion, another backend can complete the IO
+sub test_complete_foreign
+{
+ my $io_method = shift;
+ my $node = shift;
+ my ($ret, $output);
+
+ my $psql_a = $node->background_psql('postgres', on_error_stop => 0);
+ my $psql_b = $node->background_psql('postgres', on_error_stop => 0);
+
+ # Issue IO without waiting for completion, then sleep
+ $psql_a->query_safe(
+ qq(SELECT read_rel_block_ll('tbl_ok', 1, wait_complete=>false);));
+
+ # Check that another backend can read the relevant block
+ psql_like(
+ $io_method,
+ $psql_b,
+ "completing read started by sleeping backend",
+ qq(SELECT count(*) FROM tbl_ok WHERE ctid = '(1,1)' LIMIT 1),
+ qr/^1$/,
+ qr/^$/);
+
+ # Issue IO without waiting for completion, then exit
+ $psql_a->query_safe(
+ qq(SELECT read_rel_block_ll('tbl_ok', 1, wait_complete=>false);));
+ $psql_a->reconnect_and_clear();
+
+ # Check that another backend can read the relevant block
+ psql_like(
+ $io_method,
+ $psql_b,
+ "completing read started by exited backend",
+ qq(SELECT count(*) FROM tbl_ok WHERE ctid = '(1,1)' LIMIT 1),
+ qr/^1$/,
+ qr/^$/);
+
+ # Read a tbl_corr block, then sleep. The other session will retry the IO
+ # and also fail. The easiest thing to verify that seems to be to check
+ # that both are in the log.
+ my $log_location = -s $node->logfile;
+ $psql_a->query_safe(
+ qq(SELECT read_rel_block_ll('tbl_corr', 1, wait_complete=>false);));
+
+ psql_like(
+ $io_method,
+ $psql_b,
+ "completing read of tbl_corr block started by other backend",
+ qq(SELECT count(*) FROM tbl_corr WHERE ctid = '(1,1)' LIMIT 1),
+ qr/^$/,
+ qr/invalid page in block/);
+
+ # The log message issued for the read_rel_block_ll() should be logged as a LOG
+ $node->wait_for_log(qr/LOG[^\n]+invalid page in/, $log_location);
+ ok(1,
+ "$io_method: completing read of tbl_corr block started by other backend: LOG message for background read"
+ );
+
+ # But for the SELECT, it should be an ERROR
+ $log_location =
+ $node->wait_for_log(qr/ERROR[^\n]+invalid page in/, $log_location);
+ ok(1,
+ "$io_method: completing read of tbl_corr block started by other backend: ERROR message for foreground read"
+ );
+
+ $psql_a->quit();
+ $psql_b->quit();
+}
+
+sub test_inject
+{
+ my $io_method = shift;
+ my $node = shift;
+ my ($ret, $output);
+
+ my $psql = $node->background_psql('postgres', on_error_stop => 0);
+
+ # injected what we'd expect
+ $psql->query_safe(qq(SELECT inj_io_short_read_attach(8192);));
+ $psql->query_safe(qq(SELECT invalidate_rel_block('tbl_ok', 2);));
+ ($output, $ret) =
+ $psql->query(qq(SELECT count(*) FROM tbl_ok WHERE ctid = '(2, 1)';));
+ is($ret, 0,
+ "$io_method: injection point not triggering failure succeeds");
+
+ # injected a read shorter than a single block, expecting error
+ $psql->query_safe(qq(SELECT inj_io_short_read_attach(17);));
+ $psql->query_safe(qq(SELECT invalidate_rel_block('tbl_ok', 2);));
+ ($output, $ret) =
+ $psql->query(qq(SELECT count(*) FROM tbl_ok WHERE ctid = '(2, 1)';));
+ is($ret, 1, "$io_method: single block short read fails");
+ like(
+ $psql->{stderr},
+ qr/ERROR:.*could not read blocks 2\.\.2 in file "base\/.*": read only 0 of 8192 bytes/,
+ "$io_method: single block short read reports error");
+ $psql->{stderr} = '';
+
+ # shorten multi-block read to a single block, should retry
+ $psql->query_safe(
+ qq(
+SELECT invalidate_rel_block('tbl_ok', 0);
+SELECT invalidate_rel_block('tbl_ok', 1);
+SELECT invalidate_rel_block('tbl_ok', 2);
+SELECT inj_io_short_read_attach(8192);
+ ));
+ ($output, $ret) = $psql->query(qq(SELECT count(*) FROM tbl_ok;));
+ is($ret, 0, "$io_method: multi block short read is retried");
+
+ # verify that page verification errors are detected even as part of a
+ # shortened multi-block read (tbl_corr, block 1 is tbl_corred)
+ $psql->query_safe(
+ qq(
+SELECT invalidate_rel_block('tbl_corr', 0);
+SELECT invalidate_rel_block('tbl_corr', 1);
+SELECT invalidate_rel_block('tbl_corr', 2);
+SELECT inj_io_short_read_attach(8192);
+ ));
+ ($output, $ret) =
+ $psql->query(qq(SELECT count(*) FROM tbl_corr WHERE ctid < '(2, 1)'));
+ is($ret, 1,
+ "$io_method: shortened multi-block read detects invalid page");
+ like(
+ $psql->{stderr},
+ qr/ERROR:.*invalid page in block 1 of relation base\/.*/,
+ "$io_method: shortened multi-block reads reports invalid page");
+ $psql->{stderr} = '';
+
+ # trigger a hard error, should error out
+ $psql->query_safe(
+ qq(
+SELECT inj_io_short_read_attach(-errno_from_string('EIO'));
+SELECT invalidate_rel_block('tbl_ok', 2);
+ ));
+ ($output, $ret) =
+ $psql->query(qq(SELECT count(*) FROM tbl_ok; SELECT 1;));
+ is($ret, 1, "$io_method: hard IO error is detected");
+ like(
+ $psql->{stderr},
+ qr/ERROR:.*could not read blocks 2\.\.2 in file \"base\/.*\": Input\/output error/,
+ "$io_method: hard IO error is reported");
+ $psql->{stderr} = '';
+
+ $psql->query_safe(
+ qq(
+SELECT inj_io_short_read_detach();
+ ));
+
+ $psql->quit();
+}
+
+sub test_inject_worker
+{
+ my $io_method = shift;
+ my $node = shift;
+ my ($ret, $output);
+
+ my $psql = $node->background_psql('postgres', on_error_stop => 0);
+
+ # trigger a failure to reopen, should error out, but should recover
+ $psql->query_safe(
+ qq(
+SELECT inj_io_reopen_attach();
+SELECT invalidate_rel_block('tbl_ok', 1);
+ ));
+ ($output, $ret) = $psql->query(qq(SELECT count(*) FROM tbl_ok;));
+ is($ret, 1, "$io_method: failure to open is detected");
+ like(
+ $psql->{stderr},
+ qr/ERROR:.*could not read blocks 1\.\.1 in file "base\/.*": No such file or directory/,
+ "$io_method: failure to open is reported");
+ $psql->{stderr} = '';
+
+ $psql->query_safe(
+ qq(
+SELECT inj_io_reopen_detach();
+ ));
+
+ # check that we indeed recover
+ ($output, $ret) = $psql->query(qq(SELECT count(*) FROM tbl_ok;));
+ is($ret, 0, "$io_method: recovers from failure to open ");
+
+
+ $psql->quit();
+}
+
+sub test_generic
+{
+ my $io_method = shift;
+ my $node = shift;
+
+ is($node->safe_psql('postgres', 'SHOW io_method'),
+ $io_method, "$io_method: io_method set correctly");
+
+ $node->safe_psql(
+ 'postgres', qq(
+CREATE EXTENSION test_aio;
+CREATE TABLE tbl_corr(data int not null) WITH (AUTOVACUUM_ENABLED = false);
+CREATE TABLE tbl_ok(data int not null) WITH (AUTOVACUUM_ENABLED = false);
+
+INSERT INTO tbl_corr SELECT generate_series(1, 10000);
+INSERT INTO tbl_ok SELECT generate_series(1, 10000);
+SELECT grow_rel('tbl_corr', 16);
+SELECT grow_rel('tbl_ok', 16);
+
+SELECT corrupt_rel_block('tbl_corr', 1);
+CHECKPOINT;
+));
+
+ test_handle($io_method, $node);
+ test_io_error($io_method, $node);
+ test_batch($io_method, $node);
+ test_startwait_io($io_method, $node);
+ test_complete_foreign($io_method, $node);
+
+ SKIP:
+ {
+ skip 'Injection points not supported by this build', 1
+ unless $ENV{enable_injection_points} eq 'yes';
+ test_inject($io_method, $node);
+ }
+}
diff --git a/src/test/modules/test_aio/test_aio--1.0.sql b/src/test/modules/test_aio/test_aio--1.0.sql
new file mode 100644
index 00000000000..3d4e9bb070f
--- /dev/null
+++ b/src/test/modules/test_aio/test_aio--1.0.sql
@@ -0,0 +1,97 @@
+/* src/test/modules/test_aio/test_aio--1.0.sql */
+
+-- complain if script is sourced in psql, rather than via CREATE EXTENSION
+\echo Use "CREATE EXTENSION test_aio" to load this file. \quit
+
+
+CREATE FUNCTION errno_from_string(sym text)
+RETURNS pg_catalog.int4 STRICT
+AS 'MODULE_PATHNAME' LANGUAGE C;
+
+
+CREATE FUNCTION grow_rel(rel regclass, nblocks int)
+RETURNS pg_catalog.void STRICT
+AS 'MODULE_PATHNAME' LANGUAGE C;
+
+
+CREATE FUNCTION corrupt_rel_block(rel regclass, blockno int)
+RETURNS pg_catalog.void STRICT
+AS 'MODULE_PATHNAME' LANGUAGE C;
+
+CREATE FUNCTION read_rel_block_ll(rel regclass, blockno int, wait_complete bool)
+RETURNS pg_catalog.void STRICT
+AS 'MODULE_PATHNAME' LANGUAGE C;
+
+CREATE FUNCTION invalidate_rel_block(rel regclass, blockno int)
+RETURNS pg_catalog.void STRICT
+AS 'MODULE_PATHNAME' LANGUAGE C;
+
+CREATE FUNCTION buffer_create_toy(rel regclass, blockno int4)
+RETURNS pg_catalog.int4 STRICT
+AS 'MODULE_PATHNAME' LANGUAGE C;
+
+CREATE FUNCTION buffer_call_start_io(buffer int, for_input bool, nowait bool)
+RETURNS pg_catalog.bool STRICT
+AS 'MODULE_PATHNAME' LANGUAGE C;
+
+CREATE FUNCTION buffer_call_terminate_io(buffer int, for_input bool, succeed bool, io_error bool, syncio bool)
+RETURNS pg_catalog.void STRICT
+AS 'MODULE_PATHNAME' LANGUAGE C;
+
+
+
+/*
+ * Handle related functions
+ */
+CREATE FUNCTION handle_get_and_error()
+RETURNS pg_catalog.void STRICT
+AS 'MODULE_PATHNAME' LANGUAGE C;
+
+CREATE FUNCTION handle_get_twice()
+RETURNS pg_catalog.void STRICT
+AS 'MODULE_PATHNAME' LANGUAGE C;
+
+CREATE FUNCTION handle_get()
+RETURNS pg_catalog.void STRICT
+AS 'MODULE_PATHNAME' LANGUAGE C;
+
+CREATE FUNCTION handle_get_release()
+RETURNS pg_catalog.void STRICT
+AS 'MODULE_PATHNAME' LANGUAGE C;
+
+CREATE FUNCTION handle_release_last()
+RETURNS pg_catalog.void STRICT
+AS 'MODULE_PATHNAME' LANGUAGE C;
+
+
+/*
+ * Batchmode related functions
+ */
+CREATE FUNCTION batch_start()
+RETURNS pg_catalog.void STRICT
+AS 'MODULE_PATHNAME' LANGUAGE C;
+
+CREATE FUNCTION batch_end()
+RETURNS pg_catalog.void STRICT
+AS 'MODULE_PATHNAME' LANGUAGE C;
+
+
+
+/*
+ * Injection point related functions
+ */
+CREATE FUNCTION inj_io_short_read_attach(result int)
+RETURNS pg_catalog.void STRICT
+AS 'MODULE_PATHNAME' LANGUAGE C;
+
+CREATE FUNCTION inj_io_short_read_detach()
+RETURNS pg_catalog.void STRICT
+AS 'MODULE_PATHNAME' LANGUAGE C;
+
+CREATE FUNCTION inj_io_reopen_attach()
+RETURNS pg_catalog.void STRICT
+AS 'MODULE_PATHNAME' LANGUAGE C;
+
+CREATE FUNCTION inj_io_reopen_detach()
+RETURNS pg_catalog.void STRICT
+AS 'MODULE_PATHNAME' LANGUAGE C;
diff --git a/src/test/modules/test_aio/test_aio.c b/src/test/modules/test_aio/test_aio.c
new file mode 100644
index 00000000000..81b4d732206
--- /dev/null
+++ b/src/test/modules/test_aio/test_aio.c
@@ -0,0 +1,616 @@
+/*-------------------------------------------------------------------------
+ *
+ * delay_execution.c
+ * Test module to allow delay between parsing and execution of a query.
+ *
+ * The delay is implemented by taking and immediately releasing a specified
+ * advisory lock. If another process has previously taken that lock, the
+ * current process will be blocked until the lock is released; otherwise,
+ * there's no effect. This allows an isolationtester script to reliably
+ * test behaviors where some specified action happens in another backend
+ * between parsing and execution of any desired query.
+ *
+ * Copyright (c) 2020-2025, PostgreSQL Global Development Group
+ *
+ * IDENTIFICATION
+ * src/test/modules/delay_execution/delay_execution.c
+ *
+ *-------------------------------------------------------------------------
+ */
+
+#include "postgres.h"
+
+#include "access/relation.h"
+#include "fmgr.h"
+#include "storage/aio.h"
+#include "storage/aio_internal.h"
+#include "storage/buf_internals.h"
+#include "storage/bufmgr.h"
+#include "storage/ipc.h"
+#include "storage/lwlock.h"
+#include "utils/builtins.h"
+#include "utils/injection_point.h"
+#include "utils/rel.h"
+
+
+PG_MODULE_MAGIC;
+
+
+typedef struct InjIoErrorState
+{
+ bool enabled_short_read;
+ bool enabled_reopen;
+
+ bool short_read_result_set;
+ int short_read_result;
+} InjIoErrorState;
+
+static InjIoErrorState * inj_io_error_state;
+
+/* Shared memory init callbacks */
+static shmem_request_hook_type prev_shmem_request_hook = NULL;
+static shmem_startup_hook_type prev_shmem_startup_hook = NULL;
+
+
+static PgAioHandle *last_handle;
+
+
+
+static void
+test_aio_shmem_request(void)
+{
+ if (prev_shmem_request_hook)
+ prev_shmem_request_hook();
+
+ RequestAddinShmemSpace(sizeof(InjIoErrorState));
+}
+
+static void
+test_aio_shmem_startup(void)
+{
+ bool found;
+
+ if (prev_shmem_startup_hook)
+ prev_shmem_startup_hook();
+
+ /* Create or attach to the shared memory state */
+ LWLockAcquire(AddinShmemInitLock, LW_EXCLUSIVE);
+
+ inj_io_error_state = ShmemInitStruct("injection_points",
+ sizeof(InjIoErrorState),
+ &found);
+
+ if (!found)
+ {
+ /*
+ * First time through, so initialize. This is shared with the dynamic
+ * initialization using a DSM.
+ */
+ inj_io_error_state->enabled_short_read = false;
+ inj_io_error_state->enabled_reopen = false;
+
+#ifdef USE_INJECTION_POINTS
+ InjectionPointAttach("AIO_PROCESS_COMPLETION_BEFORE_SHARED",
+ "test_aio",
+ "inj_io_short_read",
+ NULL,
+ 0);
+ InjectionPointLoad("AIO_PROCESS_COMPLETION_BEFORE_SHARED");
+
+ InjectionPointAttach("AIO_WORKER_AFTER_REOPEN",
+ "test_aio",
+ "inj_io_reopen",
+ NULL,
+ 0);
+ InjectionPointLoad("AIO_WORKER_AFTER_REOPEN");
+
+#endif
+ }
+ else
+ {
+#ifdef USE_INJECTION_POINTS
+ InjectionPointLoad("AIO_PROCESS_COMPLETION_BEFORE_SHARED");
+ InjectionPointLoad("AIO_WORKER_AFTER_REOPEN");
+ elog(LOG, "injection point loaded");
+#endif
+ }
+
+ LWLockRelease(AddinShmemInitLock);
+}
+
+void
+_PG_init(void)
+{
+ if (!process_shared_preload_libraries_in_progress)
+ return;
+
+ /* Shared memory initialization */
+ prev_shmem_request_hook = shmem_request_hook;
+ shmem_request_hook = test_aio_shmem_request;
+ prev_shmem_startup_hook = shmem_startup_hook;
+ shmem_startup_hook = test_aio_shmem_startup;
+}
+
+
+PG_FUNCTION_INFO_V1(errno_from_string);
+Datum
+errno_from_string(PG_FUNCTION_ARGS)
+{
+ const char *sym = text_to_cstring(PG_GETARG_TEXT_PP(0));
+
+ if (strcmp(sym, "EIO") == 0)
+ PG_RETURN_INT32(EIO);
+ else if (strcmp(sym, "EAGAIN") == 0)
+ PG_RETURN_INT32(EAGAIN);
+ else if (strcmp(sym, "EINTR") == 0)
+ PG_RETURN_INT32(EINTR);
+ else if (strcmp(sym, "ENOSPC") == 0)
+ PG_RETURN_INT32(ENOSPC);
+ else if (strcmp(sym, "EROFS") == 0)
+ PG_RETURN_INT32(EROFS);
+
+ ereport(ERROR,
+ errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ errmsg_internal("%s is not a supported errno value", sym));
+ PG_RETURN_INT32(0);
+}
+
+
+PG_FUNCTION_INFO_V1(grow_rel);
+Datum
+grow_rel(PG_FUNCTION_ARGS)
+{
+ Oid relid = PG_GETARG_OID(0);
+ uint32 nblocks = PG_GETARG_UINT32(1);
+ Relation rel;
+#define MAX_BUFFERS_TO_EXTEND_BY 64
+ Buffer victim_buffers[MAX_BUFFERS_TO_EXTEND_BY];
+
+ rel = relation_open(relid, AccessExclusiveLock);
+
+ while (nblocks > 0)
+ {
+ uint32 extend_by_pages;
+
+ extend_by_pages = Min(nblocks, MAX_BUFFERS_TO_EXTEND_BY);
+
+ ExtendBufferedRelBy(BMR_REL(rel),
+ MAIN_FORKNUM,
+ NULL,
+ 0,
+ extend_by_pages,
+ victim_buffers,
+ &extend_by_pages);
+
+ nblocks -= extend_by_pages;
+
+ for (uint32 i = 0; i < extend_by_pages; i++)
+ {
+ ReleaseBuffer(victim_buffers[i]);
+ }
+ }
+
+ relation_close(rel, NoLock);
+
+ PG_RETURN_VOID();
+}
+
+PG_FUNCTION_INFO_V1(corrupt_rel_block);
+Datum
+corrupt_rel_block(PG_FUNCTION_ARGS)
+{
+ Oid relid = PG_GETARG_OID(0);
+ BlockNumber blkno = PG_GETARG_UINT32(1);
+ Relation rel;
+ Buffer buf;
+ Page page;
+ PageHeader ph;
+
+ rel = relation_open(relid, AccessExclusiveLock);
+
+ buf = ReadBuffer(rel, blkno);
+ page = BufferGetPage(buf);
+
+ LockBuffer(buf, BUFFER_LOCK_EXCLUSIVE);
+
+ MarkBufferDirty(buf);
+
+ PageInit(page, BufferGetPageSize(buf), 0);
+
+ ph = (PageHeader) page;
+ ph->pd_special = BLCKSZ + 1;
+
+ FlushOneBuffer(buf);
+
+ LockBuffer(buf, BUFFER_LOCK_UNLOCK);
+
+ ReleaseBuffer(buf);
+
+ EvictUnpinnedBuffer(buf);
+
+ relation_close(rel, NoLock);
+
+ PG_RETURN_VOID();
+}
+
+/*
+ * Ensures a buffer for rel & blkno is in shared buffers, without actually
+ * caring about the buffer contents. Used to set up test scenarios.
+ */
+static Buffer
+create_toy_buffer(Relation rel, BlockNumber blkno)
+{
+ Buffer buf;
+ BufferDesc *buf_hdr;
+ uint32 buf_state;
+ bool was_pinned = false;
+
+ /* read buffer without erroring out */
+ buf = ReadBufferExtended(rel, MAIN_FORKNUM, blkno, RBM_ZERO_ON_ERROR, NULL);
+
+ buf_hdr = GetBufferDescriptor(buf - 1);
+
+ buf_state = LockBufHdr(buf_hdr);
+
+ /*
+ * We should be the only backend accessing this buffer. This is just a
+ * small bit of belt-and-suspenders defense, none of this code should ever
+ * run in a cluster with real data.
+ */
+ if (BUF_STATE_GET_REFCOUNT(buf_state) > 1)
+ was_pinned = true;
+ else
+ buf_state &= ~(BM_VALID | BM_DIRTY);
+
+ UnlockBufHdr(buf_hdr, buf_state);
+
+ if (was_pinned)
+ elog(ERROR, "toy buffer %d was already pinned",
+ buf);
+
+ return buf;
+}
+
+PG_FUNCTION_INFO_V1(read_rel_block_ll);
+Datum
+read_rel_block_ll(PG_FUNCTION_ARGS)
+{
+ Oid relid = PG_GETARG_OID(0);
+ BlockNumber blkno = PG_GETARG_UINT32(1);
+ bool wait_complete = PG_GETARG_BOOL(2);
+ Relation rel;
+ Buffer buf;
+ Page pages[1];
+ PgAioReturn ior;
+ PgAioHandle *ioh;
+ PgAioWaitRef iow;
+ SMgrRelation smgr;
+
+ rel = relation_open(relid, AccessExclusiveLock);
+
+ buf = create_toy_buffer(rel, blkno);
+
+ pages[0] = BufferGetBlock(buf);
+
+ ioh = pgaio_io_acquire(CurrentResourceOwner, &ior);
+ pgaio_io_get_wref(ioh, &iow);
+
+ smgr = RelationGetSmgr(rel);
+
+ StartBufferIO(GetBufferDescriptor(buf - 1), true, false);
+
+ pgaio_io_set_handle_data_32(ioh, (uint32 *) &buf, 1);
+ pgaio_io_register_callbacks(ioh, PGAIO_HCB_SHARED_BUFFER_READV, 0);
+
+ elog(LOG, "about to smgrstartreadv");
+ smgrstartreadv(ioh, smgr, MAIN_FORKNUM, blkno,
+ (void *) pages, 1);
+
+ ReleaseBuffer(buf);
+
+ if (wait_complete)
+ {
+ pgaio_wref_wait(&iow);
+
+ if (ior.result.status != ARS_OK)
+ pgaio_result_report(ior.result, &ior.target_data,
+ ior.result.status == ARS_PARTIAL ? WARNING : ERROR);
+ }
+
+ relation_close(rel, NoLock);
+
+ PG_RETURN_VOID();
+}
+
+PG_FUNCTION_INFO_V1(invalidate_rel_block);
+Datum
+invalidate_rel_block(PG_FUNCTION_ARGS)
+{
+ Oid relid = PG_GETARG_OID(0);
+ BlockNumber blkno = PG_GETARG_UINT32(1);
+ Relation rel;
+ PrefetchBufferResult pr;
+ Buffer buf;
+
+ rel = relation_open(relid, AccessExclusiveLock);
+
+ /*
+ * This is a gross hack, but there's no other API exposed that allows to
+ * get a buffer ID without actually reading the block in.
+ */
+ pr = PrefetchBuffer(rel, MAIN_FORKNUM, blkno);
+ buf = pr.recent_buffer;
+
+ if (BufferIsValid(buf))
+ {
+ /* if the buffer contents aren't valid, this'll return false */
+ if (ReadRecentBuffer(rel->rd_locator, MAIN_FORKNUM, blkno, buf))
+ {
+ LockBuffer(buf, BUFFER_LOCK_EXCLUSIVE);
+ FlushOneBuffer(buf);
+ LockBuffer(buf, BUFFER_LOCK_UNLOCK);
+ ReleaseBuffer(buf);
+
+ if (!EvictUnpinnedBuffer(buf))
+ elog(ERROR, "couldn't evict");
+ }
+ }
+
+ relation_close(rel, AccessExclusiveLock);
+
+ PG_RETURN_VOID();
+}
+
+PG_FUNCTION_INFO_V1(buffer_create_toy);
+Datum
+buffer_create_toy(PG_FUNCTION_ARGS)
+{
+ Oid relid = PG_GETARG_OID(0);
+ BlockNumber blkno = PG_GETARG_UINT32(1);
+ Relation rel;
+ Buffer buf;
+
+ rel = relation_open(relid, AccessExclusiveLock);
+
+ buf = create_toy_buffer(rel, blkno);
+ ReleaseBuffer(buf);
+
+ relation_close(rel, NoLock);
+
+ PG_RETURN_INT32(buf);
+}
+
+PG_FUNCTION_INFO_V1(buffer_call_start_io);
+Datum
+buffer_call_start_io(PG_FUNCTION_ARGS)
+{
+ Buffer buf = PG_GETARG_INT32(0);
+ bool for_input = PG_GETARG_BOOL(1);
+ bool nowait = PG_GETARG_BOOL(2);
+ bool can_start;
+
+ can_start = StartBufferIO(GetBufferDescriptor(buf - 1), for_input, nowait);
+
+ /*
+ * Tor tests we don't want the resowner release preventing us from
+ * orchestrating odd scenarios.
+ */
+ if (can_start)
+ ResourceOwnerForgetBufferIO(CurrentResourceOwner,
+ buf);
+
+ ereport(LOG,
+ errmsg("buffer %d after StartBufferIO: %s",
+ buf, DebugPrintBufferRefcount(buf)),
+ errhidestmt(true), errhidecontext(true));
+
+ PG_RETURN_BOOL(can_start);
+}
+
+/*
+CREATE FUNCTION buffer_call_terminate_io(buffer int, for_input bool, failed bool, syncio bool)
+RETURNS pg_catalog.void STRICT
+AS 'MODULE_PATHNAME' LANGUAGE C;
+*/
+PG_FUNCTION_INFO_V1(buffer_call_terminate_io);
+Datum
+buffer_call_terminate_io(PG_FUNCTION_ARGS)
+{
+ Buffer buf = PG_GETARG_INT32(0);
+ bool for_input = PG_GETARG_BOOL(1);
+ bool succeed = PG_GETARG_BOOL(2);
+ bool io_error = PG_GETARG_BOOL(3);
+ bool syncio = PG_GETARG_BOOL(4);
+ bool clear_dirty = false;
+ uint32 set_flag_bits = 0;
+
+ if (io_error)
+ set_flag_bits |= BM_IO_ERROR;
+
+ if (for_input)
+ {
+ clear_dirty = false;
+
+ if (succeed)
+ set_flag_bits |= BM_VALID;
+ }
+ else
+ {
+ if (succeed)
+ clear_dirty = true;
+ }
+
+ ereport(LOG,
+ errmsg("buffer %d before TerminateBufferIO: %s",
+ buf, DebugPrintBufferRefcount(buf)),
+ errhidestmt(true), errhidecontext(true));
+
+ TerminateBufferIO(GetBufferDescriptor(buf - 1), clear_dirty, set_flag_bits,
+ false, syncio);
+
+ ereport(LOG,
+ errmsg("buffer %d after TerminateBufferIO: %s",
+ buf, DebugPrintBufferRefcount(buf)),
+ errhidestmt(true), errhidecontext(true));
+
+ PG_RETURN_VOID();
+}
+
+PG_FUNCTION_INFO_V1(handle_get);
+Datum
+handle_get(PG_FUNCTION_ARGS)
+{
+ last_handle = pgaio_io_acquire(CurrentResourceOwner, NULL);
+
+ PG_RETURN_VOID();
+}
+
+PG_FUNCTION_INFO_V1(handle_release_last);
+Datum
+handle_release_last(PG_FUNCTION_ARGS)
+{
+ if (!last_handle)
+ elog(ERROR, "no handle");
+
+ pgaio_io_release(last_handle);
+
+ PG_RETURN_VOID();
+}
+
+PG_FUNCTION_INFO_V1(handle_get_and_error);
+Datum
+handle_get_and_error(PG_FUNCTION_ARGS)
+{
+ pgaio_io_acquire(CurrentResourceOwner, NULL);
+
+ elog(ERROR, "as you command");
+ PG_RETURN_VOID();
+}
+
+PG_FUNCTION_INFO_V1(handle_get_twice);
+Datum
+handle_get_twice(PG_FUNCTION_ARGS)
+{
+ pgaio_io_acquire(CurrentResourceOwner, NULL);
+ pgaio_io_acquire(CurrentResourceOwner, NULL);
+
+ PG_RETURN_VOID();
+}
+
+PG_FUNCTION_INFO_V1(handle_get_release);
+Datum
+handle_get_release(PG_FUNCTION_ARGS)
+{
+ PgAioHandle *handle;
+
+ handle = pgaio_io_acquire(CurrentResourceOwner, NULL);
+ pgaio_io_release(handle);
+
+ PG_RETURN_VOID();
+}
+
+PG_FUNCTION_INFO_V1(batch_start);
+Datum
+batch_start(PG_FUNCTION_ARGS)
+{
+ pgaio_enter_batchmode();
+ PG_RETURN_VOID();
+}
+
+PG_FUNCTION_INFO_V1(batch_end);
+Datum
+batch_end(PG_FUNCTION_ARGS)
+{
+ pgaio_exit_batchmode();
+ PG_RETURN_VOID();
+}
+
+#ifdef USE_INJECTION_POINTS
+extern PGDLLEXPORT void inj_io_short_read(const char *name, const void *private_data);
+extern PGDLLEXPORT void inj_io_reopen(const char *name, const void *private_data);
+
+void
+inj_io_short_read(const char *name, const void *private_data)
+{
+ PgAioHandle *ioh;
+
+ elog(LOG, "short read called: %d", inj_io_error_state->enabled_short_read);
+
+ if (inj_io_error_state->enabled_short_read)
+ {
+ ioh = pgaio_inj_io_get();
+
+ if (inj_io_error_state->short_read_result_set)
+ {
+ elog(LOG, "short read, changing result from %d to %d",
+ ioh->result, inj_io_error_state->short_read_result);
+
+ ioh->result = inj_io_error_state->short_read_result;
+ }
+ }
+}
+
+void
+inj_io_reopen(const char *name, const void *private_data)
+{
+ elog(LOG, "reopen called: %d", inj_io_error_state->enabled_reopen);
+
+ if (inj_io_error_state->enabled_reopen)
+ {
+ elog(ERROR, "injection point triggering failure to reopen ");
+ }
+}
+#endif
+
+PG_FUNCTION_INFO_V1(inj_io_short_read_attach);
+Datum
+inj_io_short_read_attach(PG_FUNCTION_ARGS)
+{
+#ifdef USE_INJECTION_POINTS
+ inj_io_error_state->enabled_short_read = true;
+ inj_io_error_state->short_read_result_set = !PG_ARGISNULL(0);
+ if (inj_io_error_state->short_read_result_set)
+ inj_io_error_state->short_read_result = PG_GETARG_INT32(0);
+#else
+ elog(ERROR, "injection points not supported");
+#endif
+
+ PG_RETURN_VOID();
+}
+
+PG_FUNCTION_INFO_V1(inj_io_short_read_detach);
+Datum
+inj_io_short_read_detach(PG_FUNCTION_ARGS)
+{
+#ifdef USE_INJECTION_POINTS
+ inj_io_error_state->enabled_short_read = false;
+#else
+ elog(ERROR, "injection points not supported");
+#endif
+ PG_RETURN_VOID();
+}
+
+PG_FUNCTION_INFO_V1(inj_io_reopen_attach);
+Datum
+inj_io_reopen_attach(PG_FUNCTION_ARGS)
+{
+#ifdef USE_INJECTION_POINTS
+ inj_io_error_state->enabled_reopen = true;
+#else
+ elog(ERROR, "injection points not supported");
+#endif
+
+ PG_RETURN_VOID();
+}
+
+PG_FUNCTION_INFO_V1(inj_io_reopen_detach);
+Datum
+inj_io_reopen_detach(PG_FUNCTION_ARGS)
+{
+#ifdef USE_INJECTION_POINTS
+ inj_io_error_state->enabled_reopen = false;
+#else
+ elog(ERROR, "injection points not supported");
+#endif
+ PG_RETURN_VOID();
+}
diff --git a/src/test/modules/test_aio/test_aio.control b/src/test/modules/test_aio/test_aio.control
new file mode 100644
index 00000000000..cd91c3ed16b
--- /dev/null
+++ b/src/test/modules/test_aio/test_aio.control
@@ -0,0 +1,3 @@
+comment = 'Test code for AIO'
+default_version = '1.0'
+module_pathname = '$libdir/test_aio'
--
2.48.1.76.g4e746b1a31.dirty
[text/x-diff] v2.7-0027-aio-Implement-smgr-md-fd-write-support.patch (12.7K, ../../4b3f32ug3cayekysqlgspz2qjmeb7lca3gvazayglxr2m3d4dv@il33accgsji7/28-v2.7-0027-aio-Implement-smgr-md-fd-write-support.patch)
download | inline diff:
From a21e56cebe1186057859563cde2fe3a3e79f33d8 Mon Sep 17 00:00:00 2001
From: Andres Freund <andres@anarazel.de>
Date: Sun, 9 Mar 2025 19:35:04 -0400
Subject: [PATCH v2.7 27/35] aio: Implement smgr/md/fd write support
TODO:
- register_dirty_segment_aio() can error out in edge cases
---
src/include/storage/fd.h | 1 +
src/include/storage/md.h | 5 +
src/include/storage/smgr.h | 5 +
src/backend/storage/aio/aio_callback.c | 1 +
src/backend/storage/file/fd.c | 28 ++++
src/backend/storage/smgr/md.c | 189 +++++++++++++++++++++++++
src/backend/storage/smgr/smgr.c | 22 +++
7 files changed, 251 insertions(+)
diff --git a/src/include/storage/fd.h b/src/include/storage/fd.h
index b77d8e5e30e..2cc7c5a4761 100644
--- a/src/include/storage/fd.h
+++ b/src/include/storage/fd.h
@@ -112,6 +112,7 @@ extern int FilePrefetch(File file, off_t offset, off_t amount, uint32 wait_event
extern ssize_t FileReadV(File file, const struct iovec *iov, int iovcnt, off_t offset, uint32 wait_event_info);
extern ssize_t FileWriteV(File file, const struct iovec *iov, int iovcnt, off_t offset, uint32 wait_event_info);
extern int FileStartReadV(struct PgAioHandle *ioh, File file, int iovcnt, off_t offset, uint32 wait_event_info);
+extern int FileStartWriteV(struct PgAioHandle *ioh, File file, int iovcnt, off_t offset, uint32 wait_event_info);
extern int FileSync(File file, uint32 wait_event_info);
extern int FileZero(File file, off_t offset, off_t amount, uint32 wait_event_info);
extern int FileFallocate(File file, off_t offset, off_t amount, uint32 wait_event_info);
diff --git a/src/include/storage/md.h b/src/include/storage/md.h
index 9d7131eff43..47ae6c36c94 100644
--- a/src/include/storage/md.h
+++ b/src/include/storage/md.h
@@ -21,6 +21,7 @@
#include "storage/sync.h"
extern const PgAioHandleCallbacks aio_md_readv_cb;
+extern const PgAioHandleCallbacks aio_md_writev_cb;
/* md storage manager functionality */
extern void mdinit(void);
@@ -45,6 +46,10 @@ extern void mdstartreadv(PgAioHandle *ioh,
extern void mdwritev(SMgrRelation reln, ForkNumber forknum,
BlockNumber blocknum,
const void **buffers, BlockNumber nblocks, bool skipFsync);
+extern void mdstartwritev(PgAioHandle *ioh,
+ SMgrRelation reln, ForkNumber forknum,
+ BlockNumber blocknum,
+ const void **buffers, BlockNumber nblocks, bool skipFsync);
extern void mdwriteback(SMgrRelation reln, ForkNumber forknum,
BlockNumber blocknum, BlockNumber nblocks);
extern BlockNumber mdnblocks(SMgrRelation reln, ForkNumber forknum);
diff --git a/src/include/storage/smgr.h b/src/include/storage/smgr.h
index 856ebcda350..f00b3763ac9 100644
--- a/src/include/storage/smgr.h
+++ b/src/include/storage/smgr.h
@@ -108,6 +108,11 @@ extern void smgrwritev(SMgrRelation reln, ForkNumber forknum,
BlockNumber blocknum,
const void **buffers, BlockNumber nblocks,
bool skipFsync);
+extern void smgrstartwritev(PgAioHandle *ioh,
+ SMgrRelation reln, ForkNumber forknum,
+ BlockNumber blocknum,
+ const void **buffers, BlockNumber nblocks,
+ bool skipFsync);
extern void smgrwriteback(SMgrRelation reln, ForkNumber forknum,
BlockNumber blocknum, BlockNumber nblocks);
extern BlockNumber smgrnblocks(SMgrRelation reln, ForkNumber forknum);
diff --git a/src/backend/storage/aio/aio_callback.c b/src/backend/storage/aio/aio_callback.c
index f76f74ba166..fb6ac058a09 100644
--- a/src/backend/storage/aio/aio_callback.c
+++ b/src/backend/storage/aio/aio_callback.c
@@ -41,6 +41,7 @@ static const PgAioHandleCallbacksEntry aio_handle_cbs[] = {
CALLBACK_ENTRY(PGAIO_HCB_INVALID, aio_invalid_cb),
CALLBACK_ENTRY(PGAIO_HCB_MD_READV, aio_md_readv_cb),
+ CALLBACK_ENTRY(PGAIO_HCB_MD_WRITEV, aio_md_writev_cb),
CALLBACK_ENTRY(PGAIO_HCB_SHARED_BUFFER_READV, aio_shared_buffer_readv_cb),
diff --git a/src/backend/storage/file/fd.c b/src/backend/storage/file/fd.c
index be8c4c2d60d..65abebefbfb 100644
--- a/src/backend/storage/file/fd.c
+++ b/src/backend/storage/file/fd.c
@@ -2346,6 +2346,34 @@ retry:
return returnCode;
}
+int
+FileStartWriteV(PgAioHandle *ioh, File file,
+ int iovcnt, off_t offset,
+ uint32 wait_event_info)
+{
+ int returnCode;
+ Vfd *vfdP;
+
+ Assert(FileIsValid(file));
+
+ DO_DB(elog(LOG, "FileStartWriteV: %d (%s) " INT64_FORMAT " %d",
+ file, VfdCache[file].fileName,
+ (int64) offset,
+ iovcnt));
+
+ returnCode = FileAccess(file);
+ if (returnCode < 0)
+ return returnCode;
+
+ vfdP = &VfdCache[file];
+
+ /* FIXME: think about / reimplement temp_file_limit */
+
+ pgaio_io_prep_writev(ioh, vfdP->fd, iovcnt, offset);
+
+ return 0;
+}
+
int
FileSync(File file, uint32 wait_event_info)
{
diff --git a/src/backend/storage/smgr/md.c b/src/backend/storage/smgr/md.c
index 11d4d5a7aea..a61fc14805e 100644
--- a/src/backend/storage/smgr/md.c
+++ b/src/backend/storage/smgr/md.c
@@ -155,12 +155,19 @@ static BlockNumber _mdnblocks(SMgrRelation reln, ForkNumber forknum,
static PgAioResult md_readv_complete(PgAioHandle *ioh, PgAioResult prior_result, uint8 cb_data);
static void md_readv_report(PgAioResult result, const PgAioTargetData *target_data, int elevel);
+static PgAioResult md_writev_complete(PgAioHandle *ioh, PgAioResult prior_result, uint8 cb_data);
+static void md_writev_report(PgAioResult result, const PgAioTargetData *target_data, int elevel);
const PgAioHandleCallbacks aio_md_readv_cb = {
.complete_shared = md_readv_complete,
.report = md_readv_report,
};
+const PgAioHandleCallbacks aio_md_writev_cb = {
+ .complete_shared = md_writev_complete,
+ .report = md_writev_report,
+};
+
static inline int
_mdfd_open_flags(void)
@@ -1107,6 +1114,56 @@ mdwritev(SMgrRelation reln, ForkNumber forknum, BlockNumber blocknum,
}
}
+/*
+ * mdstartwritev() -- Asynchronous version of mdrwritev().
+ */
+void
+mdstartwritev(PgAioHandle *ioh,
+ SMgrRelation reln, ForkNumber forknum, BlockNumber blocknum,
+ const void **buffers, BlockNumber nblocks, bool skipFsync)
+{
+ off_t seekpos;
+ MdfdVec *v;
+ BlockNumber nblocks_this_segment;
+ struct iovec *iov;
+ int iovcnt;
+
+ v = _mdfd_getseg(reln, forknum, blocknum, false,
+ EXTENSION_FAIL | EXTENSION_CREATE_RECOVERY);
+
+ seekpos = (off_t) BLCKSZ * (blocknum % ((BlockNumber) RELSEG_SIZE));
+
+ Assert(seekpos < (off_t) BLCKSZ * RELSEG_SIZE);
+
+ nblocks_this_segment =
+ Min(nblocks,
+ RELSEG_SIZE - (blocknum % ((BlockNumber) RELSEG_SIZE)));
+
+ if (nblocks_this_segment != nblocks)
+ elog(ERROR, "write crossing segment boundary");
+
+ iovcnt = pgaio_io_get_iovec(ioh, &iov);
+
+ Assert(nblocks <= iovcnt);
+
+ iovcnt = buffers_to_iovec(iov, unconstify(void **, buffers), nblocks_this_segment);
+
+ Assert(iovcnt <= nblocks_this_segment);
+
+ if (!(io_direct_flags & IO_DIRECT_DATA))
+ pgaio_io_set_flag(ioh, PGAIO_HF_BUFFERED);
+
+ pgaio_io_set_target_smgr(ioh,
+ reln,
+ forknum,
+ blocknum,
+ nblocks,
+ skipFsync);
+ pgaio_io_register_callbacks(ioh, PGAIO_HCB_MD_WRITEV, 0);
+
+ FileStartWriteV(ioh, v->mdfd_vfd, iovcnt, seekpos, WAIT_EVENT_DATA_FILE_WRITE);
+}
+
/*
* mdwriteback() -- Tell the kernel to write pages back to storage.
@@ -1495,6 +1552,40 @@ register_dirty_segment(SMgrRelation reln, ForkNumber forknum, MdfdVec *seg)
}
}
+/*
+ * Like register_dirty_segment(), except for use by AIO. In the completion
+ * callback we don't have access to the MdfdVec (the completion callback might
+ * be executed in a different backend than the issuing backend), therefore we
+ * have to implement this slightly differently.
+ */
+static void
+register_dirty_segment_aio(RelFileLocator locator, ForkNumber forknum, uint64 segno)
+{
+ FileTag tag;
+
+ INIT_MD_FILETAG(tag, locator, forknum, segno);
+
+ /*
+ * Can't block here waiting for checkpointer to accept our sync request,
+ * as checkpointer might be waiting for this AIO to finish if offloaded to
+ * a worker.
+ */
+ if (!RegisterSyncRequest(&tag, SYNC_REQUEST, false /* retryOnError */ ))
+ {
+ char path[MAXPGPATH];
+
+ ereport(DEBUG1,
+ (errmsg_internal("could not forward fsync request because request queue is full")));
+
+ /* reuse mdsyncfiletag() to avoid duplicating code */
+ if (mdsyncfiletag(&tag, path))
+ ereport(data_sync_elevel(ERROR),
+ (errcode_for_file_access(),
+ errmsg("could not fsync file \"%s\": %m",
+ path)));
+ }
+}
+
/*
* register_unlink_segment() -- Schedule a file to be deleted after next checkpoint
*/
@@ -2017,3 +2108,101 @@ md_readv_report(PgAioResult result, const PgAioTargetData *td, int elevel)
td->smgr.nblocks * (size_t) BLCKSZ));
}
}
+
+/*
+ * AIO completion callback for mdstartwritev().
+ */
+static PgAioResult
+md_writev_complete(PgAioHandle *ioh, PgAioResult prior_result, uint8 cb_data)
+{
+ PgAioTargetData *td = pgaio_io_get_target_data(ioh);
+ PgAioResult result = prior_result;
+
+ if (prior_result.result < 0)
+ {
+ result.status = ARS_ERROR;
+ result.id = PGAIO_HCB_MD_WRITEV;
+ /* For "hard" errors, track the error number in error_data */
+ result.error_data = -prior_result.result;
+ result.result = 0;
+
+ pgaio_result_report(result, td, LOG);
+
+ return result;
+ }
+
+ /*
+ * The smgr API operates in blocks, therefore convert the result from
+ * bytes to blocks.
+ */
+ result.result /= BLCKSZ;
+
+ if (result.result == 0)
+ {
+ /* consider 0 blocks written a failure */
+ result.status = ARS_ERROR;
+ result.id = PGAIO_HCB_MD_WRITEV;
+ result.error_data = 0;
+
+ pgaio_result_report(result, td, LOG);
+
+ return result;
+ }
+
+ if (result.status != ARS_ERROR &&
+ result.result < td->smgr.nblocks)
+ {
+ /* partial writes should be retried at upper level */
+ result.status = ARS_PARTIAL;
+ result.id = PGAIO_HCB_MD_WRITEV;
+ }
+
+ if (!td->smgr.skip_fsync)
+ register_dirty_segment_aio(td->smgr.rlocator, td->smgr.forkNum,
+ td->smgr.blockNum / ((BlockNumber) RELSEG_SIZE));
+
+ return result;
+}
+
+/*
+ * AIO error reporting callback for mdstartwritev().
+ */
+static void
+md_writev_report(PgAioResult result, const PgAioTargetData *td, int elevel)
+{
+ RelPathStr path;
+
+ path = relpathbackend(td->smgr.rlocator,
+ td->smgr.is_temp ? MyProcNumber : INVALID_PROC_NUMBER,
+ td->smgr.forkNum);
+
+ if (result.error_data != 0)
+ {
+ errno = result.error_data; /* for errcode_for_file_access() */
+
+ ereport(elevel,
+ errcode_for_file_access(),
+ errmsg("could not write blocks %u..%u in file \"%s\": %m",
+ td->smgr.blockNum,
+ td->smgr.blockNum + td->smgr.nblocks,
+ path.str)
+ );
+ }
+ else
+ {
+ /*
+ * NB: This will typically only be output in debug messages, while
+ * retrying a partial IO.
+ */
+ ereport(elevel,
+ errcode(ERRCODE_DATA_CORRUPTED),
+ errmsg("could not write blocks %u..%u in file \"%s\": wrote only %zu of %zu bytes",
+ td->smgr.blockNum,
+ td->smgr.blockNum + td->smgr.nblocks - 1,
+ path.str,
+ result.result * (size_t) BLCKSZ,
+ td->smgr.nblocks * (size_t) BLCKSZ
+ )
+ );
+ }
+}
diff --git a/src/backend/storage/smgr/smgr.c b/src/backend/storage/smgr/smgr.c
index 607c14ee173..088b189543b 100644
--- a/src/backend/storage/smgr/smgr.c
+++ b/src/backend/storage/smgr/smgr.c
@@ -102,6 +102,11 @@ typedef struct f_smgr
BlockNumber blocknum,
const void **buffers, BlockNumber nblocks,
bool skipFsync);
+ void (*smgr_startwritev) (PgAioHandle *ioh,
+ SMgrRelation reln, ForkNumber forknum,
+ BlockNumber blocknum,
+ const void **buffers, BlockNumber nblocks,
+ bool skipFsync);
void (*smgr_writeback) (SMgrRelation reln, ForkNumber forknum,
BlockNumber blocknum, BlockNumber nblocks);
BlockNumber (*smgr_nblocks) (SMgrRelation reln, ForkNumber forknum);
@@ -129,6 +134,7 @@ static const f_smgr smgrsw[] = {
.smgr_readv = mdreadv,
.smgr_startreadv = mdstartreadv,
.smgr_writev = mdwritev,
+ .smgr_startwritev = mdstartwritev,
.smgr_writeback = mdwriteback,
.smgr_nblocks = mdnblocks,
.smgr_truncate = mdtruncate,
@@ -691,6 +697,22 @@ smgrwritev(SMgrRelation reln, ForkNumber forknum, BlockNumber blocknum,
buffers, nblocks, skipFsync);
}
+/*
+ * smgrstartwritev() -- asynchronous version of smgrwritev()
+ *
+ * This starts an asynchronous writev IO with the IO handle `ioh`. Other than
+ * `ioh` all parameters are the same as smgrwritev().
+ */
+void
+smgrstartwritev(PgAioHandle *ioh,
+ SMgrRelation reln, ForkNumber forknum, BlockNumber blocknum,
+ const void **buffers, BlockNumber nblocks, bool skipFsync)
+{
+ smgrsw[reln->smgr_which].smgr_startwritev(ioh,
+ reln, forknum, blocknum, buffers,
+ nblocks, skipFsync);
+}
+
/*
* smgrwriteback() -- Trigger kernel writeback for the supplied range of
* blocks.
--
2.48.1.76.g4e746b1a31.dirty
[text/x-diff] v2.7-0028-aio-Add-bounce-buffers.patch (23.1K, ../../4b3f32ug3cayekysqlgspz2qjmeb7lca3gvazayglxr2m3d4dv@il33accgsji7/29-v2.7-0028-aio-Add-bounce-buffers.patch)
download | inline diff:
From b1996adba85d9f00279a3e249da3647394073dce Mon Sep 17 00:00:00 2001
From: Andres Freund <andres@anarazel.de>
Date: Fri, 7 Mar 2025 15:47:38 -0500
Subject: [PATCH v2.7 28/35] aio: Add bounce buffers
---
src/include/storage/aio.h | 15 ++
src/include/storage/aio_internal.h | 34 ++++
src/include/storage/aio_types.h | 2 +
src/include/utils/resowner.h | 2 +
src/backend/storage/aio/README.md | 27 +++
src/backend/storage/aio/aio.c | 178 ++++++++++++++++++
src/backend/storage/aio/aio_init.c | 123 ++++++++++++
src/backend/utils/misc/guc_tables.c | 13 ++
src/backend/utils/misc/postgresql.conf.sample | 2 +
src/backend/utils/resowner/resowner.c | 25 ++-
src/test/modules/test_aio/test_aio--1.0.sql | 21 +++
src/test/modules/test_aio/test_aio.c | 55 ++++++
src/tools/pgindent/typedefs.list | 1 +
13 files changed, 496 insertions(+), 2 deletions(-)
diff --git a/src/include/storage/aio.h b/src/include/storage/aio.h
index a4d9b51acfe..3047dc77ce1 100644
--- a/src/include/storage/aio.h
+++ b/src/include/storage/aio.h
@@ -337,6 +337,20 @@ extern bool pgaio_have_staged(void);
+/* --------------------------------------------------------------------------------
+ * Bounce Buffers
+ * --------------------------------------------------------------------------------
+ */
+
+extern PgAioBounceBuffer *pgaio_bounce_buffer_get(void);
+extern void pgaio_io_assoc_bounce_buffer(PgAioHandle *ioh, PgAioBounceBuffer *bb);
+extern uint32 pgaio_bounce_buffer_id(PgAioBounceBuffer *bb);
+extern void pgaio_bounce_buffer_release(PgAioBounceBuffer *bb);
+extern char *pgaio_bounce_buffer_buffer(PgAioBounceBuffer *bb);
+extern void pgaio_bounce_buffer_release_resowner(struct dlist_node *bb_node, bool on_error);
+
+
+
/* --------------------------------------------------------------------------------
* Other
* --------------------------------------------------------------------------------
@@ -349,6 +363,7 @@ extern void pgaio_closing_fd(int fd);
/* GUCs */
extern PGDLLIMPORT int io_method;
extern PGDLLIMPORT int io_max_concurrency;
+extern PGDLLIMPORT int io_bounce_buffers;
#endif /* AIO_H */
diff --git a/src/include/storage/aio_internal.h b/src/include/storage/aio_internal.h
index 2580e34e43b..ca6cbde11a6 100644
--- a/src/include/storage/aio_internal.h
+++ b/src/include/storage/aio_internal.h
@@ -122,6 +122,12 @@ struct PgAioHandle
/* raw result of the IO operation */
int32 result;
+ /*
+ * List of bounce_buffers owned by IO. It would suffice to use an index
+ * based linked list here.
+ */
+ slist_head bounce_buffers;
+
/**
* In which list the handle is registered, depends on the state:
* - IDLE, in per-backend list
@@ -177,11 +183,24 @@ struct PgAioHandle
};
+/* typedef is in aio_types.h */
+struct PgAioBounceBuffer
+{
+ slist_node node;
+ struct ResourceOwnerData *resowner;
+ dlist_node resowner_node;
+ char *buffer;
+};
+
+
typedef struct PgAioBackend
{
/* index into PgAioCtl->io_handles */
uint32 io_handle_off;
+ /* index into PgAioCtl->bounce_buffers */
+ uint32 bounce_buffers_off;
+
/* IO Handles that currently are not used */
dclist_head idle_ios;
@@ -212,6 +231,12 @@ typedef struct PgAioBackend
* IOs being appended at the end.
*/
dclist_head in_flight_ios;
+
+ /* Bounce Buffers that currently are not used */
+ slist_head idle_bbs;
+
+ /* see handed_out_io */
+ PgAioBounceBuffer *handed_out_bb;
} PgAioBackend;
@@ -239,6 +264,15 @@ typedef struct PgAioCtl
uint32 io_handle_count;
PgAioHandle *io_handles;
+
+ /*
+ * To perform AIO on buffers that are not located in shared memory (either
+ * because they are not in shared memory or because we need to operate on
+ * a copy, as e.g. the case for writes when checksums are in use)
+ */
+ uint32 bounce_buffers_count;
+ PgAioBounceBuffer *bounce_buffers;
+ char *bounce_buffers_data;
} PgAioCtl;
diff --git a/src/include/storage/aio_types.h b/src/include/storage/aio_types.h
index b0b9a2a5c97..09968e17855 100644
--- a/src/include/storage/aio_types.h
+++ b/src/include/storage/aio_types.h
@@ -118,4 +118,6 @@ typedef struct PgAioReturn
} PgAioReturn;
+typedef struct PgAioBounceBuffer PgAioBounceBuffer;
+
#endif /* AIO_TYPES_H */
diff --git a/src/include/utils/resowner.h b/src/include/utils/resowner.h
index aede4bfc820..7e2ec224169 100644
--- a/src/include/utils/resowner.h
+++ b/src/include/utils/resowner.h
@@ -168,5 +168,7 @@ extern void ResourceOwnerForgetLock(ResourceOwner owner, struct LOCALLOCK *local
struct dlist_node;
extern void ResourceOwnerRememberAioHandle(ResourceOwner owner, struct dlist_node *ioh_node);
extern void ResourceOwnerForgetAioHandle(ResourceOwner owner, struct dlist_node *ioh_node);
+extern void ResourceOwnerRememberAioBounceBuffer(ResourceOwner owner, struct dlist_node *bb_node);
+extern void ResourceOwnerForgetAioBounceBuffer(ResourceOwner owner, struct dlist_node *bb_node);
#endif /* RESOWNER_H */
diff --git a/src/backend/storage/aio/README.md b/src/backend/storage/aio/README.md
index b7dfb80b4b2..bb0bfc71ea6 100644
--- a/src/backend/storage/aio/README.md
+++ b/src/backend/storage/aio/README.md
@@ -405,6 +405,33 @@ shared memory no less!), completion callbacks instead have to encode errors in
a more compact format that can be converted into an error message.
+### AIO Bounce Buffers
+
+For some uses of AIO there is no convenient memory location as the source /
+destination of an AIO. E.g. when data checksums are enabled, writes from
+shared buffers currently cannot be done directly from shared buffers, as a
+shared buffer lock still allows some modification, e.g., for hint bits(see
+`FlushBuffer()`). If the write were done in-place, such modifications can
+cause the checksum to fail.
+
+For synchronous IO this is solved by copying the buffer to separate memory
+before computing the checksum and using that copy as the source buffer for the
+AIO.
+
+However, for AIO that is not a workable solution:
+- Instead of a single buffer many buffers are required, as many IOs might be
+ in flight
+- When using the [worker method](#worker), the source/target of IO needs to be
+ in shared memory, otherwise the workers won't be able to access the memory.
+
+The AIO subsystem addresses this by providing a limited number of bounce
+buffers that can be used as the source / target for IO. A bounce buffer be
+acquired with `pgaio_bounce_buffer_get()` and multiple bounce buffers can be
+associated with an AIO Handle with `pgaio_io_assoc_bounce_buffer()`.
+
+Bounce buffers are automatically released when the IO completes.
+
+
## Helpers
Using the low-level AIO API introduces too much complexity to do so all over
diff --git a/src/backend/storage/aio/aio.c b/src/backend/storage/aio/aio.c
index f31e52f7fe0..a9c351eb0dc 100644
--- a/src/backend/storage/aio/aio.c
+++ b/src/backend/storage/aio/aio.c
@@ -62,6 +62,8 @@ static PgAioHandle *pgaio_io_from_wref(PgAioWaitRef *iow, uint64 *ref_generation
static const char *pgaio_io_state_get_name(PgAioHandleState s);
static void pgaio_io_wait(PgAioHandle *ioh, uint64 ref_generation);
+static void pgaio_bounce_buffer_wait_for_free(void);
+
/* Options for io_method. */
const struct config_enum_entry io_method_options[] = {
@@ -76,6 +78,7 @@ const struct config_enum_entry io_method_options[] = {
/* GUCs */
int io_method = DEFAULT_IO_METHOD;
int io_max_concurrency = -1;
+int io_bounce_buffers = -1;
/* global control for AIO */
PgAioCtl *pgaio_ctl;
@@ -663,6 +666,21 @@ pgaio_io_reclaim(PgAioHandle *ioh)
}
}
+ /* reclaim all associated bounce buffers */
+ if (!slist_is_empty(&ioh->bounce_buffers))
+ {
+ slist_mutable_iter it;
+
+ slist_foreach_modify(it, &ioh->bounce_buffers)
+ {
+ PgAioBounceBuffer *bb = slist_container(PgAioBounceBuffer, node, it.cur);
+
+ slist_delete_current(&it);
+
+ slist_push_head(&pgaio_my_backend->idle_bbs, &bb->node);
+ }
+ }
+
if (ioh->resowner)
{
ResourceOwnerForgetAioHandle(ioh->resowner, &ioh->resowner_node);
@@ -1044,6 +1062,166 @@ pgaio_submit_staged(void)
+/* --------------------------------------------------------------------------------
+ * Functions primarily related to PgAioBounceBuffer
+ * --------------------------------------------------------------------------------
+ */
+
+PgAioBounceBuffer *
+pgaio_bounce_buffer_get(void)
+{
+ PgAioBounceBuffer *bb = NULL;
+ slist_node *node;
+
+ if (pgaio_my_backend->handed_out_bb != NULL)
+ elog(ERROR, "can only hand out one BB");
+
+ /*
+ * XXX: It probably is not a good idea to have bounce buffers be per
+ * backend, that's a fair bit of memory.
+ */
+ if (slist_is_empty(&pgaio_my_backend->idle_bbs))
+ {
+ pgaio_bounce_buffer_wait_for_free();
+ }
+
+ node = slist_pop_head_node(&pgaio_my_backend->idle_bbs);
+ bb = slist_container(PgAioBounceBuffer, node, node);
+
+ pgaio_my_backend->handed_out_bb = bb;
+
+ bb->resowner = CurrentResourceOwner;
+ ResourceOwnerRememberAioBounceBuffer(bb->resowner, &bb->resowner_node);
+
+ return bb;
+}
+
+void
+pgaio_io_assoc_bounce_buffer(PgAioHandle *ioh, PgAioBounceBuffer *bb)
+{
+ if (pgaio_my_backend->handed_out_bb != bb)
+ elog(ERROR, "can only assign handed out BB");
+ pgaio_my_backend->handed_out_bb = NULL;
+
+ /*
+ * There can be many bounce buffers assigned in case of vectorized IOs.
+ */
+ slist_push_head(&ioh->bounce_buffers, &bb->node);
+
+ /* once associated with an IO, the IO has ownership */
+ ResourceOwnerForgetAioBounceBuffer(bb->resowner, &bb->resowner_node);
+ bb->resowner = NULL;
+}
+
+uint32
+pgaio_bounce_buffer_id(PgAioBounceBuffer *bb)
+{
+ return bb - pgaio_ctl->bounce_buffers;
+}
+
+void
+pgaio_bounce_buffer_release(PgAioBounceBuffer *bb)
+{
+ if (pgaio_my_backend->handed_out_bb != bb)
+ elog(ERROR, "can only release handed out BB");
+
+ slist_push_head(&pgaio_my_backend->idle_bbs, &bb->node);
+ pgaio_my_backend->handed_out_bb = NULL;
+
+ ResourceOwnerForgetAioBounceBuffer(bb->resowner, &bb->resowner_node);
+ bb->resowner = NULL;
+}
+
+void
+pgaio_bounce_buffer_release_resowner(dlist_node *bb_node, bool on_error)
+{
+ PgAioBounceBuffer *bb = dlist_container(PgAioBounceBuffer, resowner_node, bb_node);
+
+ Assert(bb->resowner);
+
+ if (!on_error)
+ elog(WARNING, "leaked AIO bounce buffer");
+
+ pgaio_bounce_buffer_release(bb);
+}
+
+char *
+pgaio_bounce_buffer_buffer(PgAioBounceBuffer *bb)
+{
+ return bb->buffer;
+}
+
+static void
+pgaio_bounce_buffer_wait_for_free(void)
+{
+ static uint32 lastpos = 0;
+
+ if (pgaio_my_backend->num_staged_ios > 0)
+ {
+ pgaio_debug(DEBUG2, "submitting %d, while acquiring free bb",
+ pgaio_my_backend->num_staged_ios);
+ pgaio_submit_staged();
+ }
+
+ for (uint32 i = lastpos; i < lastpos + io_max_concurrency; i++)
+ {
+ uint32 thisoff = pgaio_my_backend->io_handle_off + (i % io_max_concurrency);
+ PgAioHandle *ioh = &pgaio_ctl->io_handles[thisoff];
+
+ switch (ioh->state)
+ {
+ case PGAIO_HS_IDLE:
+ case PGAIO_HS_HANDED_OUT:
+ continue;
+ case PGAIO_HS_DEFINED: /* should have been submitted above */
+ case PGAIO_HS_STAGED:
+ elog(ERROR, "shouldn't get here with io:%d in state %d",
+ pgaio_io_get_id(ioh), ioh->state);
+ break;
+ case PGAIO_HS_COMPLETED_IO:
+ case PGAIO_HS_SUBMITTED:
+ if (!slist_is_empty(&ioh->bounce_buffers))
+ {
+ pgaio_debug_io(DEBUG2, ioh,
+ "waiting for IO to reclaim BB with %d in flight",
+ dclist_count(&pgaio_my_backend->in_flight_ios));
+
+ /* see comment in pgaio_io_wait_for_free() about raciness */
+ pgaio_io_wait(ioh, ioh->generation);
+
+ if (slist_is_empty(&pgaio_my_backend->idle_bbs))
+ elog(WARNING, "empty after wait");
+
+ if (!slist_is_empty(&pgaio_my_backend->idle_bbs))
+ {
+ lastpos = i;
+ return;
+ }
+ }
+ break;
+ case PGAIO_HS_COMPLETED_SHARED:
+ case PGAIO_HS_COMPLETED_LOCAL:
+ /* reclaim */
+ pgaio_io_reclaim(ioh);
+
+ if (!slist_is_empty(&pgaio_my_backend->idle_bbs))
+ {
+ lastpos = i;
+ return;
+ }
+ break;
+ }
+ }
+
+ /*
+ * The submission above could have caused the IO to complete at any time.
+ */
+ if (slist_is_empty(&pgaio_my_backend->idle_bbs))
+ elog(PANIC, "no more bbs");
+}
+
+
+
/* --------------------------------------------------------------------------------
* Other
* --------------------------------------------------------------------------------
diff --git a/src/backend/storage/aio/aio_init.c b/src/backend/storage/aio/aio_init.c
index d0d916ea5c9..c32d8e6cb3d 100644
--- a/src/backend/storage/aio/aio_init.c
+++ b/src/backend/storage/aio/aio_init.c
@@ -82,6 +82,32 @@ AioHandleDataShmemSize(void)
io_max_concurrency));
}
+static Size
+AioBounceBufferDescShmemSize(void)
+{
+ Size sz;
+
+ /* PgAioBounceBuffer itself */
+ sz = mul_size(sizeof(PgAioBounceBuffer),
+ mul_size(AioProcs(), io_bounce_buffers));
+
+ return sz;
+}
+
+static Size
+AioBounceBufferDataShmemSize(void)
+{
+ Size sz;
+
+ /* and the associated buffer */
+ sz = mul_size(BLCKSZ,
+ mul_size(io_bounce_buffers, AioProcs()));
+ /* memory for alignment */
+ sz += BLCKSZ;
+
+ return sz;
+}
+
/*
* Choose a suitable value for io_max_concurrency.
*
@@ -107,6 +133,33 @@ AioChooseMaxConcurrency(void)
return Min(max_proportional_pins, 64);
}
+/*
+ * Choose a suitable value for io_bounce_buffers.
+ *
+ * It's very unlikely to be useful to allocate more bounce buffers for each
+ * backend than the backend is allowed to pin. Additionally, bounce buffers
+ * currently are used for writes, it seems very uncommon for more than 10% of
+ * shared_buffers to be written out concurrently.
+ *
+ * XXX: This quickly can take up significant amounts of memory, the logic
+ * should probably fine tuned.
+ */
+static int
+AioChooseBounceBuffers(void)
+{
+ uint32 max_backends;
+ int max_proportional_pins;
+
+ /* Similar logic to LimitAdditionalPins() */
+ max_backends = MaxBackends + NUM_AUXILIARY_PROCS;
+ max_proportional_pins = (NBuffers / 10) / max_backends;
+
+ max_proportional_pins = Max(max_proportional_pins, 1);
+
+ /* apply upper limit */
+ return Min(max_proportional_pins, 256);
+}
+
Size
AioShmemSize(void)
{
@@ -131,11 +184,31 @@ AioShmemSize(void)
PGC_S_OVERRIDE);
}
+
+ /*
+ * If io_bounce_buffers is -1, we automatically choose a suitable value.
+ *
+ * See also comment above.
+ */
+ if (io_bounce_buffers == -1)
+ {
+ char buf[32];
+
+ snprintf(buf, sizeof(buf), "%d", AioChooseBounceBuffers());
+ SetConfigOption("io_bounce_buffers", buf, PGC_POSTMASTER,
+ PGC_S_DYNAMIC_DEFAULT);
+ if (io_bounce_buffers == -1) /* failed to apply it? */
+ SetConfigOption("io_bounce_buffers", buf, PGC_POSTMASTER,
+ PGC_S_OVERRIDE);
+ }
+
sz = add_size(sz, AioCtlShmemSize());
sz = add_size(sz, AioBackendShmemSize());
sz = add_size(sz, AioHandleShmemSize());
sz = add_size(sz, AioHandleIOVShmemSize());
sz = add_size(sz, AioHandleDataShmemSize());
+ sz = add_size(sz, AioBounceBufferDescShmemSize());
+ sz = add_size(sz, AioBounceBufferDataShmemSize());
if (pgaio_method_ops->shmem_size)
sz = add_size(sz, pgaio_method_ops->shmem_size());
@@ -150,6 +223,9 @@ AioShmemInit(void)
uint32 io_handle_off = 0;
uint32 iovec_off = 0;
uint32 per_backend_iovecs = io_max_concurrency * PG_IOV_MAX;
+ uint32 bounce_buffers_off = 0;
+ uint32 per_backend_bb = io_bounce_buffers;
+ char *bounce_buffers_data;
pgaio_ctl = (PgAioCtl *)
ShmemInitStruct("AioCtl", AioCtlShmemSize(), &found);
@@ -161,6 +237,7 @@ AioShmemInit(void)
pgaio_ctl->io_handle_count = AioProcs() * io_max_concurrency;
pgaio_ctl->iovec_count = AioProcs() * per_backend_iovecs;
+ pgaio_ctl->bounce_buffers_count = AioProcs() * per_backend_bb;
pgaio_ctl->backend_state = (PgAioBackend *)
ShmemInitStruct("AioBackend", AioBackendShmemSize(), &found);
@@ -173,6 +250,40 @@ AioShmemInit(void)
pgaio_ctl->handle_data = (uint64 *)
ShmemInitStruct("AioHandleData", AioHandleDataShmemSize(), &found);
+ pgaio_ctl->bounce_buffers = (PgAioBounceBuffer *)
+ ShmemInitStruct("AioBounceBufferDesc", AioBounceBufferDescShmemSize(),
+ &found);
+
+ bounce_buffers_data =
+ ShmemInitStruct("AioBounceBufferData", AioBounceBufferDataShmemSize(),
+ &found);
+ bounce_buffers_data =
+ (char *) TYPEALIGN(BLCKSZ, (uintptr_t) bounce_buffers_data);
+ pgaio_ctl->bounce_buffers_data = bounce_buffers_data;
+
+
+ /* Initialize IO handles. */
+ for (uint64 i = 0; i < pgaio_ctl->io_handle_count; i++)
+ {
+ PgAioHandle *ioh = &pgaio_ctl->io_handles[i];
+
+ ioh->op = PGAIO_OP_INVALID;
+ ioh->target = PGAIO_TID_INVALID;
+ ioh->state = PGAIO_HS_IDLE;
+
+ slist_init(&ioh->bounce_buffers);
+ }
+
+ /* Initialize Bounce Buffers. */
+ for (uint64 i = 0; i < pgaio_ctl->bounce_buffers_count; i++)
+ {
+ PgAioBounceBuffer *bb = &pgaio_ctl->bounce_buffers[i];
+
+ bb->buffer = bounce_buffers_data;
+ bounce_buffers_data += BLCKSZ;
+ }
+
+
for (int procno = 0; procno < AioProcs(); procno++)
{
PgAioBackend *bs = &pgaio_ctl->backend_state[procno];
@@ -180,9 +291,13 @@ AioShmemInit(void)
bs->io_handle_off = io_handle_off;
io_handle_off += io_max_concurrency;
+ bs->bounce_buffers_off = bounce_buffers_off;
+ bounce_buffers_off += per_backend_bb;
+
dclist_init(&bs->idle_ios);
memset(bs->staged_ios, 0, sizeof(PgAioHandle *) * PGAIO_SUBMIT_BATCH_SIZE);
dclist_init(&bs->in_flight_ios);
+ slist_init(&bs->idle_bbs);
/* initialize per-backend IOs */
for (int i = 0; i < io_max_concurrency; i++)
@@ -204,6 +319,14 @@ AioShmemInit(void)
dclist_push_tail(&bs->idle_ios, &ioh->node);
iovec_off += PG_IOV_MAX;
}
+
+ /* initialize per-backend bounce buffers */
+ for (int i = 0; i < per_backend_bb; i++)
+ {
+ PgAioBounceBuffer *bb = &pgaio_ctl->bounce_buffers[bs->bounce_buffers_off + i];
+
+ slist_push_head(&bs->idle_bbs, &bb->node);
+ }
}
out:
diff --git a/src/backend/utils/misc/guc_tables.c b/src/backend/utils/misc/guc_tables.c
index d4e63c75cb6..3a375b92652 100644
--- a/src/backend/utils/misc/guc_tables.c
+++ b/src/backend/utils/misc/guc_tables.c
@@ -3259,6 +3259,19 @@ struct config_int ConfigureNamesInt[] =
check_io_max_concurrency, NULL, NULL
},
+ {
+ {"io_bounce_buffers",
+ PGC_POSTMASTER,
+ RESOURCES_IO,
+ gettext_noop("Number of IO Bounce Buffers reserved for each backend."),
+ NULL,
+ GUC_UNIT_BLOCKS
+ },
+ &io_bounce_buffers,
+ -1, -1, 4096,
+ NULL, NULL, NULL
+ },
+
{
{"io_workers",
PGC_SIGHUP,
diff --git a/src/backend/utils/misc/postgresql.conf.sample b/src/backend/utils/misc/postgresql.conf.sample
index c9e9e850a99..24e1af803b0 100644
--- a/src/backend/utils/misc/postgresql.conf.sample
+++ b/src/backend/utils/misc/postgresql.conf.sample
@@ -209,6 +209,8 @@
# -1 sets based on shared_buffers
# (change requires restart)
#io_workers = 3 # 1-32;
+#io_bounce_buffers = -1 # -1 sets based on shared_buffers
+ # (change requires restart)
# - Worker Processes -
diff --git a/src/backend/utils/resowner/resowner.c b/src/backend/utils/resowner/resowner.c
index d39f3e1b655..81e7e27965a 100644
--- a/src/backend/utils/resowner/resowner.c
+++ b/src/backend/utils/resowner/resowner.c
@@ -159,10 +159,11 @@ struct ResourceOwnerData
LOCALLOCK *locks[MAX_RESOWNER_LOCKS]; /* list of owned locks */
/*
- * AIO handles need be registered in critical sections and therefore
- * cannot use the normal ResourceElem mechanism.
+ * AIO handles & bounce buffers need be registered in critical sections
+ * and therefore cannot use the normal ResourceElem mechanism.
*/
dlist_head aio_handles;
+ dlist_head aio_bounce_buffers;
};
@@ -434,6 +435,7 @@ ResourceOwnerCreate(ResourceOwner parent, const char *name)
}
dlist_init(&owner->aio_handles);
+ dlist_init(&owner->aio_bounce_buffers);
return owner;
}
@@ -742,6 +744,13 @@ ResourceOwnerReleaseInternal(ResourceOwner owner,
pgaio_io_release_resowner(node, !isCommit);
}
+
+ while (!dlist_is_empty(&owner->aio_bounce_buffers))
+ {
+ dlist_node *node = dlist_head_node(&owner->aio_bounce_buffers);
+
+ pgaio_bounce_buffer_release_resowner(node, !isCommit);
+ }
}
else if (phase == RESOURCE_RELEASE_LOCKS)
{
@@ -1111,3 +1120,15 @@ ResourceOwnerForgetAioHandle(ResourceOwner owner, struct dlist_node *ioh_node)
{
dlist_delete_from(&owner->aio_handles, ioh_node);
}
+
+void
+ResourceOwnerRememberAioBounceBuffer(ResourceOwner owner, struct dlist_node *ioh_node)
+{
+ dlist_push_tail(&owner->aio_bounce_buffers, ioh_node);
+}
+
+void
+ResourceOwnerForgetAioBounceBuffer(ResourceOwner owner, struct dlist_node *ioh_node)
+{
+ dlist_delete_from(&owner->aio_bounce_buffers, ioh_node);
+}
diff --git a/src/test/modules/test_aio/test_aio--1.0.sql b/src/test/modules/test_aio/test_aio--1.0.sql
index 3d4e9bb070f..16f248619fc 100644
--- a/src/test/modules/test_aio/test_aio--1.0.sql
+++ b/src/test/modules/test_aio/test_aio--1.0.sql
@@ -76,6 +76,27 @@ RETURNS pg_catalog.void STRICT
AS 'MODULE_PATHNAME' LANGUAGE C;
+CREATE FUNCTION bb_get_and_error()
+RETURNS pg_catalog.void STRICT
+AS 'MODULE_PATHNAME' LANGUAGE C;
+
+CREATE FUNCTION bb_get_twice()
+RETURNS pg_catalog.void STRICT
+AS 'MODULE_PATHNAME' LANGUAGE C;
+
+CREATE FUNCTION bb_get()
+RETURNS pg_catalog.void STRICT
+AS 'MODULE_PATHNAME' LANGUAGE C;
+
+CREATE FUNCTION bb_get_release()
+RETURNS pg_catalog.void STRICT
+AS 'MODULE_PATHNAME' LANGUAGE C;
+
+CREATE FUNCTION bb_release_last()
+RETURNS pg_catalog.void STRICT
+AS 'MODULE_PATHNAME' LANGUAGE C;
+
+
/*
* Injection point related functions
diff --git a/src/test/modules/test_aio/test_aio.c b/src/test/modules/test_aio/test_aio.c
index 81b4d732206..60c01b251fe 100644
--- a/src/test/modules/test_aio/test_aio.c
+++ b/src/test/modules/test_aio/test_aio.c
@@ -53,6 +53,7 @@ static shmem_startup_hook_type prev_shmem_startup_hook = NULL;
static PgAioHandle *last_handle;
+static PgAioBounceBuffer *last_bb;
@@ -525,6 +526,60 @@ batch_end(PG_FUNCTION_ARGS)
PG_RETURN_VOID();
}
+PG_FUNCTION_INFO_V1(bb_get);
+Datum
+bb_get(PG_FUNCTION_ARGS)
+{
+ last_bb = pgaio_bounce_buffer_get();
+
+ PG_RETURN_VOID();
+}
+
+PG_FUNCTION_INFO_V1(bb_release_last);
+Datum
+bb_release_last(PG_FUNCTION_ARGS)
+{
+ if (!last_bb)
+ elog(ERROR, "no bb");
+
+ pgaio_bounce_buffer_release(last_bb);
+
+ PG_RETURN_VOID();
+}
+
+PG_FUNCTION_INFO_V1(bb_get_and_error);
+Datum
+bb_get_and_error(PG_FUNCTION_ARGS)
+{
+ pgaio_bounce_buffer_get();
+
+ elog(ERROR, "as you command");
+ PG_RETURN_VOID();
+}
+
+PG_FUNCTION_INFO_V1(bb_get_twice);
+Datum
+bb_get_twice(PG_FUNCTION_ARGS)
+{
+ pgaio_bounce_buffer_get();
+ pgaio_bounce_buffer_get();
+
+ PG_RETURN_VOID();
+}
+
+
+PG_FUNCTION_INFO_V1(bb_get_release);
+Datum
+bb_get_release(PG_FUNCTION_ARGS)
+{
+ PgAioBounceBuffer *bb;
+
+ bb = pgaio_bounce_buffer_get();
+ pgaio_bounce_buffer_release(bb);
+
+ PG_RETURN_VOID();
+}
+
#ifdef USE_INJECTION_POINTS
extern PGDLLEXPORT void inj_io_short_read(const char *name, const void *private_data);
extern PGDLLEXPORT void inj_io_reopen(const char *name, const void *private_data);
diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list
index 7b9affbeb80..596b9a56265 100644
--- a/src/tools/pgindent/typedefs.list
+++ b/src/tools/pgindent/typedefs.list
@@ -2132,6 +2132,7 @@ PermutationStep
PermutationStepBlocker
PermutationStepBlockerType
PgAioBackend
+PgAioBounceBuffer
PgAioCtl
PgAioHandle
PgAioHandleCallbackID
--
2.48.1.76.g4e746b1a31.dirty
[text/x-diff] v2.7-0029-bufmgr-Implement-AIO-write-support.patch (9.8K, ../../4b3f32ug3cayekysqlgspz2qjmeb7lca3gvazayglxr2m3d4dv@il33accgsji7/30-v2.7-0029-bufmgr-Implement-AIO-write-support.patch)
download | inline diff:
From d870435980b1bacb429af7480ab45d2339f8d331 Mon Sep 17 00:00:00 2001
From: Andres Freund <andres@anarazel.de>
Date: Tue, 11 Mar 2025 11:33:13 -0400
Subject: [PATCH v2.7 29/35] bufmgr: Implement AIO write support
As of this commit there are no users of these AIO facilities, that'll come in
later commits.
Author:
Reviewed-By:
Discussion: https://postgr.es/m/
Backpatch:
---
src/include/storage/aio.h | 2 +
src/include/storage/bufmgr.h | 2 +
src/backend/storage/aio/aio_callback.c | 2 +
src/backend/storage/buffer/bufmgr.c | 180 +++++++++++++++++++++++++
4 files changed, 186 insertions(+)
diff --git a/src/include/storage/aio.h b/src/include/storage/aio.h
index 3047dc77ce1..192abc5a712 100644
--- a/src/include/storage/aio.h
+++ b/src/include/storage/aio.h
@@ -186,8 +186,10 @@ typedef enum PgAioHandleCallbackID
PGAIO_HCB_MD_WRITEV,
PGAIO_HCB_SHARED_BUFFER_READV,
+ PGAIO_HCB_SHARED_BUFFER_WRITEV,
PGAIO_HCB_LOCAL_BUFFER_READV,
+ PGAIO_HCB_LOCAL_BUFFER_WRITEV,
} PgAioHandleCallbackID;
diff --git a/src/include/storage/bufmgr.h b/src/include/storage/bufmgr.h
index cd921bdad49..f59d29113cc 100644
--- a/src/include/storage/bufmgr.h
+++ b/src/include/storage/bufmgr.h
@@ -175,7 +175,9 @@ extern PGDLLIMPORT int backend_flush_after;
extern PGDLLIMPORT int bgwriter_flush_after;
extern const PgAioHandleCallbacks aio_shared_buffer_readv_cb;
+extern const PgAioHandleCallbacks aio_shared_buffer_writev_cb;
extern const PgAioHandleCallbacks aio_local_buffer_readv_cb;
+extern const PgAioHandleCallbacks aio_local_buffer_writev_cb;
/* in buf_init.c */
extern PGDLLIMPORT char *BufferBlocks;
diff --git a/src/backend/storage/aio/aio_callback.c b/src/backend/storage/aio/aio_callback.c
index fb6ac058a09..7162f722e3c 100644
--- a/src/backend/storage/aio/aio_callback.c
+++ b/src/backend/storage/aio/aio_callback.c
@@ -44,8 +44,10 @@ static const PgAioHandleCallbacksEntry aio_handle_cbs[] = {
CALLBACK_ENTRY(PGAIO_HCB_MD_WRITEV, aio_md_writev_cb),
CALLBACK_ENTRY(PGAIO_HCB_SHARED_BUFFER_READV, aio_shared_buffer_readv_cb),
+ CALLBACK_ENTRY(PGAIO_HCB_SHARED_BUFFER_WRITEV, aio_shared_buffer_writev_cb),
CALLBACK_ENTRY(PGAIO_HCB_LOCAL_BUFFER_READV, aio_local_buffer_readv_cb),
+ CALLBACK_ENTRY(PGAIO_HCB_LOCAL_BUFFER_WRITEV, aio_local_buffer_writev_cb),
#undef CALLBACK_ENTRY
};
diff --git a/src/backend/storage/buffer/bufmgr.c b/src/backend/storage/buffer/bufmgr.c
index 55187eefa5d..19f62900259 100644
--- a/src/backend/storage/buffer/bufmgr.c
+++ b/src/backend/storage/buffer/bufmgr.c
@@ -5357,7 +5357,15 @@ LockBuffer(Buffer buffer, int mode)
else if (mode == BUFFER_LOCK_SHARE)
LWLockAcquire(BufferDescriptorGetContentLock(buf), LW_SHARED);
else if (mode == BUFFER_LOCK_EXCLUSIVE)
+ {
+ /*
+ * FIXME: Wait for AIO writes, otherwise there would be a risk of
+ * deadlock. This isn't entirely trivial to do in a race-free way, IO
+ * could be started between us checking whether there is IO and
+ * enqueueing ourselves for the lock.
+ */
LWLockAcquire(BufferDescriptorGetContentLock(buf), LW_EXCLUSIVE);
+ }
else
elog(ERROR, "unrecognized buffer lock mode: %d", mode);
}
@@ -5372,6 +5380,19 @@ ConditionalLockBuffer(Buffer buffer)
{
BufferDesc *buf;
+ /*
+ * FIXME: Wait for AIO writes. Some code does not deal well
+ * ConditionalLockBuffer() continuously failing, e.g.
+ * spginsert()->spgdoinsert() ends up busy-looping (easily reproducible by
+ * just making this function always fail and running the regression
+ * tests). While that code could be fixed, it'd be hard to find all
+ * problematic places.
+ *
+ * It would be OK to wait for the IO as waiting for IO completion does not
+ * need to wait for any locks that could lead to an undetected deadlock or
+ * such.
+ */
+
Assert(BufferIsPinned(buffer));
if (BufferIsLocal(buffer))
return true; /* act as though we got it */
@@ -5444,6 +5465,11 @@ LockBufferForCleanup(Buffer buffer)
{
uint32 buf_state;
+ /*
+ * FIXME: LockBuffer()'s handling of in-progress writes (once
+ * implemented) should suffice to deal with deadlock risk.
+ */
+
/* Try to acquire lock */
LockBuffer(buffer, BUFFER_LOCK_EXCLUSIVE);
buf_state = LockBufHdr(bufHdr);
@@ -5591,6 +5617,14 @@ ConditionalLockBufferForCleanup(Buffer buffer)
Assert(BufferIsValid(buffer));
+ /*
+ * FIXME: Should wait for IO for the same reason as in
+ * ConditionalLockBuffer(). Needs to happen before the
+ * ConditionalLockBuffer() call below, as we'd never reach the
+ * ConditionalLockBuffer() call due the buffer pin held for the duration
+ * of the IO.
+ */
+
if (BufferIsLocal(buffer))
{
refcount = LocalRefCount[-buffer - 1];
@@ -6652,12 +6686,131 @@ buffer_readv_report(PgAioResult result, const PgAioTargetData *target_data, int
);
}
+/*
+ * Helper for AIO writev completion callbacks, supporting both shared and temp
+ * buffers. Gets called once for each buffer in a multi-page write.
+ */
+static pg_attribute_always_inline PgAioResult
+buffer_writev_complete_one(uint8 buf_off, Buffer buffer, uint8 flags,
+ bool failed, bool is_temp)
+{
+ BufferDesc *buf_hdr = is_temp ?
+ GetLocalBufferDescriptor(-buffer - 1)
+ : GetBufferDescriptor(buffer - 1);
+ PgAioResult result;
+ bool clear_dirty;
+ uint32 set_flag_bits;
+
+#ifdef USE_ASSERT_CHECKING
+ {
+ uint32 buf_state = pg_atomic_read_u32(&buf_hdr->state);
+
+ Assert(buf_state & BM_VALID);
+ Assert(buf_state & BM_TAG_VALID);
+ /* temp buffers don't use BM_IO_IN_PROGRESS */
+ if (!is_temp)
+ Assert(buf_state & BM_IO_IN_PROGRESS);
+ Assert(buf_state & BM_DIRTY);
+ }
+#endif
+
+ result.status = ARS_OK;
+
+ clear_dirty = failed ? false : true;
+ set_flag_bits = failed ? BM_IO_ERROR : 0;
+
+ if (is_temp)
+ TerminateLocalBufferIO(buf_hdr, clear_dirty, set_flag_bits, false);
+ else
+ TerminateBufferIO(buf_hdr, clear_dirty, set_flag_bits, false, false);
+
+ /*
+ * The initiator of IO is not managing the lock (i.e. we called
+ * LWLockDisown()), we are.
+ */
+ if (!is_temp)
+ LWLockReleaseDisowned(BufferDescriptorGetContentLock(buf_hdr),
+ LW_SHARED);
+
+ /* FIXME: tracepoint */
+
+ return result;
+}
+
+/*
+ * Perform completion handling of a single AIO write. This write may cover
+ * multiple blocks / buffers.
+ *
+ * Shared between shared and local buffers, to reduce code duplication.
+ */
+static pg_attribute_always_inline PgAioResult
+buffer_writev_complete(PgAioHandle *ioh, PgAioResult prior_result,
+ uint8 cb_data, bool is_temp)
+{
+ PgAioResult result = prior_result;
+ PgAioTargetData *td = pgaio_io_get_target_data(ioh);
+ uint64 *io_data;
+ uint8 handle_data_len;
+
+ if (is_temp)
+ {
+ Assert(td->smgr.is_temp);
+ Assert(pgaio_io_get_owner(ioh) == MyProcNumber);
+ }
+ else
+ Assert(!td->smgr.is_temp);
+
+ /*
+ * Iterate over all the buffers affected by this IO and call appropriate
+ * per-buffer completion function for each buffer.
+ */
+ io_data = pgaio_io_get_handle_data(ioh, &handle_data_len);
+ for (uint8 buf_off = 0; buf_off < handle_data_len; buf_off++)
+ {
+ Buffer buf = io_data[buf_off];
+ PgAioResult buf_result;
+ bool failed;
+
+ Assert(BufferIsValid(buf));
+
+ /*
+ * If the entire failed on a lower-level, each buffer needs to be
+ * marked as failed. In case of a partial read, some buffers may be
+ * ok.
+ */
+ failed =
+ prior_result.status == ARS_ERROR
+ || prior_result.result <= buf_off;
+
+ buf_result = buffer_writev_complete_one(buf_off, buf, cb_data, failed,
+ is_temp);
+
+ /*
+ * If there wasn't any prior error and the IO for this page failed in
+ * some form, set the whole IO's to the page's result.
+ */
+ if (result.status != ARS_ERROR && buf_result.status != ARS_OK)
+ {
+ result = buf_result;
+ pgaio_result_report(result, td, LOG);
+ }
+ }
+
+ return result;
+}
+
static void
shared_buffer_readv_stage(PgAioHandle *ioh, uint8 cb_data)
{
buffer_stage_common(ioh, false, false);
}
+static void
+shared_buffer_writev_stage(PgAioHandle *ioh, uint8 cb_data)
+{
+ buffer_stage_common(ioh, true, false);
+}
+
static PgAioResult
shared_buffer_readv_complete(PgAioHandle *ioh, PgAioResult prior_result,
uint8 cb_data)
@@ -6665,6 +6818,13 @@ shared_buffer_readv_complete(PgAioHandle *ioh, PgAioResult prior_result,
return buffer_readv_complete(ioh, prior_result, cb_data, false);
}
+static PgAioResult
+shared_buffer_writev_complete(PgAioHandle *ioh, PgAioResult prior_result,
+ uint8 cb_data)
+{
+ return buffer_writev_complete(ioh, prior_result, cb_data, false);
+}
+
static void
local_buffer_readv_stage(PgAioHandle *ioh, uint8 cb_data)
{
@@ -6678,6 +6838,17 @@ local_buffer_readv_complete(PgAioHandle *ioh, PgAioResult prior_result,
return buffer_readv_complete(ioh, prior_result, cb_data, true);
}
+static void
+local_buffer_writev_stage(PgAioHandle *ioh, uint8 cb_data)
+{
+ /*
+ * Currently this is unreachable as the only write support is for
+ * checkpointer / bgwriter, which don't deal with local buffers.
+ */
+ elog(ERROR, "should be unreachable");
+}
+
+
/* readv callback is is passed READ_BUFFERS_* flags as callback data */
const PgAioHandleCallbacks aio_shared_buffer_readv_cb = {
.stage = shared_buffer_readv_stage,
@@ -6685,6 +6856,11 @@ const PgAioHandleCallbacks aio_shared_buffer_readv_cb = {
.report = buffer_readv_report,
};
+const PgAioHandleCallbacks aio_shared_buffer_writev_cb = {
+ .stage = shared_buffer_writev_stage,
+ .complete_shared = shared_buffer_writev_complete,
+};
+
/* readv callback is is passed READ_BUFFERS_* flags as callback data */
const PgAioHandleCallbacks aio_local_buffer_readv_cb = {
.stage = local_buffer_readv_stage,
@@ -6698,3 +6874,7 @@ const PgAioHandleCallbacks aio_local_buffer_readv_cb = {
.complete_local = local_buffer_readv_complete,
.report = buffer_readv_report,
};
+
+const PgAioHandleCallbacks aio_local_buffer_writev_cb = {
+ .stage = local_buffer_writev_stage,
+};
--
2.48.1.76.g4e746b1a31.dirty
[text/x-diff] v2.7-0030-aio-Add-IO-queue-helper.patch (7.4K, ../../4b3f32ug3cayekysqlgspz2qjmeb7lca3gvazayglxr2m3d4dv@il33accgsji7/31-v2.7-0030-aio-Add-IO-queue-helper.patch)
download | inline diff:
From 0e0e0841a9c10030058669fde1311d741eace15a Mon Sep 17 00:00:00 2001
From: Andres Freund <andres@anarazel.de>
Date: Wed, 22 Jan 2025 13:44:50 -0500
Subject: [PATCH v2.7 30/35] aio: Add IO queue helper
This is likely never going to anywhere - Thomas Munro is working on something
more complete. But I needed a way to exercise aio for checkpointer / bgwriter.
---
src/include/storage/io_queue.h | 33 +++++
src/backend/storage/aio/Makefile | 1 +
src/backend/storage/aio/io_queue.c | 204 ++++++++++++++++++++++++++++
src/backend/storage/aio/meson.build | 1 +
src/tools/pgindent/typedefs.list | 2 +
5 files changed, 241 insertions(+)
create mode 100644 src/include/storage/io_queue.h
create mode 100644 src/backend/storage/aio/io_queue.c
diff --git a/src/include/storage/io_queue.h b/src/include/storage/io_queue.h
new file mode 100644
index 00000000000..92b1e9afe6f
--- /dev/null
+++ b/src/include/storage/io_queue.h
@@ -0,0 +1,33 @@
+/*-------------------------------------------------------------------------
+ *
+ * io_queue.h
+ * Mechanism for tracking many IOs
+ *
+ *
+ * Portions Copyright (c) 1996-2025, PostgreSQL Global Development Group
+ * Portions Copyright (c) 1994, Regents of the University of California
+ *
+ * src/include/storage/io_queue.h
+ *
+ *-------------------------------------------------------------------------
+ */
+#ifndef IO_QUEUE_H
+#define IO_QUEUE_H
+
+#include "storage/aio_types.h"
+
+struct IOQueue;
+typedef struct IOQueue IOQueue;
+
+struct PgAioWaitRef;
+
+extern IOQueue *io_queue_create(int depth, int flags);
+extern void io_queue_track(IOQueue *ioq, const PgAioWaitRef *iow);
+extern void io_queue_wait_one(IOQueue *ioq);
+extern void io_queue_wait_all(IOQueue *ioq);
+extern bool io_queue_is_empty(IOQueue *ioq);
+extern void io_queue_reserve(IOQueue *ioq);
+extern PgAioHandle *io_queue_acquire_io(IOQueue *ioq);
+extern void io_queue_free(IOQueue *ioq);
+
+#endif /* IO_QUEUE_H */
diff --git a/src/backend/storage/aio/Makefile b/src/backend/storage/aio/Makefile
index 3f2469cc399..86fa4276fda 100644
--- a/src/backend/storage/aio/Makefile
+++ b/src/backend/storage/aio/Makefile
@@ -15,6 +15,7 @@ OBJS = \
aio_init.o \
aio_io.o \
aio_target.o \
+ io_queue.o \
method_io_uring.o \
method_sync.o \
method_worker.o \
diff --git a/src/backend/storage/aio/io_queue.c b/src/backend/storage/aio/io_queue.c
new file mode 100644
index 00000000000..526aa1d5e06
--- /dev/null
+++ b/src/backend/storage/aio/io_queue.c
@@ -0,0 +1,204 @@
+/*-------------------------------------------------------------------------
+ *
+ * io_queue.c
+ * AIO - Mechanism for tracking many IOs
+ *
+ * Portions Copyright (c) 1996-2025, PostgreSQL Global Development Group
+ * Portions Copyright (c) 1994, Regents of the University of California
+ *
+ * IDENTIFICATION
+ * src/backend/storage/aio/io_queue.c
+ *
+ *-------------------------------------------------------------------------
+ */
+
+#include "postgres.h"
+
+#include "lib/ilist.h"
+#include "storage/aio.h"
+#include "storage/io_queue.h"
+#include "utils/resowner.h"
+
+
+
+typedef struct TrackedIO
+{
+ PgAioWaitRef iow;
+ dlist_node node;
+} TrackedIO;
+
+struct IOQueue
+{
+ int depth;
+ int unsubmitted;
+
+ bool has_reserved;
+
+ dclist_head idle;
+ dclist_head in_progress;
+
+ TrackedIO tracked_ios[FLEXIBLE_ARRAY_MEMBER];
+};
+
+
+IOQueue *
+io_queue_create(int depth, int flags)
+{
+ size_t sz;
+ IOQueue *ioq;
+
+ sz = offsetof(IOQueue, tracked_ios)
+ + sizeof(TrackedIO) * depth;
+
+ ioq = palloc0(sz);
+
+ ioq->depth = 0;
+
+ for (int i = 0; i < depth; i++)
+ {
+ TrackedIO *tio = &ioq->tracked_ios[i];
+
+ pgaio_wref_clear(&tio->iow);
+ dclist_push_tail(&ioq->idle, &tio->node);
+ }
+
+ return ioq;
+}
+
+void
+io_queue_wait_one(IOQueue *ioq)
+{
+ /* submit all pending IO before waiting */
+ pgaio_submit_staged();
+
+ while (!dclist_is_empty(&ioq->in_progress))
+ {
+ /* FIXME: Should we really pop here already? */
+ dlist_node *node = dclist_pop_head_node(&ioq->in_progress);
+ TrackedIO *tio = dclist_container(TrackedIO, node, node);
+
+ pgaio_wref_wait(&tio->iow);
+ dclist_push_head(&ioq->idle, &tio->node);
+ }
+}
+
+void
+io_queue_reserve(IOQueue *ioq)
+{
+ if (ioq->has_reserved)
+ return;
+
+ if (dclist_is_empty(&ioq->idle))
+ io_queue_wait_one(ioq);
+
+ Assert(!dclist_is_empty(&ioq->idle));
+
+ ioq->has_reserved = true;
+}
+
+PgAioHandle *
+io_queue_acquire_io(IOQueue *ioq)
+{
+ PgAioHandle *ioh;
+
+ io_queue_reserve(ioq);
+
+ Assert(!dclist_is_empty(&ioq->idle));
+
+ if (!io_queue_is_empty(ioq))
+ {
+ ioh = pgaio_io_acquire_nb(CurrentResourceOwner, NULL);
+ if (ioh == NULL)
+ {
+ /*
+ * Need to wait for all IOs, blocking might not be legal in the
+ * context.
+ *
+ * XXX: This doesn't make a whole lot of sense, we're also
+ * blocking here. What was I smoking when I wrote the above?
+ */
+ io_queue_wait_all(ioq);
+ ioh = pgaio_io_acquire(CurrentResourceOwner, NULL);
+ }
+ }
+ else
+ {
+ ioh = pgaio_io_acquire(CurrentResourceOwner, NULL);
+ }
+
+ return ioh;
+}
+
+void
+io_queue_track(IOQueue *ioq, const struct PgAioWaitRef *iow)
+{
+ dlist_node *node;
+ TrackedIO *tio;
+
+ Assert(ioq->has_reserved);
+ ioq->has_reserved = false;
+
+ Assert(!dclist_is_empty(&ioq->idle));
+
+ node = dclist_pop_head_node(&ioq->idle);
+ tio = dclist_container(TrackedIO, node, node);
+
+ tio->iow = *iow;
+
+ dclist_push_tail(&ioq->in_progress, &tio->node);
+
+ ioq->unsubmitted++;
+
+ /*
+ * XXX: Should have some smarter logic here. We don't want to wait too
+ * long to submit, that'll mean we're more likely to block. But we also
+ * don't want to have the overhead of submitting every IO individually.
+ */
+ if (ioq->unsubmitted >= 4)
+ {
+ pgaio_submit_staged();
+ ioq->unsubmitted = 0;
+ }
+}
+
+void
+io_queue_wait_all(IOQueue *ioq)
+{
+ /* submit all pending IO before waiting */
+ pgaio_submit_staged();
+
+ while (!dclist_is_empty(&ioq->in_progress))
+ {
+ /* wait for the last IO to minimize unnecessary wakeups */
+ dlist_node *node = dclist_tail_node(&ioq->in_progress);
+ TrackedIO *tio = dclist_container(TrackedIO, node, node);
+
+ if (!pgaio_wref_check_done(&tio->iow))
+ {
+ ereport(DEBUG3,
+ errmsg("io_queue_wait_all for io:%d",
+ pgaio_wref_get_id(&tio->iow)),
+ errhidestmt(true),
+ errhidecontext(true));
+
+ pgaio_wref_wait(&tio->iow);
+ }
+
+ dclist_delete_from(&ioq->in_progress, &tio->node);
+ dclist_push_head(&ioq->idle, &tio->node);
+ }
+}
+
+bool
+io_queue_is_empty(IOQueue *ioq)
+{
+ return dclist_is_empty(&ioq->in_progress);
+}
+
+void
+io_queue_free(IOQueue *ioq)
+{
+ io_queue_wait_all(ioq);
+
+ pfree(ioq);
+}
diff --git a/src/backend/storage/aio/meson.build b/src/backend/storage/aio/meson.build
index da6df2d3654..270c4a64428 100644
--- a/src/backend/storage/aio/meson.build
+++ b/src/backend/storage/aio/meson.build
@@ -7,6 +7,7 @@ backend_sources += files(
'aio_init.c',
'aio_io.c',
'aio_target.c',
+ 'io_queue.c',
'method_io_uring.c',
'method_sync.c',
'method_worker.c',
diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list
index 596b9a56265..21b26196e0c 100644
--- a/src/tools/pgindent/typedefs.list
+++ b/src/tools/pgindent/typedefs.list
@@ -1191,6 +1191,7 @@ IOContext
IOFuncSelector
IOObject
IOOp
+IOQueue
IO_STATUS_BLOCK
IPCompareMethod
ITEM
@@ -3015,6 +3016,7 @@ TocEntry
TokenAuxData
TokenizedAuthLine
TrackItem
+TrackedIO
TransApplyAction
TransInvalidationInfo
TransState
--
2.48.1.76.g4e746b1a31.dirty
[text/x-diff] v2.7-0031-bufmgr-use-AIO-in-checkpointer-bgwriter.patch (31.1K, ../../4b3f32ug3cayekysqlgspz2qjmeb7lca3gvazayglxr2m3d4dv@il33accgsji7/32-v2.7-0031-bufmgr-use-AIO-in-checkpointer-bgwriter.patch)
download | inline diff:
From 4f652aeac3caad90aaede62e10c39793bed57576 Mon Sep 17 00:00:00 2001
From: Andres Freund <andres@anarazel.de>
Date: Wed, 22 Jan 2025 13:44:52 -0500
Subject: [PATCH v2.7 31/35] bufmgr: use AIO in checkpointer, bgwriter
This is far from ready - just included to be able to exercise AIO writes and
get some preliminary numbers. In all likelihood this will instead be based
on-top of work by Thomas Munro instead of the preceding commit.
---
src/include/postmaster/bgwriter.h | 3 +-
src/include/storage/buf_internals.h | 2 +
src/include/storage/bufmgr.h | 3 +-
src/include/storage/bufpage.h | 1 +
src/backend/postmaster/bgwriter.c | 19 +-
src/backend/postmaster/checkpointer.c | 11 +-
src/backend/storage/buffer/bufmgr.c | 588 +++++++++++++++++++++++---
src/backend/storage/page/bufpage.c | 10 +
src/tools/pgindent/typedefs.list | 1 +
9 files changed, 580 insertions(+), 58 deletions(-)
diff --git a/src/include/postmaster/bgwriter.h b/src/include/postmaster/bgwriter.h
index 4fd717169f0..4208d5f2c97 100644
--- a/src/include/postmaster/bgwriter.h
+++ b/src/include/postmaster/bgwriter.h
@@ -31,7 +31,8 @@ extern void BackgroundWriterMain(const void *startup_data, size_t startup_data_l
extern void CheckpointerMain(const void *startup_data, size_t startup_data_len) pg_attribute_noreturn();
extern void RequestCheckpoint(int flags);
-extern void CheckpointWriteDelay(int flags, double progress);
+struct IOQueue;
+extern void CheckpointWriteDelay(struct IOQueue *ioq, int flags, double progress);
extern bool ForwardSyncRequest(const FileTag *ftag, SyncRequestType type);
diff --git a/src/include/storage/buf_internals.h b/src/include/storage/buf_internals.h
index 4b1f52a9fc8..a3df3192d12 100644
--- a/src/include/storage/buf_internals.h
+++ b/src/include/storage/buf_internals.h
@@ -21,6 +21,8 @@
#include "storage/buf.h"
#include "storage/bufmgr.h"
#include "storage/condition_variable.h"
+#include "storage/io_queue.h"
+#include "storage/latch.h"
#include "storage/lwlock.h"
#include "storage/procnumber.h"
#include "storage/shmem.h"
diff --git a/src/include/storage/bufmgr.h b/src/include/storage/bufmgr.h
index f59d29113cc..98502bf69f9 100644
--- a/src/include/storage/bufmgr.h
+++ b/src/include/storage/bufmgr.h
@@ -298,7 +298,8 @@ extern bool ConditionalLockBufferForCleanup(Buffer buffer);
extern bool IsBufferCleanupOK(Buffer buffer);
extern bool HoldingBufferPinThatDelaysRecovery(void);
-extern bool BgBufferSync(struct WritebackContext *wb_context);
+struct IOQueue;
+extern bool BgBufferSync(struct IOQueue *ioq, struct WritebackContext *wb_context);
extern uint32 GetSoftPinLimit(void);
extern uint32 GetSoftLocalPinLimit(void);
diff --git a/src/include/storage/bufpage.h b/src/include/storage/bufpage.h
index 6646b6f6371..9c045e81857 100644
--- a/src/include/storage/bufpage.h
+++ b/src/include/storage/bufpage.h
@@ -509,5 +509,6 @@ extern bool PageIndexTupleOverwrite(Page page, OffsetNumber offnum,
Item newtup, Size newsize);
extern char *PageSetChecksumCopy(Page page, BlockNumber blkno);
extern void PageSetChecksumInplace(Page page, BlockNumber blkno);
+extern bool PageNeedsChecksumCopy(Page page);
#endif /* BUFPAGE_H */
diff --git a/src/backend/postmaster/bgwriter.c b/src/backend/postmaster/bgwriter.c
index 72f5acceec7..6e8801a39e3 100644
--- a/src/backend/postmaster/bgwriter.c
+++ b/src/backend/postmaster/bgwriter.c
@@ -38,11 +38,13 @@
#include "postmaster/auxprocess.h"
#include "postmaster/bgwriter.h"
#include "postmaster/interrupt.h"
+#include "storage/aio.h"
#include "storage/aio_subsys.h"
#include "storage/buf_internals.h"
#include "storage/bufmgr.h"
#include "storage/condition_variable.h"
#include "storage/fd.h"
+#include "storage/io_queue.h"
#include "storage/lwlock.h"
#include "storage/proc.h"
#include "storage/procsignal.h"
@@ -90,6 +92,7 @@ BackgroundWriterMain(const void *startup_data, size_t startup_data_len)
sigjmp_buf local_sigjmp_buf;
MemoryContext bgwriter_context;
bool prev_hibernate;
+ IOQueue *ioq;
WritebackContext wb_context;
Assert(startup_data_len == 0);
@@ -131,6 +134,7 @@ BackgroundWriterMain(const void *startup_data, size_t startup_data_len)
ALLOCSET_DEFAULT_SIZES);
MemoryContextSwitchTo(bgwriter_context);
+ ioq = io_queue_create(128, 0);
WritebackContextInit(&wb_context, &bgwriter_flush_after);
/*
@@ -228,12 +232,22 @@ BackgroundWriterMain(const void *startup_data, size_t startup_data_len)
/* Clear any already-pending wakeups */
ResetLatch(MyLatch);
+ /*
+ * FIXME: this is theoretically racy, but I didn't want to copy
+ * ProcessMainLoopInterrupts() remaining body here.
+ */
+ if (ShutdownRequestPending)
+ {
+ io_queue_wait_all(ioq);
+ io_queue_free(ioq);
+ }
+
ProcessMainLoopInterrupts();
/*
* Do one cycle of dirty-buffer writing.
*/
- can_hibernate = BgBufferSync(&wb_context);
+ can_hibernate = BgBufferSync(ioq, &wb_context);
/* Report pending statistics to the cumulative stats system */
pgstat_report_bgwriter();
@@ -250,6 +264,9 @@ BackgroundWriterMain(const void *startup_data, size_t startup_data_len)
smgrdestroyall();
}
+ /* finish IO before sleeping, to avoid blocking other backends */
+ io_queue_wait_all(ioq);
+
/*
* Log a new xl_running_xacts every now and then so replication can
* get into a consistent state faster (think of suboverflowed
diff --git a/src/backend/postmaster/checkpointer.c b/src/backend/postmaster/checkpointer.c
index fda91ffd1ce..904fe167eb4 100644
--- a/src/backend/postmaster/checkpointer.c
+++ b/src/backend/postmaster/checkpointer.c
@@ -49,10 +49,12 @@
#include "postmaster/bgwriter.h"
#include "postmaster/interrupt.h"
#include "replication/syncrep.h"
+#include "storage/aio.h"
#include "storage/aio_subsys.h"
#include "storage/bufmgr.h"
#include "storage/condition_variable.h"
#include "storage/fd.h"
+#include "storage/io_queue.h"
#include "storage/ipc.h"
#include "storage/lwlock.h"
#include "storage/pmsignal.h"
@@ -766,7 +768,7 @@ ImmediateCheckpointRequested(void)
* fraction between 0.0 meaning none, and 1.0 meaning all done.
*/
void
-CheckpointWriteDelay(int flags, double progress)
+CheckpointWriteDelay(IOQueue *ioq, int flags, double progress)
{
static int absorb_counter = WRITES_PER_ABSORB;
@@ -800,6 +802,13 @@ CheckpointWriteDelay(int flags, double progress)
/* Report interim statistics to the cumulative stats system */
pgstat_report_checkpointer();
+ /*
+ * Ensure all pending IO is submitted to avoid unnecessary delays for
+ * other processes.
+ */
+ io_queue_wait_all(ioq);
+
+
/*
* This sleep used to be connected to bgwriter_delay, typically 200ms.
* That resulted in more frequent wakeups if not much work to do.
diff --git a/src/backend/storage/buffer/bufmgr.c b/src/backend/storage/buffer/bufmgr.c
index 19f62900259..9483596e63a 100644
--- a/src/backend/storage/buffer/bufmgr.c
+++ b/src/backend/storage/buffer/bufmgr.c
@@ -52,6 +52,7 @@
#include "storage/buf_internals.h"
#include "storage/bufmgr.h"
#include "storage/fd.h"
+#include "storage/io_queue.h"
#include "storage/ipc.h"
#include "storage/lmgr.h"
#include "storage/proc.h"
@@ -76,6 +77,7 @@
/* Bits in SyncOneBuffer's return value */
#define BUF_WRITTEN 0x01
#define BUF_REUSABLE 0x02
+#define BUF_CANT_MERGE 0x04
#define RELS_BSEARCH_THRESHOLD 20
@@ -512,8 +514,6 @@ static void UnpinBuffer(BufferDesc *buf);
static void UnpinBufferNoOwner(BufferDesc *buf);
static void BufferSync(int flags);
static uint32 WaitBufHdrUnlocked(BufferDesc *buf);
-static int SyncOneBuffer(int buf_id, bool skip_recently_used,
- WritebackContext *wb_context);
static void WaitIO(BufferDesc *buf);
static void AbortBufferIO(Buffer buffer);
static void shared_buffer_write_error_callback(void *arg);
@@ -529,6 +529,7 @@ static bool AsyncReadBuffers(ReadBuffersOperation *operation,
static Buffer GetVictimBuffer(BufferAccessStrategy strategy, IOContext io_context);
static void FlushBuffer(BufferDesc *buf, SMgrRelation reln,
IOObject io_object, IOContext io_context);
+
static void FindAndDropRelationBuffers(RelFileLocator rlocator,
ForkNumber forkNum,
BlockNumber nForkBlock,
@@ -3148,6 +3149,57 @@ UnpinBufferNoOwner(BufferDesc *buf)
}
}
+typedef struct BuffersToWrite
+{
+ int nbuffers;
+ BufferTag start_at_tag;
+ uint32 max_combine;
+
+ XLogRecPtr max_lsn;
+
+ PgAioHandle *ioh;
+ PgAioWaitRef iow;
+
+ uint64 total_writes;
+
+ Buffer buffers[IOV_MAX];
+ PgAioBounceBuffer *bounce_buffers[IOV_MAX];
+ const void *data_ptrs[IOV_MAX];
+} BuffersToWrite;
+
+static int PrepareToWriteBuffer(BuffersToWrite *to_write, Buffer buf,
+ bool skip_recently_used,
+ IOQueue *ioq, WritebackContext *wb_context);
+
+static void WriteBuffers(BuffersToWrite *to_write,
+ IOQueue *ioq, WritebackContext *wb_context);
+
+static void
+BuffersToWriteInit(BuffersToWrite *to_write,
+ IOQueue *ioq, WritebackContext *wb_context)
+{
+ to_write->total_writes = 0;
+ to_write->nbuffers = 0;
+ to_write->ioh = NULL;
+ pgaio_wref_clear(&to_write->iow);
+ to_write->max_lsn = InvalidXLogRecPtr;
+
+ pgaio_enter_batchmode();
+}
+
+static void
+BuffersToWriteEnd(BuffersToWrite *to_write)
+{
+ if (to_write->ioh != NULL)
+ {
+ pgaio_io_release(to_write->ioh);
+ to_write->ioh = NULL;
+ }
+
+ pgaio_exit_batchmode();
+}
+
+
#define ST_SORT sort_checkpoint_bufferids
#define ST_ELEMENT_TYPE CkptSortItem
#define ST_COMPARE(a, b) ckpt_buforder_comparator(a, b)
@@ -3179,7 +3231,10 @@ BufferSync(int flags)
binaryheap *ts_heap;
int i;
int mask = BM_DIRTY;
+ IOQueue *ioq;
WritebackContext wb_context;
+ BuffersToWrite to_write;
+ int max_combine;
/*
* Unless this is a shutdown checkpoint or we have been explicitly told,
@@ -3241,7 +3296,9 @@ BufferSync(int flags)
if (num_to_scan == 0)
return; /* nothing to do */
+ ioq = io_queue_create(512, 0);
WritebackContextInit(&wb_context, &checkpoint_flush_after);
+ max_combine = Min(io_bounce_buffers, io_combine_limit);
TRACE_POSTGRESQL_BUFFER_SYNC_START(NBuffers, num_to_scan);
@@ -3349,48 +3406,91 @@ BufferSync(int flags)
*/
num_processed = 0;
num_written = 0;
+
+ BuffersToWriteInit(&to_write, ioq, &wb_context);
+
while (!binaryheap_empty(ts_heap))
{
BufferDesc *bufHdr = NULL;
CkptTsStatus *ts_stat = (CkptTsStatus *)
DatumGetPointer(binaryheap_first(ts_heap));
+ bool batch_continue = true;
- buf_id = CkptBufferIds[ts_stat->index].buf_id;
- Assert(buf_id != -1);
-
- bufHdr = GetBufferDescriptor(buf_id);
-
- num_processed++;
+ Assert(ts_stat->num_scanned <= ts_stat->num_to_scan);
/*
- * We don't need to acquire the lock here, because we're only looking
- * at a single bit. It's possible that someone else writes the buffer
- * and clears the flag right after we check, but that doesn't matter
- * since SyncOneBuffer will then do nothing. However, there is a
- * further race condition: it's conceivable that between the time we
- * examine the bit here and the time SyncOneBuffer acquires the lock,
- * someone else not only wrote the buffer but replaced it with another
- * page and dirtied it. In that improbable case, SyncOneBuffer will
- * write the buffer though we didn't need to. It doesn't seem worth
- * guarding against this, though.
+ * Collect a batch of buffers to write out from the current
+ * tablespace. That causes some imbalance between the tablespaces, but
+ * that's more than outweighed by the efficiency gain due to batching.
*/
- if (pg_atomic_read_u32(&bufHdr->state) & BM_CHECKPOINT_NEEDED)
+ while (batch_continue &&
+ to_write.nbuffers < max_combine &&
+ ts_stat->num_scanned < ts_stat->num_to_scan)
{
- if (SyncOneBuffer(buf_id, false, &wb_context) & BUF_WRITTEN)
+ buf_id = CkptBufferIds[ts_stat->index].buf_id;
+ Assert(buf_id != -1);
+
+ bufHdr = GetBufferDescriptor(buf_id);
+
+ num_processed++;
+
+ /*
+ * We don't need to acquire the lock here, because we're only
+ * looking at a single bit. It's possible that someone else writes
+ * the buffer and clears the flag right after we check, but that
+ * doesn't matter since SyncOneBuffer will then do nothing.
+ * However, there is a further race condition: it's conceivable
+ * that between the time we examine the bit here and the time
+ * SyncOneBuffer acquires the lock, someone else not only wrote
+ * the buffer but replaced it with another page and dirtied it. In
+ * that improbable case, SyncOneBuffer will write the buffer
+ * though we didn't need to. It doesn't seem worth guarding
+ * against this, though.
+ */
+ if (pg_atomic_read_u32(&bufHdr->state) & BM_CHECKPOINT_NEEDED)
{
- TRACE_POSTGRESQL_BUFFER_SYNC_WRITTEN(buf_id);
- PendingCheckpointerStats.buffers_written++;
- num_written++;
+ int result = PrepareToWriteBuffer(&to_write, buf_id + 1, false,
+ ioq, &wb_context);
+
+ if (result & BUF_CANT_MERGE)
+ {
+ Assert(to_write.nbuffers > 0);
+ WriteBuffers(&to_write, ioq, &wb_context);
+
+ result = PrepareToWriteBuffer(&to_write, buf_id + 1, false,
+ ioq, &wb_context);
+ Assert(result != BUF_CANT_MERGE);
+ }
+
+ if (result & BUF_WRITTEN)
+ {
+ TRACE_POSTGRESQL_BUFFER_SYNC_WRITTEN(buf_id);
+ PendingCheckpointerStats.buffers_written++;
+ num_written++;
+ }
+ else
+ {
+ batch_continue = false;
+ }
}
+ else
+ {
+ if (to_write.nbuffers > 0)
+ WriteBuffers(&to_write, ioq, &wb_context);
+ }
+
+ /*
+ * Measure progress independent of actually having to flush the
+ * buffer - otherwise writing become unbalanced.
+ */
+ ts_stat->progress += ts_stat->progress_slice;
+ ts_stat->num_scanned++;
+ ts_stat->index++;
}
- /*
- * Measure progress independent of actually having to flush the buffer
- * - otherwise writing become unbalanced.
- */
- ts_stat->progress += ts_stat->progress_slice;
- ts_stat->num_scanned++;
- ts_stat->index++;
+ if (to_write.nbuffers > 0)
+ WriteBuffers(&to_write, ioq, &wb_context);
+
/* Have all the buffers from the tablespace been processed? */
if (ts_stat->num_scanned == ts_stat->num_to_scan)
@@ -3408,15 +3508,23 @@ BufferSync(int flags)
*
* (This will check for barrier events even if it doesn't sleep.)
*/
- CheckpointWriteDelay(flags, (double) num_processed / num_to_scan);
+ CheckpointWriteDelay(ioq, flags, (double) num_processed / num_to_scan);
}
+ Assert(to_write.nbuffers == 0);
+ io_queue_wait_all(ioq);
+
/*
* Issue all pending flushes. Only checkpointer calls BufferSync(), so
* IOContext will always be IOCONTEXT_NORMAL.
*/
IssuePendingWritebacks(&wb_context, IOCONTEXT_NORMAL);
+ io_queue_wait_all(ioq); /* IssuePendingWritebacks might have added
+ * more */
+ io_queue_free(ioq);
+ BuffersToWriteEnd(&to_write);
+
pfree(per_ts_stat);
per_ts_stat = NULL;
binaryheap_free(ts_heap);
@@ -3442,7 +3550,7 @@ BufferSync(int flags)
* bgwriter_lru_maxpages to 0.)
*/
bool
-BgBufferSync(WritebackContext *wb_context)
+BgBufferSync(IOQueue *ioq, WritebackContext *wb_context)
{
/* info obtained from freelist.c */
int strategy_buf_id;
@@ -3485,6 +3593,9 @@ BgBufferSync(WritebackContext *wb_context)
long new_strategy_delta;
uint32 new_recent_alloc;
+ BuffersToWrite to_write;
+ int max_combine;
+
/*
* Find out where the freelist clock sweep currently is, and how many
* buffer allocations have happened since our last call.
@@ -3505,6 +3616,8 @@ BgBufferSync(WritebackContext *wb_context)
return true;
}
+ max_combine = Min(io_bounce_buffers, io_combine_limit);
+
/*
* Compute strategy_delta = how many buffers have been scanned by the
* clock sweep since last time. If first time through, assume none. Then
@@ -3661,11 +3774,25 @@ BgBufferSync(WritebackContext *wb_context)
num_written = 0;
reusable_buffers = reusable_buffers_est;
+ BuffersToWriteInit(&to_write, ioq, wb_context);
+
/* Execute the LRU scan */
while (num_to_scan > 0 && reusable_buffers < upcoming_alloc_est)
{
- int sync_state = SyncOneBuffer(next_to_clean, true,
- wb_context);
+ int sync_state;
+
+ sync_state = PrepareToWriteBuffer(&to_write, next_to_clean + 1,
+ true, ioq, wb_context);
+ if (sync_state & BUF_CANT_MERGE)
+ {
+ Assert(to_write.nbuffers > 0);
+
+ WriteBuffers(&to_write, ioq, wb_context);
+
+ sync_state = PrepareToWriteBuffer(&to_write, next_to_clean + 1,
+ true, ioq, wb_context);
+ Assert(sync_state != BUF_CANT_MERGE);
+ }
if (++next_to_clean >= NBuffers)
{
@@ -3676,6 +3803,13 @@ BgBufferSync(WritebackContext *wb_context)
if (sync_state & BUF_WRITTEN)
{
+ Assert(sync_state & BUF_REUSABLE);
+
+ if (to_write.nbuffers == max_combine)
+ {
+ WriteBuffers(&to_write, ioq, wb_context);
+ }
+
reusable_buffers++;
if (++num_written >= bgwriter_lru_maxpages)
{
@@ -3687,6 +3821,11 @@ BgBufferSync(WritebackContext *wb_context)
reusable_buffers++;
}
+ if (to_write.nbuffers > 0)
+ WriteBuffers(&to_write, ioq, wb_context);
+
+ BuffersToWriteEnd(&to_write);
+
PendingBgWriterStats.buf_written_clean += num_written;
#ifdef BGW_DEBUG
@@ -3725,8 +3864,66 @@ BgBufferSync(WritebackContext *wb_context)
return (bufs_to_lap == 0 && recent_alloc == 0);
}
+static inline bool
+BufferTagsSameRel(const BufferTag *tag1, const BufferTag *tag2)
+{
+ return (tag1->spcOid == tag2->spcOid) &&
+ (tag1->dbOid == tag2->dbOid) &&
+ (tag1->relNumber == tag2->relNumber) &&
+ (tag1->forkNum == tag2->forkNum)
+ ;
+}
+
+static bool
+CanMergeWrite(BuffersToWrite *to_write, BufferDesc *cur_buf_hdr)
+{
+ BlockNumber cur_block = cur_buf_hdr->tag.blockNum;
+
+ Assert(to_write->nbuffers > 0); /* can't merge with nothing */
+ Assert(to_write->start_at_tag.relNumber != InvalidOid);
+ Assert(to_write->start_at_tag.blockNum != InvalidBlockNumber);
+
+ Assert(to_write->ioh != NULL);
+
+ /*
+ * First check if the blocknumber is one that we could actually merge,
+ * that's cheaper than checking the tablespace/db/relnumber/fork match.
+ */
+ if (to_write->start_at_tag.blockNum + to_write->nbuffers != cur_block)
+ return false;
+
+ if (!BufferTagsSameRel(&to_write->start_at_tag, &cur_buf_hdr->tag))
+ return false;
+
+ /*
+ * Need to check with smgr how large a write we're allowed to make. To
+ * reduce the overhead of the smgr check, only inquire once, when
+ * processing the first to-be-merged buffer. That avoids the overhead in
+ * the common case of writing out buffers that definitely not mergeable.
+ */
+ if (to_write->nbuffers == 1)
+ {
+ SMgrRelation smgr;
+
+ smgr = smgropen(BufTagGetRelFileLocator(&to_write->start_at_tag), INVALID_PROC_NUMBER);
+
+ to_write->max_combine = smgrmaxcombine(smgr,
+ to_write->start_at_tag.forkNum,
+ to_write->start_at_tag.blockNum);
+ }
+ else
+ {
+ Assert(to_write->max_combine > 0);
+ }
+
+ if (to_write->start_at_tag.blockNum + to_write->max_combine <= cur_block)
+ return false;
+
+ return true;
+}
+
/*
- * SyncOneBuffer -- process a single buffer during syncing.
+ * PrepareToWriteBuffer -- process a single buffer during syncing.
*
* If skip_recently_used is true, we don't write currently-pinned buffers, nor
* buffers marked recently used, as these are not replacement candidates.
@@ -3735,22 +3932,50 @@ BgBufferSync(WritebackContext *wb_context)
* BUF_WRITTEN: we wrote the buffer.
* BUF_REUSABLE: buffer is available for replacement, ie, it has
* pin count 0 and usage count 0.
+ * BUF_CANT_MERGE: can't combine this write with prior writes, caller needs
+ * to issue those first
*
* (BUF_WRITTEN could be set in error if FlushBuffer finds the buffer clean
* after locking it, but we don't care all that much.)
*/
static int
-SyncOneBuffer(int buf_id, bool skip_recently_used, WritebackContext *wb_context)
+PrepareToWriteBuffer(BuffersToWrite *to_write, Buffer buf,
+ bool skip_recently_used,
+ IOQueue *ioq, WritebackContext *wb_context)
{
- BufferDesc *bufHdr = GetBufferDescriptor(buf_id);
+ BufferDesc *cur_buf_hdr = GetBufferDescriptor(buf - 1);
+ uint32 buf_state;
int result = 0;
- uint32 buf_state;
- BufferTag tag;
+ XLogRecPtr cur_buf_lsn;
+ LWLock *content_lock;
+ bool may_block;
+
+ /*
+ * Check if this buffer can be written out together with already prepared
+ * writes. We check before we have pinned the buffer, so the buffer can be
+ * written out and replaced between this check and us pinning the buffer -
+ * we'll recheck below. The reason for the pre-check is that we don't want
+ * to pin the buffer just to find out that we can't merge the IO.
+ */
+ if (to_write->nbuffers != 0)
+ {
+ if (!CanMergeWrite(to_write, cur_buf_hdr))
+ {
+ result |= BUF_CANT_MERGE;
+ return result;
+ }
+ }
+ else
+ {
+ to_write->start_at_tag = cur_buf_hdr->tag;
+ }
/* Make sure we can handle the pin */
ReservePrivateRefCountEntry();
ResourceOwnerEnlarge(CurrentResourceOwner);
+ /* XXX: Should also check if we are allowed to pin one more buffer */
+
/*
* Check whether buffer needs writing.
*
@@ -3760,7 +3985,7 @@ SyncOneBuffer(int buf_id, bool skip_recently_used, WritebackContext *wb_context)
* don't worry because our checkpoint.redo points before log record for
* upcoming changes and so we are not required to write such dirty buffer.
*/
- buf_state = LockBufHdr(bufHdr);
+ buf_state = LockBufHdr(cur_buf_hdr);
if (BUF_STATE_GET_REFCOUNT(buf_state) == 0 &&
BUF_STATE_GET_USAGECOUNT(buf_state) == 0)
@@ -3769,40 +3994,294 @@ SyncOneBuffer(int buf_id, bool skip_recently_used, WritebackContext *wb_context)
}
else if (skip_recently_used)
{
+#if 0
+ elog(LOG, "at block %d: skip recent with nbuffers %d",
+ cur_buf_hdr->tag.blockNum, to_write->nbuffers);
+#endif
/* Caller told us not to write recently-used buffers */
- UnlockBufHdr(bufHdr, buf_state);
+ UnlockBufHdr(cur_buf_hdr, buf_state);
return result;
}
if (!(buf_state & BM_VALID) || !(buf_state & BM_DIRTY))
{
/* It's clean, so nothing to do */
- UnlockBufHdr(bufHdr, buf_state);
+ UnlockBufHdr(cur_buf_hdr, buf_state);
return result;
}
+ /* pin the buffer, from now on its identity can't change anymore */
+ PinBuffer_Locked(cur_buf_hdr);
+
+ /*
+ * Acquire IO, if needed, now that it's likely that we'll need to write.
+ */
+ if (to_write->ioh == NULL)
+ {
+ /* otherwise we should already have acquired a handle */
+ Assert(to_write->nbuffers == 0);
+
+ to_write->ioh = io_queue_acquire_io(ioq);
+ pgaio_io_get_wref(to_write->ioh, &to_write->iow);
+ }
+
/*
- * Pin it, share-lock it, write it. (FlushBuffer will do nothing if the
- * buffer is clean by the time we've locked it.)
+ * If we are merging, check if the buffer's identity possibly changed
+ * while we hadn't yet pinned it.
+ *
+ * XXX: It might be worth checking if we still want to write the buffer
+ * out, e.g. it could have been replaced with a buffer that doesn't have
+ * BM_CHECKPOINT_NEEDED set.
*/
- PinBuffer_Locked(bufHdr);
- LWLockAcquire(BufferDescriptorGetContentLock(bufHdr), LW_SHARED);
+ if (to_write->nbuffers != 0)
+ {
+ if (!CanMergeWrite(to_write, cur_buf_hdr))
+ {
+ elog(LOG, "changed identity");
+ UnpinBuffer(cur_buf_hdr);
+
+ result |= BUF_CANT_MERGE;
+
+ return result;
+ }
+ }
+
+ may_block = to_write->nbuffers == 0
+ && !pgaio_have_staged()
+ && io_queue_is_empty(ioq)
+ ;
+ content_lock = BufferDescriptorGetContentLock(cur_buf_hdr);
+
+ if (!may_block)
+ {
+ if (LWLockConditionalAcquire(content_lock, LW_SHARED))
+ {
+ /* done */
+ }
+ else if (to_write->nbuffers == 0)
+ {
+ /*
+ * Need to wait for all prior IO to finish before blocking for
+ * lock acquisition, to avoid the risk a deadlock due to us
+ * waiting for another backend that is waiting for our unsubmitted
+ * IO to complete.
+ */
+ pgaio_submit_staged();
+ io_queue_wait_all(ioq);
+
+ elog(DEBUG2, "at block %u: can't block, nbuffers = 0",
+ cur_buf_hdr->tag.blockNum
+ );
+
+ may_block = to_write->nbuffers == 0
+ && !pgaio_have_staged()
+ && io_queue_is_empty(ioq)
+ ;
+ Assert(may_block);
+
+ LWLockAcquire(content_lock, LW_SHARED);
+ }
+ else
+ {
+ elog(DEBUG2, "at block %d: can't block nbuffers = %d",
+ cur_buf_hdr->tag.blockNum,
+ to_write->nbuffers);
- FlushBuffer(bufHdr, NULL, IOOBJECT_RELATION, IOCONTEXT_NORMAL);
+ UnpinBuffer(cur_buf_hdr);
+ result |= BUF_CANT_MERGE;
+ Assert(to_write->nbuffers > 0);
- LWLockRelease(BufferDescriptorGetContentLock(bufHdr));
+ return result;
+ }
+ }
+ else
+ {
+ LWLockAcquire(content_lock, LW_SHARED);
+ }
- tag = bufHdr->tag;
+ if (!may_block)
+ {
+ if (!StartBufferIO(cur_buf_hdr, false, !may_block))
+ {
+ pgaio_submit_staged();
+ io_queue_wait_all(ioq);
- UnpinBuffer(bufHdr);
+ may_block = io_queue_is_empty(ioq) && to_write->nbuffers == 0 && !pgaio_have_staged();
+
+ if (!StartBufferIO(cur_buf_hdr, false, !may_block))
+ {
+ elog(DEBUG2, "at block %d: non-waitable StartBufferIO returns false, %d",
+ cur_buf_hdr->tag.blockNum,
+ may_block);
+
+ /*
+ * FIXME: can't tell whether this is because the buffer has
+ * been cleaned
+ */
+ if (!may_block)
+ {
+ result |= BUF_CANT_MERGE;
+ Assert(to_write->nbuffers > 0);
+ }
+ LWLockRelease(content_lock);
+ UnpinBuffer(cur_buf_hdr);
+
+ return result;
+ }
+ }
+ }
+ else
+ {
+ if (!StartBufferIO(cur_buf_hdr, false, false))
+ {
+ elog(DEBUG2, "waitable StartBufferIO returns false");
+ LWLockRelease(content_lock);
+ UnpinBuffer(cur_buf_hdr);
+
+ /*
+ * FIXME: Historically we returned BUF_WRITTEN in this case, which
+ * seems wrong
+ */
+ return result;
+ }
+ }
/*
- * SyncOneBuffer() is only called by checkpointer and bgwriter, so
- * IOContext will always be IOCONTEXT_NORMAL.
+ * Run PageGetLSN while holding header lock, since we don't have the
+ * buffer locked exclusively in all cases.
*/
- ScheduleBufferTagForWriteback(wb_context, IOCONTEXT_NORMAL, &tag);
+ buf_state = LockBufHdr(cur_buf_hdr);
+
+ cur_buf_lsn = BufferGetLSN(cur_buf_hdr);
+
+ /* To check if block content changes while flushing. - vadim 01/17/97 */
+ buf_state &= ~BM_JUST_DIRTIED;
+
+ UnlockBufHdr(cur_buf_hdr, buf_state);
+
+ to_write->buffers[to_write->nbuffers] = buf;
+ to_write->nbuffers++;
+
+ if (buf_state & BM_PERMANENT &&
+ (to_write->max_lsn == InvalidXLogRecPtr || to_write->max_lsn < cur_buf_lsn))
+ {
+ to_write->max_lsn = cur_buf_lsn;
+ }
+
+ result |= BUF_WRITTEN;
+
+ return result;
+}
+
+static void
+WriteBuffers(BuffersToWrite *to_write,
+ IOQueue *ioq, WritebackContext *wb_context)
+{
+ SMgrRelation smgr;
+ Buffer first_buf;
+ BufferDesc *first_buf_hdr;
+ bool needs_checksum;
+
+ Assert(to_write->nbuffers > 0 && to_write->nbuffers <= io_combine_limit);
+
+ first_buf = to_write->buffers[0];
+ first_buf_hdr = GetBufferDescriptor(first_buf - 1);
+
+ smgr = smgropen(BufTagGetRelFileLocator(&first_buf_hdr->tag), INVALID_PROC_NUMBER);
+
+ /*
+ * Force XLOG flush up to buffer's LSN. This implements the basic WAL
+ * rule that log updates must hit disk before any of the data-file changes
+ * they describe do.
+ *
+ * However, this rule does not apply to unlogged relations, which will be
+ * lost after a crash anyway. Most unlogged relation pages do not bear
+ * LSNs since we never emit WAL records for them, and therefore flushing
+ * up through the buffer LSN would be useless, but harmless. However,
+ * GiST indexes use LSNs internally to track page-splits, and therefore
+ * unlogged GiST pages bear "fake" LSNs generated by
+ * GetFakeLSNForUnloggedRel. It is unlikely but possible that the fake
+ * LSN counter could advance past the WAL insertion point; and if it did
+ * happen, attempting to flush WAL through that location would fail, with
+ * disastrous system-wide consequences. To make sure that can't happen,
+ * skip the flush if the buffer isn't permanent.
+ */
+ if (to_write->max_lsn != InvalidXLogRecPtr)
+ XLogFlush(to_write->max_lsn);
+
+ /*
+ * Now it's safe to write buffer to disk. Note that no one else should
+ * have been able to write it while we were busy with log flushing because
+ * only one process at a time can set the BM_IO_IN_PROGRESS bit.
+ */
+
+ for (int nbuf = 0; nbuf < to_write->nbuffers; nbuf++)
+ {
+ Buffer cur_buf = to_write->buffers[nbuf];
+ BufferDesc *cur_buf_hdr = GetBufferDescriptor(cur_buf - 1);
+ Block bufBlock;
+ char *bufToWrite;
+
+ bufBlock = BufHdrGetBlock(cur_buf_hdr);
+ needs_checksum = PageNeedsChecksumCopy((Page) bufBlock);
+
+ /*
+ * Update page checksum if desired. Since we have only shared lock on
+ * the buffer, other processes might be updating hint bits in it, so
+ * we must copy the page to a bounce buffer if we do checksumming.
+ */
+ if (needs_checksum)
+ {
+ PgAioBounceBuffer *bb = pgaio_bounce_buffer_get();
+
+ pgaio_io_assoc_bounce_buffer(to_write->ioh, bb);
+
+ bufToWrite = pgaio_bounce_buffer_buffer(bb);
+ memcpy(bufToWrite, bufBlock, BLCKSZ);
+ PageSetChecksumInplace((Page) bufToWrite, cur_buf_hdr->tag.blockNum);
+ }
+ else
+ {
+ bufToWrite = bufBlock;
+ }
+
+ to_write->data_ptrs[nbuf] = bufToWrite;
+ }
+
+ pgaio_io_set_handle_data_32(to_write->ioh,
+ (uint32 *) to_write->buffers,
+ to_write->nbuffers);
+ pgaio_io_register_callbacks(to_write->ioh, PGAIO_HCB_SHARED_BUFFER_WRITEV, 0);
+
+ smgrstartwritev(to_write->ioh, smgr,
+ BufTagGetForkNum(&first_buf_hdr->tag),
+ first_buf_hdr->tag.blockNum,
+ to_write->data_ptrs,
+ to_write->nbuffers,
+ false);
+ pgstat_count_io_op(IOOBJECT_RELATION, IOCONTEXT_NORMAL,
+ IOOP_WRITE, 1, BLCKSZ * to_write->nbuffers);
+
+
+ for (int nbuf = 0; nbuf < to_write->nbuffers; nbuf++)
+ {
+ Buffer cur_buf = to_write->buffers[nbuf];
+ BufferDesc *cur_buf_hdr = GetBufferDescriptor(cur_buf - 1);
+
+ UnpinBuffer(cur_buf_hdr);
+ }
+
+ io_queue_track(ioq, &to_write->iow);
+ to_write->total_writes++;
- return result | BUF_WRITTEN;
+ /* clear state for next write */
+ to_write->nbuffers = 0;
+ to_write->start_at_tag.relNumber = InvalidOid;
+ to_write->start_at_tag.blockNum = InvalidBlockNumber;
+ to_write->max_combine = 0;
+ to_write->max_lsn = InvalidXLogRecPtr;
+ to_write->ioh = NULL;
+ pgaio_wref_clear(&to_write->iow);
}
/*
@@ -4176,6 +4655,7 @@ FlushBuffer(BufferDesc *buf, SMgrRelation reln, IOObject io_object,
error_context_stack = errcallback.previous;
}
+
/*
* RelationGetNumberOfBlocksInFork
* Determines the current number of pages in the specified relation fork.
diff --git a/src/backend/storage/page/bufpage.c b/src/backend/storage/page/bufpage.c
index ecc81aacfc3..71dc6f559dd 100644
--- a/src/backend/storage/page/bufpage.c
+++ b/src/backend/storage/page/bufpage.c
@@ -1480,6 +1480,16 @@ PageIndexTupleOverwrite(Page page, OffsetNumber offnum,
return true;
}
+bool
+PageNeedsChecksumCopy(Page page)
+{
+ if (PageIsNew(page))
+ return false;
+
+ /* If we don't need a checksum, just return the passed-in data */
+ return DataChecksumsEnabled();
+}
+
/*
* Set checksum for a page in shared buffers.
diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list
index 21b26196e0c..88c9a1285d7 100644
--- a/src/tools/pgindent/typedefs.list
+++ b/src/tools/pgindent/typedefs.list
@@ -348,6 +348,7 @@ BufferManagerRelation
BufferStrategyControl
BufferTag
BufferUsage
+BuffersToWrite
BuildAccumulator
BuiltinScript
BulkInsertState
--
2.48.1.76.g4e746b1a31.dirty
[text/x-diff] v2.7-0032-Ensure-a-resowner-exists-for-all-paths-that-may.patch (2.6K, ../../4b3f32ug3cayekysqlgspz2qjmeb7lca3gvazayglxr2m3d4dv@il33accgsji7/33-v2.7-0032-Ensure-a-resowner-exists-for-all-paths-that-may.patch)
download | inline diff:
From b79a011ac5ccaccc9a2bcd7be7176b10d676d7a5 Mon Sep 17 00:00:00 2001
From: Andres Freund <andres@anarazel.de>
Date: Tue, 8 Oct 2024 14:34:38 -0400
Subject: [PATCH v2.7 32/35] Ensure a resowner exists for all paths that may
perform AIO
Reviewed-by: Noah Misch <noah@leadboat.com>
Discussion: https://postgr.es/m/1f6b50a7-38ef-4d87-8246-786d39f46ab9@iki.fi
---
src/backend/bootstrap/bootstrap.c | 7 +++++++
src/backend/replication/logical/logical.c | 6 ++++++
src/backend/utils/init/postinit.c | 6 +++++-
3 files changed, 18 insertions(+), 1 deletion(-)
diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c
index 6db864892d0..e554504e1f0 100644
--- a/src/backend/bootstrap/bootstrap.c
+++ b/src/backend/bootstrap/bootstrap.c
@@ -361,8 +361,15 @@ BootstrapModeMain(int argc, char *argv[], bool check_only)
BaseInit();
bootstrap_signals();
+
+ /* need a resowner for IO during BootStrapXLOG() */
+ CreateAuxProcessResourceOwner();
+
BootStrapXLOG(bootstrap_data_checksum_version);
+ ReleaseAuxProcessResources(true);
+ CurrentResourceOwner = NULL;
+
/*
* To ensure that src/common/link-canary.c is linked into the backend, we
* must call it from somewhere. Here is as good as anywhere.
diff --git a/src/backend/replication/logical/logical.c b/src/backend/replication/logical/logical.c
index 8ea846bfc3b..2b936714040 100644
--- a/src/backend/replication/logical/logical.c
+++ b/src/backend/replication/logical/logical.c
@@ -386,6 +386,12 @@ CreateInitDecodingContext(const char *plugin,
slot->data.plugin = plugin_name;
SpinLockRelease(&slot->mutex);
+ if (CurrentResourceOwner == NULL)
+ {
+ Assert(am_walsender);
+ CurrentResourceOwner = AuxProcessResourceOwner;
+ }
+
if (XLogRecPtrIsInvalid(restart_lsn))
ReplicationSlotReserveWal();
else
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index 7958ea11b73..222e24bcb08 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -785,8 +785,12 @@ InitPostgres(const char *in_dbname, Oid dboid,
* We don't yet have an aux-process resource owner, but StartupXLOG
* and ShutdownXLOG will need one. Hence, create said resource owner
* (and register a callback to clean it up after ShutdownXLOG runs).
+ *
+ * In bootstrap mode CreateAuxProcessResourceOwner() was already
+ * called in BootstrapModeMain().
*/
- CreateAuxProcessResourceOwner();
+ if (!bootstrap)
+ CreateAuxProcessResourceOwner();
StartupXLOG();
/* Release (and warn about) any buffer pins leaked in StartupXLOG */
--
2.48.1.76.g4e746b1a31.dirty
[text/x-diff] v2.7-0033-Temporary-Increase-BAS_BULKREAD-size.patch (1.3K, ../../4b3f32ug3cayekysqlgspz2qjmeb7lca3gvazayglxr2m3d4dv@il33accgsji7/34-v2.7-0033-Temporary-Increase-BAS_BULKREAD-size.patch)
download | inline diff:
From 4fdb6e8083cf54dce497b4487f1685d8c90f72db Mon Sep 17 00:00:00 2001
From: Andres Freund <andres@anarazel.de>
Date: Sun, 1 Sep 2024 00:42:27 -0400
Subject: [PATCH v2.7 33/35] Temporary: Increase BAS_BULKREAD size
Without this we only can execute very little AIO for sequential scans, as
there's just not enough buffers in the ring. This isn't the right fix, as
just increasing the ring size can have negative performance implications in
workloads where the kernel has all the data cached.
Author:
Reviewed-By:
Discussion: https://postgr.es/m/
Backpatch:
---
src/backend/storage/buffer/freelist.c | 7 ++++++-
1 file changed, 6 insertions(+), 1 deletion(-)
diff --git a/src/backend/storage/buffer/freelist.c b/src/backend/storage/buffer/freelist.c
index 336715b6c63..b72a5957a20 100644
--- a/src/backend/storage/buffer/freelist.c
+++ b/src/backend/storage/buffer/freelist.c
@@ -555,7 +555,12 @@ GetAccessStrategy(BufferAccessStrategyType btype)
return NULL;
case BAS_BULKREAD:
- ring_size_kb = 256;
+
+ /*
+ * FIXME: Temporary increase to allow large enough streaming reads
+ * to actually benefit from AIO. This needs a better solution.
+ */
+ ring_size_kb = 2 * 1024;
break;
case BAS_BULKWRITE:
ring_size_kb = 16 * 1024;
--
2.48.1.76.g4e746b1a31.dirty
^ permalink raw reply [nested|flat] 153+ messages in thread
* Re: AIO v2.5
2025-03-04 19:00 AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-10 18:23 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-11 19:41 ` Re: AIO v2.5 Noah Misch <noah@leadboat.com>
2025-03-11 23:55 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-12 03:57 ` Re: AIO v2.5 Noah Misch <noah@leadboat.com>
2025-03-12 17:06 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-12 23:23 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
@ 2025-03-13 10:53 ` Antonin Houska <ah@cybertec.at>
2025-03-14 19:43 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
0 siblings, 1 reply; 153+ messages in thread
From: Antonin Houska @ 2025-03-13 10:53 UTC (permalink / raw)
To: Andres Freund <andres@anarazel.de>; +Cc: Noah Misch <noah@leadboat.com>; pgsql-hackers@postgresql.org, Thomas Munro <thomas.munro@gmail.com>; Heikki Linnakangas <hlinnaka@iki.fi>; Robert Haas <robertmhaas@gmail.com>; Jakub Wartak <jakub.wartak@enterprisedb.com>; Jelte Fennema-Nio <postgres@jeltef.nl>
Andres Freund <andres@anarazel.de> wrote:
> Attached is v2.7, with the following changes:
Attached are a few proposals for minor comment fixes.
Besides that, it occurred to me when I was trying to get familiar with the
patch set (respectable work, btw) that an additional Assert() statement could
make sense:
diff --git a/src/backend/storage/aio/aio.c b/src/backend/storage/aio/aio.c
index a9c351eb0dc..325688f0f23 100644
--- a/src/backend/storage/aio/aio.c
+++ b/src/backend/storage/aio/aio.c
@@ -413,6 +413,7 @@ pgaio_io_stage(PgAioHandle *ioh, PgAioOp op)
bool needs_synchronous;
Assert(ioh->state == PGAIO_HS_HANDED_OUT);
+ Assert(pgaio_my_backend->handed_out_io == ioh);
Assert(pgaio_io_has_target(ioh));
ioh->op = op;
--
Antonin Houska
Web: https://www.cybertec-postgresql.com
Attachments:
[text/x-diff] aio_comments.diff (1.9K, ../../17906.1741863183@localhost/2-aio_comments.diff)
download | inline diff:
diff --git a/src/backend/storage/aio/aio_io.c b/src/backend/storage/aio/aio_io.c
index 89376ff4040..7879e556263 100644
--- a/src/backend/storage/aio/aio_io.c
+++ b/src/backend/storage/aio/aio_io.c
@@ -107,7 +107,7 @@ pgaio_io_prep_writev(PgAioHandle *ioh,
/*
* Execute IO operation synchronously. This is implemented here, not in
- * method_sync.c, because other IO methods lso might use it / fall back to it.
+ * method_sync.c, because other IO methods also might use it / fall back to it.
*/
void
pgaio_io_perform_synchronously(PgAioHandle *ioh)
diff --git a/src/backend/storage/buffer/bufmgr.c b/src/backend/storage/buffer/bufmgr.c
index 9483596e63a..df9357c78c0 100644
--- a/src/backend/storage/buffer/bufmgr.c
+++ b/src/backend/storage/buffer/bufmgr.c
@@ -7329,7 +7329,7 @@ local_buffer_writev_stage(PgAioHandle *ioh, uint8 cb_data)
}
-/* readv callback is is passed READ_BUFFERS_* flags as callback data */
+/* readv callback is passed READ_BUFFERS_* flags as callback data */
const PgAioHandleCallbacks aio_shared_buffer_readv_cb = {
.stage = shared_buffer_readv_stage,
.complete_shared = shared_buffer_readv_complete,
diff --git a/src/include/storage/aio_internal.h b/src/include/storage/aio_internal.h
index ca6cbde11a6..cf4eb102adc 100644
--- a/src/include/storage/aio_internal.h
+++ b/src/include/storage/aio_internal.h
@@ -205,9 +205,9 @@ typedef struct PgAioBackend
dclist_head idle_ios;
/*
- * Only one IO may be returned by pgaio_io_acquire()/pgaio_io_acquire()
+ * Only one IO may be returned by pgaio_io_acquire()/pgaio_io_acquire_nb()
* without having been either defined (by actually associating it with IO)
- * or by released (with pgaio_io_release()). This restriction is necessary
+ * or released (with pgaio_io_release()). This restriction is necessary
* to guarantee that we always can acquire an IO. ->handed_out_io is used
* to enforce that rule.
*/
^ permalink raw reply [nested|flat] 153+ messages in thread
* Re: AIO v2.5
2025-03-04 19:00 AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-10 18:23 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-11 19:41 ` Re: AIO v2.5 Noah Misch <noah@leadboat.com>
2025-03-11 23:55 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-12 03:57 ` Re: AIO v2.5 Noah Misch <noah@leadboat.com>
2025-03-12 17:06 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-12 23:23 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-13 10:53 ` Re: AIO v2.5 Antonin Houska <ah@cybertec.at>
@ 2025-03-14 19:43 ` Andres Freund <andres@anarazel.de>
2025-03-14 19:58 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-17 20:00 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-17 20:53 ` Re: AIO v2.5 Melanie Plageman <melanieplageman@gmail.com>
2025-06-30 06:58 ` Re: AIO v2.5 Antonin Houska <ah@cybertec.at>
0 siblings, 4 replies; 153+ messages in thread
From: Andres Freund @ 2025-03-14 19:43 UTC (permalink / raw)
To: Antonin Houska <ah@cybertec.at>; +Cc: Noah Misch <noah@leadboat.com>; pgsql-hackers@postgresql.org, Thomas Munro <thomas.munro@gmail.com>; Heikki Linnakangas <hlinnaka@iki.fi>; Robert Haas <robertmhaas@gmail.com>; Jakub Wartak <jakub.wartak@enterprisedb.com>; Jelte Fennema-Nio <postgres@jeltef.nl>
Hi,
On 2025-03-13 11:53:03 +0100, Antonin Houska wrote:
> Attached are a few proposals for minor comment fixes.
Thanks, applied.
> Besides that, it occurred to me when I was trying to get familiar with the
> patch set (respectable work, btw) that an additional Assert() statement could
> make sense:
Yea, it does. I added it to another place as well.
Attached is v2.8 with the following changes:
- I wasn't happy with the way StartReadBuffers(), WaitReadBuffers() and
AsyncReadBuffers() interacted. The io_method=sync path in particular was too
cute by half, calling WaitReadBuffers() from within WaitReadBuffers().
I think the new state considerably better.
Plenty other smaller changes as part of that. One worth calling out is that
ReadBuffersCanStartIO() now submits staged IO before actually blocking. Not
the prettiest code, but I think it's ok.
- Added a function to assert the sanitiy of a ReadBuffersOperation
While doing the refactoring for the prior point, I temporarily had a bug
that returned buffers for which IO wasn't actually performed. Surprisingly
the only assertion that triggered was when that buffer was read again by
another operation, because it had been marked dirty, despite never being
valid.
Now there's a function that can be used to check that the buffers referenced
by a ReadBuffersOperation are in a sane state.
I guess this could be committed independently, but it'd not be entirely
trivial to extract, so I'm currently leaning against doing that.
- Previously VacuumCostActive accounting happened after IO completion. But
that doesn't seem quite right, it'd allow to submit a lot of IO at
once. It's now moved to AsyncReadBuffers()
- With io_method=sync or with worker and temp tables, smgrstartreadv() would
actually execute the IO. But the time accounting was done entirely around
pgaio_wref_wait(). Now it's done in both places.
- Rebased onto newer version of Thomas' read_stream.c changes
With that newer version the integration with read stream for actually doing
AIO is a bit simpler. There's one FIXME in the patch, because I don't
really understand what a comment is referring to.
I also split out a more experimental patch to make more efficient use of
batching in read stream, the heuristics are more complicated, and it works
well enough without.
- I added a commit to clean up the buffer access accounting for the case that
a buffer was read in concurrently. That IMO is somewhat bogus on master, and
it seemed to get more bogus with AIO.
- Integrated Antonin Houska's fixes and Assert suggestion
- Added a patch to address the smgr.c/md.c interrupt issue (a problem in master), see
https://postgr.es/m/3vae7l5ozvqtxmd7rr7zaeq3qkuipz365u3rtim5t5wdkr6f4g@vkgf2fogjirl
I think the reasonable next steps are:
- Commit "localbuf: *" commits
- Commit temp table tests, likely after lowering the minimum temp_buffers setting
- Pursue a fix of the smgr interupt issue on the referenced thread
This can happen in parallel with AIO patches up to
"aio: Implement support for reads in smgr/md/fd"
- Commit the core AIO infrastructure patch after doing a few more passes
- Commit IO worker support
- In parallel: Find a way to deal with the set_max_safe_fds() issue that we've
been discussing on this thread recently. As that only affects io_uring, it
doesn't have to block other patches going in.
- Do a round of review of the read_stream changes Thomas recently posted (and
that are also included here)
- Try to get some more review for the bufmgr.c related changes. I've whacked
them around a fair bit lately.
- Try to get Thomas to review my read_stream.c changes
Open items:
- The upstream BAS_BULKREAD is so small that throughput is substantially worse
once a table reaches 1/4 shared_buffers. That patch in the patchset as-is is
probably not good enough, although I am not sure about that
- The set_max_safe_fds() issue for io_uring
- Right now effective_io_concurrency cannot be set > 0 on Windows and other
platforms that lack posix_fadvise. But with AIO we can read ahead without
posix_fadvise().
It'd not really make anything worse than today to not remove the limit, but
it'd be pretty weird to prevent windows etc from benefiting from AIO. Need
to look around and see whether it would require anything other than doc
changes.
Greetings,
Andres Freund
Attachments:
[text/x-diff] v2.8-0001-aio-Basic-subsystem-initialization.patch (25.1K, ../../3yxd5r23zly5bytvgyktbxtxq2r3gbpi7xd4dugevh3h4w4q6c@lu6oatjjpltz/2-v2.8-0001-aio-Basic-subsystem-initialization.patch)
download | inline diff:
From f430ba47d6f72d37ec772b0f2e1875158cafbefe Mon Sep 17 00:00:00 2001
From: Andres Freund <andres@anarazel.de>
Date: Tue, 4 Mar 2025 10:44:56 -0500
Subject: [PATCH v2.8 01/38] aio: Basic subsystem initialization
This commit just does the minimal wiring up of the AIO subsystem, to be fully
added in the next commit, to the rest of the system. The next commit contains
more details about motivation and architecture.
This commit is kept separate to make it easier to review, reviewing how the
AIO subsystem works is a big enough task on its own an rather separate from
reviewing the changes across the tree.
We discussed squashing this commit with the main commit before merging AIO,
but there has been a mild preference for keeping it separate.
Discussion: https://postgr.es/m/uvrtrknj4kdytuboidbhwclo4gxhswwcpgadptsjvjqcluzmah%40brqs62irg4dt
---
src/include/storage/aio.h | 38 ++++++++
src/include/storage/aio_subsys.h | 33 +++++++
src/include/utils/guc.h | 1 +
src/include/utils/guc_hooks.h | 2 +
src/include/utils/resowner.h | 5 ++
src/backend/access/transam/xact.c | 12 +++
src/backend/postmaster/autovacuum.c | 2 +
src/backend/postmaster/bgwriter.c | 2 +
src/backend/postmaster/checkpointer.c | 2 +
src/backend/postmaster/pgarch.c | 2 +
src/backend/postmaster/walsummarizer.c | 2 +
src/backend/postmaster/walwriter.c | 2 +
src/backend/replication/walsender.c | 2 +
src/backend/storage/aio/Makefile | 2 +
src/backend/storage/aio/aio.c | 90 +++++++++++++++++++
src/backend/storage/aio/aio_init.c | 37 ++++++++
src/backend/storage/aio/meson.build | 2 +
src/backend/storage/ipc/ipci.c | 3 +
src/backend/utils/init/postinit.c | 7 ++
src/backend/utils/misc/guc_tables.c | 23 +++++
src/backend/utils/misc/postgresql.conf.sample | 6 ++
src/backend/utils/resowner/resowner.c | 29 ++++++
doc/src/sgml/config.sgml | 51 +++++++++++
src/tools/pgindent/typedefs.list | 1 +
24 files changed, 356 insertions(+)
create mode 100644 src/include/storage/aio.h
create mode 100644 src/include/storage/aio_subsys.h
create mode 100644 src/backend/storage/aio/aio.c
create mode 100644 src/backend/storage/aio/aio_init.c
diff --git a/src/include/storage/aio.h b/src/include/storage/aio.h
new file mode 100644
index 00000000000..e79d5343038
--- /dev/null
+++ b/src/include/storage/aio.h
@@ -0,0 +1,38 @@
+/*-------------------------------------------------------------------------
+ *
+ * aio.h
+ * Main AIO interface
+ *
+ *
+ * Portions Copyright (c) 1996-2025, PostgreSQL Global Development Group
+ * Portions Copyright (c) 1994, Regents of the University of California
+ *
+ * src/include/storage/aio.h
+ *
+ *-------------------------------------------------------------------------
+ */
+#ifndef AIO_H
+#define AIO_H
+
+
+
+/* Enum for io_method GUC. */
+typedef enum IoMethod
+{
+ IOMETHOD_SYNC = 0,
+} IoMethod;
+
+/* We'll default to synchronous execution. */
+#define DEFAULT_IO_METHOD IOMETHOD_SYNC
+
+
+struct dlist_node;
+extern void pgaio_io_release_resowner(struct dlist_node *ioh_node, bool on_error);
+
+
+/* GUCs */
+extern PGDLLIMPORT int io_method;
+extern PGDLLIMPORT int io_max_concurrency;
+
+
+#endif /* AIO_H */
diff --git a/src/include/storage/aio_subsys.h b/src/include/storage/aio_subsys.h
new file mode 100644
index 00000000000..e4faf692a38
--- /dev/null
+++ b/src/include/storage/aio_subsys.h
@@ -0,0 +1,33 @@
+/*-------------------------------------------------------------------------
+ *
+ * aio_subsys.h
+ * Interaction with AIO as a subsystem, rather than actually issuing AIO
+ *
+ * This header is for AIO related functionality that's being called by files
+ * that don't perform AIO but interact with it in some form. E.g. postmaster.c
+ * and shared memory initialization need to initialize AIO but don't perform
+ * AIO.
+ *
+ * Portions Copyright (c) 1996-2025, PostgreSQL Global Development Group
+ * Portions Copyright (c) 1994, Regents of the University of California
+ *
+ * src/include/storage/aio_subsys.h
+ *
+ *-------------------------------------------------------------------------
+ */
+#ifndef AIO_SUBSYS_H
+#define AIO_SUBSYS_H
+
+
+/* aio_init.c */
+extern Size AioShmemSize(void);
+extern void AioShmemInit(void);
+
+extern void pgaio_init_backend(void);
+
+
+/* aio.c */
+extern void pgaio_error_cleanup(void);
+extern void AtEOXact_Aio(bool is_commit);
+
+#endif /* AIO_SUBSYS_H */
diff --git a/src/include/utils/guc.h b/src/include/utils/guc.h
index 24444cbc365..f619100467d 100644
--- a/src/include/utils/guc.h
+++ b/src/include/utils/guc.h
@@ -318,6 +318,7 @@ extern PGDLLIMPORT bool optimize_bounded_sort;
*/
extern PGDLLIMPORT const struct config_enum_entry archive_mode_options[];
extern PGDLLIMPORT const struct config_enum_entry dynamic_shared_memory_options[];
+extern PGDLLIMPORT const struct config_enum_entry io_method_options[];
extern PGDLLIMPORT const struct config_enum_entry recovery_target_action_options[];
extern PGDLLIMPORT const struct config_enum_entry wal_level_options[];
extern PGDLLIMPORT const struct config_enum_entry wal_sync_method_options[];
diff --git a/src/include/utils/guc_hooks.h b/src/include/utils/guc_hooks.h
index 9a0d8ec85c7..a3eba8fbe21 100644
--- a/src/include/utils/guc_hooks.h
+++ b/src/include/utils/guc_hooks.h
@@ -64,6 +64,8 @@ extern bool check_default_with_oids(bool *newval, void **extra,
extern bool check_effective_io_concurrency(int *newval, void **extra,
GucSource source);
extern bool check_huge_page_size(int *newval, void **extra, GucSource source);
+extern void assign_io_method(int newval, void *extra);
+extern bool check_io_max_concurrency(int *newval, void **extra, GucSource source);
extern const char *show_in_hot_standby(void);
extern bool check_locale_messages(char **newval, void **extra, GucSource source);
extern void assign_locale_messages(const char *newval, void *extra);
diff --git a/src/include/utils/resowner.h b/src/include/utils/resowner.h
index e8d452ca7ee..aede4bfc820 100644
--- a/src/include/utils/resowner.h
+++ b/src/include/utils/resowner.h
@@ -164,4 +164,9 @@ struct LOCALLOCK;
extern void ResourceOwnerRememberLock(ResourceOwner owner, struct LOCALLOCK *locallock);
extern void ResourceOwnerForgetLock(ResourceOwner owner, struct LOCALLOCK *locallock);
+/* special support for AIO */
+struct dlist_node;
+extern void ResourceOwnerRememberAioHandle(ResourceOwner owner, struct dlist_node *ioh_node);
+extern void ResourceOwnerForgetAioHandle(ResourceOwner owner, struct dlist_node *ioh_node);
+
#endif /* RESOWNER_H */
diff --git a/src/backend/access/transam/xact.c b/src/backend/access/transam/xact.c
index 1b4f21a88d3..b885513f765 100644
--- a/src/backend/access/transam/xact.c
+++ b/src/backend/access/transam/xact.c
@@ -51,6 +51,7 @@
#include "replication/origin.h"
#include "replication/snapbuild.h"
#include "replication/syncrep.h"
+#include "storage/aio_subsys.h"
#include "storage/condition_variable.h"
#include "storage/fd.h"
#include "storage/lmgr.h"
@@ -2411,6 +2412,8 @@ CommitTransaction(void)
RESOURCE_RELEASE_BEFORE_LOCKS,
true, true);
+ AtEOXact_Aio(true);
+
/* Check we've released all buffer pins */
AtEOXact_Buffers(true);
@@ -2716,6 +2719,8 @@ PrepareTransaction(void)
RESOURCE_RELEASE_BEFORE_LOCKS,
true, true);
+ AtEOXact_Aio(true);
+
/* Check we've released all buffer pins */
AtEOXact_Buffers(true);
@@ -2830,6 +2835,8 @@ AbortTransaction(void)
pgstat_report_wait_end();
pgstat_progress_end_command();
+ pgaio_error_cleanup();
+
/* Clean up buffer content locks, too */
UnlockBuffers();
@@ -2960,6 +2967,7 @@ AbortTransaction(void)
ResourceOwnerRelease(TopTransactionResourceOwner,
RESOURCE_RELEASE_BEFORE_LOCKS,
false, true);
+ AtEOXact_Aio(false);
AtEOXact_Buffers(false);
AtEOXact_RelationCache(false);
AtEOXact_TypeCache();
@@ -5232,6 +5240,9 @@ AbortSubTransaction(void)
pgstat_report_wait_end();
pgstat_progress_end_command();
+
+ pgaio_error_cleanup();
+
UnlockBuffers();
/* Reset WAL record construction state */
@@ -5326,6 +5337,7 @@ AbortSubTransaction(void)
RESOURCE_RELEASE_BEFORE_LOCKS,
false, false);
+ AtEOXact_Aio(false);
AtEOSubXact_RelationCache(false, s->subTransactionId,
s->parent->subTransactionId);
AtEOSubXact_TypeCache();
diff --git a/src/backend/postmaster/autovacuum.c b/src/backend/postmaster/autovacuum.c
index 71c34027c88..2513a8ef8a6 100644
--- a/src/backend/postmaster/autovacuum.c
+++ b/src/backend/postmaster/autovacuum.c
@@ -88,6 +88,7 @@
#include "postmaster/autovacuum.h"
#include "postmaster/interrupt.h"
#include "postmaster/postmaster.h"
+#include "storage/aio_subsys.h"
#include "storage/bufmgr.h"
#include "storage/ipc.h"
#include "storage/latch.h"
@@ -465,6 +466,7 @@ AutoVacLauncherMain(const void *startup_data, size_t startup_data_len)
*/
LWLockReleaseAll();
pgstat_report_wait_end();
+ pgaio_error_cleanup();
UnlockBuffers();
/* this is probably dead code, but let's be safe: */
if (AuxProcessResourceOwner)
diff --git a/src/backend/postmaster/bgwriter.c b/src/backend/postmaster/bgwriter.c
index a688cc5d2a1..72f5acceec7 100644
--- a/src/backend/postmaster/bgwriter.c
+++ b/src/backend/postmaster/bgwriter.c
@@ -38,6 +38,7 @@
#include "postmaster/auxprocess.h"
#include "postmaster/bgwriter.h"
#include "postmaster/interrupt.h"
+#include "storage/aio_subsys.h"
#include "storage/buf_internals.h"
#include "storage/bufmgr.h"
#include "storage/condition_variable.h"
@@ -168,6 +169,7 @@ BackgroundWriterMain(const void *startup_data, size_t startup_data_len)
*/
LWLockReleaseAll();
ConditionVariableCancelSleep();
+ pgaio_error_cleanup();
UnlockBuffers();
ReleaseAuxProcessResources(false);
AtEOXact_Buffers(false);
diff --git a/src/backend/postmaster/checkpointer.c b/src/backend/postmaster/checkpointer.c
index 0e228d143a0..fda91ffd1ce 100644
--- a/src/backend/postmaster/checkpointer.c
+++ b/src/backend/postmaster/checkpointer.c
@@ -49,6 +49,7 @@
#include "postmaster/bgwriter.h"
#include "postmaster/interrupt.h"
#include "replication/syncrep.h"
+#include "storage/aio_subsys.h"
#include "storage/bufmgr.h"
#include "storage/condition_variable.h"
#include "storage/fd.h"
@@ -276,6 +277,7 @@ CheckpointerMain(const void *startup_data, size_t startup_data_len)
LWLockReleaseAll();
ConditionVariableCancelSleep();
pgstat_report_wait_end();
+ pgaio_error_cleanup();
UnlockBuffers();
ReleaseAuxProcessResources(false);
AtEOXact_Buffers(false);
diff --git a/src/backend/postmaster/pgarch.c b/src/backend/postmaster/pgarch.c
index dbe4e1d426b..7e622ae4bd2 100644
--- a/src/backend/postmaster/pgarch.c
+++ b/src/backend/postmaster/pgarch.c
@@ -40,6 +40,7 @@
#include "postmaster/interrupt.h"
#include "postmaster/pgarch.h"
#include "storage/condition_variable.h"
+#include "storage/aio_subsys.h"
#include "storage/fd.h"
#include "storage/ipc.h"
#include "storage/latch.h"
@@ -568,6 +569,7 @@ pgarch_archiveXlog(char *xlog)
LWLockReleaseAll();
ConditionVariableCancelSleep();
pgstat_report_wait_end();
+ pgaio_error_cleanup();
ReleaseAuxProcessResources(false);
AtEOXact_Files(false);
AtEOXact_HashTables(false);
diff --git a/src/backend/postmaster/walsummarizer.c b/src/backend/postmaster/walsummarizer.c
index ccba0f84e6e..0fec4f1f871 100644
--- a/src/backend/postmaster/walsummarizer.c
+++ b/src/backend/postmaster/walsummarizer.c
@@ -38,6 +38,7 @@
#include "postmaster/interrupt.h"
#include "postmaster/walsummarizer.h"
#include "replication/walreceiver.h"
+#include "storage/aio_subsys.h"
#include "storage/fd.h"
#include "storage/ipc.h"
#include "storage/latch.h"
@@ -289,6 +290,7 @@ WalSummarizerMain(const void *startup_data, size_t startup_data_len)
LWLockReleaseAll();
ConditionVariableCancelSleep();
pgstat_report_wait_end();
+ pgaio_error_cleanup();
ReleaseAuxProcessResources(false);
AtEOXact_Files(false);
AtEOXact_HashTables(false);
diff --git a/src/backend/postmaster/walwriter.c b/src/backend/postmaster/walwriter.c
index 0380601bcbb..fd92c8b7a33 100644
--- a/src/backend/postmaster/walwriter.c
+++ b/src/backend/postmaster/walwriter.c
@@ -51,6 +51,7 @@
#include "postmaster/auxprocess.h"
#include "postmaster/interrupt.h"
#include "postmaster/walwriter.h"
+#include "storage/aio_subsys.h"
#include "storage/bufmgr.h"
#include "storage/condition_variable.h"
#include "storage/fd.h"
@@ -164,6 +165,7 @@ WalWriterMain(const void *startup_data, size_t startup_data_len)
LWLockReleaseAll();
ConditionVariableCancelSleep();
pgstat_report_wait_end();
+ pgaio_error_cleanup();
UnlockBuffers();
ReleaseAuxProcessResources(false);
AtEOXact_Buffers(false);
diff --git a/src/backend/replication/walsender.c b/src/backend/replication/walsender.c
index d96121b3aad..1028919aecb 100644
--- a/src/backend/replication/walsender.c
+++ b/src/backend/replication/walsender.c
@@ -79,6 +79,7 @@
#include "replication/walsender.h"
#include "replication/walsender_private.h"
#include "storage/condition_variable.h"
+#include "storage/aio_subsys.h"
#include "storage/fd.h"
#include "storage/ipc.h"
#include "storage/pmsignal.h"
@@ -327,6 +328,7 @@ WalSndErrorCleanup(void)
LWLockReleaseAll();
ConditionVariableCancelSleep();
pgstat_report_wait_end();
+ pgaio_error_cleanup();
if (xlogreader != NULL && xlogreader->seg.ws_file >= 0)
wal_segment_close(xlogreader);
diff --git a/src/backend/storage/aio/Makefile b/src/backend/storage/aio/Makefile
index 2f29a9ec4d1..eaeaeeee8e3 100644
--- a/src/backend/storage/aio/Makefile
+++ b/src/backend/storage/aio/Makefile
@@ -9,6 +9,8 @@ top_builddir = ../../../..
include $(top_builddir)/src/Makefile.global
OBJS = \
+ aio.o \
+ aio_init.o \
read_stream.o
include $(top_srcdir)/src/backend/common.mk
diff --git a/src/backend/storage/aio/aio.c b/src/backend/storage/aio/aio.c
new file mode 100644
index 00000000000..828a94efdc3
--- /dev/null
+++ b/src/backend/storage/aio/aio.c
@@ -0,0 +1,90 @@
+/*-------------------------------------------------------------------------
+ *
+ * aio.c
+ * AIO - Core Logic
+ *
+ * Portions Copyright (c) 1996-2025, PostgreSQL Global Development Group
+ * Portions Copyright (c) 1994, Regents of the University of California
+ *
+ * IDENTIFICATION
+ * src/backend/storage/aio/aio.c
+ *
+ *-------------------------------------------------------------------------
+ */
+
+#include "postgres.h"
+
+#include "lib/ilist.h"
+#include "storage/aio.h"
+#include "storage/aio_subsys.h"
+#include "utils/guc.h"
+#include "utils/guc_hooks.h"
+
+
+/* Options for io_method. */
+const struct config_enum_entry io_method_options[] = {
+ {"sync", IOMETHOD_SYNC, false},
+ {NULL, 0, false}
+};
+
+/* GUCs */
+int io_method = DEFAULT_IO_METHOD;
+int io_max_concurrency = -1;
+
+
+
+/*
+ * Release IO handle during resource owner cleanup.
+ */
+void
+pgaio_io_release_resowner(dlist_node *ioh_node, bool on_error)
+{
+}
+
+/*
+ * Perform AIO related cleanup after an error.
+ *
+ * This should be called early in the error recovery paths, as later steps may
+ * need to issue AIO (e.g. to record a transaction abort WAL record).
+ */
+void
+pgaio_error_cleanup(void)
+{
+}
+
+/*
+ * Perform AIO related checks at (sub-)transactional boundaries.
+ *
+ * This should be called late during (sub-)transactional commit/abort, after
+ * all steps that might need to perform AIO, so that we can verify that the
+ * AIO subsystem is in a valid state at the end of a transaction.
+ */
+void
+AtEOXact_Aio(bool is_commit)
+{
+}
+
+void
+assign_io_method(int newval, void *extra)
+{
+}
+
+bool
+check_io_max_concurrency(int *newval, void **extra, GucSource source)
+{
+ if (*newval == -1)
+ {
+ /*
+ * Auto-tuning will be applied later during startup, as auto-tuning
+ * depends on the value of various GUCs.
+ */
+ return true;
+ }
+ else if (*newval == 0)
+ {
+ GUC_check_errdetail("Only -1 or values bigger than 0 are valid.");
+ return false;
+ }
+
+ return true;
+}
diff --git a/src/backend/storage/aio/aio_init.c b/src/backend/storage/aio/aio_init.c
new file mode 100644
index 00000000000..aeacc144149
--- /dev/null
+++ b/src/backend/storage/aio/aio_init.c
@@ -0,0 +1,37 @@
+/*-------------------------------------------------------------------------
+ *
+ * aio_init.c
+ * AIO - Subsystem Initialization
+ *
+ * Portions Copyright (c) 1996-2025, PostgreSQL Global Development Group
+ * Portions Copyright (c) 1994, Regents of the University of California
+ *
+ * IDENTIFICATION
+ * src/backend/storage/aio/aio_init.c
+ *
+ *-------------------------------------------------------------------------
+ */
+
+#include "postgres.h"
+
+#include "storage/aio_subsys.h"
+
+
+
+Size
+AioShmemSize(void)
+{
+ Size sz = 0;
+
+ return sz;
+}
+
+void
+AioShmemInit(void)
+{
+}
+
+void
+pgaio_init_backend(void)
+{
+}
diff --git a/src/backend/storage/aio/meson.build b/src/backend/storage/aio/meson.build
index 8abe0eb4863..c822fd4ddf7 100644
--- a/src/backend/storage/aio/meson.build
+++ b/src/backend/storage/aio/meson.build
@@ -1,5 +1,7 @@
# Copyright (c) 2024-2025, PostgreSQL Global Development Group
backend_sources += files(
+ 'aio.c',
+ 'aio_init.c',
'read_stream.c',
)
diff --git a/src/backend/storage/ipc/ipci.c b/src/backend/storage/ipc/ipci.c
index 174eed70367..2fa045e6b0f 100644
--- a/src/backend/storage/ipc/ipci.c
+++ b/src/backend/storage/ipc/ipci.c
@@ -37,6 +37,7 @@
#include "replication/slotsync.h"
#include "replication/walreceiver.h"
#include "replication/walsender.h"
+#include "storage/aio_subsys.h"
#include "storage/bufmgr.h"
#include "storage/dsm.h"
#include "storage/dsm_registry.h"
@@ -148,6 +149,7 @@ CalculateShmemSize(int *num_semaphores)
size = add_size(size, WaitEventCustomShmemSize());
size = add_size(size, InjectionPointShmemSize());
size = add_size(size, SlotSyncShmemSize());
+ size = add_size(size, AioShmemSize());
/* include additional requested shmem from preload libraries */
size = add_size(size, total_addin_request);
@@ -340,6 +342,7 @@ CreateOrAttachShmemStructs(void)
StatsShmemInit();
WaitEventCustomShmemInit();
InjectionPointShmemInit();
+ AioShmemInit();
}
/*
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index 4b2faf1ba9d..7958ea11b73 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -43,6 +43,7 @@
#include "replication/slot.h"
#include "replication/slotsync.h"
#include "replication/walsender.h"
+#include "storage/aio_subsys.h"
#include "storage/bufmgr.h"
#include "storage/fd.h"
#include "storage/ipc.h"
@@ -635,6 +636,12 @@ BaseInit(void)
*/
pgstat_initialize();
+ /*
+ * Initialize AIO before infrastructure that might need to actually
+ * execute AIO.
+ */
+ pgaio_init_backend();
+
/* Do local initialization of storage and buffer managers */
InitSync();
smgrinit();
diff --git a/src/backend/utils/misc/guc_tables.c b/src/backend/utils/misc/guc_tables.c
index 9c0b10ad4dc..0d3ebf06a95 100644
--- a/src/backend/utils/misc/guc_tables.c
+++ b/src/backend/utils/misc/guc_tables.c
@@ -72,6 +72,7 @@
#include "replication/slot.h"
#include "replication/slotsync.h"
#include "replication/syncrep.h"
+#include "storage/aio.h"
#include "storage/bufmgr.h"
#include "storage/bufpage.h"
#include "storage/large_object.h"
@@ -3254,6 +3255,18 @@ struct config_int ConfigureNamesInt[] =
NULL, NULL, NULL
},
+ {
+ {"io_max_concurrency",
+ PGC_POSTMASTER,
+ RESOURCES_IO,
+ gettext_noop("Max number of IOs that one process can execute simultaneously."),
+ NULL,
+ },
+ &io_max_concurrency,
+ -1, -1, 1024,
+ check_io_max_concurrency, NULL, NULL
+ },
+
{
{"backend_flush_after", PGC_USERSET, RESOURCES_IO,
gettext_noop("Number of pages after which previously performed writes are flushed to disk."),
@@ -5311,6 +5324,16 @@ struct config_enum ConfigureNamesEnum[] =
NULL, NULL, NULL
},
+ {
+ {"io_method", PGC_POSTMASTER, RESOURCES_IO,
+ gettext_noop("Selects the method for executing asynchronous I/O."),
+ NULL
+ },
+ &io_method,
+ DEFAULT_IO_METHOD, io_method_options,
+ NULL, assign_io_method, NULL
+ },
+
/* End-of-list marker */
{
{NULL, 0, 0, NULL, NULL}, NULL, 0, NULL, NULL, NULL, NULL
diff --git a/src/backend/utils/misc/postgresql.conf.sample b/src/backend/utils/misc/postgresql.conf.sample
index 8de86e0c945..43c2ec2153e 100644
--- a/src/backend/utils/misc/postgresql.conf.sample
+++ b/src/backend/utils/misc/postgresql.conf.sample
@@ -202,6 +202,12 @@
#maintenance_io_concurrency = 10 # 1-1000; 0 disables prefetching
#io_combine_limit = 128kB # usually 1-32 blocks (depends on OS)
+#io_method = sync # sync (change requires restart)
+#io_max_concurrency = -1 # Max number of IOs that one process
+ # can execute simultaneously
+ # -1 sets based on shared_buffers
+ # (change requires restart)
+
# - Worker Processes -
#max_worker_processes = 8 # (change requires restart)
diff --git a/src/backend/utils/resowner/resowner.c b/src/backend/utils/resowner/resowner.c
index ac5ca4a765e..d39f3e1b655 100644
--- a/src/backend/utils/resowner/resowner.c
+++ b/src/backend/utils/resowner/resowner.c
@@ -47,6 +47,8 @@
#include "common/hashfn.h"
#include "common/int.h"
+#include "lib/ilist.h"
+#include "storage/aio.h"
#include "storage/ipc.h"
#include "storage/predicate.h"
#include "storage/proc.h"
@@ -155,6 +157,12 @@ struct ResourceOwnerData
/* The local locks cache. */
LOCALLOCK *locks[MAX_RESOWNER_LOCKS]; /* list of owned locks */
+
+ /*
+ * AIO handles need be registered in critical sections and therefore
+ * cannot use the normal ResourceElem mechanism.
+ */
+ dlist_head aio_handles;
};
@@ -425,6 +433,8 @@ ResourceOwnerCreate(ResourceOwner parent, const char *name)
parent->firstchild = owner;
}
+ dlist_init(&owner->aio_handles);
+
return owner;
}
@@ -725,6 +735,13 @@ ResourceOwnerReleaseInternal(ResourceOwner owner,
* so issue warnings. In the abort case, just clean up quietly.
*/
ResourceOwnerReleaseAll(owner, phase, isCommit);
+
+ while (!dlist_is_empty(&owner->aio_handles))
+ {
+ dlist_node *node = dlist_head_node(&owner->aio_handles);
+
+ pgaio_io_release_resowner(node, !isCommit);
+ }
}
else if (phase == RESOURCE_RELEASE_LOCKS)
{
@@ -1082,3 +1099,15 @@ ResourceOwnerForgetLock(ResourceOwner owner, LOCALLOCK *locallock)
elog(ERROR, "lock reference %p is not owned by resource owner %s",
locallock, owner->name);
}
+
+void
+ResourceOwnerRememberAioHandle(ResourceOwner owner, struct dlist_node *ioh_node)
+{
+ dlist_push_tail(&owner->aio_handles, ioh_node);
+}
+
+void
+ResourceOwnerForgetAioHandle(ResourceOwner owner, struct dlist_node *ioh_node)
+{
+ dlist_delete_from(&owner->aio_handles, ioh_node);
+}
diff --git a/doc/src/sgml/config.sgml b/doc/src/sgml/config.sgml
index 3d62c8bd274..7ec18bb7627 100644
--- a/doc/src/sgml/config.sgml
+++ b/doc/src/sgml/config.sgml
@@ -2638,6 +2638,57 @@ include_dir 'conf.d'
</para>
</listitem>
</varlistentry>
+
+ <varlistentry id="guc-io-max-concurrency" xreflabel="io_max_concurrency">
+ <term><varname>io_max_concurrency</varname> (<type>integer</type>)
+ <indexterm>
+ <primary><varname>io_max_concurrency</varname> configuration parameter</primary>
+ </indexterm>
+ </term>
+ <listitem>
+ <para>
+ Controls the maximum number of I/O operations that one process can
+ execute simultaneously.
+ </para>
+ <para>
+ The default setting of <literal>-1</literal> selects a number based
+ on <xref linkend="guc-shared-buffers"/> and the maximum number of
+ processes (<xref linkend="guc-max-connections"/>, <xref
+ linkend="guc-autovacuum-worker-slots"/>, <xref
+ linkend="guc-max-worker-processes"/> and <xref
+ linkend="guc-max-wal-senders"/>), but not more than
+ <literal>64</literal>.
+ </para>
+ <para>
+ This parameter can only be set at server start.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry id="guc-io-method" xreflabel="io_method">
+ <term><varname>io_method</varname> (<type>enum</type>)
+ <indexterm>
+ <primary><varname>io_method</varname> configuration parameter</primary>
+ </indexterm>
+ </term>
+ <listitem>
+ <para>
+ Selects the method for executing asynchronous I/O.
+ Possible values are:
+ <itemizedlist>
+ <listitem>
+ <para>
+ <literal>sync</literal> (execute asynchronous-eligible I/O synchronously)
+ </para>
+ </listitem>
+ </itemizedlist>
+ </para>
+ <para>
+ This parameter can only be set at server start.
+ </para>
+ </listitem>
+ </varlistentry>
+
</variablelist>
</sect2>
diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list
index 93339ef3c58..3c9e823f07e 100644
--- a/src/tools/pgindent/typedefs.list
+++ b/src/tools/pgindent/typedefs.list
@@ -1279,6 +1279,7 @@ IntoClause
InvalMessageArray
InvalidationInfo
InvalidationMsgsGroup
+IoMethod
IpcMemoryId
IpcMemoryKey
IpcMemoryState
--
2.48.1.76.g4e746b1a31.dirty
[text/x-diff] v2.8-0002-aio-Add-core-asynchronous-I-O-infrastructure.patch (89.8K, ../../3yxd5r23zly5bytvgyktbxtxq2r3gbpi7xd4dugevh3h4w4q6c@lu6oatjjpltz/3-v2.8-0002-aio-Add-core-asynchronous-I-O-infrastructure.patch)
download | inline diff:
From ad3a1330e96ce3f3f271cea29d5e58e83a832527 Mon Sep 17 00:00:00 2001
From: Andres Freund <andres@anarazel.de>
Date: Fri, 7 Mar 2025 10:14:31 -0500
Subject: [PATCH v2.8 02/38] aio: Add core asynchronous I/O infrastructure
The main motivations to use AIO in PostgreSQL are:
a) Reduce the time spent waiting for IO by issuing IO sufficiently early.
In a few places we have approximated this using posix_fadvise() based
prefetching, but that is fairly limited (no completion feedback, double the
syscalls, only works with buffered IO, only works on some OSs).
b) Allow to use Direct-I/O (DIO).
DIO can offload most of the work for IO to hardware and thus increase
throughput / decrease CPU utilization, as well as reduce latency. While we
have gained the ability to configure DIO in d4e71df6, it is not yet usable
for real world workloads, as every IO is executed synchronously.
For portability, the new AIO infrastructure allows to implement AIO using
different methods. The choice of the AIO method is controlled by a new
io_method GUC. As of this commit, the only implemented method is "sync",
i.e. AIO is not actually executed asynchronously. The "sync" method exists to
allow to bypass most of the new code initially.
Subsequent commits will introduce additional IO methods, including a
cross-platform method implemented using worker processes and a linux specific
method using io_uring.
To allow different parts of postgres to use AIO, the core AIO infrastructure
does not need to know what kind of files it is operating on. The necessary
behavioral differences for different files are abstracted as "AIO
Targets". One example target would be smgr. For boring portability reasons all
targets currently need to be added to an array in aio_target.c. This commit
does not implement any AIO targets, just the infrastructure for them. The smgr
target will be added in a later commit.
Completion (and other events) of IOs for one type of file (i.e. one AIO
target) need to be reacted to differently based on the IO operation and the
callsite. This is made possible by callbacks that can be registered on
IOs. E.g. an smgr read into a local buffer does not need to update the
corresponding BufferDesc (as there is none), but a read into shared buffers
does. This commit does not contain any callbacks, they will be added in
subsequent commits.
For now the AIO infrastructure only understands READV and WRITEV operations,
but it is expected that more operations will be added. E.g. fsync/fdatasync,
flush_range and network operations like send/recv.
As of this commit nothing uses the AIO infrastructure. Other commits will add
an smgr target, md.c and bufmgr.c callbacks and then finally use AIO for
read_stream.c IO, which, in one fell swoop, will convert all read stream users
to AIO.
The goal is to use AIO in many more places. There are patches to use AIO for
checkpointer and bgwriter that are reasonably close to being ready. There also
are prototypes to use it for WAL, relation extension, backend writes and many
more. Those prototypes were important to ensure the design of the AIO
subsystem is not too limiting (e.g. WAL writes need to happen in critical
sections, which influenced a lot of the design).
A future commit will add an AIO README explaining the AIO architecture and how
to use the AIO subsystem. The README is added later, as it references details
only added in later commits.
Many many more people than the folks named below have contributed with
feedback, work on semi-independent patches etc. E.g. various folks have
contributed patches to use the read stream infrastructure (added by Thomas in
b5a9b18cd0b) in more places. Similarly, a *lot* of folks have contributed to
the CI infrastructure, that I started to work on to make adding AIO feasible.
Some of the work by contributors has gone into the "v1" prototype of AIO,
which heavily influenced the current design of the AIO subsystem. None of the
code from that directly survives, but without the prototype, the current
version of the AIO infrastructure would not exist.
Similarly, the reviewers below have not necessarily looked at the current
design or the whole infrastructure, but have provided very valuable input. I
am to blame for problems, not they.
Author: Andres Freund <andres@anarazel.de>
Co-authored-by: Thomas Munro <thomas.munro@gmail.com>
Co-authored-by: Nazir Bilal Yavuz <byavuz81@gmail.com>
Co-authored-by: Melanie Plageman <melanieplageman@gmail.com>
Reviewed-by: Heikki Linnakangas <hlinnaka@iki.fi>
Reviewed-by: Noah Misch <noah@leadboat.com>
Reviewed-by: Jakub Wartak <jakub.wartak@enterprisedb.com>
Reviewed-by: Melanie Plageman <melanieplageman@gmail.com>
Reviewed-by: Robert Haas <robertmhaas@gmail.com>
Reviewed-by: Dmitry Dolgov <9erthalion6@gmail.com>
Reviewed-by: Antonin Houska <ah@cybertec.at>
Discussion: https://postgr.es/m/uvrtrknj4kdytuboidbhwclo4gxhswwcpgadptsjvjqcluzmah%40brqs62irg4dt
Discussion: https://postgr.es/m/20210223100344.llw5an2aklengrmn@alap3.anarazel.de
Discussion: https://postgr.es/m/stj36ea6yyhoxtqkhpieia2z4krnam7qyetc57rfezgk4zgapf@gcnactj4z56m
---
src/include/storage/aio.h | 304 +++++
src/include/storage/aio_internal.h | 395 ++++++
src/include/storage/aio_types.h | 117 ++
src/backend/storage/aio/Makefile | 4 +
src/backend/storage/aio/aio.c | 1131 +++++++++++++++++
src/backend/storage/aio/aio_callback.c | 307 +++++
src/backend/storage/aio/aio_init.c | 187 +++
src/backend/storage/aio/aio_io.c | 182 +++
src/backend/storage/aio/aio_target.c | 118 ++
src/backend/storage/aio/meson.build | 4 +
src/backend/storage/aio/method_sync.c | 47 +
.../utils/activity/wait_event_names.txt | 1 +
src/tools/pgindent/typedefs.list | 21 +
13 files changed, 2818 insertions(+)
create mode 100644 src/include/storage/aio_internal.h
create mode 100644 src/include/storage/aio_types.h
create mode 100644 src/backend/storage/aio/aio_callback.c
create mode 100644 src/backend/storage/aio/aio_io.c
create mode 100644 src/backend/storage/aio/aio_target.c
create mode 100644 src/backend/storage/aio/method_sync.c
diff --git a/src/include/storage/aio.h b/src/include/storage/aio.h
index e79d5343038..a0a0f23f3a5 100644
--- a/src/include/storage/aio.h
+++ b/src/include/storage/aio.h
@@ -3,6 +3,10 @@
* aio.h
* Main AIO interface
*
+ * This is the header to include when actually issuing AIO. When just
+ * declaring functions involving an AIO related type, it might suffice to
+ * include aio_types.h. Initialization related functions are in the dedicated
+ * aio_init.h.
*
* Portions Copyright (c) 1996-2025, PostgreSQL Global Development Group
* Portions Copyright (c) 1994, Regents of the University of California
@@ -14,6 +18,9 @@
#ifndef AIO_H
#define AIO_H
+#include "storage/aio_types.h"
+#include "storage/procnumber.h"
+
/* Enum for io_method GUC. */
@@ -26,9 +33,306 @@ typedef enum IoMethod
#define DEFAULT_IO_METHOD IOMETHOD_SYNC
+/*
+ * Flags for an IO that can be set with pgaio_io_set_flag().
+ */
+typedef enum PgAioHandleFlags
+{
+ /*
+ * The IO references backend local memory.
+ *
+ * This needs to be set on an IO whenever the IO references process-local
+ * memory. Some IO methods do not support executing IO that references
+ * process local memory and thus need to fall back to executing IO
+ * synchronously for IOs with this flag set.
+ *
+ * Required for correctness.
+ */
+ PGAIO_HF_REFERENCES_LOCAL = 1 << 1,
+
+ /*
+ * Hint that IO will be executed synchronously.
+ *
+ * This can make it a bit cheaper to execute synchronous IO via the AIO
+ * interface, to avoid needing an AIO and non-AIO version of code.
+ *
+ * Advantageous to set, if applicable, but not required for correctness.
+ */
+ PGAIO_HF_SYNCHRONOUS = 1 << 0,
+
+ /*
+ * IO is using buffered IO, used to control heuristic in some IO methods.
+ *
+ * Advantageous to set, if applicable, but not required for correctness.
+ */
+ PGAIO_HF_BUFFERED = 1 << 2,
+} PgAioHandleFlags;
+
+/*
+ * The IO operations supported by the AIO subsystem.
+ *
+ * This could be in aio_internal.h, as it is not pubicly referenced, but
+ * PgAioOpData currently *does* need to be public, therefore keeping this
+ * public seems to make sense.
+ */
+typedef enum PgAioOp
+{
+ /* intentionally the zero value, to help catch zeroed memory etc */
+ PGAIO_OP_INVALID = 0,
+
+ PGAIO_OP_READV,
+ PGAIO_OP_WRITEV,
+
+ /**
+ * In the near term we'll need at least:
+ * - fsync / fdatasync
+ * - flush_range
+ *
+ * Eventually we'll additionally want at least:
+ * - send
+ * - recv
+ * - accept
+ **/
+} PgAioOp;
+
+#define PGAIO_OP_COUNT (PGAIO_OP_WRITEV + 1)
+
+
+/*
+ * On what is IO being performed?
+ *
+ * PgAioTargetID specific behaviour should be implemented in
+ * aio_target.c.
+ */
+typedef enum PgAioTargetID
+{
+ /* intentionally the zero value, to help catch zeroed memory etc */
+ PGAIO_TID_INVALID = 0,
+} PgAioTargetID;
+
+#define PGAIO_TID_COUNT (PGAIO_TID_INVALID + 1)
+
+
+/*
+ * Data necessary for support IO operations (see PgAioOp).
+ *
+ * NB: Note that the FDs in here may *not* be relied upon for re-issuing
+ * requests (e.g. for partial reads/writes) - the FD might be from another
+ * process, or closed since. That's not a problem for IOs waiting to be issued
+ * only because the queue is flushed when closing an FD.
+ */
+typedef union
+{
+ struct
+ {
+ int fd;
+ uint16 iov_length;
+ uint64 offset;
+ } read;
+
+ struct
+ {
+ int fd;
+ uint16 iov_length;
+ uint64 offset;
+ } write;
+} PgAioOpData;
+
+
+/*
+ * Information the object that IO is executed on. Mostly callbacks that
+ * operate on PgAioTargetData.
+ *
+ * typedef is in aio_types.h
+ */
+struct PgAioTargetInfo
+{
+ void (*reopen) (PgAioHandle *ioh);
+
+ char *(*describe_identity) (const PgAioTargetData *sd);
+
+ const char *name;
+};
+
+
+/*
+ * IDs for callbacks that can be registered on an IO.
+ *
+ * Callbacks are identified by an ID rather than a function pointer. There are
+ * two main reasons:
+ *
+ * 1) Memory within PgAioHandle is precious, due to the number of PgAioHandle
+ * structs in pre-allocated shared memory.
+ *
+ * 2) Due to EXEC_BACKEND function pointers are not necessarily stable between
+ * different backends, therefore function pointers cannot directly be in
+ * shared memory.
+ *
+ * Without 2), we could fairly easily allow to add new callbacks, by filling a
+ * ID->pointer mapping table on demand. In the presence of 2 that's still
+ * doable, but harder, because every process has to re-register the pointers
+ * so that a local ID->"backend local pointer" mapping can be maintained.
+ */
+typedef enum PgAioHandleCallbackID
+{
+ PGAIO_HCB_INVALID,
+} PgAioHandleCallbackID;
+
+
+typedef void (*PgAioHandleCallbackStage) (PgAioHandle *ioh, uint8 cb_flags);
+typedef PgAioResult (*PgAioHandleCallbackComplete) (PgAioHandle *ioh, PgAioResult prior_result, uint8 cb_flags);
+typedef void (*PgAioHandleCallbackReport) (PgAioResult result, const PgAioTargetData *target_data, int elevel);
+
+/* typedef is in aio_types.h */
+struct PgAioHandleCallbacks
+{
+ /*
+ * Prepare resources affected by the IO for execution. This could e.g.
+ * include moving ownership of buffer pins to the AIO subsystem.
+ */
+ PgAioHandleCallbackStage stage;
+
+ /*
+ * Update the state of resources affected by the IO to reflect completion
+ * of the IO. This could e.g. include updating shared buffer state to
+ * signal the IO has finished.
+ *
+ * The _shared suffix indicates that this is executed by the backend that
+ * completed the IO, which may or may not be the backend that issued the
+ * IO. Obviously the callback thus can only modify resources in shared
+ * memory.
+ *
+ * The latest registered callback is called first. This allows
+ * higher-level code to register callbacks that can rely on callbacks
+ * registered by lower-level code to already have been executed.
+ *
+ * NB: This is called in a critical section. Errors can be signalled by
+ * the callback's return value, it's the responsibility of the IO's issuer
+ * to react appropriately.
+ */
+ PgAioHandleCallbackComplete complete_shared;
+
+ /*
+ * Like complete_shared, except called in the issuing backend.
+ *
+ * This variant of the completion callback is useful when backend-local
+ * state has to be updated to reflect the IO's completion. E.g. a
+ * temporary buffer's BufferDesc isn't accessible in complete_shared.
+ *
+ * Local callbacks are only called after complete_shared for all
+ * registered callbacks has been called.
+ */
+ PgAioHandleCallbackComplete complete_local;
+
+ /*
+ * Report the result of an IO operation. This is e.g. used to raise an
+ * error after an IO failed at the appropriate time (i.e. not when the IO
+ * failed, but under control of the code that issued the IO).
+ */
+ PgAioHandleCallbackReport report;
+};
+
+
+
+/*
+ * How many callbacks can be registered for one IO handle. Currently we only
+ * need two, but it's not hard to imagine needing a few more.
+ */
+#define PGAIO_HANDLE_MAX_CALLBACKS 4
+
+
+
+/* --------------------------------------------------------------------------------
+ * IO Handles
+ * --------------------------------------------------------------------------------
+ */
+
+/* functions in aio.c */
+struct ResourceOwnerData;
+extern PgAioHandle *pgaio_io_acquire(struct ResourceOwnerData *resowner, PgAioReturn *ret);
+extern PgAioHandle *pgaio_io_acquire_nb(struct ResourceOwnerData *resowner, PgAioReturn *ret);
+
+extern void pgaio_io_release(PgAioHandle *ioh);
struct dlist_node;
extern void pgaio_io_release_resowner(struct dlist_node *ioh_node, bool on_error);
+extern void pgaio_io_set_flag(PgAioHandle *ioh, PgAioHandleFlags flag);
+
+extern int pgaio_io_get_id(PgAioHandle *ioh);
+extern ProcNumber pgaio_io_get_owner(PgAioHandle *ioh);
+
+extern void pgaio_io_get_wref(PgAioHandle *ioh, PgAioWaitRef *iow);
+
+/* functions in aio_io.c */
+struct iovec;
+extern int pgaio_io_get_iovec(PgAioHandle *ioh, struct iovec **iov);
+
+extern PgAioOpData *pgaio_io_get_op_data(PgAioHandle *ioh);
+
+extern void pgaio_io_prep_readv(PgAioHandle *ioh,
+ int fd, int iovcnt, uint64 offset);
+extern void pgaio_io_prep_writev(PgAioHandle *ioh,
+ int fd, int iovcnt, uint64 offset);
+
+/* functions in aio_target.c */
+extern void pgaio_io_set_target(PgAioHandle *ioh, PgAioTargetID targetid);
+extern bool pgaio_io_has_target(PgAioHandle *ioh);
+extern PgAioTargetData *pgaio_io_get_target_data(PgAioHandle *ioh);
+extern char *pgaio_io_get_target_description(PgAioHandle *ioh);
+
+/* functions in aio_callback.c */
+extern void pgaio_io_register_callbacks(PgAioHandle *ioh, PgAioHandleCallbackID cb_id,
+ uint8 cb_data);
+extern void pgaio_io_set_handle_data_64(PgAioHandle *ioh, uint64 *data, uint8 len);
+extern void pgaio_io_set_handle_data_32(PgAioHandle *ioh, uint32 *data, uint8 len);
+extern uint64 *pgaio_io_get_handle_data(PgAioHandle *ioh, uint8 *len);
+
+
+
+/* --------------------------------------------------------------------------------
+ * IO Wait References
+ * --------------------------------------------------------------------------------
+ */
+
+extern void pgaio_wref_clear(PgAioWaitRef *iow);
+extern bool pgaio_wref_valid(PgAioWaitRef *iow);
+extern int pgaio_wref_get_id(PgAioWaitRef *iow);
+
+extern void pgaio_wref_wait(PgAioWaitRef *iow);
+extern bool pgaio_wref_check_done(PgAioWaitRef *iow);
+
+
+
+/* --------------------------------------------------------------------------------
+ * IO Result
+ * --------------------------------------------------------------------------------
+ */
+
+extern void pgaio_result_report(PgAioResult result, const PgAioTargetData *target_data,
+ int elevel);
+
+
+
+/* --------------------------------------------------------------------------------
+ * Actions on multiple IOs.
+ * --------------------------------------------------------------------------------
+ */
+
+extern void pgaio_enter_batchmode(void);
+extern void pgaio_exit_batchmode(void);
+extern void pgaio_submit_staged(void);
+extern bool pgaio_have_staged(void);
+
+
+
+/* --------------------------------------------------------------------------------
+ * Other
+ * --------------------------------------------------------------------------------
+ */
+
+extern void pgaio_closing_fd(int fd);
+
+
/* GUCs */
extern PGDLLIMPORT int io_method;
diff --git a/src/include/storage/aio_internal.h b/src/include/storage/aio_internal.h
new file mode 100644
index 00000000000..52b7751efa8
--- /dev/null
+++ b/src/include/storage/aio_internal.h
@@ -0,0 +1,395 @@
+/*-------------------------------------------------------------------------
+ *
+ * aio_internal.h
+ * AIO related declarations that should only be used by the AIO subsystem
+ * internally.
+ *
+ *
+ * Portions Copyright (c) 1996-2025, PostgreSQL Global Development Group
+ * Portions Copyright (c) 1994, Regents of the University of California
+ *
+ * src/include/storage/aio_internal.h
+ *
+ *-------------------------------------------------------------------------
+ */
+#ifndef AIO_INTERNAL_H
+#define AIO_INTERNAL_H
+
+
+#include "lib/ilist.h"
+#include "port/pg_iovec.h"
+#include "storage/aio.h"
+#include "storage/condition_variable.h"
+
+
+/*
+ * The maximum number of IOs that can be batch submitted at once.
+ */
+#define PGAIO_SUBMIT_BATCH_SIZE 32
+
+
+
+/*
+ * State machine for handles. With some exceptions, noted below, handles move
+ * linearly through all states.
+ *
+ * State changes should all go through pgaio_io_update_state().
+ */
+typedef enum PgAioHandleState
+{
+ /* not in use */
+ PGAIO_HS_IDLE = 0,
+
+ /*
+ * Returned by pgaio_io_acquire(). The next state is either DEFINED (if
+ * pgaio_io_prep_*() is called), or IDLE (if pgaio_io_release() is
+ * called).
+ */
+ PGAIO_HS_HANDED_OUT,
+
+ /*
+ * pgaio_io_prep_*() has been called, but IO is not yet staged. At this
+ * point the handle has all the information for the IO to be executed.
+ */
+ PGAIO_HS_DEFINED,
+
+ /*
+ * stage() callbacks have been called, handle ready to be submitted for
+ * execution. Unless in batchmode (see c.f. pgaio_enter_batchmode()), the
+ * IO will be submitted immediately after.
+ */
+ PGAIO_HS_STAGED,
+
+ /* IO has been submitted to the IO method for execution */
+ PGAIO_HS_SUBMITTED,
+
+ /* IO finished, but result has not yet been processed */
+ PGAIO_HS_COMPLETED_IO,
+
+ /*
+ * IO completed, shared completion has been called.
+ *
+ * If the IO completion occurs in the issuing backend, local callbacks
+ * will immediately be called. Otherwise the handle stays in
+ * COMPLETED_SHARED until the issuing backend waits for the completion of
+ * the IO.
+ */
+ PGAIO_HS_COMPLETED_SHARED,
+
+ /*
+ * IO completed, local completion has been called.
+ *
+ * After this the handle will be made reusable and go into IDLE state.
+ */
+ PGAIO_HS_COMPLETED_LOCAL,
+} PgAioHandleState;
+
+
+struct ResourceOwnerData;
+
+/* typedef is in aio_types.h */
+struct PgAioHandle
+{
+ /* all state updates should go through pgaio_io_update_state() */
+ PgAioHandleState state:8;
+
+ /* what are we operating on */
+ PgAioTargetID target:8;
+
+ /* which IO operation */
+ PgAioOp op:8;
+
+ /* bitfield of PgAioHandleFlags */
+ uint8 flags;
+
+ uint8 num_callbacks;
+
+ /* using the proper type here would use more space */
+ uint8 callbacks[PGAIO_HANDLE_MAX_CALLBACKS];
+
+ /* data forwarded to each callback */
+ uint8 callbacks_data[PGAIO_HANDLE_MAX_CALLBACKS];
+
+ /*
+ * Length of data associated with handle using
+ * pgaio_io_set_handle_data_*().
+ */
+ uint8 handle_data_len;
+
+ /* XXX: could be optimized out with some pointer math */
+ int32 owner_procno;
+
+ /* raw result of the IO operation */
+ int32 result;
+
+ /**
+ * In which list the handle is registered, depends on the state:
+ * - IDLE, in per-backend list
+ * - HANDED_OUT - not in a list
+ * - DEFINED - in per-backend staged list
+ * - STAGED - in per-backend staged list
+ * - SUBMITTED - in issuer's in_flight list
+ * - COMPLETED_IO - in issuer's in_flight list
+ * - COMPLETED_SHARED - in issuer's in_flight list
+ **/
+ dlist_node node;
+
+ struct ResourceOwnerData *resowner;
+ dlist_node resowner_node;
+
+ /* incremented every time the IO handle is reused */
+ uint64 generation;
+
+ /*
+ * To wait for the IO to complete other backends can wait on this CV. Note
+ * that, if in SUBMITTED state, a waiter first needs to check if it needs
+ * to do work via IoMethodOps->wait_one().
+ */
+ ConditionVariable cv;
+
+ /* result of shared callback, passed to issuer callback */
+ PgAioResult distilled_result;
+
+ /*
+ * Index into PgAioCtl->iovecs and PgAioCtl->handle_data.
+ *
+ * At the moment there's no need to differentiate between the two, but
+ * that won't necessarily stay that way.
+ */
+ uint32 iovec_off;
+
+ /*
+ * If not NULL, this memory location will be updated with information
+ * about the IOs completion iff the issuing backend learns about the IOs
+ * completion.
+ */
+ PgAioReturn *report_return;
+
+ /* Data necessary for the IO to be performed */
+ PgAioOpData op_data;
+
+ /*
+ * Data necessary to identify the object undergoing IO to higher-level
+ * code. Needs to be sufficient to allow another backend to reopen the
+ * file.
+ */
+ PgAioTargetData target_data;
+};
+
+
+typedef struct PgAioBackend
+{
+ /* index into PgAioCtl->io_handles */
+ uint32 io_handle_off;
+
+ /* IO Handles that currently are not used */
+ dclist_head idle_ios;
+
+ /*
+ * Only one IO may be returned by pgaio_io_acquire()/pgaio_io_acquire_nb()
+ * without having been either defined (by actually associating it with IO)
+ * or released (with pgaio_io_release()). This restriction is necessary to
+ * guarantee that we always can acquire an IO. ->handed_out_io is used to
+ * enforce that rule.
+ */
+ PgAioHandle *handed_out_io;
+
+ /* Are we currently in batchmode? See pgaio_enter_batchmode(). */
+ bool in_batchmode;
+
+ /*
+ * IOs that are defined, but not yet submitted.
+ */
+ uint16 num_staged_ios;
+ PgAioHandle *staged_ios[PGAIO_SUBMIT_BATCH_SIZE];
+
+ /*
+ * List of in-flight IOs. Also contains IOs that aren't strictly speaking
+ * in-flight anymore, but have been waited-for and completed by another
+ * backend. Once this backend sees such an IO it'll be reclaimed.
+ *
+ * The list is ordered by submission time, with more recently submitted
+ * IOs being appended at the end.
+ */
+ dclist_head in_flight_ios;
+} PgAioBackend;
+
+
+typedef struct PgAioCtl
+{
+ int backend_state_count;
+ PgAioBackend *backend_state;
+
+ /*
+ * Array of iovec structs. Each iovec is owned by a specific backend. The
+ * allocation is in PgAioCtl to allow the maximum number of iovecs for
+ * individual IOs to be configurable with PGC_POSTMASTER GUC.
+ */
+ uint32 iovec_count;
+ struct iovec *iovecs;
+
+ /*
+ * For, e.g., an IO covering multiple buffers in shared / temp buffers, we
+ * need to get Buffer IDs during completion to be able to change the
+ * BufferDesc state accordingly. This space can be used to store e.g.
+ * Buffer IDs. Note that the actual iovec might be shorter than this,
+ * because we combine neighboring pages into one larger iovec entry.
+ */
+ uint64 *handle_data;
+
+ uint32 io_handle_count;
+ PgAioHandle *io_handles;
+} PgAioCtl;
+
+
+
+/*
+ * Callbacks used to implement an IO method.
+ */
+typedef struct IoMethodOps
+{
+ /* global initialization */
+
+ /*
+ * Amount of additional shared memory to reserve for the io_method. Called
+ * just like a normal ipci.c style *Size() function. Optional.
+ */
+ size_t (*shmem_size) (void);
+
+ /*
+ * Initialize shared memory. First time is true if AIO's shared memory was
+ * just initialized, false otherwise. Optional.
+ */
+ void (*shmem_init) (bool first_time);
+
+ /*
+ * Per-backend initialization. Optional.
+ */
+ void (*init_backend) (void);
+
+
+ /* handling of IOs */
+
+ /* optional */
+ bool (*needs_synchronous_execution) (PgAioHandle *ioh);
+
+ /*
+ * Start executing passed in IOs.
+ *
+ * Will not be called if ->needs_synchronous_execution() returned true.
+ *
+ * num_staged_ios is <= PGAIO_SUBMIT_BATCH_SIZE.
+ *
+ * Always called in a critical section.
+ */
+ int (*submit) (uint16 num_staged_ios, PgAioHandle **staged_ios);
+
+ /*
+ * Wait for the IO to complete. Optional.
+ *
+ * If not provided, it needs to be guaranteed that the IO method calls
+ * pgaio_io_process_completion() without further interaction by the
+ * issuing backend.
+ */
+ void (*wait_one) (PgAioHandle *ioh,
+ uint64 ref_generation);
+} IoMethodOps;
+
+
+/* aio.c */
+extern bool pgaio_io_was_recycled(PgAioHandle *ioh, uint64 ref_generation, PgAioHandleState *state);
+extern void pgaio_io_stage(PgAioHandle *ioh, PgAioOp op);
+extern void pgaio_io_process_completion(PgAioHandle *ioh, int result);
+extern void pgaio_io_prepare_submit(PgAioHandle *ioh);
+extern bool pgaio_io_needs_synchronous_execution(PgAioHandle *ioh);
+extern const char *pgaio_io_get_state_name(PgAioHandle *ioh);
+const char *pgaio_result_status_string(PgAioResultStatus rs);
+extern void pgaio_shutdown(int code, Datum arg);
+
+/* aio_callback.c */
+extern void pgaio_io_call_stage(PgAioHandle *ioh);
+extern void pgaio_io_call_complete_shared(PgAioHandle *ioh);
+extern void pgaio_io_call_complete_local(PgAioHandle *ioh);
+
+/* aio_io.c */
+extern void pgaio_io_perform_synchronously(PgAioHandle *ioh);
+extern const char *pgaio_io_get_op_name(PgAioHandle *ioh);
+
+/* aio_target.c */
+extern bool pgaio_io_can_reopen(PgAioHandle *ioh);
+extern void pgaio_io_reopen(PgAioHandle *ioh);
+extern const char *pgaio_io_get_target_name(PgAioHandle *ioh);
+
+
+/*
+ * The AIO subsystem has fairly verbose debug logging support. This can be
+ * enabled/disabled at build time. The reason for this is that
+ * a) the verbosity can make debugging things on higher levels hard
+ * b) even if logging can be skipped due to elevel checks, it still causes a
+ * measurable slowdown
+ *
+ * XXX: This likely should be eventually be disabled by default, at least in
+ * non-assert builds.
+ */
+#define PGAIO_VERBOSE 1
+
+/*
+ * Simple ereport() wrapper that only logs if PGAIO_VERBOSE is defined.
+ *
+ * This intentionally still compiles the code, guarded by a constant if (0),
+ * if verbose logging is disabled, to make it less likely that debug logging
+ * is silently broken.
+ *
+ * The current definition requires passing at least one argument.
+ */
+#define pgaio_debug(elevel, msg, ...) \
+ do { \
+ if (PGAIO_VERBOSE) \
+ ereport(elevel, \
+ errhidestmt(true), errhidecontext(true), \
+ errmsg_internal(msg, \
+ __VA_ARGS__)); \
+ } while(0)
+
+/*
+ * Simple ereport() wrapper. Note that the definition requires passing at
+ * least one argument.
+ */
+#define pgaio_debug_io(elevel, ioh, msg, ...) \
+ pgaio_debug(elevel, "io %-10d|op %-5s|target %-4s|state %-16s: " msg, \
+ pgaio_io_get_id(ioh), \
+ pgaio_io_get_op_name(ioh), \
+ pgaio_io_get_target_name(ioh), \
+ pgaio_io_get_state_name(ioh), \
+ __VA_ARGS__)
+
+
+#ifdef USE_INJECTION_POINTS
+
+extern void pgaio_io_call_inj(PgAioHandle *ioh, const char *injection_point);
+
+/* just for use in tests, from within injection points */
+extern PgAioHandle *pgaio_inj_io_get(void);
+
+#else
+
+#define pgaio_io_call_inj(ioh, injection_point) (void) 0
+
+/*
+ * no fallback for pgaio_inj_io_get, all code using injection points better be
+ * guarded by USE_INJECTION_POINTS.
+ */
+
+#endif
+
+
+/* Declarations for the tables of function pointers exposed by each IO method. */
+extern PGDLLIMPORT const IoMethodOps pgaio_sync_ops;
+
+extern PGDLLIMPORT const IoMethodOps *pgaio_method_ops;
+extern PGDLLIMPORT PgAioCtl *pgaio_ctl;
+extern PGDLLIMPORT PgAioBackend *pgaio_my_backend;
+
+
+
+#endif /* AIO_INTERNAL_H */
diff --git a/src/include/storage/aio_types.h b/src/include/storage/aio_types.h
new file mode 100644
index 00000000000..a5cc658efbd
--- /dev/null
+++ b/src/include/storage/aio_types.h
@@ -0,0 +1,117 @@
+/*-------------------------------------------------------------------------
+ *
+ * aio_types.h
+ * AIO related types that are useful to include separately, to reduce the
+ * "include burden".
+ *
+ *
+ * Portions Copyright (c) 1996-2025, PostgreSQL Global Development Group
+ * Portions Copyright (c) 1994, Regents of the University of California
+ *
+ * src/include/storage/aio_types.h
+ *
+ *-------------------------------------------------------------------------
+ */
+#ifndef AIO_TYPES_H
+#define AIO_TYPES_H
+
+#include "storage/block.h"
+#include "storage/relfilelocator.h"
+
+
+typedef struct PgAioHandle PgAioHandle;
+typedef struct PgAioHandleCallbacks PgAioHandleCallbacks;
+typedef struct PgAioTargetInfo PgAioTargetInfo;
+
+/*
+ * A reference to an IO that can be used to wait for the IO (using
+ * pgaio_wref_wait()) to complete.
+ *
+ * These can be passed across process boundaries.
+ */
+typedef struct PgAioWaitRef
+{
+ /* internal ID identifying the specific PgAioHandle */
+ uint32 aio_index;
+
+ /*
+ * IO handles are reused. To detect if a handle was reused, and thereby
+ * avoid unnecessarily waiting for a newer IO, each time the handle is
+ * reused a generation number is increased.
+ *
+ * To avoid requiring alignment sufficient for an int64, split the
+ * generation into two.
+ */
+ uint32 generation_upper;
+ uint32 generation_lower;
+} PgAioWaitRef;
+
+
+/*
+ * Information identifying what the IO is being performed on.
+ *
+ * This needs sufficient information to
+ *
+ * a) Reopen the file for the IO if the IO is executed in a context that
+ * cannot use the FD provided initially (e.g. because the IO is executed in
+ * a worker process).
+ *
+ * b) Describe the object the IO is performed on in log / error messages.
+ */
+typedef union PgAioTargetData
+{
+ /* just as an example placeholder for later */
+ struct
+ {
+ uint32 queue_id;
+ } wal;
+} PgAioTargetData;
+
+
+/*
+ * The status of an AIO operation.
+ */
+typedef enum PgAioResultStatus
+{
+ ARS_UNKNOWN, /* not yet completed / uninitialized */
+ ARS_OK,
+ ARS_PARTIAL, /* did not fully succeed, but no error */
+ ARS_ERROR,
+} PgAioResultStatus;
+
+
+/*
+ * Result of IO operation, visible only to the initiator of IO.
+ */
+typedef struct PgAioResult
+{
+ /*
+ * This is of type PgAioHandleCallbackID, but can't use a bitfield of an
+ * enum, because some compilers treat enums as signed.
+ */
+ uint32 id:8;
+
+ /* of type PgAioResultStatus, see above */
+ uint32 status:2;
+
+ /* meaning defined by callback->error */
+ uint32 error_data:22;
+
+ int32 result;
+} PgAioResult;
+
+
+/*
+ * Combination of PgAioResult with minimal metadata about the IO.
+ *
+ * Contains sufficient information to be able, in case the IO [partially]
+ * fails, to log/raise an error under control of the IO issuing code.
+ */
+typedef struct PgAioReturn
+{
+ PgAioResult result;
+ PgAioTargetData target_data;
+} PgAioReturn;
+
+
+#endif /* AIO_TYPES_H */
diff --git a/src/backend/storage/aio/Makefile b/src/backend/storage/aio/Makefile
index eaeaeeee8e3..89f821ea7e1 100644
--- a/src/backend/storage/aio/Makefile
+++ b/src/backend/storage/aio/Makefile
@@ -10,7 +10,11 @@ include $(top_builddir)/src/Makefile.global
OBJS = \
aio.o \
+ aio_callback.o \
aio_init.o \
+ aio_io.o \
+ aio_target.o \
+ method_sync.o \
read_stream.o
include $(top_srcdir)/src/backend/common.mk
diff --git a/src/backend/storage/aio/aio.c b/src/backend/storage/aio/aio.c
index 828a94efdc3..b02349cbf99 100644
--- a/src/backend/storage/aio/aio.c
+++ b/src/backend/storage/aio/aio.c
@@ -3,6 +3,28 @@
* aio.c
* AIO - Core Logic
*
+ * For documentation about how AIO works on a higher level, including a
+ * schematic example, see README.md.
+ *
+ *
+ * AIO is a complicated subsystem. To keep things navigable it is split across
+ * a number of files:
+ *
+ * - method_*.c - different ways of executing AIO (e.g. worker process)
+ *
+ * - aio_target.c - IO on different kinds of targets
+ *
+ * - aio_io.c - method-independent code for specific IO ops (e.g. readv)
+ *
+ * - aio_callback.c - callbacks at IO operation lifecycle events
+ *
+ * - aio_init.c - per-server and per-backend initialization
+ *
+ * - aio.c - all other topics
+ *
+ * - read_stream.c - helper for reading buffered relation data
+ *
+ *
* Portions Copyright (c) 1996-2025, PostgreSQL Global Development Group
* Portions Copyright (c) 1994, Regents of the University of California
*
@@ -15,10 +37,28 @@
#include "postgres.h"
#include "lib/ilist.h"
+#include "miscadmin.h"
+#include "port/atomics.h"
#include "storage/aio.h"
+#include "storage/aio_internal.h"
#include "storage/aio_subsys.h"
#include "utils/guc.h"
#include "utils/guc_hooks.h"
+#include "utils/resowner.h"
+#include "utils/wait_event_types.h"
+
+#ifdef USE_INJECTION_POINTS
+#include "utils/injection_point.h"
+#endif
+
+
+static inline void pgaio_io_update_state(PgAioHandle *ioh, PgAioHandleState new_state);
+static void pgaio_io_reclaim(PgAioHandle *ioh);
+static void pgaio_io_resowner_register(PgAioHandle *ioh);
+static void pgaio_io_wait_for_free(void);
+static PgAioHandle *pgaio_io_from_wref(PgAioWaitRef *iow, uint64 *ref_generation);
+static const char *pgaio_io_state_get_name(PgAioHandleState s);
+static void pgaio_io_wait(PgAioHandle *ioh, uint64 ref_generation);
/* Options for io_method. */
@@ -31,7 +71,182 @@ const struct config_enum_entry io_method_options[] = {
int io_method = DEFAULT_IO_METHOD;
int io_max_concurrency = -1;
+/* global control for AIO */
+PgAioCtl *pgaio_ctl;
+/* current backend's per-backend state */
+PgAioBackend *pgaio_my_backend;
+
+
+static const IoMethodOps *const pgaio_method_ops_table[] = {
+ [IOMETHOD_SYNC] = &pgaio_sync_ops,
+};
+
+/* callbacks for the configured io_method, set by assign_io_method */
+const IoMethodOps *pgaio_method_ops;
+
+
+/*
+ * Currently there's no infrastructure to pass arguments to injection points,
+ * so we instead set this up for the duration of the injection point
+ * invocation. See pgaio_io_call_inj().
+ */
+#ifdef USE_INJECTION_POINTS
+static PgAioHandle *pgaio_inj_cur_handle;
+#endif
+
+
+
+/* --------------------------------------------------------------------------------
+ * Public Functions related to PgAioHandle
+ * --------------------------------------------------------------------------------
+ */
+
+/*
+ * Acquire an AioHandle, waiting for IO completion if necessary.
+ *
+ * Each backend can only have one AIO handle that has been "handed out" to
+ * code, but not yet submitted or released. This restriction is necessary to
+ * ensure that it is possible for code to wait for an unused handle by waiting
+ * for in-flight IO to complete. There is a limited number of handles in each
+ * backend, if multiple handles could be handed out without being submitted,
+ * waiting for all in-flight IO to complete would not guarantee that handles
+ * free up.
+ *
+ * It is cheap to acquire an IO handle, unless all handles are in use. In that
+ * case this function waits for the oldest IO to complete. In case that is not
+ * desirable, see pgaio_io_acquire_nb().
+ *
+ * If a handle was acquired but then does not turn out to be needed,
+ * e.g. because pgaio_io_acquire() is called before starting an IO in a
+ * critical section, the handle needs to be released with pgaio_io_release().
+ *
+ *
+ * To react to the completion of the IO as soon as it is known to have
+ * completed, callbacks can be registered with pgaio_io_register_callbacks().
+ *
+ * To actually execute IO using the returned handle, the pgaio_io_prep_*()
+ * family of functions is used. In many cases the pgaio_io_prep_*() call will
+ * not be done directly by code that acquired the handle, but by lower level
+ * code that gets passed the handle. E.g. if code in bufmgr.c wants to perform
+ * AIO, it typically will pass the handle to smgr.c, which will pass it on to
+ * md.c, on to fd.c, which then finally calls pgaio_io_prep_*(). This
+ * forwarding allows the various layers to react to the IO's completion by
+ * registering callbacks. These callbacks in turn can translate a lower
+ * layer's result into a result understandable by a higher layer.
+ *
+ * During pgaio_io_prep_*() the IO is staged (i.e. prepared for execution but
+ * not submitted to the kernel). Unless in batchmode
+ * (c.f. pgaio_enter_batchmode()) the IO will also have been submitted for
+ * execution. Note that, whether in batchmode or not, the IO might even
+ * complete before the functions return.
+ *
+ * After pgaio_io_prep_*() the AioHandle is "consumed" and may not be
+ * referenced by the IO issuing code. To e.g. wait for IO, references to the
+ * IO can be established with pgaio_io_get_wref() *before* pgaio_io_prep_*()
+ * is called. pgaio_wref_wait() can be used to wait for the IO to complete.
+ *
+ *
+ * To know if the IO [partially] succeeded or failed, a PgAioReturn * can be
+ * passed to pgaio_io_acquire(). Once the issuing backend has called
+ * pgaio_wref_wait(), the PgAioReturn contains information about whether the
+ * operation succeeded and details about the first failure, if any. The error
+ * can be raised / logged with pgaio_result_report().
+ *
+ * The lifetime of the memory pointed to be *ret needs to be at least as long
+ * as the passed in resowner. If the resowner releases resources before the IO
+ * completes (typically due to an error), the reference to *ret will be
+ * cleared. In case of resowner cleanup *ret will not be updated with the
+ * results of the IO operation.
+ */
+PgAioHandle *
+pgaio_io_acquire(struct ResourceOwnerData *resowner, PgAioReturn *ret)
+{
+ PgAioHandle *h;
+
+ while (true)
+ {
+ h = pgaio_io_acquire_nb(resowner, ret);
+
+ if (h != NULL)
+ return h;
+
+ /*
+ * Evidently all handles by this backend are in use. Just wait for
+ * some to complete.
+ */
+ pgaio_io_wait_for_free();
+ }
+}
+
+/*
+ * Acquire an AioHandle, returning NULL if no handles are free.
+ *
+ * See pgaio_io_acquire(). The only difference is that this function will return
+ * NULL if there are no idle handles, instead of blocking.
+ */
+PgAioHandle *
+pgaio_io_acquire_nb(struct ResourceOwnerData *resowner, PgAioReturn *ret)
+{
+ if (pgaio_my_backend->num_staged_ios >= PGAIO_SUBMIT_BATCH_SIZE)
+ {
+ Assert(pgaio_my_backend->num_staged_ios == PGAIO_SUBMIT_BATCH_SIZE);
+ pgaio_submit_staged();
+ }
+
+ if (pgaio_my_backend->handed_out_io)
+ {
+ ereport(ERROR,
+ errmsg("API violation: Only one IO can be handed out"));
+ }
+
+ if (!dclist_is_empty(&pgaio_my_backend->idle_ios))
+ {
+ dlist_node *ion = dclist_pop_head_node(&pgaio_my_backend->idle_ios);
+ PgAioHandle *ioh = dclist_container(PgAioHandle, node, ion);
+
+ Assert(ioh->state == PGAIO_HS_IDLE);
+ Assert(ioh->owner_procno == MyProcNumber);
+
+ pgaio_io_update_state(ioh, PGAIO_HS_HANDED_OUT);
+ pgaio_my_backend->handed_out_io = ioh;
+
+ if (resowner)
+ pgaio_io_resowner_register(ioh);
+
+ if (ret)
+ {
+ ioh->report_return = ret;
+ ret->result.status = ARS_UNKNOWN;
+ }
+
+ return ioh;
+ }
+
+ return NULL;
+}
+
+/*
+ * Release IO handle that turned out to not be required.
+ *
+ * See pgaio_io_acquire() for more details.
+ */
+void
+pgaio_io_release(PgAioHandle *ioh)
+{
+ if (ioh == pgaio_my_backend->handed_out_io)
+ {
+ Assert(ioh->state == PGAIO_HS_HANDED_OUT);
+ Assert(ioh->resowner);
+
+ pgaio_my_backend->handed_out_io = NULL;
+ pgaio_io_reclaim(ioh);
+ }
+ else
+ {
+ elog(ERROR, "release in unexpected state");
+ }
+}
/*
* Release IO handle during resource owner cleanup.
@@ -39,8 +254,793 @@ int io_max_concurrency = -1;
void
pgaio_io_release_resowner(dlist_node *ioh_node, bool on_error)
{
+ PgAioHandle *ioh = dlist_container(PgAioHandle, resowner_node, ioh_node);
+
+ Assert(ioh->resowner);
+
+ ResourceOwnerForgetAioHandle(ioh->resowner, &ioh->resowner_node);
+ ioh->resowner = NULL;
+
+ switch (ioh->state)
+ {
+ case PGAIO_HS_IDLE:
+ elog(ERROR, "unexpected");
+ break;
+ case PGAIO_HS_HANDED_OUT:
+ Assert(ioh == pgaio_my_backend->handed_out_io || pgaio_my_backend->handed_out_io == NULL);
+
+ if (ioh == pgaio_my_backend->handed_out_io)
+ {
+ pgaio_my_backend->handed_out_io = NULL;
+ if (!on_error)
+ elog(WARNING, "leaked AIO handle");
+ }
+
+ pgaio_io_reclaim(ioh);
+ break;
+ case PGAIO_HS_DEFINED:
+ case PGAIO_HS_STAGED:
+ if (!on_error)
+ elog(WARNING, "AIO handle was not submitted");
+ pgaio_submit_staged();
+ break;
+ case PGAIO_HS_SUBMITTED:
+ case PGAIO_HS_COMPLETED_IO:
+ case PGAIO_HS_COMPLETED_SHARED:
+ case PGAIO_HS_COMPLETED_LOCAL:
+ /* this is expected to happen */
+ break;
+ }
+
+ /*
+ * Need to unregister the reporting of the IO's result, the memory it's
+ * referencing likely has gone away.
+ */
+ if (ioh->report_return)
+ ioh->report_return = NULL;
}
+/*
+ * Add a [set of] flags to the IO.
+ *
+ * Note that this combines flags with already set flags, rather than set flags
+ * to explicitly the passed in parameters. This is to allow multiple callsites
+ * to set flags.
+ */
+void
+pgaio_io_set_flag(PgAioHandle *ioh, PgAioHandleFlags flag)
+{
+ Assert(ioh->state == PGAIO_HS_HANDED_OUT);
+
+ ioh->flags |= flag;
+}
+
+/*
+ * Returns an ID uniquely identifying the IO handle. This is only really
+ * useful for logging, as handles are reused across multiple IOs.
+ */
+int
+pgaio_io_get_id(PgAioHandle *ioh)
+{
+ Assert(ioh >= pgaio_ctl->io_handles &&
+ ioh < (pgaio_ctl->io_handles + pgaio_ctl->io_handle_count));
+ return ioh - pgaio_ctl->io_handles;
+}
+
+/*
+ * Return the ProcNumber for the process that can use an IO handle. The
+ * mapping from IO handles to PGPROCs is static, therefore this even works
+ * when the corresponding PGPROC is not in use.
+ */
+ProcNumber
+pgaio_io_get_owner(PgAioHandle *ioh)
+{
+ return ioh->owner_procno;
+}
+
+/*
+ * Return a wait reference for the IO. Only wait references can be used to
+ * wait for an IOs completion, as handles themselves can be reused after
+ * completion. See also the comment above pgaio_io_acquire().
+ */
+void
+pgaio_io_get_wref(PgAioHandle *ioh, PgAioWaitRef *iow)
+{
+ Assert(ioh->state == PGAIO_HS_HANDED_OUT ||
+ ioh->state == PGAIO_HS_DEFINED ||
+ ioh->state == PGAIO_HS_STAGED);
+ Assert(ioh->generation != 0);
+
+ iow->aio_index = ioh - pgaio_ctl->io_handles;
+ iow->generation_upper = (uint32) (ioh->generation >> 32);
+ iow->generation_lower = (uint32) ioh->generation;
+}
+
+
+
+/* --------------------------------------------------------------------------------
+ * Internal Functions related to PgAioHandle
+ * --------------------------------------------------------------------------------
+ */
+
+static inline void
+pgaio_io_update_state(PgAioHandle *ioh, PgAioHandleState new_state)
+{
+ pgaio_debug_io(DEBUG5, ioh,
+ "updating state to %s",
+ pgaio_io_state_get_name(new_state));
+
+ /*
+ * Ensure the changes signified by the new state are visible before the
+ * new state becomes visible.
+ */
+ pg_write_barrier();
+
+ ioh->state = new_state;
+}
+
+static void
+pgaio_io_resowner_register(PgAioHandle *ioh)
+{
+ Assert(!ioh->resowner);
+ Assert(CurrentResourceOwner);
+
+ ResourceOwnerRememberAioHandle(CurrentResourceOwner, &ioh->resowner_node);
+ ioh->resowner = CurrentResourceOwner;
+}
+
+/*
+ * Stage IO for execution and, if appropriate, submit it immediately.
+ *
+ * Should only be called from pgaio_io_prep_*().
+ */
+void
+pgaio_io_stage(PgAioHandle *ioh, PgAioOp op)
+{
+ bool needs_synchronous;
+
+ Assert(ioh->state == PGAIO_HS_HANDED_OUT);
+ Assert(pgaio_my_backend->handed_out_io == ioh);
+ Assert(pgaio_io_has_target(ioh));
+
+ ioh->op = op;
+ ioh->result = 0;
+
+ pgaio_io_update_state(ioh, PGAIO_HS_DEFINED);
+
+ /* allow a new IO to be staged */
+ pgaio_my_backend->handed_out_io = NULL;
+
+ pgaio_io_call_stage(ioh);
+
+ pgaio_io_update_state(ioh, PGAIO_HS_STAGED);
+
+ /*
+ * Synchronous execution has to be executed, well, synchronously, so check
+ * that first.
+ */
+ needs_synchronous = pgaio_io_needs_synchronous_execution(ioh);
+
+ pgaio_debug_io(DEBUG3, ioh,
+ "prepared (synchronous: %d, in_batch: %d)",
+ needs_synchronous, pgaio_my_backend->in_batchmode);
+
+ if (!needs_synchronous)
+ {
+ pgaio_my_backend->staged_ios[pgaio_my_backend->num_staged_ios++] = ioh;
+ Assert(pgaio_my_backend->num_staged_ios <= PGAIO_SUBMIT_BATCH_SIZE);
+
+ /*
+ * Unless code explicitly opted into batching IOs, submit the IO
+ * immediately.
+ */
+ if (!pgaio_my_backend->in_batchmode)
+ pgaio_submit_staged();
+ }
+ else
+ {
+ pgaio_io_prepare_submit(ioh);
+ pgaio_io_perform_synchronously(ioh);
+ }
+}
+
+bool
+pgaio_io_needs_synchronous_execution(PgAioHandle *ioh)
+{
+ /*
+ * If the caller said to execute the IO synchronously, do so.
+ *
+ * XXX: We could optimize the logic when to execute synchronously by first
+ * checking if there are other IOs in flight and only synchronously
+ * executing if not. Unclear whether that'll be sufficiently common to be
+ * worth worrying about.
+ */
+ if (ioh->flags & PGAIO_HF_SYNCHRONOUS)
+ return true;
+
+ /* Check if the IO method requires synchronous execution of IO */
+ if (pgaio_method_ops->needs_synchronous_execution)
+ return pgaio_method_ops->needs_synchronous_execution(ioh);
+
+ return false;
+}
+
+/*
+ * Handle IO being processed by IO method.
+ *
+ * Should be called by IO methods / synchronous IO execution, just before the
+ * IO is performed.
+ */
+void
+pgaio_io_prepare_submit(PgAioHandle *ioh)
+{
+ pgaio_io_update_state(ioh, PGAIO_HS_SUBMITTED);
+
+ dclist_push_tail(&pgaio_my_backend->in_flight_ios, &ioh->node);
+}
+
+/*
+ * Handle IO getting completed by a method.
+ *
+ * Should be called by IO methods / synchronous IO execution, just after the
+ * IO has been performed.
+ *
+ * Expects to be called in a critical section. We expect IOs to be usable for
+ * WAL etc, which requires being able to execute completion callbacks in a
+ * critical section.
+ */
+void
+pgaio_io_process_completion(PgAioHandle *ioh, int result)
+{
+ Assert(ioh->state == PGAIO_HS_SUBMITTED);
+
+ Assert(CritSectionCount > 0);
+
+ ioh->result = result;
+
+ pgaio_io_update_state(ioh, PGAIO_HS_COMPLETED_IO);
+
+ pgaio_io_call_inj(ioh, "AIO_PROCESS_COMPLETION_BEFORE_SHARED");
+
+ pgaio_io_call_complete_shared(ioh);
+
+ pgaio_io_update_state(ioh, PGAIO_HS_COMPLETED_SHARED);
+
+ /* condition variable broadcast ensures state is visible before wakeup */
+ ConditionVariableBroadcast(&ioh->cv);
+
+ /* contains call to pgaio_io_call_complete_local() */
+ if (ioh->owner_procno == MyProcNumber)
+ pgaio_io_reclaim(ioh);
+}
+
+/*
+ * Has the IO completed and thus the IO handle been reused?
+ *
+ * This is useful when waiting for IO completion at a low level (e.g. in an IO
+ * method's ->wait_one() callback).
+ */
+bool
+pgaio_io_was_recycled(PgAioHandle *ioh, uint64 ref_generation, PgAioHandleState *state)
+{
+ *state = ioh->state;
+ pg_read_barrier();
+
+ return ioh->generation != ref_generation;
+}
+
+/*
+ * Wait for IO to complete. External code should never use this, outside of
+ * the AIO subsystem waits are only allowed via pgaio_wref_wait().
+ */
+static void
+pgaio_io_wait(PgAioHandle *ioh, uint64 ref_generation)
+{
+ PgAioHandleState state;
+ bool am_owner;
+
+ am_owner = ioh->owner_procno == MyProcNumber;
+
+ if (pgaio_io_was_recycled(ioh, ref_generation, &state))
+ return;
+
+ if (am_owner)
+ {
+ if (state != PGAIO_HS_SUBMITTED
+ && state != PGAIO_HS_COMPLETED_IO
+ && state != PGAIO_HS_COMPLETED_SHARED
+ && state != PGAIO_HS_COMPLETED_LOCAL)
+ {
+ elog(PANIC, "waiting for own IO in wrong state: %d",
+ state);
+ }
+ }
+
+ while (true)
+ {
+ if (pgaio_io_was_recycled(ioh, ref_generation, &state))
+ return;
+
+ switch (state)
+ {
+ case PGAIO_HS_IDLE:
+ case PGAIO_HS_HANDED_OUT:
+ elog(ERROR, "IO in wrong state: %d", state);
+ break;
+
+ case PGAIO_HS_SUBMITTED:
+
+ /*
+ * If we need to wait via the IO method, do so now. Don't
+ * check via the IO method if the issuing backend is executing
+ * the IO synchronously.
+ */
+ if (pgaio_method_ops->wait_one && !(ioh->flags & PGAIO_HF_SYNCHRONOUS))
+ {
+ pgaio_method_ops->wait_one(ioh, ref_generation);
+ continue;
+ }
+ /* fallthrough */
+
+ /* waiting for owner to submit */
+ case PGAIO_HS_DEFINED:
+ case PGAIO_HS_STAGED:
+ /* waiting for reaper to complete */
+ /* fallthrough */
+ case PGAIO_HS_COMPLETED_IO:
+ /* shouldn't be able to hit this otherwise */
+ Assert(IsUnderPostmaster);
+ /* ensure we're going to get woken up */
+ ConditionVariablePrepareToSleep(&ioh->cv);
+
+ while (!pgaio_io_was_recycled(ioh, ref_generation, &state))
+ {
+ if (state == PGAIO_HS_COMPLETED_SHARED ||
+ state == PGAIO_HS_COMPLETED_LOCAL)
+ break;
+ ConditionVariableSleep(&ioh->cv, WAIT_EVENT_AIO_IO_COMPLETION);
+ }
+
+ ConditionVariableCancelSleep();
+ break;
+
+ case PGAIO_HS_COMPLETED_SHARED:
+ case PGAIO_HS_COMPLETED_LOCAL:
+ /* see above */
+ if (am_owner)
+ pgaio_io_reclaim(ioh);
+ return;
+ }
+ }
+}
+
+/*
+ * Make IO handle ready to be reused after IO has completed or after the
+ * handle has been released without being used.
+ */
+static void
+pgaio_io_reclaim(PgAioHandle *ioh)
+{
+ /* This is only ok if it's our IO */
+ Assert(ioh->owner_procno == MyProcNumber);
+
+ /*
+ * It's a bit ugly, but right now the easiest place to put the execution
+ * of shared completion callbacks is this function, as we need to execute
+ * local callbacks just before reclaiming at multiple callsites.
+ */
+ if (ioh->state == PGAIO_HS_COMPLETED_SHARED)
+ {
+ pgaio_io_call_complete_local(ioh);
+ pgaio_io_update_state(ioh, PGAIO_HS_COMPLETED_LOCAL);
+ }
+
+ pgaio_debug_io(DEBUG4, ioh,
+ "reclaiming: distilled_result: (status %s, id %u, error_data %d), raw_result: %d",
+ pgaio_result_status_string(ioh->distilled_result.status),
+ ioh->distilled_result.id,
+ ioh->distilled_result.error_data,
+ ioh->result);
+
+ /* if the IO has been defined, we might need to do more work */
+ if (ioh->state != PGAIO_HS_HANDED_OUT)
+ {
+ dclist_delete_from(&pgaio_my_backend->in_flight_ios, &ioh->node);
+
+ if (ioh->report_return)
+ {
+ ioh->report_return->result = ioh->distilled_result;
+ ioh->report_return->target_data = ioh->target_data;
+ }
+ }
+
+ if (ioh->resowner)
+ {
+ ResourceOwnerForgetAioHandle(ioh->resowner, &ioh->resowner_node);
+ ioh->resowner = NULL;
+ }
+
+ Assert(!ioh->resowner);
+
+ ioh->op = PGAIO_OP_INVALID;
+ ioh->target = PGAIO_TID_INVALID;
+ ioh->flags = 0;
+ ioh->num_callbacks = 0;
+ ioh->handle_data_len = 0;
+ ioh->report_return = NULL;
+ ioh->result = 0;
+ ioh->distilled_result.status = ARS_UNKNOWN;
+
+ /* XXX: the barrier is probably superfluous */
+ pg_write_barrier();
+ ioh->generation++;
+
+ pgaio_io_update_state(ioh, PGAIO_HS_IDLE);
+
+ /*
+ * We push the IO to the head of the idle IO list, that seems more cache
+ * efficient in cases where only a few IOs are used.
+ */
+ dclist_push_head(&pgaio_my_backend->idle_ios, &ioh->node);
+}
+
+/*
+ * Wait for an IO handle to become usable.
+ *
+ * This only really is useful for pgaio_io_acquire().
+ */
+static void
+pgaio_io_wait_for_free(void)
+{
+ int reclaimed = 0;
+
+ pgaio_debug(DEBUG2, "waiting for self with %d pending",
+ pgaio_my_backend->num_staged_ios);
+
+ /*
+ * First check if any of our IOs actually have completed - when using
+ * worker, that'll often be the case. We could do so as part of the loop
+ * below, but that'd potentially lead us to wait for some IO submitted
+ * before.
+ */
+ for (int i = 0; i < io_max_concurrency; i++)
+ {
+ PgAioHandle *ioh = &pgaio_ctl->io_handles[pgaio_my_backend->io_handle_off + i];
+
+ if (ioh->state == PGAIO_HS_COMPLETED_SHARED)
+ {
+ pgaio_io_reclaim(ioh);
+ reclaimed++;
+ }
+ }
+
+ if (reclaimed > 0)
+ return;
+
+ /*
+ * If we have any unsubmitted IOs, submit them now. We'll start waiting in
+ * a second, so it's better they're in flight. This also addresses the
+ * edge-case that all IOs are unsubmitted.
+ */
+ if (pgaio_my_backend->num_staged_ios > 0)
+ pgaio_submit_staged();
+
+ if (dclist_count(&pgaio_my_backend->in_flight_ios) == 0)
+ elog(ERROR, "no free IOs despite no in-flight IOs");
+
+ /*
+ * Wait for the oldest in-flight IO to complete.
+ *
+ * XXX: Reusing the general IO wait is suboptimal, we don't need to wait
+ * for that specific IO to complete, we just need *any* IO to complete.
+ */
+ {
+ PgAioHandle *ioh = dclist_head_element(PgAioHandle, node, &pgaio_my_backend->in_flight_ios);
+
+ switch (ioh->state)
+ {
+ /* should not be in in-flight list */
+ case PGAIO_HS_IDLE:
+ case PGAIO_HS_DEFINED:
+ case PGAIO_HS_HANDED_OUT:
+ case PGAIO_HS_STAGED:
+ case PGAIO_HS_COMPLETED_LOCAL:
+ elog(ERROR, "shouldn't get here with io:%d in state %d",
+ pgaio_io_get_id(ioh), ioh->state);
+ break;
+
+ case PGAIO_HS_COMPLETED_IO:
+ case PGAIO_HS_SUBMITTED:
+ pgaio_debug_io(DEBUG2, ioh,
+ "waiting for free io with %d in flight",
+ dclist_count(&pgaio_my_backend->in_flight_ios));
+
+ /*
+ * In a more general case this would be racy, because the
+ * generation could increase after we read ioh->state above.
+ * But we are only looking at IOs by the current backend and
+ * the IO can only be recycled by this backend.
+ */
+ pgaio_io_wait(ioh, ioh->generation);
+ break;
+
+ case PGAIO_HS_COMPLETED_SHARED:
+ /* it's possible that another backend just finished this IO */
+ pgaio_io_reclaim(ioh);
+ break;
+ }
+
+ if (dclist_count(&pgaio_my_backend->idle_ios) == 0)
+ elog(PANIC, "no idle IOs after waiting");
+ return;
+ }
+}
+
+/*
+ * Internal - code outside of AIO should never need this and it'd be hard for
+ * such code to be safe.
+ */
+static PgAioHandle *
+pgaio_io_from_wref(PgAioWaitRef *iow, uint64 *ref_generation)
+{
+ PgAioHandle *ioh;
+
+ Assert(iow->aio_index < pgaio_ctl->io_handle_count);
+
+ ioh = &pgaio_ctl->io_handles[iow->aio_index];
+
+ *ref_generation = ((uint64) iow->generation_upper) << 32 |
+ iow->generation_lower;
+
+ Assert(*ref_generation != 0);
+
+ return ioh;
+}
+
+static const char *
+pgaio_io_state_get_name(PgAioHandleState s)
+{
+#define PGAIO_HS_TOSTR_CASE(sym) case PGAIO_HS_##sym: return #sym
+ switch (s)
+ {
+ PGAIO_HS_TOSTR_CASE(IDLE);
+ PGAIO_HS_TOSTR_CASE(HANDED_OUT);
+ PGAIO_HS_TOSTR_CASE(DEFINED);
+ PGAIO_HS_TOSTR_CASE(STAGED);
+ PGAIO_HS_TOSTR_CASE(SUBMITTED);
+ PGAIO_HS_TOSTR_CASE(COMPLETED_IO);
+ PGAIO_HS_TOSTR_CASE(COMPLETED_SHARED);
+ PGAIO_HS_TOSTR_CASE(COMPLETED_LOCAL);
+ }
+#undef PGAIO_HS_TOSTR_CASE
+
+ return NULL; /* silence compiler */
+}
+
+const char *
+pgaio_io_get_state_name(PgAioHandle *ioh)
+{
+ return pgaio_io_state_get_name(ioh->state);
+}
+
+const char *
+pgaio_result_status_string(PgAioResultStatus rs)
+{
+ switch (rs)
+ {
+ case ARS_UNKNOWN:
+ return "UNKNOWN";
+ case ARS_OK:
+ return "OK";
+ case ARS_PARTIAL:
+ return "PARTIAL";
+ case ARS_ERROR:
+ return "ERROR";
+ }
+
+ return NULL; /* silence compiler */
+}
+
+
+
+/* --------------------------------------------------------------------------------
+ * Functions primarily related to IO Wait References
+ * --------------------------------------------------------------------------------
+ */
+
+/*
+ * Mark a wait reference as invalid
+ */
+void
+pgaio_wref_clear(PgAioWaitRef *iow)
+{
+ iow->aio_index = PG_UINT32_MAX;
+}
+
+/* Is the wait reference valid? */
+bool
+pgaio_wref_valid(PgAioWaitRef *iow)
+{
+ return iow->aio_index != PG_UINT32_MAX;
+}
+
+/*
+ * Similar to pgaio_io_get_id(), just for wait references.
+ */
+int
+pgaio_wref_get_id(PgAioWaitRef *iow)
+{
+ Assert(pgaio_wref_valid(iow));
+ return iow->aio_index;
+}
+
+/*
+ * Wait for the IO to have completed. Can be called in any process, not just
+ * in the issuing backend.
+ */
+void
+pgaio_wref_wait(PgAioWaitRef *iow)
+{
+ uint64 ref_generation;
+ PgAioHandle *ioh;
+
+ ioh = pgaio_io_from_wref(iow, &ref_generation);
+
+ pgaio_io_wait(ioh, ref_generation);
+}
+
+/*
+ * Check if the referenced IO completed, without blocking.
+ */
+bool
+pgaio_wref_check_done(PgAioWaitRef *iow)
+{
+ uint64 ref_generation;
+ PgAioHandleState state;
+ bool am_owner;
+ PgAioHandle *ioh;
+
+ ioh = pgaio_io_from_wref(iow, &ref_generation);
+
+ if (pgaio_io_was_recycled(ioh, ref_generation, &state))
+ return true;
+
+ if (state == PGAIO_HS_IDLE)
+ return true;
+
+ am_owner = ioh->owner_procno == MyProcNumber;
+
+ if (state == PGAIO_HS_COMPLETED_SHARED ||
+ state == PGAIO_HS_COMPLETED_LOCAL)
+ {
+ if (am_owner)
+ pgaio_io_reclaim(ioh);
+ return true;
+ }
+
+ /*
+ * XXX: It likely would be worth checking in with the io method, to give
+ * the IO method a chance to check if there are completion events queued.
+ */
+
+ return false;
+}
+
+
+
+/* --------------------------------------------------------------------------------
+ * Actions on multiple IOs.
+ * --------------------------------------------------------------------------------
+ */
+
+/*
+ * Submit IOs in batches going forward.
+ *
+ * Submitting multiple IOs at once can be substantially faster than doing so
+ * one-by-one. At the same time submitting multiple IOs at once requires more
+ * care to avoid deadlocks.
+ *
+ * Consider backend A staging an IO for buffer 1 and then trying to start IO
+ * on buffer 2, while backend B does the inverse. If A submitted the IO before
+ * moving on to buffer 2, this works just fine, B will wait for the IO to
+ * complete. But if batching were used, each backend will wait for IO that has
+ * not yet been submitted to complete, i.e. forever.
+ *
+ * End batch submission mode with pgaio_exit_batchmode(). (Throwing errors is
+ * allowed; error recovery will end the batch.)
+ *
+ * To avoid deadlocks, code needs to ensure that it will not wait for another
+ * backend while there is unsubmitted IO. E.g. by using conditional lock
+ * acquisition when acquiring buffer locks. To check if there currently are
+ * staged IOs, call pgaio_have_staged() and to submit all staged IOs call
+ * pgaio_submit_staged().
+ *
+ * It is not allowed to enter batchmode while already in batchmode, it's
+ * unlikely to ever be needed, as code needs to be explicitly aware of being
+ * called in batchmode, to avoid the deadlock risks explained above.
+ *
+ * Note that IOs may get submitted before pgaio_exit_batchmode() is called,
+ * e.g. because too many IOs have been staged or because pgaio_submit_staged()
+ * was called.
+ */
+void
+pgaio_enter_batchmode(void)
+{
+ if (pgaio_my_backend->in_batchmode)
+ elog(ERROR, "starting batch while batch already in progress");
+ pgaio_my_backend->in_batchmode = true;
+}
+
+/*
+ * Stop submitting IOs in batches.
+ */
+void
+pgaio_exit_batchmode(void)
+{
+ Assert(pgaio_my_backend->in_batchmode);
+
+ pgaio_submit_staged();
+ pgaio_my_backend->in_batchmode = false;
+}
+
+/*
+ * Are there staged but unsubmitted IOs?
+ *
+ * See comment above pgaio_enter_batchmode() for why code may need to check if
+ * there is IO in that state.
+ */
+bool
+pgaio_have_staged(void)
+{
+ Assert(pgaio_my_backend->in_batchmode ||
+ pgaio_my_backend->num_staged_ios == 0);
+ return pgaio_my_backend->num_staged_ios > 0;
+}
+
+/*
+ * Submit all staged but not yet submitted IOs.
+ *
+ * Unless in batch mode, this never needs to be called, as IOs get submitted
+ * as soon as possible. While in batchmode pgaio_submit_staged() can be called
+ * before waiting on another backend, to avoid the risk of deadlocks. See
+ * pgaio_enter_batchmode().
+ */
+void
+pgaio_submit_staged(void)
+{
+ int total_submitted = 0;
+ int did_submit;
+
+ if (pgaio_my_backend->num_staged_ios == 0)
+ return;
+
+
+ START_CRIT_SECTION();
+
+ did_submit = pgaio_method_ops->submit(pgaio_my_backend->num_staged_ios,
+ pgaio_my_backend->staged_ios);
+
+ END_CRIT_SECTION();
+
+ total_submitted += did_submit;
+
+ Assert(total_submitted == did_submit);
+
+ pgaio_my_backend->num_staged_ios = 0;
+
+ pgaio_debug(DEBUG4,
+ "aio: submitted %d IOs",
+ total_submitted);
+}
+
+
+
+/* --------------------------------------------------------------------------------
+ * Other
+ * --------------------------------------------------------------------------------
+ */
+
+
/*
* Perform AIO related cleanup after an error.
*
@@ -50,6 +1050,22 @@ pgaio_io_release_resowner(dlist_node *ioh_node, bool on_error)
void
pgaio_error_cleanup(void)
{
+ /*
+ * It is possible that code errored out after pgaio_enter_batchmode() but
+ * before pgaio_exit_batchmode() was called. In that case we need to
+ * submit the IO now.
+ */
+ if (pgaio_my_backend->in_batchmode)
+ {
+ pgaio_my_backend->in_batchmode = false;
+
+ pgaio_submit_staged();
+ }
+
+ /*
+ * As we aren't in batchmode, there shouldn't be any unsubmitted IOs.
+ */
+ Assert(pgaio_my_backend->num_staged_ios == 0);
}
/*
@@ -62,11 +1078,86 @@ pgaio_error_cleanup(void)
void
AtEOXact_Aio(bool is_commit)
{
+ /*
+ * We should never be in batch mode at transactional boundaries. In case
+ * an error was thrown while in batch mode, pgaio_error_cleanup() should
+ * have exited batchmode.
+ *
+ * In case we are in batchmode somehow, make sure to submit all staged
+ * IOs, other backends may need them to complete to continue.
+ */
+ if (pgaio_my_backend->in_batchmode)
+ {
+ pgaio_error_cleanup();
+ elog(WARNING, "open AIO batch at end of (sub-)transaction");
+ }
+
+ /*
+ * As we aren't in batchmode, there shouldn't be any unsubmitted IOs.
+ */
+ Assert(pgaio_my_backend->num_staged_ios == 0);
+}
+
+/*
+ * Need to submit staged but not yet submitted IOs using the fd, otherwise
+ * the IO would end up targeting something bogus.
+ */
+void
+pgaio_closing_fd(int fd)
+{
+ /*
+ * Might be called before AIO is initialized or in a subprocess that
+ * doesn't use AIO.
+ */
+ if (!pgaio_my_backend)
+ return;
+
+ /*
+ * For now just submit all staged IOs - we could be more selective, but
+ * it's probably not worth it.
+ */
+ pgaio_submit_staged();
+}
+
+/*
+ * Registered as before_shmem_exit() callback in pgaio_init_backend()
+ */
+void
+pgaio_shutdown(int code, Datum arg)
+{
+ Assert(pgaio_my_backend);
+ Assert(!pgaio_my_backend->handed_out_io);
+
+ /* first clean up resources as we would at a transaction boundary */
+ AtEOXact_Aio(code == 0);
+
+ /*
+ * Before exiting, make sure that all IOs are finished. That has two main
+ * purposes:
+ *
+ * - Some kernel-level AIO mechanisms don't deal well with the issuer of
+ * an AIO exiting before IO completed
+ *
+ * - It'd be confusing to see partially finished IOs in stats views etc
+ */
+ while (!dclist_is_empty(&pgaio_my_backend->in_flight_ios))
+ {
+ PgAioHandle *ioh = dclist_head_element(PgAioHandle, node, &pgaio_my_backend->in_flight_ios);
+
+ /* see comment in pgaio_io_wait_for_free() about raciness */
+ pgaio_io_wait(ioh, ioh->generation);
+ }
+
+ pgaio_my_backend = NULL;
}
void
assign_io_method(int newval, void *extra)
{
+ Assert(pgaio_method_ops_table[newval] != NULL);
+ Assert(newval < lengthof(io_method_options));
+
+ pgaio_method_ops = pgaio_method_ops_table[newval];
}
bool
@@ -88,3 +1179,43 @@ check_io_max_concurrency(int *newval, void **extra, GucSource source)
return true;
}
+
+
+
+/* --------------------------------------------------------------------------------
+ * Injection point support
+ * --------------------------------------------------------------------------------
+ */
+
+#ifdef USE_INJECTION_POINTS
+
+/*
+ * Call injection point with support for pgaio_inj_io_get().
+ */
+void
+pgaio_io_call_inj(PgAioHandle *ioh, const char *injection_point)
+{
+ pgaio_inj_cur_handle = ioh;
+
+ PG_TRY();
+ {
+ InjectionPointCached(injection_point);
+ }
+ PG_FINALLY();
+ {
+ pgaio_inj_cur_handle = NULL;
+ }
+ PG_END_TRY();
+}
+
+/*
+ * Return IO associated with injection point invocation. This is only needed
+ * as injection points currently don't support arguments.
+ */
+PgAioHandle *
+pgaio_inj_io_get(void)
+{
+ return pgaio_inj_cur_handle;
+}
+
+#endif
diff --git a/src/backend/storage/aio/aio_callback.c b/src/backend/storage/aio/aio_callback.c
new file mode 100644
index 00000000000..7392b55322c
--- /dev/null
+++ b/src/backend/storage/aio/aio_callback.c
@@ -0,0 +1,307 @@
+/*-------------------------------------------------------------------------
+ *
+ * aio_callback.c
+ * AIO - Functionality related to callbacks that can be registered on IO
+ * Handles
+ *
+ * Portions Copyright (c) 1996-2025, PostgreSQL Global Development Group
+ * Portions Copyright (c) 1994, Regents of the University of California
+ *
+ * IDENTIFICATION
+ * src/backend/storage/aio/aio_callback.c
+ *
+ *-------------------------------------------------------------------------
+ */
+
+#include "postgres.h"
+
+#include "miscadmin.h"
+#include "storage/aio.h"
+#include "storage/aio_internal.h"
+
+
+/* just to have something to put into the aio_handle_cbs */
+static const struct PgAioHandleCallbacks aio_invalid_cb = {0};
+
+typedef struct PgAioHandleCallbacksEntry
+{
+ const PgAioHandleCallbacks *const cb;
+ const char *const name;
+} PgAioHandleCallbacksEntry;
+
+/*
+ * Callback definition for the callbacks that can be registered on an IO
+ * handle. See PgAioHandleCallbackID's definition for an explanation for why
+ * callbacks are not identified by a pointer.
+ */
+static const PgAioHandleCallbacksEntry aio_handle_cbs[] = {
+#define CALLBACK_ENTRY(id, callback) [id] = {.cb = &callback, .name = #callback}
+ CALLBACK_ENTRY(PGAIO_HCB_INVALID, aio_invalid_cb),
+#undef CALLBACK_ENTRY
+};
+
+
+
+/* --------------------------------------------------------------------------------
+ * Public callback related functions operating on IO Handles
+ * --------------------------------------------------------------------------------
+ */
+
+/*
+ * Register callback for the IO handle.
+ *
+ * Only a limited number (PGAIO_HANDLE_MAX_CALLBACKS) of callbacks can be
+ * registered for each IO.
+ *
+ * Callbacks need to be registered before [indirectly] calling
+ * pgaio_io_prep_*(), as the IO may be executed immediately.
+ *
+ * A callback can be passed a small bit of data, e.g. to indicate whether to
+ * zero a buffer if it is invalid.
+ *
+ *
+ * Note that callbacks are executed in critical sections. This is necessary
+ * to be able to execute IO in critical sections (consider e.g. WAL
+ * logging). To perform AIO we first need to acquire a handle, which, if there
+ * are no free handles, requires waiting for IOs to complete and to execute
+ * their completion callbacks.
+ *
+ * Callbacks may be executed in the issuing backend but also in another
+ * backend (because that backend is waiting for the IO) or in IO workers (if
+ * io_method=worker is used).
+ *
+ *
+ * See PgAioHandleCallbackID's definition for an explanation for why
+ * callbacks are not identified by a pointer.
+ */
+void
+pgaio_io_register_callbacks(PgAioHandle *ioh, PgAioHandleCallbackID cb_id,
+ uint8 cb_data)
+{
+ const PgAioHandleCallbacksEntry *ce = &aio_handle_cbs[cb_id];
+
+ if (cb_id >= lengthof(aio_handle_cbs))
+ elog(ERROR, "callback %d is out of range", cb_id);
+ if (aio_handle_cbs[cb_id].cb->complete_shared == NULL &&
+ aio_handle_cbs[cb_id].cb->complete_local == NULL)
+ elog(ERROR, "callback %d does not have completion callback", cb_id);
+ if (ioh->num_callbacks >= PGAIO_HANDLE_MAX_CALLBACKS)
+ elog(PANIC, "too many callbacks, the max is %d", PGAIO_HANDLE_MAX_CALLBACKS);
+ ioh->callbacks[ioh->num_callbacks] = cb_id;
+ ioh->callbacks_data[ioh->num_callbacks] = cb_data;
+
+ pgaio_debug_io(DEBUG3, ioh,
+ "adding cb #%d, id %d/%s",
+ ioh->num_callbacks + 1,
+ cb_id, ce->name);
+
+ ioh->num_callbacks++;
+}
+
+/*
+ * Associate an array of data with the Handle. This is e.g. useful to the
+ * transport knowledge about which buffers a multi-block IO affects to
+ * completion callbacks.
+ *
+ * Right now this can be done only once for each IO, even though multiple
+ * callbacks can be registered. There aren't any known usecases requiring more
+ * and the required amount of shared memory does add up, so it doesn't seem
+ * worth multiplying memory usage by PGAIO_HANDLE_MAX_CALLBACKS.
+ */
+void
+pgaio_io_set_handle_data_64(PgAioHandle *ioh, uint64 *data, uint8 len)
+{
+ Assert(ioh->state == PGAIO_HS_HANDED_OUT);
+ Assert(ioh->handle_data_len == 0);
+ Assert(len <= PG_IOV_MAX);
+
+ for (int i = 0; i < len; i++)
+ pgaio_ctl->handle_data[ioh->iovec_off + i] = data[i];
+ ioh->handle_data_len = len;
+}
+
+/*
+ * Convenience version of pgaio_io_set_handle_data_64() that converts a 32bit
+ * array to a 64bit array. Without it callers would end up needing to
+ * open-code equivalent code.
+ */
+void
+pgaio_io_set_handle_data_32(PgAioHandle *ioh, uint32 *data, uint8 len)
+{
+ Assert(ioh->state == PGAIO_HS_HANDED_OUT);
+ Assert(ioh->handle_data_len == 0);
+ Assert(len <= PG_IOV_MAX);
+
+ for (int i = 0; i < len; i++)
+ pgaio_ctl->handle_data[ioh->iovec_off + i] = data[i];
+ ioh->handle_data_len = len;
+}
+
+/*
+ * Return data set with pgaio_io_set_handle_data_*().
+ */
+uint64 *
+pgaio_io_get_handle_data(PgAioHandle *ioh, uint8 *len)
+{
+ Assert(ioh->handle_data_len > 0);
+
+ *len = ioh->handle_data_len;
+
+ return &pgaio_ctl->handle_data[ioh->iovec_off];
+}
+
+
+
+/* --------------------------------------------------------------------------------
+ * Public IO Result related functions
+ * --------------------------------------------------------------------------------
+ */
+
+void
+pgaio_result_report(PgAioResult result, const PgAioTargetData *target_data, int elevel)
+{
+ PgAioHandleCallbackID cb_id = result.id;
+ const PgAioHandleCallbacksEntry *ce = &aio_handle_cbs[cb_id];
+
+ Assert(result.status != ARS_UNKNOWN);
+ Assert(result.status != ARS_OK);
+
+ if (ce->cb->report == NULL)
+ elog(ERROR, "callback %d/%s does not have report callback",
+ result.id, ce->name);
+
+ ce->cb->report(result, target_data, elevel);
+}
+
+
+
+/* --------------------------------------------------------------------------------
+ * Internal callback related functions operating on IO Handles
+ * --------------------------------------------------------------------------------
+ */
+
+/*
+ * Internal function which invokes ->stage for all the registered callbacks.
+ */
+void
+pgaio_io_call_stage(PgAioHandle *ioh)
+{
+ Assert(ioh->target > PGAIO_TID_INVALID && ioh->target < PGAIO_TID_COUNT);
+ Assert(ioh->op > PGAIO_OP_INVALID && ioh->op < PGAIO_OP_COUNT);
+
+ for (int i = ioh->num_callbacks; i > 0; i--)
+ {
+ PgAioHandleCallbackID cb_id = ioh->callbacks[i - 1];
+ uint8 cb_data = ioh->callbacks_data[i - 1];
+ const PgAioHandleCallbacksEntry *ce = &aio_handle_cbs[cb_id];
+
+ if (!ce->cb->stage)
+ continue;
+
+ pgaio_debug_io(DEBUG3, ioh,
+ "calling cb #%d %d/%s->stage(%u)",
+ i, cb_id, ce->name, cb_data);
+ ce->cb->stage(ioh, cb_data);
+ }
+}
+
+/*
+ * Internal function which invokes ->complete_shared for all the registered
+ * callbacks.
+ */
+void
+pgaio_io_call_complete_shared(PgAioHandle *ioh)
+{
+ PgAioResult result;
+
+ START_CRIT_SECTION();
+
+ Assert(ioh->target > PGAIO_TID_INVALID && ioh->target < PGAIO_TID_COUNT);
+ Assert(ioh->op > PGAIO_OP_INVALID && ioh->op < PGAIO_OP_COUNT);
+
+ result.status = ARS_OK; /* low level IO is always considered OK */
+ result.result = ioh->result;
+ result.id = PGAIO_HCB_INVALID;
+ result.error_data = 0;
+
+ /*
+ * Call callbacks with the last registered (innermost) callback first.
+ * Each callback can modify the result forwarded to the next callback.
+ */
+ for (int i = ioh->num_callbacks; i > 0; i--)
+ {
+ PgAioHandleCallbackID cb_id = ioh->callbacks[i - 1];
+ uint8 cb_data = ioh->callbacks_data[i - 1];
+ const PgAioHandleCallbacksEntry *ce = &aio_handle_cbs[cb_id];
+
+ if (!ce->cb->complete_shared)
+ continue;
+
+ pgaio_debug_io(DEBUG4, ioh,
+ "calling cb #%d, id %d/%s->complete_shared(%u) with distilled result: (status %s, id %u, error_data %d, result %d)",
+ i, cb_id, ce->name,
+ cb_data,
+ pgaio_result_status_string(result.status),
+ result.id, result.error_data, result.result);
+ result = ce->cb->complete_shared(ioh, result, cb_data);
+ }
+
+ ioh->distilled_result = result;
+
+ pgaio_debug_io(DEBUG3, ioh,
+ "after shared completion: distilled result: (status %s, id %u, error_data: %d, result %d), raw_result: %d",
+ pgaio_result_status_string(result.status),
+ result.id, result.error_data, result.result,
+ ioh->result);
+
+ END_CRIT_SECTION();
+}
+
+/*
+ * Internal function which invokes ->complete_local for all the registered
+ * callbacks.
+ *
+ * XXX: It'd be nice to deduplicate with pgaio_io_call_complete_shared().
+ */
+void
+pgaio_io_call_complete_local(PgAioHandle *ioh)
+{
+ PgAioResult result;
+
+ START_CRIT_SECTION();
+
+ Assert(ioh->target > PGAIO_TID_INVALID && ioh->target < PGAIO_TID_COUNT);
+ Assert(ioh->op > PGAIO_OP_INVALID && ioh->op < PGAIO_OP_COUNT);
+
+ /* start with distilled result from shared callback */
+ result = ioh->distilled_result;
+
+ for (int i = ioh->num_callbacks; i > 0; i--)
+ {
+ PgAioHandleCallbackID cb_id = ioh->callbacks[i - 1];
+ uint8 cb_data = ioh->callbacks_data[i - 1];
+ const PgAioHandleCallbacksEntry *ce = &aio_handle_cbs[cb_id];
+
+ if (!ce->cb->complete_local)
+ continue;
+
+ pgaio_debug_io(DEBUG4, ioh,
+ "calling cb #%d, id %d/%s->complete_local(%u) with distilled result: status %s, id %u, error_data %d, result %d",
+ i, cb_id, ce->name, cb_data,
+ pgaio_result_status_string(result.status),
+ result.id, result.error_data, result.result);
+ result = ce->cb->complete_local(ioh, result, cb_data);
+ }
+
+ /*
+ * Note that we don't save the result in ioh->distilled_result, the local
+ * callback's result should not ever matter to other waiters.
+ */
+ pgaio_debug_io(DEBUG3, ioh,
+ "after local completion: distilled result: (status %s, id %u, error_data %d, result %d), raw_result: %d",
+ pgaio_result_status_string(result.status),
+ result.id, result.error_data, result.result,
+ ioh->result);
+
+ END_CRIT_SECTION();
+}
diff --git a/src/backend/storage/aio/aio_init.c b/src/backend/storage/aio/aio_init.c
index aeacc144149..e6c3e40f7d4 100644
--- a/src/backend/storage/aio/aio_init.c
+++ b/src/backend/storage/aio/aio_init.c
@@ -14,24 +14,211 @@
#include "postgres.h"
+#include "miscadmin.h"
+#include "storage/aio.h"
+#include "storage/aio_internal.h"
#include "storage/aio_subsys.h"
+#include "storage/ipc.h"
+#include "storage/proc.h"
+#include "storage/shmem.h"
+#include "utils/guc.h"
+static Size
+AioCtlShmemSize(void)
+{
+ Size sz;
+
+ /* pgaio_ctl itself */
+ sz = offsetof(PgAioCtl, io_handles);
+
+ return sz;
+}
+
+static uint32
+AioProcs(void)
+{
+ return MaxBackends + NUM_AUXILIARY_PROCS;
+}
+
+static Size
+AioBackendShmemSize(void)
+{
+ return mul_size(AioProcs(), sizeof(PgAioBackend));
+}
+
+static Size
+AioHandleShmemSize(void)
+{
+ Size sz;
+
+ /* ios */
+ sz = mul_size(AioProcs(),
+ mul_size(io_max_concurrency, sizeof(PgAioHandle)));
+
+ return sz;
+}
+
+static Size
+AioHandleIOVShmemSize(void)
+{
+ return mul_size(sizeof(struct iovec),
+ mul_size(mul_size(PG_IOV_MAX, AioProcs()),
+ io_max_concurrency));
+}
+
+static Size
+AioHandleDataShmemSize(void)
+{
+ return mul_size(sizeof(uint64),
+ mul_size(mul_size(PG_IOV_MAX, AioProcs()),
+ io_max_concurrency));
+}
+
+/*
+ * Choose a suitable value for io_max_concurrency.
+ *
+ * It's unlikely that we could have more IOs in flight than buffers that we
+ * would be allowed to pin.
+ *
+ * On the upper end, apply a cap too - just because shared_buffers is large,
+ * it doesn't make sense have millions of buffers undergo IO concurrently.
+ */
+static int
+AioChooseMaxConcurrency(void)
+{
+ uint32 max_backends;
+ int max_proportional_pins;
+
+ /* Similar logic to LimitAdditionalPins() */
+ max_backends = MaxBackends + NUM_AUXILIARY_PROCS;
+ max_proportional_pins = NBuffers / max_backends;
+
+ max_proportional_pins = Max(max_proportional_pins, 1);
+
+ /* apply upper limit */
+ return Min(max_proportional_pins, 64);
+}
+
Size
AioShmemSize(void)
{
Size sz = 0;
+ /*
+ * We prefer to report this value's source as PGC_S_DYNAMIC_DEFAULT.
+ * However, if the DBA explicitly set io_max_concurrency = -1 in the
+ * config file, then PGC_S_DYNAMIC_DEFAULT will fail to override that and
+ * we must force
+ *
+ */
+ if (io_max_concurrency == -1)
+ {
+ char buf[32];
+
+ snprintf(buf, sizeof(buf), "%d", AioChooseMaxConcurrency());
+ SetConfigOption("io_max_concurrency", buf, PGC_POSTMASTER,
+ PGC_S_DYNAMIC_DEFAULT);
+ if (io_max_concurrency == -1) /* failed to apply it? */
+ SetConfigOption("io_max_concurrency", buf, PGC_POSTMASTER,
+ PGC_S_OVERRIDE);
+ }
+
+ sz = add_size(sz, AioCtlShmemSize());
+ sz = add_size(sz, AioBackendShmemSize());
+ sz = add_size(sz, AioHandleShmemSize());
+ sz = add_size(sz, AioHandleIOVShmemSize());
+ sz = add_size(sz, AioHandleDataShmemSize());
+
+ if (pgaio_method_ops->shmem_size)
+ sz = add_size(sz, pgaio_method_ops->shmem_size());
+
return sz;
}
void
AioShmemInit(void)
{
+ bool found;
+ uint32 io_handle_off = 0;
+ uint32 iovec_off = 0;
+ uint32 per_backend_iovecs = io_max_concurrency * PG_IOV_MAX;
+
+ pgaio_ctl = (PgAioCtl *)
+ ShmemInitStruct("AioCtl", AioCtlShmemSize(), &found);
+
+ if (found)
+ goto out;
+
+ memset(pgaio_ctl, 0, AioCtlShmemSize());
+
+ pgaio_ctl->io_handle_count = AioProcs() * io_max_concurrency;
+ pgaio_ctl->iovec_count = AioProcs() * per_backend_iovecs;
+
+ pgaio_ctl->backend_state = (PgAioBackend *)
+ ShmemInitStruct("AioBackend", AioBackendShmemSize(), &found);
+
+ pgaio_ctl->io_handles = (PgAioHandle *)
+ ShmemInitStruct("AioHandle", AioHandleShmemSize(), &found);
+
+ pgaio_ctl->iovecs = (struct iovec *)
+ ShmemInitStruct("AioHandleIOV", AioHandleIOVShmemSize(), &found);
+ pgaio_ctl->handle_data = (uint64 *)
+ ShmemInitStruct("AioHandleData", AioHandleDataShmemSize(), &found);
+
+ for (int procno = 0; procno < AioProcs(); procno++)
+ {
+ PgAioBackend *bs = &pgaio_ctl->backend_state[procno];
+
+ bs->io_handle_off = io_handle_off;
+ io_handle_off += io_max_concurrency;
+
+ dclist_init(&bs->idle_ios);
+ memset(bs->staged_ios, 0, sizeof(PgAioHandle *) * PGAIO_SUBMIT_BATCH_SIZE);
+ dclist_init(&bs->in_flight_ios);
+
+ /* initialize per-backend IOs */
+ for (int i = 0; i < io_max_concurrency; i++)
+ {
+ PgAioHandle *ioh = &pgaio_ctl->io_handles[bs->io_handle_off + i];
+
+ ioh->generation = 1;
+ ioh->owner_procno = procno;
+ ioh->iovec_off = iovec_off;
+ ioh->handle_data_len = 0;
+ ioh->report_return = NULL;
+ ioh->resowner = NULL;
+ ioh->num_callbacks = 0;
+ ioh->distilled_result.status = ARS_UNKNOWN;
+ ioh->flags = 0;
+
+ ConditionVariableInit(&ioh->cv);
+
+ dclist_push_tail(&bs->idle_ios, &ioh->node);
+ iovec_off += PG_IOV_MAX;
+ }
+ }
+
+out:
+ /* Initialize IO method specific resources. */
+ if (pgaio_method_ops->shmem_init)
+ pgaio_method_ops->shmem_init(!found);
}
void
pgaio_init_backend(void)
{
+ /* shouldn't be initialized twice */
+ Assert(!pgaio_my_backend);
+
+ if (MyProc == NULL || MyProcNumber >= AioProcs())
+ elog(ERROR, "aio requires a normal PGPROC");
+
+ pgaio_my_backend = &pgaio_ctl->backend_state[MyProcNumber];
+
+ if (pgaio_method_ops->init_backend)
+ pgaio_method_ops->init_backend();
+
+ before_shmem_exit(pgaio_shutdown, 0);
}
diff --git a/src/backend/storage/aio/aio_io.c b/src/backend/storage/aio/aio_io.c
new file mode 100644
index 00000000000..df4931a84a5
--- /dev/null
+++ b/src/backend/storage/aio/aio_io.c
@@ -0,0 +1,182 @@
+/*-------------------------------------------------------------------------
+ *
+ * aio_io.c
+ * AIO - Low Level IO Handling
+ *
+ * Functions related to associating IO operations to IO Handles and IO-method
+ * independent support functions for actually performing IO.
+ *
+ *
+ * Portions Copyright (c) 1996-2025, PostgreSQL Global Development Group
+ * Portions Copyright (c) 1994, Regents of the University of California
+ *
+ * IDENTIFICATION
+ * src/backend/storage/aio/aio_io.c
+ *
+ *-------------------------------------------------------------------------
+ */
+
+#include "postgres.h"
+
+#include "miscadmin.h"
+#include "storage/aio.h"
+#include "storage/aio_internal.h"
+#include "storage/fd.h"
+#include "utils/wait_event.h"
+
+
+static void pgaio_io_before_prep(PgAioHandle *ioh);
+
+
+
+/* --------------------------------------------------------------------------------
+ * Public IO related functions operating on IO Handles
+ * --------------------------------------------------------------------------------
+ */
+
+/*
+ * Scatter/gather IO needs to associate an iovec with the Handle. To support
+ * worker mode this data needs to be in shared memory.
+ *
+ * XXX: Right now the amount of space available for each IO is
+ * PG_IOV_MAX. While it's tempting to use the io_combine_limit GUC, that's
+ * PGC_USERSET, so we can't allocate shared memory based on that.
+ */
+int
+pgaio_io_get_iovec(PgAioHandle *ioh, struct iovec **iov)
+{
+ Assert(ioh->state == PGAIO_HS_HANDED_OUT);
+
+ *iov = &pgaio_ctl->iovecs[ioh->iovec_off];
+
+ return PG_IOV_MAX;
+}
+
+PgAioOpData *
+pgaio_io_get_op_data(PgAioHandle *ioh)
+{
+ return &ioh->op_data;
+}
+
+
+
+/* --------------------------------------------------------------------------------
+ * "Preparation" routines for individual IO operations
+ *
+ * These are called by the code actually initiating an IO, to associate the IO
+ * specific data with an AIO handle.
+ *
+ * Each of the preparation routines first needs to call
+ * pgaio_io_before_prep(), then fill IO specific fields in the handle and then
+ * finally call pgaio_io_stage().
+ * --------------------------------------------------------------------------------
+ */
+
+void
+pgaio_io_prep_readv(PgAioHandle *ioh,
+ int fd, int iovcnt, uint64 offset)
+{
+ pgaio_io_before_prep(ioh);
+
+ ioh->op_data.read.fd = fd;
+ ioh->op_data.read.offset = offset;
+ ioh->op_data.read.iov_length = iovcnt;
+
+ pgaio_io_stage(ioh, PGAIO_OP_READV);
+}
+
+void
+pgaio_io_prep_writev(PgAioHandle *ioh,
+ int fd, int iovcnt, uint64 offset)
+{
+ pgaio_io_before_prep(ioh);
+
+ ioh->op_data.write.fd = fd;
+ ioh->op_data.write.offset = offset;
+ ioh->op_data.write.iov_length = iovcnt;
+
+ pgaio_io_stage(ioh, PGAIO_OP_WRITEV);
+}
+
+
+
+/* --------------------------------------------------------------------------------
+ * Internal IO related functions operating on IO Handles
+ * --------------------------------------------------------------------------------
+ */
+
+/*
+ * Execute IO operation synchronously. This is implemented here, not in
+ * method_sync.c, because other IO methods also might use it / fall back to
+ * it.
+ */
+void
+pgaio_io_perform_synchronously(PgAioHandle *ioh)
+{
+ ssize_t result = 0;
+ struct iovec *iov = &pgaio_ctl->iovecs[ioh->iovec_off];
+
+ START_CRIT_SECTION();
+
+ /* Perform IO. */
+ switch (ioh->op)
+ {
+ case PGAIO_OP_READV:
+ pgstat_report_wait_start(WAIT_EVENT_DATA_FILE_READ);
+ result = pg_preadv(ioh->op_data.read.fd, iov,
+ ioh->op_data.read.iov_length,
+ ioh->op_data.read.offset);
+ pgstat_report_wait_end();
+ break;
+ case PGAIO_OP_WRITEV:
+ pgstat_report_wait_start(WAIT_EVENT_DATA_FILE_WRITE);
+ result = pg_pwritev(ioh->op_data.write.fd, iov,
+ ioh->op_data.write.iov_length,
+ ioh->op_data.write.offset);
+ pgstat_report_wait_end();
+ break;
+ case PGAIO_OP_INVALID:
+ elog(ERROR, "trying to execute invalid IO operation");
+ }
+
+ ioh->result = result < 0 ? -errno : result;
+
+ pgaio_io_process_completion(ioh, ioh->result);
+
+ END_CRIT_SECTION();
+}
+
+/*
+ * Helper function to be called by IO operation preparation functions, before
+ * any data in the handle is set. Mostly to centralize assertions.
+ */
+static void
+pgaio_io_before_prep(PgAioHandle *ioh)
+{
+ Assert(ioh->state == PGAIO_HS_HANDED_OUT);
+ Assert(pgaio_my_backend->handed_out_io == ioh);
+ Assert(pgaio_io_has_target(ioh));
+ Assert(ioh->op == PGAIO_OP_INVALID);
+}
+
+/*
+ * Could be made part of the public interface, but it's not clear there's
+ * really a use case for that.
+ */
+const char *
+pgaio_io_get_op_name(PgAioHandle *ioh)
+{
+ Assert(ioh->op >= 0 && ioh->op < PGAIO_OP_COUNT);
+
+ switch (ioh->op)
+ {
+ case PGAIO_OP_INVALID:
+ return "invalid";
+ case PGAIO_OP_READV:
+ return "read";
+ case PGAIO_OP_WRITEV:
+ return "write";
+ }
+
+ return NULL; /* silence compiler */
+}
diff --git a/src/backend/storage/aio/aio_target.c b/src/backend/storage/aio/aio_target.c
new file mode 100644
index 00000000000..3fa813fd592
--- /dev/null
+++ b/src/backend/storage/aio/aio_target.c
@@ -0,0 +1,118 @@
+/*-------------------------------------------------------------------------
+ *
+ * aio_target.c
+ * AIO - Functionality related to executing IO for different targets
+ *
+ * Portions Copyright (c) 1996-2025, PostgreSQL Global Development Group
+ * Portions Copyright (c) 1994, Regents of the University of California
+ *
+ * IDENTIFICATION
+ * src/backend/storage/aio/aio_target.c
+ *
+ *-------------------------------------------------------------------------
+ */
+
+#include "postgres.h"
+
+#include "storage/aio.h"
+#include "storage/aio_internal.h"
+
+
+/*
+ * Registry for entities that can be the target of AIO.
+ *
+ * To support executing using worker processes, the file descriptor for an IO
+ * may need to be be reopened in a different process. This is done via the
+ * PgAioTargetInfo.reopen callback.
+ */
+static const PgAioTargetInfo *pgaio_target_info[] = {
+ [PGAIO_TID_INVALID] = &(PgAioTargetInfo) {
+ .name = "invalid",
+ },
+};
+
+
+
+/* --------------------------------------------------------------------------------
+ * Public target related functions operating on IO Handles
+ * --------------------------------------------------------------------------------
+ */
+
+bool
+pgaio_io_has_target(PgAioHandle *ioh)
+{
+ return ioh->target != PGAIO_TID_INVALID;
+}
+
+/*
+ * Return the name for the target associated with the IO. Mostly useful for
+ * debugging/logging.
+ */
+const char *
+pgaio_io_get_target_name(PgAioHandle *ioh)
+{
+ Assert(ioh->target >= 0 && ioh->target < PGAIO_TID_COUNT);
+
+ return pgaio_target_info[ioh->target]->name;
+}
+
+/*
+ * Assign a target to the IO.
+ *
+ * This has to be called exactly once before pgaio_io_prep_*() is called.
+ */
+void
+pgaio_io_set_target(PgAioHandle *ioh, PgAioTargetID targetid)
+{
+ Assert(ioh->state == PGAIO_HS_HANDED_OUT);
+ Assert(ioh->target == PGAIO_TID_INVALID);
+
+ ioh->target = targetid;
+}
+
+PgAioTargetData *
+pgaio_io_get_target_data(PgAioHandle *ioh)
+{
+ return &ioh->target_data;
+}
+
+/*
+ * Return a stringified description of the IO's target.
+ *
+ * The string is localized and allocated in the current memory context.
+ */
+char *
+pgaio_io_get_target_description(PgAioHandle *ioh)
+{
+ return pgaio_target_info[ioh->target]->describe_identity(&ioh->target_data);
+}
+
+
+
+/* --------------------------------------------------------------------------------
+ * Internal target related functions operating on IO Handles
+ * --------------------------------------------------------------------------------
+ */
+
+/*
+ * Internal: Check if pgaio_io_reopen() is available for the IO.
+ */
+bool
+pgaio_io_can_reopen(PgAioHandle *ioh)
+{
+ return pgaio_target_info[ioh->target]->reopen != NULL;
+}
+
+/*
+ * Internal: Before executing an IO outside of the context of the process the
+ * IO has been prepared in, the file descriptor has to be reopened - any FD
+ * referenced in the IO itself, won't be valid in the separate process.
+ */
+void
+pgaio_io_reopen(PgAioHandle *ioh)
+{
+ Assert(ioh->target >= 0 && ioh->target < PGAIO_TID_COUNT);
+ Assert(ioh->op >= 0 && ioh->op < PGAIO_OP_COUNT);
+
+ pgaio_target_info[ioh->target]->reopen(ioh);
+}
diff --git a/src/backend/storage/aio/meson.build b/src/backend/storage/aio/meson.build
index c822fd4ddf7..2c26089d52e 100644
--- a/src/backend/storage/aio/meson.build
+++ b/src/backend/storage/aio/meson.build
@@ -2,6 +2,10 @@
backend_sources += files(
'aio.c',
+ 'aio_callback.c',
'aio_init.c',
+ 'aio_io.c',
+ 'aio_target.c',
+ 'method_sync.c',
'read_stream.c',
)
diff --git a/src/backend/storage/aio/method_sync.c b/src/backend/storage/aio/method_sync.c
new file mode 100644
index 00000000000..902c2428d41
--- /dev/null
+++ b/src/backend/storage/aio/method_sync.c
@@ -0,0 +1,47 @@
+/*-------------------------------------------------------------------------
+ *
+ * method_sync.c
+ * AIO - perform "AIO" by executing it synchronously
+ *
+ * This method is mainly to check if AIO use causes regressions. Other IO
+ * methods might also fall back to the synchronous method for functionality
+ * they cannot provide.
+ *
+ * Portions Copyright (c) 1996-2025, PostgreSQL Global Development Group
+ * Portions Copyright (c) 1994, Regents of the University of California
+ *
+ * IDENTIFICATION
+ * src/backend/storage/aio/method_sync.c
+ *
+ *-------------------------------------------------------------------------
+ */
+
+#include "postgres.h"
+
+#include "storage/aio.h"
+#include "storage/aio_internal.h"
+
+static bool pgaio_sync_needs_synchronous_execution(PgAioHandle *ioh);
+static int pgaio_sync_submit(uint16 num_staged_ios, PgAioHandle **staged_ios);
+
+
+const IoMethodOps pgaio_sync_ops = {
+ .needs_synchronous_execution = pgaio_sync_needs_synchronous_execution,
+ .submit = pgaio_sync_submit,
+};
+
+
+
+static bool
+pgaio_sync_needs_synchronous_execution(PgAioHandle *ioh)
+{
+ return true;
+}
+
+static int
+pgaio_sync_submit(uint16 num_staged_ios, PgAioHandle **staged_ios)
+{
+ elog(ERROR, "IO should have been executed synchronously");
+
+ return 0;
+}
diff --git a/src/backend/utils/activity/wait_event_names.txt b/src/backend/utils/activity/wait_event_names.txt
index 3c594415bfd..b44e4908b25 100644
--- a/src/backend/utils/activity/wait_event_names.txt
+++ b/src/backend/utils/activity/wait_event_names.txt
@@ -191,6 +191,7 @@ ABI_compatibility:
Section: ClassName - WaitEventIO
+AIO_IO_COMPLETION "Waiting for IO completion."
BASEBACKUP_READ "Waiting for base backup to read from a file."
BASEBACKUP_SYNC "Waiting for data written by a base backup to reach durable storage."
BASEBACKUP_WRITE "Waiting for base backup to write to a file."
diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list
index 3c9e823f07e..f4261145353 100644
--- a/src/tools/pgindent/typedefs.list
+++ b/src/tools/pgindent/typedefs.list
@@ -1280,6 +1280,7 @@ InvalMessageArray
InvalidationInfo
InvalidationMsgsGroup
IoMethod
+IoMethodOps
IpcMemoryId
IpcMemoryKey
IpcMemoryState
@@ -2127,6 +2128,26 @@ Permutation
PermutationStep
PermutationStepBlocker
PermutationStepBlockerType
+PgAioBackend
+PgAioCtl
+PgAioHandle
+PgAioHandleCallbackID
+PgAioHandleCallbackStage
+PgAioHandleCallbackComplete
+PgAioHandleCallbackReport
+PgAioHandleCallbacks
+PgAioHandleCallbacksEntry
+PgAioHandleFlags
+PgAioHandleState
+PgAioOp
+PgAioOpData
+PgAioResult
+PgAioResultStatus
+PgAioReturn
+PgAioTargetData
+PgAioTargetID
+PgAioTargetInfo
+PgAioWaitRef
PgArchData
PgBackendGSSStatus
PgBackendSSLStatus
--
2.48.1.76.g4e746b1a31.dirty
[text/x-diff] v2.8-0003-aio-Infrastructure-for-io_method-worker.patch (24.9K, ../../3yxd5r23zly5bytvgyktbxtxq2r3gbpi7xd4dugevh3h4w4q6c@lu6oatjjpltz/4-v2.8-0003-aio-Infrastructure-for-io_method-worker.patch)
download | inline diff:
From ecba88025db2d79cb32f8f4b92175ea8e6b0716d Mon Sep 17 00:00:00 2001
From: Andres Freund <andres@anarazel.de>
Date: Mon, 17 Feb 2025 17:57:00 -0500
Subject: [PATCH v2.8 03/38] aio: Infrastructure for io_method=worker
This commit contains the basic, system-wide, infrastructure for
io_method=worker. It does not yet actually execute IO, this commit just
provides the infrastructure for running IO workers, kept separate for easier
review.
The number of IO workers can be adjusted with a PGC_SIGHUP GUC. Eventually
we'd like to make the number of workers dynamically scale up/down based on the
current "IO load".
To allow the number of IO workers to be increased without a restart, we need
to reserve PGPROC entries for the workers unconditionally. This has been
judged to be worth the cost. If it turns out to be problematic, we can
introduce a PGC_POSTMASTER GUC to control the maximum number.
Reviewed-by: Noah Misch <noah@leadboat.com>
Co-authored-by: Thomas Munro <thomas.munro@gmail.com>
Co-authored-by: Andres Freund <andres@anarazel.de>
Discussion: https://postgr.es/m/uvrtrknj4kdytuboidbhwclo4gxhswwcpgadptsjvjqcluzmah%40brqs62irg4dt
Discussion: https://postgr.es/m/20210223100344.llw5an2aklengrmn@alap3.anarazel.de
Discussion: https://postgr.es/m/stj36ea6yyhoxtqkhpieia2z4krnam7qyetc57rfezgk4zgapf@gcnactj4z56m
---
src/include/miscadmin.h | 2 +
src/include/storage/aio_subsys.h | 4 +
src/include/storage/io_worker.h | 22 +++
src/include/storage/proc.h | 4 +-
src/backend/postmaster/launch_backend.c | 2 +
src/backend/postmaster/pmchild.c | 1 +
src/backend/postmaster/postmaster.c | 174 ++++++++++++++++--
src/backend/storage/aio/Makefile | 1 +
src/backend/storage/aio/meson.build | 1 +
src/backend/storage/aio/method_worker.c | 88 +++++++++
src/backend/tcop/postgres.c | 2 +
src/backend/utils/activity/pgstat_backend.c | 1 +
src/backend/utils/activity/pgstat_io.c | 1 +
.../utils/activity/wait_event_names.txt | 1 +
src/backend/utils/init/miscinit.c | 3 +
src/backend/utils/misc/guc_tables.c | 13 ++
src/backend/utils/misc/postgresql.conf.sample | 1 +
doc/src/sgml/config.sgml | 19 ++
src/test/regress/expected/stats.out | 10 +-
19 files changed, 336 insertions(+), 14 deletions(-)
create mode 100644 src/include/storage/io_worker.h
create mode 100644 src/backend/storage/aio/method_worker.c
diff --git a/src/include/miscadmin.h b/src/include/miscadmin.h
index 6f16794eb63..603d0424354 100644
--- a/src/include/miscadmin.h
+++ b/src/include/miscadmin.h
@@ -360,6 +360,7 @@ typedef enum BackendType
B_ARCHIVER,
B_BG_WRITER,
B_CHECKPOINTER,
+ B_IO_WORKER,
B_STARTUP,
B_WAL_RECEIVER,
B_WAL_SUMMARIZER,
@@ -389,6 +390,7 @@ extern PGDLLIMPORT BackendType MyBackendType;
#define AmWalReceiverProcess() (MyBackendType == B_WAL_RECEIVER)
#define AmWalSummarizerProcess() (MyBackendType == B_WAL_SUMMARIZER)
#define AmWalWriterProcess() (MyBackendType == B_WAL_WRITER)
+#define AmIoWorkerProcess() (MyBackendType == B_IO_WORKER)
#define AmSpecialWorkerProcess() \
(AmAutoVacuumLauncherProcess() || \
diff --git a/src/include/storage/aio_subsys.h b/src/include/storage/aio_subsys.h
index e4faf692a38..ed00d5c47cd 100644
--- a/src/include/storage/aio_subsys.h
+++ b/src/include/storage/aio_subsys.h
@@ -30,4 +30,8 @@ extern void pgaio_init_backend(void);
extern void pgaio_error_cleanup(void);
extern void AtEOXact_Aio(bool is_commit);
+
+/* aio_worker.c */
+extern bool pgaio_workers_enabled(void);
+
#endif /* AIO_SUBSYS_H */
diff --git a/src/include/storage/io_worker.h b/src/include/storage/io_worker.h
new file mode 100644
index 00000000000..7bde7e89c8a
--- /dev/null
+++ b/src/include/storage/io_worker.h
@@ -0,0 +1,22 @@
+/*-------------------------------------------------------------------------
+ *
+ * io_worker.h
+ * IO worker for implementing AIO "ourselves"
+ *
+ *
+ * Portions Copyright (c) 1996-2025, PostgreSQL Global Development Group
+ * Portions Copyright (c) 1994, Regents of the University of California
+ *
+ * src/include/storage/io.h
+ *
+ *-------------------------------------------------------------------------
+ */
+#ifndef IO_WORKER_H
+#define IO_WORKER_H
+
+
+pg_noreturn extern void IoWorkerMain(const void *startup_data, size_t startup_data_len);
+
+extern PGDLLIMPORT int io_workers;
+
+#endif /* IO_WORKER_H */
diff --git a/src/include/storage/proc.h b/src/include/storage/proc.h
index 0750ec3c474..f51b03d3822 100644
--- a/src/include/storage/proc.h
+++ b/src/include/storage/proc.h
@@ -449,7 +449,9 @@ extern PGDLLIMPORT PGPROC *PreparedXactProcs;
* 2 slots, but WAL writer is launched only after startup has exited, so we
* only need 6 slots.
*/
-#define NUM_AUXILIARY_PROCS 6
+#define MAX_IO_WORKERS 32
+#define NUM_AUXILIARY_PROCS (6 + MAX_IO_WORKERS)
+
/* configurable options */
extern PGDLLIMPORT int DeadlockTimeout;
diff --git a/src/backend/postmaster/launch_backend.c b/src/backend/postmaster/launch_backend.c
index 77fb877dbad..bf6b55ee830 100644
--- a/src/backend/postmaster/launch_backend.c
+++ b/src/backend/postmaster/launch_backend.c
@@ -48,6 +48,7 @@
#include "replication/slotsync.h"
#include "replication/walreceiver.h"
#include "storage/dsm.h"
+#include "storage/io_worker.h"
#include "storage/pg_shmem.h"
#include "tcop/backend_startup.h"
#include "utils/memutils.h"
@@ -197,6 +198,7 @@ static child_process_kind child_process_kinds[] = {
[B_ARCHIVER] = {"archiver", PgArchiverMain, true},
[B_BG_WRITER] = {"bgwriter", BackgroundWriterMain, true},
[B_CHECKPOINTER] = {"checkpointer", CheckpointerMain, true},
+ [B_IO_WORKER] = {"io_worker", IoWorkerMain, true},
[B_STARTUP] = {"startup", StartupProcessMain, true},
[B_WAL_RECEIVER] = {"wal_receiver", WalReceiverMain, true},
[B_WAL_SUMMARIZER] = {"wal_summarizer", WalSummarizerMain, true},
diff --git a/src/backend/postmaster/pmchild.c b/src/backend/postmaster/pmchild.c
index 0d473226c3a..cde1d23a4ca 100644
--- a/src/backend/postmaster/pmchild.c
+++ b/src/backend/postmaster/pmchild.c
@@ -101,6 +101,7 @@ InitPostmasterChildSlots(void)
pmchild_pools[B_AUTOVAC_WORKER].size = autovacuum_worker_slots;
pmchild_pools[B_BG_WORKER].size = max_worker_processes;
+ pmchild_pools[B_IO_WORKER].size = MAX_IO_WORKERS;
/*
* There can be only one of each of these running at a time. They each
diff --git a/src/backend/postmaster/postmaster.c b/src/backend/postmaster/postmaster.c
index d13846298bd..f301e43743f 100644
--- a/src/backend/postmaster/postmaster.c
+++ b/src/backend/postmaster/postmaster.c
@@ -108,9 +108,12 @@
#include "replication/logicallauncher.h"
#include "replication/slotsync.h"
#include "replication/walsender.h"
+#include "storage/aio_subsys.h"
#include "storage/fd.h"
+#include "storage/io_worker.h"
#include "storage/ipc.h"
#include "storage/pmsignal.h"
+#include "storage/proc.h"
#include "tcop/backend_startup.h"
#include "tcop/tcopprot.h"
#include "utils/datetime.h"
@@ -340,6 +343,7 @@ typedef enum
* ckpt */
PM_WAIT_XLOG_ARCHIVAL, /* waiting for archiver and walsenders to
* finish */
+ PM_WAIT_IO_WORKERS, /* waiting for io workers to exit */
PM_WAIT_CHECKPOINTER, /* waiting for checkpointer to shut down */
PM_WAIT_DEAD_END, /* waiting for dead-end children to exit */
PM_NO_CHILDREN, /* all important children have exited */
@@ -402,6 +406,10 @@ bool LoadedSSL = false;
static DNSServiceRef bonjour_sdref = NULL;
#endif
+/* State for IO worker management. */
+static int io_worker_count = 0;
+static PMChild *io_worker_children[MAX_IO_WORKERS];
+
/*
* postmaster.c - function prototypes
*/
@@ -436,6 +444,8 @@ static void TerminateChildren(int signal);
static int CountChildren(BackendTypeMask targetMask);
static void LaunchMissingBackgroundProcesses(void);
static void maybe_start_bgworkers(void);
+static bool maybe_reap_io_worker(int pid);
+static void maybe_adjust_io_workers(void);
static bool CreateOptsFile(int argc, char *argv[], char *fullprogname);
static PMChild *StartChildProcess(BackendType type);
static void StartSysLogger(void);
@@ -1365,6 +1375,11 @@ PostmasterMain(int argc, char *argv[])
*/
AddToDataDirLockFile(LOCK_FILE_LINE_PM_STATUS, PM_STATUS_STARTING);
+ UpdatePMState(PM_STARTUP);
+
+ /* Make sure we can perform I/O while starting up. */
+ maybe_adjust_io_workers();
+
/* Start bgwriter and checkpointer so they can help with recovery */
if (CheckpointerPMChild == NULL)
CheckpointerPMChild = StartChildProcess(B_CHECKPOINTER);
@@ -1377,7 +1392,6 @@ PostmasterMain(int argc, char *argv[])
StartupPMChild = StartChildProcess(B_STARTUP);
Assert(StartupPMChild != NULL);
StartupStatus = STARTUP_RUNNING;
- UpdatePMState(PM_STARTUP);
/* Some workers may be scheduled to start now */
maybe_start_bgworkers();
@@ -2502,6 +2516,16 @@ process_pm_child_exit(void)
continue;
}
+ /* Was it an IO worker? */
+ if (maybe_reap_io_worker(pid))
+ {
+ if (!EXIT_STATUS_0(exitstatus) && !EXIT_STATUS_1(exitstatus))
+ HandleChildCrash(pid, exitstatus, _("io worker"));
+
+ maybe_adjust_io_workers();
+ continue;
+ }
+
/*
* Was it a backend or a background worker?
*/
@@ -2723,6 +2747,7 @@ HandleFatalError(QuitSignalReason reason, bool consider_sigabrt)
case PM_WAIT_XLOG_SHUTDOWN:
case PM_WAIT_XLOG_ARCHIVAL:
case PM_WAIT_CHECKPOINTER:
+ case PM_WAIT_IO_WORKERS:
/*
* NB: Similar code exists in PostmasterStateMachine()'s handling
@@ -2905,20 +2930,21 @@ PostmasterStateMachine(void)
/*
* If we are doing crash recovery or an immediate shutdown then we
- * expect archiver, checkpointer and walsender to exit as well,
- * otherwise not.
+ * expect archiver, checkpointer, io workers and walsender to exit as
+ * well, otherwise not.
*/
if (FatalError || Shutdown >= ImmediateShutdown)
targetMask = btmask_add(targetMask,
B_CHECKPOINTER,
B_ARCHIVER,
+ B_IO_WORKER,
B_WAL_SENDER);
/*
- * Normally walsenders and archiver will continue running; they will
- * be terminated later after writing the checkpoint record. We also
- * let dead-end children to keep running for now. The syslogger
- * process exits last.
+ * Normally archiver, checkpointer, IO workers and walsenders will
+ * continue running; they will be terminated later after writing the
+ * checkpoint record. We also let dead-end children to keep running
+ * for now. The syslogger process exits last.
*
* This assertion checks that we have covered all backend types,
* either by including them in targetMask, or by noting here that they
@@ -2933,12 +2959,13 @@ PostmasterStateMachine(void)
B_LOGGER);
/*
- * Archiver, checkpointer and walsender may or may not be in
- * targetMask already.
+ * Archiver, checkpointer, IO workers, and walsender may or may
+ * not be in targetMask already.
*/
remainMask = btmask_add(remainMask,
B_ARCHIVER,
B_CHECKPOINTER,
+ B_IO_WORKER,
B_WAL_SENDER);
/* these are not real postmaster children */
@@ -3039,11 +3066,25 @@ PostmasterStateMachine(void)
{
/*
* PM_WAIT_XLOG_ARCHIVAL state ends when there are no children other
- * than checkpointer, dead-end children and logger left. There
+ * than checkpointer, io workers and dead-end children left. There
* shouldn't be any regular backends left by now anyway; what we're
* really waiting for is for walsenders and archiver to exit.
*/
- if (CountChildren(btmask_all_except(B_CHECKPOINTER, B_LOGGER, B_DEAD_END_BACKEND)) == 0)
+ if (CountChildren(btmask_all_except(B_CHECKPOINTER, B_IO_WORKER,
+ B_LOGGER, B_DEAD_END_BACKEND)) == 0)
+ {
+ UpdatePMState(PM_WAIT_IO_WORKERS);
+ SignalChildren(SIGUSR2, btmask(B_IO_WORKER));
+ }
+ }
+
+ if (pmState == PM_WAIT_IO_WORKERS)
+ {
+ /*
+ * PM_WAIT_IO_WORKERS state ends when there's only checkpointer and
+ * dead_end children left.
+ */
+ if (io_worker_count == 0)
{
UpdatePMState(PM_WAIT_CHECKPOINTER);
@@ -3171,10 +3212,14 @@ PostmasterStateMachine(void)
/* re-create shared memory and semaphores */
CreateSharedMemoryAndSemaphores();
+ UpdatePMState(PM_STARTUP);
+
+ /* Make sure we can perform I/O while starting up. */
+ maybe_adjust_io_workers();
+
StartupPMChild = StartChildProcess(B_STARTUP);
Assert(StartupPMChild != NULL);
StartupStatus = STARTUP_RUNNING;
- UpdatePMState(PM_STARTUP);
/* crash recovery started, reset SIGKILL flag */
AbortStartTime = 0;
@@ -3198,6 +3243,7 @@ pmstate_name(PMState state)
PM_TOSTR_CASE(PM_WAIT_BACKENDS);
PM_TOSTR_CASE(PM_WAIT_XLOG_SHUTDOWN);
PM_TOSTR_CASE(PM_WAIT_XLOG_ARCHIVAL);
+ PM_TOSTR_CASE(PM_WAIT_IO_WORKERS);
PM_TOSTR_CASE(PM_WAIT_DEAD_END);
PM_TOSTR_CASE(PM_WAIT_CHECKPOINTER);
PM_TOSTR_CASE(PM_NO_CHILDREN);
@@ -3235,6 +3281,16 @@ LaunchMissingBackgroundProcesses(void)
if (SysLoggerPMChild == NULL && Logging_collector)
StartSysLogger();
+ /*
+ * The number of configured configured workers might have changed, or a
+ * prior start of a worker might have failed. Check if we need to
+ * start/stop any workers.
+ *
+ * A config file change will always lead to this function being called, so
+ * we always will process the config change in a timely manner.
+ */
+ maybe_adjust_io_workers();
+
/*
* The checkpointer and the background writer are active from the start,
* until shutdown is initiated.
@@ -4120,6 +4176,7 @@ bgworker_should_start_now(BgWorkerStartTime start_time)
case PM_WAIT_DEAD_END:
case PM_WAIT_XLOG_ARCHIVAL:
case PM_WAIT_XLOG_SHUTDOWN:
+ case PM_WAIT_IO_WORKERS:
case PM_WAIT_BACKENDS:
case PM_STOP_BACKENDS:
break;
@@ -4270,6 +4327,99 @@ maybe_start_bgworkers(void)
}
}
+static bool
+maybe_reap_io_worker(int pid)
+{
+ for (int id = 0; id < MAX_IO_WORKERS; ++id)
+ {
+ if (io_worker_children[id] &&
+ io_worker_children[id]->pid == pid)
+ {
+ ReleasePostmasterChildSlot(io_worker_children[id]);
+
+ --io_worker_count;
+ io_worker_children[id] = NULL;
+ return true;
+ }
+ }
+ return false;
+}
+
+/*
+ * Start or stop IO workers, to close the gap between the number of running
+ * workers and the number of configured workers. Used to respond to change of
+ * the io_workers GUC (by increasing and decreasing the number of workers), as
+ * well as workers terminating in response to errors (by starting
+ * "replacement" workers).
+ */
+static void
+maybe_adjust_io_workers(void)
+{
+ if (!pgaio_workers_enabled())
+ return;
+
+ /*
+ * If we're in final shutting down state, then we're just waiting for all
+ * processes to exit.
+ */
+ if (pmState >= PM_WAIT_IO_WORKERS)
+ return;
+
+ /* Don't start new workers during an immediate shutdown either. */
+ if (Shutdown >= ImmediateShutdown)
+ return;
+
+ /*
+ * Don't start new workers if we're in the shutdown phase of a crash
+ * restart. But we *do* need to start if we're already starting up again.
+ */
+ if (FatalError && pmState >= PM_STOP_BACKENDS)
+ return;
+
+ Assert(pmState < PM_WAIT_IO_WORKERS);
+
+ /* Not enough running? */
+ while (io_worker_count < io_workers)
+ {
+ PMChild *child;
+ int id;
+
+ /* find unused entry in io_worker_children array */
+ for (id = 0; id < MAX_IO_WORKERS; ++id)
+ {
+ if (io_worker_children[id] == NULL)
+ break;
+ }
+ if (id == MAX_IO_WORKERS)
+ elog(ERROR, "could not find a free IO worker ID");
+
+ /* Try to launch one. */
+ child = StartChildProcess(B_IO_WORKER);
+ if (child != NULL)
+ {
+ io_worker_children[id] = child;
+ ++io_worker_count;
+ }
+ else
+ break; /* XXX try again soon? */
+ }
+
+ /* Too many running? */
+ if (io_worker_count > io_workers)
+ {
+ /* ask the IO worker in the highest slot to exit */
+ for (int id = MAX_IO_WORKERS - 1; id >= 0; --id)
+ {
+ if (io_worker_children[id] != NULL)
+ {
+ kill(io_worker_children[id]->pid, SIGUSR2);
+ break;
+ }
+ }
+ }
+}
+
+
/*
* When a backend asks to be notified about worker state changes, we
* set a flag in its backend entry. The background worker machinery needs
diff --git a/src/backend/storage/aio/Makefile b/src/backend/storage/aio/Makefile
index 89f821ea7e1..f51c34a37f8 100644
--- a/src/backend/storage/aio/Makefile
+++ b/src/backend/storage/aio/Makefile
@@ -15,6 +15,7 @@ OBJS = \
aio_io.o \
aio_target.o \
method_sync.o \
+ method_worker.o \
read_stream.o
include $(top_srcdir)/src/backend/common.mk
diff --git a/src/backend/storage/aio/meson.build b/src/backend/storage/aio/meson.build
index 2c26089d52e..74f94c6e40b 100644
--- a/src/backend/storage/aio/meson.build
+++ b/src/backend/storage/aio/meson.build
@@ -7,5 +7,6 @@ backend_sources += files(
'aio_io.c',
'aio_target.c',
'method_sync.c',
+ 'method_worker.c',
'read_stream.c',
)
diff --git a/src/backend/storage/aio/method_worker.c b/src/backend/storage/aio/method_worker.c
new file mode 100644
index 00000000000..0ef9ef93e2b
--- /dev/null
+++ b/src/backend/storage/aio/method_worker.c
@@ -0,0 +1,88 @@
+/*-------------------------------------------------------------------------
+ *
+ * method_worker.c
+ * AIO - perform AIO using worker processes
+ *
+ * Portions Copyright (c) 1996-2025, PostgreSQL Global Development Group
+ * Portions Copyright (c) 1994, Regents of the University of California
+ *
+ * IDENTIFICATION
+ * src/backend/storage/aio/method_worker.c
+ *
+ *-------------------------------------------------------------------------
+ */
+
+#include "postgres.h"
+
+#include "libpq/pqsignal.h"
+#include "miscadmin.h"
+#include "postmaster/auxprocess.h"
+#include "postmaster/interrupt.h"
+#include "storage/aio_subsys.h"
+#include "storage/io_worker.h"
+#include "storage/ipc.h"
+#include "storage/latch.h"
+#include "storage/proc.h"
+#include "tcop/tcopprot.h"
+#include "utils/wait_event.h"
+
+
+/* GUCs */
+int io_workers = 3;
+
+
+void
+IoWorkerMain(const void *startup_data, size_t startup_data_len)
+{
+ sigjmp_buf local_sigjmp_buf;
+
+ MyBackendType = B_IO_WORKER;
+ AuxiliaryProcessMainCommon();
+
+ pqsignal(SIGHUP, SignalHandlerForConfigReload);
+ pqsignal(SIGINT, die); /* to allow manually triggering worker restart */
+
+ /*
+ * Ignore SIGTERM, will get explicit shutdown via SIGUSR2 later in the
+ * shutdown sequence, similar to checkpointer.
+ */
+ pqsignal(SIGTERM, SIG_IGN);
+ /* SIGQUIT handler was already set up by InitPostmasterChild */
+ pqsignal(SIGALRM, SIG_IGN);
+ pqsignal(SIGPIPE, SIG_IGN);
+ pqsignal(SIGUSR1, procsignal_sigusr1_handler);
+ pqsignal(SIGUSR2, SignalHandlerForShutdownRequest);
+
+ /* see PostgresMain() */
+ if (sigsetjmp(local_sigjmp_buf, 1) != 0)
+ {
+ error_context_stack = NULL;
+ HOLD_INTERRUPTS();
+
+ EmitErrorReport();
+
+ proc_exit(1);
+ }
+
+ /* We can now handle ereport(ERROR) */
+ PG_exception_stack = &local_sigjmp_buf;
+
+ sigprocmask(SIG_SETMASK, &UnBlockSig, NULL);
+
+ while (!ShutdownRequestPending)
+ {
+ WaitLatch(MyLatch, WL_LATCH_SET | WL_EXIT_ON_PM_DEATH, -1,
+ WAIT_EVENT_IO_WORKER_MAIN);
+ ResetLatch(MyLatch);
+ CHECK_FOR_INTERRUPTS();
+ }
+
+ proc_exit(0);
+}
+
+bool
+pgaio_workers_enabled(void)
+{
+ /* placeholder for future commit */
+ return false;
+}
diff --git a/src/backend/tcop/postgres.c b/src/backend/tcop/postgres.c
index 55ab2da299b..fcb23239d07 100644
--- a/src/backend/tcop/postgres.c
+++ b/src/backend/tcop/postgres.c
@@ -3316,6 +3316,8 @@ ProcessInterrupts(void)
(errcode(ERRCODE_ADMIN_SHUTDOWN),
errmsg("terminating background worker \"%s\" due to administrator command",
MyBgworkerEntry->bgw_type)));
+ else if (AmIoWorkerProcess())
+ proc_exit(0);
else
ereport(FATAL,
(errcode(ERRCODE_ADMIN_SHUTDOWN),
diff --git a/src/backend/utils/activity/pgstat_backend.c b/src/backend/utils/activity/pgstat_backend.c
index a8cb54a7732..5518a18e060 100644
--- a/src/backend/utils/activity/pgstat_backend.c
+++ b/src/backend/utils/activity/pgstat_backend.c
@@ -375,6 +375,7 @@ pgstat_tracks_backend_bktype(BackendType bktype)
case B_LOGGER:
case B_BG_WRITER:
case B_CHECKPOINTER:
+ case B_IO_WORKER:
case B_STARTUP:
return false;
diff --git a/src/backend/utils/activity/pgstat_io.c b/src/backend/utils/activity/pgstat_io.c
index eb575025596..c8de9c9e2d3 100644
--- a/src/backend/utils/activity/pgstat_io.c
+++ b/src/backend/utils/activity/pgstat_io.c
@@ -376,6 +376,7 @@ pgstat_tracks_io_bktype(BackendType bktype)
case B_BG_WORKER:
case B_BG_WRITER:
case B_CHECKPOINTER:
+ case B_IO_WORKER:
case B_SLOTSYNC_WORKER:
case B_STANDALONE_BACKEND:
case B_STARTUP:
diff --git a/src/backend/utils/activity/wait_event_names.txt b/src/backend/utils/activity/wait_event_names.txt
index b44e4908b25..3f6dc3876b4 100644
--- a/src/backend/utils/activity/wait_event_names.txt
+++ b/src/backend/utils/activity/wait_event_names.txt
@@ -57,6 +57,7 @@ BGWRITER_HIBERNATE "Waiting in background writer process, hibernating."
BGWRITER_MAIN "Waiting in main loop of background writer process."
CHECKPOINTER_MAIN "Waiting in main loop of checkpointer process."
CHECKPOINTER_SHUTDOWN "Waiting for checkpointer process to be terminated."
+IO_WORKER_MAIN "Waiting in main loop of IO Worker process."
LOGICAL_APPLY_MAIN "Waiting in main loop of logical replication apply process."
LOGICAL_LAUNCHER_MAIN "Waiting in main loop of logical replication launcher process."
LOGICAL_PARALLEL_APPLY_MAIN "Waiting in main loop of logical replication parallel apply process."
diff --git a/src/backend/utils/init/miscinit.c b/src/backend/utils/init/miscinit.c
index dc3521457c7..43b4dbccc3d 100644
--- a/src/backend/utils/init/miscinit.c
+++ b/src/backend/utils/init/miscinit.c
@@ -293,6 +293,9 @@ GetBackendTypeDesc(BackendType backendType)
case B_CHECKPOINTER:
backendDesc = gettext_noop("checkpointer");
break;
+ case B_IO_WORKER:
+ backendDesc = gettext_noop("io worker");
+ break;
case B_LOGGER:
backendDesc = gettext_noop("logger");
break;
diff --git a/src/backend/utils/misc/guc_tables.c b/src/backend/utils/misc/guc_tables.c
index 0d3ebf06a95..4cc19bef686 100644
--- a/src/backend/utils/misc/guc_tables.c
+++ b/src/backend/utils/misc/guc_tables.c
@@ -75,6 +75,7 @@
#include "storage/aio.h"
#include "storage/bufmgr.h"
#include "storage/bufpage.h"
+#include "storage/io_worker.h"
#include "storage/large_object.h"
#include "storage/pg_shmem.h"
#include "storage/predicate.h"
@@ -3267,6 +3268,18 @@ struct config_int ConfigureNamesInt[] =
check_io_max_concurrency, NULL, NULL
},
+ {
+ {"io_workers",
+ PGC_SIGHUP,
+ RESOURCES_IO,
+ gettext_noop("Number of IO worker processes, for io_method=worker."),
+ NULL,
+ },
+ &io_workers,
+ 3, 1, MAX_IO_WORKERS,
+ NULL, NULL, NULL
+ },
+
{
{"backend_flush_after", PGC_USERSET, RESOURCES_IO,
gettext_noop("Number of pages after which previously performed writes are flushed to disk."),
diff --git a/src/backend/utils/misc/postgresql.conf.sample b/src/backend/utils/misc/postgresql.conf.sample
index 43c2ec2153e..a4049ff0d9d 100644
--- a/src/backend/utils/misc/postgresql.conf.sample
+++ b/src/backend/utils/misc/postgresql.conf.sample
@@ -207,6 +207,7 @@
# can execute simultaneously
# -1 sets based on shared_buffers
# (change requires restart)
+#io_workers = 3 # 1-32;
# - Worker Processes -
diff --git a/doc/src/sgml/config.sgml b/doc/src/sgml/config.sgml
index 7ec18bb7627..baefb7af4cb 100644
--- a/doc/src/sgml/config.sgml
+++ b/doc/src/sgml/config.sgml
@@ -2689,6 +2689,25 @@ include_dir 'conf.d'
</listitem>
</varlistentry>
+ <varlistentry id="guc-io-workers" xreflabel="io_workers">
+ <term><varname>io_workers</varname> (<type>int</type>)
+ <indexterm>
+ <primary><varname>io_workers</varname> configuration parameter</primary>
+ </indexterm>
+ </term>
+ <listitem>
+ <para>
+ Selects the number of I/O worker processes to use. The default is
+ 3. This parameter can only be set in the
+ <filename>postgresql.conf</filename> file or on the server command
+ line.
+ </para>
+ <para>
+ Only has an effect if <xref linkend="guc-io-method"/> is set to
+ <literal>worker</literal>.
+ </para>
+ </listitem>
+ </varlistentry>
</variablelist>
</sect2>
diff --git a/src/test/regress/expected/stats.out b/src/test/regress/expected/stats.out
index f77caacc17d..cd08a2ca0af 100644
--- a/src/test/regress/expected/stats.out
+++ b/src/test/regress/expected/stats.out
@@ -51,6 +51,14 @@ client backend|relation|vacuum
client backend|temp relation|normal
client backend|wal|init
client backend|wal|normal
+io worker|relation|bulkread
+io worker|relation|bulkwrite
+io worker|relation|init
+io worker|relation|normal
+io worker|relation|vacuum
+io worker|temp relation|normal
+io worker|wal|init
+io worker|wal|normal
slotsync worker|relation|bulkread
slotsync worker|relation|bulkwrite
slotsync worker|relation|init
@@ -87,7 +95,7 @@ walsummarizer|wal|init
walsummarizer|wal|normal
walwriter|wal|init
walwriter|wal|normal
-(71 rows)
+(79 rows)
\a
-- ensure that both seqscan and indexscan plans are allowed
SET enable_seqscan TO on;
--
2.48.1.76.g4e746b1a31.dirty
[text/x-diff] v2.8-0004-aio-Add-io_method-worker.patch (20.2K, ../../3yxd5r23zly5bytvgyktbxtxq2r3gbpi7xd4dugevh3h4w4q6c@lu6oatjjpltz/5-v2.8-0004-aio-Add-io_method-worker.patch)
download | inline diff:
From 9afb73981705ff2973bbdd52bbf044b4adaa1f3d Mon Sep 17 00:00:00 2001
From: Andres Freund <andres@anarazel.de>
Date: Fri, 31 Jan 2025 13:46:35 -0500
Subject: [PATCH v2.8 04/38] aio: Add io_method=worker
The previous commit introduced the infrastructure to start io_workers. This
commit actually makes the workers execute IOs.
IO workers consume IOs from a shared memory submission queue, run traditional
synchronous system calls, and perform the shared completion handling
immediately. Client code submits most requests by pushing IOs into the
submission queue, and waits (if necessary) using condition variables. Some
IOs cannot be performed in another process due to lack of infrastructure for
reopening the file, and must processed synchronously by the client code when
submitted.
For now the default io_method is changed to "worker". We should re-evaluate
that around beta1, we might want to be careful and set the default to "sync"
for 18.
Reviewed-by: Noah Misch <noah@leadboat.com>
Co-authored-by: Thomas Munro <thomas.munro@gmail.com>
Co-authored-by: Andres Freund <andres@anarazel.de>
Discussion: https://postgr.es/m/uvrtrknj4kdytuboidbhwclo4gxhswwcpgadptsjvjqcluzmah%40brqs62irg4dt
Discussion: https://postgr.es/m/20210223100344.llw5an2aklengrmn@alap3.anarazel.de
Discussion: https://postgr.es/m/stj36ea6yyhoxtqkhpieia2z4krnam7qyetc57rfezgk4zgapf@gcnactj4z56m
---
src/include/storage/aio.h | 5 +-
src/include/storage/aio_internal.h | 1 +
src/include/storage/lwlocklist.h | 1 +
src/backend/storage/aio/aio.c | 2 +
src/backend/storage/aio/aio_init.c | 9 +
src/backend/storage/aio/method_worker.c | 431 +++++++++++++++++-
.../utils/activity/wait_event_names.txt | 1 +
src/backend/utils/misc/postgresql.conf.sample | 2 +-
doc/src/sgml/config.sgml | 5 +
src/tools/pgindent/typedefs.list | 3 +
10 files changed, 452 insertions(+), 8 deletions(-)
diff --git a/src/include/storage/aio.h b/src/include/storage/aio.h
index a0a0f23f3a5..769ea33ea96 100644
--- a/src/include/storage/aio.h
+++ b/src/include/storage/aio.h
@@ -27,10 +27,11 @@
typedef enum IoMethod
{
IOMETHOD_SYNC = 0,
+ IOMETHOD_WORKER,
} IoMethod;
-/* We'll default to synchronous execution. */
-#define DEFAULT_IO_METHOD IOMETHOD_SYNC
+/* We'll default to worker based execution. */
+#define DEFAULT_IO_METHOD IOMETHOD_WORKER
/*
diff --git a/src/include/storage/aio_internal.h b/src/include/storage/aio_internal.h
index 52b7751efa8..51f63586467 100644
--- a/src/include/storage/aio_internal.h
+++ b/src/include/storage/aio_internal.h
@@ -385,6 +385,7 @@ extern PgAioHandle *pgaio_inj_io_get(void);
/* Declarations for the tables of function pointers exposed by each IO method. */
extern PGDLLIMPORT const IoMethodOps pgaio_sync_ops;
+extern PGDLLIMPORT const IoMethodOps pgaio_worker_ops;
extern PGDLLIMPORT const IoMethodOps *pgaio_method_ops;
extern PGDLLIMPORT PgAioCtl *pgaio_ctl;
diff --git a/src/include/storage/lwlocklist.h b/src/include/storage/lwlocklist.h
index cf565452382..932024b1b0b 100644
--- a/src/include/storage/lwlocklist.h
+++ b/src/include/storage/lwlocklist.h
@@ -83,3 +83,4 @@ PG_LWLOCK(49, WALSummarizer)
PG_LWLOCK(50, DSMRegistry)
PG_LWLOCK(51, InjectionPoint)
PG_LWLOCK(52, SerialControl)
+PG_LWLOCK(53, AioWorkerSubmissionQueue)
diff --git a/src/backend/storage/aio/aio.c b/src/backend/storage/aio/aio.c
index b02349cbf99..f0bb2939d08 100644
--- a/src/backend/storage/aio/aio.c
+++ b/src/backend/storage/aio/aio.c
@@ -64,6 +64,7 @@ static void pgaio_io_wait(PgAioHandle *ioh, uint64 ref_generation);
/* Options for io_method. */
const struct config_enum_entry io_method_options[] = {
{"sync", IOMETHOD_SYNC, false},
+ {"worker", IOMETHOD_WORKER, false},
{NULL, 0, false}
};
@@ -80,6 +81,7 @@ PgAioBackend *pgaio_my_backend;
static const IoMethodOps *const pgaio_method_ops_table[] = {
[IOMETHOD_SYNC] = &pgaio_sync_ops,
+ [IOMETHOD_WORKER] = &pgaio_worker_ops,
};
/* callbacks for the configured io_method, set by assign_io_method */
diff --git a/src/backend/storage/aio/aio_init.c b/src/backend/storage/aio/aio_init.c
index e6c3e40f7d4..d0d916ea5c9 100644
--- a/src/backend/storage/aio/aio_init.c
+++ b/src/backend/storage/aio/aio_init.c
@@ -18,6 +18,7 @@
#include "storage/aio.h"
#include "storage/aio_internal.h"
#include "storage/aio_subsys.h"
+#include "storage/io_worker.h"
#include "storage/ipc.h"
#include "storage/proc.h"
#include "storage/shmem.h"
@@ -39,6 +40,11 @@ AioCtlShmemSize(void)
static uint32
AioProcs(void)
{
+ /*
+ * While AIO workers don't need their own AIO context, we can't currently
+ * guarantee nothing gets assigned to the a ProcNumber for an IO worker if
+ * we just subtracted MAX_IO_WORKERS.
+ */
return MaxBackends + NUM_AUXILIARY_PROCS;
}
@@ -212,6 +218,9 @@ pgaio_init_backend(void)
/* shouldn't be initialized twice */
Assert(!pgaio_my_backend);
+ if (MyBackendType == B_IO_WORKER)
+ return;
+
if (MyProc == NULL || MyProcNumber >= AioProcs())
elog(ERROR, "aio requires a normal PGPROC");
diff --git a/src/backend/storage/aio/method_worker.c b/src/backend/storage/aio/method_worker.c
index 0ef9ef93e2b..117c9a87db7 100644
--- a/src/backend/storage/aio/method_worker.c
+++ b/src/backend/storage/aio/method_worker.c
@@ -3,6 +3,21 @@
* method_worker.c
* AIO - perform AIO using worker processes
*
+ * IO workers consume IOs from a shared memory submission queue, run
+ * traditional synchronous system calls, and perform the shared completion
+ * handling immediately. Client code submits most requests by pushing IOs
+ * into the submission queue, and waits (if necessary) using condition
+ * variables. Some IOs cannot be performed in another process due to lack of
+ * infrastructure for reopening the file, and must processed synchronously by
+ * the client code when submitted.
+ *
+ * So that the submitter can make just one system call when submitting a batch
+ * of IOs, wakeups "fan out"; each woken IO worker can wake two more. XXX This
+ * could be improved by using futexes instead of latches to wake N waiters.
+ *
+ * This method of AIO is available in all builds on all operating systems, and
+ * is the default.
+ *
* Portions Copyright (c) 1996-2025, PostgreSQL Global Development Group
* Portions Copyright (c) 1994, Regents of the University of California
*
@@ -16,25 +31,325 @@
#include "libpq/pqsignal.h"
#include "miscadmin.h"
+#include "port/pg_bitutils.h"
#include "postmaster/auxprocess.h"
#include "postmaster/interrupt.h"
+#include "storage/aio.h"
+#include "storage/aio_internal.h"
#include "storage/aio_subsys.h"
#include "storage/io_worker.h"
#include "storage/ipc.h"
#include "storage/latch.h"
#include "storage/proc.h"
#include "tcop/tcopprot.h"
+#include "utils/ps_status.h"
#include "utils/wait_event.h"
+/* How many workers should each worker wake up if needed? */
+#define IO_WORKER_WAKEUP_FANOUT 2
+
+
+typedef struct AioWorkerSubmissionQueue
+{
+ uint32 size;
+ uint32 mask;
+ uint32 head;
+ uint32 tail;
+ uint32 ios[FLEXIBLE_ARRAY_MEMBER];
+} AioWorkerSubmissionQueue;
+
+typedef struct AioWorkerSlot
+{
+ Latch *latch;
+ bool in_use;
+} AioWorkerSlot;
+
+typedef struct AioWorkerControl
+{
+ uint64 idle_worker_mask;
+ AioWorkerSlot workers[FLEXIBLE_ARRAY_MEMBER];
+} AioWorkerControl;
+
+
+static size_t pgaio_worker_shmem_size(void);
+static void pgaio_worker_shmem_init(bool first_time);
+
+static bool pgaio_worker_needs_synchronous_execution(PgAioHandle *ioh);
+static int pgaio_worker_submit(uint16 num_staged_ios, PgAioHandle **staged_ios);
+
+
+const IoMethodOps pgaio_worker_ops = {
+ .shmem_size = pgaio_worker_shmem_size,
+ .shmem_init = pgaio_worker_shmem_init,
+
+ .needs_synchronous_execution = pgaio_worker_needs_synchronous_execution,
+ .submit = pgaio_worker_submit,
+};
+
+
/* GUCs */
int io_workers = 3;
+static int io_worker_queue_size = 64;
+static int MyIoWorkerId;
+static AioWorkerSubmissionQueue *io_worker_submission_queue;
+static AioWorkerControl *io_worker_control;
+
+
+static size_t
+pgaio_worker_shmem_size(void)
+{
+ return
+ offsetof(AioWorkerSubmissionQueue, ios) +
+ sizeof(uint32) * MAX_IO_WORKERS * io_max_concurrency +
+ offsetof(AioWorkerControl, workers) +
+ sizeof(AioWorkerSlot) * MAX_IO_WORKERS;
+}
+
+static void
+pgaio_worker_shmem_init(bool first_time)
+{
+ bool found;
+ int size;
+
+ /* Round size up to next power of two so we can make a mask. */
+ size = pg_nextpower2_32(io_worker_queue_size);
+
+ io_worker_submission_queue =
+ ShmemInitStruct("AioWorkerSubmissionQueue",
+ offsetof(AioWorkerSubmissionQueue, ios) +
+ sizeof(uint32) * size,
+ &found);
+ if (!found)
+ {
+ io_worker_submission_queue->size = size;
+ io_worker_submission_queue->head = 0;
+ io_worker_submission_queue->tail = 0;
+ }
+
+ io_worker_control =
+ ShmemInitStruct("AioWorkerControl",
+ offsetof(AioWorkerControl, workers) +
+ sizeof(AioWorkerSlot) * io_workers,
+ &found);
+ if (!found)
+ {
+ io_worker_control->idle_worker_mask = 0;
+ for (int i = 0; i < io_workers; ++i)
+ {
+ io_worker_control->workers[i].latch = NULL;
+ io_worker_control->workers[i].in_use = false;
+ }
+ }
+}
+
+static int
+pgaio_choose_idle_worker(void)
+{
+ int worker;
+
+ if (io_worker_control->idle_worker_mask == 0)
+ return -1;
+
+ /* Find the lowest bit position, and clear it. */
+ worker = pg_rightmost_one_pos64(io_worker_control->idle_worker_mask);
+ io_worker_control->idle_worker_mask &= ~(UINT64_C(1) << worker);
+
+ return worker;
+}
+
+static bool
+pgaio_worker_submission_queue_insert(PgAioHandle *ioh)
+{
+ AioWorkerSubmissionQueue *queue;
+ uint32 new_head;
+
+ queue = io_worker_submission_queue;
+ new_head = (queue->head + 1) & (queue->size - 1);
+ if (new_head == queue->tail)
+ {
+ pgaio_debug(DEBUG3, "io queue is full, at %u elements",
+ io_worker_submission_queue->size);
+ return false; /* full */
+ }
+
+ queue->ios[queue->head] = pgaio_io_get_id(ioh);
+ queue->head = new_head;
+
+ return true;
+}
+
+static uint32
+pgaio_worker_submission_queue_consume(void)
+{
+ AioWorkerSubmissionQueue *queue;
+ uint32 result;
+
+ queue = io_worker_submission_queue;
+ if (queue->tail == queue->head)
+ return UINT32_MAX; /* empty */
+
+ result = queue->ios[queue->tail];
+ queue->tail = (queue->tail + 1) & (queue->size - 1);
+
+ return result;
+}
+
+static uint32
+pgaio_worker_submission_queue_depth(void)
+{
+ uint32 head;
+ uint32 tail;
+
+ head = io_worker_submission_queue->head;
+ tail = io_worker_submission_queue->tail;
+
+ if (tail > head)
+ head += io_worker_submission_queue->size;
+
+ Assert(head >= tail);
+
+ return head - tail;
+}
+
+static bool
+pgaio_worker_needs_synchronous_execution(PgAioHandle *ioh)
+{
+ return
+ !IsUnderPostmaster
+ || ioh->flags & PGAIO_HF_REFERENCES_LOCAL
+ || !pgaio_io_can_reopen(ioh);
+}
+
+static void
+pgaio_worker_submit_internal(int nios, PgAioHandle *ios[])
+{
+ PgAioHandle *synchronous_ios[PGAIO_SUBMIT_BATCH_SIZE];
+ int nsync = 0;
+ Latch *wakeup = NULL;
+ int worker;
+
+ Assert(nios <= PGAIO_SUBMIT_BATCH_SIZE);
+
+ LWLockAcquire(AioWorkerSubmissionQueueLock, LW_EXCLUSIVE);
+ for (int i = 0; i < nios; ++i)
+ {
+ Assert(!pgaio_worker_needs_synchronous_execution(ios[i]));
+ if (!pgaio_worker_submission_queue_insert(ios[i]))
+ {
+ /*
+ * We'll do it synchronously, but only after we've sent as many as
+ * we can to workers, to maximize concurrency.
+ */
+ synchronous_ios[nsync++] = ios[i];
+ continue;
+ }
+
+ if (wakeup == NULL)
+ {
+ /* Choose an idle worker to wake up if we haven't already. */
+ worker = pgaio_choose_idle_worker();
+ if (worker >= 0)
+ wakeup = io_worker_control->workers[worker].latch;
+
+ pgaio_debug_io(DEBUG4, ios[i],
+ "choosing worker %d",
+ worker);
+ }
+ }
+ LWLockRelease(AioWorkerSubmissionQueueLock);
+
+ if (wakeup)
+ SetLatch(wakeup);
+
+ /* Run whatever is left synchronously. */
+ if (nsync > 0)
+ {
+ for (int i = 0; i < nsync; ++i)
+ {
+ pgaio_io_perform_synchronously(synchronous_ios[i]);
+ }
+ }
+}
+
+static int
+pgaio_worker_submit(uint16 num_staged_ios, PgAioHandle **staged_ios)
+{
+ for (int i = 0; i < num_staged_ios; i++)
+ {
+ PgAioHandle *ioh = staged_ios[i];
+
+ pgaio_io_prepare_submit(ioh);
+ }
+
+ pgaio_worker_submit_internal(num_staged_ios, staged_ios);
+
+ return num_staged_ios;
+}
+
+/*
+ * on_shmem_exit() callback that releases the worker's slot in
+ * io_worker_control.
+ */
+static void
+pgaio_worker_die(int code, Datum arg)
+{
+ LWLockAcquire(AioWorkerSubmissionQueueLock, LW_EXCLUSIVE);
+ Assert(io_worker_control->workers[MyIoWorkerId].in_use);
+ Assert(io_worker_control->workers[MyIoWorkerId].latch == MyLatch);
+
+ io_worker_control->workers[MyIoWorkerId].in_use = false;
+ io_worker_control->workers[MyIoWorkerId].latch = NULL;
+ LWLockRelease(AioWorkerSubmissionQueueLock);
+}
+
+/*
+ * Register the worker in shared memory, assign MyWorkerId and register a
+ * shutdown callback to release registration.
+ */
+static void
+pgaio_worker_register(void)
+{
+ MyIoWorkerId = -1;
+
+ /*
+ * XXX: This could do with more fine-grained locking. But it's also not
+ * very common for the number of workers to change at the moment...
+ */
+ LWLockAcquire(AioWorkerSubmissionQueueLock, LW_EXCLUSIVE);
+
+ for (int i = 0; i < io_workers; ++i)
+ {
+ if (!io_worker_control->workers[i].in_use)
+ {
+ Assert(io_worker_control->workers[i].latch == NULL);
+ io_worker_control->workers[i].in_use = true;
+ MyIoWorkerId = i;
+ break;
+ }
+ else
+ Assert(io_worker_control->workers[i].latch != NULL);
+ }
+
+ if (MyIoWorkerId == -1)
+ elog(ERROR, "couldn't find a free worker slot");
+
+ io_worker_control->idle_worker_mask |= (UINT64_C(1) << MyIoWorkerId);
+ io_worker_control->workers[MyIoWorkerId].latch = MyLatch;
+ LWLockRelease(AioWorkerSubmissionQueueLock);
+
+ on_shmem_exit(pgaio_worker_die, 0);
+}
+
void
IoWorkerMain(const void *startup_data, size_t startup_data_len)
{
sigjmp_buf local_sigjmp_buf;
+ PgAioHandle *volatile error_ioh = NULL;
+ volatile int error_errno = 0;
+ char cmd[128];
MyBackendType = B_IO_WORKER;
AuxiliaryProcessMainCommon();
@@ -53,6 +368,12 @@ IoWorkerMain(const void *startup_data, size_t startup_data_len)
pqsignal(SIGUSR1, procsignal_sigusr1_handler);
pqsignal(SIGUSR2, SignalHandlerForShutdownRequest);
+ /* also registers a shutdown callback to unregister */
+ pgaio_worker_register();
+
+ sprintf(cmd, "io worker: %d", MyIoWorkerId);
+ set_ps_display(cmd);
+
/* see PostgresMain() */
if (sigsetjmp(local_sigjmp_buf, 1) != 0)
{
@@ -61,6 +382,27 @@ IoWorkerMain(const void *startup_data, size_t startup_data_len)
EmitErrorReport();
+ /*
+ * In the - very unlikely - case that the IO failed in a way that
+ * raises an error we need to mark the IO as failed.
+ *
+ * Need to do just enough error recovery so that we can mark the IO as
+ * failed and then exit (postmaster will start a new worker).
+ */
+ LWLockReleaseAll();
+
+ if (error_ioh != NULL)
+ {
+ /* should never fail without setting error_errno */
+ Assert(error_errno != 0);
+
+ errno = error_errno;
+
+ START_CRIT_SECTION();
+ pgaio_io_process_completion(error_ioh, -error_errno);
+ END_CRIT_SECTION();
+ }
+
proc_exit(1);
}
@@ -71,9 +413,89 @@ IoWorkerMain(const void *startup_data, size_t startup_data_len)
while (!ShutdownRequestPending)
{
- WaitLatch(MyLatch, WL_LATCH_SET | WL_EXIT_ON_PM_DEATH, -1,
- WAIT_EVENT_IO_WORKER_MAIN);
- ResetLatch(MyLatch);
+ uint32 io_index;
+ Latch *latches[IO_WORKER_WAKEUP_FANOUT];
+ int nlatches = 0;
+ int nwakeups = 0;
+ int worker;
+
+ /* Try to get a job to do. */
+ LWLockAcquire(AioWorkerSubmissionQueueLock, LW_EXCLUSIVE);
+ if ((io_index = pgaio_worker_submission_queue_consume()) == UINT32_MAX)
+ {
+ /*
+ * Nothing to do. Mark self idle.
+ *
+ * XXX: Invent some kind of back pressure to reduce useless
+ * wakeups?
+ */
+ io_worker_control->idle_worker_mask |= (UINT64_C(1) << MyIoWorkerId);
+ }
+ else
+ {
+ /* Got one. Clear idle flag. */
+ io_worker_control->idle_worker_mask &= ~(UINT64_C(1) << MyIoWorkerId);
+
+ /* See if we can wake up some peers. */
+ nwakeups = Min(pgaio_worker_submission_queue_depth(),
+ IO_WORKER_WAKEUP_FANOUT);
+ for (int i = 0; i < nwakeups; ++i)
+ {
+ if ((worker = pgaio_choose_idle_worker()) < 0)
+ break;
+ latches[nlatches++] = io_worker_control->workers[worker].latch;
+ }
+ }
+ LWLockRelease(AioWorkerSubmissionQueueLock);
+
+ for (int i = 0; i < nlatches; ++i)
+ SetLatch(latches[i]);
+
+ if (io_index != UINT32_MAX)
+ {
+ PgAioHandle *ioh = NULL;
+
+ ioh = &pgaio_ctl->io_handles[io_index];
+ error_ioh = ioh;
+
+ pgaio_debug_io(DEBUG4, ioh,
+ "worker %d processing IO",
+ MyIoWorkerId);
+
+ /*
+ * It's very unlikely, but possible, that reopen fails. E.g. due
+ * to memory allocations failing or file permissions changing or
+ * such. In that case we need to fail the IO.
+ *
+ * There's not really a good errno we can report here.
+ */
+ error_errno = ENOENT;
+ pgaio_io_reopen(ioh);
+
+ /*
+ * To be able to exercise the reopen-fails path, allow injection
+ * points to trigger a failure at this point.
+ */
+ pgaio_io_call_inj(ioh, "AIO_WORKER_AFTER_REOPEN");
+
+ error_errno = 0;
+ error_ioh = NULL;
+
+ /*
+ * We don't expect this to ever fail with ERROR or FATAL, no need
+ * to keep error_ioh set to the IO.
+ * pgaio_io_perform_synchronously() contains a critical section to
+ * ensure we don't accidentally fail.
+ */
+ pgaio_io_perform_synchronously(ioh);
+ }
+ else
+ {
+ WaitLatch(MyLatch, WL_LATCH_SET | WL_EXIT_ON_PM_DEATH, -1,
+ WAIT_EVENT_IO_WORKER_MAIN);
+ ResetLatch(MyLatch);
+ }
+
CHECK_FOR_INTERRUPTS();
}
@@ -83,6 +505,5 @@ IoWorkerMain(const void *startup_data, size_t startup_data_len)
bool
pgaio_workers_enabled(void)
{
- /* placeholder for future commit */
- return false;
+ return io_method == IOMETHOD_WORKER;
}
diff --git a/src/backend/utils/activity/wait_event_names.txt b/src/backend/utils/activity/wait_event_names.txt
index 3f6dc3876b4..9fa12a555e8 100644
--- a/src/backend/utils/activity/wait_event_names.txt
+++ b/src/backend/utils/activity/wait_event_names.txt
@@ -348,6 +348,7 @@ WALSummarizer "Waiting to read or update WAL summarization state."
DSMRegistry "Waiting to read or update the dynamic shared memory registry."
InjectionPoint "Waiting to read or update information related to injection points."
SerialControl "Waiting to read or update shared <filename>pg_serial</filename> state."
+AioWorkerSubmissionQueue "Waiting to access AIO worker submission queue."
#
# END OF PREDEFINED LWLOCKS (DO NOT CHANGE THIS LINE)
diff --git a/src/backend/utils/misc/postgresql.conf.sample b/src/backend/utils/misc/postgresql.conf.sample
index a4049ff0d9d..75a844a9474 100644
--- a/src/backend/utils/misc/postgresql.conf.sample
+++ b/src/backend/utils/misc/postgresql.conf.sample
@@ -202,7 +202,7 @@
#maintenance_io_concurrency = 10 # 1-1000; 0 disables prefetching
#io_combine_limit = 128kB # usually 1-32 blocks (depends on OS)
-#io_method = sync # sync (change requires restart)
+#io_method = worker # worker, sync (change requires restart)
#io_max_concurrency = -1 # Max number of IOs that one process
# can execute simultaneously
# -1 sets based on shared_buffers
diff --git a/doc/src/sgml/config.sgml b/doc/src/sgml/config.sgml
index baefb7af4cb..108caea1fed 100644
--- a/doc/src/sgml/config.sgml
+++ b/doc/src/sgml/config.sgml
@@ -2676,6 +2676,11 @@ include_dir 'conf.d'
Selects the method for executing asynchronous I/O.
Possible values are:
<itemizedlist>
+ <listitem>
+ <para>
+ <literal>worker</literal> (execute asynchronous I/O using worker processes)
+ </para>
+ </listitem>
<listitem>
<para>
<literal>sync</literal> (execute asynchronous-eligible I/O synchronously)
diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list
index f4261145353..4d8b5c7f1d6 100644
--- a/src/tools/pgindent/typedefs.list
+++ b/src/tools/pgindent/typedefs.list
@@ -55,6 +55,9 @@ AggStrategy
AggTransInfo
Aggref
AggregateInstrumentation
+AioWorkerControl
+AioWorkerSlot
+AioWorkerSubmissionQueue
AlenState
Alias
AllocBlock
--
2.48.1.76.g4e746b1a31.dirty
[text/x-diff] v2.8-0005-aio-Add-liburing-dependency.patch (13.4K, ../../3yxd5r23zly5bytvgyktbxtxq2r3gbpi7xd4dugevh3h4w4q6c@lu6oatjjpltz/6-v2.8-0005-aio-Add-liburing-dependency.patch)
download | inline diff:
From fd2957cc938a5c071144a8d1500bda52bf2411d4 Mon Sep 17 00:00:00 2001
From: Andres Freund <andres@anarazel.de>
Date: Thu, 20 Feb 2025 17:56:05 -0500
Subject: [PATCH v2.8 05/38] aio: Add liburing dependency
Will be used in a subsequent commit, to implement io_method=io_uring. Kept
separate for easier review.
Reviewed-by: Noah Misch <noah@leadboat.com>
Discussion: https://postgr.es/m/uvrtrknj4kdytuboidbhwclo4gxhswwcpgadptsjvjqcluzmah%40brqs62irg4dt
---
meson.build | 14 ++++
meson_options.txt | 3 +
configure.ac | 11 +++
src/makefiles/meson.build | 3 +
src/include/pg_config.h.in | 3 +
src/backend/Makefile | 7 +-
doc/src/sgml/installation.sgml | 34 ++++++++
configure | 138 +++++++++++++++++++++++++++++++++
.cirrus.tasks.yml | 1 +
src/Makefile.global.in | 4 +
10 files changed, 215 insertions(+), 3 deletions(-)
diff --git a/meson.build b/meson.build
index 13c13748e5d..bed36153c7b 100644
--- a/meson.build
+++ b/meson.build
@@ -950,6 +950,18 @@ endif
+###############################################################
+# Library: liburing
+###############################################################
+
+liburingopt = get_option('liburing')
+liburing = dependency('liburing', required: liburingopt)
+if liburing.found()
+ cdata.set('USE_LIBURING', 1)
+endif
+
+
+
###############################################################
# Library: libxml
###############################################################
@@ -3168,6 +3180,7 @@ backend_both_deps += [
icu_i18n,
ldap,
libintl,
+ liburing,
libxml,
lz4,
pam,
@@ -3823,6 +3836,7 @@ if meson.version().version_compare('>=0.57')
'icu': icu,
'ldap': ldap,
'libcurl': libcurl,
+ 'liburing': liburing,
'libxml': libxml,
'libxslt': libxslt,
'llvm': llvm,
diff --git a/meson_options.txt b/meson_options.txt
index 702c4517145..dd7126da3a7 100644
--- a/meson_options.txt
+++ b/meson_options.txt
@@ -106,6 +106,9 @@ option('libcurl', type : 'feature', value: 'auto',
option('libedit_preferred', type: 'boolean', value: false,
description: 'Prefer BSD Libedit over GNU Readline')
+option('liburing', type : 'feature', value: 'auto',
+ description: 'io_uring support, for asynchronous I/O')
+
option('libxml', type: 'feature', value: 'auto',
description: 'XML support')
diff --git a/configure.ac b/configure.ac
index b6d02f5ecc7..3e2d0dc59c0 100644
--- a/configure.ac
+++ b/configure.ac
@@ -975,6 +975,14 @@ AC_SUBST(with_readline)
PGAC_ARG_BOOL(with, libedit-preferred, no,
[prefer BSD Libedit over GNU Readline])
+#
+# liburing
+#
+AC_MSG_CHECKING([whether to build with liburing support])
+PGAC_ARG_BOOL(with, liburing, no, [io_uring support, for asynchronous I/O],
+ [AC_DEFINE([USE_LIBURING], 1, [Define to build with io_uring support. (--with-liburing)])])
+AC_MSG_RESULT([$with_liburing])
+AC_SUBST(with_liburing)
#
# UUID library
@@ -1463,6 +1471,9 @@ elif test "$with_uuid" = ossp ; then
fi
AC_SUBST(UUID_LIBS)
+if test "$with_liburing" = yes; then
+ PKG_CHECK_MODULES(LIBURING, liburing)
+fi
##
## Header files
diff --git a/src/makefiles/meson.build b/src/makefiles/meson.build
index 60e13d50235..46d8da070e8 100644
--- a/src/makefiles/meson.build
+++ b/src/makefiles/meson.build
@@ -199,6 +199,8 @@ pgxs_empty = [
'PTHREAD_CFLAGS', 'PTHREAD_LIBS',
'ICU_LIBS',
+
+ 'LIBURING_CFLAGS', 'LIBURING_LIBS',
]
if host_system == 'windows' and cc.get_argument_syntax() != 'msvc'
@@ -230,6 +232,7 @@ pgxs_deps = {
'icu': icu,
'ldap': ldap,
'libcurl': libcurl,
+ 'liburing': liburing,
'libxml': libxml,
'libxslt': libxslt,
'llvm': llvm,
diff --git a/src/include/pg_config.h.in b/src/include/pg_config.h.in
index db6454090d2..f2422241133 100644
--- a/src/include/pg_config.h.in
+++ b/src/include/pg_config.h.in
@@ -672,6 +672,9 @@
/* Define to 1 to build with libcurl support. (--with-libcurl) */
#undef USE_LIBCURL
+/* Define to build with io_uring support. (--with-liburing) */
+#undef USE_LIBURING
+
/* Define to 1 to build with XML support. (--with-libxml) */
#undef USE_LIBXML
diff --git a/src/backend/Makefile b/src/backend/Makefile
index 42d4a28e5aa..7344c8c7f5c 100644
--- a/src/backend/Makefile
+++ b/src/backend/Makefile
@@ -43,9 +43,10 @@ OBJS = \
$(top_builddir)/src/common/libpgcommon_srv.a \
$(top_builddir)/src/port/libpgport_srv.a
-# We put libpgport and libpgcommon into OBJS, so remove it from LIBS; also add
-# libldap and ICU
-LIBS := $(filter-out -lpgport -lpgcommon, $(LIBS)) $(LDAP_LIBS_BE) $(ICU_LIBS)
+# We put libpgport and libpgcommon into OBJS, so remove it from LIBS.
+LIBS := $(filter-out -lpgport -lpgcommon, $(LIBS))
+# The backend conditionally needs libraries that most executables don't need.
+LIBS += $(LDAP_LIBS_BE) $(ICU_LIBS) $(LIBURING_LIBS)
# The backend doesn't need everything that's in LIBS, however
LIBS := $(filter-out -lreadline -ledit -ltermcap -lncurses -lcurses, $(LIBS))
diff --git a/doc/src/sgml/installation.sgml b/doc/src/sgml/installation.sgml
index e076cefa3b9..cc28f041330 100644
--- a/doc/src/sgml/installation.sgml
+++ b/doc/src/sgml/installation.sgml
@@ -1156,6 +1156,24 @@ build-postgresql:
</listitem>
</varlistentry>
+ <varlistentry id="configure-option-with-liburing">
+ <term><option>--with-liburing</option></term>
+ <listitem>
+ <para>
+ Build with liburing, enabling io_uring support for asynchronous I/O.
+ </para>
+ <para>
+ To detect the required compiler and linker options, PostgreSQL will
+ query <command>pkg-config</command>.
+ </para>
+ <para>
+ To use a liburing installation that is in an unusual location, you
+ can set <command>pkg-config</command>-related environment
+ variables (see its documentation).
+ </para>
+ </listitem>
+ </varlistentry>
+
<varlistentry id="configure-option-with-libxml">
<term><option>--with-libxml</option></term>
<listitem>
@@ -2611,6 +2629,22 @@ ninja install
</listitem>
</varlistentry>
+ <varlistentry id="configure-with-liburing-meson">
+ <term><option>-Dliburing={ auto | enabled | disabled }</option></term>
+ <listitem>
+ <para>
+ Build with liburing, enabling io_uring support for asynchronous I/O.
+ Defaults to auto.
+ </para>
+
+ <para>
+ To use a liburing installation that is in an unusual location, you
+ can set <command>pkg-config</command>-related environment
+ variables (see its documentation).
+ </para>
+ </listitem>
+ </varlistentry>
+
<varlistentry id="configure-with-libxml-meson">
<term><option>-Dlibxml={ auto | enabled | disabled }</option></term>
<listitem>
diff --git a/configure b/configure
index 93fddd69981..8ffcdfc2e1c 100755
--- a/configure
+++ b/configure
@@ -651,6 +651,8 @@ LIBOBJS
OPENSSL
ZSTD
LZ4
+LIBURING_LIBS
+LIBURING_CFLAGS
UUID_LIBS
LDAP_LIBS_BE
LDAP_LIBS_FE
@@ -712,6 +714,7 @@ LIBCURL_LIBS
LIBCURL_CFLAGS
with_libcurl
with_uuid
+with_liburing
with_readline
with_systemd
with_selinux
@@ -865,6 +868,7 @@ with_selinux
with_systemd
with_readline
with_libedit_preferred
+with_liburing
with_uuid
with_ossp_uuid
with_libcurl
@@ -911,6 +915,8 @@ LDFLAGS_EX
LDFLAGS_SL
PERL
PYTHON
+LIBURING_CFLAGS
+LIBURING_LIBS
MSGFMT
TCLSH'
@@ -1578,6 +1584,7 @@ Optional Packages:
--without-readline do not use GNU Readline nor BSD Libedit for editing
--with-libedit-preferred
prefer BSD Libedit over GNU Readline
+ --with-liburing io_uring support, for asynchronous I/O
--with-uuid=LIB build contrib/uuid-ossp using LIB (bsd,e2fs,ossp)
--with-ossp-uuid obsolete spelling of --with-uuid=ossp
--with-libcurl build with libcurl support
@@ -1629,6 +1636,10 @@ Some influential environment variables:
LDFLAGS_SL extra linker flags for linking shared libraries only
PERL Perl program
PYTHON Python program
+ LIBURING_CFLAGS
+ C compiler flags for LIBURING, overriding pkg-config
+ LIBURING_LIBS
+ linker flags for LIBURING, overriding pkg-config
MSGFMT msgfmt program for NLS
TCLSH Tcl interpreter program (tclsh)
@@ -8692,6 +8703,40 @@ fi
+#
+# liburing
+#
+{ $as_echo "$as_me:${as_lineno-$LINENO}: checking whether to build with liburing support" >&5
+$as_echo_n "checking whether to build with liburing support... " >&6; }
+
+
+
+# Check whether --with-liburing was given.
+if test "${with_liburing+set}" = set; then :
+ withval=$with_liburing;
+ case $withval in
+ yes)
+
+$as_echo "#define USE_LIBURING 1" >>confdefs.h
+
+ ;;
+ no)
+ :
+ ;;
+ *)
+ as_fn_error $? "no argument expected for --with-liburing option" "$LINENO" 5
+ ;;
+ esac
+
+else
+ with_liburing=no
+
+fi
+
+
+{ $as_echo "$as_me:${as_lineno-$LINENO}: result: $with_liburing" >&5
+$as_echo "$with_liburing" >&6; }
+
#
# UUID library
@@ -13444,6 +13489,99 @@ fi
fi
+if test "$with_liburing" = yes; then
+
+pkg_failed=no
+{ $as_echo "$as_me:${as_lineno-$LINENO}: checking for liburing" >&5
+$as_echo_n "checking for liburing... " >&6; }
+
+if test -n "$LIBURING_CFLAGS"; then
+ pkg_cv_LIBURING_CFLAGS="$LIBURING_CFLAGS"
+ elif test -n "$PKG_CONFIG"; then
+ if test -n "$PKG_CONFIG" && \
+ { { $as_echo "$as_me:${as_lineno-$LINENO}: \$PKG_CONFIG --exists --print-errors \"liburing\""; } >&5
+ ($PKG_CONFIG --exists --print-errors "liburing") 2>&5
+ ac_status=$?
+ $as_echo "$as_me:${as_lineno-$LINENO}: \$? = $ac_status" >&5
+ test $ac_status = 0; }; then
+ pkg_cv_LIBURING_CFLAGS=`$PKG_CONFIG --cflags "liburing" 2>/dev/null`
+ test "x$?" != "x0" && pkg_failed=yes
+else
+ pkg_failed=yes
+fi
+ else
+ pkg_failed=untried
+fi
+if test -n "$LIBURING_LIBS"; then
+ pkg_cv_LIBURING_LIBS="$LIBURING_LIBS"
+ elif test -n "$PKG_CONFIG"; then
+ if test -n "$PKG_CONFIG" && \
+ { { $as_echo "$as_me:${as_lineno-$LINENO}: \$PKG_CONFIG --exists --print-errors \"liburing\""; } >&5
+ ($PKG_CONFIG --exists --print-errors "liburing") 2>&5
+ ac_status=$?
+ $as_echo "$as_me:${as_lineno-$LINENO}: \$? = $ac_status" >&5
+ test $ac_status = 0; }; then
+ pkg_cv_LIBURING_LIBS=`$PKG_CONFIG --libs "liburing" 2>/dev/null`
+ test "x$?" != "x0" && pkg_failed=yes
+else
+ pkg_failed=yes
+fi
+ else
+ pkg_failed=untried
+fi
+
+
+
+if test $pkg_failed = yes; then
+ { $as_echo "$as_me:${as_lineno-$LINENO}: result: no" >&5
+$as_echo "no" >&6; }
+
+if $PKG_CONFIG --atleast-pkgconfig-version 0.20; then
+ _pkg_short_errors_supported=yes
+else
+ _pkg_short_errors_supported=no
+fi
+ if test $_pkg_short_errors_supported = yes; then
+ LIBURING_PKG_ERRORS=`$PKG_CONFIG --short-errors --print-errors --cflags --libs "liburing" 2>&1`
+ else
+ LIBURING_PKG_ERRORS=`$PKG_CONFIG --print-errors --cflags --libs "liburing" 2>&1`
+ fi
+ # Put the nasty error message in config.log where it belongs
+ echo "$LIBURING_PKG_ERRORS" >&5
+
+ as_fn_error $? "Package requirements (liburing) were not met:
+
+$LIBURING_PKG_ERRORS
+
+Consider adjusting the PKG_CONFIG_PATH environment variable if you
+installed software in a non-standard prefix.
+
+Alternatively, you may set the environment variables LIBURING_CFLAGS
+and LIBURING_LIBS to avoid the need to call pkg-config.
+See the pkg-config man page for more details." "$LINENO" 5
+elif test $pkg_failed = untried; then
+ { $as_echo "$as_me:${as_lineno-$LINENO}: result: no" >&5
+$as_echo "no" >&6; }
+ { { $as_echo "$as_me:${as_lineno-$LINENO}: error: in \`$ac_pwd':" >&5
+$as_echo "$as_me: error: in \`$ac_pwd':" >&2;}
+as_fn_error $? "The pkg-config script could not be found or is too old. Make sure it
+is in your PATH or set the PKG_CONFIG environment variable to the full
+path to pkg-config.
+
+Alternatively, you may set the environment variables LIBURING_CFLAGS
+and LIBURING_LIBS to avoid the need to call pkg-config.
+See the pkg-config man page for more details.
+
+To get pkg-config, see <http://pkg-config.freedesktop.org/>.
+See \`config.log' for more details" "$LINENO" 5; }
+else
+ LIBURING_CFLAGS=$pkg_cv_LIBURING_CFLAGS
+ LIBURING_LIBS=$pkg_cv_LIBURING_LIBS
+ { $as_echo "$as_me:${as_lineno-$LINENO}: result: yes" >&5
+$as_echo "yes" >&6; }
+
+fi
+fi
##
## Header files
diff --git a/.cirrus.tasks.yml b/.cirrus.tasks.yml
index 5849cbb839a..10d277f5659 100644
--- a/.cirrus.tasks.yml
+++ b/.cirrus.tasks.yml
@@ -471,6 +471,7 @@ task:
--enable-cassert --enable-injection-points --enable-debug \
--enable-tap-tests --enable-nls \
--with-segsize-blocks=6 \
+ --with-liburing \
\
${LINUX_CONFIGURE_FEATURES} \
\
diff --git a/src/Makefile.global.in b/src/Makefile.global.in
index 3b620bac5ac..d45f5fe4ed7 100644
--- a/src/Makefile.global.in
+++ b/src/Makefile.global.in
@@ -191,6 +191,7 @@ with_gssapi = @with_gssapi@
with_krb_srvnam = @with_krb_srvnam@
with_ldap = @with_ldap@
with_libcurl = @with_libcurl@
+with_liburing = @with_liburing@
with_libxml = @with_libxml@
with_libxslt = @with_libxslt@
with_llvm = @with_llvm@
@@ -217,6 +218,9 @@ krb_srvtab = @krb_srvtab@
ICU_CFLAGS = @ICU_CFLAGS@
ICU_LIBS = @ICU_LIBS@
+LIBURING_CFLAGS = @LIBURING_CFLAGS@
+LIBURING_LIBS = @LIBURING_LIBS@
+
TCLSH = @TCLSH@
TCL_LIBS = @TCL_LIBS@
TCL_LIB_SPEC = @TCL_LIB_SPEC@
--
2.48.1.76.g4e746b1a31.dirty
[text/x-diff] v2.8-0006-aio-Add-io_method-io_uring.patch (20.2K, ../../3yxd5r23zly5bytvgyktbxtxq2r3gbpi7xd4dugevh3h4w4q6c@lu6oatjjpltz/7-v2.8-0006-aio-Add-io_method-io_uring.patch)
download | inline diff:
From de04dc4c26133a0b08436733e1542990cca81a61 Mon Sep 17 00:00:00 2001
From: Andres Freund <andres@anarazel.de>
Date: Tue, 11 Feb 2025 14:29:37 -0500
Subject: [PATCH v2.8 06/38] aio: Add io_method=io_uring
io_uring, can be considerably faster than io_method=worker. It is, however,
linux specific and requires an additional compile-time dependency (liburing).
Reviewed-by: Jakub Wartak <jakub.wartak@enterprisedb.com>
Discussion: https://postgr.es/m/uvrtrknj4kdytuboidbhwclo4gxhswwcpgadptsjvjqcluzmah%40brqs62irg4dt
Discussion: https://postgr.es/m/20210223100344.llw5an2aklengrmn@alap3.anarazel.de
Discussion: https://postgr.es/m/stj36ea6yyhoxtqkhpieia2z4krnam7qyetc57rfezgk4zgapf@gcnactj4z56m
---
src/include/storage/aio.h | 3 +
src/include/storage/aio_internal.h | 3 +
src/include/storage/lwlock.h | 1 +
src/backend/storage/aio/Makefile | 1 +
src/backend/storage/aio/aio.c | 6 +
src/backend/storage/aio/meson.build | 1 +
src/backend/storage/aio/method_io_uring.c | 440 ++++++++++++++++++
src/backend/storage/lmgr/lwlock.c | 1 +
.../utils/activity/wait_event_names.txt | 2 +
src/backend/utils/misc/postgresql.conf.sample | 3 +-
doc/src/sgml/config.sgml | 6 +
.cirrus.tasks.yml | 3 +
src/tools/pgindent/typedefs.list | 1 +
13 files changed, 470 insertions(+), 1 deletion(-)
create mode 100644 src/backend/storage/aio/method_io_uring.c
diff --git a/src/include/storage/aio.h b/src/include/storage/aio.h
index 769ea33ea96..8bc43c33505 100644
--- a/src/include/storage/aio.h
+++ b/src/include/storage/aio.h
@@ -28,6 +28,9 @@ typedef enum IoMethod
{
IOMETHOD_SYNC = 0,
IOMETHOD_WORKER,
+#ifdef USE_LIBURING
+ IOMETHOD_IO_URING,
+#endif
} IoMethod;
/* We'll default to worker based execution. */
diff --git a/src/include/storage/aio_internal.h b/src/include/storage/aio_internal.h
index 51f63586467..98ae66921f5 100644
--- a/src/include/storage/aio_internal.h
+++ b/src/include/storage/aio_internal.h
@@ -386,6 +386,9 @@ extern PgAioHandle *pgaio_inj_io_get(void);
/* Declarations for the tables of function pointers exposed by each IO method. */
extern PGDLLIMPORT const IoMethodOps pgaio_sync_ops;
extern PGDLLIMPORT const IoMethodOps pgaio_worker_ops;
+#ifdef USE_LIBURING
+extern PGDLLIMPORT const IoMethodOps pgaio_uring_ops;
+#endif
extern PGDLLIMPORT const IoMethodOps *pgaio_method_ops;
extern PGDLLIMPORT PgAioCtl *pgaio_ctl;
diff --git a/src/include/storage/lwlock.h b/src/include/storage/lwlock.h
index ffa03189e2d..4df1d25c045 100644
--- a/src/include/storage/lwlock.h
+++ b/src/include/storage/lwlock.h
@@ -218,6 +218,7 @@ typedef enum BuiltinTrancheIds
LWTRANCHE_SUBTRANS_SLRU,
LWTRANCHE_XACT_SLRU,
LWTRANCHE_PARALLEL_VACUUM_DSA,
+ LWTRANCHE_AIO_URING_COMPLETION,
LWTRANCHE_FIRST_USER_DEFINED,
} BuiltinTrancheIds;
diff --git a/src/backend/storage/aio/Makefile b/src/backend/storage/aio/Makefile
index f51c34a37f8..c06c50771e0 100644
--- a/src/backend/storage/aio/Makefile
+++ b/src/backend/storage/aio/Makefile
@@ -14,6 +14,7 @@ OBJS = \
aio_init.o \
aio_io.o \
aio_target.o \
+ method_io_uring.o \
method_sync.o \
method_worker.o \
read_stream.o
diff --git a/src/backend/storage/aio/aio.c b/src/backend/storage/aio/aio.c
index f0bb2939d08..58d384b8403 100644
--- a/src/backend/storage/aio/aio.c
+++ b/src/backend/storage/aio/aio.c
@@ -65,6 +65,9 @@ static void pgaio_io_wait(PgAioHandle *ioh, uint64 ref_generation);
const struct config_enum_entry io_method_options[] = {
{"sync", IOMETHOD_SYNC, false},
{"worker", IOMETHOD_WORKER, false},
+#ifdef USE_LIBURING
+ {"io_uring", IOMETHOD_IO_URING, false},
+#endif
{NULL, 0, false}
};
@@ -82,6 +85,9 @@ PgAioBackend *pgaio_my_backend;
static const IoMethodOps *const pgaio_method_ops_table[] = {
[IOMETHOD_SYNC] = &pgaio_sync_ops,
[IOMETHOD_WORKER] = &pgaio_worker_ops,
+#ifdef USE_LIBURING
+ [IOMETHOD_IO_URING] = &pgaio_uring_ops,
+#endif
};
/* callbacks for the configured io_method, set by assign_io_method */
diff --git a/src/backend/storage/aio/meson.build b/src/backend/storage/aio/meson.build
index 74f94c6e40b..2f0f03d8071 100644
--- a/src/backend/storage/aio/meson.build
+++ b/src/backend/storage/aio/meson.build
@@ -6,6 +6,7 @@ backend_sources += files(
'aio_init.c',
'aio_io.c',
'aio_target.c',
+ 'method_io_uring.c',
'method_sync.c',
'method_worker.c',
'read_stream.c',
diff --git a/src/backend/storage/aio/method_io_uring.c b/src/backend/storage/aio/method_io_uring.c
new file mode 100644
index 00000000000..588177eaa4f
--- /dev/null
+++ b/src/backend/storage/aio/method_io_uring.c
@@ -0,0 +1,440 @@
+/*-------------------------------------------------------------------------
+ *
+ * method_io_uring.c
+ * AIO - perform AIO using Linux' io_uring
+ *
+ * For now we create one io_uring instance for each backend. These io_uring
+ * instances have to be created in postmaster, during startup, to allow other
+ * backends to process IO completions, if the issuing backend is currently
+ * busy doing other things. Other backends may not use another backend's
+ * io_uring instance to submit IO, that'd require additional locking that
+ * would likely be harmful for performance.
+ *
+ * We likely will want to introduce a backend-local io_uring instance in the
+ * future, e.g. for FE/BE network IO.
+ *
+ * Portions Copyright (c) 1996-2025, PostgreSQL Global Development Group
+ * Portions Copyright (c) 1994, Regents of the University of California
+ *
+ * IDENTIFICATION
+ * src/backend/storage/aio/method_io_uring.c
+ *
+ *-------------------------------------------------------------------------
+ */
+
+#include "postgres.h"
+
+#ifdef USE_LIBURING
+
+#include <liburing.h>
+
+#include "pgstat.h"
+#include "port/pg_iovec.h"
+#include "storage/aio.h"
+#include "storage/aio_internal.h"
+#include "storage/fd.h"
+#include "storage/proc.h"
+#include "storage/shmem.h"
+
+
+/* number of completions processed at once */
+#define PGAIO_MAX_LOCAL_COMPLETED_IO 32
+
+
+/* Entry points for IoMethodOps. */
+static size_t pgaio_uring_shmem_size(void);
+static void pgaio_uring_shmem_init(bool first_time);
+static void pgaio_uring_init_backend(void);
+static int pgaio_uring_submit(uint16 num_staged_ios, PgAioHandle **staged_ios);
+static void pgaio_uring_wait_one(PgAioHandle *ioh, uint64 ref_generation);
+
+/* helper functions */
+static void pgaio_uring_sq_from_io(PgAioHandle *ioh, struct io_uring_sqe *sqe);
+
+
+const IoMethodOps pgaio_uring_ops = {
+ .shmem_size = pgaio_uring_shmem_size,
+ .shmem_init = pgaio_uring_shmem_init,
+ .init_backend = pgaio_uring_init_backend,
+
+ .submit = pgaio_uring_submit,
+ .wait_one = pgaio_uring_wait_one,
+};
+
+/*
+ * Per-backend state when using io_method=io_uring
+ *
+ * Align the whole struct to a cacheline boundary, to prevent false sharing
+ * between completion_lock and prior backend's io_uring_ring.
+ */
+typedef struct pg_attribute_aligned (PG_CACHE_LINE_SIZE)
+PgAioUringContext
+{
+ /*
+ * Multiple backends can process completions for this backend's io_uring
+ * instance (e.g. when the backend issuing IO is busy doing something
+ * else). To make that safe we have to ensure that only a single backend
+ * gets io completions from the io_uring instance at a time.
+ */
+ LWLock completion_lock;
+
+ struct io_uring io_uring_ring;
+} PgAioUringContext;
+
+/* PgAioUringContexts for all backends */
+static PgAioUringContext *pgaio_uring_contexts;
+
+/* the current backend's context */
+static PgAioUringContext *pgaio_my_uring_context;
+
+
+static uint32
+pgaio_uring_procs(void)
+{
+ /*
+ * We can subtract MAX_IO_WORKERS here as io workers are never used at the
+ * same time as io_method=io_uring.
+ */
+ return MaxBackends + NUM_AUXILIARY_PROCS - MAX_IO_WORKERS;
+}
+
+static Size
+pgaio_uring_context_shmem_size(void)
+{
+ return mul_size(pgaio_uring_procs(), sizeof(PgAioUringContext));
+}
+
+static size_t
+pgaio_uring_shmem_size(void)
+{
+ return pgaio_uring_context_shmem_size();
+}
+
+static void
+pgaio_uring_shmem_init(bool first_time)
+{
+ uint32 TotalProcs = MaxBackends + NUM_AUXILIARY_PROCS - MAX_IO_WORKERS;
+ bool found;
+
+ pgaio_uring_contexts = (PgAioUringContext *)
+ ShmemInitStruct("AioUring", pgaio_uring_shmem_size(), &found);
+
+ if (found)
+ return;
+
+ for (int contextno = 0; contextno < TotalProcs; contextno++)
+ {
+ PgAioUringContext *context = &pgaio_uring_contexts[contextno];
+ int ret;
+
+ /*
+ * FIXME: Right now a high TotalProcs will cause problems in two ways:
+ *
+ * - RLIMIT_NOFILE needs to be big enough to allow MaxBackends fds
+ * created in postmaster, with spare space for max_files_per_process
+ * additional FDs
+ *
+ * - set_max_safe_fds() subtracts the number of already used FDs from
+ * max_files_per_process, ending up with a low limit or even erroring
+ * out.
+ *
+ * XXX: Newer versions of io_uring support sharing the workers that
+ * execute some asynchronous IOs between io_uring instances. It might
+ * be worth using that - also need to evaluate if that causes
+ * noticeable additional contention?
+ */
+ ret = io_uring_queue_init(io_max_concurrency, &context->io_uring_ring, 0);
+ if (ret < 0)
+ elog(ERROR, "io_uring_queue_init failed: %s", strerror(-ret));
+
+ LWLockInitialize(&context->completion_lock, LWTRANCHE_AIO_URING_COMPLETION);
+ }
+}
+
+static void
+pgaio_uring_init_backend(void)
+{
+ Assert(MyProcNumber < pgaio_uring_procs());
+
+ pgaio_my_uring_context = &pgaio_uring_contexts[MyProcNumber];
+}
+
+static int
+pgaio_uring_submit(uint16 num_staged_ios, PgAioHandle **staged_ios)
+{
+ struct io_uring *uring_instance = &pgaio_my_uring_context->io_uring_ring;
+ int in_flight_before = dclist_count(&pgaio_my_backend->in_flight_ios);
+
+ Assert(num_staged_ios <= PGAIO_SUBMIT_BATCH_SIZE);
+
+ for (int i = 0; i < num_staged_ios; i++)
+ {
+ PgAioHandle *ioh = staged_ios[i];
+ struct io_uring_sqe *sqe;
+
+ sqe = io_uring_get_sqe(uring_instance);
+
+ if (!sqe)
+ elog(ERROR, "io_uring submission queue is unexpectedly full");
+
+ pgaio_io_prepare_submit(ioh);
+ pgaio_uring_sq_from_io(ioh, sqe);
+
+ /*
+ * io_uring executes IO in process context if possible. That's
+ * generally good, as it reduces context switching. When performing a
+ * lot of buffered IO that means that copying between page cache and
+ * userspace memory happens in the foreground, as it can't be
+ * offloaded to DMA hardware as is possible when using direct IO. When
+ * executing a lot of buffered IO this causes io_uring to be slower
+ * than worker mode, as worker mode parallelizes the copying. io_uring
+ * can be told to offload work to worker threads instead.
+ *
+ * If an IO is buffered IO and we already have IOs in flight or
+ * multiple IOs are being submitted, we thus tell io_uring to execute
+ * the IO in the background. We don't do so for the first few IOs
+ * being submitted as executing in this process' context has lower
+ * latency.
+ */
+ if (in_flight_before > 4 && (ioh->flags & PGAIO_HF_BUFFERED))
+ io_uring_sqe_set_flags(sqe, IOSQE_ASYNC);
+
+ in_flight_before++;
+ }
+
+ while (true)
+ {
+ int ret;
+
+ pgstat_report_wait_start(WAIT_EVENT_AIO_IO_URING_SUBMIT);
+ ret = io_uring_submit(uring_instance);
+ pgstat_report_wait_end();
+
+ if (ret == -EINTR)
+ {
+ pgaio_debug(DEBUG3,
+ "aio method uring: submit EINTR, nios: %d",
+ num_staged_ios);
+ }
+ else if (ret < 0)
+ {
+ /*
+ * The io_uring_enter() manpage suggests that the appropriate
+ * reaction to EAGAIN is:
+ *
+ * "The application should wait for some completions and try
+ * again"
+ *
+ * However, it seems unlikely that that would help in our case, as
+ * we apply a low limit to the number of outstanding IOs and thus
+ * also outstanding completions, making it unlikely that we'd get
+ * EAGAIN while the OS is in good working order.
+ *
+ * Additionally, it would be problematic to just wait here, our
+ * caller might hold critical locks. It'd possibly lead to
+ * delaying the crash-restart that seems likely to occur when the
+ * kernel is under such heavy memory pressure.
+ *
+ * Update errno to to allow %m to work.
+ */
+ errno = -ret;
+ elog(PANIC, "io_uring submit failed: %m");
+ }
+ else if (ret != num_staged_ios)
+ {
+ /* likely unreachable, but if it is, we would need to re-submit */
+ elog(PANIC, "io_uring submit submitted only %d of %d",
+ ret, num_staged_ios);
+ }
+ else
+ {
+ pgaio_debug(DEBUG4,
+ "aio method uring: submitted %d IOs",
+ num_staged_ios);
+ break;
+ }
+ }
+
+ return num_staged_ios;
+}
+
+static void
+pgaio_uring_drain_locked(PgAioUringContext *context)
+{
+ int ready;
+ int orig_ready;
+
+ Assert(LWLockHeldByMeInMode(&context->completion_lock, LW_EXCLUSIVE));
+
+ /*
+ * Don't drain more events than available right now. Otherwise it's
+ * plausible that one backend could get stuck, for a while, receiving CQEs
+ * without actually processing them.
+ */
+ orig_ready = ready = io_uring_cq_ready(&context->io_uring_ring);
+
+ while (ready > 0)
+ {
+ struct io_uring_cqe *cqes[PGAIO_MAX_LOCAL_COMPLETED_IO];
+ uint32 ncqes;
+
+ START_CRIT_SECTION();
+ ncqes =
+ io_uring_peek_batch_cqe(&context->io_uring_ring,
+ cqes,
+ Min(PGAIO_MAX_LOCAL_COMPLETED_IO, ready));
+ Assert(ncqes <= ready);
+
+ ready -= ncqes;
+
+ for (int i = 0; i < ncqes; i++)
+ {
+ struct io_uring_cqe *cqe = cqes[i];
+ PgAioHandle *ioh;
+
+ ioh = io_uring_cqe_get_data(cqe);
+ io_uring_cqe_seen(&context->io_uring_ring, cqe);
+
+ pgaio_io_process_completion(ioh, cqe->res);
+ }
+
+ END_CRIT_SECTION();
+
+ pgaio_debug(DEBUG3,
+ "drained %d/%d, now expecting %d",
+ ncqes, orig_ready, io_uring_cq_ready(&context->io_uring_ring));
+ }
+}
+
+static void
+pgaio_uring_wait_one(PgAioHandle *ioh, uint64 ref_generation)
+{
+ PgAioHandleState state;
+ ProcNumber owner_procno = ioh->owner_procno;
+ PgAioUringContext *owner_context = &pgaio_uring_contexts[owner_procno];
+ bool expect_cqe;
+ int waited = 0;
+
+ /*
+ * XXX: It would be nice to have a smarter locking scheme, nearly all the
+ * time the backend owning the ring will consume the completions, making
+ * the locking unnecessarily expensive.
+ */
+ LWLockAcquire(&owner_context->completion_lock, LW_EXCLUSIVE);
+
+ while (true)
+ {
+ pgaio_debug_io(DEBUG3, ioh,
+ "wait_one io_gen: %llu, ref_gen: %llu, cycle %d",
+ (long long unsigned) ref_generation,
+ (long long unsigned) ioh->generation,
+ waited);
+
+ if (pgaio_io_was_recycled(ioh, ref_generation, &state) ||
+ state != PGAIO_HS_SUBMITTED)
+ {
+ /* the IO was completed by another backend */
+ break;
+ }
+ else if (io_uring_cq_ready(&owner_context->io_uring_ring))
+ {
+ /* no need to wait in the kernel, io_uring has a completion */
+ expect_cqe = true;
+ }
+ else
+ {
+ int ret;
+ struct io_uring_cqe *cqes;
+
+ /* need to wait in the kernel */
+ pgstat_report_wait_start(WAIT_EVENT_AIO_IO_URING_COMPLETION);
+ ret = io_uring_wait_cqes(&owner_context->io_uring_ring, &cqes, 1, NULL, NULL);
+ pgstat_report_wait_end();
+
+ if (ret == -EINTR)
+ {
+ continue;
+ }
+ else if (ret != 0)
+ {
+ /* see comment after io_uring_submit() */
+ errno = -ret;
+ elog(PANIC, "io_uring wait failed: %m");
+ }
+ else
+ {
+ Assert(cqes != NULL);
+ expect_cqe = true;
+ waited++;
+ }
+ }
+
+ if (expect_cqe)
+ {
+ pgaio_uring_drain_locked(owner_context);
+ }
+ }
+
+ LWLockRelease(&owner_context->completion_lock);
+
+ pgaio_debug(DEBUG3,
+ "wait_one with %d sleeps",
+ waited);
+}
+
+static void
+pgaio_uring_sq_from_io(PgAioHandle *ioh, struct io_uring_sqe *sqe)
+{
+ struct iovec *iov;
+
+ switch (ioh->op)
+ {
+ case PGAIO_OP_READV:
+ iov = &pgaio_ctl->iovecs[ioh->iovec_off];
+ if (ioh->op_data.read.iov_length == 1)
+ {
+ io_uring_prep_read(sqe,
+ ioh->op_data.read.fd,
+ iov->iov_base,
+ iov->iov_len,
+ ioh->op_data.read.offset);
+ }
+ else
+ {
+ io_uring_prep_readv(sqe,
+ ioh->op_data.read.fd,
+ iov,
+ ioh->op_data.read.iov_length,
+ ioh->op_data.read.offset);
+
+ }
+ break;
+
+ case PGAIO_OP_WRITEV:
+ iov = &pgaio_ctl->iovecs[ioh->iovec_off];
+ if (ioh->op_data.write.iov_length == 1)
+ {
+ io_uring_prep_write(sqe,
+ ioh->op_data.write.fd,
+ iov->iov_base,
+ iov->iov_len,
+ ioh->op_data.write.offset);
+ }
+ else
+ {
+ io_uring_prep_writev(sqe,
+ ioh->op_data.write.fd,
+ iov,
+ ioh->op_data.write.iov_length,
+ ioh->op_data.write.offset);
+ }
+ break;
+
+ case PGAIO_OP_INVALID:
+ elog(ERROR, "trying to prepare invalid IO operation for execution");
+ }
+
+ io_uring_sqe_set_data(sqe, ioh);
+}
+
+#endif /* USE_LIBURING */
diff --git a/src/backend/storage/lmgr/lwlock.c b/src/backend/storage/lmgr/lwlock.c
index 5702c35bb91..3df29658f18 100644
--- a/src/backend/storage/lmgr/lwlock.c
+++ b/src/backend/storage/lmgr/lwlock.c
@@ -177,6 +177,7 @@ static const char *const BuiltinTrancheNames[] = {
[LWTRANCHE_SUBTRANS_SLRU] = "SubtransSLRU",
[LWTRANCHE_XACT_SLRU] = "XactSLRU",
[LWTRANCHE_PARALLEL_VACUUM_DSA] = "ParallelVacuumDSA",
+ [LWTRANCHE_AIO_URING_COMPLETION] = "AioUringCompletion",
};
StaticAssertDecl(lengthof(BuiltinTrancheNames) ==
diff --git a/src/backend/utils/activity/wait_event_names.txt b/src/backend/utils/activity/wait_event_names.txt
index 9fa12a555e8..44eef41f841 100644
--- a/src/backend/utils/activity/wait_event_names.txt
+++ b/src/backend/utils/activity/wait_event_names.txt
@@ -192,6 +192,8 @@ ABI_compatibility:
Section: ClassName - WaitEventIO
+AIO_IO_URING_SUBMIT "Waiting for IO submission via io_uring."
+AIO_IO_URING_COMPLETION "Waiting for IO completion via io_uring."
AIO_IO_COMPLETION "Waiting for IO completion."
BASEBACKUP_READ "Waiting for base backup to read from a file."
BASEBACKUP_SYNC "Waiting for data written by a base backup to reach durable storage."
diff --git a/src/backend/utils/misc/postgresql.conf.sample b/src/backend/utils/misc/postgresql.conf.sample
index 75a844a9474..7e8c3dcb175 100644
--- a/src/backend/utils/misc/postgresql.conf.sample
+++ b/src/backend/utils/misc/postgresql.conf.sample
@@ -202,7 +202,8 @@
#maintenance_io_concurrency = 10 # 1-1000; 0 disables prefetching
#io_combine_limit = 128kB # usually 1-32 blocks (depends on OS)
-#io_method = worker # worker, sync (change requires restart)
+#io_method = worker # worker, io_uring, sync
+ # (change requires restart)
#io_max_concurrency = -1 # Max number of IOs that one process
# can execute simultaneously
# -1 sets based on shared_buffers
diff --git a/doc/src/sgml/config.sgml b/doc/src/sgml/config.sgml
index 108caea1fed..8d53478f53a 100644
--- a/doc/src/sgml/config.sgml
+++ b/doc/src/sgml/config.sgml
@@ -2681,6 +2681,12 @@ include_dir 'conf.d'
<literal>worker</literal> (execute asynchronous I/O using worker processes)
</para>
</listitem>
+ <listitem>
+ <para>
+ <literal>io_uring</literal> (execute asynchronous I/O using
+ io_uring, if available)
+ </para>
+ </listitem>
<listitem>
<para>
<literal>sync</literal> (execute asynchronous-eligible I/O synchronously)
diff --git a/.cirrus.tasks.yml b/.cirrus.tasks.yml
index 10d277f5659..86a1fa9bbdb 100644
--- a/.cirrus.tasks.yml
+++ b/.cirrus.tasks.yml
@@ -493,11 +493,14 @@ task:
# - Uses undefined behaviour and alignment sanitizers, sanitizer failures
# are typically printed in the server log
# - Test both 64bit and 32 bit builds
+ # - uses io_method=io_uring
- name: Linux - Debian Bookworm - Meson
env:
CCACHE_MAXSIZE: "400M" # tests two different builds
SANITIZER_FLAGS: -fsanitize=alignment,undefined
+ PG_TEST_INITDB_EXTRA_OPTS: >-
+ -c io_method=io_uring
configure_script: |
su postgres <<-EOF
diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list
index 4d8b5c7f1d6..1748befca16 100644
--- a/src/tools/pgindent/typedefs.list
+++ b/src/tools/pgindent/typedefs.list
@@ -2150,6 +2150,7 @@ PgAioReturn
PgAioTargetData
PgAioTargetID
PgAioTargetInfo
+PgAioUringContext
PgAioWaitRef
PgArchData
PgBackendGSSStatus
--
2.48.1.76.g4e746b1a31.dirty
[text/x-diff] v2.8-0007-smgr-Hold-interrupts-in-most-smgr-functions.patch (9.4K, ../../3yxd5r23zly5bytvgyktbxtxq2r3gbpi7xd4dugevh3h4w4q6c@lu6oatjjpltz/8-v2.8-0007-smgr-Hold-interrupts-in-most-smgr-functions.patch)
download | inline diff:
From 01e1021461650fd66cc2bcada5da73d2d8441f98 Mon Sep 17 00:00:00 2001
From: Andres Freund <andres@anarazel.de>
Date: Thu, 13 Mar 2025 14:48:36 -0400
Subject: [PATCH v2.8 07/38] smgr: Hold interrupts in most smgr functions
We need to hold interrupts across most of the smgr.c/md.c functions, as
otherwise interrupt processing, e.g. due to a debug elog/ereport, can trigger
procsignal processing, which in turn can trigger smgrreleaseall(). As the
relevant code is not reentrant we quickly end up in a bad situation.
It seems better to put the HOLD_INTERRUPTS()/RESUME_INTERRUPTS() in smgr.c,
instead of trying to push them down to md.c where possible: For one, every
smgr implementation would be vulnerable, for another, a good bit of smgr.c
code itself is affected too.
Discussion: https://postgr.es/m/3vae7l5ozvqtxmd7rr7zaeq3qkuipz365u3rtim5t5wdkr6f4g@vkgf2fogjirl
---
src/backend/storage/smgr/smgr.c | 94 +++++++++++++++++++++++++++++++--
1 file changed, 91 insertions(+), 3 deletions(-)
diff --git a/src/backend/storage/smgr/smgr.c b/src/backend/storage/smgr/smgr.c
index ebe35c04de5..8787ce9b18f 100644
--- a/src/backend/storage/smgr/smgr.c
+++ b/src/backend/storage/smgr/smgr.c
@@ -40,6 +40,15 @@
* themselves, as there could pointers to them in active use. See
* smgrrelease() and smgrreleaseall().
*
+ * NB: We need to hold interrupts across most of the functions in this file,
+ * as otherwise interrupt processing, e.g. due to a debug elog/ereport, can
+ * trigger procsignal processing, which in turn can trigger
+ * smgrreleaseall(). None of the relevant code is reentrant. It seems better
+ * to put the HOLD_INTERRUPTS()/RESUME_INTERRUPTS() here, instead of trying to
+ * push them down to md.c where possible: For one, every smgr implementation
+ * would be vulnerable, for another, a good bit of smgr.c code itself is
+ * affected too.
+ *
* Portions Copyright (c) 1996-2025, PostgreSQL Global Development Group
* Portions Copyright (c) 1994, Regents of the University of California
*
@@ -53,6 +62,7 @@
#include "access/xlogutils.h"
#include "lib/ilist.h"
+#include "miscadmin.h"
#include "storage/bufmgr.h"
#include "storage/ipc.h"
#include "storage/md.h"
@@ -158,12 +168,16 @@ smgrinit(void)
{
int i;
+ HOLD_INTERRUPTS();
+
for (i = 0; i < NSmgr; i++)
{
if (smgrsw[i].smgr_init)
smgrsw[i].smgr_init();
}
+ RESUME_INTERRUPTS();
+
/* register the shutdown proc */
on_proc_exit(smgrshutdown, 0);
}
@@ -176,11 +190,13 @@ smgrshutdown(int code, Datum arg)
{
int i;
+ HOLD_INTERRUPTS();
for (i = 0; i < NSmgr; i++)
{
if (smgrsw[i].smgr_shutdown)
smgrsw[i].smgr_shutdown();
}
+ RESUME_INTERRUPTS();
}
/*
@@ -206,6 +222,8 @@ smgropen(RelFileLocator rlocator, ProcNumber backend)
Assert(RelFileNumberIsValid(rlocator.relNumber));
+ HOLD_INTERRUPTS();
+
if (SMgrRelationHash == NULL)
{
/* First time through: initialize the hash table */
@@ -242,6 +260,8 @@ smgropen(RelFileLocator rlocator, ProcNumber backend)
dlist_push_tail(&unpinned_relns, &reln->node);
}
+ RESUME_INTERRUPTS();
+
return reln;
}
@@ -283,6 +303,8 @@ smgrdestroy(SMgrRelation reln)
Assert(reln->pincount == 0);
+ HOLD_INTERRUPTS();
+
for (forknum = 0; forknum <= MAX_FORKNUM; forknum++)
smgrsw[reln->smgr_which].smgr_close(reln, forknum);
@@ -292,6 +314,8 @@ smgrdestroy(SMgrRelation reln)
&(reln->smgr_rlocator),
HASH_REMOVE, NULL) == NULL)
elog(ERROR, "SMgrRelation hashtable corrupted");
+
+ RESUME_INTERRUPTS();
}
/*
@@ -302,12 +326,16 @@ smgrdestroy(SMgrRelation reln)
void
smgrrelease(SMgrRelation reln)
{
+ HOLD_INTERRUPTS();
+
for (ForkNumber forknum = 0; forknum <= MAX_FORKNUM; forknum++)
{
smgrsw[reln->smgr_which].smgr_close(reln, forknum);
reln->smgr_cached_nblocks[forknum] = InvalidBlockNumber;
}
reln->smgr_targblock = InvalidBlockNumber;
+
+ RESUME_INTERRUPTS();
}
/*
@@ -336,6 +364,8 @@ smgrdestroyall(void)
{
dlist_mutable_iter iter;
+ HOLD_INTERRUPTS();
+
/*
* Zap all unpinned SMgrRelations. We rely on smgrdestroy() to remove
* each one from the list.
@@ -347,6 +377,8 @@ smgrdestroyall(void)
smgrdestroy(rel);
}
+
+ RESUME_INTERRUPTS();
}
/*
@@ -362,12 +394,16 @@ smgrreleaseall(void)
if (SMgrRelationHash == NULL)
return;
+ HOLD_INTERRUPTS();
+
hash_seq_init(&status, SMgrRelationHash);
while ((reln = (SMgrRelation) hash_seq_search(&status)) != NULL)
{
smgrrelease(reln);
}
+
+ RESUME_INTERRUPTS();
}
/*
@@ -400,7 +436,13 @@ smgrreleaserellocator(RelFileLocatorBackend rlocator)
bool
smgrexists(SMgrRelation reln, ForkNumber forknum)
{
- return smgrsw[reln->smgr_which].smgr_exists(reln, forknum);
+ bool ret;
+
+ HOLD_INTERRUPTS();
+ ret = smgrsw[reln->smgr_which].smgr_exists(reln, forknum);
+ RESUME_INTERRUPTS();
+
+ return ret;
}
/*
@@ -413,7 +455,9 @@ smgrexists(SMgrRelation reln, ForkNumber forknum)
void
smgrcreate(SMgrRelation reln, ForkNumber forknum, bool isRedo)
{
+ HOLD_INTERRUPTS();
smgrsw[reln->smgr_which].smgr_create(reln, forknum, isRedo);
+ RESUME_INTERRUPTS();
}
/*
@@ -434,6 +478,8 @@ smgrdosyncall(SMgrRelation *rels, int nrels)
if (nrels == 0)
return;
+ HOLD_INTERRUPTS();
+
FlushRelationsAllBuffers(rels, nrels);
/*
@@ -449,6 +495,8 @@ smgrdosyncall(SMgrRelation *rels, int nrels)
smgrsw[which].smgr_immedsync(rels[i], forknum);
}
}
+
+ RESUME_INTERRUPTS();
}
/*
@@ -471,6 +519,8 @@ smgrdounlinkall(SMgrRelation *rels, int nrels, bool isRedo)
if (nrels == 0)
return;
+ HOLD_INTERRUPTS();
+
/*
* Get rid of any remaining buffers for the relations. bufmgr will just
* drop them without bothering to write the contents.
@@ -522,6 +572,8 @@ smgrdounlinkall(SMgrRelation *rels, int nrels, bool isRedo)
}
pfree(rlocators);
+
+ RESUME_INTERRUPTS();
}
@@ -538,6 +590,8 @@ void
smgrextend(SMgrRelation reln, ForkNumber forknum, BlockNumber blocknum,
const void *buffer, bool skipFsync)
{
+ HOLD_INTERRUPTS();
+
smgrsw[reln->smgr_which].smgr_extend(reln, forknum, blocknum,
buffer, skipFsync);
@@ -550,6 +604,8 @@ smgrextend(SMgrRelation reln, ForkNumber forknum, BlockNumber blocknum,
reln->smgr_cached_nblocks[forknum] = blocknum + 1;
else
reln->smgr_cached_nblocks[forknum] = InvalidBlockNumber;
+
+ RESUME_INTERRUPTS();
}
/*
@@ -563,6 +619,8 @@ void
smgrzeroextend(SMgrRelation reln, ForkNumber forknum, BlockNumber blocknum,
int nblocks, bool skipFsync)
{
+ HOLD_INTERRUPTS();
+
smgrsw[reln->smgr_which].smgr_zeroextend(reln, forknum, blocknum,
nblocks, skipFsync);
@@ -575,6 +633,8 @@ smgrzeroextend(SMgrRelation reln, ForkNumber forknum, BlockNumber blocknum,
reln->smgr_cached_nblocks[forknum] = blocknum + nblocks;
else
reln->smgr_cached_nblocks[forknum] = InvalidBlockNumber;
+
+ RESUME_INTERRUPTS();
}
/*
@@ -588,7 +648,13 @@ bool
smgrprefetch(SMgrRelation reln, ForkNumber forknum, BlockNumber blocknum,
int nblocks)
{
- return smgrsw[reln->smgr_which].smgr_prefetch(reln, forknum, blocknum, nblocks);
+ bool ret;
+
+ HOLD_INTERRUPTS();
+ ret = smgrsw[reln->smgr_which].smgr_prefetch(reln, forknum, blocknum, nblocks);
+ RESUME_INTERRUPTS();
+
+ return ret;
}
/*
@@ -601,7 +667,13 @@ uint32
smgrmaxcombine(SMgrRelation reln, ForkNumber forknum,
BlockNumber blocknum)
{
- return smgrsw[reln->smgr_which].smgr_maxcombine(reln, forknum, blocknum);
+ uint32 ret;
+
+ HOLD_INTERRUPTS();
+ ret = smgrsw[reln->smgr_which].smgr_maxcombine(reln, forknum, blocknum);
+ RESUME_INTERRUPTS();
+
+ return ret;
}
/*
@@ -619,8 +691,10 @@ void
smgrreadv(SMgrRelation reln, ForkNumber forknum, BlockNumber blocknum,
void **buffers, BlockNumber nblocks)
{
+ HOLD_INTERRUPTS();
smgrsw[reln->smgr_which].smgr_readv(reln, forknum, blocknum, buffers,
nblocks);
+ RESUME_INTERRUPTS();
}
/*
@@ -653,8 +727,10 @@ void
smgrwritev(SMgrRelation reln, ForkNumber forknum, BlockNumber blocknum,
const void **buffers, BlockNumber nblocks, bool skipFsync)
{
+ HOLD_INTERRUPTS();
smgrsw[reln->smgr_which].smgr_writev(reln, forknum, blocknum,
buffers, nblocks, skipFsync);
+ RESUME_INTERRUPTS();
}
/*
@@ -665,8 +741,10 @@ void
smgrwriteback(SMgrRelation reln, ForkNumber forknum, BlockNumber blocknum,
BlockNumber nblocks)
{
+ HOLD_INTERRUPTS();
smgrsw[reln->smgr_which].smgr_writeback(reln, forknum, blocknum,
nblocks);
+ RESUME_INTERRUPTS();
}
/*
@@ -683,10 +761,14 @@ smgrnblocks(SMgrRelation reln, ForkNumber forknum)
if (result != InvalidBlockNumber)
return result;
+ HOLD_INTERRUPTS();
+
result = smgrsw[reln->smgr_which].smgr_nblocks(reln, forknum);
reln->smgr_cached_nblocks[forknum] = result;
+ RESUME_INTERRUPTS();
+
return result;
}
@@ -731,6 +813,8 @@ smgrtruncate(SMgrRelation reln, ForkNumber *forknum, int nforks,
{
int i;
+ Assert(!INTERRUPTS_CAN_BE_PROCESSED());
+
/*
* Get rid of any buffers for the about-to-be-deleted blocks. bufmgr will
* just drop them without bothering to write the contents.
@@ -784,7 +868,9 @@ smgrtruncate(SMgrRelation reln, ForkNumber *forknum, int nforks,
void
smgrregistersync(SMgrRelation reln, ForkNumber forknum)
{
+ HOLD_INTERRUPTS();
smgrsw[reln->smgr_which].smgr_registersync(reln, forknum);
+ RESUME_INTERRUPTS();
}
/*
@@ -816,7 +902,9 @@ smgrregistersync(SMgrRelation reln, ForkNumber forknum)
void
smgrimmedsync(SMgrRelation reln, ForkNumber forknum)
{
+ HOLD_INTERRUPTS();
smgrsw[reln->smgr_which].smgr_immedsync(reln, forknum);
+ RESUME_INTERRUPTS();
}
/*
--
2.48.1.76.g4e746b1a31.dirty
[text/x-diff] v2.8-0008-aio-Implement-support-for-reads-in-smgr-md-fd.patch (21.9K, ../../3yxd5r23zly5bytvgyktbxtxq2r3gbpi7xd4dugevh3h4w4q6c@lu6oatjjpltz/9-v2.8-0008-aio-Implement-support-for-reads-in-smgr-md-fd.patch)
download | inline diff:
From a844a39186f5b29471da96feaf7f570c7a22afa3 Mon Sep 17 00:00:00 2001
From: Andres Freund <andres@anarazel.de>
Date: Thu, 13 Mar 2025 14:58:29 -0400
Subject: [PATCH v2.8 08/38] aio: Implement support for reads in smgr/md/fd
This implements the following:
1) An smgr AIO target, for AIO on smgr files. This should be usable not just
for md.c but also other SMGR implementation if we ever get them.
2) readv support in fd.c, which requires a small bit of infrastructure work in
fd.c
3) smgr.c and md.c support for readv
There still is nothing performing AIO, but as of this commit it would be
possible.
As part of this change FileGetRawDesc() actually ensures that the file is
opened - previously it was basically not usable. It's used to reopen a file in
IO workers.
Discussion: https://postgr.es/m/uvrtrknj4kdytuboidbhwclo4gxhswwcpgadptsjvjqcluzmah%40brqs62irg4dt
Discussion: https://postgr.es/m/20210223100344.llw5an2aklengrmn@alap3.anarazel.de
Discussion: https://postgr.es/m/stj36ea6yyhoxtqkhpieia2z4krnam7qyetc57rfezgk4zgapf@gcnactj4z56m
---
src/include/storage/aio.h | 6 +-
src/include/storage/aio_types.h | 10 +-
src/include/storage/fd.h | 3 +
src/include/storage/md.h | 7 +
src/include/storage/smgr.h | 14 ++
src/backend/storage/aio/aio_callback.c | 5 +-
src/backend/storage/aio/aio_target.c | 2 +
src/backend/storage/file/fd.c | 40 ++++++
src/backend/storage/smgr/md.c | 176 +++++++++++++++++++++++++
src/backend/storage/smgr/smgr.c | 132 +++++++++++++++++++
10 files changed, 390 insertions(+), 5 deletions(-)
diff --git a/src/include/storage/aio.h b/src/include/storage/aio.h
index 8bc43c33505..d02c3d27fae 100644
--- a/src/include/storage/aio.h
+++ b/src/include/storage/aio.h
@@ -112,9 +112,10 @@ typedef enum PgAioTargetID
{
/* intentionally the zero value, to help catch zeroed memory etc */
PGAIO_TID_INVALID = 0,
+ PGAIO_TID_SMGR,
} PgAioTargetID;
-#define PGAIO_TID_COUNT (PGAIO_TID_INVALID + 1)
+#define PGAIO_TID_COUNT (PGAIO_TID_SMGR + 1)
/*
@@ -180,6 +181,9 @@ struct PgAioTargetInfo
typedef enum PgAioHandleCallbackID
{
PGAIO_HCB_INVALID,
+
+ PGAIO_HCB_MD_READV,
+ PGAIO_HCB_MD_WRITEV,
} PgAioHandleCallbackID;
diff --git a/src/include/storage/aio_types.h b/src/include/storage/aio_types.h
index a5cc658efbd..b0b9a2a5c97 100644
--- a/src/include/storage/aio_types.h
+++ b/src/include/storage/aio_types.h
@@ -60,11 +60,15 @@ typedef struct PgAioWaitRef
*/
typedef union PgAioTargetData
{
- /* just as an example placeholder for later */
struct
{
- uint32 queue_id;
- } wal;
+ RelFileLocator rlocator; /* physical relation identifier */
+ BlockNumber blockNum; /* blknum relative to begin of reln */
+ BlockNumber nblocks;
+ ForkNumber forkNum:8; /* don't waste 4 byte for four values */
+ bool is_temp:1; /* proc can be inferred by owning AIO */
+ bool skip_fsync:1;
+ } smgr;
} PgAioTargetData;
diff --git a/src/include/storage/fd.h b/src/include/storage/fd.h
index e3067ab6597..b77d8e5e30e 100644
--- a/src/include/storage/fd.h
+++ b/src/include/storage/fd.h
@@ -101,6 +101,8 @@ extern PGDLLIMPORT int max_safe_fds;
* prototypes for functions in fd.c
*/
+struct PgAioHandle;
+
/* Operations on virtual Files --- equivalent to Unix kernel file ops */
extern File PathNameOpenFile(const char *fileName, int fileFlags);
extern File PathNameOpenFilePerm(const char *fileName, int fileFlags, mode_t fileMode);
@@ -109,6 +111,7 @@ extern void FileClose(File file);
extern int FilePrefetch(File file, off_t offset, off_t amount, uint32 wait_event_info);
extern ssize_t FileReadV(File file, const struct iovec *iov, int iovcnt, off_t offset, uint32 wait_event_info);
extern ssize_t FileWriteV(File file, const struct iovec *iov, int iovcnt, off_t offset, uint32 wait_event_info);
+extern int FileStartReadV(struct PgAioHandle *ioh, File file, int iovcnt, off_t offset, uint32 wait_event_info);
extern int FileSync(File file, uint32 wait_event_info);
extern int FileZero(File file, off_t offset, off_t amount, uint32 wait_event_info);
extern int FileFallocate(File file, off_t offset, off_t amount, uint32 wait_event_info);
diff --git a/src/include/storage/md.h b/src/include/storage/md.h
index 05bf537066e..9d7131eff43 100644
--- a/src/include/storage/md.h
+++ b/src/include/storage/md.h
@@ -14,11 +14,14 @@
#ifndef MD_H
#define MD_H
+#include "storage/aio_types.h"
#include "storage/block.h"
#include "storage/relfilelocator.h"
#include "storage/smgr.h"
#include "storage/sync.h"
+extern const PgAioHandleCallbacks aio_md_readv_cb;
+
/* md storage manager functionality */
extern void mdinit(void);
extern void mdopen(SMgrRelation reln);
@@ -36,6 +39,9 @@ extern uint32 mdmaxcombine(SMgrRelation reln, ForkNumber forknum,
BlockNumber blocknum);
extern void mdreadv(SMgrRelation reln, ForkNumber forknum, BlockNumber blocknum,
void **buffers, BlockNumber nblocks);
+extern void mdstartreadv(PgAioHandle *ioh,
+ SMgrRelation reln, ForkNumber forknum, BlockNumber blocknum,
+ void **buffers, BlockNumber nblocks);
extern void mdwritev(SMgrRelation reln, ForkNumber forknum,
BlockNumber blocknum,
const void **buffers, BlockNumber nblocks, bool skipFsync);
@@ -46,6 +52,7 @@ extern void mdtruncate(SMgrRelation reln, ForkNumber forknum,
BlockNumber old_blocks, BlockNumber nblocks);
extern void mdimmedsync(SMgrRelation reln, ForkNumber forknum);
extern void mdregistersync(SMgrRelation reln, ForkNumber forknum);
+extern int mdfd(SMgrRelation reln, ForkNumber forknum, BlockNumber blocknum, uint32 *off);
extern void ForgetDatabaseSyncRequests(Oid dbid);
extern void DropRelationFiles(RelFileLocator *delrels, int ndelrels, bool isRedo);
diff --git a/src/include/storage/smgr.h b/src/include/storage/smgr.h
index 4016b206ad6..856ebcda350 100644
--- a/src/include/storage/smgr.h
+++ b/src/include/storage/smgr.h
@@ -15,6 +15,7 @@
#define SMGR_H
#include "lib/ilist.h"
+#include "storage/aio_types.h"
#include "storage/block.h"
#include "storage/relfilelocator.h"
@@ -73,6 +74,8 @@ typedef SMgrRelationData *SMgrRelation;
#define SmgrIsTemp(smgr) \
RelFileLocatorBackendIsTemp((smgr)->smgr_rlocator)
+extern const PgAioTargetInfo aio_smgr_target_info;
+
extern void smgrinit(void);
extern SMgrRelation smgropen(RelFileLocator rlocator, ProcNumber backend);
extern bool smgrexists(SMgrRelation reln, ForkNumber forknum);
@@ -97,6 +100,10 @@ extern uint32 smgrmaxcombine(SMgrRelation reln, ForkNumber forknum,
extern void smgrreadv(SMgrRelation reln, ForkNumber forknum,
BlockNumber blocknum,
void **buffers, BlockNumber nblocks);
+extern void smgrstartreadv(PgAioHandle *ioh,
+ SMgrRelation reln, ForkNumber forknum,
+ BlockNumber blocknum,
+ void **buffers, BlockNumber nblocks);
extern void smgrwritev(SMgrRelation reln, ForkNumber forknum,
BlockNumber blocknum,
const void **buffers, BlockNumber nblocks,
@@ -127,4 +134,11 @@ smgrwrite(SMgrRelation reln, ForkNumber forknum, BlockNumber blocknum,
smgrwritev(reln, forknum, blocknum, &buffer, 1, skipFsync);
}
+extern void pgaio_io_set_target_smgr(PgAioHandle *ioh,
+ SMgrRelationData *smgr,
+ ForkNumber forknum,
+ BlockNumber blocknum,
+ int nblocks,
+ bool skip_fsync);
+
#endif /* SMGR_H */
diff --git a/src/backend/storage/aio/aio_callback.c b/src/backend/storage/aio/aio_callback.c
index 7392b55322c..6a21c82396a 100644
--- a/src/backend/storage/aio/aio_callback.c
+++ b/src/backend/storage/aio/aio_callback.c
@@ -18,10 +18,11 @@
#include "miscadmin.h"
#include "storage/aio.h"
#include "storage/aio_internal.h"
+#include "storage/md.h"
/* just to have something to put into the aio_handle_cbs */
-static const struct PgAioHandleCallbacks aio_invalid_cb = {0};
+static const PgAioHandleCallbacks aio_invalid_cb = {0};
typedef struct PgAioHandleCallbacksEntry
{
@@ -37,6 +38,8 @@ typedef struct PgAioHandleCallbacksEntry
static const PgAioHandleCallbacksEntry aio_handle_cbs[] = {
#define CALLBACK_ENTRY(id, callback) [id] = {.cb = &callback, .name = #callback}
CALLBACK_ENTRY(PGAIO_HCB_INVALID, aio_invalid_cb),
+
+ CALLBACK_ENTRY(PGAIO_HCB_MD_READV, aio_md_readv_cb),
#undef CALLBACK_ENTRY
};
diff --git a/src/backend/storage/aio/aio_target.c b/src/backend/storage/aio/aio_target.c
index 3fa813fd592..aab86a358da 100644
--- a/src/backend/storage/aio/aio_target.c
+++ b/src/backend/storage/aio/aio_target.c
@@ -16,6 +16,7 @@
#include "storage/aio.h"
#include "storage/aio_internal.h"
+#include "storage/smgr.h"
/*
@@ -29,6 +30,7 @@ static const PgAioTargetInfo *pgaio_target_info[] = {
[PGAIO_TID_INVALID] = &(PgAioTargetInfo) {
.name = "invalid",
},
+ [PGAIO_TID_SMGR] = &aio_smgr_target_info,
};
diff --git a/src/backend/storage/file/fd.c b/src/backend/storage/file/fd.c
index 62f1185859f..34588ed5167 100644
--- a/src/backend/storage/file/fd.c
+++ b/src/backend/storage/file/fd.c
@@ -94,6 +94,7 @@
#include "miscadmin.h"
#include "pgstat.h"
#include "postmaster/startup.h"
+#include "storage/aio.h"
#include "storage/fd.h"
#include "storage/ipc.h"
#include "utils/guc.h"
@@ -1294,6 +1295,8 @@ LruDelete(File file)
vfdP = &VfdCache[file];
+ pgaio_closing_fd(vfdP->fd);
+
/*
* Close the file. We aren't expecting this to fail; if it does, better
* to leak the FD than to mess up our internal state.
@@ -1987,6 +1990,8 @@ FileClose(File file)
if (!FileIsNotOpen(file))
{
+ pgaio_closing_fd(vfdP->fd);
+
/* close the file */
if (close(vfdP->fd) != 0)
{
@@ -2210,6 +2215,32 @@ retry:
return returnCode;
}
+int
+FileStartReadV(PgAioHandle *ioh, File file,
+ int iovcnt, off_t offset,
+ uint32 wait_event_info)
+{
+ int returnCode;
+ Vfd *vfdP;
+
+ Assert(FileIsValid(file));
+
+ DO_DB(elog(LOG, "FileStartReadV: %d (%s) " INT64_FORMAT " %d",
+ file, VfdCache[file].fileName,
+ (int64) offset,
+ iovcnt));
+
+ returnCode = FileAccess(file);
+ if (returnCode < 0)
+ return returnCode;
+
+ vfdP = &VfdCache[file];
+
+ pgaio_io_prep_readv(ioh, vfdP->fd, iovcnt, offset);
+
+ return 0;
+}
+
ssize_t
FileWriteV(File file, const struct iovec *iov, int iovcnt, off_t offset,
uint32 wait_event_info)
@@ -2498,6 +2529,12 @@ FilePathName(File file)
int
FileGetRawDesc(File file)
{
+ int returnCode;
+
+ returnCode = FileAccess(file);
+ if (returnCode < 0)
+ return returnCode;
+
Assert(FileIsValid(file));
return VfdCache[file].fd;
}
@@ -2778,6 +2815,7 @@ FreeDesc(AllocateDesc *desc)
result = closedir(desc->desc.dir);
break;
case AllocateDescRawFD:
+ pgaio_closing_fd(desc->desc.fd);
result = close(desc->desc.fd);
break;
default:
@@ -2846,6 +2884,8 @@ CloseTransientFile(int fd)
/* Only get here if someone passes us a file not in allocatedDescs */
elog(WARNING, "fd passed to CloseTransientFile was not obtained from OpenTransientFile");
+ pgaio_closing_fd(fd);
+
return close(fd);
}
diff --git a/src/backend/storage/smgr/md.c b/src/backend/storage/smgr/md.c
index f3220f98dc4..11d4d5a7aea 100644
--- a/src/backend/storage/smgr/md.c
+++ b/src/backend/storage/smgr/md.c
@@ -31,6 +31,7 @@
#include "miscadmin.h"
#include "pg_trace.h"
#include "pgstat.h"
+#include "storage/aio.h"
#include "storage/bufmgr.h"
#include "storage/fd.h"
#include "storage/md.h"
@@ -152,6 +153,15 @@ static MdfdVec *_mdfd_getseg(SMgrRelation reln, ForkNumber forknum,
static BlockNumber _mdnblocks(SMgrRelation reln, ForkNumber forknum,
MdfdVec *seg);
+static PgAioResult md_readv_complete(PgAioHandle *ioh, PgAioResult prior_result, uint8 cb_data);
+static void md_readv_report(PgAioResult result, const PgAioTargetData *target_data, int elevel);
+
+const PgAioHandleCallbacks aio_md_readv_cb = {
+ .complete_shared = md_readv_complete,
+ .report = md_readv_report,
+};
+
+
static inline int
_mdfd_open_flags(void)
{
@@ -937,6 +947,61 @@ mdreadv(SMgrRelation reln, ForkNumber forknum, BlockNumber blocknum,
}
}
+/*
+ * mdstartreadv() -- Asynchronous version of mdreadv().
+ */
+void
+mdstartreadv(PgAioHandle *ioh,
+ SMgrRelation reln, ForkNumber forknum, BlockNumber blocknum,
+ void **buffers, BlockNumber nblocks)
+{
+ off_t seekpos;
+ MdfdVec *v;
+ BlockNumber nblocks_this_segment;
+ struct iovec *iov;
+ int iovcnt;
+
+ v = _mdfd_getseg(reln, forknum, blocknum, false,
+ EXTENSION_FAIL | EXTENSION_CREATE_RECOVERY);
+
+ seekpos = (off_t) BLCKSZ * (blocknum % ((BlockNumber) RELSEG_SIZE));
+
+ Assert(seekpos < (off_t) BLCKSZ * RELSEG_SIZE);
+
+ nblocks_this_segment =
+ Min(nblocks,
+ RELSEG_SIZE - (blocknum % ((BlockNumber) RELSEG_SIZE)));
+
+ if (nblocks_this_segment != nblocks)
+ elog(ERROR, "read crossing segment boundary");
+
+ iovcnt = pgaio_io_get_iovec(ioh, &iov);
+
+ Assert(nblocks <= iovcnt);
+
+ iovcnt = buffers_to_iovec(iov, buffers, nblocks_this_segment);
+
+ Assert(iovcnt <= nblocks_this_segment);
+
+ if (!(io_direct_flags & IO_DIRECT_DATA))
+ pgaio_io_set_flag(ioh, PGAIO_HF_BUFFERED);
+
+ pgaio_io_set_target_smgr(ioh,
+ reln,
+ forknum,
+ blocknum,
+ nblocks,
+ false);
+ pgaio_io_register_callbacks(ioh, PGAIO_HCB_MD_READV, 0);
+
+ FileStartReadV(ioh, v->mdfd_vfd, iovcnt, seekpos, WAIT_EVENT_DATA_FILE_READ);
+
+ /*
+ * The error checks corresponding to the post-read checks in mdread() are
+ * in md_readv_complete().
+ */
+}
+
/*
* mdwritev() -- Write the supplied blocks at the appropriate location.
*
@@ -1365,6 +1430,21 @@ mdimmedsync(SMgrRelation reln, ForkNumber forknum)
}
}
+int
+mdfd(SMgrRelation reln, ForkNumber forknum, BlockNumber blocknum, uint32 *off)
+{
+ MdfdVec *v = mdopenfork(reln, forknum, EXTENSION_FAIL);
+
+ v = _mdfd_getseg(reln, forknum, blocknum, false,
+ EXTENSION_FAIL);
+
+ *off = (off_t) BLCKSZ * (blocknum % ((BlockNumber) RELSEG_SIZE));
+
+ Assert(*off < (off_t) BLCKSZ * RELSEG_SIZE);
+
+ return FileGetRawDesc(v->mdfd_vfd);
+}
+
/*
* register_dirty_segment() -- Mark a relation segment as needing fsync
*
@@ -1841,3 +1921,99 @@ mdfiletagmatches(const FileTag *ftag, const FileTag *candidate)
*/
return ftag->rlocator.dbOid == candidate->rlocator.dbOid;
}
+
+/*
+ * AIO completion callback for mdstartreadv().
+ */
+static PgAioResult
+md_readv_complete(PgAioHandle *ioh, PgAioResult prior_result, uint8 cb_data)
+{
+ PgAioTargetData *td = pgaio_io_get_target_data(ioh);
+ PgAioResult result = prior_result;
+
+ if (prior_result.result < 0)
+ {
+ result.status = ARS_ERROR;
+ result.id = PGAIO_HCB_MD_READV;
+ /* For "hard" errors, track the error number in error_data */
+ result.error_data = -prior_result.result;
+ result.result = 0;
+
+ pgaio_result_report(result, td, LOG);
+
+ return result;
+ }
+
+ /*
+ * The smgr API operates in blocks, therefore convert the result from
+ * bytes to blocks.
+ */
+ result.result /= BLCKSZ;
+
+ if (result.result == 0)
+ {
+ /* consider 0 blocks read a failure */
+ result.status = ARS_ERROR;
+ result.id = PGAIO_HCB_MD_READV;
+ result.error_data = 0;
+
+ pgaio_result_report(result, td, LOG);
+
+ return result;
+ }
+
+ if (result.status != ARS_ERROR &&
+ result.result < td->smgr.nblocks)
+ {
+ /* partial reads should be retried at upper level */
+ result.status = ARS_PARTIAL;
+ result.id = PGAIO_HCB_MD_READV;
+ }
+
+ return result;
+}
+
+/*
+ * AIO error reporting callback for mdstartreadv().
+ *
+ * Errors are encoded as follows:
+ * - PgAioResult.error_data != 0 encodes IO that failed with errno != 0
+ * - PgAioResult.error_data == 0 encodes IO that didn't read all data
+ */
+static void
+md_readv_report(PgAioResult result, const PgAioTargetData *td, int elevel)
+{
+ RelPathStr path;
+
+ path = relpathbackend(td->smgr.rlocator,
+ td->smgr.is_temp ? MyProcNumber : INVALID_PROC_NUMBER,
+ td->smgr.forkNum);
+
+ if (result.error_data != 0)
+ {
+ /* for errcode_for_file_access() and %m */
+ errno = result.error_data;
+
+ ereport(elevel,
+ errcode_for_file_access(),
+ errmsg("could not read blocks %u..%u in file \"%s\": %m",
+ td->smgr.blockNum,
+ td->smgr.blockNum + td->smgr.nblocks - 1,
+ path.str));
+ }
+ else
+ {
+ /*
+ * NB: This will typically only be output in debug messages, while
+ * retrying a partial IO.
+ */
+ ereport(elevel,
+ errcode(ERRCODE_DATA_CORRUPTED),
+ errmsg("could not read blocks %u..%u in file \"%s\": read only %zu of %zu bytes",
+ td->smgr.blockNum,
+ td->smgr.blockNum + td->smgr.nblocks - 1,
+ path.str,
+ result.result * (size_t) BLCKSZ,
+ td->smgr.nblocks * (size_t) BLCKSZ));
+ }
+}
diff --git a/src/backend/storage/smgr/smgr.c b/src/backend/storage/smgr/smgr.c
index 8787ce9b18f..53fd6af4799 100644
--- a/src/backend/storage/smgr/smgr.c
+++ b/src/backend/storage/smgr/smgr.c
@@ -63,6 +63,7 @@
#include "access/xlogutils.h"
#include "lib/ilist.h"
#include "miscadmin.h"
+#include "storage/aio.h"
#include "storage/bufmgr.h"
#include "storage/ipc.h"
#include "storage/md.h"
@@ -103,6 +104,10 @@ typedef struct f_smgr
void (*smgr_readv) (SMgrRelation reln, ForkNumber forknum,
BlockNumber blocknum,
void **buffers, BlockNumber nblocks);
+ void (*smgr_startreadv) (PgAioHandle *ioh,
+ SMgrRelation reln, ForkNumber forknum,
+ BlockNumber blocknum,
+ void **buffers, BlockNumber nblocks);
void (*smgr_writev) (SMgrRelation reln, ForkNumber forknum,
BlockNumber blocknum,
const void **buffers, BlockNumber nblocks,
@@ -114,6 +119,7 @@ typedef struct f_smgr
BlockNumber old_blocks, BlockNumber nblocks);
void (*smgr_immedsync) (SMgrRelation reln, ForkNumber forknum);
void (*smgr_registersync) (SMgrRelation reln, ForkNumber forknum);
+ int (*smgr_fd) (SMgrRelation reln, ForkNumber forknum, BlockNumber blocknum, uint32 *off);
} f_smgr;
static const f_smgr smgrsw[] = {
@@ -131,12 +137,14 @@ static const f_smgr smgrsw[] = {
.smgr_prefetch = mdprefetch,
.smgr_maxcombine = mdmaxcombine,
.smgr_readv = mdreadv,
+ .smgr_startreadv = mdstartreadv,
.smgr_writev = mdwritev,
.smgr_writeback = mdwriteback,
.smgr_nblocks = mdnblocks,
.smgr_truncate = mdtruncate,
.smgr_immedsync = mdimmedsync,
.smgr_registersync = mdregistersync,
+ .smgr_fd = mdfd,
}
};
@@ -154,6 +162,16 @@ static dlist_head unpinned_relns;
static void smgrshutdown(int code, Datum arg);
static void smgrdestroy(SMgrRelation reln);
+static void smgr_aio_reopen(PgAioHandle *ioh);
+static char *smgr_aio_describe_identity(const PgAioTargetData *sd);
+
+
+const PgAioTargetInfo aio_smgr_target_info = {
+ .name = "smgr",
+ .reopen = smgr_aio_reopen,
+ .describe_identity = smgr_aio_describe_identity,
+};
+
/*
* smgrinit(), smgrshutdown() -- Initialize or shut down storage
@@ -697,6 +715,24 @@ smgrreadv(SMgrRelation reln, ForkNumber forknum, BlockNumber blocknum,
RESUME_INTERRUPTS();
}
+/*
+ * smgrstartreadv() -- asynchronous version of smgrreadv()
+ *
+ * This starts an asynchronous readv IO with the IO handle `ioh`. Other than
+ * `ioh` all parameters are the same as smgrreadv().
+ */
+void
+smgrstartreadv(PgAioHandle *ioh,
+ SMgrRelation reln, ForkNumber forknum, BlockNumber blocknum,
+ void **buffers, BlockNumber nblocks)
+{
+ HOLD_INTERRUPTS();
+ smgrsw[reln->smgr_which].smgr_startreadv(ioh,
+ reln, forknum, blocknum, buffers,
+ nblocks);
+ RESUME_INTERRUPTS();
+}
+
/*
* smgrwritev() -- Write the supplied buffers out.
*
@@ -907,6 +943,25 @@ smgrimmedsync(SMgrRelation reln, ForkNumber forknum)
RESUME_INTERRUPTS();
}
+/*
+ * Return fd for the specified block number and update *off to the appropriate
+ * position.
+ *
+ * This is only to be used for when AIO needs to perform the IO in a different
+ * process than where it was issued (e.g. in an IO worker).
+ */
+static int
+smgrfd(SMgrRelation reln, ForkNumber forknum, BlockNumber blocknum, uint32 *off)
+{
+ int fd;
+
+ HOLD_INTERRUPTS();
+ fd = smgrsw[reln->smgr_which].smgr_fd(reln, forknum, blocknum, off);
+ RESUME_INTERRUPTS();
+
+ return fd;
+}
+
/*
* AtEOXact_SMgr
*
@@ -935,3 +990,80 @@ ProcessBarrierSmgrRelease(void)
smgrreleaseall();
return true;
}
+
+/*
+ * Set target of the IO handle to be smgr and initialize all the relevant
+ * pieces of data.
+ */
+void
+pgaio_io_set_target_smgr(PgAioHandle *ioh,
+ SMgrRelationData *smgr,
+ ForkNumber forknum,
+ BlockNumber blocknum,
+ int nblocks,
+ bool skip_fsync)
+{
+ PgAioTargetData *sd = pgaio_io_get_target_data(ioh);
+
+ pgaio_io_set_target(ioh, PGAIO_TID_SMGR);
+
+ /* backend is implied via IO owner */
+ sd->smgr.rlocator = smgr->smgr_rlocator.locator;
+ sd->smgr.forkNum = forknum;
+ sd->smgr.blockNum = blocknum;
+ sd->smgr.nblocks = nblocks;
+ sd->smgr.is_temp = SmgrIsTemp(smgr);
+ /* Temp relations should never be fsync'd */
+ sd->smgr.skip_fsync = skip_fsync && !SmgrIsTemp(smgr);
+}
+
+/*
+ * Callback for smgr AIO targets, to reopen the file (e.g. because the IO is
+ * executed in a worker).
+ */
+static void
+smgr_aio_reopen(PgAioHandle *ioh)
+{
+ PgAioTargetData *sd = pgaio_io_get_target_data(ioh);
+ PgAioOpData *od = pgaio_io_get_op_data(ioh);
+ SMgrRelation reln;
+ ProcNumber procno;
+ uint32 off;
+
+ if (sd->smgr.is_temp)
+ procno = pgaio_io_get_owner(ioh);
+ else
+ procno = INVALID_PROC_NUMBER;
+
+ reln = smgropen(sd->smgr.rlocator, procno);
+ od->read.fd = smgrfd(reln, sd->smgr.forkNum, sd->smgr.blockNum, &off);
+ Assert(off == od->read.offset);
+}
+
+/*
+ * Callback for smgr AIO targets, to describe the target of the IO.
+ */
+static char *
+smgr_aio_describe_identity(const PgAioTargetData *sd)
+{
+ RelPathStr path;
+ char *desc;
+
+ path = relpathbackend(sd->smgr.rlocator,
+ sd->smgr.is_temp ? MyProcNumber : INVALID_PROC_NUMBER,
+ sd->smgr.forkNum);
+
+ if (sd->smgr.nblocks == 0)
+ desc = psprintf(_("file \"%s\""), path.str);
+ else if (sd->smgr.nblocks == 1)
+ desc = psprintf(_("block %u in file \"%s\""),
+ sd->smgr.blockNum,
+ path.str);
+ else
+ desc = psprintf(_("blocks %u..%u in file \"%s\""),
+ sd->smgr.blockNum,
+ sd->smgr.blockNum + sd->smgr.nblocks - 1,
+ path.str);
+
+ return desc;
+}
--
2.48.1.76.g4e746b1a31.dirty
[text/x-diff] v2.8-0009-aio-Add-README.md-explaining-higher-level-desig.patch (19.0K, ../../3yxd5r23zly5bytvgyktbxtxq2r3gbpi7xd4dugevh3h4w4q6c@lu6oatjjpltz/10-v2.8-0009-aio-Add-README.md-explaining-higher-level-desig.patch)
download | inline diff:
From 62823ca4274f855baf5c2d5101936776cc69cd1c Mon Sep 17 00:00:00 2001
From: Andres Freund <andres@anarazel.de>
Date: Wed, 22 Jan 2025 13:44:37 -0500
Subject: [PATCH v2.8 09/38] aio: Add README.md explaining higher level design
Reviewed-by: Noah Misch <noah@leadboat.com>
Reviewed-by: Heikki Linnakangas <hlinnaka@iki.fi>
Discussion: https://postgr.es/m/uvrtrknj4kdytuboidbhwclo4gxhswwcpgadptsjvjqcluzmah%40brqs62irg4dt
Discussion: https://postgr.es/m/20210223100344.llw5an2aklengrmn@alap3.anarazel.de
Discussion: https://postgr.es/m/stj36ea6yyhoxtqkhpieia2z4krnam7qyetc57rfezgk4zgapf@gcnactj4z56m
---
src/backend/storage/aio/README.md | 423 ++++++++++++++++++++++++++++++
src/backend/storage/aio/aio.c | 2 +
2 files changed, 425 insertions(+)
create mode 100644 src/backend/storage/aio/README.md
diff --git a/src/backend/storage/aio/README.md b/src/backend/storage/aio/README.md
new file mode 100644
index 00000000000..b7dfb80b4b2
--- /dev/null
+++ b/src/backend/storage/aio/README.md
@@ -0,0 +1,423 @@
+# Asynchronous & Direct IO
+
+## Motivation
+
+### Why Asynchronous IO
+
+Until the introduction of asynchronous IO Postgres relied on the operating
+system to hide the cost of synchronous IO from Postgres. While this worked
+surprisingly well in a lot of workloads, it does not do as good a job on
+prefetching and controlled writeback as we would like.
+
+There are important expensive operations like `fdatasync()` where the operating
+system cannot hide the storage latency. This is particularly important for WAL
+writes, where the ability to asynchronously issue `fdatasync()` or O_DSYNC
+writes can yield significantly higher throughput.
+
+
+### Why Direct / unbuffered IO
+
+The main reasons to want to use Direct IO are:
+
+- Lower CPU usage / higher throughput. Particularly on modern storage buffered
+ writes are bottlenecked by the operating system having to copy data from the
+ kernel's page cache to postgres buffer pool using the CPU. Whereas direct IO
+ can often move the data directly between the storage devices and postgres'
+ buffer cache, using DMA. While that transfer is ongoing, the CPU is free to
+ perform other work.
+- Reduced latency - Direct IO can have substantially lower latency than
+ buffered IO, which can be impactful for OLTP workloads bottlenecked by WAL
+ write latency.
+- Avoiding double buffering between operating system cache and postgres'
+ shared_buffers.
+- Better control over the timing and pace of dirty data writeback.
+
+
+The main reasons *not* to use Direct IO are:
+
+- Without AIO, Direct IO is unusably slow for most purposes.
+- Even with AIO, many parts of postgres need to be modified to perform
+ explicit prefetching.
+- In situations where shared_buffers cannot be set appropriately large,
+ e.g. because there are many different postgres instances hosted on shared
+ hardware, performance will often be worse than when using buffered IO.
+
+
+## AIO Usage Example
+
+In many cases code that can benefit from AIO does not directly have to
+interact with the AIO interface, but can use AIO via higher-level
+abstractions. See [Helpers](#helpers).
+
+In this example, a buffer will be read into shared buffers.
+
+```C
+/*
+ * Result of the operation, only to be accessed in this backend.
+ */
+PgAioReturn ioret;
+
+/*
+ * Acquire an AIO Handle, ioret will get result upon completion.
+ *
+ * Note that ioret needs to stay alive until the IO completes or
+ * CurrentResourceOwner is released (i.e. an error is thrown).
+ */
+PgAioHandle *ioh = pgaio_io_acquire(CurrentResourceOwner, &ioret);
+
+/*
+ * Reference that can be used to wait for the IO we initiate below. This
+ * reference can reside in local or shared memory and waited upon by any
+ * process. An arbitrary number of references can be made for each IO.
+ */
+PgAioWaitRef iow;
+
+pgaio_io_get_wref(ioh, &iow);
+
+/*
+ * Arrange for shared buffer completion callbacks to be called upon completion
+ * of the IO. This callback will update the buffer descriptors associated with
+ * the AioHandle, which e.g. allows other backends to access the buffer.
+ *
+ * A callback can be passed a small bit of data, e.g. to indicate whether to
+ * zero a buffer if it is invalid.
+ *
+ * Multiple completion callbacks can be registered for each handle.
+ */
+pgaio_io_register_callbacks(ioh, PGAIO_HCB_SHARED_BUFFER_READV, 0);
+
+/*
+ * The completion callback needs to know which buffers to update when the IO
+ * completes. As the AIO subsystem does not know about buffers, we have to
+ * associate this information with the AioHandle, for use by the completion
+ * callback registered above.
+ *
+ * In this example we're reading only a single buffer, hence the 1.
+ */
+pgaio_io_set_handle_data_32(ioh, (uint32 *) buffer, 1);
+
+/*
+ * Pass the AIO handle to lower-level function. When operating on the level of
+ * buffers, we don't know how exactly the IO is performed, that is the
+ * responsibility of the storage manager implementation.
+ *
+ * E.g. md.c needs to translate block numbers into offsets in segments.
+ *
+ * Once the IO handle has been handed off to smgstartreadv(), it may not
+ * further be used, as the IO may immediately get executed below
+ * smgrstartreadv() and the handle reused for another IO.
+ *
+ * To issue multiple IOs in an efficient way, a caller can call
+ * pgaio_enter_batchmode() before starting multiple IOs, and end that batch
+ * with pgaio_exit_batchmode(). Note that one needs to be careful while there
+ * may be unsubmitted IOs, as another backend may need to wait for one of the
+ * unsubmitted IOs. If this backend then had to to wait for the other backend,
+ * it'd end in an undetected deadlock. See pgaio_enter_batchmode() for more
+ * details.
+ *
+ * Note that even while in batchmode an IO might get submitted immediately,
+ * e.g. due to reaching a limit on the number of unsubmitted IOs, and even
+ * complete before smgrstartreadv() returns.
+ */
+smgrstartreadv(ioh, operation->smgr, forknum, blkno,
+ BufferGetBlock(buffer), 1);
+
+/*
+ * To benefit from AIO, it is beneficial to perform other work, including
+ * submitting other IOs, before waiting for the IO to complete. Otherwise
+ * we could just have used synchronous, blocking IO.
+ */
+perform_other_work();
+
+/*
+ * We did some other work and now need the IO operation to have completed to
+ * continue.
+ */
+pgaio_wref_wait(&iow);
+
+/*
+ * At this point the IO has completed. We do not yet know whether it succeeded
+ * or failed, however. The buffer's state has been updated, which allows other
+ * backends to use the buffer (if the IO succeeded), or retry the IO (if it
+ * failed).
+ *
+ * Note that in case the IO has failed, a LOG message may have been emitted,
+ * but no ERROR has been raised. This is crucial, as another backend waiting
+ * for this IO should not see an ERROR.
+ *
+ * To check whether the operation succeeded, and to raise an ERROR, or if more
+ * appropriate LOG, the PgAioReturn we passed to pgaio_io_acquire() is used.
+ */
+if (ioret.result.status == ARS_ERROR)
+ pgaio_result_report(aio_ret.result, &aio_ret.target_data, ERROR);
+
+/*
+ * Besides having succeeded completely, the IO could also have partially
+ * completed. If we e.g. tried to read many blocks at once, the read might have
+ * only succeeded for the first few blocks.
+ *
+ * If the IO partially succeeded and this backend needs all blocks to have
+ * completed, this backend needs to reissue the IO for the remaining buffers.
+ * The AIO subsystem cannot handle this retry transparently.
+ *
+ * As this example is already long, and we only read a single block, we'll just
+ * error out if there's a partial read.
+ */
+if (ioret.result.status == ARS_PARTIAL)
+ pgaio_result_report(aio_ret.result, &aio_ret.target_data, ERROR);
+
+/*
+ * The IO succeeded, so we can use the buffer now.
+ */
+```
+
+
+## Design Criteria & Motivation
+
+### Deadlock and Starvation Dangers due to AIO
+
+Using AIO in a naive way can easily lead to deadlocks in an environment where
+the source/target of AIO are shared resources, like pages in postgres'
+shared_buffers.
+
+Consider one backend performing readahead on a table, initiating IO for a
+number of buffers ahead of the current "scan position". If that backend then
+performs some operation that blocks, or even just is slow, the IO completion
+for the asynchronously initiated read may not be processed.
+
+This AIO implementation solves this problem by requiring that AIO methods
+either allow AIO completions to be processed by any backend in the system
+(e.g. io_uring), or to guarantee that AIO processing will happen even when the
+issuing backend is blocked (e.g. worker mode, which offloads completion
+processing to the AIO workers).
+
+
+### IO can be started in critical sections
+
+Using AIO for WAL writes can reduce the overhead of WAL logging substantially:
+
+- AIO allows to start WAL writes eagerly, so they complete before needing to
+ wait
+- AIO allows to have multiple WAL flushes in progress at the same time
+- AIO makes it more realistic to use O\_DIRECT + O\_DSYNC, which can reduce
+ the number of roundtrips to storage on some OSs and storage HW (buffered IO
+ and direct IO without O_DSYNC needs to issue a write and after the write's
+ completion a cache cache flush, whereas O\_DIRECT + O\_DSYNC can use a
+ single Force Unit Access (FUA) write).
+
+The need to be able to execute IO in critical sections has substantial design
+implication on the AIO subsystem. Mainly because completing IOs (see prior
+section) needs to be possible within a critical section, even if the
+to-be-completed IO itself was not issued in a critical section. Consider
+e.g. the case of a backend first starting a number of writes from shared
+buffers and then starting to flush the WAL. Because only a limited amount of
+IO can be in-progress at the same time, initiating IO for flushing the WAL may
+require to first complete IO that was started earlier.
+
+
+### State for AIO needs to live in shared memory
+
+Because postgres uses a process model and because AIOs need to be
+complete-able by any backend much of the state of the AIO subsystem needs to
+live in shared memory.
+
+In an `EXEC_BACKEND` build, a backend's executable code and other process
+local state is not necessarily mapped to the same addresses in each process
+due to ASLR. This means that the shared memory cannot contain pointers to
+callbacks.
+
+
+## Design of the AIO Subsystem
+
+
+### AIO Methods
+
+To achieve portability and performance, multiple methods of performing AIO are
+implemented and others are likely worth adding in the future.
+
+
+#### Synchronous Mode
+
+`io_method=sync` does not actually perform AIO but allows to use the AIO API
+while performing synchronous IO. This can be useful for debugging. The code
+for the synchronous mode is also used as a fallback by e.g. the [worker
+mode](#worker) uses it to execute IO that cannot be executed by workers.
+
+
+#### Worker
+
+`io_method=worker` is available on every platform postgres runs on, and
+implements asynchronous IO - from the view of the issuing process - by
+dispatching the IO to one of several worker processes performing the IO in a
+synchronous manner.
+
+
+#### io_uring
+
+`io_method=io_uring` is available on Linux 5.1+. In contrast to worker mode it
+dispatches all IO from within the process, lowering context switch rate /
+latency.
+
+
+### AIO Handles
+
+The central API piece for postgres' AIO abstraction are AIO handles. To
+execute an IO one first has to acquire an IO handle (`pgaio_io_acquire()`) and
+then "defined", i.e. associate an IO operation with the handle.
+
+Often AIO handles are acquired on a higher level and then passed to a lower
+level to be fully defined. E.g., for IO to/from shared buffers, bufmgr.c
+routines acquire the handle, which is then passed through smgr.c, md.c to be
+finally fully defined in fd.c.
+
+The functions used at the lowest level to define the operation are
+`pgaio_io_prep_*()`.
+
+Because acquisition of an IO handle
+[must always succeed](#io-can-be-started-in-critical-sections)
+and the number of AIO Handles
+[has to be limited](#state-for-aio-needs-to-live-in-shared-memory)
+AIO handles can be reused as soon as they have completed. Obviously code needs
+to be able to react to IO completion. State can be updated using
+[AIO Completion callbacks](#aio-callbacks)
+and the issuing backend can provide a backend local variable to receive the
+result of the IO, as described in
+[AIO Result](#aio-results).
+An IO can be waited for, by both the issuing and any other backend, using
+[AIO References](#aio-wait-references).
+
+
+Because an AIO Handle is not executable just after calling
+`pgaio_io_acquire()` and because `pgaio_io_acquire()` needs to always succeed
+(absent a PANIC), only a single AIO Handle may be acquired (i.e. returned by
+`pgaio_io_acquire()`) without causing the IO to have been defined (by,
+potentially indirectly, causing `pgaio_io_prep_*()` to have been
+called). Otherwise a backend could trivially self-deadlock by using up all AIO
+Handles without the ability to wait for some of the IOs to complete.
+
+If it turns out that an AIO Handle is not needed, e.g., because the handle was
+acquired before holding a contended lock, it can be released without being
+defined using `pgaio_io_release()`.
+
+
+### AIO Callbacks
+
+Commonly several layers need to react to completion of an IO. E.g. for a read
+md.c needs to check if the IO outright failed or was shorter than needed,
+bufmgr.c needs to verify the page looks valid and bufmgr.c needs to update the
+BufferDesc to update the buffer's state.
+
+The fact that several layers / subsystems need to react to IO completion poses
+a few challenges:
+
+- Upper layers should not need to know details of lower layers. E.g. bufmgr.c
+ should not assume the IO will pass through md.c. Therefore upper levels
+ cannot know what lower layers would consider an error.
+
+- Lower layers should not need to know about upper layers. E.g. smgr APIs are
+ used going through shared buffers but are also used bypassing shared
+ buffers. This means that e.g. md.c is not in a position to validate
+ checksums.
+
+- Having code in the AIO subsystem for every possible combination of layers
+ would lead to a lot of duplication.
+
+The "solution" to this the ability to associate multiple completion callbacks
+with a handle. E.g. bufmgr.c can have a callback to update the BufferDesc
+state and to verify the page and md.c can have another callback to check if
+the IO operation was successful.
+
+As [mentioned](#state-for-aio-needs-to-live-in-shared-memory), shared memory
+currently cannot contain function pointers. Because of that completion
+callbacks are not directly identified by function pointers but by IDs
+(`PgAioHandleCallbackID`). A substantial added benefit is that that
+allows callbacks to be identified by much smaller amount of memory (a single
+byte currently).
+
+In addition to completion, AIO callbacks also are called to "stage" an
+IO. This is, e.g., used to increase buffer reference counts to account for the
+AIO subsystem referencing the buffer, which is required to handle the case
+where the issuing backend errors out and releases its own pins while the IO is
+still ongoing.
+
+As [explained earlier](#io-can-be-started-in-critical-sections) IO completions
+need to be safe to execute in critical sections. To allow the backend that
+issued the IO to error out in case of failure [AIO Result](#aio-results) can
+be used.
+
+
+### AIO Targets
+
+In addition to the completion callbacks describe above, each AIO Handle has
+exactly one "target". Each target has some space inside an AIO Handle with
+information specific to the target and can provide callbacks to allow to
+reopen the underlying file (required for worker mode) and to describe the IO
+operation (used for debug logging and error messages).
+
+I.e., if two different uses of AIO can describe the identity of the file being
+operated on the same way, it likely makes sense to use the same
+target. E.g. different smgr implementations can describe IO with
+RelFileLocator, ForkNumber and BlockNumber and can thus share a target. In
+contrast, IO for a WAL file would be described with TimeLineID and XLogRecPtr
+and it would not make sense to use the same target for smgr and WAL.
+
+
+### AIO Wait References
+
+As [described above](#aio-handles), AIO Handles can be reused immediately
+after completion and therefore cannot be used to wait for completion of the
+IO. Waiting is enabled using AIO wait references, which do not just identify
+an AIO Handle but also include the handles "generation".
+
+A reference to an AIO Handle can be acquired using `pgaio_io_get_wref()` and
+then waited upon using `pgaio_wref_wait()`.
+
+
+### AIO Results
+
+As AIO completion callbacks
+[are executed in critical sections](#io-can-be-started-in-critical-sections)
+and [may be executed by any backend](#deadlock-and-starvation-dangers-due-to-aio)
+completion callbacks cannot be used to, e.g., make the query that triggered an
+IO ERROR out.
+
+To allow to react to failing IOs the issuing backend can pass a pointer to a
+`PgAioReturn` in backend local memory. Before an AIO Handle is reused the
+`PgAioReturn` is filled with information about the IO. This includes
+information about whether the IO was successful (as a value of
+`PgAioResultStatus`) and enough information to raise an error in case of a
+failure (via `pgaio_result_report()`, with the error details encoded in
+`PgAioResult`).
+
+
+### AIO Errors
+
+It would be very convenient to have shared completion callbacks encode the
+details of errors as an `ErrorData` that could be raised at a later
+time. Unfortunately doing so would require allocating memory. While elog.c can
+guarantee (well, kinda) that logging a message will not run out of memory,
+that only works because a very limited number of messages are in the process
+of being logged. With AIO a large number of concurrently issued AIOs might
+fail.
+
+To avoid the need for preallocating a potentially large amount of memory (in
+shared memory no less!), completion callbacks instead have to encode errors in
+a more compact format that can be converted into an error message.
+
+
+## Helpers
+
+Using the low-level AIO API introduces too much complexity to do so all over
+the tree. Most uses of AIO should be done via reusable, higher-level,
+helpers.
+
+
+### Read Stream
+
+A common and very beneficial use of AIO are reads where a substantial number
+of to-be-read locations are known ahead of time. E.g., for a sequential scan
+the set of blocks that need to be read can be determined solely by knowing the
+current position and checking the buffer mapping table.
+
+The [Read Stream](../../../include/storage/read_stream.h) interface makes it
+comparatively easy to use AIO for such use cases.
diff --git a/src/backend/storage/aio/aio.c b/src/backend/storage/aio/aio.c
index 58d384b8403..3f126762a22 100644
--- a/src/backend/storage/aio/aio.c
+++ b/src/backend/storage/aio/aio.c
@@ -24,6 +24,8 @@
*
* - read_stream.c - helper for reading buffered relation data
*
+ * - README.md - higher-level overview over AIO
+ *
*
* Portions Copyright (c) 1996-2025, PostgreSQL Global Development Group
* Portions Copyright (c) 1994, Regents of the University of California
--
2.48.1.76.g4e746b1a31.dirty
[text/x-diff] v2.8-0010-aio-Add-pg_aios-view.patch (9.7K, ../../3yxd5r23zly5bytvgyktbxtxq2r3gbpi7xd4dugevh3h4w4q6c@lu6oatjjpltz/11-v2.8-0010-aio-Add-pg_aios-view.patch)
download | inline diff:
From e3be5af02530055dab22dad00d5bccdd4a4dba3a Mon Sep 17 00:00:00 2001
From: Andres Freund <andres@anarazel.de>
Date: Wed, 22 Jan 2025 13:44:40 -0500
Subject: [PATCH v2.8 10/38] aio: Add pg_aios view
Author:
Reviewed-by:
Discussion: https://postgr.es/m/
Backpatch:
---
src/include/catalog/pg_proc.dat | 10 ++
src/backend/catalog/system_views.sql | 3 +
src/backend/storage/aio/Makefile | 1 +
src/backend/storage/aio/aio_funcs.c | 219 +++++++++++++++++++++++++++
src/backend/storage/aio/meson.build | 1 +
src/test/regress/expected/rules.out | 16 ++
6 files changed, 250 insertions(+)
create mode 100644 src/backend/storage/aio/aio_funcs.c
diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat
index 890822eaf79..017971011f3 100644
--- a/src/include/catalog/pg_proc.dat
+++ b/src/include/catalog/pg_proc.dat
@@ -12479,4 +12479,14 @@
proargtypes => 'int4',
prosrc => 'gist_stratnum_common' },
+# AIO related functions
+{ oid => '9200', descr => 'information about in-progress asynchronous IOs',
+ proname => 'pg_get_aios', prorows => '100', proretset => 't',
+ provolatile => 'v', proparallel => 'r', prorettype => 'record', proargtypes => '',
+ proallargtypes => '{int4,int4,int8,text,text,int8,int8,text,int2,int4,text,text,bool,bool,bool}',
+ proargmodes => '{o,o,o,o,o,o,o,o,o,o,o,o,o,o,o}',
+ proargnames => '{pid,io_id,io_generation,state,operation,offset,length,target,handle_data_len,raw_result,result,target_desc,f_sync,f_localmem,f_buffered}',
+ prosrc => 'pg_get_aios' },
+
+
]
diff --git a/src/backend/catalog/system_views.sql b/src/backend/catalog/system_views.sql
index a4d2cfdcaf5..73902a763d1 100644
--- a/src/backend/catalog/system_views.sql
+++ b/src/backend/catalog/system_views.sql
@@ -1390,3 +1390,6 @@ CREATE VIEW pg_stat_subscription_stats AS
CREATE VIEW pg_wait_events AS
SELECT * FROM pg_get_wait_events();
+
+CREATE VIEW pg_aios AS
+ SELECT * FROM pg_get_aios();
diff --git a/src/backend/storage/aio/Makefile b/src/backend/storage/aio/Makefile
index c06c50771e0..3f2469cc399 100644
--- a/src/backend/storage/aio/Makefile
+++ b/src/backend/storage/aio/Makefile
@@ -11,6 +11,7 @@ include $(top_builddir)/src/Makefile.global
OBJS = \
aio.o \
aio_callback.o \
+ aio_funcs.o \
aio_init.o \
aio_io.o \
aio_target.o \
diff --git a/src/backend/storage/aio/aio_funcs.c b/src/backend/storage/aio/aio_funcs.c
new file mode 100644
index 00000000000..58883fd61d8
--- /dev/null
+++ b/src/backend/storage/aio/aio_funcs.c
@@ -0,0 +1,219 @@
+/*-------------------------------------------------------------------------
+ *
+ * aio_funcs.c
+ * AIO - SQL interface for AIO
+ *
+ *
+ * Portions Copyright (c) 1996-2025, PostgreSQL Global Development Group
+ * Portions Copyright (c) 1994, Regents of the University of California
+ *
+ * IDENTIFICATION
+ * src/backend/storage/aio/aio_funcs.c
+ *
+ *-------------------------------------------------------------------------
+ */
+
+#include "postgres.h"
+
+#include "storage/aio.h"
+#include "storage/aio_internal.h"
+#include "utils/builtins.h"
+#include "funcapi.h"
+#include "storage/proc.h"
+
+
+/*
+ * Byte length of an iovec.
+ */
+static size_t
+iov_byte_length(const struct iovec *iov, int cnt)
+{
+ size_t len = 0;
+
+ for (int i = 0; i < cnt; i++)
+ {
+ len += iov[i].iov_len;
+ }
+
+ return len;
+}
+
+Datum
+pg_get_aios(PG_FUNCTION_ARGS)
+{
+ ReturnSetInfo *rsinfo = (ReturnSetInfo *) fcinfo->resultinfo;
+
+ InitMaterializedSRF(fcinfo, 0);
+
+#define PG_GET_AIOS_COLS 15
+
+ for (uint64 i = 0; i < pgaio_ctl->io_handle_count; i++)
+ {
+ PgAioHandle *live_ioh = &pgaio_ctl->io_handles[i];
+ uint32 ioh_id = pgaio_io_get_id(live_ioh);
+ Datum values[PG_GET_AIOS_COLS] = {0};
+ bool nulls[PG_GET_AIOS_COLS] = {0};
+ ProcNumber owner;
+ PGPROC *owner_proc;
+ int32 owner_pid;
+ PgAioHandleState start_state;
+ uint64 start_generation;
+ PgAioHandle ioh_copy;
+ struct iovec iov_copy[PG_IOV_MAX];
+
+retry:
+
+ /*
+ * There is no lock that could prevent the state of the IO to advance
+ * concurrently - and we don't want to introduce one, as that would
+ * introduce atomics into a very common path. Instead we
+ *
+ * 1) Determine the state + generation of the IO.
+ *
+ * 2) Copy the IO to local memory.
+ *
+ * 3) Check if state or generation of the IO changed. If the state
+ * changed, retry, if the generation changed don't display the IO.
+ */
+
+ /* 1) from above */
+ start_generation = live_ioh->generation;
+ pg_read_barrier();
+ start_state = live_ioh->state;
+
+ if (start_state == PGAIO_HS_IDLE)
+ continue;
+
+ /* 2) from above */
+ memcpy(&ioh_copy, live_ioh, sizeof(PgAioHandle));
+
+ /*
+ * Safe to copy even if no iovec is used - we always reserve the
+ * required space.
+ */
+ memcpy(&iov_copy, &pgaio_ctl->iovecs[ioh_copy.iovec_off],
+ PG_IOV_MAX * sizeof(struct iovec));
+
+ /*
+ * Copy information about owner before 3) below, if the process exited
+ * it'd have to wait for the IO to finish first, which we would detect
+ * in 3).
+ */
+ owner = ioh_copy.owner_procno;
+ owner_proc = GetPGProcByNumber(owner);
+ owner_pid = owner_proc->pid;
+
+ /* 3) from above */
+ pg_read_barrier();
+
+ /*
+ * The IO completed and a new one was started with the same ID. Don't
+ * display it - it really started after this function was called.
+ * There be a risk of a livelock if we just retried endlessly, if IOs
+ * complete very quickly.
+ */
+ if (live_ioh->generation != start_generation)
+ continue;
+
+ /*
+ * The IOs state changed while we were "rendering" it. Just start from
+ * scratch. There's no risk of a livelock here, as an IO has a limited
+ * sets of states it can be in, and state changes go only in a single
+ * direction.
+ */
+ if (live_ioh->state != start_state)
+ goto retry;
+
+ /*
+ * Now that we have copied the IO into local memory and checked that
+ * it's still in the same state, we are not allowed to access "live"
+ * memory anymore. To make it slightly easier to catch such cases, set
+ * the "live" pointers to NULL.
+ */
+ live_ioh = NULL;
+ owner_proc = NULL;
+
+
+ /* column: owning pid */
+ if (owner_pid != 0)
+ values[0] = Int32GetDatum(owner_pid);
+ else
+ nulls[0] = false;
+
+ /* column: IO's id */
+ values[1] = ioh_id;
+
+ /* column: IO's generation */
+ values[2] = Int64GetDatum(start_generation);
+
+ /* column: IO's state */
+ values[3] = CStringGetTextDatum(pgaio_io_get_state_name(&ioh_copy));
+
+ /*
+ * If the IO is in PGAIO_HS_HANDED_OUT state, none of it's fields are
+ * valid yet (or are in the process of being set). Therefore we don't
+ * want to display any other columns.
+ */
+ if (start_state == PGAIO_HS_HANDED_OUT)
+ {
+ memset(nulls + 4, 1, (lengthof(nulls) - 4) * sizeof(bool));
+ goto display;
+ }
+
+ /* column: IO's operation */
+ values[4] = CStringGetTextDatum(pgaio_io_get_op_name(&ioh_copy));
+
+ /* columns: details about the IO's operation */
+ switch (ioh_copy.op)
+ {
+ case PGAIO_OP_INVALID:
+ nulls[5] = true;
+ nulls[6] = true;
+ break;
+ case PGAIO_OP_READV:
+ values[5] = Int64GetDatum(ioh_copy.op_data.read.offset);
+ values[6] =
+ Int64GetDatum(iov_byte_length(iov_copy, ioh_copy.op_data.read.iov_length));
+ break;
+ case PGAIO_OP_WRITEV:
+ values[5] = Int64GetDatum(ioh_copy.op_data.write.offset);
+ values[6] =
+ Int64GetDatum(iov_byte_length(iov_copy, ioh_copy.op_data.write.iov_length));
+ break;
+ }
+
+ /* column: IO's target */
+ values[7] = CStringGetTextDatum(pgaio_io_get_target_name(&ioh_copy));
+
+ /* column: length of IO's data array */
+ values[8] = Int16GetDatum(ioh_copy.handle_data_len);
+
+ /* column: raw result (i.e. some form of syscall return value) */
+ if (start_state == PGAIO_HS_COMPLETED_IO
+ || start_state == PGAIO_HS_COMPLETED_SHARED)
+ values[9] = Int32GetDatum(ioh_copy.result);
+ else
+ nulls[9] = true;
+
+ /*
+ * column: result in the higher level representation (unknown if not
+ * finished)
+ */
+ values[10] =
+ CStringGetTextDatum(pgaio_result_status_string(ioh_copy.distilled_result.status));
+
+ /* column: target description */
+ values[11] = CStringGetTextDatum(pgaio_io_get_target_description(&ioh_copy));
+
+ /* columns: one for each flag */
+ values[12] = BoolGetDatum(ioh_copy.flags & PGAIO_HF_SYNCHRONOUS);
+ values[13] = BoolGetDatum(ioh_copy.flags & PGAIO_HF_REFERENCES_LOCAL);
+ values[14] = BoolGetDatum(ioh_copy.flags & PGAIO_HF_BUFFERED);
+
+display:
+
+ tuplestore_putvalues(rsinfo->setResult, rsinfo->setDesc, values, nulls);
+ }
+
+ return (Datum) 0;
+}
diff --git a/src/backend/storage/aio/meson.build b/src/backend/storage/aio/meson.build
index 2f0f03d8071..da6df2d3654 100644
--- a/src/backend/storage/aio/meson.build
+++ b/src/backend/storage/aio/meson.build
@@ -3,6 +3,7 @@
backend_sources += files(
'aio.c',
'aio_callback.c',
+ 'aio_funcs.c',
'aio_init.c',
'aio_io.c',
'aio_target.c',
diff --git a/src/test/regress/expected/rules.out b/src/test/regress/expected/rules.out
index 62f69ac20b2..6570d0cddce 100644
--- a/src/test/regress/expected/rules.out
+++ b/src/test/regress/expected/rules.out
@@ -1286,6 +1286,22 @@ drop table cchild;
SELECT viewname, definition FROM pg_views
WHERE schemaname = 'pg_catalog'
ORDER BY viewname;
+pg_aios| SELECT pid,
+ io_id,
+ io_generation,
+ state,
+ operation,
+ "offset",
+ length,
+ target,
+ handle_data_len,
+ raw_result,
+ result,
+ target_desc,
+ f_sync,
+ f_localmem,
+ f_buffered
+ FROM pg_get_aios() pg_get_aios(pid, io_id, io_generation, state, operation, "offset", length, target, handle_data_len, raw_result, result, target_desc, f_sync, f_localmem, f_buffered);
pg_available_extension_versions| SELECT e.name,
e.version,
(x.extname IS NOT NULL) AS installed,
--
2.48.1.76.g4e746b1a31.dirty
[text/x-diff] v2.8-0011-Improve-read_stream.c-advice-for-big-random-chu.patch (7.6K, ../../3yxd5r23zly5bytvgyktbxtxq2r3gbpi7xd4dugevh3h4w4q6c@lu6oatjjpltz/12-v2.8-0011-Improve-read_stream.c-advice-for-big-random-chu.patch)
download | inline diff:
From 3dcfa20437c7febf65ef7ad7dbc1a3c4f0bf6980 Mon Sep 17 00:00:00 2001
From: Thomas Munro <thomas.munro@gmail.com>
Date: Tue, 18 Feb 2025 15:59:13 +1300
Subject: [PATCH v2.8 11/38] Improve read_stream.c advice for big random
chunks.
read_stream.c tries not to issue readahead advice when it thinks the
kernel's own readahead should be active, ie when using buffered I/O and
reading sequential blocks. It previously gave up too easily, and issued
advice only for the first read of size up to io_combine_limit in a
larger sequential range of blocks after a random jump. The following
read could suffer an avoidable I/O stall.
Fix, by issuing advice until the corresponding pread() calls catch up
with the start of the region we're currently issuing advice for, if
ever. That's when the kernel actually sees the sequential pattern and
has any chance of helping. That won't happen until the sequential
region fills the whole look-ahead window, so advice is now much more
aggressive, while still disabled for purely sequential streams.
While refactoring the advice logic, also get rid of the suppress_advice
argument that was passed around between functions, since
read_stream_start_pending_read() can make that determination itself, and
it's better to keep all that logic in one place.
Revealed by Tomas Vondra's disk access pattern tests with Melanie
Plageman's Bitmap Heap Scan patch.
Reviewed-by: Andres Freund <andres@anarazel.de> (earlier version)
Reported-by: Melanie Plageman <melanieplageman@gmail.com>
Reported-by: Tomas Vondra <tomas@vondra.me>
Reported-by: Andres Freund <andres@anarazel.de>
Tested-by: Melanie Plageman <melanieplageman@gmail.com>
Discussion: https://postgr.es/m/CA%2BhUKGK_%3D4CVmMHvsHjOVrK6t4F%3DLBpFzsrr3R%2BaJYN8kcTfWg%40mail.gmail.com
Discussion: https://postgr.es/m/CA%2BhUKGJ3HSWciQCz8ekP1Zn7N213RfA4nbuotQawfpq23%2Bw-5Q%40mail.gmail.com
---
src/backend/storage/aio/read_stream.c | 71 +++++++++++++++++++--------
1 file changed, 50 insertions(+), 21 deletions(-)
diff --git a/src/backend/storage/aio/read_stream.c b/src/backend/storage/aio/read_stream.c
index 175f8410baf..f991373359a 100644
--- a/src/backend/storage/aio/read_stream.c
+++ b/src/backend/storage/aio/read_stream.c
@@ -133,6 +133,7 @@ struct ReadStream
/* Next expected block, for detecting sequential access. */
BlockNumber seq_blocknum;
+ BlockNumber seq_until_processed;
/* The read operation we are currently preparing. */
BlockNumber pending_read_blocknum;
@@ -238,11 +239,11 @@ read_stream_unget_block(ReadStream *stream, BlockNumber blocknum)
* distance to a level that prevents look-ahead until buffers are released.
*/
static bool
-read_stream_start_pending_read(ReadStream *stream, bool suppress_advice)
+read_stream_start_pending_read(ReadStream *stream)
{
bool need_wait;
int nblocks;
- int flags;
+ int flags = 0;
int16 io_index;
int16 overflow;
int16 buffer_index;
@@ -262,16 +263,36 @@ read_stream_start_pending_read(ReadStream *stream, bool suppress_advice)
else
Assert(stream->next_buffer_index == stream->oldest_buffer_index);
- /*
- * If advice hasn't been suppressed, this system supports it, and this
- * isn't a strictly sequential pattern, then we'll issue advice.
- */
- if (!suppress_advice &&
- stream->advice_enabled &&
- stream->pending_read_blocknum != stream->seq_blocknum)
- flags = READ_BUFFERS_ISSUE_ADVICE;
- else
- flags = 0;
+ /* Do we need to issue read-ahead advice? */
+ if (stream->advice_enabled)
+ {
+ bool ahead;
+
+ /*
+ * We only issue advice if we're actually far enough ahead that we
+ * won't immediately have to call WaitReadBuffers().
+ */
+ ahead = stream->pinned_buffers > 0 ||
+ stream->pending_read_nblocks < stream->distance;
+
+ if (stream->pending_read_blocknum == stream->seq_blocknum)
+ {
+ /*
+ * Sequential: issue advice only until the WaitReadBuffers() calls
+ * catch up with the first advice issued for this sequential
+ * region, so the kernel can see sequential access.
+ */
+ if (stream->seq_until_processed != InvalidBlockNumber && ahead)
+ flags = READ_BUFFERS_ISSUE_ADVICE;
+ }
+ else
+ {
+ /* Random jump: start tracking new region. */
+ stream->seq_until_processed = stream->pending_read_blocknum;
+ if (ahead)
+ flags = READ_BUFFERS_ISSUE_ADVICE;
+ }
+ }
/* How many more buffers is this backend allowed? */
if (stream->temporary)
@@ -360,7 +381,7 @@ read_stream_start_pending_read(ReadStream *stream, bool suppress_advice)
}
static void
-read_stream_look_ahead(ReadStream *stream, bool suppress_advice)
+read_stream_look_ahead(ReadStream *stream)
{
while (stream->ios_in_progress < stream->max_ios &&
stream->pinned_buffers + stream->pending_read_nblocks < stream->distance)
@@ -371,8 +392,7 @@ read_stream_look_ahead(ReadStream *stream, bool suppress_advice)
if (stream->pending_read_nblocks == stream->io_combine_limit)
{
- read_stream_start_pending_read(stream, suppress_advice);
- suppress_advice = false;
+ read_stream_start_pending_read(stream);
continue;
}
@@ -405,15 +425,13 @@ read_stream_look_ahead(ReadStream *stream, bool suppress_advice)
/* We have to start the pending read before we can build another. */
while (stream->pending_read_nblocks > 0)
{
- if (!read_stream_start_pending_read(stream, suppress_advice) ||
+ if (!read_stream_start_pending_read(stream) ||
stream->ios_in_progress == stream->max_ios)
{
/* We've hit the buffer or I/O limit. Rewind and stop here. */
read_stream_unget_block(stream, blocknum);
return;
}
-
- suppress_advice = false;
}
/* This is the start of a new pending read. */
@@ -437,7 +455,7 @@ read_stream_look_ahead(ReadStream *stream, bool suppress_advice)
stream->pinned_buffers == 0) ||
stream->distance == 0) &&
stream->ios_in_progress < stream->max_ios)
- read_stream_start_pending_read(stream, suppress_advice);
+ read_stream_start_pending_read(stream);
/*
* There should always be something pinned when we leave this function,
@@ -613,6 +631,8 @@ read_stream_begin_impl(int flags,
stream->callback = callback;
stream->callback_private_data = callback_private_data;
stream->buffered_blocknum = InvalidBlockNumber;
+ stream->seq_blocknum = InvalidBlockNumber;
+ stream->seq_until_processed = InvalidBlockNumber;
stream->temporary = SmgrIsTemp(smgr);
/*
@@ -793,7 +813,7 @@ read_stream_next_buffer(ReadStream *stream, void **per_buffer_data)
* space for more, but if we're just starting up we'll need to crank
* the handle to get started.
*/
- read_stream_look_ahead(stream, true);
+ read_stream_look_ahead(stream);
/* End of stream reached? */
if (stream->pinned_buffers == 0)
@@ -838,6 +858,15 @@ read_stream_next_buffer(ReadStream *stream, void **per_buffer_data)
distance = stream->distance * 2;
distance = Min(distance, stream->max_pinned_buffers);
stream->distance = distance;
+
+ /*
+ * If we've caught up with the first advice issued for the current
+ * sequential region, cancel further advice until the next random
+ * jump. The kernel should be able to see the pattern now that
+ * we're actually making sequential preadv() calls.
+ */
+ if (stream->ios[io_index].op.blocknum == stream->seq_until_processed)
+ stream->seq_until_processed = InvalidBlockNumber;
}
else
{
@@ -899,7 +928,7 @@ read_stream_next_buffer(ReadStream *stream, void **per_buffer_data)
stream->oldest_buffer_index = 0;
/* Prepare for the next call. */
- read_stream_look_ahead(stream, false);
+ read_stream_look_ahead(stream);
#ifndef READ_STREAM_DISABLE_FAST_PATH
/* See if we can take the fast path for all-cached scans next time. */
--
2.48.1.76.g4e746b1a31.dirty
[text/x-diff] v2.8-0012-Simplify-distance-heuristics-in-read_stream.c.patch (7.4K, ../../3yxd5r23zly5bytvgyktbxtxq2r3gbpi7xd4dugevh3h4w4q6c@lu6oatjjpltz/13-v2.8-0012-Simplify-distance-heuristics-in-read_stream.c.patch)
download | inline diff:
From c0e57e66e59f7b7e98414c280d9e9d1c60c1f768 Mon Sep 17 00:00:00 2001
From: Thomas Munro <thomas.munro@gmail.com>
Date: Wed, 19 Feb 2025 01:25:40 +1300
Subject: [PATCH v2.8 12/38] Simplify distance heuristics in read_stream.c.
Previously, sequential reads would cause the look-ahead distance to
move towards io_combine_limit, on the basis that we wouldn't be issuing
advice so we wouldn't benefit from looking ahead further. That's not
really true, as we could suffer avoidable stalls when encountering
random jumps after sequential regions, for example in Bitmap Heap Scans
(with a proposed patch), and it is also incompatible with AIO plans,
where you always have to look ahead to start I/O.
Simplify the algorithm: now cache hits alone make the look-ahead
distance drop off, and cache misses make it grow rapidly as before.
Random vs sequential heuristics are no longer taken into consideration.
Reviewed-by: Andres Freund <andres@anarazel.de> (earlier version)
Tested-by: Melanie Plageman <melanieplageman@gmail.com>
Discussion: https://postgr.es/m/CA%2BhUKGK_%3D4CVmMHvsHjOVrK6t4F%3DLBpFzsrr3R%2BaJYN8kcTfWg%40mail.gmail.com
---
src/backend/storage/aio/read_stream.c | 94 ++++++++++-----------------
1 file changed, 34 insertions(+), 60 deletions(-)
diff --git a/src/backend/storage/aio/read_stream.c b/src/backend/storage/aio/read_stream.c
index f991373359a..ef30930b9b2 100644
--- a/src/backend/storage/aio/read_stream.c
+++ b/src/backend/storage/aio/read_stream.c
@@ -17,30 +17,12 @@
* pending read. When that isn't possible, the existing pending read is sent
* to StartReadBuffers() so that a new one can begin to form.
*
- * The algorithm for controlling the look-ahead distance tries to classify the
- * stream into three ideal behaviors:
- *
- * A) No I/O is necessary, because the requested blocks are fully cached
- * already. There is no benefit to looking ahead more than one block, so
- * distance is 1. This is the default initial assumption.
- *
- * B) I/O is necessary, but read-ahead advice is undesirable because the
- * access is sequential and we can rely on the kernel's read-ahead heuristics,
- * or impossible because direct I/O is enabled, or the system doesn't support
- * read-ahead advice. There is no benefit in looking ahead more than
- * io_combine_limit, because in this case the only goal is larger read system
- * calls. Looking further ahead would pin many buffers and perform
- * speculative work for no benefit.
- *
- * C) I/O is necessary, it appears to be random, and this system supports
- * read-ahead advice. We'll look further ahead in order to reach the
- * configured level of I/O concurrency.
- *
- * The distance increases rapidly and decays slowly, so that it moves towards
- * those levels as different I/O patterns are discovered. For example, a
- * sequential scan of fully cached data doesn't bother looking ahead, but a
- * sequential scan that hits a region of uncached blocks will start issuing
- * increasingly wide read calls until it plateaus at io_combine_limit.
+ * The algorithm for controlling the look-ahead distance is based on recent
+ * cache hit and miss history. When no I/O is necessary, there is no benefit
+ * in looking ahead more than one block. This is the default initial
+ * assumption, but when blocks needing I/O are streamed, the distance is
+ * increased rapidly to try to benefit from I/O combining and concurrency. It
+ * is reduced gradually when cached blocks are streamed.
*
* The main data structure is a circular queue of buffers of size
* max_pinned_buffers plus some extra space for technical reasons, ready to be
@@ -337,7 +319,7 @@ read_stream_start_pending_read(ReadStream *stream)
/* Remember whether we need to wait before returning this buffer. */
if (!need_wait)
{
- /* Look-ahead distance decays, no I/O necessary (behavior A). */
+ /* Look-ahead distance decays, no I/O necessary. */
if (stream->distance > 1)
stream->distance--;
}
@@ -518,6 +500,15 @@ read_stream_begin_impl(int flags,
else
max_ios = get_tablespace_io_concurrency(tablespace_id);
+ /*
+ * XXX Since we don't have asynchronous I/O yet, if direct I/O is enabled
+ * then just behave as though I/O concurrency is set to 0. Otherwise we
+ * would look ahead pinning many buffers for no benefit, for lack of
+ * advice and AIO.
+ */
+ if (io_direct_flags & IO_DIRECT_DATA)
+ max_ios = 0;
+
/* Cap to INT16_MAX to avoid overflowing below */
max_ios = Min(max_ios, PG_INT16_MAX);
@@ -638,7 +629,7 @@ read_stream_begin_impl(int flags,
/*
* Skip the initial ramp-up phase if the caller says we're going to be
* reading the whole relation. This way we start out assuming we'll be
- * doing full io_combine_limit sized reads (behavior B).
+ * doing full io_combine_limit sized reads.
*/
if (flags & READ_STREAM_FULL)
stream->distance = Min(max_pinned_buffers, stream->io_combine_limit);
@@ -729,10 +720,10 @@ read_stream_next_buffer(ReadStream *stream, void **per_buffer_data)
#ifndef READ_STREAM_DISABLE_FAST_PATH
/*
- * A fast path for all-cached scans (behavior A). This is the same as the
- * usual algorithm, but it is specialized for no I/O and no per-buffer
- * data, so we can skip the queue management code, stay in the same buffer
- * slot and use singular StartReadBuffer().
+ * A fast path for all-cached scans. This is the same as the usual
+ * algorithm, but it is specialized for no I/O and no per-buffer data, so
+ * we can skip the queue management code, stay in the same buffer slot and
+ * use singular StartReadBuffer().
*/
if (likely(stream->fast_path))
{
@@ -852,37 +843,20 @@ read_stream_next_buffer(ReadStream *stream, void **per_buffer_data)
if (++stream->oldest_io_index == stream->max_ios)
stream->oldest_io_index = 0;
- if (stream->ios[io_index].op.flags & READ_BUFFERS_ISSUE_ADVICE)
- {
- /* Distance ramps up fast (behavior C). */
- distance = stream->distance * 2;
- distance = Min(distance, stream->max_pinned_buffers);
- stream->distance = distance;
+ /* Look-ahead distance ramps up quickly after we do I/O. */
+ distance = stream->distance * 2;
+ distance = Min(distance, stream->max_pinned_buffers);
+ stream->distance = distance;
- /*
- * If we've caught up with the first advice issued for the current
- * sequential region, cancel further advice until the next random
- * jump. The kernel should be able to see the pattern now that
- * we're actually making sequential preadv() calls.
- */
- if (stream->ios[io_index].op.blocknum == stream->seq_until_processed)
- stream->seq_until_processed = InvalidBlockNumber;
- }
- else
- {
- /* No advice; move towards io_combine_limit (behavior B). */
- if (stream->distance > stream->io_combine_limit)
- {
- stream->distance--;
- }
- else
- {
- distance = stream->distance * 2;
- distance = Min(distance, stream->io_combine_limit);
- distance = Min(distance, stream->max_pinned_buffers);
- stream->distance = distance;
- }
- }
+ /*
+ * If we've caught up with the first advice issued for the current
+ * sequential region, cancel further advice until the next random
+ * jump. The kernel should be able to see the pattern now that we're
+ * actually making sequential preadv() calls.
+ */
+ if (stream->advice_enabled &&
+ stream->ios[io_index].op.blocknum == stream->seq_until_processed)
+ stream->seq_until_processed = InvalidBlockNumber;
}
#ifdef CLOBBER_FREED_MEMORY
--
2.48.1.76.g4e746b1a31.dirty
[text/x-diff] v2.8-0013-Support-buffer-forwarding-in-read_stream.c.patch (8.7K, ../../3yxd5r23zly5bytvgyktbxtxq2r3gbpi7xd4dugevh3h4w4q6c@lu6oatjjpltz/14-v2.8-0013-Support-buffer-forwarding-in-read_stream.c.patch)
download | inline diff:
From fcd1ffd6c68ae2c3b6687a4917fd4300cb9db3af Mon Sep 17 00:00:00 2001
From: Thomas Munro <thomas.munro@gmail.com>
Date: Thu, 30 Jan 2025 11:42:03 +1300
Subject: [PATCH v2.8 13/38] Support buffer forwarding in read_stream.c.
In preparation for a following change to the buffer manager, teach read
stream to keep track of buffers that were "forwarded" from one call to
StartReadBuffers() to the next.
Since StartReadBuffers() buffers argument will become an in/out
argument, we need to initialize the buffer queue entries with
InvalidBuffer. We don't want to do that up front, because we try to
keep stream initialization cheap and code that uses the fast path stays
in one single buffer queue element. Satisfy both goals by initializing
the queue incrementally on the first cycle.
Discussion: https://postgr.es/m/CA%2BhUKGK_%3D4CVmMHvsHjOVrK6t4F%3DLBpFzsrr3R%2BaJYN8kcTfWg%40mail.gmail.com
---
src/backend/storage/aio/read_stream.c | 103 +++++++++++++++++++++++---
1 file changed, 93 insertions(+), 10 deletions(-)
diff --git a/src/backend/storage/aio/read_stream.c b/src/backend/storage/aio/read_stream.c
index ef30930b9b2..0f1525f46c5 100644
--- a/src/backend/storage/aio/read_stream.c
+++ b/src/backend/storage/aio/read_stream.c
@@ -95,8 +95,10 @@ struct ReadStream
int16 ios_in_progress;
int16 queue_size;
int16 max_pinned_buffers;
+ int16 forwarded_buffers;
int16 pinned_buffers;
int16 distance;
+ int16 initialized_buffers;
bool advice_enabled;
bool temporary;
@@ -224,8 +226,10 @@ static bool
read_stream_start_pending_read(ReadStream *stream)
{
bool need_wait;
+ int requested_nblocks;
int nblocks;
int flags = 0;
+ int forwarded;
int16 io_index;
int16 overflow;
int16 buffer_index;
@@ -276,11 +280,20 @@ read_stream_start_pending_read(ReadStream *stream)
}
}
- /* How many more buffers is this backend allowed? */
+ /*
+ * How many more buffers is this backend allowed?
+ *
+ * If we already have some forwarded buffers, we can certainly use those.
+ * They are already pinned, and are mapped to the starting blocks of the
+ * pending read, they just don't have any I/O started yet and are not
+ * counted in stream->pinned_buffers.
+ */
if (stream->temporary)
buffer_limit = Min(GetAdditionalLocalPinLimit(), PG_INT16_MAX);
else
buffer_limit = Min(GetAdditionalPinLimit(), PG_INT16_MAX);
+ Assert(stream->forwarded_buffers <= stream->pending_read_nblocks);
+ buffer_limit += stream->forwarded_buffers;
if (buffer_limit == 0 && stream->pinned_buffers == 0)
buffer_limit = 1; /* guarantee progress */
@@ -307,8 +320,31 @@ read_stream_start_pending_read(ReadStream *stream)
* We say how many blocks we want to read, but it may be smaller on return
* if the buffer manager decides to shorten the read.
*/
+ requested_nblocks = Min(buffer_limit, stream->pending_read_nblocks);
+ nblocks = requested_nblocks;
buffer_index = stream->next_buffer_index;
io_index = stream->next_io_index;
+
+ /*
+ * The first time around the queue we initialize it as we go, including
+ * the overflow zone, because otherwise the entries would appear as
+ * forwarded buffers. This avoids initializing the whole queue up front
+ * in cases where it is large but we don't ever use it due to the
+ * all-cached fast path or small scans.
+ */
+ while (stream->initialized_buffers < buffer_index + nblocks)
+ stream->buffers[stream->initialized_buffers++] = InvalidBuffer;
+
+ /*
+ * Start the I/O. Any buffers that are not InvalidBuffer will be
+ * interpreted as already pinned, forwarded by an earlier call to
+ * StartReadBuffers(), and must map to the expected blocks. The nblocks
+ * value may be smaller on return indicating the size of the I/O that
+ * could be started. Buffers beyond the output nblocks number may also
+ * have been pinned without starting I/O due to various edge cases. In
+ * that case we'll just leave them in the queue ahead of us, "forwarded"
+ * to the next call, avoiding the need to unpin/repin.
+ */
need_wait = StartReadBuffers(&stream->ios[io_index].op,
&stream->buffers[buffer_index],
stream->pending_read_blocknum,
@@ -337,16 +373,35 @@ read_stream_start_pending_read(ReadStream *stream)
stream->seq_blocknum = stream->pending_read_blocknum + nblocks;
}
+ /*
+ * How many pins were acquired but forwarded to the next call? These need
+ * to be passed to the next StartReadBuffers() call, or released if the
+ * stream ends early. We need the number for accounting purposes, since
+ * they are not counted in stream->pinned_buffers but we already hold
+ * them.
+ */
+ forwarded = 0;
+ while (nblocks + forwarded < requested_nblocks &&
+ stream->buffers[buffer_index + nblocks + forwarded] != InvalidBuffer)
+ forwarded++;
+ stream->forwarded_buffers = forwarded;
+
/*
* We gave a contiguous range of buffer space to StartReadBuffers(), but
- * we want it to wrap around at queue_size. Slide overflowing buffers to
- * the front of the array.
+ * we want it to wrap around at queue_size. Copy overflowing buffers to
+ * the front of the array where they'll be consumed, but also leave a copy
+ * in the overflow zone which the I/O operation has a pointer to (it needs
+ * a contiguous array). Both copies will be cleared when the buffers are
+ * handed to the consumer.
*/
- overflow = (buffer_index + nblocks) - stream->queue_size;
+ overflow = (buffer_index + nblocks + forwarded) - stream->queue_size;
if (overflow > 0)
- memmove(&stream->buffers[0],
- &stream->buffers[stream->queue_size],
- sizeof(stream->buffers[0]) * overflow);
+ {
+ Assert(overflow < stream->queue_size); /* can't overlap */
+ memcpy(&stream->buffers[0],
+ &stream->buffers[stream->queue_size],
+ sizeof(stream->buffers[0]) * overflow);
+ }
/* Compute location of start of next read, without using % operator. */
buffer_index += nblocks;
@@ -731,10 +786,12 @@ read_stream_next_buffer(ReadStream *stream, void **per_buffer_data)
/* Fast path assumptions. */
Assert(stream->ios_in_progress == 0);
+ Assert(stream->forwarded_buffers == 0);
Assert(stream->pinned_buffers == 1);
Assert(stream->distance == 1);
Assert(stream->pending_read_nblocks == 0);
Assert(stream->per_buffer_data_size == 0);
+ Assert(stream->initialized_buffers > stream->oldest_buffer_index);
/* We're going to return the buffer we pinned last time. */
oldest_buffer_index = stream->oldest_buffer_index;
@@ -783,6 +840,7 @@ read_stream_next_buffer(ReadStream *stream, void **per_buffer_data)
stream->distance = 0;
stream->oldest_buffer_index = stream->next_buffer_index;
stream->pinned_buffers = 0;
+ stream->buffers[oldest_buffer_index] = InvalidBuffer;
}
stream->fast_path = false;
@@ -859,10 +917,15 @@ read_stream_next_buffer(ReadStream *stream, void **per_buffer_data)
stream->seq_until_processed = InvalidBlockNumber;
}
-#ifdef CLOBBER_FREED_MEMORY
- /* Clobber old buffer for debugging purposes. */
+ /*
+ * We must zap this queue entry, or else it would appear as a forwarded
+ * buffer. If it's potentially in the overflow zone (ie it wrapped around
+ * the queue), also zap that copy.
+ */
stream->buffers[oldest_buffer_index] = InvalidBuffer;
-#endif
+ if (oldest_buffer_index < io_combine_limit - 1)
+ stream->buffers[stream->queue_size + oldest_buffer_index] =
+ InvalidBuffer;
#if defined(CLOBBER_FREED_MEMORY) || defined(USE_VALGRIND)
@@ -907,6 +970,7 @@ read_stream_next_buffer(ReadStream *stream, void **per_buffer_data)
#ifndef READ_STREAM_DISABLE_FAST_PATH
/* See if we can take the fast path for all-cached scans next time. */
if (stream->ios_in_progress == 0 &&
+ stream->forwarded_buffers == 0 &&
stream->pinned_buffers == 1 &&
stream->distance == 1 &&
stream->pending_read_nblocks == 0 &&
@@ -942,6 +1006,7 @@ read_stream_next_block(ReadStream *stream, BufferAccessStrategy *strategy)
void
read_stream_reset(ReadStream *stream)
{
+ int16 index;
Buffer buffer;
/* Stop looking ahead. */
@@ -955,6 +1020,24 @@ read_stream_reset(ReadStream *stream)
while ((buffer = read_stream_next_buffer(stream, NULL)) != InvalidBuffer)
ReleaseBuffer(buffer);
+ /* Unpin any unused forwarded buffers. */
+ index = stream->next_buffer_index;
+ while (index < stream->initialized_buffers &&
+ (buffer = stream->buffers[index]) != InvalidBuffer)
+ {
+ Assert(stream->forwarded_buffers > 0);
+ stream->forwarded_buffers--;
+ ReleaseBuffer(buffer);
+
+ stream->buffers[index] = InvalidBuffer;
+ if (index < io_combine_limit - 1)
+ stream->buffers[stream->queue_size + index] = InvalidBuffer;
+
+ if (++index == stream->queue_size)
+ index = 0;
+ }
+
+ Assert(stream->forwarded_buffers == 0);
Assert(stream->pinned_buffers == 0);
Assert(stream->ios_in_progress == 0);
--
2.48.1.76.g4e746b1a31.dirty
[text/x-diff] v2.8-0014-Support-buffer-forwarding-in-StartReadBuffers.patch (10.5K, ../../3yxd5r23zly5bytvgyktbxtxq2r3gbpi7xd4dugevh3h4w4q6c@lu6oatjjpltz/15-v2.8-0014-Support-buffer-forwarding-in-StartReadBuffers.patch)
download | inline diff:
From 3195c6f626461fc798657a974463bbb6871efbe5 Mon Sep 17 00:00:00 2001
From: Thomas Munro <thomas.munro@gmail.com>
Date: Mon, 10 Feb 2025 21:55:40 +1300
Subject: [PATCH v2.8 14/38] Support buffer forwarding in StartReadBuffers().
Sometimes we have to perform a short read because we hit a cached block
that ends a contiguous run of blocks requiring I/O. We don't want
StartReadBuffers() to have to start more than one I/O, so we stop there.
We also don't want to have to unpin the cached block (and repin it
later), so previously we'd silently pretend the hit was part of the I/O,
and just leave it out of the read from disk. Now, we'll "forward" it to
the next call. We still write it to the buffers[] array for the caller
to pass back to us later, but it's not included in *nblocks.
This policy means that we no longer mix hits and misses in a single
operation's results, so we avoid the requirement to call
WaitReadBuffers(), which might stall, before the caller can make use of
the hits. The caller will get the hit in the next call instead, and
know that it doesn't have to wait. That's important for later work on
out-of-order read streams that minimize I/O stalls.
This also makes life easier for proposed work on true AIO, which
occasionally needs to split a large I/O after pinning all the buffers,
while the current coding only ever forwards a single bookending hit.
This API is natural for read_stream.c: it just leaves forwarded buffers
where they are in its circular queue, where the next call will pick them
up and continue, minimizing pin churn.
If we ever think of a good reason to disable this feature, i.e. for
other users of StartReadBuffers() that don't want to deal with forwarded
buffers, then we could add a flag for that. For now read_steam.c is the
only user.
Discussion: https://postgr.es/m/CA%2BhUKGK_%3D4CVmMHvsHjOVrK6t4F%3DLBpFzsrr3R%2BaJYN8kcTfWg%40mail.gmail.com
---
src/include/storage/bufmgr.h | 1 -
src/backend/storage/buffer/bufmgr.c | 128 ++++++++++++++++++++--------
2 files changed, 91 insertions(+), 38 deletions(-)
diff --git a/src/include/storage/bufmgr.h b/src/include/storage/bufmgr.h
index 79a89f87fcc..690cdde793a 100644
--- a/src/include/storage/bufmgr.h
+++ b/src/include/storage/bufmgr.h
@@ -130,7 +130,6 @@ struct ReadBuffersOperation
BlockNumber blocknum;
int flags;
int16 nblocks;
- int16 io_buffers_len;
};
typedef struct ReadBuffersOperation ReadBuffersOperation;
diff --git a/src/backend/storage/buffer/bufmgr.c b/src/backend/storage/buffer/bufmgr.c
index 8243f4b2445..ef519b4c356 100644
--- a/src/backend/storage/buffer/bufmgr.c
+++ b/src/backend/storage/buffer/bufmgr.c
@@ -1257,10 +1257,10 @@ StartReadBuffersImpl(ReadBuffersOperation *operation,
Buffer *buffers,
BlockNumber blockNum,
int *nblocks,
- int flags)
+ int flags,
+ bool allow_forwarding)
{
int actual_nblocks = *nblocks;
- int io_buffers_len = 0;
int maxcombine = 0;
Assert(*nblocks > 0);
@@ -1270,30 +1270,80 @@ StartReadBuffersImpl(ReadBuffersOperation *operation,
{
bool found;
- buffers[i] = PinBufferForBlock(operation->rel,
- operation->smgr,
- operation->persistence,
- operation->forknum,
- blockNum + i,
- operation->strategy,
- &found);
+ if (allow_forwarding && buffers[i] != InvalidBuffer)
+ {
+ BufferDesc *bufHdr;
+
+ /*
+ * This is a buffer that was pinned by an earlier call to
+ * StartReadBuffers(), but couldn't be handled in one operation at
+ * that time. The operation was split, and the caller has passed
+ * an already pinned buffer back to us to handle the rest of the
+ * operation. It must continue at the expected block number.
+ */
+ Assert(BufferGetBlockNumber(buffers[i]) == blockNum + i);
+
+ /*
+ * It might be an already valid buffer (a hit) that followed the
+ * final contiguous block of an earlier I/O (a miss) marking the
+ * end of it, or a buffer that some other backend has since made
+ * valid by performing the I/O for us, in which case we can handle
+ * it as a hit now. It is safe to check for a BM_VALID flag with
+ * a relaxed load, because we got a fresh view of it while pinning
+ * it in the previous call.
+ *
+ * On the other hand if we don't see BM_VALID yet, it must be an
+ * I/O that was split by the previous call and we need to try to
+ * start a new I/O from this block. We're also racing against any
+ * other backend that might start the I/O or even manage to mark
+ * it BM_VALID after this check, but StartBufferIO() will handle
+ * those cases.
+ */
+ if (BufferIsLocal(buffers[i]))
+ bufHdr = GetLocalBufferDescriptor(-buffers[i] - 1);
+ else
+ bufHdr = GetBufferDescriptor(buffers[i] - 1);
+ found = pg_atomic_read_u32(&bufHdr->state) & BM_VALID;
+ }
+ else
+ {
+ buffers[i] = PinBufferForBlock(operation->rel,
+ operation->smgr,
+ operation->persistence,
+ operation->forknum,
+ blockNum + i,
+ operation->strategy,
+ &found);
+ }
if (found)
{
/*
- * Terminate the read as soon as we get a hit. It could be a
- * single buffer hit, or it could be a hit that follows a readable
- * range. We don't want to create more than one readable range,
- * so we stop here.
+ * We have a hit. If it's the first block in the requested range,
+ * we can return it immediately and report that WaitReadBuffers()
+ * does not need to be called. If the initial value of *nblocks
+ * was larger, the caller will have to call again for the rest.
*/
- actual_nblocks = i + 1;
+ if (i == 0)
+ {
+ *nblocks = 1;
+ return false;
+ }
+
+ /*
+ * Otherwise we already have an I/O to perform, but this block
+ * can't be included as it is already valid. Split the I/O here.
+ * There may or may not be more blocks requiring I/O after this
+ * one, we haven't checked, but it can't be contiguous with this
+ * hit in the way. We'll leave this buffer pinned, forwarding it
+ * to the next call, avoiding the need to unpin it here and re-pin
+ * it in the next call.
+ */
+ actual_nblocks = i;
break;
}
else
{
- /* Extend the readable range to cover this block. */
- io_buffers_len++;
-
/*
* Check how many blocks we can cover with the same IO. The smgr
* implementation might e.g. be limited due to a segment boundary.
@@ -1314,15 +1364,11 @@ StartReadBuffersImpl(ReadBuffersOperation *operation,
}
*nblocks = actual_nblocks;
- if (likely(io_buffers_len == 0))
- return false;
-
/* Populate information needed for I/O. */
operation->buffers = buffers;
operation->blocknum = blockNum;
operation->flags = flags;
operation->nblocks = actual_nblocks;
- operation->io_buffers_len = io_buffers_len;
if (flags & READ_BUFFERS_ISSUE_ADVICE)
{
@@ -1337,7 +1383,7 @@ StartReadBuffersImpl(ReadBuffersOperation *operation,
smgrprefetch(operation->smgr,
operation->forknum,
blockNum,
- operation->io_buffers_len);
+ actual_nblocks);
}
/* Indicate that WaitReadBuffers() should be called. */
@@ -1351,11 +1397,21 @@ StartReadBuffersImpl(ReadBuffersOperation *operation,
* actual number, which may be fewer than requested. Caller sets some of the
* members of operation; see struct definition.
*
+ * The initial contents of the elements of buffers up to *nblocks should
+ * either be InvalidBuffer or an already-pinned buffer that was left by an
+ * preceding call to StartReadBuffers() that had to be split. On return, some
+ * elements of buffers may hold pinned buffers beyond the number indicated by
+ * the updated value of *nblocks. Operations are split on boundaries known to
+ * smgr (eg md.c segment boundaries that require crossing into a different
+ * underlying file), or when already cached blocks are found in the buffer
+ * that prevent the formation of a contiguous read.
+ *
* If false is returned, no I/O is necessary. If true is returned, one I/O
* has been started, and WaitReadBuffers() must be called with the same
* operation object before the buffers are accessed. Along with the operation
* object, the caller-supplied array of buffers must remain valid until
- * WaitReadBuffers() is called.
+ * WaitReadBuffers() is called, and any forwarded buffers must also be
+ * preserved for a future call unless explicitly released.
*
* Currently the I/O is only started with optional operating system advice if
* requested by the caller with READ_BUFFERS_ISSUE_ADVICE, and the real I/O
@@ -1369,13 +1425,18 @@ StartReadBuffers(ReadBuffersOperation *operation,
int *nblocks,
int flags)
{
- return StartReadBuffersImpl(operation, buffers, blockNum, nblocks, flags);
+ return StartReadBuffersImpl(operation, buffers, blockNum, nblocks, flags,
+ true /* expect forwarded buffers */ );
}
/*
* Single block version of the StartReadBuffers(). This might save a few
* instructions when called from another translation unit, because it is
* specialized for nblocks == 1.
+ *
+ * This version does not support "forwarded" buffers: they cannot be created
+ * by reading only one block, and the current contents of *buffer is ignored
+ * on entry.
*/
bool
StartReadBuffer(ReadBuffersOperation *operation,
@@ -1386,7 +1447,8 @@ StartReadBuffer(ReadBuffersOperation *operation,
int nblocks = 1;
bool result;
- result = StartReadBuffersImpl(operation, buffer, blocknum, &nblocks, flags);
+ result = StartReadBuffersImpl(operation, buffer, blocknum, &nblocks, flags,
+ false /* single block, no forwarding */ );
Assert(nblocks == 1); /* single block can't be short */
return result;
@@ -1416,24 +1478,16 @@ WaitReadBuffers(ReadBuffersOperation *operation)
IOObject io_object;
char persistence;
- /*
- * Currently operations are only allowed to include a read of some range,
- * with an optional extra buffer that is already pinned at the end. So
- * nblocks can be at most one more than io_buffers_len.
- */
- Assert((operation->nblocks == operation->io_buffers_len) ||
- (operation->nblocks == operation->io_buffers_len + 1));
-
/* Find the range of the physical read we need to perform. */
- nblocks = operation->io_buffers_len;
- if (nblocks == 0)
- return; /* nothing to do */
-
+ nblocks = operation->nblocks;
buffers = &operation->buffers[0];
blocknum = operation->blocknum;
forknum = operation->forknum;
persistence = operation->persistence;
+ Assert(nblocks > 0);
+ Assert(nblocks <= MAX_IO_COMBINE_LIMIT);
+
if (persistence == RELPERSISTENCE_TEMP)
{
io_context = IOCONTEXT_NORMAL;
--
2.48.1.76.g4e746b1a31.dirty
[text/x-diff] v2.8-0015-tests-Expand-temp-table-tests-to-some-pin-relat.patch (9.6K, ../../3yxd5r23zly5bytvgyktbxtxq2r3gbpi7xd4dugevh3h4w4q6c@lu6oatjjpltz/16-v2.8-0015-tests-Expand-temp-table-tests-to-some-pin-relat.patch)
download | inline diff:
From db791c683bbc295ab8cc83a3e3ac2b11064a40f9 Mon Sep 17 00:00:00 2001
From: Andres Freund <andres@anarazel.de>
Date: Tue, 4 Mar 2025 11:07:47 -0500
Subject: [PATCH v2.8 15/38] tests: Expand temp table tests to some pin related
matters
Author:
Reviewed-By:
Discussion: https://postgr.es/m/
Backpatch:
---
src/test/regress/expected/temp.out | 156 +++++++++++++++++++++++++++++
src/test/regress/parallel_schedule | 2 +-
src/test/regress/sql/temp.sql | 107 ++++++++++++++++++++
3 files changed, 264 insertions(+), 1 deletion(-)
diff --git a/src/test/regress/expected/temp.out b/src/test/regress/expected/temp.out
index 2a246a7e123..91fe519b1cc 100644
--- a/src/test/regress/expected/temp.out
+++ b/src/test/regress/expected/temp.out
@@ -410,3 +410,159 @@ SELECT current_schema() ~ 'pg_temp' AS is_temp_schema;
PREPARE TRANSACTION 'twophase_search';
ERROR: cannot PREPARE a transaction that has operated on temporary objects
+-- Tests to verify we recover correctly from exhausting buffer pins and
+-- related matters.
+-- use lower possible buffer limit to make the test cheaper
+\c
+SET temp_buffers = 100;
+CREATE TEMPORARY TABLE test_temp(a int not null unique, b TEXT not null, cnt int not null);
+INSERT INTO test_temp SELECT generate_series(1, 10000) as id, repeat('a', 200), 0;
+-- should be at least 2x as large than temp_buffers
+SELECT pg_relation_size('test_temp') / current_setting('block_size')::int8 > 200;
+ ?column?
+----------
+ t
+(1 row)
+
+-- Don't want cursor names and plpgsql function lines in the error messages
+\set VERBOSITY terse
+/* helper function to create cursors for each page in [p_start, p_end] */
+CREATE FUNCTION test_temp_pin(p_start int, p_end int)
+RETURNS void
+LANGUAGE plpgsql
+AS $f$
+ DECLARE
+ cursorname text;
+ query text;
+ BEGIN
+ FOR i IN p_start..p_end LOOP
+ cursorname = 'c_'||i;
+ query = format($q$DECLARE %I CURSOR FOR SELECT ctid FROM test_temp WHERE ctid >= '( %s, 1)'::tid $q$, cursorname, i);
+ EXECUTE query;
+ EXECUTE 'FETCH NEXT FROM '||cursorname;
+ -- for test development
+ -- RAISE NOTICE '%: %', cursorname, query;
+ END LOOP;
+ END;
+$f$;
+-- Test overflow of temp table buffers is handled correctly
+BEGIN;
+-- should work, below max
+SELECT test_temp_pin(0, 9);
+ test_temp_pin
+---------------
+
+(1 row)
+
+-- should fail, too many buffers pinned
+SELECT test_temp_pin(10, 105);
+ERROR: no empty local buffer available
+ROLLBACK;
+BEGIN;
+-- have some working cursors to test after errors
+SELECT test_temp_pin(0, 9);
+ test_temp_pin
+---------------
+
+(1 row)
+
+FETCH NEXT FROM c_3;
+ ctid
+-------
+ (3,2)
+(1 row)
+
+-- exhaust buffer pins in subtrans, check things work after
+SAVEPOINT rescue_me;
+SELECT test_temp_pin(10, 105);
+ERROR: no empty local buffer available
+ROLLBACK TO SAVEPOINT rescue_me;
+-- pre-subtrans cursors continue to work
+FETCH NEXT FROM c_3;
+ ctid
+-------
+ (3,3)
+(1 row)
+
+-- new cursors with pins can be created after subtrans rollback
+SELECT test_temp_pin(10, 95);
+ test_temp_pin
+---------------
+
+(1 row)
+
+-- Check that read streams deal with lower number of pins available
+SELECT count(*), max(a) max_a, min(a) min_a, max(cnt) max_cnt FROM test_temp;
+ count | max_a | min_a | max_cnt
+-------+-------+-------+---------
+ 10000 | 10000 | 1 | 0
+(1 row)
+
+ROLLBACK;
+-- Check that temp tables with existing cursors can't be dropped.
+BEGIN;
+SELECT test_temp_pin(0, 1);
+ test_temp_pin
+---------------
+
+(1 row)
+
+DROP TABLE test_temp;
+ERROR: cannot DROP TABLE "test_temp" because it is being used by active queries in this session
+COMMIT;
+-- Check that temp tables with existing cursors can't be dropped.
+BEGIN;
+SELECT test_temp_pin(0, 1);
+ test_temp_pin
+---------------
+
+(1 row)
+
+TRUNCATE test_temp;
+ERROR: cannot TRUNCATE "test_temp" because it is being used by active queries in this session
+COMMIT;
+-- Check that temp tables that are dropped in transaction that's rolled back
+-- preserve buffer contents
+SELECT count(*), max(a) max_a, min(a) min_a, max(cnt) max_cnt FROM test_temp;
+ count | max_a | min_a | max_cnt
+-------+-------+-------+---------
+ 10000 | 10000 | 1 | 0
+(1 row)
+
+INSERT INTO test_temp(a, b, cnt) VALUES (-1, '', 0);
+BEGIN;
+INSERT INTO test_temp(a, b, cnt) VALUES (-2, '', 0);
+DROP TABLE test_temp;
+ROLLBACK;
+SELECT count(*), max(a) max_a, min(a) min_a, max(cnt) max_cnt FROM test_temp;
+ count | max_a | min_a | max_cnt
+-------+-------+-------+---------
+ 10001 | 10000 | -1 | 0
+(1 row)
+
+-- Check that temp table drop is transactional and preserves dirty
+-- buffer contents
+UPDATE test_temp SET cnt = cnt + 1 WHERE a = -1;
+BEGIN;
+DROP TABLE test_temp;
+ROLLBACK;
+SELECT count(*), max(a) max_a, min(a) min_a, max(cnt) max_cnt FROM test_temp;
+ count | max_a | min_a | max_cnt
+-------+-------+-------+---------
+ 10001 | 10000 | -1 | 1
+(1 row)
+
+-- Check that temp table truncation is transactional and preserves dirty
+-- buffer contents
+UPDATE test_temp SET cnt = cnt + 1 WHERE a = -1;
+BEGIN;
+TRUNCATE test_temp;
+ROLLBACK;
+SELECT count(*), max(a) max_a, min(a) min_a, max(cnt) max_cnt FROM test_temp;
+ count | max_a | min_a | max_cnt
+-------+-------+-------+---------
+ 10001 | 10000 | -1 | 2
+(1 row)
+
+-- cleanup
+DROP FUNCTION test_temp_pin(int, int);
diff --git a/src/test/regress/parallel_schedule b/src/test/regress/parallel_schedule
index 37b6d21e1f9..0a35f2f8f6a 100644
--- a/src/test/regress/parallel_schedule
+++ b/src/test/regress/parallel_schedule
@@ -108,7 +108,7 @@ test: json jsonb json_encoding jsonpath jsonpath_encoding jsonb_jsonpath sqljson
# ----------
# Another group of parallel tests
# with depends on create_misc
-# NB: temp.sql does a reconnect which transiently uses 2 connections,
+# NB: temp.sql does reconnects which transiently use 2 connections,
# so keep this parallel group to at most 19 tests
# ----------
test: plancache limit plpgsql copy2 temp domain rangefuncs prepare conversion truncate alter_table sequence polymorphism rowtypes returning largeobject with xml
diff --git a/src/test/regress/sql/temp.sql b/src/test/regress/sql/temp.sql
index 2a487a1ef7f..12091f968de 100644
--- a/src/test/regress/sql/temp.sql
+++ b/src/test/regress/sql/temp.sql
@@ -311,3 +311,110 @@ SET search_path TO 'pg_temp';
BEGIN;
SELECT current_schema() ~ 'pg_temp' AS is_temp_schema;
PREPARE TRANSACTION 'twophase_search';
+
+
+-- Tests to verify we recover correctly from exhausting buffer pins and
+-- related matters.
+
+-- use lower possible buffer limit to make the test cheaper
+\c
+SET temp_buffers = 100;
+
+CREATE TEMPORARY TABLE test_temp(a int not null unique, b TEXT not null, cnt int not null);
+INSERT INTO test_temp SELECT generate_series(1, 10000) as id, repeat('a', 200), 0;
+-- should be at least 2x as large than temp_buffers
+SELECT pg_relation_size('test_temp') / current_setting('block_size')::int8 > 200;
+
+-- Don't want cursor names and plpgsql function lines in the error messages
+\set VERBOSITY terse
+
+/* helper function to create cursors for each page in [p_start, p_end] */
+CREATE FUNCTION test_temp_pin(p_start int, p_end int)
+RETURNS void
+LANGUAGE plpgsql
+AS $f$
+ DECLARE
+ cursorname text;
+ query text;
+ BEGIN
+ FOR i IN p_start..p_end LOOP
+ cursorname = 'c_'||i;
+ query = format($q$DECLARE %I CURSOR FOR SELECT ctid FROM test_temp WHERE ctid >= '( %s, 1)'::tid $q$, cursorname, i);
+ EXECUTE query;
+ EXECUTE 'FETCH NEXT FROM '||cursorname;
+ -- for test development
+ -- RAISE NOTICE '%: %', cursorname, query;
+ END LOOP;
+ END;
+$f$;
+
+
+-- Test overflow of temp table buffers is handled correctly
+BEGIN;
+-- should work, below max
+SELECT test_temp_pin(0, 9);
+-- should fail, too many buffers pinned
+SELECT test_temp_pin(10, 105);
+ROLLBACK;
+
+BEGIN;
+-- have some working cursors to test after errors
+SELECT test_temp_pin(0, 9);
+FETCH NEXT FROM c_3;
+-- exhaust buffer pins in subtrans, check things work after
+SAVEPOINT rescue_me;
+SELECT test_temp_pin(10, 105);
+ROLLBACK TO SAVEPOINT rescue_me;
+-- pre-subtrans cursors continue to work
+FETCH NEXT FROM c_3;
+
+-- new cursors with pins can be created after subtrans rollback
+SELECT test_temp_pin(10, 95);
+
+-- Check that read streams deal with lower number of pins available
+SELECT count(*), max(a) max_a, min(a) min_a, max(cnt) max_cnt FROM test_temp;
+
+ROLLBACK;
+
+
+-- Check that temp tables with existing cursors can't be dropped.
+BEGIN;
+SELECT test_temp_pin(0, 1);
+DROP TABLE test_temp;
+COMMIT;
+
+-- Check that temp tables with existing cursors can't be dropped.
+BEGIN;
+SELECT test_temp_pin(0, 1);
+TRUNCATE test_temp;
+COMMIT;
+
+-- Check that temp tables that are dropped in transaction that's rolled back
+-- preserve buffer contents
+SELECT count(*), max(a) max_a, min(a) min_a, max(cnt) max_cnt FROM test_temp;
+INSERT INTO test_temp(a, b, cnt) VALUES (-1, '', 0);
+BEGIN;
+INSERT INTO test_temp(a, b, cnt) VALUES (-2, '', 0);
+DROP TABLE test_temp;
+ROLLBACK;
+SELECT count(*), max(a) max_a, min(a) min_a, max(cnt) max_cnt FROM test_temp;
+
+-- Check that temp table drop is transactional and preserves dirty
+-- buffer contents
+UPDATE test_temp SET cnt = cnt + 1 WHERE a = -1;
+BEGIN;
+DROP TABLE test_temp;
+ROLLBACK;
+SELECT count(*), max(a) max_a, min(a) min_a, max(cnt) max_cnt FROM test_temp;
+
+-- Check that temp table truncation is transactional and preserves dirty
+-- buffer contents
+UPDATE test_temp SET cnt = cnt + 1 WHERE a = -1;
+BEGIN;
+TRUNCATE test_temp;
+ROLLBACK;
+SELECT count(*), max(a) max_a, min(a) min_a, max(cnt) max_cnt FROM test_temp;
+
+
+-- cleanup
+DROP FUNCTION test_temp_pin(int, int);
--
2.48.1.76.g4e746b1a31.dirty
[text/x-diff] v2.8-0016-localbuf-Fix-dangerous-coding-pattern-in-GetLoc.patch (2.9K, ../../3yxd5r23zly5bytvgyktbxtxq2r3gbpi7xd4dugevh3h4w4q6c@lu6oatjjpltz/17-v2.8-0016-localbuf-Fix-dangerous-coding-pattern-in-GetLoc.patch)
download | inline diff:
From a096e1ccb3e2e175053070e27f2c0b02061b8c0d Mon Sep 17 00:00:00 2001
From: Andres Freund <andres@anarazel.de>
Date: Sun, 9 Mar 2025 20:11:51 -0400
Subject: [PATCH v2.8 16/38] localbuf: Fix dangerous coding pattern in
GetLocalVictimBuffer()
If PinLocalBuffer() were to modify the buf_state, the buf_state in
GetLocalVictimBuffer() would be out of date. Currently that does not happen,
as PinLocalBuffer() only modifies the buf_state if adjust_usagecount=true and
GetLocalVictimBuffer() passes false.
However, it's easy to make this not the case anymore - it cost me a few hours
to debug the consequences.
The minimal fix would be to just refetch the buf_state after after calling
PinLocalBuffer(), but the same danger exists in later parts of the
function. Instead, declare buf_state in the narrower scopes and re-read the
state in conditional branches. Besides being safer, it also fits well with
an upcoming set of cleanup patches that move the contents of the conditional
branches in GetLocalVictimBuffer() into helper functions.
I "broke" this in 794f2594479.
Arguably this should be backpatched, but as the relevant functions are not
exported and there is no actual misbehaviour, I chose to not backpatch, at
least for now.
Reviewed-by: Melanie Plageman <melanieplageman@gmail.com>
Discussion: https://postgr.es/m/CAAKRu_b9anbWzEs5AAF9WCvcEVmgz-1AkHSQ-CLLy-p7WHzvFw@mail.gmail.com
---
src/backend/storage/buffer/localbuf.c | 9 +++++----
1 file changed, 5 insertions(+), 4 deletions(-)
diff --git a/src/backend/storage/buffer/localbuf.c b/src/backend/storage/buffer/localbuf.c
index 919f600ed41..456a2232f22 100644
--- a/src/backend/storage/buffer/localbuf.c
+++ b/src/backend/storage/buffer/localbuf.c
@@ -178,7 +178,6 @@ GetLocalVictimBuffer(void)
{
int victim_bufid;
int trycounter;
- uint32 buf_state;
BufferDesc *bufHdr;
ResourceOwnerEnlarge(CurrentResourceOwner);
@@ -199,7 +198,7 @@ GetLocalVictimBuffer(void)
if (LocalRefCount[victim_bufid] == 0)
{
- buf_state = pg_atomic_read_u32(&bufHdr->state);
+ uint32 buf_state = pg_atomic_read_u32(&bufHdr->state);
if (BUF_STATE_GET_USAGECOUNT(buf_state) > 0)
{
@@ -233,8 +232,9 @@ GetLocalVictimBuffer(void)
* this buffer is not referenced but it might still be dirty. if that's
* the case, write it out before reusing it!
*/
- if (buf_state & BM_DIRTY)
+ if (pg_atomic_read_u32(&bufHdr->state) & BM_DIRTY)
{
+ uint32 buf_state = pg_atomic_read_u32(&bufHdr->state);
instr_time io_start;
SMgrRelation oreln;
Page localpage = (char *) LocalBufHdrGetBlock(bufHdr);
@@ -267,8 +267,9 @@ GetLocalVictimBuffer(void)
/*
* Remove the victim buffer from the hashtable and mark as invalid.
*/
- if (buf_state & BM_TAG_VALID)
+ if (pg_atomic_read_u32(&bufHdr->state) & BM_TAG_VALID)
{
+ uint32 buf_state = pg_atomic_read_u32(&bufHdr->state);
LocalBufferLookupEnt *hresult;
hresult = (LocalBufferLookupEnt *)
--
2.48.1.76.g4e746b1a31.dirty
[text/x-diff] v2.8-0017-localbuf-Introduce-InvalidateLocalBuffer.patch (6.0K, ../../3yxd5r23zly5bytvgyktbxtxq2r3gbpi7xd4dugevh3h4w4q6c@lu6oatjjpltz/18-v2.8-0017-localbuf-Introduce-InvalidateLocalBuffer.patch)
download | inline diff:
From 896913e86ab7789ff24f890c99b3c2c8843f0e9a Mon Sep 17 00:00:00 2001
From: Andres Freund <andres@anarazel.de>
Date: Sun, 9 Mar 2025 20:25:22 -0400
Subject: [PATCH v2.8 17/38] localbuf: Introduce InvalidateLocalBuffer()
Previously, there were three copies of this code, two of them
identical. There's no good reason for that.
This change is nice on its own, but the main motivation is the AIO patchset,
which needs to add extra checks the deduplicated code, which of course is
easier if there is only one version.
Reviewed-by: Melanie Plageman <melanieplageman@gmail.com>
Discussion: https://postgr.es/m/CAAKRu_b9anbWzEs5AAF9WCvcEVmgz-1AkHSQ-CLLy-p7WHzvFw@mail.gmail.com
---
src/backend/storage/buffer/localbuf.c | 92 +++++++++++++--------------
1 file changed, 44 insertions(+), 48 deletions(-)
diff --git a/src/backend/storage/buffer/localbuf.c b/src/backend/storage/buffer/localbuf.c
index 456a2232f22..5331091132d 100644
--- a/src/backend/storage/buffer/localbuf.c
+++ b/src/backend/storage/buffer/localbuf.c
@@ -56,6 +56,7 @@ static int NLocalPinnedBuffers = 0;
static void InitLocalBuffers(void);
static Block GetLocalBufferStorage(void);
static Buffer GetLocalVictimBuffer(void);
+static void InvalidateLocalBuffer(BufferDesc *bufHdr, bool check_unreferenced);
/*
@@ -269,17 +270,7 @@ GetLocalVictimBuffer(void)
*/
if (pg_atomic_read_u32(&bufHdr->state) & BM_TAG_VALID)
{
- uint32 buf_state = pg_atomic_read_u32(&bufHdr->state);
- LocalBufferLookupEnt *hresult;
-
- hresult = (LocalBufferLookupEnt *)
- hash_search(LocalBufHash, &bufHdr->tag, HASH_REMOVE, NULL);
- if (!hresult) /* shouldn't happen */
- elog(ERROR, "local buffer hash table corrupted");
- /* mark buffer invalid just in case hash insert fails */
- ClearBufferTag(&bufHdr->tag);
- buf_state &= ~(BUF_FLAG_MASK | BUF_USAGECOUNT_MASK);
- pg_atomic_unlocked_write_u32(&bufHdr->state, buf_state);
+ InvalidateLocalBuffer(bufHdr, false);
pgstat_count_io_op(IOOBJECT_TEMP_RELATION, IOCONTEXT_NORMAL, IOOP_EVICT, 1, 0);
}
@@ -492,6 +483,46 @@ MarkLocalBufferDirty(Buffer buffer)
pg_atomic_unlocked_write_u32(&bufHdr->state, buf_state);
}
+/*
+ * InvalidateLocalBuffer -- mark a local buffer invalid.
+ *
+ * If check_unreferenced is true, error out if the buffer is still
+ * pinned. Passing false is appropriate when calling InvalidateLocalBuffer()
+ * as part of changing the identity of a buffer, instead of just dropping the
+ * buffer.
+ *
+ * See also InvalidateBuffer().
+ */
+static void
+InvalidateLocalBuffer(BufferDesc *bufHdr, bool check_unreferenced)
+{
+ Buffer buffer = BufferDescriptorGetBuffer(bufHdr);
+ int bufid = -buffer - 1;
+ uint32 buf_state;
+ LocalBufferLookupEnt *hresult;
+
+ buf_state = pg_atomic_read_u32(&bufHdr->state);
+
+ if (check_unreferenced && LocalRefCount[bufid] != 0)
+ elog(ERROR, "block %u of %s is still referenced (local %u)",
+ bufHdr->tag.blockNum,
+ relpathbackend(BufTagGetRelFileLocator(&bufHdr->tag),
+ MyProcNumber,
+ BufTagGetForkNum(&bufHdr->tag)).str,
+ LocalRefCount[bufid]);
+
+ /* Remove entry from hashtable */
+ hresult = (LocalBufferLookupEnt *)
+ hash_search(LocalBufHash, &bufHdr->tag, HASH_REMOVE, NULL);
+ if (!hresult) /* shouldn't happen */
+ elog(ERROR, "local buffer hash table corrupted");
+ /* Mark buffer invalid */
+ ClearBufferTag(&bufHdr->tag);
+ buf_state &= ~BUF_FLAG_MASK;
+ buf_state &= ~BUF_USAGECOUNT_MASK;
+ pg_atomic_unlocked_write_u32(&bufHdr->state, buf_state);
+}
+
/*
* DropRelationLocalBuffers
* This function removes from the buffer pool all the pages of the
@@ -512,7 +543,6 @@ DropRelationLocalBuffers(RelFileLocator rlocator, ForkNumber forkNum,
for (i = 0; i < NLocBuffer; i++)
{
BufferDesc *bufHdr = GetLocalBufferDescriptor(i);
- LocalBufferLookupEnt *hresult;
uint32 buf_state;
buf_state = pg_atomic_read_u32(&bufHdr->state);
@@ -522,24 +552,7 @@ DropRelationLocalBuffers(RelFileLocator rlocator, ForkNumber forkNum,
BufTagGetForkNum(&bufHdr->tag) == forkNum &&
bufHdr->tag.blockNum >= firstDelBlock)
{
- if (LocalRefCount[i] != 0)
- elog(ERROR, "block %u of %s is still referenced (local %u)",
- bufHdr->tag.blockNum,
- relpathbackend(BufTagGetRelFileLocator(&bufHdr->tag),
- MyProcNumber,
- BufTagGetForkNum(&bufHdr->tag)).str,
- LocalRefCount[i]);
-
- /* Remove entry from hashtable */
- hresult = (LocalBufferLookupEnt *)
- hash_search(LocalBufHash, &bufHdr->tag, HASH_REMOVE, NULL);
- if (!hresult) /* shouldn't happen */
- elog(ERROR, "local buffer hash table corrupted");
- /* Mark buffer invalid */
- ClearBufferTag(&bufHdr->tag);
- buf_state &= ~BUF_FLAG_MASK;
- buf_state &= ~BUF_USAGECOUNT_MASK;
- pg_atomic_unlocked_write_u32(&bufHdr->state, buf_state);
+ InvalidateLocalBuffer(bufHdr, true);
}
}
}
@@ -559,7 +572,6 @@ DropRelationAllLocalBuffers(RelFileLocator rlocator)
for (i = 0; i < NLocBuffer; i++)
{
BufferDesc *bufHdr = GetLocalBufferDescriptor(i);
- LocalBufferLookupEnt *hresult;
uint32 buf_state;
buf_state = pg_atomic_read_u32(&bufHdr->state);
@@ -567,23 +579,7 @@ DropRelationAllLocalBuffers(RelFileLocator rlocator)
if ((buf_state & BM_TAG_VALID) &&
BufTagMatchesRelFileLocator(&bufHdr->tag, &rlocator))
{
- if (LocalRefCount[i] != 0)
- elog(ERROR, "block %u of %s is still referenced (local %u)",
- bufHdr->tag.blockNum,
- relpathbackend(BufTagGetRelFileLocator(&bufHdr->tag),
- MyProcNumber,
- BufTagGetForkNum(&bufHdr->tag)).str,
- LocalRefCount[i]);
- /* Remove entry from hashtable */
- hresult = (LocalBufferLookupEnt *)
- hash_search(LocalBufHash, &bufHdr->tag, HASH_REMOVE, NULL);
- if (!hresult) /* shouldn't happen */
- elog(ERROR, "local buffer hash table corrupted");
- /* Mark buffer invalid */
- ClearBufferTag(&bufHdr->tag);
- buf_state &= ~BUF_FLAG_MASK;
- buf_state &= ~BUF_USAGECOUNT_MASK;
- pg_atomic_unlocked_write_u32(&bufHdr->state, buf_state);
+ InvalidateLocalBuffer(bufHdr, true);
}
}
}
--
2.48.1.76.g4e746b1a31.dirty
[text/x-diff] v2.8-0018-localbuf-Introduce-TerminateLocalBufferIO.patch (5.8K, ../../3yxd5r23zly5bytvgyktbxtxq2r3gbpi7xd4dugevh3h4w4q6c@lu6oatjjpltz/19-v2.8-0018-localbuf-Introduce-TerminateLocalBufferIO.patch)
download | inline diff:
From 109d2a3b679eb0f8ee9dd25fe6cd885f79f34e9c Mon Sep 17 00:00:00 2001
From: Andres Freund <andres@anarazel.de>
Date: Sun, 9 Mar 2025 19:55:05 -0400
Subject: [PATCH v2.8 18/38] localbuf: Introduce TerminateLocalBufferIO()
Previously TerminateLocalBufferIO() was open-coded in multiple places, which
doesn't seem like a great idea. While TerminateLocalBufferIO() currently is
rather simple, an upcoming patch requires additional code to be added to
TerminateLocalBufferIO(), making this modification particularly worthwhile.
For some reason FlushRelationBuffers() previously cleared BM_JUST_DIRTIED,
even though that's never set for temporary buffers. This is not carried over
as part of this change.
Reviewed-by: Melanie Plageman <melanieplageman@gmail.com>
Discussion: https://postgr.es/m/CAAKRu_b9anbWzEs5AAF9WCvcEVmgz-1AkHSQ-CLLy-p7WHzvFw@mail.gmail.com
---
src/include/storage/buf_internals.h | 2 ++
src/backend/storage/buffer/bufmgr.c | 32 ++++++++-------------------
src/backend/storage/buffer/localbuf.c | 29 +++++++++++++++++++++---
3 files changed, 37 insertions(+), 26 deletions(-)
diff --git a/src/include/storage/buf_internals.h b/src/include/storage/buf_internals.h
index 8b32fb108b0..4611a60d3e0 100644
--- a/src/include/storage/buf_internals.h
+++ b/src/include/storage/buf_internals.h
@@ -471,6 +471,8 @@ extern BlockNumber ExtendBufferedRelLocal(BufferManagerRelation bmr,
Buffer *buffers,
uint32 *extended_by);
extern void MarkLocalBufferDirty(Buffer buffer);
+extern void TerminateLocalBufferIO(BufferDesc *bufHdr, bool clear_dirty,
+ uint32 set_flag_bits);
extern void DropRelationLocalBuffers(RelFileLocator rlocator,
ForkNumber forkNum,
BlockNumber firstDelBlock);
diff --git a/src/backend/storage/buffer/bufmgr.c b/src/backend/storage/buffer/bufmgr.c
index ef519b4c356..d2e7c283179 100644
--- a/src/backend/storage/buffer/bufmgr.c
+++ b/src/backend/storage/buffer/bufmgr.c
@@ -1072,19 +1072,11 @@ ZeroAndLockBuffer(Buffer buffer, ReadBufferMode mode, bool already_valid)
if (!isLocalBuf)
LWLockAcquire(BufferDescriptorGetContentLock(bufHdr), LW_EXCLUSIVE);
+ /* Set BM_VALID, terminate IO, and wake up any waiters */
if (isLocalBuf)
- {
- /* Only need to adjust flags */
- uint32 buf_state = pg_atomic_read_u32(&bufHdr->state);
-
- buf_state |= BM_VALID;
- pg_atomic_unlocked_write_u32(&bufHdr->state, buf_state);
- }
+ TerminateLocalBufferIO(bufHdr, false, BM_VALID);
else
- {
- /* Set BM_VALID, terminate IO, and wake up any waiters */
TerminateBufferIO(bufHdr, false, BM_VALID, true);
- }
}
else if (!isLocalBuf)
{
@@ -1608,19 +1600,11 @@ WaitReadBuffers(ReadBuffersOperation *operation)
relpath(operation->smgr->smgr_rlocator, forknum).str)));
}
- /* Terminate I/O and set BM_VALID. */
+ /* Set BM_VALID, terminate IO, and wake up any waiters */
if (persistence == RELPERSISTENCE_TEMP)
- {
- uint32 buf_state = pg_atomic_read_u32(&bufHdr->state);
-
- buf_state |= BM_VALID;
- pg_atomic_unlocked_write_u32(&bufHdr->state, buf_state);
- }
+ TerminateLocalBufferIO(bufHdr, false, BM_VALID);
else
- {
- /* Set BM_VALID, terminate IO, and wake up any waiters */
TerminateBufferIO(bufHdr, false, BM_VALID, true);
- }
/* Report I/Os as completing individually. */
TRACE_POSTGRESQL_BUFFER_READ_DONE(forknum, io_first_block + j,
@@ -4555,8 +4539,7 @@ FlushRelationBuffers(Relation rel)
IOCONTEXT_NORMAL, IOOP_WRITE,
io_start, 1, BLCKSZ);
- buf_state &= ~(BM_DIRTY | BM_JUST_DIRTIED);
- pg_atomic_unlocked_write_u32(&bufHdr->state, buf_state);
+ TerminateLocalBufferIO(bufHdr, true, 0);
pgBufferUsage.local_blks_written++;
@@ -5643,8 +5626,11 @@ TerminateBufferIO(BufferDesc *buf, bool clear_dirty, uint32 set_flag_bits,
buf_state = LockBufHdr(buf);
Assert(buf_state & BM_IO_IN_PROGRESS);
+ buf_state &= ~BM_IO_IN_PROGRESS;
+
+ /* Clear earlier errors, if this IO failed, it'll be marked again */
+ buf_state &= ~BM_IO_ERROR;
- buf_state &= ~(BM_IO_IN_PROGRESS | BM_IO_ERROR);
if (clear_dirty && !(buf_state & BM_JUST_DIRTIED))
buf_state &= ~(BM_DIRTY | BM_CHECKPOINT_NEEDED);
diff --git a/src/backend/storage/buffer/localbuf.c b/src/backend/storage/buffer/localbuf.c
index 5331091132d..86b1c4c7c68 100644
--- a/src/backend/storage/buffer/localbuf.c
+++ b/src/backend/storage/buffer/localbuf.c
@@ -235,7 +235,6 @@ GetLocalVictimBuffer(void)
*/
if (pg_atomic_read_u32(&bufHdr->state) & BM_DIRTY)
{
- uint32 buf_state = pg_atomic_read_u32(&bufHdr->state);
instr_time io_start;
SMgrRelation oreln;
Page localpage = (char *) LocalBufHdrGetBlock(bufHdr);
@@ -259,8 +258,7 @@ GetLocalVictimBuffer(void)
IOOP_WRITE, io_start, 1, BLCKSZ);
/* Mark not-dirty now in case we error out below */
- buf_state &= ~BM_DIRTY;
- pg_atomic_unlocked_write_u32(&bufHdr->state, buf_state);
+ TerminateLocalBufferIO(bufHdr, true, 0);
pgBufferUsage.local_blks_written++;
}
@@ -483,6 +481,31 @@ MarkLocalBufferDirty(Buffer buffer)
pg_atomic_unlocked_write_u32(&bufHdr->state, buf_state);
}
+/*
+ * Like TerminateBufferIO, but for local buffers
+ */
+void
+TerminateLocalBufferIO(BufferDesc *bufHdr, bool clear_dirty, uint32 set_flag_bits)
+{
+ /* Only need to adjust flags */
+ uint32 buf_state = pg_atomic_read_u32(&bufHdr->state);
+
+ /* BM_IO_IN_PROGRESS isn't currently used for local buffers */
+
+ /* Clear earlier errors, if this IO failed, it'll be marked again */
+ buf_state &= ~BM_IO_ERROR;
+
+ if (clear_dirty)
+ buf_state &= ~BM_DIRTY;
+
+ buf_state |= set_flag_bits;
+ pg_atomic_unlocked_write_u32(&bufHdr->state, buf_state);
+
+ /* local buffers don't track IO using resowners */
+
+ /* local buffers don't use the IO CV, as no other process can see buffer */
+}
+
/*
* InvalidateLocalBuffer -- mark a local buffer invalid.
*
--
2.48.1.76.g4e746b1a31.dirty
[text/x-diff] v2.8-0019-localbuf-Introduce-FlushLocalBuffer.patch (5.4K, ../../3yxd5r23zly5bytvgyktbxtxq2r3gbpi7xd4dugevh3h4w4q6c@lu6oatjjpltz/20-v2.8-0019-localbuf-Introduce-FlushLocalBuffer.patch)
download | inline diff:
From 1f62831d5e559777753fc090e0b63c01d6f88086 Mon Sep 17 00:00:00 2001
From: Andres Freund <andres@anarazel.de>
Date: Sun, 9 Mar 2025 18:33:35 -0400
Subject: [PATCH v2.8 19/38] localbuf: Introduce FlushLocalBuffer()
Previously we had two paths implementing writing out temporary table
buffers. For shared buffers, the logic for that is centralized in
FlushBuffer(). Introduce FlushLocalBuffer() to do the same for local buffers.
Besides being a nice cleanup on its own, it also makes an upcoming change
slightly easier.
Reviewed-by: Melanie Plageman <melanieplageman@gmail.com>
Discussion: https://postgr.es/m/CAAKRu_b9anbWzEs5AAF9WCvcEVmgz-1AkHSQ-CLLy-p7WHzvFw@mail.gmail.com
---
src/include/storage/buf_internals.h | 1 +
src/backend/storage/buffer/bufmgr.c | 22 +--------
src/backend/storage/buffer/localbuf.c | 64 +++++++++++++++------------
3 files changed, 38 insertions(+), 49 deletions(-)
diff --git a/src/include/storage/buf_internals.h b/src/include/storage/buf_internals.h
index 4611a60d3e0..90bc7e0db7b 100644
--- a/src/include/storage/buf_internals.h
+++ b/src/include/storage/buf_internals.h
@@ -473,6 +473,7 @@ extern BlockNumber ExtendBufferedRelLocal(BufferManagerRelation bmr,
extern void MarkLocalBufferDirty(Buffer buffer);
extern void TerminateLocalBufferIO(BufferDesc *bufHdr, bool clear_dirty,
uint32 set_flag_bits);
+extern void FlushLocalBuffer(BufferDesc *bufHdr, SMgrRelation reln);
extern void DropRelationLocalBuffers(RelFileLocator rlocator,
ForkNumber forkNum,
BlockNumber firstDelBlock);
diff --git a/src/backend/storage/buffer/bufmgr.c b/src/backend/storage/buffer/bufmgr.c
index d2e7c283179..4a7cf3f3cc4 100644
--- a/src/backend/storage/buffer/bufmgr.c
+++ b/src/backend/storage/buffer/bufmgr.c
@@ -4507,7 +4507,6 @@ FlushRelationBuffers(Relation rel)
for (i = 0; i < NLocBuffer; i++)
{
uint32 buf_state;
- instr_time io_start;
bufHdr = GetLocalBufferDescriptor(i);
if (BufTagMatchesRelFileLocator(&bufHdr->tag, &rel->rd_locator) &&
@@ -4515,9 +4514,6 @@ FlushRelationBuffers(Relation rel)
(BM_VALID | BM_DIRTY)) == (BM_VALID | BM_DIRTY))
{
ErrorContextCallback errcallback;
- Page localpage;
-
- localpage = (char *) LocalBufHdrGetBlock(bufHdr);
/* Setup error traceback support for ereport() */
errcallback.callback = local_buffer_write_error_callback;
@@ -4525,23 +4521,7 @@ FlushRelationBuffers(Relation rel)
errcallback.previous = error_context_stack;
error_context_stack = &errcallback;
- PageSetChecksumInplace(localpage, bufHdr->tag.blockNum);
-
- io_start = pgstat_prepare_io_time(track_io_timing);
-
- smgrwrite(srel,
- BufTagGetForkNum(&bufHdr->tag),
- bufHdr->tag.blockNum,
- localpage,
- false);
-
- pgstat_count_io_op_time(IOOBJECT_TEMP_RELATION,
- IOCONTEXT_NORMAL, IOOP_WRITE,
- io_start, 1, BLCKSZ);
-
- TerminateLocalBufferIO(bufHdr, true, 0);
-
- pgBufferUsage.local_blks_written++;
+ FlushLocalBuffer(bufHdr, srel);
/* Pop the error context stack */
error_context_stack = errcallback.previous;
diff --git a/src/backend/storage/buffer/localbuf.c b/src/backend/storage/buffer/localbuf.c
index 86b1c4c7c68..8efde05c0a5 100644
--- a/src/backend/storage/buffer/localbuf.c
+++ b/src/backend/storage/buffer/localbuf.c
@@ -174,6 +174,41 @@ LocalBufferAlloc(SMgrRelation smgr, ForkNumber forkNum, BlockNumber blockNum,
return bufHdr;
}
+/*
+ * Like FlushBuffer(), just for local buffers.
+ */
+void
+FlushLocalBuffer(BufferDesc *bufHdr, SMgrRelation reln)
+{
+ instr_time io_start;
+ Page localpage = (char *) LocalBufHdrGetBlock(bufHdr);
+
+ /* Find smgr relation for buffer */
+ if (reln == NULL)
+ reln = smgropen(BufTagGetRelFileLocator(&bufHdr->tag),
+ MyProcNumber);
+
+ PageSetChecksumInplace(localpage, bufHdr->tag.blockNum);
+
+ io_start = pgstat_prepare_io_time(track_io_timing);
+
+ /* And write... */
+ smgrwrite(reln,
+ BufTagGetForkNum(&bufHdr->tag),
+ bufHdr->tag.blockNum,
+ localpage,
+ false);
+
+ /* Temporary table I/O does not use Buffer Access Strategies */
+ pgstat_count_io_op_time(IOOBJECT_TEMP_RELATION, IOCONTEXT_NORMAL,
+ IOOP_WRITE, io_start, 1, BLCKSZ);
+
+ /* Mark not-dirty */
+ TerminateLocalBufferIO(bufHdr, true, 0);
+
+ pgBufferUsage.local_blks_written++;
+}
+
static Buffer
GetLocalVictimBuffer(void)
{
@@ -234,34 +269,7 @@ GetLocalVictimBuffer(void)
* the case, write it out before reusing it!
*/
if (pg_atomic_read_u32(&bufHdr->state) & BM_DIRTY)
- {
- instr_time io_start;
- SMgrRelation oreln;
- Page localpage = (char *) LocalBufHdrGetBlock(bufHdr);
-
- /* Find smgr relation for buffer */
- oreln = smgropen(BufTagGetRelFileLocator(&bufHdr->tag), MyProcNumber);
-
- PageSetChecksumInplace(localpage, bufHdr->tag.blockNum);
-
- io_start = pgstat_prepare_io_time(track_io_timing);
-
- /* And write... */
- smgrwrite(oreln,
- BufTagGetForkNum(&bufHdr->tag),
- bufHdr->tag.blockNum,
- localpage,
- false);
-
- /* Temporary table I/O does not use Buffer Access Strategies */
- pgstat_count_io_op_time(IOOBJECT_TEMP_RELATION, IOCONTEXT_NORMAL,
- IOOP_WRITE, io_start, 1, BLCKSZ);
-
- /* Mark not-dirty now in case we error out below */
- TerminateLocalBufferIO(bufHdr, true, 0);
-
- pgBufferUsage.local_blks_written++;
- }
+ FlushLocalBuffer(bufHdr, NULL);
/*
* Remove the victim buffer from the hashtable and mark as invalid.
--
2.48.1.76.g4e746b1a31.dirty
[text/x-diff] v2.8-0020-localbuf-Introduce-StartLocalBufferIO.patch (4.9K, ../../3yxd5r23zly5bytvgyktbxtxq2r3gbpi7xd4dugevh3h4w4q6c@lu6oatjjpltz/21-v2.8-0020-localbuf-Introduce-StartLocalBufferIO.patch)
download | inline diff:
From 4460435cd2170123f2f316fbaed2b1bd8578393e Mon Sep 17 00:00:00 2001
From: Andres Freund <andres@anarazel.de>
Date: Sun, 9 Mar 2025 18:44:36 -0400
Subject: [PATCH v2.8 20/38] localbuf: Introduce StartLocalBufferIO()
To initiate IO on a shared buffer we have StartBufferIO(). For temporary table
buffers no similar function exists - likely because the code for that
currently is very simple due to the lack of concurrency.
However, the upcoming AIO support will make it possible to re-encounter a
local buffer, while the buffer already is the target of IO. In that case we
need to wait for already in-progress IO to complete. This commit makes it
easier to add the necessary code, by introducing StartLocalBufferIO().
Reviewed-by: Melanie Plageman <melanieplageman@gmail.com>
Discussion: https://postgr.es/m/CAAKRu_b9anbWzEs5AAF9WCvcEVmgz-1AkHSQ-CLLy-p7WHzvFw@mail.gmail.com
---
src/include/storage/buf_internals.h | 1 +
src/backend/storage/buffer/bufmgr.c | 8 ++----
src/backend/storage/buffer/localbuf.c | 36 +++++++++++++++++++++++++++
3 files changed, 39 insertions(+), 6 deletions(-)
diff --git a/src/include/storage/buf_internals.h b/src/include/storage/buf_internals.h
index 90bc7e0db7b..9327f60c44c 100644
--- a/src/include/storage/buf_internals.h
+++ b/src/include/storage/buf_internals.h
@@ -473,6 +473,7 @@ extern BlockNumber ExtendBufferedRelLocal(BufferManagerRelation bmr,
extern void MarkLocalBufferDirty(Buffer buffer);
extern void TerminateLocalBufferIO(BufferDesc *bufHdr, bool clear_dirty,
uint32 set_flag_bits);
+extern bool StartLocalBufferIO(BufferDesc *bufHdr, bool forInput);
extern void FlushLocalBuffer(BufferDesc *bufHdr, SMgrRelation reln);
extern void DropRelationLocalBuffers(RelFileLocator rlocator,
ForkNumber forkNum,
diff --git a/src/backend/storage/buffer/bufmgr.c b/src/backend/storage/buffer/bufmgr.c
index 4a7cf3f3cc4..95911b51d58 100644
--- a/src/backend/storage/buffer/bufmgr.c
+++ b/src/backend/storage/buffer/bufmgr.c
@@ -1038,7 +1038,7 @@ ZeroAndLockBuffer(Buffer buffer, ReadBufferMode mode, bool already_valid)
{
/* Simple case for non-shared buffers. */
bufHdr = GetLocalBufferDescriptor(-buffer - 1);
- need_to_zero = (pg_atomic_read_u32(&bufHdr->state) & BM_VALID) == 0;
+ need_to_zero = StartLocalBufferIO(bufHdr, true);
}
else
{
@@ -1450,11 +1450,7 @@ static inline bool
WaitReadBuffersCanStartIO(Buffer buffer, bool nowait)
{
if (BufferIsLocal(buffer))
- {
- BufferDesc *bufHdr = GetLocalBufferDescriptor(-buffer - 1);
-
- return (pg_atomic_read_u32(&bufHdr->state) & BM_VALID) == 0;
- }
+ return StartLocalBufferIO(GetLocalBufferDescriptor(-buffer - 1), true);
else
return StartBufferIO(GetBufferDescriptor(buffer - 1), true, nowait);
}
diff --git a/src/backend/storage/buffer/localbuf.c b/src/backend/storage/buffer/localbuf.c
index 8efde05c0a5..f172a5c7820 100644
--- a/src/backend/storage/buffer/localbuf.c
+++ b/src/backend/storage/buffer/localbuf.c
@@ -183,6 +183,13 @@ FlushLocalBuffer(BufferDesc *bufHdr, SMgrRelation reln)
instr_time io_start;
Page localpage = (char *) LocalBufHdrGetBlock(bufHdr);
+ /*
+ * Try to start an I/O operation. There currently are no reasons for
+ * StartLocalBufferIO to return false, so we raise an error in that case.
+ */
+ if (!StartLocalBufferIO(bufHdr, false))
+ elog(ERROR, "failed to start write IO on local buffer");
+
/* Find smgr relation for buffer */
if (reln == NULL)
reln = smgropen(BufTagGetRelFileLocator(&bufHdr->tag),
@@ -406,11 +413,17 @@ ExtendBufferedRelLocal(BufferManagerRelation bmr,
PinLocalBuffer(existing_hdr, false);
buffers[i] = BufferDescriptorGetBuffer(existing_hdr);
+ /*
+ * Clear the BM_VALID bit, do StartLocalBufferIO() and proceed.
+ */
buf_state = pg_atomic_read_u32(&existing_hdr->state);
Assert(buf_state & BM_TAG_VALID);
Assert(!(buf_state & BM_DIRTY));
buf_state &= ~BM_VALID;
pg_atomic_unlocked_write_u32(&existing_hdr->state, buf_state);
+
+ /* no need to loop for local buffers */
+ StartLocalBufferIO(existing_hdr, true);
}
else
{
@@ -425,6 +438,8 @@ ExtendBufferedRelLocal(BufferManagerRelation bmr,
pg_atomic_unlocked_write_u32(&victim_buf_hdr->state, buf_state);
hresult->id = victim_buf_id;
+
+ StartLocalBufferIO(victim_buf_hdr, true);
}
}
@@ -489,6 +504,27 @@ MarkLocalBufferDirty(Buffer buffer)
pg_atomic_unlocked_write_u32(&bufHdr->state, buf_state);
}
+/*
+ * Like StartBufferIO, but for local buffers
+ */
+bool
+StartLocalBufferIO(BufferDesc *bufHdr, bool forInput)
+{
+ uint32 buf_state = pg_atomic_read_u32(&bufHdr->state);
+
+ if (forInput ? (buf_state & BM_VALID) : !(buf_state & BM_DIRTY))
+ {
+ /* someone else already did the I/O */
+ return false;
+ }
+
+ /* BM_IO_IN_PROGRESS isn't currently used for local buffers */
+
+ /* local buffers don't track IO using resowners */
+
+ return true;
+}
+
/*
* Like TerminateBufferIO, but for local buffers
*/
--
2.48.1.76.g4e746b1a31.dirty
[text/x-diff] v2.8-0021-localbuf-Track-pincount-in-BufferDesc-as-well.patch (4.5K, ../../3yxd5r23zly5bytvgyktbxtxq2r3gbpi7xd4dugevh3h4w4q6c@lu6oatjjpltz/22-v2.8-0021-localbuf-Track-pincount-in-BufferDesc-as-well.patch)
download | inline diff:
From feb0718a8c17f42d3d3e597d010601574f2a81f0 Mon Sep 17 00:00:00 2001
From: Andres Freund <andres@anarazel.de>
Date: Thu, 20 Feb 2025 18:04:48 -0500
Subject: [PATCH v2.8 21/38] localbuf: Track pincount in BufferDesc as well
For AIO on temporary table buffers the AIO subsystem needs to be able to
ensure a pin on a buffer while AIO is going on, even if the IO issuing query
errors out. Tracking the buffer in LocalRefCount does not work, as it would
cause CheckForLocalBufferLeaks() to assert out.
Instead, also track the refcount in BufferDesc.state, not just
LocalRefCount. This also makes local buffers behave a bit more akin to shared
buffers.
Note that we still don't need locking, AIO completion callbacks for local
buffers are executed in the issuing session (i.e. nobody else has access to
the BufferDesc).
Discussion: https://postgr.es/m/uvrtrknj4kdytuboidbhwclo4gxhswwcpgadptsjvjqcluzmah%40brqs62irg4dt
---
src/backend/storage/buffer/bufmgr.c | 22 +++++++++++++++++++++
src/backend/storage/buffer/localbuf.c | 28 +++++++++++++++++++++++++--
2 files changed, 48 insertions(+), 2 deletions(-)
diff --git a/src/backend/storage/buffer/bufmgr.c b/src/backend/storage/buffer/bufmgr.c
index 95911b51d58..a9643930974 100644
--- a/src/backend/storage/buffer/bufmgr.c
+++ b/src/backend/storage/buffer/bufmgr.c
@@ -5393,6 +5393,18 @@ ConditionalLockBufferForCleanup(Buffer buffer)
Assert(refcount > 0);
if (refcount != 1)
return false;
+
+ /*
+ * Check that the AIO subsystem doesn't have a pin. Likely not
+ * possible today, but better safe than sorry.
+ */
+ bufHdr = GetLocalBufferDescriptor(-buffer - 1);
+ buf_state = pg_atomic_read_u32(&bufHdr->state);
+ refcount = BUF_STATE_GET_REFCOUNT(buf_state);
+ Assert(refcount > 0);
+ if (refcount != 1)
+ return false;
+
/* Nobody else to wait for */
return true;
}
@@ -5446,6 +5458,16 @@ IsBufferCleanupOK(Buffer buffer)
/* There should be exactly one pin */
if (LocalRefCount[-buffer - 1] != 1)
return false;
+
+ /*
+ * Check that the AIO subsystem doesn't have a pin. Likely not
+ * possible today, but better safe than sorry.
+ */
+ bufHdr = GetLocalBufferDescriptor(-buffer - 1);
+ buf_state = pg_atomic_read_u32(&bufHdr->state);
+ if (BUF_STATE_GET_REFCOUNT(buf_state) != 1)
+ return false;
+
/* Nobody else to wait for */
return true;
}
diff --git a/src/backend/storage/buffer/localbuf.c b/src/backend/storage/buffer/localbuf.c
index f172a5c7820..3a722321533 100644
--- a/src/backend/storage/buffer/localbuf.c
+++ b/src/backend/storage/buffer/localbuf.c
@@ -249,6 +249,13 @@ GetLocalVictimBuffer(void)
pg_atomic_unlocked_write_u32(&bufHdr->state, buf_state);
trycounter = NLocBuffer;
}
+ else if (BUF_STATE_GET_REFCOUNT(buf_state) > 0)
+ {
+ /*
+ * This can be reached if the backend initiated AIO for this
+ * buffer and then errored out.
+ */
+ }
else
{
/* Found a usable buffer */
@@ -570,7 +577,13 @@ InvalidateLocalBuffer(BufferDesc *bufHdr, bool check_unreferenced)
buf_state = pg_atomic_read_u32(&bufHdr->state);
- if (check_unreferenced && LocalRefCount[bufid] != 0)
+ /*
+ * We need to test not just LocalRefCount[bufid] but also the BufferDesc
+ * itself, as the latter is used to represent a pin by the AIO subsystem.
+ * This can happen if AIO is initiated and then the query errors out.
+ */
+ if (check_unreferenced &&
+ (LocalRefCount[bufid] != 0 || BUF_STATE_GET_REFCOUNT(buf_state) != 0))
elog(ERROR, "block %u of %s is still referenced (local %u)",
bufHdr->tag.blockNum,
relpathbackend(BufTagGetRelFileLocator(&bufHdr->tag),
@@ -744,12 +757,13 @@ PinLocalBuffer(BufferDesc *buf_hdr, bool adjust_usagecount)
if (LocalRefCount[bufid] == 0)
{
NLocalPinnedBuffers++;
+ buf_state += BUF_REFCOUNT_ONE;
if (adjust_usagecount &&
BUF_STATE_GET_USAGECOUNT(buf_state) < BM_MAX_USAGE_COUNT)
{
buf_state += BUF_USAGECOUNT_ONE;
- pg_atomic_unlocked_write_u32(&buf_hdr->state, buf_state);
}
+ pg_atomic_unlocked_write_u32(&buf_hdr->state, buf_state);
}
LocalRefCount[bufid]++;
ResourceOwnerRememberBuffer(CurrentResourceOwner,
@@ -775,7 +789,17 @@ UnpinLocalBufferNoOwner(Buffer buffer)
Assert(NLocalPinnedBuffers > 0);
if (--LocalRefCount[buffid] == 0)
+ {
+ BufferDesc *buf_hdr = GetLocalBufferDescriptor(buffid);
+ uint32 buf_state;
+
NLocalPinnedBuffers--;
+
+ buf_state = pg_atomic_read_u32(&buf_hdr->state);
+ Assert(BUF_STATE_GET_REFCOUNT(buf_state) > 0);
+ buf_state -= BUF_REFCOUNT_ONE;
+ pg_atomic_unlocked_write_u32(&buf_hdr->state, buf_state);
+ }
}
/*
--
2.48.1.76.g4e746b1a31.dirty
[text/x-diff] v2.8-0022-bufmgr-Implement-AIO-read-support.patch (25.4K, ../../3yxd5r23zly5bytvgyktbxtxq2r3gbpi7xd4dugevh3h4w4q6c@lu6oatjjpltz/23-v2.8-0022-bufmgr-Implement-AIO-read-support.patch)
download | inline diff:
From e2cf561cb8edb8012cb5996c094d216475ba6e21 Mon Sep 17 00:00:00 2001
From: Andres Freund <andres@anarazel.de>
Date: Tue, 11 Mar 2025 13:51:06 -0400
Subject: [PATCH v2.8 22/38] bufmgr: Implement AIO read support
This implements the following:
- Addition of callbacks to maintain buffer state on completion of a readv
- Addition of a wait reference to BufferDesc, to allow backends to wait for
IOs
- StartBufferIO(), WaitIO(), TerminateBufferIO() support for waiting AIO
As of this commit nothing issues AIO on shared buffers. A future commit will
update StartReadBuffers() to do so.
Todo:
- deduplicate shared/local buffer completion callbacks
- deduplicate LockBufferForCleanup() support
- function naming pattern (buffer_readv_complete_common() calls
LocalBufferCompleteRead/SharedBufferCompleteRead, which looks ugly)
Discussion: https://postgr.es/m/uvrtrknj4kdytuboidbhwclo4gxhswwcpgadptsjvjqcluzmah%40brqs62irg4dt
Discussion: https://postgr.es/m/20210223100344.llw5an2aklengrmn@alap3.anarazel.de
Discussion: https://postgr.es/m/stj36ea6yyhoxtqkhpieia2z4krnam7qyetc57rfezgk4zgapf@gcnactj4z56m
---
src/include/storage/aio.h | 4 +
src/include/storage/buf_internals.h | 7 +-
src/include/storage/bufmgr.h | 4 +
src/backend/storage/aio/aio_callback.c | 5 +
src/backend/storage/buffer/buf_init.c | 3 +
src/backend/storage/buffer/bufmgr.c | 440 +++++++++++++++++++++++--
src/backend/storage/buffer/localbuf.c | 45 ++-
7 files changed, 466 insertions(+), 42 deletions(-)
diff --git a/src/include/storage/aio.h b/src/include/storage/aio.h
index d02c3d27fae..a4d9b51acfe 100644
--- a/src/include/storage/aio.h
+++ b/src/include/storage/aio.h
@@ -184,6 +184,10 @@ typedef enum PgAioHandleCallbackID
PGAIO_HCB_MD_READV,
PGAIO_HCB_MD_WRITEV,
+
+ PGAIO_HCB_SHARED_BUFFER_READV,
+
+ PGAIO_HCB_LOCAL_BUFFER_READV,
} PgAioHandleCallbackID;
diff --git a/src/include/storage/buf_internals.h b/src/include/storage/buf_internals.h
index 9327f60c44c..8a52ace37cd 100644
--- a/src/include/storage/buf_internals.h
+++ b/src/include/storage/buf_internals.h
@@ -17,6 +17,7 @@
#include "pgstat.h"
#include "port/atomics.h"
+#include "storage/aio_types.h"
#include "storage/buf.h"
#include "storage/bufmgr.h"
#include "storage/condition_variable.h"
@@ -264,6 +265,8 @@ typedef struct BufferDesc
int wait_backend_pgprocno; /* backend of pin-count waiter */
int freeNext; /* link in freelist chain */
+
+ PgAioWaitRef io_wref; /* set iff AIO is in progress */
LWLock content_lock; /* to lock access to buffer contents */
} BufferDesc;
@@ -472,8 +475,8 @@ extern BlockNumber ExtendBufferedRelLocal(BufferManagerRelation bmr,
uint32 *extended_by);
extern void MarkLocalBufferDirty(Buffer buffer);
extern void TerminateLocalBufferIO(BufferDesc *bufHdr, bool clear_dirty,
- uint32 set_flag_bits);
-extern bool StartLocalBufferIO(BufferDesc *bufHdr, bool forInput);
+ uint32 set_flag_bits, bool syncio);
+extern bool StartLocalBufferIO(BufferDesc *bufHdr, bool forInput, bool nowait);
extern void FlushLocalBuffer(BufferDesc *bufHdr, SMgrRelation reln);
extern void DropRelationLocalBuffers(RelFileLocator rlocator,
ForkNumber forkNum,
diff --git a/src/include/storage/bufmgr.h b/src/include/storage/bufmgr.h
index 690cdde793a..eafc94d7ed1 100644
--- a/src/include/storage/bufmgr.h
+++ b/src/include/storage/bufmgr.h
@@ -15,6 +15,7 @@
#define BUFMGR_H
#include "port/pg_iovec.h"
+#include "storage/aio_types.h"
#include "storage/block.h"
#include "storage/buf.h"
#include "storage/bufpage.h"
@@ -168,6 +169,9 @@ extern PGDLLIMPORT int checkpoint_flush_after;
extern PGDLLIMPORT int backend_flush_after;
extern PGDLLIMPORT int bgwriter_flush_after;
+extern const PgAioHandleCallbacks aio_shared_buffer_readv_cb;
+extern const PgAioHandleCallbacks aio_local_buffer_readv_cb;
+
/* in buf_init.c */
extern PGDLLIMPORT char *BufferBlocks;
diff --git a/src/backend/storage/aio/aio_callback.c b/src/backend/storage/aio/aio_callback.c
index 6a21c82396a..f76f74ba166 100644
--- a/src/backend/storage/aio/aio_callback.c
+++ b/src/backend/storage/aio/aio_callback.c
@@ -18,6 +18,7 @@
#include "miscadmin.h"
#include "storage/aio.h"
#include "storage/aio_internal.h"
+#include "storage/bufmgr.h"
#include "storage/md.h"
@@ -40,6 +41,10 @@ static const PgAioHandleCallbacksEntry aio_handle_cbs[] = {
CALLBACK_ENTRY(PGAIO_HCB_INVALID, aio_invalid_cb),
CALLBACK_ENTRY(PGAIO_HCB_MD_READV, aio_md_readv_cb),
+
+ CALLBACK_ENTRY(PGAIO_HCB_SHARED_BUFFER_READV, aio_shared_buffer_readv_cb),
+
+ CALLBACK_ENTRY(PGAIO_HCB_LOCAL_BUFFER_READV, aio_local_buffer_readv_cb),
#undef CALLBACK_ENTRY
};
diff --git a/src/backend/storage/buffer/buf_init.c b/src/backend/storage/buffer/buf_init.c
index ed1f8e03190..ed1dc488a42 100644
--- a/src/backend/storage/buffer/buf_init.c
+++ b/src/backend/storage/buffer/buf_init.c
@@ -14,6 +14,7 @@
*/
#include "postgres.h"
+#include "storage/aio.h"
#include "storage/buf_internals.h"
#include "storage/bufmgr.h"
@@ -125,6 +126,8 @@ BufferManagerShmemInit(void)
buf->buf_id = i;
+ pgaio_wref_clear(&buf->io_wref);
+
/*
* Initially link all the buffers together as unused. Subsequent
* management of this list is done by freelist.c.
diff --git a/src/backend/storage/buffer/bufmgr.c b/src/backend/storage/buffer/bufmgr.c
index a9643930974..54add34f1a2 100644
--- a/src/backend/storage/buffer/bufmgr.c
+++ b/src/backend/storage/buffer/bufmgr.c
@@ -48,6 +48,7 @@
#include "pg_trace.h"
#include "pgstat.h"
#include "postmaster/bgwriter.h"
+#include "storage/aio.h"
#include "storage/buf_internals.h"
#include "storage/bufmgr.h"
#include "storage/fd.h"
@@ -516,7 +517,8 @@ static int SyncOneBuffer(int buf_id, bool skip_recently_used,
static void WaitIO(BufferDesc *buf);
static bool StartBufferIO(BufferDesc *buf, bool forInput, bool nowait);
static void TerminateBufferIO(BufferDesc *buf, bool clear_dirty,
- uint32 set_flag_bits, bool forget_owner);
+ uint32 set_flag_bits, bool forget_owner,
+ bool syncio);
static void AbortBufferIO(Buffer buffer);
static void shared_buffer_write_error_callback(void *arg);
static void local_buffer_write_error_callback(void *arg);
@@ -1038,7 +1040,7 @@ ZeroAndLockBuffer(Buffer buffer, ReadBufferMode mode, bool already_valid)
{
/* Simple case for non-shared buffers. */
bufHdr = GetLocalBufferDescriptor(-buffer - 1);
- need_to_zero = StartLocalBufferIO(bufHdr, true);
+ need_to_zero = StartLocalBufferIO(bufHdr, true, false);
}
else
{
@@ -1074,9 +1076,9 @@ ZeroAndLockBuffer(Buffer buffer, ReadBufferMode mode, bool already_valid)
/* Set BM_VALID, terminate IO, and wake up any waiters */
if (isLocalBuf)
- TerminateLocalBufferIO(bufHdr, false, BM_VALID);
+ TerminateLocalBufferIO(bufHdr, false, BM_VALID, true);
else
- TerminateBufferIO(bufHdr, false, BM_VALID, true);
+ TerminateBufferIO(bufHdr, false, BM_VALID, true, true);
}
else if (!isLocalBuf)
{
@@ -1450,7 +1452,8 @@ static inline bool
WaitReadBuffersCanStartIO(Buffer buffer, bool nowait)
{
if (BufferIsLocal(buffer))
- return StartLocalBufferIO(GetLocalBufferDescriptor(-buffer - 1), true);
+ return StartLocalBufferIO(GetLocalBufferDescriptor(-buffer - 1),
+ true, nowait);
else
return StartBufferIO(GetBufferDescriptor(buffer - 1), true, nowait);
}
@@ -1598,9 +1601,9 @@ WaitReadBuffers(ReadBuffersOperation *operation)
/* Set BM_VALID, terminate IO, and wake up any waiters */
if (persistence == RELPERSISTENCE_TEMP)
- TerminateLocalBufferIO(bufHdr, false, BM_VALID);
+ TerminateLocalBufferIO(bufHdr, false, BM_VALID, true);
else
- TerminateBufferIO(bufHdr, false, BM_VALID, true);
+ TerminateBufferIO(bufHdr, false, BM_VALID, true, true);
/* Report I/Os as completing individually. */
TRACE_POSTGRESQL_BUFFER_READ_DONE(forknum, io_first_block + j,
@@ -2503,7 +2506,7 @@ ExtendBufferedRelShared(BufferManagerRelation bmr,
if (lock)
LWLockAcquire(BufferDescriptorGetContentLock(buf_hdr), LW_EXCLUSIVE);
- TerminateBufferIO(buf_hdr, false, BM_VALID, true);
+ TerminateBufferIO(buf_hdr, false, BM_VALID, true, true);
}
pgBufferUsage.shared_blks_written += extend_by;
@@ -2849,6 +2852,44 @@ PinBuffer_Locked(BufferDesc *buf)
ResourceOwnerRememberBuffer(CurrentResourceOwner, b);
}
+/*
+ * Support for waking up another backend that is waiting for the cleanup lock
+ * to be released using BM_PIN_COUNT_WAITER.
+ *
+ * See LockBufferForCleanup().
+ *
+ * Expected to be called just after releasing a buffer pin (in a BufferDesc,
+ * not just reducing the backend-local pincount for the buffer).
+ */
+static void
+WakePinCountWaiter(BufferDesc *buf)
+{
+ /*
+ * Acquire the buffer header lock, re-check that there's a waiter. Another
+ * backend could have unpinned this buffer, and already woken up the
+ * waiter.
+ *
+ * There's no danger of the buffer being replaced after we unpinned it
+ * above, as it's pinned by the waiter. The waiter removes
+ * BM_PIN_COUNT_WAITER if it stops waiting for a reason other than this
+ * backend waking it up.
+ */
+ uint32 buf_state = LockBufHdr(buf);
+
+ if ((buf_state & BM_PIN_COUNT_WAITER) &&
+ BUF_STATE_GET_REFCOUNT(buf_state) == 1)
+ {
+ /* we just released the last pin other than the waiter's */
+ int wait_backend_pgprocno = buf->wait_backend_pgprocno;
+
+ buf_state &= ~BM_PIN_COUNT_WAITER;
+ UnlockBufHdr(buf, buf_state);
+ ProcSendSignal(wait_backend_pgprocno);
+ }
+ else
+ UnlockBufHdr(buf, buf_state);
+}
+
/*
* UnpinBuffer -- make buffer available for replacement.
*
@@ -2917,29 +2958,8 @@ UnpinBufferNoOwner(BufferDesc *buf)
/* Support LockBufferForCleanup() */
if (buf_state & BM_PIN_COUNT_WAITER)
- {
- /*
- * Acquire the buffer header lock, re-check that there's a waiter.
- * Another backend could have unpinned this buffer, and already
- * woken up the waiter. There's no danger of the buffer being
- * replaced after we unpinned it above, as it's pinned by the
- * waiter.
- */
- buf_state = LockBufHdr(buf);
+ WakePinCountWaiter(buf);
- if ((buf_state & BM_PIN_COUNT_WAITER) &&
- BUF_STATE_GET_REFCOUNT(buf_state) == 1)
- {
- /* we just released the last pin other than the waiter's */
- int wait_backend_pgprocno = buf->wait_backend_pgprocno;
-
- buf_state &= ~BM_PIN_COUNT_WAITER;
- UnlockBufHdr(buf, buf_state);
- ProcSendSignal(wait_backend_pgprocno);
- }
- else
- UnlockBufHdr(buf, buf_state);
- }
ForgetPrivateRefCountEntry(ref);
}
}
@@ -3959,7 +3979,7 @@ FlushBuffer(BufferDesc *buf, SMgrRelation reln, IOObject io_object,
* Mark the buffer as clean (unless BM_JUST_DIRTIED has become set) and
* end the BM_IO_IN_PROGRESS state.
*/
- TerminateBufferIO(buf, true, 0, true);
+ TerminateBufferIO(buf, true, 0, true, true);
TRACE_POSTGRESQL_BUFFER_FLUSH_DONE(BufTagGetForkNum(&buf->tag),
buf->tag.blockNum,
@@ -5518,6 +5538,7 @@ WaitIO(BufferDesc *buf)
for (;;)
{
uint32 buf_state;
+ PgAioWaitRef iow;
/*
* It may not be necessary to acquire the spinlock to check the flag
@@ -5525,10 +5546,19 @@ WaitIO(BufferDesc *buf)
* play it safe.
*/
buf_state = LockBufHdr(buf);
+ iow = buf->io_wref;
UnlockBufHdr(buf, buf_state);
if (!(buf_state & BM_IO_IN_PROGRESS))
break;
+
+ if (pgaio_wref_valid(&iow))
+ {
+ pgaio_wref_wait(&iow);
+ ConditionVariablePrepareToSleep(cv);
+ continue;
+ }
+
ConditionVariableSleep(cv, WAIT_EVENT_BUFFER_IO);
}
ConditionVariableCancelSleep();
@@ -5617,7 +5647,7 @@ StartBufferIO(BufferDesc *buf, bool forInput, bool nowait)
*/
static void
TerminateBufferIO(BufferDesc *buf, bool clear_dirty, uint32 set_flag_bits,
- bool forget_owner)
+ bool forget_owner, bool syncio)
{
uint32 buf_state;
@@ -5632,6 +5662,14 @@ TerminateBufferIO(BufferDesc *buf, bool clear_dirty, uint32 set_flag_bits,
if (clear_dirty && !(buf_state & BM_JUST_DIRTIED))
buf_state &= ~(BM_DIRTY | BM_CHECKPOINT_NEEDED);
+ if (!syncio)
+ {
+ /* release ownership by the AIO subsystem */
+ Assert(BUF_STATE_GET_REFCOUNT(buf_state) > 0);
+ buf_state -= BUF_REFCOUNT_ONE;
+ pgaio_wref_clear(&buf->io_wref);
+ }
+
buf_state |= set_flag_bits;
UnlockBufHdr(buf, buf_state);
@@ -5640,6 +5678,17 @@ TerminateBufferIO(BufferDesc *buf, bool clear_dirty, uint32 set_flag_bits,
BufferDescriptorGetBuffer(buf));
ConditionVariableBroadcast(BufferDescriptorGetIOCV(buf));
+
+ /*
+ * Support LockBufferForCleanup()
+ *
+ * If we just released a pin, need to do BM_PIN_COUNT_WAITER handling.
+ * Most of the time the current backend will hold another pin preventing
+ * that from happening, but that's e.g. not the case when completing an IO
+ * another backend started.
+ */
+ if (buf_state & BM_PIN_COUNT_WAITER)
+ WakePinCountWaiter(buf);
}
/*
@@ -5688,7 +5737,7 @@ AbortBufferIO(Buffer buffer)
}
}
- TerminateBufferIO(buf_hdr, false, BM_IO_ERROR, false);
+ TerminateBufferIO(buf_hdr, false, BM_IO_ERROR, false, true);
}
/*
@@ -6139,3 +6188,328 @@ EvictUnpinnedBuffer(Buffer buf)
return result;
}
+
+/*
+ * Helper for AIO staging callback for both reads and writes as well as temp
+ * and shared buffers.
+ */
+static pg_attribute_always_inline void
+buffer_stage_common(PgAioHandle *ioh, bool is_write, bool is_temp)
+{
+ uint64 *io_data;
+ uint8 handle_data_len;
+ PgAioWaitRef io_ref;
+ BufferTag first PG_USED_FOR_ASSERTS_ONLY = {0};
+
+ io_data = pgaio_io_get_handle_data(ioh, &handle_data_len);
+
+ pgaio_io_get_wref(ioh, &io_ref);
+
+ for (int i = 0; i < handle_data_len; i++)
+ {
+ Buffer buffer = (Buffer) io_data[i];
+ BufferDesc *buf_hdr = is_temp ?
+ GetLocalBufferDescriptor(-buffer - 1)
+ : GetBufferDescriptor(buffer - 1);
+ uint32 buf_state;
+
+ /*
+ * Check that all the buffers are actually ones that could conceivably
+ * be done in one IO, i.e. are sequential.
+ */
+ if (i == 0)
+ first = buf_hdr->tag;
+ else
+ {
+ Assert(buf_hdr->tag.relNumber == first.relNumber);
+ Assert(buf_hdr->tag.blockNum == first.blockNum + i);
+ }
+
+ if (is_temp)
+ buf_state = pg_atomic_read_u32(&buf_hdr->state);
+ else
+ buf_state = LockBufHdr(buf_hdr);
+
+ Assert(buf_state & BM_TAG_VALID);
+ if (is_write)
+ {
+ Assert(buf_state & BM_VALID);
+ Assert(buf_state & BM_DIRTY);
+ }
+ else
+ {
+ Assert(!(buf_state & BM_VALID));
+ Assert(!(buf_state & BM_DIRTY));
+ }
+
+ /* temp buffers don't use BM_IO_IN_PROGRESS */
+ if (!is_temp)
+ Assert(buf_state & BM_IO_IN_PROGRESS);
+
+ /* we better have ensured the buffer is present until now */
+ Assert(BUF_STATE_GET_REFCOUNT(buf_state) >= 1);
+
+ /*
+ * Reflect that the buffer is now owned by the subsystem.
+ *
+ * For local buffers: This can't be done just in LocalRefCount as one
+ * might initially think, as this backend could error out while AIO is
+ * still in progress, releasing all the pins by the backend itself.
+ */
+ buf_state += BUF_REFCOUNT_ONE;
+ buf_hdr->io_wref = io_ref;
+
+ if (is_temp)
+ pg_atomic_unlocked_write_u32(&buf_hdr->state, buf_state);
+ else
+ UnlockBufHdr(buf_hdr, buf_state);
+
+ if (is_write && !is_temp)
+ {
+ LWLock *content_lock;
+
+ content_lock = BufferDescriptorGetContentLock(buf_hdr);
+
+ Assert(LWLockHeldByMe(content_lock));
+
+ /*
+ * Lock is now owned by AIO subsystem.
+ */
+ LWLockDisown(content_lock);
+ }
+
+ /*
+ * Stop tracking this buffer via the resowner - the AIO system now
+ * keeps track.
+ */
+ if (!is_temp)
+ ResourceOwnerForgetBufferIO(CurrentResourceOwner, buffer);
+ }
+}
+
+/*
+ * Helper for AIO readv completion callbacks, supporting both shared and temp
+ * buffers. Gets called once for each buffer in a multi-page read.
+ */
+static pg_attribute_always_inline PgAioResult
+buffer_readv_complete_one(uint8 buf_off, Buffer buffer, uint8 flags,
+ bool failed, bool is_temp)
+{
+ BufferDesc *buf_hdr = is_temp ?
+ GetLocalBufferDescriptor(-buffer - 1)
+ : GetBufferDescriptor(buffer - 1);
+ BufferTag tag = buf_hdr->tag;
+ char *bufdata = BufferGetBlock(buffer);
+ PgAioResult result;
+ uint32 set_flag_bits;
+
+#ifdef USE_ASSERT_CHECKING
+ {
+ uint32 buf_state = pg_atomic_read_u32(&buf_hdr->state);
+
+ Assert(buf_state & BM_TAG_VALID);
+ Assert(!(buf_state & BM_VALID));
+ /* temp buffers don't use BM_IO_IN_PROGRESS */
+ if (!is_temp)
+ Assert(buf_state & BM_IO_IN_PROGRESS);
+ Assert(!(buf_state & BM_DIRTY));
+ }
+#endif
+
+ result.status = ARS_OK;
+
+ /* check for garbage data */
+ if (!failed &&
+ !PageIsVerifiedExtended((Page) bufdata, tag.blockNum,
+ PIV_LOG_WARNING | PIV_REPORT_STAT))
+ {
+ RelFileLocator rlocator = BufTagGetRelFileLocator(&tag);
+
+ if ((flags & READ_BUFFERS_ZERO_ON_ERROR) || zero_damaged_pages)
+ {
+ ereport(LOG,
+ (errcode(ERRCODE_DATA_CORRUPTED),
+ errmsg("invalid page in block %u of relation %s; zeroing out page",
+ tag.blockNum,
+ relpathbackend(rlocator,
+ is_temp ? MyProcNumber : INVALID_PROC_NUMBER,
+ tag.forkNum).str)));
+ memset(bufdata, 0, BLCKSZ);
+ }
+ else
+ {
+ /* mark buffer as having failed */
+ failed = true;
+
+ /* encode error for buffer_readv_report */
+ result.status = ARS_ERROR;
+ if (is_temp)
+ result.id = PGAIO_HCB_LOCAL_BUFFER_READV;
+ else
+ result.id = PGAIO_HCB_SHARED_BUFFER_READV;
+ result.error_data = buf_off;
+ }
+ }
+
+ /* Terminate I/O and set BM_VALID. */
+ set_flag_bits = failed ? BM_IO_ERROR : BM_VALID;
+ if (is_temp)
+ TerminateLocalBufferIO(buf_hdr, false, set_flag_bits, false);
+ else
+ TerminateBufferIO(buf_hdr, false, set_flag_bits, false, false);
+
+ /*
+ * Call the BUFFER_READ_DONE tracepoint in the callback, even though the
+ * callback may not be executed in the same backend that called
+ * BUFFER_READ_START. The alternative would be to defer calling the
+ * tracepoint to a later point (e.g. the local completion callback for
+ * shared buffer reads), which seems even less helpful and would also
+ * imply some overhead.
+ */
+ TRACE_POSTGRESQL_BUFFER_READ_DONE(tag.forkNum,
+ tag.blockNum,
+ tag.spcOid,
+ tag.dbOid,
+ tag.relNumber,
+ is_temp ? MyProcNumber : INVALID_PROC_NUMBER,
+ false);
+
+ return result;
+}
+
+/*
+ * Perform completion handling of a single AIO read. This read may cover
+ * multiple blocks / buffers.
+ *
+ * Shared between shared and local buffers, to reduce code duplication.
+ */
+static pg_attribute_always_inline PgAioResult
+buffer_readv_complete(PgAioHandle *ioh, PgAioResult prior_result,
+ uint8 cb_data, bool is_temp)
+{
+ PgAioResult result = prior_result;
+ PgAioTargetData *td = pgaio_io_get_target_data(ioh);
+ uint64 *io_data;
+ uint8 handle_data_len;
+
+ if (is_temp)
+ {
+ Assert(td->smgr.is_temp);
+ Assert(pgaio_io_get_owner(ioh) == MyProcNumber);
+ }
+ else
+ Assert(!td->smgr.is_temp);
+
+ /*
+ * Iterate over all the buffers affected by this IO and call appropriate
+ * per-buffer completion function for each buffer.
+ */
+ io_data = pgaio_io_get_handle_data(ioh, &handle_data_len);
+ for (uint8 buf_off = 0; buf_off < handle_data_len; buf_off++)
+ {
+ Buffer buf = io_data[buf_off];
+ PgAioResult buf_result;
+ bool failed;
+
+ Assert(BufferIsValid(buf));
+
+ /*
+ * If the entire failed on a lower-level, each buffer needs to be
+ * marked as failed. In case of a partial read, some buffers may be
+ * ok.
+ */
+ failed =
+ prior_result.status == ARS_ERROR
+ || prior_result.result <= buf_off;
+
+ buf_result = buffer_readv_complete_one(buf_off, buf, cb_data, failed,
+ is_temp);
+
+ /*
+ * If there wasn't any prior error and the IO for this page failed in
+ * some form, set the whole IO's to the page's result.
+ */
+ if (result.status != ARS_ERROR && buf_result.status != ARS_OK)
+ {
+ result = buf_result;
+ pgaio_result_report(result, td, LOG);
+ }
+ }
+
+ return result;
+}
+
+/*
+ * AIO error reporting callback for aio_shared_buffer_readv_cb and
+ * aio_local_buffer_readv_cb.
+ *
+ * Errors are encoded as follows:
+ * - the only error is page verification failing
+ * - result.error_data is the offset of the first page that failed
+ * verification in a larger IO
+ */
+static void
+buffer_readv_report(PgAioResult result, const PgAioTargetData *target_data, int elevel)
+{
+ ProcNumber errProc;
+
+ if (target_data->smgr.is_temp)
+ errProc = MyProcNumber;
+ else
+ errProc = INVALID_PROC_NUMBER;
+
+ ereport(elevel,
+ errcode(ERRCODE_DATA_CORRUPTED),
+ errmsg("invalid page in block %u of relation %s",
+ target_data->smgr.blockNum + result.error_data,
+ relpathbackend(target_data->smgr.rlocator, errProc, target_data->smgr.forkNum).str
+ )
+ );
+}
+
+static void
+shared_buffer_readv_stage(PgAioHandle *ioh, uint8 cb_data)
+{
+ buffer_stage_common(ioh, false, false);
+}
+
+static PgAioResult
+shared_buffer_readv_complete(PgAioHandle *ioh, PgAioResult prior_result,
+ uint8 cb_data)
+{
+ return buffer_readv_complete(ioh, prior_result, cb_data, false);
+}
+
+static void
+local_buffer_readv_stage(PgAioHandle *ioh, uint8 cb_data)
+{
+ buffer_stage_common(ioh, false, true);
+}
+
+static PgAioResult
+local_buffer_readv_complete(PgAioHandle *ioh, PgAioResult prior_result,
+ uint8 cb_data)
+{
+ return buffer_readv_complete(ioh, prior_result, cb_data, true);
+}
+
+/* readv callback is is passed READ_BUFFERS_* flags as callback data */
+const PgAioHandleCallbacks aio_shared_buffer_readv_cb = {
+ .stage = shared_buffer_readv_stage,
+ .complete_shared = shared_buffer_readv_complete,
+ .report = buffer_readv_report,
+};
+
+/* readv callback is is passed READ_BUFFERS_* flags as callback data */
+const PgAioHandleCallbacks aio_local_buffer_readv_cb = {
+ .stage = local_buffer_readv_stage,
+
+ /*
+ * Note that this, in contrast to the shared_buffers case, uses
+ * complete_local, as only the issuing backend has access to the required
+ * datastructures. This is important in case the IO completion may be
+ * consumed incidentally by another backend.
+ */
+ .complete_local = local_buffer_readv_complete,
+ .report = buffer_readv_report,
+};
diff --git a/src/backend/storage/buffer/localbuf.c b/src/backend/storage/buffer/localbuf.c
index 3a722321533..b18c4c25143 100644
--- a/src/backend/storage/buffer/localbuf.c
+++ b/src/backend/storage/buffer/localbuf.c
@@ -17,7 +17,9 @@
#include "access/parallel.h"
#include "executor/instrument.h"
+#include "pg_trace.h"
#include "pgstat.h"
+#include "storage/aio.h"
#include "storage/buf_internals.h"
#include "storage/bufmgr.h"
#include "storage/fd.h"
@@ -187,7 +189,7 @@ FlushLocalBuffer(BufferDesc *bufHdr, SMgrRelation reln)
* Try to start an I/O operation. There currently are no reasons for
* StartLocalBufferIO to return false, so we raise an error in that case.
*/
- if (!StartLocalBufferIO(bufHdr, false))
+ if (!StartLocalBufferIO(bufHdr, false, false))
elog(ERROR, "failed to start write IO on local buffer");
/* Find smgr relation for buffer */
@@ -211,7 +213,7 @@ FlushLocalBuffer(BufferDesc *bufHdr, SMgrRelation reln)
IOOP_WRITE, io_start, 1, BLCKSZ);
/* Mark not-dirty */
- TerminateLocalBufferIO(bufHdr, true, 0);
+ TerminateLocalBufferIO(bufHdr, true, 0, true);
pgBufferUsage.local_blks_written++;
}
@@ -430,7 +432,7 @@ ExtendBufferedRelLocal(BufferManagerRelation bmr,
pg_atomic_unlocked_write_u32(&existing_hdr->state, buf_state);
/* no need to loop for local buffers */
- StartLocalBufferIO(existing_hdr, true);
+ StartLocalBufferIO(existing_hdr, true, false);
}
else
{
@@ -446,7 +448,7 @@ ExtendBufferedRelLocal(BufferManagerRelation bmr,
hresult->id = victim_buf_id;
- StartLocalBufferIO(victim_buf_hdr, true);
+ StartLocalBufferIO(victim_buf_hdr, true, false);
}
}
@@ -515,10 +517,25 @@ MarkLocalBufferDirty(Buffer buffer)
* Like StartBufferIO, but for local buffers
*/
bool
-StartLocalBufferIO(BufferDesc *bufHdr, bool forInput)
+StartLocalBufferIO(BufferDesc *bufHdr, bool forInput, bool nowait)
{
- uint32 buf_state = pg_atomic_read_u32(&bufHdr->state);
+ uint32 buf_state;
+ /*
+ * The buffer could have IO in progress, e.g. when there are two scans of
+ * the same relation. Either wait for the other IO or return false.
+ */
+ if (pgaio_wref_valid(&bufHdr->io_wref))
+ {
+ PgAioWaitRef iow = bufHdr->io_wref;
+
+ if (nowait)
+ return false;
+
+ pgaio_wref_wait(&iow);
+ }
+
+ buf_state = pg_atomic_read_u32(&bufHdr->state);
if (forInput ? (buf_state & BM_VALID) : !(buf_state & BM_DIRTY))
{
/* someone else already did the I/O */
@@ -536,7 +553,8 @@ StartLocalBufferIO(BufferDesc *bufHdr, bool forInput)
* Like TerminateBufferIO, but for local buffers
*/
void
-TerminateLocalBufferIO(BufferDesc *bufHdr, bool clear_dirty, uint32 set_flag_bits)
+TerminateLocalBufferIO(BufferDesc *bufHdr, bool clear_dirty, uint32 set_flag_bits,
+ bool syncio)
{
/* Only need to adjust flags */
uint32 buf_state = pg_atomic_read_u32(&bufHdr->state);
@@ -549,6 +567,17 @@ TerminateLocalBufferIO(BufferDesc *bufHdr, bool clear_dirty, uint32 set_flag_bit
if (clear_dirty)
buf_state &= ~BM_DIRTY;
+ if (!syncio)
+ {
+ /*
+ * Release pin held by IO subsystem, see also
+ * local_buffer_readv_prepare().
+ */
+ Assert(BUF_STATE_GET_REFCOUNT(buf_state) > 0);
+ buf_state -= BUF_REFCOUNT_ONE;
+ pgaio_wref_clear(&bufHdr->io_wref);
+ }
+
buf_state |= set_flag_bits;
pg_atomic_unlocked_write_u32(&bufHdr->state, buf_state);
@@ -714,6 +743,8 @@ InitLocalBuffers(void)
*/
buf->buf_id = -i - 2;
+ pgaio_wref_clear(&buf->io_wref);
+
/*
* Intentionally do not initialize the buffer's atomic variable
* (besides zeroing the underlying memory above). That way we get
--
2.48.1.76.g4e746b1a31.dirty
[text/x-diff] v2.8-0023-bufmgr-Improve-stats-when-buffer-was-read-in-co.patch (3.0K, ../../3yxd5r23zly5bytvgyktbxtxq2r3gbpi7xd4dugevh3h4w4q6c@lu6oatjjpltz/24-v2.8-0023-bufmgr-Improve-stats-when-buffer-was-read-in-co.patch)
download | inline diff:
From af6ca07ccd4d6917086e3fea2224a962ed396aa6 Mon Sep 17 00:00:00 2001
From: Andres Freund <andres@anarazel.de>
Date: Fri, 14 Mar 2025 13:03:55 -0400
Subject: [PATCH v2.8 23/38] bufmgr: Improve stats when buffer was read in
concurrently
Author:
Reviewed-by:
Discussion: https://postgr.es/m/
Backpatch:
---
src/backend/storage/buffer/bufmgr.c | 41 ++++++++++++++++++-----------
1 file changed, 26 insertions(+), 15 deletions(-)
diff --git a/src/backend/storage/buffer/bufmgr.c b/src/backend/storage/buffer/bufmgr.c
index 54add34f1a2..d783b33f78e 100644
--- a/src/backend/storage/buffer/bufmgr.c
+++ b/src/backend/storage/buffer/bufmgr.c
@@ -1490,19 +1490,6 @@ WaitReadBuffers(ReadBuffersOperation *operation)
io_object = IOOBJECT_RELATION;
}
- /*
- * We count all these blocks as read by this backend. This is traditional
- * behavior, but might turn out to be not true if we find that someone
- * else has beaten us and completed the read of some of these blocks. In
- * that case the system globally double-counts, but we traditionally don't
- * count this as a "hit", and we don't have a separate counter for "miss,
- * but another backend completed the read".
- */
- if (persistence == RELPERSISTENCE_TEMP)
- pgBufferUsage.local_blks_read += nblocks;
- else
- pgBufferUsage.shared_blks_read += nblocks;
-
for (int i = 0; i < nblocks; ++i)
{
int io_buffers_len;
@@ -1520,8 +1507,13 @@ WaitReadBuffers(ReadBuffersOperation *operation)
if (!WaitReadBuffersCanStartIO(buffers[i], false))
{
/*
- * Report this as a 'hit' for this backend, even though it must
- * have started out as a miss in PinBufferForBlock().
+ * Report and track this as a 'hit' for this backend, even though
+ * it must have started out as a miss in PinBufferForBlock().
+ *
+ * Some of the accesses would otherwise never be counted (e.g.
+ * pgBufferUsage) or counted as a miss (e.g.
+ * pgstat_count_buffer_hit(), as we always call
+ * pgstat_count_buffer_read()).
*/
TRACE_POSTGRESQL_BUFFER_READ_DONE(forknum, blocknum + i,
operation->smgr->smgr_rlocator.locator.spcOid,
@@ -1529,6 +1521,20 @@ WaitReadBuffers(ReadBuffersOperation *operation)
operation->smgr->smgr_rlocator.locator.relNumber,
operation->smgr->smgr_rlocator.backend,
true);
+
+ if (persistence == RELPERSISTENCE_TEMP)
+ pgBufferUsage.local_blks_hit += 1;
+ else
+ pgBufferUsage.shared_blks_hit += 1;
+
+ if (operation->rel)
+ pgstat_count_buffer_hit(operation->rel);
+
+ pgstat_count_io_op(io_object, io_context, IOOP_HIT, 1, 0);
+
+ if (VacuumCostActive)
+ VacuumCostBalance += VacuumCostPageHit;
+
continue;
}
@@ -1614,6 +1620,11 @@ WaitReadBuffers(ReadBuffersOperation *operation)
false);
}
+ if (persistence == RELPERSISTENCE_TEMP)
+ pgBufferUsage.local_blks_read += io_buffers_len;
+ else
+ pgBufferUsage.shared_blks_read += io_buffers_len;
+
if (VacuumCostActive)
VacuumCostBalance += VacuumCostPageMiss * io_buffers_len;
}
--
2.48.1.76.g4e746b1a31.dirty
[text/x-diff] v2.8-0024-bufmgr-Use-AIO-in-StartReadBuffers.patch (26.9K, ../../3yxd5r23zly5bytvgyktbxtxq2r3gbpi7xd4dugevh3h4w4q6c@lu6oatjjpltz/25-v2.8-0024-bufmgr-Use-AIO-in-StartReadBuffers.patch)
download | inline diff:
From 5403662d9fd2730066dd3ce3083b4f4f1a3d147a Mon Sep 17 00:00:00 2001
From: Andres Freund <andres@anarazel.de>
Date: Fri, 14 Mar 2025 13:48:47 -0400
Subject: [PATCH v2.8 24/38] bufmgr: Use AIO in StartReadBuffers()
This finally introduces the first actual AIO user. StartReadBuffers() now uses
the AIO routines to issue IO. This converts a lot of callers to use the AIO
infrastructure.
As the implementation of StartReadBuffers() is also used by the functions for
reading individual blocks (StartReadBuffer() and through that
ReadBufferExtended()) this means all buffered read IO passes through the AIO
paths. However, as those are synchronous reads, actually performing the IO
asynchronously would be rarely beneficial. Instead such IOs are flagged to
always be executed synchronously. This way we don't have to duplicate a fair
bit of code.
When io_method=sync is used, the IO patterns generated after this change are
the same as before, i.e. actual reads are only issued in WaitReadBuffers() and
StartReadBuffers() may issue prefetch requests. This allows to bypass most of
the actual asynchronicity, which is important to make a change as big as this
less risky.
One thing worth calling out is that, if IO is actually executed
asynchronously, the precise meaning of what track_io_timing is measuring has
changed. Previously it tracked the time for each IO, but that does not make
sense when multiple IOs are executed concurrently. Now it only measures the
time actually spent waiting for IO.
While AIO is now actually used, the logic in read_stream.c will often prevent
using sufficiently many concurrent IOs. That will be addressed in the next
commits.
Co-authored-by: Andres Freund <andres@anarazel.de>
Co-authored-by: Thomas Munro <thomas.munro@gmail.com>
Discussion: https://postgr.es/m/uvrtrknj4kdytuboidbhwclo4gxhswwcpgadptsjvjqcluzmah%40brqs62irg4dt
Discussion: https://postgr.es/m/20210223100344.llw5an2aklengrmn@alap3.anarazel.de
Discussion: https://postgr.es/m/stj36ea6yyhoxtqkhpieia2z4krnam7qyetc57rfezgk4zgapf@gcnactj4z56m
---
src/include/storage/bufmgr.h | 6 +
src/backend/storage/buffer/bufmgr.c | 614 +++++++++++++++++++++-------
2 files changed, 477 insertions(+), 143 deletions(-)
diff --git a/src/include/storage/bufmgr.h b/src/include/storage/bufmgr.h
index eafc94d7ed1..ca4ca22c345 100644
--- a/src/include/storage/bufmgr.h
+++ b/src/include/storage/bufmgr.h
@@ -112,6 +112,9 @@ typedef struct BufferManagerRelation
#define READ_BUFFERS_ZERO_ON_ERROR (1 << 0)
/* Call smgrprefetch() if I/O necessary. */
#define READ_BUFFERS_ISSUE_ADVICE (1 << 1)
+/* IO will immediately be waited for */
+#define READ_BUFFERS_SYNCHRONOUSLY (1 << 2)
+
struct ReadBuffersOperation
{
@@ -131,6 +134,9 @@ struct ReadBuffersOperation
BlockNumber blocknum;
int flags;
int16 nblocks;
+ int16 nblocks_done;
+ PgAioWaitRef io_wref;
+ PgAioReturn io_return;
};
typedef struct ReadBuffersOperation ReadBuffersOperation;
diff --git a/src/backend/storage/buffer/bufmgr.c b/src/backend/storage/buffer/bufmgr.c
index d783b33f78e..2691757016c 100644
--- a/src/backend/storage/buffer/bufmgr.c
+++ b/src/backend/storage/buffer/bufmgr.c
@@ -528,6 +528,8 @@ static inline BufferDesc *BufferAlloc(SMgrRelation smgr,
BlockNumber blockNum,
BufferAccessStrategy strategy,
bool *foundPtr, IOContext io_context);
+static bool AsyncReadBuffers(ReadBuffersOperation *operation, int *nblocks);
+static void CheckReadBuffersOperation(ReadBuffersOperation *operation, bool is_complete);
static Buffer GetVictimBuffer(BufferAccessStrategy strategy, IOContext io_context);
static void FlushBuffer(BufferDesc *buf, SMgrRelation reln,
IOObject io_object, IOContext io_context);
@@ -1228,10 +1230,14 @@ ReadBuffer_common(Relation rel, SMgrRelation smgr, char smgr_persistence,
return buffer;
}
+ /*
+ * Signal that we are going to immediately wait. If we're immediately
+ * waiting, there is no benefit in actually executing the IO
+ * asynchronously, it would just add dispatch overhead.
+ */
+ flags = READ_BUFFERS_SYNCHRONOUSLY;
if (mode == RBM_ZERO_ON_ERROR)
- flags = READ_BUFFERS_ZERO_ON_ERROR;
- else
- flags = 0;
+ flags |= READ_BUFFERS_ZERO_ON_ERROR;
operation.smgr = smgr;
operation.rel = rel;
operation.persistence = persistence;
@@ -1256,9 +1262,11 @@ StartReadBuffersImpl(ReadBuffersOperation *operation,
{
int actual_nblocks = *nblocks;
int maxcombine = 0;
+ bool did_start_io;
Assert(*nblocks > 0);
Assert(*nblocks <= MAX_IO_COMBINE_LIMIT);
+ Assert(*nblocks == 1 || allow_forwarding);
for (int i = 0; i < actual_nblocks; ++i)
{
@@ -1297,6 +1305,7 @@ StartReadBuffersImpl(ReadBuffersOperation *operation,
bufHdr = GetLocalBufferDescriptor(-buffers[i] - 1);
else
bufHdr = GetBufferDescriptor(buffers[i] - 1);
+ Assert(pg_atomic_read_u32(&bufHdr->state) & BM_TAG_VALID);
found = pg_atomic_read_u32(&bufHdr->state) & BM_VALID;
}
else
@@ -1321,6 +1330,20 @@ StartReadBuffersImpl(ReadBuffersOperation *operation,
if (i == 0)
{
*nblocks = 1;
+
+#ifdef USE_ASSERT_CHECKING
+
+ /*
+ * Initialize enough of ReadBuffersOperation to make
+ * CheckReadBuffersOperation() work. Outside of assertions
+ * that's not necessary when no IO is issued.
+ */
+ operation->buffers = buffers;
+ operation->blocknum = blockNum;
+ operation->nblocks = 1;
+ operation->nblocks_done = 1;
+ CheckReadBuffersOperation(operation, true);
+#endif
return false;
}
@@ -1363,25 +1386,74 @@ StartReadBuffersImpl(ReadBuffersOperation *operation,
operation->blocknum = blockNum;
operation->flags = flags;
operation->nblocks = actual_nblocks;
+ operation->nblocks_done = 0;
+ pgaio_wref_clear(&operation->io_wref);
- if (flags & READ_BUFFERS_ISSUE_ADVICE)
+ /*
+ * When using AIO, start the IO in the background. If not, issue prefetch
+ * requests if desired by the caller.
+ *
+ * The reason we have a dedicated path for IOMETHOD_SYNC here is to
+ * de-risk the introduction of AIO somewhat. It's a large architectural
+ * change, with lots of chances for unanticipated performance effects.
+ *
+ * Use of IOMETHOD_SYNC already leads to not actually performing IO
+ * asynchronously, but without the check here we'd execute IO earlier than
+ * we used to. Eventually this IOMETHOD_SYNC specific path should go away.
+ */
+ if (io_method != IOMETHOD_SYNC)
{
/*
- * In theory we should only do this if PinBufferForBlock() had to
- * allocate new buffers above. That way, if two calls to
- * StartReadBuffers() were made for the same blocks before
- * WaitReadBuffers(), only the first would issue the advice. That'd be
- * a better simulation of true asynchronous I/O, which would only
- * start the I/O once, but isn't done here for simplicity.
+ * Try to start IO asynchronously. It's possible that no IO needs to
+ * be started, if another backend already performed the IO.
+ *
+ * Note that if an IO is started, it might not cover the entire
+ * requested range, e.g. because an intermediary block has been read
+ * in by another backend. In that case any "trailing" buffers we
+ * already pinned above will be "forwarded" by read_stream.c to the
+ * next call to StartReadBuffers().
+ *
+ * This is signalled to the caller by decrementing *nblocks *and*
+ * reducing operation->nblocks. The latter is done here, but not below
+ * WaitReadBuffers(), as in WaitReadBuffers() we can't "shorten" the
+ * overall read size anymore, we need to retry until done in its
+ * entirety or until failed.
*/
- smgrprefetch(operation->smgr,
- operation->forknum,
- blockNum,
- actual_nblocks);
+ did_start_io = AsyncReadBuffers(operation, nblocks);
+
+ operation->nblocks = *nblocks;
+ }
+ else
+ {
+ operation->flags |= READ_BUFFERS_SYNCHRONOUSLY;
+
+ if (flags & READ_BUFFERS_ISSUE_ADVICE)
+ {
+ /*
+ * In theory we should only do this if PinBufferForBlock() had to
+ * allocate new buffers above. That way, if two calls to
+ * StartReadBuffers() were made for the same blocks before
+ * WaitReadBuffers(), only the first would issue the advice.
+ * That'd be a better simulation of true asynchronous I/O, which
+ * would only start the I/O once, but isn't done here for
+ * simplicity.
+ */
+ smgrprefetch(operation->smgr,
+ operation->forknum,
+ blockNum,
+ actual_nblocks);
+ }
+
+ /*
+ * Indicate that WaitReadBuffers() should be called. WaitReadBuffers()
+ * will initiate the necessary IO.
+ */
+ did_start_io = true;
}
- /* Indicate that WaitReadBuffers() should be called. */
- return true;
+ CheckReadBuffersOperation(operation, !did_start_io);
+
+ return did_start_io;
}
/*
@@ -1448,8 +1520,35 @@ StartReadBuffer(ReadBuffersOperation *operation,
return result;
}
+/*
+ * Perform sanity checks on the ReadBuffersOperation.
+ */
+static void
+CheckReadBuffersOperation(ReadBuffersOperation *operation, bool is_complete)
+{
+#ifdef USE_ASSERT_CHECKING
+ Assert(operation->nblocks_done <= operation->nblocks);
+ Assert(!is_complete || operation->nblocks == operation->nblocks_done);
+
+ for (int i = 0; i < operation->nblocks; i++)
+ {
+ Buffer buffer = operation->buffers[i];
+ BufferDesc *buf_hdr = BufferIsLocal(buffer) ?
+ GetLocalBufferDescriptor(-buffer - 1) :
+ GetBufferDescriptor(buffer - 1);
+
+ Assert(BufferGetBlockNumber(buffer) == operation->blocknum + i);
+ Assert(pg_atomic_read_u32(&buf_hdr->state) & BM_TAG_VALID);
+
+ if (i < operation->nblocks_done)
+ Assert(pg_atomic_read_u32(&buf_hdr->state) & BM_VALID);
+ }
+#endif
+}
+
+/* helper for ReadBuffersCanStartIO(), to avoid repetition */
static inline bool
-WaitReadBuffersCanStartIO(Buffer buffer, bool nowait)
+ReadBuffersCanStartIOOnce(Buffer buffer, bool nowait)
{
if (BufferIsLocal(buffer))
return StartLocalBufferIO(GetLocalBufferDescriptor(-buffer - 1),
@@ -1458,31 +1557,243 @@ WaitReadBuffersCanStartIO(Buffer buffer, bool nowait)
return StartBufferIO(GetBufferDescriptor(buffer - 1), true, nowait);
}
-void
-WaitReadBuffers(ReadBuffersOperation *operation)
+/*
+ * Helper for AsyncReadBuffers that tries to initiate IO on the buffer.
+ */
+static inline bool
+ReadBuffersCanStartIO(Buffer buffer, bool nowait)
{
- Buffer *buffers;
- int nblocks;
- BlockNumber blocknum;
- ForkNumber forknum;
- IOContext io_context;
- IOObject io_object;
- char persistence;
-
- /* Find the range of the physical read we need to perform. */
- nblocks = operation->nblocks;
- buffers = &operation->buffers[0];
- blocknum = operation->blocknum;
- forknum = operation->forknum;
- persistence = operation->persistence;
+ /*
+ * If this backend currently has staged IO, we we need to submit the
+ * pending IO before waiting for the right to issue IO, to avoid the
+ * potential for deadlocks (and, more commonly, unnecessary delays for
+ * other backends).
+ */
+ if (!nowait && pgaio_have_staged())
+ {
+ if (ReadBuffersCanStartIOOnce(buffer, true))
+ return true;
+
+ /*
+ * Unfortunately a false returned StartBufferIO() doesn't allow to
+ * distinguish between the the buffer already being valid and IO
+ * already being in progress. Since IO already being in progress is
+ * quite rare, this approach seems fine.
+ */
+ pgaio_submit_staged();
+ }
+
+ return ReadBuffersCanStartIOOnce(buffer, nowait);
+}
+
+/*
+ * Helper for WaitReadBuffers() that processes the results of a readv
+ * operation, raising an error if necessary.
+ */
+static void
+ProcessReadBuffersResult(ReadBuffersOperation *operation)
+{
+ PgAioReturn *aio_ret = &operation->io_return;
+ int nblocks = 0;
+
+ Assert(pgaio_wref_valid(&operation->io_wref));
+ Assert(aio_ret->result.status != ARS_UNKNOWN);
+
+ /*
+ * SMGR reports the number of blocks sucessfully read as the result of the
+ * IO operation. Thus we can simply add that to ->nblocks_done.
+ */
+
+ if (likely(aio_ret->result.status == ARS_OK))
+ nblocks = aio_ret->result.result;
+ else if (aio_ret->result.status == ARS_PARTIAL)
+ {
+ /*
+ * We'll retry, so we just emit a debug message to the server log (or
+ * not even that in prod scenarios).
+ */
+ pgaio_result_report(aio_ret->result, &aio_ret->target_data, DEBUG1);
+ nblocks = aio_ret->result.result;
+
+ elog(DEBUG3, "partial read, will retry");
+
+ }
+ else if (aio_ret->result.status == ARS_ERROR)
+ {
+ pgaio_result_report(aio_ret->result, &aio_ret->target_data, ERROR);
+ nblocks = 0; /* silence compiler */
+ }
Assert(nblocks > 0);
Assert(nblocks <= MAX_IO_COMBINE_LIMIT);
+ operation->nblocks_done += nblocks;
+
+ Assert(operation->nblocks_done <= operation->nblocks);
+}
+
+void
+WaitReadBuffers(ReadBuffersOperation *operation)
+{
+ PgAioReturn *aio_ret = &operation->io_return;
+ IOContext io_context;
+ IOObject io_object;
+
+ if (operation->persistence == RELPERSISTENCE_TEMP)
+ {
+ io_context = IOCONTEXT_NORMAL;
+ io_object = IOOBJECT_TEMP_RELATION;
+ }
+ else
+ {
+ io_context = IOContextForStrategy(operation->strategy);
+ io_object = IOOBJECT_RELATION;
+ }
+
+ /*
+ * If we get here without an IO operation having been issued, the
+ * io_method == IOMETHOD_SYNC path must have been used. Otherwise the
+ * caller should not have called WaitReadBuffers().
+ *
+ * In the case of IOMETHOD_SYNC, we start - as we used to before the
+ * introducing of AIO - the IO in WaitReadBuffers(). This is done as part
+ * of the retry logic below, no extra code is required.
+ *
+ * This path is expected to eventually go away.
+ */
+ if (!pgaio_wref_valid(&operation->io_wref) && io_method != IOMETHOD_SYNC)
+ elog(ERROR, "waiting for read operation that didn't read");
+
+ /*
+ * To handle partial reads, and IOMETHOD_SYNC, we re-issue IO until we're
+ * done. We may need multiple retries, not just because we could get
+ * multiple short reads, but also because some of the remaining to-be-read
+ * buffers may have been read in by other backends, limiting the IO size.
+ */
+ while (true)
+ {
+ int nblocks;
+
+ CheckReadBuffersOperation(operation, false);
+
+ /*
+ * If there is an IO associated with the operation, we may need to
+ * wait for it. It's possible for there to be no IO if
+ */
+ if (pgaio_wref_valid(&operation->io_wref))
+ {
+ /*
+ * Track the time spent waiting for the IO to complete. As
+ * tracking a wait even if we don't actually need to wait
+ *
+ * a) is not cheap, due to the timestamping overhead
+ *
+ * b) reports some time as waiting, even if we never waited
+ *
+ * we first check if we already know the IO is complete.
+ */
+ if (aio_ret->result.status == ARS_UNKNOWN &&
+ !pgaio_wref_check_done(&operation->io_wref))
+ {
+ instr_time io_start = pgstat_prepare_io_time(track_io_timing);
+
+ pgaio_wref_wait(&operation->io_wref);
+
+ /*
+ * The IO operation itself was already counted earlier, in
+ * AsyncReadBuffers(), this just accounts for the wait time.
+ */
+ pgstat_count_io_op_time(io_object, io_context, IOOP_READ,
+ io_start, 0, 0);
+ }
+ else
+ {
+ Assert(pgaio_wref_check_done(&operation->io_wref));
+ }
+
+ /*
+ * We now are sure the IO completed. Check the results. This
+ * includes reporting on errors if there were any.
+ */
+ ProcessReadBuffersResult(operation);
+ }
+
+ /*
+ * Most of the the time the one IO we started will read in everything.
+ * But we need to deal with short reads and buffers not needing IO
+ * anymore.
+ */
+ if (operation->nblocks_done == operation->nblocks)
+ break;
+
+ CHECK_FOR_INTERRUPTS();
+
+ /*
+ * This may only complete the IO partially, either because some
+ * buffers were already valid, or because of a short read.
+ *
+ * NB: In contrast to after the AsyncReadBuffers() call in
+ * StartReadBuffers(), we do *not* reduce
+ * ReadBuffersOperation->nblocks here, callers expect the full
+ * operation to be completed at this point (as more operations may
+ * have been queued).
+ */
+ AsyncReadBuffers(operation, &nblocks);
+ }
+
+ CheckReadBuffersOperation(operation, true);
+
+ /* NB: READ_DONE tracepoint was already executed in completion callback */
+}
+
+/*
+ * Initiate IO for the ReadBuffersOperation
+ *
+ * This function only starts a single IO at a time. The size of the IO may be
+ * limited to below the to-be-read blocks if one of the buffers has
+ * concurrently been read in. If the first to-be-read buffer is already valid,
+ * no IO will be issued.
+ *
+ * To support retries after short reads, the first operation->nblocks_done is
+ * buffers are skipped.
+ *
+ * On return *nblocks_progres is updated to reflect the number of buffers
+ * affected by the call. If the first buffer is valid, *nblocks_progress is
+ * set to 1 and operation->nblocks_done is incremented.
+ *
+ * Returns true if IO was initiated, false if no IO was necessary.
+ */
+static bool
+AsyncReadBuffers(ReadBuffersOperation *operation, int *nblocks_progress)
+{
+ Buffer *buffers = &operation->buffers[0];
+ int flags = operation->flags;
+ BlockNumber blocknum = operation->blocknum;
+ ForkNumber forknum = operation->forknum;
+ char persistence = operation->persistence;
+ int16 nblocks_done = operation->nblocks_done;
+ Buffer *io_buffers = &operation->buffers[nblocks_done];
+ int io_buffers_len = 0;
+ PgAioHandle *ioh;
+ uint32 ioh_flags = 0;
+ void *io_pages[MAX_IO_COMBINE_LIMIT];
+ IOContext io_context;
+ IOObject io_object;
+ bool did_start_io;
+
+ /*
+ * When this IO is executed synchronously, either because the caller will
+ * immediately block waiting for the IO or because IOMETHOD_SYNC is used,
+ * the AIO subsystem needs to know.
+ */
+ if (flags & READ_BUFFERS_SYNCHRONOUSLY)
+ ioh_flags |= PGAIO_HF_SYNCHRONOUS;
+
if (persistence == RELPERSISTENCE_TEMP)
{
io_context = IOCONTEXT_NORMAL;
io_object = IOOBJECT_TEMP_RELATION;
+ ioh_flags |= PGAIO_HF_REFERENCES_LOCAL;
}
else
{
@@ -1490,58 +1801,90 @@ WaitReadBuffers(ReadBuffersOperation *operation)
io_object = IOOBJECT_RELATION;
}
- for (int i = 0; i < nblocks; ++i)
+ /*
+ * Get IO before ReadBuffersCanStartIO, as pgaio_io_acquire() might block,
+ * which we don't want after setting IO_IN_PROGRESS.
+ *
+ * If IO needs to be submitted before we can get a handle, submit already
+ * staged IO first, so that other backends don't need to wait. There
+ * wouldn't be a deadlock risk, as pgaio_io_acquire() just needs to wait
+ * for already submitted IO, which doesn't require additional locks, but
+ * it could still cause undesirable waits.
+ *
+ * A secondary benefit is that this would allows us to measure the time in
+ * pgaio_io_acquire() without causing undue timer overhead in the common,
+ * non-blocking, case. However, currently the pgstats infrastructure
+ * doesn't really allow that, as it a) asserts that an operation can't
+ * have time without operations b) doesn't have an API to report
+ * "accumulated" time.
+ */
+ ioh = pgaio_io_acquire_nb(CurrentResourceOwner, &operation->io_return);
+ if (unlikely(!ioh))
+ {
+ pgaio_submit_staged();
+
+ ioh = pgaio_io_acquire(CurrentResourceOwner, &operation->io_return);
+ }
+
+ /*
+ * Check if we can start IO on the buffer.
+ *
+ * If an I/O is already in progress in another backend, we want to wait
+ * for the outcome: either done, or something went wrong and we will
+ * retry.
+ */
+ if (!ReadBuffersCanStartIO(buffers[nblocks_done], false))
+ {
+ /*
+ * Someone else has already completed this block, we're done.
+ *
+ * When IO is necessary, ->nblocks_done is updated in
+ * ProcessReadBuffersResult(), but that is not called if no IO is
+ * necessary. Thus update here.
+ */
+ operation->nblocks_done += 1;
+ *nblocks_progress = 1;
+
+ pgaio_io_release(ioh);
+ pgaio_wref_clear(&operation->io_wref);
+ did_start_io = false;
+
+ /*
+ * Report and track this as a 'hit' for this backend, even though it
+ * must have started out as a miss in PinBufferForBlock().
+ *
+ * Some of the accesses would otherwise never be counted (e.g.
+ * pgBufferUsage) or counted as a miss (e.g.
+ * pgstat_count_buffer_hit(), as we always call
+ * pgstat_count_buffer_read()).
+ */
+ TRACE_POSTGRESQL_BUFFER_READ_DONE(forknum, blocknum + operation->nblocks_done,
+ operation->smgr->smgr_rlocator.locator.spcOid,
+ operation->smgr->smgr_rlocator.locator.dbOid,
+ operation->smgr->smgr_rlocator.locator.relNumber,
+ operation->smgr->smgr_rlocator.backend,
+ true);
+
+ if (persistence == RELPERSISTENCE_TEMP)
+ pgBufferUsage.local_blks_hit += 1;
+ else
+ pgBufferUsage.shared_blks_hit += 1;
+
+ if (operation->rel)
+ pgstat_count_buffer_hit(operation->rel);
+
+ pgstat_count_io_op(io_object, io_context, IOOP_HIT, 1, 0);
+
+ if (VacuumCostActive)
+ VacuumCostBalance += VacuumCostPageHit;
+ }
+ else
{
- int io_buffers_len;
- Buffer io_buffers[MAX_IO_COMBINE_LIMIT];
- void *io_pages[MAX_IO_COMBINE_LIMIT];
instr_time io_start;
- BlockNumber io_first_block;
-
- /*
- * Skip this block if someone else has already completed it. If an
- * I/O is already in progress in another backend, this will wait for
- * the outcome: either done, or something went wrong and we will
- * retry.
- */
- if (!WaitReadBuffersCanStartIO(buffers[i], false))
- {
- /*
- * Report and track this as a 'hit' for this backend, even though
- * it must have started out as a miss in PinBufferForBlock().
- *
- * Some of the accesses would otherwise never be counted (e.g.
- * pgBufferUsage) or counted as a miss (e.g.
- * pgstat_count_buffer_hit(), as we always call
- * pgstat_count_buffer_read()).
- */
- TRACE_POSTGRESQL_BUFFER_READ_DONE(forknum, blocknum + i,
- operation->smgr->smgr_rlocator.locator.spcOid,
- operation->smgr->smgr_rlocator.locator.dbOid,
- operation->smgr->smgr_rlocator.locator.relNumber,
- operation->smgr->smgr_rlocator.backend,
- true);
-
- if (persistence == RELPERSISTENCE_TEMP)
- pgBufferUsage.local_blks_hit += 1;
- else
- pgBufferUsage.shared_blks_hit += 1;
-
- if (operation->rel)
- pgstat_count_buffer_hit(operation->rel);
-
- pgstat_count_io_op(io_object, io_context, IOOP_HIT, 1, 0);
-
- if (VacuumCostActive)
- VacuumCostBalance += VacuumCostPageHit;
-
- continue;
- }
/* We found a buffer that we need to read in. */
- io_buffers[0] = buffers[i];
- io_pages[0] = BufferGetBlock(buffers[i]);
- io_first_block = blocknum + i;
+ Assert(io_buffers[0] == buffers[nblocks_done]);
+ io_pages[0] = BufferGetBlock(buffers[nblocks_done]);
io_buffers_len = 1;
/*
@@ -1549,85 +1892,70 @@ WaitReadBuffers(ReadBuffersOperation *operation)
* buffers at the same time? In this case we don't wait if we see an
* I/O already in progress. We already hold BM_IO_IN_PROGRESS for the
* head block, so we should get on with that I/O as soon as possible.
- * We'll come back to this block again, above.
+ *
+ * We'll come back to this block in the next call to
+ * StartReadBuffers() -> AsyncReadBuffers().
*/
- while ((i + 1) < nblocks &&
- WaitReadBuffersCanStartIO(buffers[i + 1], true))
+ for (int i = nblocks_done + 1; i < operation->nblocks; i++)
{
+ if (!ReadBuffersCanStartIO(buffers[i], true))
+ break;
/* Must be consecutive block numbers. */
- Assert(BufferGetBlockNumber(buffers[i + 1]) ==
- BufferGetBlockNumber(buffers[i]) + 1);
+ Assert(BufferGetBlockNumber(buffers[i - 1]) ==
+ BufferGetBlockNumber(buffers[i]) - 1);
+ Assert(io_buffers[io_buffers_len] == buffers[i]);
- io_buffers[io_buffers_len] = buffers[++i];
io_pages[io_buffers_len++] = BufferGetBlock(buffers[i]);
}
+ /* get a reference to wait for in WaitReadBuffers() */
+ pgaio_io_get_wref(ioh, &operation->io_wref);
+
+ /* provide the list of buffers to the completion callbacks */
+ pgaio_io_set_handle_data_32(ioh, (uint32 *) io_buffers, io_buffers_len);
+
+ pgaio_io_register_callbacks(ioh,
+ persistence == RELPERSISTENCE_TEMP ?
+ PGAIO_HCB_LOCAL_BUFFER_READV :
+ PGAIO_HCB_SHARED_BUFFER_READV,
+ flags);
+
+ pgaio_io_set_flag(ioh, ioh_flags);
+
+ /* ---
+ * Even though we're trying to issue IO asynchronously, track the time
+ * in smgrstartreadv():
+ * - if io_method == IOMETHOD_SYNC, we will always perform the IO
+ * immediately
+ * - the io method might not support the IO (e.g. worker IO for a temp
+ * table)
+ * ---
+ */
io_start = pgstat_prepare_io_time(track_io_timing);
- smgrreadv(operation->smgr, forknum, io_first_block, io_pages, io_buffers_len);
- pgstat_count_io_op_time(io_object, io_context, IOOP_READ, io_start,
- 1, io_buffers_len * BLCKSZ);
-
- /* Verify each block we read, and terminate the I/O. */
- for (int j = 0; j < io_buffers_len; ++j)
- {
- BufferDesc *bufHdr;
- Block bufBlock;
-
- if (persistence == RELPERSISTENCE_TEMP)
- {
- bufHdr = GetLocalBufferDescriptor(-io_buffers[j] - 1);
- bufBlock = LocalBufHdrGetBlock(bufHdr);
- }
- else
- {
- bufHdr = GetBufferDescriptor(io_buffers[j] - 1);
- bufBlock = BufHdrGetBlock(bufHdr);
- }
-
- /* check for garbage data */
- if (!PageIsVerifiedExtended((Page) bufBlock, io_first_block + j,
- PIV_LOG_WARNING | PIV_REPORT_STAT))
- {
- if ((operation->flags & READ_BUFFERS_ZERO_ON_ERROR) || zero_damaged_pages)
- {
- ereport(WARNING,
- (errcode(ERRCODE_DATA_CORRUPTED),
- errmsg("invalid page in block %u of relation %s; zeroing out page",
- io_first_block + j,
- relpath(operation->smgr->smgr_rlocator, forknum).str)));
- memset(bufBlock, 0, BLCKSZ);
- }
- else
- ereport(ERROR,
- (errcode(ERRCODE_DATA_CORRUPTED),
- errmsg("invalid page in block %u of relation %s",
- io_first_block + j,
- relpath(operation->smgr->smgr_rlocator, forknum).str)));
- }
-
- /* Set BM_VALID, terminate IO, and wake up any waiters */
- if (persistence == RELPERSISTENCE_TEMP)
- TerminateLocalBufferIO(bufHdr, false, BM_VALID, true);
- else
- TerminateBufferIO(bufHdr, false, BM_VALID, true, true);
-
- /* Report I/Os as completing individually. */
- TRACE_POSTGRESQL_BUFFER_READ_DONE(forknum, io_first_block + j,
- operation->smgr->smgr_rlocator.locator.spcOid,
- operation->smgr->smgr_rlocator.locator.dbOid,
- operation->smgr->smgr_rlocator.locator.relNumber,
- operation->smgr->smgr_rlocator.backend,
- false);
- }
+ smgrstartreadv(ioh, operation->smgr, forknum,
+ blocknum + nblocks_done,
+ io_pages, io_buffers_len);
+ pgstat_count_io_op_time(io_object, io_context, IOOP_READ,
+ io_start, 1, *nblocks_progress * BLCKSZ);
if (persistence == RELPERSISTENCE_TEMP)
pgBufferUsage.local_blks_read += io_buffers_len;
else
pgBufferUsage.shared_blks_read += io_buffers_len;
+ /*
+ * Track vacuum cost when issuing IO, not after waiting for it.
+ * Otherwise we could end up issuing a lot of IO in a short timespan,
+ * despite a low cost limit.
+ */
if (VacuumCostActive)
VacuumCostBalance += VacuumCostPageMiss * io_buffers_len;
+
+ *nblocks_progress = io_buffers_len;
+ did_start_io = true;
}
+
+ return did_start_io;
}
/*
--
2.48.1.76.g4e746b1a31.dirty
[text/x-diff] v2.8-0025-docs-Reframe-track_io_timing-related-docs-as-wa.patch (5.3K, ../../3yxd5r23zly5bytvgyktbxtxq2r3gbpi7xd4dugevh3h4w4q6c@lu6oatjjpltz/26-v2.8-0025-docs-Reframe-track_io_timing-related-docs-as-wa.patch)
download | inline diff:
From fff6267519bc2161b6b5c76923c69489f30f96f3 Mon Sep 17 00:00:00 2001
From: Andres Freund <andres@anarazel.de>
Date: Wed, 12 Mar 2025 13:25:16 -0400
Subject: [PATCH v2.8 25/38] docs: Reframe track_io_timing related docs as wait
time
With AIO it does not make sense anymore to track the time for each individual
IO, as multiple IOs can be in-flight at the same time. Instead we now track
the time spent *waiting* for IOs.
This should be reflected in the docs. While, so far, we only do a subset of
reads, and no other operations, via AIO, describing the GUC and view columns
as measuring IO waits is accurate for synchronous and asynchronous IO.
Discussion: https://postgr.es/m/5dzyoduxlvfg55oqtjyjehez5uoq6hnwgzor4kkybkfdgkj7ag@rbi4gsmzaczk
---
doc/src/sgml/config.sgml | 4 ++--
doc/src/sgml/monitoring.sgml | 12 ++++++------
2 files changed, 8 insertions(+), 8 deletions(-)
diff --git a/doc/src/sgml/config.sgml b/doc/src/sgml/config.sgml
index 8d53478f53a..69638d7c1f9 100644
--- a/doc/src/sgml/config.sgml
+++ b/doc/src/sgml/config.sgml
@@ -8549,7 +8549,7 @@ COPY postgres_log FROM '/full/path/to/logfile.csv' WITH csv;
</term>
<listitem>
<para>
- Enables timing of database I/O calls. This parameter is off by
+ Enables timing of database I/O waits. This parameter is off by
default, as it will repeatedly query the operating system for
the current time, which may cause significant overhead on some
platforms. You can use the <xref linkend="pgtesttiming"/> tool to
@@ -8583,7 +8583,7 @@ COPY postgres_log FROM '/full/path/to/logfile.csv' WITH csv;
</term>
<listitem>
<para>
- Enables timing of WAL I/O calls. This parameter is off by default,
+ Enables timing of WAL I/O waits. This parameter is off by default,
as it will repeatedly query the operating system for the current time,
which may cause significant overhead on some platforms.
You can use the <application>pg_test_timing</application> tool to
diff --git a/doc/src/sgml/monitoring.sgml b/doc/src/sgml/monitoring.sgml
index aaa6586d3a4..7598340072f 100644
--- a/doc/src/sgml/monitoring.sgml
+++ b/doc/src/sgml/monitoring.sgml
@@ -2733,7 +2733,7 @@ description | Waiting for a newly initialized WAL file to reach durable storage
<structfield>read_time</structfield> <type>double precision</type>
</para>
<para>
- Time spent in read operations in milliseconds (if
+ Time spent waiting for read operations in milliseconds (if
<xref linkend="guc-track-io-timing"/> is enabled and
<varname>object</varname> is not <literal>wal</literal>,
or if <xref linkend="guc-track-wal-io-timing"/> is enabled
@@ -2771,7 +2771,7 @@ description | Waiting for a newly initialized WAL file to reach durable storage
<structfield>write_time</structfield> <type>double precision</type>
</para>
<para>
- Time spent in write operations in milliseconds (if
+ Time spent waiting for write operations in milliseconds (if
<xref linkend="guc-track-io-timing"/> is enabled and
<varname>object</varname> is not <literal>wal</literal>,
or if <xref linkend="guc-track-wal-io-timing"/> is enabled
@@ -2799,7 +2799,7 @@ description | Waiting for a newly initialized WAL file to reach durable storage
<structfield>writeback_time</structfield> <type>double precision</type>
</para>
<para>
- Time spent in writeback operations in milliseconds (if
+ Time spent waiting for writeback operations in milliseconds (if
<xref linkend="guc-track-io-timing"/> is enabled, otherwise zero). This
includes the time spent queueing write-out requests and, potentially,
the time spent to write out the dirty data.
@@ -2835,7 +2835,7 @@ description | Waiting for a newly initialized WAL file to reach durable storage
<structfield>extend_time</structfield> <type>double precision</type>
</para>
<para>
- Time spent in extend operations in milliseconds. (if
+ Time spent waiting for extend operations in milliseconds. (if
<xref linkend="guc-track-io-timing"/> is enabled and
<varname>object</varname> is not <literal>wal</literal>,
or if <xref linkend="guc-track-wal-io-timing"/> is enabled
@@ -2909,7 +2909,7 @@ description | Waiting for a newly initialized WAL file to reach durable storage
<structfield>fsync_time</structfield> <type>double precision</type>
</para>
<para>
- Time spent in fsync operations in milliseconds (if
+ Time spent waiting for fsync operations in milliseconds (if
<xref linkend="guc-track-io-timing"/> is enabled and
<varname>object</varname> is not <literal>wal</literal>,
or if <xref linkend="guc-track-wal-io-timing"/> is enabled
@@ -2996,7 +2996,7 @@ description | Waiting for a newly initialized WAL file to reach durable storage
<note>
<para>
- Columns tracking I/O time will only be non-zero when
+ Columns tracking I/O wait time will only be non-zero when
<xref linkend="guc-track-io-timing"/> is enabled. The user should be
careful when referencing these columns in combination with their
corresponding I/O operations in case <varname>track_io_timing</varname>
--
2.48.1.76.g4e746b1a31.dirty
[text/x-diff] v2.8-0026-aio-Basic-read_stream-adjustments-for-real-AIO.patch (4.5K, ../../3yxd5r23zly5bytvgyktbxtxq2r3gbpi7xd4dugevh3h4w4q6c@lu6oatjjpltz/27-v2.8-0026-aio-Basic-read_stream-adjustments-for-real-AIO.patch)
download | inline diff:
From 6ddf6724030ee0c09bd8027252e244b78440514f Mon Sep 17 00:00:00 2001
From: Andres Freund <andres@anarazel.de>
Date: Fri, 14 Mar 2025 11:39:34 -0400
Subject: [PATCH v2.8 26/38] aio: Basic read_stream adjustments for real AIO
Adapt the read stream logic for real AIO:
- If AIO is enabled, we shouldn't issue advice, but if it isn't, we should
continue issuing advice
- AIO benefits from reading ahead with direct IO
- While in read_stream_look_ahead(), we can use AIO batch submission mode for
increased efficiency
There is one comment talking about max_ios logic with "real asynchronous I/O"
that I am not sure about, so I left it alone for now.
There are further improvements we should consider, e.g. waiting to issue IOs
until we can issue multiple IOs at once. But that's left for a future change,
since it would involve additional heuristics.
---
src/backend/storage/aio/read_stream.c | 39 ++++++++++++++++++++-------
1 file changed, 29 insertions(+), 10 deletions(-)
diff --git a/src/backend/storage/aio/read_stream.c b/src/backend/storage/aio/read_stream.c
index 0f1525f46c5..b44cc358f29 100644
--- a/src/backend/storage/aio/read_stream.c
+++ b/src/backend/storage/aio/read_stream.c
@@ -72,6 +72,7 @@
#include "postgres.h"
#include "miscadmin.h"
+#include "storage/aio.h"
#include "storage/fd.h"
#include "storage/smgr.h"
#include "storage/read_stream.h"
@@ -99,6 +100,7 @@ struct ReadStream
int16 pinned_buffers;
int16 distance;
int16 initialized_buffers;
+ bool sync_mode;
bool advice_enabled;
bool temporary;
@@ -420,6 +422,13 @@ read_stream_start_pending_read(ReadStream *stream)
static void
read_stream_look_ahead(ReadStream *stream)
{
+ /*
+ * Allow amortizing the cost of submitting IO over multiple IOs. This
+ * requires that we don't do any operations that could lead to a deadlock
+ * with staged-but-unsubmitted IO.
+ */
+ pgaio_enter_batchmode();
+
while (stream->ios_in_progress < stream->max_ios &&
stream->pinned_buffers + stream->pending_read_nblocks < stream->distance)
{
@@ -467,6 +476,7 @@ read_stream_look_ahead(ReadStream *stream)
{
/* We've hit the buffer or I/O limit. Rewind and stop here. */
read_stream_unget_block(stream, blocknum);
+ pgaio_exit_batchmode();
return;
}
}
@@ -501,6 +511,8 @@ read_stream_look_ahead(ReadStream *stream)
* time.
*/
Assert(stream->pinned_buffers > 0 || stream->distance == 0);
+
+ pgaio_exit_batchmode();
}
/*
@@ -556,12 +568,12 @@ read_stream_begin_impl(int flags,
max_ios = get_tablespace_io_concurrency(tablespace_id);
/*
- * XXX Since we don't have asynchronous I/O yet, if direct I/O is enabled
- * then just behave as though I/O concurrency is set to 0. Otherwise we
- * would look ahead pinning many buffers for no benefit, for lack of
- * advice and AIO.
+ * If real asynchronous I/O is disabled, and direct I/O is enabled, just
+ * behave as though I/O concurrency is set to 0. Otherwise we would look
+ * ahead pinning many buffers for no benefit, as the advice-based
+ * readahead doesn't support direct I/O.
*/
- if (io_direct_flags & IO_DIRECT_DATA)
+ if (io_method == IOMETHOD_SYNC && (io_direct_flags & IO_DIRECT_DATA))
max_ios = 0;
/* Cap to INT16_MAX to avoid overflowing below */
@@ -641,15 +653,19 @@ read_stream_begin_impl(int flags,
stream->per_buffer_data = (void *)
MAXALIGN(&stream->ios[Max(1, max_ios)]);
+ stream->sync_mode = io_method == IOMETHOD_SYNC;
+
#ifdef USE_PREFETCH
/*
- * This system supports prefetching advice. We can use it as long as
- * direct I/O isn't enabled, the caller hasn't promised sequential access
- * (overriding our detection heuristics), and max_ios hasn't been set to
- * zero.
+ * This system supports prefetching advice.
+ *
+ * Issue advice only if AIO is not used, direct I/O isn't enabled, the
+ * caller hasn't promised sequential access (overriding our detection
+ * heuristics), and max_ios hasn't been set to zero.
*/
- if ((io_direct_flags & IO_DIRECT_DATA) == 0 &&
+ if (stream->sync_mode &&
+ (io_direct_flags & IO_DIRECT_DATA) == 0 &&
(flags & READ_STREAM_SEQUENTIAL) == 0 &&
max_ios > 0)
stream->advice_enabled = true;
@@ -659,6 +675,9 @@ read_stream_begin_impl(int flags,
* For now, max_ios = 0 is interpreted as max_ios = 1 with advice disabled
* above. If we had real asynchronous I/O we might need a slightly
* different definition.
+ *
+ * FIXME: Not sure what different definition we would need? I guess we
+ * could add the READ_BUFFERS_SYNCHRONOUSLY flag automatically?
*/
if (max_ios == 0)
max_ios = 1;
--
2.48.1.76.g4e746b1a31.dirty
[text/x-diff] v2.8-0027-aio-Experimental-heuristics-to-increase-batchin.patch (1.6K, ../../3yxd5r23zly5bytvgyktbxtxq2r3gbpi7xd4dugevh3h4w4q6c@lu6oatjjpltz/28-v2.8-0027-aio-Experimental-heuristics-to-increase-batchin.patch)
download | inline diff:
From bb43dc77fb2e4e9f7244fbd4680d713040d520ce Mon Sep 17 00:00:00 2001
From: Andres Freund <andres@anarazel.de>
Date: Fri, 14 Mar 2025 12:16:17 -0400
Subject: [PATCH v2.8 27/38] aio: Experimental heuristics to increase batching
in read_stream.c
Author:
Reviewed-by:
Discussion: https://postgr.es/m/
Backpatch:
---
src/backend/storage/aio/read_stream.c | 24 ++++++++++++++++++++++++
1 file changed, 24 insertions(+)
diff --git a/src/backend/storage/aio/read_stream.c b/src/backend/storage/aio/read_stream.c
index b44cc358f29..0667efe31ca 100644
--- a/src/backend/storage/aio/read_stream.c
+++ b/src/backend/storage/aio/read_stream.c
@@ -422,6 +422,30 @@ read_stream_start_pending_read(ReadStream *stream)
static void
read_stream_look_ahead(ReadStream *stream)
{
+ /*
+ * Batch-submitting multiple IOs is more efficient than doing so
+ * one-by-one. If we just ramp up to the max, we'll only be allowed to
+ * submit one io_combine_limit sized IO. Defer submitting IO in that case.
+ *
+ * FIXME: This needs better heuristics.
+ */
+#if 1
+ if (!stream->sync_mode && stream->distance > (io_combine_limit * 8))
+ {
+ if (stream->pinned_buffers + stream->pending_read_nblocks > ((stream->distance * 3) / 4))
+ {
+#if 0
+ ereport(LOG,
+ errmsg("reduce reduce reduce: pinned: %d, pending: %d, distance: %d",
+ stream->pinned_buffers,
+ stream->pending_read_nblocks,
+ stream->distance));
+#endif
+ return;
+ }
+ }
+#endif
+
/*
* Allow amortizing the cost of submitting IO over multiple IOs. This
* requires that we don't do any operations that could lead to a deadlock
--
2.48.1.76.g4e746b1a31.dirty
[text/x-diff] v2.8-0028-aio-Add-test_aio-module.patch (42.4K, ../../3yxd5r23zly5bytvgyktbxtxq2r3gbpi7xd4dugevh3h4w4q6c@lu6oatjjpltz/29-v2.8-0028-aio-Add-test_aio-module.patch)
download | inline diff:
From 016a1d9a237305b8e8967c485c99bd2e40d0f209 Mon Sep 17 00:00:00 2001
From: Andres Freund <andres@anarazel.de>
Date: Wed, 22 Jan 2025 13:44:54 -0500
Subject: [PATCH v2.8 28/38] aio: Add test_aio module
---
src/include/storage/buf_internals.h | 6 +
src/backend/storage/buffer/bufmgr.c | 8 +-
src/test/modules/Makefile | 1 +
src/test/modules/meson.build | 1 +
src/test/modules/test_aio/.gitignore | 2 +
src/test/modules/test_aio/Makefile | 27 +
src/test/modules/test_aio/meson.build | 37 ++
src/test/modules/test_aio/t/001_aio.pl | 674 ++++++++++++++++++++
src/test/modules/test_aio/test_aio--1.0.sql | 97 +++
src/test/modules/test_aio/test_aio.c | 616 ++++++++++++++++++
src/test/modules/test_aio/test_aio.control | 3 +
11 files changed, 1466 insertions(+), 6 deletions(-)
create mode 100644 src/test/modules/test_aio/.gitignore
create mode 100644 src/test/modules/test_aio/Makefile
create mode 100644 src/test/modules/test_aio/meson.build
create mode 100644 src/test/modules/test_aio/t/001_aio.pl
create mode 100644 src/test/modules/test_aio/test_aio--1.0.sql
create mode 100644 src/test/modules/test_aio/test_aio.c
create mode 100644 src/test/modules/test_aio/test_aio.control
diff --git a/src/include/storage/buf_internals.h b/src/include/storage/buf_internals.h
index 8a52ace37cd..4b1f52a9fc8 100644
--- a/src/include/storage/buf_internals.h
+++ b/src/include/storage/buf_internals.h
@@ -434,6 +434,12 @@ extern void IssuePendingWritebacks(WritebackContext *wb_context, IOContext io_co
extern void ScheduleBufferTagForWriteback(WritebackContext *wb_context,
IOContext io_context, BufferTag *tag);
+/* solely to make it easier to write tests */
+extern bool StartBufferIO(BufferDesc *buf, bool forInput, bool nowait);
+extern void TerminateBufferIO(BufferDesc *buf, bool clear_dirty, uint32 set_flag_bits,
+ bool forget_owner, bool syncio);
+
+
/* freelist.c */
extern IOContext IOContextForStrategy(BufferAccessStrategy strategy);
extern BufferDesc *StrategyGetBuffer(BufferAccessStrategy strategy,
diff --git a/src/backend/storage/buffer/bufmgr.c b/src/backend/storage/buffer/bufmgr.c
index 2691757016c..860886d4d90 100644
--- a/src/backend/storage/buffer/bufmgr.c
+++ b/src/backend/storage/buffer/bufmgr.c
@@ -515,10 +515,6 @@ static uint32 WaitBufHdrUnlocked(BufferDesc *buf);
static int SyncOneBuffer(int buf_id, bool skip_recently_used,
WritebackContext *wb_context);
static void WaitIO(BufferDesc *buf);
-static bool StartBufferIO(BufferDesc *buf, bool forInput, bool nowait);
-static void TerminateBufferIO(BufferDesc *buf, bool clear_dirty,
- uint32 set_flag_bits, bool forget_owner,
- bool syncio);
static void AbortBufferIO(Buffer buffer);
static void shared_buffer_write_error_callback(void *arg);
static void local_buffer_write_error_callback(void *arg);
@@ -5927,7 +5923,7 @@ WaitIO(BufferDesc *buf)
* find out if they can perform the I/O as part of a larger operation, without
* waiting for the answer or distinguishing the reasons why not.
*/
-static bool
+bool
StartBufferIO(BufferDesc *buf, bool forInput, bool nowait)
{
uint32 buf_state;
@@ -5984,7 +5980,7 @@ StartBufferIO(BufferDesc *buf, bool forInput, bool nowait)
* resource owner. (forget_owner=false is used when the resource owner itself
* is being released)
*/
-static void
+void
TerminateBufferIO(BufferDesc *buf, bool clear_dirty, uint32 set_flag_bits,
bool forget_owner, bool syncio)
{
diff --git a/src/test/modules/Makefile b/src/test/modules/Makefile
index 4e4be3fa511..aa1d27bbed3 100644
--- a/src/test/modules/Makefile
+++ b/src/test/modules/Makefile
@@ -14,6 +14,7 @@ SUBDIRS = \
oauth_validator \
plsample \
spgist_name_ops \
+ test_aio \
test_bloomfilter \
test_copy_callbacks \
test_custom_rmgrs \
diff --git a/src/test/modules/meson.build b/src/test/modules/meson.build
index 2b057451473..4fda1acfb32 100644
--- a/src/test/modules/meson.build
+++ b/src/test/modules/meson.build
@@ -1,5 +1,6 @@
# Copyright (c) 2022-2025, PostgreSQL Global Development Group
+subdir('test_aio')
subdir('brin')
subdir('commit_ts')
subdir('delay_execution')
diff --git a/src/test/modules/test_aio/.gitignore b/src/test/modules/test_aio/.gitignore
new file mode 100644
index 00000000000..716e17f5a2a
--- /dev/null
+++ b/src/test/modules/test_aio/.gitignore
@@ -0,0 +1,2 @@
+# Generated subdirectories
+/tmp_check/
diff --git a/src/test/modules/test_aio/Makefile b/src/test/modules/test_aio/Makefile
new file mode 100644
index 00000000000..87d5315ba00
--- /dev/null
+++ b/src/test/modules/test_aio/Makefile
@@ -0,0 +1,27 @@
+# src/test/modules/delay_execution/Makefile
+
+PGFILEDESC = "test_aio - test code for AIO"
+
+MODULE_big = test_aio
+OBJS = \
+ $(WIN32RES) \
+ test_aio.o
+
+EXTENSION = test_aio
+DATA = test_aio--1.0.sql
+
+TAP_TESTS = 1
+
+export enable_injection_points
+export with_liburing
+
+ifdef USE_PGXS
+PG_CONFIG = pg_config
+PGXS := $(shell $(PG_CONFIG) --pgxs)
+include $(PGXS)
+else
+subdir = src/test/modules/test_aio
+top_builddir = ../../../..
+include $(top_builddir)/src/Makefile.global
+include $(top_srcdir)/contrib/contrib-global.mk
+endif
diff --git a/src/test/modules/test_aio/meson.build b/src/test/modules/test_aio/meson.build
new file mode 100644
index 00000000000..ac846b2b6f3
--- /dev/null
+++ b/src/test/modules/test_aio/meson.build
@@ -0,0 +1,37 @@
+# Copyright (c) 2022-2024, PostgreSQL Global Development Group
+
+test_aio_sources = files(
+ 'test_aio.c',
+)
+
+if host_system == 'windows'
+ test_aio_sources += rc_lib_gen.process(win32ver_rc, extra_args: [
+ '--NAME', 'test_aio',
+ '--FILEDESC', 'test_aio - test code for AIO',])
+endif
+
+test_aio = shared_module('test_aio',
+ test_aio_sources,
+ kwargs: pg_test_mod_args,
+)
+test_install_libs += test_aio
+
+test_install_data += files(
+ 'test_aio.control',
+ 'test_aio--1.0.sql',
+)
+
+tests += {
+ 'name': 'test_aio',
+ 'sd': meson.current_source_dir(),
+ 'bd': meson.current_build_dir(),
+ 'tap': {
+ 'env': {
+ 'enable_injection_points': get_option('injection_points') ? 'yes' : 'no',
+ 'with_liburing': liburing.found() ? 'yes' : 'no',
+ },
+ 'tests': [
+ 't/001_aio.pl',
+ ],
+ },
+}
diff --git a/src/test/modules/test_aio/t/001_aio.pl b/src/test/modules/test_aio/t/001_aio.pl
new file mode 100644
index 00000000000..ecd1f3c2f17
--- /dev/null
+++ b/src/test/modules/test_aio/t/001_aio.pl
@@ -0,0 +1,674 @@
+# Copyright (c) 2025, PostgreSQL Global Development Group
+
+use strict;
+use warnings FATAL => 'all';
+
+use PostgreSQL::Test::Cluster;
+use PostgreSQL::Test::Utils;
+use Test::More;
+
+
+###
+# Test io_method=worker
+###
+my $node_worker = create_node('worker');
+$node_worker->start();
+
+test_generic('worker', $node_worker);
+SKIP:
+{
+ skip 'Injection points not supported by this build', 1
+ unless $ENV{enable_injection_points} eq 'yes';
+ test_inject_worker('worker', $node_worker);
+}
+
+$node_worker->stop();
+
+
+###
+# Test io_method=io_uring
+###
+
+if ($ENV{with_liburing} eq 'yes')
+{
+ my $node_uring = create_node('io_uring');
+ $node_uring->start();
+ test_generic('io_uring', $node_uring);
+ $node_uring->stop();
+}
+
+
+###
+# Test io_method=sync
+###
+
+my $node_sync = create_node('sync');
+
+# just to have one test not use the default auto-tuning
+
+$node_sync->append_conf(
+ 'postgresql.conf', qq(
+io_max_concurrency=4
+));
+
+$node_sync->start();
+test_generic('sync', $node_sync);
+$node_sync->stop();
+
+done_testing();
+
+
+###
+# Test Helpers
+###
+
+sub create_node
+{
+ my $io_method = shift;
+
+ my $node = PostgreSQL::Test::Cluster->new($io_method);
+
+ # Want to test initdb for each IO method, otherwise we could just reuse
+ # the cluster.
+ #
+ # Unfortunately Cluster::init() puts PG_TEST_INITDB_EXTRA_OPTS after the
+ # options specified by ->extra, if somebody puts -c io_method=xyz in
+ # PG_TEST_INITDB_EXTRA_OPTS it would break this test. Fix that up if we
+ # detect it.
+ local $ENV{PG_TEST_INITDB_EXTRA_OPTS} = $ENV{PG_TEST_INITDB_EXTRA_OPTS};
+ if (defined $ENV{PG_TEST_INITDB_EXTRA_OPTS} &&
+ $ENV{PG_TEST_INITDB_EXTRA_OPTS} =~ m/io_method=/)
+ {
+ $ENV{PG_TEST_INITDB_EXTRA_OPTS} .= " -c io_method=$io_method";
+ }
+
+ $node->init(extra => [ '-c', "io_method=$io_method" ]);
+
+ $node->append_conf(
+ 'postgresql.conf', qq(
+shared_preload_libraries=test_aio
+log_min_messages = 'DEBUG3'
+log_statement=all
+restart_after_crash=false
+));
+
+ ok(1, "$io_method: initdb");
+
+ return $node;
+}
+
+sub psql_like
+{
+ my $io_method = shift;
+ my $psql = shift;
+ my $name = shift;
+ my $sql = shift;
+ my $expected_stdout = shift;
+ my $expected_stderr = shift;
+ my ($cmdret, $output);
+
+ ($output, $cmdret) = $psql->query($sql);
+
+ like($output, $expected_stdout, "$io_method: $name: expected stdout");
+ like($psql->{stderr}, $expected_stderr,
+ "$io_method: $name: expected stderr");
+ $psql->{stderr} = '';
+}
+
+sub query_wait_block
+{
+ my $io_method = shift;
+ my $node = shift;
+ my $psql = shift;
+ my $name = shift;
+ my $sql = shift;
+ my $waitfor = shift;
+
+ my $pid = $psql->query_safe('SELECT pg_backend_pid()');
+
+ $psql->{stdin} .= qq($sql;\n);
+ $psql->{run}->pump_nb();
+ ok(1, "$io_method: $name: issued sql");
+
+ $node->poll_query_until('postgres',
+ qq(SELECT wait_event FROM pg_stat_activity WHERE pid = $pid),
+ $waitfor,);
+ ok(1, "$io_method: $name: observed $waitfor wait event");
+}
+
+
+###
+# Sub-tests
+###
+
+sub test_handle
+{
+ my $io_method = shift;
+ my $node = shift;
+
+ my $psql = $node->background_psql('postgres', on_error_stop => 0);
+
+ # leak warning: implicit xact
+ psql_like(
+ $io_method,
+ $psql,
+ "handle_get() leak in implicit xact",
+ qq(SELECT handle_get()),
+ qr/^$/,
+ qr/leaked AIO handle/,
+ "$io_method: leaky handle_get() warns");
+
+ # leak warning: explicit xact
+ psql_like(
+ $io_method, $psql,
+ "handle_get() leak in explicit xact",
+ qq(BEGIN; SELECT handle_get(); COMMIT),
+ qr/^$/, qr/leaked AIO handle/);
+
+
+ # leak warning: explicit xact, rollback
+ psql_like(
+ $io_method,
+ $psql,
+ "handle_get() leak in explicit xact, rollback",
+ qq(BEGIN; SELECT handle_get(); ROLLBACK;),
+ qr/^$/,
+ qr/leaked AIO handle/);
+
+ # leak warning: subtrans
+ psql_like(
+ $io_method,
+ $psql,
+ "handle_get() leak in subxact",
+ qq(BEGIN; SAVEPOINT foo; SELECT handle_get(); COMMIT;),
+ qr/^$/,
+ qr/leaked AIO handle/);
+
+ # leak warning + error: released in different command (thus resowner)
+ psql_like(
+ $io_method,
+ $psql,
+ "handle_release() in different command",
+ qq(BEGIN; SELECT handle_get(); SELECT handle_release_last(); COMMIT;),
+ qr/^$/,
+ qr/leaked AIO handle.*release in unexpected state/ms);
+
+ # no leak, release in same command
+ psql_like(
+ $io_method,
+ $psql,
+ "handle_release() in same command",
+ qq(BEGIN; SELECT handle_get() UNION ALL SELECT handle_release_last(); COMMIT;),
+ qr/^$/,
+ qr/^$/);
+
+ # normal handle use
+ psql_like($io_method, $psql, "handle_get_release()",
+ qq(SELECT handle_get_release()),
+ qr/^$/, qr/^$/);
+
+ # should error out, API violation
+ psql_like($io_method, $psql, "handle_get_twice()",
+ qq(SELECT handle_get_release()),
+ qr/^$/, qr/^$/);
+
+ # recover after error in implicit xact
+ psql_like(
+ $io_method,
+ $psql,
+ "handle error recovery in implicit xact",
+ qq(SELECT handle_get_and_error(); SELECT 'ok', handle_get_release()),
+ qr/^|ok$/,
+ qr/ERROR.*as you command/);
+
+ # recover after error in implicit xact
+ psql_like(
+ $io_method,
+ $psql,
+ "handle error recovery in explicit xact",
+ qq(BEGIN; SELECT handle_get_and_error(); SELECT handle_get_release(), 'ok'; COMMIT;),
+ qr/^|ok$/,
+ qr/ERROR.*as you command/);
+
+ # recover after error in subtrans
+ psql_like(
+ $io_method,
+ $psql,
+ "handle error recovery in explicit subxact",
+ qq(BEGIN; SAVEPOINT foo; SELECT handle_get_and_error(); ROLLBACK TO SAVEPOINT foo; SELECT handle_get_release(); ROLLBACK;),
+ qr/^|ok$/,
+ qr/ERROR.*as you command/);
+
+ $psql->quit();
+}
+
+
+sub test_batch
+{
+ my $io_method = shift;
+ my $node = shift;
+
+ my $psql = $node->background_psql('postgres', on_error_stop => 0);
+
+ # leak warning & recovery: implicit xact
+ psql_like(
+ $io_method,
+ $psql,
+ "batch_start() leak & cleanup in implicit xact",
+ qq(SELECT batch_start()),
+ qr/^$/,
+ qr/open AIO batch at end/,
+ "$io_method: leaky batch_start() warns");
+
+ # leak warning & recovery: explicit xact
+ psql_like(
+ $io_method,
+ $psql,
+ "batch_start() leak & cleanup in explicit xact",
+ qq(BEGIN; SELECT batch_start(); COMMIT;),
+ qr/^$/,
+ qr/open AIO batch at end/,
+ "$io_method: leaky batch_start() warns");
+
+
+ # leak warning & recovery: explicit xact, rollback
+ #
+ # XXX: This doesn't fail right now, due to not getting a chance to do
+ # something at transaction command commit. That's not a correctness issue,
+ # it just means it's a bit harder to find buggy code.
+ #psql_like($io_method, $psql,
+ # "batch_start() leak & cleanup after abort",
+ # qq(BEGIN; SELECT batch_start(); ROLLBACK;),
+ # qr/^$/,
+ # qr/open AIO batch at end/, "$io_method: leaky batch_start() warns");
+
+ # no warning, batch closed in same command
+ psql_like(
+ $io_method,
+ $psql,
+ "batch_start(), batch_end() works",
+ qq(SELECT batch_start() UNION ALL SELECT batch_end()),
+ qr/^$/,
+ qr/^$/,
+ "$io_method: batch_start(), batch_end()");
+
+ $psql->quit();
+}
+
+sub test_io_error
+{
+ my $io_method = shift;
+ my $node = shift;
+ my ($ret, $output);
+
+ my $psql = $node->background_psql('postgres', on_error_stop => 0);
+
+ # verify the error is reported in custom C code
+ ($output, $ret) = $psql->query(
+ qq(SELECT read_rel_block_ll('tbl_corr', 1, wait_complete=>true);));
+ is($ret, 1, "$io_method: read_rel_block_ll() of tbl_corr page fails");
+ like(
+ $psql->{stderr},
+ qr/invalid page in block 1 of relation base\/.*/,
+ "$io_method: read_rel_block_ll() of tbl_corr page reports error");
+ $psql->{stderr} = '';
+
+ # verify the error is reported for bufmgr reads
+ ($output, $ret) =
+ $psql->query(qq(SELECT count(*) FROM tbl_corr WHERE ctid = '(1, 1)'));
+ is($ret, 1, "$io_method: tid scan reading tbl_corr block fails");
+ like(
+ $psql->{stderr},
+ qr/invalid page in block 1 of relation base\/.*/,
+ "$io_method: tid scan reading tbl_corr block reports error");
+ $psql->{stderr} = '';
+
+ # verify the error is reported for bufmgr reads
+ ($output, $ret) =
+ $psql->query(qq(SELECT count(*) FROM tbl_corr WHERE ctid = '(1, 1)'));
+ is($ret, 1, "$io_method: sequential scan reading tbl_corr block fails");
+ like(
+ $psql->{stderr},
+ qr/invalid page in block 1 of relation base\/.*/,
+ "$io_method: sequential scan reading tbl_corr block reports error");
+ $psql->{stderr} = '';
+
+ $psql->quit();
+}
+
+# Test interplay between StartBufferIO and TerminateBufferIO
+sub test_startwait_io
+{
+ my $io_method = shift;
+ my $node = shift;
+ my ($ret, $output);
+
+ my $psql_a = $node->background_psql('postgres', on_error_stop => 0);
+ my $psql_b = $node->background_psql('postgres', on_error_stop => 0);
+
+ # create a buffer we can play around with
+ ($output, $ret) =
+ $psql_a->query(qq(SELECT buffer_create_toy('tbl_ok', 1);));
+ is($ret, 0, "$io_method: create toy succeeds");
+ like($output, qr/^\d+$/, "$io_method: create toy returns numeric");
+ my $buf_id = $output;
+
+ # check that one backend can perform StartBufferIO
+ psql_like(
+ $io_method,
+ $psql_a,
+ "first StartBufferIO",
+ qq(SELECT buffer_call_start_io($buf_id, for_input=>true, nowait=>false);),
+ qr/^t$/,
+ qr/^$/);
+
+ # but not twice on the same buffer (non-waiting)
+ psql_like(
+ $io_method,
+ $psql_a,
+ "second StartBufferIO fails, same session",
+ qq(SELECT buffer_call_start_io($buf_id, for_input=>true, nowait=>true);),
+ qr/^f$/,
+ qr/^$/);
+ psql_like(
+ $io_method,
+ $psql_b,
+ "second StartBufferIO fails, other session",
+ qq(SELECT buffer_call_start_io($buf_id, for_input=>true, nowait=>true);),
+ qr/^f$/,
+ qr/^$/);
+
+ # start io in a different session, will block
+ query_wait_block(
+ $io_method,
+ $node,
+ $psql_b,
+ "blocking start buffer io",
+ qq(SELECT buffer_call_start_io($buf_id, for_input=>true, nowait=>false);),
+ "BufferIo");
+
+ # Terminate the IO, without marking it as success, this should trigger the
+ # waiting session to be able to start the io
+ ($output, $ret) = $psql_a->query(
+ qq(SELECT buffer_call_terminate_io($buf_id, for_input=>true, succeed=>false, io_error=>false, syncio=>true);)
+ );
+ is($ret, 0,
+ "$io_method: blocking start buffer io, terminating io, not valid");
+
+ # Because the IO was terminated, but not marked as valid, second session should get the right to start io
+ pump_until($psql_b->{run}, $psql_b->{timeout}, \$psql_b->{stdout}, qr/t/);
+ ok(1, "$io_method: blocking start buffer io, can start io");
+
+ # terminate the IO again
+ $psql_b->query_safe(
+ qq(SELECT buffer_call_terminate_io($buf_id, for_input=>true, succeed=>false, io_error=>false, syncio=>true);)
+ );
+
+
+ # same as the above scenario, but mark IO as having succeeded
+ ($output, $ret) =
+ $psql_a->query(qq(SELECT buffer_call_start_io($buf_id, true, false);));
+ is($ret, 0,
+ "$io_method: blocking buffer io w/ success: first start buffer io succeeds"
+ );
+ is($output, "t",
+ "$io_method: blocking buffer io w/ success: first start buffer io returns true"
+ );
+
+
+ # start io in a different session, will block
+ query_wait_block(
+ $io_method,
+ $node,
+ $psql_b,
+ "blocking start buffer io",
+ qq(SELECT buffer_call_start_io($buf_id, for_input=>true, nowait=>false);),
+ "BufferIo");
+
+ ($output, $ret) = $psql_a->query(
+ qq(SELECT buffer_call_terminate_io($buf_id, for_input=>true, succeed=>true, io_error=>false, syncio=>true);)
+ );
+ is($ret, 0,
+ "$io_method: blocking start buffer io, terminating IO, valid");
+
+ # Because the IO was terminated, and marked as valid, second session should complete but not need io
+ pump_until($psql_b->{run}, $psql_b->{timeout}, \$psql_b->{stdout}, qr/f/);
+ ok(1, "$io_method: blocking start buffer io, no need to start io");
+
+
+ # buffer is valid now, make it invalid again
+ $buf_id = $psql_a->query_safe(qq(SELECT buffer_create_toy('tbl_ok', 1);));
+
+ $psql_a->quit();
+ $psql_b->quit();
+}
+
+# Test that if the backend issuing a read doesn't wait for the IO's
+# completion, another backend can complete the IO
+sub test_complete_foreign
+{
+ my $io_method = shift;
+ my $node = shift;
+ my ($ret, $output);
+
+ my $psql_a = $node->background_psql('postgres', on_error_stop => 0);
+ my $psql_b = $node->background_psql('postgres', on_error_stop => 0);
+
+ # Issue IO without waiting for completion, then sleep
+ $psql_a->query_safe(
+ qq(SELECT read_rel_block_ll('tbl_ok', 1, wait_complete=>false);));
+
+ # Check that another backend can read the relevant block
+ psql_like(
+ $io_method,
+ $psql_b,
+ "completing read started by sleeping backend",
+ qq(SELECT count(*) FROM tbl_ok WHERE ctid = '(1,1)' LIMIT 1),
+ qr/^1$/,
+ qr/^$/);
+
+ # Issue IO without waiting for completion, then exit
+ $psql_a->query_safe(
+ qq(SELECT read_rel_block_ll('tbl_ok', 1, wait_complete=>false);));
+ $psql_a->reconnect_and_clear();
+
+ # Check that another backend can read the relevant block
+ psql_like(
+ $io_method,
+ $psql_b,
+ "completing read started by exited backend",
+ qq(SELECT count(*) FROM tbl_ok WHERE ctid = '(1,1)' LIMIT 1),
+ qr/^1$/,
+ qr/^$/);
+
+ # Read a tbl_corr block, then sleep. The other session will retry the IO
+ # and also fail. The easiest thing to verify that seems to be to check
+ # that both are in the log.
+ my $log_location = -s $node->logfile;
+ $psql_a->query_safe(
+ qq(SELECT read_rel_block_ll('tbl_corr', 1, wait_complete=>false);));
+
+ psql_like(
+ $io_method,
+ $psql_b,
+ "completing read of tbl_corr block started by other backend",
+ qq(SELECT count(*) FROM tbl_corr WHERE ctid = '(1,1)' LIMIT 1),
+ qr/^$/,
+ qr/invalid page in block/);
+
+ # The log message issued for the read_rel_block_ll() should be logged as a LOG
+ $node->wait_for_log(qr/LOG[^\n]+invalid page in/, $log_location);
+ ok(1,
+ "$io_method: completing read of tbl_corr block started by other backend: LOG message for background read"
+ );
+
+ # But for the SELECT, it should be an ERROR
+ $log_location =
+ $node->wait_for_log(qr/ERROR[^\n]+invalid page in/, $log_location);
+ ok(1,
+ "$io_method: completing read of tbl_corr block started by other backend: ERROR message for foreground read"
+ );
+
+ $psql_a->quit();
+ $psql_b->quit();
+}
+
+sub test_inject
+{
+ my $io_method = shift;
+ my $node = shift;
+ my ($ret, $output);
+
+ my $psql = $node->background_psql('postgres', on_error_stop => 0);
+
+ # injected what we'd expect
+ $psql->query_safe(qq(SELECT inj_io_short_read_attach(8192);));
+ $psql->query_safe(qq(SELECT invalidate_rel_block('tbl_ok', 2);));
+ ($output, $ret) =
+ $psql->query(qq(SELECT count(*) FROM tbl_ok WHERE ctid = '(2, 1)';));
+ is($ret, 0,
+ "$io_method: injection point not triggering failure succeeds");
+
+ # injected a read shorter than a single block, expecting error
+ $psql->query_safe(qq(SELECT inj_io_short_read_attach(17);));
+ $psql->query_safe(qq(SELECT invalidate_rel_block('tbl_ok', 2);));
+ ($output, $ret) =
+ $psql->query(qq(SELECT count(*) FROM tbl_ok WHERE ctid = '(2, 1)';));
+ is($ret, 1, "$io_method: single block short read fails");
+ like(
+ $psql->{stderr},
+ qr/ERROR:.*could not read blocks 2\.\.2 in file "base\/.*": read only 0 of 8192 bytes/,
+ "$io_method: single block short read reports error");
+ $psql->{stderr} = '';
+
+ # shorten multi-block read to a single block, should retry
+ $psql->query_safe(
+ qq(
+SELECT invalidate_rel_block('tbl_ok', 0);
+SELECT invalidate_rel_block('tbl_ok', 1);
+SELECT invalidate_rel_block('tbl_ok', 2);
+SELECT invalidate_rel_block('tbl_ok', 3);
+/* gap */
+SELECT invalidate_rel_block('tbl_ok', 5);
+SELECT invalidate_rel_block('tbl_ok', 6);
+SELECT invalidate_rel_block('tbl_ok', 7);
+SELECT invalidate_rel_block('tbl_ok', 8);
+
+SELECT inj_io_short_read_attach(8192);
+ ));
+ ($output, $ret) = $psql->query(qq(SELECT count(*) FROM tbl_ok;));
+ is($ret, 0, "$io_method: multi block short read is retried");
+
+ # verify that page verification errors are detected even as part of a
+ # shortened multi-block read (tbl_corr, block 1 is tbl_corred)
+ $psql->query_safe(
+ qq(
+SELECT invalidate_rel_block('tbl_corr', 0);
+SELECT invalidate_rel_block('tbl_corr', 1);
+SELECT invalidate_rel_block('tbl_corr', 2);
+SELECT inj_io_short_read_attach(8192);
+ ));
+ ($output, $ret) =
+ $psql->query(qq(SELECT count(*) FROM tbl_corr WHERE ctid < '(2, 1)'));
+ is($ret, 1,
+ "$io_method: shortened multi-block read detects invalid page");
+ like(
+ $psql->{stderr},
+ qr/ERROR:.*invalid page in block 1 of relation base\/.*/,
+ "$io_method: shortened multi-block reads reports invalid page");
+ $psql->{stderr} = '';
+
+ # trigger a hard error, should error out
+ $psql->query_safe(
+ qq(
+SELECT inj_io_short_read_attach(-errno_from_string('EIO'));
+SELECT invalidate_rel_block('tbl_ok', 2);
+ ));
+ ($output, $ret) =
+ $psql->query(qq(SELECT count(*) FROM tbl_ok; SELECT 1;));
+ is($ret, 1, "$io_method: hard IO error is detected");
+ like(
+ $psql->{stderr},
+ qr/ERROR:.*could not read blocks 2\.\.2 in file \"base\/.*\": Input\/output error/,
+ "$io_method: hard IO error is reported");
+ $psql->{stderr} = '';
+
+ $psql->query_safe(
+ qq(
+SELECT inj_io_short_read_detach();
+ ));
+
+ $psql->quit();
+}
+
+sub test_inject_worker
+{
+ my $io_method = shift;
+ my $node = shift;
+ my ($ret, $output);
+
+ my $psql = $node->background_psql('postgres', on_error_stop => 0);
+
+ # trigger a failure to reopen, should error out, but should recover
+ $psql->query_safe(
+ qq(
+SELECT inj_io_reopen_attach();
+SELECT invalidate_rel_block('tbl_ok', 1);
+ ));
+ ($output, $ret) = $psql->query(qq(SELECT count(*) FROM tbl_ok;));
+ is($ret, 1, "$io_method: failure to open is detected");
+ like(
+ $psql->{stderr},
+ qr/ERROR:.*could not read blocks 1\.\.1 in file "base\/.*": No such file or directory/,
+ "$io_method: failure to open is reported");
+ $psql->{stderr} = '';
+
+ $psql->query_safe(
+ qq(
+SELECT inj_io_reopen_detach();
+ ));
+
+ # check that we indeed recover
+ ($output, $ret) = $psql->query(qq(SELECT count(*) FROM tbl_ok;));
+ is($ret, 0, "$io_method: recovers from failure to open ");
+
+
+ $psql->quit();
+}
+
+sub test_generic
+{
+ my $io_method = shift;
+ my $node = shift;
+
+ is($node->safe_psql('postgres', 'SHOW io_method'),
+ $io_method, "$io_method: io_method set correctly");
+
+ $node->safe_psql(
+ 'postgres', qq(
+CREATE EXTENSION test_aio;
+CREATE TABLE tbl_corr(data int not null) WITH (AUTOVACUUM_ENABLED = false);
+CREATE TABLE tbl_ok(data int not null) WITH (AUTOVACUUM_ENABLED = false);
+
+INSERT INTO tbl_corr SELECT generate_series(1, 10000);
+INSERT INTO tbl_ok SELECT generate_series(1, 10000);
+SELECT grow_rel('tbl_corr', 16);
+SELECT grow_rel('tbl_ok', 16);
+
+SELECT corrupt_rel_block('tbl_corr', 1);
+CHECKPOINT;
+));
+
+ test_handle($io_method, $node);
+ test_io_error($io_method, $node);
+ test_batch($io_method, $node);
+ test_startwait_io($io_method, $node);
+ test_complete_foreign($io_method, $node);
+
+ SKIP:
+ {
+ skip 'Injection points not supported by this build', 1
+ unless $ENV{enable_injection_points} eq 'yes';
+ test_inject($io_method, $node);
+ }
+}
diff --git a/src/test/modules/test_aio/test_aio--1.0.sql b/src/test/modules/test_aio/test_aio--1.0.sql
new file mode 100644
index 00000000000..3d4e9bb070f
--- /dev/null
+++ b/src/test/modules/test_aio/test_aio--1.0.sql
@@ -0,0 +1,97 @@
+/* src/test/modules/test_aio/test_aio--1.0.sql */
+
+-- complain if script is sourced in psql, rather than via CREATE EXTENSION
+\echo Use "CREATE EXTENSION test_aio" to load this file. \quit
+
+
+CREATE FUNCTION errno_from_string(sym text)
+RETURNS pg_catalog.int4 STRICT
+AS 'MODULE_PATHNAME' LANGUAGE C;
+
+
+CREATE FUNCTION grow_rel(rel regclass, nblocks int)
+RETURNS pg_catalog.void STRICT
+AS 'MODULE_PATHNAME' LANGUAGE C;
+
+
+CREATE FUNCTION corrupt_rel_block(rel regclass, blockno int)
+RETURNS pg_catalog.void STRICT
+AS 'MODULE_PATHNAME' LANGUAGE C;
+
+CREATE FUNCTION read_rel_block_ll(rel regclass, blockno int, wait_complete bool)
+RETURNS pg_catalog.void STRICT
+AS 'MODULE_PATHNAME' LANGUAGE C;
+
+CREATE FUNCTION invalidate_rel_block(rel regclass, blockno int)
+RETURNS pg_catalog.void STRICT
+AS 'MODULE_PATHNAME' LANGUAGE C;
+
+CREATE FUNCTION buffer_create_toy(rel regclass, blockno int4)
+RETURNS pg_catalog.int4 STRICT
+AS 'MODULE_PATHNAME' LANGUAGE C;
+
+CREATE FUNCTION buffer_call_start_io(buffer int, for_input bool, nowait bool)
+RETURNS pg_catalog.bool STRICT
+AS 'MODULE_PATHNAME' LANGUAGE C;
+
+CREATE FUNCTION buffer_call_terminate_io(buffer int, for_input bool, succeed bool, io_error bool, syncio bool)
+RETURNS pg_catalog.void STRICT
+AS 'MODULE_PATHNAME' LANGUAGE C;
+
+
+
+/*
+ * Handle related functions
+ */
+CREATE FUNCTION handle_get_and_error()
+RETURNS pg_catalog.void STRICT
+AS 'MODULE_PATHNAME' LANGUAGE C;
+
+CREATE FUNCTION handle_get_twice()
+RETURNS pg_catalog.void STRICT
+AS 'MODULE_PATHNAME' LANGUAGE C;
+
+CREATE FUNCTION handle_get()
+RETURNS pg_catalog.void STRICT
+AS 'MODULE_PATHNAME' LANGUAGE C;
+
+CREATE FUNCTION handle_get_release()
+RETURNS pg_catalog.void STRICT
+AS 'MODULE_PATHNAME' LANGUAGE C;
+
+CREATE FUNCTION handle_release_last()
+RETURNS pg_catalog.void STRICT
+AS 'MODULE_PATHNAME' LANGUAGE C;
+
+
+/*
+ * Batchmode related functions
+ */
+CREATE FUNCTION batch_start()
+RETURNS pg_catalog.void STRICT
+AS 'MODULE_PATHNAME' LANGUAGE C;
+
+CREATE FUNCTION batch_end()
+RETURNS pg_catalog.void STRICT
+AS 'MODULE_PATHNAME' LANGUAGE C;
+
+
+
+/*
+ * Injection point related functions
+ */
+CREATE FUNCTION inj_io_short_read_attach(result int)
+RETURNS pg_catalog.void STRICT
+AS 'MODULE_PATHNAME' LANGUAGE C;
+
+CREATE FUNCTION inj_io_short_read_detach()
+RETURNS pg_catalog.void STRICT
+AS 'MODULE_PATHNAME' LANGUAGE C;
+
+CREATE FUNCTION inj_io_reopen_attach()
+RETURNS pg_catalog.void STRICT
+AS 'MODULE_PATHNAME' LANGUAGE C;
+
+CREATE FUNCTION inj_io_reopen_detach()
+RETURNS pg_catalog.void STRICT
+AS 'MODULE_PATHNAME' LANGUAGE C;
diff --git a/src/test/modules/test_aio/test_aio.c b/src/test/modules/test_aio/test_aio.c
new file mode 100644
index 00000000000..81b4d732206
--- /dev/null
+++ b/src/test/modules/test_aio/test_aio.c
@@ -0,0 +1,616 @@
+/*-------------------------------------------------------------------------
+ *
+ * delay_execution.c
+ * Test module to allow delay between parsing and execution of a query.
+ *
+ * The delay is implemented by taking and immediately releasing a specified
+ * advisory lock. If another process has previously taken that lock, the
+ * current process will be blocked until the lock is released; otherwise,
+ * there's no effect. This allows an isolationtester script to reliably
+ * test behaviors where some specified action happens in another backend
+ * between parsing and execution of any desired query.
+ *
+ * Copyright (c) 2020-2025, PostgreSQL Global Development Group
+ *
+ * IDENTIFICATION
+ * src/test/modules/delay_execution/delay_execution.c
+ *
+ *-------------------------------------------------------------------------
+ */
+
+#include "postgres.h"
+
+#include "access/relation.h"
+#include "fmgr.h"
+#include "storage/aio.h"
+#include "storage/aio_internal.h"
+#include "storage/buf_internals.h"
+#include "storage/bufmgr.h"
+#include "storage/ipc.h"
+#include "storage/lwlock.h"
+#include "utils/builtins.h"
+#include "utils/injection_point.h"
+#include "utils/rel.h"
+
+
+PG_MODULE_MAGIC;
+
+
+typedef struct InjIoErrorState
+{
+ bool enabled_short_read;
+ bool enabled_reopen;
+
+ bool short_read_result_set;
+ int short_read_result;
+} InjIoErrorState;
+
+static InjIoErrorState * inj_io_error_state;
+
+/* Shared memory init callbacks */
+static shmem_request_hook_type prev_shmem_request_hook = NULL;
+static shmem_startup_hook_type prev_shmem_startup_hook = NULL;
+
+
+static PgAioHandle *last_handle;
+
+
+
+static void
+test_aio_shmem_request(void)
+{
+ if (prev_shmem_request_hook)
+ prev_shmem_request_hook();
+
+ RequestAddinShmemSpace(sizeof(InjIoErrorState));
+}
+
+static void
+test_aio_shmem_startup(void)
+{
+ bool found;
+
+ if (prev_shmem_startup_hook)
+ prev_shmem_startup_hook();
+
+ /* Create or attach to the shared memory state */
+ LWLockAcquire(AddinShmemInitLock, LW_EXCLUSIVE);
+
+ inj_io_error_state = ShmemInitStruct("injection_points",
+ sizeof(InjIoErrorState),
+ &found);
+
+ if (!found)
+ {
+ /*
+ * First time through, so initialize. This is shared with the dynamic
+ * initialization using a DSM.
+ */
+ inj_io_error_state->enabled_short_read = false;
+ inj_io_error_state->enabled_reopen = false;
+
+#ifdef USE_INJECTION_POINTS
+ InjectionPointAttach("AIO_PROCESS_COMPLETION_BEFORE_SHARED",
+ "test_aio",
+ "inj_io_short_read",
+ NULL,
+ 0);
+ InjectionPointLoad("AIO_PROCESS_COMPLETION_BEFORE_SHARED");
+
+ InjectionPointAttach("AIO_WORKER_AFTER_REOPEN",
+ "test_aio",
+ "inj_io_reopen",
+ NULL,
+ 0);
+ InjectionPointLoad("AIO_WORKER_AFTER_REOPEN");
+
+#endif
+ }
+ else
+ {
+#ifdef USE_INJECTION_POINTS
+ InjectionPointLoad("AIO_PROCESS_COMPLETION_BEFORE_SHARED");
+ InjectionPointLoad("AIO_WORKER_AFTER_REOPEN");
+ elog(LOG, "injection point loaded");
+#endif
+ }
+
+ LWLockRelease(AddinShmemInitLock);
+}
+
+void
+_PG_init(void)
+{
+ if (!process_shared_preload_libraries_in_progress)
+ return;
+
+ /* Shared memory initialization */
+ prev_shmem_request_hook = shmem_request_hook;
+ shmem_request_hook = test_aio_shmem_request;
+ prev_shmem_startup_hook = shmem_startup_hook;
+ shmem_startup_hook = test_aio_shmem_startup;
+}
+
+
+PG_FUNCTION_INFO_V1(errno_from_string);
+Datum
+errno_from_string(PG_FUNCTION_ARGS)
+{
+ const char *sym = text_to_cstring(PG_GETARG_TEXT_PP(0));
+
+ if (strcmp(sym, "EIO") == 0)
+ PG_RETURN_INT32(EIO);
+ else if (strcmp(sym, "EAGAIN") == 0)
+ PG_RETURN_INT32(EAGAIN);
+ else if (strcmp(sym, "EINTR") == 0)
+ PG_RETURN_INT32(EINTR);
+ else if (strcmp(sym, "ENOSPC") == 0)
+ PG_RETURN_INT32(ENOSPC);
+ else if (strcmp(sym, "EROFS") == 0)
+ PG_RETURN_INT32(EROFS);
+
+ ereport(ERROR,
+ errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ errmsg_internal("%s is not a supported errno value", sym));
+ PG_RETURN_INT32(0);
+}
+
+
+PG_FUNCTION_INFO_V1(grow_rel);
+Datum
+grow_rel(PG_FUNCTION_ARGS)
+{
+ Oid relid = PG_GETARG_OID(0);
+ uint32 nblocks = PG_GETARG_UINT32(1);
+ Relation rel;
+#define MAX_BUFFERS_TO_EXTEND_BY 64
+ Buffer victim_buffers[MAX_BUFFERS_TO_EXTEND_BY];
+
+ rel = relation_open(relid, AccessExclusiveLock);
+
+ while (nblocks > 0)
+ {
+ uint32 extend_by_pages;
+
+ extend_by_pages = Min(nblocks, MAX_BUFFERS_TO_EXTEND_BY);
+
+ ExtendBufferedRelBy(BMR_REL(rel),
+ MAIN_FORKNUM,
+ NULL,
+ 0,
+ extend_by_pages,
+ victim_buffers,
+ &extend_by_pages);
+
+ nblocks -= extend_by_pages;
+
+ for (uint32 i = 0; i < extend_by_pages; i++)
+ {
+ ReleaseBuffer(victim_buffers[i]);
+ }
+ }
+
+ relation_close(rel, NoLock);
+
+ PG_RETURN_VOID();
+}
+
+PG_FUNCTION_INFO_V1(corrupt_rel_block);
+Datum
+corrupt_rel_block(PG_FUNCTION_ARGS)
+{
+ Oid relid = PG_GETARG_OID(0);
+ BlockNumber blkno = PG_GETARG_UINT32(1);
+ Relation rel;
+ Buffer buf;
+ Page page;
+ PageHeader ph;
+
+ rel = relation_open(relid, AccessExclusiveLock);
+
+ buf = ReadBuffer(rel, blkno);
+ page = BufferGetPage(buf);
+
+ LockBuffer(buf, BUFFER_LOCK_EXCLUSIVE);
+
+ MarkBufferDirty(buf);
+
+ PageInit(page, BufferGetPageSize(buf), 0);
+
+ ph = (PageHeader) page;
+ ph->pd_special = BLCKSZ + 1;
+
+ FlushOneBuffer(buf);
+
+ LockBuffer(buf, BUFFER_LOCK_UNLOCK);
+
+ ReleaseBuffer(buf);
+
+ EvictUnpinnedBuffer(buf);
+
+ relation_close(rel, NoLock);
+
+ PG_RETURN_VOID();
+}
+
+/*
+ * Ensures a buffer for rel & blkno is in shared buffers, without actually
+ * caring about the buffer contents. Used to set up test scenarios.
+ */
+static Buffer
+create_toy_buffer(Relation rel, BlockNumber blkno)
+{
+ Buffer buf;
+ BufferDesc *buf_hdr;
+ uint32 buf_state;
+ bool was_pinned = false;
+
+ /* read buffer without erroring out */
+ buf = ReadBufferExtended(rel, MAIN_FORKNUM, blkno, RBM_ZERO_ON_ERROR, NULL);
+
+ buf_hdr = GetBufferDescriptor(buf - 1);
+
+ buf_state = LockBufHdr(buf_hdr);
+
+ /*
+ * We should be the only backend accessing this buffer. This is just a
+ * small bit of belt-and-suspenders defense, none of this code should ever
+ * run in a cluster with real data.
+ */
+ if (BUF_STATE_GET_REFCOUNT(buf_state) > 1)
+ was_pinned = true;
+ else
+ buf_state &= ~(BM_VALID | BM_DIRTY);
+
+ UnlockBufHdr(buf_hdr, buf_state);
+
+ if (was_pinned)
+ elog(ERROR, "toy buffer %d was already pinned",
+ buf);
+
+ return buf;
+}
+
+PG_FUNCTION_INFO_V1(read_rel_block_ll);
+Datum
+read_rel_block_ll(PG_FUNCTION_ARGS)
+{
+ Oid relid = PG_GETARG_OID(0);
+ BlockNumber blkno = PG_GETARG_UINT32(1);
+ bool wait_complete = PG_GETARG_BOOL(2);
+ Relation rel;
+ Buffer buf;
+ Page pages[1];
+ PgAioReturn ior;
+ PgAioHandle *ioh;
+ PgAioWaitRef iow;
+ SMgrRelation smgr;
+
+ rel = relation_open(relid, AccessExclusiveLock);
+
+ buf = create_toy_buffer(rel, blkno);
+
+ pages[0] = BufferGetBlock(buf);
+
+ ioh = pgaio_io_acquire(CurrentResourceOwner, &ior);
+ pgaio_io_get_wref(ioh, &iow);
+
+ smgr = RelationGetSmgr(rel);
+
+ StartBufferIO(GetBufferDescriptor(buf - 1), true, false);
+
+ pgaio_io_set_handle_data_32(ioh, (uint32 *) &buf, 1);
+ pgaio_io_register_callbacks(ioh, PGAIO_HCB_SHARED_BUFFER_READV, 0);
+
+ elog(LOG, "about to smgrstartreadv");
+ smgrstartreadv(ioh, smgr, MAIN_FORKNUM, blkno,
+ (void *) pages, 1);
+
+ ReleaseBuffer(buf);
+
+ if (wait_complete)
+ {
+ pgaio_wref_wait(&iow);
+
+ if (ior.result.status != ARS_OK)
+ pgaio_result_report(ior.result, &ior.target_data,
+ ior.result.status == ARS_PARTIAL ? WARNING : ERROR);
+ }
+
+ relation_close(rel, NoLock);
+
+ PG_RETURN_VOID();
+}
+
+PG_FUNCTION_INFO_V1(invalidate_rel_block);
+Datum
+invalidate_rel_block(PG_FUNCTION_ARGS)
+{
+ Oid relid = PG_GETARG_OID(0);
+ BlockNumber blkno = PG_GETARG_UINT32(1);
+ Relation rel;
+ PrefetchBufferResult pr;
+ Buffer buf;
+
+ rel = relation_open(relid, AccessExclusiveLock);
+
+ /*
+ * This is a gross hack, but there's no other API exposed that allows to
+ * get a buffer ID without actually reading the block in.
+ */
+ pr = PrefetchBuffer(rel, MAIN_FORKNUM, blkno);
+ buf = pr.recent_buffer;
+
+ if (BufferIsValid(buf))
+ {
+ /* if the buffer contents aren't valid, this'll return false */
+ if (ReadRecentBuffer(rel->rd_locator, MAIN_FORKNUM, blkno, buf))
+ {
+ LockBuffer(buf, BUFFER_LOCK_EXCLUSIVE);
+ FlushOneBuffer(buf);
+ LockBuffer(buf, BUFFER_LOCK_UNLOCK);
+ ReleaseBuffer(buf);
+
+ if (!EvictUnpinnedBuffer(buf))
+ elog(ERROR, "couldn't evict");
+ }
+ }
+
+ relation_close(rel, AccessExclusiveLock);
+
+ PG_RETURN_VOID();
+}
+
+PG_FUNCTION_INFO_V1(buffer_create_toy);
+Datum
+buffer_create_toy(PG_FUNCTION_ARGS)
+{
+ Oid relid = PG_GETARG_OID(0);
+ BlockNumber blkno = PG_GETARG_UINT32(1);
+ Relation rel;
+ Buffer buf;
+
+ rel = relation_open(relid, AccessExclusiveLock);
+
+ buf = create_toy_buffer(rel, blkno);
+ ReleaseBuffer(buf);
+
+ relation_close(rel, NoLock);
+
+ PG_RETURN_INT32(buf);
+}
+
+PG_FUNCTION_INFO_V1(buffer_call_start_io);
+Datum
+buffer_call_start_io(PG_FUNCTION_ARGS)
+{
+ Buffer buf = PG_GETARG_INT32(0);
+ bool for_input = PG_GETARG_BOOL(1);
+ bool nowait = PG_GETARG_BOOL(2);
+ bool can_start;
+
+ can_start = StartBufferIO(GetBufferDescriptor(buf - 1), for_input, nowait);
+
+ /*
+ * Tor tests we don't want the resowner release preventing us from
+ * orchestrating odd scenarios.
+ */
+ if (can_start)
+ ResourceOwnerForgetBufferIO(CurrentResourceOwner,
+ buf);
+
+ ereport(LOG,
+ errmsg("buffer %d after StartBufferIO: %s",
+ buf, DebugPrintBufferRefcount(buf)),
+ errhidestmt(true), errhidecontext(true));
+
+ PG_RETURN_BOOL(can_start);
+}
+
+/*
+CREATE FUNCTION buffer_call_terminate_io(buffer int, for_input bool, failed bool, syncio bool)
+RETURNS pg_catalog.void STRICT
+AS 'MODULE_PATHNAME' LANGUAGE C;
+*/
+PG_FUNCTION_INFO_V1(buffer_call_terminate_io);
+Datum
+buffer_call_terminate_io(PG_FUNCTION_ARGS)
+{
+ Buffer buf = PG_GETARG_INT32(0);
+ bool for_input = PG_GETARG_BOOL(1);
+ bool succeed = PG_GETARG_BOOL(2);
+ bool io_error = PG_GETARG_BOOL(3);
+ bool syncio = PG_GETARG_BOOL(4);
+ bool clear_dirty = false;
+ uint32 set_flag_bits = 0;
+
+ if (io_error)
+ set_flag_bits |= BM_IO_ERROR;
+
+ if (for_input)
+ {
+ clear_dirty = false;
+
+ if (succeed)
+ set_flag_bits |= BM_VALID;
+ }
+ else
+ {
+ if (succeed)
+ clear_dirty = true;
+ }
+
+ ereport(LOG,
+ errmsg("buffer %d before TerminateBufferIO: %s",
+ buf, DebugPrintBufferRefcount(buf)),
+ errhidestmt(true), errhidecontext(true));
+
+ TerminateBufferIO(GetBufferDescriptor(buf - 1), clear_dirty, set_flag_bits,
+ false, syncio);
+
+ ereport(LOG,
+ errmsg("buffer %d after TerminateBufferIO: %s",
+ buf, DebugPrintBufferRefcount(buf)),
+ errhidestmt(true), errhidecontext(true));
+
+ PG_RETURN_VOID();
+}
+
+PG_FUNCTION_INFO_V1(handle_get);
+Datum
+handle_get(PG_FUNCTION_ARGS)
+{
+ last_handle = pgaio_io_acquire(CurrentResourceOwner, NULL);
+
+ PG_RETURN_VOID();
+}
+
+PG_FUNCTION_INFO_V1(handle_release_last);
+Datum
+handle_release_last(PG_FUNCTION_ARGS)
+{
+ if (!last_handle)
+ elog(ERROR, "no handle");
+
+ pgaio_io_release(last_handle);
+
+ PG_RETURN_VOID();
+}
+
+PG_FUNCTION_INFO_V1(handle_get_and_error);
+Datum
+handle_get_and_error(PG_FUNCTION_ARGS)
+{
+ pgaio_io_acquire(CurrentResourceOwner, NULL);
+
+ elog(ERROR, "as you command");
+ PG_RETURN_VOID();
+}
+
+PG_FUNCTION_INFO_V1(handle_get_twice);
+Datum
+handle_get_twice(PG_FUNCTION_ARGS)
+{
+ pgaio_io_acquire(CurrentResourceOwner, NULL);
+ pgaio_io_acquire(CurrentResourceOwner, NULL);
+
+ PG_RETURN_VOID();
+}
+
+PG_FUNCTION_INFO_V1(handle_get_release);
+Datum
+handle_get_release(PG_FUNCTION_ARGS)
+{
+ PgAioHandle *handle;
+
+ handle = pgaio_io_acquire(CurrentResourceOwner, NULL);
+ pgaio_io_release(handle);
+
+ PG_RETURN_VOID();
+}
+
+PG_FUNCTION_INFO_V1(batch_start);
+Datum
+batch_start(PG_FUNCTION_ARGS)
+{
+ pgaio_enter_batchmode();
+ PG_RETURN_VOID();
+}
+
+PG_FUNCTION_INFO_V1(batch_end);
+Datum
+batch_end(PG_FUNCTION_ARGS)
+{
+ pgaio_exit_batchmode();
+ PG_RETURN_VOID();
+}
+
+#ifdef USE_INJECTION_POINTS
+extern PGDLLEXPORT void inj_io_short_read(const char *name, const void *private_data);
+extern PGDLLEXPORT void inj_io_reopen(const char *name, const void *private_data);
+
+void
+inj_io_short_read(const char *name, const void *private_data)
+{
+ PgAioHandle *ioh;
+
+ elog(LOG, "short read called: %d", inj_io_error_state->enabled_short_read);
+
+ if (inj_io_error_state->enabled_short_read)
+ {
+ ioh = pgaio_inj_io_get();
+
+ if (inj_io_error_state->short_read_result_set)
+ {
+ elog(LOG, "short read, changing result from %d to %d",
+ ioh->result, inj_io_error_state->short_read_result);
+
+ ioh->result = inj_io_error_state->short_read_result;
+ }
+ }
+}
+
+void
+inj_io_reopen(const char *name, const void *private_data)
+{
+ elog(LOG, "reopen called: %d", inj_io_error_state->enabled_reopen);
+
+ if (inj_io_error_state->enabled_reopen)
+ {
+ elog(ERROR, "injection point triggering failure to reopen ");
+ }
+}
+#endif
+
+PG_FUNCTION_INFO_V1(inj_io_short_read_attach);
+Datum
+inj_io_short_read_attach(PG_FUNCTION_ARGS)
+{
+#ifdef USE_INJECTION_POINTS
+ inj_io_error_state->enabled_short_read = true;
+ inj_io_error_state->short_read_result_set = !PG_ARGISNULL(0);
+ if (inj_io_error_state->short_read_result_set)
+ inj_io_error_state->short_read_result = PG_GETARG_INT32(0);
+#else
+ elog(ERROR, "injection points not supported");
+#endif
+
+ PG_RETURN_VOID();
+}
+
+PG_FUNCTION_INFO_V1(inj_io_short_read_detach);
+Datum
+inj_io_short_read_detach(PG_FUNCTION_ARGS)
+{
+#ifdef USE_INJECTION_POINTS
+ inj_io_error_state->enabled_short_read = false;
+#else
+ elog(ERROR, "injection points not supported");
+#endif
+ PG_RETURN_VOID();
+}
+
+PG_FUNCTION_INFO_V1(inj_io_reopen_attach);
+Datum
+inj_io_reopen_attach(PG_FUNCTION_ARGS)
+{
+#ifdef USE_INJECTION_POINTS
+ inj_io_error_state->enabled_reopen = true;
+#else
+ elog(ERROR, "injection points not supported");
+#endif
+
+ PG_RETURN_VOID();
+}
+
+PG_FUNCTION_INFO_V1(inj_io_reopen_detach);
+Datum
+inj_io_reopen_detach(PG_FUNCTION_ARGS)
+{
+#ifdef USE_INJECTION_POINTS
+ inj_io_error_state->enabled_reopen = false;
+#else
+ elog(ERROR, "injection points not supported");
+#endif
+ PG_RETURN_VOID();
+}
diff --git a/src/test/modules/test_aio/test_aio.control b/src/test/modules/test_aio/test_aio.control
new file mode 100644
index 00000000000..cd91c3ed16b
--- /dev/null
+++ b/src/test/modules/test_aio/test_aio.control
@@ -0,0 +1,3 @@
+comment = 'Test code for AIO'
+default_version = '1.0'
+module_pathname = '$libdir/test_aio'
--
2.48.1.76.g4e746b1a31.dirty
[text/x-diff] v2.8-0029-aio-Implement-smgr-md-fd-write-support.patch (12.7K, ../../3yxd5r23zly5bytvgyktbxtxq2r3gbpi7xd4dugevh3h4w4q6c@lu6oatjjpltz/30-v2.8-0029-aio-Implement-smgr-md-fd-write-support.patch)
download | inline diff:
From 3df9228c566d98f8a0a60c9d49a16054334674e2 Mon Sep 17 00:00:00 2001
From: Andres Freund <andres@anarazel.de>
Date: Sun, 9 Mar 2025 19:35:04 -0400
Subject: [PATCH v2.8 29/38] aio: Implement smgr/md/fd write support
TODO:
- register_dirty_segment_aio() can error out in edge cases
---
src/include/storage/fd.h | 1 +
src/include/storage/md.h | 5 +
src/include/storage/smgr.h | 5 +
src/backend/storage/aio/aio_callback.c | 1 +
src/backend/storage/file/fd.c | 28 ++++
src/backend/storage/smgr/md.c | 189 +++++++++++++++++++++++++
src/backend/storage/smgr/smgr.c | 22 +++
7 files changed, 251 insertions(+)
diff --git a/src/include/storage/fd.h b/src/include/storage/fd.h
index b77d8e5e30e..2cc7c5a4761 100644
--- a/src/include/storage/fd.h
+++ b/src/include/storage/fd.h
@@ -112,6 +112,7 @@ extern int FilePrefetch(File file, off_t offset, off_t amount, uint32 wait_event
extern ssize_t FileReadV(File file, const struct iovec *iov, int iovcnt, off_t offset, uint32 wait_event_info);
extern ssize_t FileWriteV(File file, const struct iovec *iov, int iovcnt, off_t offset, uint32 wait_event_info);
extern int FileStartReadV(struct PgAioHandle *ioh, File file, int iovcnt, off_t offset, uint32 wait_event_info);
+extern int FileStartWriteV(struct PgAioHandle *ioh, File file, int iovcnt, off_t offset, uint32 wait_event_info);
extern int FileSync(File file, uint32 wait_event_info);
extern int FileZero(File file, off_t offset, off_t amount, uint32 wait_event_info);
extern int FileFallocate(File file, off_t offset, off_t amount, uint32 wait_event_info);
diff --git a/src/include/storage/md.h b/src/include/storage/md.h
index 9d7131eff43..47ae6c36c94 100644
--- a/src/include/storage/md.h
+++ b/src/include/storage/md.h
@@ -21,6 +21,7 @@
#include "storage/sync.h"
extern const PgAioHandleCallbacks aio_md_readv_cb;
+extern const PgAioHandleCallbacks aio_md_writev_cb;
/* md storage manager functionality */
extern void mdinit(void);
@@ -45,6 +46,10 @@ extern void mdstartreadv(PgAioHandle *ioh,
extern void mdwritev(SMgrRelation reln, ForkNumber forknum,
BlockNumber blocknum,
const void **buffers, BlockNumber nblocks, bool skipFsync);
+extern void mdstartwritev(PgAioHandle *ioh,
+ SMgrRelation reln, ForkNumber forknum,
+ BlockNumber blocknum,
+ const void **buffers, BlockNumber nblocks, bool skipFsync);
extern void mdwriteback(SMgrRelation reln, ForkNumber forknum,
BlockNumber blocknum, BlockNumber nblocks);
extern BlockNumber mdnblocks(SMgrRelation reln, ForkNumber forknum);
diff --git a/src/include/storage/smgr.h b/src/include/storage/smgr.h
index 856ebcda350..f00b3763ac9 100644
--- a/src/include/storage/smgr.h
+++ b/src/include/storage/smgr.h
@@ -108,6 +108,11 @@ extern void smgrwritev(SMgrRelation reln, ForkNumber forknum,
BlockNumber blocknum,
const void **buffers, BlockNumber nblocks,
bool skipFsync);
+extern void smgrstartwritev(PgAioHandle *ioh,
+ SMgrRelation reln, ForkNumber forknum,
+ BlockNumber blocknum,
+ const void **buffers, BlockNumber nblocks,
+ bool skipFsync);
extern void smgrwriteback(SMgrRelation reln, ForkNumber forknum,
BlockNumber blocknum, BlockNumber nblocks);
extern BlockNumber smgrnblocks(SMgrRelation reln, ForkNumber forknum);
diff --git a/src/backend/storage/aio/aio_callback.c b/src/backend/storage/aio/aio_callback.c
index f76f74ba166..fb6ac058a09 100644
--- a/src/backend/storage/aio/aio_callback.c
+++ b/src/backend/storage/aio/aio_callback.c
@@ -41,6 +41,7 @@ static const PgAioHandleCallbacksEntry aio_handle_cbs[] = {
CALLBACK_ENTRY(PGAIO_HCB_INVALID, aio_invalid_cb),
CALLBACK_ENTRY(PGAIO_HCB_MD_READV, aio_md_readv_cb),
+ CALLBACK_ENTRY(PGAIO_HCB_MD_WRITEV, aio_md_writev_cb),
CALLBACK_ENTRY(PGAIO_HCB_SHARED_BUFFER_READV, aio_shared_buffer_readv_cb),
diff --git a/src/backend/storage/file/fd.c b/src/backend/storage/file/fd.c
index 34588ed5167..46cbcb2b688 100644
--- a/src/backend/storage/file/fd.c
+++ b/src/backend/storage/file/fd.c
@@ -2346,6 +2346,34 @@ retry:
return returnCode;
}
+int
+FileStartWriteV(PgAioHandle *ioh, File file,
+ int iovcnt, off_t offset,
+ uint32 wait_event_info)
+{
+ int returnCode;
+ Vfd *vfdP;
+
+ Assert(FileIsValid(file));
+
+ DO_DB(elog(LOG, "FileStartWriteV: %d (%s) " INT64_FORMAT " %d",
+ file, VfdCache[file].fileName,
+ (int64) offset,
+ iovcnt));
+
+ returnCode = FileAccess(file);
+ if (returnCode < 0)
+ return returnCode;
+
+ vfdP = &VfdCache[file];
+
+ /* FIXME: think about / reimplement temp_file_limit */
+
+ pgaio_io_prep_writev(ioh, vfdP->fd, iovcnt, offset);
+
+ return 0;
+}
+
int
FileSync(File file, uint32 wait_event_info)
{
diff --git a/src/backend/storage/smgr/md.c b/src/backend/storage/smgr/md.c
index 11d4d5a7aea..a61fc14805e 100644
--- a/src/backend/storage/smgr/md.c
+++ b/src/backend/storage/smgr/md.c
@@ -155,12 +155,19 @@ static BlockNumber _mdnblocks(SMgrRelation reln, ForkNumber forknum,
static PgAioResult md_readv_complete(PgAioHandle *ioh, PgAioResult prior_result, uint8 cb_data);
static void md_readv_report(PgAioResult result, const PgAioTargetData *target_data, int elevel);
+static PgAioResult md_writev_complete(PgAioHandle *ioh, PgAioResult prior_result, uint8 cb_data);
+static void md_writev_report(PgAioResult result, const PgAioTargetData *target_data, int elevel);
const PgAioHandleCallbacks aio_md_readv_cb = {
.complete_shared = md_readv_complete,
.report = md_readv_report,
};
+const PgAioHandleCallbacks aio_md_writev_cb = {
+ .complete_shared = md_writev_complete,
+ .report = md_writev_report,
+};
+
static inline int
_mdfd_open_flags(void)
@@ -1107,6 +1114,56 @@ mdwritev(SMgrRelation reln, ForkNumber forknum, BlockNumber blocknum,
}
}
+/*
+ * mdstartwritev() -- Asynchronous version of mdrwritev().
+ */
+void
+mdstartwritev(PgAioHandle *ioh,
+ SMgrRelation reln, ForkNumber forknum, BlockNumber blocknum,
+ const void **buffers, BlockNumber nblocks, bool skipFsync)
+{
+ off_t seekpos;
+ MdfdVec *v;
+ BlockNumber nblocks_this_segment;
+ struct iovec *iov;
+ int iovcnt;
+
+ v = _mdfd_getseg(reln, forknum, blocknum, false,
+ EXTENSION_FAIL | EXTENSION_CREATE_RECOVERY);
+
+ seekpos = (off_t) BLCKSZ * (blocknum % ((BlockNumber) RELSEG_SIZE));
+
+ Assert(seekpos < (off_t) BLCKSZ * RELSEG_SIZE);
+
+ nblocks_this_segment =
+ Min(nblocks,
+ RELSEG_SIZE - (blocknum % ((BlockNumber) RELSEG_SIZE)));
+
+ if (nblocks_this_segment != nblocks)
+ elog(ERROR, "write crossing segment boundary");
+
+ iovcnt = pgaio_io_get_iovec(ioh, &iov);
+
+ Assert(nblocks <= iovcnt);
+
+ iovcnt = buffers_to_iovec(iov, unconstify(void **, buffers), nblocks_this_segment);
+
+ Assert(iovcnt <= nblocks_this_segment);
+
+ if (!(io_direct_flags & IO_DIRECT_DATA))
+ pgaio_io_set_flag(ioh, PGAIO_HF_BUFFERED);
+
+ pgaio_io_set_target_smgr(ioh,
+ reln,
+ forknum,
+ blocknum,
+ nblocks,
+ skipFsync);
+ pgaio_io_register_callbacks(ioh, PGAIO_HCB_MD_WRITEV, 0);
+
+ FileStartWriteV(ioh, v->mdfd_vfd, iovcnt, seekpos, WAIT_EVENT_DATA_FILE_WRITE);
+}
+
/*
* mdwriteback() -- Tell the kernel to write pages back to storage.
@@ -1495,6 +1552,40 @@ register_dirty_segment(SMgrRelation reln, ForkNumber forknum, MdfdVec *seg)
}
}
+/*
+ * Like register_dirty_segment(), except for use by AIO. In the completion
+ * callback we don't have access to the MdfdVec (the completion callback might
+ * be executed in a different backend than the issuing backend), therefore we
+ * have to implement this slightly differently.
+ */
+static void
+register_dirty_segment_aio(RelFileLocator locator, ForkNumber forknum, uint64 segno)
+{
+ FileTag tag;
+
+ INIT_MD_FILETAG(tag, locator, forknum, segno);
+
+ /*
+ * Can't block here waiting for checkpointer to accept our sync request,
+ * as checkpointer might be waiting for this AIO to finish if offloaded to
+ * a worker.
+ */
+ if (!RegisterSyncRequest(&tag, SYNC_REQUEST, false /* retryOnError */ ))
+ {
+ char path[MAXPGPATH];
+
+ ereport(DEBUG1,
+ (errmsg_internal("could not forward fsync request because request queue is full")));
+
+ /* reuse mdsyncfiletag() to avoid duplicating code */
+ if (mdsyncfiletag(&tag, path))
+ ereport(data_sync_elevel(ERROR),
+ (errcode_for_file_access(),
+ errmsg("could not fsync file \"%s\": %m",
+ path)));
+ }
+}
+
/*
* register_unlink_segment() -- Schedule a file to be deleted after next checkpoint
*/
@@ -2017,3 +2108,101 @@ md_readv_report(PgAioResult result, const PgAioTargetData *td, int elevel)
td->smgr.nblocks * (size_t) BLCKSZ));
}
}
+
+/*
+ * AIO completion callback for mdstartwritev().
+ */
+static PgAioResult
+md_writev_complete(PgAioHandle *ioh, PgAioResult prior_result, uint8 cb_data)
+{
+ PgAioTargetData *td = pgaio_io_get_target_data(ioh);
+ PgAioResult result = prior_result;
+
+ if (prior_result.result < 0)
+ {
+ result.status = ARS_ERROR;
+ result.id = PGAIO_HCB_MD_WRITEV;
+ /* For "hard" errors, track the error number in error_data */
+ result.error_data = -prior_result.result;
+ result.result = 0;
+
+ pgaio_result_report(result, td, LOG);
+
+ return result;
+ }
+
+ /*
+ * The smgr API operates in blocks, therefore convert the result from
+ * bytes to blocks.
+ */
+ result.result /= BLCKSZ;
+
+ if (result.result == 0)
+ {
+ /* consider 0 blocks written a failure */
+ result.status = ARS_ERROR;
+ result.id = PGAIO_HCB_MD_WRITEV;
+ result.error_data = 0;
+
+ pgaio_result_report(result, td, LOG);
+
+ return result;
+ }
+
+ if (result.status != ARS_ERROR &&
+ result.result < td->smgr.nblocks)
+ {
+ /* partial writes should be retried at upper level */
+ result.status = ARS_PARTIAL;
+ result.id = PGAIO_HCB_MD_WRITEV;
+ }
+
+ if (!td->smgr.skip_fsync)
+ register_dirty_segment_aio(td->smgr.rlocator, td->smgr.forkNum,
+ td->smgr.blockNum / ((BlockNumber) RELSEG_SIZE));
+
+ return result;
+}
+
+/*
+ * AIO error reporting callback for mdstartwritev().
+ */
+static void
+md_writev_report(PgAioResult result, const PgAioTargetData *td, int elevel)
+{
+ RelPathStr path;
+
+ path = relpathbackend(td->smgr.rlocator,
+ td->smgr.is_temp ? MyProcNumber : INVALID_PROC_NUMBER,
+ td->smgr.forkNum);
+
+ if (result.error_data != 0)
+ {
+ errno = result.error_data; /* for errcode_for_file_access() */
+
+ ereport(elevel,
+ errcode_for_file_access(),
+ errmsg("could not write blocks %u..%u in file \"%s\": %m",
+ td->smgr.blockNum,
+ td->smgr.blockNum + td->smgr.nblocks,
+ path.str)
+ );
+ }
+ else
+ {
+ /*
+ * NB: This will typically only be output in debug messages, while
+ * retrying a partial IO.
+ */
+ ereport(elevel,
+ errcode(ERRCODE_DATA_CORRUPTED),
+ errmsg("could not write blocks %u..%u in file \"%s\": wrote only %zu of %zu bytes",
+ td->smgr.blockNum,
+ td->smgr.blockNum + td->smgr.nblocks - 1,
+ path.str,
+ result.result * (size_t) BLCKSZ,
+ td->smgr.nblocks * (size_t) BLCKSZ
+ )
+ );
+ }
+}
diff --git a/src/backend/storage/smgr/smgr.c b/src/backend/storage/smgr/smgr.c
index 53fd6af4799..cd515954e74 100644
--- a/src/backend/storage/smgr/smgr.c
+++ b/src/backend/storage/smgr/smgr.c
@@ -112,6 +112,11 @@ typedef struct f_smgr
BlockNumber blocknum,
const void **buffers, BlockNumber nblocks,
bool skipFsync);
+ void (*smgr_startwritev) (PgAioHandle *ioh,
+ SMgrRelation reln, ForkNumber forknum,
+ BlockNumber blocknum,
+ const void **buffers, BlockNumber nblocks,
+ bool skipFsync);
void (*smgr_writeback) (SMgrRelation reln, ForkNumber forknum,
BlockNumber blocknum, BlockNumber nblocks);
BlockNumber (*smgr_nblocks) (SMgrRelation reln, ForkNumber forknum);
@@ -139,6 +144,7 @@ static const f_smgr smgrsw[] = {
.smgr_readv = mdreadv,
.smgr_startreadv = mdstartreadv,
.smgr_writev = mdwritev,
+ .smgr_startwritev = mdstartwritev,
.smgr_writeback = mdwriteback,
.smgr_nblocks = mdnblocks,
.smgr_truncate = mdtruncate,
@@ -769,6 +775,22 @@ smgrwritev(SMgrRelation reln, ForkNumber forknum, BlockNumber blocknum,
RESUME_INTERRUPTS();
}
+/*
+ * smgrstartwritev() -- asynchronous version of smgrwritev()
+ *
+ * This starts an asynchronous writev IO with the IO handle `ioh`. Other than
+ * `ioh` all parameters are the same as smgrwritev().
+ */
+void
+smgrstartwritev(PgAioHandle *ioh,
+ SMgrRelation reln, ForkNumber forknum, BlockNumber blocknum,
+ const void **buffers, BlockNumber nblocks, bool skipFsync)
+{
+ smgrsw[reln->smgr_which].smgr_startwritev(ioh,
+ reln, forknum, blocknum, buffers,
+ nblocks, skipFsync);
+}
+
/*
* smgrwriteback() -- Trigger kernel writeback for the supplied range of
* blocks.
--
2.48.1.76.g4e746b1a31.dirty
[text/x-diff] v2.8-0030-aio-Add-bounce-buffers.patch (23.1K, ../../3yxd5r23zly5bytvgyktbxtxq2r3gbpi7xd4dugevh3h4w4q6c@lu6oatjjpltz/31-v2.8-0030-aio-Add-bounce-buffers.patch)
download | inline diff:
From 408511c945fb6231227a066f1c14d40ecbd58070 Mon Sep 17 00:00:00 2001
From: Andres Freund <andres@anarazel.de>
Date: Fri, 7 Mar 2025 15:47:38 -0500
Subject: [PATCH v2.8 30/38] aio: Add bounce buffers
---
src/include/storage/aio.h | 15 ++
src/include/storage/aio_internal.h | 34 ++++
src/include/storage/aio_types.h | 2 +
src/include/utils/resowner.h | 2 +
src/backend/storage/aio/README.md | 27 +++
src/backend/storage/aio/aio.c | 178 ++++++++++++++++++
src/backend/storage/aio/aio_init.c | 123 ++++++++++++
src/backend/utils/misc/guc_tables.c | 13 ++
src/backend/utils/misc/postgresql.conf.sample | 2 +
src/backend/utils/resowner/resowner.c | 25 ++-
src/test/modules/test_aio/test_aio--1.0.sql | 21 +++
src/test/modules/test_aio/test_aio.c | 55 ++++++
src/tools/pgindent/typedefs.list | 1 +
13 files changed, 496 insertions(+), 2 deletions(-)
diff --git a/src/include/storage/aio.h b/src/include/storage/aio.h
index a4d9b51acfe..3047dc77ce1 100644
--- a/src/include/storage/aio.h
+++ b/src/include/storage/aio.h
@@ -337,6 +337,20 @@ extern bool pgaio_have_staged(void);
+/* --------------------------------------------------------------------------------
+ * Bounce Buffers
+ * --------------------------------------------------------------------------------
+ */
+
+extern PgAioBounceBuffer *pgaio_bounce_buffer_get(void);
+extern void pgaio_io_assoc_bounce_buffer(PgAioHandle *ioh, PgAioBounceBuffer *bb);
+extern uint32 pgaio_bounce_buffer_id(PgAioBounceBuffer *bb);
+extern void pgaio_bounce_buffer_release(PgAioBounceBuffer *bb);
+extern char *pgaio_bounce_buffer_buffer(PgAioBounceBuffer *bb);
+extern void pgaio_bounce_buffer_release_resowner(struct dlist_node *bb_node, bool on_error);
+
+
+
/* --------------------------------------------------------------------------------
* Other
* --------------------------------------------------------------------------------
@@ -349,6 +363,7 @@ extern void pgaio_closing_fd(int fd);
/* GUCs */
extern PGDLLIMPORT int io_method;
extern PGDLLIMPORT int io_max_concurrency;
+extern PGDLLIMPORT int io_bounce_buffers;
#endif /* AIO_H */
diff --git a/src/include/storage/aio_internal.h b/src/include/storage/aio_internal.h
index 98ae66921f5..166a66f7062 100644
--- a/src/include/storage/aio_internal.h
+++ b/src/include/storage/aio_internal.h
@@ -122,6 +122,12 @@ struct PgAioHandle
/* raw result of the IO operation */
int32 result;
+ /*
+ * List of bounce_buffers owned by IO. It would suffice to use an index
+ * based linked list here.
+ */
+ slist_head bounce_buffers;
+
/**
* In which list the handle is registered, depends on the state:
* - IDLE, in per-backend list
@@ -177,11 +183,24 @@ struct PgAioHandle
};
+/* typedef is in aio_types.h */
+struct PgAioBounceBuffer
+{
+ slist_node node;
+ struct ResourceOwnerData *resowner;
+ dlist_node resowner_node;
+ char *buffer;
+};
+
+
typedef struct PgAioBackend
{
/* index into PgAioCtl->io_handles */
uint32 io_handle_off;
+ /* index into PgAioCtl->bounce_buffers */
+ uint32 bounce_buffers_off;
+
/* IO Handles that currently are not used */
dclist_head idle_ios;
@@ -212,6 +231,12 @@ typedef struct PgAioBackend
* IOs being appended at the end.
*/
dclist_head in_flight_ios;
+
+ /* Bounce Buffers that currently are not used */
+ slist_head idle_bbs;
+
+ /* see handed_out_io */
+ PgAioBounceBuffer *handed_out_bb;
} PgAioBackend;
@@ -239,6 +264,15 @@ typedef struct PgAioCtl
uint32 io_handle_count;
PgAioHandle *io_handles;
+
+ /*
+ * To perform AIO on buffers that are not located in shared memory (either
+ * because they are not in shared memory or because we need to operate on
+ * a copy, as e.g. the case for writes when checksums are in use)
+ */
+ uint32 bounce_buffers_count;
+ PgAioBounceBuffer *bounce_buffers;
+ char *bounce_buffers_data;
} PgAioCtl;
diff --git a/src/include/storage/aio_types.h b/src/include/storage/aio_types.h
index b0b9a2a5c97..09968e17855 100644
--- a/src/include/storage/aio_types.h
+++ b/src/include/storage/aio_types.h
@@ -118,4 +118,6 @@ typedef struct PgAioReturn
} PgAioReturn;
+typedef struct PgAioBounceBuffer PgAioBounceBuffer;
+
#endif /* AIO_TYPES_H */
diff --git a/src/include/utils/resowner.h b/src/include/utils/resowner.h
index aede4bfc820..7e2ec224169 100644
--- a/src/include/utils/resowner.h
+++ b/src/include/utils/resowner.h
@@ -168,5 +168,7 @@ extern void ResourceOwnerForgetLock(ResourceOwner owner, struct LOCALLOCK *local
struct dlist_node;
extern void ResourceOwnerRememberAioHandle(ResourceOwner owner, struct dlist_node *ioh_node);
extern void ResourceOwnerForgetAioHandle(ResourceOwner owner, struct dlist_node *ioh_node);
+extern void ResourceOwnerRememberAioBounceBuffer(ResourceOwner owner, struct dlist_node *bb_node);
+extern void ResourceOwnerForgetAioBounceBuffer(ResourceOwner owner, struct dlist_node *bb_node);
#endif /* RESOWNER_H */
diff --git a/src/backend/storage/aio/README.md b/src/backend/storage/aio/README.md
index b7dfb80b4b2..bb0bfc71ea6 100644
--- a/src/backend/storage/aio/README.md
+++ b/src/backend/storage/aio/README.md
@@ -405,6 +405,33 @@ shared memory no less!), completion callbacks instead have to encode errors in
a more compact format that can be converted into an error message.
+### AIO Bounce Buffers
+
+For some uses of AIO there is no convenient memory location as the source /
+destination of an AIO. E.g. when data checksums are enabled, writes from
+shared buffers currently cannot be done directly from shared buffers, as a
+shared buffer lock still allows some modification, e.g., for hint bits(see
+`FlushBuffer()`). If the write were done in-place, such modifications can
+cause the checksum to fail.
+
+For synchronous IO this is solved by copying the buffer to separate memory
+before computing the checksum and using that copy as the source buffer for the
+AIO.
+
+However, for AIO that is not a workable solution:
+- Instead of a single buffer many buffers are required, as many IOs might be
+ in flight
+- When using the [worker method](#worker), the source/target of IO needs to be
+ in shared memory, otherwise the workers won't be able to access the memory.
+
+The AIO subsystem addresses this by providing a limited number of bounce
+buffers that can be used as the source / target for IO. A bounce buffer be
+acquired with `pgaio_bounce_buffer_get()` and multiple bounce buffers can be
+associated with an AIO Handle with `pgaio_io_assoc_bounce_buffer()`.
+
+Bounce buffers are automatically released when the IO completes.
+
+
## Helpers
Using the low-level AIO API introduces too much complexity to do so all over
diff --git a/src/backend/storage/aio/aio.c b/src/backend/storage/aio/aio.c
index 3f126762a22..8ad6be06ca1 100644
--- a/src/backend/storage/aio/aio.c
+++ b/src/backend/storage/aio/aio.c
@@ -62,6 +62,8 @@ static PgAioHandle *pgaio_io_from_wref(PgAioWaitRef *iow, uint64 *ref_generation
static const char *pgaio_io_state_get_name(PgAioHandleState s);
static void pgaio_io_wait(PgAioHandle *ioh, uint64 ref_generation);
+static void pgaio_bounce_buffer_wait_for_free(void);
+
/* Options for io_method. */
const struct config_enum_entry io_method_options[] = {
@@ -76,6 +78,7 @@ const struct config_enum_entry io_method_options[] = {
/* GUCs */
int io_method = DEFAULT_IO_METHOD;
int io_max_concurrency = -1;
+int io_bounce_buffers = -1;
/* global control for AIO */
PgAioCtl *pgaio_ctl;
@@ -664,6 +667,21 @@ pgaio_io_reclaim(PgAioHandle *ioh)
}
}
+ /* reclaim all associated bounce buffers */
+ if (!slist_is_empty(&ioh->bounce_buffers))
+ {
+ slist_mutable_iter it;
+
+ slist_foreach_modify(it, &ioh->bounce_buffers)
+ {
+ PgAioBounceBuffer *bb = slist_container(PgAioBounceBuffer, node, it.cur);
+
+ slist_delete_current(&it);
+
+ slist_push_head(&pgaio_my_backend->idle_bbs, &bb->node);
+ }
+ }
+
if (ioh->resowner)
{
ResourceOwnerForgetAioHandle(ioh->resowner, &ioh->resowner_node);
@@ -1045,6 +1063,166 @@ pgaio_submit_staged(void)
+/* --------------------------------------------------------------------------------
+ * Functions primarily related to PgAioBounceBuffer
+ * --------------------------------------------------------------------------------
+ */
+
+PgAioBounceBuffer *
+pgaio_bounce_buffer_get(void)
+{
+ PgAioBounceBuffer *bb = NULL;
+ slist_node *node;
+
+ if (pgaio_my_backend->handed_out_bb != NULL)
+ elog(ERROR, "can only hand out one BB");
+
+ /*
+ * XXX: It probably is not a good idea to have bounce buffers be per
+ * backend, that's a fair bit of memory.
+ */
+ if (slist_is_empty(&pgaio_my_backend->idle_bbs))
+ {
+ pgaio_bounce_buffer_wait_for_free();
+ }
+
+ node = slist_pop_head_node(&pgaio_my_backend->idle_bbs);
+ bb = slist_container(PgAioBounceBuffer, node, node);
+
+ pgaio_my_backend->handed_out_bb = bb;
+
+ bb->resowner = CurrentResourceOwner;
+ ResourceOwnerRememberAioBounceBuffer(bb->resowner, &bb->resowner_node);
+
+ return bb;
+}
+
+void
+pgaio_io_assoc_bounce_buffer(PgAioHandle *ioh, PgAioBounceBuffer *bb)
+{
+ if (pgaio_my_backend->handed_out_bb != bb)
+ elog(ERROR, "can only assign handed out BB");
+ pgaio_my_backend->handed_out_bb = NULL;
+
+ /*
+ * There can be many bounce buffers assigned in case of vectorized IOs.
+ */
+ slist_push_head(&ioh->bounce_buffers, &bb->node);
+
+ /* once associated with an IO, the IO has ownership */
+ ResourceOwnerForgetAioBounceBuffer(bb->resowner, &bb->resowner_node);
+ bb->resowner = NULL;
+}
+
+uint32
+pgaio_bounce_buffer_id(PgAioBounceBuffer *bb)
+{
+ return bb - pgaio_ctl->bounce_buffers;
+}
+
+void
+pgaio_bounce_buffer_release(PgAioBounceBuffer *bb)
+{
+ if (pgaio_my_backend->handed_out_bb != bb)
+ elog(ERROR, "can only release handed out BB");
+
+ slist_push_head(&pgaio_my_backend->idle_bbs, &bb->node);
+ pgaio_my_backend->handed_out_bb = NULL;
+
+ ResourceOwnerForgetAioBounceBuffer(bb->resowner, &bb->resowner_node);
+ bb->resowner = NULL;
+}
+
+void
+pgaio_bounce_buffer_release_resowner(dlist_node *bb_node, bool on_error)
+{
+ PgAioBounceBuffer *bb = dlist_container(PgAioBounceBuffer, resowner_node, bb_node);
+
+ Assert(bb->resowner);
+
+ if (!on_error)
+ elog(WARNING, "leaked AIO bounce buffer");
+
+ pgaio_bounce_buffer_release(bb);
+}
+
+char *
+pgaio_bounce_buffer_buffer(PgAioBounceBuffer *bb)
+{
+ return bb->buffer;
+}
+
+static void
+pgaio_bounce_buffer_wait_for_free(void)
+{
+ static uint32 lastpos = 0;
+
+ if (pgaio_my_backend->num_staged_ios > 0)
+ {
+ pgaio_debug(DEBUG2, "submitting %d, while acquiring free bb",
+ pgaio_my_backend->num_staged_ios);
+ pgaio_submit_staged();
+ }
+
+ for (uint32 i = lastpos; i < lastpos + io_max_concurrency; i++)
+ {
+ uint32 thisoff = pgaio_my_backend->io_handle_off + (i % io_max_concurrency);
+ PgAioHandle *ioh = &pgaio_ctl->io_handles[thisoff];
+
+ switch (ioh->state)
+ {
+ case PGAIO_HS_IDLE:
+ case PGAIO_HS_HANDED_OUT:
+ continue;
+ case PGAIO_HS_DEFINED: /* should have been submitted above */
+ case PGAIO_HS_STAGED:
+ elog(ERROR, "shouldn't get here with io:%d in state %d",
+ pgaio_io_get_id(ioh), ioh->state);
+ break;
+ case PGAIO_HS_COMPLETED_IO:
+ case PGAIO_HS_SUBMITTED:
+ if (!slist_is_empty(&ioh->bounce_buffers))
+ {
+ pgaio_debug_io(DEBUG2, ioh,
+ "waiting for IO to reclaim BB with %d in flight",
+ dclist_count(&pgaio_my_backend->in_flight_ios));
+
+ /* see comment in pgaio_io_wait_for_free() about raciness */
+ pgaio_io_wait(ioh, ioh->generation);
+
+ if (slist_is_empty(&pgaio_my_backend->idle_bbs))
+ elog(WARNING, "empty after wait");
+
+ if (!slist_is_empty(&pgaio_my_backend->idle_bbs))
+ {
+ lastpos = i;
+ return;
+ }
+ }
+ break;
+ case PGAIO_HS_COMPLETED_SHARED:
+ case PGAIO_HS_COMPLETED_LOCAL:
+ /* reclaim */
+ pgaio_io_reclaim(ioh);
+
+ if (!slist_is_empty(&pgaio_my_backend->idle_bbs))
+ {
+ lastpos = i;
+ return;
+ }
+ break;
+ }
+ }
+
+ /*
+ * The submission above could have caused the IO to complete at any time.
+ */
+ if (slist_is_empty(&pgaio_my_backend->idle_bbs))
+ elog(PANIC, "no more bbs");
+}
+
+
+
/* --------------------------------------------------------------------------------
* Other
* --------------------------------------------------------------------------------
diff --git a/src/backend/storage/aio/aio_init.c b/src/backend/storage/aio/aio_init.c
index d0d916ea5c9..c32d8e6cb3d 100644
--- a/src/backend/storage/aio/aio_init.c
+++ b/src/backend/storage/aio/aio_init.c
@@ -82,6 +82,32 @@ AioHandleDataShmemSize(void)
io_max_concurrency));
}
+static Size
+AioBounceBufferDescShmemSize(void)
+{
+ Size sz;
+
+ /* PgAioBounceBuffer itself */
+ sz = mul_size(sizeof(PgAioBounceBuffer),
+ mul_size(AioProcs(), io_bounce_buffers));
+
+ return sz;
+}
+
+static Size
+AioBounceBufferDataShmemSize(void)
+{
+ Size sz;
+
+ /* and the associated buffer */
+ sz = mul_size(BLCKSZ,
+ mul_size(io_bounce_buffers, AioProcs()));
+ /* memory for alignment */
+ sz += BLCKSZ;
+
+ return sz;
+}
+
/*
* Choose a suitable value for io_max_concurrency.
*
@@ -107,6 +133,33 @@ AioChooseMaxConcurrency(void)
return Min(max_proportional_pins, 64);
}
+/*
+ * Choose a suitable value for io_bounce_buffers.
+ *
+ * It's very unlikely to be useful to allocate more bounce buffers for each
+ * backend than the backend is allowed to pin. Additionally, bounce buffers
+ * currently are used for writes, it seems very uncommon for more than 10% of
+ * shared_buffers to be written out concurrently.
+ *
+ * XXX: This quickly can take up significant amounts of memory, the logic
+ * should probably fine tuned.
+ */
+static int
+AioChooseBounceBuffers(void)
+{
+ uint32 max_backends;
+ int max_proportional_pins;
+
+ /* Similar logic to LimitAdditionalPins() */
+ max_backends = MaxBackends + NUM_AUXILIARY_PROCS;
+ max_proportional_pins = (NBuffers / 10) / max_backends;
+
+ max_proportional_pins = Max(max_proportional_pins, 1);
+
+ /* apply upper limit */
+ return Min(max_proportional_pins, 256);
+}
+
Size
AioShmemSize(void)
{
@@ -131,11 +184,31 @@ AioShmemSize(void)
PGC_S_OVERRIDE);
}
+
+ /*
+ * If io_bounce_buffers is -1, we automatically choose a suitable value.
+ *
+ * See also comment above.
+ */
+ if (io_bounce_buffers == -1)
+ {
+ char buf[32];
+
+ snprintf(buf, sizeof(buf), "%d", AioChooseBounceBuffers());
+ SetConfigOption("io_bounce_buffers", buf, PGC_POSTMASTER,
+ PGC_S_DYNAMIC_DEFAULT);
+ if (io_bounce_buffers == -1) /* failed to apply it? */
+ SetConfigOption("io_bounce_buffers", buf, PGC_POSTMASTER,
+ PGC_S_OVERRIDE);
+ }
+
sz = add_size(sz, AioCtlShmemSize());
sz = add_size(sz, AioBackendShmemSize());
sz = add_size(sz, AioHandleShmemSize());
sz = add_size(sz, AioHandleIOVShmemSize());
sz = add_size(sz, AioHandleDataShmemSize());
+ sz = add_size(sz, AioBounceBufferDescShmemSize());
+ sz = add_size(sz, AioBounceBufferDataShmemSize());
if (pgaio_method_ops->shmem_size)
sz = add_size(sz, pgaio_method_ops->shmem_size());
@@ -150,6 +223,9 @@ AioShmemInit(void)
uint32 io_handle_off = 0;
uint32 iovec_off = 0;
uint32 per_backend_iovecs = io_max_concurrency * PG_IOV_MAX;
+ uint32 bounce_buffers_off = 0;
+ uint32 per_backend_bb = io_bounce_buffers;
+ char *bounce_buffers_data;
pgaio_ctl = (PgAioCtl *)
ShmemInitStruct("AioCtl", AioCtlShmemSize(), &found);
@@ -161,6 +237,7 @@ AioShmemInit(void)
pgaio_ctl->io_handle_count = AioProcs() * io_max_concurrency;
pgaio_ctl->iovec_count = AioProcs() * per_backend_iovecs;
+ pgaio_ctl->bounce_buffers_count = AioProcs() * per_backend_bb;
pgaio_ctl->backend_state = (PgAioBackend *)
ShmemInitStruct("AioBackend", AioBackendShmemSize(), &found);
@@ -173,6 +250,40 @@ AioShmemInit(void)
pgaio_ctl->handle_data = (uint64 *)
ShmemInitStruct("AioHandleData", AioHandleDataShmemSize(), &found);
+ pgaio_ctl->bounce_buffers = (PgAioBounceBuffer *)
+ ShmemInitStruct("AioBounceBufferDesc", AioBounceBufferDescShmemSize(),
+ &found);
+
+ bounce_buffers_data =
+ ShmemInitStruct("AioBounceBufferData", AioBounceBufferDataShmemSize(),
+ &found);
+ bounce_buffers_data =
+ (char *) TYPEALIGN(BLCKSZ, (uintptr_t) bounce_buffers_data);
+ pgaio_ctl->bounce_buffers_data = bounce_buffers_data;
+
+
+ /* Initialize IO handles. */
+ for (uint64 i = 0; i < pgaio_ctl->io_handle_count; i++)
+ {
+ PgAioHandle *ioh = &pgaio_ctl->io_handles[i];
+
+ ioh->op = PGAIO_OP_INVALID;
+ ioh->target = PGAIO_TID_INVALID;
+ ioh->state = PGAIO_HS_IDLE;
+
+ slist_init(&ioh->bounce_buffers);
+ }
+
+ /* Initialize Bounce Buffers. */
+ for (uint64 i = 0; i < pgaio_ctl->bounce_buffers_count; i++)
+ {
+ PgAioBounceBuffer *bb = &pgaio_ctl->bounce_buffers[i];
+
+ bb->buffer = bounce_buffers_data;
+ bounce_buffers_data += BLCKSZ;
+ }
+
+
for (int procno = 0; procno < AioProcs(); procno++)
{
PgAioBackend *bs = &pgaio_ctl->backend_state[procno];
@@ -180,9 +291,13 @@ AioShmemInit(void)
bs->io_handle_off = io_handle_off;
io_handle_off += io_max_concurrency;
+ bs->bounce_buffers_off = bounce_buffers_off;
+ bounce_buffers_off += per_backend_bb;
+
dclist_init(&bs->idle_ios);
memset(bs->staged_ios, 0, sizeof(PgAioHandle *) * PGAIO_SUBMIT_BATCH_SIZE);
dclist_init(&bs->in_flight_ios);
+ slist_init(&bs->idle_bbs);
/* initialize per-backend IOs */
for (int i = 0; i < io_max_concurrency; i++)
@@ -204,6 +319,14 @@ AioShmemInit(void)
dclist_push_tail(&bs->idle_ios, &ioh->node);
iovec_off += PG_IOV_MAX;
}
+
+ /* initialize per-backend bounce buffers */
+ for (int i = 0; i < per_backend_bb; i++)
+ {
+ PgAioBounceBuffer *bb = &pgaio_ctl->bounce_buffers[bs->bounce_buffers_off + i];
+
+ slist_push_head(&bs->idle_bbs, &bb->node);
+ }
}
out:
diff --git a/src/backend/utils/misc/guc_tables.c b/src/backend/utils/misc/guc_tables.c
index 4cc19bef686..7290793c9a1 100644
--- a/src/backend/utils/misc/guc_tables.c
+++ b/src/backend/utils/misc/guc_tables.c
@@ -3268,6 +3268,19 @@ struct config_int ConfigureNamesInt[] =
check_io_max_concurrency, NULL, NULL
},
+ {
+ {"io_bounce_buffers",
+ PGC_POSTMASTER,
+ RESOURCES_IO,
+ gettext_noop("Number of IO Bounce Buffers reserved for each backend."),
+ NULL,
+ GUC_UNIT_BLOCKS
+ },
+ &io_bounce_buffers,
+ -1, -1, 4096,
+ NULL, NULL, NULL
+ },
+
{
{"io_workers",
PGC_SIGHUP,
diff --git a/src/backend/utils/misc/postgresql.conf.sample b/src/backend/utils/misc/postgresql.conf.sample
index 7e8c3dcb175..ea71e9075ff 100644
--- a/src/backend/utils/misc/postgresql.conf.sample
+++ b/src/backend/utils/misc/postgresql.conf.sample
@@ -209,6 +209,8 @@
# -1 sets based on shared_buffers
# (change requires restart)
#io_workers = 3 # 1-32;
+#io_bounce_buffers = -1 # -1 sets based on shared_buffers
+ # (change requires restart)
# - Worker Processes -
diff --git a/src/backend/utils/resowner/resowner.c b/src/backend/utils/resowner/resowner.c
index d39f3e1b655..81e7e27965a 100644
--- a/src/backend/utils/resowner/resowner.c
+++ b/src/backend/utils/resowner/resowner.c
@@ -159,10 +159,11 @@ struct ResourceOwnerData
LOCALLOCK *locks[MAX_RESOWNER_LOCKS]; /* list of owned locks */
/*
- * AIO handles need be registered in critical sections and therefore
- * cannot use the normal ResourceElem mechanism.
+ * AIO handles & bounce buffers need be registered in critical sections
+ * and therefore cannot use the normal ResourceElem mechanism.
*/
dlist_head aio_handles;
+ dlist_head aio_bounce_buffers;
};
@@ -434,6 +435,7 @@ ResourceOwnerCreate(ResourceOwner parent, const char *name)
}
dlist_init(&owner->aio_handles);
+ dlist_init(&owner->aio_bounce_buffers);
return owner;
}
@@ -742,6 +744,13 @@ ResourceOwnerReleaseInternal(ResourceOwner owner,
pgaio_io_release_resowner(node, !isCommit);
}
+
+ while (!dlist_is_empty(&owner->aio_bounce_buffers))
+ {
+ dlist_node *node = dlist_head_node(&owner->aio_bounce_buffers);
+
+ pgaio_bounce_buffer_release_resowner(node, !isCommit);
+ }
}
else if (phase == RESOURCE_RELEASE_LOCKS)
{
@@ -1111,3 +1120,15 @@ ResourceOwnerForgetAioHandle(ResourceOwner owner, struct dlist_node *ioh_node)
{
dlist_delete_from(&owner->aio_handles, ioh_node);
}
+
+void
+ResourceOwnerRememberAioBounceBuffer(ResourceOwner owner, struct dlist_node *ioh_node)
+{
+ dlist_push_tail(&owner->aio_bounce_buffers, ioh_node);
+}
+
+void
+ResourceOwnerForgetAioBounceBuffer(ResourceOwner owner, struct dlist_node *ioh_node)
+{
+ dlist_delete_from(&owner->aio_bounce_buffers, ioh_node);
+}
diff --git a/src/test/modules/test_aio/test_aio--1.0.sql b/src/test/modules/test_aio/test_aio--1.0.sql
index 3d4e9bb070f..16f248619fc 100644
--- a/src/test/modules/test_aio/test_aio--1.0.sql
+++ b/src/test/modules/test_aio/test_aio--1.0.sql
@@ -76,6 +76,27 @@ RETURNS pg_catalog.void STRICT
AS 'MODULE_PATHNAME' LANGUAGE C;
+CREATE FUNCTION bb_get_and_error()
+RETURNS pg_catalog.void STRICT
+AS 'MODULE_PATHNAME' LANGUAGE C;
+
+CREATE FUNCTION bb_get_twice()
+RETURNS pg_catalog.void STRICT
+AS 'MODULE_PATHNAME' LANGUAGE C;
+
+CREATE FUNCTION bb_get()
+RETURNS pg_catalog.void STRICT
+AS 'MODULE_PATHNAME' LANGUAGE C;
+
+CREATE FUNCTION bb_get_release()
+RETURNS pg_catalog.void STRICT
+AS 'MODULE_PATHNAME' LANGUAGE C;
+
+CREATE FUNCTION bb_release_last()
+RETURNS pg_catalog.void STRICT
+AS 'MODULE_PATHNAME' LANGUAGE C;
+
+
/*
* Injection point related functions
diff --git a/src/test/modules/test_aio/test_aio.c b/src/test/modules/test_aio/test_aio.c
index 81b4d732206..60c01b251fe 100644
--- a/src/test/modules/test_aio/test_aio.c
+++ b/src/test/modules/test_aio/test_aio.c
@@ -53,6 +53,7 @@ static shmem_startup_hook_type prev_shmem_startup_hook = NULL;
static PgAioHandle *last_handle;
+static PgAioBounceBuffer *last_bb;
@@ -525,6 +526,60 @@ batch_end(PG_FUNCTION_ARGS)
PG_RETURN_VOID();
}
+PG_FUNCTION_INFO_V1(bb_get);
+Datum
+bb_get(PG_FUNCTION_ARGS)
+{
+ last_bb = pgaio_bounce_buffer_get();
+
+ PG_RETURN_VOID();
+}
+
+PG_FUNCTION_INFO_V1(bb_release_last);
+Datum
+bb_release_last(PG_FUNCTION_ARGS)
+{
+ if (!last_bb)
+ elog(ERROR, "no bb");
+
+ pgaio_bounce_buffer_release(last_bb);
+
+ PG_RETURN_VOID();
+}
+
+PG_FUNCTION_INFO_V1(bb_get_and_error);
+Datum
+bb_get_and_error(PG_FUNCTION_ARGS)
+{
+ pgaio_bounce_buffer_get();
+
+ elog(ERROR, "as you command");
+ PG_RETURN_VOID();
+}
+
+PG_FUNCTION_INFO_V1(bb_get_twice);
+Datum
+bb_get_twice(PG_FUNCTION_ARGS)
+{
+ pgaio_bounce_buffer_get();
+ pgaio_bounce_buffer_get();
+
+ PG_RETURN_VOID();
+}
+
+
+PG_FUNCTION_INFO_V1(bb_get_release);
+Datum
+bb_get_release(PG_FUNCTION_ARGS)
+{
+ PgAioBounceBuffer *bb;
+
+ bb = pgaio_bounce_buffer_get();
+ pgaio_bounce_buffer_release(bb);
+
+ PG_RETURN_VOID();
+}
+
#ifdef USE_INJECTION_POINTS
extern PGDLLEXPORT void inj_io_short_read(const char *name, const void *private_data);
extern PGDLLEXPORT void inj_io_reopen(const char *name, const void *private_data);
diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list
index 1748befca16..6ffd4e82976 100644
--- a/src/tools/pgindent/typedefs.list
+++ b/src/tools/pgindent/typedefs.list
@@ -2132,6 +2132,7 @@ PermutationStep
PermutationStepBlocker
PermutationStepBlockerType
PgAioBackend
+PgAioBounceBuffer
PgAioCtl
PgAioHandle
PgAioHandleCallbackID
--
2.48.1.76.g4e746b1a31.dirty
[text/x-diff] v2.8-0031-bufmgr-Implement-AIO-write-support.patch (9.9K, ../../3yxd5r23zly5bytvgyktbxtxq2r3gbpi7xd4dugevh3h4w4q6c@lu6oatjjpltz/32-v2.8-0031-bufmgr-Implement-AIO-write-support.patch)
download | inline diff:
From 3b7b8829481e08137d42fbd660976923dc8b293a Mon Sep 17 00:00:00 2001
From: Andres Freund <andres@anarazel.de>
Date: Tue, 11 Mar 2025 11:33:13 -0400
Subject: [PATCH v2.8 31/38] bufmgr: Implement AIO write support
As of this commit there are no users of these AIO facilities, that'll come in
later commits.
Author:
Reviewed-By:
Discussion: https://postgr.es/m/
Backpatch:
---
src/include/storage/aio.h | 2 +
src/include/storage/bufmgr.h | 2 +
src/backend/storage/aio/aio_callback.c | 2 +
src/backend/storage/buffer/bufmgr.c | 182 ++++++++++++++++++++++++-
4 files changed, 187 insertions(+), 1 deletion(-)
diff --git a/src/include/storage/aio.h b/src/include/storage/aio.h
index 3047dc77ce1..192abc5a712 100644
--- a/src/include/storage/aio.h
+++ b/src/include/storage/aio.h
@@ -186,8 +186,10 @@ typedef enum PgAioHandleCallbackID
PGAIO_HCB_MD_WRITEV,
PGAIO_HCB_SHARED_BUFFER_READV,
+ PGAIO_HCB_SHARED_BUFFER_WRITEV,
PGAIO_HCB_LOCAL_BUFFER_READV,
+ PGAIO_HCB_LOCAL_BUFFER_WRITEV,
} PgAioHandleCallbackID;
diff --git a/src/include/storage/bufmgr.h b/src/include/storage/bufmgr.h
index ca4ca22c345..fffc76d11d9 100644
--- a/src/include/storage/bufmgr.h
+++ b/src/include/storage/bufmgr.h
@@ -176,7 +176,9 @@ extern PGDLLIMPORT int backend_flush_after;
extern PGDLLIMPORT int bgwriter_flush_after;
extern const PgAioHandleCallbacks aio_shared_buffer_readv_cb;
+extern const PgAioHandleCallbacks aio_shared_buffer_writev_cb;
extern const PgAioHandleCallbacks aio_local_buffer_readv_cb;
+extern const PgAioHandleCallbacks aio_local_buffer_writev_cb;
/* in buf_init.c */
extern PGDLLIMPORT char *BufferBlocks;
diff --git a/src/backend/storage/aio/aio_callback.c b/src/backend/storage/aio/aio_callback.c
index fb6ac058a09..7162f722e3c 100644
--- a/src/backend/storage/aio/aio_callback.c
+++ b/src/backend/storage/aio/aio_callback.c
@@ -44,8 +44,10 @@ static const PgAioHandleCallbacksEntry aio_handle_cbs[] = {
CALLBACK_ENTRY(PGAIO_HCB_MD_WRITEV, aio_md_writev_cb),
CALLBACK_ENTRY(PGAIO_HCB_SHARED_BUFFER_READV, aio_shared_buffer_readv_cb),
+ CALLBACK_ENTRY(PGAIO_HCB_SHARED_BUFFER_WRITEV, aio_shared_buffer_writev_cb),
CALLBACK_ENTRY(PGAIO_HCB_LOCAL_BUFFER_READV, aio_local_buffer_readv_cb),
+ CALLBACK_ENTRY(PGAIO_HCB_LOCAL_BUFFER_WRITEV, aio_local_buffer_writev_cb),
#undef CALLBACK_ENTRY
};
diff --git a/src/backend/storage/buffer/bufmgr.c b/src/backend/storage/buffer/bufmgr.c
index 860886d4d90..886fbaf63f3 100644
--- a/src/backend/storage/buffer/bufmgr.c
+++ b/src/backend/storage/buffer/bufmgr.c
@@ -5507,7 +5507,15 @@ LockBuffer(Buffer buffer, int mode)
else if (mode == BUFFER_LOCK_SHARE)
LWLockAcquire(BufferDescriptorGetContentLock(buf), LW_SHARED);
else if (mode == BUFFER_LOCK_EXCLUSIVE)
+ {
+ /*
+ * FIXME: Wait for AIO writes, otherwise there would be a risk of
+ * deadlock. This isn't entirely trivial to do in a race-free way, IO
+ * could be started between us checking whether there is IO and
+ * enqueueing ourselves for the lock.
+ */
LWLockAcquire(BufferDescriptorGetContentLock(buf), LW_EXCLUSIVE);
+ }
else
elog(ERROR, "unrecognized buffer lock mode: %d", mode);
}
@@ -5522,6 +5530,19 @@ ConditionalLockBuffer(Buffer buffer)
{
BufferDesc *buf;
+ /*
+ * FIXME: Wait for AIO writes. Some code does not deal well
+ * ConditionalLockBuffer() continuously failing, e.g.
+ * spginsert()->spgdoinsert() ends up busy-looping (easily reproducible by
+ * just making this function always fail and running the regression
+ * tests). While that code could be fixed, it'd be hard to find all
+ * problematic places.
+ *
+ * It would be OK to wait for the IO as waiting for IO completion does not
+ * need to wait for any locks that could lead to an undetected deadlock or
+ * such.
+ */
+
Assert(BufferIsPinned(buffer));
if (BufferIsLocal(buffer))
return true; /* act as though we got it */
@@ -5594,6 +5615,11 @@ LockBufferForCleanup(Buffer buffer)
{
uint32 buf_state;
+ /*
+ * FIXME: LockBuffer()'s handling of in-progress writes (once
+ * implemented) should suffice to deal with deadlock risk.
+ */
+
/* Try to acquire lock */
LockBuffer(buffer, BUFFER_LOCK_EXCLUSIVE);
buf_state = LockBufHdr(bufHdr);
@@ -5741,6 +5767,14 @@ ConditionalLockBufferForCleanup(Buffer buffer)
Assert(BufferIsValid(buffer));
+ /*
+ * FIXME: Should wait for IO for the same reason as in
+ * ConditionalLockBuffer(). Needs to happen before the
+ * ConditionalLockBuffer() call below, as we'd never reach the
+ * ConditionalLockBuffer() call due the buffer pin held for the duration
+ * of the IO.
+ */
+
if (BufferIsLocal(buffer))
{
refcount = LocalRefCount[-buffer - 1];
@@ -6802,12 +6836,131 @@ buffer_readv_report(PgAioResult result, const PgAioTargetData *target_data, int
);
}
+/*
+ * Helper for AIO writev completion callbacks, supporting both shared and temp
+ * buffers. Gets called once for each buffer in a multi-page write.
+ */
+static pg_attribute_always_inline PgAioResult
+buffer_writev_complete_one(uint8 buf_off, Buffer buffer, uint8 flags,
+ bool failed, bool is_temp)
+{
+ BufferDesc *buf_hdr = is_temp ?
+ GetLocalBufferDescriptor(-buffer - 1)
+ : GetBufferDescriptor(buffer - 1);
+ PgAioResult result;
+ bool clear_dirty;
+ uint32 set_flag_bits;
+
+#ifdef USE_ASSERT_CHECKING
+ {
+ uint32 buf_state = pg_atomic_read_u32(&buf_hdr->state);
+
+ Assert(buf_state & BM_VALID);
+ Assert(buf_state & BM_TAG_VALID);
+ /* temp buffers don't use BM_IO_IN_PROGRESS */
+ if (!is_temp)
+ Assert(buf_state & BM_IO_IN_PROGRESS);
+ Assert(buf_state & BM_DIRTY);
+ }
+#endif
+
+ result.status = ARS_OK;
+
+ clear_dirty = failed ? false : true;
+ set_flag_bits = failed ? BM_IO_ERROR : 0;
+
+ if (is_temp)
+ TerminateLocalBufferIO(buf_hdr, clear_dirty, set_flag_bits, false);
+ else
+ TerminateBufferIO(buf_hdr, clear_dirty, set_flag_bits, false, false);
+
+ /*
+ * The initiator of IO is not managing the lock (i.e. we called
+ * LWLockDisown()), we are.
+ */
+ if (!is_temp)
+ LWLockReleaseDisowned(BufferDescriptorGetContentLock(buf_hdr),
+ LW_SHARED);
+
+ /* FIXME: tracepoint */
+
+ return result;
+}
+
+/*
+ * Perform completion handling of a single AIO write. This write may cover
+ * multiple blocks / buffers.
+ *
+ * Shared between shared and local buffers, to reduce code duplication.
+ */
+static pg_attribute_always_inline PgAioResult
+buffer_writev_complete(PgAioHandle *ioh, PgAioResult prior_result,
+ uint8 cb_data, bool is_temp)
+{
+ PgAioResult result = prior_result;
+ PgAioTargetData *td = pgaio_io_get_target_data(ioh);
+ uint64 *io_data;
+ uint8 handle_data_len;
+
+ if (is_temp)
+ {
+ Assert(td->smgr.is_temp);
+ Assert(pgaio_io_get_owner(ioh) == MyProcNumber);
+ }
+ else
+ Assert(!td->smgr.is_temp);
+
+ /*
+ * Iterate over all the buffers affected by this IO and call appropriate
+ * per-buffer completion function for each buffer.
+ */
+ io_data = pgaio_io_get_handle_data(ioh, &handle_data_len);
+ for (uint8 buf_off = 0; buf_off < handle_data_len; buf_off++)
+ {
+ Buffer buf = io_data[buf_off];
+ PgAioResult buf_result;
+ bool failed;
+
+ Assert(BufferIsValid(buf));
+
+ /*
+ * If the entire failed on a lower-level, each buffer needs to be
+ * marked as failed. In case of a partial read, some buffers may be
+ * ok.
+ */
+ failed =
+ prior_result.status == ARS_ERROR
+ || prior_result.result <= buf_off;
+
+ buf_result = buffer_writev_complete_one(buf_off, buf, cb_data, failed,
+ is_temp);
+
+ /*
+ * If there wasn't any prior error and the IO for this page failed in
+ * some form, set the whole IO's to the page's result.
+ */
+ if (result.status != ARS_ERROR && buf_result.status != ARS_OK)
+ {
+ result = buf_result;
+ pgaio_result_report(result, td, LOG);
+ }
+ }
+
+ return result;
+}
+
static void
shared_buffer_readv_stage(PgAioHandle *ioh, uint8 cb_data)
{
buffer_stage_common(ioh, false, false);
}
+static void
+shared_buffer_writev_stage(PgAioHandle *ioh, uint8 cb_data)
+{
+ buffer_stage_common(ioh, true, false);
+}
+
static PgAioResult
shared_buffer_readv_complete(PgAioHandle *ioh, PgAioResult prior_result,
uint8 cb_data)
@@ -6815,6 +6968,13 @@ shared_buffer_readv_complete(PgAioHandle *ioh, PgAioResult prior_result,
return buffer_readv_complete(ioh, prior_result, cb_data, false);
}
+static PgAioResult
+shared_buffer_writev_complete(PgAioHandle *ioh, PgAioResult prior_result,
+ uint8 cb_data)
+{
+ return buffer_writev_complete(ioh, prior_result, cb_data, false);
+}
+
static void
local_buffer_readv_stage(PgAioHandle *ioh, uint8 cb_data)
{
@@ -6828,13 +6988,29 @@ local_buffer_readv_complete(PgAioHandle *ioh, PgAioResult prior_result,
return buffer_readv_complete(ioh, prior_result, cb_data, true);
}
-/* readv callback is is passed READ_BUFFERS_* flags as callback data */
+static void
+local_buffer_writev_stage(PgAioHandle *ioh, uint8 cb_data)
+{
+ /*
+ * Currently this is unreachable as the only write support is for
+ * checkpointer / bgwriter, which don't deal with local buffers.
+ */
+ elog(ERROR, "should be unreachable");
+}
+
+
+/* readv callback is passed READ_BUFFERS_* flags as callback data */
const PgAioHandleCallbacks aio_shared_buffer_readv_cb = {
.stage = shared_buffer_readv_stage,
.complete_shared = shared_buffer_readv_complete,
.report = buffer_readv_report,
};
+const PgAioHandleCallbacks aio_shared_buffer_writev_cb = {
+ .stage = shared_buffer_writev_stage,
+ .complete_shared = shared_buffer_writev_complete,
+};
+
/* readv callback is is passed READ_BUFFERS_* flags as callback data */
const PgAioHandleCallbacks aio_local_buffer_readv_cb = {
.stage = local_buffer_readv_stage,
@@ -6848,3 +7024,7 @@ const PgAioHandleCallbacks aio_local_buffer_readv_cb = {
.complete_local = local_buffer_readv_complete,
.report = buffer_readv_report,
};
+
+const PgAioHandleCallbacks aio_local_buffer_writev_cb = {
+ .stage = local_buffer_writev_stage,
+};
--
2.48.1.76.g4e746b1a31.dirty
[text/x-diff] v2.8-0032-aio-Add-IO-queue-helper.patch (7.4K, ../../3yxd5r23zly5bytvgyktbxtxq2r3gbpi7xd4dugevh3h4w4q6c@lu6oatjjpltz/33-v2.8-0032-aio-Add-IO-queue-helper.patch)
download | inline diff:
From 0a47b96406ba63839ffe9bde619229cad8fc135e Mon Sep 17 00:00:00 2001
From: Andres Freund <andres@anarazel.de>
Date: Wed, 22 Jan 2025 13:44:50 -0500
Subject: [PATCH v2.8 32/38] aio: Add IO queue helper
This is likely never going to anywhere - Thomas Munro is working on something
more complete. But I needed a way to exercise aio for checkpointer / bgwriter.
---
src/include/storage/io_queue.h | 33 +++++
src/backend/storage/aio/Makefile | 1 +
src/backend/storage/aio/io_queue.c | 204 ++++++++++++++++++++++++++++
src/backend/storage/aio/meson.build | 1 +
src/tools/pgindent/typedefs.list | 2 +
5 files changed, 241 insertions(+)
create mode 100644 src/include/storage/io_queue.h
create mode 100644 src/backend/storage/aio/io_queue.c
diff --git a/src/include/storage/io_queue.h b/src/include/storage/io_queue.h
new file mode 100644
index 00000000000..92b1e9afe6f
--- /dev/null
+++ b/src/include/storage/io_queue.h
@@ -0,0 +1,33 @@
+/*-------------------------------------------------------------------------
+ *
+ * io_queue.h
+ * Mechanism for tracking many IOs
+ *
+ *
+ * Portions Copyright (c) 1996-2025, PostgreSQL Global Development Group
+ * Portions Copyright (c) 1994, Regents of the University of California
+ *
+ * src/include/storage/io_queue.h
+ *
+ *-------------------------------------------------------------------------
+ */
+#ifndef IO_QUEUE_H
+#define IO_QUEUE_H
+
+#include "storage/aio_types.h"
+
+struct IOQueue;
+typedef struct IOQueue IOQueue;
+
+struct PgAioWaitRef;
+
+extern IOQueue *io_queue_create(int depth, int flags);
+extern void io_queue_track(IOQueue *ioq, const PgAioWaitRef *iow);
+extern void io_queue_wait_one(IOQueue *ioq);
+extern void io_queue_wait_all(IOQueue *ioq);
+extern bool io_queue_is_empty(IOQueue *ioq);
+extern void io_queue_reserve(IOQueue *ioq);
+extern PgAioHandle *io_queue_acquire_io(IOQueue *ioq);
+extern void io_queue_free(IOQueue *ioq);
+
+#endif /* IO_QUEUE_H */
diff --git a/src/backend/storage/aio/Makefile b/src/backend/storage/aio/Makefile
index 3f2469cc399..86fa4276fda 100644
--- a/src/backend/storage/aio/Makefile
+++ b/src/backend/storage/aio/Makefile
@@ -15,6 +15,7 @@ OBJS = \
aio_init.o \
aio_io.o \
aio_target.o \
+ io_queue.o \
method_io_uring.o \
method_sync.o \
method_worker.o \
diff --git a/src/backend/storage/aio/io_queue.c b/src/backend/storage/aio/io_queue.c
new file mode 100644
index 00000000000..526aa1d5e06
--- /dev/null
+++ b/src/backend/storage/aio/io_queue.c
@@ -0,0 +1,204 @@
+/*-------------------------------------------------------------------------
+ *
+ * io_queue.c
+ * AIO - Mechanism for tracking many IOs
+ *
+ * Portions Copyright (c) 1996-2025, PostgreSQL Global Development Group
+ * Portions Copyright (c) 1994, Regents of the University of California
+ *
+ * IDENTIFICATION
+ * src/backend/storage/aio/io_queue.c
+ *
+ *-------------------------------------------------------------------------
+ */
+
+#include "postgres.h"
+
+#include "lib/ilist.h"
+#include "storage/aio.h"
+#include "storage/io_queue.h"
+#include "utils/resowner.h"
+
+
+
+typedef struct TrackedIO
+{
+ PgAioWaitRef iow;
+ dlist_node node;
+} TrackedIO;
+
+struct IOQueue
+{
+ int depth;
+ int unsubmitted;
+
+ bool has_reserved;
+
+ dclist_head idle;
+ dclist_head in_progress;
+
+ TrackedIO tracked_ios[FLEXIBLE_ARRAY_MEMBER];
+};
+
+
+IOQueue *
+io_queue_create(int depth, int flags)
+{
+ size_t sz;
+ IOQueue *ioq;
+
+ sz = offsetof(IOQueue, tracked_ios)
+ + sizeof(TrackedIO) * depth;
+
+ ioq = palloc0(sz);
+
+ ioq->depth = 0;
+
+ for (int i = 0; i < depth; i++)
+ {
+ TrackedIO *tio = &ioq->tracked_ios[i];
+
+ pgaio_wref_clear(&tio->iow);
+ dclist_push_tail(&ioq->idle, &tio->node);
+ }
+
+ return ioq;
+}
+
+void
+io_queue_wait_one(IOQueue *ioq)
+{
+ /* submit all pending IO before waiting */
+ pgaio_submit_staged();
+
+ while (!dclist_is_empty(&ioq->in_progress))
+ {
+ /* FIXME: Should we really pop here already? */
+ dlist_node *node = dclist_pop_head_node(&ioq->in_progress);
+ TrackedIO *tio = dclist_container(TrackedIO, node, node);
+
+ pgaio_wref_wait(&tio->iow);
+ dclist_push_head(&ioq->idle, &tio->node);
+ }
+}
+
+void
+io_queue_reserve(IOQueue *ioq)
+{
+ if (ioq->has_reserved)
+ return;
+
+ if (dclist_is_empty(&ioq->idle))
+ io_queue_wait_one(ioq);
+
+ Assert(!dclist_is_empty(&ioq->idle));
+
+ ioq->has_reserved = true;
+}
+
+PgAioHandle *
+io_queue_acquire_io(IOQueue *ioq)
+{
+ PgAioHandle *ioh;
+
+ io_queue_reserve(ioq);
+
+ Assert(!dclist_is_empty(&ioq->idle));
+
+ if (!io_queue_is_empty(ioq))
+ {
+ ioh = pgaio_io_acquire_nb(CurrentResourceOwner, NULL);
+ if (ioh == NULL)
+ {
+ /*
+ * Need to wait for all IOs, blocking might not be legal in the
+ * context.
+ *
+ * XXX: This doesn't make a whole lot of sense, we're also
+ * blocking here. What was I smoking when I wrote the above?
+ */
+ io_queue_wait_all(ioq);
+ ioh = pgaio_io_acquire(CurrentResourceOwner, NULL);
+ }
+ }
+ else
+ {
+ ioh = pgaio_io_acquire(CurrentResourceOwner, NULL);
+ }
+
+ return ioh;
+}
+
+void
+io_queue_track(IOQueue *ioq, const struct PgAioWaitRef *iow)
+{
+ dlist_node *node;
+ TrackedIO *tio;
+
+ Assert(ioq->has_reserved);
+ ioq->has_reserved = false;
+
+ Assert(!dclist_is_empty(&ioq->idle));
+
+ node = dclist_pop_head_node(&ioq->idle);
+ tio = dclist_container(TrackedIO, node, node);
+
+ tio->iow = *iow;
+
+ dclist_push_tail(&ioq->in_progress, &tio->node);
+
+ ioq->unsubmitted++;
+
+ /*
+ * XXX: Should have some smarter logic here. We don't want to wait too
+ * long to submit, that'll mean we're more likely to block. But we also
+ * don't want to have the overhead of submitting every IO individually.
+ */
+ if (ioq->unsubmitted >= 4)
+ {
+ pgaio_submit_staged();
+ ioq->unsubmitted = 0;
+ }
+}
+
+void
+io_queue_wait_all(IOQueue *ioq)
+{
+ /* submit all pending IO before waiting */
+ pgaio_submit_staged();
+
+ while (!dclist_is_empty(&ioq->in_progress))
+ {
+ /* wait for the last IO to minimize unnecessary wakeups */
+ dlist_node *node = dclist_tail_node(&ioq->in_progress);
+ TrackedIO *tio = dclist_container(TrackedIO, node, node);
+
+ if (!pgaio_wref_check_done(&tio->iow))
+ {
+ ereport(DEBUG3,
+ errmsg("io_queue_wait_all for io:%d",
+ pgaio_wref_get_id(&tio->iow)),
+ errhidestmt(true),
+ errhidecontext(true));
+
+ pgaio_wref_wait(&tio->iow);
+ }
+
+ dclist_delete_from(&ioq->in_progress, &tio->node);
+ dclist_push_head(&ioq->idle, &tio->node);
+ }
+}
+
+bool
+io_queue_is_empty(IOQueue *ioq)
+{
+ return dclist_is_empty(&ioq->in_progress);
+}
+
+void
+io_queue_free(IOQueue *ioq)
+{
+ io_queue_wait_all(ioq);
+
+ pfree(ioq);
+}
diff --git a/src/backend/storage/aio/meson.build b/src/backend/storage/aio/meson.build
index da6df2d3654..270c4a64428 100644
--- a/src/backend/storage/aio/meson.build
+++ b/src/backend/storage/aio/meson.build
@@ -7,6 +7,7 @@ backend_sources += files(
'aio_init.c',
'aio_io.c',
'aio_target.c',
+ 'io_queue.c',
'method_io_uring.c',
'method_sync.c',
'method_worker.c',
diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list
index 6ffd4e82976..26c622824e4 100644
--- a/src/tools/pgindent/typedefs.list
+++ b/src/tools/pgindent/typedefs.list
@@ -1191,6 +1191,7 @@ IOContext
IOFuncSelector
IOObject
IOOp
+IOQueue
IO_STATUS_BLOCK
IPCompareMethod
ITEM
@@ -3016,6 +3017,7 @@ TocEntry
TokenAuxData
TokenizedAuthLine
TrackItem
+TrackedIO
TransApplyAction
TransInvalidationInfo
TransState
--
2.48.1.76.g4e746b1a31.dirty
[text/x-diff] v2.8-0033-bufmgr-use-AIO-in-checkpointer-bgwriter.patch (31.1K, ../../3yxd5r23zly5bytvgyktbxtxq2r3gbpi7xd4dugevh3h4w4q6c@lu6oatjjpltz/34-v2.8-0033-bufmgr-use-AIO-in-checkpointer-bgwriter.patch)
download | inline diff:
From 44f06b0060d94daeb3d467e9f4f364016fdc3615 Mon Sep 17 00:00:00 2001
From: Andres Freund <andres@anarazel.de>
Date: Wed, 22 Jan 2025 13:44:52 -0500
Subject: [PATCH v2.8 33/38] bufmgr: use AIO in checkpointer, bgwriter
This is far from ready - just included to be able to exercise AIO writes and
get some preliminary numbers. In all likelihood this will instead be based
on-top of work by Thomas Munro instead of the preceding commit.
---
src/include/postmaster/bgwriter.h | 3 +-
src/include/storage/buf_internals.h | 2 +
src/include/storage/bufmgr.h | 3 +-
src/include/storage/bufpage.h | 1 +
src/backend/postmaster/bgwriter.c | 19 +-
src/backend/postmaster/checkpointer.c | 11 +-
src/backend/storage/buffer/bufmgr.c | 588 +++++++++++++++++++++++---
src/backend/storage/page/bufpage.c | 10 +
src/tools/pgindent/typedefs.list | 1 +
9 files changed, 580 insertions(+), 58 deletions(-)
diff --git a/src/include/postmaster/bgwriter.h b/src/include/postmaster/bgwriter.h
index 800ecbfd13b..a8081d411b6 100644
--- a/src/include/postmaster/bgwriter.h
+++ b/src/include/postmaster/bgwriter.h
@@ -31,7 +31,8 @@ pg_noreturn extern void BackgroundWriterMain(const void *startup_data, size_t st
pg_noreturn extern void CheckpointerMain(const void *startup_data, size_t startup_data_len);
extern void RequestCheckpoint(int flags);
-extern void CheckpointWriteDelay(int flags, double progress);
+struct IOQueue;
+extern void CheckpointWriteDelay(struct IOQueue *ioq, int flags, double progress);
extern bool ForwardSyncRequest(const FileTag *ftag, SyncRequestType type);
diff --git a/src/include/storage/buf_internals.h b/src/include/storage/buf_internals.h
index 4b1f52a9fc8..a3df3192d12 100644
--- a/src/include/storage/buf_internals.h
+++ b/src/include/storage/buf_internals.h
@@ -21,6 +21,8 @@
#include "storage/buf.h"
#include "storage/bufmgr.h"
#include "storage/condition_variable.h"
+#include "storage/io_queue.h"
+#include "storage/latch.h"
#include "storage/lwlock.h"
#include "storage/procnumber.h"
#include "storage/shmem.h"
diff --git a/src/include/storage/bufmgr.h b/src/include/storage/bufmgr.h
index fffc76d11d9..30d965ec069 100644
--- a/src/include/storage/bufmgr.h
+++ b/src/include/storage/bufmgr.h
@@ -299,7 +299,8 @@ extern bool ConditionalLockBufferForCleanup(Buffer buffer);
extern bool IsBufferCleanupOK(Buffer buffer);
extern bool HoldingBufferPinThatDelaysRecovery(void);
-extern bool BgBufferSync(struct WritebackContext *wb_context);
+struct IOQueue;
+extern bool BgBufferSync(struct IOQueue *ioq, struct WritebackContext *wb_context);
extern uint32 GetPinLimit(void);
extern uint32 GetLocalPinLimit(void);
diff --git a/src/include/storage/bufpage.h b/src/include/storage/bufpage.h
index 6646b6f6371..9c045e81857 100644
--- a/src/include/storage/bufpage.h
+++ b/src/include/storage/bufpage.h
@@ -509,5 +509,6 @@ extern bool PageIndexTupleOverwrite(Page page, OffsetNumber offnum,
Item newtup, Size newsize);
extern char *PageSetChecksumCopy(Page page, BlockNumber blkno);
extern void PageSetChecksumInplace(Page page, BlockNumber blkno);
+extern bool PageNeedsChecksumCopy(Page page);
#endif /* BUFPAGE_H */
diff --git a/src/backend/postmaster/bgwriter.c b/src/backend/postmaster/bgwriter.c
index 72f5acceec7..6e8801a39e3 100644
--- a/src/backend/postmaster/bgwriter.c
+++ b/src/backend/postmaster/bgwriter.c
@@ -38,11 +38,13 @@
#include "postmaster/auxprocess.h"
#include "postmaster/bgwriter.h"
#include "postmaster/interrupt.h"
+#include "storage/aio.h"
#include "storage/aio_subsys.h"
#include "storage/buf_internals.h"
#include "storage/bufmgr.h"
#include "storage/condition_variable.h"
#include "storage/fd.h"
+#include "storage/io_queue.h"
#include "storage/lwlock.h"
#include "storage/proc.h"
#include "storage/procsignal.h"
@@ -90,6 +92,7 @@ BackgroundWriterMain(const void *startup_data, size_t startup_data_len)
sigjmp_buf local_sigjmp_buf;
MemoryContext bgwriter_context;
bool prev_hibernate;
+ IOQueue *ioq;
WritebackContext wb_context;
Assert(startup_data_len == 0);
@@ -131,6 +134,7 @@ BackgroundWriterMain(const void *startup_data, size_t startup_data_len)
ALLOCSET_DEFAULT_SIZES);
MemoryContextSwitchTo(bgwriter_context);
+ ioq = io_queue_create(128, 0);
WritebackContextInit(&wb_context, &bgwriter_flush_after);
/*
@@ -228,12 +232,22 @@ BackgroundWriterMain(const void *startup_data, size_t startup_data_len)
/* Clear any already-pending wakeups */
ResetLatch(MyLatch);
+ /*
+ * FIXME: this is theoretically racy, but I didn't want to copy
+ * ProcessMainLoopInterrupts() remaining body here.
+ */
+ if (ShutdownRequestPending)
+ {
+ io_queue_wait_all(ioq);
+ io_queue_free(ioq);
+ }
+
ProcessMainLoopInterrupts();
/*
* Do one cycle of dirty-buffer writing.
*/
- can_hibernate = BgBufferSync(&wb_context);
+ can_hibernate = BgBufferSync(ioq, &wb_context);
/* Report pending statistics to the cumulative stats system */
pgstat_report_bgwriter();
@@ -250,6 +264,9 @@ BackgroundWriterMain(const void *startup_data, size_t startup_data_len)
smgrdestroyall();
}
+ /* finish IO before sleeping, to avoid blocking other backends */
+ io_queue_wait_all(ioq);
+
/*
* Log a new xl_running_xacts every now and then so replication can
* get into a consistent state faster (think of suboverflowed
diff --git a/src/backend/postmaster/checkpointer.c b/src/backend/postmaster/checkpointer.c
index fda91ffd1ce..904fe167eb4 100644
--- a/src/backend/postmaster/checkpointer.c
+++ b/src/backend/postmaster/checkpointer.c
@@ -49,10 +49,12 @@
#include "postmaster/bgwriter.h"
#include "postmaster/interrupt.h"
#include "replication/syncrep.h"
+#include "storage/aio.h"
#include "storage/aio_subsys.h"
#include "storage/bufmgr.h"
#include "storage/condition_variable.h"
#include "storage/fd.h"
+#include "storage/io_queue.h"
#include "storage/ipc.h"
#include "storage/lwlock.h"
#include "storage/pmsignal.h"
@@ -766,7 +768,7 @@ ImmediateCheckpointRequested(void)
* fraction between 0.0 meaning none, and 1.0 meaning all done.
*/
void
-CheckpointWriteDelay(int flags, double progress)
+CheckpointWriteDelay(IOQueue *ioq, int flags, double progress)
{
static int absorb_counter = WRITES_PER_ABSORB;
@@ -800,6 +802,13 @@ CheckpointWriteDelay(int flags, double progress)
/* Report interim statistics to the cumulative stats system */
pgstat_report_checkpointer();
+ /*
+ * Ensure all pending IO is submitted to avoid unnecessary delays for
+ * other processes.
+ */
+ io_queue_wait_all(ioq);
+
+
/*
* This sleep used to be connected to bgwriter_delay, typically 200ms.
* That resulted in more frequent wakeups if not much work to do.
diff --git a/src/backend/storage/buffer/bufmgr.c b/src/backend/storage/buffer/bufmgr.c
index 886fbaf63f3..3312b713c8c 100644
--- a/src/backend/storage/buffer/bufmgr.c
+++ b/src/backend/storage/buffer/bufmgr.c
@@ -52,6 +52,7 @@
#include "storage/buf_internals.h"
#include "storage/bufmgr.h"
#include "storage/fd.h"
+#include "storage/io_queue.h"
#include "storage/ipc.h"
#include "storage/lmgr.h"
#include "storage/proc.h"
@@ -76,6 +77,7 @@
/* Bits in SyncOneBuffer's return value */
#define BUF_WRITTEN 0x01
#define BUF_REUSABLE 0x02
+#define BUF_CANT_MERGE 0x04
#define RELS_BSEARCH_THRESHOLD 20
@@ -512,8 +514,6 @@ static void UnpinBuffer(BufferDesc *buf);
static void UnpinBufferNoOwner(BufferDesc *buf);
static void BufferSync(int flags);
static uint32 WaitBufHdrUnlocked(BufferDesc *buf);
-static int SyncOneBuffer(int buf_id, bool skip_recently_used,
- WritebackContext *wb_context);
static void WaitIO(BufferDesc *buf);
static void AbortBufferIO(Buffer buffer);
static void shared_buffer_write_error_callback(void *arg);
@@ -529,6 +529,7 @@ static void CheckReadBuffersOperation(ReadBuffersOperation *operation, bool is_c
static Buffer GetVictimBuffer(BufferAccessStrategy strategy, IOContext io_context);
static void FlushBuffer(BufferDesc *buf, SMgrRelation reln,
IOObject io_object, IOContext io_context);
+
static void FindAndDropRelationBuffers(RelFileLocator rlocator,
ForkNumber forkNum,
BlockNumber nForkBlock,
@@ -3299,6 +3300,57 @@ UnpinBufferNoOwner(BufferDesc *buf)
}
}
+typedef struct BuffersToWrite
+{
+ int nbuffers;
+ BufferTag start_at_tag;
+ uint32 max_combine;
+
+ XLogRecPtr max_lsn;
+
+ PgAioHandle *ioh;
+ PgAioWaitRef iow;
+
+ uint64 total_writes;
+
+ Buffer buffers[IOV_MAX];
+ PgAioBounceBuffer *bounce_buffers[IOV_MAX];
+ const void *data_ptrs[IOV_MAX];
+} BuffersToWrite;
+
+static int PrepareToWriteBuffer(BuffersToWrite *to_write, Buffer buf,
+ bool skip_recently_used,
+ IOQueue *ioq, WritebackContext *wb_context);
+
+static void WriteBuffers(BuffersToWrite *to_write,
+ IOQueue *ioq, WritebackContext *wb_context);
+
+static void
+BuffersToWriteInit(BuffersToWrite *to_write,
+ IOQueue *ioq, WritebackContext *wb_context)
+{
+ to_write->total_writes = 0;
+ to_write->nbuffers = 0;
+ to_write->ioh = NULL;
+ pgaio_wref_clear(&to_write->iow);
+ to_write->max_lsn = InvalidXLogRecPtr;
+
+ pgaio_enter_batchmode();
+}
+
+static void
+BuffersToWriteEnd(BuffersToWrite *to_write)
+{
+ if (to_write->ioh != NULL)
+ {
+ pgaio_io_release(to_write->ioh);
+ to_write->ioh = NULL;
+ }
+
+ pgaio_exit_batchmode();
+}
+
+
#define ST_SORT sort_checkpoint_bufferids
#define ST_ELEMENT_TYPE CkptSortItem
#define ST_COMPARE(a, b) ckpt_buforder_comparator(a, b)
@@ -3330,7 +3382,10 @@ BufferSync(int flags)
binaryheap *ts_heap;
int i;
int mask = BM_DIRTY;
+ IOQueue *ioq;
WritebackContext wb_context;
+ BuffersToWrite to_write;
+ int max_combine;
/*
* Unless this is a shutdown checkpoint or we have been explicitly told,
@@ -3392,7 +3447,9 @@ BufferSync(int flags)
if (num_to_scan == 0)
return; /* nothing to do */
+ ioq = io_queue_create(512, 0);
WritebackContextInit(&wb_context, &checkpoint_flush_after);
+ max_combine = Min(io_bounce_buffers, io_combine_limit);
TRACE_POSTGRESQL_BUFFER_SYNC_START(NBuffers, num_to_scan);
@@ -3500,48 +3557,91 @@ BufferSync(int flags)
*/
num_processed = 0;
num_written = 0;
+
+ BuffersToWriteInit(&to_write, ioq, &wb_context);
+
while (!binaryheap_empty(ts_heap))
{
BufferDesc *bufHdr = NULL;
CkptTsStatus *ts_stat = (CkptTsStatus *)
DatumGetPointer(binaryheap_first(ts_heap));
+ bool batch_continue = true;
- buf_id = CkptBufferIds[ts_stat->index].buf_id;
- Assert(buf_id != -1);
-
- bufHdr = GetBufferDescriptor(buf_id);
-
- num_processed++;
+ Assert(ts_stat->num_scanned <= ts_stat->num_to_scan);
/*
- * We don't need to acquire the lock here, because we're only looking
- * at a single bit. It's possible that someone else writes the buffer
- * and clears the flag right after we check, but that doesn't matter
- * since SyncOneBuffer will then do nothing. However, there is a
- * further race condition: it's conceivable that between the time we
- * examine the bit here and the time SyncOneBuffer acquires the lock,
- * someone else not only wrote the buffer but replaced it with another
- * page and dirtied it. In that improbable case, SyncOneBuffer will
- * write the buffer though we didn't need to. It doesn't seem worth
- * guarding against this, though.
+ * Collect a batch of buffers to write out from the current
+ * tablespace. That causes some imbalance between the tablespaces, but
+ * that's more than outweighed by the efficiency gain due to batching.
*/
- if (pg_atomic_read_u32(&bufHdr->state) & BM_CHECKPOINT_NEEDED)
+ while (batch_continue &&
+ to_write.nbuffers < max_combine &&
+ ts_stat->num_scanned < ts_stat->num_to_scan)
{
- if (SyncOneBuffer(buf_id, false, &wb_context) & BUF_WRITTEN)
+ buf_id = CkptBufferIds[ts_stat->index].buf_id;
+ Assert(buf_id != -1);
+
+ bufHdr = GetBufferDescriptor(buf_id);
+
+ num_processed++;
+
+ /*
+ * We don't need to acquire the lock here, because we're only
+ * looking at a single bit. It's possible that someone else writes
+ * the buffer and clears the flag right after we check, but that
+ * doesn't matter since SyncOneBuffer will then do nothing.
+ * However, there is a further race condition: it's conceivable
+ * that between the time we examine the bit here and the time
+ * SyncOneBuffer acquires the lock, someone else not only wrote
+ * the buffer but replaced it with another page and dirtied it. In
+ * that improbable case, SyncOneBuffer will write the buffer
+ * though we didn't need to. It doesn't seem worth guarding
+ * against this, though.
+ */
+ if (pg_atomic_read_u32(&bufHdr->state) & BM_CHECKPOINT_NEEDED)
{
- TRACE_POSTGRESQL_BUFFER_SYNC_WRITTEN(buf_id);
- PendingCheckpointerStats.buffers_written++;
- num_written++;
+ int result = PrepareToWriteBuffer(&to_write, buf_id + 1, false,
+ ioq, &wb_context);
+
+ if (result & BUF_CANT_MERGE)
+ {
+ Assert(to_write.nbuffers > 0);
+ WriteBuffers(&to_write, ioq, &wb_context);
+
+ result = PrepareToWriteBuffer(&to_write, buf_id + 1, false,
+ ioq, &wb_context);
+ Assert(result != BUF_CANT_MERGE);
+ }
+
+ if (result & BUF_WRITTEN)
+ {
+ TRACE_POSTGRESQL_BUFFER_SYNC_WRITTEN(buf_id);
+ PendingCheckpointerStats.buffers_written++;
+ num_written++;
+ }
+ else
+ {
+ batch_continue = false;
+ }
}
+ else
+ {
+ if (to_write.nbuffers > 0)
+ WriteBuffers(&to_write, ioq, &wb_context);
+ }
+
+ /*
+ * Measure progress independent of actually having to flush the
+ * buffer - otherwise writing become unbalanced.
+ */
+ ts_stat->progress += ts_stat->progress_slice;
+ ts_stat->num_scanned++;
+ ts_stat->index++;
}
- /*
- * Measure progress independent of actually having to flush the buffer
- * - otherwise writing become unbalanced.
- */
- ts_stat->progress += ts_stat->progress_slice;
- ts_stat->num_scanned++;
- ts_stat->index++;
+ if (to_write.nbuffers > 0)
+ WriteBuffers(&to_write, ioq, &wb_context);
+
/* Have all the buffers from the tablespace been processed? */
if (ts_stat->num_scanned == ts_stat->num_to_scan)
@@ -3559,15 +3659,23 @@ BufferSync(int flags)
*
* (This will check for barrier events even if it doesn't sleep.)
*/
- CheckpointWriteDelay(flags, (double) num_processed / num_to_scan);
+ CheckpointWriteDelay(ioq, flags, (double) num_processed / num_to_scan);
}
+ Assert(to_write.nbuffers == 0);
+ io_queue_wait_all(ioq);
+
/*
* Issue all pending flushes. Only checkpointer calls BufferSync(), so
* IOContext will always be IOCONTEXT_NORMAL.
*/
IssuePendingWritebacks(&wb_context, IOCONTEXT_NORMAL);
+ io_queue_wait_all(ioq); /* IssuePendingWritebacks might have added
+ * more */
+ io_queue_free(ioq);
+ BuffersToWriteEnd(&to_write);
+
pfree(per_ts_stat);
per_ts_stat = NULL;
binaryheap_free(ts_heap);
@@ -3593,7 +3701,7 @@ BufferSync(int flags)
* bgwriter_lru_maxpages to 0.)
*/
bool
-BgBufferSync(WritebackContext *wb_context)
+BgBufferSync(IOQueue *ioq, WritebackContext *wb_context)
{
/* info obtained from freelist.c */
int strategy_buf_id;
@@ -3636,6 +3744,9 @@ BgBufferSync(WritebackContext *wb_context)
long new_strategy_delta;
uint32 new_recent_alloc;
+ BuffersToWrite to_write;
+ int max_combine;
+
/*
* Find out where the freelist clock sweep currently is, and how many
* buffer allocations have happened since our last call.
@@ -3656,6 +3767,8 @@ BgBufferSync(WritebackContext *wb_context)
return true;
}
+ max_combine = Min(io_bounce_buffers, io_combine_limit);
+
/*
* Compute strategy_delta = how many buffers have been scanned by the
* clock sweep since last time. If first time through, assume none. Then
@@ -3812,11 +3925,25 @@ BgBufferSync(WritebackContext *wb_context)
num_written = 0;
reusable_buffers = reusable_buffers_est;
+ BuffersToWriteInit(&to_write, ioq, wb_context);
+
/* Execute the LRU scan */
while (num_to_scan > 0 && reusable_buffers < upcoming_alloc_est)
{
- int sync_state = SyncOneBuffer(next_to_clean, true,
- wb_context);
+ int sync_state;
+
+ sync_state = PrepareToWriteBuffer(&to_write, next_to_clean + 1,
+ true, ioq, wb_context);
+ if (sync_state & BUF_CANT_MERGE)
+ {
+ Assert(to_write.nbuffers > 0);
+
+ WriteBuffers(&to_write, ioq, wb_context);
+
+ sync_state = PrepareToWriteBuffer(&to_write, next_to_clean + 1,
+ true, ioq, wb_context);
+ Assert(sync_state != BUF_CANT_MERGE);
+ }
if (++next_to_clean >= NBuffers)
{
@@ -3827,6 +3954,13 @@ BgBufferSync(WritebackContext *wb_context)
if (sync_state & BUF_WRITTEN)
{
+ Assert(sync_state & BUF_REUSABLE);
+
+ if (to_write.nbuffers == max_combine)
+ {
+ WriteBuffers(&to_write, ioq, wb_context);
+ }
+
reusable_buffers++;
if (++num_written >= bgwriter_lru_maxpages)
{
@@ -3838,6 +3972,11 @@ BgBufferSync(WritebackContext *wb_context)
reusable_buffers++;
}
+ if (to_write.nbuffers > 0)
+ WriteBuffers(&to_write, ioq, wb_context);
+
+ BuffersToWriteEnd(&to_write);
+
PendingBgWriterStats.buf_written_clean += num_written;
#ifdef BGW_DEBUG
@@ -3876,8 +4015,66 @@ BgBufferSync(WritebackContext *wb_context)
return (bufs_to_lap == 0 && recent_alloc == 0);
}
+static inline bool
+BufferTagsSameRel(const BufferTag *tag1, const BufferTag *tag2)
+{
+ return (tag1->spcOid == tag2->spcOid) &&
+ (tag1->dbOid == tag2->dbOid) &&
+ (tag1->relNumber == tag2->relNumber) &&
+ (tag1->forkNum == tag2->forkNum)
+ ;
+}
+
+static bool
+CanMergeWrite(BuffersToWrite *to_write, BufferDesc *cur_buf_hdr)
+{
+ BlockNumber cur_block = cur_buf_hdr->tag.blockNum;
+
+ Assert(to_write->nbuffers > 0); /* can't merge with nothing */
+ Assert(to_write->start_at_tag.relNumber != InvalidOid);
+ Assert(to_write->start_at_tag.blockNum != InvalidBlockNumber);
+
+ Assert(to_write->ioh != NULL);
+
+ /*
+ * First check if the blocknumber is one that we could actually merge,
+ * that's cheaper than checking the tablespace/db/relnumber/fork match.
+ */
+ if (to_write->start_at_tag.blockNum + to_write->nbuffers != cur_block)
+ return false;
+
+ if (!BufferTagsSameRel(&to_write->start_at_tag, &cur_buf_hdr->tag))
+ return false;
+
+ /*
+ * Need to check with smgr how large a write we're allowed to make. To
+ * reduce the overhead of the smgr check, only inquire once, when
+ * processing the first to-be-merged buffer. That avoids the overhead in
+ * the common case of writing out buffers that definitely not mergeable.
+ */
+ if (to_write->nbuffers == 1)
+ {
+ SMgrRelation smgr;
+
+ smgr = smgropen(BufTagGetRelFileLocator(&to_write->start_at_tag), INVALID_PROC_NUMBER);
+
+ to_write->max_combine = smgrmaxcombine(smgr,
+ to_write->start_at_tag.forkNum,
+ to_write->start_at_tag.blockNum);
+ }
+ else
+ {
+ Assert(to_write->max_combine > 0);
+ }
+
+ if (to_write->start_at_tag.blockNum + to_write->max_combine <= cur_block)
+ return false;
+
+ return true;
+}
+
/*
- * SyncOneBuffer -- process a single buffer during syncing.
+ * PrepareToWriteBuffer -- process a single buffer during syncing.
*
* If skip_recently_used is true, we don't write currently-pinned buffers, nor
* buffers marked recently used, as these are not replacement candidates.
@@ -3886,22 +4083,50 @@ BgBufferSync(WritebackContext *wb_context)
* BUF_WRITTEN: we wrote the buffer.
* BUF_REUSABLE: buffer is available for replacement, ie, it has
* pin count 0 and usage count 0.
+ * BUF_CANT_MERGE: can't combine this write with prior writes, caller needs
+ * to issue those first
*
* (BUF_WRITTEN could be set in error if FlushBuffer finds the buffer clean
* after locking it, but we don't care all that much.)
*/
static int
-SyncOneBuffer(int buf_id, bool skip_recently_used, WritebackContext *wb_context)
+PrepareToWriteBuffer(BuffersToWrite *to_write, Buffer buf,
+ bool skip_recently_used,
+ IOQueue *ioq, WritebackContext *wb_context)
{
- BufferDesc *bufHdr = GetBufferDescriptor(buf_id);
+ BufferDesc *cur_buf_hdr = GetBufferDescriptor(buf - 1);
+ uint32 buf_state;
int result = 0;
- uint32 buf_state;
- BufferTag tag;
+ XLogRecPtr cur_buf_lsn;
+ LWLock *content_lock;
+ bool may_block;
+
+ /*
+ * Check if this buffer can be written out together with already prepared
+ * writes. We check before we have pinned the buffer, so the buffer can be
+ * written out and replaced between this check and us pinning the buffer -
+ * we'll recheck below. The reason for the pre-check is that we don't want
+ * to pin the buffer just to find out that we can't merge the IO.
+ */
+ if (to_write->nbuffers != 0)
+ {
+ if (!CanMergeWrite(to_write, cur_buf_hdr))
+ {
+ result |= BUF_CANT_MERGE;
+ return result;
+ }
+ }
+ else
+ {
+ to_write->start_at_tag = cur_buf_hdr->tag;
+ }
/* Make sure we can handle the pin */
ReservePrivateRefCountEntry();
ResourceOwnerEnlarge(CurrentResourceOwner);
+ /* XXX: Should also check if we are allowed to pin one more buffer */
+
/*
* Check whether buffer needs writing.
*
@@ -3911,7 +4136,7 @@ SyncOneBuffer(int buf_id, bool skip_recently_used, WritebackContext *wb_context)
* don't worry because our checkpoint.redo points before log record for
* upcoming changes and so we are not required to write such dirty buffer.
*/
- buf_state = LockBufHdr(bufHdr);
+ buf_state = LockBufHdr(cur_buf_hdr);
if (BUF_STATE_GET_REFCOUNT(buf_state) == 0 &&
BUF_STATE_GET_USAGECOUNT(buf_state) == 0)
@@ -3920,40 +4145,294 @@ SyncOneBuffer(int buf_id, bool skip_recently_used, WritebackContext *wb_context)
}
else if (skip_recently_used)
{
+#if 0
+ elog(LOG, "at block %d: skip recent with nbuffers %d",
+ cur_buf_hdr->tag.blockNum, to_write->nbuffers);
+#endif
/* Caller told us not to write recently-used buffers */
- UnlockBufHdr(bufHdr, buf_state);
+ UnlockBufHdr(cur_buf_hdr, buf_state);
return result;
}
if (!(buf_state & BM_VALID) || !(buf_state & BM_DIRTY))
{
/* It's clean, so nothing to do */
- UnlockBufHdr(bufHdr, buf_state);
+ UnlockBufHdr(cur_buf_hdr, buf_state);
return result;
}
+ /* pin the buffer, from now on its identity can't change anymore */
+ PinBuffer_Locked(cur_buf_hdr);
+
+ /*
+ * Acquire IO, if needed, now that it's likely that we'll need to write.
+ */
+ if (to_write->ioh == NULL)
+ {
+ /* otherwise we should already have acquired a handle */
+ Assert(to_write->nbuffers == 0);
+
+ to_write->ioh = io_queue_acquire_io(ioq);
+ pgaio_io_get_wref(to_write->ioh, &to_write->iow);
+ }
+
/*
- * Pin it, share-lock it, write it. (FlushBuffer will do nothing if the
- * buffer is clean by the time we've locked it.)
+ * If we are merging, check if the buffer's identity possibly changed
+ * while we hadn't yet pinned it.
+ *
+ * XXX: It might be worth checking if we still want to write the buffer
+ * out, e.g. it could have been replaced with a buffer that doesn't have
+ * BM_CHECKPOINT_NEEDED set.
*/
- PinBuffer_Locked(bufHdr);
- LWLockAcquire(BufferDescriptorGetContentLock(bufHdr), LW_SHARED);
+ if (to_write->nbuffers != 0)
+ {
+ if (!CanMergeWrite(to_write, cur_buf_hdr))
+ {
+ elog(LOG, "changed identity");
+ UnpinBuffer(cur_buf_hdr);
+
+ result |= BUF_CANT_MERGE;
+
+ return result;
+ }
+ }
+
+ may_block = to_write->nbuffers == 0
+ && !pgaio_have_staged()
+ && io_queue_is_empty(ioq)
+ ;
+ content_lock = BufferDescriptorGetContentLock(cur_buf_hdr);
+
+ if (!may_block)
+ {
+ if (LWLockConditionalAcquire(content_lock, LW_SHARED))
+ {
+ /* done */
+ }
+ else if (to_write->nbuffers == 0)
+ {
+ /*
+ * Need to wait for all prior IO to finish before blocking for
+ * lock acquisition, to avoid the risk a deadlock due to us
+ * waiting for another backend that is waiting for our unsubmitted
+ * IO to complete.
+ */
+ pgaio_submit_staged();
+ io_queue_wait_all(ioq);
+
+ elog(DEBUG2, "at block %u: can't block, nbuffers = 0",
+ cur_buf_hdr->tag.blockNum
+ );
+
+ may_block = to_write->nbuffers == 0
+ && !pgaio_have_staged()
+ && io_queue_is_empty(ioq)
+ ;
+ Assert(may_block);
+
+ LWLockAcquire(content_lock, LW_SHARED);
+ }
+ else
+ {
+ elog(DEBUG2, "at block %d: can't block nbuffers = %d",
+ cur_buf_hdr->tag.blockNum,
+ to_write->nbuffers);
- FlushBuffer(bufHdr, NULL, IOOBJECT_RELATION, IOCONTEXT_NORMAL);
+ UnpinBuffer(cur_buf_hdr);
+ result |= BUF_CANT_MERGE;
+ Assert(to_write->nbuffers > 0);
- LWLockRelease(BufferDescriptorGetContentLock(bufHdr));
+ return result;
+ }
+ }
+ else
+ {
+ LWLockAcquire(content_lock, LW_SHARED);
+ }
- tag = bufHdr->tag;
+ if (!may_block)
+ {
+ if (!StartBufferIO(cur_buf_hdr, false, !may_block))
+ {
+ pgaio_submit_staged();
+ io_queue_wait_all(ioq);
- UnpinBuffer(bufHdr);
+ may_block = io_queue_is_empty(ioq) && to_write->nbuffers == 0 && !pgaio_have_staged();
+
+ if (!StartBufferIO(cur_buf_hdr, false, !may_block))
+ {
+ elog(DEBUG2, "at block %d: non-waitable StartBufferIO returns false, %d",
+ cur_buf_hdr->tag.blockNum,
+ may_block);
+
+ /*
+ * FIXME: can't tell whether this is because the buffer has
+ * been cleaned
+ */
+ if (!may_block)
+ {
+ result |= BUF_CANT_MERGE;
+ Assert(to_write->nbuffers > 0);
+ }
+ LWLockRelease(content_lock);
+ UnpinBuffer(cur_buf_hdr);
+
+ return result;
+ }
+ }
+ }
+ else
+ {
+ if (!StartBufferIO(cur_buf_hdr, false, false))
+ {
+ elog(DEBUG2, "waitable StartBufferIO returns false");
+ LWLockRelease(content_lock);
+ UnpinBuffer(cur_buf_hdr);
+
+ /*
+ * FIXME: Historically we returned BUF_WRITTEN in this case, which
+ * seems wrong
+ */
+ return result;
+ }
+ }
/*
- * SyncOneBuffer() is only called by checkpointer and bgwriter, so
- * IOContext will always be IOCONTEXT_NORMAL.
+ * Run PageGetLSN while holding header lock, since we don't have the
+ * buffer locked exclusively in all cases.
*/
- ScheduleBufferTagForWriteback(wb_context, IOCONTEXT_NORMAL, &tag);
+ buf_state = LockBufHdr(cur_buf_hdr);
+
+ cur_buf_lsn = BufferGetLSN(cur_buf_hdr);
+
+ /* To check if block content changes while flushing. - vadim 01/17/97 */
+ buf_state &= ~BM_JUST_DIRTIED;
+
+ UnlockBufHdr(cur_buf_hdr, buf_state);
+
+ to_write->buffers[to_write->nbuffers] = buf;
+ to_write->nbuffers++;
+
+ if (buf_state & BM_PERMANENT &&
+ (to_write->max_lsn == InvalidXLogRecPtr || to_write->max_lsn < cur_buf_lsn))
+ {
+ to_write->max_lsn = cur_buf_lsn;
+ }
+
+ result |= BUF_WRITTEN;
+
+ return result;
+}
+
+static void
+WriteBuffers(BuffersToWrite *to_write,
+ IOQueue *ioq, WritebackContext *wb_context)
+{
+ SMgrRelation smgr;
+ Buffer first_buf;
+ BufferDesc *first_buf_hdr;
+ bool needs_checksum;
+
+ Assert(to_write->nbuffers > 0 && to_write->nbuffers <= io_combine_limit);
+
+ first_buf = to_write->buffers[0];
+ first_buf_hdr = GetBufferDescriptor(first_buf - 1);
+
+ smgr = smgropen(BufTagGetRelFileLocator(&first_buf_hdr->tag), INVALID_PROC_NUMBER);
+
+ /*
+ * Force XLOG flush up to buffer's LSN. This implements the basic WAL
+ * rule that log updates must hit disk before any of the data-file changes
+ * they describe do.
+ *
+ * However, this rule does not apply to unlogged relations, which will be
+ * lost after a crash anyway. Most unlogged relation pages do not bear
+ * LSNs since we never emit WAL records for them, and therefore flushing
+ * up through the buffer LSN would be useless, but harmless. However,
+ * GiST indexes use LSNs internally to track page-splits, and therefore
+ * unlogged GiST pages bear "fake" LSNs generated by
+ * GetFakeLSNForUnloggedRel. It is unlikely but possible that the fake
+ * LSN counter could advance past the WAL insertion point; and if it did
+ * happen, attempting to flush WAL through that location would fail, with
+ * disastrous system-wide consequences. To make sure that can't happen,
+ * skip the flush if the buffer isn't permanent.
+ */
+ if (to_write->max_lsn != InvalidXLogRecPtr)
+ XLogFlush(to_write->max_lsn);
+
+ /*
+ * Now it's safe to write buffer to disk. Note that no one else should
+ * have been able to write it while we were busy with log flushing because
+ * only one process at a time can set the BM_IO_IN_PROGRESS bit.
+ */
+
+ for (int nbuf = 0; nbuf < to_write->nbuffers; nbuf++)
+ {
+ Buffer cur_buf = to_write->buffers[nbuf];
+ BufferDesc *cur_buf_hdr = GetBufferDescriptor(cur_buf - 1);
+ Block bufBlock;
+ char *bufToWrite;
+
+ bufBlock = BufHdrGetBlock(cur_buf_hdr);
+ needs_checksum = PageNeedsChecksumCopy((Page) bufBlock);
+
+ /*
+ * Update page checksum if desired. Since we have only shared lock on
+ * the buffer, other processes might be updating hint bits in it, so
+ * we must copy the page to a bounce buffer if we do checksumming.
+ */
+ if (needs_checksum)
+ {
+ PgAioBounceBuffer *bb = pgaio_bounce_buffer_get();
+
+ pgaio_io_assoc_bounce_buffer(to_write->ioh, bb);
+
+ bufToWrite = pgaio_bounce_buffer_buffer(bb);
+ memcpy(bufToWrite, bufBlock, BLCKSZ);
+ PageSetChecksumInplace((Page) bufToWrite, cur_buf_hdr->tag.blockNum);
+ }
+ else
+ {
+ bufToWrite = bufBlock;
+ }
+
+ to_write->data_ptrs[nbuf] = bufToWrite;
+ }
+
+ pgaio_io_set_handle_data_32(to_write->ioh,
+ (uint32 *) to_write->buffers,
+ to_write->nbuffers);
+ pgaio_io_register_callbacks(to_write->ioh, PGAIO_HCB_SHARED_BUFFER_WRITEV, 0);
+
+ smgrstartwritev(to_write->ioh, smgr,
+ BufTagGetForkNum(&first_buf_hdr->tag),
+ first_buf_hdr->tag.blockNum,
+ to_write->data_ptrs,
+ to_write->nbuffers,
+ false);
+ pgstat_count_io_op(IOOBJECT_RELATION, IOCONTEXT_NORMAL,
+ IOOP_WRITE, 1, BLCKSZ * to_write->nbuffers);
+
+
+ for (int nbuf = 0; nbuf < to_write->nbuffers; nbuf++)
+ {
+ Buffer cur_buf = to_write->buffers[nbuf];
+ BufferDesc *cur_buf_hdr = GetBufferDescriptor(cur_buf - 1);
+
+ UnpinBuffer(cur_buf_hdr);
+ }
+
+ io_queue_track(ioq, &to_write->iow);
+ to_write->total_writes++;
- return result | BUF_WRITTEN;
+ /* clear state for next write */
+ to_write->nbuffers = 0;
+ to_write->start_at_tag.relNumber = InvalidOid;
+ to_write->start_at_tag.blockNum = InvalidBlockNumber;
+ to_write->max_combine = 0;
+ to_write->max_lsn = InvalidXLogRecPtr;
+ to_write->ioh = NULL;
+ pgaio_wref_clear(&to_write->iow);
}
/*
@@ -4326,6 +4805,7 @@ FlushBuffer(BufferDesc *buf, SMgrRelation reln, IOObject io_object,
error_context_stack = errcallback.previous;
}
+
/*
* RelationGetNumberOfBlocksInFork
* Determines the current number of pages in the specified relation fork.
diff --git a/src/backend/storage/page/bufpage.c b/src/backend/storage/page/bufpage.c
index ecc81aacfc3..71dc6f559dd 100644
--- a/src/backend/storage/page/bufpage.c
+++ b/src/backend/storage/page/bufpage.c
@@ -1480,6 +1480,16 @@ PageIndexTupleOverwrite(Page page, OffsetNumber offnum,
return true;
}
+bool
+PageNeedsChecksumCopy(Page page)
+{
+ if (PageIsNew(page))
+ return false;
+
+ /* If we don't need a checksum, just return the passed-in data */
+ return DataChecksumsEnabled();
+}
+
/*
* Set checksum for a page in shared buffers.
diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list
index 26c622824e4..26a9c1d4ba6 100644
--- a/src/tools/pgindent/typedefs.list
+++ b/src/tools/pgindent/typedefs.list
@@ -348,6 +348,7 @@ BufferManagerRelation
BufferStrategyControl
BufferTag
BufferUsage
+BuffersToWrite
BuildAccumulator
BuiltinScript
BulkInsertState
--
2.48.1.76.g4e746b1a31.dirty
[text/x-diff] v2.8-0034-Ensure-a-resowner-exists-for-all-paths-that-may.patch (2.6K, ../../3yxd5r23zly5bytvgyktbxtxq2r3gbpi7xd4dugevh3h4w4q6c@lu6oatjjpltz/35-v2.8-0034-Ensure-a-resowner-exists-for-all-paths-that-may.patch)
download | inline diff:
From ee17a601fd0903d54b7858dd3e81a03db6b1d003 Mon Sep 17 00:00:00 2001
From: Andres Freund <andres@anarazel.de>
Date: Tue, 8 Oct 2024 14:34:38 -0400
Subject: [PATCH v2.8 34/38] Ensure a resowner exists for all paths that may
perform AIO
Reviewed-by: Noah Misch <noah@leadboat.com>
Discussion: https://postgr.es/m/1f6b50a7-38ef-4d87-8246-786d39f46ab9@iki.fi
---
src/backend/bootstrap/bootstrap.c | 7 +++++++
src/backend/replication/logical/logical.c | 6 ++++++
src/backend/utils/init/postinit.c | 6 +++++-
3 files changed, 18 insertions(+), 1 deletion(-)
diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c
index 6db864892d0..e554504e1f0 100644
--- a/src/backend/bootstrap/bootstrap.c
+++ b/src/backend/bootstrap/bootstrap.c
@@ -361,8 +361,15 @@ BootstrapModeMain(int argc, char *argv[], bool check_only)
BaseInit();
bootstrap_signals();
+
+ /* need a resowner for IO during BootStrapXLOG() */
+ CreateAuxProcessResourceOwner();
+
BootStrapXLOG(bootstrap_data_checksum_version);
+ ReleaseAuxProcessResources(true);
+ CurrentResourceOwner = NULL;
+
/*
* To ensure that src/common/link-canary.c is linked into the backend, we
* must call it from somewhere. Here is as good as anywhere.
diff --git a/src/backend/replication/logical/logical.c b/src/backend/replication/logical/logical.c
index 8ea846bfc3b..2b936714040 100644
--- a/src/backend/replication/logical/logical.c
+++ b/src/backend/replication/logical/logical.c
@@ -386,6 +386,12 @@ CreateInitDecodingContext(const char *plugin,
slot->data.plugin = plugin_name;
SpinLockRelease(&slot->mutex);
+ if (CurrentResourceOwner == NULL)
+ {
+ Assert(am_walsender);
+ CurrentResourceOwner = AuxProcessResourceOwner;
+ }
+
if (XLogRecPtrIsInvalid(restart_lsn))
ReplicationSlotReserveWal();
else
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index 7958ea11b73..222e24bcb08 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -785,8 +785,12 @@ InitPostgres(const char *in_dbname, Oid dboid,
* We don't yet have an aux-process resource owner, but StartupXLOG
* and ShutdownXLOG will need one. Hence, create said resource owner
* (and register a callback to clean it up after ShutdownXLOG runs).
+ *
+ * In bootstrap mode CreateAuxProcessResourceOwner() was already
+ * called in BootstrapModeMain().
*/
- CreateAuxProcessResourceOwner();
+ if (!bootstrap)
+ CreateAuxProcessResourceOwner();
StartupXLOG();
/* Release (and warn about) any buffer pins leaked in StartupXLOG */
--
2.48.1.76.g4e746b1a31.dirty
[text/x-diff] v2.8-0035-Temporary-Increase-BAS_BULKREAD-size.patch (1.3K, ../../3yxd5r23zly5bytvgyktbxtxq2r3gbpi7xd4dugevh3h4w4q6c@lu6oatjjpltz/36-v2.8-0035-Temporary-Increase-BAS_BULKREAD-size.patch)
download | inline diff:
From ee763e18fabaa01879c6aa444283ced36d3c34a1 Mon Sep 17 00:00:00 2001
From: Andres Freund <andres@anarazel.de>
Date: Sun, 1 Sep 2024 00:42:27 -0400
Subject: [PATCH v2.8 35/38] Temporary: Increase BAS_BULKREAD size
Without this we only can execute very little AIO for sequential scans, as
there's just not enough buffers in the ring. This isn't the right fix, as
just increasing the ring size can have negative performance implications in
workloads where the kernel has all the data cached.
Author:
Reviewed-By:
Discussion: https://postgr.es/m/
Backpatch:
---
src/backend/storage/buffer/freelist.c | 7 ++++++-
1 file changed, 6 insertions(+), 1 deletion(-)
diff --git a/src/backend/storage/buffer/freelist.c b/src/backend/storage/buffer/freelist.c
index 336715b6c63..b72a5957a20 100644
--- a/src/backend/storage/buffer/freelist.c
+++ b/src/backend/storage/buffer/freelist.c
@@ -555,7 +555,12 @@ GetAccessStrategy(BufferAccessStrategyType btype)
return NULL;
case BAS_BULKREAD:
- ring_size_kb = 256;
+
+ /*
+ * FIXME: Temporary increase to allow large enough streaming reads
+ * to actually benefit from AIO. This needs a better solution.
+ */
+ ring_size_kb = 2 * 1024;
break;
case BAS_BULKWRITE:
ring_size_kb = 16 * 1024;
--
2.48.1.76.g4e746b1a31.dirty
[text/x-diff] v2.8-0036-WIP-Use-MAP_POPULATE.patch (1.1K, ../../3yxd5r23zly5bytvgyktbxtxq2r3gbpi7xd4dugevh3h4w4q6c@lu6oatjjpltz/37-v2.8-0036-WIP-Use-MAP_POPULATE.patch)
download | inline diff:
From 6cb92c23b3975602f02fb777516fb8aff57e440a Mon Sep 17 00:00:00 2001
From: Andres Freund <andres@anarazel.de>
Date: Tue, 31 Dec 2024 13:25:56 -0500
Subject: [PATCH v2.8 36/38] WIP: Use MAP_POPULATE
For benchmarking it's quite annoying that the first time a memory is touched
has completely different perf characteristics than subsequent accesses. Using
MAP_POPULATE reduces that substantially.
---
src/backend/port/sysv_shmem.c | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/src/backend/port/sysv_shmem.c b/src/backend/port/sysv_shmem.c
index 197926d44f6..a700b02d5a1 100644
--- a/src/backend/port/sysv_shmem.c
+++ b/src/backend/port/sysv_shmem.c
@@ -620,7 +620,7 @@ CreateAnonymousSegment(Size *size)
allocsize += hugepagesize - (allocsize % hugepagesize);
ptr = mmap(NULL, allocsize, PROT_READ | PROT_WRITE,
- PG_MMAP_FLAGS | mmap_flags, -1, 0);
+ PG_MMAP_FLAGS | MAP_POPULATE | mmap_flags, -1, 0);
mmap_errno = errno;
if (huge_pages == HUGE_PAGES_TRY && ptr == MAP_FAILED)
elog(DEBUG1, "mmap(%zu) with MAP_HUGETLB failed, huge pages disabled: %m",
--
2.48.1.76.g4e746b1a31.dirty
^ permalink raw reply [nested|flat] 153+ messages in thread
* Re: AIO v2.5
2025-03-04 19:00 AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-10 18:23 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-11 19:41 ` Re: AIO v2.5 Noah Misch <noah@leadboat.com>
2025-03-11 23:55 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-12 03:57 ` Re: AIO v2.5 Noah Misch <noah@leadboat.com>
2025-03-12 17:06 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-12 23:23 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-13 10:53 ` Re: AIO v2.5 Antonin Houska <ah@cybertec.at>
2025-03-14 19:43 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
@ 2025-03-14 19:58 ` Andres Freund <andres@anarazel.de>
3 siblings, 0 replies; 153+ messages in thread
From: Andres Freund @ 2025-03-14 19:58 UTC (permalink / raw)
To: Antonin Houska <ah@cybertec.at>; +Cc: Noah Misch <noah@leadboat.com>; pgsql-hackers@postgresql.org, Thomas Munro <thomas.munro@gmail.com>; Heikki Linnakangas <hlinnaka@iki.fi>; Robert Haas <robertmhaas@gmail.com>; Jakub Wartak <jakub.wartak@enterprisedb.com>; Jelte Fennema-Nio <postgres@jeltef.nl>
Hi,
On 2025-03-14 15:43:15 -0400, Andres Freund wrote:
> Open items:
>
> - The upstream BAS_BULKREAD is so small that throughput is substantially worse
> once a table reaches 1/4 shared_buffers. That patch in the patchset as-is is
> probably not good enough, although I am not sure about that
>
>
> - The set_max_safe_fds() issue for io_uring
>
>
> - Right now effective_io_concurrency cannot be set > 0 on Windows and other
> platforms that lack posix_fadvise. But with AIO we can read ahead without
> posix_fadvise().
>
> It'd not really make anything worse than today to not remove the limit, but
> it'd be pretty weird to prevent windows etc from benefiting from AIO. Need
> to look around and see whether it would require anything other than doc
> changes.
A fourth, smaller, question:
- Should the docs for debug_io_direct be rephrased and if so, how?
Without read-stream-AIO debug_io_direct=data has completely unusable
performance if there's ever any data IO - and if there's no IO there's no
point in using the option.
Now there is a certain set of workloads where performance with
debug_io_direct=data can be better than master, sometimes substantially
so. But at the same time, without support for at least:
- AIO writes for at least checkpointer, bgwriter
doing one synchronous IO for each buffer is ... slow.
- read-streamified index vacuuming
And probably also:
- AIO-ified writes for writes executed by backends, e.g. due to strategies
Doing one synchronous IO for each buffer is ... slow. And e.g. with COPY
we do a *lot* of those. OTOH, it could be fine if most modifications are
done via INSERTs instead of COPY.
- prefetching for non-BHS index accesses
Without prefetching, a well correlated index-range scan will be orders of
magnitude slower with DIO.
- Anything bypassing shared_buffers, like RelationCopyStorage() or
bulk_write.c will be extremely slow
The only saving grace is that these aren't all *that* common.
Due to those constraints I think it's pretty clear we can't remove the debug_
prefix at this time.
Perhaps it's worth going from
<para>
Currently this feature reduces performance, and is intended for
developer testing only.
</para>
to
<para>
Currently this feature reduces performance in many workloads, and is
intended for testing only.
</para>
I.e. qualify the downside with "many workloads" and widen the audience ever so
slightly?
Greetings,
Andres Freund
^ permalink raw reply [nested|flat] 153+ messages in thread
* Re: AIO v2.5
2025-03-04 19:00 AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-10 18:23 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-11 19:41 ` Re: AIO v2.5 Noah Misch <noah@leadboat.com>
2025-03-11 23:55 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-12 03:57 ` Re: AIO v2.5 Noah Misch <noah@leadboat.com>
2025-03-12 17:06 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-12 23:23 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-13 10:53 ` Re: AIO v2.5 Antonin Houska <ah@cybertec.at>
2025-03-14 19:43 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
@ 2025-03-17 20:00 ` Andres Freund <andres@anarazel.de>
2025-03-18 20:12 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
3 siblings, 1 reply; 153+ messages in thread
From: Andres Freund @ 2025-03-17 20:00 UTC (permalink / raw)
To: Antonin Houska <ah@cybertec.at>; +Cc: Noah Misch <noah@leadboat.com>; pgsql-hackers@postgresql.org, Thomas Munro <thomas.munro@gmail.com>; Heikki Linnakangas <hlinnaka@iki.fi>; Robert Haas <robertmhaas@gmail.com>; Jakub Wartak <jakub.wartak@enterprisedb.com>; Jelte Fennema-Nio <postgres@jeltef.nl>
Hi,
Attached is v2.9 with the following, fairly small, changes:
- Rebased ontop of the additional committed read stream patches
- Committed the localbuf: refactorings (but not the change to expand
refcounting of local buffers, that seems a bit more dependent on the rest)
- Committed the temp table test after some annoying debugging
https://postgr.es/m/w5wr26ijzp7xz2qrxkt6dzvmmn2tn6tn5fp64y6gq5iuvg43hw%40v4guo6x776dq
- Some rephrasing and moving of comments in the first two commits
- There was a small bug in the smgr reopen call I found when reviewing, the
PgAioOpData->read.fd field was referenced for both reads and writes, which
failed to fail because both read/write use a compatible struct layout.
Unless I hear otherwise, I plan to commit the first two patches fairly soon,
followed by the worker support patches a few buildfarm cycles later.
I'm sure there's a bunch of stuff worth improving in the AIO infrastructure
and I can't imagine a project of this size not having bugs. But I think it's
in a state where that's better worked out in tree, with broader exposure and
testing.
Greetings,
Andres Freund
Attachments:
[text/x-diff] v2.9-0001-aio-Basic-subsystem-initialization.patch (25.1K, ../../oav52ytakcvfnk52w5o5wvxdjavxth2tgptxt25gnfmh2pdcpk@omp3xukdu5if/2-v2.9-0001-aio-Basic-subsystem-initialization.patch)
download | inline diff:
From de5db91a35226eb03ba6d643fffc74ee9a89c74b Mon Sep 17 00:00:00 2001
From: Andres Freund <andres@anarazel.de>
Date: Mon, 17 Mar 2025 14:17:28 -0400
Subject: [PATCH v2.9 01/30] aio: Basic subsystem initialization
This commit just does the minimal wiring up of the AIO subsystem, to be fully
added in the next commit, to the rest of the system. The next commit contains
more details about motivation and architecture.
This commit is kept separate to make it easier to review, separating the
changes across the tree from the new subsystem itself.
We discussed squashing this commit with the main commit before merging AIO,
but there has been a mild preference for keeping it separate.
Reviewed-by: Heikki Linnakangas <hlinnaka@iki.fi>
Reviewed-by: Noah Misch <noah@leadboat.com>
Discussion: https://postgr.es/m/uvrtrknj4kdytuboidbhwclo4gxhswwcpgadptsjvjqcluzmah%40brqs62irg4dt
---
src/include/storage/aio.h | 38 ++++++++
src/include/storage/aio_subsys.h | 33 +++++++
src/include/utils/guc.h | 1 +
src/include/utils/guc_hooks.h | 2 +
src/include/utils/resowner.h | 5 ++
src/backend/access/transam/xact.c | 12 +++
src/backend/postmaster/autovacuum.c | 2 +
src/backend/postmaster/bgwriter.c | 2 +
src/backend/postmaster/checkpointer.c | 2 +
src/backend/postmaster/pgarch.c | 2 +
src/backend/postmaster/walsummarizer.c | 2 +
src/backend/postmaster/walwriter.c | 2 +
src/backend/replication/walsender.c | 2 +
src/backend/storage/aio/Makefile | 2 +
src/backend/storage/aio/aio.c | 90 +++++++++++++++++++
src/backend/storage/aio/aio_init.c | 37 ++++++++
src/backend/storage/aio/meson.build | 2 +
src/backend/storage/ipc/ipci.c | 3 +
src/backend/utils/init/postinit.c | 7 ++
src/backend/utils/misc/guc_tables.c | 23 +++++
src/backend/utils/misc/postgresql.conf.sample | 6 ++
src/backend/utils/resowner/resowner.c | 29 ++++++
doc/src/sgml/config.sgml | 51 +++++++++++
src/tools/pgindent/typedefs.list | 1 +
24 files changed, 356 insertions(+)
create mode 100644 src/include/storage/aio.h
create mode 100644 src/include/storage/aio_subsys.h
create mode 100644 src/backend/storage/aio/aio.c
create mode 100644 src/backend/storage/aio/aio_init.c
diff --git a/src/include/storage/aio.h b/src/include/storage/aio.h
new file mode 100644
index 00000000000..e79d5343038
--- /dev/null
+++ b/src/include/storage/aio.h
@@ -0,0 +1,38 @@
+/*-------------------------------------------------------------------------
+ *
+ * aio.h
+ * Main AIO interface
+ *
+ *
+ * Portions Copyright (c) 1996-2025, PostgreSQL Global Development Group
+ * Portions Copyright (c) 1994, Regents of the University of California
+ *
+ * src/include/storage/aio.h
+ *
+ *-------------------------------------------------------------------------
+ */
+#ifndef AIO_H
+#define AIO_H
+
+
+
+/* Enum for io_method GUC. */
+typedef enum IoMethod
+{
+ IOMETHOD_SYNC = 0,
+} IoMethod;
+
+/* We'll default to synchronous execution. */
+#define DEFAULT_IO_METHOD IOMETHOD_SYNC
+
+
+struct dlist_node;
+extern void pgaio_io_release_resowner(struct dlist_node *ioh_node, bool on_error);
+
+
+/* GUCs */
+extern PGDLLIMPORT int io_method;
+extern PGDLLIMPORT int io_max_concurrency;
+
+
+#endif /* AIO_H */
diff --git a/src/include/storage/aio_subsys.h b/src/include/storage/aio_subsys.h
new file mode 100644
index 00000000000..2a6ca1c27a9
--- /dev/null
+++ b/src/include/storage/aio_subsys.h
@@ -0,0 +1,33 @@
+/*-------------------------------------------------------------------------
+ *
+ * aio_subsys.h
+ * Interaction with AIO as a subsystem, rather than actually issuing AIO
+ *
+ * This header is for AIO related functionality that's being called by files
+ * that don't perform AIO, but interact with the AIO subsystem in some
+ * form. E.g. postmaster.c and shared memory initialization need to initialize
+ * AIO but don't perform AIO.
+ *
+ * Portions Copyright (c) 1996-2025, PostgreSQL Global Development Group
+ * Portions Copyright (c) 1994, Regents of the University of California
+ *
+ * src/include/storage/aio_subsys.h
+ *
+ *-------------------------------------------------------------------------
+ */
+#ifndef AIO_SUBSYS_H
+#define AIO_SUBSYS_H
+
+
+/* aio_init.c */
+extern Size AioShmemSize(void);
+extern void AioShmemInit(void);
+
+extern void pgaio_init_backend(void);
+
+
+/* aio.c */
+extern void pgaio_error_cleanup(void);
+extern void AtEOXact_Aio(bool is_commit);
+
+#endif /* AIO_SUBSYS_H */
diff --git a/src/include/utils/guc.h b/src/include/utils/guc.h
index 24444cbc365..f619100467d 100644
--- a/src/include/utils/guc.h
+++ b/src/include/utils/guc.h
@@ -318,6 +318,7 @@ extern PGDLLIMPORT bool optimize_bounded_sort;
*/
extern PGDLLIMPORT const struct config_enum_entry archive_mode_options[];
extern PGDLLIMPORT const struct config_enum_entry dynamic_shared_memory_options[];
+extern PGDLLIMPORT const struct config_enum_entry io_method_options[];
extern PGDLLIMPORT const struct config_enum_entry recovery_target_action_options[];
extern PGDLLIMPORT const struct config_enum_entry wal_level_options[];
extern PGDLLIMPORT const struct config_enum_entry wal_sync_method_options[];
diff --git a/src/include/utils/guc_hooks.h b/src/include/utils/guc_hooks.h
index 9a0d8ec85c7..a3eba8fbe21 100644
--- a/src/include/utils/guc_hooks.h
+++ b/src/include/utils/guc_hooks.h
@@ -64,6 +64,8 @@ extern bool check_default_with_oids(bool *newval, void **extra,
extern bool check_effective_io_concurrency(int *newval, void **extra,
GucSource source);
extern bool check_huge_page_size(int *newval, void **extra, GucSource source);
+extern void assign_io_method(int newval, void *extra);
+extern bool check_io_max_concurrency(int *newval, void **extra, GucSource source);
extern const char *show_in_hot_standby(void);
extern bool check_locale_messages(char **newval, void **extra, GucSource source);
extern void assign_locale_messages(const char *newval, void *extra);
diff --git a/src/include/utils/resowner.h b/src/include/utils/resowner.h
index e8d452ca7ee..aede4bfc820 100644
--- a/src/include/utils/resowner.h
+++ b/src/include/utils/resowner.h
@@ -164,4 +164,9 @@ struct LOCALLOCK;
extern void ResourceOwnerRememberLock(ResourceOwner owner, struct LOCALLOCK *locallock);
extern void ResourceOwnerForgetLock(ResourceOwner owner, struct LOCALLOCK *locallock);
+/* special support for AIO */
+struct dlist_node;
+extern void ResourceOwnerRememberAioHandle(ResourceOwner owner, struct dlist_node *ioh_node);
+extern void ResourceOwnerForgetAioHandle(ResourceOwner owner, struct dlist_node *ioh_node);
+
#endif /* RESOWNER_H */
diff --git a/src/backend/access/transam/xact.c b/src/backend/access/transam/xact.c
index 1b4f21a88d3..b885513f765 100644
--- a/src/backend/access/transam/xact.c
+++ b/src/backend/access/transam/xact.c
@@ -51,6 +51,7 @@
#include "replication/origin.h"
#include "replication/snapbuild.h"
#include "replication/syncrep.h"
+#include "storage/aio_subsys.h"
#include "storage/condition_variable.h"
#include "storage/fd.h"
#include "storage/lmgr.h"
@@ -2411,6 +2412,8 @@ CommitTransaction(void)
RESOURCE_RELEASE_BEFORE_LOCKS,
true, true);
+ AtEOXact_Aio(true);
+
/* Check we've released all buffer pins */
AtEOXact_Buffers(true);
@@ -2716,6 +2719,8 @@ PrepareTransaction(void)
RESOURCE_RELEASE_BEFORE_LOCKS,
true, true);
+ AtEOXact_Aio(true);
+
/* Check we've released all buffer pins */
AtEOXact_Buffers(true);
@@ -2830,6 +2835,8 @@ AbortTransaction(void)
pgstat_report_wait_end();
pgstat_progress_end_command();
+ pgaio_error_cleanup();
+
/* Clean up buffer content locks, too */
UnlockBuffers();
@@ -2960,6 +2967,7 @@ AbortTransaction(void)
ResourceOwnerRelease(TopTransactionResourceOwner,
RESOURCE_RELEASE_BEFORE_LOCKS,
false, true);
+ AtEOXact_Aio(false);
AtEOXact_Buffers(false);
AtEOXact_RelationCache(false);
AtEOXact_TypeCache();
@@ -5232,6 +5240,9 @@ AbortSubTransaction(void)
pgstat_report_wait_end();
pgstat_progress_end_command();
+
+ pgaio_error_cleanup();
+
UnlockBuffers();
/* Reset WAL record construction state */
@@ -5326,6 +5337,7 @@ AbortSubTransaction(void)
RESOURCE_RELEASE_BEFORE_LOCKS,
false, false);
+ AtEOXact_Aio(false);
AtEOSubXact_RelationCache(false, s->subTransactionId,
s->parent->subTransactionId);
AtEOSubXact_TypeCache();
diff --git a/src/backend/postmaster/autovacuum.c b/src/backend/postmaster/autovacuum.c
index 71c34027c88..2513a8ef8a6 100644
--- a/src/backend/postmaster/autovacuum.c
+++ b/src/backend/postmaster/autovacuum.c
@@ -88,6 +88,7 @@
#include "postmaster/autovacuum.h"
#include "postmaster/interrupt.h"
#include "postmaster/postmaster.h"
+#include "storage/aio_subsys.h"
#include "storage/bufmgr.h"
#include "storage/ipc.h"
#include "storage/latch.h"
@@ -465,6 +466,7 @@ AutoVacLauncherMain(const void *startup_data, size_t startup_data_len)
*/
LWLockReleaseAll();
pgstat_report_wait_end();
+ pgaio_error_cleanup();
UnlockBuffers();
/* this is probably dead code, but let's be safe: */
if (AuxProcessResourceOwner)
diff --git a/src/backend/postmaster/bgwriter.c b/src/backend/postmaster/bgwriter.c
index a688cc5d2a1..72f5acceec7 100644
--- a/src/backend/postmaster/bgwriter.c
+++ b/src/backend/postmaster/bgwriter.c
@@ -38,6 +38,7 @@
#include "postmaster/auxprocess.h"
#include "postmaster/bgwriter.h"
#include "postmaster/interrupt.h"
+#include "storage/aio_subsys.h"
#include "storage/buf_internals.h"
#include "storage/bufmgr.h"
#include "storage/condition_variable.h"
@@ -168,6 +169,7 @@ BackgroundWriterMain(const void *startup_data, size_t startup_data_len)
*/
LWLockReleaseAll();
ConditionVariableCancelSleep();
+ pgaio_error_cleanup();
UnlockBuffers();
ReleaseAuxProcessResources(false);
AtEOXact_Buffers(false);
diff --git a/src/backend/postmaster/checkpointer.c b/src/backend/postmaster/checkpointer.c
index 0e228d143a0..fda91ffd1ce 100644
--- a/src/backend/postmaster/checkpointer.c
+++ b/src/backend/postmaster/checkpointer.c
@@ -49,6 +49,7 @@
#include "postmaster/bgwriter.h"
#include "postmaster/interrupt.h"
#include "replication/syncrep.h"
+#include "storage/aio_subsys.h"
#include "storage/bufmgr.h"
#include "storage/condition_variable.h"
#include "storage/fd.h"
@@ -276,6 +277,7 @@ CheckpointerMain(const void *startup_data, size_t startup_data_len)
LWLockReleaseAll();
ConditionVariableCancelSleep();
pgstat_report_wait_end();
+ pgaio_error_cleanup();
UnlockBuffers();
ReleaseAuxProcessResources(false);
AtEOXact_Buffers(false);
diff --git a/src/backend/postmaster/pgarch.c b/src/backend/postmaster/pgarch.c
index dbe4e1d426b..7e622ae4bd2 100644
--- a/src/backend/postmaster/pgarch.c
+++ b/src/backend/postmaster/pgarch.c
@@ -40,6 +40,7 @@
#include "postmaster/interrupt.h"
#include "postmaster/pgarch.h"
#include "storage/condition_variable.h"
+#include "storage/aio_subsys.h"
#include "storage/fd.h"
#include "storage/ipc.h"
#include "storage/latch.h"
@@ -568,6 +569,7 @@ pgarch_archiveXlog(char *xlog)
LWLockReleaseAll();
ConditionVariableCancelSleep();
pgstat_report_wait_end();
+ pgaio_error_cleanup();
ReleaseAuxProcessResources(false);
AtEOXact_Files(false);
AtEOXact_HashTables(false);
diff --git a/src/backend/postmaster/walsummarizer.c b/src/backend/postmaster/walsummarizer.c
index ccba0f84e6e..0fec4f1f871 100644
--- a/src/backend/postmaster/walsummarizer.c
+++ b/src/backend/postmaster/walsummarizer.c
@@ -38,6 +38,7 @@
#include "postmaster/interrupt.h"
#include "postmaster/walsummarizer.h"
#include "replication/walreceiver.h"
+#include "storage/aio_subsys.h"
#include "storage/fd.h"
#include "storage/ipc.h"
#include "storage/latch.h"
@@ -289,6 +290,7 @@ WalSummarizerMain(const void *startup_data, size_t startup_data_len)
LWLockReleaseAll();
ConditionVariableCancelSleep();
pgstat_report_wait_end();
+ pgaio_error_cleanup();
ReleaseAuxProcessResources(false);
AtEOXact_Files(false);
AtEOXact_HashTables(false);
diff --git a/src/backend/postmaster/walwriter.c b/src/backend/postmaster/walwriter.c
index 0380601bcbb..fd92c8b7a33 100644
--- a/src/backend/postmaster/walwriter.c
+++ b/src/backend/postmaster/walwriter.c
@@ -51,6 +51,7 @@
#include "postmaster/auxprocess.h"
#include "postmaster/interrupt.h"
#include "postmaster/walwriter.h"
+#include "storage/aio_subsys.h"
#include "storage/bufmgr.h"
#include "storage/condition_variable.h"
#include "storage/fd.h"
@@ -164,6 +165,7 @@ WalWriterMain(const void *startup_data, size_t startup_data_len)
LWLockReleaseAll();
ConditionVariableCancelSleep();
pgstat_report_wait_end();
+ pgaio_error_cleanup();
UnlockBuffers();
ReleaseAuxProcessResources(false);
AtEOXact_Buffers(false);
diff --git a/src/backend/replication/walsender.c b/src/backend/replication/walsender.c
index d96121b3aad..1028919aecb 100644
--- a/src/backend/replication/walsender.c
+++ b/src/backend/replication/walsender.c
@@ -79,6 +79,7 @@
#include "replication/walsender.h"
#include "replication/walsender_private.h"
#include "storage/condition_variable.h"
+#include "storage/aio_subsys.h"
#include "storage/fd.h"
#include "storage/ipc.h"
#include "storage/pmsignal.h"
@@ -327,6 +328,7 @@ WalSndErrorCleanup(void)
LWLockReleaseAll();
ConditionVariableCancelSleep();
pgstat_report_wait_end();
+ pgaio_error_cleanup();
if (xlogreader != NULL && xlogreader->seg.ws_file >= 0)
wal_segment_close(xlogreader);
diff --git a/src/backend/storage/aio/Makefile b/src/backend/storage/aio/Makefile
index 2f29a9ec4d1..eaeaeeee8e3 100644
--- a/src/backend/storage/aio/Makefile
+++ b/src/backend/storage/aio/Makefile
@@ -9,6 +9,8 @@ top_builddir = ../../../..
include $(top_builddir)/src/Makefile.global
OBJS = \
+ aio.o \
+ aio_init.o \
read_stream.o
include $(top_srcdir)/src/backend/common.mk
diff --git a/src/backend/storage/aio/aio.c b/src/backend/storage/aio/aio.c
new file mode 100644
index 00000000000..828a94efdc3
--- /dev/null
+++ b/src/backend/storage/aio/aio.c
@@ -0,0 +1,90 @@
+/*-------------------------------------------------------------------------
+ *
+ * aio.c
+ * AIO - Core Logic
+ *
+ * Portions Copyright (c) 1996-2025, PostgreSQL Global Development Group
+ * Portions Copyright (c) 1994, Regents of the University of California
+ *
+ * IDENTIFICATION
+ * src/backend/storage/aio/aio.c
+ *
+ *-------------------------------------------------------------------------
+ */
+
+#include "postgres.h"
+
+#include "lib/ilist.h"
+#include "storage/aio.h"
+#include "storage/aio_subsys.h"
+#include "utils/guc.h"
+#include "utils/guc_hooks.h"
+
+
+/* Options for io_method. */
+const struct config_enum_entry io_method_options[] = {
+ {"sync", IOMETHOD_SYNC, false},
+ {NULL, 0, false}
+};
+
+/* GUCs */
+int io_method = DEFAULT_IO_METHOD;
+int io_max_concurrency = -1;
+
+
+
+/*
+ * Release IO handle during resource owner cleanup.
+ */
+void
+pgaio_io_release_resowner(dlist_node *ioh_node, bool on_error)
+{
+}
+
+/*
+ * Perform AIO related cleanup after an error.
+ *
+ * This should be called early in the error recovery paths, as later steps may
+ * need to issue AIO (e.g. to record a transaction abort WAL record).
+ */
+void
+pgaio_error_cleanup(void)
+{
+}
+
+/*
+ * Perform AIO related checks at (sub-)transactional boundaries.
+ *
+ * This should be called late during (sub-)transactional commit/abort, after
+ * all steps that might need to perform AIO, so that we can verify that the
+ * AIO subsystem is in a valid state at the end of a transaction.
+ */
+void
+AtEOXact_Aio(bool is_commit)
+{
+}
+
+void
+assign_io_method(int newval, void *extra)
+{
+}
+
+bool
+check_io_max_concurrency(int *newval, void **extra, GucSource source)
+{
+ if (*newval == -1)
+ {
+ /*
+ * Auto-tuning will be applied later during startup, as auto-tuning
+ * depends on the value of various GUCs.
+ */
+ return true;
+ }
+ else if (*newval == 0)
+ {
+ GUC_check_errdetail("Only -1 or values bigger than 0 are valid.");
+ return false;
+ }
+
+ return true;
+}
diff --git a/src/backend/storage/aio/aio_init.c b/src/backend/storage/aio/aio_init.c
new file mode 100644
index 00000000000..aeacc144149
--- /dev/null
+++ b/src/backend/storage/aio/aio_init.c
@@ -0,0 +1,37 @@
+/*-------------------------------------------------------------------------
+ *
+ * aio_init.c
+ * AIO - Subsystem Initialization
+ *
+ * Portions Copyright (c) 1996-2025, PostgreSQL Global Development Group
+ * Portions Copyright (c) 1994, Regents of the University of California
+ *
+ * IDENTIFICATION
+ * src/backend/storage/aio/aio_init.c
+ *
+ *-------------------------------------------------------------------------
+ */
+
+#include "postgres.h"
+
+#include "storage/aio_subsys.h"
+
+
+
+Size
+AioShmemSize(void)
+{
+ Size sz = 0;
+
+ return sz;
+}
+
+void
+AioShmemInit(void)
+{
+}
+
+void
+pgaio_init_backend(void)
+{
+}
diff --git a/src/backend/storage/aio/meson.build b/src/backend/storage/aio/meson.build
index 8abe0eb4863..c822fd4ddf7 100644
--- a/src/backend/storage/aio/meson.build
+++ b/src/backend/storage/aio/meson.build
@@ -1,5 +1,7 @@
# Copyright (c) 2024-2025, PostgreSQL Global Development Group
backend_sources += files(
+ 'aio.c',
+ 'aio_init.c',
'read_stream.c',
)
diff --git a/src/backend/storage/ipc/ipci.c b/src/backend/storage/ipc/ipci.c
index 174eed70367..2fa045e6b0f 100644
--- a/src/backend/storage/ipc/ipci.c
+++ b/src/backend/storage/ipc/ipci.c
@@ -37,6 +37,7 @@
#include "replication/slotsync.h"
#include "replication/walreceiver.h"
#include "replication/walsender.h"
+#include "storage/aio_subsys.h"
#include "storage/bufmgr.h"
#include "storage/dsm.h"
#include "storage/dsm_registry.h"
@@ -148,6 +149,7 @@ CalculateShmemSize(int *num_semaphores)
size = add_size(size, WaitEventCustomShmemSize());
size = add_size(size, InjectionPointShmemSize());
size = add_size(size, SlotSyncShmemSize());
+ size = add_size(size, AioShmemSize());
/* include additional requested shmem from preload libraries */
size = add_size(size, total_addin_request);
@@ -340,6 +342,7 @@ CreateOrAttachShmemStructs(void)
StatsShmemInit();
WaitEventCustomShmemInit();
InjectionPointShmemInit();
+ AioShmemInit();
}
/*
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index 4b2faf1ba9d..7958ea11b73 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -43,6 +43,7 @@
#include "replication/slot.h"
#include "replication/slotsync.h"
#include "replication/walsender.h"
+#include "storage/aio_subsys.h"
#include "storage/bufmgr.h"
#include "storage/fd.h"
#include "storage/ipc.h"
@@ -635,6 +636,12 @@ BaseInit(void)
*/
pgstat_initialize();
+ /*
+ * Initialize AIO before infrastructure that might need to actually
+ * execute AIO.
+ */
+ pgaio_init_backend();
+
/* Do local initialization of storage and buffer managers */
InitSync();
smgrinit();
diff --git a/src/backend/utils/misc/guc_tables.c b/src/backend/utils/misc/guc_tables.c
index 9c0b10ad4dc..0d3ebf06a95 100644
--- a/src/backend/utils/misc/guc_tables.c
+++ b/src/backend/utils/misc/guc_tables.c
@@ -72,6 +72,7 @@
#include "replication/slot.h"
#include "replication/slotsync.h"
#include "replication/syncrep.h"
+#include "storage/aio.h"
#include "storage/bufmgr.h"
#include "storage/bufpage.h"
#include "storage/large_object.h"
@@ -3254,6 +3255,18 @@ struct config_int ConfigureNamesInt[] =
NULL, NULL, NULL
},
+ {
+ {"io_max_concurrency",
+ PGC_POSTMASTER,
+ RESOURCES_IO,
+ gettext_noop("Max number of IOs that one process can execute simultaneously."),
+ NULL,
+ },
+ &io_max_concurrency,
+ -1, -1, 1024,
+ check_io_max_concurrency, NULL, NULL
+ },
+
{
{"backend_flush_after", PGC_USERSET, RESOURCES_IO,
gettext_noop("Number of pages after which previously performed writes are flushed to disk."),
@@ -5311,6 +5324,16 @@ struct config_enum ConfigureNamesEnum[] =
NULL, NULL, NULL
},
+ {
+ {"io_method", PGC_POSTMASTER, RESOURCES_IO,
+ gettext_noop("Selects the method for executing asynchronous I/O."),
+ NULL
+ },
+ &io_method,
+ DEFAULT_IO_METHOD, io_method_options,
+ NULL, assign_io_method, NULL
+ },
+
/* End-of-list marker */
{
{NULL, 0, 0, NULL, NULL}, NULL, 0, NULL, NULL, NULL, NULL
diff --git a/src/backend/utils/misc/postgresql.conf.sample b/src/backend/utils/misc/postgresql.conf.sample
index 8de86e0c945..43c2ec2153e 100644
--- a/src/backend/utils/misc/postgresql.conf.sample
+++ b/src/backend/utils/misc/postgresql.conf.sample
@@ -202,6 +202,12 @@
#maintenance_io_concurrency = 10 # 1-1000; 0 disables prefetching
#io_combine_limit = 128kB # usually 1-32 blocks (depends on OS)
+#io_method = sync # sync (change requires restart)
+#io_max_concurrency = -1 # Max number of IOs that one process
+ # can execute simultaneously
+ # -1 sets based on shared_buffers
+ # (change requires restart)
+
# - Worker Processes -
#max_worker_processes = 8 # (change requires restart)
diff --git a/src/backend/utils/resowner/resowner.c b/src/backend/utils/resowner/resowner.c
index ac5ca4a765e..d39f3e1b655 100644
--- a/src/backend/utils/resowner/resowner.c
+++ b/src/backend/utils/resowner/resowner.c
@@ -47,6 +47,8 @@
#include "common/hashfn.h"
#include "common/int.h"
+#include "lib/ilist.h"
+#include "storage/aio.h"
#include "storage/ipc.h"
#include "storage/predicate.h"
#include "storage/proc.h"
@@ -155,6 +157,12 @@ struct ResourceOwnerData
/* The local locks cache. */
LOCALLOCK *locks[MAX_RESOWNER_LOCKS]; /* list of owned locks */
+
+ /*
+ * AIO handles need be registered in critical sections and therefore
+ * cannot use the normal ResourceElem mechanism.
+ */
+ dlist_head aio_handles;
};
@@ -425,6 +433,8 @@ ResourceOwnerCreate(ResourceOwner parent, const char *name)
parent->firstchild = owner;
}
+ dlist_init(&owner->aio_handles);
+
return owner;
}
@@ -725,6 +735,13 @@ ResourceOwnerReleaseInternal(ResourceOwner owner,
* so issue warnings. In the abort case, just clean up quietly.
*/
ResourceOwnerReleaseAll(owner, phase, isCommit);
+
+ while (!dlist_is_empty(&owner->aio_handles))
+ {
+ dlist_node *node = dlist_head_node(&owner->aio_handles);
+
+ pgaio_io_release_resowner(node, !isCommit);
+ }
}
else if (phase == RESOURCE_RELEASE_LOCKS)
{
@@ -1082,3 +1099,15 @@ ResourceOwnerForgetLock(ResourceOwner owner, LOCALLOCK *locallock)
elog(ERROR, "lock reference %p is not owned by resource owner %s",
locallock, owner->name);
}
+
+void
+ResourceOwnerRememberAioHandle(ResourceOwner owner, struct dlist_node *ioh_node)
+{
+ dlist_push_tail(&owner->aio_handles, ioh_node);
+}
+
+void
+ResourceOwnerForgetAioHandle(ResourceOwner owner, struct dlist_node *ioh_node)
+{
+ dlist_delete_from(&owner->aio_handles, ioh_node);
+}
diff --git a/doc/src/sgml/config.sgml b/doc/src/sgml/config.sgml
index 3d62c8bd274..7ec18bb7627 100644
--- a/doc/src/sgml/config.sgml
+++ b/doc/src/sgml/config.sgml
@@ -2638,6 +2638,57 @@ include_dir 'conf.d'
</para>
</listitem>
</varlistentry>
+
+ <varlistentry id="guc-io-max-concurrency" xreflabel="io_max_concurrency">
+ <term><varname>io_max_concurrency</varname> (<type>integer</type>)
+ <indexterm>
+ <primary><varname>io_max_concurrency</varname> configuration parameter</primary>
+ </indexterm>
+ </term>
+ <listitem>
+ <para>
+ Controls the maximum number of I/O operations that one process can
+ execute simultaneously.
+ </para>
+ <para>
+ The default setting of <literal>-1</literal> selects a number based
+ on <xref linkend="guc-shared-buffers"/> and the maximum number of
+ processes (<xref linkend="guc-max-connections"/>, <xref
+ linkend="guc-autovacuum-worker-slots"/>, <xref
+ linkend="guc-max-worker-processes"/> and <xref
+ linkend="guc-max-wal-senders"/>), but not more than
+ <literal>64</literal>.
+ </para>
+ <para>
+ This parameter can only be set at server start.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry id="guc-io-method" xreflabel="io_method">
+ <term><varname>io_method</varname> (<type>enum</type>)
+ <indexterm>
+ <primary><varname>io_method</varname> configuration parameter</primary>
+ </indexterm>
+ </term>
+ <listitem>
+ <para>
+ Selects the method for executing asynchronous I/O.
+ Possible values are:
+ <itemizedlist>
+ <listitem>
+ <para>
+ <literal>sync</literal> (execute asynchronous-eligible I/O synchronously)
+ </para>
+ </listitem>
+ </itemizedlist>
+ </para>
+ <para>
+ This parameter can only be set at server start.
+ </para>
+ </listitem>
+ </varlistentry>
+
</variablelist>
</sect2>
diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list
index 93339ef3c58..3c9e823f07e 100644
--- a/src/tools/pgindent/typedefs.list
+++ b/src/tools/pgindent/typedefs.list
@@ -1279,6 +1279,7 @@ IntoClause
InvalMessageArray
InvalidationInfo
InvalidationMsgsGroup
+IoMethod
IpcMemoryId
IpcMemoryKey
IpcMemoryState
--
2.48.1.76.g4e746b1a31.dirty
[text/x-diff] v2.9-0002-aio-Add-core-asynchronous-I-O-infrastructure.patch (90.3K, ../../oav52ytakcvfnk52w5o5wvxdjavxth2tgptxt25gnfmh2pdcpk@omp3xukdu5if/3-v2.9-0002-aio-Add-core-asynchronous-I-O-infrastructure.patch)
download | inline diff:
From 8d8ec39d441a6b9c0b1a01bdec3849a981195af5 Mon Sep 17 00:00:00 2001
From: Andres Freund <andres@anarazel.de>
Date: Sat, 15 Mar 2025 12:29:57 -0400
Subject: [PATCH v2.9 02/30] aio: Add core asynchronous I/O infrastructure
The main motivations to use AIO in PostgreSQL are:
a) Reduce the time spent waiting for IO by issuing IO sufficiently early.
In a few places we have approximated this using posix_fadvise() based
prefetching, but that is fairly limited (no completion feedback, double the
syscalls, only works with buffered IO, only works on some OSs).
b) Allow to use Direct-I/O (DIO).
DIO can offload most of the work for IO to hardware and thus increase
throughput / decrease CPU utilization, as well as reduce latency. While we
have gained the ability to configure DIO in d4e71df6, it is not yet usable
for real world workloads, as every IO is executed synchronously.
For portability, the new AIO infrastructure allows to implement AIO using
different methods. The choice of the AIO method is controlled by the new
io_method GUC. As of this commit, the only implemented method is "sync",
i.e. AIO is not actually executed asynchronously. The "sync" method exists to
allow to bypass most of the new code initially.
Subsequent commits will introduce additional IO methods, including a
cross-platform method implemented using worker processes and a linux specific
method using io_uring.
To allow different parts of postgres to use AIO, the core AIO infrastructure
does not need to know what kind of files it is operating on. The necessary
behavioral differences for different files are abstracted as "AIO
Targets". One example target would be smgr. For boring portability reasons,
all targets currently need to be added to an array in aio_target.c. This
commit does not implement any AIO targets, just the infrastructure for
them. The smgr target will be added in a later commit.
Completion (and other events) of IOs for one type of file (i.e. one AIO
target) need to be reacted to differently, based on the IO operation and the
callsite. This is made possible by callbacks that can be registered on
IOs. E.g. an smgr read into a local buffer does not need to update the
corresponding BufferDesc (as there is none), but a read into shared buffers
does. This commit does not contain any callbacks, they will be added in
subsequent commits.
For now the AIO infrastructure only understands READV and WRITEV operations,
but it is expected that more operations will be added. E.g. fsync/fdatasync,
flush_range and network operations like send/recv.
As of this commit, nothing uses the AIO infrastructure. Later commits will add
an smgr target, md.c and bufmgr.c callbacks and then finally use AIO for
read_stream.c IO, which, in one fell swoop, will convert all read stream users
to AIO.
The goal is to use AIO in many more places. There are patches to use AIO for
checkpointer and bgwriter that are reasonably close to being ready. There also
are prototypes to use it for WAL, relation extension, backend writes and many
more. Those prototypes were important to ensure the design of the AIO
subsystem is not too limiting (e.g. WAL writes need to happen in critical
sections, which influenced a lot of the design).
A future commit will add an AIO README explaining the AIO architecture and how
to use the AIO subsystem. The README is added later, as it references details
only added in later commits.
Many many more people than the folks named below have contributed with
feedback, work on semi-independent patches etc. E.g. various folks have
contributed patches to use the read stream infrastructure (added by Thomas in
b5a9b18cd0b) in more places. Similarly, a *lot* of folks have contributed to
the CI infrastructure, which I had started to work on to make adding AIO
feasible.
Some of the work by contributors has gone into the "v1" prototype of AIO,
which heavily influenced the current design of the AIO subsystem. None of the
code from that directly survives, but without the prototype, the current
version of the AIO infrastructure would not exist.
Similarly, the reviewers below have not necessarily looked at the current
design or the whole infrastructure, but have provided very valuable input. I
am to blame for problems, not they.
Author: Andres Freund <andres@anarazel.de>
Co-authored-by: Thomas Munro <thomas.munro@gmail.com>
Co-authored-by: Nazir Bilal Yavuz <byavuz81@gmail.com>
Co-authored-by: Melanie Plageman <melanieplageman@gmail.com>
Reviewed-by: Heikki Linnakangas <hlinnaka@iki.fi>
Reviewed-by: Noah Misch <noah@leadboat.com>
Reviewed-by: Jakub Wartak <jakub.wartak@enterprisedb.com>
Reviewed-by: Melanie Plageman <melanieplageman@gmail.com>
Reviewed-by: Robert Haas <robertmhaas@gmail.com>
Reviewed-by: Dmitry Dolgov <9erthalion6@gmail.com>
Reviewed-by: Antonin Houska <ah@cybertec.at>
Discussion: https://postgr.es/m/uvrtrknj4kdytuboidbhwclo4gxhswwcpgadptsjvjqcluzmah%40brqs62irg4dt
Discussion: https://postgr.es/m/20210223100344.llw5an2aklengrmn@alap3.anarazel.de
Discussion: https://postgr.es/m/stj36ea6yyhoxtqkhpieia2z4krnam7qyetc57rfezgk4zgapf@gcnactj4z56m
---
src/include/storage/aio.h | 311 +++++
src/include/storage/aio_internal.h | 395 ++++++
src/include/storage/aio_types.h | 117 ++
src/backend/storage/aio/Makefile | 4 +
src/backend/storage/aio/aio.c | 1130 +++++++++++++++++
src/backend/storage/aio/aio_callback.c | 308 +++++
src/backend/storage/aio/aio_init.c | 198 +++
src/backend/storage/aio/aio_io.c | 184 +++
src/backend/storage/aio/aio_target.c | 114 ++
src/backend/storage/aio/meson.build | 4 +
src/backend/storage/aio/method_sync.c | 47 +
.../utils/activity/wait_event_names.txt | 1 +
src/tools/pgindent/typedefs.list | 21 +
13 files changed, 2834 insertions(+)
create mode 100644 src/include/storage/aio_internal.h
create mode 100644 src/include/storage/aio_types.h
create mode 100644 src/backend/storage/aio/aio_callback.c
create mode 100644 src/backend/storage/aio/aio_io.c
create mode 100644 src/backend/storage/aio/aio_target.c
create mode 100644 src/backend/storage/aio/method_sync.c
diff --git a/src/include/storage/aio.h b/src/include/storage/aio.h
index e79d5343038..f48a4962089 100644
--- a/src/include/storage/aio.h
+++ b/src/include/storage/aio.h
@@ -3,6 +3,10 @@
* aio.h
* Main AIO interface
*
+ * This is the header to include when actually issuing AIO. When just
+ * declaring functions involving an AIO related type, it might suffice to
+ * include aio_types.h. Initialization related functions are in the dedicated
+ * aio_init.h.
*
* Portions Copyright (c) 1996-2025, PostgreSQL Global Development Group
* Portions Copyright (c) 1994, Regents of the University of California
@@ -14,6 +18,9 @@
#ifndef AIO_H
#define AIO_H
+#include "storage/aio_types.h"
+#include "storage/procnumber.h"
+
/* Enum for io_method GUC. */
@@ -26,9 +33,313 @@ typedef enum IoMethod
#define DEFAULT_IO_METHOD IOMETHOD_SYNC
+/*
+ * Flags for an IO that can be set with pgaio_io_set_flag().
+ */
+typedef enum PgAioHandleFlags
+{
+ /*
+ * The IO references backend local memory.
+ *
+ * This needs to be set on an IO whenever the IO references process-local
+ * memory. Some IO methods do not support executing IO that references
+ * process local memory and thus need to fall back to executing IO
+ * synchronously for IOs with this flag set.
+ *
+ * Required for correctness.
+ */
+ PGAIO_HF_REFERENCES_LOCAL = 1 << 1,
+
+ /*
+ * Hint that IO will be executed synchronously.
+ *
+ * This can make it a bit cheaper to execute synchronous IO via the AIO
+ * interface, to avoid needing an AIO and non-AIO version of code.
+ *
+ * Advantageous to set, if applicable, but not required for correctness.
+ */
+ PGAIO_HF_SYNCHRONOUS = 1 << 0,
+
+ /*
+ * IO is using buffered IO, used to control heuristic in some IO methods.
+ *
+ * Advantageous to set, if applicable, but not required for correctness.
+ */
+ PGAIO_HF_BUFFERED = 1 << 2,
+} PgAioHandleFlags;
+
+/*
+ * The IO operations supported by the AIO subsystem.
+ *
+ * This could be in aio_internal.h, as it is not pubicly referenced, but
+ * PgAioOpData currently *does* need to be public, therefore keeping this
+ * public seems to make sense.
+ */
+typedef enum PgAioOp
+{
+ /* intentionally the zero value, to help catch zeroed memory etc */
+ PGAIO_OP_INVALID = 0,
+
+ PGAIO_OP_READV,
+ PGAIO_OP_WRITEV,
+
+ /**
+ * In the near term we'll need at least:
+ * - fsync / fdatasync
+ * - flush_range
+ *
+ * Eventually we'll additionally want at least:
+ * - send
+ * - recv
+ * - accept
+ **/
+} PgAioOp;
+
+#define PGAIO_OP_COUNT (PGAIO_OP_WRITEV + 1)
+
+
+/*
+ * On what is IO being performed?
+ *
+ * PgAioTargetID specific behaviour should be implemented in
+ * aio_target.c.
+ */
+typedef enum PgAioTargetID
+{
+ /* intentionally the zero value, to help catch zeroed memory etc */
+ PGAIO_TID_INVALID = 0,
+} PgAioTargetID;
+
+#define PGAIO_TID_COUNT (PGAIO_TID_INVALID + 1)
+
+
+/*
+ * Data necessary for support IO operations (see PgAioOp).
+ *
+ * NB: Note that the FDs in here may *not* be relied upon for re-issuing
+ * requests (e.g. for partial reads/writes or in an IO worker) - the FD might
+ * be from another process, or closed since. That's not a problem for staged
+ * IOs, as all staged IOs are submitted when closing an FD.
+ */
+typedef union
+{
+ struct
+ {
+ int fd;
+ uint16 iov_length;
+ uint64 offset;
+ } read;
+
+ struct
+ {
+ int fd;
+ uint16 iov_length;
+ uint64 offset;
+ } write;
+} PgAioOpData;
+
+
+/*
+ * Information the object that IO is executed on. Mostly callbacks that
+ * operate on PgAioTargetData.
+ *
+ * typedef is in aio_types.h
+ */
+struct PgAioTargetInfo
+{
+ /*
+ * To support executing using worker processes, the file descriptor for an
+ * IO may need to be be reopened in a different process.
+ */
+ void (*reopen) (PgAioHandle *ioh);
+
+ /* describe the target of the IO, used for log messages and views */
+ char *(*describe_identity) (const PgAioTargetData *sd);
+
+ /* name of the target, used in log messages / views */
+ const char *name;
+};
+
+
+/*
+ * IDs for callbacks that can be registered on an IO.
+ *
+ * Callbacks are identified by an ID rather than a function pointer. There are
+ * two main reasons:
+ *
+ * 1) Memory within PgAioHandle is precious, due to the number of PgAioHandle
+ * structs in pre-allocated shared memory.
+ *
+ * 2) Due to EXEC_BACKEND function pointers are not necessarily stable between
+ * different backends, therefore function pointers cannot directly be in
+ * shared memory.
+ *
+ * Without 2), we could fairly easily allow to add new callbacks, by filling a
+ * ID->pointer mapping table on demand. In the presence of 2 that's still
+ * doable, but harder, because every process has to re-register the pointers
+ * so that a local ID->"backend local pointer" mapping can be maintained.
+ */
+typedef enum PgAioHandleCallbackID
+{
+ PGAIO_HCB_INVALID,
+} PgAioHandleCallbackID;
+
+
+typedef void (*PgAioHandleCallbackStage) (PgAioHandle *ioh, uint8 cb_flags);
+typedef PgAioResult (*PgAioHandleCallbackComplete) (PgAioHandle *ioh, PgAioResult prior_result, uint8 cb_flags);
+typedef void (*PgAioHandleCallbackReport) (PgAioResult result, const PgAioTargetData *target_data, int elevel);
+
+/* typedef is in aio_types.h */
+struct PgAioHandleCallbacks
+{
+ /*
+ * Prepare resources affected by the IO for execution. This could e.g.
+ * include moving ownership of buffer pins to the AIO subsystem.
+ */
+ PgAioHandleCallbackStage stage;
+
+ /*
+ * Update the state of resources affected by the IO to reflect completion
+ * of the IO. This could e.g. include updating shared buffer state to
+ * signal the IO has finished.
+ *
+ * The _shared suffix indicates that this is executed by the backend that
+ * completed the IO, which may or may not be the backend that issued the
+ * IO. Obviously the callback thus can only modify resources in shared
+ * memory.
+ *
+ * The latest registered callback is called first. This allows
+ * higher-level code to register callbacks that can rely on callbacks
+ * registered by lower-level code to already have been executed.
+ *
+ * NB: This is called in a critical section. Errors can be signalled by
+ * the callback's return value, it's the responsibility of the IO's issuer
+ * to react appropriately.
+ */
+ PgAioHandleCallbackComplete complete_shared;
+
+ /*
+ * Like complete_shared, except called in the issuing backend.
+ *
+ * This variant of the completion callback is useful when backend-local
+ * state has to be updated to reflect the IO's completion. E.g. a
+ * temporary buffer's BufferDesc isn't accessible in complete_shared.
+ *
+ * Local callbacks are only called after complete_shared for all
+ * registered callbacks has been called.
+ */
+ PgAioHandleCallbackComplete complete_local;
+
+ /*
+ * Report the result of an IO operation. This is e.g. used to raise an
+ * error after an IO failed at the appropriate time (i.e. not when the IO
+ * failed, but under control of the code that issued the IO).
+ */
+ PgAioHandleCallbackReport report;
+};
+
+
+
+/*
+ * How many callbacks can be registered for one IO handle. Currently we only
+ * need two, but it's not hard to imagine needing a few more.
+ */
+#define PGAIO_HANDLE_MAX_CALLBACKS 4
+
+
+
+/* --------------------------------------------------------------------------------
+ * IO Handles
+ * --------------------------------------------------------------------------------
+ */
+
+/* functions in aio.c */
+struct ResourceOwnerData;
+extern PgAioHandle *pgaio_io_acquire(struct ResourceOwnerData *resowner, PgAioReturn *ret);
+extern PgAioHandle *pgaio_io_acquire_nb(struct ResourceOwnerData *resowner, PgAioReturn *ret);
+
+extern void pgaio_io_release(PgAioHandle *ioh);
struct dlist_node;
extern void pgaio_io_release_resowner(struct dlist_node *ioh_node, bool on_error);
+extern void pgaio_io_set_flag(PgAioHandle *ioh, PgAioHandleFlags flag);
+
+extern int pgaio_io_get_id(PgAioHandle *ioh);
+extern ProcNumber pgaio_io_get_owner(PgAioHandle *ioh);
+
+extern void pgaio_io_get_wref(PgAioHandle *ioh, PgAioWaitRef *iow);
+
+/* functions in aio_io.c */
+struct iovec;
+extern int pgaio_io_get_iovec(PgAioHandle *ioh, struct iovec **iov);
+
+extern PgAioOp pgaio_io_get_op(PgAioHandle *ioh);
+extern PgAioOpData *pgaio_io_get_op_data(PgAioHandle *ioh);
+
+extern void pgaio_io_prep_readv(PgAioHandle *ioh,
+ int fd, int iovcnt, uint64 offset);
+extern void pgaio_io_prep_writev(PgAioHandle *ioh,
+ int fd, int iovcnt, uint64 offset);
+
+/* functions in aio_target.c */
+extern void pgaio_io_set_target(PgAioHandle *ioh, PgAioTargetID targetid);
+extern bool pgaio_io_has_target(PgAioHandle *ioh);
+extern PgAioTargetData *pgaio_io_get_target_data(PgAioHandle *ioh);
+extern char *pgaio_io_get_target_description(PgAioHandle *ioh);
+
+/* functions in aio_callback.c */
+extern void pgaio_io_register_callbacks(PgAioHandle *ioh, PgAioHandleCallbackID cb_id,
+ uint8 cb_data);
+extern void pgaio_io_set_handle_data_64(PgAioHandle *ioh, uint64 *data, uint8 len);
+extern void pgaio_io_set_handle_data_32(PgAioHandle *ioh, uint32 *data, uint8 len);
+extern uint64 *pgaio_io_get_handle_data(PgAioHandle *ioh, uint8 *len);
+
+
+
+/* --------------------------------------------------------------------------------
+ * IO Wait References
+ * --------------------------------------------------------------------------------
+ */
+
+extern void pgaio_wref_clear(PgAioWaitRef *iow);
+extern bool pgaio_wref_valid(PgAioWaitRef *iow);
+extern int pgaio_wref_get_id(PgAioWaitRef *iow);
+
+extern void pgaio_wref_wait(PgAioWaitRef *iow);
+extern bool pgaio_wref_check_done(PgAioWaitRef *iow);
+
+
+
+/* --------------------------------------------------------------------------------
+ * IO Result
+ * --------------------------------------------------------------------------------
+ */
+
+extern void pgaio_result_report(PgAioResult result, const PgAioTargetData *target_data,
+ int elevel);
+
+
+
+/* --------------------------------------------------------------------------------
+ * Actions on multiple IOs.
+ * --------------------------------------------------------------------------------
+ */
+
+extern void pgaio_enter_batchmode(void);
+extern void pgaio_exit_batchmode(void);
+extern void pgaio_submit_staged(void);
+extern bool pgaio_have_staged(void);
+
+
+
+/* --------------------------------------------------------------------------------
+ * Other
+ * --------------------------------------------------------------------------------
+ */
+
+extern void pgaio_closing_fd(int fd);
+
+
/* GUCs */
extern PGDLLIMPORT int io_method;
diff --git a/src/include/storage/aio_internal.h b/src/include/storage/aio_internal.h
new file mode 100644
index 00000000000..52b7751efa8
--- /dev/null
+++ b/src/include/storage/aio_internal.h
@@ -0,0 +1,395 @@
+/*-------------------------------------------------------------------------
+ *
+ * aio_internal.h
+ * AIO related declarations that should only be used by the AIO subsystem
+ * internally.
+ *
+ *
+ * Portions Copyright (c) 1996-2025, PostgreSQL Global Development Group
+ * Portions Copyright (c) 1994, Regents of the University of California
+ *
+ * src/include/storage/aio_internal.h
+ *
+ *-------------------------------------------------------------------------
+ */
+#ifndef AIO_INTERNAL_H
+#define AIO_INTERNAL_H
+
+
+#include "lib/ilist.h"
+#include "port/pg_iovec.h"
+#include "storage/aio.h"
+#include "storage/condition_variable.h"
+
+
+/*
+ * The maximum number of IOs that can be batch submitted at once.
+ */
+#define PGAIO_SUBMIT_BATCH_SIZE 32
+
+
+
+/*
+ * State machine for handles. With some exceptions, noted below, handles move
+ * linearly through all states.
+ *
+ * State changes should all go through pgaio_io_update_state().
+ */
+typedef enum PgAioHandleState
+{
+ /* not in use */
+ PGAIO_HS_IDLE = 0,
+
+ /*
+ * Returned by pgaio_io_acquire(). The next state is either DEFINED (if
+ * pgaio_io_prep_*() is called), or IDLE (if pgaio_io_release() is
+ * called).
+ */
+ PGAIO_HS_HANDED_OUT,
+
+ /*
+ * pgaio_io_prep_*() has been called, but IO is not yet staged. At this
+ * point the handle has all the information for the IO to be executed.
+ */
+ PGAIO_HS_DEFINED,
+
+ /*
+ * stage() callbacks have been called, handle ready to be submitted for
+ * execution. Unless in batchmode (see c.f. pgaio_enter_batchmode()), the
+ * IO will be submitted immediately after.
+ */
+ PGAIO_HS_STAGED,
+
+ /* IO has been submitted to the IO method for execution */
+ PGAIO_HS_SUBMITTED,
+
+ /* IO finished, but result has not yet been processed */
+ PGAIO_HS_COMPLETED_IO,
+
+ /*
+ * IO completed, shared completion has been called.
+ *
+ * If the IO completion occurs in the issuing backend, local callbacks
+ * will immediately be called. Otherwise the handle stays in
+ * COMPLETED_SHARED until the issuing backend waits for the completion of
+ * the IO.
+ */
+ PGAIO_HS_COMPLETED_SHARED,
+
+ /*
+ * IO completed, local completion has been called.
+ *
+ * After this the handle will be made reusable and go into IDLE state.
+ */
+ PGAIO_HS_COMPLETED_LOCAL,
+} PgAioHandleState;
+
+
+struct ResourceOwnerData;
+
+/* typedef is in aio_types.h */
+struct PgAioHandle
+{
+ /* all state updates should go through pgaio_io_update_state() */
+ PgAioHandleState state:8;
+
+ /* what are we operating on */
+ PgAioTargetID target:8;
+
+ /* which IO operation */
+ PgAioOp op:8;
+
+ /* bitfield of PgAioHandleFlags */
+ uint8 flags;
+
+ uint8 num_callbacks;
+
+ /* using the proper type here would use more space */
+ uint8 callbacks[PGAIO_HANDLE_MAX_CALLBACKS];
+
+ /* data forwarded to each callback */
+ uint8 callbacks_data[PGAIO_HANDLE_MAX_CALLBACKS];
+
+ /*
+ * Length of data associated with handle using
+ * pgaio_io_set_handle_data_*().
+ */
+ uint8 handle_data_len;
+
+ /* XXX: could be optimized out with some pointer math */
+ int32 owner_procno;
+
+ /* raw result of the IO operation */
+ int32 result;
+
+ /**
+ * In which list the handle is registered, depends on the state:
+ * - IDLE, in per-backend list
+ * - HANDED_OUT - not in a list
+ * - DEFINED - in per-backend staged list
+ * - STAGED - in per-backend staged list
+ * - SUBMITTED - in issuer's in_flight list
+ * - COMPLETED_IO - in issuer's in_flight list
+ * - COMPLETED_SHARED - in issuer's in_flight list
+ **/
+ dlist_node node;
+
+ struct ResourceOwnerData *resowner;
+ dlist_node resowner_node;
+
+ /* incremented every time the IO handle is reused */
+ uint64 generation;
+
+ /*
+ * To wait for the IO to complete other backends can wait on this CV. Note
+ * that, if in SUBMITTED state, a waiter first needs to check if it needs
+ * to do work via IoMethodOps->wait_one().
+ */
+ ConditionVariable cv;
+
+ /* result of shared callback, passed to issuer callback */
+ PgAioResult distilled_result;
+
+ /*
+ * Index into PgAioCtl->iovecs and PgAioCtl->handle_data.
+ *
+ * At the moment there's no need to differentiate between the two, but
+ * that won't necessarily stay that way.
+ */
+ uint32 iovec_off;
+
+ /*
+ * If not NULL, this memory location will be updated with information
+ * about the IOs completion iff the issuing backend learns about the IOs
+ * completion.
+ */
+ PgAioReturn *report_return;
+
+ /* Data necessary for the IO to be performed */
+ PgAioOpData op_data;
+
+ /*
+ * Data necessary to identify the object undergoing IO to higher-level
+ * code. Needs to be sufficient to allow another backend to reopen the
+ * file.
+ */
+ PgAioTargetData target_data;
+};
+
+
+typedef struct PgAioBackend
+{
+ /* index into PgAioCtl->io_handles */
+ uint32 io_handle_off;
+
+ /* IO Handles that currently are not used */
+ dclist_head idle_ios;
+
+ /*
+ * Only one IO may be returned by pgaio_io_acquire()/pgaio_io_acquire_nb()
+ * without having been either defined (by actually associating it with IO)
+ * or released (with pgaio_io_release()). This restriction is necessary to
+ * guarantee that we always can acquire an IO. ->handed_out_io is used to
+ * enforce that rule.
+ */
+ PgAioHandle *handed_out_io;
+
+ /* Are we currently in batchmode? See pgaio_enter_batchmode(). */
+ bool in_batchmode;
+
+ /*
+ * IOs that are defined, but not yet submitted.
+ */
+ uint16 num_staged_ios;
+ PgAioHandle *staged_ios[PGAIO_SUBMIT_BATCH_SIZE];
+
+ /*
+ * List of in-flight IOs. Also contains IOs that aren't strictly speaking
+ * in-flight anymore, but have been waited-for and completed by another
+ * backend. Once this backend sees such an IO it'll be reclaimed.
+ *
+ * The list is ordered by submission time, with more recently submitted
+ * IOs being appended at the end.
+ */
+ dclist_head in_flight_ios;
+} PgAioBackend;
+
+
+typedef struct PgAioCtl
+{
+ int backend_state_count;
+ PgAioBackend *backend_state;
+
+ /*
+ * Array of iovec structs. Each iovec is owned by a specific backend. The
+ * allocation is in PgAioCtl to allow the maximum number of iovecs for
+ * individual IOs to be configurable with PGC_POSTMASTER GUC.
+ */
+ uint32 iovec_count;
+ struct iovec *iovecs;
+
+ /*
+ * For, e.g., an IO covering multiple buffers in shared / temp buffers, we
+ * need to get Buffer IDs during completion to be able to change the
+ * BufferDesc state accordingly. This space can be used to store e.g.
+ * Buffer IDs. Note that the actual iovec might be shorter than this,
+ * because we combine neighboring pages into one larger iovec entry.
+ */
+ uint64 *handle_data;
+
+ uint32 io_handle_count;
+ PgAioHandle *io_handles;
+} PgAioCtl;
+
+
+
+/*
+ * Callbacks used to implement an IO method.
+ */
+typedef struct IoMethodOps
+{
+ /* global initialization */
+
+ /*
+ * Amount of additional shared memory to reserve for the io_method. Called
+ * just like a normal ipci.c style *Size() function. Optional.
+ */
+ size_t (*shmem_size) (void);
+
+ /*
+ * Initialize shared memory. First time is true if AIO's shared memory was
+ * just initialized, false otherwise. Optional.
+ */
+ void (*shmem_init) (bool first_time);
+
+ /*
+ * Per-backend initialization. Optional.
+ */
+ void (*init_backend) (void);
+
+
+ /* handling of IOs */
+
+ /* optional */
+ bool (*needs_synchronous_execution) (PgAioHandle *ioh);
+
+ /*
+ * Start executing passed in IOs.
+ *
+ * Will not be called if ->needs_synchronous_execution() returned true.
+ *
+ * num_staged_ios is <= PGAIO_SUBMIT_BATCH_SIZE.
+ *
+ * Always called in a critical section.
+ */
+ int (*submit) (uint16 num_staged_ios, PgAioHandle **staged_ios);
+
+ /*
+ * Wait for the IO to complete. Optional.
+ *
+ * If not provided, it needs to be guaranteed that the IO method calls
+ * pgaio_io_process_completion() without further interaction by the
+ * issuing backend.
+ */
+ void (*wait_one) (PgAioHandle *ioh,
+ uint64 ref_generation);
+} IoMethodOps;
+
+
+/* aio.c */
+extern bool pgaio_io_was_recycled(PgAioHandle *ioh, uint64 ref_generation, PgAioHandleState *state);
+extern void pgaio_io_stage(PgAioHandle *ioh, PgAioOp op);
+extern void pgaio_io_process_completion(PgAioHandle *ioh, int result);
+extern void pgaio_io_prepare_submit(PgAioHandle *ioh);
+extern bool pgaio_io_needs_synchronous_execution(PgAioHandle *ioh);
+extern const char *pgaio_io_get_state_name(PgAioHandle *ioh);
+const char *pgaio_result_status_string(PgAioResultStatus rs);
+extern void pgaio_shutdown(int code, Datum arg);
+
+/* aio_callback.c */
+extern void pgaio_io_call_stage(PgAioHandle *ioh);
+extern void pgaio_io_call_complete_shared(PgAioHandle *ioh);
+extern void pgaio_io_call_complete_local(PgAioHandle *ioh);
+
+/* aio_io.c */
+extern void pgaio_io_perform_synchronously(PgAioHandle *ioh);
+extern const char *pgaio_io_get_op_name(PgAioHandle *ioh);
+
+/* aio_target.c */
+extern bool pgaio_io_can_reopen(PgAioHandle *ioh);
+extern void pgaio_io_reopen(PgAioHandle *ioh);
+extern const char *pgaio_io_get_target_name(PgAioHandle *ioh);
+
+
+/*
+ * The AIO subsystem has fairly verbose debug logging support. This can be
+ * enabled/disabled at build time. The reason for this is that
+ * a) the verbosity can make debugging things on higher levels hard
+ * b) even if logging can be skipped due to elevel checks, it still causes a
+ * measurable slowdown
+ *
+ * XXX: This likely should be eventually be disabled by default, at least in
+ * non-assert builds.
+ */
+#define PGAIO_VERBOSE 1
+
+/*
+ * Simple ereport() wrapper that only logs if PGAIO_VERBOSE is defined.
+ *
+ * This intentionally still compiles the code, guarded by a constant if (0),
+ * if verbose logging is disabled, to make it less likely that debug logging
+ * is silently broken.
+ *
+ * The current definition requires passing at least one argument.
+ */
+#define pgaio_debug(elevel, msg, ...) \
+ do { \
+ if (PGAIO_VERBOSE) \
+ ereport(elevel, \
+ errhidestmt(true), errhidecontext(true), \
+ errmsg_internal(msg, \
+ __VA_ARGS__)); \
+ } while(0)
+
+/*
+ * Simple ereport() wrapper. Note that the definition requires passing at
+ * least one argument.
+ */
+#define pgaio_debug_io(elevel, ioh, msg, ...) \
+ pgaio_debug(elevel, "io %-10d|op %-5s|target %-4s|state %-16s: " msg, \
+ pgaio_io_get_id(ioh), \
+ pgaio_io_get_op_name(ioh), \
+ pgaio_io_get_target_name(ioh), \
+ pgaio_io_get_state_name(ioh), \
+ __VA_ARGS__)
+
+
+#ifdef USE_INJECTION_POINTS
+
+extern void pgaio_io_call_inj(PgAioHandle *ioh, const char *injection_point);
+
+/* just for use in tests, from within injection points */
+extern PgAioHandle *pgaio_inj_io_get(void);
+
+#else
+
+#define pgaio_io_call_inj(ioh, injection_point) (void) 0
+
+/*
+ * no fallback for pgaio_inj_io_get, all code using injection points better be
+ * guarded by USE_INJECTION_POINTS.
+ */
+
+#endif
+
+
+/* Declarations for the tables of function pointers exposed by each IO method. */
+extern PGDLLIMPORT const IoMethodOps pgaio_sync_ops;
+
+extern PGDLLIMPORT const IoMethodOps *pgaio_method_ops;
+extern PGDLLIMPORT PgAioCtl *pgaio_ctl;
+extern PGDLLIMPORT PgAioBackend *pgaio_my_backend;
+
+
+
+#endif /* AIO_INTERNAL_H */
diff --git a/src/include/storage/aio_types.h b/src/include/storage/aio_types.h
new file mode 100644
index 00000000000..a5cc658efbd
--- /dev/null
+++ b/src/include/storage/aio_types.h
@@ -0,0 +1,117 @@
+/*-------------------------------------------------------------------------
+ *
+ * aio_types.h
+ * AIO related types that are useful to include separately, to reduce the
+ * "include burden".
+ *
+ *
+ * Portions Copyright (c) 1996-2025, PostgreSQL Global Development Group
+ * Portions Copyright (c) 1994, Regents of the University of California
+ *
+ * src/include/storage/aio_types.h
+ *
+ *-------------------------------------------------------------------------
+ */
+#ifndef AIO_TYPES_H
+#define AIO_TYPES_H
+
+#include "storage/block.h"
+#include "storage/relfilelocator.h"
+
+
+typedef struct PgAioHandle PgAioHandle;
+typedef struct PgAioHandleCallbacks PgAioHandleCallbacks;
+typedef struct PgAioTargetInfo PgAioTargetInfo;
+
+/*
+ * A reference to an IO that can be used to wait for the IO (using
+ * pgaio_wref_wait()) to complete.
+ *
+ * These can be passed across process boundaries.
+ */
+typedef struct PgAioWaitRef
+{
+ /* internal ID identifying the specific PgAioHandle */
+ uint32 aio_index;
+
+ /*
+ * IO handles are reused. To detect if a handle was reused, and thereby
+ * avoid unnecessarily waiting for a newer IO, each time the handle is
+ * reused a generation number is increased.
+ *
+ * To avoid requiring alignment sufficient for an int64, split the
+ * generation into two.
+ */
+ uint32 generation_upper;
+ uint32 generation_lower;
+} PgAioWaitRef;
+
+
+/*
+ * Information identifying what the IO is being performed on.
+ *
+ * This needs sufficient information to
+ *
+ * a) Reopen the file for the IO if the IO is executed in a context that
+ * cannot use the FD provided initially (e.g. because the IO is executed in
+ * a worker process).
+ *
+ * b) Describe the object the IO is performed on in log / error messages.
+ */
+typedef union PgAioTargetData
+{
+ /* just as an example placeholder for later */
+ struct
+ {
+ uint32 queue_id;
+ } wal;
+} PgAioTargetData;
+
+
+/*
+ * The status of an AIO operation.
+ */
+typedef enum PgAioResultStatus
+{
+ ARS_UNKNOWN, /* not yet completed / uninitialized */
+ ARS_OK,
+ ARS_PARTIAL, /* did not fully succeed, but no error */
+ ARS_ERROR,
+} PgAioResultStatus;
+
+
+/*
+ * Result of IO operation, visible only to the initiator of IO.
+ */
+typedef struct PgAioResult
+{
+ /*
+ * This is of type PgAioHandleCallbackID, but can't use a bitfield of an
+ * enum, because some compilers treat enums as signed.
+ */
+ uint32 id:8;
+
+ /* of type PgAioResultStatus, see above */
+ uint32 status:2;
+
+ /* meaning defined by callback->error */
+ uint32 error_data:22;
+
+ int32 result;
+} PgAioResult;
+
+
+/*
+ * Combination of PgAioResult with minimal metadata about the IO.
+ *
+ * Contains sufficient information to be able, in case the IO [partially]
+ * fails, to log/raise an error under control of the IO issuing code.
+ */
+typedef struct PgAioReturn
+{
+ PgAioResult result;
+ PgAioTargetData target_data;
+} PgAioReturn;
+
+
+#endif /* AIO_TYPES_H */
diff --git a/src/backend/storage/aio/Makefile b/src/backend/storage/aio/Makefile
index eaeaeeee8e3..89f821ea7e1 100644
--- a/src/backend/storage/aio/Makefile
+++ b/src/backend/storage/aio/Makefile
@@ -10,7 +10,11 @@ include $(top_builddir)/src/Makefile.global
OBJS = \
aio.o \
+ aio_callback.o \
aio_init.o \
+ aio_io.o \
+ aio_target.o \
+ method_sync.o \
read_stream.o
include $(top_srcdir)/src/backend/common.mk
diff --git a/src/backend/storage/aio/aio.c b/src/backend/storage/aio/aio.c
index 828a94efdc3..4d5439c73fd 100644
--- a/src/backend/storage/aio/aio.c
+++ b/src/backend/storage/aio/aio.c
@@ -3,6 +3,28 @@
* aio.c
* AIO - Core Logic
*
+ * For documentation about how AIO works on a higher level, including a
+ * schematic example, see README.md.
+ *
+ *
+ * AIO is a complicated subsystem. To keep things navigable, it is split
+ * across a number of files:
+ *
+ * - method_*.c - different ways of executing AIO (e.g. worker process)
+ *
+ * - aio_target.c - IO on different kinds of targets
+ *
+ * - aio_io.c - method-independent code for specific IO ops (e.g. readv)
+ *
+ * - aio_callback.c - callbacks at IO operation lifecycle events
+ *
+ * - aio_init.c - per-server and per-backend initialization
+ *
+ * - aio.c - all other topics
+ *
+ * - read_stream.c - helper for reading buffered relation data
+ *
+ *
* Portions Copyright (c) 1996-2025, PostgreSQL Global Development Group
* Portions Copyright (c) 1994, Regents of the University of California
*
@@ -15,10 +37,28 @@
#include "postgres.h"
#include "lib/ilist.h"
+#include "miscadmin.h"
+#include "port/atomics.h"
#include "storage/aio.h"
+#include "storage/aio_internal.h"
#include "storage/aio_subsys.h"
#include "utils/guc.h"
#include "utils/guc_hooks.h"
+#include "utils/resowner.h"
+#include "utils/wait_event_types.h"
+
+#ifdef USE_INJECTION_POINTS
+#include "utils/injection_point.h"
+#endif
+
+
+static inline void pgaio_io_update_state(PgAioHandle *ioh, PgAioHandleState new_state);
+static void pgaio_io_reclaim(PgAioHandle *ioh);
+static void pgaio_io_resowner_register(PgAioHandle *ioh);
+static void pgaio_io_wait_for_free(void);
+static PgAioHandle *pgaio_io_from_wref(PgAioWaitRef *iow, uint64 *ref_generation);
+static const char *pgaio_io_state_get_name(PgAioHandleState s);
+static void pgaio_io_wait(PgAioHandle *ioh, uint64 ref_generation);
/* Options for io_method. */
@@ -31,7 +71,179 @@ const struct config_enum_entry io_method_options[] = {
int io_method = DEFAULT_IO_METHOD;
int io_max_concurrency = -1;
+/* global control for AIO */
+PgAioCtl *pgaio_ctl;
+/* current backend's per-backend state */
+PgAioBackend *pgaio_my_backend;
+
+
+static const IoMethodOps *const pgaio_method_ops_table[] = {
+ [IOMETHOD_SYNC] = &pgaio_sync_ops,
+};
+
+/* callbacks for the configured io_method, set by assign_io_method */
+const IoMethodOps *pgaio_method_ops;
+
+
+/*
+ * Currently there's no infrastructure to pass arguments to injection points,
+ * so we instead set this up for the duration of the injection point
+ * invocation. See pgaio_io_call_inj().
+ */
+#ifdef USE_INJECTION_POINTS
+static PgAioHandle *pgaio_inj_cur_handle;
+#endif
+
+
+
+/* --------------------------------------------------------------------------------
+ * Public Functions related to PgAioHandle
+ * --------------------------------------------------------------------------------
+ */
+
+/*
+ * Acquire an AioHandle, waiting for IO completion if necessary.
+ *
+ * Each backend can only have one AIO handle that has been "handed out" to
+ * code, but not yet submitted or released. This restriction is necessary to
+ * ensure that it is possible for code to wait for an unused handle by waiting
+ * for in-flight IO to complete. There is a limited number of handles in each
+ * backend, if multiple handles could be handed out without being submitted,
+ * waiting for all in-flight IO to complete would not guarantee that handles
+ * free up.
+ *
+ * It is cheap to acquire an IO handle, unless all handles are in use. In that
+ * case this function waits for the oldest IO to complete. If that is not
+ * desirable, use pgaio_io_acquire_nb().
+ *
+ * If a handle was acquired but then does not turn out to be needed,
+ * e.g. because pgaio_io_acquire() is called before starting an IO in a
+ * critical section, the handle needs to be released with pgaio_io_release().
+ *
+ *
+ * To react to the completion of the IO as soon as it is known to have
+ * completed, callbacks can be registered with pgaio_io_register_callbacks().
+ *
+ * To actually execute IO using the returned handle, the pgaio_io_prep_*()
+ * family of functions is used. In many cases the pgaio_io_prep_*() call will
+ * not be done directly by code that acquired the handle, but by lower level
+ * code that gets passed the handle. E.g. if code in bufmgr.c wants to perform
+ * AIO, it typically will pass the handle to smgr.c, which will pass it on to
+ * md.c, on to fd.c, which then finally calls pgaio_io_prep_*(). This
+ * forwarding allows the various layers to react to the IO's completion by
+ * registering callbacks. These callbacks in turn can translate a lower
+ * layer's result into a result understandable by a higher layer.
+ *
+ * During pgaio_io_prep_*() the IO is staged (i.e. prepared for execution but
+ * not submitted to the kernel). Unless in batchmode
+ * (c.f. pgaio_enter_batchmode()), the IO will also get submitted for
+ * execution. Note that, whether in batchmode or not, the IO might even
+ * complete before the functions return.
+ *
+ * After pgaio_io_prep_*() the AioHandle is "consumed" and may not be
+ * referenced by the IO issuing code. To e.g. wait for IO, references to the
+ * IO can be established with pgaio_io_get_wref() *before* pgaio_io_prep_*()
+ * is called. pgaio_wref_wait() can be used to wait for the IO to complete.
+ *
+ *
+ * To know if the IO [partially] succeeded or failed, a PgAioReturn * can be
+ * passed to pgaio_io_acquire(). Once the issuing backend has called
+ * pgaio_wref_wait(), the PgAioReturn contains information about whether the
+ * operation succeeded and details about the first failure, if any. The error
+ * can be raised / logged with pgaio_result_report().
+ *
+ * The lifetime of the memory pointed to be *ret needs to be at least as long
+ * as the passed in resowner. If the resowner releases resources before the IO
+ * completes (typically due to an error), the reference to *ret will be
+ * cleared. In case of resowner cleanup *ret will not be updated with the
+ * results of the IO operation.
+ */
+PgAioHandle *
+pgaio_io_acquire(struct ResourceOwnerData *resowner, PgAioReturn *ret)
+{
+ PgAioHandle *h;
+
+ while (true)
+ {
+ h = pgaio_io_acquire_nb(resowner, ret);
+
+ if (h != NULL)
+ return h;
+
+ /*
+ * Evidently all handles by this backend are in use. Just wait for
+ * some to complete.
+ */
+ pgaio_io_wait_for_free();
+ }
+}
+
+/*
+ * Acquire an AioHandle, returning NULL if no handles are free.
+ *
+ * See pgaio_io_acquire(). The only difference is that this function will return
+ * NULL if there are no idle handles, instead of blocking.
+ */
+PgAioHandle *
+pgaio_io_acquire_nb(struct ResourceOwnerData *resowner, PgAioReturn *ret)
+{
+ if (pgaio_my_backend->num_staged_ios >= PGAIO_SUBMIT_BATCH_SIZE)
+ {
+ Assert(pgaio_my_backend->num_staged_ios == PGAIO_SUBMIT_BATCH_SIZE);
+ pgaio_submit_staged();
+ }
+
+ if (pgaio_my_backend->handed_out_io)
+ elog(ERROR, "API violation: Only one IO can be handed out");
+
+ if (!dclist_is_empty(&pgaio_my_backend->idle_ios))
+ {
+ dlist_node *ion = dclist_pop_head_node(&pgaio_my_backend->idle_ios);
+ PgAioHandle *ioh = dclist_container(PgAioHandle, node, ion);
+
+ Assert(ioh->state == PGAIO_HS_IDLE);
+ Assert(ioh->owner_procno == MyProcNumber);
+
+ pgaio_io_update_state(ioh, PGAIO_HS_HANDED_OUT);
+ pgaio_my_backend->handed_out_io = ioh;
+
+ if (resowner)
+ pgaio_io_resowner_register(ioh);
+
+ if (ret)
+ {
+ ioh->report_return = ret;
+ ret->result.status = ARS_UNKNOWN;
+ }
+
+ return ioh;
+ }
+
+ return NULL;
+}
+
+/*
+ * Release IO handle that turned out to not be required.
+ *
+ * See pgaio_io_acquire() for more details.
+ */
+void
+pgaio_io_release(PgAioHandle *ioh)
+{
+ if (ioh == pgaio_my_backend->handed_out_io)
+ {
+ Assert(ioh->state == PGAIO_HS_HANDED_OUT);
+ Assert(ioh->resowner);
+
+ pgaio_my_backend->handed_out_io = NULL;
+ pgaio_io_reclaim(ioh);
+ }
+ else
+ {
+ elog(ERROR, "release in unexpected state");
+ }
+}
/*
* Release IO handle during resource owner cleanup.
@@ -39,8 +251,795 @@ int io_max_concurrency = -1;
void
pgaio_io_release_resowner(dlist_node *ioh_node, bool on_error)
{
+ PgAioHandle *ioh = dlist_container(PgAioHandle, resowner_node, ioh_node);
+
+ Assert(ioh->resowner);
+
+ ResourceOwnerForgetAioHandle(ioh->resowner, &ioh->resowner_node);
+ ioh->resowner = NULL;
+
+ switch (ioh->state)
+ {
+ case PGAIO_HS_IDLE:
+ elog(ERROR, "unexpected");
+ break;
+ case PGAIO_HS_HANDED_OUT:
+ Assert(ioh == pgaio_my_backend->handed_out_io || pgaio_my_backend->handed_out_io == NULL);
+
+ if (ioh == pgaio_my_backend->handed_out_io)
+ {
+ pgaio_my_backend->handed_out_io = NULL;
+ if (!on_error)
+ elog(WARNING, "leaked AIO handle");
+ }
+
+ pgaio_io_reclaim(ioh);
+ break;
+ case PGAIO_HS_DEFINED:
+ case PGAIO_HS_STAGED:
+ if (!on_error)
+ elog(WARNING, "AIO handle was not submitted");
+ pgaio_submit_staged();
+ break;
+ case PGAIO_HS_SUBMITTED:
+ case PGAIO_HS_COMPLETED_IO:
+ case PGAIO_HS_COMPLETED_SHARED:
+ case PGAIO_HS_COMPLETED_LOCAL:
+ /* this is expected to happen */
+ break;
+ }
+
+ /*
+ * Need to unregister the reporting of the IO's result, the memory it's
+ * referencing likely has gone away.
+ */
+ if (ioh->report_return)
+ ioh->report_return = NULL;
}
+/*
+ * Add a [set of] flags to the IO.
+ *
+ * Note that this combines flags with already set flags, rather than set flags
+ * to explicitly the passed in parameters. This is to allow multiple callsites
+ * to set flags.
+ */
+void
+pgaio_io_set_flag(PgAioHandle *ioh, PgAioHandleFlags flag)
+{
+ Assert(ioh->state == PGAIO_HS_HANDED_OUT);
+
+ ioh->flags |= flag;
+}
+
+/*
+ * Returns an ID uniquely identifying the IO handle. This is only really
+ * useful for logging, as handles are reused across multiple IOs.
+ */
+int
+pgaio_io_get_id(PgAioHandle *ioh)
+{
+ Assert(ioh >= pgaio_ctl->io_handles &&
+ ioh < (pgaio_ctl->io_handles + pgaio_ctl->io_handle_count));
+ return ioh - pgaio_ctl->io_handles;
+}
+
+/*
+ * Return the ProcNumber for the process that can use an IO handle. The
+ * mapping from IO handles to PGPROCs is static, therefore this even works
+ * when the corresponding PGPROC is not in use.
+ */
+ProcNumber
+pgaio_io_get_owner(PgAioHandle *ioh)
+{
+ return ioh->owner_procno;
+}
+
+/*
+ * Return a wait reference for the IO. Only wait references can be used to
+ * wait for an IOs completion, as handles themselves can be reused after
+ * completion. See also the comment above pgaio_io_acquire().
+ */
+void
+pgaio_io_get_wref(PgAioHandle *ioh, PgAioWaitRef *iow)
+{
+ Assert(ioh->state == PGAIO_HS_HANDED_OUT ||
+ ioh->state == PGAIO_HS_DEFINED ||
+ ioh->state == PGAIO_HS_STAGED);
+ Assert(ioh->generation != 0);
+
+ iow->aio_index = ioh - pgaio_ctl->io_handles;
+ iow->generation_upper = (uint32) (ioh->generation >> 32);
+ iow->generation_lower = (uint32) ioh->generation;
+}
+
+
+
+/* --------------------------------------------------------------------------------
+ * Internal Functions related to PgAioHandle
+ * --------------------------------------------------------------------------------
+ */
+
+static inline void
+pgaio_io_update_state(PgAioHandle *ioh, PgAioHandleState new_state)
+{
+ pgaio_debug_io(DEBUG5, ioh,
+ "updating state to %s",
+ pgaio_io_state_get_name(new_state));
+
+ /*
+ * Ensure the changes signified by the new state are visible before the
+ * new state becomes visible.
+ */
+ pg_write_barrier();
+
+ ioh->state = new_state;
+}
+
+static void
+pgaio_io_resowner_register(PgAioHandle *ioh)
+{
+ Assert(!ioh->resowner);
+ Assert(CurrentResourceOwner);
+
+ ResourceOwnerRememberAioHandle(CurrentResourceOwner, &ioh->resowner_node);
+ ioh->resowner = CurrentResourceOwner;
+}
+
+/*
+ * Stage IO for execution and, if appropriate, submit it immediately.
+ *
+ * Should only be called from pgaio_io_prep_*().
+ */
+void
+pgaio_io_stage(PgAioHandle *ioh, PgAioOp op)
+{
+ bool needs_synchronous;
+
+ Assert(ioh->state == PGAIO_HS_HANDED_OUT);
+ Assert(pgaio_my_backend->handed_out_io == ioh);
+ Assert(pgaio_io_has_target(ioh));
+
+ ioh->op = op;
+ ioh->result = 0;
+
+ pgaio_io_update_state(ioh, PGAIO_HS_DEFINED);
+
+ /* allow a new IO to be staged */
+ pgaio_my_backend->handed_out_io = NULL;
+
+ pgaio_io_call_stage(ioh);
+
+ pgaio_io_update_state(ioh, PGAIO_HS_STAGED);
+
+ /*
+ * Synchronous execution has to be executed, well, synchronously, so check
+ * that first.
+ */
+ needs_synchronous = pgaio_io_needs_synchronous_execution(ioh);
+
+ pgaio_debug_io(DEBUG3, ioh,
+ "prepared (synchronous: %d, in_batch: %d)",
+ needs_synchronous, pgaio_my_backend->in_batchmode);
+
+ if (!needs_synchronous)
+ {
+ pgaio_my_backend->staged_ios[pgaio_my_backend->num_staged_ios++] = ioh;
+ Assert(pgaio_my_backend->num_staged_ios <= PGAIO_SUBMIT_BATCH_SIZE);
+
+ /*
+ * Unless code explicitly opted into batching IOs, submit the IO
+ * immediately.
+ */
+ if (!pgaio_my_backend->in_batchmode)
+ pgaio_submit_staged();
+ }
+ else
+ {
+ pgaio_io_prepare_submit(ioh);
+ pgaio_io_perform_synchronously(ioh);
+ }
+}
+
+bool
+pgaio_io_needs_synchronous_execution(PgAioHandle *ioh)
+{
+ /*
+ * If the caller said to execute the IO synchronously, do so.
+ *
+ * XXX: We could optimize the logic when to execute synchronously by first
+ * checking if there are other IOs in flight and only synchronously
+ * executing if not. Unclear whether that'll be sufficiently common to be
+ * worth worrying about.
+ */
+ if (ioh->flags & PGAIO_HF_SYNCHRONOUS)
+ return true;
+
+ /* Check if the IO method requires synchronous execution of IO */
+ if (pgaio_method_ops->needs_synchronous_execution)
+ return pgaio_method_ops->needs_synchronous_execution(ioh);
+
+ return false;
+}
+
+/*
+ * Handle IO being processed by IO method.
+ *
+ * Should be called by IO methods / synchronous IO execution, just before the
+ * IO is performed.
+ */
+void
+pgaio_io_prepare_submit(PgAioHandle *ioh)
+{
+ pgaio_io_update_state(ioh, PGAIO_HS_SUBMITTED);
+
+ dclist_push_tail(&pgaio_my_backend->in_flight_ios, &ioh->node);
+}
+
+/*
+ * Handle IO getting completed by a method.
+ *
+ * Should be called by IO methods / synchronous IO execution, just after the
+ * IO has been performed.
+ *
+ * Expects to be called in a critical section. We expect IOs to be usable for
+ * WAL etc, which requires being able to execute completion callbacks in a
+ * critical section.
+ */
+void
+pgaio_io_process_completion(PgAioHandle *ioh, int result)
+{
+ Assert(ioh->state == PGAIO_HS_SUBMITTED);
+
+ Assert(CritSectionCount > 0);
+
+ ioh->result = result;
+
+ pgaio_io_update_state(ioh, PGAIO_HS_COMPLETED_IO);
+
+ pgaio_io_call_inj(ioh, "AIO_PROCESS_COMPLETION_BEFORE_SHARED");
+
+ pgaio_io_call_complete_shared(ioh);
+
+ pgaio_io_update_state(ioh, PGAIO_HS_COMPLETED_SHARED);
+
+ /* condition variable broadcast ensures state is visible before wakeup */
+ ConditionVariableBroadcast(&ioh->cv);
+
+ /* contains call to pgaio_io_call_complete_local() */
+ if (ioh->owner_procno == MyProcNumber)
+ pgaio_io_reclaim(ioh);
+}
+
+/*
+ * Has the IO completed and thus the IO handle been reused?
+ *
+ * This is useful when waiting for IO completion at a low level (e.g. in an IO
+ * method's ->wait_one() callback).
+ */
+bool
+pgaio_io_was_recycled(PgAioHandle *ioh, uint64 ref_generation, PgAioHandleState *state)
+{
+ *state = ioh->state;
+ pg_read_barrier();
+
+ return ioh->generation != ref_generation;
+}
+
+/*
+ * Wait for IO to complete. External code should never use this, outside of
+ * the AIO subsystem waits are only allowed via pgaio_wref_wait().
+ */
+static void
+pgaio_io_wait(PgAioHandle *ioh, uint64 ref_generation)
+{
+ PgAioHandleState state;
+ bool am_owner;
+
+ am_owner = ioh->owner_procno == MyProcNumber;
+
+ if (pgaio_io_was_recycled(ioh, ref_generation, &state))
+ return;
+
+ if (am_owner)
+ {
+ if (state != PGAIO_HS_SUBMITTED
+ && state != PGAIO_HS_COMPLETED_IO
+ && state != PGAIO_HS_COMPLETED_SHARED
+ && state != PGAIO_HS_COMPLETED_LOCAL)
+ {
+ elog(PANIC, "waiting for own IO in wrong state: %d",
+ state);
+ }
+ }
+
+ while (true)
+ {
+ if (pgaio_io_was_recycled(ioh, ref_generation, &state))
+ return;
+
+ switch (state)
+ {
+ case PGAIO_HS_IDLE:
+ case PGAIO_HS_HANDED_OUT:
+ elog(ERROR, "IO in wrong state: %d", state);
+ break;
+
+ case PGAIO_HS_SUBMITTED:
+
+ /*
+ * If we need to wait via the IO method, do so now. Don't
+ * check via the IO method if the issuing backend is executing
+ * the IO synchronously.
+ */
+ if (pgaio_method_ops->wait_one && !(ioh->flags & PGAIO_HF_SYNCHRONOUS))
+ {
+ pgaio_method_ops->wait_one(ioh, ref_generation);
+ continue;
+ }
+ /* fallthrough */
+
+ /* waiting for owner to submit */
+ case PGAIO_HS_DEFINED:
+ case PGAIO_HS_STAGED:
+ /* waiting for reaper to complete */
+ /* fallthrough */
+ case PGAIO_HS_COMPLETED_IO:
+ /* shouldn't be able to hit this otherwise */
+ Assert(IsUnderPostmaster);
+ /* ensure we're going to get woken up */
+ ConditionVariablePrepareToSleep(&ioh->cv);
+
+ while (!pgaio_io_was_recycled(ioh, ref_generation, &state))
+ {
+ if (state == PGAIO_HS_COMPLETED_SHARED ||
+ state == PGAIO_HS_COMPLETED_LOCAL)
+ break;
+ ConditionVariableSleep(&ioh->cv, WAIT_EVENT_AIO_IO_COMPLETION);
+ }
+
+ ConditionVariableCancelSleep();
+ break;
+
+ case PGAIO_HS_COMPLETED_SHARED:
+ case PGAIO_HS_COMPLETED_LOCAL:
+ /* see above */
+ if (am_owner)
+ pgaio_io_reclaim(ioh);
+ return;
+ }
+ }
+}
+
+/*
+ * Make IO handle ready to be reused after IO has completed or after the
+ * handle has been released without being used.
+ */
+static void
+pgaio_io_reclaim(PgAioHandle *ioh)
+{
+ /* This is only ok if it's our IO */
+ Assert(ioh->owner_procno == MyProcNumber);
+ Assert(ioh->state != PGAIO_HS_IDLE);
+
+ /*
+ * It's a bit ugly, but right now the easiest place to put the execution
+ * of shared completion callbacks is this function, as we need to execute
+ * local callbacks just before reclaiming at multiple callsites.
+ */
+ if (ioh->state == PGAIO_HS_COMPLETED_SHARED)
+ {
+ pgaio_io_call_complete_local(ioh);
+ pgaio_io_update_state(ioh, PGAIO_HS_COMPLETED_LOCAL);
+ }
+
+ pgaio_debug_io(DEBUG4, ioh,
+ "reclaiming: distilled_result: (status %s, id %u, error_data %d), raw_result: %d",
+ pgaio_result_status_string(ioh->distilled_result.status),
+ ioh->distilled_result.id,
+ ioh->distilled_result.error_data,
+ ioh->result);
+
+ /* if the IO has been defined, we might need to do more work */
+ if (ioh->state != PGAIO_HS_HANDED_OUT)
+ {
+ dclist_delete_from(&pgaio_my_backend->in_flight_ios, &ioh->node);
+
+ if (ioh->report_return)
+ {
+ ioh->report_return->result = ioh->distilled_result;
+ ioh->report_return->target_data = ioh->target_data;
+ }
+ }
+
+ if (ioh->resowner)
+ {
+ ResourceOwnerForgetAioHandle(ioh->resowner, &ioh->resowner_node);
+ ioh->resowner = NULL;
+ }
+
+ Assert(!ioh->resowner);
+
+ ioh->op = PGAIO_OP_INVALID;
+ ioh->target = PGAIO_TID_INVALID;
+ ioh->flags = 0;
+ ioh->num_callbacks = 0;
+ ioh->handle_data_len = 0;
+ ioh->report_return = NULL;
+ ioh->result = 0;
+ ioh->distilled_result.status = ARS_UNKNOWN;
+
+ /* XXX: the barrier is probably superfluous */
+ pg_write_barrier();
+ ioh->generation++;
+
+ pgaio_io_update_state(ioh, PGAIO_HS_IDLE);
+
+ /*
+ * We push the IO to the head of the idle IO list, that seems more cache
+ * efficient in cases where only a few IOs are used.
+ */
+ dclist_push_head(&pgaio_my_backend->idle_ios, &ioh->node);
+}
+
+/*
+ * Wait for an IO handle to become usable.
+ *
+ * This only really is useful for pgaio_io_acquire().
+ */
+static void
+pgaio_io_wait_for_free(void)
+{
+ int reclaimed = 0;
+
+ pgaio_debug(DEBUG2, "waiting for self with %d pending",
+ pgaio_my_backend->num_staged_ios);
+
+ /*
+ * First check if any of our IOs actually have completed - when using
+ * worker, that'll often be the case. We could do so as part of the loop
+ * below, but that'd potentially lead us to wait for some IO submitted
+ * before.
+ */
+ for (int i = 0; i < io_max_concurrency; i++)
+ {
+ PgAioHandle *ioh = &pgaio_ctl->io_handles[pgaio_my_backend->io_handle_off + i];
+
+ if (ioh->state == PGAIO_HS_COMPLETED_SHARED)
+ {
+ pgaio_io_reclaim(ioh);
+ reclaimed++;
+ }
+ }
+
+ if (reclaimed > 0)
+ return;
+
+ /*
+ * If we have any unsubmitted IOs, submit them now. We'll start waiting in
+ * a second, so it's better they're in flight. This also addresses the
+ * edge-case that all IOs are unsubmitted.
+ */
+ if (pgaio_my_backend->num_staged_ios > 0)
+ pgaio_submit_staged();
+
+ if (dclist_count(&pgaio_my_backend->in_flight_ios) == 0)
+ elog(ERROR, "no free IOs despite no in-flight IOs");
+
+ /*
+ * Wait for the oldest in-flight IO to complete.
+ *
+ * XXX: Reusing the general IO wait is suboptimal, we don't need to wait
+ * for that specific IO to complete, we just need *any* IO to complete.
+ */
+ {
+ PgAioHandle *ioh = dclist_head_element(PgAioHandle, node,
+ &pgaio_my_backend->in_flight_ios);
+
+ switch (ioh->state)
+ {
+ /* should not be in in-flight list */
+ case PGAIO_HS_IDLE:
+ case PGAIO_HS_DEFINED:
+ case PGAIO_HS_HANDED_OUT:
+ case PGAIO_HS_STAGED:
+ case PGAIO_HS_COMPLETED_LOCAL:
+ elog(ERROR, "shouldn't get here with io:%d in state %d",
+ pgaio_io_get_id(ioh), ioh->state);
+ break;
+
+ case PGAIO_HS_COMPLETED_IO:
+ case PGAIO_HS_SUBMITTED:
+ pgaio_debug_io(DEBUG2, ioh,
+ "waiting for free io with %d in flight",
+ dclist_count(&pgaio_my_backend->in_flight_ios));
+
+ /*
+ * In a more general case this would be racy, because the
+ * generation could increase after we read ioh->state above.
+ * But we are only looking at IOs by the current backend and
+ * the IO can only be recycled by this backend.
+ */
+ pgaio_io_wait(ioh, ioh->generation);
+ break;
+
+ case PGAIO_HS_COMPLETED_SHARED:
+ /* it's possible that another backend just finished this IO */
+ pgaio_io_reclaim(ioh);
+ break;
+ }
+
+ if (dclist_count(&pgaio_my_backend->idle_ios) == 0)
+ elog(PANIC, "no idle IO after waiting for IO to terminate");
+ return;
+ }
+}
+
+/*
+ * Internal - code outside of AIO should never need this and it'd be hard for
+ * such code to be safe.
+ */
+static PgAioHandle *
+pgaio_io_from_wref(PgAioWaitRef *iow, uint64 *ref_generation)
+{
+ PgAioHandle *ioh;
+
+ Assert(iow->aio_index < pgaio_ctl->io_handle_count);
+
+ ioh = &pgaio_ctl->io_handles[iow->aio_index];
+
+ *ref_generation = ((uint64) iow->generation_upper) << 32 |
+ iow->generation_lower;
+
+ Assert(*ref_generation != 0);
+
+ return ioh;
+}
+
+static const char *
+pgaio_io_state_get_name(PgAioHandleState s)
+{
+#define PGAIO_HS_TOSTR_CASE(sym) case PGAIO_HS_##sym: return #sym
+ switch (s)
+ {
+ PGAIO_HS_TOSTR_CASE(IDLE);
+ PGAIO_HS_TOSTR_CASE(HANDED_OUT);
+ PGAIO_HS_TOSTR_CASE(DEFINED);
+ PGAIO_HS_TOSTR_CASE(STAGED);
+ PGAIO_HS_TOSTR_CASE(SUBMITTED);
+ PGAIO_HS_TOSTR_CASE(COMPLETED_IO);
+ PGAIO_HS_TOSTR_CASE(COMPLETED_SHARED);
+ PGAIO_HS_TOSTR_CASE(COMPLETED_LOCAL);
+ }
+#undef PGAIO_HS_TOSTR_CASE
+
+ return NULL; /* silence compiler */
+}
+
+const char *
+pgaio_io_get_state_name(PgAioHandle *ioh)
+{
+ return pgaio_io_state_get_name(ioh->state);
+}
+
+const char *
+pgaio_result_status_string(PgAioResultStatus rs)
+{
+ switch (rs)
+ {
+ case ARS_UNKNOWN:
+ return "UNKNOWN";
+ case ARS_OK:
+ return "OK";
+ case ARS_PARTIAL:
+ return "PARTIAL";
+ case ARS_ERROR:
+ return "ERROR";
+ }
+
+ return NULL; /* silence compiler */
+}
+
+
+
+/* --------------------------------------------------------------------------------
+ * Functions primarily related to IO Wait References
+ * --------------------------------------------------------------------------------
+ */
+
+/*
+ * Mark a wait reference as invalid
+ */
+void
+pgaio_wref_clear(PgAioWaitRef *iow)
+{
+ iow->aio_index = PG_UINT32_MAX;
+}
+
+/* Is the wait reference valid? */
+bool
+pgaio_wref_valid(PgAioWaitRef *iow)
+{
+ return iow->aio_index != PG_UINT32_MAX;
+}
+
+/*
+ * Similar to pgaio_io_get_id(), just for wait references.
+ */
+int
+pgaio_wref_get_id(PgAioWaitRef *iow)
+{
+ Assert(pgaio_wref_valid(iow));
+ return iow->aio_index;
+}
+
+/*
+ * Wait for the IO to have completed. Can be called in any process, not just
+ * in the issuing backend.
+ */
+void
+pgaio_wref_wait(PgAioWaitRef *iow)
+{
+ uint64 ref_generation;
+ PgAioHandle *ioh;
+
+ ioh = pgaio_io_from_wref(iow, &ref_generation);
+
+ pgaio_io_wait(ioh, ref_generation);
+}
+
+/*
+ * Check if the referenced IO completed, without blocking.
+ */
+bool
+pgaio_wref_check_done(PgAioWaitRef *iow)
+{
+ uint64 ref_generation;
+ PgAioHandleState state;
+ bool am_owner;
+ PgAioHandle *ioh;
+
+ ioh = pgaio_io_from_wref(iow, &ref_generation);
+
+ if (pgaio_io_was_recycled(ioh, ref_generation, &state))
+ return true;
+
+ if (state == PGAIO_HS_IDLE)
+ return true;
+
+ am_owner = ioh->owner_procno == MyProcNumber;
+
+ if (state == PGAIO_HS_COMPLETED_SHARED ||
+ state == PGAIO_HS_COMPLETED_LOCAL)
+ {
+ if (am_owner)
+ pgaio_io_reclaim(ioh);
+ return true;
+ }
+
+ /*
+ * XXX: It likely would be worth checking in with the io method, to give
+ * the IO method a chance to check if there are completion events queued.
+ */
+
+ return false;
+}
+
+
+
+/* --------------------------------------------------------------------------------
+ * Actions on multiple IOs.
+ * --------------------------------------------------------------------------------
+ */
+
+/*
+ * Submit IOs in batches going forward.
+ *
+ * Submitting multiple IOs at once can be substantially faster than doing so
+ * one-by-one. At the same time, submitting multiple IOs at once requires more
+ * care to avoid deadlocks.
+ *
+ * Consider backend A staging an IO for buffer 1 and then trying to start IO
+ * on buffer 2, while backend B does the inverse. If A submitted the IO before
+ * moving on to buffer 2, this works just fine, B will wait for the IO to
+ * complete. But if batching were used, each backend will wait for IO that has
+ * not yet been submitted to complete, i.e. forever.
+ *
+ * End batch submission mode with pgaio_exit_batchmode(). (Throwing errors is
+ * allowed; error recovery will end the batch.)
+ *
+ * To avoid deadlocks, code needs to ensure that it will not wait for another
+ * backend while there is unsubmitted IO. E.g. by using conditional lock
+ * acquisition when acquiring buffer locks. To check if there currently are
+ * staged IOs, call pgaio_have_staged() and to submit all staged IOs call
+ * pgaio_submit_staged().
+ *
+ * It is not allowed to enter batchmode while already in batchmode, it's
+ * unlikely to ever be needed, as code needs to be explicitly aware of being
+ * called in batchmode, to avoid the deadlock risks explained above.
+ *
+ * Note that IOs may get submitted before pgaio_exit_batchmode() is called,
+ * e.g. because too many IOs have been staged or because pgaio_submit_staged()
+ * was called.
+ */
+void
+pgaio_enter_batchmode(void)
+{
+ if (pgaio_my_backend->in_batchmode)
+ elog(ERROR, "starting batch while batch already in progress");
+ pgaio_my_backend->in_batchmode = true;
+}
+
+/*
+ * Stop submitting IOs in batches.
+ */
+void
+pgaio_exit_batchmode(void)
+{
+ Assert(pgaio_my_backend->in_batchmode);
+
+ pgaio_submit_staged();
+ pgaio_my_backend->in_batchmode = false;
+}
+
+/*
+ * Are there staged but unsubmitted IOs?
+ *
+ * See comment above pgaio_enter_batchmode() for why code may need to check if
+ * there is IO in that state.
+ */
+bool
+pgaio_have_staged(void)
+{
+ Assert(pgaio_my_backend->in_batchmode ||
+ pgaio_my_backend->num_staged_ios == 0);
+ return pgaio_my_backend->num_staged_ios > 0;
+}
+
+/*
+ * Submit all staged but not yet submitted IOs.
+ *
+ * Unless in batch mode, this never needs to be called, as IOs get submitted
+ * as soon as possible. While in batchmode pgaio_submit_staged() can be called
+ * before waiting on another backend, to avoid the risk of deadlocks. See
+ * pgaio_enter_batchmode().
+ */
+void
+pgaio_submit_staged(void)
+{
+ int total_submitted = 0;
+ int did_submit;
+
+ if (pgaio_my_backend->num_staged_ios == 0)
+ return;
+
+
+ START_CRIT_SECTION();
+
+ did_submit = pgaio_method_ops->submit(pgaio_my_backend->num_staged_ios,
+ pgaio_my_backend->staged_ios);
+
+ END_CRIT_SECTION();
+
+ total_submitted += did_submit;
+
+ Assert(total_submitted == did_submit);
+
+ pgaio_my_backend->num_staged_ios = 0;
+
+ pgaio_debug(DEBUG4,
+ "aio: submitted %d IOs",
+ total_submitted);
+}
+
+
+
+/* --------------------------------------------------------------------------------
+ * Other
+ * --------------------------------------------------------------------------------
+ */
+
+
/*
* Perform AIO related cleanup after an error.
*
@@ -50,6 +1049,22 @@ pgaio_io_release_resowner(dlist_node *ioh_node, bool on_error)
void
pgaio_error_cleanup(void)
{
+ /*
+ * It is possible that code errored out after pgaio_enter_batchmode() but
+ * before pgaio_exit_batchmode() was called. In that case we need to
+ * submit the IO now.
+ */
+ if (pgaio_my_backend->in_batchmode)
+ {
+ pgaio_my_backend->in_batchmode = false;
+
+ pgaio_submit_staged();
+ }
+
+ /*
+ * As we aren't in batchmode, there shouldn't be any unsubmitted IOs.
+ */
+ Assert(pgaio_my_backend->num_staged_ios == 0);
}
/*
@@ -62,11 +1077,86 @@ pgaio_error_cleanup(void)
void
AtEOXact_Aio(bool is_commit)
{
+ /*
+ * We should never be in batch mode at transactional boundaries. In case
+ * an error was thrown while in batch mode, pgaio_error_cleanup() should
+ * have exited batchmode.
+ *
+ * In case we are in batchmode somehow, make sure to submit all staged
+ * IOs, other backends may need them to complete to continue.
+ */
+ if (pgaio_my_backend->in_batchmode)
+ {
+ pgaio_error_cleanup();
+ elog(WARNING, "open AIO batch at end of (sub-)transaction");
+ }
+
+ /*
+ * As we aren't in batchmode, there shouldn't be any unsubmitted IOs.
+ */
+ Assert(pgaio_my_backend->num_staged_ios == 0);
+}
+
+/*
+ * Need to submit staged but not yet submitted IOs using the fd, otherwise
+ * the IO would end up targeting something bogus.
+ */
+void
+pgaio_closing_fd(int fd)
+{
+ /*
+ * Might be called before AIO is initialized or in a subprocess that
+ * doesn't use AIO.
+ */
+ if (!pgaio_my_backend)
+ return;
+
+ /*
+ * For now just submit all staged IOs - we could be more selective, but
+ * it's probably not worth it.
+ */
+ pgaio_submit_staged();
+}
+
+/*
+ * Registered as before_shmem_exit() callback in pgaio_init_backend()
+ */
+void
+pgaio_shutdown(int code, Datum arg)
+{
+ Assert(pgaio_my_backend);
+ Assert(!pgaio_my_backend->handed_out_io);
+
+ /* first clean up resources as we would at a transaction boundary */
+ AtEOXact_Aio(code == 0);
+
+ /*
+ * Before exiting, make sure that all IOs are finished. That has two main
+ * purposes:
+ *
+ * - Some kernel-level AIO mechanisms don't deal well with the issuer of
+ * an AIO exiting before IO completed
+ *
+ * - It'd be confusing to see partially finished IOs in stats views etc
+ */
+ while (!dclist_is_empty(&pgaio_my_backend->in_flight_ios))
+ {
+ PgAioHandle *ioh = dclist_head_element(PgAioHandle, node, &pgaio_my_backend->in_flight_ios);
+
+ /* see comment in pgaio_io_wait_for_free() about raciness */
+ pgaio_io_wait(ioh, ioh->generation);
+ }
+
+ pgaio_my_backend = NULL;
}
void
assign_io_method(int newval, void *extra)
{
+ Assert(pgaio_method_ops_table[newval] != NULL);
+ Assert(newval < lengthof(io_method_options));
+
+ pgaio_method_ops = pgaio_method_ops_table[newval];
}
bool
@@ -88,3 +1178,43 @@ check_io_max_concurrency(int *newval, void **extra, GucSource source)
return true;
}
+
+
+
+/* --------------------------------------------------------------------------------
+ * Injection point support
+ * --------------------------------------------------------------------------------
+ */
+
+#ifdef USE_INJECTION_POINTS
+
+/*
+ * Call injection point with support for pgaio_inj_io_get().
+ */
+void
+pgaio_io_call_inj(PgAioHandle *ioh, const char *injection_point)
+{
+ pgaio_inj_cur_handle = ioh;
+
+ PG_TRY();
+ {
+ InjectionPointCached(injection_point);
+ }
+ PG_FINALLY();
+ {
+ pgaio_inj_cur_handle = NULL;
+ }
+ PG_END_TRY();
+}
+
+/*
+ * Return IO associated with injection point invocation. This is only needed
+ * as injection points currently don't support arguments.
+ */
+PgAioHandle *
+pgaio_inj_io_get(void)
+{
+ return pgaio_inj_cur_handle;
+}
+
+#endif
diff --git a/src/backend/storage/aio/aio_callback.c b/src/backend/storage/aio/aio_callback.c
new file mode 100644
index 00000000000..d5a2cca28f1
--- /dev/null
+++ b/src/backend/storage/aio/aio_callback.c
@@ -0,0 +1,308 @@
+/*-------------------------------------------------------------------------
+ *
+ * aio_callback.c
+ * AIO - Functionality related to callbacks that can be registered on IO
+ * Handles
+ *
+ * Portions Copyright (c) 1996-2025, PostgreSQL Global Development Group
+ * Portions Copyright (c) 1994, Regents of the University of California
+ *
+ * IDENTIFICATION
+ * src/backend/storage/aio/aio_callback.c
+ *
+ *-------------------------------------------------------------------------
+ */
+
+#include "postgres.h"
+
+#include "miscadmin.h"
+#include "storage/aio.h"
+#include "storage/aio_internal.h"
+
+
+/* just to have something to put into aio_handle_cbs */
+static const PgAioHandleCallbacks aio_invalid_cb = {0};
+
+typedef struct PgAioHandleCallbacksEntry
+{
+ const PgAioHandleCallbacks *const cb;
+ const char *const name;
+} PgAioHandleCallbacksEntry;
+
+/*
+ * Callback definition for the callbacks that can be registered on an IO
+ * handle. See PgAioHandleCallbackID's definition for an explanation for why
+ * callbacks are not identified by a pointer.
+ */
+static const PgAioHandleCallbacksEntry aio_handle_cbs[] = {
+#define CALLBACK_ENTRY(id, callback) [id] = {.cb = &callback, .name = #callback}
+ CALLBACK_ENTRY(PGAIO_HCB_INVALID, aio_invalid_cb),
+#undef CALLBACK_ENTRY
+};
+
+
+
+/* --------------------------------------------------------------------------------
+ * Public callback related functions operating on IO Handles
+ * --------------------------------------------------------------------------------
+ */
+
+/*
+ * Register callback for the IO handle.
+ *
+ * Only a limited number (PGAIO_HANDLE_MAX_CALLBACKS) of callbacks can be
+ * registered for each IO.
+ *
+ * Callbacks need to be registered before [indirectly] calling
+ * pgaio_io_prep_*(), as the IO may be executed immediately.
+ *
+ * A callback can be passed a small bit of data, e.g. to indicate whether to
+ * zero a buffer if it is invalid.
+ *
+ *
+ * Note that callbacks are executed in critical sections. This is necessary
+ * to be able to execute IO in critical sections (consider e.g. WAL
+ * logging). To perform AIO we first need to acquire a handle, which, if there
+ * are no free handles, requires waiting for IOs to complete and to execute
+ * their completion callbacks.
+ *
+ * Callbacks may be executed in the issuing backend but also in another
+ * backend (because that backend is waiting for the IO) or in IO workers (if
+ * io_method=worker is used).
+ *
+ *
+ * See PgAioHandleCallbackID's definition for an explanation for why
+ * callbacks are not identified by a pointer.
+ */
+void
+pgaio_io_register_callbacks(PgAioHandle *ioh, PgAioHandleCallbackID cb_id,
+ uint8 cb_data)
+{
+ const PgAioHandleCallbacksEntry *ce = &aio_handle_cbs[cb_id];
+
+ if (cb_id >= lengthof(aio_handle_cbs))
+ elog(ERROR, "callback %d is out of range", cb_id);
+ if (aio_handle_cbs[cb_id].cb->complete_shared == NULL &&
+ aio_handle_cbs[cb_id].cb->complete_local == NULL)
+ elog(ERROR, "callback %d does not have a completion callback", cb_id);
+ if (ioh->num_callbacks >= PGAIO_HANDLE_MAX_CALLBACKS)
+ elog(PANIC, "too many callbacks, the max is %d",
+ PGAIO_HANDLE_MAX_CALLBACKS);
+ ioh->callbacks[ioh->num_callbacks] = cb_id;
+ ioh->callbacks_data[ioh->num_callbacks] = cb_data;
+
+ pgaio_debug_io(DEBUG3, ioh,
+ "adding cb #%d, id %d/%s",
+ ioh->num_callbacks + 1,
+ cb_id, ce->name);
+
+ ioh->num_callbacks++;
+}
+
+/*
+ * Associate an array of data with the Handle. This is e.g. useful to the
+ * transport knowledge about which buffers a multi-block IO affects to
+ * completion callbacks.
+ *
+ * Right now this can be done only once for each IO, even though multiple
+ * callbacks can be registered. There aren't any known usecases requiring more
+ * and the required amount of shared memory does add up, so it doesn't seem
+ * worth multiplying memory usage by PGAIO_HANDLE_MAX_CALLBACKS.
+ */
+void
+pgaio_io_set_handle_data_64(PgAioHandle *ioh, uint64 *data, uint8 len)
+{
+ Assert(ioh->state == PGAIO_HS_HANDED_OUT);
+ Assert(ioh->handle_data_len == 0);
+ Assert(len <= PG_IOV_MAX);
+
+ for (int i = 0; i < len; i++)
+ pgaio_ctl->handle_data[ioh->iovec_off + i] = data[i];
+ ioh->handle_data_len = len;
+}
+
+/*
+ * Convenience version of pgaio_io_set_handle_data_64() that converts a 32bit
+ * array to a 64bit array. Without it callers would end up needing to
+ * open-code equivalent code.
+ */
+void
+pgaio_io_set_handle_data_32(PgAioHandle *ioh, uint32 *data, uint8 len)
+{
+ Assert(ioh->state == PGAIO_HS_HANDED_OUT);
+ Assert(ioh->handle_data_len == 0);
+ Assert(len <= PG_IOV_MAX);
+
+ for (int i = 0; i < len; i++)
+ pgaio_ctl->handle_data[ioh->iovec_off + i] = data[i];
+ ioh->handle_data_len = len;
+}
+
+/*
+ * Return data set with pgaio_io_set_handle_data_*().
+ */
+uint64 *
+pgaio_io_get_handle_data(PgAioHandle *ioh, uint8 *len)
+{
+ Assert(ioh->handle_data_len > 0);
+
+ *len = ioh->handle_data_len;
+
+ return &pgaio_ctl->handle_data[ioh->iovec_off];
+}
+
+
+
+/* --------------------------------------------------------------------------------
+ * Public IO Result related functions
+ * --------------------------------------------------------------------------------
+ */
+
+void
+pgaio_result_report(PgAioResult result, const PgAioTargetData *target_data, int elevel)
+{
+ PgAioHandleCallbackID cb_id = result.id;
+ const PgAioHandleCallbacksEntry *ce = &aio_handle_cbs[cb_id];
+
+ Assert(result.status != ARS_UNKNOWN);
+ Assert(result.status != ARS_OK);
+
+ if (ce->cb->report == NULL)
+ elog(ERROR, "callback %d/%s does not have report callback",
+ result.id, ce->name);
+
+ ce->cb->report(result, target_data, elevel);
+}
+
+
+
+/* --------------------------------------------------------------------------------
+ * Internal callback related functions operating on IO Handles
+ * --------------------------------------------------------------------------------
+ */
+
+/*
+ * Internal function which invokes ->stage for all the registered callbacks.
+ */
+void
+pgaio_io_call_stage(PgAioHandle *ioh)
+{
+ Assert(ioh->target > PGAIO_TID_INVALID && ioh->target < PGAIO_TID_COUNT);
+ Assert(ioh->op > PGAIO_OP_INVALID && ioh->op < PGAIO_OP_COUNT);
+
+ for (int i = ioh->num_callbacks; i > 0; i--)
+ {
+ PgAioHandleCallbackID cb_id = ioh->callbacks[i - 1];
+ uint8 cb_data = ioh->callbacks_data[i - 1];
+ const PgAioHandleCallbacksEntry *ce = &aio_handle_cbs[cb_id];
+
+ if (!ce->cb->stage)
+ continue;
+
+ pgaio_debug_io(DEBUG3, ioh,
+ "calling cb #%d %d/%s->stage(%u)",
+ i, cb_id, ce->name, cb_data);
+ ce->cb->stage(ioh, cb_data);
+ }
+}
+
+/*
+ * Internal function which invokes ->complete_shared for all the registered
+ * callbacks.
+ */
+void
+pgaio_io_call_complete_shared(PgAioHandle *ioh)
+{
+ PgAioResult result;
+
+ START_CRIT_SECTION();
+
+ Assert(ioh->target > PGAIO_TID_INVALID && ioh->target < PGAIO_TID_COUNT);
+ Assert(ioh->op > PGAIO_OP_INVALID && ioh->op < PGAIO_OP_COUNT);
+
+ result.status = ARS_OK; /* low level IO is always considered OK */
+ result.result = ioh->result;
+ result.id = PGAIO_HCB_INVALID;
+ result.error_data = 0;
+
+ /*
+ * Call callbacks with the last registered (innermost) callback first.
+ * Each callback can modify the result forwarded to the next callback.
+ */
+ for (int i = ioh->num_callbacks; i > 0; i--)
+ {
+ PgAioHandleCallbackID cb_id = ioh->callbacks[i - 1];
+ uint8 cb_data = ioh->callbacks_data[i - 1];
+ const PgAioHandleCallbacksEntry *ce = &aio_handle_cbs[cb_id];
+
+ if (!ce->cb->complete_shared)
+ continue;
+
+ pgaio_debug_io(DEBUG4, ioh,
+ "calling cb #%d, id %d/%s->complete_shared(%u) with distilled result: (status %s, id %u, error_data %d, result %d)",
+ i, cb_id, ce->name,
+ cb_data,
+ pgaio_result_status_string(result.status),
+ result.id, result.error_data, result.result);
+ result = ce->cb->complete_shared(ioh, result, cb_data);
+ }
+
+ ioh->distilled_result = result;
+
+ pgaio_debug_io(DEBUG3, ioh,
+ "after shared completion: distilled result: (status %s, id %u, error_data: %d, result %d), raw_result: %d",
+ pgaio_result_status_string(result.status),
+ result.id, result.error_data, result.result,
+ ioh->result);
+
+ END_CRIT_SECTION();
+}
+
+/*
+ * Internal function which invokes ->complete_local for all the registered
+ * callbacks.
+ *
+ * XXX: It'd be nice to deduplicate with pgaio_io_call_complete_shared().
+ */
+void
+pgaio_io_call_complete_local(PgAioHandle *ioh)
+{
+ PgAioResult result;
+
+ START_CRIT_SECTION();
+
+ Assert(ioh->target > PGAIO_TID_INVALID && ioh->target < PGAIO_TID_COUNT);
+ Assert(ioh->op > PGAIO_OP_INVALID && ioh->op < PGAIO_OP_COUNT);
+
+ /* start with distilled result from shared callback */
+ result = ioh->distilled_result;
+
+ for (int i = ioh->num_callbacks; i > 0; i--)
+ {
+ PgAioHandleCallbackID cb_id = ioh->callbacks[i - 1];
+ uint8 cb_data = ioh->callbacks_data[i - 1];
+ const PgAioHandleCallbacksEntry *ce = &aio_handle_cbs[cb_id];
+
+ if (!ce->cb->complete_local)
+ continue;
+
+ pgaio_debug_io(DEBUG4, ioh,
+ "calling cb #%d, id %d/%s->complete_local(%u) with distilled result: status %s, id %u, error_data %d, result %d",
+ i, cb_id, ce->name, cb_data,
+ pgaio_result_status_string(result.status),
+ result.id, result.error_data, result.result);
+ result = ce->cb->complete_local(ioh, result, cb_data);
+ }
+
+ /*
+ * Note that we don't save the result in ioh->distilled_result, the local
+ * callback's result should not ever matter to other waiters.
+ */
+ pgaio_debug_io(DEBUG3, ioh,
+ "after local completion: distilled result: (status %s, id %u, error_data %d, result %d), raw_result: %d",
+ pgaio_result_status_string(result.status),
+ result.id, result.error_data, result.result,
+ ioh->result);
+
+ END_CRIT_SECTION();
+}
diff --git a/src/backend/storage/aio/aio_init.c b/src/backend/storage/aio/aio_init.c
index aeacc144149..6fe55510fae 100644
--- a/src/backend/storage/aio/aio_init.c
+++ b/src/backend/storage/aio/aio_init.c
@@ -14,24 +14,222 @@
#include "postgres.h"
+#include "miscadmin.h"
+#include "storage/aio.h"
+#include "storage/aio_internal.h"
#include "storage/aio_subsys.h"
+#include "storage/ipc.h"
+#include "storage/proc.h"
+#include "storage/shmem.h"
+#include "utils/guc.h"
+static Size
+AioCtlShmemSize(void)
+{
+ Size sz;
+
+ /* pgaio_ctl itself */
+ sz = offsetof(PgAioCtl, io_handles);
+
+ return sz;
+}
+
+static uint32
+AioProcs(void)
+{
+ return MaxBackends + NUM_AUXILIARY_PROCS;
+}
+
+static Size
+AioBackendShmemSize(void)
+{
+ return mul_size(AioProcs(), sizeof(PgAioBackend));
+}
+
+static Size
+AioHandleShmemSize(void)
+{
+ Size sz;
+
+ /* verify AioChooseMaxConcurrency() did its thing */
+ Assert(io_max_concurrency > 0);
+
+ /* io handles */
+ sz = mul_size(AioProcs(),
+ mul_size(io_max_concurrency, sizeof(PgAioHandle)));
+
+ return sz;
+}
+
+static Size
+AioHandleIOVShmemSize(void)
+{
+ /*
+ * Each IO handle can have an PG_IOV_MAX long iovec.
+ *
+ * XXX: Right now the amount of space available for each IO is PG_IOV_MAX.
+ * While it's tempting to use the io_combine_limit GUC, that's
+ * PGC_USERSET, so we can't allocate shared memory based on that.
+ */
+ return mul_size(sizeof(struct iovec),
+ mul_size(mul_size(PG_IOV_MAX, AioProcs()),
+ io_max_concurrency));
+}
+
+static Size
+AioHandleDataShmemSize(void)
+{
+ /* each buffer referenced by an iovec can have associated data */
+ return mul_size(sizeof(uint64),
+ mul_size(mul_size(PG_IOV_MAX, AioProcs()),
+ io_max_concurrency));
+}
+
+/*
+ * Choose a suitable value for io_max_concurrency.
+ *
+ * It's unlikely that we could have more IOs in flight than buffers that we
+ * would be allowed to pin.
+ *
+ * On the upper end, apply a cap too - just because shared_buffers is large,
+ * it doesn't make sense have millions of buffers undergo IO concurrently.
+ */
+static int
+AioChooseMaxConcurrency(void)
+{
+ uint32 max_backends;
+ int max_proportional_pins;
+
+ /* Similar logic to LimitAdditionalPins() */
+ max_backends = MaxBackends + NUM_AUXILIARY_PROCS;
+ max_proportional_pins = NBuffers / max_backends;
+
+ max_proportional_pins = Max(max_proportional_pins, 1);
+
+ /* apply upper limit */
+ return Min(max_proportional_pins, 64);
+}
+
Size
AioShmemSize(void)
{
Size sz = 0;
+ /*
+ * We prefer to report this value's source as PGC_S_DYNAMIC_DEFAULT.
+ * However, if the DBA explicitly set io_max_concurrency = -1 in the
+ * config file, then PGC_S_DYNAMIC_DEFAULT will fail to override that and
+ * we must force the matter with PGC_S_OVERRIDE.
+ */
+ if (io_max_concurrency == -1)
+ {
+ char buf[32];
+
+ snprintf(buf, sizeof(buf), "%d", AioChooseMaxConcurrency());
+ SetConfigOption("io_max_concurrency", buf, PGC_POSTMASTER,
+ PGC_S_DYNAMIC_DEFAULT);
+ if (io_max_concurrency == -1) /* failed to apply it? */
+ SetConfigOption("io_max_concurrency", buf, PGC_POSTMASTER,
+ PGC_S_OVERRIDE);
+ }
+
+ sz = add_size(sz, AioCtlShmemSize());
+ sz = add_size(sz, AioBackendShmemSize());
+ sz = add_size(sz, AioHandleShmemSize());
+ sz = add_size(sz, AioHandleIOVShmemSize());
+ sz = add_size(sz, AioHandleDataShmemSize());
+
+ /* Reserve space for method specific resources. */
+ if (pgaio_method_ops->shmem_size)
+ sz = add_size(sz, pgaio_method_ops->shmem_size());
+
return sz;
}
void
AioShmemInit(void)
{
+ bool found;
+ uint32 io_handle_off = 0;
+ uint32 iovec_off = 0;
+ uint32 per_backend_iovecs = io_max_concurrency * PG_IOV_MAX;
+
+ pgaio_ctl = (PgAioCtl *)
+ ShmemInitStruct("AioCtl", AioCtlShmemSize(), &found);
+
+ if (found)
+ goto out;
+
+ memset(pgaio_ctl, 0, AioCtlShmemSize());
+
+ pgaio_ctl->io_handle_count = AioProcs() * io_max_concurrency;
+ pgaio_ctl->iovec_count = AioProcs() * per_backend_iovecs;
+
+ pgaio_ctl->backend_state = (PgAioBackend *)
+ ShmemInitStruct("AioBackend", AioBackendShmemSize(), &found);
+
+ pgaio_ctl->io_handles = (PgAioHandle *)
+ ShmemInitStruct("AioHandle", AioHandleShmemSize(), &found);
+
+ pgaio_ctl->iovecs = (struct iovec *)
+ ShmemInitStruct("AioHandleIOV", AioHandleIOVShmemSize(), &found);
+ pgaio_ctl->handle_data = (uint64 *)
+ ShmemInitStruct("AioHandleData", AioHandleDataShmemSize(), &found);
+
+ for (int procno = 0; procno < AioProcs(); procno++)
+ {
+ PgAioBackend *bs = &pgaio_ctl->backend_state[procno];
+
+ bs->io_handle_off = io_handle_off;
+ io_handle_off += io_max_concurrency;
+
+ dclist_init(&bs->idle_ios);
+ memset(bs->staged_ios, 0, sizeof(PgAioHandle *) * PGAIO_SUBMIT_BATCH_SIZE);
+ dclist_init(&bs->in_flight_ios);
+
+ /* initialize per-backend IOs */
+ for (int i = 0; i < io_max_concurrency; i++)
+ {
+ PgAioHandle *ioh = &pgaio_ctl->io_handles[bs->io_handle_off + i];
+
+ ioh->generation = 1;
+ ioh->owner_procno = procno;
+ ioh->iovec_off = iovec_off;
+ ioh->handle_data_len = 0;
+ ioh->report_return = NULL;
+ ioh->resowner = NULL;
+ ioh->num_callbacks = 0;
+ ioh->distilled_result.status = ARS_UNKNOWN;
+ ioh->flags = 0;
+
+ ConditionVariableInit(&ioh->cv);
+
+ dclist_push_tail(&bs->idle_ios, &ioh->node);
+ iovec_off += PG_IOV_MAX;
+ }
+ }
+
+out:
+ /* Initialize IO method specific resources. */
+ if (pgaio_method_ops->shmem_init)
+ pgaio_method_ops->shmem_init(!found);
}
void
pgaio_init_backend(void)
{
+ /* shouldn't be initialized twice */
+ Assert(!pgaio_my_backend);
+
+ if (MyProc == NULL || MyProcNumber >= AioProcs())
+ elog(ERROR, "aio requires a normal PGPROC");
+
+ pgaio_my_backend = &pgaio_ctl->backend_state[MyProcNumber];
+
+ if (pgaio_method_ops->init_backend)
+ pgaio_method_ops->init_backend();
+
+ before_shmem_exit(pgaio_shutdown, 0);
}
diff --git a/src/backend/storage/aio/aio_io.c b/src/backend/storage/aio/aio_io.c
new file mode 100644
index 00000000000..36d2c1f492d
--- /dev/null
+++ b/src/backend/storage/aio/aio_io.c
@@ -0,0 +1,184 @@
+/*-------------------------------------------------------------------------
+ *
+ * aio_io.c
+ * AIO - Low Level IO Handling
+ *
+ * Functions related to associating IO operations to IO Handles and IO-method
+ * independent support functions for actually performing IO.
+ *
+ *
+ * Portions Copyright (c) 1996-2025, PostgreSQL Global Development Group
+ * Portions Copyright (c) 1994, Regents of the University of California
+ *
+ * IDENTIFICATION
+ * src/backend/storage/aio/aio_io.c
+ *
+ *-------------------------------------------------------------------------
+ */
+
+#include "postgres.h"
+
+#include "miscadmin.h"
+#include "storage/aio.h"
+#include "storage/aio_internal.h"
+#include "storage/fd.h"
+#include "utils/wait_event.h"
+
+
+static void pgaio_io_before_prep(PgAioHandle *ioh);
+
+
+
+/* --------------------------------------------------------------------------------
+ * Public IO related functions operating on IO Handles
+ * --------------------------------------------------------------------------------
+ */
+
+/*
+ * Scatter/gather IO needs to associate an iovec with the Handle. To support
+ * worker mode this data needs to be in shared memory.
+ */
+int
+pgaio_io_get_iovec(PgAioHandle *ioh, struct iovec **iov)
+{
+ Assert(ioh->state == PGAIO_HS_HANDED_OUT);
+
+ *iov = &pgaio_ctl->iovecs[ioh->iovec_off];
+
+ return PG_IOV_MAX;
+}
+
+PgAioOp
+pgaio_io_get_op(PgAioHandle *ioh)
+{
+ return ioh->op;
+}
+
+PgAioOpData *
+pgaio_io_get_op_data(PgAioHandle *ioh)
+{
+ return &ioh->op_data;
+}
+
+
+
+/* --------------------------------------------------------------------------------
+ * "Preparation" routines for individual IO operations
+ *
+ * These are called by the code actually initiating an IO, to associate the IO
+ * specific data with an AIO handle.
+ *
+ * Each of the preparation routines first needs to call
+ * pgaio_io_before_prep(), then fill IO specific fields in the handle and then
+ * finally call pgaio_io_stage().
+ * --------------------------------------------------------------------------------
+ */
+
+void
+pgaio_io_prep_readv(PgAioHandle *ioh,
+ int fd, int iovcnt, uint64 offset)
+{
+ pgaio_io_before_prep(ioh);
+
+ ioh->op_data.read.fd = fd;
+ ioh->op_data.read.offset = offset;
+ ioh->op_data.read.iov_length = iovcnt;
+
+ pgaio_io_stage(ioh, PGAIO_OP_READV);
+}
+
+void
+pgaio_io_prep_writev(PgAioHandle *ioh,
+ int fd, int iovcnt, uint64 offset)
+{
+ pgaio_io_before_prep(ioh);
+
+ ioh->op_data.write.fd = fd;
+ ioh->op_data.write.offset = offset;
+ ioh->op_data.write.iov_length = iovcnt;
+
+ pgaio_io_stage(ioh, PGAIO_OP_WRITEV);
+}
+
+
+
+/* --------------------------------------------------------------------------------
+ * Internal IO related functions operating on IO Handles
+ * --------------------------------------------------------------------------------
+ */
+
+/*
+ * Execute IO operation synchronously. This is implemented here, not in
+ * method_sync.c, because other IO methods also might use it / fall back to
+ * it.
+ */
+void
+pgaio_io_perform_synchronously(PgAioHandle *ioh)
+{
+ ssize_t result = 0;
+ struct iovec *iov = &pgaio_ctl->iovecs[ioh->iovec_off];
+
+ START_CRIT_SECTION();
+
+ /* Perform IO. */
+ switch (ioh->op)
+ {
+ case PGAIO_OP_READV:
+ pgstat_report_wait_start(WAIT_EVENT_DATA_FILE_READ);
+ result = pg_preadv(ioh->op_data.read.fd, iov,
+ ioh->op_data.read.iov_length,
+ ioh->op_data.read.offset);
+ pgstat_report_wait_end();
+ break;
+ case PGAIO_OP_WRITEV:
+ pgstat_report_wait_start(WAIT_EVENT_DATA_FILE_WRITE);
+ result = pg_pwritev(ioh->op_data.write.fd, iov,
+ ioh->op_data.write.iov_length,
+ ioh->op_data.write.offset);
+ pgstat_report_wait_end();
+ break;
+ case PGAIO_OP_INVALID:
+ elog(ERROR, "trying to execute invalid IO operation");
+ }
+
+ ioh->result = result < 0 ? -errno : result;
+
+ pgaio_io_process_completion(ioh, ioh->result);
+
+ END_CRIT_SECTION();
+}
+
+/*
+ * Helper function to be called by IO operation preparation functions, before
+ * any data in the handle is set. Mostly to centralize assertions.
+ */
+static void
+pgaio_io_before_prep(PgAioHandle *ioh)
+{
+ Assert(ioh->state == PGAIO_HS_HANDED_OUT);
+ Assert(pgaio_my_backend->handed_out_io == ioh);
+ Assert(pgaio_io_has_target(ioh));
+ Assert(ioh->op == PGAIO_OP_INVALID);
+}
+
+/*
+ * Could be made part of the public interface, but it's not clear there's
+ * really a use case for that.
+ */
+const char *
+pgaio_io_get_op_name(PgAioHandle *ioh)
+{
+ Assert(ioh->op >= 0 && ioh->op < PGAIO_OP_COUNT);
+
+ switch (ioh->op)
+ {
+ case PGAIO_OP_INVALID:
+ return "invalid";
+ case PGAIO_OP_READV:
+ return "read";
+ case PGAIO_OP_WRITEV:
+ return "write";
+ }
+
+ return NULL; /* silence compiler */
+}
diff --git a/src/backend/storage/aio/aio_target.c b/src/backend/storage/aio/aio_target.c
new file mode 100644
index 00000000000..b01406a6a52
--- /dev/null
+++ b/src/backend/storage/aio/aio_target.c
@@ -0,0 +1,114 @@
+/*-------------------------------------------------------------------------
+ *
+ * aio_target.c
+ * AIO - Functionality related to executing IO for different targets
+ *
+ * Portions Copyright (c) 1996-2025, PostgreSQL Global Development Group
+ * Portions Copyright (c) 1994, Regents of the University of California
+ *
+ * IDENTIFICATION
+ * src/backend/storage/aio/aio_target.c
+ *
+ *-------------------------------------------------------------------------
+ */
+
+#include "postgres.h"
+
+#include "storage/aio.h"
+#include "storage/aio_internal.h"
+
+
+/*
+ * Registry for entities that can be the target of AIO.
+ */
+static const PgAioTargetInfo *pgaio_target_info[] = {
+ [PGAIO_TID_INVALID] = &(PgAioTargetInfo) {
+ .name = "invalid",
+ },
+};
+
+
+
+/* --------------------------------------------------------------------------------
+ * Public target related functions operating on IO Handles
+ * --------------------------------------------------------------------------------
+ */
+
+bool
+pgaio_io_has_target(PgAioHandle *ioh)
+{
+ return ioh->target != PGAIO_TID_INVALID;
+}
+
+/*
+ * Return the name for the target associated with the IO. Mostly useful for
+ * debugging/logging.
+ */
+const char *
+pgaio_io_get_target_name(PgAioHandle *ioh)
+{
+ Assert(ioh->target >= 0 && ioh->target < PGAIO_TID_COUNT);
+
+ return pgaio_target_info[ioh->target]->name;
+}
+
+/*
+ * Assign a target to the IO.
+ *
+ * This has to be called exactly once before pgaio_io_prep_*() is called.
+ */
+void
+pgaio_io_set_target(PgAioHandle *ioh, PgAioTargetID targetid)
+{
+ Assert(ioh->state == PGAIO_HS_HANDED_OUT);
+ Assert(ioh->target == PGAIO_TID_INVALID);
+
+ ioh->target = targetid;
+}
+
+PgAioTargetData *
+pgaio_io_get_target_data(PgAioHandle *ioh)
+{
+ return &ioh->target_data;
+}
+
+/*
+ * Return a stringified description of the IO's target.
+ *
+ * The string is localized and allocated in the current memory context.
+ */
+char *
+pgaio_io_get_target_description(PgAioHandle *ioh)
+{
+ return pgaio_target_info[ioh->target]->describe_identity(&ioh->target_data);
+}
+
+
+
+/* --------------------------------------------------------------------------------
+ * Internal target related functions operating on IO Handles
+ * --------------------------------------------------------------------------------
+ */
+
+/*
+ * Internal: Check if pgaio_io_reopen() is available for the IO.
+ */
+bool
+pgaio_io_can_reopen(PgAioHandle *ioh)
+{
+ return pgaio_target_info[ioh->target]->reopen != NULL;
+}
+
+/*
+ * Internal: Before executing an IO outside of the context of the process the
+ * IO has been prepared in, the file descriptor has to be reopened - any FD
+ * referenced in the IO itself, won't be valid in the separate process.
+ */
+void
+pgaio_io_reopen(PgAioHandle *ioh)
+{
+ Assert(ioh->target >= 0 && ioh->target < PGAIO_TID_COUNT);
+ Assert(ioh->op >= 0 && ioh->op < PGAIO_OP_COUNT);
+
+ pgaio_target_info[ioh->target]->reopen(ioh);
+}
diff --git a/src/backend/storage/aio/meson.build b/src/backend/storage/aio/meson.build
index c822fd4ddf7..2c26089d52e 100644
--- a/src/backend/storage/aio/meson.build
+++ b/src/backend/storage/aio/meson.build
@@ -2,6 +2,10 @@
backend_sources += files(
'aio.c',
+ 'aio_callback.c',
'aio_init.c',
+ 'aio_io.c',
+ 'aio_target.c',
+ 'method_sync.c',
'read_stream.c',
)
diff --git a/src/backend/storage/aio/method_sync.c b/src/backend/storage/aio/method_sync.c
new file mode 100644
index 00000000000..902c2428d41
--- /dev/null
+++ b/src/backend/storage/aio/method_sync.c
@@ -0,0 +1,47 @@
+/*-------------------------------------------------------------------------
+ *
+ * method_sync.c
+ * AIO - perform "AIO" by executing it synchronously
+ *
+ * This method is mainly to check if AIO use causes regressions. Other IO
+ * methods might also fall back to the synchronous method for functionality
+ * they cannot provide.
+ *
+ * Portions Copyright (c) 1996-2025, PostgreSQL Global Development Group
+ * Portions Copyright (c) 1994, Regents of the University of California
+ *
+ * IDENTIFICATION
+ * src/backend/storage/aio/method_sync.c
+ *
+ *-------------------------------------------------------------------------
+ */
+
+#include "postgres.h"
+
+#include "storage/aio.h"
+#include "storage/aio_internal.h"
+
+static bool pgaio_sync_needs_synchronous_execution(PgAioHandle *ioh);
+static int pgaio_sync_submit(uint16 num_staged_ios, PgAioHandle **staged_ios);
+
+
+const IoMethodOps pgaio_sync_ops = {
+ .needs_synchronous_execution = pgaio_sync_needs_synchronous_execution,
+ .submit = pgaio_sync_submit,
+};
+
+
+
+static bool
+pgaio_sync_needs_synchronous_execution(PgAioHandle *ioh)
+{
+ return true;
+}
+
+static int
+pgaio_sync_submit(uint16 num_staged_ios, PgAioHandle **staged_ios)
+{
+ elog(ERROR, "IO should have been executed synchronously");
+
+ return 0;
+}
diff --git a/src/backend/utils/activity/wait_event_names.txt b/src/backend/utils/activity/wait_event_names.txt
index 3c594415bfd..b44e4908b25 100644
--- a/src/backend/utils/activity/wait_event_names.txt
+++ b/src/backend/utils/activity/wait_event_names.txt
@@ -191,6 +191,7 @@ ABI_compatibility:
Section: ClassName - WaitEventIO
+AIO_IO_COMPLETION "Waiting for IO completion."
BASEBACKUP_READ "Waiting for base backup to read from a file."
BASEBACKUP_SYNC "Waiting for data written by a base backup to reach durable storage."
BASEBACKUP_WRITE "Waiting for base backup to write to a file."
diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list
index 3c9e823f07e..f4261145353 100644
--- a/src/tools/pgindent/typedefs.list
+++ b/src/tools/pgindent/typedefs.list
@@ -1280,6 +1280,7 @@ InvalMessageArray
InvalidationInfo
InvalidationMsgsGroup
IoMethod
+IoMethodOps
IpcMemoryId
IpcMemoryKey
IpcMemoryState
@@ -2127,6 +2128,26 @@ Permutation
PermutationStep
PermutationStepBlocker
PermutationStepBlockerType
+PgAioBackend
+PgAioCtl
+PgAioHandle
+PgAioHandleCallbackID
+PgAioHandleCallbackStage
+PgAioHandleCallbackComplete
+PgAioHandleCallbackReport
+PgAioHandleCallbacks
+PgAioHandleCallbacksEntry
+PgAioHandleFlags
+PgAioHandleState
+PgAioOp
+PgAioOpData
+PgAioResult
+PgAioResultStatus
+PgAioReturn
+PgAioTargetData
+PgAioTargetID
+PgAioTargetInfo
+PgAioWaitRef
PgArchData
PgBackendGSSStatus
PgBackendSSLStatus
--
2.48.1.76.g4e746b1a31.dirty
[text/x-diff] v2.9-0003-aio-Infrastructure-for-io_method-worker.patch (24.9K, ../../oav52ytakcvfnk52w5o5wvxdjavxth2tgptxt25gnfmh2pdcpk@omp3xukdu5if/4-v2.9-0003-aio-Infrastructure-for-io_method-worker.patch)
download | inline diff:
From 8db5243ad68b6802e401fc1d465d01072ae794cb Mon Sep 17 00:00:00 2001
From: Andres Freund <andres@anarazel.de>
Date: Sat, 15 Mar 2025 12:29:57 -0400
Subject: [PATCH v2.9 03/30] aio: Infrastructure for io_method=worker
This commit contains the basic, system-wide, infrastructure for
io_method=worker. It does not yet actually execute IO, this commit just
provides the infrastructure for running IO workers, kept separate for easier
review.
The number of IO workers can be adjusted with a PGC_SIGHUP GUC. Eventually
we'd like to make the number of workers dynamically scale up/down based on the
current "IO load".
To allow the number of IO workers to be increased without a restart, we need
to reserve PGPROC entries for the workers unconditionally. This has been
judged to be worth the cost. If it turns out to be problematic, we can
introduce a PGC_POSTMASTER GUC to control the maximum number.
Reviewed-by: Noah Misch <noah@leadboat.com>
Co-authored-by: Thomas Munro <thomas.munro@gmail.com>
Co-authored-by: Andres Freund <andres@anarazel.de>
Discussion: https://postgr.es/m/uvrtrknj4kdytuboidbhwclo4gxhswwcpgadptsjvjqcluzmah%40brqs62irg4dt
Discussion: https://postgr.es/m/20210223100344.llw5an2aklengrmn@alap3.anarazel.de
Discussion: https://postgr.es/m/stj36ea6yyhoxtqkhpieia2z4krnam7qyetc57rfezgk4zgapf@gcnactj4z56m
---
src/include/miscadmin.h | 2 +
src/include/storage/aio_subsys.h | 4 +
src/include/storage/io_worker.h | 22 +++
src/include/storage/proc.h | 4 +-
src/backend/postmaster/launch_backend.c | 2 +
src/backend/postmaster/pmchild.c | 1 +
src/backend/postmaster/postmaster.c | 174 ++++++++++++++++--
src/backend/storage/aio/Makefile | 1 +
src/backend/storage/aio/meson.build | 1 +
src/backend/storage/aio/method_worker.c | 88 +++++++++
src/backend/tcop/postgres.c | 2 +
src/backend/utils/activity/pgstat_backend.c | 1 +
src/backend/utils/activity/pgstat_io.c | 1 +
.../utils/activity/wait_event_names.txt | 1 +
src/backend/utils/init/miscinit.c | 3 +
src/backend/utils/misc/guc_tables.c | 13 ++
src/backend/utils/misc/postgresql.conf.sample | 1 +
doc/src/sgml/config.sgml | 19 ++
src/test/regress/expected/stats.out | 10 +-
19 files changed, 336 insertions(+), 14 deletions(-)
create mode 100644 src/include/storage/io_worker.h
create mode 100644 src/backend/storage/aio/method_worker.c
diff --git a/src/include/miscadmin.h b/src/include/miscadmin.h
index 6f16794eb63..603d0424354 100644
--- a/src/include/miscadmin.h
+++ b/src/include/miscadmin.h
@@ -360,6 +360,7 @@ typedef enum BackendType
B_ARCHIVER,
B_BG_WRITER,
B_CHECKPOINTER,
+ B_IO_WORKER,
B_STARTUP,
B_WAL_RECEIVER,
B_WAL_SUMMARIZER,
@@ -389,6 +390,7 @@ extern PGDLLIMPORT BackendType MyBackendType;
#define AmWalReceiverProcess() (MyBackendType == B_WAL_RECEIVER)
#define AmWalSummarizerProcess() (MyBackendType == B_WAL_SUMMARIZER)
#define AmWalWriterProcess() (MyBackendType == B_WAL_WRITER)
+#define AmIoWorkerProcess() (MyBackendType == B_IO_WORKER)
#define AmSpecialWorkerProcess() \
(AmAutoVacuumLauncherProcess() || \
diff --git a/src/include/storage/aio_subsys.h b/src/include/storage/aio_subsys.h
index 2a6ca1c27a9..8a8ce87f62a 100644
--- a/src/include/storage/aio_subsys.h
+++ b/src/include/storage/aio_subsys.h
@@ -30,4 +30,8 @@ extern void pgaio_init_backend(void);
extern void pgaio_error_cleanup(void);
extern void AtEOXact_Aio(bool is_commit);
+
+/* aio_worker.c */
+extern bool pgaio_workers_enabled(void);
+
#endif /* AIO_SUBSYS_H */
diff --git a/src/include/storage/io_worker.h b/src/include/storage/io_worker.h
new file mode 100644
index 00000000000..7bde7e89c8a
--- /dev/null
+++ b/src/include/storage/io_worker.h
@@ -0,0 +1,22 @@
+/*-------------------------------------------------------------------------
+ *
+ * io_worker.h
+ * IO worker for implementing AIO "ourselves"
+ *
+ *
+ * Portions Copyright (c) 1996-2025, PostgreSQL Global Development Group
+ * Portions Copyright (c) 1994, Regents of the University of California
+ *
+ * src/include/storage/io.h
+ *
+ *-------------------------------------------------------------------------
+ */
+#ifndef IO_WORKER_H
+#define IO_WORKER_H
+
+
+pg_noreturn extern void IoWorkerMain(const void *startup_data, size_t startup_data_len);
+
+extern PGDLLIMPORT int io_workers;
+
+#endif /* IO_WORKER_H */
diff --git a/src/include/storage/proc.h b/src/include/storage/proc.h
index 0750ec3c474..f51b03d3822 100644
--- a/src/include/storage/proc.h
+++ b/src/include/storage/proc.h
@@ -449,7 +449,9 @@ extern PGDLLIMPORT PGPROC *PreparedXactProcs;
* 2 slots, but WAL writer is launched only after startup has exited, so we
* only need 6 slots.
*/
-#define NUM_AUXILIARY_PROCS 6
+#define MAX_IO_WORKERS 32
+#define NUM_AUXILIARY_PROCS (6 + MAX_IO_WORKERS)
+
/* configurable options */
extern PGDLLIMPORT int DeadlockTimeout;
diff --git a/src/backend/postmaster/launch_backend.c b/src/backend/postmaster/launch_backend.c
index 77fb877dbad..bf6b55ee830 100644
--- a/src/backend/postmaster/launch_backend.c
+++ b/src/backend/postmaster/launch_backend.c
@@ -48,6 +48,7 @@
#include "replication/slotsync.h"
#include "replication/walreceiver.h"
#include "storage/dsm.h"
+#include "storage/io_worker.h"
#include "storage/pg_shmem.h"
#include "tcop/backend_startup.h"
#include "utils/memutils.h"
@@ -197,6 +198,7 @@ static child_process_kind child_process_kinds[] = {
[B_ARCHIVER] = {"archiver", PgArchiverMain, true},
[B_BG_WRITER] = {"bgwriter", BackgroundWriterMain, true},
[B_CHECKPOINTER] = {"checkpointer", CheckpointerMain, true},
+ [B_IO_WORKER] = {"io_worker", IoWorkerMain, true},
[B_STARTUP] = {"startup", StartupProcessMain, true},
[B_WAL_RECEIVER] = {"wal_receiver", WalReceiverMain, true},
[B_WAL_SUMMARIZER] = {"wal_summarizer", WalSummarizerMain, true},
diff --git a/src/backend/postmaster/pmchild.c b/src/backend/postmaster/pmchild.c
index 0d473226c3a..cde1d23a4ca 100644
--- a/src/backend/postmaster/pmchild.c
+++ b/src/backend/postmaster/pmchild.c
@@ -101,6 +101,7 @@ InitPostmasterChildSlots(void)
pmchild_pools[B_AUTOVAC_WORKER].size = autovacuum_worker_slots;
pmchild_pools[B_BG_WORKER].size = max_worker_processes;
+ pmchild_pools[B_IO_WORKER].size = MAX_IO_WORKERS;
/*
* There can be only one of each of these running at a time. They each
diff --git a/src/backend/postmaster/postmaster.c b/src/backend/postmaster/postmaster.c
index d13846298bd..f301e43743f 100644
--- a/src/backend/postmaster/postmaster.c
+++ b/src/backend/postmaster/postmaster.c
@@ -108,9 +108,12 @@
#include "replication/logicallauncher.h"
#include "replication/slotsync.h"
#include "replication/walsender.h"
+#include "storage/aio_subsys.h"
#include "storage/fd.h"
+#include "storage/io_worker.h"
#include "storage/ipc.h"
#include "storage/pmsignal.h"
+#include "storage/proc.h"
#include "tcop/backend_startup.h"
#include "tcop/tcopprot.h"
#include "utils/datetime.h"
@@ -340,6 +343,7 @@ typedef enum
* ckpt */
PM_WAIT_XLOG_ARCHIVAL, /* waiting for archiver and walsenders to
* finish */
+ PM_WAIT_IO_WORKERS, /* waiting for io workers to exit */
PM_WAIT_CHECKPOINTER, /* waiting for checkpointer to shut down */
PM_WAIT_DEAD_END, /* waiting for dead-end children to exit */
PM_NO_CHILDREN, /* all important children have exited */
@@ -402,6 +406,10 @@ bool LoadedSSL = false;
static DNSServiceRef bonjour_sdref = NULL;
#endif
+/* State for IO worker management. */
+static int io_worker_count = 0;
+static PMChild *io_worker_children[MAX_IO_WORKERS];
+
/*
* postmaster.c - function prototypes
*/
@@ -436,6 +444,8 @@ static void TerminateChildren(int signal);
static int CountChildren(BackendTypeMask targetMask);
static void LaunchMissingBackgroundProcesses(void);
static void maybe_start_bgworkers(void);
+static bool maybe_reap_io_worker(int pid);
+static void maybe_adjust_io_workers(void);
static bool CreateOptsFile(int argc, char *argv[], char *fullprogname);
static PMChild *StartChildProcess(BackendType type);
static void StartSysLogger(void);
@@ -1365,6 +1375,11 @@ PostmasterMain(int argc, char *argv[])
*/
AddToDataDirLockFile(LOCK_FILE_LINE_PM_STATUS, PM_STATUS_STARTING);
+ UpdatePMState(PM_STARTUP);
+
+ /* Make sure we can perform I/O while starting up. */
+ maybe_adjust_io_workers();
+
/* Start bgwriter and checkpointer so they can help with recovery */
if (CheckpointerPMChild == NULL)
CheckpointerPMChild = StartChildProcess(B_CHECKPOINTER);
@@ -1377,7 +1392,6 @@ PostmasterMain(int argc, char *argv[])
StartupPMChild = StartChildProcess(B_STARTUP);
Assert(StartupPMChild != NULL);
StartupStatus = STARTUP_RUNNING;
- UpdatePMState(PM_STARTUP);
/* Some workers may be scheduled to start now */
maybe_start_bgworkers();
@@ -2502,6 +2516,16 @@ process_pm_child_exit(void)
continue;
}
+ /* Was it an IO worker? */
+ if (maybe_reap_io_worker(pid))
+ {
+ if (!EXIT_STATUS_0(exitstatus) && !EXIT_STATUS_1(exitstatus))
+ HandleChildCrash(pid, exitstatus, _("io worker"));
+
+ maybe_adjust_io_workers();
+ continue;
+ }
+
/*
* Was it a backend or a background worker?
*/
@@ -2723,6 +2747,7 @@ HandleFatalError(QuitSignalReason reason, bool consider_sigabrt)
case PM_WAIT_XLOG_SHUTDOWN:
case PM_WAIT_XLOG_ARCHIVAL:
case PM_WAIT_CHECKPOINTER:
+ case PM_WAIT_IO_WORKERS:
/*
* NB: Similar code exists in PostmasterStateMachine()'s handling
@@ -2905,20 +2930,21 @@ PostmasterStateMachine(void)
/*
* If we are doing crash recovery or an immediate shutdown then we
- * expect archiver, checkpointer and walsender to exit as well,
- * otherwise not.
+ * expect archiver, checkpointer, io workers and walsender to exit as
+ * well, otherwise not.
*/
if (FatalError || Shutdown >= ImmediateShutdown)
targetMask = btmask_add(targetMask,
B_CHECKPOINTER,
B_ARCHIVER,
+ B_IO_WORKER,
B_WAL_SENDER);
/*
- * Normally walsenders and archiver will continue running; they will
- * be terminated later after writing the checkpoint record. We also
- * let dead-end children to keep running for now. The syslogger
- * process exits last.
+ * Normally archiver, checkpointer, IO workers and walsenders will
+ * continue running; they will be terminated later after writing the
+ * checkpoint record. We also let dead-end children to keep running
+ * for now. The syslogger process exits last.
*
* This assertion checks that we have covered all backend types,
* either by including them in targetMask, or by noting here that they
@@ -2933,12 +2959,13 @@ PostmasterStateMachine(void)
B_LOGGER);
/*
- * Archiver, checkpointer and walsender may or may not be in
- * targetMask already.
+ * Archiver, checkpointer, IO workers, and walsender may or may
+ * not be in targetMask already.
*/
remainMask = btmask_add(remainMask,
B_ARCHIVER,
B_CHECKPOINTER,
+ B_IO_WORKER,
B_WAL_SENDER);
/* these are not real postmaster children */
@@ -3039,11 +3066,25 @@ PostmasterStateMachine(void)
{
/*
* PM_WAIT_XLOG_ARCHIVAL state ends when there are no children other
- * than checkpointer, dead-end children and logger left. There
+ * than checkpointer, io workers and dead-end children left. There
* shouldn't be any regular backends left by now anyway; what we're
* really waiting for is for walsenders and archiver to exit.
*/
- if (CountChildren(btmask_all_except(B_CHECKPOINTER, B_LOGGER, B_DEAD_END_BACKEND)) == 0)
+ if (CountChildren(btmask_all_except(B_CHECKPOINTER, B_IO_WORKER,
+ B_LOGGER, B_DEAD_END_BACKEND)) == 0)
+ {
+ UpdatePMState(PM_WAIT_IO_WORKERS);
+ SignalChildren(SIGUSR2, btmask(B_IO_WORKER));
+ }
+ }
+
+ if (pmState == PM_WAIT_IO_WORKERS)
+ {
+ /*
+ * PM_WAIT_IO_WORKERS state ends when there's only checkpointer and
+ * dead_end children left.
+ */
+ if (io_worker_count == 0)
{
UpdatePMState(PM_WAIT_CHECKPOINTER);
@@ -3171,10 +3212,14 @@ PostmasterStateMachine(void)
/* re-create shared memory and semaphores */
CreateSharedMemoryAndSemaphores();
+ UpdatePMState(PM_STARTUP);
+
+ /* Make sure we can perform I/O while starting up. */
+ maybe_adjust_io_workers();
+
StartupPMChild = StartChildProcess(B_STARTUP);
Assert(StartupPMChild != NULL);
StartupStatus = STARTUP_RUNNING;
- UpdatePMState(PM_STARTUP);
/* crash recovery started, reset SIGKILL flag */
AbortStartTime = 0;
@@ -3198,6 +3243,7 @@ pmstate_name(PMState state)
PM_TOSTR_CASE(PM_WAIT_BACKENDS);
PM_TOSTR_CASE(PM_WAIT_XLOG_SHUTDOWN);
PM_TOSTR_CASE(PM_WAIT_XLOG_ARCHIVAL);
+ PM_TOSTR_CASE(PM_WAIT_IO_WORKERS);
PM_TOSTR_CASE(PM_WAIT_DEAD_END);
PM_TOSTR_CASE(PM_WAIT_CHECKPOINTER);
PM_TOSTR_CASE(PM_NO_CHILDREN);
@@ -3235,6 +3281,16 @@ LaunchMissingBackgroundProcesses(void)
if (SysLoggerPMChild == NULL && Logging_collector)
StartSysLogger();
+ /*
+ * The number of configured configured workers might have changed, or a
+ * prior start of a worker might have failed. Check if we need to
+ * start/stop any workers.
+ *
+ * A config file change will always lead to this function being called, so
+ * we always will process the config change in a timely manner.
+ */
+ maybe_adjust_io_workers();
+
/*
* The checkpointer and the background writer are active from the start,
* until shutdown is initiated.
@@ -4120,6 +4176,7 @@ bgworker_should_start_now(BgWorkerStartTime start_time)
case PM_WAIT_DEAD_END:
case PM_WAIT_XLOG_ARCHIVAL:
case PM_WAIT_XLOG_SHUTDOWN:
+ case PM_WAIT_IO_WORKERS:
case PM_WAIT_BACKENDS:
case PM_STOP_BACKENDS:
break;
@@ -4270,6 +4327,99 @@ maybe_start_bgworkers(void)
}
}
+static bool
+maybe_reap_io_worker(int pid)
+{
+ for (int id = 0; id < MAX_IO_WORKERS; ++id)
+ {
+ if (io_worker_children[id] &&
+ io_worker_children[id]->pid == pid)
+ {
+ ReleasePostmasterChildSlot(io_worker_children[id]);
+
+ --io_worker_count;
+ io_worker_children[id] = NULL;
+ return true;
+ }
+ }
+ return false;
+}
+
+/*
+ * Start or stop IO workers, to close the gap between the number of running
+ * workers and the number of configured workers. Used to respond to change of
+ * the io_workers GUC (by increasing and decreasing the number of workers), as
+ * well as workers terminating in response to errors (by starting
+ * "replacement" workers).
+ */
+static void
+maybe_adjust_io_workers(void)
+{
+ if (!pgaio_workers_enabled())
+ return;
+
+ /*
+ * If we're in final shutting down state, then we're just waiting for all
+ * processes to exit.
+ */
+ if (pmState >= PM_WAIT_IO_WORKERS)
+ return;
+
+ /* Don't start new workers during an immediate shutdown either. */
+ if (Shutdown >= ImmediateShutdown)
+ return;
+
+ /*
+ * Don't start new workers if we're in the shutdown phase of a crash
+ * restart. But we *do* need to start if we're already starting up again.
+ */
+ if (FatalError && pmState >= PM_STOP_BACKENDS)
+ return;
+
+ Assert(pmState < PM_WAIT_IO_WORKERS);
+
+ /* Not enough running? */
+ while (io_worker_count < io_workers)
+ {
+ PMChild *child;
+ int id;
+
+ /* find unused entry in io_worker_children array */
+ for (id = 0; id < MAX_IO_WORKERS; ++id)
+ {
+ if (io_worker_children[id] == NULL)
+ break;
+ }
+ if (id == MAX_IO_WORKERS)
+ elog(ERROR, "could not find a free IO worker ID");
+
+ /* Try to launch one. */
+ child = StartChildProcess(B_IO_WORKER);
+ if (child != NULL)
+ {
+ io_worker_children[id] = child;
+ ++io_worker_count;
+ }
+ else
+ break; /* XXX try again soon? */
+ }
+
+ /* Too many running? */
+ if (io_worker_count > io_workers)
+ {
+ /* ask the IO worker in the highest slot to exit */
+ for (int id = MAX_IO_WORKERS - 1; id >= 0; --id)
+ {
+ if (io_worker_children[id] != NULL)
+ {
+ kill(io_worker_children[id]->pid, SIGUSR2);
+ break;
+ }
+ }
+ }
+}
+
+
/*
* When a backend asks to be notified about worker state changes, we
* set a flag in its backend entry. The background worker machinery needs
diff --git a/src/backend/storage/aio/Makefile b/src/backend/storage/aio/Makefile
index 89f821ea7e1..f51c34a37f8 100644
--- a/src/backend/storage/aio/Makefile
+++ b/src/backend/storage/aio/Makefile
@@ -15,6 +15,7 @@ OBJS = \
aio_io.o \
aio_target.o \
method_sync.o \
+ method_worker.o \
read_stream.o
include $(top_srcdir)/src/backend/common.mk
diff --git a/src/backend/storage/aio/meson.build b/src/backend/storage/aio/meson.build
index 2c26089d52e..74f94c6e40b 100644
--- a/src/backend/storage/aio/meson.build
+++ b/src/backend/storage/aio/meson.build
@@ -7,5 +7,6 @@ backend_sources += files(
'aio_io.c',
'aio_target.c',
'method_sync.c',
+ 'method_worker.c',
'read_stream.c',
)
diff --git a/src/backend/storage/aio/method_worker.c b/src/backend/storage/aio/method_worker.c
new file mode 100644
index 00000000000..0ef9ef93e2b
--- /dev/null
+++ b/src/backend/storage/aio/method_worker.c
@@ -0,0 +1,88 @@
+/*-------------------------------------------------------------------------
+ *
+ * method_worker.c
+ * AIO - perform AIO using worker processes
+ *
+ * Portions Copyright (c) 1996-2025, PostgreSQL Global Development Group
+ * Portions Copyright (c) 1994, Regents of the University of California
+ *
+ * IDENTIFICATION
+ * src/backend/storage/aio/method_worker.c
+ *
+ *-------------------------------------------------------------------------
+ */
+
+#include "postgres.h"
+
+#include "libpq/pqsignal.h"
+#include "miscadmin.h"
+#include "postmaster/auxprocess.h"
+#include "postmaster/interrupt.h"
+#include "storage/aio_subsys.h"
+#include "storage/io_worker.h"
+#include "storage/ipc.h"
+#include "storage/latch.h"
+#include "storage/proc.h"
+#include "tcop/tcopprot.h"
+#include "utils/wait_event.h"
+
+
+/* GUCs */
+int io_workers = 3;
+
+
+void
+IoWorkerMain(const void *startup_data, size_t startup_data_len)
+{
+ sigjmp_buf local_sigjmp_buf;
+
+ MyBackendType = B_IO_WORKER;
+ AuxiliaryProcessMainCommon();
+
+ pqsignal(SIGHUP, SignalHandlerForConfigReload);
+ pqsignal(SIGINT, die); /* to allow manually triggering worker restart */
+
+ /*
+ * Ignore SIGTERM, will get explicit shutdown via SIGUSR2 later in the
+ * shutdown sequence, similar to checkpointer.
+ */
+ pqsignal(SIGTERM, SIG_IGN);
+ /* SIGQUIT handler was already set up by InitPostmasterChild */
+ pqsignal(SIGALRM, SIG_IGN);
+ pqsignal(SIGPIPE, SIG_IGN);
+ pqsignal(SIGUSR1, procsignal_sigusr1_handler);
+ pqsignal(SIGUSR2, SignalHandlerForShutdownRequest);
+
+ /* see PostgresMain() */
+ if (sigsetjmp(local_sigjmp_buf, 1) != 0)
+ {
+ error_context_stack = NULL;
+ HOLD_INTERRUPTS();
+
+ EmitErrorReport();
+
+ proc_exit(1);
+ }
+
+ /* We can now handle ereport(ERROR) */
+ PG_exception_stack = &local_sigjmp_buf;
+
+ sigprocmask(SIG_SETMASK, &UnBlockSig, NULL);
+
+ while (!ShutdownRequestPending)
+ {
+ WaitLatch(MyLatch, WL_LATCH_SET | WL_EXIT_ON_PM_DEATH, -1,
+ WAIT_EVENT_IO_WORKER_MAIN);
+ ResetLatch(MyLatch);
+ CHECK_FOR_INTERRUPTS();
+ }
+
+ proc_exit(0);
+}
+
+bool
+pgaio_workers_enabled(void)
+{
+ /* placeholder for future commit */
+ return false;
+}
diff --git a/src/backend/tcop/postgres.c b/src/backend/tcop/postgres.c
index 55ab2da299b..fcb23239d07 100644
--- a/src/backend/tcop/postgres.c
+++ b/src/backend/tcop/postgres.c
@@ -3316,6 +3316,8 @@ ProcessInterrupts(void)
(errcode(ERRCODE_ADMIN_SHUTDOWN),
errmsg("terminating background worker \"%s\" due to administrator command",
MyBgworkerEntry->bgw_type)));
+ else if (AmIoWorkerProcess())
+ proc_exit(0);
else
ereport(FATAL,
(errcode(ERRCODE_ADMIN_SHUTDOWN),
diff --git a/src/backend/utils/activity/pgstat_backend.c b/src/backend/utils/activity/pgstat_backend.c
index a8cb54a7732..5518a18e060 100644
--- a/src/backend/utils/activity/pgstat_backend.c
+++ b/src/backend/utils/activity/pgstat_backend.c
@@ -375,6 +375,7 @@ pgstat_tracks_backend_bktype(BackendType bktype)
case B_LOGGER:
case B_BG_WRITER:
case B_CHECKPOINTER:
+ case B_IO_WORKER:
case B_STARTUP:
return false;
diff --git a/src/backend/utils/activity/pgstat_io.c b/src/backend/utils/activity/pgstat_io.c
index eb575025596..c8de9c9e2d3 100644
--- a/src/backend/utils/activity/pgstat_io.c
+++ b/src/backend/utils/activity/pgstat_io.c
@@ -376,6 +376,7 @@ pgstat_tracks_io_bktype(BackendType bktype)
case B_BG_WORKER:
case B_BG_WRITER:
case B_CHECKPOINTER:
+ case B_IO_WORKER:
case B_SLOTSYNC_WORKER:
case B_STANDALONE_BACKEND:
case B_STARTUP:
diff --git a/src/backend/utils/activity/wait_event_names.txt b/src/backend/utils/activity/wait_event_names.txt
index b44e4908b25..3f6dc3876b4 100644
--- a/src/backend/utils/activity/wait_event_names.txt
+++ b/src/backend/utils/activity/wait_event_names.txt
@@ -57,6 +57,7 @@ BGWRITER_HIBERNATE "Waiting in background writer process, hibernating."
BGWRITER_MAIN "Waiting in main loop of background writer process."
CHECKPOINTER_MAIN "Waiting in main loop of checkpointer process."
CHECKPOINTER_SHUTDOWN "Waiting for checkpointer process to be terminated."
+IO_WORKER_MAIN "Waiting in main loop of IO Worker process."
LOGICAL_APPLY_MAIN "Waiting in main loop of logical replication apply process."
LOGICAL_LAUNCHER_MAIN "Waiting in main loop of logical replication launcher process."
LOGICAL_PARALLEL_APPLY_MAIN "Waiting in main loop of logical replication parallel apply process."
diff --git a/src/backend/utils/init/miscinit.c b/src/backend/utils/init/miscinit.c
index dc3521457c7..43b4dbccc3d 100644
--- a/src/backend/utils/init/miscinit.c
+++ b/src/backend/utils/init/miscinit.c
@@ -293,6 +293,9 @@ GetBackendTypeDesc(BackendType backendType)
case B_CHECKPOINTER:
backendDesc = gettext_noop("checkpointer");
break;
+ case B_IO_WORKER:
+ backendDesc = gettext_noop("io worker");
+ break;
case B_LOGGER:
backendDesc = gettext_noop("logger");
break;
diff --git a/src/backend/utils/misc/guc_tables.c b/src/backend/utils/misc/guc_tables.c
index 0d3ebf06a95..4cc19bef686 100644
--- a/src/backend/utils/misc/guc_tables.c
+++ b/src/backend/utils/misc/guc_tables.c
@@ -75,6 +75,7 @@
#include "storage/aio.h"
#include "storage/bufmgr.h"
#include "storage/bufpage.h"
+#include "storage/io_worker.h"
#include "storage/large_object.h"
#include "storage/pg_shmem.h"
#include "storage/predicate.h"
@@ -3267,6 +3268,18 @@ struct config_int ConfigureNamesInt[] =
check_io_max_concurrency, NULL, NULL
},
+ {
+ {"io_workers",
+ PGC_SIGHUP,
+ RESOURCES_IO,
+ gettext_noop("Number of IO worker processes, for io_method=worker."),
+ NULL,
+ },
+ &io_workers,
+ 3, 1, MAX_IO_WORKERS,
+ NULL, NULL, NULL
+ },
+
{
{"backend_flush_after", PGC_USERSET, RESOURCES_IO,
gettext_noop("Number of pages after which previously performed writes are flushed to disk."),
diff --git a/src/backend/utils/misc/postgresql.conf.sample b/src/backend/utils/misc/postgresql.conf.sample
index 43c2ec2153e..a4049ff0d9d 100644
--- a/src/backend/utils/misc/postgresql.conf.sample
+++ b/src/backend/utils/misc/postgresql.conf.sample
@@ -207,6 +207,7 @@
# can execute simultaneously
# -1 sets based on shared_buffers
# (change requires restart)
+#io_workers = 3 # 1-32;
# - Worker Processes -
diff --git a/doc/src/sgml/config.sgml b/doc/src/sgml/config.sgml
index 7ec18bb7627..baefb7af4cb 100644
--- a/doc/src/sgml/config.sgml
+++ b/doc/src/sgml/config.sgml
@@ -2689,6 +2689,25 @@ include_dir 'conf.d'
</listitem>
</varlistentry>
+ <varlistentry id="guc-io-workers" xreflabel="io_workers">
+ <term><varname>io_workers</varname> (<type>int</type>)
+ <indexterm>
+ <primary><varname>io_workers</varname> configuration parameter</primary>
+ </indexterm>
+ </term>
+ <listitem>
+ <para>
+ Selects the number of I/O worker processes to use. The default is
+ 3. This parameter can only be set in the
+ <filename>postgresql.conf</filename> file or on the server command
+ line.
+ </para>
+ <para>
+ Only has an effect if <xref linkend="guc-io-method"/> is set to
+ <literal>worker</literal>.
+ </para>
+ </listitem>
+ </varlistentry>
</variablelist>
</sect2>
diff --git a/src/test/regress/expected/stats.out b/src/test/regress/expected/stats.out
index f77caacc17d..cd08a2ca0af 100644
--- a/src/test/regress/expected/stats.out
+++ b/src/test/regress/expected/stats.out
@@ -51,6 +51,14 @@ client backend|relation|vacuum
client backend|temp relation|normal
client backend|wal|init
client backend|wal|normal
+io worker|relation|bulkread
+io worker|relation|bulkwrite
+io worker|relation|init
+io worker|relation|normal
+io worker|relation|vacuum
+io worker|temp relation|normal
+io worker|wal|init
+io worker|wal|normal
slotsync worker|relation|bulkread
slotsync worker|relation|bulkwrite
slotsync worker|relation|init
@@ -87,7 +95,7 @@ walsummarizer|wal|init
walsummarizer|wal|normal
walwriter|wal|init
walwriter|wal|normal
-(71 rows)
+(79 rows)
\a
-- ensure that both seqscan and indexscan plans are allowed
SET enable_seqscan TO on;
--
2.48.1.76.g4e746b1a31.dirty
[text/x-diff] v2.9-0004-aio-Add-io_method-worker.patch (20.2K, ../../oav52ytakcvfnk52w5o5wvxdjavxth2tgptxt25gnfmh2pdcpk@omp3xukdu5if/5-v2.9-0004-aio-Add-io_method-worker.patch)
download | inline diff:
From 5fb2be750b2513df5a88769051b114f530ded0fc Mon Sep 17 00:00:00 2001
From: Andres Freund <andres@anarazel.de>
Date: Sat, 15 Mar 2025 12:29:57 -0400
Subject: [PATCH v2.9 04/30] aio: Add io_method=worker
The previous commit introduced the infrastructure to start io_workers. This
commit actually makes the workers execute IOs.
IO workers consume IOs from a shared memory submission queue, run traditional
synchronous system calls, and perform the shared completion handling
immediately. Client code submits most requests by pushing IOs into the
submission queue, and waits (if necessary) using condition variables. Some
IOs cannot be performed in another process due to lack of infrastructure for
reopening the file, and must processed synchronously by the client code when
submitted.
For now the default io_method is changed to "worker". We should re-evaluate
that around beta1, we might want to be careful and set the default to "sync"
for 18.
Reviewed-by: Noah Misch <noah@leadboat.com>
Co-authored-by: Thomas Munro <thomas.munro@gmail.com>
Co-authored-by: Andres Freund <andres@anarazel.de>
Discussion: https://postgr.es/m/uvrtrknj4kdytuboidbhwclo4gxhswwcpgadptsjvjqcluzmah%40brqs62irg4dt
Discussion: https://postgr.es/m/20210223100344.llw5an2aklengrmn@alap3.anarazel.de
Discussion: https://postgr.es/m/stj36ea6yyhoxtqkhpieia2z4krnam7qyetc57rfezgk4zgapf@gcnactj4z56m
---
src/include/storage/aio.h | 5 +-
src/include/storage/aio_internal.h | 1 +
src/include/storage/lwlocklist.h | 1 +
src/backend/storage/aio/aio.c | 2 +
src/backend/storage/aio/aio_init.c | 9 +
src/backend/storage/aio/method_worker.c | 431 +++++++++++++++++-
.../utils/activity/wait_event_names.txt | 1 +
src/backend/utils/misc/postgresql.conf.sample | 2 +-
doc/src/sgml/config.sgml | 5 +
src/tools/pgindent/typedefs.list | 3 +
10 files changed, 452 insertions(+), 8 deletions(-)
diff --git a/src/include/storage/aio.h b/src/include/storage/aio.h
index f48a4962089..7b6b7d20a85 100644
--- a/src/include/storage/aio.h
+++ b/src/include/storage/aio.h
@@ -27,10 +27,11 @@
typedef enum IoMethod
{
IOMETHOD_SYNC = 0,
+ IOMETHOD_WORKER,
} IoMethod;
-/* We'll default to synchronous execution. */
-#define DEFAULT_IO_METHOD IOMETHOD_SYNC
+/* We'll default to worker based execution. */
+#define DEFAULT_IO_METHOD IOMETHOD_WORKER
/*
diff --git a/src/include/storage/aio_internal.h b/src/include/storage/aio_internal.h
index 52b7751efa8..51f63586467 100644
--- a/src/include/storage/aio_internal.h
+++ b/src/include/storage/aio_internal.h
@@ -385,6 +385,7 @@ extern PgAioHandle *pgaio_inj_io_get(void);
/* Declarations for the tables of function pointers exposed by each IO method. */
extern PGDLLIMPORT const IoMethodOps pgaio_sync_ops;
+extern PGDLLIMPORT const IoMethodOps pgaio_worker_ops;
extern PGDLLIMPORT const IoMethodOps *pgaio_method_ops;
extern PGDLLIMPORT PgAioCtl *pgaio_ctl;
diff --git a/src/include/storage/lwlocklist.h b/src/include/storage/lwlocklist.h
index cf565452382..932024b1b0b 100644
--- a/src/include/storage/lwlocklist.h
+++ b/src/include/storage/lwlocklist.h
@@ -83,3 +83,4 @@ PG_LWLOCK(49, WALSummarizer)
PG_LWLOCK(50, DSMRegistry)
PG_LWLOCK(51, InjectionPoint)
PG_LWLOCK(52, SerialControl)
+PG_LWLOCK(53, AioWorkerSubmissionQueue)
diff --git a/src/backend/storage/aio/aio.c b/src/backend/storage/aio/aio.c
index 4d5439c73fd..3ed4b1dfdac 100644
--- a/src/backend/storage/aio/aio.c
+++ b/src/backend/storage/aio/aio.c
@@ -64,6 +64,7 @@ static void pgaio_io_wait(PgAioHandle *ioh, uint64 ref_generation);
/* Options for io_method. */
const struct config_enum_entry io_method_options[] = {
{"sync", IOMETHOD_SYNC, false},
+ {"worker", IOMETHOD_WORKER, false},
{NULL, 0, false}
};
@@ -80,6 +81,7 @@ PgAioBackend *pgaio_my_backend;
static const IoMethodOps *const pgaio_method_ops_table[] = {
[IOMETHOD_SYNC] = &pgaio_sync_ops,
+ [IOMETHOD_WORKER] = &pgaio_worker_ops,
};
/* callbacks for the configured io_method, set by assign_io_method */
diff --git a/src/backend/storage/aio/aio_init.c b/src/backend/storage/aio/aio_init.c
index 6fe55510fae..4e405ce7ca8 100644
--- a/src/backend/storage/aio/aio_init.c
+++ b/src/backend/storage/aio/aio_init.c
@@ -18,6 +18,7 @@
#include "storage/aio.h"
#include "storage/aio_internal.h"
#include "storage/aio_subsys.h"
+#include "storage/io_worker.h"
#include "storage/ipc.h"
#include "storage/proc.h"
#include "storage/shmem.h"
@@ -39,6 +40,11 @@ AioCtlShmemSize(void)
static uint32
AioProcs(void)
{
+ /*
+ * While AIO workers don't need their own AIO context, we can't currently
+ * guarantee nothing gets assigned to the a ProcNumber for an IO worker if
+ * we just subtracted MAX_IO_WORKERS.
+ */
return MaxBackends + NUM_AUXILIARY_PROCS;
}
@@ -223,6 +229,9 @@ pgaio_init_backend(void)
/* shouldn't be initialized twice */
Assert(!pgaio_my_backend);
+ if (MyBackendType == B_IO_WORKER)
+ return;
+
if (MyProc == NULL || MyProcNumber >= AioProcs())
elog(ERROR, "aio requires a normal PGPROC");
diff --git a/src/backend/storage/aio/method_worker.c b/src/backend/storage/aio/method_worker.c
index 0ef9ef93e2b..117c9a87db7 100644
--- a/src/backend/storage/aio/method_worker.c
+++ b/src/backend/storage/aio/method_worker.c
@@ -3,6 +3,21 @@
* method_worker.c
* AIO - perform AIO using worker processes
*
+ * IO workers consume IOs from a shared memory submission queue, run
+ * traditional synchronous system calls, and perform the shared completion
+ * handling immediately. Client code submits most requests by pushing IOs
+ * into the submission queue, and waits (if necessary) using condition
+ * variables. Some IOs cannot be performed in another process due to lack of
+ * infrastructure for reopening the file, and must processed synchronously by
+ * the client code when submitted.
+ *
+ * So that the submitter can make just one system call when submitting a batch
+ * of IOs, wakeups "fan out"; each woken IO worker can wake two more. XXX This
+ * could be improved by using futexes instead of latches to wake N waiters.
+ *
+ * This method of AIO is available in all builds on all operating systems, and
+ * is the default.
+ *
* Portions Copyright (c) 1996-2025, PostgreSQL Global Development Group
* Portions Copyright (c) 1994, Regents of the University of California
*
@@ -16,25 +31,325 @@
#include "libpq/pqsignal.h"
#include "miscadmin.h"
+#include "port/pg_bitutils.h"
#include "postmaster/auxprocess.h"
#include "postmaster/interrupt.h"
+#include "storage/aio.h"
+#include "storage/aio_internal.h"
#include "storage/aio_subsys.h"
#include "storage/io_worker.h"
#include "storage/ipc.h"
#include "storage/latch.h"
#include "storage/proc.h"
#include "tcop/tcopprot.h"
+#include "utils/ps_status.h"
#include "utils/wait_event.h"
+/* How many workers should each worker wake up if needed? */
+#define IO_WORKER_WAKEUP_FANOUT 2
+
+
+typedef struct AioWorkerSubmissionQueue
+{
+ uint32 size;
+ uint32 mask;
+ uint32 head;
+ uint32 tail;
+ uint32 ios[FLEXIBLE_ARRAY_MEMBER];
+} AioWorkerSubmissionQueue;
+
+typedef struct AioWorkerSlot
+{
+ Latch *latch;
+ bool in_use;
+} AioWorkerSlot;
+
+typedef struct AioWorkerControl
+{
+ uint64 idle_worker_mask;
+ AioWorkerSlot workers[FLEXIBLE_ARRAY_MEMBER];
+} AioWorkerControl;
+
+
+static size_t pgaio_worker_shmem_size(void);
+static void pgaio_worker_shmem_init(bool first_time);
+
+static bool pgaio_worker_needs_synchronous_execution(PgAioHandle *ioh);
+static int pgaio_worker_submit(uint16 num_staged_ios, PgAioHandle **staged_ios);
+
+
+const IoMethodOps pgaio_worker_ops = {
+ .shmem_size = pgaio_worker_shmem_size,
+ .shmem_init = pgaio_worker_shmem_init,
+
+ .needs_synchronous_execution = pgaio_worker_needs_synchronous_execution,
+ .submit = pgaio_worker_submit,
+};
+
+
/* GUCs */
int io_workers = 3;
+static int io_worker_queue_size = 64;
+static int MyIoWorkerId;
+static AioWorkerSubmissionQueue *io_worker_submission_queue;
+static AioWorkerControl *io_worker_control;
+
+
+static size_t
+pgaio_worker_shmem_size(void)
+{
+ return
+ offsetof(AioWorkerSubmissionQueue, ios) +
+ sizeof(uint32) * MAX_IO_WORKERS * io_max_concurrency +
+ offsetof(AioWorkerControl, workers) +
+ sizeof(AioWorkerSlot) * MAX_IO_WORKERS;
+}
+
+static void
+pgaio_worker_shmem_init(bool first_time)
+{
+ bool found;
+ int size;
+
+ /* Round size up to next power of two so we can make a mask. */
+ size = pg_nextpower2_32(io_worker_queue_size);
+
+ io_worker_submission_queue =
+ ShmemInitStruct("AioWorkerSubmissionQueue",
+ offsetof(AioWorkerSubmissionQueue, ios) +
+ sizeof(uint32) * size,
+ &found);
+ if (!found)
+ {
+ io_worker_submission_queue->size = size;
+ io_worker_submission_queue->head = 0;
+ io_worker_submission_queue->tail = 0;
+ }
+
+ io_worker_control =
+ ShmemInitStruct("AioWorkerControl",
+ offsetof(AioWorkerControl, workers) +
+ sizeof(AioWorkerSlot) * io_workers,
+ &found);
+ if (!found)
+ {
+ io_worker_control->idle_worker_mask = 0;
+ for (int i = 0; i < io_workers; ++i)
+ {
+ io_worker_control->workers[i].latch = NULL;
+ io_worker_control->workers[i].in_use = false;
+ }
+ }
+}
+
+static int
+pgaio_choose_idle_worker(void)
+{
+ int worker;
+
+ if (io_worker_control->idle_worker_mask == 0)
+ return -1;
+
+ /* Find the lowest bit position, and clear it. */
+ worker = pg_rightmost_one_pos64(io_worker_control->idle_worker_mask);
+ io_worker_control->idle_worker_mask &= ~(UINT64_C(1) << worker);
+
+ return worker;
+}
+
+static bool
+pgaio_worker_submission_queue_insert(PgAioHandle *ioh)
+{
+ AioWorkerSubmissionQueue *queue;
+ uint32 new_head;
+
+ queue = io_worker_submission_queue;
+ new_head = (queue->head + 1) & (queue->size - 1);
+ if (new_head == queue->tail)
+ {
+ pgaio_debug(DEBUG3, "io queue is full, at %u elements",
+ io_worker_submission_queue->size);
+ return false; /* full */
+ }
+
+ queue->ios[queue->head] = pgaio_io_get_id(ioh);
+ queue->head = new_head;
+
+ return true;
+}
+
+static uint32
+pgaio_worker_submission_queue_consume(void)
+{
+ AioWorkerSubmissionQueue *queue;
+ uint32 result;
+
+ queue = io_worker_submission_queue;
+ if (queue->tail == queue->head)
+ return UINT32_MAX; /* empty */
+
+ result = queue->ios[queue->tail];
+ queue->tail = (queue->tail + 1) & (queue->size - 1);
+
+ return result;
+}
+
+static uint32
+pgaio_worker_submission_queue_depth(void)
+{
+ uint32 head;
+ uint32 tail;
+
+ head = io_worker_submission_queue->head;
+ tail = io_worker_submission_queue->tail;
+
+ if (tail > head)
+ head += io_worker_submission_queue->size;
+
+ Assert(head >= tail);
+
+ return head - tail;
+}
+
+static bool
+pgaio_worker_needs_synchronous_execution(PgAioHandle *ioh)
+{
+ return
+ !IsUnderPostmaster
+ || ioh->flags & PGAIO_HF_REFERENCES_LOCAL
+ || !pgaio_io_can_reopen(ioh);
+}
+
+static void
+pgaio_worker_submit_internal(int nios, PgAioHandle *ios[])
+{
+ PgAioHandle *synchronous_ios[PGAIO_SUBMIT_BATCH_SIZE];
+ int nsync = 0;
+ Latch *wakeup = NULL;
+ int worker;
+
+ Assert(nios <= PGAIO_SUBMIT_BATCH_SIZE);
+
+ LWLockAcquire(AioWorkerSubmissionQueueLock, LW_EXCLUSIVE);
+ for (int i = 0; i < nios; ++i)
+ {
+ Assert(!pgaio_worker_needs_synchronous_execution(ios[i]));
+ if (!pgaio_worker_submission_queue_insert(ios[i]))
+ {
+ /*
+ * We'll do it synchronously, but only after we've sent as many as
+ * we can to workers, to maximize concurrency.
+ */
+ synchronous_ios[nsync++] = ios[i];
+ continue;
+ }
+
+ if (wakeup == NULL)
+ {
+ /* Choose an idle worker to wake up if we haven't already. */
+ worker = pgaio_choose_idle_worker();
+ if (worker >= 0)
+ wakeup = io_worker_control->workers[worker].latch;
+
+ pgaio_debug_io(DEBUG4, ios[i],
+ "choosing worker %d",
+ worker);
+ }
+ }
+ LWLockRelease(AioWorkerSubmissionQueueLock);
+
+ if (wakeup)
+ SetLatch(wakeup);
+
+ /* Run whatever is left synchronously. */
+ if (nsync > 0)
+ {
+ for (int i = 0; i < nsync; ++i)
+ {
+ pgaio_io_perform_synchronously(synchronous_ios[i]);
+ }
+ }
+}
+
+static int
+pgaio_worker_submit(uint16 num_staged_ios, PgAioHandle **staged_ios)
+{
+ for (int i = 0; i < num_staged_ios; i++)
+ {
+ PgAioHandle *ioh = staged_ios[i];
+
+ pgaio_io_prepare_submit(ioh);
+ }
+
+ pgaio_worker_submit_internal(num_staged_ios, staged_ios);
+
+ return num_staged_ios;
+}
+
+/*
+ * on_shmem_exit() callback that releases the worker's slot in
+ * io_worker_control.
+ */
+static void
+pgaio_worker_die(int code, Datum arg)
+{
+ LWLockAcquire(AioWorkerSubmissionQueueLock, LW_EXCLUSIVE);
+ Assert(io_worker_control->workers[MyIoWorkerId].in_use);
+ Assert(io_worker_control->workers[MyIoWorkerId].latch == MyLatch);
+
+ io_worker_control->workers[MyIoWorkerId].in_use = false;
+ io_worker_control->workers[MyIoWorkerId].latch = NULL;
+ LWLockRelease(AioWorkerSubmissionQueueLock);
+}
+
+/*
+ * Register the worker in shared memory, assign MyWorkerId and register a
+ * shutdown callback to release registration.
+ */
+static void
+pgaio_worker_register(void)
+{
+ MyIoWorkerId = -1;
+
+ /*
+ * XXX: This could do with more fine-grained locking. But it's also not
+ * very common for the number of workers to change at the moment...
+ */
+ LWLockAcquire(AioWorkerSubmissionQueueLock, LW_EXCLUSIVE);
+
+ for (int i = 0; i < io_workers; ++i)
+ {
+ if (!io_worker_control->workers[i].in_use)
+ {
+ Assert(io_worker_control->workers[i].latch == NULL);
+ io_worker_control->workers[i].in_use = true;
+ MyIoWorkerId = i;
+ break;
+ }
+ else
+ Assert(io_worker_control->workers[i].latch != NULL);
+ }
+
+ if (MyIoWorkerId == -1)
+ elog(ERROR, "couldn't find a free worker slot");
+
+ io_worker_control->idle_worker_mask |= (UINT64_C(1) << MyIoWorkerId);
+ io_worker_control->workers[MyIoWorkerId].latch = MyLatch;
+ LWLockRelease(AioWorkerSubmissionQueueLock);
+
+ on_shmem_exit(pgaio_worker_die, 0);
+}
+
void
IoWorkerMain(const void *startup_data, size_t startup_data_len)
{
sigjmp_buf local_sigjmp_buf;
+ PgAioHandle *volatile error_ioh = NULL;
+ volatile int error_errno = 0;
+ char cmd[128];
MyBackendType = B_IO_WORKER;
AuxiliaryProcessMainCommon();
@@ -53,6 +368,12 @@ IoWorkerMain(const void *startup_data, size_t startup_data_len)
pqsignal(SIGUSR1, procsignal_sigusr1_handler);
pqsignal(SIGUSR2, SignalHandlerForShutdownRequest);
+ /* also registers a shutdown callback to unregister */
+ pgaio_worker_register();
+
+ sprintf(cmd, "io worker: %d", MyIoWorkerId);
+ set_ps_display(cmd);
+
/* see PostgresMain() */
if (sigsetjmp(local_sigjmp_buf, 1) != 0)
{
@@ -61,6 +382,27 @@ IoWorkerMain(const void *startup_data, size_t startup_data_len)
EmitErrorReport();
+ /*
+ * In the - very unlikely - case that the IO failed in a way that
+ * raises an error we need to mark the IO as failed.
+ *
+ * Need to do just enough error recovery so that we can mark the IO as
+ * failed and then exit (postmaster will start a new worker).
+ */
+ LWLockReleaseAll();
+
+ if (error_ioh != NULL)
+ {
+ /* should never fail without setting error_errno */
+ Assert(error_errno != 0);
+
+ errno = error_errno;
+
+ START_CRIT_SECTION();
+ pgaio_io_process_completion(error_ioh, -error_errno);
+ END_CRIT_SECTION();
+ }
+
proc_exit(1);
}
@@ -71,9 +413,89 @@ IoWorkerMain(const void *startup_data, size_t startup_data_len)
while (!ShutdownRequestPending)
{
- WaitLatch(MyLatch, WL_LATCH_SET | WL_EXIT_ON_PM_DEATH, -1,
- WAIT_EVENT_IO_WORKER_MAIN);
- ResetLatch(MyLatch);
+ uint32 io_index;
+ Latch *latches[IO_WORKER_WAKEUP_FANOUT];
+ int nlatches = 0;
+ int nwakeups = 0;
+ int worker;
+
+ /* Try to get a job to do. */
+ LWLockAcquire(AioWorkerSubmissionQueueLock, LW_EXCLUSIVE);
+ if ((io_index = pgaio_worker_submission_queue_consume()) == UINT32_MAX)
+ {
+ /*
+ * Nothing to do. Mark self idle.
+ *
+ * XXX: Invent some kind of back pressure to reduce useless
+ * wakeups?
+ */
+ io_worker_control->idle_worker_mask |= (UINT64_C(1) << MyIoWorkerId);
+ }
+ else
+ {
+ /* Got one. Clear idle flag. */
+ io_worker_control->idle_worker_mask &= ~(UINT64_C(1) << MyIoWorkerId);
+
+ /* See if we can wake up some peers. */
+ nwakeups = Min(pgaio_worker_submission_queue_depth(),
+ IO_WORKER_WAKEUP_FANOUT);
+ for (int i = 0; i < nwakeups; ++i)
+ {
+ if ((worker = pgaio_choose_idle_worker()) < 0)
+ break;
+ latches[nlatches++] = io_worker_control->workers[worker].latch;
+ }
+ }
+ LWLockRelease(AioWorkerSubmissionQueueLock);
+
+ for (int i = 0; i < nlatches; ++i)
+ SetLatch(latches[i]);
+
+ if (io_index != UINT32_MAX)
+ {
+ PgAioHandle *ioh = NULL;
+
+ ioh = &pgaio_ctl->io_handles[io_index];
+ error_ioh = ioh;
+
+ pgaio_debug_io(DEBUG4, ioh,
+ "worker %d processing IO",
+ MyIoWorkerId);
+
+ /*
+ * It's very unlikely, but possible, that reopen fails. E.g. due
+ * to memory allocations failing or file permissions changing or
+ * such. In that case we need to fail the IO.
+ *
+ * There's not really a good errno we can report here.
+ */
+ error_errno = ENOENT;
+ pgaio_io_reopen(ioh);
+
+ /*
+ * To be able to exercise the reopen-fails path, allow injection
+ * points to trigger a failure at this point.
+ */
+ pgaio_io_call_inj(ioh, "AIO_WORKER_AFTER_REOPEN");
+
+ error_errno = 0;
+ error_ioh = NULL;
+
+ /*
+ * We don't expect this to ever fail with ERROR or FATAL, no need
+ * to keep error_ioh set to the IO.
+ * pgaio_io_perform_synchronously() contains a critical section to
+ * ensure we don't accidentally fail.
+ */
+ pgaio_io_perform_synchronously(ioh);
+ }
+ else
+ {
+ WaitLatch(MyLatch, WL_LATCH_SET | WL_EXIT_ON_PM_DEATH, -1,
+ WAIT_EVENT_IO_WORKER_MAIN);
+ ResetLatch(MyLatch);
+ }
+
CHECK_FOR_INTERRUPTS();
}
@@ -83,6 +505,5 @@ IoWorkerMain(const void *startup_data, size_t startup_data_len)
bool
pgaio_workers_enabled(void)
{
- /* placeholder for future commit */
- return false;
+ return io_method == IOMETHOD_WORKER;
}
diff --git a/src/backend/utils/activity/wait_event_names.txt b/src/backend/utils/activity/wait_event_names.txt
index 3f6dc3876b4..9fa12a555e8 100644
--- a/src/backend/utils/activity/wait_event_names.txt
+++ b/src/backend/utils/activity/wait_event_names.txt
@@ -348,6 +348,7 @@ WALSummarizer "Waiting to read or update WAL summarization state."
DSMRegistry "Waiting to read or update the dynamic shared memory registry."
InjectionPoint "Waiting to read or update information related to injection points."
SerialControl "Waiting to read or update shared <filename>pg_serial</filename> state."
+AioWorkerSubmissionQueue "Waiting to access AIO worker submission queue."
#
# END OF PREDEFINED LWLOCKS (DO NOT CHANGE THIS LINE)
diff --git a/src/backend/utils/misc/postgresql.conf.sample b/src/backend/utils/misc/postgresql.conf.sample
index a4049ff0d9d..75a844a9474 100644
--- a/src/backend/utils/misc/postgresql.conf.sample
+++ b/src/backend/utils/misc/postgresql.conf.sample
@@ -202,7 +202,7 @@
#maintenance_io_concurrency = 10 # 1-1000; 0 disables prefetching
#io_combine_limit = 128kB # usually 1-32 blocks (depends on OS)
-#io_method = sync # sync (change requires restart)
+#io_method = worker # worker, sync (change requires restart)
#io_max_concurrency = -1 # Max number of IOs that one process
# can execute simultaneously
# -1 sets based on shared_buffers
diff --git a/doc/src/sgml/config.sgml b/doc/src/sgml/config.sgml
index baefb7af4cb..108caea1fed 100644
--- a/doc/src/sgml/config.sgml
+++ b/doc/src/sgml/config.sgml
@@ -2676,6 +2676,11 @@ include_dir 'conf.d'
Selects the method for executing asynchronous I/O.
Possible values are:
<itemizedlist>
+ <listitem>
+ <para>
+ <literal>worker</literal> (execute asynchronous I/O using worker processes)
+ </para>
+ </listitem>
<listitem>
<para>
<literal>sync</literal> (execute asynchronous-eligible I/O synchronously)
diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list
index f4261145353..4d8b5c7f1d6 100644
--- a/src/tools/pgindent/typedefs.list
+++ b/src/tools/pgindent/typedefs.list
@@ -55,6 +55,9 @@ AggStrategy
AggTransInfo
Aggref
AggregateInstrumentation
+AioWorkerControl
+AioWorkerSlot
+AioWorkerSubmissionQueue
AlenState
Alias
AllocBlock
--
2.48.1.76.g4e746b1a31.dirty
[text/x-diff] v2.9-0005-aio-Add-liburing-dependency.patch (13.4K, ../../oav52ytakcvfnk52w5o5wvxdjavxth2tgptxt25gnfmh2pdcpk@omp3xukdu5if/6-v2.9-0005-aio-Add-liburing-dependency.patch)
download | inline diff:
From 958c27172c8945c072c475b28e534cc0ebce74b4 Mon Sep 17 00:00:00 2001
From: Andres Freund <andres@anarazel.de>
Date: Sat, 15 Mar 2025 12:29:57 -0400
Subject: [PATCH v2.9 05/30] aio: Add liburing dependency
Will be used in a subsequent commit, to implement io_method=io_uring. Kept
separate for easier review.
Reviewed-by: Noah Misch <noah@leadboat.com>
Discussion: https://postgr.es/m/uvrtrknj4kdytuboidbhwclo4gxhswwcpgadptsjvjqcluzmah%40brqs62irg4dt
---
meson.build | 14 ++++
meson_options.txt | 3 +
configure.ac | 11 +++
src/makefiles/meson.build | 3 +
src/include/pg_config.h.in | 3 +
src/backend/Makefile | 7 +-
doc/src/sgml/installation.sgml | 34 ++++++++
configure | 138 +++++++++++++++++++++++++++++++++
.cirrus.tasks.yml | 1 +
src/Makefile.global.in | 4 +
10 files changed, 215 insertions(+), 3 deletions(-)
diff --git a/meson.build b/meson.build
index 13c13748e5d..bed36153c7b 100644
--- a/meson.build
+++ b/meson.build
@@ -950,6 +950,18 @@ endif
+###############################################################
+# Library: liburing
+###############################################################
+
+liburingopt = get_option('liburing')
+liburing = dependency('liburing', required: liburingopt)
+if liburing.found()
+ cdata.set('USE_LIBURING', 1)
+endif
+
+
+
###############################################################
# Library: libxml
###############################################################
@@ -3168,6 +3180,7 @@ backend_both_deps += [
icu_i18n,
ldap,
libintl,
+ liburing,
libxml,
lz4,
pam,
@@ -3823,6 +3836,7 @@ if meson.version().version_compare('>=0.57')
'icu': icu,
'ldap': ldap,
'libcurl': libcurl,
+ 'liburing': liburing,
'libxml': libxml,
'libxslt': libxslt,
'llvm': llvm,
diff --git a/meson_options.txt b/meson_options.txt
index 702c4517145..dd7126da3a7 100644
--- a/meson_options.txt
+++ b/meson_options.txt
@@ -106,6 +106,9 @@ option('libcurl', type : 'feature', value: 'auto',
option('libedit_preferred', type: 'boolean', value: false,
description: 'Prefer BSD Libedit over GNU Readline')
+option('liburing', type : 'feature', value: 'auto',
+ description: 'io_uring support, for asynchronous I/O')
+
option('libxml', type: 'feature', value: 'auto',
description: 'XML support')
diff --git a/configure.ac b/configure.ac
index b6d02f5ecc7..3e2d0dc59c0 100644
--- a/configure.ac
+++ b/configure.ac
@@ -975,6 +975,14 @@ AC_SUBST(with_readline)
PGAC_ARG_BOOL(with, libedit-preferred, no,
[prefer BSD Libedit over GNU Readline])
+#
+# liburing
+#
+AC_MSG_CHECKING([whether to build with liburing support])
+PGAC_ARG_BOOL(with, liburing, no, [io_uring support, for asynchronous I/O],
+ [AC_DEFINE([USE_LIBURING], 1, [Define to build with io_uring support. (--with-liburing)])])
+AC_MSG_RESULT([$with_liburing])
+AC_SUBST(with_liburing)
#
# UUID library
@@ -1463,6 +1471,9 @@ elif test "$with_uuid" = ossp ; then
fi
AC_SUBST(UUID_LIBS)
+if test "$with_liburing" = yes; then
+ PKG_CHECK_MODULES(LIBURING, liburing)
+fi
##
## Header files
diff --git a/src/makefiles/meson.build b/src/makefiles/meson.build
index 60e13d50235..46d8da070e8 100644
--- a/src/makefiles/meson.build
+++ b/src/makefiles/meson.build
@@ -199,6 +199,8 @@ pgxs_empty = [
'PTHREAD_CFLAGS', 'PTHREAD_LIBS',
'ICU_LIBS',
+
+ 'LIBURING_CFLAGS', 'LIBURING_LIBS',
]
if host_system == 'windows' and cc.get_argument_syntax() != 'msvc'
@@ -230,6 +232,7 @@ pgxs_deps = {
'icu': icu,
'ldap': ldap,
'libcurl': libcurl,
+ 'liburing': liburing,
'libxml': libxml,
'libxslt': libxslt,
'llvm': llvm,
diff --git a/src/include/pg_config.h.in b/src/include/pg_config.h.in
index db6454090d2..f2422241133 100644
--- a/src/include/pg_config.h.in
+++ b/src/include/pg_config.h.in
@@ -672,6 +672,9 @@
/* Define to 1 to build with libcurl support. (--with-libcurl) */
#undef USE_LIBCURL
+/* Define to build with io_uring support. (--with-liburing) */
+#undef USE_LIBURING
+
/* Define to 1 to build with XML support. (--with-libxml) */
#undef USE_LIBXML
diff --git a/src/backend/Makefile b/src/backend/Makefile
index 42d4a28e5aa..7344c8c7f5c 100644
--- a/src/backend/Makefile
+++ b/src/backend/Makefile
@@ -43,9 +43,10 @@ OBJS = \
$(top_builddir)/src/common/libpgcommon_srv.a \
$(top_builddir)/src/port/libpgport_srv.a
-# We put libpgport and libpgcommon into OBJS, so remove it from LIBS; also add
-# libldap and ICU
-LIBS := $(filter-out -lpgport -lpgcommon, $(LIBS)) $(LDAP_LIBS_BE) $(ICU_LIBS)
+# We put libpgport and libpgcommon into OBJS, so remove it from LIBS.
+LIBS := $(filter-out -lpgport -lpgcommon, $(LIBS))
+# The backend conditionally needs libraries that most executables don't need.
+LIBS += $(LDAP_LIBS_BE) $(ICU_LIBS) $(LIBURING_LIBS)
# The backend doesn't need everything that's in LIBS, however
LIBS := $(filter-out -lreadline -ledit -ltermcap -lncurses -lcurses, $(LIBS))
diff --git a/doc/src/sgml/installation.sgml b/doc/src/sgml/installation.sgml
index e076cefa3b9..cc28f041330 100644
--- a/doc/src/sgml/installation.sgml
+++ b/doc/src/sgml/installation.sgml
@@ -1156,6 +1156,24 @@ build-postgresql:
</listitem>
</varlistentry>
+ <varlistentry id="configure-option-with-liburing">
+ <term><option>--with-liburing</option></term>
+ <listitem>
+ <para>
+ Build with liburing, enabling io_uring support for asynchronous I/O.
+ </para>
+ <para>
+ To detect the required compiler and linker options, PostgreSQL will
+ query <command>pkg-config</command>.
+ </para>
+ <para>
+ To use a liburing installation that is in an unusual location, you
+ can set <command>pkg-config</command>-related environment
+ variables (see its documentation).
+ </para>
+ </listitem>
+ </varlistentry>
+
<varlistentry id="configure-option-with-libxml">
<term><option>--with-libxml</option></term>
<listitem>
@@ -2611,6 +2629,22 @@ ninja install
</listitem>
</varlistentry>
+ <varlistentry id="configure-with-liburing-meson">
+ <term><option>-Dliburing={ auto | enabled | disabled }</option></term>
+ <listitem>
+ <para>
+ Build with liburing, enabling io_uring support for asynchronous I/O.
+ Defaults to auto.
+ </para>
+
+ <para>
+ To use a liburing installation that is in an unusual location, you
+ can set <command>pkg-config</command>-related environment
+ variables (see its documentation).
+ </para>
+ </listitem>
+ </varlistentry>
+
<varlistentry id="configure-with-libxml-meson">
<term><option>-Dlibxml={ auto | enabled | disabled }</option></term>
<listitem>
diff --git a/configure b/configure
index 93fddd69981..8ffcdfc2e1c 100755
--- a/configure
+++ b/configure
@@ -651,6 +651,8 @@ LIBOBJS
OPENSSL
ZSTD
LZ4
+LIBURING_LIBS
+LIBURING_CFLAGS
UUID_LIBS
LDAP_LIBS_BE
LDAP_LIBS_FE
@@ -712,6 +714,7 @@ LIBCURL_LIBS
LIBCURL_CFLAGS
with_libcurl
with_uuid
+with_liburing
with_readline
with_systemd
with_selinux
@@ -865,6 +868,7 @@ with_selinux
with_systemd
with_readline
with_libedit_preferred
+with_liburing
with_uuid
with_ossp_uuid
with_libcurl
@@ -911,6 +915,8 @@ LDFLAGS_EX
LDFLAGS_SL
PERL
PYTHON
+LIBURING_CFLAGS
+LIBURING_LIBS
MSGFMT
TCLSH'
@@ -1578,6 +1584,7 @@ Optional Packages:
--without-readline do not use GNU Readline nor BSD Libedit for editing
--with-libedit-preferred
prefer BSD Libedit over GNU Readline
+ --with-liburing io_uring support, for asynchronous I/O
--with-uuid=LIB build contrib/uuid-ossp using LIB (bsd,e2fs,ossp)
--with-ossp-uuid obsolete spelling of --with-uuid=ossp
--with-libcurl build with libcurl support
@@ -1629,6 +1636,10 @@ Some influential environment variables:
LDFLAGS_SL extra linker flags for linking shared libraries only
PERL Perl program
PYTHON Python program
+ LIBURING_CFLAGS
+ C compiler flags for LIBURING, overriding pkg-config
+ LIBURING_LIBS
+ linker flags for LIBURING, overriding pkg-config
MSGFMT msgfmt program for NLS
TCLSH Tcl interpreter program (tclsh)
@@ -8692,6 +8703,40 @@ fi
+#
+# liburing
+#
+{ $as_echo "$as_me:${as_lineno-$LINENO}: checking whether to build with liburing support" >&5
+$as_echo_n "checking whether to build with liburing support... " >&6; }
+
+
+
+# Check whether --with-liburing was given.
+if test "${with_liburing+set}" = set; then :
+ withval=$with_liburing;
+ case $withval in
+ yes)
+
+$as_echo "#define USE_LIBURING 1" >>confdefs.h
+
+ ;;
+ no)
+ :
+ ;;
+ *)
+ as_fn_error $? "no argument expected for --with-liburing option" "$LINENO" 5
+ ;;
+ esac
+
+else
+ with_liburing=no
+
+fi
+
+
+{ $as_echo "$as_me:${as_lineno-$LINENO}: result: $with_liburing" >&5
+$as_echo "$with_liburing" >&6; }
+
#
# UUID library
@@ -13444,6 +13489,99 @@ fi
fi
+if test "$with_liburing" = yes; then
+
+pkg_failed=no
+{ $as_echo "$as_me:${as_lineno-$LINENO}: checking for liburing" >&5
+$as_echo_n "checking for liburing... " >&6; }
+
+if test -n "$LIBURING_CFLAGS"; then
+ pkg_cv_LIBURING_CFLAGS="$LIBURING_CFLAGS"
+ elif test -n "$PKG_CONFIG"; then
+ if test -n "$PKG_CONFIG" && \
+ { { $as_echo "$as_me:${as_lineno-$LINENO}: \$PKG_CONFIG --exists --print-errors \"liburing\""; } >&5
+ ($PKG_CONFIG --exists --print-errors "liburing") 2>&5
+ ac_status=$?
+ $as_echo "$as_me:${as_lineno-$LINENO}: \$? = $ac_status" >&5
+ test $ac_status = 0; }; then
+ pkg_cv_LIBURING_CFLAGS=`$PKG_CONFIG --cflags "liburing" 2>/dev/null`
+ test "x$?" != "x0" && pkg_failed=yes
+else
+ pkg_failed=yes
+fi
+ else
+ pkg_failed=untried
+fi
+if test -n "$LIBURING_LIBS"; then
+ pkg_cv_LIBURING_LIBS="$LIBURING_LIBS"
+ elif test -n "$PKG_CONFIG"; then
+ if test -n "$PKG_CONFIG" && \
+ { { $as_echo "$as_me:${as_lineno-$LINENO}: \$PKG_CONFIG --exists --print-errors \"liburing\""; } >&5
+ ($PKG_CONFIG --exists --print-errors "liburing") 2>&5
+ ac_status=$?
+ $as_echo "$as_me:${as_lineno-$LINENO}: \$? = $ac_status" >&5
+ test $ac_status = 0; }; then
+ pkg_cv_LIBURING_LIBS=`$PKG_CONFIG --libs "liburing" 2>/dev/null`
+ test "x$?" != "x0" && pkg_failed=yes
+else
+ pkg_failed=yes
+fi
+ else
+ pkg_failed=untried
+fi
+
+
+
+if test $pkg_failed = yes; then
+ { $as_echo "$as_me:${as_lineno-$LINENO}: result: no" >&5
+$as_echo "no" >&6; }
+
+if $PKG_CONFIG --atleast-pkgconfig-version 0.20; then
+ _pkg_short_errors_supported=yes
+else
+ _pkg_short_errors_supported=no
+fi
+ if test $_pkg_short_errors_supported = yes; then
+ LIBURING_PKG_ERRORS=`$PKG_CONFIG --short-errors --print-errors --cflags --libs "liburing" 2>&1`
+ else
+ LIBURING_PKG_ERRORS=`$PKG_CONFIG --print-errors --cflags --libs "liburing" 2>&1`
+ fi
+ # Put the nasty error message in config.log where it belongs
+ echo "$LIBURING_PKG_ERRORS" >&5
+
+ as_fn_error $? "Package requirements (liburing) were not met:
+
+$LIBURING_PKG_ERRORS
+
+Consider adjusting the PKG_CONFIG_PATH environment variable if you
+installed software in a non-standard prefix.
+
+Alternatively, you may set the environment variables LIBURING_CFLAGS
+and LIBURING_LIBS to avoid the need to call pkg-config.
+See the pkg-config man page for more details." "$LINENO" 5
+elif test $pkg_failed = untried; then
+ { $as_echo "$as_me:${as_lineno-$LINENO}: result: no" >&5
+$as_echo "no" >&6; }
+ { { $as_echo "$as_me:${as_lineno-$LINENO}: error: in \`$ac_pwd':" >&5
+$as_echo "$as_me: error: in \`$ac_pwd':" >&2;}
+as_fn_error $? "The pkg-config script could not be found or is too old. Make sure it
+is in your PATH or set the PKG_CONFIG environment variable to the full
+path to pkg-config.
+
+Alternatively, you may set the environment variables LIBURING_CFLAGS
+and LIBURING_LIBS to avoid the need to call pkg-config.
+See the pkg-config man page for more details.
+
+To get pkg-config, see <http://pkg-config.freedesktop.org/>.
+See \`config.log' for more details" "$LINENO" 5; }
+else
+ LIBURING_CFLAGS=$pkg_cv_LIBURING_CFLAGS
+ LIBURING_LIBS=$pkg_cv_LIBURING_LIBS
+ { $as_echo "$as_me:${as_lineno-$LINENO}: result: yes" >&5
+$as_echo "yes" >&6; }
+
+fi
+fi
##
## Header files
diff --git a/.cirrus.tasks.yml b/.cirrus.tasks.yml
index 5849cbb839a..10d277f5659 100644
--- a/.cirrus.tasks.yml
+++ b/.cirrus.tasks.yml
@@ -471,6 +471,7 @@ task:
--enable-cassert --enable-injection-points --enable-debug \
--enable-tap-tests --enable-nls \
--with-segsize-blocks=6 \
+ --with-liburing \
\
${LINUX_CONFIGURE_FEATURES} \
\
diff --git a/src/Makefile.global.in b/src/Makefile.global.in
index 3b620bac5ac..d45f5fe4ed7 100644
--- a/src/Makefile.global.in
+++ b/src/Makefile.global.in
@@ -191,6 +191,7 @@ with_gssapi = @with_gssapi@
with_krb_srvnam = @with_krb_srvnam@
with_ldap = @with_ldap@
with_libcurl = @with_libcurl@
+with_liburing = @with_liburing@
with_libxml = @with_libxml@
with_libxslt = @with_libxslt@
with_llvm = @with_llvm@
@@ -217,6 +218,9 @@ krb_srvtab = @krb_srvtab@
ICU_CFLAGS = @ICU_CFLAGS@
ICU_LIBS = @ICU_LIBS@
+LIBURING_CFLAGS = @LIBURING_CFLAGS@
+LIBURING_LIBS = @LIBURING_LIBS@
+
TCLSH = @TCLSH@
TCL_LIBS = @TCL_LIBS@
TCL_LIB_SPEC = @TCL_LIB_SPEC@
--
2.48.1.76.g4e746b1a31.dirty
[text/x-diff] v2.9-0006-aio-Add-io_method-io_uring.patch (20.2K, ../../oav52ytakcvfnk52w5o5wvxdjavxth2tgptxt25gnfmh2pdcpk@omp3xukdu5if/7-v2.9-0006-aio-Add-io_method-io_uring.patch)
download | inline diff:
From e61ff69e39bbd81dc184f5eade772203e10f1b29 Mon Sep 17 00:00:00 2001
From: Andres Freund <andres@anarazel.de>
Date: Sat, 15 Mar 2025 12:29:57 -0400
Subject: [PATCH v2.9 06/30] aio: Add io_method=io_uring
io_uring, can be considerably faster than io_method=worker. It is, however,
linux specific and requires an additional compile-time dependency (liburing).
Reviewed-by: Jakub Wartak <jakub.wartak@enterprisedb.com>
Discussion: https://postgr.es/m/uvrtrknj4kdytuboidbhwclo4gxhswwcpgadptsjvjqcluzmah%40brqs62irg4dt
Discussion: https://postgr.es/m/20210223100344.llw5an2aklengrmn@alap3.anarazel.de
Discussion: https://postgr.es/m/stj36ea6yyhoxtqkhpieia2z4krnam7qyetc57rfezgk4zgapf@gcnactj4z56m
---
src/include/storage/aio.h | 3 +
src/include/storage/aio_internal.h | 3 +
src/include/storage/lwlock.h | 1 +
src/backend/storage/aio/Makefile | 1 +
src/backend/storage/aio/aio.c | 6 +
src/backend/storage/aio/meson.build | 1 +
src/backend/storage/aio/method_io_uring.c | 440 ++++++++++++++++++
src/backend/storage/lmgr/lwlock.c | 1 +
.../utils/activity/wait_event_names.txt | 2 +
src/backend/utils/misc/postgresql.conf.sample | 3 +-
doc/src/sgml/config.sgml | 6 +
.cirrus.tasks.yml | 3 +
src/tools/pgindent/typedefs.list | 1 +
13 files changed, 470 insertions(+), 1 deletion(-)
create mode 100644 src/backend/storage/aio/method_io_uring.c
diff --git a/src/include/storage/aio.h b/src/include/storage/aio.h
index 7b6b7d20a85..53b1080b17f 100644
--- a/src/include/storage/aio.h
+++ b/src/include/storage/aio.h
@@ -28,6 +28,9 @@ typedef enum IoMethod
{
IOMETHOD_SYNC = 0,
IOMETHOD_WORKER,
+#ifdef USE_LIBURING
+ IOMETHOD_IO_URING,
+#endif
} IoMethod;
/* We'll default to worker based execution. */
diff --git a/src/include/storage/aio_internal.h b/src/include/storage/aio_internal.h
index 51f63586467..98ae66921f5 100644
--- a/src/include/storage/aio_internal.h
+++ b/src/include/storage/aio_internal.h
@@ -386,6 +386,9 @@ extern PgAioHandle *pgaio_inj_io_get(void);
/* Declarations for the tables of function pointers exposed by each IO method. */
extern PGDLLIMPORT const IoMethodOps pgaio_sync_ops;
extern PGDLLIMPORT const IoMethodOps pgaio_worker_ops;
+#ifdef USE_LIBURING
+extern PGDLLIMPORT const IoMethodOps pgaio_uring_ops;
+#endif
extern PGDLLIMPORT const IoMethodOps *pgaio_method_ops;
extern PGDLLIMPORT PgAioCtl *pgaio_ctl;
diff --git a/src/include/storage/lwlock.h b/src/include/storage/lwlock.h
index ffa03189e2d..4df1d25c045 100644
--- a/src/include/storage/lwlock.h
+++ b/src/include/storage/lwlock.h
@@ -218,6 +218,7 @@ typedef enum BuiltinTrancheIds
LWTRANCHE_SUBTRANS_SLRU,
LWTRANCHE_XACT_SLRU,
LWTRANCHE_PARALLEL_VACUUM_DSA,
+ LWTRANCHE_AIO_URING_COMPLETION,
LWTRANCHE_FIRST_USER_DEFINED,
} BuiltinTrancheIds;
diff --git a/src/backend/storage/aio/Makefile b/src/backend/storage/aio/Makefile
index f51c34a37f8..c06c50771e0 100644
--- a/src/backend/storage/aio/Makefile
+++ b/src/backend/storage/aio/Makefile
@@ -14,6 +14,7 @@ OBJS = \
aio_init.o \
aio_io.o \
aio_target.o \
+ method_io_uring.o \
method_sync.o \
method_worker.o \
read_stream.o
diff --git a/src/backend/storage/aio/aio.c b/src/backend/storage/aio/aio.c
index 3ed4b1dfdac..2e210335872 100644
--- a/src/backend/storage/aio/aio.c
+++ b/src/backend/storage/aio/aio.c
@@ -65,6 +65,9 @@ static void pgaio_io_wait(PgAioHandle *ioh, uint64 ref_generation);
const struct config_enum_entry io_method_options[] = {
{"sync", IOMETHOD_SYNC, false},
{"worker", IOMETHOD_WORKER, false},
+#ifdef USE_LIBURING
+ {"io_uring", IOMETHOD_IO_URING, false},
+#endif
{NULL, 0, false}
};
@@ -82,6 +85,9 @@ PgAioBackend *pgaio_my_backend;
static const IoMethodOps *const pgaio_method_ops_table[] = {
[IOMETHOD_SYNC] = &pgaio_sync_ops,
[IOMETHOD_WORKER] = &pgaio_worker_ops,
+#ifdef USE_LIBURING
+ [IOMETHOD_IO_URING] = &pgaio_uring_ops,
+#endif
};
/* callbacks for the configured io_method, set by assign_io_method */
diff --git a/src/backend/storage/aio/meson.build b/src/backend/storage/aio/meson.build
index 74f94c6e40b..2f0f03d8071 100644
--- a/src/backend/storage/aio/meson.build
+++ b/src/backend/storage/aio/meson.build
@@ -6,6 +6,7 @@ backend_sources += files(
'aio_init.c',
'aio_io.c',
'aio_target.c',
+ 'method_io_uring.c',
'method_sync.c',
'method_worker.c',
'read_stream.c',
diff --git a/src/backend/storage/aio/method_io_uring.c b/src/backend/storage/aio/method_io_uring.c
new file mode 100644
index 00000000000..588177eaa4f
--- /dev/null
+++ b/src/backend/storage/aio/method_io_uring.c
@@ -0,0 +1,440 @@
+/*-------------------------------------------------------------------------
+ *
+ * method_io_uring.c
+ * AIO - perform AIO using Linux' io_uring
+ *
+ * For now we create one io_uring instance for each backend. These io_uring
+ * instances have to be created in postmaster, during startup, to allow other
+ * backends to process IO completions, if the issuing backend is currently
+ * busy doing other things. Other backends may not use another backend's
+ * io_uring instance to submit IO, that'd require additional locking that
+ * would likely be harmful for performance.
+ *
+ * We likely will want to introduce a backend-local io_uring instance in the
+ * future, e.g. for FE/BE network IO.
+ *
+ * Portions Copyright (c) 1996-2025, PostgreSQL Global Development Group
+ * Portions Copyright (c) 1994, Regents of the University of California
+ *
+ * IDENTIFICATION
+ * src/backend/storage/aio/method_io_uring.c
+ *
+ *-------------------------------------------------------------------------
+ */
+
+#include "postgres.h"
+
+#ifdef USE_LIBURING
+
+#include <liburing.h>
+
+#include "pgstat.h"
+#include "port/pg_iovec.h"
+#include "storage/aio.h"
+#include "storage/aio_internal.h"
+#include "storage/fd.h"
+#include "storage/proc.h"
+#include "storage/shmem.h"
+
+
+/* number of completions processed at once */
+#define PGAIO_MAX_LOCAL_COMPLETED_IO 32
+
+
+/* Entry points for IoMethodOps. */
+static size_t pgaio_uring_shmem_size(void);
+static void pgaio_uring_shmem_init(bool first_time);
+static void pgaio_uring_init_backend(void);
+static int pgaio_uring_submit(uint16 num_staged_ios, PgAioHandle **staged_ios);
+static void pgaio_uring_wait_one(PgAioHandle *ioh, uint64 ref_generation);
+
+/* helper functions */
+static void pgaio_uring_sq_from_io(PgAioHandle *ioh, struct io_uring_sqe *sqe);
+
+
+const IoMethodOps pgaio_uring_ops = {
+ .shmem_size = pgaio_uring_shmem_size,
+ .shmem_init = pgaio_uring_shmem_init,
+ .init_backend = pgaio_uring_init_backend,
+
+ .submit = pgaio_uring_submit,
+ .wait_one = pgaio_uring_wait_one,
+};
+
+/*
+ * Per-backend state when using io_method=io_uring
+ *
+ * Align the whole struct to a cacheline boundary, to prevent false sharing
+ * between completion_lock and prior backend's io_uring_ring.
+ */
+typedef struct pg_attribute_aligned (PG_CACHE_LINE_SIZE)
+PgAioUringContext
+{
+ /*
+ * Multiple backends can process completions for this backend's io_uring
+ * instance (e.g. when the backend issuing IO is busy doing something
+ * else). To make that safe we have to ensure that only a single backend
+ * gets io completions from the io_uring instance at a time.
+ */
+ LWLock completion_lock;
+
+ struct io_uring io_uring_ring;
+} PgAioUringContext;
+
+/* PgAioUringContexts for all backends */
+static PgAioUringContext *pgaio_uring_contexts;
+
+/* the current backend's context */
+static PgAioUringContext *pgaio_my_uring_context;
+
+
+static uint32
+pgaio_uring_procs(void)
+{
+ /*
+ * We can subtract MAX_IO_WORKERS here as io workers are never used at the
+ * same time as io_method=io_uring.
+ */
+ return MaxBackends + NUM_AUXILIARY_PROCS - MAX_IO_WORKERS;
+}
+
+static Size
+pgaio_uring_context_shmem_size(void)
+{
+ return mul_size(pgaio_uring_procs(), sizeof(PgAioUringContext));
+}
+
+static size_t
+pgaio_uring_shmem_size(void)
+{
+ return pgaio_uring_context_shmem_size();
+}
+
+static void
+pgaio_uring_shmem_init(bool first_time)
+{
+ uint32 TotalProcs = MaxBackends + NUM_AUXILIARY_PROCS - MAX_IO_WORKERS;
+ bool found;
+
+ pgaio_uring_contexts = (PgAioUringContext *)
+ ShmemInitStruct("AioUring", pgaio_uring_shmem_size(), &found);
+
+ if (found)
+ return;
+
+ for (int contextno = 0; contextno < TotalProcs; contextno++)
+ {
+ PgAioUringContext *context = &pgaio_uring_contexts[contextno];
+ int ret;
+
+ /*
+ * FIXME: Right now a high TotalProcs will cause problems in two ways:
+ *
+ * - RLIMIT_NOFILE needs to be big enough to allow MaxBackends fds
+ * created in postmaster, with spare space for max_files_per_process
+ * additional FDs
+ *
+ * - set_max_safe_fds() subtracts the number of already used FDs from
+ * max_files_per_process, ending up with a low limit or even erroring
+ * out.
+ *
+ * XXX: Newer versions of io_uring support sharing the workers that
+ * execute some asynchronous IOs between io_uring instances. It might
+ * be worth using that - also need to evaluate if that causes
+ * noticeable additional contention?
+ */
+ ret = io_uring_queue_init(io_max_concurrency, &context->io_uring_ring, 0);
+ if (ret < 0)
+ elog(ERROR, "io_uring_queue_init failed: %s", strerror(-ret));
+
+ LWLockInitialize(&context->completion_lock, LWTRANCHE_AIO_URING_COMPLETION);
+ }
+}
+
+static void
+pgaio_uring_init_backend(void)
+{
+ Assert(MyProcNumber < pgaio_uring_procs());
+
+ pgaio_my_uring_context = &pgaio_uring_contexts[MyProcNumber];
+}
+
+static int
+pgaio_uring_submit(uint16 num_staged_ios, PgAioHandle **staged_ios)
+{
+ struct io_uring *uring_instance = &pgaio_my_uring_context->io_uring_ring;
+ int in_flight_before = dclist_count(&pgaio_my_backend->in_flight_ios);
+
+ Assert(num_staged_ios <= PGAIO_SUBMIT_BATCH_SIZE);
+
+ for (int i = 0; i < num_staged_ios; i++)
+ {
+ PgAioHandle *ioh = staged_ios[i];
+ struct io_uring_sqe *sqe;
+
+ sqe = io_uring_get_sqe(uring_instance);
+
+ if (!sqe)
+ elog(ERROR, "io_uring submission queue is unexpectedly full");
+
+ pgaio_io_prepare_submit(ioh);
+ pgaio_uring_sq_from_io(ioh, sqe);
+
+ /*
+ * io_uring executes IO in process context if possible. That's
+ * generally good, as it reduces context switching. When performing a
+ * lot of buffered IO that means that copying between page cache and
+ * userspace memory happens in the foreground, as it can't be
+ * offloaded to DMA hardware as is possible when using direct IO. When
+ * executing a lot of buffered IO this causes io_uring to be slower
+ * than worker mode, as worker mode parallelizes the copying. io_uring
+ * can be told to offload work to worker threads instead.
+ *
+ * If an IO is buffered IO and we already have IOs in flight or
+ * multiple IOs are being submitted, we thus tell io_uring to execute
+ * the IO in the background. We don't do so for the first few IOs
+ * being submitted as executing in this process' context has lower
+ * latency.
+ */
+ if (in_flight_before > 4 && (ioh->flags & PGAIO_HF_BUFFERED))
+ io_uring_sqe_set_flags(sqe, IOSQE_ASYNC);
+
+ in_flight_before++;
+ }
+
+ while (true)
+ {
+ int ret;
+
+ pgstat_report_wait_start(WAIT_EVENT_AIO_IO_URING_SUBMIT);
+ ret = io_uring_submit(uring_instance);
+ pgstat_report_wait_end();
+
+ if (ret == -EINTR)
+ {
+ pgaio_debug(DEBUG3,
+ "aio method uring: submit EINTR, nios: %d",
+ num_staged_ios);
+ }
+ else if (ret < 0)
+ {
+ /*
+ * The io_uring_enter() manpage suggests that the appropriate
+ * reaction to EAGAIN is:
+ *
+ * "The application should wait for some completions and try
+ * again"
+ *
+ * However, it seems unlikely that that would help in our case, as
+ * we apply a low limit to the number of outstanding IOs and thus
+ * also outstanding completions, making it unlikely that we'd get
+ * EAGAIN while the OS is in good working order.
+ *
+ * Additionally, it would be problematic to just wait here, our
+ * caller might hold critical locks. It'd possibly lead to
+ * delaying the crash-restart that seems likely to occur when the
+ * kernel is under such heavy memory pressure.
+ *
+ * Update errno to to allow %m to work.
+ */
+ errno = -ret;
+ elog(PANIC, "io_uring submit failed: %m");
+ }
+ else if (ret != num_staged_ios)
+ {
+ /* likely unreachable, but if it is, we would need to re-submit */
+ elog(PANIC, "io_uring submit submitted only %d of %d",
+ ret, num_staged_ios);
+ }
+ else
+ {
+ pgaio_debug(DEBUG4,
+ "aio method uring: submitted %d IOs",
+ num_staged_ios);
+ break;
+ }
+ }
+
+ return num_staged_ios;
+}
+
+static void
+pgaio_uring_drain_locked(PgAioUringContext *context)
+{
+ int ready;
+ int orig_ready;
+
+ Assert(LWLockHeldByMeInMode(&context->completion_lock, LW_EXCLUSIVE));
+
+ /*
+ * Don't drain more events than available right now. Otherwise it's
+ * plausible that one backend could get stuck, for a while, receiving CQEs
+ * without actually processing them.
+ */
+ orig_ready = ready = io_uring_cq_ready(&context->io_uring_ring);
+
+ while (ready > 0)
+ {
+ struct io_uring_cqe *cqes[PGAIO_MAX_LOCAL_COMPLETED_IO];
+ uint32 ncqes;
+
+ START_CRIT_SECTION();
+ ncqes =
+ io_uring_peek_batch_cqe(&context->io_uring_ring,
+ cqes,
+ Min(PGAIO_MAX_LOCAL_COMPLETED_IO, ready));
+ Assert(ncqes <= ready);
+
+ ready -= ncqes;
+
+ for (int i = 0; i < ncqes; i++)
+ {
+ struct io_uring_cqe *cqe = cqes[i];
+ PgAioHandle *ioh;
+
+ ioh = io_uring_cqe_get_data(cqe);
+ io_uring_cqe_seen(&context->io_uring_ring, cqe);
+
+ pgaio_io_process_completion(ioh, cqe->res);
+ }
+
+ END_CRIT_SECTION();
+
+ pgaio_debug(DEBUG3,
+ "drained %d/%d, now expecting %d",
+ ncqes, orig_ready, io_uring_cq_ready(&context->io_uring_ring));
+ }
+}
+
+static void
+pgaio_uring_wait_one(PgAioHandle *ioh, uint64 ref_generation)
+{
+ PgAioHandleState state;
+ ProcNumber owner_procno = ioh->owner_procno;
+ PgAioUringContext *owner_context = &pgaio_uring_contexts[owner_procno];
+ bool expect_cqe;
+ int waited = 0;
+
+ /*
+ * XXX: It would be nice to have a smarter locking scheme, nearly all the
+ * time the backend owning the ring will consume the completions, making
+ * the locking unnecessarily expensive.
+ */
+ LWLockAcquire(&owner_context->completion_lock, LW_EXCLUSIVE);
+
+ while (true)
+ {
+ pgaio_debug_io(DEBUG3, ioh,
+ "wait_one io_gen: %llu, ref_gen: %llu, cycle %d",
+ (long long unsigned) ref_generation,
+ (long long unsigned) ioh->generation,
+ waited);
+
+ if (pgaio_io_was_recycled(ioh, ref_generation, &state) ||
+ state != PGAIO_HS_SUBMITTED)
+ {
+ /* the IO was completed by another backend */
+ break;
+ }
+ else if (io_uring_cq_ready(&owner_context->io_uring_ring))
+ {
+ /* no need to wait in the kernel, io_uring has a completion */
+ expect_cqe = true;
+ }
+ else
+ {
+ int ret;
+ struct io_uring_cqe *cqes;
+
+ /* need to wait in the kernel */
+ pgstat_report_wait_start(WAIT_EVENT_AIO_IO_URING_COMPLETION);
+ ret = io_uring_wait_cqes(&owner_context->io_uring_ring, &cqes, 1, NULL, NULL);
+ pgstat_report_wait_end();
+
+ if (ret == -EINTR)
+ {
+ continue;
+ }
+ else if (ret != 0)
+ {
+ /* see comment after io_uring_submit() */
+ errno = -ret;
+ elog(PANIC, "io_uring wait failed: %m");
+ }
+ else
+ {
+ Assert(cqes != NULL);
+ expect_cqe = true;
+ waited++;
+ }
+ }
+
+ if (expect_cqe)
+ {
+ pgaio_uring_drain_locked(owner_context);
+ }
+ }
+
+ LWLockRelease(&owner_context->completion_lock);
+
+ pgaio_debug(DEBUG3,
+ "wait_one with %d sleeps",
+ waited);
+}
+
+static void
+pgaio_uring_sq_from_io(PgAioHandle *ioh, struct io_uring_sqe *sqe)
+{
+ struct iovec *iov;
+
+ switch (ioh->op)
+ {
+ case PGAIO_OP_READV:
+ iov = &pgaio_ctl->iovecs[ioh->iovec_off];
+ if (ioh->op_data.read.iov_length == 1)
+ {
+ io_uring_prep_read(sqe,
+ ioh->op_data.read.fd,
+ iov->iov_base,
+ iov->iov_len,
+ ioh->op_data.read.offset);
+ }
+ else
+ {
+ io_uring_prep_readv(sqe,
+ ioh->op_data.read.fd,
+ iov,
+ ioh->op_data.read.iov_length,
+ ioh->op_data.read.offset);
+
+ }
+ break;
+
+ case PGAIO_OP_WRITEV:
+ iov = &pgaio_ctl->iovecs[ioh->iovec_off];
+ if (ioh->op_data.write.iov_length == 1)
+ {
+ io_uring_prep_write(sqe,
+ ioh->op_data.write.fd,
+ iov->iov_base,
+ iov->iov_len,
+ ioh->op_data.write.offset);
+ }
+ else
+ {
+ io_uring_prep_writev(sqe,
+ ioh->op_data.write.fd,
+ iov,
+ ioh->op_data.write.iov_length,
+ ioh->op_data.write.offset);
+ }
+ break;
+
+ case PGAIO_OP_INVALID:
+ elog(ERROR, "trying to prepare invalid IO operation for execution");
+ }
+
+ io_uring_sqe_set_data(sqe, ioh);
+}
+
+#endif /* USE_LIBURING */
diff --git a/src/backend/storage/lmgr/lwlock.c b/src/backend/storage/lmgr/lwlock.c
index 5702c35bb91..3df29658f18 100644
--- a/src/backend/storage/lmgr/lwlock.c
+++ b/src/backend/storage/lmgr/lwlock.c
@@ -177,6 +177,7 @@ static const char *const BuiltinTrancheNames[] = {
[LWTRANCHE_SUBTRANS_SLRU] = "SubtransSLRU",
[LWTRANCHE_XACT_SLRU] = "XactSLRU",
[LWTRANCHE_PARALLEL_VACUUM_DSA] = "ParallelVacuumDSA",
+ [LWTRANCHE_AIO_URING_COMPLETION] = "AioUringCompletion",
};
StaticAssertDecl(lengthof(BuiltinTrancheNames) ==
diff --git a/src/backend/utils/activity/wait_event_names.txt b/src/backend/utils/activity/wait_event_names.txt
index 9fa12a555e8..44eef41f841 100644
--- a/src/backend/utils/activity/wait_event_names.txt
+++ b/src/backend/utils/activity/wait_event_names.txt
@@ -192,6 +192,8 @@ ABI_compatibility:
Section: ClassName - WaitEventIO
+AIO_IO_URING_SUBMIT "Waiting for IO submission via io_uring."
+AIO_IO_URING_COMPLETION "Waiting for IO completion via io_uring."
AIO_IO_COMPLETION "Waiting for IO completion."
BASEBACKUP_READ "Waiting for base backup to read from a file."
BASEBACKUP_SYNC "Waiting for data written by a base backup to reach durable storage."
diff --git a/src/backend/utils/misc/postgresql.conf.sample b/src/backend/utils/misc/postgresql.conf.sample
index 75a844a9474..7e8c3dcb175 100644
--- a/src/backend/utils/misc/postgresql.conf.sample
+++ b/src/backend/utils/misc/postgresql.conf.sample
@@ -202,7 +202,8 @@
#maintenance_io_concurrency = 10 # 1-1000; 0 disables prefetching
#io_combine_limit = 128kB # usually 1-32 blocks (depends on OS)
-#io_method = worker # worker, sync (change requires restart)
+#io_method = worker # worker, io_uring, sync
+ # (change requires restart)
#io_max_concurrency = -1 # Max number of IOs that one process
# can execute simultaneously
# -1 sets based on shared_buffers
diff --git a/doc/src/sgml/config.sgml b/doc/src/sgml/config.sgml
index 108caea1fed..8d53478f53a 100644
--- a/doc/src/sgml/config.sgml
+++ b/doc/src/sgml/config.sgml
@@ -2681,6 +2681,12 @@ include_dir 'conf.d'
<literal>worker</literal> (execute asynchronous I/O using worker processes)
</para>
</listitem>
+ <listitem>
+ <para>
+ <literal>io_uring</literal> (execute asynchronous I/O using
+ io_uring, if available)
+ </para>
+ </listitem>
<listitem>
<para>
<literal>sync</literal> (execute asynchronous-eligible I/O synchronously)
diff --git a/.cirrus.tasks.yml b/.cirrus.tasks.yml
index 10d277f5659..86a1fa9bbdb 100644
--- a/.cirrus.tasks.yml
+++ b/.cirrus.tasks.yml
@@ -493,11 +493,14 @@ task:
# - Uses undefined behaviour and alignment sanitizers, sanitizer failures
# are typically printed in the server log
# - Test both 64bit and 32 bit builds
+ # - uses io_method=io_uring
- name: Linux - Debian Bookworm - Meson
env:
CCACHE_MAXSIZE: "400M" # tests two different builds
SANITIZER_FLAGS: -fsanitize=alignment,undefined
+ PG_TEST_INITDB_EXTRA_OPTS: >-
+ -c io_method=io_uring
configure_script: |
su postgres <<-EOF
diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list
index 4d8b5c7f1d6..1748befca16 100644
--- a/src/tools/pgindent/typedefs.list
+++ b/src/tools/pgindent/typedefs.list
@@ -2150,6 +2150,7 @@ PgAioReturn
PgAioTargetData
PgAioTargetID
PgAioTargetInfo
+PgAioUringContext
PgAioWaitRef
PgArchData
PgBackendGSSStatus
--
2.48.1.76.g4e746b1a31.dirty
[text/x-diff] v2.9-0007-smgr-Hold-interrupts-in-most-smgr-functions.patch (9.4K, ../../oav52ytakcvfnk52w5o5wvxdjavxth2tgptxt25gnfmh2pdcpk@omp3xukdu5if/8-v2.9-0007-smgr-Hold-interrupts-in-most-smgr-functions.patch)
download | inline diff:
From da8a868fd73f96604426a232f8300f50bc26e425 Mon Sep 17 00:00:00 2001
From: Andres Freund <andres@anarazel.de>
Date: Sat, 15 Mar 2025 12:29:57 -0400
Subject: [PATCH v2.9 07/30] smgr: Hold interrupts in most smgr functions
We need to hold interrupts across most of the smgr.c/md.c functions, as
otherwise interrupt processing, e.g. due to a debug elog/ereport, can trigger
procsignal processing, which in turn can trigger smgrreleaseall(). As the
relevant code is not reentrant we quickly end up in a bad situation.
It seems better to put the HOLD_INTERRUPTS()/RESUME_INTERRUPTS() in smgr.c,
instead of trying to push them down to md.c where possible: For one, every
smgr implementation would be vulnerable, for another, a good bit of smgr.c
code itself is affected too.
Discussion: https://postgr.es/m/3vae7l5ozvqtxmd7rr7zaeq3qkuipz365u3rtim5t5wdkr6f4g@vkgf2fogjirl
---
src/backend/storage/smgr/smgr.c | 94 +++++++++++++++++++++++++++++++--
1 file changed, 91 insertions(+), 3 deletions(-)
diff --git a/src/backend/storage/smgr/smgr.c b/src/backend/storage/smgr/smgr.c
index ebe35c04de5..8787ce9b18f 100644
--- a/src/backend/storage/smgr/smgr.c
+++ b/src/backend/storage/smgr/smgr.c
@@ -40,6 +40,15 @@
* themselves, as there could pointers to them in active use. See
* smgrrelease() and smgrreleaseall().
*
+ * NB: We need to hold interrupts across most of the functions in this file,
+ * as otherwise interrupt processing, e.g. due to a debug elog/ereport, can
+ * trigger procsignal processing, which in turn can trigger
+ * smgrreleaseall(). None of the relevant code is reentrant. It seems better
+ * to put the HOLD_INTERRUPTS()/RESUME_INTERRUPTS() here, instead of trying to
+ * push them down to md.c where possible: For one, every smgr implementation
+ * would be vulnerable, for another, a good bit of smgr.c code itself is
+ * affected too.
+ *
* Portions Copyright (c) 1996-2025, PostgreSQL Global Development Group
* Portions Copyright (c) 1994, Regents of the University of California
*
@@ -53,6 +62,7 @@
#include "access/xlogutils.h"
#include "lib/ilist.h"
+#include "miscadmin.h"
#include "storage/bufmgr.h"
#include "storage/ipc.h"
#include "storage/md.h"
@@ -158,12 +168,16 @@ smgrinit(void)
{
int i;
+ HOLD_INTERRUPTS();
+
for (i = 0; i < NSmgr; i++)
{
if (smgrsw[i].smgr_init)
smgrsw[i].smgr_init();
}
+ RESUME_INTERRUPTS();
+
/* register the shutdown proc */
on_proc_exit(smgrshutdown, 0);
}
@@ -176,11 +190,13 @@ smgrshutdown(int code, Datum arg)
{
int i;
+ HOLD_INTERRUPTS();
for (i = 0; i < NSmgr; i++)
{
if (smgrsw[i].smgr_shutdown)
smgrsw[i].smgr_shutdown();
}
+ RESUME_INTERRUPTS();
}
/*
@@ -206,6 +222,8 @@ smgropen(RelFileLocator rlocator, ProcNumber backend)
Assert(RelFileNumberIsValid(rlocator.relNumber));
+ HOLD_INTERRUPTS();
+
if (SMgrRelationHash == NULL)
{
/* First time through: initialize the hash table */
@@ -242,6 +260,8 @@ smgropen(RelFileLocator rlocator, ProcNumber backend)
dlist_push_tail(&unpinned_relns, &reln->node);
}
+ RESUME_INTERRUPTS();
+
return reln;
}
@@ -283,6 +303,8 @@ smgrdestroy(SMgrRelation reln)
Assert(reln->pincount == 0);
+ HOLD_INTERRUPTS();
+
for (forknum = 0; forknum <= MAX_FORKNUM; forknum++)
smgrsw[reln->smgr_which].smgr_close(reln, forknum);
@@ -292,6 +314,8 @@ smgrdestroy(SMgrRelation reln)
&(reln->smgr_rlocator),
HASH_REMOVE, NULL) == NULL)
elog(ERROR, "SMgrRelation hashtable corrupted");
+
+ RESUME_INTERRUPTS();
}
/*
@@ -302,12 +326,16 @@ smgrdestroy(SMgrRelation reln)
void
smgrrelease(SMgrRelation reln)
{
+ HOLD_INTERRUPTS();
+
for (ForkNumber forknum = 0; forknum <= MAX_FORKNUM; forknum++)
{
smgrsw[reln->smgr_which].smgr_close(reln, forknum);
reln->smgr_cached_nblocks[forknum] = InvalidBlockNumber;
}
reln->smgr_targblock = InvalidBlockNumber;
+
+ RESUME_INTERRUPTS();
}
/*
@@ -336,6 +364,8 @@ smgrdestroyall(void)
{
dlist_mutable_iter iter;
+ HOLD_INTERRUPTS();
+
/*
* Zap all unpinned SMgrRelations. We rely on smgrdestroy() to remove
* each one from the list.
@@ -347,6 +377,8 @@ smgrdestroyall(void)
smgrdestroy(rel);
}
+
+ RESUME_INTERRUPTS();
}
/*
@@ -362,12 +394,16 @@ smgrreleaseall(void)
if (SMgrRelationHash == NULL)
return;
+ HOLD_INTERRUPTS();
+
hash_seq_init(&status, SMgrRelationHash);
while ((reln = (SMgrRelation) hash_seq_search(&status)) != NULL)
{
smgrrelease(reln);
}
+
+ RESUME_INTERRUPTS();
}
/*
@@ -400,7 +436,13 @@ smgrreleaserellocator(RelFileLocatorBackend rlocator)
bool
smgrexists(SMgrRelation reln, ForkNumber forknum)
{
- return smgrsw[reln->smgr_which].smgr_exists(reln, forknum);
+ bool ret;
+
+ HOLD_INTERRUPTS();
+ ret = smgrsw[reln->smgr_which].smgr_exists(reln, forknum);
+ RESUME_INTERRUPTS();
+
+ return ret;
}
/*
@@ -413,7 +455,9 @@ smgrexists(SMgrRelation reln, ForkNumber forknum)
void
smgrcreate(SMgrRelation reln, ForkNumber forknum, bool isRedo)
{
+ HOLD_INTERRUPTS();
smgrsw[reln->smgr_which].smgr_create(reln, forknum, isRedo);
+ RESUME_INTERRUPTS();
}
/*
@@ -434,6 +478,8 @@ smgrdosyncall(SMgrRelation *rels, int nrels)
if (nrels == 0)
return;
+ HOLD_INTERRUPTS();
+
FlushRelationsAllBuffers(rels, nrels);
/*
@@ -449,6 +495,8 @@ smgrdosyncall(SMgrRelation *rels, int nrels)
smgrsw[which].smgr_immedsync(rels[i], forknum);
}
}
+
+ RESUME_INTERRUPTS();
}
/*
@@ -471,6 +519,8 @@ smgrdounlinkall(SMgrRelation *rels, int nrels, bool isRedo)
if (nrels == 0)
return;
+ HOLD_INTERRUPTS();
+
/*
* Get rid of any remaining buffers for the relations. bufmgr will just
* drop them without bothering to write the contents.
@@ -522,6 +572,8 @@ smgrdounlinkall(SMgrRelation *rels, int nrels, bool isRedo)
}
pfree(rlocators);
+
+ RESUME_INTERRUPTS();
}
@@ -538,6 +590,8 @@ void
smgrextend(SMgrRelation reln, ForkNumber forknum, BlockNumber blocknum,
const void *buffer, bool skipFsync)
{
+ HOLD_INTERRUPTS();
+
smgrsw[reln->smgr_which].smgr_extend(reln, forknum, blocknum,
buffer, skipFsync);
@@ -550,6 +604,8 @@ smgrextend(SMgrRelation reln, ForkNumber forknum, BlockNumber blocknum,
reln->smgr_cached_nblocks[forknum] = blocknum + 1;
else
reln->smgr_cached_nblocks[forknum] = InvalidBlockNumber;
+
+ RESUME_INTERRUPTS();
}
/*
@@ -563,6 +619,8 @@ void
smgrzeroextend(SMgrRelation reln, ForkNumber forknum, BlockNumber blocknum,
int nblocks, bool skipFsync)
{
+ HOLD_INTERRUPTS();
+
smgrsw[reln->smgr_which].smgr_zeroextend(reln, forknum, blocknum,
nblocks, skipFsync);
@@ -575,6 +633,8 @@ smgrzeroextend(SMgrRelation reln, ForkNumber forknum, BlockNumber blocknum,
reln->smgr_cached_nblocks[forknum] = blocknum + nblocks;
else
reln->smgr_cached_nblocks[forknum] = InvalidBlockNumber;
+
+ RESUME_INTERRUPTS();
}
/*
@@ -588,7 +648,13 @@ bool
smgrprefetch(SMgrRelation reln, ForkNumber forknum, BlockNumber blocknum,
int nblocks)
{
- return smgrsw[reln->smgr_which].smgr_prefetch(reln, forknum, blocknum, nblocks);
+ bool ret;
+
+ HOLD_INTERRUPTS();
+ ret = smgrsw[reln->smgr_which].smgr_prefetch(reln, forknum, blocknum, nblocks);
+ RESUME_INTERRUPTS();
+
+ return ret;
}
/*
@@ -601,7 +667,13 @@ uint32
smgrmaxcombine(SMgrRelation reln, ForkNumber forknum,
BlockNumber blocknum)
{
- return smgrsw[reln->smgr_which].smgr_maxcombine(reln, forknum, blocknum);
+ uint32 ret;
+
+ HOLD_INTERRUPTS();
+ ret = smgrsw[reln->smgr_which].smgr_maxcombine(reln, forknum, blocknum);
+ RESUME_INTERRUPTS();
+
+ return ret;
}
/*
@@ -619,8 +691,10 @@ void
smgrreadv(SMgrRelation reln, ForkNumber forknum, BlockNumber blocknum,
void **buffers, BlockNumber nblocks)
{
+ HOLD_INTERRUPTS();
smgrsw[reln->smgr_which].smgr_readv(reln, forknum, blocknum, buffers,
nblocks);
+ RESUME_INTERRUPTS();
}
/*
@@ -653,8 +727,10 @@ void
smgrwritev(SMgrRelation reln, ForkNumber forknum, BlockNumber blocknum,
const void **buffers, BlockNumber nblocks, bool skipFsync)
{
+ HOLD_INTERRUPTS();
smgrsw[reln->smgr_which].smgr_writev(reln, forknum, blocknum,
buffers, nblocks, skipFsync);
+ RESUME_INTERRUPTS();
}
/*
@@ -665,8 +741,10 @@ void
smgrwriteback(SMgrRelation reln, ForkNumber forknum, BlockNumber blocknum,
BlockNumber nblocks)
{
+ HOLD_INTERRUPTS();
smgrsw[reln->smgr_which].smgr_writeback(reln, forknum, blocknum,
nblocks);
+ RESUME_INTERRUPTS();
}
/*
@@ -683,10 +761,14 @@ smgrnblocks(SMgrRelation reln, ForkNumber forknum)
if (result != InvalidBlockNumber)
return result;
+ HOLD_INTERRUPTS();
+
result = smgrsw[reln->smgr_which].smgr_nblocks(reln, forknum);
reln->smgr_cached_nblocks[forknum] = result;
+ RESUME_INTERRUPTS();
+
return result;
}
@@ -731,6 +813,8 @@ smgrtruncate(SMgrRelation reln, ForkNumber *forknum, int nforks,
{
int i;
+ Assert(!INTERRUPTS_CAN_BE_PROCESSED());
+
/*
* Get rid of any buffers for the about-to-be-deleted blocks. bufmgr will
* just drop them without bothering to write the contents.
@@ -784,7 +868,9 @@ smgrtruncate(SMgrRelation reln, ForkNumber *forknum, int nforks,
void
smgrregistersync(SMgrRelation reln, ForkNumber forknum)
{
+ HOLD_INTERRUPTS();
smgrsw[reln->smgr_which].smgr_registersync(reln, forknum);
+ RESUME_INTERRUPTS();
}
/*
@@ -816,7 +902,9 @@ smgrregistersync(SMgrRelation reln, ForkNumber forknum)
void
smgrimmedsync(SMgrRelation reln, ForkNumber forknum)
{
+ HOLD_INTERRUPTS();
smgrsw[reln->smgr_which].smgr_immedsync(reln, forknum);
+ RESUME_INTERRUPTS();
}
/*
--
2.48.1.76.g4e746b1a31.dirty
[text/x-diff] v2.9-0008-aio-Implement-support-for-reads-in-smgr-md-fd.patch (22.0K, ../../oav52ytakcvfnk52w5o5wvxdjavxth2tgptxt25gnfmh2pdcpk@omp3xukdu5if/9-v2.9-0008-aio-Implement-support-for-reads-in-smgr-md-fd.patch)
download | inline diff:
From 478fe41b2ec999c394ff5848ff3ec5d725ad2598 Mon Sep 17 00:00:00 2001
From: Andres Freund <andres@anarazel.de>
Date: Sat, 15 Mar 2025 12:29:57 -0400
Subject: [PATCH v2.9 08/30] aio: Implement support for reads in smgr/md/fd
This implements the following:
1) An smgr AIO target, for AIO on smgr files. This should be usable not just
for md.c but also other SMGR implementation if we ever get them.
2) readv support in fd.c, which requires a small bit of infrastructure work in
fd.c
3) smgr.c and md.c support for readv
There still is nothing performing AIO, but as of this commit it would be
possible.
As part of this change FileGetRawDesc() actually ensures that the file is
opened - previously it was basically not usable. It's used to reopen a file in
IO workers.
Discussion: https://postgr.es/m/uvrtrknj4kdytuboidbhwclo4gxhswwcpgadptsjvjqcluzmah%40brqs62irg4dt
Discussion: https://postgr.es/m/20210223100344.llw5an2aklengrmn@alap3.anarazel.de
Discussion: https://postgr.es/m/stj36ea6yyhoxtqkhpieia2z4krnam7qyetc57rfezgk4zgapf@gcnactj4z56m
---
src/include/storage/aio.h | 6 +-
src/include/storage/aio_types.h | 10 +-
src/include/storage/fd.h | 3 +
src/include/storage/md.h | 7 +
src/include/storage/smgr.h | 14 ++
src/backend/storage/aio/aio_callback.c | 3 +
src/backend/storage/aio/aio_target.c | 2 +
src/backend/storage/file/fd.c | 40 ++++++
src/backend/storage/smgr/md.c | 176 +++++++++++++++++++++++++
src/backend/storage/smgr/smgr.c | 144 ++++++++++++++++++++
10 files changed, 401 insertions(+), 4 deletions(-)
diff --git a/src/include/storage/aio.h b/src/include/storage/aio.h
index 53b1080b17f..4fe7a6a8b00 100644
--- a/src/include/storage/aio.h
+++ b/src/include/storage/aio.h
@@ -112,9 +112,10 @@ typedef enum PgAioTargetID
{
/* intentionally the zero value, to help catch zeroed memory etc */
PGAIO_TID_INVALID = 0,
+ PGAIO_TID_SMGR,
} PgAioTargetID;
-#define PGAIO_TID_COUNT (PGAIO_TID_INVALID + 1)
+#define PGAIO_TID_COUNT (PGAIO_TID_SMGR + 1)
/*
@@ -186,6 +187,9 @@ struct PgAioTargetInfo
typedef enum PgAioHandleCallbackID
{
PGAIO_HCB_INVALID,
+
+ PGAIO_HCB_MD_READV,
+ PGAIO_HCB_MD_WRITEV,
} PgAioHandleCallbackID;
diff --git a/src/include/storage/aio_types.h b/src/include/storage/aio_types.h
index a5cc658efbd..b0b9a2a5c97 100644
--- a/src/include/storage/aio_types.h
+++ b/src/include/storage/aio_types.h
@@ -60,11 +60,15 @@ typedef struct PgAioWaitRef
*/
typedef union PgAioTargetData
{
- /* just as an example placeholder for later */
struct
{
- uint32 queue_id;
- } wal;
+ RelFileLocator rlocator; /* physical relation identifier */
+ BlockNumber blockNum; /* blknum relative to begin of reln */
+ BlockNumber nblocks;
+ ForkNumber forkNum:8; /* don't waste 4 byte for four values */
+ bool is_temp:1; /* proc can be inferred by owning AIO */
+ bool skip_fsync:1;
+ } smgr;
} PgAioTargetData;
diff --git a/src/include/storage/fd.h b/src/include/storage/fd.h
index e3067ab6597..b77d8e5e30e 100644
--- a/src/include/storage/fd.h
+++ b/src/include/storage/fd.h
@@ -101,6 +101,8 @@ extern PGDLLIMPORT int max_safe_fds;
* prototypes for functions in fd.c
*/
+struct PgAioHandle;
+
/* Operations on virtual Files --- equivalent to Unix kernel file ops */
extern File PathNameOpenFile(const char *fileName, int fileFlags);
extern File PathNameOpenFilePerm(const char *fileName, int fileFlags, mode_t fileMode);
@@ -109,6 +111,7 @@ extern void FileClose(File file);
extern int FilePrefetch(File file, off_t offset, off_t amount, uint32 wait_event_info);
extern ssize_t FileReadV(File file, const struct iovec *iov, int iovcnt, off_t offset, uint32 wait_event_info);
extern ssize_t FileWriteV(File file, const struct iovec *iov, int iovcnt, off_t offset, uint32 wait_event_info);
+extern int FileStartReadV(struct PgAioHandle *ioh, File file, int iovcnt, off_t offset, uint32 wait_event_info);
extern int FileSync(File file, uint32 wait_event_info);
extern int FileZero(File file, off_t offset, off_t amount, uint32 wait_event_info);
extern int FileFallocate(File file, off_t offset, off_t amount, uint32 wait_event_info);
diff --git a/src/include/storage/md.h b/src/include/storage/md.h
index 05bf537066e..9d7131eff43 100644
--- a/src/include/storage/md.h
+++ b/src/include/storage/md.h
@@ -14,11 +14,14 @@
#ifndef MD_H
#define MD_H
+#include "storage/aio_types.h"
#include "storage/block.h"
#include "storage/relfilelocator.h"
#include "storage/smgr.h"
#include "storage/sync.h"
+extern const PgAioHandleCallbacks aio_md_readv_cb;
+
/* md storage manager functionality */
extern void mdinit(void);
extern void mdopen(SMgrRelation reln);
@@ -36,6 +39,9 @@ extern uint32 mdmaxcombine(SMgrRelation reln, ForkNumber forknum,
BlockNumber blocknum);
extern void mdreadv(SMgrRelation reln, ForkNumber forknum, BlockNumber blocknum,
void **buffers, BlockNumber nblocks);
+extern void mdstartreadv(PgAioHandle *ioh,
+ SMgrRelation reln, ForkNumber forknum, BlockNumber blocknum,
+ void **buffers, BlockNumber nblocks);
extern void mdwritev(SMgrRelation reln, ForkNumber forknum,
BlockNumber blocknum,
const void **buffers, BlockNumber nblocks, bool skipFsync);
@@ -46,6 +52,7 @@ extern void mdtruncate(SMgrRelation reln, ForkNumber forknum,
BlockNumber old_blocks, BlockNumber nblocks);
extern void mdimmedsync(SMgrRelation reln, ForkNumber forknum);
extern void mdregistersync(SMgrRelation reln, ForkNumber forknum);
+extern int mdfd(SMgrRelation reln, ForkNumber forknum, BlockNumber blocknum, uint32 *off);
extern void ForgetDatabaseSyncRequests(Oid dbid);
extern void DropRelationFiles(RelFileLocator *delrels, int ndelrels, bool isRedo);
diff --git a/src/include/storage/smgr.h b/src/include/storage/smgr.h
index 4016b206ad6..856ebcda350 100644
--- a/src/include/storage/smgr.h
+++ b/src/include/storage/smgr.h
@@ -15,6 +15,7 @@
#define SMGR_H
#include "lib/ilist.h"
+#include "storage/aio_types.h"
#include "storage/block.h"
#include "storage/relfilelocator.h"
@@ -73,6 +74,8 @@ typedef SMgrRelationData *SMgrRelation;
#define SmgrIsTemp(smgr) \
RelFileLocatorBackendIsTemp((smgr)->smgr_rlocator)
+extern const PgAioTargetInfo aio_smgr_target_info;
+
extern void smgrinit(void);
extern SMgrRelation smgropen(RelFileLocator rlocator, ProcNumber backend);
extern bool smgrexists(SMgrRelation reln, ForkNumber forknum);
@@ -97,6 +100,10 @@ extern uint32 smgrmaxcombine(SMgrRelation reln, ForkNumber forknum,
extern void smgrreadv(SMgrRelation reln, ForkNumber forknum,
BlockNumber blocknum,
void **buffers, BlockNumber nblocks);
+extern void smgrstartreadv(PgAioHandle *ioh,
+ SMgrRelation reln, ForkNumber forknum,
+ BlockNumber blocknum,
+ void **buffers, BlockNumber nblocks);
extern void smgrwritev(SMgrRelation reln, ForkNumber forknum,
BlockNumber blocknum,
const void **buffers, BlockNumber nblocks,
@@ -127,4 +134,11 @@ smgrwrite(SMgrRelation reln, ForkNumber forknum, BlockNumber blocknum,
smgrwritev(reln, forknum, blocknum, &buffer, 1, skipFsync);
}
+extern void pgaio_io_set_target_smgr(PgAioHandle *ioh,
+ SMgrRelationData *smgr,
+ ForkNumber forknum,
+ BlockNumber blocknum,
+ int nblocks,
+ bool skip_fsync);
+
#endif /* SMGR_H */
diff --git a/src/backend/storage/aio/aio_callback.c b/src/backend/storage/aio/aio_callback.c
index d5a2cca28f1..2d48aa7c1b3 100644
--- a/src/backend/storage/aio/aio_callback.c
+++ b/src/backend/storage/aio/aio_callback.c
@@ -18,6 +18,7 @@
#include "miscadmin.h"
#include "storage/aio.h"
#include "storage/aio_internal.h"
+#include "storage/md.h"
/* just to have something to put into aio_handle_cbs */
@@ -37,6 +38,8 @@ typedef struct PgAioHandleCallbacksEntry
static const PgAioHandleCallbacksEntry aio_handle_cbs[] = {
#define CALLBACK_ENTRY(id, callback) [id] = {.cb = &callback, .name = #callback}
CALLBACK_ENTRY(PGAIO_HCB_INVALID, aio_invalid_cb),
+
+ CALLBACK_ENTRY(PGAIO_HCB_MD_READV, aio_md_readv_cb),
#undef CALLBACK_ENTRY
};
diff --git a/src/backend/storage/aio/aio_target.c b/src/backend/storage/aio/aio_target.c
index b01406a6a52..6133b04f350 100644
--- a/src/backend/storage/aio/aio_target.c
+++ b/src/backend/storage/aio/aio_target.c
@@ -16,6 +16,7 @@
#include "storage/aio.h"
#include "storage/aio_internal.h"
+#include "storage/smgr.h"
/*
@@ -25,6 +26,7 @@ static const PgAioTargetInfo *pgaio_target_info[] = {
[PGAIO_TID_INVALID] = &(PgAioTargetInfo) {
.name = "invalid",
},
+ [PGAIO_TID_SMGR] = &aio_smgr_target_info,
};
diff --git a/src/backend/storage/file/fd.c b/src/backend/storage/file/fd.c
index 62f1185859f..34588ed5167 100644
--- a/src/backend/storage/file/fd.c
+++ b/src/backend/storage/file/fd.c
@@ -94,6 +94,7 @@
#include "miscadmin.h"
#include "pgstat.h"
#include "postmaster/startup.h"
+#include "storage/aio.h"
#include "storage/fd.h"
#include "storage/ipc.h"
#include "utils/guc.h"
@@ -1294,6 +1295,8 @@ LruDelete(File file)
vfdP = &VfdCache[file];
+ pgaio_closing_fd(vfdP->fd);
+
/*
* Close the file. We aren't expecting this to fail; if it does, better
* to leak the FD than to mess up our internal state.
@@ -1987,6 +1990,8 @@ FileClose(File file)
if (!FileIsNotOpen(file))
{
+ pgaio_closing_fd(vfdP->fd);
+
/* close the file */
if (close(vfdP->fd) != 0)
{
@@ -2210,6 +2215,32 @@ retry:
return returnCode;
}
+int
+FileStartReadV(PgAioHandle *ioh, File file,
+ int iovcnt, off_t offset,
+ uint32 wait_event_info)
+{
+ int returnCode;
+ Vfd *vfdP;
+
+ Assert(FileIsValid(file));
+
+ DO_DB(elog(LOG, "FileStartReadV: %d (%s) " INT64_FORMAT " %d",
+ file, VfdCache[file].fileName,
+ (int64) offset,
+ iovcnt));
+
+ returnCode = FileAccess(file);
+ if (returnCode < 0)
+ return returnCode;
+
+ vfdP = &VfdCache[file];
+
+ pgaio_io_prep_readv(ioh, vfdP->fd, iovcnt, offset);
+
+ return 0;
+}
+
ssize_t
FileWriteV(File file, const struct iovec *iov, int iovcnt, off_t offset,
uint32 wait_event_info)
@@ -2498,6 +2529,12 @@ FilePathName(File file)
int
FileGetRawDesc(File file)
{
+ int returnCode;
+
+ returnCode = FileAccess(file);
+ if (returnCode < 0)
+ return returnCode;
+
Assert(FileIsValid(file));
return VfdCache[file].fd;
}
@@ -2778,6 +2815,7 @@ FreeDesc(AllocateDesc *desc)
result = closedir(desc->desc.dir);
break;
case AllocateDescRawFD:
+ pgaio_closing_fd(desc->desc.fd);
result = close(desc->desc.fd);
break;
default:
@@ -2846,6 +2884,8 @@ CloseTransientFile(int fd)
/* Only get here if someone passes us a file not in allocatedDescs */
elog(WARNING, "fd passed to CloseTransientFile was not obtained from OpenTransientFile");
+ pgaio_closing_fd(fd);
+
return close(fd);
}
diff --git a/src/backend/storage/smgr/md.c b/src/backend/storage/smgr/md.c
index f3220f98dc4..11d4d5a7aea 100644
--- a/src/backend/storage/smgr/md.c
+++ b/src/backend/storage/smgr/md.c
@@ -31,6 +31,7 @@
#include "miscadmin.h"
#include "pg_trace.h"
#include "pgstat.h"
+#include "storage/aio.h"
#include "storage/bufmgr.h"
#include "storage/fd.h"
#include "storage/md.h"
@@ -152,6 +153,15 @@ static MdfdVec *_mdfd_getseg(SMgrRelation reln, ForkNumber forknum,
static BlockNumber _mdnblocks(SMgrRelation reln, ForkNumber forknum,
MdfdVec *seg);
+static PgAioResult md_readv_complete(PgAioHandle *ioh, PgAioResult prior_result, uint8 cb_data);
+static void md_readv_report(PgAioResult result, const PgAioTargetData *target_data, int elevel);
+
+const PgAioHandleCallbacks aio_md_readv_cb = {
+ .complete_shared = md_readv_complete,
+ .report = md_readv_report,
+};
+
+
static inline int
_mdfd_open_flags(void)
{
@@ -937,6 +947,61 @@ mdreadv(SMgrRelation reln, ForkNumber forknum, BlockNumber blocknum,
}
}
+/*
+ * mdstartreadv() -- Asynchronous version of mdreadv().
+ */
+void
+mdstartreadv(PgAioHandle *ioh,
+ SMgrRelation reln, ForkNumber forknum, BlockNumber blocknum,
+ void **buffers, BlockNumber nblocks)
+{
+ off_t seekpos;
+ MdfdVec *v;
+ BlockNumber nblocks_this_segment;
+ struct iovec *iov;
+ int iovcnt;
+
+ v = _mdfd_getseg(reln, forknum, blocknum, false,
+ EXTENSION_FAIL | EXTENSION_CREATE_RECOVERY);
+
+ seekpos = (off_t) BLCKSZ * (blocknum % ((BlockNumber) RELSEG_SIZE));
+
+ Assert(seekpos < (off_t) BLCKSZ * RELSEG_SIZE);
+
+ nblocks_this_segment =
+ Min(nblocks,
+ RELSEG_SIZE - (blocknum % ((BlockNumber) RELSEG_SIZE)));
+
+ if (nblocks_this_segment != nblocks)
+ elog(ERROR, "read crossing segment boundary");
+
+ iovcnt = pgaio_io_get_iovec(ioh, &iov);
+
+ Assert(nblocks <= iovcnt);
+
+ iovcnt = buffers_to_iovec(iov, buffers, nblocks_this_segment);
+
+ Assert(iovcnt <= nblocks_this_segment);
+
+ if (!(io_direct_flags & IO_DIRECT_DATA))
+ pgaio_io_set_flag(ioh, PGAIO_HF_BUFFERED);
+
+ pgaio_io_set_target_smgr(ioh,
+ reln,
+ forknum,
+ blocknum,
+ nblocks,
+ false);
+ pgaio_io_register_callbacks(ioh, PGAIO_HCB_MD_READV, 0);
+
+ FileStartReadV(ioh, v->mdfd_vfd, iovcnt, seekpos, WAIT_EVENT_DATA_FILE_READ);
+
+ /*
+ * The error checks corresponding to the post-read checks in mdread() are
+ * in md_readv_complete().
+ */
+}
+
/*
* mdwritev() -- Write the supplied blocks at the appropriate location.
*
@@ -1365,6 +1430,21 @@ mdimmedsync(SMgrRelation reln, ForkNumber forknum)
}
}
+int
+mdfd(SMgrRelation reln, ForkNumber forknum, BlockNumber blocknum, uint32 *off)
+{
+ MdfdVec *v = mdopenfork(reln, forknum, EXTENSION_FAIL);
+
+ v = _mdfd_getseg(reln, forknum, blocknum, false,
+ EXTENSION_FAIL);
+
+ *off = (off_t) BLCKSZ * (blocknum % ((BlockNumber) RELSEG_SIZE));
+
+ Assert(*off < (off_t) BLCKSZ * RELSEG_SIZE);
+
+ return FileGetRawDesc(v->mdfd_vfd);
+}
+
/*
* register_dirty_segment() -- Mark a relation segment as needing fsync
*
@@ -1841,3 +1921,99 @@ mdfiletagmatches(const FileTag *ftag, const FileTag *candidate)
*/
return ftag->rlocator.dbOid == candidate->rlocator.dbOid;
}
+
+/*
+ * AIO completion callback for mdstartreadv().
+ */
+static PgAioResult
+md_readv_complete(PgAioHandle *ioh, PgAioResult prior_result, uint8 cb_data)
+{
+ PgAioTargetData *td = pgaio_io_get_target_data(ioh);
+ PgAioResult result = prior_result;
+
+ if (prior_result.result < 0)
+ {
+ result.status = ARS_ERROR;
+ result.id = PGAIO_HCB_MD_READV;
+ /* For "hard" errors, track the error number in error_data */
+ result.error_data = -prior_result.result;
+ result.result = 0;
+
+ pgaio_result_report(result, td, LOG);
+
+ return result;
+ }
+
+ /*
+ * The smgr API operates in blocks, therefore convert the result from
+ * bytes to blocks.
+ */
+ result.result /= BLCKSZ;
+
+ if (result.result == 0)
+ {
+ /* consider 0 blocks read a failure */
+ result.status = ARS_ERROR;
+ result.id = PGAIO_HCB_MD_READV;
+ result.error_data = 0;
+
+ pgaio_result_report(result, td, LOG);
+
+ return result;
+ }
+
+ if (result.status != ARS_ERROR &&
+ result.result < td->smgr.nblocks)
+ {
+ /* partial reads should be retried at upper level */
+ result.status = ARS_PARTIAL;
+ result.id = PGAIO_HCB_MD_READV;
+ }
+
+ return result;
+}
+
+/*
+ * AIO error reporting callback for mdstartreadv().
+ *
+ * Errors are encoded as follows:
+ * - PgAioResult.error_data != 0 encodes IO that failed with errno != 0
+ * - PgAioResult.error_data == 0 encodes IO that didn't read all data
+ */
+static void
+md_readv_report(PgAioResult result, const PgAioTargetData *td, int elevel)
+{
+ RelPathStr path;
+
+ path = relpathbackend(td->smgr.rlocator,
+ td->smgr.is_temp ? MyProcNumber : INVALID_PROC_NUMBER,
+ td->smgr.forkNum);
+
+ if (result.error_data != 0)
+ {
+ /* for errcode_for_file_access() and %m */
+ errno = result.error_data;
+
+ ereport(elevel,
+ errcode_for_file_access(),
+ errmsg("could not read blocks %u..%u in file \"%s\": %m",
+ td->smgr.blockNum,
+ td->smgr.blockNum + td->smgr.nblocks - 1,
+ path.str));
+ }
+ else
+ {
+ /*
+ * NB: This will typically only be output in debug messages, while
+ * retrying a partial IO.
+ */
+ ereport(elevel,
+ errcode(ERRCODE_DATA_CORRUPTED),
+ errmsg("could not read blocks %u..%u in file \"%s\": read only %zu of %zu bytes",
+ td->smgr.blockNum,
+ td->smgr.blockNum + td->smgr.nblocks - 1,
+ path.str,
+ result.result * (size_t) BLCKSZ,
+ td->smgr.nblocks * (size_t) BLCKSZ));
+ }
+}
diff --git a/src/backend/storage/smgr/smgr.c b/src/backend/storage/smgr/smgr.c
index 8787ce9b18f..06b2acf8fea 100644
--- a/src/backend/storage/smgr/smgr.c
+++ b/src/backend/storage/smgr/smgr.c
@@ -63,6 +63,7 @@
#include "access/xlogutils.h"
#include "lib/ilist.h"
#include "miscadmin.h"
+#include "storage/aio.h"
#include "storage/bufmgr.h"
#include "storage/ipc.h"
#include "storage/md.h"
@@ -103,6 +104,10 @@ typedef struct f_smgr
void (*smgr_readv) (SMgrRelation reln, ForkNumber forknum,
BlockNumber blocknum,
void **buffers, BlockNumber nblocks);
+ void (*smgr_startreadv) (PgAioHandle *ioh,
+ SMgrRelation reln, ForkNumber forknum,
+ BlockNumber blocknum,
+ void **buffers, BlockNumber nblocks);
void (*smgr_writev) (SMgrRelation reln, ForkNumber forknum,
BlockNumber blocknum,
const void **buffers, BlockNumber nblocks,
@@ -114,6 +119,7 @@ typedef struct f_smgr
BlockNumber old_blocks, BlockNumber nblocks);
void (*smgr_immedsync) (SMgrRelation reln, ForkNumber forknum);
void (*smgr_registersync) (SMgrRelation reln, ForkNumber forknum);
+ int (*smgr_fd) (SMgrRelation reln, ForkNumber forknum, BlockNumber blocknum, uint32 *off);
} f_smgr;
static const f_smgr smgrsw[] = {
@@ -131,12 +137,14 @@ static const f_smgr smgrsw[] = {
.smgr_prefetch = mdprefetch,
.smgr_maxcombine = mdmaxcombine,
.smgr_readv = mdreadv,
+ .smgr_startreadv = mdstartreadv,
.smgr_writev = mdwritev,
.smgr_writeback = mdwriteback,
.smgr_nblocks = mdnblocks,
.smgr_truncate = mdtruncate,
.smgr_immedsync = mdimmedsync,
.smgr_registersync = mdregistersync,
+ .smgr_fd = mdfd,
}
};
@@ -154,6 +162,16 @@ static dlist_head unpinned_relns;
static void smgrshutdown(int code, Datum arg);
static void smgrdestroy(SMgrRelation reln);
+static void smgr_aio_reopen(PgAioHandle *ioh);
+static char *smgr_aio_describe_identity(const PgAioTargetData *sd);
+
+
+const PgAioTargetInfo aio_smgr_target_info = {
+ .name = "smgr",
+ .reopen = smgr_aio_reopen,
+ .describe_identity = smgr_aio_describe_identity,
+};
+
/*
* smgrinit(), smgrshutdown() -- Initialize or shut down storage
@@ -697,6 +715,24 @@ smgrreadv(SMgrRelation reln, ForkNumber forknum, BlockNumber blocknum,
RESUME_INTERRUPTS();
}
+/*
+ * smgrstartreadv() -- asynchronous version of smgrreadv()
+ *
+ * This starts an asynchronous readv IO with the IO handle `ioh`. Other than
+ * `ioh` all parameters are the same as smgrreadv().
+ */
+void
+smgrstartreadv(PgAioHandle *ioh,
+ SMgrRelation reln, ForkNumber forknum, BlockNumber blocknum,
+ void **buffers, BlockNumber nblocks)
+{
+ HOLD_INTERRUPTS();
+ smgrsw[reln->smgr_which].smgr_startreadv(ioh,
+ reln, forknum, blocknum, buffers,
+ nblocks);
+ RESUME_INTERRUPTS();
+}
+
/*
* smgrwritev() -- Write the supplied buffers out.
*
@@ -907,6 +943,25 @@ smgrimmedsync(SMgrRelation reln, ForkNumber forknum)
RESUME_INTERRUPTS();
}
+/*
+ * Return fd for the specified block number and update *off to the appropriate
+ * position.
+ *
+ * This is only to be used for when AIO needs to perform the IO in a different
+ * process than where it was issued (e.g. in an IO worker).
+ */
+static int
+smgrfd(SMgrRelation reln, ForkNumber forknum, BlockNumber blocknum, uint32 *off)
+{
+ int fd;
+
+ HOLD_INTERRUPTS();
+ fd = smgrsw[reln->smgr_which].smgr_fd(reln, forknum, blocknum, off);
+ RESUME_INTERRUPTS();
+
+ return fd;
+}
+
/*
* AtEOXact_SMgr
*
@@ -935,3 +990,92 @@ ProcessBarrierSmgrRelease(void)
smgrreleaseall();
return true;
}
+
+/*
+ * Set target of the IO handle to be smgr and initialize all the relevant
+ * pieces of data.
+ */
+void
+pgaio_io_set_target_smgr(PgAioHandle *ioh,
+ SMgrRelationData *smgr,
+ ForkNumber forknum,
+ BlockNumber blocknum,
+ int nblocks,
+ bool skip_fsync)
+{
+ PgAioTargetData *sd = pgaio_io_get_target_data(ioh);
+
+ pgaio_io_set_target(ioh, PGAIO_TID_SMGR);
+
+ /* backend is implied via IO owner */
+ sd->smgr.rlocator = smgr->smgr_rlocator.locator;
+ sd->smgr.forkNum = forknum;
+ sd->smgr.blockNum = blocknum;
+ sd->smgr.nblocks = nblocks;
+ sd->smgr.is_temp = SmgrIsTemp(smgr);
+ /* Temp relations should never be fsync'd */
+ sd->smgr.skip_fsync = skip_fsync && !SmgrIsTemp(smgr);
+}
+
+/*
+ * Callback for smgr AIO targets, to reopen the file (e.g. because the IO is
+ * executed in a worker).
+ */
+static void
+smgr_aio_reopen(PgAioHandle *ioh)
+{
+ PgAioTargetData *sd = pgaio_io_get_target_data(ioh);
+ PgAioOpData *od = pgaio_io_get_op_data(ioh);
+ SMgrRelation reln;
+ ProcNumber procno;
+ uint32 off;
+
+ if (sd->smgr.is_temp)
+ procno = pgaio_io_get_owner(ioh);
+ else
+ procno = INVALID_PROC_NUMBER;
+
+ reln = smgropen(sd->smgr.rlocator, procno);
+ switch (pgaio_io_get_op(ioh))
+ {
+ case PGAIO_OP_INVALID:
+ pg_unreachable();
+ break;
+ case PGAIO_OP_READV:
+ od->read.fd = smgrfd(reln, sd->smgr.forkNum, sd->smgr.blockNum, &off);
+ Assert(off == od->read.offset);
+ break;
+ case PGAIO_OP_WRITEV:
+ od->write.fd = smgrfd(reln, sd->smgr.forkNum, sd->smgr.blockNum, &off);
+ Assert(off == od->write.offset);
+ break;
+ }
+}
+
+/*
+ * Callback for smgr AIO targets, to describe the target of the IO.
+ */
+static char *
+smgr_aio_describe_identity(const PgAioTargetData *sd)
+{
+ RelPathStr path;
+ char *desc;
+
+ path = relpathbackend(sd->smgr.rlocator,
+ sd->smgr.is_temp ? MyProcNumber : INVALID_PROC_NUMBER,
+ sd->smgr.forkNum);
+
+ if (sd->smgr.nblocks == 0)
+ desc = psprintf(_("file \"%s\""), path.str);
+ else if (sd->smgr.nblocks == 1)
+ desc = psprintf(_("block %u in file \"%s\""),
+ sd->smgr.blockNum,
+ path.str);
+ else
+ desc = psprintf(_("blocks %u..%u in file \"%s\""),
+ sd->smgr.blockNum,
+ sd->smgr.blockNum + sd->smgr.nblocks - 1,
+ path.str);
+
+ return desc;
+}
--
2.48.1.76.g4e746b1a31.dirty
[text/x-diff] v2.9-0009-aio-Add-README.md-explaining-higher-level-desig.patch (19.0K, ../../oav52ytakcvfnk52w5o5wvxdjavxth2tgptxt25gnfmh2pdcpk@omp3xukdu5if/10-v2.9-0009-aio-Add-README.md-explaining-higher-level-desig.patch)
download | inline diff:
From 6b1664d4a346e1bce5f2b7fc8abfe49044e45a9e Mon Sep 17 00:00:00 2001
From: Andres Freund <andres@anarazel.de>
Date: Sat, 15 Mar 2025 12:29:57 -0400
Subject: [PATCH v2.9 09/30] aio: Add README.md explaining higher level design
Reviewed-by: Noah Misch <noah@leadboat.com>
Reviewed-by: Heikki Linnakangas <hlinnaka@iki.fi>
Discussion: https://postgr.es/m/uvrtrknj4kdytuboidbhwclo4gxhswwcpgadptsjvjqcluzmah%40brqs62irg4dt
Discussion: https://postgr.es/m/20210223100344.llw5an2aklengrmn@alap3.anarazel.de
Discussion: https://postgr.es/m/stj36ea6yyhoxtqkhpieia2z4krnam7qyetc57rfezgk4zgapf@gcnactj4z56m
---
src/backend/storage/aio/README.md | 423 ++++++++++++++++++++++++++++++
src/backend/storage/aio/aio.c | 2 +
2 files changed, 425 insertions(+)
create mode 100644 src/backend/storage/aio/README.md
diff --git a/src/backend/storage/aio/README.md b/src/backend/storage/aio/README.md
new file mode 100644
index 00000000000..b7dfb80b4b2
--- /dev/null
+++ b/src/backend/storage/aio/README.md
@@ -0,0 +1,423 @@
+# Asynchronous & Direct IO
+
+## Motivation
+
+### Why Asynchronous IO
+
+Until the introduction of asynchronous IO Postgres relied on the operating
+system to hide the cost of synchronous IO from Postgres. While this worked
+surprisingly well in a lot of workloads, it does not do as good a job on
+prefetching and controlled writeback as we would like.
+
+There are important expensive operations like `fdatasync()` where the operating
+system cannot hide the storage latency. This is particularly important for WAL
+writes, where the ability to asynchronously issue `fdatasync()` or O_DSYNC
+writes can yield significantly higher throughput.
+
+
+### Why Direct / unbuffered IO
+
+The main reasons to want to use Direct IO are:
+
+- Lower CPU usage / higher throughput. Particularly on modern storage buffered
+ writes are bottlenecked by the operating system having to copy data from the
+ kernel's page cache to postgres buffer pool using the CPU. Whereas direct IO
+ can often move the data directly between the storage devices and postgres'
+ buffer cache, using DMA. While that transfer is ongoing, the CPU is free to
+ perform other work.
+- Reduced latency - Direct IO can have substantially lower latency than
+ buffered IO, which can be impactful for OLTP workloads bottlenecked by WAL
+ write latency.
+- Avoiding double buffering between operating system cache and postgres'
+ shared_buffers.
+- Better control over the timing and pace of dirty data writeback.
+
+
+The main reasons *not* to use Direct IO are:
+
+- Without AIO, Direct IO is unusably slow for most purposes.
+- Even with AIO, many parts of postgres need to be modified to perform
+ explicit prefetching.
+- In situations where shared_buffers cannot be set appropriately large,
+ e.g. because there are many different postgres instances hosted on shared
+ hardware, performance will often be worse than when using buffered IO.
+
+
+## AIO Usage Example
+
+In many cases code that can benefit from AIO does not directly have to
+interact with the AIO interface, but can use AIO via higher-level
+abstractions. See [Helpers](#helpers).
+
+In this example, a buffer will be read into shared buffers.
+
+```C
+/*
+ * Result of the operation, only to be accessed in this backend.
+ */
+PgAioReturn ioret;
+
+/*
+ * Acquire an AIO Handle, ioret will get result upon completion.
+ *
+ * Note that ioret needs to stay alive until the IO completes or
+ * CurrentResourceOwner is released (i.e. an error is thrown).
+ */
+PgAioHandle *ioh = pgaio_io_acquire(CurrentResourceOwner, &ioret);
+
+/*
+ * Reference that can be used to wait for the IO we initiate below. This
+ * reference can reside in local or shared memory and waited upon by any
+ * process. An arbitrary number of references can be made for each IO.
+ */
+PgAioWaitRef iow;
+
+pgaio_io_get_wref(ioh, &iow);
+
+/*
+ * Arrange for shared buffer completion callbacks to be called upon completion
+ * of the IO. This callback will update the buffer descriptors associated with
+ * the AioHandle, which e.g. allows other backends to access the buffer.
+ *
+ * A callback can be passed a small bit of data, e.g. to indicate whether to
+ * zero a buffer if it is invalid.
+ *
+ * Multiple completion callbacks can be registered for each handle.
+ */
+pgaio_io_register_callbacks(ioh, PGAIO_HCB_SHARED_BUFFER_READV, 0);
+
+/*
+ * The completion callback needs to know which buffers to update when the IO
+ * completes. As the AIO subsystem does not know about buffers, we have to
+ * associate this information with the AioHandle, for use by the completion
+ * callback registered above.
+ *
+ * In this example we're reading only a single buffer, hence the 1.
+ */
+pgaio_io_set_handle_data_32(ioh, (uint32 *) buffer, 1);
+
+/*
+ * Pass the AIO handle to lower-level function. When operating on the level of
+ * buffers, we don't know how exactly the IO is performed, that is the
+ * responsibility of the storage manager implementation.
+ *
+ * E.g. md.c needs to translate block numbers into offsets in segments.
+ *
+ * Once the IO handle has been handed off to smgstartreadv(), it may not
+ * further be used, as the IO may immediately get executed below
+ * smgrstartreadv() and the handle reused for another IO.
+ *
+ * To issue multiple IOs in an efficient way, a caller can call
+ * pgaio_enter_batchmode() before starting multiple IOs, and end that batch
+ * with pgaio_exit_batchmode(). Note that one needs to be careful while there
+ * may be unsubmitted IOs, as another backend may need to wait for one of the
+ * unsubmitted IOs. If this backend then had to to wait for the other backend,
+ * it'd end in an undetected deadlock. See pgaio_enter_batchmode() for more
+ * details.
+ *
+ * Note that even while in batchmode an IO might get submitted immediately,
+ * e.g. due to reaching a limit on the number of unsubmitted IOs, and even
+ * complete before smgrstartreadv() returns.
+ */
+smgrstartreadv(ioh, operation->smgr, forknum, blkno,
+ BufferGetBlock(buffer), 1);
+
+/*
+ * To benefit from AIO, it is beneficial to perform other work, including
+ * submitting other IOs, before waiting for the IO to complete. Otherwise
+ * we could just have used synchronous, blocking IO.
+ */
+perform_other_work();
+
+/*
+ * We did some other work and now need the IO operation to have completed to
+ * continue.
+ */
+pgaio_wref_wait(&iow);
+
+/*
+ * At this point the IO has completed. We do not yet know whether it succeeded
+ * or failed, however. The buffer's state has been updated, which allows other
+ * backends to use the buffer (if the IO succeeded), or retry the IO (if it
+ * failed).
+ *
+ * Note that in case the IO has failed, a LOG message may have been emitted,
+ * but no ERROR has been raised. This is crucial, as another backend waiting
+ * for this IO should not see an ERROR.
+ *
+ * To check whether the operation succeeded, and to raise an ERROR, or if more
+ * appropriate LOG, the PgAioReturn we passed to pgaio_io_acquire() is used.
+ */
+if (ioret.result.status == ARS_ERROR)
+ pgaio_result_report(aio_ret.result, &aio_ret.target_data, ERROR);
+
+/*
+ * Besides having succeeded completely, the IO could also have partially
+ * completed. If we e.g. tried to read many blocks at once, the read might have
+ * only succeeded for the first few blocks.
+ *
+ * If the IO partially succeeded and this backend needs all blocks to have
+ * completed, this backend needs to reissue the IO for the remaining buffers.
+ * The AIO subsystem cannot handle this retry transparently.
+ *
+ * As this example is already long, and we only read a single block, we'll just
+ * error out if there's a partial read.
+ */
+if (ioret.result.status == ARS_PARTIAL)
+ pgaio_result_report(aio_ret.result, &aio_ret.target_data, ERROR);
+
+/*
+ * The IO succeeded, so we can use the buffer now.
+ */
+```
+
+
+## Design Criteria & Motivation
+
+### Deadlock and Starvation Dangers due to AIO
+
+Using AIO in a naive way can easily lead to deadlocks in an environment where
+the source/target of AIO are shared resources, like pages in postgres'
+shared_buffers.
+
+Consider one backend performing readahead on a table, initiating IO for a
+number of buffers ahead of the current "scan position". If that backend then
+performs some operation that blocks, or even just is slow, the IO completion
+for the asynchronously initiated read may not be processed.
+
+This AIO implementation solves this problem by requiring that AIO methods
+either allow AIO completions to be processed by any backend in the system
+(e.g. io_uring), or to guarantee that AIO processing will happen even when the
+issuing backend is blocked (e.g. worker mode, which offloads completion
+processing to the AIO workers).
+
+
+### IO can be started in critical sections
+
+Using AIO for WAL writes can reduce the overhead of WAL logging substantially:
+
+- AIO allows to start WAL writes eagerly, so they complete before needing to
+ wait
+- AIO allows to have multiple WAL flushes in progress at the same time
+- AIO makes it more realistic to use O\_DIRECT + O\_DSYNC, which can reduce
+ the number of roundtrips to storage on some OSs and storage HW (buffered IO
+ and direct IO without O_DSYNC needs to issue a write and after the write's
+ completion a cache cache flush, whereas O\_DIRECT + O\_DSYNC can use a
+ single Force Unit Access (FUA) write).
+
+The need to be able to execute IO in critical sections has substantial design
+implication on the AIO subsystem. Mainly because completing IOs (see prior
+section) needs to be possible within a critical section, even if the
+to-be-completed IO itself was not issued in a critical section. Consider
+e.g. the case of a backend first starting a number of writes from shared
+buffers and then starting to flush the WAL. Because only a limited amount of
+IO can be in-progress at the same time, initiating IO for flushing the WAL may
+require to first complete IO that was started earlier.
+
+
+### State for AIO needs to live in shared memory
+
+Because postgres uses a process model and because AIOs need to be
+complete-able by any backend much of the state of the AIO subsystem needs to
+live in shared memory.
+
+In an `EXEC_BACKEND` build, a backend's executable code and other process
+local state is not necessarily mapped to the same addresses in each process
+due to ASLR. This means that the shared memory cannot contain pointers to
+callbacks.
+
+
+## Design of the AIO Subsystem
+
+
+### AIO Methods
+
+To achieve portability and performance, multiple methods of performing AIO are
+implemented and others are likely worth adding in the future.
+
+
+#### Synchronous Mode
+
+`io_method=sync` does not actually perform AIO but allows to use the AIO API
+while performing synchronous IO. This can be useful for debugging. The code
+for the synchronous mode is also used as a fallback by e.g. the [worker
+mode](#worker) uses it to execute IO that cannot be executed by workers.
+
+
+#### Worker
+
+`io_method=worker` is available on every platform postgres runs on, and
+implements asynchronous IO - from the view of the issuing process - by
+dispatching the IO to one of several worker processes performing the IO in a
+synchronous manner.
+
+
+#### io_uring
+
+`io_method=io_uring` is available on Linux 5.1+. In contrast to worker mode it
+dispatches all IO from within the process, lowering context switch rate /
+latency.
+
+
+### AIO Handles
+
+The central API piece for postgres' AIO abstraction are AIO handles. To
+execute an IO one first has to acquire an IO handle (`pgaio_io_acquire()`) and
+then "defined", i.e. associate an IO operation with the handle.
+
+Often AIO handles are acquired on a higher level and then passed to a lower
+level to be fully defined. E.g., for IO to/from shared buffers, bufmgr.c
+routines acquire the handle, which is then passed through smgr.c, md.c to be
+finally fully defined in fd.c.
+
+The functions used at the lowest level to define the operation are
+`pgaio_io_prep_*()`.
+
+Because acquisition of an IO handle
+[must always succeed](#io-can-be-started-in-critical-sections)
+and the number of AIO Handles
+[has to be limited](#state-for-aio-needs-to-live-in-shared-memory)
+AIO handles can be reused as soon as they have completed. Obviously code needs
+to be able to react to IO completion. State can be updated using
+[AIO Completion callbacks](#aio-callbacks)
+and the issuing backend can provide a backend local variable to receive the
+result of the IO, as described in
+[AIO Result](#aio-results).
+An IO can be waited for, by both the issuing and any other backend, using
+[AIO References](#aio-wait-references).
+
+
+Because an AIO Handle is not executable just after calling
+`pgaio_io_acquire()` and because `pgaio_io_acquire()` needs to always succeed
+(absent a PANIC), only a single AIO Handle may be acquired (i.e. returned by
+`pgaio_io_acquire()`) without causing the IO to have been defined (by,
+potentially indirectly, causing `pgaio_io_prep_*()` to have been
+called). Otherwise a backend could trivially self-deadlock by using up all AIO
+Handles without the ability to wait for some of the IOs to complete.
+
+If it turns out that an AIO Handle is not needed, e.g., because the handle was
+acquired before holding a contended lock, it can be released without being
+defined using `pgaio_io_release()`.
+
+
+### AIO Callbacks
+
+Commonly several layers need to react to completion of an IO. E.g. for a read
+md.c needs to check if the IO outright failed or was shorter than needed,
+bufmgr.c needs to verify the page looks valid and bufmgr.c needs to update the
+BufferDesc to update the buffer's state.
+
+The fact that several layers / subsystems need to react to IO completion poses
+a few challenges:
+
+- Upper layers should not need to know details of lower layers. E.g. bufmgr.c
+ should not assume the IO will pass through md.c. Therefore upper levels
+ cannot know what lower layers would consider an error.
+
+- Lower layers should not need to know about upper layers. E.g. smgr APIs are
+ used going through shared buffers but are also used bypassing shared
+ buffers. This means that e.g. md.c is not in a position to validate
+ checksums.
+
+- Having code in the AIO subsystem for every possible combination of layers
+ would lead to a lot of duplication.
+
+The "solution" to this the ability to associate multiple completion callbacks
+with a handle. E.g. bufmgr.c can have a callback to update the BufferDesc
+state and to verify the page and md.c can have another callback to check if
+the IO operation was successful.
+
+As [mentioned](#state-for-aio-needs-to-live-in-shared-memory), shared memory
+currently cannot contain function pointers. Because of that completion
+callbacks are not directly identified by function pointers but by IDs
+(`PgAioHandleCallbackID`). A substantial added benefit is that that
+allows callbacks to be identified by much smaller amount of memory (a single
+byte currently).
+
+In addition to completion, AIO callbacks also are called to "stage" an
+IO. This is, e.g., used to increase buffer reference counts to account for the
+AIO subsystem referencing the buffer, which is required to handle the case
+where the issuing backend errors out and releases its own pins while the IO is
+still ongoing.
+
+As [explained earlier](#io-can-be-started-in-critical-sections) IO completions
+need to be safe to execute in critical sections. To allow the backend that
+issued the IO to error out in case of failure [AIO Result](#aio-results) can
+be used.
+
+
+### AIO Targets
+
+In addition to the completion callbacks describe above, each AIO Handle has
+exactly one "target". Each target has some space inside an AIO Handle with
+information specific to the target and can provide callbacks to allow to
+reopen the underlying file (required for worker mode) and to describe the IO
+operation (used for debug logging and error messages).
+
+I.e., if two different uses of AIO can describe the identity of the file being
+operated on the same way, it likely makes sense to use the same
+target. E.g. different smgr implementations can describe IO with
+RelFileLocator, ForkNumber and BlockNumber and can thus share a target. In
+contrast, IO for a WAL file would be described with TimeLineID and XLogRecPtr
+and it would not make sense to use the same target for smgr and WAL.
+
+
+### AIO Wait References
+
+As [described above](#aio-handles), AIO Handles can be reused immediately
+after completion and therefore cannot be used to wait for completion of the
+IO. Waiting is enabled using AIO wait references, which do not just identify
+an AIO Handle but also include the handles "generation".
+
+A reference to an AIO Handle can be acquired using `pgaio_io_get_wref()` and
+then waited upon using `pgaio_wref_wait()`.
+
+
+### AIO Results
+
+As AIO completion callbacks
+[are executed in critical sections](#io-can-be-started-in-critical-sections)
+and [may be executed by any backend](#deadlock-and-starvation-dangers-due-to-aio)
+completion callbacks cannot be used to, e.g., make the query that triggered an
+IO ERROR out.
+
+To allow to react to failing IOs the issuing backend can pass a pointer to a
+`PgAioReturn` in backend local memory. Before an AIO Handle is reused the
+`PgAioReturn` is filled with information about the IO. This includes
+information about whether the IO was successful (as a value of
+`PgAioResultStatus`) and enough information to raise an error in case of a
+failure (via `pgaio_result_report()`, with the error details encoded in
+`PgAioResult`).
+
+
+### AIO Errors
+
+It would be very convenient to have shared completion callbacks encode the
+details of errors as an `ErrorData` that could be raised at a later
+time. Unfortunately doing so would require allocating memory. While elog.c can
+guarantee (well, kinda) that logging a message will not run out of memory,
+that only works because a very limited number of messages are in the process
+of being logged. With AIO a large number of concurrently issued AIOs might
+fail.
+
+To avoid the need for preallocating a potentially large amount of memory (in
+shared memory no less!), completion callbacks instead have to encode errors in
+a more compact format that can be converted into an error message.
+
+
+## Helpers
+
+Using the low-level AIO API introduces too much complexity to do so all over
+the tree. Most uses of AIO should be done via reusable, higher-level,
+helpers.
+
+
+### Read Stream
+
+A common and very beneficial use of AIO are reads where a substantial number
+of to-be-read locations are known ahead of time. E.g., for a sequential scan
+the set of blocks that need to be read can be determined solely by knowing the
+current position and checking the buffer mapping table.
+
+The [Read Stream](../../../include/storage/read_stream.h) interface makes it
+comparatively easy to use AIO for such use cases.
diff --git a/src/backend/storage/aio/aio.c b/src/backend/storage/aio/aio.c
index 2e210335872..a0504ee84ee 100644
--- a/src/backend/storage/aio/aio.c
+++ b/src/backend/storage/aio/aio.c
@@ -24,6 +24,8 @@
*
* - read_stream.c - helper for reading buffered relation data
*
+ * - README.md - higher-level overview over AIO
+ *
*
* Portions Copyright (c) 1996-2025, PostgreSQL Global Development Group
* Portions Copyright (c) 1994, Regents of the University of California
--
2.48.1.76.g4e746b1a31.dirty
[text/x-diff] v2.9-0010-aio-Add-pg_aios-view.patch (9.7K, ../../oav52ytakcvfnk52w5o5wvxdjavxth2tgptxt25gnfmh2pdcpk@omp3xukdu5if/11-v2.9-0010-aio-Add-pg_aios-view.patch)
download | inline diff:
From 876c89f4643b08e733771320b193872e337c8e1e Mon Sep 17 00:00:00 2001
From: Andres Freund <andres@anarazel.de>
Date: Sat, 15 Mar 2025 12:29:57 -0400
Subject: [PATCH v2.9 10/30] aio: Add pg_aios view
Author:
Reviewed-by:
Discussion: https://postgr.es/m/
Backpatch:
---
src/include/catalog/pg_proc.dat | 10 ++
src/backend/catalog/system_views.sql | 3 +
src/backend/storage/aio/Makefile | 1 +
src/backend/storage/aio/aio_funcs.c | 219 +++++++++++++++++++++++++++
src/backend/storage/aio/meson.build | 1 +
src/test/regress/expected/rules.out | 16 ++
6 files changed, 250 insertions(+)
create mode 100644 src/backend/storage/aio/aio_funcs.c
diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat
index 890822eaf79..017971011f3 100644
--- a/src/include/catalog/pg_proc.dat
+++ b/src/include/catalog/pg_proc.dat
@@ -12479,4 +12479,14 @@
proargtypes => 'int4',
prosrc => 'gist_stratnum_common' },
+# AIO related functions
+{ oid => '9200', descr => 'information about in-progress asynchronous IOs',
+ proname => 'pg_get_aios', prorows => '100', proretset => 't',
+ provolatile => 'v', proparallel => 'r', prorettype => 'record', proargtypes => '',
+ proallargtypes => '{int4,int4,int8,text,text,int8,int8,text,int2,int4,text,text,bool,bool,bool}',
+ proargmodes => '{o,o,o,o,o,o,o,o,o,o,o,o,o,o,o}',
+ proargnames => '{pid,io_id,io_generation,state,operation,offset,length,target,handle_data_len,raw_result,result,target_desc,f_sync,f_localmem,f_buffered}',
+ prosrc => 'pg_get_aios' },
+
+
]
diff --git a/src/backend/catalog/system_views.sql b/src/backend/catalog/system_views.sql
index a4d2cfdcaf5..73902a763d1 100644
--- a/src/backend/catalog/system_views.sql
+++ b/src/backend/catalog/system_views.sql
@@ -1390,3 +1390,6 @@ CREATE VIEW pg_stat_subscription_stats AS
CREATE VIEW pg_wait_events AS
SELECT * FROM pg_get_wait_events();
+
+CREATE VIEW pg_aios AS
+ SELECT * FROM pg_get_aios();
diff --git a/src/backend/storage/aio/Makefile b/src/backend/storage/aio/Makefile
index c06c50771e0..3f2469cc399 100644
--- a/src/backend/storage/aio/Makefile
+++ b/src/backend/storage/aio/Makefile
@@ -11,6 +11,7 @@ include $(top_builddir)/src/Makefile.global
OBJS = \
aio.o \
aio_callback.o \
+ aio_funcs.o \
aio_init.o \
aio_io.o \
aio_target.o \
diff --git a/src/backend/storage/aio/aio_funcs.c b/src/backend/storage/aio/aio_funcs.c
new file mode 100644
index 00000000000..58883fd61d8
--- /dev/null
+++ b/src/backend/storage/aio/aio_funcs.c
@@ -0,0 +1,219 @@
+/*-------------------------------------------------------------------------
+ *
+ * aio_funcs.c
+ * AIO - SQL interface for AIO
+ *
+ *
+ * Portions Copyright (c) 1996-2025, PostgreSQL Global Development Group
+ * Portions Copyright (c) 1994, Regents of the University of California
+ *
+ * IDENTIFICATION
+ * src/backend/storage/aio/aio_funcs.c
+ *
+ *-------------------------------------------------------------------------
+ */
+
+#include "postgres.h"
+
+#include "storage/aio.h"
+#include "storage/aio_internal.h"
+#include "utils/builtins.h"
+#include "funcapi.h"
+#include "storage/proc.h"
+
+
+/*
+ * Byte length of an iovec.
+ */
+static size_t
+iov_byte_length(const struct iovec *iov, int cnt)
+{
+ size_t len = 0;
+
+ for (int i = 0; i < cnt; i++)
+ {
+ len += iov[i].iov_len;
+ }
+
+ return len;
+}
+
+Datum
+pg_get_aios(PG_FUNCTION_ARGS)
+{
+ ReturnSetInfo *rsinfo = (ReturnSetInfo *) fcinfo->resultinfo;
+
+ InitMaterializedSRF(fcinfo, 0);
+
+#define PG_GET_AIOS_COLS 15
+
+ for (uint64 i = 0; i < pgaio_ctl->io_handle_count; i++)
+ {
+ PgAioHandle *live_ioh = &pgaio_ctl->io_handles[i];
+ uint32 ioh_id = pgaio_io_get_id(live_ioh);
+ Datum values[PG_GET_AIOS_COLS] = {0};
+ bool nulls[PG_GET_AIOS_COLS] = {0};
+ ProcNumber owner;
+ PGPROC *owner_proc;
+ int32 owner_pid;
+ PgAioHandleState start_state;
+ uint64 start_generation;
+ PgAioHandle ioh_copy;
+ struct iovec iov_copy[PG_IOV_MAX];
+
+retry:
+
+ /*
+ * There is no lock that could prevent the state of the IO to advance
+ * concurrently - and we don't want to introduce one, as that would
+ * introduce atomics into a very common path. Instead we
+ *
+ * 1) Determine the state + generation of the IO.
+ *
+ * 2) Copy the IO to local memory.
+ *
+ * 3) Check if state or generation of the IO changed. If the state
+ * changed, retry, if the generation changed don't display the IO.
+ */
+
+ /* 1) from above */
+ start_generation = live_ioh->generation;
+ pg_read_barrier();
+ start_state = live_ioh->state;
+
+ if (start_state == PGAIO_HS_IDLE)
+ continue;
+
+ /* 2) from above */
+ memcpy(&ioh_copy, live_ioh, sizeof(PgAioHandle));
+
+ /*
+ * Safe to copy even if no iovec is used - we always reserve the
+ * required space.
+ */
+ memcpy(&iov_copy, &pgaio_ctl->iovecs[ioh_copy.iovec_off],
+ PG_IOV_MAX * sizeof(struct iovec));
+
+ /*
+ * Copy information about owner before 3) below, if the process exited
+ * it'd have to wait for the IO to finish first, which we would detect
+ * in 3).
+ */
+ owner = ioh_copy.owner_procno;
+ owner_proc = GetPGProcByNumber(owner);
+ owner_pid = owner_proc->pid;
+
+ /* 3) from above */
+ pg_read_barrier();
+
+ /*
+ * The IO completed and a new one was started with the same ID. Don't
+ * display it - it really started after this function was called.
+ * There be a risk of a livelock if we just retried endlessly, if IOs
+ * complete very quickly.
+ */
+ if (live_ioh->generation != start_generation)
+ continue;
+
+ /*
+ * The IOs state changed while we were "rendering" it. Just start from
+ * scratch. There's no risk of a livelock here, as an IO has a limited
+ * sets of states it can be in, and state changes go only in a single
+ * direction.
+ */
+ if (live_ioh->state != start_state)
+ goto retry;
+
+ /*
+ * Now that we have copied the IO into local memory and checked that
+ * it's still in the same state, we are not allowed to access "live"
+ * memory anymore. To make it slightly easier to catch such cases, set
+ * the "live" pointers to NULL.
+ */
+ live_ioh = NULL;
+ owner_proc = NULL;
+
+
+ /* column: owning pid */
+ if (owner_pid != 0)
+ values[0] = Int32GetDatum(owner_pid);
+ else
+ nulls[0] = false;
+
+ /* column: IO's id */
+ values[1] = ioh_id;
+
+ /* column: IO's generation */
+ values[2] = Int64GetDatum(start_generation);
+
+ /* column: IO's state */
+ values[3] = CStringGetTextDatum(pgaio_io_get_state_name(&ioh_copy));
+
+ /*
+ * If the IO is in PGAIO_HS_HANDED_OUT state, none of it's fields are
+ * valid yet (or are in the process of being set). Therefore we don't
+ * want to display any other columns.
+ */
+ if (start_state == PGAIO_HS_HANDED_OUT)
+ {
+ memset(nulls + 4, 1, (lengthof(nulls) - 4) * sizeof(bool));
+ goto display;
+ }
+
+ /* column: IO's operation */
+ values[4] = CStringGetTextDatum(pgaio_io_get_op_name(&ioh_copy));
+
+ /* columns: details about the IO's operation */
+ switch (ioh_copy.op)
+ {
+ case PGAIO_OP_INVALID:
+ nulls[5] = true;
+ nulls[6] = true;
+ break;
+ case PGAIO_OP_READV:
+ values[5] = Int64GetDatum(ioh_copy.op_data.read.offset);
+ values[6] =
+ Int64GetDatum(iov_byte_length(iov_copy, ioh_copy.op_data.read.iov_length));
+ break;
+ case PGAIO_OP_WRITEV:
+ values[5] = Int64GetDatum(ioh_copy.op_data.write.offset);
+ values[6] =
+ Int64GetDatum(iov_byte_length(iov_copy, ioh_copy.op_data.write.iov_length));
+ break;
+ }
+
+ /* column: IO's target */
+ values[7] = CStringGetTextDatum(pgaio_io_get_target_name(&ioh_copy));
+
+ /* column: length of IO's data array */
+ values[8] = Int16GetDatum(ioh_copy.handle_data_len);
+
+ /* column: raw result (i.e. some form of syscall return value) */
+ if (start_state == PGAIO_HS_COMPLETED_IO
+ || start_state == PGAIO_HS_COMPLETED_SHARED)
+ values[9] = Int32GetDatum(ioh_copy.result);
+ else
+ nulls[9] = true;
+
+ /*
+ * column: result in the higher level representation (unknown if not
+ * finished)
+ */
+ values[10] =
+ CStringGetTextDatum(pgaio_result_status_string(ioh_copy.distilled_result.status));
+
+ /* column: target description */
+ values[11] = CStringGetTextDatum(pgaio_io_get_target_description(&ioh_copy));
+
+ /* columns: one for each flag */
+ values[12] = BoolGetDatum(ioh_copy.flags & PGAIO_HF_SYNCHRONOUS);
+ values[13] = BoolGetDatum(ioh_copy.flags & PGAIO_HF_REFERENCES_LOCAL);
+ values[14] = BoolGetDatum(ioh_copy.flags & PGAIO_HF_BUFFERED);
+
+display:
+
+ tuplestore_putvalues(rsinfo->setResult, rsinfo->setDesc, values, nulls);
+ }
+
+ return (Datum) 0;
+}
diff --git a/src/backend/storage/aio/meson.build b/src/backend/storage/aio/meson.build
index 2f0f03d8071..da6df2d3654 100644
--- a/src/backend/storage/aio/meson.build
+++ b/src/backend/storage/aio/meson.build
@@ -3,6 +3,7 @@
backend_sources += files(
'aio.c',
'aio_callback.c',
+ 'aio_funcs.c',
'aio_init.c',
'aio_io.c',
'aio_target.c',
diff --git a/src/test/regress/expected/rules.out b/src/test/regress/expected/rules.out
index 62f69ac20b2..6570d0cddce 100644
--- a/src/test/regress/expected/rules.out
+++ b/src/test/regress/expected/rules.out
@@ -1286,6 +1286,22 @@ drop table cchild;
SELECT viewname, definition FROM pg_views
WHERE schemaname = 'pg_catalog'
ORDER BY viewname;
+pg_aios| SELECT pid,
+ io_id,
+ io_generation,
+ state,
+ operation,
+ "offset",
+ length,
+ target,
+ handle_data_len,
+ raw_result,
+ result,
+ target_desc,
+ f_sync,
+ f_localmem,
+ f_buffered
+ FROM pg_get_aios() pg_get_aios(pid, io_id, io_generation, state, operation, "offset", length, target, handle_data_len, raw_result, result, target_desc, f_sync, f_localmem, f_buffered);
pg_available_extension_versions| SELECT e.name,
e.version,
(x.extname IS NOT NULL) AS installed,
--
2.48.1.76.g4e746b1a31.dirty
[text/x-diff] v2.9-0011-Support-buffer-forwarding-in-read_stream.c.patch (8.7K, ../../oav52ytakcvfnk52w5o5wvxdjavxth2tgptxt25gnfmh2pdcpk@omp3xukdu5if/12-v2.9-0011-Support-buffer-forwarding-in-read_stream.c.patch)
download | inline diff:
From f14f6d89d9e9d90959f3e36b6ac850e1d79d23bd Mon Sep 17 00:00:00 2001
From: Thomas Munro <thomas.munro@gmail.com>
Date: Sat, 15 Mar 2025 12:29:57 -0400
Subject: [PATCH v2.9 11/30] Support buffer forwarding in read_stream.c.
In preparation for a following change to the buffer manager, teach
streams to keep track of buffers that were "forwarded" from one call to
StartReadBuffers() to the next.
Since StartReadBuffers() buffers argument will become an in/out
argument, we need to initialize the buffer queue entries with
InvalidBuffer. We don't want to do that up front, because we try to
keep stream initialization cheap for code that uses the fast path and
stays in one single buffer queue element. Satisfy both goals by
initializing the queue incrementally on the first cycle.
Discussion: https://postgr.es/m/CA%2BhUKGK_%3D4CVmMHvsHjOVrK6t4F%3DLBpFzsrr3R%2BaJYN8kcTfWg%40mail.gmail.com
---
src/backend/storage/aio/read_stream.c | 103 +++++++++++++++++++++++---
1 file changed, 93 insertions(+), 10 deletions(-)
diff --git a/src/backend/storage/aio/read_stream.c b/src/backend/storage/aio/read_stream.c
index d65fa07b44c..cdf4b5a86a2 100644
--- a/src/backend/storage/aio/read_stream.c
+++ b/src/backend/storage/aio/read_stream.c
@@ -95,8 +95,10 @@ struct ReadStream
int16 ios_in_progress;
int16 queue_size;
int16 max_pinned_buffers;
+ int16 forwarded_buffers;
int16 pinned_buffers;
int16 distance;
+ int16 initialized_buffers;
bool advice_enabled;
bool temporary;
@@ -224,8 +226,10 @@ static bool
read_stream_start_pending_read(ReadStream *stream)
{
bool need_wait;
+ int requested_nblocks;
int nblocks;
int flags;
+ int forwarded;
int16 io_index;
int16 overflow;
int16 buffer_index;
@@ -272,11 +276,20 @@ read_stream_start_pending_read(ReadStream *stream)
}
}
- /* How many more buffers is this backend allowed? */
+ /*
+ * How many more buffers is this backend allowed?
+ *
+ * If we already have some forwarded buffers, we can certainly use those.
+ * They are already pinned, and are mapped to the starting blocks of the
+ * pending read, they just don't have any I/O started yet and are not
+ * counted in stream->pinned_buffers.
+ */
if (stream->temporary)
buffer_limit = Min(GetAdditionalLocalPinLimit(), PG_INT16_MAX);
else
buffer_limit = Min(GetAdditionalPinLimit(), PG_INT16_MAX);
+ Assert(stream->forwarded_buffers <= stream->pending_read_nblocks);
+ buffer_limit += stream->forwarded_buffers;
if (buffer_limit == 0 && stream->pinned_buffers == 0)
buffer_limit = 1; /* guarantee progress */
@@ -303,8 +316,31 @@ read_stream_start_pending_read(ReadStream *stream)
* We say how many blocks we want to read, but it may be smaller on return
* if the buffer manager decides to shorten the read.
*/
+ requested_nblocks = Min(buffer_limit, stream->pending_read_nblocks);
+ nblocks = requested_nblocks;
buffer_index = stream->next_buffer_index;
io_index = stream->next_io_index;
+
+ /*
+ * The first time around the queue we initialize it as we go, including
+ * the overflow zone, because otherwise the entries would appear as
+ * forwarded buffers. This avoids initializing the whole queue up front
+ * in cases where it is large but we don't ever use it due to the
+ * all-cached fast path or small scans.
+ */
+ while (stream->initialized_buffers < buffer_index + nblocks)
+ stream->buffers[stream->initialized_buffers++] = InvalidBuffer;
+
+ /*
+ * Start the I/O. Any buffers that are not InvalidBuffer will be
+ * interpreted as already pinned, forwarded by an earlier call to
+ * StartReadBuffers(), and must map to the expected blocks. The nblocks
+ * value may be smaller on return indicating the size of the I/O that
+ * could be started. Buffers beyond the output nblocks number may also
+ * have been pinned without starting I/O due to various edge cases. In
+ * that case we'll just leave them in the queue ahead of us, "forwarded"
+ * to the next call, avoiding the need to unpin/repin.
+ */
need_wait = StartReadBuffers(&stream->ios[io_index].op,
&stream->buffers[buffer_index],
stream->pending_read_blocknum,
@@ -333,16 +369,35 @@ read_stream_start_pending_read(ReadStream *stream)
stream->seq_blocknum = stream->pending_read_blocknum + nblocks;
}
+ /*
+ * How many pins were acquired but forwarded to the next call? These need
+ * to be passed to the next StartReadBuffers() call, or released if the
+ * stream ends early. We need the number for accounting purposes, since
+ * they are not counted in stream->pinned_buffers but we already hold
+ * them.
+ */
+ forwarded = 0;
+ while (nblocks + forwarded < requested_nblocks &&
+ stream->buffers[buffer_index + nblocks + forwarded] != InvalidBuffer)
+ forwarded++;
+ stream->forwarded_buffers = forwarded;
+
/*
* We gave a contiguous range of buffer space to StartReadBuffers(), but
- * we want it to wrap around at queue_size. Slide overflowing buffers to
- * the front of the array.
+ * we want it to wrap around at queue_size. Copy overflowing buffers to
+ * the front of the array where they'll be consumed, but also leave a copy
+ * in the overflow zone which the I/O operation has a pointer to (it needs
+ * a contiguous array). Both copies will be cleared when the buffers are
+ * handed to the consumer.
*/
- overflow = (buffer_index + nblocks) - stream->queue_size;
+ overflow = (buffer_index + nblocks + forwarded) - stream->queue_size;
if (overflow > 0)
- memmove(&stream->buffers[0],
- &stream->buffers[stream->queue_size],
- sizeof(stream->buffers[0]) * overflow);
+ {
+ Assert(overflow < stream->queue_size); /* can't overlap */
+ memcpy(&stream->buffers[0],
+ &stream->buffers[stream->queue_size],
+ sizeof(stream->buffers[0]) * overflow);
+ }
/* Compute location of start of next read, without using % operator. */
buffer_index += nblocks;
@@ -718,10 +773,12 @@ read_stream_next_buffer(ReadStream *stream, void **per_buffer_data)
/* Fast path assumptions. */
Assert(stream->ios_in_progress == 0);
+ Assert(stream->forwarded_buffers == 0);
Assert(stream->pinned_buffers == 1);
Assert(stream->distance == 1);
Assert(stream->pending_read_nblocks == 0);
Assert(stream->per_buffer_data_size == 0);
+ Assert(stream->initialized_buffers > stream->oldest_buffer_index);
/* We're going to return the buffer we pinned last time. */
oldest_buffer_index = stream->oldest_buffer_index;
@@ -770,6 +827,7 @@ read_stream_next_buffer(ReadStream *stream, void **per_buffer_data)
stream->distance = 0;
stream->oldest_buffer_index = stream->next_buffer_index;
stream->pinned_buffers = 0;
+ stream->buffers[oldest_buffer_index] = InvalidBuffer;
}
stream->fast_path = false;
@@ -845,10 +903,15 @@ read_stream_next_buffer(ReadStream *stream, void **per_buffer_data)
stream->seq_until_processed = InvalidBlockNumber;
}
-#ifdef CLOBBER_FREED_MEMORY
- /* Clobber old buffer for debugging purposes. */
+ /*
+ * We must zap this queue entry, or else it would appear as a forwarded
+ * buffer. If it's potentially in the overflow zone (ie it wrapped around
+ * the queue), also zap that copy.
+ */
stream->buffers[oldest_buffer_index] = InvalidBuffer;
-#endif
+ if (oldest_buffer_index < io_combine_limit - 1)
+ stream->buffers[stream->queue_size + oldest_buffer_index] =
+ InvalidBuffer;
#if defined(CLOBBER_FREED_MEMORY) || defined(USE_VALGRIND)
@@ -893,6 +956,7 @@ read_stream_next_buffer(ReadStream *stream, void **per_buffer_data)
#ifndef READ_STREAM_DISABLE_FAST_PATH
/* See if we can take the fast path for all-cached scans next time. */
if (stream->ios_in_progress == 0 &&
+ stream->forwarded_buffers == 0 &&
stream->pinned_buffers == 1 &&
stream->distance == 1 &&
stream->pending_read_nblocks == 0 &&
@@ -928,6 +992,7 @@ read_stream_next_block(ReadStream *stream, BufferAccessStrategy *strategy)
void
read_stream_reset(ReadStream *stream)
{
+ int16 index;
Buffer buffer;
/* Stop looking ahead. */
@@ -941,6 +1006,24 @@ read_stream_reset(ReadStream *stream)
while ((buffer = read_stream_next_buffer(stream, NULL)) != InvalidBuffer)
ReleaseBuffer(buffer);
+ /* Unpin any unused forwarded buffers. */
+ index = stream->next_buffer_index;
+ while (index < stream->initialized_buffers &&
+ (buffer = stream->buffers[index]) != InvalidBuffer)
+ {
+ Assert(stream->forwarded_buffers > 0);
+ stream->forwarded_buffers--;
+ ReleaseBuffer(buffer);
+
+ stream->buffers[index] = InvalidBuffer;
+ if (index < io_combine_limit - 1)
+ stream->buffers[stream->queue_size + index] = InvalidBuffer;
+
+ if (++index == stream->queue_size)
+ index = 0;
+ }
+
+ Assert(stream->forwarded_buffers == 0);
Assert(stream->pinned_buffers == 0);
Assert(stream->ios_in_progress == 0);
--
2.48.1.76.g4e746b1a31.dirty
[text/x-diff] v2.9-0012-Support-buffer-forwarding-in-StartReadBuffers.patch (10.5K, ../../oav52ytakcvfnk52w5o5wvxdjavxth2tgptxt25gnfmh2pdcpk@omp3xukdu5if/13-v2.9-0012-Support-buffer-forwarding-in-StartReadBuffers.patch)
download | inline diff:
From e811294f9390f0178f02d9627854fe82b1857ff8 Mon Sep 17 00:00:00 2001
From: Thomas Munro <thomas.munro@gmail.com>
Date: Sat, 15 Mar 2025 12:29:57 -0400
Subject: [PATCH v2.9 12/30] Support buffer forwarding in StartReadBuffers().
Sometimes we have to perform a short read because we hit a cached block
that ends a contiguous run of blocks requiring I/O. We don't want
StartReadBuffers() to have to start more than one I/O, so we stop there.
We also don't want to have to unpin the cached block (and repin it
later), so previously we'd silently pretend the hit was part of the I/O,
and just leave it out of the read from disk. Now, we'll "forward" it to
the next call. We still write it to the buffers[] array for the caller
to pass back to us later, but it's not included in *nblocks.
This policy means that we no longer mix hits and misses in a single
operation's results, so we avoid the requirement to call
WaitReadBuffers(), which might stall, before the caller can make use of
the hits. The caller will get the hit in the next call instead, and
know that it doesn't have to wait. That's important for later work on
out-of-order read streams that minimize I/O stalls.
This also makes life easier for proposed work on true AIO, which
occasionally needs to split a large I/O after pinning all the buffers,
while the current coding only ever forwards a single bookending hit.
This API is natural for read_stream.c: it just leaves forwarded buffers
where they are in its circular queue, where the next call will pick them
up and continue, minimizing pin churn.
If we ever think of a good reason to disable this feature, i.e. for
other users of StartReadBuffers() that don't want to deal with forwarded
buffers, then we could add a flag for that. For now read_steam.c is the
only user.
Discussion: https://postgr.es/m/CA%2BhUKGK_%3D4CVmMHvsHjOVrK6t4F%3DLBpFzsrr3R%2BaJYN8kcTfWg%40mail.gmail.com
---
src/include/storage/bufmgr.h | 1 -
src/backend/storage/buffer/bufmgr.c | 128 ++++++++++++++++++++--------
2 files changed, 91 insertions(+), 38 deletions(-)
diff --git a/src/include/storage/bufmgr.h b/src/include/storage/bufmgr.h
index 79a89f87fcc..690cdde793a 100644
--- a/src/include/storage/bufmgr.h
+++ b/src/include/storage/bufmgr.h
@@ -130,7 +130,6 @@ struct ReadBuffersOperation
BlockNumber blocknum;
int flags;
int16 nblocks;
- int16 io_buffers_len;
};
typedef struct ReadBuffersOperation ReadBuffersOperation;
diff --git a/src/backend/storage/buffer/bufmgr.c b/src/backend/storage/buffer/bufmgr.c
index 79ca9d18d07..95911b51d58 100644
--- a/src/backend/storage/buffer/bufmgr.c
+++ b/src/backend/storage/buffer/bufmgr.c
@@ -1249,10 +1249,10 @@ StartReadBuffersImpl(ReadBuffersOperation *operation,
Buffer *buffers,
BlockNumber blockNum,
int *nblocks,
- int flags)
+ int flags,
+ bool allow_forwarding)
{
int actual_nblocks = *nblocks;
- int io_buffers_len = 0;
int maxcombine = 0;
Assert(*nblocks > 0);
@@ -1262,30 +1262,80 @@ StartReadBuffersImpl(ReadBuffersOperation *operation,
{
bool found;
- buffers[i] = PinBufferForBlock(operation->rel,
- operation->smgr,
- operation->persistence,
- operation->forknum,
- blockNum + i,
- operation->strategy,
- &found);
+ if (allow_forwarding && buffers[i] != InvalidBuffer)
+ {
+ BufferDesc *bufHdr;
+
+ /*
+ * This is a buffer that was pinned by an earlier call to
+ * StartReadBuffers(), but couldn't be handled in one operation at
+ * that time. The operation was split, and the caller has passed
+ * an already pinned buffer back to us to handle the rest of the
+ * operation. It must continue at the expected block number.
+ */
+ Assert(BufferGetBlockNumber(buffers[i]) == blockNum + i);
+
+ /*
+ * It might be an already valid buffer (a hit) that followed the
+ * final contiguous block of an earlier I/O (a miss) marking the
+ * end of it, or a buffer that some other backend has since made
+ * valid by performing the I/O for us, in which case we can handle
+ * it as a hit now. It is safe to check for a BM_VALID flag with
+ * a relaxed load, because we got a fresh view of it while pinning
+ * it in the previous call.
+ *
+ * On the other hand if we don't see BM_VALID yet, it must be an
+ * I/O that was split by the previous call and we need to try to
+ * start a new I/O from this block. We're also racing against any
+ * other backend that might start the I/O or even manage to mark
+ * it BM_VALID after this check, but StartBufferIO() will handle
+ * those cases.
+ */
+ if (BufferIsLocal(buffers[i]))
+ bufHdr = GetLocalBufferDescriptor(-buffers[i] - 1);
+ else
+ bufHdr = GetBufferDescriptor(buffers[i] - 1);
+ found = pg_atomic_read_u32(&bufHdr->state) & BM_VALID;
+ }
+ else
+ {
+ buffers[i] = PinBufferForBlock(operation->rel,
+ operation->smgr,
+ operation->persistence,
+ operation->forknum,
+ blockNum + i,
+ operation->strategy,
+ &found);
+ }
if (found)
{
/*
- * Terminate the read as soon as we get a hit. It could be a
- * single buffer hit, or it could be a hit that follows a readable
- * range. We don't want to create more than one readable range,
- * so we stop here.
+ * We have a hit. If it's the first block in the requested range,
+ * we can return it immediately and report that WaitReadBuffers()
+ * does not need to be called. If the initial value of *nblocks
+ * was larger, the caller will have to call again for the rest.
*/
- actual_nblocks = i + 1;
+ if (i == 0)
+ {
+ *nblocks = 1;
+ return false;
+ }
+
+ /*
+ * Otherwise we already have an I/O to perform, but this block
+ * can't be included as it is already valid. Split the I/O here.
+ * There may or may not be more blocks requiring I/O after this
+ * one, we haven't checked, but it can't be contiguous with this
+ * hit in the way. We'll leave this buffer pinned, forwarding it
+ * to the next call, avoiding the need to unpin it here and re-pin
+ * it in the next call.
+ */
+ actual_nblocks = i;
break;
}
else
{
- /* Extend the readable range to cover this block. */
- io_buffers_len++;
-
/*
* Check how many blocks we can cover with the same IO. The smgr
* implementation might e.g. be limited due to a segment boundary.
@@ -1306,15 +1356,11 @@ StartReadBuffersImpl(ReadBuffersOperation *operation,
}
*nblocks = actual_nblocks;
- if (likely(io_buffers_len == 0))
- return false;
-
/* Populate information needed for I/O. */
operation->buffers = buffers;
operation->blocknum = blockNum;
operation->flags = flags;
operation->nblocks = actual_nblocks;
- operation->io_buffers_len = io_buffers_len;
if (flags & READ_BUFFERS_ISSUE_ADVICE)
{
@@ -1329,7 +1375,7 @@ StartReadBuffersImpl(ReadBuffersOperation *operation,
smgrprefetch(operation->smgr,
operation->forknum,
blockNum,
- operation->io_buffers_len);
+ actual_nblocks);
}
/* Indicate that WaitReadBuffers() should be called. */
@@ -1343,11 +1389,21 @@ StartReadBuffersImpl(ReadBuffersOperation *operation,
* actual number, which may be fewer than requested. Caller sets some of the
* members of operation; see struct definition.
*
+ * The initial contents of the elements of buffers up to *nblocks should
+ * either be InvalidBuffer or an already-pinned buffer that was left by an
+ * preceding call to StartReadBuffers() that had to be split. On return, some
+ * elements of buffers may hold pinned buffers beyond the number indicated by
+ * the updated value of *nblocks. Operations are split on boundaries known to
+ * smgr (eg md.c segment boundaries that require crossing into a different
+ * underlying file), or when already cached blocks are found in the buffer
+ * that prevent the formation of a contiguous read.
+ *
* If false is returned, no I/O is necessary. If true is returned, one I/O
* has been started, and WaitReadBuffers() must be called with the same
* operation object before the buffers are accessed. Along with the operation
* object, the caller-supplied array of buffers must remain valid until
- * WaitReadBuffers() is called.
+ * WaitReadBuffers() is called, and any forwarded buffers must also be
+ * preserved for a future call unless explicitly released.
*
* Currently the I/O is only started with optional operating system advice if
* requested by the caller with READ_BUFFERS_ISSUE_ADVICE, and the real I/O
@@ -1361,13 +1417,18 @@ StartReadBuffers(ReadBuffersOperation *operation,
int *nblocks,
int flags)
{
- return StartReadBuffersImpl(operation, buffers, blockNum, nblocks, flags);
+ return StartReadBuffersImpl(operation, buffers, blockNum, nblocks, flags,
+ true /* expect forwarded buffers */ );
}
/*
* Single block version of the StartReadBuffers(). This might save a few
* instructions when called from another translation unit, because it is
* specialized for nblocks == 1.
+ *
+ * This version does not support "forwarded" buffers: they cannot be created
+ * by reading only one block, and the current contents of *buffer is ignored
+ * on entry.
*/
bool
StartReadBuffer(ReadBuffersOperation *operation,
@@ -1378,7 +1439,8 @@ StartReadBuffer(ReadBuffersOperation *operation,
int nblocks = 1;
bool result;
- result = StartReadBuffersImpl(operation, buffer, blocknum, &nblocks, flags);
+ result = StartReadBuffersImpl(operation, buffer, blocknum, &nblocks, flags,
+ false /* single block, no forwarding */ );
Assert(nblocks == 1); /* single block can't be short */
return result;
@@ -1404,24 +1466,16 @@ WaitReadBuffers(ReadBuffersOperation *operation)
IOObject io_object;
char persistence;
- /*
- * Currently operations are only allowed to include a read of some range,
- * with an optional extra buffer that is already pinned at the end. So
- * nblocks can be at most one more than io_buffers_len.
- */
- Assert((operation->nblocks == operation->io_buffers_len) ||
- (operation->nblocks == operation->io_buffers_len + 1));
-
/* Find the range of the physical read we need to perform. */
- nblocks = operation->io_buffers_len;
- if (nblocks == 0)
- return; /* nothing to do */
-
+ nblocks = operation->nblocks;
buffers = &operation->buffers[0];
blocknum = operation->blocknum;
forknum = operation->forknum;
persistence = operation->persistence;
+ Assert(nblocks > 0);
+ Assert(nblocks <= MAX_IO_COMBINE_LIMIT);
+
if (persistence == RELPERSISTENCE_TEMP)
{
io_context = IOCONTEXT_NORMAL;
--
2.48.1.76.g4e746b1a31.dirty
[text/x-diff] v2.9-0013-localbuf-Track-pincount-in-BufferDesc-as-well.patch (4.5K, ../../oav52ytakcvfnk52w5o5wvxdjavxth2tgptxt25gnfmh2pdcpk@omp3xukdu5if/14-v2.9-0013-localbuf-Track-pincount-in-BufferDesc-as-well.patch)
download | inline diff:
From ff815ec0ea2ef1b3b5e30dc44e6f952353ec4b91 Mon Sep 17 00:00:00 2001
From: Andres Freund <andres@anarazel.de>
Date: Sat, 15 Mar 2025 12:29:57 -0400
Subject: [PATCH v2.9 13/30] localbuf: Track pincount in BufferDesc as well
For AIO on temporary table buffers the AIO subsystem needs to be able to
ensure a pin on a buffer while AIO is going on, even if the IO issuing query
errors out. Tracking the buffer in LocalRefCount does not work, as it would
cause CheckForLocalBufferLeaks() to assert out.
Instead, also track the refcount in BufferDesc.state, not just
LocalRefCount. This also makes local buffers behave a bit more akin to shared
buffers.
Note that we still don't need locking, AIO completion callbacks for local
buffers are executed in the issuing session (i.e. nobody else has access to
the BufferDesc).
Discussion: https://postgr.es/m/uvrtrknj4kdytuboidbhwclo4gxhswwcpgadptsjvjqcluzmah%40brqs62irg4dt
---
src/backend/storage/buffer/bufmgr.c | 22 +++++++++++++++++++++
src/backend/storage/buffer/localbuf.c | 28 +++++++++++++++++++++++++--
2 files changed, 48 insertions(+), 2 deletions(-)
diff --git a/src/backend/storage/buffer/bufmgr.c b/src/backend/storage/buffer/bufmgr.c
index 95911b51d58..a9643930974 100644
--- a/src/backend/storage/buffer/bufmgr.c
+++ b/src/backend/storage/buffer/bufmgr.c
@@ -5393,6 +5393,18 @@ ConditionalLockBufferForCleanup(Buffer buffer)
Assert(refcount > 0);
if (refcount != 1)
return false;
+
+ /*
+ * Check that the AIO subsystem doesn't have a pin. Likely not
+ * possible today, but better safe than sorry.
+ */
+ bufHdr = GetLocalBufferDescriptor(-buffer - 1);
+ buf_state = pg_atomic_read_u32(&bufHdr->state);
+ refcount = BUF_STATE_GET_REFCOUNT(buf_state);
+ Assert(refcount > 0);
+ if (refcount != 1)
+ return false;
+
/* Nobody else to wait for */
return true;
}
@@ -5446,6 +5458,16 @@ IsBufferCleanupOK(Buffer buffer)
/* There should be exactly one pin */
if (LocalRefCount[-buffer - 1] != 1)
return false;
+
+ /*
+ * Check that the AIO subsystem doesn't have a pin. Likely not
+ * possible today, but better safe than sorry.
+ */
+ bufHdr = GetLocalBufferDescriptor(-buffer - 1);
+ buf_state = pg_atomic_read_u32(&bufHdr->state);
+ if (BUF_STATE_GET_REFCOUNT(buf_state) != 1)
+ return false;
+
/* Nobody else to wait for */
return true;
}
diff --git a/src/backend/storage/buffer/localbuf.c b/src/backend/storage/buffer/localbuf.c
index f172a5c7820..3a722321533 100644
--- a/src/backend/storage/buffer/localbuf.c
+++ b/src/backend/storage/buffer/localbuf.c
@@ -249,6 +249,13 @@ GetLocalVictimBuffer(void)
pg_atomic_unlocked_write_u32(&bufHdr->state, buf_state);
trycounter = NLocBuffer;
}
+ else if (BUF_STATE_GET_REFCOUNT(buf_state) > 0)
+ {
+ /*
+ * This can be reached if the backend initiated AIO for this
+ * buffer and then errored out.
+ */
+ }
else
{
/* Found a usable buffer */
@@ -570,7 +577,13 @@ InvalidateLocalBuffer(BufferDesc *bufHdr, bool check_unreferenced)
buf_state = pg_atomic_read_u32(&bufHdr->state);
- if (check_unreferenced && LocalRefCount[bufid] != 0)
+ /*
+ * We need to test not just LocalRefCount[bufid] but also the BufferDesc
+ * itself, as the latter is used to represent a pin by the AIO subsystem.
+ * This can happen if AIO is initiated and then the query errors out.
+ */
+ if (check_unreferenced &&
+ (LocalRefCount[bufid] != 0 || BUF_STATE_GET_REFCOUNT(buf_state) != 0))
elog(ERROR, "block %u of %s is still referenced (local %u)",
bufHdr->tag.blockNum,
relpathbackend(BufTagGetRelFileLocator(&bufHdr->tag),
@@ -744,12 +757,13 @@ PinLocalBuffer(BufferDesc *buf_hdr, bool adjust_usagecount)
if (LocalRefCount[bufid] == 0)
{
NLocalPinnedBuffers++;
+ buf_state += BUF_REFCOUNT_ONE;
if (adjust_usagecount &&
BUF_STATE_GET_USAGECOUNT(buf_state) < BM_MAX_USAGE_COUNT)
{
buf_state += BUF_USAGECOUNT_ONE;
- pg_atomic_unlocked_write_u32(&buf_hdr->state, buf_state);
}
+ pg_atomic_unlocked_write_u32(&buf_hdr->state, buf_state);
}
LocalRefCount[bufid]++;
ResourceOwnerRememberBuffer(CurrentResourceOwner,
@@ -775,7 +789,17 @@ UnpinLocalBufferNoOwner(Buffer buffer)
Assert(NLocalPinnedBuffers > 0);
if (--LocalRefCount[buffid] == 0)
+ {
+ BufferDesc *buf_hdr = GetLocalBufferDescriptor(buffid);
+ uint32 buf_state;
+
NLocalPinnedBuffers--;
+
+ buf_state = pg_atomic_read_u32(&buf_hdr->state);
+ Assert(BUF_STATE_GET_REFCOUNT(buf_state) > 0);
+ buf_state -= BUF_REFCOUNT_ONE;
+ pg_atomic_unlocked_write_u32(&buf_hdr->state, buf_state);
+ }
}
/*
--
2.48.1.76.g4e746b1a31.dirty
[text/x-diff] v2.9-0014-bufmgr-Implement-AIO-read-support.patch (25.4K, ../../oav52ytakcvfnk52w5o5wvxdjavxth2tgptxt25gnfmh2pdcpk@omp3xukdu5if/15-v2.9-0014-bufmgr-Implement-AIO-read-support.patch)
download | inline diff:
From 56160476c2e0bd8da81582f25a073e1db120fdb1 Mon Sep 17 00:00:00 2001
From: Andres Freund <andres@anarazel.de>
Date: Sat, 15 Mar 2025 12:29:57 -0400
Subject: [PATCH v2.9 14/30] bufmgr: Implement AIO read support
This implements the following:
- Addition of callbacks to maintain buffer state on completion of a readv
- Addition of a wait reference to BufferDesc, to allow backends to wait for
IOs
- StartBufferIO(), WaitIO(), TerminateBufferIO() support for waiting AIO
As of this commit nothing issues AIO on shared buffers. A future commit will
update StartReadBuffers() to do so.
Todo:
- deduplicate shared/local buffer completion callbacks
- deduplicate LockBufferForCleanup() support
- function naming pattern (buffer_readv_complete_common() calls
LocalBufferCompleteRead/SharedBufferCompleteRead, which looks ugly)
Discussion: https://postgr.es/m/uvrtrknj4kdytuboidbhwclo4gxhswwcpgadptsjvjqcluzmah%40brqs62irg4dt
Discussion: https://postgr.es/m/20210223100344.llw5an2aklengrmn@alap3.anarazel.de
Discussion: https://postgr.es/m/stj36ea6yyhoxtqkhpieia2z4krnam7qyetc57rfezgk4zgapf@gcnactj4z56m
---
src/include/storage/aio.h | 4 +
src/include/storage/buf_internals.h | 7 +-
src/include/storage/bufmgr.h | 4 +
src/backend/storage/aio/aio_callback.c | 5 +
src/backend/storage/buffer/buf_init.c | 3 +
src/backend/storage/buffer/bufmgr.c | 440 +++++++++++++++++++++++--
src/backend/storage/buffer/localbuf.c | 45 ++-
7 files changed, 466 insertions(+), 42 deletions(-)
diff --git a/src/include/storage/aio.h b/src/include/storage/aio.h
index 4fe7a6a8b00..c0db8f9754f 100644
--- a/src/include/storage/aio.h
+++ b/src/include/storage/aio.h
@@ -190,6 +190,10 @@ typedef enum PgAioHandleCallbackID
PGAIO_HCB_MD_READV,
PGAIO_HCB_MD_WRITEV,
+
+ PGAIO_HCB_SHARED_BUFFER_READV,
+
+ PGAIO_HCB_LOCAL_BUFFER_READV,
} PgAioHandleCallbackID;
diff --git a/src/include/storage/buf_internals.h b/src/include/storage/buf_internals.h
index 9327f60c44c..8a52ace37cd 100644
--- a/src/include/storage/buf_internals.h
+++ b/src/include/storage/buf_internals.h
@@ -17,6 +17,7 @@
#include "pgstat.h"
#include "port/atomics.h"
+#include "storage/aio_types.h"
#include "storage/buf.h"
#include "storage/bufmgr.h"
#include "storage/condition_variable.h"
@@ -264,6 +265,8 @@ typedef struct BufferDesc
int wait_backend_pgprocno; /* backend of pin-count waiter */
int freeNext; /* link in freelist chain */
+
+ PgAioWaitRef io_wref; /* set iff AIO is in progress */
LWLock content_lock; /* to lock access to buffer contents */
} BufferDesc;
@@ -472,8 +475,8 @@ extern BlockNumber ExtendBufferedRelLocal(BufferManagerRelation bmr,
uint32 *extended_by);
extern void MarkLocalBufferDirty(Buffer buffer);
extern void TerminateLocalBufferIO(BufferDesc *bufHdr, bool clear_dirty,
- uint32 set_flag_bits);
-extern bool StartLocalBufferIO(BufferDesc *bufHdr, bool forInput);
+ uint32 set_flag_bits, bool syncio);
+extern bool StartLocalBufferIO(BufferDesc *bufHdr, bool forInput, bool nowait);
extern void FlushLocalBuffer(BufferDesc *bufHdr, SMgrRelation reln);
extern void DropRelationLocalBuffers(RelFileLocator rlocator,
ForkNumber forkNum,
diff --git a/src/include/storage/bufmgr.h b/src/include/storage/bufmgr.h
index 690cdde793a..eafc94d7ed1 100644
--- a/src/include/storage/bufmgr.h
+++ b/src/include/storage/bufmgr.h
@@ -15,6 +15,7 @@
#define BUFMGR_H
#include "port/pg_iovec.h"
+#include "storage/aio_types.h"
#include "storage/block.h"
#include "storage/buf.h"
#include "storage/bufpage.h"
@@ -168,6 +169,9 @@ extern PGDLLIMPORT int checkpoint_flush_after;
extern PGDLLIMPORT int backend_flush_after;
extern PGDLLIMPORT int bgwriter_flush_after;
+extern const PgAioHandleCallbacks aio_shared_buffer_readv_cb;
+extern const PgAioHandleCallbacks aio_local_buffer_readv_cb;
+
/* in buf_init.c */
extern PGDLLIMPORT char *BufferBlocks;
diff --git a/src/backend/storage/aio/aio_callback.c b/src/backend/storage/aio/aio_callback.c
index 2d48aa7c1b3..2537b1914d8 100644
--- a/src/backend/storage/aio/aio_callback.c
+++ b/src/backend/storage/aio/aio_callback.c
@@ -18,6 +18,7 @@
#include "miscadmin.h"
#include "storage/aio.h"
#include "storage/aio_internal.h"
+#include "storage/bufmgr.h"
#include "storage/md.h"
@@ -40,6 +41,10 @@ static const PgAioHandleCallbacksEntry aio_handle_cbs[] = {
CALLBACK_ENTRY(PGAIO_HCB_INVALID, aio_invalid_cb),
CALLBACK_ENTRY(PGAIO_HCB_MD_READV, aio_md_readv_cb),
+
+ CALLBACK_ENTRY(PGAIO_HCB_SHARED_BUFFER_READV, aio_shared_buffer_readv_cb),
+
+ CALLBACK_ENTRY(PGAIO_HCB_LOCAL_BUFFER_READV, aio_local_buffer_readv_cb),
#undef CALLBACK_ENTRY
};
diff --git a/src/backend/storage/buffer/buf_init.c b/src/backend/storage/buffer/buf_init.c
index ed1f8e03190..ed1dc488a42 100644
--- a/src/backend/storage/buffer/buf_init.c
+++ b/src/backend/storage/buffer/buf_init.c
@@ -14,6 +14,7 @@
*/
#include "postgres.h"
+#include "storage/aio.h"
#include "storage/buf_internals.h"
#include "storage/bufmgr.h"
@@ -125,6 +126,8 @@ BufferManagerShmemInit(void)
buf->buf_id = i;
+ pgaio_wref_clear(&buf->io_wref);
+
/*
* Initially link all the buffers together as unused. Subsequent
* management of this list is done by freelist.c.
diff --git a/src/backend/storage/buffer/bufmgr.c b/src/backend/storage/buffer/bufmgr.c
index a9643930974..54add34f1a2 100644
--- a/src/backend/storage/buffer/bufmgr.c
+++ b/src/backend/storage/buffer/bufmgr.c
@@ -48,6 +48,7 @@
#include "pg_trace.h"
#include "pgstat.h"
#include "postmaster/bgwriter.h"
+#include "storage/aio.h"
#include "storage/buf_internals.h"
#include "storage/bufmgr.h"
#include "storage/fd.h"
@@ -516,7 +517,8 @@ static int SyncOneBuffer(int buf_id, bool skip_recently_used,
static void WaitIO(BufferDesc *buf);
static bool StartBufferIO(BufferDesc *buf, bool forInput, bool nowait);
static void TerminateBufferIO(BufferDesc *buf, bool clear_dirty,
- uint32 set_flag_bits, bool forget_owner);
+ uint32 set_flag_bits, bool forget_owner,
+ bool syncio);
static void AbortBufferIO(Buffer buffer);
static void shared_buffer_write_error_callback(void *arg);
static void local_buffer_write_error_callback(void *arg);
@@ -1038,7 +1040,7 @@ ZeroAndLockBuffer(Buffer buffer, ReadBufferMode mode, bool already_valid)
{
/* Simple case for non-shared buffers. */
bufHdr = GetLocalBufferDescriptor(-buffer - 1);
- need_to_zero = StartLocalBufferIO(bufHdr, true);
+ need_to_zero = StartLocalBufferIO(bufHdr, true, false);
}
else
{
@@ -1074,9 +1076,9 @@ ZeroAndLockBuffer(Buffer buffer, ReadBufferMode mode, bool already_valid)
/* Set BM_VALID, terminate IO, and wake up any waiters */
if (isLocalBuf)
- TerminateLocalBufferIO(bufHdr, false, BM_VALID);
+ TerminateLocalBufferIO(bufHdr, false, BM_VALID, true);
else
- TerminateBufferIO(bufHdr, false, BM_VALID, true);
+ TerminateBufferIO(bufHdr, false, BM_VALID, true, true);
}
else if (!isLocalBuf)
{
@@ -1450,7 +1452,8 @@ static inline bool
WaitReadBuffersCanStartIO(Buffer buffer, bool nowait)
{
if (BufferIsLocal(buffer))
- return StartLocalBufferIO(GetLocalBufferDescriptor(-buffer - 1), true);
+ return StartLocalBufferIO(GetLocalBufferDescriptor(-buffer - 1),
+ true, nowait);
else
return StartBufferIO(GetBufferDescriptor(buffer - 1), true, nowait);
}
@@ -1598,9 +1601,9 @@ WaitReadBuffers(ReadBuffersOperation *operation)
/* Set BM_VALID, terminate IO, and wake up any waiters */
if (persistence == RELPERSISTENCE_TEMP)
- TerminateLocalBufferIO(bufHdr, false, BM_VALID);
+ TerminateLocalBufferIO(bufHdr, false, BM_VALID, true);
else
- TerminateBufferIO(bufHdr, false, BM_VALID, true);
+ TerminateBufferIO(bufHdr, false, BM_VALID, true, true);
/* Report I/Os as completing individually. */
TRACE_POSTGRESQL_BUFFER_READ_DONE(forknum, io_first_block + j,
@@ -2503,7 +2506,7 @@ ExtendBufferedRelShared(BufferManagerRelation bmr,
if (lock)
LWLockAcquire(BufferDescriptorGetContentLock(buf_hdr), LW_EXCLUSIVE);
- TerminateBufferIO(buf_hdr, false, BM_VALID, true);
+ TerminateBufferIO(buf_hdr, false, BM_VALID, true, true);
}
pgBufferUsage.shared_blks_written += extend_by;
@@ -2849,6 +2852,44 @@ PinBuffer_Locked(BufferDesc *buf)
ResourceOwnerRememberBuffer(CurrentResourceOwner, b);
}
+/*
+ * Support for waking up another backend that is waiting for the cleanup lock
+ * to be released using BM_PIN_COUNT_WAITER.
+ *
+ * See LockBufferForCleanup().
+ *
+ * Expected to be called just after releasing a buffer pin (in a BufferDesc,
+ * not just reducing the backend-local pincount for the buffer).
+ */
+static void
+WakePinCountWaiter(BufferDesc *buf)
+{
+ /*
+ * Acquire the buffer header lock, re-check that there's a waiter. Another
+ * backend could have unpinned this buffer, and already woken up the
+ * waiter.
+ *
+ * There's no danger of the buffer being replaced after we unpinned it
+ * above, as it's pinned by the waiter. The waiter removes
+ * BM_PIN_COUNT_WAITER if it stops waiting for a reason other than this
+ * backend waking it up.
+ */
+ uint32 buf_state = LockBufHdr(buf);
+
+ if ((buf_state & BM_PIN_COUNT_WAITER) &&
+ BUF_STATE_GET_REFCOUNT(buf_state) == 1)
+ {
+ /* we just released the last pin other than the waiter's */
+ int wait_backend_pgprocno = buf->wait_backend_pgprocno;
+
+ buf_state &= ~BM_PIN_COUNT_WAITER;
+ UnlockBufHdr(buf, buf_state);
+ ProcSendSignal(wait_backend_pgprocno);
+ }
+ else
+ UnlockBufHdr(buf, buf_state);
+}
+
/*
* UnpinBuffer -- make buffer available for replacement.
*
@@ -2917,29 +2958,8 @@ UnpinBufferNoOwner(BufferDesc *buf)
/* Support LockBufferForCleanup() */
if (buf_state & BM_PIN_COUNT_WAITER)
- {
- /*
- * Acquire the buffer header lock, re-check that there's a waiter.
- * Another backend could have unpinned this buffer, and already
- * woken up the waiter. There's no danger of the buffer being
- * replaced after we unpinned it above, as it's pinned by the
- * waiter.
- */
- buf_state = LockBufHdr(buf);
+ WakePinCountWaiter(buf);
- if ((buf_state & BM_PIN_COUNT_WAITER) &&
- BUF_STATE_GET_REFCOUNT(buf_state) == 1)
- {
- /* we just released the last pin other than the waiter's */
- int wait_backend_pgprocno = buf->wait_backend_pgprocno;
-
- buf_state &= ~BM_PIN_COUNT_WAITER;
- UnlockBufHdr(buf, buf_state);
- ProcSendSignal(wait_backend_pgprocno);
- }
- else
- UnlockBufHdr(buf, buf_state);
- }
ForgetPrivateRefCountEntry(ref);
}
}
@@ -3959,7 +3979,7 @@ FlushBuffer(BufferDesc *buf, SMgrRelation reln, IOObject io_object,
* Mark the buffer as clean (unless BM_JUST_DIRTIED has become set) and
* end the BM_IO_IN_PROGRESS state.
*/
- TerminateBufferIO(buf, true, 0, true);
+ TerminateBufferIO(buf, true, 0, true, true);
TRACE_POSTGRESQL_BUFFER_FLUSH_DONE(BufTagGetForkNum(&buf->tag),
buf->tag.blockNum,
@@ -5518,6 +5538,7 @@ WaitIO(BufferDesc *buf)
for (;;)
{
uint32 buf_state;
+ PgAioWaitRef iow;
/*
* It may not be necessary to acquire the spinlock to check the flag
@@ -5525,10 +5546,19 @@ WaitIO(BufferDesc *buf)
* play it safe.
*/
buf_state = LockBufHdr(buf);
+ iow = buf->io_wref;
UnlockBufHdr(buf, buf_state);
if (!(buf_state & BM_IO_IN_PROGRESS))
break;
+
+ if (pgaio_wref_valid(&iow))
+ {
+ pgaio_wref_wait(&iow);
+ ConditionVariablePrepareToSleep(cv);
+ continue;
+ }
+
ConditionVariableSleep(cv, WAIT_EVENT_BUFFER_IO);
}
ConditionVariableCancelSleep();
@@ -5617,7 +5647,7 @@ StartBufferIO(BufferDesc *buf, bool forInput, bool nowait)
*/
static void
TerminateBufferIO(BufferDesc *buf, bool clear_dirty, uint32 set_flag_bits,
- bool forget_owner)
+ bool forget_owner, bool syncio)
{
uint32 buf_state;
@@ -5632,6 +5662,14 @@ TerminateBufferIO(BufferDesc *buf, bool clear_dirty, uint32 set_flag_bits,
if (clear_dirty && !(buf_state & BM_JUST_DIRTIED))
buf_state &= ~(BM_DIRTY | BM_CHECKPOINT_NEEDED);
+ if (!syncio)
+ {
+ /* release ownership by the AIO subsystem */
+ Assert(BUF_STATE_GET_REFCOUNT(buf_state) > 0);
+ buf_state -= BUF_REFCOUNT_ONE;
+ pgaio_wref_clear(&buf->io_wref);
+ }
+
buf_state |= set_flag_bits;
UnlockBufHdr(buf, buf_state);
@@ -5640,6 +5678,17 @@ TerminateBufferIO(BufferDesc *buf, bool clear_dirty, uint32 set_flag_bits,
BufferDescriptorGetBuffer(buf));
ConditionVariableBroadcast(BufferDescriptorGetIOCV(buf));
+
+ /*
+ * Support LockBufferForCleanup()
+ *
+ * If we just released a pin, need to do BM_PIN_COUNT_WAITER handling.
+ * Most of the time the current backend will hold another pin preventing
+ * that from happening, but that's e.g. not the case when completing an IO
+ * another backend started.
+ */
+ if (buf_state & BM_PIN_COUNT_WAITER)
+ WakePinCountWaiter(buf);
}
/*
@@ -5688,7 +5737,7 @@ AbortBufferIO(Buffer buffer)
}
}
- TerminateBufferIO(buf_hdr, false, BM_IO_ERROR, false);
+ TerminateBufferIO(buf_hdr, false, BM_IO_ERROR, false, true);
}
/*
@@ -6139,3 +6188,328 @@ EvictUnpinnedBuffer(Buffer buf)
return result;
}
+
+/*
+ * Helper for AIO staging callback for both reads and writes as well as temp
+ * and shared buffers.
+ */
+static pg_attribute_always_inline void
+buffer_stage_common(PgAioHandle *ioh, bool is_write, bool is_temp)
+{
+ uint64 *io_data;
+ uint8 handle_data_len;
+ PgAioWaitRef io_ref;
+ BufferTag first PG_USED_FOR_ASSERTS_ONLY = {0};
+
+ io_data = pgaio_io_get_handle_data(ioh, &handle_data_len);
+
+ pgaio_io_get_wref(ioh, &io_ref);
+
+ for (int i = 0; i < handle_data_len; i++)
+ {
+ Buffer buffer = (Buffer) io_data[i];
+ BufferDesc *buf_hdr = is_temp ?
+ GetLocalBufferDescriptor(-buffer - 1)
+ : GetBufferDescriptor(buffer - 1);
+ uint32 buf_state;
+
+ /*
+ * Check that all the buffers are actually ones that could conceivably
+ * be done in one IO, i.e. are sequential.
+ */
+ if (i == 0)
+ first = buf_hdr->tag;
+ else
+ {
+ Assert(buf_hdr->tag.relNumber == first.relNumber);
+ Assert(buf_hdr->tag.blockNum == first.blockNum + i);
+ }
+
+ if (is_temp)
+ buf_state = pg_atomic_read_u32(&buf_hdr->state);
+ else
+ buf_state = LockBufHdr(buf_hdr);
+
+ Assert(buf_state & BM_TAG_VALID);
+ if (is_write)
+ {
+ Assert(buf_state & BM_VALID);
+ Assert(buf_state & BM_DIRTY);
+ }
+ else
+ {
+ Assert(!(buf_state & BM_VALID));
+ Assert(!(buf_state & BM_DIRTY));
+ }
+
+ /* temp buffers don't use BM_IO_IN_PROGRESS */
+ if (!is_temp)
+ Assert(buf_state & BM_IO_IN_PROGRESS);
+
+ /* we better have ensured the buffer is present until now */
+ Assert(BUF_STATE_GET_REFCOUNT(buf_state) >= 1);
+
+ /*
+ * Reflect that the buffer is now owned by the subsystem.
+ *
+ * For local buffers: This can't be done just in LocalRefCount as one
+ * might initially think, as this backend could error out while AIO is
+ * still in progress, releasing all the pins by the backend itself.
+ */
+ buf_state += BUF_REFCOUNT_ONE;
+ buf_hdr->io_wref = io_ref;
+
+ if (is_temp)
+ pg_atomic_unlocked_write_u32(&buf_hdr->state, buf_state);
+ else
+ UnlockBufHdr(buf_hdr, buf_state);
+
+ if (is_write && !is_temp)
+ {
+ LWLock *content_lock;
+
+ content_lock = BufferDescriptorGetContentLock(buf_hdr);
+
+ Assert(LWLockHeldByMe(content_lock));
+
+ /*
+ * Lock is now owned by AIO subsystem.
+ */
+ LWLockDisown(content_lock);
+ }
+
+ /*
+ * Stop tracking this buffer via the resowner - the AIO system now
+ * keeps track.
+ */
+ if (!is_temp)
+ ResourceOwnerForgetBufferIO(CurrentResourceOwner, buffer);
+ }
+}
+
+/*
+ * Helper for AIO readv completion callbacks, supporting both shared and temp
+ * buffers. Gets called once for each buffer in a multi-page read.
+ */
+static pg_attribute_always_inline PgAioResult
+buffer_readv_complete_one(uint8 buf_off, Buffer buffer, uint8 flags,
+ bool failed, bool is_temp)
+{
+ BufferDesc *buf_hdr = is_temp ?
+ GetLocalBufferDescriptor(-buffer - 1)
+ : GetBufferDescriptor(buffer - 1);
+ BufferTag tag = buf_hdr->tag;
+ char *bufdata = BufferGetBlock(buffer);
+ PgAioResult result;
+ uint32 set_flag_bits;
+
+#ifdef USE_ASSERT_CHECKING
+ {
+ uint32 buf_state = pg_atomic_read_u32(&buf_hdr->state);
+
+ Assert(buf_state & BM_TAG_VALID);
+ Assert(!(buf_state & BM_VALID));
+ /* temp buffers don't use BM_IO_IN_PROGRESS */
+ if (!is_temp)
+ Assert(buf_state & BM_IO_IN_PROGRESS);
+ Assert(!(buf_state & BM_DIRTY));
+ }
+#endif
+
+ result.status = ARS_OK;
+
+ /* check for garbage data */
+ if (!failed &&
+ !PageIsVerifiedExtended((Page) bufdata, tag.blockNum,
+ PIV_LOG_WARNING | PIV_REPORT_STAT))
+ {
+ RelFileLocator rlocator = BufTagGetRelFileLocator(&tag);
+
+ if ((flags & READ_BUFFERS_ZERO_ON_ERROR) || zero_damaged_pages)
+ {
+ ereport(LOG,
+ (errcode(ERRCODE_DATA_CORRUPTED),
+ errmsg("invalid page in block %u of relation %s; zeroing out page",
+ tag.blockNum,
+ relpathbackend(rlocator,
+ is_temp ? MyProcNumber : INVALID_PROC_NUMBER,
+ tag.forkNum).str)));
+ memset(bufdata, 0, BLCKSZ);
+ }
+ else
+ {
+ /* mark buffer as having failed */
+ failed = true;
+
+ /* encode error for buffer_readv_report */
+ result.status = ARS_ERROR;
+ if (is_temp)
+ result.id = PGAIO_HCB_LOCAL_BUFFER_READV;
+ else
+ result.id = PGAIO_HCB_SHARED_BUFFER_READV;
+ result.error_data = buf_off;
+ }
+ }
+
+ /* Terminate I/O and set BM_VALID. */
+ set_flag_bits = failed ? BM_IO_ERROR : BM_VALID;
+ if (is_temp)
+ TerminateLocalBufferIO(buf_hdr, false, set_flag_bits, false);
+ else
+ TerminateBufferIO(buf_hdr, false, set_flag_bits, false, false);
+
+ /*
+ * Call the BUFFER_READ_DONE tracepoint in the callback, even though the
+ * callback may not be executed in the same backend that called
+ * BUFFER_READ_START. The alternative would be to defer calling the
+ * tracepoint to a later point (e.g. the local completion callback for
+ * shared buffer reads), which seems even less helpful and would also
+ * imply some overhead.
+ */
+ TRACE_POSTGRESQL_BUFFER_READ_DONE(tag.forkNum,
+ tag.blockNum,
+ tag.spcOid,
+ tag.dbOid,
+ tag.relNumber,
+ is_temp ? MyProcNumber : INVALID_PROC_NUMBER,
+ false);
+
+ return result;
+}
+
+/*
+ * Perform completion handling of a single AIO read. This read may cover
+ * multiple blocks / buffers.
+ *
+ * Shared between shared and local buffers, to reduce code duplication.
+ */
+static pg_attribute_always_inline PgAioResult
+buffer_readv_complete(PgAioHandle *ioh, PgAioResult prior_result,
+ uint8 cb_data, bool is_temp)
+{
+ PgAioResult result = prior_result;
+ PgAioTargetData *td = pgaio_io_get_target_data(ioh);
+ uint64 *io_data;
+ uint8 handle_data_len;
+
+ if (is_temp)
+ {
+ Assert(td->smgr.is_temp);
+ Assert(pgaio_io_get_owner(ioh) == MyProcNumber);
+ }
+ else
+ Assert(!td->smgr.is_temp);
+
+ /*
+ * Iterate over all the buffers affected by this IO and call appropriate
+ * per-buffer completion function for each buffer.
+ */
+ io_data = pgaio_io_get_handle_data(ioh, &handle_data_len);
+ for (uint8 buf_off = 0; buf_off < handle_data_len; buf_off++)
+ {
+ Buffer buf = io_data[buf_off];
+ PgAioResult buf_result;
+ bool failed;
+
+ Assert(BufferIsValid(buf));
+
+ /*
+ * If the entire failed on a lower-level, each buffer needs to be
+ * marked as failed. In case of a partial read, some buffers may be
+ * ok.
+ */
+ failed =
+ prior_result.status == ARS_ERROR
+ || prior_result.result <= buf_off;
+
+ buf_result = buffer_readv_complete_one(buf_off, buf, cb_data, failed,
+ is_temp);
+
+ /*
+ * If there wasn't any prior error and the IO for this page failed in
+ * some form, set the whole IO's to the page's result.
+ */
+ if (result.status != ARS_ERROR && buf_result.status != ARS_OK)
+ {
+ result = buf_result;
+ pgaio_result_report(result, td, LOG);
+ }
+ }
+
+ return result;
+}
+
+/*
+ * AIO error reporting callback for aio_shared_buffer_readv_cb and
+ * aio_local_buffer_readv_cb.
+ *
+ * Errors are encoded as follows:
+ * - the only error is page verification failing
+ * - result.error_data is the offset of the first page that failed
+ * verification in a larger IO
+ */
+static void
+buffer_readv_report(PgAioResult result, const PgAioTargetData *target_data, int elevel)
+{
+ ProcNumber errProc;
+
+ if (target_data->smgr.is_temp)
+ errProc = MyProcNumber;
+ else
+ errProc = INVALID_PROC_NUMBER;
+
+ ereport(elevel,
+ errcode(ERRCODE_DATA_CORRUPTED),
+ errmsg("invalid page in block %u of relation %s",
+ target_data->smgr.blockNum + result.error_data,
+ relpathbackend(target_data->smgr.rlocator, errProc, target_data->smgr.forkNum).str
+ )
+ );
+}
+
+static void
+shared_buffer_readv_stage(PgAioHandle *ioh, uint8 cb_data)
+{
+ buffer_stage_common(ioh, false, false);
+}
+
+static PgAioResult
+shared_buffer_readv_complete(PgAioHandle *ioh, PgAioResult prior_result,
+ uint8 cb_data)
+{
+ return buffer_readv_complete(ioh, prior_result, cb_data, false);
+}
+
+static void
+local_buffer_readv_stage(PgAioHandle *ioh, uint8 cb_data)
+{
+ buffer_stage_common(ioh, false, true);
+}
+
+static PgAioResult
+local_buffer_readv_complete(PgAioHandle *ioh, PgAioResult prior_result,
+ uint8 cb_data)
+{
+ return buffer_readv_complete(ioh, prior_result, cb_data, true);
+}
+
+/* readv callback is is passed READ_BUFFERS_* flags as callback data */
+const PgAioHandleCallbacks aio_shared_buffer_readv_cb = {
+ .stage = shared_buffer_readv_stage,
+ .complete_shared = shared_buffer_readv_complete,
+ .report = buffer_readv_report,
+};
+
+/* readv callback is is passed READ_BUFFERS_* flags as callback data */
+const PgAioHandleCallbacks aio_local_buffer_readv_cb = {
+ .stage = local_buffer_readv_stage,
+
+ /*
+ * Note that this, in contrast to the shared_buffers case, uses
+ * complete_local, as only the issuing backend has access to the required
+ * datastructures. This is important in case the IO completion may be
+ * consumed incidentally by another backend.
+ */
+ .complete_local = local_buffer_readv_complete,
+ .report = buffer_readv_report,
+};
diff --git a/src/backend/storage/buffer/localbuf.c b/src/backend/storage/buffer/localbuf.c
index 3a722321533..b18c4c25143 100644
--- a/src/backend/storage/buffer/localbuf.c
+++ b/src/backend/storage/buffer/localbuf.c
@@ -17,7 +17,9 @@
#include "access/parallel.h"
#include "executor/instrument.h"
+#include "pg_trace.h"
#include "pgstat.h"
+#include "storage/aio.h"
#include "storage/buf_internals.h"
#include "storage/bufmgr.h"
#include "storage/fd.h"
@@ -187,7 +189,7 @@ FlushLocalBuffer(BufferDesc *bufHdr, SMgrRelation reln)
* Try to start an I/O operation. There currently are no reasons for
* StartLocalBufferIO to return false, so we raise an error in that case.
*/
- if (!StartLocalBufferIO(bufHdr, false))
+ if (!StartLocalBufferIO(bufHdr, false, false))
elog(ERROR, "failed to start write IO on local buffer");
/* Find smgr relation for buffer */
@@ -211,7 +213,7 @@ FlushLocalBuffer(BufferDesc *bufHdr, SMgrRelation reln)
IOOP_WRITE, io_start, 1, BLCKSZ);
/* Mark not-dirty */
- TerminateLocalBufferIO(bufHdr, true, 0);
+ TerminateLocalBufferIO(bufHdr, true, 0, true);
pgBufferUsage.local_blks_written++;
}
@@ -430,7 +432,7 @@ ExtendBufferedRelLocal(BufferManagerRelation bmr,
pg_atomic_unlocked_write_u32(&existing_hdr->state, buf_state);
/* no need to loop for local buffers */
- StartLocalBufferIO(existing_hdr, true);
+ StartLocalBufferIO(existing_hdr, true, false);
}
else
{
@@ -446,7 +448,7 @@ ExtendBufferedRelLocal(BufferManagerRelation bmr,
hresult->id = victim_buf_id;
- StartLocalBufferIO(victim_buf_hdr, true);
+ StartLocalBufferIO(victim_buf_hdr, true, false);
}
}
@@ -515,10 +517,25 @@ MarkLocalBufferDirty(Buffer buffer)
* Like StartBufferIO, but for local buffers
*/
bool
-StartLocalBufferIO(BufferDesc *bufHdr, bool forInput)
+StartLocalBufferIO(BufferDesc *bufHdr, bool forInput, bool nowait)
{
- uint32 buf_state = pg_atomic_read_u32(&bufHdr->state);
+ uint32 buf_state;
+ /*
+ * The buffer could have IO in progress, e.g. when there are two scans of
+ * the same relation. Either wait for the other IO or return false.
+ */
+ if (pgaio_wref_valid(&bufHdr->io_wref))
+ {
+ PgAioWaitRef iow = bufHdr->io_wref;
+
+ if (nowait)
+ return false;
+
+ pgaio_wref_wait(&iow);
+ }
+
+ buf_state = pg_atomic_read_u32(&bufHdr->state);
if (forInput ? (buf_state & BM_VALID) : !(buf_state & BM_DIRTY))
{
/* someone else already did the I/O */
@@ -536,7 +553,8 @@ StartLocalBufferIO(BufferDesc *bufHdr, bool forInput)
* Like TerminateBufferIO, but for local buffers
*/
void
-TerminateLocalBufferIO(BufferDesc *bufHdr, bool clear_dirty, uint32 set_flag_bits)
+TerminateLocalBufferIO(BufferDesc *bufHdr, bool clear_dirty, uint32 set_flag_bits,
+ bool syncio)
{
/* Only need to adjust flags */
uint32 buf_state = pg_atomic_read_u32(&bufHdr->state);
@@ -549,6 +567,17 @@ TerminateLocalBufferIO(BufferDesc *bufHdr, bool clear_dirty, uint32 set_flag_bit
if (clear_dirty)
buf_state &= ~BM_DIRTY;
+ if (!syncio)
+ {
+ /*
+ * Release pin held by IO subsystem, see also
+ * local_buffer_readv_prepare().
+ */
+ Assert(BUF_STATE_GET_REFCOUNT(buf_state) > 0);
+ buf_state -= BUF_REFCOUNT_ONE;
+ pgaio_wref_clear(&bufHdr->io_wref);
+ }
+
buf_state |= set_flag_bits;
pg_atomic_unlocked_write_u32(&bufHdr->state, buf_state);
@@ -714,6 +743,8 @@ InitLocalBuffers(void)
*/
buf->buf_id = -i - 2;
+ pgaio_wref_clear(&buf->io_wref);
+
/*
* Intentionally do not initialize the buffer's atomic variable
* (besides zeroing the underlying memory above). That way we get
--
2.48.1.76.g4e746b1a31.dirty
[text/x-diff] v2.9-0015-bufmgr-Improve-stats-when-buffer-was-read-in-co.patch (3.0K, ../../oav52ytakcvfnk52w5o5wvxdjavxth2tgptxt25gnfmh2pdcpk@omp3xukdu5if/16-v2.9-0015-bufmgr-Improve-stats-when-buffer-was-read-in-co.patch)
download | inline diff:
From 4bab968b504fae9b1ebe7445d182a74619de4306 Mon Sep 17 00:00:00 2001
From: Andres Freund <andres@anarazel.de>
Date: Sat, 15 Mar 2025 12:29:57 -0400
Subject: [PATCH v2.9 15/30] bufmgr: Improve stats when buffer was read in
concurrently
Author:
Reviewed-by:
Discussion: https://postgr.es/m/
Backpatch:
---
src/backend/storage/buffer/bufmgr.c | 41 ++++++++++++++++++-----------
1 file changed, 26 insertions(+), 15 deletions(-)
diff --git a/src/backend/storage/buffer/bufmgr.c b/src/backend/storage/buffer/bufmgr.c
index 54add34f1a2..d783b33f78e 100644
--- a/src/backend/storage/buffer/bufmgr.c
+++ b/src/backend/storage/buffer/bufmgr.c
@@ -1490,19 +1490,6 @@ WaitReadBuffers(ReadBuffersOperation *operation)
io_object = IOOBJECT_RELATION;
}
- /*
- * We count all these blocks as read by this backend. This is traditional
- * behavior, but might turn out to be not true if we find that someone
- * else has beaten us and completed the read of some of these blocks. In
- * that case the system globally double-counts, but we traditionally don't
- * count this as a "hit", and we don't have a separate counter for "miss,
- * but another backend completed the read".
- */
- if (persistence == RELPERSISTENCE_TEMP)
- pgBufferUsage.local_blks_read += nblocks;
- else
- pgBufferUsage.shared_blks_read += nblocks;
-
for (int i = 0; i < nblocks; ++i)
{
int io_buffers_len;
@@ -1520,8 +1507,13 @@ WaitReadBuffers(ReadBuffersOperation *operation)
if (!WaitReadBuffersCanStartIO(buffers[i], false))
{
/*
- * Report this as a 'hit' for this backend, even though it must
- * have started out as a miss in PinBufferForBlock().
+ * Report and track this as a 'hit' for this backend, even though
+ * it must have started out as a miss in PinBufferForBlock().
+ *
+ * Some of the accesses would otherwise never be counted (e.g.
+ * pgBufferUsage) or counted as a miss (e.g.
+ * pgstat_count_buffer_hit(), as we always call
+ * pgstat_count_buffer_read()).
*/
TRACE_POSTGRESQL_BUFFER_READ_DONE(forknum, blocknum + i,
operation->smgr->smgr_rlocator.locator.spcOid,
@@ -1529,6 +1521,20 @@ WaitReadBuffers(ReadBuffersOperation *operation)
operation->smgr->smgr_rlocator.locator.relNumber,
operation->smgr->smgr_rlocator.backend,
true);
+
+ if (persistence == RELPERSISTENCE_TEMP)
+ pgBufferUsage.local_blks_hit += 1;
+ else
+ pgBufferUsage.shared_blks_hit += 1;
+
+ if (operation->rel)
+ pgstat_count_buffer_hit(operation->rel);
+
+ pgstat_count_io_op(io_object, io_context, IOOP_HIT, 1, 0);
+
+ if (VacuumCostActive)
+ VacuumCostBalance += VacuumCostPageHit;
+
continue;
}
@@ -1614,6 +1620,11 @@ WaitReadBuffers(ReadBuffersOperation *operation)
false);
}
+ if (persistence == RELPERSISTENCE_TEMP)
+ pgBufferUsage.local_blks_read += io_buffers_len;
+ else
+ pgBufferUsage.shared_blks_read += io_buffers_len;
+
if (VacuumCostActive)
VacuumCostBalance += VacuumCostPageMiss * io_buffers_len;
}
--
2.48.1.76.g4e746b1a31.dirty
[text/x-diff] v2.9-0016-bufmgr-Use-AIO-in-StartReadBuffers.patch (26.9K, ../../oav52ytakcvfnk52w5o5wvxdjavxth2tgptxt25gnfmh2pdcpk@omp3xukdu5if/17-v2.9-0016-bufmgr-Use-AIO-in-StartReadBuffers.patch)
download | inline diff:
From 82ce7620bc540caaf639e386f2c458039b21c4e8 Mon Sep 17 00:00:00 2001
From: Andres Freund <andres@anarazel.de>
Date: Sat, 15 Mar 2025 12:29:57 -0400
Subject: [PATCH v2.9 16/30] bufmgr: Use AIO in StartReadBuffers()
This finally introduces the first actual AIO user. StartReadBuffers() now uses
the AIO routines to issue IO. This converts a lot of callers to use the AIO
infrastructure.
As the implementation of StartReadBuffers() is also used by the functions for
reading individual blocks (StartReadBuffer() and through that
ReadBufferExtended()) this means all buffered read IO passes through the AIO
paths. However, as those are synchronous reads, actually performing the IO
asynchronously would be rarely beneficial. Instead such IOs are flagged to
always be executed synchronously. This way we don't have to duplicate a fair
bit of code.
When io_method=sync is used, the IO patterns generated after this change are
the same as before, i.e. actual reads are only issued in WaitReadBuffers() and
StartReadBuffers() may issue prefetch requests. This allows to bypass most of
the actual asynchronicity, which is important to make a change as big as this
less risky.
One thing worth calling out is that, if IO is actually executed
asynchronously, the precise meaning of what track_io_timing is measuring has
changed. Previously it tracked the time for each IO, but that does not make
sense when multiple IOs are executed concurrently. Now it only measures the
time actually spent waiting for IO.
While AIO is now actually used, the logic in read_stream.c will often prevent
using sufficiently many concurrent IOs. That will be addressed in the next
commits.
Co-authored-by: Andres Freund <andres@anarazel.de>
Co-authored-by: Thomas Munro <thomas.munro@gmail.com>
Discussion: https://postgr.es/m/uvrtrknj4kdytuboidbhwclo4gxhswwcpgadptsjvjqcluzmah%40brqs62irg4dt
Discussion: https://postgr.es/m/20210223100344.llw5an2aklengrmn@alap3.anarazel.de
Discussion: https://postgr.es/m/stj36ea6yyhoxtqkhpieia2z4krnam7qyetc57rfezgk4zgapf@gcnactj4z56m
---
src/include/storage/bufmgr.h | 6 +
src/backend/storage/buffer/bufmgr.c | 614 +++++++++++++++++++++-------
2 files changed, 477 insertions(+), 143 deletions(-)
diff --git a/src/include/storage/bufmgr.h b/src/include/storage/bufmgr.h
index eafc94d7ed1..ca4ca22c345 100644
--- a/src/include/storage/bufmgr.h
+++ b/src/include/storage/bufmgr.h
@@ -112,6 +112,9 @@ typedef struct BufferManagerRelation
#define READ_BUFFERS_ZERO_ON_ERROR (1 << 0)
/* Call smgrprefetch() if I/O necessary. */
#define READ_BUFFERS_ISSUE_ADVICE (1 << 1)
+/* IO will immediately be waited for */
+#define READ_BUFFERS_SYNCHRONOUSLY (1 << 2)
+
struct ReadBuffersOperation
{
@@ -131,6 +134,9 @@ struct ReadBuffersOperation
BlockNumber blocknum;
int flags;
int16 nblocks;
+ int16 nblocks_done;
+ PgAioWaitRef io_wref;
+ PgAioReturn io_return;
};
typedef struct ReadBuffersOperation ReadBuffersOperation;
diff --git a/src/backend/storage/buffer/bufmgr.c b/src/backend/storage/buffer/bufmgr.c
index d783b33f78e..2691757016c 100644
--- a/src/backend/storage/buffer/bufmgr.c
+++ b/src/backend/storage/buffer/bufmgr.c
@@ -528,6 +528,8 @@ static inline BufferDesc *BufferAlloc(SMgrRelation smgr,
BlockNumber blockNum,
BufferAccessStrategy strategy,
bool *foundPtr, IOContext io_context);
+static bool AsyncReadBuffers(ReadBuffersOperation *operation, int *nblocks);
+static void CheckReadBuffersOperation(ReadBuffersOperation *operation, bool is_complete);
static Buffer GetVictimBuffer(BufferAccessStrategy strategy, IOContext io_context);
static void FlushBuffer(BufferDesc *buf, SMgrRelation reln,
IOObject io_object, IOContext io_context);
@@ -1228,10 +1230,14 @@ ReadBuffer_common(Relation rel, SMgrRelation smgr, char smgr_persistence,
return buffer;
}
+ /*
+ * Signal that we are going to immediately wait. If we're immediately
+ * waiting, there is no benefit in actually executing the IO
+ * asynchronously, it would just add dispatch overhead.
+ */
+ flags = READ_BUFFERS_SYNCHRONOUSLY;
if (mode == RBM_ZERO_ON_ERROR)
- flags = READ_BUFFERS_ZERO_ON_ERROR;
- else
- flags = 0;
+ flags |= READ_BUFFERS_ZERO_ON_ERROR;
operation.smgr = smgr;
operation.rel = rel;
operation.persistence = persistence;
@@ -1256,9 +1262,11 @@ StartReadBuffersImpl(ReadBuffersOperation *operation,
{
int actual_nblocks = *nblocks;
int maxcombine = 0;
+ bool did_start_io;
Assert(*nblocks > 0);
Assert(*nblocks <= MAX_IO_COMBINE_LIMIT);
+ Assert(*nblocks == 1 || allow_forwarding);
for (int i = 0; i < actual_nblocks; ++i)
{
@@ -1297,6 +1305,7 @@ StartReadBuffersImpl(ReadBuffersOperation *operation,
bufHdr = GetLocalBufferDescriptor(-buffers[i] - 1);
else
bufHdr = GetBufferDescriptor(buffers[i] - 1);
+ Assert(pg_atomic_read_u32(&bufHdr->state) & BM_TAG_VALID);
found = pg_atomic_read_u32(&bufHdr->state) & BM_VALID;
}
else
@@ -1321,6 +1330,20 @@ StartReadBuffersImpl(ReadBuffersOperation *operation,
if (i == 0)
{
*nblocks = 1;
+
+#ifdef USE_ASSERT_CHECKING
+
+ /*
+ * Initialize enough of ReadBuffersOperation to make
+ * CheckReadBuffersOperation() work. Outside of assertions
+ * that's not necessary when no IO is issued.
+ */
+ operation->buffers = buffers;
+ operation->blocknum = blockNum;
+ operation->nblocks = 1;
+ operation->nblocks_done = 1;
+ CheckReadBuffersOperation(operation, true);
+#endif
return false;
}
@@ -1363,25 +1386,74 @@ StartReadBuffersImpl(ReadBuffersOperation *operation,
operation->blocknum = blockNum;
operation->flags = flags;
operation->nblocks = actual_nblocks;
+ operation->nblocks_done = 0;
+ pgaio_wref_clear(&operation->io_wref);
- if (flags & READ_BUFFERS_ISSUE_ADVICE)
+ /*
+ * When using AIO, start the IO in the background. If not, issue prefetch
+ * requests if desired by the caller.
+ *
+ * The reason we have a dedicated path for IOMETHOD_SYNC here is to
+ * de-risk the introduction of AIO somewhat. It's a large architectural
+ * change, with lots of chances for unanticipated performance effects.
+ *
+ * Use of IOMETHOD_SYNC already leads to not actually performing IO
+ * asynchronously, but without the check here we'd execute IO earlier than
+ * we used to. Eventually this IOMETHOD_SYNC specific path should go away.
+ */
+ if (io_method != IOMETHOD_SYNC)
{
/*
- * In theory we should only do this if PinBufferForBlock() had to
- * allocate new buffers above. That way, if two calls to
- * StartReadBuffers() were made for the same blocks before
- * WaitReadBuffers(), only the first would issue the advice. That'd be
- * a better simulation of true asynchronous I/O, which would only
- * start the I/O once, but isn't done here for simplicity.
+ * Try to start IO asynchronously. It's possible that no IO needs to
+ * be started, if another backend already performed the IO.
+ *
+ * Note that if an IO is started, it might not cover the entire
+ * requested range, e.g. because an intermediary block has been read
+ * in by another backend. In that case any "trailing" buffers we
+ * already pinned above will be "forwarded" by read_stream.c to the
+ * next call to StartReadBuffers().
+ *
+ * This is signalled to the caller by decrementing *nblocks *and*
+ * reducing operation->nblocks. The latter is done here, but not below
+ * WaitReadBuffers(), as in WaitReadBuffers() we can't "shorten" the
+ * overall read size anymore, we need to retry until done in its
+ * entirety or until failed.
*/
- smgrprefetch(operation->smgr,
- operation->forknum,
- blockNum,
- actual_nblocks);
+ did_start_io = AsyncReadBuffers(operation, nblocks);
+
+ operation->nblocks = *nblocks;
+ }
+ else
+ {
+ operation->flags |= READ_BUFFERS_SYNCHRONOUSLY;
+
+ if (flags & READ_BUFFERS_ISSUE_ADVICE)
+ {
+ /*
+ * In theory we should only do this if PinBufferForBlock() had to
+ * allocate new buffers above. That way, if two calls to
+ * StartReadBuffers() were made for the same blocks before
+ * WaitReadBuffers(), only the first would issue the advice.
+ * That'd be a better simulation of true asynchronous I/O, which
+ * would only start the I/O once, but isn't done here for
+ * simplicity.
+ */
+ smgrprefetch(operation->smgr,
+ operation->forknum,
+ blockNum,
+ actual_nblocks);
+ }
+
+ /*
+ * Indicate that WaitReadBuffers() should be called. WaitReadBuffers()
+ * will initiate the necessary IO.
+ */
+ did_start_io = true;
}
- /* Indicate that WaitReadBuffers() should be called. */
- return true;
+ CheckReadBuffersOperation(operation, !did_start_io);
+
+ return did_start_io;
}
/*
@@ -1448,8 +1520,35 @@ StartReadBuffer(ReadBuffersOperation *operation,
return result;
}
+/*
+ * Perform sanity checks on the ReadBuffersOperation.
+ */
+static void
+CheckReadBuffersOperation(ReadBuffersOperation *operation, bool is_complete)
+{
+#ifdef USE_ASSERT_CHECKING
+ Assert(operation->nblocks_done <= operation->nblocks);
+ Assert(!is_complete || operation->nblocks == operation->nblocks_done);
+
+ for (int i = 0; i < operation->nblocks; i++)
+ {
+ Buffer buffer = operation->buffers[i];
+ BufferDesc *buf_hdr = BufferIsLocal(buffer) ?
+ GetLocalBufferDescriptor(-buffer - 1) :
+ GetBufferDescriptor(buffer - 1);
+
+ Assert(BufferGetBlockNumber(buffer) == operation->blocknum + i);
+ Assert(pg_atomic_read_u32(&buf_hdr->state) & BM_TAG_VALID);
+
+ if (i < operation->nblocks_done)
+ Assert(pg_atomic_read_u32(&buf_hdr->state) & BM_VALID);
+ }
+#endif
+}
+
+/* helper for ReadBuffersCanStartIO(), to avoid repetition */
static inline bool
-WaitReadBuffersCanStartIO(Buffer buffer, bool nowait)
+ReadBuffersCanStartIOOnce(Buffer buffer, bool nowait)
{
if (BufferIsLocal(buffer))
return StartLocalBufferIO(GetLocalBufferDescriptor(-buffer - 1),
@@ -1458,31 +1557,243 @@ WaitReadBuffersCanStartIO(Buffer buffer, bool nowait)
return StartBufferIO(GetBufferDescriptor(buffer - 1), true, nowait);
}
-void
-WaitReadBuffers(ReadBuffersOperation *operation)
+/*
+ * Helper for AsyncReadBuffers that tries to initiate IO on the buffer.
+ */
+static inline bool
+ReadBuffersCanStartIO(Buffer buffer, bool nowait)
{
- Buffer *buffers;
- int nblocks;
- BlockNumber blocknum;
- ForkNumber forknum;
- IOContext io_context;
- IOObject io_object;
- char persistence;
-
- /* Find the range of the physical read we need to perform. */
- nblocks = operation->nblocks;
- buffers = &operation->buffers[0];
- blocknum = operation->blocknum;
- forknum = operation->forknum;
- persistence = operation->persistence;
+ /*
+ * If this backend currently has staged IO, we we need to submit the
+ * pending IO before waiting for the right to issue IO, to avoid the
+ * potential for deadlocks (and, more commonly, unnecessary delays for
+ * other backends).
+ */
+ if (!nowait && pgaio_have_staged())
+ {
+ if (ReadBuffersCanStartIOOnce(buffer, true))
+ return true;
+
+ /*
+ * Unfortunately a false returned StartBufferIO() doesn't allow to
+ * distinguish between the the buffer already being valid and IO
+ * already being in progress. Since IO already being in progress is
+ * quite rare, this approach seems fine.
+ */
+ pgaio_submit_staged();
+ }
+
+ return ReadBuffersCanStartIOOnce(buffer, nowait);
+}
+
+/*
+ * Helper for WaitReadBuffers() that processes the results of a readv
+ * operation, raising an error if necessary.
+ */
+static void
+ProcessReadBuffersResult(ReadBuffersOperation *operation)
+{
+ PgAioReturn *aio_ret = &operation->io_return;
+ int nblocks = 0;
+
+ Assert(pgaio_wref_valid(&operation->io_wref));
+ Assert(aio_ret->result.status != ARS_UNKNOWN);
+
+ /*
+ * SMGR reports the number of blocks sucessfully read as the result of the
+ * IO operation. Thus we can simply add that to ->nblocks_done.
+ */
+
+ if (likely(aio_ret->result.status == ARS_OK))
+ nblocks = aio_ret->result.result;
+ else if (aio_ret->result.status == ARS_PARTIAL)
+ {
+ /*
+ * We'll retry, so we just emit a debug message to the server log (or
+ * not even that in prod scenarios).
+ */
+ pgaio_result_report(aio_ret->result, &aio_ret->target_data, DEBUG1);
+ nblocks = aio_ret->result.result;
+
+ elog(DEBUG3, "partial read, will retry");
+
+ }
+ else if (aio_ret->result.status == ARS_ERROR)
+ {
+ pgaio_result_report(aio_ret->result, &aio_ret->target_data, ERROR);
+ nblocks = 0; /* silence compiler */
+ }
Assert(nblocks > 0);
Assert(nblocks <= MAX_IO_COMBINE_LIMIT);
+ operation->nblocks_done += nblocks;
+
+ Assert(operation->nblocks_done <= operation->nblocks);
+}
+
+void
+WaitReadBuffers(ReadBuffersOperation *operation)
+{
+ PgAioReturn *aio_ret = &operation->io_return;
+ IOContext io_context;
+ IOObject io_object;
+
+ if (operation->persistence == RELPERSISTENCE_TEMP)
+ {
+ io_context = IOCONTEXT_NORMAL;
+ io_object = IOOBJECT_TEMP_RELATION;
+ }
+ else
+ {
+ io_context = IOContextForStrategy(operation->strategy);
+ io_object = IOOBJECT_RELATION;
+ }
+
+ /*
+ * If we get here without an IO operation having been issued, the
+ * io_method == IOMETHOD_SYNC path must have been used. Otherwise the
+ * caller should not have called WaitReadBuffers().
+ *
+ * In the case of IOMETHOD_SYNC, we start - as we used to before the
+ * introducing of AIO - the IO in WaitReadBuffers(). This is done as part
+ * of the retry logic below, no extra code is required.
+ *
+ * This path is expected to eventually go away.
+ */
+ if (!pgaio_wref_valid(&operation->io_wref) && io_method != IOMETHOD_SYNC)
+ elog(ERROR, "waiting for read operation that didn't read");
+
+ /*
+ * To handle partial reads, and IOMETHOD_SYNC, we re-issue IO until we're
+ * done. We may need multiple retries, not just because we could get
+ * multiple short reads, but also because some of the remaining to-be-read
+ * buffers may have been read in by other backends, limiting the IO size.
+ */
+ while (true)
+ {
+ int nblocks;
+
+ CheckReadBuffersOperation(operation, false);
+
+ /*
+ * If there is an IO associated with the operation, we may need to
+ * wait for it. It's possible for there to be no IO if
+ */
+ if (pgaio_wref_valid(&operation->io_wref))
+ {
+ /*
+ * Track the time spent waiting for the IO to complete. As
+ * tracking a wait even if we don't actually need to wait
+ *
+ * a) is not cheap, due to the timestamping overhead
+ *
+ * b) reports some time as waiting, even if we never waited
+ *
+ * we first check if we already know the IO is complete.
+ */
+ if (aio_ret->result.status == ARS_UNKNOWN &&
+ !pgaio_wref_check_done(&operation->io_wref))
+ {
+ instr_time io_start = pgstat_prepare_io_time(track_io_timing);
+
+ pgaio_wref_wait(&operation->io_wref);
+
+ /*
+ * The IO operation itself was already counted earlier, in
+ * AsyncReadBuffers(), this just accounts for the wait time.
+ */
+ pgstat_count_io_op_time(io_object, io_context, IOOP_READ,
+ io_start, 0, 0);
+ }
+ else
+ {
+ Assert(pgaio_wref_check_done(&operation->io_wref));
+ }
+
+ /*
+ * We now are sure the IO completed. Check the results. This
+ * includes reporting on errors if there were any.
+ */
+ ProcessReadBuffersResult(operation);
+ }
+
+ /*
+ * Most of the the time the one IO we started will read in everything.
+ * But we need to deal with short reads and buffers not needing IO
+ * anymore.
+ */
+ if (operation->nblocks_done == operation->nblocks)
+ break;
+
+ CHECK_FOR_INTERRUPTS();
+
+ /*
+ * This may only complete the IO partially, either because some
+ * buffers were already valid, or because of a short read.
+ *
+ * NB: In contrast to after the AsyncReadBuffers() call in
+ * StartReadBuffers(), we do *not* reduce
+ * ReadBuffersOperation->nblocks here, callers expect the full
+ * operation to be completed at this point (as more operations may
+ * have been queued).
+ */
+ AsyncReadBuffers(operation, &nblocks);
+ }
+
+ CheckReadBuffersOperation(operation, true);
+
+ /* NB: READ_DONE tracepoint was already executed in completion callback */
+}
+
+/*
+ * Initiate IO for the ReadBuffersOperation
+ *
+ * This function only starts a single IO at a time. The size of the IO may be
+ * limited to below the to-be-read blocks if one of the buffers has
+ * concurrently been read in. If the first to-be-read buffer is already valid,
+ * no IO will be issued.
+ *
+ * To support retries after short reads, the first operation->nblocks_done is
+ * buffers are skipped.
+ *
+ * On return *nblocks_progres is updated to reflect the number of buffers
+ * affected by the call. If the first buffer is valid, *nblocks_progress is
+ * set to 1 and operation->nblocks_done is incremented.
+ *
+ * Returns true if IO was initiated, false if no IO was necessary.
+ */
+static bool
+AsyncReadBuffers(ReadBuffersOperation *operation, int *nblocks_progress)
+{
+ Buffer *buffers = &operation->buffers[0];
+ int flags = operation->flags;
+ BlockNumber blocknum = operation->blocknum;
+ ForkNumber forknum = operation->forknum;
+ char persistence = operation->persistence;
+ int16 nblocks_done = operation->nblocks_done;
+ Buffer *io_buffers = &operation->buffers[nblocks_done];
+ int io_buffers_len = 0;
+ PgAioHandle *ioh;
+ uint32 ioh_flags = 0;
+ void *io_pages[MAX_IO_COMBINE_LIMIT];
+ IOContext io_context;
+ IOObject io_object;
+ bool did_start_io;
+
+ /*
+ * When this IO is executed synchronously, either because the caller will
+ * immediately block waiting for the IO or because IOMETHOD_SYNC is used,
+ * the AIO subsystem needs to know.
+ */
+ if (flags & READ_BUFFERS_SYNCHRONOUSLY)
+ ioh_flags |= PGAIO_HF_SYNCHRONOUS;
+
if (persistence == RELPERSISTENCE_TEMP)
{
io_context = IOCONTEXT_NORMAL;
io_object = IOOBJECT_TEMP_RELATION;
+ ioh_flags |= PGAIO_HF_REFERENCES_LOCAL;
}
else
{
@@ -1490,58 +1801,90 @@ WaitReadBuffers(ReadBuffersOperation *operation)
io_object = IOOBJECT_RELATION;
}
- for (int i = 0; i < nblocks; ++i)
+ /*
+ * Get IO before ReadBuffersCanStartIO, as pgaio_io_acquire() might block,
+ * which we don't want after setting IO_IN_PROGRESS.
+ *
+ * If IO needs to be submitted before we can get a handle, submit already
+ * staged IO first, so that other backends don't need to wait. There
+ * wouldn't be a deadlock risk, as pgaio_io_acquire() just needs to wait
+ * for already submitted IO, which doesn't require additional locks, but
+ * it could still cause undesirable waits.
+ *
+ * A secondary benefit is that this would allows us to measure the time in
+ * pgaio_io_acquire() without causing undue timer overhead in the common,
+ * non-blocking, case. However, currently the pgstats infrastructure
+ * doesn't really allow that, as it a) asserts that an operation can't
+ * have time without operations b) doesn't have an API to report
+ * "accumulated" time.
+ */
+ ioh = pgaio_io_acquire_nb(CurrentResourceOwner, &operation->io_return);
+ if (unlikely(!ioh))
+ {
+ pgaio_submit_staged();
+
+ ioh = pgaio_io_acquire(CurrentResourceOwner, &operation->io_return);
+ }
+
+ /*
+ * Check if we can start IO on the buffer.
+ *
+ * If an I/O is already in progress in another backend, we want to wait
+ * for the outcome: either done, or something went wrong and we will
+ * retry.
+ */
+ if (!ReadBuffersCanStartIO(buffers[nblocks_done], false))
+ {
+ /*
+ * Someone else has already completed this block, we're done.
+ *
+ * When IO is necessary, ->nblocks_done is updated in
+ * ProcessReadBuffersResult(), but that is not called if no IO is
+ * necessary. Thus update here.
+ */
+ operation->nblocks_done += 1;
+ *nblocks_progress = 1;
+
+ pgaio_io_release(ioh);
+ pgaio_wref_clear(&operation->io_wref);
+ did_start_io = false;
+
+ /*
+ * Report and track this as a 'hit' for this backend, even though it
+ * must have started out as a miss in PinBufferForBlock().
+ *
+ * Some of the accesses would otherwise never be counted (e.g.
+ * pgBufferUsage) or counted as a miss (e.g.
+ * pgstat_count_buffer_hit(), as we always call
+ * pgstat_count_buffer_read()).
+ */
+ TRACE_POSTGRESQL_BUFFER_READ_DONE(forknum, blocknum + operation->nblocks_done,
+ operation->smgr->smgr_rlocator.locator.spcOid,
+ operation->smgr->smgr_rlocator.locator.dbOid,
+ operation->smgr->smgr_rlocator.locator.relNumber,
+ operation->smgr->smgr_rlocator.backend,
+ true);
+
+ if (persistence == RELPERSISTENCE_TEMP)
+ pgBufferUsage.local_blks_hit += 1;
+ else
+ pgBufferUsage.shared_blks_hit += 1;
+
+ if (operation->rel)
+ pgstat_count_buffer_hit(operation->rel);
+
+ pgstat_count_io_op(io_object, io_context, IOOP_HIT, 1, 0);
+
+ if (VacuumCostActive)
+ VacuumCostBalance += VacuumCostPageHit;
+ }
+ else
{
- int io_buffers_len;
- Buffer io_buffers[MAX_IO_COMBINE_LIMIT];
- void *io_pages[MAX_IO_COMBINE_LIMIT];
instr_time io_start;
- BlockNumber io_first_block;
-
- /*
- * Skip this block if someone else has already completed it. If an
- * I/O is already in progress in another backend, this will wait for
- * the outcome: either done, or something went wrong and we will
- * retry.
- */
- if (!WaitReadBuffersCanStartIO(buffers[i], false))
- {
- /*
- * Report and track this as a 'hit' for this backend, even though
- * it must have started out as a miss in PinBufferForBlock().
- *
- * Some of the accesses would otherwise never be counted (e.g.
- * pgBufferUsage) or counted as a miss (e.g.
- * pgstat_count_buffer_hit(), as we always call
- * pgstat_count_buffer_read()).
- */
- TRACE_POSTGRESQL_BUFFER_READ_DONE(forknum, blocknum + i,
- operation->smgr->smgr_rlocator.locator.spcOid,
- operation->smgr->smgr_rlocator.locator.dbOid,
- operation->smgr->smgr_rlocator.locator.relNumber,
- operation->smgr->smgr_rlocator.backend,
- true);
-
- if (persistence == RELPERSISTENCE_TEMP)
- pgBufferUsage.local_blks_hit += 1;
- else
- pgBufferUsage.shared_blks_hit += 1;
-
- if (operation->rel)
- pgstat_count_buffer_hit(operation->rel);
-
- pgstat_count_io_op(io_object, io_context, IOOP_HIT, 1, 0);
-
- if (VacuumCostActive)
- VacuumCostBalance += VacuumCostPageHit;
-
- continue;
- }
/* We found a buffer that we need to read in. */
- io_buffers[0] = buffers[i];
- io_pages[0] = BufferGetBlock(buffers[i]);
- io_first_block = blocknum + i;
+ Assert(io_buffers[0] == buffers[nblocks_done]);
+ io_pages[0] = BufferGetBlock(buffers[nblocks_done]);
io_buffers_len = 1;
/*
@@ -1549,85 +1892,70 @@ WaitReadBuffers(ReadBuffersOperation *operation)
* buffers at the same time? In this case we don't wait if we see an
* I/O already in progress. We already hold BM_IO_IN_PROGRESS for the
* head block, so we should get on with that I/O as soon as possible.
- * We'll come back to this block again, above.
+ *
+ * We'll come back to this block in the next call to
+ * StartReadBuffers() -> AsyncReadBuffers().
*/
- while ((i + 1) < nblocks &&
- WaitReadBuffersCanStartIO(buffers[i + 1], true))
+ for (int i = nblocks_done + 1; i < operation->nblocks; i++)
{
+ if (!ReadBuffersCanStartIO(buffers[i], true))
+ break;
/* Must be consecutive block numbers. */
- Assert(BufferGetBlockNumber(buffers[i + 1]) ==
- BufferGetBlockNumber(buffers[i]) + 1);
+ Assert(BufferGetBlockNumber(buffers[i - 1]) ==
+ BufferGetBlockNumber(buffers[i]) - 1);
+ Assert(io_buffers[io_buffers_len] == buffers[i]);
- io_buffers[io_buffers_len] = buffers[++i];
io_pages[io_buffers_len++] = BufferGetBlock(buffers[i]);
}
+ /* get a reference to wait for in WaitReadBuffers() */
+ pgaio_io_get_wref(ioh, &operation->io_wref);
+
+ /* provide the list of buffers to the completion callbacks */
+ pgaio_io_set_handle_data_32(ioh, (uint32 *) io_buffers, io_buffers_len);
+
+ pgaio_io_register_callbacks(ioh,
+ persistence == RELPERSISTENCE_TEMP ?
+ PGAIO_HCB_LOCAL_BUFFER_READV :
+ PGAIO_HCB_SHARED_BUFFER_READV,
+ flags);
+
+ pgaio_io_set_flag(ioh, ioh_flags);
+
+ /* ---
+ * Even though we're trying to issue IO asynchronously, track the time
+ * in smgrstartreadv():
+ * - if io_method == IOMETHOD_SYNC, we will always perform the IO
+ * immediately
+ * - the io method might not support the IO (e.g. worker IO for a temp
+ * table)
+ * ---
+ */
io_start = pgstat_prepare_io_time(track_io_timing);
- smgrreadv(operation->smgr, forknum, io_first_block, io_pages, io_buffers_len);
- pgstat_count_io_op_time(io_object, io_context, IOOP_READ, io_start,
- 1, io_buffers_len * BLCKSZ);
-
- /* Verify each block we read, and terminate the I/O. */
- for (int j = 0; j < io_buffers_len; ++j)
- {
- BufferDesc *bufHdr;
- Block bufBlock;
-
- if (persistence == RELPERSISTENCE_TEMP)
- {
- bufHdr = GetLocalBufferDescriptor(-io_buffers[j] - 1);
- bufBlock = LocalBufHdrGetBlock(bufHdr);
- }
- else
- {
- bufHdr = GetBufferDescriptor(io_buffers[j] - 1);
- bufBlock = BufHdrGetBlock(bufHdr);
- }
-
- /* check for garbage data */
- if (!PageIsVerifiedExtended((Page) bufBlock, io_first_block + j,
- PIV_LOG_WARNING | PIV_REPORT_STAT))
- {
- if ((operation->flags & READ_BUFFERS_ZERO_ON_ERROR) || zero_damaged_pages)
- {
- ereport(WARNING,
- (errcode(ERRCODE_DATA_CORRUPTED),
- errmsg("invalid page in block %u of relation %s; zeroing out page",
- io_first_block + j,
- relpath(operation->smgr->smgr_rlocator, forknum).str)));
- memset(bufBlock, 0, BLCKSZ);
- }
- else
- ereport(ERROR,
- (errcode(ERRCODE_DATA_CORRUPTED),
- errmsg("invalid page in block %u of relation %s",
- io_first_block + j,
- relpath(operation->smgr->smgr_rlocator, forknum).str)));
- }
-
- /* Set BM_VALID, terminate IO, and wake up any waiters */
- if (persistence == RELPERSISTENCE_TEMP)
- TerminateLocalBufferIO(bufHdr, false, BM_VALID, true);
- else
- TerminateBufferIO(bufHdr, false, BM_VALID, true, true);
-
- /* Report I/Os as completing individually. */
- TRACE_POSTGRESQL_BUFFER_READ_DONE(forknum, io_first_block + j,
- operation->smgr->smgr_rlocator.locator.spcOid,
- operation->smgr->smgr_rlocator.locator.dbOid,
- operation->smgr->smgr_rlocator.locator.relNumber,
- operation->smgr->smgr_rlocator.backend,
- false);
- }
+ smgrstartreadv(ioh, operation->smgr, forknum,
+ blocknum + nblocks_done,
+ io_pages, io_buffers_len);
+ pgstat_count_io_op_time(io_object, io_context, IOOP_READ,
+ io_start, 1, *nblocks_progress * BLCKSZ);
if (persistence == RELPERSISTENCE_TEMP)
pgBufferUsage.local_blks_read += io_buffers_len;
else
pgBufferUsage.shared_blks_read += io_buffers_len;
+ /*
+ * Track vacuum cost when issuing IO, not after waiting for it.
+ * Otherwise we could end up issuing a lot of IO in a short timespan,
+ * despite a low cost limit.
+ */
if (VacuumCostActive)
VacuumCostBalance += VacuumCostPageMiss * io_buffers_len;
+
+ *nblocks_progress = io_buffers_len;
+ did_start_io = true;
}
+
+ return did_start_io;
}
/*
--
2.48.1.76.g4e746b1a31.dirty
[text/x-diff] v2.9-0017-docs-Reframe-track_io_timing-related-docs-as-wa.patch (5.3K, ../../oav52ytakcvfnk52w5o5wvxdjavxth2tgptxt25gnfmh2pdcpk@omp3xukdu5if/18-v2.9-0017-docs-Reframe-track_io_timing-related-docs-as-wa.patch)
download | inline diff:
From b2f6cddcf8f00c757d2c0e4679416de6d3e2c66e Mon Sep 17 00:00:00 2001
From: Andres Freund <andres@anarazel.de>
Date: Sat, 15 Mar 2025 12:29:57 -0400
Subject: [PATCH v2.9 17/30] docs: Reframe track_io_timing related docs as wait
time
With AIO it does not make sense anymore to track the time for each individual
IO, as multiple IOs can be in-flight at the same time. Instead we now track
the time spent *waiting* for IOs.
This should be reflected in the docs. While, so far, we only do a subset of
reads, and no other operations, via AIO, describing the GUC and view columns
as measuring IO waits is accurate for synchronous and asynchronous IO.
Discussion: https://postgr.es/m/5dzyoduxlvfg55oqtjyjehez5uoq6hnwgzor4kkybkfdgkj7ag@rbi4gsmzaczk
---
doc/src/sgml/config.sgml | 4 ++--
doc/src/sgml/monitoring.sgml | 12 ++++++------
2 files changed, 8 insertions(+), 8 deletions(-)
diff --git a/doc/src/sgml/config.sgml b/doc/src/sgml/config.sgml
index 8d53478f53a..69638d7c1f9 100644
--- a/doc/src/sgml/config.sgml
+++ b/doc/src/sgml/config.sgml
@@ -8549,7 +8549,7 @@ COPY postgres_log FROM '/full/path/to/logfile.csv' WITH csv;
</term>
<listitem>
<para>
- Enables timing of database I/O calls. This parameter is off by
+ Enables timing of database I/O waits. This parameter is off by
default, as it will repeatedly query the operating system for
the current time, which may cause significant overhead on some
platforms. You can use the <xref linkend="pgtesttiming"/> tool to
@@ -8583,7 +8583,7 @@ COPY postgres_log FROM '/full/path/to/logfile.csv' WITH csv;
</term>
<listitem>
<para>
- Enables timing of WAL I/O calls. This parameter is off by default,
+ Enables timing of WAL I/O waits. This parameter is off by default,
as it will repeatedly query the operating system for the current time,
which may cause significant overhead on some platforms.
You can use the <application>pg_test_timing</application> tool to
diff --git a/doc/src/sgml/monitoring.sgml b/doc/src/sgml/monitoring.sgml
index aaa6586d3a4..7598340072f 100644
--- a/doc/src/sgml/monitoring.sgml
+++ b/doc/src/sgml/monitoring.sgml
@@ -2733,7 +2733,7 @@ description | Waiting for a newly initialized WAL file to reach durable storage
<structfield>read_time</structfield> <type>double precision</type>
</para>
<para>
- Time spent in read operations in milliseconds (if
+ Time spent waiting for read operations in milliseconds (if
<xref linkend="guc-track-io-timing"/> is enabled and
<varname>object</varname> is not <literal>wal</literal>,
or if <xref linkend="guc-track-wal-io-timing"/> is enabled
@@ -2771,7 +2771,7 @@ description | Waiting for a newly initialized WAL file to reach durable storage
<structfield>write_time</structfield> <type>double precision</type>
</para>
<para>
- Time spent in write operations in milliseconds (if
+ Time spent waiting for write operations in milliseconds (if
<xref linkend="guc-track-io-timing"/> is enabled and
<varname>object</varname> is not <literal>wal</literal>,
or if <xref linkend="guc-track-wal-io-timing"/> is enabled
@@ -2799,7 +2799,7 @@ description | Waiting for a newly initialized WAL file to reach durable storage
<structfield>writeback_time</structfield> <type>double precision</type>
</para>
<para>
- Time spent in writeback operations in milliseconds (if
+ Time spent waiting for writeback operations in milliseconds (if
<xref linkend="guc-track-io-timing"/> is enabled, otherwise zero). This
includes the time spent queueing write-out requests and, potentially,
the time spent to write out the dirty data.
@@ -2835,7 +2835,7 @@ description | Waiting for a newly initialized WAL file to reach durable storage
<structfield>extend_time</structfield> <type>double precision</type>
</para>
<para>
- Time spent in extend operations in milliseconds. (if
+ Time spent waiting for extend operations in milliseconds. (if
<xref linkend="guc-track-io-timing"/> is enabled and
<varname>object</varname> is not <literal>wal</literal>,
or if <xref linkend="guc-track-wal-io-timing"/> is enabled
@@ -2909,7 +2909,7 @@ description | Waiting for a newly initialized WAL file to reach durable storage
<structfield>fsync_time</structfield> <type>double precision</type>
</para>
<para>
- Time spent in fsync operations in milliseconds (if
+ Time spent waiting for fsync operations in milliseconds (if
<xref linkend="guc-track-io-timing"/> is enabled and
<varname>object</varname> is not <literal>wal</literal>,
or if <xref linkend="guc-track-wal-io-timing"/> is enabled
@@ -2996,7 +2996,7 @@ description | Waiting for a newly initialized WAL file to reach durable storage
<note>
<para>
- Columns tracking I/O time will only be non-zero when
+ Columns tracking I/O wait time will only be non-zero when
<xref linkend="guc-track-io-timing"/> is enabled. The user should be
careful when referencing these columns in combination with their
corresponding I/O operations in case <varname>track_io_timing</varname>
--
2.48.1.76.g4e746b1a31.dirty
[text/x-diff] v2.9-0018-aio-Basic-read_stream-adjustments-for-real-AIO.patch (3.8K, ../../oav52ytakcvfnk52w5o5wvxdjavxth2tgptxt25gnfmh2pdcpk@omp3xukdu5if/19-v2.9-0018-aio-Basic-read_stream-adjustments-for-real-AIO.patch)
download | inline diff:
From 00d41c3c9fa6e9f66fac4422afebd88505bd7f22 Mon Sep 17 00:00:00 2001
From: Andres Freund <andres@anarazel.de>
Date: Sat, 15 Mar 2025 12:29:57 -0400
Subject: [PATCH v2.9 18/30] aio: Basic read_stream adjustments for real AIO
Adapt the read stream logic for real AIO:
- If AIO is enabled, we shouldn't issue advice, but if it isn't, we should
continue issuing advice
- AIO benefits from reading ahead with direct IO
- While in read_stream_look_ahead(), we can use AIO batch submission mode for
increased efficiency
There is one comment talking about max_ios logic with "real asynchronous I/O"
that I am not sure about, so I left it alone for now.
There are further improvements we should consider, e.g. waiting to issue IOs
until we can issue multiple IOs at once. But that's left for a future change,
since it would involve additional heuristics.
---
src/backend/storage/aio/read_stream.c | 29 ++++++++++++++++++++++-----
1 file changed, 24 insertions(+), 5 deletions(-)
diff --git a/src/backend/storage/aio/read_stream.c b/src/backend/storage/aio/read_stream.c
index cdf4b5a86a2..d7b395b86a3 100644
--- a/src/backend/storage/aio/read_stream.c
+++ b/src/backend/storage/aio/read_stream.c
@@ -72,6 +72,7 @@
#include "postgres.h"
#include "miscadmin.h"
+#include "storage/aio.h"
#include "storage/fd.h"
#include "storage/smgr.h"
#include "storage/read_stream.h"
@@ -99,6 +100,7 @@ struct ReadStream
int16 pinned_buffers;
int16 distance;
int16 initialized_buffers;
+ bool sync_mode;
bool advice_enabled;
bool temporary;
@@ -416,6 +418,13 @@ read_stream_start_pending_read(ReadStream *stream)
static void
read_stream_look_ahead(ReadStream *stream)
{
+ /*
+ * Allow amortizing the cost of submitting IO over multiple IOs. This
+ * requires that we don't do any operations that could lead to a deadlock
+ * with staged-but-unsubmitted IO.
+ */
+ pgaio_enter_batchmode();
+
while (stream->ios_in_progress < stream->max_ios &&
stream->pinned_buffers + stream->pending_read_nblocks < stream->distance)
{
@@ -463,6 +472,7 @@ read_stream_look_ahead(ReadStream *stream)
{
/* We've hit the buffer or I/O limit. Rewind and stop here. */
read_stream_unget_block(stream, blocknum);
+ pgaio_exit_batchmode();
return;
}
}
@@ -497,6 +507,8 @@ read_stream_look_ahead(ReadStream *stream)
* time.
*/
Assert(stream->pinned_buffers > 0 || stream->distance == 0);
+
+ pgaio_exit_batchmode();
}
/*
@@ -628,15 +640,19 @@ read_stream_begin_impl(int flags,
stream->per_buffer_data = (void *)
MAXALIGN(&stream->ios[Max(1, max_ios)]);
+ stream->sync_mode = io_method == IOMETHOD_SYNC;
+
#ifdef USE_PREFETCH
/*
- * This system supports prefetching advice. We can use it as long as
- * direct I/O isn't enabled, the caller hasn't promised sequential access
- * (overriding our detection heuristics), and max_ios hasn't been set to
- * zero.
+ * This system supports prefetching advice.
+ *
+ * Issue advice only if AIO is not used, direct I/O isn't enabled, the
+ * caller hasn't promised sequential access (overriding our detection
+ * heuristics), and max_ios hasn't been set to zero.
*/
- if ((io_direct_flags & IO_DIRECT_DATA) == 0 &&
+ if (stream->sync_mode &&
+ (io_direct_flags & IO_DIRECT_DATA) == 0 &&
(flags & READ_STREAM_SEQUENTIAL) == 0 &&
max_ios > 0)
stream->advice_enabled = true;
@@ -646,6 +662,9 @@ read_stream_begin_impl(int flags,
* For now, max_ios = 0 is interpreted as max_ios = 1 with advice disabled
* above. If we had real asynchronous I/O we might need a slightly
* different definition.
+ *
+ * FIXME: Not sure what different definition we would need? I guess we
+ * could add the READ_BUFFERS_SYNCHRONOUSLY flag automatically?
*/
if (max_ios == 0)
max_ios = 1;
--
2.48.1.76.g4e746b1a31.dirty
[text/x-diff] v2.9-0019-aio-Experimental-heuristics-to-increase-batchin.patch (1.6K, ../../oav52ytakcvfnk52w5o5wvxdjavxth2tgptxt25gnfmh2pdcpk@omp3xukdu5if/20-v2.9-0019-aio-Experimental-heuristics-to-increase-batchin.patch)
download | inline diff:
From 6c3d70000a3805f8961b7cb7d74e4da4a79bedce Mon Sep 17 00:00:00 2001
From: Andres Freund <andres@anarazel.de>
Date: Sat, 15 Mar 2025 12:29:57 -0400
Subject: [PATCH v2.9 19/30] aio: Experimental heuristics to increase batching
in read_stream.c
Author:
Reviewed-by:
Discussion: https://postgr.es/m/
Backpatch:
---
src/backend/storage/aio/read_stream.c | 24 ++++++++++++++++++++++++
1 file changed, 24 insertions(+)
diff --git a/src/backend/storage/aio/read_stream.c b/src/backend/storage/aio/read_stream.c
index d7b395b86a3..887be2b3961 100644
--- a/src/backend/storage/aio/read_stream.c
+++ b/src/backend/storage/aio/read_stream.c
@@ -418,6 +418,30 @@ read_stream_start_pending_read(ReadStream *stream)
static void
read_stream_look_ahead(ReadStream *stream)
{
+ /*
+ * Batch-submitting multiple IOs is more efficient than doing so
+ * one-by-one. If we just ramp up to the max, we'll only be allowed to
+ * submit one io_combine_limit sized IO. Defer submitting IO in that case.
+ *
+ * FIXME: This needs better heuristics.
+ */
+#if 1
+ if (!stream->sync_mode && stream->distance > (io_combine_limit * 8))
+ {
+ if (stream->pinned_buffers + stream->pending_read_nblocks > ((stream->distance * 3) / 4))
+ {
+#if 0
+ ereport(LOG,
+ errmsg("reduce reduce reduce: pinned: %d, pending: %d, distance: %d",
+ stream->pinned_buffers,
+ stream->pending_read_nblocks,
+ stream->distance));
+#endif
+ return;
+ }
+ }
+#endif
+
/*
* Allow amortizing the cost of submitting IO over multiple IOs. This
* requires that we don't do any operations that could lead to a deadlock
--
2.48.1.76.g4e746b1a31.dirty
[text/x-diff] v2.9-0020-aio-Add-test_aio-module.patch (42.4K, ../../oav52ytakcvfnk52w5o5wvxdjavxth2tgptxt25gnfmh2pdcpk@omp3xukdu5if/21-v2.9-0020-aio-Add-test_aio-module.patch)
download | inline diff:
From 76e4646099e84780dc3a294ce50a9089261c91b9 Mon Sep 17 00:00:00 2001
From: Andres Freund <andres@anarazel.de>
Date: Sat, 15 Mar 2025 12:29:57 -0400
Subject: [PATCH v2.9 20/30] aio: Add test_aio module
---
src/include/storage/buf_internals.h | 6 +
src/backend/storage/buffer/bufmgr.c | 8 +-
src/test/modules/Makefile | 1 +
src/test/modules/meson.build | 1 +
src/test/modules/test_aio/.gitignore | 2 +
src/test/modules/test_aio/Makefile | 27 +
src/test/modules/test_aio/meson.build | 37 ++
src/test/modules/test_aio/t/001_aio.pl | 674 ++++++++++++++++++++
src/test/modules/test_aio/test_aio--1.0.sql | 97 +++
src/test/modules/test_aio/test_aio.c | 616 ++++++++++++++++++
src/test/modules/test_aio/test_aio.control | 3 +
11 files changed, 1466 insertions(+), 6 deletions(-)
create mode 100644 src/test/modules/test_aio/.gitignore
create mode 100644 src/test/modules/test_aio/Makefile
create mode 100644 src/test/modules/test_aio/meson.build
create mode 100644 src/test/modules/test_aio/t/001_aio.pl
create mode 100644 src/test/modules/test_aio/test_aio--1.0.sql
create mode 100644 src/test/modules/test_aio/test_aio.c
create mode 100644 src/test/modules/test_aio/test_aio.control
diff --git a/src/include/storage/buf_internals.h b/src/include/storage/buf_internals.h
index 8a52ace37cd..4b1f52a9fc8 100644
--- a/src/include/storage/buf_internals.h
+++ b/src/include/storage/buf_internals.h
@@ -434,6 +434,12 @@ extern void IssuePendingWritebacks(WritebackContext *wb_context, IOContext io_co
extern void ScheduleBufferTagForWriteback(WritebackContext *wb_context,
IOContext io_context, BufferTag *tag);
+/* solely to make it easier to write tests */
+extern bool StartBufferIO(BufferDesc *buf, bool forInput, bool nowait);
+extern void TerminateBufferIO(BufferDesc *buf, bool clear_dirty, uint32 set_flag_bits,
+ bool forget_owner, bool syncio);
+
+
/* freelist.c */
extern IOContext IOContextForStrategy(BufferAccessStrategy strategy);
extern BufferDesc *StrategyGetBuffer(BufferAccessStrategy strategy,
diff --git a/src/backend/storage/buffer/bufmgr.c b/src/backend/storage/buffer/bufmgr.c
index 2691757016c..860886d4d90 100644
--- a/src/backend/storage/buffer/bufmgr.c
+++ b/src/backend/storage/buffer/bufmgr.c
@@ -515,10 +515,6 @@ static uint32 WaitBufHdrUnlocked(BufferDesc *buf);
static int SyncOneBuffer(int buf_id, bool skip_recently_used,
WritebackContext *wb_context);
static void WaitIO(BufferDesc *buf);
-static bool StartBufferIO(BufferDesc *buf, bool forInput, bool nowait);
-static void TerminateBufferIO(BufferDesc *buf, bool clear_dirty,
- uint32 set_flag_bits, bool forget_owner,
- bool syncio);
static void AbortBufferIO(Buffer buffer);
static void shared_buffer_write_error_callback(void *arg);
static void local_buffer_write_error_callback(void *arg);
@@ -5927,7 +5923,7 @@ WaitIO(BufferDesc *buf)
* find out if they can perform the I/O as part of a larger operation, without
* waiting for the answer or distinguishing the reasons why not.
*/
-static bool
+bool
StartBufferIO(BufferDesc *buf, bool forInput, bool nowait)
{
uint32 buf_state;
@@ -5984,7 +5980,7 @@ StartBufferIO(BufferDesc *buf, bool forInput, bool nowait)
* resource owner. (forget_owner=false is used when the resource owner itself
* is being released)
*/
-static void
+void
TerminateBufferIO(BufferDesc *buf, bool clear_dirty, uint32 set_flag_bits,
bool forget_owner, bool syncio)
{
diff --git a/src/test/modules/Makefile b/src/test/modules/Makefile
index 4e4be3fa511..aa1d27bbed3 100644
--- a/src/test/modules/Makefile
+++ b/src/test/modules/Makefile
@@ -14,6 +14,7 @@ SUBDIRS = \
oauth_validator \
plsample \
spgist_name_ops \
+ test_aio \
test_bloomfilter \
test_copy_callbacks \
test_custom_rmgrs \
diff --git a/src/test/modules/meson.build b/src/test/modules/meson.build
index 2b057451473..4fda1acfb32 100644
--- a/src/test/modules/meson.build
+++ b/src/test/modules/meson.build
@@ -1,5 +1,6 @@
# Copyright (c) 2022-2025, PostgreSQL Global Development Group
+subdir('test_aio')
subdir('brin')
subdir('commit_ts')
subdir('delay_execution')
diff --git a/src/test/modules/test_aio/.gitignore b/src/test/modules/test_aio/.gitignore
new file mode 100644
index 00000000000..716e17f5a2a
--- /dev/null
+++ b/src/test/modules/test_aio/.gitignore
@@ -0,0 +1,2 @@
+# Generated subdirectories
+/tmp_check/
diff --git a/src/test/modules/test_aio/Makefile b/src/test/modules/test_aio/Makefile
new file mode 100644
index 00000000000..87d5315ba00
--- /dev/null
+++ b/src/test/modules/test_aio/Makefile
@@ -0,0 +1,27 @@
+# src/test/modules/delay_execution/Makefile
+
+PGFILEDESC = "test_aio - test code for AIO"
+
+MODULE_big = test_aio
+OBJS = \
+ $(WIN32RES) \
+ test_aio.o
+
+EXTENSION = test_aio
+DATA = test_aio--1.0.sql
+
+TAP_TESTS = 1
+
+export enable_injection_points
+export with_liburing
+
+ifdef USE_PGXS
+PG_CONFIG = pg_config
+PGXS := $(shell $(PG_CONFIG) --pgxs)
+include $(PGXS)
+else
+subdir = src/test/modules/test_aio
+top_builddir = ../../../..
+include $(top_builddir)/src/Makefile.global
+include $(top_srcdir)/contrib/contrib-global.mk
+endif
diff --git a/src/test/modules/test_aio/meson.build b/src/test/modules/test_aio/meson.build
new file mode 100644
index 00000000000..ac846b2b6f3
--- /dev/null
+++ b/src/test/modules/test_aio/meson.build
@@ -0,0 +1,37 @@
+# Copyright (c) 2022-2024, PostgreSQL Global Development Group
+
+test_aio_sources = files(
+ 'test_aio.c',
+)
+
+if host_system == 'windows'
+ test_aio_sources += rc_lib_gen.process(win32ver_rc, extra_args: [
+ '--NAME', 'test_aio',
+ '--FILEDESC', 'test_aio - test code for AIO',])
+endif
+
+test_aio = shared_module('test_aio',
+ test_aio_sources,
+ kwargs: pg_test_mod_args,
+)
+test_install_libs += test_aio
+
+test_install_data += files(
+ 'test_aio.control',
+ 'test_aio--1.0.sql',
+)
+
+tests += {
+ 'name': 'test_aio',
+ 'sd': meson.current_source_dir(),
+ 'bd': meson.current_build_dir(),
+ 'tap': {
+ 'env': {
+ 'enable_injection_points': get_option('injection_points') ? 'yes' : 'no',
+ 'with_liburing': liburing.found() ? 'yes' : 'no',
+ },
+ 'tests': [
+ 't/001_aio.pl',
+ ],
+ },
+}
diff --git a/src/test/modules/test_aio/t/001_aio.pl b/src/test/modules/test_aio/t/001_aio.pl
new file mode 100644
index 00000000000..ecd1f3c2f17
--- /dev/null
+++ b/src/test/modules/test_aio/t/001_aio.pl
@@ -0,0 +1,674 @@
+# Copyright (c) 2025, PostgreSQL Global Development Group
+
+use strict;
+use warnings FATAL => 'all';
+
+use PostgreSQL::Test::Cluster;
+use PostgreSQL::Test::Utils;
+use Test::More;
+
+
+###
+# Test io_method=worker
+###
+my $node_worker = create_node('worker');
+$node_worker->start();
+
+test_generic('worker', $node_worker);
+SKIP:
+{
+ skip 'Injection points not supported by this build', 1
+ unless $ENV{enable_injection_points} eq 'yes';
+ test_inject_worker('worker', $node_worker);
+}
+
+$node_worker->stop();
+
+
+###
+# Test io_method=io_uring
+###
+
+if ($ENV{with_liburing} eq 'yes')
+{
+ my $node_uring = create_node('io_uring');
+ $node_uring->start();
+ test_generic('io_uring', $node_uring);
+ $node_uring->stop();
+}
+
+
+###
+# Test io_method=sync
+###
+
+my $node_sync = create_node('sync');
+
+# just to have one test not use the default auto-tuning
+
+$node_sync->append_conf(
+ 'postgresql.conf', qq(
+io_max_concurrency=4
+));
+
+$node_sync->start();
+test_generic('sync', $node_sync);
+$node_sync->stop();
+
+done_testing();
+
+
+###
+# Test Helpers
+###
+
+sub create_node
+{
+ my $io_method = shift;
+
+ my $node = PostgreSQL::Test::Cluster->new($io_method);
+
+ # Want to test initdb for each IO method, otherwise we could just reuse
+ # the cluster.
+ #
+ # Unfortunately Cluster::init() puts PG_TEST_INITDB_EXTRA_OPTS after the
+ # options specified by ->extra, if somebody puts -c io_method=xyz in
+ # PG_TEST_INITDB_EXTRA_OPTS it would break this test. Fix that up if we
+ # detect it.
+ local $ENV{PG_TEST_INITDB_EXTRA_OPTS} = $ENV{PG_TEST_INITDB_EXTRA_OPTS};
+ if (defined $ENV{PG_TEST_INITDB_EXTRA_OPTS} &&
+ $ENV{PG_TEST_INITDB_EXTRA_OPTS} =~ m/io_method=/)
+ {
+ $ENV{PG_TEST_INITDB_EXTRA_OPTS} .= " -c io_method=$io_method";
+ }
+
+ $node->init(extra => [ '-c', "io_method=$io_method" ]);
+
+ $node->append_conf(
+ 'postgresql.conf', qq(
+shared_preload_libraries=test_aio
+log_min_messages = 'DEBUG3'
+log_statement=all
+restart_after_crash=false
+));
+
+ ok(1, "$io_method: initdb");
+
+ return $node;
+}
+
+sub psql_like
+{
+ my $io_method = shift;
+ my $psql = shift;
+ my $name = shift;
+ my $sql = shift;
+ my $expected_stdout = shift;
+ my $expected_stderr = shift;
+ my ($cmdret, $output);
+
+ ($output, $cmdret) = $psql->query($sql);
+
+ like($output, $expected_stdout, "$io_method: $name: expected stdout");
+ like($psql->{stderr}, $expected_stderr,
+ "$io_method: $name: expected stderr");
+ $psql->{stderr} = '';
+}
+
+sub query_wait_block
+{
+ my $io_method = shift;
+ my $node = shift;
+ my $psql = shift;
+ my $name = shift;
+ my $sql = shift;
+ my $waitfor = shift;
+
+ my $pid = $psql->query_safe('SELECT pg_backend_pid()');
+
+ $psql->{stdin} .= qq($sql;\n);
+ $psql->{run}->pump_nb();
+ ok(1, "$io_method: $name: issued sql");
+
+ $node->poll_query_until('postgres',
+ qq(SELECT wait_event FROM pg_stat_activity WHERE pid = $pid),
+ $waitfor,);
+ ok(1, "$io_method: $name: observed $waitfor wait event");
+}
+
+
+###
+# Sub-tests
+###
+
+sub test_handle
+{
+ my $io_method = shift;
+ my $node = shift;
+
+ my $psql = $node->background_psql('postgres', on_error_stop => 0);
+
+ # leak warning: implicit xact
+ psql_like(
+ $io_method,
+ $psql,
+ "handle_get() leak in implicit xact",
+ qq(SELECT handle_get()),
+ qr/^$/,
+ qr/leaked AIO handle/,
+ "$io_method: leaky handle_get() warns");
+
+ # leak warning: explicit xact
+ psql_like(
+ $io_method, $psql,
+ "handle_get() leak in explicit xact",
+ qq(BEGIN; SELECT handle_get(); COMMIT),
+ qr/^$/, qr/leaked AIO handle/);
+
+
+ # leak warning: explicit xact, rollback
+ psql_like(
+ $io_method,
+ $psql,
+ "handle_get() leak in explicit xact, rollback",
+ qq(BEGIN; SELECT handle_get(); ROLLBACK;),
+ qr/^$/,
+ qr/leaked AIO handle/);
+
+ # leak warning: subtrans
+ psql_like(
+ $io_method,
+ $psql,
+ "handle_get() leak in subxact",
+ qq(BEGIN; SAVEPOINT foo; SELECT handle_get(); COMMIT;),
+ qr/^$/,
+ qr/leaked AIO handle/);
+
+ # leak warning + error: released in different command (thus resowner)
+ psql_like(
+ $io_method,
+ $psql,
+ "handle_release() in different command",
+ qq(BEGIN; SELECT handle_get(); SELECT handle_release_last(); COMMIT;),
+ qr/^$/,
+ qr/leaked AIO handle.*release in unexpected state/ms);
+
+ # no leak, release in same command
+ psql_like(
+ $io_method,
+ $psql,
+ "handle_release() in same command",
+ qq(BEGIN; SELECT handle_get() UNION ALL SELECT handle_release_last(); COMMIT;),
+ qr/^$/,
+ qr/^$/);
+
+ # normal handle use
+ psql_like($io_method, $psql, "handle_get_release()",
+ qq(SELECT handle_get_release()),
+ qr/^$/, qr/^$/);
+
+ # should error out, API violation
+ psql_like($io_method, $psql, "handle_get_twice()",
+ qq(SELECT handle_get_release()),
+ qr/^$/, qr/^$/);
+
+ # recover after error in implicit xact
+ psql_like(
+ $io_method,
+ $psql,
+ "handle error recovery in implicit xact",
+ qq(SELECT handle_get_and_error(); SELECT 'ok', handle_get_release()),
+ qr/^|ok$/,
+ qr/ERROR.*as you command/);
+
+ # recover after error in implicit xact
+ psql_like(
+ $io_method,
+ $psql,
+ "handle error recovery in explicit xact",
+ qq(BEGIN; SELECT handle_get_and_error(); SELECT handle_get_release(), 'ok'; COMMIT;),
+ qr/^|ok$/,
+ qr/ERROR.*as you command/);
+
+ # recover after error in subtrans
+ psql_like(
+ $io_method,
+ $psql,
+ "handle error recovery in explicit subxact",
+ qq(BEGIN; SAVEPOINT foo; SELECT handle_get_and_error(); ROLLBACK TO SAVEPOINT foo; SELECT handle_get_release(); ROLLBACK;),
+ qr/^|ok$/,
+ qr/ERROR.*as you command/);
+
+ $psql->quit();
+}
+
+
+sub test_batch
+{
+ my $io_method = shift;
+ my $node = shift;
+
+ my $psql = $node->background_psql('postgres', on_error_stop => 0);
+
+ # leak warning & recovery: implicit xact
+ psql_like(
+ $io_method,
+ $psql,
+ "batch_start() leak & cleanup in implicit xact",
+ qq(SELECT batch_start()),
+ qr/^$/,
+ qr/open AIO batch at end/,
+ "$io_method: leaky batch_start() warns");
+
+ # leak warning & recovery: explicit xact
+ psql_like(
+ $io_method,
+ $psql,
+ "batch_start() leak & cleanup in explicit xact",
+ qq(BEGIN; SELECT batch_start(); COMMIT;),
+ qr/^$/,
+ qr/open AIO batch at end/,
+ "$io_method: leaky batch_start() warns");
+
+
+ # leak warning & recovery: explicit xact, rollback
+ #
+ # XXX: This doesn't fail right now, due to not getting a chance to do
+ # something at transaction command commit. That's not a correctness issue,
+ # it just means it's a bit harder to find buggy code.
+ #psql_like($io_method, $psql,
+ # "batch_start() leak & cleanup after abort",
+ # qq(BEGIN; SELECT batch_start(); ROLLBACK;),
+ # qr/^$/,
+ # qr/open AIO batch at end/, "$io_method: leaky batch_start() warns");
+
+ # no warning, batch closed in same command
+ psql_like(
+ $io_method,
+ $psql,
+ "batch_start(), batch_end() works",
+ qq(SELECT batch_start() UNION ALL SELECT batch_end()),
+ qr/^$/,
+ qr/^$/,
+ "$io_method: batch_start(), batch_end()");
+
+ $psql->quit();
+}
+
+sub test_io_error
+{
+ my $io_method = shift;
+ my $node = shift;
+ my ($ret, $output);
+
+ my $psql = $node->background_psql('postgres', on_error_stop => 0);
+
+ # verify the error is reported in custom C code
+ ($output, $ret) = $psql->query(
+ qq(SELECT read_rel_block_ll('tbl_corr', 1, wait_complete=>true);));
+ is($ret, 1, "$io_method: read_rel_block_ll() of tbl_corr page fails");
+ like(
+ $psql->{stderr},
+ qr/invalid page in block 1 of relation base\/.*/,
+ "$io_method: read_rel_block_ll() of tbl_corr page reports error");
+ $psql->{stderr} = '';
+
+ # verify the error is reported for bufmgr reads
+ ($output, $ret) =
+ $psql->query(qq(SELECT count(*) FROM tbl_corr WHERE ctid = '(1, 1)'));
+ is($ret, 1, "$io_method: tid scan reading tbl_corr block fails");
+ like(
+ $psql->{stderr},
+ qr/invalid page in block 1 of relation base\/.*/,
+ "$io_method: tid scan reading tbl_corr block reports error");
+ $psql->{stderr} = '';
+
+ # verify the error is reported for bufmgr reads
+ ($output, $ret) =
+ $psql->query(qq(SELECT count(*) FROM tbl_corr WHERE ctid = '(1, 1)'));
+ is($ret, 1, "$io_method: sequential scan reading tbl_corr block fails");
+ like(
+ $psql->{stderr},
+ qr/invalid page in block 1 of relation base\/.*/,
+ "$io_method: sequential scan reading tbl_corr block reports error");
+ $psql->{stderr} = '';
+
+ $psql->quit();
+}
+
+# Test interplay between StartBufferIO and TerminateBufferIO
+sub test_startwait_io
+{
+ my $io_method = shift;
+ my $node = shift;
+ my ($ret, $output);
+
+ my $psql_a = $node->background_psql('postgres', on_error_stop => 0);
+ my $psql_b = $node->background_psql('postgres', on_error_stop => 0);
+
+ # create a buffer we can play around with
+ ($output, $ret) =
+ $psql_a->query(qq(SELECT buffer_create_toy('tbl_ok', 1);));
+ is($ret, 0, "$io_method: create toy succeeds");
+ like($output, qr/^\d+$/, "$io_method: create toy returns numeric");
+ my $buf_id = $output;
+
+ # check that one backend can perform StartBufferIO
+ psql_like(
+ $io_method,
+ $psql_a,
+ "first StartBufferIO",
+ qq(SELECT buffer_call_start_io($buf_id, for_input=>true, nowait=>false);),
+ qr/^t$/,
+ qr/^$/);
+
+ # but not twice on the same buffer (non-waiting)
+ psql_like(
+ $io_method,
+ $psql_a,
+ "second StartBufferIO fails, same session",
+ qq(SELECT buffer_call_start_io($buf_id, for_input=>true, nowait=>true);),
+ qr/^f$/,
+ qr/^$/);
+ psql_like(
+ $io_method,
+ $psql_b,
+ "second StartBufferIO fails, other session",
+ qq(SELECT buffer_call_start_io($buf_id, for_input=>true, nowait=>true);),
+ qr/^f$/,
+ qr/^$/);
+
+ # start io in a different session, will block
+ query_wait_block(
+ $io_method,
+ $node,
+ $psql_b,
+ "blocking start buffer io",
+ qq(SELECT buffer_call_start_io($buf_id, for_input=>true, nowait=>false);),
+ "BufferIo");
+
+ # Terminate the IO, without marking it as success, this should trigger the
+ # waiting session to be able to start the io
+ ($output, $ret) = $psql_a->query(
+ qq(SELECT buffer_call_terminate_io($buf_id, for_input=>true, succeed=>false, io_error=>false, syncio=>true);)
+ );
+ is($ret, 0,
+ "$io_method: blocking start buffer io, terminating io, not valid");
+
+ # Because the IO was terminated, but not marked as valid, second session should get the right to start io
+ pump_until($psql_b->{run}, $psql_b->{timeout}, \$psql_b->{stdout}, qr/t/);
+ ok(1, "$io_method: blocking start buffer io, can start io");
+
+ # terminate the IO again
+ $psql_b->query_safe(
+ qq(SELECT buffer_call_terminate_io($buf_id, for_input=>true, succeed=>false, io_error=>false, syncio=>true);)
+ );
+
+
+ # same as the above scenario, but mark IO as having succeeded
+ ($output, $ret) =
+ $psql_a->query(qq(SELECT buffer_call_start_io($buf_id, true, false);));
+ is($ret, 0,
+ "$io_method: blocking buffer io w/ success: first start buffer io succeeds"
+ );
+ is($output, "t",
+ "$io_method: blocking buffer io w/ success: first start buffer io returns true"
+ );
+
+
+ # start io in a different session, will block
+ query_wait_block(
+ $io_method,
+ $node,
+ $psql_b,
+ "blocking start buffer io",
+ qq(SELECT buffer_call_start_io($buf_id, for_input=>true, nowait=>false);),
+ "BufferIo");
+
+ ($output, $ret) = $psql_a->query(
+ qq(SELECT buffer_call_terminate_io($buf_id, for_input=>true, succeed=>true, io_error=>false, syncio=>true);)
+ );
+ is($ret, 0,
+ "$io_method: blocking start buffer io, terminating IO, valid");
+
+ # Because the IO was terminated, and marked as valid, second session should complete but not need io
+ pump_until($psql_b->{run}, $psql_b->{timeout}, \$psql_b->{stdout}, qr/f/);
+ ok(1, "$io_method: blocking start buffer io, no need to start io");
+
+
+ # buffer is valid now, make it invalid again
+ $buf_id = $psql_a->query_safe(qq(SELECT buffer_create_toy('tbl_ok', 1);));
+
+ $psql_a->quit();
+ $psql_b->quit();
+}
+
+# Test that if the backend issuing a read doesn't wait for the IO's
+# completion, another backend can complete the IO
+sub test_complete_foreign
+{
+ my $io_method = shift;
+ my $node = shift;
+ my ($ret, $output);
+
+ my $psql_a = $node->background_psql('postgres', on_error_stop => 0);
+ my $psql_b = $node->background_psql('postgres', on_error_stop => 0);
+
+ # Issue IO without waiting for completion, then sleep
+ $psql_a->query_safe(
+ qq(SELECT read_rel_block_ll('tbl_ok', 1, wait_complete=>false);));
+
+ # Check that another backend can read the relevant block
+ psql_like(
+ $io_method,
+ $psql_b,
+ "completing read started by sleeping backend",
+ qq(SELECT count(*) FROM tbl_ok WHERE ctid = '(1,1)' LIMIT 1),
+ qr/^1$/,
+ qr/^$/);
+
+ # Issue IO without waiting for completion, then exit
+ $psql_a->query_safe(
+ qq(SELECT read_rel_block_ll('tbl_ok', 1, wait_complete=>false);));
+ $psql_a->reconnect_and_clear();
+
+ # Check that another backend can read the relevant block
+ psql_like(
+ $io_method,
+ $psql_b,
+ "completing read started by exited backend",
+ qq(SELECT count(*) FROM tbl_ok WHERE ctid = '(1,1)' LIMIT 1),
+ qr/^1$/,
+ qr/^$/);
+
+ # Read a tbl_corr block, then sleep. The other session will retry the IO
+ # and also fail. The easiest thing to verify that seems to be to check
+ # that both are in the log.
+ my $log_location = -s $node->logfile;
+ $psql_a->query_safe(
+ qq(SELECT read_rel_block_ll('tbl_corr', 1, wait_complete=>false);));
+
+ psql_like(
+ $io_method,
+ $psql_b,
+ "completing read of tbl_corr block started by other backend",
+ qq(SELECT count(*) FROM tbl_corr WHERE ctid = '(1,1)' LIMIT 1),
+ qr/^$/,
+ qr/invalid page in block/);
+
+ # The log message issued for the read_rel_block_ll() should be logged as a LOG
+ $node->wait_for_log(qr/LOG[^\n]+invalid page in/, $log_location);
+ ok(1,
+ "$io_method: completing read of tbl_corr block started by other backend: LOG message for background read"
+ );
+
+ # But for the SELECT, it should be an ERROR
+ $log_location =
+ $node->wait_for_log(qr/ERROR[^\n]+invalid page in/, $log_location);
+ ok(1,
+ "$io_method: completing read of tbl_corr block started by other backend: ERROR message for foreground read"
+ );
+
+ $psql_a->quit();
+ $psql_b->quit();
+}
+
+sub test_inject
+{
+ my $io_method = shift;
+ my $node = shift;
+ my ($ret, $output);
+
+ my $psql = $node->background_psql('postgres', on_error_stop => 0);
+
+ # injected what we'd expect
+ $psql->query_safe(qq(SELECT inj_io_short_read_attach(8192);));
+ $psql->query_safe(qq(SELECT invalidate_rel_block('tbl_ok', 2);));
+ ($output, $ret) =
+ $psql->query(qq(SELECT count(*) FROM tbl_ok WHERE ctid = '(2, 1)';));
+ is($ret, 0,
+ "$io_method: injection point not triggering failure succeeds");
+
+ # injected a read shorter than a single block, expecting error
+ $psql->query_safe(qq(SELECT inj_io_short_read_attach(17);));
+ $psql->query_safe(qq(SELECT invalidate_rel_block('tbl_ok', 2);));
+ ($output, $ret) =
+ $psql->query(qq(SELECT count(*) FROM tbl_ok WHERE ctid = '(2, 1)';));
+ is($ret, 1, "$io_method: single block short read fails");
+ like(
+ $psql->{stderr},
+ qr/ERROR:.*could not read blocks 2\.\.2 in file "base\/.*": read only 0 of 8192 bytes/,
+ "$io_method: single block short read reports error");
+ $psql->{stderr} = '';
+
+ # shorten multi-block read to a single block, should retry
+ $psql->query_safe(
+ qq(
+SELECT invalidate_rel_block('tbl_ok', 0);
+SELECT invalidate_rel_block('tbl_ok', 1);
+SELECT invalidate_rel_block('tbl_ok', 2);
+SELECT invalidate_rel_block('tbl_ok', 3);
+/* gap */
+SELECT invalidate_rel_block('tbl_ok', 5);
+SELECT invalidate_rel_block('tbl_ok', 6);
+SELECT invalidate_rel_block('tbl_ok', 7);
+SELECT invalidate_rel_block('tbl_ok', 8);
+
+SELECT inj_io_short_read_attach(8192);
+ ));
+ ($output, $ret) = $psql->query(qq(SELECT count(*) FROM tbl_ok;));
+ is($ret, 0, "$io_method: multi block short read is retried");
+
+ # verify that page verification errors are detected even as part of a
+ # shortened multi-block read (tbl_corr, block 1 is tbl_corred)
+ $psql->query_safe(
+ qq(
+SELECT invalidate_rel_block('tbl_corr', 0);
+SELECT invalidate_rel_block('tbl_corr', 1);
+SELECT invalidate_rel_block('tbl_corr', 2);
+SELECT inj_io_short_read_attach(8192);
+ ));
+ ($output, $ret) =
+ $psql->query(qq(SELECT count(*) FROM tbl_corr WHERE ctid < '(2, 1)'));
+ is($ret, 1,
+ "$io_method: shortened multi-block read detects invalid page");
+ like(
+ $psql->{stderr},
+ qr/ERROR:.*invalid page in block 1 of relation base\/.*/,
+ "$io_method: shortened multi-block reads reports invalid page");
+ $psql->{stderr} = '';
+
+ # trigger a hard error, should error out
+ $psql->query_safe(
+ qq(
+SELECT inj_io_short_read_attach(-errno_from_string('EIO'));
+SELECT invalidate_rel_block('tbl_ok', 2);
+ ));
+ ($output, $ret) =
+ $psql->query(qq(SELECT count(*) FROM tbl_ok; SELECT 1;));
+ is($ret, 1, "$io_method: hard IO error is detected");
+ like(
+ $psql->{stderr},
+ qr/ERROR:.*could not read blocks 2\.\.2 in file \"base\/.*\": Input\/output error/,
+ "$io_method: hard IO error is reported");
+ $psql->{stderr} = '';
+
+ $psql->query_safe(
+ qq(
+SELECT inj_io_short_read_detach();
+ ));
+
+ $psql->quit();
+}
+
+sub test_inject_worker
+{
+ my $io_method = shift;
+ my $node = shift;
+ my ($ret, $output);
+
+ my $psql = $node->background_psql('postgres', on_error_stop => 0);
+
+ # trigger a failure to reopen, should error out, but should recover
+ $psql->query_safe(
+ qq(
+SELECT inj_io_reopen_attach();
+SELECT invalidate_rel_block('tbl_ok', 1);
+ ));
+ ($output, $ret) = $psql->query(qq(SELECT count(*) FROM tbl_ok;));
+ is($ret, 1, "$io_method: failure to open is detected");
+ like(
+ $psql->{stderr},
+ qr/ERROR:.*could not read blocks 1\.\.1 in file "base\/.*": No such file or directory/,
+ "$io_method: failure to open is reported");
+ $psql->{stderr} = '';
+
+ $psql->query_safe(
+ qq(
+SELECT inj_io_reopen_detach();
+ ));
+
+ # check that we indeed recover
+ ($output, $ret) = $psql->query(qq(SELECT count(*) FROM tbl_ok;));
+ is($ret, 0, "$io_method: recovers from failure to open ");
+
+
+ $psql->quit();
+}
+
+sub test_generic
+{
+ my $io_method = shift;
+ my $node = shift;
+
+ is($node->safe_psql('postgres', 'SHOW io_method'),
+ $io_method, "$io_method: io_method set correctly");
+
+ $node->safe_psql(
+ 'postgres', qq(
+CREATE EXTENSION test_aio;
+CREATE TABLE tbl_corr(data int not null) WITH (AUTOVACUUM_ENABLED = false);
+CREATE TABLE tbl_ok(data int not null) WITH (AUTOVACUUM_ENABLED = false);
+
+INSERT INTO tbl_corr SELECT generate_series(1, 10000);
+INSERT INTO tbl_ok SELECT generate_series(1, 10000);
+SELECT grow_rel('tbl_corr', 16);
+SELECT grow_rel('tbl_ok', 16);
+
+SELECT corrupt_rel_block('tbl_corr', 1);
+CHECKPOINT;
+));
+
+ test_handle($io_method, $node);
+ test_io_error($io_method, $node);
+ test_batch($io_method, $node);
+ test_startwait_io($io_method, $node);
+ test_complete_foreign($io_method, $node);
+
+ SKIP:
+ {
+ skip 'Injection points not supported by this build', 1
+ unless $ENV{enable_injection_points} eq 'yes';
+ test_inject($io_method, $node);
+ }
+}
diff --git a/src/test/modules/test_aio/test_aio--1.0.sql b/src/test/modules/test_aio/test_aio--1.0.sql
new file mode 100644
index 00000000000..3d4e9bb070f
--- /dev/null
+++ b/src/test/modules/test_aio/test_aio--1.0.sql
@@ -0,0 +1,97 @@
+/* src/test/modules/test_aio/test_aio--1.0.sql */
+
+-- complain if script is sourced in psql, rather than via CREATE EXTENSION
+\echo Use "CREATE EXTENSION test_aio" to load this file. \quit
+
+
+CREATE FUNCTION errno_from_string(sym text)
+RETURNS pg_catalog.int4 STRICT
+AS 'MODULE_PATHNAME' LANGUAGE C;
+
+
+CREATE FUNCTION grow_rel(rel regclass, nblocks int)
+RETURNS pg_catalog.void STRICT
+AS 'MODULE_PATHNAME' LANGUAGE C;
+
+
+CREATE FUNCTION corrupt_rel_block(rel regclass, blockno int)
+RETURNS pg_catalog.void STRICT
+AS 'MODULE_PATHNAME' LANGUAGE C;
+
+CREATE FUNCTION read_rel_block_ll(rel regclass, blockno int, wait_complete bool)
+RETURNS pg_catalog.void STRICT
+AS 'MODULE_PATHNAME' LANGUAGE C;
+
+CREATE FUNCTION invalidate_rel_block(rel regclass, blockno int)
+RETURNS pg_catalog.void STRICT
+AS 'MODULE_PATHNAME' LANGUAGE C;
+
+CREATE FUNCTION buffer_create_toy(rel regclass, blockno int4)
+RETURNS pg_catalog.int4 STRICT
+AS 'MODULE_PATHNAME' LANGUAGE C;
+
+CREATE FUNCTION buffer_call_start_io(buffer int, for_input bool, nowait bool)
+RETURNS pg_catalog.bool STRICT
+AS 'MODULE_PATHNAME' LANGUAGE C;
+
+CREATE FUNCTION buffer_call_terminate_io(buffer int, for_input bool, succeed bool, io_error bool, syncio bool)
+RETURNS pg_catalog.void STRICT
+AS 'MODULE_PATHNAME' LANGUAGE C;
+
+
+
+/*
+ * Handle related functions
+ */
+CREATE FUNCTION handle_get_and_error()
+RETURNS pg_catalog.void STRICT
+AS 'MODULE_PATHNAME' LANGUAGE C;
+
+CREATE FUNCTION handle_get_twice()
+RETURNS pg_catalog.void STRICT
+AS 'MODULE_PATHNAME' LANGUAGE C;
+
+CREATE FUNCTION handle_get()
+RETURNS pg_catalog.void STRICT
+AS 'MODULE_PATHNAME' LANGUAGE C;
+
+CREATE FUNCTION handle_get_release()
+RETURNS pg_catalog.void STRICT
+AS 'MODULE_PATHNAME' LANGUAGE C;
+
+CREATE FUNCTION handle_release_last()
+RETURNS pg_catalog.void STRICT
+AS 'MODULE_PATHNAME' LANGUAGE C;
+
+
+/*
+ * Batchmode related functions
+ */
+CREATE FUNCTION batch_start()
+RETURNS pg_catalog.void STRICT
+AS 'MODULE_PATHNAME' LANGUAGE C;
+
+CREATE FUNCTION batch_end()
+RETURNS pg_catalog.void STRICT
+AS 'MODULE_PATHNAME' LANGUAGE C;
+
+
+
+/*
+ * Injection point related functions
+ */
+CREATE FUNCTION inj_io_short_read_attach(result int)
+RETURNS pg_catalog.void STRICT
+AS 'MODULE_PATHNAME' LANGUAGE C;
+
+CREATE FUNCTION inj_io_short_read_detach()
+RETURNS pg_catalog.void STRICT
+AS 'MODULE_PATHNAME' LANGUAGE C;
+
+CREATE FUNCTION inj_io_reopen_attach()
+RETURNS pg_catalog.void STRICT
+AS 'MODULE_PATHNAME' LANGUAGE C;
+
+CREATE FUNCTION inj_io_reopen_detach()
+RETURNS pg_catalog.void STRICT
+AS 'MODULE_PATHNAME' LANGUAGE C;
diff --git a/src/test/modules/test_aio/test_aio.c b/src/test/modules/test_aio/test_aio.c
new file mode 100644
index 00000000000..81b4d732206
--- /dev/null
+++ b/src/test/modules/test_aio/test_aio.c
@@ -0,0 +1,616 @@
+/*-------------------------------------------------------------------------
+ *
+ * delay_execution.c
+ * Test module to allow delay between parsing and execution of a query.
+ *
+ * The delay is implemented by taking and immediately releasing a specified
+ * advisory lock. If another process has previously taken that lock, the
+ * current process will be blocked until the lock is released; otherwise,
+ * there's no effect. This allows an isolationtester script to reliably
+ * test behaviors where some specified action happens in another backend
+ * between parsing and execution of any desired query.
+ *
+ * Copyright (c) 2020-2025, PostgreSQL Global Development Group
+ *
+ * IDENTIFICATION
+ * src/test/modules/delay_execution/delay_execution.c
+ *
+ *-------------------------------------------------------------------------
+ */
+
+#include "postgres.h"
+
+#include "access/relation.h"
+#include "fmgr.h"
+#include "storage/aio.h"
+#include "storage/aio_internal.h"
+#include "storage/buf_internals.h"
+#include "storage/bufmgr.h"
+#include "storage/ipc.h"
+#include "storage/lwlock.h"
+#include "utils/builtins.h"
+#include "utils/injection_point.h"
+#include "utils/rel.h"
+
+
+PG_MODULE_MAGIC;
+
+
+typedef struct InjIoErrorState
+{
+ bool enabled_short_read;
+ bool enabled_reopen;
+
+ bool short_read_result_set;
+ int short_read_result;
+} InjIoErrorState;
+
+static InjIoErrorState * inj_io_error_state;
+
+/* Shared memory init callbacks */
+static shmem_request_hook_type prev_shmem_request_hook = NULL;
+static shmem_startup_hook_type prev_shmem_startup_hook = NULL;
+
+
+static PgAioHandle *last_handle;
+
+
+
+static void
+test_aio_shmem_request(void)
+{
+ if (prev_shmem_request_hook)
+ prev_shmem_request_hook();
+
+ RequestAddinShmemSpace(sizeof(InjIoErrorState));
+}
+
+static void
+test_aio_shmem_startup(void)
+{
+ bool found;
+
+ if (prev_shmem_startup_hook)
+ prev_shmem_startup_hook();
+
+ /* Create or attach to the shared memory state */
+ LWLockAcquire(AddinShmemInitLock, LW_EXCLUSIVE);
+
+ inj_io_error_state = ShmemInitStruct("injection_points",
+ sizeof(InjIoErrorState),
+ &found);
+
+ if (!found)
+ {
+ /*
+ * First time through, so initialize. This is shared with the dynamic
+ * initialization using a DSM.
+ */
+ inj_io_error_state->enabled_short_read = false;
+ inj_io_error_state->enabled_reopen = false;
+
+#ifdef USE_INJECTION_POINTS
+ InjectionPointAttach("AIO_PROCESS_COMPLETION_BEFORE_SHARED",
+ "test_aio",
+ "inj_io_short_read",
+ NULL,
+ 0);
+ InjectionPointLoad("AIO_PROCESS_COMPLETION_BEFORE_SHARED");
+
+ InjectionPointAttach("AIO_WORKER_AFTER_REOPEN",
+ "test_aio",
+ "inj_io_reopen",
+ NULL,
+ 0);
+ InjectionPointLoad("AIO_WORKER_AFTER_REOPEN");
+
+#endif
+ }
+ else
+ {
+#ifdef USE_INJECTION_POINTS
+ InjectionPointLoad("AIO_PROCESS_COMPLETION_BEFORE_SHARED");
+ InjectionPointLoad("AIO_WORKER_AFTER_REOPEN");
+ elog(LOG, "injection point loaded");
+#endif
+ }
+
+ LWLockRelease(AddinShmemInitLock);
+}
+
+void
+_PG_init(void)
+{
+ if (!process_shared_preload_libraries_in_progress)
+ return;
+
+ /* Shared memory initialization */
+ prev_shmem_request_hook = shmem_request_hook;
+ shmem_request_hook = test_aio_shmem_request;
+ prev_shmem_startup_hook = shmem_startup_hook;
+ shmem_startup_hook = test_aio_shmem_startup;
+}
+
+
+PG_FUNCTION_INFO_V1(errno_from_string);
+Datum
+errno_from_string(PG_FUNCTION_ARGS)
+{
+ const char *sym = text_to_cstring(PG_GETARG_TEXT_PP(0));
+
+ if (strcmp(sym, "EIO") == 0)
+ PG_RETURN_INT32(EIO);
+ else if (strcmp(sym, "EAGAIN") == 0)
+ PG_RETURN_INT32(EAGAIN);
+ else if (strcmp(sym, "EINTR") == 0)
+ PG_RETURN_INT32(EINTR);
+ else if (strcmp(sym, "ENOSPC") == 0)
+ PG_RETURN_INT32(ENOSPC);
+ else if (strcmp(sym, "EROFS") == 0)
+ PG_RETURN_INT32(EROFS);
+
+ ereport(ERROR,
+ errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ errmsg_internal("%s is not a supported errno value", sym));
+ PG_RETURN_INT32(0);
+}
+
+
+PG_FUNCTION_INFO_V1(grow_rel);
+Datum
+grow_rel(PG_FUNCTION_ARGS)
+{
+ Oid relid = PG_GETARG_OID(0);
+ uint32 nblocks = PG_GETARG_UINT32(1);
+ Relation rel;
+#define MAX_BUFFERS_TO_EXTEND_BY 64
+ Buffer victim_buffers[MAX_BUFFERS_TO_EXTEND_BY];
+
+ rel = relation_open(relid, AccessExclusiveLock);
+
+ while (nblocks > 0)
+ {
+ uint32 extend_by_pages;
+
+ extend_by_pages = Min(nblocks, MAX_BUFFERS_TO_EXTEND_BY);
+
+ ExtendBufferedRelBy(BMR_REL(rel),
+ MAIN_FORKNUM,
+ NULL,
+ 0,
+ extend_by_pages,
+ victim_buffers,
+ &extend_by_pages);
+
+ nblocks -= extend_by_pages;
+
+ for (uint32 i = 0; i < extend_by_pages; i++)
+ {
+ ReleaseBuffer(victim_buffers[i]);
+ }
+ }
+
+ relation_close(rel, NoLock);
+
+ PG_RETURN_VOID();
+}
+
+PG_FUNCTION_INFO_V1(corrupt_rel_block);
+Datum
+corrupt_rel_block(PG_FUNCTION_ARGS)
+{
+ Oid relid = PG_GETARG_OID(0);
+ BlockNumber blkno = PG_GETARG_UINT32(1);
+ Relation rel;
+ Buffer buf;
+ Page page;
+ PageHeader ph;
+
+ rel = relation_open(relid, AccessExclusiveLock);
+
+ buf = ReadBuffer(rel, blkno);
+ page = BufferGetPage(buf);
+
+ LockBuffer(buf, BUFFER_LOCK_EXCLUSIVE);
+
+ MarkBufferDirty(buf);
+
+ PageInit(page, BufferGetPageSize(buf), 0);
+
+ ph = (PageHeader) page;
+ ph->pd_special = BLCKSZ + 1;
+
+ FlushOneBuffer(buf);
+
+ LockBuffer(buf, BUFFER_LOCK_UNLOCK);
+
+ ReleaseBuffer(buf);
+
+ EvictUnpinnedBuffer(buf);
+
+ relation_close(rel, NoLock);
+
+ PG_RETURN_VOID();
+}
+
+/*
+ * Ensures a buffer for rel & blkno is in shared buffers, without actually
+ * caring about the buffer contents. Used to set up test scenarios.
+ */
+static Buffer
+create_toy_buffer(Relation rel, BlockNumber blkno)
+{
+ Buffer buf;
+ BufferDesc *buf_hdr;
+ uint32 buf_state;
+ bool was_pinned = false;
+
+ /* read buffer without erroring out */
+ buf = ReadBufferExtended(rel, MAIN_FORKNUM, blkno, RBM_ZERO_ON_ERROR, NULL);
+
+ buf_hdr = GetBufferDescriptor(buf - 1);
+
+ buf_state = LockBufHdr(buf_hdr);
+
+ /*
+ * We should be the only backend accessing this buffer. This is just a
+ * small bit of belt-and-suspenders defense, none of this code should ever
+ * run in a cluster with real data.
+ */
+ if (BUF_STATE_GET_REFCOUNT(buf_state) > 1)
+ was_pinned = true;
+ else
+ buf_state &= ~(BM_VALID | BM_DIRTY);
+
+ UnlockBufHdr(buf_hdr, buf_state);
+
+ if (was_pinned)
+ elog(ERROR, "toy buffer %d was already pinned",
+ buf);
+
+ return buf;
+}
+
+PG_FUNCTION_INFO_V1(read_rel_block_ll);
+Datum
+read_rel_block_ll(PG_FUNCTION_ARGS)
+{
+ Oid relid = PG_GETARG_OID(0);
+ BlockNumber blkno = PG_GETARG_UINT32(1);
+ bool wait_complete = PG_GETARG_BOOL(2);
+ Relation rel;
+ Buffer buf;
+ Page pages[1];
+ PgAioReturn ior;
+ PgAioHandle *ioh;
+ PgAioWaitRef iow;
+ SMgrRelation smgr;
+
+ rel = relation_open(relid, AccessExclusiveLock);
+
+ buf = create_toy_buffer(rel, blkno);
+
+ pages[0] = BufferGetBlock(buf);
+
+ ioh = pgaio_io_acquire(CurrentResourceOwner, &ior);
+ pgaio_io_get_wref(ioh, &iow);
+
+ smgr = RelationGetSmgr(rel);
+
+ StartBufferIO(GetBufferDescriptor(buf - 1), true, false);
+
+ pgaio_io_set_handle_data_32(ioh, (uint32 *) &buf, 1);
+ pgaio_io_register_callbacks(ioh, PGAIO_HCB_SHARED_BUFFER_READV, 0);
+
+ elog(LOG, "about to smgrstartreadv");
+ smgrstartreadv(ioh, smgr, MAIN_FORKNUM, blkno,
+ (void *) pages, 1);
+
+ ReleaseBuffer(buf);
+
+ if (wait_complete)
+ {
+ pgaio_wref_wait(&iow);
+
+ if (ior.result.status != ARS_OK)
+ pgaio_result_report(ior.result, &ior.target_data,
+ ior.result.status == ARS_PARTIAL ? WARNING : ERROR);
+ }
+
+ relation_close(rel, NoLock);
+
+ PG_RETURN_VOID();
+}
+
+PG_FUNCTION_INFO_V1(invalidate_rel_block);
+Datum
+invalidate_rel_block(PG_FUNCTION_ARGS)
+{
+ Oid relid = PG_GETARG_OID(0);
+ BlockNumber blkno = PG_GETARG_UINT32(1);
+ Relation rel;
+ PrefetchBufferResult pr;
+ Buffer buf;
+
+ rel = relation_open(relid, AccessExclusiveLock);
+
+ /*
+ * This is a gross hack, but there's no other API exposed that allows to
+ * get a buffer ID without actually reading the block in.
+ */
+ pr = PrefetchBuffer(rel, MAIN_FORKNUM, blkno);
+ buf = pr.recent_buffer;
+
+ if (BufferIsValid(buf))
+ {
+ /* if the buffer contents aren't valid, this'll return false */
+ if (ReadRecentBuffer(rel->rd_locator, MAIN_FORKNUM, blkno, buf))
+ {
+ LockBuffer(buf, BUFFER_LOCK_EXCLUSIVE);
+ FlushOneBuffer(buf);
+ LockBuffer(buf, BUFFER_LOCK_UNLOCK);
+ ReleaseBuffer(buf);
+
+ if (!EvictUnpinnedBuffer(buf))
+ elog(ERROR, "couldn't evict");
+ }
+ }
+
+ relation_close(rel, AccessExclusiveLock);
+
+ PG_RETURN_VOID();
+}
+
+PG_FUNCTION_INFO_V1(buffer_create_toy);
+Datum
+buffer_create_toy(PG_FUNCTION_ARGS)
+{
+ Oid relid = PG_GETARG_OID(0);
+ BlockNumber blkno = PG_GETARG_UINT32(1);
+ Relation rel;
+ Buffer buf;
+
+ rel = relation_open(relid, AccessExclusiveLock);
+
+ buf = create_toy_buffer(rel, blkno);
+ ReleaseBuffer(buf);
+
+ relation_close(rel, NoLock);
+
+ PG_RETURN_INT32(buf);
+}
+
+PG_FUNCTION_INFO_V1(buffer_call_start_io);
+Datum
+buffer_call_start_io(PG_FUNCTION_ARGS)
+{
+ Buffer buf = PG_GETARG_INT32(0);
+ bool for_input = PG_GETARG_BOOL(1);
+ bool nowait = PG_GETARG_BOOL(2);
+ bool can_start;
+
+ can_start = StartBufferIO(GetBufferDescriptor(buf - 1), for_input, nowait);
+
+ /*
+ * Tor tests we don't want the resowner release preventing us from
+ * orchestrating odd scenarios.
+ */
+ if (can_start)
+ ResourceOwnerForgetBufferIO(CurrentResourceOwner,
+ buf);
+
+ ereport(LOG,
+ errmsg("buffer %d after StartBufferIO: %s",
+ buf, DebugPrintBufferRefcount(buf)),
+ errhidestmt(true), errhidecontext(true));
+
+ PG_RETURN_BOOL(can_start);
+}
+
+/*
+CREATE FUNCTION buffer_call_terminate_io(buffer int, for_input bool, failed bool, syncio bool)
+RETURNS pg_catalog.void STRICT
+AS 'MODULE_PATHNAME' LANGUAGE C;
+*/
+PG_FUNCTION_INFO_V1(buffer_call_terminate_io);
+Datum
+buffer_call_terminate_io(PG_FUNCTION_ARGS)
+{
+ Buffer buf = PG_GETARG_INT32(0);
+ bool for_input = PG_GETARG_BOOL(1);
+ bool succeed = PG_GETARG_BOOL(2);
+ bool io_error = PG_GETARG_BOOL(3);
+ bool syncio = PG_GETARG_BOOL(4);
+ bool clear_dirty = false;
+ uint32 set_flag_bits = 0;
+
+ if (io_error)
+ set_flag_bits |= BM_IO_ERROR;
+
+ if (for_input)
+ {
+ clear_dirty = false;
+
+ if (succeed)
+ set_flag_bits |= BM_VALID;
+ }
+ else
+ {
+ if (succeed)
+ clear_dirty = true;
+ }
+
+ ereport(LOG,
+ errmsg("buffer %d before TerminateBufferIO: %s",
+ buf, DebugPrintBufferRefcount(buf)),
+ errhidestmt(true), errhidecontext(true));
+
+ TerminateBufferIO(GetBufferDescriptor(buf - 1), clear_dirty, set_flag_bits,
+ false, syncio);
+
+ ereport(LOG,
+ errmsg("buffer %d after TerminateBufferIO: %s",
+ buf, DebugPrintBufferRefcount(buf)),
+ errhidestmt(true), errhidecontext(true));
+
+ PG_RETURN_VOID();
+}
+
+PG_FUNCTION_INFO_V1(handle_get);
+Datum
+handle_get(PG_FUNCTION_ARGS)
+{
+ last_handle = pgaio_io_acquire(CurrentResourceOwner, NULL);
+
+ PG_RETURN_VOID();
+}
+
+PG_FUNCTION_INFO_V1(handle_release_last);
+Datum
+handle_release_last(PG_FUNCTION_ARGS)
+{
+ if (!last_handle)
+ elog(ERROR, "no handle");
+
+ pgaio_io_release(last_handle);
+
+ PG_RETURN_VOID();
+}
+
+PG_FUNCTION_INFO_V1(handle_get_and_error);
+Datum
+handle_get_and_error(PG_FUNCTION_ARGS)
+{
+ pgaio_io_acquire(CurrentResourceOwner, NULL);
+
+ elog(ERROR, "as you command");
+ PG_RETURN_VOID();
+}
+
+PG_FUNCTION_INFO_V1(handle_get_twice);
+Datum
+handle_get_twice(PG_FUNCTION_ARGS)
+{
+ pgaio_io_acquire(CurrentResourceOwner, NULL);
+ pgaio_io_acquire(CurrentResourceOwner, NULL);
+
+ PG_RETURN_VOID();
+}
+
+PG_FUNCTION_INFO_V1(handle_get_release);
+Datum
+handle_get_release(PG_FUNCTION_ARGS)
+{
+ PgAioHandle *handle;
+
+ handle = pgaio_io_acquire(CurrentResourceOwner, NULL);
+ pgaio_io_release(handle);
+
+ PG_RETURN_VOID();
+}
+
+PG_FUNCTION_INFO_V1(batch_start);
+Datum
+batch_start(PG_FUNCTION_ARGS)
+{
+ pgaio_enter_batchmode();
+ PG_RETURN_VOID();
+}
+
+PG_FUNCTION_INFO_V1(batch_end);
+Datum
+batch_end(PG_FUNCTION_ARGS)
+{
+ pgaio_exit_batchmode();
+ PG_RETURN_VOID();
+}
+
+#ifdef USE_INJECTION_POINTS
+extern PGDLLEXPORT void inj_io_short_read(const char *name, const void *private_data);
+extern PGDLLEXPORT void inj_io_reopen(const char *name, const void *private_data);
+
+void
+inj_io_short_read(const char *name, const void *private_data)
+{
+ PgAioHandle *ioh;
+
+ elog(LOG, "short read called: %d", inj_io_error_state->enabled_short_read);
+
+ if (inj_io_error_state->enabled_short_read)
+ {
+ ioh = pgaio_inj_io_get();
+
+ if (inj_io_error_state->short_read_result_set)
+ {
+ elog(LOG, "short read, changing result from %d to %d",
+ ioh->result, inj_io_error_state->short_read_result);
+
+ ioh->result = inj_io_error_state->short_read_result;
+ }
+ }
+}
+
+void
+inj_io_reopen(const char *name, const void *private_data)
+{
+ elog(LOG, "reopen called: %d", inj_io_error_state->enabled_reopen);
+
+ if (inj_io_error_state->enabled_reopen)
+ {
+ elog(ERROR, "injection point triggering failure to reopen ");
+ }
+}
+#endif
+
+PG_FUNCTION_INFO_V1(inj_io_short_read_attach);
+Datum
+inj_io_short_read_attach(PG_FUNCTION_ARGS)
+{
+#ifdef USE_INJECTION_POINTS
+ inj_io_error_state->enabled_short_read = true;
+ inj_io_error_state->short_read_result_set = !PG_ARGISNULL(0);
+ if (inj_io_error_state->short_read_result_set)
+ inj_io_error_state->short_read_result = PG_GETARG_INT32(0);
+#else
+ elog(ERROR, "injection points not supported");
+#endif
+
+ PG_RETURN_VOID();
+}
+
+PG_FUNCTION_INFO_V1(inj_io_short_read_detach);
+Datum
+inj_io_short_read_detach(PG_FUNCTION_ARGS)
+{
+#ifdef USE_INJECTION_POINTS
+ inj_io_error_state->enabled_short_read = false;
+#else
+ elog(ERROR, "injection points not supported");
+#endif
+ PG_RETURN_VOID();
+}
+
+PG_FUNCTION_INFO_V1(inj_io_reopen_attach);
+Datum
+inj_io_reopen_attach(PG_FUNCTION_ARGS)
+{
+#ifdef USE_INJECTION_POINTS
+ inj_io_error_state->enabled_reopen = true;
+#else
+ elog(ERROR, "injection points not supported");
+#endif
+
+ PG_RETURN_VOID();
+}
+
+PG_FUNCTION_INFO_V1(inj_io_reopen_detach);
+Datum
+inj_io_reopen_detach(PG_FUNCTION_ARGS)
+{
+#ifdef USE_INJECTION_POINTS
+ inj_io_error_state->enabled_reopen = false;
+#else
+ elog(ERROR, "injection points not supported");
+#endif
+ PG_RETURN_VOID();
+}
diff --git a/src/test/modules/test_aio/test_aio.control b/src/test/modules/test_aio/test_aio.control
new file mode 100644
index 00000000000..cd91c3ed16b
--- /dev/null
+++ b/src/test/modules/test_aio/test_aio.control
@@ -0,0 +1,3 @@
+comment = 'Test code for AIO'
+default_version = '1.0'
+module_pathname = '$libdir/test_aio'
--
2.48.1.76.g4e746b1a31.dirty
[text/x-diff] v2.9-0021-aio-Implement-smgr-md-fd-write-support.patch (12.7K, ../../oav52ytakcvfnk52w5o5wvxdjavxth2tgptxt25gnfmh2pdcpk@omp3xukdu5if/22-v2.9-0021-aio-Implement-smgr-md-fd-write-support.patch)
download | inline diff:
From 5aa9adf453330e025acc3de8a29522cb54359362 Mon Sep 17 00:00:00 2001
From: Andres Freund <andres@anarazel.de>
Date: Sat, 15 Mar 2025 12:29:58 -0400
Subject: [PATCH v2.9 21/30] aio: Implement smgr/md/fd write support
TODO:
- register_dirty_segment_aio() can error out in edge cases
---
src/include/storage/fd.h | 1 +
src/include/storage/md.h | 5 +
src/include/storage/smgr.h | 5 +
src/backend/storage/aio/aio_callback.c | 1 +
src/backend/storage/file/fd.c | 28 ++++
src/backend/storage/smgr/md.c | 189 +++++++++++++++++++++++++
src/backend/storage/smgr/smgr.c | 22 +++
7 files changed, 251 insertions(+)
diff --git a/src/include/storage/fd.h b/src/include/storage/fd.h
index b77d8e5e30e..2cc7c5a4761 100644
--- a/src/include/storage/fd.h
+++ b/src/include/storage/fd.h
@@ -112,6 +112,7 @@ extern int FilePrefetch(File file, off_t offset, off_t amount, uint32 wait_event
extern ssize_t FileReadV(File file, const struct iovec *iov, int iovcnt, off_t offset, uint32 wait_event_info);
extern ssize_t FileWriteV(File file, const struct iovec *iov, int iovcnt, off_t offset, uint32 wait_event_info);
extern int FileStartReadV(struct PgAioHandle *ioh, File file, int iovcnt, off_t offset, uint32 wait_event_info);
+extern int FileStartWriteV(struct PgAioHandle *ioh, File file, int iovcnt, off_t offset, uint32 wait_event_info);
extern int FileSync(File file, uint32 wait_event_info);
extern int FileZero(File file, off_t offset, off_t amount, uint32 wait_event_info);
extern int FileFallocate(File file, off_t offset, off_t amount, uint32 wait_event_info);
diff --git a/src/include/storage/md.h b/src/include/storage/md.h
index 9d7131eff43..47ae6c36c94 100644
--- a/src/include/storage/md.h
+++ b/src/include/storage/md.h
@@ -21,6 +21,7 @@
#include "storage/sync.h"
extern const PgAioHandleCallbacks aio_md_readv_cb;
+extern const PgAioHandleCallbacks aio_md_writev_cb;
/* md storage manager functionality */
extern void mdinit(void);
@@ -45,6 +46,10 @@ extern void mdstartreadv(PgAioHandle *ioh,
extern void mdwritev(SMgrRelation reln, ForkNumber forknum,
BlockNumber blocknum,
const void **buffers, BlockNumber nblocks, bool skipFsync);
+extern void mdstartwritev(PgAioHandle *ioh,
+ SMgrRelation reln, ForkNumber forknum,
+ BlockNumber blocknum,
+ const void **buffers, BlockNumber nblocks, bool skipFsync);
extern void mdwriteback(SMgrRelation reln, ForkNumber forknum,
BlockNumber blocknum, BlockNumber nblocks);
extern BlockNumber mdnblocks(SMgrRelation reln, ForkNumber forknum);
diff --git a/src/include/storage/smgr.h b/src/include/storage/smgr.h
index 856ebcda350..f00b3763ac9 100644
--- a/src/include/storage/smgr.h
+++ b/src/include/storage/smgr.h
@@ -108,6 +108,11 @@ extern void smgrwritev(SMgrRelation reln, ForkNumber forknum,
BlockNumber blocknum,
const void **buffers, BlockNumber nblocks,
bool skipFsync);
+extern void smgrstartwritev(PgAioHandle *ioh,
+ SMgrRelation reln, ForkNumber forknum,
+ BlockNumber blocknum,
+ const void **buffers, BlockNumber nblocks,
+ bool skipFsync);
extern void smgrwriteback(SMgrRelation reln, ForkNumber forknum,
BlockNumber blocknum, BlockNumber nblocks);
extern BlockNumber smgrnblocks(SMgrRelation reln, ForkNumber forknum);
diff --git a/src/backend/storage/aio/aio_callback.c b/src/backend/storage/aio/aio_callback.c
index 2537b1914d8..d1ea1ee145b 100644
--- a/src/backend/storage/aio/aio_callback.c
+++ b/src/backend/storage/aio/aio_callback.c
@@ -41,6 +41,7 @@ static const PgAioHandleCallbacksEntry aio_handle_cbs[] = {
CALLBACK_ENTRY(PGAIO_HCB_INVALID, aio_invalid_cb),
CALLBACK_ENTRY(PGAIO_HCB_MD_READV, aio_md_readv_cb),
+ CALLBACK_ENTRY(PGAIO_HCB_MD_WRITEV, aio_md_writev_cb),
CALLBACK_ENTRY(PGAIO_HCB_SHARED_BUFFER_READV, aio_shared_buffer_readv_cb),
diff --git a/src/backend/storage/file/fd.c b/src/backend/storage/file/fd.c
index 34588ed5167..46cbcb2b688 100644
--- a/src/backend/storage/file/fd.c
+++ b/src/backend/storage/file/fd.c
@@ -2346,6 +2346,34 @@ retry:
return returnCode;
}
+int
+FileStartWriteV(PgAioHandle *ioh, File file,
+ int iovcnt, off_t offset,
+ uint32 wait_event_info)
+{
+ int returnCode;
+ Vfd *vfdP;
+
+ Assert(FileIsValid(file));
+
+ DO_DB(elog(LOG, "FileStartWriteV: %d (%s) " INT64_FORMAT " %d",
+ file, VfdCache[file].fileName,
+ (int64) offset,
+ iovcnt));
+
+ returnCode = FileAccess(file);
+ if (returnCode < 0)
+ return returnCode;
+
+ vfdP = &VfdCache[file];
+
+ /* FIXME: think about / reimplement temp_file_limit */
+
+ pgaio_io_prep_writev(ioh, vfdP->fd, iovcnt, offset);
+
+ return 0;
+}
+
int
FileSync(File file, uint32 wait_event_info)
{
diff --git a/src/backend/storage/smgr/md.c b/src/backend/storage/smgr/md.c
index 11d4d5a7aea..a61fc14805e 100644
--- a/src/backend/storage/smgr/md.c
+++ b/src/backend/storage/smgr/md.c
@@ -155,12 +155,19 @@ static BlockNumber _mdnblocks(SMgrRelation reln, ForkNumber forknum,
static PgAioResult md_readv_complete(PgAioHandle *ioh, PgAioResult prior_result, uint8 cb_data);
static void md_readv_report(PgAioResult result, const PgAioTargetData *target_data, int elevel);
+static PgAioResult md_writev_complete(PgAioHandle *ioh, PgAioResult prior_result, uint8 cb_data);
+static void md_writev_report(PgAioResult result, const PgAioTargetData *target_data, int elevel);
const PgAioHandleCallbacks aio_md_readv_cb = {
.complete_shared = md_readv_complete,
.report = md_readv_report,
};
+const PgAioHandleCallbacks aio_md_writev_cb = {
+ .complete_shared = md_writev_complete,
+ .report = md_writev_report,
+};
+
static inline int
_mdfd_open_flags(void)
@@ -1107,6 +1114,56 @@ mdwritev(SMgrRelation reln, ForkNumber forknum, BlockNumber blocknum,
}
}
+/*
+ * mdstartwritev() -- Asynchronous version of mdrwritev().
+ */
+void
+mdstartwritev(PgAioHandle *ioh,
+ SMgrRelation reln, ForkNumber forknum, BlockNumber blocknum,
+ const void **buffers, BlockNumber nblocks, bool skipFsync)
+{
+ off_t seekpos;
+ MdfdVec *v;
+ BlockNumber nblocks_this_segment;
+ struct iovec *iov;
+ int iovcnt;
+
+ v = _mdfd_getseg(reln, forknum, blocknum, false,
+ EXTENSION_FAIL | EXTENSION_CREATE_RECOVERY);
+
+ seekpos = (off_t) BLCKSZ * (blocknum % ((BlockNumber) RELSEG_SIZE));
+
+ Assert(seekpos < (off_t) BLCKSZ * RELSEG_SIZE);
+
+ nblocks_this_segment =
+ Min(nblocks,
+ RELSEG_SIZE - (blocknum % ((BlockNumber) RELSEG_SIZE)));
+
+ if (nblocks_this_segment != nblocks)
+ elog(ERROR, "write crossing segment boundary");
+
+ iovcnt = pgaio_io_get_iovec(ioh, &iov);
+
+ Assert(nblocks <= iovcnt);
+
+ iovcnt = buffers_to_iovec(iov, unconstify(void **, buffers), nblocks_this_segment);
+
+ Assert(iovcnt <= nblocks_this_segment);
+
+ if (!(io_direct_flags & IO_DIRECT_DATA))
+ pgaio_io_set_flag(ioh, PGAIO_HF_BUFFERED);
+
+ pgaio_io_set_target_smgr(ioh,
+ reln,
+ forknum,
+ blocknum,
+ nblocks,
+ skipFsync);
+ pgaio_io_register_callbacks(ioh, PGAIO_HCB_MD_WRITEV, 0);
+
+ FileStartWriteV(ioh, v->mdfd_vfd, iovcnt, seekpos, WAIT_EVENT_DATA_FILE_WRITE);
+}
+
/*
* mdwriteback() -- Tell the kernel to write pages back to storage.
@@ -1495,6 +1552,40 @@ register_dirty_segment(SMgrRelation reln, ForkNumber forknum, MdfdVec *seg)
}
}
+/*
+ * Like register_dirty_segment(), except for use by AIO. In the completion
+ * callback we don't have access to the MdfdVec (the completion callback might
+ * be executed in a different backend than the issuing backend), therefore we
+ * have to implement this slightly differently.
+ */
+static void
+register_dirty_segment_aio(RelFileLocator locator, ForkNumber forknum, uint64 segno)
+{
+ FileTag tag;
+
+ INIT_MD_FILETAG(tag, locator, forknum, segno);
+
+ /*
+ * Can't block here waiting for checkpointer to accept our sync request,
+ * as checkpointer might be waiting for this AIO to finish if offloaded to
+ * a worker.
+ */
+ if (!RegisterSyncRequest(&tag, SYNC_REQUEST, false /* retryOnError */ ))
+ {
+ char path[MAXPGPATH];
+
+ ereport(DEBUG1,
+ (errmsg_internal("could not forward fsync request because request queue is full")));
+
+ /* reuse mdsyncfiletag() to avoid duplicating code */
+ if (mdsyncfiletag(&tag, path))
+ ereport(data_sync_elevel(ERROR),
+ (errcode_for_file_access(),
+ errmsg("could not fsync file \"%s\": %m",
+ path)));
+ }
+}
+
/*
* register_unlink_segment() -- Schedule a file to be deleted after next checkpoint
*/
@@ -2017,3 +2108,101 @@ md_readv_report(PgAioResult result, const PgAioTargetData *td, int elevel)
td->smgr.nblocks * (size_t) BLCKSZ));
}
}
+
+/*
+ * AIO completion callback for mdstartwritev().
+ */
+static PgAioResult
+md_writev_complete(PgAioHandle *ioh, PgAioResult prior_result, uint8 cb_data)
+{
+ PgAioTargetData *td = pgaio_io_get_target_data(ioh);
+ PgAioResult result = prior_result;
+
+ if (prior_result.result < 0)
+ {
+ result.status = ARS_ERROR;
+ result.id = PGAIO_HCB_MD_WRITEV;
+ /* For "hard" errors, track the error number in error_data */
+ result.error_data = -prior_result.result;
+ result.result = 0;
+
+ pgaio_result_report(result, td, LOG);
+
+ return result;
+ }
+
+ /*
+ * The smgr API operates in blocks, therefore convert the result from
+ * bytes to blocks.
+ */
+ result.result /= BLCKSZ;
+
+ if (result.result == 0)
+ {
+ /* consider 0 blocks written a failure */
+ result.status = ARS_ERROR;
+ result.id = PGAIO_HCB_MD_WRITEV;
+ result.error_data = 0;
+
+ pgaio_result_report(result, td, LOG);
+
+ return result;
+ }
+
+ if (result.status != ARS_ERROR &&
+ result.result < td->smgr.nblocks)
+ {
+ /* partial writes should be retried at upper level */
+ result.status = ARS_PARTIAL;
+ result.id = PGAIO_HCB_MD_WRITEV;
+ }
+
+ if (!td->smgr.skip_fsync)
+ register_dirty_segment_aio(td->smgr.rlocator, td->smgr.forkNum,
+ td->smgr.blockNum / ((BlockNumber) RELSEG_SIZE));
+
+ return result;
+}
+
+/*
+ * AIO error reporting callback for mdstartwritev().
+ */
+static void
+md_writev_report(PgAioResult result, const PgAioTargetData *td, int elevel)
+{
+ RelPathStr path;
+
+ path = relpathbackend(td->smgr.rlocator,
+ td->smgr.is_temp ? MyProcNumber : INVALID_PROC_NUMBER,
+ td->smgr.forkNum);
+
+ if (result.error_data != 0)
+ {
+ errno = result.error_data; /* for errcode_for_file_access() */
+
+ ereport(elevel,
+ errcode_for_file_access(),
+ errmsg("could not write blocks %u..%u in file \"%s\": %m",
+ td->smgr.blockNum,
+ td->smgr.blockNum + td->smgr.nblocks,
+ path.str)
+ );
+ }
+ else
+ {
+ /*
+ * NB: This will typically only be output in debug messages, while
+ * retrying a partial IO.
+ */
+ ereport(elevel,
+ errcode(ERRCODE_DATA_CORRUPTED),
+ errmsg("could not write blocks %u..%u in file \"%s\": wrote only %zu of %zu bytes",
+ td->smgr.blockNum,
+ td->smgr.blockNum + td->smgr.nblocks - 1,
+ path.str,
+ result.result * (size_t) BLCKSZ,
+ td->smgr.nblocks * (size_t) BLCKSZ
+ )
+ );
+ }
+}
diff --git a/src/backend/storage/smgr/smgr.c b/src/backend/storage/smgr/smgr.c
index 06b2acf8fea..86be5cb61bc 100644
--- a/src/backend/storage/smgr/smgr.c
+++ b/src/backend/storage/smgr/smgr.c
@@ -112,6 +112,11 @@ typedef struct f_smgr
BlockNumber blocknum,
const void **buffers, BlockNumber nblocks,
bool skipFsync);
+ void (*smgr_startwritev) (PgAioHandle *ioh,
+ SMgrRelation reln, ForkNumber forknum,
+ BlockNumber blocknum,
+ const void **buffers, BlockNumber nblocks,
+ bool skipFsync);
void (*smgr_writeback) (SMgrRelation reln, ForkNumber forknum,
BlockNumber blocknum, BlockNumber nblocks);
BlockNumber (*smgr_nblocks) (SMgrRelation reln, ForkNumber forknum);
@@ -139,6 +144,7 @@ static const f_smgr smgrsw[] = {
.smgr_readv = mdreadv,
.smgr_startreadv = mdstartreadv,
.smgr_writev = mdwritev,
+ .smgr_startwritev = mdstartwritev,
.smgr_writeback = mdwriteback,
.smgr_nblocks = mdnblocks,
.smgr_truncate = mdtruncate,
@@ -769,6 +775,22 @@ smgrwritev(SMgrRelation reln, ForkNumber forknum, BlockNumber blocknum,
RESUME_INTERRUPTS();
}
+/*
+ * smgrstartwritev() -- asynchronous version of smgrwritev()
+ *
+ * This starts an asynchronous writev IO with the IO handle `ioh`. Other than
+ * `ioh` all parameters are the same as smgrwritev().
+ */
+void
+smgrstartwritev(PgAioHandle *ioh,
+ SMgrRelation reln, ForkNumber forknum, BlockNumber blocknum,
+ const void **buffers, BlockNumber nblocks, bool skipFsync)
+{
+ smgrsw[reln->smgr_which].smgr_startwritev(ioh,
+ reln, forknum, blocknum, buffers,
+ nblocks, skipFsync);
+}
+
/*
* smgrwriteback() -- Trigger kernel writeback for the supplied range of
* blocks.
--
2.48.1.76.g4e746b1a31.dirty
[text/x-diff] v2.9-0022-aio-Add-bounce-buffers.patch (23.1K, ../../oav52ytakcvfnk52w5o5wvxdjavxth2tgptxt25gnfmh2pdcpk@omp3xukdu5if/23-v2.9-0022-aio-Add-bounce-buffers.patch)
download | inline diff:
From 7a16b6dc189cb8ebde7d1054ab7633d9776f3a93 Mon Sep 17 00:00:00 2001
From: Andres Freund <andres@anarazel.de>
Date: Sat, 15 Mar 2025 12:29:58 -0400
Subject: [PATCH v2.9 22/30] aio: Add bounce buffers
---
src/include/storage/aio.h | 15 ++
src/include/storage/aio_internal.h | 34 ++++
src/include/storage/aio_types.h | 2 +
src/include/utils/resowner.h | 2 +
src/backend/storage/aio/README.md | 27 +++
src/backend/storage/aio/aio.c | 178 ++++++++++++++++++
src/backend/storage/aio/aio_init.c | 123 ++++++++++++
src/backend/utils/misc/guc_tables.c | 13 ++
src/backend/utils/misc/postgresql.conf.sample | 2 +
src/backend/utils/resowner/resowner.c | 25 ++-
src/test/modules/test_aio/test_aio--1.0.sql | 21 +++
src/test/modules/test_aio/test_aio.c | 55 ++++++
src/tools/pgindent/typedefs.list | 1 +
13 files changed, 496 insertions(+), 2 deletions(-)
diff --git a/src/include/storage/aio.h b/src/include/storage/aio.h
index c0db8f9754f..c673627a358 100644
--- a/src/include/storage/aio.h
+++ b/src/include/storage/aio.h
@@ -344,6 +344,20 @@ extern bool pgaio_have_staged(void);
+/* --------------------------------------------------------------------------------
+ * Bounce Buffers
+ * --------------------------------------------------------------------------------
+ */
+
+extern PgAioBounceBuffer *pgaio_bounce_buffer_get(void);
+extern void pgaio_io_assoc_bounce_buffer(PgAioHandle *ioh, PgAioBounceBuffer *bb);
+extern uint32 pgaio_bounce_buffer_id(PgAioBounceBuffer *bb);
+extern void pgaio_bounce_buffer_release(PgAioBounceBuffer *bb);
+extern char *pgaio_bounce_buffer_buffer(PgAioBounceBuffer *bb);
+extern void pgaio_bounce_buffer_release_resowner(struct dlist_node *bb_node, bool on_error);
+
+
+
/* --------------------------------------------------------------------------------
* Other
* --------------------------------------------------------------------------------
@@ -356,6 +370,7 @@ extern void pgaio_closing_fd(int fd);
/* GUCs */
extern PGDLLIMPORT int io_method;
extern PGDLLIMPORT int io_max_concurrency;
+extern PGDLLIMPORT int io_bounce_buffers;
#endif /* AIO_H */
diff --git a/src/include/storage/aio_internal.h b/src/include/storage/aio_internal.h
index 98ae66921f5..166a66f7062 100644
--- a/src/include/storage/aio_internal.h
+++ b/src/include/storage/aio_internal.h
@@ -122,6 +122,12 @@ struct PgAioHandle
/* raw result of the IO operation */
int32 result;
+ /*
+ * List of bounce_buffers owned by IO. It would suffice to use an index
+ * based linked list here.
+ */
+ slist_head bounce_buffers;
+
/**
* In which list the handle is registered, depends on the state:
* - IDLE, in per-backend list
@@ -177,11 +183,24 @@ struct PgAioHandle
};
+/* typedef is in aio_types.h */
+struct PgAioBounceBuffer
+{
+ slist_node node;
+ struct ResourceOwnerData *resowner;
+ dlist_node resowner_node;
+ char *buffer;
+};
+
+
typedef struct PgAioBackend
{
/* index into PgAioCtl->io_handles */
uint32 io_handle_off;
+ /* index into PgAioCtl->bounce_buffers */
+ uint32 bounce_buffers_off;
+
/* IO Handles that currently are not used */
dclist_head idle_ios;
@@ -212,6 +231,12 @@ typedef struct PgAioBackend
* IOs being appended at the end.
*/
dclist_head in_flight_ios;
+
+ /* Bounce Buffers that currently are not used */
+ slist_head idle_bbs;
+
+ /* see handed_out_io */
+ PgAioBounceBuffer *handed_out_bb;
} PgAioBackend;
@@ -239,6 +264,15 @@ typedef struct PgAioCtl
uint32 io_handle_count;
PgAioHandle *io_handles;
+
+ /*
+ * To perform AIO on buffers that are not located in shared memory (either
+ * because they are not in shared memory or because we need to operate on
+ * a copy, as e.g. the case for writes when checksums are in use)
+ */
+ uint32 bounce_buffers_count;
+ PgAioBounceBuffer *bounce_buffers;
+ char *bounce_buffers_data;
} PgAioCtl;
diff --git a/src/include/storage/aio_types.h b/src/include/storage/aio_types.h
index b0b9a2a5c97..09968e17855 100644
--- a/src/include/storage/aio_types.h
+++ b/src/include/storage/aio_types.h
@@ -118,4 +118,6 @@ typedef struct PgAioReturn
} PgAioReturn;
+typedef struct PgAioBounceBuffer PgAioBounceBuffer;
+
#endif /* AIO_TYPES_H */
diff --git a/src/include/utils/resowner.h b/src/include/utils/resowner.h
index aede4bfc820..7e2ec224169 100644
--- a/src/include/utils/resowner.h
+++ b/src/include/utils/resowner.h
@@ -168,5 +168,7 @@ extern void ResourceOwnerForgetLock(ResourceOwner owner, struct LOCALLOCK *local
struct dlist_node;
extern void ResourceOwnerRememberAioHandle(ResourceOwner owner, struct dlist_node *ioh_node);
extern void ResourceOwnerForgetAioHandle(ResourceOwner owner, struct dlist_node *ioh_node);
+extern void ResourceOwnerRememberAioBounceBuffer(ResourceOwner owner, struct dlist_node *bb_node);
+extern void ResourceOwnerForgetAioBounceBuffer(ResourceOwner owner, struct dlist_node *bb_node);
#endif /* RESOWNER_H */
diff --git a/src/backend/storage/aio/README.md b/src/backend/storage/aio/README.md
index b7dfb80b4b2..bb0bfc71ea6 100644
--- a/src/backend/storage/aio/README.md
+++ b/src/backend/storage/aio/README.md
@@ -405,6 +405,33 @@ shared memory no less!), completion callbacks instead have to encode errors in
a more compact format that can be converted into an error message.
+### AIO Bounce Buffers
+
+For some uses of AIO there is no convenient memory location as the source /
+destination of an AIO. E.g. when data checksums are enabled, writes from
+shared buffers currently cannot be done directly from shared buffers, as a
+shared buffer lock still allows some modification, e.g., for hint bits(see
+`FlushBuffer()`). If the write were done in-place, such modifications can
+cause the checksum to fail.
+
+For synchronous IO this is solved by copying the buffer to separate memory
+before computing the checksum and using that copy as the source buffer for the
+AIO.
+
+However, for AIO that is not a workable solution:
+- Instead of a single buffer many buffers are required, as many IOs might be
+ in flight
+- When using the [worker method](#worker), the source/target of IO needs to be
+ in shared memory, otherwise the workers won't be able to access the memory.
+
+The AIO subsystem addresses this by providing a limited number of bounce
+buffers that can be used as the source / target for IO. A bounce buffer be
+acquired with `pgaio_bounce_buffer_get()` and multiple bounce buffers can be
+associated with an AIO Handle with `pgaio_io_assoc_bounce_buffer()`.
+
+Bounce buffers are automatically released when the IO completes.
+
+
## Helpers
Using the low-level AIO API introduces too much complexity to do so all over
diff --git a/src/backend/storage/aio/aio.c b/src/backend/storage/aio/aio.c
index a0504ee84ee..46d83da60f7 100644
--- a/src/backend/storage/aio/aio.c
+++ b/src/backend/storage/aio/aio.c
@@ -62,6 +62,8 @@ static PgAioHandle *pgaio_io_from_wref(PgAioWaitRef *iow, uint64 *ref_generation
static const char *pgaio_io_state_get_name(PgAioHandleState s);
static void pgaio_io_wait(PgAioHandle *ioh, uint64 ref_generation);
+static void pgaio_bounce_buffer_wait_for_free(void);
+
/* Options for io_method. */
const struct config_enum_entry io_method_options[] = {
@@ -76,6 +78,7 @@ const struct config_enum_entry io_method_options[] = {
/* GUCs */
int io_method = DEFAULT_IO_METHOD;
int io_max_concurrency = -1;
+int io_bounce_buffers = -1;
/* global control for AIO */
PgAioCtl *pgaio_ctl;
@@ -662,6 +665,21 @@ pgaio_io_reclaim(PgAioHandle *ioh)
}
}
+ /* reclaim all associated bounce buffers */
+ if (!slist_is_empty(&ioh->bounce_buffers))
+ {
+ slist_mutable_iter it;
+
+ slist_foreach_modify(it, &ioh->bounce_buffers)
+ {
+ PgAioBounceBuffer *bb = slist_container(PgAioBounceBuffer, node, it.cur);
+
+ slist_delete_current(&it);
+
+ slist_push_head(&pgaio_my_backend->idle_bbs, &bb->node);
+ }
+ }
+
if (ioh->resowner)
{
ResourceOwnerForgetAioHandle(ioh->resowner, &ioh->resowner_node);
@@ -1044,6 +1062,166 @@ pgaio_submit_staged(void)
+/* --------------------------------------------------------------------------------
+ * Functions primarily related to PgAioBounceBuffer
+ * --------------------------------------------------------------------------------
+ */
+
+PgAioBounceBuffer *
+pgaio_bounce_buffer_get(void)
+{
+ PgAioBounceBuffer *bb = NULL;
+ slist_node *node;
+
+ if (pgaio_my_backend->handed_out_bb != NULL)
+ elog(ERROR, "can only hand out one BB");
+
+ /*
+ * XXX: It probably is not a good idea to have bounce buffers be per
+ * backend, that's a fair bit of memory.
+ */
+ if (slist_is_empty(&pgaio_my_backend->idle_bbs))
+ {
+ pgaio_bounce_buffer_wait_for_free();
+ }
+
+ node = slist_pop_head_node(&pgaio_my_backend->idle_bbs);
+ bb = slist_container(PgAioBounceBuffer, node, node);
+
+ pgaio_my_backend->handed_out_bb = bb;
+
+ bb->resowner = CurrentResourceOwner;
+ ResourceOwnerRememberAioBounceBuffer(bb->resowner, &bb->resowner_node);
+
+ return bb;
+}
+
+void
+pgaio_io_assoc_bounce_buffer(PgAioHandle *ioh, PgAioBounceBuffer *bb)
+{
+ if (pgaio_my_backend->handed_out_bb != bb)
+ elog(ERROR, "can only assign handed out BB");
+ pgaio_my_backend->handed_out_bb = NULL;
+
+ /*
+ * There can be many bounce buffers assigned in case of vectorized IOs.
+ */
+ slist_push_head(&ioh->bounce_buffers, &bb->node);
+
+ /* once associated with an IO, the IO has ownership */
+ ResourceOwnerForgetAioBounceBuffer(bb->resowner, &bb->resowner_node);
+ bb->resowner = NULL;
+}
+
+uint32
+pgaio_bounce_buffer_id(PgAioBounceBuffer *bb)
+{
+ return bb - pgaio_ctl->bounce_buffers;
+}
+
+void
+pgaio_bounce_buffer_release(PgAioBounceBuffer *bb)
+{
+ if (pgaio_my_backend->handed_out_bb != bb)
+ elog(ERROR, "can only release handed out BB");
+
+ slist_push_head(&pgaio_my_backend->idle_bbs, &bb->node);
+ pgaio_my_backend->handed_out_bb = NULL;
+
+ ResourceOwnerForgetAioBounceBuffer(bb->resowner, &bb->resowner_node);
+ bb->resowner = NULL;
+}
+
+void
+pgaio_bounce_buffer_release_resowner(dlist_node *bb_node, bool on_error)
+{
+ PgAioBounceBuffer *bb = dlist_container(PgAioBounceBuffer, resowner_node, bb_node);
+
+ Assert(bb->resowner);
+
+ if (!on_error)
+ elog(WARNING, "leaked AIO bounce buffer");
+
+ pgaio_bounce_buffer_release(bb);
+}
+
+char *
+pgaio_bounce_buffer_buffer(PgAioBounceBuffer *bb)
+{
+ return bb->buffer;
+}
+
+static void
+pgaio_bounce_buffer_wait_for_free(void)
+{
+ static uint32 lastpos = 0;
+
+ if (pgaio_my_backend->num_staged_ios > 0)
+ {
+ pgaio_debug(DEBUG2, "submitting %d, while acquiring free bb",
+ pgaio_my_backend->num_staged_ios);
+ pgaio_submit_staged();
+ }
+
+ for (uint32 i = lastpos; i < lastpos + io_max_concurrency; i++)
+ {
+ uint32 thisoff = pgaio_my_backend->io_handle_off + (i % io_max_concurrency);
+ PgAioHandle *ioh = &pgaio_ctl->io_handles[thisoff];
+
+ switch (ioh->state)
+ {
+ case PGAIO_HS_IDLE:
+ case PGAIO_HS_HANDED_OUT:
+ continue;
+ case PGAIO_HS_DEFINED: /* should have been submitted above */
+ case PGAIO_HS_STAGED:
+ elog(ERROR, "shouldn't get here with io:%d in state %d",
+ pgaio_io_get_id(ioh), ioh->state);
+ break;
+ case PGAIO_HS_COMPLETED_IO:
+ case PGAIO_HS_SUBMITTED:
+ if (!slist_is_empty(&ioh->bounce_buffers))
+ {
+ pgaio_debug_io(DEBUG2, ioh,
+ "waiting for IO to reclaim BB with %d in flight",
+ dclist_count(&pgaio_my_backend->in_flight_ios));
+
+ /* see comment in pgaio_io_wait_for_free() about raciness */
+ pgaio_io_wait(ioh, ioh->generation);
+
+ if (slist_is_empty(&pgaio_my_backend->idle_bbs))
+ elog(WARNING, "empty after wait");
+
+ if (!slist_is_empty(&pgaio_my_backend->idle_bbs))
+ {
+ lastpos = i;
+ return;
+ }
+ }
+ break;
+ case PGAIO_HS_COMPLETED_SHARED:
+ case PGAIO_HS_COMPLETED_LOCAL:
+ /* reclaim */
+ pgaio_io_reclaim(ioh);
+
+ if (!slist_is_empty(&pgaio_my_backend->idle_bbs))
+ {
+ lastpos = i;
+ return;
+ }
+ break;
+ }
+ }
+
+ /*
+ * The submission above could have caused the IO to complete at any time.
+ */
+ if (slist_is_empty(&pgaio_my_backend->idle_bbs))
+ elog(PANIC, "no more bbs");
+}
+
+
+
/* --------------------------------------------------------------------------------
* Other
* --------------------------------------------------------------------------------
diff --git a/src/backend/storage/aio/aio_init.c b/src/backend/storage/aio/aio_init.c
index 4e405ce7ca8..b9732d989ff 100644
--- a/src/backend/storage/aio/aio_init.c
+++ b/src/backend/storage/aio/aio_init.c
@@ -93,6 +93,32 @@ AioHandleDataShmemSize(void)
io_max_concurrency));
}
+static Size
+AioBounceBufferDescShmemSize(void)
+{
+ Size sz;
+
+ /* PgAioBounceBuffer itself */
+ sz = mul_size(sizeof(PgAioBounceBuffer),
+ mul_size(AioProcs(), io_bounce_buffers));
+
+ return sz;
+}
+
+static Size
+AioBounceBufferDataShmemSize(void)
+{
+ Size sz;
+
+ /* and the associated buffer */
+ sz = mul_size(BLCKSZ,
+ mul_size(io_bounce_buffers, AioProcs()));
+ /* memory for alignment */
+ sz += BLCKSZ;
+
+ return sz;
+}
+
/*
* Choose a suitable value for io_max_concurrency.
*
@@ -118,6 +144,33 @@ AioChooseMaxConcurrency(void)
return Min(max_proportional_pins, 64);
}
+/*
+ * Choose a suitable value for io_bounce_buffers.
+ *
+ * It's very unlikely to be useful to allocate more bounce buffers for each
+ * backend than the backend is allowed to pin. Additionally, bounce buffers
+ * currently are used for writes, it seems very uncommon for more than 10% of
+ * shared_buffers to be written out concurrently.
+ *
+ * XXX: This quickly can take up significant amounts of memory, the logic
+ * should probably fine tuned.
+ */
+static int
+AioChooseBounceBuffers(void)
+{
+ uint32 max_backends;
+ int max_proportional_pins;
+
+ /* Similar logic to LimitAdditionalPins() */
+ max_backends = MaxBackends + NUM_AUXILIARY_PROCS;
+ max_proportional_pins = (NBuffers / 10) / max_backends;
+
+ max_proportional_pins = Max(max_proportional_pins, 1);
+
+ /* apply upper limit */
+ return Min(max_proportional_pins, 256);
+}
+
Size
AioShmemSize(void)
{
@@ -141,11 +194,31 @@ AioShmemSize(void)
PGC_S_OVERRIDE);
}
+
+ /*
+ * If io_bounce_buffers is -1, we automatically choose a suitable value.
+ *
+ * See also comment above.
+ */
+ if (io_bounce_buffers == -1)
+ {
+ char buf[32];
+
+ snprintf(buf, sizeof(buf), "%d", AioChooseBounceBuffers());
+ SetConfigOption("io_bounce_buffers", buf, PGC_POSTMASTER,
+ PGC_S_DYNAMIC_DEFAULT);
+ if (io_bounce_buffers == -1) /* failed to apply it? */
+ SetConfigOption("io_bounce_buffers", buf, PGC_POSTMASTER,
+ PGC_S_OVERRIDE);
+ }
+
sz = add_size(sz, AioCtlShmemSize());
sz = add_size(sz, AioBackendShmemSize());
sz = add_size(sz, AioHandleShmemSize());
sz = add_size(sz, AioHandleIOVShmemSize());
sz = add_size(sz, AioHandleDataShmemSize());
+ sz = add_size(sz, AioBounceBufferDescShmemSize());
+ sz = add_size(sz, AioBounceBufferDataShmemSize());
/* Reserve space for method specific resources. */
if (pgaio_method_ops->shmem_size)
@@ -161,6 +234,9 @@ AioShmemInit(void)
uint32 io_handle_off = 0;
uint32 iovec_off = 0;
uint32 per_backend_iovecs = io_max_concurrency * PG_IOV_MAX;
+ uint32 bounce_buffers_off = 0;
+ uint32 per_backend_bb = io_bounce_buffers;
+ char *bounce_buffers_data;
pgaio_ctl = (PgAioCtl *)
ShmemInitStruct("AioCtl", AioCtlShmemSize(), &found);
@@ -172,6 +248,7 @@ AioShmemInit(void)
pgaio_ctl->io_handle_count = AioProcs() * io_max_concurrency;
pgaio_ctl->iovec_count = AioProcs() * per_backend_iovecs;
+ pgaio_ctl->bounce_buffers_count = AioProcs() * per_backend_bb;
pgaio_ctl->backend_state = (PgAioBackend *)
ShmemInitStruct("AioBackend", AioBackendShmemSize(), &found);
@@ -184,6 +261,40 @@ AioShmemInit(void)
pgaio_ctl->handle_data = (uint64 *)
ShmemInitStruct("AioHandleData", AioHandleDataShmemSize(), &found);
+ pgaio_ctl->bounce_buffers = (PgAioBounceBuffer *)
+ ShmemInitStruct("AioBounceBufferDesc", AioBounceBufferDescShmemSize(),
+ &found);
+
+ bounce_buffers_data =
+ ShmemInitStruct("AioBounceBufferData", AioBounceBufferDataShmemSize(),
+ &found);
+ bounce_buffers_data =
+ (char *) TYPEALIGN(BLCKSZ, (uintptr_t) bounce_buffers_data);
+ pgaio_ctl->bounce_buffers_data = bounce_buffers_data;
+
+
+ /* Initialize IO handles. */
+ for (uint64 i = 0; i < pgaio_ctl->io_handle_count; i++)
+ {
+ PgAioHandle *ioh = &pgaio_ctl->io_handles[i];
+
+ ioh->op = PGAIO_OP_INVALID;
+ ioh->target = PGAIO_TID_INVALID;
+ ioh->state = PGAIO_HS_IDLE;
+
+ slist_init(&ioh->bounce_buffers);
+ }
+
+ /* Initialize Bounce Buffers. */
+ for (uint64 i = 0; i < pgaio_ctl->bounce_buffers_count; i++)
+ {
+ PgAioBounceBuffer *bb = &pgaio_ctl->bounce_buffers[i];
+
+ bb->buffer = bounce_buffers_data;
+ bounce_buffers_data += BLCKSZ;
+ }
+
+
for (int procno = 0; procno < AioProcs(); procno++)
{
PgAioBackend *bs = &pgaio_ctl->backend_state[procno];
@@ -191,9 +302,13 @@ AioShmemInit(void)
bs->io_handle_off = io_handle_off;
io_handle_off += io_max_concurrency;
+ bs->bounce_buffers_off = bounce_buffers_off;
+ bounce_buffers_off += per_backend_bb;
+
dclist_init(&bs->idle_ios);
memset(bs->staged_ios, 0, sizeof(PgAioHandle *) * PGAIO_SUBMIT_BATCH_SIZE);
dclist_init(&bs->in_flight_ios);
+ slist_init(&bs->idle_bbs);
/* initialize per-backend IOs */
for (int i = 0; i < io_max_concurrency; i++)
@@ -215,6 +330,14 @@ AioShmemInit(void)
dclist_push_tail(&bs->idle_ios, &ioh->node);
iovec_off += PG_IOV_MAX;
}
+
+ /* initialize per-backend bounce buffers */
+ for (int i = 0; i < per_backend_bb; i++)
+ {
+ PgAioBounceBuffer *bb = &pgaio_ctl->bounce_buffers[bs->bounce_buffers_off + i];
+
+ slist_push_head(&bs->idle_bbs, &bb->node);
+ }
}
out:
diff --git a/src/backend/utils/misc/guc_tables.c b/src/backend/utils/misc/guc_tables.c
index 4cc19bef686..7290793c9a1 100644
--- a/src/backend/utils/misc/guc_tables.c
+++ b/src/backend/utils/misc/guc_tables.c
@@ -3268,6 +3268,19 @@ struct config_int ConfigureNamesInt[] =
check_io_max_concurrency, NULL, NULL
},
+ {
+ {"io_bounce_buffers",
+ PGC_POSTMASTER,
+ RESOURCES_IO,
+ gettext_noop("Number of IO Bounce Buffers reserved for each backend."),
+ NULL,
+ GUC_UNIT_BLOCKS
+ },
+ &io_bounce_buffers,
+ -1, -1, 4096,
+ NULL, NULL, NULL
+ },
+
{
{"io_workers",
PGC_SIGHUP,
diff --git a/src/backend/utils/misc/postgresql.conf.sample b/src/backend/utils/misc/postgresql.conf.sample
index 7e8c3dcb175..ea71e9075ff 100644
--- a/src/backend/utils/misc/postgresql.conf.sample
+++ b/src/backend/utils/misc/postgresql.conf.sample
@@ -209,6 +209,8 @@
# -1 sets based on shared_buffers
# (change requires restart)
#io_workers = 3 # 1-32;
+#io_bounce_buffers = -1 # -1 sets based on shared_buffers
+ # (change requires restart)
# - Worker Processes -
diff --git a/src/backend/utils/resowner/resowner.c b/src/backend/utils/resowner/resowner.c
index d39f3e1b655..81e7e27965a 100644
--- a/src/backend/utils/resowner/resowner.c
+++ b/src/backend/utils/resowner/resowner.c
@@ -159,10 +159,11 @@ struct ResourceOwnerData
LOCALLOCK *locks[MAX_RESOWNER_LOCKS]; /* list of owned locks */
/*
- * AIO handles need be registered in critical sections and therefore
- * cannot use the normal ResourceElem mechanism.
+ * AIO handles & bounce buffers need be registered in critical sections
+ * and therefore cannot use the normal ResourceElem mechanism.
*/
dlist_head aio_handles;
+ dlist_head aio_bounce_buffers;
};
@@ -434,6 +435,7 @@ ResourceOwnerCreate(ResourceOwner parent, const char *name)
}
dlist_init(&owner->aio_handles);
+ dlist_init(&owner->aio_bounce_buffers);
return owner;
}
@@ -742,6 +744,13 @@ ResourceOwnerReleaseInternal(ResourceOwner owner,
pgaio_io_release_resowner(node, !isCommit);
}
+
+ while (!dlist_is_empty(&owner->aio_bounce_buffers))
+ {
+ dlist_node *node = dlist_head_node(&owner->aio_bounce_buffers);
+
+ pgaio_bounce_buffer_release_resowner(node, !isCommit);
+ }
}
else if (phase == RESOURCE_RELEASE_LOCKS)
{
@@ -1111,3 +1120,15 @@ ResourceOwnerForgetAioHandle(ResourceOwner owner, struct dlist_node *ioh_node)
{
dlist_delete_from(&owner->aio_handles, ioh_node);
}
+
+void
+ResourceOwnerRememberAioBounceBuffer(ResourceOwner owner, struct dlist_node *ioh_node)
+{
+ dlist_push_tail(&owner->aio_bounce_buffers, ioh_node);
+}
+
+void
+ResourceOwnerForgetAioBounceBuffer(ResourceOwner owner, struct dlist_node *ioh_node)
+{
+ dlist_delete_from(&owner->aio_bounce_buffers, ioh_node);
+}
diff --git a/src/test/modules/test_aio/test_aio--1.0.sql b/src/test/modules/test_aio/test_aio--1.0.sql
index 3d4e9bb070f..16f248619fc 100644
--- a/src/test/modules/test_aio/test_aio--1.0.sql
+++ b/src/test/modules/test_aio/test_aio--1.0.sql
@@ -76,6 +76,27 @@ RETURNS pg_catalog.void STRICT
AS 'MODULE_PATHNAME' LANGUAGE C;
+CREATE FUNCTION bb_get_and_error()
+RETURNS pg_catalog.void STRICT
+AS 'MODULE_PATHNAME' LANGUAGE C;
+
+CREATE FUNCTION bb_get_twice()
+RETURNS pg_catalog.void STRICT
+AS 'MODULE_PATHNAME' LANGUAGE C;
+
+CREATE FUNCTION bb_get()
+RETURNS pg_catalog.void STRICT
+AS 'MODULE_PATHNAME' LANGUAGE C;
+
+CREATE FUNCTION bb_get_release()
+RETURNS pg_catalog.void STRICT
+AS 'MODULE_PATHNAME' LANGUAGE C;
+
+CREATE FUNCTION bb_release_last()
+RETURNS pg_catalog.void STRICT
+AS 'MODULE_PATHNAME' LANGUAGE C;
+
+
/*
* Injection point related functions
diff --git a/src/test/modules/test_aio/test_aio.c b/src/test/modules/test_aio/test_aio.c
index 81b4d732206..60c01b251fe 100644
--- a/src/test/modules/test_aio/test_aio.c
+++ b/src/test/modules/test_aio/test_aio.c
@@ -53,6 +53,7 @@ static shmem_startup_hook_type prev_shmem_startup_hook = NULL;
static PgAioHandle *last_handle;
+static PgAioBounceBuffer *last_bb;
@@ -525,6 +526,60 @@ batch_end(PG_FUNCTION_ARGS)
PG_RETURN_VOID();
}
+PG_FUNCTION_INFO_V1(bb_get);
+Datum
+bb_get(PG_FUNCTION_ARGS)
+{
+ last_bb = pgaio_bounce_buffer_get();
+
+ PG_RETURN_VOID();
+}
+
+PG_FUNCTION_INFO_V1(bb_release_last);
+Datum
+bb_release_last(PG_FUNCTION_ARGS)
+{
+ if (!last_bb)
+ elog(ERROR, "no bb");
+
+ pgaio_bounce_buffer_release(last_bb);
+
+ PG_RETURN_VOID();
+}
+
+PG_FUNCTION_INFO_V1(bb_get_and_error);
+Datum
+bb_get_and_error(PG_FUNCTION_ARGS)
+{
+ pgaio_bounce_buffer_get();
+
+ elog(ERROR, "as you command");
+ PG_RETURN_VOID();
+}
+
+PG_FUNCTION_INFO_V1(bb_get_twice);
+Datum
+bb_get_twice(PG_FUNCTION_ARGS)
+{
+ pgaio_bounce_buffer_get();
+ pgaio_bounce_buffer_get();
+
+ PG_RETURN_VOID();
+}
+
+
+PG_FUNCTION_INFO_V1(bb_get_release);
+Datum
+bb_get_release(PG_FUNCTION_ARGS)
+{
+ PgAioBounceBuffer *bb;
+
+ bb = pgaio_bounce_buffer_get();
+ pgaio_bounce_buffer_release(bb);
+
+ PG_RETURN_VOID();
+}
+
#ifdef USE_INJECTION_POINTS
extern PGDLLEXPORT void inj_io_short_read(const char *name, const void *private_data);
extern PGDLLEXPORT void inj_io_reopen(const char *name, const void *private_data);
diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list
index 1748befca16..6ffd4e82976 100644
--- a/src/tools/pgindent/typedefs.list
+++ b/src/tools/pgindent/typedefs.list
@@ -2132,6 +2132,7 @@ PermutationStep
PermutationStepBlocker
PermutationStepBlockerType
PgAioBackend
+PgAioBounceBuffer
PgAioCtl
PgAioHandle
PgAioHandleCallbackID
--
2.48.1.76.g4e746b1a31.dirty
[text/x-diff] v2.9-0023-bufmgr-Implement-AIO-write-support.patch (9.9K, ../../oav52ytakcvfnk52w5o5wvxdjavxth2tgptxt25gnfmh2pdcpk@omp3xukdu5if/24-v2.9-0023-bufmgr-Implement-AIO-write-support.patch)
download | inline diff:
From f4fd610ea674d1f6d601b7e620b81f5c7698f892 Mon Sep 17 00:00:00 2001
From: Andres Freund <andres@anarazel.de>
Date: Sat, 15 Mar 2025 12:29:58 -0400
Subject: [PATCH v2.9 23/30] bufmgr: Implement AIO write support
As of this commit there are no users of these AIO facilities, that'll come in
later commits.
Author:
Reviewed-By:
Discussion: https://postgr.es/m/
Backpatch:
---
src/include/storage/aio.h | 2 +
src/include/storage/bufmgr.h | 2 +
src/backend/storage/aio/aio_callback.c | 2 +
src/backend/storage/buffer/bufmgr.c | 182 ++++++++++++++++++++++++-
4 files changed, 187 insertions(+), 1 deletion(-)
diff --git a/src/include/storage/aio.h b/src/include/storage/aio.h
index c673627a358..2eeabb8e01a 100644
--- a/src/include/storage/aio.h
+++ b/src/include/storage/aio.h
@@ -192,8 +192,10 @@ typedef enum PgAioHandleCallbackID
PGAIO_HCB_MD_WRITEV,
PGAIO_HCB_SHARED_BUFFER_READV,
+ PGAIO_HCB_SHARED_BUFFER_WRITEV,
PGAIO_HCB_LOCAL_BUFFER_READV,
+ PGAIO_HCB_LOCAL_BUFFER_WRITEV,
} PgAioHandleCallbackID;
diff --git a/src/include/storage/bufmgr.h b/src/include/storage/bufmgr.h
index ca4ca22c345..fffc76d11d9 100644
--- a/src/include/storage/bufmgr.h
+++ b/src/include/storage/bufmgr.h
@@ -176,7 +176,9 @@ extern PGDLLIMPORT int backend_flush_after;
extern PGDLLIMPORT int bgwriter_flush_after;
extern const PgAioHandleCallbacks aio_shared_buffer_readv_cb;
+extern const PgAioHandleCallbacks aio_shared_buffer_writev_cb;
extern const PgAioHandleCallbacks aio_local_buffer_readv_cb;
+extern const PgAioHandleCallbacks aio_local_buffer_writev_cb;
/* in buf_init.c */
extern PGDLLIMPORT char *BufferBlocks;
diff --git a/src/backend/storage/aio/aio_callback.c b/src/backend/storage/aio/aio_callback.c
index d1ea1ee145b..d7c261a964c 100644
--- a/src/backend/storage/aio/aio_callback.c
+++ b/src/backend/storage/aio/aio_callback.c
@@ -44,8 +44,10 @@ static const PgAioHandleCallbacksEntry aio_handle_cbs[] = {
CALLBACK_ENTRY(PGAIO_HCB_MD_WRITEV, aio_md_writev_cb),
CALLBACK_ENTRY(PGAIO_HCB_SHARED_BUFFER_READV, aio_shared_buffer_readv_cb),
+ CALLBACK_ENTRY(PGAIO_HCB_SHARED_BUFFER_WRITEV, aio_shared_buffer_writev_cb),
CALLBACK_ENTRY(PGAIO_HCB_LOCAL_BUFFER_READV, aio_local_buffer_readv_cb),
+ CALLBACK_ENTRY(PGAIO_HCB_LOCAL_BUFFER_WRITEV, aio_local_buffer_writev_cb),
#undef CALLBACK_ENTRY
};
diff --git a/src/backend/storage/buffer/bufmgr.c b/src/backend/storage/buffer/bufmgr.c
index 860886d4d90..886fbaf63f3 100644
--- a/src/backend/storage/buffer/bufmgr.c
+++ b/src/backend/storage/buffer/bufmgr.c
@@ -5507,7 +5507,15 @@ LockBuffer(Buffer buffer, int mode)
else if (mode == BUFFER_LOCK_SHARE)
LWLockAcquire(BufferDescriptorGetContentLock(buf), LW_SHARED);
else if (mode == BUFFER_LOCK_EXCLUSIVE)
+ {
+ /*
+ * FIXME: Wait for AIO writes, otherwise there would be a risk of
+ * deadlock. This isn't entirely trivial to do in a race-free way, IO
+ * could be started between us checking whether there is IO and
+ * enqueueing ourselves for the lock.
+ */
LWLockAcquire(BufferDescriptorGetContentLock(buf), LW_EXCLUSIVE);
+ }
else
elog(ERROR, "unrecognized buffer lock mode: %d", mode);
}
@@ -5522,6 +5530,19 @@ ConditionalLockBuffer(Buffer buffer)
{
BufferDesc *buf;
+ /*
+ * FIXME: Wait for AIO writes. Some code does not deal well
+ * ConditionalLockBuffer() continuously failing, e.g.
+ * spginsert()->spgdoinsert() ends up busy-looping (easily reproducible by
+ * just making this function always fail and running the regression
+ * tests). While that code could be fixed, it'd be hard to find all
+ * problematic places.
+ *
+ * It would be OK to wait for the IO as waiting for IO completion does not
+ * need to wait for any locks that could lead to an undetected deadlock or
+ * such.
+ */
+
Assert(BufferIsPinned(buffer));
if (BufferIsLocal(buffer))
return true; /* act as though we got it */
@@ -5594,6 +5615,11 @@ LockBufferForCleanup(Buffer buffer)
{
uint32 buf_state;
+ /*
+ * FIXME: LockBuffer()'s handling of in-progress writes (once
+ * implemented) should suffice to deal with deadlock risk.
+ */
+
/* Try to acquire lock */
LockBuffer(buffer, BUFFER_LOCK_EXCLUSIVE);
buf_state = LockBufHdr(bufHdr);
@@ -5741,6 +5767,14 @@ ConditionalLockBufferForCleanup(Buffer buffer)
Assert(BufferIsValid(buffer));
+ /*
+ * FIXME: Should wait for IO for the same reason as in
+ * ConditionalLockBuffer(). Needs to happen before the
+ * ConditionalLockBuffer() call below, as we'd never reach the
+ * ConditionalLockBuffer() call due the buffer pin held for the duration
+ * of the IO.
+ */
+
if (BufferIsLocal(buffer))
{
refcount = LocalRefCount[-buffer - 1];
@@ -6802,12 +6836,131 @@ buffer_readv_report(PgAioResult result, const PgAioTargetData *target_data, int
);
}
+/*
+ * Helper for AIO writev completion callbacks, supporting both shared and temp
+ * buffers. Gets called once for each buffer in a multi-page write.
+ */
+static pg_attribute_always_inline PgAioResult
+buffer_writev_complete_one(uint8 buf_off, Buffer buffer, uint8 flags,
+ bool failed, bool is_temp)
+{
+ BufferDesc *buf_hdr = is_temp ?
+ GetLocalBufferDescriptor(-buffer - 1)
+ : GetBufferDescriptor(buffer - 1);
+ PgAioResult result;
+ bool clear_dirty;
+ uint32 set_flag_bits;
+
+#ifdef USE_ASSERT_CHECKING
+ {
+ uint32 buf_state = pg_atomic_read_u32(&buf_hdr->state);
+
+ Assert(buf_state & BM_VALID);
+ Assert(buf_state & BM_TAG_VALID);
+ /* temp buffers don't use BM_IO_IN_PROGRESS */
+ if (!is_temp)
+ Assert(buf_state & BM_IO_IN_PROGRESS);
+ Assert(buf_state & BM_DIRTY);
+ }
+#endif
+
+ result.status = ARS_OK;
+
+ clear_dirty = failed ? false : true;
+ set_flag_bits = failed ? BM_IO_ERROR : 0;
+
+ if (is_temp)
+ TerminateLocalBufferIO(buf_hdr, clear_dirty, set_flag_bits, false);
+ else
+ TerminateBufferIO(buf_hdr, clear_dirty, set_flag_bits, false, false);
+
+ /*
+ * The initiator of IO is not managing the lock (i.e. we called
+ * LWLockDisown()), we are.
+ */
+ if (!is_temp)
+ LWLockReleaseDisowned(BufferDescriptorGetContentLock(buf_hdr),
+ LW_SHARED);
+
+ /* FIXME: tracepoint */
+
+ return result;
+}
+
+/*
+ * Perform completion handling of a single AIO write. This write may cover
+ * multiple blocks / buffers.
+ *
+ * Shared between shared and local buffers, to reduce code duplication.
+ */
+static pg_attribute_always_inline PgAioResult
+buffer_writev_complete(PgAioHandle *ioh, PgAioResult prior_result,
+ uint8 cb_data, bool is_temp)
+{
+ PgAioResult result = prior_result;
+ PgAioTargetData *td = pgaio_io_get_target_data(ioh);
+ uint64 *io_data;
+ uint8 handle_data_len;
+
+ if (is_temp)
+ {
+ Assert(td->smgr.is_temp);
+ Assert(pgaio_io_get_owner(ioh) == MyProcNumber);
+ }
+ else
+ Assert(!td->smgr.is_temp);
+
+ /*
+ * Iterate over all the buffers affected by this IO and call appropriate
+ * per-buffer completion function for each buffer.
+ */
+ io_data = pgaio_io_get_handle_data(ioh, &handle_data_len);
+ for (uint8 buf_off = 0; buf_off < handle_data_len; buf_off++)
+ {
+ Buffer buf = io_data[buf_off];
+ PgAioResult buf_result;
+ bool failed;
+
+ Assert(BufferIsValid(buf));
+
+ /*
+ * If the entire failed on a lower-level, each buffer needs to be
+ * marked as failed. In case of a partial read, some buffers may be
+ * ok.
+ */
+ failed =
+ prior_result.status == ARS_ERROR
+ || prior_result.result <= buf_off;
+
+ buf_result = buffer_writev_complete_one(buf_off, buf, cb_data, failed,
+ is_temp);
+
+ /*
+ * If there wasn't any prior error and the IO for this page failed in
+ * some form, set the whole IO's to the page's result.
+ */
+ if (result.status != ARS_ERROR && buf_result.status != ARS_OK)
+ {
+ result = buf_result;
+ pgaio_result_report(result, td, LOG);
+ }
+ }
+
+ return result;
+}
+
static void
shared_buffer_readv_stage(PgAioHandle *ioh, uint8 cb_data)
{
buffer_stage_common(ioh, false, false);
}
+static void
+shared_buffer_writev_stage(PgAioHandle *ioh, uint8 cb_data)
+{
+ buffer_stage_common(ioh, true, false);
+}
+
static PgAioResult
shared_buffer_readv_complete(PgAioHandle *ioh, PgAioResult prior_result,
uint8 cb_data)
@@ -6815,6 +6968,13 @@ shared_buffer_readv_complete(PgAioHandle *ioh, PgAioResult prior_result,
return buffer_readv_complete(ioh, prior_result, cb_data, false);
}
+static PgAioResult
+shared_buffer_writev_complete(PgAioHandle *ioh, PgAioResult prior_result,
+ uint8 cb_data)
+{
+ return buffer_writev_complete(ioh, prior_result, cb_data, false);
+}
+
static void
local_buffer_readv_stage(PgAioHandle *ioh, uint8 cb_data)
{
@@ -6828,13 +6988,29 @@ local_buffer_readv_complete(PgAioHandle *ioh, PgAioResult prior_result,
return buffer_readv_complete(ioh, prior_result, cb_data, true);
}
-/* readv callback is is passed READ_BUFFERS_* flags as callback data */
+static void
+local_buffer_writev_stage(PgAioHandle *ioh, uint8 cb_data)
+{
+ /*
+ * Currently this is unreachable as the only write support is for
+ * checkpointer / bgwriter, which don't deal with local buffers.
+ */
+ elog(ERROR, "should be unreachable");
+}
+
+
+/* readv callback is passed READ_BUFFERS_* flags as callback data */
const PgAioHandleCallbacks aio_shared_buffer_readv_cb = {
.stage = shared_buffer_readv_stage,
.complete_shared = shared_buffer_readv_complete,
.report = buffer_readv_report,
};
+const PgAioHandleCallbacks aio_shared_buffer_writev_cb = {
+ .stage = shared_buffer_writev_stage,
+ .complete_shared = shared_buffer_writev_complete,
+};
+
/* readv callback is is passed READ_BUFFERS_* flags as callback data */
const PgAioHandleCallbacks aio_local_buffer_readv_cb = {
.stage = local_buffer_readv_stage,
@@ -6848,3 +7024,7 @@ const PgAioHandleCallbacks aio_local_buffer_readv_cb = {
.complete_local = local_buffer_readv_complete,
.report = buffer_readv_report,
};
+
+const PgAioHandleCallbacks aio_local_buffer_writev_cb = {
+ .stage = local_buffer_writev_stage,
+};
--
2.48.1.76.g4e746b1a31.dirty
[text/x-diff] v2.9-0024-aio-Add-IO-queue-helper.patch (7.4K, ../../oav52ytakcvfnk52w5o5wvxdjavxth2tgptxt25gnfmh2pdcpk@omp3xukdu5if/25-v2.9-0024-aio-Add-IO-queue-helper.patch)
download | inline diff:
From 7bdb0f91ee2666574c73d0d5d3f6c57f7c0add91 Mon Sep 17 00:00:00 2001
From: Andres Freund <andres@anarazel.de>
Date: Sat, 15 Mar 2025 12:29:58 -0400
Subject: [PATCH v2.9 24/30] aio: Add IO queue helper
This is likely never going to anywhere - Thomas Munro is working on something
more complete. But I needed a way to exercise aio for checkpointer / bgwriter.
---
src/include/storage/io_queue.h | 33 +++++
src/backend/storage/aio/Makefile | 1 +
src/backend/storage/aio/io_queue.c | 204 ++++++++++++++++++++++++++++
src/backend/storage/aio/meson.build | 1 +
src/tools/pgindent/typedefs.list | 2 +
5 files changed, 241 insertions(+)
create mode 100644 src/include/storage/io_queue.h
create mode 100644 src/backend/storage/aio/io_queue.c
diff --git a/src/include/storage/io_queue.h b/src/include/storage/io_queue.h
new file mode 100644
index 00000000000..92b1e9afe6f
--- /dev/null
+++ b/src/include/storage/io_queue.h
@@ -0,0 +1,33 @@
+/*-------------------------------------------------------------------------
+ *
+ * io_queue.h
+ * Mechanism for tracking many IOs
+ *
+ *
+ * Portions Copyright (c) 1996-2025, PostgreSQL Global Development Group
+ * Portions Copyright (c) 1994, Regents of the University of California
+ *
+ * src/include/storage/io_queue.h
+ *
+ *-------------------------------------------------------------------------
+ */
+#ifndef IO_QUEUE_H
+#define IO_QUEUE_H
+
+#include "storage/aio_types.h"
+
+struct IOQueue;
+typedef struct IOQueue IOQueue;
+
+struct PgAioWaitRef;
+
+extern IOQueue *io_queue_create(int depth, int flags);
+extern void io_queue_track(IOQueue *ioq, const PgAioWaitRef *iow);
+extern void io_queue_wait_one(IOQueue *ioq);
+extern void io_queue_wait_all(IOQueue *ioq);
+extern bool io_queue_is_empty(IOQueue *ioq);
+extern void io_queue_reserve(IOQueue *ioq);
+extern PgAioHandle *io_queue_acquire_io(IOQueue *ioq);
+extern void io_queue_free(IOQueue *ioq);
+
+#endif /* IO_QUEUE_H */
diff --git a/src/backend/storage/aio/Makefile b/src/backend/storage/aio/Makefile
index 3f2469cc399..86fa4276fda 100644
--- a/src/backend/storage/aio/Makefile
+++ b/src/backend/storage/aio/Makefile
@@ -15,6 +15,7 @@ OBJS = \
aio_init.o \
aio_io.o \
aio_target.o \
+ io_queue.o \
method_io_uring.o \
method_sync.o \
method_worker.o \
diff --git a/src/backend/storage/aio/io_queue.c b/src/backend/storage/aio/io_queue.c
new file mode 100644
index 00000000000..526aa1d5e06
--- /dev/null
+++ b/src/backend/storage/aio/io_queue.c
@@ -0,0 +1,204 @@
+/*-------------------------------------------------------------------------
+ *
+ * io_queue.c
+ * AIO - Mechanism for tracking many IOs
+ *
+ * Portions Copyright (c) 1996-2025, PostgreSQL Global Development Group
+ * Portions Copyright (c) 1994, Regents of the University of California
+ *
+ * IDENTIFICATION
+ * src/backend/storage/aio/io_queue.c
+ *
+ *-------------------------------------------------------------------------
+ */
+
+#include "postgres.h"
+
+#include "lib/ilist.h"
+#include "storage/aio.h"
+#include "storage/io_queue.h"
+#include "utils/resowner.h"
+
+
+
+typedef struct TrackedIO
+{
+ PgAioWaitRef iow;
+ dlist_node node;
+} TrackedIO;
+
+struct IOQueue
+{
+ int depth;
+ int unsubmitted;
+
+ bool has_reserved;
+
+ dclist_head idle;
+ dclist_head in_progress;
+
+ TrackedIO tracked_ios[FLEXIBLE_ARRAY_MEMBER];
+};
+
+
+IOQueue *
+io_queue_create(int depth, int flags)
+{
+ size_t sz;
+ IOQueue *ioq;
+
+ sz = offsetof(IOQueue, tracked_ios)
+ + sizeof(TrackedIO) * depth;
+
+ ioq = palloc0(sz);
+
+ ioq->depth = 0;
+
+ for (int i = 0; i < depth; i++)
+ {
+ TrackedIO *tio = &ioq->tracked_ios[i];
+
+ pgaio_wref_clear(&tio->iow);
+ dclist_push_tail(&ioq->idle, &tio->node);
+ }
+
+ return ioq;
+}
+
+void
+io_queue_wait_one(IOQueue *ioq)
+{
+ /* submit all pending IO before waiting */
+ pgaio_submit_staged();
+
+ while (!dclist_is_empty(&ioq->in_progress))
+ {
+ /* FIXME: Should we really pop here already? */
+ dlist_node *node = dclist_pop_head_node(&ioq->in_progress);
+ TrackedIO *tio = dclist_container(TrackedIO, node, node);
+
+ pgaio_wref_wait(&tio->iow);
+ dclist_push_head(&ioq->idle, &tio->node);
+ }
+}
+
+void
+io_queue_reserve(IOQueue *ioq)
+{
+ if (ioq->has_reserved)
+ return;
+
+ if (dclist_is_empty(&ioq->idle))
+ io_queue_wait_one(ioq);
+
+ Assert(!dclist_is_empty(&ioq->idle));
+
+ ioq->has_reserved = true;
+}
+
+PgAioHandle *
+io_queue_acquire_io(IOQueue *ioq)
+{
+ PgAioHandle *ioh;
+
+ io_queue_reserve(ioq);
+
+ Assert(!dclist_is_empty(&ioq->idle));
+
+ if (!io_queue_is_empty(ioq))
+ {
+ ioh = pgaio_io_acquire_nb(CurrentResourceOwner, NULL);
+ if (ioh == NULL)
+ {
+ /*
+ * Need to wait for all IOs, blocking might not be legal in the
+ * context.
+ *
+ * XXX: This doesn't make a whole lot of sense, we're also
+ * blocking here. What was I smoking when I wrote the above?
+ */
+ io_queue_wait_all(ioq);
+ ioh = pgaio_io_acquire(CurrentResourceOwner, NULL);
+ }
+ }
+ else
+ {
+ ioh = pgaio_io_acquire(CurrentResourceOwner, NULL);
+ }
+
+ return ioh;
+}
+
+void
+io_queue_track(IOQueue *ioq, const struct PgAioWaitRef *iow)
+{
+ dlist_node *node;
+ TrackedIO *tio;
+
+ Assert(ioq->has_reserved);
+ ioq->has_reserved = false;
+
+ Assert(!dclist_is_empty(&ioq->idle));
+
+ node = dclist_pop_head_node(&ioq->idle);
+ tio = dclist_container(TrackedIO, node, node);
+
+ tio->iow = *iow;
+
+ dclist_push_tail(&ioq->in_progress, &tio->node);
+
+ ioq->unsubmitted++;
+
+ /*
+ * XXX: Should have some smarter logic here. We don't want to wait too
+ * long to submit, that'll mean we're more likely to block. But we also
+ * don't want to have the overhead of submitting every IO individually.
+ */
+ if (ioq->unsubmitted >= 4)
+ {
+ pgaio_submit_staged();
+ ioq->unsubmitted = 0;
+ }
+}
+
+void
+io_queue_wait_all(IOQueue *ioq)
+{
+ /* submit all pending IO before waiting */
+ pgaio_submit_staged();
+
+ while (!dclist_is_empty(&ioq->in_progress))
+ {
+ /* wait for the last IO to minimize unnecessary wakeups */
+ dlist_node *node = dclist_tail_node(&ioq->in_progress);
+ TrackedIO *tio = dclist_container(TrackedIO, node, node);
+
+ if (!pgaio_wref_check_done(&tio->iow))
+ {
+ ereport(DEBUG3,
+ errmsg("io_queue_wait_all for io:%d",
+ pgaio_wref_get_id(&tio->iow)),
+ errhidestmt(true),
+ errhidecontext(true));
+
+ pgaio_wref_wait(&tio->iow);
+ }
+
+ dclist_delete_from(&ioq->in_progress, &tio->node);
+ dclist_push_head(&ioq->idle, &tio->node);
+ }
+}
+
+bool
+io_queue_is_empty(IOQueue *ioq)
+{
+ return dclist_is_empty(&ioq->in_progress);
+}
+
+void
+io_queue_free(IOQueue *ioq)
+{
+ io_queue_wait_all(ioq);
+
+ pfree(ioq);
+}
diff --git a/src/backend/storage/aio/meson.build b/src/backend/storage/aio/meson.build
index da6df2d3654..270c4a64428 100644
--- a/src/backend/storage/aio/meson.build
+++ b/src/backend/storage/aio/meson.build
@@ -7,6 +7,7 @@ backend_sources += files(
'aio_init.c',
'aio_io.c',
'aio_target.c',
+ 'io_queue.c',
'method_io_uring.c',
'method_sync.c',
'method_worker.c',
diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list
index 6ffd4e82976..26c622824e4 100644
--- a/src/tools/pgindent/typedefs.list
+++ b/src/tools/pgindent/typedefs.list
@@ -1191,6 +1191,7 @@ IOContext
IOFuncSelector
IOObject
IOOp
+IOQueue
IO_STATUS_BLOCK
IPCompareMethod
ITEM
@@ -3016,6 +3017,7 @@ TocEntry
TokenAuxData
TokenizedAuthLine
TrackItem
+TrackedIO
TransApplyAction
TransInvalidationInfo
TransState
--
2.48.1.76.g4e746b1a31.dirty
[text/x-diff] v2.9-0025-bufmgr-use-AIO-in-checkpointer-bgwriter.patch (31.1K, ../../oav52ytakcvfnk52w5o5wvxdjavxth2tgptxt25gnfmh2pdcpk@omp3xukdu5if/26-v2.9-0025-bufmgr-use-AIO-in-checkpointer-bgwriter.patch)
download | inline diff:
From 12321aa81bf8063dcd0859c13c40d8bff1c9b9d9 Mon Sep 17 00:00:00 2001
From: Andres Freund <andres@anarazel.de>
Date: Sat, 15 Mar 2025 12:29:58 -0400
Subject: [PATCH v2.9 25/30] bufmgr: use AIO in checkpointer, bgwriter
This is far from ready - just included to be able to exercise AIO writes and
get some preliminary numbers. In all likelihood this will instead be based
on-top of work by Thomas Munro instead of the preceding commit.
---
src/include/postmaster/bgwriter.h | 3 +-
src/include/storage/buf_internals.h | 2 +
src/include/storage/bufmgr.h | 3 +-
src/include/storage/bufpage.h | 1 +
src/backend/postmaster/bgwriter.c | 19 +-
src/backend/postmaster/checkpointer.c | 11 +-
src/backend/storage/buffer/bufmgr.c | 588 +++++++++++++++++++++++---
src/backend/storage/page/bufpage.c | 10 +
src/tools/pgindent/typedefs.list | 1 +
9 files changed, 580 insertions(+), 58 deletions(-)
diff --git a/src/include/postmaster/bgwriter.h b/src/include/postmaster/bgwriter.h
index 800ecbfd13b..a8081d411b6 100644
--- a/src/include/postmaster/bgwriter.h
+++ b/src/include/postmaster/bgwriter.h
@@ -31,7 +31,8 @@ pg_noreturn extern void BackgroundWriterMain(const void *startup_data, size_t st
pg_noreturn extern void CheckpointerMain(const void *startup_data, size_t startup_data_len);
extern void RequestCheckpoint(int flags);
-extern void CheckpointWriteDelay(int flags, double progress);
+struct IOQueue;
+extern void CheckpointWriteDelay(struct IOQueue *ioq, int flags, double progress);
extern bool ForwardSyncRequest(const FileTag *ftag, SyncRequestType type);
diff --git a/src/include/storage/buf_internals.h b/src/include/storage/buf_internals.h
index 4b1f52a9fc8..a3df3192d12 100644
--- a/src/include/storage/buf_internals.h
+++ b/src/include/storage/buf_internals.h
@@ -21,6 +21,8 @@
#include "storage/buf.h"
#include "storage/bufmgr.h"
#include "storage/condition_variable.h"
+#include "storage/io_queue.h"
+#include "storage/latch.h"
#include "storage/lwlock.h"
#include "storage/procnumber.h"
#include "storage/shmem.h"
diff --git a/src/include/storage/bufmgr.h b/src/include/storage/bufmgr.h
index fffc76d11d9..30d965ec069 100644
--- a/src/include/storage/bufmgr.h
+++ b/src/include/storage/bufmgr.h
@@ -299,7 +299,8 @@ extern bool ConditionalLockBufferForCleanup(Buffer buffer);
extern bool IsBufferCleanupOK(Buffer buffer);
extern bool HoldingBufferPinThatDelaysRecovery(void);
-extern bool BgBufferSync(struct WritebackContext *wb_context);
+struct IOQueue;
+extern bool BgBufferSync(struct IOQueue *ioq, struct WritebackContext *wb_context);
extern uint32 GetPinLimit(void);
extern uint32 GetLocalPinLimit(void);
diff --git a/src/include/storage/bufpage.h b/src/include/storage/bufpage.h
index 6646b6f6371..9c045e81857 100644
--- a/src/include/storage/bufpage.h
+++ b/src/include/storage/bufpage.h
@@ -509,5 +509,6 @@ extern bool PageIndexTupleOverwrite(Page page, OffsetNumber offnum,
Item newtup, Size newsize);
extern char *PageSetChecksumCopy(Page page, BlockNumber blkno);
extern void PageSetChecksumInplace(Page page, BlockNumber blkno);
+extern bool PageNeedsChecksumCopy(Page page);
#endif /* BUFPAGE_H */
diff --git a/src/backend/postmaster/bgwriter.c b/src/backend/postmaster/bgwriter.c
index 72f5acceec7..6e8801a39e3 100644
--- a/src/backend/postmaster/bgwriter.c
+++ b/src/backend/postmaster/bgwriter.c
@@ -38,11 +38,13 @@
#include "postmaster/auxprocess.h"
#include "postmaster/bgwriter.h"
#include "postmaster/interrupt.h"
+#include "storage/aio.h"
#include "storage/aio_subsys.h"
#include "storage/buf_internals.h"
#include "storage/bufmgr.h"
#include "storage/condition_variable.h"
#include "storage/fd.h"
+#include "storage/io_queue.h"
#include "storage/lwlock.h"
#include "storage/proc.h"
#include "storage/procsignal.h"
@@ -90,6 +92,7 @@ BackgroundWriterMain(const void *startup_data, size_t startup_data_len)
sigjmp_buf local_sigjmp_buf;
MemoryContext bgwriter_context;
bool prev_hibernate;
+ IOQueue *ioq;
WritebackContext wb_context;
Assert(startup_data_len == 0);
@@ -131,6 +134,7 @@ BackgroundWriterMain(const void *startup_data, size_t startup_data_len)
ALLOCSET_DEFAULT_SIZES);
MemoryContextSwitchTo(bgwriter_context);
+ ioq = io_queue_create(128, 0);
WritebackContextInit(&wb_context, &bgwriter_flush_after);
/*
@@ -228,12 +232,22 @@ BackgroundWriterMain(const void *startup_data, size_t startup_data_len)
/* Clear any already-pending wakeups */
ResetLatch(MyLatch);
+ /*
+ * FIXME: this is theoretically racy, but I didn't want to copy
+ * ProcessMainLoopInterrupts() remaining body here.
+ */
+ if (ShutdownRequestPending)
+ {
+ io_queue_wait_all(ioq);
+ io_queue_free(ioq);
+ }
+
ProcessMainLoopInterrupts();
/*
* Do one cycle of dirty-buffer writing.
*/
- can_hibernate = BgBufferSync(&wb_context);
+ can_hibernate = BgBufferSync(ioq, &wb_context);
/* Report pending statistics to the cumulative stats system */
pgstat_report_bgwriter();
@@ -250,6 +264,9 @@ BackgroundWriterMain(const void *startup_data, size_t startup_data_len)
smgrdestroyall();
}
+ /* finish IO before sleeping, to avoid blocking other backends */
+ io_queue_wait_all(ioq);
+
/*
* Log a new xl_running_xacts every now and then so replication can
* get into a consistent state faster (think of suboverflowed
diff --git a/src/backend/postmaster/checkpointer.c b/src/backend/postmaster/checkpointer.c
index fda91ffd1ce..904fe167eb4 100644
--- a/src/backend/postmaster/checkpointer.c
+++ b/src/backend/postmaster/checkpointer.c
@@ -49,10 +49,12 @@
#include "postmaster/bgwriter.h"
#include "postmaster/interrupt.h"
#include "replication/syncrep.h"
+#include "storage/aio.h"
#include "storage/aio_subsys.h"
#include "storage/bufmgr.h"
#include "storage/condition_variable.h"
#include "storage/fd.h"
+#include "storage/io_queue.h"
#include "storage/ipc.h"
#include "storage/lwlock.h"
#include "storage/pmsignal.h"
@@ -766,7 +768,7 @@ ImmediateCheckpointRequested(void)
* fraction between 0.0 meaning none, and 1.0 meaning all done.
*/
void
-CheckpointWriteDelay(int flags, double progress)
+CheckpointWriteDelay(IOQueue *ioq, int flags, double progress)
{
static int absorb_counter = WRITES_PER_ABSORB;
@@ -800,6 +802,13 @@ CheckpointWriteDelay(int flags, double progress)
/* Report interim statistics to the cumulative stats system */
pgstat_report_checkpointer();
+ /*
+ * Ensure all pending IO is submitted to avoid unnecessary delays for
+ * other processes.
+ */
+ io_queue_wait_all(ioq);
+
+
/*
* This sleep used to be connected to bgwriter_delay, typically 200ms.
* That resulted in more frequent wakeups if not much work to do.
diff --git a/src/backend/storage/buffer/bufmgr.c b/src/backend/storage/buffer/bufmgr.c
index 886fbaf63f3..3312b713c8c 100644
--- a/src/backend/storage/buffer/bufmgr.c
+++ b/src/backend/storage/buffer/bufmgr.c
@@ -52,6 +52,7 @@
#include "storage/buf_internals.h"
#include "storage/bufmgr.h"
#include "storage/fd.h"
+#include "storage/io_queue.h"
#include "storage/ipc.h"
#include "storage/lmgr.h"
#include "storage/proc.h"
@@ -76,6 +77,7 @@
/* Bits in SyncOneBuffer's return value */
#define BUF_WRITTEN 0x01
#define BUF_REUSABLE 0x02
+#define BUF_CANT_MERGE 0x04
#define RELS_BSEARCH_THRESHOLD 20
@@ -512,8 +514,6 @@ static void UnpinBuffer(BufferDesc *buf);
static void UnpinBufferNoOwner(BufferDesc *buf);
static void BufferSync(int flags);
static uint32 WaitBufHdrUnlocked(BufferDesc *buf);
-static int SyncOneBuffer(int buf_id, bool skip_recently_used,
- WritebackContext *wb_context);
static void WaitIO(BufferDesc *buf);
static void AbortBufferIO(Buffer buffer);
static void shared_buffer_write_error_callback(void *arg);
@@ -529,6 +529,7 @@ static void CheckReadBuffersOperation(ReadBuffersOperation *operation, bool is_c
static Buffer GetVictimBuffer(BufferAccessStrategy strategy, IOContext io_context);
static void FlushBuffer(BufferDesc *buf, SMgrRelation reln,
IOObject io_object, IOContext io_context);
+
static void FindAndDropRelationBuffers(RelFileLocator rlocator,
ForkNumber forkNum,
BlockNumber nForkBlock,
@@ -3299,6 +3300,57 @@ UnpinBufferNoOwner(BufferDesc *buf)
}
}
+typedef struct BuffersToWrite
+{
+ int nbuffers;
+ BufferTag start_at_tag;
+ uint32 max_combine;
+
+ XLogRecPtr max_lsn;
+
+ PgAioHandle *ioh;
+ PgAioWaitRef iow;
+
+ uint64 total_writes;
+
+ Buffer buffers[IOV_MAX];
+ PgAioBounceBuffer *bounce_buffers[IOV_MAX];
+ const void *data_ptrs[IOV_MAX];
+} BuffersToWrite;
+
+static int PrepareToWriteBuffer(BuffersToWrite *to_write, Buffer buf,
+ bool skip_recently_used,
+ IOQueue *ioq, WritebackContext *wb_context);
+
+static void WriteBuffers(BuffersToWrite *to_write,
+ IOQueue *ioq, WritebackContext *wb_context);
+
+static void
+BuffersToWriteInit(BuffersToWrite *to_write,
+ IOQueue *ioq, WritebackContext *wb_context)
+{
+ to_write->total_writes = 0;
+ to_write->nbuffers = 0;
+ to_write->ioh = NULL;
+ pgaio_wref_clear(&to_write->iow);
+ to_write->max_lsn = InvalidXLogRecPtr;
+
+ pgaio_enter_batchmode();
+}
+
+static void
+BuffersToWriteEnd(BuffersToWrite *to_write)
+{
+ if (to_write->ioh != NULL)
+ {
+ pgaio_io_release(to_write->ioh);
+ to_write->ioh = NULL;
+ }
+
+ pgaio_exit_batchmode();
+}
+
+
#define ST_SORT sort_checkpoint_bufferids
#define ST_ELEMENT_TYPE CkptSortItem
#define ST_COMPARE(a, b) ckpt_buforder_comparator(a, b)
@@ -3330,7 +3382,10 @@ BufferSync(int flags)
binaryheap *ts_heap;
int i;
int mask = BM_DIRTY;
+ IOQueue *ioq;
WritebackContext wb_context;
+ BuffersToWrite to_write;
+ int max_combine;
/*
* Unless this is a shutdown checkpoint or we have been explicitly told,
@@ -3392,7 +3447,9 @@ BufferSync(int flags)
if (num_to_scan == 0)
return; /* nothing to do */
+ ioq = io_queue_create(512, 0);
WritebackContextInit(&wb_context, &checkpoint_flush_after);
+ max_combine = Min(io_bounce_buffers, io_combine_limit);
TRACE_POSTGRESQL_BUFFER_SYNC_START(NBuffers, num_to_scan);
@@ -3500,48 +3557,91 @@ BufferSync(int flags)
*/
num_processed = 0;
num_written = 0;
+
+ BuffersToWriteInit(&to_write, ioq, &wb_context);
+
while (!binaryheap_empty(ts_heap))
{
BufferDesc *bufHdr = NULL;
CkptTsStatus *ts_stat = (CkptTsStatus *)
DatumGetPointer(binaryheap_first(ts_heap));
+ bool batch_continue = true;
- buf_id = CkptBufferIds[ts_stat->index].buf_id;
- Assert(buf_id != -1);
-
- bufHdr = GetBufferDescriptor(buf_id);
-
- num_processed++;
+ Assert(ts_stat->num_scanned <= ts_stat->num_to_scan);
/*
- * We don't need to acquire the lock here, because we're only looking
- * at a single bit. It's possible that someone else writes the buffer
- * and clears the flag right after we check, but that doesn't matter
- * since SyncOneBuffer will then do nothing. However, there is a
- * further race condition: it's conceivable that between the time we
- * examine the bit here and the time SyncOneBuffer acquires the lock,
- * someone else not only wrote the buffer but replaced it with another
- * page and dirtied it. In that improbable case, SyncOneBuffer will
- * write the buffer though we didn't need to. It doesn't seem worth
- * guarding against this, though.
+ * Collect a batch of buffers to write out from the current
+ * tablespace. That causes some imbalance between the tablespaces, but
+ * that's more than outweighed by the efficiency gain due to batching.
*/
- if (pg_atomic_read_u32(&bufHdr->state) & BM_CHECKPOINT_NEEDED)
+ while (batch_continue &&
+ to_write.nbuffers < max_combine &&
+ ts_stat->num_scanned < ts_stat->num_to_scan)
{
- if (SyncOneBuffer(buf_id, false, &wb_context) & BUF_WRITTEN)
+ buf_id = CkptBufferIds[ts_stat->index].buf_id;
+ Assert(buf_id != -1);
+
+ bufHdr = GetBufferDescriptor(buf_id);
+
+ num_processed++;
+
+ /*
+ * We don't need to acquire the lock here, because we're only
+ * looking at a single bit. It's possible that someone else writes
+ * the buffer and clears the flag right after we check, but that
+ * doesn't matter since SyncOneBuffer will then do nothing.
+ * However, there is a further race condition: it's conceivable
+ * that between the time we examine the bit here and the time
+ * SyncOneBuffer acquires the lock, someone else not only wrote
+ * the buffer but replaced it with another page and dirtied it. In
+ * that improbable case, SyncOneBuffer will write the buffer
+ * though we didn't need to. It doesn't seem worth guarding
+ * against this, though.
+ */
+ if (pg_atomic_read_u32(&bufHdr->state) & BM_CHECKPOINT_NEEDED)
{
- TRACE_POSTGRESQL_BUFFER_SYNC_WRITTEN(buf_id);
- PendingCheckpointerStats.buffers_written++;
- num_written++;
+ int result = PrepareToWriteBuffer(&to_write, buf_id + 1, false,
+ ioq, &wb_context);
+
+ if (result & BUF_CANT_MERGE)
+ {
+ Assert(to_write.nbuffers > 0);
+ WriteBuffers(&to_write, ioq, &wb_context);
+
+ result = PrepareToWriteBuffer(&to_write, buf_id + 1, false,
+ ioq, &wb_context);
+ Assert(result != BUF_CANT_MERGE);
+ }
+
+ if (result & BUF_WRITTEN)
+ {
+ TRACE_POSTGRESQL_BUFFER_SYNC_WRITTEN(buf_id);
+ PendingCheckpointerStats.buffers_written++;
+ num_written++;
+ }
+ else
+ {
+ batch_continue = false;
+ }
}
+ else
+ {
+ if (to_write.nbuffers > 0)
+ WriteBuffers(&to_write, ioq, &wb_context);
+ }
+
+ /*
+ * Measure progress independent of actually having to flush the
+ * buffer - otherwise writing become unbalanced.
+ */
+ ts_stat->progress += ts_stat->progress_slice;
+ ts_stat->num_scanned++;
+ ts_stat->index++;
}
- /*
- * Measure progress independent of actually having to flush the buffer
- * - otherwise writing become unbalanced.
- */
- ts_stat->progress += ts_stat->progress_slice;
- ts_stat->num_scanned++;
- ts_stat->index++;
+ if (to_write.nbuffers > 0)
+ WriteBuffers(&to_write, ioq, &wb_context);
+
/* Have all the buffers from the tablespace been processed? */
if (ts_stat->num_scanned == ts_stat->num_to_scan)
@@ -3559,15 +3659,23 @@ BufferSync(int flags)
*
* (This will check for barrier events even if it doesn't sleep.)
*/
- CheckpointWriteDelay(flags, (double) num_processed / num_to_scan);
+ CheckpointWriteDelay(ioq, flags, (double) num_processed / num_to_scan);
}
+ Assert(to_write.nbuffers == 0);
+ io_queue_wait_all(ioq);
+
/*
* Issue all pending flushes. Only checkpointer calls BufferSync(), so
* IOContext will always be IOCONTEXT_NORMAL.
*/
IssuePendingWritebacks(&wb_context, IOCONTEXT_NORMAL);
+ io_queue_wait_all(ioq); /* IssuePendingWritebacks might have added
+ * more */
+ io_queue_free(ioq);
+ BuffersToWriteEnd(&to_write);
+
pfree(per_ts_stat);
per_ts_stat = NULL;
binaryheap_free(ts_heap);
@@ -3593,7 +3701,7 @@ BufferSync(int flags)
* bgwriter_lru_maxpages to 0.)
*/
bool
-BgBufferSync(WritebackContext *wb_context)
+BgBufferSync(IOQueue *ioq, WritebackContext *wb_context)
{
/* info obtained from freelist.c */
int strategy_buf_id;
@@ -3636,6 +3744,9 @@ BgBufferSync(WritebackContext *wb_context)
long new_strategy_delta;
uint32 new_recent_alloc;
+ BuffersToWrite to_write;
+ int max_combine;
+
/*
* Find out where the freelist clock sweep currently is, and how many
* buffer allocations have happened since our last call.
@@ -3656,6 +3767,8 @@ BgBufferSync(WritebackContext *wb_context)
return true;
}
+ max_combine = Min(io_bounce_buffers, io_combine_limit);
+
/*
* Compute strategy_delta = how many buffers have been scanned by the
* clock sweep since last time. If first time through, assume none. Then
@@ -3812,11 +3925,25 @@ BgBufferSync(WritebackContext *wb_context)
num_written = 0;
reusable_buffers = reusable_buffers_est;
+ BuffersToWriteInit(&to_write, ioq, wb_context);
+
/* Execute the LRU scan */
while (num_to_scan > 0 && reusable_buffers < upcoming_alloc_est)
{
- int sync_state = SyncOneBuffer(next_to_clean, true,
- wb_context);
+ int sync_state;
+
+ sync_state = PrepareToWriteBuffer(&to_write, next_to_clean + 1,
+ true, ioq, wb_context);
+ if (sync_state & BUF_CANT_MERGE)
+ {
+ Assert(to_write.nbuffers > 0);
+
+ WriteBuffers(&to_write, ioq, wb_context);
+
+ sync_state = PrepareToWriteBuffer(&to_write, next_to_clean + 1,
+ true, ioq, wb_context);
+ Assert(sync_state != BUF_CANT_MERGE);
+ }
if (++next_to_clean >= NBuffers)
{
@@ -3827,6 +3954,13 @@ BgBufferSync(WritebackContext *wb_context)
if (sync_state & BUF_WRITTEN)
{
+ Assert(sync_state & BUF_REUSABLE);
+
+ if (to_write.nbuffers == max_combine)
+ {
+ WriteBuffers(&to_write, ioq, wb_context);
+ }
+
reusable_buffers++;
if (++num_written >= bgwriter_lru_maxpages)
{
@@ -3838,6 +3972,11 @@ BgBufferSync(WritebackContext *wb_context)
reusable_buffers++;
}
+ if (to_write.nbuffers > 0)
+ WriteBuffers(&to_write, ioq, wb_context);
+
+ BuffersToWriteEnd(&to_write);
+
PendingBgWriterStats.buf_written_clean += num_written;
#ifdef BGW_DEBUG
@@ -3876,8 +4015,66 @@ BgBufferSync(WritebackContext *wb_context)
return (bufs_to_lap == 0 && recent_alloc == 0);
}
+static inline bool
+BufferTagsSameRel(const BufferTag *tag1, const BufferTag *tag2)
+{
+ return (tag1->spcOid == tag2->spcOid) &&
+ (tag1->dbOid == tag2->dbOid) &&
+ (tag1->relNumber == tag2->relNumber) &&
+ (tag1->forkNum == tag2->forkNum)
+ ;
+}
+
+static bool
+CanMergeWrite(BuffersToWrite *to_write, BufferDesc *cur_buf_hdr)
+{
+ BlockNumber cur_block = cur_buf_hdr->tag.blockNum;
+
+ Assert(to_write->nbuffers > 0); /* can't merge with nothing */
+ Assert(to_write->start_at_tag.relNumber != InvalidOid);
+ Assert(to_write->start_at_tag.blockNum != InvalidBlockNumber);
+
+ Assert(to_write->ioh != NULL);
+
+ /*
+ * First check if the blocknumber is one that we could actually merge,
+ * that's cheaper than checking the tablespace/db/relnumber/fork match.
+ */
+ if (to_write->start_at_tag.blockNum + to_write->nbuffers != cur_block)
+ return false;
+
+ if (!BufferTagsSameRel(&to_write->start_at_tag, &cur_buf_hdr->tag))
+ return false;
+
+ /*
+ * Need to check with smgr how large a write we're allowed to make. To
+ * reduce the overhead of the smgr check, only inquire once, when
+ * processing the first to-be-merged buffer. That avoids the overhead in
+ * the common case of writing out buffers that definitely not mergeable.
+ */
+ if (to_write->nbuffers == 1)
+ {
+ SMgrRelation smgr;
+
+ smgr = smgropen(BufTagGetRelFileLocator(&to_write->start_at_tag), INVALID_PROC_NUMBER);
+
+ to_write->max_combine = smgrmaxcombine(smgr,
+ to_write->start_at_tag.forkNum,
+ to_write->start_at_tag.blockNum);
+ }
+ else
+ {
+ Assert(to_write->max_combine > 0);
+ }
+
+ if (to_write->start_at_tag.blockNum + to_write->max_combine <= cur_block)
+ return false;
+
+ return true;
+}
+
/*
- * SyncOneBuffer -- process a single buffer during syncing.
+ * PrepareToWriteBuffer -- process a single buffer during syncing.
*
* If skip_recently_used is true, we don't write currently-pinned buffers, nor
* buffers marked recently used, as these are not replacement candidates.
@@ -3886,22 +4083,50 @@ BgBufferSync(WritebackContext *wb_context)
* BUF_WRITTEN: we wrote the buffer.
* BUF_REUSABLE: buffer is available for replacement, ie, it has
* pin count 0 and usage count 0.
+ * BUF_CANT_MERGE: can't combine this write with prior writes, caller needs
+ * to issue those first
*
* (BUF_WRITTEN could be set in error if FlushBuffer finds the buffer clean
* after locking it, but we don't care all that much.)
*/
static int
-SyncOneBuffer(int buf_id, bool skip_recently_used, WritebackContext *wb_context)
+PrepareToWriteBuffer(BuffersToWrite *to_write, Buffer buf,
+ bool skip_recently_used,
+ IOQueue *ioq, WritebackContext *wb_context)
{
- BufferDesc *bufHdr = GetBufferDescriptor(buf_id);
+ BufferDesc *cur_buf_hdr = GetBufferDescriptor(buf - 1);
+ uint32 buf_state;
int result = 0;
- uint32 buf_state;
- BufferTag tag;
+ XLogRecPtr cur_buf_lsn;
+ LWLock *content_lock;
+ bool may_block;
+
+ /*
+ * Check if this buffer can be written out together with already prepared
+ * writes. We check before we have pinned the buffer, so the buffer can be
+ * written out and replaced between this check and us pinning the buffer -
+ * we'll recheck below. The reason for the pre-check is that we don't want
+ * to pin the buffer just to find out that we can't merge the IO.
+ */
+ if (to_write->nbuffers != 0)
+ {
+ if (!CanMergeWrite(to_write, cur_buf_hdr))
+ {
+ result |= BUF_CANT_MERGE;
+ return result;
+ }
+ }
+ else
+ {
+ to_write->start_at_tag = cur_buf_hdr->tag;
+ }
/* Make sure we can handle the pin */
ReservePrivateRefCountEntry();
ResourceOwnerEnlarge(CurrentResourceOwner);
+ /* XXX: Should also check if we are allowed to pin one more buffer */
+
/*
* Check whether buffer needs writing.
*
@@ -3911,7 +4136,7 @@ SyncOneBuffer(int buf_id, bool skip_recently_used, WritebackContext *wb_context)
* don't worry because our checkpoint.redo points before log record for
* upcoming changes and so we are not required to write such dirty buffer.
*/
- buf_state = LockBufHdr(bufHdr);
+ buf_state = LockBufHdr(cur_buf_hdr);
if (BUF_STATE_GET_REFCOUNT(buf_state) == 0 &&
BUF_STATE_GET_USAGECOUNT(buf_state) == 0)
@@ -3920,40 +4145,294 @@ SyncOneBuffer(int buf_id, bool skip_recently_used, WritebackContext *wb_context)
}
else if (skip_recently_used)
{
+#if 0
+ elog(LOG, "at block %d: skip recent with nbuffers %d",
+ cur_buf_hdr->tag.blockNum, to_write->nbuffers);
+#endif
/* Caller told us not to write recently-used buffers */
- UnlockBufHdr(bufHdr, buf_state);
+ UnlockBufHdr(cur_buf_hdr, buf_state);
return result;
}
if (!(buf_state & BM_VALID) || !(buf_state & BM_DIRTY))
{
/* It's clean, so nothing to do */
- UnlockBufHdr(bufHdr, buf_state);
+ UnlockBufHdr(cur_buf_hdr, buf_state);
return result;
}
+ /* pin the buffer, from now on its identity can't change anymore */
+ PinBuffer_Locked(cur_buf_hdr);
+
+ /*
+ * Acquire IO, if needed, now that it's likely that we'll need to write.
+ */
+ if (to_write->ioh == NULL)
+ {
+ /* otherwise we should already have acquired a handle */
+ Assert(to_write->nbuffers == 0);
+
+ to_write->ioh = io_queue_acquire_io(ioq);
+ pgaio_io_get_wref(to_write->ioh, &to_write->iow);
+ }
+
/*
- * Pin it, share-lock it, write it. (FlushBuffer will do nothing if the
- * buffer is clean by the time we've locked it.)
+ * If we are merging, check if the buffer's identity possibly changed
+ * while we hadn't yet pinned it.
+ *
+ * XXX: It might be worth checking if we still want to write the buffer
+ * out, e.g. it could have been replaced with a buffer that doesn't have
+ * BM_CHECKPOINT_NEEDED set.
*/
- PinBuffer_Locked(bufHdr);
- LWLockAcquire(BufferDescriptorGetContentLock(bufHdr), LW_SHARED);
+ if (to_write->nbuffers != 0)
+ {
+ if (!CanMergeWrite(to_write, cur_buf_hdr))
+ {
+ elog(LOG, "changed identity");
+ UnpinBuffer(cur_buf_hdr);
+
+ result |= BUF_CANT_MERGE;
+
+ return result;
+ }
+ }
+
+ may_block = to_write->nbuffers == 0
+ && !pgaio_have_staged()
+ && io_queue_is_empty(ioq)
+ ;
+ content_lock = BufferDescriptorGetContentLock(cur_buf_hdr);
+
+ if (!may_block)
+ {
+ if (LWLockConditionalAcquire(content_lock, LW_SHARED))
+ {
+ /* done */
+ }
+ else if (to_write->nbuffers == 0)
+ {
+ /*
+ * Need to wait for all prior IO to finish before blocking for
+ * lock acquisition, to avoid the risk a deadlock due to us
+ * waiting for another backend that is waiting for our unsubmitted
+ * IO to complete.
+ */
+ pgaio_submit_staged();
+ io_queue_wait_all(ioq);
+
+ elog(DEBUG2, "at block %u: can't block, nbuffers = 0",
+ cur_buf_hdr->tag.blockNum
+ );
+
+ may_block = to_write->nbuffers == 0
+ && !pgaio_have_staged()
+ && io_queue_is_empty(ioq)
+ ;
+ Assert(may_block);
+
+ LWLockAcquire(content_lock, LW_SHARED);
+ }
+ else
+ {
+ elog(DEBUG2, "at block %d: can't block nbuffers = %d",
+ cur_buf_hdr->tag.blockNum,
+ to_write->nbuffers);
- FlushBuffer(bufHdr, NULL, IOOBJECT_RELATION, IOCONTEXT_NORMAL);
+ UnpinBuffer(cur_buf_hdr);
+ result |= BUF_CANT_MERGE;
+ Assert(to_write->nbuffers > 0);
- LWLockRelease(BufferDescriptorGetContentLock(bufHdr));
+ return result;
+ }
+ }
+ else
+ {
+ LWLockAcquire(content_lock, LW_SHARED);
+ }
- tag = bufHdr->tag;
+ if (!may_block)
+ {
+ if (!StartBufferIO(cur_buf_hdr, false, !may_block))
+ {
+ pgaio_submit_staged();
+ io_queue_wait_all(ioq);
- UnpinBuffer(bufHdr);
+ may_block = io_queue_is_empty(ioq) && to_write->nbuffers == 0 && !pgaio_have_staged();
+
+ if (!StartBufferIO(cur_buf_hdr, false, !may_block))
+ {
+ elog(DEBUG2, "at block %d: non-waitable StartBufferIO returns false, %d",
+ cur_buf_hdr->tag.blockNum,
+ may_block);
+
+ /*
+ * FIXME: can't tell whether this is because the buffer has
+ * been cleaned
+ */
+ if (!may_block)
+ {
+ result |= BUF_CANT_MERGE;
+ Assert(to_write->nbuffers > 0);
+ }
+ LWLockRelease(content_lock);
+ UnpinBuffer(cur_buf_hdr);
+
+ return result;
+ }
+ }
+ }
+ else
+ {
+ if (!StartBufferIO(cur_buf_hdr, false, false))
+ {
+ elog(DEBUG2, "waitable StartBufferIO returns false");
+ LWLockRelease(content_lock);
+ UnpinBuffer(cur_buf_hdr);
+
+ /*
+ * FIXME: Historically we returned BUF_WRITTEN in this case, which
+ * seems wrong
+ */
+ return result;
+ }
+ }
/*
- * SyncOneBuffer() is only called by checkpointer and bgwriter, so
- * IOContext will always be IOCONTEXT_NORMAL.
+ * Run PageGetLSN while holding header lock, since we don't have the
+ * buffer locked exclusively in all cases.
*/
- ScheduleBufferTagForWriteback(wb_context, IOCONTEXT_NORMAL, &tag);
+ buf_state = LockBufHdr(cur_buf_hdr);
+
+ cur_buf_lsn = BufferGetLSN(cur_buf_hdr);
+
+ /* To check if block content changes while flushing. - vadim 01/17/97 */
+ buf_state &= ~BM_JUST_DIRTIED;
+
+ UnlockBufHdr(cur_buf_hdr, buf_state);
+
+ to_write->buffers[to_write->nbuffers] = buf;
+ to_write->nbuffers++;
+
+ if (buf_state & BM_PERMANENT &&
+ (to_write->max_lsn == InvalidXLogRecPtr || to_write->max_lsn < cur_buf_lsn))
+ {
+ to_write->max_lsn = cur_buf_lsn;
+ }
+
+ result |= BUF_WRITTEN;
+
+ return result;
+}
+
+static void
+WriteBuffers(BuffersToWrite *to_write,
+ IOQueue *ioq, WritebackContext *wb_context)
+{
+ SMgrRelation smgr;
+ Buffer first_buf;
+ BufferDesc *first_buf_hdr;
+ bool needs_checksum;
+
+ Assert(to_write->nbuffers > 0 && to_write->nbuffers <= io_combine_limit);
+
+ first_buf = to_write->buffers[0];
+ first_buf_hdr = GetBufferDescriptor(first_buf - 1);
+
+ smgr = smgropen(BufTagGetRelFileLocator(&first_buf_hdr->tag), INVALID_PROC_NUMBER);
+
+ /*
+ * Force XLOG flush up to buffer's LSN. This implements the basic WAL
+ * rule that log updates must hit disk before any of the data-file changes
+ * they describe do.
+ *
+ * However, this rule does not apply to unlogged relations, which will be
+ * lost after a crash anyway. Most unlogged relation pages do not bear
+ * LSNs since we never emit WAL records for them, and therefore flushing
+ * up through the buffer LSN would be useless, but harmless. However,
+ * GiST indexes use LSNs internally to track page-splits, and therefore
+ * unlogged GiST pages bear "fake" LSNs generated by
+ * GetFakeLSNForUnloggedRel. It is unlikely but possible that the fake
+ * LSN counter could advance past the WAL insertion point; and if it did
+ * happen, attempting to flush WAL through that location would fail, with
+ * disastrous system-wide consequences. To make sure that can't happen,
+ * skip the flush if the buffer isn't permanent.
+ */
+ if (to_write->max_lsn != InvalidXLogRecPtr)
+ XLogFlush(to_write->max_lsn);
+
+ /*
+ * Now it's safe to write buffer to disk. Note that no one else should
+ * have been able to write it while we were busy with log flushing because
+ * only one process at a time can set the BM_IO_IN_PROGRESS bit.
+ */
+
+ for (int nbuf = 0; nbuf < to_write->nbuffers; nbuf++)
+ {
+ Buffer cur_buf = to_write->buffers[nbuf];
+ BufferDesc *cur_buf_hdr = GetBufferDescriptor(cur_buf - 1);
+ Block bufBlock;
+ char *bufToWrite;
+
+ bufBlock = BufHdrGetBlock(cur_buf_hdr);
+ needs_checksum = PageNeedsChecksumCopy((Page) bufBlock);
+
+ /*
+ * Update page checksum if desired. Since we have only shared lock on
+ * the buffer, other processes might be updating hint bits in it, so
+ * we must copy the page to a bounce buffer if we do checksumming.
+ */
+ if (needs_checksum)
+ {
+ PgAioBounceBuffer *bb = pgaio_bounce_buffer_get();
+
+ pgaio_io_assoc_bounce_buffer(to_write->ioh, bb);
+
+ bufToWrite = pgaio_bounce_buffer_buffer(bb);
+ memcpy(bufToWrite, bufBlock, BLCKSZ);
+ PageSetChecksumInplace((Page) bufToWrite, cur_buf_hdr->tag.blockNum);
+ }
+ else
+ {
+ bufToWrite = bufBlock;
+ }
+
+ to_write->data_ptrs[nbuf] = bufToWrite;
+ }
+
+ pgaio_io_set_handle_data_32(to_write->ioh,
+ (uint32 *) to_write->buffers,
+ to_write->nbuffers);
+ pgaio_io_register_callbacks(to_write->ioh, PGAIO_HCB_SHARED_BUFFER_WRITEV, 0);
+
+ smgrstartwritev(to_write->ioh, smgr,
+ BufTagGetForkNum(&first_buf_hdr->tag),
+ first_buf_hdr->tag.blockNum,
+ to_write->data_ptrs,
+ to_write->nbuffers,
+ false);
+ pgstat_count_io_op(IOOBJECT_RELATION, IOCONTEXT_NORMAL,
+ IOOP_WRITE, 1, BLCKSZ * to_write->nbuffers);
+
+
+ for (int nbuf = 0; nbuf < to_write->nbuffers; nbuf++)
+ {
+ Buffer cur_buf = to_write->buffers[nbuf];
+ BufferDesc *cur_buf_hdr = GetBufferDescriptor(cur_buf - 1);
+
+ UnpinBuffer(cur_buf_hdr);
+ }
+
+ io_queue_track(ioq, &to_write->iow);
+ to_write->total_writes++;
- return result | BUF_WRITTEN;
+ /* clear state for next write */
+ to_write->nbuffers = 0;
+ to_write->start_at_tag.relNumber = InvalidOid;
+ to_write->start_at_tag.blockNum = InvalidBlockNumber;
+ to_write->max_combine = 0;
+ to_write->max_lsn = InvalidXLogRecPtr;
+ to_write->ioh = NULL;
+ pgaio_wref_clear(&to_write->iow);
}
/*
@@ -4326,6 +4805,7 @@ FlushBuffer(BufferDesc *buf, SMgrRelation reln, IOObject io_object,
error_context_stack = errcallback.previous;
}
+
/*
* RelationGetNumberOfBlocksInFork
* Determines the current number of pages in the specified relation fork.
diff --git a/src/backend/storage/page/bufpage.c b/src/backend/storage/page/bufpage.c
index ecc81aacfc3..71dc6f559dd 100644
--- a/src/backend/storage/page/bufpage.c
+++ b/src/backend/storage/page/bufpage.c
@@ -1480,6 +1480,16 @@ PageIndexTupleOverwrite(Page page, OffsetNumber offnum,
return true;
}
+bool
+PageNeedsChecksumCopy(Page page)
+{
+ if (PageIsNew(page))
+ return false;
+
+ /* If we don't need a checksum, just return the passed-in data */
+ return DataChecksumsEnabled();
+}
+
/*
* Set checksum for a page in shared buffers.
diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list
index 26c622824e4..26a9c1d4ba6 100644
--- a/src/tools/pgindent/typedefs.list
+++ b/src/tools/pgindent/typedefs.list
@@ -348,6 +348,7 @@ BufferManagerRelation
BufferStrategyControl
BufferTag
BufferUsage
+BuffersToWrite
BuildAccumulator
BuiltinScript
BulkInsertState
--
2.48.1.76.g4e746b1a31.dirty
[text/x-diff] v2.9-0026-Ensure-a-resowner-exists-for-all-paths-that-may.patch (2.6K, ../../oav52ytakcvfnk52w5o5wvxdjavxth2tgptxt25gnfmh2pdcpk@omp3xukdu5if/27-v2.9-0026-Ensure-a-resowner-exists-for-all-paths-that-may.patch)
download | inline diff:
From 2800508280cf377fd5e39ae064357e9d79ad9807 Mon Sep 17 00:00:00 2001
From: Andres Freund <andres@anarazel.de>
Date: Sat, 15 Mar 2025 12:29:58 -0400
Subject: [PATCH v2.9 26/30] Ensure a resowner exists for all paths that may
perform AIO
Reviewed-by: Noah Misch <noah@leadboat.com>
Discussion: https://postgr.es/m/1f6b50a7-38ef-4d87-8246-786d39f46ab9@iki.fi
---
src/backend/bootstrap/bootstrap.c | 7 +++++++
src/backend/replication/logical/logical.c | 6 ++++++
src/backend/utils/init/postinit.c | 6 +++++-
3 files changed, 18 insertions(+), 1 deletion(-)
diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c
index 6db864892d0..e554504e1f0 100644
--- a/src/backend/bootstrap/bootstrap.c
+++ b/src/backend/bootstrap/bootstrap.c
@@ -361,8 +361,15 @@ BootstrapModeMain(int argc, char *argv[], bool check_only)
BaseInit();
bootstrap_signals();
+
+ /* need a resowner for IO during BootStrapXLOG() */
+ CreateAuxProcessResourceOwner();
+
BootStrapXLOG(bootstrap_data_checksum_version);
+ ReleaseAuxProcessResources(true);
+ CurrentResourceOwner = NULL;
+
/*
* To ensure that src/common/link-canary.c is linked into the backend, we
* must call it from somewhere. Here is as good as anywhere.
diff --git a/src/backend/replication/logical/logical.c b/src/backend/replication/logical/logical.c
index 8ea846bfc3b..2b936714040 100644
--- a/src/backend/replication/logical/logical.c
+++ b/src/backend/replication/logical/logical.c
@@ -386,6 +386,12 @@ CreateInitDecodingContext(const char *plugin,
slot->data.plugin = plugin_name;
SpinLockRelease(&slot->mutex);
+ if (CurrentResourceOwner == NULL)
+ {
+ Assert(am_walsender);
+ CurrentResourceOwner = AuxProcessResourceOwner;
+ }
+
if (XLogRecPtrIsInvalid(restart_lsn))
ReplicationSlotReserveWal();
else
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index 7958ea11b73..222e24bcb08 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -785,8 +785,12 @@ InitPostgres(const char *in_dbname, Oid dboid,
* We don't yet have an aux-process resource owner, but StartupXLOG
* and ShutdownXLOG will need one. Hence, create said resource owner
* (and register a callback to clean it up after ShutdownXLOG runs).
+ *
+ * In bootstrap mode CreateAuxProcessResourceOwner() was already
+ * called in BootstrapModeMain().
*/
- CreateAuxProcessResourceOwner();
+ if (!bootstrap)
+ CreateAuxProcessResourceOwner();
StartupXLOG();
/* Release (and warn about) any buffer pins leaked in StartupXLOG */
--
2.48.1.76.g4e746b1a31.dirty
[text/x-diff] v2.9-0027-Temporary-Increase-BAS_BULKREAD-size.patch (1.3K, ../../oav52ytakcvfnk52w5o5wvxdjavxth2tgptxt25gnfmh2pdcpk@omp3xukdu5if/28-v2.9-0027-Temporary-Increase-BAS_BULKREAD-size.patch)
download | inline diff:
From 4c148856b1c38d69882a4d867a9343d758724218 Mon Sep 17 00:00:00 2001
From: Andres Freund <andres@anarazel.de>
Date: Sat, 15 Mar 2025 12:29:58 -0400
Subject: [PATCH v2.9 27/30] Temporary: Increase BAS_BULKREAD size
Without this we only can execute very little AIO for sequential scans, as
there's just not enough buffers in the ring. This isn't the right fix, as
just increasing the ring size can have negative performance implications in
workloads where the kernel has all the data cached.
Author:
Reviewed-By:
Discussion: https://postgr.es/m/
Backpatch:
---
src/backend/storage/buffer/freelist.c | 7 ++++++-
1 file changed, 6 insertions(+), 1 deletion(-)
diff --git a/src/backend/storage/buffer/freelist.c b/src/backend/storage/buffer/freelist.c
index 336715b6c63..b72a5957a20 100644
--- a/src/backend/storage/buffer/freelist.c
+++ b/src/backend/storage/buffer/freelist.c
@@ -555,7 +555,12 @@ GetAccessStrategy(BufferAccessStrategyType btype)
return NULL;
case BAS_BULKREAD:
- ring_size_kb = 256;
+
+ /*
+ * FIXME: Temporary increase to allow large enough streaming reads
+ * to actually benefit from AIO. This needs a better solution.
+ */
+ ring_size_kb = 2 * 1024;
break;
case BAS_BULKWRITE:
ring_size_kb = 16 * 1024;
--
2.48.1.76.g4e746b1a31.dirty
[text/x-diff] v2.9-0028-WIP-Use-MAP_POPULATE.patch (1.1K, ../../oav52ytakcvfnk52w5o5wvxdjavxth2tgptxt25gnfmh2pdcpk@omp3xukdu5if/29-v2.9-0028-WIP-Use-MAP_POPULATE.patch)
download | inline diff:
From 01750e92f72263b959fab2037269069fab3b0e3f Mon Sep 17 00:00:00 2001
From: Andres Freund <andres@anarazel.de>
Date: Sat, 15 Mar 2025 12:29:58 -0400
Subject: [PATCH v2.9 28/30] WIP: Use MAP_POPULATE
For benchmarking it's quite annoying that the first time a memory is touched
has completely different perf characteristics than subsequent accesses. Using
MAP_POPULATE reduces that substantially.
---
src/backend/port/sysv_shmem.c | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/src/backend/port/sysv_shmem.c b/src/backend/port/sysv_shmem.c
index 197926d44f6..a700b02d5a1 100644
--- a/src/backend/port/sysv_shmem.c
+++ b/src/backend/port/sysv_shmem.c
@@ -620,7 +620,7 @@ CreateAnonymousSegment(Size *size)
allocsize += hugepagesize - (allocsize % hugepagesize);
ptr = mmap(NULL, allocsize, PROT_READ | PROT_WRITE,
- PG_MMAP_FLAGS | mmap_flags, -1, 0);
+ PG_MMAP_FLAGS | MAP_POPULATE | mmap_flags, -1, 0);
mmap_errno = errno;
if (huge_pages == HUGE_PAGES_TRY && ptr == MAP_FAILED)
elog(DEBUG1, "mmap(%zu) with MAP_HUGETLB failed, huge pages disabled: %m",
--
2.48.1.76.g4e746b1a31.dirty
^ permalink raw reply [nested|flat] 153+ messages in thread
* Re: AIO v2.5
2025-03-04 19:00 AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-10 18:23 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-11 19:41 ` Re: AIO v2.5 Noah Misch <noah@leadboat.com>
2025-03-11 23:55 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-12 03:57 ` Re: AIO v2.5 Noah Misch <noah@leadboat.com>
2025-03-12 17:06 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-12 23:23 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-13 10:53 ` Re: AIO v2.5 Antonin Houska <ah@cybertec.at>
2025-03-14 19:43 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-17 20:00 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
@ 2025-03-18 20:12 ` Andres Freund <andres@anarazel.de>
2025-03-19 01:00 ` Re: AIO v2.5 Melanie Plageman <melanieplageman@gmail.com>
2025-03-19 14:47 ` Re: AIO v2.5 Melanie Plageman <melanieplageman@gmail.com>
2025-03-19 17:20 ` Re: AIO v2.5 Melanie Plageman <melanieplageman@gmail.com>
2025-03-19 21:25 ` Re: AIO v2.5 Noah Misch <noah@leadboat.com>
2025-03-20 08:39 ` Re: AIO v2.5 Jakub Wartak <jakub.wartak@enterprisedb.com>
2025-03-21 01:58 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-04-22 22:38 ` Re: AIO v2.5 Tomas Vondra <tomas@vondra.me>
0 siblings, 7 replies; 153+ messages in thread
From: Andres Freund @ 2025-03-18 20:12 UTC (permalink / raw)
To: Antonin Houska <ah@cybertec.at>; +Cc: Noah Misch <noah@leadboat.com>; pgsql-hackers@postgresql.org, Thomas Munro <thomas.munro@gmail.com>; Heikki Linnakangas <hlinnaka@iki.fi>; Robert Haas <robertmhaas@gmail.com>; Jakub Wartak <jakub.wartak@enterprisedb.com>; Jelte Fennema-Nio <postgres@jeltef.nl>
Hi,
Attached is v2.10, with the following changes:
- committed core AIO infrastructure patch
A few cosmetic changes
- committed io_method=worker
Two non-cosmetic changes:
- The shmem allocation functions over-estimated the amount of shared memory
required.
- pgaio_worker_shmem_init() should initialize up to MAX_IO_WORKERS, not just
io_workers, since the latter is intentionally PGC_SIGHUP (found by Thomas)
- Bunch of typo fixes found by searching for repeated words
Thomas found one and then I searched for more.
- Added Melanie's patch to allow effective_io_concurrency to be set on windows etc
- Fixed a reference to an outdated function reference (thanks to Bilal)
- Reordered patches to be a bit more in dependency order
E.g. "bufmgr: Implement AIO read support" doesn't depend on Thomas' "buffer
forwarding" patches and thus can be commited before those go in.
Next steps:
- Decide what to do about the smgr interrupt issue
I guess it could be deferred, based on the argument it only matters with a
sufficiently high debug level, but I don't feel comfortable with that.
I think it'd be reasonable to just go with the patch I sent on the other
thread (and included here).
- Get somebody to look at
"bufmgr: Improve stats when buffer was read in concurrently"
This arguably fixes a bug, or just weird behaviour, on master.
- Address the set_max_safe_fds() issue and once resolved, commit io_uring
method
Can happen concurrently with next steps
- Commit "aio: Implement support for reads in smgr/md/fd"
- Get somebody to do one more pass at bufmgr related commits, I think they
could use a less in-the-weeds eye.
That's the following commits:
- localbuf: Track pincount in BufferDesc as well
- bufmgr: Implement AIO read support
- bufmgr: Use AIO in StartReadBuffers()
- aio: Basic read_stream adjustments for real AIO
Questions / Unresolved:
- Write support isn't going to land in 18, but there is a tiny bit of code
regarding writes in the code for bufmgr IO. I guess I could move that to a
later commit?
I'm inclined to leave it, the structure of the code only really makes
knowing that it's going to be shared between reads & writes.
- pg_aios view name
Greetings,
Andres Freund
Attachments:
[text/x-diff] v2.10-0001-smgr-Hold-interrupts-in-most-smgr-functions.patch (10.2K, ../../zp7obmaqdd6b5u2vhpw27hovpjirtqvjakkbkaz4vju7tohyul@4kfobe7ipgxm/2-v2.10-0001-smgr-Hold-interrupts-in-most-smgr-functions.patch)
download | inline diff:
From 47448e59a3d4d33267b10d1efd964ad358164d47 Mon Sep 17 00:00:00 2001
From: Andres Freund <andres@anarazel.de>
Date: Tue, 18 Mar 2025 14:40:05 -0400
Subject: [PATCH v2.10 01/28] smgr: Hold interrupts in most smgr functions
We need to hold interrupts across most of the smgr.c/md.c functions, as
otherwise interrupt processing, e.g. due to a debug elog/ereport, can trigger
procsignal processing, which in turn can trigger smgrreleaseall(). As the
relevant code is not reentrant, we quickly end up in a bad situation.
The only reason we haven't noticed this before is that there is only one
non-error ereport called in affected routines, in register_dirty_segments(),
and that one is extremely rarely reached. If one enables fd.c's FDDEBUG it's
easy to reproduce crashes.
It seems better to put the HOLD_INTERRUPTS()/RESUME_INTERRUPTS() in smgr.c,
instead of trying to push them down to md.c where possible: For one, every
smgr implementation would be vulnerable, for another, a good bit of smgr.c
code itself is affected too.
Eventually we might want a more targeted solution, allowing e.g. a networked
smgr implementation to be interrupted, but many other, more complicated,
problems would need to be fixed for that to be viable (e.g. smgr.c is often
called with interrupts already held).
Discussion: https://postgr.es/m/3vae7l5ozvqtxmd7rr7zaeq3qkuipz365u3rtim5t5wdkr6f4g@vkgf2fogjirl
---
src/backend/storage/smgr/smgr.c | 97 ++++++++++++++++++++++++++++++++-
1 file changed, 94 insertions(+), 3 deletions(-)
diff --git a/src/backend/storage/smgr/smgr.c b/src/backend/storage/smgr/smgr.c
index e6cbb9b4ca2..24971304b85 100644
--- a/src/backend/storage/smgr/smgr.c
+++ b/src/backend/storage/smgr/smgr.c
@@ -40,6 +40,18 @@
* themselves, as there could pointers to them in active use. See
* smgrrelease() and smgrreleaseall().
*
+ * NB: We need to hold interrupts across most of the functions in this file,
+ * as otherwise interrupt processing, e.g. due to a debug elog/ereport, can
+ * trigger procsignal processing, which in turn can trigger
+ * smgrreleaseall(). None of the relevant code is reentrant. It seems better
+ * to put the HOLD_INTERRUPTS()/RESUME_INTERRUPTS() here, instead of trying to
+ * push them down to md.c where possible: For one, every smgr implementation
+ * would be vulnerable, for another, a good bit of smgr.c code itself is
+ * affected too. Eventually we might want a more targeted solution, allowing
+ * e.g. a networked smgr implementation to be interrupted, but many other,
+ * more complicated, problems would need to be fixed for that to be viable
+ * (e.g. smgr.c is often called with interrupts already held).
+ *
* Portions Copyright (c) 1996-2025, PostgreSQL Global Development Group
* Portions Copyright (c) 1994, Regents of the University of California
*
@@ -53,6 +65,7 @@
#include "access/xlogutils.h"
#include "lib/ilist.h"
+#include "miscadmin.h"
#include "storage/bufmgr.h"
#include "storage/ipc.h"
#include "storage/md.h"
@@ -158,12 +171,16 @@ smgrinit(void)
{
int i;
+ HOLD_INTERRUPTS();
+
for (i = 0; i < NSmgr; i++)
{
if (smgrsw[i].smgr_init)
smgrsw[i].smgr_init();
}
+ RESUME_INTERRUPTS();
+
/* register the shutdown proc */
on_proc_exit(smgrshutdown, 0);
}
@@ -176,11 +193,13 @@ smgrshutdown(int code, Datum arg)
{
int i;
+ HOLD_INTERRUPTS();
for (i = 0; i < NSmgr; i++)
{
if (smgrsw[i].smgr_shutdown)
smgrsw[i].smgr_shutdown();
}
+ RESUME_INTERRUPTS();
}
/*
@@ -206,6 +225,8 @@ smgropen(RelFileLocator rlocator, ProcNumber backend)
Assert(RelFileNumberIsValid(rlocator.relNumber));
+ HOLD_INTERRUPTS();
+
if (SMgrRelationHash == NULL)
{
/* First time through: initialize the hash table */
@@ -242,6 +263,8 @@ smgropen(RelFileLocator rlocator, ProcNumber backend)
smgrsw[reln->smgr_which].smgr_open(reln);
}
+ RESUME_INTERRUPTS();
+
return reln;
}
@@ -283,6 +306,8 @@ smgrdestroy(SMgrRelation reln)
Assert(reln->pincount == 0);
+ HOLD_INTERRUPTS();
+
for (forknum = 0; forknum <= MAX_FORKNUM; forknum++)
smgrsw[reln->smgr_which].smgr_close(reln, forknum);
@@ -292,6 +317,8 @@ smgrdestroy(SMgrRelation reln)
&(reln->smgr_rlocator),
HASH_REMOVE, NULL) == NULL)
elog(ERROR, "SMgrRelation hashtable corrupted");
+
+ RESUME_INTERRUPTS();
}
/*
@@ -302,12 +329,16 @@ smgrdestroy(SMgrRelation reln)
void
smgrrelease(SMgrRelation reln)
{
+ HOLD_INTERRUPTS();
+
for (ForkNumber forknum = 0; forknum <= MAX_FORKNUM; forknum++)
{
smgrsw[reln->smgr_which].smgr_close(reln, forknum);
reln->smgr_cached_nblocks[forknum] = InvalidBlockNumber;
}
reln->smgr_targblock = InvalidBlockNumber;
+
+ RESUME_INTERRUPTS();
}
/*
@@ -336,6 +367,8 @@ smgrdestroyall(void)
{
dlist_mutable_iter iter;
+ HOLD_INTERRUPTS();
+
/*
* Zap all unpinned SMgrRelations. We rely on smgrdestroy() to remove
* each one from the list.
@@ -347,6 +380,8 @@ smgrdestroyall(void)
smgrdestroy(rel);
}
+
+ RESUME_INTERRUPTS();
}
/*
@@ -362,12 +397,16 @@ smgrreleaseall(void)
if (SMgrRelationHash == NULL)
return;
+ HOLD_INTERRUPTS();
+
hash_seq_init(&status, SMgrRelationHash);
while ((reln = (SMgrRelation) hash_seq_search(&status)) != NULL)
{
smgrrelease(reln);
}
+
+ RESUME_INTERRUPTS();
}
/*
@@ -400,7 +439,13 @@ smgrreleaserellocator(RelFileLocatorBackend rlocator)
bool
smgrexists(SMgrRelation reln, ForkNumber forknum)
{
- return smgrsw[reln->smgr_which].smgr_exists(reln, forknum);
+ bool ret;
+
+ HOLD_INTERRUPTS();
+ ret = smgrsw[reln->smgr_which].smgr_exists(reln, forknum);
+ RESUME_INTERRUPTS();
+
+ return ret;
}
/*
@@ -413,7 +458,9 @@ smgrexists(SMgrRelation reln, ForkNumber forknum)
void
smgrcreate(SMgrRelation reln, ForkNumber forknum, bool isRedo)
{
+ HOLD_INTERRUPTS();
smgrsw[reln->smgr_which].smgr_create(reln, forknum, isRedo);
+ RESUME_INTERRUPTS();
}
/*
@@ -434,6 +481,8 @@ smgrdosyncall(SMgrRelation *rels, int nrels)
if (nrels == 0)
return;
+ HOLD_INTERRUPTS();
+
FlushRelationsAllBuffers(rels, nrels);
/*
@@ -449,6 +498,8 @@ smgrdosyncall(SMgrRelation *rels, int nrels)
smgrsw[which].smgr_immedsync(rels[i], forknum);
}
}
+
+ RESUME_INTERRUPTS();
}
/*
@@ -471,6 +522,8 @@ smgrdounlinkall(SMgrRelation *rels, int nrels, bool isRedo)
if (nrels == 0)
return;
+ HOLD_INTERRUPTS();
+
/*
* Get rid of any remaining buffers for the relations. bufmgr will just
* drop them without bothering to write the contents.
@@ -522,6 +575,8 @@ smgrdounlinkall(SMgrRelation *rels, int nrels, bool isRedo)
}
pfree(rlocators);
+
+ RESUME_INTERRUPTS();
}
@@ -538,6 +593,8 @@ void
smgrextend(SMgrRelation reln, ForkNumber forknum, BlockNumber blocknum,
const void *buffer, bool skipFsync)
{
+ HOLD_INTERRUPTS();
+
smgrsw[reln->smgr_which].smgr_extend(reln, forknum, blocknum,
buffer, skipFsync);
@@ -550,6 +607,8 @@ smgrextend(SMgrRelation reln, ForkNumber forknum, BlockNumber blocknum,
reln->smgr_cached_nblocks[forknum] = blocknum + 1;
else
reln->smgr_cached_nblocks[forknum] = InvalidBlockNumber;
+
+ RESUME_INTERRUPTS();
}
/*
@@ -563,6 +622,8 @@ void
smgrzeroextend(SMgrRelation reln, ForkNumber forknum, BlockNumber blocknum,
int nblocks, bool skipFsync)
{
+ HOLD_INTERRUPTS();
+
smgrsw[reln->smgr_which].smgr_zeroextend(reln, forknum, blocknum,
nblocks, skipFsync);
@@ -575,6 +636,8 @@ smgrzeroextend(SMgrRelation reln, ForkNumber forknum, BlockNumber blocknum,
reln->smgr_cached_nblocks[forknum] = blocknum + nblocks;
else
reln->smgr_cached_nblocks[forknum] = InvalidBlockNumber;
+
+ RESUME_INTERRUPTS();
}
/*
@@ -588,7 +651,13 @@ bool
smgrprefetch(SMgrRelation reln, ForkNumber forknum, BlockNumber blocknum,
int nblocks)
{
- return smgrsw[reln->smgr_which].smgr_prefetch(reln, forknum, blocknum, nblocks);
+ bool ret;
+
+ HOLD_INTERRUPTS();
+ ret = smgrsw[reln->smgr_which].smgr_prefetch(reln, forknum, blocknum, nblocks);
+ RESUME_INTERRUPTS();
+
+ return ret;
}
/*
@@ -601,7 +670,13 @@ uint32
smgrmaxcombine(SMgrRelation reln, ForkNumber forknum,
BlockNumber blocknum)
{
- return smgrsw[reln->smgr_which].smgr_maxcombine(reln, forknum, blocknum);
+ uint32 ret;
+
+ HOLD_INTERRUPTS();
+ ret = smgrsw[reln->smgr_which].smgr_maxcombine(reln, forknum, blocknum);
+ RESUME_INTERRUPTS();
+
+ return ret;
}
/*
@@ -619,8 +694,10 @@ void
smgrreadv(SMgrRelation reln, ForkNumber forknum, BlockNumber blocknum,
void **buffers, BlockNumber nblocks)
{
+ HOLD_INTERRUPTS();
smgrsw[reln->smgr_which].smgr_readv(reln, forknum, blocknum, buffers,
nblocks);
+ RESUME_INTERRUPTS();
}
/*
@@ -653,8 +730,10 @@ void
smgrwritev(SMgrRelation reln, ForkNumber forknum, BlockNumber blocknum,
const void **buffers, BlockNumber nblocks, bool skipFsync)
{
+ HOLD_INTERRUPTS();
smgrsw[reln->smgr_which].smgr_writev(reln, forknum, blocknum,
buffers, nblocks, skipFsync);
+ RESUME_INTERRUPTS();
}
/*
@@ -665,8 +744,10 @@ void
smgrwriteback(SMgrRelation reln, ForkNumber forknum, BlockNumber blocknum,
BlockNumber nblocks)
{
+ HOLD_INTERRUPTS();
smgrsw[reln->smgr_which].smgr_writeback(reln, forknum, blocknum,
nblocks);
+ RESUME_INTERRUPTS();
}
/*
@@ -683,10 +764,14 @@ smgrnblocks(SMgrRelation reln, ForkNumber forknum)
if (result != InvalidBlockNumber)
return result;
+ HOLD_INTERRUPTS();
+
result = smgrsw[reln->smgr_which].smgr_nblocks(reln, forknum);
reln->smgr_cached_nblocks[forknum] = result;
+ RESUME_INTERRUPTS();
+
return result;
}
@@ -731,6 +816,8 @@ smgrtruncate(SMgrRelation reln, ForkNumber *forknum, int nforks,
{
int i;
+ Assert(!INTERRUPTS_CAN_BE_PROCESSED());
+
/*
* Get rid of any buffers for the about-to-be-deleted blocks. bufmgr will
* just drop them without bothering to write the contents.
@@ -784,7 +871,9 @@ smgrtruncate(SMgrRelation reln, ForkNumber *forknum, int nforks,
void
smgrregistersync(SMgrRelation reln, ForkNumber forknum)
{
+ HOLD_INTERRUPTS();
smgrsw[reln->smgr_which].smgr_registersync(reln, forknum);
+ RESUME_INTERRUPTS();
}
/*
@@ -816,7 +905,9 @@ smgrregistersync(SMgrRelation reln, ForkNumber forknum)
void
smgrimmedsync(SMgrRelation reln, ForkNumber forknum)
{
+ HOLD_INTERRUPTS();
smgrsw[reln->smgr_which].smgr_immedsync(reln, forknum);
+ RESUME_INTERRUPTS();
}
/*
--
2.48.1.76.g4e746b1a31.dirty
[text/x-diff] v2.10-0002-bufmgr-Improve-stats-when-buffer-is-read-in-co.patch (3.4K, ../../zp7obmaqdd6b5u2vhpw27hovpjirtqvjakkbkaz4vju7tohyul@4kfobe7ipgxm/3-v2.10-0002-bufmgr-Improve-stats-when-buffer-is-read-in-co.patch)
download | inline diff:
From 59783c7585e36043ebb2384937226c054d42c766 Mon Sep 17 00:00:00 2001
From: Andres Freund <andres@anarazel.de>
Date: Tue, 18 Mar 2025 14:40:06 -0400
Subject: [PATCH v2.10 02/28] bufmgr: Improve stats when buffer is read in
concurrently
Previously we would count buffers that were read in concurently as being read
by the current backend, which leads to double-counting the read IOs
globally. Similarly, the buffer hit in that case was never attributed to
pg_stat_io and relation-level IO statistics, making cache hit ratios look
worse. We also did not account for the cache hit in vacuum cost balancing.
This behaviour has historically grown, but there doesn't seem to be any reason
to just continue down that path.
---
src/backend/storage/buffer/bufmgr.c | 41 ++++++++++++++++++-----------
1 file changed, 26 insertions(+), 15 deletions(-)
diff --git a/src/backend/storage/buffer/bufmgr.c b/src/backend/storage/buffer/bufmgr.c
index 79ca9d18d07..e76ff1f3224 100644
--- a/src/backend/storage/buffer/bufmgr.c
+++ b/src/backend/storage/buffer/bufmgr.c
@@ -1433,19 +1433,6 @@ WaitReadBuffers(ReadBuffersOperation *operation)
io_object = IOOBJECT_RELATION;
}
- /*
- * We count all these blocks as read by this backend. This is traditional
- * behavior, but might turn out to be not true if we find that someone
- * else has beaten us and completed the read of some of these blocks. In
- * that case the system globally double-counts, but we traditionally don't
- * count this as a "hit", and we don't have a separate counter for "miss,
- * but another backend completed the read".
- */
- if (persistence == RELPERSISTENCE_TEMP)
- pgBufferUsage.local_blks_read += nblocks;
- else
- pgBufferUsage.shared_blks_read += nblocks;
-
for (int i = 0; i < nblocks; ++i)
{
int io_buffers_len;
@@ -1463,8 +1450,13 @@ WaitReadBuffers(ReadBuffersOperation *operation)
if (!WaitReadBuffersCanStartIO(buffers[i], false))
{
/*
- * Report this as a 'hit' for this backend, even though it must
- * have started out as a miss in PinBufferForBlock().
+ * Report and track this as a 'hit' for this backend, even though
+ * it must have started out as a miss in PinBufferForBlock().
+ *
+ * Some of the accesses would otherwise never be counted (e.g.
+ * pgBufferUsage) or counted as a miss (e.g.
+ * pgstat_count_buffer_hit(), as we always call
+ * pgstat_count_buffer_read()).
*/
TRACE_POSTGRESQL_BUFFER_READ_DONE(forknum, blocknum + i,
operation->smgr->smgr_rlocator.locator.spcOid,
@@ -1472,6 +1464,20 @@ WaitReadBuffers(ReadBuffersOperation *operation)
operation->smgr->smgr_rlocator.locator.relNumber,
operation->smgr->smgr_rlocator.backend,
true);
+
+ if (persistence == RELPERSISTENCE_TEMP)
+ pgBufferUsage.local_blks_hit += 1;
+ else
+ pgBufferUsage.shared_blks_hit += 1;
+
+ if (operation->rel)
+ pgstat_count_buffer_hit(operation->rel);
+
+ pgstat_count_io_op(io_object, io_context, IOOP_HIT, 1, 0);
+
+ if (VacuumCostActive)
+ VacuumCostBalance += VacuumCostPageHit;
+
continue;
}
@@ -1557,6 +1563,11 @@ WaitReadBuffers(ReadBuffersOperation *operation)
false);
}
+ if (persistence == RELPERSISTENCE_TEMP)
+ pgBufferUsage.local_blks_read += io_buffers_len;
+ else
+ pgBufferUsage.shared_blks_read += io_buffers_len;
+
if (VacuumCostActive)
VacuumCostBalance += VacuumCostPageMiss * io_buffers_len;
}
--
2.48.1.76.g4e746b1a31.dirty
[text/x-diff] v2.10-0003-aio-Add-liburing-dependency.patch (13.4K, ../../zp7obmaqdd6b5u2vhpw27hovpjirtqvjakkbkaz4vju7tohyul@4kfobe7ipgxm/4-v2.10-0003-aio-Add-liburing-dependency.patch)
download | inline diff:
From 81dd8ccc81c4173228c0b7b9a3088fd1fb7b0a52 Mon Sep 17 00:00:00 2001
From: Andres Freund <andres@anarazel.de>
Date: Tue, 18 Mar 2025 14:40:05 -0400
Subject: [PATCH v2.10 03/28] aio: Add liburing dependency
Will be used in a subsequent commit, to implement io_method=io_uring. Kept
separate for easier review.
Reviewed-by: Noah Misch <noah@leadboat.com>
Discussion: https://postgr.es/m/uvrtrknj4kdytuboidbhwclo4gxhswwcpgadptsjvjqcluzmah%40brqs62irg4dt
---
meson.build | 14 ++++
meson_options.txt | 3 +
configure.ac | 11 +++
src/makefiles/meson.build | 3 +
src/include/pg_config.h.in | 3 +
src/backend/Makefile | 7 +-
doc/src/sgml/installation.sgml | 34 ++++++++
configure | 138 +++++++++++++++++++++++++++++++++
.cirrus.tasks.yml | 1 +
src/Makefile.global.in | 4 +
10 files changed, 215 insertions(+), 3 deletions(-)
diff --git a/meson.build b/meson.build
index 13c13748e5d..bed36153c7b 100644
--- a/meson.build
+++ b/meson.build
@@ -950,6 +950,18 @@ endif
+###############################################################
+# Library: liburing
+###############################################################
+
+liburingopt = get_option('liburing')
+liburing = dependency('liburing', required: liburingopt)
+if liburing.found()
+ cdata.set('USE_LIBURING', 1)
+endif
+
+
+
###############################################################
# Library: libxml
###############################################################
@@ -3168,6 +3180,7 @@ backend_both_deps += [
icu_i18n,
ldap,
libintl,
+ liburing,
libxml,
lz4,
pam,
@@ -3823,6 +3836,7 @@ if meson.version().version_compare('>=0.57')
'icu': icu,
'ldap': ldap,
'libcurl': libcurl,
+ 'liburing': liburing,
'libxml': libxml,
'libxslt': libxslt,
'llvm': llvm,
diff --git a/meson_options.txt b/meson_options.txt
index 702c4517145..dd7126da3a7 100644
--- a/meson_options.txt
+++ b/meson_options.txt
@@ -106,6 +106,9 @@ option('libcurl', type : 'feature', value: 'auto',
option('libedit_preferred', type: 'boolean', value: false,
description: 'Prefer BSD Libedit over GNU Readline')
+option('liburing', type : 'feature', value: 'auto',
+ description: 'io_uring support, for asynchronous I/O')
+
option('libxml', type: 'feature', value: 'auto',
description: 'XML support')
diff --git a/configure.ac b/configure.ac
index b6d02f5ecc7..3e2d0dc59c0 100644
--- a/configure.ac
+++ b/configure.ac
@@ -975,6 +975,14 @@ AC_SUBST(with_readline)
PGAC_ARG_BOOL(with, libedit-preferred, no,
[prefer BSD Libedit over GNU Readline])
+#
+# liburing
+#
+AC_MSG_CHECKING([whether to build with liburing support])
+PGAC_ARG_BOOL(with, liburing, no, [io_uring support, for asynchronous I/O],
+ [AC_DEFINE([USE_LIBURING], 1, [Define to build with io_uring support. (--with-liburing)])])
+AC_MSG_RESULT([$with_liburing])
+AC_SUBST(with_liburing)
#
# UUID library
@@ -1463,6 +1471,9 @@ elif test "$with_uuid" = ossp ; then
fi
AC_SUBST(UUID_LIBS)
+if test "$with_liburing" = yes; then
+ PKG_CHECK_MODULES(LIBURING, liburing)
+fi
##
## Header files
diff --git a/src/makefiles/meson.build b/src/makefiles/meson.build
index 60e13d50235..46d8da070e8 100644
--- a/src/makefiles/meson.build
+++ b/src/makefiles/meson.build
@@ -199,6 +199,8 @@ pgxs_empty = [
'PTHREAD_CFLAGS', 'PTHREAD_LIBS',
'ICU_LIBS',
+
+ 'LIBURING_CFLAGS', 'LIBURING_LIBS',
]
if host_system == 'windows' and cc.get_argument_syntax() != 'msvc'
@@ -230,6 +232,7 @@ pgxs_deps = {
'icu': icu,
'ldap': ldap,
'libcurl': libcurl,
+ 'liburing': liburing,
'libxml': libxml,
'libxslt': libxslt,
'llvm': llvm,
diff --git a/src/include/pg_config.h.in b/src/include/pg_config.h.in
index db6454090d2..f2422241133 100644
--- a/src/include/pg_config.h.in
+++ b/src/include/pg_config.h.in
@@ -672,6 +672,9 @@
/* Define to 1 to build with libcurl support. (--with-libcurl) */
#undef USE_LIBCURL
+/* Define to build with io_uring support. (--with-liburing) */
+#undef USE_LIBURING
+
/* Define to 1 to build with XML support. (--with-libxml) */
#undef USE_LIBXML
diff --git a/src/backend/Makefile b/src/backend/Makefile
index 42d4a28e5aa..7344c8c7f5c 100644
--- a/src/backend/Makefile
+++ b/src/backend/Makefile
@@ -43,9 +43,10 @@ OBJS = \
$(top_builddir)/src/common/libpgcommon_srv.a \
$(top_builddir)/src/port/libpgport_srv.a
-# We put libpgport and libpgcommon into OBJS, so remove it from LIBS; also add
-# libldap and ICU
-LIBS := $(filter-out -lpgport -lpgcommon, $(LIBS)) $(LDAP_LIBS_BE) $(ICU_LIBS)
+# We put libpgport and libpgcommon into OBJS, so remove it from LIBS.
+LIBS := $(filter-out -lpgport -lpgcommon, $(LIBS))
+# The backend conditionally needs libraries that most executables don't need.
+LIBS += $(LDAP_LIBS_BE) $(ICU_LIBS) $(LIBURING_LIBS)
# The backend doesn't need everything that's in LIBS, however
LIBS := $(filter-out -lreadline -ledit -ltermcap -lncurses -lcurses, $(LIBS))
diff --git a/doc/src/sgml/installation.sgml b/doc/src/sgml/installation.sgml
index e076cefa3b9..cc28f041330 100644
--- a/doc/src/sgml/installation.sgml
+++ b/doc/src/sgml/installation.sgml
@@ -1156,6 +1156,24 @@ build-postgresql:
</listitem>
</varlistentry>
+ <varlistentry id="configure-option-with-liburing">
+ <term><option>--with-liburing</option></term>
+ <listitem>
+ <para>
+ Build with liburing, enabling io_uring support for asynchronous I/O.
+ </para>
+ <para>
+ To detect the required compiler and linker options, PostgreSQL will
+ query <command>pkg-config</command>.
+ </para>
+ <para>
+ To use a liburing installation that is in an unusual location, you
+ can set <command>pkg-config</command>-related environment
+ variables (see its documentation).
+ </para>
+ </listitem>
+ </varlistentry>
+
<varlistentry id="configure-option-with-libxml">
<term><option>--with-libxml</option></term>
<listitem>
@@ -2611,6 +2629,22 @@ ninja install
</listitem>
</varlistentry>
+ <varlistentry id="configure-with-liburing-meson">
+ <term><option>-Dliburing={ auto | enabled | disabled }</option></term>
+ <listitem>
+ <para>
+ Build with liburing, enabling io_uring support for asynchronous I/O.
+ Defaults to auto.
+ </para>
+
+ <para>
+ To use a liburing installation that is in an unusual location, you
+ can set <command>pkg-config</command>-related environment
+ variables (see its documentation).
+ </para>
+ </listitem>
+ </varlistentry>
+
<varlistentry id="configure-with-libxml-meson">
<term><option>-Dlibxml={ auto | enabled | disabled }</option></term>
<listitem>
diff --git a/configure b/configure
index 93fddd69981..8ffcdfc2e1c 100755
--- a/configure
+++ b/configure
@@ -651,6 +651,8 @@ LIBOBJS
OPENSSL
ZSTD
LZ4
+LIBURING_LIBS
+LIBURING_CFLAGS
UUID_LIBS
LDAP_LIBS_BE
LDAP_LIBS_FE
@@ -712,6 +714,7 @@ LIBCURL_LIBS
LIBCURL_CFLAGS
with_libcurl
with_uuid
+with_liburing
with_readline
with_systemd
with_selinux
@@ -865,6 +868,7 @@ with_selinux
with_systemd
with_readline
with_libedit_preferred
+with_liburing
with_uuid
with_ossp_uuid
with_libcurl
@@ -911,6 +915,8 @@ LDFLAGS_EX
LDFLAGS_SL
PERL
PYTHON
+LIBURING_CFLAGS
+LIBURING_LIBS
MSGFMT
TCLSH'
@@ -1578,6 +1584,7 @@ Optional Packages:
--without-readline do not use GNU Readline nor BSD Libedit for editing
--with-libedit-preferred
prefer BSD Libedit over GNU Readline
+ --with-liburing io_uring support, for asynchronous I/O
--with-uuid=LIB build contrib/uuid-ossp using LIB (bsd,e2fs,ossp)
--with-ossp-uuid obsolete spelling of --with-uuid=ossp
--with-libcurl build with libcurl support
@@ -1629,6 +1636,10 @@ Some influential environment variables:
LDFLAGS_SL extra linker flags for linking shared libraries only
PERL Perl program
PYTHON Python program
+ LIBURING_CFLAGS
+ C compiler flags for LIBURING, overriding pkg-config
+ LIBURING_LIBS
+ linker flags for LIBURING, overriding pkg-config
MSGFMT msgfmt program for NLS
TCLSH Tcl interpreter program (tclsh)
@@ -8692,6 +8703,40 @@ fi
+#
+# liburing
+#
+{ $as_echo "$as_me:${as_lineno-$LINENO}: checking whether to build with liburing support" >&5
+$as_echo_n "checking whether to build with liburing support... " >&6; }
+
+
+
+# Check whether --with-liburing was given.
+if test "${with_liburing+set}" = set; then :
+ withval=$with_liburing;
+ case $withval in
+ yes)
+
+$as_echo "#define USE_LIBURING 1" >>confdefs.h
+
+ ;;
+ no)
+ :
+ ;;
+ *)
+ as_fn_error $? "no argument expected for --with-liburing option" "$LINENO" 5
+ ;;
+ esac
+
+else
+ with_liburing=no
+
+fi
+
+
+{ $as_echo "$as_me:${as_lineno-$LINENO}: result: $with_liburing" >&5
+$as_echo "$with_liburing" >&6; }
+
#
# UUID library
@@ -13444,6 +13489,99 @@ fi
fi
+if test "$with_liburing" = yes; then
+
+pkg_failed=no
+{ $as_echo "$as_me:${as_lineno-$LINENO}: checking for liburing" >&5
+$as_echo_n "checking for liburing... " >&6; }
+
+if test -n "$LIBURING_CFLAGS"; then
+ pkg_cv_LIBURING_CFLAGS="$LIBURING_CFLAGS"
+ elif test -n "$PKG_CONFIG"; then
+ if test -n "$PKG_CONFIG" && \
+ { { $as_echo "$as_me:${as_lineno-$LINENO}: \$PKG_CONFIG --exists --print-errors \"liburing\""; } >&5
+ ($PKG_CONFIG --exists --print-errors "liburing") 2>&5
+ ac_status=$?
+ $as_echo "$as_me:${as_lineno-$LINENO}: \$? = $ac_status" >&5
+ test $ac_status = 0; }; then
+ pkg_cv_LIBURING_CFLAGS=`$PKG_CONFIG --cflags "liburing" 2>/dev/null`
+ test "x$?" != "x0" && pkg_failed=yes
+else
+ pkg_failed=yes
+fi
+ else
+ pkg_failed=untried
+fi
+if test -n "$LIBURING_LIBS"; then
+ pkg_cv_LIBURING_LIBS="$LIBURING_LIBS"
+ elif test -n "$PKG_CONFIG"; then
+ if test -n "$PKG_CONFIG" && \
+ { { $as_echo "$as_me:${as_lineno-$LINENO}: \$PKG_CONFIG --exists --print-errors \"liburing\""; } >&5
+ ($PKG_CONFIG --exists --print-errors "liburing") 2>&5
+ ac_status=$?
+ $as_echo "$as_me:${as_lineno-$LINENO}: \$? = $ac_status" >&5
+ test $ac_status = 0; }; then
+ pkg_cv_LIBURING_LIBS=`$PKG_CONFIG --libs "liburing" 2>/dev/null`
+ test "x$?" != "x0" && pkg_failed=yes
+else
+ pkg_failed=yes
+fi
+ else
+ pkg_failed=untried
+fi
+
+
+
+if test $pkg_failed = yes; then
+ { $as_echo "$as_me:${as_lineno-$LINENO}: result: no" >&5
+$as_echo "no" >&6; }
+
+if $PKG_CONFIG --atleast-pkgconfig-version 0.20; then
+ _pkg_short_errors_supported=yes
+else
+ _pkg_short_errors_supported=no
+fi
+ if test $_pkg_short_errors_supported = yes; then
+ LIBURING_PKG_ERRORS=`$PKG_CONFIG --short-errors --print-errors --cflags --libs "liburing" 2>&1`
+ else
+ LIBURING_PKG_ERRORS=`$PKG_CONFIG --print-errors --cflags --libs "liburing" 2>&1`
+ fi
+ # Put the nasty error message in config.log where it belongs
+ echo "$LIBURING_PKG_ERRORS" >&5
+
+ as_fn_error $? "Package requirements (liburing) were not met:
+
+$LIBURING_PKG_ERRORS
+
+Consider adjusting the PKG_CONFIG_PATH environment variable if you
+installed software in a non-standard prefix.
+
+Alternatively, you may set the environment variables LIBURING_CFLAGS
+and LIBURING_LIBS to avoid the need to call pkg-config.
+See the pkg-config man page for more details." "$LINENO" 5
+elif test $pkg_failed = untried; then
+ { $as_echo "$as_me:${as_lineno-$LINENO}: result: no" >&5
+$as_echo "no" >&6; }
+ { { $as_echo "$as_me:${as_lineno-$LINENO}: error: in \`$ac_pwd':" >&5
+$as_echo "$as_me: error: in \`$ac_pwd':" >&2;}
+as_fn_error $? "The pkg-config script could not be found or is too old. Make sure it
+is in your PATH or set the PKG_CONFIG environment variable to the full
+path to pkg-config.
+
+Alternatively, you may set the environment variables LIBURING_CFLAGS
+and LIBURING_LIBS to avoid the need to call pkg-config.
+See the pkg-config man page for more details.
+
+To get pkg-config, see <http://pkg-config.freedesktop.org/>.
+See \`config.log' for more details" "$LINENO" 5; }
+else
+ LIBURING_CFLAGS=$pkg_cv_LIBURING_CFLAGS
+ LIBURING_LIBS=$pkg_cv_LIBURING_LIBS
+ { $as_echo "$as_me:${as_lineno-$LINENO}: result: yes" >&5
+$as_echo "yes" >&6; }
+
+fi
+fi
##
## Header files
diff --git a/.cirrus.tasks.yml b/.cirrus.tasks.yml
index 5849cbb839a..10d277f5659 100644
--- a/.cirrus.tasks.yml
+++ b/.cirrus.tasks.yml
@@ -471,6 +471,7 @@ task:
--enable-cassert --enable-injection-points --enable-debug \
--enable-tap-tests --enable-nls \
--with-segsize-blocks=6 \
+ --with-liburing \
\
${LINUX_CONFIGURE_FEATURES} \
\
diff --git a/src/Makefile.global.in b/src/Makefile.global.in
index 3b620bac5ac..d45f5fe4ed7 100644
--- a/src/Makefile.global.in
+++ b/src/Makefile.global.in
@@ -191,6 +191,7 @@ with_gssapi = @with_gssapi@
with_krb_srvnam = @with_krb_srvnam@
with_ldap = @with_ldap@
with_libcurl = @with_libcurl@
+with_liburing = @with_liburing@
with_libxml = @with_libxml@
with_libxslt = @with_libxslt@
with_llvm = @with_llvm@
@@ -217,6 +218,9 @@ krb_srvtab = @krb_srvtab@
ICU_CFLAGS = @ICU_CFLAGS@
ICU_LIBS = @ICU_LIBS@
+LIBURING_CFLAGS = @LIBURING_CFLAGS@
+LIBURING_LIBS = @LIBURING_LIBS@
+
TCLSH = @TCLSH@
TCL_LIBS = @TCL_LIBS@
TCL_LIB_SPEC = @TCL_LIB_SPEC@
--
2.48.1.76.g4e746b1a31.dirty
[text/x-diff] v2.10-0004-aio-Add-io_method-io_uring.patch (20.2K, ../../zp7obmaqdd6b5u2vhpw27hovpjirtqvjakkbkaz4vju7tohyul@4kfobe7ipgxm/5-v2.10-0004-aio-Add-io_method-io_uring.patch)
download | inline diff:
From fcf8cb09e32be7e7c0b193e153085cc355b05c6a Mon Sep 17 00:00:00 2001
From: Andres Freund <andres@anarazel.de>
Date: Tue, 18 Mar 2025 14:40:05 -0400
Subject: [PATCH v2.10 04/28] aio: Add io_method=io_uring
io_uring, can be considerably faster than io_method=worker. It is, however,
linux specific and requires an additional compile-time dependency (liburing).
Reviewed-by: Jakub Wartak <jakub.wartak@enterprisedb.com>
Discussion: https://postgr.es/m/uvrtrknj4kdytuboidbhwclo4gxhswwcpgadptsjvjqcluzmah%40brqs62irg4dt
Discussion: https://postgr.es/m/20210223100344.llw5an2aklengrmn@alap3.anarazel.de
Discussion: https://postgr.es/m/stj36ea6yyhoxtqkhpieia2z4krnam7qyetc57rfezgk4zgapf@gcnactj4z56m
---
src/include/storage/aio.h | 3 +
src/include/storage/aio_internal.h | 3 +
src/include/storage/lwlock.h | 1 +
src/backend/storage/aio/Makefile | 1 +
src/backend/storage/aio/aio.c | 6 +
src/backend/storage/aio/meson.build | 1 +
src/backend/storage/aio/method_io_uring.c | 440 ++++++++++++++++++
src/backend/storage/lmgr/lwlock.c | 1 +
.../utils/activity/wait_event_names.txt | 2 +
src/backend/utils/misc/postgresql.conf.sample | 3 +-
doc/src/sgml/config.sgml | 6 +
.cirrus.tasks.yml | 3 +
src/tools/pgindent/typedefs.list | 1 +
13 files changed, 470 insertions(+), 1 deletion(-)
create mode 100644 src/backend/storage/aio/method_io_uring.c
diff --git a/src/include/storage/aio.h b/src/include/storage/aio.h
index 7b6b7d20a85..53b1080b17f 100644
--- a/src/include/storage/aio.h
+++ b/src/include/storage/aio.h
@@ -28,6 +28,9 @@ typedef enum IoMethod
{
IOMETHOD_SYNC = 0,
IOMETHOD_WORKER,
+#ifdef USE_LIBURING
+ IOMETHOD_IO_URING,
+#endif
} IoMethod;
/* We'll default to worker based execution. */
diff --git a/src/include/storage/aio_internal.h b/src/include/storage/aio_internal.h
index 108fe61c7b4..da4d9c341f0 100644
--- a/src/include/storage/aio_internal.h
+++ b/src/include/storage/aio_internal.h
@@ -386,6 +386,9 @@ extern PgAioHandle *pgaio_inj_io_get(void);
/* Declarations for the tables of function pointers exposed by each IO method. */
extern PGDLLIMPORT const IoMethodOps pgaio_sync_ops;
extern PGDLLIMPORT const IoMethodOps pgaio_worker_ops;
+#ifdef USE_LIBURING
+extern PGDLLIMPORT const IoMethodOps pgaio_uring_ops;
+#endif
extern PGDLLIMPORT const IoMethodOps *pgaio_method_ops;
extern PGDLLIMPORT PgAioCtl *pgaio_ctl;
diff --git a/src/include/storage/lwlock.h b/src/include/storage/lwlock.h
index ffa03189e2d..4df1d25c045 100644
--- a/src/include/storage/lwlock.h
+++ b/src/include/storage/lwlock.h
@@ -218,6 +218,7 @@ typedef enum BuiltinTrancheIds
LWTRANCHE_SUBTRANS_SLRU,
LWTRANCHE_XACT_SLRU,
LWTRANCHE_PARALLEL_VACUUM_DSA,
+ LWTRANCHE_AIO_URING_COMPLETION,
LWTRANCHE_FIRST_USER_DEFINED,
} BuiltinTrancheIds;
diff --git a/src/backend/storage/aio/Makefile b/src/backend/storage/aio/Makefile
index f51c34a37f8..c06c50771e0 100644
--- a/src/backend/storage/aio/Makefile
+++ b/src/backend/storage/aio/Makefile
@@ -14,6 +14,7 @@ OBJS = \
aio_init.o \
aio_io.o \
aio_target.o \
+ method_io_uring.o \
method_sync.o \
method_worker.o \
read_stream.o
diff --git a/src/backend/storage/aio/aio.c b/src/backend/storage/aio/aio.c
index 3ed4b1dfdac..2e210335872 100644
--- a/src/backend/storage/aio/aio.c
+++ b/src/backend/storage/aio/aio.c
@@ -65,6 +65,9 @@ static void pgaio_io_wait(PgAioHandle *ioh, uint64 ref_generation);
const struct config_enum_entry io_method_options[] = {
{"sync", IOMETHOD_SYNC, false},
{"worker", IOMETHOD_WORKER, false},
+#ifdef USE_LIBURING
+ {"io_uring", IOMETHOD_IO_URING, false},
+#endif
{NULL, 0, false}
};
@@ -82,6 +85,9 @@ PgAioBackend *pgaio_my_backend;
static const IoMethodOps *const pgaio_method_ops_table[] = {
[IOMETHOD_SYNC] = &pgaio_sync_ops,
[IOMETHOD_WORKER] = &pgaio_worker_ops,
+#ifdef USE_LIBURING
+ [IOMETHOD_IO_URING] = &pgaio_uring_ops,
+#endif
};
/* callbacks for the configured io_method, set by assign_io_method */
diff --git a/src/backend/storage/aio/meson.build b/src/backend/storage/aio/meson.build
index 74f94c6e40b..2f0f03d8071 100644
--- a/src/backend/storage/aio/meson.build
+++ b/src/backend/storage/aio/meson.build
@@ -6,6 +6,7 @@ backend_sources += files(
'aio_init.c',
'aio_io.c',
'aio_target.c',
+ 'method_io_uring.c',
'method_sync.c',
'method_worker.c',
'read_stream.c',
diff --git a/src/backend/storage/aio/method_io_uring.c b/src/backend/storage/aio/method_io_uring.c
new file mode 100644
index 00000000000..6360c5621d1
--- /dev/null
+++ b/src/backend/storage/aio/method_io_uring.c
@@ -0,0 +1,440 @@
+/*-------------------------------------------------------------------------
+ *
+ * method_io_uring.c
+ * AIO - perform AIO using Linux' io_uring
+ *
+ * For now we create one io_uring instance for each backend. These io_uring
+ * instances have to be created in postmaster, during startup, to allow other
+ * backends to process IO completions, if the issuing backend is currently
+ * busy doing other things. Other backends may not use another backend's
+ * io_uring instance to submit IO, that'd require additional locking that
+ * would likely be harmful for performance.
+ *
+ * We likely will want to introduce a backend-local io_uring instance in the
+ * future, e.g. for FE/BE network IO.
+ *
+ * Portions Copyright (c) 1996-2025, PostgreSQL Global Development Group
+ * Portions Copyright (c) 1994, Regents of the University of California
+ *
+ * IDENTIFICATION
+ * src/backend/storage/aio/method_io_uring.c
+ *
+ *-------------------------------------------------------------------------
+ */
+
+#include "postgres.h"
+
+#ifdef USE_LIBURING
+
+#include <liburing.h>
+
+#include "pgstat.h"
+#include "port/pg_iovec.h"
+#include "storage/aio.h"
+#include "storage/aio_internal.h"
+#include "storage/fd.h"
+#include "storage/proc.h"
+#include "storage/shmem.h"
+
+
+/* number of completions processed at once */
+#define PGAIO_MAX_LOCAL_COMPLETED_IO 32
+
+
+/* Entry points for IoMethodOps. */
+static size_t pgaio_uring_shmem_size(void);
+static void pgaio_uring_shmem_init(bool first_time);
+static void pgaio_uring_init_backend(void);
+static int pgaio_uring_submit(uint16 num_staged_ios, PgAioHandle **staged_ios);
+static void pgaio_uring_wait_one(PgAioHandle *ioh, uint64 ref_generation);
+
+/* helper functions */
+static void pgaio_uring_sq_from_io(PgAioHandle *ioh, struct io_uring_sqe *sqe);
+
+
+const IoMethodOps pgaio_uring_ops = {
+ .shmem_size = pgaio_uring_shmem_size,
+ .shmem_init = pgaio_uring_shmem_init,
+ .init_backend = pgaio_uring_init_backend,
+
+ .submit = pgaio_uring_submit,
+ .wait_one = pgaio_uring_wait_one,
+};
+
+/*
+ * Per-backend state when using io_method=io_uring
+ *
+ * Align the whole struct to a cacheline boundary, to prevent false sharing
+ * between completion_lock and prior backend's io_uring_ring.
+ */
+typedef struct pg_attribute_aligned (PG_CACHE_LINE_SIZE)
+PgAioUringContext
+{
+ /*
+ * Multiple backends can process completions for this backend's io_uring
+ * instance (e.g. when the backend issuing IO is busy doing something
+ * else). To make that safe we have to ensure that only a single backend
+ * gets io completions from the io_uring instance at a time.
+ */
+ LWLock completion_lock;
+
+ struct io_uring io_uring_ring;
+} PgAioUringContext;
+
+/* PgAioUringContexts for all backends */
+static PgAioUringContext *pgaio_uring_contexts;
+
+/* the current backend's context */
+static PgAioUringContext *pgaio_my_uring_context;
+
+
+static uint32
+pgaio_uring_procs(void)
+{
+ /*
+ * We can subtract MAX_IO_WORKERS here as io workers are never used at the
+ * same time as io_method=io_uring.
+ */
+ return MaxBackends + NUM_AUXILIARY_PROCS - MAX_IO_WORKERS;
+}
+
+static Size
+pgaio_uring_context_shmem_size(void)
+{
+ return mul_size(pgaio_uring_procs(), sizeof(PgAioUringContext));
+}
+
+static size_t
+pgaio_uring_shmem_size(void)
+{
+ return pgaio_uring_context_shmem_size();
+}
+
+static void
+pgaio_uring_shmem_init(bool first_time)
+{
+ uint32 TotalProcs = MaxBackends + NUM_AUXILIARY_PROCS - MAX_IO_WORKERS;
+ bool found;
+
+ pgaio_uring_contexts = (PgAioUringContext *)
+ ShmemInitStruct("AioUring", pgaio_uring_shmem_size(), &found);
+
+ if (found)
+ return;
+
+ for (int contextno = 0; contextno < TotalProcs; contextno++)
+ {
+ PgAioUringContext *context = &pgaio_uring_contexts[contextno];
+ int ret;
+
+ /*
+ * FIXME: Right now a high TotalProcs will cause problems in two ways:
+ *
+ * - RLIMIT_NOFILE needs to be big enough to allow MaxBackends fds
+ * created in postmaster, with spare space for max_files_per_process
+ * additional FDs
+ *
+ * - set_max_safe_fds() subtracts the number of already used FDs from
+ * max_files_per_process, ending up with a low limit or even erroring
+ * out.
+ *
+ * XXX: Newer versions of io_uring support sharing the workers that
+ * execute some asynchronous IOs between io_uring instances. It might
+ * be worth using that - also need to evaluate if that causes
+ * noticeable additional contention?
+ */
+ ret = io_uring_queue_init(io_max_concurrency, &context->io_uring_ring, 0);
+ if (ret < 0)
+ elog(ERROR, "io_uring_queue_init failed: %s", strerror(-ret));
+
+ LWLockInitialize(&context->completion_lock, LWTRANCHE_AIO_URING_COMPLETION);
+ }
+}
+
+static void
+pgaio_uring_init_backend(void)
+{
+ Assert(MyProcNumber < pgaio_uring_procs());
+
+ pgaio_my_uring_context = &pgaio_uring_contexts[MyProcNumber];
+}
+
+static int
+pgaio_uring_submit(uint16 num_staged_ios, PgAioHandle **staged_ios)
+{
+ struct io_uring *uring_instance = &pgaio_my_uring_context->io_uring_ring;
+ int in_flight_before = dclist_count(&pgaio_my_backend->in_flight_ios);
+
+ Assert(num_staged_ios <= PGAIO_SUBMIT_BATCH_SIZE);
+
+ for (int i = 0; i < num_staged_ios; i++)
+ {
+ PgAioHandle *ioh = staged_ios[i];
+ struct io_uring_sqe *sqe;
+
+ sqe = io_uring_get_sqe(uring_instance);
+
+ if (!sqe)
+ elog(ERROR, "io_uring submission queue is unexpectedly full");
+
+ pgaio_io_prepare_submit(ioh);
+ pgaio_uring_sq_from_io(ioh, sqe);
+
+ /*
+ * io_uring executes IO in process context if possible. That's
+ * generally good, as it reduces context switching. When performing a
+ * lot of buffered IO that means that copying between page cache and
+ * userspace memory happens in the foreground, as it can't be
+ * offloaded to DMA hardware as is possible when using direct IO. When
+ * executing a lot of buffered IO this causes io_uring to be slower
+ * than worker mode, as worker mode parallelizes the copying. io_uring
+ * can be told to offload work to worker threads instead.
+ *
+ * If an IO is buffered IO and we already have IOs in flight or
+ * multiple IOs are being submitted, we thus tell io_uring to execute
+ * the IO in the background. We don't do so for the first few IOs
+ * being submitted as executing in this process' context has lower
+ * latency.
+ */
+ if (in_flight_before > 4 && (ioh->flags & PGAIO_HF_BUFFERED))
+ io_uring_sqe_set_flags(sqe, IOSQE_ASYNC);
+
+ in_flight_before++;
+ }
+
+ while (true)
+ {
+ int ret;
+
+ pgstat_report_wait_start(WAIT_EVENT_AIO_IO_URING_SUBMIT);
+ ret = io_uring_submit(uring_instance);
+ pgstat_report_wait_end();
+
+ if (ret == -EINTR)
+ {
+ pgaio_debug(DEBUG3,
+ "aio method uring: submit EINTR, nios: %d",
+ num_staged_ios);
+ }
+ else if (ret < 0)
+ {
+ /*
+ * The io_uring_enter() manpage suggests that the appropriate
+ * reaction to EAGAIN is:
+ *
+ * "The application should wait for some completions and try
+ * again"
+ *
+ * However, it seems unlikely that that would help in our case, as
+ * we apply a low limit to the number of outstanding IOs and thus
+ * also outstanding completions, making it unlikely that we'd get
+ * EAGAIN while the OS is in good working order.
+ *
+ * Additionally, it would be problematic to just wait here, our
+ * caller might hold critical locks. It'd possibly lead to
+ * delaying the crash-restart that seems likely to occur when the
+ * kernel is under such heavy memory pressure.
+ *
+ * Update errno to allow %m to work.
+ */
+ errno = -ret;
+ elog(PANIC, "io_uring submit failed: %m");
+ }
+ else if (ret != num_staged_ios)
+ {
+ /* likely unreachable, but if it is, we would need to re-submit */
+ elog(PANIC, "io_uring submit submitted only %d of %d",
+ ret, num_staged_ios);
+ }
+ else
+ {
+ pgaio_debug(DEBUG4,
+ "aio method uring: submitted %d IOs",
+ num_staged_ios);
+ break;
+ }
+ }
+
+ return num_staged_ios;
+}
+
+static void
+pgaio_uring_drain_locked(PgAioUringContext *context)
+{
+ int ready;
+ int orig_ready;
+
+ Assert(LWLockHeldByMeInMode(&context->completion_lock, LW_EXCLUSIVE));
+
+ /*
+ * Don't drain more events than available right now. Otherwise it's
+ * plausible that one backend could get stuck, for a while, receiving CQEs
+ * without actually processing them.
+ */
+ orig_ready = ready = io_uring_cq_ready(&context->io_uring_ring);
+
+ while (ready > 0)
+ {
+ struct io_uring_cqe *cqes[PGAIO_MAX_LOCAL_COMPLETED_IO];
+ uint32 ncqes;
+
+ START_CRIT_SECTION();
+ ncqes =
+ io_uring_peek_batch_cqe(&context->io_uring_ring,
+ cqes,
+ Min(PGAIO_MAX_LOCAL_COMPLETED_IO, ready));
+ Assert(ncqes <= ready);
+
+ ready -= ncqes;
+
+ for (int i = 0; i < ncqes; i++)
+ {
+ struct io_uring_cqe *cqe = cqes[i];
+ PgAioHandle *ioh;
+
+ ioh = io_uring_cqe_get_data(cqe);
+ io_uring_cqe_seen(&context->io_uring_ring, cqe);
+
+ pgaio_io_process_completion(ioh, cqe->res);
+ }
+
+ END_CRIT_SECTION();
+
+ pgaio_debug(DEBUG3,
+ "drained %d/%d, now expecting %d",
+ ncqes, orig_ready, io_uring_cq_ready(&context->io_uring_ring));
+ }
+}
+
+static void
+pgaio_uring_wait_one(PgAioHandle *ioh, uint64 ref_generation)
+{
+ PgAioHandleState state;
+ ProcNumber owner_procno = ioh->owner_procno;
+ PgAioUringContext *owner_context = &pgaio_uring_contexts[owner_procno];
+ bool expect_cqe;
+ int waited = 0;
+
+ /*
+ * XXX: It would be nice to have a smarter locking scheme, nearly all the
+ * time the backend owning the ring will consume the completions, making
+ * the locking unnecessarily expensive.
+ */
+ LWLockAcquire(&owner_context->completion_lock, LW_EXCLUSIVE);
+
+ while (true)
+ {
+ pgaio_debug_io(DEBUG3, ioh,
+ "wait_one io_gen: %llu, ref_gen: %llu, cycle %d",
+ (long long unsigned) ref_generation,
+ (long long unsigned) ioh->generation,
+ waited);
+
+ if (pgaio_io_was_recycled(ioh, ref_generation, &state) ||
+ state != PGAIO_HS_SUBMITTED)
+ {
+ /* the IO was completed by another backend */
+ break;
+ }
+ else if (io_uring_cq_ready(&owner_context->io_uring_ring))
+ {
+ /* no need to wait in the kernel, io_uring has a completion */
+ expect_cqe = true;
+ }
+ else
+ {
+ int ret;
+ struct io_uring_cqe *cqes;
+
+ /* need to wait in the kernel */
+ pgstat_report_wait_start(WAIT_EVENT_AIO_IO_URING_COMPLETION);
+ ret = io_uring_wait_cqes(&owner_context->io_uring_ring, &cqes, 1, NULL, NULL);
+ pgstat_report_wait_end();
+
+ if (ret == -EINTR)
+ {
+ continue;
+ }
+ else if (ret != 0)
+ {
+ /* see comment after io_uring_submit() */
+ errno = -ret;
+ elog(PANIC, "io_uring wait failed: %m");
+ }
+ else
+ {
+ Assert(cqes != NULL);
+ expect_cqe = true;
+ waited++;
+ }
+ }
+
+ if (expect_cqe)
+ {
+ pgaio_uring_drain_locked(owner_context);
+ }
+ }
+
+ LWLockRelease(&owner_context->completion_lock);
+
+ pgaio_debug(DEBUG3,
+ "wait_one with %d sleeps",
+ waited);
+}
+
+static void
+pgaio_uring_sq_from_io(PgAioHandle *ioh, struct io_uring_sqe *sqe)
+{
+ struct iovec *iov;
+
+ switch (ioh->op)
+ {
+ case PGAIO_OP_READV:
+ iov = &pgaio_ctl->iovecs[ioh->iovec_off];
+ if (ioh->op_data.read.iov_length == 1)
+ {
+ io_uring_prep_read(sqe,
+ ioh->op_data.read.fd,
+ iov->iov_base,
+ iov->iov_len,
+ ioh->op_data.read.offset);
+ }
+ else
+ {
+ io_uring_prep_readv(sqe,
+ ioh->op_data.read.fd,
+ iov,
+ ioh->op_data.read.iov_length,
+ ioh->op_data.read.offset);
+
+ }
+ break;
+
+ case PGAIO_OP_WRITEV:
+ iov = &pgaio_ctl->iovecs[ioh->iovec_off];
+ if (ioh->op_data.write.iov_length == 1)
+ {
+ io_uring_prep_write(sqe,
+ ioh->op_data.write.fd,
+ iov->iov_base,
+ iov->iov_len,
+ ioh->op_data.write.offset);
+ }
+ else
+ {
+ io_uring_prep_writev(sqe,
+ ioh->op_data.write.fd,
+ iov,
+ ioh->op_data.write.iov_length,
+ ioh->op_data.write.offset);
+ }
+ break;
+
+ case PGAIO_OP_INVALID:
+ elog(ERROR, "trying to prepare invalid IO operation for execution");
+ }
+
+ io_uring_sqe_set_data(sqe, ioh);
+}
+
+#endif /* USE_LIBURING */
diff --git a/src/backend/storage/lmgr/lwlock.c b/src/backend/storage/lmgr/lwlock.c
index 5702c35bb91..3df29658f18 100644
--- a/src/backend/storage/lmgr/lwlock.c
+++ b/src/backend/storage/lmgr/lwlock.c
@@ -177,6 +177,7 @@ static const char *const BuiltinTrancheNames[] = {
[LWTRANCHE_SUBTRANS_SLRU] = "SubtransSLRU",
[LWTRANCHE_XACT_SLRU] = "XactSLRU",
[LWTRANCHE_PARALLEL_VACUUM_DSA] = "ParallelVacuumDSA",
+ [LWTRANCHE_AIO_URING_COMPLETION] = "AioUringCompletion",
};
StaticAssertDecl(lengthof(BuiltinTrancheNames) ==
diff --git a/src/backend/utils/activity/wait_event_names.txt b/src/backend/utils/activity/wait_event_names.txt
index 9fa12a555e8..44eef41f841 100644
--- a/src/backend/utils/activity/wait_event_names.txt
+++ b/src/backend/utils/activity/wait_event_names.txt
@@ -192,6 +192,8 @@ ABI_compatibility:
Section: ClassName - WaitEventIO
+AIO_IO_URING_SUBMIT "Waiting for IO submission via io_uring."
+AIO_IO_URING_COMPLETION "Waiting for IO completion via io_uring."
AIO_IO_COMPLETION "Waiting for IO completion."
BASEBACKUP_READ "Waiting for base backup to read from a file."
BASEBACKUP_SYNC "Waiting for data written by a base backup to reach durable storage."
diff --git a/src/backend/utils/misc/postgresql.conf.sample b/src/backend/utils/misc/postgresql.conf.sample
index beb05a89501..458f0815a12 100644
--- a/src/backend/utils/misc/postgresql.conf.sample
+++ b/src/backend/utils/misc/postgresql.conf.sample
@@ -202,7 +202,8 @@
#maintenance_io_concurrency = 16 # 1-1000; 0 disables prefetching
#io_combine_limit = 128kB # usually 1-32 blocks (depends on OS)
-#io_method = worker # worker, sync (change requires restart)
+#io_method = worker # worker, io_uring, sync
+ # (change requires restart)
#io_max_concurrency = -1 # Max number of IOs that one process
# can execute simultaneously
# -1 sets based on shared_buffers
diff --git a/doc/src/sgml/config.sgml b/doc/src/sgml/config.sgml
index 9e9c02cde83..1156c389c2c 100644
--- a/doc/src/sgml/config.sgml
+++ b/doc/src/sgml/config.sgml
@@ -2681,6 +2681,12 @@ include_dir 'conf.d'
<literal>worker</literal> (execute asynchronous I/O using worker processes)
</para>
</listitem>
+ <listitem>
+ <para>
+ <literal>io_uring</literal> (execute asynchronous I/O using
+ io_uring, if available)
+ </para>
+ </listitem>
<listitem>
<para>
<literal>sync</literal> (execute asynchronous-eligible I/O synchronously)
diff --git a/.cirrus.tasks.yml b/.cirrus.tasks.yml
index 10d277f5659..86a1fa9bbdb 100644
--- a/.cirrus.tasks.yml
+++ b/.cirrus.tasks.yml
@@ -493,11 +493,14 @@ task:
# - Uses undefined behaviour and alignment sanitizers, sanitizer failures
# are typically printed in the server log
# - Test both 64bit and 32 bit builds
+ # - uses io_method=io_uring
- name: Linux - Debian Bookworm - Meson
env:
CCACHE_MAXSIZE: "400M" # tests two different builds
SANITIZER_FLAGS: -fsanitize=alignment,undefined
+ PG_TEST_INITDB_EXTRA_OPTS: >-
+ -c io_method=io_uring
configure_script: |
su postgres <<-EOF
diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list
index bfa276d2d35..53abd54edd8 100644
--- a/src/tools/pgindent/typedefs.list
+++ b/src/tools/pgindent/typedefs.list
@@ -2150,6 +2150,7 @@ PgAioReturn
PgAioTargetData
PgAioTargetID
PgAioTargetInfo
+PgAioUringContext
PgAioWaitRef
PgArchData
PgBackendGSSStatus
--
2.48.1.76.g4e746b1a31.dirty
[text/x-diff] v2.10-0005-aio-Implement-support-for-reads-in-smgr-md-fd.patch (22.0K, ../../zp7obmaqdd6b5u2vhpw27hovpjirtqvjakkbkaz4vju7tohyul@4kfobe7ipgxm/6-v2.10-0005-aio-Implement-support-for-reads-in-smgr-md-fd.patch)
download | inline diff:
From f82753ec15143863fd593153d561ae7b55dbf6bc Mon Sep 17 00:00:00 2001
From: Andres Freund <andres@anarazel.de>
Date: Tue, 18 Mar 2025 14:40:05 -0400
Subject: [PATCH v2.10 05/28] aio: Implement support for reads in smgr/md/fd
This implements the following:
1) An smgr AIO target, for AIO on smgr files. This should be usable not just
for md.c but also other SMGR implementation if we ever get them.
2) readv support in fd.c, which requires a small bit of infrastructure work in
fd.c
3) smgr.c and md.c support for readv
There still is nothing performing AIO, but as of this commit it would be
possible.
As part of this change FileGetRawDesc() actually ensures that the file is
opened - previously it was basically not usable. It's used to reopen a file in
IO workers.
Discussion: https://postgr.es/m/uvrtrknj4kdytuboidbhwclo4gxhswwcpgadptsjvjqcluzmah%40brqs62irg4dt
Discussion: https://postgr.es/m/20210223100344.llw5an2aklengrmn@alap3.anarazel.de
Discussion: https://postgr.es/m/stj36ea6yyhoxtqkhpieia2z4krnam7qyetc57rfezgk4zgapf@gcnactj4z56m
---
src/include/storage/aio.h | 6 +-
src/include/storage/aio_types.h | 10 +-
src/include/storage/fd.h | 3 +
src/include/storage/md.h | 7 +
src/include/storage/smgr.h | 14 ++
src/backend/storage/aio/aio_callback.c | 3 +
src/backend/storage/aio/aio_target.c | 2 +
src/backend/storage/file/fd.c | 40 ++++++
src/backend/storage/smgr/md.c | 176 +++++++++++++++++++++++++
src/backend/storage/smgr/smgr.c | 144 ++++++++++++++++++++
10 files changed, 401 insertions(+), 4 deletions(-)
diff --git a/src/include/storage/aio.h b/src/include/storage/aio.h
index 53b1080b17f..4fe7a6a8b00 100644
--- a/src/include/storage/aio.h
+++ b/src/include/storage/aio.h
@@ -112,9 +112,10 @@ typedef enum PgAioTargetID
{
/* intentionally the zero value, to help catch zeroed memory etc */
PGAIO_TID_INVALID = 0,
+ PGAIO_TID_SMGR,
} PgAioTargetID;
-#define PGAIO_TID_COUNT (PGAIO_TID_INVALID + 1)
+#define PGAIO_TID_COUNT (PGAIO_TID_SMGR + 1)
/*
@@ -186,6 +187,9 @@ struct PgAioTargetInfo
typedef enum PgAioHandleCallbackID
{
PGAIO_HCB_INVALID,
+
+ PGAIO_HCB_MD_READV,
+ PGAIO_HCB_MD_WRITEV,
} PgAioHandleCallbackID;
diff --git a/src/include/storage/aio_types.h b/src/include/storage/aio_types.h
index a5cc658efbd..b0b9a2a5c97 100644
--- a/src/include/storage/aio_types.h
+++ b/src/include/storage/aio_types.h
@@ -60,11 +60,15 @@ typedef struct PgAioWaitRef
*/
typedef union PgAioTargetData
{
- /* just as an example placeholder for later */
struct
{
- uint32 queue_id;
- } wal;
+ RelFileLocator rlocator; /* physical relation identifier */
+ BlockNumber blockNum; /* blknum relative to begin of reln */
+ BlockNumber nblocks;
+ ForkNumber forkNum:8; /* don't waste 4 byte for four values */
+ bool is_temp:1; /* proc can be inferred by owning AIO */
+ bool skip_fsync:1;
+ } smgr;
} PgAioTargetData;
diff --git a/src/include/storage/fd.h b/src/include/storage/fd.h
index e3067ab6597..b77d8e5e30e 100644
--- a/src/include/storage/fd.h
+++ b/src/include/storage/fd.h
@@ -101,6 +101,8 @@ extern PGDLLIMPORT int max_safe_fds;
* prototypes for functions in fd.c
*/
+struct PgAioHandle;
+
/* Operations on virtual Files --- equivalent to Unix kernel file ops */
extern File PathNameOpenFile(const char *fileName, int fileFlags);
extern File PathNameOpenFilePerm(const char *fileName, int fileFlags, mode_t fileMode);
@@ -109,6 +111,7 @@ extern void FileClose(File file);
extern int FilePrefetch(File file, off_t offset, off_t amount, uint32 wait_event_info);
extern ssize_t FileReadV(File file, const struct iovec *iov, int iovcnt, off_t offset, uint32 wait_event_info);
extern ssize_t FileWriteV(File file, const struct iovec *iov, int iovcnt, off_t offset, uint32 wait_event_info);
+extern int FileStartReadV(struct PgAioHandle *ioh, File file, int iovcnt, off_t offset, uint32 wait_event_info);
extern int FileSync(File file, uint32 wait_event_info);
extern int FileZero(File file, off_t offset, off_t amount, uint32 wait_event_info);
extern int FileFallocate(File file, off_t offset, off_t amount, uint32 wait_event_info);
diff --git a/src/include/storage/md.h b/src/include/storage/md.h
index 05bf537066e..9d7131eff43 100644
--- a/src/include/storage/md.h
+++ b/src/include/storage/md.h
@@ -14,11 +14,14 @@
#ifndef MD_H
#define MD_H
+#include "storage/aio_types.h"
#include "storage/block.h"
#include "storage/relfilelocator.h"
#include "storage/smgr.h"
#include "storage/sync.h"
+extern const PgAioHandleCallbacks aio_md_readv_cb;
+
/* md storage manager functionality */
extern void mdinit(void);
extern void mdopen(SMgrRelation reln);
@@ -36,6 +39,9 @@ extern uint32 mdmaxcombine(SMgrRelation reln, ForkNumber forknum,
BlockNumber blocknum);
extern void mdreadv(SMgrRelation reln, ForkNumber forknum, BlockNumber blocknum,
void **buffers, BlockNumber nblocks);
+extern void mdstartreadv(PgAioHandle *ioh,
+ SMgrRelation reln, ForkNumber forknum, BlockNumber blocknum,
+ void **buffers, BlockNumber nblocks);
extern void mdwritev(SMgrRelation reln, ForkNumber forknum,
BlockNumber blocknum,
const void **buffers, BlockNumber nblocks, bool skipFsync);
@@ -46,6 +52,7 @@ extern void mdtruncate(SMgrRelation reln, ForkNumber forknum,
BlockNumber old_blocks, BlockNumber nblocks);
extern void mdimmedsync(SMgrRelation reln, ForkNumber forknum);
extern void mdregistersync(SMgrRelation reln, ForkNumber forknum);
+extern int mdfd(SMgrRelation reln, ForkNumber forknum, BlockNumber blocknum, uint32 *off);
extern void ForgetDatabaseSyncRequests(Oid dbid);
extern void DropRelationFiles(RelFileLocator *delrels, int ndelrels, bool isRedo);
diff --git a/src/include/storage/smgr.h b/src/include/storage/smgr.h
index 4016b206ad6..856ebcda350 100644
--- a/src/include/storage/smgr.h
+++ b/src/include/storage/smgr.h
@@ -15,6 +15,7 @@
#define SMGR_H
#include "lib/ilist.h"
+#include "storage/aio_types.h"
#include "storage/block.h"
#include "storage/relfilelocator.h"
@@ -73,6 +74,8 @@ typedef SMgrRelationData *SMgrRelation;
#define SmgrIsTemp(smgr) \
RelFileLocatorBackendIsTemp((smgr)->smgr_rlocator)
+extern const PgAioTargetInfo aio_smgr_target_info;
+
extern void smgrinit(void);
extern SMgrRelation smgropen(RelFileLocator rlocator, ProcNumber backend);
extern bool smgrexists(SMgrRelation reln, ForkNumber forknum);
@@ -97,6 +100,10 @@ extern uint32 smgrmaxcombine(SMgrRelation reln, ForkNumber forknum,
extern void smgrreadv(SMgrRelation reln, ForkNumber forknum,
BlockNumber blocknum,
void **buffers, BlockNumber nblocks);
+extern void smgrstartreadv(PgAioHandle *ioh,
+ SMgrRelation reln, ForkNumber forknum,
+ BlockNumber blocknum,
+ void **buffers, BlockNumber nblocks);
extern void smgrwritev(SMgrRelation reln, ForkNumber forknum,
BlockNumber blocknum,
const void **buffers, BlockNumber nblocks,
@@ -127,4 +134,11 @@ smgrwrite(SMgrRelation reln, ForkNumber forknum, BlockNumber blocknum,
smgrwritev(reln, forknum, blocknum, &buffer, 1, skipFsync);
}
+extern void pgaio_io_set_target_smgr(PgAioHandle *ioh,
+ SMgrRelationData *smgr,
+ ForkNumber forknum,
+ BlockNumber blocknum,
+ int nblocks,
+ bool skip_fsync);
+
#endif /* SMGR_H */
diff --git a/src/backend/storage/aio/aio_callback.c b/src/backend/storage/aio/aio_callback.c
index d5a2cca28f1..2d48aa7c1b3 100644
--- a/src/backend/storage/aio/aio_callback.c
+++ b/src/backend/storage/aio/aio_callback.c
@@ -18,6 +18,7 @@
#include "miscadmin.h"
#include "storage/aio.h"
#include "storage/aio_internal.h"
+#include "storage/md.h"
/* just to have something to put into aio_handle_cbs */
@@ -37,6 +38,8 @@ typedef struct PgAioHandleCallbacksEntry
static const PgAioHandleCallbacksEntry aio_handle_cbs[] = {
#define CALLBACK_ENTRY(id, callback) [id] = {.cb = &callback, .name = #callback}
CALLBACK_ENTRY(PGAIO_HCB_INVALID, aio_invalid_cb),
+
+ CALLBACK_ENTRY(PGAIO_HCB_MD_READV, aio_md_readv_cb),
#undef CALLBACK_ENTRY
};
diff --git a/src/backend/storage/aio/aio_target.c b/src/backend/storage/aio/aio_target.c
index b01406a6a52..6133b04f350 100644
--- a/src/backend/storage/aio/aio_target.c
+++ b/src/backend/storage/aio/aio_target.c
@@ -16,6 +16,7 @@
#include "storage/aio.h"
#include "storage/aio_internal.h"
+#include "storage/smgr.h"
/*
@@ -25,6 +26,7 @@ static const PgAioTargetInfo *pgaio_target_info[] = {
[PGAIO_TID_INVALID] = &(PgAioTargetInfo) {
.name = "invalid",
},
+ [PGAIO_TID_SMGR] = &aio_smgr_target_info,
};
diff --git a/src/backend/storage/file/fd.c b/src/backend/storage/file/fd.c
index 62f1185859f..34588ed5167 100644
--- a/src/backend/storage/file/fd.c
+++ b/src/backend/storage/file/fd.c
@@ -94,6 +94,7 @@
#include "miscadmin.h"
#include "pgstat.h"
#include "postmaster/startup.h"
+#include "storage/aio.h"
#include "storage/fd.h"
#include "storage/ipc.h"
#include "utils/guc.h"
@@ -1294,6 +1295,8 @@ LruDelete(File file)
vfdP = &VfdCache[file];
+ pgaio_closing_fd(vfdP->fd);
+
/*
* Close the file. We aren't expecting this to fail; if it does, better
* to leak the FD than to mess up our internal state.
@@ -1987,6 +1990,8 @@ FileClose(File file)
if (!FileIsNotOpen(file))
{
+ pgaio_closing_fd(vfdP->fd);
+
/* close the file */
if (close(vfdP->fd) != 0)
{
@@ -2210,6 +2215,32 @@ retry:
return returnCode;
}
+int
+FileStartReadV(PgAioHandle *ioh, File file,
+ int iovcnt, off_t offset,
+ uint32 wait_event_info)
+{
+ int returnCode;
+ Vfd *vfdP;
+
+ Assert(FileIsValid(file));
+
+ DO_DB(elog(LOG, "FileStartReadV: %d (%s) " INT64_FORMAT " %d",
+ file, VfdCache[file].fileName,
+ (int64) offset,
+ iovcnt));
+
+ returnCode = FileAccess(file);
+ if (returnCode < 0)
+ return returnCode;
+
+ vfdP = &VfdCache[file];
+
+ pgaio_io_prep_readv(ioh, vfdP->fd, iovcnt, offset);
+
+ return 0;
+}
+
ssize_t
FileWriteV(File file, const struct iovec *iov, int iovcnt, off_t offset,
uint32 wait_event_info)
@@ -2498,6 +2529,12 @@ FilePathName(File file)
int
FileGetRawDesc(File file)
{
+ int returnCode;
+
+ returnCode = FileAccess(file);
+ if (returnCode < 0)
+ return returnCode;
+
Assert(FileIsValid(file));
return VfdCache[file].fd;
}
@@ -2778,6 +2815,7 @@ FreeDesc(AllocateDesc *desc)
result = closedir(desc->desc.dir);
break;
case AllocateDescRawFD:
+ pgaio_closing_fd(desc->desc.fd);
result = close(desc->desc.fd);
break;
default:
@@ -2846,6 +2884,8 @@ CloseTransientFile(int fd)
/* Only get here if someone passes us a file not in allocatedDescs */
elog(WARNING, "fd passed to CloseTransientFile was not obtained from OpenTransientFile");
+ pgaio_closing_fd(fd);
+
return close(fd);
}
diff --git a/src/backend/storage/smgr/md.c b/src/backend/storage/smgr/md.c
index f3220f98dc4..11d4d5a7aea 100644
--- a/src/backend/storage/smgr/md.c
+++ b/src/backend/storage/smgr/md.c
@@ -31,6 +31,7 @@
#include "miscadmin.h"
#include "pg_trace.h"
#include "pgstat.h"
+#include "storage/aio.h"
#include "storage/bufmgr.h"
#include "storage/fd.h"
#include "storage/md.h"
@@ -152,6 +153,15 @@ static MdfdVec *_mdfd_getseg(SMgrRelation reln, ForkNumber forknum,
static BlockNumber _mdnblocks(SMgrRelation reln, ForkNumber forknum,
MdfdVec *seg);
+static PgAioResult md_readv_complete(PgAioHandle *ioh, PgAioResult prior_result, uint8 cb_data);
+static void md_readv_report(PgAioResult result, const PgAioTargetData *target_data, int elevel);
+
+const PgAioHandleCallbacks aio_md_readv_cb = {
+ .complete_shared = md_readv_complete,
+ .report = md_readv_report,
+};
+
+
static inline int
_mdfd_open_flags(void)
{
@@ -937,6 +947,61 @@ mdreadv(SMgrRelation reln, ForkNumber forknum, BlockNumber blocknum,
}
}
+/*
+ * mdstartreadv() -- Asynchronous version of mdreadv().
+ */
+void
+mdstartreadv(PgAioHandle *ioh,
+ SMgrRelation reln, ForkNumber forknum, BlockNumber blocknum,
+ void **buffers, BlockNumber nblocks)
+{
+ off_t seekpos;
+ MdfdVec *v;
+ BlockNumber nblocks_this_segment;
+ struct iovec *iov;
+ int iovcnt;
+
+ v = _mdfd_getseg(reln, forknum, blocknum, false,
+ EXTENSION_FAIL | EXTENSION_CREATE_RECOVERY);
+
+ seekpos = (off_t) BLCKSZ * (blocknum % ((BlockNumber) RELSEG_SIZE));
+
+ Assert(seekpos < (off_t) BLCKSZ * RELSEG_SIZE);
+
+ nblocks_this_segment =
+ Min(nblocks,
+ RELSEG_SIZE - (blocknum % ((BlockNumber) RELSEG_SIZE)));
+
+ if (nblocks_this_segment != nblocks)
+ elog(ERROR, "read crossing segment boundary");
+
+ iovcnt = pgaio_io_get_iovec(ioh, &iov);
+
+ Assert(nblocks <= iovcnt);
+
+ iovcnt = buffers_to_iovec(iov, buffers, nblocks_this_segment);
+
+ Assert(iovcnt <= nblocks_this_segment);
+
+ if (!(io_direct_flags & IO_DIRECT_DATA))
+ pgaio_io_set_flag(ioh, PGAIO_HF_BUFFERED);
+
+ pgaio_io_set_target_smgr(ioh,
+ reln,
+ forknum,
+ blocknum,
+ nblocks,
+ false);
+ pgaio_io_register_callbacks(ioh, PGAIO_HCB_MD_READV, 0);
+
+ FileStartReadV(ioh, v->mdfd_vfd, iovcnt, seekpos, WAIT_EVENT_DATA_FILE_READ);
+
+ /*
+ * The error checks corresponding to the post-read checks in mdread() are
+ * in md_readv_complete().
+ */
+}
+
/*
* mdwritev() -- Write the supplied blocks at the appropriate location.
*
@@ -1365,6 +1430,21 @@ mdimmedsync(SMgrRelation reln, ForkNumber forknum)
}
}
+int
+mdfd(SMgrRelation reln, ForkNumber forknum, BlockNumber blocknum, uint32 *off)
+{
+ MdfdVec *v = mdopenfork(reln, forknum, EXTENSION_FAIL);
+
+ v = _mdfd_getseg(reln, forknum, blocknum, false,
+ EXTENSION_FAIL);
+
+ *off = (off_t) BLCKSZ * (blocknum % ((BlockNumber) RELSEG_SIZE));
+
+ Assert(*off < (off_t) BLCKSZ * RELSEG_SIZE);
+
+ return FileGetRawDesc(v->mdfd_vfd);
+}
+
/*
* register_dirty_segment() -- Mark a relation segment as needing fsync
*
@@ -1841,3 +1921,99 @@ mdfiletagmatches(const FileTag *ftag, const FileTag *candidate)
*/
return ftag->rlocator.dbOid == candidate->rlocator.dbOid;
}
+
+/*
+ * AIO completion callback for mdstartreadv().
+ */
+static PgAioResult
+md_readv_complete(PgAioHandle *ioh, PgAioResult prior_result, uint8 cb_data)
+{
+ PgAioTargetData *td = pgaio_io_get_target_data(ioh);
+ PgAioResult result = prior_result;
+
+ if (prior_result.result < 0)
+ {
+ result.status = ARS_ERROR;
+ result.id = PGAIO_HCB_MD_READV;
+ /* For "hard" errors, track the error number in error_data */
+ result.error_data = -prior_result.result;
+ result.result = 0;
+
+ pgaio_result_report(result, td, LOG);
+
+ return result;
+ }
+
+ /*
+ * The smgr API operates in blocks, therefore convert the result from
+ * bytes to blocks.
+ */
+ result.result /= BLCKSZ;
+
+ if (result.result == 0)
+ {
+ /* consider 0 blocks read a failure */
+ result.status = ARS_ERROR;
+ result.id = PGAIO_HCB_MD_READV;
+ result.error_data = 0;
+
+ pgaio_result_report(result, td, LOG);
+
+ return result;
+ }
+
+ if (result.status != ARS_ERROR &&
+ result.result < td->smgr.nblocks)
+ {
+ /* partial reads should be retried at upper level */
+ result.status = ARS_PARTIAL;
+ result.id = PGAIO_HCB_MD_READV;
+ }
+
+ return result;
+}
+
+/*
+ * AIO error reporting callback for mdstartreadv().
+ *
+ * Errors are encoded as follows:
+ * - PgAioResult.error_data != 0 encodes IO that failed with errno != 0
+ * - PgAioResult.error_data == 0 encodes IO that didn't read all data
+ */
+static void
+md_readv_report(PgAioResult result, const PgAioTargetData *td, int elevel)
+{
+ RelPathStr path;
+
+ path = relpathbackend(td->smgr.rlocator,
+ td->smgr.is_temp ? MyProcNumber : INVALID_PROC_NUMBER,
+ td->smgr.forkNum);
+
+ if (result.error_data != 0)
+ {
+ /* for errcode_for_file_access() and %m */
+ errno = result.error_data;
+
+ ereport(elevel,
+ errcode_for_file_access(),
+ errmsg("could not read blocks %u..%u in file \"%s\": %m",
+ td->smgr.blockNum,
+ td->smgr.blockNum + td->smgr.nblocks - 1,
+ path.str));
+ }
+ else
+ {
+ /*
+ * NB: This will typically only be output in debug messages, while
+ * retrying a partial IO.
+ */
+ ereport(elevel,
+ errcode(ERRCODE_DATA_CORRUPTED),
+ errmsg("could not read blocks %u..%u in file \"%s\": read only %zu of %zu bytes",
+ td->smgr.blockNum,
+ td->smgr.blockNum + td->smgr.nblocks - 1,
+ path.str,
+ result.result * (size_t) BLCKSZ,
+ td->smgr.nblocks * (size_t) BLCKSZ));
+ }
+}
diff --git a/src/backend/storage/smgr/smgr.c b/src/backend/storage/smgr/smgr.c
index 24971304b85..70bf6fd824a 100644
--- a/src/backend/storage/smgr/smgr.c
+++ b/src/backend/storage/smgr/smgr.c
@@ -66,6 +66,7 @@
#include "access/xlogutils.h"
#include "lib/ilist.h"
#include "miscadmin.h"
+#include "storage/aio.h"
#include "storage/bufmgr.h"
#include "storage/ipc.h"
#include "storage/md.h"
@@ -106,6 +107,10 @@ typedef struct f_smgr
void (*smgr_readv) (SMgrRelation reln, ForkNumber forknum,
BlockNumber blocknum,
void **buffers, BlockNumber nblocks);
+ void (*smgr_startreadv) (PgAioHandle *ioh,
+ SMgrRelation reln, ForkNumber forknum,
+ BlockNumber blocknum,
+ void **buffers, BlockNumber nblocks);
void (*smgr_writev) (SMgrRelation reln, ForkNumber forknum,
BlockNumber blocknum,
const void **buffers, BlockNumber nblocks,
@@ -117,6 +122,7 @@ typedef struct f_smgr
BlockNumber old_blocks, BlockNumber nblocks);
void (*smgr_immedsync) (SMgrRelation reln, ForkNumber forknum);
void (*smgr_registersync) (SMgrRelation reln, ForkNumber forknum);
+ int (*smgr_fd) (SMgrRelation reln, ForkNumber forknum, BlockNumber blocknum, uint32 *off);
} f_smgr;
static const f_smgr smgrsw[] = {
@@ -134,12 +140,14 @@ static const f_smgr smgrsw[] = {
.smgr_prefetch = mdprefetch,
.smgr_maxcombine = mdmaxcombine,
.smgr_readv = mdreadv,
+ .smgr_startreadv = mdstartreadv,
.smgr_writev = mdwritev,
.smgr_writeback = mdwriteback,
.smgr_nblocks = mdnblocks,
.smgr_truncate = mdtruncate,
.smgr_immedsync = mdimmedsync,
.smgr_registersync = mdregistersync,
+ .smgr_fd = mdfd,
}
};
@@ -157,6 +165,16 @@ static dlist_head unpinned_relns;
static void smgrshutdown(int code, Datum arg);
static void smgrdestroy(SMgrRelation reln);
+static void smgr_aio_reopen(PgAioHandle *ioh);
+static char *smgr_aio_describe_identity(const PgAioTargetData *sd);
+
+
+const PgAioTargetInfo aio_smgr_target_info = {
+ .name = "smgr",
+ .reopen = smgr_aio_reopen,
+ .describe_identity = smgr_aio_describe_identity,
+};
+
/*
* smgrinit(), smgrshutdown() -- Initialize or shut down storage
@@ -700,6 +718,24 @@ smgrreadv(SMgrRelation reln, ForkNumber forknum, BlockNumber blocknum,
RESUME_INTERRUPTS();
}
+/*
+ * smgrstartreadv() -- asynchronous version of smgrreadv()
+ *
+ * This starts an asynchronous readv IO with the IO handle `ioh`. Other than
+ * `ioh` all parameters are the same as smgrreadv().
+ */
+void
+smgrstartreadv(PgAioHandle *ioh,
+ SMgrRelation reln, ForkNumber forknum, BlockNumber blocknum,
+ void **buffers, BlockNumber nblocks)
+{
+ HOLD_INTERRUPTS();
+ smgrsw[reln->smgr_which].smgr_startreadv(ioh,
+ reln, forknum, blocknum, buffers,
+ nblocks);
+ RESUME_INTERRUPTS();
+}
+
/*
* smgrwritev() -- Write the supplied buffers out.
*
@@ -910,6 +946,25 @@ smgrimmedsync(SMgrRelation reln, ForkNumber forknum)
RESUME_INTERRUPTS();
}
+/*
+ * Return fd for the specified block number and update *off to the appropriate
+ * position.
+ *
+ * This is only to be used for when AIO needs to perform the IO in a different
+ * process than where it was issued (e.g. in an IO worker).
+ */
+static int
+smgrfd(SMgrRelation reln, ForkNumber forknum, BlockNumber blocknum, uint32 *off)
+{
+ int fd;
+
+ HOLD_INTERRUPTS();
+ fd = smgrsw[reln->smgr_which].smgr_fd(reln, forknum, blocknum, off);
+ RESUME_INTERRUPTS();
+
+ return fd;
+}
+
/*
* AtEOXact_SMgr
*
@@ -938,3 +993,92 @@ ProcessBarrierSmgrRelease(void)
smgrreleaseall();
return true;
}
+
+/*
+ * Set target of the IO handle to be smgr and initialize all the relevant
+ * pieces of data.
+ */
+void
+pgaio_io_set_target_smgr(PgAioHandle *ioh,
+ SMgrRelationData *smgr,
+ ForkNumber forknum,
+ BlockNumber blocknum,
+ int nblocks,
+ bool skip_fsync)
+{
+ PgAioTargetData *sd = pgaio_io_get_target_data(ioh);
+
+ pgaio_io_set_target(ioh, PGAIO_TID_SMGR);
+
+ /* backend is implied via IO owner */
+ sd->smgr.rlocator = smgr->smgr_rlocator.locator;
+ sd->smgr.forkNum = forknum;
+ sd->smgr.blockNum = blocknum;
+ sd->smgr.nblocks = nblocks;
+ sd->smgr.is_temp = SmgrIsTemp(smgr);
+ /* Temp relations should never be fsync'd */
+ sd->smgr.skip_fsync = skip_fsync && !SmgrIsTemp(smgr);
+}
+
+/*
+ * Callback for smgr AIO targets, to reopen the file (e.g. because the IO is
+ * executed in a worker).
+ */
+static void
+smgr_aio_reopen(PgAioHandle *ioh)
+{
+ PgAioTargetData *sd = pgaio_io_get_target_data(ioh);
+ PgAioOpData *od = pgaio_io_get_op_data(ioh);
+ SMgrRelation reln;
+ ProcNumber procno;
+ uint32 off;
+
+ if (sd->smgr.is_temp)
+ procno = pgaio_io_get_owner(ioh);
+ else
+ procno = INVALID_PROC_NUMBER;
+
+ reln = smgropen(sd->smgr.rlocator, procno);
+ switch (pgaio_io_get_op(ioh))
+ {
+ case PGAIO_OP_INVALID:
+ pg_unreachable();
+ break;
+ case PGAIO_OP_READV:
+ od->read.fd = smgrfd(reln, sd->smgr.forkNum, sd->smgr.blockNum, &off);
+ Assert(off == od->read.offset);
+ break;
+ case PGAIO_OP_WRITEV:
+ od->write.fd = smgrfd(reln, sd->smgr.forkNum, sd->smgr.blockNum, &off);
+ Assert(off == od->write.offset);
+ break;
+ }
+}
+
+/*
+ * Callback for smgr AIO targets, to describe the target of the IO.
+ */
+static char *
+smgr_aio_describe_identity(const PgAioTargetData *sd)
+{
+ RelPathStr path;
+ char *desc;
+
+ path = relpathbackend(sd->smgr.rlocator,
+ sd->smgr.is_temp ? MyProcNumber : INVALID_PROC_NUMBER,
+ sd->smgr.forkNum);
+
+ if (sd->smgr.nblocks == 0)
+ desc = psprintf(_("file \"%s\""), path.str);
+ else if (sd->smgr.nblocks == 1)
+ desc = psprintf(_("block %u in file \"%s\""),
+ sd->smgr.blockNum,
+ path.str);
+ else
+ desc = psprintf(_("blocks %u..%u in file \"%s\""),
+ sd->smgr.blockNum,
+ sd->smgr.blockNum + sd->smgr.nblocks - 1,
+ path.str);
+
+ return desc;
+}
--
2.48.1.76.g4e746b1a31.dirty
[text/x-diff] v2.10-0006-aio-Add-README.md-explaining-higher-level-desi.patch (19.0K, ../../zp7obmaqdd6b5u2vhpw27hovpjirtqvjakkbkaz4vju7tohyul@4kfobe7ipgxm/7-v2.10-0006-aio-Add-README.md-explaining-higher-level-desi.patch)
download | inline diff:
From dd729535150b66fa9879de1ef69fd3f03d4c4cf8 Mon Sep 17 00:00:00 2001
From: Andres Freund <andres@anarazel.de>
Date: Tue, 18 Mar 2025 14:40:06 -0400
Subject: [PATCH v2.10 06/28] aio: Add README.md explaining higher level design
Reviewed-by: Noah Misch <noah@leadboat.com>
Reviewed-by: Heikki Linnakangas <hlinnaka@iki.fi>
Discussion: https://postgr.es/m/uvrtrknj4kdytuboidbhwclo4gxhswwcpgadptsjvjqcluzmah%40brqs62irg4dt
Discussion: https://postgr.es/m/20210223100344.llw5an2aklengrmn@alap3.anarazel.de
Discussion: https://postgr.es/m/stj36ea6yyhoxtqkhpieia2z4krnam7qyetc57rfezgk4zgapf@gcnactj4z56m
---
src/backend/storage/aio/README.md | 423 ++++++++++++++++++++++++++++++
src/backend/storage/aio/aio.c | 2 +
2 files changed, 425 insertions(+)
create mode 100644 src/backend/storage/aio/README.md
diff --git a/src/backend/storage/aio/README.md b/src/backend/storage/aio/README.md
new file mode 100644
index 00000000000..6df4826163e
--- /dev/null
+++ b/src/backend/storage/aio/README.md
@@ -0,0 +1,423 @@
+# Asynchronous & Direct IO
+
+## Motivation
+
+### Why Asynchronous IO
+
+Until the introduction of asynchronous IO Postgres relied on the operating
+system to hide the cost of synchronous IO from Postgres. While this worked
+surprisingly well in a lot of workloads, it does not do as good a job on
+prefetching and controlled writeback as we would like.
+
+There are important expensive operations like `fdatasync()` where the operating
+system cannot hide the storage latency. This is particularly important for WAL
+writes, where the ability to asynchronously issue `fdatasync()` or O_DSYNC
+writes can yield significantly higher throughput.
+
+
+### Why Direct / unbuffered IO
+
+The main reasons to want to use Direct IO are:
+
+- Lower CPU usage / higher throughput. Particularly on modern storage buffered
+ writes are bottlenecked by the operating system having to copy data from the
+ kernel's page cache to postgres buffer pool using the CPU. Whereas direct IO
+ can often move the data directly between the storage devices and postgres'
+ buffer cache, using DMA. While that transfer is ongoing, the CPU is free to
+ perform other work.
+- Reduced latency - Direct IO can have substantially lower latency than
+ buffered IO, which can be impactful for OLTP workloads bottlenecked by WAL
+ write latency.
+- Avoiding double buffering between operating system cache and postgres'
+ shared_buffers.
+- Better control over the timing and pace of dirty data writeback.
+
+
+The main reasons *not* to use Direct IO are:
+
+- Without AIO, Direct IO is unusably slow for most purposes.
+- Even with AIO, many parts of postgres need to be modified to perform
+ explicit prefetching.
+- In situations where shared_buffers cannot be set appropriately large,
+ e.g. because there are many different postgres instances hosted on shared
+ hardware, performance will often be worse than when using buffered IO.
+
+
+## AIO Usage Example
+
+In many cases code that can benefit from AIO does not directly have to
+interact with the AIO interface, but can use AIO via higher-level
+abstractions. See [Helpers](#helpers).
+
+In this example, a buffer will be read into shared buffers.
+
+```C
+/*
+ * Result of the operation, only to be accessed in this backend.
+ */
+PgAioReturn ioret;
+
+/*
+ * Acquire an AIO Handle, ioret will get result upon completion.
+ *
+ * Note that ioret needs to stay alive until the IO completes or
+ * CurrentResourceOwner is released (i.e. an error is thrown).
+ */
+PgAioHandle *ioh = pgaio_io_acquire(CurrentResourceOwner, &ioret);
+
+/*
+ * Reference that can be used to wait for the IO we initiate below. This
+ * reference can reside in local or shared memory and waited upon by any
+ * process. An arbitrary number of references can be made for each IO.
+ */
+PgAioWaitRef iow;
+
+pgaio_io_get_wref(ioh, &iow);
+
+/*
+ * Arrange for shared buffer completion callbacks to be called upon completion
+ * of the IO. This callback will update the buffer descriptors associated with
+ * the AioHandle, which e.g. allows other backends to access the buffer.
+ *
+ * A callback can be passed a small bit of data, e.g. to indicate whether to
+ * zero a buffer if it is invalid.
+ *
+ * Multiple completion callbacks can be registered for each handle.
+ */
+pgaio_io_register_callbacks(ioh, PGAIO_HCB_SHARED_BUFFER_READV, 0);
+
+/*
+ * The completion callback needs to know which buffers to update when the IO
+ * completes. As the AIO subsystem does not know about buffers, we have to
+ * associate this information with the AioHandle, for use by the completion
+ * callback registered above.
+ *
+ * In this example we're reading only a single buffer, hence the 1.
+ */
+pgaio_io_set_handle_data_32(ioh, (uint32 *) buffer, 1);
+
+/*
+ * Pass the AIO handle to lower-level function. When operating on the level of
+ * buffers, we don't know how exactly the IO is performed, that is the
+ * responsibility of the storage manager implementation.
+ *
+ * E.g. md.c needs to translate block numbers into offsets in segments.
+ *
+ * Once the IO handle has been handed off to smgstartreadv(), it may not
+ * further be used, as the IO may immediately get executed below
+ * smgrstartreadv() and the handle reused for another IO.
+ *
+ * To issue multiple IOs in an efficient way, a caller can call
+ * pgaio_enter_batchmode() before starting multiple IOs, and end that batch
+ * with pgaio_exit_batchmode(). Note that one needs to be careful while there
+ * may be unsubmitted IOs, as another backend may need to wait for one of the
+ * unsubmitted IOs. If this backend then had to wait for the other backend,
+ * it'd end in an undetected deadlock. See pgaio_enter_batchmode() for more
+ * details.
+ *
+ * Note that even while in batchmode an IO might get submitted immediately,
+ * e.g. due to reaching a limit on the number of unsubmitted IOs, and even
+ * complete before smgrstartreadv() returns.
+ */
+smgrstartreadv(ioh, operation->smgr, forknum, blkno,
+ BufferGetBlock(buffer), 1);
+
+/*
+ * To benefit from AIO, it is beneficial to perform other work, including
+ * submitting other IOs, before waiting for the IO to complete. Otherwise
+ * we could just have used synchronous, blocking IO.
+ */
+perform_other_work();
+
+/*
+ * We did some other work and now need the IO operation to have completed to
+ * continue.
+ */
+pgaio_wref_wait(&iow);
+
+/*
+ * At this point the IO has completed. We do not yet know whether it succeeded
+ * or failed, however. The buffer's state has been updated, which allows other
+ * backends to use the buffer (if the IO succeeded), or retry the IO (if it
+ * failed).
+ *
+ * Note that in case the IO has failed, a LOG message may have been emitted,
+ * but no ERROR has been raised. This is crucial, as another backend waiting
+ * for this IO should not see an ERROR.
+ *
+ * To check whether the operation succeeded, and to raise an ERROR, or if more
+ * appropriate LOG, the PgAioReturn we passed to pgaio_io_acquire() is used.
+ */
+if (ioret.result.status == ARS_ERROR)
+ pgaio_result_report(aio_ret.result, &aio_ret.target_data, ERROR);
+
+/*
+ * Besides having succeeded completely, the IO could also have partially
+ * completed. If we e.g. tried to read many blocks at once, the read might have
+ * only succeeded for the first few blocks.
+ *
+ * If the IO partially succeeded and this backend needs all blocks to have
+ * completed, this backend needs to reissue the IO for the remaining buffers.
+ * The AIO subsystem cannot handle this retry transparently.
+ *
+ * As this example is already long, and we only read a single block, we'll just
+ * error out if there's a partial read.
+ */
+if (ioret.result.status == ARS_PARTIAL)
+ pgaio_result_report(aio_ret.result, &aio_ret.target_data, ERROR);
+
+/*
+ * The IO succeeded, so we can use the buffer now.
+ */
+```
+
+
+## Design Criteria & Motivation
+
+### Deadlock and Starvation Dangers due to AIO
+
+Using AIO in a naive way can easily lead to deadlocks in an environment where
+the source/target of AIO are shared resources, like pages in postgres'
+shared_buffers.
+
+Consider one backend performing readahead on a table, initiating IO for a
+number of buffers ahead of the current "scan position". If that backend then
+performs some operation that blocks, or even just is slow, the IO completion
+for the asynchronously initiated read may not be processed.
+
+This AIO implementation solves this problem by requiring that AIO methods
+either allow AIO completions to be processed by any backend in the system
+(e.g. io_uring), or to guarantee that AIO processing will happen even when the
+issuing backend is blocked (e.g. worker mode, which offloads completion
+processing to the AIO workers).
+
+
+### IO can be started in critical sections
+
+Using AIO for WAL writes can reduce the overhead of WAL logging substantially:
+
+- AIO allows to start WAL writes eagerly, so they complete before needing to
+ wait
+- AIO allows to have multiple WAL flushes in progress at the same time
+- AIO makes it more realistic to use O\_DIRECT + O\_DSYNC, which can reduce
+ the number of roundtrips to storage on some OSs and storage HW (buffered IO
+ and direct IO without O_DSYNC needs to issue a write and after the write's
+ completion a cache flush, whereas O\_DIRECT + O\_DSYNC can use a single
+ Force Unit Access (FUA) write).
+
+The need to be able to execute IO in critical sections has substantial design
+implication on the AIO subsystem. Mainly because completing IOs (see prior
+section) needs to be possible within a critical section, even if the
+to-be-completed IO itself was not issued in a critical section. Consider
+e.g. the case of a backend first starting a number of writes from shared
+buffers and then starting to flush the WAL. Because only a limited amount of
+IO can be in-progress at the same time, initiating IO for flushing the WAL may
+require to first complete IO that was started earlier.
+
+
+### State for AIO needs to live in shared memory
+
+Because postgres uses a process model and because AIOs need to be
+complete-able by any backend much of the state of the AIO subsystem needs to
+live in shared memory.
+
+In an `EXEC_BACKEND` build, a backend's executable code and other process
+local state is not necessarily mapped to the same addresses in each process
+due to ASLR. This means that the shared memory cannot contain pointers to
+callbacks.
+
+
+## Design of the AIO Subsystem
+
+
+### AIO Methods
+
+To achieve portability and performance, multiple methods of performing AIO are
+implemented and others are likely worth adding in the future.
+
+
+#### Synchronous Mode
+
+`io_method=sync` does not actually perform AIO but allows to use the AIO API
+while performing synchronous IO. This can be useful for debugging. The code
+for the synchronous mode is also used as a fallback by e.g. the [worker
+mode](#worker) uses it to execute IO that cannot be executed by workers.
+
+
+#### Worker
+
+`io_method=worker` is available on every platform postgres runs on, and
+implements asynchronous IO - from the view of the issuing process - by
+dispatching the IO to one of several worker processes performing the IO in a
+synchronous manner.
+
+
+#### io_uring
+
+`io_method=io_uring` is available on Linux 5.1+. In contrast to worker mode it
+dispatches all IO from within the process, lowering context switch rate /
+latency.
+
+
+### AIO Handles
+
+The central API piece for postgres' AIO abstraction are AIO handles. To
+execute an IO one first has to acquire an IO handle (`pgaio_io_acquire()`) and
+then "defined", i.e. associate an IO operation with the handle.
+
+Often AIO handles are acquired on a higher level and then passed to a lower
+level to be fully defined. E.g., for IO to/from shared buffers, bufmgr.c
+routines acquire the handle, which is then passed through smgr.c, md.c to be
+finally fully defined in fd.c.
+
+The functions used at the lowest level to define the operation are
+`pgaio_io_prep_*()`.
+
+Because acquisition of an IO handle
+[must always succeed](#io-can-be-started-in-critical-sections)
+and the number of AIO Handles
+[has to be limited](#state-for-aio-needs-to-live-in-shared-memory)
+AIO handles can be reused as soon as they have completed. Obviously code needs
+to be able to react to IO completion. State can be updated using
+[AIO Completion callbacks](#aio-callbacks)
+and the issuing backend can provide a backend local variable to receive the
+result of the IO, as described in
+[AIO Result](#aio-results).
+An IO can be waited for, by both the issuing and any other backend, using
+[AIO References](#aio-wait-references).
+
+
+Because an AIO Handle is not executable just after calling
+`pgaio_io_acquire()` and because `pgaio_io_acquire()` needs to always succeed
+(absent a PANIC), only a single AIO Handle may be acquired (i.e. returned by
+`pgaio_io_acquire()`) without causing the IO to have been defined (by,
+potentially indirectly, causing `pgaio_io_prep_*()` to have been
+called). Otherwise a backend could trivially self-deadlock by using up all AIO
+Handles without the ability to wait for some of the IOs to complete.
+
+If it turns out that an AIO Handle is not needed, e.g., because the handle was
+acquired before holding a contended lock, it can be released without being
+defined using `pgaio_io_release()`.
+
+
+### AIO Callbacks
+
+Commonly several layers need to react to completion of an IO. E.g. for a read
+md.c needs to check if the IO outright failed or was shorter than needed,
+bufmgr.c needs to verify the page looks valid and bufmgr.c needs to update the
+BufferDesc to update the buffer's state.
+
+The fact that several layers / subsystems need to react to IO completion poses
+a few challenges:
+
+- Upper layers should not need to know details of lower layers. E.g. bufmgr.c
+ should not assume the IO will pass through md.c. Therefore upper levels
+ cannot know what lower layers would consider an error.
+
+- Lower layers should not need to know about upper layers. E.g. smgr APIs are
+ used going through shared buffers but are also used bypassing shared
+ buffers. This means that e.g. md.c is not in a position to validate
+ checksums.
+
+- Having code in the AIO subsystem for every possible combination of layers
+ would lead to a lot of duplication.
+
+The "solution" to this the ability to associate multiple completion callbacks
+with a handle. E.g. bufmgr.c can have a callback to update the BufferDesc
+state and to verify the page and md.c can have another callback to check if
+the IO operation was successful.
+
+As [mentioned](#state-for-aio-needs-to-live-in-shared-memory), shared memory
+currently cannot contain function pointers. Because of that completion
+callbacks are not directly identified by function pointers but by IDs
+(`PgAioHandleCallbackID`). A substantial added benefit is that that
+allows callbacks to be identified by much smaller amount of memory (a single
+byte currently).
+
+In addition to completion, AIO callbacks also are called to "stage" an
+IO. This is, e.g., used to increase buffer reference counts to account for the
+AIO subsystem referencing the buffer, which is required to handle the case
+where the issuing backend errors out and releases its own pins while the IO is
+still ongoing.
+
+As [explained earlier](#io-can-be-started-in-critical-sections) IO completions
+need to be safe to execute in critical sections. To allow the backend that
+issued the IO to error out in case of failure [AIO Result](#aio-results) can
+be used.
+
+
+### AIO Targets
+
+In addition to the completion callbacks describe above, each AIO Handle has
+exactly one "target". Each target has some space inside an AIO Handle with
+information specific to the target and can provide callbacks to allow to
+reopen the underlying file (required for worker mode) and to describe the IO
+operation (used for debug logging and error messages).
+
+I.e., if two different uses of AIO can describe the identity of the file being
+operated on the same way, it likely makes sense to use the same
+target. E.g. different smgr implementations can describe IO with
+RelFileLocator, ForkNumber and BlockNumber and can thus share a target. In
+contrast, IO for a WAL file would be described with TimeLineID and XLogRecPtr
+and it would not make sense to use the same target for smgr and WAL.
+
+
+### AIO Wait References
+
+As [described above](#aio-handles), AIO Handles can be reused immediately
+after completion and therefore cannot be used to wait for completion of the
+IO. Waiting is enabled using AIO wait references, which do not just identify
+an AIO Handle but also include the handles "generation".
+
+A reference to an AIO Handle can be acquired using `pgaio_io_get_wref()` and
+then waited upon using `pgaio_wref_wait()`.
+
+
+### AIO Results
+
+As AIO completion callbacks
+[are executed in critical sections](#io-can-be-started-in-critical-sections)
+and [may be executed by any backend](#deadlock-and-starvation-dangers-due-to-aio)
+completion callbacks cannot be used to, e.g., make the query that triggered an
+IO ERROR out.
+
+To allow to react to failing IOs the issuing backend can pass a pointer to a
+`PgAioReturn` in backend local memory. Before an AIO Handle is reused the
+`PgAioReturn` is filled with information about the IO. This includes
+information about whether the IO was successful (as a value of
+`PgAioResultStatus`) and enough information to raise an error in case of a
+failure (via `pgaio_result_report()`, with the error details encoded in
+`PgAioResult`).
+
+
+### AIO Errors
+
+It would be very convenient to have shared completion callbacks encode the
+details of errors as an `ErrorData` that could be raised at a later
+time. Unfortunately doing so would require allocating memory. While elog.c can
+guarantee (well, kinda) that logging a message will not run out of memory,
+that only works because a very limited number of messages are in the process
+of being logged. With AIO a large number of concurrently issued AIOs might
+fail.
+
+To avoid the need for preallocating a potentially large amount of memory (in
+shared memory no less!), completion callbacks instead have to encode errors in
+a more compact format that can be converted into an error message.
+
+
+## Helpers
+
+Using the low-level AIO API introduces too much complexity to do so all over
+the tree. Most uses of AIO should be done via reusable, higher-level,
+helpers.
+
+
+### Read Stream
+
+A common and very beneficial use of AIO are reads where a substantial number
+of to-be-read locations are known ahead of time. E.g., for a sequential scan
+the set of blocks that need to be read can be determined solely by knowing the
+current position and checking the buffer mapping table.
+
+The [Read Stream](../../../include/storage/read_stream.h) interface makes it
+comparatively easy to use AIO for such use cases.
diff --git a/src/backend/storage/aio/aio.c b/src/backend/storage/aio/aio.c
index 2e210335872..a0504ee84ee 100644
--- a/src/backend/storage/aio/aio.c
+++ b/src/backend/storage/aio/aio.c
@@ -24,6 +24,8 @@
*
* - read_stream.c - helper for reading buffered relation data
*
+ * - README.md - higher-level overview over AIO
+ *
*
* Portions Copyright (c) 1996-2025, PostgreSQL Global Development Group
* Portions Copyright (c) 1994, Regents of the University of California
--
2.48.1.76.g4e746b1a31.dirty
[text/x-diff] v2.10-0007-localbuf-Track-pincount-in-BufferDesc-as-well.patch (4.5K, ../../zp7obmaqdd6b5u2vhpw27hovpjirtqvjakkbkaz4vju7tohyul@4kfobe7ipgxm/8-v2.10-0007-localbuf-Track-pincount-in-BufferDesc-as-well.patch)
download | inline diff:
From 03799b214e7365defc7b149ce7cf5235297d0580 Mon Sep 17 00:00:00 2001
From: Andres Freund <andres@anarazel.de>
Date: Tue, 18 Mar 2025 14:40:06 -0400
Subject: [PATCH v2.10 07/28] localbuf: Track pincount in BufferDesc as well
For AIO on temporary table buffers the AIO subsystem needs to be able to
ensure a pin on a buffer while AIO is going on, even if the IO issuing query
errors out. Tracking the buffer in LocalRefCount does not work, as it would
cause CheckForLocalBufferLeaks() to assert out.
Instead, also track the refcount in BufferDesc.state, not just
LocalRefCount. This also makes local buffers behave a bit more akin to shared
buffers.
Note that we still don't need locking, AIO completion callbacks for local
buffers are executed in the issuing session (i.e. nobody else has access to
the BufferDesc).
Discussion: https://postgr.es/m/uvrtrknj4kdytuboidbhwclo4gxhswwcpgadptsjvjqcluzmah%40brqs62irg4dt
---
src/backend/storage/buffer/bufmgr.c | 22 +++++++++++++++++++++
src/backend/storage/buffer/localbuf.c | 28 +++++++++++++++++++++++++--
2 files changed, 48 insertions(+), 2 deletions(-)
diff --git a/src/backend/storage/buffer/bufmgr.c b/src/backend/storage/buffer/bufmgr.c
index e76ff1f3224..5fd57bc8d95 100644
--- a/src/backend/storage/buffer/bufmgr.c
+++ b/src/backend/storage/buffer/bufmgr.c
@@ -5350,6 +5350,18 @@ ConditionalLockBufferForCleanup(Buffer buffer)
Assert(refcount > 0);
if (refcount != 1)
return false;
+
+ /*
+ * Check that the AIO subsystem doesn't have a pin. Likely not
+ * possible today, but better safe than sorry.
+ */
+ bufHdr = GetLocalBufferDescriptor(-buffer - 1);
+ buf_state = pg_atomic_read_u32(&bufHdr->state);
+ refcount = BUF_STATE_GET_REFCOUNT(buf_state);
+ Assert(refcount > 0);
+ if (refcount != 1)
+ return false;
+
/* Nobody else to wait for */
return true;
}
@@ -5403,6 +5415,16 @@ IsBufferCleanupOK(Buffer buffer)
/* There should be exactly one pin */
if (LocalRefCount[-buffer - 1] != 1)
return false;
+
+ /*
+ * Check that the AIO subsystem doesn't have a pin. Likely not
+ * possible today, but better safe than sorry.
+ */
+ bufHdr = GetLocalBufferDescriptor(-buffer - 1);
+ buf_state = pg_atomic_read_u32(&bufHdr->state);
+ if (BUF_STATE_GET_REFCOUNT(buf_state) != 1)
+ return false;
+
/* Nobody else to wait for */
return true;
}
diff --git a/src/backend/storage/buffer/localbuf.c b/src/backend/storage/buffer/localbuf.c
index f172a5c7820..3a722321533 100644
--- a/src/backend/storage/buffer/localbuf.c
+++ b/src/backend/storage/buffer/localbuf.c
@@ -249,6 +249,13 @@ GetLocalVictimBuffer(void)
pg_atomic_unlocked_write_u32(&bufHdr->state, buf_state);
trycounter = NLocBuffer;
}
+ else if (BUF_STATE_GET_REFCOUNT(buf_state) > 0)
+ {
+ /*
+ * This can be reached if the backend initiated AIO for this
+ * buffer and then errored out.
+ */
+ }
else
{
/* Found a usable buffer */
@@ -570,7 +577,13 @@ InvalidateLocalBuffer(BufferDesc *bufHdr, bool check_unreferenced)
buf_state = pg_atomic_read_u32(&bufHdr->state);
- if (check_unreferenced && LocalRefCount[bufid] != 0)
+ /*
+ * We need to test not just LocalRefCount[bufid] but also the BufferDesc
+ * itself, as the latter is used to represent a pin by the AIO subsystem.
+ * This can happen if AIO is initiated and then the query errors out.
+ */
+ if (check_unreferenced &&
+ (LocalRefCount[bufid] != 0 || BUF_STATE_GET_REFCOUNT(buf_state) != 0))
elog(ERROR, "block %u of %s is still referenced (local %u)",
bufHdr->tag.blockNum,
relpathbackend(BufTagGetRelFileLocator(&bufHdr->tag),
@@ -744,12 +757,13 @@ PinLocalBuffer(BufferDesc *buf_hdr, bool adjust_usagecount)
if (LocalRefCount[bufid] == 0)
{
NLocalPinnedBuffers++;
+ buf_state += BUF_REFCOUNT_ONE;
if (adjust_usagecount &&
BUF_STATE_GET_USAGECOUNT(buf_state) < BM_MAX_USAGE_COUNT)
{
buf_state += BUF_USAGECOUNT_ONE;
- pg_atomic_unlocked_write_u32(&buf_hdr->state, buf_state);
}
+ pg_atomic_unlocked_write_u32(&buf_hdr->state, buf_state);
}
LocalRefCount[bufid]++;
ResourceOwnerRememberBuffer(CurrentResourceOwner,
@@ -775,7 +789,17 @@ UnpinLocalBufferNoOwner(Buffer buffer)
Assert(NLocalPinnedBuffers > 0);
if (--LocalRefCount[buffid] == 0)
+ {
+ BufferDesc *buf_hdr = GetLocalBufferDescriptor(buffid);
+ uint32 buf_state;
+
NLocalPinnedBuffers--;
+
+ buf_state = pg_atomic_read_u32(&buf_hdr->state);
+ Assert(BUF_STATE_GET_REFCOUNT(buf_state) > 0);
+ buf_state -= BUF_REFCOUNT_ONE;
+ pg_atomic_unlocked_write_u32(&buf_hdr->state, buf_state);
+ }
}
/*
--
2.48.1.76.g4e746b1a31.dirty
[text/x-diff] v2.10-0008-bufmgr-Implement-AIO-read-support.patch (25.3K, ../../zp7obmaqdd6b5u2vhpw27hovpjirtqvjakkbkaz4vju7tohyul@4kfobe7ipgxm/9-v2.10-0008-bufmgr-Implement-AIO-read-support.patch)
download | inline diff:
From 4824a2a39e7cc395d2c05a341186721b0bf38273 Mon Sep 17 00:00:00 2001
From: Andres Freund <andres@anarazel.de>
Date: Tue, 18 Mar 2025 14:40:06 -0400
Subject: [PATCH v2.10 08/28] bufmgr: Implement AIO read support
This implements the following:
- Addition of callbacks to maintain buffer state on completion of a readv
- Addition of a wait reference to BufferDesc, to allow backends to wait for
IOs
- StartBufferIO(), WaitIO(), TerminateBufferIO() support for waiting AIO
As of this commit nothing issues AIO on shared buffers. A future commit will
update StartReadBuffers() to do so.
Reviewed-by: Noah Misch <noah@leadboat.com>
Reviewed-by: Nazir Bilal Yavuz <byavuz81@gmail.com>
Discussion: https://postgr.es/m/uvrtrknj4kdytuboidbhwclo4gxhswwcpgadptsjvjqcluzmah%40brqs62irg4dt
Discussion: https://postgr.es/m/20210223100344.llw5an2aklengrmn@alap3.anarazel.de
Discussion: https://postgr.es/m/stj36ea6yyhoxtqkhpieia2z4krnam7qyetc57rfezgk4zgapf@gcnactj4z56m
---
src/include/storage/aio.h | 4 +
src/include/storage/buf_internals.h | 7 +-
src/include/storage/bufmgr.h | 4 +
src/backend/storage/aio/aio_callback.c | 5 +
src/backend/storage/buffer/buf_init.c | 3 +
src/backend/storage/buffer/bufmgr.c | 440 +++++++++++++++++++++++--
src/backend/storage/buffer/localbuf.c | 42 ++-
7 files changed, 463 insertions(+), 42 deletions(-)
diff --git a/src/include/storage/aio.h b/src/include/storage/aio.h
index 4fe7a6a8b00..c0db8f9754f 100644
--- a/src/include/storage/aio.h
+++ b/src/include/storage/aio.h
@@ -190,6 +190,10 @@ typedef enum PgAioHandleCallbackID
PGAIO_HCB_MD_READV,
PGAIO_HCB_MD_WRITEV,
+
+ PGAIO_HCB_SHARED_BUFFER_READV,
+
+ PGAIO_HCB_LOCAL_BUFFER_READV,
} PgAioHandleCallbackID;
diff --git a/src/include/storage/buf_internals.h b/src/include/storage/buf_internals.h
index 9327f60c44c..8a52ace37cd 100644
--- a/src/include/storage/buf_internals.h
+++ b/src/include/storage/buf_internals.h
@@ -17,6 +17,7 @@
#include "pgstat.h"
#include "port/atomics.h"
+#include "storage/aio_types.h"
#include "storage/buf.h"
#include "storage/bufmgr.h"
#include "storage/condition_variable.h"
@@ -264,6 +265,8 @@ typedef struct BufferDesc
int wait_backend_pgprocno; /* backend of pin-count waiter */
int freeNext; /* link in freelist chain */
+
+ PgAioWaitRef io_wref; /* set iff AIO is in progress */
LWLock content_lock; /* to lock access to buffer contents */
} BufferDesc;
@@ -472,8 +475,8 @@ extern BlockNumber ExtendBufferedRelLocal(BufferManagerRelation bmr,
uint32 *extended_by);
extern void MarkLocalBufferDirty(Buffer buffer);
extern void TerminateLocalBufferIO(BufferDesc *bufHdr, bool clear_dirty,
- uint32 set_flag_bits);
-extern bool StartLocalBufferIO(BufferDesc *bufHdr, bool forInput);
+ uint32 set_flag_bits, bool syncio);
+extern bool StartLocalBufferIO(BufferDesc *bufHdr, bool forInput, bool nowait);
extern void FlushLocalBuffer(BufferDesc *bufHdr, SMgrRelation reln);
extern void DropRelationLocalBuffers(RelFileLocator rlocator,
ForkNumber forkNum,
diff --git a/src/include/storage/bufmgr.h b/src/include/storage/bufmgr.h
index 7f5def6bada..3225bce605f 100644
--- a/src/include/storage/bufmgr.h
+++ b/src/include/storage/bufmgr.h
@@ -15,6 +15,7 @@
#define BUFMGR_H
#include "port/pg_iovec.h"
+#include "storage/aio_types.h"
#include "storage/block.h"
#include "storage/buf.h"
#include "storage/bufpage.h"
@@ -169,6 +170,9 @@ extern PGDLLIMPORT int checkpoint_flush_after;
extern PGDLLIMPORT int backend_flush_after;
extern PGDLLIMPORT int bgwriter_flush_after;
+extern const PgAioHandleCallbacks aio_shared_buffer_readv_cb;
+extern const PgAioHandleCallbacks aio_local_buffer_readv_cb;
+
/* in buf_init.c */
extern PGDLLIMPORT char *BufferBlocks;
diff --git a/src/backend/storage/aio/aio_callback.c b/src/backend/storage/aio/aio_callback.c
index 2d48aa7c1b3..2537b1914d8 100644
--- a/src/backend/storage/aio/aio_callback.c
+++ b/src/backend/storage/aio/aio_callback.c
@@ -18,6 +18,7 @@
#include "miscadmin.h"
#include "storage/aio.h"
#include "storage/aio_internal.h"
+#include "storage/bufmgr.h"
#include "storage/md.h"
@@ -40,6 +41,10 @@ static const PgAioHandleCallbacksEntry aio_handle_cbs[] = {
CALLBACK_ENTRY(PGAIO_HCB_INVALID, aio_invalid_cb),
CALLBACK_ENTRY(PGAIO_HCB_MD_READV, aio_md_readv_cb),
+
+ CALLBACK_ENTRY(PGAIO_HCB_SHARED_BUFFER_READV, aio_shared_buffer_readv_cb),
+
+ CALLBACK_ENTRY(PGAIO_HCB_LOCAL_BUFFER_READV, aio_local_buffer_readv_cb),
#undef CALLBACK_ENTRY
};
diff --git a/src/backend/storage/buffer/buf_init.c b/src/backend/storage/buffer/buf_init.c
index ed1f8e03190..ed1dc488a42 100644
--- a/src/backend/storage/buffer/buf_init.c
+++ b/src/backend/storage/buffer/buf_init.c
@@ -14,6 +14,7 @@
*/
#include "postgres.h"
+#include "storage/aio.h"
#include "storage/buf_internals.h"
#include "storage/bufmgr.h"
@@ -125,6 +126,8 @@ BufferManagerShmemInit(void)
buf->buf_id = i;
+ pgaio_wref_clear(&buf->io_wref);
+
/*
* Initially link all the buffers together as unused. Subsequent
* management of this list is done by freelist.c.
diff --git a/src/backend/storage/buffer/bufmgr.c b/src/backend/storage/buffer/bufmgr.c
index 5fd57bc8d95..c3cdcd8d4ad 100644
--- a/src/backend/storage/buffer/bufmgr.c
+++ b/src/backend/storage/buffer/bufmgr.c
@@ -48,6 +48,7 @@
#include "pg_trace.h"
#include "pgstat.h"
#include "postmaster/bgwriter.h"
+#include "storage/aio.h"
#include "storage/buf_internals.h"
#include "storage/bufmgr.h"
#include "storage/fd.h"
@@ -516,7 +517,8 @@ static int SyncOneBuffer(int buf_id, bool skip_recently_used,
static void WaitIO(BufferDesc *buf);
static bool StartBufferIO(BufferDesc *buf, bool forInput, bool nowait);
static void TerminateBufferIO(BufferDesc *buf, bool clear_dirty,
- uint32 set_flag_bits, bool forget_owner);
+ uint32 set_flag_bits, bool forget_owner,
+ bool syncio);
static void AbortBufferIO(Buffer buffer);
static void shared_buffer_write_error_callback(void *arg);
static void local_buffer_write_error_callback(void *arg);
@@ -1038,7 +1040,7 @@ ZeroAndLockBuffer(Buffer buffer, ReadBufferMode mode, bool already_valid)
{
/* Simple case for non-shared buffers. */
bufHdr = GetLocalBufferDescriptor(-buffer - 1);
- need_to_zero = StartLocalBufferIO(bufHdr, true);
+ need_to_zero = StartLocalBufferIO(bufHdr, true, false);
}
else
{
@@ -1074,9 +1076,9 @@ ZeroAndLockBuffer(Buffer buffer, ReadBufferMode mode, bool already_valid)
/* Set BM_VALID, terminate IO, and wake up any waiters */
if (isLocalBuf)
- TerminateLocalBufferIO(bufHdr, false, BM_VALID);
+ TerminateLocalBufferIO(bufHdr, false, BM_VALID, true);
else
- TerminateBufferIO(bufHdr, false, BM_VALID, true);
+ TerminateBufferIO(bufHdr, false, BM_VALID, true, true);
}
else if (!isLocalBuf)
{
@@ -1388,7 +1390,8 @@ static inline bool
WaitReadBuffersCanStartIO(Buffer buffer, bool nowait)
{
if (BufferIsLocal(buffer))
- return StartLocalBufferIO(GetLocalBufferDescriptor(-buffer - 1), true);
+ return StartLocalBufferIO(GetLocalBufferDescriptor(-buffer - 1),
+ true, nowait);
else
return StartBufferIO(GetBufferDescriptor(buffer - 1), true, nowait);
}
@@ -1550,9 +1553,9 @@ WaitReadBuffers(ReadBuffersOperation *operation)
/* Set BM_VALID, terminate IO, and wake up any waiters */
if (persistence == RELPERSISTENCE_TEMP)
- TerminateLocalBufferIO(bufHdr, false, BM_VALID);
+ TerminateLocalBufferIO(bufHdr, false, BM_VALID, true);
else
- TerminateBufferIO(bufHdr, false, BM_VALID, true);
+ TerminateBufferIO(bufHdr, false, BM_VALID, true, true);
/* Report I/Os as completing individually. */
TRACE_POSTGRESQL_BUFFER_READ_DONE(forknum, io_first_block + j,
@@ -2460,7 +2463,7 @@ ExtendBufferedRelShared(BufferManagerRelation bmr,
if (lock)
LWLockAcquire(BufferDescriptorGetContentLock(buf_hdr), LW_EXCLUSIVE);
- TerminateBufferIO(buf_hdr, false, BM_VALID, true);
+ TerminateBufferIO(buf_hdr, false, BM_VALID, true, true);
}
pgBufferUsage.shared_blks_written += extend_by;
@@ -2806,6 +2809,44 @@ PinBuffer_Locked(BufferDesc *buf)
ResourceOwnerRememberBuffer(CurrentResourceOwner, b);
}
+/*
+ * Support for waking up another backend that is waiting for the cleanup lock
+ * to be released using BM_PIN_COUNT_WAITER.
+ *
+ * See LockBufferForCleanup().
+ *
+ * Expected to be called just after releasing a buffer pin (in a BufferDesc,
+ * not just reducing the backend-local pincount for the buffer).
+ */
+static void
+WakePinCountWaiter(BufferDesc *buf)
+{
+ /*
+ * Acquire the buffer header lock, re-check that there's a waiter. Another
+ * backend could have unpinned this buffer, and already woken up the
+ * waiter.
+ *
+ * There's no danger of the buffer being replaced after we unpinned it
+ * above, as it's pinned by the waiter. The waiter removes
+ * BM_PIN_COUNT_WAITER if it stops waiting for a reason other than this
+ * backend waking it up.
+ */
+ uint32 buf_state = LockBufHdr(buf);
+
+ if ((buf_state & BM_PIN_COUNT_WAITER) &&
+ BUF_STATE_GET_REFCOUNT(buf_state) == 1)
+ {
+ /* we just released the last pin other than the waiter's */
+ int wait_backend_pgprocno = buf->wait_backend_pgprocno;
+
+ buf_state &= ~BM_PIN_COUNT_WAITER;
+ UnlockBufHdr(buf, buf_state);
+ ProcSendSignal(wait_backend_pgprocno);
+ }
+ else
+ UnlockBufHdr(buf, buf_state);
+}
+
/*
* UnpinBuffer -- make buffer available for replacement.
*
@@ -2874,29 +2915,8 @@ UnpinBufferNoOwner(BufferDesc *buf)
/* Support LockBufferForCleanup() */
if (buf_state & BM_PIN_COUNT_WAITER)
- {
- /*
- * Acquire the buffer header lock, re-check that there's a waiter.
- * Another backend could have unpinned this buffer, and already
- * woken up the waiter. There's no danger of the buffer being
- * replaced after we unpinned it above, as it's pinned by the
- * waiter.
- */
- buf_state = LockBufHdr(buf);
+ WakePinCountWaiter(buf);
- if ((buf_state & BM_PIN_COUNT_WAITER) &&
- BUF_STATE_GET_REFCOUNT(buf_state) == 1)
- {
- /* we just released the last pin other than the waiter's */
- int wait_backend_pgprocno = buf->wait_backend_pgprocno;
-
- buf_state &= ~BM_PIN_COUNT_WAITER;
- UnlockBufHdr(buf, buf_state);
- ProcSendSignal(wait_backend_pgprocno);
- }
- else
- UnlockBufHdr(buf, buf_state);
- }
ForgetPrivateRefCountEntry(ref);
}
}
@@ -3916,7 +3936,7 @@ FlushBuffer(BufferDesc *buf, SMgrRelation reln, IOObject io_object,
* Mark the buffer as clean (unless BM_JUST_DIRTIED has become set) and
* end the BM_IO_IN_PROGRESS state.
*/
- TerminateBufferIO(buf, true, 0, true);
+ TerminateBufferIO(buf, true, 0, true, true);
TRACE_POSTGRESQL_BUFFER_FLUSH_DONE(BufTagGetForkNum(&buf->tag),
buf->tag.blockNum,
@@ -5475,6 +5495,7 @@ WaitIO(BufferDesc *buf)
for (;;)
{
uint32 buf_state;
+ PgAioWaitRef iow;
/*
* It may not be necessary to acquire the spinlock to check the flag
@@ -5482,10 +5503,19 @@ WaitIO(BufferDesc *buf)
* play it safe.
*/
buf_state = LockBufHdr(buf);
+ iow = buf->io_wref;
UnlockBufHdr(buf, buf_state);
if (!(buf_state & BM_IO_IN_PROGRESS))
break;
+
+ if (pgaio_wref_valid(&iow))
+ {
+ pgaio_wref_wait(&iow);
+ ConditionVariablePrepareToSleep(cv);
+ continue;
+ }
+
ConditionVariableSleep(cv, WAIT_EVENT_BUFFER_IO);
}
ConditionVariableCancelSleep();
@@ -5574,7 +5604,7 @@ StartBufferIO(BufferDesc *buf, bool forInput, bool nowait)
*/
static void
TerminateBufferIO(BufferDesc *buf, bool clear_dirty, uint32 set_flag_bits,
- bool forget_owner)
+ bool forget_owner, bool syncio)
{
uint32 buf_state;
@@ -5589,6 +5619,14 @@ TerminateBufferIO(BufferDesc *buf, bool clear_dirty, uint32 set_flag_bits,
if (clear_dirty && !(buf_state & BM_JUST_DIRTIED))
buf_state &= ~(BM_DIRTY | BM_CHECKPOINT_NEEDED);
+ if (!syncio)
+ {
+ /* release ownership by the AIO subsystem */
+ Assert(BUF_STATE_GET_REFCOUNT(buf_state) > 0);
+ buf_state -= BUF_REFCOUNT_ONE;
+ pgaio_wref_clear(&buf->io_wref);
+ }
+
buf_state |= set_flag_bits;
UnlockBufHdr(buf, buf_state);
@@ -5597,6 +5635,17 @@ TerminateBufferIO(BufferDesc *buf, bool clear_dirty, uint32 set_flag_bits,
BufferDescriptorGetBuffer(buf));
ConditionVariableBroadcast(BufferDescriptorGetIOCV(buf));
+
+ /*
+ * Support LockBufferForCleanup()
+ *
+ * If we just released a pin, need to do BM_PIN_COUNT_WAITER handling.
+ * Most of the time the current backend will hold another pin preventing
+ * that from happening, but that's e.g. not the case when completing an IO
+ * another backend started.
+ */
+ if (buf_state & BM_PIN_COUNT_WAITER)
+ WakePinCountWaiter(buf);
}
/*
@@ -5645,7 +5694,7 @@ AbortBufferIO(Buffer buffer)
}
}
- TerminateBufferIO(buf_hdr, false, BM_IO_ERROR, false);
+ TerminateBufferIO(buf_hdr, false, BM_IO_ERROR, false, true);
}
/*
@@ -6096,3 +6145,328 @@ EvictUnpinnedBuffer(Buffer buf)
return result;
}
+
+/*
+ * Helper for AIO staging callback for both reads and writes as well as temp
+ * and shared buffers.
+ */
+static pg_attribute_always_inline void
+buffer_stage_common(PgAioHandle *ioh, bool is_write, bool is_temp)
+{
+ uint64 *io_data;
+ uint8 handle_data_len;
+ PgAioWaitRef io_ref;
+ BufferTag first PG_USED_FOR_ASSERTS_ONLY = {0};
+
+ io_data = pgaio_io_get_handle_data(ioh, &handle_data_len);
+
+ pgaio_io_get_wref(ioh, &io_ref);
+
+ for (int i = 0; i < handle_data_len; i++)
+ {
+ Buffer buffer = (Buffer) io_data[i];
+ BufferDesc *buf_hdr = is_temp ?
+ GetLocalBufferDescriptor(-buffer - 1)
+ : GetBufferDescriptor(buffer - 1);
+ uint32 buf_state;
+
+ /*
+ * Check that all the buffers are actually ones that could conceivably
+ * be done in one IO, i.e. are sequential.
+ */
+ if (i == 0)
+ first = buf_hdr->tag;
+ else
+ {
+ Assert(buf_hdr->tag.relNumber == first.relNumber);
+ Assert(buf_hdr->tag.blockNum == first.blockNum + i);
+ }
+
+ if (is_temp)
+ buf_state = pg_atomic_read_u32(&buf_hdr->state);
+ else
+ buf_state = LockBufHdr(buf_hdr);
+
+ Assert(buf_state & BM_TAG_VALID);
+ if (is_write)
+ {
+ Assert(buf_state & BM_VALID);
+ Assert(buf_state & BM_DIRTY);
+ }
+ else
+ {
+ Assert(!(buf_state & BM_VALID));
+ Assert(!(buf_state & BM_DIRTY));
+ }
+
+ /* temp buffers don't use BM_IO_IN_PROGRESS */
+ if (!is_temp)
+ Assert(buf_state & BM_IO_IN_PROGRESS);
+
+ /* we better have ensured the buffer is present until now */
+ Assert(BUF_STATE_GET_REFCOUNT(buf_state) >= 1);
+
+ /*
+ * Reflect that the buffer is now owned by the subsystem.
+ *
+ * For local buffers: This can't be done just in LocalRefCount as one
+ * might initially think, as this backend could error out while AIO is
+ * still in progress, releasing all the pins by the backend itself.
+ */
+ buf_state += BUF_REFCOUNT_ONE;
+ buf_hdr->io_wref = io_ref;
+
+ if (is_temp)
+ pg_atomic_unlocked_write_u32(&buf_hdr->state, buf_state);
+ else
+ UnlockBufHdr(buf_hdr, buf_state);
+
+ if (is_write && !is_temp)
+ {
+ LWLock *content_lock;
+
+ content_lock = BufferDescriptorGetContentLock(buf_hdr);
+
+ Assert(LWLockHeldByMe(content_lock));
+
+ /*
+ * Lock is now owned by AIO subsystem.
+ */
+ LWLockDisown(content_lock);
+ }
+
+ /*
+ * Stop tracking this buffer via the resowner - the AIO system now
+ * keeps track.
+ */
+ if (!is_temp)
+ ResourceOwnerForgetBufferIO(CurrentResourceOwner, buffer);
+ }
+}
+
+/*
+ * Helper for AIO readv completion callbacks, supporting both shared and temp
+ * buffers. Gets called once for each buffer in a multi-page read.
+ */
+static pg_attribute_always_inline PgAioResult
+buffer_readv_complete_one(uint8 buf_off, Buffer buffer, uint8 flags,
+ bool failed, bool is_temp)
+{
+ BufferDesc *buf_hdr = is_temp ?
+ GetLocalBufferDescriptor(-buffer - 1)
+ : GetBufferDescriptor(buffer - 1);
+ BufferTag tag = buf_hdr->tag;
+ char *bufdata = BufferGetBlock(buffer);
+ PgAioResult result;
+ uint32 set_flag_bits;
+
+#ifdef USE_ASSERT_CHECKING
+ {
+ uint32 buf_state = pg_atomic_read_u32(&buf_hdr->state);
+
+ Assert(buf_state & BM_TAG_VALID);
+ Assert(!(buf_state & BM_VALID));
+ /* temp buffers don't use BM_IO_IN_PROGRESS */
+ if (!is_temp)
+ Assert(buf_state & BM_IO_IN_PROGRESS);
+ Assert(!(buf_state & BM_DIRTY));
+ }
+#endif
+
+ result.status = ARS_OK;
+
+ /* check for garbage data */
+ if (!failed &&
+ !PageIsVerifiedExtended((Page) bufdata, tag.blockNum,
+ PIV_LOG_WARNING | PIV_REPORT_STAT))
+ {
+ RelFileLocator rlocator = BufTagGetRelFileLocator(&tag);
+
+ if ((flags & READ_BUFFERS_ZERO_ON_ERROR) || zero_damaged_pages)
+ {
+ ereport(LOG,
+ (errcode(ERRCODE_DATA_CORRUPTED),
+ errmsg("invalid page in block %u of relation %s; zeroing out page",
+ tag.blockNum,
+ relpathbackend(rlocator,
+ is_temp ? MyProcNumber : INVALID_PROC_NUMBER,
+ tag.forkNum).str)));
+ memset(bufdata, 0, BLCKSZ);
+ }
+ else
+ {
+ /* mark buffer as having failed */
+ failed = true;
+
+ /* encode error for buffer_readv_report */
+ result.status = ARS_ERROR;
+ if (is_temp)
+ result.id = PGAIO_HCB_LOCAL_BUFFER_READV;
+ else
+ result.id = PGAIO_HCB_SHARED_BUFFER_READV;
+ result.error_data = buf_off;
+ }
+ }
+
+ /* Terminate I/O and set BM_VALID. */
+ set_flag_bits = failed ? BM_IO_ERROR : BM_VALID;
+ if (is_temp)
+ TerminateLocalBufferIO(buf_hdr, false, set_flag_bits, false);
+ else
+ TerminateBufferIO(buf_hdr, false, set_flag_bits, false, false);
+
+ /*
+ * Call the BUFFER_READ_DONE tracepoint in the callback, even though the
+ * callback may not be executed in the same backend that called
+ * BUFFER_READ_START. The alternative would be to defer calling the
+ * tracepoint to a later point (e.g. the local completion callback for
+ * shared buffer reads), which seems even less helpful and would also
+ * imply some overhead.
+ */
+ TRACE_POSTGRESQL_BUFFER_READ_DONE(tag.forkNum,
+ tag.blockNum,
+ tag.spcOid,
+ tag.dbOid,
+ tag.relNumber,
+ is_temp ? MyProcNumber : INVALID_PROC_NUMBER,
+ false);
+
+ return result;
+}
+
+/*
+ * Perform completion handling of a single AIO read. This read may cover
+ * multiple blocks / buffers.
+ *
+ * Shared between shared and local buffers, to reduce code duplication.
+ */
+static pg_attribute_always_inline PgAioResult
+buffer_readv_complete(PgAioHandle *ioh, PgAioResult prior_result,
+ uint8 cb_data, bool is_temp)
+{
+ PgAioResult result = prior_result;
+ PgAioTargetData *td = pgaio_io_get_target_data(ioh);
+ uint64 *io_data;
+ uint8 handle_data_len;
+
+ if (is_temp)
+ {
+ Assert(td->smgr.is_temp);
+ Assert(pgaio_io_get_owner(ioh) == MyProcNumber);
+ }
+ else
+ Assert(!td->smgr.is_temp);
+
+ /*
+ * Iterate over all the buffers affected by this IO and call appropriate
+ * per-buffer completion function for each buffer.
+ */
+ io_data = pgaio_io_get_handle_data(ioh, &handle_data_len);
+ for (uint8 buf_off = 0; buf_off < handle_data_len; buf_off++)
+ {
+ Buffer buf = io_data[buf_off];
+ PgAioResult buf_result;
+ bool failed;
+
+ Assert(BufferIsValid(buf));
+
+ /*
+ * If the entire failed on a lower-level, each buffer needs to be
+ * marked as failed. In case of a partial read, some buffers may be
+ * ok.
+ */
+ failed =
+ prior_result.status == ARS_ERROR
+ || prior_result.result <= buf_off;
+
+ buf_result = buffer_readv_complete_one(buf_off, buf, cb_data, failed,
+ is_temp);
+
+ /*
+ * If there wasn't any prior error and the IO for this page failed in
+ * some form, set the whole IO's to the page's result.
+ */
+ if (result.status != ARS_ERROR && buf_result.status != ARS_OK)
+ {
+ result = buf_result;
+ pgaio_result_report(result, td, LOG);
+ }
+ }
+
+ return result;
+}
+
+/*
+ * AIO error reporting callback for aio_shared_buffer_readv_cb and
+ * aio_local_buffer_readv_cb.
+ *
+ * Errors are encoded as follows:
+ * - the only error is page verification failing
+ * - result.error_data is the offset of the first page that failed
+ * verification in a larger IO
+ */
+static void
+buffer_readv_report(PgAioResult result, const PgAioTargetData *target_data, int elevel)
+{
+ ProcNumber errProc;
+
+ if (target_data->smgr.is_temp)
+ errProc = MyProcNumber;
+ else
+ errProc = INVALID_PROC_NUMBER;
+
+ ereport(elevel,
+ errcode(ERRCODE_DATA_CORRUPTED),
+ errmsg("invalid page in block %u of relation %s",
+ target_data->smgr.blockNum + result.error_data,
+ relpathbackend(target_data->smgr.rlocator, errProc, target_data->smgr.forkNum).str
+ )
+ );
+}
+
+static void
+shared_buffer_readv_stage(PgAioHandle *ioh, uint8 cb_data)
+{
+ buffer_stage_common(ioh, false, false);
+}
+
+static PgAioResult
+shared_buffer_readv_complete(PgAioHandle *ioh, PgAioResult prior_result,
+ uint8 cb_data)
+{
+ return buffer_readv_complete(ioh, prior_result, cb_data, false);
+}
+
+static void
+local_buffer_readv_stage(PgAioHandle *ioh, uint8 cb_data)
+{
+ buffer_stage_common(ioh, false, true);
+}
+
+static PgAioResult
+local_buffer_readv_complete(PgAioHandle *ioh, PgAioResult prior_result,
+ uint8 cb_data)
+{
+ return buffer_readv_complete(ioh, prior_result, cb_data, true);
+}
+
+/* readv callback is passed READ_BUFFERS_* flags as callback data */
+const PgAioHandleCallbacks aio_shared_buffer_readv_cb = {
+ .stage = shared_buffer_readv_stage,
+ .complete_shared = shared_buffer_readv_complete,
+ .report = buffer_readv_report,
+};
+
+/* readv callback is passed READ_BUFFERS_* flags as callback data */
+const PgAioHandleCallbacks aio_local_buffer_readv_cb = {
+ .stage = local_buffer_readv_stage,
+
+ /*
+ * Note that this, in contrast to the shared_buffers case, uses
+ * complete_local, as only the issuing backend has access to the required
+ * datastructures. This is important in case the IO completion may be
+ * consumed incidentally by another backend.
+ */
+ .complete_local = local_buffer_readv_complete,
+ .report = buffer_readv_report,
+};
diff --git a/src/backend/storage/buffer/localbuf.c b/src/backend/storage/buffer/localbuf.c
index 3a722321533..a58e917a788 100644
--- a/src/backend/storage/buffer/localbuf.c
+++ b/src/backend/storage/buffer/localbuf.c
@@ -17,7 +17,9 @@
#include "access/parallel.h"
#include "executor/instrument.h"
+#include "pg_trace.h"
#include "pgstat.h"
+#include "storage/aio.h"
#include "storage/buf_internals.h"
#include "storage/bufmgr.h"
#include "storage/fd.h"
@@ -187,7 +189,7 @@ FlushLocalBuffer(BufferDesc *bufHdr, SMgrRelation reln)
* Try to start an I/O operation. There currently are no reasons for
* StartLocalBufferIO to return false, so we raise an error in that case.
*/
- if (!StartLocalBufferIO(bufHdr, false))
+ if (!StartLocalBufferIO(bufHdr, false, false))
elog(ERROR, "failed to start write IO on local buffer");
/* Find smgr relation for buffer */
@@ -211,7 +213,7 @@ FlushLocalBuffer(BufferDesc *bufHdr, SMgrRelation reln)
IOOP_WRITE, io_start, 1, BLCKSZ);
/* Mark not-dirty */
- TerminateLocalBufferIO(bufHdr, true, 0);
+ TerminateLocalBufferIO(bufHdr, true, 0, true);
pgBufferUsage.local_blks_written++;
}
@@ -430,7 +432,7 @@ ExtendBufferedRelLocal(BufferManagerRelation bmr,
pg_atomic_unlocked_write_u32(&existing_hdr->state, buf_state);
/* no need to loop for local buffers */
- StartLocalBufferIO(existing_hdr, true);
+ StartLocalBufferIO(existing_hdr, true, false);
}
else
{
@@ -446,7 +448,7 @@ ExtendBufferedRelLocal(BufferManagerRelation bmr,
hresult->id = victim_buf_id;
- StartLocalBufferIO(victim_buf_hdr, true);
+ StartLocalBufferIO(victim_buf_hdr, true, false);
}
}
@@ -515,10 +517,25 @@ MarkLocalBufferDirty(Buffer buffer)
* Like StartBufferIO, but for local buffers
*/
bool
-StartLocalBufferIO(BufferDesc *bufHdr, bool forInput)
+StartLocalBufferIO(BufferDesc *bufHdr, bool forInput, bool nowait)
{
- uint32 buf_state = pg_atomic_read_u32(&bufHdr->state);
+ uint32 buf_state;
+ /*
+ * The buffer could have IO in progress, e.g. when there are two scans of
+ * the same relation. Either wait for the other IO or return false.
+ */
+ if (pgaio_wref_valid(&bufHdr->io_wref))
+ {
+ PgAioWaitRef iow = bufHdr->io_wref;
+
+ if (nowait)
+ return false;
+
+ pgaio_wref_wait(&iow);
+ }
+
+ buf_state = pg_atomic_read_u32(&bufHdr->state);
if (forInput ? (buf_state & BM_VALID) : !(buf_state & BM_DIRTY))
{
/* someone else already did the I/O */
@@ -536,7 +553,8 @@ StartLocalBufferIO(BufferDesc *bufHdr, bool forInput)
* Like TerminateBufferIO, but for local buffers
*/
void
-TerminateLocalBufferIO(BufferDesc *bufHdr, bool clear_dirty, uint32 set_flag_bits)
+TerminateLocalBufferIO(BufferDesc *bufHdr, bool clear_dirty, uint32 set_flag_bits,
+ bool syncio)
{
/* Only need to adjust flags */
uint32 buf_state = pg_atomic_read_u32(&bufHdr->state);
@@ -549,6 +567,14 @@ TerminateLocalBufferIO(BufferDesc *bufHdr, bool clear_dirty, uint32 set_flag_bit
if (clear_dirty)
buf_state &= ~BM_DIRTY;
+ if (!syncio)
+ {
+ /* release pin held by IO subsystem, see also buffer_stage_common() */
+ Assert(BUF_STATE_GET_REFCOUNT(buf_state) > 0);
+ buf_state -= BUF_REFCOUNT_ONE;
+ pgaio_wref_clear(&bufHdr->io_wref);
+ }
+
buf_state |= set_flag_bits;
pg_atomic_unlocked_write_u32(&bufHdr->state, buf_state);
@@ -714,6 +740,8 @@ InitLocalBuffers(void)
*/
buf->buf_id = -i - 2;
+ pgaio_wref_clear(&buf->io_wref);
+
/*
* Intentionally do not initialize the buffer's atomic variable
* (besides zeroing the underlying memory above). That way we get
--
2.48.1.76.g4e746b1a31.dirty
[text/x-diff] v2.10-0009-Support-buffer-forwarding-in-read_stream.c.patch (8.7K, ../../zp7obmaqdd6b5u2vhpw27hovpjirtqvjakkbkaz4vju7tohyul@4kfobe7ipgxm/10-v2.10-0009-Support-buffer-forwarding-in-read_stream.c.patch)
download | inline diff:
From 8ead766192b863a8738208e358f07d845a91677a Mon Sep 17 00:00:00 2001
From: Thomas Munro <thomas.munro@gmail.com>
Date: Tue, 18 Mar 2025 14:40:06 -0400
Subject: [PATCH v2.10 09/28] Support buffer forwarding in read_stream.c.
In preparation for a following change to the buffer manager, teach
streams to keep track of buffers that were "forwarded" from one call to
StartReadBuffers() to the next.
Since StartReadBuffers() buffers argument will become an in/out
argument, we need to initialize the buffer queue entries with
InvalidBuffer. We don't want to do that up front, because we try to
keep stream initialization cheap for code that uses the fast path and
stays in one single buffer queue element. Satisfy both goals by
initializing the queue incrementally on the first cycle.
Discussion: https://postgr.es/m/CA%2BhUKGK_%3D4CVmMHvsHjOVrK6t4F%3DLBpFzsrr3R%2BaJYN8kcTfWg%40mail.gmail.com
---
src/backend/storage/aio/read_stream.c | 103 +++++++++++++++++++++++---
1 file changed, 93 insertions(+), 10 deletions(-)
diff --git a/src/backend/storage/aio/read_stream.c b/src/backend/storage/aio/read_stream.c
index d65fa07b44c..cdf4b5a86a2 100644
--- a/src/backend/storage/aio/read_stream.c
+++ b/src/backend/storage/aio/read_stream.c
@@ -95,8 +95,10 @@ struct ReadStream
int16 ios_in_progress;
int16 queue_size;
int16 max_pinned_buffers;
+ int16 forwarded_buffers;
int16 pinned_buffers;
int16 distance;
+ int16 initialized_buffers;
bool advice_enabled;
bool temporary;
@@ -224,8 +226,10 @@ static bool
read_stream_start_pending_read(ReadStream *stream)
{
bool need_wait;
+ int requested_nblocks;
int nblocks;
int flags;
+ int forwarded;
int16 io_index;
int16 overflow;
int16 buffer_index;
@@ -272,11 +276,20 @@ read_stream_start_pending_read(ReadStream *stream)
}
}
- /* How many more buffers is this backend allowed? */
+ /*
+ * How many more buffers is this backend allowed?
+ *
+ * If we already have some forwarded buffers, we can certainly use those.
+ * They are already pinned, and are mapped to the starting blocks of the
+ * pending read, they just don't have any I/O started yet and are not
+ * counted in stream->pinned_buffers.
+ */
if (stream->temporary)
buffer_limit = Min(GetAdditionalLocalPinLimit(), PG_INT16_MAX);
else
buffer_limit = Min(GetAdditionalPinLimit(), PG_INT16_MAX);
+ Assert(stream->forwarded_buffers <= stream->pending_read_nblocks);
+ buffer_limit += stream->forwarded_buffers;
if (buffer_limit == 0 && stream->pinned_buffers == 0)
buffer_limit = 1; /* guarantee progress */
@@ -303,8 +316,31 @@ read_stream_start_pending_read(ReadStream *stream)
* We say how many blocks we want to read, but it may be smaller on return
* if the buffer manager decides to shorten the read.
*/
+ requested_nblocks = Min(buffer_limit, stream->pending_read_nblocks);
+ nblocks = requested_nblocks;
buffer_index = stream->next_buffer_index;
io_index = stream->next_io_index;
+
+ /*
+ * The first time around the queue we initialize it as we go, including
+ * the overflow zone, because otherwise the entries would appear as
+ * forwarded buffers. This avoids initializing the whole queue up front
+ * in cases where it is large but we don't ever use it due to the
+ * all-cached fast path or small scans.
+ */
+ while (stream->initialized_buffers < buffer_index + nblocks)
+ stream->buffers[stream->initialized_buffers++] = InvalidBuffer;
+
+ /*
+ * Start the I/O. Any buffers that are not InvalidBuffer will be
+ * interpreted as already pinned, forwarded by an earlier call to
+ * StartReadBuffers(), and must map to the expected blocks. The nblocks
+ * value may be smaller on return indicating the size of the I/O that
+ * could be started. Buffers beyond the output nblocks number may also
+ * have been pinned without starting I/O due to various edge cases. In
+ * that case we'll just leave them in the queue ahead of us, "forwarded"
+ * to the next call, avoiding the need to unpin/repin.
+ */
need_wait = StartReadBuffers(&stream->ios[io_index].op,
&stream->buffers[buffer_index],
stream->pending_read_blocknum,
@@ -333,16 +369,35 @@ read_stream_start_pending_read(ReadStream *stream)
stream->seq_blocknum = stream->pending_read_blocknum + nblocks;
}
+ /*
+ * How many pins were acquired but forwarded to the next call? These need
+ * to be passed to the next StartReadBuffers() call, or released if the
+ * stream ends early. We need the number for accounting purposes, since
+ * they are not counted in stream->pinned_buffers but we already hold
+ * them.
+ */
+ forwarded = 0;
+ while (nblocks + forwarded < requested_nblocks &&
+ stream->buffers[buffer_index + nblocks + forwarded] != InvalidBuffer)
+ forwarded++;
+ stream->forwarded_buffers = forwarded;
+
/*
* We gave a contiguous range of buffer space to StartReadBuffers(), but
- * we want it to wrap around at queue_size. Slide overflowing buffers to
- * the front of the array.
+ * we want it to wrap around at queue_size. Copy overflowing buffers to
+ * the front of the array where they'll be consumed, but also leave a copy
+ * in the overflow zone which the I/O operation has a pointer to (it needs
+ * a contiguous array). Both copies will be cleared when the buffers are
+ * handed to the consumer.
*/
- overflow = (buffer_index + nblocks) - stream->queue_size;
+ overflow = (buffer_index + nblocks + forwarded) - stream->queue_size;
if (overflow > 0)
- memmove(&stream->buffers[0],
- &stream->buffers[stream->queue_size],
- sizeof(stream->buffers[0]) * overflow);
+ {
+ Assert(overflow < stream->queue_size); /* can't overlap */
+ memcpy(&stream->buffers[0],
+ &stream->buffers[stream->queue_size],
+ sizeof(stream->buffers[0]) * overflow);
+ }
/* Compute location of start of next read, without using % operator. */
buffer_index += nblocks;
@@ -718,10 +773,12 @@ read_stream_next_buffer(ReadStream *stream, void **per_buffer_data)
/* Fast path assumptions. */
Assert(stream->ios_in_progress == 0);
+ Assert(stream->forwarded_buffers == 0);
Assert(stream->pinned_buffers == 1);
Assert(stream->distance == 1);
Assert(stream->pending_read_nblocks == 0);
Assert(stream->per_buffer_data_size == 0);
+ Assert(stream->initialized_buffers > stream->oldest_buffer_index);
/* We're going to return the buffer we pinned last time. */
oldest_buffer_index = stream->oldest_buffer_index;
@@ -770,6 +827,7 @@ read_stream_next_buffer(ReadStream *stream, void **per_buffer_data)
stream->distance = 0;
stream->oldest_buffer_index = stream->next_buffer_index;
stream->pinned_buffers = 0;
+ stream->buffers[oldest_buffer_index] = InvalidBuffer;
}
stream->fast_path = false;
@@ -845,10 +903,15 @@ read_stream_next_buffer(ReadStream *stream, void **per_buffer_data)
stream->seq_until_processed = InvalidBlockNumber;
}
-#ifdef CLOBBER_FREED_MEMORY
- /* Clobber old buffer for debugging purposes. */
+ /*
+ * We must zap this queue entry, or else it would appear as a forwarded
+ * buffer. If it's potentially in the overflow zone (ie it wrapped around
+ * the queue), also zap that copy.
+ */
stream->buffers[oldest_buffer_index] = InvalidBuffer;
-#endif
+ if (oldest_buffer_index < io_combine_limit - 1)
+ stream->buffers[stream->queue_size + oldest_buffer_index] =
+ InvalidBuffer;
#if defined(CLOBBER_FREED_MEMORY) || defined(USE_VALGRIND)
@@ -893,6 +956,7 @@ read_stream_next_buffer(ReadStream *stream, void **per_buffer_data)
#ifndef READ_STREAM_DISABLE_FAST_PATH
/* See if we can take the fast path for all-cached scans next time. */
if (stream->ios_in_progress == 0 &&
+ stream->forwarded_buffers == 0 &&
stream->pinned_buffers == 1 &&
stream->distance == 1 &&
stream->pending_read_nblocks == 0 &&
@@ -928,6 +992,7 @@ read_stream_next_block(ReadStream *stream, BufferAccessStrategy *strategy)
void
read_stream_reset(ReadStream *stream)
{
+ int16 index;
Buffer buffer;
/* Stop looking ahead. */
@@ -941,6 +1006,24 @@ read_stream_reset(ReadStream *stream)
while ((buffer = read_stream_next_buffer(stream, NULL)) != InvalidBuffer)
ReleaseBuffer(buffer);
+ /* Unpin any unused forwarded buffers. */
+ index = stream->next_buffer_index;
+ while (index < stream->initialized_buffers &&
+ (buffer = stream->buffers[index]) != InvalidBuffer)
+ {
+ Assert(stream->forwarded_buffers > 0);
+ stream->forwarded_buffers--;
+ ReleaseBuffer(buffer);
+
+ stream->buffers[index] = InvalidBuffer;
+ if (index < io_combine_limit - 1)
+ stream->buffers[stream->queue_size + index] = InvalidBuffer;
+
+ if (++index == stream->queue_size)
+ index = 0;
+ }
+
+ Assert(stream->forwarded_buffers == 0);
Assert(stream->pinned_buffers == 0);
Assert(stream->ios_in_progress == 0);
--
2.48.1.76.g4e746b1a31.dirty
[text/x-diff] v2.10-0010-Support-buffer-forwarding-in-StartReadBuffers.patch (10.5K, ../../zp7obmaqdd6b5u2vhpw27hovpjirtqvjakkbkaz4vju7tohyul@4kfobe7ipgxm/11-v2.10-0010-Support-buffer-forwarding-in-StartReadBuffers.patch)
download | inline diff:
From 791fe26b9ea4fd646186f2f41fbc185fd75aa6c3 Mon Sep 17 00:00:00 2001
From: Thomas Munro <thomas.munro@gmail.com>
Date: Tue, 18 Mar 2025 14:40:06 -0400
Subject: [PATCH v2.10 10/28] Support buffer forwarding in StartReadBuffers().
Sometimes we have to perform a short read because we hit a cached block
that ends a contiguous run of blocks requiring I/O. We don't want
StartReadBuffers() to have to start more than one I/O, so we stop there.
We also don't want to have to unpin the cached block (and repin it
later), so previously we'd silently pretend the hit was part of the I/O,
and just leave it out of the read from disk. Now, we'll "forward" it to
the next call. We still write it to the buffers[] array for the caller
to pass back to us later, but it's not included in *nblocks.
This policy means that we no longer mix hits and misses in a single
operation's results, so we avoid the requirement to call
WaitReadBuffers(), which might stall, before the caller can make use of
the hits. The caller will get the hit in the next call instead, and
know that it doesn't have to wait. That's important for later work on
out-of-order read streams that minimize I/O stalls.
This also makes life easier for proposed work on true AIO, which
occasionally needs to split a large I/O after pinning all the buffers,
while the current coding only ever forwards a single bookending hit.
This API is natural for read_stream.c: it just leaves forwarded buffers
where they are in its circular queue, where the next call will pick them
up and continue, minimizing pin churn.
If we ever think of a good reason to disable this feature, i.e. for
other users of StartReadBuffers() that don't want to deal with forwarded
buffers, then we could add a flag for that. For now read_steam.c is the
only user.
Discussion: https://postgr.es/m/CA%2BhUKGK_%3D4CVmMHvsHjOVrK6t4F%3DLBpFzsrr3R%2BaJYN8kcTfWg%40mail.gmail.com
---
src/include/storage/bufmgr.h | 1 -
src/backend/storage/buffer/bufmgr.c | 128 ++++++++++++++++++++--------
2 files changed, 91 insertions(+), 38 deletions(-)
diff --git a/src/include/storage/bufmgr.h b/src/include/storage/bufmgr.h
index 3225bce605f..dcdef3e8164 100644
--- a/src/include/storage/bufmgr.h
+++ b/src/include/storage/bufmgr.h
@@ -131,7 +131,6 @@ struct ReadBuffersOperation
BlockNumber blocknum;
int flags;
int16 nblocks;
- int16 io_buffers_len;
};
typedef struct ReadBuffersOperation ReadBuffersOperation;
diff --git a/src/backend/storage/buffer/bufmgr.c b/src/backend/storage/buffer/bufmgr.c
index c3cdcd8d4ad..7c0d6b85c0d 100644
--- a/src/backend/storage/buffer/bufmgr.c
+++ b/src/backend/storage/buffer/bufmgr.c
@@ -1251,10 +1251,10 @@ StartReadBuffersImpl(ReadBuffersOperation *operation,
Buffer *buffers,
BlockNumber blockNum,
int *nblocks,
- int flags)
+ int flags,
+ bool allow_forwarding)
{
int actual_nblocks = *nblocks;
- int io_buffers_len = 0;
int maxcombine = 0;
Assert(*nblocks > 0);
@@ -1264,30 +1264,80 @@ StartReadBuffersImpl(ReadBuffersOperation *operation,
{
bool found;
- buffers[i] = PinBufferForBlock(operation->rel,
- operation->smgr,
- operation->persistence,
- operation->forknum,
- blockNum + i,
- operation->strategy,
- &found);
+ if (allow_forwarding && buffers[i] != InvalidBuffer)
+ {
+ BufferDesc *bufHdr;
+
+ /*
+ * This is a buffer that was pinned by an earlier call to
+ * StartReadBuffers(), but couldn't be handled in one operation at
+ * that time. The operation was split, and the caller has passed
+ * an already pinned buffer back to us to handle the rest of the
+ * operation. It must continue at the expected block number.
+ */
+ Assert(BufferGetBlockNumber(buffers[i]) == blockNum + i);
+
+ /*
+ * It might be an already valid buffer (a hit) that followed the
+ * final contiguous block of an earlier I/O (a miss) marking the
+ * end of it, or a buffer that some other backend has since made
+ * valid by performing the I/O for us, in which case we can handle
+ * it as a hit now. It is safe to check for a BM_VALID flag with
+ * a relaxed load, because we got a fresh view of it while pinning
+ * it in the previous call.
+ *
+ * On the other hand if we don't see BM_VALID yet, it must be an
+ * I/O that was split by the previous call and we need to try to
+ * start a new I/O from this block. We're also racing against any
+ * other backend that might start the I/O or even manage to mark
+ * it BM_VALID after this check, but StartBufferIO() will handle
+ * those cases.
+ */
+ if (BufferIsLocal(buffers[i]))
+ bufHdr = GetLocalBufferDescriptor(-buffers[i] - 1);
+ else
+ bufHdr = GetBufferDescriptor(buffers[i] - 1);
+ found = pg_atomic_read_u32(&bufHdr->state) & BM_VALID;
+ }
+ else
+ {
+ buffers[i] = PinBufferForBlock(operation->rel,
+ operation->smgr,
+ operation->persistence,
+ operation->forknum,
+ blockNum + i,
+ operation->strategy,
+ &found);
+ }
if (found)
{
/*
- * Terminate the read as soon as we get a hit. It could be a
- * single buffer hit, or it could be a hit that follows a readable
- * range. We don't want to create more than one readable range,
- * so we stop here.
+ * We have a hit. If it's the first block in the requested range,
+ * we can return it immediately and report that WaitReadBuffers()
+ * does not need to be called. If the initial value of *nblocks
+ * was larger, the caller will have to call again for the rest.
*/
- actual_nblocks = i + 1;
+ if (i == 0)
+ {
+ *nblocks = 1;
+ return false;
+ }
+
+ /*
+ * Otherwise we already have an I/O to perform, but this block
+ * can't be included as it is already valid. Split the I/O here.
+ * There may or may not be more blocks requiring I/O after this
+ * one, we haven't checked, but it can't be contiguous with this
+ * hit in the way. We'll leave this buffer pinned, forwarding it
+ * to the next call, avoiding the need to unpin it here and re-pin
+ * it in the next call.
+ */
+ actual_nblocks = i;
break;
}
else
{
- /* Extend the readable range to cover this block. */
- io_buffers_len++;
-
/*
* Check how many blocks we can cover with the same IO. The smgr
* implementation might e.g. be limited due to a segment boundary.
@@ -1308,15 +1358,11 @@ StartReadBuffersImpl(ReadBuffersOperation *operation,
}
*nblocks = actual_nblocks;
- if (likely(io_buffers_len == 0))
- return false;
-
/* Populate information needed for I/O. */
operation->buffers = buffers;
operation->blocknum = blockNum;
operation->flags = flags;
operation->nblocks = actual_nblocks;
- operation->io_buffers_len = io_buffers_len;
if (flags & READ_BUFFERS_ISSUE_ADVICE)
{
@@ -1331,7 +1377,7 @@ StartReadBuffersImpl(ReadBuffersOperation *operation,
smgrprefetch(operation->smgr,
operation->forknum,
blockNum,
- operation->io_buffers_len);
+ actual_nblocks);
}
/* Indicate that WaitReadBuffers() should be called. */
@@ -1345,11 +1391,21 @@ StartReadBuffersImpl(ReadBuffersOperation *operation,
* actual number, which may be fewer than requested. Caller sets some of the
* members of operation; see struct definition.
*
+ * The initial contents of the elements of buffers up to *nblocks should
+ * either be InvalidBuffer or an already-pinned buffer that was left by an
+ * preceding call to StartReadBuffers() that had to be split. On return, some
+ * elements of buffers may hold pinned buffers beyond the number indicated by
+ * the updated value of *nblocks. Operations are split on boundaries known to
+ * smgr (eg md.c segment boundaries that require crossing into a different
+ * underlying file), or when already cached blocks are found in the buffer
+ * that prevent the formation of a contiguous read.
+ *
* If false is returned, no I/O is necessary. If true is returned, one I/O
* has been started, and WaitReadBuffers() must be called with the same
* operation object before the buffers are accessed. Along with the operation
* object, the caller-supplied array of buffers must remain valid until
- * WaitReadBuffers() is called.
+ * WaitReadBuffers() is called, and any forwarded buffers must also be
+ * preserved for a future call unless explicitly released.
*
* Currently the I/O is only started with optional operating system advice if
* requested by the caller with READ_BUFFERS_ISSUE_ADVICE, and the real I/O
@@ -1363,13 +1419,18 @@ StartReadBuffers(ReadBuffersOperation *operation,
int *nblocks,
int flags)
{
- return StartReadBuffersImpl(operation, buffers, blockNum, nblocks, flags);
+ return StartReadBuffersImpl(operation, buffers, blockNum, nblocks, flags,
+ true /* expect forwarded buffers */ );
}
/*
* Single block version of the StartReadBuffers(). This might save a few
* instructions when called from another translation unit, because it is
* specialized for nblocks == 1.
+ *
+ * This version does not support "forwarded" buffers: they cannot be created
+ * by reading only one block, and the current contents of *buffer is ignored
+ * on entry.
*/
bool
StartReadBuffer(ReadBuffersOperation *operation,
@@ -1380,7 +1441,8 @@ StartReadBuffer(ReadBuffersOperation *operation,
int nblocks = 1;
bool result;
- result = StartReadBuffersImpl(operation, buffer, blocknum, &nblocks, flags);
+ result = StartReadBuffersImpl(operation, buffer, blocknum, &nblocks, flags,
+ false /* single block, no forwarding */ );
Assert(nblocks == 1); /* single block can't be short */
return result;
@@ -1407,24 +1469,16 @@ WaitReadBuffers(ReadBuffersOperation *operation)
IOObject io_object;
char persistence;
- /*
- * Currently operations are only allowed to include a read of some range,
- * with an optional extra buffer that is already pinned at the end. So
- * nblocks can be at most one more than io_buffers_len.
- */
- Assert((operation->nblocks == operation->io_buffers_len) ||
- (operation->nblocks == operation->io_buffers_len + 1));
-
/* Find the range of the physical read we need to perform. */
- nblocks = operation->io_buffers_len;
- if (nblocks == 0)
- return; /* nothing to do */
-
+ nblocks = operation->nblocks;
buffers = &operation->buffers[0];
blocknum = operation->blocknum;
forknum = operation->forknum;
persistence = operation->persistence;
+ Assert(nblocks > 0);
+ Assert(nblocks <= MAX_IO_COMBINE_LIMIT);
+
if (persistence == RELPERSISTENCE_TEMP)
{
io_context = IOCONTEXT_NORMAL;
--
2.48.1.76.g4e746b1a31.dirty
[text/x-diff] v2.10-0011-bufmgr-Use-AIO-in-StartReadBuffers.patch (27.0K, ../../zp7obmaqdd6b5u2vhpw27hovpjirtqvjakkbkaz4vju7tohyul@4kfobe7ipgxm/12-v2.10-0011-bufmgr-Use-AIO-in-StartReadBuffers.patch)
download | inline diff:
From 9ec0ef4c8bde3f13259a909e037526f2564d46b3 Mon Sep 17 00:00:00 2001
From: Andres Freund <andres@anarazel.de>
Date: Tue, 18 Mar 2025 14:40:06 -0400
Subject: [PATCH v2.10 11/28] bufmgr: Use AIO in StartReadBuffers()
This finally introduces the first actual AIO user. StartReadBuffers() now uses
the AIO routines to issue IO. This converts a lot of callers to use the AIO
infrastructure.
As the implementation of StartReadBuffers() is also used by the functions for
reading individual blocks (StartReadBuffer() and through that
ReadBufferExtended()) this means all buffered read IO passes through the AIO
paths. However, as those are synchronous reads, actually performing the IO
asynchronously would be rarely beneficial. Instead such IOs are flagged to
always be executed synchronously. This way we don't have to duplicate a fair
bit of code.
When io_method=sync is used, the IO patterns generated after this change are
the same as before, i.e. actual reads are only issued in WaitReadBuffers() and
StartReadBuffers() may issue prefetch requests. This allows to bypass most of
the actual asynchronicity, which is important to make a change as big as this
less risky.
One thing worth calling out is that, if IO is actually executed
asynchronously, the precise meaning of what track_io_timing is measuring has
changed. Previously it tracked the time for each IO, but that does not make
sense when multiple IOs are executed concurrently. Now it only measures the
time actually spent waiting for IO.
While AIO is now actually used, the logic in read_stream.c will often prevent
using sufficiently many concurrent IOs. That will be addressed in the next
commit.
Reviewed-by: Noah Misch <noah@leadboat.com>
Reviewed-by: Nazir Bilal Yavuz <byavuz81@gmail.com>
Co-authored-by: Andres Freund <andres@anarazel.de>
Co-authored-by: Thomas Munro <thomas.munro@gmail.com>
Discussion: https://postgr.es/m/uvrtrknj4kdytuboidbhwclo4gxhswwcpgadptsjvjqcluzmah%40brqs62irg4dt
Discussion: https://postgr.es/m/20210223100344.llw5an2aklengrmn@alap3.anarazel.de
Discussion: https://postgr.es/m/stj36ea6yyhoxtqkhpieia2z4krnam7qyetc57rfezgk4zgapf@gcnactj4z56m
---
src/include/storage/bufmgr.h | 6 +
src/backend/storage/buffer/bufmgr.c | 613 +++++++++++++++++++++-------
2 files changed, 476 insertions(+), 143 deletions(-)
diff --git a/src/include/storage/bufmgr.h b/src/include/storage/bufmgr.h
index dcdef3e8164..f82bd1814dc 100644
--- a/src/include/storage/bufmgr.h
+++ b/src/include/storage/bufmgr.h
@@ -112,6 +112,9 @@ typedef struct BufferManagerRelation
#define READ_BUFFERS_ZERO_ON_ERROR (1 << 0)
/* Call smgrprefetch() if I/O necessary. */
#define READ_BUFFERS_ISSUE_ADVICE (1 << 1)
+/* IO will immediately be waited for */
+#define READ_BUFFERS_SYNCHRONOUSLY (1 << 2)
+
struct ReadBuffersOperation
{
@@ -131,6 +134,9 @@ struct ReadBuffersOperation
BlockNumber blocknum;
int flags;
int16 nblocks;
+ int16 nblocks_done;
+ PgAioWaitRef io_wref;
+ PgAioReturn io_return;
};
typedef struct ReadBuffersOperation ReadBuffersOperation;
diff --git a/src/backend/storage/buffer/bufmgr.c b/src/backend/storage/buffer/bufmgr.c
index 7c0d6b85c0d..f747650036f 100644
--- a/src/backend/storage/buffer/bufmgr.c
+++ b/src/backend/storage/buffer/bufmgr.c
@@ -528,6 +528,8 @@ static inline BufferDesc *BufferAlloc(SMgrRelation smgr,
BlockNumber blockNum,
BufferAccessStrategy strategy,
bool *foundPtr, IOContext io_context);
+static bool AsyncReadBuffers(ReadBuffersOperation *operation, int *nblocks);
+static void CheckReadBuffersOperation(ReadBuffersOperation *operation, bool is_complete);
static Buffer GetVictimBuffer(BufferAccessStrategy strategy, IOContext io_context);
static void FlushBuffer(BufferDesc *buf, SMgrRelation reln,
IOObject io_object, IOContext io_context);
@@ -1228,10 +1230,14 @@ ReadBuffer_common(Relation rel, SMgrRelation smgr, char smgr_persistence,
return buffer;
}
+ /*
+ * Signal that we are going to immediately wait. If we're immediately
+ * waiting, there is no benefit in actually executing the IO
+ * asynchronously, it would just add dispatch overhead.
+ */
+ flags = READ_BUFFERS_SYNCHRONOUSLY;
if (mode == RBM_ZERO_ON_ERROR)
- flags = READ_BUFFERS_ZERO_ON_ERROR;
- else
- flags = 0;
+ flags |= READ_BUFFERS_ZERO_ON_ERROR;
operation.smgr = smgr;
operation.rel = rel;
operation.persistence = persistence;
@@ -1256,9 +1262,11 @@ StartReadBuffersImpl(ReadBuffersOperation *operation,
{
int actual_nblocks = *nblocks;
int maxcombine = 0;
+ bool did_start_io;
Assert(*nblocks > 0);
Assert(*nblocks <= MAX_IO_COMBINE_LIMIT);
+ Assert(*nblocks == 1 || allow_forwarding);
for (int i = 0; i < actual_nblocks; ++i)
{
@@ -1297,6 +1305,7 @@ StartReadBuffersImpl(ReadBuffersOperation *operation,
bufHdr = GetLocalBufferDescriptor(-buffers[i] - 1);
else
bufHdr = GetBufferDescriptor(buffers[i] - 1);
+ Assert(pg_atomic_read_u32(&bufHdr->state) & BM_TAG_VALID);
found = pg_atomic_read_u32(&bufHdr->state) & BM_VALID;
}
else
@@ -1321,6 +1330,20 @@ StartReadBuffersImpl(ReadBuffersOperation *operation,
if (i == 0)
{
*nblocks = 1;
+
+#ifdef USE_ASSERT_CHECKING
+
+ /*
+ * Initialize enough of ReadBuffersOperation to make
+ * CheckReadBuffersOperation() work. Outside of assertions
+ * that's not necessary when no IO is issued.
+ */
+ operation->buffers = buffers;
+ operation->blocknum = blockNum;
+ operation->nblocks = 1;
+ operation->nblocks_done = 1;
+ CheckReadBuffersOperation(operation, true);
+#endif
return false;
}
@@ -1363,25 +1386,74 @@ StartReadBuffersImpl(ReadBuffersOperation *operation,
operation->blocknum = blockNum;
operation->flags = flags;
operation->nblocks = actual_nblocks;
+ operation->nblocks_done = 0;
+ pgaio_wref_clear(&operation->io_wref);
- if (flags & READ_BUFFERS_ISSUE_ADVICE)
+ /*
+ * When using AIO, start the IO in the background. If not, issue prefetch
+ * requests if desired by the caller.
+ *
+ * The reason we have a dedicated path for IOMETHOD_SYNC here is to
+ * de-risk the introduction of AIO somewhat. It's a large architectural
+ * change, with lots of chances for unanticipated performance effects.
+ *
+ * Use of IOMETHOD_SYNC already leads to not actually performing IO
+ * asynchronously, but without the check here we'd execute IO earlier than
+ * we used to. Eventually this IOMETHOD_SYNC specific path should go away.
+ */
+ if (io_method != IOMETHOD_SYNC)
{
/*
- * In theory we should only do this if PinBufferForBlock() had to
- * allocate new buffers above. That way, if two calls to
- * StartReadBuffers() were made for the same blocks before
- * WaitReadBuffers(), only the first would issue the advice. That'd be
- * a better simulation of true asynchronous I/O, which would only
- * start the I/O once, but isn't done here for simplicity.
+ * Try to start IO asynchronously. It's possible that no IO needs to
+ * be started, if another backend already performed the IO.
+ *
+ * Note that if an IO is started, it might not cover the entire
+ * requested range, e.g. because an intermediary block has been read
+ * in by another backend. In that case any "trailing" buffers we
+ * already pinned above will be "forwarded" by read_stream.c to the
+ * next call to StartReadBuffers().
+ *
+ * This is signalled to the caller by decrementing *nblocks *and*
+ * reducing operation->nblocks. The latter is done here, but not below
+ * WaitReadBuffers(), as in WaitReadBuffers() we can't "shorten" the
+ * overall read size anymore, we need to retry until done in its
+ * entirety or until failed.
*/
- smgrprefetch(operation->smgr,
- operation->forknum,
- blockNum,
- actual_nblocks);
+ did_start_io = AsyncReadBuffers(operation, nblocks);
+
+ operation->nblocks = *nblocks;
+ }
+ else
+ {
+ operation->flags |= READ_BUFFERS_SYNCHRONOUSLY;
+
+ if (flags & READ_BUFFERS_ISSUE_ADVICE)
+ {
+ /*
+ * In theory we should only do this if PinBufferForBlock() had to
+ * allocate new buffers above. That way, if two calls to
+ * StartReadBuffers() were made for the same blocks before
+ * WaitReadBuffers(), only the first would issue the advice.
+ * That'd be a better simulation of true asynchronous I/O, which
+ * would only start the I/O once, but isn't done here for
+ * simplicity.
+ */
+ smgrprefetch(operation->smgr,
+ operation->forknum,
+ blockNum,
+ actual_nblocks);
+ }
+
+ /*
+ * Indicate that WaitReadBuffers() should be called. WaitReadBuffers()
+ * will initiate the necessary IO.
+ */
+ did_start_io = true;
}
- /* Indicate that WaitReadBuffers() should be called. */
- return true;
+ CheckReadBuffersOperation(operation, !did_start_io);
+
+ return did_start_io;
}
/*
@@ -1448,8 +1520,35 @@ StartReadBuffer(ReadBuffersOperation *operation,
return result;
}
+/*
+ * Perform sanity checks on the ReadBuffersOperation.
+ */
+static void
+CheckReadBuffersOperation(ReadBuffersOperation *operation, bool is_complete)
+{
+#ifdef USE_ASSERT_CHECKING
+ Assert(operation->nblocks_done <= operation->nblocks);
+ Assert(!is_complete || operation->nblocks == operation->nblocks_done);
+
+ for (int i = 0; i < operation->nblocks; i++)
+ {
+ Buffer buffer = operation->buffers[i];
+ BufferDesc *buf_hdr = BufferIsLocal(buffer) ?
+ GetLocalBufferDescriptor(-buffer - 1) :
+ GetBufferDescriptor(buffer - 1);
+
+ Assert(BufferGetBlockNumber(buffer) == operation->blocknum + i);
+ Assert(pg_atomic_read_u32(&buf_hdr->state) & BM_TAG_VALID);
+
+ if (i < operation->nblocks_done)
+ Assert(pg_atomic_read_u32(&buf_hdr->state) & BM_VALID);
+ }
+#endif
+}
+
+/* helper for ReadBuffersCanStartIO(), to avoid repetition */
static inline bool
-WaitReadBuffersCanStartIO(Buffer buffer, bool nowait)
+ReadBuffersCanStartIOOnce(Buffer buffer, bool nowait)
{
if (BufferIsLocal(buffer))
return StartLocalBufferIO(GetLocalBufferDescriptor(-buffer - 1),
@@ -1458,31 +1557,242 @@ WaitReadBuffersCanStartIO(Buffer buffer, bool nowait)
return StartBufferIO(GetBufferDescriptor(buffer - 1), true, nowait);
}
-void
-WaitReadBuffers(ReadBuffersOperation *operation)
+/*
+ * Helper for AsyncReadBuffers that tries to initiate IO on the buffer.
+ */
+static inline bool
+ReadBuffersCanStartIO(Buffer buffer, bool nowait)
{
- Buffer *buffers;
- int nblocks;
- BlockNumber blocknum;
- ForkNumber forknum;
- IOContext io_context;
- IOObject io_object;
- char persistence;
-
- /* Find the range of the physical read we need to perform. */
- nblocks = operation->nblocks;
- buffers = &operation->buffers[0];
- blocknum = operation->blocknum;
- forknum = operation->forknum;
- persistence = operation->persistence;
+ /*
+ * If this backend currently has staged IO, we need to submit the pending
+ * IO before waiting for the right to issue IO, to avoid the potential for
+ * deadlocks (and, more commonly, unnecessary delays for other backends).
+ */
+ if (!nowait && pgaio_have_staged())
+ {
+ if (ReadBuffersCanStartIOOnce(buffer, true))
+ return true;
+
+ /*
+ * Unfortunately a false returned StartBufferIO() doesn't allow to
+ * distinguish between the buffer already being valid and IO already
+ * being in progress. Since IO already being in progress is quite
+ * rare, this approach seems fine.
+ */
+ pgaio_submit_staged();
+ }
+
+ return ReadBuffersCanStartIOOnce(buffer, nowait);
+}
+
+/*
+ * Helper for WaitReadBuffers() that processes the results of a readv
+ * operation, raising an error if necessary.
+ */
+static void
+ProcessReadBuffersResult(ReadBuffersOperation *operation)
+{
+ PgAioReturn *aio_ret = &operation->io_return;
+ int nblocks = 0;
+
+ Assert(pgaio_wref_valid(&operation->io_wref));
+ Assert(aio_ret->result.status != ARS_UNKNOWN);
+
+ /*
+ * SMGR reports the number of blocks successfully read as the result of
+ * the IO operation. Thus we can simply add that to ->nblocks_done.
+ */
+
+ if (likely(aio_ret->result.status == ARS_OK))
+ nblocks = aio_ret->result.result;
+ else if (aio_ret->result.status == ARS_PARTIAL)
+ {
+ /*
+ * We'll retry, so we just emit a debug message to the server log (or
+ * not even that in prod scenarios).
+ */
+ pgaio_result_report(aio_ret->result, &aio_ret->target_data, DEBUG1);
+ nblocks = aio_ret->result.result;
+
+ elog(DEBUG3, "partial read, will retry");
+
+ }
+ else if (aio_ret->result.status == ARS_ERROR)
+ {
+ pgaio_result_report(aio_ret->result, &aio_ret->target_data, ERROR);
+ nblocks = 0; /* silence compiler */
+ }
Assert(nblocks > 0);
Assert(nblocks <= MAX_IO_COMBINE_LIMIT);
+ operation->nblocks_done += nblocks;
+
+ Assert(operation->nblocks_done <= operation->nblocks);
+}
+
+void
+WaitReadBuffers(ReadBuffersOperation *operation)
+{
+ PgAioReturn *aio_ret = &operation->io_return;
+ IOContext io_context;
+ IOObject io_object;
+
+ if (operation->persistence == RELPERSISTENCE_TEMP)
+ {
+ io_context = IOCONTEXT_NORMAL;
+ io_object = IOOBJECT_TEMP_RELATION;
+ }
+ else
+ {
+ io_context = IOContextForStrategy(operation->strategy);
+ io_object = IOOBJECT_RELATION;
+ }
+
+ /*
+ * If we get here without an IO operation having been issued, the
+ * io_method == IOMETHOD_SYNC path must have been used. Otherwise the
+ * caller should not have called WaitReadBuffers().
+ *
+ * In the case of IOMETHOD_SYNC, we start - as we used to before the
+ * introducing of AIO - the IO in WaitReadBuffers(). This is done as part
+ * of the retry logic below, no extra code is required.
+ *
+ * This path is expected to eventually go away.
+ */
+ if (!pgaio_wref_valid(&operation->io_wref) && io_method != IOMETHOD_SYNC)
+ elog(ERROR, "waiting for read operation that didn't read");
+
+ /*
+ * To handle partial reads, and IOMETHOD_SYNC, we re-issue IO until we're
+ * done. We may need multiple retries, not just because we could get
+ * multiple short reads, but also because some of the remaining to-be-read
+ * buffers may have been read in by other backends, limiting the IO size.
+ */
+ while (true)
+ {
+ int nblocks;
+
+ CheckReadBuffersOperation(operation, false);
+
+ /*
+ * If there is an IO associated with the operation, we may need to
+ * wait for it. It's possible for there to be no IO if
+ */
+ if (pgaio_wref_valid(&operation->io_wref))
+ {
+ /*
+ * Track the time spent waiting for the IO to complete. As
+ * tracking a wait even if we don't actually need to wait
+ *
+ * a) is not cheap, due to the timestamping overhead
+ *
+ * b) reports some time as waiting, even if we never waited
+ *
+ * we first check if we already know the IO is complete.
+ */
+ if (aio_ret->result.status == ARS_UNKNOWN &&
+ !pgaio_wref_check_done(&operation->io_wref))
+ {
+ instr_time io_start = pgstat_prepare_io_time(track_io_timing);
+
+ pgaio_wref_wait(&operation->io_wref);
+
+ /*
+ * The IO operation itself was already counted earlier, in
+ * AsyncReadBuffers(), this just accounts for the wait time.
+ */
+ pgstat_count_io_op_time(io_object, io_context, IOOP_READ,
+ io_start, 0, 0);
+ }
+ else
+ {
+ Assert(pgaio_wref_check_done(&operation->io_wref));
+ }
+
+ /*
+ * We now are sure the IO completed. Check the results. This
+ * includes reporting on errors if there were any.
+ */
+ ProcessReadBuffersResult(operation);
+ }
+
+ /*
+ * Most of the the the one IO we started will read in everything. But
+ * we need to deal with short reads and buffers not needing IO
+ * anymore.
+ */
+ if (operation->nblocks_done == operation->nblocks)
+ break;
+
+ CHECK_FOR_INTERRUPTS();
+
+ /*
+ * This may only complete the IO partially, either because some
+ * buffers were already valid, or because of a short read.
+ *
+ * NB: In contrast to after the AsyncReadBuffers() call in
+ * StartReadBuffers(), we do *not* reduce
+ * ReadBuffersOperation->nblocks here, callers expect the full
+ * operation to be completed at this point (as more operations may
+ * have been queued).
+ */
+ AsyncReadBuffers(operation, &nblocks);
+ }
+
+ CheckReadBuffersOperation(operation, true);
+
+ /* NB: READ_DONE tracepoint was already executed in completion callback */
+}
+
+/*
+ * Initiate IO for the ReadBuffersOperation
+ *
+ * This function only starts a single IO at a time. The size of the IO may be
+ * limited to below the to-be-read blocks if one of the buffers has
+ * concurrently been read in. If the first to-be-read buffer is already valid,
+ * no IO will be issued.
+ *
+ * To support retries after short reads, the first operation->nblocks_done is
+ * buffers are skipped.
+ *
+ * On return *nblocks_progres is updated to reflect the number of buffers
+ * affected by the call. If the first buffer is valid, *nblocks_progress is
+ * set to 1 and operation->nblocks_done is incremented.
+ *
+ * Returns true if IO was initiated, false if no IO was necessary.
+ */
+static bool
+AsyncReadBuffers(ReadBuffersOperation *operation, int *nblocks_progress)
+{
+ Buffer *buffers = &operation->buffers[0];
+ int flags = operation->flags;
+ BlockNumber blocknum = operation->blocknum;
+ ForkNumber forknum = operation->forknum;
+ char persistence = operation->persistence;
+ int16 nblocks_done = operation->nblocks_done;
+ Buffer *io_buffers = &operation->buffers[nblocks_done];
+ int io_buffers_len = 0;
+ PgAioHandle *ioh;
+ uint32 ioh_flags = 0;
+ void *io_pages[MAX_IO_COMBINE_LIMIT];
+ IOContext io_context;
+ IOObject io_object;
+ bool did_start_io;
+
+ /*
+ * When this IO is executed synchronously, either because the caller will
+ * immediately block waiting for the IO or because IOMETHOD_SYNC is used,
+ * the AIO subsystem needs to know.
+ */
+ if (flags & READ_BUFFERS_SYNCHRONOUSLY)
+ ioh_flags |= PGAIO_HF_SYNCHRONOUS;
+
if (persistence == RELPERSISTENCE_TEMP)
{
io_context = IOCONTEXT_NORMAL;
io_object = IOOBJECT_TEMP_RELATION;
+ ioh_flags |= PGAIO_HF_REFERENCES_LOCAL;
}
else
{
@@ -1490,58 +1800,90 @@ WaitReadBuffers(ReadBuffersOperation *operation)
io_object = IOOBJECT_RELATION;
}
- for (int i = 0; i < nblocks; ++i)
+ /*
+ * Get IO before ReadBuffersCanStartIO, as pgaio_io_acquire() might block,
+ * which we don't want after setting IO_IN_PROGRESS.
+ *
+ * If IO needs to be submitted before we can get a handle, submit already
+ * staged IO first, so that other backends don't need to wait. There
+ * wouldn't be a deadlock risk, as pgaio_io_acquire() just needs to wait
+ * for already submitted IO, which doesn't require additional locks, but
+ * it could still cause undesirable waits.
+ *
+ * A secondary benefit is that this would allows us to measure the time in
+ * pgaio_io_acquire() without causing undue timer overhead in the common,
+ * non-blocking, case. However, currently the pgstats infrastructure
+ * doesn't really allow that, as it a) asserts that an operation can't
+ * have time without operations b) doesn't have an API to report
+ * "accumulated" time.
+ */
+ ioh = pgaio_io_acquire_nb(CurrentResourceOwner, &operation->io_return);
+ if (unlikely(!ioh))
+ {
+ pgaio_submit_staged();
+
+ ioh = pgaio_io_acquire(CurrentResourceOwner, &operation->io_return);
+ }
+
+ /*
+ * Check if we can start IO on the buffer.
+ *
+ * If an I/O is already in progress in another backend, we want to wait
+ * for the outcome: either done, or something went wrong and we will
+ * retry.
+ */
+ if (!ReadBuffersCanStartIO(buffers[nblocks_done], false))
+ {
+ /*
+ * Someone else has already completed this block, we're done.
+ *
+ * When IO is necessary, ->nblocks_done is updated in
+ * ProcessReadBuffersResult(), but that is not called if no IO is
+ * necessary. Thus update here.
+ */
+ operation->nblocks_done += 1;
+ *nblocks_progress = 1;
+
+ pgaio_io_release(ioh);
+ pgaio_wref_clear(&operation->io_wref);
+ did_start_io = false;
+
+ /*
+ * Report and track this as a 'hit' for this backend, even though it
+ * must have started out as a miss in PinBufferForBlock().
+ *
+ * Some of the accesses would otherwise never be counted (e.g.
+ * pgBufferUsage) or counted as a miss (e.g.
+ * pgstat_count_buffer_hit(), as we always call
+ * pgstat_count_buffer_read()).
+ */
+ TRACE_POSTGRESQL_BUFFER_READ_DONE(forknum, blocknum + operation->nblocks_done,
+ operation->smgr->smgr_rlocator.locator.spcOid,
+ operation->smgr->smgr_rlocator.locator.dbOid,
+ operation->smgr->smgr_rlocator.locator.relNumber,
+ operation->smgr->smgr_rlocator.backend,
+ true);
+
+ if (persistence == RELPERSISTENCE_TEMP)
+ pgBufferUsage.local_blks_hit += 1;
+ else
+ pgBufferUsage.shared_blks_hit += 1;
+
+ if (operation->rel)
+ pgstat_count_buffer_hit(operation->rel);
+
+ pgstat_count_io_op(io_object, io_context, IOOP_HIT, 1, 0);
+
+ if (VacuumCostActive)
+ VacuumCostBalance += VacuumCostPageHit;
+ }
+ else
{
- int io_buffers_len;
- Buffer io_buffers[MAX_IO_COMBINE_LIMIT];
- void *io_pages[MAX_IO_COMBINE_LIMIT];
instr_time io_start;
- BlockNumber io_first_block;
-
- /*
- * Skip this block if someone else has already completed it. If an
- * I/O is already in progress in another backend, this will wait for
- * the outcome: either done, or something went wrong and we will
- * retry.
- */
- if (!WaitReadBuffersCanStartIO(buffers[i], false))
- {
- /*
- * Report and track this as a 'hit' for this backend, even though
- * it must have started out as a miss in PinBufferForBlock().
- *
- * Some of the accesses would otherwise never be counted (e.g.
- * pgBufferUsage) or counted as a miss (e.g.
- * pgstat_count_buffer_hit(), as we always call
- * pgstat_count_buffer_read()).
- */
- TRACE_POSTGRESQL_BUFFER_READ_DONE(forknum, blocknum + i,
- operation->smgr->smgr_rlocator.locator.spcOid,
- operation->smgr->smgr_rlocator.locator.dbOid,
- operation->smgr->smgr_rlocator.locator.relNumber,
- operation->smgr->smgr_rlocator.backend,
- true);
-
- if (persistence == RELPERSISTENCE_TEMP)
- pgBufferUsage.local_blks_hit += 1;
- else
- pgBufferUsage.shared_blks_hit += 1;
-
- if (operation->rel)
- pgstat_count_buffer_hit(operation->rel);
-
- pgstat_count_io_op(io_object, io_context, IOOP_HIT, 1, 0);
-
- if (VacuumCostActive)
- VacuumCostBalance += VacuumCostPageHit;
-
- continue;
- }
/* We found a buffer that we need to read in. */
- io_buffers[0] = buffers[i];
- io_pages[0] = BufferGetBlock(buffers[i]);
- io_first_block = blocknum + i;
+ Assert(io_buffers[0] == buffers[nblocks_done]);
+ io_pages[0] = BufferGetBlock(buffers[nblocks_done]);
io_buffers_len = 1;
/*
@@ -1549,85 +1891,70 @@ WaitReadBuffers(ReadBuffersOperation *operation)
* buffers at the same time? In this case we don't wait if we see an
* I/O already in progress. We already hold BM_IO_IN_PROGRESS for the
* head block, so we should get on with that I/O as soon as possible.
- * We'll come back to this block again, above.
+ *
+ * We'll come back to this block in the next call to
+ * StartReadBuffers() -> AsyncReadBuffers().
*/
- while ((i + 1) < nblocks &&
- WaitReadBuffersCanStartIO(buffers[i + 1], true))
+ for (int i = nblocks_done + 1; i < operation->nblocks; i++)
{
+ if (!ReadBuffersCanStartIO(buffers[i], true))
+ break;
/* Must be consecutive block numbers. */
- Assert(BufferGetBlockNumber(buffers[i + 1]) ==
- BufferGetBlockNumber(buffers[i]) + 1);
+ Assert(BufferGetBlockNumber(buffers[i - 1]) ==
+ BufferGetBlockNumber(buffers[i]) - 1);
+ Assert(io_buffers[io_buffers_len] == buffers[i]);
- io_buffers[io_buffers_len] = buffers[++i];
io_pages[io_buffers_len++] = BufferGetBlock(buffers[i]);
}
+ /* get a reference to wait for in WaitReadBuffers() */
+ pgaio_io_get_wref(ioh, &operation->io_wref);
+
+ /* provide the list of buffers to the completion callbacks */
+ pgaio_io_set_handle_data_32(ioh, (uint32 *) io_buffers, io_buffers_len);
+
+ pgaio_io_register_callbacks(ioh,
+ persistence == RELPERSISTENCE_TEMP ?
+ PGAIO_HCB_LOCAL_BUFFER_READV :
+ PGAIO_HCB_SHARED_BUFFER_READV,
+ flags);
+
+ pgaio_io_set_flag(ioh, ioh_flags);
+
+ /* ---
+ * Even though we're trying to issue IO asynchronously, track the time
+ * in smgrstartreadv():
+ * - if io_method == IOMETHOD_SYNC, we will always perform the IO
+ * immediately
+ * - the io method might not support the IO (e.g. worker IO for a temp
+ * table)
+ * ---
+ */
io_start = pgstat_prepare_io_time(track_io_timing);
- smgrreadv(operation->smgr, forknum, io_first_block, io_pages, io_buffers_len);
- pgstat_count_io_op_time(io_object, io_context, IOOP_READ, io_start,
- 1, io_buffers_len * BLCKSZ);
-
- /* Verify each block we read, and terminate the I/O. */
- for (int j = 0; j < io_buffers_len; ++j)
- {
- BufferDesc *bufHdr;
- Block bufBlock;
-
- if (persistence == RELPERSISTENCE_TEMP)
- {
- bufHdr = GetLocalBufferDescriptor(-io_buffers[j] - 1);
- bufBlock = LocalBufHdrGetBlock(bufHdr);
- }
- else
- {
- bufHdr = GetBufferDescriptor(io_buffers[j] - 1);
- bufBlock = BufHdrGetBlock(bufHdr);
- }
-
- /* check for garbage data */
- if (!PageIsVerifiedExtended((Page) bufBlock, io_first_block + j,
- PIV_LOG_WARNING | PIV_REPORT_STAT))
- {
- if ((operation->flags & READ_BUFFERS_ZERO_ON_ERROR) || zero_damaged_pages)
- {
- ereport(WARNING,
- (errcode(ERRCODE_DATA_CORRUPTED),
- errmsg("invalid page in block %u of relation %s; zeroing out page",
- io_first_block + j,
- relpath(operation->smgr->smgr_rlocator, forknum).str)));
- memset(bufBlock, 0, BLCKSZ);
- }
- else
- ereport(ERROR,
- (errcode(ERRCODE_DATA_CORRUPTED),
- errmsg("invalid page in block %u of relation %s",
- io_first_block + j,
- relpath(operation->smgr->smgr_rlocator, forknum).str)));
- }
-
- /* Set BM_VALID, terminate IO, and wake up any waiters */
- if (persistence == RELPERSISTENCE_TEMP)
- TerminateLocalBufferIO(bufHdr, false, BM_VALID, true);
- else
- TerminateBufferIO(bufHdr, false, BM_VALID, true, true);
-
- /* Report I/Os as completing individually. */
- TRACE_POSTGRESQL_BUFFER_READ_DONE(forknum, io_first_block + j,
- operation->smgr->smgr_rlocator.locator.spcOid,
- operation->smgr->smgr_rlocator.locator.dbOid,
- operation->smgr->smgr_rlocator.locator.relNumber,
- operation->smgr->smgr_rlocator.backend,
- false);
- }
+ smgrstartreadv(ioh, operation->smgr, forknum,
+ blocknum + nblocks_done,
+ io_pages, io_buffers_len);
+ pgstat_count_io_op_time(io_object, io_context, IOOP_READ,
+ io_start, 1, *nblocks_progress * BLCKSZ);
if (persistence == RELPERSISTENCE_TEMP)
pgBufferUsage.local_blks_read += io_buffers_len;
else
pgBufferUsage.shared_blks_read += io_buffers_len;
+ /*
+ * Track vacuum cost when issuing IO, not after waiting for it.
+ * Otherwise we could end up issuing a lot of IO in a short timespan,
+ * despite a low cost limit.
+ */
if (VacuumCostActive)
VacuumCostBalance += VacuumCostPageMiss * io_buffers_len;
+
+ *nblocks_progress = io_buffers_len;
+ did_start_io = true;
}
+
+ return did_start_io;
}
/*
--
2.48.1.76.g4e746b1a31.dirty
[text/x-diff] v2.10-0012-aio-Basic-read_stream-adjustments-for-real-AIO.patch (3.8K, ../../zp7obmaqdd6b5u2vhpw27hovpjirtqvjakkbkaz4vju7tohyul@4kfobe7ipgxm/13-v2.10-0012-aio-Basic-read_stream-adjustments-for-real-AIO.patch)
download | inline diff:
From ea72565d754cc7d0e7b89dedd7d97608c73172dd Mon Sep 17 00:00:00 2001
From: Andres Freund <andres@anarazel.de>
Date: Tue, 18 Mar 2025 14:40:06 -0400
Subject: [PATCH v2.10 12/28] aio: Basic read_stream adjustments for real AIO
Adapt the read stream logic for real AIO:
- If AIO is enabled, we shouldn't issue advice, but if it isn't, we should
continue issuing advice
- AIO benefits from reading ahead with direct IO
- While in read_stream_look_ahead(), we can use AIO batch submission mode for
increased efficiency
There is one comment talking about max_ios logic with "real asynchronous I/O"
that I am not sure about, so I left it alone for now.
There are further improvements we should consider, e.g. waiting to issue IOs
until we can issue multiple IOs at once. But that's left for a future change,
since it would involve additional heuristics.
---
src/backend/storage/aio/read_stream.c | 29 ++++++++++++++++++++++-----
1 file changed, 24 insertions(+), 5 deletions(-)
diff --git a/src/backend/storage/aio/read_stream.c b/src/backend/storage/aio/read_stream.c
index cdf4b5a86a2..d7b395b86a3 100644
--- a/src/backend/storage/aio/read_stream.c
+++ b/src/backend/storage/aio/read_stream.c
@@ -72,6 +72,7 @@
#include "postgres.h"
#include "miscadmin.h"
+#include "storage/aio.h"
#include "storage/fd.h"
#include "storage/smgr.h"
#include "storage/read_stream.h"
@@ -99,6 +100,7 @@ struct ReadStream
int16 pinned_buffers;
int16 distance;
int16 initialized_buffers;
+ bool sync_mode;
bool advice_enabled;
bool temporary;
@@ -416,6 +418,13 @@ read_stream_start_pending_read(ReadStream *stream)
static void
read_stream_look_ahead(ReadStream *stream)
{
+ /*
+ * Allow amortizing the cost of submitting IO over multiple IOs. This
+ * requires that we don't do any operations that could lead to a deadlock
+ * with staged-but-unsubmitted IO.
+ */
+ pgaio_enter_batchmode();
+
while (stream->ios_in_progress < stream->max_ios &&
stream->pinned_buffers + stream->pending_read_nblocks < stream->distance)
{
@@ -463,6 +472,7 @@ read_stream_look_ahead(ReadStream *stream)
{
/* We've hit the buffer or I/O limit. Rewind and stop here. */
read_stream_unget_block(stream, blocknum);
+ pgaio_exit_batchmode();
return;
}
}
@@ -497,6 +507,8 @@ read_stream_look_ahead(ReadStream *stream)
* time.
*/
Assert(stream->pinned_buffers > 0 || stream->distance == 0);
+
+ pgaio_exit_batchmode();
}
/*
@@ -628,15 +640,19 @@ read_stream_begin_impl(int flags,
stream->per_buffer_data = (void *)
MAXALIGN(&stream->ios[Max(1, max_ios)]);
+ stream->sync_mode = io_method == IOMETHOD_SYNC;
+
#ifdef USE_PREFETCH
/*
- * This system supports prefetching advice. We can use it as long as
- * direct I/O isn't enabled, the caller hasn't promised sequential access
- * (overriding our detection heuristics), and max_ios hasn't been set to
- * zero.
+ * This system supports prefetching advice.
+ *
+ * Issue advice only if AIO is not used, direct I/O isn't enabled, the
+ * caller hasn't promised sequential access (overriding our detection
+ * heuristics), and max_ios hasn't been set to zero.
*/
- if ((io_direct_flags & IO_DIRECT_DATA) == 0 &&
+ if (stream->sync_mode &&
+ (io_direct_flags & IO_DIRECT_DATA) == 0 &&
(flags & READ_STREAM_SEQUENTIAL) == 0 &&
max_ios > 0)
stream->advice_enabled = true;
@@ -646,6 +662,9 @@ read_stream_begin_impl(int flags,
* For now, max_ios = 0 is interpreted as max_ios = 1 with advice disabled
* above. If we had real asynchronous I/O we might need a slightly
* different definition.
+ *
+ * FIXME: Not sure what different definition we would need? I guess we
+ * could add the READ_BUFFERS_SYNCHRONOUSLY flag automatically?
*/
if (max_ios == 0)
max_ios = 1;
--
2.48.1.76.g4e746b1a31.dirty
[text/x-diff] v2.10-0013-docs-Reframe-track_io_timing-related-docs-as-w.patch (5.3K, ../../zp7obmaqdd6b5u2vhpw27hovpjirtqvjakkbkaz4vju7tohyul@4kfobe7ipgxm/14-v2.10-0013-docs-Reframe-track_io_timing-related-docs-as-w.patch)
download | inline diff:
From 99638b6cff10894856202d5b6a82698f4f2be9b9 Mon Sep 17 00:00:00 2001
From: Andres Freund <andres@anarazel.de>
Date: Tue, 18 Mar 2025 14:40:06 -0400
Subject: [PATCH v2.10 13/28] docs: Reframe track_io_timing related docs as
wait time
With AIO it does not make sense anymore to track the time for each individual
IO, as multiple IOs can be in-flight at the same time. Instead we now track
the time spent *waiting* for IOs.
This should be reflected in the docs. While, so far, we only do a subset of
reads, and no other operations, via AIO, describing the GUC and view columns
as measuring IO waits is accurate for synchronous and asynchronous IO.
Discussion: https://postgr.es/m/5dzyoduxlvfg55oqtjyjehez5uoq6hnwgzor4kkybkfdgkj7ag@rbi4gsmzaczk
---
doc/src/sgml/config.sgml | 4 ++--
doc/src/sgml/monitoring.sgml | 12 ++++++------
2 files changed, 8 insertions(+), 8 deletions(-)
diff --git a/doc/src/sgml/config.sgml b/doc/src/sgml/config.sgml
index 1156c389c2c..a1811713bcd 100644
--- a/doc/src/sgml/config.sgml
+++ b/doc/src/sgml/config.sgml
@@ -8549,7 +8549,7 @@ COPY postgres_log FROM '/full/path/to/logfile.csv' WITH csv;
</term>
<listitem>
<para>
- Enables timing of database I/O calls. This parameter is off by
+ Enables timing of database I/O waits. This parameter is off by
default, as it will repeatedly query the operating system for
the current time, which may cause significant overhead on some
platforms. You can use the <xref linkend="pgtesttiming"/> tool to
@@ -8583,7 +8583,7 @@ COPY postgres_log FROM '/full/path/to/logfile.csv' WITH csv;
</term>
<listitem>
<para>
- Enables timing of WAL I/O calls. This parameter is off by default,
+ Enables timing of WAL I/O waits. This parameter is off by default,
as it will repeatedly query the operating system for the current time,
which may cause significant overhead on some platforms.
You can use the <application>pg_test_timing</application> tool to
diff --git a/doc/src/sgml/monitoring.sgml b/doc/src/sgml/monitoring.sgml
index aaa6586d3a4..7598340072f 100644
--- a/doc/src/sgml/monitoring.sgml
+++ b/doc/src/sgml/monitoring.sgml
@@ -2733,7 +2733,7 @@ description | Waiting for a newly initialized WAL file to reach durable storage
<structfield>read_time</structfield> <type>double precision</type>
</para>
<para>
- Time spent in read operations in milliseconds (if
+ Time spent waiting for read operations in milliseconds (if
<xref linkend="guc-track-io-timing"/> is enabled and
<varname>object</varname> is not <literal>wal</literal>,
or if <xref linkend="guc-track-wal-io-timing"/> is enabled
@@ -2771,7 +2771,7 @@ description | Waiting for a newly initialized WAL file to reach durable storage
<structfield>write_time</structfield> <type>double precision</type>
</para>
<para>
- Time spent in write operations in milliseconds (if
+ Time spent waiting for write operations in milliseconds (if
<xref linkend="guc-track-io-timing"/> is enabled and
<varname>object</varname> is not <literal>wal</literal>,
or if <xref linkend="guc-track-wal-io-timing"/> is enabled
@@ -2799,7 +2799,7 @@ description | Waiting for a newly initialized WAL file to reach durable storage
<structfield>writeback_time</structfield> <type>double precision</type>
</para>
<para>
- Time spent in writeback operations in milliseconds (if
+ Time spent waiting for writeback operations in milliseconds (if
<xref linkend="guc-track-io-timing"/> is enabled, otherwise zero). This
includes the time spent queueing write-out requests and, potentially,
the time spent to write out the dirty data.
@@ -2835,7 +2835,7 @@ description | Waiting for a newly initialized WAL file to reach durable storage
<structfield>extend_time</structfield> <type>double precision</type>
</para>
<para>
- Time spent in extend operations in milliseconds. (if
+ Time spent waiting for extend operations in milliseconds. (if
<xref linkend="guc-track-io-timing"/> is enabled and
<varname>object</varname> is not <literal>wal</literal>,
or if <xref linkend="guc-track-wal-io-timing"/> is enabled
@@ -2909,7 +2909,7 @@ description | Waiting for a newly initialized WAL file to reach durable storage
<structfield>fsync_time</structfield> <type>double precision</type>
</para>
<para>
- Time spent in fsync operations in milliseconds (if
+ Time spent waiting for fsync operations in milliseconds (if
<xref linkend="guc-track-io-timing"/> is enabled and
<varname>object</varname> is not <literal>wal</literal>,
or if <xref linkend="guc-track-wal-io-timing"/> is enabled
@@ -2996,7 +2996,7 @@ description | Waiting for a newly initialized WAL file to reach durable storage
<note>
<para>
- Columns tracking I/O time will only be non-zero when
+ Columns tracking I/O wait time will only be non-zero when
<xref linkend="guc-track-io-timing"/> is enabled. The user should be
careful when referencing these columns in combination with their
corresponding I/O operations in case <varname>track_io_timing</varname>
--
2.48.1.76.g4e746b1a31.dirty
[text/x-diff] v2.10-0014-Enable-IO-concurrency-on-all-systems.patch (10.9K, ../../zp7obmaqdd6b5u2vhpw27hovpjirtqvjakkbkaz4vju7tohyul@4kfobe7ipgxm/15-v2.10-0014-Enable-IO-concurrency-on-all-systems.patch)
download | inline diff:
From 90286d23b91150b49290b3e58f1e2f5f669938f8 Mon Sep 17 00:00:00 2001
From: Andres Freund <andres@anarazel.de>
Date: Tue, 18 Mar 2025 14:40:06 -0400
Subject: [PATCH v2.10 14/28] Enable IO concurrency on all systems
Previously effective_io_concurrency and maintenance_io_concurrency could
not be set above 0 on machines without fadvise support. AIO enables
IO concurrency without such support.
Currently only subsystems using the read stream API will take advantage
of this. Other users of maintenance_io_concurrency (like recovery
prefetching) which leverage OS advice directly will not benefit from
this change. In those cases, maintenance_io_concurrency will have no
effect on I/O behavior.
Author: Melanie Plageman <melanieplageman@gmail.com>
Discussion: https://postgr.es/m/CAAKRu_atGgZePo=_g6T3cNtfMf0QxpvoUh5OUqa_cnPdhLd=gw@mail.gmail.com
---
src/include/storage/bufmgr.h | 6 ----
src/include/utils/guc_hooks.h | 4 ---
src/backend/access/common/reloptions.c | 8 -----
src/backend/commands/variable.c | 30 -------------------
src/backend/utils/misc/guc_tables.c | 4 +--
src/backend/utils/misc/postgresql.conf.sample | 4 +--
src/bin/initdb/initdb.c | 5 ----
doc/src/sgml/config.sgml | 15 +++++-----
doc/src/sgml/ref/alter_tablespace.sgml | 2 +-
doc/src/sgml/ref/create_tablespace.sgml | 2 +-
10 files changed, 13 insertions(+), 67 deletions(-)
diff --git a/src/include/storage/bufmgr.h b/src/include/storage/bufmgr.h
index f82bd1814dc..7f709fd7ea4 100644
--- a/src/include/storage/bufmgr.h
+++ b/src/include/storage/bufmgr.h
@@ -156,14 +156,8 @@ extern PGDLLIMPORT int bgwriter_lru_maxpages;
extern PGDLLIMPORT double bgwriter_lru_multiplier;
extern PGDLLIMPORT bool track_io_timing;
-/* only applicable when prefetching is available */
-#ifdef USE_PREFETCH
#define DEFAULT_EFFECTIVE_IO_CONCURRENCY 16
#define DEFAULT_MAINTENANCE_IO_CONCURRENCY 16
-#else
-#define DEFAULT_EFFECTIVE_IO_CONCURRENCY 0
-#define DEFAULT_MAINTENANCE_IO_CONCURRENCY 0
-#endif
extern PGDLLIMPORT int effective_io_concurrency;
extern PGDLLIMPORT int maintenance_io_concurrency;
diff --git a/src/include/utils/guc_hooks.h b/src/include/utils/guc_hooks.h
index a3eba8fbe21..74de7a6abdf 100644
--- a/src/include/utils/guc_hooks.h
+++ b/src/include/utils/guc_hooks.h
@@ -61,8 +61,6 @@ extern bool check_default_text_search_config(char **newval, void **extra, GucSou
extern void assign_default_text_search_config(const char *newval, void *extra);
extern bool check_default_with_oids(bool *newval, void **extra,
GucSource source);
-extern bool check_effective_io_concurrency(int *newval, void **extra,
- GucSource source);
extern bool check_huge_page_size(int *newval, void **extra, GucSource source);
extern void assign_io_method(int newval, void *extra);
extern bool check_io_max_concurrency(int *newval, void **extra, GucSource source);
@@ -83,8 +81,6 @@ extern bool check_log_stats(bool *newval, void **extra, GucSource source);
extern bool check_log_timezone(char **newval, void **extra, GucSource source);
extern void assign_log_timezone(const char *newval, void *extra);
extern const char *show_log_timezone(void);
-extern bool check_maintenance_io_concurrency(int *newval, void **extra,
- GucSource source);
extern void assign_maintenance_io_concurrency(int newval, void *extra);
extern bool check_max_slot_wal_keep_size(int *newval, void **extra,
GucSource source);
diff --git a/src/backend/access/common/reloptions.c b/src/backend/access/common/reloptions.c
index 59fb53e7707..3e43ad064c3 100644
--- a/src/backend/access/common/reloptions.c
+++ b/src/backend/access/common/reloptions.c
@@ -361,11 +361,7 @@ static relopt_int intRelOpts[] =
RELOPT_KIND_TABLESPACE,
ShareUpdateExclusiveLock
},
-#ifdef USE_PREFETCH
-1, 0, MAX_IO_CONCURRENCY
-#else
- 0, 0, 0
-#endif
},
{
{
@@ -374,11 +370,7 @@ static relopt_int intRelOpts[] =
RELOPT_KIND_TABLESPACE,
ShareUpdateExclusiveLock
},
-#ifdef USE_PREFETCH
-1, 0, MAX_IO_CONCURRENCY
-#else
- 0, 0, 0
-#endif
},
{
{
diff --git a/src/backend/commands/variable.c b/src/backend/commands/variable.c
index 4ad6e236d69..7188afd2860 100644
--- a/src/backend/commands/variable.c
+++ b/src/backend/commands/variable.c
@@ -1145,7 +1145,6 @@ check_cluster_name(char **newval, void **extra, GucSource source)
void
assign_maintenance_io_concurrency(int newval, void *extra)
{
-#ifdef USE_PREFETCH
/*
* Reconfigure recovery prefetching, because a setting it depends on
* changed.
@@ -1153,7 +1152,6 @@ assign_maintenance_io_concurrency(int newval, void *extra)
maintenance_io_concurrency = newval;
if (AmStartupProcess())
XLogPrefetchReconfigure();
-#endif
}
@@ -1231,34 +1229,6 @@ check_default_with_oids(bool *newval, void **extra, GucSource source)
return true;
}
-bool
-check_effective_io_concurrency(int *newval, void **extra, GucSource source)
-{
-#ifndef USE_PREFETCH
- if (*newval != 0)
- {
- GUC_check_errdetail("\"%s\" must be set to 0 on platforms that lack support for issuing read-ahead advice.",
- "effective_io_concurrency");
- return false;
- }
-#endif /* USE_PREFETCH */
- return true;
-}
-
-bool
-check_maintenance_io_concurrency(int *newval, void **extra, GucSource source)
-{
-#ifndef USE_PREFETCH
- if (*newval != 0)
- {
- GUC_check_errdetail("\"%s\" must be set to 0 on platforms that lack support for issuing read-ahead advice.",
- "maintenance_io_concurrency");
- return false;
- }
-#endif /* USE_PREFETCH */
- return true;
-}
-
bool
check_ssl(bool *newval, void **extra, GucSource source)
{
diff --git a/src/backend/utils/misc/guc_tables.c b/src/backend/utils/misc/guc_tables.c
index 60a40ed445a..ea0d9dfb841 100644
--- a/src/backend/utils/misc/guc_tables.c
+++ b/src/backend/utils/misc/guc_tables.c
@@ -3234,7 +3234,7 @@ struct config_int ConfigureNamesInt[] =
&effective_io_concurrency,
DEFAULT_EFFECTIVE_IO_CONCURRENCY,
0, MAX_IO_CONCURRENCY,
- check_effective_io_concurrency, NULL, NULL
+ NULL, NULL, NULL
},
{
@@ -3248,7 +3248,7 @@ struct config_int ConfigureNamesInt[] =
&maintenance_io_concurrency,
DEFAULT_MAINTENANCE_IO_CONCURRENCY,
0, MAX_IO_CONCURRENCY,
- check_maintenance_io_concurrency, assign_maintenance_io_concurrency,
+ NULL, assign_maintenance_io_concurrency,
NULL
},
diff --git a/src/backend/utils/misc/postgresql.conf.sample b/src/backend/utils/misc/postgresql.conf.sample
index 458f0815a12..0748601ec20 100644
--- a/src/backend/utils/misc/postgresql.conf.sample
+++ b/src/backend/utils/misc/postgresql.conf.sample
@@ -198,8 +198,8 @@
# - I/O -
#backend_flush_after = 0 # measured in pages, 0 disables
-#effective_io_concurrency = 16 # 1-1000; 0 disables prefetching
-#maintenance_io_concurrency = 16 # 1-1000; 0 disables prefetching
+#effective_io_concurrency = 16 # 1-1000; 0 disables issuing multiple simultaneous IO requests
+#maintenance_io_concurrency = 16 # 1-1000; same as effective_io_concurrency
#io_combine_limit = 128kB # usually 1-32 blocks (depends on OS)
#io_method = worker # worker, io_uring, sync
diff --git a/src/bin/initdb/initdb.c b/src/bin/initdb/initdb.c
index 21a0fe3ecd9..b9d75799ee0 100644
--- a/src/bin/initdb/initdb.c
+++ b/src/bin/initdb/initdb.c
@@ -1401,11 +1401,6 @@ setup_config(void)
repltok, true);
#endif
-#ifndef USE_PREFETCH
- conflines = replace_guc_value(conflines, "effective_io_concurrency",
- "0", true);
-#endif
-
#ifdef WIN32
conflines = replace_guc_value(conflines, "update_process_title",
"off", true);
diff --git a/doc/src/sgml/config.sgml b/doc/src/sgml/config.sgml
index a1811713bcd..865979aa8ea 100644
--- a/doc/src/sgml/config.sgml
+++ b/doc/src/sgml/config.sgml
@@ -2581,8 +2581,7 @@ include_dir 'conf.d'
session attempts to initiate in parallel. The allowed range is
<literal>1</literal> to <literal>1000</literal>, or
<literal>0</literal> to disable issuance of asynchronous I/O requests.
- The default is <literal>16</literal> on supported systems, otherwise
- <literal>0</literal>.
+ The default is <literal>16</literal>.
</para>
<para>
@@ -2593,8 +2592,8 @@ include_dir 'conf.d'
</para>
<para>
- On systems without prefetch advice support, attempting to configure
- any value other than <literal>0</literal> will error out.
+ On systems with prefetch advice support,
+ <varname>effective_io_concurrency</varname> also controls the prefetch distance.
</para>
<para>
@@ -2617,10 +2616,10 @@ include_dir 'conf.d'
for maintenance work that is done on behalf of many client sessions.
</para>
<para>
- The default is <literal>16</literal> on supported systems, otherwise
- <literal>0</literal>. This value can be overridden for tables in a
- particular tablespace by setting the tablespace parameter of the same
- name (see <xref linkend="sql-altertablespace"/>).
+ The default is <literal>16</literal>. This value can be overridden
+ for tables in a particular tablespace by setting the tablespace
+ parameter of the same name (see <xref
+ linkend="sql-altertablespace"/>).
</para>
</listitem>
</varlistentry>
diff --git a/doc/src/sgml/ref/alter_tablespace.sgml b/doc/src/sgml/ref/alter_tablespace.sgml
index 6ec863400d1..d0e08089ddb 100644
--- a/doc/src/sgml/ref/alter_tablespace.sgml
+++ b/doc/src/sgml/ref/alter_tablespace.sgml
@@ -88,7 +88,7 @@ ALTER TABLESPACE <replaceable>name</replaceable> RESET ( <replaceable class="par
and <varname>maintenance_io_concurrency</varname>.
Setting these values for a particular tablespace will override the
planner's usual estimate of the cost of reading pages from tables in
- that tablespace, and the executor's prefetching behavior, as established
+ that tablespace, and how many concurrent I/Os are issued, as established
by the configuration parameters of the
same name (see <xref linkend="guc-seq-page-cost"/>,
<xref linkend="guc-random-page-cost"/>,
diff --git a/doc/src/sgml/ref/create_tablespace.sgml b/doc/src/sgml/ref/create_tablespace.sgml
index 9d5ab025261..b77e774c53f 100644
--- a/doc/src/sgml/ref/create_tablespace.sgml
+++ b/doc/src/sgml/ref/create_tablespace.sgml
@@ -110,7 +110,7 @@ CREATE TABLESPACE <replaceable class="parameter">tablespace_name</replaceable>
and <varname>maintenance_io_concurrency</varname>.
Setting these values for a particular tablespace will override the
planner's usual estimate of the cost of reading pages from tables in
- that tablespace, and the executor's prefetching behavior, as established
+ that tablespace, and how many concurrent I/Os are issued, as established
by the configuration parameters of the
same name (see <xref linkend="guc-seq-page-cost"/>,
<xref linkend="guc-random-page-cost"/>,
--
2.48.1.76.g4e746b1a31.dirty
[text/x-diff] v2.10-0015-aio-Add-pg_aios-view.patch (9.8K, ../../zp7obmaqdd6b5u2vhpw27hovpjirtqvjakkbkaz4vju7tohyul@4kfobe7ipgxm/16-v2.10-0015-aio-Add-pg_aios-view.patch)
download | inline diff:
From e4d814dd6896d1490160edfd2d18016afb0d4bd4 Mon Sep 17 00:00:00 2001
From: Andres Freund <andres@anarazel.de>
Date: Tue, 18 Mar 2025 14:40:06 -0400
Subject: [PATCH v2.10 15/28] aio: Add pg_aios view
TODO:
- decide on name
- add docs
FIXME:
- catversion bump
Author:
Reviewed-by:
Discussion: https://postgr.es/m/
Backpatch:
---
src/include/catalog/pg_proc.dat | 10 ++
src/backend/catalog/system_views.sql | 3 +
src/backend/storage/aio/Makefile | 1 +
src/backend/storage/aio/aio_funcs.c | 219 +++++++++++++++++++++++++++
src/backend/storage/aio/meson.build | 1 +
src/test/regress/expected/rules.out | 16 ++
6 files changed, 250 insertions(+)
create mode 100644 src/backend/storage/aio/aio_funcs.c
diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat
index 890822eaf79..017971011f3 100644
--- a/src/include/catalog/pg_proc.dat
+++ b/src/include/catalog/pg_proc.dat
@@ -12479,4 +12479,14 @@
proargtypes => 'int4',
prosrc => 'gist_stratnum_common' },
+# AIO related functions
+{ oid => '9200', descr => 'information about in-progress asynchronous IOs',
+ proname => 'pg_get_aios', prorows => '100', proretset => 't',
+ provolatile => 'v', proparallel => 'r', prorettype => 'record', proargtypes => '',
+ proallargtypes => '{int4,int4,int8,text,text,int8,int8,text,int2,int4,text,text,bool,bool,bool}',
+ proargmodes => '{o,o,o,o,o,o,o,o,o,o,o,o,o,o,o}',
+ proargnames => '{pid,io_id,io_generation,state,operation,offset,length,target,handle_data_len,raw_result,result,target_desc,f_sync,f_localmem,f_buffered}',
+ prosrc => 'pg_get_aios' },
+
+
]
diff --git a/src/backend/catalog/system_views.sql b/src/backend/catalog/system_views.sql
index a4d2cfdcaf5..73902a763d1 100644
--- a/src/backend/catalog/system_views.sql
+++ b/src/backend/catalog/system_views.sql
@@ -1390,3 +1390,6 @@ CREATE VIEW pg_stat_subscription_stats AS
CREATE VIEW pg_wait_events AS
SELECT * FROM pg_get_wait_events();
+
+CREATE VIEW pg_aios AS
+ SELECT * FROM pg_get_aios();
diff --git a/src/backend/storage/aio/Makefile b/src/backend/storage/aio/Makefile
index c06c50771e0..3f2469cc399 100644
--- a/src/backend/storage/aio/Makefile
+++ b/src/backend/storage/aio/Makefile
@@ -11,6 +11,7 @@ include $(top_builddir)/src/Makefile.global
OBJS = \
aio.o \
aio_callback.o \
+ aio_funcs.o \
aio_init.o \
aio_io.o \
aio_target.o \
diff --git a/src/backend/storage/aio/aio_funcs.c b/src/backend/storage/aio/aio_funcs.c
new file mode 100644
index 00000000000..58883fd61d8
--- /dev/null
+++ b/src/backend/storage/aio/aio_funcs.c
@@ -0,0 +1,219 @@
+/*-------------------------------------------------------------------------
+ *
+ * aio_funcs.c
+ * AIO - SQL interface for AIO
+ *
+ *
+ * Portions Copyright (c) 1996-2025, PostgreSQL Global Development Group
+ * Portions Copyright (c) 1994, Regents of the University of California
+ *
+ * IDENTIFICATION
+ * src/backend/storage/aio/aio_funcs.c
+ *
+ *-------------------------------------------------------------------------
+ */
+
+#include "postgres.h"
+
+#include "storage/aio.h"
+#include "storage/aio_internal.h"
+#include "utils/builtins.h"
+#include "funcapi.h"
+#include "storage/proc.h"
+
+
+/*
+ * Byte length of an iovec.
+ */
+static size_t
+iov_byte_length(const struct iovec *iov, int cnt)
+{
+ size_t len = 0;
+
+ for (int i = 0; i < cnt; i++)
+ {
+ len += iov[i].iov_len;
+ }
+
+ return len;
+}
+
+Datum
+pg_get_aios(PG_FUNCTION_ARGS)
+{
+ ReturnSetInfo *rsinfo = (ReturnSetInfo *) fcinfo->resultinfo;
+
+ InitMaterializedSRF(fcinfo, 0);
+
+#define PG_GET_AIOS_COLS 15
+
+ for (uint64 i = 0; i < pgaio_ctl->io_handle_count; i++)
+ {
+ PgAioHandle *live_ioh = &pgaio_ctl->io_handles[i];
+ uint32 ioh_id = pgaio_io_get_id(live_ioh);
+ Datum values[PG_GET_AIOS_COLS] = {0};
+ bool nulls[PG_GET_AIOS_COLS] = {0};
+ ProcNumber owner;
+ PGPROC *owner_proc;
+ int32 owner_pid;
+ PgAioHandleState start_state;
+ uint64 start_generation;
+ PgAioHandle ioh_copy;
+ struct iovec iov_copy[PG_IOV_MAX];
+
+retry:
+
+ /*
+ * There is no lock that could prevent the state of the IO to advance
+ * concurrently - and we don't want to introduce one, as that would
+ * introduce atomics into a very common path. Instead we
+ *
+ * 1) Determine the state + generation of the IO.
+ *
+ * 2) Copy the IO to local memory.
+ *
+ * 3) Check if state or generation of the IO changed. If the state
+ * changed, retry, if the generation changed don't display the IO.
+ */
+
+ /* 1) from above */
+ start_generation = live_ioh->generation;
+ pg_read_barrier();
+ start_state = live_ioh->state;
+
+ if (start_state == PGAIO_HS_IDLE)
+ continue;
+
+ /* 2) from above */
+ memcpy(&ioh_copy, live_ioh, sizeof(PgAioHandle));
+
+ /*
+ * Safe to copy even if no iovec is used - we always reserve the
+ * required space.
+ */
+ memcpy(&iov_copy, &pgaio_ctl->iovecs[ioh_copy.iovec_off],
+ PG_IOV_MAX * sizeof(struct iovec));
+
+ /*
+ * Copy information about owner before 3) below, if the process exited
+ * it'd have to wait for the IO to finish first, which we would detect
+ * in 3).
+ */
+ owner = ioh_copy.owner_procno;
+ owner_proc = GetPGProcByNumber(owner);
+ owner_pid = owner_proc->pid;
+
+ /* 3) from above */
+ pg_read_barrier();
+
+ /*
+ * The IO completed and a new one was started with the same ID. Don't
+ * display it - it really started after this function was called.
+ * There be a risk of a livelock if we just retried endlessly, if IOs
+ * complete very quickly.
+ */
+ if (live_ioh->generation != start_generation)
+ continue;
+
+ /*
+ * The IOs state changed while we were "rendering" it. Just start from
+ * scratch. There's no risk of a livelock here, as an IO has a limited
+ * sets of states it can be in, and state changes go only in a single
+ * direction.
+ */
+ if (live_ioh->state != start_state)
+ goto retry;
+
+ /*
+ * Now that we have copied the IO into local memory and checked that
+ * it's still in the same state, we are not allowed to access "live"
+ * memory anymore. To make it slightly easier to catch such cases, set
+ * the "live" pointers to NULL.
+ */
+ live_ioh = NULL;
+ owner_proc = NULL;
+
+
+ /* column: owning pid */
+ if (owner_pid != 0)
+ values[0] = Int32GetDatum(owner_pid);
+ else
+ nulls[0] = false;
+
+ /* column: IO's id */
+ values[1] = ioh_id;
+
+ /* column: IO's generation */
+ values[2] = Int64GetDatum(start_generation);
+
+ /* column: IO's state */
+ values[3] = CStringGetTextDatum(pgaio_io_get_state_name(&ioh_copy));
+
+ /*
+ * If the IO is in PGAIO_HS_HANDED_OUT state, none of it's fields are
+ * valid yet (or are in the process of being set). Therefore we don't
+ * want to display any other columns.
+ */
+ if (start_state == PGAIO_HS_HANDED_OUT)
+ {
+ memset(nulls + 4, 1, (lengthof(nulls) - 4) * sizeof(bool));
+ goto display;
+ }
+
+ /* column: IO's operation */
+ values[4] = CStringGetTextDatum(pgaio_io_get_op_name(&ioh_copy));
+
+ /* columns: details about the IO's operation */
+ switch (ioh_copy.op)
+ {
+ case PGAIO_OP_INVALID:
+ nulls[5] = true;
+ nulls[6] = true;
+ break;
+ case PGAIO_OP_READV:
+ values[5] = Int64GetDatum(ioh_copy.op_data.read.offset);
+ values[6] =
+ Int64GetDatum(iov_byte_length(iov_copy, ioh_copy.op_data.read.iov_length));
+ break;
+ case PGAIO_OP_WRITEV:
+ values[5] = Int64GetDatum(ioh_copy.op_data.write.offset);
+ values[6] =
+ Int64GetDatum(iov_byte_length(iov_copy, ioh_copy.op_data.write.iov_length));
+ break;
+ }
+
+ /* column: IO's target */
+ values[7] = CStringGetTextDatum(pgaio_io_get_target_name(&ioh_copy));
+
+ /* column: length of IO's data array */
+ values[8] = Int16GetDatum(ioh_copy.handle_data_len);
+
+ /* column: raw result (i.e. some form of syscall return value) */
+ if (start_state == PGAIO_HS_COMPLETED_IO
+ || start_state == PGAIO_HS_COMPLETED_SHARED)
+ values[9] = Int32GetDatum(ioh_copy.result);
+ else
+ nulls[9] = true;
+
+ /*
+ * column: result in the higher level representation (unknown if not
+ * finished)
+ */
+ values[10] =
+ CStringGetTextDatum(pgaio_result_status_string(ioh_copy.distilled_result.status));
+
+ /* column: target description */
+ values[11] = CStringGetTextDatum(pgaio_io_get_target_description(&ioh_copy));
+
+ /* columns: one for each flag */
+ values[12] = BoolGetDatum(ioh_copy.flags & PGAIO_HF_SYNCHRONOUS);
+ values[13] = BoolGetDatum(ioh_copy.flags & PGAIO_HF_REFERENCES_LOCAL);
+ values[14] = BoolGetDatum(ioh_copy.flags & PGAIO_HF_BUFFERED);
+
+display:
+
+ tuplestore_putvalues(rsinfo->setResult, rsinfo->setDesc, values, nulls);
+ }
+
+ return (Datum) 0;
+}
diff --git a/src/backend/storage/aio/meson.build b/src/backend/storage/aio/meson.build
index 2f0f03d8071..da6df2d3654 100644
--- a/src/backend/storage/aio/meson.build
+++ b/src/backend/storage/aio/meson.build
@@ -3,6 +3,7 @@
backend_sources += files(
'aio.c',
'aio_callback.c',
+ 'aio_funcs.c',
'aio_init.c',
'aio_io.c',
'aio_target.c',
diff --git a/src/test/regress/expected/rules.out b/src/test/regress/expected/rules.out
index 62f69ac20b2..6570d0cddce 100644
--- a/src/test/regress/expected/rules.out
+++ b/src/test/regress/expected/rules.out
@@ -1286,6 +1286,22 @@ drop table cchild;
SELECT viewname, definition FROM pg_views
WHERE schemaname = 'pg_catalog'
ORDER BY viewname;
+pg_aios| SELECT pid,
+ io_id,
+ io_generation,
+ state,
+ operation,
+ "offset",
+ length,
+ target,
+ handle_data_len,
+ raw_result,
+ result,
+ target_desc,
+ f_sync,
+ f_localmem,
+ f_buffered
+ FROM pg_get_aios() pg_get_aios(pid, io_id, io_generation, state, operation, "offset", length, target, handle_data_len, raw_result, result, target_desc, f_sync, f_localmem, f_buffered);
pg_available_extension_versions| SELECT e.name,
e.version,
(x.extname IS NOT NULL) AS installed,
--
2.48.1.76.g4e746b1a31.dirty
[text/x-diff] v2.10-0016-aio-Experimental-heuristics-to-increase-batchi.patch (1.6K, ../../zp7obmaqdd6b5u2vhpw27hovpjirtqvjakkbkaz4vju7tohyul@4kfobe7ipgxm/17-v2.10-0016-aio-Experimental-heuristics-to-increase-batchi.patch)
download | inline diff:
From 953b5f5045607c44f5b0b720edf190c82a32d57e Mon Sep 17 00:00:00 2001
From: Andres Freund <andres@anarazel.de>
Date: Tue, 18 Mar 2025 14:40:06 -0400
Subject: [PATCH v2.10 16/28] aio: Experimental heuristics to increase batching
in read_stream.c
Author:
Reviewed-by:
Discussion: https://postgr.es/m/
Backpatch:
---
src/backend/storage/aio/read_stream.c | 24 ++++++++++++++++++++++++
1 file changed, 24 insertions(+)
diff --git a/src/backend/storage/aio/read_stream.c b/src/backend/storage/aio/read_stream.c
index d7b395b86a3..887be2b3961 100644
--- a/src/backend/storage/aio/read_stream.c
+++ b/src/backend/storage/aio/read_stream.c
@@ -418,6 +418,30 @@ read_stream_start_pending_read(ReadStream *stream)
static void
read_stream_look_ahead(ReadStream *stream)
{
+ /*
+ * Batch-submitting multiple IOs is more efficient than doing so
+ * one-by-one. If we just ramp up to the max, we'll only be allowed to
+ * submit one io_combine_limit sized IO. Defer submitting IO in that case.
+ *
+ * FIXME: This needs better heuristics.
+ */
+#if 1
+ if (!stream->sync_mode && stream->distance > (io_combine_limit * 8))
+ {
+ if (stream->pinned_buffers + stream->pending_read_nblocks > ((stream->distance * 3) / 4))
+ {
+#if 0
+ ereport(LOG,
+ errmsg("reduce reduce reduce: pinned: %d, pending: %d, distance: %d",
+ stream->pinned_buffers,
+ stream->pending_read_nblocks,
+ stream->distance));
+#endif
+ return;
+ }
+ }
+#endif
+
/*
* Allow amortizing the cost of submitting IO over multiple IOs. This
* requires that we don't do any operations that could lead to a deadlock
--
2.48.1.76.g4e746b1a31.dirty
[text/x-diff] v2.10-0017-aio-Add-test_aio-module.patch (42.4K, ../../zp7obmaqdd6b5u2vhpw27hovpjirtqvjakkbkaz4vju7tohyul@4kfobe7ipgxm/18-v2.10-0017-aio-Add-test_aio-module.patch)
download | inline diff:
From 1e191dd86704a156a7b066fbbe8b6ed993b7d703 Mon Sep 17 00:00:00 2001
From: Andres Freund <andres@anarazel.de>
Date: Tue, 18 Mar 2025 14:40:06 -0400
Subject: [PATCH v2.10 17/28] aio: Add test_aio module
---
src/include/storage/buf_internals.h | 6 +
src/backend/storage/buffer/bufmgr.c | 8 +-
src/test/modules/Makefile | 1 +
src/test/modules/meson.build | 1 +
src/test/modules/test_aio/.gitignore | 2 +
src/test/modules/test_aio/Makefile | 27 +
src/test/modules/test_aio/meson.build | 37 ++
src/test/modules/test_aio/t/001_aio.pl | 674 ++++++++++++++++++++
src/test/modules/test_aio/test_aio--1.0.sql | 97 +++
src/test/modules/test_aio/test_aio.c | 616 ++++++++++++++++++
src/test/modules/test_aio/test_aio.control | 3 +
11 files changed, 1466 insertions(+), 6 deletions(-)
create mode 100644 src/test/modules/test_aio/.gitignore
create mode 100644 src/test/modules/test_aio/Makefile
create mode 100644 src/test/modules/test_aio/meson.build
create mode 100644 src/test/modules/test_aio/t/001_aio.pl
create mode 100644 src/test/modules/test_aio/test_aio--1.0.sql
create mode 100644 src/test/modules/test_aio/test_aio.c
create mode 100644 src/test/modules/test_aio/test_aio.control
diff --git a/src/include/storage/buf_internals.h b/src/include/storage/buf_internals.h
index 8a52ace37cd..4b1f52a9fc8 100644
--- a/src/include/storage/buf_internals.h
+++ b/src/include/storage/buf_internals.h
@@ -434,6 +434,12 @@ extern void IssuePendingWritebacks(WritebackContext *wb_context, IOContext io_co
extern void ScheduleBufferTagForWriteback(WritebackContext *wb_context,
IOContext io_context, BufferTag *tag);
+/* solely to make it easier to write tests */
+extern bool StartBufferIO(BufferDesc *buf, bool forInput, bool nowait);
+extern void TerminateBufferIO(BufferDesc *buf, bool clear_dirty, uint32 set_flag_bits,
+ bool forget_owner, bool syncio);
+
+
/* freelist.c */
extern IOContext IOContextForStrategy(BufferAccessStrategy strategy);
extern BufferDesc *StrategyGetBuffer(BufferAccessStrategy strategy,
diff --git a/src/backend/storage/buffer/bufmgr.c b/src/backend/storage/buffer/bufmgr.c
index f747650036f..7259b0b3bb6 100644
--- a/src/backend/storage/buffer/bufmgr.c
+++ b/src/backend/storage/buffer/bufmgr.c
@@ -515,10 +515,6 @@ static uint32 WaitBufHdrUnlocked(BufferDesc *buf);
static int SyncOneBuffer(int buf_id, bool skip_recently_used,
WritebackContext *wb_context);
static void WaitIO(BufferDesc *buf);
-static bool StartBufferIO(BufferDesc *buf, bool forInput, bool nowait);
-static void TerminateBufferIO(BufferDesc *buf, bool clear_dirty,
- uint32 set_flag_bits, bool forget_owner,
- bool syncio);
static void AbortBufferIO(Buffer buffer);
static void shared_buffer_write_error_callback(void *arg);
static void local_buffer_write_error_callback(void *arg);
@@ -5926,7 +5922,7 @@ WaitIO(BufferDesc *buf)
* find out if they can perform the I/O as part of a larger operation, without
* waiting for the answer or distinguishing the reasons why not.
*/
-static bool
+bool
StartBufferIO(BufferDesc *buf, bool forInput, bool nowait)
{
uint32 buf_state;
@@ -5983,7 +5979,7 @@ StartBufferIO(BufferDesc *buf, bool forInput, bool nowait)
* resource owner. (forget_owner=false is used when the resource owner itself
* is being released)
*/
-static void
+void
TerminateBufferIO(BufferDesc *buf, bool clear_dirty, uint32 set_flag_bits,
bool forget_owner, bool syncio)
{
diff --git a/src/test/modules/Makefile b/src/test/modules/Makefile
index 4e4be3fa511..aa1d27bbed3 100644
--- a/src/test/modules/Makefile
+++ b/src/test/modules/Makefile
@@ -14,6 +14,7 @@ SUBDIRS = \
oauth_validator \
plsample \
spgist_name_ops \
+ test_aio \
test_bloomfilter \
test_copy_callbacks \
test_custom_rmgrs \
diff --git a/src/test/modules/meson.build b/src/test/modules/meson.build
index 2b057451473..4fda1acfb32 100644
--- a/src/test/modules/meson.build
+++ b/src/test/modules/meson.build
@@ -1,5 +1,6 @@
# Copyright (c) 2022-2025, PostgreSQL Global Development Group
+subdir('test_aio')
subdir('brin')
subdir('commit_ts')
subdir('delay_execution')
diff --git a/src/test/modules/test_aio/.gitignore b/src/test/modules/test_aio/.gitignore
new file mode 100644
index 00000000000..716e17f5a2a
--- /dev/null
+++ b/src/test/modules/test_aio/.gitignore
@@ -0,0 +1,2 @@
+# Generated subdirectories
+/tmp_check/
diff --git a/src/test/modules/test_aio/Makefile b/src/test/modules/test_aio/Makefile
new file mode 100644
index 00000000000..87d5315ba00
--- /dev/null
+++ b/src/test/modules/test_aio/Makefile
@@ -0,0 +1,27 @@
+# src/test/modules/delay_execution/Makefile
+
+PGFILEDESC = "test_aio - test code for AIO"
+
+MODULE_big = test_aio
+OBJS = \
+ $(WIN32RES) \
+ test_aio.o
+
+EXTENSION = test_aio
+DATA = test_aio--1.0.sql
+
+TAP_TESTS = 1
+
+export enable_injection_points
+export with_liburing
+
+ifdef USE_PGXS
+PG_CONFIG = pg_config
+PGXS := $(shell $(PG_CONFIG) --pgxs)
+include $(PGXS)
+else
+subdir = src/test/modules/test_aio
+top_builddir = ../../../..
+include $(top_builddir)/src/Makefile.global
+include $(top_srcdir)/contrib/contrib-global.mk
+endif
diff --git a/src/test/modules/test_aio/meson.build b/src/test/modules/test_aio/meson.build
new file mode 100644
index 00000000000..ac846b2b6f3
--- /dev/null
+++ b/src/test/modules/test_aio/meson.build
@@ -0,0 +1,37 @@
+# Copyright (c) 2022-2024, PostgreSQL Global Development Group
+
+test_aio_sources = files(
+ 'test_aio.c',
+)
+
+if host_system == 'windows'
+ test_aio_sources += rc_lib_gen.process(win32ver_rc, extra_args: [
+ '--NAME', 'test_aio',
+ '--FILEDESC', 'test_aio - test code for AIO',])
+endif
+
+test_aio = shared_module('test_aio',
+ test_aio_sources,
+ kwargs: pg_test_mod_args,
+)
+test_install_libs += test_aio
+
+test_install_data += files(
+ 'test_aio.control',
+ 'test_aio--1.0.sql',
+)
+
+tests += {
+ 'name': 'test_aio',
+ 'sd': meson.current_source_dir(),
+ 'bd': meson.current_build_dir(),
+ 'tap': {
+ 'env': {
+ 'enable_injection_points': get_option('injection_points') ? 'yes' : 'no',
+ 'with_liburing': liburing.found() ? 'yes' : 'no',
+ },
+ 'tests': [
+ 't/001_aio.pl',
+ ],
+ },
+}
diff --git a/src/test/modules/test_aio/t/001_aio.pl b/src/test/modules/test_aio/t/001_aio.pl
new file mode 100644
index 00000000000..ecd1f3c2f17
--- /dev/null
+++ b/src/test/modules/test_aio/t/001_aio.pl
@@ -0,0 +1,674 @@
+# Copyright (c) 2025, PostgreSQL Global Development Group
+
+use strict;
+use warnings FATAL => 'all';
+
+use PostgreSQL::Test::Cluster;
+use PostgreSQL::Test::Utils;
+use Test::More;
+
+
+###
+# Test io_method=worker
+###
+my $node_worker = create_node('worker');
+$node_worker->start();
+
+test_generic('worker', $node_worker);
+SKIP:
+{
+ skip 'Injection points not supported by this build', 1
+ unless $ENV{enable_injection_points} eq 'yes';
+ test_inject_worker('worker', $node_worker);
+}
+
+$node_worker->stop();
+
+
+###
+# Test io_method=io_uring
+###
+
+if ($ENV{with_liburing} eq 'yes')
+{
+ my $node_uring = create_node('io_uring');
+ $node_uring->start();
+ test_generic('io_uring', $node_uring);
+ $node_uring->stop();
+}
+
+
+###
+# Test io_method=sync
+###
+
+my $node_sync = create_node('sync');
+
+# just to have one test not use the default auto-tuning
+
+$node_sync->append_conf(
+ 'postgresql.conf', qq(
+io_max_concurrency=4
+));
+
+$node_sync->start();
+test_generic('sync', $node_sync);
+$node_sync->stop();
+
+done_testing();
+
+
+###
+# Test Helpers
+###
+
+sub create_node
+{
+ my $io_method = shift;
+
+ my $node = PostgreSQL::Test::Cluster->new($io_method);
+
+ # Want to test initdb for each IO method, otherwise we could just reuse
+ # the cluster.
+ #
+ # Unfortunately Cluster::init() puts PG_TEST_INITDB_EXTRA_OPTS after the
+ # options specified by ->extra, if somebody puts -c io_method=xyz in
+ # PG_TEST_INITDB_EXTRA_OPTS it would break this test. Fix that up if we
+ # detect it.
+ local $ENV{PG_TEST_INITDB_EXTRA_OPTS} = $ENV{PG_TEST_INITDB_EXTRA_OPTS};
+ if (defined $ENV{PG_TEST_INITDB_EXTRA_OPTS} &&
+ $ENV{PG_TEST_INITDB_EXTRA_OPTS} =~ m/io_method=/)
+ {
+ $ENV{PG_TEST_INITDB_EXTRA_OPTS} .= " -c io_method=$io_method";
+ }
+
+ $node->init(extra => [ '-c', "io_method=$io_method" ]);
+
+ $node->append_conf(
+ 'postgresql.conf', qq(
+shared_preload_libraries=test_aio
+log_min_messages = 'DEBUG3'
+log_statement=all
+restart_after_crash=false
+));
+
+ ok(1, "$io_method: initdb");
+
+ return $node;
+}
+
+sub psql_like
+{
+ my $io_method = shift;
+ my $psql = shift;
+ my $name = shift;
+ my $sql = shift;
+ my $expected_stdout = shift;
+ my $expected_stderr = shift;
+ my ($cmdret, $output);
+
+ ($output, $cmdret) = $psql->query($sql);
+
+ like($output, $expected_stdout, "$io_method: $name: expected stdout");
+ like($psql->{stderr}, $expected_stderr,
+ "$io_method: $name: expected stderr");
+ $psql->{stderr} = '';
+}
+
+sub query_wait_block
+{
+ my $io_method = shift;
+ my $node = shift;
+ my $psql = shift;
+ my $name = shift;
+ my $sql = shift;
+ my $waitfor = shift;
+
+ my $pid = $psql->query_safe('SELECT pg_backend_pid()');
+
+ $psql->{stdin} .= qq($sql;\n);
+ $psql->{run}->pump_nb();
+ ok(1, "$io_method: $name: issued sql");
+
+ $node->poll_query_until('postgres',
+ qq(SELECT wait_event FROM pg_stat_activity WHERE pid = $pid),
+ $waitfor,);
+ ok(1, "$io_method: $name: observed $waitfor wait event");
+}
+
+
+###
+# Sub-tests
+###
+
+sub test_handle
+{
+ my $io_method = shift;
+ my $node = shift;
+
+ my $psql = $node->background_psql('postgres', on_error_stop => 0);
+
+ # leak warning: implicit xact
+ psql_like(
+ $io_method,
+ $psql,
+ "handle_get() leak in implicit xact",
+ qq(SELECT handle_get()),
+ qr/^$/,
+ qr/leaked AIO handle/,
+ "$io_method: leaky handle_get() warns");
+
+ # leak warning: explicit xact
+ psql_like(
+ $io_method, $psql,
+ "handle_get() leak in explicit xact",
+ qq(BEGIN; SELECT handle_get(); COMMIT),
+ qr/^$/, qr/leaked AIO handle/);
+
+
+ # leak warning: explicit xact, rollback
+ psql_like(
+ $io_method,
+ $psql,
+ "handle_get() leak in explicit xact, rollback",
+ qq(BEGIN; SELECT handle_get(); ROLLBACK;),
+ qr/^$/,
+ qr/leaked AIO handle/);
+
+ # leak warning: subtrans
+ psql_like(
+ $io_method,
+ $psql,
+ "handle_get() leak in subxact",
+ qq(BEGIN; SAVEPOINT foo; SELECT handle_get(); COMMIT;),
+ qr/^$/,
+ qr/leaked AIO handle/);
+
+ # leak warning + error: released in different command (thus resowner)
+ psql_like(
+ $io_method,
+ $psql,
+ "handle_release() in different command",
+ qq(BEGIN; SELECT handle_get(); SELECT handle_release_last(); COMMIT;),
+ qr/^$/,
+ qr/leaked AIO handle.*release in unexpected state/ms);
+
+ # no leak, release in same command
+ psql_like(
+ $io_method,
+ $psql,
+ "handle_release() in same command",
+ qq(BEGIN; SELECT handle_get() UNION ALL SELECT handle_release_last(); COMMIT;),
+ qr/^$/,
+ qr/^$/);
+
+ # normal handle use
+ psql_like($io_method, $psql, "handle_get_release()",
+ qq(SELECT handle_get_release()),
+ qr/^$/, qr/^$/);
+
+ # should error out, API violation
+ psql_like($io_method, $psql, "handle_get_twice()",
+ qq(SELECT handle_get_release()),
+ qr/^$/, qr/^$/);
+
+ # recover after error in implicit xact
+ psql_like(
+ $io_method,
+ $psql,
+ "handle error recovery in implicit xact",
+ qq(SELECT handle_get_and_error(); SELECT 'ok', handle_get_release()),
+ qr/^|ok$/,
+ qr/ERROR.*as you command/);
+
+ # recover after error in implicit xact
+ psql_like(
+ $io_method,
+ $psql,
+ "handle error recovery in explicit xact",
+ qq(BEGIN; SELECT handle_get_and_error(); SELECT handle_get_release(), 'ok'; COMMIT;),
+ qr/^|ok$/,
+ qr/ERROR.*as you command/);
+
+ # recover after error in subtrans
+ psql_like(
+ $io_method,
+ $psql,
+ "handle error recovery in explicit subxact",
+ qq(BEGIN; SAVEPOINT foo; SELECT handle_get_and_error(); ROLLBACK TO SAVEPOINT foo; SELECT handle_get_release(); ROLLBACK;),
+ qr/^|ok$/,
+ qr/ERROR.*as you command/);
+
+ $psql->quit();
+}
+
+
+sub test_batch
+{
+ my $io_method = shift;
+ my $node = shift;
+
+ my $psql = $node->background_psql('postgres', on_error_stop => 0);
+
+ # leak warning & recovery: implicit xact
+ psql_like(
+ $io_method,
+ $psql,
+ "batch_start() leak & cleanup in implicit xact",
+ qq(SELECT batch_start()),
+ qr/^$/,
+ qr/open AIO batch at end/,
+ "$io_method: leaky batch_start() warns");
+
+ # leak warning & recovery: explicit xact
+ psql_like(
+ $io_method,
+ $psql,
+ "batch_start() leak & cleanup in explicit xact",
+ qq(BEGIN; SELECT batch_start(); COMMIT;),
+ qr/^$/,
+ qr/open AIO batch at end/,
+ "$io_method: leaky batch_start() warns");
+
+
+ # leak warning & recovery: explicit xact, rollback
+ #
+ # XXX: This doesn't fail right now, due to not getting a chance to do
+ # something at transaction command commit. That's not a correctness issue,
+ # it just means it's a bit harder to find buggy code.
+ #psql_like($io_method, $psql,
+ # "batch_start() leak & cleanup after abort",
+ # qq(BEGIN; SELECT batch_start(); ROLLBACK;),
+ # qr/^$/,
+ # qr/open AIO batch at end/, "$io_method: leaky batch_start() warns");
+
+ # no warning, batch closed in same command
+ psql_like(
+ $io_method,
+ $psql,
+ "batch_start(), batch_end() works",
+ qq(SELECT batch_start() UNION ALL SELECT batch_end()),
+ qr/^$/,
+ qr/^$/,
+ "$io_method: batch_start(), batch_end()");
+
+ $psql->quit();
+}
+
+sub test_io_error
+{
+ my $io_method = shift;
+ my $node = shift;
+ my ($ret, $output);
+
+ my $psql = $node->background_psql('postgres', on_error_stop => 0);
+
+ # verify the error is reported in custom C code
+ ($output, $ret) = $psql->query(
+ qq(SELECT read_rel_block_ll('tbl_corr', 1, wait_complete=>true);));
+ is($ret, 1, "$io_method: read_rel_block_ll() of tbl_corr page fails");
+ like(
+ $psql->{stderr},
+ qr/invalid page in block 1 of relation base\/.*/,
+ "$io_method: read_rel_block_ll() of tbl_corr page reports error");
+ $psql->{stderr} = '';
+
+ # verify the error is reported for bufmgr reads
+ ($output, $ret) =
+ $psql->query(qq(SELECT count(*) FROM tbl_corr WHERE ctid = '(1, 1)'));
+ is($ret, 1, "$io_method: tid scan reading tbl_corr block fails");
+ like(
+ $psql->{stderr},
+ qr/invalid page in block 1 of relation base\/.*/,
+ "$io_method: tid scan reading tbl_corr block reports error");
+ $psql->{stderr} = '';
+
+ # verify the error is reported for bufmgr reads
+ ($output, $ret) =
+ $psql->query(qq(SELECT count(*) FROM tbl_corr WHERE ctid = '(1, 1)'));
+ is($ret, 1, "$io_method: sequential scan reading tbl_corr block fails");
+ like(
+ $psql->{stderr},
+ qr/invalid page in block 1 of relation base\/.*/,
+ "$io_method: sequential scan reading tbl_corr block reports error");
+ $psql->{stderr} = '';
+
+ $psql->quit();
+}
+
+# Test interplay between StartBufferIO and TerminateBufferIO
+sub test_startwait_io
+{
+ my $io_method = shift;
+ my $node = shift;
+ my ($ret, $output);
+
+ my $psql_a = $node->background_psql('postgres', on_error_stop => 0);
+ my $psql_b = $node->background_psql('postgres', on_error_stop => 0);
+
+ # create a buffer we can play around with
+ ($output, $ret) =
+ $psql_a->query(qq(SELECT buffer_create_toy('tbl_ok', 1);));
+ is($ret, 0, "$io_method: create toy succeeds");
+ like($output, qr/^\d+$/, "$io_method: create toy returns numeric");
+ my $buf_id = $output;
+
+ # check that one backend can perform StartBufferIO
+ psql_like(
+ $io_method,
+ $psql_a,
+ "first StartBufferIO",
+ qq(SELECT buffer_call_start_io($buf_id, for_input=>true, nowait=>false);),
+ qr/^t$/,
+ qr/^$/);
+
+ # but not twice on the same buffer (non-waiting)
+ psql_like(
+ $io_method,
+ $psql_a,
+ "second StartBufferIO fails, same session",
+ qq(SELECT buffer_call_start_io($buf_id, for_input=>true, nowait=>true);),
+ qr/^f$/,
+ qr/^$/);
+ psql_like(
+ $io_method,
+ $psql_b,
+ "second StartBufferIO fails, other session",
+ qq(SELECT buffer_call_start_io($buf_id, for_input=>true, nowait=>true);),
+ qr/^f$/,
+ qr/^$/);
+
+ # start io in a different session, will block
+ query_wait_block(
+ $io_method,
+ $node,
+ $psql_b,
+ "blocking start buffer io",
+ qq(SELECT buffer_call_start_io($buf_id, for_input=>true, nowait=>false);),
+ "BufferIo");
+
+ # Terminate the IO, without marking it as success, this should trigger the
+ # waiting session to be able to start the io
+ ($output, $ret) = $psql_a->query(
+ qq(SELECT buffer_call_terminate_io($buf_id, for_input=>true, succeed=>false, io_error=>false, syncio=>true);)
+ );
+ is($ret, 0,
+ "$io_method: blocking start buffer io, terminating io, not valid");
+
+ # Because the IO was terminated, but not marked as valid, second session should get the right to start io
+ pump_until($psql_b->{run}, $psql_b->{timeout}, \$psql_b->{stdout}, qr/t/);
+ ok(1, "$io_method: blocking start buffer io, can start io");
+
+ # terminate the IO again
+ $psql_b->query_safe(
+ qq(SELECT buffer_call_terminate_io($buf_id, for_input=>true, succeed=>false, io_error=>false, syncio=>true);)
+ );
+
+
+ # same as the above scenario, but mark IO as having succeeded
+ ($output, $ret) =
+ $psql_a->query(qq(SELECT buffer_call_start_io($buf_id, true, false);));
+ is($ret, 0,
+ "$io_method: blocking buffer io w/ success: first start buffer io succeeds"
+ );
+ is($output, "t",
+ "$io_method: blocking buffer io w/ success: first start buffer io returns true"
+ );
+
+
+ # start io in a different session, will block
+ query_wait_block(
+ $io_method,
+ $node,
+ $psql_b,
+ "blocking start buffer io",
+ qq(SELECT buffer_call_start_io($buf_id, for_input=>true, nowait=>false);),
+ "BufferIo");
+
+ ($output, $ret) = $psql_a->query(
+ qq(SELECT buffer_call_terminate_io($buf_id, for_input=>true, succeed=>true, io_error=>false, syncio=>true);)
+ );
+ is($ret, 0,
+ "$io_method: blocking start buffer io, terminating IO, valid");
+
+ # Because the IO was terminated, and marked as valid, second session should complete but not need io
+ pump_until($psql_b->{run}, $psql_b->{timeout}, \$psql_b->{stdout}, qr/f/);
+ ok(1, "$io_method: blocking start buffer io, no need to start io");
+
+
+ # buffer is valid now, make it invalid again
+ $buf_id = $psql_a->query_safe(qq(SELECT buffer_create_toy('tbl_ok', 1);));
+
+ $psql_a->quit();
+ $psql_b->quit();
+}
+
+# Test that if the backend issuing a read doesn't wait for the IO's
+# completion, another backend can complete the IO
+sub test_complete_foreign
+{
+ my $io_method = shift;
+ my $node = shift;
+ my ($ret, $output);
+
+ my $psql_a = $node->background_psql('postgres', on_error_stop => 0);
+ my $psql_b = $node->background_psql('postgres', on_error_stop => 0);
+
+ # Issue IO without waiting for completion, then sleep
+ $psql_a->query_safe(
+ qq(SELECT read_rel_block_ll('tbl_ok', 1, wait_complete=>false);));
+
+ # Check that another backend can read the relevant block
+ psql_like(
+ $io_method,
+ $psql_b,
+ "completing read started by sleeping backend",
+ qq(SELECT count(*) FROM tbl_ok WHERE ctid = '(1,1)' LIMIT 1),
+ qr/^1$/,
+ qr/^$/);
+
+ # Issue IO without waiting for completion, then exit
+ $psql_a->query_safe(
+ qq(SELECT read_rel_block_ll('tbl_ok', 1, wait_complete=>false);));
+ $psql_a->reconnect_and_clear();
+
+ # Check that another backend can read the relevant block
+ psql_like(
+ $io_method,
+ $psql_b,
+ "completing read started by exited backend",
+ qq(SELECT count(*) FROM tbl_ok WHERE ctid = '(1,1)' LIMIT 1),
+ qr/^1$/,
+ qr/^$/);
+
+ # Read a tbl_corr block, then sleep. The other session will retry the IO
+ # and also fail. The easiest thing to verify that seems to be to check
+ # that both are in the log.
+ my $log_location = -s $node->logfile;
+ $psql_a->query_safe(
+ qq(SELECT read_rel_block_ll('tbl_corr', 1, wait_complete=>false);));
+
+ psql_like(
+ $io_method,
+ $psql_b,
+ "completing read of tbl_corr block started by other backend",
+ qq(SELECT count(*) FROM tbl_corr WHERE ctid = '(1,1)' LIMIT 1),
+ qr/^$/,
+ qr/invalid page in block/);
+
+ # The log message issued for the read_rel_block_ll() should be logged as a LOG
+ $node->wait_for_log(qr/LOG[^\n]+invalid page in/, $log_location);
+ ok(1,
+ "$io_method: completing read of tbl_corr block started by other backend: LOG message for background read"
+ );
+
+ # But for the SELECT, it should be an ERROR
+ $log_location =
+ $node->wait_for_log(qr/ERROR[^\n]+invalid page in/, $log_location);
+ ok(1,
+ "$io_method: completing read of tbl_corr block started by other backend: ERROR message for foreground read"
+ );
+
+ $psql_a->quit();
+ $psql_b->quit();
+}
+
+sub test_inject
+{
+ my $io_method = shift;
+ my $node = shift;
+ my ($ret, $output);
+
+ my $psql = $node->background_psql('postgres', on_error_stop => 0);
+
+ # injected what we'd expect
+ $psql->query_safe(qq(SELECT inj_io_short_read_attach(8192);));
+ $psql->query_safe(qq(SELECT invalidate_rel_block('tbl_ok', 2);));
+ ($output, $ret) =
+ $psql->query(qq(SELECT count(*) FROM tbl_ok WHERE ctid = '(2, 1)';));
+ is($ret, 0,
+ "$io_method: injection point not triggering failure succeeds");
+
+ # injected a read shorter than a single block, expecting error
+ $psql->query_safe(qq(SELECT inj_io_short_read_attach(17);));
+ $psql->query_safe(qq(SELECT invalidate_rel_block('tbl_ok', 2);));
+ ($output, $ret) =
+ $psql->query(qq(SELECT count(*) FROM tbl_ok WHERE ctid = '(2, 1)';));
+ is($ret, 1, "$io_method: single block short read fails");
+ like(
+ $psql->{stderr},
+ qr/ERROR:.*could not read blocks 2\.\.2 in file "base\/.*": read only 0 of 8192 bytes/,
+ "$io_method: single block short read reports error");
+ $psql->{stderr} = '';
+
+ # shorten multi-block read to a single block, should retry
+ $psql->query_safe(
+ qq(
+SELECT invalidate_rel_block('tbl_ok', 0);
+SELECT invalidate_rel_block('tbl_ok', 1);
+SELECT invalidate_rel_block('tbl_ok', 2);
+SELECT invalidate_rel_block('tbl_ok', 3);
+/* gap */
+SELECT invalidate_rel_block('tbl_ok', 5);
+SELECT invalidate_rel_block('tbl_ok', 6);
+SELECT invalidate_rel_block('tbl_ok', 7);
+SELECT invalidate_rel_block('tbl_ok', 8);
+
+SELECT inj_io_short_read_attach(8192);
+ ));
+ ($output, $ret) = $psql->query(qq(SELECT count(*) FROM tbl_ok;));
+ is($ret, 0, "$io_method: multi block short read is retried");
+
+ # verify that page verification errors are detected even as part of a
+ # shortened multi-block read (tbl_corr, block 1 is tbl_corred)
+ $psql->query_safe(
+ qq(
+SELECT invalidate_rel_block('tbl_corr', 0);
+SELECT invalidate_rel_block('tbl_corr', 1);
+SELECT invalidate_rel_block('tbl_corr', 2);
+SELECT inj_io_short_read_attach(8192);
+ ));
+ ($output, $ret) =
+ $psql->query(qq(SELECT count(*) FROM tbl_corr WHERE ctid < '(2, 1)'));
+ is($ret, 1,
+ "$io_method: shortened multi-block read detects invalid page");
+ like(
+ $psql->{stderr},
+ qr/ERROR:.*invalid page in block 1 of relation base\/.*/,
+ "$io_method: shortened multi-block reads reports invalid page");
+ $psql->{stderr} = '';
+
+ # trigger a hard error, should error out
+ $psql->query_safe(
+ qq(
+SELECT inj_io_short_read_attach(-errno_from_string('EIO'));
+SELECT invalidate_rel_block('tbl_ok', 2);
+ ));
+ ($output, $ret) =
+ $psql->query(qq(SELECT count(*) FROM tbl_ok; SELECT 1;));
+ is($ret, 1, "$io_method: hard IO error is detected");
+ like(
+ $psql->{stderr},
+ qr/ERROR:.*could not read blocks 2\.\.2 in file \"base\/.*\": Input\/output error/,
+ "$io_method: hard IO error is reported");
+ $psql->{stderr} = '';
+
+ $psql->query_safe(
+ qq(
+SELECT inj_io_short_read_detach();
+ ));
+
+ $psql->quit();
+}
+
+sub test_inject_worker
+{
+ my $io_method = shift;
+ my $node = shift;
+ my ($ret, $output);
+
+ my $psql = $node->background_psql('postgres', on_error_stop => 0);
+
+ # trigger a failure to reopen, should error out, but should recover
+ $psql->query_safe(
+ qq(
+SELECT inj_io_reopen_attach();
+SELECT invalidate_rel_block('tbl_ok', 1);
+ ));
+ ($output, $ret) = $psql->query(qq(SELECT count(*) FROM tbl_ok;));
+ is($ret, 1, "$io_method: failure to open is detected");
+ like(
+ $psql->{stderr},
+ qr/ERROR:.*could not read blocks 1\.\.1 in file "base\/.*": No such file or directory/,
+ "$io_method: failure to open is reported");
+ $psql->{stderr} = '';
+
+ $psql->query_safe(
+ qq(
+SELECT inj_io_reopen_detach();
+ ));
+
+ # check that we indeed recover
+ ($output, $ret) = $psql->query(qq(SELECT count(*) FROM tbl_ok;));
+ is($ret, 0, "$io_method: recovers from failure to open ");
+
+
+ $psql->quit();
+}
+
+sub test_generic
+{
+ my $io_method = shift;
+ my $node = shift;
+
+ is($node->safe_psql('postgres', 'SHOW io_method'),
+ $io_method, "$io_method: io_method set correctly");
+
+ $node->safe_psql(
+ 'postgres', qq(
+CREATE EXTENSION test_aio;
+CREATE TABLE tbl_corr(data int not null) WITH (AUTOVACUUM_ENABLED = false);
+CREATE TABLE tbl_ok(data int not null) WITH (AUTOVACUUM_ENABLED = false);
+
+INSERT INTO tbl_corr SELECT generate_series(1, 10000);
+INSERT INTO tbl_ok SELECT generate_series(1, 10000);
+SELECT grow_rel('tbl_corr', 16);
+SELECT grow_rel('tbl_ok', 16);
+
+SELECT corrupt_rel_block('tbl_corr', 1);
+CHECKPOINT;
+));
+
+ test_handle($io_method, $node);
+ test_io_error($io_method, $node);
+ test_batch($io_method, $node);
+ test_startwait_io($io_method, $node);
+ test_complete_foreign($io_method, $node);
+
+ SKIP:
+ {
+ skip 'Injection points not supported by this build', 1
+ unless $ENV{enable_injection_points} eq 'yes';
+ test_inject($io_method, $node);
+ }
+}
diff --git a/src/test/modules/test_aio/test_aio--1.0.sql b/src/test/modules/test_aio/test_aio--1.0.sql
new file mode 100644
index 00000000000..3d4e9bb070f
--- /dev/null
+++ b/src/test/modules/test_aio/test_aio--1.0.sql
@@ -0,0 +1,97 @@
+/* src/test/modules/test_aio/test_aio--1.0.sql */
+
+-- complain if script is sourced in psql, rather than via CREATE EXTENSION
+\echo Use "CREATE EXTENSION test_aio" to load this file. \quit
+
+
+CREATE FUNCTION errno_from_string(sym text)
+RETURNS pg_catalog.int4 STRICT
+AS 'MODULE_PATHNAME' LANGUAGE C;
+
+
+CREATE FUNCTION grow_rel(rel regclass, nblocks int)
+RETURNS pg_catalog.void STRICT
+AS 'MODULE_PATHNAME' LANGUAGE C;
+
+
+CREATE FUNCTION corrupt_rel_block(rel regclass, blockno int)
+RETURNS pg_catalog.void STRICT
+AS 'MODULE_PATHNAME' LANGUAGE C;
+
+CREATE FUNCTION read_rel_block_ll(rel regclass, blockno int, wait_complete bool)
+RETURNS pg_catalog.void STRICT
+AS 'MODULE_PATHNAME' LANGUAGE C;
+
+CREATE FUNCTION invalidate_rel_block(rel regclass, blockno int)
+RETURNS pg_catalog.void STRICT
+AS 'MODULE_PATHNAME' LANGUAGE C;
+
+CREATE FUNCTION buffer_create_toy(rel regclass, blockno int4)
+RETURNS pg_catalog.int4 STRICT
+AS 'MODULE_PATHNAME' LANGUAGE C;
+
+CREATE FUNCTION buffer_call_start_io(buffer int, for_input bool, nowait bool)
+RETURNS pg_catalog.bool STRICT
+AS 'MODULE_PATHNAME' LANGUAGE C;
+
+CREATE FUNCTION buffer_call_terminate_io(buffer int, for_input bool, succeed bool, io_error bool, syncio bool)
+RETURNS pg_catalog.void STRICT
+AS 'MODULE_PATHNAME' LANGUAGE C;
+
+
+
+/*
+ * Handle related functions
+ */
+CREATE FUNCTION handle_get_and_error()
+RETURNS pg_catalog.void STRICT
+AS 'MODULE_PATHNAME' LANGUAGE C;
+
+CREATE FUNCTION handle_get_twice()
+RETURNS pg_catalog.void STRICT
+AS 'MODULE_PATHNAME' LANGUAGE C;
+
+CREATE FUNCTION handle_get()
+RETURNS pg_catalog.void STRICT
+AS 'MODULE_PATHNAME' LANGUAGE C;
+
+CREATE FUNCTION handle_get_release()
+RETURNS pg_catalog.void STRICT
+AS 'MODULE_PATHNAME' LANGUAGE C;
+
+CREATE FUNCTION handle_release_last()
+RETURNS pg_catalog.void STRICT
+AS 'MODULE_PATHNAME' LANGUAGE C;
+
+
+/*
+ * Batchmode related functions
+ */
+CREATE FUNCTION batch_start()
+RETURNS pg_catalog.void STRICT
+AS 'MODULE_PATHNAME' LANGUAGE C;
+
+CREATE FUNCTION batch_end()
+RETURNS pg_catalog.void STRICT
+AS 'MODULE_PATHNAME' LANGUAGE C;
+
+
+
+/*
+ * Injection point related functions
+ */
+CREATE FUNCTION inj_io_short_read_attach(result int)
+RETURNS pg_catalog.void STRICT
+AS 'MODULE_PATHNAME' LANGUAGE C;
+
+CREATE FUNCTION inj_io_short_read_detach()
+RETURNS pg_catalog.void STRICT
+AS 'MODULE_PATHNAME' LANGUAGE C;
+
+CREATE FUNCTION inj_io_reopen_attach()
+RETURNS pg_catalog.void STRICT
+AS 'MODULE_PATHNAME' LANGUAGE C;
+
+CREATE FUNCTION inj_io_reopen_detach()
+RETURNS pg_catalog.void STRICT
+AS 'MODULE_PATHNAME' LANGUAGE C;
diff --git a/src/test/modules/test_aio/test_aio.c b/src/test/modules/test_aio/test_aio.c
new file mode 100644
index 00000000000..81b4d732206
--- /dev/null
+++ b/src/test/modules/test_aio/test_aio.c
@@ -0,0 +1,616 @@
+/*-------------------------------------------------------------------------
+ *
+ * delay_execution.c
+ * Test module to allow delay between parsing and execution of a query.
+ *
+ * The delay is implemented by taking and immediately releasing a specified
+ * advisory lock. If another process has previously taken that lock, the
+ * current process will be blocked until the lock is released; otherwise,
+ * there's no effect. This allows an isolationtester script to reliably
+ * test behaviors where some specified action happens in another backend
+ * between parsing and execution of any desired query.
+ *
+ * Copyright (c) 2020-2025, PostgreSQL Global Development Group
+ *
+ * IDENTIFICATION
+ * src/test/modules/delay_execution/delay_execution.c
+ *
+ *-------------------------------------------------------------------------
+ */
+
+#include "postgres.h"
+
+#include "access/relation.h"
+#include "fmgr.h"
+#include "storage/aio.h"
+#include "storage/aio_internal.h"
+#include "storage/buf_internals.h"
+#include "storage/bufmgr.h"
+#include "storage/ipc.h"
+#include "storage/lwlock.h"
+#include "utils/builtins.h"
+#include "utils/injection_point.h"
+#include "utils/rel.h"
+
+
+PG_MODULE_MAGIC;
+
+
+typedef struct InjIoErrorState
+{
+ bool enabled_short_read;
+ bool enabled_reopen;
+
+ bool short_read_result_set;
+ int short_read_result;
+} InjIoErrorState;
+
+static InjIoErrorState * inj_io_error_state;
+
+/* Shared memory init callbacks */
+static shmem_request_hook_type prev_shmem_request_hook = NULL;
+static shmem_startup_hook_type prev_shmem_startup_hook = NULL;
+
+
+static PgAioHandle *last_handle;
+
+
+
+static void
+test_aio_shmem_request(void)
+{
+ if (prev_shmem_request_hook)
+ prev_shmem_request_hook();
+
+ RequestAddinShmemSpace(sizeof(InjIoErrorState));
+}
+
+static void
+test_aio_shmem_startup(void)
+{
+ bool found;
+
+ if (prev_shmem_startup_hook)
+ prev_shmem_startup_hook();
+
+ /* Create or attach to the shared memory state */
+ LWLockAcquire(AddinShmemInitLock, LW_EXCLUSIVE);
+
+ inj_io_error_state = ShmemInitStruct("injection_points",
+ sizeof(InjIoErrorState),
+ &found);
+
+ if (!found)
+ {
+ /*
+ * First time through, so initialize. This is shared with the dynamic
+ * initialization using a DSM.
+ */
+ inj_io_error_state->enabled_short_read = false;
+ inj_io_error_state->enabled_reopen = false;
+
+#ifdef USE_INJECTION_POINTS
+ InjectionPointAttach("AIO_PROCESS_COMPLETION_BEFORE_SHARED",
+ "test_aio",
+ "inj_io_short_read",
+ NULL,
+ 0);
+ InjectionPointLoad("AIO_PROCESS_COMPLETION_BEFORE_SHARED");
+
+ InjectionPointAttach("AIO_WORKER_AFTER_REOPEN",
+ "test_aio",
+ "inj_io_reopen",
+ NULL,
+ 0);
+ InjectionPointLoad("AIO_WORKER_AFTER_REOPEN");
+
+#endif
+ }
+ else
+ {
+#ifdef USE_INJECTION_POINTS
+ InjectionPointLoad("AIO_PROCESS_COMPLETION_BEFORE_SHARED");
+ InjectionPointLoad("AIO_WORKER_AFTER_REOPEN");
+ elog(LOG, "injection point loaded");
+#endif
+ }
+
+ LWLockRelease(AddinShmemInitLock);
+}
+
+void
+_PG_init(void)
+{
+ if (!process_shared_preload_libraries_in_progress)
+ return;
+
+ /* Shared memory initialization */
+ prev_shmem_request_hook = shmem_request_hook;
+ shmem_request_hook = test_aio_shmem_request;
+ prev_shmem_startup_hook = shmem_startup_hook;
+ shmem_startup_hook = test_aio_shmem_startup;
+}
+
+
+PG_FUNCTION_INFO_V1(errno_from_string);
+Datum
+errno_from_string(PG_FUNCTION_ARGS)
+{
+ const char *sym = text_to_cstring(PG_GETARG_TEXT_PP(0));
+
+ if (strcmp(sym, "EIO") == 0)
+ PG_RETURN_INT32(EIO);
+ else if (strcmp(sym, "EAGAIN") == 0)
+ PG_RETURN_INT32(EAGAIN);
+ else if (strcmp(sym, "EINTR") == 0)
+ PG_RETURN_INT32(EINTR);
+ else if (strcmp(sym, "ENOSPC") == 0)
+ PG_RETURN_INT32(ENOSPC);
+ else if (strcmp(sym, "EROFS") == 0)
+ PG_RETURN_INT32(EROFS);
+
+ ereport(ERROR,
+ errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ errmsg_internal("%s is not a supported errno value", sym));
+ PG_RETURN_INT32(0);
+}
+
+
+PG_FUNCTION_INFO_V1(grow_rel);
+Datum
+grow_rel(PG_FUNCTION_ARGS)
+{
+ Oid relid = PG_GETARG_OID(0);
+ uint32 nblocks = PG_GETARG_UINT32(1);
+ Relation rel;
+#define MAX_BUFFERS_TO_EXTEND_BY 64
+ Buffer victim_buffers[MAX_BUFFERS_TO_EXTEND_BY];
+
+ rel = relation_open(relid, AccessExclusiveLock);
+
+ while (nblocks > 0)
+ {
+ uint32 extend_by_pages;
+
+ extend_by_pages = Min(nblocks, MAX_BUFFERS_TO_EXTEND_BY);
+
+ ExtendBufferedRelBy(BMR_REL(rel),
+ MAIN_FORKNUM,
+ NULL,
+ 0,
+ extend_by_pages,
+ victim_buffers,
+ &extend_by_pages);
+
+ nblocks -= extend_by_pages;
+
+ for (uint32 i = 0; i < extend_by_pages; i++)
+ {
+ ReleaseBuffer(victim_buffers[i]);
+ }
+ }
+
+ relation_close(rel, NoLock);
+
+ PG_RETURN_VOID();
+}
+
+PG_FUNCTION_INFO_V1(corrupt_rel_block);
+Datum
+corrupt_rel_block(PG_FUNCTION_ARGS)
+{
+ Oid relid = PG_GETARG_OID(0);
+ BlockNumber blkno = PG_GETARG_UINT32(1);
+ Relation rel;
+ Buffer buf;
+ Page page;
+ PageHeader ph;
+
+ rel = relation_open(relid, AccessExclusiveLock);
+
+ buf = ReadBuffer(rel, blkno);
+ page = BufferGetPage(buf);
+
+ LockBuffer(buf, BUFFER_LOCK_EXCLUSIVE);
+
+ MarkBufferDirty(buf);
+
+ PageInit(page, BufferGetPageSize(buf), 0);
+
+ ph = (PageHeader) page;
+ ph->pd_special = BLCKSZ + 1;
+
+ FlushOneBuffer(buf);
+
+ LockBuffer(buf, BUFFER_LOCK_UNLOCK);
+
+ ReleaseBuffer(buf);
+
+ EvictUnpinnedBuffer(buf);
+
+ relation_close(rel, NoLock);
+
+ PG_RETURN_VOID();
+}
+
+/*
+ * Ensures a buffer for rel & blkno is in shared buffers, without actually
+ * caring about the buffer contents. Used to set up test scenarios.
+ */
+static Buffer
+create_toy_buffer(Relation rel, BlockNumber blkno)
+{
+ Buffer buf;
+ BufferDesc *buf_hdr;
+ uint32 buf_state;
+ bool was_pinned = false;
+
+ /* read buffer without erroring out */
+ buf = ReadBufferExtended(rel, MAIN_FORKNUM, blkno, RBM_ZERO_ON_ERROR, NULL);
+
+ buf_hdr = GetBufferDescriptor(buf - 1);
+
+ buf_state = LockBufHdr(buf_hdr);
+
+ /*
+ * We should be the only backend accessing this buffer. This is just a
+ * small bit of belt-and-suspenders defense, none of this code should ever
+ * run in a cluster with real data.
+ */
+ if (BUF_STATE_GET_REFCOUNT(buf_state) > 1)
+ was_pinned = true;
+ else
+ buf_state &= ~(BM_VALID | BM_DIRTY);
+
+ UnlockBufHdr(buf_hdr, buf_state);
+
+ if (was_pinned)
+ elog(ERROR, "toy buffer %d was already pinned",
+ buf);
+
+ return buf;
+}
+
+PG_FUNCTION_INFO_V1(read_rel_block_ll);
+Datum
+read_rel_block_ll(PG_FUNCTION_ARGS)
+{
+ Oid relid = PG_GETARG_OID(0);
+ BlockNumber blkno = PG_GETARG_UINT32(1);
+ bool wait_complete = PG_GETARG_BOOL(2);
+ Relation rel;
+ Buffer buf;
+ Page pages[1];
+ PgAioReturn ior;
+ PgAioHandle *ioh;
+ PgAioWaitRef iow;
+ SMgrRelation smgr;
+
+ rel = relation_open(relid, AccessExclusiveLock);
+
+ buf = create_toy_buffer(rel, blkno);
+
+ pages[0] = BufferGetBlock(buf);
+
+ ioh = pgaio_io_acquire(CurrentResourceOwner, &ior);
+ pgaio_io_get_wref(ioh, &iow);
+
+ smgr = RelationGetSmgr(rel);
+
+ StartBufferIO(GetBufferDescriptor(buf - 1), true, false);
+
+ pgaio_io_set_handle_data_32(ioh, (uint32 *) &buf, 1);
+ pgaio_io_register_callbacks(ioh, PGAIO_HCB_SHARED_BUFFER_READV, 0);
+
+ elog(LOG, "about to smgrstartreadv");
+ smgrstartreadv(ioh, smgr, MAIN_FORKNUM, blkno,
+ (void *) pages, 1);
+
+ ReleaseBuffer(buf);
+
+ if (wait_complete)
+ {
+ pgaio_wref_wait(&iow);
+
+ if (ior.result.status != ARS_OK)
+ pgaio_result_report(ior.result, &ior.target_data,
+ ior.result.status == ARS_PARTIAL ? WARNING : ERROR);
+ }
+
+ relation_close(rel, NoLock);
+
+ PG_RETURN_VOID();
+}
+
+PG_FUNCTION_INFO_V1(invalidate_rel_block);
+Datum
+invalidate_rel_block(PG_FUNCTION_ARGS)
+{
+ Oid relid = PG_GETARG_OID(0);
+ BlockNumber blkno = PG_GETARG_UINT32(1);
+ Relation rel;
+ PrefetchBufferResult pr;
+ Buffer buf;
+
+ rel = relation_open(relid, AccessExclusiveLock);
+
+ /*
+ * This is a gross hack, but there's no other API exposed that allows to
+ * get a buffer ID without actually reading the block in.
+ */
+ pr = PrefetchBuffer(rel, MAIN_FORKNUM, blkno);
+ buf = pr.recent_buffer;
+
+ if (BufferIsValid(buf))
+ {
+ /* if the buffer contents aren't valid, this'll return false */
+ if (ReadRecentBuffer(rel->rd_locator, MAIN_FORKNUM, blkno, buf))
+ {
+ LockBuffer(buf, BUFFER_LOCK_EXCLUSIVE);
+ FlushOneBuffer(buf);
+ LockBuffer(buf, BUFFER_LOCK_UNLOCK);
+ ReleaseBuffer(buf);
+
+ if (!EvictUnpinnedBuffer(buf))
+ elog(ERROR, "couldn't evict");
+ }
+ }
+
+ relation_close(rel, AccessExclusiveLock);
+
+ PG_RETURN_VOID();
+}
+
+PG_FUNCTION_INFO_V1(buffer_create_toy);
+Datum
+buffer_create_toy(PG_FUNCTION_ARGS)
+{
+ Oid relid = PG_GETARG_OID(0);
+ BlockNumber blkno = PG_GETARG_UINT32(1);
+ Relation rel;
+ Buffer buf;
+
+ rel = relation_open(relid, AccessExclusiveLock);
+
+ buf = create_toy_buffer(rel, blkno);
+ ReleaseBuffer(buf);
+
+ relation_close(rel, NoLock);
+
+ PG_RETURN_INT32(buf);
+}
+
+PG_FUNCTION_INFO_V1(buffer_call_start_io);
+Datum
+buffer_call_start_io(PG_FUNCTION_ARGS)
+{
+ Buffer buf = PG_GETARG_INT32(0);
+ bool for_input = PG_GETARG_BOOL(1);
+ bool nowait = PG_GETARG_BOOL(2);
+ bool can_start;
+
+ can_start = StartBufferIO(GetBufferDescriptor(buf - 1), for_input, nowait);
+
+ /*
+ * Tor tests we don't want the resowner release preventing us from
+ * orchestrating odd scenarios.
+ */
+ if (can_start)
+ ResourceOwnerForgetBufferIO(CurrentResourceOwner,
+ buf);
+
+ ereport(LOG,
+ errmsg("buffer %d after StartBufferIO: %s",
+ buf, DebugPrintBufferRefcount(buf)),
+ errhidestmt(true), errhidecontext(true));
+
+ PG_RETURN_BOOL(can_start);
+}
+
+/*
+CREATE FUNCTION buffer_call_terminate_io(buffer int, for_input bool, failed bool, syncio bool)
+RETURNS pg_catalog.void STRICT
+AS 'MODULE_PATHNAME' LANGUAGE C;
+*/
+PG_FUNCTION_INFO_V1(buffer_call_terminate_io);
+Datum
+buffer_call_terminate_io(PG_FUNCTION_ARGS)
+{
+ Buffer buf = PG_GETARG_INT32(0);
+ bool for_input = PG_GETARG_BOOL(1);
+ bool succeed = PG_GETARG_BOOL(2);
+ bool io_error = PG_GETARG_BOOL(3);
+ bool syncio = PG_GETARG_BOOL(4);
+ bool clear_dirty = false;
+ uint32 set_flag_bits = 0;
+
+ if (io_error)
+ set_flag_bits |= BM_IO_ERROR;
+
+ if (for_input)
+ {
+ clear_dirty = false;
+
+ if (succeed)
+ set_flag_bits |= BM_VALID;
+ }
+ else
+ {
+ if (succeed)
+ clear_dirty = true;
+ }
+
+ ereport(LOG,
+ errmsg("buffer %d before TerminateBufferIO: %s",
+ buf, DebugPrintBufferRefcount(buf)),
+ errhidestmt(true), errhidecontext(true));
+
+ TerminateBufferIO(GetBufferDescriptor(buf - 1), clear_dirty, set_flag_bits,
+ false, syncio);
+
+ ereport(LOG,
+ errmsg("buffer %d after TerminateBufferIO: %s",
+ buf, DebugPrintBufferRefcount(buf)),
+ errhidestmt(true), errhidecontext(true));
+
+ PG_RETURN_VOID();
+}
+
+PG_FUNCTION_INFO_V1(handle_get);
+Datum
+handle_get(PG_FUNCTION_ARGS)
+{
+ last_handle = pgaio_io_acquire(CurrentResourceOwner, NULL);
+
+ PG_RETURN_VOID();
+}
+
+PG_FUNCTION_INFO_V1(handle_release_last);
+Datum
+handle_release_last(PG_FUNCTION_ARGS)
+{
+ if (!last_handle)
+ elog(ERROR, "no handle");
+
+ pgaio_io_release(last_handle);
+
+ PG_RETURN_VOID();
+}
+
+PG_FUNCTION_INFO_V1(handle_get_and_error);
+Datum
+handle_get_and_error(PG_FUNCTION_ARGS)
+{
+ pgaio_io_acquire(CurrentResourceOwner, NULL);
+
+ elog(ERROR, "as you command");
+ PG_RETURN_VOID();
+}
+
+PG_FUNCTION_INFO_V1(handle_get_twice);
+Datum
+handle_get_twice(PG_FUNCTION_ARGS)
+{
+ pgaio_io_acquire(CurrentResourceOwner, NULL);
+ pgaio_io_acquire(CurrentResourceOwner, NULL);
+
+ PG_RETURN_VOID();
+}
+
+PG_FUNCTION_INFO_V1(handle_get_release);
+Datum
+handle_get_release(PG_FUNCTION_ARGS)
+{
+ PgAioHandle *handle;
+
+ handle = pgaio_io_acquire(CurrentResourceOwner, NULL);
+ pgaio_io_release(handle);
+
+ PG_RETURN_VOID();
+}
+
+PG_FUNCTION_INFO_V1(batch_start);
+Datum
+batch_start(PG_FUNCTION_ARGS)
+{
+ pgaio_enter_batchmode();
+ PG_RETURN_VOID();
+}
+
+PG_FUNCTION_INFO_V1(batch_end);
+Datum
+batch_end(PG_FUNCTION_ARGS)
+{
+ pgaio_exit_batchmode();
+ PG_RETURN_VOID();
+}
+
+#ifdef USE_INJECTION_POINTS
+extern PGDLLEXPORT void inj_io_short_read(const char *name, const void *private_data);
+extern PGDLLEXPORT void inj_io_reopen(const char *name, const void *private_data);
+
+void
+inj_io_short_read(const char *name, const void *private_data)
+{
+ PgAioHandle *ioh;
+
+ elog(LOG, "short read called: %d", inj_io_error_state->enabled_short_read);
+
+ if (inj_io_error_state->enabled_short_read)
+ {
+ ioh = pgaio_inj_io_get();
+
+ if (inj_io_error_state->short_read_result_set)
+ {
+ elog(LOG, "short read, changing result from %d to %d",
+ ioh->result, inj_io_error_state->short_read_result);
+
+ ioh->result = inj_io_error_state->short_read_result;
+ }
+ }
+}
+
+void
+inj_io_reopen(const char *name, const void *private_data)
+{
+ elog(LOG, "reopen called: %d", inj_io_error_state->enabled_reopen);
+
+ if (inj_io_error_state->enabled_reopen)
+ {
+ elog(ERROR, "injection point triggering failure to reopen ");
+ }
+}
+#endif
+
+PG_FUNCTION_INFO_V1(inj_io_short_read_attach);
+Datum
+inj_io_short_read_attach(PG_FUNCTION_ARGS)
+{
+#ifdef USE_INJECTION_POINTS
+ inj_io_error_state->enabled_short_read = true;
+ inj_io_error_state->short_read_result_set = !PG_ARGISNULL(0);
+ if (inj_io_error_state->short_read_result_set)
+ inj_io_error_state->short_read_result = PG_GETARG_INT32(0);
+#else
+ elog(ERROR, "injection points not supported");
+#endif
+
+ PG_RETURN_VOID();
+}
+
+PG_FUNCTION_INFO_V1(inj_io_short_read_detach);
+Datum
+inj_io_short_read_detach(PG_FUNCTION_ARGS)
+{
+#ifdef USE_INJECTION_POINTS
+ inj_io_error_state->enabled_short_read = false;
+#else
+ elog(ERROR, "injection points not supported");
+#endif
+ PG_RETURN_VOID();
+}
+
+PG_FUNCTION_INFO_V1(inj_io_reopen_attach);
+Datum
+inj_io_reopen_attach(PG_FUNCTION_ARGS)
+{
+#ifdef USE_INJECTION_POINTS
+ inj_io_error_state->enabled_reopen = true;
+#else
+ elog(ERROR, "injection points not supported");
+#endif
+
+ PG_RETURN_VOID();
+}
+
+PG_FUNCTION_INFO_V1(inj_io_reopen_detach);
+Datum
+inj_io_reopen_detach(PG_FUNCTION_ARGS)
+{
+#ifdef USE_INJECTION_POINTS
+ inj_io_error_state->enabled_reopen = false;
+#else
+ elog(ERROR, "injection points not supported");
+#endif
+ PG_RETURN_VOID();
+}
diff --git a/src/test/modules/test_aio/test_aio.control b/src/test/modules/test_aio/test_aio.control
new file mode 100644
index 00000000000..cd91c3ed16b
--- /dev/null
+++ b/src/test/modules/test_aio/test_aio.control
@@ -0,0 +1,3 @@
+comment = 'Test code for AIO'
+default_version = '1.0'
+module_pathname = '$libdir/test_aio'
--
2.48.1.76.g4e746b1a31.dirty
[text/x-diff] v2.10-0018-aio-Implement-smgr-md-fd-write-support.patch (12.7K, ../../zp7obmaqdd6b5u2vhpw27hovpjirtqvjakkbkaz4vju7tohyul@4kfobe7ipgxm/19-v2.10-0018-aio-Implement-smgr-md-fd-write-support.patch)
download | inline diff:
From 1bcc000713dff3d98dcac5b590d8b5a426a8cd3c Mon Sep 17 00:00:00 2001
From: Andres Freund <andres@anarazel.de>
Date: Tue, 18 Mar 2025 14:40:06 -0400
Subject: [PATCH v2.10 18/28] aio: Implement smgr/md/fd write support
TODO:
- register_dirty_segment_aio() can error out in edge cases
---
src/include/storage/fd.h | 1 +
src/include/storage/md.h | 5 +
src/include/storage/smgr.h | 5 +
src/backend/storage/aio/aio_callback.c | 1 +
src/backend/storage/file/fd.c | 28 ++++
src/backend/storage/smgr/md.c | 189 +++++++++++++++++++++++++
src/backend/storage/smgr/smgr.c | 22 +++
7 files changed, 251 insertions(+)
diff --git a/src/include/storage/fd.h b/src/include/storage/fd.h
index b77d8e5e30e..2cc7c5a4761 100644
--- a/src/include/storage/fd.h
+++ b/src/include/storage/fd.h
@@ -112,6 +112,7 @@ extern int FilePrefetch(File file, off_t offset, off_t amount, uint32 wait_event
extern ssize_t FileReadV(File file, const struct iovec *iov, int iovcnt, off_t offset, uint32 wait_event_info);
extern ssize_t FileWriteV(File file, const struct iovec *iov, int iovcnt, off_t offset, uint32 wait_event_info);
extern int FileStartReadV(struct PgAioHandle *ioh, File file, int iovcnt, off_t offset, uint32 wait_event_info);
+extern int FileStartWriteV(struct PgAioHandle *ioh, File file, int iovcnt, off_t offset, uint32 wait_event_info);
extern int FileSync(File file, uint32 wait_event_info);
extern int FileZero(File file, off_t offset, off_t amount, uint32 wait_event_info);
extern int FileFallocate(File file, off_t offset, off_t amount, uint32 wait_event_info);
diff --git a/src/include/storage/md.h b/src/include/storage/md.h
index 9d7131eff43..47ae6c36c94 100644
--- a/src/include/storage/md.h
+++ b/src/include/storage/md.h
@@ -21,6 +21,7 @@
#include "storage/sync.h"
extern const PgAioHandleCallbacks aio_md_readv_cb;
+extern const PgAioHandleCallbacks aio_md_writev_cb;
/* md storage manager functionality */
extern void mdinit(void);
@@ -45,6 +46,10 @@ extern void mdstartreadv(PgAioHandle *ioh,
extern void mdwritev(SMgrRelation reln, ForkNumber forknum,
BlockNumber blocknum,
const void **buffers, BlockNumber nblocks, bool skipFsync);
+extern void mdstartwritev(PgAioHandle *ioh,
+ SMgrRelation reln, ForkNumber forknum,
+ BlockNumber blocknum,
+ const void **buffers, BlockNumber nblocks, bool skipFsync);
extern void mdwriteback(SMgrRelation reln, ForkNumber forknum,
BlockNumber blocknum, BlockNumber nblocks);
extern BlockNumber mdnblocks(SMgrRelation reln, ForkNumber forknum);
diff --git a/src/include/storage/smgr.h b/src/include/storage/smgr.h
index 856ebcda350..f00b3763ac9 100644
--- a/src/include/storage/smgr.h
+++ b/src/include/storage/smgr.h
@@ -108,6 +108,11 @@ extern void smgrwritev(SMgrRelation reln, ForkNumber forknum,
BlockNumber blocknum,
const void **buffers, BlockNumber nblocks,
bool skipFsync);
+extern void smgrstartwritev(PgAioHandle *ioh,
+ SMgrRelation reln, ForkNumber forknum,
+ BlockNumber blocknum,
+ const void **buffers, BlockNumber nblocks,
+ bool skipFsync);
extern void smgrwriteback(SMgrRelation reln, ForkNumber forknum,
BlockNumber blocknum, BlockNumber nblocks);
extern BlockNumber smgrnblocks(SMgrRelation reln, ForkNumber forknum);
diff --git a/src/backend/storage/aio/aio_callback.c b/src/backend/storage/aio/aio_callback.c
index 2537b1914d8..d1ea1ee145b 100644
--- a/src/backend/storage/aio/aio_callback.c
+++ b/src/backend/storage/aio/aio_callback.c
@@ -41,6 +41,7 @@ static const PgAioHandleCallbacksEntry aio_handle_cbs[] = {
CALLBACK_ENTRY(PGAIO_HCB_INVALID, aio_invalid_cb),
CALLBACK_ENTRY(PGAIO_HCB_MD_READV, aio_md_readv_cb),
+ CALLBACK_ENTRY(PGAIO_HCB_MD_WRITEV, aio_md_writev_cb),
CALLBACK_ENTRY(PGAIO_HCB_SHARED_BUFFER_READV, aio_shared_buffer_readv_cb),
diff --git a/src/backend/storage/file/fd.c b/src/backend/storage/file/fd.c
index 34588ed5167..46cbcb2b688 100644
--- a/src/backend/storage/file/fd.c
+++ b/src/backend/storage/file/fd.c
@@ -2346,6 +2346,34 @@ retry:
return returnCode;
}
+int
+FileStartWriteV(PgAioHandle *ioh, File file,
+ int iovcnt, off_t offset,
+ uint32 wait_event_info)
+{
+ int returnCode;
+ Vfd *vfdP;
+
+ Assert(FileIsValid(file));
+
+ DO_DB(elog(LOG, "FileStartWriteV: %d (%s) " INT64_FORMAT " %d",
+ file, VfdCache[file].fileName,
+ (int64) offset,
+ iovcnt));
+
+ returnCode = FileAccess(file);
+ if (returnCode < 0)
+ return returnCode;
+
+ vfdP = &VfdCache[file];
+
+ /* FIXME: think about / reimplement temp_file_limit */
+
+ pgaio_io_prep_writev(ioh, vfdP->fd, iovcnt, offset);
+
+ return 0;
+}
+
int
FileSync(File file, uint32 wait_event_info)
{
diff --git a/src/backend/storage/smgr/md.c b/src/backend/storage/smgr/md.c
index 11d4d5a7aea..a61fc14805e 100644
--- a/src/backend/storage/smgr/md.c
+++ b/src/backend/storage/smgr/md.c
@@ -155,12 +155,19 @@ static BlockNumber _mdnblocks(SMgrRelation reln, ForkNumber forknum,
static PgAioResult md_readv_complete(PgAioHandle *ioh, PgAioResult prior_result, uint8 cb_data);
static void md_readv_report(PgAioResult result, const PgAioTargetData *target_data, int elevel);
+static PgAioResult md_writev_complete(PgAioHandle *ioh, PgAioResult prior_result, uint8 cb_data);
+static void md_writev_report(PgAioResult result, const PgAioTargetData *target_data, int elevel);
const PgAioHandleCallbacks aio_md_readv_cb = {
.complete_shared = md_readv_complete,
.report = md_readv_report,
};
+const PgAioHandleCallbacks aio_md_writev_cb = {
+ .complete_shared = md_writev_complete,
+ .report = md_writev_report,
+};
+
static inline int
_mdfd_open_flags(void)
@@ -1107,6 +1114,56 @@ mdwritev(SMgrRelation reln, ForkNumber forknum, BlockNumber blocknum,
}
}
+/*
+ * mdstartwritev() -- Asynchronous version of mdrwritev().
+ */
+void
+mdstartwritev(PgAioHandle *ioh,
+ SMgrRelation reln, ForkNumber forknum, BlockNumber blocknum,
+ const void **buffers, BlockNumber nblocks, bool skipFsync)
+{
+ off_t seekpos;
+ MdfdVec *v;
+ BlockNumber nblocks_this_segment;
+ struct iovec *iov;
+ int iovcnt;
+
+ v = _mdfd_getseg(reln, forknum, blocknum, false,
+ EXTENSION_FAIL | EXTENSION_CREATE_RECOVERY);
+
+ seekpos = (off_t) BLCKSZ * (blocknum % ((BlockNumber) RELSEG_SIZE));
+
+ Assert(seekpos < (off_t) BLCKSZ * RELSEG_SIZE);
+
+ nblocks_this_segment =
+ Min(nblocks,
+ RELSEG_SIZE - (blocknum % ((BlockNumber) RELSEG_SIZE)));
+
+ if (nblocks_this_segment != nblocks)
+ elog(ERROR, "write crossing segment boundary");
+
+ iovcnt = pgaio_io_get_iovec(ioh, &iov);
+
+ Assert(nblocks <= iovcnt);
+
+ iovcnt = buffers_to_iovec(iov, unconstify(void **, buffers), nblocks_this_segment);
+
+ Assert(iovcnt <= nblocks_this_segment);
+
+ if (!(io_direct_flags & IO_DIRECT_DATA))
+ pgaio_io_set_flag(ioh, PGAIO_HF_BUFFERED);
+
+ pgaio_io_set_target_smgr(ioh,
+ reln,
+ forknum,
+ blocknum,
+ nblocks,
+ skipFsync);
+ pgaio_io_register_callbacks(ioh, PGAIO_HCB_MD_WRITEV, 0);
+
+ FileStartWriteV(ioh, v->mdfd_vfd, iovcnt, seekpos, WAIT_EVENT_DATA_FILE_WRITE);
+}
+
/*
* mdwriteback() -- Tell the kernel to write pages back to storage.
@@ -1495,6 +1552,40 @@ register_dirty_segment(SMgrRelation reln, ForkNumber forknum, MdfdVec *seg)
}
}
+/*
+ * Like register_dirty_segment(), except for use by AIO. In the completion
+ * callback we don't have access to the MdfdVec (the completion callback might
+ * be executed in a different backend than the issuing backend), therefore we
+ * have to implement this slightly differently.
+ */
+static void
+register_dirty_segment_aio(RelFileLocator locator, ForkNumber forknum, uint64 segno)
+{
+ FileTag tag;
+
+ INIT_MD_FILETAG(tag, locator, forknum, segno);
+
+ /*
+ * Can't block here waiting for checkpointer to accept our sync request,
+ * as checkpointer might be waiting for this AIO to finish if offloaded to
+ * a worker.
+ */
+ if (!RegisterSyncRequest(&tag, SYNC_REQUEST, false /* retryOnError */ ))
+ {
+ char path[MAXPGPATH];
+
+ ereport(DEBUG1,
+ (errmsg_internal("could not forward fsync request because request queue is full")));
+
+ /* reuse mdsyncfiletag() to avoid duplicating code */
+ if (mdsyncfiletag(&tag, path))
+ ereport(data_sync_elevel(ERROR),
+ (errcode_for_file_access(),
+ errmsg("could not fsync file \"%s\": %m",
+ path)));
+ }
+}
+
/*
* register_unlink_segment() -- Schedule a file to be deleted after next checkpoint
*/
@@ -2017,3 +2108,101 @@ md_readv_report(PgAioResult result, const PgAioTargetData *td, int elevel)
td->smgr.nblocks * (size_t) BLCKSZ));
}
}
+
+/*
+ * AIO completion callback for mdstartwritev().
+ */
+static PgAioResult
+md_writev_complete(PgAioHandle *ioh, PgAioResult prior_result, uint8 cb_data)
+{
+ PgAioTargetData *td = pgaio_io_get_target_data(ioh);
+ PgAioResult result = prior_result;
+
+ if (prior_result.result < 0)
+ {
+ result.status = ARS_ERROR;
+ result.id = PGAIO_HCB_MD_WRITEV;
+ /* For "hard" errors, track the error number in error_data */
+ result.error_data = -prior_result.result;
+ result.result = 0;
+
+ pgaio_result_report(result, td, LOG);
+
+ return result;
+ }
+
+ /*
+ * The smgr API operates in blocks, therefore convert the result from
+ * bytes to blocks.
+ */
+ result.result /= BLCKSZ;
+
+ if (result.result == 0)
+ {
+ /* consider 0 blocks written a failure */
+ result.status = ARS_ERROR;
+ result.id = PGAIO_HCB_MD_WRITEV;
+ result.error_data = 0;
+
+ pgaio_result_report(result, td, LOG);
+
+ return result;
+ }
+
+ if (result.status != ARS_ERROR &&
+ result.result < td->smgr.nblocks)
+ {
+ /* partial writes should be retried at upper level */
+ result.status = ARS_PARTIAL;
+ result.id = PGAIO_HCB_MD_WRITEV;
+ }
+
+ if (!td->smgr.skip_fsync)
+ register_dirty_segment_aio(td->smgr.rlocator, td->smgr.forkNum,
+ td->smgr.blockNum / ((BlockNumber) RELSEG_SIZE));
+
+ return result;
+}
+
+/*
+ * AIO error reporting callback for mdstartwritev().
+ */
+static void
+md_writev_report(PgAioResult result, const PgAioTargetData *td, int elevel)
+{
+ RelPathStr path;
+
+ path = relpathbackend(td->smgr.rlocator,
+ td->smgr.is_temp ? MyProcNumber : INVALID_PROC_NUMBER,
+ td->smgr.forkNum);
+
+ if (result.error_data != 0)
+ {
+ errno = result.error_data; /* for errcode_for_file_access() */
+
+ ereport(elevel,
+ errcode_for_file_access(),
+ errmsg("could not write blocks %u..%u in file \"%s\": %m",
+ td->smgr.blockNum,
+ td->smgr.blockNum + td->smgr.nblocks,
+ path.str)
+ );
+ }
+ else
+ {
+ /*
+ * NB: This will typically only be output in debug messages, while
+ * retrying a partial IO.
+ */
+ ereport(elevel,
+ errcode(ERRCODE_DATA_CORRUPTED),
+ errmsg("could not write blocks %u..%u in file \"%s\": wrote only %zu of %zu bytes",
+ td->smgr.blockNum,
+ td->smgr.blockNum + td->smgr.nblocks - 1,
+ path.str,
+ result.result * (size_t) BLCKSZ,
+ td->smgr.nblocks * (size_t) BLCKSZ
+ )
+ );
+ }
+}
diff --git a/src/backend/storage/smgr/smgr.c b/src/backend/storage/smgr/smgr.c
index 70bf6fd824a..51eaea69550 100644
--- a/src/backend/storage/smgr/smgr.c
+++ b/src/backend/storage/smgr/smgr.c
@@ -115,6 +115,11 @@ typedef struct f_smgr
BlockNumber blocknum,
const void **buffers, BlockNumber nblocks,
bool skipFsync);
+ void (*smgr_startwritev) (PgAioHandle *ioh,
+ SMgrRelation reln, ForkNumber forknum,
+ BlockNumber blocknum,
+ const void **buffers, BlockNumber nblocks,
+ bool skipFsync);
void (*smgr_writeback) (SMgrRelation reln, ForkNumber forknum,
BlockNumber blocknum, BlockNumber nblocks);
BlockNumber (*smgr_nblocks) (SMgrRelation reln, ForkNumber forknum);
@@ -142,6 +147,7 @@ static const f_smgr smgrsw[] = {
.smgr_readv = mdreadv,
.smgr_startreadv = mdstartreadv,
.smgr_writev = mdwritev,
+ .smgr_startwritev = mdstartwritev,
.smgr_writeback = mdwriteback,
.smgr_nblocks = mdnblocks,
.smgr_truncate = mdtruncate,
@@ -772,6 +778,22 @@ smgrwritev(SMgrRelation reln, ForkNumber forknum, BlockNumber blocknum,
RESUME_INTERRUPTS();
}
+/*
+ * smgrstartwritev() -- asynchronous version of smgrwritev()
+ *
+ * This starts an asynchronous writev IO with the IO handle `ioh`. Other than
+ * `ioh` all parameters are the same as smgrwritev().
+ */
+void
+smgrstartwritev(PgAioHandle *ioh,
+ SMgrRelation reln, ForkNumber forknum, BlockNumber blocknum,
+ const void **buffers, BlockNumber nblocks, bool skipFsync)
+{
+ smgrsw[reln->smgr_which].smgr_startwritev(ioh,
+ reln, forknum, blocknum, buffers,
+ nblocks, skipFsync);
+}
+
/*
* smgrwriteback() -- Trigger kernel writeback for the supplied range of
* blocks.
--
2.48.1.76.g4e746b1a31.dirty
[text/x-diff] v2.10-0019-aio-Add-bounce-buffers.patch (23.1K, ../../zp7obmaqdd6b5u2vhpw27hovpjirtqvjakkbkaz4vju7tohyul@4kfobe7ipgxm/20-v2.10-0019-aio-Add-bounce-buffers.patch)
download | inline diff:
From 4a2fc0878e4ec4bb029ad397e93351eee6407a0e Mon Sep 17 00:00:00 2001
From: Andres Freund <andres@anarazel.de>
Date: Tue, 18 Mar 2025 14:40:06 -0400
Subject: [PATCH v2.10 19/28] aio: Add bounce buffers
---
src/include/storage/aio.h | 15 ++
src/include/storage/aio_internal.h | 34 ++++
src/include/storage/aio_types.h | 2 +
src/include/utils/resowner.h | 2 +
src/backend/storage/aio/README.md | 27 +++
src/backend/storage/aio/aio.c | 178 ++++++++++++++++++
src/backend/storage/aio/aio_init.c | 123 ++++++++++++
src/backend/utils/misc/guc_tables.c | 13 ++
src/backend/utils/misc/postgresql.conf.sample | 2 +
src/backend/utils/resowner/resowner.c | 25 ++-
src/test/modules/test_aio/test_aio--1.0.sql | 21 +++
src/test/modules/test_aio/test_aio.c | 55 ++++++
src/tools/pgindent/typedefs.list | 1 +
13 files changed, 496 insertions(+), 2 deletions(-)
diff --git a/src/include/storage/aio.h b/src/include/storage/aio.h
index c0db8f9754f..c673627a358 100644
--- a/src/include/storage/aio.h
+++ b/src/include/storage/aio.h
@@ -344,6 +344,20 @@ extern bool pgaio_have_staged(void);
+/* --------------------------------------------------------------------------------
+ * Bounce Buffers
+ * --------------------------------------------------------------------------------
+ */
+
+extern PgAioBounceBuffer *pgaio_bounce_buffer_get(void);
+extern void pgaio_io_assoc_bounce_buffer(PgAioHandle *ioh, PgAioBounceBuffer *bb);
+extern uint32 pgaio_bounce_buffer_id(PgAioBounceBuffer *bb);
+extern void pgaio_bounce_buffer_release(PgAioBounceBuffer *bb);
+extern char *pgaio_bounce_buffer_buffer(PgAioBounceBuffer *bb);
+extern void pgaio_bounce_buffer_release_resowner(struct dlist_node *bb_node, bool on_error);
+
+
+
/* --------------------------------------------------------------------------------
* Other
* --------------------------------------------------------------------------------
@@ -356,6 +370,7 @@ extern void pgaio_closing_fd(int fd);
/* GUCs */
extern PGDLLIMPORT int io_method;
extern PGDLLIMPORT int io_max_concurrency;
+extern PGDLLIMPORT int io_bounce_buffers;
#endif /* AIO_H */
diff --git a/src/include/storage/aio_internal.h b/src/include/storage/aio_internal.h
index da4d9c341f0..4378d51908f 100644
--- a/src/include/storage/aio_internal.h
+++ b/src/include/storage/aio_internal.h
@@ -122,6 +122,12 @@ struct PgAioHandle
/* raw result of the IO operation */
int32 result;
+ /*
+ * List of bounce_buffers owned by IO. It would suffice to use an index
+ * based linked list here.
+ */
+ slist_head bounce_buffers;
+
/**
* In which list the handle is registered, depends on the state:
* - IDLE, in per-backend list
@@ -177,11 +183,24 @@ struct PgAioHandle
};
+/* typedef is in aio_types.h */
+struct PgAioBounceBuffer
+{
+ slist_node node;
+ struct ResourceOwnerData *resowner;
+ dlist_node resowner_node;
+ char *buffer;
+};
+
+
typedef struct PgAioBackend
{
/* index into PgAioCtl->io_handles */
uint32 io_handle_off;
+ /* index into PgAioCtl->bounce_buffers */
+ uint32 bounce_buffers_off;
+
/* IO Handles that currently are not used */
dclist_head idle_ios;
@@ -212,6 +231,12 @@ typedef struct PgAioBackend
* IOs being appended at the end.
*/
dclist_head in_flight_ios;
+
+ /* Bounce Buffers that currently are not used */
+ slist_head idle_bbs;
+
+ /* see handed_out_io */
+ PgAioBounceBuffer *handed_out_bb;
} PgAioBackend;
@@ -239,6 +264,15 @@ typedef struct PgAioCtl
uint32 io_handle_count;
PgAioHandle *io_handles;
+
+ /*
+ * To perform AIO on buffers that are not located in shared memory (either
+ * because they are not in shared memory or because we need to operate on
+ * a copy, as e.g. the case for writes when checksums are in use)
+ */
+ uint32 bounce_buffers_count;
+ PgAioBounceBuffer *bounce_buffers;
+ char *bounce_buffers_data;
} PgAioCtl;
diff --git a/src/include/storage/aio_types.h b/src/include/storage/aio_types.h
index b0b9a2a5c97..09968e17855 100644
--- a/src/include/storage/aio_types.h
+++ b/src/include/storage/aio_types.h
@@ -118,4 +118,6 @@ typedef struct PgAioReturn
} PgAioReturn;
+typedef struct PgAioBounceBuffer PgAioBounceBuffer;
+
#endif /* AIO_TYPES_H */
diff --git a/src/include/utils/resowner.h b/src/include/utils/resowner.h
index aede4bfc820..7e2ec224169 100644
--- a/src/include/utils/resowner.h
+++ b/src/include/utils/resowner.h
@@ -168,5 +168,7 @@ extern void ResourceOwnerForgetLock(ResourceOwner owner, struct LOCALLOCK *local
struct dlist_node;
extern void ResourceOwnerRememberAioHandle(ResourceOwner owner, struct dlist_node *ioh_node);
extern void ResourceOwnerForgetAioHandle(ResourceOwner owner, struct dlist_node *ioh_node);
+extern void ResourceOwnerRememberAioBounceBuffer(ResourceOwner owner, struct dlist_node *bb_node);
+extern void ResourceOwnerForgetAioBounceBuffer(ResourceOwner owner, struct dlist_node *bb_node);
#endif /* RESOWNER_H */
diff --git a/src/backend/storage/aio/README.md b/src/backend/storage/aio/README.md
index 6df4826163e..628018ff246 100644
--- a/src/backend/storage/aio/README.md
+++ b/src/backend/storage/aio/README.md
@@ -405,6 +405,33 @@ shared memory no less!), completion callbacks instead have to encode errors in
a more compact format that can be converted into an error message.
+### AIO Bounce Buffers
+
+For some uses of AIO there is no convenient memory location as the source /
+destination of an AIO. E.g. when data checksums are enabled, writes from
+shared buffers currently cannot be done directly from shared buffers, as a
+shared buffer lock still allows some modification, e.g., for hint bits(see
+`FlushBuffer()`). If the write were done in-place, such modifications can
+cause the checksum to fail.
+
+For synchronous IO this is solved by copying the buffer to separate memory
+before computing the checksum and using that copy as the source buffer for the
+AIO.
+
+However, for AIO that is not a workable solution:
+- Instead of a single buffer many buffers are required, as many IOs might be
+ in flight
+- When using the [worker method](#worker), the source/target of IO needs to be
+ in shared memory, otherwise the workers won't be able to access the memory.
+
+The AIO subsystem addresses this by providing a limited number of bounce
+buffers that can be used as the source / target for IO. A bounce buffer be
+acquired with `pgaio_bounce_buffer_get()` and multiple bounce buffers can be
+associated with an AIO Handle with `pgaio_io_assoc_bounce_buffer()`.
+
+Bounce buffers are automatically released when the IO completes.
+
+
## Helpers
Using the low-level AIO API introduces too much complexity to do so all over
diff --git a/src/backend/storage/aio/aio.c b/src/backend/storage/aio/aio.c
index a0504ee84ee..46d83da60f7 100644
--- a/src/backend/storage/aio/aio.c
+++ b/src/backend/storage/aio/aio.c
@@ -62,6 +62,8 @@ static PgAioHandle *pgaio_io_from_wref(PgAioWaitRef *iow, uint64 *ref_generation
static const char *pgaio_io_state_get_name(PgAioHandleState s);
static void pgaio_io_wait(PgAioHandle *ioh, uint64 ref_generation);
+static void pgaio_bounce_buffer_wait_for_free(void);
+
/* Options for io_method. */
const struct config_enum_entry io_method_options[] = {
@@ -76,6 +78,7 @@ const struct config_enum_entry io_method_options[] = {
/* GUCs */
int io_method = DEFAULT_IO_METHOD;
int io_max_concurrency = -1;
+int io_bounce_buffers = -1;
/* global control for AIO */
PgAioCtl *pgaio_ctl;
@@ -662,6 +665,21 @@ pgaio_io_reclaim(PgAioHandle *ioh)
}
}
+ /* reclaim all associated bounce buffers */
+ if (!slist_is_empty(&ioh->bounce_buffers))
+ {
+ slist_mutable_iter it;
+
+ slist_foreach_modify(it, &ioh->bounce_buffers)
+ {
+ PgAioBounceBuffer *bb = slist_container(PgAioBounceBuffer, node, it.cur);
+
+ slist_delete_current(&it);
+
+ slist_push_head(&pgaio_my_backend->idle_bbs, &bb->node);
+ }
+ }
+
if (ioh->resowner)
{
ResourceOwnerForgetAioHandle(ioh->resowner, &ioh->resowner_node);
@@ -1044,6 +1062,166 @@ pgaio_submit_staged(void)
+/* --------------------------------------------------------------------------------
+ * Functions primarily related to PgAioBounceBuffer
+ * --------------------------------------------------------------------------------
+ */
+
+PgAioBounceBuffer *
+pgaio_bounce_buffer_get(void)
+{
+ PgAioBounceBuffer *bb = NULL;
+ slist_node *node;
+
+ if (pgaio_my_backend->handed_out_bb != NULL)
+ elog(ERROR, "can only hand out one BB");
+
+ /*
+ * XXX: It probably is not a good idea to have bounce buffers be per
+ * backend, that's a fair bit of memory.
+ */
+ if (slist_is_empty(&pgaio_my_backend->idle_bbs))
+ {
+ pgaio_bounce_buffer_wait_for_free();
+ }
+
+ node = slist_pop_head_node(&pgaio_my_backend->idle_bbs);
+ bb = slist_container(PgAioBounceBuffer, node, node);
+
+ pgaio_my_backend->handed_out_bb = bb;
+
+ bb->resowner = CurrentResourceOwner;
+ ResourceOwnerRememberAioBounceBuffer(bb->resowner, &bb->resowner_node);
+
+ return bb;
+}
+
+void
+pgaio_io_assoc_bounce_buffer(PgAioHandle *ioh, PgAioBounceBuffer *bb)
+{
+ if (pgaio_my_backend->handed_out_bb != bb)
+ elog(ERROR, "can only assign handed out BB");
+ pgaio_my_backend->handed_out_bb = NULL;
+
+ /*
+ * There can be many bounce buffers assigned in case of vectorized IOs.
+ */
+ slist_push_head(&ioh->bounce_buffers, &bb->node);
+
+ /* once associated with an IO, the IO has ownership */
+ ResourceOwnerForgetAioBounceBuffer(bb->resowner, &bb->resowner_node);
+ bb->resowner = NULL;
+}
+
+uint32
+pgaio_bounce_buffer_id(PgAioBounceBuffer *bb)
+{
+ return bb - pgaio_ctl->bounce_buffers;
+}
+
+void
+pgaio_bounce_buffer_release(PgAioBounceBuffer *bb)
+{
+ if (pgaio_my_backend->handed_out_bb != bb)
+ elog(ERROR, "can only release handed out BB");
+
+ slist_push_head(&pgaio_my_backend->idle_bbs, &bb->node);
+ pgaio_my_backend->handed_out_bb = NULL;
+
+ ResourceOwnerForgetAioBounceBuffer(bb->resowner, &bb->resowner_node);
+ bb->resowner = NULL;
+}
+
+void
+pgaio_bounce_buffer_release_resowner(dlist_node *bb_node, bool on_error)
+{
+ PgAioBounceBuffer *bb = dlist_container(PgAioBounceBuffer, resowner_node, bb_node);
+
+ Assert(bb->resowner);
+
+ if (!on_error)
+ elog(WARNING, "leaked AIO bounce buffer");
+
+ pgaio_bounce_buffer_release(bb);
+}
+
+char *
+pgaio_bounce_buffer_buffer(PgAioBounceBuffer *bb)
+{
+ return bb->buffer;
+}
+
+static void
+pgaio_bounce_buffer_wait_for_free(void)
+{
+ static uint32 lastpos = 0;
+
+ if (pgaio_my_backend->num_staged_ios > 0)
+ {
+ pgaio_debug(DEBUG2, "submitting %d, while acquiring free bb",
+ pgaio_my_backend->num_staged_ios);
+ pgaio_submit_staged();
+ }
+
+ for (uint32 i = lastpos; i < lastpos + io_max_concurrency; i++)
+ {
+ uint32 thisoff = pgaio_my_backend->io_handle_off + (i % io_max_concurrency);
+ PgAioHandle *ioh = &pgaio_ctl->io_handles[thisoff];
+
+ switch (ioh->state)
+ {
+ case PGAIO_HS_IDLE:
+ case PGAIO_HS_HANDED_OUT:
+ continue;
+ case PGAIO_HS_DEFINED: /* should have been submitted above */
+ case PGAIO_HS_STAGED:
+ elog(ERROR, "shouldn't get here with io:%d in state %d",
+ pgaio_io_get_id(ioh), ioh->state);
+ break;
+ case PGAIO_HS_COMPLETED_IO:
+ case PGAIO_HS_SUBMITTED:
+ if (!slist_is_empty(&ioh->bounce_buffers))
+ {
+ pgaio_debug_io(DEBUG2, ioh,
+ "waiting for IO to reclaim BB with %d in flight",
+ dclist_count(&pgaio_my_backend->in_flight_ios));
+
+ /* see comment in pgaio_io_wait_for_free() about raciness */
+ pgaio_io_wait(ioh, ioh->generation);
+
+ if (slist_is_empty(&pgaio_my_backend->idle_bbs))
+ elog(WARNING, "empty after wait");
+
+ if (!slist_is_empty(&pgaio_my_backend->idle_bbs))
+ {
+ lastpos = i;
+ return;
+ }
+ }
+ break;
+ case PGAIO_HS_COMPLETED_SHARED:
+ case PGAIO_HS_COMPLETED_LOCAL:
+ /* reclaim */
+ pgaio_io_reclaim(ioh);
+
+ if (!slist_is_empty(&pgaio_my_backend->idle_bbs))
+ {
+ lastpos = i;
+ return;
+ }
+ break;
+ }
+ }
+
+ /*
+ * The submission above could have caused the IO to complete at any time.
+ */
+ if (slist_is_empty(&pgaio_my_backend->idle_bbs))
+ elog(PANIC, "no more bbs");
+}
+
+
+
/* --------------------------------------------------------------------------------
* Other
* --------------------------------------------------------------------------------
diff --git a/src/backend/storage/aio/aio_init.c b/src/backend/storage/aio/aio_init.c
index 4e405ce7ca8..b9732d989ff 100644
--- a/src/backend/storage/aio/aio_init.c
+++ b/src/backend/storage/aio/aio_init.c
@@ -93,6 +93,32 @@ AioHandleDataShmemSize(void)
io_max_concurrency));
}
+static Size
+AioBounceBufferDescShmemSize(void)
+{
+ Size sz;
+
+ /* PgAioBounceBuffer itself */
+ sz = mul_size(sizeof(PgAioBounceBuffer),
+ mul_size(AioProcs(), io_bounce_buffers));
+
+ return sz;
+}
+
+static Size
+AioBounceBufferDataShmemSize(void)
+{
+ Size sz;
+
+ /* and the associated buffer */
+ sz = mul_size(BLCKSZ,
+ mul_size(io_bounce_buffers, AioProcs()));
+ /* memory for alignment */
+ sz += BLCKSZ;
+
+ return sz;
+}
+
/*
* Choose a suitable value for io_max_concurrency.
*
@@ -118,6 +144,33 @@ AioChooseMaxConcurrency(void)
return Min(max_proportional_pins, 64);
}
+/*
+ * Choose a suitable value for io_bounce_buffers.
+ *
+ * It's very unlikely to be useful to allocate more bounce buffers for each
+ * backend than the backend is allowed to pin. Additionally, bounce buffers
+ * currently are used for writes, it seems very uncommon for more than 10% of
+ * shared_buffers to be written out concurrently.
+ *
+ * XXX: This quickly can take up significant amounts of memory, the logic
+ * should probably fine tuned.
+ */
+static int
+AioChooseBounceBuffers(void)
+{
+ uint32 max_backends;
+ int max_proportional_pins;
+
+ /* Similar logic to LimitAdditionalPins() */
+ max_backends = MaxBackends + NUM_AUXILIARY_PROCS;
+ max_proportional_pins = (NBuffers / 10) / max_backends;
+
+ max_proportional_pins = Max(max_proportional_pins, 1);
+
+ /* apply upper limit */
+ return Min(max_proportional_pins, 256);
+}
+
Size
AioShmemSize(void)
{
@@ -141,11 +194,31 @@ AioShmemSize(void)
PGC_S_OVERRIDE);
}
+
+ /*
+ * If io_bounce_buffers is -1, we automatically choose a suitable value.
+ *
+ * See also comment above.
+ */
+ if (io_bounce_buffers == -1)
+ {
+ char buf[32];
+
+ snprintf(buf, sizeof(buf), "%d", AioChooseBounceBuffers());
+ SetConfigOption("io_bounce_buffers", buf, PGC_POSTMASTER,
+ PGC_S_DYNAMIC_DEFAULT);
+ if (io_bounce_buffers == -1) /* failed to apply it? */
+ SetConfigOption("io_bounce_buffers", buf, PGC_POSTMASTER,
+ PGC_S_OVERRIDE);
+ }
+
sz = add_size(sz, AioCtlShmemSize());
sz = add_size(sz, AioBackendShmemSize());
sz = add_size(sz, AioHandleShmemSize());
sz = add_size(sz, AioHandleIOVShmemSize());
sz = add_size(sz, AioHandleDataShmemSize());
+ sz = add_size(sz, AioBounceBufferDescShmemSize());
+ sz = add_size(sz, AioBounceBufferDataShmemSize());
/* Reserve space for method specific resources. */
if (pgaio_method_ops->shmem_size)
@@ -161,6 +234,9 @@ AioShmemInit(void)
uint32 io_handle_off = 0;
uint32 iovec_off = 0;
uint32 per_backend_iovecs = io_max_concurrency * PG_IOV_MAX;
+ uint32 bounce_buffers_off = 0;
+ uint32 per_backend_bb = io_bounce_buffers;
+ char *bounce_buffers_data;
pgaio_ctl = (PgAioCtl *)
ShmemInitStruct("AioCtl", AioCtlShmemSize(), &found);
@@ -172,6 +248,7 @@ AioShmemInit(void)
pgaio_ctl->io_handle_count = AioProcs() * io_max_concurrency;
pgaio_ctl->iovec_count = AioProcs() * per_backend_iovecs;
+ pgaio_ctl->bounce_buffers_count = AioProcs() * per_backend_bb;
pgaio_ctl->backend_state = (PgAioBackend *)
ShmemInitStruct("AioBackend", AioBackendShmemSize(), &found);
@@ -184,6 +261,40 @@ AioShmemInit(void)
pgaio_ctl->handle_data = (uint64 *)
ShmemInitStruct("AioHandleData", AioHandleDataShmemSize(), &found);
+ pgaio_ctl->bounce_buffers = (PgAioBounceBuffer *)
+ ShmemInitStruct("AioBounceBufferDesc", AioBounceBufferDescShmemSize(),
+ &found);
+
+ bounce_buffers_data =
+ ShmemInitStruct("AioBounceBufferData", AioBounceBufferDataShmemSize(),
+ &found);
+ bounce_buffers_data =
+ (char *) TYPEALIGN(BLCKSZ, (uintptr_t) bounce_buffers_data);
+ pgaio_ctl->bounce_buffers_data = bounce_buffers_data;
+
+
+ /* Initialize IO handles. */
+ for (uint64 i = 0; i < pgaio_ctl->io_handle_count; i++)
+ {
+ PgAioHandle *ioh = &pgaio_ctl->io_handles[i];
+
+ ioh->op = PGAIO_OP_INVALID;
+ ioh->target = PGAIO_TID_INVALID;
+ ioh->state = PGAIO_HS_IDLE;
+
+ slist_init(&ioh->bounce_buffers);
+ }
+
+ /* Initialize Bounce Buffers. */
+ for (uint64 i = 0; i < pgaio_ctl->bounce_buffers_count; i++)
+ {
+ PgAioBounceBuffer *bb = &pgaio_ctl->bounce_buffers[i];
+
+ bb->buffer = bounce_buffers_data;
+ bounce_buffers_data += BLCKSZ;
+ }
+
+
for (int procno = 0; procno < AioProcs(); procno++)
{
PgAioBackend *bs = &pgaio_ctl->backend_state[procno];
@@ -191,9 +302,13 @@ AioShmemInit(void)
bs->io_handle_off = io_handle_off;
io_handle_off += io_max_concurrency;
+ bs->bounce_buffers_off = bounce_buffers_off;
+ bounce_buffers_off += per_backend_bb;
+
dclist_init(&bs->idle_ios);
memset(bs->staged_ios, 0, sizeof(PgAioHandle *) * PGAIO_SUBMIT_BATCH_SIZE);
dclist_init(&bs->in_flight_ios);
+ slist_init(&bs->idle_bbs);
/* initialize per-backend IOs */
for (int i = 0; i < io_max_concurrency; i++)
@@ -215,6 +330,14 @@ AioShmemInit(void)
dclist_push_tail(&bs->idle_ios, &ioh->node);
iovec_off += PG_IOV_MAX;
}
+
+ /* initialize per-backend bounce buffers */
+ for (int i = 0; i < per_backend_bb; i++)
+ {
+ PgAioBounceBuffer *bb = &pgaio_ctl->bounce_buffers[bs->bounce_buffers_off + i];
+
+ slist_push_head(&bs->idle_bbs, &bb->node);
+ }
}
out:
diff --git a/src/backend/utils/misc/guc_tables.c b/src/backend/utils/misc/guc_tables.c
index ea0d9dfb841..960ced7c274 100644
--- a/src/backend/utils/misc/guc_tables.c
+++ b/src/backend/utils/misc/guc_tables.c
@@ -3278,6 +3278,19 @@ struct config_int ConfigureNamesInt[] =
check_io_max_concurrency, NULL, NULL
},
+ {
+ {"io_bounce_buffers",
+ PGC_POSTMASTER,
+ RESOURCES_IO,
+ gettext_noop("Number of IO Bounce Buffers reserved for each backend."),
+ NULL,
+ GUC_UNIT_BLOCKS
+ },
+ &io_bounce_buffers,
+ -1, -1, 4096,
+ NULL, NULL, NULL
+ },
+
{
{"io_workers",
PGC_SIGHUP,
diff --git a/src/backend/utils/misc/postgresql.conf.sample b/src/backend/utils/misc/postgresql.conf.sample
index 0748601ec20..799851cc347 100644
--- a/src/backend/utils/misc/postgresql.conf.sample
+++ b/src/backend/utils/misc/postgresql.conf.sample
@@ -209,6 +209,8 @@
# -1 sets based on shared_buffers
# (change requires restart)
#io_workers = 3 # 1-32;
+#io_bounce_buffers = -1 # -1 sets based on shared_buffers
+ # (change requires restart)
# - Worker Processes -
diff --git a/src/backend/utils/resowner/resowner.c b/src/backend/utils/resowner/resowner.c
index d39f3e1b655..81e7e27965a 100644
--- a/src/backend/utils/resowner/resowner.c
+++ b/src/backend/utils/resowner/resowner.c
@@ -159,10 +159,11 @@ struct ResourceOwnerData
LOCALLOCK *locks[MAX_RESOWNER_LOCKS]; /* list of owned locks */
/*
- * AIO handles need be registered in critical sections and therefore
- * cannot use the normal ResourceElem mechanism.
+ * AIO handles & bounce buffers need be registered in critical sections
+ * and therefore cannot use the normal ResourceElem mechanism.
*/
dlist_head aio_handles;
+ dlist_head aio_bounce_buffers;
};
@@ -434,6 +435,7 @@ ResourceOwnerCreate(ResourceOwner parent, const char *name)
}
dlist_init(&owner->aio_handles);
+ dlist_init(&owner->aio_bounce_buffers);
return owner;
}
@@ -742,6 +744,13 @@ ResourceOwnerReleaseInternal(ResourceOwner owner,
pgaio_io_release_resowner(node, !isCommit);
}
+
+ while (!dlist_is_empty(&owner->aio_bounce_buffers))
+ {
+ dlist_node *node = dlist_head_node(&owner->aio_bounce_buffers);
+
+ pgaio_bounce_buffer_release_resowner(node, !isCommit);
+ }
}
else if (phase == RESOURCE_RELEASE_LOCKS)
{
@@ -1111,3 +1120,15 @@ ResourceOwnerForgetAioHandle(ResourceOwner owner, struct dlist_node *ioh_node)
{
dlist_delete_from(&owner->aio_handles, ioh_node);
}
+
+void
+ResourceOwnerRememberAioBounceBuffer(ResourceOwner owner, struct dlist_node *ioh_node)
+{
+ dlist_push_tail(&owner->aio_bounce_buffers, ioh_node);
+}
+
+void
+ResourceOwnerForgetAioBounceBuffer(ResourceOwner owner, struct dlist_node *ioh_node)
+{
+ dlist_delete_from(&owner->aio_bounce_buffers, ioh_node);
+}
diff --git a/src/test/modules/test_aio/test_aio--1.0.sql b/src/test/modules/test_aio/test_aio--1.0.sql
index 3d4e9bb070f..16f248619fc 100644
--- a/src/test/modules/test_aio/test_aio--1.0.sql
+++ b/src/test/modules/test_aio/test_aio--1.0.sql
@@ -76,6 +76,27 @@ RETURNS pg_catalog.void STRICT
AS 'MODULE_PATHNAME' LANGUAGE C;
+CREATE FUNCTION bb_get_and_error()
+RETURNS pg_catalog.void STRICT
+AS 'MODULE_PATHNAME' LANGUAGE C;
+
+CREATE FUNCTION bb_get_twice()
+RETURNS pg_catalog.void STRICT
+AS 'MODULE_PATHNAME' LANGUAGE C;
+
+CREATE FUNCTION bb_get()
+RETURNS pg_catalog.void STRICT
+AS 'MODULE_PATHNAME' LANGUAGE C;
+
+CREATE FUNCTION bb_get_release()
+RETURNS pg_catalog.void STRICT
+AS 'MODULE_PATHNAME' LANGUAGE C;
+
+CREATE FUNCTION bb_release_last()
+RETURNS pg_catalog.void STRICT
+AS 'MODULE_PATHNAME' LANGUAGE C;
+
+
/*
* Injection point related functions
diff --git a/src/test/modules/test_aio/test_aio.c b/src/test/modules/test_aio/test_aio.c
index 81b4d732206..60c01b251fe 100644
--- a/src/test/modules/test_aio/test_aio.c
+++ b/src/test/modules/test_aio/test_aio.c
@@ -53,6 +53,7 @@ static shmem_startup_hook_type prev_shmem_startup_hook = NULL;
static PgAioHandle *last_handle;
+static PgAioBounceBuffer *last_bb;
@@ -525,6 +526,60 @@ batch_end(PG_FUNCTION_ARGS)
PG_RETURN_VOID();
}
+PG_FUNCTION_INFO_V1(bb_get);
+Datum
+bb_get(PG_FUNCTION_ARGS)
+{
+ last_bb = pgaio_bounce_buffer_get();
+
+ PG_RETURN_VOID();
+}
+
+PG_FUNCTION_INFO_V1(bb_release_last);
+Datum
+bb_release_last(PG_FUNCTION_ARGS)
+{
+ if (!last_bb)
+ elog(ERROR, "no bb");
+
+ pgaio_bounce_buffer_release(last_bb);
+
+ PG_RETURN_VOID();
+}
+
+PG_FUNCTION_INFO_V1(bb_get_and_error);
+Datum
+bb_get_and_error(PG_FUNCTION_ARGS)
+{
+ pgaio_bounce_buffer_get();
+
+ elog(ERROR, "as you command");
+ PG_RETURN_VOID();
+}
+
+PG_FUNCTION_INFO_V1(bb_get_twice);
+Datum
+bb_get_twice(PG_FUNCTION_ARGS)
+{
+ pgaio_bounce_buffer_get();
+ pgaio_bounce_buffer_get();
+
+ PG_RETURN_VOID();
+}
+
+
+PG_FUNCTION_INFO_V1(bb_get_release);
+Datum
+bb_get_release(PG_FUNCTION_ARGS)
+{
+ PgAioBounceBuffer *bb;
+
+ bb = pgaio_bounce_buffer_get();
+ pgaio_bounce_buffer_release(bb);
+
+ PG_RETURN_VOID();
+}
+
#ifdef USE_INJECTION_POINTS
extern PGDLLEXPORT void inj_io_short_read(const char *name, const void *private_data);
extern PGDLLEXPORT void inj_io_reopen(const char *name, const void *private_data);
diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list
index 53abd54edd8..d633c732162 100644
--- a/src/tools/pgindent/typedefs.list
+++ b/src/tools/pgindent/typedefs.list
@@ -2132,6 +2132,7 @@ PermutationStep
PermutationStepBlocker
PermutationStepBlockerType
PgAioBackend
+PgAioBounceBuffer
PgAioCtl
PgAioHandle
PgAioHandleCallbackID
--
2.48.1.76.g4e746b1a31.dirty
[text/x-diff] v2.10-0020-bufmgr-Implement-AIO-write-support.patch (9.8K, ../../zp7obmaqdd6b5u2vhpw27hovpjirtqvjakkbkaz4vju7tohyul@4kfobe7ipgxm/21-v2.10-0020-bufmgr-Implement-AIO-write-support.patch)
download | inline diff:
From 9fee2ba1e38eaf80c972642b1507e6f2747e7a61 Mon Sep 17 00:00:00 2001
From: Andres Freund <andres@anarazel.de>
Date: Tue, 18 Mar 2025 14:40:06 -0400
Subject: [PATCH v2.10 20/28] bufmgr: Implement AIO write support
As of this commit there are no users of these AIO facilities, that'll come in
later commits.
Author:
Reviewed-By:
Discussion: https://postgr.es/m/
Backpatch:
---
src/include/storage/aio.h | 2 +
src/include/storage/bufmgr.h | 2 +
src/backend/storage/aio/aio_callback.c | 2 +
src/backend/storage/buffer/bufmgr.c | 180 +++++++++++++++++++++++++
4 files changed, 186 insertions(+)
diff --git a/src/include/storage/aio.h b/src/include/storage/aio.h
index c673627a358..2eeabb8e01a 100644
--- a/src/include/storage/aio.h
+++ b/src/include/storage/aio.h
@@ -192,8 +192,10 @@ typedef enum PgAioHandleCallbackID
PGAIO_HCB_MD_WRITEV,
PGAIO_HCB_SHARED_BUFFER_READV,
+ PGAIO_HCB_SHARED_BUFFER_WRITEV,
PGAIO_HCB_LOCAL_BUFFER_READV,
+ PGAIO_HCB_LOCAL_BUFFER_WRITEV,
} PgAioHandleCallbackID;
diff --git a/src/include/storage/bufmgr.h b/src/include/storage/bufmgr.h
index 7f709fd7ea4..2cfa5736f4b 100644
--- a/src/include/storage/bufmgr.h
+++ b/src/include/storage/bufmgr.h
@@ -170,7 +170,9 @@ extern PGDLLIMPORT int backend_flush_after;
extern PGDLLIMPORT int bgwriter_flush_after;
extern const PgAioHandleCallbacks aio_shared_buffer_readv_cb;
+extern const PgAioHandleCallbacks aio_shared_buffer_writev_cb;
extern const PgAioHandleCallbacks aio_local_buffer_readv_cb;
+extern const PgAioHandleCallbacks aio_local_buffer_writev_cb;
/* in buf_init.c */
extern PGDLLIMPORT char *BufferBlocks;
diff --git a/src/backend/storage/aio/aio_callback.c b/src/backend/storage/aio/aio_callback.c
index d1ea1ee145b..d7c261a964c 100644
--- a/src/backend/storage/aio/aio_callback.c
+++ b/src/backend/storage/aio/aio_callback.c
@@ -44,8 +44,10 @@ static const PgAioHandleCallbacksEntry aio_handle_cbs[] = {
CALLBACK_ENTRY(PGAIO_HCB_MD_WRITEV, aio_md_writev_cb),
CALLBACK_ENTRY(PGAIO_HCB_SHARED_BUFFER_READV, aio_shared_buffer_readv_cb),
+ CALLBACK_ENTRY(PGAIO_HCB_SHARED_BUFFER_WRITEV, aio_shared_buffer_writev_cb),
CALLBACK_ENTRY(PGAIO_HCB_LOCAL_BUFFER_READV, aio_local_buffer_readv_cb),
+ CALLBACK_ENTRY(PGAIO_HCB_LOCAL_BUFFER_WRITEV, aio_local_buffer_writev_cb),
#undef CALLBACK_ENTRY
};
diff --git a/src/backend/storage/buffer/bufmgr.c b/src/backend/storage/buffer/bufmgr.c
index 7259b0b3bb6..4ed342eeb3f 100644
--- a/src/backend/storage/buffer/bufmgr.c
+++ b/src/backend/storage/buffer/bufmgr.c
@@ -5506,7 +5506,15 @@ LockBuffer(Buffer buffer, int mode)
else if (mode == BUFFER_LOCK_SHARE)
LWLockAcquire(BufferDescriptorGetContentLock(buf), LW_SHARED);
else if (mode == BUFFER_LOCK_EXCLUSIVE)
+ {
+ /*
+ * FIXME: Wait for AIO writes, otherwise there would be a risk of
+ * deadlock. This isn't entirely trivial to do in a race-free way, IO
+ * could be started between us checking whether there is IO and
+ * enqueueing ourselves for the lock.
+ */
LWLockAcquire(BufferDescriptorGetContentLock(buf), LW_EXCLUSIVE);
+ }
else
elog(ERROR, "unrecognized buffer lock mode: %d", mode);
}
@@ -5521,6 +5529,19 @@ ConditionalLockBuffer(Buffer buffer)
{
BufferDesc *buf;
+ /*
+ * FIXME: Wait for AIO writes. Some code does not deal well
+ * ConditionalLockBuffer() continuously failing, e.g.
+ * spginsert()->spgdoinsert() ends up busy-looping (easily reproducible by
+ * just making this function always fail and running the regression
+ * tests). While that code could be fixed, it'd be hard to find all
+ * problematic places.
+ *
+ * It would be OK to wait for the IO as waiting for IO completion does not
+ * need to wait for any locks that could lead to an undetected deadlock or
+ * such.
+ */
+
Assert(BufferIsPinned(buffer));
if (BufferIsLocal(buffer))
return true; /* act as though we got it */
@@ -5593,6 +5614,11 @@ LockBufferForCleanup(Buffer buffer)
{
uint32 buf_state;
+ /*
+ * FIXME: LockBuffer()'s handling of in-progress writes (once
+ * implemented) should suffice to deal with deadlock risk.
+ */
+
/* Try to acquire lock */
LockBuffer(buffer, BUFFER_LOCK_EXCLUSIVE);
buf_state = LockBufHdr(bufHdr);
@@ -5740,6 +5766,14 @@ ConditionalLockBufferForCleanup(Buffer buffer)
Assert(BufferIsValid(buffer));
+ /*
+ * FIXME: Should wait for IO for the same reason as in
+ * ConditionalLockBuffer(). Needs to happen before the
+ * ConditionalLockBuffer() call below, as we'd never reach the
+ * ConditionalLockBuffer() call due the buffer pin held for the duration
+ * of the IO.
+ */
+
if (BufferIsLocal(buffer))
{
refcount = LocalRefCount[-buffer - 1];
@@ -6801,12 +6835,131 @@ buffer_readv_report(PgAioResult result, const PgAioTargetData *target_data, int
);
}
+/*
+ * Helper for AIO writev completion callbacks, supporting both shared and temp
+ * buffers. Gets called once for each buffer in a multi-page write.
+ */
+static pg_attribute_always_inline PgAioResult
+buffer_writev_complete_one(uint8 buf_off, Buffer buffer, uint8 flags,
+ bool failed, bool is_temp)
+{
+ BufferDesc *buf_hdr = is_temp ?
+ GetLocalBufferDescriptor(-buffer - 1)
+ : GetBufferDescriptor(buffer - 1);
+ PgAioResult result;
+ bool clear_dirty;
+ uint32 set_flag_bits;
+
+#ifdef USE_ASSERT_CHECKING
+ {
+ uint32 buf_state = pg_atomic_read_u32(&buf_hdr->state);
+
+ Assert(buf_state & BM_VALID);
+ Assert(buf_state & BM_TAG_VALID);
+ /* temp buffers don't use BM_IO_IN_PROGRESS */
+ if (!is_temp)
+ Assert(buf_state & BM_IO_IN_PROGRESS);
+ Assert(buf_state & BM_DIRTY);
+ }
+#endif
+
+ result.status = ARS_OK;
+
+ clear_dirty = failed ? false : true;
+ set_flag_bits = failed ? BM_IO_ERROR : 0;
+
+ if (is_temp)
+ TerminateLocalBufferIO(buf_hdr, clear_dirty, set_flag_bits, false);
+ else
+ TerminateBufferIO(buf_hdr, clear_dirty, set_flag_bits, false, false);
+
+ /*
+ * The initiator of IO is not managing the lock (i.e. we called
+ * LWLockDisown()), we are.
+ */
+ if (!is_temp)
+ LWLockReleaseDisowned(BufferDescriptorGetContentLock(buf_hdr),
+ LW_SHARED);
+
+ /* FIXME: tracepoint */
+
+ return result;
+}
+
+/*
+ * Perform completion handling of a single AIO write. This write may cover
+ * multiple blocks / buffers.
+ *
+ * Shared between shared and local buffers, to reduce code duplication.
+ */
+static pg_attribute_always_inline PgAioResult
+buffer_writev_complete(PgAioHandle *ioh, PgAioResult prior_result,
+ uint8 cb_data, bool is_temp)
+{
+ PgAioResult result = prior_result;
+ PgAioTargetData *td = pgaio_io_get_target_data(ioh);
+ uint64 *io_data;
+ uint8 handle_data_len;
+
+ if (is_temp)
+ {
+ Assert(td->smgr.is_temp);
+ Assert(pgaio_io_get_owner(ioh) == MyProcNumber);
+ }
+ else
+ Assert(!td->smgr.is_temp);
+
+ /*
+ * Iterate over all the buffers affected by this IO and call appropriate
+ * per-buffer completion function for each buffer.
+ */
+ io_data = pgaio_io_get_handle_data(ioh, &handle_data_len);
+ for (uint8 buf_off = 0; buf_off < handle_data_len; buf_off++)
+ {
+ Buffer buf = io_data[buf_off];
+ PgAioResult buf_result;
+ bool failed;
+
+ Assert(BufferIsValid(buf));
+
+ /*
+ * If the entire failed on a lower-level, each buffer needs to be
+ * marked as failed. In case of a partial read, some buffers may be
+ * ok.
+ */
+ failed =
+ prior_result.status == ARS_ERROR
+ || prior_result.result <= buf_off;
+
+ buf_result = buffer_writev_complete_one(buf_off, buf, cb_data, failed,
+ is_temp);
+
+ /*
+ * If there wasn't any prior error and the IO for this page failed in
+ * some form, set the whole IO's to the page's result.
+ */
+ if (result.status != ARS_ERROR && buf_result.status != ARS_OK)
+ {
+ result = buf_result;
+ pgaio_result_report(result, td, LOG);
+ }
+ }
+
+ return result;
+}
+
static void
shared_buffer_readv_stage(PgAioHandle *ioh, uint8 cb_data)
{
buffer_stage_common(ioh, false, false);
}
+static void
+shared_buffer_writev_stage(PgAioHandle *ioh, uint8 cb_data)
+{
+ buffer_stage_common(ioh, true, false);
+}
+
static PgAioResult
shared_buffer_readv_complete(PgAioHandle *ioh, PgAioResult prior_result,
uint8 cb_data)
@@ -6814,6 +6967,13 @@ shared_buffer_readv_complete(PgAioHandle *ioh, PgAioResult prior_result,
return buffer_readv_complete(ioh, prior_result, cb_data, false);
}
+static PgAioResult
+shared_buffer_writev_complete(PgAioHandle *ioh, PgAioResult prior_result,
+ uint8 cb_data)
+{
+ return buffer_writev_complete(ioh, prior_result, cb_data, false);
+}
+
static void
local_buffer_readv_stage(PgAioHandle *ioh, uint8 cb_data)
{
@@ -6827,6 +6987,17 @@ local_buffer_readv_complete(PgAioHandle *ioh, PgAioResult prior_result,
return buffer_readv_complete(ioh, prior_result, cb_data, true);
}
+static void
+local_buffer_writev_stage(PgAioHandle *ioh, uint8 cb_data)
+{
+ /*
+ * Currently this is unreachable as the only write support is for
+ * checkpointer / bgwriter, which don't deal with local buffers.
+ */
+ elog(ERROR, "should be unreachable");
+}
+
+
/* readv callback is passed READ_BUFFERS_* flags as callback data */
const PgAioHandleCallbacks aio_shared_buffer_readv_cb = {
.stage = shared_buffer_readv_stage,
@@ -6834,6 +7005,11 @@ const PgAioHandleCallbacks aio_shared_buffer_readv_cb = {
.report = buffer_readv_report,
};
+const PgAioHandleCallbacks aio_shared_buffer_writev_cb = {
+ .stage = shared_buffer_writev_stage,
+ .complete_shared = shared_buffer_writev_complete,
+};
+
/* readv callback is passed READ_BUFFERS_* flags as callback data */
const PgAioHandleCallbacks aio_local_buffer_readv_cb = {
.stage = local_buffer_readv_stage,
@@ -6847,3 +7023,7 @@ const PgAioHandleCallbacks aio_local_buffer_readv_cb = {
.complete_local = local_buffer_readv_complete,
.report = buffer_readv_report,
};
+
+const PgAioHandleCallbacks aio_local_buffer_writev_cb = {
+ .stage = local_buffer_writev_stage,
+};
--
2.48.1.76.g4e746b1a31.dirty
[text/x-diff] v2.10-0021-aio-Add-IO-queue-helper.patch (7.4K, ../../zp7obmaqdd6b5u2vhpw27hovpjirtqvjakkbkaz4vju7tohyul@4kfobe7ipgxm/22-v2.10-0021-aio-Add-IO-queue-helper.patch)
download | inline diff:
From 4e840b3c7ec31e8a53ab87526287a6744540dd34 Mon Sep 17 00:00:00 2001
From: Andres Freund <andres@anarazel.de>
Date: Tue, 18 Mar 2025 14:40:06 -0400
Subject: [PATCH v2.10 21/28] aio: Add IO queue helper
This is likely never going to anywhere - Thomas Munro is working on something
more complete. But I needed a way to exercise aio for checkpointer / bgwriter.
---
src/include/storage/io_queue.h | 33 +++++
src/backend/storage/aio/Makefile | 1 +
src/backend/storage/aio/io_queue.c | 204 ++++++++++++++++++++++++++++
src/backend/storage/aio/meson.build | 1 +
src/tools/pgindent/typedefs.list | 2 +
5 files changed, 241 insertions(+)
create mode 100644 src/include/storage/io_queue.h
create mode 100644 src/backend/storage/aio/io_queue.c
diff --git a/src/include/storage/io_queue.h b/src/include/storage/io_queue.h
new file mode 100644
index 00000000000..92b1e9afe6f
--- /dev/null
+++ b/src/include/storage/io_queue.h
@@ -0,0 +1,33 @@
+/*-------------------------------------------------------------------------
+ *
+ * io_queue.h
+ * Mechanism for tracking many IOs
+ *
+ *
+ * Portions Copyright (c) 1996-2025, PostgreSQL Global Development Group
+ * Portions Copyright (c) 1994, Regents of the University of California
+ *
+ * src/include/storage/io_queue.h
+ *
+ *-------------------------------------------------------------------------
+ */
+#ifndef IO_QUEUE_H
+#define IO_QUEUE_H
+
+#include "storage/aio_types.h"
+
+struct IOQueue;
+typedef struct IOQueue IOQueue;
+
+struct PgAioWaitRef;
+
+extern IOQueue *io_queue_create(int depth, int flags);
+extern void io_queue_track(IOQueue *ioq, const PgAioWaitRef *iow);
+extern void io_queue_wait_one(IOQueue *ioq);
+extern void io_queue_wait_all(IOQueue *ioq);
+extern bool io_queue_is_empty(IOQueue *ioq);
+extern void io_queue_reserve(IOQueue *ioq);
+extern PgAioHandle *io_queue_acquire_io(IOQueue *ioq);
+extern void io_queue_free(IOQueue *ioq);
+
+#endif /* IO_QUEUE_H */
diff --git a/src/backend/storage/aio/Makefile b/src/backend/storage/aio/Makefile
index 3f2469cc399..86fa4276fda 100644
--- a/src/backend/storage/aio/Makefile
+++ b/src/backend/storage/aio/Makefile
@@ -15,6 +15,7 @@ OBJS = \
aio_init.o \
aio_io.o \
aio_target.o \
+ io_queue.o \
method_io_uring.o \
method_sync.o \
method_worker.o \
diff --git a/src/backend/storage/aio/io_queue.c b/src/backend/storage/aio/io_queue.c
new file mode 100644
index 00000000000..526aa1d5e06
--- /dev/null
+++ b/src/backend/storage/aio/io_queue.c
@@ -0,0 +1,204 @@
+/*-------------------------------------------------------------------------
+ *
+ * io_queue.c
+ * AIO - Mechanism for tracking many IOs
+ *
+ * Portions Copyright (c) 1996-2025, PostgreSQL Global Development Group
+ * Portions Copyright (c) 1994, Regents of the University of California
+ *
+ * IDENTIFICATION
+ * src/backend/storage/aio/io_queue.c
+ *
+ *-------------------------------------------------------------------------
+ */
+
+#include "postgres.h"
+
+#include "lib/ilist.h"
+#include "storage/aio.h"
+#include "storage/io_queue.h"
+#include "utils/resowner.h"
+
+
+
+typedef struct TrackedIO
+{
+ PgAioWaitRef iow;
+ dlist_node node;
+} TrackedIO;
+
+struct IOQueue
+{
+ int depth;
+ int unsubmitted;
+
+ bool has_reserved;
+
+ dclist_head idle;
+ dclist_head in_progress;
+
+ TrackedIO tracked_ios[FLEXIBLE_ARRAY_MEMBER];
+};
+
+
+IOQueue *
+io_queue_create(int depth, int flags)
+{
+ size_t sz;
+ IOQueue *ioq;
+
+ sz = offsetof(IOQueue, tracked_ios)
+ + sizeof(TrackedIO) * depth;
+
+ ioq = palloc0(sz);
+
+ ioq->depth = 0;
+
+ for (int i = 0; i < depth; i++)
+ {
+ TrackedIO *tio = &ioq->tracked_ios[i];
+
+ pgaio_wref_clear(&tio->iow);
+ dclist_push_tail(&ioq->idle, &tio->node);
+ }
+
+ return ioq;
+}
+
+void
+io_queue_wait_one(IOQueue *ioq)
+{
+ /* submit all pending IO before waiting */
+ pgaio_submit_staged();
+
+ while (!dclist_is_empty(&ioq->in_progress))
+ {
+ /* FIXME: Should we really pop here already? */
+ dlist_node *node = dclist_pop_head_node(&ioq->in_progress);
+ TrackedIO *tio = dclist_container(TrackedIO, node, node);
+
+ pgaio_wref_wait(&tio->iow);
+ dclist_push_head(&ioq->idle, &tio->node);
+ }
+}
+
+void
+io_queue_reserve(IOQueue *ioq)
+{
+ if (ioq->has_reserved)
+ return;
+
+ if (dclist_is_empty(&ioq->idle))
+ io_queue_wait_one(ioq);
+
+ Assert(!dclist_is_empty(&ioq->idle));
+
+ ioq->has_reserved = true;
+}
+
+PgAioHandle *
+io_queue_acquire_io(IOQueue *ioq)
+{
+ PgAioHandle *ioh;
+
+ io_queue_reserve(ioq);
+
+ Assert(!dclist_is_empty(&ioq->idle));
+
+ if (!io_queue_is_empty(ioq))
+ {
+ ioh = pgaio_io_acquire_nb(CurrentResourceOwner, NULL);
+ if (ioh == NULL)
+ {
+ /*
+ * Need to wait for all IOs, blocking might not be legal in the
+ * context.
+ *
+ * XXX: This doesn't make a whole lot of sense, we're also
+ * blocking here. What was I smoking when I wrote the above?
+ */
+ io_queue_wait_all(ioq);
+ ioh = pgaio_io_acquire(CurrentResourceOwner, NULL);
+ }
+ }
+ else
+ {
+ ioh = pgaio_io_acquire(CurrentResourceOwner, NULL);
+ }
+
+ return ioh;
+}
+
+void
+io_queue_track(IOQueue *ioq, const struct PgAioWaitRef *iow)
+{
+ dlist_node *node;
+ TrackedIO *tio;
+
+ Assert(ioq->has_reserved);
+ ioq->has_reserved = false;
+
+ Assert(!dclist_is_empty(&ioq->idle));
+
+ node = dclist_pop_head_node(&ioq->idle);
+ tio = dclist_container(TrackedIO, node, node);
+
+ tio->iow = *iow;
+
+ dclist_push_tail(&ioq->in_progress, &tio->node);
+
+ ioq->unsubmitted++;
+
+ /*
+ * XXX: Should have some smarter logic here. We don't want to wait too
+ * long to submit, that'll mean we're more likely to block. But we also
+ * don't want to have the overhead of submitting every IO individually.
+ */
+ if (ioq->unsubmitted >= 4)
+ {
+ pgaio_submit_staged();
+ ioq->unsubmitted = 0;
+ }
+}
+
+void
+io_queue_wait_all(IOQueue *ioq)
+{
+ /* submit all pending IO before waiting */
+ pgaio_submit_staged();
+
+ while (!dclist_is_empty(&ioq->in_progress))
+ {
+ /* wait for the last IO to minimize unnecessary wakeups */
+ dlist_node *node = dclist_tail_node(&ioq->in_progress);
+ TrackedIO *tio = dclist_container(TrackedIO, node, node);
+
+ if (!pgaio_wref_check_done(&tio->iow))
+ {
+ ereport(DEBUG3,
+ errmsg("io_queue_wait_all for io:%d",
+ pgaio_wref_get_id(&tio->iow)),
+ errhidestmt(true),
+ errhidecontext(true));
+
+ pgaio_wref_wait(&tio->iow);
+ }
+
+ dclist_delete_from(&ioq->in_progress, &tio->node);
+ dclist_push_head(&ioq->idle, &tio->node);
+ }
+}
+
+bool
+io_queue_is_empty(IOQueue *ioq)
+{
+ return dclist_is_empty(&ioq->in_progress);
+}
+
+void
+io_queue_free(IOQueue *ioq)
+{
+ io_queue_wait_all(ioq);
+
+ pfree(ioq);
+}
diff --git a/src/backend/storage/aio/meson.build b/src/backend/storage/aio/meson.build
index da6df2d3654..270c4a64428 100644
--- a/src/backend/storage/aio/meson.build
+++ b/src/backend/storage/aio/meson.build
@@ -7,6 +7,7 @@ backend_sources += files(
'aio_init.c',
'aio_io.c',
'aio_target.c',
+ 'io_queue.c',
'method_io_uring.c',
'method_sync.c',
'method_worker.c',
diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list
index d633c732162..9c07fa04b49 100644
--- a/src/tools/pgindent/typedefs.list
+++ b/src/tools/pgindent/typedefs.list
@@ -1191,6 +1191,7 @@ IOContext
IOFuncSelector
IOObject
IOOp
+IOQueue
IO_STATUS_BLOCK
IPCompareMethod
ITEM
@@ -3016,6 +3017,7 @@ TocEntry
TokenAuxData
TokenizedAuthLine
TrackItem
+TrackedIO
TransApplyAction
TransInvalidationInfo
TransState
--
2.48.1.76.g4e746b1a31.dirty
[text/x-diff] v2.10-0022-bufmgr-use-AIO-in-checkpointer-bgwriter.patch (31.1K, ../../zp7obmaqdd6b5u2vhpw27hovpjirtqvjakkbkaz4vju7tohyul@4kfobe7ipgxm/23-v2.10-0022-bufmgr-use-AIO-in-checkpointer-bgwriter.patch)
download | inline diff:
From 6b684d809f160325c5eb38774c72552b5f198b23 Mon Sep 17 00:00:00 2001
From: Andres Freund <andres@anarazel.de>
Date: Tue, 18 Mar 2025 14:40:06 -0400
Subject: [PATCH v2.10 22/28] bufmgr: use AIO in checkpointer, bgwriter
This is far from ready - just included to be able to exercise AIO writes and
get some preliminary numbers. In all likelihood this will instead be based
on-top of work by Thomas Munro instead of the preceding commit.
---
src/include/postmaster/bgwriter.h | 3 +-
src/include/storage/buf_internals.h | 2 +
src/include/storage/bufmgr.h | 3 +-
src/include/storage/bufpage.h | 1 +
src/backend/postmaster/bgwriter.c | 19 +-
src/backend/postmaster/checkpointer.c | 11 +-
src/backend/storage/buffer/bufmgr.c | 588 +++++++++++++++++++++++---
src/backend/storage/page/bufpage.c | 10 +
src/tools/pgindent/typedefs.list | 1 +
9 files changed, 580 insertions(+), 58 deletions(-)
diff --git a/src/include/postmaster/bgwriter.h b/src/include/postmaster/bgwriter.h
index 800ecbfd13b..a8081d411b6 100644
--- a/src/include/postmaster/bgwriter.h
+++ b/src/include/postmaster/bgwriter.h
@@ -31,7 +31,8 @@ pg_noreturn extern void BackgroundWriterMain(const void *startup_data, size_t st
pg_noreturn extern void CheckpointerMain(const void *startup_data, size_t startup_data_len);
extern void RequestCheckpoint(int flags);
-extern void CheckpointWriteDelay(int flags, double progress);
+struct IOQueue;
+extern void CheckpointWriteDelay(struct IOQueue *ioq, int flags, double progress);
extern bool ForwardSyncRequest(const FileTag *ftag, SyncRequestType type);
diff --git a/src/include/storage/buf_internals.h b/src/include/storage/buf_internals.h
index 4b1f52a9fc8..a3df3192d12 100644
--- a/src/include/storage/buf_internals.h
+++ b/src/include/storage/buf_internals.h
@@ -21,6 +21,8 @@
#include "storage/buf.h"
#include "storage/bufmgr.h"
#include "storage/condition_variable.h"
+#include "storage/io_queue.h"
+#include "storage/latch.h"
#include "storage/lwlock.h"
#include "storage/procnumber.h"
#include "storage/shmem.h"
diff --git a/src/include/storage/bufmgr.h b/src/include/storage/bufmgr.h
index 2cfa5736f4b..025167b2118 100644
--- a/src/include/storage/bufmgr.h
+++ b/src/include/storage/bufmgr.h
@@ -293,7 +293,8 @@ extern bool ConditionalLockBufferForCleanup(Buffer buffer);
extern bool IsBufferCleanupOK(Buffer buffer);
extern bool HoldingBufferPinThatDelaysRecovery(void);
-extern bool BgBufferSync(struct WritebackContext *wb_context);
+struct IOQueue;
+extern bool BgBufferSync(struct IOQueue *ioq, struct WritebackContext *wb_context);
extern uint32 GetPinLimit(void);
extern uint32 GetLocalPinLimit(void);
diff --git a/src/include/storage/bufpage.h b/src/include/storage/bufpage.h
index 6646b6f6371..9c045e81857 100644
--- a/src/include/storage/bufpage.h
+++ b/src/include/storage/bufpage.h
@@ -509,5 +509,6 @@ extern bool PageIndexTupleOverwrite(Page page, OffsetNumber offnum,
Item newtup, Size newsize);
extern char *PageSetChecksumCopy(Page page, BlockNumber blkno);
extern void PageSetChecksumInplace(Page page, BlockNumber blkno);
+extern bool PageNeedsChecksumCopy(Page page);
#endif /* BUFPAGE_H */
diff --git a/src/backend/postmaster/bgwriter.c b/src/backend/postmaster/bgwriter.c
index 72f5acceec7..6e8801a39e3 100644
--- a/src/backend/postmaster/bgwriter.c
+++ b/src/backend/postmaster/bgwriter.c
@@ -38,11 +38,13 @@
#include "postmaster/auxprocess.h"
#include "postmaster/bgwriter.h"
#include "postmaster/interrupt.h"
+#include "storage/aio.h"
#include "storage/aio_subsys.h"
#include "storage/buf_internals.h"
#include "storage/bufmgr.h"
#include "storage/condition_variable.h"
#include "storage/fd.h"
+#include "storage/io_queue.h"
#include "storage/lwlock.h"
#include "storage/proc.h"
#include "storage/procsignal.h"
@@ -90,6 +92,7 @@ BackgroundWriterMain(const void *startup_data, size_t startup_data_len)
sigjmp_buf local_sigjmp_buf;
MemoryContext bgwriter_context;
bool prev_hibernate;
+ IOQueue *ioq;
WritebackContext wb_context;
Assert(startup_data_len == 0);
@@ -131,6 +134,7 @@ BackgroundWriterMain(const void *startup_data, size_t startup_data_len)
ALLOCSET_DEFAULT_SIZES);
MemoryContextSwitchTo(bgwriter_context);
+ ioq = io_queue_create(128, 0);
WritebackContextInit(&wb_context, &bgwriter_flush_after);
/*
@@ -228,12 +232,22 @@ BackgroundWriterMain(const void *startup_data, size_t startup_data_len)
/* Clear any already-pending wakeups */
ResetLatch(MyLatch);
+ /*
+ * FIXME: this is theoretically racy, but I didn't want to copy
+ * ProcessMainLoopInterrupts() remaining body here.
+ */
+ if (ShutdownRequestPending)
+ {
+ io_queue_wait_all(ioq);
+ io_queue_free(ioq);
+ }
+
ProcessMainLoopInterrupts();
/*
* Do one cycle of dirty-buffer writing.
*/
- can_hibernate = BgBufferSync(&wb_context);
+ can_hibernate = BgBufferSync(ioq, &wb_context);
/* Report pending statistics to the cumulative stats system */
pgstat_report_bgwriter();
@@ -250,6 +264,9 @@ BackgroundWriterMain(const void *startup_data, size_t startup_data_len)
smgrdestroyall();
}
+ /* finish IO before sleeping, to avoid blocking other backends */
+ io_queue_wait_all(ioq);
+
/*
* Log a new xl_running_xacts every now and then so replication can
* get into a consistent state faster (think of suboverflowed
diff --git a/src/backend/postmaster/checkpointer.c b/src/backend/postmaster/checkpointer.c
index fda91ffd1ce..904fe167eb4 100644
--- a/src/backend/postmaster/checkpointer.c
+++ b/src/backend/postmaster/checkpointer.c
@@ -49,10 +49,12 @@
#include "postmaster/bgwriter.h"
#include "postmaster/interrupt.h"
#include "replication/syncrep.h"
+#include "storage/aio.h"
#include "storage/aio_subsys.h"
#include "storage/bufmgr.h"
#include "storage/condition_variable.h"
#include "storage/fd.h"
+#include "storage/io_queue.h"
#include "storage/ipc.h"
#include "storage/lwlock.h"
#include "storage/pmsignal.h"
@@ -766,7 +768,7 @@ ImmediateCheckpointRequested(void)
* fraction between 0.0 meaning none, and 1.0 meaning all done.
*/
void
-CheckpointWriteDelay(int flags, double progress)
+CheckpointWriteDelay(IOQueue *ioq, int flags, double progress)
{
static int absorb_counter = WRITES_PER_ABSORB;
@@ -800,6 +802,13 @@ CheckpointWriteDelay(int flags, double progress)
/* Report interim statistics to the cumulative stats system */
pgstat_report_checkpointer();
+ /*
+ * Ensure all pending IO is submitted to avoid unnecessary delays for
+ * other processes.
+ */
+ io_queue_wait_all(ioq);
+
+
/*
* This sleep used to be connected to bgwriter_delay, typically 200ms.
* That resulted in more frequent wakeups if not much work to do.
diff --git a/src/backend/storage/buffer/bufmgr.c b/src/backend/storage/buffer/bufmgr.c
index 4ed342eeb3f..306267c068a 100644
--- a/src/backend/storage/buffer/bufmgr.c
+++ b/src/backend/storage/buffer/bufmgr.c
@@ -52,6 +52,7 @@
#include "storage/buf_internals.h"
#include "storage/bufmgr.h"
#include "storage/fd.h"
+#include "storage/io_queue.h"
#include "storage/ipc.h"
#include "storage/lmgr.h"
#include "storage/proc.h"
@@ -76,6 +77,7 @@
/* Bits in SyncOneBuffer's return value */
#define BUF_WRITTEN 0x01
#define BUF_REUSABLE 0x02
+#define BUF_CANT_MERGE 0x04
#define RELS_BSEARCH_THRESHOLD 20
@@ -512,8 +514,6 @@ static void UnpinBuffer(BufferDesc *buf);
static void UnpinBufferNoOwner(BufferDesc *buf);
static void BufferSync(int flags);
static uint32 WaitBufHdrUnlocked(BufferDesc *buf);
-static int SyncOneBuffer(int buf_id, bool skip_recently_used,
- WritebackContext *wb_context);
static void WaitIO(BufferDesc *buf);
static void AbortBufferIO(Buffer buffer);
static void shared_buffer_write_error_callback(void *arg);
@@ -529,6 +529,7 @@ static void CheckReadBuffersOperation(ReadBuffersOperation *operation, bool is_c
static Buffer GetVictimBuffer(BufferAccessStrategy strategy, IOContext io_context);
static void FlushBuffer(BufferDesc *buf, SMgrRelation reln,
IOObject io_object, IOContext io_context);
+
static void FindAndDropRelationBuffers(RelFileLocator rlocator,
ForkNumber forkNum,
BlockNumber nForkBlock,
@@ -3298,6 +3299,57 @@ UnpinBufferNoOwner(BufferDesc *buf)
}
}
+typedef struct BuffersToWrite
+{
+ int nbuffers;
+ BufferTag start_at_tag;
+ uint32 max_combine;
+
+ XLogRecPtr max_lsn;
+
+ PgAioHandle *ioh;
+ PgAioWaitRef iow;
+
+ uint64 total_writes;
+
+ Buffer buffers[IOV_MAX];
+ PgAioBounceBuffer *bounce_buffers[IOV_MAX];
+ const void *data_ptrs[IOV_MAX];
+} BuffersToWrite;
+
+static int PrepareToWriteBuffer(BuffersToWrite *to_write, Buffer buf,
+ bool skip_recently_used,
+ IOQueue *ioq, WritebackContext *wb_context);
+
+static void WriteBuffers(BuffersToWrite *to_write,
+ IOQueue *ioq, WritebackContext *wb_context);
+
+static void
+BuffersToWriteInit(BuffersToWrite *to_write,
+ IOQueue *ioq, WritebackContext *wb_context)
+{
+ to_write->total_writes = 0;
+ to_write->nbuffers = 0;
+ to_write->ioh = NULL;
+ pgaio_wref_clear(&to_write->iow);
+ to_write->max_lsn = InvalidXLogRecPtr;
+
+ pgaio_enter_batchmode();
+}
+
+static void
+BuffersToWriteEnd(BuffersToWrite *to_write)
+{
+ if (to_write->ioh != NULL)
+ {
+ pgaio_io_release(to_write->ioh);
+ to_write->ioh = NULL;
+ }
+
+ pgaio_exit_batchmode();
+}
+
+
#define ST_SORT sort_checkpoint_bufferids
#define ST_ELEMENT_TYPE CkptSortItem
#define ST_COMPARE(a, b) ckpt_buforder_comparator(a, b)
@@ -3329,7 +3381,10 @@ BufferSync(int flags)
binaryheap *ts_heap;
int i;
int mask = BM_DIRTY;
+ IOQueue *ioq;
WritebackContext wb_context;
+ BuffersToWrite to_write;
+ int max_combine;
/*
* Unless this is a shutdown checkpoint or we have been explicitly told,
@@ -3391,7 +3446,9 @@ BufferSync(int flags)
if (num_to_scan == 0)
return; /* nothing to do */
+ ioq = io_queue_create(512, 0);
WritebackContextInit(&wb_context, &checkpoint_flush_after);
+ max_combine = Min(io_bounce_buffers, io_combine_limit);
TRACE_POSTGRESQL_BUFFER_SYNC_START(NBuffers, num_to_scan);
@@ -3499,48 +3556,91 @@ BufferSync(int flags)
*/
num_processed = 0;
num_written = 0;
+
+ BuffersToWriteInit(&to_write, ioq, &wb_context);
+
while (!binaryheap_empty(ts_heap))
{
BufferDesc *bufHdr = NULL;
CkptTsStatus *ts_stat = (CkptTsStatus *)
DatumGetPointer(binaryheap_first(ts_heap));
+ bool batch_continue = true;
- buf_id = CkptBufferIds[ts_stat->index].buf_id;
- Assert(buf_id != -1);
-
- bufHdr = GetBufferDescriptor(buf_id);
-
- num_processed++;
+ Assert(ts_stat->num_scanned <= ts_stat->num_to_scan);
/*
- * We don't need to acquire the lock here, because we're only looking
- * at a single bit. It's possible that someone else writes the buffer
- * and clears the flag right after we check, but that doesn't matter
- * since SyncOneBuffer will then do nothing. However, there is a
- * further race condition: it's conceivable that between the time we
- * examine the bit here and the time SyncOneBuffer acquires the lock,
- * someone else not only wrote the buffer but replaced it with another
- * page and dirtied it. In that improbable case, SyncOneBuffer will
- * write the buffer though we didn't need to. It doesn't seem worth
- * guarding against this, though.
+ * Collect a batch of buffers to write out from the current
+ * tablespace. That causes some imbalance between the tablespaces, but
+ * that's more than outweighed by the efficiency gain due to batching.
*/
- if (pg_atomic_read_u32(&bufHdr->state) & BM_CHECKPOINT_NEEDED)
+ while (batch_continue &&
+ to_write.nbuffers < max_combine &&
+ ts_stat->num_scanned < ts_stat->num_to_scan)
{
- if (SyncOneBuffer(buf_id, false, &wb_context) & BUF_WRITTEN)
+ buf_id = CkptBufferIds[ts_stat->index].buf_id;
+ Assert(buf_id != -1);
+
+ bufHdr = GetBufferDescriptor(buf_id);
+
+ num_processed++;
+
+ /*
+ * We don't need to acquire the lock here, because we're only
+ * looking at a single bit. It's possible that someone else writes
+ * the buffer and clears the flag right after we check, but that
+ * doesn't matter since SyncOneBuffer will then do nothing.
+ * However, there is a further race condition: it's conceivable
+ * that between the time we examine the bit here and the time
+ * SyncOneBuffer acquires the lock, someone else not only wrote
+ * the buffer but replaced it with another page and dirtied it. In
+ * that improbable case, SyncOneBuffer will write the buffer
+ * though we didn't need to. It doesn't seem worth guarding
+ * against this, though.
+ */
+ if (pg_atomic_read_u32(&bufHdr->state) & BM_CHECKPOINT_NEEDED)
{
- TRACE_POSTGRESQL_BUFFER_SYNC_WRITTEN(buf_id);
- PendingCheckpointerStats.buffers_written++;
- num_written++;
+ int result = PrepareToWriteBuffer(&to_write, buf_id + 1, false,
+ ioq, &wb_context);
+
+ if (result & BUF_CANT_MERGE)
+ {
+ Assert(to_write.nbuffers > 0);
+ WriteBuffers(&to_write, ioq, &wb_context);
+
+ result = PrepareToWriteBuffer(&to_write, buf_id + 1, false,
+ ioq, &wb_context);
+ Assert(result != BUF_CANT_MERGE);
+ }
+
+ if (result & BUF_WRITTEN)
+ {
+ TRACE_POSTGRESQL_BUFFER_SYNC_WRITTEN(buf_id);
+ PendingCheckpointerStats.buffers_written++;
+ num_written++;
+ }
+ else
+ {
+ batch_continue = false;
+ }
}
+ else
+ {
+ if (to_write.nbuffers > 0)
+ WriteBuffers(&to_write, ioq, &wb_context);
+ }
+
+ /*
+ * Measure progress independent of actually having to flush the
+ * buffer - otherwise writing become unbalanced.
+ */
+ ts_stat->progress += ts_stat->progress_slice;
+ ts_stat->num_scanned++;
+ ts_stat->index++;
}
- /*
- * Measure progress independent of actually having to flush the buffer
- * - otherwise writing become unbalanced.
- */
- ts_stat->progress += ts_stat->progress_slice;
- ts_stat->num_scanned++;
- ts_stat->index++;
+ if (to_write.nbuffers > 0)
+ WriteBuffers(&to_write, ioq, &wb_context);
+
/* Have all the buffers from the tablespace been processed? */
if (ts_stat->num_scanned == ts_stat->num_to_scan)
@@ -3558,15 +3658,23 @@ BufferSync(int flags)
*
* (This will check for barrier events even if it doesn't sleep.)
*/
- CheckpointWriteDelay(flags, (double) num_processed / num_to_scan);
+ CheckpointWriteDelay(ioq, flags, (double) num_processed / num_to_scan);
}
+ Assert(to_write.nbuffers == 0);
+ io_queue_wait_all(ioq);
+
/*
* Issue all pending flushes. Only checkpointer calls BufferSync(), so
* IOContext will always be IOCONTEXT_NORMAL.
*/
IssuePendingWritebacks(&wb_context, IOCONTEXT_NORMAL);
+ io_queue_wait_all(ioq); /* IssuePendingWritebacks might have added
+ * more */
+ io_queue_free(ioq);
+ BuffersToWriteEnd(&to_write);
+
pfree(per_ts_stat);
per_ts_stat = NULL;
binaryheap_free(ts_heap);
@@ -3592,7 +3700,7 @@ BufferSync(int flags)
* bgwriter_lru_maxpages to 0.)
*/
bool
-BgBufferSync(WritebackContext *wb_context)
+BgBufferSync(IOQueue *ioq, WritebackContext *wb_context)
{
/* info obtained from freelist.c */
int strategy_buf_id;
@@ -3635,6 +3743,9 @@ BgBufferSync(WritebackContext *wb_context)
long new_strategy_delta;
uint32 new_recent_alloc;
+ BuffersToWrite to_write;
+ int max_combine;
+
/*
* Find out where the freelist clock sweep currently is, and how many
* buffer allocations have happened since our last call.
@@ -3655,6 +3766,8 @@ BgBufferSync(WritebackContext *wb_context)
return true;
}
+ max_combine = Min(io_bounce_buffers, io_combine_limit);
+
/*
* Compute strategy_delta = how many buffers have been scanned by the
* clock sweep since last time. If first time through, assume none. Then
@@ -3811,11 +3924,25 @@ BgBufferSync(WritebackContext *wb_context)
num_written = 0;
reusable_buffers = reusable_buffers_est;
+ BuffersToWriteInit(&to_write, ioq, wb_context);
+
/* Execute the LRU scan */
while (num_to_scan > 0 && reusable_buffers < upcoming_alloc_est)
{
- int sync_state = SyncOneBuffer(next_to_clean, true,
- wb_context);
+ int sync_state;
+
+ sync_state = PrepareToWriteBuffer(&to_write, next_to_clean + 1,
+ true, ioq, wb_context);
+ if (sync_state & BUF_CANT_MERGE)
+ {
+ Assert(to_write.nbuffers > 0);
+
+ WriteBuffers(&to_write, ioq, wb_context);
+
+ sync_state = PrepareToWriteBuffer(&to_write, next_to_clean + 1,
+ true, ioq, wb_context);
+ Assert(sync_state != BUF_CANT_MERGE);
+ }
if (++next_to_clean >= NBuffers)
{
@@ -3826,6 +3953,13 @@ BgBufferSync(WritebackContext *wb_context)
if (sync_state & BUF_WRITTEN)
{
+ Assert(sync_state & BUF_REUSABLE);
+
+ if (to_write.nbuffers == max_combine)
+ {
+ WriteBuffers(&to_write, ioq, wb_context);
+ }
+
reusable_buffers++;
if (++num_written >= bgwriter_lru_maxpages)
{
@@ -3837,6 +3971,11 @@ BgBufferSync(WritebackContext *wb_context)
reusable_buffers++;
}
+ if (to_write.nbuffers > 0)
+ WriteBuffers(&to_write, ioq, wb_context);
+
+ BuffersToWriteEnd(&to_write);
+
PendingBgWriterStats.buf_written_clean += num_written;
#ifdef BGW_DEBUG
@@ -3875,8 +4014,66 @@ BgBufferSync(WritebackContext *wb_context)
return (bufs_to_lap == 0 && recent_alloc == 0);
}
+static inline bool
+BufferTagsSameRel(const BufferTag *tag1, const BufferTag *tag2)
+{
+ return (tag1->spcOid == tag2->spcOid) &&
+ (tag1->dbOid == tag2->dbOid) &&
+ (tag1->relNumber == tag2->relNumber) &&
+ (tag1->forkNum == tag2->forkNum)
+ ;
+}
+
+static bool
+CanMergeWrite(BuffersToWrite *to_write, BufferDesc *cur_buf_hdr)
+{
+ BlockNumber cur_block = cur_buf_hdr->tag.blockNum;
+
+ Assert(to_write->nbuffers > 0); /* can't merge with nothing */
+ Assert(to_write->start_at_tag.relNumber != InvalidOid);
+ Assert(to_write->start_at_tag.blockNum != InvalidBlockNumber);
+
+ Assert(to_write->ioh != NULL);
+
+ /*
+ * First check if the blocknumber is one that we could actually merge,
+ * that's cheaper than checking the tablespace/db/relnumber/fork match.
+ */
+ if (to_write->start_at_tag.blockNum + to_write->nbuffers != cur_block)
+ return false;
+
+ if (!BufferTagsSameRel(&to_write->start_at_tag, &cur_buf_hdr->tag))
+ return false;
+
+ /*
+ * Need to check with smgr how large a write we're allowed to make. To
+ * reduce the overhead of the smgr check, only inquire once, when
+ * processing the first to-be-merged buffer. That avoids the overhead in
+ * the common case of writing out buffers that definitely not mergeable.
+ */
+ if (to_write->nbuffers == 1)
+ {
+ SMgrRelation smgr;
+
+ smgr = smgropen(BufTagGetRelFileLocator(&to_write->start_at_tag), INVALID_PROC_NUMBER);
+
+ to_write->max_combine = smgrmaxcombine(smgr,
+ to_write->start_at_tag.forkNum,
+ to_write->start_at_tag.blockNum);
+ }
+ else
+ {
+ Assert(to_write->max_combine > 0);
+ }
+
+ if (to_write->start_at_tag.blockNum + to_write->max_combine <= cur_block)
+ return false;
+
+ return true;
+}
+
/*
- * SyncOneBuffer -- process a single buffer during syncing.
+ * PrepareToWriteBuffer -- process a single buffer during syncing.
*
* If skip_recently_used is true, we don't write currently-pinned buffers, nor
* buffers marked recently used, as these are not replacement candidates.
@@ -3885,22 +4082,50 @@ BgBufferSync(WritebackContext *wb_context)
* BUF_WRITTEN: we wrote the buffer.
* BUF_REUSABLE: buffer is available for replacement, ie, it has
* pin count 0 and usage count 0.
+ * BUF_CANT_MERGE: can't combine this write with prior writes, caller needs
+ * to issue those first
*
* (BUF_WRITTEN could be set in error if FlushBuffer finds the buffer clean
* after locking it, but we don't care all that much.)
*/
static int
-SyncOneBuffer(int buf_id, bool skip_recently_used, WritebackContext *wb_context)
+PrepareToWriteBuffer(BuffersToWrite *to_write, Buffer buf,
+ bool skip_recently_used,
+ IOQueue *ioq, WritebackContext *wb_context)
{
- BufferDesc *bufHdr = GetBufferDescriptor(buf_id);
+ BufferDesc *cur_buf_hdr = GetBufferDescriptor(buf - 1);
+ uint32 buf_state;
int result = 0;
- uint32 buf_state;
- BufferTag tag;
+ XLogRecPtr cur_buf_lsn;
+ LWLock *content_lock;
+ bool may_block;
+
+ /*
+ * Check if this buffer can be written out together with already prepared
+ * writes. We check before we have pinned the buffer, so the buffer can be
+ * written out and replaced between this check and us pinning the buffer -
+ * we'll recheck below. The reason for the pre-check is that we don't want
+ * to pin the buffer just to find out that we can't merge the IO.
+ */
+ if (to_write->nbuffers != 0)
+ {
+ if (!CanMergeWrite(to_write, cur_buf_hdr))
+ {
+ result |= BUF_CANT_MERGE;
+ return result;
+ }
+ }
+ else
+ {
+ to_write->start_at_tag = cur_buf_hdr->tag;
+ }
/* Make sure we can handle the pin */
ReservePrivateRefCountEntry();
ResourceOwnerEnlarge(CurrentResourceOwner);
+ /* XXX: Should also check if we are allowed to pin one more buffer */
+
/*
* Check whether buffer needs writing.
*
@@ -3910,7 +4135,7 @@ SyncOneBuffer(int buf_id, bool skip_recently_used, WritebackContext *wb_context)
* don't worry because our checkpoint.redo points before log record for
* upcoming changes and so we are not required to write such dirty buffer.
*/
- buf_state = LockBufHdr(bufHdr);
+ buf_state = LockBufHdr(cur_buf_hdr);
if (BUF_STATE_GET_REFCOUNT(buf_state) == 0 &&
BUF_STATE_GET_USAGECOUNT(buf_state) == 0)
@@ -3919,40 +4144,294 @@ SyncOneBuffer(int buf_id, bool skip_recently_used, WritebackContext *wb_context)
}
else if (skip_recently_used)
{
+#if 0
+ elog(LOG, "at block %d: skip recent with nbuffers %d",
+ cur_buf_hdr->tag.blockNum, to_write->nbuffers);
+#endif
/* Caller told us not to write recently-used buffers */
- UnlockBufHdr(bufHdr, buf_state);
+ UnlockBufHdr(cur_buf_hdr, buf_state);
return result;
}
if (!(buf_state & BM_VALID) || !(buf_state & BM_DIRTY))
{
/* It's clean, so nothing to do */
- UnlockBufHdr(bufHdr, buf_state);
+ UnlockBufHdr(cur_buf_hdr, buf_state);
return result;
}
+ /* pin the buffer, from now on its identity can't change anymore */
+ PinBuffer_Locked(cur_buf_hdr);
+
+ /*
+ * Acquire IO, if needed, now that it's likely that we'll need to write.
+ */
+ if (to_write->ioh == NULL)
+ {
+ /* otherwise we should already have acquired a handle */
+ Assert(to_write->nbuffers == 0);
+
+ to_write->ioh = io_queue_acquire_io(ioq);
+ pgaio_io_get_wref(to_write->ioh, &to_write->iow);
+ }
+
/*
- * Pin it, share-lock it, write it. (FlushBuffer will do nothing if the
- * buffer is clean by the time we've locked it.)
+ * If we are merging, check if the buffer's identity possibly changed
+ * while we hadn't yet pinned it.
+ *
+ * XXX: It might be worth checking if we still want to write the buffer
+ * out, e.g. it could have been replaced with a buffer that doesn't have
+ * BM_CHECKPOINT_NEEDED set.
*/
- PinBuffer_Locked(bufHdr);
- LWLockAcquire(BufferDescriptorGetContentLock(bufHdr), LW_SHARED);
+ if (to_write->nbuffers != 0)
+ {
+ if (!CanMergeWrite(to_write, cur_buf_hdr))
+ {
+ elog(LOG, "changed identity");
+ UnpinBuffer(cur_buf_hdr);
+
+ result |= BUF_CANT_MERGE;
+
+ return result;
+ }
+ }
+
+ may_block = to_write->nbuffers == 0
+ && !pgaio_have_staged()
+ && io_queue_is_empty(ioq)
+ ;
+ content_lock = BufferDescriptorGetContentLock(cur_buf_hdr);
+
+ if (!may_block)
+ {
+ if (LWLockConditionalAcquire(content_lock, LW_SHARED))
+ {
+ /* done */
+ }
+ else if (to_write->nbuffers == 0)
+ {
+ /*
+ * Need to wait for all prior IO to finish before blocking for
+ * lock acquisition, to avoid the risk a deadlock due to us
+ * waiting for another backend that is waiting for our unsubmitted
+ * IO to complete.
+ */
+ pgaio_submit_staged();
+ io_queue_wait_all(ioq);
+
+ elog(DEBUG2, "at block %u: can't block, nbuffers = 0",
+ cur_buf_hdr->tag.blockNum
+ );
+
+ may_block = to_write->nbuffers == 0
+ && !pgaio_have_staged()
+ && io_queue_is_empty(ioq)
+ ;
+ Assert(may_block);
+
+ LWLockAcquire(content_lock, LW_SHARED);
+ }
+ else
+ {
+ elog(DEBUG2, "at block %d: can't block nbuffers = %d",
+ cur_buf_hdr->tag.blockNum,
+ to_write->nbuffers);
- FlushBuffer(bufHdr, NULL, IOOBJECT_RELATION, IOCONTEXT_NORMAL);
+ UnpinBuffer(cur_buf_hdr);
+ result |= BUF_CANT_MERGE;
+ Assert(to_write->nbuffers > 0);
- LWLockRelease(BufferDescriptorGetContentLock(bufHdr));
+ return result;
+ }
+ }
+ else
+ {
+ LWLockAcquire(content_lock, LW_SHARED);
+ }
- tag = bufHdr->tag;
+ if (!may_block)
+ {
+ if (!StartBufferIO(cur_buf_hdr, false, !may_block))
+ {
+ pgaio_submit_staged();
+ io_queue_wait_all(ioq);
- UnpinBuffer(bufHdr);
+ may_block = io_queue_is_empty(ioq) && to_write->nbuffers == 0 && !pgaio_have_staged();
+
+ if (!StartBufferIO(cur_buf_hdr, false, !may_block))
+ {
+ elog(DEBUG2, "at block %d: non-waitable StartBufferIO returns false, %d",
+ cur_buf_hdr->tag.blockNum,
+ may_block);
+
+ /*
+ * FIXME: can't tell whether this is because the buffer has
+ * been cleaned
+ */
+ if (!may_block)
+ {
+ result |= BUF_CANT_MERGE;
+ Assert(to_write->nbuffers > 0);
+ }
+ LWLockRelease(content_lock);
+ UnpinBuffer(cur_buf_hdr);
+
+ return result;
+ }
+ }
+ }
+ else
+ {
+ if (!StartBufferIO(cur_buf_hdr, false, false))
+ {
+ elog(DEBUG2, "waitable StartBufferIO returns false");
+ LWLockRelease(content_lock);
+ UnpinBuffer(cur_buf_hdr);
+
+ /*
+ * FIXME: Historically we returned BUF_WRITTEN in this case, which
+ * seems wrong
+ */
+ return result;
+ }
+ }
/*
- * SyncOneBuffer() is only called by checkpointer and bgwriter, so
- * IOContext will always be IOCONTEXT_NORMAL.
+ * Run PageGetLSN while holding header lock, since we don't have the
+ * buffer locked exclusively in all cases.
*/
- ScheduleBufferTagForWriteback(wb_context, IOCONTEXT_NORMAL, &tag);
+ buf_state = LockBufHdr(cur_buf_hdr);
+
+ cur_buf_lsn = BufferGetLSN(cur_buf_hdr);
+
+ /* To check if block content changes while flushing. - vadim 01/17/97 */
+ buf_state &= ~BM_JUST_DIRTIED;
+
+ UnlockBufHdr(cur_buf_hdr, buf_state);
+
+ to_write->buffers[to_write->nbuffers] = buf;
+ to_write->nbuffers++;
+
+ if (buf_state & BM_PERMANENT &&
+ (to_write->max_lsn == InvalidXLogRecPtr || to_write->max_lsn < cur_buf_lsn))
+ {
+ to_write->max_lsn = cur_buf_lsn;
+ }
+
+ result |= BUF_WRITTEN;
+
+ return result;
+}
+
+static void
+WriteBuffers(BuffersToWrite *to_write,
+ IOQueue *ioq, WritebackContext *wb_context)
+{
+ SMgrRelation smgr;
+ Buffer first_buf;
+ BufferDesc *first_buf_hdr;
+ bool needs_checksum;
+
+ Assert(to_write->nbuffers > 0 && to_write->nbuffers <= io_combine_limit);
+
+ first_buf = to_write->buffers[0];
+ first_buf_hdr = GetBufferDescriptor(first_buf - 1);
+
+ smgr = smgropen(BufTagGetRelFileLocator(&first_buf_hdr->tag), INVALID_PROC_NUMBER);
+
+ /*
+ * Force XLOG flush up to buffer's LSN. This implements the basic WAL
+ * rule that log updates must hit disk before any of the data-file changes
+ * they describe do.
+ *
+ * However, this rule does not apply to unlogged relations, which will be
+ * lost after a crash anyway. Most unlogged relation pages do not bear
+ * LSNs since we never emit WAL records for them, and therefore flushing
+ * up through the buffer LSN would be useless, but harmless. However,
+ * GiST indexes use LSNs internally to track page-splits, and therefore
+ * unlogged GiST pages bear "fake" LSNs generated by
+ * GetFakeLSNForUnloggedRel. It is unlikely but possible that the fake
+ * LSN counter could advance past the WAL insertion point; and if it did
+ * happen, attempting to flush WAL through that location would fail, with
+ * disastrous system-wide consequences. To make sure that can't happen,
+ * skip the flush if the buffer isn't permanent.
+ */
+ if (to_write->max_lsn != InvalidXLogRecPtr)
+ XLogFlush(to_write->max_lsn);
+
+ /*
+ * Now it's safe to write buffer to disk. Note that no one else should
+ * have been able to write it while we were busy with log flushing because
+ * only one process at a time can set the BM_IO_IN_PROGRESS bit.
+ */
+
+ for (int nbuf = 0; nbuf < to_write->nbuffers; nbuf++)
+ {
+ Buffer cur_buf = to_write->buffers[nbuf];
+ BufferDesc *cur_buf_hdr = GetBufferDescriptor(cur_buf - 1);
+ Block bufBlock;
+ char *bufToWrite;
+
+ bufBlock = BufHdrGetBlock(cur_buf_hdr);
+ needs_checksum = PageNeedsChecksumCopy((Page) bufBlock);
+
+ /*
+ * Update page checksum if desired. Since we have only shared lock on
+ * the buffer, other processes might be updating hint bits in it, so
+ * we must copy the page to a bounce buffer if we do checksumming.
+ */
+ if (needs_checksum)
+ {
+ PgAioBounceBuffer *bb = pgaio_bounce_buffer_get();
+
+ pgaio_io_assoc_bounce_buffer(to_write->ioh, bb);
+
+ bufToWrite = pgaio_bounce_buffer_buffer(bb);
+ memcpy(bufToWrite, bufBlock, BLCKSZ);
+ PageSetChecksumInplace((Page) bufToWrite, cur_buf_hdr->tag.blockNum);
+ }
+ else
+ {
+ bufToWrite = bufBlock;
+ }
+
+ to_write->data_ptrs[nbuf] = bufToWrite;
+ }
+
+ pgaio_io_set_handle_data_32(to_write->ioh,
+ (uint32 *) to_write->buffers,
+ to_write->nbuffers);
+ pgaio_io_register_callbacks(to_write->ioh, PGAIO_HCB_SHARED_BUFFER_WRITEV, 0);
+
+ smgrstartwritev(to_write->ioh, smgr,
+ BufTagGetForkNum(&first_buf_hdr->tag),
+ first_buf_hdr->tag.blockNum,
+ to_write->data_ptrs,
+ to_write->nbuffers,
+ false);
+ pgstat_count_io_op(IOOBJECT_RELATION, IOCONTEXT_NORMAL,
+ IOOP_WRITE, 1, BLCKSZ * to_write->nbuffers);
+
+
+ for (int nbuf = 0; nbuf < to_write->nbuffers; nbuf++)
+ {
+ Buffer cur_buf = to_write->buffers[nbuf];
+ BufferDesc *cur_buf_hdr = GetBufferDescriptor(cur_buf - 1);
+
+ UnpinBuffer(cur_buf_hdr);
+ }
+
+ io_queue_track(ioq, &to_write->iow);
+ to_write->total_writes++;
- return result | BUF_WRITTEN;
+ /* clear state for next write */
+ to_write->nbuffers = 0;
+ to_write->start_at_tag.relNumber = InvalidOid;
+ to_write->start_at_tag.blockNum = InvalidBlockNumber;
+ to_write->max_combine = 0;
+ to_write->max_lsn = InvalidXLogRecPtr;
+ to_write->ioh = NULL;
+ pgaio_wref_clear(&to_write->iow);
}
/*
@@ -4325,6 +4804,7 @@ FlushBuffer(BufferDesc *buf, SMgrRelation reln, IOObject io_object,
error_context_stack = errcallback.previous;
}
+
/*
* RelationGetNumberOfBlocksInFork
* Determines the current number of pages in the specified relation fork.
diff --git a/src/backend/storage/page/bufpage.c b/src/backend/storage/page/bufpage.c
index ecc81aacfc3..71dc6f559dd 100644
--- a/src/backend/storage/page/bufpage.c
+++ b/src/backend/storage/page/bufpage.c
@@ -1480,6 +1480,16 @@ PageIndexTupleOverwrite(Page page, OffsetNumber offnum,
return true;
}
+bool
+PageNeedsChecksumCopy(Page page)
+{
+ if (PageIsNew(page))
+ return false;
+
+ /* If we don't need a checksum, just return the passed-in data */
+ return DataChecksumsEnabled();
+}
+
/*
* Set checksum for a page in shared buffers.
diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list
index 9c07fa04b49..96144af6a4a 100644
--- a/src/tools/pgindent/typedefs.list
+++ b/src/tools/pgindent/typedefs.list
@@ -348,6 +348,7 @@ BufferManagerRelation
BufferStrategyControl
BufferTag
BufferUsage
+BuffersToWrite
BuildAccumulator
BuiltinScript
BulkInsertState
--
2.48.1.76.g4e746b1a31.dirty
[text/x-diff] v2.10-0023-Ensure-a-resowner-exists-for-all-paths-that-ma.patch (2.6K, ../../zp7obmaqdd6b5u2vhpw27hovpjirtqvjakkbkaz4vju7tohyul@4kfobe7ipgxm/24-v2.10-0023-Ensure-a-resowner-exists-for-all-paths-that-ma.patch)
download | inline diff:
From 7452c48c0728ceffacb7220ef2b156f898a64259 Mon Sep 17 00:00:00 2001
From: Andres Freund <andres@anarazel.de>
Date: Tue, 18 Mar 2025 14:40:06 -0400
Subject: [PATCH v2.10 23/28] Ensure a resowner exists for all paths that may
perform AIO
Reviewed-by: Noah Misch <noah@leadboat.com>
Discussion: https://postgr.es/m/1f6b50a7-38ef-4d87-8246-786d39f46ab9@iki.fi
---
src/backend/bootstrap/bootstrap.c | 7 +++++++
src/backend/replication/logical/logical.c | 6 ++++++
src/backend/utils/init/postinit.c | 6 +++++-
3 files changed, 18 insertions(+), 1 deletion(-)
diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c
index 6db864892d0..e554504e1f0 100644
--- a/src/backend/bootstrap/bootstrap.c
+++ b/src/backend/bootstrap/bootstrap.c
@@ -361,8 +361,15 @@ BootstrapModeMain(int argc, char *argv[], bool check_only)
BaseInit();
bootstrap_signals();
+
+ /* need a resowner for IO during BootStrapXLOG() */
+ CreateAuxProcessResourceOwner();
+
BootStrapXLOG(bootstrap_data_checksum_version);
+ ReleaseAuxProcessResources(true);
+ CurrentResourceOwner = NULL;
+
/*
* To ensure that src/common/link-canary.c is linked into the backend, we
* must call it from somewhere. Here is as good as anywhere.
diff --git a/src/backend/replication/logical/logical.c b/src/backend/replication/logical/logical.c
index 8ea846bfc3b..2b936714040 100644
--- a/src/backend/replication/logical/logical.c
+++ b/src/backend/replication/logical/logical.c
@@ -386,6 +386,12 @@ CreateInitDecodingContext(const char *plugin,
slot->data.plugin = plugin_name;
SpinLockRelease(&slot->mutex);
+ if (CurrentResourceOwner == NULL)
+ {
+ Assert(am_walsender);
+ CurrentResourceOwner = AuxProcessResourceOwner;
+ }
+
if (XLogRecPtrIsInvalid(restart_lsn))
ReplicationSlotReserveWal();
else
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index 7958ea11b73..222e24bcb08 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -785,8 +785,12 @@ InitPostgres(const char *in_dbname, Oid dboid,
* We don't yet have an aux-process resource owner, but StartupXLOG
* and ShutdownXLOG will need one. Hence, create said resource owner
* (and register a callback to clean it up after ShutdownXLOG runs).
+ *
+ * In bootstrap mode CreateAuxProcessResourceOwner() was already
+ * called in BootstrapModeMain().
*/
- CreateAuxProcessResourceOwner();
+ if (!bootstrap)
+ CreateAuxProcessResourceOwner();
StartupXLOG();
/* Release (and warn about) any buffer pins leaked in StartupXLOG */
--
2.48.1.76.g4e746b1a31.dirty
[text/x-diff] v2.10-0024-Temporary-Increase-BAS_BULKREAD-size.patch (1.3K, ../../zp7obmaqdd6b5u2vhpw27hovpjirtqvjakkbkaz4vju7tohyul@4kfobe7ipgxm/25-v2.10-0024-Temporary-Increase-BAS_BULKREAD-size.patch)
download | inline diff:
From 3143cf15ab619862a243c2b2f8410b1d6c08d215 Mon Sep 17 00:00:00 2001
From: Andres Freund <andres@anarazel.de>
Date: Tue, 18 Mar 2025 14:40:06 -0400
Subject: [PATCH v2.10 24/28] Temporary: Increase BAS_BULKREAD size
Without this we only can execute very little AIO for sequential scans, as
there's just not enough buffers in the ring. This isn't the right fix, as
just increasing the ring size can have negative performance implications in
workloads where the kernel has all the data cached.
Author:
Reviewed-By:
Discussion: https://postgr.es/m/
Backpatch:
---
src/backend/storage/buffer/freelist.c | 7 ++++++-
1 file changed, 6 insertions(+), 1 deletion(-)
diff --git a/src/backend/storage/buffer/freelist.c b/src/backend/storage/buffer/freelist.c
index 336715b6c63..b72a5957a20 100644
--- a/src/backend/storage/buffer/freelist.c
+++ b/src/backend/storage/buffer/freelist.c
@@ -555,7 +555,12 @@ GetAccessStrategy(BufferAccessStrategyType btype)
return NULL;
case BAS_BULKREAD:
- ring_size_kb = 256;
+
+ /*
+ * FIXME: Temporary increase to allow large enough streaming reads
+ * to actually benefit from AIO. This needs a better solution.
+ */
+ ring_size_kb = 2 * 1024;
break;
case BAS_BULKWRITE:
ring_size_kb = 16 * 1024;
--
2.48.1.76.g4e746b1a31.dirty
[text/x-diff] v2.10-0025-WIP-Use-MAP_POPULATE.patch (1.1K, ../../zp7obmaqdd6b5u2vhpw27hovpjirtqvjakkbkaz4vju7tohyul@4kfobe7ipgxm/26-v2.10-0025-WIP-Use-MAP_POPULATE.patch)
download | inline diff:
From 9a7ad7dc2bd2d46d31d9c60d3f4456fa44296b07 Mon Sep 17 00:00:00 2001
From: Andres Freund <andres@anarazel.de>
Date: Tue, 18 Mar 2025 14:40:06 -0400
Subject: [PATCH v2.10 25/28] WIP: Use MAP_POPULATE
For benchmarking it's quite annoying that the first time a memory is touched
has completely different perf characteristics than subsequent accesses. Using
MAP_POPULATE reduces that substantially.
---
src/backend/port/sysv_shmem.c | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/src/backend/port/sysv_shmem.c b/src/backend/port/sysv_shmem.c
index 197926d44f6..a700b02d5a1 100644
--- a/src/backend/port/sysv_shmem.c
+++ b/src/backend/port/sysv_shmem.c
@@ -620,7 +620,7 @@ CreateAnonymousSegment(Size *size)
allocsize += hugepagesize - (allocsize % hugepagesize);
ptr = mmap(NULL, allocsize, PROT_READ | PROT_WRITE,
- PG_MMAP_FLAGS | mmap_flags, -1, 0);
+ PG_MMAP_FLAGS | MAP_POPULATE | mmap_flags, -1, 0);
mmap_errno = errno;
if (huge_pages == HUGE_PAGES_TRY && ptr == MAP_FAILED)
elog(DEBUG1, "mmap(%zu) with MAP_HUGETLB failed, huge pages disabled: %m",
--
2.48.1.76.g4e746b1a31.dirty
^ permalink raw reply [nested|flat] 153+ messages in thread
* Re: AIO v2.5
2025-03-04 19:00 AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-10 18:23 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-11 19:41 ` Re: AIO v2.5 Noah Misch <noah@leadboat.com>
2025-03-11 23:55 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-12 03:57 ` Re: AIO v2.5 Noah Misch <noah@leadboat.com>
2025-03-12 17:06 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-12 23:23 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-13 10:53 ` Re: AIO v2.5 Antonin Houska <ah@cybertec.at>
2025-03-14 19:43 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-17 20:00 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-18 20:12 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
@ 2025-03-19 01:00 ` Melanie Plageman <melanieplageman@gmail.com>
2025-03-19 01:50 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
6 siblings, 1 reply; 153+ messages in thread
From: Melanie Plageman @ 2025-03-19 01:00 UTC (permalink / raw)
To: Andres Freund <andres@anarazel.de>; +Cc: Antonin Houska <ah@cybertec.at>; Noah Misch <noah@leadboat.com>; pgsql-hackers@postgresql.org, Thomas Munro <thomas.munro@gmail.com>; Heikki Linnakangas <hlinnaka@iki.fi>; Robert Haas <robertmhaas@gmail.com>; Jakub Wartak <jakub.wartak@enterprisedb.com>; Jelte Fennema-Nio <postgres@jeltef.nl>
On Tue, Mar 18, 2025 at 4:12 PM Andres Freund <andres@anarazel.de> wrote:
> Attached is v2.10,
This is a review of 0008: bufmgr: Implement AIO read support
I'm afraid it is more of a cosmetic review than a sign-off on the
patch's correctness, but perhaps it will help future readers who may
have the same questions I did.
In the commit message:
bufmgr: Implement AIO read support
This implements the following:
- Addition of callbacks to maintain buffer state on completion of a readv
- Addition of a wait reference to BufferDesc, to allow backends to wait for
IOs
- StartBufferIO(), WaitIO(), TerminateBufferIO() support for waiting AIO
I think it might be nice to say something about allowing backends to
complete IOs issued by other backends.
@@ -40,6 +41,10 @@ static const PgAioHandleCallbacksEntry aio_handle_cbs[] = {
CALLBACK_ENTRY(PGAIO_HCB_INVALID, aio_invalid_cb),
CALLBACK_ENTRY(PGAIO_HCB_MD_READV, aio_md_readv_cb),
+
+ CALLBACK_ENTRY(PGAIO_HCB_SHARED_BUFFER_READV, aio_shared_buffer_readv_cb),
+
+ CALLBACK_ENTRY(PGAIO_HCB_LOCAL_BUFFER_READV, aio_local_buffer_readv_cb),
#undef CALLBACK_ENTRY
};
I personally can't quite figure out why the read and write callbacks
are defined differently than the stage, complete, and report
callbacks. I know there is a comment above PgAioHandleCallbackID about
something about this, but it didn't really clarify it for me. Maybe
you can put a block comment at the top of aio_callback.c? Or perhaps I
just need to study it more...
@@ -5482,10 +5503,19 @@ WaitIO(BufferDesc *buf)
+ if (pgaio_wref_valid(&iow))
+ {
+ pgaio_wref_wait(&iow);
+ ConditionVariablePrepareToSleep(cv);
+ continue;
+ }
I'd add some comment above this. I reread it many times, and I still
only _think_ I understand what it does. I think the reason we need
ConditionVariablePrepareToSleep() again is because pgaio_io_wait() may
have called ConditionVariableCancelSleep() so we need to
ConditionVariablePrepareToSleep() again (it was done already at the
top of Wait())?
I'll admit I find the CV API quite confusing, so maybe I'm just
misunderstanding it.
Maybe worth mentioning in the commit message about why WaitIO() has to
work differently for AIO than sync IO.
/*
* Support LockBufferForCleanup()
*
* If we just released a pin, need to do BM_PIN_COUNT_WAITER handling.
* Most of the time the current backend will hold another pin preventing
* that from happening, but that's e.g. not the case when completing an IO
* another backend started.
*/
I found this wording a bit confusing. what about this:
We may have just released the last pin other than the waiter's. In most cases,
this backend holds another pin on the buffer. But, if, for example, this
backend is completing an IO issued by another backend, it may be time to wake
the waiter.
/*
* Helper for AIO staging callback for both reads and writes as well as temp
* and shared buffers.
*/
static pg_attribute_always_inline void
buffer_stage_common(PgAioHandle *ioh, bool is_write, bool is_temp)
I think buffer_stage_common() needs the function comment to explain
what unit it is operating on.
It is called "buffer_" singular but then it loops through io_data
which appears to contain multiple buffers
/*
* Check that all the buffers are actually ones that could conceivably
* be done in one IO, i.e. are sequential.
*/
if (i == 0)
first = buf_hdr->tag;
else
{
Assert(buf_hdr->tag.relNumber == first.relNumber);
Assert(buf_hdr->tag.blockNum == first.blockNum + i);
}
So it is interesting to me that this validation is done at this level.
Enforcing sequentialness doesn't seem like it would be intrinsically
related to or required to stage IOs. And there isn't really anything
in this function that seems like it would require it either. Usually
an assert is pretty close to the thing it is protecting.
Oh and I think the end of the loop in buffer_stage_common() would look
nicer with a small refactor with the resulting code looking like this:
/* temp buffers don't use BM_IO_IN_PROGRESS */
Assert(!is_temp || (buf_state & BM_IO_IN_PROGRESS));
/* we better have ensured the buffer is present until now */
Assert(BUF_STATE_GET_REFCOUNT(buf_state) >= 1);
/*
* Reflect that the buffer is now owned by the subsystem.
*
* For local buffers: This can't be done just in LocalRefCount as one
* might initially think, as this backend could error out while AIO is
* still in progress, releasing all the pins by the backend itself.
*/
buf_state += BUF_REFCOUNT_ONE;
buf_hdr->io_wref = io_ref;
if (is_temp)
{
pg_atomic_unlocked_write_u32(&buf_hdr->state, buf_state);
continue;
}
UnlockBufHdr(buf_hdr, buf_state);
if (is_write)
{
LWLock *content_lock;
content_lock = BufferDescriptorGetContentLock(buf_hdr);
Assert(LWLockHeldByMe(content_lock));
/*
* Lock is now owned by AIO subsystem.
*/
LWLockDisown(content_lock);
}
/*
* Stop tracking this buffer via the resowner - the AIO system now
* keeps track.
*/
ResourceOwnerForgetBufferIO(CurrentResourceOwner, buffer);
}
In buffer_readv_complete(), this comment
/*
* Iterate over all the buffers affected by this IO and call appropriate
* per-buffer completion function for each buffer.
*/
makes it sound like we might invoke different completion functions (like invoke
the completion callback), but that isn't what happens here.
failed =
prior_result.status == ARS_ERROR
|| prior_result.result <= buf_off;
Though not introduced in this commit, I will say that I find the ARS prefix not
particularly helpful. Though not as brief, something like AIO_RS_ERROR would
probably be more clear.
@@ -515,10 +517,25 @@ MarkLocalBufferDirty(Buffer buffer)
* Like StartBufferIO, but for local buffers
*/
bool
-StartLocalBufferIO(BufferDesc *bufHdr, bool forInput)
+StartLocalBufferIO(BufferDesc *bufHdr, bool forInput, bool nowait)
{
- uint32 buf_state = pg_atomic_read_u32(&bufHdr->state);
+ uint32 buf_state;
+
+ /*
+ * The buffer could have IO in progress, e.g. when there are two scans of
+ * the same relation. Either wait for the other IO or return false.
+ */
+ if (pgaio_wref_valid(&bufHdr->io_wref))
+ {
+ PgAioWaitRef iow = bufHdr->io_wref;
+
+ if (nowait)
+ return false;
+
+ pgaio_wref_wait(&iow);
+ }
+ buf_state = pg_atomic_read_u32(&bufHdr->state);
if (forInput ? (buf_state & BM_VALID) : !(buf_state & BM_DIRTY))
{
/* someone else already did the I/O */
I'd move this comment ("someone else already did") outside of the if
statement so it kind of separates it into three clear cases:
1) the IO is in progress and you can wait on it if you want, 2) the IO
is completed by someone else (is this possible for local buffers
without AIO?) 3) you can start the IO
- Melanie
^ permalink raw reply [nested|flat] 153+ messages in thread
* Re: AIO v2.5
2025-03-04 19:00 AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-10 18:23 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-11 19:41 ` Re: AIO v2.5 Noah Misch <noah@leadboat.com>
2025-03-11 23:55 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-12 03:57 ` Re: AIO v2.5 Noah Misch <noah@leadboat.com>
2025-03-12 17:06 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-12 23:23 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-13 10:53 ` Re: AIO v2.5 Antonin Houska <ah@cybertec.at>
2025-03-14 19:43 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-17 20:00 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-18 20:12 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-19 01:00 ` Re: AIO v2.5 Melanie Plageman <melanieplageman@gmail.com>
@ 2025-03-19 01:50 ` Andres Freund <andres@anarazel.de>
0 siblings, 0 replies; 153+ messages in thread
From: Andres Freund @ 2025-03-19 01:50 UTC (permalink / raw)
To: Melanie Plageman <melanieplageman@gmail.com>; +Cc: Antonin Houska <ah@cybertec.at>; Noah Misch <noah@leadboat.com>; pgsql-hackers@postgresql.org, Thomas Munro <thomas.munro@gmail.com>; Heikki Linnakangas <hlinnaka@iki.fi>; Robert Haas <robertmhaas@gmail.com>; Jakub Wartak <jakub.wartak@enterprisedb.com>; Jelte Fennema-Nio <postgres@jeltef.nl>
Hi,
On 2025-03-18 21:00:17 -0400, Melanie Plageman wrote:
> On Tue, Mar 18, 2025 at 4:12 PM Andres Freund <andres@anarazel.de> wrote:
> > Attached is v2.10,
>
> This is a review of 0008: bufmgr: Implement AIO read support
>
> I'm afraid it is more of a cosmetic review than a sign-off on the
> patch's correctness, but perhaps it will help future readers who may
> have the same questions I did.
I think that's actually an important level of review. I'm, as odd as that
sounds, more confident about the architectural stuff than about
"understandability" etc.
> In the commit message:
> bufmgr: Implement AIO read support
>
> This implements the following:
> - Addition of callbacks to maintain buffer state on completion of a readv
> - Addition of a wait reference to BufferDesc, to allow backends to wait for
> IOs
> - StartBufferIO(), WaitIO(), TerminateBufferIO() support for waiting AIO
>
> I think it might be nice to say something about allowing backends to
> complete IOs issued by other backends.
Hm, I'd have said that's basically implied by the way AIO works (as outlined
in the added README.md), but I can think of a way to mention it here.
> @@ -40,6 +41,10 @@ static const PgAioHandleCallbacksEntry aio_handle_cbs[] = {
> CALLBACK_ENTRY(PGAIO_HCB_INVALID, aio_invalid_cb),
>
> CALLBACK_ENTRY(PGAIO_HCB_MD_READV, aio_md_readv_cb),
> +
> + CALLBACK_ENTRY(PGAIO_HCB_SHARED_BUFFER_READV, aio_shared_buffer_readv_cb),
> +
> + CALLBACK_ENTRY(PGAIO_HCB_LOCAL_BUFFER_READV, aio_local_buffer_readv_cb),
> #undef CALLBACK_ENTRY
> };
>
> I personally can't quite figure out why the read and write callbacks
> are defined differently than the stage, complete, and report
> callbacks. I know there is a comment above PgAioHandleCallbackID about
> something about this, but it didn't really clarify it for me. Maybe
> you can put a block comment at the top of aio_callback.c? Or perhaps I
> just need to study it more...
They're not implemented differently - PgAioHandleCallbacks (which is what is
contained in aio_handle_cbs, just with a name added) all have a stage,
complete and report callbacks.
E.g. for SHARED_BUFFER_READV you have a stage (to transfer the buffer pins to
the AIO subsystem), a shared completion (to verify the page, to remove
BM_IO_IN_PROGRESS and set BM_VALID/BM_IO_ERROR, as appropriate) and a report
callback (to report a page validation error).
Maybe more of the relevant types and functions should have been plural, but
then it becomes very awkward to talk about the separate registrations of
multiple callbacks (i.e. the set of callbacks for md.c and the set of
callbacks for bufmgr.c).
> @@ -5482,10 +5503,19 @@ WaitIO(BufferDesc *buf)
> + if (pgaio_wref_valid(&iow))
> + {
> + pgaio_wref_wait(&iow);
> + ConditionVariablePrepareToSleep(cv);
> + continue;
> + }
>
> I'd add some comment above this. I reread it many times, and I still
> only _think_ I understand what it does. I think the reason we need
> ConditionVariablePrepareToSleep() again is because pgaio_io_wait() may
> have called ConditionVariableCancelSleep() so we need to
> ConditionVariablePrepareToSleep() again (it was done already at the
> top of Wait())?
Oh, yes, that definitely needs a comment. I've been marinating in this for so
long that it seems obvious, but if I take a step back, it's not at all
obvious.
The issue is that pgaio_wref_wait() internally waits on a *different*
condition variable than the BufferDesc's CV. The consequences of not doing
this would be fairly mild - the next ConditionVariableSleep would prepare to
sleep and return immediately - but it's unnecessary.
> Maybe worth mentioning in the commit message about why WaitIO() has to
> work differently for AIO than sync IO.
K.
> /*
> * Support LockBufferForCleanup()
> *
> * If we just released a pin, need to do BM_PIN_COUNT_WAITER handling.
> * Most of the time the current backend will hold another pin preventing
> * that from happening, but that's e.g. not the case when completing an IO
> * another backend started.
> */
>
> I found this wording a bit confusing. what about this:
>
> We may have just released the last pin other than the waiter's. In most cases,
> this backend holds another pin on the buffer. But, if, for example, this
> backend is completing an IO issued by another backend, it may be time to wake
> the waiter.
WFM.
> /*
> * Helper for AIO staging callback for both reads and writes as well as temp
> * and shared buffers.
> */
> static pg_attribute_always_inline void
> buffer_stage_common(PgAioHandle *ioh, bool is_write, bool is_temp)
>
> I think buffer_stage_common() needs the function comment to explain
> what unit it is operating on.
> It is called "buffer_" singular but then it loops through io_data
> which appears to contain multiple buffers
Hm. Yea. Originally it was just for readv and was duplicated for writes. The
vectorized bit hinted at being for multiple buffers.
> /*
> * Check that all the buffers are actually ones that could conceivably
> * be done in one IO, i.e. are sequential.
> */
> if (i == 0)
> first = buf_hdr->tag;
> else
> {
> Assert(buf_hdr->tag.relNumber == first.relNumber);
> Assert(buf_hdr->tag.blockNum == first.blockNum + i);
> }
>
> So it is interesting to me that this validation is done at this level.
> Enforcing sequentialness doesn't seem like it would be intrinsically
> related to or required to stage IOs. And there isn't really anything
> in this function that seems like it would require it either. Usually
> an assert is pretty close to the thing it is protecting.
Staging is the last buffer-aware thing that happens before IO is actually
executed. If you were to do a readv() into *non* buffers that aren't for
sequential blocks, you would get bogus buffer pool contents, because obviously
it doesn't make sense to read data for block N+1 into the buffer for block N+3
or whatnot.
The assertions did find bugs during development, fwiw.
> Oh and I think the end of the loop in buffer_stage_common() would look
> nicer with a small refactor with the resulting code looking like this:
>
> /* temp buffers don't use BM_IO_IN_PROGRESS */
> Assert(!is_temp || (buf_state & BM_IO_IN_PROGRESS));
>
> /* we better have ensured the buffer is present until now */
> Assert(BUF_STATE_GET_REFCOUNT(buf_state) >= 1);
>
> /*
> * Reflect that the buffer is now owned by the subsystem.
> *
> * For local buffers: This can't be done just in LocalRefCount as one
> * might initially think, as this backend could error out while AIO is
> * still in progress, releasing all the pins by the backend itself.
> */
> buf_state += BUF_REFCOUNT_ONE;
> buf_hdr->io_wref = io_ref;
>
> if (is_temp)
> {
> pg_atomic_unlocked_write_u32(&buf_hdr->state, buf_state);
> continue;
> }
>
> UnlockBufHdr(buf_hdr, buf_state);
>
> if (is_write)
> {
> LWLock *content_lock;
>
> content_lock = BufferDescriptorGetContentLock(buf_hdr);
>
> Assert(LWLockHeldByMe(content_lock));
>
> /*
> * Lock is now owned by AIO subsystem.
> */
> LWLockDisown(content_lock);
> }
>
> /*
> * Stop tracking this buffer via the resowner - the AIO system now
> * keeps track.
> */
> ResourceOwnerForgetBufferIO(CurrentResourceOwner, buffer);
> }
I don't particularly like this, I'd like to make the logic for shared and
local buffers more similar over time. E.g. by also tracking local buffer IO
via resowner.
> In buffer_readv_complete(), this comment
>
> /*
> * Iterate over all the buffers affected by this IO and call appropriate
> * per-buffer completion function for each buffer.
> */
>
> makes it sound like we might invoke different completion functions (like invoke
> the completion callback), but that isn't what happens here.
Oops, that's how it used to work, but it doesn't anymore, because it ended up
with too much duplication.
> failed =
> prior_result.status == ARS_ERROR
> || prior_result.result <= buf_off;
>
> Though not introduced in this commit, I will say that I find the ARS prefix not
> particularly helpful. Though not as brief, something like AIO_RS_ERROR would
> probably be more clear.
Fair enough. I'd go for PGAIO_RS_ERROR etc though.
> @@ -515,10 +517,25 @@ MarkLocalBufferDirty(Buffer buffer)
> * Like StartBufferIO, but for local buffers
> */
> bool
> -StartLocalBufferIO(BufferDesc *bufHdr, bool forInput)
> +StartLocalBufferIO(BufferDesc *bufHdr, bool forInput, bool nowait)
> {
> - uint32 buf_state = pg_atomic_read_u32(&bufHdr->state);
> + uint32 buf_state;
> +
> + /*
> + * The buffer could have IO in progress, e.g. when there are two scans of
> + * the same relation. Either wait for the other IO or return false.
> + */
> + if (pgaio_wref_valid(&bufHdr->io_wref))
> + {
> + PgAioWaitRef iow = bufHdr->io_wref;
> +
> + if (nowait)
> + return false;
> +
> + pgaio_wref_wait(&iow);
> + }
>
> + buf_state = pg_atomic_read_u32(&bufHdr->state);
> if (forInput ? (buf_state & BM_VALID) : !(buf_state & BM_DIRTY))
> {
> /* someone else already did the I/O */
>
> I'd move this comment ("someone else already did") outside of the if
> statement so it kind of separates it into three clear cases:
FWIW it's inside because that's how StartBufferIOs comment has been for a fair
while...
> 1) the IO is in progress and you can wait on it if you want,
> 2) the IO is completed by someone else (is this possible for local buffers
> without AIO?)
No, that's not possible without AIO.
> 3) you can start the IO
I'll give it a go.
Thanks for the review!
Greetings,
Andres Freund
^ permalink raw reply [nested|flat] 153+ messages in thread
* Re: AIO v2.5
2025-03-04 19:00 AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-10 18:23 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-11 19:41 ` Re: AIO v2.5 Noah Misch <noah@leadboat.com>
2025-03-11 23:55 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-12 03:57 ` Re: AIO v2.5 Noah Misch <noah@leadboat.com>
2025-03-12 17:06 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-12 23:23 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-13 10:53 ` Re: AIO v2.5 Antonin Houska <ah@cybertec.at>
2025-03-14 19:43 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-17 20:00 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-18 20:12 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
@ 2025-03-19 14:47 ` Melanie Plageman <melanieplageman@gmail.com>
6 siblings, 0 replies; 153+ messages in thread
From: Melanie Plageman @ 2025-03-19 14:47 UTC (permalink / raw)
To: Andres Freund <andres@anarazel.de>; +Cc: Antonin Houska <ah@cybertec.at>; Noah Misch <noah@leadboat.com>; pgsql-hackers@postgresql.org, Thomas Munro <thomas.munro@gmail.com>; Heikki Linnakangas <hlinnaka@iki.fi>; Robert Haas <robertmhaas@gmail.com>; Jakub Wartak <jakub.wartak@enterprisedb.com>; Jelte Fennema-Nio <postgres@jeltef.nl>
On Tue, Mar 18, 2025 at 4:12 PM Andres Freund <andres@anarazel.de> wrote:
>
> Attached is v2.10
This is a review of 0002: bufmgr: Improve stats when buffer is read
in concurrently
In the commit message, it might be worth distinguishing that
pg_stat_io and vacuum didn't double count reads, they under-counted
hits. pgBufferUsage and relation-level stats (pg_stat_all_tables etc)
overcounted reads and undercounted hits.
Quick example:
On master, if we try to read 7 blocks and 3 were hits and 2 were
completed by someone else then
- pg_stat_io and VacuumCostBalance would record 3 hits and 2 reads,
which looks like 2 misses.
- pgBufferUsage would record 3 hits and 4 reads, which looks like 4 misses.
- pg_stat_all_tables would record 3 hits and 7 reads, which looks like 4 misses.
The correct number of misses is 2 misses comprising 5 hits and 2 reads
(or 7 reads and 5 hits for pg_stat_all_tables which does the math
later).
@@ -1463,8 +1450,13 @@ WaitReadBuffers(ReadBuffersOperation *operation)
if (!WaitReadBuffersCanStartIO(buffers[i], false))
{
/*
- * Report this as a 'hit' for this backend, even though it must
- * have started out as a miss in PinBufferForBlock().
+ * Report and track this as a 'hit' for this backend, even though
+ * it must have started out as a miss in PinBufferForBlock().
+ *
+ * Some of the accesses would otherwise never be counted (e.g.
+ * pgBufferUsage) or counted as a miss (e.g.
+ * pgstat_count_buffer_hit(), as we always call
+ * pgstat_count_buffer_read()).
*/
I think this comment should be changed. It reads like something
written when discovering this problem and not like something useful in
the future. I think you can probably drop the whole second paragraph.
You could make it even more clear by mentioning that the other backend
will count it as a read.
Otherwise, LGTM
- Melanie
^ permalink raw reply [nested|flat] 153+ messages in thread
* Re: AIO v2.5
2025-03-04 19:00 AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-10 18:23 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-11 19:41 ` Re: AIO v2.5 Noah Misch <noah@leadboat.com>
2025-03-11 23:55 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-12 03:57 ` Re: AIO v2.5 Noah Misch <noah@leadboat.com>
2025-03-12 17:06 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-12 23:23 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-13 10:53 ` Re: AIO v2.5 Antonin Houska <ah@cybertec.at>
2025-03-14 19:43 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-17 20:00 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-18 20:12 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
@ 2025-03-19 17:20 ` Melanie Plageman <melanieplageman@gmail.com>
2025-03-19 19:28 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
6 siblings, 1 reply; 153+ messages in thread
From: Melanie Plageman @ 2025-03-19 17:20 UTC (permalink / raw)
To: Andres Freund <andres@anarazel.de>; +Cc: Antonin Houska <ah@cybertec.at>; Noah Misch <noah@leadboat.com>; pgsql-hackers@postgresql.org, Thomas Munro <thomas.munro@gmail.com>; Heikki Linnakangas <hlinnaka@iki.fi>; Robert Haas <robertmhaas@gmail.com>; Jakub Wartak <jakub.wartak@enterprisedb.com>; Jelte Fennema-Nio <postgres@jeltef.nl>
On Tue, Mar 18, 2025 at 4:12 PM Andres Freund <andres@anarazel.de> wrote:
>
> Attached is v2.10,
I noticed a few comments could be improved in 0011: bufmgr: Use AIO
in StartReadBuffers()
In WaitReadBuffers(), this comment is incomplete:
/*
- * Skip this block if someone else has already completed it. If an
- * I/O is already in progress in another backend, this will wait for
- * the outcome: either done, or something went wrong and we will
- * retry.
+ * If there is an IO associated with the operation, we may need to
+ * wait for it. It's possible for there to be no IO if
*/
In WaitReadBuffers(), too many thes
/*
* Most of the the the one IO we started will read in everything. But
* we need to deal with short reads and buffers not needing IO
* anymore.
*/
In ReadBuffersCanStartIO()
+ /*
+ * Unfortunately a false returned StartBufferIO() doesn't allow to
+ * distinguish between the buffer already being valid and IO already
+ * being in progress. Since IO already being in progress is quite
+ * rare, this approach seems fine.
+ */
maybe reword "a false returned StartBufferIO()"
Above and in AsyncReadBuffers()
* To support retries after short reads, the first operation->nblocks_done is
* buffers are skipped.
can't quite understand this
+ * On return *nblocks_progres is updated to reflect the number of buffers
progress spelled wrong
* A secondary benefit is that this would allows us to measure the time in
* pgaio_io_acquire() without causing undue timer overhead in the common,
* non-blocking, case. However, currently the pgstats infrastructure
* doesn't really allow that, as it a) asserts that an operation can't
* have time without operations b) doesn't have an API to report
* "accumulated" time.
*/
allows->allow
What would the time spent in pgaio_io_acquire() be reported as? Time
submitting IOs? Time waiting for a handle? And what is "accumulated"
time here? It seems like you just add the time to the running total
and that is already accumulated.
- Melanie
^ permalink raw reply [nested|flat] 153+ messages in thread
* Re: AIO v2.5
2025-03-04 19:00 AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-10 18:23 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-11 19:41 ` Re: AIO v2.5 Noah Misch <noah@leadboat.com>
2025-03-11 23:55 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-12 03:57 ` Re: AIO v2.5 Noah Misch <noah@leadboat.com>
2025-03-12 17:06 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-12 23:23 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-13 10:53 ` Re: AIO v2.5 Antonin Houska <ah@cybertec.at>
2025-03-14 19:43 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-17 20:00 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-18 20:12 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-19 17:20 ` Re: AIO v2.5 Melanie Plageman <melanieplageman@gmail.com>
@ 2025-03-19 19:28 ` Andres Freund <andres@anarazel.de>
0 siblings, 0 replies; 153+ messages in thread
From: Andres Freund @ 2025-03-19 19:28 UTC (permalink / raw)
To: Melanie Plageman <melanieplageman@gmail.com>; +Cc: Antonin Houska <ah@cybertec.at>; Noah Misch <noah@leadboat.com>; pgsql-hackers@postgresql.org, Thomas Munro <thomas.munro@gmail.com>; Heikki Linnakangas <hlinnaka@iki.fi>; Robert Haas <robertmhaas@gmail.com>; Jakub Wartak <jakub.wartak@enterprisedb.com>; Jelte Fennema-Nio <postgres@jeltef.nl>
Hi,
On 2025-03-19 13:20:17 -0400, Melanie Plageman wrote:
> On Tue, Mar 18, 2025 at 4:12 PM Andres Freund <andres@anarazel.de> wrote:
> >
> > Attached is v2.10,
>
> I noticed a few comments could be improved in 0011: bufmgr: Use AIO
> in StartReadBuffers()
> [...]
Yep.
> Above and in AsyncReadBuffers()
>
> * To support retries after short reads, the first operation->nblocks_done is
> * buffers are skipped.
>
> can't quite understand this
Heh, yea, it's easy to misunderstand. "short read" in the sense of a partial
read, i.e. a preadv() that only read some of the blocks, not all. I'm
replacing the "short" with partial.
(also removed the superfluous "is")
> * A secondary benefit is that this would allows us to measure the time in
> * pgaio_io_acquire() without causing undue timer overhead in the common,
> * non-blocking, case. However, currently the pgstats infrastructure
> * doesn't really allow that, as it a) asserts that an operation can't
> * have time without operations b) doesn't have an API to report
> * "accumulated" time.
> */
>
> allows->allow
>
> What would the time spent in pgaio_io_acquire() be reported as?
I'd report it as additional time for the IO we're trying to start, as that
wait would otherwise not happen.
> And what is "accumulated" time here? It seems like you just add the time to
> the running total and that is already accumulated.
Afaict there currently is no way to report a time delta to
pgstat. pgstat_count_io_op_time() computes the time since
pgstat_prepare_io_time(). Due to the assertions that time cannot be reported
for an operation with a zero count, we can't just do two
pgstat_prepare_io_time(); ...; pgstat_count_io_op_time();
twice, with the first one passing cnt=0.
Greetings,
Andres Freund
^ permalink raw reply [nested|flat] 153+ messages in thread
* Re: AIO v2.5
2025-03-04 19:00 AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-10 18:23 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-11 19:41 ` Re: AIO v2.5 Noah Misch <noah@leadboat.com>
2025-03-11 23:55 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-12 03:57 ` Re: AIO v2.5 Noah Misch <noah@leadboat.com>
2025-03-12 17:06 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-12 23:23 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-13 10:53 ` Re: AIO v2.5 Antonin Houska <ah@cybertec.at>
2025-03-14 19:43 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-17 20:00 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-18 20:12 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
@ 2025-03-19 21:25 ` Noah Misch <noah@leadboat.com>
2025-03-19 22:17 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
6 siblings, 1 reply; 153+ messages in thread
From: Noah Misch @ 2025-03-19 21:25 UTC (permalink / raw)
To: Andres Freund <andres@anarazel.de>; +Cc: Antonin Houska <ah@cybertec.at>; pgsql-hackers@postgresql.org, Thomas Munro <thomas.munro@gmail.com>; Heikki Linnakangas <hlinnaka@iki.fi>; Robert Haas <robertmhaas@gmail.com>; Jakub Wartak <jakub.wartak@enterprisedb.com>; Jelte Fennema-Nio <postgres@jeltef.nl>
On Wed, Mar 12, 2025 at 01:06:03PM -0400, Andres Freund wrote:
> On 2025-03-11 20:57:43 -0700, Noah Misch wrote:
> > - Like you say, "redefine max_files_per_process to be about the number of
> > files each *backend* will additionally open". It will become normal that
> > each backend's actual FD list length is max_files_per_process + MaxBackends
> > if io_method=io_uring. Outcome is not unlike
> > v6-0002-Bump-postmaster-soft-open-file-limit-RLIMIT_NOFIL.patch +
> > v6-0003-Reflect-the-value-of-max_safe_fds-in-max_files_pe.patch but we don't
> > mutate max_files_per_process. Benchmark results should not change beyond
> > the inter-major-version noise level unless one sets io_method=io_uring. I'm
> > feeling best about this one, but I've not been thinking about it long.
>
> Yea, I think that's something probably worth doing separately from Jelte's
> patch. I do think that it'd be rather helpful to have jelte's patch to
> increase NOFILE in addition though.
Agreed.
> > > > > +static void
> > > > > +maybe_adjust_io_workers(void)
> > > >
> > > > This also restarts workers that exit, so perhaps name it
> > > > start_io_workers_if_missing().
> > >
> > > But it also stops IO workers if necessary?
> >
> > Good point. Maybe just add a comment like "start or stop IO workers to close
> > the gap between the running count and the configured count intent".
>
> It's now
> /*
> * Start or stop IO workers, to close the gap between the number of running
> * workers and the number of configured workers. Used to respond to change of
> * the io_workers GUC (by increasing and decreasing the number of workers), as
> * well as workers terminating in response to errors (by starting
> * "replacement" workers).
> */
Excellent.
> > > > > +{
> > > > ...
> > > > > + /* Try to launch one. */
> > > > > + child = StartChildProcess(B_IO_WORKER);
> > > > > + if (child != NULL)
> > > > > + {
> > > > > + io_worker_children[id] = child;
> > > > > + ++io_worker_count;
> > > > > + }
> > > > > + else
> > > > > + break; /* XXX try again soon? */
> > > >
> > > > Can LaunchMissingBackgroundProcesses() become the sole caller of this
> > > > function, replacing the current mix of callers? That would be more conducive
> > > > to promptly doing the right thing after launch failure.
> > >
> > > I'm not sure that'd be a good idea - right now IO workers are started before
> > > the startup process, as the startup process might need to perform IO. If we
> > > started it only later in ServerLoop() we'd potentially do a fair bit of work,
> > > including starting checkpointer, bgwriter, bgworkers before we started IO
> > > workers. That shouldn't actively break anything, but it would likely make
> > > things slower.
> >
> > I missed that. How about keeping the two calls associated with PM_STARTUP but
> > replacing the assign_io_workers() and process_pm_child_exit() calls with one
> > in LaunchMissingBackgroundProcesses()?
>
> I think replacing the call in assign_io_workers() is a good idea, that way we
> don't need assign_io_workers().
>
> Less convinced it's a good idea to do the same for process_pm_child_exit() -
> if IO workers errored out we'll launch backends etc before we get to
> LaunchMissingBackgroundProcesses(). That's not a fundamental problem, but
> seems a bit odd.
Works for me.
> I think LaunchMissingBackgroundProcesses() should be split into one that
> starts aux processes and one that starts bgworkers. The one maintaining aux
> processes should be called before we start backends, the latter not.
That makes sense, though I've not thought about it much.
> > > > > + /*
> > > > > + * It's very unlikely, but possible, that reopen fails. E.g. due
> > > > > + * to memory allocations failing or file permissions changing or
> > > > > + * such. In that case we need to fail the IO.
> > > > > + *
> > > > > + * There's not really a good errno we can report here.
> > > > > + */
> > > > > + error_errno = ENOENT;
> > > >
> > > > Agreed there's not a good errno, but let's use a fake errno that we're mighty
> > > > unlikely to confuse with an actual case of libc returning that errno. Like
> > > > one of EBADF or EOWNERDEAD.
> > >
> > > Can we rely on that to be present on all platforms, including windows?
> >
> > I expect EBADF is universal. EBADF would be fine.
>
> Hm, that's actually an error that could happen for other reasons, and IMO
> would be more confusing than ENOENT. The latter describes the issue to a
> reasonable extent, EBADFD seems like it would be more confusing.
>
> I'm not sure it's worth investing time in this - it really shouldn't happen,
> and we probably have bigger problems than the error code if it does. But if we
> do want to do something, I think I can see a way to report a dedicated error
> message for this.
I agree it's not worth much investment. Let's leave that one as-is. We can
always change it further if the not-really-good errno shows up too much.
> > https://github.com/coreutils/gnulib/blob/master/doc/posix-headers/errno.texi
> > lists some OSs not having it, the newest of which looks like NetBSD 9.3
> > (2022). We could use it and add a #define for platforms lacking it.
>
> What would we define it as? I guess we could just pick a high value, but...
Some second-best value, but I withdraw that idea.
On Wed, Mar 12, 2025 at 07:23:47PM -0400, Andres Freund wrote:
> Attached is v2.7, with the following changes:
> Unresolved:
>
> - Whether to continue starting new workers in process_pm_child_exit()
I'm fine with that continuing. It's hurting ~nothing.
> - What to name the view (currently pg_aios). I'm inclined to go for
> pg_io_handles right now.
I like pg_aios mildly better than pg_io_handles, since "handle" sounds
implementation-centric.
On Fri, Mar 14, 2025 at 03:43:15PM -0400, Andres Freund wrote:
> Attached is v2.8 with the following changes:
> - In parallel: Find a way to deal with the set_max_safe_fds() issue that we've
> been discussing on this thread recently. As that only affects io_uring, it
> doesn't have to block other patches going in.
As above, I like the "redefine" option.
> - Right now effective_io_concurrency cannot be set > 0 on Windows and other
> platforms that lack posix_fadvise. But with AIO we can read ahead without
> posix_fadvise().
>
> It'd not really make anything worse than today to not remove the limit, but
> it'd be pretty weird to prevent windows etc from benefiting from AIO. Need
> to look around and see whether it would require anything other than doc
> changes.
Worth changing, but non-blocking.
On Fri, Mar 14, 2025 at 03:58:43PM -0400, Andres Freund wrote:
> - Should the docs for debug_io_direct be rephrased and if so, how?
> Perhaps it's worth going from
>
> <para>
> Currently this feature reduces performance, and is intended for
> developer testing only.
> </para>
> to
> <para>
> Currently this feature reduces performance in many workloads, and is
> intended for testing only.
> </para>
>
> I.e. qualify the downside with "many workloads" and widen the audience ever so
> slightly?
Yes, that's good.
Other than the smgr patch review sent on its own thread, I've not yet reviewed
any of these patches comprehensively. Given the speed of change, I felt it
was time to flush comments buffered since 2025-03-11:
commit 0284401 wrote:
> aio: Basic subsystem initialization
> @@ -465,6 +466,7 @@ AutoVacLauncherMain(const void *startup_data, size_t startup_data_len)
> */
> LWLockReleaseAll();
> pgstat_report_wait_end();
> + pgaio_error_cleanup();
AutoVacLauncherMain(), BackgroundWriterMain(), CheckpointerMain(), and
WalWriterMain() call AtEOXact_Buffers() but not AtEOXact_Aio(). Is that
proper? They do call pgaio_error_cleanup() as seen here, so the only loss is
some asserts. (The load-bearing part does get done.)
commit da72269 wrote:
> aio: Add core asynchronous I/O infrastructure
> + * This could be in aio_internal.h, as it is not pubicly referenced, but
typo -> publicly
commit 55b454d wrote:
> aio: Infrastructure for io_method=worker
> + /* Try to launch one. */
> + child = StartChildProcess(B_IO_WORKER);
> + if (child != NULL)
> + {
> + io_worker_children[id] = child;
> + ++io_worker_count;
> + }
> + else
> + break; /* XXX try again soon? */
I'd change the comment to something like one of:
retry after DetermineSleepTime()
next LaunchMissingBackgroundProcesses() will retry in <60s
On Tue, Mar 18, 2025 at 04:12:18PM -0400, Andres Freund wrote:
> - Decide what to do about the smgr interrupt issue
Replied on that thread. It's essentially ready.
> Questions / Unresolved:
>
> - Write support isn't going to land in 18, but there is a tiny bit of code
> regarding writes in the code for bufmgr IO. I guess I could move that to a
> later commit?
>
> I'm inclined to leave it, the structure of the code only really makes
> knowing that it's going to be shared between reads & writes.
Fine to leave it.
> - pg_aios view name
Covered above.
> Subject: [PATCH v2.10 08/28] bufmgr: Implement AIO read support
Some comments about BM_IO_IN_PROGRESS may need updates. This paragraph:
* The BM_IO_IN_PROGRESS flag acts as a kind of lock, used to wait for I/O on a
buffer to complete (and in releases before 14, it was accompanied by a
per-buffer LWLock). The process doing a read or write sets the flag for the
duration, and processes that need to wait for it to be cleared sleep on a
condition variable.
And these individual lines from "git grep BM_IO_IN_PROGRESS":
* I/O already in progress. We already hold BM_IO_IN_PROGRESS for the
* only one process at a time can set the BM_IO_IN_PROGRESS bit.
* only one process at a time can set the BM_IO_IN_PROGRESS bit.
* i.e at most one BM_IO_IN_PROGRESS bit is set per proc.
The last especially. For the other three lines and the paragraph, the notion
of a process "holding" BM_IO_IN_PROGRESS or being the process to "set" it or
being the process "doing a read" becomes less significant when one process
starts the IO and another completes it.
> + /* we better have ensured the buffer is present until now */
> + Assert(BUF_STATE_GET_REFCOUNT(buf_state) >= 1);
I'd delete that comment; to me, the assertion alone is clearer.
> + ereport(LOG,
> + (errcode(ERRCODE_DATA_CORRUPTED),
> + errmsg("invalid page in block %u of relation %s; zeroing out page",
This is changing level s/WARNING/LOG/. That seems orthogonal to the patch's
goals; is it needed? If so, I recommend splitting it out as a preliminary
patch, to highlight the behavior change for release notes.
> +/*
> + * Perform completion handling of a single AIO read. This read may cover
> + * multiple blocks / buffers.
> + *
> + * Shared between shared and local buffers, to reduce code duplication.
> + */
> +static pg_attribute_always_inline PgAioResult
> +buffer_readv_complete(PgAioHandle *ioh, PgAioResult prior_result,
> + uint8 cb_data, bool is_temp)
> +{
> + PgAioResult result = prior_result;
> + PgAioTargetData *td = pgaio_io_get_target_data(ioh);
> + uint64 *io_data;
> + uint8 handle_data_len;
> +
> + if (is_temp)
> + {
> + Assert(td->smgr.is_temp);
> + Assert(pgaio_io_get_owner(ioh) == MyProcNumber);
> + }
> + else
> + Assert(!td->smgr.is_temp);
> +
> + /*
> + * Iterate over all the buffers affected by this IO and call appropriate
> + * per-buffer completion function for each buffer.
> + */
> + io_data = pgaio_io_get_handle_data(ioh, &handle_data_len);
> + for (uint8 buf_off = 0; buf_off < handle_data_len; buf_off++)
> + {
> + Buffer buf = io_data[buf_off];
> + PgAioResult buf_result;
> + bool failed;
> +
> + Assert(BufferIsValid(buf));
> +
> + /*
> + * If the entire failed on a lower-level, each buffer needs to be
Missing word, probably fix like:
s,entire failed on a lower-level,entire I/O failed on a lower level,
> + * marked as failed. In case of a partial read, some buffers may be
> + * ok.
> + */
> + failed =
> + prior_result.status == ARS_ERROR
> + || prior_result.result <= buf_off;
I didn't run an experiment to check the following, but I think this should be
s/<=/</. Suppose we requested two blocks and read some amount of bytes
[1*BLCKSZ, 2*BLSCKSZ - 1]. md_readv_complete will store result=1. buf_off==0
should compute failed=false here, but buf_off==1 should compute failed=true.
I see this relies on md_readv_complete having converted "result" to blocks.
Was there some win from doing that as opposed to doing the division here?
Division here ("blocks_read = prior_result.result / BLCKSZ") would feel easier
to follow, to me.
> +
> + buf_result = buffer_readv_complete_one(buf_off, buf, cb_data, failed,
> + is_temp);
> +
> + /*
> + * If there wasn't any prior error and the IO for this page failed in
> + * some form, set the whole IO's to the page's result.
s/the IO for this page/page verification/
s/IO's/IO's result/
> + */
> + if (result.status != ARS_ERROR && buf_result.status != ARS_OK)
> + {
> + result = buf_result;
> + pgaio_result_report(result, td, LOG);
> + }
> + }
> +
> + return result;
> +}
^ permalink raw reply [nested|flat] 153+ messages in thread
* Re: AIO v2.5
2025-03-04 19:00 AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-10 18:23 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-11 19:41 ` Re: AIO v2.5 Noah Misch <noah@leadboat.com>
2025-03-11 23:55 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-12 03:57 ` Re: AIO v2.5 Noah Misch <noah@leadboat.com>
2025-03-12 17:06 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-12 23:23 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-13 10:53 ` Re: AIO v2.5 Antonin Houska <ah@cybertec.at>
2025-03-14 19:43 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-17 20:00 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-18 20:12 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-19 21:25 ` Re: AIO v2.5 Noah Misch <noah@leadboat.com>
@ 2025-03-19 22:17 ` Andres Freund <andres@anarazel.de>
2025-03-20 01:11 ` Re: AIO v2.5 Noah Misch <noah@leadboat.com>
2025-03-20 17:05 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
0 siblings, 2 replies; 153+ messages in thread
From: Andres Freund @ 2025-03-19 22:17 UTC (permalink / raw)
To: Noah Misch <noah@leadboat.com>; +Cc: Antonin Houska <ah@cybertec.at>; pgsql-hackers@postgresql.org, Thomas Munro <thomas.munro@gmail.com>; Heikki Linnakangas <hlinnaka@iki.fi>; Robert Haas <robertmhaas@gmail.com>; Jakub Wartak <jakub.wartak@enterprisedb.com>; Jelte Fennema-Nio <postgres@jeltef.nl>
Hi,
On 2025-03-19 14:25:30 -0700, Noah Misch wrote:
> On Wed, Mar 12, 2025 at 01:06:03PM -0400, Andres Freund wrote:
> > - Right now effective_io_concurrency cannot be set > 0 on Windows and other
> > platforms that lack posix_fadvise. But with AIO we can read ahead without
> > posix_fadvise().
> >
> > It'd not really make anything worse than today to not remove the limit, but
> > it'd be pretty weird to prevent windows etc from benefiting from AIO. Need
> > to look around and see whether it would require anything other than doc
> > changes.
>
> Worth changing, but non-blocking.
Thankfully Melanie submitted a patch for that...
> Other than the smgr patch review sent on its own thread, I've not yet reviewed
> any of these patches comprehensively. Given the speed of change, I felt it
> was time to flush comments buffered since 2025-03-11:
Thanks!
> commit 0284401 wrote:
> > aio: Basic subsystem initialization
>
> > @@ -465,6 +466,7 @@ AutoVacLauncherMain(const void *startup_data, size_t startup_data_len)
> > */
> > LWLockReleaseAll();
> > pgstat_report_wait_end();
> > + pgaio_error_cleanup();
>
> AutoVacLauncherMain(), BackgroundWriterMain(), CheckpointerMain(), and
> WalWriterMain() call AtEOXact_Buffers() but not AtEOXact_Aio(). Is that
> proper? They do call pgaio_error_cleanup() as seen here, so the only loss is
> some asserts. (The load-bearing part does get done.)
I don't think it's particularly good that we use the AtEOXact_* functions in
the sigsetjmp blocks, that feels like a weird mixup of infrastructure to
me. So this was intentional.
> commit da72269 wrote:
> > aio: Add core asynchronous I/O infrastructure
>
> > + * This could be in aio_internal.h, as it is not pubicly referenced, but
>
> typo -> publicly
/me has a red face.
> commit 55b454d wrote:
> > aio: Infrastructure for io_method=worker
>
> > + /* Try to launch one. */
> > + child = StartChildProcess(B_IO_WORKER);
> > + if (child != NULL)
> > + {
> > + io_worker_children[id] = child;
> > + ++io_worker_count;
> > + }
> > + else
> > + break; /* XXX try again soon? */
>
> I'd change the comment to something like one of:
>
> retry after DetermineSleepTime()
> next LaunchMissingBackgroundProcesses() will retry in <60s
Hm, we retry more frequently that that if there are new connections... Maybe
just "try again next time"?
> On Tue, Mar 18, 2025 at 04:12:18PM -0400, Andres Freund wrote:
> > - Decide what to do about the smgr interrupt issue
>
> Replied on that thread. It's essentially ready.
Cool, will reply there in a bit.
> > Subject: [PATCH v2.10 08/28] bufmgr: Implement AIO read support
>
> Some comments about BM_IO_IN_PROGRESS may need updates. This paragraph:
>
> * The BM_IO_IN_PROGRESS flag acts as a kind of lock, used to wait for I/O on a
> buffer to complete (and in releases before 14, it was accompanied by a
> per-buffer LWLock). The process doing a read or write sets the flag for the
> duration, and processes that need to wait for it to be cleared sleep on a
> condition variable.
First draft:
* The BM_IO_IN_PROGRESS flag acts as a kind of lock, used to wait for I/O on a
buffer to complete (and in releases before 14, it was accompanied by a
per-buffer LWLock). The process start a read or write sets the flag. When the
I/O is completed, be it by the process that initiated the I/O or by another
process, the flag is removed and the Buffer's condition variable is signalled.
Processes that need to wait for the I/O to complete can wait for asynchronous
I/O to using BufferDesc->io_wref and for BM_IO_IN_PROGRESS to be unset by
sleeping on the buffer's condition variable.
> And these individual lines from "git grep BM_IO_IN_PROGRESS":
>
> * i.e at most one BM_IO_IN_PROGRESS bit is set per proc.
>
> The last especially.
Huh - yea. This isn't a "new" issue, I think I missed this comment in 16's
12f3867f5534. I think the comment can just be deleted?
> * I/O already in progress. We already hold BM_IO_IN_PROGRESS for the
> * only one process at a time can set the BM_IO_IN_PROGRESS bit.
> * only one process at a time can set the BM_IO_IN_PROGRESS bit.
> For the other three lines and the paragraph, the notion
> of a process "holding" BM_IO_IN_PROGRESS or being the process to "set" it or
> being the process "doing a read" becomes less significant when one process
> starts the IO and another completes it.
Hm. I think they'd be ok as-is, but we can probably improve them. Maybe
* Now it's safe to write buffer to disk. Note that no one else should
* have been able to write it while we were busy with log flushing because
* we got the exclusive right to perform I/O by setting the
* BM_IO_IN_PROGRESS bit.
> > + /* we better have ensured the buffer is present until now */
> > + Assert(BUF_STATE_GET_REFCOUNT(buf_state) >= 1);
>
> I'd delete that comment; to me, the assertion alone is clearer.
Ok.
> > + ereport(LOG,
> > + (errcode(ERRCODE_DATA_CORRUPTED),
> > + errmsg("invalid page in block %u of relation %s; zeroing out page",
>
> This is changing level s/WARNING/LOG/. That seems orthogonal to the patch's
> goals; is it needed? If so, I recommend splitting it out as a preliminary
> patch, to highlight the behavior change for release notes.
No, it's not needed. I think I looked over the patch at some point and
considered the log-level wrong according to our guidelines and thought I'd
broken it.
> > + /*
> > + * If the entire failed on a lower-level, each buffer needs to be
>
> Missing word, probably fix like:
> s,entire failed on a lower-level,entire I/O failed on a lower level,
Yep.
> > + * marked as failed. In case of a partial read, some buffers may be
> > + * ok.
> > + */
> > + failed =
> > + prior_result.status == ARS_ERROR
> > + || prior_result.result <= buf_off;
>
> I didn't run an experiment to check the following, but I think this should be
> s/<=/</. Suppose we requested two blocks and read some amount of bytes
> [1*BLCKSZ, 2*BLSCKSZ - 1]. md_readv_complete will store result=1. buf_off==0
> should compute failed=false here, but buf_off==1 should compute failed=true.
Huh, you might be right. I thought I wrote a test for this, I wonder why it
didn't catch the problem...
> I see this relies on md_readv_complete having converted "result" to blocks.
> Was there some win from doing that as opposed to doing the division here?
> Division here ("blocks_read = prior_result.result / BLCKSZ") would feel easier
> to follow, to me.
It seemed like that would be wrong layering - what if we had an smgr that
could store data in a compressed format? The raw read would be of a smaller
size. The smgr API deals in BlockNumbers, only the md.c layer should know
about bytes.
> > +
> > + buf_result = buffer_readv_complete_one(buf_off, buf, cb_data, failed,
> > + is_temp);
> > +
> > + /*
> > + * If there wasn't any prior error and the IO for this page failed in
> > + * some form, set the whole IO's to the page's result.
>
> s/the IO for this page/page verification/
> s/IO's/IO's result/
Agreed.
Thanks for the review!
Greetings,
Andres Freund
^ permalink raw reply [nested|flat] 153+ messages in thread
* Re: AIO v2.5
2025-03-04 19:00 AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-10 18:23 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-11 19:41 ` Re: AIO v2.5 Noah Misch <noah@leadboat.com>
2025-03-11 23:55 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-12 03:57 ` Re: AIO v2.5 Noah Misch <noah@leadboat.com>
2025-03-12 17:06 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-12 23:23 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-13 10:53 ` Re: AIO v2.5 Antonin Houska <ah@cybertec.at>
2025-03-14 19:43 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-17 20:00 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-18 20:12 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-19 21:25 ` Re: AIO v2.5 Noah Misch <noah@leadboat.com>
2025-03-19 22:17 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
@ 2025-03-20 01:11 ` Noah Misch <noah@leadboat.com>
2025-03-20 18:54 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
1 sibling, 1 reply; 153+ messages in thread
From: Noah Misch @ 2025-03-20 01:11 UTC (permalink / raw)
To: Andres Freund <andres@anarazel.de>; +Cc: Antonin Houska <ah@cybertec.at>; pgsql-hackers@postgresql.org, Thomas Munro <thomas.munro@gmail.com>; Heikki Linnakangas <hlinnaka@iki.fi>; Robert Haas <robertmhaas@gmail.com>; Jakub Wartak <jakub.wartak@enterprisedb.com>; Jelte Fennema-Nio <postgres@jeltef.nl>
On Wed, Mar 19, 2025 at 06:17:37PM -0400, Andres Freund wrote:
> On 2025-03-19 14:25:30 -0700, Noah Misch wrote:
> > commit 55b454d wrote:
> > > aio: Infrastructure for io_method=worker
> >
> > > + /* Try to launch one. */
> > > + child = StartChildProcess(B_IO_WORKER);
> > > + if (child != NULL)
> > > + {
> > > + io_worker_children[id] = child;
> > > + ++io_worker_count;
> > > + }
> > > + else
> > > + break; /* XXX try again soon? */
> >
> > I'd change the comment to something like one of:
> >
> > retry after DetermineSleepTime()
> > next LaunchMissingBackgroundProcesses() will retry in <60s
>
> Hm, we retry more frequently that that if there are new connections... Maybe
> just "try again next time"?
Works for me.
> > On Tue, Mar 18, 2025 at 04:12:18PM -0400, Andres Freund wrote:
> > > Subject: [PATCH v2.10 08/28] bufmgr: Implement AIO read support
> >
> > Some comments about BM_IO_IN_PROGRESS may need updates. This paragraph:
> >
> > * The BM_IO_IN_PROGRESS flag acts as a kind of lock, used to wait for I/O on a
> > buffer to complete (and in releases before 14, it was accompanied by a
> > per-buffer LWLock). The process doing a read or write sets the flag for the
> > duration, and processes that need to wait for it to be cleared sleep on a
> > condition variable.
>
> First draft:
> * The BM_IO_IN_PROGRESS flag acts as a kind of lock, used to wait for I/O on a
> buffer to complete (and in releases before 14, it was accompanied by a
> per-buffer LWLock). The process start a read or write sets the flag. When the
s/start/starting/
> I/O is completed, be it by the process that initiated the I/O or by another
> process, the flag is removed and the Buffer's condition variable is signalled.
> Processes that need to wait for the I/O to complete can wait for asynchronous
> I/O to using BufferDesc->io_wref and for BM_IO_IN_PROGRESS to be unset by
s/to using/by using/
> sleeping on the buffer's condition variable.
Sounds good.
> > And these individual lines from "git grep BM_IO_IN_PROGRESS":
> >
> > * i.e at most one BM_IO_IN_PROGRESS bit is set per proc.
> >
> > The last especially.
>
> Huh - yea. This isn't a "new" issue, I think I missed this comment in 16's
> 12f3867f5534. I think the comment can just be deleted?
Hmm, yes, it's orthogonal to $SUBJECT and deletion works fine.
> > * I/O already in progress. We already hold BM_IO_IN_PROGRESS for the
> > * only one process at a time can set the BM_IO_IN_PROGRESS bit.
> > * only one process at a time can set the BM_IO_IN_PROGRESS bit.
>
> > For the other three lines and the paragraph, the notion
> > of a process "holding" BM_IO_IN_PROGRESS or being the process to "set" it or
> > being the process "doing a read" becomes less significant when one process
> > starts the IO and another completes it.
>
> Hm. I think they'd be ok as-is, but we can probably improve them. Maybe
Looking again, I agree they're okay.
>
> * Now it's safe to write buffer to disk. Note that no one else should
> * have been able to write it while we were busy with log flushing because
> * we got the exclusive right to perform I/O by setting the
> * BM_IO_IN_PROGRESS bit.
That's fine too. Maybe s/perform/stage/ or s/perform/start/.
> > I see this relies on md_readv_complete having converted "result" to blocks.
> > Was there some win from doing that as opposed to doing the division here?
> > Division here ("blocks_read = prior_result.result / BLCKSZ") would feel easier
> > to follow, to me.
>
> It seemed like that would be wrong layering - what if we had an smgr that
> could store data in a compressed format? The raw read would be of a smaller
> size. The smgr API deals in BlockNumbers, only the md.c layer should know
> about bytes.
I hadn't thought of that. That's a good reason.
^ permalink raw reply [nested|flat] 153+ messages in thread
* Re: AIO v2.5
2025-03-04 19:00 AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-10 18:23 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-11 19:41 ` Re: AIO v2.5 Noah Misch <noah@leadboat.com>
2025-03-11 23:55 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-12 03:57 ` Re: AIO v2.5 Noah Misch <noah@leadboat.com>
2025-03-12 17:06 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-12 23:23 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-13 10:53 ` Re: AIO v2.5 Antonin Houska <ah@cybertec.at>
2025-03-14 19:43 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-17 20:00 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-18 20:12 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-19 21:25 ` Re: AIO v2.5 Noah Misch <noah@leadboat.com>
2025-03-19 22:17 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-20 01:11 ` Re: AIO v2.5 Noah Misch <noah@leadboat.com>
@ 2025-03-20 18:54 ` Andres Freund <andres@anarazel.de>
2025-03-20 19:52 ` Re: AIO v2.5 Noah Misch <noah@leadboat.com>
0 siblings, 1 reply; 153+ messages in thread
From: Andres Freund @ 2025-03-20 18:54 UTC (permalink / raw)
To: Noah Misch <noah@leadboat.com>; +Cc: Antonin Houska <ah@cybertec.at>; pgsql-hackers@postgresql.org, Thomas Munro <thomas.munro@gmail.com>; Heikki Linnakangas <hlinnaka@iki.fi>; Robert Haas <robertmhaas@gmail.com>; Jakub Wartak <jakub.wartak@enterprisedb.com>; Jelte Fennema-Nio <postgres@jeltef.nl>
Hi,
On 2025-03-19 18:11:18 -0700, Noah Misch wrote:
> On Wed, Mar 19, 2025 at 06:17:37PM -0400, Andres Freund wrote:
> > On 2025-03-19 14:25:30 -0700, Noah Misch wrote:
> > Hm, we retry more frequently that that if there are new connections... Maybe
> > just "try again next time"?
>
> Works for me.
>
> > > And these individual lines from "git grep BM_IO_IN_PROGRESS":
> > >
> > > * i.e at most one BM_IO_IN_PROGRESS bit is set per proc.
> > >
> > > The last especially.
> >
> > Huh - yea. This isn't a "new" issue, I think I missed this comment in 16's
> > 12f3867f5534. I think the comment can just be deleted?
>
> Hmm, yes, it's orthogonal to $SUBJECT and deletion works fine.
>
> > > * I/O already in progress. We already hold BM_IO_IN_PROGRESS for the
> > > * only one process at a time can set the BM_IO_IN_PROGRESS bit.
> > > * only one process at a time can set the BM_IO_IN_PROGRESS bit.
> >
> > > For the other three lines and the paragraph, the notion
> > > of a process "holding" BM_IO_IN_PROGRESS or being the process to "set" it or
> > > being the process "doing a read" becomes less significant when one process
> > > starts the IO and another completes it.
> >
> > Hm. I think they'd be ok as-is, but we can probably improve them. Maybe
>
> Looking again, I agree they're okay.
>
> >
> > * Now it's safe to write buffer to disk. Note that no one else should
> > * have been able to write it while we were busy with log flushing because
> > * we got the exclusive right to perform I/O by setting the
> > * BM_IO_IN_PROGRESS bit.
>
> That's fine too. Maybe s/perform/stage/ or s/perform/start/.
I put these comment changes into their own patch, as it seemed confusing to
change them as part of one of the already queued commits.
> > > I see this relies on md_readv_complete having converted "result" to blocks.
> > > Was there some win from doing that as opposed to doing the division here?
> > > Division here ("blocks_read = prior_result.result / BLCKSZ") would feel easier
> > > to follow, to me.
> >
> > It seemed like that would be wrong layering - what if we had an smgr that
> > could store data in a compressed format? The raw read would be of a smaller
> > size. The smgr API deals in BlockNumbers, only the md.c layer should know
> > about bytes.
>
> I hadn't thought of that. That's a good reason.
I thought that was better documented, but alas, it wasn't. How about updating
the documentation of smgrstartreadv to the following:
/*
* smgrstartreadv() -- asynchronous version of smgrreadv()
*
* This starts an asynchronous readv IO using the IO handle `ioh`. Other than
* `ioh` all parameters are the same as smgrreadv().
*
* Completion callbacks above smgr will be passed the result as the number of
* successfully read blocks if the read [partially] succeeds. This maintains
* the abstraction that smgr operates on the level of blocks, rather than
* bytes.
*/
I briefly had a bug in test_aio's injection point that lead to *increasing*
the number of bytes successfully read. That triggered an assertion failure in
bufmgr.c, but not closer to the problem. Is it worth adding an assert against
that to md_readv_complete? Can't quite decide.
Greetings,
Andres Freund
^ permalink raw reply [nested|flat] 153+ messages in thread
* Re: AIO v2.5
2025-03-04 19:00 AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-10 18:23 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-11 19:41 ` Re: AIO v2.5 Noah Misch <noah@leadboat.com>
2025-03-11 23:55 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-12 03:57 ` Re: AIO v2.5 Noah Misch <noah@leadboat.com>
2025-03-12 17:06 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-12 23:23 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-13 10:53 ` Re: AIO v2.5 Antonin Houska <ah@cybertec.at>
2025-03-14 19:43 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-17 20:00 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-18 20:12 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-19 21:25 ` Re: AIO v2.5 Noah Misch <noah@leadboat.com>
2025-03-19 22:17 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-20 01:11 ` Re: AIO v2.5 Noah Misch <noah@leadboat.com>
2025-03-20 18:54 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
@ 2025-03-20 19:52 ` Noah Misch <noah@leadboat.com>
0 siblings, 0 replies; 153+ messages in thread
From: Noah Misch @ 2025-03-20 19:52 UTC (permalink / raw)
To: Andres Freund <andres@anarazel.de>; +Cc: Antonin Houska <ah@cybertec.at>; pgsql-hackers@postgresql.org, Thomas Munro <thomas.munro@gmail.com>; Heikki Linnakangas <hlinnaka@iki.fi>; Robert Haas <robertmhaas@gmail.com>; Jakub Wartak <jakub.wartak@enterprisedb.com>; Jelte Fennema-Nio <postgres@jeltef.nl>
On Thu, Mar 20, 2025 at 02:54:14PM -0400, Andres Freund wrote:
> On 2025-03-19 18:11:18 -0700, Noah Misch wrote:
> > On Wed, Mar 19, 2025 at 06:17:37PM -0400, Andres Freund wrote:
> > > On 2025-03-19 14:25:30 -0700, Noah Misch wrote:
> > > > I see this relies on md_readv_complete having converted "result" to blocks.
> > > > Was there some win from doing that as opposed to doing the division here?
> > > > Division here ("blocks_read = prior_result.result / BLCKSZ") would feel easier
> > > > to follow, to me.
> > >
> > > It seemed like that would be wrong layering - what if we had an smgr that
> > > could store data in a compressed format? The raw read would be of a smaller
> > > size. The smgr API deals in BlockNumbers, only the md.c layer should know
> > > about bytes.
> >
> > I hadn't thought of that. That's a good reason.
>
> I thought that was better documented, but alas, it wasn't. How about updating
> the documentation of smgrstartreadv to the following:
>
> /*
> * smgrstartreadv() -- asynchronous version of smgrreadv()
> *
> * This starts an asynchronous readv IO using the IO handle `ioh`. Other than
> * `ioh` all parameters are the same as smgrreadv().
> *
> * Completion callbacks above smgr will be passed the result as the number of
> * successfully read blocks if the read [partially] succeeds. This maintains
> * the abstraction that smgr operates on the level of blocks, rather than
> * bytes.
> */
That's good. Possibly add "(Buffers for blocks not successfully read might
bear unspecified modifications, up to the full nblocks.)"
In a bit of over-thinking this, I wondered if shared_buffer_readv_complete
would be better named shared_buffer_smgrreadv_complete, to emphasize the
smgrreadv semantics. PGAIO_HCB_SHARED_BUFFER_READV likewise. But I tend to
think not. smgrreadv() has no "result" concept, so the symmetry is limited.
> I briefly had a bug in test_aio's injection point that lead to *increasing*
> the number of bytes successfully read. That triggered an assertion failure in
> bufmgr.c, but not closer to the problem. Is it worth adding an assert against
> that to md_readv_complete? Can't quite decide.
I'd lean yes, if in doubt.
^ permalink raw reply [nested|flat] 153+ messages in thread
* Re: AIO v2.5
2025-03-04 19:00 AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-10 18:23 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-11 19:41 ` Re: AIO v2.5 Noah Misch <noah@leadboat.com>
2025-03-11 23:55 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-12 03:57 ` Re: AIO v2.5 Noah Misch <noah@leadboat.com>
2025-03-12 17:06 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-12 23:23 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-13 10:53 ` Re: AIO v2.5 Antonin Houska <ah@cybertec.at>
2025-03-14 19:43 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-17 20:00 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-18 20:12 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-19 21:25 ` Re: AIO v2.5 Noah Misch <noah@leadboat.com>
2025-03-19 22:17 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
@ 2025-03-20 17:05 ` Andres Freund <andres@anarazel.de>
2025-03-20 18:19 ` Re: AIO v2.5 Noah Misch <noah@leadboat.com>
1 sibling, 1 reply; 153+ messages in thread
From: Andres Freund @ 2025-03-20 17:05 UTC (permalink / raw)
To: Noah Misch <noah@leadboat.com>; +Cc: Antonin Houska <ah@cybertec.at>; pgsql-hackers@postgresql.org, Thomas Munro <thomas.munro@gmail.com>; Heikki Linnakangas <hlinnaka@iki.fi>; Robert Haas <robertmhaas@gmail.com>; Jakub Wartak <jakub.wartak@enterprisedb.com>; Jelte Fennema-Nio <postgres@jeltef.nl>
Hi,
On 2025-03-19 18:17:37 -0400, Andres Freund wrote:
> On 2025-03-19 14:25:30 -0700, Noah Misch wrote:
> > > + * marked as failed. In case of a partial read, some buffers may be
> > > + * ok.
> > > + */
> > > + failed =
> > > + prior_result.status == ARS_ERROR
> > > + || prior_result.result <= buf_off;
> >
> > I didn't run an experiment to check the following, but I think this should be
> > s/<=/</. Suppose we requested two blocks and read some amount of bytes
> > [1*BLCKSZ, 2*BLSCKSZ - 1]. md_readv_complete will store result=1. buf_off==0
> > should compute failed=false here, but buf_off==1 should compute failed=true.
>
> Huh, you might be right. I thought I wrote a test for this, I wonder why it
> didn't catch the problem...
It was correct as-is. With result=1 you get precisely the result you describe
as the desired outcome, no?
prior_result.result <= buf_off
->
1 <= 0 -> failed = 0
1 <= 1 -> failed = 1
but if it were < as you suggest:
prior_result.result < buf_off
->
1 < 0 -> failed = 0
1 < 1 -> failed = 0
I.e. we would assume that the second buffer also completed.
What does concern me is that the existing tests do *not* catch the problem if
I turn "<=" into "<". The second buffer in this case wrongly gets marked as
valid. We do retry the read (because bufmgr.c thinks only one block was read),
but find the buffer to already be valid.
The reason the test doesn't fail, is that the way I set up the "short read"
tests. The injection point runs after the IO completed and just modifies the
result. However, the actual buffer contents still got modified.
The easiest way around that seems to be to have the injection point actually
zero out the remaining memory. Not pretty, but it'd be harder to just submit
shortend IOs in multiple IO methods. It'd be even better if we could
trivially use something like randomize_mem(), but it's only conditionally
compiled...
Greetings,
Andres Freund
^ permalink raw reply [nested|flat] 153+ messages in thread
* Re: AIO v2.5
2025-03-04 19:00 AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-10 18:23 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-11 19:41 ` Re: AIO v2.5 Noah Misch <noah@leadboat.com>
2025-03-11 23:55 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-12 03:57 ` Re: AIO v2.5 Noah Misch <noah@leadboat.com>
2025-03-12 17:06 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-12 23:23 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-13 10:53 ` Re: AIO v2.5 Antonin Houska <ah@cybertec.at>
2025-03-14 19:43 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-17 20:00 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-18 20:12 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-19 21:25 ` Re: AIO v2.5 Noah Misch <noah@leadboat.com>
2025-03-19 22:17 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-20 17:05 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
@ 2025-03-20 18:19 ` Noah Misch <noah@leadboat.com>
0 siblings, 0 replies; 153+ messages in thread
From: Noah Misch @ 2025-03-20 18:19 UTC (permalink / raw)
To: Andres Freund <andres@anarazel.de>; +Cc: Antonin Houska <ah@cybertec.at>; pgsql-hackers@postgresql.org, Thomas Munro <thomas.munro@gmail.com>; Heikki Linnakangas <hlinnaka@iki.fi>; Robert Haas <robertmhaas@gmail.com>; Jakub Wartak <jakub.wartak@enterprisedb.com>; Jelte Fennema-Nio <postgres@jeltef.nl>
On Thu, Mar 20, 2025 at 01:05:05PM -0400, Andres Freund wrote:
> On 2025-03-19 18:17:37 -0400, Andres Freund wrote:
> > On 2025-03-19 14:25:30 -0700, Noah Misch wrote:
> > > > + * marked as failed. In case of a partial read, some buffers may be
> > > > + * ok.
> > > > + */
> > > > + failed =
> > > > + prior_result.status == ARS_ERROR
> > > > + || prior_result.result <= buf_off;
> > >
> > > I didn't run an experiment to check the following, but I think this should be
> > > s/<=/</. Suppose we requested two blocks and read some amount of bytes
> > > [1*BLCKSZ, 2*BLSCKSZ - 1]. md_readv_complete will store result=1. buf_off==0
> > > should compute failed=false here, but buf_off==1 should compute failed=true.
> >
> > Huh, you might be right. I thought I wrote a test for this, I wonder why it
> > didn't catch the problem...
>
> It was correct as-is. With result=1 you get precisely the result you describe
> as the desired outcome, no?
> prior_result.result <= buf_off
> ->
> 1 <= 0 -> failed = 0
> 1 <= 1 -> failed = 1
>
> but if it were < as you suggest:
>
> prior_result.result < buf_off
> ->
> 1 < 0 -> failed = 0
> 1 < 1 -> failed = 0
>
> I.e. we would assume that the second buffer also completed.
That's right. I see it now. My mistake.
> What does concern me is that the existing tests do *not* catch the problem if
> I turn "<=" into "<". The second buffer in this case wrongly gets marked as
> valid. We do retry the read (because bufmgr.c thinks only one block was read),
> but find the buffer to already be valid.
>
> The reason the test doesn't fail, is that the way I set up the "short read"
> tests. The injection point runs after the IO completed and just modifies the
> result. However, the actual buffer contents still got modified.
>
>
> The easiest way around that seems to be to have the injection point actually
> zero out the remaining memory.
Sounds reasonable and sufficient.
FYI, I've resumed the comprehensive review. That's still ongoing.
^ permalink raw reply [nested|flat] 153+ messages in thread
* Re: AIO v2.5
2025-03-04 19:00 AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-10 18:23 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-11 19:41 ` Re: AIO v2.5 Noah Misch <noah@leadboat.com>
2025-03-11 23:55 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-12 03:57 ` Re: AIO v2.5 Noah Misch <noah@leadboat.com>
2025-03-12 17:06 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-12 23:23 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-13 10:53 ` Re: AIO v2.5 Antonin Houska <ah@cybertec.at>
2025-03-14 19:43 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-17 20:00 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-18 20:12 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
@ 2025-03-20 08:39 ` Jakub Wartak <jakub.wartak@enterprisedb.com>
6 siblings, 0 replies; 153+ messages in thread
From: Jakub Wartak @ 2025-03-20 08:39 UTC (permalink / raw)
To: Andres Freund <andres@anarazel.de>; +Cc: Antonin Houska <ah@cybertec.at>; Noah Misch <noah@leadboat.com>; pgsql-hackers@postgresql.org, Thomas Munro <thomas.munro@gmail.com>; Heikki Linnakangas <hlinnaka@iki.fi>; Robert Haas <robertmhaas@gmail.com>; Jelte Fennema-Nio <postgres@jeltef.nl>
On Tue, Mar 18, 2025 at 9:12 PM Andres Freund <andres@anarazel.de> wrote:
>
> Hi,
>
> Attached is v2.10, with the following changes:
>
> - committed core AIO infrastructure patch
Hi, yay, It's happening.jpg ;)
Some thoughts about 2.10-0004:
What do you think about putting there into (io_uring patch) info
about the need to ensure that kernel.io_uring_disabled sysctl is on ?
(some distros might shut it down) E.g. in doc/src/sgml/config.sgml
after io_method = <listitems>... there could be
--- a/doc/src/sgml/config.sgml
+++ b/doc/src/sgml/config.sgml
<literal>io_uring</literal> (execute asynchronous I/O using
io_uring, if available)
[..]
and then add something like:
+ "At present io_method=io_uring is supported only on Linux and
requires Linux's sysctl kernel.io_uring_disabled (if present) to be at
value 0 (enabled) or 1 (with kernel.io_uring_group set to PostgreSQL's
GID)."
Rationale: it seems that at least RHEL 9.x will have this knob present
(but e.g. RHEL 8.10 even kernel-ml 6.4.2 doesn't, as this seems to
come with 6.6+ but saw somewhere that somebody had issues with this on
probably backported kernel in Rocky 9.x ). Also further googling for I
have found also that mysql can throw - when executed from
podmap/docker: "mysqld: io_uring_queue_init() failed with ENOSYS:
check seccomp filters, and the kernel version (newer than 5.1
required)"
and this leaves this with two probable follow-up questions when
adjusting this sentence:
a. shouldn't we add some sentence about containers/namespaces/seccomp
allowing this ?
b. and/or shouldn't we reference in docs a minimum kernel version
(this is somewhat wild, liburing could be installed and compiled
against, but runtime kernel would be < 5.1 ?)
-J.
^ permalink raw reply [nested|flat] 153+ messages in thread
* Re: AIO v2.5
2025-03-04 19:00 AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-10 18:23 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-11 19:41 ` Re: AIO v2.5 Noah Misch <noah@leadboat.com>
2025-03-11 23:55 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-12 03:57 ` Re: AIO v2.5 Noah Misch <noah@leadboat.com>
2025-03-12 17:06 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-12 23:23 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-13 10:53 ` Re: AIO v2.5 Antonin Houska <ah@cybertec.at>
2025-03-14 19:43 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-17 20:00 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-18 20:12 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
@ 2025-03-21 01:58 ` Andres Freund <andres@anarazel.de>
2025-03-23 00:20 ` Re: AIO v2.5 Noah Misch <noah@leadboat.com>
2025-03-23 02:09 ` Re: AIO v2.5 Noah Misch <noah@leadboat.com>
2025-03-24 00:29 ` Re: AIO v2.5 Noah Misch <noah@leadboat.com>
2025-03-25 01:18 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-25 02:20 ` Re: AIO v2.5 Noah Misch <noah@leadboat.com>
6 siblings, 5 replies; 153+ messages in thread
From: Andres Freund @ 2025-03-21 01:58 UTC (permalink / raw)
To: Antonin Houska <ah@cybertec.at>; +Cc: Noah Misch <noah@leadboat.com>; pgsql-hackers@postgresql.org, Thomas Munro <thomas.munro@gmail.com>; Heikki Linnakangas <hlinnaka@iki.fi>; Robert Haas <robertmhaas@gmail.com>; Jakub Wartak <jakub.wartak@enterprisedb.com>; Jelte Fennema-Nio <postgres@jeltef.nl>
Hi,
Attached v2.11, with the following changes:
- Pushed the smgr interrupt change, as discussed on the dedicated thread
- Pushed "bufmgr: Improve stats when a buffer is read in concurrently"
It was reviewed by Melanie and there didn't seem to be any reason to wait
further.
- Addressed feedback from Melanie
- Addressed feedback from Noah
- Added a new commit: aio: Change prefix of PgAioResultStatus values to PGAIO_RS_
As suggested/requested by Melanie. I think she's unfortunately right.
- Added a patch for some comment fixups for code that's either older or
already pushed
- Added an error check for FileStartReadV() failing
FileStartReadV() actually can fail, if the file can't be re-opened. I
thought it'd be important for the error message to differ from the one
that's issued for read actually failing, so I went with:
"could not start reading blocks %u..%u in file \"%s\": %m"
but I'm not sure how good that is.
- Added a new commit to redefine set_max_safe_fds() to not subtract
already_open fds from max_files_per_process
This prevents io_method=io_uring from failing when RLIMIT_NOFILE is high
enough, but more than max_files_per_process io_uring instances need to be
created.
- Improved error message if io_uring_queue_init() fails
Added errhint()s for likely cases of failure.
Added errcode(). I was tempted to use errcode_for_file_access(), but that
doesn't support ENOSYS - perhaps I should add that instead?
- Disable io_uring method when using EXEC_BACKEND, they're not compatible
I chose to do this with a define aio.h, but I guess we could also do it at
configure time? That seems more complicated though - how would we even know
that EXEC_BACKEND is used on non-windows?
Not sure yet how to best disable testing io_uring in this case. We can't
just query EXEC_BACKEND from pg_config.h unfortunately. I guess making the
initdb not fail and checking the error log would work, but that doesn't work
nicely with Cluster.pm.
- Changed test_aio injection short-read injection point to zero out the rest
of the IO, otherwise some tests fail to fail even if a bug in retries of
partial reads is introduced
- Improved method_io_uring.c includes a bit (no pgstat.h)
Questions:
- We only "look" at BM_IO_ERROR for writes, isn't that somewhat weird?
See AbortBufferIO(Buffer buffer)
It doesn't really matter for the patchset, but it just strikes me as an oddity.
Greetings,
Andres Freund
Attachments:
[text/x-diff] v2.11-0001-aio-bufmgr-Comment-fixes.patch (2.8K, ../../6ak556uyqiptdwjaci4kbi5eykwkmzqgkbtkyaosjnopjhncrc@2v4ac2jwyz22/2-v2.11-0001-aio-bufmgr-Comment-fixes.patch)
download | inline diff:
From 3432ae5d2421ab49bce015c6a64bb256ae36625d Mon Sep 17 00:00:00 2001
From: Andres Freund <andres@anarazel.de>
Date: Thu, 20 Mar 2025 11:32:10 -0400
Subject: [PATCH v2.11 01/27] aio, bufmgr: Comment fixes
Some of these comments have been wrong for a while (12f3867f5534), some I
recently introduced (da7226993fd, 55b454d0e14). This also updates a comment in
FlushBuffer(), which will be copied in a future commit.
These changes seem big enough to be worth doing in separate commits.
Suggested-by: Noah Misch <noah@leadboat.com>
Discussion: https://postgr.es/m/20250319212530.80.nmisch@google.com
---
src/include/storage/aio.h | 2 +-
src/backend/postmaster/postmaster.c | 2 +-
src/backend/storage/buffer/bufmgr.c | 10 ++++------
3 files changed, 6 insertions(+), 8 deletions(-)
diff --git a/src/include/storage/aio.h b/src/include/storage/aio.h
index 7b6b7d20a85..a221c6c3d9a 100644
--- a/src/include/storage/aio.h
+++ b/src/include/storage/aio.h
@@ -72,7 +72,7 @@ typedef enum PgAioHandleFlags
/*
* The IO operations supported by the AIO subsystem.
*
- * This could be in aio_internal.h, as it is not pubicly referenced, but
+ * This could be in aio_internal.h, as it is not publicly referenced, but
* PgAioOpData currently *does* need to be public, therefore keeping this
* public seems to make sense.
*/
diff --git a/src/backend/postmaster/postmaster.c b/src/backend/postmaster/postmaster.c
index a0c37532d2f..c966c2e83af 100644
--- a/src/backend/postmaster/postmaster.c
+++ b/src/backend/postmaster/postmaster.c
@@ -4401,7 +4401,7 @@ maybe_adjust_io_workers(void)
++io_worker_count;
}
else
- break; /* XXX try again soon? */
+ break; /* try again next time */
}
/* Too many running? */
diff --git a/src/backend/storage/buffer/bufmgr.c b/src/backend/storage/buffer/bufmgr.c
index 14fc1bd1248..37fb09d8b92 100644
--- a/src/backend/storage/buffer/bufmgr.c
+++ b/src/backend/storage/buffer/bufmgr.c
@@ -3864,9 +3864,10 @@ FlushBuffer(BufferDesc *buf, SMgrRelation reln, IOObject io_object,
XLogFlush(recptr);
/*
- * Now it's safe to write buffer to disk. Note that no one else should
- * have been able to write it while we were busy with log flushing because
- * only one process at a time can set the BM_IO_IN_PROGRESS bit.
+ * Now it's safe to write the buffer to disk. Note that no one else should
+ * have been able to write it, while we were busy with log flushing,
+ * because we got the exclusive right to perform I/O by setting the
+ * BM_IO_IN_PROGRESS bit.
*/
bufBlock = BufHdrGetBlock(buf);
@@ -5434,9 +5435,6 @@ IsBufferCleanupOK(Buffer buffer)
/*
* Functions for buffer I/O handling
*
- * Note: We assume that nested buffer I/O never occurs.
- * i.e at most one BM_IO_IN_PROGRESS bit is set per proc.
- *
* Also note that these are used only for shared buffers, not local ones.
*/
--
2.48.1.76.g4e746b1a31.dirty
[text/x-diff] v2.11-0002-aio-Change-prefix-of-PgAioResultStatus-values-.patch (4.0K, ../../6ak556uyqiptdwjaci4kbi5eykwkmzqgkbtkyaosjnopjhncrc@2v4ac2jwyz22/3-v2.11-0002-aio-Change-prefix-of-PgAioResultStatus-values-.patch)
download | inline diff:
From 3a7fd5b1c1e21955f1bd696873f2836e429f6801 Mon Sep 17 00:00:00 2001
From: Andres Freund <andres@anarazel.de>
Date: Wed, 19 Mar 2025 11:30:26 -0400
Subject: [PATCH v2.11 02/27] aio: Change prefix of PgAioResultStatus values to
PGAIO_RS_
The previous prefix wasn't consistent with the naming of other AIO related
enum values.
Reported-by: Melanie Plageman <melanieplageman@gmail.com>
Discussion: https://postgr.es/m/CAAKRu_Yb+JzQpNsgUxCB0gBi+sE-mi_HmcJF6ALnmO4W+UgwpA@mail.gmail.com
---
src/include/storage/aio_types.h | 8 ++++----
src/backend/storage/aio/aio.c | 12 ++++++------
src/backend/storage/aio/aio_callback.c | 6 +++---
src/backend/storage/aio/aio_init.c | 2 +-
4 files changed, 14 insertions(+), 14 deletions(-)
diff --git a/src/include/storage/aio_types.h b/src/include/storage/aio_types.h
index a5cc658efbd..dddda3a3e2f 100644
--- a/src/include/storage/aio_types.h
+++ b/src/include/storage/aio_types.h
@@ -73,10 +73,10 @@ typedef union PgAioTargetData
*/
typedef enum PgAioResultStatus
{
- ARS_UNKNOWN, /* not yet completed / uninitialized */
- ARS_OK,
- ARS_PARTIAL, /* did not fully succeed, but no error */
- ARS_ERROR,
+ PGAIO_RS_UNKNOWN, /* not yet completed / uninitialized */
+ PGAIO_RS_OK,
+ PGAIO_RS_PARTIAL, /* did not fully succeed, but no error */
+ PGAIO_RS_ERROR,
} PgAioResultStatus;
diff --git a/src/backend/storage/aio/aio.c b/src/backend/storage/aio/aio.c
index 3ed4b1dfdac..29f57f9cd1c 100644
--- a/src/backend/storage/aio/aio.c
+++ b/src/backend/storage/aio/aio.c
@@ -216,7 +216,7 @@ pgaio_io_acquire_nb(struct ResourceOwnerData *resowner, PgAioReturn *ret)
if (ret)
{
ioh->report_return = ret;
- ret->result.status = ARS_UNKNOWN;
+ ret->result.status = PGAIO_RS_UNKNOWN;
}
return ioh;
@@ -669,7 +669,7 @@ pgaio_io_reclaim(PgAioHandle *ioh)
ioh->handle_data_len = 0;
ioh->report_return = NULL;
ioh->result = 0;
- ioh->distilled_result.status = ARS_UNKNOWN;
+ ioh->distilled_result.status = PGAIO_RS_UNKNOWN;
/* XXX: the barrier is probably superfluous */
pg_write_barrier();
@@ -829,13 +829,13 @@ pgaio_result_status_string(PgAioResultStatus rs)
{
switch (rs)
{
- case ARS_UNKNOWN:
+ case PGAIO_RS_UNKNOWN:
return "UNKNOWN";
- case ARS_OK:
+ case PGAIO_RS_OK:
return "OK";
- case ARS_PARTIAL:
+ case PGAIO_RS_PARTIAL:
return "PARTIAL";
- case ARS_ERROR:
+ case PGAIO_RS_ERROR:
return "ERROR";
}
diff --git a/src/backend/storage/aio/aio_callback.c b/src/backend/storage/aio/aio_callback.c
index d5a2cca28f1..09f03f296f5 100644
--- a/src/backend/storage/aio/aio_callback.c
+++ b/src/backend/storage/aio/aio_callback.c
@@ -164,8 +164,8 @@ pgaio_result_report(PgAioResult result, const PgAioTargetData *target_data, int
PgAioHandleCallbackID cb_id = result.id;
const PgAioHandleCallbacksEntry *ce = &aio_handle_cbs[cb_id];
- Assert(result.status != ARS_UNKNOWN);
- Assert(result.status != ARS_OK);
+ Assert(result.status != PGAIO_RS_UNKNOWN);
+ Assert(result.status != PGAIO_RS_OK);
if (ce->cb->report == NULL)
elog(ERROR, "callback %d/%s does not have report callback",
@@ -220,7 +220,7 @@ pgaio_io_call_complete_shared(PgAioHandle *ioh)
Assert(ioh->target > PGAIO_TID_INVALID && ioh->target < PGAIO_TID_COUNT);
Assert(ioh->op > PGAIO_OP_INVALID && ioh->op < PGAIO_OP_COUNT);
- result.status = ARS_OK; /* low level IO is always considered OK */
+ result.status = PGAIO_RS_OK; /* low level IO is always considered OK */
result.result = ioh->result;
result.id = PGAIO_HCB_INVALID;
result.error_data = 0;
diff --git a/src/backend/storage/aio/aio_init.c b/src/backend/storage/aio/aio_init.c
index 2ede7e80b65..885c3940c66 100644
--- a/src/backend/storage/aio/aio_init.c
+++ b/src/backend/storage/aio/aio_init.c
@@ -202,7 +202,7 @@ AioShmemInit(void)
ioh->report_return = NULL;
ioh->resowner = NULL;
ioh->num_callbacks = 0;
- ioh->distilled_result.status = ARS_UNKNOWN;
+ ioh->distilled_result.status = PGAIO_RS_UNKNOWN;
ioh->flags = 0;
ConditionVariableInit(&ioh->cv);
--
2.48.1.76.g4e746b1a31.dirty
[text/x-diff] v2.11-0003-Redefine-max_files_per_process-to-control-addi.patch (4.4K, ../../6ak556uyqiptdwjaci4kbi5eykwkmzqgkbtkyaosjnopjhncrc@2v4ac2jwyz22/4-v2.11-0003-Redefine-max_files_per_process-to-control-addi.patch)
download | inline diff:
From 5f29c2c22ef09854687ca3dda6c7648090684648 Mon Sep 17 00:00:00 2001
From: Andres Freund <andres@anarazel.de>
Date: Wed, 19 Mar 2025 13:17:41 -0400
Subject: [PATCH v2.11 03/27] Redefine max_files_per_process to control
additionally opened files
Until now max_files_per_process=N limited each backend to open N files in
total (minus a safety factor), even if there were already more files opened in
postmaster and inherited by backends. Change max_files_per_process to control
how many additional files each process is allowed to open.
The main motivation for this is the patch to add io_method=io_uring, which
needs to open one file for each backend. Without this patch, even if
RLIMIT_NOFILE is high enough, postmaster will fail in set_max_safe_fds() if
started with a high max_connections. The cause of the failure is that, until
now, set_max_safe_fds() subtracted the already open files from
max_files_per_process.
Discussion: https://postgr.es/m/w6uiicyou7hzq47mbyejubtcyb2rngkkf45fk4q7inue5kfbeo@bbfad3qyubvs
Discussion: https://postgr.es/m/CAGECzQQh6VSy3KG4pN1d=h9J=D1rStFCMR+t7yh_Kwj-g87aLQ@mail.gmail.com
---
src/backend/storage/file/fd.c | 14 ++++++++------
src/backend/utils/misc/guc_tables.c | 2 +-
doc/src/sgml/config.sgml | 8 ++++++--
3 files changed, 15 insertions(+), 9 deletions(-)
diff --git a/src/backend/storage/file/fd.c b/src/backend/storage/file/fd.c
index 62f1185859f..0c3a2a756e7 100644
--- a/src/backend/storage/file/fd.c
+++ b/src/backend/storage/file/fd.c
@@ -1047,16 +1047,17 @@ set_max_safe_fds(void)
/*----------
* We want to set max_safe_fds to
- * MIN(usable_fds, max_files_per_process - already_open)
+ * MIN(usable_fds, max_files_per_process)
* less the slop factor for files that are opened without consulting
- * fd.c. This ensures that we won't exceed either max_files_per_process
- * or the experimentally-determined EMFILE limit.
+ * fd.c. This ensures that we won't allow to open more than
+ * max_files_per_process, or the experimentally-determined EMFILE limit,
+ * additional files.
*----------
*/
count_usable_fds(max_files_per_process,
&usable_fds, &already_open);
- max_safe_fds = Min(usable_fds, max_files_per_process - already_open);
+ max_safe_fds = Min(usable_fds, max_files_per_process);
/*
* Take off the FDs reserved for system() etc.
@@ -1070,9 +1071,10 @@ set_max_safe_fds(void)
ereport(FATAL,
(errcode(ERRCODE_INSUFFICIENT_RESOURCES),
errmsg("insufficient file descriptors available to start server process"),
- errdetail("System allows %d, server needs at least %d.",
+ errdetail("System allows %d, server needs at least %d, %d files are already open.",
max_safe_fds + NUM_RESERVED_FDS,
- FD_MINFREE + NUM_RESERVED_FDS)));
+ FD_MINFREE + NUM_RESERVED_FDS,
+ already_open)));
elog(DEBUG2, "max_safe_fds = %d, usable_fds = %d, already_open = %d",
max_safe_fds, usable_fds, already_open);
diff --git a/src/backend/utils/misc/guc_tables.c b/src/backend/utils/misc/guc_tables.c
index 97cfd6e5a82..75bb0acdf0f 100644
--- a/src/backend/utils/misc/guc_tables.c
+++ b/src/backend/utils/misc/guc_tables.c
@@ -2680,7 +2680,7 @@ struct config_int ConfigureNamesInt[] =
{
{"max_files_per_process", PGC_POSTMASTER, RESOURCES_KERNEL,
- gettext_noop("Sets the maximum number of simultaneously open files for each server process."),
+ gettext_noop("Sets the maximum number of files each server process is allowed to open simultaneously."),
NULL
},
&max_files_per_process,
diff --git a/doc/src/sgml/config.sgml b/doc/src/sgml/config.sgml
index bdcefa8140b..c704f3f98f1 100644
--- a/doc/src/sgml/config.sgml
+++ b/doc/src/sgml/config.sgml
@@ -2378,8 +2378,12 @@ include_dir 'conf.d'
</term>
<listitem>
<para>
- Sets the maximum number of simultaneously open files allowed to each
- server subprocess. The default is one thousand files. If the kernel is enforcing
+ Sets the maximum number of open files each server subprocess is
+ allowed to open simultaneously, in addition to the files already open
+ in postmaster. The default is one thousand files.
+ </para>
+ <para>
+ If the kernel is enforcing
a safe per-process limit, you don't need to worry about this setting.
But on some platforms (notably, most BSD systems), the kernel will
allow individual processes to open many more files than the system
--
2.48.1.76.g4e746b1a31.dirty
[text/x-diff] v2.11-0004-aio-Add-liburing-dependency.patch (13.4K, ../../6ak556uyqiptdwjaci4kbi5eykwkmzqgkbtkyaosjnopjhncrc@2v4ac2jwyz22/5-v2.11-0004-aio-Add-liburing-dependency.patch)
download | inline diff:
From e5e829d3476ac26b41cb2ee0b056150d28c8aa54 Mon Sep 17 00:00:00 2001
From: Andres Freund <andres@anarazel.de>
Date: Tue, 18 Mar 2025 14:40:05 -0400
Subject: [PATCH v2.11 04/27] aio: Add liburing dependency
Will be used in a subsequent commit, to implement io_method=io_uring. Kept
separate for easier review.
Reviewed-by: Noah Misch <noah@leadboat.com>
Discussion: https://postgr.es/m/uvrtrknj4kdytuboidbhwclo4gxhswwcpgadptsjvjqcluzmah%40brqs62irg4dt
---
meson.build | 14 ++++
meson_options.txt | 3 +
configure.ac | 11 +++
src/makefiles/meson.build | 3 +
src/include/pg_config.h.in | 3 +
src/backend/Makefile | 7 +-
doc/src/sgml/installation.sgml | 34 ++++++++
configure | 138 +++++++++++++++++++++++++++++++++
.cirrus.tasks.yml | 1 +
src/Makefile.global.in | 4 +
10 files changed, 215 insertions(+), 3 deletions(-)
diff --git a/meson.build b/meson.build
index 7cf518a2765..108e3678071 100644
--- a/meson.build
+++ b/meson.build
@@ -944,6 +944,18 @@ endif
+###############################################################
+# Library: liburing
+###############################################################
+
+liburingopt = get_option('liburing')
+liburing = dependency('liburing', required: liburingopt)
+if liburing.found()
+ cdata.set('USE_LIBURING', 1)
+endif
+
+
+
###############################################################
# Library: libxml
###############################################################
@@ -3164,6 +3176,7 @@ backend_both_deps += [
icu_i18n,
ldap,
libintl,
+ liburing,
libxml,
lz4,
pam,
@@ -3819,6 +3832,7 @@ if meson.version().version_compare('>=0.57')
'icu': icu,
'ldap': ldap,
'libcurl': libcurl,
+ 'liburing': liburing,
'libxml': libxml,
'libxslt': libxslt,
'llvm': llvm,
diff --git a/meson_options.txt b/meson_options.txt
index 702c4517145..dd7126da3a7 100644
--- a/meson_options.txt
+++ b/meson_options.txt
@@ -106,6 +106,9 @@ option('libcurl', type : 'feature', value: 'auto',
option('libedit_preferred', type: 'boolean', value: false,
description: 'Prefer BSD Libedit over GNU Readline')
+option('liburing', type : 'feature', value: 'auto',
+ description: 'io_uring support, for asynchronous I/O')
+
option('libxml', type: 'feature', value: 'auto',
description: 'XML support')
diff --git a/configure.ac b/configure.ac
index b6d02f5ecc7..3e2d0dc59c0 100644
--- a/configure.ac
+++ b/configure.ac
@@ -975,6 +975,14 @@ AC_SUBST(with_readline)
PGAC_ARG_BOOL(with, libedit-preferred, no,
[prefer BSD Libedit over GNU Readline])
+#
+# liburing
+#
+AC_MSG_CHECKING([whether to build with liburing support])
+PGAC_ARG_BOOL(with, liburing, no, [io_uring support, for asynchronous I/O],
+ [AC_DEFINE([USE_LIBURING], 1, [Define to build with io_uring support. (--with-liburing)])])
+AC_MSG_RESULT([$with_liburing])
+AC_SUBST(with_liburing)
#
# UUID library
@@ -1463,6 +1471,9 @@ elif test "$with_uuid" = ossp ; then
fi
AC_SUBST(UUID_LIBS)
+if test "$with_liburing" = yes; then
+ PKG_CHECK_MODULES(LIBURING, liburing)
+fi
##
## Header files
diff --git a/src/makefiles/meson.build b/src/makefiles/meson.build
index 60e13d50235..46d8da070e8 100644
--- a/src/makefiles/meson.build
+++ b/src/makefiles/meson.build
@@ -199,6 +199,8 @@ pgxs_empty = [
'PTHREAD_CFLAGS', 'PTHREAD_LIBS',
'ICU_LIBS',
+
+ 'LIBURING_CFLAGS', 'LIBURING_LIBS',
]
if host_system == 'windows' and cc.get_argument_syntax() != 'msvc'
@@ -230,6 +232,7 @@ pgxs_deps = {
'icu': icu,
'ldap': ldap,
'libcurl': libcurl,
+ 'liburing': liburing,
'libxml': libxml,
'libxslt': libxslt,
'llvm': llvm,
diff --git a/src/include/pg_config.h.in b/src/include/pg_config.h.in
index db6454090d2..f2422241133 100644
--- a/src/include/pg_config.h.in
+++ b/src/include/pg_config.h.in
@@ -672,6 +672,9 @@
/* Define to 1 to build with libcurl support. (--with-libcurl) */
#undef USE_LIBCURL
+/* Define to build with io_uring support. (--with-liburing) */
+#undef USE_LIBURING
+
/* Define to 1 to build with XML support. (--with-libxml) */
#undef USE_LIBXML
diff --git a/src/backend/Makefile b/src/backend/Makefile
index 42d4a28e5aa..7344c8c7f5c 100644
--- a/src/backend/Makefile
+++ b/src/backend/Makefile
@@ -43,9 +43,10 @@ OBJS = \
$(top_builddir)/src/common/libpgcommon_srv.a \
$(top_builddir)/src/port/libpgport_srv.a
-# We put libpgport and libpgcommon into OBJS, so remove it from LIBS; also add
-# libldap and ICU
-LIBS := $(filter-out -lpgport -lpgcommon, $(LIBS)) $(LDAP_LIBS_BE) $(ICU_LIBS)
+# We put libpgport and libpgcommon into OBJS, so remove it from LIBS.
+LIBS := $(filter-out -lpgport -lpgcommon, $(LIBS))
+# The backend conditionally needs libraries that most executables don't need.
+LIBS += $(LDAP_LIBS_BE) $(ICU_LIBS) $(LIBURING_LIBS)
# The backend doesn't need everything that's in LIBS, however
LIBS := $(filter-out -lreadline -ledit -ltermcap -lncurses -lcurses, $(LIBS))
diff --git a/doc/src/sgml/installation.sgml b/doc/src/sgml/installation.sgml
index e076cefa3b9..cc28f041330 100644
--- a/doc/src/sgml/installation.sgml
+++ b/doc/src/sgml/installation.sgml
@@ -1156,6 +1156,24 @@ build-postgresql:
</listitem>
</varlistentry>
+ <varlistentry id="configure-option-with-liburing">
+ <term><option>--with-liburing</option></term>
+ <listitem>
+ <para>
+ Build with liburing, enabling io_uring support for asynchronous I/O.
+ </para>
+ <para>
+ To detect the required compiler and linker options, PostgreSQL will
+ query <command>pkg-config</command>.
+ </para>
+ <para>
+ To use a liburing installation that is in an unusual location, you
+ can set <command>pkg-config</command>-related environment
+ variables (see its documentation).
+ </para>
+ </listitem>
+ </varlistentry>
+
<varlistentry id="configure-option-with-libxml">
<term><option>--with-libxml</option></term>
<listitem>
@@ -2611,6 +2629,22 @@ ninja install
</listitem>
</varlistentry>
+ <varlistentry id="configure-with-liburing-meson">
+ <term><option>-Dliburing={ auto | enabled | disabled }</option></term>
+ <listitem>
+ <para>
+ Build with liburing, enabling io_uring support for asynchronous I/O.
+ Defaults to auto.
+ </para>
+
+ <para>
+ To use a liburing installation that is in an unusual location, you
+ can set <command>pkg-config</command>-related environment
+ variables (see its documentation).
+ </para>
+ </listitem>
+ </varlistentry>
+
<varlistentry id="configure-with-libxml-meson">
<term><option>-Dlibxml={ auto | enabled | disabled }</option></term>
<listitem>
diff --git a/configure b/configure
index fac1e9a4e39..e98ed535185 100755
--- a/configure
+++ b/configure
@@ -651,6 +651,8 @@ LIBOBJS
OPENSSL
ZSTD
LZ4
+LIBURING_LIBS
+LIBURING_CFLAGS
UUID_LIBS
LDAP_LIBS_BE
LDAP_LIBS_FE
@@ -712,6 +714,7 @@ LIBCURL_LIBS
LIBCURL_CFLAGS
with_libcurl
with_uuid
+with_liburing
with_readline
with_systemd
with_selinux
@@ -865,6 +868,7 @@ with_selinux
with_systemd
with_readline
with_libedit_preferred
+with_liburing
with_uuid
with_ossp_uuid
with_libcurl
@@ -911,6 +915,8 @@ LDFLAGS_EX
LDFLAGS_SL
PERL
PYTHON
+LIBURING_CFLAGS
+LIBURING_LIBS
MSGFMT
TCLSH'
@@ -1578,6 +1584,7 @@ Optional Packages:
--without-readline do not use GNU Readline nor BSD Libedit for editing
--with-libedit-preferred
prefer BSD Libedit over GNU Readline
+ --with-liburing io_uring support, for asynchronous I/O
--with-uuid=LIB build contrib/uuid-ossp using LIB (bsd,e2fs,ossp)
--with-ossp-uuid obsolete spelling of --with-uuid=ossp
--with-libcurl build with libcurl support
@@ -1629,6 +1636,10 @@ Some influential environment variables:
LDFLAGS_SL extra linker flags for linking shared libraries only
PERL Perl program
PYTHON Python program
+ LIBURING_CFLAGS
+ C compiler flags for LIBURING, overriding pkg-config
+ LIBURING_LIBS
+ linker flags for LIBURING, overriding pkg-config
MSGFMT msgfmt program for NLS
TCLSH Tcl interpreter program (tclsh)
@@ -8692,6 +8703,40 @@ fi
+#
+# liburing
+#
+{ $as_echo "$as_me:${as_lineno-$LINENO}: checking whether to build with liburing support" >&5
+$as_echo_n "checking whether to build with liburing support... " >&6; }
+
+
+
+# Check whether --with-liburing was given.
+if test "${with_liburing+set}" = set; then :
+ withval=$with_liburing;
+ case $withval in
+ yes)
+
+$as_echo "#define USE_LIBURING 1" >>confdefs.h
+
+ ;;
+ no)
+ :
+ ;;
+ *)
+ as_fn_error $? "no argument expected for --with-liburing option" "$LINENO" 5
+ ;;
+ esac
+
+else
+ with_liburing=no
+
+fi
+
+
+{ $as_echo "$as_me:${as_lineno-$LINENO}: result: $with_liburing" >&5
+$as_echo "$with_liburing" >&6; }
+
#
# UUID library
@@ -13440,6 +13485,99 @@ fi
fi
+if test "$with_liburing" = yes; then
+
+pkg_failed=no
+{ $as_echo "$as_me:${as_lineno-$LINENO}: checking for liburing" >&5
+$as_echo_n "checking for liburing... " >&6; }
+
+if test -n "$LIBURING_CFLAGS"; then
+ pkg_cv_LIBURING_CFLAGS="$LIBURING_CFLAGS"
+ elif test -n "$PKG_CONFIG"; then
+ if test -n "$PKG_CONFIG" && \
+ { { $as_echo "$as_me:${as_lineno-$LINENO}: \$PKG_CONFIG --exists --print-errors \"liburing\""; } >&5
+ ($PKG_CONFIG --exists --print-errors "liburing") 2>&5
+ ac_status=$?
+ $as_echo "$as_me:${as_lineno-$LINENO}: \$? = $ac_status" >&5
+ test $ac_status = 0; }; then
+ pkg_cv_LIBURING_CFLAGS=`$PKG_CONFIG --cflags "liburing" 2>/dev/null`
+ test "x$?" != "x0" && pkg_failed=yes
+else
+ pkg_failed=yes
+fi
+ else
+ pkg_failed=untried
+fi
+if test -n "$LIBURING_LIBS"; then
+ pkg_cv_LIBURING_LIBS="$LIBURING_LIBS"
+ elif test -n "$PKG_CONFIG"; then
+ if test -n "$PKG_CONFIG" && \
+ { { $as_echo "$as_me:${as_lineno-$LINENO}: \$PKG_CONFIG --exists --print-errors \"liburing\""; } >&5
+ ($PKG_CONFIG --exists --print-errors "liburing") 2>&5
+ ac_status=$?
+ $as_echo "$as_me:${as_lineno-$LINENO}: \$? = $ac_status" >&5
+ test $ac_status = 0; }; then
+ pkg_cv_LIBURING_LIBS=`$PKG_CONFIG --libs "liburing" 2>/dev/null`
+ test "x$?" != "x0" && pkg_failed=yes
+else
+ pkg_failed=yes
+fi
+ else
+ pkg_failed=untried
+fi
+
+
+
+if test $pkg_failed = yes; then
+ { $as_echo "$as_me:${as_lineno-$LINENO}: result: no" >&5
+$as_echo "no" >&6; }
+
+if $PKG_CONFIG --atleast-pkgconfig-version 0.20; then
+ _pkg_short_errors_supported=yes
+else
+ _pkg_short_errors_supported=no
+fi
+ if test $_pkg_short_errors_supported = yes; then
+ LIBURING_PKG_ERRORS=`$PKG_CONFIG --short-errors --print-errors --cflags --libs "liburing" 2>&1`
+ else
+ LIBURING_PKG_ERRORS=`$PKG_CONFIG --print-errors --cflags --libs "liburing" 2>&1`
+ fi
+ # Put the nasty error message in config.log where it belongs
+ echo "$LIBURING_PKG_ERRORS" >&5
+
+ as_fn_error $? "Package requirements (liburing) were not met:
+
+$LIBURING_PKG_ERRORS
+
+Consider adjusting the PKG_CONFIG_PATH environment variable if you
+installed software in a non-standard prefix.
+
+Alternatively, you may set the environment variables LIBURING_CFLAGS
+and LIBURING_LIBS to avoid the need to call pkg-config.
+See the pkg-config man page for more details." "$LINENO" 5
+elif test $pkg_failed = untried; then
+ { $as_echo "$as_me:${as_lineno-$LINENO}: result: no" >&5
+$as_echo "no" >&6; }
+ { { $as_echo "$as_me:${as_lineno-$LINENO}: error: in \`$ac_pwd':" >&5
+$as_echo "$as_me: error: in \`$ac_pwd':" >&2;}
+as_fn_error $? "The pkg-config script could not be found or is too old. Make sure it
+is in your PATH or set the PKG_CONFIG environment variable to the full
+path to pkg-config.
+
+Alternatively, you may set the environment variables LIBURING_CFLAGS
+and LIBURING_LIBS to avoid the need to call pkg-config.
+See the pkg-config man page for more details.
+
+To get pkg-config, see <http://pkg-config.freedesktop.org/>.
+See \`config.log' for more details" "$LINENO" 5; }
+else
+ LIBURING_CFLAGS=$pkg_cv_LIBURING_CFLAGS
+ LIBURING_LIBS=$pkg_cv_LIBURING_LIBS
+ { $as_echo "$as_me:${as_lineno-$LINENO}: result: yes" >&5
+$as_echo "yes" >&6; }
+
+fi
+fi
##
## Header files
diff --git a/.cirrus.tasks.yml b/.cirrus.tasks.yml
index 5849cbb839a..10d277f5659 100644
--- a/.cirrus.tasks.yml
+++ b/.cirrus.tasks.yml
@@ -471,6 +471,7 @@ task:
--enable-cassert --enable-injection-points --enable-debug \
--enable-tap-tests --enable-nls \
--with-segsize-blocks=6 \
+ --with-liburing \
\
${LINUX_CONFIGURE_FEATURES} \
\
diff --git a/src/Makefile.global.in b/src/Makefile.global.in
index 8fe9d61e82a..fba607cb5af 100644
--- a/src/Makefile.global.in
+++ b/src/Makefile.global.in
@@ -196,6 +196,7 @@ with_gssapi = @with_gssapi@
with_krb_srvnam = @with_krb_srvnam@
with_ldap = @with_ldap@
with_libcurl = @with_libcurl@
+with_liburing = @with_liburing@
with_libxml = @with_libxml@
with_libxslt = @with_libxslt@
with_llvm = @with_llvm@
@@ -222,6 +223,9 @@ krb_srvtab = @krb_srvtab@
ICU_CFLAGS = @ICU_CFLAGS@
ICU_LIBS = @ICU_LIBS@
+LIBURING_CFLAGS = @LIBURING_CFLAGS@
+LIBURING_LIBS = @LIBURING_LIBS@
+
TCLSH = @TCLSH@
TCL_LIBS = @TCL_LIBS@
TCL_LIB_SPEC = @TCL_LIB_SPEC@
--
2.48.1.76.g4e746b1a31.dirty
[text/x-diff] v2.11-0005-aio-Add-io_method-io_uring.patch (21.4K, ../../6ak556uyqiptdwjaci4kbi5eykwkmzqgkbtkyaosjnopjhncrc@2v4ac2jwyz22/6-v2.11-0005-aio-Add-io_method-io_uring.patch)
download | inline diff:
From 3642a03831aab68a70af15478eebf3b620f802eb Mon Sep 17 00:00:00 2001
From: Andres Freund <andres@anarazel.de>
Date: Tue, 18 Mar 2025 14:40:05 -0400
Subject: [PATCH v2.11 05/27] aio: Add io_method=io_uring
io_uring, can be considerably faster than io_method=worker. It is, however,
linux specific and requires an additional compile-time dependency (liburing).
Reviewed-by: Jakub Wartak <jakub.wartak@enterprisedb.com>
Discussion: https://postgr.es/m/uvrtrknj4kdytuboidbhwclo4gxhswwcpgadptsjvjqcluzmah%40brqs62irg4dt
Discussion: https://postgr.es/m/20210223100344.llw5an2aklengrmn@alap3.anarazel.de
Discussion: https://postgr.es/m/stj36ea6yyhoxtqkhpieia2z4krnam7qyetc57rfezgk4zgapf@gcnactj4z56m
---
src/include/storage/aio.h | 8 +
src/include/storage/aio_internal.h | 3 +
src/include/storage/lwlock.h | 1 +
src/backend/storage/aio/Makefile | 1 +
src/backend/storage/aio/aio.c | 6 +
src/backend/storage/aio/meson.build | 1 +
src/backend/storage/aio/method_io_uring.c | 474 ++++++++++++++++++
src/backend/storage/lmgr/lwlock.c | 1 +
.../utils/activity/wait_event_names.txt | 2 +
src/backend/utils/misc/postgresql.conf.sample | 3 +-
doc/src/sgml/config.sgml | 6 +
.cirrus.tasks.yml | 3 +
src/tools/pgindent/typedefs.list | 1 +
13 files changed, 509 insertions(+), 1 deletion(-)
create mode 100644 src/backend/storage/aio/method_io_uring.c
diff --git a/src/include/storage/aio.h b/src/include/storage/aio.h
index a221c6c3d9a..57aa1c04b8b 100644
--- a/src/include/storage/aio.h
+++ b/src/include/storage/aio.h
@@ -22,12 +22,20 @@
#include "storage/procnumber.h"
+/* io_uring is incompatible with EXEC_BACKEND */
+#if defined(USE_LIBURING) && !defined(EXEC_BACKEND)
+#define IOMETHOD_IO_URING_ENABLED
+#endif
+
/* Enum for io_method GUC. */
typedef enum IoMethod
{
IOMETHOD_SYNC = 0,
IOMETHOD_WORKER,
+#ifdef IOMETHOD_IO_URING_ENABLED
+ IOMETHOD_IO_URING,
+#endif
} IoMethod;
/* We'll default to worker based execution. */
diff --git a/src/include/storage/aio_internal.h b/src/include/storage/aio_internal.h
index 108fe61c7b4..6e6033519b2 100644
--- a/src/include/storage/aio_internal.h
+++ b/src/include/storage/aio_internal.h
@@ -386,6 +386,9 @@ extern PgAioHandle *pgaio_inj_io_get(void);
/* Declarations for the tables of function pointers exposed by each IO method. */
extern PGDLLIMPORT const IoMethodOps pgaio_sync_ops;
extern PGDLLIMPORT const IoMethodOps pgaio_worker_ops;
+#ifdef IOMETHOD_IO_URING_ENABLED
+extern PGDLLIMPORT const IoMethodOps pgaio_uring_ops;
+#endif
extern PGDLLIMPORT const IoMethodOps *pgaio_method_ops;
extern PGDLLIMPORT PgAioCtl *pgaio_ctl;
diff --git a/src/include/storage/lwlock.h b/src/include/storage/lwlock.h
index ffa03189e2d..4df1d25c045 100644
--- a/src/include/storage/lwlock.h
+++ b/src/include/storage/lwlock.h
@@ -218,6 +218,7 @@ typedef enum BuiltinTrancheIds
LWTRANCHE_SUBTRANS_SLRU,
LWTRANCHE_XACT_SLRU,
LWTRANCHE_PARALLEL_VACUUM_DSA,
+ LWTRANCHE_AIO_URING_COMPLETION,
LWTRANCHE_FIRST_USER_DEFINED,
} BuiltinTrancheIds;
diff --git a/src/backend/storage/aio/Makefile b/src/backend/storage/aio/Makefile
index f51c34a37f8..c06c50771e0 100644
--- a/src/backend/storage/aio/Makefile
+++ b/src/backend/storage/aio/Makefile
@@ -14,6 +14,7 @@ OBJS = \
aio_init.o \
aio_io.o \
aio_target.o \
+ method_io_uring.o \
method_sync.o \
method_worker.o \
read_stream.o
diff --git a/src/backend/storage/aio/aio.c b/src/backend/storage/aio/aio.c
index 29f57f9cd1c..a30c163c3de 100644
--- a/src/backend/storage/aio/aio.c
+++ b/src/backend/storage/aio/aio.c
@@ -65,6 +65,9 @@ static void pgaio_io_wait(PgAioHandle *ioh, uint64 ref_generation);
const struct config_enum_entry io_method_options[] = {
{"sync", IOMETHOD_SYNC, false},
{"worker", IOMETHOD_WORKER, false},
+#ifdef IOMETHOD_IO_URING_ENABLED
+ {"io_uring", IOMETHOD_IO_URING, false},
+#endif
{NULL, 0, false}
};
@@ -82,6 +85,9 @@ PgAioBackend *pgaio_my_backend;
static const IoMethodOps *const pgaio_method_ops_table[] = {
[IOMETHOD_SYNC] = &pgaio_sync_ops,
[IOMETHOD_WORKER] = &pgaio_worker_ops,
+#ifdef IOMETHOD_IO_URING_ENABLED
+ [IOMETHOD_IO_URING] = &pgaio_uring_ops,
+#endif
};
/* callbacks for the configured io_method, set by assign_io_method */
diff --git a/src/backend/storage/aio/meson.build b/src/backend/storage/aio/meson.build
index 74f94c6e40b..2f0f03d8071 100644
--- a/src/backend/storage/aio/meson.build
+++ b/src/backend/storage/aio/meson.build
@@ -6,6 +6,7 @@ backend_sources += files(
'aio_init.c',
'aio_io.c',
'aio_target.c',
+ 'method_io_uring.c',
'method_sync.c',
'method_worker.c',
'read_stream.c',
diff --git a/src/backend/storage/aio/method_io_uring.c b/src/backend/storage/aio/method_io_uring.c
new file mode 100644
index 00000000000..abf5273376a
--- /dev/null
+++ b/src/backend/storage/aio/method_io_uring.c
@@ -0,0 +1,474 @@
+/*-------------------------------------------------------------------------
+ *
+ * method_io_uring.c
+ * AIO - perform AIO using Linux' io_uring
+ *
+ * For now we create one io_uring instance for each backend. These io_uring
+ * instances have to be created in postmaster, during startup, to allow other
+ * backends to process IO completions, if the issuing backend is currently
+ * busy doing other things. Other backends may not use another backend's
+ * io_uring instance to submit IO, that'd require additional locking that
+ * would likely be harmful for performance.
+ *
+ * We likely will want to introduce a backend-local io_uring instance in the
+ * future, e.g. for FE/BE network IO.
+ *
+ * Portions Copyright (c) 1996-2025, PostgreSQL Global Development Group
+ * Portions Copyright (c) 1994, Regents of the University of California
+ *
+ * IDENTIFICATION
+ * src/backend/storage/aio/method_io_uring.c
+ *
+ *-------------------------------------------------------------------------
+ */
+
+#include "postgres.h"
+
+/* included early, for IOMETHOD_IO_URING_ENABLED */
+#include "storage/aio.h"
+
+#ifdef IOMETHOD_IO_URING_ENABLED
+
+#include <liburing.h>
+
+#include "miscadmin.h"
+#include "port/pg_iovec.h"
+#include "storage/aio_internal.h"
+#include "storage/fd.h"
+#include "storage/proc.h"
+#include "storage/shmem.h"
+#include "utils/wait_event.h"
+
+
+/* number of completions processed at once */
+#define PGAIO_MAX_LOCAL_COMPLETED_IO 32
+
+
+/* Entry points for IoMethodOps. */
+static size_t pgaio_uring_shmem_size(void);
+static void pgaio_uring_shmem_init(bool first_time);
+static void pgaio_uring_init_backend(void);
+static int pgaio_uring_submit(uint16 num_staged_ios, PgAioHandle **staged_ios);
+static void pgaio_uring_wait_one(PgAioHandle *ioh, uint64 ref_generation);
+
+/* helper functions */
+static void pgaio_uring_sq_from_io(PgAioHandle *ioh, struct io_uring_sqe *sqe);
+
+
+const IoMethodOps pgaio_uring_ops = {
+ .shmem_size = pgaio_uring_shmem_size,
+ .shmem_init = pgaio_uring_shmem_init,
+ .init_backend = pgaio_uring_init_backend,
+
+ .submit = pgaio_uring_submit,
+ .wait_one = pgaio_uring_wait_one,
+};
+
+/*
+ * Per-backend state when using io_method=io_uring
+ *
+ * Align the whole struct to a cacheline boundary, to prevent false sharing
+ * between completion_lock and prior backend's io_uring_ring.
+ */
+typedef struct pg_attribute_aligned (PG_CACHE_LINE_SIZE)
+PgAioUringContext
+{
+ /*
+ * Multiple backends can process completions for this backend's io_uring
+ * instance (e.g. when the backend issuing IO is busy doing something
+ * else). To make that safe we have to ensure that only a single backend
+ * gets io completions from the io_uring instance at a time.
+ */
+ LWLock completion_lock;
+
+ struct io_uring io_uring_ring;
+} PgAioUringContext;
+
+/* PgAioUringContexts for all backends */
+static PgAioUringContext *pgaio_uring_contexts;
+
+/* the current backend's context */
+static PgAioUringContext *pgaio_my_uring_context;
+
+
+static uint32
+pgaio_uring_procs(void)
+{
+ /*
+ * We can subtract MAX_IO_WORKERS here as io workers are never used at the
+ * same time as io_method=io_uring.
+ */
+ return MaxBackends + NUM_AUXILIARY_PROCS - MAX_IO_WORKERS;
+}
+
+static Size
+pgaio_uring_context_shmem_size(void)
+{
+ return mul_size(pgaio_uring_procs(), sizeof(PgAioUringContext));
+}
+
+static size_t
+pgaio_uring_shmem_size(void)
+{
+ return pgaio_uring_context_shmem_size();
+}
+
+static void
+pgaio_uring_shmem_init(bool first_time)
+{
+ int TotalProcs = MaxBackends + NUM_AUXILIARY_PROCS - MAX_IO_WORKERS;
+ bool found;
+
+ pgaio_uring_contexts = (PgAioUringContext *)
+ ShmemInitStruct("AioUring", pgaio_uring_shmem_size(), &found);
+
+ if (found)
+ return;
+
+ for (int contextno = 0; contextno < TotalProcs; contextno++)
+ {
+ PgAioUringContext *context = &pgaio_uring_contexts[contextno];
+ int ret;
+
+ /*
+ * Right now a high TotalProcs will cause problems in two ways:
+ *
+ * - RLIMIT_NOFILE needs to be big enough to allow all
+ * io_uring_queue_init() calls to succeed.
+ *
+ * - RLIMIT_NOFILE needs to be big enough to still have enough file
+ * descriptors to satisfy set_max_safe_fds() left over. Or, even
+ * better, have max_files_per_process left over FDs.
+ *
+ * We probably should adjust the soft RLIMIT_NOFILE to ensure that.
+ *
+ *
+ * XXX: Newer versions of io_uring support sharing the workers that
+ * execute some asynchronous IOs between io_uring instances. It might
+ * be worth using that - also need to evaluate if that causes
+ * noticeable additional contention?
+ */
+ ret = io_uring_queue_init(io_max_concurrency, &context->io_uring_ring, 0);
+ if (ret < 0)
+ {
+ char *hint = NULL;
+ int err = ERRCODE_INTERNAL_ERROR;
+
+ /* add hints for some failures that errno explains sufficiently */
+ if (-ret == EPERM)
+ {
+ err = ERRCODE_INSUFFICIENT_PRIVILEGE;
+ hint = _("Check if io_uring is disabled via /proc/sys/kernel/io_uring_disabled.");
+ }
+ else if (-ret == EMFILE)
+ {
+ err = ERRCODE_INSUFFICIENT_RESOURCES;
+ hint = psprintf(_("Consider increasing \"ulimit -n\" to at least %d."),
+ TotalProcs + max_files_per_process);
+ }
+ else if (-ret == ENOSYS)
+ {
+ err = ERRCODE_FEATURE_NOT_SUPPORTED;
+ hint = _("Kernel does not support io_uring.");
+ }
+
+ /* update errno to allow %m to work */
+ errno = -ret;
+
+ ereport(ERROR,
+ errcode(err),
+ errmsg("io_uring_queue_init failed: %m"),
+ hint != NULL ? errhint("%s", hint) : 0);
+ }
+
+ LWLockInitialize(&context->completion_lock, LWTRANCHE_AIO_URING_COMPLETION);
+ }
+}
+
+static void
+pgaio_uring_init_backend(void)
+{
+ Assert(MyProcNumber < pgaio_uring_procs());
+
+ pgaio_my_uring_context = &pgaio_uring_contexts[MyProcNumber];
+}
+
+static int
+pgaio_uring_submit(uint16 num_staged_ios, PgAioHandle **staged_ios)
+{
+ struct io_uring *uring_instance = &pgaio_my_uring_context->io_uring_ring;
+ int in_flight_before = dclist_count(&pgaio_my_backend->in_flight_ios);
+
+ Assert(num_staged_ios <= PGAIO_SUBMIT_BATCH_SIZE);
+
+ for (int i = 0; i < num_staged_ios; i++)
+ {
+ PgAioHandle *ioh = staged_ios[i];
+ struct io_uring_sqe *sqe;
+
+ sqe = io_uring_get_sqe(uring_instance);
+
+ if (!sqe)
+ elog(ERROR, "io_uring submission queue is unexpectedly full");
+
+ pgaio_io_prepare_submit(ioh);
+ pgaio_uring_sq_from_io(ioh, sqe);
+
+ /*
+ * io_uring executes IO in process context if possible. That's
+ * generally good, as it reduces context switching. When performing a
+ * lot of buffered IO that means that copying between page cache and
+ * userspace memory happens in the foreground, as it can't be
+ * offloaded to DMA hardware as is possible when using direct IO. When
+ * executing a lot of buffered IO this causes io_uring to be slower
+ * than worker mode, as worker mode parallelizes the copying. io_uring
+ * can be told to offload work to worker threads instead.
+ *
+ * If an IO is buffered IO and we already have IOs in flight or
+ * multiple IOs are being submitted, we thus tell io_uring to execute
+ * the IO in the background. We don't do so for the first few IOs
+ * being submitted as executing in this process' context has lower
+ * latency.
+ */
+ if (in_flight_before > 4 && (ioh->flags & PGAIO_HF_BUFFERED))
+ io_uring_sqe_set_flags(sqe, IOSQE_ASYNC);
+
+ in_flight_before++;
+ }
+
+ while (true)
+ {
+ int ret;
+
+ pgstat_report_wait_start(WAIT_EVENT_AIO_IO_URING_SUBMIT);
+ ret = io_uring_submit(uring_instance);
+ pgstat_report_wait_end();
+
+ if (ret == -EINTR)
+ {
+ pgaio_debug(DEBUG3,
+ "aio method uring: submit EINTR, nios: %d",
+ num_staged_ios);
+ }
+ else if (ret < 0)
+ {
+ /*
+ * The io_uring_enter() manpage suggests that the appropriate
+ * reaction to EAGAIN is:
+ *
+ * "The application should wait for some completions and try
+ * again"
+ *
+ * However, it seems unlikely that that would help in our case, as
+ * we apply a low limit to the number of outstanding IOs and thus
+ * also outstanding completions, making it unlikely that we'd get
+ * EAGAIN while the OS is in good working order.
+ *
+ * Additionally, it would be problematic to just wait here, our
+ * caller might hold critical locks. It'd possibly lead to
+ * delaying the crash-restart that seems likely to occur when the
+ * kernel is under such heavy memory pressure.
+ *
+ * Update errno to allow %m to work.
+ */
+ errno = -ret;
+ elog(PANIC, "io_uring submit failed: %m");
+ }
+ else if (ret != num_staged_ios)
+ {
+ /* likely unreachable, but if it is, we would need to re-submit */
+ elog(PANIC, "io_uring submit submitted only %d of %d",
+ ret, num_staged_ios);
+ }
+ else
+ {
+ pgaio_debug(DEBUG4,
+ "aio method uring: submitted %d IOs",
+ num_staged_ios);
+ break;
+ }
+ }
+
+ return num_staged_ios;
+}
+
+static void
+pgaio_uring_drain_locked(PgAioUringContext *context)
+{
+ int ready;
+ int orig_ready;
+
+ Assert(LWLockHeldByMeInMode(&context->completion_lock, LW_EXCLUSIVE));
+
+ /*
+ * Don't drain more events than available right now. Otherwise it's
+ * plausible that one backend could get stuck, for a while, receiving CQEs
+ * without actually processing them.
+ */
+ orig_ready = ready = io_uring_cq_ready(&context->io_uring_ring);
+
+ while (ready > 0)
+ {
+ struct io_uring_cqe *cqes[PGAIO_MAX_LOCAL_COMPLETED_IO];
+ uint32 ncqes;
+
+ START_CRIT_SECTION();
+ ncqes =
+ io_uring_peek_batch_cqe(&context->io_uring_ring,
+ cqes,
+ Min(PGAIO_MAX_LOCAL_COMPLETED_IO, ready));
+ Assert(ncqes <= ready);
+
+ ready -= ncqes;
+
+ for (int i = 0; i < ncqes; i++)
+ {
+ struct io_uring_cqe *cqe = cqes[i];
+ PgAioHandle *ioh;
+
+ ioh = io_uring_cqe_get_data(cqe);
+ io_uring_cqe_seen(&context->io_uring_ring, cqe);
+
+ pgaio_io_process_completion(ioh, cqe->res);
+ }
+
+ END_CRIT_SECTION();
+
+ pgaio_debug(DEBUG3,
+ "drained %d/%d, now expecting %d",
+ ncqes, orig_ready, io_uring_cq_ready(&context->io_uring_ring));
+ }
+}
+
+static void
+pgaio_uring_wait_one(PgAioHandle *ioh, uint64 ref_generation)
+{
+ PgAioHandleState state;
+ ProcNumber owner_procno = ioh->owner_procno;
+ PgAioUringContext *owner_context = &pgaio_uring_contexts[owner_procno];
+ bool expect_cqe;
+ int waited = 0;
+
+ /*
+ * XXX: It would be nice to have a smarter locking scheme, nearly all the
+ * time the backend owning the ring will consume the completions, making
+ * the locking unnecessarily expensive.
+ */
+ LWLockAcquire(&owner_context->completion_lock, LW_EXCLUSIVE);
+
+ while (true)
+ {
+ pgaio_debug_io(DEBUG3, ioh,
+ "wait_one io_gen: %llu, ref_gen: %llu, cycle %d",
+ (long long unsigned) ref_generation,
+ (long long unsigned) ioh->generation,
+ waited);
+
+ if (pgaio_io_was_recycled(ioh, ref_generation, &state) ||
+ state != PGAIO_HS_SUBMITTED)
+ {
+ /* the IO was completed by another backend */
+ break;
+ }
+ else if (io_uring_cq_ready(&owner_context->io_uring_ring))
+ {
+ /* no need to wait in the kernel, io_uring has a completion */
+ expect_cqe = true;
+ }
+ else
+ {
+ int ret;
+ struct io_uring_cqe *cqes;
+
+ /* need to wait in the kernel */
+ pgstat_report_wait_start(WAIT_EVENT_AIO_IO_URING_COMPLETION);
+ ret = io_uring_wait_cqes(&owner_context->io_uring_ring, &cqes, 1, NULL, NULL);
+ pgstat_report_wait_end();
+
+ if (ret == -EINTR)
+ {
+ continue;
+ }
+ else if (ret != 0)
+ {
+ /* see comment after io_uring_submit() */
+ errno = -ret;
+ elog(PANIC, "io_uring wait failed: %m");
+ }
+ else
+ {
+ Assert(cqes != NULL);
+ expect_cqe = true;
+ waited++;
+ }
+ }
+
+ if (expect_cqe)
+ {
+ pgaio_uring_drain_locked(owner_context);
+ }
+ }
+
+ LWLockRelease(&owner_context->completion_lock);
+
+ pgaio_debug(DEBUG3,
+ "wait_one with %d sleeps",
+ waited);
+}
+
+static void
+pgaio_uring_sq_from_io(PgAioHandle *ioh, struct io_uring_sqe *sqe)
+{
+ struct iovec *iov;
+
+ switch (ioh->op)
+ {
+ case PGAIO_OP_READV:
+ iov = &pgaio_ctl->iovecs[ioh->iovec_off];
+ if (ioh->op_data.read.iov_length == 1)
+ {
+ io_uring_prep_read(sqe,
+ ioh->op_data.read.fd,
+ iov->iov_base,
+ iov->iov_len,
+ ioh->op_data.read.offset);
+ }
+ else
+ {
+ io_uring_prep_readv(sqe,
+ ioh->op_data.read.fd,
+ iov,
+ ioh->op_data.read.iov_length,
+ ioh->op_data.read.offset);
+
+ }
+ break;
+
+ case PGAIO_OP_WRITEV:
+ iov = &pgaio_ctl->iovecs[ioh->iovec_off];
+ if (ioh->op_data.write.iov_length == 1)
+ {
+ io_uring_prep_write(sqe,
+ ioh->op_data.write.fd,
+ iov->iov_base,
+ iov->iov_len,
+ ioh->op_data.write.offset);
+ }
+ else
+ {
+ io_uring_prep_writev(sqe,
+ ioh->op_data.write.fd,
+ iov,
+ ioh->op_data.write.iov_length,
+ ioh->op_data.write.offset);
+ }
+ break;
+
+ case PGAIO_OP_INVALID:
+ elog(ERROR, "trying to prepare invalid IO operation for execution");
+ }
+
+ io_uring_sqe_set_data(sqe, ioh);
+}
+
+#endif /* IOMETHOD_IO_URING_ENABLED */
diff --git a/src/backend/storage/lmgr/lwlock.c b/src/backend/storage/lmgr/lwlock.c
index 5702c35bb91..3df29658f18 100644
--- a/src/backend/storage/lmgr/lwlock.c
+++ b/src/backend/storage/lmgr/lwlock.c
@@ -177,6 +177,7 @@ static const char *const BuiltinTrancheNames[] = {
[LWTRANCHE_SUBTRANS_SLRU] = "SubtransSLRU",
[LWTRANCHE_XACT_SLRU] = "XactSLRU",
[LWTRANCHE_PARALLEL_VACUUM_DSA] = "ParallelVacuumDSA",
+ [LWTRANCHE_AIO_URING_COMPLETION] = "AioUringCompletion",
};
StaticAssertDecl(lengthof(BuiltinTrancheNames) ==
diff --git a/src/backend/utils/activity/wait_event_names.txt b/src/backend/utils/activity/wait_event_names.txt
index 9fa12a555e8..44eef41f841 100644
--- a/src/backend/utils/activity/wait_event_names.txt
+++ b/src/backend/utils/activity/wait_event_names.txt
@@ -192,6 +192,8 @@ ABI_compatibility:
Section: ClassName - WaitEventIO
+AIO_IO_URING_SUBMIT "Waiting for IO submission via io_uring."
+AIO_IO_URING_COMPLETION "Waiting for IO completion via io_uring."
AIO_IO_COMPLETION "Waiting for IO completion."
BASEBACKUP_READ "Waiting for base backup to read from a file."
BASEBACKUP_SYNC "Waiting for data written by a base backup to reach durable storage."
diff --git a/src/backend/utils/misc/postgresql.conf.sample b/src/backend/utils/misc/postgresql.conf.sample
index 9f31e4071c7..69dc727174b 100644
--- a/src/backend/utils/misc/postgresql.conf.sample
+++ b/src/backend/utils/misc/postgresql.conf.sample
@@ -204,7 +204,8 @@
# (change requires restart)
#io_combine_limit = 128kB # usually 1-128 blocks (depends on OS)
-#io_method = worker # worker, sync (change requires restart)
+#io_method = worker # worker, io_uring, sync
+ # (change requires restart)
#io_max_concurrency = -1 # Max number of IOs that one process
# can execute simultaneously
# -1 sets based on shared_buffers
diff --git a/doc/src/sgml/config.sgml b/doc/src/sgml/config.sgml
index c704f3f98f1..4f71725036b 100644
--- a/doc/src/sgml/config.sgml
+++ b/doc/src/sgml/config.sgml
@@ -2710,6 +2710,12 @@ include_dir 'conf.d'
<literal>worker</literal> (execute asynchronous I/O using worker processes)
</para>
</listitem>
+ <listitem>
+ <para>
+ <literal>io_uring</literal> (execute asynchronous I/O using
+ io_uring, if available)
+ </para>
+ </listitem>
<listitem>
<para>
<literal>sync</literal> (execute asynchronous-eligible I/O synchronously)
diff --git a/.cirrus.tasks.yml b/.cirrus.tasks.yml
index 10d277f5659..86a1fa9bbdb 100644
--- a/.cirrus.tasks.yml
+++ b/.cirrus.tasks.yml
@@ -493,11 +493,14 @@ task:
# - Uses undefined behaviour and alignment sanitizers, sanitizer failures
# are typically printed in the server log
# - Test both 64bit and 32 bit builds
+ # - uses io_method=io_uring
- name: Linux - Debian Bookworm - Meson
env:
CCACHE_MAXSIZE: "400M" # tests two different builds
SANITIZER_FLAGS: -fsanitize=alignment,undefined
+ PG_TEST_INITDB_EXTRA_OPTS: >-
+ -c io_method=io_uring
configure_script: |
su postgres <<-EOF
diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list
index bfa276d2d35..53abd54edd8 100644
--- a/src/tools/pgindent/typedefs.list
+++ b/src/tools/pgindent/typedefs.list
@@ -2150,6 +2150,7 @@ PgAioReturn
PgAioTargetData
PgAioTargetID
PgAioTargetInfo
+PgAioUringContext
PgAioWaitRef
PgArchData
PgBackendGSSStatus
--
2.48.1.76.g4e746b1a31.dirty
[text/x-diff] v2.11-0006-aio-Implement-support-for-reads-in-smgr-md-fd.patch (22.8K, ../../6ak556uyqiptdwjaci4kbi5eykwkmzqgkbtkyaosjnopjhncrc@2v4ac2jwyz22/7-v2.11-0006-aio-Implement-support-for-reads-in-smgr-md-fd.patch)
download | inline diff:
From 5d65d6a0c7588f392305c40d16ed23dfa49927aa Mon Sep 17 00:00:00 2001
From: Andres Freund <andres@anarazel.de>
Date: Tue, 18 Mar 2025 14:40:05 -0400
Subject: [PATCH v2.11 06/27] aio: Implement support for reads in smgr/md/fd
This implements the following:
1) An smgr AIO target, for AIO on smgr files. This should be usable not just
for md.c but also other SMGR implementation if we ever get them.
2) readv support in fd.c, which requires a small bit of infrastructure work in
fd.c
3) smgr.c and md.c support for readv
There still is nothing performing AIO, but as of this commit it would be
possible.
As part of this change FileGetRawDesc() actually ensures that the file is
opened - previously it was basically not usable. It's used to reopen a file in
IO workers.
Reviewed-by: Noah Misch <noah@leadboat.com>
Discussion: https://postgr.es/m/uvrtrknj4kdytuboidbhwclo4gxhswwcpgadptsjvjqcluzmah%40brqs62irg4dt
Discussion: https://postgr.es/m/20210223100344.llw5an2aklengrmn@alap3.anarazel.de
Discussion: https://postgr.es/m/stj36ea6yyhoxtqkhpieia2z4krnam7qyetc57rfezgk4zgapf@gcnactj4z56m
---
src/include/storage/aio.h | 6 +-
src/include/storage/aio_types.h | 10 +-
src/include/storage/fd.h | 3 +
src/include/storage/md.h | 7 +
src/include/storage/smgr.h | 14 ++
src/backend/storage/aio/aio_callback.c | 3 +
src/backend/storage/aio/aio_target.c | 2 +
src/backend/storage/file/fd.c | 40 ++++++
src/backend/storage/smgr/md.c | 186 +++++++++++++++++++++++++
src/backend/storage/smgr/smgr.c | 150 ++++++++++++++++++++
10 files changed, 417 insertions(+), 4 deletions(-)
diff --git a/src/include/storage/aio.h b/src/include/storage/aio.h
index 57aa1c04b8b..1591a6c2fca 100644
--- a/src/include/storage/aio.h
+++ b/src/include/storage/aio.h
@@ -117,9 +117,10 @@ typedef enum PgAioTargetID
{
/* intentionally the zero value, to help catch zeroed memory etc */
PGAIO_TID_INVALID = 0,
+ PGAIO_TID_SMGR,
} PgAioTargetID;
-#define PGAIO_TID_COUNT (PGAIO_TID_INVALID + 1)
+#define PGAIO_TID_COUNT (PGAIO_TID_SMGR + 1)
/*
@@ -191,6 +192,9 @@ struct PgAioTargetInfo
typedef enum PgAioHandleCallbackID
{
PGAIO_HCB_INVALID,
+
+ PGAIO_HCB_MD_READV,
+ PGAIO_HCB_MD_WRITEV,
} PgAioHandleCallbackID;
diff --git a/src/include/storage/aio_types.h b/src/include/storage/aio_types.h
index dddda3a3e2f..debe8163d4e 100644
--- a/src/include/storage/aio_types.h
+++ b/src/include/storage/aio_types.h
@@ -60,11 +60,15 @@ typedef struct PgAioWaitRef
*/
typedef union PgAioTargetData
{
- /* just as an example placeholder for later */
struct
{
- uint32 queue_id;
- } wal;
+ RelFileLocator rlocator; /* physical relation identifier */
+ BlockNumber blockNum; /* blknum relative to begin of reln */
+ BlockNumber nblocks;
+ ForkNumber forkNum:8; /* don't waste 4 byte for four values */
+ bool is_temp:1; /* proc can be inferred by owning AIO */
+ bool skip_fsync:1;
+ } smgr;
} PgAioTargetData;
diff --git a/src/include/storage/fd.h b/src/include/storage/fd.h
index e3067ab6597..b77d8e5e30e 100644
--- a/src/include/storage/fd.h
+++ b/src/include/storage/fd.h
@@ -101,6 +101,8 @@ extern PGDLLIMPORT int max_safe_fds;
* prototypes for functions in fd.c
*/
+struct PgAioHandle;
+
/* Operations on virtual Files --- equivalent to Unix kernel file ops */
extern File PathNameOpenFile(const char *fileName, int fileFlags);
extern File PathNameOpenFilePerm(const char *fileName, int fileFlags, mode_t fileMode);
@@ -109,6 +111,7 @@ extern void FileClose(File file);
extern int FilePrefetch(File file, off_t offset, off_t amount, uint32 wait_event_info);
extern ssize_t FileReadV(File file, const struct iovec *iov, int iovcnt, off_t offset, uint32 wait_event_info);
extern ssize_t FileWriteV(File file, const struct iovec *iov, int iovcnt, off_t offset, uint32 wait_event_info);
+extern int FileStartReadV(struct PgAioHandle *ioh, File file, int iovcnt, off_t offset, uint32 wait_event_info);
extern int FileSync(File file, uint32 wait_event_info);
extern int FileZero(File file, off_t offset, off_t amount, uint32 wait_event_info);
extern int FileFallocate(File file, off_t offset, off_t amount, uint32 wait_event_info);
diff --git a/src/include/storage/md.h b/src/include/storage/md.h
index 05bf537066e..9d7131eff43 100644
--- a/src/include/storage/md.h
+++ b/src/include/storage/md.h
@@ -14,11 +14,14 @@
#ifndef MD_H
#define MD_H
+#include "storage/aio_types.h"
#include "storage/block.h"
#include "storage/relfilelocator.h"
#include "storage/smgr.h"
#include "storage/sync.h"
+extern const PgAioHandleCallbacks aio_md_readv_cb;
+
/* md storage manager functionality */
extern void mdinit(void);
extern void mdopen(SMgrRelation reln);
@@ -36,6 +39,9 @@ extern uint32 mdmaxcombine(SMgrRelation reln, ForkNumber forknum,
BlockNumber blocknum);
extern void mdreadv(SMgrRelation reln, ForkNumber forknum, BlockNumber blocknum,
void **buffers, BlockNumber nblocks);
+extern void mdstartreadv(PgAioHandle *ioh,
+ SMgrRelation reln, ForkNumber forknum, BlockNumber blocknum,
+ void **buffers, BlockNumber nblocks);
extern void mdwritev(SMgrRelation reln, ForkNumber forknum,
BlockNumber blocknum,
const void **buffers, BlockNumber nblocks, bool skipFsync);
@@ -46,6 +52,7 @@ extern void mdtruncate(SMgrRelation reln, ForkNumber forknum,
BlockNumber old_blocks, BlockNumber nblocks);
extern void mdimmedsync(SMgrRelation reln, ForkNumber forknum);
extern void mdregistersync(SMgrRelation reln, ForkNumber forknum);
+extern int mdfd(SMgrRelation reln, ForkNumber forknum, BlockNumber blocknum, uint32 *off);
extern void ForgetDatabaseSyncRequests(Oid dbid);
extern void DropRelationFiles(RelFileLocator *delrels, int ndelrels, bool isRedo);
diff --git a/src/include/storage/smgr.h b/src/include/storage/smgr.h
index 4016b206ad6..856ebcda350 100644
--- a/src/include/storage/smgr.h
+++ b/src/include/storage/smgr.h
@@ -15,6 +15,7 @@
#define SMGR_H
#include "lib/ilist.h"
+#include "storage/aio_types.h"
#include "storage/block.h"
#include "storage/relfilelocator.h"
@@ -73,6 +74,8 @@ typedef SMgrRelationData *SMgrRelation;
#define SmgrIsTemp(smgr) \
RelFileLocatorBackendIsTemp((smgr)->smgr_rlocator)
+extern const PgAioTargetInfo aio_smgr_target_info;
+
extern void smgrinit(void);
extern SMgrRelation smgropen(RelFileLocator rlocator, ProcNumber backend);
extern bool smgrexists(SMgrRelation reln, ForkNumber forknum);
@@ -97,6 +100,10 @@ extern uint32 smgrmaxcombine(SMgrRelation reln, ForkNumber forknum,
extern void smgrreadv(SMgrRelation reln, ForkNumber forknum,
BlockNumber blocknum,
void **buffers, BlockNumber nblocks);
+extern void smgrstartreadv(PgAioHandle *ioh,
+ SMgrRelation reln, ForkNumber forknum,
+ BlockNumber blocknum,
+ void **buffers, BlockNumber nblocks);
extern void smgrwritev(SMgrRelation reln, ForkNumber forknum,
BlockNumber blocknum,
const void **buffers, BlockNumber nblocks,
@@ -127,4 +134,11 @@ smgrwrite(SMgrRelation reln, ForkNumber forknum, BlockNumber blocknum,
smgrwritev(reln, forknum, blocknum, &buffer, 1, skipFsync);
}
+extern void pgaio_io_set_target_smgr(PgAioHandle *ioh,
+ SMgrRelationData *smgr,
+ ForkNumber forknum,
+ BlockNumber blocknum,
+ int nblocks,
+ bool skip_fsync);
+
#endif /* SMGR_H */
diff --git a/src/backend/storage/aio/aio_callback.c b/src/backend/storage/aio/aio_callback.c
index 09f03f296f5..a92fa8f678e 100644
--- a/src/backend/storage/aio/aio_callback.c
+++ b/src/backend/storage/aio/aio_callback.c
@@ -18,6 +18,7 @@
#include "miscadmin.h"
#include "storage/aio.h"
#include "storage/aio_internal.h"
+#include "storage/md.h"
/* just to have something to put into aio_handle_cbs */
@@ -37,6 +38,8 @@ typedef struct PgAioHandleCallbacksEntry
static const PgAioHandleCallbacksEntry aio_handle_cbs[] = {
#define CALLBACK_ENTRY(id, callback) [id] = {.cb = &callback, .name = #callback}
CALLBACK_ENTRY(PGAIO_HCB_INVALID, aio_invalid_cb),
+
+ CALLBACK_ENTRY(PGAIO_HCB_MD_READV, aio_md_readv_cb),
#undef CALLBACK_ENTRY
};
diff --git a/src/backend/storage/aio/aio_target.c b/src/backend/storage/aio/aio_target.c
index b01406a6a52..6133b04f350 100644
--- a/src/backend/storage/aio/aio_target.c
+++ b/src/backend/storage/aio/aio_target.c
@@ -16,6 +16,7 @@
#include "storage/aio.h"
#include "storage/aio_internal.h"
+#include "storage/smgr.h"
/*
@@ -25,6 +26,7 @@ static const PgAioTargetInfo *pgaio_target_info[] = {
[PGAIO_TID_INVALID] = &(PgAioTargetInfo) {
.name = "invalid",
},
+ [PGAIO_TID_SMGR] = &aio_smgr_target_info,
};
diff --git a/src/backend/storage/file/fd.c b/src/backend/storage/file/fd.c
index 0c3a2a756e7..f573aa8e7ca 100644
--- a/src/backend/storage/file/fd.c
+++ b/src/backend/storage/file/fd.c
@@ -94,6 +94,7 @@
#include "miscadmin.h"
#include "pgstat.h"
#include "postmaster/startup.h"
+#include "storage/aio.h"
#include "storage/fd.h"
#include "storage/ipc.h"
#include "utils/guc.h"
@@ -1296,6 +1297,8 @@ LruDelete(File file)
vfdP = &VfdCache[file];
+ pgaio_closing_fd(vfdP->fd);
+
/*
* Close the file. We aren't expecting this to fail; if it does, better
* to leak the FD than to mess up our internal state.
@@ -1989,6 +1992,8 @@ FileClose(File file)
if (!FileIsNotOpen(file))
{
+ pgaio_closing_fd(vfdP->fd);
+
/* close the file */
if (close(vfdP->fd) != 0)
{
@@ -2212,6 +2217,32 @@ retry:
return returnCode;
}
+int
+FileStartReadV(PgAioHandle *ioh, File file,
+ int iovcnt, off_t offset,
+ uint32 wait_event_info)
+{
+ int returnCode;
+ Vfd *vfdP;
+
+ Assert(FileIsValid(file));
+
+ DO_DB(elog(LOG, "FileStartReadV: %d (%s) " INT64_FORMAT " %d",
+ file, VfdCache[file].fileName,
+ (int64) offset,
+ iovcnt));
+
+ returnCode = FileAccess(file);
+ if (returnCode < 0)
+ return returnCode;
+
+ vfdP = &VfdCache[file];
+
+ pgaio_io_prep_readv(ioh, vfdP->fd, iovcnt, offset);
+
+ return 0;
+}
+
ssize_t
FileWriteV(File file, const struct iovec *iov, int iovcnt, off_t offset,
uint32 wait_event_info)
@@ -2500,6 +2531,12 @@ FilePathName(File file)
int
FileGetRawDesc(File file)
{
+ int returnCode;
+
+ returnCode = FileAccess(file);
+ if (returnCode < 0)
+ return returnCode;
+
Assert(FileIsValid(file));
return VfdCache[file].fd;
}
@@ -2780,6 +2817,7 @@ FreeDesc(AllocateDesc *desc)
result = closedir(desc->desc.dir);
break;
case AllocateDescRawFD:
+ pgaio_closing_fd(desc->desc.fd);
result = close(desc->desc.fd);
break;
default:
@@ -2848,6 +2886,8 @@ CloseTransientFile(int fd)
/* Only get here if someone passes us a file not in allocatedDescs */
elog(WARNING, "fd passed to CloseTransientFile was not obtained from OpenTransientFile");
+ pgaio_closing_fd(fd);
+
return close(fd);
}
diff --git a/src/backend/storage/smgr/md.c b/src/backend/storage/smgr/md.c
index f3220f98dc4..2218f17e7a0 100644
--- a/src/backend/storage/smgr/md.c
+++ b/src/backend/storage/smgr/md.c
@@ -31,6 +31,7 @@
#include "miscadmin.h"
#include "pg_trace.h"
#include "pgstat.h"
+#include "storage/aio.h"
#include "storage/bufmgr.h"
#include "storage/fd.h"
#include "storage/md.h"
@@ -152,6 +153,15 @@ static MdfdVec *_mdfd_getseg(SMgrRelation reln, ForkNumber forknum,
static BlockNumber _mdnblocks(SMgrRelation reln, ForkNumber forknum,
MdfdVec *seg);
+static PgAioResult md_readv_complete(PgAioHandle *ioh, PgAioResult prior_result, uint8 cb_data);
+static void md_readv_report(PgAioResult result, const PgAioTargetData *target_data, int elevel);
+
+const PgAioHandleCallbacks aio_md_readv_cb = {
+ .complete_shared = md_readv_complete,
+ .report = md_readv_report,
+};
+
+
static inline int
_mdfd_open_flags(void)
{
@@ -937,6 +947,69 @@ mdreadv(SMgrRelation reln, ForkNumber forknum, BlockNumber blocknum,
}
}
+/*
+ * mdstartreadv() -- Asynchronous version of mdreadv().
+ */
+void
+mdstartreadv(PgAioHandle *ioh,
+ SMgrRelation reln, ForkNumber forknum, BlockNumber blocknum,
+ void **buffers, BlockNumber nblocks)
+{
+ off_t seekpos;
+ MdfdVec *v;
+ BlockNumber nblocks_this_segment;
+ struct iovec *iov;
+ int iovcnt;
+ int ret;
+
+ v = _mdfd_getseg(reln, forknum, blocknum, false,
+ EXTENSION_FAIL | EXTENSION_CREATE_RECOVERY);
+
+ seekpos = (off_t) BLCKSZ * (blocknum % ((BlockNumber) RELSEG_SIZE));
+
+ Assert(seekpos < (off_t) BLCKSZ * RELSEG_SIZE);
+
+ nblocks_this_segment =
+ Min(nblocks,
+ RELSEG_SIZE - (blocknum % ((BlockNumber) RELSEG_SIZE)));
+
+ if (nblocks_this_segment != nblocks)
+ elog(ERROR, "read crossing segment boundary");
+
+ iovcnt = pgaio_io_get_iovec(ioh, &iov);
+
+ Assert(nblocks <= iovcnt);
+
+ iovcnt = buffers_to_iovec(iov, buffers, nblocks_this_segment);
+
+ Assert(iovcnt <= nblocks_this_segment);
+
+ if (!(io_direct_flags & IO_DIRECT_DATA))
+ pgaio_io_set_flag(ioh, PGAIO_HF_BUFFERED);
+
+ pgaio_io_set_target_smgr(ioh,
+ reln,
+ forknum,
+ blocknum,
+ nblocks,
+ false);
+ pgaio_io_register_callbacks(ioh, PGAIO_HCB_MD_READV, 0);
+
+ ret = FileStartReadV(ioh, v->mdfd_vfd, iovcnt, seekpos, WAIT_EVENT_DATA_FILE_READ);
+ if (ret != 0)
+ ereport(ERROR,
+ (errcode_for_file_access(),
+ errmsg("could not start reading blocks %u..%u in file \"%s\": %m",
+ blocknum,
+ blocknum + nblocks_this_segment - 1,
+ FilePathName(v->mdfd_vfd))));
+
+ /*
+ * The error checks corresponding to the post-read checks in mdread() are
+ * in md_readv_complete().
+ */
+}
+
/*
* mdwritev() -- Write the supplied blocks at the appropriate location.
*
@@ -1365,6 +1438,21 @@ mdimmedsync(SMgrRelation reln, ForkNumber forknum)
}
}
+int
+mdfd(SMgrRelation reln, ForkNumber forknum, BlockNumber blocknum, uint32 *off)
+{
+ MdfdVec *v = mdopenfork(reln, forknum, EXTENSION_FAIL);
+
+ v = _mdfd_getseg(reln, forknum, blocknum, false,
+ EXTENSION_FAIL);
+
+ *off = (off_t) BLCKSZ * (blocknum % ((BlockNumber) RELSEG_SIZE));
+
+ Assert(*off < (off_t) BLCKSZ * RELSEG_SIZE);
+
+ return FileGetRawDesc(v->mdfd_vfd);
+}
+
/*
* register_dirty_segment() -- Mark a relation segment as needing fsync
*
@@ -1841,3 +1929,101 @@ mdfiletagmatches(const FileTag *ftag, const FileTag *candidate)
*/
return ftag->rlocator.dbOid == candidate->rlocator.dbOid;
}
+
+/*
+ * AIO completion callback for mdstartreadv().
+ */
+static PgAioResult
+md_readv_complete(PgAioHandle *ioh, PgAioResult prior_result, uint8 cb_data)
+{
+ PgAioTargetData *td = pgaio_io_get_target_data(ioh);
+ PgAioResult result = prior_result;
+
+ if (prior_result.result < 0)
+ {
+ result.status = PGAIO_RS_ERROR;
+ result.id = PGAIO_HCB_MD_READV;
+ /* For "hard" errors, track the error number in error_data */
+ result.error_data = -prior_result.result;
+ result.result = 0;
+
+ pgaio_result_report(result, td, LOG);
+
+ return result;
+ }
+
+ /*
+ * As explained above smgrstartreadv(), the smgr API operates on the level
+ * of blocks, rather than bytes. Convert.
+ */
+ result.result /= BLCKSZ;
+
+ Assert(result.result <= td->smgr.nblocks);
+
+ if (result.result == 0)
+ {
+ /* consider 0 blocks read a failure */
+ result.status = PGAIO_RS_ERROR;
+ result.id = PGAIO_HCB_MD_READV;
+ result.error_data = 0;
+
+ pgaio_result_report(result, td, LOG);
+
+ return result;
+ }
+
+ if (result.status != PGAIO_RS_ERROR &&
+ result.result < td->smgr.nblocks)
+ {
+ /* partial reads should be retried at upper level */
+ result.status = PGAIO_RS_PARTIAL;
+ result.id = PGAIO_HCB_MD_READV;
+ }
+
+ return result;
+}
+
+/*
+ * AIO error reporting callback for mdstartreadv().
+ *
+ * Errors are encoded as follows:
+ * - PgAioResult.error_data != 0 encodes IO that failed with errno != 0
+ * - PgAioResult.error_data == 0 encodes IO that didn't read all data
+ */
+static void
+md_readv_report(PgAioResult result, const PgAioTargetData *td, int elevel)
+{
+ RelPathStr path;
+
+ path = relpathbackend(td->smgr.rlocator,
+ td->smgr.is_temp ? MyProcNumber : INVALID_PROC_NUMBER,
+ td->smgr.forkNum);
+
+ if (result.error_data != 0)
+ {
+ /* for errcode_for_file_access() and %m */
+ errno = result.error_data;
+
+ ereport(elevel,
+ errcode_for_file_access(),
+ errmsg("could not read blocks %u..%u in file \"%s\": %m",
+ td->smgr.blockNum,
+ td->smgr.blockNum + td->smgr.nblocks - 1,
+ path.str));
+ }
+ else
+ {
+ /*
+ * NB: This will typically only be output in debug messages, while
+ * retrying a partial IO.
+ */
+ ereport(elevel,
+ errcode(ERRCODE_DATA_CORRUPTED),
+ errmsg("could not read blocks %u..%u in file \"%s\": read only %zu of %zu bytes",
+ td->smgr.blockNum,
+ td->smgr.blockNum + td->smgr.nblocks - 1,
+ path.str,
+ result.result * (size_t) BLCKSZ,
+ td->smgr.nblocks * (size_t) BLCKSZ));
+ }
+}
diff --git a/src/backend/storage/smgr/smgr.c b/src/backend/storage/smgr/smgr.c
index af74f54b43b..545888dcdfc 100644
--- a/src/backend/storage/smgr/smgr.c
+++ b/src/backend/storage/smgr/smgr.c
@@ -66,6 +66,7 @@
#include "access/xlogutils.h"
#include "lib/ilist.h"
#include "miscadmin.h"
+#include "storage/aio.h"
#include "storage/bufmgr.h"
#include "storage/ipc.h"
#include "storage/md.h"
@@ -106,6 +107,10 @@ typedef struct f_smgr
void (*smgr_readv) (SMgrRelation reln, ForkNumber forknum,
BlockNumber blocknum,
void **buffers, BlockNumber nblocks);
+ void (*smgr_startreadv) (PgAioHandle *ioh,
+ SMgrRelation reln, ForkNumber forknum,
+ BlockNumber blocknum,
+ void **buffers, BlockNumber nblocks);
void (*smgr_writev) (SMgrRelation reln, ForkNumber forknum,
BlockNumber blocknum,
const void **buffers, BlockNumber nblocks,
@@ -117,6 +122,7 @@ typedef struct f_smgr
BlockNumber old_blocks, BlockNumber nblocks);
void (*smgr_immedsync) (SMgrRelation reln, ForkNumber forknum);
void (*smgr_registersync) (SMgrRelation reln, ForkNumber forknum);
+ int (*smgr_fd) (SMgrRelation reln, ForkNumber forknum, BlockNumber blocknum, uint32 *off);
} f_smgr;
static const f_smgr smgrsw[] = {
@@ -134,12 +140,14 @@ static const f_smgr smgrsw[] = {
.smgr_prefetch = mdprefetch,
.smgr_maxcombine = mdmaxcombine,
.smgr_readv = mdreadv,
+ .smgr_startreadv = mdstartreadv,
.smgr_writev = mdwritev,
.smgr_writeback = mdwriteback,
.smgr_nblocks = mdnblocks,
.smgr_truncate = mdtruncate,
.smgr_immedsync = mdimmedsync,
.smgr_registersync = mdregistersync,
+ .smgr_fd = mdfd,
}
};
@@ -157,6 +165,16 @@ static dlist_head unpinned_relns;
static void smgrshutdown(int code, Datum arg);
static void smgrdestroy(SMgrRelation reln);
+static void smgr_aio_reopen(PgAioHandle *ioh);
+static char *smgr_aio_describe_identity(const PgAioTargetData *sd);
+
+
+const PgAioTargetInfo aio_smgr_target_info = {
+ .name = "smgr",
+ .reopen = smgr_aio_reopen,
+ .describe_identity = smgr_aio_describe_identity,
+};
+
/*
* smgrinit(), smgrshutdown() -- Initialize or shut down storage
@@ -709,6 +727,30 @@ smgrreadv(SMgrRelation reln, ForkNumber forknum, BlockNumber blocknum,
RESUME_INTERRUPTS();
}
+/*
+ * smgrstartreadv() -- asynchronous version of smgrreadv()
+ *
+ * This starts an asynchronous readv IO using the IO handle `ioh`. Other than
+ * `ioh` all parameters are the same as smgrreadv().
+ *
+ * Completion callbacks above smgr will be passed the result as the number of
+ * successfully read blocks if the read [partially] succeeds (Buffers for
+ * blocks not successfully read might bear unspecified modifications, up to
+ * the full nblocks). This maintains the abstraction that smgr operates on the
+ * level of blocks, rather than bytes.
+ */
+void
+smgrstartreadv(PgAioHandle *ioh,
+ SMgrRelation reln, ForkNumber forknum, BlockNumber blocknum,
+ void **buffers, BlockNumber nblocks)
+{
+ HOLD_INTERRUPTS();
+ smgrsw[reln->smgr_which].smgr_startreadv(ioh,
+ reln, forknum, blocknum, buffers,
+ nblocks);
+ RESUME_INTERRUPTS();
+}
+
/*
* smgrwritev() -- Write the supplied buffers out.
*
@@ -917,6 +959,25 @@ smgrimmedsync(SMgrRelation reln, ForkNumber forknum)
RESUME_INTERRUPTS();
}
+/*
+ * Return fd for the specified block number and update *off to the appropriate
+ * position.
+ *
+ * This is only to be used for when AIO needs to perform the IO in a different
+ * process than where it was issued (e.g. in an IO worker).
+ */
+static int
+smgrfd(SMgrRelation reln, ForkNumber forknum, BlockNumber blocknum, uint32 *off)
+{
+ int fd;
+
+ HOLD_INTERRUPTS();
+ fd = smgrsw[reln->smgr_which].smgr_fd(reln, forknum, blocknum, off);
+ RESUME_INTERRUPTS();
+
+ return fd;
+}
+
/*
* AtEOXact_SMgr
*
@@ -945,3 +1006,92 @@ ProcessBarrierSmgrRelease(void)
smgrreleaseall();
return true;
}
+
+/*
+ * Set target of the IO handle to be smgr and initialize all the relevant
+ * pieces of data.
+ */
+void
+pgaio_io_set_target_smgr(PgAioHandle *ioh,
+ SMgrRelationData *smgr,
+ ForkNumber forknum,
+ BlockNumber blocknum,
+ int nblocks,
+ bool skip_fsync)
+{
+ PgAioTargetData *sd = pgaio_io_get_target_data(ioh);
+
+ pgaio_io_set_target(ioh, PGAIO_TID_SMGR);
+
+ /* backend is implied via IO owner */
+ sd->smgr.rlocator = smgr->smgr_rlocator.locator;
+ sd->smgr.forkNum = forknum;
+ sd->smgr.blockNum = blocknum;
+ sd->smgr.nblocks = nblocks;
+ sd->smgr.is_temp = SmgrIsTemp(smgr);
+ /* Temp relations should never be fsync'd */
+ sd->smgr.skip_fsync = skip_fsync && !SmgrIsTemp(smgr);
+}
+
+/*
+ * Callback for smgr AIO targets, to reopen the file (e.g. because the IO is
+ * executed in a worker).
+ */
+static void
+smgr_aio_reopen(PgAioHandle *ioh)
+{
+ PgAioTargetData *sd = pgaio_io_get_target_data(ioh);
+ PgAioOpData *od = pgaio_io_get_op_data(ioh);
+ SMgrRelation reln;
+ ProcNumber procno;
+ uint32 off;
+
+ if (sd->smgr.is_temp)
+ procno = pgaio_io_get_owner(ioh);
+ else
+ procno = INVALID_PROC_NUMBER;
+
+ reln = smgropen(sd->smgr.rlocator, procno);
+ switch (pgaio_io_get_op(ioh))
+ {
+ case PGAIO_OP_INVALID:
+ pg_unreachable();
+ break;
+ case PGAIO_OP_READV:
+ od->read.fd = smgrfd(reln, sd->smgr.forkNum, sd->smgr.blockNum, &off);
+ Assert(off == od->read.offset);
+ break;
+ case PGAIO_OP_WRITEV:
+ od->write.fd = smgrfd(reln, sd->smgr.forkNum, sd->smgr.blockNum, &off);
+ Assert(off == od->write.offset);
+ break;
+ }
+}
+
+/*
+ * Callback for smgr AIO targets, to describe the target of the IO.
+ */
+static char *
+smgr_aio_describe_identity(const PgAioTargetData *sd)
+{
+ RelPathStr path;
+ char *desc;
+
+ path = relpathbackend(sd->smgr.rlocator,
+ sd->smgr.is_temp ? MyProcNumber : INVALID_PROC_NUMBER,
+ sd->smgr.forkNum);
+
+ if (sd->smgr.nblocks == 0)
+ desc = psprintf(_("file \"%s\""), path.str);
+ else if (sd->smgr.nblocks == 1)
+ desc = psprintf(_("block %u in file \"%s\""),
+ sd->smgr.blockNum,
+ path.str);
+ else
+ desc = psprintf(_("blocks %u..%u in file \"%s\""),
+ sd->smgr.blockNum,
+ sd->smgr.blockNum + sd->smgr.nblocks - 1,
+ path.str);
+
+ return desc;
+}
--
2.48.1.76.g4e746b1a31.dirty
[text/x-diff] v2.11-0007-aio-Add-README.md-explaining-higher-level-desi.patch (19.0K, ../../6ak556uyqiptdwjaci4kbi5eykwkmzqgkbtkyaosjnopjhncrc@2v4ac2jwyz22/8-v2.11-0007-aio-Add-README.md-explaining-higher-level-desi.patch)
download | inline diff:
From e614522f8aedaf46a27c4b08fa78105cc92459d7 Mon Sep 17 00:00:00 2001
From: Andres Freund <andres@anarazel.de>
Date: Tue, 18 Mar 2025 14:40:06 -0400
Subject: [PATCH v2.11 07/27] aio: Add README.md explaining higher level design
Reviewed-by: Noah Misch <noah@leadboat.com>
Reviewed-by: Heikki Linnakangas <hlinnaka@iki.fi>
Discussion: https://postgr.es/m/uvrtrknj4kdytuboidbhwclo4gxhswwcpgadptsjvjqcluzmah%40brqs62irg4dt
Discussion: https://postgr.es/m/20210223100344.llw5an2aklengrmn@alap3.anarazel.de
Discussion: https://postgr.es/m/stj36ea6yyhoxtqkhpieia2z4krnam7qyetc57rfezgk4zgapf@gcnactj4z56m
---
src/backend/storage/aio/README.md | 423 ++++++++++++++++++++++++++++++
src/backend/storage/aio/aio.c | 2 +
2 files changed, 425 insertions(+)
create mode 100644 src/backend/storage/aio/README.md
diff --git a/src/backend/storage/aio/README.md b/src/backend/storage/aio/README.md
new file mode 100644
index 00000000000..012cf3f305a
--- /dev/null
+++ b/src/backend/storage/aio/README.md
@@ -0,0 +1,423 @@
+# Asynchronous & Direct IO
+
+## Motivation
+
+### Why Asynchronous IO
+
+Until the introduction of asynchronous IO Postgres relied on the operating
+system to hide the cost of synchronous IO from Postgres. While this worked
+surprisingly well in a lot of workloads, it does not do as good a job on
+prefetching and controlled writeback as we would like.
+
+There are important expensive operations like `fdatasync()` where the operating
+system cannot hide the storage latency. This is particularly important for WAL
+writes, where the ability to asynchronously issue `fdatasync()` or O_DSYNC
+writes can yield significantly higher throughput.
+
+
+### Why Direct / unbuffered IO
+
+The main reasons to want to use Direct IO are:
+
+- Lower CPU usage / higher throughput. Particularly on modern storage buffered
+ writes are bottlenecked by the operating system having to copy data from the
+ kernel's page cache to postgres buffer pool using the CPU. Whereas direct IO
+ can often move the data directly between the storage devices and postgres'
+ buffer cache, using DMA. While that transfer is ongoing, the CPU is free to
+ perform other work.
+- Reduced latency - Direct IO can have substantially lower latency than
+ buffered IO, which can be impactful for OLTP workloads bottlenecked by WAL
+ write latency.
+- Avoiding double buffering between operating system cache and postgres'
+ shared_buffers.
+- Better control over the timing and pace of dirty data writeback.
+
+
+The main reasons *not* to use Direct IO are:
+
+- Without AIO, Direct IO is unusably slow for most purposes.
+- Even with AIO, many parts of postgres need to be modified to perform
+ explicit prefetching.
+- In situations where shared_buffers cannot be set appropriately large,
+ e.g. because there are many different postgres instances hosted on shared
+ hardware, performance will often be worse than when using buffered IO.
+
+
+## AIO Usage Example
+
+In many cases code that can benefit from AIO does not directly have to
+interact with the AIO interface, but can use AIO via higher-level
+abstractions. See [Helpers](#helpers).
+
+In this example, a buffer will be read into shared buffers.
+
+```C
+/*
+ * Result of the operation, only to be accessed in this backend.
+ */
+PgAioReturn ioret;
+
+/*
+ * Acquire an AIO Handle, ioret will get result upon completion.
+ *
+ * Note that ioret needs to stay alive until the IO completes or
+ * CurrentResourceOwner is released (i.e. an error is thrown).
+ */
+PgAioHandle *ioh = pgaio_io_acquire(CurrentResourceOwner, &ioret);
+
+/*
+ * Reference that can be used to wait for the IO we initiate below. This
+ * reference can reside in local or shared memory and waited upon by any
+ * process. An arbitrary number of references can be made for each IO.
+ */
+PgAioWaitRef iow;
+
+pgaio_io_get_wref(ioh, &iow);
+
+/*
+ * Arrange for shared buffer completion callbacks to be called upon completion
+ * of the IO. This callback will update the buffer descriptors associated with
+ * the AioHandle, which e.g. allows other backends to access the buffer.
+ *
+ * A callback can be passed a small bit of data, e.g. to indicate whether to
+ * zero a buffer if it is invalid.
+ *
+ * Multiple completion callbacks can be registered for each handle.
+ */
+pgaio_io_register_callbacks(ioh, PGAIO_HCB_SHARED_BUFFER_READV, 0);
+
+/*
+ * The completion callback needs to know which buffers to update when the IO
+ * completes. As the AIO subsystem does not know about buffers, we have to
+ * associate this information with the AioHandle, for use by the completion
+ * callback registered above.
+ *
+ * In this example we're reading only a single buffer, hence the 1.
+ */
+pgaio_io_set_handle_data_32(ioh, (uint32 *) buffer, 1);
+
+/*
+ * Pass the AIO handle to lower-level function. When operating on the level of
+ * buffers, we don't know how exactly the IO is performed, that is the
+ * responsibility of the storage manager implementation.
+ *
+ * E.g. md.c needs to translate block numbers into offsets in segments.
+ *
+ * Once the IO handle has been handed off to smgstartreadv(), it may not
+ * further be used, as the IO may immediately get executed below
+ * smgrstartreadv() and the handle reused for another IO.
+ *
+ * To issue multiple IOs in an efficient way, a caller can call
+ * pgaio_enter_batchmode() before starting multiple IOs, and end that batch
+ * with pgaio_exit_batchmode(). Note that one needs to be careful while there
+ * may be unsubmitted IOs, as another backend may need to wait for one of the
+ * unsubmitted IOs. If this backend then had to wait for the other backend,
+ * it'd end in an undetected deadlock. See pgaio_enter_batchmode() for more
+ * details.
+ *
+ * Note that even while in batchmode an IO might get submitted immediately,
+ * e.g. due to reaching a limit on the number of unsubmitted IOs, and even
+ * complete before smgrstartreadv() returns.
+ */
+smgrstartreadv(ioh, operation->smgr, forknum, blkno,
+ BufferGetBlock(buffer), 1);
+
+/*
+ * To benefit from AIO, it is beneficial to perform other work, including
+ * submitting other IOs, before waiting for the IO to complete. Otherwise
+ * we could just have used synchronous, blocking IO.
+ */
+perform_other_work();
+
+/*
+ * We did some other work and now need the IO operation to have completed to
+ * continue.
+ */
+pgaio_wref_wait(&iow);
+
+/*
+ * At this point the IO has completed. We do not yet know whether it succeeded
+ * or failed, however. The buffer's state has been updated, which allows other
+ * backends to use the buffer (if the IO succeeded), or retry the IO (if it
+ * failed).
+ *
+ * Note that in case the IO has failed, a LOG message may have been emitted,
+ * but no ERROR has been raised. This is crucial, as another backend waiting
+ * for this IO should not see an ERROR.
+ *
+ * To check whether the operation succeeded, and to raise an ERROR, or if more
+ * appropriate LOG, the PgAioReturn we passed to pgaio_io_acquire() is used.
+ */
+if (ioret.result.status == PGAIO_RS_ERROR)
+ pgaio_result_report(aio_ret.result, &aio_ret.target_data, ERROR);
+
+/*
+ * Besides having succeeded completely, the IO could also have partially
+ * completed. If we e.g. tried to read many blocks at once, the read might have
+ * only succeeded for the first few blocks.
+ *
+ * If the IO partially succeeded and this backend needs all blocks to have
+ * completed, this backend needs to reissue the IO for the remaining buffers.
+ * The AIO subsystem cannot handle this retry transparently.
+ *
+ * As this example is already long, and we only read a single block, we'll just
+ * error out if there's a partial read.
+ */
+if (ioret.result.status == PGAIO_RS_PARTIAL)
+ pgaio_result_report(aio_ret.result, &aio_ret.target_data, ERROR);
+
+/*
+ * The IO succeeded, so we can use the buffer now.
+ */
+```
+
+
+## Design Criteria & Motivation
+
+### Deadlock and Starvation Dangers due to AIO
+
+Using AIO in a naive way can easily lead to deadlocks in an environment where
+the source/target of AIO are shared resources, like pages in postgres'
+shared_buffers.
+
+Consider one backend performing readahead on a table, initiating IO for a
+number of buffers ahead of the current "scan position". If that backend then
+performs some operation that blocks, or even just is slow, the IO completion
+for the asynchronously initiated read may not be processed.
+
+This AIO implementation solves this problem by requiring that AIO methods
+either allow AIO completions to be processed by any backend in the system
+(e.g. io_uring), or to guarantee that AIO processing will happen even when the
+issuing backend is blocked (e.g. worker mode, which offloads completion
+processing to the AIO workers).
+
+
+### IO can be started in critical sections
+
+Using AIO for WAL writes can reduce the overhead of WAL logging substantially:
+
+- AIO allows to start WAL writes eagerly, so they complete before needing to
+ wait
+- AIO allows to have multiple WAL flushes in progress at the same time
+- AIO makes it more realistic to use O\_DIRECT + O\_DSYNC, which can reduce
+ the number of roundtrips to storage on some OSs and storage HW (buffered IO
+ and direct IO without O_DSYNC needs to issue a write and after the write's
+ completion a cache flush, whereas O\_DIRECT + O\_DSYNC can use a single
+ Force Unit Access (FUA) write).
+
+The need to be able to execute IO in critical sections has substantial design
+implication on the AIO subsystem. Mainly because completing IOs (see prior
+section) needs to be possible within a critical section, even if the
+to-be-completed IO itself was not issued in a critical section. Consider
+e.g. the case of a backend first starting a number of writes from shared
+buffers and then starting to flush the WAL. Because only a limited amount of
+IO can be in-progress at the same time, initiating IO for flushing the WAL may
+require to first complete IO that was started earlier.
+
+
+### State for AIO needs to live in shared memory
+
+Because postgres uses a process model and because AIOs need to be
+complete-able by any backend much of the state of the AIO subsystem needs to
+live in shared memory.
+
+In an `EXEC_BACKEND` build, a backend's executable code and other process
+local state is not necessarily mapped to the same addresses in each process
+due to ASLR. This means that the shared memory cannot contain pointers to
+callbacks.
+
+
+## Design of the AIO Subsystem
+
+
+### AIO Methods
+
+To achieve portability and performance, multiple methods of performing AIO are
+implemented and others are likely worth adding in the future.
+
+
+#### Synchronous Mode
+
+`io_method=sync` does not actually perform AIO but allows to use the AIO API
+while performing synchronous IO. This can be useful for debugging. The code
+for the synchronous mode is also used as a fallback by e.g. the [worker
+mode](#worker) uses it to execute IO that cannot be executed by workers.
+
+
+#### Worker
+
+`io_method=worker` is available on every platform postgres runs on, and
+implements asynchronous IO - from the view of the issuing process - by
+dispatching the IO to one of several worker processes performing the IO in a
+synchronous manner.
+
+
+#### io_uring
+
+`io_method=io_uring` is available on Linux 5.1+. In contrast to worker mode it
+dispatches all IO from within the process, lowering context switch rate /
+latency.
+
+
+### AIO Handles
+
+The central API piece for postgres' AIO abstraction are AIO handles. To
+execute an IO one first has to acquire an IO handle (`pgaio_io_acquire()`) and
+then "defined", i.e. associate an IO operation with the handle.
+
+Often AIO handles are acquired on a higher level and then passed to a lower
+level to be fully defined. E.g., for IO to/from shared buffers, bufmgr.c
+routines acquire the handle, which is then passed through smgr.c, md.c to be
+finally fully defined in fd.c.
+
+The functions used at the lowest level to define the operation are
+`pgaio_io_prep_*()`.
+
+Because acquisition of an IO handle
+[must always succeed](#io-can-be-started-in-critical-sections)
+and the number of AIO Handles
+[has to be limited](#state-for-aio-needs-to-live-in-shared-memory)
+AIO handles can be reused as soon as they have completed. Obviously code needs
+to be able to react to IO completion. State can be updated using
+[AIO Completion callbacks](#aio-callbacks)
+and the issuing backend can provide a backend local variable to receive the
+result of the IO, as described in
+[AIO Result](#aio-results).
+An IO can be waited for, by both the issuing and any other backend, using
+[AIO References](#aio-wait-references).
+
+
+Because an AIO Handle is not executable just after calling
+`pgaio_io_acquire()` and because `pgaio_io_acquire()` needs to always succeed
+(absent a PANIC), only a single AIO Handle may be acquired (i.e. returned by
+`pgaio_io_acquire()`) without causing the IO to have been defined (by,
+potentially indirectly, causing `pgaio_io_prep_*()` to have been
+called). Otherwise a backend could trivially self-deadlock by using up all AIO
+Handles without the ability to wait for some of the IOs to complete.
+
+If it turns out that an AIO Handle is not needed, e.g., because the handle was
+acquired before holding a contended lock, it can be released without being
+defined using `pgaio_io_release()`.
+
+
+### AIO Callbacks
+
+Commonly several layers need to react to completion of an IO. E.g. for a read
+md.c needs to check if the IO outright failed or was shorter than needed,
+bufmgr.c needs to verify the page looks valid and bufmgr.c needs to update the
+BufferDesc to update the buffer's state.
+
+The fact that several layers / subsystems need to react to IO completion poses
+a few challenges:
+
+- Upper layers should not need to know details of lower layers. E.g. bufmgr.c
+ should not assume the IO will pass through md.c. Therefore upper levels
+ cannot know what lower layers would consider an error.
+
+- Lower layers should not need to know about upper layers. E.g. smgr APIs are
+ used going through shared buffers but are also used bypassing shared
+ buffers. This means that e.g. md.c is not in a position to validate
+ checksums.
+
+- Having code in the AIO subsystem for every possible combination of layers
+ would lead to a lot of duplication.
+
+The "solution" to this the ability to associate multiple completion callbacks
+with a handle. E.g. bufmgr.c can have a callback to update the BufferDesc
+state and to verify the page and md.c can have another callback to check if
+the IO operation was successful.
+
+As [mentioned](#state-for-aio-needs-to-live-in-shared-memory), shared memory
+currently cannot contain function pointers. Because of that completion
+callbacks are not directly identified by function pointers but by IDs
+(`PgAioHandleCallbackID`). A substantial added benefit is that that
+allows callbacks to be identified by much smaller amount of memory (a single
+byte currently).
+
+In addition to completion, AIO callbacks also are called to "stage" an
+IO. This is, e.g., used to increase buffer reference counts to account for the
+AIO subsystem referencing the buffer, which is required to handle the case
+where the issuing backend errors out and releases its own pins while the IO is
+still ongoing.
+
+As [explained earlier](#io-can-be-started-in-critical-sections) IO completions
+need to be safe to execute in critical sections. To allow the backend that
+issued the IO to error out in case of failure [AIO Result](#aio-results) can
+be used.
+
+
+### AIO Targets
+
+In addition to the completion callbacks describe above, each AIO Handle has
+exactly one "target". Each target has some space inside an AIO Handle with
+information specific to the target and can provide callbacks to allow to
+reopen the underlying file (required for worker mode) and to describe the IO
+operation (used for debug logging and error messages).
+
+I.e., if two different uses of AIO can describe the identity of the file being
+operated on the same way, it likely makes sense to use the same
+target. E.g. different smgr implementations can describe IO with
+RelFileLocator, ForkNumber and BlockNumber and can thus share a target. In
+contrast, IO for a WAL file would be described with TimeLineID and XLogRecPtr
+and it would not make sense to use the same target for smgr and WAL.
+
+
+### AIO Wait References
+
+As [described above](#aio-handles), AIO Handles can be reused immediately
+after completion and therefore cannot be used to wait for completion of the
+IO. Waiting is enabled using AIO wait references, which do not just identify
+an AIO Handle but also include the handles "generation".
+
+A reference to an AIO Handle can be acquired using `pgaio_io_get_wref()` and
+then waited upon using `pgaio_wref_wait()`.
+
+
+### AIO Results
+
+As AIO completion callbacks
+[are executed in critical sections](#io-can-be-started-in-critical-sections)
+and [may be executed by any backend](#deadlock-and-starvation-dangers-due-to-aio)
+completion callbacks cannot be used to, e.g., make the query that triggered an
+IO ERROR out.
+
+To allow to react to failing IOs the issuing backend can pass a pointer to a
+`PgAioReturn` in backend local memory. Before an AIO Handle is reused the
+`PgAioReturn` is filled with information about the IO. This includes
+information about whether the IO was successful (as a value of
+`PgAioResultStatus`) and enough information to raise an error in case of a
+failure (via `pgaio_result_report()`, with the error details encoded in
+`PgAioResult`).
+
+
+### AIO Errors
+
+It would be very convenient to have shared completion callbacks encode the
+details of errors as an `ErrorData` that could be raised at a later
+time. Unfortunately doing so would require allocating memory. While elog.c can
+guarantee (well, kinda) that logging a message will not run out of memory,
+that only works because a very limited number of messages are in the process
+of being logged. With AIO a large number of concurrently issued AIOs might
+fail.
+
+To avoid the need for preallocating a potentially large amount of memory (in
+shared memory no less!), completion callbacks instead have to encode errors in
+a more compact format that can be converted into an error message.
+
+
+## Helpers
+
+Using the low-level AIO API introduces too much complexity to do so all over
+the tree. Most uses of AIO should be done via reusable, higher-level,
+helpers.
+
+
+### Read Stream
+
+A common and very beneficial use of AIO are reads where a substantial number
+of to-be-read locations are known ahead of time. E.g., for a sequential scan
+the set of blocks that need to be read can be determined solely by knowing the
+current position and checking the buffer mapping table.
+
+The [Read Stream](../../../include/storage/read_stream.h) interface makes it
+comparatively easy to use AIO for such use cases.
diff --git a/src/backend/storage/aio/aio.c b/src/backend/storage/aio/aio.c
index a30c163c3de..3c3383b1318 100644
--- a/src/backend/storage/aio/aio.c
+++ b/src/backend/storage/aio/aio.c
@@ -24,6 +24,8 @@
*
* - read_stream.c - helper for reading buffered relation data
*
+ * - README.md - higher-level overview over AIO
+ *
*
* Portions Copyright (c) 1996-2025, PostgreSQL Global Development Group
* Portions Copyright (c) 1994, Regents of the University of California
--
2.48.1.76.g4e746b1a31.dirty
[text/x-diff] v2.11-0008-localbuf-Track-pincount-in-BufferDesc-as-well.patch (4.5K, ../../6ak556uyqiptdwjaci4kbi5eykwkmzqgkbtkyaosjnopjhncrc@2v4ac2jwyz22/9-v2.11-0008-localbuf-Track-pincount-in-BufferDesc-as-well.patch)
download | inline diff:
From 2822b36f45394855d60b10e4985ad08acb52e973 Mon Sep 17 00:00:00 2001
From: Andres Freund <andres@anarazel.de>
Date: Tue, 18 Mar 2025 14:40:06 -0400
Subject: [PATCH v2.11 08/27] localbuf: Track pincount in BufferDesc as well
For AIO on temporary table buffers the AIO subsystem needs to be able to
ensure a pin on a buffer while AIO is going on, even if the IO issuing query
errors out. Tracking the buffer in LocalRefCount does not work, as it would
cause CheckForLocalBufferLeaks() to assert out.
Instead, also track the refcount in BufferDesc.state, not just
LocalRefCount. This also makes local buffers behave a bit more akin to shared
buffers.
Note that we still don't need locking, AIO completion callbacks for local
buffers are executed in the issuing session (i.e. nobody else has access to
the BufferDesc).
Discussion: https://postgr.es/m/uvrtrknj4kdytuboidbhwclo4gxhswwcpgadptsjvjqcluzmah%40brqs62irg4dt
---
src/backend/storage/buffer/bufmgr.c | 22 +++++++++++++++++++++
src/backend/storage/buffer/localbuf.c | 28 +++++++++++++++++++++++++--
2 files changed, 48 insertions(+), 2 deletions(-)
diff --git a/src/backend/storage/buffer/bufmgr.c b/src/backend/storage/buffer/bufmgr.c
index 37fb09d8b92..aea08b242ac 100644
--- a/src/backend/storage/buffer/bufmgr.c
+++ b/src/backend/storage/buffer/bufmgr.c
@@ -5350,6 +5350,18 @@ ConditionalLockBufferForCleanup(Buffer buffer)
Assert(refcount > 0);
if (refcount != 1)
return false;
+
+ /*
+ * Check that the AIO subsystem doesn't have a pin. Likely not
+ * possible today, but better safe than sorry.
+ */
+ bufHdr = GetLocalBufferDescriptor(-buffer - 1);
+ buf_state = pg_atomic_read_u32(&bufHdr->state);
+ refcount = BUF_STATE_GET_REFCOUNT(buf_state);
+ Assert(refcount > 0);
+ if (refcount != 1)
+ return false;
+
/* Nobody else to wait for */
return true;
}
@@ -5403,6 +5415,16 @@ IsBufferCleanupOK(Buffer buffer)
/* There should be exactly one pin */
if (LocalRefCount[-buffer - 1] != 1)
return false;
+
+ /*
+ * Check that the AIO subsystem doesn't have a pin. Likely not
+ * possible today, but better safe than sorry.
+ */
+ bufHdr = GetLocalBufferDescriptor(-buffer - 1);
+ buf_state = pg_atomic_read_u32(&bufHdr->state);
+ if (BUF_STATE_GET_REFCOUNT(buf_state) != 1)
+ return false;
+
/* Nobody else to wait for */
return true;
}
diff --git a/src/backend/storage/buffer/localbuf.c b/src/backend/storage/buffer/localbuf.c
index f172a5c7820..3a722321533 100644
--- a/src/backend/storage/buffer/localbuf.c
+++ b/src/backend/storage/buffer/localbuf.c
@@ -249,6 +249,13 @@ GetLocalVictimBuffer(void)
pg_atomic_unlocked_write_u32(&bufHdr->state, buf_state);
trycounter = NLocBuffer;
}
+ else if (BUF_STATE_GET_REFCOUNT(buf_state) > 0)
+ {
+ /*
+ * This can be reached if the backend initiated AIO for this
+ * buffer and then errored out.
+ */
+ }
else
{
/* Found a usable buffer */
@@ -570,7 +577,13 @@ InvalidateLocalBuffer(BufferDesc *bufHdr, bool check_unreferenced)
buf_state = pg_atomic_read_u32(&bufHdr->state);
- if (check_unreferenced && LocalRefCount[bufid] != 0)
+ /*
+ * We need to test not just LocalRefCount[bufid] but also the BufferDesc
+ * itself, as the latter is used to represent a pin by the AIO subsystem.
+ * This can happen if AIO is initiated and then the query errors out.
+ */
+ if (check_unreferenced &&
+ (LocalRefCount[bufid] != 0 || BUF_STATE_GET_REFCOUNT(buf_state) != 0))
elog(ERROR, "block %u of %s is still referenced (local %u)",
bufHdr->tag.blockNum,
relpathbackend(BufTagGetRelFileLocator(&bufHdr->tag),
@@ -744,12 +757,13 @@ PinLocalBuffer(BufferDesc *buf_hdr, bool adjust_usagecount)
if (LocalRefCount[bufid] == 0)
{
NLocalPinnedBuffers++;
+ buf_state += BUF_REFCOUNT_ONE;
if (adjust_usagecount &&
BUF_STATE_GET_USAGECOUNT(buf_state) < BM_MAX_USAGE_COUNT)
{
buf_state += BUF_USAGECOUNT_ONE;
- pg_atomic_unlocked_write_u32(&buf_hdr->state, buf_state);
}
+ pg_atomic_unlocked_write_u32(&buf_hdr->state, buf_state);
}
LocalRefCount[bufid]++;
ResourceOwnerRememberBuffer(CurrentResourceOwner,
@@ -775,7 +789,17 @@ UnpinLocalBufferNoOwner(Buffer buffer)
Assert(NLocalPinnedBuffers > 0);
if (--LocalRefCount[buffid] == 0)
+ {
+ BufferDesc *buf_hdr = GetLocalBufferDescriptor(buffid);
+ uint32 buf_state;
+
NLocalPinnedBuffers--;
+
+ buf_state = pg_atomic_read_u32(&buf_hdr->state);
+ Assert(BUF_STATE_GET_REFCOUNT(buf_state) > 0);
+ buf_state -= BUF_REFCOUNT_ONE;
+ pg_atomic_unlocked_write_u32(&buf_hdr->state, buf_state);
+ }
}
/*
--
2.48.1.76.g4e746b1a31.dirty
[text/x-diff] v2.11-0009-bufmgr-Implement-AIO-read-support.patch (30.1K, ../../6ak556uyqiptdwjaci4kbi5eykwkmzqgkbtkyaosjnopjhncrc@2v4ac2jwyz22/10-v2.11-0009-bufmgr-Implement-AIO-read-support.patch)
download | inline diff:
From ed356428e1058b156a158c2e319668611f3539cc Mon Sep 17 00:00:00 2001
From: Andres Freund <andres@anarazel.de>
Date: Tue, 18 Mar 2025 14:40:06 -0400
Subject: [PATCH v2.11 09/27] bufmgr: Implement AIO read support
This commit implements the infrastructure to perform asynchronous reads into
the buffer pool.
To do so, it:
- Adds readv AIO callbacks for shared and local buffers
It may be worth calling out that shared buffer completions may be run in a
different backend than where the IO started.
- Adds an AIO wait reference to BufferDesc, to allow backends to wait for
in-progress asynchronous IOs
- Adapts StartBufferIO(), WaitIO(), TerminateBufferIO(), and their localbuf.c
equivalents, to be able to deal with AIO
- Moves the code to handle BM_PIN_COUNT_WAITER into a helper function, as it
now also needs to be called on IO completion
As of this commit, nothing issues AIO on shared buffers. A future commit will
update StartReadBuffers() to do so.
Reviewed-by: Noah Misch <noah@leadboat.com>
Reviewed-by: Melanie Plageman <melanieplageman@gmail.com>
Reviewed-by: Nazir Bilal Yavuz <byavuz81@gmail.com>
Discussion: https://postgr.es/m/uvrtrknj4kdytuboidbhwclo4gxhswwcpgadptsjvjqcluzmah%40brqs62irg4dt
Discussion: https://postgr.es/m/20210223100344.llw5an2aklengrmn@alap3.anarazel.de
Discussion: https://postgr.es/m/stj36ea6yyhoxtqkhpieia2z4krnam7qyetc57rfezgk4zgapf@gcnactj4z56m
---
src/include/storage/aio.h | 4 +
src/include/storage/buf_internals.h | 7 +-
src/include/storage/bufmgr.h | 4 +
src/backend/storage/aio/aio_callback.c | 5 +
src/backend/storage/buffer/README | 9 +-
src/backend/storage/buffer/buf_init.c | 3 +
src/backend/storage/buffer/bufmgr.c | 496 +++++++++++++++++++++++--
src/backend/storage/buffer/localbuf.c | 47 ++-
8 files changed, 523 insertions(+), 52 deletions(-)
diff --git a/src/include/storage/aio.h b/src/include/storage/aio.h
index 1591a6c2fca..9a0868a270c 100644
--- a/src/include/storage/aio.h
+++ b/src/include/storage/aio.h
@@ -195,6 +195,10 @@ typedef enum PgAioHandleCallbackID
PGAIO_HCB_MD_READV,
PGAIO_HCB_MD_WRITEV,
+
+ PGAIO_HCB_SHARED_BUFFER_READV,
+
+ PGAIO_HCB_LOCAL_BUFFER_READV,
} PgAioHandleCallbackID;
diff --git a/src/include/storage/buf_internals.h b/src/include/storage/buf_internals.h
index 9327f60c44c..8a52ace37cd 100644
--- a/src/include/storage/buf_internals.h
+++ b/src/include/storage/buf_internals.h
@@ -17,6 +17,7 @@
#include "pgstat.h"
#include "port/atomics.h"
+#include "storage/aio_types.h"
#include "storage/buf.h"
#include "storage/bufmgr.h"
#include "storage/condition_variable.h"
@@ -264,6 +265,8 @@ typedef struct BufferDesc
int wait_backend_pgprocno; /* backend of pin-count waiter */
int freeNext; /* link in freelist chain */
+
+ PgAioWaitRef io_wref; /* set iff AIO is in progress */
LWLock content_lock; /* to lock access to buffer contents */
} BufferDesc;
@@ -472,8 +475,8 @@ extern BlockNumber ExtendBufferedRelLocal(BufferManagerRelation bmr,
uint32 *extended_by);
extern void MarkLocalBufferDirty(Buffer buffer);
extern void TerminateLocalBufferIO(BufferDesc *bufHdr, bool clear_dirty,
- uint32 set_flag_bits);
-extern bool StartLocalBufferIO(BufferDesc *bufHdr, bool forInput);
+ uint32 set_flag_bits, bool syncio);
+extern bool StartLocalBufferIO(BufferDesc *bufHdr, bool forInput, bool nowait);
extern void FlushLocalBuffer(BufferDesc *bufHdr, SMgrRelation reln);
extern void DropRelationLocalBuffers(RelFileLocator rlocator,
ForkNumber forkNum,
diff --git a/src/include/storage/bufmgr.h b/src/include/storage/bufmgr.h
index ecc1c3a909b..b8c28fe79f4 100644
--- a/src/include/storage/bufmgr.h
+++ b/src/include/storage/bufmgr.h
@@ -15,6 +15,7 @@
#define BUFMGR_H
#include "port/pg_iovec.h"
+#include "storage/aio_types.h"
#include "storage/block.h"
#include "storage/buf.h"
#include "storage/bufpage.h"
@@ -171,6 +172,9 @@ extern PGDLLIMPORT int checkpoint_flush_after;
extern PGDLLIMPORT int backend_flush_after;
extern PGDLLIMPORT int bgwriter_flush_after;
+extern const PgAioHandleCallbacks aio_shared_buffer_readv_cb;
+extern const PgAioHandleCallbacks aio_local_buffer_readv_cb;
+
/* in buf_init.c */
extern PGDLLIMPORT char *BufferBlocks;
diff --git a/src/backend/storage/aio/aio_callback.c b/src/backend/storage/aio/aio_callback.c
index a92fa8f678e..3cdd2a6b195 100644
--- a/src/backend/storage/aio/aio_callback.c
+++ b/src/backend/storage/aio/aio_callback.c
@@ -18,6 +18,7 @@
#include "miscadmin.h"
#include "storage/aio.h"
#include "storage/aio_internal.h"
+#include "storage/bufmgr.h"
#include "storage/md.h"
@@ -40,6 +41,10 @@ static const PgAioHandleCallbacksEntry aio_handle_cbs[] = {
CALLBACK_ENTRY(PGAIO_HCB_INVALID, aio_invalid_cb),
CALLBACK_ENTRY(PGAIO_HCB_MD_READV, aio_md_readv_cb),
+
+ CALLBACK_ENTRY(PGAIO_HCB_SHARED_BUFFER_READV, aio_shared_buffer_readv_cb),
+
+ CALLBACK_ENTRY(PGAIO_HCB_LOCAL_BUFFER_READV, aio_local_buffer_readv_cb),
#undef CALLBACK_ENTRY
};
diff --git a/src/backend/storage/buffer/README b/src/backend/storage/buffer/README
index 011af7aff3e..a182fcd660c 100644
--- a/src/backend/storage/buffer/README
+++ b/src/backend/storage/buffer/README
@@ -147,9 +147,12 @@ in the buffer. It is used per the rules above.
* The BM_IO_IN_PROGRESS flag acts as a kind of lock, used to wait for I/O on a
buffer to complete (and in releases before 14, it was accompanied by a
-per-buffer LWLock). The process doing a read or write sets the flag for the
-duration, and processes that need to wait for it to be cleared sleep on a
-condition variable.
+per-buffer LWLock). The process starting a read or write sets the flag. When
+the I/O is completed, be it by the process that initiated the I/O or by
+another process, the flag is removed and the Buffer's condition variable is
+signalled. Processes that need to wait for the I/O to complete can wait for
+asynchronous I/O by using BufferDesc->io_wref and for BM_IO_IN_PROGRESS to be
+unset by sleeping on the buffer's condition variable.
Normal Buffer Replacement Strategy
diff --git a/src/backend/storage/buffer/buf_init.c b/src/backend/storage/buffer/buf_init.c
index ed1f8e03190..ed1dc488a42 100644
--- a/src/backend/storage/buffer/buf_init.c
+++ b/src/backend/storage/buffer/buf_init.c
@@ -14,6 +14,7 @@
*/
#include "postgres.h"
+#include "storage/aio.h"
#include "storage/buf_internals.h"
#include "storage/bufmgr.h"
@@ -125,6 +126,8 @@ BufferManagerShmemInit(void)
buf->buf_id = i;
+ pgaio_wref_clear(&buf->io_wref);
+
/*
* Initially link all the buffers together as unused. Subsequent
* management of this list is done by freelist.c.
diff --git a/src/backend/storage/buffer/bufmgr.c b/src/backend/storage/buffer/bufmgr.c
index aea08b242ac..ea5f504e38a 100644
--- a/src/backend/storage/buffer/bufmgr.c
+++ b/src/backend/storage/buffer/bufmgr.c
@@ -48,6 +48,7 @@
#include "pg_trace.h"
#include "pgstat.h"
#include "postmaster/bgwriter.h"
+#include "storage/aio.h"
#include "storage/buf_internals.h"
#include "storage/bufmgr.h"
#include "storage/fd.h"
@@ -519,7 +520,8 @@ static int SyncOneBuffer(int buf_id, bool skip_recently_used,
static void WaitIO(BufferDesc *buf);
static bool StartBufferIO(BufferDesc *buf, bool forInput, bool nowait);
static void TerminateBufferIO(BufferDesc *buf, bool clear_dirty,
- uint32 set_flag_bits, bool forget_owner);
+ uint32 set_flag_bits, bool forget_owner,
+ bool syncio);
static void AbortBufferIO(Buffer buffer);
static void shared_buffer_write_error_callback(void *arg);
static void local_buffer_write_error_callback(void *arg);
@@ -1041,7 +1043,7 @@ ZeroAndLockBuffer(Buffer buffer, ReadBufferMode mode, bool already_valid)
{
/* Simple case for non-shared buffers. */
bufHdr = GetLocalBufferDescriptor(-buffer - 1);
- need_to_zero = StartLocalBufferIO(bufHdr, true);
+ need_to_zero = StartLocalBufferIO(bufHdr, true, false);
}
else
{
@@ -1077,9 +1079,9 @@ ZeroAndLockBuffer(Buffer buffer, ReadBufferMode mode, bool already_valid)
/* Set BM_VALID, terminate IO, and wake up any waiters */
if (isLocalBuf)
- TerminateLocalBufferIO(bufHdr, false, BM_VALID);
+ TerminateLocalBufferIO(bufHdr, false, BM_VALID, true);
else
- TerminateBufferIO(bufHdr, false, BM_VALID, true);
+ TerminateBufferIO(bufHdr, false, BM_VALID, true, true);
}
else if (!isLocalBuf)
{
@@ -1391,7 +1393,8 @@ static inline bool
WaitReadBuffersCanStartIO(Buffer buffer, bool nowait)
{
if (BufferIsLocal(buffer))
- return StartLocalBufferIO(GetLocalBufferDescriptor(-buffer - 1), true);
+ return StartLocalBufferIO(GetLocalBufferDescriptor(-buffer - 1),
+ true, nowait);
else
return StartBufferIO(GetBufferDescriptor(buffer - 1), true, nowait);
}
@@ -1549,9 +1552,9 @@ WaitReadBuffers(ReadBuffersOperation *operation)
/* Set BM_VALID, terminate IO, and wake up any waiters */
if (persistence == RELPERSISTENCE_TEMP)
- TerminateLocalBufferIO(bufHdr, false, BM_VALID);
+ TerminateLocalBufferIO(bufHdr, false, BM_VALID, true);
else
- TerminateBufferIO(bufHdr, false, BM_VALID, true);
+ TerminateBufferIO(bufHdr, false, BM_VALID, true, true);
/* Report I/Os as completing individually. */
TRACE_POSTGRESQL_BUFFER_READ_DONE(forknum, io_first_block + j,
@@ -2459,7 +2462,7 @@ ExtendBufferedRelShared(BufferManagerRelation bmr,
if (lock)
LWLockAcquire(BufferDescriptorGetContentLock(buf_hdr), LW_EXCLUSIVE);
- TerminateBufferIO(buf_hdr, false, BM_VALID, true);
+ TerminateBufferIO(buf_hdr, false, BM_VALID, true, true);
}
pgBufferUsage.shared_blks_written += extend_by;
@@ -2805,6 +2808,44 @@ PinBuffer_Locked(BufferDesc *buf)
ResourceOwnerRememberBuffer(CurrentResourceOwner, b);
}
+/*
+ * Support for waking up another backend that is waiting for the cleanup lock
+ * to be released using BM_PIN_COUNT_WAITER.
+ *
+ * See LockBufferForCleanup().
+ *
+ * Expected to be called just after releasing a buffer pin (in a BufferDesc,
+ * not just reducing the backend-local pincount for the buffer).
+ */
+static void
+WakePinCountWaiter(BufferDesc *buf)
+{
+ /*
+ * Acquire the buffer header lock, re-check that there's a waiter. Another
+ * backend could have unpinned this buffer, and already woken up the
+ * waiter.
+ *
+ * There's no danger of the buffer being replaced after we unpinned it
+ * above, as it's pinned by the waiter. The waiter removes
+ * BM_PIN_COUNT_WAITER if it stops waiting for a reason other than this
+ * backend waking it up.
+ */
+ uint32 buf_state = LockBufHdr(buf);
+
+ if ((buf_state & BM_PIN_COUNT_WAITER) &&
+ BUF_STATE_GET_REFCOUNT(buf_state) == 1)
+ {
+ /* we just released the last pin other than the waiter's */
+ int wait_backend_pgprocno = buf->wait_backend_pgprocno;
+
+ buf_state &= ~BM_PIN_COUNT_WAITER;
+ UnlockBufHdr(buf, buf_state);
+ ProcSendSignal(wait_backend_pgprocno);
+ }
+ else
+ UnlockBufHdr(buf, buf_state);
+}
+
/*
* UnpinBuffer -- make buffer available for replacement.
*
@@ -2873,29 +2914,8 @@ UnpinBufferNoOwner(BufferDesc *buf)
/* Support LockBufferForCleanup() */
if (buf_state & BM_PIN_COUNT_WAITER)
- {
- /*
- * Acquire the buffer header lock, re-check that there's a waiter.
- * Another backend could have unpinned this buffer, and already
- * woken up the waiter. There's no danger of the buffer being
- * replaced after we unpinned it above, as it's pinned by the
- * waiter.
- */
- buf_state = LockBufHdr(buf);
+ WakePinCountWaiter(buf);
- if ((buf_state & BM_PIN_COUNT_WAITER) &&
- BUF_STATE_GET_REFCOUNT(buf_state) == 1)
- {
- /* we just released the last pin other than the waiter's */
- int wait_backend_pgprocno = buf->wait_backend_pgprocno;
-
- buf_state &= ~BM_PIN_COUNT_WAITER;
- UnlockBufHdr(buf, buf_state);
- ProcSendSignal(wait_backend_pgprocno);
- }
- else
- UnlockBufHdr(buf, buf_state);
- }
ForgetPrivateRefCountEntry(ref);
}
}
@@ -3916,7 +3936,7 @@ FlushBuffer(BufferDesc *buf, SMgrRelation reln, IOObject io_object,
* Mark the buffer as clean (unless BM_JUST_DIRTIED has become set) and
* end the BM_IO_IN_PROGRESS state.
*/
- TerminateBufferIO(buf, true, 0, true);
+ TerminateBufferIO(buf, true, 0, true, true);
TRACE_POSTGRESQL_BUFFER_FLUSH_DONE(BufTagGetForkNum(&buf->tag),
buf->tag.blockNum,
@@ -5472,6 +5492,7 @@ WaitIO(BufferDesc *buf)
for (;;)
{
uint32 buf_state;
+ PgAioWaitRef iow;
/*
* It may not be necessary to acquire the spinlock to check the flag
@@ -5479,10 +5500,40 @@ WaitIO(BufferDesc *buf)
* play it safe.
*/
buf_state = LockBufHdr(buf);
+
+ /*
+ * Copy the wait reference while holding the spinlock. This protects
+ * against a concurrent TerminateBufferIO() in another backend from
+ * clearing the wref while it's being read.
+ */
+ iow = buf->io_wref;
UnlockBufHdr(buf, buf_state);
+ /* no IO in progress, we don't need to wait */
if (!(buf_state & BM_IO_IN_PROGRESS))
break;
+
+ /*
+ * The buffer has asynchronous IO in progress, wait for it to
+ * complete.
+ */
+ if (pgaio_wref_valid(&iow))
+ {
+ pgaio_wref_wait(&iow);
+
+ /*
+ * The AIO subsystem internally uses condition variables and thus
+ * might remove this backend from the BufferDesc's CV. While that
+ * wouldn't cause a correctness issue (the first CV sleep just
+ * immediately returns if not already registered), it seems worth
+ * avoiding unnecessary loop iterations, given that we take care
+ * to do so at the start of the function.
+ */
+ ConditionVariablePrepareToSleep(cv);
+ continue;
+ }
+
+ /* wait on BufferDesc->cv, e.g. for concurrent synchronous IO */
ConditionVariableSleep(cv, WAIT_EVENT_BUFFER_IO);
}
ConditionVariableCancelSleep();
@@ -5491,13 +5542,12 @@ WaitIO(BufferDesc *buf)
/*
* StartBufferIO: begin I/O on this buffer
* (Assumptions)
- * My process is executing no IO
+ * My process is executing no IO on this buffer
* The buffer is Pinned
*
- * In some scenarios there are race conditions in which multiple backends
- * could attempt the same I/O operation concurrently. If someone else
- * has already started I/O on this buffer then we will block on the
- * I/O condition variable until he's done.
+ * In some scenarios multiple backends could attempt the same I/O operation
+ * concurrently. If someone else has already started I/O on this buffer then
+ * we will wait for completion of the IO using WaitIO().
*
* Input operations are only attempted on buffers that are not BM_VALID,
* and output operations only on buffers that are BM_VALID and BM_DIRTY,
@@ -5533,9 +5583,9 @@ StartBufferIO(BufferDesc *buf, bool forInput, bool nowait)
/* Once we get here, there is definitely no I/O active on this buffer */
+ /* Check if someone else already did the I/O */
if (forInput ? (buf_state & BM_VALID) : !(buf_state & BM_DIRTY))
{
- /* someone else already did the I/O */
UnlockBufHdr(buf, buf_state);
return false;
}
@@ -5571,7 +5621,7 @@ StartBufferIO(BufferDesc *buf, bool forInput, bool nowait)
*/
static void
TerminateBufferIO(BufferDesc *buf, bool clear_dirty, uint32 set_flag_bits,
- bool forget_owner)
+ bool forget_owner, bool syncio)
{
uint32 buf_state;
@@ -5586,6 +5636,14 @@ TerminateBufferIO(BufferDesc *buf, bool clear_dirty, uint32 set_flag_bits,
if (clear_dirty && !(buf_state & BM_JUST_DIRTIED))
buf_state &= ~(BM_DIRTY | BM_CHECKPOINT_NEEDED);
+ if (!syncio)
+ {
+ /* release ownership by the AIO subsystem */
+ Assert(BUF_STATE_GET_REFCOUNT(buf_state) > 0);
+ buf_state -= BUF_REFCOUNT_ONE;
+ pgaio_wref_clear(&buf->io_wref);
+ }
+
buf_state |= set_flag_bits;
UnlockBufHdr(buf, buf_state);
@@ -5594,6 +5652,17 @@ TerminateBufferIO(BufferDesc *buf, bool clear_dirty, uint32 set_flag_bits,
BufferDescriptorGetBuffer(buf));
ConditionVariableBroadcast(BufferDescriptorGetIOCV(buf));
+
+ /*
+ * Support LockBufferForCleanup()
+ *
+ * We may have just released the last pin other than the waiter's. In most
+ * cases, this backend holds another pin on the buffer. But, if, for
+ * example, this backend is completing an IO issued by another backend, it
+ * may be time to wake the waiter.
+ */
+ if (buf_state & BM_PIN_COUNT_WAITER)
+ WakePinCountWaiter(buf);
}
/*
@@ -5642,7 +5711,7 @@ AbortBufferIO(Buffer buffer)
}
}
- TerminateBufferIO(buf_hdr, false, BM_IO_ERROR, false);
+ TerminateBufferIO(buf_hdr, false, BM_IO_ERROR, false, true);
}
/*
@@ -6093,3 +6162,352 @@ EvictUnpinnedBuffer(Buffer buf)
return result;
}
+
+/*
+ * Generic implementation of the AIO handle staging callback for readv/writev
+ * on local/shared buffers.
+ *
+ * Each readv/writev can target multiple buffers. The buffers have already
+ * been registered with the IO handle.
+ *
+ * To make the IO ready for execution ("staging"), we need to ensure that the
+ * targeted buffers are in an appropriate state while the IO is ongoing. For
+ * that the AIO subsystem needs to have its own buffer pin, otherwise an error
+ * in this backend could lead to this backend's buffer pin being released as
+ * part of error handling, which in turn could lead to the buffer being
+ * replaced while IO is ongoing.
+ */
+static pg_attribute_always_inline void
+buffer_stage_common(PgAioHandle *ioh, bool is_write, bool is_temp)
+{
+ uint64 *io_data;
+ uint8 handle_data_len;
+ PgAioWaitRef io_ref;
+ BufferTag first PG_USED_FOR_ASSERTS_ONLY = {0};
+
+ io_data = pgaio_io_get_handle_data(ioh, &handle_data_len);
+
+ pgaio_io_get_wref(ioh, &io_ref);
+
+ /* iterate over all buffers affected by the vectored readv/writev */
+ for (int i = 0; i < handle_data_len; i++)
+ {
+ Buffer buffer = (Buffer) io_data[i];
+ BufferDesc *buf_hdr = is_temp ?
+ GetLocalBufferDescriptor(-buffer - 1)
+ : GetBufferDescriptor(buffer - 1);
+ uint32 buf_state;
+
+ /*
+ * Check that all the buffers are actually ones that could conceivably
+ * be done in one IO, i.e. are sequential. This is the last
+ * buffer-aware code before IO is actually executed and confusion
+ * about which buffers are target by IO can be hard to debug, making
+ * it worth doing extra-paranoid checks.
+ */
+ if (i == 0)
+ first = buf_hdr->tag;
+ else
+ {
+ Assert(buf_hdr->tag.relNumber == first.relNumber);
+ Assert(buf_hdr->tag.blockNum == first.blockNum + i);
+ }
+
+ if (is_temp)
+ buf_state = pg_atomic_read_u32(&buf_hdr->state);
+ else
+ buf_state = LockBufHdr(buf_hdr);
+
+ /* verify the buffer is in the expected state */
+ Assert(buf_state & BM_TAG_VALID);
+ if (is_write)
+ {
+ Assert(buf_state & BM_VALID);
+ Assert(buf_state & BM_DIRTY);
+ }
+ else
+ {
+ Assert(!(buf_state & BM_VALID));
+ Assert(!(buf_state & BM_DIRTY));
+ }
+
+ /* temp buffers don't use BM_IO_IN_PROGRESS */
+ if (!is_temp)
+ Assert(buf_state & BM_IO_IN_PROGRESS);
+
+ Assert(BUF_STATE_GET_REFCOUNT(buf_state) >= 1);
+
+ /*
+ * Reflect that the buffer is now owned by the AIO subsystem.
+ *
+ * For local buffers: This can't be done just via LocalRefCount, as
+ * one might initially think, as this backend could error out while
+ * AIO is still in progress, releasing all the pins by the backend
+ * itself.
+ *
+ * This pin is released again in TerminateBufferIO().
+ */
+ buf_state += BUF_REFCOUNT_ONE;
+ buf_hdr->io_wref = io_ref;
+
+ if (is_temp)
+ pg_atomic_unlocked_write_u32(&buf_hdr->state, buf_state);
+ else
+ UnlockBufHdr(buf_hdr, buf_state);
+
+ /*
+ * Ensure the content lock that prevents buffer modifications while
+ * the buffer is being written out is not released early due to an
+ * error.
+ */
+ if (is_write && !is_temp)
+ {
+ LWLock *content_lock;
+
+ content_lock = BufferDescriptorGetContentLock(buf_hdr);
+
+ Assert(LWLockHeldByMe(content_lock));
+
+ /*
+ * Lock is now owned by AIO subsystem.
+ */
+ LWLockDisown(content_lock);
+ }
+
+ /*
+ * Stop tracking this buffer via the resowner - the AIO system now
+ * keeps track.
+ */
+ if (!is_temp)
+ ResourceOwnerForgetBufferIO(CurrentResourceOwner, buffer);
+ }
+}
+
+/*
+ * Helper for AIO readv completion callbacks, supporting both shared and temp
+ * buffers. Gets called once for each buffer in a multi-page read.
+ */
+static pg_attribute_always_inline PgAioResult
+buffer_readv_complete_one(uint8 buf_off, Buffer buffer, uint8 flags,
+ bool failed, bool is_temp)
+{
+ BufferDesc *buf_hdr = is_temp ?
+ GetLocalBufferDescriptor(-buffer - 1)
+ : GetBufferDescriptor(buffer - 1);
+ BufferTag tag = buf_hdr->tag;
+ char *bufdata = BufferGetBlock(buffer);
+ PgAioResult result;
+ uint32 set_flag_bits;
+
+ /* check that the buffer is in the expected state for a read */
+#ifdef USE_ASSERT_CHECKING
+ {
+ uint32 buf_state = pg_atomic_read_u32(&buf_hdr->state);
+
+ Assert(buf_state & BM_TAG_VALID);
+ Assert(!(buf_state & BM_VALID));
+ /* temp buffers don't use BM_IO_IN_PROGRESS */
+ if (!is_temp)
+ Assert(buf_state & BM_IO_IN_PROGRESS);
+ Assert(!(buf_state & BM_DIRTY));
+ }
+#endif
+
+ result.status = PGAIO_RS_OK;
+
+ /* check for garbage data */
+ if (!failed &&
+ !PageIsVerifiedExtended((Page) bufdata, tag.blockNum,
+ PIV_LOG_WARNING | PIV_REPORT_STAT))
+ {
+ RelFileLocator rlocator = BufTagGetRelFileLocator(&tag);
+
+ if ((flags & READ_BUFFERS_ZERO_ON_ERROR) || zero_damaged_pages)
+ {
+ ereport(WARNING,
+ (errcode(ERRCODE_DATA_CORRUPTED),
+ errmsg("invalid page in block %u of relation %s; zeroing out page",
+ tag.blockNum,
+ relpathbackend(rlocator,
+ is_temp ? MyProcNumber : INVALID_PROC_NUMBER,
+ tag.forkNum).str)));
+ memset(bufdata, 0, BLCKSZ);
+ }
+ else
+ {
+ /* mark buffer as having failed */
+ failed = true;
+
+ /* encode error for buffer_readv_report */
+ result.status = PGAIO_RS_ERROR;
+ if (is_temp)
+ result.id = PGAIO_HCB_LOCAL_BUFFER_READV;
+ else
+ result.id = PGAIO_HCB_SHARED_BUFFER_READV;
+ result.error_data = buf_off;
+ }
+ }
+
+ /* Terminate I/O and set BM_VALID. */
+ set_flag_bits = failed ? BM_IO_ERROR : BM_VALID;
+ if (is_temp)
+ TerminateLocalBufferIO(buf_hdr, false, set_flag_bits, false);
+ else
+ TerminateBufferIO(buf_hdr, false, set_flag_bits, false, false);
+
+ /*
+ * Call the BUFFER_READ_DONE tracepoint in the callback, even though the
+ * callback may not be executed in the same backend that called
+ * BUFFER_READ_START. The alternative would be to defer calling the
+ * tracepoint to a later point (e.g. the local completion callback for
+ * shared buffer reads), which seems even less helpful and would also
+ * imply some overhead.
+ */
+ TRACE_POSTGRESQL_BUFFER_READ_DONE(tag.forkNum,
+ tag.blockNum,
+ tag.spcOid,
+ tag.dbOid,
+ tag.relNumber,
+ is_temp ? MyProcNumber : INVALID_PROC_NUMBER,
+ false);
+
+ return result;
+}
+
+/*
+ * Perform completion handling of a single AIO read. This read may cover
+ * multiple blocks / buffers.
+ *
+ * Shared between shared and local buffers, to reduce code duplication.
+ */
+static pg_attribute_always_inline PgAioResult
+buffer_readv_complete(PgAioHandle *ioh, PgAioResult prior_result,
+ uint8 cb_data, bool is_temp)
+{
+ PgAioResult result = prior_result;
+ PgAioTargetData *td = pgaio_io_get_target_data(ioh);
+ uint64 *io_data;
+ uint8 handle_data_len;
+
+ if (is_temp)
+ {
+ Assert(td->smgr.is_temp);
+ Assert(pgaio_io_get_owner(ioh) == MyProcNumber);
+ }
+ else
+ Assert(!td->smgr.is_temp);
+
+ /*
+ * Iterate over all the buffers affected by this IO and call the
+ * per-buffer completion function for each buffer.
+ */
+ io_data = pgaio_io_get_handle_data(ioh, &handle_data_len);
+ for (uint8 buf_off = 0; buf_off < handle_data_len; buf_off++)
+ {
+ Buffer buf = io_data[buf_off];
+ PgAioResult buf_result;
+ bool failed;
+
+ Assert(BufferIsValid(buf));
+
+ /*
+ * If the entire I/O failed on a lower-level, each buffer needs to be
+ * marked as failed. In case of a partial read, some buffers may be
+ * ok.
+ */
+ failed =
+ prior_result.status == PGAIO_RS_ERROR
+ || prior_result.result <= buf_off;
+
+ buf_result = buffer_readv_complete_one(buf_off, buf, cb_data, failed,
+ is_temp);
+
+ /*
+ * If there wasn't any prior error and page verification failed in
+ * some form, set the whole IO's result to the page's result.
+ */
+ if (result.status != PGAIO_RS_ERROR
+ && buf_result.status != PGAIO_RS_OK)
+ {
+ result = buf_result;
+ pgaio_result_report(result, td, LOG);
+ }
+ }
+
+ return result;
+}
+
+/*
+ * AIO error reporting callback for aio_shared_buffer_readv_cb and
+ * aio_local_buffer_readv_cb.
+ *
+ * Errors are encoded as follows:
+ * - the only error is page verification failing
+ * - result.error_data is the offset of the first page that failed
+ * verification in a larger IO
+ */
+static void
+buffer_readv_report(PgAioResult result, const PgAioTargetData *target_data,
+ int elevel)
+{
+ ProcNumber errProc;
+
+ if (target_data->smgr.is_temp)
+ errProc = MyProcNumber;
+ else
+ errProc = INVALID_PROC_NUMBER;
+
+ ereport(elevel,
+ errcode(ERRCODE_DATA_CORRUPTED),
+ errmsg("invalid page in block %u of relation %s",
+ target_data->smgr.blockNum + result.error_data,
+ relpathbackend(target_data->smgr.rlocator, errProc,
+ target_data->smgr.forkNum).str));
+}
+
+static void
+shared_buffer_readv_stage(PgAioHandle *ioh, uint8 cb_data)
+{
+ buffer_stage_common(ioh, false, false);
+}
+
+static PgAioResult
+shared_buffer_readv_complete(PgAioHandle *ioh, PgAioResult prior_result,
+ uint8 cb_data)
+{
+ return buffer_readv_complete(ioh, prior_result, cb_data, false);
+}
+
+static void
+local_buffer_readv_stage(PgAioHandle *ioh, uint8 cb_data)
+{
+ buffer_stage_common(ioh, false, true);
+}
+
+static PgAioResult
+local_buffer_readv_complete(PgAioHandle *ioh, PgAioResult prior_result,
+ uint8 cb_data)
+{
+ return buffer_readv_complete(ioh, prior_result, cb_data, true);
+}
+
+/* readv callback is passed READ_BUFFERS_* flags as callback data */
+const PgAioHandleCallbacks aio_shared_buffer_readv_cb = {
+ .stage = shared_buffer_readv_stage,
+ .complete_shared = shared_buffer_readv_complete,
+ .report = buffer_readv_report,
+};
+
+/* readv callback is passed READ_BUFFERS_* flags as callback data */
+const PgAioHandleCallbacks aio_local_buffer_readv_cb = {
+ .stage = local_buffer_readv_stage,
+
+ /*
+ * Note that this, in contrast to the shared_buffers case, uses
+ * complete_local, as only the issuing backend has access to the required
+ * datastructures. This is important in case the IO completion may be
+ * consumed incidentally by another backend.
+ */
+ .complete_local = local_buffer_readv_complete,
+ .report = buffer_readv_report,
+};
diff --git a/src/backend/storage/buffer/localbuf.c b/src/backend/storage/buffer/localbuf.c
index 3a722321533..8b6cd54d09e 100644
--- a/src/backend/storage/buffer/localbuf.c
+++ b/src/backend/storage/buffer/localbuf.c
@@ -17,7 +17,9 @@
#include "access/parallel.h"
#include "executor/instrument.h"
+#include "pg_trace.h"
#include "pgstat.h"
+#include "storage/aio.h"
#include "storage/buf_internals.h"
#include "storage/bufmgr.h"
#include "storage/fd.h"
@@ -187,7 +189,7 @@ FlushLocalBuffer(BufferDesc *bufHdr, SMgrRelation reln)
* Try to start an I/O operation. There currently are no reasons for
* StartLocalBufferIO to return false, so we raise an error in that case.
*/
- if (!StartLocalBufferIO(bufHdr, false))
+ if (!StartLocalBufferIO(bufHdr, false, false))
elog(ERROR, "failed to start write IO on local buffer");
/* Find smgr relation for buffer */
@@ -211,7 +213,7 @@ FlushLocalBuffer(BufferDesc *bufHdr, SMgrRelation reln)
IOOP_WRITE, io_start, 1, BLCKSZ);
/* Mark not-dirty */
- TerminateLocalBufferIO(bufHdr, true, 0);
+ TerminateLocalBufferIO(bufHdr, true, 0, true);
pgBufferUsage.local_blks_written++;
}
@@ -430,7 +432,7 @@ ExtendBufferedRelLocal(BufferManagerRelation bmr,
pg_atomic_unlocked_write_u32(&existing_hdr->state, buf_state);
/* no need to loop for local buffers */
- StartLocalBufferIO(existing_hdr, true);
+ StartLocalBufferIO(existing_hdr, true, false);
}
else
{
@@ -446,7 +448,7 @@ ExtendBufferedRelLocal(BufferManagerRelation bmr,
hresult->id = victim_buf_id;
- StartLocalBufferIO(victim_buf_hdr, true);
+ StartLocalBufferIO(victim_buf_hdr, true, false);
}
}
@@ -515,13 +517,31 @@ MarkLocalBufferDirty(Buffer buffer)
* Like StartBufferIO, but for local buffers
*/
bool
-StartLocalBufferIO(BufferDesc *bufHdr, bool forInput)
+StartLocalBufferIO(BufferDesc *bufHdr, bool forInput, bool nowait)
{
- uint32 buf_state = pg_atomic_read_u32(&bufHdr->state);
+ uint32 buf_state;
+ /*
+ * With AIO the buffer could have IO in progress, e.g. when there are two
+ * scans of the same relation. Either wait for the other IO or return
+ * false.
+ */
+ if (pgaio_wref_valid(&bufHdr->io_wref))
+ {
+ PgAioWaitRef iow = bufHdr->io_wref;
+
+ if (nowait)
+ return false;
+
+ pgaio_wref_wait(&iow);
+ }
+
+ /* Once we get here, there is definitely no I/O active on this buffer */
+
+ /* Check if someone else already did the I/O */
+ buf_state = pg_atomic_read_u32(&bufHdr->state);
if (forInput ? (buf_state & BM_VALID) : !(buf_state & BM_DIRTY))
{
- /* someone else already did the I/O */
return false;
}
@@ -536,7 +556,8 @@ StartLocalBufferIO(BufferDesc *bufHdr, bool forInput)
* Like TerminateBufferIO, but for local buffers
*/
void
-TerminateLocalBufferIO(BufferDesc *bufHdr, bool clear_dirty, uint32 set_flag_bits)
+TerminateLocalBufferIO(BufferDesc *bufHdr, bool clear_dirty, uint32 set_flag_bits,
+ bool syncio)
{
/* Only need to adjust flags */
uint32 buf_state = pg_atomic_read_u32(&bufHdr->state);
@@ -549,6 +570,14 @@ TerminateLocalBufferIO(BufferDesc *bufHdr, bool clear_dirty, uint32 set_flag_bit
if (clear_dirty)
buf_state &= ~BM_DIRTY;
+ if (!syncio)
+ {
+ /* release pin held by IO subsystem, see also buffer_stage_common() */
+ Assert(BUF_STATE_GET_REFCOUNT(buf_state) > 0);
+ buf_state -= BUF_REFCOUNT_ONE;
+ pgaio_wref_clear(&bufHdr->io_wref);
+ }
+
buf_state |= set_flag_bits;
pg_atomic_unlocked_write_u32(&bufHdr->state, buf_state);
@@ -714,6 +743,8 @@ InitLocalBuffers(void)
*/
buf->buf_id = -i - 2;
+ pgaio_wref_clear(&buf->io_wref);
+
/*
* Intentionally do not initialize the buffer's atomic variable
* (besides zeroing the underlying memory above). That way we get
--
2.48.1.76.g4e746b1a31.dirty
[text/x-diff] v2.11-0010-Support-buffer-forwarding-in-read_stream.c.patch (8.7K, ../../6ak556uyqiptdwjaci4kbi5eykwkmzqgkbtkyaosjnopjhncrc@2v4ac2jwyz22/11-v2.11-0010-Support-buffer-forwarding-in-read_stream.c.patch)
download | inline diff:
From 39ac7acb4944758ae9c1c243e9da09f99dafade0 Mon Sep 17 00:00:00 2001
From: Thomas Munro <thomas.munro@gmail.com>
Date: Tue, 18 Mar 2025 14:40:06 -0400
Subject: [PATCH v2.11 10/27] Support buffer forwarding in read_stream.c.
In preparation for a following change to the buffer manager, teach
streams to keep track of buffers that were "forwarded" from one call to
StartReadBuffers() to the next.
Since StartReadBuffers() buffers argument will become an in/out
argument, we need to initialize the buffer queue entries with
InvalidBuffer. We don't want to do that up front, because we try to
keep stream initialization cheap for code that uses the fast path and
stays in one single buffer queue element. Satisfy both goals by
initializing the queue incrementally on the first cycle.
Discussion: https://postgr.es/m/CA%2BhUKGK_%3D4CVmMHvsHjOVrK6t4F%3DLBpFzsrr3R%2BaJYN8kcTfWg%40mail.gmail.com
---
src/backend/storage/aio/read_stream.c | 103 +++++++++++++++++++++++---
1 file changed, 93 insertions(+), 10 deletions(-)
diff --git a/src/backend/storage/aio/read_stream.c b/src/backend/storage/aio/read_stream.c
index 45bdf819d57..18ecf4affc7 100644
--- a/src/backend/storage/aio/read_stream.c
+++ b/src/backend/storage/aio/read_stream.c
@@ -95,8 +95,10 @@ struct ReadStream
int16 ios_in_progress;
int16 queue_size;
int16 max_pinned_buffers;
+ int16 forwarded_buffers;
int16 pinned_buffers;
int16 distance;
+ int16 initialized_buffers;
bool advice_enabled;
bool temporary;
@@ -224,8 +226,10 @@ static bool
read_stream_start_pending_read(ReadStream *stream)
{
bool need_wait;
+ int requested_nblocks;
int nblocks;
int flags;
+ int forwarded;
int16 io_index;
int16 overflow;
int16 buffer_index;
@@ -272,11 +276,20 @@ read_stream_start_pending_read(ReadStream *stream)
}
}
- /* How many more buffers is this backend allowed? */
+ /*
+ * How many more buffers is this backend allowed?
+ *
+ * If we already have some forwarded buffers, we can certainly use those.
+ * They are already pinned, and are mapped to the starting blocks of the
+ * pending read, they just don't have any I/O started yet and are not
+ * counted in stream->pinned_buffers.
+ */
if (stream->temporary)
buffer_limit = Min(GetAdditionalLocalPinLimit(), PG_INT16_MAX);
else
buffer_limit = Min(GetAdditionalPinLimit(), PG_INT16_MAX);
+ Assert(stream->forwarded_buffers <= stream->pending_read_nblocks);
+ buffer_limit += stream->forwarded_buffers;
if (buffer_limit == 0 && stream->pinned_buffers == 0)
buffer_limit = 1; /* guarantee progress */
@@ -303,8 +316,31 @@ read_stream_start_pending_read(ReadStream *stream)
* We say how many blocks we want to read, but it may be smaller on return
* if the buffer manager decides to shorten the read.
*/
+ requested_nblocks = Min(buffer_limit, stream->pending_read_nblocks);
+ nblocks = requested_nblocks;
buffer_index = stream->next_buffer_index;
io_index = stream->next_io_index;
+
+ /*
+ * The first time around the queue we initialize it as we go, including
+ * the overflow zone, because otherwise the entries would appear as
+ * forwarded buffers. This avoids initializing the whole queue up front
+ * in cases where it is large but we don't ever use it due to the
+ * all-cached fast path or small scans.
+ */
+ while (stream->initialized_buffers < buffer_index + nblocks)
+ stream->buffers[stream->initialized_buffers++] = InvalidBuffer;
+
+ /*
+ * Start the I/O. Any buffers that are not InvalidBuffer will be
+ * interpreted as already pinned, forwarded by an earlier call to
+ * StartReadBuffers(), and must map to the expected blocks. The nblocks
+ * value may be smaller on return indicating the size of the I/O that
+ * could be started. Buffers beyond the output nblocks number may also
+ * have been pinned without starting I/O due to various edge cases. In
+ * that case we'll just leave them in the queue ahead of us, "forwarded"
+ * to the next call, avoiding the need to unpin/repin.
+ */
need_wait = StartReadBuffers(&stream->ios[io_index].op,
&stream->buffers[buffer_index],
stream->pending_read_blocknum,
@@ -333,16 +369,35 @@ read_stream_start_pending_read(ReadStream *stream)
stream->seq_blocknum = stream->pending_read_blocknum + nblocks;
}
+ /*
+ * How many pins were acquired but forwarded to the next call? These need
+ * to be passed to the next StartReadBuffers() call, or released if the
+ * stream ends early. We need the number for accounting purposes, since
+ * they are not counted in stream->pinned_buffers but we already hold
+ * them.
+ */
+ forwarded = 0;
+ while (nblocks + forwarded < requested_nblocks &&
+ stream->buffers[buffer_index + nblocks + forwarded] != InvalidBuffer)
+ forwarded++;
+ stream->forwarded_buffers = forwarded;
+
/*
* We gave a contiguous range of buffer space to StartReadBuffers(), but
- * we want it to wrap around at queue_size. Slide overflowing buffers to
- * the front of the array.
+ * we want it to wrap around at queue_size. Copy overflowing buffers to
+ * the front of the array where they'll be consumed, but also leave a copy
+ * in the overflow zone which the I/O operation has a pointer to (it needs
+ * a contiguous array). Both copies will be cleared when the buffers are
+ * handed to the consumer.
*/
- overflow = (buffer_index + nblocks) - stream->queue_size;
+ overflow = (buffer_index + nblocks + forwarded) - stream->queue_size;
if (overflow > 0)
- memmove(&stream->buffers[0],
- &stream->buffers[stream->queue_size],
- sizeof(stream->buffers[0]) * overflow);
+ {
+ Assert(overflow < stream->queue_size); /* can't overlap */
+ memcpy(&stream->buffers[0],
+ &stream->buffers[stream->queue_size],
+ sizeof(stream->buffers[0]) * overflow);
+ }
/* Compute location of start of next read, without using % operator. */
buffer_index += nblocks;
@@ -719,10 +774,12 @@ read_stream_next_buffer(ReadStream *stream, void **per_buffer_data)
/* Fast path assumptions. */
Assert(stream->ios_in_progress == 0);
+ Assert(stream->forwarded_buffers == 0);
Assert(stream->pinned_buffers == 1);
Assert(stream->distance == 1);
Assert(stream->pending_read_nblocks == 0);
Assert(stream->per_buffer_data_size == 0);
+ Assert(stream->initialized_buffers > stream->oldest_buffer_index);
/* We're going to return the buffer we pinned last time. */
oldest_buffer_index = stream->oldest_buffer_index;
@@ -771,6 +828,7 @@ read_stream_next_buffer(ReadStream *stream, void **per_buffer_data)
stream->distance = 0;
stream->oldest_buffer_index = stream->next_buffer_index;
stream->pinned_buffers = 0;
+ stream->buffers[oldest_buffer_index] = InvalidBuffer;
}
stream->fast_path = false;
@@ -846,10 +904,15 @@ read_stream_next_buffer(ReadStream *stream, void **per_buffer_data)
stream->seq_until_processed = InvalidBlockNumber;
}
-#ifdef CLOBBER_FREED_MEMORY
- /* Clobber old buffer for debugging purposes. */
+ /*
+ * We must zap this queue entry, or else it would appear as a forwarded
+ * buffer. If it's potentially in the overflow zone (ie it wrapped around
+ * the queue), also zap that copy.
+ */
stream->buffers[oldest_buffer_index] = InvalidBuffer;
-#endif
+ if (oldest_buffer_index < io_combine_limit - 1)
+ stream->buffers[stream->queue_size + oldest_buffer_index] =
+ InvalidBuffer;
#if defined(CLOBBER_FREED_MEMORY) || defined(USE_VALGRIND)
@@ -894,6 +957,7 @@ read_stream_next_buffer(ReadStream *stream, void **per_buffer_data)
#ifndef READ_STREAM_DISABLE_FAST_PATH
/* See if we can take the fast path for all-cached scans next time. */
if (stream->ios_in_progress == 0 &&
+ stream->forwarded_buffers == 0 &&
stream->pinned_buffers == 1 &&
stream->distance == 1 &&
stream->pending_read_nblocks == 0 &&
@@ -929,6 +993,7 @@ read_stream_next_block(ReadStream *stream, BufferAccessStrategy *strategy)
void
read_stream_reset(ReadStream *stream)
{
+ int16 index;
Buffer buffer;
/* Stop looking ahead. */
@@ -942,6 +1007,24 @@ read_stream_reset(ReadStream *stream)
while ((buffer = read_stream_next_buffer(stream, NULL)) != InvalidBuffer)
ReleaseBuffer(buffer);
+ /* Unpin any unused forwarded buffers. */
+ index = stream->next_buffer_index;
+ while (index < stream->initialized_buffers &&
+ (buffer = stream->buffers[index]) != InvalidBuffer)
+ {
+ Assert(stream->forwarded_buffers > 0);
+ stream->forwarded_buffers--;
+ ReleaseBuffer(buffer);
+
+ stream->buffers[index] = InvalidBuffer;
+ if (index < io_combine_limit - 1)
+ stream->buffers[stream->queue_size + index] = InvalidBuffer;
+
+ if (++index == stream->queue_size)
+ index = 0;
+ }
+
+ Assert(stream->forwarded_buffers == 0);
Assert(stream->pinned_buffers == 0);
Assert(stream->ios_in_progress == 0);
--
2.48.1.76.g4e746b1a31.dirty
[text/x-diff] v2.11-0011-Support-buffer-forwarding-in-StartReadBuffers.patch (10.5K, ../../6ak556uyqiptdwjaci4kbi5eykwkmzqgkbtkyaosjnopjhncrc@2v4ac2jwyz22/12-v2.11-0011-Support-buffer-forwarding-in-StartReadBuffers.patch)
download | inline diff:
From eef02887f094b1fc3fdfe2ec4343f889ec592776 Mon Sep 17 00:00:00 2001
From: Thomas Munro <thomas.munro@gmail.com>
Date: Tue, 18 Mar 2025 14:40:06 -0400
Subject: [PATCH v2.11 11/27] Support buffer forwarding in StartReadBuffers().
Sometimes we have to perform a short read because we hit a cached block
that ends a contiguous run of blocks requiring I/O. We don't want
StartReadBuffers() to have to start more than one I/O, so we stop there.
We also don't want to have to unpin the cached block (and repin it
later), so previously we'd silently pretend the hit was part of the I/O,
and just leave it out of the read from disk. Now, we'll "forward" it to
the next call. We still write it to the buffers[] array for the caller
to pass back to us later, but it's not included in *nblocks.
This policy means that we no longer mix hits and misses in a single
operation's results, so we avoid the requirement to call
WaitReadBuffers(), which might stall, before the caller can make use of
the hits. The caller will get the hit in the next call instead, and
know that it doesn't have to wait. That's important for later work on
out-of-order read streams that minimize I/O stalls.
This also makes life easier for proposed work on true AIO, which
occasionally needs to split a large I/O after pinning all the buffers,
while the current coding only ever forwards a single bookending hit.
This API is natural for read_stream.c: it just leaves forwarded buffers
where they are in its circular queue, where the next call will pick them
up and continue, minimizing pin churn.
If we ever think of a good reason to disable this feature, i.e. for
other users of StartReadBuffers() that don't want to deal with forwarded
buffers, then we could add a flag for that. For now read_steam.c is the
only user.
Discussion: https://postgr.es/m/CA%2BhUKGK_%3D4CVmMHvsHjOVrK6t4F%3DLBpFzsrr3R%2BaJYN8kcTfWg%40mail.gmail.com
---
src/include/storage/bufmgr.h | 1 -
src/backend/storage/buffer/bufmgr.c | 128 ++++++++++++++++++++--------
2 files changed, 91 insertions(+), 38 deletions(-)
diff --git a/src/include/storage/bufmgr.h b/src/include/storage/bufmgr.h
index b8c28fe79f4..5137288c96e 100644
--- a/src/include/storage/bufmgr.h
+++ b/src/include/storage/bufmgr.h
@@ -131,7 +131,6 @@ struct ReadBuffersOperation
BlockNumber blocknum;
int flags;
int16 nblocks;
- int16 io_buffers_len;
};
typedef struct ReadBuffersOperation ReadBuffersOperation;
diff --git a/src/backend/storage/buffer/bufmgr.c b/src/backend/storage/buffer/bufmgr.c
index ea5f504e38a..1a5037dd913 100644
--- a/src/backend/storage/buffer/bufmgr.c
+++ b/src/backend/storage/buffer/bufmgr.c
@@ -1254,10 +1254,10 @@ StartReadBuffersImpl(ReadBuffersOperation *operation,
Buffer *buffers,
BlockNumber blockNum,
int *nblocks,
- int flags)
+ int flags,
+ bool allow_forwarding)
{
int actual_nblocks = *nblocks;
- int io_buffers_len = 0;
int maxcombine = 0;
Assert(*nblocks > 0);
@@ -1267,30 +1267,80 @@ StartReadBuffersImpl(ReadBuffersOperation *operation,
{
bool found;
- buffers[i] = PinBufferForBlock(operation->rel,
- operation->smgr,
- operation->persistence,
- operation->forknum,
- blockNum + i,
- operation->strategy,
- &found);
+ if (allow_forwarding && buffers[i] != InvalidBuffer)
+ {
+ BufferDesc *bufHdr;
+
+ /*
+ * This is a buffer that was pinned by an earlier call to
+ * StartReadBuffers(), but couldn't be handled in one operation at
+ * that time. The operation was split, and the caller has passed
+ * an already pinned buffer back to us to handle the rest of the
+ * operation. It must continue at the expected block number.
+ */
+ Assert(BufferGetBlockNumber(buffers[i]) == blockNum + i);
+
+ /*
+ * It might be an already valid buffer (a hit) that followed the
+ * final contiguous block of an earlier I/O (a miss) marking the
+ * end of it, or a buffer that some other backend has since made
+ * valid by performing the I/O for us, in which case we can handle
+ * it as a hit now. It is safe to check for a BM_VALID flag with
+ * a relaxed load, because we got a fresh view of it while pinning
+ * it in the previous call.
+ *
+ * On the other hand if we don't see BM_VALID yet, it must be an
+ * I/O that was split by the previous call and we need to try to
+ * start a new I/O from this block. We're also racing against any
+ * other backend that might start the I/O or even manage to mark
+ * it BM_VALID after this check, but StartBufferIO() will handle
+ * those cases.
+ */
+ if (BufferIsLocal(buffers[i]))
+ bufHdr = GetLocalBufferDescriptor(-buffers[i] - 1);
+ else
+ bufHdr = GetBufferDescriptor(buffers[i] - 1);
+ found = pg_atomic_read_u32(&bufHdr->state) & BM_VALID;
+ }
+ else
+ {
+ buffers[i] = PinBufferForBlock(operation->rel,
+ operation->smgr,
+ operation->persistence,
+ operation->forknum,
+ blockNum + i,
+ operation->strategy,
+ &found);
+ }
if (found)
{
/*
- * Terminate the read as soon as we get a hit. It could be a
- * single buffer hit, or it could be a hit that follows a readable
- * range. We don't want to create more than one readable range,
- * so we stop here.
+ * We have a hit. If it's the first block in the requested range,
+ * we can return it immediately and report that WaitReadBuffers()
+ * does not need to be called. If the initial value of *nblocks
+ * was larger, the caller will have to call again for the rest.
*/
- actual_nblocks = i + 1;
+ if (i == 0)
+ {
+ *nblocks = 1;
+ return false;
+ }
+
+ /*
+ * Otherwise we already have an I/O to perform, but this block
+ * can't be included as it is already valid. Split the I/O here.
+ * There may or may not be more blocks requiring I/O after this
+ * one, we haven't checked, but it can't be contiguous with this
+ * hit in the way. We'll leave this buffer pinned, forwarding it
+ * to the next call, avoiding the need to unpin it here and re-pin
+ * it in the next call.
+ */
+ actual_nblocks = i;
break;
}
else
{
- /* Extend the readable range to cover this block. */
- io_buffers_len++;
-
/*
* Check how many blocks we can cover with the same IO. The smgr
* implementation might e.g. be limited due to a segment boundary.
@@ -1311,15 +1361,11 @@ StartReadBuffersImpl(ReadBuffersOperation *operation,
}
*nblocks = actual_nblocks;
- if (likely(io_buffers_len == 0))
- return false;
-
/* Populate information needed for I/O. */
operation->buffers = buffers;
operation->blocknum = blockNum;
operation->flags = flags;
operation->nblocks = actual_nblocks;
- operation->io_buffers_len = io_buffers_len;
if (flags & READ_BUFFERS_ISSUE_ADVICE)
{
@@ -1334,7 +1380,7 @@ StartReadBuffersImpl(ReadBuffersOperation *operation,
smgrprefetch(operation->smgr,
operation->forknum,
blockNum,
- operation->io_buffers_len);
+ actual_nblocks);
}
/* Indicate that WaitReadBuffers() should be called. */
@@ -1348,11 +1394,21 @@ StartReadBuffersImpl(ReadBuffersOperation *operation,
* actual number, which may be fewer than requested. Caller sets some of the
* members of operation; see struct definition.
*
+ * The initial contents of the elements of buffers up to *nblocks should
+ * either be InvalidBuffer or an already-pinned buffer that was left by an
+ * preceding call to StartReadBuffers() that had to be split. On return, some
+ * elements of buffers may hold pinned buffers beyond the number indicated by
+ * the updated value of *nblocks. Operations are split on boundaries known to
+ * smgr (eg md.c segment boundaries that require crossing into a different
+ * underlying file), or when already cached blocks are found in the buffer
+ * that prevent the formation of a contiguous read.
+ *
* If false is returned, no I/O is necessary. If true is returned, one I/O
* has been started, and WaitReadBuffers() must be called with the same
* operation object before the buffers are accessed. Along with the operation
* object, the caller-supplied array of buffers must remain valid until
- * WaitReadBuffers() is called.
+ * WaitReadBuffers() is called, and any forwarded buffers must also be
+ * preserved for a future call unless explicitly released.
*
* Currently the I/O is only started with optional operating system advice if
* requested by the caller with READ_BUFFERS_ISSUE_ADVICE, and the real I/O
@@ -1366,13 +1422,18 @@ StartReadBuffers(ReadBuffersOperation *operation,
int *nblocks,
int flags)
{
- return StartReadBuffersImpl(operation, buffers, blockNum, nblocks, flags);
+ return StartReadBuffersImpl(operation, buffers, blockNum, nblocks, flags,
+ true /* expect forwarded buffers */ );
}
/*
* Single block version of the StartReadBuffers(). This might save a few
* instructions when called from another translation unit, because it is
* specialized for nblocks == 1.
+ *
+ * This version does not support "forwarded" buffers: they cannot be created
+ * by reading only one block, and the current contents of *buffer is ignored
+ * on entry.
*/
bool
StartReadBuffer(ReadBuffersOperation *operation,
@@ -1383,7 +1444,8 @@ StartReadBuffer(ReadBuffersOperation *operation,
int nblocks = 1;
bool result;
- result = StartReadBuffersImpl(operation, buffer, blocknum, &nblocks, flags);
+ result = StartReadBuffersImpl(operation, buffer, blocknum, &nblocks, flags,
+ false /* single block, no forwarding */ );
Assert(nblocks == 1); /* single block can't be short */
return result;
@@ -1410,24 +1472,16 @@ WaitReadBuffers(ReadBuffersOperation *operation)
IOObject io_object;
char persistence;
- /*
- * Currently operations are only allowed to include a read of some range,
- * with an optional extra buffer that is already pinned at the end. So
- * nblocks can be at most one more than io_buffers_len.
- */
- Assert((operation->nblocks == operation->io_buffers_len) ||
- (operation->nblocks == operation->io_buffers_len + 1));
-
/* Find the range of the physical read we need to perform. */
- nblocks = operation->io_buffers_len;
- if (nblocks == 0)
- return; /* nothing to do */
-
+ nblocks = operation->nblocks;
buffers = &operation->buffers[0];
blocknum = operation->blocknum;
forknum = operation->forknum;
persistence = operation->persistence;
+ Assert(nblocks > 0);
+ Assert(nblocks <= MAX_IO_COMBINE_LIMIT);
+
if (persistence == RELPERSISTENCE_TEMP)
{
io_context = IOCONTEXT_NORMAL;
--
2.48.1.76.g4e746b1a31.dirty
[text/x-diff] v2.11-0012-bufmgr-Use-AIO-in-StartReadBuffers.patch (26.8K, ../../6ak556uyqiptdwjaci4kbi5eykwkmzqgkbtkyaosjnopjhncrc@2v4ac2jwyz22/13-v2.11-0012-bufmgr-Use-AIO-in-StartReadBuffers.patch)
download | inline diff:
From de33e6dd831449c50e571a61be39ce0fa9a72f45 Mon Sep 17 00:00:00 2001
From: Andres Freund <andres@anarazel.de>
Date: Tue, 18 Mar 2025 14:40:06 -0400
Subject: [PATCH v2.11 12/27] bufmgr: Use AIO in StartReadBuffers()
This finally introduces the first actual use of AIO. StartReadBuffers() now
uses the AIO routines to issue IO. This converts a lot of callers to use the
AIO infrastructure.
As the implementation of StartReadBuffers() is also used by the functions for
reading individual blocks (StartReadBuffer() and through that
ReadBufferExtended()) this means all buffered read IO passes through the AIO
paths. However, as those are synchronous reads, actually performing the IO
asynchronously would be rarely beneficial. Instead such IOs are flagged to
always be executed synchronously. This way we don't have to duplicate a fair
bit of code.
When io_method=sync is used, the IO patterns generated after this change are
the same as before, i.e. actual reads are only issued in WaitReadBuffers() and
StartReadBuffers() may issue prefetch requests. This allows to bypass most of
the actual asynchronicity, which is important to make a change as big as this
less risky.
One thing worth calling out is that, if IO is actually executed
asynchronously, the precise meaning of what track_io_timing is measuring has
changed. Previously it tracked the time for each IO, but that does not make
sense when multiple IOs are executed concurrently. Now it only measures the
time actually spent waiting for IO. A subsequent commit will adjust the docs
for this.
While AIO is now actually used, the logic in read_stream.c will often prevent
using sufficiently many concurrent IOs. That will be addressed in the next
commit.
Reviewed-by: Noah Misch <noah@leadboat.com>
Reviewed-by: Nazir Bilal Yavuz <byavuz81@gmail.com>
Co-authored-by: Andres Freund <andres@anarazel.de>
Co-authored-by: Thomas Munro <thomas.munro@gmail.com>
Discussion: https://postgr.es/m/uvrtrknj4kdytuboidbhwclo4gxhswwcpgadptsjvjqcluzmah%40brqs62irg4dt
Discussion: https://postgr.es/m/20210223100344.llw5an2aklengrmn@alap3.anarazel.de
Discussion: https://postgr.es/m/stj36ea6yyhoxtqkhpieia2z4krnam7qyetc57rfezgk4zgapf@gcnactj4z56m
---
src/include/storage/bufmgr.h | 6 +
src/backend/storage/buffer/bufmgr.c | 608 +++++++++++++++++++++-------
2 files changed, 474 insertions(+), 140 deletions(-)
diff --git a/src/include/storage/bufmgr.h b/src/include/storage/bufmgr.h
index 5137288c96e..7abb3bdfd9d 100644
--- a/src/include/storage/bufmgr.h
+++ b/src/include/storage/bufmgr.h
@@ -112,6 +112,9 @@ typedef struct BufferManagerRelation
#define READ_BUFFERS_ZERO_ON_ERROR (1 << 0)
/* Call smgrprefetch() if I/O necessary. */
#define READ_BUFFERS_ISSUE_ADVICE (1 << 1)
+/* IO will immediately be waited for */
+#define READ_BUFFERS_SYNCHRONOUSLY (1 << 2)
+
struct ReadBuffersOperation
{
@@ -131,6 +134,9 @@ struct ReadBuffersOperation
BlockNumber blocknum;
int flags;
int16 nblocks;
+ int16 nblocks_done;
+ PgAioWaitRef io_wref;
+ PgAioReturn io_return;
};
typedef struct ReadBuffersOperation ReadBuffersOperation;
diff --git a/src/backend/storage/buffer/bufmgr.c b/src/backend/storage/buffer/bufmgr.c
index 1a5037dd913..08d25e70bf2 100644
--- a/src/backend/storage/buffer/bufmgr.c
+++ b/src/backend/storage/buffer/bufmgr.c
@@ -531,6 +531,8 @@ static inline BufferDesc *BufferAlloc(SMgrRelation smgr,
BlockNumber blockNum,
BufferAccessStrategy strategy,
bool *foundPtr, IOContext io_context);
+static bool AsyncReadBuffers(ReadBuffersOperation *operation, int *nblocks);
+static void CheckReadBuffersOperation(ReadBuffersOperation *operation, bool is_complete);
static Buffer GetVictimBuffer(BufferAccessStrategy strategy, IOContext io_context);
static void FlushBuffer(BufferDesc *buf, SMgrRelation reln,
IOObject io_object, IOContext io_context);
@@ -1231,10 +1233,14 @@ ReadBuffer_common(Relation rel, SMgrRelation smgr, char smgr_persistence,
return buffer;
}
+ /*
+ * Signal that we are going to immediately wait. If we're immediately
+ * waiting, there is no benefit in actually executing the IO
+ * asynchronously, it would just add dispatch overhead.
+ */
+ flags = READ_BUFFERS_SYNCHRONOUSLY;
if (mode == RBM_ZERO_ON_ERROR)
- flags = READ_BUFFERS_ZERO_ON_ERROR;
- else
- flags = 0;
+ flags |= READ_BUFFERS_ZERO_ON_ERROR;
operation.smgr = smgr;
operation.rel = rel;
operation.persistence = persistence;
@@ -1259,9 +1265,11 @@ StartReadBuffersImpl(ReadBuffersOperation *operation,
{
int actual_nblocks = *nblocks;
int maxcombine = 0;
+ bool did_start_io;
Assert(*nblocks > 0);
Assert(*nblocks <= MAX_IO_COMBINE_LIMIT);
+ Assert(*nblocks == 1 || allow_forwarding);
for (int i = 0; i < actual_nblocks; ++i)
{
@@ -1300,6 +1308,7 @@ StartReadBuffersImpl(ReadBuffersOperation *operation,
bufHdr = GetLocalBufferDescriptor(-buffers[i] - 1);
else
bufHdr = GetBufferDescriptor(buffers[i] - 1);
+ Assert(pg_atomic_read_u32(&bufHdr->state) & BM_TAG_VALID);
found = pg_atomic_read_u32(&bufHdr->state) & BM_VALID;
}
else
@@ -1324,6 +1333,20 @@ StartReadBuffersImpl(ReadBuffersOperation *operation,
if (i == 0)
{
*nblocks = 1;
+
+#ifdef USE_ASSERT_CHECKING
+
+ /*
+ * Initialize enough of ReadBuffersOperation to make
+ * CheckReadBuffersOperation() work. Outside of assertions
+ * that's not necessary when no IO is issued.
+ */
+ operation->buffers = buffers;
+ operation->blocknum = blockNum;
+ operation->nblocks = 1;
+ operation->nblocks_done = 1;
+ CheckReadBuffersOperation(operation, true);
+#endif
return false;
}
@@ -1366,25 +1389,74 @@ StartReadBuffersImpl(ReadBuffersOperation *operation,
operation->blocknum = blockNum;
operation->flags = flags;
operation->nblocks = actual_nblocks;
+ operation->nblocks_done = 0;
+ pgaio_wref_clear(&operation->io_wref);
- if (flags & READ_BUFFERS_ISSUE_ADVICE)
+ /*
+ * When using AIO, start the IO in the background. If not, issue prefetch
+ * requests if desired by the caller.
+ *
+ * The reason we have a dedicated path for IOMETHOD_SYNC here is to
+ * de-risk the introduction of AIO somewhat. It's a large architectural
+ * change, with lots of chances for unanticipated performance effects.
+ *
+ * Use of IOMETHOD_SYNC already leads to not actually performing IO
+ * asynchronously, but without the check here we'd execute IO earlier than
+ * we used to. Eventually this IOMETHOD_SYNC specific path should go away.
+ */
+ if (io_method != IOMETHOD_SYNC)
{
/*
- * In theory we should only do this if PinBufferForBlock() had to
- * allocate new buffers above. That way, if two calls to
- * StartReadBuffers() were made for the same blocks before
- * WaitReadBuffers(), only the first would issue the advice. That'd be
- * a better simulation of true asynchronous I/O, which would only
- * start the I/O once, but isn't done here for simplicity.
+ * Try to start IO asynchronously. It's possible that no IO needs to
+ * be started, if another backend already performed the IO.
+ *
+ * Note that if an IO is started, it might not cover the entire
+ * requested range, e.g. because an intermediary block has been read
+ * in by another backend. In that case any "trailing" buffers we
+ * already pinned above will be "forwarded" by read_stream.c to the
+ * next call to StartReadBuffers().
+ *
+ * This is signalled to the caller by decrementing *nblocks *and*
+ * reducing operation->nblocks. The latter is done here, but not below
+ * WaitReadBuffers(), as in WaitReadBuffers() we can't "shorten" the
+ * overall read size anymore, we need to retry until done in its
+ * entirety or until failed.
*/
- smgrprefetch(operation->smgr,
- operation->forknum,
- blockNum,
- actual_nblocks);
+ did_start_io = AsyncReadBuffers(operation, nblocks);
+
+ operation->nblocks = *nblocks;
+ }
+ else
+ {
+ operation->flags |= READ_BUFFERS_SYNCHRONOUSLY;
+
+ if (flags & READ_BUFFERS_ISSUE_ADVICE)
+ {
+ /*
+ * In theory we should only do this if PinBufferForBlock() had to
+ * allocate new buffers above. That way, if two calls to
+ * StartReadBuffers() were made for the same blocks before
+ * WaitReadBuffers(), only the first would issue the advice.
+ * That'd be a better simulation of true asynchronous I/O, which
+ * would only start the I/O once, but isn't done here for
+ * simplicity.
+ */
+ smgrprefetch(operation->smgr,
+ operation->forknum,
+ blockNum,
+ actual_nblocks);
+ }
+
+ /*
+ * Indicate that WaitReadBuffers() should be called. WaitReadBuffers()
+ * will initiate the necessary IO.
+ */
+ did_start_io = true;
}
- /* Indicate that WaitReadBuffers() should be called. */
- return true;
+ CheckReadBuffersOperation(operation, !did_start_io);
+
+ return did_start_io;
}
/*
@@ -1451,8 +1523,35 @@ StartReadBuffer(ReadBuffersOperation *operation,
return result;
}
+/*
+ * Perform sanity checks on the ReadBuffersOperation.
+ */
+static void
+CheckReadBuffersOperation(ReadBuffersOperation *operation, bool is_complete)
+{
+#ifdef USE_ASSERT_CHECKING
+ Assert(operation->nblocks_done <= operation->nblocks);
+ Assert(!is_complete || operation->nblocks == operation->nblocks_done);
+
+ for (int i = 0; i < operation->nblocks; i++)
+ {
+ Buffer buffer = operation->buffers[i];
+ BufferDesc *buf_hdr = BufferIsLocal(buffer) ?
+ GetLocalBufferDescriptor(-buffer - 1) :
+ GetBufferDescriptor(buffer - 1);
+
+ Assert(BufferGetBlockNumber(buffer) == operation->blocknum + i);
+ Assert(pg_atomic_read_u32(&buf_hdr->state) & BM_TAG_VALID);
+
+ if (i < operation->nblocks_done)
+ Assert(pg_atomic_read_u32(&buf_hdr->state) & BM_VALID);
+ }
+#endif
+}
+
+/* helper for ReadBuffersCanStartIO(), to avoid repetition */
static inline bool
-WaitReadBuffersCanStartIO(Buffer buffer, bool nowait)
+ReadBuffersCanStartIOOnce(Buffer buffer, bool nowait)
{
if (BufferIsLocal(buffer))
return StartLocalBufferIO(GetLocalBufferDescriptor(-buffer - 1),
@@ -1461,31 +1560,243 @@ WaitReadBuffersCanStartIO(Buffer buffer, bool nowait)
return StartBufferIO(GetBufferDescriptor(buffer - 1), true, nowait);
}
-void
-WaitReadBuffers(ReadBuffersOperation *operation)
+/*
+ * Helper for AsyncReadBuffers that tries to get the buffer ready for IO.
+ */
+static inline bool
+ReadBuffersCanStartIO(Buffer buffer, bool nowait)
{
- Buffer *buffers;
- int nblocks;
- BlockNumber blocknum;
- ForkNumber forknum;
- IOContext io_context;
- IOObject io_object;
- char persistence;
-
- /* Find the range of the physical read we need to perform. */
- nblocks = operation->nblocks;
- buffers = &operation->buffers[0];
- blocknum = operation->blocknum;
- forknum = operation->forknum;
- persistence = operation->persistence;
+ /*
+ * If this backend currently has staged IO, we need to submit the pending
+ * IO before waiting for the right to issue IO, to avoid the potential for
+ * deadlocks (and, more commonly, unnecessary delays for other backends).
+ */
+ if (!nowait && pgaio_have_staged())
+ {
+ if (ReadBuffersCanStartIOOnce(buffer, true))
+ return true;
+
+ /*
+ * Unfortunately StartBufferIO() returning false doesn't allow to
+ * distinguish between the buffer already being valid and IO already
+ * being in progress. Since IO already being in progress is quite
+ * rare, this approach seems fine.
+ */
+ pgaio_submit_staged();
+ }
+
+ return ReadBuffersCanStartIOOnce(buffer, nowait);
+}
+
+/*
+ * Helper for WaitReadBuffers() that processes the results of a readv
+ * operation, raising an error if necessary.
+ */
+static void
+ProcessReadBuffersResult(ReadBuffersOperation *operation)
+{
+ PgAioReturn *aio_ret = &operation->io_return;
+ int nblocks = 0;
+
+ Assert(pgaio_wref_valid(&operation->io_wref));
+ Assert(aio_ret->result.status != PGAIO_RS_UNKNOWN);
+
+ /*
+ * SMGR reports the number of blocks successfully read as the result of
+ * the IO operation. Thus we can simply add that to ->nblocks_done.
+ */
+
+ if (likely(aio_ret->result.status == PGAIO_RS_OK))
+ nblocks = aio_ret->result.result;
+ else if (aio_ret->result.status == PGAIO_RS_PARTIAL)
+ {
+ /*
+ * We'll retry, so we just emit a debug message to the server log (or
+ * not even that in prod scenarios).
+ */
+ pgaio_result_report(aio_ret->result, &aio_ret->target_data, DEBUG1);
+ nblocks = aio_ret->result.result;
+
+ elog(DEBUG3, "partial read, will retry");
+
+ }
+ else if (aio_ret->result.status == PGAIO_RS_ERROR)
+ {
+ pgaio_result_report(aio_ret->result, &aio_ret->target_data, ERROR);
+ nblocks = 0; /* silence compiler */
+ }
Assert(nblocks > 0);
Assert(nblocks <= MAX_IO_COMBINE_LIMIT);
+ operation->nblocks_done += nblocks;
+
+ Assert(operation->nblocks_done <= operation->nblocks);
+}
+
+void
+WaitReadBuffers(ReadBuffersOperation *operation)
+{
+ PgAioReturn *aio_ret = &operation->io_return;
+ IOContext io_context;
+ IOObject io_object;
+
+ if (operation->persistence == RELPERSISTENCE_TEMP)
+ {
+ io_context = IOCONTEXT_NORMAL;
+ io_object = IOOBJECT_TEMP_RELATION;
+ }
+ else
+ {
+ io_context = IOContextForStrategy(operation->strategy);
+ io_object = IOOBJECT_RELATION;
+ }
+
+ /*
+ * If we get here without an IO operation having been issued, the
+ * io_method == IOMETHOD_SYNC path must have been used. Otherwise the
+ * caller should not have called WaitReadBuffers().
+ *
+ * In the case of IOMETHOD_SYNC, we start - as we used to before the
+ * introducing of AIO - the IO in WaitReadBuffers(). This is done as part
+ * of the retry logic below, no extra code is required.
+ *
+ * This path is expected to eventually go away.
+ */
+ if (!pgaio_wref_valid(&operation->io_wref) && io_method != IOMETHOD_SYNC)
+ elog(ERROR, "waiting for read operation that didn't read");
+
+ /*
+ * To handle partial reads, and IOMETHOD_SYNC, we re-issue IO until we're
+ * done. We may need multiple retries, not just because we could get
+ * multiple partial reads, but also because some of the remaining
+ * to-be-read buffers may have been read in by other backends, limiting
+ * the IO size.
+ */
+ while (true)
+ {
+ int nblocks;
+
+ CheckReadBuffersOperation(operation, false);
+
+ /*
+ * If there is an IO associated with the operation, we may need to
+ * wait for it.
+ */
+ if (pgaio_wref_valid(&operation->io_wref))
+ {
+ /*
+ * Track the time spent waiting for the IO to complete. As
+ * tracking a wait even if we don't actually need to wait
+ *
+ * a) is not cheap, due to the timestamping overhead
+ *
+ * b) reports some time as waiting, even if we never waited
+ *
+ * we first check if we already know the IO is complete.
+ */
+ if (aio_ret->result.status == PGAIO_RS_UNKNOWN &&
+ !pgaio_wref_check_done(&operation->io_wref))
+ {
+ instr_time io_start = pgstat_prepare_io_time(track_io_timing);
+
+ pgaio_wref_wait(&operation->io_wref);
+
+ /*
+ * The IO operation itself was already counted earlier, in
+ * AsyncReadBuffers(), this just accounts for the wait time.
+ */
+ pgstat_count_io_op_time(io_object, io_context, IOOP_READ,
+ io_start, 0, 0);
+ }
+ else
+ {
+ Assert(pgaio_wref_check_done(&operation->io_wref));
+ }
+
+ /*
+ * We now are sure the IO completed. Check the results. This
+ * includes reporting on errors if there were any.
+ */
+ ProcessReadBuffersResult(operation);
+ }
+
+ /*
+ * Most of the time, the one IO we already started, will read in
+ * everything. But we need to deal with partial reads and buffers not
+ * needing IO anymore.
+ */
+ if (operation->nblocks_done == operation->nblocks)
+ break;
+
+ CHECK_FOR_INTERRUPTS();
+
+ /*
+ * This may only complete the IO partially, either because some
+ * buffers were already valid, or because of a partial read.
+ *
+ * NB: In contrast to after the AsyncReadBuffers() call in
+ * StartReadBuffers(), we do *not* reduce
+ * ReadBuffersOperation->nblocks here, callers expect the full
+ * operation to be completed at this point (as more operations may
+ * have been queued).
+ */
+ AsyncReadBuffers(operation, &nblocks);
+ }
+
+ CheckReadBuffersOperation(operation, true);
+
+ /* NB: READ_DONE tracepoint was already executed in completion callback */
+}
+
+/*
+ * Initiate IO for the ReadBuffersOperation
+ *
+ * This function only starts a single IO at a time. The size of the IO may be
+ * limited to below the to-be-read blocks, if one of the buffers has
+ * concurrently been read in. If the first to-be-read buffer is already valid,
+ * no IO will be issued.
+ *
+ * To support retries after partial reads, the first operation->nblocks_done
+ * buffers are skipped.
+ *
+ * On return *nblocks_progress is updated to reflect the number of buffers
+ * affected by the call. If the first buffer is valid, *nblocks_progress is
+ * set to 1 and operation->nblocks_done is incremented.
+ *
+ * Returns true if IO was initiated, false if no IO was necessary.
+ */
+static bool
+AsyncReadBuffers(ReadBuffersOperation *operation, int *nblocks_progress)
+{
+ Buffer *buffers = &operation->buffers[0];
+ int flags = operation->flags;
+ BlockNumber blocknum = operation->blocknum;
+ ForkNumber forknum = operation->forknum;
+ char persistence = operation->persistence;
+ int16 nblocks_done = operation->nblocks_done;
+ Buffer *io_buffers = &operation->buffers[nblocks_done];
+ int io_buffers_len = 0;
+ PgAioHandle *ioh;
+ uint32 ioh_flags = 0;
+ void *io_pages[MAX_IO_COMBINE_LIMIT];
+ IOContext io_context;
+ IOObject io_object;
+ bool did_start_io;
+
+ /*
+ * When this IO is executed synchronously, either because the caller will
+ * immediately block waiting for the IO or because IOMETHOD_SYNC is used,
+ * the AIO subsystem needs to know.
+ */
+ if (flags & READ_BUFFERS_SYNCHRONOUSLY)
+ ioh_flags |= PGAIO_HF_SYNCHRONOUS;
+
if (persistence == RELPERSISTENCE_TEMP)
{
io_context = IOCONTEXT_NORMAL;
io_object = IOOBJECT_TEMP_RELATION;
+ ioh_flags |= PGAIO_HF_REFERENCES_LOCAL;
}
else
{
@@ -1493,140 +1804,157 @@ WaitReadBuffers(ReadBuffersOperation *operation)
io_object = IOOBJECT_RELATION;
}
- for (int i = 0; i < nblocks; ++i)
+ /*
+ * Get IO handle before ReadBuffersCanStartIO(), as pgaio_io_acquire()
+ * might block, which we don't want after setting IO_IN_PROGRESS.
+ *
+ * If we need to wait for IO before we can get a handle, submit already
+ * staged IO first, so that other backends don't need to wait. There
+ * wouldn't be a deadlock risk, as pgaio_io_acquire() just needs to wait
+ * for already submitted IO, which doesn't require additional locks, but
+ * it could still cause undesirable waits.
+ *
+ * A secondary benefit is that this would allow us to measure the time in
+ * pgaio_io_acquire() without causing undue timer overhead in the common,
+ * non-blocking, case. However, currently the pgstats infrastructure
+ * doesn't really allow that, as it a) asserts that an operation can't
+ * have time without operations b) doesn't have an API to report
+ * "accumulated" time.
+ */
+ ioh = pgaio_io_acquire_nb(CurrentResourceOwner, &operation->io_return);
+ if (unlikely(!ioh))
+ {
+ pgaio_submit_staged();
+
+ ioh = pgaio_io_acquire(CurrentResourceOwner, &operation->io_return);
+ }
+
+ /*
+ * Check if we can start IO on the first to-be-read buffer.
+ *
+ * If an I/O is already in progress in another backend, we want to wait
+ * for the outcome: either done, or something went wrong and we will
+ * retry.
+ */
+ if (!ReadBuffersCanStartIO(buffers[nblocks_done], false))
+ {
+ /*
+ * Someone else has already completed this block, we're done.
+ *
+ * When IO is necessary, ->nblocks_done is updated in
+ * ProcessReadBuffersResult(), but that is not called if no IO is
+ * necessary. Thus update here.
+ */
+ operation->nblocks_done += 1;
+ *nblocks_progress = 1;
+
+ pgaio_io_release(ioh);
+ pgaio_wref_clear(&operation->io_wref);
+ did_start_io = false;
+
+ /*
+ * Report and track this as a 'hit' for this backend, even though it
+ * must have started out as a miss in PinBufferForBlock(). The other
+ * backend will track this as a 'read'.
+ */
+ TRACE_POSTGRESQL_BUFFER_READ_DONE(forknum, blocknum + operation->nblocks_done,
+ operation->smgr->smgr_rlocator.locator.spcOid,
+ operation->smgr->smgr_rlocator.locator.dbOid,
+ operation->smgr->smgr_rlocator.locator.relNumber,
+ operation->smgr->smgr_rlocator.backend,
+ true);
+
+ if (persistence == RELPERSISTENCE_TEMP)
+ pgBufferUsage.local_blks_hit += 1;
+ else
+ pgBufferUsage.shared_blks_hit += 1;
+
+ if (operation->rel)
+ pgstat_count_buffer_hit(operation->rel);
+
+ pgstat_count_io_op(io_object, io_context, IOOP_HIT, 1, 0);
+
+ if (VacuumCostActive)
+ VacuumCostBalance += VacuumCostPageHit;
+ }
+ else
{
- int io_buffers_len;
- Buffer io_buffers[MAX_IO_COMBINE_LIMIT];
- void *io_pages[MAX_IO_COMBINE_LIMIT];
instr_time io_start;
- BlockNumber io_first_block;
-
- /*
- * Skip this block if someone else has already completed it. If an
- * I/O is already in progress in another backend, this will wait for
- * the outcome: either done, or something went wrong and we will
- * retry.
- */
- if (!WaitReadBuffersCanStartIO(buffers[i], false))
- {
- /*
- * Report and track this as a 'hit' for this backend, even though
- * it must have started out as a miss in PinBufferForBlock(). The
- * other backend will track this as a 'read'.
- */
- TRACE_POSTGRESQL_BUFFER_READ_DONE(forknum, blocknum + i,
- operation->smgr->smgr_rlocator.locator.spcOid,
- operation->smgr->smgr_rlocator.locator.dbOid,
- operation->smgr->smgr_rlocator.locator.relNumber,
- operation->smgr->smgr_rlocator.backend,
- true);
-
- if (persistence == RELPERSISTENCE_TEMP)
- pgBufferUsage.local_blks_hit += 1;
- else
- pgBufferUsage.shared_blks_hit += 1;
-
- if (operation->rel)
- pgstat_count_buffer_hit(operation->rel);
-
- pgstat_count_io_op(io_object, io_context, IOOP_HIT, 1, 0);
-
- if (VacuumCostActive)
- VacuumCostBalance += VacuumCostPageHit;
-
- continue;
- }
/* We found a buffer that we need to read in. */
- io_buffers[0] = buffers[i];
- io_pages[0] = BufferGetBlock(buffers[i]);
- io_first_block = blocknum + i;
+ Assert(io_buffers[0] == buffers[nblocks_done]);
+ io_pages[0] = BufferGetBlock(buffers[nblocks_done]);
io_buffers_len = 1;
/*
* How many neighboring-on-disk blocks can we scatter-read into other
* buffers at the same time? In this case we don't wait if we see an
- * I/O already in progress. We already hold BM_IO_IN_PROGRESS for the
+ * I/O already in progress. We already set BM_IO_IN_PROGRESS for the
* head block, so we should get on with that I/O as soon as possible.
- * We'll come back to this block again, above.
+ *
+ * We'll come back to this block in the next call to
+ * StartReadBuffers() -> AsyncReadBuffers().
*/
- while ((i + 1) < nblocks &&
- WaitReadBuffersCanStartIO(buffers[i + 1], true))
+ for (int i = nblocks_done + 1; i < operation->nblocks; i++)
{
+ if (!ReadBuffersCanStartIO(buffers[i], true))
+ break;
/* Must be consecutive block numbers. */
- Assert(BufferGetBlockNumber(buffers[i + 1]) ==
- BufferGetBlockNumber(buffers[i]) + 1);
+ Assert(BufferGetBlockNumber(buffers[i - 1]) ==
+ BufferGetBlockNumber(buffers[i]) - 1);
+ Assert(io_buffers[io_buffers_len] == buffers[i]);
- io_buffers[io_buffers_len] = buffers[++i];
io_pages[io_buffers_len++] = BufferGetBlock(buffers[i]);
}
+ /* get a reference to wait for in WaitReadBuffers() */
+ pgaio_io_get_wref(ioh, &operation->io_wref);
+
+ /* provide the list of buffers to the completion callbacks */
+ pgaio_io_set_handle_data_32(ioh, (uint32 *) io_buffers, io_buffers_len);
+
+ pgaio_io_register_callbacks(ioh,
+ persistence == RELPERSISTENCE_TEMP ?
+ PGAIO_HCB_LOCAL_BUFFER_READV :
+ PGAIO_HCB_SHARED_BUFFER_READV,
+ flags);
+
+ pgaio_io_set_flag(ioh, ioh_flags);
+
+ /* ---
+ * Even though we're trying to issue IO asynchronously, track the time
+ * in smgrstartreadv():
+ * - if io_method == IOMETHOD_SYNC, we will always perform the IO
+ * immediately
+ * - the io method might not support the IO (e.g. worker IO for a temp
+ * table)
+ * ---
+ */
io_start = pgstat_prepare_io_time(track_io_timing);
- smgrreadv(operation->smgr, forknum, io_first_block, io_pages, io_buffers_len);
- pgstat_count_io_op_time(io_object, io_context, IOOP_READ, io_start,
- 1, io_buffers_len * BLCKSZ);
-
- /* Verify each block we read, and terminate the I/O. */
- for (int j = 0; j < io_buffers_len; ++j)
- {
- BufferDesc *bufHdr;
- Block bufBlock;
-
- if (persistence == RELPERSISTENCE_TEMP)
- {
- bufHdr = GetLocalBufferDescriptor(-io_buffers[j] - 1);
- bufBlock = LocalBufHdrGetBlock(bufHdr);
- }
- else
- {
- bufHdr = GetBufferDescriptor(io_buffers[j] - 1);
- bufBlock = BufHdrGetBlock(bufHdr);
- }
-
- /* check for garbage data */
- if (!PageIsVerifiedExtended((Page) bufBlock, io_first_block + j,
- PIV_LOG_WARNING | PIV_REPORT_STAT))
- {
- if ((operation->flags & READ_BUFFERS_ZERO_ON_ERROR) || zero_damaged_pages)
- {
- ereport(WARNING,
- (errcode(ERRCODE_DATA_CORRUPTED),
- errmsg("invalid page in block %u of relation %s; zeroing out page",
- io_first_block + j,
- relpath(operation->smgr->smgr_rlocator, forknum).str)));
- memset(bufBlock, 0, BLCKSZ);
- }
- else
- ereport(ERROR,
- (errcode(ERRCODE_DATA_CORRUPTED),
- errmsg("invalid page in block %u of relation %s",
- io_first_block + j,
- relpath(operation->smgr->smgr_rlocator, forknum).str)));
- }
-
- /* Set BM_VALID, terminate IO, and wake up any waiters */
- if (persistence == RELPERSISTENCE_TEMP)
- TerminateLocalBufferIO(bufHdr, false, BM_VALID, true);
- else
- TerminateBufferIO(bufHdr, false, BM_VALID, true, true);
-
- /* Report I/Os as completing individually. */
- TRACE_POSTGRESQL_BUFFER_READ_DONE(forknum, io_first_block + j,
- operation->smgr->smgr_rlocator.locator.spcOid,
- operation->smgr->smgr_rlocator.locator.dbOid,
- operation->smgr->smgr_rlocator.locator.relNumber,
- operation->smgr->smgr_rlocator.backend,
- false);
- }
+ smgrstartreadv(ioh, operation->smgr, forknum,
+ blocknum + nblocks_done,
+ io_pages, io_buffers_len);
+ pgstat_count_io_op_time(io_object, io_context, IOOP_READ,
+ io_start, 1, *nblocks_progress * BLCKSZ);
if (persistence == RELPERSISTENCE_TEMP)
pgBufferUsage.local_blks_read += io_buffers_len;
else
pgBufferUsage.shared_blks_read += io_buffers_len;
+ /*
+ * Track vacuum cost when issuing IO, not after waiting for it.
+ * Otherwise we could end up issuing a lot of IO in a short timespan,
+ * despite a low cost limit.
+ */
if (VacuumCostActive)
VacuumCostBalance += VacuumCostPageMiss * io_buffers_len;
+
+ *nblocks_progress = io_buffers_len;
+ did_start_io = true;
}
+
+ return did_start_io;
}
/*
--
2.48.1.76.g4e746b1a31.dirty
[text/x-diff] v2.11-0013-aio-Basic-read_stream-adjustments-for-real-AIO.patch (3.8K, ../../6ak556uyqiptdwjaci4kbi5eykwkmzqgkbtkyaosjnopjhncrc@2v4ac2jwyz22/14-v2.11-0013-aio-Basic-read_stream-adjustments-for-real-AIO.patch)
download | inline diff:
From b6d647cffe0ec26b4d2cc03fe324a76cd60abdaf Mon Sep 17 00:00:00 2001
From: Andres Freund <andres@anarazel.de>
Date: Tue, 18 Mar 2025 14:40:06 -0400
Subject: [PATCH v2.11 13/27] aio: Basic read_stream adjustments for real AIO
Adapt the read stream logic for real AIO:
- If AIO is enabled, we shouldn't issue advice, but if it isn't, we should
continue issuing advice
- AIO benefits from reading ahead with direct IO
- While in read_stream_look_ahead(), we can use AIO batch submission mode for
increased efficiency
There is one comment talking about max_ios logic with "real asynchronous I/O"
that I am not sure about, so I left it alone for now.
There are further improvements we should consider, e.g. waiting to issue IOs
until we can issue multiple IOs at once. But that's left for a future change,
since it would involve additional heuristics.
---
src/backend/storage/aio/read_stream.c | 29 ++++++++++++++++++++++-----
1 file changed, 24 insertions(+), 5 deletions(-)
diff --git a/src/backend/storage/aio/read_stream.c b/src/backend/storage/aio/read_stream.c
index 18ecf4affc7..60a841816d8 100644
--- a/src/backend/storage/aio/read_stream.c
+++ b/src/backend/storage/aio/read_stream.c
@@ -72,6 +72,7 @@
#include "postgres.h"
#include "miscadmin.h"
+#include "storage/aio.h"
#include "storage/fd.h"
#include "storage/smgr.h"
#include "storage/read_stream.h"
@@ -99,6 +100,7 @@ struct ReadStream
int16 pinned_buffers;
int16 distance;
int16 initialized_buffers;
+ bool sync_mode;
bool advice_enabled;
bool temporary;
@@ -416,6 +418,13 @@ read_stream_start_pending_read(ReadStream *stream)
static void
read_stream_look_ahead(ReadStream *stream)
{
+ /*
+ * Allow amortizing the cost of submitting IO over multiple IOs. This
+ * requires that we don't do any operations that could lead to a deadlock
+ * with staged-but-unsubmitted IO.
+ */
+ pgaio_enter_batchmode();
+
while (stream->ios_in_progress < stream->max_ios &&
stream->pinned_buffers + stream->pending_read_nblocks < stream->distance)
{
@@ -463,6 +472,7 @@ read_stream_look_ahead(ReadStream *stream)
{
/* We've hit the buffer or I/O limit. Rewind and stop here. */
read_stream_unget_block(stream, blocknum);
+ pgaio_exit_batchmode();
return;
}
}
@@ -497,6 +507,8 @@ read_stream_look_ahead(ReadStream *stream)
* time.
*/
Assert(stream->pinned_buffers > 0 || stream->distance == 0);
+
+ pgaio_exit_batchmode();
}
/*
@@ -629,15 +641,19 @@ read_stream_begin_impl(int flags,
stream->per_buffer_data = (void *)
MAXALIGN(&stream->ios[Max(1, max_ios)]);
+ stream->sync_mode = io_method == IOMETHOD_SYNC;
+
#ifdef USE_PREFETCH
/*
- * This system supports prefetching advice. We can use it as long as
- * direct I/O isn't enabled, the caller hasn't promised sequential access
- * (overriding our detection heuristics), and max_ios hasn't been set to
- * zero.
+ * This system supports prefetching advice.
+ *
+ * Issue advice only if AIO is not used, direct I/O isn't enabled, the
+ * caller hasn't promised sequential access (overriding our detection
+ * heuristics), and max_ios hasn't been set to zero.
*/
- if ((io_direct_flags & IO_DIRECT_DATA) == 0 &&
+ if (stream->sync_mode &&
+ (io_direct_flags & IO_DIRECT_DATA) == 0 &&
(flags & READ_STREAM_SEQUENTIAL) == 0 &&
max_ios > 0)
stream->advice_enabled = true;
@@ -647,6 +663,9 @@ read_stream_begin_impl(int flags,
* For now, max_ios = 0 is interpreted as max_ios = 1 with advice disabled
* above. If we had real asynchronous I/O we might need a slightly
* different definition.
+ *
+ * FIXME: Not sure what different definition we would need? I guess we
+ * could add the READ_BUFFERS_SYNCHRONOUSLY flag automatically?
*/
if (max_ios == 0)
max_ios = 1;
--
2.48.1.76.g4e746b1a31.dirty
[text/x-diff] v2.11-0014-docs-Reframe-track_io_timing-related-docs-as-w.patch (5.3K, ../../6ak556uyqiptdwjaci4kbi5eykwkmzqgkbtkyaosjnopjhncrc@2v4ac2jwyz22/15-v2.11-0014-docs-Reframe-track_io_timing-related-docs-as-w.patch)
download | inline diff:
From f8344496f6de5c018850e8953c30a1a3c93037a4 Mon Sep 17 00:00:00 2001
From: Andres Freund <andres@anarazel.de>
Date: Tue, 18 Mar 2025 14:40:06 -0400
Subject: [PATCH v2.11 14/27] docs: Reframe track_io_timing related docs as
wait time
With AIO it does not make sense anymore to track the time for each individual
IO, as multiple IOs can be in-flight at the same time. Instead we now track
the time spent *waiting* for IOs.
This should be reflected in the docs. While, so far, we only do a subset of
reads, and no other operations, via AIO, describing the GUC and view columns
as measuring IO waits is accurate for synchronous and asynchronous IO.
Discussion: https://postgr.es/m/5dzyoduxlvfg55oqtjyjehez5uoq6hnwgzor4kkybkfdgkj7ag@rbi4gsmzaczk
---
doc/src/sgml/config.sgml | 4 ++--
doc/src/sgml/monitoring.sgml | 12 ++++++------
2 files changed, 8 insertions(+), 8 deletions(-)
diff --git a/doc/src/sgml/config.sgml b/doc/src/sgml/config.sgml
index 4f71725036b..9f63c4574f9 100644
--- a/doc/src/sgml/config.sgml
+++ b/doc/src/sgml/config.sgml
@@ -8578,7 +8578,7 @@ COPY postgres_log FROM '/full/path/to/logfile.csv' WITH csv;
</term>
<listitem>
<para>
- Enables timing of database I/O calls. This parameter is off by
+ Enables timing of database I/O waits. This parameter is off by
default, as it will repeatedly query the operating system for
the current time, which may cause significant overhead on some
platforms. You can use the <xref linkend="pgtesttiming"/> tool to
@@ -8612,7 +8612,7 @@ COPY postgres_log FROM '/full/path/to/logfile.csv' WITH csv;
</term>
<listitem>
<para>
- Enables timing of WAL I/O calls. This parameter is off by default,
+ Enables timing of WAL I/O waits. This parameter is off by default,
as it will repeatedly query the operating system for the current time,
which may cause significant overhead on some platforms.
You can use the <application>pg_test_timing</application> tool to
diff --git a/doc/src/sgml/monitoring.sgml b/doc/src/sgml/monitoring.sgml
index aaa6586d3a4..7598340072f 100644
--- a/doc/src/sgml/monitoring.sgml
+++ b/doc/src/sgml/monitoring.sgml
@@ -2733,7 +2733,7 @@ description | Waiting for a newly initialized WAL file to reach durable storage
<structfield>read_time</structfield> <type>double precision</type>
</para>
<para>
- Time spent in read operations in milliseconds (if
+ Time spent waiting for read operations in milliseconds (if
<xref linkend="guc-track-io-timing"/> is enabled and
<varname>object</varname> is not <literal>wal</literal>,
or if <xref linkend="guc-track-wal-io-timing"/> is enabled
@@ -2771,7 +2771,7 @@ description | Waiting for a newly initialized WAL file to reach durable storage
<structfield>write_time</structfield> <type>double precision</type>
</para>
<para>
- Time spent in write operations in milliseconds (if
+ Time spent waiting for write operations in milliseconds (if
<xref linkend="guc-track-io-timing"/> is enabled and
<varname>object</varname> is not <literal>wal</literal>,
or if <xref linkend="guc-track-wal-io-timing"/> is enabled
@@ -2799,7 +2799,7 @@ description | Waiting for a newly initialized WAL file to reach durable storage
<structfield>writeback_time</structfield> <type>double precision</type>
</para>
<para>
- Time spent in writeback operations in milliseconds (if
+ Time spent waiting for writeback operations in milliseconds (if
<xref linkend="guc-track-io-timing"/> is enabled, otherwise zero). This
includes the time spent queueing write-out requests and, potentially,
the time spent to write out the dirty data.
@@ -2835,7 +2835,7 @@ description | Waiting for a newly initialized WAL file to reach durable storage
<structfield>extend_time</structfield> <type>double precision</type>
</para>
<para>
- Time spent in extend operations in milliseconds. (if
+ Time spent waiting for extend operations in milliseconds. (if
<xref linkend="guc-track-io-timing"/> is enabled and
<varname>object</varname> is not <literal>wal</literal>,
or if <xref linkend="guc-track-wal-io-timing"/> is enabled
@@ -2909,7 +2909,7 @@ description | Waiting for a newly initialized WAL file to reach durable storage
<structfield>fsync_time</structfield> <type>double precision</type>
</para>
<para>
- Time spent in fsync operations in milliseconds (if
+ Time spent waiting for fsync operations in milliseconds (if
<xref linkend="guc-track-io-timing"/> is enabled and
<varname>object</varname> is not <literal>wal</literal>,
or if <xref linkend="guc-track-wal-io-timing"/> is enabled
@@ -2996,7 +2996,7 @@ description | Waiting for a newly initialized WAL file to reach durable storage
<note>
<para>
- Columns tracking I/O time will only be non-zero when
+ Columns tracking I/O wait time will only be non-zero when
<xref linkend="guc-track-io-timing"/> is enabled. The user should be
careful when referencing these columns in combination with their
corresponding I/O operations in case <varname>track_io_timing</varname>
--
2.48.1.76.g4e746b1a31.dirty
[text/x-diff] v2.11-0015-Enable-IO-concurrency-on-all-systems.patch (11.2K, ../../6ak556uyqiptdwjaci4kbi5eykwkmzqgkbtkyaosjnopjhncrc@2v4ac2jwyz22/16-v2.11-0015-Enable-IO-concurrency-on-all-systems.patch)
download | inline diff:
From 59a9323ac26e99b81d1a68762ab3953c960040bf Mon Sep 17 00:00:00 2001
From: Andres Freund <andres@anarazel.de>
Date: Wed, 19 Mar 2025 10:15:20 -0400
Subject: [PATCH v2.11 15/27] Enable IO concurrency on all systems
Previously effective_io_concurrency and maintenance_io_concurrency could
not be set above 0 on machines without fadvise support. AIO enables
IO concurrency without such support.
Currently only subsystems using the read stream API will take advantage
of this. Other users of maintenance_io_concurrency (like recovery
prefetching) which leverage OS advice directly will not benefit from
this change. In those cases, maintenance_io_concurrency will have no
effect on I/O behavior.
Author: Melanie Plageman <melanieplageman@gmail.com>
Discussion: https://postgr.es/m/CAAKRu_atGgZePo=_g6T3cNtfMf0QxpvoUh5OUqa_cnPdhLd=gw@mail.gmail.com
---
src/include/storage/bufmgr.h | 6 ----
src/include/utils/guc_hooks.h | 4 ---
src/backend/access/common/reloptions.c | 8 -----
src/backend/commands/variable.c | 30 -------------------
src/backend/utils/misc/guc_tables.c | 4 +--
src/backend/utils/misc/postgresql.conf.sample | 6 ++--
src/bin/initdb/initdb.c | 5 ----
doc/src/sgml/config.sgml | 15 +++++-----
doc/src/sgml/ref/alter_tablespace.sgml | 2 +-
doc/src/sgml/ref/create_tablespace.sgml | 2 +-
10 files changed, 14 insertions(+), 68 deletions(-)
diff --git a/src/include/storage/bufmgr.h b/src/include/storage/bufmgr.h
index 7abb3bdfd9d..784df8b00cb 100644
--- a/src/include/storage/bufmgr.h
+++ b/src/include/storage/bufmgr.h
@@ -156,14 +156,8 @@ extern PGDLLIMPORT int bgwriter_lru_maxpages;
extern PGDLLIMPORT double bgwriter_lru_multiplier;
extern PGDLLIMPORT bool track_io_timing;
-/* only applicable when prefetching is available */
-#ifdef USE_PREFETCH
#define DEFAULT_EFFECTIVE_IO_CONCURRENCY 16
#define DEFAULT_MAINTENANCE_IO_CONCURRENCY 16
-#else
-#define DEFAULT_EFFECTIVE_IO_CONCURRENCY 0
-#define DEFAULT_MAINTENANCE_IO_CONCURRENCY 0
-#endif
extern PGDLLIMPORT int effective_io_concurrency;
extern PGDLLIMPORT int maintenance_io_concurrency;
diff --git a/src/include/utils/guc_hooks.h b/src/include/utils/guc_hooks.h
index 0f1e74f96c9..799fa7ace68 100644
--- a/src/include/utils/guc_hooks.h
+++ b/src/include/utils/guc_hooks.h
@@ -61,8 +61,6 @@ extern bool check_default_text_search_config(char **newval, void **extra, GucSou
extern void assign_default_text_search_config(const char *newval, void *extra);
extern bool check_default_with_oids(bool *newval, void **extra,
GucSource source);
-extern bool check_effective_io_concurrency(int *newval, void **extra,
- GucSource source);
extern bool check_huge_page_size(int *newval, void **extra, GucSource source);
extern void assign_io_method(int newval, void *extra);
extern bool check_io_max_concurrency(int *newval, void **extra, GucSource source);
@@ -83,8 +81,6 @@ extern bool check_log_stats(bool *newval, void **extra, GucSource source);
extern bool check_log_timezone(char **newval, void **extra, GucSource source);
extern void assign_log_timezone(const char *newval, void *extra);
extern const char *show_log_timezone(void);
-extern bool check_maintenance_io_concurrency(int *newval, void **extra,
- GucSource source);
extern void assign_maintenance_io_concurrency(int newval, void *extra);
extern void assign_io_max_combine_limit(int newval, void *extra);
extern void assign_io_combine_limit(int newval, void *extra);
diff --git a/src/backend/access/common/reloptions.c b/src/backend/access/common/reloptions.c
index 645b5c00467..46c1dce222d 100644
--- a/src/backend/access/common/reloptions.c
+++ b/src/backend/access/common/reloptions.c
@@ -361,11 +361,7 @@ static relopt_int intRelOpts[] =
RELOPT_KIND_TABLESPACE,
ShareUpdateExclusiveLock
},
-#ifdef USE_PREFETCH
-1, 0, MAX_IO_CONCURRENCY
-#else
- 0, 0, 0
-#endif
},
{
{
@@ -374,11 +370,7 @@ static relopt_int intRelOpts[] =
RELOPT_KIND_TABLESPACE,
ShareUpdateExclusiveLock
},
-#ifdef USE_PREFETCH
-1, 0, MAX_IO_CONCURRENCY
-#else
- 0, 0, 0
-#endif
},
{
{
diff --git a/src/backend/commands/variable.c b/src/backend/commands/variable.c
index f550a3c0c63..d3567aa7f27 100644
--- a/src/backend/commands/variable.c
+++ b/src/backend/commands/variable.c
@@ -1145,7 +1145,6 @@ check_cluster_name(char **newval, void **extra, GucSource source)
void
assign_maintenance_io_concurrency(int newval, void *extra)
{
-#ifdef USE_PREFETCH
/*
* Reconfigure recovery prefetching, because a setting it depends on
* changed.
@@ -1153,7 +1152,6 @@ assign_maintenance_io_concurrency(int newval, void *extra)
maintenance_io_concurrency = newval;
if (AmStartupProcess())
XLogPrefetchReconfigure();
-#endif
}
/*
@@ -1249,34 +1247,6 @@ check_default_with_oids(bool *newval, void **extra, GucSource source)
return true;
}
-bool
-check_effective_io_concurrency(int *newval, void **extra, GucSource source)
-{
-#ifndef USE_PREFETCH
- if (*newval != 0)
- {
- GUC_check_errdetail("\"%s\" must be set to 0 on platforms that lack support for issuing read-ahead advice.",
- "effective_io_concurrency");
- return false;
- }
-#endif /* USE_PREFETCH */
- return true;
-}
-
-bool
-check_maintenance_io_concurrency(int *newval, void **extra, GucSource source)
-{
-#ifndef USE_PREFETCH
- if (*newval != 0)
- {
- GUC_check_errdetail("\"%s\" must be set to 0 on platforms that lack support for issuing read-ahead advice.",
- "maintenance_io_concurrency");
- return false;
- }
-#endif /* USE_PREFETCH */
- return true;
-}
-
bool
check_ssl(bool *newval, void **extra, GucSource source)
{
diff --git a/src/backend/utils/misc/guc_tables.c b/src/backend/utils/misc/guc_tables.c
index 75bb0acdf0f..2e17dbc8c9e 100644
--- a/src/backend/utils/misc/guc_tables.c
+++ b/src/backend/utils/misc/guc_tables.c
@@ -3245,7 +3245,7 @@ struct config_int ConfigureNamesInt[] =
&effective_io_concurrency,
DEFAULT_EFFECTIVE_IO_CONCURRENCY,
0, MAX_IO_CONCURRENCY,
- check_effective_io_concurrency, NULL, NULL
+ NULL, NULL, NULL
},
{
@@ -3259,7 +3259,7 @@ struct config_int ConfigureNamesInt[] =
&maintenance_io_concurrency,
DEFAULT_MAINTENANCE_IO_CONCURRENCY,
0, MAX_IO_CONCURRENCY,
- check_maintenance_io_concurrency, assign_maintenance_io_concurrency,
+ NULL, assign_maintenance_io_concurrency,
NULL
},
diff --git a/src/backend/utils/misc/postgresql.conf.sample b/src/backend/utils/misc/postgresql.conf.sample
index 69dc727174b..d5f5f3dbda4 100644
--- a/src/backend/utils/misc/postgresql.conf.sample
+++ b/src/backend/utils/misc/postgresql.conf.sample
@@ -198,11 +198,11 @@
# - I/O -
#backend_flush_after = 0 # measured in pages, 0 disables
-#effective_io_concurrency = 16 # 1-1000; 0 disables prefetching
-#maintenance_io_concurrency = 16 # 1-1000; 0 disables prefetching
+#effective_io_concurrency = 16 # 1-1000; 0 disables issuing multiple simultaneous IO requests
+#maintenance_io_concurrency = 16 # 1-1000; same as effective_io_concurrency
#io_max_combine_limit = 128kB # usually 1-128 blocks (depends on OS)
# (change requires restart)
-#io_combine_limit = 128kB # usually 1-128 blocks (depends on OS)
+#io_combine_limit = 128kB # usually 1-32 blocks (depends on OS)
#io_method = worker # worker, io_uring, sync
# (change requires restart)
diff --git a/src/bin/initdb/initdb.c b/src/bin/initdb/initdb.c
index 21a0fe3ecd9..b9d75799ee0 100644
--- a/src/bin/initdb/initdb.c
+++ b/src/bin/initdb/initdb.c
@@ -1401,11 +1401,6 @@ setup_config(void)
repltok, true);
#endif
-#ifndef USE_PREFETCH
- conflines = replace_guc_value(conflines, "effective_io_concurrency",
- "0", true);
-#endif
-
#ifdef WIN32
conflines = replace_guc_value(conflines, "update_process_title",
"off", true);
diff --git a/doc/src/sgml/config.sgml b/doc/src/sgml/config.sgml
index 9f63c4574f9..863a5d025c4 100644
--- a/doc/src/sgml/config.sgml
+++ b/doc/src/sgml/config.sgml
@@ -2585,8 +2585,7 @@ include_dir 'conf.d'
session attempts to initiate in parallel. The allowed range is
<literal>1</literal> to <literal>1000</literal>, or
<literal>0</literal> to disable issuance of asynchronous I/O requests.
- The default is <literal>16</literal> on supported systems, otherwise
- <literal>0</literal>.
+ The default is <literal>16</literal>.
</para>
<para>
@@ -2597,8 +2596,8 @@ include_dir 'conf.d'
</para>
<para>
- On systems without prefetch advice support, attempting to configure
- any value other than <literal>0</literal> will error out.
+ On systems with prefetch advice support,
+ <varname>effective_io_concurrency</varname> also controls the prefetch distance.
</para>
<para>
@@ -2621,10 +2620,10 @@ include_dir 'conf.d'
for maintenance work that is done on behalf of many client sessions.
</para>
<para>
- The default is <literal>16</literal> on supported systems, otherwise
- <literal>0</literal>. This value can be overridden for tables in a
- particular tablespace by setting the tablespace parameter of the same
- name (see <xref linkend="sql-altertablespace"/>).
+ The default is <literal>16</literal>. This value can be overridden
+ for tables in a particular tablespace by setting the tablespace
+ parameter of the same name (see <xref
+ linkend="sql-altertablespace"/>).
</para>
</listitem>
</varlistentry>
diff --git a/doc/src/sgml/ref/alter_tablespace.sgml b/doc/src/sgml/ref/alter_tablespace.sgml
index 6ec863400d1..d0e08089ddb 100644
--- a/doc/src/sgml/ref/alter_tablespace.sgml
+++ b/doc/src/sgml/ref/alter_tablespace.sgml
@@ -88,7 +88,7 @@ ALTER TABLESPACE <replaceable>name</replaceable> RESET ( <replaceable class="par
and <varname>maintenance_io_concurrency</varname>.
Setting these values for a particular tablespace will override the
planner's usual estimate of the cost of reading pages from tables in
- that tablespace, and the executor's prefetching behavior, as established
+ that tablespace, and how many concurrent I/Os are issued, as established
by the configuration parameters of the
same name (see <xref linkend="guc-seq-page-cost"/>,
<xref linkend="guc-random-page-cost"/>,
diff --git a/doc/src/sgml/ref/create_tablespace.sgml b/doc/src/sgml/ref/create_tablespace.sgml
index 9d5ab025261..b77e774c53f 100644
--- a/doc/src/sgml/ref/create_tablespace.sgml
+++ b/doc/src/sgml/ref/create_tablespace.sgml
@@ -110,7 +110,7 @@ CREATE TABLESPACE <replaceable class="parameter">tablespace_name</replaceable>
and <varname>maintenance_io_concurrency</varname>.
Setting these values for a particular tablespace will override the
planner's usual estimate of the cost of reading pages from tables in
- that tablespace, and the executor's prefetching behavior, as established
+ that tablespace, and how many concurrent I/Os are issued, as established
by the configuration parameters of the
same name (see <xref linkend="guc-seq-page-cost"/>,
<xref linkend="guc-random-page-cost"/>,
--
2.48.1.76.g4e746b1a31.dirty
[text/x-diff] v2.11-0016-aio-Add-pg_aios-view.patch (9.8K, ../../6ak556uyqiptdwjaci4kbi5eykwkmzqgkbtkyaosjnopjhncrc@2v4ac2jwyz22/17-v2.11-0016-aio-Add-pg_aios-view.patch)
download | inline diff:
From a12552dfdda77bf66f7f248ba3265220f8f848e3 Mon Sep 17 00:00:00 2001
From: Andres Freund <andres@anarazel.de>
Date: Tue, 18 Mar 2025 14:40:06 -0400
Subject: [PATCH v2.11 16/27] aio: Add pg_aios view
TODO:
- decide on name
- add docs
FIXME:
- catversion bump
Author:
Reviewed-by:
Discussion: https://postgr.es/m/
Backpatch:
---
src/include/catalog/pg_proc.dat | 10 ++
src/backend/catalog/system_views.sql | 3 +
src/backend/storage/aio/Makefile | 1 +
src/backend/storage/aio/aio_funcs.c | 219 +++++++++++++++++++++++++++
src/backend/storage/aio/meson.build | 1 +
src/test/regress/expected/rules.out | 16 ++
6 files changed, 250 insertions(+)
create mode 100644 src/backend/storage/aio/aio_funcs.c
diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat
index 890822eaf79..017971011f3 100644
--- a/src/include/catalog/pg_proc.dat
+++ b/src/include/catalog/pg_proc.dat
@@ -12479,4 +12479,14 @@
proargtypes => 'int4',
prosrc => 'gist_stratnum_common' },
+# AIO related functions
+{ oid => '9200', descr => 'information about in-progress asynchronous IOs',
+ proname => 'pg_get_aios', prorows => '100', proretset => 't',
+ provolatile => 'v', proparallel => 'r', prorettype => 'record', proargtypes => '',
+ proallargtypes => '{int4,int4,int8,text,text,int8,int8,text,int2,int4,text,text,bool,bool,bool}',
+ proargmodes => '{o,o,o,o,o,o,o,o,o,o,o,o,o,o,o}',
+ proargnames => '{pid,io_id,io_generation,state,operation,offset,length,target,handle_data_len,raw_result,result,target_desc,f_sync,f_localmem,f_buffered}',
+ prosrc => 'pg_get_aios' },
+
+
]
diff --git a/src/backend/catalog/system_views.sql b/src/backend/catalog/system_views.sql
index a4d2cfdcaf5..73902a763d1 100644
--- a/src/backend/catalog/system_views.sql
+++ b/src/backend/catalog/system_views.sql
@@ -1390,3 +1390,6 @@ CREATE VIEW pg_stat_subscription_stats AS
CREATE VIEW pg_wait_events AS
SELECT * FROM pg_get_wait_events();
+
+CREATE VIEW pg_aios AS
+ SELECT * FROM pg_get_aios();
diff --git a/src/backend/storage/aio/Makefile b/src/backend/storage/aio/Makefile
index c06c50771e0..3f2469cc399 100644
--- a/src/backend/storage/aio/Makefile
+++ b/src/backend/storage/aio/Makefile
@@ -11,6 +11,7 @@ include $(top_builddir)/src/Makefile.global
OBJS = \
aio.o \
aio_callback.o \
+ aio_funcs.o \
aio_init.o \
aio_io.o \
aio_target.o \
diff --git a/src/backend/storage/aio/aio_funcs.c b/src/backend/storage/aio/aio_funcs.c
new file mode 100644
index 00000000000..58883fd61d8
--- /dev/null
+++ b/src/backend/storage/aio/aio_funcs.c
@@ -0,0 +1,219 @@
+/*-------------------------------------------------------------------------
+ *
+ * aio_funcs.c
+ * AIO - SQL interface for AIO
+ *
+ *
+ * Portions Copyright (c) 1996-2025, PostgreSQL Global Development Group
+ * Portions Copyright (c) 1994, Regents of the University of California
+ *
+ * IDENTIFICATION
+ * src/backend/storage/aio/aio_funcs.c
+ *
+ *-------------------------------------------------------------------------
+ */
+
+#include "postgres.h"
+
+#include "storage/aio.h"
+#include "storage/aio_internal.h"
+#include "utils/builtins.h"
+#include "funcapi.h"
+#include "storage/proc.h"
+
+
+/*
+ * Byte length of an iovec.
+ */
+static size_t
+iov_byte_length(const struct iovec *iov, int cnt)
+{
+ size_t len = 0;
+
+ for (int i = 0; i < cnt; i++)
+ {
+ len += iov[i].iov_len;
+ }
+
+ return len;
+}
+
+Datum
+pg_get_aios(PG_FUNCTION_ARGS)
+{
+ ReturnSetInfo *rsinfo = (ReturnSetInfo *) fcinfo->resultinfo;
+
+ InitMaterializedSRF(fcinfo, 0);
+
+#define PG_GET_AIOS_COLS 15
+
+ for (uint64 i = 0; i < pgaio_ctl->io_handle_count; i++)
+ {
+ PgAioHandle *live_ioh = &pgaio_ctl->io_handles[i];
+ uint32 ioh_id = pgaio_io_get_id(live_ioh);
+ Datum values[PG_GET_AIOS_COLS] = {0};
+ bool nulls[PG_GET_AIOS_COLS] = {0};
+ ProcNumber owner;
+ PGPROC *owner_proc;
+ int32 owner_pid;
+ PgAioHandleState start_state;
+ uint64 start_generation;
+ PgAioHandle ioh_copy;
+ struct iovec iov_copy[PG_IOV_MAX];
+
+retry:
+
+ /*
+ * There is no lock that could prevent the state of the IO to advance
+ * concurrently - and we don't want to introduce one, as that would
+ * introduce atomics into a very common path. Instead we
+ *
+ * 1) Determine the state + generation of the IO.
+ *
+ * 2) Copy the IO to local memory.
+ *
+ * 3) Check if state or generation of the IO changed. If the state
+ * changed, retry, if the generation changed don't display the IO.
+ */
+
+ /* 1) from above */
+ start_generation = live_ioh->generation;
+ pg_read_barrier();
+ start_state = live_ioh->state;
+
+ if (start_state == PGAIO_HS_IDLE)
+ continue;
+
+ /* 2) from above */
+ memcpy(&ioh_copy, live_ioh, sizeof(PgAioHandle));
+
+ /*
+ * Safe to copy even if no iovec is used - we always reserve the
+ * required space.
+ */
+ memcpy(&iov_copy, &pgaio_ctl->iovecs[ioh_copy.iovec_off],
+ PG_IOV_MAX * sizeof(struct iovec));
+
+ /*
+ * Copy information about owner before 3) below, if the process exited
+ * it'd have to wait for the IO to finish first, which we would detect
+ * in 3).
+ */
+ owner = ioh_copy.owner_procno;
+ owner_proc = GetPGProcByNumber(owner);
+ owner_pid = owner_proc->pid;
+
+ /* 3) from above */
+ pg_read_barrier();
+
+ /*
+ * The IO completed and a new one was started with the same ID. Don't
+ * display it - it really started after this function was called.
+ * There be a risk of a livelock if we just retried endlessly, if IOs
+ * complete very quickly.
+ */
+ if (live_ioh->generation != start_generation)
+ continue;
+
+ /*
+ * The IOs state changed while we were "rendering" it. Just start from
+ * scratch. There's no risk of a livelock here, as an IO has a limited
+ * sets of states it can be in, and state changes go only in a single
+ * direction.
+ */
+ if (live_ioh->state != start_state)
+ goto retry;
+
+ /*
+ * Now that we have copied the IO into local memory and checked that
+ * it's still in the same state, we are not allowed to access "live"
+ * memory anymore. To make it slightly easier to catch such cases, set
+ * the "live" pointers to NULL.
+ */
+ live_ioh = NULL;
+ owner_proc = NULL;
+
+
+ /* column: owning pid */
+ if (owner_pid != 0)
+ values[0] = Int32GetDatum(owner_pid);
+ else
+ nulls[0] = false;
+
+ /* column: IO's id */
+ values[1] = ioh_id;
+
+ /* column: IO's generation */
+ values[2] = Int64GetDatum(start_generation);
+
+ /* column: IO's state */
+ values[3] = CStringGetTextDatum(pgaio_io_get_state_name(&ioh_copy));
+
+ /*
+ * If the IO is in PGAIO_HS_HANDED_OUT state, none of it's fields are
+ * valid yet (or are in the process of being set). Therefore we don't
+ * want to display any other columns.
+ */
+ if (start_state == PGAIO_HS_HANDED_OUT)
+ {
+ memset(nulls + 4, 1, (lengthof(nulls) - 4) * sizeof(bool));
+ goto display;
+ }
+
+ /* column: IO's operation */
+ values[4] = CStringGetTextDatum(pgaio_io_get_op_name(&ioh_copy));
+
+ /* columns: details about the IO's operation */
+ switch (ioh_copy.op)
+ {
+ case PGAIO_OP_INVALID:
+ nulls[5] = true;
+ nulls[6] = true;
+ break;
+ case PGAIO_OP_READV:
+ values[5] = Int64GetDatum(ioh_copy.op_data.read.offset);
+ values[6] =
+ Int64GetDatum(iov_byte_length(iov_copy, ioh_copy.op_data.read.iov_length));
+ break;
+ case PGAIO_OP_WRITEV:
+ values[5] = Int64GetDatum(ioh_copy.op_data.write.offset);
+ values[6] =
+ Int64GetDatum(iov_byte_length(iov_copy, ioh_copy.op_data.write.iov_length));
+ break;
+ }
+
+ /* column: IO's target */
+ values[7] = CStringGetTextDatum(pgaio_io_get_target_name(&ioh_copy));
+
+ /* column: length of IO's data array */
+ values[8] = Int16GetDatum(ioh_copy.handle_data_len);
+
+ /* column: raw result (i.e. some form of syscall return value) */
+ if (start_state == PGAIO_HS_COMPLETED_IO
+ || start_state == PGAIO_HS_COMPLETED_SHARED)
+ values[9] = Int32GetDatum(ioh_copy.result);
+ else
+ nulls[9] = true;
+
+ /*
+ * column: result in the higher level representation (unknown if not
+ * finished)
+ */
+ values[10] =
+ CStringGetTextDatum(pgaio_result_status_string(ioh_copy.distilled_result.status));
+
+ /* column: target description */
+ values[11] = CStringGetTextDatum(pgaio_io_get_target_description(&ioh_copy));
+
+ /* columns: one for each flag */
+ values[12] = BoolGetDatum(ioh_copy.flags & PGAIO_HF_SYNCHRONOUS);
+ values[13] = BoolGetDatum(ioh_copy.flags & PGAIO_HF_REFERENCES_LOCAL);
+ values[14] = BoolGetDatum(ioh_copy.flags & PGAIO_HF_BUFFERED);
+
+display:
+
+ tuplestore_putvalues(rsinfo->setResult, rsinfo->setDesc, values, nulls);
+ }
+
+ return (Datum) 0;
+}
diff --git a/src/backend/storage/aio/meson.build b/src/backend/storage/aio/meson.build
index 2f0f03d8071..da6df2d3654 100644
--- a/src/backend/storage/aio/meson.build
+++ b/src/backend/storage/aio/meson.build
@@ -3,6 +3,7 @@
backend_sources += files(
'aio.c',
'aio_callback.c',
+ 'aio_funcs.c',
'aio_init.c',
'aio_io.c',
'aio_target.c',
diff --git a/src/test/regress/expected/rules.out b/src/test/regress/expected/rules.out
index 62f69ac20b2..6570d0cddce 100644
--- a/src/test/regress/expected/rules.out
+++ b/src/test/regress/expected/rules.out
@@ -1286,6 +1286,22 @@ drop table cchild;
SELECT viewname, definition FROM pg_views
WHERE schemaname = 'pg_catalog'
ORDER BY viewname;
+pg_aios| SELECT pid,
+ io_id,
+ io_generation,
+ state,
+ operation,
+ "offset",
+ length,
+ target,
+ handle_data_len,
+ raw_result,
+ result,
+ target_desc,
+ f_sync,
+ f_localmem,
+ f_buffered
+ FROM pg_get_aios() pg_get_aios(pid, io_id, io_generation, state, operation, "offset", length, target, handle_data_len, raw_result, result, target_desc, f_sync, f_localmem, f_buffered);
pg_available_extension_versions| SELECT e.name,
e.version,
(x.extname IS NOT NULL) AS installed,
--
2.48.1.76.g4e746b1a31.dirty
[text/x-diff] v2.11-0017-aio-Add-test_aio-module.patch (45.0K, ../../6ak556uyqiptdwjaci4kbi5eykwkmzqgkbtkyaosjnopjhncrc@2v4ac2jwyz22/18-v2.11-0017-aio-Add-test_aio-module.patch)
download | inline diff:
From 7ec622ba4ff50a7b9c24ed0158376b920076259a Mon Sep 17 00:00:00 2001
From: Andres Freund <andres@anarazel.de>
Date: Tue, 18 Mar 2025 14:40:06 -0400
Subject: [PATCH v2.11 17/27] aio: Add test_aio module
---
src/include/storage/buf_internals.h | 6 +
src/backend/storage/buffer/bufmgr.c | 8 +-
src/test/modules/Makefile | 1 +
src/test/modules/meson.build | 1 +
src/test/modules/test_aio/.gitignore | 2 +
src/test/modules/test_aio/Makefile | 27 +
src/test/modules/test_aio/meson.build | 37 ++
src/test/modules/test_aio/t/001_aio.pl | 699 ++++++++++++++++++++
src/test/modules/test_aio/test_aio--1.0.sql | 97 +++
src/test/modules/test_aio/test_aio.c | 657 ++++++++++++++++++
src/test/modules/test_aio/test_aio.control | 3 +
11 files changed, 1532 insertions(+), 6 deletions(-)
create mode 100644 src/test/modules/test_aio/.gitignore
create mode 100644 src/test/modules/test_aio/Makefile
create mode 100644 src/test/modules/test_aio/meson.build
create mode 100644 src/test/modules/test_aio/t/001_aio.pl
create mode 100644 src/test/modules/test_aio/test_aio--1.0.sql
create mode 100644 src/test/modules/test_aio/test_aio.c
create mode 100644 src/test/modules/test_aio/test_aio.control
diff --git a/src/include/storage/buf_internals.h b/src/include/storage/buf_internals.h
index 8a52ace37cd..4b1f52a9fc8 100644
--- a/src/include/storage/buf_internals.h
+++ b/src/include/storage/buf_internals.h
@@ -434,6 +434,12 @@ extern void IssuePendingWritebacks(WritebackContext *wb_context, IOContext io_co
extern void ScheduleBufferTagForWriteback(WritebackContext *wb_context,
IOContext io_context, BufferTag *tag);
+/* solely to make it easier to write tests */
+extern bool StartBufferIO(BufferDesc *buf, bool forInput, bool nowait);
+extern void TerminateBufferIO(BufferDesc *buf, bool clear_dirty, uint32 set_flag_bits,
+ bool forget_owner, bool syncio);
+
+
/* freelist.c */
extern IOContext IOContextForStrategy(BufferAccessStrategy strategy);
extern BufferDesc *StrategyGetBuffer(BufferAccessStrategy strategy,
diff --git a/src/backend/storage/buffer/bufmgr.c b/src/backend/storage/buffer/bufmgr.c
index 08d25e70bf2..be306bfa4f2 100644
--- a/src/backend/storage/buffer/bufmgr.c
+++ b/src/backend/storage/buffer/bufmgr.c
@@ -518,10 +518,6 @@ static uint32 WaitBufHdrUnlocked(BufferDesc *buf);
static int SyncOneBuffer(int buf_id, bool skip_recently_used,
WritebackContext *wb_context);
static void WaitIO(BufferDesc *buf);
-static bool StartBufferIO(BufferDesc *buf, bool forInput, bool nowait);
-static void TerminateBufferIO(BufferDesc *buf, bool clear_dirty,
- uint32 set_flag_bits, bool forget_owner,
- bool syncio);
static void AbortBufferIO(Buffer buffer);
static void shared_buffer_write_error_callback(void *arg);
static void local_buffer_write_error_callback(void *arg);
@@ -5944,7 +5940,7 @@ WaitIO(BufferDesc *buf)
* find out if they can perform the I/O as part of a larger operation, without
* waiting for the answer or distinguishing the reasons why not.
*/
-static bool
+bool
StartBufferIO(BufferDesc *buf, bool forInput, bool nowait)
{
uint32 buf_state;
@@ -6001,7 +5997,7 @@ StartBufferIO(BufferDesc *buf, bool forInput, bool nowait)
* resource owner. (forget_owner=false is used when the resource owner itself
* is being released)
*/
-static void
+void
TerminateBufferIO(BufferDesc *buf, bool clear_dirty, uint32 set_flag_bits,
bool forget_owner, bool syncio)
{
diff --git a/src/test/modules/Makefile b/src/test/modules/Makefile
index 4e4be3fa511..aa1d27bbed3 100644
--- a/src/test/modules/Makefile
+++ b/src/test/modules/Makefile
@@ -14,6 +14,7 @@ SUBDIRS = \
oauth_validator \
plsample \
spgist_name_ops \
+ test_aio \
test_bloomfilter \
test_copy_callbacks \
test_custom_rmgrs \
diff --git a/src/test/modules/meson.build b/src/test/modules/meson.build
index 2b057451473..4fda1acfb32 100644
--- a/src/test/modules/meson.build
+++ b/src/test/modules/meson.build
@@ -1,5 +1,6 @@
# Copyright (c) 2022-2025, PostgreSQL Global Development Group
+subdir('test_aio')
subdir('brin')
subdir('commit_ts')
subdir('delay_execution')
diff --git a/src/test/modules/test_aio/.gitignore b/src/test/modules/test_aio/.gitignore
new file mode 100644
index 00000000000..716e17f5a2a
--- /dev/null
+++ b/src/test/modules/test_aio/.gitignore
@@ -0,0 +1,2 @@
+# Generated subdirectories
+/tmp_check/
diff --git a/src/test/modules/test_aio/Makefile b/src/test/modules/test_aio/Makefile
new file mode 100644
index 00000000000..87d5315ba00
--- /dev/null
+++ b/src/test/modules/test_aio/Makefile
@@ -0,0 +1,27 @@
+# src/test/modules/delay_execution/Makefile
+
+PGFILEDESC = "test_aio - test code for AIO"
+
+MODULE_big = test_aio
+OBJS = \
+ $(WIN32RES) \
+ test_aio.o
+
+EXTENSION = test_aio
+DATA = test_aio--1.0.sql
+
+TAP_TESTS = 1
+
+export enable_injection_points
+export with_liburing
+
+ifdef USE_PGXS
+PG_CONFIG = pg_config
+PGXS := $(shell $(PG_CONFIG) --pgxs)
+include $(PGXS)
+else
+subdir = src/test/modules/test_aio
+top_builddir = ../../../..
+include $(top_builddir)/src/Makefile.global
+include $(top_srcdir)/contrib/contrib-global.mk
+endif
diff --git a/src/test/modules/test_aio/meson.build b/src/test/modules/test_aio/meson.build
new file mode 100644
index 00000000000..ac846b2b6f3
--- /dev/null
+++ b/src/test/modules/test_aio/meson.build
@@ -0,0 +1,37 @@
+# Copyright (c) 2022-2024, PostgreSQL Global Development Group
+
+test_aio_sources = files(
+ 'test_aio.c',
+)
+
+if host_system == 'windows'
+ test_aio_sources += rc_lib_gen.process(win32ver_rc, extra_args: [
+ '--NAME', 'test_aio',
+ '--FILEDESC', 'test_aio - test code for AIO',])
+endif
+
+test_aio = shared_module('test_aio',
+ test_aio_sources,
+ kwargs: pg_test_mod_args,
+)
+test_install_libs += test_aio
+
+test_install_data += files(
+ 'test_aio.control',
+ 'test_aio--1.0.sql',
+)
+
+tests += {
+ 'name': 'test_aio',
+ 'sd': meson.current_source_dir(),
+ 'bd': meson.current_build_dir(),
+ 'tap': {
+ 'env': {
+ 'enable_injection_points': get_option('injection_points') ? 'yes' : 'no',
+ 'with_liburing': liburing.found() ? 'yes' : 'no',
+ },
+ 'tests': [
+ 't/001_aio.pl',
+ ],
+ },
+}
diff --git a/src/test/modules/test_aio/t/001_aio.pl b/src/test/modules/test_aio/t/001_aio.pl
new file mode 100644
index 00000000000..444cabf2954
--- /dev/null
+++ b/src/test/modules/test_aio/t/001_aio.pl
@@ -0,0 +1,699 @@
+# Copyright (c) 2025, PostgreSQL Global Development Group
+
+use strict;
+use warnings FATAL => 'all';
+
+use PostgreSQL::Test::Cluster;
+use PostgreSQL::Test::Utils;
+use Test::More;
+
+
+###
+# Test io_method=worker
+###
+my $node_worker = create_node('worker');
+$node_worker->start();
+
+test_generic('worker', $node_worker);
+SKIP:
+{
+ skip 'Injection points not supported by this build', 1
+ unless $ENV{enable_injection_points} eq 'yes';
+ test_inject_worker('worker', $node_worker);
+}
+
+$node_worker->stop();
+
+
+###
+# Test io_method=io_uring
+###
+
+if ($ENV{with_liburing} eq 'yes')
+{
+ my $node_uring = create_node('io_uring');
+ $node_uring->start();
+ test_generic('io_uring', $node_uring);
+ $node_uring->stop();
+}
+
+
+###
+# Test io_method=sync
+###
+
+my $node_sync = create_node('sync');
+
+# just to have one test not use the default auto-tuning
+
+$node_sync->append_conf(
+ 'postgresql.conf', qq(
+io_max_concurrency=4
+));
+
+$node_sync->start();
+test_generic('sync', $node_sync);
+$node_sync->stop();
+
+done_testing();
+
+
+###
+# Test Helpers
+###
+
+sub create_node
+{
+ local $Test::Builder::Level = $Test::Builder::Level + 1;
+
+ my $io_method = shift;
+
+ my $node = PostgreSQL::Test::Cluster->new($io_method);
+
+ # Want to test initdb for each IO method, otherwise we could just reuse
+ # the cluster.
+ #
+ # Unfortunately Cluster::init() puts PG_TEST_INITDB_EXTRA_OPTS after the
+ # options specified by ->extra, if somebody puts -c io_method=xyz in
+ # PG_TEST_INITDB_EXTRA_OPTS it would break this test. Fix that up if we
+ # detect it.
+ local $ENV{PG_TEST_INITDB_EXTRA_OPTS} = $ENV{PG_TEST_INITDB_EXTRA_OPTS};
+ if (defined $ENV{PG_TEST_INITDB_EXTRA_OPTS} &&
+ $ENV{PG_TEST_INITDB_EXTRA_OPTS} =~ m/io_method=/)
+ {
+ $ENV{PG_TEST_INITDB_EXTRA_OPTS} .= " -c io_method=$io_method";
+ }
+
+ $node->init(extra => [ '-c', "io_method=$io_method" ]);
+
+ $node->append_conf(
+ 'postgresql.conf', qq(
+shared_preload_libraries=test_aio
+log_min_messages = 'DEBUG3'
+log_statement=all
+restart_after_crash=false
+));
+
+ ok(1, "$io_method: initdb");
+
+ return $node;
+}
+
+sub psql_like
+{
+ my $io_method = shift;
+ my $psql = shift;
+ my $name = shift;
+ my $sql = shift;
+ my $expected_stdout = shift;
+ my $expected_stderr = shift;
+ my ($cmdret, $output);
+
+ ($output, $cmdret) = $psql->query($sql);
+
+ like($output, $expected_stdout, "$io_method: $name: expected stdout");
+ like($psql->{stderr}, $expected_stderr,
+ "$io_method: $name: expected stderr");
+ $psql->{stderr} = '';
+}
+
+sub query_wait_block
+{
+ my $io_method = shift;
+ my $node = shift;
+ my $psql = shift;
+ my $name = shift;
+ my $sql = shift;
+ my $waitfor = shift;
+
+ my $pid = $psql->query_safe('SELECT pg_backend_pid()');
+
+ $psql->{stdin} .= qq($sql;\n);
+ $psql->{run}->pump_nb();
+ ok(1, "$io_method: $name: issued sql");
+
+ $node->poll_query_until('postgres',
+ qq(SELECT wait_event FROM pg_stat_activity WHERE pid = $pid),
+ $waitfor,);
+ ok(1, "$io_method: $name: observed $waitfor wait event");
+}
+
+
+###
+# Sub-tests
+###
+
+sub test_handle
+{
+ my $io_method = shift;
+ my $node = shift;
+
+ my $psql = $node->background_psql('postgres', on_error_stop => 0);
+
+ # leak warning: implicit xact
+ psql_like(
+ $io_method,
+ $psql,
+ "handle_get() leak in implicit xact",
+ qq(SELECT handle_get()),
+ qr/^$/,
+ qr/leaked AIO handle/,
+ "$io_method: leaky handle_get() warns");
+
+ # leak warning: explicit xact
+ psql_like(
+ $io_method, $psql,
+ "handle_get() leak in explicit xact",
+ qq(BEGIN; SELECT handle_get(); COMMIT),
+ qr/^$/, qr/leaked AIO handle/);
+
+
+ # leak warning: explicit xact, rollback
+ psql_like(
+ $io_method,
+ $psql,
+ "handle_get() leak in explicit xact, rollback",
+ qq(BEGIN; SELECT handle_get(); ROLLBACK;),
+ qr/^$/,
+ qr/leaked AIO handle/);
+
+ # leak warning: subtrans
+ psql_like(
+ $io_method,
+ $psql,
+ "handle_get() leak in subxact",
+ qq(BEGIN; SAVEPOINT foo; SELECT handle_get(); COMMIT;),
+ qr/^$/,
+ qr/leaked AIO handle/);
+
+ # leak warning + error: released in different command (thus resowner)
+ psql_like(
+ $io_method,
+ $psql,
+ "handle_release() in different command",
+ qq(BEGIN; SELECT handle_get(); SELECT handle_release_last(); COMMIT;),
+ qr/^$/,
+ qr/leaked AIO handle.*release in unexpected state/ms);
+
+ # no leak, release in same command
+ psql_like(
+ $io_method,
+ $psql,
+ "handle_release() in same command",
+ qq(BEGIN; SELECT handle_get() UNION ALL SELECT handle_release_last(); COMMIT;),
+ qr/^$/,
+ qr/^$/);
+
+ # normal handle use
+ psql_like($io_method, $psql, "handle_get_release()",
+ qq(SELECT handle_get_release()),
+ qr/^$/, qr/^$/);
+
+ # should error out, API violation
+ psql_like($io_method, $psql, "handle_get_twice()",
+ qq(SELECT handle_get_release()),
+ qr/^$/, qr/^$/);
+
+ # recover after error in implicit xact
+ psql_like(
+ $io_method,
+ $psql,
+ "handle error recovery in implicit xact",
+ qq(SELECT handle_get_and_error(); SELECT 'ok', handle_get_release()),
+ qr/^|ok$/,
+ qr/ERROR.*as you command/);
+
+ # recover after error in implicit xact
+ psql_like(
+ $io_method,
+ $psql,
+ "handle error recovery in explicit xact",
+ qq(BEGIN; SELECT handle_get_and_error(); SELECT handle_get_release(), 'ok'; COMMIT;),
+ qr/^|ok$/,
+ qr/ERROR.*as you command/);
+
+ # recover after error in subtrans
+ psql_like(
+ $io_method,
+ $psql,
+ "handle error recovery in explicit subxact",
+ qq(BEGIN; SAVEPOINT foo; SELECT handle_get_and_error(); ROLLBACK TO SAVEPOINT foo; SELECT handle_get_release(); ROLLBACK;),
+ qr/^|ok$/,
+ qr/ERROR.*as you command/);
+
+ $psql->quit();
+}
+
+
+sub test_batch
+{
+ my $io_method = shift;
+ my $node = shift;
+
+ my $psql = $node->background_psql('postgres', on_error_stop => 0);
+
+ # leak warning & recovery: implicit xact
+ psql_like(
+ $io_method,
+ $psql,
+ "batch_start() leak & cleanup in implicit xact",
+ qq(SELECT batch_start()),
+ qr/^$/,
+ qr/open AIO batch at end/,
+ "$io_method: leaky batch_start() warns");
+
+ # leak warning & recovery: explicit xact
+ psql_like(
+ $io_method,
+ $psql,
+ "batch_start() leak & cleanup in explicit xact",
+ qq(BEGIN; SELECT batch_start(); COMMIT;),
+ qr/^$/,
+ qr/open AIO batch at end/,
+ "$io_method: leaky batch_start() warns");
+
+
+ # leak warning & recovery: explicit xact, rollback
+ #
+ # XXX: This doesn't fail right now, due to not getting a chance to do
+ # something at transaction command commit. That's not a correctness issue,
+ # it just means it's a bit harder to find buggy code.
+ #psql_like($io_method, $psql,
+ # "batch_start() leak & cleanup after abort",
+ # qq(BEGIN; SELECT batch_start(); ROLLBACK;),
+ # qr/^$/,
+ # qr/open AIO batch at end/, "$io_method: leaky batch_start() warns");
+
+ # no warning, batch closed in same command
+ psql_like(
+ $io_method,
+ $psql,
+ "batch_start(), batch_end() works",
+ qq(SELECT batch_start() UNION ALL SELECT batch_end()),
+ qr/^$/,
+ qr/^$/,
+ "$io_method: batch_start(), batch_end()");
+
+ $psql->quit();
+}
+
+sub test_io_error
+{
+ my $io_method = shift;
+ my $node = shift;
+ my ($ret, $output);
+
+ my $psql = $node->background_psql('postgres', on_error_stop => 0);
+
+ # verify the error is reported in custom C code
+ ($output, $ret) = $psql->query(
+ qq(SELECT read_rel_block_ll('tbl_corr', 1, wait_complete=>true);));
+ is($ret, 1, "$io_method: read_rel_block_ll() of tbl_corr page fails");
+ like(
+ $psql->{stderr},
+ qr/invalid page in block 1 of relation base\/.*/,
+ "$io_method: read_rel_block_ll() of tbl_corr page reports error");
+ $psql->{stderr} = '';
+
+ # verify the error is reported for bufmgr reads
+ ($output, $ret) =
+ $psql->query(qq(SELECT count(*) FROM tbl_corr WHERE ctid = '(1, 1)'));
+ is($ret, 1, "$io_method: tid scan reading tbl_corr block fails");
+ like(
+ $psql->{stderr},
+ qr/invalid page in block 1 of relation base\/.*/,
+ "$io_method: tid scan reading tbl_corr block reports error");
+ $psql->{stderr} = '';
+
+ # verify the error is reported for bufmgr reads
+ ($output, $ret) =
+ $psql->query(qq(SELECT count(*) FROM tbl_corr WHERE ctid = '(1, 1)'));
+ is($ret, 1, "$io_method: sequential scan reading tbl_corr block fails");
+ like(
+ $psql->{stderr},
+ qr/invalid page in block 1 of relation base\/.*/,
+ "$io_method: sequential scan reading tbl_corr block reports error");
+ $psql->{stderr} = '';
+
+ $psql->quit();
+}
+
+# Test interplay between StartBufferIO and TerminateBufferIO
+sub test_startwait_io
+{
+ my $io_method = shift;
+ my $node = shift;
+ my ($ret, $output);
+
+ my $psql_a = $node->background_psql('postgres', on_error_stop => 0);
+ my $psql_b = $node->background_psql('postgres', on_error_stop => 0);
+
+ # create a buffer we can play around with
+ ($output, $ret) =
+ $psql_a->query(qq(SELECT buffer_create_toy('tbl_ok', 1);));
+ is($ret, 0, "$io_method: create toy succeeds");
+ like($output, qr/^\d+$/, "$io_method: create toy returns numeric");
+ my $buf_id = $output;
+
+ # check that one backend can perform StartBufferIO
+ psql_like(
+ $io_method,
+ $psql_a,
+ "first StartBufferIO",
+ qq(SELECT buffer_call_start_io($buf_id, for_input=>true, nowait=>false);),
+ qr/^t$/,
+ qr/^$/);
+
+ # but not twice on the same buffer (non-waiting)
+ psql_like(
+ $io_method,
+ $psql_a,
+ "second StartBufferIO fails, same session",
+ qq(SELECT buffer_call_start_io($buf_id, for_input=>true, nowait=>true);),
+ qr/^f$/,
+ qr/^$/);
+ psql_like(
+ $io_method,
+ $psql_b,
+ "second StartBufferIO fails, other session",
+ qq(SELECT buffer_call_start_io($buf_id, for_input=>true, nowait=>true);),
+ qr/^f$/,
+ qr/^$/);
+
+ # start io in a different session, will block
+ query_wait_block(
+ $io_method,
+ $node,
+ $psql_b,
+ "blocking start buffer io",
+ qq(SELECT buffer_call_start_io($buf_id, for_input=>true, nowait=>false);),
+ "BufferIo");
+
+ # Terminate the IO, without marking it as success, this should trigger the
+ # waiting session to be able to start the io
+ ($output, $ret) = $psql_a->query(
+ qq(SELECT buffer_call_terminate_io($buf_id, for_input=>true, succeed=>false, io_error=>false, syncio=>true);)
+ );
+ is($ret, 0,
+ "$io_method: blocking start buffer io, terminating io, not valid");
+
+ # Because the IO was terminated, but not marked as valid, second session should get the right to start io
+ pump_until($psql_b->{run}, $psql_b->{timeout}, \$psql_b->{stdout}, qr/t/);
+ ok(1, "$io_method: blocking start buffer io, can start io");
+
+ # terminate the IO again
+ $psql_b->query_safe(
+ qq(SELECT buffer_call_terminate_io($buf_id, for_input=>true, succeed=>false, io_error=>false, syncio=>true);)
+ );
+
+
+ # same as the above scenario, but mark IO as having succeeded
+ ($output, $ret) =
+ $psql_a->query(qq(SELECT buffer_call_start_io($buf_id, true, false);));
+ is($ret, 0,
+ "$io_method: blocking buffer io w/ success: first start buffer io succeeds"
+ );
+ is($output, "t",
+ "$io_method: blocking buffer io w/ success: first start buffer io returns true"
+ );
+
+
+ # start io in a different session, will block
+ query_wait_block(
+ $io_method,
+ $node,
+ $psql_b,
+ "blocking start buffer io",
+ qq(SELECT buffer_call_start_io($buf_id, for_input=>true, nowait=>false);),
+ "BufferIo");
+
+ ($output, $ret) = $psql_a->query(
+ qq(SELECT buffer_call_terminate_io($buf_id, for_input=>true, succeed=>true, io_error=>false, syncio=>true);)
+ );
+ is($ret, 0,
+ "$io_method: blocking start buffer io, terminating IO, valid");
+
+ # Because the IO was terminated, and marked as valid, second session should complete but not need io
+ pump_until($psql_b->{run}, $psql_b->{timeout}, \$psql_b->{stdout}, qr/f/);
+ ok(1, "$io_method: blocking start buffer io, no need to start io");
+
+
+ # buffer is valid now, make it invalid again
+ $buf_id = $psql_a->query_safe(qq(SELECT buffer_create_toy('tbl_ok', 1);));
+
+ $psql_a->quit();
+ $psql_b->quit();
+}
+
+# Test that if the backend issuing a read doesn't wait for the IO's
+# completion, another backend can complete the IO
+sub test_complete_foreign
+{
+ my $io_method = shift;
+ my $node = shift;
+ my ($ret, $output);
+
+ my $psql_a = $node->background_psql('postgres', on_error_stop => 0);
+ my $psql_b = $node->background_psql('postgres', on_error_stop => 0);
+
+ # Issue IO without waiting for completion, then sleep
+ $psql_a->query_safe(
+ qq(SELECT read_rel_block_ll('tbl_ok', 1, wait_complete=>false);));
+
+ # Check that another backend can read the relevant block
+ psql_like(
+ $io_method,
+ $psql_b,
+ "completing read started by sleeping backend",
+ qq(SELECT count(*) FROM tbl_ok WHERE ctid = '(1,1)' LIMIT 1),
+ qr/^1$/,
+ qr/^$/);
+
+ # Issue IO without waiting for completion, then exit
+ $psql_a->query_safe(
+ qq(SELECT read_rel_block_ll('tbl_ok', 1, wait_complete=>false);));
+ $psql_a->reconnect_and_clear();
+
+ # Check that another backend can read the relevant block
+ psql_like(
+ $io_method,
+ $psql_b,
+ "completing read started by exited backend",
+ qq(SELECT count(*) FROM tbl_ok WHERE ctid = '(1,1)' LIMIT 1),
+ qr/^1$/,
+ qr/^$/);
+
+ # Read a tbl_corr block, then sleep. The other session will retry the IO
+ # and also fail. The easiest thing to verify that seems to be to check
+ # that both are in the log.
+ my $log_location = -s $node->logfile;
+ $psql_a->query_safe(
+ qq(SELECT read_rel_block_ll('tbl_corr', 1, wait_complete=>false);));
+
+ psql_like(
+ $io_method,
+ $psql_b,
+ "completing read of tbl_corr block started by other backend",
+ qq(SELECT count(*) FROM tbl_corr WHERE ctid = '(1,1)' LIMIT 1),
+ qr/^$/,
+ qr/invalid page in block/);
+
+ # The log message issued for the read_rel_block_ll() should be logged as a LOG
+ $node->wait_for_log(qr/LOG[^\n]+invalid page in/, $log_location);
+ ok(1,
+ "$io_method: completing read of tbl_corr block started by other backend: LOG message for background read"
+ );
+
+ # But for the SELECT, it should be an ERROR
+ $log_location =
+ $node->wait_for_log(qr/ERROR[^\n]+invalid page in/, $log_location);
+ ok(1,
+ "$io_method: completing read of tbl_corr block started by other backend: ERROR message for foreground read"
+ );
+
+ $psql_a->quit();
+ $psql_b->quit();
+}
+
+sub test_inject
+{
+ my $io_method = shift;
+ my $node = shift;
+ my ($ret, $output);
+
+ my $psql = $node->background_psql('postgres', on_error_stop => 0);
+
+ # injected what we'd expect
+ $psql->query_safe(qq(SELECT inj_io_short_read_attach(8192);));
+ $psql->query_safe(qq(SELECT invalidate_rel_block('tbl_ok', 2);));
+ ($output, $ret) =
+ $psql->query(qq(SELECT count(*) FROM tbl_ok WHERE ctid = '(2, 1)';));
+ is($ret, 0,
+ "$io_method: injection point not triggering failure succeeds");
+
+ # injected a read shorter than a single block, expecting error
+ $psql->query_safe(qq(SELECT inj_io_short_read_attach(17);));
+ $psql->query_safe(qq(SELECT invalidate_rel_block('tbl_ok', 2);));
+ ($output, $ret) =
+ $psql->query(qq(SELECT count(*) FROM tbl_ok WHERE ctid = '(2, 1)';));
+ is($ret, 1, "$io_method: single block short read fails");
+ like(
+ $psql->{stderr},
+ qr/ERROR:.*could not read blocks 2\.\.2 in file "base\/.*": read only 0 of 8192 bytes/,
+ "$io_method: single block short read reports error");
+ $psql->{stderr} = '';
+
+ # shorten multi-block read to a single block, should retry
+ my $inval_query = qq(SELECT invalidate_rel_block('tbl_ok', 0);
+SELECT invalidate_rel_block('tbl_ok', 1);
+SELECT invalidate_rel_block('tbl_ok', 2);
+SELECT invalidate_rel_block('tbl_ok', 3);
+/* gap */
+SELECT invalidate_rel_block('tbl_ok', 5);
+SELECT invalidate_rel_block('tbl_ok', 6);
+SELECT invalidate_rel_block('tbl_ok', 7);
+SELECT invalidate_rel_block('tbl_ok', 8););
+
+ $psql->query_safe($inval_query);
+ $psql->query_safe(qq(SELECT inj_io_short_read_attach(8192);));
+ ($output, $ret) = $psql->query(qq(SELECT count(*) FROM tbl_ok;));
+ is($ret, 0, "$io_method: multi block short read (1 block) is retried");
+ is($output, 10000, "$io_method: multi block short read (1 block) has correct result");
+
+ # shorten multi-block read to two blocks, should retry
+ $psql->query_safe($inval_query);
+ $psql->query_safe(qq(SELECT inj_io_short_read_attach(8192*2);));
+
+ ($output, $ret) = $psql->query(qq(SELECT count(*) FROM tbl_ok;));
+ is($ret, 0, "$io_method: multi block short read (2 blocks) is retried");
+ is($output, 10000, "$io_method: multi block short read (2 blocks) has correct result");
+
+ # verify that page verification errors are detected even as part of a
+ # shortened multi-block read (tbl_corr, block 1 is tbl_corred)
+ $psql->query_safe(
+ qq(
+SELECT invalidate_rel_block('tbl_corr', 0);
+SELECT invalidate_rel_block('tbl_corr', 1);
+SELECT invalidate_rel_block('tbl_corr', 2);
+SELECT inj_io_short_read_attach(8192);
+ ));
+ ($output, $ret) =
+ $psql->query(qq(SELECT count(*) FROM tbl_corr WHERE ctid < '(2, 1)'));
+ is($ret, 1,
+ "$io_method: shortened multi-block read detects invalid page");
+ like(
+ $psql->{stderr},
+ qr/ERROR:.*invalid page in block 1 of relation base\/.*/,
+ "$io_method: shortened multi-block reads reports invalid page");
+ $psql->{stderr} = '';
+
+ # trigger a hard error, should error out
+ $psql->query_safe(
+ qq(
+SELECT inj_io_short_read_attach(-errno_from_string('EIO'));
+SELECT invalidate_rel_block('tbl_ok', 2);
+ ));
+ ($output, $ret) =
+ $psql->query(qq(SELECT count(*) FROM tbl_ok;));
+ is($ret, 1, "$io_method: first hard IO error is detected");
+ like(
+ $psql->{stderr},
+ qr/ERROR:.*could not read blocks 2\.\.2 in file \"base\/.*\": Input\/output error/,
+ "$io_method: first hard IO error is reported");
+ $psql->{stderr} = '';
+
+ # trigger a second hard error, should error out again
+ ($output, $ret) =
+ $psql->query(qq(SELECT count(*) FROM tbl_ok;));
+ is($ret, 1, "$io_method: second hard IO error is detected");
+ like(
+ $psql->{stderr},
+ qr/ERROR:.*could not read blocks 2\.\.2 in file \"base\/.*\": Input\/output error/,
+ "$io_method: second hard IO error is reported");
+ $psql->{stderr} = '';
+
+ $psql->query_safe(
+ qq(
+SELECT inj_io_short_read_detach();
+ ));
+
+ # now the IO should be ok.
+ ($output, $ret) =
+ $psql->query(qq(SELECT count(*) FROM tbl_ok;));
+ is($ret, 0, "$io_method: recovers after hard error");
+ is($output, 10000, "$io_method: recovers after hard error, query result ok");
+
+ $psql->quit();
+}
+
+sub test_inject_worker
+{
+ my $io_method = shift;
+ my $node = shift;
+ my ($ret, $output);
+
+ my $psql = $node->background_psql('postgres', on_error_stop => 0);
+
+ # trigger a failure to reopen, should error out, but should recover
+ $psql->query_safe(
+ qq(
+SELECT inj_io_reopen_attach();
+SELECT invalidate_rel_block('tbl_ok', 1);
+ ));
+ ($output, $ret) = $psql->query(qq(SELECT count(*) FROM tbl_ok;));
+ is($ret, 1, "$io_method: failure to open: detected");
+ like(
+ $psql->{stderr},
+ qr/ERROR:.*could not read blocks 1\.\.1 in file "base\/.*": No such file or directory/,
+ "$io_method: failure to open is reported");
+ $psql->{stderr} = '';
+
+ $psql->query_safe(
+ qq(
+SELECT inj_io_reopen_detach();
+ ));
+
+ # check that we indeed recover
+ ($output, $ret) = $psql->query(qq(SELECT count(*) FROM tbl_ok;));
+ is($ret, 0, "$io_method: failure to open: recovers");
+ is($output, 10000, "$io_method: failure to open: next query ok");
+
+ $psql->quit();
+}
+
+sub test_generic
+{
+ my $io_method = shift;
+ my $node = shift;
+
+ is($node->safe_psql('postgres', 'SHOW io_method'),
+ $io_method, "$io_method: io_method set correctly");
+
+ $node->safe_psql(
+ 'postgres', qq(
+CREATE EXTENSION test_aio;
+CREATE TABLE tbl_corr(data int not null) WITH (AUTOVACUUM_ENABLED = false);
+CREATE TABLE tbl_ok(data int not null) WITH (AUTOVACUUM_ENABLED = false);
+
+INSERT INTO tbl_corr SELECT generate_series(1, 10000);
+INSERT INTO tbl_ok SELECT generate_series(1, 10000);
+SELECT grow_rel('tbl_corr', 16);
+SELECT grow_rel('tbl_ok', 16);
+
+SELECT corrupt_rel_block('tbl_corr', 1);
+CHECKPOINT;
+));
+
+ test_handle($io_method, $node);
+ test_io_error($io_method, $node);
+ test_batch($io_method, $node);
+ test_startwait_io($io_method, $node);
+ test_complete_foreign($io_method, $node);
+
+ SKIP:
+ {
+ skip 'Injection points not supported by this build', 1
+ unless $ENV{enable_injection_points} eq 'yes';
+ test_inject($io_method, $node);
+ }
+}
diff --git a/src/test/modules/test_aio/test_aio--1.0.sql b/src/test/modules/test_aio/test_aio--1.0.sql
new file mode 100644
index 00000000000..3d4e9bb070f
--- /dev/null
+++ b/src/test/modules/test_aio/test_aio--1.0.sql
@@ -0,0 +1,97 @@
+/* src/test/modules/test_aio/test_aio--1.0.sql */
+
+-- complain if script is sourced in psql, rather than via CREATE EXTENSION
+\echo Use "CREATE EXTENSION test_aio" to load this file. \quit
+
+
+CREATE FUNCTION errno_from_string(sym text)
+RETURNS pg_catalog.int4 STRICT
+AS 'MODULE_PATHNAME' LANGUAGE C;
+
+
+CREATE FUNCTION grow_rel(rel regclass, nblocks int)
+RETURNS pg_catalog.void STRICT
+AS 'MODULE_PATHNAME' LANGUAGE C;
+
+
+CREATE FUNCTION corrupt_rel_block(rel regclass, blockno int)
+RETURNS pg_catalog.void STRICT
+AS 'MODULE_PATHNAME' LANGUAGE C;
+
+CREATE FUNCTION read_rel_block_ll(rel regclass, blockno int, wait_complete bool)
+RETURNS pg_catalog.void STRICT
+AS 'MODULE_PATHNAME' LANGUAGE C;
+
+CREATE FUNCTION invalidate_rel_block(rel regclass, blockno int)
+RETURNS pg_catalog.void STRICT
+AS 'MODULE_PATHNAME' LANGUAGE C;
+
+CREATE FUNCTION buffer_create_toy(rel regclass, blockno int4)
+RETURNS pg_catalog.int4 STRICT
+AS 'MODULE_PATHNAME' LANGUAGE C;
+
+CREATE FUNCTION buffer_call_start_io(buffer int, for_input bool, nowait bool)
+RETURNS pg_catalog.bool STRICT
+AS 'MODULE_PATHNAME' LANGUAGE C;
+
+CREATE FUNCTION buffer_call_terminate_io(buffer int, for_input bool, succeed bool, io_error bool, syncio bool)
+RETURNS pg_catalog.void STRICT
+AS 'MODULE_PATHNAME' LANGUAGE C;
+
+
+
+/*
+ * Handle related functions
+ */
+CREATE FUNCTION handle_get_and_error()
+RETURNS pg_catalog.void STRICT
+AS 'MODULE_PATHNAME' LANGUAGE C;
+
+CREATE FUNCTION handle_get_twice()
+RETURNS pg_catalog.void STRICT
+AS 'MODULE_PATHNAME' LANGUAGE C;
+
+CREATE FUNCTION handle_get()
+RETURNS pg_catalog.void STRICT
+AS 'MODULE_PATHNAME' LANGUAGE C;
+
+CREATE FUNCTION handle_get_release()
+RETURNS pg_catalog.void STRICT
+AS 'MODULE_PATHNAME' LANGUAGE C;
+
+CREATE FUNCTION handle_release_last()
+RETURNS pg_catalog.void STRICT
+AS 'MODULE_PATHNAME' LANGUAGE C;
+
+
+/*
+ * Batchmode related functions
+ */
+CREATE FUNCTION batch_start()
+RETURNS pg_catalog.void STRICT
+AS 'MODULE_PATHNAME' LANGUAGE C;
+
+CREATE FUNCTION batch_end()
+RETURNS pg_catalog.void STRICT
+AS 'MODULE_PATHNAME' LANGUAGE C;
+
+
+
+/*
+ * Injection point related functions
+ */
+CREATE FUNCTION inj_io_short_read_attach(result int)
+RETURNS pg_catalog.void STRICT
+AS 'MODULE_PATHNAME' LANGUAGE C;
+
+CREATE FUNCTION inj_io_short_read_detach()
+RETURNS pg_catalog.void STRICT
+AS 'MODULE_PATHNAME' LANGUAGE C;
+
+CREATE FUNCTION inj_io_reopen_attach()
+RETURNS pg_catalog.void STRICT
+AS 'MODULE_PATHNAME' LANGUAGE C;
+
+CREATE FUNCTION inj_io_reopen_detach()
+RETURNS pg_catalog.void STRICT
+AS 'MODULE_PATHNAME' LANGUAGE C;
diff --git a/src/test/modules/test_aio/test_aio.c b/src/test/modules/test_aio/test_aio.c
new file mode 100644
index 00000000000..fc55b985dad
--- /dev/null
+++ b/src/test/modules/test_aio/test_aio.c
@@ -0,0 +1,657 @@
+/*-------------------------------------------------------------------------
+ *
+ * delay_execution.c
+ * Test module to allow delay between parsing and execution of a query.
+ *
+ * The delay is implemented by taking and immediately releasing a specified
+ * advisory lock. If another process has previously taken that lock, the
+ * current process will be blocked until the lock is released; otherwise,
+ * there's no effect. This allows an isolationtester script to reliably
+ * test behaviors where some specified action happens in another backend
+ * between parsing and execution of any desired query.
+ *
+ * Copyright (c) 2020-2025, PostgreSQL Global Development Group
+ *
+ * IDENTIFICATION
+ * src/test/modules/delay_execution/delay_execution.c
+ *
+ *-------------------------------------------------------------------------
+ */
+
+#include "postgres.h"
+
+#include "access/relation.h"
+#include "fmgr.h"
+#include "storage/aio.h"
+#include "storage/aio_internal.h"
+#include "storage/buf_internals.h"
+#include "storage/bufmgr.h"
+#include "storage/ipc.h"
+#include "storage/lwlock.h"
+#include "utils/builtins.h"
+#include "utils/injection_point.h"
+#include "utils/rel.h"
+
+
+PG_MODULE_MAGIC;
+
+
+typedef struct InjIoErrorState
+{
+ bool enabled_short_read;
+ bool enabled_reopen;
+
+ bool short_read_result_set;
+ int short_read_result;
+} InjIoErrorState;
+
+static InjIoErrorState * inj_io_error_state;
+
+/* Shared memory init callbacks */
+static shmem_request_hook_type prev_shmem_request_hook = NULL;
+static shmem_startup_hook_type prev_shmem_startup_hook = NULL;
+
+
+static PgAioHandle *last_handle;
+
+
+
+static void
+test_aio_shmem_request(void)
+{
+ if (prev_shmem_request_hook)
+ prev_shmem_request_hook();
+
+ RequestAddinShmemSpace(sizeof(InjIoErrorState));
+}
+
+static void
+test_aio_shmem_startup(void)
+{
+ bool found;
+
+ if (prev_shmem_startup_hook)
+ prev_shmem_startup_hook();
+
+ /* Create or attach to the shared memory state */
+ LWLockAcquire(AddinShmemInitLock, LW_EXCLUSIVE);
+
+ inj_io_error_state = ShmemInitStruct("injection_points",
+ sizeof(InjIoErrorState),
+ &found);
+
+ if (!found)
+ {
+ /*
+ * First time through, so initialize. This is shared with the dynamic
+ * initialization using a DSM.
+ */
+ inj_io_error_state->enabled_short_read = false;
+ inj_io_error_state->enabled_reopen = false;
+
+#ifdef USE_INJECTION_POINTS
+ InjectionPointAttach("AIO_PROCESS_COMPLETION_BEFORE_SHARED",
+ "test_aio",
+ "inj_io_short_read",
+ NULL,
+ 0);
+ InjectionPointLoad("AIO_PROCESS_COMPLETION_BEFORE_SHARED");
+
+ InjectionPointAttach("AIO_WORKER_AFTER_REOPEN",
+ "test_aio",
+ "inj_io_reopen",
+ NULL,
+ 0);
+ InjectionPointLoad("AIO_WORKER_AFTER_REOPEN");
+
+#endif
+ }
+ else
+ {
+#ifdef USE_INJECTION_POINTS
+ InjectionPointLoad("AIO_PROCESS_COMPLETION_BEFORE_SHARED");
+ InjectionPointLoad("AIO_WORKER_AFTER_REOPEN");
+ elog(LOG, "injection point loaded");
+#endif
+ }
+
+ LWLockRelease(AddinShmemInitLock);
+}
+
+void
+_PG_init(void)
+{
+ if (!process_shared_preload_libraries_in_progress)
+ return;
+
+ /* Shared memory initialization */
+ prev_shmem_request_hook = shmem_request_hook;
+ shmem_request_hook = test_aio_shmem_request;
+ prev_shmem_startup_hook = shmem_startup_hook;
+ shmem_startup_hook = test_aio_shmem_startup;
+}
+
+
+PG_FUNCTION_INFO_V1(errno_from_string);
+Datum
+errno_from_string(PG_FUNCTION_ARGS)
+{
+ const char *sym = text_to_cstring(PG_GETARG_TEXT_PP(0));
+
+ if (strcmp(sym, "EIO") == 0)
+ PG_RETURN_INT32(EIO);
+ else if (strcmp(sym, "EAGAIN") == 0)
+ PG_RETURN_INT32(EAGAIN);
+ else if (strcmp(sym, "EINTR") == 0)
+ PG_RETURN_INT32(EINTR);
+ else if (strcmp(sym, "ENOSPC") == 0)
+ PG_RETURN_INT32(ENOSPC);
+ else if (strcmp(sym, "EROFS") == 0)
+ PG_RETURN_INT32(EROFS);
+
+ ereport(ERROR,
+ errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ errmsg_internal("%s is not a supported errno value", sym));
+ PG_RETURN_INT32(0);
+}
+
+
+PG_FUNCTION_INFO_V1(grow_rel);
+Datum
+grow_rel(PG_FUNCTION_ARGS)
+{
+ Oid relid = PG_GETARG_OID(0);
+ uint32 nblocks = PG_GETARG_UINT32(1);
+ Relation rel;
+#define MAX_BUFFERS_TO_EXTEND_BY 64
+ Buffer victim_buffers[MAX_BUFFERS_TO_EXTEND_BY];
+
+ rel = relation_open(relid, AccessExclusiveLock);
+
+ while (nblocks > 0)
+ {
+ uint32 extend_by_pages;
+
+ extend_by_pages = Min(nblocks, MAX_BUFFERS_TO_EXTEND_BY);
+
+ ExtendBufferedRelBy(BMR_REL(rel),
+ MAIN_FORKNUM,
+ NULL,
+ 0,
+ extend_by_pages,
+ victim_buffers,
+ &extend_by_pages);
+
+ nblocks -= extend_by_pages;
+
+ for (uint32 i = 0; i < extend_by_pages; i++)
+ {
+ ReleaseBuffer(victim_buffers[i]);
+ }
+ }
+
+ relation_close(rel, NoLock);
+
+ PG_RETURN_VOID();
+}
+
+PG_FUNCTION_INFO_V1(corrupt_rel_block);
+Datum
+corrupt_rel_block(PG_FUNCTION_ARGS)
+{
+ Oid relid = PG_GETARG_OID(0);
+ BlockNumber blkno = PG_GETARG_UINT32(1);
+ Relation rel;
+ Buffer buf;
+ Page page;
+ PageHeader ph;
+
+ rel = relation_open(relid, AccessExclusiveLock);
+
+ buf = ReadBuffer(rel, blkno);
+ page = BufferGetPage(buf);
+
+ LockBuffer(buf, BUFFER_LOCK_EXCLUSIVE);
+
+ MarkBufferDirty(buf);
+
+ PageInit(page, BufferGetPageSize(buf), 0);
+
+ ph = (PageHeader) page;
+ ph->pd_special = BLCKSZ + 1;
+
+ FlushOneBuffer(buf);
+
+ LockBuffer(buf, BUFFER_LOCK_UNLOCK);
+
+ ReleaseBuffer(buf);
+
+ EvictUnpinnedBuffer(buf);
+
+ relation_close(rel, NoLock);
+
+ PG_RETURN_VOID();
+}
+
+/*
+ * Ensures a buffer for rel & blkno is in shared buffers, without actually
+ * caring about the buffer contents. Used to set up test scenarios.
+ */
+static Buffer
+create_toy_buffer(Relation rel, BlockNumber blkno)
+{
+ Buffer buf;
+ BufferDesc *buf_hdr;
+ uint32 buf_state;
+ bool was_pinned = false;
+
+ /* place buffer in shared buffers without erroring out */
+ buf = ReadBufferExtended(rel, MAIN_FORKNUM, blkno, RBM_ZERO_AND_LOCK, NULL);
+ LockBuffer(buf, BUFFER_LOCK_UNLOCK);
+
+ buf_hdr = GetBufferDescriptor(buf - 1);
+
+ buf_state = LockBufHdr(buf_hdr);
+
+ /*
+ * We should be the only backend accessing this buffer. This is just a
+ * small bit of belt-and-suspenders defense, none of this code should ever
+ * run in a cluster with real data.
+ */
+ if (BUF_STATE_GET_REFCOUNT(buf_state) > 1)
+ was_pinned = true;
+ else
+ buf_state &= ~(BM_VALID | BM_DIRTY);
+
+ UnlockBufHdr(buf_hdr, buf_state);
+
+ if (was_pinned)
+ elog(ERROR, "toy buffer %d was already pinned",
+ buf);
+
+ return buf;
+}
+
+PG_FUNCTION_INFO_V1(read_rel_block_ll);
+Datum
+read_rel_block_ll(PG_FUNCTION_ARGS)
+{
+ Oid relid = PG_GETARG_OID(0);
+ BlockNumber blkno = PG_GETARG_UINT32(1);
+ bool wait_complete = PG_GETARG_BOOL(2);
+ Relation rel;
+ Buffer buf;
+ Page pages[1];
+ PgAioReturn ior;
+ PgAioHandle *ioh;
+ PgAioWaitRef iow;
+ SMgrRelation smgr;
+
+ rel = relation_open(relid, AccessExclusiveLock);
+
+ buf = create_toy_buffer(rel, blkno);
+
+ pages[0] = BufferGetBlock(buf);
+
+ ioh = pgaio_io_acquire(CurrentResourceOwner, &ior);
+ pgaio_io_get_wref(ioh, &iow);
+
+ smgr = RelationGetSmgr(rel);
+
+ StartBufferIO(GetBufferDescriptor(buf - 1), true, false);
+
+ pgaio_io_set_handle_data_32(ioh, (uint32 *) &buf, 1);
+ pgaio_io_register_callbacks(ioh, PGAIO_HCB_SHARED_BUFFER_READV, 0);
+
+ elog(LOG, "about to smgrstartreadv");
+ smgrstartreadv(ioh, smgr, MAIN_FORKNUM, blkno,
+ (void *) pages, 1);
+
+ ReleaseBuffer(buf);
+
+ if (wait_complete)
+ {
+ pgaio_wref_wait(&iow);
+
+ if (ior.result.status != PGAIO_RS_OK)
+ pgaio_result_report(ior.result, &ior.target_data,
+ ior.result.status == PGAIO_RS_PARTIAL ? WARNING : ERROR);
+ }
+
+ relation_close(rel, NoLock);
+
+ PG_RETURN_VOID();
+}
+
+PG_FUNCTION_INFO_V1(invalidate_rel_block);
+Datum
+invalidate_rel_block(PG_FUNCTION_ARGS)
+{
+ Oid relid = PG_GETARG_OID(0);
+ BlockNumber blkno = PG_GETARG_UINT32(1);
+ Relation rel;
+ PrefetchBufferResult pr;
+ Buffer buf;
+
+ rel = relation_open(relid, AccessExclusiveLock);
+
+ /*
+ * This is a gross hack, but there's no other API exposed that allows to
+ * get a buffer ID without actually reading the block in.
+ */
+ pr = PrefetchBuffer(rel, MAIN_FORKNUM, blkno);
+ buf = pr.recent_buffer;
+
+ if (BufferIsValid(buf))
+ {
+ /* if the buffer contents aren't valid, this'll return false */
+ if (ReadRecentBuffer(rel->rd_locator, MAIN_FORKNUM, blkno, buf))
+ {
+ LockBuffer(buf, BUFFER_LOCK_EXCLUSIVE);
+ FlushOneBuffer(buf);
+ LockBuffer(buf, BUFFER_LOCK_UNLOCK);
+ ReleaseBuffer(buf);
+
+ if (!EvictUnpinnedBuffer(buf))
+ elog(ERROR, "couldn't evict");
+ }
+ }
+
+ relation_close(rel, AccessExclusiveLock);
+
+ PG_RETURN_VOID();
+}
+
+PG_FUNCTION_INFO_V1(buffer_create_toy);
+Datum
+buffer_create_toy(PG_FUNCTION_ARGS)
+{
+ Oid relid = PG_GETARG_OID(0);
+ BlockNumber blkno = PG_GETARG_UINT32(1);
+ Relation rel;
+ Buffer buf;
+
+ rel = relation_open(relid, AccessExclusiveLock);
+
+ buf = create_toy_buffer(rel, blkno);
+ ReleaseBuffer(buf);
+
+ relation_close(rel, NoLock);
+
+ PG_RETURN_INT32(buf);
+}
+
+PG_FUNCTION_INFO_V1(buffer_call_start_io);
+Datum
+buffer_call_start_io(PG_FUNCTION_ARGS)
+{
+ Buffer buf = PG_GETARG_INT32(0);
+ bool for_input = PG_GETARG_BOOL(1);
+ bool nowait = PG_GETARG_BOOL(2);
+ bool can_start;
+
+ can_start = StartBufferIO(GetBufferDescriptor(buf - 1), for_input, nowait);
+
+ /*
+ * Tor tests we don't want the resowner release preventing us from
+ * orchestrating odd scenarios.
+ */
+ if (can_start)
+ ResourceOwnerForgetBufferIO(CurrentResourceOwner,
+ buf);
+
+ ereport(LOG,
+ errmsg("buffer %d after StartBufferIO: %s",
+ buf, DebugPrintBufferRefcount(buf)),
+ errhidestmt(true), errhidecontext(true));
+
+ PG_RETURN_BOOL(can_start);
+}
+
+/*
+CREATE FUNCTION buffer_call_terminate_io(buffer int, for_input bool, failed bool, syncio bool)
+RETURNS pg_catalog.void STRICT
+AS 'MODULE_PATHNAME' LANGUAGE C;
+*/
+PG_FUNCTION_INFO_V1(buffer_call_terminate_io);
+Datum
+buffer_call_terminate_io(PG_FUNCTION_ARGS)
+{
+ Buffer buf = PG_GETARG_INT32(0);
+ bool for_input = PG_GETARG_BOOL(1);
+ bool succeed = PG_GETARG_BOOL(2);
+ bool io_error = PG_GETARG_BOOL(3);
+ bool syncio = PG_GETARG_BOOL(4);
+ bool clear_dirty = false;
+ uint32 set_flag_bits = 0;
+
+ if (io_error)
+ set_flag_bits |= BM_IO_ERROR;
+
+ if (for_input)
+ {
+ clear_dirty = false;
+
+ if (succeed)
+ set_flag_bits |= BM_VALID;
+ }
+ else
+ {
+ if (succeed)
+ clear_dirty = true;
+ }
+
+ ereport(LOG,
+ errmsg("buffer %d before TerminateBufferIO: %s",
+ buf, DebugPrintBufferRefcount(buf)),
+ errhidestmt(true), errhidecontext(true));
+
+ TerminateBufferIO(GetBufferDescriptor(buf - 1), clear_dirty, set_flag_bits,
+ false, syncio);
+
+ ereport(LOG,
+ errmsg("buffer %d after TerminateBufferIO: %s",
+ buf, DebugPrintBufferRefcount(buf)),
+ errhidestmt(true), errhidecontext(true));
+
+ PG_RETURN_VOID();
+}
+
+PG_FUNCTION_INFO_V1(handle_get);
+Datum
+handle_get(PG_FUNCTION_ARGS)
+{
+ last_handle = pgaio_io_acquire(CurrentResourceOwner, NULL);
+
+ PG_RETURN_VOID();
+}
+
+PG_FUNCTION_INFO_V1(handle_release_last);
+Datum
+handle_release_last(PG_FUNCTION_ARGS)
+{
+ if (!last_handle)
+ elog(ERROR, "no handle");
+
+ pgaio_io_release(last_handle);
+
+ PG_RETURN_VOID();
+}
+
+PG_FUNCTION_INFO_V1(handle_get_and_error);
+Datum
+handle_get_and_error(PG_FUNCTION_ARGS)
+{
+ pgaio_io_acquire(CurrentResourceOwner, NULL);
+
+ elog(ERROR, "as you command");
+ PG_RETURN_VOID();
+}
+
+PG_FUNCTION_INFO_V1(handle_get_twice);
+Datum
+handle_get_twice(PG_FUNCTION_ARGS)
+{
+ pgaio_io_acquire(CurrentResourceOwner, NULL);
+ pgaio_io_acquire(CurrentResourceOwner, NULL);
+
+ PG_RETURN_VOID();
+}
+
+PG_FUNCTION_INFO_V1(handle_get_release);
+Datum
+handle_get_release(PG_FUNCTION_ARGS)
+{
+ PgAioHandle *handle;
+
+ handle = pgaio_io_acquire(CurrentResourceOwner, NULL);
+ pgaio_io_release(handle);
+
+ PG_RETURN_VOID();
+}
+
+PG_FUNCTION_INFO_V1(batch_start);
+Datum
+batch_start(PG_FUNCTION_ARGS)
+{
+ pgaio_enter_batchmode();
+ PG_RETURN_VOID();
+}
+
+PG_FUNCTION_INFO_V1(batch_end);
+Datum
+batch_end(PG_FUNCTION_ARGS)
+{
+ pgaio_exit_batchmode();
+ PG_RETURN_VOID();
+}
+
+#ifdef USE_INJECTION_POINTS
+extern PGDLLEXPORT void inj_io_short_read(const char *name, const void *private_data);
+extern PGDLLEXPORT void inj_io_reopen(const char *name, const void *private_data);
+
+void
+inj_io_short_read(const char *name, const void *private_data)
+{
+ PgAioHandle *ioh;
+
+ elog(LOG, "short read called: %d", inj_io_error_state->enabled_short_read);
+
+ if (inj_io_error_state->enabled_short_read)
+ {
+ ioh = pgaio_inj_io_get();
+
+ /*
+ * Only shorten reads that are actually longer than the target size,
+ * otherwise we can trigger over-reads.
+ */
+ if (inj_io_error_state->short_read_result_set
+ && ioh->op == PGAIO_OP_READV
+ && inj_io_error_state->short_read_result <= ioh->result)
+ {
+ struct iovec *iov = &pgaio_ctl->iovecs[ioh->iovec_off];
+ int32 old_result = ioh->result;
+ int32 new_result = inj_io_error_state->short_read_result;
+ int32 processed = 0;
+
+ ereport(LOG,
+ errmsg("short read, changing result from %d to %d",
+ old_result, new_result),
+ errhidestmt(true), errhidecontext(true));
+
+ /*
+ * The underlying IO actually completed OK, and thus the "invalid"
+ * portion of the IOV actually contains valid data. That can hide
+ * a lot of problems, e.g. if we were to wrongly mark a buffer,
+ * that wasn't read according to the shortened-read, IO as valid,
+ * the contents would look valid and we might miss a bug.
+ *
+ * To avoid that, iterate through the IOV and zero out the
+ * "failed" portion of the IO.
+ */
+ for (int i = 0; i < ioh->op_data.read.iov_length; i++)
+ {
+ if (processed + iov[i].iov_len <= new_result)
+ processed += iov[i].iov_len;
+ else if (processed <= new_result)
+ {
+ uint32 ok_part = new_result - processed;
+
+ memset((char *) iov[i].iov_base + ok_part, 0, iov[i].iov_len - ok_part);
+ processed += iov[i].iov_len;
+ }
+ else
+ {
+ memset((char *) iov[i].iov_base, 0, iov[i].iov_len);
+ }
+ }
+
+ ioh->result = new_result;
+ }
+ }
+}
+
+void
+inj_io_reopen(const char *name, const void *private_data)
+{
+ elog(LOG, "reopen called: %d", inj_io_error_state->enabled_reopen);
+
+ if (inj_io_error_state->enabled_reopen)
+ {
+ elog(ERROR, "injection point triggering failure to reopen ");
+ }
+}
+#endif
+
+PG_FUNCTION_INFO_V1(inj_io_short_read_attach);
+Datum
+inj_io_short_read_attach(PG_FUNCTION_ARGS)
+{
+#ifdef USE_INJECTION_POINTS
+ inj_io_error_state->enabled_short_read = true;
+ inj_io_error_state->short_read_result_set = !PG_ARGISNULL(0);
+ if (inj_io_error_state->short_read_result_set)
+ inj_io_error_state->short_read_result = PG_GETARG_INT32(0);
+#else
+ elog(ERROR, "injection points not supported");
+#endif
+
+ PG_RETURN_VOID();
+}
+
+PG_FUNCTION_INFO_V1(inj_io_short_read_detach);
+Datum
+inj_io_short_read_detach(PG_FUNCTION_ARGS)
+{
+#ifdef USE_INJECTION_POINTS
+ inj_io_error_state->enabled_short_read = false;
+#else
+ elog(ERROR, "injection points not supported");
+#endif
+ PG_RETURN_VOID();
+}
+
+PG_FUNCTION_INFO_V1(inj_io_reopen_attach);
+Datum
+inj_io_reopen_attach(PG_FUNCTION_ARGS)
+{
+#ifdef USE_INJECTION_POINTS
+ inj_io_error_state->enabled_reopen = true;
+#else
+ elog(ERROR, "injection points not supported");
+#endif
+
+ PG_RETURN_VOID();
+}
+
+PG_FUNCTION_INFO_V1(inj_io_reopen_detach);
+Datum
+inj_io_reopen_detach(PG_FUNCTION_ARGS)
+{
+#ifdef USE_INJECTION_POINTS
+ inj_io_error_state->enabled_reopen = false;
+#else
+ elog(ERROR, "injection points not supported");
+#endif
+ PG_RETURN_VOID();
+}
diff --git a/src/test/modules/test_aio/test_aio.control b/src/test/modules/test_aio/test_aio.control
new file mode 100644
index 00000000000..cd91c3ed16b
--- /dev/null
+++ b/src/test/modules/test_aio/test_aio.control
@@ -0,0 +1,3 @@
+comment = 'Test code for AIO'
+default_version = '1.0'
+module_pathname = '$libdir/test_aio'
--
2.48.1.76.g4e746b1a31.dirty
[text/x-diff] v2.11-0018-aio-Experimental-heuristics-to-increase-batchi.patch (1.6K, ../../6ak556uyqiptdwjaci4kbi5eykwkmzqgkbtkyaosjnopjhncrc@2v4ac2jwyz22/19-v2.11-0018-aio-Experimental-heuristics-to-increase-batchi.patch)
download | inline diff:
From 564255d2458963965d29894f12b6a37dc20a37e1 Mon Sep 17 00:00:00 2001
From: Andres Freund <andres@anarazel.de>
Date: Tue, 18 Mar 2025 14:40:06 -0400
Subject: [PATCH v2.11 18/27] aio: Experimental heuristics to increase batching
in read_stream.c
Author:
Reviewed-by:
Discussion: https://postgr.es/m/
Backpatch:
---
src/backend/storage/aio/read_stream.c | 24 ++++++++++++++++++++++++
1 file changed, 24 insertions(+)
diff --git a/src/backend/storage/aio/read_stream.c b/src/backend/storage/aio/read_stream.c
index 60a841816d8..facf996608e 100644
--- a/src/backend/storage/aio/read_stream.c
+++ b/src/backend/storage/aio/read_stream.c
@@ -418,6 +418,30 @@ read_stream_start_pending_read(ReadStream *stream)
static void
read_stream_look_ahead(ReadStream *stream)
{
+ /*
+ * Batch-submitting multiple IOs is more efficient than doing so
+ * one-by-one. If we just ramp up to the max, we'll only be allowed to
+ * submit one io_combine_limit sized IO. Defer submitting IO in that case.
+ *
+ * FIXME: This needs better heuristics.
+ */
+#if 1
+ if (!stream->sync_mode && stream->distance > (io_combine_limit * 8))
+ {
+ if (stream->pinned_buffers + stream->pending_read_nblocks > ((stream->distance * 3) / 4))
+ {
+#if 0
+ ereport(LOG,
+ errmsg("reduce reduce reduce: pinned: %d, pending: %d, distance: %d",
+ stream->pinned_buffers,
+ stream->pending_read_nblocks,
+ stream->distance));
+#endif
+ return;
+ }
+ }
+#endif
+
/*
* Allow amortizing the cost of submitting IO over multiple IOs. This
* requires that we don't do any operations that could lead to a deadlock
--
2.48.1.76.g4e746b1a31.dirty
[text/x-diff] v2.11-0019-aio-Implement-smgr-md-fd-write-support.patch (13.3K, ../../6ak556uyqiptdwjaci4kbi5eykwkmzqgkbtkyaosjnopjhncrc@2v4ac2jwyz22/20-v2.11-0019-aio-Implement-smgr-md-fd-write-support.patch)
download | inline diff:
From d9dc948108a99e8ee4052873f08b2ff10d9a0477 Mon Sep 17 00:00:00 2001
From: Andres Freund <andres@anarazel.de>
Date: Tue, 18 Mar 2025 14:40:06 -0400
Subject: [PATCH v2.11 19/27] aio: Implement smgr/md/fd write support
TODO:
- register_dirty_segment_aio() can error out in edge cases
---
src/include/storage/fd.h | 1 +
src/include/storage/md.h | 5 +
src/include/storage/smgr.h | 5 +
src/backend/storage/aio/aio_callback.c | 1 +
src/backend/storage/file/fd.c | 28 ++++
src/backend/storage/smgr/md.c | 199 +++++++++++++++++++++++++
src/backend/storage/smgr/smgr.c | 29 ++++
7 files changed, 268 insertions(+)
diff --git a/src/include/storage/fd.h b/src/include/storage/fd.h
index b77d8e5e30e..2cc7c5a4761 100644
--- a/src/include/storage/fd.h
+++ b/src/include/storage/fd.h
@@ -112,6 +112,7 @@ extern int FilePrefetch(File file, off_t offset, off_t amount, uint32 wait_event
extern ssize_t FileReadV(File file, const struct iovec *iov, int iovcnt, off_t offset, uint32 wait_event_info);
extern ssize_t FileWriteV(File file, const struct iovec *iov, int iovcnt, off_t offset, uint32 wait_event_info);
extern int FileStartReadV(struct PgAioHandle *ioh, File file, int iovcnt, off_t offset, uint32 wait_event_info);
+extern int FileStartWriteV(struct PgAioHandle *ioh, File file, int iovcnt, off_t offset, uint32 wait_event_info);
extern int FileSync(File file, uint32 wait_event_info);
extern int FileZero(File file, off_t offset, off_t amount, uint32 wait_event_info);
extern int FileFallocate(File file, off_t offset, off_t amount, uint32 wait_event_info);
diff --git a/src/include/storage/md.h b/src/include/storage/md.h
index 9d7131eff43..47ae6c36c94 100644
--- a/src/include/storage/md.h
+++ b/src/include/storage/md.h
@@ -21,6 +21,7 @@
#include "storage/sync.h"
extern const PgAioHandleCallbacks aio_md_readv_cb;
+extern const PgAioHandleCallbacks aio_md_writev_cb;
/* md storage manager functionality */
extern void mdinit(void);
@@ -45,6 +46,10 @@ extern void mdstartreadv(PgAioHandle *ioh,
extern void mdwritev(SMgrRelation reln, ForkNumber forknum,
BlockNumber blocknum,
const void **buffers, BlockNumber nblocks, bool skipFsync);
+extern void mdstartwritev(PgAioHandle *ioh,
+ SMgrRelation reln, ForkNumber forknum,
+ BlockNumber blocknum,
+ const void **buffers, BlockNumber nblocks, bool skipFsync);
extern void mdwriteback(SMgrRelation reln, ForkNumber forknum,
BlockNumber blocknum, BlockNumber nblocks);
extern BlockNumber mdnblocks(SMgrRelation reln, ForkNumber forknum);
diff --git a/src/include/storage/smgr.h b/src/include/storage/smgr.h
index 856ebcda350..f00b3763ac9 100644
--- a/src/include/storage/smgr.h
+++ b/src/include/storage/smgr.h
@@ -108,6 +108,11 @@ extern void smgrwritev(SMgrRelation reln, ForkNumber forknum,
BlockNumber blocknum,
const void **buffers, BlockNumber nblocks,
bool skipFsync);
+extern void smgrstartwritev(PgAioHandle *ioh,
+ SMgrRelation reln, ForkNumber forknum,
+ BlockNumber blocknum,
+ const void **buffers, BlockNumber nblocks,
+ bool skipFsync);
extern void smgrwriteback(SMgrRelation reln, ForkNumber forknum,
BlockNumber blocknum, BlockNumber nblocks);
extern BlockNumber smgrnblocks(SMgrRelation reln, ForkNumber forknum);
diff --git a/src/backend/storage/aio/aio_callback.c b/src/backend/storage/aio/aio_callback.c
index 3cdd2a6b195..03e44b76075 100644
--- a/src/backend/storage/aio/aio_callback.c
+++ b/src/backend/storage/aio/aio_callback.c
@@ -41,6 +41,7 @@ static const PgAioHandleCallbacksEntry aio_handle_cbs[] = {
CALLBACK_ENTRY(PGAIO_HCB_INVALID, aio_invalid_cb),
CALLBACK_ENTRY(PGAIO_HCB_MD_READV, aio_md_readv_cb),
+ CALLBACK_ENTRY(PGAIO_HCB_MD_WRITEV, aio_md_writev_cb),
CALLBACK_ENTRY(PGAIO_HCB_SHARED_BUFFER_READV, aio_shared_buffer_readv_cb),
diff --git a/src/backend/storage/file/fd.c b/src/backend/storage/file/fd.c
index f573aa8e7ca..8f143c10d36 100644
--- a/src/backend/storage/file/fd.c
+++ b/src/backend/storage/file/fd.c
@@ -2348,6 +2348,34 @@ retry:
return returnCode;
}
+int
+FileStartWriteV(PgAioHandle *ioh, File file,
+ int iovcnt, off_t offset,
+ uint32 wait_event_info)
+{
+ int returnCode;
+ Vfd *vfdP;
+
+ Assert(FileIsValid(file));
+
+ DO_DB(elog(LOG, "FileStartWriteV: %d (%s) " INT64_FORMAT " %d",
+ file, VfdCache[file].fileName,
+ (int64) offset,
+ iovcnt));
+
+ returnCode = FileAccess(file);
+ if (returnCode < 0)
+ return returnCode;
+
+ vfdP = &VfdCache[file];
+
+ /* FIXME: think about / reimplement temp_file_limit */
+
+ pgaio_io_prep_writev(ioh, vfdP->fd, iovcnt, offset);
+
+ return 0;
+}
+
int
FileSync(File file, uint32 wait_event_info)
{
diff --git a/src/backend/storage/smgr/md.c b/src/backend/storage/smgr/md.c
index 2218f17e7a0..9d3dec2710c 100644
--- a/src/backend/storage/smgr/md.c
+++ b/src/backend/storage/smgr/md.c
@@ -155,12 +155,19 @@ static BlockNumber _mdnblocks(SMgrRelation reln, ForkNumber forknum,
static PgAioResult md_readv_complete(PgAioHandle *ioh, PgAioResult prior_result, uint8 cb_data);
static void md_readv_report(PgAioResult result, const PgAioTargetData *target_data, int elevel);
+static PgAioResult md_writev_complete(PgAioHandle *ioh, PgAioResult prior_result, uint8 cb_data);
+static void md_writev_report(PgAioResult result, const PgAioTargetData *target_data, int elevel);
const PgAioHandleCallbacks aio_md_readv_cb = {
.complete_shared = md_readv_complete,
.report = md_readv_report,
};
+const PgAioHandleCallbacks aio_md_writev_cb = {
+ .complete_shared = md_writev_complete,
+ .report = md_writev_report,
+};
+
static inline int
_mdfd_open_flags(void)
@@ -1115,6 +1122,64 @@ mdwritev(SMgrRelation reln, ForkNumber forknum, BlockNumber blocknum,
}
}
+/*
+ * mdstartwritev() -- Asynchronous version of mdrwritev().
+ */
+void
+mdstartwritev(PgAioHandle *ioh,
+ SMgrRelation reln, ForkNumber forknum, BlockNumber blocknum,
+ const void **buffers, BlockNumber nblocks, bool skipFsync)
+{
+ off_t seekpos;
+ MdfdVec *v;
+ BlockNumber nblocks_this_segment;
+ struct iovec *iov;
+ int iovcnt;
+ int ret;
+
+ v = _mdfd_getseg(reln, forknum, blocknum, false,
+ EXTENSION_FAIL | EXTENSION_CREATE_RECOVERY);
+
+ seekpos = (off_t) BLCKSZ * (blocknum % ((BlockNumber) RELSEG_SIZE));
+
+ Assert(seekpos < (off_t) BLCKSZ * RELSEG_SIZE);
+
+ nblocks_this_segment =
+ Min(nblocks,
+ RELSEG_SIZE - (blocknum % ((BlockNumber) RELSEG_SIZE)));
+
+ if (nblocks_this_segment != nblocks)
+ elog(ERROR, "write crossing segment boundary");
+
+ iovcnt = pgaio_io_get_iovec(ioh, &iov);
+
+ Assert(nblocks <= iovcnt);
+
+ iovcnt = buffers_to_iovec(iov, unconstify(void **, buffers), nblocks_this_segment);
+
+ Assert(iovcnt <= nblocks_this_segment);
+
+ if (!(io_direct_flags & IO_DIRECT_DATA))
+ pgaio_io_set_flag(ioh, PGAIO_HF_BUFFERED);
+
+ pgaio_io_set_target_smgr(ioh,
+ reln,
+ forknum,
+ blocknum,
+ nblocks,
+ skipFsync);
+ pgaio_io_register_callbacks(ioh, PGAIO_HCB_MD_WRITEV, 0);
+
+ ret = FileStartWriteV(ioh, v->mdfd_vfd, iovcnt, seekpos, WAIT_EVENT_DATA_FILE_WRITE);
+ if (ret != 0)
+ ereport(ERROR,
+ (errcode_for_file_access(),
+ errmsg("could not start writing blocks %u..%u in file \"%s\": %m",
+ blocknum,
+ blocknum + nblocks_this_segment - 1,
+ FilePathName(v->mdfd_vfd))));
+}
+
/*
* mdwriteback() -- Tell the kernel to write pages back to storage.
@@ -1503,6 +1568,40 @@ register_dirty_segment(SMgrRelation reln, ForkNumber forknum, MdfdVec *seg)
}
}
+/*
+ * Like register_dirty_segment(), except for use by AIO. In the completion
+ * callback we don't have access to the MdfdVec (the completion callback might
+ * be executed in a different backend than the issuing backend), therefore we
+ * have to implement this slightly differently.
+ */
+static void
+register_dirty_segment_aio(RelFileLocator locator, ForkNumber forknum, uint64 segno)
+{
+ FileTag tag;
+
+ INIT_MD_FILETAG(tag, locator, forknum, segno);
+
+ /*
+ * Can't block here waiting for checkpointer to accept our sync request,
+ * as checkpointer might be waiting for this AIO to finish if offloaded to
+ * a worker.
+ */
+ if (!RegisterSyncRequest(&tag, SYNC_REQUEST, false /* retryOnError */ ))
+ {
+ char path[MAXPGPATH];
+
+ ereport(DEBUG1,
+ (errmsg_internal("could not forward fsync request because request queue is full")));
+
+ /* reuse mdsyncfiletag() to avoid duplicating code */
+ if (mdsyncfiletag(&tag, path))
+ ereport(data_sync_elevel(ERROR),
+ (errcode_for_file_access(),
+ errmsg("could not fsync file \"%s\": %m",
+ path)));
+ }
+}
+
/*
* register_unlink_segment() -- Schedule a file to be deleted after next checkpoint
*/
@@ -2027,3 +2126,103 @@ md_readv_report(PgAioResult result, const PgAioTargetData *td, int elevel)
td->smgr.nblocks * (size_t) BLCKSZ));
}
}
+
+/*
+ * AIO completion callback for mdstartwritev().
+ */
+static PgAioResult
+md_writev_complete(PgAioHandle *ioh, PgAioResult prior_result, uint8 cb_data)
+{
+ PgAioTargetData *td = pgaio_io_get_target_data(ioh);
+ PgAioResult result = prior_result;
+
+ if (prior_result.result < 0)
+ {
+ result.status = PGAIO_RS_ERROR;
+ result.id = PGAIO_HCB_MD_WRITEV;
+ /* For "hard" errors, track the error number in error_data */
+ result.error_data = -prior_result.result;
+ result.result = 0;
+
+ pgaio_result_report(result, td, LOG);
+
+ return result;
+ }
+
+ /*
+ * As explained above smgrstartwritev(), the smgr API operates on the
+ * level of blocks, rather than bytes. Convert.
+ */
+ result.result /= BLCKSZ;
+
+ Assert(result.result <= td->smgr.nblocks);
+
+ if (result.result == 0)
+ {
+ /* consider 0 blocks written a failure */
+ result.status = PGAIO_RS_ERROR;
+ result.id = PGAIO_HCB_MD_WRITEV;
+ result.error_data = 0;
+
+ pgaio_result_report(result, td, LOG);
+
+ return result;
+ }
+
+ if (result.status != PGAIO_RS_ERROR &&
+ result.result < td->smgr.nblocks)
+ {
+ /* partial writes should be retried at upper level */
+ result.status = PGAIO_RS_PARTIAL;
+ result.id = PGAIO_HCB_MD_WRITEV;
+ }
+
+ if (!td->smgr.skip_fsync)
+ register_dirty_segment_aio(td->smgr.rlocator, td->smgr.forkNum,
+ td->smgr.blockNum / ((BlockNumber) RELSEG_SIZE));
+
+ return result;
+}
+
+/*
+ * AIO error reporting callback for mdstartwritev().
+ */
+static void
+md_writev_report(PgAioResult result, const PgAioTargetData *td, int elevel)
+{
+ RelPathStr path;
+
+ path = relpathbackend(td->smgr.rlocator,
+ td->smgr.is_temp ? MyProcNumber : INVALID_PROC_NUMBER,
+ td->smgr.forkNum);
+
+ if (result.error_data != 0)
+ {
+ errno = result.error_data; /* for errcode_for_file_access() */
+
+ ereport(elevel,
+ errcode_for_file_access(),
+ errmsg("could not write blocks %u..%u in file \"%s\": %m",
+ td->smgr.blockNum,
+ td->smgr.blockNum + td->smgr.nblocks,
+ path.str)
+ );
+ }
+ else
+ {
+ /*
+ * NB: This will typically only be output in debug messages, while
+ * retrying a partial IO.
+ */
+ ereport(elevel,
+ errcode(ERRCODE_DATA_CORRUPTED),
+ errmsg("could not write blocks %u..%u in file \"%s\": wrote only %zu of %zu bytes",
+ td->smgr.blockNum,
+ td->smgr.blockNum + td->smgr.nblocks - 1,
+ path.str,
+ result.result * (size_t) BLCKSZ,
+ td->smgr.nblocks * (size_t) BLCKSZ
+ )
+ );
+ }
+}
diff --git a/src/backend/storage/smgr/smgr.c b/src/backend/storage/smgr/smgr.c
index 545888dcdfc..aec2c0c565a 100644
--- a/src/backend/storage/smgr/smgr.c
+++ b/src/backend/storage/smgr/smgr.c
@@ -115,6 +115,11 @@ typedef struct f_smgr
BlockNumber blocknum,
const void **buffers, BlockNumber nblocks,
bool skipFsync);
+ void (*smgr_startwritev) (PgAioHandle *ioh,
+ SMgrRelation reln, ForkNumber forknum,
+ BlockNumber blocknum,
+ const void **buffers, BlockNumber nblocks,
+ bool skipFsync);
void (*smgr_writeback) (SMgrRelation reln, ForkNumber forknum,
BlockNumber blocknum, BlockNumber nblocks);
BlockNumber (*smgr_nblocks) (SMgrRelation reln, ForkNumber forknum);
@@ -142,6 +147,7 @@ static const f_smgr smgrsw[] = {
.smgr_readv = mdreadv,
.smgr_startreadv = mdstartreadv,
.smgr_writev = mdwritev,
+ .smgr_startwritev = mdstartwritev,
.smgr_writeback = mdwriteback,
.smgr_nblocks = mdnblocks,
.smgr_truncate = mdtruncate,
@@ -787,6 +793,29 @@ smgrwritev(SMgrRelation reln, ForkNumber forknum, BlockNumber blocknum,
RESUME_INTERRUPTS();
}
+/*
+ * smgrstartwritev() -- asynchronous version of smgrwritev()
+ *
+ * This starts an asynchronous writev IO using the IO handle `ioh`. Other than
+ * `ioh` all parameters are the same as smgrwritev().
+ *
+ * Completion callbacks above smgr will be passed the result as the number of
+ * successfully written blocks if the write [partially] succeeds. This
+ * maintains the abstraction that smgr operates on the level of blocks, rather
+ * than bytes.
+ */
+void
+smgrstartwritev(PgAioHandle *ioh,
+ SMgrRelation reln, ForkNumber forknum, BlockNumber blocknum,
+ const void **buffers, BlockNumber nblocks, bool skipFsync)
+{
+ HOLD_INTERRUPTS();
+ smgrsw[reln->smgr_which].smgr_startwritev(ioh,
+ reln, forknum, blocknum, buffers,
+ nblocks, skipFsync);
+ RESUME_INTERRUPTS();
+}
+
/*
* smgrwriteback() -- Trigger kernel writeback for the supplied range of
* blocks.
--
2.48.1.76.g4e746b1a31.dirty
[text/x-diff] v2.11-0020-aio-Add-bounce-buffers.patch (23.1K, ../../6ak556uyqiptdwjaci4kbi5eykwkmzqgkbtkyaosjnopjhncrc@2v4ac2jwyz22/21-v2.11-0020-aio-Add-bounce-buffers.patch)
download | inline diff:
From 8be13a1426cb9d4a9f16dc0bfee0e59c9f375742 Mon Sep 17 00:00:00 2001
From: Andres Freund <andres@anarazel.de>
Date: Tue, 18 Mar 2025 14:40:06 -0400
Subject: [PATCH v2.11 20/27] aio: Add bounce buffers
---
src/include/storage/aio.h | 15 ++
src/include/storage/aio_internal.h | 34 ++++
src/include/storage/aio_types.h | 2 +
src/include/utils/resowner.h | 2 +
src/backend/storage/aio/README.md | 27 +++
src/backend/storage/aio/aio.c | 178 ++++++++++++++++++
src/backend/storage/aio/aio_init.c | 123 ++++++++++++
src/backend/utils/misc/guc_tables.c | 13 ++
src/backend/utils/misc/postgresql.conf.sample | 2 +
src/backend/utils/resowner/resowner.c | 25 ++-
src/test/modules/test_aio/test_aio--1.0.sql | 21 +++
src/test/modules/test_aio/test_aio.c | 55 ++++++
src/tools/pgindent/typedefs.list | 1 +
13 files changed, 496 insertions(+), 2 deletions(-)
diff --git a/src/include/storage/aio.h b/src/include/storage/aio.h
index 9a0868a270c..09d6d9fe1bc 100644
--- a/src/include/storage/aio.h
+++ b/src/include/storage/aio.h
@@ -349,6 +349,20 @@ extern bool pgaio_have_staged(void);
+/* --------------------------------------------------------------------------------
+ * Bounce Buffers
+ * --------------------------------------------------------------------------------
+ */
+
+extern PgAioBounceBuffer *pgaio_bounce_buffer_get(void);
+extern void pgaio_io_assoc_bounce_buffer(PgAioHandle *ioh, PgAioBounceBuffer *bb);
+extern uint32 pgaio_bounce_buffer_id(PgAioBounceBuffer *bb);
+extern void pgaio_bounce_buffer_release(PgAioBounceBuffer *bb);
+extern char *pgaio_bounce_buffer_buffer(PgAioBounceBuffer *bb);
+extern void pgaio_bounce_buffer_release_resowner(struct dlist_node *bb_node, bool on_error);
+
+
+
/* --------------------------------------------------------------------------------
* Other
* --------------------------------------------------------------------------------
@@ -361,6 +375,7 @@ extern void pgaio_closing_fd(int fd);
/* GUCs */
extern PGDLLIMPORT int io_method;
extern PGDLLIMPORT int io_max_concurrency;
+extern PGDLLIMPORT int io_bounce_buffers;
#endif /* AIO_H */
diff --git a/src/include/storage/aio_internal.h b/src/include/storage/aio_internal.h
index 6e6033519b2..447208c8f12 100644
--- a/src/include/storage/aio_internal.h
+++ b/src/include/storage/aio_internal.h
@@ -122,6 +122,12 @@ struct PgAioHandle
/* raw result of the IO operation */
int32 result;
+ /*
+ * List of bounce_buffers owned by IO. It would suffice to use an index
+ * based linked list here.
+ */
+ slist_head bounce_buffers;
+
/**
* In which list the handle is registered, depends on the state:
* - IDLE, in per-backend list
@@ -177,11 +183,24 @@ struct PgAioHandle
};
+/* typedef is in aio_types.h */
+struct PgAioBounceBuffer
+{
+ slist_node node;
+ struct ResourceOwnerData *resowner;
+ dlist_node resowner_node;
+ char *buffer;
+};
+
+
typedef struct PgAioBackend
{
/* index into PgAioCtl->io_handles */
uint32 io_handle_off;
+ /* index into PgAioCtl->bounce_buffers */
+ uint32 bounce_buffers_off;
+
/* IO Handles that currently are not used */
dclist_head idle_ios;
@@ -212,6 +231,12 @@ typedef struct PgAioBackend
* IOs being appended at the end.
*/
dclist_head in_flight_ios;
+
+ /* Bounce Buffers that currently are not used */
+ slist_head idle_bbs;
+
+ /* see handed_out_io */
+ PgAioBounceBuffer *handed_out_bb;
} PgAioBackend;
@@ -239,6 +264,15 @@ typedef struct PgAioCtl
uint32 io_handle_count;
PgAioHandle *io_handles;
+
+ /*
+ * To perform AIO on buffers that are not located in shared memory (either
+ * because they are not in shared memory or because we need to operate on
+ * a copy, as e.g. the case for writes when checksums are in use)
+ */
+ uint32 bounce_buffers_count;
+ PgAioBounceBuffer *bounce_buffers;
+ char *bounce_buffers_data;
} PgAioCtl;
diff --git a/src/include/storage/aio_types.h b/src/include/storage/aio_types.h
index debe8163d4e..02950ee56a7 100644
--- a/src/include/storage/aio_types.h
+++ b/src/include/storage/aio_types.h
@@ -118,4 +118,6 @@ typedef struct PgAioReturn
} PgAioReturn;
+typedef struct PgAioBounceBuffer PgAioBounceBuffer;
+
#endif /* AIO_TYPES_H */
diff --git a/src/include/utils/resowner.h b/src/include/utils/resowner.h
index aede4bfc820..7e2ec224169 100644
--- a/src/include/utils/resowner.h
+++ b/src/include/utils/resowner.h
@@ -168,5 +168,7 @@ extern void ResourceOwnerForgetLock(ResourceOwner owner, struct LOCALLOCK *local
struct dlist_node;
extern void ResourceOwnerRememberAioHandle(ResourceOwner owner, struct dlist_node *ioh_node);
extern void ResourceOwnerForgetAioHandle(ResourceOwner owner, struct dlist_node *ioh_node);
+extern void ResourceOwnerRememberAioBounceBuffer(ResourceOwner owner, struct dlist_node *bb_node);
+extern void ResourceOwnerForgetAioBounceBuffer(ResourceOwner owner, struct dlist_node *bb_node);
#endif /* RESOWNER_H */
diff --git a/src/backend/storage/aio/README.md b/src/backend/storage/aio/README.md
index 012cf3f305a..57e6f2fef07 100644
--- a/src/backend/storage/aio/README.md
+++ b/src/backend/storage/aio/README.md
@@ -405,6 +405,33 @@ shared memory no less!), completion callbacks instead have to encode errors in
a more compact format that can be converted into an error message.
+### AIO Bounce Buffers
+
+For some uses of AIO there is no convenient memory location as the source /
+destination of an AIO. E.g. when data checksums are enabled, writes from
+shared buffers currently cannot be done directly from shared buffers, as a
+shared buffer lock still allows some modification, e.g., for hint bits(see
+`FlushBuffer()`). If the write were done in-place, such modifications can
+cause the checksum to fail.
+
+For synchronous IO this is solved by copying the buffer to separate memory
+before computing the checksum and using that copy as the source buffer for the
+AIO.
+
+However, for AIO that is not a workable solution:
+- Instead of a single buffer many buffers are required, as many IOs might be
+ in flight
+- When using the [worker method](#worker), the source/target of IO needs to be
+ in shared memory, otherwise the workers won't be able to access the memory.
+
+The AIO subsystem addresses this by providing a limited number of bounce
+buffers that can be used as the source / target for IO. A bounce buffer be
+acquired with `pgaio_bounce_buffer_get()` and multiple bounce buffers can be
+associated with an AIO Handle with `pgaio_io_assoc_bounce_buffer()`.
+
+Bounce buffers are automatically released when the IO completes.
+
+
## Helpers
Using the low-level AIO API introduces too much complexity to do so all over
diff --git a/src/backend/storage/aio/aio.c b/src/backend/storage/aio/aio.c
index 3c3383b1318..ef344c32c61 100644
--- a/src/backend/storage/aio/aio.c
+++ b/src/backend/storage/aio/aio.c
@@ -62,6 +62,8 @@ static PgAioHandle *pgaio_io_from_wref(PgAioWaitRef *iow, uint64 *ref_generation
static const char *pgaio_io_state_get_name(PgAioHandleState s);
static void pgaio_io_wait(PgAioHandle *ioh, uint64 ref_generation);
+static void pgaio_bounce_buffer_wait_for_free(void);
+
/* Options for io_method. */
const struct config_enum_entry io_method_options[] = {
@@ -76,6 +78,7 @@ const struct config_enum_entry io_method_options[] = {
/* GUCs */
int io_method = DEFAULT_IO_METHOD;
int io_max_concurrency = -1;
+int io_bounce_buffers = -1;
/* global control for AIO */
PgAioCtl *pgaio_ctl;
@@ -662,6 +665,21 @@ pgaio_io_reclaim(PgAioHandle *ioh)
}
}
+ /* reclaim all associated bounce buffers */
+ if (!slist_is_empty(&ioh->bounce_buffers))
+ {
+ slist_mutable_iter it;
+
+ slist_foreach_modify(it, &ioh->bounce_buffers)
+ {
+ PgAioBounceBuffer *bb = slist_container(PgAioBounceBuffer, node, it.cur);
+
+ slist_delete_current(&it);
+
+ slist_push_head(&pgaio_my_backend->idle_bbs, &bb->node);
+ }
+ }
+
if (ioh->resowner)
{
ResourceOwnerForgetAioHandle(ioh->resowner, &ioh->resowner_node);
@@ -1044,6 +1062,166 @@ pgaio_submit_staged(void)
+/* --------------------------------------------------------------------------------
+ * Functions primarily related to PgAioBounceBuffer
+ * --------------------------------------------------------------------------------
+ */
+
+PgAioBounceBuffer *
+pgaio_bounce_buffer_get(void)
+{
+ PgAioBounceBuffer *bb = NULL;
+ slist_node *node;
+
+ if (pgaio_my_backend->handed_out_bb != NULL)
+ elog(ERROR, "can only hand out one BB");
+
+ /*
+ * XXX: It probably is not a good idea to have bounce buffers be per
+ * backend, that's a fair bit of memory.
+ */
+ if (slist_is_empty(&pgaio_my_backend->idle_bbs))
+ {
+ pgaio_bounce_buffer_wait_for_free();
+ }
+
+ node = slist_pop_head_node(&pgaio_my_backend->idle_bbs);
+ bb = slist_container(PgAioBounceBuffer, node, node);
+
+ pgaio_my_backend->handed_out_bb = bb;
+
+ bb->resowner = CurrentResourceOwner;
+ ResourceOwnerRememberAioBounceBuffer(bb->resowner, &bb->resowner_node);
+
+ return bb;
+}
+
+void
+pgaio_io_assoc_bounce_buffer(PgAioHandle *ioh, PgAioBounceBuffer *bb)
+{
+ if (pgaio_my_backend->handed_out_bb != bb)
+ elog(ERROR, "can only assign handed out BB");
+ pgaio_my_backend->handed_out_bb = NULL;
+
+ /*
+ * There can be many bounce buffers assigned in case of vectorized IOs.
+ */
+ slist_push_head(&ioh->bounce_buffers, &bb->node);
+
+ /* once associated with an IO, the IO has ownership */
+ ResourceOwnerForgetAioBounceBuffer(bb->resowner, &bb->resowner_node);
+ bb->resowner = NULL;
+}
+
+uint32
+pgaio_bounce_buffer_id(PgAioBounceBuffer *bb)
+{
+ return bb - pgaio_ctl->bounce_buffers;
+}
+
+void
+pgaio_bounce_buffer_release(PgAioBounceBuffer *bb)
+{
+ if (pgaio_my_backend->handed_out_bb != bb)
+ elog(ERROR, "can only release handed out BB");
+
+ slist_push_head(&pgaio_my_backend->idle_bbs, &bb->node);
+ pgaio_my_backend->handed_out_bb = NULL;
+
+ ResourceOwnerForgetAioBounceBuffer(bb->resowner, &bb->resowner_node);
+ bb->resowner = NULL;
+}
+
+void
+pgaio_bounce_buffer_release_resowner(dlist_node *bb_node, bool on_error)
+{
+ PgAioBounceBuffer *bb = dlist_container(PgAioBounceBuffer, resowner_node, bb_node);
+
+ Assert(bb->resowner);
+
+ if (!on_error)
+ elog(WARNING, "leaked AIO bounce buffer");
+
+ pgaio_bounce_buffer_release(bb);
+}
+
+char *
+pgaio_bounce_buffer_buffer(PgAioBounceBuffer *bb)
+{
+ return bb->buffer;
+}
+
+static void
+pgaio_bounce_buffer_wait_for_free(void)
+{
+ static uint32 lastpos = 0;
+
+ if (pgaio_my_backend->num_staged_ios > 0)
+ {
+ pgaio_debug(DEBUG2, "submitting %d, while acquiring free bb",
+ pgaio_my_backend->num_staged_ios);
+ pgaio_submit_staged();
+ }
+
+ for (uint32 i = lastpos; i < lastpos + io_max_concurrency; i++)
+ {
+ uint32 thisoff = pgaio_my_backend->io_handle_off + (i % io_max_concurrency);
+ PgAioHandle *ioh = &pgaio_ctl->io_handles[thisoff];
+
+ switch (ioh->state)
+ {
+ case PGAIO_HS_IDLE:
+ case PGAIO_HS_HANDED_OUT:
+ continue;
+ case PGAIO_HS_DEFINED: /* should have been submitted above */
+ case PGAIO_HS_STAGED:
+ elog(ERROR, "shouldn't get here with io:%d in state %d",
+ pgaio_io_get_id(ioh), ioh->state);
+ break;
+ case PGAIO_HS_COMPLETED_IO:
+ case PGAIO_HS_SUBMITTED:
+ if (!slist_is_empty(&ioh->bounce_buffers))
+ {
+ pgaio_debug_io(DEBUG2, ioh,
+ "waiting for IO to reclaim BB with %d in flight",
+ dclist_count(&pgaio_my_backend->in_flight_ios));
+
+ /* see comment in pgaio_io_wait_for_free() about raciness */
+ pgaio_io_wait(ioh, ioh->generation);
+
+ if (slist_is_empty(&pgaio_my_backend->idle_bbs))
+ elog(WARNING, "empty after wait");
+
+ if (!slist_is_empty(&pgaio_my_backend->idle_bbs))
+ {
+ lastpos = i;
+ return;
+ }
+ }
+ break;
+ case PGAIO_HS_COMPLETED_SHARED:
+ case PGAIO_HS_COMPLETED_LOCAL:
+ /* reclaim */
+ pgaio_io_reclaim(ioh);
+
+ if (!slist_is_empty(&pgaio_my_backend->idle_bbs))
+ {
+ lastpos = i;
+ return;
+ }
+ break;
+ }
+ }
+
+ /*
+ * The submission above could have caused the IO to complete at any time.
+ */
+ if (slist_is_empty(&pgaio_my_backend->idle_bbs))
+ elog(PANIC, "no more bbs");
+}
+
+
+
/* --------------------------------------------------------------------------------
* Other
* --------------------------------------------------------------------------------
diff --git a/src/backend/storage/aio/aio_init.c b/src/backend/storage/aio/aio_init.c
index 885c3940c66..95b10933fed 100644
--- a/src/backend/storage/aio/aio_init.c
+++ b/src/backend/storage/aio/aio_init.c
@@ -88,6 +88,32 @@ AioHandleDataShmemSize(void)
io_max_concurrency));
}
+static Size
+AioBounceBufferDescShmemSize(void)
+{
+ Size sz;
+
+ /* PgAioBounceBuffer itself */
+ sz = mul_size(sizeof(PgAioBounceBuffer),
+ mul_size(AioProcs(), io_bounce_buffers));
+
+ return sz;
+}
+
+static Size
+AioBounceBufferDataShmemSize(void)
+{
+ Size sz;
+
+ /* and the associated buffer */
+ sz = mul_size(BLCKSZ,
+ mul_size(io_bounce_buffers, AioProcs()));
+ /* memory for alignment */
+ sz += BLCKSZ;
+
+ return sz;
+}
+
/*
* Choose a suitable value for io_max_concurrency.
*
@@ -113,6 +139,33 @@ AioChooseMaxConcurrency(void)
return Min(max_proportional_pins, 64);
}
+/*
+ * Choose a suitable value for io_bounce_buffers.
+ *
+ * It's very unlikely to be useful to allocate more bounce buffers for each
+ * backend than the backend is allowed to pin. Additionally, bounce buffers
+ * currently are used for writes, it seems very uncommon for more than 10% of
+ * shared_buffers to be written out concurrently.
+ *
+ * XXX: This quickly can take up significant amounts of memory, the logic
+ * should probably fine tuned.
+ */
+static int
+AioChooseBounceBuffers(void)
+{
+ uint32 max_backends;
+ int max_proportional_pins;
+
+ /* Similar logic to LimitAdditionalPins() */
+ max_backends = MaxBackends + NUM_AUXILIARY_PROCS;
+ max_proportional_pins = (NBuffers / 10) / max_backends;
+
+ max_proportional_pins = Max(max_proportional_pins, 1);
+
+ /* apply upper limit */
+ return Min(max_proportional_pins, 256);
+}
+
Size
AioShmemSize(void)
{
@@ -136,11 +189,31 @@ AioShmemSize(void)
PGC_S_OVERRIDE);
}
+
+ /*
+ * If io_bounce_buffers is -1, we automatically choose a suitable value.
+ *
+ * See also comment above.
+ */
+ if (io_bounce_buffers == -1)
+ {
+ char buf[32];
+
+ snprintf(buf, sizeof(buf), "%d", AioChooseBounceBuffers());
+ SetConfigOption("io_bounce_buffers", buf, PGC_POSTMASTER,
+ PGC_S_DYNAMIC_DEFAULT);
+ if (io_bounce_buffers == -1) /* failed to apply it? */
+ SetConfigOption("io_bounce_buffers", buf, PGC_POSTMASTER,
+ PGC_S_OVERRIDE);
+ }
+
sz = add_size(sz, AioCtlShmemSize());
sz = add_size(sz, AioBackendShmemSize());
sz = add_size(sz, AioHandleShmemSize());
sz = add_size(sz, AioHandleIOVShmemSize());
sz = add_size(sz, AioHandleDataShmemSize());
+ sz = add_size(sz, AioBounceBufferDescShmemSize());
+ sz = add_size(sz, AioBounceBufferDataShmemSize());
/* Reserve space for method specific resources. */
if (pgaio_method_ops->shmem_size)
@@ -156,6 +229,9 @@ AioShmemInit(void)
uint32 io_handle_off = 0;
uint32 iovec_off = 0;
uint32 per_backend_iovecs = io_max_concurrency * io_max_combine_limit;
+ uint32 bounce_buffers_off = 0;
+ uint32 per_backend_bb = io_bounce_buffers;
+ char *bounce_buffers_data;
pgaio_ctl = (PgAioCtl *)
ShmemInitStruct("AioCtl", AioCtlShmemSize(), &found);
@@ -167,6 +243,7 @@ AioShmemInit(void)
pgaio_ctl->io_handle_count = AioProcs() * io_max_concurrency;
pgaio_ctl->iovec_count = AioProcs() * per_backend_iovecs;
+ pgaio_ctl->bounce_buffers_count = AioProcs() * per_backend_bb;
pgaio_ctl->backend_state = (PgAioBackend *)
ShmemInitStruct("AioBackend", AioBackendShmemSize(), &found);
@@ -179,6 +256,40 @@ AioShmemInit(void)
pgaio_ctl->handle_data = (uint64 *)
ShmemInitStruct("AioHandleData", AioHandleDataShmemSize(), &found);
+ pgaio_ctl->bounce_buffers = (PgAioBounceBuffer *)
+ ShmemInitStruct("AioBounceBufferDesc", AioBounceBufferDescShmemSize(),
+ &found);
+
+ bounce_buffers_data =
+ ShmemInitStruct("AioBounceBufferData", AioBounceBufferDataShmemSize(),
+ &found);
+ bounce_buffers_data =
+ (char *) TYPEALIGN(BLCKSZ, (uintptr_t) bounce_buffers_data);
+ pgaio_ctl->bounce_buffers_data = bounce_buffers_data;
+
+
+ /* Initialize IO handles. */
+ for (uint64 i = 0; i < pgaio_ctl->io_handle_count; i++)
+ {
+ PgAioHandle *ioh = &pgaio_ctl->io_handles[i];
+
+ ioh->op = PGAIO_OP_INVALID;
+ ioh->target = PGAIO_TID_INVALID;
+ ioh->state = PGAIO_HS_IDLE;
+
+ slist_init(&ioh->bounce_buffers);
+ }
+
+ /* Initialize Bounce Buffers. */
+ for (uint64 i = 0; i < pgaio_ctl->bounce_buffers_count; i++)
+ {
+ PgAioBounceBuffer *bb = &pgaio_ctl->bounce_buffers[i];
+
+ bb->buffer = bounce_buffers_data;
+ bounce_buffers_data += BLCKSZ;
+ }
+
+
for (int procno = 0; procno < AioProcs(); procno++)
{
PgAioBackend *bs = &pgaio_ctl->backend_state[procno];
@@ -186,9 +297,13 @@ AioShmemInit(void)
bs->io_handle_off = io_handle_off;
io_handle_off += io_max_concurrency;
+ bs->bounce_buffers_off = bounce_buffers_off;
+ bounce_buffers_off += per_backend_bb;
+
dclist_init(&bs->idle_ios);
memset(bs->staged_ios, 0, sizeof(PgAioHandle *) * PGAIO_SUBMIT_BATCH_SIZE);
dclist_init(&bs->in_flight_ios);
+ slist_init(&bs->idle_bbs);
/* initialize per-backend IOs */
for (int i = 0; i < io_max_concurrency; i++)
@@ -210,6 +325,14 @@ AioShmemInit(void)
dclist_push_tail(&bs->idle_ios, &ioh->node);
iovec_off += io_max_combine_limit;
}
+
+ /* initialize per-backend bounce buffers */
+ for (int i = 0; i < per_backend_bb; i++)
+ {
+ PgAioBounceBuffer *bb = &pgaio_ctl->bounce_buffers[bs->bounce_buffers_off + i];
+
+ slist_push_head(&bs->idle_bbs, &bb->node);
+ }
}
out:
diff --git a/src/backend/utils/misc/guc_tables.c b/src/backend/utils/misc/guc_tables.c
index 2e17dbc8c9e..89f9b1fbab7 100644
--- a/src/backend/utils/misc/guc_tables.c
+++ b/src/backend/utils/misc/guc_tables.c
@@ -3303,6 +3303,19 @@ struct config_int ConfigureNamesInt[] =
check_io_max_concurrency, NULL, NULL
},
+ {
+ {"io_bounce_buffers",
+ PGC_POSTMASTER,
+ RESOURCES_IO,
+ gettext_noop("Number of IO Bounce Buffers reserved for each backend."),
+ NULL,
+ GUC_UNIT_BLOCKS
+ },
+ &io_bounce_buffers,
+ -1, -1, 4096,
+ NULL, NULL, NULL
+ },
+
{
{"io_workers",
PGC_SIGHUP,
diff --git a/src/backend/utils/misc/postgresql.conf.sample b/src/backend/utils/misc/postgresql.conf.sample
index d5f5f3dbda4..762194c61ad 100644
--- a/src/backend/utils/misc/postgresql.conf.sample
+++ b/src/backend/utils/misc/postgresql.conf.sample
@@ -211,6 +211,8 @@
# -1 sets based on shared_buffers
# (change requires restart)
#io_workers = 3 # 1-32;
+#io_bounce_buffers = -1 # -1 sets based on shared_buffers
+ # (change requires restart)
# - Worker Processes -
diff --git a/src/backend/utils/resowner/resowner.c b/src/backend/utils/resowner/resowner.c
index d39f3e1b655..81e7e27965a 100644
--- a/src/backend/utils/resowner/resowner.c
+++ b/src/backend/utils/resowner/resowner.c
@@ -159,10 +159,11 @@ struct ResourceOwnerData
LOCALLOCK *locks[MAX_RESOWNER_LOCKS]; /* list of owned locks */
/*
- * AIO handles need be registered in critical sections and therefore
- * cannot use the normal ResourceElem mechanism.
+ * AIO handles & bounce buffers need be registered in critical sections
+ * and therefore cannot use the normal ResourceElem mechanism.
*/
dlist_head aio_handles;
+ dlist_head aio_bounce_buffers;
};
@@ -434,6 +435,7 @@ ResourceOwnerCreate(ResourceOwner parent, const char *name)
}
dlist_init(&owner->aio_handles);
+ dlist_init(&owner->aio_bounce_buffers);
return owner;
}
@@ -742,6 +744,13 @@ ResourceOwnerReleaseInternal(ResourceOwner owner,
pgaio_io_release_resowner(node, !isCommit);
}
+
+ while (!dlist_is_empty(&owner->aio_bounce_buffers))
+ {
+ dlist_node *node = dlist_head_node(&owner->aio_bounce_buffers);
+
+ pgaio_bounce_buffer_release_resowner(node, !isCommit);
+ }
}
else if (phase == RESOURCE_RELEASE_LOCKS)
{
@@ -1111,3 +1120,15 @@ ResourceOwnerForgetAioHandle(ResourceOwner owner, struct dlist_node *ioh_node)
{
dlist_delete_from(&owner->aio_handles, ioh_node);
}
+
+void
+ResourceOwnerRememberAioBounceBuffer(ResourceOwner owner, struct dlist_node *ioh_node)
+{
+ dlist_push_tail(&owner->aio_bounce_buffers, ioh_node);
+}
+
+void
+ResourceOwnerForgetAioBounceBuffer(ResourceOwner owner, struct dlist_node *ioh_node)
+{
+ dlist_delete_from(&owner->aio_bounce_buffers, ioh_node);
+}
diff --git a/src/test/modules/test_aio/test_aio--1.0.sql b/src/test/modules/test_aio/test_aio--1.0.sql
index 3d4e9bb070f..16f248619fc 100644
--- a/src/test/modules/test_aio/test_aio--1.0.sql
+++ b/src/test/modules/test_aio/test_aio--1.0.sql
@@ -76,6 +76,27 @@ RETURNS pg_catalog.void STRICT
AS 'MODULE_PATHNAME' LANGUAGE C;
+CREATE FUNCTION bb_get_and_error()
+RETURNS pg_catalog.void STRICT
+AS 'MODULE_PATHNAME' LANGUAGE C;
+
+CREATE FUNCTION bb_get_twice()
+RETURNS pg_catalog.void STRICT
+AS 'MODULE_PATHNAME' LANGUAGE C;
+
+CREATE FUNCTION bb_get()
+RETURNS pg_catalog.void STRICT
+AS 'MODULE_PATHNAME' LANGUAGE C;
+
+CREATE FUNCTION bb_get_release()
+RETURNS pg_catalog.void STRICT
+AS 'MODULE_PATHNAME' LANGUAGE C;
+
+CREATE FUNCTION bb_release_last()
+RETURNS pg_catalog.void STRICT
+AS 'MODULE_PATHNAME' LANGUAGE C;
+
+
/*
* Injection point related functions
diff --git a/src/test/modules/test_aio/test_aio.c b/src/test/modules/test_aio/test_aio.c
index fc55b985dad..1e2127fe888 100644
--- a/src/test/modules/test_aio/test_aio.c
+++ b/src/test/modules/test_aio/test_aio.c
@@ -53,6 +53,7 @@ static shmem_startup_hook_type prev_shmem_startup_hook = NULL;
static PgAioHandle *last_handle;
+static PgAioBounceBuffer *last_bb;
@@ -526,6 +527,60 @@ batch_end(PG_FUNCTION_ARGS)
PG_RETURN_VOID();
}
+PG_FUNCTION_INFO_V1(bb_get);
+Datum
+bb_get(PG_FUNCTION_ARGS)
+{
+ last_bb = pgaio_bounce_buffer_get();
+
+ PG_RETURN_VOID();
+}
+
+PG_FUNCTION_INFO_V1(bb_release_last);
+Datum
+bb_release_last(PG_FUNCTION_ARGS)
+{
+ if (!last_bb)
+ elog(ERROR, "no bb");
+
+ pgaio_bounce_buffer_release(last_bb);
+
+ PG_RETURN_VOID();
+}
+
+PG_FUNCTION_INFO_V1(bb_get_and_error);
+Datum
+bb_get_and_error(PG_FUNCTION_ARGS)
+{
+ pgaio_bounce_buffer_get();
+
+ elog(ERROR, "as you command");
+ PG_RETURN_VOID();
+}
+
+PG_FUNCTION_INFO_V1(bb_get_twice);
+Datum
+bb_get_twice(PG_FUNCTION_ARGS)
+{
+ pgaio_bounce_buffer_get();
+ pgaio_bounce_buffer_get();
+
+ PG_RETURN_VOID();
+}
+
+
+PG_FUNCTION_INFO_V1(bb_get_release);
+Datum
+bb_get_release(PG_FUNCTION_ARGS)
+{
+ PgAioBounceBuffer *bb;
+
+ bb = pgaio_bounce_buffer_get();
+ pgaio_bounce_buffer_release(bb);
+
+ PG_RETURN_VOID();
+}
+
#ifdef USE_INJECTION_POINTS
extern PGDLLEXPORT void inj_io_short_read(const char *name, const void *private_data);
extern PGDLLEXPORT void inj_io_reopen(const char *name, const void *private_data);
diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list
index 53abd54edd8..d633c732162 100644
--- a/src/tools/pgindent/typedefs.list
+++ b/src/tools/pgindent/typedefs.list
@@ -2132,6 +2132,7 @@ PermutationStep
PermutationStepBlocker
PermutationStepBlockerType
PgAioBackend
+PgAioBounceBuffer
PgAioCtl
PgAioHandle
PgAioHandleCallbackID
--
2.48.1.76.g4e746b1a31.dirty
[text/x-diff] v2.11-0021-bufmgr-Implement-AIO-write-support.patch (9.8K, ../../6ak556uyqiptdwjaci4kbi5eykwkmzqgkbtkyaosjnopjhncrc@2v4ac2jwyz22/22-v2.11-0021-bufmgr-Implement-AIO-write-support.patch)
download | inline diff:
From 9b4e2a0758fb149e3b7dd94728f948bab74ba5f0 Mon Sep 17 00:00:00 2001
From: Andres Freund <andres@anarazel.de>
Date: Tue, 18 Mar 2025 14:40:06 -0400
Subject: [PATCH v2.11 21/27] bufmgr: Implement AIO write support
As of this commit there are no users of these AIO facilities, that'll come in
later commits.
Author:
Reviewed-By:
Discussion: https://postgr.es/m/
Backpatch:
---
src/include/storage/aio.h | 2 +
src/include/storage/bufmgr.h | 2 +
src/backend/storage/aio/aio_callback.c | 2 +
src/backend/storage/buffer/bufmgr.c | 180 +++++++++++++++++++++++++
4 files changed, 186 insertions(+)
diff --git a/src/include/storage/aio.h b/src/include/storage/aio.h
index 09d6d9fe1bc..d461485e83f 100644
--- a/src/include/storage/aio.h
+++ b/src/include/storage/aio.h
@@ -197,8 +197,10 @@ typedef enum PgAioHandleCallbackID
PGAIO_HCB_MD_WRITEV,
PGAIO_HCB_SHARED_BUFFER_READV,
+ PGAIO_HCB_SHARED_BUFFER_WRITEV,
PGAIO_HCB_LOCAL_BUFFER_READV,
+ PGAIO_HCB_LOCAL_BUFFER_WRITEV,
} PgAioHandleCallbackID;
diff --git a/src/include/storage/bufmgr.h b/src/include/storage/bufmgr.h
index 784df8b00cb..ba9bf247ddb 100644
--- a/src/include/storage/bufmgr.h
+++ b/src/include/storage/bufmgr.h
@@ -172,7 +172,9 @@ extern PGDLLIMPORT int backend_flush_after;
extern PGDLLIMPORT int bgwriter_flush_after;
extern const PgAioHandleCallbacks aio_shared_buffer_readv_cb;
+extern const PgAioHandleCallbacks aio_shared_buffer_writev_cb;
extern const PgAioHandleCallbacks aio_local_buffer_readv_cb;
+extern const PgAioHandleCallbacks aio_local_buffer_writev_cb;
/* in buf_init.c */
extern PGDLLIMPORT char *BufferBlocks;
diff --git a/src/backend/storage/aio/aio_callback.c b/src/backend/storage/aio/aio_callback.c
index 03e44b76075..c49bf4a2384 100644
--- a/src/backend/storage/aio/aio_callback.c
+++ b/src/backend/storage/aio/aio_callback.c
@@ -44,8 +44,10 @@ static const PgAioHandleCallbacksEntry aio_handle_cbs[] = {
CALLBACK_ENTRY(PGAIO_HCB_MD_WRITEV, aio_md_writev_cb),
CALLBACK_ENTRY(PGAIO_HCB_SHARED_BUFFER_READV, aio_shared_buffer_readv_cb),
+ CALLBACK_ENTRY(PGAIO_HCB_SHARED_BUFFER_WRITEV, aio_shared_buffer_writev_cb),
CALLBACK_ENTRY(PGAIO_HCB_LOCAL_BUFFER_READV, aio_local_buffer_readv_cb),
+ CALLBACK_ENTRY(PGAIO_HCB_LOCAL_BUFFER_WRITEV, aio_local_buffer_writev_cb),
#undef CALLBACK_ENTRY
};
diff --git a/src/backend/storage/buffer/bufmgr.c b/src/backend/storage/buffer/bufmgr.c
index be306bfa4f2..94f3caf2224 100644
--- a/src/backend/storage/buffer/bufmgr.c
+++ b/src/backend/storage/buffer/bufmgr.c
@@ -5507,7 +5507,15 @@ LockBuffer(Buffer buffer, int mode)
else if (mode == BUFFER_LOCK_SHARE)
LWLockAcquire(BufferDescriptorGetContentLock(buf), LW_SHARED);
else if (mode == BUFFER_LOCK_EXCLUSIVE)
+ {
+ /*
+ * FIXME: Wait for AIO writes, otherwise there would be a risk of
+ * deadlock. This isn't entirely trivial to do in a race-free way, IO
+ * could be started between us checking whether there is IO and
+ * enqueueing ourselves for the lock.
+ */
LWLockAcquire(BufferDescriptorGetContentLock(buf), LW_EXCLUSIVE);
+ }
else
elog(ERROR, "unrecognized buffer lock mode: %d", mode);
}
@@ -5522,6 +5530,19 @@ ConditionalLockBuffer(Buffer buffer)
{
BufferDesc *buf;
+ /*
+ * FIXME: Wait for AIO writes. Some code does not deal well
+ * ConditionalLockBuffer() continuously failing, e.g.
+ * spginsert()->spgdoinsert() ends up busy-looping (easily reproducible by
+ * just making this function always fail and running the regression
+ * tests). While that code could be fixed, it'd be hard to find all
+ * problematic places.
+ *
+ * It would be OK to wait for the IO as waiting for IO completion does not
+ * need to wait for any locks that could lead to an undetected deadlock or
+ * such.
+ */
+
Assert(BufferIsPinned(buffer));
if (BufferIsLocal(buffer))
return true; /* act as though we got it */
@@ -5594,6 +5615,11 @@ LockBufferForCleanup(Buffer buffer)
{
uint32 buf_state;
+ /*
+ * FIXME: LockBuffer()'s handling of in-progress writes (once
+ * implemented) should suffice to deal with deadlock risk.
+ */
+
/* Try to acquire lock */
LockBuffer(buffer, BUFFER_LOCK_EXCLUSIVE);
buf_state = LockBufHdr(bufHdr);
@@ -5741,6 +5767,14 @@ ConditionalLockBufferForCleanup(Buffer buffer)
Assert(BufferIsValid(buffer));
+ /*
+ * FIXME: Should wait for IO for the same reason as in
+ * ConditionalLockBuffer(). Needs to happen before the
+ * ConditionalLockBuffer() call below, as we'd never reach the
+ * ConditionalLockBuffer() call due the buffer pin held for the duration
+ * of the IO.
+ */
+
if (BufferIsLocal(buffer))
{
refcount = LocalRefCount[-buffer - 1];
@@ -6843,12 +6877,131 @@ buffer_readv_report(PgAioResult result, const PgAioTargetData *target_data,
target_data->smgr.forkNum).str));
}
+/*
+ * Helper for AIO writev completion callbacks, supporting both shared and temp
+ * buffers. Gets called once for each buffer in a multi-page write.
+ */
+static pg_attribute_always_inline PgAioResult
+buffer_writev_complete_one(uint8 buf_off, Buffer buffer, uint8 flags,
+ bool failed, bool is_temp)
+{
+ BufferDesc *buf_hdr = is_temp ?
+ GetLocalBufferDescriptor(-buffer - 1)
+ : GetBufferDescriptor(buffer - 1);
+ PgAioResult result;
+ bool clear_dirty;
+ uint32 set_flag_bits;
+
+#ifdef USE_ASSERT_CHECKING
+ {
+ uint32 buf_state = pg_atomic_read_u32(&buf_hdr->state);
+
+ Assert(buf_state & BM_VALID);
+ Assert(buf_state & BM_TAG_VALID);
+ /* temp buffers don't use BM_IO_IN_PROGRESS */
+ if (!is_temp)
+ Assert(buf_state & BM_IO_IN_PROGRESS);
+ Assert(buf_state & BM_DIRTY);
+ }
+#endif
+
+ result.status = PGAIO_RS_OK;
+
+ clear_dirty = failed ? false : true;
+ set_flag_bits = failed ? BM_IO_ERROR : 0;
+
+ if (is_temp)
+ TerminateLocalBufferIO(buf_hdr, clear_dirty, set_flag_bits, false);
+ else
+ TerminateBufferIO(buf_hdr, clear_dirty, set_flag_bits, false, false);
+
+ /*
+ * The initiator of IO is not managing the lock (i.e. we called
+ * LWLockDisown()), we are.
+ */
+ if (!is_temp)
+ LWLockReleaseDisowned(BufferDescriptorGetContentLock(buf_hdr),
+ LW_SHARED);
+
+ /* FIXME: tracepoint */
+
+ return result;
+}
+
+/*
+ * Perform completion handling of a single AIO write. This write may cover
+ * multiple blocks / buffers.
+ *
+ * Shared between shared and local buffers, to reduce code duplication.
+ */
+static pg_attribute_always_inline PgAioResult
+buffer_writev_complete(PgAioHandle *ioh, PgAioResult prior_result,
+ uint8 cb_data, bool is_temp)
+{
+ PgAioResult result = prior_result;
+ PgAioTargetData *td = pgaio_io_get_target_data(ioh);
+ uint64 *io_data;
+ uint8 handle_data_len;
+
+ if (is_temp)
+ {
+ Assert(td->smgr.is_temp);
+ Assert(pgaio_io_get_owner(ioh) == MyProcNumber);
+ }
+ else
+ Assert(!td->smgr.is_temp);
+
+ /*
+ * Iterate over all the buffers affected by this IO and call appropriate
+ * per-buffer completion function for each buffer.
+ */
+ io_data = pgaio_io_get_handle_data(ioh, &handle_data_len);
+ for (uint8 buf_off = 0; buf_off < handle_data_len; buf_off++)
+ {
+ Buffer buf = io_data[buf_off];
+ PgAioResult buf_result;
+ bool failed;
+
+ Assert(BufferIsValid(buf));
+
+ /*
+ * If the entire failed on a lower-level, each buffer needs to be
+ * marked as failed. In case of a partial read, some buffers may be
+ * ok.
+ */
+ failed =
+ prior_result.status == PGAIO_RS_ERROR
+ || prior_result.result <= buf_off;
+
+ buf_result = buffer_writev_complete_one(buf_off, buf, cb_data, failed,
+ is_temp);
+
+ /*
+ * If there wasn't any prior error and the IO for this page failed in
+ * some form, set the whole IO's to the page's result.
+ */
+ if (result.status != PGAIO_RS_ERROR && buf_result.status != PGAIO_RS_OK)
+ {
+ result = buf_result;
+ pgaio_result_report(result, td, LOG);
+ }
+ }
+
+ return result;
+}
+
static void
shared_buffer_readv_stage(PgAioHandle *ioh, uint8 cb_data)
{
buffer_stage_common(ioh, false, false);
}
+static void
+shared_buffer_writev_stage(PgAioHandle *ioh, uint8 cb_data)
+{
+ buffer_stage_common(ioh, true, false);
+}
+
static PgAioResult
shared_buffer_readv_complete(PgAioHandle *ioh, PgAioResult prior_result,
uint8 cb_data)
@@ -6856,6 +7009,13 @@ shared_buffer_readv_complete(PgAioHandle *ioh, PgAioResult prior_result,
return buffer_readv_complete(ioh, prior_result, cb_data, false);
}
+static PgAioResult
+shared_buffer_writev_complete(PgAioHandle *ioh, PgAioResult prior_result,
+ uint8 cb_data)
+{
+ return buffer_writev_complete(ioh, prior_result, cb_data, false);
+}
+
static void
local_buffer_readv_stage(PgAioHandle *ioh, uint8 cb_data)
{
@@ -6869,6 +7029,17 @@ local_buffer_readv_complete(PgAioHandle *ioh, PgAioResult prior_result,
return buffer_readv_complete(ioh, prior_result, cb_data, true);
}
+static void
+local_buffer_writev_stage(PgAioHandle *ioh, uint8 cb_data)
+{
+ /*
+ * Currently this is unreachable as the only write support is for
+ * checkpointer / bgwriter, which don't deal with local buffers.
+ */
+ elog(ERROR, "should be unreachable");
+}
+
+
/* readv callback is passed READ_BUFFERS_* flags as callback data */
const PgAioHandleCallbacks aio_shared_buffer_readv_cb = {
.stage = shared_buffer_readv_stage,
@@ -6876,6 +7047,11 @@ const PgAioHandleCallbacks aio_shared_buffer_readv_cb = {
.report = buffer_readv_report,
};
+const PgAioHandleCallbacks aio_shared_buffer_writev_cb = {
+ .stage = shared_buffer_writev_stage,
+ .complete_shared = shared_buffer_writev_complete,
+};
+
/* readv callback is passed READ_BUFFERS_* flags as callback data */
const PgAioHandleCallbacks aio_local_buffer_readv_cb = {
.stage = local_buffer_readv_stage,
@@ -6889,3 +7065,7 @@ const PgAioHandleCallbacks aio_local_buffer_readv_cb = {
.complete_local = local_buffer_readv_complete,
.report = buffer_readv_report,
};
+
+const PgAioHandleCallbacks aio_local_buffer_writev_cb = {
+ .stage = local_buffer_writev_stage,
+};
--
2.48.1.76.g4e746b1a31.dirty
[text/x-diff] v2.11-0022-aio-Add-IO-queue-helper.patch (7.4K, ../../6ak556uyqiptdwjaci4kbi5eykwkmzqgkbtkyaosjnopjhncrc@2v4ac2jwyz22/23-v2.11-0022-aio-Add-IO-queue-helper.patch)
download | inline diff:
From 90d1cada1a2782a6bf06c0da69860ddf3c589185 Mon Sep 17 00:00:00 2001
From: Andres Freund <andres@anarazel.de>
Date: Tue, 18 Mar 2025 14:40:06 -0400
Subject: [PATCH v2.11 22/27] aio: Add IO queue helper
This is likely never going to anywhere - Thomas Munro is working on something
more complete. But I needed a way to exercise aio for checkpointer / bgwriter.
---
src/include/storage/io_queue.h | 33 +++++
src/backend/storage/aio/Makefile | 1 +
src/backend/storage/aio/io_queue.c | 204 ++++++++++++++++++++++++++++
src/backend/storage/aio/meson.build | 1 +
src/tools/pgindent/typedefs.list | 2 +
5 files changed, 241 insertions(+)
create mode 100644 src/include/storage/io_queue.h
create mode 100644 src/backend/storage/aio/io_queue.c
diff --git a/src/include/storage/io_queue.h b/src/include/storage/io_queue.h
new file mode 100644
index 00000000000..92b1e9afe6f
--- /dev/null
+++ b/src/include/storage/io_queue.h
@@ -0,0 +1,33 @@
+/*-------------------------------------------------------------------------
+ *
+ * io_queue.h
+ * Mechanism for tracking many IOs
+ *
+ *
+ * Portions Copyright (c) 1996-2025, PostgreSQL Global Development Group
+ * Portions Copyright (c) 1994, Regents of the University of California
+ *
+ * src/include/storage/io_queue.h
+ *
+ *-------------------------------------------------------------------------
+ */
+#ifndef IO_QUEUE_H
+#define IO_QUEUE_H
+
+#include "storage/aio_types.h"
+
+struct IOQueue;
+typedef struct IOQueue IOQueue;
+
+struct PgAioWaitRef;
+
+extern IOQueue *io_queue_create(int depth, int flags);
+extern void io_queue_track(IOQueue *ioq, const PgAioWaitRef *iow);
+extern void io_queue_wait_one(IOQueue *ioq);
+extern void io_queue_wait_all(IOQueue *ioq);
+extern bool io_queue_is_empty(IOQueue *ioq);
+extern void io_queue_reserve(IOQueue *ioq);
+extern PgAioHandle *io_queue_acquire_io(IOQueue *ioq);
+extern void io_queue_free(IOQueue *ioq);
+
+#endif /* IO_QUEUE_H */
diff --git a/src/backend/storage/aio/Makefile b/src/backend/storage/aio/Makefile
index 3f2469cc399..86fa4276fda 100644
--- a/src/backend/storage/aio/Makefile
+++ b/src/backend/storage/aio/Makefile
@@ -15,6 +15,7 @@ OBJS = \
aio_init.o \
aio_io.o \
aio_target.o \
+ io_queue.o \
method_io_uring.o \
method_sync.o \
method_worker.o \
diff --git a/src/backend/storage/aio/io_queue.c b/src/backend/storage/aio/io_queue.c
new file mode 100644
index 00000000000..526aa1d5e06
--- /dev/null
+++ b/src/backend/storage/aio/io_queue.c
@@ -0,0 +1,204 @@
+/*-------------------------------------------------------------------------
+ *
+ * io_queue.c
+ * AIO - Mechanism for tracking many IOs
+ *
+ * Portions Copyright (c) 1996-2025, PostgreSQL Global Development Group
+ * Portions Copyright (c) 1994, Regents of the University of California
+ *
+ * IDENTIFICATION
+ * src/backend/storage/aio/io_queue.c
+ *
+ *-------------------------------------------------------------------------
+ */
+
+#include "postgres.h"
+
+#include "lib/ilist.h"
+#include "storage/aio.h"
+#include "storage/io_queue.h"
+#include "utils/resowner.h"
+
+
+
+typedef struct TrackedIO
+{
+ PgAioWaitRef iow;
+ dlist_node node;
+} TrackedIO;
+
+struct IOQueue
+{
+ int depth;
+ int unsubmitted;
+
+ bool has_reserved;
+
+ dclist_head idle;
+ dclist_head in_progress;
+
+ TrackedIO tracked_ios[FLEXIBLE_ARRAY_MEMBER];
+};
+
+
+IOQueue *
+io_queue_create(int depth, int flags)
+{
+ size_t sz;
+ IOQueue *ioq;
+
+ sz = offsetof(IOQueue, tracked_ios)
+ + sizeof(TrackedIO) * depth;
+
+ ioq = palloc0(sz);
+
+ ioq->depth = 0;
+
+ for (int i = 0; i < depth; i++)
+ {
+ TrackedIO *tio = &ioq->tracked_ios[i];
+
+ pgaio_wref_clear(&tio->iow);
+ dclist_push_tail(&ioq->idle, &tio->node);
+ }
+
+ return ioq;
+}
+
+void
+io_queue_wait_one(IOQueue *ioq)
+{
+ /* submit all pending IO before waiting */
+ pgaio_submit_staged();
+
+ while (!dclist_is_empty(&ioq->in_progress))
+ {
+ /* FIXME: Should we really pop here already? */
+ dlist_node *node = dclist_pop_head_node(&ioq->in_progress);
+ TrackedIO *tio = dclist_container(TrackedIO, node, node);
+
+ pgaio_wref_wait(&tio->iow);
+ dclist_push_head(&ioq->idle, &tio->node);
+ }
+}
+
+void
+io_queue_reserve(IOQueue *ioq)
+{
+ if (ioq->has_reserved)
+ return;
+
+ if (dclist_is_empty(&ioq->idle))
+ io_queue_wait_one(ioq);
+
+ Assert(!dclist_is_empty(&ioq->idle));
+
+ ioq->has_reserved = true;
+}
+
+PgAioHandle *
+io_queue_acquire_io(IOQueue *ioq)
+{
+ PgAioHandle *ioh;
+
+ io_queue_reserve(ioq);
+
+ Assert(!dclist_is_empty(&ioq->idle));
+
+ if (!io_queue_is_empty(ioq))
+ {
+ ioh = pgaio_io_acquire_nb(CurrentResourceOwner, NULL);
+ if (ioh == NULL)
+ {
+ /*
+ * Need to wait for all IOs, blocking might not be legal in the
+ * context.
+ *
+ * XXX: This doesn't make a whole lot of sense, we're also
+ * blocking here. What was I smoking when I wrote the above?
+ */
+ io_queue_wait_all(ioq);
+ ioh = pgaio_io_acquire(CurrentResourceOwner, NULL);
+ }
+ }
+ else
+ {
+ ioh = pgaio_io_acquire(CurrentResourceOwner, NULL);
+ }
+
+ return ioh;
+}
+
+void
+io_queue_track(IOQueue *ioq, const struct PgAioWaitRef *iow)
+{
+ dlist_node *node;
+ TrackedIO *tio;
+
+ Assert(ioq->has_reserved);
+ ioq->has_reserved = false;
+
+ Assert(!dclist_is_empty(&ioq->idle));
+
+ node = dclist_pop_head_node(&ioq->idle);
+ tio = dclist_container(TrackedIO, node, node);
+
+ tio->iow = *iow;
+
+ dclist_push_tail(&ioq->in_progress, &tio->node);
+
+ ioq->unsubmitted++;
+
+ /*
+ * XXX: Should have some smarter logic here. We don't want to wait too
+ * long to submit, that'll mean we're more likely to block. But we also
+ * don't want to have the overhead of submitting every IO individually.
+ */
+ if (ioq->unsubmitted >= 4)
+ {
+ pgaio_submit_staged();
+ ioq->unsubmitted = 0;
+ }
+}
+
+void
+io_queue_wait_all(IOQueue *ioq)
+{
+ /* submit all pending IO before waiting */
+ pgaio_submit_staged();
+
+ while (!dclist_is_empty(&ioq->in_progress))
+ {
+ /* wait for the last IO to minimize unnecessary wakeups */
+ dlist_node *node = dclist_tail_node(&ioq->in_progress);
+ TrackedIO *tio = dclist_container(TrackedIO, node, node);
+
+ if (!pgaio_wref_check_done(&tio->iow))
+ {
+ ereport(DEBUG3,
+ errmsg("io_queue_wait_all for io:%d",
+ pgaio_wref_get_id(&tio->iow)),
+ errhidestmt(true),
+ errhidecontext(true));
+
+ pgaio_wref_wait(&tio->iow);
+ }
+
+ dclist_delete_from(&ioq->in_progress, &tio->node);
+ dclist_push_head(&ioq->idle, &tio->node);
+ }
+}
+
+bool
+io_queue_is_empty(IOQueue *ioq)
+{
+ return dclist_is_empty(&ioq->in_progress);
+}
+
+void
+io_queue_free(IOQueue *ioq)
+{
+ io_queue_wait_all(ioq);
+
+ pfree(ioq);
+}
diff --git a/src/backend/storage/aio/meson.build b/src/backend/storage/aio/meson.build
index da6df2d3654..270c4a64428 100644
--- a/src/backend/storage/aio/meson.build
+++ b/src/backend/storage/aio/meson.build
@@ -7,6 +7,7 @@ backend_sources += files(
'aio_init.c',
'aio_io.c',
'aio_target.c',
+ 'io_queue.c',
'method_io_uring.c',
'method_sync.c',
'method_worker.c',
diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list
index d633c732162..9c07fa04b49 100644
--- a/src/tools/pgindent/typedefs.list
+++ b/src/tools/pgindent/typedefs.list
@@ -1191,6 +1191,7 @@ IOContext
IOFuncSelector
IOObject
IOOp
+IOQueue
IO_STATUS_BLOCK
IPCompareMethod
ITEM
@@ -3016,6 +3017,7 @@ TocEntry
TokenAuxData
TokenizedAuthLine
TrackItem
+TrackedIO
TransApplyAction
TransInvalidationInfo
TransState
--
2.48.1.76.g4e746b1a31.dirty
[text/x-diff] v2.11-0023-bufmgr-use-AIO-in-checkpointer-bgwriter.patch (31.3K, ../../6ak556uyqiptdwjaci4kbi5eykwkmzqgkbtkyaosjnopjhncrc@2v4ac2jwyz22/24-v2.11-0023-bufmgr-use-AIO-in-checkpointer-bgwriter.patch)
download | inline diff:
From 920a84a3fbe31183a01a711dc4fc68ba5a57497c Mon Sep 17 00:00:00 2001
From: Andres Freund <andres@anarazel.de>
Date: Tue, 18 Mar 2025 14:40:06 -0400
Subject: [PATCH v2.11 23/27] bufmgr: use AIO in checkpointer, bgwriter
This is far from ready - just included to be able to exercise AIO writes and
get some preliminary numbers. In all likelihood this will instead be based
on-top of work by Thomas Munro instead of the preceding commit.
---
src/include/postmaster/bgwriter.h | 3 +-
src/include/storage/buf_internals.h | 2 +
src/include/storage/bufmgr.h | 3 +-
src/include/storage/bufpage.h | 1 +
src/backend/postmaster/bgwriter.c | 19 +-
src/backend/postmaster/checkpointer.c | 11 +-
src/backend/storage/buffer/bufmgr.c | 594 +++++++++++++++++++++++---
src/backend/storage/page/bufpage.c | 10 +
src/tools/pgindent/typedefs.list | 1 +
9 files changed, 586 insertions(+), 58 deletions(-)
diff --git a/src/include/postmaster/bgwriter.h b/src/include/postmaster/bgwriter.h
index 800ecbfd13b..a8081d411b6 100644
--- a/src/include/postmaster/bgwriter.h
+++ b/src/include/postmaster/bgwriter.h
@@ -31,7 +31,8 @@ pg_noreturn extern void BackgroundWriterMain(const void *startup_data, size_t st
pg_noreturn extern void CheckpointerMain(const void *startup_data, size_t startup_data_len);
extern void RequestCheckpoint(int flags);
-extern void CheckpointWriteDelay(int flags, double progress);
+struct IOQueue;
+extern void CheckpointWriteDelay(struct IOQueue *ioq, int flags, double progress);
extern bool ForwardSyncRequest(const FileTag *ftag, SyncRequestType type);
diff --git a/src/include/storage/buf_internals.h b/src/include/storage/buf_internals.h
index 4b1f52a9fc8..a3df3192d12 100644
--- a/src/include/storage/buf_internals.h
+++ b/src/include/storage/buf_internals.h
@@ -21,6 +21,8 @@
#include "storage/buf.h"
#include "storage/bufmgr.h"
#include "storage/condition_variable.h"
+#include "storage/io_queue.h"
+#include "storage/latch.h"
#include "storage/lwlock.h"
#include "storage/procnumber.h"
#include "storage/shmem.h"
diff --git a/src/include/storage/bufmgr.h b/src/include/storage/bufmgr.h
index ba9bf247ddb..af5035317b7 100644
--- a/src/include/storage/bufmgr.h
+++ b/src/include/storage/bufmgr.h
@@ -295,7 +295,8 @@ extern bool ConditionalLockBufferForCleanup(Buffer buffer);
extern bool IsBufferCleanupOK(Buffer buffer);
extern bool HoldingBufferPinThatDelaysRecovery(void);
-extern bool BgBufferSync(struct WritebackContext *wb_context);
+struct IOQueue;
+extern bool BgBufferSync(struct IOQueue *ioq, struct WritebackContext *wb_context);
extern uint32 GetPinLimit(void);
extern uint32 GetLocalPinLimit(void);
diff --git a/src/include/storage/bufpage.h b/src/include/storage/bufpage.h
index 6646b6f6371..9c045e81857 100644
--- a/src/include/storage/bufpage.h
+++ b/src/include/storage/bufpage.h
@@ -509,5 +509,6 @@ extern bool PageIndexTupleOverwrite(Page page, OffsetNumber offnum,
Item newtup, Size newsize);
extern char *PageSetChecksumCopy(Page page, BlockNumber blkno);
extern void PageSetChecksumInplace(Page page, BlockNumber blkno);
+extern bool PageNeedsChecksumCopy(Page page);
#endif /* BUFPAGE_H */
diff --git a/src/backend/postmaster/bgwriter.c b/src/backend/postmaster/bgwriter.c
index 72f5acceec7..6e8801a39e3 100644
--- a/src/backend/postmaster/bgwriter.c
+++ b/src/backend/postmaster/bgwriter.c
@@ -38,11 +38,13 @@
#include "postmaster/auxprocess.h"
#include "postmaster/bgwriter.h"
#include "postmaster/interrupt.h"
+#include "storage/aio.h"
#include "storage/aio_subsys.h"
#include "storage/buf_internals.h"
#include "storage/bufmgr.h"
#include "storage/condition_variable.h"
#include "storage/fd.h"
+#include "storage/io_queue.h"
#include "storage/lwlock.h"
#include "storage/proc.h"
#include "storage/procsignal.h"
@@ -90,6 +92,7 @@ BackgroundWriterMain(const void *startup_data, size_t startup_data_len)
sigjmp_buf local_sigjmp_buf;
MemoryContext bgwriter_context;
bool prev_hibernate;
+ IOQueue *ioq;
WritebackContext wb_context;
Assert(startup_data_len == 0);
@@ -131,6 +134,7 @@ BackgroundWriterMain(const void *startup_data, size_t startup_data_len)
ALLOCSET_DEFAULT_SIZES);
MemoryContextSwitchTo(bgwriter_context);
+ ioq = io_queue_create(128, 0);
WritebackContextInit(&wb_context, &bgwriter_flush_after);
/*
@@ -228,12 +232,22 @@ BackgroundWriterMain(const void *startup_data, size_t startup_data_len)
/* Clear any already-pending wakeups */
ResetLatch(MyLatch);
+ /*
+ * FIXME: this is theoretically racy, but I didn't want to copy
+ * ProcessMainLoopInterrupts() remaining body here.
+ */
+ if (ShutdownRequestPending)
+ {
+ io_queue_wait_all(ioq);
+ io_queue_free(ioq);
+ }
+
ProcessMainLoopInterrupts();
/*
* Do one cycle of dirty-buffer writing.
*/
- can_hibernate = BgBufferSync(&wb_context);
+ can_hibernate = BgBufferSync(ioq, &wb_context);
/* Report pending statistics to the cumulative stats system */
pgstat_report_bgwriter();
@@ -250,6 +264,9 @@ BackgroundWriterMain(const void *startup_data, size_t startup_data_len)
smgrdestroyall();
}
+ /* finish IO before sleeping, to avoid blocking other backends */
+ io_queue_wait_all(ioq);
+
/*
* Log a new xl_running_xacts every now and then so replication can
* get into a consistent state faster (think of suboverflowed
diff --git a/src/backend/postmaster/checkpointer.c b/src/backend/postmaster/checkpointer.c
index fda91ffd1ce..904fe167eb4 100644
--- a/src/backend/postmaster/checkpointer.c
+++ b/src/backend/postmaster/checkpointer.c
@@ -49,10 +49,12 @@
#include "postmaster/bgwriter.h"
#include "postmaster/interrupt.h"
#include "replication/syncrep.h"
+#include "storage/aio.h"
#include "storage/aio_subsys.h"
#include "storage/bufmgr.h"
#include "storage/condition_variable.h"
#include "storage/fd.h"
+#include "storage/io_queue.h"
#include "storage/ipc.h"
#include "storage/lwlock.h"
#include "storage/pmsignal.h"
@@ -766,7 +768,7 @@ ImmediateCheckpointRequested(void)
* fraction between 0.0 meaning none, and 1.0 meaning all done.
*/
void
-CheckpointWriteDelay(int flags, double progress)
+CheckpointWriteDelay(IOQueue *ioq, int flags, double progress)
{
static int absorb_counter = WRITES_PER_ABSORB;
@@ -800,6 +802,13 @@ CheckpointWriteDelay(int flags, double progress)
/* Report interim statistics to the cumulative stats system */
pgstat_report_checkpointer();
+ /*
+ * Ensure all pending IO is submitted to avoid unnecessary delays for
+ * other processes.
+ */
+ io_queue_wait_all(ioq);
+
+
/*
* This sleep used to be connected to bgwriter_delay, typically 200ms.
* That resulted in more frequent wakeups if not much work to do.
diff --git a/src/backend/storage/buffer/bufmgr.c b/src/backend/storage/buffer/bufmgr.c
index 94f3caf2224..0909888c342 100644
--- a/src/backend/storage/buffer/bufmgr.c
+++ b/src/backend/storage/buffer/bufmgr.c
@@ -52,6 +52,7 @@
#include "storage/buf_internals.h"
#include "storage/bufmgr.h"
#include "storage/fd.h"
+#include "storage/io_queue.h"
#include "storage/ipc.h"
#include "storage/lmgr.h"
#include "storage/proc.h"
@@ -76,6 +77,7 @@
/* Bits in SyncOneBuffer's return value */
#define BUF_WRITTEN 0x01
#define BUF_REUSABLE 0x02
+#define BUF_CANT_MERGE 0x04
#define RELS_BSEARCH_THRESHOLD 20
@@ -515,8 +517,6 @@ static void UnpinBuffer(BufferDesc *buf);
static void UnpinBufferNoOwner(BufferDesc *buf);
static void BufferSync(int flags);
static uint32 WaitBufHdrUnlocked(BufferDesc *buf);
-static int SyncOneBuffer(int buf_id, bool skip_recently_used,
- WritebackContext *wb_context);
static void WaitIO(BufferDesc *buf);
static void AbortBufferIO(Buffer buffer);
static void shared_buffer_write_error_callback(void *arg);
@@ -532,6 +532,7 @@ static void CheckReadBuffersOperation(ReadBuffersOperation *operation, bool is_c
static Buffer GetVictimBuffer(BufferAccessStrategy strategy, IOContext io_context);
static void FlushBuffer(BufferDesc *buf, SMgrRelation reln,
IOObject io_object, IOContext io_context);
+
static void FindAndDropRelationBuffers(RelFileLocator rlocator,
ForkNumber forkNum,
BlockNumber nForkBlock,
@@ -3298,6 +3299,57 @@ UnpinBufferNoOwner(BufferDesc *buf)
}
}
+typedef struct BuffersToWrite
+{
+ int nbuffers;
+ BufferTag start_at_tag;
+ uint32 max_combine;
+
+ XLogRecPtr max_lsn;
+
+ PgAioHandle *ioh;
+ PgAioWaitRef iow;
+
+ uint64 total_writes;
+
+ Buffer buffers[IOV_MAX];
+ PgAioBounceBuffer *bounce_buffers[IOV_MAX];
+ const void *data_ptrs[IOV_MAX];
+} BuffersToWrite;
+
+static int PrepareToWriteBuffer(BuffersToWrite *to_write, Buffer buf,
+ bool skip_recently_used,
+ IOQueue *ioq, WritebackContext *wb_context);
+
+static void WriteBuffers(BuffersToWrite *to_write,
+ IOQueue *ioq, WritebackContext *wb_context);
+
+static void
+BuffersToWriteInit(BuffersToWrite *to_write,
+ IOQueue *ioq, WritebackContext *wb_context)
+{
+ to_write->total_writes = 0;
+ to_write->nbuffers = 0;
+ to_write->ioh = NULL;
+ pgaio_wref_clear(&to_write->iow);
+ to_write->max_lsn = InvalidXLogRecPtr;
+
+ pgaio_enter_batchmode();
+}
+
+static void
+BuffersToWriteEnd(BuffersToWrite *to_write)
+{
+ if (to_write->ioh != NULL)
+ {
+ pgaio_io_release(to_write->ioh);
+ to_write->ioh = NULL;
+ }
+
+ pgaio_exit_batchmode();
+}
+
+
#define ST_SORT sort_checkpoint_bufferids
#define ST_ELEMENT_TYPE CkptSortItem
#define ST_COMPARE(a, b) ckpt_buforder_comparator(a, b)
@@ -3329,7 +3381,10 @@ BufferSync(int flags)
binaryheap *ts_heap;
int i;
int mask = BM_DIRTY;
+ IOQueue *ioq;
WritebackContext wb_context;
+ BuffersToWrite to_write;
+ int max_combine;
/*
* Unless this is a shutdown checkpoint or we have been explicitly told,
@@ -3391,7 +3446,9 @@ BufferSync(int flags)
if (num_to_scan == 0)
return; /* nothing to do */
+ ioq = io_queue_create(512, 0);
WritebackContextInit(&wb_context, &checkpoint_flush_after);
+ max_combine = Min(io_bounce_buffers, io_combine_limit);
TRACE_POSTGRESQL_BUFFER_SYNC_START(NBuffers, num_to_scan);
@@ -3499,48 +3556,91 @@ BufferSync(int flags)
*/
num_processed = 0;
num_written = 0;
+
+ BuffersToWriteInit(&to_write, ioq, &wb_context);
+
while (!binaryheap_empty(ts_heap))
{
BufferDesc *bufHdr = NULL;
CkptTsStatus *ts_stat = (CkptTsStatus *)
DatumGetPointer(binaryheap_first(ts_heap));
+ bool batch_continue = true;
- buf_id = CkptBufferIds[ts_stat->index].buf_id;
- Assert(buf_id != -1);
-
- bufHdr = GetBufferDescriptor(buf_id);
-
- num_processed++;
+ Assert(ts_stat->num_scanned <= ts_stat->num_to_scan);
/*
- * We don't need to acquire the lock here, because we're only looking
- * at a single bit. It's possible that someone else writes the buffer
- * and clears the flag right after we check, but that doesn't matter
- * since SyncOneBuffer will then do nothing. However, there is a
- * further race condition: it's conceivable that between the time we
- * examine the bit here and the time SyncOneBuffer acquires the lock,
- * someone else not only wrote the buffer but replaced it with another
- * page and dirtied it. In that improbable case, SyncOneBuffer will
- * write the buffer though we didn't need to. It doesn't seem worth
- * guarding against this, though.
+ * Collect a batch of buffers to write out from the current
+ * tablespace. That causes some imbalance between the tablespaces, but
+ * that's more than outweighed by the efficiency gain due to batching.
*/
- if (pg_atomic_read_u32(&bufHdr->state) & BM_CHECKPOINT_NEEDED)
+ while (batch_continue &&
+ to_write.nbuffers < max_combine &&
+ ts_stat->num_scanned < ts_stat->num_to_scan)
{
- if (SyncOneBuffer(buf_id, false, &wb_context) & BUF_WRITTEN)
+ buf_id = CkptBufferIds[ts_stat->index].buf_id;
+ Assert(buf_id != -1);
+
+ bufHdr = GetBufferDescriptor(buf_id);
+
+ num_processed++;
+
+ /*
+ * We don't need to acquire the lock here, because we're only
+ * looking at a single bit. It's possible that someone else writes
+ * the buffer and clears the flag right after we check, but that
+ * doesn't matter since SyncOneBuffer will then do nothing.
+ * However, there is a further race condition: it's conceivable
+ * that between the time we examine the bit here and the time
+ * SyncOneBuffer acquires the lock, someone else not only wrote
+ * the buffer but replaced it with another page and dirtied it. In
+ * that improbable case, SyncOneBuffer will write the buffer
+ * though we didn't need to. It doesn't seem worth guarding
+ * against this, though.
+ */
+ if (pg_atomic_read_u32(&bufHdr->state) & BM_CHECKPOINT_NEEDED)
{
- TRACE_POSTGRESQL_BUFFER_SYNC_WRITTEN(buf_id);
- PendingCheckpointerStats.buffers_written++;
- num_written++;
+ int result = PrepareToWriteBuffer(&to_write, buf_id + 1, false,
+ ioq, &wb_context);
+
+ if (result & BUF_CANT_MERGE)
+ {
+ Assert(to_write.nbuffers > 0);
+ WriteBuffers(&to_write, ioq, &wb_context);
+
+ result = PrepareToWriteBuffer(&to_write, buf_id + 1, false,
+ ioq, &wb_context);
+ Assert(result != BUF_CANT_MERGE);
+ }
+
+ if (result & BUF_WRITTEN)
+ {
+ TRACE_POSTGRESQL_BUFFER_SYNC_WRITTEN(buf_id);
+ PendingCheckpointerStats.buffers_written++;
+ num_written++;
+ }
+ else
+ {
+ batch_continue = false;
+ }
}
+ else
+ {
+ if (to_write.nbuffers > 0)
+ WriteBuffers(&to_write, ioq, &wb_context);
+ }
+
+ /*
+ * Measure progress independent of actually having to flush the
+ * buffer - otherwise writing become unbalanced.
+ */
+ ts_stat->progress += ts_stat->progress_slice;
+ ts_stat->num_scanned++;
+ ts_stat->index++;
}
- /*
- * Measure progress independent of actually having to flush the buffer
- * - otherwise writing become unbalanced.
- */
- ts_stat->progress += ts_stat->progress_slice;
- ts_stat->num_scanned++;
- ts_stat->index++;
+ if (to_write.nbuffers > 0)
+ WriteBuffers(&to_write, ioq, &wb_context);
+
/* Have all the buffers from the tablespace been processed? */
if (ts_stat->num_scanned == ts_stat->num_to_scan)
@@ -3558,15 +3658,23 @@ BufferSync(int flags)
*
* (This will check for barrier events even if it doesn't sleep.)
*/
- CheckpointWriteDelay(flags, (double) num_processed / num_to_scan);
+ CheckpointWriteDelay(ioq, flags, (double) num_processed / num_to_scan);
}
+ Assert(to_write.nbuffers == 0);
+ io_queue_wait_all(ioq);
+
/*
* Issue all pending flushes. Only checkpointer calls BufferSync(), so
* IOContext will always be IOCONTEXT_NORMAL.
*/
IssuePendingWritebacks(&wb_context, IOCONTEXT_NORMAL);
+ io_queue_wait_all(ioq); /* IssuePendingWritebacks might have added
+ * more */
+ io_queue_free(ioq);
+ BuffersToWriteEnd(&to_write);
+
pfree(per_ts_stat);
per_ts_stat = NULL;
binaryheap_free(ts_heap);
@@ -3592,7 +3700,7 @@ BufferSync(int flags)
* bgwriter_lru_maxpages to 0.)
*/
bool
-BgBufferSync(WritebackContext *wb_context)
+BgBufferSync(IOQueue *ioq, WritebackContext *wb_context)
{
/* info obtained from freelist.c */
int strategy_buf_id;
@@ -3635,6 +3743,9 @@ BgBufferSync(WritebackContext *wb_context)
long new_strategy_delta;
uint32 new_recent_alloc;
+ BuffersToWrite to_write;
+ int max_combine;
+
/*
* Find out where the freelist clock sweep currently is, and how many
* buffer allocations have happened since our last call.
@@ -3655,6 +3766,8 @@ BgBufferSync(WritebackContext *wb_context)
return true;
}
+ max_combine = Min(io_bounce_buffers, io_combine_limit);
+
/*
* Compute strategy_delta = how many buffers have been scanned by the
* clock sweep since last time. If first time through, assume none. Then
@@ -3811,11 +3924,25 @@ BgBufferSync(WritebackContext *wb_context)
num_written = 0;
reusable_buffers = reusable_buffers_est;
+ BuffersToWriteInit(&to_write, ioq, wb_context);
+
/* Execute the LRU scan */
while (num_to_scan > 0 && reusable_buffers < upcoming_alloc_est)
{
- int sync_state = SyncOneBuffer(next_to_clean, true,
- wb_context);
+ int sync_state;
+
+ sync_state = PrepareToWriteBuffer(&to_write, next_to_clean + 1,
+ true, ioq, wb_context);
+ if (sync_state & BUF_CANT_MERGE)
+ {
+ Assert(to_write.nbuffers > 0);
+
+ WriteBuffers(&to_write, ioq, wb_context);
+
+ sync_state = PrepareToWriteBuffer(&to_write, next_to_clean + 1,
+ true, ioq, wb_context);
+ Assert(sync_state != BUF_CANT_MERGE);
+ }
if (++next_to_clean >= NBuffers)
{
@@ -3826,6 +3953,13 @@ BgBufferSync(WritebackContext *wb_context)
if (sync_state & BUF_WRITTEN)
{
+ Assert(sync_state & BUF_REUSABLE);
+
+ if (to_write.nbuffers == max_combine)
+ {
+ WriteBuffers(&to_write, ioq, wb_context);
+ }
+
reusable_buffers++;
if (++num_written >= bgwriter_lru_maxpages)
{
@@ -3837,6 +3971,11 @@ BgBufferSync(WritebackContext *wb_context)
reusable_buffers++;
}
+ if (to_write.nbuffers > 0)
+ WriteBuffers(&to_write, ioq, wb_context);
+
+ BuffersToWriteEnd(&to_write);
+
PendingBgWriterStats.buf_written_clean += num_written;
#ifdef BGW_DEBUG
@@ -3875,8 +4014,66 @@ BgBufferSync(WritebackContext *wb_context)
return (bufs_to_lap == 0 && recent_alloc == 0);
}
+static inline bool
+BufferTagsSameRel(const BufferTag *tag1, const BufferTag *tag2)
+{
+ return (tag1->spcOid == tag2->spcOid) &&
+ (tag1->dbOid == tag2->dbOid) &&
+ (tag1->relNumber == tag2->relNumber) &&
+ (tag1->forkNum == tag2->forkNum)
+ ;
+}
+
+static bool
+CanMergeWrite(BuffersToWrite *to_write, BufferDesc *cur_buf_hdr)
+{
+ BlockNumber cur_block = cur_buf_hdr->tag.blockNum;
+
+ Assert(to_write->nbuffers > 0); /* can't merge with nothing */
+ Assert(to_write->start_at_tag.relNumber != InvalidOid);
+ Assert(to_write->start_at_tag.blockNum != InvalidBlockNumber);
+
+ Assert(to_write->ioh != NULL);
+
+ /*
+ * First check if the blocknumber is one that we could actually merge,
+ * that's cheaper than checking the tablespace/db/relnumber/fork match.
+ */
+ if (to_write->start_at_tag.blockNum + to_write->nbuffers != cur_block)
+ return false;
+
+ if (!BufferTagsSameRel(&to_write->start_at_tag, &cur_buf_hdr->tag))
+ return false;
+
+ /*
+ * Need to check with smgr how large a write we're allowed to make. To
+ * reduce the overhead of the smgr check, only inquire once, when
+ * processing the first to-be-merged buffer. That avoids the overhead in
+ * the common case of writing out buffers that definitely not mergeable.
+ */
+ if (to_write->nbuffers == 1)
+ {
+ SMgrRelation smgr;
+
+ smgr = smgropen(BufTagGetRelFileLocator(&to_write->start_at_tag), INVALID_PROC_NUMBER);
+
+ to_write->max_combine = smgrmaxcombine(smgr,
+ to_write->start_at_tag.forkNum,
+ to_write->start_at_tag.blockNum);
+ }
+ else
+ {
+ Assert(to_write->max_combine > 0);
+ }
+
+ if (to_write->start_at_tag.blockNum + to_write->max_combine <= cur_block)
+ return false;
+
+ return true;
+}
+
/*
- * SyncOneBuffer -- process a single buffer during syncing.
+ * PrepareToWriteBuffer -- process a single buffer during syncing.
*
* If skip_recently_used is true, we don't write currently-pinned buffers, nor
* buffers marked recently used, as these are not replacement candidates.
@@ -3885,22 +4082,50 @@ BgBufferSync(WritebackContext *wb_context)
* BUF_WRITTEN: we wrote the buffer.
* BUF_REUSABLE: buffer is available for replacement, ie, it has
* pin count 0 and usage count 0.
+ * BUF_CANT_MERGE: can't combine this write with prior writes, caller needs
+ * to issue those first
*
* (BUF_WRITTEN could be set in error if FlushBuffer finds the buffer clean
* after locking it, but we don't care all that much.)
*/
static int
-SyncOneBuffer(int buf_id, bool skip_recently_used, WritebackContext *wb_context)
+PrepareToWriteBuffer(BuffersToWrite *to_write, Buffer buf,
+ bool skip_recently_used,
+ IOQueue *ioq, WritebackContext *wb_context)
{
- BufferDesc *bufHdr = GetBufferDescriptor(buf_id);
+ BufferDesc *cur_buf_hdr = GetBufferDescriptor(buf - 1);
+ uint32 buf_state;
int result = 0;
- uint32 buf_state;
- BufferTag tag;
+ XLogRecPtr cur_buf_lsn;
+ LWLock *content_lock;
+ bool may_block;
+
+ /*
+ * Check if this buffer can be written out together with already prepared
+ * writes. We check before we have pinned the buffer, so the buffer can be
+ * written out and replaced between this check and us pinning the buffer -
+ * we'll recheck below. The reason for the pre-check is that we don't want
+ * to pin the buffer just to find out that we can't merge the IO.
+ */
+ if (to_write->nbuffers != 0)
+ {
+ if (!CanMergeWrite(to_write, cur_buf_hdr))
+ {
+ result |= BUF_CANT_MERGE;
+ return result;
+ }
+ }
+ else
+ {
+ to_write->start_at_tag = cur_buf_hdr->tag;
+ }
/* Make sure we can handle the pin */
ReservePrivateRefCountEntry();
ResourceOwnerEnlarge(CurrentResourceOwner);
+ /* XXX: Should also check if we are allowed to pin one more buffer */
+
/*
* Check whether buffer needs writing.
*
@@ -3910,7 +4135,7 @@ SyncOneBuffer(int buf_id, bool skip_recently_used, WritebackContext *wb_context)
* don't worry because our checkpoint.redo points before log record for
* upcoming changes and so we are not required to write such dirty buffer.
*/
- buf_state = LockBufHdr(bufHdr);
+ buf_state = LockBufHdr(cur_buf_hdr);
if (BUF_STATE_GET_REFCOUNT(buf_state) == 0 &&
BUF_STATE_GET_USAGECOUNT(buf_state) == 0)
@@ -3919,40 +4144,300 @@ SyncOneBuffer(int buf_id, bool skip_recently_used, WritebackContext *wb_context)
}
else if (skip_recently_used)
{
+#if 0
+ elog(LOG, "at block %d: skip recent with nbuffers %d",
+ cur_buf_hdr->tag.blockNum, to_write->nbuffers);
+#endif
/* Caller told us not to write recently-used buffers */
- UnlockBufHdr(bufHdr, buf_state);
+ UnlockBufHdr(cur_buf_hdr, buf_state);
return result;
}
if (!(buf_state & BM_VALID) || !(buf_state & BM_DIRTY))
{
/* It's clean, so nothing to do */
- UnlockBufHdr(bufHdr, buf_state);
+ UnlockBufHdr(cur_buf_hdr, buf_state);
return result;
}
+ /* pin the buffer, from now on its identity can't change anymore */
+ PinBuffer_Locked(cur_buf_hdr);
+
+ /*
+ * Acquire IO, if needed, now that it's likely that we'll need to write.
+ */
+ if (to_write->ioh == NULL)
+ {
+ /* otherwise we should already have acquired a handle */
+ Assert(to_write->nbuffers == 0);
+
+ to_write->ioh = io_queue_acquire_io(ioq);
+ pgaio_io_get_wref(to_write->ioh, &to_write->iow);
+ }
+
+ /*
+ * If we are merging, check if the buffer's identity possibly changed
+ * while we hadn't yet pinned it.
+ *
+ * XXX: It might be worth checking if we still want to write the buffer
+ * out, e.g. it could have been replaced with a buffer that doesn't have
+ * BM_CHECKPOINT_NEEDED set.
+ */
+ if (to_write->nbuffers != 0)
+ {
+ if (!CanMergeWrite(to_write, cur_buf_hdr))
+ {
+ elog(LOG, "changed identity");
+ UnpinBuffer(cur_buf_hdr);
+
+ result |= BUF_CANT_MERGE;
+
+ return result;
+ }
+ }
+
+ may_block = to_write->nbuffers == 0
+ && !pgaio_have_staged()
+ && io_queue_is_empty(ioq)
+ ;
+ content_lock = BufferDescriptorGetContentLock(cur_buf_hdr);
+
+ if (!may_block)
+ {
+ if (LWLockConditionalAcquire(content_lock, LW_SHARED))
+ {
+ /* done */
+ }
+ else if (to_write->nbuffers == 0)
+ {
+ /*
+ * Need to wait for all prior IO to finish before blocking for
+ * lock acquisition, to avoid the risk a deadlock due to us
+ * waiting for another backend that is waiting for our unsubmitted
+ * IO to complete.
+ */
+ pgaio_submit_staged();
+ io_queue_wait_all(ioq);
+
+ elog(DEBUG2, "at block %u: can't block, nbuffers = 0",
+ cur_buf_hdr->tag.blockNum
+ );
+
+ may_block = to_write->nbuffers == 0
+ && !pgaio_have_staged()
+ && io_queue_is_empty(ioq)
+ ;
+ Assert(may_block);
+
+ LWLockAcquire(content_lock, LW_SHARED);
+ }
+ else
+ {
+ elog(DEBUG2, "at block %d: can't block nbuffers = %d",
+ cur_buf_hdr->tag.blockNum,
+ to_write->nbuffers);
+
+ UnpinBuffer(cur_buf_hdr);
+ result |= BUF_CANT_MERGE;
+ Assert(to_write->nbuffers > 0);
+
+ return result;
+ }
+ }
+ else
+ {
+ LWLockAcquire(content_lock, LW_SHARED);
+ }
+
+ if (!may_block)
+ {
+ if (!StartBufferIO(cur_buf_hdr, false, !may_block))
+ {
+ pgaio_submit_staged();
+ io_queue_wait_all(ioq);
+
+ may_block = io_queue_is_empty(ioq) && to_write->nbuffers == 0 && !pgaio_have_staged();
+
+ if (!StartBufferIO(cur_buf_hdr, false, !may_block))
+ {
+ elog(DEBUG2, "at block %d: non-waitable StartBufferIO returns false, %d",
+ cur_buf_hdr->tag.blockNum,
+ may_block);
+
+ /*
+ * FIXME: can't tell whether this is because the buffer has
+ * been cleaned
+ */
+ if (!may_block)
+ {
+ result |= BUF_CANT_MERGE;
+ Assert(to_write->nbuffers > 0);
+ }
+ LWLockRelease(content_lock);
+ UnpinBuffer(cur_buf_hdr);
+
+ return result;
+ }
+ }
+ }
+ else
+ {
+ if (!StartBufferIO(cur_buf_hdr, false, false))
+ {
+ elog(DEBUG2, "waitable StartBufferIO returns false");
+ LWLockRelease(content_lock);
+ UnpinBuffer(cur_buf_hdr);
+
+ /*
+ * FIXME: Historically we returned BUF_WRITTEN in this case, which
+ * seems wrong
+ */
+ return result;
+ }
+ }
+
/*
- * Pin it, share-lock it, write it. (FlushBuffer will do nothing if the
- * buffer is clean by the time we've locked it.)
+ * Run PageGetLSN while holding header lock, since we don't have the
+ * buffer locked exclusively in all cases.
*/
- PinBuffer_Locked(bufHdr);
- LWLockAcquire(BufferDescriptorGetContentLock(bufHdr), LW_SHARED);
+ buf_state = LockBufHdr(cur_buf_hdr);
+
+ cur_buf_lsn = BufferGetLSN(cur_buf_hdr);
- FlushBuffer(bufHdr, NULL, IOOBJECT_RELATION, IOCONTEXT_NORMAL);
+ /* To check if block content changes while flushing. - vadim 01/17/97 */
+ buf_state &= ~BM_JUST_DIRTIED;
- LWLockRelease(BufferDescriptorGetContentLock(bufHdr));
+ UnlockBufHdr(cur_buf_hdr, buf_state);
- tag = bufHdr->tag;
+ to_write->buffers[to_write->nbuffers] = buf;
+ to_write->nbuffers++;
- UnpinBuffer(bufHdr);
+ if (buf_state & BM_PERMANENT &&
+ (to_write->max_lsn == InvalidXLogRecPtr || to_write->max_lsn < cur_buf_lsn))
+ {
+ to_write->max_lsn = cur_buf_lsn;
+ }
+
+ result |= BUF_WRITTEN;
+
+ return result;
+}
+
+static void
+WriteBuffers(BuffersToWrite *to_write,
+ IOQueue *ioq, WritebackContext *wb_context)
+{
+ SMgrRelation smgr;
+ Buffer first_buf;
+ BufferDesc *first_buf_hdr;
+ bool needs_checksum;
+
+ Assert(to_write->nbuffers > 0 && to_write->nbuffers <= io_combine_limit);
+
+ first_buf = to_write->buffers[0];
+ first_buf_hdr = GetBufferDescriptor(first_buf - 1);
+
+ smgr = smgropen(BufTagGetRelFileLocator(&first_buf_hdr->tag), INVALID_PROC_NUMBER);
/*
- * SyncOneBuffer() is only called by checkpointer and bgwriter, so
- * IOContext will always be IOCONTEXT_NORMAL.
+ * Force XLOG flush up to buffer's LSN. This implements the basic WAL
+ * rule that log updates must hit disk before any of the data-file changes
+ * they describe do.
+ *
+ * However, this rule does not apply to unlogged relations, which will be
+ * lost after a crash anyway. Most unlogged relation pages do not bear
+ * LSNs since we never emit WAL records for them, and therefore flushing
+ * up through the buffer LSN would be useless, but harmless. However,
+ * GiST indexes use LSNs internally to track page-splits, and therefore
+ * unlogged GiST pages bear "fake" LSNs generated by
+ * GetFakeLSNForUnloggedRel. It is unlikely but possible that the fake
+ * LSN counter could advance past the WAL insertion point; and if it did
+ * happen, attempting to flush WAL through that location would fail, with
+ * disastrous system-wide consequences. To make sure that can't happen,
+ * skip the flush if the buffer isn't permanent.
*/
- ScheduleBufferTagForWriteback(wb_context, IOCONTEXT_NORMAL, &tag);
+ if (to_write->max_lsn != InvalidXLogRecPtr)
+ XLogFlush(to_write->max_lsn);
+
+ /*
+ * Now it's safe to write the buffer to disk. Note that no one else should
+ * have been able to write it, while we were busy with log flushing,
+ * because we got the exclusive right to perform I/O by setting the
+ * BM_IO_IN_PROGRESS bit.
+ */
+
+ for (int nbuf = 0; nbuf < to_write->nbuffers; nbuf++)
+ {
+ Buffer cur_buf = to_write->buffers[nbuf];
+ BufferDesc *cur_buf_hdr = GetBufferDescriptor(cur_buf - 1);
+ Block bufBlock;
+ char *bufToWrite;
+
+ bufBlock = BufHdrGetBlock(cur_buf_hdr);
+ needs_checksum = PageNeedsChecksumCopy((Page) bufBlock);
+
+ /*
+ * Update page checksum if desired. Since we have only shared lock on
+ * the buffer, other processes might be updating hint bits in it, so
+ * we must copy the page to a bounce buffer if we do checksumming.
+ */
+ if (needs_checksum)
+ {
+ PgAioBounceBuffer *bb = pgaio_bounce_buffer_get();
- return result | BUF_WRITTEN;
+ pgaio_io_assoc_bounce_buffer(to_write->ioh, bb);
+
+ bufToWrite = pgaio_bounce_buffer_buffer(bb);
+ memcpy(bufToWrite, bufBlock, BLCKSZ);
+ PageSetChecksumInplace((Page) bufToWrite, cur_buf_hdr->tag.blockNum);
+ }
+ else
+ {
+ bufToWrite = bufBlock;
+ }
+
+ to_write->data_ptrs[nbuf] = bufToWrite;
+ }
+
+ pgaio_io_set_handle_data_32(to_write->ioh,
+ (uint32 *) to_write->buffers,
+ to_write->nbuffers);
+ pgaio_io_register_callbacks(to_write->ioh, PGAIO_HCB_SHARED_BUFFER_WRITEV, 0);
+
+ smgrstartwritev(to_write->ioh, smgr,
+ BufTagGetForkNum(&first_buf_hdr->tag),
+ first_buf_hdr->tag.blockNum,
+ to_write->data_ptrs,
+ to_write->nbuffers,
+ false);
+ pgstat_count_io_op(IOOBJECT_RELATION, IOCONTEXT_NORMAL,
+ IOOP_WRITE, 1, BLCKSZ * to_write->nbuffers);
+
+
+ for (int nbuf = 0; nbuf < to_write->nbuffers; nbuf++)
+ {
+ Buffer cur_buf = to_write->buffers[nbuf];
+ BufferDesc *cur_buf_hdr = GetBufferDescriptor(cur_buf - 1);
+
+ UnpinBuffer(cur_buf_hdr);
+ }
+
+ io_queue_track(ioq, &to_write->iow);
+ to_write->total_writes++;
+
+ /* clear state for next write */
+ to_write->nbuffers = 0;
+ to_write->start_at_tag.relNumber = InvalidOid;
+ to_write->start_at_tag.blockNum = InvalidBlockNumber;
+ to_write->max_combine = 0;
+ to_write->max_lsn = InvalidXLogRecPtr;
+ to_write->ioh = NULL;
+ pgaio_wref_clear(&to_write->iow);
+
+ /*
+ * FIXME: Implement issuing writebacks (note wb_context isn't used here).
+ * Possibly needs to be integrated with io_queue.c.
+ */
}
/*
@@ -4326,6 +4811,7 @@ FlushBuffer(BufferDesc *buf, SMgrRelation reln, IOObject io_object,
error_context_stack = errcallback.previous;
}
+
/*
* RelationGetNumberOfBlocksInFork
* Determines the current number of pages in the specified relation fork.
diff --git a/src/backend/storage/page/bufpage.c b/src/backend/storage/page/bufpage.c
index ecc81aacfc3..71dc6f559dd 100644
--- a/src/backend/storage/page/bufpage.c
+++ b/src/backend/storage/page/bufpage.c
@@ -1480,6 +1480,16 @@ PageIndexTupleOverwrite(Page page, OffsetNumber offnum,
return true;
}
+bool
+PageNeedsChecksumCopy(Page page)
+{
+ if (PageIsNew(page))
+ return false;
+
+ /* If we don't need a checksum, just return the passed-in data */
+ return DataChecksumsEnabled();
+}
+
/*
* Set checksum for a page in shared buffers.
diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list
index 9c07fa04b49..96144af6a4a 100644
--- a/src/tools/pgindent/typedefs.list
+++ b/src/tools/pgindent/typedefs.list
@@ -348,6 +348,7 @@ BufferManagerRelation
BufferStrategyControl
BufferTag
BufferUsage
+BuffersToWrite
BuildAccumulator
BuiltinScript
BulkInsertState
--
2.48.1.76.g4e746b1a31.dirty
[text/x-diff] v2.11-0024-Ensure-a-resowner-exists-for-all-paths-that-ma.patch (2.6K, ../../6ak556uyqiptdwjaci4kbi5eykwkmzqgkbtkyaosjnopjhncrc@2v4ac2jwyz22/25-v2.11-0024-Ensure-a-resowner-exists-for-all-paths-that-ma.patch)
download | inline diff:
From ae835445290a70888f0969d5ce1da973ca1a728c Mon Sep 17 00:00:00 2001
From: Andres Freund <andres@anarazel.de>
Date: Tue, 18 Mar 2025 14:40:06 -0400
Subject: [PATCH v2.11 24/27] Ensure a resowner exists for all paths that may
perform AIO
Reviewed-by: Noah Misch <noah@leadboat.com>
Discussion: https://postgr.es/m/1f6b50a7-38ef-4d87-8246-786d39f46ab9@iki.fi
---
src/backend/bootstrap/bootstrap.c | 7 +++++++
src/backend/replication/logical/logical.c | 6 ++++++
src/backend/utils/init/postinit.c | 6 +++++-
3 files changed, 18 insertions(+), 1 deletion(-)
diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c
index 6db864892d0..e554504e1f0 100644
--- a/src/backend/bootstrap/bootstrap.c
+++ b/src/backend/bootstrap/bootstrap.c
@@ -361,8 +361,15 @@ BootstrapModeMain(int argc, char *argv[], bool check_only)
BaseInit();
bootstrap_signals();
+
+ /* need a resowner for IO during BootStrapXLOG() */
+ CreateAuxProcessResourceOwner();
+
BootStrapXLOG(bootstrap_data_checksum_version);
+ ReleaseAuxProcessResources(true);
+ CurrentResourceOwner = NULL;
+
/*
* To ensure that src/common/link-canary.c is linked into the backend, we
* must call it from somewhere. Here is as good as anywhere.
diff --git a/src/backend/replication/logical/logical.c b/src/backend/replication/logical/logical.c
index 8ea846bfc3b..2b936714040 100644
--- a/src/backend/replication/logical/logical.c
+++ b/src/backend/replication/logical/logical.c
@@ -386,6 +386,12 @@ CreateInitDecodingContext(const char *plugin,
slot->data.plugin = plugin_name;
SpinLockRelease(&slot->mutex);
+ if (CurrentResourceOwner == NULL)
+ {
+ Assert(am_walsender);
+ CurrentResourceOwner = AuxProcessResourceOwner;
+ }
+
if (XLogRecPtrIsInvalid(restart_lsn))
ReplicationSlotReserveWal();
else
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index 7958ea11b73..222e24bcb08 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -785,8 +785,12 @@ InitPostgres(const char *in_dbname, Oid dboid,
* We don't yet have an aux-process resource owner, but StartupXLOG
* and ShutdownXLOG will need one. Hence, create said resource owner
* (and register a callback to clean it up after ShutdownXLOG runs).
+ *
+ * In bootstrap mode CreateAuxProcessResourceOwner() was already
+ * called in BootstrapModeMain().
*/
- CreateAuxProcessResourceOwner();
+ if (!bootstrap)
+ CreateAuxProcessResourceOwner();
StartupXLOG();
/* Release (and warn about) any buffer pins leaked in StartupXLOG */
--
2.48.1.76.g4e746b1a31.dirty
[text/x-diff] v2.11-0025-Temporary-Increase-BAS_BULKREAD-size.patch (1.3K, ../../6ak556uyqiptdwjaci4kbi5eykwkmzqgkbtkyaosjnopjhncrc@2v4ac2jwyz22/26-v2.11-0025-Temporary-Increase-BAS_BULKREAD-size.patch)
download | inline diff:
From e3e9545899e6b2ea34e589f88f427472a803e2ff Mon Sep 17 00:00:00 2001
From: Andres Freund <andres@anarazel.de>
Date: Tue, 18 Mar 2025 14:40:06 -0400
Subject: [PATCH v2.11 25/27] Temporary: Increase BAS_BULKREAD size
Without this we only can execute very little AIO for sequential scans, as
there's just not enough buffers in the ring. This isn't the right fix, as
just increasing the ring size can have negative performance implications in
workloads where the kernel has all the data cached.
Author:
Reviewed-By:
Discussion: https://postgr.es/m/
Backpatch:
---
src/backend/storage/buffer/freelist.c | 7 ++++++-
1 file changed, 6 insertions(+), 1 deletion(-)
diff --git a/src/backend/storage/buffer/freelist.c b/src/backend/storage/buffer/freelist.c
index 336715b6c63..b72a5957a20 100644
--- a/src/backend/storage/buffer/freelist.c
+++ b/src/backend/storage/buffer/freelist.c
@@ -555,7 +555,12 @@ GetAccessStrategy(BufferAccessStrategyType btype)
return NULL;
case BAS_BULKREAD:
- ring_size_kb = 256;
+
+ /*
+ * FIXME: Temporary increase to allow large enough streaming reads
+ * to actually benefit from AIO. This needs a better solution.
+ */
+ ring_size_kb = 2 * 1024;
break;
case BAS_BULKWRITE:
ring_size_kb = 16 * 1024;
--
2.48.1.76.g4e746b1a31.dirty
[text/x-diff] v2.11-0026-WIP-Use-MAP_POPULATE.patch (1.1K, ../../6ak556uyqiptdwjaci4kbi5eykwkmzqgkbtkyaosjnopjhncrc@2v4ac2jwyz22/27-v2.11-0026-WIP-Use-MAP_POPULATE.patch)
download | inline diff:
From 9cba27b9fb50afe9b01d97bbfa967dcf12d69779 Mon Sep 17 00:00:00 2001
From: Andres Freund <andres@anarazel.de>
Date: Tue, 18 Mar 2025 14:40:06 -0400
Subject: [PATCH v2.11 26/27] WIP: Use MAP_POPULATE
For benchmarking it's quite annoying that the first time a memory is touched
has completely different perf characteristics than subsequent accesses. Using
MAP_POPULATE reduces that substantially.
---
src/backend/port/sysv_shmem.c | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/src/backend/port/sysv_shmem.c b/src/backend/port/sysv_shmem.c
index 197926d44f6..a700b02d5a1 100644
--- a/src/backend/port/sysv_shmem.c
+++ b/src/backend/port/sysv_shmem.c
@@ -620,7 +620,7 @@ CreateAnonymousSegment(Size *size)
allocsize += hugepagesize - (allocsize % hugepagesize);
ptr = mmap(NULL, allocsize, PROT_READ | PROT_WRITE,
- PG_MMAP_FLAGS | mmap_flags, -1, 0);
+ PG_MMAP_FLAGS | MAP_POPULATE | mmap_flags, -1, 0);
mmap_errno = errno;
if (huge_pages == HUGE_PAGES_TRY && ptr == MAP_FAILED)
elog(DEBUG1, "mmap(%zu) with MAP_HUGETLB failed, huge pages disabled: %m",
--
2.48.1.76.g4e746b1a31.dirty
[text/x-diff] v2.11-0027-StartReadBuffers-debug-stuff.patch (1.9K, ../../6ak556uyqiptdwjaci4kbi5eykwkmzqgkbtkyaosjnopjhncrc@2v4ac2jwyz22/28-v2.11-0027-StartReadBuffers-debug-stuff.patch)
download | inline diff:
From af56fb208e9b6f45de7c2f1b2d8e25214d2370c3 Mon Sep 17 00:00:00 2001
From: Andres Freund <andres@anarazel.de>
Date: Tue, 18 Mar 2025 14:40:06 -0400
Subject: [PATCH v2.11 27/27] StartReadBuffers debug stuff
---
src/backend/storage/buffer/bufmgr.c | 25 +++++++++++++++++++++++++
1 file changed, 25 insertions(+)
diff --git a/src/backend/storage/buffer/bufmgr.c b/src/backend/storage/buffer/bufmgr.c
index 0909888c342..ba695badbeb 100644
--- a/src/backend/storage/buffer/bufmgr.c
+++ b/src/backend/storage/buffer/bufmgr.c
@@ -1781,6 +1781,18 @@ AsyncReadBuffers(ReadBuffersOperation *operation, int *nblocks_progress)
IOObject io_object;
bool did_start_io;
+#if 0
+ ereport(DEBUG3,
+ errmsg("%s: op->blocks: %d, op->blocks_done: %d, *nblocks_progress: %d, first buf %d",
+ __func__,
+ operation->nblocks,
+ nblocks_done,
+ *nblocks_progress,
+ buffers[0]),
+ errhidestmt(true),
+ errhidecontext(true));
+#endif
+
/*
* When this IO is executed synchronously, either because the caller will
* immediately block waiting for the IO or because IOMETHOD_SYNC is used,
@@ -1845,6 +1857,13 @@ AsyncReadBuffers(ReadBuffersOperation *operation, int *nblocks_progress)
operation->nblocks_done += 1;
*nblocks_progress = 1;
+ ereport(DEBUG3,
+ errmsg("%s - trunc: %d",
+ __func__,
+ operation->nblocks_done),
+ errhidestmt(true),
+ errhidecontext(true));
+
pgaio_io_release(ioh);
pgaio_wref_clear(&operation->io_wref);
did_start_io = false;
@@ -1894,6 +1913,12 @@ AsyncReadBuffers(ReadBuffersOperation *operation, int *nblocks_progress)
*/
for (int i = nblocks_done + 1; i < operation->nblocks; i++)
{
+#if 0
+ /* FIXME: Remove forced short read */
+ if (i > 3)
+ break;
+#endif
+
if (!ReadBuffersCanStartIO(buffers[i], true))
break;
/* Must be consecutive block numbers. */
--
2.48.1.76.g4e746b1a31.dirty
^ permalink raw reply [nested|flat] 153+ messages in thread
* Re: AIO v2.5
2025-03-04 19:00 AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-10 18:23 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-11 19:41 ` Re: AIO v2.5 Noah Misch <noah@leadboat.com>
2025-03-11 23:55 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-12 03:57 ` Re: AIO v2.5 Noah Misch <noah@leadboat.com>
2025-03-12 17:06 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-12 23:23 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-13 10:53 ` Re: AIO v2.5 Antonin Houska <ah@cybertec.at>
2025-03-14 19:43 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-17 20:00 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-18 20:12 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-21 01:58 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
@ 2025-03-23 00:20 ` Noah Misch <noah@leadboat.com>
2025-03-23 15:11 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
4 siblings, 1 reply; 153+ messages in thread
From: Noah Misch @ 2025-03-23 00:20 UTC (permalink / raw)
To: Andres Freund <andres@anarazel.de>; +Cc: Antonin Houska <ah@cybertec.at>; pgsql-hackers@postgresql.org, Thomas Munro <thomas.munro@gmail.com>; Heikki Linnakangas <hlinnaka@iki.fi>; Robert Haas <robertmhaas@gmail.com>; Jakub Wartak <jakub.wartak@enterprisedb.com>; Jelte Fennema-Nio <postgres@jeltef.nl>
On Thu, Mar 20, 2025 at 09:58:37PM -0400, Andres Freund wrote:
> Attached v2.11, with the following changes:
> - Added an error check for FileStartReadV() failing
>
> FileStartReadV() actually can fail, if the file can't be re-opened. I
> thought it'd be important for the error message to differ from the one
> that's issued for read actually failing, so I went with:
>
> "could not start reading blocks %u..%u in file \"%s\": %m"
>
> but I'm not sure how good that is.
Message looks good.
> - Improved error message if io_uring_queue_init() fails
>
> Added errhint()s for likely cases of failure.
>
> Added errcode(). I was tempted to use errcode_for_file_access(), but that
> doesn't support ENOSYS - perhaps I should add that instead?
Either way is fine with me. ENOSYS -> ERRCODE_FEATURE_NOT_SUPPORTED is a good
general mapping to have in errcode_for_file_access(), but it's also not a
problem to keep it the way v2.11 has it.
> - Disable io_uring method when using EXEC_BACKEND, they're not compatible
>
> I chose to do this with a define aio.h, but I guess we could also do it at
> configure time? That seems more complicated though - how would we even know
> that EXEC_BACKEND is used on non-windows?
Agreed, "make PROFILE=-DEXEC_BACKEND" is a valid way to get EXEC_BACKEND.
> Not sure yet how to best disable testing io_uring in this case. We can't
> just query EXEC_BACKEND from pg_config.h unfortunately. I guess making the
> initdb not fail and checking the error log would work, but that doesn't work
> nicely with Cluster.pm.
How about "postgres -c io_method=io_uring -C <anything>":
--- a/src/test/modules/test_aio/t/001_aio.pl
+++ b/src/test/modules/test_aio/t/001_aio.pl
@@ -29,7 +29,13 @@ $node_worker->stop();
# Test io_method=io_uring
###
-if ($ENV{with_liburing} eq 'yes')
+sub have_io_uring
+{
+ local %ENV = $node_worker->_get_env(); # any node works
+ return run_log [qw(postgres -c io_method=io_uring -C io_method)];
+}
+
+if (have_io_uring())
{
my $node_uring = create_node('io_uring');
$node_uring->start();
> Questions:
>
>
> - We only "look" at BM_IO_ERROR for writes, isn't that somewhat weird?
>
> See AbortBufferIO(Buffer buffer)
>
> It doesn't really matter for the patchset, but it just strikes me as an oddity.
That caught my attention in an earlier review round, but I didn't find it
important enough to raise. It's mildly unfortunate to be setting BM_IO_ERROR
for reads when the only thing BM_IO_ERROR drives is message "Multiple failures
--- write error might be permanent." It's minor, so let's leave it that way
for the foreseeable future.
> Subject: [PATCH v2.11 01/27] aio, bufmgr: Comment fixes
Ready to commit, though other comment fixes might come up in later reviews.
One idea so far is to comment on valid states after some IoMethodOps
callbacks:
--- a/src/include/storage/aio_internal.h
+++ b/src/include/storage/aio_internal.h
@@ -310,6 +310,9 @@ typedef struct IoMethodOps
/*
* Start executing passed in IOs.
*
+ * Shall advance state to PGAIO_HS_SUBMITTED. (By the time this returns,
+ * other backends might have advanced the state further.)
+ *
* Will not be called if ->needs_synchronous_execution() returned true.
*
* num_staged_ios is <= PGAIO_SUBMIT_BATCH_SIZE.
@@ -321,6 +324,12 @@ typedef struct IoMethodOps
/*
* Wait for the IO to complete. Optional.
*
+ * On return, state shall be PGAIO_HS_COMPLETED_IO,
+ * PGAIO_HS_COMPLETED_SHARED or PGAIO_HS_COMPLETED_LOCAL. (The callback
+ * need not change the state if it's already one of those.) If state is
+ * PGAIO_HS_COMPLETED_IO, state will reach PGAIO_HS_COMPLETED_SHARED
+ * without further intervention.
+ *
* If not provided, it needs to be guaranteed that the IO method calls
* pgaio_io_process_completion() without further interaction by the
* issuing backend.
> Subject: [PATCH v2.11 02/27] aio: Change prefix of PgAioResultStatus values to
> PGAIO_RS_
Ready to commit
> Subject: [PATCH v2.11 03/27] Redefine max_files_per_process to control
> additionally opened files
Ready to commit
> Subject: [PATCH v2.11 04/27] aio: Add liburing dependency
> --- a/meson.build
> +++ b/meson.build
> @@ -944,6 +944,18 @@ endif
>
>
>
> +###############################################################
> +# Library: liburing
> +###############################################################
> +
> +liburingopt = get_option('liburing')
> +liburing = dependency('liburing', required: liburingopt)
> +if liburing.found()
> + cdata.set('USE_LIBURING', 1)
> +endif
This is a different style from other deps; is it equivalent to our standard
style? Example for lz4:
lz4opt = get_option('lz4')
if not lz4opt.disabled()
lz4 = dependency('liblz4', required: false)
# Unfortunately the dependency is named differently with cmake
if not lz4.found() # combine with above once meson 0.60.0 is required
lz4 = dependency('lz4', required: lz4opt,
method: 'cmake', modules: ['LZ4::lz4_shared'],
)
endif
if lz4.found()
cdata.set('USE_LZ4', 1)
cdata.set('HAVE_LIBLZ4', 1)
endif
else
lz4 = not_found_dep
endif
> --- a/configure.ac
> +++ b/configure.ac
> @@ -975,6 +975,14 @@ AC_SUBST(with_readline)
> PGAC_ARG_BOOL(with, libedit-preferred, no,
> [prefer BSD Libedit over GNU Readline])
>
> +#
> +# liburing
> +#
> +AC_MSG_CHECKING([whether to build with liburing support])
> +PGAC_ARG_BOOL(with, liburing, no, [io_uring support, for asynchronous I/O],
Fourth arg generally starts with "build" for args like this. I suggest "build
with io_uring support, for asynchronous I/O". Comparable options:
--with-llvm build with LLVM based JIT support
--with-tcl build Tcl modules (PL/Tcl)
--with-perl build Perl modules (PL/Perl)
--with-python build Python modules (PL/Python)
--with-gssapi build with GSSAPI support
--with-pam build with PAM support
--with-bsd-auth build with BSD Authentication support
--with-ldap build with LDAP support
--with-bonjour build with Bonjour support
--with-selinux build with SELinux support
--with-systemd build with systemd support
--with-libcurl build with libcurl support
--with-libxml build with XML support
--with-libxslt use XSLT support when building contrib/xml2
--with-lz4 build with LZ4 support
--with-zstd build with ZSTD support
> + [AC_DEFINE([USE_LIBURING], 1, [Define to build with io_uring support. (--with-liburing)])])
> +AC_MSG_RESULT([$with_liburing])
> +AC_SUBST(with_liburing)
>
> #
> # UUID library
> @@ -1463,6 +1471,9 @@ elif test "$with_uuid" = ossp ; then
> fi
> AC_SUBST(UUID_LIBS)
>
> +if test "$with_liburing" = yes; then
> + PKG_CHECK_MODULES(LIBURING, liburing)
> +fi
We usually put this right after the AC_MSG_CHECKING ... AC_SUBST block. This
currently has unrelated stuff separating them. Also, with the exception of
icu, we follow PKG_CHECK_MODULES uses by absorbing flags from pkg-config and
use AC_CHECK_LIB to add the actual "-l". By not absorbing flags, I think a
liburing in a nonstandard location would require --with-libraries and
--with-includes, unlike the other PKG_CHECK_MODULES-based dependencies. lz4
is a representative example of our standard:
```
AC_MSG_CHECKING([whether to build with LZ4 support])
PGAC_ARG_BOOL(with, lz4, no, [build with LZ4 support],
[AC_DEFINE([USE_LZ4], 1, [Define to 1 to build with LZ4 support. (--with-lz4)])])
AC_MSG_RESULT([$with_lz4])
AC_SUBST(with_lz4)
if test "$with_lz4" = yes; then
PKG_CHECK_MODULES(LZ4, liblz4)
# We only care about -I, -D, and -L switches;
# note that -llz4 will be added by AC_CHECK_LIB below.
for pgac_option in $LZ4_CFLAGS; do
case $pgac_option in
-I*|-D*) CPPFLAGS="$CPPFLAGS $pgac_option";;
esac
done
for pgac_option in $LZ4_LIBS; do
case $pgac_option in
-L*) LDFLAGS="$LDFLAGS $pgac_option";;
esac
done
fi
# ... later in file ...
if test "$with_lz4" = yes ; then
AC_CHECK_LIB(lz4, LZ4_compress_default, [], [AC_MSG_ERROR([library 'lz4' is required for LZ4 support])])
fi
```
I think it's okay to not use the AC_CHECK_LIB and rely on explicit
src/backend/Makefile code like you've done, but we shouldn't miss
CPPFLAGS/LDFLAGS (or should have a comment on why missing them is right).
> --- a/doc/src/sgml/installation.sgml
> +++ b/doc/src/sgml/installation.sgml
lz4 and other deps have a mention in <sect1 id="install-requirements">, in
addition to sections edited here.
> Subject: [PATCH v2.11 05/27] aio: Add io_method=io_uring
(Still reviewing this one.)
^ permalink raw reply [nested|flat] 153+ messages in thread
* Re: AIO v2.5
2025-03-04 19:00 AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-10 18:23 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-11 19:41 ` Re: AIO v2.5 Noah Misch <noah@leadboat.com>
2025-03-11 23:55 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-12 03:57 ` Re: AIO v2.5 Noah Misch <noah@leadboat.com>
2025-03-12 17:06 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-12 23:23 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-13 10:53 ` Re: AIO v2.5 Antonin Houska <ah@cybertec.at>
2025-03-14 19:43 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-17 20:00 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-18 20:12 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-21 01:58 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-23 00:20 ` Re: AIO v2.5 Noah Misch <noah@leadboat.com>
@ 2025-03-23 15:11 ` Andres Freund <andres@anarazel.de>
2025-03-23 15:55 ` Re: AIO v2.5 Noah Misch <noah@leadboat.com>
0 siblings, 1 reply; 153+ messages in thread
From: Andres Freund @ 2025-03-23 15:11 UTC (permalink / raw)
To: Noah Misch <noah@leadboat.com>; +Cc: Antonin Houska <ah@cybertec.at>; pgsql-hackers@postgresql.org, Thomas Munro <thomas.munro@gmail.com>; Heikki Linnakangas <hlinnaka@iki.fi>; Robert Haas <robertmhaas@gmail.com>; Jakub Wartak <jakub.wartak@enterprisedb.com>; Jelte Fennema-Nio <postgres@jeltef.nl>
Hi,
On 2025-03-22 17:20:56 -0700, Noah Misch wrote:
> On Thu, Mar 20, 2025 at 09:58:37PM -0400, Andres Freund wrote:
> > Not sure yet how to best disable testing io_uring in this case. We can't
> > just query EXEC_BACKEND from pg_config.h unfortunately. I guess making the
> > initdb not fail and checking the error log would work, but that doesn't work
> > nicely with Cluster.pm.
>
> How about "postgres -c io_method=io_uring -C <anything>":
>
> --- a/src/test/modules/test_aio/t/001_aio.pl
> +++ b/src/test/modules/test_aio/t/001_aio.pl
> @@ -29,7 +29,13 @@ $node_worker->stop();
> # Test io_method=io_uring
> ###
>
> -if ($ENV{with_liburing} eq 'yes')
> +sub have_io_uring
> +{
> + local %ENV = $node_worker->_get_env(); # any node works
> + return run_log [qw(postgres -c io_method=io_uring -C io_method)];
> +}
> +
> +if (have_io_uring())
> {
> my $node_uring = create_node('io_uring');
> $node_uring->start();
Yea, that's a good idea.
One thing that doesn't seem great is that it requires a prior node - what if
we do -c io_method=invalid' that would report the list of valid GUC options,
so we could just grep for io_uring?
It's too bad that postgres --describe-config
a) doesn't report the possible enum values
b) doesn't apply/validate -c options
> > Subject: [PATCH v2.11 01/27] aio, bufmgr: Comment fixes
>
> Ready to commit, though other comment fixes might come up in later reviews.
I'll reorder it to a bit later in the series, to accumulate a few more.
> One idea so far is to comment on valid states after some IoMethodOps
> callbacks:
>
> --- a/src/include/storage/aio_internal.h
> +++ b/src/include/storage/aio_internal.h
> @@ -310,6 +310,9 @@ typedef struct IoMethodOps
> /*
> * Start executing passed in IOs.
> *
> + * Shall advance state to PGAIO_HS_SUBMITTED. (By the time this returns,
> + * other backends might have advanced the state further.)
> + *
> * Will not be called if ->needs_synchronous_execution() returned true.
> *
> * num_staged_ios is <= PGAIO_SUBMIT_BATCH_SIZE.
> @@ -321,6 +324,12 @@ typedef struct IoMethodOps
> /*
> * Wait for the IO to complete. Optional.
> *
> + * On return, state shall be PGAIO_HS_COMPLETED_IO,
> + * PGAIO_HS_COMPLETED_SHARED or PGAIO_HS_COMPLETED_LOCAL. (The callback
> + * need not change the state if it's already one of those.) If state is
> + * PGAIO_HS_COMPLETED_IO, state will reach PGAIO_HS_COMPLETED_SHARED
> + * without further intervention.
> + *
> * If not provided, it needs to be guaranteed that the IO method calls
> * pgaio_io_process_completion() without further interaction by the
> * issuing backend.
I think these are a good idea. I added those to the copy-edit patch, with a
few more tweaks:
@@ -315,6 +315,9 @@ typedef struct IoMethodOps
/*
* Start executing passed in IOs.
*
+ * Shall advance state to at least PGAIO_HS_SUBMITTED. (By the time this
+ * returns, other backends might have advanced the state further.)
+ *
* Will not be called if ->needs_synchronous_execution() returned true.
*
* num_staged_ios is <= PGAIO_SUBMIT_BATCH_SIZE.
@@ -323,12 +326,24 @@ typedef struct IoMethodOps
*/
int (*submit) (uint16 num_staged_ios, PgAioHandle **staged_ios);
- /*
+ /* ---
* Wait for the IO to complete. Optional.
*
+ * On return, state shall be on of
+ * - PGAIO_HS_COMPLETED_IO
+ * - PGAIO_HS_COMPLETED_SHARED
+ * - PGAIO_HS_COMPLETED_LOCAL
+ *
+ * The callback must not block if the handle is already in one of those
+ * states, or has been reused (see pgaio_io_was_recycled()). If, on
+ * return, the state is PGAIO_HS_COMPLETED_IO, state will reach
+ * PGAIO_HS_COMPLETED_SHARED without further intervention by the IO
+ * method.
+ *
* If not provided, it needs to be guaranteed that the IO method calls
* pgaio_io_process_completion() without further interaction by the
* issuing backend.
+ * ---
*/
void (*wait_one) (PgAioHandle *ioh,
uint64 ref_generation);
> > Subject: [PATCH v2.11 03/27] Redefine max_files_per_process to control
> > additionally opened files
>
> Ready to commit
Cool!
> > Subject: [PATCH v2.11 04/27] aio: Add liburing dependency
>
> > --- a/meson.build
> > +++ b/meson.build
> > @@ -944,6 +944,18 @@ endif
> >
> >
> >
> > +###############################################################
> > +# Library: liburing
> > +###############################################################
> > +
> > +liburingopt = get_option('liburing')
> > +liburing = dependency('liburing', required: liburingopt)
> > +if liburing.found()
> > + cdata.set('USE_LIBURING', 1)
> > +endif
>
> This is a different style from other deps; is it equivalent to our standard
> style?
Yes - the only reason to be more complicated in the lz4 case is that we want
to fall back to other ways of looking up the dependency (primarily because of
windows. But that's not required for liburing, which oviously is linux only.
> > --- a/configure.ac
> > +++ b/configure.ac
> > @@ -975,6 +975,14 @@ AC_SUBST(with_readline)
> > PGAC_ARG_BOOL(with, libedit-preferred, no,
> > [prefer BSD Libedit over GNU Readline])
> >
> > +#
> > +# liburing
> > +#
> > +AC_MSG_CHECKING([whether to build with liburing support])
> > +PGAC_ARG_BOOL(with, liburing, no, [io_uring support, for asynchronous I/O],
>
> Fourth arg generally starts with "build" for args like this. I suggest "build
> with io_uring support, for asynchronous I/O".
WFM.
> > + [AC_DEFINE([USE_LIBURING], 1, [Define to build with io_uring support. (--with-liburing)])])
> > +AC_MSG_RESULT([$with_liburing])
> > +AC_SUBST(with_liburing)
> >
> > #
> > # UUID library
> > @@ -1463,6 +1471,9 @@ elif test "$with_uuid" = ossp ; then
> > fi
> > AC_SUBST(UUID_LIBS)
> >
> > +if test "$with_liburing" = yes; then
> > + PKG_CHECK_MODULES(LIBURING, liburing)
> > +fi
>
> We usually put this right after the AC_MSG_CHECKING ... AC_SUBST block.
We don't really seem to do that for "dependency checks" in general, e.g.
PGAC_CHECK_PERL_CONFIGS, PGAC_CHECK_PYTHON_EMBED_SETUP, PGAC_CHECK_READLINE,
dependency dependent AC_CHECK_LIB calls, .. later in configure.ac than the
defnition of the option. TBH, I've always struggled trying to discern what
the organizing principle of configure.ac is.
But you're right that the PKG_CHECK_MODULES calls are closer-by. And I'm happy
to move towards having the code for each dep all in one place, so moved.
A related thing: We seem to have no order of the $with_ checks that I can
discern. Should the liburing check be at a different place?
> This currently has unrelated stuff separating them. Also, with the
> exception of icu, we follow PKG_CHECK_MODULES uses by absorbing flags from
> pkg-config and use AC_CHECK_LIB to add the actual "-l".
I think for liburing I was trying to follow ICU's example - injecting CFLAGS
and LIBS just in the parts of the build dir that needs them.
For LIBS I think I did so:
diff --git a/src/backend/Makefile b/src/backend/Makefile
...
+# The backend conditionally needs libraries that most executables don't need.
+LIBS += $(LDAP_LIBS_BE) $(ICU_LIBS) $(LIBURING_LIBS)
But ugh, for some reason I didn't do that for LIBURING_CFLAGS. In the v1.x
version of aio I had
aio:src/backend/storage/aio/Makefile:override CPPFLAGS += $(LIBURING_CFLAGS)
but somehow lost that somewhere along the way to v2.x
I think I like targetting where ${LIB}_LIBS and ${LIB}_CFLAGS are applied more
narrowly better than just adding to the global CFLAGS, CPPFLAGS, LDFLAGS. I'm
somewhat inclined to add it LIBURING_CFLAGS in src/backend rather than
src/backend/storage/aio/ though.
But I'm also willing to do it entirely differently.
> > --- a/doc/src/sgml/installation.sgml
> > +++ b/doc/src/sgml/installation.sgml
>
> lz4 and other deps have a mention in <sect1 id="install-requirements">, in
> addition to sections edited here.
Good point.
Although once more I feel defeated by the ordering used :)
Hm, that list is rather incomplete. At least libxml, libxslt, selinux, curl,
uuid, systemd, selinux and bonjour aren't listed.
Not sure if it makes sense to add liburing, given that?
Greetings,
Andres Freund
^ permalink raw reply [nested|flat] 153+ messages in thread
* Re: AIO v2.5
2025-03-04 19:00 AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-10 18:23 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-11 19:41 ` Re: AIO v2.5 Noah Misch <noah@leadboat.com>
2025-03-11 23:55 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-12 03:57 ` Re: AIO v2.5 Noah Misch <noah@leadboat.com>
2025-03-12 17:06 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-12 23:23 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-13 10:53 ` Re: AIO v2.5 Antonin Houska <ah@cybertec.at>
2025-03-14 19:43 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-17 20:00 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-18 20:12 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-21 01:58 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-23 00:20 ` Re: AIO v2.5 Noah Misch <noah@leadboat.com>
2025-03-23 15:11 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
@ 2025-03-23 15:55 ` Noah Misch <noah@leadboat.com>
2025-03-23 16:59 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
0 siblings, 1 reply; 153+ messages in thread
From: Noah Misch @ 2025-03-23 15:55 UTC (permalink / raw)
To: Andres Freund <andres@anarazel.de>; +Cc: Antonin Houska <ah@cybertec.at>; pgsql-hackers@postgresql.org, Thomas Munro <thomas.munro@gmail.com>; Heikki Linnakangas <hlinnaka@iki.fi>; Robert Haas <robertmhaas@gmail.com>; Jakub Wartak <jakub.wartak@enterprisedb.com>; Jelte Fennema-Nio <postgres@jeltef.nl>
On Sun, Mar 23, 2025 at 11:11:53AM -0400, Andres Freund wrote:
> On 2025-03-22 17:20:56 -0700, Noah Misch wrote:
> > On Thu, Mar 20, 2025 at 09:58:37PM -0400, Andres Freund wrote:
> > > Not sure yet how to best disable testing io_uring in this case. We can't
> > > just query EXEC_BACKEND from pg_config.h unfortunately. I guess making the
> > > initdb not fail and checking the error log would work, but that doesn't work
> > > nicely with Cluster.pm.
> >
> > How about "postgres -c io_method=io_uring -C <anything>":
> >
> > --- a/src/test/modules/test_aio/t/001_aio.pl
> > +++ b/src/test/modules/test_aio/t/001_aio.pl
> > @@ -29,7 +29,13 @@ $node_worker->stop();
> > # Test io_method=io_uring
> > ###
> >
> > -if ($ENV{with_liburing} eq 'yes')
> > +sub have_io_uring
> > +{
> > + local %ENV = $node_worker->_get_env(); # any node works
> > + return run_log [qw(postgres -c io_method=io_uring -C io_method)];
> > +}
> > +
> > +if (have_io_uring())
> > {
> > my $node_uring = create_node('io_uring');
> > $node_uring->start();
>
> Yea, that's a good idea.
>
> One thing that doesn't seem great is that it requires a prior node - what if
> we do -c io_method=invalid' that would report the list of valid GUC options,
> so we could just grep for io_uring?
Works for me.
> > One idea so far is to comment on valid states after some IoMethodOps
> > callbacks:
> I think these are a good idea. I added those to the copy-edit patch, with a
> few more tweaks:
The tweaks made it better.
> > > Subject: [PATCH v2.11 04/27] aio: Add liburing dependency
> > > + [AC_DEFINE([USE_LIBURING], 1, [Define to build with io_uring support. (--with-liburing)])])
> > > +AC_MSG_RESULT([$with_liburing])
> > > +AC_SUBST(with_liburing)
> > >
> > > #
> > > # UUID library
> > > @@ -1463,6 +1471,9 @@ elif test "$with_uuid" = ossp ; then
> > > fi
> > > AC_SUBST(UUID_LIBS)
> > >
> > > +if test "$with_liburing" = yes; then
> > > + PKG_CHECK_MODULES(LIBURING, liburing)
> > > +fi
> >
> > We usually put this right after the AC_MSG_CHECKING ... AC_SUBST block.
>
> We don't really seem to do that for "dependency checks" in general, e.g.
> PGAC_CHECK_PERL_CONFIGS, PGAC_CHECK_PYTHON_EMBED_SETUP, PGAC_CHECK_READLINE,
> dependency dependent AC_CHECK_LIB calls, .. later in configure.ac than the
> defnition of the option.
AC_CHECK_LIB stays far away, yes.
> But you're right that the PKG_CHECK_MODULES calls are closer-by. And I'm happy
> to move towards having the code for each dep all in one place, so moved.
>
>
> A related thing: We seem to have no order of the $with_ checks that I can
> discern. Should the liburing check be at a different place?
No opinion on that one. It's fine.
> > This currently has unrelated stuff separating them. Also, with the
> > exception of icu, we follow PKG_CHECK_MODULES uses by absorbing flags from
> > pkg-config and use AC_CHECK_LIB to add the actual "-l".
>
> I think for liburing I was trying to follow ICU's example - injecting CFLAGS
> and LIBS just in the parts of the build dir that needs them.
>
> For LIBS I think I did so:
>
> diff --git a/src/backend/Makefile b/src/backend/Makefile
> ...
> +# The backend conditionally needs libraries that most executables don't need.
> +LIBS += $(LDAP_LIBS_BE) $(ICU_LIBS) $(LIBURING_LIBS)
>
> But ugh, for some reason I didn't do that for LIBURING_CFLAGS. In the v1.x
> version of aio I had
> aio:src/backend/storage/aio/Makefile:override CPPFLAGS += $(LIBURING_CFLAGS)
>
> but somehow lost that somewhere along the way to v2.x
>
>
> I think I like targetting where ${LIB}_LIBS and ${LIB}_CFLAGS are applied more
> narrowly better than just adding to the global CFLAGS, CPPFLAGS, LDFLAGS.
Agreed.
> somewhat inclined to add it LIBURING_CFLAGS in src/backend rather than
> src/backend/storage/aio/ though.
>
> But I'm also willing to do it entirely differently.
The CPPFLAGS addition, located wherever makes sense, resolves that point.
> > > --- a/doc/src/sgml/installation.sgml
> > > +++ b/doc/src/sgml/installation.sgml
> >
> > lz4 and other deps have a mention in <sect1 id="install-requirements">, in
> > addition to sections edited here.
>
> Good point.
>
> Although once more I feel defeated by the ordering used :)
>
> Hm, that list is rather incomplete. At least libxml, libxslt, selinux, curl,
> uuid, systemd, selinux and bonjour aren't listed.
>
> Not sure if it makes sense to add liburing, given that?
That's a lot of preexisting incompleteness. I withdraw the point about <sect1
id="install-requirements">.
Unrelated to the above, another question about io_uring:
commit da722699 wrote:
> +/*
> + * Need to submit staged but not yet submitted IOs using the fd, otherwise
> + * the IO would end up targeting something bogus.
> + */
> +void
> +pgaio_closing_fd(int fd)
An IO in PGAIO_HS_STAGED clearly blocks closing the IO's FD, and an IO in
PGAIO_HS_COMPLETED_IO clearly doesn't block that close. For io_method=worker,
closing in PGAIO_HS_SUBMITTED is okay. For io_method=io_uring, is there a
reference about it being okay to close during PGAIO_HS_SUBMITTED? I looked
awhile for an authoritative view on that, but I didn't find one. If we can
rely on io_uring_submit() returning only after the kernel has given the
io_uring its own reference to all applicable file descriptors, I expect it's
okay to close the process's FD. If the io_uring acquires its reference later
than that, I expect we shouldn't close before that later time.
^ permalink raw reply [nested|flat] 153+ messages in thread
* Re: AIO v2.5
2025-03-04 19:00 AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-10 18:23 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-11 19:41 ` Re: AIO v2.5 Noah Misch <noah@leadboat.com>
2025-03-11 23:55 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-12 03:57 ` Re: AIO v2.5 Noah Misch <noah@leadboat.com>
2025-03-12 17:06 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-12 23:23 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-13 10:53 ` Re: AIO v2.5 Antonin Houska <ah@cybertec.at>
2025-03-14 19:43 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-17 20:00 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-18 20:12 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-21 01:58 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-23 00:20 ` Re: AIO v2.5 Noah Misch <noah@leadboat.com>
2025-03-23 15:11 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-23 15:55 ` Re: AIO v2.5 Noah Misch <noah@leadboat.com>
@ 2025-03-23 16:59 ` Andres Freund <andres@anarazel.de>
2025-03-24 01:11 ` Re: AIO v2.5 Thomas Munro <thomas.munro@gmail.com>
0 siblings, 1 reply; 153+ messages in thread
From: Andres Freund @ 2025-03-23 16:59 UTC (permalink / raw)
To: Noah Misch <noah@leadboat.com>; +Cc: Antonin Houska <ah@cybertec.at>; pgsql-hackers@postgresql.org, Thomas Munro <thomas.munro@gmail.com>; Heikki Linnakangas <hlinnaka@iki.fi>; Robert Haas <robertmhaas@gmail.com>; Jakub Wartak <jakub.wartak@enterprisedb.com>; Jelte Fennema-Nio <postgres@jeltef.nl>
Hi,
On 2025-03-23 08:55:29 -0700, Noah Misch wrote:
> On Sun, Mar 23, 2025 at 11:11:53AM -0400, Andres Freund wrote:
> Unrelated to the above, another question about io_uring:
>
> commit da722699 wrote:
> > +/*
> > + * Need to submit staged but not yet submitted IOs using the fd, otherwise
> > + * the IO would end up targeting something bogus.
> > + */
> > +void
> > +pgaio_closing_fd(int fd)
>
> An IO in PGAIO_HS_STAGED clearly blocks closing the IO's FD, and an IO in
> PGAIO_HS_COMPLETED_IO clearly doesn't block that close. For io_method=worker,
> closing in PGAIO_HS_SUBMITTED is okay. For io_method=io_uring, is there a
> reference about it being okay to close during PGAIO_HS_SUBMITTED? I looked
> awhile for an authoritative view on that, but I didn't find one. If we can
> rely on io_uring_submit() returning only after the kernel has given the
> io_uring its own reference to all applicable file descriptors, I expect it's
> okay to close the process's FD. If the io_uring acquires its reference later
> than that, I expect we shouldn't close before that later time.
I'm fairly sure io_uring has its own reference for the file descriptor by the
time io_uring_enter() returns [1]. What io_uring does *not* reliably tolerate
is the issuing process *exiting* before the IO completes, even if there are
other processes attached to the same io_uring instance.
AIO v1 had a posix_aio backend, which, on several platforms, did *not*
tolerate the FD being closed before the IO completes. Because of that
IoMethodOps had a closing_fd callback, which posix_aio used to wait for the
IO's completion [2].
I've added a test case exercising this path for all io methods. But I can't
think of a way that would catch io_uring not actually holding a reference to
the fd with a high likelihood - the IO will almost always complete quickly
enough to not be able to catch that. But it still seems better than not at all
testing the path - it does catch at least the problem of pgaio_closing_fd()
not doing anything.
Greetings,
Andres Freund
[1] See
https://github.com/torvalds/linux/blob/586de92313fcab8ed84ac5f78f4d2aae2db92c59/io_uring/io_uring.c#...
called from
https://github.com/torvalds/linux/blob/586de92313fcab8ed84ac5f78f4d2aae2db92c59/io_uring/io_uring.c#...
called from
https://github.com/torvalds/linux/blob/586de92313fcab8ed84ac5f78f4d2aae2db92c59/io_uring/io_uring.c#...
in the io_uring_enter() syscall
[2] https://github.com/anarazel/postgres/blob/a08cd717b5af4e51afb25ec86623973158a72ab9/src/backend/stora...
^ permalink raw reply [nested|flat] 153+ messages in thread
* Re: AIO v2.5
2025-03-04 19:00 AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-10 18:23 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-11 19:41 ` Re: AIO v2.5 Noah Misch <noah@leadboat.com>
2025-03-11 23:55 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-12 03:57 ` Re: AIO v2.5 Noah Misch <noah@leadboat.com>
2025-03-12 17:06 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-12 23:23 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-13 10:53 ` Re: AIO v2.5 Antonin Houska <ah@cybertec.at>
2025-03-14 19:43 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-17 20:00 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-18 20:12 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-21 01:58 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-23 00:20 ` Re: AIO v2.5 Noah Misch <noah@leadboat.com>
2025-03-23 15:11 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-23 15:55 ` Re: AIO v2.5 Noah Misch <noah@leadboat.com>
2025-03-23 16:59 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
@ 2025-03-24 01:11 ` Thomas Munro <thomas.munro@gmail.com>
0 siblings, 0 replies; 153+ messages in thread
From: Thomas Munro @ 2025-03-24 01:11 UTC (permalink / raw)
To: Andres Freund <andres@anarazel.de>; +Cc: Noah Misch <noah@leadboat.com>; Antonin Houska <ah@cybertec.at>; pgsql-hackers@postgresql.org, Heikki Linnakangas <hlinnaka@iki.fi>; Robert Haas <robertmhaas@gmail.com>; Jakub Wartak <jakub.wartak@enterprisedb.com>; Jelte Fennema-Nio <postgres@jeltef.nl>
On Mon, Mar 24, 2025 at 5:59 AM Andres Freund <andres@anarazel.de> wrote:
> On 2025-03-23 08:55:29 -0700, Noah Misch wrote:
> > An IO in PGAIO_HS_STAGED clearly blocks closing the IO's FD, and an IO in
> > PGAIO_HS_COMPLETED_IO clearly doesn't block that close. For io_method=worker,
> > closing in PGAIO_HS_SUBMITTED is okay. For io_method=io_uring, is there a
> > reference about it being okay to close during PGAIO_HS_SUBMITTED? I looked
> > awhile for an authoritative view on that, but I didn't find one. If we can
> > rely on io_uring_submit() returning only after the kernel has given the
> > io_uring its own reference to all applicable file descriptors, I expect it's
> > okay to close the process's FD. If the io_uring acquires its reference later
> > than that, I expect we shouldn't close before that later time.
>
> I'm fairly sure io_uring has its own reference for the file descriptor by the
> time io_uring_enter() returns [1]. What io_uring does *not* reliably tolerate
> is the issuing process *exiting* before the IO completes, even if there are
> other processes attached to the same io_uring instance.
It is a bit strange that the documentation doesn't say that
explicitly. You can sorta-maybe-kinda infer it from the fact that
io_uring didn't originally support cancelling requests at all, maybe a
small clue that it also didn't cancel them when you closed the fd :-)
The only sane alternative would seem to be that they keep running and
have their own reference to the *file* (not the fd), which is the
actual case, and might also be inferrable at a stretch from the
io_uring_register() documentation that says it reduces overheads with
a "long term reference" reducing "per-I/O overhead". (The distant
third option/non-option is a sort of late/async binding fd as seen in
the Glibc user space POSIX AIO implementation, but that sort of
madness doesn't seem to be the sort of thing anyone working in the
kernel would entertain for a nanosecond...) Anyway, there are also
public discussions involving Mr Axboe that discuss the fact that async
operations continue to run when the associated fd is closed, eg from
people who were surprised by that when porting stuff from other
systems, which might help fill in the documentation gap a teensy bit
if people want to see something outside the source code:
https://github.com/axboe/liburing/issues/568
> AIO v1 had a posix_aio backend, which, on several platforms, did *not*
> tolerate the FD being closed before the IO completes. Because of that
> IoMethodOps had a closing_fd callback, which posix_aio used to wait for the
> IO's completion [2].
Just for the record while remembering this stuff: Windows is another
system that took the cancel-on-close approach, so the Windows IOCP
proof-of-concept patches also used that AIO v1 callback and we'll have
to think about that again if/when we want to get that stuff
going on AIO v2. I recall also speculating that it might be better to
teach the vfd system to pick another victim to close instead if an fd
was currently tied up with an asynchronous I/O for the benefit of
those cancel-on-close systems, hopefully without any happy-path
book-keeping. But just submitting staged I/O is a nice and cheap
solution for now, without them in the picture.
^ permalink raw reply [nested|flat] 153+ messages in thread
* Re: AIO v2.5
2025-03-04 19:00 AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-10 18:23 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-11 19:41 ` Re: AIO v2.5 Noah Misch <noah@leadboat.com>
2025-03-11 23:55 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-12 03:57 ` Re: AIO v2.5 Noah Misch <noah@leadboat.com>
2025-03-12 17:06 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-12 23:23 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-13 10:53 ` Re: AIO v2.5 Antonin Houska <ah@cybertec.at>
2025-03-14 19:43 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-17 20:00 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-18 20:12 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-21 01:58 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
@ 2025-03-23 02:09 ` Noah Misch <noah@leadboat.com>
2025-03-23 15:57 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
4 siblings, 1 reply; 153+ messages in thread
From: Noah Misch @ 2025-03-23 02:09 UTC (permalink / raw)
To: Andres Freund <andres@anarazel.de>; +Cc: Antonin Houska <ah@cybertec.at>; pgsql-hackers@postgresql.org, Thomas Munro <thomas.munro@gmail.com>; Heikki Linnakangas <hlinnaka@iki.fi>; Robert Haas <robertmhaas@gmail.com>; Jakub Wartak <jakub.wartak@enterprisedb.com>; Jelte Fennema-Nio <postgres@jeltef.nl>
On Thu, Mar 20, 2025 at 09:58:37PM -0400, Andres Freund wrote:
> Attached v2.11
> Subject: [PATCH v2.11 05/27] aio: Add io_method=io_uring
Apart from some isolated cosmetic points, this is ready to commit:
> + ereport(ERROR,
> + errcode(err),
> + errmsg("io_uring_queue_init failed: %m"),
> + hint != NULL ? errhint("%s", hint) : 0);
https://www.postgresql.org/docs/current/error-style-guide.html gives the example:
BAD: open() failed: %m
BETTER: could not open file %s: %m
Hence, this errmsg should change, perhaps to:
"could not setup io_uring queues: %m".
> + pgaio_debug_io(DEBUG3, ioh,
> + "wait_one io_gen: %llu, ref_gen: %llu, cycle %d",
> + (long long unsigned) ref_generation,
> + (long long unsigned) ioh->generation,
In the message string, io_gen appears before ref_gen. In the subsequent args,
the order is swapped relative to the message string.
> --- a/src/backend/utils/activity/wait_event_names.txt
> +++ b/src/backend/utils/activity/wait_event_names.txt
> @@ -192,6 +192,8 @@ ABI_compatibility:
>
> Section: ClassName - WaitEventIO
>
> +AIO_IO_URING_SUBMIT "Waiting for IO submission via io_uring."
> +AIO_IO_URING_COMPLETION "Waiting for IO completion via io_uring."
> AIO_IO_COMPLETION "Waiting for IO completion."
I'm wondering if there's an opportunity to enrich the last two wait event
names and/or descriptions. The current descriptions suggest to me more
similarity than is actually there. Inputs to the decision:
- AIO_IO_COMPLETION waits for an IO in PGAIO_HS_DEFINED, PGAIO_HS_STAGED, or
PGAIO_HS_COMPLETED_IO to reach PGAIO_HS_COMPLETED_SHARED. The three
starting states are the states where some other backend owns the next
action, so the current backend can only wait to be signaled.
- AIO_IO_URING_COMPLETION waits for the kernel to do enough so we can move
from PGAIO_HS_SUBMITTED to PGAIO_HS_COMPLETED_IO.
Possible names and descriptions, based on PgAioHandleState enum names and
comments:
AIO_IO_URING_COMPLETED_IO "Waiting for IO result via io_uring."
AIO_COMPLETED_SHARED "Waiting for IO shared completion callback."
If "shared completion callback" is too internals-focused, perhaps this:
AIO_IO_URING_COMPLETED_IO "Waiting for IO result via io_uring."
AIO_COMPLETED_SHARED "Waiting for IO completion to update shared memory."
> --- a/doc/src/sgml/config.sgml
> +++ b/doc/src/sgml/config.sgml
> @@ -2710,6 +2710,12 @@ include_dir 'conf.d'
> <literal>worker</literal> (execute asynchronous I/O using worker processes)
> </para>
> </listitem>
> + <listitem>
> + <para>
> + <literal>io_uring</literal> (execute asynchronous I/O using
> + io_uring, if available)
I feel the "if available" doesn't quite fit, since we'll fail if unavailable.
Maybe just "(execute asynchronous I/O using Linux io_uring)" with "Linux"
there to reduce surprise on other platforms.
> Subject: [PATCH v2.11 06/27] aio: Implement support for reads in smgr/md/fd
(Still reviewing this one.)
^ permalink raw reply [nested|flat] 153+ messages in thread
* Re: AIO v2.5
2025-03-04 19:00 AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-10 18:23 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-11 19:41 ` Re: AIO v2.5 Noah Misch <noah@leadboat.com>
2025-03-11 23:55 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-12 03:57 ` Re: AIO v2.5 Noah Misch <noah@leadboat.com>
2025-03-12 17:06 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-12 23:23 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-13 10:53 ` Re: AIO v2.5 Antonin Houska <ah@cybertec.at>
2025-03-14 19:43 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-17 20:00 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-18 20:12 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-21 01:58 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-23 02:09 ` Re: AIO v2.5 Noah Misch <noah@leadboat.com>
@ 2025-03-23 15:57 ` Andres Freund <andres@anarazel.de>
2025-03-23 16:32 ` Re: AIO v2.5 Noah Misch <noah@leadboat.com>
0 siblings, 1 reply; 153+ messages in thread
From: Andres Freund @ 2025-03-23 15:57 UTC (permalink / raw)
To: Noah Misch <noah@leadboat.com>; +Cc: Antonin Houska <ah@cybertec.at>; pgsql-hackers@postgresql.org, Thomas Munro <thomas.munro@gmail.com>; Heikki Linnakangas <hlinnaka@iki.fi>; Robert Haas <robertmhaas@gmail.com>; Jakub Wartak <jakub.wartak@enterprisedb.com>; Jelte Fennema-Nio <postgres@jeltef.nl>
Hi,
On 2025-03-22 19:09:55 -0700, Noah Misch wrote:
> On Thu, Mar 20, 2025 at 09:58:37PM -0400, Andres Freund wrote:
> > Attached v2.11
>
> > Subject: [PATCH v2.11 05/27] aio: Add io_method=io_uring
>
> Apart from some isolated cosmetic points, this is ready to commit:
>
> > + ereport(ERROR,
> > + errcode(err),
> > + errmsg("io_uring_queue_init failed: %m"),
> > + hint != NULL ? errhint("%s", hint) : 0);
>
> https://www.postgresql.org/docs/current/error-style-guide.html gives the example:
>
> BAD: open() failed: %m
> BETTER: could not open file %s: %m
>
> Hence, this errmsg should change, perhaps to:
> "could not setup io_uring queues: %m".
You're right. I didn't intentionally "violate" the policy, but I do have to
admit, I'm not a huge fan of that aspect, it just obfuscates what actually
failed, forcing one to look at the code or strace to figure out what precisely
failed.
(Changed)
> > + pgaio_debug_io(DEBUG3, ioh,
> > + "wait_one io_gen: %llu, ref_gen: %llu, cycle %d",
> > + (long long unsigned) ref_generation,
> > + (long long unsigned) ioh->generation,
>
> In the message string, io_gen appears before ref_gen. In the subsequent args,
> the order is swapped relative to the message string.
Oops, you're right.
> > --- a/src/backend/utils/activity/wait_event_names.txt
> > +++ b/src/backend/utils/activity/wait_event_names.txt
> > @@ -192,6 +192,8 @@ ABI_compatibility:
> >
> > Section: ClassName - WaitEventIO
> >
> > +AIO_IO_URING_SUBMIT "Waiting for IO submission via io_uring."
> > +AIO_IO_URING_COMPLETION "Waiting for IO completion via io_uring."
> > AIO_IO_COMPLETION "Waiting for IO completion."
>
> I'm wondering if there's an opportunity to enrich the last two wait event
> names and/or descriptions. The current descriptions suggest to me more
> similarity than is actually there. Inputs to the decision:
>
> - AIO_IO_COMPLETION waits for an IO in PGAIO_HS_DEFINED, PGAIO_HS_STAGED, or
> PGAIO_HS_COMPLETED_IO to reach PGAIO_HS_COMPLETED_SHARED. The three
> starting states are the states where some other backend owns the next
> action, so the current backend can only wait to be signaled.
>
> - AIO_IO_URING_COMPLETION waits for the kernel to do enough so we can move
> from PGAIO_HS_SUBMITTED to PGAIO_HS_COMPLETED_IO.
>
> Possible names and descriptions, based on PgAioHandleState enum names and
> comments:
>
> AIO_IO_URING_COMPLETED_IO "Waiting for IO result via io_uring."
> AIO_COMPLETED_SHARED "Waiting for IO shared completion callback."
>
> If "shared completion callback" is too internals-focused, perhaps this:
>
> AIO_IO_URING_COMPLETED_IO "Waiting for IO result via io_uring."
> AIO_COMPLETED_SHARED "Waiting for IO completion to update shared memory."
Hm, right now AIO_IO_COMPLETION also covers the actual "raw" execution of the
IO with io_method=worker/sync. For that AIO_COMPLETED_SHARED would be
inappropriate.
We could use a different wait event if wait for an IO via CV in
PGAIO_HS_SUBMITTED, with a small refactoring of pgaio_io_wait(). But I'm not
sure that would get you that far - we don't broadcast the CV when
transitioning from PGAIO_HS_SUBMITTED -> PGAIO_HS_COMPLETED_IO, so the wait
event would stay the same, now wrong, wait event until the shared callback
completes. Obviously waking everyone up just so they can use a differen wait
event doesn't make sense.
A more minimal change would be to narrow AIO_IO_URING_COMPLETION to
"execution" or something like that, to hint at a separation between the raw IO
being completed and the IO, including the callbacks completing.
> > --- a/doc/src/sgml/config.sgml
> > +++ b/doc/src/sgml/config.sgml
> > @@ -2710,6 +2710,12 @@ include_dir 'conf.d'
> > <literal>worker</literal> (execute asynchronous I/O using worker processes)
> > </para>
> > </listitem>
> > + <listitem>
> > + <para>
> > + <literal>io_uring</literal> (execute asynchronous I/O using
> > + io_uring, if available)
>
> I feel the "if available" doesn't quite fit, since we'll fail if unavailable.
> Maybe just "(execute asynchronous I/O using Linux io_uring)" with "Linux"
> there to reduce surprise on other platforms.
You're right, the if available can be misunderstood. But not mentioning that
it's an optional dependency seems odd too. What about something like
<para>
<literal>io_uring</literal> (execute asynchronous I/O using
io_uring, requires postgres to have been built with
<link linkend="configure-option-with-liburing"><option>--with-liburing</option></link> /
<link linkend="configure-with-liburing-meson"><option>-Dliburing</option></link>)
</para>
Should the docs for --with-liburing/-Dliburing mention it's linux only? We
don't seem to do that for things like systemd (linux), selinux (linux) and
only kinda for bonjour (macos).
Greetings,
Andres Freund
^ permalink raw reply [nested|flat] 153+ messages in thread
* Re: AIO v2.5
2025-03-04 19:00 AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-10 18:23 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-11 19:41 ` Re: AIO v2.5 Noah Misch <noah@leadboat.com>
2025-03-11 23:55 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-12 03:57 ` Re: AIO v2.5 Noah Misch <noah@leadboat.com>
2025-03-12 17:06 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-12 23:23 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-13 10:53 ` Re: AIO v2.5 Antonin Houska <ah@cybertec.at>
2025-03-14 19:43 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-17 20:00 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-18 20:12 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-21 01:58 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-23 02:09 ` Re: AIO v2.5 Noah Misch <noah@leadboat.com>
2025-03-23 15:57 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
@ 2025-03-23 16:32 ` Noah Misch <noah@leadboat.com>
2025-03-25 01:40 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
0 siblings, 1 reply; 153+ messages in thread
From: Noah Misch @ 2025-03-23 16:32 UTC (permalink / raw)
To: Andres Freund <andres@anarazel.de>; +Cc: Antonin Houska <ah@cybertec.at>; pgsql-hackers@postgresql.org, Thomas Munro <thomas.munro@gmail.com>; Heikki Linnakangas <hlinnaka@iki.fi>; Robert Haas <robertmhaas@gmail.com>; Jakub Wartak <jakub.wartak@enterprisedb.com>; Jelte Fennema-Nio <postgres@jeltef.nl>
On Sun, Mar 23, 2025 at 11:57:48AM -0400, Andres Freund wrote:
> On 2025-03-22 19:09:55 -0700, Noah Misch wrote:
> > On Thu, Mar 20, 2025 at 09:58:37PM -0400, Andres Freund wrote:
> > > Attached v2.11
> >
> > > Subject: [PATCH v2.11 05/27] aio: Add io_method=io_uring
> > > --- a/src/backend/utils/activity/wait_event_names.txt
> > > +++ b/src/backend/utils/activity/wait_event_names.txt
> > > @@ -192,6 +192,8 @@ ABI_compatibility:
> > >
> > > Section: ClassName - WaitEventIO
> > >
> > > +AIO_IO_URING_SUBMIT "Waiting for IO submission via io_uring."
> > > +AIO_IO_URING_COMPLETION "Waiting for IO completion via io_uring."
> > > AIO_IO_COMPLETION "Waiting for IO completion."
> >
> > I'm wondering if there's an opportunity to enrich the last two wait event
> > names and/or descriptions. The current descriptions suggest to me more
> > similarity than is actually there. Inputs to the decision:
> >
> > - AIO_IO_COMPLETION waits for an IO in PGAIO_HS_DEFINED, PGAIO_HS_STAGED, or
> > PGAIO_HS_COMPLETED_IO to reach PGAIO_HS_COMPLETED_SHARED. The three
> > starting states are the states where some other backend owns the next
> > action, so the current backend can only wait to be signaled.
> >
> > - AIO_IO_URING_COMPLETION waits for the kernel to do enough so we can move
> > from PGAIO_HS_SUBMITTED to PGAIO_HS_COMPLETED_IO.
> >
> > Possible names and descriptions, based on PgAioHandleState enum names and
> > comments:
> >
> > AIO_IO_URING_COMPLETED_IO "Waiting for IO result via io_uring."
> > AIO_COMPLETED_SHARED "Waiting for IO shared completion callback."
> >
> > If "shared completion callback" is too internals-focused, perhaps this:
> >
> > AIO_IO_URING_COMPLETED_IO "Waiting for IO result via io_uring."
> > AIO_COMPLETED_SHARED "Waiting for IO completion to update shared memory."
>
> Hm, right now AIO_IO_COMPLETION also covers the actual "raw" execution of the
> IO with io_method=worker/sync.
Right, it could start with the IO in PGAIO_HS_DEFINED and end with the IO in
PGAIO_HS_COMPLETED_SHARED. So another part of the wait may be the definer
doing work before exiting batch mode.
> For that AIO_COMPLETED_SHARED would be
> inappropriate.
The concept I had in mind was "waiting to reach PGAIO_HS_COMPLETED_SHARED,
whatever obstacles that involves".
Another candidate description string:
AIO_COMPLETED_SHARED "Waiting for another process to complete IO."
> We could use a different wait event if wait for an IO via CV in
> PGAIO_HS_SUBMITTED, with a small refactoring of pgaio_io_wait(). But I'm not
> sure that would get you that far - we don't broadcast the CV when
> transitioning from PGAIO_HS_SUBMITTED -> PGAIO_HS_COMPLETED_IO, so the wait
> event would stay the same, now wrong, wait event until the shared callback
> completes. Obviously waking everyone up just so they can use a differen wait
> event doesn't make sense.
Agreed. The mapping of code ranges to wait events seems fine to me. I'm mainly
trying to optimize the wait event description strings to fit those code ranges.
> A more minimal change would be to narrow AIO_IO_URING_COMPLETION to
> "execution" or something like that, to hint at a separation between the raw IO
> being completed and the IO, including the callbacks completing.
Yes, that would work for me.
> > > --- a/doc/src/sgml/config.sgml
> > > +++ b/doc/src/sgml/config.sgml
> > > @@ -2710,6 +2710,12 @@ include_dir 'conf.d'
> > > <literal>worker</literal> (execute asynchronous I/O using worker processes)
> > > </para>
> > > </listitem>
> > > + <listitem>
> > > + <para>
> > > + <literal>io_uring</literal> (execute asynchronous I/O using
> > > + io_uring, if available)
> >
> > I feel the "if available" doesn't quite fit, since we'll fail if unavailable.
> > Maybe just "(execute asynchronous I/O using Linux io_uring)" with "Linux"
> > there to reduce surprise on other platforms.
>
> You're right, the if available can be misunderstood. But not mentioning that
> it's an optional dependency seems odd too. What about something like
>
> <para>
> <literal>io_uring</literal> (execute asynchronous I/O using
> io_uring, requires postgres to have been built with
> <link linkend="configure-option-with-liburing"><option>--with-liburing</option></link> /
> <link linkend="configure-with-liburing-meson"><option>-Dliburing</option></link>)
> </para>
I'd change s/postgres to have been built/a build with/ since the SGML docs
don't use the term "postgres" that way. Otherwise, that works for me.
> Should the docs for --with-liburing/-Dliburing mention it's linux only? We
> don't seem to do that for things like systemd (linux), selinux (linux) and
> only kinda for bonjour (macos).
No need, I think.
^ permalink raw reply [nested|flat] 153+ messages in thread
* Re: AIO v2.5
2025-03-04 19:00 AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-10 18:23 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-11 19:41 ` Re: AIO v2.5 Noah Misch <noah@leadboat.com>
2025-03-11 23:55 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-12 03:57 ` Re: AIO v2.5 Noah Misch <noah@leadboat.com>
2025-03-12 17:06 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-12 23:23 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-13 10:53 ` Re: AIO v2.5 Antonin Houska <ah@cybertec.at>
2025-03-14 19:43 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-17 20:00 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-18 20:12 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-21 01:58 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-23 02:09 ` Re: AIO v2.5 Noah Misch <noah@leadboat.com>
2025-03-23 15:57 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-23 16:32 ` Re: AIO v2.5 Noah Misch <noah@leadboat.com>
@ 2025-03-25 01:40 ` Andres Freund <andres@anarazel.de>
0 siblings, 0 replies; 153+ messages in thread
From: Andres Freund @ 2025-03-25 01:40 UTC (permalink / raw)
To: Noah Misch <noah@leadboat.com>; +Cc: Antonin Houska <ah@cybertec.at>; pgsql-hackers@postgresql.org, Thomas Munro <thomas.munro@gmail.com>; Heikki Linnakangas <hlinnaka@iki.fi>; Robert Haas <robertmhaas@gmail.com>; Jakub Wartak <jakub.wartak@enterprisedb.com>; Jelte Fennema-Nio <postgres@jeltef.nl>
Hi,
On 2025-03-23 09:32:48 -0700, Noah Misch wrote:
> Another candidate description string:
>
> AIO_COMPLETED_SHARED "Waiting for another process to complete IO."
I liked that one and adopted it.
> > A more minimal change would be to narrow AIO_IO_URING_COMPLETION to
> > "execution" or something like that, to hint at a separation between the raw IO
> > being completed and the IO, including the callbacks completing.
>
> Yes, that would work for me.
I updated both the name and the description of this one to EXECUTION, but I'm
not sure I like it for the name...
> > > > --- a/doc/src/sgml/config.sgml
> > > > +++ b/doc/src/sgml/config.sgml
> > > > @@ -2710,6 +2710,12 @@ include_dir 'conf.d'
> > > > <literal>worker</literal> (execute asynchronous I/O using worker processes)
> > > > </para>
> > > > </listitem>
> > > > + <listitem>
> > > > + <para>
> > > > + <literal>io_uring</literal> (execute asynchronous I/O using
> > > > + io_uring, if available)
> > >
> > > I feel the "if available" doesn't quite fit, since we'll fail if unavailable.
> > > Maybe just "(execute asynchronous I/O using Linux io_uring)" with "Linux"
> > > there to reduce surprise on other platforms.
> >
> > You're right, the if available can be misunderstood. But not mentioning that
> > it's an optional dependency seems odd too. What about something like
> >
> > <para>
> > <literal>io_uring</literal> (execute asynchronous I/O using
> > io_uring, requires postgres to have been built with
> > <link linkend="configure-option-with-liburing"><option>--with-liburing</option></link> /
> > <link linkend="configure-with-liburing-meson"><option>-Dliburing</option></link>)
> > </para>
>
> I'd change s/postgres to have been built/a build with/ since the SGML docs
> don't use the term "postgres" that way. Otherwise, that works for me.
Went with that.
Greetings,
Andres Freund
^ permalink raw reply [nested|flat] 153+ messages in thread
* Re: AIO v2.5
2025-03-04 19:00 AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-10 18:23 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-11 19:41 ` Re: AIO v2.5 Noah Misch <noah@leadboat.com>
2025-03-11 23:55 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-12 03:57 ` Re: AIO v2.5 Noah Misch <noah@leadboat.com>
2025-03-12 17:06 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-12 23:23 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-13 10:53 ` Re: AIO v2.5 Antonin Houska <ah@cybertec.at>
2025-03-14 19:43 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-17 20:00 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-18 20:12 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-21 01:58 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
@ 2025-03-24 00:29 ` Noah Misch <noah@leadboat.com>
2025-03-24 15:43 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
4 siblings, 1 reply; 153+ messages in thread
From: Noah Misch @ 2025-03-24 00:29 UTC (permalink / raw)
To: Andres Freund <andres@anarazel.de>; +Cc: Antonin Houska <ah@cybertec.at>; pgsql-hackers@postgresql.org, Thomas Munro <thomas.munro@gmail.com>; Heikki Linnakangas <hlinnaka@iki.fi>; Robert Haas <robertmhaas@gmail.com>; Jakub Wartak <jakub.wartak@enterprisedb.com>; Jelte Fennema-Nio <postgres@jeltef.nl>
commit 247ce06b wrote:
> + pgaio_io_reopen(ioh);
> +
> + /*
> + * To be able to exercise the reopen-fails path, allow injection
> + * points to trigger a failure at this point.
> + */
> + pgaio_io_call_inj(ioh, "AIO_WORKER_AFTER_REOPEN");
> +
> + error_errno = 0;
> + error_ioh = NULL;
> +
> + /*
> + * We don't expect this to ever fail with ERROR or FATAL, no need
> + * to keep error_ioh set to the IO.
> + * pgaio_io_perform_synchronously() contains a critical section to
> + * ensure we don't accidentally fail.
> + */
> + pgaio_io_perform_synchronously(ioh);
A CHECK_FOR_INTERRUPTS() could close() the FD that pgaio_io_reopen() callee
smgr_aio_reopen() stores. Hence, I think smgrfd() should assert that
interrupts are held instead of doing its own HOLD_INTERRUPTS(), and a
HOLD_INTERRUPTS() should surround the above region of code. It's likely hard
to reproduce a problem, because pgaio_io_call_inj() does nothing in many
builds, and pgaio_io_perform_synchronously() starts by entering a critical
section.
On Thu, Mar 20, 2025 at 09:58:37PM -0400, Andres Freund wrote:
> Attached v2.11
> Subject: [PATCH v2.11 06/27] aio: Implement support for reads in smgr/md/fd
> +int
> +FileStartReadV(PgAioHandle *ioh, File file,
> + int iovcnt, off_t offset,
> + uint32 wait_event_info)
> +{
> + int returnCode;
> + Vfd *vfdP;
> +
> + Assert(FileIsValid(file));
> +
> + DO_DB(elog(LOG, "FileStartReadV: %d (%s) " INT64_FORMAT " %d",
> + file, VfdCache[file].fileName,
> + (int64) offset,
> + iovcnt));
> +
> + returnCode = FileAccess(file);
> + if (returnCode < 0)
> + return returnCode;
> +
> + vfdP = &VfdCache[file];
> +
> + pgaio_io_prep_readv(ioh, vfdP->fd, iovcnt, offset);
FileStartReadV() and pgaio_io_prep_readv() advance the IO to PGAIO_HS_STAGED
w/ batch mode, PGAIO_HS_SUBMITTED w/o batch mode. I didn't expect that from
functions so named. The "start" verb sounds to me like unconditional
PGAIO_HS_SUBMITTED, and the "prep" verb sounds like PGAIO_HS_DEFINED. I like
the "stage" verb, because it matches PGAIO_HS_STAGED, and the comment at
PGAIO_HS_STAGED succinctly covers what to expect. Hence, I recommend names
FileStageReadV, pgaio_io_stage_readv, mdstagereadv, and smgrstageread. How do
you see it?
> +/*
> + * AIO error reporting callback for mdstartreadv().
> + *
> + * Errors are encoded as follows:
> + * - PgAioResult.error_data != 0 encodes IO that failed with errno != 0
I recommend replacing "errno != 0" with either "that errno" or "errno ==
error_data".
> Subject: [PATCH v2.11 07/27] aio: Add README.md explaining higher level design
Ready for commit apart from some trivia:
> +if (ioret.result.status == PGAIO_RS_ERROR)
> + pgaio_result_report(aio_ret.result, &aio_ret.target_data, ERROR);
I think ioret and aio_ret are supposed to be the same object. If that's
right, change one of the names. Likewise elsewhere in this file.
> +The central API piece for postgres' AIO abstraction are AIO handles. To
> +execute an IO one first has to acquire an IO handle (`pgaio_io_acquire()`) and
> +then "defined", i.e. associate an IO operation with the handle.
s/"defined"/"define" it/ or similar
> +The "solution" to this the ability to associate multiple completion callbacks
s/this the/this is the/
> Subject: [PATCH v2.11 08/27] localbuf: Track pincount in BufferDesc as well
> @@ -5350,6 +5350,18 @@ ConditionalLockBufferForCleanup(Buffer buffer)
> Assert(refcount > 0);
> if (refcount != 1)
> return false;
> +
> + /*
> + * Check that the AIO subsystem doesn't have a pin. Likely not
> + * possible today, but better safe than sorry.
> + */
> + bufHdr = GetLocalBufferDescriptor(-buffer - 1);
> + buf_state = pg_atomic_read_u32(&bufHdr->state);
> + refcount = BUF_STATE_GET_REFCOUNT(buf_state);
> + Assert(refcount > 0);
> + if (refcount != 1)
> + return false;
> +
LockBufferForCleanup() should get code like this
ConditionalLockBufferForCleanup() code, either now or when "not possible
today" ends. Currently, it just assumes all local buffers are
cleanup-lockable:
/* Nobody else to wait for */
if (BufferIsLocal(buffer))
return;
> @@ -570,7 +577,13 @@ InvalidateLocalBuffer(BufferDesc *bufHdr, bool check_unreferenced)
>
> buf_state = pg_atomic_read_u32(&bufHdr->state);
>
> - if (check_unreferenced && LocalRefCount[bufid] != 0)
> + /*
> + * We need to test not just LocalRefCount[bufid] but also the BufferDesc
> + * itself, as the latter is used to represent a pin by the AIO subsystem.
> + * This can happen if AIO is initiated and then the query errors out.
> + */
> + if (check_unreferenced &&
> + (LocalRefCount[bufid] != 0 || BUF_STATE_GET_REFCOUNT(buf_state) != 0))
> elog(ERROR, "block %u of %s is still referenced (local %u)",
I didn't write a test to prove it, but I'm suspecting we'll reach the above
ERROR with this sequence:
CREATE TEMP TABLE foo ...;
[some command that starts reading a block of foo into local buffers, then ERROR with IO ongoing]
DROP TABLE foo;
DropRelationAllLocalBuffers() calls InvalidateLocalBuffer(bufHdr, true). I
think we'd need to do like pgaio_shutdown() and finish all IOs (or all IOs for
the particular rel) before InvalidateLocalBuffer(). Or use something like the
logic near elog(ERROR, "buffer is pinned in InvalidateBuffer") in
corresponding bufmgr code. I think that bufmgr ERROR is unreachable, since
only a private refcnt triggers that bufmgr ERROR. Is there something
preventing the localbuf error from being a problem? (This wouldn't require
changes to the current patch; responsibility would fall in a bufmgr AIO
patch.)
> Subject: [PATCH v2.11 09/27] bufmgr: Implement AIO read support
(Still reviewing this and later patches, but incidental observations follow.)
> +buffer_readv_complete_one(uint8 buf_off, Buffer buffer, uint8 flags,
> + bool failed, bool is_temp)
> +{
...
> + PgAioResult result;
...
> + result.status = PGAIO_RS_OK;
...
> + return result;
gcc 14.2.0 -Werror gives me:
bufmgr.c:7297:16: error: ‘result’ may be used uninitialized [-Werror=maybe-uninitialized]
Zeroing the unset fields silenced it:
--- a/src/backend/storage/buffer/bufmgr.c
+++ b/src/backend/storage/buffer/bufmgr.c
@@ -7221,3 +7221,3 @@ buffer_readv_complete_one(uint8 buf_off, Buffer buffer, uint8 flags,
char *bufdata = BufferGetBlock(buffer);
- PgAioResult result;
+ PgAioResult result = { .status = PGAIO_RS_OK };
uint32 set_flag_bits;
@@ -7238,4 +7238,2 @@ buffer_readv_complete_one(uint8 buf_off, Buffer buffer, uint8 flags,
- result.status = PGAIO_RS_OK;
-
/* check for garbage data */
> Subject: [PATCH v2.11 13/27] aio: Basic read_stream adjustments for real AIO
> @@ -416,6 +418,13 @@ read_stream_start_pending_read(ReadStream *stream)
> static void
> read_stream_look_ahead(ReadStream *stream)
> {
> + /*
> + * Allow amortizing the cost of submitting IO over multiple IOs. This
> + * requires that we don't do any operations that could lead to a deadlock
> + * with staged-but-unsubmitted IO.
> + */
> + pgaio_enter_batchmode();
We call read_stream_get_block() while in batchmode, so the stream callback
needs to be ready for that. A complicated case is
collect_corrupt_items_read_stream_next_block(), which may do its own buffer
I/O to read in a vmbuffer for VM_ALL_FROZEN(). That's feeling to me like a
recipe for corner cases reaching ERROR "starting batch while batch already in
progress". Are there mitigating factors?
> Subject: [PATCH v2.11 17/27] aio: Add test_aio module
> + # verify that page verification errors are detected even as part of a
> + # shortened multi-block read (tbl_corr, block 1 is tbl_corred)
Is "tbl_corred" a typo of something?
> --- /dev/null
> +++ b/src/test/modules/test_aio/test_aio.c
> @@ -0,0 +1,657 @@
> +/*-------------------------------------------------------------------------
> + *
> + * delay_execution.c
> + * Test module to allow delay between parsing and execution of a query.
> + *
> + * The delay is implemented by taking and immediately releasing a specified
> + * advisory lock. If another process has previously taken that lock, the
> + * current process will be blocked until the lock is released; otherwise,
> + * there's no effect. This allows an isolationtester script to reliably
> + * test behaviors where some specified action happens in another backend
> + * between parsing and execution of any desired query.
> + *
> + * Copyright (c) 2020-2025, PostgreSQL Global Development Group
> + *
> + * IDENTIFICATION
> + * src/test/modules/delay_execution/delay_execution.c
Header comment is surviving from copy-paste of delay_execution.c.
> + * Tor tests we don't want the resowner release preventing us from
s/Tor/For/
^ permalink raw reply [nested|flat] 153+ messages in thread
* Re: AIO v2.5
2025-03-04 19:00 AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-10 18:23 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-11 19:41 ` Re: AIO v2.5 Noah Misch <noah@leadboat.com>
2025-03-11 23:55 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-12 03:57 ` Re: AIO v2.5 Noah Misch <noah@leadboat.com>
2025-03-12 17:06 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-12 23:23 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-13 10:53 ` Re: AIO v2.5 Antonin Houska <ah@cybertec.at>
2025-03-14 19:43 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-17 20:00 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-18 20:12 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-21 01:58 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-24 00:29 ` Re: AIO v2.5 Noah Misch <noah@leadboat.com>
@ 2025-03-24 15:43 ` Andres Freund <andres@anarazel.de>
2025-03-24 22:55 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
0 siblings, 1 reply; 153+ messages in thread
From: Andres Freund @ 2025-03-24 15:43 UTC (permalink / raw)
To: Noah Misch <noah@leadboat.com>; +Cc: Antonin Houska <ah@cybertec.at>; pgsql-hackers@postgresql.org, Thomas Munro <thomas.munro@gmail.com>; Heikki Linnakangas <hlinnaka@iki.fi>; Robert Haas <robertmhaas@gmail.com>; Jakub Wartak <jakub.wartak@enterprisedb.com>; Jelte Fennema-Nio <postgres@jeltef.nl>
Hi,
On 2025-03-23 17:29:39 -0700, Noah Misch wrote:
> commit 247ce06b wrote:
> > + pgaio_io_reopen(ioh);
> > +
> > + /*
> > + * To be able to exercise the reopen-fails path, allow injection
> > + * points to trigger a failure at this point.
> > + */
> > + pgaio_io_call_inj(ioh, "AIO_WORKER_AFTER_REOPEN");
> > +
> > + error_errno = 0;
> > + error_ioh = NULL;
> > +
> > + /*
> > + * We don't expect this to ever fail with ERROR or FATAL, no need
> > + * to keep error_ioh set to the IO.
> > + * pgaio_io_perform_synchronously() contains a critical section to
> > + * ensure we don't accidentally fail.
> > + */
> > + pgaio_io_perform_synchronously(ioh);
>
> A CHECK_FOR_INTERRUPTS() could close() the FD that pgaio_io_reopen() callee
> smgr_aio_reopen() stores. Hence, I think smgrfd() should assert that
> interrupts are held instead of doing its own HOLD_INTERRUPTS(), and a
> HOLD_INTERRUPTS() should surround the above region of code. It's likely hard
> to reproduce a problem, because pgaio_io_call_inj() does nothing in many
> builds, and pgaio_io_perform_synchronously() starts by entering a critical
> section.
Hm, I guess you're right - it would be pretty bonkers for the injection to
process interrupts, but its much better to clarify the code to make that not
an option. Once doing that it seemed we should also have a similar assertion
in pgaio_io_before_prep() would be appropriate.
> On Thu, Mar 20, 2025 at 09:58:37PM -0400, Andres Freund wrote:
> > Attached v2.11
>
> > Subject: [PATCH v2.11 06/27] aio: Implement support for reads in smgr/md/fd
>
> > +int
> > +FileStartReadV(PgAioHandle *ioh, File file,
> > + int iovcnt, off_t offset,
> > + uint32 wait_event_info)
> > +{
> > + int returnCode;
> > + Vfd *vfdP;
> > +
> > + Assert(FileIsValid(file));
> > +
> > + DO_DB(elog(LOG, "FileStartReadV: %d (%s) " INT64_FORMAT " %d",
> > + file, VfdCache[file].fileName,
> > + (int64) offset,
> > + iovcnt));
> > +
> > + returnCode = FileAccess(file);
> > + if (returnCode < 0)
> > + return returnCode;
> > +
> > + vfdP = &VfdCache[file];
> > +
> > + pgaio_io_prep_readv(ioh, vfdP->fd, iovcnt, offset);
>
> FileStartReadV() and pgaio_io_prep_readv() advance the IO to PGAIO_HS_STAGED
> w/ batch mode, PGAIO_HS_SUBMITTED w/o batch mode. I didn't expect that from
> functions so named. The "start" verb sounds to me like unconditional
> PGAIO_HS_SUBMITTED, and the "prep" verb sounds like PGAIO_HS_DEFINED. I like
> the "stage" verb, because it matches PGAIO_HS_STAGED, and the comment at
> PGAIO_HS_STAGED succinctly covers what to expect. Hence, I recommend names
> FileStageReadV, pgaio_io_stage_readv, mdstagereadv, and smgrstageread. How do
> you see it?
I have a surprisingly strong negative reaction to that proposed naming. To me
the staging is a distinct step that happens *after* the IO is fully
defined. Making all the layered calls that lead up to that named that way
would IMO be a bad idea.
I however don't particularly like the *start* or *prep* names, I've gone back
and forth on those a couple times. I could see "begin" work uniformly across
those.
> > +/*
> > + * AIO error reporting callback for mdstartreadv().
> > + *
> > + * Errors are encoded as follows:
> > + * - PgAioResult.error_data != 0 encodes IO that failed with errno != 0
>
> I recommend replacing "errno != 0" with either "that errno" or "errno ==
> error_data".
Done.
> > Subject: [PATCH v2.11 07/27] aio: Add README.md explaining higher level design
>
> Ready for commit apart from some trivia:
Great.
> > +if (ioret.result.status == PGAIO_RS_ERROR)
> > + pgaio_result_report(aio_ret.result, &aio_ret.target_data, ERROR);
>
> I think ioret and aio_ret are supposed to be the same object. If that's
> right, change one of the names. Likewise elsewhere in this file.
You're right.
> > +The central API piece for postgres' AIO abstraction are AIO handles. To
> > +execute an IO one first has to acquire an IO handle (`pgaio_io_acquire()`) and
> > +then "defined", i.e. associate an IO operation with the handle.
>
> s/"defined"/"define" it/ or similar
>
> > +The "solution" to this the ability to associate multiple completion callbacks
>
> s/this the/this is the/
Applied.
> > Subject: [PATCH v2.11 08/27] localbuf: Track pincount in BufferDesc as well
>
> > @@ -5350,6 +5350,18 @@ ConditionalLockBufferForCleanup(Buffer buffer)
> > Assert(refcount > 0);
> > if (refcount != 1)
> > return false;
> > +
> > + /*
> > + * Check that the AIO subsystem doesn't have a pin. Likely not
> > + * possible today, but better safe than sorry.
> > + */
> > + bufHdr = GetLocalBufferDescriptor(-buffer - 1);
> > + buf_state = pg_atomic_read_u32(&bufHdr->state);
> > + refcount = BUF_STATE_GET_REFCOUNT(buf_state);
> > + Assert(refcount > 0);
> > + if (refcount != 1)
> > + return false;
> > +
>
> LockBufferForCleanup() should get code like this
> ConditionalLockBufferForCleanup() code, either now or when "not possible
> today" ends. Currently, it just assumes all local buffers are
> cleanup-lockable:
>
> /* Nobody else to wait for */
> if (BufferIsLocal(buffer))
> return;
Kinda, yes, kinda no? LockBufferForCleanup() assumes, even for shared
buffers, that the current backend can't be doing anything that conflicts with
acquiring a buffer pin - note that it doesn't check the backend local pincount
for shared buffers either.
LockBufferForCleanup() kind of has to make that assumption, because there's no
way to wait for yourself to release another pin, because obviously waiting in
LockBufferForCleanup() would prevent that from ever happening.
It's somewhat disheartening that the comments for LockBufferForCleanup() don't
mention that somehow the caller needs to be sure not to be called with other
pins on a relation. Nor does LockBufferForCleanup() have any asserts checking
how many backend-local pins exist.
Leaving documentation / asserts aside, I think this is largely a safe
assumption given current callers. With one exception, it's all vacuum or
recovery related code - as vacuum can't run in a transaction, we can't
conflict with another pin by the same backend.
The one exception is heap_surgery.c - it doesn't quite seem safe, the
surrounding (or another query with a cursor) could have a pin on the target
block. The most obvious fix would be to use CheckTableNotInUse(), but that
might break some reasonable uses. Or maybe it should just not use a cleanup
lock, it's not obvious to me why it uses one. But tbh, I don't care too much,
given what heap_surgery is.
> > @@ -570,7 +577,13 @@ InvalidateLocalBuffer(BufferDesc *bufHdr, bool check_unreferenced)
> >
> > buf_state = pg_atomic_read_u32(&bufHdr->state);
> >
> > - if (check_unreferenced && LocalRefCount[bufid] != 0)
> > + /*
> > + * We need to test not just LocalRefCount[bufid] but also the BufferDesc
> > + * itself, as the latter is used to represent a pin by the AIO subsystem.
> > + * This can happen if AIO is initiated and then the query errors out.
> > + */
> > + if (check_unreferenced &&
> > + (LocalRefCount[bufid] != 0 || BUF_STATE_GET_REFCOUNT(buf_state) != 0))
> > elog(ERROR, "block %u of %s is still referenced (local %u)",
>
> I didn't write a test to prove it, but I'm suspecting we'll reach the above
> ERROR with this sequence:
>
> CREATE TEMP TABLE foo ...;
> [some command that starts reading a block of foo into local buffers, then ERROR with IO ongoing]
> DROP TABLE foo;
That seems plausible. I'll try to write a test after this email.
> DropRelationAllLocalBuffers() calls InvalidateLocalBuffer(bufHdr, true). I
> think we'd need to do like pgaio_shutdown() and finish all IOs (or all IOs for
> the particular rel) before InvalidateLocalBuffer(). Or use something like the
> logic near elog(ERROR, "buffer is pinned in InvalidateBuffer") in
> corresponding bufmgr code.
Just waiting for the IO in InvalidateBuffer() does seem like the best bet to
me. It's going to be pretty rarely reached, waiting for all concurrent IO
seems unnecessarily heavyweight. I don't think it matters much today, but once
we do things like asynchronously writing back buffers or WAL, the situation
will be different.
I think this points to the comment above the WaitIO() in InvalidateBuffer()
needing a bit of adapting - an in-progress read can trigger the WaitIO as
well. Something like:
/*
* We assume the reason for it to be pinned is that either we were
* asynchronously reading the page in before erroring out or someone else
* is flushing the page out. Wait for the IO to finish. (This could be
* an infinite loop if the refcount is messed up... it would be nice to
* time out after awhile, but there seems no way to be sure how many loops
* may be needed. Note that if the other guy has pinned the buffer but
* not yet done StartBufferIO, WaitIO will fall through and we'll
* effectively be busy-looping here.)
*/
> > +buffer_readv_complete_one(uint8 buf_off, Buffer buffer, uint8 flags,
> > + bool failed, bool is_temp)
> > +{
> ...
> > + PgAioResult result;
> ...
> > + result.status = PGAIO_RS_OK;
> ...
> > + return result;
>
> gcc 14.2.0 -Werror gives me:
>
> bufmgr.c:7297:16: error: ‘result’ may be used uninitialized [-Werror=maybe-uninitialized]
Gngngng. Since when is it a bug for some fields of a struct to be
uninitialized, as long they're not used?
Interestingly I don't see that warning, despite also using gcc 14.2.0.
I'll just move to your solution, but it seems odd.
> > Subject: [PATCH v2.11 13/27] aio: Basic read_stream adjustments for real AIO
>
> > @@ -416,6 +418,13 @@ read_stream_start_pending_read(ReadStream *stream)
> > static void
> > read_stream_look_ahead(ReadStream *stream)
> > {
> > + /*
> > + * Allow amortizing the cost of submitting IO over multiple IOs. This
> > + * requires that we don't do any operations that could lead to a deadlock
> > + * with staged-but-unsubmitted IO.
> > + */
> > + pgaio_enter_batchmode();
>
> We call read_stream_get_block() while in batchmode, so the stream callback
> needs to be ready for that. A complicated case is
> collect_corrupt_items_read_stream_next_block(), which may do its own buffer
> I/O to read in a vmbuffer for VM_ALL_FROZEN(). That's feeling to me like a
> recipe for corner cases reaching ERROR "starting batch while batch already in
> progress". Are there mitigating factors?
Ugh, yes, you're right. heap_vac_scan_next_block() is also affected.
I don't think "starting batch while batch already in progress" is the real
issue though - it seems easy enough to avoid starting another batch inside,
partially because current cases seem unlikely to need to do batchable IO
inside. What worries me more is that code might block while there's
unsubmitted IO - which seems entirely plausible.
I can see a few approaches:
1) Declare that all read stream callbacks have to be careful and cope with
batch mode
I'm not sure how viable that is, not starting batches seems ok, but
ensuring that the code doesn't block is a different story.
2) Have read stream users opt-in to batching
Presumably via a flag like READ_STREAM_USE_BATCHING. That'd be easy enough
to implement and to add to the callsites where that's fine.
3) Teach read stream to "look ahead" far enough to determine all the blocks
that could be issued in a batch outside of batchmode
I think that's probably not a great idea, it'd lead us to looking further
ahead than we really need to, which could increase "unfairness" in
e.g. parallel sequential scan.
4) Just defer using batch mode for now
It's a nice win with io_uring for random IO, e.g. from bitmap heap scans ,
but there's no need to immediately solve this.
I think regardless of what we go for, it's worth splitting
"aio: Basic read_stream adjustments for real AIO"
into the actually basic parts (i.e. introducing sync_mode) from the not
actually so basic parts (i.e. batching).
I suspect that 2) would be the best approach. Only the read stream user knows
what it needs to do in the callback.
>
> > Subject: [PATCH v2.11 17/27] aio: Add test_aio module
>
> > + # verify that page verification errors are detected even as part of a
> > + # shortened multi-block read (tbl_corr, block 1 is tbl_corred)
>
> Is "tbl_corred" a typo of something?
I think that was a search&replace of the table name gone wrong. It was just
supposed to be "corrupted".
> > + *
> > + * IDENTIFICATION
> > + * src/test/modules/delay_execution/delay_execution.c
>
> Header comment is surviving from copy-paste of delay_execution.c.
Oh, how I hate these pointless comments. Fixed.
Greetings,
Andres Freund
^ permalink raw reply [nested|flat] 153+ messages in thread
* Re: AIO v2.5
2025-03-04 19:00 AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-10 18:23 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-11 19:41 ` Re: AIO v2.5 Noah Misch <noah@leadboat.com>
2025-03-11 23:55 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-12 03:57 ` Re: AIO v2.5 Noah Misch <noah@leadboat.com>
2025-03-12 17:06 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-12 23:23 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-13 10:53 ` Re: AIO v2.5 Antonin Houska <ah@cybertec.at>
2025-03-14 19:43 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-17 20:00 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-18 20:12 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-21 01:58 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-24 00:29 ` Re: AIO v2.5 Noah Misch <noah@leadboat.com>
2025-03-24 15:43 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
@ 2025-03-24 22:55 ` Andres Freund <andres@anarazel.de>
2025-03-25 00:07 ` Re: AIO v2.5 Thomas Munro <thomas.munro@gmail.com>
0 siblings, 1 reply; 153+ messages in thread
From: Andres Freund @ 2025-03-24 22:55 UTC (permalink / raw)
To: Noah Misch <noah@leadboat.com>; +Cc: Antonin Houska <ah@cybertec.at>; pgsql-hackers@postgresql.org, Thomas Munro <thomas.munro@gmail.com>; Heikki Linnakangas <hlinnaka@iki.fi>; Robert Haas <robertmhaas@gmail.com>; Jakub Wartak <jakub.wartak@enterprisedb.com>; Jelte Fennema-Nio <postgres@jeltef.nl>
Hi,
On 2025-03-24 11:43:47 -0400, Andres Freund wrote:
> > I didn't write a test to prove it, but I'm suspecting we'll reach the above
> > ERROR with this sequence:
> >
> > CREATE TEMP TABLE foo ...;
> > [some command that starts reading a block of foo into local buffers, then ERROR with IO ongoing]
> > DROP TABLE foo;
>
> That seems plausible. I'll try to write a test after this email.
FWIW, a test did indeed confirm that. Luckily:
> > DropRelationAllLocalBuffers() calls InvalidateLocalBuffer(bufHdr, true). I
> > think we'd need to do like pgaio_shutdown() and finish all IOs (or all IOs for
> > the particular rel) before InvalidateLocalBuffer(). Or use something like the
> > logic near elog(ERROR, "buffer is pinned in InvalidateBuffer") in
> > corresponding bufmgr code.
>
> Just waiting for the IO in InvalidateBuffer() does seem like the best bet to
> me.
This did indeed resolve the issue.
I've extended the testsuite to test for that and a bunch more things. Working
on sending out a new version...
> > We call read_stream_get_block() while in batchmode, so the stream callback
> > needs to be ready for that. A complicated case is
> > collect_corrupt_items_read_stream_next_block(), which may do its own buffer
> > I/O to read in a vmbuffer for VM_ALL_FROZEN(). That's feeling to me like a
> > recipe for corner cases reaching ERROR "starting batch while batch already in
> > progress". Are there mitigating factors?
>
> Ugh, yes, you're right. heap_vac_scan_next_block() is also affected.
>
> I don't think "starting batch while batch already in progress" is the real
> issue though - it seems easy enough to avoid starting another batch inside,
> partially because current cases seem unlikely to need to do batchable IO
> inside. What worries me more is that code might block while there's
> unsubmitted IO - which seems entirely plausible.
>
>
> I can see a few approaches:
>
> 1) Declare that all read stream callbacks have to be careful and cope with
> batch mode
>
> I'm not sure how viable that is, not starting batches seems ok, but
> ensuring that the code doesn't block is a different story.
>
>
> 2) Have read stream users opt-in to batching
>
> Presumably via a flag like READ_STREAM_USE_BATCHING. That'd be easy enough
> to implement and to add to the callsites where that's fine.
>
>
> 3) Teach read stream to "look ahead" far enough to determine all the blocks
> that could be issued in a batch outside of batchmode
>
> I think that's probably not a great idea, it'd lead us to looking further
> ahead than we really need to, which could increase "unfairness" in
> e.g. parallel sequential scan.
>
>
> 4) Just defer using batch mode for now
>
> It's a nice win with io_uring for random IO, e.g. from bitmap heap scans ,
> but there's no need to immediately solve this.
>
>
> I think regardless of what we go for, it's worth splitting
> "aio: Basic read_stream adjustments for real AIO"
> into the actually basic parts (i.e. introducing sync_mode) from the not
> actually so basic parts (i.e. batching).
>
>
> I suspect that 2) would be the best approach. Only the read stream user knows
> what it needs to do in the callback.
I still think 2) would be the best option.
Writing a patch for that.
If a callback may sometimes need to block, it can still opt into
READ_STREAM_USE_BATCHING, by submitting all staged IO before blocking.
The hardest part is to explain the flag. Here's my current attempt:
/* ---
* Opt-in to using AIO batchmode.
*
* Submitting IO in larger batches can be more efficient than doing so
* one-by-one, particularly for many small reads. It does, however, require
* the ReadStreamBlockNumberCB callback to abide by the restrictions of AIO
* batching (c.f. pgaio_enter_batchmode()). Basically, the callback may not:
* a) block without first calling pgaio_submit_staged(), unless a
* to-be-waited-on lock cannot be part of a deadlock, e.g. because it is
* never acquired in a nested fashion
* b) directly or indirectly start another batch pgaio_enter_batchmode()
*
* As this requires care and is nontrivial in some cases, batching is only
* used with explicit opt-in.
* ---
*/
#define READ_STREAM_USE_BATCHING 0x08
Greetings,
Andres Freund
^ permalink raw reply [nested|flat] 153+ messages in thread
* Re: AIO v2.5
2025-03-04 19:00 AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-10 18:23 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-11 19:41 ` Re: AIO v2.5 Noah Misch <noah@leadboat.com>
2025-03-11 23:55 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-12 03:57 ` Re: AIO v2.5 Noah Misch <noah@leadboat.com>
2025-03-12 17:06 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-12 23:23 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-13 10:53 ` Re: AIO v2.5 Antonin Houska <ah@cybertec.at>
2025-03-14 19:43 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-17 20:00 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-18 20:12 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-21 01:58 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-24 00:29 ` Re: AIO v2.5 Noah Misch <noah@leadboat.com>
2025-03-24 15:43 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-24 22:55 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
@ 2025-03-25 00:07 ` Thomas Munro <thomas.munro@gmail.com>
2025-03-25 00:18 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-25 00:45 ` Re: AIO v2.5 Noah Misch <noah@leadboat.com>
0 siblings, 2 replies; 153+ messages in thread
From: Thomas Munro @ 2025-03-25 00:07 UTC (permalink / raw)
To: Andres Freund <andres@anarazel.de>; +Cc: Noah Misch <noah@leadboat.com>; Antonin Houska <ah@cybertec.at>; pgsql-hackers@postgresql.org, Heikki Linnakangas <hlinnaka@iki.fi>; Robert Haas <robertmhaas@gmail.com>; Jakub Wartak <jakub.wartak@enterprisedb.com>; Jelte Fennema-Nio <postgres@jeltef.nl>
On Tue, Mar 25, 2025 at 11:55 AM Andres Freund <andres@anarazel.de> wrote:
> If a callback may sometimes need to block, it can still opt into
> READ_STREAM_USE_BATCHING, by submitting all staged IO before blocking.
>
> The hardest part is to explain the flag. Here's my current attempt:
>
> /* ---
> * Opt-in to using AIO batchmode.
> *
> * Submitting IO in larger batches can be more efficient than doing so
> * one-by-one, particularly for many small reads. It does, however, require
> * the ReadStreamBlockNumberCB callback to abide by the restrictions of AIO
> * batching (c.f. pgaio_enter_batchmode()). Basically, the callback may not:
> * a) block without first calling pgaio_submit_staged(), unless a
> * to-be-waited-on lock cannot be part of a deadlock, e.g. because it is
> * never acquired in a nested fashion
> * b) directly or indirectly start another batch pgaio_enter_batchmode()
> *
> * As this requires care and is nontrivial in some cases, batching is only
> * used with explicit opt-in.
> * ---
> */
> #define READ_STREAM_USE_BATCHING 0x08
+1
I wonder if something more like READ_STREAM_CALLBACK_BATCHMODE_AWARE
would be better, to highlight that you are making a declaration about
a property of your callback, not just turning on an independent
go-fast feature... I fished those words out of the main (?)
description of this topic atop pgaio_enter_batchmode(). Just a
thought, IDK.
^ permalink raw reply [nested|flat] 153+ messages in thread
* Re: AIO v2.5
2025-03-04 19:00 AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-10 18:23 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-11 19:41 ` Re: AIO v2.5 Noah Misch <noah@leadboat.com>
2025-03-11 23:55 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-12 03:57 ` Re: AIO v2.5 Noah Misch <noah@leadboat.com>
2025-03-12 17:06 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-12 23:23 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-13 10:53 ` Re: AIO v2.5 Antonin Houska <ah@cybertec.at>
2025-03-14 19:43 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-17 20:00 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-18 20:12 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-21 01:58 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-24 00:29 ` Re: AIO v2.5 Noah Misch <noah@leadboat.com>
2025-03-24 15:43 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-24 22:55 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-25 00:07 ` Re: AIO v2.5 Thomas Munro <thomas.munro@gmail.com>
@ 2025-03-25 00:18 ` Andres Freund <andres@anarazel.de>
1 sibling, 0 replies; 153+ messages in thread
From: Andres Freund @ 2025-03-25 00:18 UTC (permalink / raw)
To: Thomas Munro <thomas.munro@gmail.com>; +Cc: Noah Misch <noah@leadboat.com>; Antonin Houska <ah@cybertec.at>; pgsql-hackers@postgresql.org, Heikki Linnakangas <hlinnaka@iki.fi>; Robert Haas <robertmhaas@gmail.com>; Jakub Wartak <jakub.wartak@enterprisedb.com>; Jelte Fennema-Nio <postgres@jeltef.nl>
Hi,
On 2025-03-25 13:07:49 +1300, Thomas Munro wrote:
> On Tue, Mar 25, 2025 at 11:55 AM Andres Freund <andres@anarazel.de> wrote:
> > #define READ_STREAM_USE_BATCHING 0x08
>
> +1
>
> I wonder if something more like READ_STREAM_CALLBACK_BATCHMODE_AWARE
> would be better, to highlight that you are making a declaration about
> a property of your callback, not just turning on an independent
> go-fast feature... I fished those words out of the main (?)
> description of this topic atop pgaio_enter_batchmode(). Just a
> thought, IDK.
The relevant lines are already very deeply indented, so I'm a bit wary of such
a long name. I think we'd basically have to use a separate flags variable
everywhere and that is annoying due to us following C89 variable declaration
positions...
Greetings,
Andres Freund
^ permalink raw reply [nested|flat] 153+ messages in thread
* Re: AIO v2.5
2025-03-04 19:00 AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-10 18:23 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-11 19:41 ` Re: AIO v2.5 Noah Misch <noah@leadboat.com>
2025-03-11 23:55 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-12 03:57 ` Re: AIO v2.5 Noah Misch <noah@leadboat.com>
2025-03-12 17:06 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-12 23:23 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-13 10:53 ` Re: AIO v2.5 Antonin Houska <ah@cybertec.at>
2025-03-14 19:43 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-17 20:00 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-18 20:12 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-21 01:58 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-24 00:29 ` Re: AIO v2.5 Noah Misch <noah@leadboat.com>
2025-03-24 15:43 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-24 22:55 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-25 00:07 ` Re: AIO v2.5 Thomas Munro <thomas.munro@gmail.com>
@ 2025-03-25 00:45 ` Noah Misch <noah@leadboat.com>
2025-03-25 02:30 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
1 sibling, 1 reply; 153+ messages in thread
From: Noah Misch @ 2025-03-25 00:45 UTC (permalink / raw)
To: Andres Freund <andres@anarazel.de>; Thomas Munro <thomas.munro@gmail.com>; +Cc: Antonin Houska <ah@cybertec.at>; pgsql-hackers@postgresql.org, Heikki Linnakangas <hlinnaka@iki.fi>; Robert Haas <robertmhaas@gmail.com>; Jakub Wartak <jakub.wartak@enterprisedb.com>; Jelte Fennema-Nio <postgres@jeltef.nl>
On Mon, Mar 24, 2025 at 11:43:47AM -0400, Andres Freund wrote:
> On 2025-03-23 17:29:39 -0700, Noah Misch wrote:
> > commit 247ce06b wrote:
> > > + pgaio_io_reopen(ioh);
> > > +
> > > + /*
> > > + * To be able to exercise the reopen-fails path, allow injection
> > > + * points to trigger a failure at this point.
> > > + */
> > > + pgaio_io_call_inj(ioh, "AIO_WORKER_AFTER_REOPEN");
> > > +
> > > + error_errno = 0;
> > > + error_ioh = NULL;
> > > +
> > > + /*
> > > + * We don't expect this to ever fail with ERROR or FATAL, no need
> > > + * to keep error_ioh set to the IO.
> > > + * pgaio_io_perform_synchronously() contains a critical section to
> > > + * ensure we don't accidentally fail.
> > > + */
> > > + pgaio_io_perform_synchronously(ioh);
> >
> > A CHECK_FOR_INTERRUPTS() could close() the FD that pgaio_io_reopen() callee
> > smgr_aio_reopen() stores. Hence, I think smgrfd() should assert that
> > interrupts are held instead of doing its own HOLD_INTERRUPTS(), and a
> > HOLD_INTERRUPTS() should surround the above region of code. It's likely hard
> > to reproduce a problem, because pgaio_io_call_inj() does nothing in many
> > builds, and pgaio_io_perform_synchronously() starts by entering a critical
> > section.
>
> Hm, I guess you're right - it would be pretty bonkers for the injection to
> process interrupts, but its much better to clarify the code to make that not
> an option. Once doing that it seemed we should also have a similar assertion
> in pgaio_io_before_prep() would be appropriate.
Agreed. Following that line of thinking, the io_uring case needs to
HOLD_INTERRUPTS() (or hold smgrrelease() specifically) all the way from
pgaio_io_before_prep() to PGAIO_HS_SUBMITTED. The fd has to stay valid until
io_uring_submit().
(We may be due for a test mode that does smgrreleaseall() at every
CHECK_FOR_INTERRUPTS()?)
> > On Thu, Mar 20, 2025 at 09:58:37PM -0400, Andres Freund wrote:
> > > Subject: [PATCH v2.11 06/27] aio: Implement support for reads in smgr/md/fd
> >
> > > +int
> > > +FileStartReadV(PgAioHandle *ioh, File file,
> > > + int iovcnt, off_t offset,
> > > + uint32 wait_event_info)
> > > +{
> > > + int returnCode;
> > > + Vfd *vfdP;
> > > +
> > > + Assert(FileIsValid(file));
> > > +
> > > + DO_DB(elog(LOG, "FileStartReadV: %d (%s) " INT64_FORMAT " %d",
> > > + file, VfdCache[file].fileName,
> > > + (int64) offset,
> > > + iovcnt));
> > > +
> > > + returnCode = FileAccess(file);
> > > + if (returnCode < 0)
> > > + return returnCode;
> > > +
> > > + vfdP = &VfdCache[file];
> > > +
> > > + pgaio_io_prep_readv(ioh, vfdP->fd, iovcnt, offset);
> >
> > FileStartReadV() and pgaio_io_prep_readv() advance the IO to PGAIO_HS_STAGED
> > w/ batch mode, PGAIO_HS_SUBMITTED w/o batch mode. I didn't expect that from
> > functions so named. The "start" verb sounds to me like unconditional
> > PGAIO_HS_SUBMITTED, and the "prep" verb sounds like PGAIO_HS_DEFINED. I like
> > the "stage" verb, because it matches PGAIO_HS_STAGED, and the comment at
> > PGAIO_HS_STAGED succinctly covers what to expect. Hence, I recommend names
> > FileStageReadV, pgaio_io_stage_readv, mdstagereadv, and smgrstageread. How do
> > you see it?
>
> I have a surprisingly strong negative reaction to that proposed naming. To me
> the staging is a distinct step that happens *after* the IO is fully
> defined. Making all the layered calls that lead up to that named that way
> would IMO be a bad idea.
As a general naming principle, I think the name of a function that advances
through multiple named steps should mention the last step. Naming the
function after just a non-last step feels weird to me. For example, serving a
meal consists of steps menu_define, mix_ingredients, and plate_food. It would
be weird to me if a function called meal_menu_define() mixed ingredients or
plated food, but it's fine if meal_plate_food() does all three steps. A
second strategy is to name both the first and last steps:
meal_define_menu_thru_plate_food() is fine apart from being long. A third
strategy is to have meal_plate_food() assert than meal_mix_ingredients() has
been called.
I wouldn't mind "staging" as a distinct step, but I think today's API
boundaries hide the distinction. PGAIO_HS_DEFINED is a temporary state during
a pgaio_io_stage() call, so the process that defines and stages the IO can
observe PGAIO_HS_DEFINED only while pgaio_io_stage() is on the stack.
The aforementioned "third strategy" could map to having distinct
smgrdefinereadv() and smgrstagereadv(). I don't know how well that would work
out overall. I wouldn't be optimistic about that winning, but I mention it
for completeness.
> I however don't particularly like the *start* or *prep* names, I've gone back
> and forth on those a couple times. I could see "begin" work uniformly across
> those.
For ease of new readers understanding things, I think it helps for the
functions that advance PgAioHandleState to have names that use words from
PgAioHandleState. It's one less mapping to get into the reader's head.
"Begin", "Start" and "prep" are all outside that taxonomy, making the reader
learn how to map them to the taxonomy. What reward does the reader get at the
end of that exercise? I'm not seeing one, but please do tell me what I'm
missing here.
> > > Subject: [PATCH v2.11 08/27] localbuf: Track pincount in BufferDesc as well
> >
> > > @@ -5350,6 +5350,18 @@ ConditionalLockBufferForCleanup(Buffer buffer)
> > > Assert(refcount > 0);
> > > if (refcount != 1)
> > > return false;
> > > +
> > > + /*
> > > + * Check that the AIO subsystem doesn't have a pin. Likely not
> > > + * possible today, but better safe than sorry.
> > > + */
> > > + bufHdr = GetLocalBufferDescriptor(-buffer - 1);
> > > + buf_state = pg_atomic_read_u32(&bufHdr->state);
> > > + refcount = BUF_STATE_GET_REFCOUNT(buf_state);
> > > + Assert(refcount > 0);
> > > + if (refcount != 1)
> > > + return false;
> > > +
> >
> > LockBufferForCleanup() should get code like this
> > ConditionalLockBufferForCleanup() code, either now or when "not possible
> > today" ends. Currently, it just assumes all local buffers are
> > cleanup-lockable:
> >
> > /* Nobody else to wait for */
> > if (BufferIsLocal(buffer))
> > return;
>
> Kinda, yes, kinda no? LockBufferForCleanup() assumes, even for shared
> buffers, that the current backend can't be doing anything that conflicts with
> acquiring a buffer pin - note that it doesn't check the backend local pincount
> for shared buffers either.
It checks the local pincount via callee CheckBufferIsPinnedOnce().
As the patch stands, LockBufferForCleanup() can succeed when
ConditionalLockBufferForCleanup() would have returned false. I'm not seeking
to raise the overall standard of *Cleanup() family of functions, but I am
trying to keep members of that family agreeing on the standard.
Like the comment, I expect it's academic today. I expect it will stay
academic. Anything that does a cleanup will start by reading the buffer,
which will resolve any refcnt the AIO subsystems holds for a read. If there's
an AIO write, the LockBuffer(buffer, BUFFER_LOCK_EXCLUSIVE) will block on
that. How about just removing the ConditionalLockBufferForCleanup() changes
or replacing them with a comment (like the present paragraph)?
> I think this points to the comment above the WaitIO() in InvalidateBuffer()
> needing a bit of adapting - an in-progress read can trigger the WaitIO as
> well. Something like:
>
> /*
> * We assume the reason for it to be pinned is that either we were
> * asynchronously reading the page in before erroring out or someone else
> * is flushing the page out. Wait for the IO to finish. (This could be
> * an infinite loop if the refcount is messed up... it would be nice to
> * time out after awhile, but there seems no way to be sure how many loops
> * may be needed. Note that if the other guy has pinned the buffer but
> * not yet done StartBufferIO, WaitIO will fall through and we'll
> * effectively be busy-looping here.)
> */
Agreed.
> > > +buffer_readv_complete_one(uint8 buf_off, Buffer buffer, uint8 flags,
> > > + bool failed, bool is_temp)
> > > +{
> > ...
> > > + PgAioResult result;
> > ...
> > > + result.status = PGAIO_RS_OK;
> > ...
> > > + return result;
> >
> > gcc 14.2.0 -Werror gives me:
> >
> > bufmgr.c:7297:16: error: ‘result’ may be used uninitialized [-Werror=maybe-uninitialized]
>
> Gngngng. Since when is it a bug for some fields of a struct to be
> uninitialized, as long they're not used?
>
> Interestingly I don't see that warning, despite also using gcc 14.2.0.
I badly neglected to mention my non-default flags:
CFLAGS='-O2 -fno-sanitize-recover=all -fsanitize=address,alignment,undefined --param=max-vartrack-size=150000000 -ftrivial-auto-var-init=pattern'
COPT=-Werror -Wno-error=array-bounds
Final CFLAGS, including the ones "configure" elects on its own:
configure: using CFLAGS=-Wall -Wmissing-prototypes -Wpointer-arith -Wdeclaration-after-statement -Werror=vla -Werror=unguarded-availability-new -Wendif-labels -Wmissing-format-attribute -Wcast-function-type -Wformat-security -Wmissing-variable-declarations -fno-strict-aliasing -fwrapv -fexcess-precision=standard -Wno-unused-command-line-argument -Wno-compound-token-split-by-macro -Wno-format-truncation -Wno-cast-function-type-strict -g -O2 -fno-sanitize-recover=all -fsanitize=address,alignment,undefined --param=max-vartrack-size=150000000 -ftrivial-auto-var-init=pattern
(I use -Wno-error=array-bounds because the sanitizer options elicit a lot of
those warnings. Today's master is free from maybe-uninitialized warnings in
this configuration, though.)
> I'll just move to your solution, but it seems odd.
Got it.
> I think regardless of what we go for, it's worth splitting
> "aio: Basic read_stream adjustments for real AIO"
> into the actually basic parts (i.e. introducing sync_mode) from the not
> actually so basic parts (i.e. batching).
Fair.
On Mon, Mar 24, 2025 at 06:55:22PM -0400, Andres Freund wrote:
> Hi,
>
> On 2025-03-24 11:43:47 -0400, Andres Freund wrote:
> > > I didn't write a test to prove it, but I'm suspecting we'll reach the above
> > > ERROR with this sequence:
> > >
> > > CREATE TEMP TABLE foo ...;
> > > [some command that starts reading a block of foo into local buffers, then ERROR with IO ongoing]
> > > DROP TABLE foo;
> >
> > That seems plausible. I'll try to write a test after this email.
>
> FWIW, a test did indeed confirm that. Luckily:
>
> > > DropRelationAllLocalBuffers() calls InvalidateLocalBuffer(bufHdr, true). I
> > > think we'd need to do like pgaio_shutdown() and finish all IOs (or all IOs for
> > > the particular rel) before InvalidateLocalBuffer(). Or use something like the
> > > logic near elog(ERROR, "buffer is pinned in InvalidateBuffer") in
> > > corresponding bufmgr code.
> >
> > Just waiting for the IO in InvalidateBuffer() does seem like the best bet to
> > me.
>
> This did indeed resolve the issue.
I'm happy with that approach.
On Tue, Mar 25, 2025 at 01:07:49PM +1300, Thomas Munro wrote:
> On Tue, Mar 25, 2025 at 11:55 AM Andres Freund <andres@anarazel.de> wrote:
> > If a callback may sometimes need to block, it can still opt into
> > READ_STREAM_USE_BATCHING, by submitting all staged IO before blocking.
> >
> > The hardest part is to explain the flag. Here's my current attempt:
> >
> > /* ---
> > * Opt-in to using AIO batchmode.
> > *
> > * Submitting IO in larger batches can be more efficient than doing so
> > * one-by-one, particularly for many small reads. It does, however, require
> > * the ReadStreamBlockNumberCB callback to abide by the restrictions of AIO
> > * batching (c.f. pgaio_enter_batchmode()). Basically, the callback may not:
> > * a) block without first calling pgaio_submit_staged(), unless a
> > * to-be-waited-on lock cannot be part of a deadlock, e.g. because it is
> > * never acquired in a nested fashion
> > * b) directly or indirectly start another batch pgaio_enter_batchmode()
I think a callback could still do:
pgaio_exit_batchmode()
... arbitrary code that might reach pgaio_enter_batchmode() ...
pgaio_enter_batchmode()
> > *
> > * As this requires care and is nontrivial in some cases, batching is only
> > * used with explicit opt-in.
> > * ---
> > */
> > #define READ_STREAM_USE_BATCHING 0x08
>
> +1
Agreed. It's simple, and there's no loss of generality.
> I wonder if something more like READ_STREAM_CALLBACK_BATCHMODE_AWARE
> would be better, to highlight that you are making a declaration about
> a property of your callback, not just turning on an independent
> go-fast feature... I fished those words out of the main (?)
> description of this topic atop pgaio_enter_batchmode(). Just a
> thought, IDK.
Good points. I lean toward your renaming suggestion, or shortening to
READ_STREAM_BATCHMODE_AWARE or READ_STREAM_BATCH_OK. I'm also fine with the
original name, though.
Thanks,
nm
^ permalink raw reply [nested|flat] 153+ messages in thread
* Re: AIO v2.5
2025-03-04 19:00 AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-10 18:23 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-11 19:41 ` Re: AIO v2.5 Noah Misch <noah@leadboat.com>
2025-03-11 23:55 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-12 03:57 ` Re: AIO v2.5 Noah Misch <noah@leadboat.com>
2025-03-12 17:06 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-12 23:23 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-13 10:53 ` Re: AIO v2.5 Antonin Houska <ah@cybertec.at>
2025-03-14 19:43 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-17 20:00 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-18 20:12 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-21 01:58 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-24 00:29 ` Re: AIO v2.5 Noah Misch <noah@leadboat.com>
2025-03-24 15:43 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-24 22:55 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-25 00:07 ` Re: AIO v2.5 Thomas Munro <thomas.munro@gmail.com>
2025-03-25 00:45 ` Re: AIO v2.5 Noah Misch <noah@leadboat.com>
@ 2025-03-25 02:30 ` Andres Freund <andres@anarazel.de>
2025-03-25 13:33 ` Re: AIO v2.5 Noah Misch <noah@leadboat.com>
0 siblings, 1 reply; 153+ messages in thread
From: Andres Freund @ 2025-03-25 02:30 UTC (permalink / raw)
To: Noah Misch <noah@leadboat.com>; +Cc: Thomas Munro <thomas.munro@gmail.com>; Antonin Houska <ah@cybertec.at>; pgsql-hackers@postgresql.org, Heikki Linnakangas <hlinnaka@iki.fi>; Robert Haas <robertmhaas@gmail.com>; Jakub Wartak <jakub.wartak@enterprisedb.com>; Jelte Fennema-Nio <postgres@jeltef.nl>
Hi,
On 2025-03-24 17:45:37 -0700, Noah Misch wrote:
> (We may be due for a test mode that does smgrreleaseall() at every
> CHECK_FOR_INTERRUPTS()?)
I suspect we are. I'm a bit afraid of even trying...
...
It's extremely slow - but at least the main regression as well as the aio tests pass!
> > I however don't particularly like the *start* or *prep* names, I've gone back
> > and forth on those a couple times. I could see "begin" work uniformly across
> > those.
>
> For ease of new readers understanding things, I think it helps for the
> functions that advance PgAioHandleState to have names that use words from
> PgAioHandleState. It's one less mapping to get into the reader's head.
Unfortunately for me it's kind of the opposite in this case, see below.
> "Begin", "Start" and "prep" are all outside that taxonomy, making the reader
> learn how to map them to the taxonomy. What reward does the reader get at the
> end of that exercise? I'm not seeing one, but please do tell me what I'm
> missing here.
Because the end state varies, depending on the number of previously staged
IOs, the IO method and whether batchmode is enabled, I think it's better if
the "function naming pattern" (i.e. FileStartReadv, smgrstartreadv etc) is
*not* aligned with an internal state name. It will just mislead readers to
think that there's a deterministic mapping when that does not exist.
That's not an excuse for pgaio_io_prep* though, that's a pointlessly different
naming that I just stopped seeing.
I'll try to think more about this, perhaps I can make myself see your POV
more.
> > > > Subject: [PATCH v2.11 08/27] localbuf: Track pincount in BufferDesc as well
> > > LockBufferForCleanup() should get code like this
> > > ConditionalLockBufferForCleanup() code, either now or when "not possible
> > > today" ends. Currently, it just assumes all local buffers are
> > > cleanup-lockable:
> > >
> > > /* Nobody else to wait for */
> > > if (BufferIsLocal(buffer))
> > > return;
> >
> > Kinda, yes, kinda no? LockBufferForCleanup() assumes, even for shared
> > buffers, that the current backend can't be doing anything that conflicts with
> > acquiring a buffer pin - note that it doesn't check the backend local pincount
> > for shared buffers either.
>
> It checks the local pincount via callee CheckBufferIsPinnedOnce().
In exactly one of the callers :/
> As the patch stands, LockBufferForCleanup() can succeed when
> ConditionalLockBufferForCleanup() would have returned false.
That's already true today, right? In master ConditionalLockBufferForCleanup()
for temp buffers checks LocalRefCount, whereas LockBufferForCleanup() doesn't.
I think I agree with your suggestion further below, but independent of that, I
don't see how the current modification in the patch makes the worse.
Historically this behaviour of LockBufferForCleanup() kinda somewhat makes
sense - the only place we use LockBufferForCleanup() is in a non-transactional
command i.e. vacuum / index vacuum. So LockBufferForCleanup() turns out to
only be safe in that context.
> Like the comment, I expect it's academic today. I expect it will stay
> academic. Anything that does a cleanup will start by reading the buffer,
> which will resolve any refcnt the AIO subsystems holds for a read. If there's
> an AIO write, the LockBuffer(buffer, BUFFER_LOCK_EXCLUSIVE) will block on
> that. How about just removing the ConditionalLockBufferForCleanup() changes
> or replacing them with a comment (like the present paragraph)?
I think we'll need an expanded version of what I suggest once we have writes -
but as you say, it shouldn't matter as long as we only have reads. So I think
moving the relevant changes, with adjusted caveats, to the bufmgr: write
change makes sense.
> > > /* ---
> > > * Opt-in to using AIO batchmode.
> > > *
> > > * Submitting IO in larger batches can be more efficient than doing so
> > > * one-by-one, particularly for many small reads. It does, however, require
> > > * the ReadStreamBlockNumberCB callback to abide by the restrictions of AIO
> > > * batching (c.f. pgaio_enter_batchmode()). Basically, the callback may not:
> > > * a) block without first calling pgaio_submit_staged(), unless a
> > > * to-be-waited-on lock cannot be part of a deadlock, e.g. because it is
> > > * never acquired in a nested fashion
> > > * b) directly or indirectly start another batch pgaio_enter_batchmode()
>
> I think a callback could still do:
>
> pgaio_exit_batchmode()
> ... arbitrary code that might reach pgaio_enter_batchmode() ...
> pgaio_enter_batchmode()
Yea - but I somehow doubt there are many cases where it makes sense to
deep-queue IOs within the callback. The cases I can think of are things like
ensuring the right VM buffer is in s_b. But if it turns out to be necessary,
what you seuggest would be an out.
Do you think it's worth mentioning the above workaround? I'm mildly inclined
that not.
If it turns out to be actually useful to do nested batching, we can change it
so that nested batching *is* allowed, that'd not be hard.
> > > *
> > > * As this requires care and is nontrivial in some cases, batching is only
> > > * used with explicit opt-in.
> > > * ---
> > > */
> > > #define READ_STREAM_USE_BATCHING 0x08
> >
> > +1
>
> Agreed. It's simple, and there's no loss of generality.
>
> > I wonder if something more like READ_STREAM_CALLBACK_BATCHMODE_AWARE
> > would be better, to highlight that you are making a declaration about
> > a property of your callback, not just turning on an independent
> > go-fast feature... I fished those words out of the main (?)
> > description of this topic atop pgaio_enter_batchmode(). Just a
> > thought, IDK.
>
> Good points. I lean toward your renaming suggestion, or shortening to
> READ_STREAM_BATCHMODE_AWARE or READ_STREAM_BATCH_OK. I'm also fine with the
> original name, though.
I'm ok with all of these. In order of preference:
1) READ_STREAM_USE_BATCHING or READ_STREAM_BATCH_OK
2) READ_STREAM_BATCHMODE_AWARE
3) READ_STREAM_CALLBACK_BATCHMODE_AWARE
Greetings,
Andres Freund
^ permalink raw reply [nested|flat] 153+ messages in thread
* Re: AIO v2.5
2025-03-04 19:00 AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-10 18:23 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-11 19:41 ` Re: AIO v2.5 Noah Misch <noah@leadboat.com>
2025-03-11 23:55 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-12 03:57 ` Re: AIO v2.5 Noah Misch <noah@leadboat.com>
2025-03-12 17:06 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-12 23:23 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-13 10:53 ` Re: AIO v2.5 Antonin Houska <ah@cybertec.at>
2025-03-14 19:43 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-17 20:00 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-18 20:12 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-21 01:58 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-24 00:29 ` Re: AIO v2.5 Noah Misch <noah@leadboat.com>
2025-03-24 15:43 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-24 22:55 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-25 00:07 ` Re: AIO v2.5 Thomas Munro <thomas.munro@gmail.com>
2025-03-25 00:45 ` Re: AIO v2.5 Noah Misch <noah@leadboat.com>
2025-03-25 02:30 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
@ 2025-03-25 13:33 ` Noah Misch <noah@leadboat.com>
2025-03-25 15:26 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
0 siblings, 1 reply; 153+ messages in thread
From: Noah Misch @ 2025-03-25 13:33 UTC (permalink / raw)
To: Andres Freund <andres@anarazel.de>; +Cc: Thomas Munro <thomas.munro@gmail.com>; Antonin Houska <ah@cybertec.at>; pgsql-hackers@postgresql.org, Heikki Linnakangas <hlinnaka@iki.fi>; Robert Haas <robertmhaas@gmail.com>; Jakub Wartak <jakub.wartak@enterprisedb.com>; Jelte Fennema-Nio <postgres@jeltef.nl>
On Mon, Mar 24, 2025 at 10:30:27PM -0400, Andres Freund wrote:
> On 2025-03-24 17:45:37 -0700, Noah Misch wrote:
> > (We may be due for a test mode that does smgrreleaseall() at every
> > CHECK_FOR_INTERRUPTS()?)
>
> I suspect we are. I'm a bit afraid of even trying...
>
> ...
>
> It's extremely slow - but at least the main regression as well as the aio tests pass!
One less thing!
> > > I however don't particularly like the *start* or *prep* names, I've gone back
> > > and forth on those a couple times. I could see "begin" work uniformly across
> > > those.
> >
> > For ease of new readers understanding things, I think it helps for the
> > functions that advance PgAioHandleState to have names that use words from
> > PgAioHandleState. It's one less mapping to get into the reader's head.
>
> Unfortunately for me it's kind of the opposite in this case, see below.
>
>
> > "Begin", "Start" and "prep" are all outside that taxonomy, making the reader
> > learn how to map them to the taxonomy. What reward does the reader get at the
> > end of that exercise? I'm not seeing one, but please do tell me what I'm
> > missing here.
>
> Because the end state varies, depending on the number of previously staged
> IOs, the IO method and whether batchmode is enabled, I think it's better if
> the "function naming pattern" (i.e. FileStartReadv, smgrstartreadv etc) is
> *not* aligned with an internal state name. It will just mislead readers to
> think that there's a deterministic mapping when that does not exist.
That's fair. Could we provide the mapping in a comment, something like the
following?
--- a/src/include/storage/aio_internal.h
+++ b/src/include/storage/aio_internal.h
@@ -34,5 +34,10 @@
* linearly through all states.
*
- * State changes should all go through pgaio_io_update_state().
+ * State changes should all go through pgaio_io_update_state(). Its callers
+ * use these naming conventions:
+ *
+ * - A "start" function (e.g. FileStartReadV()) moves an IO from
+ * PGAIO_HS_HANDED_OUT to at least PGAIO_HS_STAGED and at most
+ * PGAIO_HS_COMPLETED_LOCAL.
*/
typedef enum PgAioHandleState
> That's not an excuse for pgaio_io_prep* though, that's a pointlessly different
> naming that I just stopped seeing.
>
> I'll try to think more about this, perhaps I can make myself see your POV
> more.
> > As the patch stands, LockBufferForCleanup() can succeed when
> > ConditionalLockBufferForCleanup() would have returned false.
>
> That's already true today, right? In master ConditionalLockBufferForCleanup()
> for temp buffers checks LocalRefCount, whereas LockBufferForCleanup() doesn't.
I'm finding a LocalRefCount check under LockBufferForCleanup:
LockBufferForCleanup(Buffer buffer)
{
...
CheckBufferIsPinnedOnce(buffer);
CheckBufferIsPinnedOnce(Buffer buffer)
{
if (BufferIsLocal(buffer))
{
if (LocalRefCount[-buffer - 1] != 1)
elog(ERROR, "incorrect local pin count: %d",
LocalRefCount[-buffer - 1]);
}
else
{
if (GetPrivateRefCount(buffer) != 1)
elog(ERROR, "incorrect local pin count: %d",
GetPrivateRefCount(buffer));
}
}
> > Like the comment, I expect it's academic today. I expect it will stay
> > academic. Anything that does a cleanup will start by reading the buffer,
> > which will resolve any refcnt the AIO subsystems holds for a read. If there's
> > an AIO write, the LockBuffer(buffer, BUFFER_LOCK_EXCLUSIVE) will block on
> > that. How about just removing the ConditionalLockBufferForCleanup() changes
> > or replacing them with a comment (like the present paragraph)?
>
> I think we'll need an expanded version of what I suggest once we have writes -
> but as you say, it shouldn't matter as long as we only have reads. So I think
> moving the relevant changes, with adjusted caveats, to the bufmgr: write
> change makes sense.
Moving those changes works for me. I'm not currently seeing the need under
writes, but that may get clearer upon reaching those patches.
> > > > /* ---
> > > > * Opt-in to using AIO batchmode.
> > > > *
> > > > * Submitting IO in larger batches can be more efficient than doing so
> > > > * one-by-one, particularly for many small reads. It does, however, require
> > > > * the ReadStreamBlockNumberCB callback to abide by the restrictions of AIO
> > > > * batching (c.f. pgaio_enter_batchmode()). Basically, the callback may not:
> > > > * a) block without first calling pgaio_submit_staged(), unless a
> > > > * to-be-waited-on lock cannot be part of a deadlock, e.g. because it is
> > > > * never acquired in a nested fashion
> > > > * b) directly or indirectly start another batch pgaio_enter_batchmode()
> >
> > I think a callback could still do:
> >
> > pgaio_exit_batchmode()
> > ... arbitrary code that might reach pgaio_enter_batchmode() ...
> > pgaio_enter_batchmode()
>
> Yea - but I somehow doubt there are many cases where it makes sense to
> deep-queue IOs within the callback. The cases I can think of are things like
> ensuring the right VM buffer is in s_b. But if it turns out to be necessary,
> what you seuggest would be an out.
I don't foresee a callback specifically wanting to batch, but callbacks might
call into other infrastructure that can elect to batch. The exit+reenter
pattern would be better than adding no-batch options to other infrastructure.
> Do you think it's worth mentioning the above workaround? I'm mildly inclined
> that not.
Perhaps not in that detail, but perhaps we can rephrase (b) to not imply
exit+reenter is banned. Maybe "(b) start another batch (without first exiting
one)". It's also fine as-is, though.
> If it turns out to be actually useful to do nested batching, we can change it
> so that nested batching *is* allowed, that'd not be hard.
Good point.
> I'm ok with all of these. In order of preference:
>
> 1) READ_STREAM_USE_BATCHING or READ_STREAM_BATCH_OK
> 2) READ_STREAM_BATCHMODE_AWARE
> 3) READ_STREAM_CALLBACK_BATCHMODE_AWARE
Same for me.
^ permalink raw reply [nested|flat] 153+ messages in thread
* Re: AIO v2.5
2025-03-04 19:00 AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-10 18:23 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-11 19:41 ` Re: AIO v2.5 Noah Misch <noah@leadboat.com>
2025-03-11 23:55 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-12 03:57 ` Re: AIO v2.5 Noah Misch <noah@leadboat.com>
2025-03-12 17:06 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-12 23:23 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-13 10:53 ` Re: AIO v2.5 Antonin Houska <ah@cybertec.at>
2025-03-14 19:43 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-17 20:00 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-18 20:12 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-21 01:58 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-24 00:29 ` Re: AIO v2.5 Noah Misch <noah@leadboat.com>
2025-03-24 15:43 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-24 22:55 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-25 00:07 ` Re: AIO v2.5 Thomas Munro <thomas.munro@gmail.com>
2025-03-25 00:45 ` Re: AIO v2.5 Noah Misch <noah@leadboat.com>
2025-03-25 02:30 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-25 13:33 ` Re: AIO v2.5 Noah Misch <noah@leadboat.com>
@ 2025-03-25 15:26 ` Andres Freund <andres@anarazel.de>
2025-03-25 15:58 ` Re: AIO v2.5 Noah Misch <noah@leadboat.com>
0 siblings, 1 reply; 153+ messages in thread
From: Andres Freund @ 2025-03-25 15:26 UTC (permalink / raw)
To: Noah Misch <noah@leadboat.com>; +Cc: Thomas Munro <thomas.munro@gmail.com>; Antonin Houska <ah@cybertec.at>; pgsql-hackers@postgresql.org, Heikki Linnakangas <hlinnaka@iki.fi>; Robert Haas <robertmhaas@gmail.com>; Jakub Wartak <jakub.wartak@enterprisedb.com>; Jelte Fennema-Nio <postgres@jeltef.nl>
Hi,
On 2025-03-25 06:33:21 -0700, Noah Misch wrote:
> On Mon, Mar 24, 2025 at 10:30:27PM -0400, Andres Freund wrote:
> > On 2025-03-24 17:45:37 -0700, Noah Misch wrote:
> > > (We may be due for a test mode that does smgrreleaseall() at every
> > > CHECK_FOR_INTERRUPTS()?)
> >
> > I suspect we are. I'm a bit afraid of even trying...
> >
> > ...
> >
> > It's extremely slow - but at least the main regression as well as the aio tests pass!
>
> One less thing!
Unfortunately I'm now doubting the thoroughness of my check - while I made
every CFI() execute smgrreleaseall(), I didn't trigger CFI() in cases where we
trigger it conditionally. E.g. elog(DEBUGN, ...) only executes a CFI if
log_min_messages <= DEBUGN...
I'll try that in a bit.
> > Because the end state varies, depending on the number of previously staged
> > IOs, the IO method and whether batchmode is enabled, I think it's better if
> > the "function naming pattern" (i.e. FileStartReadv, smgrstartreadv etc) is
> > *not* aligned with an internal state name. It will just mislead readers to
> > think that there's a deterministic mapping when that does not exist.
>
> That's fair. Could we provide the mapping in a comment, something like the
> following?
Yes!
I wonder if it should also be duplicated or referenced elsewhere, although I
am not sure where precisely.
> --- a/src/include/storage/aio_internal.h
> +++ b/src/include/storage/aio_internal.h
> @@ -34,5 +34,10 @@
> * linearly through all states.
> *
> - * State changes should all go through pgaio_io_update_state().
> + * State changes should all go through pgaio_io_update_state(). Its callers
> + * use these naming conventions:
> + *
> + * - A "start" function (e.g. FileStartReadV()) moves an IO from
> + * PGAIO_HS_HANDED_OUT to at least PGAIO_HS_STAGED and at most
> + * PGAIO_HS_COMPLETED_LOCAL.
> */
> typedef enum PgAioHandleState
One detail I'm not sure about: The above change is correct, but perhaps a bit
misleading, because we can actually go "back" to IDLE. Not sure how to best
phrase that though.
> > That's not an excuse for pgaio_io_prep* though, that's a pointlessly different
> > naming that I just stopped seeing.
I assume you're on board with renaming _io_prep* to _io_start_*?
> > I'll try to think more about this, perhaps I can make myself see your POV
> > more.
>
> > > As the patch stands, LockBufferForCleanup() can succeed when
> > > ConditionalLockBufferForCleanup() would have returned false.
> >
> > That's already true today, right? In master ConditionalLockBufferForCleanup()
> > for temp buffers checks LocalRefCount, whereas LockBufferForCleanup() doesn't.
>
> I'm finding a LocalRefCount check under LockBufferForCleanup:
I guess I should have stopped looking at code / replying before my last email
last night... Not sure how I missed that.
> CheckBufferIsPinnedOnce(Buffer buffer)
> {
> if (BufferIsLocal(buffer))
> {
> if (LocalRefCount[-buffer - 1] != 1)
> elog(ERROR, "incorrect local pin count: %d",
> LocalRefCount[-buffer - 1]);
> }
> else
> {
> if (GetPrivateRefCount(buffer) != 1)
> elog(ERROR, "incorrect local pin count: %d",
> GetPrivateRefCount(buffer));
> }
> }
Pretty random orthogonal thought, that I was reminded of by the above code
snippet:
It sure seems we should at some point get rid of LocalRefCount[] and just use
the GetPrivateRefCount() infrastructure for both shared and local buffers. I
don't think the GetPrivateRefCount() infrastructure cares about
local/non-local, leaving a few asserts aside. If we do that, and start to use
BM_IO_IN_PROGRESS, combined with ResourceOwnerRememberBufferIO(), the set of
differences between shared and local buffers would be a lot smaller.
> > > Like the comment, I expect it's academic today. I expect it will stay
> > > academic. Anything that does a cleanup will start by reading the buffer,
> > > which will resolve any refcnt the AIO subsystems holds for a read. If there's
> > > an AIO write, the LockBuffer(buffer, BUFFER_LOCK_EXCLUSIVE) will block on
> > > that. How about just removing the ConditionalLockBufferForCleanup() changes
> > > or replacing them with a comment (like the present paragraph)?
> >
> > I think we'll need an expanded version of what I suggest once we have writes -
> > but as you say, it shouldn't matter as long as we only have reads. So I think
> > moving the relevant changes, with adjusted caveats, to the bufmgr: write
> > change makes sense.
>
> Moving those changes works for me. I'm not currently seeing the need under
> writes, but that may get clearer upon reaching those patches.
FWIW, I don't think it's currently worth looking at the write side in detail,
there's enough required changes to make that not necessarily the best use of
your time at this point. At least:
- Write logic needs to be rebased ontop of the patch series to not hit bit
dirty buffers while IO is going on
The performance impact of doing the memory copies is rather substantial, as
on intel memory bandwidth is *the* IO bottleneck even just for the checksum
computation, without a copy. That makes the memory copy for something like
bounce buffers hurt really badly.
And the memory usage of bounce buffers is also really concerning.
And even without checksums, several filesystems *really* don't like buffers
getting modified during DIO writes. Which I think would mean we ought to use
bounce buffers for *all* writes, which would impose a *very* substantial
overhead (basically removing the benefit of DMA happening off-cpu).
- Right now the sync.c integration with smgr.c/md.c isn't properly safe to use
in a critical section
The only reason it doesn't immediately fail is that it's reasonably rare
that RegisterSyncRequest() fails *and* either:
- smgropen()->hash_search(HASH_ENTER) decides to resize the hash table, even
though the lookup is guaranteed to suceed for io_method=worker.
- an io_method=uring completion is run in a different backend and smgropen()
needs to build a new entry and thus needs to allocate memory
For a bit I thought this could be worked around easily enough by not doing
an smgropen() in mdsyncfiletag(), or adding a "fallible" smgropen() and
instead just opening the file directly. That actually does kinda solve the
problem, but only because the memory allocation in PathNameOpenFile()
uses malloc(), not palloc() and thus doesn't trigger
- I think it requires new lwlock.c infrastructure (as v1 of aio had), to make
LockBuffer(BUFFER_LOCK_EXCLUSIVE) etc wait in a concurrency safe manner for
in-progress writes
I can think of ways to solve this purely in bufmgr.c, but only in ways that
would cause other problems (e.g. setting BM_IO_IN_PROGRESS before waiting
for an exclusive lock) and/or expensive.
- My current set of patches doesn't implement bgwriter_flush_after,
checkpointer_flush_after
I think that's not too hard to do, it's mainly round tuits.
- temp_file_limit is not respected by aio writes
I guess that could be ok if AIO writes are only used by checkpointer /
bgwriter, but we need to figure out a way to deal with that. Perhaps by
redesigning temp_file_limit, the current implementation seems like rather
substantial layering violation.
- Too much duplicated code, as there's the aio and non-aio write paths. That
might be ok for a bit.
I updated the commit messages of the relevant commits with the above, there
were abbreviated versions of most of the above, but not in enough detail for
anybody but me (and maybe not even that).
> > Do you think it's worth mentioning the above workaround? I'm mildly inclined
> > that not.
>
> Perhaps not in that detail, but perhaps we can rephrase (b) to not imply
> exit+reenter is banned. Maybe "(b) start another batch (without first exiting
> one)". It's also fine as-is, though.
I updated it to:
* b) start another batch (without first exiting batchmode and re-entering
* before returning)
> > I'm ok with all of these. In order of preference:
> >
> > 1) READ_STREAM_USE_BATCHING or READ_STREAM_BATCH_OK
> > 2) READ_STREAM_BATCHMODE_AWARE
> > 3) READ_STREAM_CALLBACK_BATCHMODE_AWARE
>
> Same for me.
For now I'll leave it at READ_STREAM_USE_BATCHING, but if Thomas has a
preference I'll go for whatever we have a majority for.
Greetings,
Andres Freund
^ permalink raw reply [nested|flat] 153+ messages in thread
* Re: AIO v2.5
2025-03-04 19:00 AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-10 18:23 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-11 19:41 ` Re: AIO v2.5 Noah Misch <noah@leadboat.com>
2025-03-11 23:55 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-12 03:57 ` Re: AIO v2.5 Noah Misch <noah@leadboat.com>
2025-03-12 17:06 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-12 23:23 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-13 10:53 ` Re: AIO v2.5 Antonin Houska <ah@cybertec.at>
2025-03-14 19:43 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-17 20:00 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-18 20:12 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-21 01:58 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-24 00:29 ` Re: AIO v2.5 Noah Misch <noah@leadboat.com>
2025-03-24 15:43 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-24 22:55 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-25 00:07 ` Re: AIO v2.5 Thomas Munro <thomas.munro@gmail.com>
2025-03-25 00:45 ` Re: AIO v2.5 Noah Misch <noah@leadboat.com>
2025-03-25 02:30 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-25 13:33 ` Re: AIO v2.5 Noah Misch <noah@leadboat.com>
2025-03-25 15:26 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
@ 2025-03-25 15:58 ` Noah Misch <noah@leadboat.com>
2025-03-25 18:58 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
0 siblings, 1 reply; 153+ messages in thread
From: Noah Misch @ 2025-03-25 15:58 UTC (permalink / raw)
To: Andres Freund <andres@anarazel.de>; +Cc: Thomas Munro <thomas.munro@gmail.com>; Antonin Houska <ah@cybertec.at>; pgsql-hackers@postgresql.org, Heikki Linnakangas <hlinnaka@iki.fi>; Robert Haas <robertmhaas@gmail.com>; Jakub Wartak <jakub.wartak@enterprisedb.com>; Jelte Fennema-Nio <postgres@jeltef.nl>
On Tue, Mar 25, 2025 at 11:26:14AM -0400, Andres Freund wrote:
> On 2025-03-25 06:33:21 -0700, Noah Misch wrote:
> > On Mon, Mar 24, 2025 at 10:30:27PM -0400, Andres Freund wrote:
> > > On 2025-03-24 17:45:37 -0700, Noah Misch wrote:
> > > > (We may be due for a test mode that does smgrreleaseall() at every
> > > > CHECK_FOR_INTERRUPTS()?)
> > >
> > > I suspect we are. I'm a bit afraid of even trying...
> > >
> > > ...
> > >
> > > It's extremely slow - but at least the main regression as well as the aio tests pass!
> >
> > One less thing!
>
> Unfortunately I'm now doubting the thoroughness of my check - while I made
> every CFI() execute smgrreleaseall(), I didn't trigger CFI() in cases where we
> trigger it conditionally. E.g. elog(DEBUGN, ...) only executes a CFI if
> log_min_messages <= DEBUGN...
>
> I'll try that in a bit.
While having nagging thoughts that we might be releasing FDs before io_uring
gets them into kernel custody, I tried this hack to maximize FD turnover:
static void
ReleaseLruFiles(void)
{
#if 0
while (nfile + numAllocatedDescs + numExternalFDs >= max_safe_fds)
{
if (!ReleaseLruFile())
break;
}
#else
while (ReleaseLruFile())
;
#endif
}
"make check" with default settings (io_method=worker) passes, but
io_method=io_uring in the TEMP_CONFIG file got different diffs in each of two
runs. s/#if 0/#if 1/ (restore normal FD turnover) removes the failures.
Here's the richer of the two diffs:
diff -U3 src/test/regress/expected/sanity_check.out src/test/regress/results/sanity_check.out
--- src/test/regress/expected/sanity_check.out 2024-10-24 12:43:25.741817594 -0700
+++ src/test/regress/results/sanity_check.out 2025-03-25 08:27:44.875151566 -0700
@@ -1,4 +1,7 @@
VACUUM;
+ERROR: index "pg_enum_oid_index" contains corrupted page at block 2
+HINT: Please REINDEX it.
+CONTEXT: while vacuuming index "pg_enum_oid_index" of relation "pg_catalog.pg_enum"
--
-- Sanity check: every system catalog that has OIDs should have
-- a unique index on OID. This ensures that the OIDs will be unique,
diff -U3 src/test/regress/expected/oidjoins.out src/test/regress/results/oidjoins.out
--- src/test/regress/expected/oidjoins.out 2023-07-06 19:58:07.686364439 -0700
+++ src/test/regress/results/oidjoins.out 2025-03-25 08:28:02.584335458 -0700
@@ -233,6 +233,8 @@
NOTICE: checking pg_policy {polrelid} => pg_class {oid}
NOTICE: checking pg_policy {polroles} => pg_authid {oid}
NOTICE: checking pg_default_acl {defaclrole} => pg_authid {oid}
+WARNING: FK VIOLATION IN pg_default_acl({defaclrole}): ("(1,5)",0)
+WARNING: FK VIOLATION IN pg_default_acl({defaclrole}): ("(1,7)",402654464)
NOTICE: checking pg_default_acl {defaclnamespace} => pg_namespace {oid}
NOTICE: checking pg_init_privs {classoid} => pg_class {oid}
NOTICE: checking pg_seclabel {classoid} => pg_class {oid}
> > > Because the end state varies, depending on the number of previously staged
> > > IOs, the IO method and whether batchmode is enabled, I think it's better if
> > > the "function naming pattern" (i.e. FileStartReadv, smgrstartreadv etc) is
> > > *not* aligned with an internal state name. It will just mislead readers to
> > > think that there's a deterministic mapping when that does not exist.
> >
> > That's fair. Could we provide the mapping in a comment, something like the
> > following?
>
> Yes!
>
> I wonder if it should also be duplicated or referenced elsewhere, although I
> am not sure where precisely.
I considered the README.md also, but adding that wasn't an obvious win.
> > --- a/src/include/storage/aio_internal.h
> > +++ b/src/include/storage/aio_internal.h
> > @@ -34,5 +34,10 @@
> > * linearly through all states.
> > *
> > - * State changes should all go through pgaio_io_update_state().
> > + * State changes should all go through pgaio_io_update_state(). Its callers
> > + * use these naming conventions:
> > + *
> > + * - A "start" function (e.g. FileStartReadV()) moves an IO from
> > + * PGAIO_HS_HANDED_OUT to at least PGAIO_HS_STAGED and at most
> > + * PGAIO_HS_COMPLETED_LOCAL.
> > */
> > typedef enum PgAioHandleState
>
> One detail I'm not sure about: The above change is correct, but perhaps a bit
> misleading, because we can actually go "back" to IDLE. Not sure how to best
> phrase that though.
Not sure either. Maybe the above could change to "to PGAIO_HS_STAGED or any
subsequent state" and the comment at PGAIO_HS_STAGED could say like "Once in
this state, concurrent activity could move the IO all the way to
PGAIO_HS_COMPLETED_LOCAL and recycle it back to IDLE."
> > > That's not an excuse for pgaio_io_prep* though, that's a pointlessly different
> > > naming that I just stopped seeing.
>
> I assume you're on board with renaming _io_prep* to _io_start_*?
Yes.
> > > I'll try to think more about this, perhaps I can make myself see your POV
> > > more.
> > CheckBufferIsPinnedOnce(Buffer buffer)
> > {
> > if (BufferIsLocal(buffer))
> > {
> > if (LocalRefCount[-buffer - 1] != 1)
> > elog(ERROR, "incorrect local pin count: %d",
> > LocalRefCount[-buffer - 1]);
> > }
> > else
> > {
> > if (GetPrivateRefCount(buffer) != 1)
> > elog(ERROR, "incorrect local pin count: %d",
> > GetPrivateRefCount(buffer));
> > }
> > }
>
> Pretty random orthogonal thought, that I was reminded of by the above code
> snippet:
>
> It sure seems we should at some point get rid of LocalRefCount[] and just use
> the GetPrivateRefCount() infrastructure for both shared and local buffers. I
> don't think the GetPrivateRefCount() infrastructure cares about
> local/non-local, leaving a few asserts aside. If we do that, and start to use
> BM_IO_IN_PROGRESS, combined with ResourceOwnerRememberBufferIO(), the set of
> differences between shared and local buffers would be a lot smaller.
That sounds promising.
> > > > Like the comment, I expect it's academic today. I expect it will stay
> > > > academic. Anything that does a cleanup will start by reading the buffer,
> > > > which will resolve any refcnt the AIO subsystems holds for a read. If there's
> > > > an AIO write, the LockBuffer(buffer, BUFFER_LOCK_EXCLUSIVE) will block on
> > > > that. How about just removing the ConditionalLockBufferForCleanup() changes
> > > > or replacing them with a comment (like the present paragraph)?
> > >
> > > I think we'll need an expanded version of what I suggest once we have writes -
> > > but as you say, it shouldn't matter as long as we only have reads. So I think
> > > moving the relevant changes, with adjusted caveats, to the bufmgr: write
> > > change makes sense.
> >
> > Moving those changes works for me. I'm not currently seeing the need under
> > writes, but that may get clearer upon reaching those patches.
>
> FWIW, I don't think it's currently worth looking at the write side in detail,
Got it. (I meant I didn't see a first-principles need, not that I had deduced
lack of need from a specific writes implementation.)
> > > Do you think it's worth mentioning the above workaround? I'm mildly inclined
> > > that not.
> >
> > Perhaps not in that detail, but perhaps we can rephrase (b) to not imply
> > exit+reenter is banned. Maybe "(b) start another batch (without first exiting
> > one)". It's also fine as-is, though.
>
> I updated it to:
>
> * b) start another batch (without first exiting batchmode and re-entering
> * before returning)
That's good.
^ permalink raw reply [nested|flat] 153+ messages in thread
* Re: AIO v2.5
2025-03-04 19:00 AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-10 18:23 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-11 19:41 ` Re: AIO v2.5 Noah Misch <noah@leadboat.com>
2025-03-11 23:55 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-12 03:57 ` Re: AIO v2.5 Noah Misch <noah@leadboat.com>
2025-03-12 17:06 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-12 23:23 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-13 10:53 ` Re: AIO v2.5 Antonin Houska <ah@cybertec.at>
2025-03-14 19:43 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-17 20:00 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-18 20:12 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-21 01:58 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-24 00:29 ` Re: AIO v2.5 Noah Misch <noah@leadboat.com>
2025-03-24 15:43 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-24 22:55 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-25 00:07 ` Re: AIO v2.5 Thomas Munro <thomas.munro@gmail.com>
2025-03-25 00:45 ` Re: AIO v2.5 Noah Misch <noah@leadboat.com>
2025-03-25 02:30 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-25 13:33 ` Re: AIO v2.5 Noah Misch <noah@leadboat.com>
2025-03-25 15:26 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-25 15:58 ` Re: AIO v2.5 Noah Misch <noah@leadboat.com>
@ 2025-03-25 18:58 ` Andres Freund <andres@anarazel.de>
2025-03-25 19:39 ` Re: AIO v2.5 Noah Misch <noah@leadboat.com>
0 siblings, 1 reply; 153+ messages in thread
From: Andres Freund @ 2025-03-25 18:58 UTC (permalink / raw)
To: Noah Misch <noah@leadboat.com>; +Cc: Thomas Munro <thomas.munro@gmail.com>; Antonin Houska <ah@cybertec.at>; pgsql-hackers@postgresql.org, Heikki Linnakangas <hlinnaka@iki.fi>; Robert Haas <robertmhaas@gmail.com>; Jakub Wartak <jakub.wartak@enterprisedb.com>; Jelte Fennema-Nio <postgres@jeltef.nl>
Hi,
On 2025-03-25 08:58:08 -0700, Noah Misch wrote:
> While having nagging thoughts that we might be releasing FDs before io_uring
> gets them into kernel custody, I tried this hack to maximize FD turnover:
>
> static void
> ReleaseLruFiles(void)
> {
> #if 0
> while (nfile + numAllocatedDescs + numExternalFDs >= max_safe_fds)
> {
> if (!ReleaseLruFile())
> break;
> }
> #else
> while (ReleaseLruFile())
> ;
> #endif
> }
>
> "make check" with default settings (io_method=worker) passes, but
> io_method=io_uring in the TEMP_CONFIG file got different diffs in each of two
> runs. s/#if 0/#if 1/ (restore normal FD turnover) removes the failures.
> Here's the richer of the two diffs:
Yikes. That's a very good catch.
I spent a bit of time debugging this. I think I see what's going on - it turns
out that the kernel does *not* open the FDs during io_uring_enter() if
IOSQE_ASYNC is specified [1]. Which we do add heuristically, in an attempt to
avoid a small but measurable slowdown for sequential scans that are fully
buffered (c.f. pgaio_uring_submit()). If I disable that heuristic, your patch
above passes all tests here.
I don't know if that's an intentional or unintentional behavioral difference.
There are 2 1/2 ways around this:
1) Stop using IOSQE_ASYNC heuristic
2a) Wait for all in-flight IOs when any FD gets closed
2b) Wait for all in-flight IOs using FD when it gets closed
Given that we have clear evidence that io_uring doesn't completely support
closing FDs while IOs are in flight, be it a bug or intentional, it seems
clearly better to go for 2a or 2b.
Greetings,
Andres Freund
[1] Instead files are opened when the queue entry is being worked on
instead. Interestingly that only happens when the IO is *explicitly*
requested to be executed in the workqueue with IOSQE_ASYNC, not when it's
put there because it couldn't be done in a non-blocking way.
^ permalink raw reply [nested|flat] 153+ messages in thread
* Re: AIO v2.5
2025-03-04 19:00 AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-10 18:23 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-11 19:41 ` Re: AIO v2.5 Noah Misch <noah@leadboat.com>
2025-03-11 23:55 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-12 03:57 ` Re: AIO v2.5 Noah Misch <noah@leadboat.com>
2025-03-12 17:06 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-12 23:23 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-13 10:53 ` Re: AIO v2.5 Antonin Houska <ah@cybertec.at>
2025-03-14 19:43 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-17 20:00 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-18 20:12 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-21 01:58 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-24 00:29 ` Re: AIO v2.5 Noah Misch <noah@leadboat.com>
2025-03-24 15:43 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-24 22:55 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-25 00:07 ` Re: AIO v2.5 Thomas Munro <thomas.munro@gmail.com>
2025-03-25 00:45 ` Re: AIO v2.5 Noah Misch <noah@leadboat.com>
2025-03-25 02:30 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-25 13:33 ` Re: AIO v2.5 Noah Misch <noah@leadboat.com>
2025-03-25 15:26 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-25 15:58 ` Re: AIO v2.5 Noah Misch <noah@leadboat.com>
2025-03-25 18:58 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
@ 2025-03-25 19:39 ` Noah Misch <noah@leadboat.com>
2025-03-25 20:07 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
0 siblings, 1 reply; 153+ messages in thread
From: Noah Misch @ 2025-03-25 19:39 UTC (permalink / raw)
To: Andres Freund <andres@anarazel.de>; +Cc: Thomas Munro <thomas.munro@gmail.com>; Antonin Houska <ah@cybertec.at>; pgsql-hackers@postgresql.org, Heikki Linnakangas <hlinnaka@iki.fi>; Robert Haas <robertmhaas@gmail.com>; Jakub Wartak <jakub.wartak@enterprisedb.com>; Jelte Fennema-Nio <postgres@jeltef.nl>
On Tue, Mar 25, 2025 at 02:58:37PM -0400, Andres Freund wrote:
> On 2025-03-25 08:58:08 -0700, Noah Misch wrote:
> > While having nagging thoughts that we might be releasing FDs before io_uring
> > gets them into kernel custody, I tried this hack to maximize FD turnover:
> >
> > static void
> > ReleaseLruFiles(void)
> > {
> > #if 0
> > while (nfile + numAllocatedDescs + numExternalFDs >= max_safe_fds)
> > {
> > if (!ReleaseLruFile())
> > break;
> > }
> > #else
> > while (ReleaseLruFile())
> > ;
> > #endif
> > }
> >
> > "make check" with default settings (io_method=worker) passes, but
> > io_method=io_uring in the TEMP_CONFIG file got different diffs in each of two
> > runs. s/#if 0/#if 1/ (restore normal FD turnover) removes the failures.
> > Here's the richer of the two diffs:
>
> Yikes. That's a very good catch.
>
> I spent a bit of time debugging this. I think I see what's going on - it turns
> out that the kernel does *not* open the FDs during io_uring_enter() if
> IOSQE_ASYNC is specified [1]. Which we do add heuristically, in an attempt to
> avoid a small but measurable slowdown for sequential scans that are fully
> buffered (c.f. pgaio_uring_submit()). If I disable that heuristic, your patch
> above passes all tests here.
Same result here. As an additional data point, I tried adding this so every
reopen gets a new FD number (leaks FDs wildly):
--- a/src/backend/storage/file/fd.c
+++ b/src/backend/storage/file/fd.c
@@ -1304,5 +1304,5 @@ LruDelete(File file)
* to leak the FD than to mess up our internal state.
*/
- if (close(vfdP->fd) != 0)
+ if (dup2(2, vfdP->fd) != vfdP->fd)
elog(vfdP->fdstate & FD_TEMP_FILE_LIMIT ? LOG : data_sync_elevel(LOG),
"could not close file \"%s\": %m", vfdP->fileName);
The same "make check" w/ TEMP_CONFIG io_method=io_uring passes with the
combination of that and the max-turnover change to ReleaseLruFiles().
> I don't know if that's an intentional or unintentional behavioral difference.
>
> There are 2 1/2 ways around this:
>
> 1) Stop using IOSQE_ASYNC heuristic
> 2a) Wait for all in-flight IOs when any FD gets closed
> 2b) Wait for all in-flight IOs using FD when it gets closed
>
> Given that we have clear evidence that io_uring doesn't completely support
> closing FDs while IOs are in flight, be it a bug or intentional, it seems
> clearly better to go for 2a or 2b.
Agreed. If a workload spends significant time on fd.c closing files, I
suspect that workload already won't have impressive benchmark numbers.
Performance-seeking workloads will already want to tune FD usage high enough
to keep FDs long-lived. So (1) clearly loses, and neither (2a) nor (2b)
clearly beats the other. I'd try (2b) first but, if complicated, quickly
abandon it in favor of (2a). What other considerations could be important?
^ permalink raw reply [nested|flat] 153+ messages in thread
* Re: AIO v2.5
2025-03-04 19:00 AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-10 18:23 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-11 19:41 ` Re: AIO v2.5 Noah Misch <noah@leadboat.com>
2025-03-11 23:55 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-12 03:57 ` Re: AIO v2.5 Noah Misch <noah@leadboat.com>
2025-03-12 17:06 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-12 23:23 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-13 10:53 ` Re: AIO v2.5 Antonin Houska <ah@cybertec.at>
2025-03-14 19:43 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-17 20:00 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-18 20:12 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-21 01:58 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-24 00:29 ` Re: AIO v2.5 Noah Misch <noah@leadboat.com>
2025-03-24 15:43 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-24 22:55 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-25 00:07 ` Re: AIO v2.5 Thomas Munro <thomas.munro@gmail.com>
2025-03-25 00:45 ` Re: AIO v2.5 Noah Misch <noah@leadboat.com>
2025-03-25 02:30 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-25 13:33 ` Re: AIO v2.5 Noah Misch <noah@leadboat.com>
2025-03-25 15:26 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-25 15:58 ` Re: AIO v2.5 Noah Misch <noah@leadboat.com>
2025-03-25 18:58 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-25 19:39 ` Re: AIO v2.5 Noah Misch <noah@leadboat.com>
@ 2025-03-25 20:07 ` Andres Freund <andres@anarazel.de>
2025-03-25 20:18 ` Re: AIO v2.5 Noah Misch <noah@leadboat.com>
0 siblings, 1 reply; 153+ messages in thread
From: Andres Freund @ 2025-03-25 20:07 UTC (permalink / raw)
To: Noah Misch <noah@leadboat.com>; +Cc: Thomas Munro <thomas.munro@gmail.com>; Antonin Houska <ah@cybertec.at>; pgsql-hackers@postgresql.org, Heikki Linnakangas <hlinnaka@iki.fi>; Robert Haas <robertmhaas@gmail.com>; Jakub Wartak <jakub.wartak@enterprisedb.com>; Jelte Fennema-Nio <postgres@jeltef.nl>
Hi,
On 2025-03-25 12:39:56 -0700, Noah Misch wrote:
> On Tue, Mar 25, 2025 at 02:58:37PM -0400, Andres Freund wrote:
> > I don't know if that's an intentional or unintentional behavioral difference.
> >
> > There are 2 1/2 ways around this:
> >
> > 1) Stop using IOSQE_ASYNC heuristic
> > 2a) Wait for all in-flight IOs when any FD gets closed
> > 2b) Wait for all in-flight IOs using FD when it gets closed
> >
> > Given that we have clear evidence that io_uring doesn't completely support
> > closing FDs while IOs are in flight, be it a bug or intentional, it seems
> > clearly better to go for 2a or 2b.
>
> Agreed. If a workload spends significant time on fd.c closing files, I
> suspect that workload already won't have impressive benchmark numbers.
> Performance-seeking workloads will already want to tune FD usage high enough
> to keep FDs long-lived. So (1) clearly loses, and neither (2a) nor (2b)
> clearly beats the other. I'd try (2b) first but, if complicated, quickly
> abandon it in favor of (2a). What other considerations could be important?
The only other consideration I can think of is whether this should happen for
all io_methods or not.
I'm inclined to do it via a bool in IoMethodOps, but I guess one could argue
it's a bit weird to have a bool in a struct called *Ops.
Greetings,
Andres Freund
^ permalink raw reply [nested|flat] 153+ messages in thread
* Re: AIO v2.5
2025-03-04 19:00 AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-10 18:23 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-11 19:41 ` Re: AIO v2.5 Noah Misch <noah@leadboat.com>
2025-03-11 23:55 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-12 03:57 ` Re: AIO v2.5 Noah Misch <noah@leadboat.com>
2025-03-12 17:06 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-12 23:23 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-13 10:53 ` Re: AIO v2.5 Antonin Houska <ah@cybertec.at>
2025-03-14 19:43 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-17 20:00 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-18 20:12 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-21 01:58 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-24 00:29 ` Re: AIO v2.5 Noah Misch <noah@leadboat.com>
2025-03-24 15:43 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-24 22:55 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-25 00:07 ` Re: AIO v2.5 Thomas Munro <thomas.munro@gmail.com>
2025-03-25 00:45 ` Re: AIO v2.5 Noah Misch <noah@leadboat.com>
2025-03-25 02:30 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-25 13:33 ` Re: AIO v2.5 Noah Misch <noah@leadboat.com>
2025-03-25 15:26 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-25 15:58 ` Re: AIO v2.5 Noah Misch <noah@leadboat.com>
2025-03-25 18:58 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-25 19:39 ` Re: AIO v2.5 Noah Misch <noah@leadboat.com>
2025-03-25 20:07 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
@ 2025-03-25 20:18 ` Noah Misch <noah@leadboat.com>
2025-03-25 20:56 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
0 siblings, 1 reply; 153+ messages in thread
From: Noah Misch @ 2025-03-25 20:18 UTC (permalink / raw)
To: Andres Freund <andres@anarazel.de>; +Cc: Thomas Munro <thomas.munro@gmail.com>; Antonin Houska <ah@cybertec.at>; pgsql-hackers@postgresql.org, Heikki Linnakangas <hlinnaka@iki.fi>; Robert Haas <robertmhaas@gmail.com>; Jakub Wartak <jakub.wartak@enterprisedb.com>; Jelte Fennema-Nio <postgres@jeltef.nl>
On Tue, Mar 25, 2025 at 04:07:35PM -0400, Andres Freund wrote:
> On 2025-03-25 12:39:56 -0700, Noah Misch wrote:
> > On Tue, Mar 25, 2025 at 02:58:37PM -0400, Andres Freund wrote:
> > > There are 2 1/2 ways around this:
> > >
> > > 1) Stop using IOSQE_ASYNC heuristic
> > > 2a) Wait for all in-flight IOs when any FD gets closed
> > > 2b) Wait for all in-flight IOs using FD when it gets closed
> > >
> > > Given that we have clear evidence that io_uring doesn't completely support
> > > closing FDs while IOs are in flight, be it a bug or intentional, it seems
> > > clearly better to go for 2a or 2b.
> >
> > Agreed. If a workload spends significant time on fd.c closing files, I
> > suspect that workload already won't have impressive benchmark numbers.
> > Performance-seeking workloads will already want to tune FD usage high enough
> > to keep FDs long-lived. So (1) clearly loses, and neither (2a) nor (2b)
> > clearly beats the other. I'd try (2b) first but, if complicated, quickly
> > abandon it in favor of (2a). What other considerations could be important?
>
> The only other consideration I can think of is whether this should happen for
> all io_methods or not.
Either way is fine, I think.
> I'm inclined to do it via a bool in IoMethodOps, but I guess one could argue
> it's a bit weird to have a bool in a struct called *Ops.
That wouldn't bother me. IndexAmRoutine has many bools, and "Ops" is
basically a synonym of "Routine".
^ permalink raw reply [nested|flat] 153+ messages in thread
* Re: AIO v2.5
2025-03-04 19:00 AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-10 18:23 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-11 19:41 ` Re: AIO v2.5 Noah Misch <noah@leadboat.com>
2025-03-11 23:55 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-12 03:57 ` Re: AIO v2.5 Noah Misch <noah@leadboat.com>
2025-03-12 17:06 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-12 23:23 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-13 10:53 ` Re: AIO v2.5 Antonin Houska <ah@cybertec.at>
2025-03-14 19:43 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-17 20:00 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-18 20:12 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-21 01:58 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-24 00:29 ` Re: AIO v2.5 Noah Misch <noah@leadboat.com>
2025-03-24 15:43 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-24 22:55 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-25 00:07 ` Re: AIO v2.5 Thomas Munro <thomas.munro@gmail.com>
2025-03-25 00:45 ` Re: AIO v2.5 Noah Misch <noah@leadboat.com>
2025-03-25 02:30 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-25 13:33 ` Re: AIO v2.5 Noah Misch <noah@leadboat.com>
2025-03-25 15:26 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-25 15:58 ` Re: AIO v2.5 Noah Misch <noah@leadboat.com>
2025-03-25 18:58 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-25 19:39 ` Re: AIO v2.5 Noah Misch <noah@leadboat.com>
2025-03-25 20:07 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-25 20:18 ` Re: AIO v2.5 Noah Misch <noah@leadboat.com>
@ 2025-03-25 20:56 ` Andres Freund <andres@anarazel.de>
2025-03-25 22:44 ` Re: AIO v2.5 Noah Misch <noah@leadboat.com>
0 siblings, 1 reply; 153+ messages in thread
From: Andres Freund @ 2025-03-25 20:56 UTC (permalink / raw)
To: Noah Misch <noah@leadboat.com>; +Cc: Thomas Munro <thomas.munro@gmail.com>; Antonin Houska <ah@cybertec.at>; pgsql-hackers@postgresql.org, Heikki Linnakangas <hlinnaka@iki.fi>; Robert Haas <robertmhaas@gmail.com>; Jakub Wartak <jakub.wartak@enterprisedb.com>; Jelte Fennema-Nio <postgres@jeltef.nl>
Hi,
On 2025-03-25 13:18:50 -0700, Noah Misch wrote:
> On Tue, Mar 25, 2025 at 04:07:35PM -0400, Andres Freund wrote:
> > On 2025-03-25 12:39:56 -0700, Noah Misch wrote:
> > > On Tue, Mar 25, 2025 at 02:58:37PM -0400, Andres Freund wrote:
> > > > There are 2 1/2 ways around this:
> > > >
> > > > 1) Stop using IOSQE_ASYNC heuristic
> > > > 2a) Wait for all in-flight IOs when any FD gets closed
> > > > 2b) Wait for all in-flight IOs using FD when it gets closed
> > > >
> > > > Given that we have clear evidence that io_uring doesn't completely support
> > > > closing FDs while IOs are in flight, be it a bug or intentional, it seems
> > > > clearly better to go for 2a or 2b.
> > >
> > > Agreed. If a workload spends significant time on fd.c closing files, I
> > > suspect that workload already won't have impressive benchmark numbers.
> > > Performance-seeking workloads will already want to tune FD usage high enough
> > > to keep FDs long-lived. So (1) clearly loses, and neither (2a) nor (2b)
> > > clearly beats the other. I'd try (2b) first but, if complicated, quickly
> > > abandon it in favor of (2a). What other considerations could be important?
> >
> > The only other consideration I can think of is whether this should happen for
> > all io_methods or not.
>
> Either way is fine, I think.
Here's a draft incremental patch (attached as a .fixup to avoid triggering
cfbot) implementing 2b).
> > I'm inclined to do it via a bool in IoMethodOps, but I guess one could argue
> > it's a bit weird to have a bool in a struct called *Ops.
>
> That wouldn't bother me. IndexAmRoutine has many bools, and "Ops" is
> basically a synonym of "Routine".
Cool. Done that way.
The repeated-iteration approach taken in pgaio_closing_fd() isn't the
prettiest, but it's hard to to imagine that ever being a noticeable.
This survives a testrun where I use your torture patch and where I force all
IOs to use ASYNC. Previously that did not get very far. I also did verify
that, if I allow a small number of FDs, we do not wrongly wait for all IOs.
Greetings,
Andres Freund
commit 290de30c0e4a87ed3bd6cdef478376d39a922912
Author: Andres Freund <andres@anarazel.de>
Date: 2025-03-25 16:50:27 -0400
wip: aio: io_uring wait for in-flight IOs
Author:
Reviewed-by:
Discussion: https://postgr.es/m/
Backpatch:
diff --git a/src/include/storage/aio_internal.h b/src/include/storage/aio_internal.h
index a46d4393ebc..352610f60a8 100644
--- a/src/include/storage/aio_internal.h
+++ b/src/include/storage/aio_internal.h
@@ -287,6 +287,15 @@ typedef struct PgAioCtl
*/
typedef struct IoMethodOps
{
+ /* properties */
+
+ /*
+ * If an FD is about to be closed, do we need to wait for all in-flight
+ * IOs referencing that FD?
+ */
+ bool wait_on_fd_before_close;
+
+
/* global initialization */
/*
@@ -368,6 +377,7 @@ extern PgAioResult pgaio_io_call_complete_local(PgAioHandle *ioh);
/* aio_io.c */
extern void pgaio_io_perform_synchronously(PgAioHandle *ioh);
extern const char *pgaio_io_get_op_name(PgAioHandle *ioh);
+extern bool pgaio_io_uses_fd(PgAioHandle *ioh, int fd);
/* aio_target.c */
extern bool pgaio_io_can_reopen(PgAioHandle *ioh);
diff --git a/src/backend/storage/aio/aio.c b/src/backend/storage/aio/aio.c
index c512495dce3..8816ccfa0d7 100644
--- a/src/backend/storage/aio/aio.c
+++ b/src/backend/storage/aio/aio.c
@@ -1303,6 +1303,41 @@ pgaio_closing_fd(int fd)
* it's probably not worth it.
*/
pgaio_submit_staged();
+
+ /*
+ * If requested by the IO method, wait for all IOs that use the
+ * to-be-closed FD.
+ */
+ if (pgaio_method_ops->wait_on_fd_before_close)
+ {
+ /*
+ * As waiting for one IO to complete may complete multiple IOs, we
+ * can't just use a mutable list iterator. The maximum number of
+ * in-flight IOs is fairly small, so just restart the loop after
+ * waiting for an IO.
+ */
+ while (!dclist_is_empty(&pgaio_my_backend->in_flight_ios))
+ {
+ dlist_iter iter;
+ PgAioHandle *ioh = NULL;
+
+ dclist_foreach(iter, &pgaio_my_backend->in_flight_ios)
+ {
+ ioh = dclist_container(PgAioHandle, node, iter.cur);
+
+ if (pgaio_io_uses_fd(ioh, fd))
+ break;
+ else
+ ioh = NULL;
+ }
+
+ if (!ioh)
+ break;
+
+ /* see comment in pgaio_io_wait_for_free() about raciness */
+ pgaio_io_wait(ioh, ioh->generation);
+ }
+ }
}
/*
diff --git a/src/backend/storage/aio/aio_io.c b/src/backend/storage/aio/aio_io.c
index 97930d687c2..108021fa377 100644
--- a/src/backend/storage/aio/aio_io.c
+++ b/src/backend/storage/aio/aio_io.c
@@ -186,3 +186,25 @@ pgaio_io_get_op_name(PgAioHandle *ioh)
return NULL; /* silence compiler */
}
+
+/*
+ * Used to determine if an IO needs to be waited upon before the file
+ * descriptor can be closed.
+ */
+bool
+pgaio_io_uses_fd(PgAioHandle *ioh, int fd)
+{
+ Assert(ioh->state >= PGAIO_HS_DEFINED);
+
+ switch (ioh->op)
+ {
+ case PGAIO_OP_READV:
+ return ioh->op_data.read.fd == fd;
+ case PGAIO_OP_WRITEV:
+ return ioh->op_data.write.fd == fd;
+ case PGAIO_OP_INVALID:
+ return false;
+ }
+
+ return false; /* silence compiler */
+}
diff --git a/src/backend/storage/aio/method_io_uring.c b/src/backend/storage/aio/method_io_uring.c
index aab09793684..3b299dcf388 100644
--- a/src/backend/storage/aio/method_io_uring.c
+++ b/src/backend/storage/aio/method_io_uring.c
@@ -58,6 +58,15 @@ static void pgaio_uring_sq_from_io(PgAioHandle *ioh, struct io_uring_sqe *sqe);
const IoMethodOps pgaio_uring_ops = {
+ /*
+ * While io_uring mostly is OK with FDs getting closed while the IO is in
+ * flight, that is not true for IOs submitted with IOSQE_ASYNC.
+ *
+ * See
+ * https://postgr.es/m/5ons2rtmwarqqhhexb3dnqulw5rjgwgoct57vpdau4rujlrffj%403fls6d2mkiwc
+ */
+ .wait_on_fd_before_close = true,
+
.shmem_size = pgaio_uring_shmem_size,
.shmem_init = pgaio_uring_shmem_init,
.init_backend = pgaio_uring_init_backend,
Attachments:
[text/plain] open_fd.fixup (3.7K, ../../xp62aoh554ufuz64aizgozxxjc6cltftthaa5my7sxzlwiefbm@f2pqjvkffwqo/2-open_fd.fixup)
download | inline diff:
commit 290de30c0e4a87ed3bd6cdef478376d39a922912
Author: Andres Freund <andres@anarazel.de>
Date: 2025-03-25 16:50:27 -0400
wip: aio: io_uring wait for in-flight IOs
Author:
Reviewed-by:
Discussion: https://postgr.es/m/
Backpatch:
diff --git a/src/include/storage/aio_internal.h b/src/include/storage/aio_internal.h
index a46d4393ebc..352610f60a8 100644
--- a/src/include/storage/aio_internal.h
+++ b/src/include/storage/aio_internal.h
@@ -287,6 +287,15 @@ typedef struct PgAioCtl
*/
typedef struct IoMethodOps
{
+ /* properties */
+
+ /*
+ * If an FD is about to be closed, do we need to wait for all in-flight
+ * IOs referencing that FD?
+ */
+ bool wait_on_fd_before_close;
+
+
/* global initialization */
/*
@@ -368,6 +377,7 @@ extern PgAioResult pgaio_io_call_complete_local(PgAioHandle *ioh);
/* aio_io.c */
extern void pgaio_io_perform_synchronously(PgAioHandle *ioh);
extern const char *pgaio_io_get_op_name(PgAioHandle *ioh);
+extern bool pgaio_io_uses_fd(PgAioHandle *ioh, int fd);
/* aio_target.c */
extern bool pgaio_io_can_reopen(PgAioHandle *ioh);
diff --git a/src/backend/storage/aio/aio.c b/src/backend/storage/aio/aio.c
index c512495dce3..8816ccfa0d7 100644
--- a/src/backend/storage/aio/aio.c
+++ b/src/backend/storage/aio/aio.c
@@ -1303,6 +1303,41 @@ pgaio_closing_fd(int fd)
* it's probably not worth it.
*/
pgaio_submit_staged();
+
+ /*
+ * If requested by the IO method, wait for all IOs that use the
+ * to-be-closed FD.
+ */
+ if (pgaio_method_ops->wait_on_fd_before_close)
+ {
+ /*
+ * As waiting for one IO to complete may complete multiple IOs, we
+ * can't just use a mutable list iterator. The maximum number of
+ * in-flight IOs is fairly small, so just restart the loop after
+ * waiting for an IO.
+ */
+ while (!dclist_is_empty(&pgaio_my_backend->in_flight_ios))
+ {
+ dlist_iter iter;
+ PgAioHandle *ioh = NULL;
+
+ dclist_foreach(iter, &pgaio_my_backend->in_flight_ios)
+ {
+ ioh = dclist_container(PgAioHandle, node, iter.cur);
+
+ if (pgaio_io_uses_fd(ioh, fd))
+ break;
+ else
+ ioh = NULL;
+ }
+
+ if (!ioh)
+ break;
+
+ /* see comment in pgaio_io_wait_for_free() about raciness */
+ pgaio_io_wait(ioh, ioh->generation);
+ }
+ }
}
/*
diff --git a/src/backend/storage/aio/aio_io.c b/src/backend/storage/aio/aio_io.c
index 97930d687c2..108021fa377 100644
--- a/src/backend/storage/aio/aio_io.c
+++ b/src/backend/storage/aio/aio_io.c
@@ -186,3 +186,25 @@ pgaio_io_get_op_name(PgAioHandle *ioh)
return NULL; /* silence compiler */
}
+
+/*
+ * Used to determine if an IO needs to be waited upon before the file
+ * descriptor can be closed.
+ */
+bool
+pgaio_io_uses_fd(PgAioHandle *ioh, int fd)
+{
+ Assert(ioh->state >= PGAIO_HS_DEFINED);
+
+ switch (ioh->op)
+ {
+ case PGAIO_OP_READV:
+ return ioh->op_data.read.fd == fd;
+ case PGAIO_OP_WRITEV:
+ return ioh->op_data.write.fd == fd;
+ case PGAIO_OP_INVALID:
+ return false;
+ }
+
+ return false; /* silence compiler */
+}
diff --git a/src/backend/storage/aio/method_io_uring.c b/src/backend/storage/aio/method_io_uring.c
index aab09793684..3b299dcf388 100644
--- a/src/backend/storage/aio/method_io_uring.c
+++ b/src/backend/storage/aio/method_io_uring.c
@@ -58,6 +58,15 @@ static void pgaio_uring_sq_from_io(PgAioHandle *ioh, struct io_uring_sqe *sqe);
const IoMethodOps pgaio_uring_ops = {
+ /*
+ * While io_uring mostly is OK with FDs getting closed while the IO is in
+ * flight, that is not true for IOs submitted with IOSQE_ASYNC.
+ *
+ * See
+ * https://postgr.es/m/5ons2rtmwarqqhhexb3dnqulw5rjgwgoct57vpdau4rujlrffj%403fls6d2mkiwc
+ */
+ .wait_on_fd_before_close = true,
+
.shmem_size = pgaio_uring_shmem_size,
.shmem_init = pgaio_uring_shmem_init,
.init_backend = pgaio_uring_init_backend,
^ permalink raw reply [nested|flat] 153+ messages in thread
* Re: AIO v2.5
2025-03-04 19:00 AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-10 18:23 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-11 19:41 ` Re: AIO v2.5 Noah Misch <noah@leadboat.com>
2025-03-11 23:55 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-12 03:57 ` Re: AIO v2.5 Noah Misch <noah@leadboat.com>
2025-03-12 17:06 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-12 23:23 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-13 10:53 ` Re: AIO v2.5 Antonin Houska <ah@cybertec.at>
2025-03-14 19:43 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-17 20:00 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-18 20:12 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-21 01:58 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-24 00:29 ` Re: AIO v2.5 Noah Misch <noah@leadboat.com>
2025-03-24 15:43 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-24 22:55 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-25 00:07 ` Re: AIO v2.5 Thomas Munro <thomas.munro@gmail.com>
2025-03-25 00:45 ` Re: AIO v2.5 Noah Misch <noah@leadboat.com>
2025-03-25 02:30 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-25 13:33 ` Re: AIO v2.5 Noah Misch <noah@leadboat.com>
2025-03-25 15:26 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-25 15:58 ` Re: AIO v2.5 Noah Misch <noah@leadboat.com>
2025-03-25 18:58 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-25 19:39 ` Re: AIO v2.5 Noah Misch <noah@leadboat.com>
2025-03-25 20:07 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-25 20:18 ` Re: AIO v2.5 Noah Misch <noah@leadboat.com>
2025-03-25 20:56 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
@ 2025-03-25 22:44 ` Noah Misch <noah@leadboat.com>
0 siblings, 0 replies; 153+ messages in thread
From: Noah Misch @ 2025-03-25 22:44 UTC (permalink / raw)
To: Andres Freund <andres@anarazel.de>; +Cc: Thomas Munro <thomas.munro@gmail.com>; Antonin Houska <ah@cybertec.at>; pgsql-hackers@postgresql.org, Heikki Linnakangas <hlinnaka@iki.fi>; Robert Haas <robertmhaas@gmail.com>; Jakub Wartak <jakub.wartak@enterprisedb.com>; Jelte Fennema-Nio <postgres@jeltef.nl>
On Tue, Mar 25, 2025 at 04:56:53PM -0400, Andres Freund wrote:
> The repeated-iteration approach taken in pgaio_closing_fd() isn't the
> prettiest, but it's hard to to imagine that ever being a noticeable.
Yep. I've reviewed the fixup code, and it looks all good.
> This survives a testrun where I use your torture patch and where I force all
> IOs to use ASYNC. Previously that did not get very far. I also did verify
> that, if I allow a small number of FDs, we do not wrongly wait for all IOs.
I, too, see the test diffs gone.
^ permalink raw reply [nested|flat] 153+ messages in thread
* Re: AIO v2.5
2025-03-04 19:00 AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-10 18:23 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-11 19:41 ` Re: AIO v2.5 Noah Misch <noah@leadboat.com>
2025-03-11 23:55 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-12 03:57 ` Re: AIO v2.5 Noah Misch <noah@leadboat.com>
2025-03-12 17:06 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-12 23:23 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-13 10:53 ` Re: AIO v2.5 Antonin Houska <ah@cybertec.at>
2025-03-14 19:43 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-17 20:00 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-18 20:12 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-21 01:58 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
@ 2025-03-25 01:18 ` Andres Freund <andres@anarazel.de>
2025-03-25 04:10 ` Re: AIO v2.5 Thomas Munro <thomas.munro@gmail.com>
2025-03-26 00:19 ` Re: AIO v2.5 Noah Misch <noah@leadboat.com>
2025-03-26 18:31 ` Re: AIO v2.5 Noah Misch <noah@leadboat.com>
2025-03-26 21:20 ` Re: AIO v2.5 Thom Brown <thom@linux.com>
2025-03-27 01:07 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
4 siblings, 5 replies; 153+ messages in thread
From: Andres Freund @ 2025-03-25 01:18 UTC (permalink / raw)
To: Antonin Houska <ah@cybertec.at>; +Cc: Noah Misch <noah@leadboat.com>; pgsql-hackers@postgresql.org, Thomas Munro <thomas.munro@gmail.com>; Heikki Linnakangas <hlinnaka@iki.fi>; Robert Haas <robertmhaas@gmail.com>; Jakub Wartak <jakub.wartak@enterprisedb.com>; Jelte Fennema-Nio <postgres@jeltef.nl>
Hi,
Attached v2.12, with the following changes:
- Pushed the max_files_per_process change
I plan to look at what parts of Jelte's change is worth doing ontop.
Thanks for the review Noah.
- Rebased over Thomas' commit of the remaining read stream changes
Yay!
- Addressed Noah's review comments
- Added another test to test_aio/, to test that changing io_workers while
running works, and that workers are restarted if terminated
Written by Bilal
- Made InvalidateLocalBuffer wait for IO if necessary
As reported / suggested by Noah
- Added tests for dropping tables with ongoing IO
This failed, as Noah predicted, without the InvalidateLocalBuffer() change.
- Added a commit to explicitly hold interrupts in workers after
pgaio_io_reopen()
As suggested by Noah.
- Added a commit to fix a logic error around what gets passed to
ioh->report_return - this lead to temporary buffer validation errors not
being reported
Discovered while extending the tests, as noted in the next point.
I could see a few different "formulations" of this change (e.g. the
report_return stuff could be populated by pgaio_io_call_complete_local()
instead), but I don't think it matters much.
- Add temporary table coverage to test_aio
This required hanged test_aio.c to cope with temporary tables as well.
- io_uring tests don't run anymore when built with EXEC_BACKEND and liburing
enabled
- Split the read stream patch into two
Noah, quite rightly, pointed out that it's not safe to use batching if the
next-block callback may block (or start its own batch). The best idea seems
to be to make users of read stream opt-in to batching. I've done that in a
patch that uses where it seems safe without doing extra work. See also the
commit message.
- Added a commit to add I/O, Asynchronous I/O glossary and acronym entries
- Docs for pg_aios
- Renamed pg_aios.offset to off, to avoid use of a keyword
- Updated the io_uring wait event name while waiting for IOs to complete to
AIO_IO_URING_COMPLETION and updated the description of AIO_IO_COMPLETION to
"Waiting for another process to complete IO."
I think this is a mix of different suggestions by Noah.
TODO:
- There are more tests in test_aio that should be expanded to run for temp
tables as well, not just normal tables
- Add an explicit test for the checksum verification in the completion callback
There is an existing test for testing an invalid page due to page header
verification in test_aio, but not for checksum failures.
I think it's indirectly covered (e.g. in amcheck), but seems better to test
it explicitly.
Wonder if it's worth adding some coverage for when checksums are disabled?
Probably not necessary?
Greetings,
Andres Freund
Attachments:
[text/x-diff] v2.12-0001-aio-Be-more-paranoid-about-interrupts.patch (2.5K, ../../cxi6ssym62v7ywkt367qda3xik5jhsrcycwedyfqacinxooys5@pwarur37bosc/2-v2.12-0001-aio-Be-more-paranoid-about-interrupts.patch)
download | inline diff:
From 326719b771e2a600b8eca864ab3ca180958b2f5e Mon Sep 17 00:00:00 2001
From: Andres Freund <andres@anarazel.de>
Date: Mon, 24 Mar 2025 10:38:38 -0400
Subject: [PATCH v2.12 01/28] aio: Be more paranoid about interrupts
As reported by Noah, it's possible, although practically very unlikely, that
interrupts could be processed in between pgaio_io_reopen() and
pgaio_io_perform_synchronously(). Prevent that by explicitly holding
interrupts.
It also seems good to add an assertion to pgaio_io_before_prep() to ensure
that interrupts are held, as otherwise FDs referenced by the IO could be
closed during interrupt processing. All code in the aio series currently runs
the code with interrupts held, but it seems better to be paranoid.
Reported-by: Noah Misch <noah@leadboat.com>
Discussion: https://postgr.es/m/20250324002939.5c.nmisch@google.com
---
src/backend/storage/aio/aio_io.c | 6 ++++++
src/backend/storage/aio/method_worker.c | 9 +++++++++
2 files changed, 15 insertions(+)
diff --git a/src/backend/storage/aio/aio_io.c b/src/backend/storage/aio/aio_io.c
index 36d2c1f492d..cc6d999a6fb 100644
--- a/src/backend/storage/aio/aio_io.c
+++ b/src/backend/storage/aio/aio_io.c
@@ -159,6 +159,12 @@ pgaio_io_before_prep(PgAioHandle *ioh)
Assert(pgaio_my_backend->handed_out_io == ioh);
Assert(pgaio_io_has_target(ioh));
Assert(ioh->op == PGAIO_OP_INVALID);
+
+ /*
+ * Otherwise the FDs referenced by the IO could be closed due to interrupt
+ * processing.
+ */
+ Assert(!INTERRUPTS_CAN_BE_PROCESSED());
}
/*
diff --git a/src/backend/storage/aio/method_worker.c b/src/backend/storage/aio/method_worker.c
index 2be6bb8972b..4a7853d13fa 100644
--- a/src/backend/storage/aio/method_worker.c
+++ b/src/backend/storage/aio/method_worker.c
@@ -476,6 +476,13 @@ IoWorkerMain(const void *startup_data, size_t startup_data_len)
"worker %d processing IO",
MyIoWorkerId);
+ /*
+ * Prevent interrupts between pgaio_io_reopen() and
+ * pgaio_io_perform_synchronously() that otherwise could lead to
+ * the FD getting closed in that window.
+ */
+ HOLD_INTERRUPTS();
+
/*
* It's very unlikely, but possible, that reopen fails. E.g. due
* to memory allocations failing or file permissions changing or
@@ -502,6 +509,8 @@ IoWorkerMain(const void *startup_data, size_t startup_data_len)
* ensure we don't accidentally fail.
*/
pgaio_io_perform_synchronously(ioh);
+
+ RESUME_INTERRUPTS();
}
else
{
--
2.48.1.76.g4e746b1a31.dirty
[text/x-diff] v2.12-0002-aio-Pass-result-of-local-callbacks-to-report_r.patch (4.1K, ../../cxi6ssym62v7ywkt367qda3xik5jhsrcycwedyfqacinxooys5@pwarur37bosc/3-v2.12-0002-aio-Pass-result-of-local-callbacks-to-report_r.patch)
download | inline diff:
From 04aa1bd244da3e4fb6a756247af66b8ecb61f505 Mon Sep 17 00:00:00 2001
From: Andres Freund <andres@anarazel.de>
Date: Mon, 24 Mar 2025 17:06:54 -0400
Subject: [PATCH v2.12 02/28] aio: Pass result of local callbacks to
->report_return
Otherwise the results of e.g. temp table buffer verification errors will not
reach bufmgr.c. Obviously that's not right.
---
src/include/storage/aio_internal.h | 2 +-
src/backend/storage/aio/aio.c | 22 +++++++++++-----------
src/backend/storage/aio/aio_callback.c | 11 +++++++++--
3 files changed, 21 insertions(+), 14 deletions(-)
diff --git a/src/include/storage/aio_internal.h b/src/include/storage/aio_internal.h
index 108fe61c7b4..d5f64416870 100644
--- a/src/include/storage/aio_internal.h
+++ b/src/include/storage/aio_internal.h
@@ -309,7 +309,7 @@ extern void pgaio_shutdown(int code, Datum arg);
/* aio_callback.c */
extern void pgaio_io_call_stage(PgAioHandle *ioh);
extern void pgaio_io_call_complete_shared(PgAioHandle *ioh);
-extern void pgaio_io_call_complete_local(PgAioHandle *ioh);
+extern PgAioResult pgaio_io_call_complete_local(PgAioHandle *ioh);
/* aio_io.c */
extern void pgaio_io_perform_synchronously(PgAioHandle *ioh);
diff --git a/src/backend/storage/aio/aio.c b/src/backend/storage/aio/aio.c
index 29f57f9cd1c..1fd82842718 100644
--- a/src/backend/storage/aio/aio.c
+++ b/src/backend/storage/aio/aio.c
@@ -626,13 +626,21 @@ pgaio_io_reclaim(PgAioHandle *ioh)
/*
* It's a bit ugly, but right now the easiest place to put the execution
- * of shared completion callbacks is this function, as we need to execute
+ * of local completion callbacks is this function, as we need to execute
* local callbacks just before reclaiming at multiple callsites.
*/
if (ioh->state == PGAIO_HS_COMPLETED_SHARED)
{
- pgaio_io_call_complete_local(ioh);
+ PgAioResult local_result;
+
+ local_result = pgaio_io_call_complete_local(ioh);
pgaio_io_update_state(ioh, PGAIO_HS_COMPLETED_LOCAL);
+
+ if (ioh->report_return)
+ {
+ ioh->report_return->result = local_result;
+ ioh->report_return->target_data = ioh->target_data;
+ }
}
pgaio_debug_io(DEBUG4, ioh,
@@ -642,18 +650,10 @@ pgaio_io_reclaim(PgAioHandle *ioh)
ioh->distilled_result.error_data,
ioh->result);
- /* if the IO has been defined, we might need to do more work */
+ /* if the IO has been defined, it's on the in-flight list, remove */
if (ioh->state != PGAIO_HS_HANDED_OUT)
- {
dclist_delete_from(&pgaio_my_backend->in_flight_ios, &ioh->node);
- if (ioh->report_return)
- {
- ioh->report_return->result = ioh->distilled_result;
- ioh->report_return->target_data = ioh->target_data;
- }
- }
-
if (ioh->resowner)
{
ResourceOwnerForgetAioHandle(ioh->resowner, &ioh->resowner_node);
diff --git a/src/backend/storage/aio/aio_callback.c b/src/backend/storage/aio/aio_callback.c
index 09f03f296f5..d32df1626ba 100644
--- a/src/backend/storage/aio/aio_callback.c
+++ b/src/backend/storage/aio/aio_callback.c
@@ -262,9 +262,12 @@ pgaio_io_call_complete_shared(PgAioHandle *ioh)
* Internal function which invokes ->complete_local for all the registered
* callbacks.
*
+ * Returns ioh->distilled_result after, possibly, being modified by local
+ * callbacks.
+ *
* XXX: It'd be nice to deduplicate with pgaio_io_call_complete_shared().
*/
-void
+PgAioResult
pgaio_io_call_complete_local(PgAioHandle *ioh)
{
PgAioResult result;
@@ -296,7 +299,9 @@ pgaio_io_call_complete_local(PgAioHandle *ioh)
/*
* Note that we don't save the result in ioh->distilled_result, the local
- * callback's result should not ever matter to other waiters.
+ * callback's result should not ever matter to other waiters. However, the
+ * local backend does care, so we return the result as modified by local
+ * callbacks, which then can be passed to ioh->report_return->result.
*/
pgaio_debug_io(DEBUG3, ioh,
"after local completion: distilled result: (status %s, id %u, error_data %d, result %d), raw_result: %d",
@@ -305,4 +310,6 @@ pgaio_io_call_complete_local(PgAioHandle *ioh)
ioh->result);
END_CRIT_SECTION();
+
+ return result;
}
--
2.48.1.76.g4e746b1a31.dirty
[text/x-diff] v2.12-0003-aio-Add-liburing-dependency.patch (13.6K, ../../cxi6ssym62v7ywkt367qda3xik5jhsrcycwedyfqacinxooys5@pwarur37bosc/4-v2.12-0003-aio-Add-liburing-dependency.patch)
download | inline diff:
From 0cf2230ae482bb6d7f6c362235427a3d3d64459a Mon Sep 17 00:00:00 2001
From: Andres Freund <andres@anarazel.de>
Date: Tue, 18 Mar 2025 14:40:05 -0400
Subject: [PATCH v2.12 03/28] aio: Add liburing dependency
Will be used in a subsequent commit, to implement io_method=io_uring. Kept
separate for easier review.
Reviewed-by: Noah Misch <noah@leadboat.com>
Discussion: https://postgr.es/m/uvrtrknj4kdytuboidbhwclo4gxhswwcpgadptsjvjqcluzmah%40brqs62irg4dt
---
meson.build | 14 ++++
meson_options.txt | 3 +
configure.ac | 12 +++
src/makefiles/meson.build | 3 +
src/include/pg_config.h.in | 3 +
src/backend/Makefile | 7 +-
doc/src/sgml/installation.sgml | 34 ++++++++
configure | 139 +++++++++++++++++++++++++++++++++
.cirrus.tasks.yml | 1 +
src/Makefile.global.in | 6 +-
10 files changed, 218 insertions(+), 4 deletions(-)
diff --git a/meson.build b/meson.build
index 7cf518a2765..108e3678071 100644
--- a/meson.build
+++ b/meson.build
@@ -944,6 +944,18 @@ endif
+###############################################################
+# Library: liburing
+###############################################################
+
+liburingopt = get_option('liburing')
+liburing = dependency('liburing', required: liburingopt)
+if liburing.found()
+ cdata.set('USE_LIBURING', 1)
+endif
+
+
+
###############################################################
# Library: libxml
###############################################################
@@ -3164,6 +3176,7 @@ backend_both_deps += [
icu_i18n,
ldap,
libintl,
+ liburing,
libxml,
lz4,
pam,
@@ -3819,6 +3832,7 @@ if meson.version().version_compare('>=0.57')
'icu': icu,
'ldap': ldap,
'libcurl': libcurl,
+ 'liburing': liburing,
'libxml': libxml,
'libxslt': libxslt,
'llvm': llvm,
diff --git a/meson_options.txt b/meson_options.txt
index 702c4517145..dd7126da3a7 100644
--- a/meson_options.txt
+++ b/meson_options.txt
@@ -106,6 +106,9 @@ option('libcurl', type : 'feature', value: 'auto',
option('libedit_preferred', type: 'boolean', value: false,
description: 'Prefer BSD Libedit over GNU Readline')
+option('liburing', type : 'feature', value: 'auto',
+ description: 'io_uring support, for asynchronous I/O')
+
option('libxml', type: 'feature', value: 'auto',
description: 'XML support')
diff --git a/configure.ac b/configure.ac
index b6d02f5ecc7..ecbc2734829 100644
--- a/configure.ac
+++ b/configure.ac
@@ -975,6 +975,18 @@ AC_SUBST(with_readline)
PGAC_ARG_BOOL(with, libedit-preferred, no,
[prefer BSD Libedit over GNU Readline])
+#
+# liburing
+#
+AC_MSG_CHECKING([whether to build with liburing support])
+PGAC_ARG_BOOL(with, liburing, no, [build with io_uring support, for asynchronous I/O],
+ [AC_DEFINE([USE_LIBURING], 1, [Define to build with io_uring support. (--with-liburing)])])
+AC_MSG_RESULT([$with_liburing])
+AC_SUBST(with_liburing)
+
+if test "$with_liburing" = yes; then
+ PKG_CHECK_MODULES(LIBURING, liburing)
+fi
#
# UUID library
diff --git a/src/makefiles/meson.build b/src/makefiles/meson.build
index 60e13d50235..46d8da070e8 100644
--- a/src/makefiles/meson.build
+++ b/src/makefiles/meson.build
@@ -199,6 +199,8 @@ pgxs_empty = [
'PTHREAD_CFLAGS', 'PTHREAD_LIBS',
'ICU_LIBS',
+
+ 'LIBURING_CFLAGS', 'LIBURING_LIBS',
]
if host_system == 'windows' and cc.get_argument_syntax() != 'msvc'
@@ -230,6 +232,7 @@ pgxs_deps = {
'icu': icu,
'ldap': ldap,
'libcurl': libcurl,
+ 'liburing': liburing,
'libxml': libxml,
'libxslt': libxslt,
'llvm': llvm,
diff --git a/src/include/pg_config.h.in b/src/include/pg_config.h.in
index db6454090d2..f2422241133 100644
--- a/src/include/pg_config.h.in
+++ b/src/include/pg_config.h.in
@@ -672,6 +672,9 @@
/* Define to 1 to build with libcurl support. (--with-libcurl) */
#undef USE_LIBCURL
+/* Define to build with io_uring support. (--with-liburing) */
+#undef USE_LIBURING
+
/* Define to 1 to build with XML support. (--with-libxml) */
#undef USE_LIBXML
diff --git a/src/backend/Makefile b/src/backend/Makefile
index 42d4a28e5aa..7344c8c7f5c 100644
--- a/src/backend/Makefile
+++ b/src/backend/Makefile
@@ -43,9 +43,10 @@ OBJS = \
$(top_builddir)/src/common/libpgcommon_srv.a \
$(top_builddir)/src/port/libpgport_srv.a
-# We put libpgport and libpgcommon into OBJS, so remove it from LIBS; also add
-# libldap and ICU
-LIBS := $(filter-out -lpgport -lpgcommon, $(LIBS)) $(LDAP_LIBS_BE) $(ICU_LIBS)
+# We put libpgport and libpgcommon into OBJS, so remove it from LIBS.
+LIBS := $(filter-out -lpgport -lpgcommon, $(LIBS))
+# The backend conditionally needs libraries that most executables don't need.
+LIBS += $(LDAP_LIBS_BE) $(ICU_LIBS) $(LIBURING_LIBS)
# The backend doesn't need everything that's in LIBS, however
LIBS := $(filter-out -lreadline -ledit -ltermcap -lncurses -lcurses, $(LIBS))
diff --git a/doc/src/sgml/installation.sgml b/doc/src/sgml/installation.sgml
index e076cefa3b9..cc28f041330 100644
--- a/doc/src/sgml/installation.sgml
+++ b/doc/src/sgml/installation.sgml
@@ -1156,6 +1156,24 @@ build-postgresql:
</listitem>
</varlistentry>
+ <varlistentry id="configure-option-with-liburing">
+ <term><option>--with-liburing</option></term>
+ <listitem>
+ <para>
+ Build with liburing, enabling io_uring support for asynchronous I/O.
+ </para>
+ <para>
+ To detect the required compiler and linker options, PostgreSQL will
+ query <command>pkg-config</command>.
+ </para>
+ <para>
+ To use a liburing installation that is in an unusual location, you
+ can set <command>pkg-config</command>-related environment
+ variables (see its documentation).
+ </para>
+ </listitem>
+ </varlistentry>
+
<varlistentry id="configure-option-with-libxml">
<term><option>--with-libxml</option></term>
<listitem>
@@ -2611,6 +2629,22 @@ ninja install
</listitem>
</varlistentry>
+ <varlistentry id="configure-with-liburing-meson">
+ <term><option>-Dliburing={ auto | enabled | disabled }</option></term>
+ <listitem>
+ <para>
+ Build with liburing, enabling io_uring support for asynchronous I/O.
+ Defaults to auto.
+ </para>
+
+ <para>
+ To use a liburing installation that is in an unusual location, you
+ can set <command>pkg-config</command>-related environment
+ variables (see its documentation).
+ </para>
+ </listitem>
+ </varlistentry>
+
<varlistentry id="configure-with-libxml-meson">
<term><option>-Dlibxml={ auto | enabled | disabled }</option></term>
<listitem>
diff --git a/configure b/configure
index fac1e9a4e39..c6d762dc999 100755
--- a/configure
+++ b/configure
@@ -712,6 +712,9 @@ LIBCURL_LIBS
LIBCURL_CFLAGS
with_libcurl
with_uuid
+LIBURING_LIBS
+LIBURING_CFLAGS
+with_liburing
with_readline
with_systemd
with_selinux
@@ -865,6 +868,7 @@ with_selinux
with_systemd
with_readline
with_libedit_preferred
+with_liburing
with_uuid
with_ossp_uuid
with_libcurl
@@ -898,6 +902,8 @@ PKG_CONFIG_PATH
PKG_CONFIG_LIBDIR
ICU_CFLAGS
ICU_LIBS
+LIBURING_CFLAGS
+LIBURING_LIBS
LIBCURL_CFLAGS
LIBCURL_LIBS
XML2_CONFIG
@@ -1578,6 +1584,7 @@ Optional Packages:
--without-readline do not use GNU Readline nor BSD Libedit for editing
--with-libedit-preferred
prefer BSD Libedit over GNU Readline
+ --with-liburing build with io_uring support, for asynchronous I/O
--with-uuid=LIB build contrib/uuid-ossp using LIB (bsd,e2fs,ossp)
--with-ossp-uuid obsolete spelling of --with-uuid=ossp
--with-libcurl build with libcurl support
@@ -1614,6 +1621,10 @@ Some influential environment variables:
path overriding pkg-config's built-in search path
ICU_CFLAGS C compiler flags for ICU, overriding pkg-config
ICU_LIBS linker flags for ICU, overriding pkg-config
+ LIBURING_CFLAGS
+ C compiler flags for LIBURING, overriding pkg-config
+ LIBURING_LIBS
+ linker flags for LIBURING, overriding pkg-config
LIBCURL_CFLAGS
C compiler flags for LIBCURL, overriding pkg-config
LIBCURL_LIBS
@@ -8692,6 +8703,134 @@ fi
+#
+# liburing
+#
+{ $as_echo "$as_me:${as_lineno-$LINENO}: checking whether to build with liburing support" >&5
+$as_echo_n "checking whether to build with liburing support... " >&6; }
+
+
+
+# Check whether --with-liburing was given.
+if test "${with_liburing+set}" = set; then :
+ withval=$with_liburing;
+ case $withval in
+ yes)
+
+$as_echo "#define USE_LIBURING 1" >>confdefs.h
+
+ ;;
+ no)
+ :
+ ;;
+ *)
+ as_fn_error $? "no argument expected for --with-liburing option" "$LINENO" 5
+ ;;
+ esac
+
+else
+ with_liburing=no
+
+fi
+
+
+{ $as_echo "$as_me:${as_lineno-$LINENO}: result: $with_liburing" >&5
+$as_echo "$with_liburing" >&6; }
+
+
+if test "$with_liburing" = yes; then
+
+pkg_failed=no
+{ $as_echo "$as_me:${as_lineno-$LINENO}: checking for liburing" >&5
+$as_echo_n "checking for liburing... " >&6; }
+
+if test -n "$LIBURING_CFLAGS"; then
+ pkg_cv_LIBURING_CFLAGS="$LIBURING_CFLAGS"
+ elif test -n "$PKG_CONFIG"; then
+ if test -n "$PKG_CONFIG" && \
+ { { $as_echo "$as_me:${as_lineno-$LINENO}: \$PKG_CONFIG --exists --print-errors \"liburing\""; } >&5
+ ($PKG_CONFIG --exists --print-errors "liburing") 2>&5
+ ac_status=$?
+ $as_echo "$as_me:${as_lineno-$LINENO}: \$? = $ac_status" >&5
+ test $ac_status = 0; }; then
+ pkg_cv_LIBURING_CFLAGS=`$PKG_CONFIG --cflags "liburing" 2>/dev/null`
+ test "x$?" != "x0" && pkg_failed=yes
+else
+ pkg_failed=yes
+fi
+ else
+ pkg_failed=untried
+fi
+if test -n "$LIBURING_LIBS"; then
+ pkg_cv_LIBURING_LIBS="$LIBURING_LIBS"
+ elif test -n "$PKG_CONFIG"; then
+ if test -n "$PKG_CONFIG" && \
+ { { $as_echo "$as_me:${as_lineno-$LINENO}: \$PKG_CONFIG --exists --print-errors \"liburing\""; } >&5
+ ($PKG_CONFIG --exists --print-errors "liburing") 2>&5
+ ac_status=$?
+ $as_echo "$as_me:${as_lineno-$LINENO}: \$? = $ac_status" >&5
+ test $ac_status = 0; }; then
+ pkg_cv_LIBURING_LIBS=`$PKG_CONFIG --libs "liburing" 2>/dev/null`
+ test "x$?" != "x0" && pkg_failed=yes
+else
+ pkg_failed=yes
+fi
+ else
+ pkg_failed=untried
+fi
+
+
+
+if test $pkg_failed = yes; then
+ { $as_echo "$as_me:${as_lineno-$LINENO}: result: no" >&5
+$as_echo "no" >&6; }
+
+if $PKG_CONFIG --atleast-pkgconfig-version 0.20; then
+ _pkg_short_errors_supported=yes
+else
+ _pkg_short_errors_supported=no
+fi
+ if test $_pkg_short_errors_supported = yes; then
+ LIBURING_PKG_ERRORS=`$PKG_CONFIG --short-errors --print-errors --cflags --libs "liburing" 2>&1`
+ else
+ LIBURING_PKG_ERRORS=`$PKG_CONFIG --print-errors --cflags --libs "liburing" 2>&1`
+ fi
+ # Put the nasty error message in config.log where it belongs
+ echo "$LIBURING_PKG_ERRORS" >&5
+
+ as_fn_error $? "Package requirements (liburing) were not met:
+
+$LIBURING_PKG_ERRORS
+
+Consider adjusting the PKG_CONFIG_PATH environment variable if you
+installed software in a non-standard prefix.
+
+Alternatively, you may set the environment variables LIBURING_CFLAGS
+and LIBURING_LIBS to avoid the need to call pkg-config.
+See the pkg-config man page for more details." "$LINENO" 5
+elif test $pkg_failed = untried; then
+ { $as_echo "$as_me:${as_lineno-$LINENO}: result: no" >&5
+$as_echo "no" >&6; }
+ { { $as_echo "$as_me:${as_lineno-$LINENO}: error: in \`$ac_pwd':" >&5
+$as_echo "$as_me: error: in \`$ac_pwd':" >&2;}
+as_fn_error $? "The pkg-config script could not be found or is too old. Make sure it
+is in your PATH or set the PKG_CONFIG environment variable to the full
+path to pkg-config.
+
+Alternatively, you may set the environment variables LIBURING_CFLAGS
+and LIBURING_LIBS to avoid the need to call pkg-config.
+See the pkg-config man page for more details.
+
+To get pkg-config, see <http://pkg-config.freedesktop.org/>.
+See \`config.log' for more details" "$LINENO" 5; }
+else
+ LIBURING_CFLAGS=$pkg_cv_LIBURING_CFLAGS
+ LIBURING_LIBS=$pkg_cv_LIBURING_LIBS
+ { $as_echo "$as_me:${as_lineno-$LINENO}: result: yes" >&5
+$as_echo "yes" >&6; }
+
+fi
+fi
#
# UUID library
diff --git a/.cirrus.tasks.yml b/.cirrus.tasks.yml
index 5849cbb839a..10d277f5659 100644
--- a/.cirrus.tasks.yml
+++ b/.cirrus.tasks.yml
@@ -471,6 +471,7 @@ task:
--enable-cassert --enable-injection-points --enable-debug \
--enable-tap-tests --enable-nls \
--with-segsize-blocks=6 \
+ --with-liburing \
\
${LINUX_CONFIGURE_FEATURES} \
\
diff --git a/src/Makefile.global.in b/src/Makefile.global.in
index 8fe9d61e82a..cce29a37ac5 100644
--- a/src/Makefile.global.in
+++ b/src/Makefile.global.in
@@ -196,6 +196,7 @@ with_gssapi = @with_gssapi@
with_krb_srvnam = @with_krb_srvnam@
with_ldap = @with_ldap@
with_libcurl = @with_libcurl@
+with_liburing = @with_liburing@
with_libxml = @with_libxml@
with_libxslt = @with_libxslt@
with_llvm = @with_llvm@
@@ -222,6 +223,9 @@ krb_srvtab = @krb_srvtab@
ICU_CFLAGS = @ICU_CFLAGS@
ICU_LIBS = @ICU_LIBS@
+LIBURING_CFLAGS = @LIBURING_CFLAGS@
+LIBURING_LIBS = @LIBURING_LIBS@
+
TCLSH = @TCLSH@
TCL_LIBS = @TCL_LIBS@
TCL_LIB_SPEC = @TCL_LIB_SPEC@
@@ -246,7 +250,7 @@ CPP = @CPP@
CPPFLAGS = @CPPFLAGS@
PG_SYSROOT = @PG_SYSROOT@
-override CPPFLAGS := $(ICU_CFLAGS) $(CPPFLAGS)
+override CPPFLAGS := $(ICU_CFLAGS) $(LIBURING_CFLAGS) $(CPPFLAGS)
ifdef PGXS
override CPPFLAGS := -I$(includedir_server) -I$(includedir_internal) $(CPPFLAGS)
--
2.48.1.76.g4e746b1a31.dirty
[text/x-diff] v2.12-0004-aio-Add-io_method-io_uring.patch (22.2K, ../../cxi6ssym62v7ywkt367qda3xik5jhsrcycwedyfqacinxooys5@pwarur37bosc/5-v2.12-0004-aio-Add-io_method-io_uring.patch)
download | inline diff:
From f492f8d0ea3f00aeda25a119b55799c230807f3b Mon Sep 17 00:00:00 2001
From: Andres Freund <andres@anarazel.de>
Date: Tue, 18 Mar 2025 14:40:05 -0400
Subject: [PATCH v2.12 04/28] aio: Add io_method=io_uring
Performing AIO using io_uring can be considerably faster than
io_method=worker, particularly when lots of small IOs are issued, as the
context-switch overhead for worker based AIO becomes more significant.
io_uring, however, is linux specific and requires an additional compile-time
dependency (liburing).
This implementation is fairly simple and there are substantial optimization
opportunities.
The description of the existing AIO_IO_COMPLETION wait event is updated to
make the difference between it and the new AIO_IO_URING_EXECUTION clearer.
Reviewed-by: Noah Misch <noah@leadboat.com>
Reviewed-by: Jakub Wartak <jakub.wartak@enterprisedb.com>
Discussion: https://postgr.es/m/uvrtrknj4kdytuboidbhwclo4gxhswwcpgadptsjvjqcluzmah%40brqs62irg4dt
Discussion: https://postgr.es/m/20210223100344.llw5an2aklengrmn@alap3.anarazel.de
Discussion: https://postgr.es/m/stj36ea6yyhoxtqkhpieia2z4krnam7qyetc57rfezgk4zgapf@gcnactj4z56m
---
src/include/storage/aio.h | 8 +
src/include/storage/aio_internal.h | 3 +
src/include/storage/lwlock.h | 1 +
src/backend/storage/aio/Makefile | 1 +
src/backend/storage/aio/aio.c | 6 +
src/backend/storage/aio/meson.build | 1 +
src/backend/storage/aio/method_io_uring.c | 475 ++++++++++++++++++
src/backend/storage/lmgr/lwlock.c | 1 +
.../utils/activity/wait_event_names.txt | 4 +-
src/backend/utils/misc/postgresql.conf.sample | 3 +-
doc/src/sgml/config.sgml | 8 +
.cirrus.tasks.yml | 3 +
src/tools/pgindent/typedefs.list | 1 +
13 files changed, 513 insertions(+), 2 deletions(-)
create mode 100644 src/backend/storage/aio/method_io_uring.c
diff --git a/src/include/storage/aio.h b/src/include/storage/aio.h
index 7b6b7d20a85..8d84f2b9710 100644
--- a/src/include/storage/aio.h
+++ b/src/include/storage/aio.h
@@ -22,12 +22,20 @@
#include "storage/procnumber.h"
+/* io_uring is incompatible with EXEC_BACKEND */
+#if defined(USE_LIBURING) && !defined(EXEC_BACKEND)
+#define IOMETHOD_IO_URING_ENABLED
+#endif
+
/* Enum for io_method GUC. */
typedef enum IoMethod
{
IOMETHOD_SYNC = 0,
IOMETHOD_WORKER,
+#ifdef IOMETHOD_IO_URING_ENABLED
+ IOMETHOD_IO_URING,
+#endif
} IoMethod;
/* We'll default to worker based execution. */
diff --git a/src/include/storage/aio_internal.h b/src/include/storage/aio_internal.h
index d5f64416870..400c44206dd 100644
--- a/src/include/storage/aio_internal.h
+++ b/src/include/storage/aio_internal.h
@@ -386,6 +386,9 @@ extern PgAioHandle *pgaio_inj_io_get(void);
/* Declarations for the tables of function pointers exposed by each IO method. */
extern PGDLLIMPORT const IoMethodOps pgaio_sync_ops;
extern PGDLLIMPORT const IoMethodOps pgaio_worker_ops;
+#ifdef IOMETHOD_IO_URING_ENABLED
+extern PGDLLIMPORT const IoMethodOps pgaio_uring_ops;
+#endif
extern PGDLLIMPORT const IoMethodOps *pgaio_method_ops;
extern PGDLLIMPORT PgAioCtl *pgaio_ctl;
diff --git a/src/include/storage/lwlock.h b/src/include/storage/lwlock.h
index ffa03189e2d..4df1d25c045 100644
--- a/src/include/storage/lwlock.h
+++ b/src/include/storage/lwlock.h
@@ -218,6 +218,7 @@ typedef enum BuiltinTrancheIds
LWTRANCHE_SUBTRANS_SLRU,
LWTRANCHE_XACT_SLRU,
LWTRANCHE_PARALLEL_VACUUM_DSA,
+ LWTRANCHE_AIO_URING_COMPLETION,
LWTRANCHE_FIRST_USER_DEFINED,
} BuiltinTrancheIds;
diff --git a/src/backend/storage/aio/Makefile b/src/backend/storage/aio/Makefile
index f51c34a37f8..c06c50771e0 100644
--- a/src/backend/storage/aio/Makefile
+++ b/src/backend/storage/aio/Makefile
@@ -14,6 +14,7 @@ OBJS = \
aio_init.o \
aio_io.o \
aio_target.o \
+ method_io_uring.o \
method_sync.o \
method_worker.o \
read_stream.o
diff --git a/src/backend/storage/aio/aio.c b/src/backend/storage/aio/aio.c
index 1fd82842718..e52b01c6bc1 100644
--- a/src/backend/storage/aio/aio.c
+++ b/src/backend/storage/aio/aio.c
@@ -65,6 +65,9 @@ static void pgaio_io_wait(PgAioHandle *ioh, uint64 ref_generation);
const struct config_enum_entry io_method_options[] = {
{"sync", IOMETHOD_SYNC, false},
{"worker", IOMETHOD_WORKER, false},
+#ifdef IOMETHOD_IO_URING_ENABLED
+ {"io_uring", IOMETHOD_IO_URING, false},
+#endif
{NULL, 0, false}
};
@@ -82,6 +85,9 @@ PgAioBackend *pgaio_my_backend;
static const IoMethodOps *const pgaio_method_ops_table[] = {
[IOMETHOD_SYNC] = &pgaio_sync_ops,
[IOMETHOD_WORKER] = &pgaio_worker_ops,
+#ifdef IOMETHOD_IO_URING_ENABLED
+ [IOMETHOD_IO_URING] = &pgaio_uring_ops,
+#endif
};
/* callbacks for the configured io_method, set by assign_io_method */
diff --git a/src/backend/storage/aio/meson.build b/src/backend/storage/aio/meson.build
index 74f94c6e40b..2f0f03d8071 100644
--- a/src/backend/storage/aio/meson.build
+++ b/src/backend/storage/aio/meson.build
@@ -6,6 +6,7 @@ backend_sources += files(
'aio_init.c',
'aio_io.c',
'aio_target.c',
+ 'method_io_uring.c',
'method_sync.c',
'method_worker.c',
'read_stream.c',
diff --git a/src/backend/storage/aio/method_io_uring.c b/src/backend/storage/aio/method_io_uring.c
new file mode 100644
index 00000000000..7bd1594d7c7
--- /dev/null
+++ b/src/backend/storage/aio/method_io_uring.c
@@ -0,0 +1,475 @@
+/*-------------------------------------------------------------------------
+ *
+ * method_io_uring.c
+ * AIO - perform AIO using Linux' io_uring
+ *
+ * For now we create one io_uring instance for each backend. These io_uring
+ * instances have to be created in postmaster, during startup, to allow other
+ * backends to process IO completions, if the issuing backend is currently
+ * busy doing other things. Other backends may not use another backend's
+ * io_uring instance to submit IO, that'd require additional locking that
+ * would likely be harmful for performance.
+ *
+ * We likely will want to introduce a backend-local io_uring instance in the
+ * future, e.g. for FE/BE network IO.
+ *
+ * Portions Copyright (c) 1996-2025, PostgreSQL Global Development Group
+ * Portions Copyright (c) 1994, Regents of the University of California
+ *
+ * IDENTIFICATION
+ * src/backend/storage/aio/method_io_uring.c
+ *
+ *-------------------------------------------------------------------------
+ */
+
+#include "postgres.h"
+
+/* included early, for IOMETHOD_IO_URING_ENABLED */
+#include "storage/aio.h"
+
+#ifdef IOMETHOD_IO_URING_ENABLED
+
+#include <liburing.h>
+
+#include "miscadmin.h"
+#include "storage/aio_internal.h"
+#include "storage/fd.h"
+#include "storage/proc.h"
+#include "storage/shmem.h"
+#include "storage/lwlock.h"
+#include "storage/procnumber.h"
+#include "utils/wait_event.h"
+
+
+/* number of completions processed at once */
+#define PGAIO_MAX_LOCAL_COMPLETED_IO 32
+
+
+/* Entry points for IoMethodOps. */
+static size_t pgaio_uring_shmem_size(void);
+static void pgaio_uring_shmem_init(bool first_time);
+static void pgaio_uring_init_backend(void);
+static int pgaio_uring_submit(uint16 num_staged_ios, PgAioHandle **staged_ios);
+static void pgaio_uring_wait_one(PgAioHandle *ioh, uint64 ref_generation);
+
+/* helper functions */
+static void pgaio_uring_sq_from_io(PgAioHandle *ioh, struct io_uring_sqe *sqe);
+
+
+const IoMethodOps pgaio_uring_ops = {
+ .shmem_size = pgaio_uring_shmem_size,
+ .shmem_init = pgaio_uring_shmem_init,
+ .init_backend = pgaio_uring_init_backend,
+
+ .submit = pgaio_uring_submit,
+ .wait_one = pgaio_uring_wait_one,
+};
+
+/*
+ * Per-backend state when using io_method=io_uring
+ *
+ * Align the whole struct to a cacheline boundary, to prevent false sharing
+ * between completion_lock and prior backend's io_uring_ring.
+ */
+typedef struct pg_attribute_aligned (PG_CACHE_LINE_SIZE)
+PgAioUringContext
+{
+ /*
+ * Multiple backends can process completions for this backend's io_uring
+ * instance (e.g. when the backend issuing IO is busy doing something
+ * else). To make that safe we have to ensure that only a single backend
+ * gets io completions from the io_uring instance at a time.
+ */
+ LWLock completion_lock;
+
+ struct io_uring io_uring_ring;
+} PgAioUringContext;
+
+/* PgAioUringContexts for all backends */
+static PgAioUringContext *pgaio_uring_contexts;
+
+/* the current backend's context */
+static PgAioUringContext *pgaio_my_uring_context;
+
+
+static uint32
+pgaio_uring_procs(void)
+{
+ /*
+ * We can subtract MAX_IO_WORKERS here as io workers are never used at the
+ * same time as io_method=io_uring.
+ */
+ return MaxBackends + NUM_AUXILIARY_PROCS - MAX_IO_WORKERS;
+}
+
+static Size
+pgaio_uring_context_shmem_size(void)
+{
+ return mul_size(pgaio_uring_procs(), sizeof(PgAioUringContext));
+}
+
+static size_t
+pgaio_uring_shmem_size(void)
+{
+ return pgaio_uring_context_shmem_size();
+}
+
+static void
+pgaio_uring_shmem_init(bool first_time)
+{
+ int TotalProcs = MaxBackends + NUM_AUXILIARY_PROCS - MAX_IO_WORKERS;
+ bool found;
+
+ pgaio_uring_contexts = (PgAioUringContext *)
+ ShmemInitStruct("AioUring", pgaio_uring_shmem_size(), &found);
+
+ if (found)
+ return;
+
+ for (int contextno = 0; contextno < TotalProcs; contextno++)
+ {
+ PgAioUringContext *context = &pgaio_uring_contexts[contextno];
+ int ret;
+
+ /*
+ * Right now a high TotalProcs will cause problems in two ways:
+ *
+ * - RLIMIT_NOFILE needs to be big enough to allow all
+ * io_uring_queue_init() calls to succeed.
+ *
+ * - RLIMIT_NOFILE needs to be big enough to still have enough file
+ * descriptors to satisfy set_max_safe_fds() left over. Or, even
+ * better, have max_files_per_process left over FDs.
+ *
+ * We probably should adjust the soft RLIMIT_NOFILE to ensure that.
+ *
+ *
+ * XXX: Newer versions of io_uring support sharing the workers that
+ * execute some asynchronous IOs between io_uring instances. It might
+ * be worth using that - also need to evaluate if that causes
+ * noticeable additional contention?
+ */
+ ret = io_uring_queue_init(io_max_concurrency, &context->io_uring_ring, 0);
+ if (ret < 0)
+ {
+ char *hint = NULL;
+ int err = ERRCODE_INTERNAL_ERROR;
+
+ /* add hints for some failures that errno explains sufficiently */
+ if (-ret == EPERM)
+ {
+ err = ERRCODE_INSUFFICIENT_PRIVILEGE;
+ hint = _("Check if io_uring is disabled via /proc/sys/kernel/io_uring_disabled.");
+ }
+ else if (-ret == EMFILE)
+ {
+ err = ERRCODE_INSUFFICIENT_RESOURCES;
+ hint = psprintf(_("Consider increasing \"ulimit -n\" to at least %d."),
+ TotalProcs + max_files_per_process);
+ }
+ else if (-ret == ENOSYS)
+ {
+ err = ERRCODE_FEATURE_NOT_SUPPORTED;
+ hint = _("Kernel does not support io_uring.");
+ }
+
+ /* update errno to allow %m to work */
+ errno = -ret;
+
+ ereport(ERROR,
+ errcode(err),
+ errmsg("could not setup io_uring queue: %m"),
+ hint != NULL ? errhint("%s", hint) : 0);
+ }
+
+ LWLockInitialize(&context->completion_lock, LWTRANCHE_AIO_URING_COMPLETION);
+ }
+}
+
+static void
+pgaio_uring_init_backend(void)
+{
+ Assert(MyProcNumber < pgaio_uring_procs());
+
+ pgaio_my_uring_context = &pgaio_uring_contexts[MyProcNumber];
+}
+
+static int
+pgaio_uring_submit(uint16 num_staged_ios, PgAioHandle **staged_ios)
+{
+ struct io_uring *uring_instance = &pgaio_my_uring_context->io_uring_ring;
+ int in_flight_before = dclist_count(&pgaio_my_backend->in_flight_ios);
+
+ Assert(num_staged_ios <= PGAIO_SUBMIT_BATCH_SIZE);
+
+ for (int i = 0; i < num_staged_ios; i++)
+ {
+ PgAioHandle *ioh = staged_ios[i];
+ struct io_uring_sqe *sqe;
+
+ sqe = io_uring_get_sqe(uring_instance);
+
+ if (!sqe)
+ elog(ERROR, "io_uring submission queue is unexpectedly full");
+
+ pgaio_io_prepare_submit(ioh);
+ pgaio_uring_sq_from_io(ioh, sqe);
+
+ /*
+ * io_uring executes IO in process context if possible. That's
+ * generally good, as it reduces context switching. When performing a
+ * lot of buffered IO that means that copying between page cache and
+ * userspace memory happens in the foreground, as it can't be
+ * offloaded to DMA hardware as is possible when using direct IO. When
+ * executing a lot of buffered IO this causes io_uring to be slower
+ * than worker mode, as worker mode parallelizes the copying. io_uring
+ * can be told to offload work to worker threads instead.
+ *
+ * If an IO is buffered IO and we already have IOs in flight or
+ * multiple IOs are being submitted, we thus tell io_uring to execute
+ * the IO in the background. We don't do so for the first few IOs
+ * being submitted as executing in this process' context has lower
+ * latency.
+ */
+ if (in_flight_before > 4 && (ioh->flags & PGAIO_HF_BUFFERED))
+ io_uring_sqe_set_flags(sqe, IOSQE_ASYNC);
+
+ in_flight_before++;
+ }
+
+ while (true)
+ {
+ int ret;
+
+ pgstat_report_wait_start(WAIT_EVENT_AIO_IO_URING_SUBMIT);
+ ret = io_uring_submit(uring_instance);
+ pgstat_report_wait_end();
+
+ if (ret == -EINTR)
+ {
+ pgaio_debug(DEBUG3,
+ "aio method uring: submit EINTR, nios: %d",
+ num_staged_ios);
+ }
+ else if (ret < 0)
+ {
+ /*
+ * The io_uring_enter() manpage suggests that the appropriate
+ * reaction to EAGAIN is:
+ *
+ * "The application should wait for some completions and try
+ * again"
+ *
+ * However, it seems unlikely that that would help in our case, as
+ * we apply a low limit to the number of outstanding IOs and thus
+ * also outstanding completions, making it unlikely that we'd get
+ * EAGAIN while the OS is in good working order.
+ *
+ * Additionally, it would be problematic to just wait here, our
+ * caller might hold critical locks. It'd possibly lead to
+ * delaying the crash-restart that seems likely to occur when the
+ * kernel is under such heavy memory pressure.
+ *
+ * Update errno to allow %m to work.
+ */
+ errno = -ret;
+ elog(PANIC, "io_uring submit failed: %m");
+ }
+ else if (ret != num_staged_ios)
+ {
+ /* likely unreachable, but if it is, we would need to re-submit */
+ elog(PANIC, "io_uring submit submitted only %d of %d",
+ ret, num_staged_ios);
+ }
+ else
+ {
+ pgaio_debug(DEBUG4,
+ "aio method uring: submitted %d IOs",
+ num_staged_ios);
+ break;
+ }
+ }
+
+ return num_staged_ios;
+}
+
+static void
+pgaio_uring_drain_locked(PgAioUringContext *context)
+{
+ int ready;
+ int orig_ready;
+
+ Assert(LWLockHeldByMeInMode(&context->completion_lock, LW_EXCLUSIVE));
+
+ /*
+ * Don't drain more events than available right now. Otherwise it's
+ * plausible that one backend could get stuck, for a while, receiving CQEs
+ * without actually processing them.
+ */
+ orig_ready = ready = io_uring_cq_ready(&context->io_uring_ring);
+
+ while (ready > 0)
+ {
+ struct io_uring_cqe *cqes[PGAIO_MAX_LOCAL_COMPLETED_IO];
+ uint32 ncqes;
+
+ START_CRIT_SECTION();
+ ncqes =
+ io_uring_peek_batch_cqe(&context->io_uring_ring,
+ cqes,
+ Min(PGAIO_MAX_LOCAL_COMPLETED_IO, ready));
+ Assert(ncqes <= ready);
+
+ ready -= ncqes;
+
+ for (int i = 0; i < ncqes; i++)
+ {
+ struct io_uring_cqe *cqe = cqes[i];
+ PgAioHandle *ioh;
+
+ ioh = io_uring_cqe_get_data(cqe);
+ io_uring_cqe_seen(&context->io_uring_ring, cqe);
+
+ pgaio_io_process_completion(ioh, cqe->res);
+ }
+
+ END_CRIT_SECTION();
+
+ pgaio_debug(DEBUG3,
+ "drained %d/%d, now expecting %d",
+ ncqes, orig_ready, io_uring_cq_ready(&context->io_uring_ring));
+ }
+}
+
+static void
+pgaio_uring_wait_one(PgAioHandle *ioh, uint64 ref_generation)
+{
+ PgAioHandleState state;
+ ProcNumber owner_procno = ioh->owner_procno;
+ PgAioUringContext *owner_context = &pgaio_uring_contexts[owner_procno];
+ bool expect_cqe;
+ int waited = 0;
+
+ /*
+ * XXX: It would be nice to have a smarter locking scheme, nearly all the
+ * time the backend owning the ring will consume the completions, making
+ * the locking unnecessarily expensive.
+ */
+ LWLockAcquire(&owner_context->completion_lock, LW_EXCLUSIVE);
+
+ while (true)
+ {
+ pgaio_debug_io(DEBUG3, ioh,
+ "wait_one io_gen: %llu, ref_gen: %llu, cycle %d",
+ (long long unsigned) ioh->generation,
+ (long long unsigned) ref_generation,
+ waited);
+
+ if (pgaio_io_was_recycled(ioh, ref_generation, &state) ||
+ state != PGAIO_HS_SUBMITTED)
+ {
+ /* the IO was completed by another backend */
+ break;
+ }
+ else if (io_uring_cq_ready(&owner_context->io_uring_ring))
+ {
+ /* no need to wait in the kernel, io_uring has a completion */
+ expect_cqe = true;
+ }
+ else
+ {
+ int ret;
+ struct io_uring_cqe *cqes;
+
+ /* need to wait in the kernel */
+ pgstat_report_wait_start(WAIT_EVENT_AIO_IO_URING_EXECUTION);
+ ret = io_uring_wait_cqes(&owner_context->io_uring_ring, &cqes, 1, NULL, NULL);
+ pgstat_report_wait_end();
+
+ if (ret == -EINTR)
+ {
+ continue;
+ }
+ else if (ret != 0)
+ {
+ /* see comment after io_uring_submit() */
+ errno = -ret;
+ elog(PANIC, "io_uring wait failed: %m");
+ }
+ else
+ {
+ Assert(cqes != NULL);
+ expect_cqe = true;
+ waited++;
+ }
+ }
+
+ if (expect_cqe)
+ {
+ pgaio_uring_drain_locked(owner_context);
+ }
+ }
+
+ LWLockRelease(&owner_context->completion_lock);
+
+ pgaio_debug(DEBUG3,
+ "wait_one with %d sleeps",
+ waited);
+}
+
+static void
+pgaio_uring_sq_from_io(PgAioHandle *ioh, struct io_uring_sqe *sqe)
+{
+ struct iovec *iov;
+
+ switch (ioh->op)
+ {
+ case PGAIO_OP_READV:
+ iov = &pgaio_ctl->iovecs[ioh->iovec_off];
+ if (ioh->op_data.read.iov_length == 1)
+ {
+ io_uring_prep_read(sqe,
+ ioh->op_data.read.fd,
+ iov->iov_base,
+ iov->iov_len,
+ ioh->op_data.read.offset);
+ }
+ else
+ {
+ io_uring_prep_readv(sqe,
+ ioh->op_data.read.fd,
+ iov,
+ ioh->op_data.read.iov_length,
+ ioh->op_data.read.offset);
+
+ }
+ break;
+
+ case PGAIO_OP_WRITEV:
+ iov = &pgaio_ctl->iovecs[ioh->iovec_off];
+ if (ioh->op_data.write.iov_length == 1)
+ {
+ io_uring_prep_write(sqe,
+ ioh->op_data.write.fd,
+ iov->iov_base,
+ iov->iov_len,
+ ioh->op_data.write.offset);
+ }
+ else
+ {
+ io_uring_prep_writev(sqe,
+ ioh->op_data.write.fd,
+ iov,
+ ioh->op_data.write.iov_length,
+ ioh->op_data.write.offset);
+ }
+ break;
+
+ case PGAIO_OP_INVALID:
+ elog(ERROR, "trying to prepare invalid IO operation for execution");
+ }
+
+ io_uring_sqe_set_data(sqe, ioh);
+}
+
+#endif /* IOMETHOD_IO_URING_ENABLED */
diff --git a/src/backend/storage/lmgr/lwlock.c b/src/backend/storage/lmgr/lwlock.c
index 5702c35bb91..3df29658f18 100644
--- a/src/backend/storage/lmgr/lwlock.c
+++ b/src/backend/storage/lmgr/lwlock.c
@@ -177,6 +177,7 @@ static const char *const BuiltinTrancheNames[] = {
[LWTRANCHE_SUBTRANS_SLRU] = "SubtransSLRU",
[LWTRANCHE_XACT_SLRU] = "XactSLRU",
[LWTRANCHE_PARALLEL_VACUUM_DSA] = "ParallelVacuumDSA",
+ [LWTRANCHE_AIO_URING_COMPLETION] = "AioUringCompletion",
};
StaticAssertDecl(lengthof(BuiltinTrancheNames) ==
diff --git a/src/backend/utils/activity/wait_event_names.txt b/src/backend/utils/activity/wait_event_names.txt
index 9fa12a555e8..4f44648aca8 100644
--- a/src/backend/utils/activity/wait_event_names.txt
+++ b/src/backend/utils/activity/wait_event_names.txt
@@ -192,7 +192,9 @@ ABI_compatibility:
Section: ClassName - WaitEventIO
-AIO_IO_COMPLETION "Waiting for IO completion."
+AIO_IO_COMPLETION "Waiting for another process to complete IO."
+AIO_IO_URING_SUBMIT "Waiting for IO submission via io_uring."
+AIO_IO_URING_EXECUTION "Waiting for IO execution via io_uring."
BASEBACKUP_READ "Waiting for base backup to read from a file."
BASEBACKUP_SYNC "Waiting for data written by a base backup to reach durable storage."
BASEBACKUP_WRITE "Waiting for base backup to write to a file."
diff --git a/src/backend/utils/misc/postgresql.conf.sample b/src/backend/utils/misc/postgresql.conf.sample
index 0b9e3066bde..2246ccb85a7 100644
--- a/src/backend/utils/misc/postgresql.conf.sample
+++ b/src/backend/utils/misc/postgresql.conf.sample
@@ -204,7 +204,8 @@
# (change requires restart)
#io_combine_limit = 128kB # usually 1-128 blocks (depends on OS)
-#io_method = worker # worker, sync (change requires restart)
+#io_method = worker # worker, io_uring, sync
+ # (change requires restart)
#io_max_concurrency = -1 # Max number of IOs that one process
# can execute simultaneously
# -1 sets based on shared_buffers
diff --git a/doc/src/sgml/config.sgml b/doc/src/sgml/config.sgml
index 69fc93dffc4..f86135fbe1d 100644
--- a/doc/src/sgml/config.sgml
+++ b/doc/src/sgml/config.sgml
@@ -2710,6 +2710,14 @@ include_dir 'conf.d'
<literal>worker</literal> (execute asynchronous I/O using worker processes)
</para>
</listitem>
+ <listitem>
+ <para>
+ <literal>io_uring</literal> (execute asynchronous I/O using
+ io_uring, requires a build with
+ <link linkend="configure-option-with-liburing"><option>--with-liburing</option></link> /
+ <link linkend="configure-with-liburing-meson"><option>-Dliburing</option></link>)
+ </para>
+ </listitem>
<listitem>
<para>
<literal>sync</literal> (execute asynchronous-eligible I/O synchronously)
diff --git a/.cirrus.tasks.yml b/.cirrus.tasks.yml
index 10d277f5659..86a1fa9bbdb 100644
--- a/.cirrus.tasks.yml
+++ b/.cirrus.tasks.yml
@@ -493,11 +493,14 @@ task:
# - Uses undefined behaviour and alignment sanitizers, sanitizer failures
# are typically printed in the server log
# - Test both 64bit and 32 bit builds
+ # - uses io_method=io_uring
- name: Linux - Debian Bookworm - Meson
env:
CCACHE_MAXSIZE: "400M" # tests two different builds
SANITIZER_FLAGS: -fsanitize=alignment,undefined
+ PG_TEST_INITDB_EXTRA_OPTS: >-
+ -c io_method=io_uring
configure_script: |
su postgres <<-EOF
diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list
index 3fbf5a4c212..e253f4937a2 100644
--- a/src/tools/pgindent/typedefs.list
+++ b/src/tools/pgindent/typedefs.list
@@ -2151,6 +2151,7 @@ PgAioReturn
PgAioTargetData
PgAioTargetID
PgAioTargetInfo
+PgAioUringContext
PgAioWaitRef
PgArchData
PgBackendGSSStatus
--
2.48.1.76.g4e746b1a31.dirty
[text/x-diff] v2.12-0005-aio-Implement-support-for-reads-in-smgr-md-fd.patch (23.1K, ../../cxi6ssym62v7ywkt367qda3xik5jhsrcycwedyfqacinxooys5@pwarur37bosc/6-v2.12-0005-aio-Implement-support-for-reads-in-smgr-md-fd.patch)
download | inline diff:
From 5e4c86d04b9ec0c45e0162628af182ecfc5bdb52 Mon Sep 17 00:00:00 2001
From: Andres Freund <andres@anarazel.de>
Date: Tue, 18 Mar 2025 14:40:05 -0400
Subject: [PATCH v2.12 05/28] aio: Implement support for reads in smgr/md/fd
This implements the following:
1) An smgr AIO target, for AIO on smgr files. This should be usable not just
for md.c but also other SMGR implementation if we ever get them.
2) readv support in fd.c, which requires a small bit of infrastructure work in
fd.c
3) smgr.c and md.c support for readv
There still is nothing performing AIO, but as of this commit it would be
possible.
As part of this change FileGetRawDesc() actually ensures that the file is
opened - previously it was basically not usable. It's used to reopen a file in
IO workers.
Reviewed-by: Noah Misch <noah@leadboat.com>
Discussion: https://postgr.es/m/uvrtrknj4kdytuboidbhwclo4gxhswwcpgadptsjvjqcluzmah%40brqs62irg4dt
Discussion: https://postgr.es/m/20210223100344.llw5an2aklengrmn@alap3.anarazel.de
Discussion: https://postgr.es/m/stj36ea6yyhoxtqkhpieia2z4krnam7qyetc57rfezgk4zgapf@gcnactj4z56m
---
src/include/storage/aio.h | 6 +-
src/include/storage/aio_types.h | 10 +-
src/include/storage/fd.h | 3 +
src/include/storage/md.h | 7 +
src/include/storage/smgr.h | 14 ++
src/backend/storage/aio/aio_callback.c | 3 +
src/backend/storage/aio/aio_target.c | 2 +
src/backend/storage/file/fd.c | 40 ++++++
src/backend/storage/smgr/md.c | 186 +++++++++++++++++++++++++
src/backend/storage/smgr/smgr.c | 160 +++++++++++++++++++++
10 files changed, 427 insertions(+), 4 deletions(-)
diff --git a/src/include/storage/aio.h b/src/include/storage/aio.h
index 8d84f2b9710..1be7821b29b 100644
--- a/src/include/storage/aio.h
+++ b/src/include/storage/aio.h
@@ -117,9 +117,10 @@ typedef enum PgAioTargetID
{
/* intentionally the zero value, to help catch zeroed memory etc */
PGAIO_TID_INVALID = 0,
+ PGAIO_TID_SMGR,
} PgAioTargetID;
-#define PGAIO_TID_COUNT (PGAIO_TID_INVALID + 1)
+#define PGAIO_TID_COUNT (PGAIO_TID_SMGR + 1)
/*
@@ -191,6 +192,9 @@ struct PgAioTargetInfo
typedef enum PgAioHandleCallbackID
{
PGAIO_HCB_INVALID,
+
+ PGAIO_HCB_MD_READV,
+ PGAIO_HCB_MD_WRITEV,
} PgAioHandleCallbackID;
diff --git a/src/include/storage/aio_types.h b/src/include/storage/aio_types.h
index dddda3a3e2f..debe8163d4e 100644
--- a/src/include/storage/aio_types.h
+++ b/src/include/storage/aio_types.h
@@ -60,11 +60,15 @@ typedef struct PgAioWaitRef
*/
typedef union PgAioTargetData
{
- /* just as an example placeholder for later */
struct
{
- uint32 queue_id;
- } wal;
+ RelFileLocator rlocator; /* physical relation identifier */
+ BlockNumber blockNum; /* blknum relative to begin of reln */
+ BlockNumber nblocks;
+ ForkNumber forkNum:8; /* don't waste 4 byte for four values */
+ bool is_temp:1; /* proc can be inferred by owning AIO */
+ bool skip_fsync:1;
+ } smgr;
} PgAioTargetData;
diff --git a/src/include/storage/fd.h b/src/include/storage/fd.h
index e3067ab6597..b77d8e5e30e 100644
--- a/src/include/storage/fd.h
+++ b/src/include/storage/fd.h
@@ -101,6 +101,8 @@ extern PGDLLIMPORT int max_safe_fds;
* prototypes for functions in fd.c
*/
+struct PgAioHandle;
+
/* Operations on virtual Files --- equivalent to Unix kernel file ops */
extern File PathNameOpenFile(const char *fileName, int fileFlags);
extern File PathNameOpenFilePerm(const char *fileName, int fileFlags, mode_t fileMode);
@@ -109,6 +111,7 @@ extern void FileClose(File file);
extern int FilePrefetch(File file, off_t offset, off_t amount, uint32 wait_event_info);
extern ssize_t FileReadV(File file, const struct iovec *iov, int iovcnt, off_t offset, uint32 wait_event_info);
extern ssize_t FileWriteV(File file, const struct iovec *iov, int iovcnt, off_t offset, uint32 wait_event_info);
+extern int FileStartReadV(struct PgAioHandle *ioh, File file, int iovcnt, off_t offset, uint32 wait_event_info);
extern int FileSync(File file, uint32 wait_event_info);
extern int FileZero(File file, off_t offset, off_t amount, uint32 wait_event_info);
extern int FileFallocate(File file, off_t offset, off_t amount, uint32 wait_event_info);
diff --git a/src/include/storage/md.h b/src/include/storage/md.h
index 05bf537066e..9d7131eff43 100644
--- a/src/include/storage/md.h
+++ b/src/include/storage/md.h
@@ -14,11 +14,14 @@
#ifndef MD_H
#define MD_H
+#include "storage/aio_types.h"
#include "storage/block.h"
#include "storage/relfilelocator.h"
#include "storage/smgr.h"
#include "storage/sync.h"
+extern const PgAioHandleCallbacks aio_md_readv_cb;
+
/* md storage manager functionality */
extern void mdinit(void);
extern void mdopen(SMgrRelation reln);
@@ -36,6 +39,9 @@ extern uint32 mdmaxcombine(SMgrRelation reln, ForkNumber forknum,
BlockNumber blocknum);
extern void mdreadv(SMgrRelation reln, ForkNumber forknum, BlockNumber blocknum,
void **buffers, BlockNumber nblocks);
+extern void mdstartreadv(PgAioHandle *ioh,
+ SMgrRelation reln, ForkNumber forknum, BlockNumber blocknum,
+ void **buffers, BlockNumber nblocks);
extern void mdwritev(SMgrRelation reln, ForkNumber forknum,
BlockNumber blocknum,
const void **buffers, BlockNumber nblocks, bool skipFsync);
@@ -46,6 +52,7 @@ extern void mdtruncate(SMgrRelation reln, ForkNumber forknum,
BlockNumber old_blocks, BlockNumber nblocks);
extern void mdimmedsync(SMgrRelation reln, ForkNumber forknum);
extern void mdregistersync(SMgrRelation reln, ForkNumber forknum);
+extern int mdfd(SMgrRelation reln, ForkNumber forknum, BlockNumber blocknum, uint32 *off);
extern void ForgetDatabaseSyncRequests(Oid dbid);
extern void DropRelationFiles(RelFileLocator *delrels, int ndelrels, bool isRedo);
diff --git a/src/include/storage/smgr.h b/src/include/storage/smgr.h
index 4016b206ad6..856ebcda350 100644
--- a/src/include/storage/smgr.h
+++ b/src/include/storage/smgr.h
@@ -15,6 +15,7 @@
#define SMGR_H
#include "lib/ilist.h"
+#include "storage/aio_types.h"
#include "storage/block.h"
#include "storage/relfilelocator.h"
@@ -73,6 +74,8 @@ typedef SMgrRelationData *SMgrRelation;
#define SmgrIsTemp(smgr) \
RelFileLocatorBackendIsTemp((smgr)->smgr_rlocator)
+extern const PgAioTargetInfo aio_smgr_target_info;
+
extern void smgrinit(void);
extern SMgrRelation smgropen(RelFileLocator rlocator, ProcNumber backend);
extern bool smgrexists(SMgrRelation reln, ForkNumber forknum);
@@ -97,6 +100,10 @@ extern uint32 smgrmaxcombine(SMgrRelation reln, ForkNumber forknum,
extern void smgrreadv(SMgrRelation reln, ForkNumber forknum,
BlockNumber blocknum,
void **buffers, BlockNumber nblocks);
+extern void smgrstartreadv(PgAioHandle *ioh,
+ SMgrRelation reln, ForkNumber forknum,
+ BlockNumber blocknum,
+ void **buffers, BlockNumber nblocks);
extern void smgrwritev(SMgrRelation reln, ForkNumber forknum,
BlockNumber blocknum,
const void **buffers, BlockNumber nblocks,
@@ -127,4 +134,11 @@ smgrwrite(SMgrRelation reln, ForkNumber forknum, BlockNumber blocknum,
smgrwritev(reln, forknum, blocknum, &buffer, 1, skipFsync);
}
+extern void pgaio_io_set_target_smgr(PgAioHandle *ioh,
+ SMgrRelationData *smgr,
+ ForkNumber forknum,
+ BlockNumber blocknum,
+ int nblocks,
+ bool skip_fsync);
+
#endif /* SMGR_H */
diff --git a/src/backend/storage/aio/aio_callback.c b/src/backend/storage/aio/aio_callback.c
index d32df1626ba..54970337bf3 100644
--- a/src/backend/storage/aio/aio_callback.c
+++ b/src/backend/storage/aio/aio_callback.c
@@ -18,6 +18,7 @@
#include "miscadmin.h"
#include "storage/aio.h"
#include "storage/aio_internal.h"
+#include "storage/md.h"
/* just to have something to put into aio_handle_cbs */
@@ -37,6 +38,8 @@ typedef struct PgAioHandleCallbacksEntry
static const PgAioHandleCallbacksEntry aio_handle_cbs[] = {
#define CALLBACK_ENTRY(id, callback) [id] = {.cb = &callback, .name = #callback}
CALLBACK_ENTRY(PGAIO_HCB_INVALID, aio_invalid_cb),
+
+ CALLBACK_ENTRY(PGAIO_HCB_MD_READV, aio_md_readv_cb),
#undef CALLBACK_ENTRY
};
diff --git a/src/backend/storage/aio/aio_target.c b/src/backend/storage/aio/aio_target.c
index b01406a6a52..6133b04f350 100644
--- a/src/backend/storage/aio/aio_target.c
+++ b/src/backend/storage/aio/aio_target.c
@@ -16,6 +16,7 @@
#include "storage/aio.h"
#include "storage/aio_internal.h"
+#include "storage/smgr.h"
/*
@@ -25,6 +26,7 @@ static const PgAioTargetInfo *pgaio_target_info[] = {
[PGAIO_TID_INVALID] = &(PgAioTargetInfo) {
.name = "invalid",
},
+ [PGAIO_TID_SMGR] = &aio_smgr_target_info,
};
diff --git a/src/backend/storage/file/fd.c b/src/backend/storage/file/fd.c
index 0c3a2a756e7..f573aa8e7ca 100644
--- a/src/backend/storage/file/fd.c
+++ b/src/backend/storage/file/fd.c
@@ -94,6 +94,7 @@
#include "miscadmin.h"
#include "pgstat.h"
#include "postmaster/startup.h"
+#include "storage/aio.h"
#include "storage/fd.h"
#include "storage/ipc.h"
#include "utils/guc.h"
@@ -1296,6 +1297,8 @@ LruDelete(File file)
vfdP = &VfdCache[file];
+ pgaio_closing_fd(vfdP->fd);
+
/*
* Close the file. We aren't expecting this to fail; if it does, better
* to leak the FD than to mess up our internal state.
@@ -1989,6 +1992,8 @@ FileClose(File file)
if (!FileIsNotOpen(file))
{
+ pgaio_closing_fd(vfdP->fd);
+
/* close the file */
if (close(vfdP->fd) != 0)
{
@@ -2212,6 +2217,32 @@ retry:
return returnCode;
}
+int
+FileStartReadV(PgAioHandle *ioh, File file,
+ int iovcnt, off_t offset,
+ uint32 wait_event_info)
+{
+ int returnCode;
+ Vfd *vfdP;
+
+ Assert(FileIsValid(file));
+
+ DO_DB(elog(LOG, "FileStartReadV: %d (%s) " INT64_FORMAT " %d",
+ file, VfdCache[file].fileName,
+ (int64) offset,
+ iovcnt));
+
+ returnCode = FileAccess(file);
+ if (returnCode < 0)
+ return returnCode;
+
+ vfdP = &VfdCache[file];
+
+ pgaio_io_prep_readv(ioh, vfdP->fd, iovcnt, offset);
+
+ return 0;
+}
+
ssize_t
FileWriteV(File file, const struct iovec *iov, int iovcnt, off_t offset,
uint32 wait_event_info)
@@ -2500,6 +2531,12 @@ FilePathName(File file)
int
FileGetRawDesc(File file)
{
+ int returnCode;
+
+ returnCode = FileAccess(file);
+ if (returnCode < 0)
+ return returnCode;
+
Assert(FileIsValid(file));
return VfdCache[file].fd;
}
@@ -2780,6 +2817,7 @@ FreeDesc(AllocateDesc *desc)
result = closedir(desc->desc.dir);
break;
case AllocateDescRawFD:
+ pgaio_closing_fd(desc->desc.fd);
result = close(desc->desc.fd);
break;
default:
@@ -2848,6 +2886,8 @@ CloseTransientFile(int fd)
/* Only get here if someone passes us a file not in allocatedDescs */
elog(WARNING, "fd passed to CloseTransientFile was not obtained from OpenTransientFile");
+ pgaio_closing_fd(fd);
+
return close(fd);
}
diff --git a/src/backend/storage/smgr/md.c b/src/backend/storage/smgr/md.c
index f3220f98dc4..2218f17e7a0 100644
--- a/src/backend/storage/smgr/md.c
+++ b/src/backend/storage/smgr/md.c
@@ -31,6 +31,7 @@
#include "miscadmin.h"
#include "pg_trace.h"
#include "pgstat.h"
+#include "storage/aio.h"
#include "storage/bufmgr.h"
#include "storage/fd.h"
#include "storage/md.h"
@@ -152,6 +153,15 @@ static MdfdVec *_mdfd_getseg(SMgrRelation reln, ForkNumber forknum,
static BlockNumber _mdnblocks(SMgrRelation reln, ForkNumber forknum,
MdfdVec *seg);
+static PgAioResult md_readv_complete(PgAioHandle *ioh, PgAioResult prior_result, uint8 cb_data);
+static void md_readv_report(PgAioResult result, const PgAioTargetData *target_data, int elevel);
+
+const PgAioHandleCallbacks aio_md_readv_cb = {
+ .complete_shared = md_readv_complete,
+ .report = md_readv_report,
+};
+
+
static inline int
_mdfd_open_flags(void)
{
@@ -937,6 +947,69 @@ mdreadv(SMgrRelation reln, ForkNumber forknum, BlockNumber blocknum,
}
}
+/*
+ * mdstartreadv() -- Asynchronous version of mdreadv().
+ */
+void
+mdstartreadv(PgAioHandle *ioh,
+ SMgrRelation reln, ForkNumber forknum, BlockNumber blocknum,
+ void **buffers, BlockNumber nblocks)
+{
+ off_t seekpos;
+ MdfdVec *v;
+ BlockNumber nblocks_this_segment;
+ struct iovec *iov;
+ int iovcnt;
+ int ret;
+
+ v = _mdfd_getseg(reln, forknum, blocknum, false,
+ EXTENSION_FAIL | EXTENSION_CREATE_RECOVERY);
+
+ seekpos = (off_t) BLCKSZ * (blocknum % ((BlockNumber) RELSEG_SIZE));
+
+ Assert(seekpos < (off_t) BLCKSZ * RELSEG_SIZE);
+
+ nblocks_this_segment =
+ Min(nblocks,
+ RELSEG_SIZE - (blocknum % ((BlockNumber) RELSEG_SIZE)));
+
+ if (nblocks_this_segment != nblocks)
+ elog(ERROR, "read crossing segment boundary");
+
+ iovcnt = pgaio_io_get_iovec(ioh, &iov);
+
+ Assert(nblocks <= iovcnt);
+
+ iovcnt = buffers_to_iovec(iov, buffers, nblocks_this_segment);
+
+ Assert(iovcnt <= nblocks_this_segment);
+
+ if (!(io_direct_flags & IO_DIRECT_DATA))
+ pgaio_io_set_flag(ioh, PGAIO_HF_BUFFERED);
+
+ pgaio_io_set_target_smgr(ioh,
+ reln,
+ forknum,
+ blocknum,
+ nblocks,
+ false);
+ pgaio_io_register_callbacks(ioh, PGAIO_HCB_MD_READV, 0);
+
+ ret = FileStartReadV(ioh, v->mdfd_vfd, iovcnt, seekpos, WAIT_EVENT_DATA_FILE_READ);
+ if (ret != 0)
+ ereport(ERROR,
+ (errcode_for_file_access(),
+ errmsg("could not start reading blocks %u..%u in file \"%s\": %m",
+ blocknum,
+ blocknum + nblocks_this_segment - 1,
+ FilePathName(v->mdfd_vfd))));
+
+ /*
+ * The error checks corresponding to the post-read checks in mdread() are
+ * in md_readv_complete().
+ */
+}
+
/*
* mdwritev() -- Write the supplied blocks at the appropriate location.
*
@@ -1365,6 +1438,21 @@ mdimmedsync(SMgrRelation reln, ForkNumber forknum)
}
}
+int
+mdfd(SMgrRelation reln, ForkNumber forknum, BlockNumber blocknum, uint32 *off)
+{
+ MdfdVec *v = mdopenfork(reln, forknum, EXTENSION_FAIL);
+
+ v = _mdfd_getseg(reln, forknum, blocknum, false,
+ EXTENSION_FAIL);
+
+ *off = (off_t) BLCKSZ * (blocknum % ((BlockNumber) RELSEG_SIZE));
+
+ Assert(*off < (off_t) BLCKSZ * RELSEG_SIZE);
+
+ return FileGetRawDesc(v->mdfd_vfd);
+}
+
/*
* register_dirty_segment() -- Mark a relation segment as needing fsync
*
@@ -1841,3 +1929,101 @@ mdfiletagmatches(const FileTag *ftag, const FileTag *candidate)
*/
return ftag->rlocator.dbOid == candidate->rlocator.dbOid;
}
+
+/*
+ * AIO completion callback for mdstartreadv().
+ */
+static PgAioResult
+md_readv_complete(PgAioHandle *ioh, PgAioResult prior_result, uint8 cb_data)
+{
+ PgAioTargetData *td = pgaio_io_get_target_data(ioh);
+ PgAioResult result = prior_result;
+
+ if (prior_result.result < 0)
+ {
+ result.status = PGAIO_RS_ERROR;
+ result.id = PGAIO_HCB_MD_READV;
+ /* For "hard" errors, track the error number in error_data */
+ result.error_data = -prior_result.result;
+ result.result = 0;
+
+ pgaio_result_report(result, td, LOG);
+
+ return result;
+ }
+
+ /*
+ * As explained above smgrstartreadv(), the smgr API operates on the level
+ * of blocks, rather than bytes. Convert.
+ */
+ result.result /= BLCKSZ;
+
+ Assert(result.result <= td->smgr.nblocks);
+
+ if (result.result == 0)
+ {
+ /* consider 0 blocks read a failure */
+ result.status = PGAIO_RS_ERROR;
+ result.id = PGAIO_HCB_MD_READV;
+ result.error_data = 0;
+
+ pgaio_result_report(result, td, LOG);
+
+ return result;
+ }
+
+ if (result.status != PGAIO_RS_ERROR &&
+ result.result < td->smgr.nblocks)
+ {
+ /* partial reads should be retried at upper level */
+ result.status = PGAIO_RS_PARTIAL;
+ result.id = PGAIO_HCB_MD_READV;
+ }
+
+ return result;
+}
+
+/*
+ * AIO error reporting callback for mdstartreadv().
+ *
+ * Errors are encoded as follows:
+ * - PgAioResult.error_data != 0 encodes IO that failed with errno != 0
+ * - PgAioResult.error_data == 0 encodes IO that didn't read all data
+ */
+static void
+md_readv_report(PgAioResult result, const PgAioTargetData *td, int elevel)
+{
+ RelPathStr path;
+
+ path = relpathbackend(td->smgr.rlocator,
+ td->smgr.is_temp ? MyProcNumber : INVALID_PROC_NUMBER,
+ td->smgr.forkNum);
+
+ if (result.error_data != 0)
+ {
+ /* for errcode_for_file_access() and %m */
+ errno = result.error_data;
+
+ ereport(elevel,
+ errcode_for_file_access(),
+ errmsg("could not read blocks %u..%u in file \"%s\": %m",
+ td->smgr.blockNum,
+ td->smgr.blockNum + td->smgr.nblocks - 1,
+ path.str));
+ }
+ else
+ {
+ /*
+ * NB: This will typically only be output in debug messages, while
+ * retrying a partial IO.
+ */
+ ereport(elevel,
+ errcode(ERRCODE_DATA_CORRUPTED),
+ errmsg("could not read blocks %u..%u in file \"%s\": read only %zu of %zu bytes",
+ td->smgr.blockNum,
+ td->smgr.blockNum + td->smgr.nblocks - 1,
+ path.str,
+ result.result * (size_t) BLCKSZ,
+ td->smgr.nblocks * (size_t) BLCKSZ));
+ }
+}
diff --git a/src/backend/storage/smgr/smgr.c b/src/backend/storage/smgr/smgr.c
index af74f54b43b..bf8f57b410a 100644
--- a/src/backend/storage/smgr/smgr.c
+++ b/src/backend/storage/smgr/smgr.c
@@ -66,6 +66,7 @@
#include "access/xlogutils.h"
#include "lib/ilist.h"
#include "miscadmin.h"
+#include "storage/aio.h"
#include "storage/bufmgr.h"
#include "storage/ipc.h"
#include "storage/md.h"
@@ -106,6 +107,10 @@ typedef struct f_smgr
void (*smgr_readv) (SMgrRelation reln, ForkNumber forknum,
BlockNumber blocknum,
void **buffers, BlockNumber nblocks);
+ void (*smgr_startreadv) (PgAioHandle *ioh,
+ SMgrRelation reln, ForkNumber forknum,
+ BlockNumber blocknum,
+ void **buffers, BlockNumber nblocks);
void (*smgr_writev) (SMgrRelation reln, ForkNumber forknum,
BlockNumber blocknum,
const void **buffers, BlockNumber nblocks,
@@ -117,6 +122,7 @@ typedef struct f_smgr
BlockNumber old_blocks, BlockNumber nblocks);
void (*smgr_immedsync) (SMgrRelation reln, ForkNumber forknum);
void (*smgr_registersync) (SMgrRelation reln, ForkNumber forknum);
+ int (*smgr_fd) (SMgrRelation reln, ForkNumber forknum, BlockNumber blocknum, uint32 *off);
} f_smgr;
static const f_smgr smgrsw[] = {
@@ -134,12 +140,14 @@ static const f_smgr smgrsw[] = {
.smgr_prefetch = mdprefetch,
.smgr_maxcombine = mdmaxcombine,
.smgr_readv = mdreadv,
+ .smgr_startreadv = mdstartreadv,
.smgr_writev = mdwritev,
.smgr_writeback = mdwriteback,
.smgr_nblocks = mdnblocks,
.smgr_truncate = mdtruncate,
.smgr_immedsync = mdimmedsync,
.smgr_registersync = mdregistersync,
+ .smgr_fd = mdfd,
}
};
@@ -157,6 +165,16 @@ static dlist_head unpinned_relns;
static void smgrshutdown(int code, Datum arg);
static void smgrdestroy(SMgrRelation reln);
+static void smgr_aio_reopen(PgAioHandle *ioh);
+static char *smgr_aio_describe_identity(const PgAioTargetData *sd);
+
+
+const PgAioTargetInfo aio_smgr_target_info = {
+ .name = "smgr",
+ .reopen = smgr_aio_reopen,
+ .describe_identity = smgr_aio_describe_identity,
+};
+
/*
* smgrinit(), smgrshutdown() -- Initialize or shut down storage
@@ -709,6 +727,30 @@ smgrreadv(SMgrRelation reln, ForkNumber forknum, BlockNumber blocknum,
RESUME_INTERRUPTS();
}
+/*
+ * smgrstartreadv() -- asynchronous version of smgrreadv()
+ *
+ * This starts an asynchronous readv IO using the IO handle `ioh`. Other than
+ * `ioh` all parameters are the same as smgrreadv().
+ *
+ * Completion callbacks above smgr will be passed the result as the number of
+ * successfully read blocks if the read [partially] succeeds (Buffers for
+ * blocks not successfully read might bear unspecified modifications, up to
+ * the full nblocks). This maintains the abstraction that smgr operates on the
+ * level of blocks, rather than bytes.
+ */
+void
+smgrstartreadv(PgAioHandle *ioh,
+ SMgrRelation reln, ForkNumber forknum, BlockNumber blocknum,
+ void **buffers, BlockNumber nblocks)
+{
+ HOLD_INTERRUPTS();
+ smgrsw[reln->smgr_which].smgr_startreadv(ioh,
+ reln, forknum, blocknum, buffers,
+ nblocks);
+ RESUME_INTERRUPTS();
+}
+
/*
* smgrwritev() -- Write the supplied buffers out.
*
@@ -917,6 +959,29 @@ smgrimmedsync(SMgrRelation reln, ForkNumber forknum)
RESUME_INTERRUPTS();
}
+/*
+ * Return fd for the specified block number and update *off to the appropriate
+ * position.
+ *
+ * This is only to be used for when AIO needs to perform the IO in a different
+ * process than where it was issued (e.g. in an IO worker).
+ */
+static int
+smgrfd(SMgrRelation reln, ForkNumber forknum, BlockNumber blocknum, uint32 *off)
+{
+ int fd;
+
+ /*
+ * The caller needs to prevent interrupts from being processed, otherwise
+ * the FD could be closed prematurely.
+ */
+ Assert(!INTERRUPTS_CAN_BE_PROCESSED());
+
+ fd = smgrsw[reln->smgr_which].smgr_fd(reln, forknum, blocknum, off);
+
+ return fd;
+}
+
/*
* AtEOXact_SMgr
*
@@ -945,3 +1010,98 @@ ProcessBarrierSmgrRelease(void)
smgrreleaseall();
return true;
}
+
+/*
+ * Set target of the IO handle to be smgr and initialize all the relevant
+ * pieces of data.
+ */
+void
+pgaio_io_set_target_smgr(PgAioHandle *ioh,
+ SMgrRelationData *smgr,
+ ForkNumber forknum,
+ BlockNumber blocknum,
+ int nblocks,
+ bool skip_fsync)
+{
+ PgAioTargetData *sd = pgaio_io_get_target_data(ioh);
+
+ pgaio_io_set_target(ioh, PGAIO_TID_SMGR);
+
+ /* backend is implied via IO owner */
+ sd->smgr.rlocator = smgr->smgr_rlocator.locator;
+ sd->smgr.forkNum = forknum;
+ sd->smgr.blockNum = blocknum;
+ sd->smgr.nblocks = nblocks;
+ sd->smgr.is_temp = SmgrIsTemp(smgr);
+ /* Temp relations should never be fsync'd */
+ sd->smgr.skip_fsync = skip_fsync && !SmgrIsTemp(smgr);
+}
+
+/*
+ * Callback for smgr AIO targets, to reopen the file (e.g. because the IO is
+ * executed in a worker).
+ */
+static void
+smgr_aio_reopen(PgAioHandle *ioh)
+{
+ PgAioTargetData *sd = pgaio_io_get_target_data(ioh);
+ PgAioOpData *od = pgaio_io_get_op_data(ioh);
+ SMgrRelation reln;
+ ProcNumber procno;
+ uint32 off;
+
+ /*
+ * The caller needs to prevent interrupts from being processed, otherwise
+ * the FD could be closed again before we get to executing the IO.
+ */
+ Assert(!INTERRUPTS_CAN_BE_PROCESSED());
+
+ if (sd->smgr.is_temp)
+ procno = pgaio_io_get_owner(ioh);
+ else
+ procno = INVALID_PROC_NUMBER;
+
+ reln = smgropen(sd->smgr.rlocator, procno);
+ switch (pgaio_io_get_op(ioh))
+ {
+ case PGAIO_OP_INVALID:
+ pg_unreachable();
+ break;
+ case PGAIO_OP_READV:
+ od->read.fd = smgrfd(reln, sd->smgr.forkNum, sd->smgr.blockNum, &off);
+ Assert(off == od->read.offset);
+ break;
+ case PGAIO_OP_WRITEV:
+ od->write.fd = smgrfd(reln, sd->smgr.forkNum, sd->smgr.blockNum, &off);
+ Assert(off == od->write.offset);
+ break;
+ }
+}
+
+/*
+ * Callback for smgr AIO targets, to describe the target of the IO.
+ */
+static char *
+smgr_aio_describe_identity(const PgAioTargetData *sd)
+{
+ RelPathStr path;
+ char *desc;
+
+ path = relpathbackend(sd->smgr.rlocator,
+ sd->smgr.is_temp ? MyProcNumber : INVALID_PROC_NUMBER,
+ sd->smgr.forkNum);
+
+ if (sd->smgr.nblocks == 0)
+ desc = psprintf(_("file \"%s\""), path.str);
+ else if (sd->smgr.nblocks == 1)
+ desc = psprintf(_("block %u in file \"%s\""),
+ sd->smgr.blockNum,
+ path.str);
+ else
+ desc = psprintf(_("blocks %u..%u in file \"%s\""),
+ sd->smgr.blockNum,
+ sd->smgr.blockNum + sd->smgr.nblocks - 1,
+ path.str);
+
+ return desc;
+}
--
2.48.1.76.g4e746b1a31.dirty
[text/x-diff] v2.12-0006-aio-Add-README.md-explaining-higher-level-desi.patch (19.0K, ../../cxi6ssym62v7ywkt367qda3xik5jhsrcycwedyfqacinxooys5@pwarur37bosc/7-v2.12-0006-aio-Add-README.md-explaining-higher-level-desi.patch)
download | inline diff:
From 04c07d2dddcc771c76f1104eede1888d5e2d209c Mon Sep 17 00:00:00 2001
From: Andres Freund <andres@anarazel.de>
Date: Tue, 18 Mar 2025 14:40:06 -0400
Subject: [PATCH v2.12 06/28] aio: Add README.md explaining higher level design
Reviewed-by: Noah Misch <noah@leadboat.com>
Reviewed-by: Heikki Linnakangas <hlinnaka@iki.fi>
Discussion: https://postgr.es/m/uvrtrknj4kdytuboidbhwclo4gxhswwcpgadptsjvjqcluzmah%40brqs62irg4dt
Discussion: https://postgr.es/m/20210223100344.llw5an2aklengrmn@alap3.anarazel.de
Discussion: https://postgr.es/m/stj36ea6yyhoxtqkhpieia2z4krnam7qyetc57rfezgk4zgapf@gcnactj4z56m
---
src/backend/storage/aio/README.md | 423 ++++++++++++++++++++++++++++++
src/backend/storage/aio/aio.c | 2 +
2 files changed, 425 insertions(+)
create mode 100644 src/backend/storage/aio/README.md
diff --git a/src/backend/storage/aio/README.md b/src/backend/storage/aio/README.md
new file mode 100644
index 00000000000..24eec5776cd
--- /dev/null
+++ b/src/backend/storage/aio/README.md
@@ -0,0 +1,423 @@
+# Asynchronous & Direct IO
+
+## Motivation
+
+### Why Asynchronous IO
+
+Until the introduction of asynchronous IO Postgres relied on the operating
+system to hide the cost of synchronous IO from Postgres. While this worked
+surprisingly well in a lot of workloads, it does not do as good a job on
+prefetching and controlled writeback as we would like.
+
+There are important expensive operations like `fdatasync()` where the operating
+system cannot hide the storage latency. This is particularly important for WAL
+writes, where the ability to asynchronously issue `fdatasync()` or O_DSYNC
+writes can yield significantly higher throughput.
+
+
+### Why Direct / unbuffered IO
+
+The main reasons to want to use Direct IO are:
+
+- Lower CPU usage / higher throughput. Particularly on modern storage buffered
+ writes are bottlenecked by the operating system having to copy data from the
+ kernel's page cache to postgres buffer pool using the CPU. Whereas direct IO
+ can often move the data directly between the storage devices and postgres'
+ buffer cache, using DMA. While that transfer is ongoing, the CPU is free to
+ perform other work.
+- Reduced latency - Direct IO can have substantially lower latency than
+ buffered IO, which can be impactful for OLTP workloads bottlenecked by WAL
+ write latency.
+- Avoiding double buffering between operating system cache and postgres'
+ shared_buffers.
+- Better control over the timing and pace of dirty data writeback.
+
+
+The main reasons *not* to use Direct IO are:
+
+- Without AIO, Direct IO is unusably slow for most purposes.
+- Even with AIO, many parts of postgres need to be modified to perform
+ explicit prefetching.
+- In situations where shared_buffers cannot be set appropriately large,
+ e.g. because there are many different postgres instances hosted on shared
+ hardware, performance will often be worse than when using buffered IO.
+
+
+## AIO Usage Example
+
+In many cases code that can benefit from AIO does not directly have to
+interact with the AIO interface, but can use AIO via higher-level
+abstractions. See [Helpers](#helpers).
+
+In this example, a buffer will be read into shared buffers.
+
+```C
+/*
+ * Result of the operation, only to be accessed in this backend.
+ */
+PgAioReturn ioret;
+
+/*
+ * Acquire an AIO Handle, ioret will get result upon completion.
+ *
+ * Note that ioret needs to stay alive until the IO completes or
+ * CurrentResourceOwner is released (i.e. an error is thrown).
+ */
+PgAioHandle *ioh = pgaio_io_acquire(CurrentResourceOwner, &ioret);
+
+/*
+ * Reference that can be used to wait for the IO we initiate below. This
+ * reference can reside in local or shared memory and waited upon by any
+ * process. An arbitrary number of references can be made for each IO.
+ */
+PgAioWaitRef iow;
+
+pgaio_io_get_wref(ioh, &iow);
+
+/*
+ * Arrange for shared buffer completion callbacks to be called upon completion
+ * of the IO. This callback will update the buffer descriptors associated with
+ * the AioHandle, which e.g. allows other backends to access the buffer.
+ *
+ * A callback can be passed a small bit of data, e.g. to indicate whether to
+ * zero a buffer if it is invalid.
+ *
+ * Multiple completion callbacks can be registered for each handle.
+ */
+pgaio_io_register_callbacks(ioh, PGAIO_HCB_SHARED_BUFFER_READV, 0);
+
+/*
+ * The completion callback needs to know which buffers to update when the IO
+ * completes. As the AIO subsystem does not know about buffers, we have to
+ * associate this information with the AioHandle, for use by the completion
+ * callback registered above.
+ *
+ * In this example we're reading only a single buffer, hence the 1.
+ */
+pgaio_io_set_handle_data_32(ioh, (uint32 *) buffer, 1);
+
+/*
+ * Pass the AIO handle to lower-level function. When operating on the level of
+ * buffers, we don't know how exactly the IO is performed, that is the
+ * responsibility of the storage manager implementation.
+ *
+ * E.g. md.c needs to translate block numbers into offsets in segments.
+ *
+ * Once the IO handle has been handed off to smgstartreadv(), it may not
+ * further be used, as the IO may immediately get executed below
+ * smgrstartreadv() and the handle reused for another IO.
+ *
+ * To issue multiple IOs in an efficient way, a caller can call
+ * pgaio_enter_batchmode() before starting multiple IOs, and end that batch
+ * with pgaio_exit_batchmode(). Note that one needs to be careful while there
+ * may be unsubmitted IOs, as another backend may need to wait for one of the
+ * unsubmitted IOs. If this backend then had to wait for the other backend,
+ * it'd end in an undetected deadlock. See pgaio_enter_batchmode() for more
+ * details.
+ *
+ * Note that even while in batchmode an IO might get submitted immediately,
+ * e.g. due to reaching a limit on the number of unsubmitted IOs, and even
+ * complete before smgrstartreadv() returns.
+ */
+smgrstartreadv(ioh, operation->smgr, forknum, blkno,
+ BufferGetBlock(buffer), 1);
+
+/*
+ * To benefit from AIO, it is beneficial to perform other work, including
+ * submitting other IOs, before waiting for the IO to complete. Otherwise
+ * we could just have used synchronous, blocking IO.
+ */
+perform_other_work();
+
+/*
+ * We did some other work and now need the IO operation to have completed to
+ * continue.
+ */
+pgaio_wref_wait(&iow);
+
+/*
+ * At this point the IO has completed. We do not yet know whether it succeeded
+ * or failed, however. The buffer's state has been updated, which allows other
+ * backends to use the buffer (if the IO succeeded), or retry the IO (if it
+ * failed).
+ *
+ * Note that in case the IO has failed, a LOG message may have been emitted,
+ * but no ERROR has been raised. This is crucial, as another backend waiting
+ * for this IO should not see an ERROR.
+ *
+ * To check whether the operation succeeded, and to raise an ERROR, or if more
+ * appropriate LOG, the PgAioReturn we passed to pgaio_io_acquire() is used.
+ */
+if (ioret.result.status == PGAIO_RS_ERROR)
+ pgaio_result_report(ioret.result, &ioret.target_data, ERROR);
+
+/*
+ * Besides having succeeded completely, the IO could also have partially
+ * completed. If we e.g. tried to read many blocks at once, the read might have
+ * only succeeded for the first few blocks.
+ *
+ * If the IO partially succeeded and this backend needs all blocks to have
+ * completed, this backend needs to reissue the IO for the remaining buffers.
+ * The AIO subsystem cannot handle this retry transparently.
+ *
+ * As this example is already long, and we only read a single block, we'll just
+ * error out if there's a partial read.
+ */
+if (ioret.result.status == PGAIO_RS_PARTIAL)
+ pgaio_result_report(ioret.result, &ioret.target_data, ERROR);
+
+/*
+ * The IO succeeded, so we can use the buffer now.
+ */
+```
+
+
+## Design Criteria & Motivation
+
+### Deadlock and Starvation Dangers due to AIO
+
+Using AIO in a naive way can easily lead to deadlocks in an environment where
+the source/target of AIO are shared resources, like pages in postgres'
+shared_buffers.
+
+Consider one backend performing readahead on a table, initiating IO for a
+number of buffers ahead of the current "scan position". If that backend then
+performs some operation that blocks, or even just is slow, the IO completion
+for the asynchronously initiated read may not be processed.
+
+This AIO implementation solves this problem by requiring that AIO methods
+either allow AIO completions to be processed by any backend in the system
+(e.g. io_uring), or to guarantee that AIO processing will happen even when the
+issuing backend is blocked (e.g. worker mode, which offloads completion
+processing to the AIO workers).
+
+
+### IO can be started in critical sections
+
+Using AIO for WAL writes can reduce the overhead of WAL logging substantially:
+
+- AIO allows to start WAL writes eagerly, so they complete before needing to
+ wait
+- AIO allows to have multiple WAL flushes in progress at the same time
+- AIO makes it more realistic to use O\_DIRECT + O\_DSYNC, which can reduce
+ the number of roundtrips to storage on some OSs and storage HW (buffered IO
+ and direct IO without O_DSYNC needs to issue a write and after the write's
+ completion a cache flush, whereas O\_DIRECT + O\_DSYNC can use a single
+ Force Unit Access (FUA) write).
+
+The need to be able to execute IO in critical sections has substantial design
+implication on the AIO subsystem. Mainly because completing IOs (see prior
+section) needs to be possible within a critical section, even if the
+to-be-completed IO itself was not issued in a critical section. Consider
+e.g. the case of a backend first starting a number of writes from shared
+buffers and then starting to flush the WAL. Because only a limited amount of
+IO can be in-progress at the same time, initiating IO for flushing the WAL may
+require to first complete IO that was started earlier.
+
+
+### State for AIO needs to live in shared memory
+
+Because postgres uses a process model and because AIOs need to be
+complete-able by any backend much of the state of the AIO subsystem needs to
+live in shared memory.
+
+In an `EXEC_BACKEND` build, a backend's executable code and other process
+local state is not necessarily mapped to the same addresses in each process
+due to ASLR. This means that the shared memory cannot contain pointers to
+callbacks.
+
+
+## Design of the AIO Subsystem
+
+
+### AIO Methods
+
+To achieve portability and performance, multiple methods of performing AIO are
+implemented and others are likely worth adding in the future.
+
+
+#### Synchronous Mode
+
+`io_method=sync` does not actually perform AIO but allows to use the AIO API
+while performing synchronous IO. This can be useful for debugging. The code
+for the synchronous mode is also used as a fallback by e.g. the [worker
+mode](#worker) uses it to execute IO that cannot be executed by workers.
+
+
+#### Worker
+
+`io_method=worker` is available on every platform postgres runs on, and
+implements asynchronous IO - from the view of the issuing process - by
+dispatching the IO to one of several worker processes performing the IO in a
+synchronous manner.
+
+
+#### io_uring
+
+`io_method=io_uring` is available on Linux 5.1+. In contrast to worker mode it
+dispatches all IO from within the process, lowering context switch rate /
+latency.
+
+
+### AIO Handles
+
+The central API piece for postgres' AIO abstraction are AIO handles. To
+execute an IO one first has to acquire an IO handle (`pgaio_io_acquire()`) and
+then "define" it, i.e. associate an IO operation with the handle.
+
+Often AIO handles are acquired on a higher level and then passed to a lower
+level to be fully defined. E.g., for IO to/from shared buffers, bufmgr.c
+routines acquire the handle, which is then passed through smgr.c, md.c to be
+finally fully defined in fd.c.
+
+The functions used at the lowest level to define the operation are
+`pgaio_io_prep_*()`.
+
+Because acquisition of an IO handle
+[must always succeed](#io-can-be-started-in-critical-sections)
+and the number of AIO Handles
+[has to be limited](#state-for-aio-needs-to-live-in-shared-memory)
+AIO handles can be reused as soon as they have completed. Obviously code needs
+to be able to react to IO completion. State can be updated using
+[AIO Completion callbacks](#aio-callbacks)
+and the issuing backend can provide a backend local variable to receive the
+result of the IO, as described in
+[AIO Result](#aio-results).
+An IO can be waited for, by both the issuing and any other backend, using
+[AIO References](#aio-wait-references).
+
+
+Because an AIO Handle is not executable just after calling
+`pgaio_io_acquire()` and because `pgaio_io_acquire()` needs to always succeed
+(absent a PANIC), only a single AIO Handle may be acquired (i.e. returned by
+`pgaio_io_acquire()`) without causing the IO to have been defined (by,
+potentially indirectly, causing `pgaio_io_prep_*()` to have been
+called). Otherwise a backend could trivially self-deadlock by using up all AIO
+Handles without the ability to wait for some of the IOs to complete.
+
+If it turns out that an AIO Handle is not needed, e.g., because the handle was
+acquired before holding a contended lock, it can be released without being
+defined using `pgaio_io_release()`.
+
+
+### AIO Callbacks
+
+Commonly several layers need to react to completion of an IO. E.g. for a read
+md.c needs to check if the IO outright failed or was shorter than needed,
+bufmgr.c needs to verify the page looks valid and bufmgr.c needs to update the
+BufferDesc to update the buffer's state.
+
+The fact that several layers / subsystems need to react to IO completion poses
+a few challenges:
+
+- Upper layers should not need to know details of lower layers. E.g. bufmgr.c
+ should not assume the IO will pass through md.c. Therefore upper levels
+ cannot know what lower layers would consider an error.
+
+- Lower layers should not need to know about upper layers. E.g. smgr APIs are
+ used going through shared buffers but are also used bypassing shared
+ buffers. This means that e.g. md.c is not in a position to validate
+ checksums.
+
+- Having code in the AIO subsystem for every possible combination of layers
+ would lead to a lot of duplication.
+
+The "solution" to this is the ability to associate multiple completion
+callbacks with a handle. E.g. bufmgr.c can have a callback to update the
+BufferDesc state and to verify the page and md.c can have another callback to
+check if the IO operation was successful.
+
+As [mentioned](#state-for-aio-needs-to-live-in-shared-memory), shared memory
+currently cannot contain function pointers. Because of that completion
+callbacks are not directly identified by function pointers but by IDs
+(`PgAioHandleCallbackID`). A substantial added benefit is that that
+allows callbacks to be identified by much smaller amount of memory (a single
+byte currently).
+
+In addition to completion, AIO callbacks also are called to "stage" an
+IO. This is, e.g., used to increase buffer reference counts to account for the
+AIO subsystem referencing the buffer, which is required to handle the case
+where the issuing backend errors out and releases its own pins while the IO is
+still ongoing.
+
+As [explained earlier](#io-can-be-started-in-critical-sections) IO completions
+need to be safe to execute in critical sections. To allow the backend that
+issued the IO to error out in case of failure [AIO Result](#aio-results) can
+be used.
+
+
+### AIO Targets
+
+In addition to the completion callbacks describe above, each AIO Handle has
+exactly one "target". Each target has some space inside an AIO Handle with
+information specific to the target and can provide callbacks to allow to
+reopen the underlying file (required for worker mode) and to describe the IO
+operation (used for debug logging and error messages).
+
+I.e., if two different uses of AIO can describe the identity of the file being
+operated on the same way, it likely makes sense to use the same
+target. E.g. different smgr implementations can describe IO with
+RelFileLocator, ForkNumber and BlockNumber and can thus share a target. In
+contrast, IO for a WAL file would be described with TimeLineID and XLogRecPtr
+and it would not make sense to use the same target for smgr and WAL.
+
+
+### AIO Wait References
+
+As [described above](#aio-handles), AIO Handles can be reused immediately
+after completion and therefore cannot be used to wait for completion of the
+IO. Waiting is enabled using AIO wait references, which do not just identify
+an AIO Handle but also include the handles "generation".
+
+A reference to an AIO Handle can be acquired using `pgaio_io_get_wref()` and
+then waited upon using `pgaio_wref_wait()`.
+
+
+### AIO Results
+
+As AIO completion callbacks
+[are executed in critical sections](#io-can-be-started-in-critical-sections)
+and [may be executed by any backend](#deadlock-and-starvation-dangers-due-to-aio)
+completion callbacks cannot be used to, e.g., make the query that triggered an
+IO ERROR out.
+
+To allow to react to failing IOs the issuing backend can pass a pointer to a
+`PgAioReturn` in backend local memory. Before an AIO Handle is reused the
+`PgAioReturn` is filled with information about the IO. This includes
+information about whether the IO was successful (as a value of
+`PgAioResultStatus`) and enough information to raise an error in case of a
+failure (via `pgaio_result_report()`, with the error details encoded in
+`PgAioResult`).
+
+
+### AIO Errors
+
+It would be very convenient to have shared completion callbacks encode the
+details of errors as an `ErrorData` that could be raised at a later
+time. Unfortunately doing so would require allocating memory. While elog.c can
+guarantee (well, kinda) that logging a message will not run out of memory,
+that only works because a very limited number of messages are in the process
+of being logged. With AIO a large number of concurrently issued AIOs might
+fail.
+
+To avoid the need for preallocating a potentially large amount of memory (in
+shared memory no less!), completion callbacks instead have to encode errors in
+a more compact format that can be converted into an error message.
+
+
+## Helpers
+
+Using the low-level AIO API introduces too much complexity to do so all over
+the tree. Most uses of AIO should be done via reusable, higher-level,
+helpers.
+
+
+### Read Stream
+
+A common and very beneficial use of AIO are reads where a substantial number
+of to-be-read locations are known ahead of time. E.g., for a sequential scan
+the set of blocks that need to be read can be determined solely by knowing the
+current position and checking the buffer mapping table.
+
+The [Read Stream](../../../include/storage/read_stream.h) interface makes it
+comparatively easy to use AIO for such use cases.
diff --git a/src/backend/storage/aio/aio.c b/src/backend/storage/aio/aio.c
index e52b01c6bc1..0d52f72dfdc 100644
--- a/src/backend/storage/aio/aio.c
+++ b/src/backend/storage/aio/aio.c
@@ -24,6 +24,8 @@
*
* - read_stream.c - helper for reading buffered relation data
*
+ * - README.md - higher-level overview over AIO
+ *
*
* Portions Copyright (c) 1996-2025, PostgreSQL Global Development Group
* Portions Copyright (c) 1994, Regents of the University of California
--
2.48.1.76.g4e746b1a31.dirty
[text/x-diff] v2.12-0007-localbuf-Track-pincount-in-BufferDesc-as-well.patch (4.6K, ../../cxi6ssym62v7ywkt367qda3xik5jhsrcycwedyfqacinxooys5@pwarur37bosc/8-v2.12-0007-localbuf-Track-pincount-in-BufferDesc-as-well.patch)
download | inline diff:
From 2a5998a947d3cb1d1a027de26ffc7225b4e1bcc5 Mon Sep 17 00:00:00 2001
From: Andres Freund <andres@anarazel.de>
Date: Tue, 18 Mar 2025 14:40:06 -0400
Subject: [PATCH v2.12 07/28] localbuf: Track pincount in BufferDesc as well
For AIO on temporary table buffers the AIO subsystem needs to be able to
ensure a pin on a buffer while AIO is going on, even if the IO issuing query
errors out. Tracking the buffer in LocalRefCount does not work, as it would
cause CheckForLocalBufferLeaks() to assert out.
Instead, also track the refcount in BufferDesc.state, not just
LocalRefCount. This also makes local buffers behave a bit more akin to shared
buffers.
Note that we still don't need locking, AIO completion callbacks for local
buffers are executed in the issuing session (i.e. nobody else has access to
the BufferDesc).
Reviewed-by: Noah Misch <noah@leadboat.com>
Discussion: https://postgr.es/m/uvrtrknj4kdytuboidbhwclo4gxhswwcpgadptsjvjqcluzmah%40brqs62irg4dt
---
src/backend/storage/buffer/bufmgr.c | 22 +++++++++++++++++++++
src/backend/storage/buffer/localbuf.c | 28 +++++++++++++++++++++++++--
2 files changed, 48 insertions(+), 2 deletions(-)
diff --git a/src/backend/storage/buffer/bufmgr.c b/src/backend/storage/buffer/bufmgr.c
index 323382dcfa8..a2c4338f089 100644
--- a/src/backend/storage/buffer/bufmgr.c
+++ b/src/backend/storage/buffer/bufmgr.c
@@ -5404,6 +5404,18 @@ ConditionalLockBufferForCleanup(Buffer buffer)
Assert(refcount > 0);
if (refcount != 1)
return false;
+
+ /*
+ * Check that the AIO subsystem doesn't have a pin. Likely not
+ * possible today, but better safe than sorry.
+ */
+ bufHdr = GetLocalBufferDescriptor(-buffer - 1);
+ buf_state = pg_atomic_read_u32(&bufHdr->state);
+ refcount = BUF_STATE_GET_REFCOUNT(buf_state);
+ Assert(refcount > 0);
+ if (refcount != 1)
+ return false;
+
/* Nobody else to wait for */
return true;
}
@@ -5457,6 +5469,16 @@ IsBufferCleanupOK(Buffer buffer)
/* There should be exactly one pin */
if (LocalRefCount[-buffer - 1] != 1)
return false;
+
+ /*
+ * Check that the AIO subsystem doesn't have a pin. Likely not
+ * possible today, but better safe than sorry.
+ */
+ bufHdr = GetLocalBufferDescriptor(-buffer - 1);
+ buf_state = pg_atomic_read_u32(&bufHdr->state);
+ if (BUF_STATE_GET_REFCOUNT(buf_state) != 1)
+ return false;
+
/* Nobody else to wait for */
return true;
}
diff --git a/src/backend/storage/buffer/localbuf.c b/src/backend/storage/buffer/localbuf.c
index f172a5c7820..3a722321533 100644
--- a/src/backend/storage/buffer/localbuf.c
+++ b/src/backend/storage/buffer/localbuf.c
@@ -249,6 +249,13 @@ GetLocalVictimBuffer(void)
pg_atomic_unlocked_write_u32(&bufHdr->state, buf_state);
trycounter = NLocBuffer;
}
+ else if (BUF_STATE_GET_REFCOUNT(buf_state) > 0)
+ {
+ /*
+ * This can be reached if the backend initiated AIO for this
+ * buffer and then errored out.
+ */
+ }
else
{
/* Found a usable buffer */
@@ -570,7 +577,13 @@ InvalidateLocalBuffer(BufferDesc *bufHdr, bool check_unreferenced)
buf_state = pg_atomic_read_u32(&bufHdr->state);
- if (check_unreferenced && LocalRefCount[bufid] != 0)
+ /*
+ * We need to test not just LocalRefCount[bufid] but also the BufferDesc
+ * itself, as the latter is used to represent a pin by the AIO subsystem.
+ * This can happen if AIO is initiated and then the query errors out.
+ */
+ if (check_unreferenced &&
+ (LocalRefCount[bufid] != 0 || BUF_STATE_GET_REFCOUNT(buf_state) != 0))
elog(ERROR, "block %u of %s is still referenced (local %u)",
bufHdr->tag.blockNum,
relpathbackend(BufTagGetRelFileLocator(&bufHdr->tag),
@@ -744,12 +757,13 @@ PinLocalBuffer(BufferDesc *buf_hdr, bool adjust_usagecount)
if (LocalRefCount[bufid] == 0)
{
NLocalPinnedBuffers++;
+ buf_state += BUF_REFCOUNT_ONE;
if (adjust_usagecount &&
BUF_STATE_GET_USAGECOUNT(buf_state) < BM_MAX_USAGE_COUNT)
{
buf_state += BUF_USAGECOUNT_ONE;
- pg_atomic_unlocked_write_u32(&buf_hdr->state, buf_state);
}
+ pg_atomic_unlocked_write_u32(&buf_hdr->state, buf_state);
}
LocalRefCount[bufid]++;
ResourceOwnerRememberBuffer(CurrentResourceOwner,
@@ -775,7 +789,17 @@ UnpinLocalBufferNoOwner(Buffer buffer)
Assert(NLocalPinnedBuffers > 0);
if (--LocalRefCount[buffid] == 0)
+ {
+ BufferDesc *buf_hdr = GetLocalBufferDescriptor(buffid);
+ uint32 buf_state;
+
NLocalPinnedBuffers--;
+
+ buf_state = pg_atomic_read_u32(&buf_hdr->state);
+ Assert(BUF_STATE_GET_REFCOUNT(buf_state) > 0);
+ buf_state -= BUF_REFCOUNT_ONE;
+ pg_atomic_unlocked_write_u32(&buf_hdr->state, buf_state);
+ }
}
/*
--
2.48.1.76.g4e746b1a31.dirty
[text/x-diff] v2.12-0008-bufmgr-Implement-AIO-read-support.patch (31.8K, ../../cxi6ssym62v7ywkt367qda3xik5jhsrcycwedyfqacinxooys5@pwarur37bosc/9-v2.12-0008-bufmgr-Implement-AIO-read-support.patch)
download | inline diff:
From 6718b7ef56c779dceb4b188a653fdc5e5d5a6dad Mon Sep 17 00:00:00 2001
From: Andres Freund <andres@anarazel.de>
Date: Tue, 18 Mar 2025 14:40:06 -0400
Subject: [PATCH v2.12 08/28] bufmgr: Implement AIO read support
This commit implements the infrastructure to perform asynchronous reads into
the buffer pool.
To do so, it:
- Adds readv AIO callbacks for shared and local buffers
It may be worth calling out that shared buffer completions may be run in a
different backend than where the IO started.
- Adds an AIO wait reference to BufferDesc, to allow backends to wait for
in-progress asynchronous IOs
- Adapts StartBufferIO(), WaitIO(), TerminateBufferIO(), and their localbuf.c
equivalents, to be able to deal with AIO
- Moves the code to handle BM_PIN_COUNT_WAITER into a helper function, as it
now also needs to be called on IO completion
As of this commit, nothing issues AIO on shared buffers. A future commit will
update StartReadBuffers() to do so.
Reviewed-by: Noah Misch <noah@leadboat.com>
Reviewed-by: Melanie Plageman <melanieplageman@gmail.com>
Reviewed-by: Nazir Bilal Yavuz <byavuz81@gmail.com>
Discussion: https://postgr.es/m/uvrtrknj4kdytuboidbhwclo4gxhswwcpgadptsjvjqcluzmah%40brqs62irg4dt
Discussion: https://postgr.es/m/20210223100344.llw5an2aklengrmn@alap3.anarazel.de
Discussion: https://postgr.es/m/stj36ea6yyhoxtqkhpieia2z4krnam7qyetc57rfezgk4zgapf@gcnactj4z56m
---
src/include/storage/aio.h | 4 +
src/include/storage/buf_internals.h | 7 +-
src/include/storage/bufmgr.h | 4 +
src/backend/storage/aio/aio_callback.c | 5 +
src/backend/storage/buffer/README | 9 +-
src/backend/storage/buffer/buf_init.c | 3 +
src/backend/storage/buffer/bufmgr.c | 509 ++++++++++++++++++++++---
src/backend/storage/buffer/localbuf.c | 60 ++-
8 files changed, 542 insertions(+), 59 deletions(-)
diff --git a/src/include/storage/aio.h b/src/include/storage/aio.h
index 1be7821b29b..92f29cfdb71 100644
--- a/src/include/storage/aio.h
+++ b/src/include/storage/aio.h
@@ -195,6 +195,10 @@ typedef enum PgAioHandleCallbackID
PGAIO_HCB_MD_READV,
PGAIO_HCB_MD_WRITEV,
+
+ PGAIO_HCB_SHARED_BUFFER_READV,
+
+ PGAIO_HCB_LOCAL_BUFFER_READV,
} PgAioHandleCallbackID;
diff --git a/src/include/storage/buf_internals.h b/src/include/storage/buf_internals.h
index 9327f60c44c..8a52ace37cd 100644
--- a/src/include/storage/buf_internals.h
+++ b/src/include/storage/buf_internals.h
@@ -17,6 +17,7 @@
#include "pgstat.h"
#include "port/atomics.h"
+#include "storage/aio_types.h"
#include "storage/buf.h"
#include "storage/bufmgr.h"
#include "storage/condition_variable.h"
@@ -264,6 +265,8 @@ typedef struct BufferDesc
int wait_backend_pgprocno; /* backend of pin-count waiter */
int freeNext; /* link in freelist chain */
+
+ PgAioWaitRef io_wref; /* set iff AIO is in progress */
LWLock content_lock; /* to lock access to buffer contents */
} BufferDesc;
@@ -472,8 +475,8 @@ extern BlockNumber ExtendBufferedRelLocal(BufferManagerRelation bmr,
uint32 *extended_by);
extern void MarkLocalBufferDirty(Buffer buffer);
extern void TerminateLocalBufferIO(BufferDesc *bufHdr, bool clear_dirty,
- uint32 set_flag_bits);
-extern bool StartLocalBufferIO(BufferDesc *bufHdr, bool forInput);
+ uint32 set_flag_bits, bool syncio);
+extern bool StartLocalBufferIO(BufferDesc *bufHdr, bool forInput, bool nowait);
extern void FlushLocalBuffer(BufferDesc *bufHdr, SMgrRelation reln);
extern void DropRelationLocalBuffers(RelFileLocator rlocator,
ForkNumber forkNum,
diff --git a/src/include/storage/bufmgr.h b/src/include/storage/bufmgr.h
index 538b890a51d..5137288c96e 100644
--- a/src/include/storage/bufmgr.h
+++ b/src/include/storage/bufmgr.h
@@ -15,6 +15,7 @@
#define BUFMGR_H
#include "port/pg_iovec.h"
+#include "storage/aio_types.h"
#include "storage/block.h"
#include "storage/buf.h"
#include "storage/bufpage.h"
@@ -170,6 +171,9 @@ extern PGDLLIMPORT int checkpoint_flush_after;
extern PGDLLIMPORT int backend_flush_after;
extern PGDLLIMPORT int bgwriter_flush_after;
+extern const PgAioHandleCallbacks aio_shared_buffer_readv_cb;
+extern const PgAioHandleCallbacks aio_local_buffer_readv_cb;
+
/* in buf_init.c */
extern PGDLLIMPORT char *BufferBlocks;
diff --git a/src/backend/storage/aio/aio_callback.c b/src/backend/storage/aio/aio_callback.c
index 54970337bf3..566060c12cb 100644
--- a/src/backend/storage/aio/aio_callback.c
+++ b/src/backend/storage/aio/aio_callback.c
@@ -18,6 +18,7 @@
#include "miscadmin.h"
#include "storage/aio.h"
#include "storage/aio_internal.h"
+#include "storage/bufmgr.h"
#include "storage/md.h"
@@ -40,6 +41,10 @@ static const PgAioHandleCallbacksEntry aio_handle_cbs[] = {
CALLBACK_ENTRY(PGAIO_HCB_INVALID, aio_invalid_cb),
CALLBACK_ENTRY(PGAIO_HCB_MD_READV, aio_md_readv_cb),
+
+ CALLBACK_ENTRY(PGAIO_HCB_SHARED_BUFFER_READV, aio_shared_buffer_readv_cb),
+
+ CALLBACK_ENTRY(PGAIO_HCB_LOCAL_BUFFER_READV, aio_local_buffer_readv_cb),
#undef CALLBACK_ENTRY
};
diff --git a/src/backend/storage/buffer/README b/src/backend/storage/buffer/README
index 011af7aff3e..a182fcd660c 100644
--- a/src/backend/storage/buffer/README
+++ b/src/backend/storage/buffer/README
@@ -147,9 +147,12 @@ in the buffer. It is used per the rules above.
* The BM_IO_IN_PROGRESS flag acts as a kind of lock, used to wait for I/O on a
buffer to complete (and in releases before 14, it was accompanied by a
-per-buffer LWLock). The process doing a read or write sets the flag for the
-duration, and processes that need to wait for it to be cleared sleep on a
-condition variable.
+per-buffer LWLock). The process starting a read or write sets the flag. When
+the I/O is completed, be it by the process that initiated the I/O or by
+another process, the flag is removed and the Buffer's condition variable is
+signalled. Processes that need to wait for the I/O to complete can wait for
+asynchronous I/O by using BufferDesc->io_wref and for BM_IO_IN_PROGRESS to be
+unset by sleeping on the buffer's condition variable.
Normal Buffer Replacement Strategy
diff --git a/src/backend/storage/buffer/buf_init.c b/src/backend/storage/buffer/buf_init.c
index ed1f8e03190..ed1dc488a42 100644
--- a/src/backend/storage/buffer/buf_init.c
+++ b/src/backend/storage/buffer/buf_init.c
@@ -14,6 +14,7 @@
*/
#include "postgres.h"
+#include "storage/aio.h"
#include "storage/buf_internals.h"
#include "storage/bufmgr.h"
@@ -125,6 +126,8 @@ BufferManagerShmemInit(void)
buf->buf_id = i;
+ pgaio_wref_clear(&buf->io_wref);
+
/*
* Initially link all the buffers together as unused. Subsequent
* management of this list is done by freelist.c.
diff --git a/src/backend/storage/buffer/bufmgr.c b/src/backend/storage/buffer/bufmgr.c
index a2c4338f089..3dad509b4d3 100644
--- a/src/backend/storage/buffer/bufmgr.c
+++ b/src/backend/storage/buffer/bufmgr.c
@@ -48,6 +48,7 @@
#include "pg_trace.h"
#include "pgstat.h"
#include "postmaster/bgwriter.h"
+#include "storage/aio.h"
#include "storage/buf_internals.h"
#include "storage/bufmgr.h"
#include "storage/fd.h"
@@ -519,7 +520,8 @@ static int SyncOneBuffer(int buf_id, bool skip_recently_used,
static void WaitIO(BufferDesc *buf);
static bool StartBufferIO(BufferDesc *buf, bool forInput, bool nowait);
static void TerminateBufferIO(BufferDesc *buf, bool clear_dirty,
- uint32 set_flag_bits, bool forget_owner);
+ uint32 set_flag_bits, bool forget_owner,
+ bool syncio);
static void AbortBufferIO(Buffer buffer);
static void shared_buffer_write_error_callback(void *arg);
static void local_buffer_write_error_callback(void *arg);
@@ -1041,7 +1043,7 @@ ZeroAndLockBuffer(Buffer buffer, ReadBufferMode mode, bool already_valid)
{
/* Simple case for non-shared buffers. */
bufHdr = GetLocalBufferDescriptor(-buffer - 1);
- need_to_zero = StartLocalBufferIO(bufHdr, true);
+ need_to_zero = StartLocalBufferIO(bufHdr, true, false);
}
else
{
@@ -1077,9 +1079,9 @@ ZeroAndLockBuffer(Buffer buffer, ReadBufferMode mode, bool already_valid)
/* Set BM_VALID, terminate IO, and wake up any waiters */
if (isLocalBuf)
- TerminateLocalBufferIO(bufHdr, false, BM_VALID);
+ TerminateLocalBufferIO(bufHdr, false, BM_VALID, true);
else
- TerminateBufferIO(bufHdr, false, BM_VALID, true);
+ TerminateBufferIO(bufHdr, false, BM_VALID, true, true);
}
else if (!isLocalBuf)
{
@@ -1454,7 +1456,8 @@ static inline bool
WaitReadBuffersCanStartIO(Buffer buffer, bool nowait)
{
if (BufferIsLocal(buffer))
- return StartLocalBufferIO(GetLocalBufferDescriptor(-buffer - 1), true);
+ return StartLocalBufferIO(GetLocalBufferDescriptor(-buffer - 1),
+ true, nowait);
else
return StartBufferIO(GetBufferDescriptor(buffer - 1), true, nowait);
}
@@ -1604,9 +1607,9 @@ WaitReadBuffers(ReadBuffersOperation *operation)
/* Set BM_VALID, terminate IO, and wake up any waiters */
if (persistence == RELPERSISTENCE_TEMP)
- TerminateLocalBufferIO(bufHdr, false, BM_VALID);
+ TerminateLocalBufferIO(bufHdr, false, BM_VALID, true);
else
- TerminateBufferIO(bufHdr, false, BM_VALID, true);
+ TerminateBufferIO(bufHdr, false, BM_VALID, true, true);
/* Report I/Os as completing individually. */
TRACE_POSTGRESQL_BUFFER_READ_DONE(forknum, io_first_block + j,
@@ -1868,13 +1871,14 @@ retry:
}
/*
- * We assume the only reason for it to be pinned is that someone else is
- * flushing the page out. Wait for them to finish. (This could be an
- * infinite loop if the refcount is messed up... it would be nice to time
- * out after awhile, but there seems no way to be sure how many loops may
- * be needed. Note that if the other guy has pinned the buffer but not
- * yet done StartBufferIO, WaitIO will fall through and we'll effectively
- * be busy-looping here.)
+ * We assume the reason for it to be pinned is that either we were
+ * asynchronously reading the page in before erroring out or someone else
+ * is flushing the page out. Wait for the IO to finish. (This could be
+ * an infinite loop if the refcount is messed up... it would be nice to
+ * time out after awhile, but there seems no way to be sure how many loops
+ * may be needed. Note that if the other guy has pinned the buffer but
+ * not yet done StartBufferIO, WaitIO will fall through and we'll
+ * effectively be busy-looping here.)
*/
if (BUF_STATE_GET_REFCOUNT(buf_state) != 0)
{
@@ -2514,7 +2518,7 @@ ExtendBufferedRelShared(BufferManagerRelation bmr,
if (lock)
LWLockAcquire(BufferDescriptorGetContentLock(buf_hdr), LW_EXCLUSIVE);
- TerminateBufferIO(buf_hdr, false, BM_VALID, true);
+ TerminateBufferIO(buf_hdr, false, BM_VALID, true, true);
}
pgBufferUsage.shared_blks_written += extend_by;
@@ -2860,6 +2864,44 @@ PinBuffer_Locked(BufferDesc *buf)
ResourceOwnerRememberBuffer(CurrentResourceOwner, b);
}
+/*
+ * Support for waking up another backend that is waiting for the cleanup lock
+ * to be released using BM_PIN_COUNT_WAITER.
+ *
+ * See LockBufferForCleanup().
+ *
+ * Expected to be called just after releasing a buffer pin (in a BufferDesc,
+ * not just reducing the backend-local pincount for the buffer).
+ */
+static void
+WakePinCountWaiter(BufferDesc *buf)
+{
+ /*
+ * Acquire the buffer header lock, re-check that there's a waiter. Another
+ * backend could have unpinned this buffer, and already woken up the
+ * waiter.
+ *
+ * There's no danger of the buffer being replaced after we unpinned it
+ * above, as it's pinned by the waiter. The waiter removes
+ * BM_PIN_COUNT_WAITER if it stops waiting for a reason other than this
+ * backend waking it up.
+ */
+ uint32 buf_state = LockBufHdr(buf);
+
+ if ((buf_state & BM_PIN_COUNT_WAITER) &&
+ BUF_STATE_GET_REFCOUNT(buf_state) == 1)
+ {
+ /* we just released the last pin other than the waiter's */
+ int wait_backend_pgprocno = buf->wait_backend_pgprocno;
+
+ buf_state &= ~BM_PIN_COUNT_WAITER;
+ UnlockBufHdr(buf, buf_state);
+ ProcSendSignal(wait_backend_pgprocno);
+ }
+ else
+ UnlockBufHdr(buf, buf_state);
+}
+
/*
* UnpinBuffer -- make buffer available for replacement.
*
@@ -2928,29 +2970,8 @@ UnpinBufferNoOwner(BufferDesc *buf)
/* Support LockBufferForCleanup() */
if (buf_state & BM_PIN_COUNT_WAITER)
- {
- /*
- * Acquire the buffer header lock, re-check that there's a waiter.
- * Another backend could have unpinned this buffer, and already
- * woken up the waiter. There's no danger of the buffer being
- * replaced after we unpinned it above, as it's pinned by the
- * waiter.
- */
- buf_state = LockBufHdr(buf);
+ WakePinCountWaiter(buf);
- if ((buf_state & BM_PIN_COUNT_WAITER) &&
- BUF_STATE_GET_REFCOUNT(buf_state) == 1)
- {
- /* we just released the last pin other than the waiter's */
- int wait_backend_pgprocno = buf->wait_backend_pgprocno;
-
- buf_state &= ~BM_PIN_COUNT_WAITER;
- UnlockBufHdr(buf, buf_state);
- ProcSendSignal(wait_backend_pgprocno);
- }
- else
- UnlockBufHdr(buf, buf_state);
- }
ForgetPrivateRefCountEntry(ref);
}
}
@@ -3970,7 +3991,7 @@ FlushBuffer(BufferDesc *buf, SMgrRelation reln, IOObject io_object,
* Mark the buffer as clean (unless BM_JUST_DIRTIED has become set) and
* end the BM_IO_IN_PROGRESS state.
*/
- TerminateBufferIO(buf, true, 0, true);
+ TerminateBufferIO(buf, true, 0, true, true);
TRACE_POSTGRESQL_BUFFER_FLUSH_DONE(BufTagGetForkNum(&buf->tag),
buf->tag.blockNum,
@@ -5529,6 +5550,7 @@ WaitIO(BufferDesc *buf)
for (;;)
{
uint32 buf_state;
+ PgAioWaitRef iow;
/*
* It may not be necessary to acquire the spinlock to check the flag
@@ -5536,10 +5558,40 @@ WaitIO(BufferDesc *buf)
* play it safe.
*/
buf_state = LockBufHdr(buf);
+
+ /*
+ * Copy the wait reference while holding the spinlock. This protects
+ * against a concurrent TerminateBufferIO() in another backend from
+ * clearing the wref while it's being read.
+ */
+ iow = buf->io_wref;
UnlockBufHdr(buf, buf_state);
+ /* no IO in progress, we don't need to wait */
if (!(buf_state & BM_IO_IN_PROGRESS))
break;
+
+ /*
+ * The buffer has asynchronous IO in progress, wait for it to
+ * complete.
+ */
+ if (pgaio_wref_valid(&iow))
+ {
+ pgaio_wref_wait(&iow);
+
+ /*
+ * The AIO subsystem internally uses condition variables and thus
+ * might remove this backend from the BufferDesc's CV. While that
+ * wouldn't cause a correctness issue (the first CV sleep just
+ * immediately returns if not already registered), it seems worth
+ * avoiding unnecessary loop iterations, given that we take care
+ * to do so at the start of the function.
+ */
+ ConditionVariablePrepareToSleep(cv);
+ continue;
+ }
+
+ /* wait on BufferDesc->cv, e.g. for concurrent synchronous IO */
ConditionVariableSleep(cv, WAIT_EVENT_BUFFER_IO);
}
ConditionVariableCancelSleep();
@@ -5548,13 +5600,12 @@ WaitIO(BufferDesc *buf)
/*
* StartBufferIO: begin I/O on this buffer
* (Assumptions)
- * My process is executing no IO
+ * My process is executing no IO on this buffer
* The buffer is Pinned
*
- * In some scenarios there are race conditions in which multiple backends
- * could attempt the same I/O operation concurrently. If someone else
- * has already started I/O on this buffer then we will block on the
- * I/O condition variable until he's done.
+ * In some scenarios multiple backends could attempt the same I/O operation
+ * concurrently. If someone else has already started I/O on this buffer then
+ * we will wait for completion of the IO using WaitIO().
*
* Input operations are only attempted on buffers that are not BM_VALID,
* and output operations only on buffers that are BM_VALID and BM_DIRTY,
@@ -5590,9 +5641,9 @@ StartBufferIO(BufferDesc *buf, bool forInput, bool nowait)
/* Once we get here, there is definitely no I/O active on this buffer */
+ /* Check if someone else already did the I/O */
if (forInput ? (buf_state & BM_VALID) : !(buf_state & BM_DIRTY))
{
- /* someone else already did the I/O */
UnlockBufHdr(buf, buf_state);
return false;
}
@@ -5628,7 +5679,7 @@ StartBufferIO(BufferDesc *buf, bool forInput, bool nowait)
*/
static void
TerminateBufferIO(BufferDesc *buf, bool clear_dirty, uint32 set_flag_bits,
- bool forget_owner)
+ bool forget_owner, bool syncio)
{
uint32 buf_state;
@@ -5643,6 +5694,14 @@ TerminateBufferIO(BufferDesc *buf, bool clear_dirty, uint32 set_flag_bits,
if (clear_dirty && !(buf_state & BM_JUST_DIRTIED))
buf_state &= ~(BM_DIRTY | BM_CHECKPOINT_NEEDED);
+ if (!syncio)
+ {
+ /* release ownership by the AIO subsystem */
+ Assert(BUF_STATE_GET_REFCOUNT(buf_state) > 0);
+ buf_state -= BUF_REFCOUNT_ONE;
+ pgaio_wref_clear(&buf->io_wref);
+ }
+
buf_state |= set_flag_bits;
UnlockBufHdr(buf, buf_state);
@@ -5651,6 +5710,17 @@ TerminateBufferIO(BufferDesc *buf, bool clear_dirty, uint32 set_flag_bits,
BufferDescriptorGetBuffer(buf));
ConditionVariableBroadcast(BufferDescriptorGetIOCV(buf));
+
+ /*
+ * Support LockBufferForCleanup()
+ *
+ * We may have just released the last pin other than the waiter's. In most
+ * cases, this backend holds another pin on the buffer. But, if, for
+ * example, this backend is completing an IO issued by another backend, it
+ * may be time to wake the waiter.
+ */
+ if (buf_state & BM_PIN_COUNT_WAITER)
+ WakePinCountWaiter(buf);
}
/*
@@ -5699,7 +5769,7 @@ AbortBufferIO(Buffer buffer)
}
}
- TerminateBufferIO(buf_hdr, false, BM_IO_ERROR, false);
+ TerminateBufferIO(buf_hdr, false, BM_IO_ERROR, false, true);
}
/*
@@ -6150,3 +6220,350 @@ EvictUnpinnedBuffer(Buffer buf)
return result;
}
+
+/*
+ * Generic implementation of the AIO handle staging callback for readv/writev
+ * on local/shared buffers.
+ *
+ * Each readv/writev can target multiple buffers. The buffers have already
+ * been registered with the IO handle.
+ *
+ * To make the IO ready for execution ("staging"), we need to ensure that the
+ * targeted buffers are in an appropriate state while the IO is ongoing. For
+ * that the AIO subsystem needs to have its own buffer pin, otherwise an error
+ * in this backend could lead to this backend's buffer pin being released as
+ * part of error handling, which in turn could lead to the buffer being
+ * replaced while IO is ongoing.
+ */
+static pg_attribute_always_inline void
+buffer_stage_common(PgAioHandle *ioh, bool is_write, bool is_temp)
+{
+ uint64 *io_data;
+ uint8 handle_data_len;
+ PgAioWaitRef io_ref;
+ BufferTag first PG_USED_FOR_ASSERTS_ONLY = {0};
+
+ io_data = pgaio_io_get_handle_data(ioh, &handle_data_len);
+
+ pgaio_io_get_wref(ioh, &io_ref);
+
+ /* iterate over all buffers affected by the vectored readv/writev */
+ for (int i = 0; i < handle_data_len; i++)
+ {
+ Buffer buffer = (Buffer) io_data[i];
+ BufferDesc *buf_hdr = is_temp ?
+ GetLocalBufferDescriptor(-buffer - 1)
+ : GetBufferDescriptor(buffer - 1);
+ uint32 buf_state;
+
+ /*
+ * Check that all the buffers are actually ones that could conceivably
+ * be done in one IO, i.e. are sequential. This is the last
+ * buffer-aware code before IO is actually executed and confusion
+ * about which buffers are target by IO can be hard to debug, making
+ * it worth doing extra-paranoid checks.
+ */
+ if (i == 0)
+ first = buf_hdr->tag;
+ else
+ {
+ Assert(buf_hdr->tag.relNumber == first.relNumber);
+ Assert(buf_hdr->tag.blockNum == first.blockNum + i);
+ }
+
+ if (is_temp)
+ buf_state = pg_atomic_read_u32(&buf_hdr->state);
+ else
+ buf_state = LockBufHdr(buf_hdr);
+
+ /* verify the buffer is in the expected state */
+ Assert(buf_state & BM_TAG_VALID);
+ if (is_write)
+ {
+ Assert(buf_state & BM_VALID);
+ Assert(buf_state & BM_DIRTY);
+ }
+ else
+ {
+ Assert(!(buf_state & BM_VALID));
+ Assert(!(buf_state & BM_DIRTY));
+ }
+
+ /* temp buffers don't use BM_IO_IN_PROGRESS */
+ if (!is_temp)
+ Assert(buf_state & BM_IO_IN_PROGRESS);
+
+ Assert(BUF_STATE_GET_REFCOUNT(buf_state) >= 1);
+
+ /*
+ * Reflect that the buffer is now owned by the AIO subsystem.
+ *
+ * For local buffers: This can't be done just via LocalRefCount, as
+ * one might initially think, as this backend could error out while
+ * AIO is still in progress, releasing all the pins by the backend
+ * itself.
+ *
+ * This pin is released again in TerminateBufferIO().
+ */
+ buf_state += BUF_REFCOUNT_ONE;
+ buf_hdr->io_wref = io_ref;
+
+ if (is_temp)
+ pg_atomic_unlocked_write_u32(&buf_hdr->state, buf_state);
+ else
+ UnlockBufHdr(buf_hdr, buf_state);
+
+ /*
+ * Ensure the content lock that prevents buffer modifications while
+ * the buffer is being written out is not released early due to an
+ * error.
+ */
+ if (is_write && !is_temp)
+ {
+ LWLock *content_lock;
+
+ content_lock = BufferDescriptorGetContentLock(buf_hdr);
+
+ Assert(LWLockHeldByMe(content_lock));
+
+ /*
+ * Lock is now owned by AIO subsystem.
+ */
+ LWLockDisown(content_lock);
+ }
+
+ /*
+ * Stop tracking this buffer via the resowner - the AIO system now
+ * keeps track.
+ */
+ if (!is_temp)
+ ResourceOwnerForgetBufferIO(CurrentResourceOwner, buffer);
+ }
+}
+
+/*
+ * Helper for AIO readv completion callbacks, supporting both shared and temp
+ * buffers. Gets called once for each buffer in a multi-page read.
+ */
+static pg_attribute_always_inline PgAioResult
+buffer_readv_complete_one(uint8 buf_off, Buffer buffer, uint8 flags,
+ bool failed, bool is_temp)
+{
+ BufferDesc *buf_hdr = is_temp ?
+ GetLocalBufferDescriptor(-buffer - 1)
+ : GetBufferDescriptor(buffer - 1);
+ BufferTag tag = buf_hdr->tag;
+ char *bufdata = BufferGetBlock(buffer);
+ PgAioResult result = {.status = PGAIO_RS_OK};
+ uint32 set_flag_bits;
+
+ /* check that the buffer is in the expected state for a read */
+#ifdef USE_ASSERT_CHECKING
+ {
+ uint32 buf_state = pg_atomic_read_u32(&buf_hdr->state);
+
+ Assert(buf_state & BM_TAG_VALID);
+ Assert(!(buf_state & BM_VALID));
+ /* temp buffers don't use BM_IO_IN_PROGRESS */
+ if (!is_temp)
+ Assert(buf_state & BM_IO_IN_PROGRESS);
+ Assert(!(buf_state & BM_DIRTY));
+ }
+#endif
+
+ /* check for garbage data */
+ if (!failed &&
+ !PageIsVerifiedExtended((Page) bufdata, tag.blockNum,
+ PIV_LOG_WARNING | PIV_REPORT_STAT))
+ {
+ RelFileLocator rlocator = BufTagGetRelFileLocator(&tag);
+
+ if ((flags & READ_BUFFERS_ZERO_ON_ERROR) || zero_damaged_pages)
+ {
+ ereport(WARNING,
+ (errcode(ERRCODE_DATA_CORRUPTED),
+ errmsg("invalid page in block %u of relation %s; zeroing out page",
+ tag.blockNum,
+ relpathbackend(rlocator,
+ is_temp ? MyProcNumber : INVALID_PROC_NUMBER,
+ tag.forkNum).str)));
+ memset(bufdata, 0, BLCKSZ);
+ }
+ else
+ {
+ /* mark buffer as having failed */
+ failed = true;
+
+ /* encode error for buffer_readv_report */
+ result.status = PGAIO_RS_ERROR;
+ if (is_temp)
+ result.id = PGAIO_HCB_LOCAL_BUFFER_READV;
+ else
+ result.id = PGAIO_HCB_SHARED_BUFFER_READV;
+ result.error_data = buf_off;
+ }
+ }
+
+ /* Terminate I/O and set BM_VALID. */
+ set_flag_bits = failed ? BM_IO_ERROR : BM_VALID;
+ if (is_temp)
+ TerminateLocalBufferIO(buf_hdr, false, set_flag_bits, false);
+ else
+ TerminateBufferIO(buf_hdr, false, set_flag_bits, false, false);
+
+ /*
+ * Call the BUFFER_READ_DONE tracepoint in the callback, even though the
+ * callback may not be executed in the same backend that called
+ * BUFFER_READ_START. The alternative would be to defer calling the
+ * tracepoint to a later point (e.g. the local completion callback for
+ * shared buffer reads), which seems even less helpful and would also
+ * imply some overhead.
+ */
+ TRACE_POSTGRESQL_BUFFER_READ_DONE(tag.forkNum,
+ tag.blockNum,
+ tag.spcOid,
+ tag.dbOid,
+ tag.relNumber,
+ is_temp ? MyProcNumber : INVALID_PROC_NUMBER,
+ false);
+
+ return result;
+}
+
+/*
+ * Perform completion handling of a single AIO read. This read may cover
+ * multiple blocks / buffers.
+ *
+ * Shared between shared and local buffers, to reduce code duplication.
+ */
+static pg_attribute_always_inline PgAioResult
+buffer_readv_complete(PgAioHandle *ioh, PgAioResult prior_result,
+ uint8 cb_data, bool is_temp)
+{
+ PgAioResult result = prior_result;
+ PgAioTargetData *td = pgaio_io_get_target_data(ioh);
+ uint64 *io_data;
+ uint8 handle_data_len;
+
+ if (is_temp)
+ {
+ Assert(td->smgr.is_temp);
+ Assert(pgaio_io_get_owner(ioh) == MyProcNumber);
+ }
+ else
+ Assert(!td->smgr.is_temp);
+
+ /*
+ * Iterate over all the buffers affected by this IO and call the
+ * per-buffer completion function for each buffer.
+ */
+ io_data = pgaio_io_get_handle_data(ioh, &handle_data_len);
+ for (uint8 buf_off = 0; buf_off < handle_data_len; buf_off++)
+ {
+ Buffer buf = io_data[buf_off];
+ PgAioResult buf_result;
+ bool failed;
+
+ Assert(BufferIsValid(buf));
+
+ /*
+ * If the entire I/O failed on a lower-level, each buffer needs to be
+ * marked as failed. In case of a partial read, some buffers may be
+ * ok.
+ */
+ failed =
+ prior_result.status == PGAIO_RS_ERROR
+ || prior_result.result <= buf_off;
+
+ buf_result = buffer_readv_complete_one(buf_off, buf, cb_data, failed,
+ is_temp);
+
+ /*
+ * If there wasn't any prior error and page verification failed in
+ * some form, set the whole IO's result to the page's result.
+ */
+ if (result.status != PGAIO_RS_ERROR
+ && buf_result.status != PGAIO_RS_OK)
+ {
+ result = buf_result;
+ pgaio_result_report(result, td, LOG);
+ }
+ }
+
+ return result;
+}
+
+/*
+ * AIO error reporting callback for aio_shared_buffer_readv_cb and
+ * aio_local_buffer_readv_cb.
+ *
+ * Errors are encoded as follows:
+ * - the only error is page verification failing
+ * - result.error_data is the offset of the first page that failed
+ * verification in a larger IO
+ */
+static void
+buffer_readv_report(PgAioResult result, const PgAioTargetData *target_data,
+ int elevel)
+{
+ ProcNumber errProc;
+
+ if (target_data->smgr.is_temp)
+ errProc = MyProcNumber;
+ else
+ errProc = INVALID_PROC_NUMBER;
+
+ ereport(elevel,
+ errcode(ERRCODE_DATA_CORRUPTED),
+ errmsg("invalid page in block %u of relation %s",
+ target_data->smgr.blockNum + result.error_data,
+ relpathbackend(target_data->smgr.rlocator, errProc,
+ target_data->smgr.forkNum).str));
+}
+
+static void
+shared_buffer_readv_stage(PgAioHandle *ioh, uint8 cb_data)
+{
+ buffer_stage_common(ioh, false, false);
+}
+
+static PgAioResult
+shared_buffer_readv_complete(PgAioHandle *ioh, PgAioResult prior_result,
+ uint8 cb_data)
+{
+ return buffer_readv_complete(ioh, prior_result, cb_data, false);
+}
+
+static void
+local_buffer_readv_stage(PgAioHandle *ioh, uint8 cb_data)
+{
+ buffer_stage_common(ioh, false, true);
+}
+
+static PgAioResult
+local_buffer_readv_complete(PgAioHandle *ioh, PgAioResult prior_result,
+ uint8 cb_data)
+{
+ return buffer_readv_complete(ioh, prior_result, cb_data, true);
+}
+
+/* readv callback is passed READ_BUFFERS_* flags as callback data */
+const PgAioHandleCallbacks aio_shared_buffer_readv_cb = {
+ .stage = shared_buffer_readv_stage,
+ .complete_shared = shared_buffer_readv_complete,
+ .report = buffer_readv_report,
+};
+
+/* readv callback is passed READ_BUFFERS_* flags as callback data */
+const PgAioHandleCallbacks aio_local_buffer_readv_cb = {
+ .stage = local_buffer_readv_stage,
+
+ /*
+ * Note that this, in contrast to the shared_buffers case, uses
+ * complete_local, as only the issuing backend has access to the required
+ * datastructures. This is important in case the IO completion may be
+ * consumed incidentally by another backend.
+ */
+ .complete_local = local_buffer_readv_complete,
+ .report = buffer_readv_report,
+};
diff --git a/src/backend/storage/buffer/localbuf.c b/src/backend/storage/buffer/localbuf.c
index 3a722321533..3b81a5661c7 100644
--- a/src/backend/storage/buffer/localbuf.c
+++ b/src/backend/storage/buffer/localbuf.c
@@ -17,7 +17,9 @@
#include "access/parallel.h"
#include "executor/instrument.h"
+#include "pg_trace.h"
#include "pgstat.h"
+#include "storage/aio.h"
#include "storage/buf_internals.h"
#include "storage/bufmgr.h"
#include "storage/fd.h"
@@ -187,7 +189,7 @@ FlushLocalBuffer(BufferDesc *bufHdr, SMgrRelation reln)
* Try to start an I/O operation. There currently are no reasons for
* StartLocalBufferIO to return false, so we raise an error in that case.
*/
- if (!StartLocalBufferIO(bufHdr, false))
+ if (!StartLocalBufferIO(bufHdr, false, false))
elog(ERROR, "failed to start write IO on local buffer");
/* Find smgr relation for buffer */
@@ -211,7 +213,7 @@ FlushLocalBuffer(BufferDesc *bufHdr, SMgrRelation reln)
IOOP_WRITE, io_start, 1, BLCKSZ);
/* Mark not-dirty */
- TerminateLocalBufferIO(bufHdr, true, 0);
+ TerminateLocalBufferIO(bufHdr, true, 0, true);
pgBufferUsage.local_blks_written++;
}
@@ -430,7 +432,7 @@ ExtendBufferedRelLocal(BufferManagerRelation bmr,
pg_atomic_unlocked_write_u32(&existing_hdr->state, buf_state);
/* no need to loop for local buffers */
- StartLocalBufferIO(existing_hdr, true);
+ StartLocalBufferIO(existing_hdr, true, false);
}
else
{
@@ -446,7 +448,7 @@ ExtendBufferedRelLocal(BufferManagerRelation bmr,
hresult->id = victim_buf_id;
- StartLocalBufferIO(victim_buf_hdr, true);
+ StartLocalBufferIO(victim_buf_hdr, true, false);
}
}
@@ -515,13 +517,31 @@ MarkLocalBufferDirty(Buffer buffer)
* Like StartBufferIO, but for local buffers
*/
bool
-StartLocalBufferIO(BufferDesc *bufHdr, bool forInput)
+StartLocalBufferIO(BufferDesc *bufHdr, bool forInput, bool nowait)
{
- uint32 buf_state = pg_atomic_read_u32(&bufHdr->state);
+ uint32 buf_state;
+ /*
+ * With AIO the buffer could have IO in progress, e.g. when there are two
+ * scans of the same relation. Either wait for the other IO or return
+ * false.
+ */
+ if (pgaio_wref_valid(&bufHdr->io_wref))
+ {
+ PgAioWaitRef iow = bufHdr->io_wref;
+
+ if (nowait)
+ return false;
+
+ pgaio_wref_wait(&iow);
+ }
+
+ /* Once we get here, there is definitely no I/O active on this buffer */
+
+ /* Check if someone else already did the I/O */
+ buf_state = pg_atomic_read_u32(&bufHdr->state);
if (forInput ? (buf_state & BM_VALID) : !(buf_state & BM_DIRTY))
{
- /* someone else already did the I/O */
return false;
}
@@ -536,7 +556,8 @@ StartLocalBufferIO(BufferDesc *bufHdr, bool forInput)
* Like TerminateBufferIO, but for local buffers
*/
void
-TerminateLocalBufferIO(BufferDesc *bufHdr, bool clear_dirty, uint32 set_flag_bits)
+TerminateLocalBufferIO(BufferDesc *bufHdr, bool clear_dirty, uint32 set_flag_bits,
+ bool syncio)
{
/* Only need to adjust flags */
uint32 buf_state = pg_atomic_read_u32(&bufHdr->state);
@@ -549,6 +570,14 @@ TerminateLocalBufferIO(BufferDesc *bufHdr, bool clear_dirty, uint32 set_flag_bit
if (clear_dirty)
buf_state &= ~BM_DIRTY;
+ if (!syncio)
+ {
+ /* release pin held by IO subsystem, see also buffer_stage_common() */
+ Assert(BUF_STATE_GET_REFCOUNT(buf_state) > 0);
+ buf_state -= BUF_REFCOUNT_ONE;
+ pgaio_wref_clear(&bufHdr->io_wref);
+ }
+
buf_state |= set_flag_bits;
pg_atomic_unlocked_write_u32(&bufHdr->state, buf_state);
@@ -575,6 +604,19 @@ InvalidateLocalBuffer(BufferDesc *bufHdr, bool check_unreferenced)
uint32 buf_state;
LocalBufferLookupEnt *hresult;
+ /*
+ * It's possible that we started IO on this buffer before e.g. aborting
+ * the transaction that created a table. We need to wait for that IO to
+ * complete before removing / reusing the buffer.
+ */
+ if (pgaio_wref_valid(&bufHdr->io_wref))
+ {
+ PgAioWaitRef iow = bufHdr->io_wref;
+
+ pgaio_wref_wait(&iow);
+ Assert(!pgaio_wref_valid(&bufHdr->io_wref));
+ }
+
buf_state = pg_atomic_read_u32(&bufHdr->state);
/*
@@ -714,6 +756,8 @@ InitLocalBuffers(void)
*/
buf->buf_id = -i - 2;
+ pgaio_wref_clear(&buf->io_wref);
+
/*
* Intentionally do not initialize the buffer's atomic variable
* (besides zeroing the underlying memory above). That way we get
--
2.48.1.76.g4e746b1a31.dirty
[text/x-diff] v2.12-0009-bufmgr-Use-AIO-in-StartReadBuffers.patch (26.6K, ../../cxi6ssym62v7ywkt367qda3xik5jhsrcycwedyfqacinxooys5@pwarur37bosc/10-v2.12-0009-bufmgr-Use-AIO-in-StartReadBuffers.patch)
download | inline diff:
From 9effa7a99730091e3f458dd765c651df70d6b2a3 Mon Sep 17 00:00:00 2001
From: Andres Freund <andres@anarazel.de>
Date: Tue, 18 Mar 2025 14:40:06 -0400
Subject: [PATCH v2.12 09/28] bufmgr: Use AIO in StartReadBuffers()
This finally introduces the first actual use of AIO. StartReadBuffers() now
uses the AIO routines to issue IO. This converts a lot of callers to use the
AIO infrastructure.
As the implementation of StartReadBuffers() is also used by the functions for
reading individual blocks (StartReadBuffer() and through that
ReadBufferExtended()) this means all buffered read IO passes through the AIO
paths. However, as those are synchronous reads, actually performing the IO
asynchronously would be rarely beneficial. Instead such IOs are flagged to
always be executed synchronously. This way we don't have to duplicate a fair
bit of code.
When io_method=sync is used, the IO patterns generated after this change are
the same as before, i.e. actual reads are only issued in WaitReadBuffers() and
StartReadBuffers() may issue prefetch requests. This allows to bypass most of
the actual asynchronicity, which is important to make a change as big as this
less risky.
One thing worth calling out is that, if IO is actually executed
asynchronously, the precise meaning of what track_io_timing is measuring has
changed. Previously it tracked the time for each IO, but that does not make
sense when multiple IOs are executed concurrently. Now it only measures the
time actually spent waiting for IO. A subsequent commit will adjust the docs
for this.
While AIO is now actually used, the logic in read_stream.c will often prevent
using sufficiently many concurrent IOs. That will be addressed in the next
commit.
Reviewed-by: Noah Misch <noah@leadboat.com>
Reviewed-by: Nazir Bilal Yavuz <byavuz81@gmail.com>
Co-authored-by: Andres Freund <andres@anarazel.de>
Co-authored-by: Thomas Munro <thomas.munro@gmail.com>
Discussion: https://postgr.es/m/uvrtrknj4kdytuboidbhwclo4gxhswwcpgadptsjvjqcluzmah%40brqs62irg4dt
Discussion: https://postgr.es/m/20210223100344.llw5an2aklengrmn@alap3.anarazel.de
Discussion: https://postgr.es/m/stj36ea6yyhoxtqkhpieia2z4krnam7qyetc57rfezgk4zgapf@gcnactj4z56m
---
src/include/storage/bufmgr.h | 6 +
src/backend/storage/buffer/bufmgr.c | 607 +++++++++++++++++++++-------
2 files changed, 473 insertions(+), 140 deletions(-)
diff --git a/src/include/storage/bufmgr.h b/src/include/storage/bufmgr.h
index 5137288c96e..7abb3bdfd9d 100644
--- a/src/include/storage/bufmgr.h
+++ b/src/include/storage/bufmgr.h
@@ -112,6 +112,9 @@ typedef struct BufferManagerRelation
#define READ_BUFFERS_ZERO_ON_ERROR (1 << 0)
/* Call smgrprefetch() if I/O necessary. */
#define READ_BUFFERS_ISSUE_ADVICE (1 << 1)
+/* IO will immediately be waited for */
+#define READ_BUFFERS_SYNCHRONOUSLY (1 << 2)
+
struct ReadBuffersOperation
{
@@ -131,6 +134,9 @@ struct ReadBuffersOperation
BlockNumber blocknum;
int flags;
int16 nblocks;
+ int16 nblocks_done;
+ PgAioWaitRef io_wref;
+ PgAioReturn io_return;
};
typedef struct ReadBuffersOperation ReadBuffersOperation;
diff --git a/src/backend/storage/buffer/bufmgr.c b/src/backend/storage/buffer/bufmgr.c
index 3dad509b4d3..883c5918a33 100644
--- a/src/backend/storage/buffer/bufmgr.c
+++ b/src/backend/storage/buffer/bufmgr.c
@@ -531,6 +531,8 @@ static inline BufferDesc *BufferAlloc(SMgrRelation smgr,
BlockNumber blockNum,
BufferAccessStrategy strategy,
bool *foundPtr, IOContext io_context);
+static bool AsyncReadBuffers(ReadBuffersOperation *operation, int *nblocks);
+static void CheckReadBuffersOperation(ReadBuffersOperation *operation, bool is_complete);
static Buffer GetVictimBuffer(BufferAccessStrategy strategy, IOContext io_context);
static void FlushBuffer(BufferDesc *buf, SMgrRelation reln,
IOObject io_object, IOContext io_context);
@@ -1231,10 +1233,14 @@ ReadBuffer_common(Relation rel, SMgrRelation smgr, char smgr_persistence,
return buffer;
}
+ /*
+ * Signal that we are going to immediately wait. If we're immediately
+ * waiting, there is no benefit in actually executing the IO
+ * asynchronously, it would just add dispatch overhead.
+ */
+ flags = READ_BUFFERS_SYNCHRONOUSLY;
if (mode == RBM_ZERO_ON_ERROR)
- flags = READ_BUFFERS_ZERO_ON_ERROR;
- else
- flags = 0;
+ flags |= READ_BUFFERS_ZERO_ON_ERROR;
operation.smgr = smgr;
operation.rel = rel;
operation.persistence = persistence;
@@ -1259,10 +1265,12 @@ StartReadBuffersImpl(ReadBuffersOperation *operation,
{
int actual_nblocks = *nblocks;
int maxcombine = 0;
+ bool did_start_io;
Assert(*nblocks == 1 || allow_forwarding);
Assert(*nblocks > 0);
Assert(*nblocks <= MAX_IO_COMBINE_LIMIT);
+ Assert(*nblocks == 1 || allow_forwarding);
for (int i = 0; i < actual_nblocks; ++i)
{
@@ -1326,6 +1334,20 @@ StartReadBuffersImpl(ReadBuffersOperation *operation,
if (i == 0)
{
*nblocks = 1;
+
+#ifdef USE_ASSERT_CHECKING
+
+ /*
+ * Initialize enough of ReadBuffersOperation to make
+ * CheckReadBuffersOperation() work. Outside of assertions
+ * that's not necessary when no IO is issued.
+ */
+ operation->buffers = buffers;
+ operation->blocknum = blockNum;
+ operation->nblocks = 1;
+ operation->nblocks_done = 1;
+ CheckReadBuffersOperation(operation, true);
+#endif
return false;
}
@@ -1368,25 +1390,74 @@ StartReadBuffersImpl(ReadBuffersOperation *operation,
operation->blocknum = blockNum;
operation->flags = flags;
operation->nblocks = actual_nblocks;
+ operation->nblocks_done = 0;
+ pgaio_wref_clear(&operation->io_wref);
- if (flags & READ_BUFFERS_ISSUE_ADVICE)
+ /*
+ * When using AIO, start the IO in the background. If not, issue prefetch
+ * requests if desired by the caller.
+ *
+ * The reason we have a dedicated path for IOMETHOD_SYNC here is to
+ * de-risk the introduction of AIO somewhat. It's a large architectural
+ * change, with lots of chances for unanticipated performance effects.
+ *
+ * Use of IOMETHOD_SYNC already leads to not actually performing IO
+ * asynchronously, but without the check here we'd execute IO earlier than
+ * we used to. Eventually this IOMETHOD_SYNC specific path should go away.
+ */
+ if (io_method != IOMETHOD_SYNC)
{
/*
- * In theory we should only do this if PinBufferForBlock() had to
- * allocate new buffers above. That way, if two calls to
- * StartReadBuffers() were made for the same blocks before
- * WaitReadBuffers(), only the first would issue the advice. That'd be
- * a better simulation of true asynchronous I/O, which would only
- * start the I/O once, but isn't done here for simplicity.
+ * Try to start IO asynchronously. It's possible that no IO needs to
+ * be started, if another backend already performed the IO.
+ *
+ * Note that if an IO is started, it might not cover the entire
+ * requested range, e.g. because an intermediary block has been read
+ * in by another backend. In that case any "trailing" buffers we
+ * already pinned above will be "forwarded" by read_stream.c to the
+ * next call to StartReadBuffers().
+ *
+ * This is signalled to the caller by decrementing *nblocks *and*
+ * reducing operation->nblocks. The latter is done here, but not below
+ * WaitReadBuffers(), as in WaitReadBuffers() we can't "shorten" the
+ * overall read size anymore, we need to retry until done in its
+ * entirety or until failed.
*/
- smgrprefetch(operation->smgr,
- operation->forknum,
- blockNum,
- actual_nblocks);
+ did_start_io = AsyncReadBuffers(operation, nblocks);
+
+ operation->nblocks = *nblocks;
+ }
+ else
+ {
+ operation->flags |= READ_BUFFERS_SYNCHRONOUSLY;
+
+ if (flags & READ_BUFFERS_ISSUE_ADVICE)
+ {
+ /*
+ * In theory we should only do this if PinBufferForBlock() had to
+ * allocate new buffers above. That way, if two calls to
+ * StartReadBuffers() were made for the same blocks before
+ * WaitReadBuffers(), only the first would issue the advice.
+ * That'd be a better simulation of true asynchronous I/O, which
+ * would only start the I/O once, but isn't done here for
+ * simplicity.
+ */
+ smgrprefetch(operation->smgr,
+ operation->forknum,
+ blockNum,
+ actual_nblocks);
+ }
+
+ /*
+ * Indicate that WaitReadBuffers() should be called. WaitReadBuffers()
+ * will initiate the necessary IO.
+ */
+ did_start_io = true;
}
- /* Indicate that WaitReadBuffers() should be called. */
- return true;
+ CheckReadBuffersOperation(operation, !did_start_io);
+
+ return did_start_io;
}
/*
@@ -1452,8 +1523,35 @@ StartReadBuffer(ReadBuffersOperation *operation,
return result;
}
+/*
+ * Perform sanity checks on the ReadBuffersOperation.
+ */
+static void
+CheckReadBuffersOperation(ReadBuffersOperation *operation, bool is_complete)
+{
+#ifdef USE_ASSERT_CHECKING
+ Assert(operation->nblocks_done <= operation->nblocks);
+ Assert(!is_complete || operation->nblocks == operation->nblocks_done);
+
+ for (int i = 0; i < operation->nblocks; i++)
+ {
+ Buffer buffer = operation->buffers[i];
+ BufferDesc *buf_hdr = BufferIsLocal(buffer) ?
+ GetLocalBufferDescriptor(-buffer - 1) :
+ GetBufferDescriptor(buffer - 1);
+
+ Assert(BufferGetBlockNumber(buffer) == operation->blocknum + i);
+ Assert(pg_atomic_read_u32(&buf_hdr->state) & BM_TAG_VALID);
+
+ if (i < operation->nblocks_done)
+ Assert(pg_atomic_read_u32(&buf_hdr->state) & BM_VALID);
+ }
+#endif
+}
+
+/* helper for ReadBuffersCanStartIO(), to avoid repetition */
static inline bool
-WaitReadBuffersCanStartIO(Buffer buffer, bool nowait)
+ReadBuffersCanStartIOOnce(Buffer buffer, bool nowait)
{
if (BufferIsLocal(buffer))
return StartLocalBufferIO(GetLocalBufferDescriptor(-buffer - 1),
@@ -1462,31 +1560,243 @@ WaitReadBuffersCanStartIO(Buffer buffer, bool nowait)
return StartBufferIO(GetBufferDescriptor(buffer - 1), true, nowait);
}
-void
-WaitReadBuffers(ReadBuffersOperation *operation)
+/*
+ * Helper for AsyncReadBuffers that tries to get the buffer ready for IO.
+ */
+static inline bool
+ReadBuffersCanStartIO(Buffer buffer, bool nowait)
{
- Buffer *buffers;
- int nblocks;
- BlockNumber blocknum;
- ForkNumber forknum;
- IOContext io_context;
- IOObject io_object;
- char persistence;
-
- /* Find the range of the physical read we need to perform. */
- nblocks = operation->nblocks;
- buffers = &operation->buffers[0];
- blocknum = operation->blocknum;
- forknum = operation->forknum;
- persistence = operation->persistence;
+ /*
+ * If this backend currently has staged IO, we need to submit the pending
+ * IO before waiting for the right to issue IO, to avoid the potential for
+ * deadlocks (and, more commonly, unnecessary delays for other backends).
+ */
+ if (!nowait && pgaio_have_staged())
+ {
+ if (ReadBuffersCanStartIOOnce(buffer, true))
+ return true;
+
+ /*
+ * Unfortunately StartBufferIO() returning false doesn't allow to
+ * distinguish between the buffer already being valid and IO already
+ * being in progress. Since IO already being in progress is quite
+ * rare, this approach seems fine.
+ */
+ pgaio_submit_staged();
+ }
+
+ return ReadBuffersCanStartIOOnce(buffer, nowait);
+}
+
+/*
+ * Helper for WaitReadBuffers() that processes the results of a readv
+ * operation, raising an error if necessary.
+ */
+static void
+ProcessReadBuffersResult(ReadBuffersOperation *operation)
+{
+ PgAioReturn *aio_ret = &operation->io_return;
+ int nblocks = 0;
+
+ Assert(pgaio_wref_valid(&operation->io_wref));
+ Assert(aio_ret->result.status != PGAIO_RS_UNKNOWN);
+
+ /*
+ * SMGR reports the number of blocks successfully read as the result of
+ * the IO operation. Thus we can simply add that to ->nblocks_done.
+ */
+
+ if (likely(aio_ret->result.status == PGAIO_RS_OK))
+ nblocks = aio_ret->result.result;
+ else if (aio_ret->result.status == PGAIO_RS_PARTIAL)
+ {
+ /*
+ * We'll retry, so we just emit a debug message to the server log (or
+ * not even that in prod scenarios).
+ */
+ pgaio_result_report(aio_ret->result, &aio_ret->target_data, DEBUG1);
+ nblocks = aio_ret->result.result;
+
+ elog(DEBUG3, "partial read, will retry");
+
+ }
+ else if (aio_ret->result.status == PGAIO_RS_ERROR)
+ {
+ pgaio_result_report(aio_ret->result, &aio_ret->target_data, ERROR);
+ nblocks = 0; /* silence compiler */
+ }
Assert(nblocks > 0);
Assert(nblocks <= MAX_IO_COMBINE_LIMIT);
+ operation->nblocks_done += nblocks;
+
+ Assert(operation->nblocks_done <= operation->nblocks);
+}
+
+void
+WaitReadBuffers(ReadBuffersOperation *operation)
+{
+ PgAioReturn *aio_ret = &operation->io_return;
+ IOContext io_context;
+ IOObject io_object;
+
+ if (operation->persistence == RELPERSISTENCE_TEMP)
+ {
+ io_context = IOCONTEXT_NORMAL;
+ io_object = IOOBJECT_TEMP_RELATION;
+ }
+ else
+ {
+ io_context = IOContextForStrategy(operation->strategy);
+ io_object = IOOBJECT_RELATION;
+ }
+
+ /*
+ * If we get here without an IO operation having been issued, the
+ * io_method == IOMETHOD_SYNC path must have been used. Otherwise the
+ * caller should not have called WaitReadBuffers().
+ *
+ * In the case of IOMETHOD_SYNC, we start - as we used to before the
+ * introducing of AIO - the IO in WaitReadBuffers(). This is done as part
+ * of the retry logic below, no extra code is required.
+ *
+ * This path is expected to eventually go away.
+ */
+ if (!pgaio_wref_valid(&operation->io_wref) && io_method != IOMETHOD_SYNC)
+ elog(ERROR, "waiting for read operation that didn't read");
+
+ /*
+ * To handle partial reads, and IOMETHOD_SYNC, we re-issue IO until we're
+ * done. We may need multiple retries, not just because we could get
+ * multiple partial reads, but also because some of the remaining
+ * to-be-read buffers may have been read in by other backends, limiting
+ * the IO size.
+ */
+ while (true)
+ {
+ int nblocks;
+
+ CheckReadBuffersOperation(operation, false);
+
+ /*
+ * If there is an IO associated with the operation, we may need to
+ * wait for it.
+ */
+ if (pgaio_wref_valid(&operation->io_wref))
+ {
+ /*
+ * Track the time spent waiting for the IO to complete. As
+ * tracking a wait even if we don't actually need to wait
+ *
+ * a) is not cheap, due to the timestamping overhead
+ *
+ * b) reports some time as waiting, even if we never waited
+ *
+ * we first check if we already know the IO is complete.
+ */
+ if (aio_ret->result.status == PGAIO_RS_UNKNOWN &&
+ !pgaio_wref_check_done(&operation->io_wref))
+ {
+ instr_time io_start = pgstat_prepare_io_time(track_io_timing);
+
+ pgaio_wref_wait(&operation->io_wref);
+
+ /*
+ * The IO operation itself was already counted earlier, in
+ * AsyncReadBuffers(), this just accounts for the wait time.
+ */
+ pgstat_count_io_op_time(io_object, io_context, IOOP_READ,
+ io_start, 0, 0);
+ }
+ else
+ {
+ Assert(pgaio_wref_check_done(&operation->io_wref));
+ }
+
+ /*
+ * We now are sure the IO completed. Check the results. This
+ * includes reporting on errors if there were any.
+ */
+ ProcessReadBuffersResult(operation);
+ }
+
+ /*
+ * Most of the time, the one IO we already started, will read in
+ * everything. But we need to deal with partial reads and buffers not
+ * needing IO anymore.
+ */
+ if (operation->nblocks_done == operation->nblocks)
+ break;
+
+ CHECK_FOR_INTERRUPTS();
+
+ /*
+ * This may only complete the IO partially, either because some
+ * buffers were already valid, or because of a partial read.
+ *
+ * NB: In contrast to after the AsyncReadBuffers() call in
+ * StartReadBuffers(), we do *not* reduce
+ * ReadBuffersOperation->nblocks here, callers expect the full
+ * operation to be completed at this point (as more operations may
+ * have been queued).
+ */
+ AsyncReadBuffers(operation, &nblocks);
+ }
+
+ CheckReadBuffersOperation(operation, true);
+
+ /* NB: READ_DONE tracepoint was already executed in completion callback */
+}
+
+/*
+ * Initiate IO for the ReadBuffersOperation
+ *
+ * This function only starts a single IO at a time. The size of the IO may be
+ * limited to below the to-be-read blocks, if one of the buffers has
+ * concurrently been read in. If the first to-be-read buffer is already valid,
+ * no IO will be issued.
+ *
+ * To support retries after partial reads, the first operation->nblocks_done
+ * buffers are skipped.
+ *
+ * On return *nblocks_progress is updated to reflect the number of buffers
+ * affected by the call. If the first buffer is valid, *nblocks_progress is
+ * set to 1 and operation->nblocks_done is incremented.
+ *
+ * Returns true if IO was initiated, false if no IO was necessary.
+ */
+static bool
+AsyncReadBuffers(ReadBuffersOperation *operation, int *nblocks_progress)
+{
+ Buffer *buffers = &operation->buffers[0];
+ int flags = operation->flags;
+ BlockNumber blocknum = operation->blocknum;
+ ForkNumber forknum = operation->forknum;
+ char persistence = operation->persistence;
+ int16 nblocks_done = operation->nblocks_done;
+ Buffer *io_buffers = &operation->buffers[nblocks_done];
+ int io_buffers_len = 0;
+ PgAioHandle *ioh;
+ uint32 ioh_flags = 0;
+ void *io_pages[MAX_IO_COMBINE_LIMIT];
+ IOContext io_context;
+ IOObject io_object;
+ bool did_start_io;
+
+ /*
+ * When this IO is executed synchronously, either because the caller will
+ * immediately block waiting for the IO or because IOMETHOD_SYNC is used,
+ * the AIO subsystem needs to know.
+ */
+ if (flags & READ_BUFFERS_SYNCHRONOUSLY)
+ ioh_flags |= PGAIO_HF_SYNCHRONOUS;
+
if (persistence == RELPERSISTENCE_TEMP)
{
io_context = IOCONTEXT_NORMAL;
io_object = IOOBJECT_TEMP_RELATION;
+ ioh_flags |= PGAIO_HF_REFERENCES_LOCAL;
}
else
{
@@ -1494,140 +1804,157 @@ WaitReadBuffers(ReadBuffersOperation *operation)
io_object = IOOBJECT_RELATION;
}
- for (int i = 0; i < nblocks; ++i)
+ /*
+ * Get IO handle before ReadBuffersCanStartIO(), as pgaio_io_acquire()
+ * might block, which we don't want after setting IO_IN_PROGRESS.
+ *
+ * If we need to wait for IO before we can get a handle, submit already
+ * staged IO first, so that other backends don't need to wait. There
+ * wouldn't be a deadlock risk, as pgaio_io_acquire() just needs to wait
+ * for already submitted IO, which doesn't require additional locks, but
+ * it could still cause undesirable waits.
+ *
+ * A secondary benefit is that this would allow us to measure the time in
+ * pgaio_io_acquire() without causing undue timer overhead in the common,
+ * non-blocking, case. However, currently the pgstats infrastructure
+ * doesn't really allow that, as it a) asserts that an operation can't
+ * have time without operations b) doesn't have an API to report
+ * "accumulated" time.
+ */
+ ioh = pgaio_io_acquire_nb(CurrentResourceOwner, &operation->io_return);
+ if (unlikely(!ioh))
+ {
+ pgaio_submit_staged();
+
+ ioh = pgaio_io_acquire(CurrentResourceOwner, &operation->io_return);
+ }
+
+ /*
+ * Check if we can start IO on the first to-be-read buffer.
+ *
+ * If an I/O is already in progress in another backend, we want to wait
+ * for the outcome: either done, or something went wrong and we will
+ * retry.
+ */
+ if (!ReadBuffersCanStartIO(buffers[nblocks_done], false))
+ {
+ /*
+ * Someone else has already completed this block, we're done.
+ *
+ * When IO is necessary, ->nblocks_done is updated in
+ * ProcessReadBuffersResult(), but that is not called if no IO is
+ * necessary. Thus update here.
+ */
+ operation->nblocks_done += 1;
+ *nblocks_progress = 1;
+
+ pgaio_io_release(ioh);
+ pgaio_wref_clear(&operation->io_wref);
+ did_start_io = false;
+
+ /*
+ * Report and track this as a 'hit' for this backend, even though it
+ * must have started out as a miss in PinBufferForBlock(). The other
+ * backend will track this as a 'read'.
+ */
+ TRACE_POSTGRESQL_BUFFER_READ_DONE(forknum, blocknum + operation->nblocks_done,
+ operation->smgr->smgr_rlocator.locator.spcOid,
+ operation->smgr->smgr_rlocator.locator.dbOid,
+ operation->smgr->smgr_rlocator.locator.relNumber,
+ operation->smgr->smgr_rlocator.backend,
+ true);
+
+ if (persistence == RELPERSISTENCE_TEMP)
+ pgBufferUsage.local_blks_hit += 1;
+ else
+ pgBufferUsage.shared_blks_hit += 1;
+
+ if (operation->rel)
+ pgstat_count_buffer_hit(operation->rel);
+
+ pgstat_count_io_op(io_object, io_context, IOOP_HIT, 1, 0);
+
+ if (VacuumCostActive)
+ VacuumCostBalance += VacuumCostPageHit;
+ }
+ else
{
- int io_buffers_len;
- Buffer io_buffers[MAX_IO_COMBINE_LIMIT];
- void *io_pages[MAX_IO_COMBINE_LIMIT];
instr_time io_start;
- BlockNumber io_first_block;
-
- /*
- * Skip this block if someone else has already completed it. If an
- * I/O is already in progress in another backend, this will wait for
- * the outcome: either done, or something went wrong and we will
- * retry.
- */
- if (!WaitReadBuffersCanStartIO(buffers[i], false))
- {
- /*
- * Report and track this as a 'hit' for this backend, even though
- * it must have started out as a miss in PinBufferForBlock(). The
- * other backend will track this as a 'read'.
- */
- TRACE_POSTGRESQL_BUFFER_READ_DONE(forknum, blocknum + i,
- operation->smgr->smgr_rlocator.locator.spcOid,
- operation->smgr->smgr_rlocator.locator.dbOid,
- operation->smgr->smgr_rlocator.locator.relNumber,
- operation->smgr->smgr_rlocator.backend,
- true);
-
- if (persistence == RELPERSISTENCE_TEMP)
- pgBufferUsage.local_blks_hit += 1;
- else
- pgBufferUsage.shared_blks_hit += 1;
-
- if (operation->rel)
- pgstat_count_buffer_hit(operation->rel);
-
- pgstat_count_io_op(io_object, io_context, IOOP_HIT, 1, 0);
-
- if (VacuumCostActive)
- VacuumCostBalance += VacuumCostPageHit;
-
- continue;
- }
/* We found a buffer that we need to read in. */
- io_buffers[0] = buffers[i];
- io_pages[0] = BufferGetBlock(buffers[i]);
- io_first_block = blocknum + i;
+ Assert(io_buffers[0] == buffers[nblocks_done]);
+ io_pages[0] = BufferGetBlock(buffers[nblocks_done]);
io_buffers_len = 1;
/*
* How many neighboring-on-disk blocks can we scatter-read into other
* buffers at the same time? In this case we don't wait if we see an
- * I/O already in progress. We already hold BM_IO_IN_PROGRESS for the
+ * I/O already in progress. We already set BM_IO_IN_PROGRESS for the
* head block, so we should get on with that I/O as soon as possible.
- * We'll come back to this block again, above.
+ *
+ * We'll come back to this block in the next call to
+ * StartReadBuffers() -> AsyncReadBuffers().
*/
- while ((i + 1) < nblocks &&
- WaitReadBuffersCanStartIO(buffers[i + 1], true))
+ for (int i = nblocks_done + 1; i < operation->nblocks; i++)
{
+ if (!ReadBuffersCanStartIO(buffers[i], true))
+ break;
/* Must be consecutive block numbers. */
- Assert(BufferGetBlockNumber(buffers[i + 1]) ==
- BufferGetBlockNumber(buffers[i]) + 1);
+ Assert(BufferGetBlockNumber(buffers[i - 1]) ==
+ BufferGetBlockNumber(buffers[i]) - 1);
+ Assert(io_buffers[io_buffers_len] == buffers[i]);
- io_buffers[io_buffers_len] = buffers[++i];
io_pages[io_buffers_len++] = BufferGetBlock(buffers[i]);
}
+ /* get a reference to wait for in WaitReadBuffers() */
+ pgaio_io_get_wref(ioh, &operation->io_wref);
+
+ /* provide the list of buffers to the completion callbacks */
+ pgaio_io_set_handle_data_32(ioh, (uint32 *) io_buffers, io_buffers_len);
+
+ pgaio_io_register_callbacks(ioh,
+ persistence == RELPERSISTENCE_TEMP ?
+ PGAIO_HCB_LOCAL_BUFFER_READV :
+ PGAIO_HCB_SHARED_BUFFER_READV,
+ flags);
+
+ pgaio_io_set_flag(ioh, ioh_flags);
+
+ /* ---
+ * Even though we're trying to issue IO asynchronously, track the time
+ * in smgrstartreadv():
+ * - if io_method == IOMETHOD_SYNC, we will always perform the IO
+ * immediately
+ * - the io method might not support the IO (e.g. worker IO for a temp
+ * table)
+ * ---
+ */
io_start = pgstat_prepare_io_time(track_io_timing);
- smgrreadv(operation->smgr, forknum, io_first_block, io_pages, io_buffers_len);
- pgstat_count_io_op_time(io_object, io_context, IOOP_READ, io_start,
- 1, io_buffers_len * BLCKSZ);
-
- /* Verify each block we read, and terminate the I/O. */
- for (int j = 0; j < io_buffers_len; ++j)
- {
- BufferDesc *bufHdr;
- Block bufBlock;
-
- if (persistence == RELPERSISTENCE_TEMP)
- {
- bufHdr = GetLocalBufferDescriptor(-io_buffers[j] - 1);
- bufBlock = LocalBufHdrGetBlock(bufHdr);
- }
- else
- {
- bufHdr = GetBufferDescriptor(io_buffers[j] - 1);
- bufBlock = BufHdrGetBlock(bufHdr);
- }
-
- /* check for garbage data */
- if (!PageIsVerifiedExtended((Page) bufBlock, io_first_block + j,
- PIV_LOG_WARNING | PIV_REPORT_STAT))
- {
- if ((operation->flags & READ_BUFFERS_ZERO_ON_ERROR) || zero_damaged_pages)
- {
- ereport(WARNING,
- (errcode(ERRCODE_DATA_CORRUPTED),
- errmsg("invalid page in block %u of relation %s; zeroing out page",
- io_first_block + j,
- relpath(operation->smgr->smgr_rlocator, forknum).str)));
- memset(bufBlock, 0, BLCKSZ);
- }
- else
- ereport(ERROR,
- (errcode(ERRCODE_DATA_CORRUPTED),
- errmsg("invalid page in block %u of relation %s",
- io_first_block + j,
- relpath(operation->smgr->smgr_rlocator, forknum).str)));
- }
-
- /* Set BM_VALID, terminate IO, and wake up any waiters */
- if (persistence == RELPERSISTENCE_TEMP)
- TerminateLocalBufferIO(bufHdr, false, BM_VALID, true);
- else
- TerminateBufferIO(bufHdr, false, BM_VALID, true, true);
-
- /* Report I/Os as completing individually. */
- TRACE_POSTGRESQL_BUFFER_READ_DONE(forknum, io_first_block + j,
- operation->smgr->smgr_rlocator.locator.spcOid,
- operation->smgr->smgr_rlocator.locator.dbOid,
- operation->smgr->smgr_rlocator.locator.relNumber,
- operation->smgr->smgr_rlocator.backend,
- false);
- }
+ smgrstartreadv(ioh, operation->smgr, forknum,
+ blocknum + nblocks_done,
+ io_pages, io_buffers_len);
+ pgstat_count_io_op_time(io_object, io_context, IOOP_READ,
+ io_start, 1, *nblocks_progress * BLCKSZ);
if (persistence == RELPERSISTENCE_TEMP)
pgBufferUsage.local_blks_read += io_buffers_len;
else
pgBufferUsage.shared_blks_read += io_buffers_len;
+ /*
+ * Track vacuum cost when issuing IO, not after waiting for it.
+ * Otherwise we could end up issuing a lot of IO in a short timespan,
+ * despite a low cost limit.
+ */
if (VacuumCostActive)
VacuumCostBalance += VacuumCostPageMiss * io_buffers_len;
+
+ *nblocks_progress = io_buffers_len;
+ did_start_io = true;
}
+
+ return did_start_io;
}
/*
--
2.48.1.76.g4e746b1a31.dirty
[text/x-diff] v2.12-0010-aio-Basic-read_stream-adjustments-for-real-AIO.patch (2.8K, ../../cxi6ssym62v7ywkt367qda3xik5jhsrcycwedyfqacinxooys5@pwarur37bosc/11-v2.12-0010-aio-Basic-read_stream-adjustments-for-real-AIO.patch)
download | inline diff:
From 194a0c62a14cac960e046f566a337115adf82cda Mon Sep 17 00:00:00 2001
From: Andres Freund <andres@anarazel.de>
Date: Tue, 18 Mar 2025 14:40:06 -0400
Subject: [PATCH v2.12 10/28] aio: Basic read_stream adjustments for real AIO
Adapt the read stream logic for real AIO:
- If AIO is enabled, we shouldn't issue advice, but if it isn't, we should
continue issuing advice
- AIO benefits from reading ahead with direct IO
There is one comment talking about max_ios logic with "real asynchronous I/O"
that I am not sure about, so I left it alone for now.
There are further improvements we should consider:
- While in read_stream_look_ahead(), we can use AIO batch submission mode for
increased efficiency
- We could wait to issue IOs until we can issue multiple IOs at once
---
src/backend/storage/aio/read_stream.c | 19 ++++++++++++++-----
1 file changed, 14 insertions(+), 5 deletions(-)
diff --git a/src/backend/storage/aio/read_stream.c b/src/backend/storage/aio/read_stream.c
index c60e37e7f7f..df16530d673 100644
--- a/src/backend/storage/aio/read_stream.c
+++ b/src/backend/storage/aio/read_stream.c
@@ -72,6 +72,7 @@
#include "postgres.h"
#include "miscadmin.h"
+#include "storage/aio.h"
#include "storage/fd.h"
#include "storage/smgr.h"
#include "storage/read_stream.h"
@@ -99,6 +100,7 @@ struct ReadStream
int16 pinned_buffers;
int16 distance;
int16 initialized_buffers;
+ bool sync_mode; /* using io_method=sync */
bool advice_enabled;
bool temporary;
@@ -613,15 +615,19 @@ read_stream_begin_impl(int flags,
stream->per_buffer_data = (void *)
MAXALIGN(&stream->ios[Max(1, max_ios)]);
+ stream->sync_mode = io_method == IOMETHOD_SYNC;
+
#ifdef USE_PREFETCH
/*
- * This system supports prefetching advice. We can use it as long as
- * direct I/O isn't enabled, the caller hasn't promised sequential access
- * (overriding our detection heuristics), and max_ios hasn't been set to
- * zero.
+ * This system supports prefetching advice.
+ *
+ * Issue advice only if AIO is not used, direct I/O isn't enabled, the
+ * caller hasn't promised sequential access (overriding our detection
+ * heuristics), and max_ios hasn't been set to zero.
*/
- if ((io_direct_flags & IO_DIRECT_DATA) == 0 &&
+ if (stream->sync_mode &&
+ (io_direct_flags & IO_DIRECT_DATA) == 0 &&
(flags & READ_STREAM_SEQUENTIAL) == 0 &&
max_ios > 0)
stream->advice_enabled = true;
@@ -631,6 +637,9 @@ read_stream_begin_impl(int flags,
* For now, max_ios = 0 is interpreted as max_ios = 1 with advice disabled
* above. If we had real asynchronous I/O we might need a slightly
* different definition.
+ *
+ * FIXME: Not sure what different definition we would need? I guess we
+ * could add the READ_BUFFERS_SYNCHRONOUSLY flag automatically?
*/
if (max_ios == 0)
max_ios = 1;
--
2.48.1.76.g4e746b1a31.dirty
[text/x-diff] v2.12-0011-read_stream-Introduce-and-use-optional-batchmo.patch (12.2K, ../../cxi6ssym62v7ywkt367qda3xik5jhsrcycwedyfqacinxooys5@pwarur37bosc/12-v2.12-0011-read_stream-Introduce-and-use-optional-batchmo.patch)
download | inline diff:
From f8bdb4f465c80a59bb25fa737de290053d9d8239 Mon Sep 17 00:00:00 2001
From: Andres Freund <andres@anarazel.de>
Date: Mon, 24 Mar 2025 17:30:42 -0400
Subject: [PATCH v2.12 11/28] read_stream: Introduce and use optional batchmode
support
Submitting IO in larger batches can be more efficient than doing so
one-by-one, particularly for many small reads. It does, however, require
the ReadStreamBlockNumberCB callback to abide by the restrictions of AIO
batching (c.f. pgaio_enter_batchmode()). Basically, the callback may not:
a) block without first calling pgaio_submit_staged(), unless a
to-be-waited-on lock cannot be part of a deadlock, e.g. because it is
never held while waiting for IO.
b) directly or indirectly start another batch pgaio_enter_batchmode()
As this requires care and is nontrivial in some cases, batching is only
used with explicit opt-in.
This patch adds an explicit flag (READ_STREAM_USE_BATCHING) to read_stream and
uses it where appropriate.
There are two cases where batching would likely be beneficial, but where we
aren't using it yet:
1) bitmap heap scans, because the callback reads the VM
This should soon be solved, because we are planning to remove the use of
the VM, due to that not being sound.
2) The first phase of heap vacuum
This could be made to support batchmode, but would require some care.
Discussion: https://postgr.es/m/uvrtrknj4kdytuboidbhwclo4gxhswwcpgadptsjvjqcluzmah%40brqs62irg4dt
---
src/include/storage/read_stream.h | 19 +++++++++++++++++++
src/backend/access/gist/gistvacuum.c | 8 +++++++-
src/backend/access/heap/heapam.c | 18 +++++++++++++++++-
src/backend/access/heap/vacuumlazy.c | 21 ++++++++++++++++++---
src/backend/access/nbtree/nbtree.c | 8 +++++++-
src/backend/access/spgist/spgvacuum.c | 8 +++++++-
src/backend/commands/analyze.c | 7 ++++++-
src/backend/storage/aio/read_stream.c | 16 ++++++++++++++++
contrib/pg_prewarm/pg_prewarm.c | 7 ++++++-
contrib/pg_visibility/pg_visibility.c | 8 +++++++-
10 files changed, 110 insertions(+), 10 deletions(-)
diff --git a/src/include/storage/read_stream.h b/src/include/storage/read_stream.h
index c11d8ce3300..18761ac7df5 100644
--- a/src/include/storage/read_stream.h
+++ b/src/include/storage/read_stream.h
@@ -42,6 +42,25 @@
*/
#define READ_STREAM_FULL 0x04
+/* ---
+ * Opt-in to using AIO batchmode.
+ *
+ * Submitting IO in larger batches can be more efficient than doing so
+ * one-by-one, particularly for many small reads. It does, however, require
+ * the ReadStreamBlockNumberCB callback to abide by the restrictions of AIO
+ * batching (c.f. pgaio_enter_batchmode()). Basically, the callback may not:
+ * a) block without first calling pgaio_submit_staged(), unless a
+ * to-be-waited-on lock cannot be part of a deadlock, e.g. because it is
+ * never held while waiting for IO.
+ *
+ * b) directly or indirectly start another batch pgaio_enter_batchmode()
+ *
+ * As this requires care and is nontrivial in some cases, batching is only
+ * used with explicit opt-in.
+ * ---
+ */
+#define READ_STREAM_USE_BATCHING 0x08
+
struct ReadStream;
typedef struct ReadStream ReadStream;
diff --git a/src/backend/access/gist/gistvacuum.c b/src/backend/access/gist/gistvacuum.c
index 20b1bb5dbac..ce9d78d78d6 100644
--- a/src/backend/access/gist/gistvacuum.c
+++ b/src/backend/access/gist/gistvacuum.c
@@ -210,7 +210,13 @@ gistvacuumscan(IndexVacuumInfo *info, IndexBulkDeleteResult *stats,
needLock = !RELATION_IS_LOCAL(rel);
p.current_blocknum = GIST_ROOT_BLKNO;
- stream = read_stream_begin_relation(READ_STREAM_FULL,
+
+ /*
+ * It is safe to use batchmode as block_range_read_stream_cb takes no
+ * locks.
+ */
+ stream = read_stream_begin_relation(READ_STREAM_FULL |
+ READ_STREAM_USE_BATCHING,
info->strategy,
rel,
MAIN_FORKNUM,
diff --git a/src/backend/access/heap/heapam.c b/src/backend/access/heap/heapam.c
index b12b583c4d9..c8357660776 100644
--- a/src/backend/access/heap/heapam.c
+++ b/src/backend/access/heap/heapam.c
@@ -1206,7 +1206,15 @@ heap_beginscan(Relation relation, Snapshot snapshot,
else
cb = heap_scan_stream_read_next_serial;
- scan->rs_read_stream = read_stream_begin_relation(READ_STREAM_SEQUENTIAL,
+ /* ---
+ * It is safe to use batchmode as the only locks taken by `cb`
+ * are never taken while waiting for IO:
+ * - SyncScanLock is used in the non-parallel case
+ * - in the parallel case, only spinlocks and atomics are used
+ * ---
+ */
+ scan->rs_read_stream = read_stream_begin_relation(READ_STREAM_SEQUENTIAL |
+ READ_STREAM_USE_BATCHING,
scan->rs_strategy,
scan->rs_base.rs_rd,
MAIN_FORKNUM,
@@ -1216,6 +1224,14 @@ heap_beginscan(Relation relation, Snapshot snapshot,
}
else if (scan->rs_base.rs_flags & SO_TYPE_BITMAPSCAN)
{
+ /*
+ * Currently we can't trivially use batching, due to the
+ * VM_ALL_VISIBLE check in bitmapheap_stream_read_next. While that
+ * could be made safe, we are about to remove the all-visible logic
+ * from bitmap scans due to its unsoundness.
+ *
+ * FIXME: Should be changed soon!
+ */
scan->rs_read_stream = read_stream_begin_relation(READ_STREAM_DEFAULT,
scan->rs_strategy,
scan->rs_base.rs_rd,
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index 2cbcf5e5db2..82883492128 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -1225,7 +1225,12 @@ lazy_scan_heap(LVRelState *vacrel)
vacrel->next_unskippable_eager_scanned = false;
vacrel->next_unskippable_vmbuffer = InvalidBuffer;
- /* Set up the read stream for vacuum's first pass through the heap */
+ /*
+ * Set up the read stream for vacuum's first pass through the heap.
+ *
+ * This could be made safe for READ_STREAM_USE_BATCHING, but only with
+ * explicit work in heap_vac_scan_next_block.
+ */
stream = read_stream_begin_relation(READ_STREAM_MAINTENANCE,
vacrel->bstrategy,
vacrel->rel,
@@ -2669,6 +2674,8 @@ lazy_vacuum_all_indexes(LVRelState *vacrel)
* Read stream callback for vacuum's third phase (second pass over the heap).
* Gets the next block from the TID store and returns it or InvalidBlockNumber
* if there are no further blocks to vacuum.
+ *
+ * NB: Assumed to be safe to use with READ_STREAM_USE_BATCHING.
*/
static BlockNumber
vacuum_reap_lp_read_stream_next(ReadStream *stream,
@@ -2732,8 +2739,16 @@ lazy_vacuum_heap_rel(LVRelState *vacrel)
iter = TidStoreBeginIterate(vacrel->dead_items);
- /* Set up the read stream for vacuum's second pass through the heap */
- stream = read_stream_begin_relation(READ_STREAM_MAINTENANCE,
+ /*
+ * Set up the read stream for vacuum's second pass through the heap.
+ *
+ * It is safe to use batchmode, as vacuum_reap_lp_read_stream_next() does
+ * not need to wait for IO and does not perform locking. Once we support
+ * parallelism it should still be fine, as presumably the holder of locks
+ * would never be blocked by IO while holding the lock.
+ */
+ stream = read_stream_begin_relation(READ_STREAM_MAINTENANCE |
+ READ_STREAM_USE_BATCHING,
vacrel->bstrategy,
vacrel->rel,
MAIN_FORKNUM,
diff --git a/src/backend/access/nbtree/nbtree.c b/src/backend/access/nbtree/nbtree.c
index 80b04d6ca2a..4a0bf069f99 100644
--- a/src/backend/access/nbtree/nbtree.c
+++ b/src/backend/access/nbtree/nbtree.c
@@ -1064,7 +1064,13 @@ btvacuumscan(IndexVacuumInfo *info, IndexBulkDeleteResult *stats,
needLock = !RELATION_IS_LOCAL(rel);
p.current_blocknum = BTREE_METAPAGE + 1;
- stream = read_stream_begin_relation(READ_STREAM_FULL,
+
+ /*
+ * It is safe to use batchmode as block_range_read_stream_cb takes no
+ * locks.
+ */
+ stream = read_stream_begin_relation(READ_STREAM_FULL |
+ READ_STREAM_USE_BATCHING,
info->strategy,
rel,
MAIN_FORKNUM,
diff --git a/src/backend/access/spgist/spgvacuum.c b/src/backend/access/spgist/spgvacuum.c
index 77deb226b7e..b3df2d89074 100644
--- a/src/backend/access/spgist/spgvacuum.c
+++ b/src/backend/access/spgist/spgvacuum.c
@@ -822,7 +822,13 @@ spgvacuumscan(spgBulkDeleteState *bds)
/* We can skip locking for new or temp relations */
needLock = !RELATION_IS_LOCAL(index);
p.current_blocknum = SPGIST_METAPAGE_BLKNO + 1;
- stream = read_stream_begin_relation(READ_STREAM_FULL,
+
+ /*
+ * It is safe to use batchmode as block_range_read_stream_cb takes no
+ * locks.
+ */
+ stream = read_stream_begin_relation(READ_STREAM_FULL |
+ READ_STREAM_USE_BATCHING,
bds->info->strategy,
index,
MAIN_FORKNUM,
diff --git a/src/backend/commands/analyze.c b/src/backend/commands/analyze.c
index 2b5fbdcbd82..fbf618e6abb 100644
--- a/src/backend/commands/analyze.c
+++ b/src/backend/commands/analyze.c
@@ -1237,7 +1237,12 @@ acquire_sample_rows(Relation onerel, int elevel,
scan = table_beginscan_analyze(onerel);
slot = table_slot_create(onerel, NULL);
- stream = read_stream_begin_relation(READ_STREAM_MAINTENANCE,
+ /*
+ * It is safe to use batching, as block_sampling_read_stream_next never
+ * blocks.
+ */
+ stream = read_stream_begin_relation(READ_STREAM_MAINTENANCE |
+ READ_STREAM_USE_BATCHING,
vac_strategy,
scan->rs_rd,
MAIN_FORKNUM,
diff --git a/src/backend/storage/aio/read_stream.c b/src/backend/storage/aio/read_stream.c
index df16530d673..f675168e89a 100644
--- a/src/backend/storage/aio/read_stream.c
+++ b/src/backend/storage/aio/read_stream.c
@@ -101,6 +101,7 @@ struct ReadStream
int16 distance;
int16 initialized_buffers;
bool sync_mode; /* using io_method=sync */
+ bool batch_mode; /* READ_STREAM_USE_BATCHING */
bool advice_enabled;
bool temporary;
@@ -402,6 +403,15 @@ read_stream_start_pending_read(ReadStream *stream)
static void
read_stream_look_ahead(ReadStream *stream)
{
+ /*
+ * Allow amortizing the cost of submitting IO over multiple IOs. This
+ * requires that we don't do any operations that could lead to a deadlock
+ * with staged-but-unsubmitted IO. The callback needs to opt-in to being
+ * careful.
+ */
+ if (stream->batch_mode)
+ pgaio_enter_batchmode();
+
while (stream->ios_in_progress < stream->max_ios &&
stream->pinned_buffers + stream->pending_read_nblocks < stream->distance)
{
@@ -449,6 +459,8 @@ read_stream_look_ahead(ReadStream *stream)
{
/* We've hit the buffer or I/O limit. Rewind and stop here. */
read_stream_unget_block(stream, blocknum);
+ if (stream->batch_mode)
+ pgaio_exit_batchmode();
return;
}
}
@@ -483,6 +495,9 @@ read_stream_look_ahead(ReadStream *stream)
* time.
*/
Assert(stream->pinned_buffers > 0 || stream->distance == 0);
+
+ if (stream->batch_mode)
+ pgaio_exit_batchmode();
}
/*
@@ -616,6 +631,7 @@ read_stream_begin_impl(int flags,
MAXALIGN(&stream->ios[Max(1, max_ios)]);
stream->sync_mode = io_method == IOMETHOD_SYNC;
+ stream->batch_mode = flags & READ_STREAM_USE_BATCHING;
#ifdef USE_PREFETCH
diff --git a/contrib/pg_prewarm/pg_prewarm.c b/contrib/pg_prewarm/pg_prewarm.c
index a2f0ac4af0c..05ea40c793a 100644
--- a/contrib/pg_prewarm/pg_prewarm.c
+++ b/contrib/pg_prewarm/pg_prewarm.c
@@ -195,7 +195,12 @@ pg_prewarm(PG_FUNCTION_ARGS)
p.current_blocknum = first_block;
p.last_exclusive = last_block + 1;
- stream = read_stream_begin_relation(READ_STREAM_FULL,
+ /*
+ * It is safe to use batchmode as block_range_read_stream_cb takes no
+ * locks.
+ */
+ stream = read_stream_begin_relation(READ_STREAM_FULL |
+ READ_STREAM_USE_BATCHING,
NULL,
rel,
forkNumber,
diff --git a/contrib/pg_visibility/pg_visibility.c b/contrib/pg_visibility/pg_visibility.c
index 7f268a18a74..8a9fbb5f1b8 100644
--- a/contrib/pg_visibility/pg_visibility.c
+++ b/contrib/pg_visibility/pg_visibility.c
@@ -523,7 +523,13 @@ collect_visibility_data(Oid relid, bool include_pd)
{
p.current_blocknum = 0;
p.last_exclusive = nblocks;
- stream = read_stream_begin_relation(READ_STREAM_FULL,
+
+ /*
+ * It is safe to use batchmode as block_range_read_stream_cb takes no
+ * locks.
+ */
+ stream = read_stream_begin_relation(READ_STREAM_FULL |
+ READ_STREAM_USE_BATCHING,
bstrategy,
rel,
MAIN_FORKNUM,
--
2.48.1.76.g4e746b1a31.dirty
[text/x-diff] v2.12-0012-docs-Reframe-track_io_timing-related-docs-as-w.patch (5.3K, ../../cxi6ssym62v7ywkt367qda3xik5jhsrcycwedyfqacinxooys5@pwarur37bosc/13-v2.12-0012-docs-Reframe-track_io_timing-related-docs-as-w.patch)
download | inline diff:
From 156355b98c690d2193dfe4637a7c73faa0b3f56f Mon Sep 17 00:00:00 2001
From: Andres Freund <andres@anarazel.de>
Date: Tue, 18 Mar 2025 14:40:06 -0400
Subject: [PATCH v2.12 12/28] docs: Reframe track_io_timing related docs as
wait time
With AIO it does not make sense anymore to track the time for each individual
IO, as multiple IOs can be in-flight at the same time. Instead we now track
the time spent *waiting* for IOs.
This should be reflected in the docs. While, so far, we only do a subset of
reads, and no other operations, via AIO, describing the GUC and view columns
as measuring IO waits is accurate for synchronous and asynchronous IO.
Discussion: https://postgr.es/m/5dzyoduxlvfg55oqtjyjehez5uoq6hnwgzor4kkybkfdgkj7ag@rbi4gsmzaczk
---
doc/src/sgml/config.sgml | 4 ++--
doc/src/sgml/monitoring.sgml | 12 ++++++------
2 files changed, 8 insertions(+), 8 deletions(-)
diff --git a/doc/src/sgml/config.sgml b/doc/src/sgml/config.sgml
index f86135fbe1d..13773c05ff4 100644
--- a/doc/src/sgml/config.sgml
+++ b/doc/src/sgml/config.sgml
@@ -8568,7 +8568,7 @@ COPY postgres_log FROM '/full/path/to/logfile.csv' WITH csv;
</term>
<listitem>
<para>
- Enables timing of database I/O calls. This parameter is off by
+ Enables timing of database I/O waits. This parameter is off by
default, as it will repeatedly query the operating system for
the current time, which may cause significant overhead on some
platforms. You can use the <xref linkend="pgtesttiming"/> tool to
@@ -8602,7 +8602,7 @@ COPY postgres_log FROM '/full/path/to/logfile.csv' WITH csv;
</term>
<listitem>
<para>
- Enables timing of WAL I/O calls. This parameter is off by default,
+ Enables timing of WAL I/O waits. This parameter is off by default,
as it will repeatedly query the operating system for the current time,
which may cause significant overhead on some platforms.
You can use the <application>pg_test_timing</application> tool to
diff --git a/doc/src/sgml/monitoring.sgml b/doc/src/sgml/monitoring.sgml
index 0960f5ba94a..3b6df434c85 100644
--- a/doc/src/sgml/monitoring.sgml
+++ b/doc/src/sgml/monitoring.sgml
@@ -2745,7 +2745,7 @@ description | Waiting for a newly initialized WAL file to reach durable storage
<structfield>read_time</structfield> <type>double precision</type>
</para>
<para>
- Time spent in read operations in milliseconds (if
+ Time spent waiting for read operations in milliseconds (if
<xref linkend="guc-track-io-timing"/> is enabled and
<varname>object</varname> is not <literal>wal</literal>,
or if <xref linkend="guc-track-wal-io-timing"/> is enabled
@@ -2783,7 +2783,7 @@ description | Waiting for a newly initialized WAL file to reach durable storage
<structfield>write_time</structfield> <type>double precision</type>
</para>
<para>
- Time spent in write operations in milliseconds (if
+ Time spent waiting for write operations in milliseconds (if
<xref linkend="guc-track-io-timing"/> is enabled and
<varname>object</varname> is not <literal>wal</literal>,
or if <xref linkend="guc-track-wal-io-timing"/> is enabled
@@ -2811,7 +2811,7 @@ description | Waiting for a newly initialized WAL file to reach durable storage
<structfield>writeback_time</structfield> <type>double precision</type>
</para>
<para>
- Time spent in writeback operations in milliseconds (if
+ Time spent waiting for writeback operations in milliseconds (if
<xref linkend="guc-track-io-timing"/> is enabled, otherwise zero). This
includes the time spent queueing write-out requests and, potentially,
the time spent to write out the dirty data.
@@ -2847,7 +2847,7 @@ description | Waiting for a newly initialized WAL file to reach durable storage
<structfield>extend_time</structfield> <type>double precision</type>
</para>
<para>
- Time spent in extend operations in milliseconds. (if
+ Time spent waiting for extend operations in milliseconds. (if
<xref linkend="guc-track-io-timing"/> is enabled and
<varname>object</varname> is not <literal>wal</literal>,
or if <xref linkend="guc-track-wal-io-timing"/> is enabled
@@ -2921,7 +2921,7 @@ description | Waiting for a newly initialized WAL file to reach durable storage
<structfield>fsync_time</structfield> <type>double precision</type>
</para>
<para>
- Time spent in fsync operations in milliseconds (if
+ Time spent waiting for fsync operations in milliseconds (if
<xref linkend="guc-track-io-timing"/> is enabled and
<varname>object</varname> is not <literal>wal</literal>,
or if <xref linkend="guc-track-wal-io-timing"/> is enabled
@@ -3008,7 +3008,7 @@ description | Waiting for a newly initialized WAL file to reach durable storage
<note>
<para>
- Columns tracking I/O time will only be non-zero when
+ Columns tracking I/O wait time will only be non-zero when
<xref linkend="guc-track-io-timing"/> is enabled. The user should be
careful when referencing these columns in combination with their
corresponding I/O operations in case <varname>track_io_timing</varname>
--
2.48.1.76.g4e746b1a31.dirty
[text/x-diff] v2.12-0013-Enable-IO-concurrency-on-all-systems.patch (11.2K, ../../cxi6ssym62v7ywkt367qda3xik5jhsrcycwedyfqacinxooys5@pwarur37bosc/14-v2.12-0013-Enable-IO-concurrency-on-all-systems.patch)
download | inline diff:
From 8f98d20223aaf9a3c7965bbf6e64a0ff02ea1931 Mon Sep 17 00:00:00 2001
From: Andres Freund <andres@anarazel.de>
Date: Wed, 19 Mar 2025 10:15:20 -0400
Subject: [PATCH v2.12 13/28] Enable IO concurrency on all systems
Previously effective_io_concurrency and maintenance_io_concurrency could
not be set above 0 on machines without fadvise support. AIO enables
IO concurrency without such support.
Currently only subsystems using the read stream API will take advantage
of this. Other users of maintenance_io_concurrency (like recovery
prefetching) which leverage OS advice directly will not benefit from
this change. In those cases, maintenance_io_concurrency will have no
effect on I/O behavior.
Author: Melanie Plageman <melanieplageman@gmail.com>
Discussion: https://postgr.es/m/CAAKRu_atGgZePo=_g6T3cNtfMf0QxpvoUh5OUqa_cnPdhLd=gw@mail.gmail.com
---
src/include/storage/bufmgr.h | 6 ----
src/include/utils/guc_hooks.h | 4 ---
src/backend/access/common/reloptions.c | 8 -----
src/backend/commands/variable.c | 30 -------------------
src/backend/utils/misc/guc_tables.c | 4 +--
src/backend/utils/misc/postgresql.conf.sample | 6 ++--
src/bin/initdb/initdb.c | 5 ----
doc/src/sgml/config.sgml | 15 +++++-----
doc/src/sgml/ref/alter_tablespace.sgml | 2 +-
doc/src/sgml/ref/create_tablespace.sgml | 2 +-
10 files changed, 14 insertions(+), 68 deletions(-)
diff --git a/src/include/storage/bufmgr.h b/src/include/storage/bufmgr.h
index 7abb3bdfd9d..784df8b00cb 100644
--- a/src/include/storage/bufmgr.h
+++ b/src/include/storage/bufmgr.h
@@ -156,14 +156,8 @@ extern PGDLLIMPORT int bgwriter_lru_maxpages;
extern PGDLLIMPORT double bgwriter_lru_multiplier;
extern PGDLLIMPORT bool track_io_timing;
-/* only applicable when prefetching is available */
-#ifdef USE_PREFETCH
#define DEFAULT_EFFECTIVE_IO_CONCURRENCY 16
#define DEFAULT_MAINTENANCE_IO_CONCURRENCY 16
-#else
-#define DEFAULT_EFFECTIVE_IO_CONCURRENCY 0
-#define DEFAULT_MAINTENANCE_IO_CONCURRENCY 0
-#endif
extern PGDLLIMPORT int effective_io_concurrency;
extern PGDLLIMPORT int maintenance_io_concurrency;
diff --git a/src/include/utils/guc_hooks.h b/src/include/utils/guc_hooks.h
index 0f1e74f96c9..799fa7ace68 100644
--- a/src/include/utils/guc_hooks.h
+++ b/src/include/utils/guc_hooks.h
@@ -61,8 +61,6 @@ extern bool check_default_text_search_config(char **newval, void **extra, GucSou
extern void assign_default_text_search_config(const char *newval, void *extra);
extern bool check_default_with_oids(bool *newval, void **extra,
GucSource source);
-extern bool check_effective_io_concurrency(int *newval, void **extra,
- GucSource source);
extern bool check_huge_page_size(int *newval, void **extra, GucSource source);
extern void assign_io_method(int newval, void *extra);
extern bool check_io_max_concurrency(int *newval, void **extra, GucSource source);
@@ -83,8 +81,6 @@ extern bool check_log_stats(bool *newval, void **extra, GucSource source);
extern bool check_log_timezone(char **newval, void **extra, GucSource source);
extern void assign_log_timezone(const char *newval, void *extra);
extern const char *show_log_timezone(void);
-extern bool check_maintenance_io_concurrency(int *newval, void **extra,
- GucSource source);
extern void assign_maintenance_io_concurrency(int newval, void *extra);
extern void assign_io_max_combine_limit(int newval, void *extra);
extern void assign_io_combine_limit(int newval, void *extra);
diff --git a/src/backend/access/common/reloptions.c b/src/backend/access/common/reloptions.c
index 645b5c00467..46c1dce222d 100644
--- a/src/backend/access/common/reloptions.c
+++ b/src/backend/access/common/reloptions.c
@@ -361,11 +361,7 @@ static relopt_int intRelOpts[] =
RELOPT_KIND_TABLESPACE,
ShareUpdateExclusiveLock
},
-#ifdef USE_PREFETCH
-1, 0, MAX_IO_CONCURRENCY
-#else
- 0, 0, 0
-#endif
},
{
{
@@ -374,11 +370,7 @@ static relopt_int intRelOpts[] =
RELOPT_KIND_TABLESPACE,
ShareUpdateExclusiveLock
},
-#ifdef USE_PREFETCH
-1, 0, MAX_IO_CONCURRENCY
-#else
- 0, 0, 0
-#endif
},
{
{
diff --git a/src/backend/commands/variable.c b/src/backend/commands/variable.c
index f550a3c0c63..d3567aa7f27 100644
--- a/src/backend/commands/variable.c
+++ b/src/backend/commands/variable.c
@@ -1145,7 +1145,6 @@ check_cluster_name(char **newval, void **extra, GucSource source)
void
assign_maintenance_io_concurrency(int newval, void *extra)
{
-#ifdef USE_PREFETCH
/*
* Reconfigure recovery prefetching, because a setting it depends on
* changed.
@@ -1153,7 +1152,6 @@ assign_maintenance_io_concurrency(int newval, void *extra)
maintenance_io_concurrency = newval;
if (AmStartupProcess())
XLogPrefetchReconfigure();
-#endif
}
/*
@@ -1249,34 +1247,6 @@ check_default_with_oids(bool *newval, void **extra, GucSource source)
return true;
}
-bool
-check_effective_io_concurrency(int *newval, void **extra, GucSource source)
-{
-#ifndef USE_PREFETCH
- if (*newval != 0)
- {
- GUC_check_errdetail("\"%s\" must be set to 0 on platforms that lack support for issuing read-ahead advice.",
- "effective_io_concurrency");
- return false;
- }
-#endif /* USE_PREFETCH */
- return true;
-}
-
-bool
-check_maintenance_io_concurrency(int *newval, void **extra, GucSource source)
-{
-#ifndef USE_PREFETCH
- if (*newval != 0)
- {
- GUC_check_errdetail("\"%s\" must be set to 0 on platforms that lack support for issuing read-ahead advice.",
- "maintenance_io_concurrency");
- return false;
- }
-#endif /* USE_PREFETCH */
- return true;
-}
-
bool
check_ssl(bool *newval, void **extra, GucSource source)
{
diff --git a/src/backend/utils/misc/guc_tables.c b/src/backend/utils/misc/guc_tables.c
index 989825d3a9c..5d729102f46 100644
--- a/src/backend/utils/misc/guc_tables.c
+++ b/src/backend/utils/misc/guc_tables.c
@@ -3245,7 +3245,7 @@ struct config_int ConfigureNamesInt[] =
&effective_io_concurrency,
DEFAULT_EFFECTIVE_IO_CONCURRENCY,
0, MAX_IO_CONCURRENCY,
- check_effective_io_concurrency, NULL, NULL
+ NULL, NULL, NULL
},
{
@@ -3259,7 +3259,7 @@ struct config_int ConfigureNamesInt[] =
&maintenance_io_concurrency,
DEFAULT_MAINTENANCE_IO_CONCURRENCY,
0, MAX_IO_CONCURRENCY,
- check_maintenance_io_concurrency, assign_maintenance_io_concurrency,
+ NULL, assign_maintenance_io_concurrency,
NULL
},
diff --git a/src/backend/utils/misc/postgresql.conf.sample b/src/backend/utils/misc/postgresql.conf.sample
index 2246ccb85a7..771fe4cbe35 100644
--- a/src/backend/utils/misc/postgresql.conf.sample
+++ b/src/backend/utils/misc/postgresql.conf.sample
@@ -198,11 +198,11 @@
# - I/O -
#backend_flush_after = 0 # measured in pages, 0 disables
-#effective_io_concurrency = 16 # 1-1000; 0 disables prefetching
-#maintenance_io_concurrency = 16 # 1-1000; 0 disables prefetching
+#effective_io_concurrency = 16 # 1-1000; 0 disables issuing multiple simultaneous IO requests
+#maintenance_io_concurrency = 16 # 1-1000; same as effective_io_concurrency
#io_max_combine_limit = 128kB # usually 1-128 blocks (depends on OS)
# (change requires restart)
-#io_combine_limit = 128kB # usually 1-128 blocks (depends on OS)
+#io_combine_limit = 128kB # usually 1-32 blocks (depends on OS)
#io_method = worker # worker, io_uring, sync
# (change requires restart)
diff --git a/src/bin/initdb/initdb.c b/src/bin/initdb/initdb.c
index 21a0fe3ecd9..b9d75799ee0 100644
--- a/src/bin/initdb/initdb.c
+++ b/src/bin/initdb/initdb.c
@@ -1401,11 +1401,6 @@ setup_config(void)
repltok, true);
#endif
-#ifndef USE_PREFETCH
- conflines = replace_guc_value(conflines, "effective_io_concurrency",
- "0", true);
-#endif
-
#ifdef WIN32
conflines = replace_guc_value(conflines, "update_process_title",
"off", true);
diff --git a/doc/src/sgml/config.sgml b/doc/src/sgml/config.sgml
index 13773c05ff4..e447ade5c2b 100644
--- a/doc/src/sgml/config.sgml
+++ b/doc/src/sgml/config.sgml
@@ -2585,8 +2585,7 @@ include_dir 'conf.d'
session attempts to initiate in parallel. The allowed range is
<literal>1</literal> to <literal>1000</literal>, or
<literal>0</literal> to disable issuance of asynchronous I/O requests.
- The default is <literal>16</literal> on supported systems, otherwise
- <literal>0</literal>.
+ The default is <literal>16</literal>.
</para>
<para>
@@ -2597,8 +2596,8 @@ include_dir 'conf.d'
</para>
<para>
- On systems without prefetch advice support, attempting to configure
- any value other than <literal>0</literal> will error out.
+ On systems with prefetch advice support,
+ <varname>effective_io_concurrency</varname> also controls the prefetch distance.
</para>
<para>
@@ -2621,10 +2620,10 @@ include_dir 'conf.d'
for maintenance work that is done on behalf of many client sessions.
</para>
<para>
- The default is <literal>16</literal> on supported systems, otherwise
- <literal>0</literal>. This value can be overridden for tables in a
- particular tablespace by setting the tablespace parameter of the same
- name (see <xref linkend="sql-altertablespace"/>).
+ The default is <literal>16</literal>. This value can be overridden
+ for tables in a particular tablespace by setting the tablespace
+ parameter of the same name (see <xref
+ linkend="sql-altertablespace"/>).
</para>
</listitem>
</varlistentry>
diff --git a/doc/src/sgml/ref/alter_tablespace.sgml b/doc/src/sgml/ref/alter_tablespace.sgml
index 6ec863400d1..d0e08089ddb 100644
--- a/doc/src/sgml/ref/alter_tablespace.sgml
+++ b/doc/src/sgml/ref/alter_tablespace.sgml
@@ -88,7 +88,7 @@ ALTER TABLESPACE <replaceable>name</replaceable> RESET ( <replaceable class="par
and <varname>maintenance_io_concurrency</varname>.
Setting these values for a particular tablespace will override the
planner's usual estimate of the cost of reading pages from tables in
- that tablespace, and the executor's prefetching behavior, as established
+ that tablespace, and how many concurrent I/Os are issued, as established
by the configuration parameters of the
same name (see <xref linkend="guc-seq-page-cost"/>,
<xref linkend="guc-random-page-cost"/>,
diff --git a/doc/src/sgml/ref/create_tablespace.sgml b/doc/src/sgml/ref/create_tablespace.sgml
index 9d5ab025261..b77e774c53f 100644
--- a/doc/src/sgml/ref/create_tablespace.sgml
+++ b/doc/src/sgml/ref/create_tablespace.sgml
@@ -110,7 +110,7 @@ CREATE TABLESPACE <replaceable class="parameter">tablespace_name</replaceable>
and <varname>maintenance_io_concurrency</varname>.
Setting these values for a particular tablespace will override the
planner's usual estimate of the cost of reading pages from tables in
- that tablespace, and the executor's prefetching behavior, as established
+ that tablespace, and how many concurrent I/Os are issued, as established
by the configuration parameters of the
same name (see <xref linkend="guc-seq-page-cost"/>,
<xref linkend="guc-random-page-cost"/>,
--
2.48.1.76.g4e746b1a31.dirty
[text/x-diff] v2.12-0014-docs-Add-acronym-and-glossary-entries-for-I-O-.patch (3.5K, ../../cxi6ssym62v7ywkt367qda3xik5jhsrcycwedyfqacinxooys5@pwarur37bosc/15-v2.12-0014-docs-Add-acronym-and-glossary-entries-for-I-O-.patch)
download | inline diff:
From c4afa7789e9c326d3bf29e26916bca08fc0718a6 Mon Sep 17 00:00:00 2001
From: Andres Freund <andres@anarazel.de>
Date: Fri, 21 Mar 2025 15:06:35 -0400
Subject: [PATCH v2.12 14/28] docs: Add acronym and glossary entries for I/O
and AIO
These could use a lot more polish.
I did not actually reference the new entries yet, because I don't really
understand what our policy for that is.
---
doc/src/sgml/acronyms.sgml | 18 +++++++++++++++++
doc/src/sgml/glossary.sgml | 40 ++++++++++++++++++++++++++++++++++++++
2 files changed, 58 insertions(+)
diff --git a/doc/src/sgml/acronyms.sgml b/doc/src/sgml/acronyms.sgml
index 58d0d90fece..2f906e9f018 100644
--- a/doc/src/sgml/acronyms.sgml
+++ b/doc/src/sgml/acronyms.sgml
@@ -9,6 +9,15 @@
<variablelist>
+ <varlistentry>
+ <term><acronym>AIO</acronym></term>
+ <listitem>
+ <para>
+ <link linkend="glossary-aio">Asynchronous <acronym>I/O</acronym></link>
+ </para>
+ </listitem>
+ </varlistentry>
+
<varlistentry>
<term><acronym>ACL</acronym></term>
<listitem>
@@ -354,6 +363,15 @@
</listitem>
</varlistentry>
+ <varlistentry>
+ <term><acronym>I/O</acronym></term>
+ <listitem>
+ <para>
+ <link linkend="glossary-io">Input/Output</link>
+ </para>
+ </listitem>
+ </varlistentry>
+
<varlistentry>
<term><acronym>ISO</acronym></term>
<listitem>
diff --git a/doc/src/sgml/glossary.sgml b/doc/src/sgml/glossary.sgml
index c0f812e3f5e..6ca2237ea1d 100644
--- a/doc/src/sgml/glossary.sgml
+++ b/doc/src/sgml/glossary.sgml
@@ -7,6 +7,7 @@
</para>
<glosslist>
+
<glossentry id="glossary-acid">
<glossterm>ACID</glossterm>
<glossdef>
@@ -81,6 +82,31 @@
</glossdef>
</glossentry>
+ <glossentry id="glossary-aio">
+ <glossterm>Asynchronous <acronym>I/O</acronym></glossterm>
+ <acronym>AIO</acronym>
+ <indexterm>
+ <primary>Asynchronous <acronym>I/O</acronym></primary>
+ </indexterm>
+ <glossdef>
+ <para>
+ Asynchronous <acronym>I/O</acronym> (<acronym>AIO</acronym>) describes
+ performing <acronym>I/O</acronym> in a non-blocking way (asynchronously),
+ in contrast to synchronous <acronym>I/O</acronym>, which blocks for the
+ entire duration of the <acronym>I/O</acronym>.
+ </para>
+ <para>
+ With <acronym>AIO</acronym>, starting an <acronym>I/O</acronym> operation
+ is separated from waiting for the result of the operation, allowing
+ multiple <acronym>I/O</acronym> operations to be initiated concurrently,
+ as well as performing <acronym>CPU</acronym> heavy operations
+ concurrently with <acronym>I/O</acronym>. The price for that increased
+ concurrency is increased complexity.
+ </para>
+ <glossseealso otherterm="glossary-io" />
+ </glossdef>
+ </glossentry>
+
<glossentry id="glossary-atomic">
<glossterm>Atomic</glossterm>
<glossdef>
@@ -938,6 +964,20 @@
</glossdef>
</glossentry>
+ <glossentry id="glossary-io">
+ <glossterm>Input/Output</glossterm>
+ <acronym>I/O</acronym>
+ <glossdef>
+ <para>
+ Input/Output (<acronym>I/O</acronym>) describes the communication between
+ a program and peripheral devices. In the context of database systems,
+ <acronym>I/O</acronym> commonly, but not exclusively, refers to
+ interaction with storage devices or the network.
+ </para>
+ <glossseealso otherterm="glossary-aio" />
+ </glossdef>
+ </glossentry>
+
<glossentry id="glossary-insert">
<glossterm>Insert</glossterm>
<glossdef>
--
2.48.1.76.g4e746b1a31.dirty
[text/x-diff] v2.12-0015-aio-Add-pg_aios-view.patch (18.8K, ../../cxi6ssym62v7ywkt367qda3xik5jhsrcycwedyfqacinxooys5@pwarur37bosc/16-v2.12-0015-aio-Add-pg_aios-view.patch)
download | inline diff:
From 98a0a84107d692d043511b1282dc5c924b83e5d5 Mon Sep 17 00:00:00 2001
From: Andres Freund <andres@anarazel.de>
Date: Tue, 18 Mar 2025 14:40:06 -0400
Subject: [PATCH v2.12 15/28] aio: Add pg_aios view
FIXME:
- catversion bump before commit
---
src/include/catalog/pg_proc.dat | 10 +
src/backend/catalog/system_views.sql | 3 +
src/backend/storage/aio/Makefile | 1 +
src/backend/storage/aio/aio_funcs.c | 225 +++++++++++++++++++++
src/backend/storage/aio/meson.build | 1 +
doc/src/sgml/system-views.sgml | 280 +++++++++++++++++++++++++++
src/test/regress/expected/rules.out | 16 ++
7 files changed, 536 insertions(+)
create mode 100644 src/backend/storage/aio/aio_funcs.c
diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat
index 0d29ef50ff2..cf5855ccf35 100644
--- a/src/include/catalog/pg_proc.dat
+++ b/src/include/catalog/pg_proc.dat
@@ -12479,4 +12479,14 @@
proargtypes => 'int4',
prosrc => 'gist_stratnum_common' },
+# AIO related functions
+{ oid => '9200', descr => 'information about in-progress asynchronous IOs',
+ proname => 'pg_get_aios', prorows => '100', proretset => 't',
+ provolatile => 'v', proparallel => 'r', prorettype => 'record', proargtypes => '',
+ proallargtypes => '{int4,int4,int8,text,text,int8,int8,text,int2,int4,text,text,bool,bool,bool}',
+ proargmodes => '{o,o,o,o,o,o,o,o,o,o,o,o,o,o,o}',
+ proargnames => '{pid,io_id,io_generation,state,operation,off,length,target,handle_data_len,raw_result,result,target_desc,f_sync,f_localmem,f_buffered}',
+ prosrc => 'pg_get_aios' },
+
+
]
diff --git a/src/backend/catalog/system_views.sql b/src/backend/catalog/system_views.sql
index 31d269b7ee0..37fd4bc2566 100644
--- a/src/backend/catalog/system_views.sql
+++ b/src/backend/catalog/system_views.sql
@@ -1391,3 +1391,6 @@ CREATE VIEW pg_stat_subscription_stats AS
CREATE VIEW pg_wait_events AS
SELECT * FROM pg_get_wait_events();
+
+CREATE VIEW pg_aios AS
+ SELECT * FROM pg_get_aios();
diff --git a/src/backend/storage/aio/Makefile b/src/backend/storage/aio/Makefile
index c06c50771e0..3f2469cc399 100644
--- a/src/backend/storage/aio/Makefile
+++ b/src/backend/storage/aio/Makefile
@@ -11,6 +11,7 @@ include $(top_builddir)/src/Makefile.global
OBJS = \
aio.o \
aio_callback.o \
+ aio_funcs.o \
aio_init.o \
aio_io.o \
aio_target.o \
diff --git a/src/backend/storage/aio/aio_funcs.c b/src/backend/storage/aio/aio_funcs.c
new file mode 100644
index 00000000000..625966a515e
--- /dev/null
+++ b/src/backend/storage/aio/aio_funcs.c
@@ -0,0 +1,225 @@
+/*-------------------------------------------------------------------------
+ *
+ * aio_funcs.c
+ * AIO - SQL interface for AIO
+ *
+ *
+ * Portions Copyright (c) 1996-2025, PostgreSQL Global Development Group
+ * Portions Copyright (c) 1994, Regents of the University of California
+ *
+ * IDENTIFICATION
+ * src/backend/storage/aio/aio_funcs.c
+ *
+ *-------------------------------------------------------------------------
+ */
+
+#include "postgres.h"
+
+#include "fmgr.h"
+#include "funcapi.h"
+#include "nodes/execnodes.h"
+#include "port/atomics.h"
+#include "storage/aio_internal.h"
+#include "storage/lock.h"
+#include "storage/proc.h"
+#include "storage/procnumber.h"
+#include "utils/builtins.h"
+#include "utils/fmgrprotos.h"
+#include "utils/tuplestore.h"
+
+
+/*
+ * Byte length of an iovec.
+ */
+static size_t
+iov_byte_length(const struct iovec *iov, int cnt)
+{
+ size_t len = 0;
+
+ for (int i = 0; i < cnt; i++)
+ {
+ len += iov[i].iov_len;
+ }
+
+ return len;
+}
+
+Datum
+pg_get_aios(PG_FUNCTION_ARGS)
+{
+ ReturnSetInfo *rsinfo = (ReturnSetInfo *) fcinfo->resultinfo;
+
+ InitMaterializedSRF(fcinfo, 0);
+
+#define PG_GET_AIOS_COLS 15
+
+ for (uint64 i = 0; i < pgaio_ctl->io_handle_count; i++)
+ {
+ PgAioHandle *live_ioh = &pgaio_ctl->io_handles[i];
+ uint32 ioh_id = pgaio_io_get_id(live_ioh);
+ Datum values[PG_GET_AIOS_COLS] = {0};
+ bool nulls[PG_GET_AIOS_COLS] = {0};
+ ProcNumber owner;
+ PGPROC *owner_proc;
+ int32 owner_pid;
+ PgAioHandleState start_state;
+ uint64 start_generation;
+ PgAioHandle ioh_copy;
+ struct iovec iov_copy[PG_IOV_MAX];
+
+retry:
+
+ /*
+ * There is no lock that could prevent the state of the IO to advance
+ * concurrently - and we don't want to introduce one, as that would
+ * introduce atomics into a very common path. Instead we
+ *
+ * 1) Determine the state + generation of the IO.
+ *
+ * 2) Copy the IO to local memory.
+ *
+ * 3) Check if state or generation of the IO changed. If the state
+ * changed, retry, if the generation changed don't display the IO.
+ */
+
+ /* 1) from above */
+ start_generation = live_ioh->generation;
+ pg_read_barrier();
+ start_state = live_ioh->state;
+
+ if (start_state == PGAIO_HS_IDLE)
+ continue;
+
+ /* 2) from above */
+ memcpy(&ioh_copy, live_ioh, sizeof(PgAioHandle));
+
+ /*
+ * Safe to copy even if no iovec is used - we always reserve the
+ * required space.
+ */
+ memcpy(&iov_copy, &pgaio_ctl->iovecs[ioh_copy.iovec_off],
+ PG_IOV_MAX * sizeof(struct iovec));
+
+ /*
+ * Copy information about owner before 3) below, if the process exited
+ * it'd have to wait for the IO to finish first, which we would detect
+ * in 3).
+ */
+ owner = ioh_copy.owner_procno;
+ owner_proc = GetPGProcByNumber(owner);
+ owner_pid = owner_proc->pid;
+
+ /* 3) from above */
+ pg_read_barrier();
+
+ /*
+ * The IO completed and a new one was started with the same ID. Don't
+ * display it - it really started after this function was called.
+ * There be a risk of a livelock if we just retried endlessly, if IOs
+ * complete very quickly.
+ */
+ if (live_ioh->generation != start_generation)
+ continue;
+
+ /*
+ * The IOs state changed while we were "rendering" it. Just start from
+ * scratch. There's no risk of a livelock here, as an IO has a limited
+ * sets of states it can be in, and state changes go only in a single
+ * direction.
+ */
+ if (live_ioh->state != start_state)
+ goto retry;
+
+ /*
+ * Now that we have copied the IO into local memory and checked that
+ * it's still in the same state, we are not allowed to access "live"
+ * memory anymore. To make it slightly easier to catch such cases, set
+ * the "live" pointers to NULL.
+ */
+ live_ioh = NULL;
+ owner_proc = NULL;
+
+
+ /* column: owning pid */
+ if (owner_pid != 0)
+ values[0] = Int32GetDatum(owner_pid);
+ else
+ nulls[0] = false;
+
+ /* column: IO's id */
+ values[1] = ioh_id;
+
+ /* column: IO's generation */
+ values[2] = Int64GetDatum(start_generation);
+
+ /* column: IO's state */
+ values[3] = CStringGetTextDatum(pgaio_io_get_state_name(&ioh_copy));
+
+ /*
+ * If the IO is in PGAIO_HS_HANDED_OUT state, none of the following
+ * fields are valid yet (or are in the process of being set).
+ * Therefore we don't want to display any other columns.
+ */
+ if (start_state == PGAIO_HS_HANDED_OUT)
+ {
+ memset(nulls + 4, 1, (lengthof(nulls) - 4) * sizeof(bool));
+ goto display;
+ }
+
+ /* column: IO's operation */
+ values[4] = CStringGetTextDatum(pgaio_io_get_op_name(&ioh_copy));
+
+ /* columns: details about the IO's operation (offset, length) */
+ switch (ioh_copy.op)
+ {
+ case PGAIO_OP_INVALID:
+ nulls[5] = true;
+ nulls[6] = true;
+ break;
+ case PGAIO_OP_READV:
+ values[5] = Int64GetDatum(ioh_copy.op_data.read.offset);
+ values[6] =
+ Int64GetDatum(iov_byte_length(iov_copy, ioh_copy.op_data.read.iov_length));
+ break;
+ case PGAIO_OP_WRITEV:
+ values[5] = Int64GetDatum(ioh_copy.op_data.write.offset);
+ values[6] =
+ Int64GetDatum(iov_byte_length(iov_copy, ioh_copy.op_data.write.iov_length));
+ break;
+ }
+
+ /* column: IO's target */
+ values[7] = CStringGetTextDatum(pgaio_io_get_target_name(&ioh_copy));
+
+ /* column: length of IO's data array */
+ values[8] = Int16GetDatum(ioh_copy.handle_data_len);
+
+ /* column: raw result (i.e. some form of syscall return value) */
+ if (start_state == PGAIO_HS_COMPLETED_IO
+ || start_state == PGAIO_HS_COMPLETED_SHARED
+ || start_state == PGAIO_HS_COMPLETED_LOCAL)
+ values[9] = Int32GetDatum(ioh_copy.result);
+ else
+ nulls[9] = true;
+
+ /*
+ * column: result in the higher level representation (unknown if not
+ * finished)
+ */
+ values[10] =
+ CStringGetTextDatum(pgaio_result_status_string(ioh_copy.distilled_result.status));
+
+ /* column: target description */
+ values[11] = CStringGetTextDatum(pgaio_io_get_target_description(&ioh_copy));
+
+ /* columns: one for each flag */
+ values[12] = BoolGetDatum(ioh_copy.flags & PGAIO_HF_SYNCHRONOUS);
+ values[13] = BoolGetDatum(ioh_copy.flags & PGAIO_HF_REFERENCES_LOCAL);
+ values[14] = BoolGetDatum(ioh_copy.flags & PGAIO_HF_BUFFERED);
+
+display:
+ tuplestore_putvalues(rsinfo->setResult, rsinfo->setDesc, values, nulls);
+ }
+
+ return (Datum) 0;
+}
diff --git a/src/backend/storage/aio/meson.build b/src/backend/storage/aio/meson.build
index 2f0f03d8071..da6df2d3654 100644
--- a/src/backend/storage/aio/meson.build
+++ b/src/backend/storage/aio/meson.build
@@ -3,6 +3,7 @@
backend_sources += files(
'aio.c',
'aio_callback.c',
+ 'aio_funcs.c',
'aio_init.c',
'aio_io.c',
'aio_target.c',
diff --git a/doc/src/sgml/system-views.sgml b/doc/src/sgml/system-views.sgml
index 3f5a306247e..ec6e554e533 100644
--- a/doc/src/sgml/system-views.sgml
+++ b/doc/src/sgml/system-views.sgml
@@ -51,6 +51,11 @@
</thead>
<tbody>
+ <row>
+ <entry><link linkend="view-pg-aios"><structname>pg_aios</structname></link></entry>
+ <entry>In-use asynchronous IO handles</entry>
+ </row>
+
<row>
<entry><link linkend="view-pg-available-extensions"><structname>pg_available_extensions</structname></link></entry>
<entry>available extensions</entry>
@@ -231,6 +236,281 @@
</table>
</sect1>
+ <sect1 id="view-pg-aios">
+ <title><structname>pg_aios</structname></title>
+
+ <indexterm zone="view-pg-aios">
+ <primary>pg_aios</primary>
+ </indexterm>
+
+ <para>
+ The <structname>pg_aios</structname> view lists all <xref
+ linkend="glossary-aio"/> handles that are currently in-use. An I/O handle
+ is used to reference an I/O operation that is being prepared, executed or
+ is in the process of completing. <structname>pg_aios</structname> contains
+ one row for each I/O handle.
+ </para>
+
+ <para>
+ This view is mainly useful for developers of
+ <productname>PostgreSQL</productname>, but may also be useful when tuning
+ <productname>PostgreSQL</productname>.
+ </para>
+
+ <table>
+ <title><structname>pg_aios</structname> Columns</title>
+ <tgroup cols="1">
+ <thead>
+ <row>
+ <entry role="catalog_table_entry"><para role="column_definition">
+ Column Type
+ </para>
+ <para>
+ Description
+ </para></entry>
+ </row>
+ </thead>
+
+ <tbody>
+ <row>
+ <entry role="catalog_table_entry"><para role="column_definition">
+ <structfield>pid</structfield> <type>int4</type>
+ </para>
+ <para>
+ Process ID of the server process that is issuing this I/O.
+ </para></entry>
+ </row>
+
+ <row>
+ <entry role="catalog_table_entry"><para role="column_definition">
+ <structfield>io_id</structfield> <type>int4</type>
+ </para>
+ <para>
+ Identifier of the I/O handle. Handles are reused once the I/O
+ completed (or if the handle is released before I/O is started). On reuse
+ <link linkend="view-pg-aios-io-generation">
+ <structname>pg_aios</structname>.<structfield>io_generation</structfield>
+ </link>
+ is incremented.
+ </para></entry>
+ </row>
+
+ <row>
+ <entry role="catalog_table_entry" id="view-pg-aios-io-generation"><para role="column_definition">
+ <structfield>io_generation</structfield> <type>int8</type>
+ </para>
+ <para>
+ Generation of the I/O handle.
+ </para></entry>
+ </row>
+
+ <row>
+ <entry role="catalog_table_entry"><para role="column_definition">
+ <structfield>state</structfield> <type>text</type>
+ </para>
+ <para>
+ State of the I/O handle:
+ <itemizedlist>
+ <listitem>
+ <para>
+ <literal>HANDED_OUT</literal>, referenced by code but not yet used
+ </para>
+ </listitem>
+ <listitem>
+ <para>
+ <literal>DEFINED</literal>, information necessary for execution is known
+ </para>
+ </listitem>
+ <listitem>
+ <para>
+ <literal>STAGED</literal>, ready for execution
+ </para>
+ </listitem>
+ <listitem>
+ <para>
+ <literal>SUBMITTED</literal>, submitted for execution
+ </para>
+ </listitem>
+ <listitem>
+ <para>
+ <literal>COMPLETED_IO</literal>, finished, but result has not yet been processed
+ </para>
+ </listitem>
+ <listitem>
+ <para>
+ <literal>COMPLETED_SHARED</literal>, shared completion processing completed
+ </para>
+ </listitem>
+ <listitem>
+ <para>
+ <literal>COMPLETED_LOCAL</literal>, backend local completion processing completed
+ </para>
+ </listitem>
+ </itemizedlist>
+ </para></entry>
+ </row>
+
+ <row>
+ <entry role="catalog_table_entry"><para role="column_definition">
+ <structfield>operation</structfield> <type>text</type>
+ </para>
+ <para>
+ Operation performed using the I/O handle:
+ <itemizedlist>
+ <listitem>
+ <para>
+ <literal>invalid</literal>, not yet known
+ </para>
+ </listitem>
+ <listitem>
+ <para>
+ <literal>readv</literal>, a vectored read
+ </para>
+ </listitem>
+ <listitem>
+ <para>
+ <literal>writev</literal>, a vectored write
+ </para>
+ </listitem>
+ </itemizedlist>
+ </para></entry>
+ </row>
+
+ <row>
+ <entry role="catalog_table_entry"><para role="column_definition">
+ <structfield>off</structfield> <type>int8</type>
+ </para>
+ <para>
+ Offset of the I/O operation.
+ </para></entry>
+ </row>
+
+ <row>
+ <entry role="catalog_table_entry"><para role="column_definition">
+ <structfield>length</structfield> <type>int8</type>
+ </para>
+ <para>
+ Length of the I/O operation.
+ </para></entry>
+ </row>
+
+ <row>
+ <entry role="catalog_table_entry"><para role="column_definition">
+ <structfield>target</structfield> <type>text</type>
+ </para>
+ <para>
+ What kind of object is the I/O targeting:
+ <itemizedlist spacing="compact">
+ <listitem>
+ <para>
+ <literal>smgr</literal>, I/O on postgres relations
+ </para>
+ </listitem>
+ </itemizedlist>
+ </para></entry>
+ </row>
+
+ <row>
+ <entry role="catalog_table_entry"><para role="column_definition">
+ <structfield>handle_data_len</structfield> <type>int2</type>
+ </para>
+ <para>
+ Length of the data associated with the I/O operation. For I/O to/from
+ <xref linkend="guc-shared-buffers"/> and <xref
+ linkend="guc-temp-buffers"/>, this indicates the number of buffers the
+ I/O is operating on.
+ </para></entry>
+ </row>
+
+ <row>
+ <entry role="catalog_table_entry"><para role="column_definition">
+ <structfield>raw_result</structfield> <type>int4</type>
+ </para>
+ <para>
+ Low-level result of the I/O operation, or NULL if the operation has not
+ yet completed.
+ </para></entry>
+ </row>
+
+ <row>
+ <entry role="catalog_table_entry"><para role="column_definition">
+ <structfield>result</structfield> <type>text</type>
+ </para>
+ <para>
+ High-level result of the I/O operation:
+ <itemizedlist>
+ <listitem>
+ <para>
+ <literal>UNKNOWN</literal> means that the result of the
+ operation is not yet known.
+ </para>
+ </listitem>
+ <listitem>
+ <para>
+ <literal>OK</literal> means the I/O completed successfully.
+ </para>
+ </listitem>
+ <listitem>
+ <para>
+ <literal>PARTIAL</literal> means that the I/O completed without
+ error, but did not process all data. Commonly callers will need to
+ retry and perform the remainder of the work in a separate I/O.
+ </para>
+ </listitem>
+ <listitem>
+ <para>
+ <literal>ERROR</literal> mean the I/O failed with an error.
+ </para>
+ </listitem>
+ </itemizedlist>
+ </para></entry>
+ </row>
+
+ <row>
+ <entry role="catalog_table_entry"><para role="column_definition">
+ <structfield>target_desc</structfield> <type>text</type>
+ </para>
+ <para>
+ Description of what the I/O operation is targeting.
+ </para></entry>
+ </row>
+
+ <row>
+ <entry role="catalog_table_entry"><para role="column_definition">
+ <structfield>f_sync</structfield> <type>bool</type>
+ </para>
+ <para>
+ Flag indicating whether the I/O is executed synchronously.
+ </para></entry>
+ </row>
+
+ <row>
+ <entry role="catalog_table_entry"><para role="column_definition">
+ <structfield>f_localmem</structfield> <type>bool</type>
+ </para>
+ <para>
+ Flag indicating whether the I/O references process local memory.
+ </para></entry>
+ </row>
+
+ <row>
+ <entry role="catalog_table_entry"><para role="column_definition">
+ <structfield>f_buffered</structfield> <type>bool</type>
+ </para>
+ <para>
+ Flag indicating whether the I/O is buffered I/O.
+ </para></entry>
+ </row>
+
+ </tbody>
+ </tgroup>
+ </table>
+
+ <para>
+ The <structname>pg_aios</structname> view is read-only.
+ </para>
+ </sect1>
+
<sect1 id="view-pg-available-extensions">
<title><structname>pg_available_extensions</structname></title>
diff --git a/src/test/regress/expected/rules.out b/src/test/regress/expected/rules.out
index 47478969135..d9533deb04e 100644
--- a/src/test/regress/expected/rules.out
+++ b/src/test/regress/expected/rules.out
@@ -1286,6 +1286,22 @@ drop table cchild;
SELECT viewname, definition FROM pg_views
WHERE schemaname = 'pg_catalog'
ORDER BY viewname;
+pg_aios| SELECT pid,
+ io_id,
+ io_generation,
+ state,
+ operation,
+ off,
+ length,
+ target,
+ handle_data_len,
+ raw_result,
+ result,
+ target_desc,
+ f_sync,
+ f_localmem,
+ f_buffered
+ FROM pg_get_aios() pg_get_aios(pid, io_id, io_generation, state, operation, off, length, target, handle_data_len, raw_result, result, target_desc, f_sync, f_localmem, f_buffered);
pg_available_extension_versions| SELECT e.name,
e.version,
(x.extname IS NOT NULL) AS installed,
--
2.48.1.76.g4e746b1a31.dirty
[text/x-diff] v2.12-0016-aio-Add-test_aio-module.patch (55.9K, ../../cxi6ssym62v7ywkt367qda3xik5jhsrcycwedyfqacinxooys5@pwarur37bosc/17-v2.12-0016-aio-Add-test_aio-module.patch)
download | inline diff:
From 8dbc0b33b7c4e4f26404208d1277be8ca4331f3f Mon Sep 17 00:00:00 2001
From: Andres Freund <andres@anarazel.de>
Date: Tue, 18 Mar 2025 14:40:06 -0400
Subject: [PATCH v2.12 16/28] aio: Add test_aio module
To make the tests possible, a few functions from bufmgr.c/localbuf.c had to be
exported, via buf_internals.h.
Reviewed-by: Noah Misch <noah@leadboat.com>
Co-authored-by: Andres Freund <andres@anarazel.de>
Co-authored-by: Nazir Bilal Yavuz <byavuz81@gmail.com>
Discussion: https://postgr.es/m/uvrtrknj4kdytuboidbhwclo4gxhswwcpgadptsjvjqcluzmah%40brqs62irg4dt
---
src/include/storage/buf_internals.h | 7 +
src/backend/storage/buffer/bufmgr.c | 8 +-
src/backend/storage/buffer/localbuf.c | 3 +-
src/test/modules/Makefile | 1 +
src/test/modules/meson.build | 1 +
src/test/modules/test_aio/.gitignore | 2 +
src/test/modules/test_aio/Makefile | 26 +
src/test/modules/test_aio/meson.build | 37 +
src/test/modules/test_aio/t/001_aio.pl | 906 ++++++++++++++++++
src/test/modules/test_aio/t/002_io_workers.pl | 125 +++
src/test/modules/test_aio/test_aio--1.0.sql | 101 ++
src/test/modules/test_aio/test_aio.c | 712 ++++++++++++++
src/test/modules/test_aio/test_aio.control | 3 +
13 files changed, 1924 insertions(+), 8 deletions(-)
create mode 100644 src/test/modules/test_aio/.gitignore
create mode 100644 src/test/modules/test_aio/Makefile
create mode 100644 src/test/modules/test_aio/meson.build
create mode 100644 src/test/modules/test_aio/t/001_aio.pl
create mode 100644 src/test/modules/test_aio/t/002_io_workers.pl
create mode 100644 src/test/modules/test_aio/test_aio--1.0.sql
create mode 100644 src/test/modules/test_aio/test_aio.c
create mode 100644 src/test/modules/test_aio/test_aio.control
diff --git a/src/include/storage/buf_internals.h b/src/include/storage/buf_internals.h
index 8a52ace37cd..6821a710e46 100644
--- a/src/include/storage/buf_internals.h
+++ b/src/include/storage/buf_internals.h
@@ -434,6 +434,12 @@ extern void IssuePendingWritebacks(WritebackContext *wb_context, IOContext io_co
extern void ScheduleBufferTagForWriteback(WritebackContext *wb_context,
IOContext io_context, BufferTag *tag);
+/* solely to make it easier to write tests */
+extern bool StartBufferIO(BufferDesc *buf, bool forInput, bool nowait);
+extern void TerminateBufferIO(BufferDesc *buf, bool clear_dirty, uint32 set_flag_bits,
+ bool forget_owner, bool syncio);
+
+
/* freelist.c */
extern IOContext IOContextForStrategy(BufferAccessStrategy strategy);
extern BufferDesc *StrategyGetBuffer(BufferAccessStrategy strategy,
@@ -478,6 +484,7 @@ extern void TerminateLocalBufferIO(BufferDesc *bufHdr, bool clear_dirty,
uint32 set_flag_bits, bool syncio);
extern bool StartLocalBufferIO(BufferDesc *bufHdr, bool forInput, bool nowait);
extern void FlushLocalBuffer(BufferDesc *bufHdr, SMgrRelation reln);
+extern void InvalidateLocalBuffer(BufferDesc *bufHdr, bool check_unreferenced);
extern void DropRelationLocalBuffers(RelFileLocator rlocator,
ForkNumber forkNum,
BlockNumber firstDelBlock);
diff --git a/src/backend/storage/buffer/bufmgr.c b/src/backend/storage/buffer/bufmgr.c
index 883c5918a33..fcc5eb5654e 100644
--- a/src/backend/storage/buffer/bufmgr.c
+++ b/src/backend/storage/buffer/bufmgr.c
@@ -518,10 +518,6 @@ static uint32 WaitBufHdrUnlocked(BufferDesc *buf);
static int SyncOneBuffer(int buf_id, bool skip_recently_used,
WritebackContext *wb_context);
static void WaitIO(BufferDesc *buf);
-static bool StartBufferIO(BufferDesc *buf, bool forInput, bool nowait);
-static void TerminateBufferIO(BufferDesc *buf, bool clear_dirty,
- uint32 set_flag_bits, bool forget_owner,
- bool syncio);
static void AbortBufferIO(Buffer buffer);
static void shared_buffer_write_error_callback(void *arg);
static void local_buffer_write_error_callback(void *arg);
@@ -5947,7 +5943,7 @@ WaitIO(BufferDesc *buf)
* find out if they can perform the I/O as part of a larger operation, without
* waiting for the answer or distinguishing the reasons why not.
*/
-static bool
+bool
StartBufferIO(BufferDesc *buf, bool forInput, bool nowait)
{
uint32 buf_state;
@@ -6004,7 +6000,7 @@ StartBufferIO(BufferDesc *buf, bool forInput, bool nowait)
* resource owner. (forget_owner=false is used when the resource owner itself
* is being released)
*/
-static void
+void
TerminateBufferIO(BufferDesc *buf, bool clear_dirty, uint32 set_flag_bits,
bool forget_owner, bool syncio)
{
diff --git a/src/backend/storage/buffer/localbuf.c b/src/backend/storage/buffer/localbuf.c
index 3b81a5661c7..ffc1a7c93a2 100644
--- a/src/backend/storage/buffer/localbuf.c
+++ b/src/backend/storage/buffer/localbuf.c
@@ -58,7 +58,6 @@ static int NLocalPinnedBuffers = 0;
static void InitLocalBuffers(void);
static Block GetLocalBufferStorage(void);
static Buffer GetLocalVictimBuffer(void);
-static void InvalidateLocalBuffer(BufferDesc *bufHdr, bool check_unreferenced);
/*
@@ -596,7 +595,7 @@ TerminateLocalBufferIO(BufferDesc *bufHdr, bool clear_dirty, uint32 set_flag_bit
*
* See also InvalidateBuffer().
*/
-static void
+void
InvalidateLocalBuffer(BufferDesc *bufHdr, bool check_unreferenced)
{
Buffer buffer = BufferDescriptorGetBuffer(bufHdr);
diff --git a/src/test/modules/Makefile b/src/test/modules/Makefile
index 4e4be3fa511..aa1d27bbed3 100644
--- a/src/test/modules/Makefile
+++ b/src/test/modules/Makefile
@@ -14,6 +14,7 @@ SUBDIRS = \
oauth_validator \
plsample \
spgist_name_ops \
+ test_aio \
test_bloomfilter \
test_copy_callbacks \
test_custom_rmgrs \
diff --git a/src/test/modules/meson.build b/src/test/modules/meson.build
index 2b057451473..4fda1acfb32 100644
--- a/src/test/modules/meson.build
+++ b/src/test/modules/meson.build
@@ -1,5 +1,6 @@
# Copyright (c) 2022-2025, PostgreSQL Global Development Group
+subdir('test_aio')
subdir('brin')
subdir('commit_ts')
subdir('delay_execution')
diff --git a/src/test/modules/test_aio/.gitignore b/src/test/modules/test_aio/.gitignore
new file mode 100644
index 00000000000..716e17f5a2a
--- /dev/null
+++ b/src/test/modules/test_aio/.gitignore
@@ -0,0 +1,2 @@
+# Generated subdirectories
+/tmp_check/
diff --git a/src/test/modules/test_aio/Makefile b/src/test/modules/test_aio/Makefile
new file mode 100644
index 00000000000..0a99b68d56d
--- /dev/null
+++ b/src/test/modules/test_aio/Makefile
@@ -0,0 +1,26 @@
+# src/test/modules/delay_execution/Makefile
+
+PGFILEDESC = "test_aio - test code for AIO"
+
+MODULE_big = test_aio
+OBJS = \
+ $(WIN32RES) \
+ test_aio.o
+
+EXTENSION = test_aio
+DATA = test_aio--1.0.sql
+
+TAP_TESTS = 1
+
+export enable_injection_points
+
+ifdef USE_PGXS
+PG_CONFIG = pg_config
+PGXS := $(shell $(PG_CONFIG) --pgxs)
+include $(PGXS)
+else
+subdir = src/test/modules/test_aio
+top_builddir = ../../../..
+include $(top_builddir)/src/Makefile.global
+include $(top_srcdir)/contrib/contrib-global.mk
+endif
diff --git a/src/test/modules/test_aio/meson.build b/src/test/modules/test_aio/meson.build
new file mode 100644
index 00000000000..fd638eb4416
--- /dev/null
+++ b/src/test/modules/test_aio/meson.build
@@ -0,0 +1,37 @@
+# Copyright (c) 2022-2024, PostgreSQL Global Development Group
+
+test_aio_sources = files(
+ 'test_aio.c',
+)
+
+if host_system == 'windows'
+ test_aio_sources += rc_lib_gen.process(win32ver_rc, extra_args: [
+ '--NAME', 'test_aio',
+ '--FILEDESC', 'test_aio - test code for AIO',])
+endif
+
+test_aio = shared_module('test_aio',
+ test_aio_sources,
+ kwargs: pg_test_mod_args,
+)
+test_install_libs += test_aio
+
+test_install_data += files(
+ 'test_aio.control',
+ 'test_aio--1.0.sql',
+)
+
+tests += {
+ 'name': 'test_aio',
+ 'sd': meson.current_source_dir(),
+ 'bd': meson.current_build_dir(),
+ 'tap': {
+ 'env': {
+ 'enable_injection_points': get_option('injection_points') ? 'yes' : 'no',
+ },
+ 'tests': [
+ 't/001_aio.pl',
+ 't/002_io_workers.pl',
+ ],
+ },
+}
diff --git a/src/test/modules/test_aio/t/001_aio.pl b/src/test/modules/test_aio/t/001_aio.pl
new file mode 100644
index 00000000000..b0beddb568e
--- /dev/null
+++ b/src/test/modules/test_aio/t/001_aio.pl
@@ -0,0 +1,906 @@
+# Copyright (c) 2025, PostgreSQL Global Development Group
+
+use strict;
+use warnings FATAL => 'all';
+
+use PostgreSQL::Test::Cluster;
+use PostgreSQL::Test::Utils;
+use Test::More;
+
+
+###
+# Test io_method=worker
+###
+my $node_worker = create_node('worker');
+$node_worker->start();
+
+test_generic('worker', $node_worker);
+SKIP:
+{
+ skip 'Injection points not supported by this build', 1
+ unless $ENV{enable_injection_points} eq 'yes';
+ test_inject_worker('worker', $node_worker);
+}
+
+$node_worker->stop();
+
+
+###
+# Test io_method=io_uring
+###
+
+if (have_io_uring())
+{
+ my $node_uring = create_node('io_uring');
+ $node_uring->start();
+ test_generic('io_uring', $node_uring);
+ $node_uring->stop();
+}
+
+
+###
+# Test io_method=sync
+###
+
+my $node_sync = create_node('sync');
+
+# just to have one test not use the default auto-tuning
+
+$node_sync->append_conf(
+ 'postgresql.conf', qq(
+io_max_concurrency=4
+temp_buffers=100
+));
+
+$node_sync->start();
+test_generic('sync', $node_sync);
+$node_sync->stop();
+
+done_testing();
+
+
+###
+# Test Helpers
+###
+
+sub create_node
+{
+ local $Test::Builder::Level = $Test::Builder::Level + 1;
+
+ my $io_method = shift;
+
+ my $node = PostgreSQL::Test::Cluster->new($io_method);
+
+ # Want to test initdb for each IO method, otherwise we could just reuse
+ # the cluster.
+ #
+ # Unfortunately Cluster::init() puts PG_TEST_INITDB_EXTRA_OPTS after the
+ # options specified by ->extra, if somebody puts -c io_method=xyz in
+ # PG_TEST_INITDB_EXTRA_OPTS it would break this test. Fix that up if we
+ # detect it.
+ local $ENV{PG_TEST_INITDB_EXTRA_OPTS} = $ENV{PG_TEST_INITDB_EXTRA_OPTS};
+ if (defined $ENV{PG_TEST_INITDB_EXTRA_OPTS}
+ && $ENV{PG_TEST_INITDB_EXTRA_OPTS} =~ m/io_method=/)
+ {
+ $ENV{PG_TEST_INITDB_EXTRA_OPTS} .= " -c io_method=$io_method";
+ }
+
+ $node->init(extra => [ '-c', "io_method=$io_method" ]);
+
+ $node->append_conf(
+ 'postgresql.conf', qq(
+shared_preload_libraries=test_aio
+log_min_messages = 'DEBUG3'
+log_statement=all
+restart_after_crash=false
+));
+
+ ok(1, "$io_method: initdb");
+
+ return $node;
+}
+
+sub have_io_uring
+{
+ # To detect if io_uring is supported, we look at the error message for
+ # assigning an invalid value to an enum GUC, which lists all the valid
+ # options. We need to use -C to deal with running as administrator on
+ # windows, the superuser check is omitted if -C is used.
+ my ($stdout, $stderr) =
+ run_command [qw(postgres -C invalid -c io_method=invalid)];
+ die "can't determine supported io_method values"
+ unless $stderr =~ m/Available values: ([^\.]+)\./;
+ my $methods = $1;
+ note "supported io_method values are: $methods";
+
+ return ($methods =~ m/io_uring/) ? 1 : 0;
+}
+
+sub psql_like
+{
+ my $io_method = shift;
+ my $psql = shift;
+ my $name = shift;
+ my $sql = shift;
+ my $expected_stdout = shift;
+ my $expected_stderr = shift;
+ my ($cmdret, $output);
+
+ ($output, $cmdret) = $psql->query($sql);
+
+ like($output, $expected_stdout, "$io_method: $name: expected stdout");
+ like($psql->{stderr}, $expected_stderr,
+ "$io_method: $name: expected stderr");
+ $psql->{stderr} = '';
+
+ return $output;
+}
+
+sub query_wait_block
+{
+ my $io_method = shift;
+ my $node = shift;
+ my $psql = shift;
+ my $name = shift;
+ my $sql = shift;
+ my $waitfor = shift;
+
+ my $pid = $psql->query_safe('SELECT pg_backend_pid()');
+
+ $psql->{stdin} .= qq($sql;\n);
+ $psql->{run}->pump_nb();
+ ok(1, "$io_method: $name: issued sql");
+
+ $node->poll_query_until('postgres',
+ qq(SELECT wait_event FROM pg_stat_activity WHERE pid = $pid),
+ $waitfor,);
+ ok(1, "$io_method: $name: observed $waitfor wait event");
+}
+
+
+###
+# Sub-tests
+###
+
+sub test_handle
+{
+ my $io_method = shift;
+ my $node = shift;
+
+ my $psql = $node->background_psql('postgres', on_error_stop => 0);
+
+ # leak warning: implicit xact
+ psql_like(
+ $io_method,
+ $psql,
+ "handle_get() leak in implicit xact",
+ qq(SELECT handle_get()),
+ qr/^$/,
+ qr/leaked AIO handle/,
+ "$io_method: leaky handle_get() warns");
+
+ # leak warning: explicit xact
+ psql_like(
+ $io_method, $psql,
+ "handle_get() leak in explicit xact",
+ qq(BEGIN; SELECT handle_get(); COMMIT),
+ qr/^$/, qr/leaked AIO handle/);
+
+
+ # leak warning: explicit xact, rollback
+ psql_like(
+ $io_method,
+ $psql,
+ "handle_get() leak in explicit xact, rollback",
+ qq(BEGIN; SELECT handle_get(); ROLLBACK;),
+ qr/^$/,
+ qr/leaked AIO handle/);
+
+ # leak warning: subtrans
+ psql_like(
+ $io_method,
+ $psql,
+ "handle_get() leak in subxact",
+ qq(BEGIN; SAVEPOINT foo; SELECT handle_get(); COMMIT;),
+ qr/^$/,
+ qr/leaked AIO handle/);
+
+ # leak warning + error: released in different command (thus resowner)
+ psql_like(
+ $io_method,
+ $psql,
+ "handle_release() in different command",
+ qq(BEGIN; SELECT handle_get(); SELECT handle_release_last(); COMMIT;),
+ qr/^$/,
+ qr/leaked AIO handle.*release in unexpected state/ms);
+
+ # no leak, release in same command
+ psql_like(
+ $io_method,
+ $psql,
+ "handle_release() in same command",
+ qq(BEGIN; SELECT handle_get() UNION ALL SELECT handle_release_last(); COMMIT;),
+ qr/^$/,
+ qr/^$/);
+
+ # normal handle use
+ psql_like($io_method, $psql, "handle_get_release()",
+ qq(SELECT handle_get_release()),
+ qr/^$/, qr/^$/);
+
+ # should error out, API violation
+ psql_like($io_method, $psql, "handle_get_twice()",
+ qq(SELECT handle_get_release()),
+ qr/^$/, qr/^$/);
+
+ # recover after error in implicit xact
+ psql_like(
+ $io_method,
+ $psql,
+ "handle error recovery in implicit xact",
+ qq(SELECT handle_get_and_error(); SELECT 'ok', handle_get_release()),
+ qr/^|ok$/,
+ qr/ERROR.*as you command/);
+
+ # recover after error in implicit xact
+ psql_like(
+ $io_method,
+ $psql,
+ "handle error recovery in explicit xact",
+ qq(BEGIN; SELECT handle_get_and_error(); SELECT handle_get_release(), 'ok'; COMMIT;),
+ qr/^|ok$/,
+ qr/ERROR.*as you command/);
+
+ # recover after error in subtrans
+ psql_like(
+ $io_method,
+ $psql,
+ "handle error recovery in explicit subxact",
+ qq(BEGIN; SAVEPOINT foo; SELECT handle_get_and_error(); ROLLBACK TO SAVEPOINT foo; SELECT handle_get_release(); ROLLBACK;),
+ qr/^|ok$/,
+ qr/ERROR.*as you command/);
+
+ $psql->quit();
+}
+
+
+sub test_batch
+{
+ my $io_method = shift;
+ my $node = shift;
+
+ my $psql = $node->background_psql('postgres', on_error_stop => 0);
+
+ # leak warning & recovery: implicit xact
+ psql_like(
+ $io_method,
+ $psql,
+ "batch_start() leak & cleanup in implicit xact",
+ qq(SELECT batch_start()),
+ qr/^$/,
+ qr/open AIO batch at end/,
+ "$io_method: leaky batch_start() warns");
+
+ # leak warning & recovery: explicit xact
+ psql_like(
+ $io_method,
+ $psql,
+ "batch_start() leak & cleanup in explicit xact",
+ qq(BEGIN; SELECT batch_start(); COMMIT;),
+ qr/^$/,
+ qr/open AIO batch at end/,
+ "$io_method: leaky batch_start() warns");
+
+
+ # leak warning & recovery: explicit xact, rollback
+ #
+ # XXX: This doesn't fail right now, due to not getting a chance to do
+ # something at transaction command commit. That's not a correctness issue,
+ # it just means it's a bit harder to find buggy code.
+ #psql_like($io_method, $psql,
+ # "batch_start() leak & cleanup after abort",
+ # qq(BEGIN; SELECT batch_start(); ROLLBACK;),
+ # qr/^$/,
+ # qr/open AIO batch at end/, "$io_method: leaky batch_start() warns");
+
+ # no warning, batch closed in same command
+ psql_like(
+ $io_method,
+ $psql,
+ "batch_start(), batch_end() works",
+ qq(SELECT batch_start() UNION ALL SELECT batch_end()),
+ qr/^$/,
+ qr/^$/,
+ "$io_method: batch_start(), batch_end()");
+
+ $psql->quit();
+}
+
+sub test_io_error
+{
+ my $io_method = shift;
+ my $node = shift;
+ my ($ret, $output);
+
+ my $psql = $node->background_psql('postgres', on_error_stop => 0);
+
+ $psql->query_safe(
+ qq(
+CREATE TEMPORARY TABLE tmp_corr(data int not null);
+INSERT INTO tmp_corr SELECT generate_series(1, 10000);
+SELECT corrupt_rel_block('tmp_corr', 1);
+));
+
+ foreach my $tblname (qw(tbl_corr tmp_corr))
+ {
+ my $invalid_page_re =
+ $tblname eq 'tbl_corr'
+ ? qr/invalid page in block 1 of relation base\/\d+\/\d+/
+ : qr/invalid page in block 1 of relation base\/\d+\/t\d+_\d+/;
+
+ # verify the error is reported in custom C code
+ psql_like(
+ $io_method,
+ $psql,
+ "read_rel_block_ll() of $tblname page",
+ qq(SELECT read_rel_block_ll('$tblname', 1, wait_complete=>true)),
+ qr/^$/,
+ $invalid_page_re);
+
+ # verify the error is reported for bufmgr reads, seq scan
+ psql_like(
+ $io_method, $psql,
+ "sequential scan of $tblname block fails",
+ qq(SELECT count(*) FROM $tblname),
+ qr/^$/, $invalid_page_re);
+
+ # verify the error is reported for bufmgr reads, tid scan
+ psql_like(
+ $io_method,
+ $psql,
+ "tid scan of $tblname block fails",
+ qq(SELECT count(*) FROM $tblname WHERE ctid = '(1, 1)'),
+ qr/^$/,
+ $invalid_page_re);
+ }
+
+ $psql->quit();
+}
+
+# Test interplay between StartBufferIO and TerminateBufferIO
+sub test_startwait_io
+{
+ my $io_method = shift;
+ my $node = shift;
+ my ($ret, $output);
+
+ my $psql_a = $node->background_psql('postgres', on_error_stop => 0);
+ my $psql_b = $node->background_psql('postgres', on_error_stop => 0);
+
+
+ ### Verify behavior for normal tables
+
+ # create a buffer we can play around with
+ my $buf_id = psql_like(
+ $io_method, $psql_a,
+ "creation of toy buffer succeeds",
+ qq(SELECT buffer_create_toy('tbl_ok', 1)),
+ qr/^\d+$/, qr/^$/);
+
+ # check that one backend can perform StartBufferIO
+ psql_like(
+ $io_method,
+ $psql_a,
+ "first StartBufferIO",
+ qq(SELECT buffer_call_start_io($buf_id, for_input=>true, nowait=>false);),
+ qr/^t$/,
+ qr/^$/);
+
+ # but not twice on the same buffer (non-waiting)
+ psql_like(
+ $io_method,
+ $psql_a,
+ "second StartBufferIO fails, same session",
+ qq(SELECT buffer_call_start_io($buf_id, for_input=>true, nowait=>true);),
+ qr/^f$/,
+ qr/^$/);
+ psql_like(
+ $io_method,
+ $psql_b,
+ "second StartBufferIO fails, other session",
+ qq(SELECT buffer_call_start_io($buf_id, for_input=>true, nowait=>true);),
+ qr/^f$/,
+ qr/^$/);
+
+ # start io in a different session, will block
+ query_wait_block(
+ $io_method,
+ $node,
+ $psql_b,
+ "blocking start buffer io",
+ qq(SELECT buffer_call_start_io($buf_id, for_input=>true, nowait=>false);),
+ "BufferIo");
+
+ # Terminate the IO, without marking it as success, this should trigger the
+ # waiting session to be able to start the io
+ psql_like(
+ $io_method,
+ $psql_a,
+ "blocking start buffer io, terminating io, not valid",
+ qq(SELECT buffer_call_terminate_io($buf_id, for_input=>true, succeed=>false, io_error=>false, syncio=>true)),
+ qr/^$/,
+ qr/^$/);
+
+
+ # Because the IO was terminated, but not marked as valid, second session should get the right to start io
+ pump_until($psql_b->{run}, $psql_b->{timeout}, \$psql_b->{stdout}, qr/t/);
+ ok(1, "$io_method: blocking start buffer io, can start io");
+
+ # terminate the IO again
+ $psql_b->query_safe(
+ qq(SELECT buffer_call_terminate_io($buf_id, for_input=>true, succeed=>false, io_error=>false, syncio=>true);)
+ );
+
+
+ # same as the above scenario, but mark IO as having succeeded
+ psql_like(
+ $io_method,
+ $psql_a,
+ "blocking buffer io w/ success: first start buffer io",
+ qq(SELECT buffer_call_start_io($buf_id, for_input=>true, nowait=>false);),
+ qr/^t$/,
+ qr/^$/);
+
+ # start io in a different session, will block
+ query_wait_block(
+ $io_method,
+ $node,
+ $psql_b,
+ "blocking start buffer io",
+ qq(SELECT buffer_call_start_io($buf_id, for_input=>true, nowait=>false);),
+ "BufferIo");
+
+ # Terminate the IO, marking it as success
+ psql_like(
+ $io_method,
+ $psql_a,
+ "blocking start buffer io, terminating io, valid",
+ qq(SELECT buffer_call_terminate_io($buf_id, for_input=>true, succeed=>true, io_error=>false, syncio=>true)),
+ qr/^$/,
+ qr/^$/);
+
+ # Because the IO was terminated, and marked as valid, second session should complete but not need io
+ pump_until($psql_b->{run}, $psql_b->{timeout}, \$psql_b->{stdout}, qr/f/);
+ ok(1, "$io_method: blocking start buffer io, no need to start io");
+
+ # buffer is valid now, make it invalid again
+ $psql_a->query_safe(qq(SELECT buffer_create_toy('tbl_ok', 1);));
+
+
+ ### Verify behavior for temporary tables
+
+ # Can't unfortunately share the code with the normal table case, there are
+ # too many behavioral differences.
+
+ # create a buffer we can play around with
+ $psql_a->query_safe(
+ qq(
+CREATE TEMPORARY TABLE tmp_ok(data int not null);
+INSERT INTO tmp_ok SELECT generate_series(1, 10000);
+));
+ $buf_id = $psql_a->query_safe(qq(SELECT buffer_create_toy('tmp_ok', 3);));
+
+ # check that one backend can perform StartLocalBufferIO
+ psql_like(
+ $io_method,
+ $psql_a,
+ "first StartLocalBufferIO",
+ qq(SELECT buffer_call_start_io($buf_id, for_input=>true, nowait=>true);),
+ qr/^t$/,
+ qr/^$/);
+
+ # Because local buffers don't use IO_IN_PROGRESS, a second StartLocalBufer
+ # succeeds as well. This test mostly serves as a documentation of that
+ # fact. If we had actually started IO, it'd be different.
+ psql_like(
+ $io_method,
+ $psql_a,
+ "second StartLocalBufferIO succeeds, same session",
+ qq(SELECT buffer_call_start_io($buf_id, for_input=>true, nowait=>true);),
+ qr/^t$/,
+ qr/^$/);
+
+ # Terminate the IO again, without marking it as a success
+ $psql_a->query_safe(
+ qq(SELECT buffer_call_terminate_io($buf_id, for_input=>true, succeed=>false, io_error=>false, syncio=>true);)
+ );
+ psql_like(
+ $io_method,
+ $psql_a,
+ "StartLocalBufferIO after not marking valid succeeds, same session",
+ qq(SELECT buffer_call_start_io($buf_id, for_input=>true, nowait=>true);),
+ qr/^t$/,
+ qr/^$/);
+
+ # Terminate the IO again, marking it as a success
+ $psql_a->query_safe(
+ qq(SELECT buffer_call_terminate_io($buf_id, for_input=>true, succeed=>true, io_error=>false, syncio=>true);)
+ );
+
+ # Now another StartLocalBufferIO should fail, this time because the buffer
+ # is already valid.
+ psql_like(
+ $io_method,
+ $psql_a,
+ "StartLocalBufferIO after marking valid fails",
+ qq(SELECT buffer_call_start_io($buf_id, for_input=>true, nowait=>false);),
+ qr/^f$/,
+ qr/^$/);
+
+ $psql_a->quit();
+ $psql_b->quit();
+}
+
+# Test that if the backend issuing a read doesn't wait for the IO's
+# completion, another backend can complete the IO
+sub test_complete_foreign
+{
+ my $io_method = shift;
+ my $node = shift;
+ my ($ret, $output);
+
+ my $psql_a = $node->background_psql('postgres', on_error_stop => 0);
+ my $psql_b = $node->background_psql('postgres', on_error_stop => 0);
+
+ # Issue IO without waiting for completion, then sleep
+ $psql_a->query_safe(
+ qq(SELECT read_rel_block_ll('tbl_ok', 1, wait_complete=>false);));
+
+ # Check that another backend can read the relevant block
+ psql_like(
+ $io_method,
+ $psql_b,
+ "completing read started by sleeping backend",
+ qq(SELECT count(*) FROM tbl_ok WHERE ctid = '(1,1)' LIMIT 1),
+ qr/^1$/,
+ qr/^$/);
+
+ # Issue IO without waiting for completion, then exit
+ $psql_a->query_safe(
+ qq(SELECT read_rel_block_ll('tbl_ok', 1, wait_complete=>false);));
+ $psql_a->reconnect_and_clear();
+
+ # Check that another backend can read the relevant block
+ psql_like(
+ $io_method,
+ $psql_b,
+ "completing read started by exited backend",
+ qq(SELECT count(*) FROM tbl_ok WHERE ctid = '(1,1)' LIMIT 1),
+ qr/^1$/,
+ qr/^$/);
+
+ # Read a tbl_corr block, then sleep. The other session will retry the IO
+ # and also fail. The easiest thing to verify that seems to be to check
+ # that both are in the log.
+ my $log_location = -s $node->logfile;
+ $psql_a->query_safe(
+ qq(SELECT read_rel_block_ll('tbl_corr', 1, wait_complete=>false);));
+
+ psql_like(
+ $io_method,
+ $psql_b,
+ "completing read of tbl_corr block started by other backend",
+ qq(SELECT count(*) FROM tbl_corr WHERE ctid = '(1,1)' LIMIT 1),
+ qr/^$/,
+ qr/invalid page in block/);
+
+ # The log message issued for the read_rel_block_ll() should be logged as a LOG
+ $node->wait_for_log(qr/LOG[^\n]+invalid page in/, $log_location);
+ ok(1,
+ "$io_method: completing read of tbl_corr block started by other backend: LOG message for background read"
+ );
+
+ # But for the SELECT, it should be an ERROR
+ $log_location =
+ $node->wait_for_log(qr/ERROR[^\n]+invalid page in/, $log_location);
+ ok(1,
+ "$io_method: completing read of tbl_corr block started by other backend: ERROR message for foreground read"
+ );
+
+ $psql_a->quit();
+ $psql_b->quit();
+}
+
+# Test that we deal correctly with FDs being closed while IO is in progress
+sub test_close_fd
+{
+ my $io_method = shift;
+ my $node = shift;
+ my ($ret, $output);
+
+ my $psql = $node->background_psql('postgres', on_error_stop => 0);
+
+ psql_like(
+ $io_method,
+ $psql,
+ "close all FDs after read, waiting for results",
+ qq(
+ SELECT read_rel_block_ll('tbl_ok', 1,
+ wait_complete=>true,
+ batchmode_enter=>true,
+ smgrreleaseall=>true,
+ batchmode_exit=>true
+ );),
+ qr/^$/,
+ qr/^$/);
+
+ psql_like(
+ $io_method,
+ $psql,
+ "close all FDs after read, no waiting",
+ qq(
+ SELECT read_rel_block_ll('tbl_ok', 1,
+ wait_complete=>false,
+ batchmode_enter=>true,
+ smgrreleaseall=>true,
+ batchmode_exit=>true
+ );),
+ qr/^$/,
+ qr/^$/);
+
+ # Check that another backend can read the relevant block
+ psql_like(
+ $io_method,
+ $psql,
+ "close all FDs after read, no waiting, query works",
+ qq(SELECT count(*) FROM tbl_ok WHERE ctid = '(1,1)' LIMIT 1),
+ qr/^1$/,
+ qr/^$/);
+
+ $psql->quit();
+}
+
+sub test_inject
+{
+ my $io_method = shift;
+ my $node = shift;
+ my ($ret, $output);
+
+ my $psql = $node->background_psql('postgres', on_error_stop => 0);
+
+ # injected what we'd expect
+ $psql->query_safe(qq(SELECT inj_io_short_read_attach(8192);));
+ $psql->query_safe(qq(SELECT invalidate_rel_block('tbl_ok', 2);));
+ psql_like(
+ $io_method, $psql,
+ "injection point not triggering failure",
+ qq(SELECT count(*) FROM tbl_ok WHERE ctid = '(2, 1)'),
+ qr/^1$/, qr/^$/);
+
+ # injected a read shorter than a single block, expecting error
+ $psql->query_safe(qq(SELECT inj_io_short_read_attach(17);));
+ $psql->query_safe(qq(SELECT invalidate_rel_block('tbl_ok', 2);));
+ psql_like(
+ $io_method,
+ $psql,
+ "single block short read fails",
+ qq(SELECT count(*) FROM tbl_ok WHERE ctid = '(2, 1)'),
+ qr/^$/,
+ qr/ERROR:.*could not read blocks 2\.\.2 in file "base\/.*": read only 0 of 8192 bytes/
+ );
+
+ # shorten multi-block read to a single block, should retry
+ my $inval_query = qq(SELECT invalidate_rel_block('tbl_ok', 0);
+SELECT invalidate_rel_block('tbl_ok', 1);
+SELECT invalidate_rel_block('tbl_ok', 2);
+SELECT invalidate_rel_block('tbl_ok', 3);
+/* gap */
+SELECT invalidate_rel_block('tbl_ok', 5);
+SELECT invalidate_rel_block('tbl_ok', 6);
+SELECT invalidate_rel_block('tbl_ok', 7);
+SELECT invalidate_rel_block('tbl_ok', 8););
+
+ $psql->query_safe($inval_query);
+ $psql->query_safe(qq(SELECT inj_io_short_read_attach(8192);));
+ psql_like(
+ $io_method, $psql,
+ "multi block short read (1 block) is retried",
+ qq(SELECT count(*) FROM tbl_ok),
+ qr/^10000$/, qr/^$/);
+
+ # shorten multi-block read to two blocks, should retry
+ $psql->query_safe($inval_query);
+ $psql->query_safe(qq(SELECT inj_io_short_read_attach(8192*2);));
+
+ psql_like(
+ $io_method, $psql,
+ "multi block short read (2 blocks) is retried",
+ qq(SELECT count(*) FROM tbl_ok),
+ qr/^10000$/, qr/^$/);
+
+ # verify that page verification errors are detected even as part of a
+ # shortened multi-block read (tbl_corr, block 1 is corrupted)
+ $psql->query_safe(
+ qq(
+SELECT invalidate_rel_block('tbl_corr', 0);
+SELECT invalidate_rel_block('tbl_corr', 1);
+SELECT invalidate_rel_block('tbl_corr', 2);
+SELECT inj_io_short_read_attach(8192);
+ ));
+
+ psql_like(
+ $io_method,
+ $psql,
+ "shortened multi-block read detects invalid page",
+ qq(SELECT count(*) FROM tbl_corr WHERE ctid < '(2, 1)'),
+ qr/^$/,
+ qr/ERROR:.*invalid page in block 1 of relation base\/.*/);
+
+ # trigger a hard error, should error out
+ $psql->query_safe(
+ qq(
+SELECT inj_io_short_read_attach(-errno_from_string('EIO'));
+SELECT invalidate_rel_block('tbl_ok', 2);
+ ));
+
+ psql_like(
+ $io_method,
+ $psql,
+ "first hard IO error is reported",
+ qq(SELECT count(*) FROM tbl_ok),
+ qr/^$/,
+ qr/ERROR:.*could not read blocks 2\.\.2 in file \"base\/.*\": Input\/output error/
+ );
+
+ psql_like(
+ $io_method,
+ $psql,
+ "second hard IO error is reported",
+ qq(SELECT count(*) FROM tbl_ok),
+ qr/^$/,
+ qr/ERROR:.*could not read blocks 2\.\.2 in file \"base\/.*\": Input\/output error/
+ );
+
+ $psql->query_safe(qq(SELECT inj_io_short_read_detach()));
+
+ # now the IO should be ok.
+ psql_like(
+ $io_method, $psql,
+ "recovers after hard error",
+ qq(SELECT count(*) FROM tbl_ok),
+ qr/^10000$/, qr/^$/);
+
+ $psql->quit();
+}
+
+sub test_inject_worker
+{
+ my $io_method = shift;
+ my $node = shift;
+ my ($ret, $output);
+
+ my $psql = $node->background_psql('postgres', on_error_stop => 0);
+
+ # trigger a failure to reopen, should error out, but should recover
+ $psql->query_safe(
+ qq(
+SELECT inj_io_reopen_attach();
+SELECT invalidate_rel_block('tbl_ok', 1);
+ ));
+
+ psql_like(
+ $io_method,
+ $psql,
+ "failure to open: detected",
+ qq(SELECT count(*) FROM tbl_ok),
+ qr/^$/,
+ qr/ERROR:.*could not read blocks 1\.\.1 in file "base\/.*": No such file or directory/
+ );
+
+ $psql->query_safe(qq(SELECT inj_io_reopen_detach();));
+
+ # check that we indeed recover
+ psql_like(
+ $io_method, $psql,
+ "failure to open: recovers",
+ qq(SELECT count(*) FROM tbl_ok),
+ qr/^10000$/, qr/^$/);
+
+ $psql->quit();
+}
+
+sub test_invalidate
+{
+ my $io_method = shift;
+ my $node = shift;
+
+ my $psql = $node->background_psql('postgres', on_error_stop => 0);
+
+ foreach my $persistency (qw(normal unlogged temporary))
+ {
+ my $sql_persistency = $persistency eq 'normal' ? '' : $persistency;
+ my $tblname = $persistency . '_transactional';
+
+ my $create_sql = qq(
+CREATE $sql_persistency TABLE $tblname (id int not null, data text not null);
+INSERT INTO $tblname(id, data) SELECT generate_series(1, 10000) as id, repeat('a', 200);
+);
+
+ # Verify that outstanding read IO does not cause problems with
+ # AbortTransaction -> smgrDoPendingDeletes -> smgrdounlinkall -> ...
+ # -> Invalidate[Local]Buffer.
+ $psql->query_safe("BEGIN; $create_sql;");
+ $psql->query_safe(
+ qq(
+SELECT read_rel_block_ll('$tblname', 1, wait_complete=>false);
+));
+ psql_like(
+ $io_method,
+ $psql,
+ "rollback of newly created $persistency table with outstanding IO",
+ qq(ROLLBACK),
+ qr/^$/,
+ qr/^$/);
+
+ # Verify that outstanding read IO does not cause problems with
+ # CommitTransaction -> smgrDoPendingDeletes -> smgrdounlinkall -> ...
+ # -> Invalidate[Local]Buffer.
+ $psql->query_safe("BEGIN; $create_sql; COMMIT;");
+ $psql->query_safe(
+ qq(
+BEGIN;
+SELECT read_rel_block_ll('$tblname', 1, wait_complete=>false);
+));
+
+ psql_like(
+ $io_method, $psql,
+ "drop $persistency table with outstanding IO",
+ qq(DROP TABLE $tblname),
+ qr/^$/, qr/^$/);
+
+ psql_like($io_method, $psql,
+ "commit after drop $persistency table with outstanding IO",
+ qq(COMMIT), qr/^$/, qr/^$/);
+ }
+
+ $psql->quit();
+}
+
+sub test_generic
+{
+ my $io_method = shift;
+ my $node = shift;
+
+ is($node->safe_psql('postgres', 'SHOW io_method'),
+ $io_method, "$io_method: io_method set correctly");
+
+ $node->safe_psql(
+ 'postgres', qq(
+CREATE EXTENSION test_aio;
+CREATE TABLE tbl_corr(data int not null) WITH (AUTOVACUUM_ENABLED = false);
+CREATE TABLE tbl_ok(data int not null) WITH (AUTOVACUUM_ENABLED = false);
+
+INSERT INTO tbl_corr SELECT generate_series(1, 10000);
+INSERT INTO tbl_ok SELECT generate_series(1, 10000);
+SELECT grow_rel('tbl_corr', 16);
+SELECT grow_rel('tbl_ok', 16);
+
+SELECT corrupt_rel_block('tbl_corr', 1);
+CHECKPOINT;
+));
+
+ test_handle($io_method, $node);
+ test_io_error($io_method, $node);
+ test_batch($io_method, $node);
+ test_startwait_io($io_method, $node);
+ test_complete_foreign($io_method, $node);
+ test_close_fd($io_method, $node);
+ test_invalidate($io_method, $node);
+
+ SKIP:
+ {
+ skip 'Injection points not supported by this build', 1
+ unless $ENV{enable_injection_points} eq 'yes';
+ test_inject($io_method, $node);
+ }
+}
diff --git a/src/test/modules/test_aio/t/002_io_workers.pl b/src/test/modules/test_aio/t/002_io_workers.pl
new file mode 100644
index 00000000000..169119275ee
--- /dev/null
+++ b/src/test/modules/test_aio/t/002_io_workers.pl
@@ -0,0 +1,125 @@
+# Copyright (c) 2025, PostgreSQL Global Development Group
+
+use strict;
+use warnings FATAL => 'all';
+
+use PostgreSQL::Test::Cluster;
+use PostgreSQL::Test::Utils;
+use Test::More;
+use List::Util qw(sample);
+
+
+my $node = PostgreSQL::Test::Cluster->new('worker');
+$node->init();
+$node->append_conf(
+ 'postgresql.conf', qq(
+io_method=worker
+));
+
+$node->start();
+
+# Test changing the number of I/O worker processes while also evaluating the
+# handling of their termination.
+test_number_of_io_workers_dynamic($node);
+
+$node->stop();
+
+done_testing();
+
+
+sub test_number_of_io_workers_dynamic
+{
+ my $node = shift;
+
+ my $prev_worker_count = $node->safe_psql('postgres', 'SHOW io_workers');
+
+ # Verify that worker count can't be set to 0
+ change_number_of_io_workers($node, 0, $prev_worker_count, 1);
+
+ # Verify that worker count can't be set to 33 (above the max)
+ change_number_of_io_workers($node, 33, $prev_worker_count, 1);
+
+ # Try changing IO workers to a random value and verify that the worker
+ # count ends up as expected. Always test the min/max of workers.
+ #
+ # Valid range for io_workers is [1, 32]. 8 tests in total seems
+ # reasonable.
+ my @io_workers_range = sample 6, 1 ... 32;
+ foreach my $worker_count (1, 32, @io_workers_range)
+ {
+ $prev_worker_count =
+ change_number_of_io_workers($node, $worker_count,
+ $prev_worker_count, 0);
+ }
+}
+
+sub change_number_of_io_workers
+{
+ my $node = shift;
+ my $worker_count = shift;
+ my $prev_worker_count = shift;
+ my $expect_failure = shift;
+ my ($result, $stdout, $stderr);
+
+ ($result, $stdout, $stderr) =
+ $node->psql('postgres', "ALTER SYSTEM SET io_workers = $worker_count");
+ $node->safe_psql('postgres', 'SELECT pg_reload_conf()');
+
+ if ($expect_failure)
+ {
+ ok( $stderr =~
+ /$worker_count is outside the valid range for parameter "io_workers"/,
+ "updating number of io_workers to $worker_count failed, as expected"
+ );
+
+ return $prev_worker_count;
+ }
+ else
+ {
+ is( $node->safe_psql('postgres', 'SHOW io_workers'),
+ $worker_count,
+ "updating number of io_workers from $prev_worker_count to $worker_count"
+ );
+
+ check_io_worker_count($node, $worker_count);
+ terminate_io_worker($node, $worker_count);
+ check_io_worker_count($node, $worker_count);
+
+ return $worker_count;
+ }
+}
+
+sub terminate_io_worker
+{
+ my $node = shift;
+ my $worker_count = shift;
+ my ($pid, $ret);
+
+ # Select a random io worker
+ $pid = $node->safe_psql(
+ 'postgres',
+ qq(SELECT pid FROM pg_stat_activity WHERE
+ backend_type = 'io worker' ORDER BY RANDOM() LIMIT 1));
+
+ # terminate IO worker with SIGINT
+ is(PostgreSQL::Test::Utils::system_log('pg_ctl', 'kill', 'INT', $pid),
+ 0, "random io worker process signalled with INT");
+
+ # Check that worker exits
+ ok( $node->poll_query_until(
+ 'postgres',
+ qq(SELECT COUNT(*) FROM pg_stat_activity WHERE pid = $pid), '0'),
+ "random io worker process exited after signal");
+}
+
+sub check_io_worker_count
+{
+ my $node = shift;
+ my $worker_count = shift;
+
+ ok( $node->poll_query_until(
+ 'postgres',
+ qq(SELECT COUNT(*) FROM pg_stat_activity WHERE backend_type = 'io worker'),
+ $worker_count),
+ "io worker count is $worker_count");
+}
diff --git a/src/test/modules/test_aio/test_aio--1.0.sql b/src/test/modules/test_aio/test_aio--1.0.sql
new file mode 100644
index 00000000000..602615e39be
--- /dev/null
+++ b/src/test/modules/test_aio/test_aio--1.0.sql
@@ -0,0 +1,101 @@
+/* src/test/modules/test_aio/test_aio--1.0.sql */
+
+-- complain if script is sourced in psql, rather than via CREATE EXTENSION
+\echo Use "CREATE EXTENSION test_aio" to load this file. \quit
+
+
+CREATE FUNCTION errno_from_string(sym text)
+RETURNS pg_catalog.int4 STRICT
+AS 'MODULE_PATHNAME' LANGUAGE C;
+
+
+CREATE FUNCTION grow_rel(rel regclass, nblocks int)
+RETURNS pg_catalog.void STRICT
+AS 'MODULE_PATHNAME' LANGUAGE C;
+
+
+CREATE FUNCTION corrupt_rel_block(rel regclass, blockno int)
+RETURNS pg_catalog.void STRICT
+AS 'MODULE_PATHNAME' LANGUAGE C;
+
+CREATE FUNCTION read_rel_block_ll(rel regclass, blockno int,
+ wait_complete bool,
+ batchmode_enter bool DEFAULT false,
+ smgrreleaseall bool DEFAULT false,
+ batchmode_exit bool DEFAULT false)
+RETURNS pg_catalog.void STRICT
+AS 'MODULE_PATHNAME' LANGUAGE C;
+
+CREATE FUNCTION invalidate_rel_block(rel regclass, blockno int)
+RETURNS pg_catalog.void STRICT
+AS 'MODULE_PATHNAME' LANGUAGE C;
+
+CREATE FUNCTION buffer_create_toy(rel regclass, blockno int4)
+RETURNS pg_catalog.int4 STRICT
+AS 'MODULE_PATHNAME' LANGUAGE C;
+
+CREATE FUNCTION buffer_call_start_io(buffer int, for_input bool, nowait bool)
+RETURNS pg_catalog.bool STRICT
+AS 'MODULE_PATHNAME' LANGUAGE C;
+
+CREATE FUNCTION buffer_call_terminate_io(buffer int, for_input bool, succeed bool, io_error bool, syncio bool)
+RETURNS pg_catalog.void STRICT
+AS 'MODULE_PATHNAME' LANGUAGE C;
+
+
+
+/*
+ * Handle related functions
+ */
+CREATE FUNCTION handle_get_and_error()
+RETURNS pg_catalog.void STRICT
+AS 'MODULE_PATHNAME' LANGUAGE C;
+
+CREATE FUNCTION handle_get_twice()
+RETURNS pg_catalog.void STRICT
+AS 'MODULE_PATHNAME' LANGUAGE C;
+
+CREATE FUNCTION handle_get()
+RETURNS pg_catalog.void STRICT
+AS 'MODULE_PATHNAME' LANGUAGE C;
+
+CREATE FUNCTION handle_get_release()
+RETURNS pg_catalog.void STRICT
+AS 'MODULE_PATHNAME' LANGUAGE C;
+
+CREATE FUNCTION handle_release_last()
+RETURNS pg_catalog.void STRICT
+AS 'MODULE_PATHNAME' LANGUAGE C;
+
+
+/*
+ * Batchmode related functions
+ */
+CREATE FUNCTION batch_start()
+RETURNS pg_catalog.void STRICT
+AS 'MODULE_PATHNAME' LANGUAGE C;
+
+CREATE FUNCTION batch_end()
+RETURNS pg_catalog.void STRICT
+AS 'MODULE_PATHNAME' LANGUAGE C;
+
+
+
+/*
+ * Injection point related functions
+ */
+CREATE FUNCTION inj_io_short_read_attach(result int)
+RETURNS pg_catalog.void STRICT
+AS 'MODULE_PATHNAME' LANGUAGE C;
+
+CREATE FUNCTION inj_io_short_read_detach()
+RETURNS pg_catalog.void STRICT
+AS 'MODULE_PATHNAME' LANGUAGE C;
+
+CREATE FUNCTION inj_io_reopen_attach()
+RETURNS pg_catalog.void STRICT
+AS 'MODULE_PATHNAME' LANGUAGE C;
+
+CREATE FUNCTION inj_io_reopen_detach()
+RETURNS pg_catalog.void STRICT
+AS 'MODULE_PATHNAME' LANGUAGE C;
diff --git a/src/test/modules/test_aio/test_aio.c b/src/test/modules/test_aio/test_aio.c
new file mode 100644
index 00000000000..0f34892a17e
--- /dev/null
+++ b/src/test/modules/test_aio/test_aio.c
@@ -0,0 +1,712 @@
+/*-------------------------------------------------------------------------
+ *
+ * delay_execution.c
+ * Test module to allow delay between parsing and execution of a query.
+ *
+ * The delay is implemented by taking and immediately releasing a specified
+ * advisory lock. If another process has previously taken that lock, the
+ * current process will be blocked until the lock is released; otherwise,
+ * there's no effect. This allows an isolationtester script to reliably
+ * test behaviors where some specified action happens in another backend
+ * between parsing and execution of any desired query.
+ *
+ * Copyright (c) 2020-2025, PostgreSQL Global Development Group
+ *
+ * IDENTIFICATION
+ * src/test/modules/test_aio/test_aio.c
+ *
+ *-------------------------------------------------------------------------
+ */
+
+#include "postgres.h"
+
+#include "access/relation.h"
+#include "fmgr.h"
+#include "storage/aio.h"
+#include "storage/aio_internal.h"
+#include "storage/buf_internals.h"
+#include "storage/bufmgr.h"
+#include "storage/ipc.h"
+#include "storage/lwlock.h"
+#include "utils/builtins.h"
+#include "utils/injection_point.h"
+#include "utils/rel.h"
+
+
+PG_MODULE_MAGIC;
+
+
+typedef struct InjIoErrorState
+{
+ bool enabled_short_read;
+ bool enabled_reopen;
+
+ bool short_read_result_set;
+ int short_read_result;
+} InjIoErrorState;
+
+static InjIoErrorState * inj_io_error_state;
+
+/* Shared memory init callbacks */
+static shmem_request_hook_type prev_shmem_request_hook = NULL;
+static shmem_startup_hook_type prev_shmem_startup_hook = NULL;
+
+
+static PgAioHandle *last_handle;
+
+
+
+static void
+test_aio_shmem_request(void)
+{
+ if (prev_shmem_request_hook)
+ prev_shmem_request_hook();
+
+ RequestAddinShmemSpace(sizeof(InjIoErrorState));
+}
+
+static void
+test_aio_shmem_startup(void)
+{
+ bool found;
+
+ if (prev_shmem_startup_hook)
+ prev_shmem_startup_hook();
+
+ /* Create or attach to the shared memory state */
+ LWLockAcquire(AddinShmemInitLock, LW_EXCLUSIVE);
+
+ inj_io_error_state = ShmemInitStruct("injection_points",
+ sizeof(InjIoErrorState),
+ &found);
+
+ if (!found)
+ {
+ /*
+ * First time through, so initialize. This is shared with the dynamic
+ * initialization using a DSM.
+ */
+ inj_io_error_state->enabled_short_read = false;
+ inj_io_error_state->enabled_reopen = false;
+
+#ifdef USE_INJECTION_POINTS
+ InjectionPointAttach("AIO_PROCESS_COMPLETION_BEFORE_SHARED",
+ "test_aio",
+ "inj_io_short_read",
+ NULL,
+ 0);
+ InjectionPointLoad("AIO_PROCESS_COMPLETION_BEFORE_SHARED");
+
+ InjectionPointAttach("AIO_WORKER_AFTER_REOPEN",
+ "test_aio",
+ "inj_io_reopen",
+ NULL,
+ 0);
+ InjectionPointLoad("AIO_WORKER_AFTER_REOPEN");
+
+#endif
+ }
+ else
+ {
+#ifdef USE_INJECTION_POINTS
+ InjectionPointLoad("AIO_PROCESS_COMPLETION_BEFORE_SHARED");
+ InjectionPointLoad("AIO_WORKER_AFTER_REOPEN");
+ elog(LOG, "injection point loaded");
+#endif
+ }
+
+ LWLockRelease(AddinShmemInitLock);
+}
+
+void
+_PG_init(void)
+{
+ if (!process_shared_preload_libraries_in_progress)
+ return;
+
+ /* Shared memory initialization */
+ prev_shmem_request_hook = shmem_request_hook;
+ shmem_request_hook = test_aio_shmem_request;
+ prev_shmem_startup_hook = shmem_startup_hook;
+ shmem_startup_hook = test_aio_shmem_startup;
+}
+
+
+PG_FUNCTION_INFO_V1(errno_from_string);
+Datum
+errno_from_string(PG_FUNCTION_ARGS)
+{
+ const char *sym = text_to_cstring(PG_GETARG_TEXT_PP(0));
+
+ if (strcmp(sym, "EIO") == 0)
+ PG_RETURN_INT32(EIO);
+ else if (strcmp(sym, "EAGAIN") == 0)
+ PG_RETURN_INT32(EAGAIN);
+ else if (strcmp(sym, "EINTR") == 0)
+ PG_RETURN_INT32(EINTR);
+ else if (strcmp(sym, "ENOSPC") == 0)
+ PG_RETURN_INT32(ENOSPC);
+ else if (strcmp(sym, "EROFS") == 0)
+ PG_RETURN_INT32(EROFS);
+
+ ereport(ERROR,
+ errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ errmsg_internal("%s is not a supported errno value", sym));
+ PG_RETURN_INT32(0);
+}
+
+
+PG_FUNCTION_INFO_V1(grow_rel);
+Datum
+grow_rel(PG_FUNCTION_ARGS)
+{
+ Oid relid = PG_GETARG_OID(0);
+ uint32 nblocks = PG_GETARG_UINT32(1);
+ Relation rel;
+#define MAX_BUFFERS_TO_EXTEND_BY 64
+ Buffer victim_buffers[MAX_BUFFERS_TO_EXTEND_BY];
+
+ rel = relation_open(relid, AccessExclusiveLock);
+
+ while (nblocks > 0)
+ {
+ uint32 extend_by_pages;
+
+ extend_by_pages = Min(nblocks, MAX_BUFFERS_TO_EXTEND_BY);
+
+ ExtendBufferedRelBy(BMR_REL(rel),
+ MAIN_FORKNUM,
+ NULL,
+ 0,
+ extend_by_pages,
+ victim_buffers,
+ &extend_by_pages);
+
+ nblocks -= extend_by_pages;
+
+ for (uint32 i = 0; i < extend_by_pages; i++)
+ {
+ ReleaseBuffer(victim_buffers[i]);
+ }
+ }
+
+ relation_close(rel, NoLock);
+
+ PG_RETURN_VOID();
+}
+
+PG_FUNCTION_INFO_V1(corrupt_rel_block);
+Datum
+corrupt_rel_block(PG_FUNCTION_ARGS)
+{
+ Oid relid = PG_GETARG_OID(0);
+ BlockNumber blkno = PG_GETARG_UINT32(1);
+ Relation rel;
+ Buffer buf;
+ Page page;
+ PageHeader ph;
+
+ rel = relation_open(relid, AccessExclusiveLock);
+
+ buf = ReadBuffer(rel, blkno);
+ page = BufferGetPage(buf);
+
+ LockBuffer(buf, BUFFER_LOCK_EXCLUSIVE);
+
+ MarkBufferDirty(buf);
+
+ PageInit(page, BufferGetPageSize(buf), 0);
+
+ ph = (PageHeader) page;
+ ph->pd_special = BLCKSZ + 1;
+
+ /* ensure the on-disk state is updated */
+ if (BufferIsLocal(buf))
+ FlushLocalBuffer(GetLocalBufferDescriptor(-buf - 1), NULL);
+ else
+ FlushOneBuffer(buf);
+
+ LockBuffer(buf, BUFFER_LOCK_UNLOCK);
+
+ ReleaseBuffer(buf);
+
+ /*
+ * Don't want to have a buffer in-memory that's marked valid where the
+ * on-disk contents are invalid.
+ *
+ * NB: This is racy, better don't copy this to non-test code.
+ */
+ if (BufferIsLocal(buf))
+ InvalidateLocalBuffer(GetLocalBufferDescriptor(-buf - 1), true);
+ else
+ EvictUnpinnedBuffer(buf);
+
+ relation_close(rel, NoLock);
+
+ PG_RETURN_VOID();
+}
+
+/*
+ * Ensures a buffer for rel & blkno is in shared buffers, without actually
+ * caring about the buffer contents. Used to set up test scenarios.
+ */
+static Buffer
+create_toy_buffer(Relation rel, BlockNumber blkno)
+{
+ Buffer buf;
+ BufferDesc *buf_hdr;
+ uint32 buf_state;
+ bool was_pinned = false;
+
+ /* place buffer in shared buffers without erroring out */
+ buf = ReadBufferExtended(rel, MAIN_FORKNUM, blkno, RBM_ZERO_AND_LOCK, NULL);
+ LockBuffer(buf, BUFFER_LOCK_UNLOCK);
+
+ if (RelationUsesLocalBuffers(rel))
+ {
+ buf_hdr = GetLocalBufferDescriptor(-buf - 1);
+ buf_state = pg_atomic_read_u32(&buf_hdr->state);
+ }
+ else
+ {
+ buf_hdr = GetBufferDescriptor(buf - 1);
+ buf_state = LockBufHdr(buf_hdr);
+ }
+
+ /*
+ * We should be the only backend accessing this buffer. This is just a
+ * small bit of belt-and-suspenders defense, none of this code should ever
+ * run in a cluster with real data.
+ */
+ if (BUF_STATE_GET_REFCOUNT(buf_state) > 1)
+ was_pinned = true;
+ else
+ buf_state &= ~(BM_VALID | BM_DIRTY);
+
+ if (RelationUsesLocalBuffers(rel))
+ pg_atomic_unlocked_write_u32(&buf_hdr->state, buf_state);
+ else
+ UnlockBufHdr(buf_hdr, buf_state);
+
+ if (was_pinned)
+ elog(ERROR, "toy buffer %d was already pinned",
+ buf);
+
+ return buf;
+}
+
+PG_FUNCTION_INFO_V1(read_rel_block_ll);
+Datum
+read_rel_block_ll(PG_FUNCTION_ARGS)
+{
+ Oid relid = PG_GETARG_OID(0);
+ BlockNumber blkno = PG_GETARG_UINT32(1);
+ bool wait_complete = PG_GETARG_BOOL(2);
+ bool batchmode_enter = PG_GETARG_BOOL(3);
+ bool call_smgrreleaseall = PG_GETARG_BOOL(4);
+ bool batchmode_exit = PG_GETARG_BOOL(5);
+ Relation rel;
+ Buffer buf;
+ BufferDesc *buf_hdr;
+ Page pages[1];
+ PgAioReturn ior;
+ PgAioHandle *ioh;
+ PgAioWaitRef iow;
+ SMgrRelation smgr;
+
+ rel = relation_open(relid, AccessExclusiveLock);
+
+ buf = create_toy_buffer(rel, blkno);
+
+ pages[0] = BufferGetBlock(buf);
+
+ ioh = pgaio_io_acquire(CurrentResourceOwner, &ior);
+ pgaio_io_get_wref(ioh, &iow);
+
+ smgr = RelationGetSmgr(rel);
+
+ if (RelationUsesLocalBuffers(rel))
+ {
+ buf_hdr = GetLocalBufferDescriptor(-buf - 1);
+ StartLocalBufferIO(buf_hdr, true, false);
+ pgaio_io_set_flag(ioh, PGAIO_HF_REFERENCES_LOCAL);
+ }
+ else
+ {
+ buf_hdr = GetBufferDescriptor(buf - 1);
+ StartBufferIO(buf_hdr, true, false);
+ }
+
+ pgaio_io_set_handle_data_32(ioh, (uint32 *) &buf, 1);
+
+ pgaio_io_register_callbacks(ioh,
+ RelationUsesLocalBuffers(rel) ?
+ PGAIO_HCB_LOCAL_BUFFER_READV :
+ PGAIO_HCB_SHARED_BUFFER_READV,
+ 0);
+
+ if (batchmode_enter)
+ pgaio_enter_batchmode();
+
+ elog(LOG, "about to smgrstartreadv");
+ smgrstartreadv(ioh, smgr, MAIN_FORKNUM, blkno,
+ (void *) pages, 1);
+
+ if (call_smgrreleaseall)
+ smgrreleaseall();
+
+ if (batchmode_exit)
+ pgaio_exit_batchmode();
+
+ ReleaseBuffer(buf);
+
+ if (wait_complete)
+ {
+ pgaio_wref_wait(&iow);
+
+ if (ior.result.status != PGAIO_RS_OK)
+ pgaio_result_report(ior.result, &ior.target_data,
+ ior.result.status == PGAIO_RS_PARTIAL ? WARNING : ERROR);
+ }
+
+ relation_close(rel, NoLock);
+
+ PG_RETURN_VOID();
+}
+
+PG_FUNCTION_INFO_V1(invalidate_rel_block);
+Datum
+invalidate_rel_block(PG_FUNCTION_ARGS)
+{
+ Oid relid = PG_GETARG_OID(0);
+ BlockNumber blkno = PG_GETARG_UINT32(1);
+ Relation rel;
+ PrefetchBufferResult pr;
+ Buffer buf;
+
+ rel = relation_open(relid, AccessExclusiveLock);
+
+ /*
+ * This is a gross hack, but there's no other API exposed that allows to
+ * get a buffer ID without actually reading the block in.
+ */
+ pr = PrefetchBuffer(rel, MAIN_FORKNUM, blkno);
+ buf = pr.recent_buffer;
+
+ if (BufferIsValid(buf))
+ {
+ /* if the buffer contents aren't valid, this'll return false */
+ if (ReadRecentBuffer(rel->rd_locator, MAIN_FORKNUM, blkno, buf))
+ {
+ LockBuffer(buf, BUFFER_LOCK_EXCLUSIVE);
+ FlushOneBuffer(buf);
+ LockBuffer(buf, BUFFER_LOCK_UNLOCK);
+ ReleaseBuffer(buf);
+
+ if (!EvictUnpinnedBuffer(buf))
+ elog(ERROR, "couldn't evict");
+ }
+ }
+
+ relation_close(rel, AccessExclusiveLock);
+
+ PG_RETURN_VOID();
+}
+
+PG_FUNCTION_INFO_V1(buffer_create_toy);
+Datum
+buffer_create_toy(PG_FUNCTION_ARGS)
+{
+ Oid relid = PG_GETARG_OID(0);
+ BlockNumber blkno = PG_GETARG_UINT32(1);
+ Relation rel;
+ Buffer buf;
+
+ rel = relation_open(relid, AccessExclusiveLock);
+
+ buf = create_toy_buffer(rel, blkno);
+ ReleaseBuffer(buf);
+
+ relation_close(rel, NoLock);
+
+ PG_RETURN_INT32(buf);
+}
+
+PG_FUNCTION_INFO_V1(buffer_call_start_io);
+Datum
+buffer_call_start_io(PG_FUNCTION_ARGS)
+{
+ Buffer buf = PG_GETARG_INT32(0);
+ bool for_input = PG_GETARG_BOOL(1);
+ bool nowait = PG_GETARG_BOOL(2);
+ bool can_start;
+
+ if (BufferIsLocal(buf))
+ can_start = StartLocalBufferIO(GetLocalBufferDescriptor(-buf - 1),
+ for_input, nowait);
+ else
+ can_start = StartBufferIO(GetBufferDescriptor(buf - 1),
+ for_input, nowait);
+
+ /*
+ * For tests we don't want the resowner release preventing us from
+ * orchestrating odd scenarios.
+ */
+ if (can_start && !BufferIsLocal(buf))
+ ResourceOwnerForgetBufferIO(CurrentResourceOwner,
+ buf);
+
+ ereport(LOG,
+ errmsg("buffer %d after StartBufferIO: %s",
+ buf, DebugPrintBufferRefcount(buf)),
+ errhidestmt(true), errhidecontext(true));
+
+ PG_RETURN_BOOL(can_start);
+}
+
+PG_FUNCTION_INFO_V1(buffer_call_terminate_io);
+Datum
+buffer_call_terminate_io(PG_FUNCTION_ARGS)
+{
+ Buffer buf = PG_GETARG_INT32(0);
+ bool for_input = PG_GETARG_BOOL(1);
+ bool succeed = PG_GETARG_BOOL(2);
+ bool io_error = PG_GETARG_BOOL(3);
+ bool syncio = PG_GETARG_BOOL(4);
+ bool clear_dirty = false;
+ uint32 set_flag_bits = 0;
+
+ if (io_error)
+ set_flag_bits |= BM_IO_ERROR;
+
+ if (for_input)
+ {
+ clear_dirty = false;
+
+ if (succeed)
+ set_flag_bits |= BM_VALID;
+ }
+ else
+ {
+ if (succeed)
+ clear_dirty = true;
+ }
+
+ ereport(LOG,
+ errmsg("buffer %d before Terminate[Local]BufferIO: %s",
+ buf, DebugPrintBufferRefcount(buf)),
+ errhidestmt(true), errhidecontext(true));
+
+ if (BufferIsLocal(buf))
+ TerminateLocalBufferIO(GetLocalBufferDescriptor(-buf - 1),
+ clear_dirty, set_flag_bits, syncio);
+ else
+ TerminateBufferIO(GetBufferDescriptor(buf - 1),
+ clear_dirty, set_flag_bits, false, syncio);
+
+ ereport(LOG,
+ errmsg("buffer %d after Terminate[Local]BufferIO: %s",
+ buf, DebugPrintBufferRefcount(buf)),
+ errhidestmt(true), errhidecontext(true));
+
+ PG_RETURN_VOID();
+}
+
+PG_FUNCTION_INFO_V1(handle_get);
+Datum
+handle_get(PG_FUNCTION_ARGS)
+{
+ last_handle = pgaio_io_acquire(CurrentResourceOwner, NULL);
+
+ PG_RETURN_VOID();
+}
+
+PG_FUNCTION_INFO_V1(handle_release_last);
+Datum
+handle_release_last(PG_FUNCTION_ARGS)
+{
+ if (!last_handle)
+ elog(ERROR, "no handle");
+
+ pgaio_io_release(last_handle);
+
+ PG_RETURN_VOID();
+}
+
+PG_FUNCTION_INFO_V1(handle_get_and_error);
+Datum
+handle_get_and_error(PG_FUNCTION_ARGS)
+{
+ pgaio_io_acquire(CurrentResourceOwner, NULL);
+
+ elog(ERROR, "as you command");
+ PG_RETURN_VOID();
+}
+
+PG_FUNCTION_INFO_V1(handle_get_twice);
+Datum
+handle_get_twice(PG_FUNCTION_ARGS)
+{
+ pgaio_io_acquire(CurrentResourceOwner, NULL);
+ pgaio_io_acquire(CurrentResourceOwner, NULL);
+
+ PG_RETURN_VOID();
+}
+
+PG_FUNCTION_INFO_V1(handle_get_release);
+Datum
+handle_get_release(PG_FUNCTION_ARGS)
+{
+ PgAioHandle *handle;
+
+ handle = pgaio_io_acquire(CurrentResourceOwner, NULL);
+ pgaio_io_release(handle);
+
+ PG_RETURN_VOID();
+}
+
+PG_FUNCTION_INFO_V1(batch_start);
+Datum
+batch_start(PG_FUNCTION_ARGS)
+{
+ pgaio_enter_batchmode();
+ PG_RETURN_VOID();
+}
+
+PG_FUNCTION_INFO_V1(batch_end);
+Datum
+batch_end(PG_FUNCTION_ARGS)
+{
+ pgaio_exit_batchmode();
+ PG_RETURN_VOID();
+}
+
+#ifdef USE_INJECTION_POINTS
+extern PGDLLEXPORT void inj_io_short_read(const char *name, const void *private_data);
+extern PGDLLEXPORT void inj_io_reopen(const char *name, const void *private_data);
+
+void
+inj_io_short_read(const char *name, const void *private_data)
+{
+ PgAioHandle *ioh;
+
+ elog(LOG, "short read called: %d", inj_io_error_state->enabled_short_read);
+
+ if (inj_io_error_state->enabled_short_read)
+ {
+ ioh = pgaio_inj_io_get();
+
+ /*
+ * Only shorten reads that are actually longer than the target size,
+ * otherwise we can trigger over-reads.
+ */
+ if (inj_io_error_state->short_read_result_set
+ && ioh->op == PGAIO_OP_READV
+ && inj_io_error_state->short_read_result <= ioh->result)
+ {
+ struct iovec *iov = &pgaio_ctl->iovecs[ioh->iovec_off];
+ int32 old_result = ioh->result;
+ int32 new_result = inj_io_error_state->short_read_result;
+ int32 processed = 0;
+
+ ereport(LOG,
+ errmsg("short read, changing result from %d to %d",
+ old_result, new_result),
+ errhidestmt(true), errhidecontext(true));
+
+ /*
+ * The underlying IO actually completed OK, and thus the "invalid"
+ * portion of the IOV actually contains valid data. That can hide
+ * a lot of problems, e.g. if we were to wrongly mark a buffer,
+ * that wasn't read according to the shortened-read, IO as valid,
+ * the contents would look valid and we might miss a bug.
+ *
+ * To avoid that, iterate through the IOV and zero out the
+ * "failed" portion of the IO.
+ */
+ for (int i = 0; i < ioh->op_data.read.iov_length; i++)
+ {
+ if (processed + iov[i].iov_len <= new_result)
+ processed += iov[i].iov_len;
+ else if (processed <= new_result)
+ {
+ uint32 ok_part = new_result - processed;
+
+ memset((char *) iov[i].iov_base + ok_part, 0, iov[i].iov_len - ok_part);
+ processed += iov[i].iov_len;
+ }
+ else
+ {
+ memset((char *) iov[i].iov_base, 0, iov[i].iov_len);
+ }
+ }
+
+ ioh->result = new_result;
+ }
+ }
+}
+
+void
+inj_io_reopen(const char *name, const void *private_data)
+{
+ elog(LOG, "reopen called: %d", inj_io_error_state->enabled_reopen);
+
+ if (inj_io_error_state->enabled_reopen)
+ {
+ elog(ERROR, "injection point triggering failure to reopen ");
+ }
+}
+#endif
+
+PG_FUNCTION_INFO_V1(inj_io_short_read_attach);
+Datum
+inj_io_short_read_attach(PG_FUNCTION_ARGS)
+{
+#ifdef USE_INJECTION_POINTS
+ inj_io_error_state->enabled_short_read = true;
+ inj_io_error_state->short_read_result_set = !PG_ARGISNULL(0);
+ if (inj_io_error_state->short_read_result_set)
+ inj_io_error_state->short_read_result = PG_GETARG_INT32(0);
+#else
+ elog(ERROR, "injection points not supported");
+#endif
+
+ PG_RETURN_VOID();
+}
+
+PG_FUNCTION_INFO_V1(inj_io_short_read_detach);
+Datum
+inj_io_short_read_detach(PG_FUNCTION_ARGS)
+{
+#ifdef USE_INJECTION_POINTS
+ inj_io_error_state->enabled_short_read = false;
+#else
+ elog(ERROR, "injection points not supported");
+#endif
+ PG_RETURN_VOID();
+}
+
+PG_FUNCTION_INFO_V1(inj_io_reopen_attach);
+Datum
+inj_io_reopen_attach(PG_FUNCTION_ARGS)
+{
+#ifdef USE_INJECTION_POINTS
+ inj_io_error_state->enabled_reopen = true;
+#else
+ elog(ERROR, "injection points not supported");
+#endif
+
+ PG_RETURN_VOID();
+}
+
+PG_FUNCTION_INFO_V1(inj_io_reopen_detach);
+Datum
+inj_io_reopen_detach(PG_FUNCTION_ARGS)
+{
+#ifdef USE_INJECTION_POINTS
+ inj_io_error_state->enabled_reopen = false;
+#else
+ elog(ERROR, "injection points not supported");
+#endif
+ PG_RETURN_VOID();
+}
diff --git a/src/test/modules/test_aio/test_aio.control b/src/test/modules/test_aio/test_aio.control
new file mode 100644
index 00000000000..cd91c3ed16b
--- /dev/null
+++ b/src/test/modules/test_aio/test_aio.control
@@ -0,0 +1,3 @@
+comment = 'Test code for AIO'
+default_version = '1.0'
+module_pathname = '$libdir/test_aio'
--
2.48.1.76.g4e746b1a31.dirty
[text/x-diff] v2.12-0017-aio-bufmgr-Comment-fixes.patch (4.3K, ../../cxi6ssym62v7ywkt367qda3xik5jhsrcycwedyfqacinxooys5@pwarur37bosc/18-v2.12-0017-aio-bufmgr-Comment-fixes.patch)
download | inline diff:
From 71ad880d55e58e603d7a03ac1e57af00e8611826 Mon Sep 17 00:00:00 2001
From: Andres Freund <andres@anarazel.de>
Date: Thu, 20 Mar 2025 11:32:10 -0400
Subject: [PATCH v2.12 17/28] aio, bufmgr: Comment fixes
Some of these comments have been wrong for a while (12f3867f5534), some I
recently introduced (da7226993fd, 55b454d0e14). This also updates a comment in
FlushBuffer(), which will be copied in a future commit.
These changes seem big enough to be worth doing in separate commits.
Suggested-by: Noah Misch <noah@leadboat.com>
Discussion: https://postgr.es/m/20250319212530.80.nmisch@google.com
---
src/include/storage/aio.h | 2 +-
src/include/storage/aio_internal.h | 17 ++++++++++++++++-
src/backend/postmaster/postmaster.c | 2 +-
src/backend/storage/buffer/bufmgr.c | 10 ++++------
4 files changed, 22 insertions(+), 9 deletions(-)
diff --git a/src/include/storage/aio.h b/src/include/storage/aio.h
index 92f29cfdb71..9a0868a270c 100644
--- a/src/include/storage/aio.h
+++ b/src/include/storage/aio.h
@@ -80,7 +80,7 @@ typedef enum PgAioHandleFlags
/*
* The IO operations supported by the AIO subsystem.
*
- * This could be in aio_internal.h, as it is not pubicly referenced, but
+ * This could be in aio_internal.h, as it is not publicly referenced, but
* PgAioOpData currently *does* need to be public, therefore keeping this
* public seems to make sense.
*/
diff --git a/src/include/storage/aio_internal.h b/src/include/storage/aio_internal.h
index 400c44206dd..0dff909e4ad 100644
--- a/src/include/storage/aio_internal.h
+++ b/src/include/storage/aio_internal.h
@@ -276,6 +276,9 @@ typedef struct IoMethodOps
/*
* Start executing passed in IOs.
*
+ * Shall advance state to at least PGAIO_HS_SUBMITTED. (By the time this
+ * returns, other backends might have advanced the state further.)
+ *
* Will not be called if ->needs_synchronous_execution() returned true.
*
* num_staged_ios is <= PGAIO_SUBMIT_BATCH_SIZE.
@@ -284,12 +287,24 @@ typedef struct IoMethodOps
*/
int (*submit) (uint16 num_staged_ios, PgAioHandle **staged_ios);
- /*
+ /* ---
* Wait for the IO to complete. Optional.
*
+ * On return, state shall be on of
+ * - PGAIO_HS_COMPLETED_IO
+ * - PGAIO_HS_COMPLETED_SHARED
+ * - PGAIO_HS_COMPLETED_LOCAL
+ *
+ * The callback must not block if the handle is already in one of those
+ * states, or has been reused (see pgaio_io_was_recycled()). If, on
+ * return, the state is PGAIO_HS_COMPLETED_IO, state will reach
+ * PGAIO_HS_COMPLETED_SHARED without further intervention by the IO
+ * method.
+ *
* If not provided, it needs to be guaranteed that the IO method calls
* pgaio_io_process_completion() without further interaction by the
* issuing backend.
+ * ---
*/
void (*wait_one) (PgAioHandle *ioh,
uint64 ref_generation);
diff --git a/src/backend/postmaster/postmaster.c b/src/backend/postmaster/postmaster.c
index a0c37532d2f..c966c2e83af 100644
--- a/src/backend/postmaster/postmaster.c
+++ b/src/backend/postmaster/postmaster.c
@@ -4401,7 +4401,7 @@ maybe_adjust_io_workers(void)
++io_worker_count;
}
else
- break; /* XXX try again soon? */
+ break; /* try again next time */
}
/* Too many running? */
diff --git a/src/backend/storage/buffer/bufmgr.c b/src/backend/storage/buffer/bufmgr.c
index fcc5eb5654e..7f9b58003b3 100644
--- a/src/backend/storage/buffer/bufmgr.c
+++ b/src/backend/storage/buffer/bufmgr.c
@@ -4263,9 +4263,10 @@ FlushBuffer(BufferDesc *buf, SMgrRelation reln, IOObject io_object,
XLogFlush(recptr);
/*
- * Now it's safe to write buffer to disk. Note that no one else should
- * have been able to write it while we were busy with log flushing because
- * only one process at a time can set the BM_IO_IN_PROGRESS bit.
+ * Now it's safe to write the buffer to disk. Note that no one else should
+ * have been able to write it, while we were busy with log flushing,
+ * because we got the exclusive right to perform I/O by setting the
+ * BM_IO_IN_PROGRESS bit.
*/
bufBlock = BufHdrGetBlock(buf);
@@ -5855,9 +5856,6 @@ IsBufferCleanupOK(Buffer buffer)
/*
* Functions for buffer I/O handling
*
- * Note: We assume that nested buffer I/O never occurs.
- * i.e at most one BM_IO_IN_PROGRESS bit is set per proc.
- *
* Also note that these are used only for shared buffers, not local ones.
*/
--
2.48.1.76.g4e746b1a31.dirty
[text/x-diff] v2.12-0018-aio-Experimental-heuristics-to-increase-batchi.patch (1.6K, ../../cxi6ssym62v7ywkt367qda3xik5jhsrcycwedyfqacinxooys5@pwarur37bosc/19-v2.12-0018-aio-Experimental-heuristics-to-increase-batchi.patch)
download | inline diff:
From 9d1583dc515e77019cb97f3a302b2d178ae87585 Mon Sep 17 00:00:00 2001
From: Andres Freund <andres@anarazel.de>
Date: Tue, 18 Mar 2025 14:40:06 -0400
Subject: [PATCH v2.12 18/28] aio: Experimental heuristics to increase batching
in read_stream.c
Author:
Reviewed-by:
Discussion: https://postgr.es/m/
Backpatch:
---
src/backend/storage/aio/read_stream.c | 24 ++++++++++++++++++++++++
1 file changed, 24 insertions(+)
diff --git a/src/backend/storage/aio/read_stream.c b/src/backend/storage/aio/read_stream.c
index f675168e89a..039d7dc71a5 100644
--- a/src/backend/storage/aio/read_stream.c
+++ b/src/backend/storage/aio/read_stream.c
@@ -403,6 +403,30 @@ read_stream_start_pending_read(ReadStream *stream)
static void
read_stream_look_ahead(ReadStream *stream)
{
+ /*
+ * Batch-submitting multiple IOs is more efficient than doing so
+ * one-by-one. If we just ramp up to the max, we'll only be allowed to
+ * submit one io_combine_limit sized IO. Defer submitting IO in that case.
+ *
+ * FIXME: This needs better heuristics.
+ */
+#if 1
+ if (!stream->sync_mode && stream->distance > (io_combine_limit * 8))
+ {
+ if (stream->pinned_buffers + stream->pending_read_nblocks > ((stream->distance * 3) / 4))
+ {
+#if 0
+ ereport(LOG,
+ errmsg("reduce reduce reduce: pinned: %d, pending: %d, distance: %d",
+ stream->pinned_buffers,
+ stream->pending_read_nblocks,
+ stream->distance));
+#endif
+ return;
+ }
+ }
+#endif
+
/*
* Allow amortizing the cost of submitting IO over multiple IOs. This
* requires that we don't do any operations that could lead to a deadlock
--
2.48.1.76.g4e746b1a31.dirty
[text/x-diff] v2.12-0019-aio-Implement-smgr-md-fd-write-support.patch (13.7K, ../../cxi6ssym62v7ywkt367qda3xik5jhsrcycwedyfqacinxooys5@pwarur37bosc/20-v2.12-0019-aio-Implement-smgr-md-fd-write-support.patch)
download | inline diff:
From 4b718a6e20de7a6fdee662e327b986b4f429e55f Mon Sep 17 00:00:00 2001
From: Andres Freund <andres@anarazel.de>
Date: Tue, 18 Mar 2025 14:40:06 -0400
Subject: [PATCH v2.12 19/28] aio: Implement smgr/md/fd write support
TODO:
- register_dirty_segment_aio() can error out in edge cases
---
src/include/storage/fd.h | 1 +
src/include/storage/md.h | 5 +
src/include/storage/smgr.h | 5 +
src/backend/storage/aio/aio_callback.c | 1 +
src/backend/storage/file/fd.c | 28 ++++
src/backend/storage/smgr/md.c | 201 ++++++++++++++++++++++++-
src/backend/storage/smgr/smgr.c | 29 ++++
7 files changed, 269 insertions(+), 1 deletion(-)
diff --git a/src/include/storage/fd.h b/src/include/storage/fd.h
index b77d8e5e30e..2cc7c5a4761 100644
--- a/src/include/storage/fd.h
+++ b/src/include/storage/fd.h
@@ -112,6 +112,7 @@ extern int FilePrefetch(File file, off_t offset, off_t amount, uint32 wait_event
extern ssize_t FileReadV(File file, const struct iovec *iov, int iovcnt, off_t offset, uint32 wait_event_info);
extern ssize_t FileWriteV(File file, const struct iovec *iov, int iovcnt, off_t offset, uint32 wait_event_info);
extern int FileStartReadV(struct PgAioHandle *ioh, File file, int iovcnt, off_t offset, uint32 wait_event_info);
+extern int FileStartWriteV(struct PgAioHandle *ioh, File file, int iovcnt, off_t offset, uint32 wait_event_info);
extern int FileSync(File file, uint32 wait_event_info);
extern int FileZero(File file, off_t offset, off_t amount, uint32 wait_event_info);
extern int FileFallocate(File file, off_t offset, off_t amount, uint32 wait_event_info);
diff --git a/src/include/storage/md.h b/src/include/storage/md.h
index 9d7131eff43..47ae6c36c94 100644
--- a/src/include/storage/md.h
+++ b/src/include/storage/md.h
@@ -21,6 +21,7 @@
#include "storage/sync.h"
extern const PgAioHandleCallbacks aio_md_readv_cb;
+extern const PgAioHandleCallbacks aio_md_writev_cb;
/* md storage manager functionality */
extern void mdinit(void);
@@ -45,6 +46,10 @@ extern void mdstartreadv(PgAioHandle *ioh,
extern void mdwritev(SMgrRelation reln, ForkNumber forknum,
BlockNumber blocknum,
const void **buffers, BlockNumber nblocks, bool skipFsync);
+extern void mdstartwritev(PgAioHandle *ioh,
+ SMgrRelation reln, ForkNumber forknum,
+ BlockNumber blocknum,
+ const void **buffers, BlockNumber nblocks, bool skipFsync);
extern void mdwriteback(SMgrRelation reln, ForkNumber forknum,
BlockNumber blocknum, BlockNumber nblocks);
extern BlockNumber mdnblocks(SMgrRelation reln, ForkNumber forknum);
diff --git a/src/include/storage/smgr.h b/src/include/storage/smgr.h
index 856ebcda350..f00b3763ac9 100644
--- a/src/include/storage/smgr.h
+++ b/src/include/storage/smgr.h
@@ -108,6 +108,11 @@ extern void smgrwritev(SMgrRelation reln, ForkNumber forknum,
BlockNumber blocknum,
const void **buffers, BlockNumber nblocks,
bool skipFsync);
+extern void smgrstartwritev(PgAioHandle *ioh,
+ SMgrRelation reln, ForkNumber forknum,
+ BlockNumber blocknum,
+ const void **buffers, BlockNumber nblocks,
+ bool skipFsync);
extern void smgrwriteback(SMgrRelation reln, ForkNumber forknum,
BlockNumber blocknum, BlockNumber nblocks);
extern BlockNumber smgrnblocks(SMgrRelation reln, ForkNumber forknum);
diff --git a/src/backend/storage/aio/aio_callback.c b/src/backend/storage/aio/aio_callback.c
index 566060c12cb..12b0f1ebfa6 100644
--- a/src/backend/storage/aio/aio_callback.c
+++ b/src/backend/storage/aio/aio_callback.c
@@ -41,6 +41,7 @@ static const PgAioHandleCallbacksEntry aio_handle_cbs[] = {
CALLBACK_ENTRY(PGAIO_HCB_INVALID, aio_invalid_cb),
CALLBACK_ENTRY(PGAIO_HCB_MD_READV, aio_md_readv_cb),
+ CALLBACK_ENTRY(PGAIO_HCB_MD_WRITEV, aio_md_writev_cb),
CALLBACK_ENTRY(PGAIO_HCB_SHARED_BUFFER_READV, aio_shared_buffer_readv_cb),
diff --git a/src/backend/storage/file/fd.c b/src/backend/storage/file/fd.c
index f573aa8e7ca..8f143c10d36 100644
--- a/src/backend/storage/file/fd.c
+++ b/src/backend/storage/file/fd.c
@@ -2348,6 +2348,34 @@ retry:
return returnCode;
}
+int
+FileStartWriteV(PgAioHandle *ioh, File file,
+ int iovcnt, off_t offset,
+ uint32 wait_event_info)
+{
+ int returnCode;
+ Vfd *vfdP;
+
+ Assert(FileIsValid(file));
+
+ DO_DB(elog(LOG, "FileStartWriteV: %d (%s) " INT64_FORMAT " %d",
+ file, VfdCache[file].fileName,
+ (int64) offset,
+ iovcnt));
+
+ returnCode = FileAccess(file);
+ if (returnCode < 0)
+ return returnCode;
+
+ vfdP = &VfdCache[file];
+
+ /* FIXME: think about / reimplement temp_file_limit */
+
+ pgaio_io_prep_writev(ioh, vfdP->fd, iovcnt, offset);
+
+ return 0;
+}
+
int
FileSync(File file, uint32 wait_event_info)
{
diff --git a/src/backend/storage/smgr/md.c b/src/backend/storage/smgr/md.c
index 2218f17e7a0..f4a7af201ba 100644
--- a/src/backend/storage/smgr/md.c
+++ b/src/backend/storage/smgr/md.c
@@ -155,12 +155,19 @@ static BlockNumber _mdnblocks(SMgrRelation reln, ForkNumber forknum,
static PgAioResult md_readv_complete(PgAioHandle *ioh, PgAioResult prior_result, uint8 cb_data);
static void md_readv_report(PgAioResult result, const PgAioTargetData *target_data, int elevel);
+static PgAioResult md_writev_complete(PgAioHandle *ioh, PgAioResult prior_result, uint8 cb_data);
+static void md_writev_report(PgAioResult result, const PgAioTargetData *target_data, int elevel);
const PgAioHandleCallbacks aio_md_readv_cb = {
.complete_shared = md_readv_complete,
.report = md_readv_report,
};
+const PgAioHandleCallbacks aio_md_writev_cb = {
+ .complete_shared = md_writev_complete,
+ .report = md_writev_report,
+};
+
static inline int
_mdfd_open_flags(void)
@@ -1115,6 +1122,64 @@ mdwritev(SMgrRelation reln, ForkNumber forknum, BlockNumber blocknum,
}
}
+/*
+ * mdstartwritev() -- Asynchronous version of mdrwritev().
+ */
+void
+mdstartwritev(PgAioHandle *ioh,
+ SMgrRelation reln, ForkNumber forknum, BlockNumber blocknum,
+ const void **buffers, BlockNumber nblocks, bool skipFsync)
+{
+ off_t seekpos;
+ MdfdVec *v;
+ BlockNumber nblocks_this_segment;
+ struct iovec *iov;
+ int iovcnt;
+ int ret;
+
+ v = _mdfd_getseg(reln, forknum, blocknum, false,
+ EXTENSION_FAIL | EXTENSION_CREATE_RECOVERY);
+
+ seekpos = (off_t) BLCKSZ * (blocknum % ((BlockNumber) RELSEG_SIZE));
+
+ Assert(seekpos < (off_t) BLCKSZ * RELSEG_SIZE);
+
+ nblocks_this_segment =
+ Min(nblocks,
+ RELSEG_SIZE - (blocknum % ((BlockNumber) RELSEG_SIZE)));
+
+ if (nblocks_this_segment != nblocks)
+ elog(ERROR, "write crossing segment boundary");
+
+ iovcnt = pgaio_io_get_iovec(ioh, &iov);
+
+ Assert(nblocks <= iovcnt);
+
+ iovcnt = buffers_to_iovec(iov, unconstify(void **, buffers), nblocks_this_segment);
+
+ Assert(iovcnt <= nblocks_this_segment);
+
+ if (!(io_direct_flags & IO_DIRECT_DATA))
+ pgaio_io_set_flag(ioh, PGAIO_HF_BUFFERED);
+
+ pgaio_io_set_target_smgr(ioh,
+ reln,
+ forknum,
+ blocknum,
+ nblocks,
+ skipFsync);
+ pgaio_io_register_callbacks(ioh, PGAIO_HCB_MD_WRITEV, 0);
+
+ ret = FileStartWriteV(ioh, v->mdfd_vfd, iovcnt, seekpos, WAIT_EVENT_DATA_FILE_WRITE);
+ if (ret != 0)
+ ereport(ERROR,
+ (errcode_for_file_access(),
+ errmsg("could not start writing blocks %u..%u in file \"%s\": %m",
+ blocknum,
+ blocknum + nblocks_this_segment - 1,
+ FilePathName(v->mdfd_vfd))));
+}
+
/*
* mdwriteback() -- Tell the kernel to write pages back to storage.
@@ -1503,6 +1568,40 @@ register_dirty_segment(SMgrRelation reln, ForkNumber forknum, MdfdVec *seg)
}
}
+/*
+ * Like register_dirty_segment(), except for use by AIO. In the completion
+ * callback we don't have access to the MdfdVec (the completion callback might
+ * be executed in a different backend than the issuing backend), therefore we
+ * have to implement this slightly differently.
+ */
+static void
+register_dirty_segment_aio(RelFileLocator locator, ForkNumber forknum, uint64 segno)
+{
+ FileTag tag;
+
+ INIT_MD_FILETAG(tag, locator, forknum, segno);
+
+ /*
+ * Can't block here waiting for checkpointer to accept our sync request,
+ * as checkpointer might be waiting for this AIO to finish if offloaded to
+ * a worker.
+ */
+ if (!RegisterSyncRequest(&tag, SYNC_REQUEST, false /* retryOnError */ ))
+ {
+ char path[MAXPGPATH];
+
+ ereport(DEBUG1,
+ (errmsg_internal("could not forward fsync request because request queue is full")));
+
+ /* reuse mdsyncfiletag() to avoid duplicating code */
+ if (mdsyncfiletag(&tag, path))
+ ereport(data_sync_elevel(ERROR),
+ (errcode_for_file_access(),
+ errmsg("could not fsync file \"%s\": %m",
+ path)));
+ }
+}
+
/*
* register_unlink_segment() -- Schedule a file to be deleted after next checkpoint
*/
@@ -1987,7 +2086,7 @@ md_readv_complete(PgAioHandle *ioh, PgAioResult prior_result, uint8 cb_data)
* AIO error reporting callback for mdstartreadv().
*
* Errors are encoded as follows:
- * - PgAioResult.error_data != 0 encodes IO that failed with errno != 0
+ * - PgAioResult.error_data != 0 encodes IO that failed with that errno
* - PgAioResult.error_data == 0 encodes IO that didn't read all data
*/
static void
@@ -2027,3 +2126,103 @@ md_readv_report(PgAioResult result, const PgAioTargetData *td, int elevel)
td->smgr.nblocks * (size_t) BLCKSZ));
}
}
+
+/*
+ * AIO completion callback for mdstartwritev().
+ */
+static PgAioResult
+md_writev_complete(PgAioHandle *ioh, PgAioResult prior_result, uint8 cb_data)
+{
+ PgAioTargetData *td = pgaio_io_get_target_data(ioh);
+ PgAioResult result = prior_result;
+
+ if (prior_result.result < 0)
+ {
+ result.status = PGAIO_RS_ERROR;
+ result.id = PGAIO_HCB_MD_WRITEV;
+ /* For "hard" errors, track the error number in error_data */
+ result.error_data = -prior_result.result;
+ result.result = 0;
+
+ pgaio_result_report(result, td, LOG);
+
+ return result;
+ }
+
+ /*
+ * As explained above smgrstartwritev(), the smgr API operates on the
+ * level of blocks, rather than bytes. Convert.
+ */
+ result.result /= BLCKSZ;
+
+ Assert(result.result <= td->smgr.nblocks);
+
+ if (result.result == 0)
+ {
+ /* consider 0 blocks written a failure */
+ result.status = PGAIO_RS_ERROR;
+ result.id = PGAIO_HCB_MD_WRITEV;
+ result.error_data = 0;
+
+ pgaio_result_report(result, td, LOG);
+
+ return result;
+ }
+
+ if (result.status != PGAIO_RS_ERROR &&
+ result.result < td->smgr.nblocks)
+ {
+ /* partial writes should be retried at upper level */
+ result.status = PGAIO_RS_PARTIAL;
+ result.id = PGAIO_HCB_MD_WRITEV;
+ }
+
+ if (!td->smgr.skip_fsync)
+ register_dirty_segment_aio(td->smgr.rlocator, td->smgr.forkNum,
+ td->smgr.blockNum / ((BlockNumber) RELSEG_SIZE));
+
+ return result;
+}
+
+/*
+ * AIO error reporting callback for mdstartwritev().
+ */
+static void
+md_writev_report(PgAioResult result, const PgAioTargetData *td, int elevel)
+{
+ RelPathStr path;
+
+ path = relpathbackend(td->smgr.rlocator,
+ td->smgr.is_temp ? MyProcNumber : INVALID_PROC_NUMBER,
+ td->smgr.forkNum);
+
+ if (result.error_data != 0)
+ {
+ errno = result.error_data; /* for errcode_for_file_access() */
+
+ ereport(elevel,
+ errcode_for_file_access(),
+ errmsg("could not write blocks %u..%u in file \"%s\": %m",
+ td->smgr.blockNum,
+ td->smgr.blockNum + td->smgr.nblocks,
+ path.str)
+ );
+ }
+ else
+ {
+ /*
+ * NB: This will typically only be output in debug messages, while
+ * retrying a partial IO.
+ */
+ ereport(elevel,
+ errcode(ERRCODE_DATA_CORRUPTED),
+ errmsg("could not write blocks %u..%u in file \"%s\": wrote only %zu of %zu bytes",
+ td->smgr.blockNum,
+ td->smgr.blockNum + td->smgr.nblocks - 1,
+ path.str,
+ result.result * (size_t) BLCKSZ,
+ td->smgr.nblocks * (size_t) BLCKSZ
+ )
+ );
+ }
+}
diff --git a/src/backend/storage/smgr/smgr.c b/src/backend/storage/smgr/smgr.c
index bf8f57b410a..e56d6a2597b 100644
--- a/src/backend/storage/smgr/smgr.c
+++ b/src/backend/storage/smgr/smgr.c
@@ -115,6 +115,11 @@ typedef struct f_smgr
BlockNumber blocknum,
const void **buffers, BlockNumber nblocks,
bool skipFsync);
+ void (*smgr_startwritev) (PgAioHandle *ioh,
+ SMgrRelation reln, ForkNumber forknum,
+ BlockNumber blocknum,
+ const void **buffers, BlockNumber nblocks,
+ bool skipFsync);
void (*smgr_writeback) (SMgrRelation reln, ForkNumber forknum,
BlockNumber blocknum, BlockNumber nblocks);
BlockNumber (*smgr_nblocks) (SMgrRelation reln, ForkNumber forknum);
@@ -142,6 +147,7 @@ static const f_smgr smgrsw[] = {
.smgr_readv = mdreadv,
.smgr_startreadv = mdstartreadv,
.smgr_writev = mdwritev,
+ .smgr_startwritev = mdstartwritev,
.smgr_writeback = mdwriteback,
.smgr_nblocks = mdnblocks,
.smgr_truncate = mdtruncate,
@@ -787,6 +793,29 @@ smgrwritev(SMgrRelation reln, ForkNumber forknum, BlockNumber blocknum,
RESUME_INTERRUPTS();
}
+/*
+ * smgrstartwritev() -- asynchronous version of smgrwritev()
+ *
+ * This starts an asynchronous writev IO using the IO handle `ioh`. Other than
+ * `ioh` all parameters are the same as smgrwritev().
+ *
+ * Completion callbacks above smgr will be passed the result as the number of
+ * successfully written blocks if the write [partially] succeeds. This
+ * maintains the abstraction that smgr operates on the level of blocks, rather
+ * than bytes.
+ */
+void
+smgrstartwritev(PgAioHandle *ioh,
+ SMgrRelation reln, ForkNumber forknum, BlockNumber blocknum,
+ const void **buffers, BlockNumber nblocks, bool skipFsync)
+{
+ HOLD_INTERRUPTS();
+ smgrsw[reln->smgr_which].smgr_startwritev(ioh,
+ reln, forknum, blocknum, buffers,
+ nblocks, skipFsync);
+ RESUME_INTERRUPTS();
+}
+
/*
* smgrwriteback() -- Trigger kernel writeback for the supplied range of
* blocks.
--
2.48.1.76.g4e746b1a31.dirty
[text/x-diff] v2.12-0020-aio-Add-bounce-buffers.patch (23.2K, ../../cxi6ssym62v7ywkt367qda3xik5jhsrcycwedyfqacinxooys5@pwarur37bosc/21-v2.12-0020-aio-Add-bounce-buffers.patch)
download | inline diff:
From 5836eeb338e7866f9c4bcd75a2db92cf58bf0995 Mon Sep 17 00:00:00 2001
From: Andres Freund <andres@anarazel.de>
Date: Tue, 18 Mar 2025 14:40:06 -0400
Subject: [PATCH v2.12 20/28] aio: Add bounce buffers
---
src/include/storage/aio.h | 15 ++
src/include/storage/aio_internal.h | 34 ++++
src/include/storage/aio_types.h | 2 +
src/include/utils/resowner.h | 2 +
src/backend/storage/aio/README.md | 27 +++
src/backend/storage/aio/aio.c | 178 ++++++++++++++++++
src/backend/storage/aio/aio_init.c | 123 ++++++++++++
src/backend/utils/misc/guc_tables.c | 13 ++
src/backend/utils/misc/postgresql.conf.sample | 2 +
src/backend/utils/resowner/resowner.c | 25 ++-
src/test/modules/test_aio/test_aio--1.0.sql | 21 +++
src/test/modules/test_aio/test_aio.c | 55 ++++++
src/tools/pgindent/typedefs.list | 1 +
13 files changed, 496 insertions(+), 2 deletions(-)
diff --git a/src/include/storage/aio.h b/src/include/storage/aio.h
index 9a0868a270c..09d6d9fe1bc 100644
--- a/src/include/storage/aio.h
+++ b/src/include/storage/aio.h
@@ -349,6 +349,20 @@ extern bool pgaio_have_staged(void);
+/* --------------------------------------------------------------------------------
+ * Bounce Buffers
+ * --------------------------------------------------------------------------------
+ */
+
+extern PgAioBounceBuffer *pgaio_bounce_buffer_get(void);
+extern void pgaio_io_assoc_bounce_buffer(PgAioHandle *ioh, PgAioBounceBuffer *bb);
+extern uint32 pgaio_bounce_buffer_id(PgAioBounceBuffer *bb);
+extern void pgaio_bounce_buffer_release(PgAioBounceBuffer *bb);
+extern char *pgaio_bounce_buffer_buffer(PgAioBounceBuffer *bb);
+extern void pgaio_bounce_buffer_release_resowner(struct dlist_node *bb_node, bool on_error);
+
+
+
/* --------------------------------------------------------------------------------
* Other
* --------------------------------------------------------------------------------
@@ -361,6 +375,7 @@ extern void pgaio_closing_fd(int fd);
/* GUCs */
extern PGDLLIMPORT int io_method;
extern PGDLLIMPORT int io_max_concurrency;
+extern PGDLLIMPORT int io_bounce_buffers;
#endif /* AIO_H */
diff --git a/src/include/storage/aio_internal.h b/src/include/storage/aio_internal.h
index 0dff909e4ad..776c6985220 100644
--- a/src/include/storage/aio_internal.h
+++ b/src/include/storage/aio_internal.h
@@ -122,6 +122,12 @@ struct PgAioHandle
/* raw result of the IO operation */
int32 result;
+ /*
+ * List of bounce_buffers owned by IO. It would suffice to use an index
+ * based linked list here.
+ */
+ slist_head bounce_buffers;
+
/**
* In which list the handle is registered, depends on the state:
* - IDLE, in per-backend list
@@ -177,11 +183,24 @@ struct PgAioHandle
};
+/* typedef is in aio_types.h */
+struct PgAioBounceBuffer
+{
+ slist_node node;
+ struct ResourceOwnerData *resowner;
+ dlist_node resowner_node;
+ char *buffer;
+};
+
+
typedef struct PgAioBackend
{
/* index into PgAioCtl->io_handles */
uint32 io_handle_off;
+ /* index into PgAioCtl->bounce_buffers */
+ uint32 bounce_buffers_off;
+
/* IO Handles that currently are not used */
dclist_head idle_ios;
@@ -212,6 +231,12 @@ typedef struct PgAioBackend
* IOs being appended at the end.
*/
dclist_head in_flight_ios;
+
+ /* Bounce Buffers that currently are not used */
+ slist_head idle_bbs;
+
+ /* see handed_out_io */
+ PgAioBounceBuffer *handed_out_bb;
} PgAioBackend;
@@ -239,6 +264,15 @@ typedef struct PgAioCtl
uint32 io_handle_count;
PgAioHandle *io_handles;
+
+ /*
+ * To perform AIO on buffers that are not located in shared memory (either
+ * because they are not in shared memory or because we need to operate on
+ * a copy, as e.g. the case for writes when checksums are in use)
+ */
+ uint32 bounce_buffers_count;
+ PgAioBounceBuffer *bounce_buffers;
+ char *bounce_buffers_data;
} PgAioCtl;
diff --git a/src/include/storage/aio_types.h b/src/include/storage/aio_types.h
index debe8163d4e..02950ee56a7 100644
--- a/src/include/storage/aio_types.h
+++ b/src/include/storage/aio_types.h
@@ -118,4 +118,6 @@ typedef struct PgAioReturn
} PgAioReturn;
+typedef struct PgAioBounceBuffer PgAioBounceBuffer;
+
#endif /* AIO_TYPES_H */
diff --git a/src/include/utils/resowner.h b/src/include/utils/resowner.h
index aede4bfc820..7e2ec224169 100644
--- a/src/include/utils/resowner.h
+++ b/src/include/utils/resowner.h
@@ -168,5 +168,7 @@ extern void ResourceOwnerForgetLock(ResourceOwner owner, struct LOCALLOCK *local
struct dlist_node;
extern void ResourceOwnerRememberAioHandle(ResourceOwner owner, struct dlist_node *ioh_node);
extern void ResourceOwnerForgetAioHandle(ResourceOwner owner, struct dlist_node *ioh_node);
+extern void ResourceOwnerRememberAioBounceBuffer(ResourceOwner owner, struct dlist_node *bb_node);
+extern void ResourceOwnerForgetAioBounceBuffer(ResourceOwner owner, struct dlist_node *bb_node);
#endif /* RESOWNER_H */
diff --git a/src/backend/storage/aio/README.md b/src/backend/storage/aio/README.md
index 24eec5776cd..b7d43ba8422 100644
--- a/src/backend/storage/aio/README.md
+++ b/src/backend/storage/aio/README.md
@@ -405,6 +405,33 @@ shared memory no less!), completion callbacks instead have to encode errors in
a more compact format that can be converted into an error message.
+### AIO Bounce Buffers
+
+For some uses of AIO there is no convenient memory location as the source /
+destination of an AIO. E.g. when data checksums are enabled, writes from
+shared buffers currently cannot be done directly from shared buffers, as a
+shared buffer lock still allows some modification, e.g., for hint bits(see
+`FlushBuffer()`). If the write were done in-place, such modifications can
+cause the checksum to fail.
+
+For synchronous IO this is solved by copying the buffer to separate memory
+before computing the checksum and using that copy as the source buffer for the
+AIO.
+
+However, for AIO that is not a workable solution:
+- Instead of a single buffer many buffers are required, as many IOs might be
+ in flight
+- When using the [worker method](#worker), the source/target of IO needs to be
+ in shared memory, otherwise the workers won't be able to access the memory.
+
+The AIO subsystem addresses this by providing a limited number of bounce
+buffers that can be used as the source / target for IO. A bounce buffer be
+acquired with `pgaio_bounce_buffer_get()` and multiple bounce buffers can be
+associated with an AIO Handle with `pgaio_io_assoc_bounce_buffer()`.
+
+Bounce buffers are automatically released when the IO completes.
+
+
## Helpers
Using the low-level AIO API introduces too much complexity to do so all over
diff --git a/src/backend/storage/aio/aio.c b/src/backend/storage/aio/aio.c
index 0d52f72dfdc..f50ae7cdc23 100644
--- a/src/backend/storage/aio/aio.c
+++ b/src/backend/storage/aio/aio.c
@@ -62,6 +62,8 @@ static PgAioHandle *pgaio_io_from_wref(PgAioWaitRef *iow, uint64 *ref_generation
static const char *pgaio_io_state_get_name(PgAioHandleState s);
static void pgaio_io_wait(PgAioHandle *ioh, uint64 ref_generation);
+static void pgaio_bounce_buffer_wait_for_free(void);
+
/* Options for io_method. */
const struct config_enum_entry io_method_options[] = {
@@ -76,6 +78,7 @@ const struct config_enum_entry io_method_options[] = {
/* GUCs */
int io_method = DEFAULT_IO_METHOD;
int io_max_concurrency = -1;
+int io_bounce_buffers = -1;
/* global control for AIO */
PgAioCtl *pgaio_ctl;
@@ -662,6 +665,21 @@ pgaio_io_reclaim(PgAioHandle *ioh)
if (ioh->state != PGAIO_HS_HANDED_OUT)
dclist_delete_from(&pgaio_my_backend->in_flight_ios, &ioh->node);
+ /* reclaim all associated bounce buffers */
+ if (!slist_is_empty(&ioh->bounce_buffers))
+ {
+ slist_mutable_iter it;
+
+ slist_foreach_modify(it, &ioh->bounce_buffers)
+ {
+ PgAioBounceBuffer *bb = slist_container(PgAioBounceBuffer, node, it.cur);
+
+ slist_delete_current(&it);
+
+ slist_push_head(&pgaio_my_backend->idle_bbs, &bb->node);
+ }
+ }
+
if (ioh->resowner)
{
ResourceOwnerForgetAioHandle(ioh->resowner, &ioh->resowner_node);
@@ -1044,6 +1062,166 @@ pgaio_submit_staged(void)
+/* --------------------------------------------------------------------------------
+ * Functions primarily related to PgAioBounceBuffer
+ * --------------------------------------------------------------------------------
+ */
+
+PgAioBounceBuffer *
+pgaio_bounce_buffer_get(void)
+{
+ PgAioBounceBuffer *bb = NULL;
+ slist_node *node;
+
+ if (pgaio_my_backend->handed_out_bb != NULL)
+ elog(ERROR, "can only hand out one BB");
+
+ /*
+ * XXX: It probably is not a good idea to have bounce buffers be per
+ * backend, that's a fair bit of memory.
+ */
+ if (slist_is_empty(&pgaio_my_backend->idle_bbs))
+ {
+ pgaio_bounce_buffer_wait_for_free();
+ }
+
+ node = slist_pop_head_node(&pgaio_my_backend->idle_bbs);
+ bb = slist_container(PgAioBounceBuffer, node, node);
+
+ pgaio_my_backend->handed_out_bb = bb;
+
+ bb->resowner = CurrentResourceOwner;
+ ResourceOwnerRememberAioBounceBuffer(bb->resowner, &bb->resowner_node);
+
+ return bb;
+}
+
+void
+pgaio_io_assoc_bounce_buffer(PgAioHandle *ioh, PgAioBounceBuffer *bb)
+{
+ if (pgaio_my_backend->handed_out_bb != bb)
+ elog(ERROR, "can only assign handed out BB");
+ pgaio_my_backend->handed_out_bb = NULL;
+
+ /*
+ * There can be many bounce buffers assigned in case of vectorized IOs.
+ */
+ slist_push_head(&ioh->bounce_buffers, &bb->node);
+
+ /* once associated with an IO, the IO has ownership */
+ ResourceOwnerForgetAioBounceBuffer(bb->resowner, &bb->resowner_node);
+ bb->resowner = NULL;
+}
+
+uint32
+pgaio_bounce_buffer_id(PgAioBounceBuffer *bb)
+{
+ return bb - pgaio_ctl->bounce_buffers;
+}
+
+void
+pgaio_bounce_buffer_release(PgAioBounceBuffer *bb)
+{
+ if (pgaio_my_backend->handed_out_bb != bb)
+ elog(ERROR, "can only release handed out BB");
+
+ slist_push_head(&pgaio_my_backend->idle_bbs, &bb->node);
+ pgaio_my_backend->handed_out_bb = NULL;
+
+ ResourceOwnerForgetAioBounceBuffer(bb->resowner, &bb->resowner_node);
+ bb->resowner = NULL;
+}
+
+void
+pgaio_bounce_buffer_release_resowner(dlist_node *bb_node, bool on_error)
+{
+ PgAioBounceBuffer *bb = dlist_container(PgAioBounceBuffer, resowner_node, bb_node);
+
+ Assert(bb->resowner);
+
+ if (!on_error)
+ elog(WARNING, "leaked AIO bounce buffer");
+
+ pgaio_bounce_buffer_release(bb);
+}
+
+char *
+pgaio_bounce_buffer_buffer(PgAioBounceBuffer *bb)
+{
+ return bb->buffer;
+}
+
+static void
+pgaio_bounce_buffer_wait_for_free(void)
+{
+ static uint32 lastpos = 0;
+
+ if (pgaio_my_backend->num_staged_ios > 0)
+ {
+ pgaio_debug(DEBUG2, "submitting %d, while acquiring free bb",
+ pgaio_my_backend->num_staged_ios);
+ pgaio_submit_staged();
+ }
+
+ for (uint32 i = lastpos; i < lastpos + io_max_concurrency; i++)
+ {
+ uint32 thisoff = pgaio_my_backend->io_handle_off + (i % io_max_concurrency);
+ PgAioHandle *ioh = &pgaio_ctl->io_handles[thisoff];
+
+ switch (ioh->state)
+ {
+ case PGAIO_HS_IDLE:
+ case PGAIO_HS_HANDED_OUT:
+ continue;
+ case PGAIO_HS_DEFINED: /* should have been submitted above */
+ case PGAIO_HS_STAGED:
+ elog(ERROR, "shouldn't get here with io:%d in state %d",
+ pgaio_io_get_id(ioh), ioh->state);
+ break;
+ case PGAIO_HS_COMPLETED_IO:
+ case PGAIO_HS_SUBMITTED:
+ if (!slist_is_empty(&ioh->bounce_buffers))
+ {
+ pgaio_debug_io(DEBUG2, ioh,
+ "waiting for IO to reclaim BB with %d in flight",
+ dclist_count(&pgaio_my_backend->in_flight_ios));
+
+ /* see comment in pgaio_io_wait_for_free() about raciness */
+ pgaio_io_wait(ioh, ioh->generation);
+
+ if (slist_is_empty(&pgaio_my_backend->idle_bbs))
+ elog(WARNING, "empty after wait");
+
+ if (!slist_is_empty(&pgaio_my_backend->idle_bbs))
+ {
+ lastpos = i;
+ return;
+ }
+ }
+ break;
+ case PGAIO_HS_COMPLETED_SHARED:
+ case PGAIO_HS_COMPLETED_LOCAL:
+ /* reclaim */
+ pgaio_io_reclaim(ioh);
+
+ if (!slist_is_empty(&pgaio_my_backend->idle_bbs))
+ {
+ lastpos = i;
+ return;
+ }
+ break;
+ }
+ }
+
+ /*
+ * The submission above could have caused the IO to complete at any time.
+ */
+ if (slist_is_empty(&pgaio_my_backend->idle_bbs))
+ elog(PANIC, "no more bbs");
+}
+
+
+
/* --------------------------------------------------------------------------------
* Other
* --------------------------------------------------------------------------------
diff --git a/src/backend/storage/aio/aio_init.c b/src/backend/storage/aio/aio_init.c
index 885c3940c66..95b10933fed 100644
--- a/src/backend/storage/aio/aio_init.c
+++ b/src/backend/storage/aio/aio_init.c
@@ -88,6 +88,32 @@ AioHandleDataShmemSize(void)
io_max_concurrency));
}
+static Size
+AioBounceBufferDescShmemSize(void)
+{
+ Size sz;
+
+ /* PgAioBounceBuffer itself */
+ sz = mul_size(sizeof(PgAioBounceBuffer),
+ mul_size(AioProcs(), io_bounce_buffers));
+
+ return sz;
+}
+
+static Size
+AioBounceBufferDataShmemSize(void)
+{
+ Size sz;
+
+ /* and the associated buffer */
+ sz = mul_size(BLCKSZ,
+ mul_size(io_bounce_buffers, AioProcs()));
+ /* memory for alignment */
+ sz += BLCKSZ;
+
+ return sz;
+}
+
/*
* Choose a suitable value for io_max_concurrency.
*
@@ -113,6 +139,33 @@ AioChooseMaxConcurrency(void)
return Min(max_proportional_pins, 64);
}
+/*
+ * Choose a suitable value for io_bounce_buffers.
+ *
+ * It's very unlikely to be useful to allocate more bounce buffers for each
+ * backend than the backend is allowed to pin. Additionally, bounce buffers
+ * currently are used for writes, it seems very uncommon for more than 10% of
+ * shared_buffers to be written out concurrently.
+ *
+ * XXX: This quickly can take up significant amounts of memory, the logic
+ * should probably fine tuned.
+ */
+static int
+AioChooseBounceBuffers(void)
+{
+ uint32 max_backends;
+ int max_proportional_pins;
+
+ /* Similar logic to LimitAdditionalPins() */
+ max_backends = MaxBackends + NUM_AUXILIARY_PROCS;
+ max_proportional_pins = (NBuffers / 10) / max_backends;
+
+ max_proportional_pins = Max(max_proportional_pins, 1);
+
+ /* apply upper limit */
+ return Min(max_proportional_pins, 256);
+}
+
Size
AioShmemSize(void)
{
@@ -136,11 +189,31 @@ AioShmemSize(void)
PGC_S_OVERRIDE);
}
+
+ /*
+ * If io_bounce_buffers is -1, we automatically choose a suitable value.
+ *
+ * See also comment above.
+ */
+ if (io_bounce_buffers == -1)
+ {
+ char buf[32];
+
+ snprintf(buf, sizeof(buf), "%d", AioChooseBounceBuffers());
+ SetConfigOption("io_bounce_buffers", buf, PGC_POSTMASTER,
+ PGC_S_DYNAMIC_DEFAULT);
+ if (io_bounce_buffers == -1) /* failed to apply it? */
+ SetConfigOption("io_bounce_buffers", buf, PGC_POSTMASTER,
+ PGC_S_OVERRIDE);
+ }
+
sz = add_size(sz, AioCtlShmemSize());
sz = add_size(sz, AioBackendShmemSize());
sz = add_size(sz, AioHandleShmemSize());
sz = add_size(sz, AioHandleIOVShmemSize());
sz = add_size(sz, AioHandleDataShmemSize());
+ sz = add_size(sz, AioBounceBufferDescShmemSize());
+ sz = add_size(sz, AioBounceBufferDataShmemSize());
/* Reserve space for method specific resources. */
if (pgaio_method_ops->shmem_size)
@@ -156,6 +229,9 @@ AioShmemInit(void)
uint32 io_handle_off = 0;
uint32 iovec_off = 0;
uint32 per_backend_iovecs = io_max_concurrency * io_max_combine_limit;
+ uint32 bounce_buffers_off = 0;
+ uint32 per_backend_bb = io_bounce_buffers;
+ char *bounce_buffers_data;
pgaio_ctl = (PgAioCtl *)
ShmemInitStruct("AioCtl", AioCtlShmemSize(), &found);
@@ -167,6 +243,7 @@ AioShmemInit(void)
pgaio_ctl->io_handle_count = AioProcs() * io_max_concurrency;
pgaio_ctl->iovec_count = AioProcs() * per_backend_iovecs;
+ pgaio_ctl->bounce_buffers_count = AioProcs() * per_backend_bb;
pgaio_ctl->backend_state = (PgAioBackend *)
ShmemInitStruct("AioBackend", AioBackendShmemSize(), &found);
@@ -179,6 +256,40 @@ AioShmemInit(void)
pgaio_ctl->handle_data = (uint64 *)
ShmemInitStruct("AioHandleData", AioHandleDataShmemSize(), &found);
+ pgaio_ctl->bounce_buffers = (PgAioBounceBuffer *)
+ ShmemInitStruct("AioBounceBufferDesc", AioBounceBufferDescShmemSize(),
+ &found);
+
+ bounce_buffers_data =
+ ShmemInitStruct("AioBounceBufferData", AioBounceBufferDataShmemSize(),
+ &found);
+ bounce_buffers_data =
+ (char *) TYPEALIGN(BLCKSZ, (uintptr_t) bounce_buffers_data);
+ pgaio_ctl->bounce_buffers_data = bounce_buffers_data;
+
+
+ /* Initialize IO handles. */
+ for (uint64 i = 0; i < pgaio_ctl->io_handle_count; i++)
+ {
+ PgAioHandle *ioh = &pgaio_ctl->io_handles[i];
+
+ ioh->op = PGAIO_OP_INVALID;
+ ioh->target = PGAIO_TID_INVALID;
+ ioh->state = PGAIO_HS_IDLE;
+
+ slist_init(&ioh->bounce_buffers);
+ }
+
+ /* Initialize Bounce Buffers. */
+ for (uint64 i = 0; i < pgaio_ctl->bounce_buffers_count; i++)
+ {
+ PgAioBounceBuffer *bb = &pgaio_ctl->bounce_buffers[i];
+
+ bb->buffer = bounce_buffers_data;
+ bounce_buffers_data += BLCKSZ;
+ }
+
+
for (int procno = 0; procno < AioProcs(); procno++)
{
PgAioBackend *bs = &pgaio_ctl->backend_state[procno];
@@ -186,9 +297,13 @@ AioShmemInit(void)
bs->io_handle_off = io_handle_off;
io_handle_off += io_max_concurrency;
+ bs->bounce_buffers_off = bounce_buffers_off;
+ bounce_buffers_off += per_backend_bb;
+
dclist_init(&bs->idle_ios);
memset(bs->staged_ios, 0, sizeof(PgAioHandle *) * PGAIO_SUBMIT_BATCH_SIZE);
dclist_init(&bs->in_flight_ios);
+ slist_init(&bs->idle_bbs);
/* initialize per-backend IOs */
for (int i = 0; i < io_max_concurrency; i++)
@@ -210,6 +325,14 @@ AioShmemInit(void)
dclist_push_tail(&bs->idle_ios, &ioh->node);
iovec_off += io_max_combine_limit;
}
+
+ /* initialize per-backend bounce buffers */
+ for (int i = 0; i < per_backend_bb; i++)
+ {
+ PgAioBounceBuffer *bb = &pgaio_ctl->bounce_buffers[bs->bounce_buffers_off + i];
+
+ slist_push_head(&bs->idle_bbs, &bb->node);
+ }
}
out:
diff --git a/src/backend/utils/misc/guc_tables.c b/src/backend/utils/misc/guc_tables.c
index 5d729102f46..e885a682258 100644
--- a/src/backend/utils/misc/guc_tables.c
+++ b/src/backend/utils/misc/guc_tables.c
@@ -3303,6 +3303,19 @@ struct config_int ConfigureNamesInt[] =
check_io_max_concurrency, NULL, NULL
},
+ {
+ {"io_bounce_buffers",
+ PGC_POSTMASTER,
+ RESOURCES_IO,
+ gettext_noop("Number of IO Bounce Buffers reserved for each backend."),
+ NULL,
+ GUC_UNIT_BLOCKS
+ },
+ &io_bounce_buffers,
+ -1, -1, 4096,
+ NULL, NULL, NULL
+ },
+
{
{"io_workers",
PGC_SIGHUP,
diff --git a/src/backend/utils/misc/postgresql.conf.sample b/src/backend/utils/misc/postgresql.conf.sample
index 771fe4cbe35..ef9585ac991 100644
--- a/src/backend/utils/misc/postgresql.conf.sample
+++ b/src/backend/utils/misc/postgresql.conf.sample
@@ -211,6 +211,8 @@
# -1 sets based on shared_buffers
# (change requires restart)
#io_workers = 3 # 1-32;
+#io_bounce_buffers = -1 # -1 sets based on shared_buffers
+ # (change requires restart)
# - Worker Processes -
diff --git a/src/backend/utils/resowner/resowner.c b/src/backend/utils/resowner/resowner.c
index d39f3e1b655..81e7e27965a 100644
--- a/src/backend/utils/resowner/resowner.c
+++ b/src/backend/utils/resowner/resowner.c
@@ -159,10 +159,11 @@ struct ResourceOwnerData
LOCALLOCK *locks[MAX_RESOWNER_LOCKS]; /* list of owned locks */
/*
- * AIO handles need be registered in critical sections and therefore
- * cannot use the normal ResourceElem mechanism.
+ * AIO handles & bounce buffers need be registered in critical sections
+ * and therefore cannot use the normal ResourceElem mechanism.
*/
dlist_head aio_handles;
+ dlist_head aio_bounce_buffers;
};
@@ -434,6 +435,7 @@ ResourceOwnerCreate(ResourceOwner parent, const char *name)
}
dlist_init(&owner->aio_handles);
+ dlist_init(&owner->aio_bounce_buffers);
return owner;
}
@@ -742,6 +744,13 @@ ResourceOwnerReleaseInternal(ResourceOwner owner,
pgaio_io_release_resowner(node, !isCommit);
}
+
+ while (!dlist_is_empty(&owner->aio_bounce_buffers))
+ {
+ dlist_node *node = dlist_head_node(&owner->aio_bounce_buffers);
+
+ pgaio_bounce_buffer_release_resowner(node, !isCommit);
+ }
}
else if (phase == RESOURCE_RELEASE_LOCKS)
{
@@ -1111,3 +1120,15 @@ ResourceOwnerForgetAioHandle(ResourceOwner owner, struct dlist_node *ioh_node)
{
dlist_delete_from(&owner->aio_handles, ioh_node);
}
+
+void
+ResourceOwnerRememberAioBounceBuffer(ResourceOwner owner, struct dlist_node *ioh_node)
+{
+ dlist_push_tail(&owner->aio_bounce_buffers, ioh_node);
+}
+
+void
+ResourceOwnerForgetAioBounceBuffer(ResourceOwner owner, struct dlist_node *ioh_node)
+{
+ dlist_delete_from(&owner->aio_bounce_buffers, ioh_node);
+}
diff --git a/src/test/modules/test_aio/test_aio--1.0.sql b/src/test/modules/test_aio/test_aio--1.0.sql
index 602615e39be..3187bd859b7 100644
--- a/src/test/modules/test_aio/test_aio--1.0.sql
+++ b/src/test/modules/test_aio/test_aio--1.0.sql
@@ -80,6 +80,27 @@ RETURNS pg_catalog.void STRICT
AS 'MODULE_PATHNAME' LANGUAGE C;
+CREATE FUNCTION bb_get_and_error()
+RETURNS pg_catalog.void STRICT
+AS 'MODULE_PATHNAME' LANGUAGE C;
+
+CREATE FUNCTION bb_get_twice()
+RETURNS pg_catalog.void STRICT
+AS 'MODULE_PATHNAME' LANGUAGE C;
+
+CREATE FUNCTION bb_get()
+RETURNS pg_catalog.void STRICT
+AS 'MODULE_PATHNAME' LANGUAGE C;
+
+CREATE FUNCTION bb_get_release()
+RETURNS pg_catalog.void STRICT
+AS 'MODULE_PATHNAME' LANGUAGE C;
+
+CREATE FUNCTION bb_release_last()
+RETURNS pg_catalog.void STRICT
+AS 'MODULE_PATHNAME' LANGUAGE C;
+
+
/*
* Injection point related functions
diff --git a/src/test/modules/test_aio/test_aio.c b/src/test/modules/test_aio/test_aio.c
index 0f34892a17e..2e12e60d3fb 100644
--- a/src/test/modules/test_aio/test_aio.c
+++ b/src/test/modules/test_aio/test_aio.c
@@ -53,6 +53,7 @@ static shmem_startup_hook_type prev_shmem_startup_hook = NULL;
static PgAioHandle *last_handle;
+static PgAioBounceBuffer *last_bb;
@@ -581,6 +582,60 @@ batch_end(PG_FUNCTION_ARGS)
PG_RETURN_VOID();
}
+PG_FUNCTION_INFO_V1(bb_get);
+Datum
+bb_get(PG_FUNCTION_ARGS)
+{
+ last_bb = pgaio_bounce_buffer_get();
+
+ PG_RETURN_VOID();
+}
+
+PG_FUNCTION_INFO_V1(bb_release_last);
+Datum
+bb_release_last(PG_FUNCTION_ARGS)
+{
+ if (!last_bb)
+ elog(ERROR, "no bb");
+
+ pgaio_bounce_buffer_release(last_bb);
+
+ PG_RETURN_VOID();
+}
+
+PG_FUNCTION_INFO_V1(bb_get_and_error);
+Datum
+bb_get_and_error(PG_FUNCTION_ARGS)
+{
+ pgaio_bounce_buffer_get();
+
+ elog(ERROR, "as you command");
+ PG_RETURN_VOID();
+}
+
+PG_FUNCTION_INFO_V1(bb_get_twice);
+Datum
+bb_get_twice(PG_FUNCTION_ARGS)
+{
+ pgaio_bounce_buffer_get();
+ pgaio_bounce_buffer_get();
+
+ PG_RETURN_VOID();
+}
+
+
+PG_FUNCTION_INFO_V1(bb_get_release);
+Datum
+bb_get_release(PG_FUNCTION_ARGS)
+{
+ PgAioBounceBuffer *bb;
+
+ bb = pgaio_bounce_buffer_get();
+ pgaio_bounce_buffer_release(bb);
+
+ PG_RETURN_VOID();
+}
+
#ifdef USE_INJECTION_POINTS
extern PGDLLEXPORT void inj_io_short_read(const char *name, const void *private_data);
extern PGDLLEXPORT void inj_io_reopen(const char *name, const void *private_data);
diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list
index e253f4937a2..cabe406294a 100644
--- a/src/tools/pgindent/typedefs.list
+++ b/src/tools/pgindent/typedefs.list
@@ -2133,6 +2133,7 @@ PermutationStep
PermutationStepBlocker
PermutationStepBlockerType
PgAioBackend
+PgAioBounceBuffer
PgAioCtl
PgAioHandle
PgAioHandleCallbackID
--
2.48.1.76.g4e746b1a31.dirty
[text/x-diff] v2.12-0021-bufmgr-Implement-AIO-write-support.patch (9.8K, ../../cxi6ssym62v7ywkt367qda3xik5jhsrcycwedyfqacinxooys5@pwarur37bosc/22-v2.12-0021-bufmgr-Implement-AIO-write-support.patch)
download | inline diff:
From e7b00b7ad5db7ca79010eea3abcd626bffc8e25d Mon Sep 17 00:00:00 2001
From: Andres Freund <andres@anarazel.de>
Date: Tue, 18 Mar 2025 14:40:06 -0400
Subject: [PATCH v2.12 21/28] bufmgr: Implement AIO write support
As of this commit there are no users of these AIO facilities, that'll come in
later commits.
Author:
Reviewed-By:
Discussion: https://postgr.es/m/
Backpatch:
---
src/include/storage/aio.h | 2 +
src/include/storage/bufmgr.h | 2 +
src/backend/storage/aio/aio_callback.c | 2 +
src/backend/storage/buffer/bufmgr.c | 178 +++++++++++++++++++++++++
4 files changed, 184 insertions(+)
diff --git a/src/include/storage/aio.h b/src/include/storage/aio.h
index 09d6d9fe1bc..d461485e83f 100644
--- a/src/include/storage/aio.h
+++ b/src/include/storage/aio.h
@@ -197,8 +197,10 @@ typedef enum PgAioHandleCallbackID
PGAIO_HCB_MD_WRITEV,
PGAIO_HCB_SHARED_BUFFER_READV,
+ PGAIO_HCB_SHARED_BUFFER_WRITEV,
PGAIO_HCB_LOCAL_BUFFER_READV,
+ PGAIO_HCB_LOCAL_BUFFER_WRITEV,
} PgAioHandleCallbackID;
diff --git a/src/include/storage/bufmgr.h b/src/include/storage/bufmgr.h
index 784df8b00cb..ba9bf247ddb 100644
--- a/src/include/storage/bufmgr.h
+++ b/src/include/storage/bufmgr.h
@@ -172,7 +172,9 @@ extern PGDLLIMPORT int backend_flush_after;
extern PGDLLIMPORT int bgwriter_flush_after;
extern const PgAioHandleCallbacks aio_shared_buffer_readv_cb;
+extern const PgAioHandleCallbacks aio_shared_buffer_writev_cb;
extern const PgAioHandleCallbacks aio_local_buffer_readv_cb;
+extern const PgAioHandleCallbacks aio_local_buffer_writev_cb;
/* in buf_init.c */
extern PGDLLIMPORT char *BufferBlocks;
diff --git a/src/backend/storage/aio/aio_callback.c b/src/backend/storage/aio/aio_callback.c
index 12b0f1ebfa6..2a7c6f4f8ad 100644
--- a/src/backend/storage/aio/aio_callback.c
+++ b/src/backend/storage/aio/aio_callback.c
@@ -44,8 +44,10 @@ static const PgAioHandleCallbacksEntry aio_handle_cbs[] = {
CALLBACK_ENTRY(PGAIO_HCB_MD_WRITEV, aio_md_writev_cb),
CALLBACK_ENTRY(PGAIO_HCB_SHARED_BUFFER_READV, aio_shared_buffer_readv_cb),
+ CALLBACK_ENTRY(PGAIO_HCB_SHARED_BUFFER_WRITEV, aio_shared_buffer_writev_cb),
CALLBACK_ENTRY(PGAIO_HCB_LOCAL_BUFFER_READV, aio_local_buffer_readv_cb),
+ CALLBACK_ENTRY(PGAIO_HCB_LOCAL_BUFFER_WRITEV, aio_local_buffer_writev_cb),
#undef CALLBACK_ENTRY
};
diff --git a/src/backend/storage/buffer/bufmgr.c b/src/backend/storage/buffer/bufmgr.c
index 7f9b58003b3..d037aa76489 100644
--- a/src/backend/storage/buffer/bufmgr.c
+++ b/src/backend/storage/buffer/bufmgr.c
@@ -5508,7 +5508,15 @@ LockBuffer(Buffer buffer, int mode)
else if (mode == BUFFER_LOCK_SHARE)
LWLockAcquire(BufferDescriptorGetContentLock(buf), LW_SHARED);
else if (mode == BUFFER_LOCK_EXCLUSIVE)
+ {
+ /*
+ * FIXME: Wait for AIO writes, otherwise there would be a risk of
+ * deadlock. This isn't entirely trivial to do in a race-free way, IO
+ * could be started between us checking whether there is IO and
+ * enqueueing ourselves for the lock.
+ */
LWLockAcquire(BufferDescriptorGetContentLock(buf), LW_EXCLUSIVE);
+ }
else
elog(ERROR, "unrecognized buffer lock mode: %d", mode);
}
@@ -5523,6 +5531,19 @@ ConditionalLockBuffer(Buffer buffer)
{
BufferDesc *buf;
+ /*
+ * FIXME: Wait for AIO writes. Some code does not deal well
+ * ConditionalLockBuffer() continuously failing, e.g.
+ * spginsert()->spgdoinsert() ends up busy-looping (easily reproducible by
+ * just making this function always fail and running the regression
+ * tests). While that code could be fixed, it'd be hard to find all
+ * problematic places.
+ *
+ * It would be OK to wait for the IO as waiting for IO completion does not
+ * need to wait for any locks that could lead to an undetected deadlock or
+ * such.
+ */
+
Assert(BufferIsPinned(buffer));
if (BufferIsLocal(buffer))
return true; /* act as though we got it */
@@ -5595,6 +5616,11 @@ LockBufferForCleanup(Buffer buffer)
{
uint32 buf_state;
+ /*
+ * FIXME: LockBuffer()'s handling of in-progress writes (once
+ * implemented) should suffice to deal with deadlock risk.
+ */
+
/* Try to acquire lock */
LockBuffer(buffer, BUFFER_LOCK_EXCLUSIVE);
buf_state = LockBufHdr(bufHdr);
@@ -5742,6 +5768,14 @@ ConditionalLockBufferForCleanup(Buffer buffer)
Assert(BufferIsValid(buffer));
+ /*
+ * FIXME: Should wait for IO for the same reason as in
+ * ConditionalLockBuffer(). Needs to happen before the
+ * ConditionalLockBuffer() call below, as we'd never reach the
+ * ConditionalLockBuffer() call due the buffer pin held for the duration
+ * of the IO.
+ */
+
if (BufferIsLocal(buffer))
{
refcount = LocalRefCount[-buffer - 1];
@@ -6842,12 +6876,129 @@ buffer_readv_report(PgAioResult result, const PgAioTargetData *target_data,
target_data->smgr.forkNum).str));
}
+/*
+ * Helper for AIO writev completion callbacks, supporting both shared and temp
+ * buffers. Gets called once for each buffer in a multi-page write.
+ */
+static pg_attribute_always_inline PgAioResult
+buffer_writev_complete_one(uint8 buf_off, Buffer buffer, uint8 flags,
+ bool failed, bool is_temp)
+{
+ BufferDesc *buf_hdr = is_temp ?
+ GetLocalBufferDescriptor(-buffer - 1)
+ : GetBufferDescriptor(buffer - 1);
+ PgAioResult result = {.status = PGAIO_RS_OK};
+ bool clear_dirty;
+ uint32 set_flag_bits;
+
+#ifdef USE_ASSERT_CHECKING
+ {
+ uint32 buf_state = pg_atomic_read_u32(&buf_hdr->state);
+
+ Assert(buf_state & BM_VALID);
+ Assert(buf_state & BM_TAG_VALID);
+ /* temp buffers don't use BM_IO_IN_PROGRESS */
+ if (!is_temp)
+ Assert(buf_state & BM_IO_IN_PROGRESS);
+ Assert(buf_state & BM_DIRTY);
+ }
+#endif
+
+ clear_dirty = failed ? false : true;
+ set_flag_bits = failed ? BM_IO_ERROR : 0;
+
+ if (is_temp)
+ TerminateLocalBufferIO(buf_hdr, clear_dirty, set_flag_bits, false);
+ else
+ TerminateBufferIO(buf_hdr, clear_dirty, set_flag_bits, false, false);
+
+ /*
+ * The initiator of IO is not managing the lock (i.e. we called
+ * LWLockDisown()), we are.
+ */
+ if (!is_temp)
+ LWLockReleaseDisowned(BufferDescriptorGetContentLock(buf_hdr),
+ LW_SHARED);
+
+ /* FIXME: tracepoint */
+
+ return result;
+}
+
+/*
+ * Perform completion handling of a single AIO write. This write may cover
+ * multiple blocks / buffers.
+ *
+ * Shared between shared and local buffers, to reduce code duplication.
+ */
+static pg_attribute_always_inline PgAioResult
+buffer_writev_complete(PgAioHandle *ioh, PgAioResult prior_result,
+ uint8 cb_data, bool is_temp)
+{
+ PgAioResult result = prior_result;
+ PgAioTargetData *td = pgaio_io_get_target_data(ioh);
+ uint64 *io_data;
+ uint8 handle_data_len;
+
+ if (is_temp)
+ {
+ Assert(td->smgr.is_temp);
+ Assert(pgaio_io_get_owner(ioh) == MyProcNumber);
+ }
+ else
+ Assert(!td->smgr.is_temp);
+
+ /*
+ * Iterate over all the buffers affected by this IO and call appropriate
+ * per-buffer completion function for each buffer.
+ */
+ io_data = pgaio_io_get_handle_data(ioh, &handle_data_len);
+ for (uint8 buf_off = 0; buf_off < handle_data_len; buf_off++)
+ {
+ Buffer buf = io_data[buf_off];
+ PgAioResult buf_result;
+ bool failed;
+
+ Assert(BufferIsValid(buf));
+
+ /*
+ * If the entire failed on a lower-level, each buffer needs to be
+ * marked as failed. In case of a partial read, some buffers may be
+ * ok.
+ */
+ failed =
+ prior_result.status == PGAIO_RS_ERROR
+ || prior_result.result <= buf_off;
+
+ buf_result = buffer_writev_complete_one(buf_off, buf, cb_data, failed,
+ is_temp);
+
+ /*
+ * If there wasn't any prior error and the IO for this page failed in
+ * some form, set the whole IO's to the page's result.
+ */
+ if (result.status != PGAIO_RS_ERROR && buf_result.status != PGAIO_RS_OK)
+ {
+ result = buf_result;
+ pgaio_result_report(result, td, LOG);
+ }
+ }
+
+ return result;
+}
+
static void
shared_buffer_readv_stage(PgAioHandle *ioh, uint8 cb_data)
{
buffer_stage_common(ioh, false, false);
}
+static void
+shared_buffer_writev_stage(PgAioHandle *ioh, uint8 cb_data)
+{
+ buffer_stage_common(ioh, true, false);
+}
+
static PgAioResult
shared_buffer_readv_complete(PgAioHandle *ioh, PgAioResult prior_result,
uint8 cb_data)
@@ -6855,6 +7006,13 @@ shared_buffer_readv_complete(PgAioHandle *ioh, PgAioResult prior_result,
return buffer_readv_complete(ioh, prior_result, cb_data, false);
}
+static PgAioResult
+shared_buffer_writev_complete(PgAioHandle *ioh, PgAioResult prior_result,
+ uint8 cb_data)
+{
+ return buffer_writev_complete(ioh, prior_result, cb_data, false);
+}
+
static void
local_buffer_readv_stage(PgAioHandle *ioh, uint8 cb_data)
{
@@ -6868,6 +7026,17 @@ local_buffer_readv_complete(PgAioHandle *ioh, PgAioResult prior_result,
return buffer_readv_complete(ioh, prior_result, cb_data, true);
}
+static void
+local_buffer_writev_stage(PgAioHandle *ioh, uint8 cb_data)
+{
+ /*
+ * Currently this is unreachable as the only write support is for
+ * checkpointer / bgwriter, which don't deal with local buffers.
+ */
+ elog(ERROR, "should be unreachable");
+}
+
+
/* readv callback is passed READ_BUFFERS_* flags as callback data */
const PgAioHandleCallbacks aio_shared_buffer_readv_cb = {
.stage = shared_buffer_readv_stage,
@@ -6875,6 +7044,11 @@ const PgAioHandleCallbacks aio_shared_buffer_readv_cb = {
.report = buffer_readv_report,
};
+const PgAioHandleCallbacks aio_shared_buffer_writev_cb = {
+ .stage = shared_buffer_writev_stage,
+ .complete_shared = shared_buffer_writev_complete,
+};
+
/* readv callback is passed READ_BUFFERS_* flags as callback data */
const PgAioHandleCallbacks aio_local_buffer_readv_cb = {
.stage = local_buffer_readv_stage,
@@ -6888,3 +7062,7 @@ const PgAioHandleCallbacks aio_local_buffer_readv_cb = {
.complete_local = local_buffer_readv_complete,
.report = buffer_readv_report,
};
+
+const PgAioHandleCallbacks aio_local_buffer_writev_cb = {
+ .stage = local_buffer_writev_stage,
+};
--
2.48.1.76.g4e746b1a31.dirty
[text/x-diff] v2.12-0022-aio-Add-IO-queue-helper.patch (7.4K, ../../cxi6ssym62v7ywkt367qda3xik5jhsrcycwedyfqacinxooys5@pwarur37bosc/23-v2.12-0022-aio-Add-IO-queue-helper.patch)
download | inline diff:
From 68ac3a336c08bc50747c7528fa635da260c31d55 Mon Sep 17 00:00:00 2001
From: Andres Freund <andres@anarazel.de>
Date: Tue, 18 Mar 2025 14:40:06 -0400
Subject: [PATCH v2.12 22/28] aio: Add IO queue helper
This is likely never going to anywhere - Thomas Munro is working on something
more complete. But I needed a way to exercise aio for checkpointer / bgwriter.
---
src/include/storage/io_queue.h | 33 +++++
src/backend/storage/aio/Makefile | 1 +
src/backend/storage/aio/io_queue.c | 204 ++++++++++++++++++++++++++++
src/backend/storage/aio/meson.build | 1 +
src/tools/pgindent/typedefs.list | 2 +
5 files changed, 241 insertions(+)
create mode 100644 src/include/storage/io_queue.h
create mode 100644 src/backend/storage/aio/io_queue.c
diff --git a/src/include/storage/io_queue.h b/src/include/storage/io_queue.h
new file mode 100644
index 00000000000..92b1e9afe6f
--- /dev/null
+++ b/src/include/storage/io_queue.h
@@ -0,0 +1,33 @@
+/*-------------------------------------------------------------------------
+ *
+ * io_queue.h
+ * Mechanism for tracking many IOs
+ *
+ *
+ * Portions Copyright (c) 1996-2025, PostgreSQL Global Development Group
+ * Portions Copyright (c) 1994, Regents of the University of California
+ *
+ * src/include/storage/io_queue.h
+ *
+ *-------------------------------------------------------------------------
+ */
+#ifndef IO_QUEUE_H
+#define IO_QUEUE_H
+
+#include "storage/aio_types.h"
+
+struct IOQueue;
+typedef struct IOQueue IOQueue;
+
+struct PgAioWaitRef;
+
+extern IOQueue *io_queue_create(int depth, int flags);
+extern void io_queue_track(IOQueue *ioq, const PgAioWaitRef *iow);
+extern void io_queue_wait_one(IOQueue *ioq);
+extern void io_queue_wait_all(IOQueue *ioq);
+extern bool io_queue_is_empty(IOQueue *ioq);
+extern void io_queue_reserve(IOQueue *ioq);
+extern PgAioHandle *io_queue_acquire_io(IOQueue *ioq);
+extern void io_queue_free(IOQueue *ioq);
+
+#endif /* IO_QUEUE_H */
diff --git a/src/backend/storage/aio/Makefile b/src/backend/storage/aio/Makefile
index 3f2469cc399..86fa4276fda 100644
--- a/src/backend/storage/aio/Makefile
+++ b/src/backend/storage/aio/Makefile
@@ -15,6 +15,7 @@ OBJS = \
aio_init.o \
aio_io.o \
aio_target.o \
+ io_queue.o \
method_io_uring.o \
method_sync.o \
method_worker.o \
diff --git a/src/backend/storage/aio/io_queue.c b/src/backend/storage/aio/io_queue.c
new file mode 100644
index 00000000000..526aa1d5e06
--- /dev/null
+++ b/src/backend/storage/aio/io_queue.c
@@ -0,0 +1,204 @@
+/*-------------------------------------------------------------------------
+ *
+ * io_queue.c
+ * AIO - Mechanism for tracking many IOs
+ *
+ * Portions Copyright (c) 1996-2025, PostgreSQL Global Development Group
+ * Portions Copyright (c) 1994, Regents of the University of California
+ *
+ * IDENTIFICATION
+ * src/backend/storage/aio/io_queue.c
+ *
+ *-------------------------------------------------------------------------
+ */
+
+#include "postgres.h"
+
+#include "lib/ilist.h"
+#include "storage/aio.h"
+#include "storage/io_queue.h"
+#include "utils/resowner.h"
+
+
+
+typedef struct TrackedIO
+{
+ PgAioWaitRef iow;
+ dlist_node node;
+} TrackedIO;
+
+struct IOQueue
+{
+ int depth;
+ int unsubmitted;
+
+ bool has_reserved;
+
+ dclist_head idle;
+ dclist_head in_progress;
+
+ TrackedIO tracked_ios[FLEXIBLE_ARRAY_MEMBER];
+};
+
+
+IOQueue *
+io_queue_create(int depth, int flags)
+{
+ size_t sz;
+ IOQueue *ioq;
+
+ sz = offsetof(IOQueue, tracked_ios)
+ + sizeof(TrackedIO) * depth;
+
+ ioq = palloc0(sz);
+
+ ioq->depth = 0;
+
+ for (int i = 0; i < depth; i++)
+ {
+ TrackedIO *tio = &ioq->tracked_ios[i];
+
+ pgaio_wref_clear(&tio->iow);
+ dclist_push_tail(&ioq->idle, &tio->node);
+ }
+
+ return ioq;
+}
+
+void
+io_queue_wait_one(IOQueue *ioq)
+{
+ /* submit all pending IO before waiting */
+ pgaio_submit_staged();
+
+ while (!dclist_is_empty(&ioq->in_progress))
+ {
+ /* FIXME: Should we really pop here already? */
+ dlist_node *node = dclist_pop_head_node(&ioq->in_progress);
+ TrackedIO *tio = dclist_container(TrackedIO, node, node);
+
+ pgaio_wref_wait(&tio->iow);
+ dclist_push_head(&ioq->idle, &tio->node);
+ }
+}
+
+void
+io_queue_reserve(IOQueue *ioq)
+{
+ if (ioq->has_reserved)
+ return;
+
+ if (dclist_is_empty(&ioq->idle))
+ io_queue_wait_one(ioq);
+
+ Assert(!dclist_is_empty(&ioq->idle));
+
+ ioq->has_reserved = true;
+}
+
+PgAioHandle *
+io_queue_acquire_io(IOQueue *ioq)
+{
+ PgAioHandle *ioh;
+
+ io_queue_reserve(ioq);
+
+ Assert(!dclist_is_empty(&ioq->idle));
+
+ if (!io_queue_is_empty(ioq))
+ {
+ ioh = pgaio_io_acquire_nb(CurrentResourceOwner, NULL);
+ if (ioh == NULL)
+ {
+ /*
+ * Need to wait for all IOs, blocking might not be legal in the
+ * context.
+ *
+ * XXX: This doesn't make a whole lot of sense, we're also
+ * blocking here. What was I smoking when I wrote the above?
+ */
+ io_queue_wait_all(ioq);
+ ioh = pgaio_io_acquire(CurrentResourceOwner, NULL);
+ }
+ }
+ else
+ {
+ ioh = pgaio_io_acquire(CurrentResourceOwner, NULL);
+ }
+
+ return ioh;
+}
+
+void
+io_queue_track(IOQueue *ioq, const struct PgAioWaitRef *iow)
+{
+ dlist_node *node;
+ TrackedIO *tio;
+
+ Assert(ioq->has_reserved);
+ ioq->has_reserved = false;
+
+ Assert(!dclist_is_empty(&ioq->idle));
+
+ node = dclist_pop_head_node(&ioq->idle);
+ tio = dclist_container(TrackedIO, node, node);
+
+ tio->iow = *iow;
+
+ dclist_push_tail(&ioq->in_progress, &tio->node);
+
+ ioq->unsubmitted++;
+
+ /*
+ * XXX: Should have some smarter logic here. We don't want to wait too
+ * long to submit, that'll mean we're more likely to block. But we also
+ * don't want to have the overhead of submitting every IO individually.
+ */
+ if (ioq->unsubmitted >= 4)
+ {
+ pgaio_submit_staged();
+ ioq->unsubmitted = 0;
+ }
+}
+
+void
+io_queue_wait_all(IOQueue *ioq)
+{
+ /* submit all pending IO before waiting */
+ pgaio_submit_staged();
+
+ while (!dclist_is_empty(&ioq->in_progress))
+ {
+ /* wait for the last IO to minimize unnecessary wakeups */
+ dlist_node *node = dclist_tail_node(&ioq->in_progress);
+ TrackedIO *tio = dclist_container(TrackedIO, node, node);
+
+ if (!pgaio_wref_check_done(&tio->iow))
+ {
+ ereport(DEBUG3,
+ errmsg("io_queue_wait_all for io:%d",
+ pgaio_wref_get_id(&tio->iow)),
+ errhidestmt(true),
+ errhidecontext(true));
+
+ pgaio_wref_wait(&tio->iow);
+ }
+
+ dclist_delete_from(&ioq->in_progress, &tio->node);
+ dclist_push_head(&ioq->idle, &tio->node);
+ }
+}
+
+bool
+io_queue_is_empty(IOQueue *ioq)
+{
+ return dclist_is_empty(&ioq->in_progress);
+}
+
+void
+io_queue_free(IOQueue *ioq)
+{
+ io_queue_wait_all(ioq);
+
+ pfree(ioq);
+}
diff --git a/src/backend/storage/aio/meson.build b/src/backend/storage/aio/meson.build
index da6df2d3654..270c4a64428 100644
--- a/src/backend/storage/aio/meson.build
+++ b/src/backend/storage/aio/meson.build
@@ -7,6 +7,7 @@ backend_sources += files(
'aio_init.c',
'aio_io.c',
'aio_target.c',
+ 'io_queue.c',
'method_io_uring.c',
'method_sync.c',
'method_worker.c',
diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list
index cabe406294a..0e40687bb43 100644
--- a/src/tools/pgindent/typedefs.list
+++ b/src/tools/pgindent/typedefs.list
@@ -1192,6 +1192,7 @@ IOContext
IOFuncSelector
IOObject
IOOp
+IOQueue
IO_STATUS_BLOCK
IPCompareMethod
ITEM
@@ -3017,6 +3018,7 @@ TocEntry
TokenAuxData
TokenizedAuthLine
TrackItem
+TrackedIO
TransApplyAction
TransInvalidationInfo
TransState
--
2.48.1.76.g4e746b1a31.dirty
[text/x-diff] v2.12-0023-bufmgr-use-AIO-in-checkpointer-bgwriter.patch (31.3K, ../../cxi6ssym62v7ywkt367qda3xik5jhsrcycwedyfqacinxooys5@pwarur37bosc/24-v2.12-0023-bufmgr-use-AIO-in-checkpointer-bgwriter.patch)
download | inline diff:
From 9b79a6489e9a7721fb947306a5e248aac52b3656 Mon Sep 17 00:00:00 2001
From: Andres Freund <andres@anarazel.de>
Date: Tue, 18 Mar 2025 14:40:06 -0400
Subject: [PATCH v2.12 23/28] bufmgr: use AIO in checkpointer, bgwriter
This is far from ready - just included to be able to exercise AIO writes and
get some preliminary numbers. In all likelihood this will instead be based
on-top of work by Thomas Munro instead of the preceding commit.
---
src/include/postmaster/bgwriter.h | 3 +-
src/include/storage/buf_internals.h | 2 +
src/include/storage/bufmgr.h | 3 +-
src/include/storage/bufpage.h | 1 +
src/backend/postmaster/bgwriter.c | 19 +-
src/backend/postmaster/checkpointer.c | 11 +-
src/backend/storage/buffer/bufmgr.c | 594 +++++++++++++++++++++++---
src/backend/storage/page/bufpage.c | 10 +
src/tools/pgindent/typedefs.list | 1 +
9 files changed, 586 insertions(+), 58 deletions(-)
diff --git a/src/include/postmaster/bgwriter.h b/src/include/postmaster/bgwriter.h
index 800ecbfd13b..a8081d411b6 100644
--- a/src/include/postmaster/bgwriter.h
+++ b/src/include/postmaster/bgwriter.h
@@ -31,7 +31,8 @@ pg_noreturn extern void BackgroundWriterMain(const void *startup_data, size_t st
pg_noreturn extern void CheckpointerMain(const void *startup_data, size_t startup_data_len);
extern void RequestCheckpoint(int flags);
-extern void CheckpointWriteDelay(int flags, double progress);
+struct IOQueue;
+extern void CheckpointWriteDelay(struct IOQueue *ioq, int flags, double progress);
extern bool ForwardSyncRequest(const FileTag *ftag, SyncRequestType type);
diff --git a/src/include/storage/buf_internals.h b/src/include/storage/buf_internals.h
index 6821a710e46..234e093cd04 100644
--- a/src/include/storage/buf_internals.h
+++ b/src/include/storage/buf_internals.h
@@ -21,6 +21,8 @@
#include "storage/buf.h"
#include "storage/bufmgr.h"
#include "storage/condition_variable.h"
+#include "storage/io_queue.h"
+#include "storage/latch.h"
#include "storage/lwlock.h"
#include "storage/procnumber.h"
#include "storage/shmem.h"
diff --git a/src/include/storage/bufmgr.h b/src/include/storage/bufmgr.h
index ba9bf247ddb..af5035317b7 100644
--- a/src/include/storage/bufmgr.h
+++ b/src/include/storage/bufmgr.h
@@ -295,7 +295,8 @@ extern bool ConditionalLockBufferForCleanup(Buffer buffer);
extern bool IsBufferCleanupOK(Buffer buffer);
extern bool HoldingBufferPinThatDelaysRecovery(void);
-extern bool BgBufferSync(struct WritebackContext *wb_context);
+struct IOQueue;
+extern bool BgBufferSync(struct IOQueue *ioq, struct WritebackContext *wb_context);
extern uint32 GetPinLimit(void);
extern uint32 GetLocalPinLimit(void);
diff --git a/src/include/storage/bufpage.h b/src/include/storage/bufpage.h
index 6646b6f6371..9c045e81857 100644
--- a/src/include/storage/bufpage.h
+++ b/src/include/storage/bufpage.h
@@ -509,5 +509,6 @@ extern bool PageIndexTupleOverwrite(Page page, OffsetNumber offnum,
Item newtup, Size newsize);
extern char *PageSetChecksumCopy(Page page, BlockNumber blkno);
extern void PageSetChecksumInplace(Page page, BlockNumber blkno);
+extern bool PageNeedsChecksumCopy(Page page);
#endif /* BUFPAGE_H */
diff --git a/src/backend/postmaster/bgwriter.c b/src/backend/postmaster/bgwriter.c
index 72f5acceec7..6e8801a39e3 100644
--- a/src/backend/postmaster/bgwriter.c
+++ b/src/backend/postmaster/bgwriter.c
@@ -38,11 +38,13 @@
#include "postmaster/auxprocess.h"
#include "postmaster/bgwriter.h"
#include "postmaster/interrupt.h"
+#include "storage/aio.h"
#include "storage/aio_subsys.h"
#include "storage/buf_internals.h"
#include "storage/bufmgr.h"
#include "storage/condition_variable.h"
#include "storage/fd.h"
+#include "storage/io_queue.h"
#include "storage/lwlock.h"
#include "storage/proc.h"
#include "storage/procsignal.h"
@@ -90,6 +92,7 @@ BackgroundWriterMain(const void *startup_data, size_t startup_data_len)
sigjmp_buf local_sigjmp_buf;
MemoryContext bgwriter_context;
bool prev_hibernate;
+ IOQueue *ioq;
WritebackContext wb_context;
Assert(startup_data_len == 0);
@@ -131,6 +134,7 @@ BackgroundWriterMain(const void *startup_data, size_t startup_data_len)
ALLOCSET_DEFAULT_SIZES);
MemoryContextSwitchTo(bgwriter_context);
+ ioq = io_queue_create(128, 0);
WritebackContextInit(&wb_context, &bgwriter_flush_after);
/*
@@ -228,12 +232,22 @@ BackgroundWriterMain(const void *startup_data, size_t startup_data_len)
/* Clear any already-pending wakeups */
ResetLatch(MyLatch);
+ /*
+ * FIXME: this is theoretically racy, but I didn't want to copy
+ * ProcessMainLoopInterrupts() remaining body here.
+ */
+ if (ShutdownRequestPending)
+ {
+ io_queue_wait_all(ioq);
+ io_queue_free(ioq);
+ }
+
ProcessMainLoopInterrupts();
/*
* Do one cycle of dirty-buffer writing.
*/
- can_hibernate = BgBufferSync(&wb_context);
+ can_hibernate = BgBufferSync(ioq, &wb_context);
/* Report pending statistics to the cumulative stats system */
pgstat_report_bgwriter();
@@ -250,6 +264,9 @@ BackgroundWriterMain(const void *startup_data, size_t startup_data_len)
smgrdestroyall();
}
+ /* finish IO before sleeping, to avoid blocking other backends */
+ io_queue_wait_all(ioq);
+
/*
* Log a new xl_running_xacts every now and then so replication can
* get into a consistent state faster (think of suboverflowed
diff --git a/src/backend/postmaster/checkpointer.c b/src/backend/postmaster/checkpointer.c
index fda91ffd1ce..904fe167eb4 100644
--- a/src/backend/postmaster/checkpointer.c
+++ b/src/backend/postmaster/checkpointer.c
@@ -49,10 +49,12 @@
#include "postmaster/bgwriter.h"
#include "postmaster/interrupt.h"
#include "replication/syncrep.h"
+#include "storage/aio.h"
#include "storage/aio_subsys.h"
#include "storage/bufmgr.h"
#include "storage/condition_variable.h"
#include "storage/fd.h"
+#include "storage/io_queue.h"
#include "storage/ipc.h"
#include "storage/lwlock.h"
#include "storage/pmsignal.h"
@@ -766,7 +768,7 @@ ImmediateCheckpointRequested(void)
* fraction between 0.0 meaning none, and 1.0 meaning all done.
*/
void
-CheckpointWriteDelay(int flags, double progress)
+CheckpointWriteDelay(IOQueue *ioq, int flags, double progress)
{
static int absorb_counter = WRITES_PER_ABSORB;
@@ -800,6 +802,13 @@ CheckpointWriteDelay(int flags, double progress)
/* Report interim statistics to the cumulative stats system */
pgstat_report_checkpointer();
+ /*
+ * Ensure all pending IO is submitted to avoid unnecessary delays for
+ * other processes.
+ */
+ io_queue_wait_all(ioq);
+
+
/*
* This sleep used to be connected to bgwriter_delay, typically 200ms.
* That resulted in more frequent wakeups if not much work to do.
diff --git a/src/backend/storage/buffer/bufmgr.c b/src/backend/storage/buffer/bufmgr.c
index d037aa76489..8702614b5f3 100644
--- a/src/backend/storage/buffer/bufmgr.c
+++ b/src/backend/storage/buffer/bufmgr.c
@@ -52,6 +52,7 @@
#include "storage/buf_internals.h"
#include "storage/bufmgr.h"
#include "storage/fd.h"
+#include "storage/io_queue.h"
#include "storage/ipc.h"
#include "storage/lmgr.h"
#include "storage/proc.h"
@@ -76,6 +77,7 @@
/* Bits in SyncOneBuffer's return value */
#define BUF_WRITTEN 0x01
#define BUF_REUSABLE 0x02
+#define BUF_CANT_MERGE 0x04
#define RELS_BSEARCH_THRESHOLD 20
@@ -515,8 +517,6 @@ static void UnpinBuffer(BufferDesc *buf);
static void UnpinBufferNoOwner(BufferDesc *buf);
static void BufferSync(int flags);
static uint32 WaitBufHdrUnlocked(BufferDesc *buf);
-static int SyncOneBuffer(int buf_id, bool skip_recently_used,
- WritebackContext *wb_context);
static void WaitIO(BufferDesc *buf);
static void AbortBufferIO(Buffer buffer);
static void shared_buffer_write_error_callback(void *arg);
@@ -532,6 +532,7 @@ static void CheckReadBuffersOperation(ReadBuffersOperation *operation, bool is_c
static Buffer GetVictimBuffer(BufferAccessStrategy strategy, IOContext io_context);
static void FlushBuffer(BufferDesc *buf, SMgrRelation reln,
IOObject io_object, IOContext io_context);
+
static void FindAndDropRelationBuffers(RelFileLocator rlocator,
ForkNumber forkNum,
BlockNumber nForkBlock,
@@ -3299,6 +3300,57 @@ UnpinBufferNoOwner(BufferDesc *buf)
}
}
+typedef struct BuffersToWrite
+{
+ int nbuffers;
+ BufferTag start_at_tag;
+ uint32 max_combine;
+
+ XLogRecPtr max_lsn;
+
+ PgAioHandle *ioh;
+ PgAioWaitRef iow;
+
+ uint64 total_writes;
+
+ Buffer buffers[IOV_MAX];
+ PgAioBounceBuffer *bounce_buffers[IOV_MAX];
+ const void *data_ptrs[IOV_MAX];
+} BuffersToWrite;
+
+static int PrepareToWriteBuffer(BuffersToWrite *to_write, Buffer buf,
+ bool skip_recently_used,
+ IOQueue *ioq, WritebackContext *wb_context);
+
+static void WriteBuffers(BuffersToWrite *to_write,
+ IOQueue *ioq, WritebackContext *wb_context);
+
+static void
+BuffersToWriteInit(BuffersToWrite *to_write,
+ IOQueue *ioq, WritebackContext *wb_context)
+{
+ to_write->total_writes = 0;
+ to_write->nbuffers = 0;
+ to_write->ioh = NULL;
+ pgaio_wref_clear(&to_write->iow);
+ to_write->max_lsn = InvalidXLogRecPtr;
+
+ pgaio_enter_batchmode();
+}
+
+static void
+BuffersToWriteEnd(BuffersToWrite *to_write)
+{
+ if (to_write->ioh != NULL)
+ {
+ pgaio_io_release(to_write->ioh);
+ to_write->ioh = NULL;
+ }
+
+ pgaio_exit_batchmode();
+}
+
+
#define ST_SORT sort_checkpoint_bufferids
#define ST_ELEMENT_TYPE CkptSortItem
#define ST_COMPARE(a, b) ckpt_buforder_comparator(a, b)
@@ -3330,7 +3382,10 @@ BufferSync(int flags)
binaryheap *ts_heap;
int i;
int mask = BM_DIRTY;
+ IOQueue *ioq;
WritebackContext wb_context;
+ BuffersToWrite to_write;
+ int max_combine;
/*
* Unless this is a shutdown checkpoint or we have been explicitly told,
@@ -3392,7 +3447,9 @@ BufferSync(int flags)
if (num_to_scan == 0)
return; /* nothing to do */
+ ioq = io_queue_create(512, 0);
WritebackContextInit(&wb_context, &checkpoint_flush_after);
+ max_combine = Min(io_bounce_buffers, io_combine_limit);
TRACE_POSTGRESQL_BUFFER_SYNC_START(NBuffers, num_to_scan);
@@ -3500,48 +3557,91 @@ BufferSync(int flags)
*/
num_processed = 0;
num_written = 0;
+
+ BuffersToWriteInit(&to_write, ioq, &wb_context);
+
while (!binaryheap_empty(ts_heap))
{
BufferDesc *bufHdr = NULL;
CkptTsStatus *ts_stat = (CkptTsStatus *)
DatumGetPointer(binaryheap_first(ts_heap));
+ bool batch_continue = true;
- buf_id = CkptBufferIds[ts_stat->index].buf_id;
- Assert(buf_id != -1);
-
- bufHdr = GetBufferDescriptor(buf_id);
-
- num_processed++;
+ Assert(ts_stat->num_scanned <= ts_stat->num_to_scan);
/*
- * We don't need to acquire the lock here, because we're only looking
- * at a single bit. It's possible that someone else writes the buffer
- * and clears the flag right after we check, but that doesn't matter
- * since SyncOneBuffer will then do nothing. However, there is a
- * further race condition: it's conceivable that between the time we
- * examine the bit here and the time SyncOneBuffer acquires the lock,
- * someone else not only wrote the buffer but replaced it with another
- * page and dirtied it. In that improbable case, SyncOneBuffer will
- * write the buffer though we didn't need to. It doesn't seem worth
- * guarding against this, though.
+ * Collect a batch of buffers to write out from the current
+ * tablespace. That causes some imbalance between the tablespaces, but
+ * that's more than outweighed by the efficiency gain due to batching.
*/
- if (pg_atomic_read_u32(&bufHdr->state) & BM_CHECKPOINT_NEEDED)
+ while (batch_continue &&
+ to_write.nbuffers < max_combine &&
+ ts_stat->num_scanned < ts_stat->num_to_scan)
{
- if (SyncOneBuffer(buf_id, false, &wb_context) & BUF_WRITTEN)
+ buf_id = CkptBufferIds[ts_stat->index].buf_id;
+ Assert(buf_id != -1);
+
+ bufHdr = GetBufferDescriptor(buf_id);
+
+ num_processed++;
+
+ /*
+ * We don't need to acquire the lock here, because we're only
+ * looking at a single bit. It's possible that someone else writes
+ * the buffer and clears the flag right after we check, but that
+ * doesn't matter since SyncOneBuffer will then do nothing.
+ * However, there is a further race condition: it's conceivable
+ * that between the time we examine the bit here and the time
+ * SyncOneBuffer acquires the lock, someone else not only wrote
+ * the buffer but replaced it with another page and dirtied it. In
+ * that improbable case, SyncOneBuffer will write the buffer
+ * though we didn't need to. It doesn't seem worth guarding
+ * against this, though.
+ */
+ if (pg_atomic_read_u32(&bufHdr->state) & BM_CHECKPOINT_NEEDED)
{
- TRACE_POSTGRESQL_BUFFER_SYNC_WRITTEN(buf_id);
- PendingCheckpointerStats.buffers_written++;
- num_written++;
+ int result = PrepareToWriteBuffer(&to_write, buf_id + 1, false,
+ ioq, &wb_context);
+
+ if (result & BUF_CANT_MERGE)
+ {
+ Assert(to_write.nbuffers > 0);
+ WriteBuffers(&to_write, ioq, &wb_context);
+
+ result = PrepareToWriteBuffer(&to_write, buf_id + 1, false,
+ ioq, &wb_context);
+ Assert(result != BUF_CANT_MERGE);
+ }
+
+ if (result & BUF_WRITTEN)
+ {
+ TRACE_POSTGRESQL_BUFFER_SYNC_WRITTEN(buf_id);
+ PendingCheckpointerStats.buffers_written++;
+ num_written++;
+ }
+ else
+ {
+ batch_continue = false;
+ }
}
+ else
+ {
+ if (to_write.nbuffers > 0)
+ WriteBuffers(&to_write, ioq, &wb_context);
+ }
+
+ /*
+ * Measure progress independent of actually having to flush the
+ * buffer - otherwise writing become unbalanced.
+ */
+ ts_stat->progress += ts_stat->progress_slice;
+ ts_stat->num_scanned++;
+ ts_stat->index++;
}
- /*
- * Measure progress independent of actually having to flush the buffer
- * - otherwise writing become unbalanced.
- */
- ts_stat->progress += ts_stat->progress_slice;
- ts_stat->num_scanned++;
- ts_stat->index++;
+ if (to_write.nbuffers > 0)
+ WriteBuffers(&to_write, ioq, &wb_context);
+
/* Have all the buffers from the tablespace been processed? */
if (ts_stat->num_scanned == ts_stat->num_to_scan)
@@ -3559,15 +3659,23 @@ BufferSync(int flags)
*
* (This will check for barrier events even if it doesn't sleep.)
*/
- CheckpointWriteDelay(flags, (double) num_processed / num_to_scan);
+ CheckpointWriteDelay(ioq, flags, (double) num_processed / num_to_scan);
}
+ Assert(to_write.nbuffers == 0);
+ io_queue_wait_all(ioq);
+
/*
* Issue all pending flushes. Only checkpointer calls BufferSync(), so
* IOContext will always be IOCONTEXT_NORMAL.
*/
IssuePendingWritebacks(&wb_context, IOCONTEXT_NORMAL);
+ io_queue_wait_all(ioq); /* IssuePendingWritebacks might have added
+ * more */
+ io_queue_free(ioq);
+ BuffersToWriteEnd(&to_write);
+
pfree(per_ts_stat);
per_ts_stat = NULL;
binaryheap_free(ts_heap);
@@ -3593,7 +3701,7 @@ BufferSync(int flags)
* bgwriter_lru_maxpages to 0.)
*/
bool
-BgBufferSync(WritebackContext *wb_context)
+BgBufferSync(IOQueue *ioq, WritebackContext *wb_context)
{
/* info obtained from freelist.c */
int strategy_buf_id;
@@ -3636,6 +3744,9 @@ BgBufferSync(WritebackContext *wb_context)
long new_strategy_delta;
uint32 new_recent_alloc;
+ BuffersToWrite to_write;
+ int max_combine;
+
/*
* Find out where the freelist clock sweep currently is, and how many
* buffer allocations have happened since our last call.
@@ -3656,6 +3767,8 @@ BgBufferSync(WritebackContext *wb_context)
return true;
}
+ max_combine = Min(io_bounce_buffers, io_combine_limit);
+
/*
* Compute strategy_delta = how many buffers have been scanned by the
* clock sweep since last time. If first time through, assume none. Then
@@ -3812,11 +3925,25 @@ BgBufferSync(WritebackContext *wb_context)
num_written = 0;
reusable_buffers = reusable_buffers_est;
+ BuffersToWriteInit(&to_write, ioq, wb_context);
+
/* Execute the LRU scan */
while (num_to_scan > 0 && reusable_buffers < upcoming_alloc_est)
{
- int sync_state = SyncOneBuffer(next_to_clean, true,
- wb_context);
+ int sync_state;
+
+ sync_state = PrepareToWriteBuffer(&to_write, next_to_clean + 1,
+ true, ioq, wb_context);
+ if (sync_state & BUF_CANT_MERGE)
+ {
+ Assert(to_write.nbuffers > 0);
+
+ WriteBuffers(&to_write, ioq, wb_context);
+
+ sync_state = PrepareToWriteBuffer(&to_write, next_to_clean + 1,
+ true, ioq, wb_context);
+ Assert(sync_state != BUF_CANT_MERGE);
+ }
if (++next_to_clean >= NBuffers)
{
@@ -3827,6 +3954,13 @@ BgBufferSync(WritebackContext *wb_context)
if (sync_state & BUF_WRITTEN)
{
+ Assert(sync_state & BUF_REUSABLE);
+
+ if (to_write.nbuffers == max_combine)
+ {
+ WriteBuffers(&to_write, ioq, wb_context);
+ }
+
reusable_buffers++;
if (++num_written >= bgwriter_lru_maxpages)
{
@@ -3838,6 +3972,11 @@ BgBufferSync(WritebackContext *wb_context)
reusable_buffers++;
}
+ if (to_write.nbuffers > 0)
+ WriteBuffers(&to_write, ioq, wb_context);
+
+ BuffersToWriteEnd(&to_write);
+
PendingBgWriterStats.buf_written_clean += num_written;
#ifdef BGW_DEBUG
@@ -3876,8 +4015,66 @@ BgBufferSync(WritebackContext *wb_context)
return (bufs_to_lap == 0 && recent_alloc == 0);
}
+static inline bool
+BufferTagsSameRel(const BufferTag *tag1, const BufferTag *tag2)
+{
+ return (tag1->spcOid == tag2->spcOid) &&
+ (tag1->dbOid == tag2->dbOid) &&
+ (tag1->relNumber == tag2->relNumber) &&
+ (tag1->forkNum == tag2->forkNum)
+ ;
+}
+
+static bool
+CanMergeWrite(BuffersToWrite *to_write, BufferDesc *cur_buf_hdr)
+{
+ BlockNumber cur_block = cur_buf_hdr->tag.blockNum;
+
+ Assert(to_write->nbuffers > 0); /* can't merge with nothing */
+ Assert(to_write->start_at_tag.relNumber != InvalidOid);
+ Assert(to_write->start_at_tag.blockNum != InvalidBlockNumber);
+
+ Assert(to_write->ioh != NULL);
+
+ /*
+ * First check if the blocknumber is one that we could actually merge,
+ * that's cheaper than checking the tablespace/db/relnumber/fork match.
+ */
+ if (to_write->start_at_tag.blockNum + to_write->nbuffers != cur_block)
+ return false;
+
+ if (!BufferTagsSameRel(&to_write->start_at_tag, &cur_buf_hdr->tag))
+ return false;
+
+ /*
+ * Need to check with smgr how large a write we're allowed to make. To
+ * reduce the overhead of the smgr check, only inquire once, when
+ * processing the first to-be-merged buffer. That avoids the overhead in
+ * the common case of writing out buffers that definitely not mergeable.
+ */
+ if (to_write->nbuffers == 1)
+ {
+ SMgrRelation smgr;
+
+ smgr = smgropen(BufTagGetRelFileLocator(&to_write->start_at_tag), INVALID_PROC_NUMBER);
+
+ to_write->max_combine = smgrmaxcombine(smgr,
+ to_write->start_at_tag.forkNum,
+ to_write->start_at_tag.blockNum);
+ }
+ else
+ {
+ Assert(to_write->max_combine > 0);
+ }
+
+ if (to_write->start_at_tag.blockNum + to_write->max_combine <= cur_block)
+ return false;
+
+ return true;
+}
+
/*
- * SyncOneBuffer -- process a single buffer during syncing.
+ * PrepareToWriteBuffer -- process a single buffer during syncing.
*
* If skip_recently_used is true, we don't write currently-pinned buffers, nor
* buffers marked recently used, as these are not replacement candidates.
@@ -3886,22 +4083,50 @@ BgBufferSync(WritebackContext *wb_context)
* BUF_WRITTEN: we wrote the buffer.
* BUF_REUSABLE: buffer is available for replacement, ie, it has
* pin count 0 and usage count 0.
+ * BUF_CANT_MERGE: can't combine this write with prior writes, caller needs
+ * to issue those first
*
* (BUF_WRITTEN could be set in error if FlushBuffer finds the buffer clean
* after locking it, but we don't care all that much.)
*/
static int
-SyncOneBuffer(int buf_id, bool skip_recently_used, WritebackContext *wb_context)
+PrepareToWriteBuffer(BuffersToWrite *to_write, Buffer buf,
+ bool skip_recently_used,
+ IOQueue *ioq, WritebackContext *wb_context)
{
- BufferDesc *bufHdr = GetBufferDescriptor(buf_id);
+ BufferDesc *cur_buf_hdr = GetBufferDescriptor(buf - 1);
+ uint32 buf_state;
int result = 0;
- uint32 buf_state;
- BufferTag tag;
+ XLogRecPtr cur_buf_lsn;
+ LWLock *content_lock;
+ bool may_block;
+
+ /*
+ * Check if this buffer can be written out together with already prepared
+ * writes. We check before we have pinned the buffer, so the buffer can be
+ * written out and replaced between this check and us pinning the buffer -
+ * we'll recheck below. The reason for the pre-check is that we don't want
+ * to pin the buffer just to find out that we can't merge the IO.
+ */
+ if (to_write->nbuffers != 0)
+ {
+ if (!CanMergeWrite(to_write, cur_buf_hdr))
+ {
+ result |= BUF_CANT_MERGE;
+ return result;
+ }
+ }
+ else
+ {
+ to_write->start_at_tag = cur_buf_hdr->tag;
+ }
/* Make sure we can handle the pin */
ReservePrivateRefCountEntry();
ResourceOwnerEnlarge(CurrentResourceOwner);
+ /* XXX: Should also check if we are allowed to pin one more buffer */
+
/*
* Check whether buffer needs writing.
*
@@ -3911,7 +4136,7 @@ SyncOneBuffer(int buf_id, bool skip_recently_used, WritebackContext *wb_context)
* don't worry because our checkpoint.redo points before log record for
* upcoming changes and so we are not required to write such dirty buffer.
*/
- buf_state = LockBufHdr(bufHdr);
+ buf_state = LockBufHdr(cur_buf_hdr);
if (BUF_STATE_GET_REFCOUNT(buf_state) == 0 &&
BUF_STATE_GET_USAGECOUNT(buf_state) == 0)
@@ -3920,40 +4145,300 @@ SyncOneBuffer(int buf_id, bool skip_recently_used, WritebackContext *wb_context)
}
else if (skip_recently_used)
{
+#if 0
+ elog(LOG, "at block %d: skip recent with nbuffers %d",
+ cur_buf_hdr->tag.blockNum, to_write->nbuffers);
+#endif
/* Caller told us not to write recently-used buffers */
- UnlockBufHdr(bufHdr, buf_state);
+ UnlockBufHdr(cur_buf_hdr, buf_state);
return result;
}
if (!(buf_state & BM_VALID) || !(buf_state & BM_DIRTY))
{
/* It's clean, so nothing to do */
- UnlockBufHdr(bufHdr, buf_state);
+ UnlockBufHdr(cur_buf_hdr, buf_state);
return result;
}
+ /* pin the buffer, from now on its identity can't change anymore */
+ PinBuffer_Locked(cur_buf_hdr);
+
+ /*
+ * Acquire IO, if needed, now that it's likely that we'll need to write.
+ */
+ if (to_write->ioh == NULL)
+ {
+ /* otherwise we should already have acquired a handle */
+ Assert(to_write->nbuffers == 0);
+
+ to_write->ioh = io_queue_acquire_io(ioq);
+ pgaio_io_get_wref(to_write->ioh, &to_write->iow);
+ }
+
+ /*
+ * If we are merging, check if the buffer's identity possibly changed
+ * while we hadn't yet pinned it.
+ *
+ * XXX: It might be worth checking if we still want to write the buffer
+ * out, e.g. it could have been replaced with a buffer that doesn't have
+ * BM_CHECKPOINT_NEEDED set.
+ */
+ if (to_write->nbuffers != 0)
+ {
+ if (!CanMergeWrite(to_write, cur_buf_hdr))
+ {
+ elog(LOG, "changed identity");
+ UnpinBuffer(cur_buf_hdr);
+
+ result |= BUF_CANT_MERGE;
+
+ return result;
+ }
+ }
+
+ may_block = to_write->nbuffers == 0
+ && !pgaio_have_staged()
+ && io_queue_is_empty(ioq)
+ ;
+ content_lock = BufferDescriptorGetContentLock(cur_buf_hdr);
+
+ if (!may_block)
+ {
+ if (LWLockConditionalAcquire(content_lock, LW_SHARED))
+ {
+ /* done */
+ }
+ else if (to_write->nbuffers == 0)
+ {
+ /*
+ * Need to wait for all prior IO to finish before blocking for
+ * lock acquisition, to avoid the risk a deadlock due to us
+ * waiting for another backend that is waiting for our unsubmitted
+ * IO to complete.
+ */
+ pgaio_submit_staged();
+ io_queue_wait_all(ioq);
+
+ elog(DEBUG2, "at block %u: can't block, nbuffers = 0",
+ cur_buf_hdr->tag.blockNum
+ );
+
+ may_block = to_write->nbuffers == 0
+ && !pgaio_have_staged()
+ && io_queue_is_empty(ioq)
+ ;
+ Assert(may_block);
+
+ LWLockAcquire(content_lock, LW_SHARED);
+ }
+ else
+ {
+ elog(DEBUG2, "at block %d: can't block nbuffers = %d",
+ cur_buf_hdr->tag.blockNum,
+ to_write->nbuffers);
+
+ UnpinBuffer(cur_buf_hdr);
+ result |= BUF_CANT_MERGE;
+ Assert(to_write->nbuffers > 0);
+
+ return result;
+ }
+ }
+ else
+ {
+ LWLockAcquire(content_lock, LW_SHARED);
+ }
+
+ if (!may_block)
+ {
+ if (!StartBufferIO(cur_buf_hdr, false, !may_block))
+ {
+ pgaio_submit_staged();
+ io_queue_wait_all(ioq);
+
+ may_block = io_queue_is_empty(ioq) && to_write->nbuffers == 0 && !pgaio_have_staged();
+
+ if (!StartBufferIO(cur_buf_hdr, false, !may_block))
+ {
+ elog(DEBUG2, "at block %d: non-waitable StartBufferIO returns false, %d",
+ cur_buf_hdr->tag.blockNum,
+ may_block);
+
+ /*
+ * FIXME: can't tell whether this is because the buffer has
+ * been cleaned
+ */
+ if (!may_block)
+ {
+ result |= BUF_CANT_MERGE;
+ Assert(to_write->nbuffers > 0);
+ }
+ LWLockRelease(content_lock);
+ UnpinBuffer(cur_buf_hdr);
+
+ return result;
+ }
+ }
+ }
+ else
+ {
+ if (!StartBufferIO(cur_buf_hdr, false, false))
+ {
+ elog(DEBUG2, "waitable StartBufferIO returns false");
+ LWLockRelease(content_lock);
+ UnpinBuffer(cur_buf_hdr);
+
+ /*
+ * FIXME: Historically we returned BUF_WRITTEN in this case, which
+ * seems wrong
+ */
+ return result;
+ }
+ }
+
/*
- * Pin it, share-lock it, write it. (FlushBuffer will do nothing if the
- * buffer is clean by the time we've locked it.)
+ * Run PageGetLSN while holding header lock, since we don't have the
+ * buffer locked exclusively in all cases.
*/
- PinBuffer_Locked(bufHdr);
- LWLockAcquire(BufferDescriptorGetContentLock(bufHdr), LW_SHARED);
+ buf_state = LockBufHdr(cur_buf_hdr);
+
+ cur_buf_lsn = BufferGetLSN(cur_buf_hdr);
- FlushBuffer(bufHdr, NULL, IOOBJECT_RELATION, IOCONTEXT_NORMAL);
+ /* To check if block content changes while flushing. - vadim 01/17/97 */
+ buf_state &= ~BM_JUST_DIRTIED;
- LWLockRelease(BufferDescriptorGetContentLock(bufHdr));
+ UnlockBufHdr(cur_buf_hdr, buf_state);
- tag = bufHdr->tag;
+ to_write->buffers[to_write->nbuffers] = buf;
+ to_write->nbuffers++;
- UnpinBuffer(bufHdr);
+ if (buf_state & BM_PERMANENT &&
+ (to_write->max_lsn == InvalidXLogRecPtr || to_write->max_lsn < cur_buf_lsn))
+ {
+ to_write->max_lsn = cur_buf_lsn;
+ }
+
+ result |= BUF_WRITTEN;
+
+ return result;
+}
+
+static void
+WriteBuffers(BuffersToWrite *to_write,
+ IOQueue *ioq, WritebackContext *wb_context)
+{
+ SMgrRelation smgr;
+ Buffer first_buf;
+ BufferDesc *first_buf_hdr;
+ bool needs_checksum;
+
+ Assert(to_write->nbuffers > 0 && to_write->nbuffers <= io_combine_limit);
+
+ first_buf = to_write->buffers[0];
+ first_buf_hdr = GetBufferDescriptor(first_buf - 1);
+
+ smgr = smgropen(BufTagGetRelFileLocator(&first_buf_hdr->tag), INVALID_PROC_NUMBER);
/*
- * SyncOneBuffer() is only called by checkpointer and bgwriter, so
- * IOContext will always be IOCONTEXT_NORMAL.
+ * Force XLOG flush up to buffer's LSN. This implements the basic WAL
+ * rule that log updates must hit disk before any of the data-file changes
+ * they describe do.
+ *
+ * However, this rule does not apply to unlogged relations, which will be
+ * lost after a crash anyway. Most unlogged relation pages do not bear
+ * LSNs since we never emit WAL records for them, and therefore flushing
+ * up through the buffer LSN would be useless, but harmless. However,
+ * GiST indexes use LSNs internally to track page-splits, and therefore
+ * unlogged GiST pages bear "fake" LSNs generated by
+ * GetFakeLSNForUnloggedRel. It is unlikely but possible that the fake
+ * LSN counter could advance past the WAL insertion point; and if it did
+ * happen, attempting to flush WAL through that location would fail, with
+ * disastrous system-wide consequences. To make sure that can't happen,
+ * skip the flush if the buffer isn't permanent.
*/
- ScheduleBufferTagForWriteback(wb_context, IOCONTEXT_NORMAL, &tag);
+ if (to_write->max_lsn != InvalidXLogRecPtr)
+ XLogFlush(to_write->max_lsn);
+
+ /*
+ * Now it's safe to write the buffer to disk. Note that no one else should
+ * have been able to write it, while we were busy with log flushing,
+ * because we got the exclusive right to perform I/O by setting the
+ * BM_IO_IN_PROGRESS bit.
+ */
+
+ for (int nbuf = 0; nbuf < to_write->nbuffers; nbuf++)
+ {
+ Buffer cur_buf = to_write->buffers[nbuf];
+ BufferDesc *cur_buf_hdr = GetBufferDescriptor(cur_buf - 1);
+ Block bufBlock;
+ char *bufToWrite;
+
+ bufBlock = BufHdrGetBlock(cur_buf_hdr);
+ needs_checksum = PageNeedsChecksumCopy((Page) bufBlock);
+
+ /*
+ * Update page checksum if desired. Since we have only shared lock on
+ * the buffer, other processes might be updating hint bits in it, so
+ * we must copy the page to a bounce buffer if we do checksumming.
+ */
+ if (needs_checksum)
+ {
+ PgAioBounceBuffer *bb = pgaio_bounce_buffer_get();
- return result | BUF_WRITTEN;
+ pgaio_io_assoc_bounce_buffer(to_write->ioh, bb);
+
+ bufToWrite = pgaio_bounce_buffer_buffer(bb);
+ memcpy(bufToWrite, bufBlock, BLCKSZ);
+ PageSetChecksumInplace((Page) bufToWrite, cur_buf_hdr->tag.blockNum);
+ }
+ else
+ {
+ bufToWrite = bufBlock;
+ }
+
+ to_write->data_ptrs[nbuf] = bufToWrite;
+ }
+
+ pgaio_io_set_handle_data_32(to_write->ioh,
+ (uint32 *) to_write->buffers,
+ to_write->nbuffers);
+ pgaio_io_register_callbacks(to_write->ioh, PGAIO_HCB_SHARED_BUFFER_WRITEV, 0);
+
+ smgrstartwritev(to_write->ioh, smgr,
+ BufTagGetForkNum(&first_buf_hdr->tag),
+ first_buf_hdr->tag.blockNum,
+ to_write->data_ptrs,
+ to_write->nbuffers,
+ false);
+ pgstat_count_io_op(IOOBJECT_RELATION, IOCONTEXT_NORMAL,
+ IOOP_WRITE, 1, BLCKSZ * to_write->nbuffers);
+
+
+ for (int nbuf = 0; nbuf < to_write->nbuffers; nbuf++)
+ {
+ Buffer cur_buf = to_write->buffers[nbuf];
+ BufferDesc *cur_buf_hdr = GetBufferDescriptor(cur_buf - 1);
+
+ UnpinBuffer(cur_buf_hdr);
+ }
+
+ io_queue_track(ioq, &to_write->iow);
+ to_write->total_writes++;
+
+ /* clear state for next write */
+ to_write->nbuffers = 0;
+ to_write->start_at_tag.relNumber = InvalidOid;
+ to_write->start_at_tag.blockNum = InvalidBlockNumber;
+ to_write->max_combine = 0;
+ to_write->max_lsn = InvalidXLogRecPtr;
+ to_write->ioh = NULL;
+ pgaio_wref_clear(&to_write->iow);
+
+ /*
+ * FIXME: Implement issuing writebacks (note wb_context isn't used here).
+ * Possibly needs to be integrated with io_queue.c.
+ */
}
/*
@@ -4327,6 +4812,7 @@ FlushBuffer(BufferDesc *buf, SMgrRelation reln, IOObject io_object,
error_context_stack = errcallback.previous;
}
+
/*
* RelationGetNumberOfBlocksInFork
* Determines the current number of pages in the specified relation fork.
diff --git a/src/backend/storage/page/bufpage.c b/src/backend/storage/page/bufpage.c
index ecc81aacfc3..71dc6f559dd 100644
--- a/src/backend/storage/page/bufpage.c
+++ b/src/backend/storage/page/bufpage.c
@@ -1480,6 +1480,16 @@ PageIndexTupleOverwrite(Page page, OffsetNumber offnum,
return true;
}
+bool
+PageNeedsChecksumCopy(Page page)
+{
+ if (PageIsNew(page))
+ return false;
+
+ /* If we don't need a checksum, just return the passed-in data */
+ return DataChecksumsEnabled();
+}
+
/*
* Set checksum for a page in shared buffers.
diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list
index 0e40687bb43..125fcf35bd8 100644
--- a/src/tools/pgindent/typedefs.list
+++ b/src/tools/pgindent/typedefs.list
@@ -348,6 +348,7 @@ BufferManagerRelation
BufferStrategyControl
BufferTag
BufferUsage
+BuffersToWrite
BuildAccumulator
BuiltinScript
BulkInsertState
--
2.48.1.76.g4e746b1a31.dirty
[text/x-diff] v2.12-0024-Ensure-a-resowner-exists-for-all-paths-that-ma.patch (2.6K, ../../cxi6ssym62v7ywkt367qda3xik5jhsrcycwedyfqacinxooys5@pwarur37bosc/25-v2.12-0024-Ensure-a-resowner-exists-for-all-paths-that-ma.patch)
download | inline diff:
From 7d6bc867506568482bd40c48268bf586c8470d06 Mon Sep 17 00:00:00 2001
From: Andres Freund <andres@anarazel.de>
Date: Tue, 18 Mar 2025 14:40:06 -0400
Subject: [PATCH v2.12 24/28] Ensure a resowner exists for all paths that may
perform AIO
Reviewed-by: Noah Misch <noah@leadboat.com>
Discussion: https://postgr.es/m/1f6b50a7-38ef-4d87-8246-786d39f46ab9@iki.fi
---
src/backend/bootstrap/bootstrap.c | 7 +++++++
src/backend/replication/logical/logical.c | 6 ++++++
src/backend/utils/init/postinit.c | 6 +++++-
3 files changed, 18 insertions(+), 1 deletion(-)
diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c
index 6db864892d0..e554504e1f0 100644
--- a/src/backend/bootstrap/bootstrap.c
+++ b/src/backend/bootstrap/bootstrap.c
@@ -361,8 +361,15 @@ BootstrapModeMain(int argc, char *argv[], bool check_only)
BaseInit();
bootstrap_signals();
+
+ /* need a resowner for IO during BootStrapXLOG() */
+ CreateAuxProcessResourceOwner();
+
BootStrapXLOG(bootstrap_data_checksum_version);
+ ReleaseAuxProcessResources(true);
+ CurrentResourceOwner = NULL;
+
/*
* To ensure that src/common/link-canary.c is linked into the backend, we
* must call it from somewhere. Here is as good as anywhere.
diff --git a/src/backend/replication/logical/logical.c b/src/backend/replication/logical/logical.c
index 8ea846bfc3b..2b936714040 100644
--- a/src/backend/replication/logical/logical.c
+++ b/src/backend/replication/logical/logical.c
@@ -386,6 +386,12 @@ CreateInitDecodingContext(const char *plugin,
slot->data.plugin = plugin_name;
SpinLockRelease(&slot->mutex);
+ if (CurrentResourceOwner == NULL)
+ {
+ Assert(am_walsender);
+ CurrentResourceOwner = AuxProcessResourceOwner;
+ }
+
if (XLogRecPtrIsInvalid(restart_lsn))
ReplicationSlotReserveWal();
else
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index 7958ea11b73..222e24bcb08 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -785,8 +785,12 @@ InitPostgres(const char *in_dbname, Oid dboid,
* We don't yet have an aux-process resource owner, but StartupXLOG
* and ShutdownXLOG will need one. Hence, create said resource owner
* (and register a callback to clean it up after ShutdownXLOG runs).
+ *
+ * In bootstrap mode CreateAuxProcessResourceOwner() was already
+ * called in BootstrapModeMain().
*/
- CreateAuxProcessResourceOwner();
+ if (!bootstrap)
+ CreateAuxProcessResourceOwner();
StartupXLOG();
/* Release (and warn about) any buffer pins leaked in StartupXLOG */
--
2.48.1.76.g4e746b1a31.dirty
[text/x-diff] v2.12-0025-Temporary-Increase-BAS_BULKREAD-size.patch (1.3K, ../../cxi6ssym62v7ywkt367qda3xik5jhsrcycwedyfqacinxooys5@pwarur37bosc/26-v2.12-0025-Temporary-Increase-BAS_BULKREAD-size.patch)
download | inline diff:
From d798396cff0a52848210550a5e21785e02cb54b6 Mon Sep 17 00:00:00 2001
From: Andres Freund <andres@anarazel.de>
Date: Tue, 18 Mar 2025 14:40:06 -0400
Subject: [PATCH v2.12 25/28] Temporary: Increase BAS_BULKREAD size
Without this we only can execute very little AIO for sequential scans, as
there's just not enough buffers in the ring. This isn't the right fix, as
just increasing the ring size can have negative performance implications in
workloads where the kernel has all the data cached.
Author:
Reviewed-By:
Discussion: https://postgr.es/m/
Backpatch:
---
src/backend/storage/buffer/freelist.c | 7 ++++++-
1 file changed, 6 insertions(+), 1 deletion(-)
diff --git a/src/backend/storage/buffer/freelist.c b/src/backend/storage/buffer/freelist.c
index 336715b6c63..b72a5957a20 100644
--- a/src/backend/storage/buffer/freelist.c
+++ b/src/backend/storage/buffer/freelist.c
@@ -555,7 +555,12 @@ GetAccessStrategy(BufferAccessStrategyType btype)
return NULL;
case BAS_BULKREAD:
- ring_size_kb = 256;
+
+ /*
+ * FIXME: Temporary increase to allow large enough streaming reads
+ * to actually benefit from AIO. This needs a better solution.
+ */
+ ring_size_kb = 2 * 1024;
break;
case BAS_BULKWRITE:
ring_size_kb = 16 * 1024;
--
2.48.1.76.g4e746b1a31.dirty
[text/x-diff] v2.12-0026-WIP-Use-MAP_POPULATE.patch (1.1K, ../../cxi6ssym62v7ywkt367qda3xik5jhsrcycwedyfqacinxooys5@pwarur37bosc/27-v2.12-0026-WIP-Use-MAP_POPULATE.patch)
download | inline diff:
From 82e31a4a036727b32bb32a6051bf4b2d2ea634cd Mon Sep 17 00:00:00 2001
From: Andres Freund <andres@anarazel.de>
Date: Tue, 18 Mar 2025 14:40:06 -0400
Subject: [PATCH v2.12 26/28] WIP: Use MAP_POPULATE
For benchmarking it's quite annoying that the first time a memory is touched
has completely different perf characteristics than subsequent accesses. Using
MAP_POPULATE reduces that substantially.
---
src/backend/port/sysv_shmem.c | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/src/backend/port/sysv_shmem.c b/src/backend/port/sysv_shmem.c
index 197926d44f6..a700b02d5a1 100644
--- a/src/backend/port/sysv_shmem.c
+++ b/src/backend/port/sysv_shmem.c
@@ -620,7 +620,7 @@ CreateAnonymousSegment(Size *size)
allocsize += hugepagesize - (allocsize % hugepagesize);
ptr = mmap(NULL, allocsize, PROT_READ | PROT_WRITE,
- PG_MMAP_FLAGS | mmap_flags, -1, 0);
+ PG_MMAP_FLAGS | MAP_POPULATE | mmap_flags, -1, 0);
mmap_errno = errno;
if (huge_pages == HUGE_PAGES_TRY && ptr == MAP_FAILED)
elog(DEBUG1, "mmap(%zu) with MAP_HUGETLB failed, huge pages disabled: %m",
--
2.48.1.76.g4e746b1a31.dirty
^ permalink raw reply [nested|flat] 153+ messages in thread
* Re: AIO v2.5
2025-03-04 19:00 AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-10 18:23 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-11 19:41 ` Re: AIO v2.5 Noah Misch <noah@leadboat.com>
2025-03-11 23:55 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-12 03:57 ` Re: AIO v2.5 Noah Misch <noah@leadboat.com>
2025-03-12 17:06 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-12 23:23 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-13 10:53 ` Re: AIO v2.5 Antonin Houska <ah@cybertec.at>
2025-03-14 19:43 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-17 20:00 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-18 20:12 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-21 01:58 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-25 01:18 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
@ 2025-03-25 04:10 ` Thomas Munro <thomas.munro@gmail.com>
2025-03-25 15:33 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
4 siblings, 1 reply; 153+ messages in thread
From: Thomas Munro @ 2025-03-25 04:10 UTC (permalink / raw)
To: Andres Freund <andres@anarazel.de>; +Cc: Antonin Houska <ah@cybertec.at>; Noah Misch <noah@leadboat.com>; pgsql-hackers@postgresql.org, Heikki Linnakangas <hlinnaka@iki.fi>; Robert Haas <robertmhaas@gmail.com>; Jakub Wartak <jakub.wartak@enterprisedb.com>; Jelte Fennema-Nio <postgres@jeltef.nl>
On Tue, Mar 25, 2025 at 2:18 PM Andres Freund <andres@anarazel.de> wrote:
> Attached v2.12, with the following changes:
Here's a tiny fixup to make io_concurrency=0 turn on
READ_BUFFERS_SYNCHRONOUSLY as mooted in a FIXME. Without this, AIO
will still run at level 1 even if you asked for 0. Feel free to
squash, or ignore and I'll push it later, whatever suits... (tested on
the tip of your public aio-2 branch).
Attachments:
[application/octet-stream] 0001-fixup-to-v2.12-0010.fixup (3.7K, ../../CA+hUKG+8SC2=AD3bC0Pn85aMXm-PE2JSFGhC=MFVJvNQLObZeA@mail.gmail.com/2-0001-fixup-to-v2.12-0010.fixup)
download
^ permalink raw reply [nested|flat] 153+ messages in thread
* Re: AIO v2.5
2025-03-04 19:00 AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-10 18:23 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-11 19:41 ` Re: AIO v2.5 Noah Misch <noah@leadboat.com>
2025-03-11 23:55 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-12 03:57 ` Re: AIO v2.5 Noah Misch <noah@leadboat.com>
2025-03-12 17:06 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-12 23:23 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-13 10:53 ` Re: AIO v2.5 Antonin Houska <ah@cybertec.at>
2025-03-14 19:43 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-17 20:00 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-18 20:12 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-21 01:58 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-25 01:18 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-25 04:10 ` Re: AIO v2.5 Thomas Munro <thomas.munro@gmail.com>
@ 2025-03-25 15:33 ` Andres Freund <andres@anarazel.de>
0 siblings, 0 replies; 153+ messages in thread
From: Andres Freund @ 2025-03-25 15:33 UTC (permalink / raw)
To: Thomas Munro <thomas.munro@gmail.com>; +Cc: Antonin Houska <ah@cybertec.at>; Noah Misch <noah@leadboat.com>; pgsql-hackers@postgresql.org, Heikki Linnakangas <hlinnaka@iki.fi>; Robert Haas <robertmhaas@gmail.com>; Jakub Wartak <jakub.wartak@enterprisedb.com>; Jelte Fennema-Nio <postgres@jeltef.nl>
Hi,
On 2025-03-25 17:10:19 +1300, Thomas Munro wrote:
> On Tue, Mar 25, 2025 at 2:18 PM Andres Freund <andres@anarazel.de> wrote:
> > Attached v2.12, with the following changes:
>
> Here's a tiny fixup to make io_concurrency=0 turn on
> READ_BUFFERS_SYNCHRONOUSLY as mooted in a FIXME. Without this, AIO
> will still run at level 1 even if you asked for 0. Feel free to
> squash, or ignore and I'll push it later, whatever suits... (tested on
> the tip of your public aio-2 branch).
Thanks! I squashed it into "aio: Basic read_stream adjustments for real AIO"
and updated the commit message to account for that.
Greetings,
Andres Freund
^ permalink raw reply [nested|flat] 153+ messages in thread
* Re: AIO v2.5
2025-03-04 19:00 AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-10 18:23 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-11 19:41 ` Re: AIO v2.5 Noah Misch <noah@leadboat.com>
2025-03-11 23:55 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-12 03:57 ` Re: AIO v2.5 Noah Misch <noah@leadboat.com>
2025-03-12 17:06 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-12 23:23 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-13 10:53 ` Re: AIO v2.5 Antonin Houska <ah@cybertec.at>
2025-03-14 19:43 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-17 20:00 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-18 20:12 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-21 01:58 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-25 01:18 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
@ 2025-03-26 00:19 ` Noah Misch <noah@leadboat.com>
2025-03-26 20:33 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
4 siblings, 1 reply; 153+ messages in thread
From: Noah Misch @ 2025-03-26 00:19 UTC (permalink / raw)
To: Andres Freund <andres@anarazel.de>; +Cc: Antonin Houska <ah@cybertec.at>; pgsql-hackers@postgresql.org, Thomas Munro <thomas.munro@gmail.com>; Heikki Linnakangas <hlinnaka@iki.fi>; Robert Haas <robertmhaas@gmail.com>; Jakub Wartak <jakub.wartak@enterprisedb.com>; Jelte Fennema-Nio <postgres@jeltef.nl>
On Mon, Mar 24, 2025 at 09:18:06PM -0400, Andres Freund wrote:
> Attached v2.12, with the following changes:
> TODO:
> Wonder if it's worth adding some coverage for when checksums are disabled?
> Probably not necessary?
Probably not necessary, agreed. Orthogonal to AIO, it's likely worth a CI
"SPECIAL" and/or buildfarm animal that runs all tests w/ checksums disabled.
> Subject: [PATCH v2.12 01/28] aio: Be more paranoid about interrupts
Ready for commit
> Subject: [PATCH v2.12 02/28] aio: Pass result of local callbacks to
> ->report_return
Ready for commit w/ up to one cosmetic change:
> @@ -296,7 +299,9 @@ pgaio_io_call_complete_local(PgAioHandle *ioh)
>
> /*
> * Note that we don't save the result in ioh->distilled_result, the local
> - * callback's result should not ever matter to other waiters.
> + * callback's result should not ever matter to other waiters. However, the
> + * local backend does care, so we return the result as modified by local
> + * callbacks, which then can be passed to ioh->report_return->result.
> */
> pgaio_debug_io(DEBUG3, ioh,
> "after local completion: distilled result: (status %s, id %u, error_data %d, result %d), raw_result: %d",
Should this debug message remove the word "distilled", since this commit
solidifies distilled_result as referring to the complete_shared result?
> Subject: [PATCH v2.12 03/28] aio: Add liburing dependency
Ready for commit
> Subject: [PATCH v2.12 04/28] aio: Add io_method=io_uring
Ready for commit w/ open_fd.fixup
> Subject: [PATCH v2.12 05/28] aio: Implement support for reads in smgr/md/fd
Ready for commit w/ up to two cosmetic changes:
> +/*
> + * AIO error reporting callback for mdstartreadv().
> + *
> + * Errors are encoded as follows:
> + * - PgAioResult.error_data != 0 encodes IO that failed with errno != 0
I recommend replacing "errno != 0" with either "that errno" or "errno ==
error_data".
Second, the aio_internal.h comment changes discussed in
postgr.es/m/20250325155808.f7.nmisch@google.com and earlier.
> Subject: [PATCH v2.12 06/28] aio: Add README.md explaining higher level design
Ready for commit
(This and the previous patch have three spots that would change with the
s/prep/start/ renames. No opinion on whether to rename before or rename
after.)
> Subject: [PATCH v2.12 07/28] localbuf: Track pincount in BufferDesc as well
The plan here looks good:
postgr.es/m/dbeeaize47y7esifdrinpa2l7cqqb67k72exvuf3appyxywjnc@7bt76mozhcy2
> Subject: [PATCH v2.12 08/28] bufmgr: Implement AIO read support
See review here and later discussion:
postgr.es/m/20250325022037.91.nmisch@google.com
> Subject: [PATCH v2.12 09/28] bufmgr: Use AIO in StartReadBuffers()
Ready for commit after a batch of small things, all but one of which have no
implications beyond code cosmetics. This is my first comprehensive review of
this patch. I like the test coverage (by the end of the patch series). For
anyone else following, I found "diff -w" helpful for the bufmgr.c changes.
That's because a key part is former WaitReadBuffers() code moving up an
indentation level to its home in new subroutine AsyncReadBuffers().
> Assert(*nblocks == 1 || allow_forwarding);
> Assert(*nblocks > 0);
> Assert(*nblocks <= MAX_IO_COMBINE_LIMIT);
> + Assert(*nblocks == 1 || allow_forwarding);
Duplicates the assert three lines back.
> + nblocks = aio_ret->result.result;
> +
> + elog(DEBUG3, "partial read, will retry");
> +
> + }
> + else if (aio_ret->result.status == PGAIO_RS_ERROR)
> + {
> + pgaio_result_report(aio_ret->result, &aio_ret->target_data, ERROR);
> + nblocks = 0; /* silence compiler */
> + }
>
> Assert(nblocks > 0);
> Assert(nblocks <= MAX_IO_COMBINE_LIMIT);
>
> + operation->nblocks_done += nblocks;
I struggled somewhat from the variety of "nblocks" variables: this local
nblocks, operation->nblocks, actual_nblocks, and *nblocks in/out parameters of
some functions. No one of them is clearly wrong to use the name, and some of
these names are preexisting. That said, if you see opportunities to push in
the direction of more-specific names, I'd welcome it.
For example, this local variable could become add_to_nblocks_done instead.
> + AsyncReadBuffers(operation, &nblocks);
I suggest renaming s/nblocks/ignored_nblocks_progress/ here.
> + * If we need to wait for IO before we can get a handle, submit already
> + * staged IO first, so that other backends don't need to wait. There
s/already staged/already-staged/. Normally I'd skip this as nitpicking, but I
misread this particular sentence twice, as "submit" being the subject that
"staged" something. (It's still nitpicking, alas.)
> /*
> * How many neighboring-on-disk blocks can we scatter-read into other
> * buffers at the same time? In this case we don't wait if we see an
> - * I/O already in progress. We already hold BM_IO_IN_PROGRESS for the
> + * I/O already in progress. We already set BM_IO_IN_PROGRESS for the
> * head block, so we should get on with that I/O as soon as possible.
> - * We'll come back to this block again, above.
> + *
> + * We'll come back to this block in the next call to
> + * StartReadBuffers() -> AsyncReadBuffers().
Did this mean to say "WaitReadBuffers() -> AsyncReadBuffers()"? I'm guessing
so, since WaitReadBuffers() is the one that loops. It might be referring to
read_stream_start_pending_read()'s next StartReadBuffers(), though.
I think this could just delete the last sentence. The function header comment
already mentions the possibility of reading a subset of the request. This
spot doesn't need to detail how the higher layers come back to here.
> + smgrstartreadv(ioh, operation->smgr, forknum,
> + blocknum + nblocks_done,
> + io_pages, io_buffers_len);
> + pgstat_count_io_op_time(io_object, io_context, IOOP_READ,
> + io_start, 1, *nblocks_progress * BLCKSZ);
We don't assign *nblocks_progress until lower in the function, so I think
"io_buffers_len" should replace "*nblocks_progress" here. (This is my only
non-cosmetic comment on this patch.)
> Subject: [PATCH v2.12 10/28] aio: Basic read_stream adjustments for real AIO
(Still reviewing this and later patches, but incidental observations follow.)
> Subject: [PATCH v2.12 16/28] aio: Add test_aio module
> +use List::Util qw(sample);
sample() is new in 2020:
https://metacpan.org/release/PEVANS/Scalar-List-Utils-1.68/source/Changes#L100
Hence, I'd expect some buildfarm failures. I'd try to use shuffle(), then
take the first N elements.
> +++ b/src/test/modules/test_aio/test_aio.c
> @@ -0,0 +1,712 @@
> +/*-------------------------------------------------------------------------
> + *
> + * delay_execution.c
> + * Test module to allow delay between parsing and execution of a query.
> + *
> + * The delay is implemented by taking and immediately releasing a specified
> + * advisory lock. If another process has previously taken that lock, the
> + * current process will be blocked until the lock is released; otherwise,
> + * there's no effect. This allows an isolationtester script to reliably
> + * test behaviors where some specified action happens in another backend
> + * between parsing and execution of any desired query.
> + *
> + * Copyright (c) 2020-2025, PostgreSQL Global Development Group
> + *
> + * IDENTIFICATION
> + * src/test/modules/test_aio/test_aio.c
To elaborate on my last review, the entire header comment was a copy from
delay_execution.c. v2.12 fixes the IDENTIFICATION, but the rest needs
updates.
Thanks,
nm
^ permalink raw reply [nested|flat] 153+ messages in thread
* Re: AIO v2.5
2025-03-04 19:00 AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-10 18:23 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-11 19:41 ` Re: AIO v2.5 Noah Misch <noah@leadboat.com>
2025-03-11 23:55 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-12 03:57 ` Re: AIO v2.5 Noah Misch <noah@leadboat.com>
2025-03-12 17:06 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-12 23:23 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-13 10:53 ` Re: AIO v2.5 Antonin Houska <ah@cybertec.at>
2025-03-14 19:43 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-17 20:00 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-18 20:12 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-21 01:58 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-25 01:18 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-26 00:19 ` Re: AIO v2.5 Noah Misch <noah@leadboat.com>
@ 2025-03-26 20:33 ` Andres Freund <andres@anarazel.de>
2025-03-26 21:21 ` Re: AIO v2.5 Noah Misch <noah@leadboat.com>
0 siblings, 1 reply; 153+ messages in thread
From: Andres Freund @ 2025-03-26 20:33 UTC (permalink / raw)
To: Noah Misch <noah@leadboat.com>; +Cc: Antonin Houska <ah@cybertec.at>; pgsql-hackers@postgresql.org, Thomas Munro <thomas.munro@gmail.com>; Heikki Linnakangas <hlinnaka@iki.fi>; Robert Haas <robertmhaas@gmail.com>; Jakub Wartak <jakub.wartak@enterprisedb.com>; Jelte Fennema-Nio <postgres@jeltef.nl>
Hi,
On 2025-03-25 17:19:15 -0700, Noah Misch wrote:
> On Mon, Mar 24, 2025 at 09:18:06PM -0400, Andres Freund wrote:
> > @@ -296,7 +299,9 @@ pgaio_io_call_complete_local(PgAioHandle *ioh)
> >
> > /*
> > * Note that we don't save the result in ioh->distilled_result, the local
> > - * callback's result should not ever matter to other waiters.
> > + * callback's result should not ever matter to other waiters. However, the
> > + * local backend does care, so we return the result as modified by local
> > + * callbacks, which then can be passed to ioh->report_return->result.
> > */
> > pgaio_debug_io(DEBUG3, ioh,
> > "after local completion: distilled result: (status %s, id %u, error_data %d, result %d), raw_result: %d",
>
> Should this debug message remove the word "distilled", since this commit
> solidifies distilled_result as referring to the complete_shared result?
Good point, updated.
> > Subject: [PATCH v2.12 01/28] aio: Be more paranoid about interrupts
> Ready for commit
> > Subject: [PATCH v2.12 02/28] aio: Pass result of local callbacks to
> > ->report_return
>
> Ready for commit w/ up to one cosmetic change:
>
And pushed. Together with the s/pgaio_io_prep_/s/pgaio_io_start_/ renaming
we've been discussing. Btw, I figured out the origin of that, I was just
mirroring the liburing API...
Thanks again for the reviews.
> > Subject: [PATCH v2.12 03/28] aio: Add liburing dependency
>
> Ready for commit
>
>
> > Subject: [PATCH v2.12 04/28] aio: Add io_method=io_uring
>
> Ready for commit w/ open_fd.fixup
Yay. Planning to push those soon.
> > Subject: [PATCH v2.12 05/28] aio: Implement support for reads in smgr/md/fd
>
> Ready for commit w/ up to two cosmetic changes:
Cool.
> > +/*
> > + * AIO error reporting callback for mdstartreadv().
> > + *
> > + * Errors are encoded as follows:
> > + * - PgAioResult.error_data != 0 encodes IO that failed with errno != 0
>
> I recommend replacing "errno != 0" with either "that errno" or "errno ==
> error_data".
Applied.
> Second, the aio_internal.h comment changes discussed in
> postgr.es/m/20250325155808.f7.nmisch@google.com and earlier.
Here's my current version of that:
* Note that the externally visible functions to start IO
* (e.g. FileStartReadV(), via pgaio_io_start_readv()) move an IO from
* PGAIO_HS_HANDED_OUT to at least PGAIO_HS_STAGED and at most
* PGAIO_HS_COMPLETED_LOCAL (at which point the handle will be reused).
Does that work?
I think I'll push that as part of the comment updates patch instead of
"Implement support for reads in smgr/md/fd", unless you see a reason to do so
differently. I'd have done it in the patch to s/prep/start/, but then it would
reference functions that don't exist yet...
> > Subject: [PATCH v2.12 06/28] aio: Add README.md explaining higher level design
>
> Ready for commit
Cool.
Comments in it reference PGAIO_HCB_SHARED_BUFFER_READV, so I'm inclined to
reorder it until after "bufmgr: Implement AIO read support".
There's also a small change in a new patch in the series (not yet sent), due
to the changes related to emitting WARNINGs about checksum failures to the
client connection. I think that part is fine, but...
> (This and the previous patch have three spots that would change with the
> s/prep/start/ renames. No opinion on whether to rename before or rename
> after.)
I thought it'd be better to do the renaming first.
> > Subject: [PATCH v2.12 07/28] localbuf: Track pincount in BufferDesc as well
>
> The plan here looks good:
> postgr.es/m/dbeeaize47y7esifdrinpa2l7cqqb67k72exvuf3appyxywjnc@7bt76mozhcy2
> > Subject: [PATCH v2.12 08/28] bufmgr: Implement AIO read support
>
> See review here and later discussion:
> postgr.es/m/20250325022037.91.nmisch@google.com
I'm working on a version with those addressed.
> > Subject: [PATCH v2.12 09/28] bufmgr: Use AIO in StartReadBuffers()
>
> Ready for commit after a batch of small things, all but one of which have no
> implications beyond code cosmetics.
Yay.
> I like the test coverage (by the end of the patch series).
I'm really shocked just how bad our test coverage for a lot of this is today
:(
> For anyone else following, I found "diff -w" helpful for the bufmgr.c
> changes. That's because a key part is former WaitReadBuffers() code moving
> up an indentation level to its home in new subroutine AsyncReadBuffers().
For reviewing changes that move stuff around a lot I find this rather helpful:
git diff --color-moved --color-moved-ws=ignore-space-change
That highlights removed code differently from moved code, and due to
ignore-space-change considers code that changed just due to space changes, to
be moved.
> > Assert(*nblocks == 1 || allow_forwarding);
> > Assert(*nblocks > 0);
> > Assert(*nblocks <= MAX_IO_COMBINE_LIMIT);
> > + Assert(*nblocks == 1 || allow_forwarding);
>
> Duplicates the assert three lines back.
Ah, it was moved into ce1a75c4fea, which I didn't notice while rebasing...
> > + nblocks = aio_ret->result.result;
> > +
> > + elog(DEBUG3, "partial read, will retry");
> > +
> > + }
> > + else if (aio_ret->result.status == PGAIO_RS_ERROR)
> > + {
> > + pgaio_result_report(aio_ret->result, &aio_ret->target_data, ERROR);
> > + nblocks = 0; /* silence compiler */
> > + }
> >
> > Assert(nblocks > 0);
> > Assert(nblocks <= MAX_IO_COMBINE_LIMIT);
> >
> > + operation->nblocks_done += nblocks;
>
> I struggled somewhat from the variety of "nblocks" variables: this local
> nblocks, operation->nblocks, actual_nblocks, and *nblocks in/out parameters of
> some functions. No one of them is clearly wrong to use the name, and some of
> these names are preexisting. That said, if you see opportunities to push in
> the direction of more-specific names, I'd welcome it.
>
> For example, this local variable could become add_to_nblocks_done instead.
I named it "newly_read_blocks", hope that works?
> > + AsyncReadBuffers(operation, &nblocks);
>
> I suggest renaming s/nblocks/ignored_nblocks_progress/ here.
Adopted.
Unfortunately I didn't see a good way to reduce the amount of the other
nblocks variables, as they are all, I think, preexisting.
> > + * If we need to wait for IO before we can get a handle, submit already
> > + * staged IO first, so that other backends don't need to wait. There
>
> s/already staged/already-staged/. Normally I'd skip this as nitpicking, but I
> misread this particular sentence twice, as "submit" being the subject that
> "staged" something. (It's still nitpicking, alas.)
Makes sense - it doesn't help that it was at a linebreak...
> > /*
> > * How many neighboring-on-disk blocks can we scatter-read into other
> > * buffers at the same time? In this case we don't wait if we see an
> > - * I/O already in progress. We already hold BM_IO_IN_PROGRESS for the
> > + * I/O already in progress. We already set BM_IO_IN_PROGRESS for the
> > * head block, so we should get on with that I/O as soon as possible.
> > - * We'll come back to this block again, above.
> > + *
> > + * We'll come back to this block in the next call to
> > + * StartReadBuffers() -> AsyncReadBuffers().
>
> Did this mean to say "WaitReadBuffers() -> AsyncReadBuffers()"? I'm guessing
> so, since WaitReadBuffers() is the one that loops. It might be referring to
> read_stream_start_pending_read()'s next StartReadBuffers(), though.
I was referring to the latter, as that is the more common case (it's pretty
easy to hit if you e.g. have multiple sequential scans on the same table
going).
> I think this could just delete the last sentence. The function header comment
> already mentions the possibility of reading a subset of the request. This
> spot doesn't need to detail how the higher layers come back to here.
Agreed.
> > + smgrstartreadv(ioh, operation->smgr, forknum,
> > + blocknum + nblocks_done,
> > + io_pages, io_buffers_len);
> > + pgstat_count_io_op_time(io_object, io_context, IOOP_READ,
> > + io_start, 1, *nblocks_progress * BLCKSZ);
>
> We don't assign *nblocks_progress until lower in the function, so I think
> "io_buffers_len" should replace "*nblocks_progress" here. (This is my only
> non-cosmetic comment on this patch.)
Good catch!
> > Subject: [PATCH v2.12 16/28] aio: Add test_aio module
>
> > +use List::Util qw(sample);
>
> sample() is new in 2020:
> https://metacpan.org/release/PEVANS/Scalar-List-Utils-1.68/source/Changes#L100
>
> Hence, I'd expect some buildfarm failures. I'd try to use shuffle(), then
> take the first N elements.
Hah. Bilal's patch was using shuffe(). I wanted to reduce the number of
iterations and first did as you suggest and then saw that there's a nicer
way...
Done that way again...
> > +++ b/src/test/modules/test_aio/test_aio.c
> > @@ -0,0 +1,712 @@
> > +/*-------------------------------------------------------------------------
> > + *
> > + * delay_execution.c
> > + * Test module to allow delay between parsing and execution of a query.
> > + *
> > + * The delay is implemented by taking and immediately releasing a specified
> > + * advisory lock. If another process has previously taken that lock, the
> > + * current process will be blocked until the lock is released; otherwise,
> > + * there's no effect. This allows an isolationtester script to reliably
> > + * test behaviors where some specified action happens in another backend
> > + * between parsing and execution of any desired query.
> > + *
> > + * Copyright (c) 2020-2025, PostgreSQL Global Development Group
> > + *
> > + * IDENTIFICATION
> > + * src/test/modules/test_aio/test_aio.c
>
> To elaborate on my last review, the entire header comment was a copy from
> delay_execution.c. v2.12 fixes the IDENTIFICATION, but the rest needs
> updates.
I was really too tired that day... Embarassing.
Greetings,
Andres Freund
^ permalink raw reply [nested|flat] 153+ messages in thread
* Re: AIO v2.5
2025-03-04 19:00 AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-10 18:23 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-11 19:41 ` Re: AIO v2.5 Noah Misch <noah@leadboat.com>
2025-03-11 23:55 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-12 03:57 ` Re: AIO v2.5 Noah Misch <noah@leadboat.com>
2025-03-12 17:06 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-12 23:23 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-13 10:53 ` Re: AIO v2.5 Antonin Houska <ah@cybertec.at>
2025-03-14 19:43 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-17 20:00 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-18 20:12 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-21 01:58 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-25 01:18 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-26 00:19 ` Re: AIO v2.5 Noah Misch <noah@leadboat.com>
2025-03-26 20:33 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
@ 2025-03-26 21:21 ` Noah Misch <noah@leadboat.com>
0 siblings, 0 replies; 153+ messages in thread
From: Noah Misch @ 2025-03-26 21:21 UTC (permalink / raw)
To: Andres Freund <andres@anarazel.de>; +Cc: Antonin Houska <ah@cybertec.at>; pgsql-hackers@postgresql.org, Thomas Munro <thomas.munro@gmail.com>; Heikki Linnakangas <hlinnaka@iki.fi>; Robert Haas <robertmhaas@gmail.com>; Jakub Wartak <jakub.wartak@enterprisedb.com>; Jelte Fennema-Nio <postgres@jeltef.nl>
On Wed, Mar 26, 2025 at 04:33:49PM -0400, Andres Freund wrote:
> On 2025-03-25 17:19:15 -0700, Noah Misch wrote:
> > On Mon, Mar 24, 2025 at 09:18:06PM -0400, Andres Freund wrote:
> > Second, the aio_internal.h comment changes discussed in
> > postgr.es/m/20250325155808.f7.nmisch@google.com and earlier.
>
> Here's my current version of that:
>
> * Note that the externally visible functions to start IO
> * (e.g. FileStartReadV(), via pgaio_io_start_readv()) move an IO from
> * PGAIO_HS_HANDED_OUT to at least PGAIO_HS_STAGED and at most
> * PGAIO_HS_COMPLETED_LOCAL (at which point the handle will be reused).
>
> Does that work?
Yes.
> I think I'll push that as part of the comment updates patch instead of
> "Implement support for reads in smgr/md/fd", unless you see a reason to do so
> differently. I'd have done it in the patch to s/prep/start/, but then it would
> reference functions that don't exist yet...
Agreed.
> > > Subject: [PATCH v2.12 06/28] aio: Add README.md explaining higher level design
> >
> > Ready for commit
>
> Cool.
>
> Comments in it reference PGAIO_HCB_SHARED_BUFFER_READV, so I'm inclined to
> reorder it until after "bufmgr: Implement AIO read support".
Agreed.
> > For example, this local variable could become add_to_nblocks_done instead.
>
> I named it "newly_read_blocks", hope that works?
Yes.
^ permalink raw reply [nested|flat] 153+ messages in thread
* Re: AIO v2.5
2025-03-04 19:00 AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-10 18:23 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-11 19:41 ` Re: AIO v2.5 Noah Misch <noah@leadboat.com>
2025-03-11 23:55 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-12 03:57 ` Re: AIO v2.5 Noah Misch <noah@leadboat.com>
2025-03-12 17:06 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-12 23:23 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-13 10:53 ` Re: AIO v2.5 Antonin Houska <ah@cybertec.at>
2025-03-14 19:43 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-17 20:00 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-18 20:12 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-21 01:58 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-25 01:18 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
@ 2025-03-26 18:31 ` Noah Misch <noah@leadboat.com>
2025-03-26 21:41 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
4 siblings, 1 reply; 153+ messages in thread
From: Noah Misch @ 2025-03-26 18:31 UTC (permalink / raw)
To: Andres Freund <andres@anarazel.de>; +Cc: Antonin Houska <ah@cybertec.at>; pgsql-hackers@postgresql.org, Thomas Munro <thomas.munro@gmail.com>; Heikki Linnakangas <hlinnaka@iki.fi>; Robert Haas <robertmhaas@gmail.com>; Jakub Wartak <jakub.wartak@enterprisedb.com>; Jelte Fennema-Nio <postgres@jeltef.nl>
I reviewed everything up to and including "[PATCH v2.12 17/28] aio, bufmgr:
Comment fixes", the last patch before write support.
postgr.es/m/20250326001915.bc.nmisch@google.com covered patches 1-9, and this
email covers patches 10-17. All remaining review comments are minor, so I've
marked the commitfest entry Ready for Committer. If there's anything you'd
like re-reviewed before you commit it, feel free to bring it to my attention.
Thanks for getting the feature to this stage!
On Mon, Mar 24, 2025 at 09:18:06PM -0400, Andres Freund wrote:
> Subject: [PATCH v2.12 10/28] aio: Basic read_stream adjustments for real AIO
> @@ -631,6 +637,9 @@ read_stream_begin_impl(int flags,
> * For now, max_ios = 0 is interpreted as max_ios = 1 with advice disabled
> * above. If we had real asynchronous I/O we might need a slightly
> * different definition.
> + *
> + * FIXME: Not sure what different definition we would need? I guess we
> + * could add the READ_BUFFERS_SYNCHRONOUSLY flag automatically?
I think we don't need a different definition. max_ios comes from
effective_io_concurrency and similar settings. The above comment's definition
of max_ios=0 matches that GUC's documented behavior:
The allowed range is
<literal>1</literal> to <literal>1000</literal>, or
<literal>0</literal> to disable issuance of asynchronous I/O requests.
I'll guess the comment meant that "advice disabled" is a no-op for AIO, so we
could reasonably argue to have effective_io_concurrency=0 distinguish itself
from effective_io_concurrency=1 in some different way for AIO. Equally,
there's no hurry to use that freedom to distinguish them.
> Subject: [PATCH v2.12 11/28] read_stream: Introduce and use optional batchmode
> support
> This patch adds an explicit flag (READ_STREAM_USE_BATCHING) to read_stream and
> uses it where appropriate.
I'd also use the new flag on the read_stream_begin_smgr_relation() call in
RelationCopyStorageUsingBuffer(). It uses block_range_read_stream_cb, and
other streams of that callback rightly use the flag.
> + * b) directly or indirectly start another batch pgaio_enter_batchmode()
Needs new wording from end of postgr.es/m/20250325155808.f7.nmisch@google.com
> Subject: [PATCH v2.12 12/28] docs: Reframe track_io_timing related docs as
> wait time
> Subject: [PATCH v2.12 13/28] Enable IO concurrency on all systems
Consider also updating this comment to stop focusing on prefetch; I think
changing that aligns with the patch's other changes:
/*
* How many buffers PrefetchBuffer callers should try to stay ahead of their
* ReadBuffer calls by. Zero means "never prefetch". This value is only used
* for buffers not belonging to tablespaces that have their
* effective_io_concurrency parameter set.
*/
int effective_io_concurrency = DEFAULT_EFFECTIVE_IO_CONCURRENCY;
> -#io_combine_limit = 128kB # usually 1-128 blocks (depends on OS)
> +#io_combine_limit = 128kB # usually 1-32 blocks (depends on OS)
I think "usually 1-128" remains right given:
GUC_UNIT_BLOCKS
#define MAX_IO_COMBINE_LIMIT PG_IOV_MAX
#define PG_IOV_MAX Min(IOV_MAX, 128)
> - On systems without prefetch advice support, attempting to configure
> - any value other than <literal>0</literal> will error out.
> + On systems with prefetch advice support,
> + <varname>effective_io_concurrency</varname> also controls the prefetch distance.
Wrap the last line.
> Subject: [PATCH v2.12 14/28] docs: Add acronym and glossary entries for I/O
> and AIO
> These could use a lot more polish.
To me, it's fine as-is.
> I did not actually reference the new entries yet, because I don't really
> understand what our policy for that is.
I haven't seen much of a policy on that.
> Subject: [PATCH v2.12 15/28] aio: Add pg_aios view
> +retry:
> +
> + /*
> + * There is no lock that could prevent the state of the IO to advance
> + * concurrently - and we don't want to introduce one, as that would
> + * introduce atomics into a very common path. Instead we
> + *
> + * 1) Determine the state + generation of the IO.
> + *
> + * 2) Copy the IO to local memory.
> + *
> + * 3) Check if state or generation of the IO changed. If the state
> + * changed, retry, if the generation changed don't display the IO.
> + */
> +
> + /* 1) from above */
> + start_generation = live_ioh->generation;
> + pg_read_barrier();
Based on the "really started after this function was called" and "no risk of a
livelock here" comments below, I think "retry:" should be here. We don't
want to livelock in the form of chasing ever-growing start_generation numbers.
> + /*
> + * The IO completed and a new one was started with the same ID. Don't
> + * display it - it really started after this function was called.
> + * There be a risk of a livelock if we just retried endlessly, if IOs
> + * complete very quickly.
> + */
> + if (live_ioh->generation != start_generation)
> + continue;
> +
> + /*
> + * The IOs state changed while we were "rendering" it. Just start from
s/IOs/IO's/
> + * scratch. There's no risk of a livelock here, as an IO has a limited
> + * sets of states it can be in, and state changes go only in a single
> + * direction.
> + */
> + if (live_ioh->state != start_state)
> + goto retry;
> + <entry role="catalog_table_entry"><para role="column_definition">
> + <structfield>target</structfield> <type>text</type>
> + </para>
> + <para>
> + What kind of object is the I/O targeting:
> + <itemizedlist spacing="compact">
> + <listitem>
> + <para>
> + <literal>smgr</literal>, I/O on postgres relations
s/postgres relations/relations/ since SGML docs don't use the term "postgres"
that way.
> Subject: [PATCH v2.12 16/28] aio: Add test_aio module
> --- a/src/test/modules/meson.build
> +++ b/src/test/modules/meson.build
> @@ -1,5 +1,6 @@
> # Copyright (c) 2022-2025, PostgreSQL Global Development Group
>
> +subdir('test_aio')
> subdir('brin')
List is alphabetized; please preserve that.
> +++ b/src/test/modules/test_aio/Makefile
> @@ -0,0 +1,26 @@
> +# src/test/modules/delay_execution/Makefile
Update filename in comment.
> +++ b/src/test/modules/test_aio/meson.build
> @@ -0,0 +1,37 @@
> +# Copyright (c) 2022-2024, PostgreSQL Global Development Group
s/2024/2025/
> --- /dev/null
> +++ b/src/test/modules/test_aio/t/001_aio.pl
s/ {4}/\t/g on this file. It's mostly \t now, with some exceptions.
> + test_inject_worker('worker', $node_worker);
What do we expect to happen if autovacuum or checkpointer runs one of these
injection points? I'm guessing it would at most make that process fail
without affecting the test outcome. If so, that's fine.
> + $waitfor,);
s/,//
> + # normal handle use
> + psql_like($io_method, $psql, "handle_get_release()",
> + qq(SELECT handle_get_release()),
> + qr/^$/, qr/^$/);
> +
> + # should error out, API violation
> + psql_like($io_method, $psql, "handle_get_twice()",
> + qq(SELECT handle_get_release()),
> + qr/^$/, qr/^$/);
Last two lines are a clone of the previous psql_like() call. I guess this
wants to instead call handle_get_twice() and check for some stderr.
> + "read_rel_block_ll() of $tblname page",
What does "_ll" stand for?
> + # Issue IO without waiting for completion, then exit
> + $psql_a->query_safe(
> + qq(SELECT read_rel_block_ll('tbl_ok', 1, wait_complete=>false);));
> + $psql_a->reconnect_and_clear();
> +
> + # Check that another backend can read the relevant block
> + psql_like(
> + $io_method,
> + $psql_b,
> + "completing read started by exited backend",
I think the exiting backend's pgaio_shutdown() completed it.
> +sub test_inject
This deserves a brief comment on the behaviors being tested, like the previous
functions have. It seems to be about short reads and hard failures like EIO.
> Subject: [PATCH v2.12 17/28] aio, bufmgr: Comment fixes
^ permalink raw reply [nested|flat] 153+ messages in thread
* Re: AIO v2.5
2025-03-04 19:00 AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-10 18:23 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-11 19:41 ` Re: AIO v2.5 Noah Misch <noah@leadboat.com>
2025-03-11 23:55 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-12 03:57 ` Re: AIO v2.5 Noah Misch <noah@leadboat.com>
2025-03-12 17:06 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-12 23:23 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-13 10:53 ` Re: AIO v2.5 Antonin Houska <ah@cybertec.at>
2025-03-14 19:43 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-17 20:00 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-18 20:12 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-21 01:58 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-25 01:18 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-26 18:31 ` Re: AIO v2.5 Noah Misch <noah@leadboat.com>
@ 2025-03-26 21:41 ` Andres Freund <andres@anarazel.de>
2025-03-26 21:52 ` Re: AIO v2.5 Thomas Munro <thomas.munro@gmail.com>
0 siblings, 1 reply; 153+ messages in thread
From: Andres Freund @ 2025-03-26 21:41 UTC (permalink / raw)
To: Noah Misch <noah@leadboat.com>; +Cc: Antonin Houska <ah@cybertec.at>; pgsql-hackers@postgresql.org, Thomas Munro <thomas.munro@gmail.com>; Heikki Linnakangas <hlinnaka@iki.fi>; Robert Haas <robertmhaas@gmail.com>; Jakub Wartak <jakub.wartak@enterprisedb.com>; Jelte Fennema-Nio <postgres@jeltef.nl>
Hi,
On 2025-03-26 11:31:02 -0700, Noah Misch wrote:
> I reviewed everything up to and including "[PATCH v2.12 17/28] aio, bufmgr:
> Comment fixes", the last patch before write support.
Thanks!
> postgr.es/m/20250326001915.bc.nmisch@google.com covered patches 1-9, and this
> email covers patches 10-17. All remaining review comments are minor, so I've
> marked the commitfest entry Ready for Committer. If there's anything you'd
> like re-reviewed before you commit it, feel free to bring it to my attention.
> Thanks for getting the feature to this stage!
As part of our discussion around the WARNING stuff I did make some changes,
it'd be good if you could look at those once I send them. While I squashed
the rest of the changes (addressing review comments) into their base commits,
I left the error-reporting related bits and pieces in fixup commits, to make
that easier.
> On Mon, Mar 24, 2025 at 09:18:06PM -0400, Andres Freund wrote:
> > Subject: [PATCH v2.12 10/28] aio: Basic read_stream adjustments for real AIO
>
> > @@ -631,6 +637,9 @@ read_stream_begin_impl(int flags,
> > * For now, max_ios = 0 is interpreted as max_ios = 1 with advice disabled
> > * above. If we had real asynchronous I/O we might need a slightly
> > * different definition.
> > + *
> > + * FIXME: Not sure what different definition we would need? I guess we
> > + * could add the READ_BUFFERS_SYNCHRONOUSLY flag automatically?
>
> I think we don't need a different definition. max_ios comes from
> effective_io_concurrency and similar settings. The above comment's definition
> of max_ios=0 matches that GUC's documented behavior:
>
> The allowed range is
> <literal>1</literal> to <literal>1000</literal>, or
> <literal>0</literal> to disable issuance of asynchronous I/O requests.
>
> I'll guess the comment meant that "advice disabled" is a no-op for AIO, so we
> could reasonably argue to have effective_io_concurrency=0 distinguish itself
> from effective_io_concurrency=1 in some different way for AIO. Equally,
> there's no hurry to use that freedom to distinguish them.
Thomas has since provided an implementation of what he was thinking of when
writing that comment:
https://postgr.es/m/CA%2BhUKG%2B8SC2%3DAD3bC0Pn85aMXm-PE2JSFGhC%3DMFVJvNQLObZeA%40mail.gmail.com
I squashed that into "aio: Basic read_stream adjustments for real AIO".
> > Subject: [PATCH v2.12 11/28] read_stream: Introduce and use optional batchmode
> > support
>
> > This patch adds an explicit flag (READ_STREAM_USE_BATCHING) to read_stream and
> > uses it where appropriate.
>
> I'd also use the new flag on the read_stream_begin_smgr_relation() call in
> RelationCopyStorageUsingBuffer(). It uses block_range_read_stream_cb, and
> other streams of that callback rightly use the flag.
Ah, yes. I had searched for all read_stream_begin_relation(), but not for
_smgr...
> > + * b) directly or indirectly start another batch pgaio_enter_batchmode()
>
> Needs new wording from end of postgr.es/m/20250325155808.f7.nmisch@google.com
Locally it's that, just need to send out a new version...
*
* b) start another batch (without first exiting batchmode and re-entering
* before returning)
> > Subject: [PATCH v2.12 13/28] Enable IO concurrency on all systems
>
> Consider also updating this comment to stop focusing on prefetch; I think
> changing that aligns with the patch's other changes:
>
> /*
> * How many buffers PrefetchBuffer callers should try to stay ahead of their
> * ReadBuffer calls by. Zero means "never prefetch". This value is only used
> * for buffers not belonging to tablespaces that have their
> * effective_io_concurrency parameter set.
> */
> int effective_io_concurrency = DEFAULT_EFFECTIVE_IO_CONCURRENCY;
Good point. Although I suspect it might be worth adjusting this, and also the
config.sgml bit about effective_io_concurrency separately. That seems like it
might take an iteration or two.
> > -#io_combine_limit = 128kB # usually 1-128 blocks (depends on OS)
> > +#io_combine_limit = 128kB # usually 1-32 blocks (depends on OS)
>
> I think "usually 1-128" remains right given:
> GUC_UNIT_BLOCKS
> #define MAX_IO_COMBINE_LIMIT PG_IOV_MAX
> #define PG_IOV_MAX Min(IOV_MAX, 128)
You're right. I think I got this wrong when rebasing over conflicts due to
06fb5612c97.
> > - On systems without prefetch advice support, attempting to configure
> > - any value other than <literal>0</literal> will error out.
> > + On systems with prefetch advice support,
> > + <varname>effective_io_concurrency</varname> also controls the prefetch distance.
>
> Wrap the last line.
Done.
> > Subject: [PATCH v2.12 14/28] docs: Add acronym and glossary entries for I/O
> > and AIO
>
> > These could use a lot more polish.
>
> To me, it's fine as-is.
Cool.
> > I did not actually reference the new entries yet, because I don't really
> > understand what our policy for that is.
>
> I haven't seen much of a policy on that.
That's sure what it looks like to me :/
>
> > Subject: [PATCH v2.12 15/28] aio: Add pg_aios view
> > +retry:
> > +
> > + /*
> > + * There is no lock that could prevent the state of the IO to advance
> > + * concurrently - and we don't want to introduce one, as that would
> > + * introduce atomics into a very common path. Instead we
> > + *
> > + * 1) Determine the state + generation of the IO.
> > + *
> > + * 2) Copy the IO to local memory.
> > + *
> > + * 3) Check if state or generation of the IO changed. If the state
> > + * changed, retry, if the generation changed don't display the IO.
> > + */
> > +
> > + /* 1) from above */
> > + start_generation = live_ioh->generation;
> > + pg_read_barrier();
>
> Based on the "really started after this function was called" and "no risk of a
> livelock here" comments below, I think "retry:" should be here. We don't
> want to livelock in the form of chasing ever-growing start_generation numbers.
You're right.
> > + * scratch. There's no risk of a livelock here, as an IO has a limited
> > + * sets of states it can be in, and state changes go only in a single
> > + * direction.
> > + */
> > + if (live_ioh->state != start_state)
> > + goto retry;
>
> > + <entry role="catalog_table_entry"><para role="column_definition">
> > + <structfield>target</structfield> <type>text</type>
> > + </para>
> > + <para>
> > + What kind of object is the I/O targeting:
> > + <itemizedlist spacing="compact">
> > + <listitem>
> > + <para>
> > + <literal>smgr</literal>, I/O on postgres relations
>
> s/postgres relations/relations/ since SGML docs don't use the term "postgres"
> that way.
Not sure what I was even trying to express with "postgres relations" vs plain
"relations" here...
>
> > Subject: [PATCH v2.12 16/28] aio: Add test_aio module
>
> > --- a/src/test/modules/meson.build
> > +++ b/src/test/modules/meson.build
> > @@ -1,5 +1,6 @@
> > # Copyright (c) 2022-2025, PostgreSQL Global Development Group
> >
> > +subdir('test_aio')
> > subdir('brin')
>
> List is alphabetized; please preserve that.
>
> > +++ b/src/test/modules/test_aio/Makefile
> > @@ -0,0 +1,26 @@
> > +# src/test/modules/delay_execution/Makefile
>
> Update filename in comment.
>
> > +++ b/src/test/modules/test_aio/meson.build
> > @@ -0,0 +1,37 @@
> > +# Copyright (c) 2022-2024, PostgreSQL Global Development Group
>
> s/2024/2025/
Done.
> > --- /dev/null
> > +++ b/src/test/modules/test_aio/t/001_aio.pl
>
> s/ {4}/\t/g on this file. It's mostly \t now, with some exceptions.
Huh. No idea how that happened.
> > + test_inject_worker('worker', $node_worker);
>
> What do we expect to happen if autovacuum or checkpointer runs one of these
> injection points? I'm guessing it would at most make that process fail
> without affecting the test outcome. If so, that's fine.
Autovacuum I disabled on the relations, to prevent that.
I think checkpointer should behave as you describe, although I could wonder if
it could confuse wait_for_log() based checks - but even so, I think that would
at worst lead to a test missing a bug, in extremely rare circumstances.
I tried triggering that condition, but it's pretty hard to hit, even after
lowering checkpoint_timeout to 1s and looping in the tests.
> > + # normal handle use
> > + psql_like($io_method, $psql, "handle_get_release()",
> > + qq(SELECT handle_get_release()),
> > + qr/^$/, qr/^$/);
> > +
> > + # should error out, API violation
> > + psql_like($io_method, $psql, "handle_get_twice()",
> > + qq(SELECT handle_get_release()),
> > + qr/^$/, qr/^$/);
>
> Last two lines are a clone of the previous psql_like() call. I guess this
> wants to instead call handle_get_twice() and check for some stderr.
Indeed.
> > + "read_rel_block_ll() of $tblname page",
>
> What does "_ll" stand for?
"low level". I added a C comment:
/*
* A "low level" read. This does similar things to what
* StartReadBuffers()/WaitReadBuffers() do, but provides more control (and
* less sanity).
*/
> > + # Issue IO without waiting for completion, then exit
> > + $psql_a->query_safe(
> > + qq(SELECT read_rel_block_ll('tbl_ok', 1, wait_complete=>false);));
> > + $psql_a->reconnect_and_clear();
> > +
> > + # Check that another backend can read the relevant block
> > + psql_like(
> > + $io_method,
> > + $psql_b,
> > + "completing read started by exited backend",
>
> I think the exiting backend's pgaio_shutdown() completed it.
I wrote the test precisely to exercise that path, otherwise it's pretty hard
to reach. It does seem to reach the path reasonably reliably, although it's
much harder to catch that causing problems, as the IO is typically too fast.
> > +sub test_inject
>
> This deserves a brief comment on the behaviors being tested, like the previous
> functions have. It seems to be about short reads and hard failures like EIO.
Done.
Greetings,
Andres Freund
^ permalink raw reply [nested|flat] 153+ messages in thread
* Re: AIO v2.5
2025-03-04 19:00 AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-10 18:23 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-11 19:41 ` Re: AIO v2.5 Noah Misch <noah@leadboat.com>
2025-03-11 23:55 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-12 03:57 ` Re: AIO v2.5 Noah Misch <noah@leadboat.com>
2025-03-12 17:06 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-12 23:23 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-13 10:53 ` Re: AIO v2.5 Antonin Houska <ah@cybertec.at>
2025-03-14 19:43 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-17 20:00 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-18 20:12 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-21 01:58 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-25 01:18 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-26 18:31 ` Re: AIO v2.5 Noah Misch <noah@leadboat.com>
2025-03-26 21:41 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
@ 2025-03-26 21:52 ` Thomas Munro <thomas.munro@gmail.com>
2025-03-30 23:13 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
0 siblings, 1 reply; 153+ messages in thread
From: Thomas Munro @ 2025-03-26 21:52 UTC (permalink / raw)
To: Andres Freund <andres@anarazel.de>; +Cc: Noah Misch <noah@leadboat.com>; Antonin Houska <ah@cybertec.at>; pgsql-hackers@postgresql.org, Heikki Linnakangas <hlinnaka@iki.fi>; Robert Haas <robertmhaas@gmail.com>; Jakub Wartak <jakub.wartak@enterprisedb.com>; Jelte Fennema-Nio <postgres@jeltef.nl>
On Thu, Mar 27, 2025 at 10:41 AM Andres Freund <andres@anarazel.de> wrote:
> > > Subject: [PATCH v2.12 13/28] Enable IO concurrency on all systems
> >
> > Consider also updating this comment to stop focusing on prefetch; I think
> > changing that aligns with the patch's other changes:
> >
> > /*
> > * How many buffers PrefetchBuffer callers should try to stay ahead of their
> > * ReadBuffer calls by. Zero means "never prefetch". This value is only used
> > * for buffers not belonging to tablespaces that have their
> > * effective_io_concurrency parameter set.
> > */
> > int effective_io_concurrency = DEFAULT_EFFECTIVE_IO_CONCURRENCY;
>
> Good point. Although I suspect it might be worth adjusting this, and also the
> config.sgml bit about effective_io_concurrency separately. That seems like it
> might take an iteration or two.
+1 for rewriting that separately from this work on the code (I can
have a crack at that if you want). For the comment, my suggestion
would be something like:
"Default limit on the level of concurrency that each I/O stream
(currently, ReadStream but in future other kinds of streams) can use.
Zero means that I/O is always performed synchronously, ie not
concurrently with query execution. This value can be overridden at the
tablespace level with the parameter of the same name. Note that
streams performing I/O not classified as single-session work respect
maintenance_io_concurrency instead."
^ permalink raw reply [nested|flat] 153+ messages in thread
* Re: AIO v2.5
2025-03-04 19:00 AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-10 18:23 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-11 19:41 ` Re: AIO v2.5 Noah Misch <noah@leadboat.com>
2025-03-11 23:55 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-12 03:57 ` Re: AIO v2.5 Noah Misch <noah@leadboat.com>
2025-03-12 17:06 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-12 23:23 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-13 10:53 ` Re: AIO v2.5 Antonin Houska <ah@cybertec.at>
2025-03-14 19:43 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-17 20:00 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-18 20:12 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-21 01:58 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-25 01:18 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-26 18:31 ` Re: AIO v2.5 Noah Misch <noah@leadboat.com>
2025-03-26 21:41 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-26 21:52 ` Re: AIO v2.5 Thomas Munro <thomas.munro@gmail.com>
@ 2025-03-30 23:13 ` Andres Freund <andres@anarazel.de>
0 siblings, 0 replies; 153+ messages in thread
From: Andres Freund @ 2025-03-30 23:13 UTC (permalink / raw)
To: Thomas Munro <thomas.munro@gmail.com>; +Cc: Noah Misch <noah@leadboat.com>; Antonin Houska <ah@cybertec.at>; pgsql-hackers@postgresql.org, Heikki Linnakangas <hlinnaka@iki.fi>; Robert Haas <robertmhaas@gmail.com>; Jakub Wartak <jakub.wartak@enterprisedb.com>; Jelte Fennema-Nio <postgres@jeltef.nl>
Hi,
On 2025-03-27 10:52:10 +1300, Thomas Munro wrote:
> On Thu, Mar 27, 2025 at 10:41 AM Andres Freund <andres@anarazel.de> wrote:
> > > > Subject: [PATCH v2.12 13/28] Enable IO concurrency on all systems
> > >
> > > Consider also updating this comment to stop focusing on prefetch; I think
> > > changing that aligns with the patch's other changes:
> > >
> > > /*
> > > * How many buffers PrefetchBuffer callers should try to stay ahead of their
> > > * ReadBuffer calls by. Zero means "never prefetch". This value is only used
> > > * for buffers not belonging to tablespaces that have their
> > > * effective_io_concurrency parameter set.
> > > */
> > > int effective_io_concurrency = DEFAULT_EFFECTIVE_IO_CONCURRENCY;
> >
> > Good point. Although I suspect it might be worth adjusting this, and also the
> > config.sgml bit about effective_io_concurrency separately. That seems like it
> > might take an iteration or two.
>
> +1 for rewriting that separately from this work on the code (I can
> have a crack at that if you want).
You taking a crack at that would be appreciated!
> For the comment, my suggestion would be something like:
>
> "Default limit on the level of concurrency that each I/O stream
> (currently, ReadStream but in future other kinds of streams) can use.
> Zero means that I/O is always performed synchronously, ie not
> concurrently with query execution. This value can be overridden at the
> tablespace level with the parameter of the same name. Note that
> streams performing I/O not classified as single-session work respect
> maintenance_io_concurrency instead."
Generally sounds good. I do wonder if the last sentence could be made a bit
simpler, it took me a few seconds to parse "not classified as single-session".
Maybe "classified as performing work for multiple sessions respect
maintenance_io_concurrency instead."?
Greetings,
Andres Freund
^ permalink raw reply [nested|flat] 153+ messages in thread
* Re: AIO v2.5
2025-03-04 19:00 AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-10 18:23 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-11 19:41 ` Re: AIO v2.5 Noah Misch <noah@leadboat.com>
2025-03-11 23:55 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-12 03:57 ` Re: AIO v2.5 Noah Misch <noah@leadboat.com>
2025-03-12 17:06 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-12 23:23 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-13 10:53 ` Re: AIO v2.5 Antonin Houska <ah@cybertec.at>
2025-03-14 19:43 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-17 20:00 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-18 20:12 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-21 01:58 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-25 01:18 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
@ 2025-03-26 21:20 ` Thom Brown <thom@linux.com>
2025-03-26 21:42 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
4 siblings, 1 reply; 153+ messages in thread
From: Thom Brown @ 2025-03-26 21:20 UTC (permalink / raw)
To: Andres Freund <andres@anarazel.de>; +Cc: Antonin Houska <ah@cybertec.at>; Noah Misch <noah@leadboat.com>; pgsql-hackers@postgresql.org, Thomas Munro <thomas.munro@gmail.com>; Heikki Linnakangas <hlinnaka@iki.fi>; Robert Haas <robertmhaas@gmail.com>; Jakub Wartak <jakub.wartak@enterprisedb.com>; Jelte Fennema-Nio <postgres@jeltef.nl>
On Tue, 25 Mar 2025 at 01:18, Andres Freund <andres@anarazel.de> wrote:
>
> Hi,
>
> Attached v2.12, with the following changes:
I took a quick gander through this just out of curiosity (yes, I know
I'm late), and found these show-stoppers:
v2.12-0015-aio-Add-pg_aios-view.patch:
+ <literal>ERROR</literal> mean the I/O failed with an error.
s/mean/means/
v2.12-0021-bufmgr-Implement-AIO-write-support.patch
+shared buffer lock still allows some modification, e.g., for hint bits(see
s/bits\(see/bits \(see)
+buffers that can be used as the source / target for IO. A bounce buffer be
s/be/can be/
Regards
Thom
^ permalink raw reply [nested|flat] 153+ messages in thread
* Re: AIO v2.5
2025-03-04 19:00 AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-10 18:23 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-11 19:41 ` Re: AIO v2.5 Noah Misch <noah@leadboat.com>
2025-03-11 23:55 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-12 03:57 ` Re: AIO v2.5 Noah Misch <noah@leadboat.com>
2025-03-12 17:06 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-12 23:23 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-13 10:53 ` Re: AIO v2.5 Antonin Houska <ah@cybertec.at>
2025-03-14 19:43 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-17 20:00 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-18 20:12 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-21 01:58 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-25 01:18 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-26 21:20 ` Re: AIO v2.5 Thom Brown <thom@linux.com>
@ 2025-03-26 21:42 ` Andres Freund <andres@anarazel.de>
0 siblings, 0 replies; 153+ messages in thread
From: Andres Freund @ 2025-03-26 21:42 UTC (permalink / raw)
To: Thom Brown <thom@linux.com>; +Cc: Antonin Houska <ah@cybertec.at>; Noah Misch <noah@leadboat.com>; pgsql-hackers@postgresql.org, Thomas Munro <thomas.munro@gmail.com>; Heikki Linnakangas <hlinnaka@iki.fi>; Robert Haas <robertmhaas@gmail.com>; Jakub Wartak <jakub.wartak@enterprisedb.com>; Jelte Fennema-Nio <postgres@jeltef.nl>
Hi,
On 2025-03-26 21:20:47 +0000, Thom Brown wrote:
> I took a quick gander through this just out of curiosity (yes, I know
> I'm late), and found these show-stoppers:
>
> v2.12-0015-aio-Add-pg_aios-view.patch:
>
> + <literal>ERROR</literal> mean the I/O failed with an error.
>
> s/mean/means/
>
>
> v2.12-0021-bufmgr-Implement-AIO-write-support.patch
>
> +shared buffer lock still allows some modification, e.g., for hint bits(see
>
> s/bits\(see/bits \(see)
>
> +buffers that can be used as the source / target for IO. A bounce buffer be
>
> s/be/can be/
Thanks! Squashed into my local tree.
Greetings,
Andres Freund
^ permalink raw reply [nested|flat] 153+ messages in thread
* Re: AIO v2.5
2025-03-04 19:00 AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-10 18:23 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-11 19:41 ` Re: AIO v2.5 Noah Misch <noah@leadboat.com>
2025-03-11 23:55 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-12 03:57 ` Re: AIO v2.5 Noah Misch <noah@leadboat.com>
2025-03-12 17:06 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-12 23:23 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-13 10:53 ` Re: AIO v2.5 Antonin Houska <ah@cybertec.at>
2025-03-14 19:43 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-17 20:00 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-18 20:12 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-21 01:58 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-25 01:18 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
@ 2025-03-27 01:07 ` Andres Freund <andres@anarazel.de>
2025-03-27 20:58 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
4 siblings, 1 reply; 153+ messages in thread
From: Andres Freund @ 2025-03-27 01:07 UTC (permalink / raw)
To: Noah Misch <noah@leadboat.com>; +Cc: pgsql-hackers@postgresql.org, Thomas Munro <thomas.munro@gmail.com>; Heikki Linnakangas <hlinnaka@iki.fi>; Robert Haas <robertmhaas@gmail.com>; Jakub Wartak <jakub.wartak@enterprisedb.com>; Jelte Fennema-Nio <postgres@jeltef.nl>; Antonin Houska <ah@cybertec.at>
Hi,
Attached v2.13, with the following changes:
Changes:
- Pushed a fair number of commits
A lot of thanks goes to Noah's detailed reviews!
- As Noah pointed out, the zero_damaged_pages warning could be emitted in an
io worker or another backend, but omitted in the backend that started the IO
To address that:
1) I added a new commit "aio: Add WARNING result status"
(itself trivial)
2) I changed buffer_readv_complete() to encode the warning/error in a more
detailed way than before (was_zeroed, first_invalid_off, count_invalid)
As part of that I put the encoding/decoding into a static inline
3) Tracking the number of invalid buffers was awkward with
buffer_readv_complete_one() returning a PgAioResult. Now it just
reports whether it found an invalid page with an out argument.
4) As discussed there now is a different error messages for the case of
multiple invalid pages
The code is a bit awkward to avoid code duplication, curious whether
that's seen as acceptable? I could just duplicate the entire ereport()
instead.
5) The WARNING in the callback is now a LOG, as it will be sent to the
client as a WARNING explicitly when the IO's results are processed
I actually chose LOG_SERVER_ONLY - that seemed slightly better than just
LOG? But not at all sure.
There's a comment explaining this now too.
Noah, I think this set of changes would benefit from another round of
review. I left these changes in "squash-later: " commits, to make it easier
to see / think about.
- Added a comment about the pgaio_result_report() in md_readv_complete(). I
changed it to LOG_SERVER_ONLY as well , but I'm not at all sure about that.
- Previously the buffer completion callback checked zero_damaged_pages - but
that's not right, the GUC hopefully is only set on a per-session basis
I solved that by having AsyncReadBuffers() add ZERO_ON_ERROR to the flags if
zero_damaged_pages is configured.
Also added a comment explaining that we probably should eventually use a
separate flag, so we can adjust the errcode etc differently.
- Explicit test for zero_damaged_pages and ZERO_ON_ERROR
As part of that I made read_rel_block_ll() support reading multiple
blocks. That makes it a lot easier to verify that we handle cases like a
4-block read where 2,3 are invalid correctly.
- I removed the code that "localbuf: Track pincount in BufferDesc as well"
added to ConditionalLockBufferForCleanup() and IsBufferCleanupOK() as discussed
Right now the situations that the code was worried don't exist yet, as we
only support reads.
I added a comment about not needing to worry about that yet to "bufmgr:
Implement AIO read support". And then changed that comment to a FIXME in the
write patches.
- Squashed Thomas' change to make io_concurrency=0 really not use AIO
- Lots of other review comments by Noah addressed
- Merged typo fixes by Thom Brown
TODO:
- There are more tests in test_aio that should be expanded to run for temp
tables as well, not just normal tables
- Add an explicit test for the checksum verification in the completion callback
There is an existing test for testing an invalid page due to page header
verification in test_aio, but not for checksum failures.
I think it's indirectly covered (e.g. in amcheck), but seems better to test
it explicitly.
- Add error context callbacks for io worker and "foreign" IO completion
Greetings,
Andres Freund
Attachments:
[text/x-diff] v2.13-0001-aio-Implement-support-for-reads-in-smgr-md-fd.patch (23.6K, ../../o2dtdrlc2xqpjycsfuvv7u6zyrqj7jfxqu32et3lyelkwce2jy@3k3fwdiiu73v/2-v2.13-0001-aio-Implement-support-for-reads-in-smgr-md-fd.patch)
download | inline diff:
From 592615eaae6d420127d6ab12f3023679337c3bce Mon Sep 17 00:00:00 2001
From: Andres Freund <andres@anarazel.de>
Date: Tue, 18 Mar 2025 14:40:05 -0400
Subject: [PATCH v2.13 01/28] aio: Implement support for reads in smgr/md/fd
This implements the following:
1) An smgr AIO target, for AIO on smgr files. This should be usable not just
for md.c but also other SMGR implementation if we ever get them.
2) readv support in fd.c, which requires a small bit of infrastructure work in
fd.c
3) smgr.c and md.c support for readv
There still is nothing performing AIO, but as of this commit it would be
possible.
As part of this change FileGetRawDesc() actually ensures that the file is
opened - previously it was basically not usable. It's used to reopen a file in
IO workers.
Reviewed-by: Noah Misch <noah@leadboat.com>
Discussion: https://postgr.es/m/uvrtrknj4kdytuboidbhwclo4gxhswwcpgadptsjvjqcluzmah%40brqs62irg4dt
Discussion: https://postgr.es/m/20210223100344.llw5an2aklengrmn@alap3.anarazel.de
Discussion: https://postgr.es/m/stj36ea6yyhoxtqkhpieia2z4krnam7qyetc57rfezgk4zgapf@gcnactj4z56m
---
src/include/storage/aio.h | 6 +-
src/include/storage/aio_types.h | 10 +-
src/include/storage/fd.h | 3 +
src/include/storage/md.h | 7 +
src/include/storage/smgr.h | 14 ++
src/backend/storage/aio/aio_callback.c | 3 +
src/backend/storage/aio/aio_target.c | 2 +
src/backend/storage/file/fd.c | 40 +++++
src/backend/storage/smgr/md.c | 196 +++++++++++++++++++++++++
src/backend/storage/smgr/smgr.c | 160 ++++++++++++++++++++
10 files changed, 437 insertions(+), 4 deletions(-)
diff --git a/src/include/storage/aio.h b/src/include/storage/aio.h
index cc987556e14..c386fc96d90 100644
--- a/src/include/storage/aio.h
+++ b/src/include/storage/aio.h
@@ -117,9 +117,10 @@ typedef enum PgAioTargetID
{
/* intentionally the zero value, to help catch zeroed memory etc */
PGAIO_TID_INVALID = 0,
+ PGAIO_TID_SMGR,
} PgAioTargetID;
-#define PGAIO_TID_COUNT (PGAIO_TID_INVALID + 1)
+#define PGAIO_TID_COUNT (PGAIO_TID_SMGR + 1)
/*
@@ -191,6 +192,9 @@ struct PgAioTargetInfo
typedef enum PgAioHandleCallbackID
{
PGAIO_HCB_INVALID,
+
+ PGAIO_HCB_MD_READV,
+ PGAIO_HCB_MD_WRITEV,
} PgAioHandleCallbackID;
diff --git a/src/include/storage/aio_types.h b/src/include/storage/aio_types.h
index dddda3a3e2f..debe8163d4e 100644
--- a/src/include/storage/aio_types.h
+++ b/src/include/storage/aio_types.h
@@ -60,11 +60,15 @@ typedef struct PgAioWaitRef
*/
typedef union PgAioTargetData
{
- /* just as an example placeholder for later */
struct
{
- uint32 queue_id;
- } wal;
+ RelFileLocator rlocator; /* physical relation identifier */
+ BlockNumber blockNum; /* blknum relative to begin of reln */
+ BlockNumber nblocks;
+ ForkNumber forkNum:8; /* don't waste 4 byte for four values */
+ bool is_temp:1; /* proc can be inferred by owning AIO */
+ bool skip_fsync:1;
+ } smgr;
} PgAioTargetData;
diff --git a/src/include/storage/fd.h b/src/include/storage/fd.h
index e3067ab6597..b77d8e5e30e 100644
--- a/src/include/storage/fd.h
+++ b/src/include/storage/fd.h
@@ -101,6 +101,8 @@ extern PGDLLIMPORT int max_safe_fds;
* prototypes for functions in fd.c
*/
+struct PgAioHandle;
+
/* Operations on virtual Files --- equivalent to Unix kernel file ops */
extern File PathNameOpenFile(const char *fileName, int fileFlags);
extern File PathNameOpenFilePerm(const char *fileName, int fileFlags, mode_t fileMode);
@@ -109,6 +111,7 @@ extern void FileClose(File file);
extern int FilePrefetch(File file, off_t offset, off_t amount, uint32 wait_event_info);
extern ssize_t FileReadV(File file, const struct iovec *iov, int iovcnt, off_t offset, uint32 wait_event_info);
extern ssize_t FileWriteV(File file, const struct iovec *iov, int iovcnt, off_t offset, uint32 wait_event_info);
+extern int FileStartReadV(struct PgAioHandle *ioh, File file, int iovcnt, off_t offset, uint32 wait_event_info);
extern int FileSync(File file, uint32 wait_event_info);
extern int FileZero(File file, off_t offset, off_t amount, uint32 wait_event_info);
extern int FileFallocate(File file, off_t offset, off_t amount, uint32 wait_event_info);
diff --git a/src/include/storage/md.h b/src/include/storage/md.h
index 05bf537066e..9d7131eff43 100644
--- a/src/include/storage/md.h
+++ b/src/include/storage/md.h
@@ -14,11 +14,14 @@
#ifndef MD_H
#define MD_H
+#include "storage/aio_types.h"
#include "storage/block.h"
#include "storage/relfilelocator.h"
#include "storage/smgr.h"
#include "storage/sync.h"
+extern const PgAioHandleCallbacks aio_md_readv_cb;
+
/* md storage manager functionality */
extern void mdinit(void);
extern void mdopen(SMgrRelation reln);
@@ -36,6 +39,9 @@ extern uint32 mdmaxcombine(SMgrRelation reln, ForkNumber forknum,
BlockNumber blocknum);
extern void mdreadv(SMgrRelation reln, ForkNumber forknum, BlockNumber blocknum,
void **buffers, BlockNumber nblocks);
+extern void mdstartreadv(PgAioHandle *ioh,
+ SMgrRelation reln, ForkNumber forknum, BlockNumber blocknum,
+ void **buffers, BlockNumber nblocks);
extern void mdwritev(SMgrRelation reln, ForkNumber forknum,
BlockNumber blocknum,
const void **buffers, BlockNumber nblocks, bool skipFsync);
@@ -46,6 +52,7 @@ extern void mdtruncate(SMgrRelation reln, ForkNumber forknum,
BlockNumber old_blocks, BlockNumber nblocks);
extern void mdimmedsync(SMgrRelation reln, ForkNumber forknum);
extern void mdregistersync(SMgrRelation reln, ForkNumber forknum);
+extern int mdfd(SMgrRelation reln, ForkNumber forknum, BlockNumber blocknum, uint32 *off);
extern void ForgetDatabaseSyncRequests(Oid dbid);
extern void DropRelationFiles(RelFileLocator *delrels, int ndelrels, bool isRedo);
diff --git a/src/include/storage/smgr.h b/src/include/storage/smgr.h
index 4016b206ad6..856ebcda350 100644
--- a/src/include/storage/smgr.h
+++ b/src/include/storage/smgr.h
@@ -15,6 +15,7 @@
#define SMGR_H
#include "lib/ilist.h"
+#include "storage/aio_types.h"
#include "storage/block.h"
#include "storage/relfilelocator.h"
@@ -73,6 +74,8 @@ typedef SMgrRelationData *SMgrRelation;
#define SmgrIsTemp(smgr) \
RelFileLocatorBackendIsTemp((smgr)->smgr_rlocator)
+extern const PgAioTargetInfo aio_smgr_target_info;
+
extern void smgrinit(void);
extern SMgrRelation smgropen(RelFileLocator rlocator, ProcNumber backend);
extern bool smgrexists(SMgrRelation reln, ForkNumber forknum);
@@ -97,6 +100,10 @@ extern uint32 smgrmaxcombine(SMgrRelation reln, ForkNumber forknum,
extern void smgrreadv(SMgrRelation reln, ForkNumber forknum,
BlockNumber blocknum,
void **buffers, BlockNumber nblocks);
+extern void smgrstartreadv(PgAioHandle *ioh,
+ SMgrRelation reln, ForkNumber forknum,
+ BlockNumber blocknum,
+ void **buffers, BlockNumber nblocks);
extern void smgrwritev(SMgrRelation reln, ForkNumber forknum,
BlockNumber blocknum,
const void **buffers, BlockNumber nblocks,
@@ -127,4 +134,11 @@ smgrwrite(SMgrRelation reln, ForkNumber forknum, BlockNumber blocknum,
smgrwritev(reln, forknum, blocknum, &buffer, 1, skipFsync);
}
+extern void pgaio_io_set_target_smgr(PgAioHandle *ioh,
+ SMgrRelationData *smgr,
+ ForkNumber forknum,
+ BlockNumber blocknum,
+ int nblocks,
+ bool skip_fsync);
+
#endif /* SMGR_H */
diff --git a/src/backend/storage/aio/aio_callback.c b/src/backend/storage/aio/aio_callback.c
index 413ea3247c2..53db6e194af 100644
--- a/src/backend/storage/aio/aio_callback.c
+++ b/src/backend/storage/aio/aio_callback.c
@@ -18,6 +18,7 @@
#include "miscadmin.h"
#include "storage/aio.h"
#include "storage/aio_internal.h"
+#include "storage/md.h"
/* just to have something to put into aio_handle_cbs */
@@ -37,6 +38,8 @@ typedef struct PgAioHandleCallbacksEntry
static const PgAioHandleCallbacksEntry aio_handle_cbs[] = {
#define CALLBACK_ENTRY(id, callback) [id] = {.cb = &callback, .name = #callback}
CALLBACK_ENTRY(PGAIO_HCB_INVALID, aio_invalid_cb),
+
+ CALLBACK_ENTRY(PGAIO_HCB_MD_READV, aio_md_readv_cb),
#undef CALLBACK_ENTRY
};
diff --git a/src/backend/storage/aio/aio_target.c b/src/backend/storage/aio/aio_target.c
index 536c7f91f5e..ac6c74f4ff2 100644
--- a/src/backend/storage/aio/aio_target.c
+++ b/src/backend/storage/aio/aio_target.c
@@ -16,6 +16,7 @@
#include "storage/aio.h"
#include "storage/aio_internal.h"
+#include "storage/smgr.h"
/*
@@ -25,6 +26,7 @@ static const PgAioTargetInfo *pgaio_target_info[] = {
[PGAIO_TID_INVALID] = &(PgAioTargetInfo) {
.name = "invalid",
},
+ [PGAIO_TID_SMGR] = &aio_smgr_target_info,
};
diff --git a/src/backend/storage/file/fd.c b/src/backend/storage/file/fd.c
index 0c3a2a756e7..9e1cdf7ddfe 100644
--- a/src/backend/storage/file/fd.c
+++ b/src/backend/storage/file/fd.c
@@ -94,6 +94,7 @@
#include "miscadmin.h"
#include "pgstat.h"
#include "postmaster/startup.h"
+#include "storage/aio.h"
#include "storage/fd.h"
#include "storage/ipc.h"
#include "utils/guc.h"
@@ -1296,6 +1297,8 @@ LruDelete(File file)
vfdP = &VfdCache[file];
+ pgaio_closing_fd(vfdP->fd);
+
/*
* Close the file. We aren't expecting this to fail; if it does, better
* to leak the FD than to mess up our internal state.
@@ -1989,6 +1992,8 @@ FileClose(File file)
if (!FileIsNotOpen(file))
{
+ pgaio_closing_fd(vfdP->fd);
+
/* close the file */
if (close(vfdP->fd) != 0)
{
@@ -2212,6 +2217,32 @@ retry:
return returnCode;
}
+int
+FileStartReadV(PgAioHandle *ioh, File file,
+ int iovcnt, off_t offset,
+ uint32 wait_event_info)
+{
+ int returnCode;
+ Vfd *vfdP;
+
+ Assert(FileIsValid(file));
+
+ DO_DB(elog(LOG, "FileStartReadV: %d (%s) " INT64_FORMAT " %d",
+ file, VfdCache[file].fileName,
+ (int64) offset,
+ iovcnt));
+
+ returnCode = FileAccess(file);
+ if (returnCode < 0)
+ return returnCode;
+
+ vfdP = &VfdCache[file];
+
+ pgaio_io_start_readv(ioh, vfdP->fd, iovcnt, offset);
+
+ return 0;
+}
+
ssize_t
FileWriteV(File file, const struct iovec *iov, int iovcnt, off_t offset,
uint32 wait_event_info)
@@ -2500,6 +2531,12 @@ FilePathName(File file)
int
FileGetRawDesc(File file)
{
+ int returnCode;
+
+ returnCode = FileAccess(file);
+ if (returnCode < 0)
+ return returnCode;
+
Assert(FileIsValid(file));
return VfdCache[file].fd;
}
@@ -2780,6 +2817,7 @@ FreeDesc(AllocateDesc *desc)
result = closedir(desc->desc.dir);
break;
case AllocateDescRawFD:
+ pgaio_closing_fd(desc->desc.fd);
result = close(desc->desc.fd);
break;
default:
@@ -2848,6 +2886,8 @@ CloseTransientFile(int fd)
/* Only get here if someone passes us a file not in allocatedDescs */
elog(WARNING, "fd passed to CloseTransientFile was not obtained from OpenTransientFile");
+ pgaio_closing_fd(fd);
+
return close(fd);
}
diff --git a/src/backend/storage/smgr/md.c b/src/backend/storage/smgr/md.c
index f3220f98dc4..eedba1e5794 100644
--- a/src/backend/storage/smgr/md.c
+++ b/src/backend/storage/smgr/md.c
@@ -31,6 +31,7 @@
#include "miscadmin.h"
#include "pg_trace.h"
#include "pgstat.h"
+#include "storage/aio.h"
#include "storage/bufmgr.h"
#include "storage/fd.h"
#include "storage/md.h"
@@ -152,6 +153,15 @@ static MdfdVec *_mdfd_getseg(SMgrRelation reln, ForkNumber forknum,
static BlockNumber _mdnblocks(SMgrRelation reln, ForkNumber forknum,
MdfdVec *seg);
+static PgAioResult md_readv_complete(PgAioHandle *ioh, PgAioResult prior_result, uint8 cb_data);
+static void md_readv_report(PgAioResult result, const PgAioTargetData *target_data, int elevel);
+
+const PgAioHandleCallbacks aio_md_readv_cb = {
+ .complete_shared = md_readv_complete,
+ .report = md_readv_report,
+};
+
+
static inline int
_mdfd_open_flags(void)
{
@@ -937,6 +947,69 @@ mdreadv(SMgrRelation reln, ForkNumber forknum, BlockNumber blocknum,
}
}
+/*
+ * mdstartreadv() -- Asynchronous version of mdreadv().
+ */
+void
+mdstartreadv(PgAioHandle *ioh,
+ SMgrRelation reln, ForkNumber forknum, BlockNumber blocknum,
+ void **buffers, BlockNumber nblocks)
+{
+ off_t seekpos;
+ MdfdVec *v;
+ BlockNumber nblocks_this_segment;
+ struct iovec *iov;
+ int iovcnt;
+ int ret;
+
+ v = _mdfd_getseg(reln, forknum, blocknum, false,
+ EXTENSION_FAIL | EXTENSION_CREATE_RECOVERY);
+
+ seekpos = (off_t) BLCKSZ * (blocknum % ((BlockNumber) RELSEG_SIZE));
+
+ Assert(seekpos < (off_t) BLCKSZ * RELSEG_SIZE);
+
+ nblocks_this_segment =
+ Min(nblocks,
+ RELSEG_SIZE - (blocknum % ((BlockNumber) RELSEG_SIZE)));
+
+ if (nblocks_this_segment != nblocks)
+ elog(ERROR, "read crossing segment boundary");
+
+ iovcnt = pgaio_io_get_iovec(ioh, &iov);
+
+ Assert(nblocks <= iovcnt);
+
+ iovcnt = buffers_to_iovec(iov, buffers, nblocks_this_segment);
+
+ Assert(iovcnt <= nblocks_this_segment);
+
+ if (!(io_direct_flags & IO_DIRECT_DATA))
+ pgaio_io_set_flag(ioh, PGAIO_HF_BUFFERED);
+
+ pgaio_io_set_target_smgr(ioh,
+ reln,
+ forknum,
+ blocknum,
+ nblocks,
+ false);
+ pgaio_io_register_callbacks(ioh, PGAIO_HCB_MD_READV, 0);
+
+ ret = FileStartReadV(ioh, v->mdfd_vfd, iovcnt, seekpos, WAIT_EVENT_DATA_FILE_READ);
+ if (ret != 0)
+ ereport(ERROR,
+ (errcode_for_file_access(),
+ errmsg("could not start reading blocks %u..%u in file \"%s\": %m",
+ blocknum,
+ blocknum + nblocks_this_segment - 1,
+ FilePathName(v->mdfd_vfd))));
+
+ /*
+ * The error checks corresponding to the post-read checks in mdread() are
+ * in md_readv_complete().
+ */
+}
+
/*
* mdwritev() -- Write the supplied blocks at the appropriate location.
*
@@ -1365,6 +1438,21 @@ mdimmedsync(SMgrRelation reln, ForkNumber forknum)
}
}
+int
+mdfd(SMgrRelation reln, ForkNumber forknum, BlockNumber blocknum, uint32 *off)
+{
+ MdfdVec *v = mdopenfork(reln, forknum, EXTENSION_FAIL);
+
+ v = _mdfd_getseg(reln, forknum, blocknum, false,
+ EXTENSION_FAIL);
+
+ *off = (off_t) BLCKSZ * (blocknum % ((BlockNumber) RELSEG_SIZE));
+
+ Assert(*off < (off_t) BLCKSZ * RELSEG_SIZE);
+
+ return FileGetRawDesc(v->mdfd_vfd);
+}
+
/*
* register_dirty_segment() -- Mark a relation segment as needing fsync
*
@@ -1841,3 +1929,111 @@ mdfiletagmatches(const FileTag *ftag, const FileTag *candidate)
*/
return ftag->rlocator.dbOid == candidate->rlocator.dbOid;
}
+
+/*
+ * AIO completion callback for mdstartreadv().
+ */
+static PgAioResult
+md_readv_complete(PgAioHandle *ioh, PgAioResult prior_result, uint8 cb_data)
+{
+ PgAioTargetData *td = pgaio_io_get_target_data(ioh);
+ PgAioResult result = prior_result;
+
+ if (prior_result.result < 0)
+ {
+ result.status = PGAIO_RS_ERROR;
+ result.id = PGAIO_HCB_MD_READV;
+ /* For "hard" errors, track the error number in error_data */
+ result.error_data = -prior_result.result;
+ result.result = 0;
+
+ /*
+ * Immediately log a message about the IO error, but only to the
+ * server log. The reason to do so immediately is that the originator
+ * might not process the query result immediately (because it is busy
+ * doing another part of query processing) or at all (e.g. if it was
+ * cancelled or errored out due to another IO also failing). The
+ * issuer of the IO will emit an ERROR when processing the IO's
+ * results
+ */
+ pgaio_result_report(result, td, LOG_SERVER_ONLY);
+
+ return result;
+ }
+
+ /*
+ * As explained above smgrstartreadv(), the smgr API operates on the level
+ * of blocks, rather than bytes. Convert.
+ */
+ result.result /= BLCKSZ;
+
+ Assert(result.result <= td->smgr.nblocks);
+
+ if (result.result == 0)
+ {
+ /* consider 0 blocks read a failure */
+ result.status = PGAIO_RS_ERROR;
+ result.id = PGAIO_HCB_MD_READV;
+ result.error_data = 0;
+
+ /* see comment above the "hard error" case */
+ pgaio_result_report(result, td, LOG_SERVER_ONLY);
+
+ return result;
+ }
+
+ if (result.status != PGAIO_RS_ERROR &&
+ result.result < td->smgr.nblocks)
+ {
+ /* partial reads should be retried at upper level */
+ result.status = PGAIO_RS_PARTIAL;
+ result.id = PGAIO_HCB_MD_READV;
+ }
+
+ return result;
+}
+
+/*
+ * AIO error reporting callback for mdstartreadv().
+ *
+ * Errors are encoded as follows:
+ * - PgAioResult.error_data != 0 encodes IO that failed with errno != 0
+ * - PgAioResult.error_data == 0 encodes IO that didn't read all data
+ */
+static void
+md_readv_report(PgAioResult result, const PgAioTargetData *td, int elevel)
+{
+ RelPathStr path;
+
+ path = relpathbackend(td->smgr.rlocator,
+ td->smgr.is_temp ? MyProcNumber : INVALID_PROC_NUMBER,
+ td->smgr.forkNum);
+
+ if (result.error_data != 0)
+ {
+ /* for errcode_for_file_access() and %m */
+ errno = result.error_data;
+
+ ereport(elevel,
+ errcode_for_file_access(),
+ errmsg("could not read blocks %u..%u in file \"%s\": %m",
+ td->smgr.blockNum,
+ td->smgr.blockNum + td->smgr.nblocks - 1,
+ path.str));
+ }
+ else
+ {
+ /*
+ * NB: This will typically only be output in debug messages, while
+ * retrying a partial IO.
+ */
+ ereport(elevel,
+ errcode(ERRCODE_DATA_CORRUPTED),
+ errmsg("could not read blocks %u..%u in file \"%s\": read only %zu of %zu bytes",
+ td->smgr.blockNum,
+ td->smgr.blockNum + td->smgr.nblocks - 1,
+ path.str,
+ result.result * (size_t) BLCKSZ,
+ td->smgr.nblocks * (size_t) BLCKSZ));
+ }
+}
diff --git a/src/backend/storage/smgr/smgr.c b/src/backend/storage/smgr/smgr.c
index af74f54b43b..bf8f57b410a 100644
--- a/src/backend/storage/smgr/smgr.c
+++ b/src/backend/storage/smgr/smgr.c
@@ -66,6 +66,7 @@
#include "access/xlogutils.h"
#include "lib/ilist.h"
#include "miscadmin.h"
+#include "storage/aio.h"
#include "storage/bufmgr.h"
#include "storage/ipc.h"
#include "storage/md.h"
@@ -106,6 +107,10 @@ typedef struct f_smgr
void (*smgr_readv) (SMgrRelation reln, ForkNumber forknum,
BlockNumber blocknum,
void **buffers, BlockNumber nblocks);
+ void (*smgr_startreadv) (PgAioHandle *ioh,
+ SMgrRelation reln, ForkNumber forknum,
+ BlockNumber blocknum,
+ void **buffers, BlockNumber nblocks);
void (*smgr_writev) (SMgrRelation reln, ForkNumber forknum,
BlockNumber blocknum,
const void **buffers, BlockNumber nblocks,
@@ -117,6 +122,7 @@ typedef struct f_smgr
BlockNumber old_blocks, BlockNumber nblocks);
void (*smgr_immedsync) (SMgrRelation reln, ForkNumber forknum);
void (*smgr_registersync) (SMgrRelation reln, ForkNumber forknum);
+ int (*smgr_fd) (SMgrRelation reln, ForkNumber forknum, BlockNumber blocknum, uint32 *off);
} f_smgr;
static const f_smgr smgrsw[] = {
@@ -134,12 +140,14 @@ static const f_smgr smgrsw[] = {
.smgr_prefetch = mdprefetch,
.smgr_maxcombine = mdmaxcombine,
.smgr_readv = mdreadv,
+ .smgr_startreadv = mdstartreadv,
.smgr_writev = mdwritev,
.smgr_writeback = mdwriteback,
.smgr_nblocks = mdnblocks,
.smgr_truncate = mdtruncate,
.smgr_immedsync = mdimmedsync,
.smgr_registersync = mdregistersync,
+ .smgr_fd = mdfd,
}
};
@@ -157,6 +165,16 @@ static dlist_head unpinned_relns;
static void smgrshutdown(int code, Datum arg);
static void smgrdestroy(SMgrRelation reln);
+static void smgr_aio_reopen(PgAioHandle *ioh);
+static char *smgr_aio_describe_identity(const PgAioTargetData *sd);
+
+
+const PgAioTargetInfo aio_smgr_target_info = {
+ .name = "smgr",
+ .reopen = smgr_aio_reopen,
+ .describe_identity = smgr_aio_describe_identity,
+};
+
/*
* smgrinit(), smgrshutdown() -- Initialize or shut down storage
@@ -709,6 +727,30 @@ smgrreadv(SMgrRelation reln, ForkNumber forknum, BlockNumber blocknum,
RESUME_INTERRUPTS();
}
+/*
+ * smgrstartreadv() -- asynchronous version of smgrreadv()
+ *
+ * This starts an asynchronous readv IO using the IO handle `ioh`. Other than
+ * `ioh` all parameters are the same as smgrreadv().
+ *
+ * Completion callbacks above smgr will be passed the result as the number of
+ * successfully read blocks if the read [partially] succeeds (Buffers for
+ * blocks not successfully read might bear unspecified modifications, up to
+ * the full nblocks). This maintains the abstraction that smgr operates on the
+ * level of blocks, rather than bytes.
+ */
+void
+smgrstartreadv(PgAioHandle *ioh,
+ SMgrRelation reln, ForkNumber forknum, BlockNumber blocknum,
+ void **buffers, BlockNumber nblocks)
+{
+ HOLD_INTERRUPTS();
+ smgrsw[reln->smgr_which].smgr_startreadv(ioh,
+ reln, forknum, blocknum, buffers,
+ nblocks);
+ RESUME_INTERRUPTS();
+}
+
/*
* smgrwritev() -- Write the supplied buffers out.
*
@@ -917,6 +959,29 @@ smgrimmedsync(SMgrRelation reln, ForkNumber forknum)
RESUME_INTERRUPTS();
}
+/*
+ * Return fd for the specified block number and update *off to the appropriate
+ * position.
+ *
+ * This is only to be used for when AIO needs to perform the IO in a different
+ * process than where it was issued (e.g. in an IO worker).
+ */
+static int
+smgrfd(SMgrRelation reln, ForkNumber forknum, BlockNumber blocknum, uint32 *off)
+{
+ int fd;
+
+ /*
+ * The caller needs to prevent interrupts from being processed, otherwise
+ * the FD could be closed prematurely.
+ */
+ Assert(!INTERRUPTS_CAN_BE_PROCESSED());
+
+ fd = smgrsw[reln->smgr_which].smgr_fd(reln, forknum, blocknum, off);
+
+ return fd;
+}
+
/*
* AtEOXact_SMgr
*
@@ -945,3 +1010,98 @@ ProcessBarrierSmgrRelease(void)
smgrreleaseall();
return true;
}
+
+/*
+ * Set target of the IO handle to be smgr and initialize all the relevant
+ * pieces of data.
+ */
+void
+pgaio_io_set_target_smgr(PgAioHandle *ioh,
+ SMgrRelationData *smgr,
+ ForkNumber forknum,
+ BlockNumber blocknum,
+ int nblocks,
+ bool skip_fsync)
+{
+ PgAioTargetData *sd = pgaio_io_get_target_data(ioh);
+
+ pgaio_io_set_target(ioh, PGAIO_TID_SMGR);
+
+ /* backend is implied via IO owner */
+ sd->smgr.rlocator = smgr->smgr_rlocator.locator;
+ sd->smgr.forkNum = forknum;
+ sd->smgr.blockNum = blocknum;
+ sd->smgr.nblocks = nblocks;
+ sd->smgr.is_temp = SmgrIsTemp(smgr);
+ /* Temp relations should never be fsync'd */
+ sd->smgr.skip_fsync = skip_fsync && !SmgrIsTemp(smgr);
+}
+
+/*
+ * Callback for smgr AIO targets, to reopen the file (e.g. because the IO is
+ * executed in a worker).
+ */
+static void
+smgr_aio_reopen(PgAioHandle *ioh)
+{
+ PgAioTargetData *sd = pgaio_io_get_target_data(ioh);
+ PgAioOpData *od = pgaio_io_get_op_data(ioh);
+ SMgrRelation reln;
+ ProcNumber procno;
+ uint32 off;
+
+ /*
+ * The caller needs to prevent interrupts from being processed, otherwise
+ * the FD could be closed again before we get to executing the IO.
+ */
+ Assert(!INTERRUPTS_CAN_BE_PROCESSED());
+
+ if (sd->smgr.is_temp)
+ procno = pgaio_io_get_owner(ioh);
+ else
+ procno = INVALID_PROC_NUMBER;
+
+ reln = smgropen(sd->smgr.rlocator, procno);
+ switch (pgaio_io_get_op(ioh))
+ {
+ case PGAIO_OP_INVALID:
+ pg_unreachable();
+ break;
+ case PGAIO_OP_READV:
+ od->read.fd = smgrfd(reln, sd->smgr.forkNum, sd->smgr.blockNum, &off);
+ Assert(off == od->read.offset);
+ break;
+ case PGAIO_OP_WRITEV:
+ od->write.fd = smgrfd(reln, sd->smgr.forkNum, sd->smgr.blockNum, &off);
+ Assert(off == od->write.offset);
+ break;
+ }
+}
+
+/*
+ * Callback for smgr AIO targets, to describe the target of the IO.
+ */
+static char *
+smgr_aio_describe_identity(const PgAioTargetData *sd)
+{
+ RelPathStr path;
+ char *desc;
+
+ path = relpathbackend(sd->smgr.rlocator,
+ sd->smgr.is_temp ? MyProcNumber : INVALID_PROC_NUMBER,
+ sd->smgr.forkNum);
+
+ if (sd->smgr.nblocks == 0)
+ desc = psprintf(_("file \"%s\""), path.str);
+ else if (sd->smgr.nblocks == 1)
+ desc = psprintf(_("block %u in file \"%s\""),
+ sd->smgr.blockNum,
+ path.str);
+ else
+ desc = psprintf(_("blocks %u..%u in file \"%s\""),
+ sd->smgr.blockNum,
+ sd->smgr.blockNum + sd->smgr.nblocks - 1,
+ path.str);
+
+ return desc;
+}
--
2.48.1.76.g4e746b1a31.dirty
[text/x-diff] v2.13-0002-docs-Add-acronym-and-glossary-entries-for-I-O-.patch (3.5K, ../../o2dtdrlc2xqpjycsfuvv7u6zyrqj7jfxqu32et3lyelkwce2jy@3k3fwdiiu73v/3-v2.13-0002-docs-Add-acronym-and-glossary-entries-for-I-O-.patch)
download | inline diff:
From b6f0959552a56bb598f91f8e44c313cbd938d908 Mon Sep 17 00:00:00 2001
From: Andres Freund <andres@anarazel.de>
Date: Fri, 21 Mar 2025 15:06:35 -0400
Subject: [PATCH v2.13 02/28] docs: Add acronym and glossary entries for I/O
and AIO
These are fairly basic, but better than nothing. While there are several
opportunities to link to these entries, this patch does not add any. They will
however be referenced by future patches.
Reviewed-by: Noah Misch <noah@leadboat.com>
Discussion: https://postgr.es/m/20250326183102.92.nmisch@google.com
---
doc/src/sgml/acronyms.sgml | 18 ++++++++++++++++++
doc/src/sgml/glossary.sgml | 39 ++++++++++++++++++++++++++++++++++++++
2 files changed, 57 insertions(+)
diff --git a/doc/src/sgml/acronyms.sgml b/doc/src/sgml/acronyms.sgml
index 58d0d90fece..2f906e9f018 100644
--- a/doc/src/sgml/acronyms.sgml
+++ b/doc/src/sgml/acronyms.sgml
@@ -9,6 +9,15 @@
<variablelist>
+ <varlistentry>
+ <term><acronym>AIO</acronym></term>
+ <listitem>
+ <para>
+ <link linkend="glossary-aio">Asynchronous <acronym>I/O</acronym></link>
+ </para>
+ </listitem>
+ </varlistentry>
+
<varlistentry>
<term><acronym>ACL</acronym></term>
<listitem>
@@ -354,6 +363,15 @@
</listitem>
</varlistentry>
+ <varlistentry>
+ <term><acronym>I/O</acronym></term>
+ <listitem>
+ <para>
+ <link linkend="glossary-io">Input/Output</link>
+ </para>
+ </listitem>
+ </varlistentry>
+
<varlistentry>
<term><acronym>ISO</acronym></term>
<listitem>
diff --git a/doc/src/sgml/glossary.sgml b/doc/src/sgml/glossary.sgml
index c0f812e3f5e..b88cac598e9 100644
--- a/doc/src/sgml/glossary.sgml
+++ b/doc/src/sgml/glossary.sgml
@@ -81,6 +81,31 @@
</glossdef>
</glossentry>
+ <glossentry id="glossary-aio">
+ <glossterm>Asynchronous <acronym>I/O</acronym></glossterm>
+ <acronym>AIO</acronym>
+ <indexterm>
+ <primary>Asynchronous <acronym>I/O</acronym></primary>
+ </indexterm>
+ <glossdef>
+ <para>
+ Asynchronous <acronym>I/O</acronym> (<acronym>AIO</acronym>) describes
+ performing <acronym>I/O</acronym> in a non-blocking way (asynchronously),
+ in contrast to synchronous <acronym>I/O</acronym>, which blocks for the
+ entire duration of the <acronym>I/O</acronym>.
+ </para>
+ <para>
+ With <acronym>AIO</acronym>, starting an <acronym>I/O</acronym> operation
+ is separated from waiting for the result of the operation, allowing
+ multiple <acronym>I/O</acronym> operations to be initiated concurrently,
+ as well as performing <acronym>CPU</acronym> heavy operations
+ concurrently with <acronym>I/O</acronym>. The price for that increased
+ concurrency is increased complexity.
+ </para>
+ <glossseealso otherterm="glossary-io" />
+ </glossdef>
+ </glossentry>
+
<glossentry id="glossary-atomic">
<glossterm>Atomic</glossterm>
<glossdef>
@@ -938,6 +963,20 @@
</glossdef>
</glossentry>
+ <glossentry id="glossary-io">
+ <glossterm>Input/Output</glossterm>
+ <acronym>I/O</acronym>
+ <glossdef>
+ <para>
+ Input/Output (<acronym>I/O</acronym>) describes the communication between
+ a program and peripheral devices. In the context of database systems,
+ <acronym>I/O</acronym> commonly, but not exclusively, refers to
+ interaction with storage devices or the network.
+ </para>
+ <glossseealso otherterm="glossary-aio" />
+ </glossdef>
+ </glossentry>
+
<glossentry id="glossary-insert">
<glossterm>Insert</glossterm>
<glossdef>
--
2.48.1.76.g4e746b1a31.dirty
[text/x-diff] v2.13-0003-aio-Add-pg_aios-view.patch (19.4K, ../../o2dtdrlc2xqpjycsfuvv7u6zyrqj7jfxqu32et3lyelkwce2jy@3k3fwdiiu73v/4-v2.13-0003-aio-Add-pg_aios-view.patch)
download | inline diff:
From 23d1b3b0a0a0a1629fc7a9fa6c432feee8139cf9 Mon Sep 17 00:00:00 2001
From: Andres Freund <andres@anarazel.de>
Date: Tue, 18 Mar 2025 14:40:06 -0400
Subject: [PATCH v2.13 03/28] aio: Add pg_aios view
The new view lists all IO handles that are currently in use and is mainly
useful for PG developers, but may also be useful when tuning PG.
FIXME:
- catversion bump before commit
Reviewed-by: Noah Misch <noah@leadboat.com>
Discussion: https://postgr.es/m/uvrtrknj4kdytuboidbhwclo4gxhswwcpgadptsjvjqcluzmah%40brqs62irg4dt
---
src/include/catalog/pg_proc.dat | 10 +
src/backend/catalog/system_views.sql | 3 +
src/backend/storage/aio/Makefile | 1 +
src/backend/storage/aio/aio_funcs.c | 225 +++++++++++++++++++++
src/backend/storage/aio/meson.build | 1 +
doc/src/sgml/system-views.sgml | 288 +++++++++++++++++++++++++++
src/test/regress/expected/rules.out | 16 ++
7 files changed, 544 insertions(+)
create mode 100644 src/backend/storage/aio/aio_funcs.c
diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat
index 8b68b16d79d..52dbce05a45 100644
--- a/src/include/catalog/pg_proc.dat
+++ b/src/include/catalog/pg_proc.dat
@@ -12493,4 +12493,14 @@
proargtypes => 'int4',
prosrc => 'gist_stratnum_common' },
+# AIO related functions
+{ oid => '9200', descr => 'information about in-progress asynchronous IOs',
+ proname => 'pg_get_aios', prorows => '100', proretset => 't',
+ provolatile => 'v', proparallel => 'r', prorettype => 'record', proargtypes => '',
+ proallargtypes => '{int4,int4,int8,text,text,int8,int8,text,int2,int4,text,text,bool,bool,bool}',
+ proargmodes => '{o,o,o,o,o,o,o,o,o,o,o,o,o,o,o}',
+ proargnames => '{pid,io_id,io_generation,state,operation,off,length,target,handle_data_len,raw_result,result,target_desc,f_sync,f_localmem,f_buffered}',
+ prosrc => 'pg_get_aios' },
+
+
]
diff --git a/src/backend/catalog/system_views.sql b/src/backend/catalog/system_views.sql
index 31d269b7ee0..37fd4bc2566 100644
--- a/src/backend/catalog/system_views.sql
+++ b/src/backend/catalog/system_views.sql
@@ -1391,3 +1391,6 @@ CREATE VIEW pg_stat_subscription_stats AS
CREATE VIEW pg_wait_events AS
SELECT * FROM pg_get_wait_events();
+
+CREATE VIEW pg_aios AS
+ SELECT * FROM pg_get_aios();
diff --git a/src/backend/storage/aio/Makefile b/src/backend/storage/aio/Makefile
index c06c50771e0..3f2469cc399 100644
--- a/src/backend/storage/aio/Makefile
+++ b/src/backend/storage/aio/Makefile
@@ -11,6 +11,7 @@ include $(top_builddir)/src/Makefile.global
OBJS = \
aio.o \
aio_callback.o \
+ aio_funcs.o \
aio_init.o \
aio_io.o \
aio_target.o \
diff --git a/src/backend/storage/aio/aio_funcs.c b/src/backend/storage/aio/aio_funcs.c
new file mode 100644
index 00000000000..765fb16aef4
--- /dev/null
+++ b/src/backend/storage/aio/aio_funcs.c
@@ -0,0 +1,225 @@
+/*-------------------------------------------------------------------------
+ *
+ * aio_funcs.c
+ * AIO - SQL interface for AIO
+ *
+ *
+ * Portions Copyright (c) 1996-2025, PostgreSQL Global Development Group
+ * Portions Copyright (c) 1994, Regents of the University of California
+ *
+ * IDENTIFICATION
+ * src/backend/storage/aio/aio_funcs.c
+ *
+ *-------------------------------------------------------------------------
+ */
+
+#include "postgres.h"
+
+#include "fmgr.h"
+#include "funcapi.h"
+#include "nodes/execnodes.h"
+#include "port/atomics.h"
+#include "storage/aio_internal.h"
+#include "storage/lock.h"
+#include "storage/proc.h"
+#include "storage/procnumber.h"
+#include "utils/builtins.h"
+#include "utils/fmgrprotos.h"
+#include "utils/tuplestore.h"
+
+
+/*
+ * Byte length of an iovec.
+ */
+static size_t
+iov_byte_length(const struct iovec *iov, int cnt)
+{
+ size_t len = 0;
+
+ for (int i = 0; i < cnt; i++)
+ {
+ len += iov[i].iov_len;
+ }
+
+ return len;
+}
+
+Datum
+pg_get_aios(PG_FUNCTION_ARGS)
+{
+ ReturnSetInfo *rsinfo = (ReturnSetInfo *) fcinfo->resultinfo;
+
+ InitMaterializedSRF(fcinfo, 0);
+
+#define PG_GET_AIOS_COLS 15
+
+ for (uint64 i = 0; i < pgaio_ctl->io_handle_count; i++)
+ {
+ PgAioHandle *live_ioh = &pgaio_ctl->io_handles[i];
+ uint32 ioh_id = pgaio_io_get_id(live_ioh);
+ Datum values[PG_GET_AIOS_COLS] = {0};
+ bool nulls[PG_GET_AIOS_COLS] = {0};
+ ProcNumber owner;
+ PGPROC *owner_proc;
+ int32 owner_pid;
+ PgAioHandleState start_state;
+ uint64 start_generation;
+ PgAioHandle ioh_copy;
+ struct iovec iov_copy[PG_IOV_MAX];
+
+
+ /*
+ * There is no lock that could prevent the state of the IO to advance
+ * concurrently - and we don't want to introduce one, as that would
+ * introduce atomics into a very common path. Instead we
+ *
+ * 1) Determine the state + generation of the IO.
+ *
+ * 2) Copy the IO to local memory.
+ *
+ * 3) Check if state or generation of the IO changed. If the state
+ * changed, retry, if the generation changed don't display the IO.
+ */
+
+ /* 1) from above */
+ start_generation = live_ioh->generation;
+ pg_read_barrier();
+ start_state = live_ioh->state;
+
+retry:
+ if (start_state == PGAIO_HS_IDLE)
+ continue;
+
+ /* 2) from above */
+ memcpy(&ioh_copy, live_ioh, sizeof(PgAioHandle));
+
+ /*
+ * Safe to copy even if no iovec is used - we always reserve the
+ * required space.
+ */
+ memcpy(&iov_copy, &pgaio_ctl->iovecs[ioh_copy.iovec_off],
+ PG_IOV_MAX * sizeof(struct iovec));
+
+ /*
+ * Copy information about owner before 3) below, if the process exited
+ * it'd have to wait for the IO to finish first, which we would detect
+ * in 3).
+ */
+ owner = ioh_copy.owner_procno;
+ owner_proc = GetPGProcByNumber(owner);
+ owner_pid = owner_proc->pid;
+
+ /* 3) from above */
+ pg_read_barrier();
+
+ /*
+ * The IO completed and a new one was started with the same ID. Don't
+ * display it - it really started after this function was called.
+ * There be a risk of a livelock if we just retried endlessly, if IOs
+ * complete very quickly.
+ */
+ if (live_ioh->generation != start_generation)
+ continue;
+
+ /*
+ * The IO's state changed while we were "rendering" it. Just start
+ * from scratch. There's no risk of a livelock here, as an IO has a
+ * limited sets of states it can be in, and state changes go only in a
+ * single direction.
+ */
+ if (live_ioh->state != start_state)
+ goto retry;
+
+ /*
+ * Now that we have copied the IO into local memory and checked that
+ * it's still in the same state, we are not allowed to access "live"
+ * memory anymore. To make it slightly easier to catch such cases, set
+ * the "live" pointers to NULL.
+ */
+ live_ioh = NULL;
+ owner_proc = NULL;
+
+
+ /* column: owning pid */
+ if (owner_pid != 0)
+ values[0] = Int32GetDatum(owner_pid);
+ else
+ nulls[0] = false;
+
+ /* column: IO's id */
+ values[1] = ioh_id;
+
+ /* column: IO's generation */
+ values[2] = Int64GetDatum(start_generation);
+
+ /* column: IO's state */
+ values[3] = CStringGetTextDatum(pgaio_io_get_state_name(&ioh_copy));
+
+ /*
+ * If the IO is in PGAIO_HS_HANDED_OUT state, none of the following
+ * fields are valid yet (or are in the process of being set).
+ * Therefore we don't want to display any other columns.
+ */
+ if (start_state == PGAIO_HS_HANDED_OUT)
+ {
+ memset(nulls + 4, 1, (lengthof(nulls) - 4) * sizeof(bool));
+ goto display;
+ }
+
+ /* column: IO's operation */
+ values[4] = CStringGetTextDatum(pgaio_io_get_op_name(&ioh_copy));
+
+ /* columns: details about the IO's operation (offset, length) */
+ switch (ioh_copy.op)
+ {
+ case PGAIO_OP_INVALID:
+ nulls[5] = true;
+ nulls[6] = true;
+ break;
+ case PGAIO_OP_READV:
+ values[5] = Int64GetDatum(ioh_copy.op_data.read.offset);
+ values[6] =
+ Int64GetDatum(iov_byte_length(iov_copy, ioh_copy.op_data.read.iov_length));
+ break;
+ case PGAIO_OP_WRITEV:
+ values[5] = Int64GetDatum(ioh_copy.op_data.write.offset);
+ values[6] =
+ Int64GetDatum(iov_byte_length(iov_copy, ioh_copy.op_data.write.iov_length));
+ break;
+ }
+
+ /* column: IO's target */
+ values[7] = CStringGetTextDatum(pgaio_io_get_target_name(&ioh_copy));
+
+ /* column: length of IO's data array */
+ values[8] = Int16GetDatum(ioh_copy.handle_data_len);
+
+ /* column: raw result (i.e. some form of syscall return value) */
+ if (start_state == PGAIO_HS_COMPLETED_IO
+ || start_state == PGAIO_HS_COMPLETED_SHARED
+ || start_state == PGAIO_HS_COMPLETED_LOCAL)
+ values[9] = Int32GetDatum(ioh_copy.result);
+ else
+ nulls[9] = true;
+
+ /*
+ * column: result in the higher level representation (unknown if not
+ * finished)
+ */
+ values[10] =
+ CStringGetTextDatum(pgaio_result_status_string(ioh_copy.distilled_result.status));
+
+ /* column: target description */
+ values[11] = CStringGetTextDatum(pgaio_io_get_target_description(&ioh_copy));
+
+ /* columns: one for each flag */
+ values[12] = BoolGetDatum(ioh_copy.flags & PGAIO_HF_SYNCHRONOUS);
+ values[13] = BoolGetDatum(ioh_copy.flags & PGAIO_HF_REFERENCES_LOCAL);
+ values[14] = BoolGetDatum(ioh_copy.flags & PGAIO_HF_BUFFERED);
+
+display:
+ tuplestore_putvalues(rsinfo->setResult, rsinfo->setDesc, values, nulls);
+ }
+
+ return (Datum) 0;
+}
diff --git a/src/backend/storage/aio/meson.build b/src/backend/storage/aio/meson.build
index 2f0f03d8071..da6df2d3654 100644
--- a/src/backend/storage/aio/meson.build
+++ b/src/backend/storage/aio/meson.build
@@ -3,6 +3,7 @@
backend_sources += files(
'aio.c',
'aio_callback.c',
+ 'aio_funcs.c',
'aio_init.c',
'aio_io.c',
'aio_target.c',
diff --git a/doc/src/sgml/system-views.sgml b/doc/src/sgml/system-views.sgml
index 3f5a306247e..5154512b16b 100644
--- a/doc/src/sgml/system-views.sgml
+++ b/doc/src/sgml/system-views.sgml
@@ -51,6 +51,11 @@
</thead>
<tbody>
+ <row>
+ <entry><link linkend="view-pg-aios"><structname>pg_aios</structname></link></entry>
+ <entry>In-use asynchronous IO handles</entry>
+ </row>
+
<row>
<entry><link linkend="view-pg-available-extensions"><structname>pg_available_extensions</structname></link></entry>
<entry>available extensions</entry>
@@ -231,6 +236,289 @@
</table>
</sect1>
+ <sect1 id="view-pg-aios">
+ <title><structname>pg_aios</structname></title>
+
+ <indexterm zone="view-pg-aios">
+ <primary>pg_aios</primary>
+ </indexterm>
+
+ <para>
+ The <structname>pg_aios</structname> view lists all <xref
+ linkend="glossary-aio"/> handles that are currently in-use. An I/O handle
+ is used to reference an I/O operation that is being prepared, executed or
+ is in the process of completing. <structname>pg_aios</structname> contains
+ one row for each I/O handle.
+ </para>
+
+ <para>
+ This view is mainly useful for developers of
+ <productname>PostgreSQL</productname>, but may also be useful when tuning
+ <productname>PostgreSQL</productname>.
+ </para>
+
+ <table>
+ <title><structname>pg_aios</structname> Columns</title>
+ <tgroup cols="1">
+ <thead>
+ <row>
+ <entry role="catalog_table_entry"><para role="column_definition">
+ Column Type
+ </para>
+ <para>
+ Description
+ </para></entry>
+ </row>
+ </thead>
+
+ <tbody>
+ <row>
+ <entry role="catalog_table_entry"><para role="column_definition">
+ <structfield>pid</structfield> <type>int4</type>
+ </para>
+ <para>
+ Process ID of the server process that is issuing this I/O.
+ </para></entry>
+ </row>
+
+ <row>
+ <entry role="catalog_table_entry"><para role="column_definition">
+ <structfield>io_id</structfield> <type>int4</type>
+ </para>
+ <para>
+ Identifier of the I/O handle. Handles are reused once the I/O
+ completed (or if the handle is released before I/O is started). On reuse
+ <link linkend="view-pg-aios-io-generation">
+ <structname>pg_aios</structname>.<structfield>io_generation</structfield>
+ </link>
+ is incremented.
+ </para></entry>
+ </row>
+
+ <row>
+ <entry role="catalog_table_entry" id="view-pg-aios-io-generation"><para role="column_definition">
+ <structfield>io_generation</structfield> <type>int8</type>
+ </para>
+ <para>
+ Generation of the I/O handle.
+ </para></entry>
+ </row>
+
+ <row>
+ <entry role="catalog_table_entry"><para role="column_definition">
+ <structfield>state</structfield> <type>text</type>
+ </para>
+ <para>
+ State of the I/O handle:
+ <itemizedlist>
+ <listitem>
+ <para>
+ <literal>HANDED_OUT</literal>, referenced by code but not yet used
+ </para>
+ </listitem>
+ <listitem>
+ <para>
+ <literal>DEFINED</literal>, information necessary for execution is known
+ </para>
+ </listitem>
+ <listitem>
+ <para>
+ <literal>STAGED</literal>, ready for execution
+ </para>
+ </listitem>
+ <listitem>
+ <para>
+ <literal>SUBMITTED</literal>, submitted for execution
+ </para>
+ </listitem>
+ <listitem>
+ <para>
+ <literal>COMPLETED_IO</literal>, finished, but result has not yet been processed
+ </para>
+ </listitem>
+ <listitem>
+ <para>
+ <literal>COMPLETED_SHARED</literal>, shared completion processing completed
+ </para>
+ </listitem>
+ <listitem>
+ <para>
+ <literal>COMPLETED_LOCAL</literal>, backend local completion processing completed
+ </para>
+ </listitem>
+ </itemizedlist>
+ </para></entry>
+ </row>
+
+ <row>
+ <entry role="catalog_table_entry"><para role="column_definition">
+ <structfield>operation</structfield> <type>text</type>
+ </para>
+ <para>
+ Operation performed using the I/O handle:
+ <itemizedlist>
+ <listitem>
+ <para>
+ <literal>invalid</literal>, not yet known
+ </para>
+ </listitem>
+ <listitem>
+ <para>
+ <literal>readv</literal>, a vectored read
+ </para>
+ </listitem>
+ <listitem>
+ <para>
+ <literal>writev</literal>, a vectored write
+ </para>
+ </listitem>
+ </itemizedlist>
+ </para></entry>
+ </row>
+
+ <row>
+ <entry role="catalog_table_entry"><para role="column_definition">
+ <structfield>off</structfield> <type>int8</type>
+ </para>
+ <para>
+ Offset of the I/O operation.
+ </para></entry>
+ </row>
+
+ <row>
+ <entry role="catalog_table_entry"><para role="column_definition">
+ <structfield>length</structfield> <type>int8</type>
+ </para>
+ <para>
+ Length of the I/O operation.
+ </para></entry>
+ </row>
+
+ <row>
+ <entry role="catalog_table_entry"><para role="column_definition">
+ <structfield>target</structfield> <type>text</type>
+ </para>
+ <para>
+ What kind of object is the I/O targeting:
+ <itemizedlist spacing="compact">
+ <listitem>
+ <para>
+ <literal>smgr</literal>, I/O on relations
+ </para>
+ </listitem>
+ </itemizedlist>
+ </para></entry>
+ </row>
+
+ <row>
+ <entry role="catalog_table_entry"><para role="column_definition">
+ <structfield>handle_data_len</structfield> <type>int2</type>
+ </para>
+ <para>
+ Length of the data associated with the I/O operation. For I/O to/from
+ <xref linkend="guc-shared-buffers"/> and <xref
+ linkend="guc-temp-buffers"/>, this indicates the number of buffers the
+ I/O is operating on.
+ </para></entry>
+ </row>
+
+ <row>
+ <entry role="catalog_table_entry"><para role="column_definition">
+ <structfield>raw_result</structfield> <type>int4</type>
+ </para>
+ <para>
+ Low-level result of the I/O operation, or NULL if the operation has not
+ yet completed.
+ </para></entry>
+ </row>
+
+ <row>
+ <entry role="catalog_table_entry"><para role="column_definition">
+ <structfield>result</structfield> <type>text</type>
+ </para>
+ <para>
+ High-level result of the I/O operation:
+ <itemizedlist>
+ <listitem>
+ <para>
+ <literal>UNKNOWN</literal> means that the result of the
+ operation is not yet known.
+ </para>
+ </listitem>
+ <listitem>
+ <para>
+ <literal>OK</literal> means the I/O completed successfully.
+ </para>
+ </listitem>
+ <listitem>
+ <para>
+ <literal>PARTIAL</literal> means that the I/O completed without
+ error, but did not process all data. Commonly callers will need to
+ retry and perform the remainder of the work in a separate I/O.
+ </para>
+ </listitem>
+ <listitem>
+ <para>
+ <literal>WARNING</literal> means that the I/O completed without
+ error, but that execution of the IO triggered a warning. E.g. when
+ encountering a corrupted buffer with <xref
+ linkend="guc-zero-damaged-pages"/> enabled.
+ </para>
+ </listitem>
+ <listitem>
+ <para>
+ <literal>ERROR</literal> means the I/O failed with an error.
+ </para>
+ </listitem>
+ </itemizedlist>
+ </para></entry>
+ </row>
+
+ <row>
+ <entry role="catalog_table_entry"><para role="column_definition">
+ <structfield>target_desc</structfield> <type>text</type>
+ </para>
+ <para>
+ Description of what the I/O operation is targeting.
+ </para></entry>
+ </row>
+
+ <row>
+ <entry role="catalog_table_entry"><para role="column_definition">
+ <structfield>f_sync</structfield> <type>bool</type>
+ </para>
+ <para>
+ Flag indicating whether the I/O is executed synchronously.
+ </para></entry>
+ </row>
+
+ <row>
+ <entry role="catalog_table_entry"><para role="column_definition">
+ <structfield>f_localmem</structfield> <type>bool</type>
+ </para>
+ <para>
+ Flag indicating whether the I/O references process local memory.
+ </para></entry>
+ </row>
+
+ <row>
+ <entry role="catalog_table_entry"><para role="column_definition">
+ <structfield>f_buffered</structfield> <type>bool</type>
+ </para>
+ <para>
+ Flag indicating whether the I/O is buffered I/O.
+ </para></entry>
+ </row>
+
+ </tbody>
+ </tgroup>
+ </table>
+
+ <para>
+ The <structname>pg_aios</structname> view is read-only.
+ </para>
+ </sect1>
+
<sect1 id="view-pg-available-extensions">
<title><structname>pg_available_extensions</structname></title>
diff --git a/src/test/regress/expected/rules.out b/src/test/regress/expected/rules.out
index 47478969135..d9533deb04e 100644
--- a/src/test/regress/expected/rules.out
+++ b/src/test/regress/expected/rules.out
@@ -1286,6 +1286,22 @@ drop table cchild;
SELECT viewname, definition FROM pg_views
WHERE schemaname = 'pg_catalog'
ORDER BY viewname;
+pg_aios| SELECT pid,
+ io_id,
+ io_generation,
+ state,
+ operation,
+ off,
+ length,
+ target,
+ handle_data_len,
+ raw_result,
+ result,
+ target_desc,
+ f_sync,
+ f_localmem,
+ f_buffered
+ FROM pg_get_aios() pg_get_aios(pid, io_id, io_generation, state, operation, off, length, target, handle_data_len, raw_result, result, target_desc, f_sync, f_localmem, f_buffered);
pg_available_extension_versions| SELECT e.name,
e.version,
(x.extname IS NOT NULL) AS installed,
--
2.48.1.76.g4e746b1a31.dirty
[text/x-diff] v2.13-0004-localbuf-Track-pincount-in-BufferDesc-as-well.patch (3.3K, ../../o2dtdrlc2xqpjycsfuvv7u6zyrqj7jfxqu32et3lyelkwce2jy@3k3fwdiiu73v/5-v2.13-0004-localbuf-Track-pincount-in-BufferDesc-as-well.patch)
download | inline diff:
From d93f53ebe1d73e4fce87cfbc73c92dcd186a9317 Mon Sep 17 00:00:00 2001
From: Andres Freund <andres@anarazel.de>
Date: Tue, 18 Mar 2025 14:40:06 -0400
Subject: [PATCH v2.13 04/28] localbuf: Track pincount in BufferDesc as well
For AIO on temporary table buffers the AIO subsystem needs to be able to
ensure a pin on a buffer while AIO is going on, even if the IO issuing query
errors out. Tracking the buffer in LocalRefCount does not work, as it would
cause CheckForLocalBufferLeaks() to assert out.
Instead, also track the refcount in BufferDesc.state, not just
LocalRefCount. This also makes local buffers behave a bit more akin to shared
buffers.
Note that we still don't need locking, AIO completion callbacks for local
buffers are executed in the issuing session (i.e. nobody else has access to
the BufferDesc).
Reviewed-by: Noah Misch <noah@leadboat.com>
Discussion: https://postgr.es/m/uvrtrknj4kdytuboidbhwclo4gxhswwcpgadptsjvjqcluzmah%40brqs62irg4dt
---
src/backend/storage/buffer/localbuf.c | 28 +++++++++++++++++++++++++--
1 file changed, 26 insertions(+), 2 deletions(-)
diff --git a/src/backend/storage/buffer/localbuf.c b/src/backend/storage/buffer/localbuf.c
index f172a5c7820..3a722321533 100644
--- a/src/backend/storage/buffer/localbuf.c
+++ b/src/backend/storage/buffer/localbuf.c
@@ -249,6 +249,13 @@ GetLocalVictimBuffer(void)
pg_atomic_unlocked_write_u32(&bufHdr->state, buf_state);
trycounter = NLocBuffer;
}
+ else if (BUF_STATE_GET_REFCOUNT(buf_state) > 0)
+ {
+ /*
+ * This can be reached if the backend initiated AIO for this
+ * buffer and then errored out.
+ */
+ }
else
{
/* Found a usable buffer */
@@ -570,7 +577,13 @@ InvalidateLocalBuffer(BufferDesc *bufHdr, bool check_unreferenced)
buf_state = pg_atomic_read_u32(&bufHdr->state);
- if (check_unreferenced && LocalRefCount[bufid] != 0)
+ /*
+ * We need to test not just LocalRefCount[bufid] but also the BufferDesc
+ * itself, as the latter is used to represent a pin by the AIO subsystem.
+ * This can happen if AIO is initiated and then the query errors out.
+ */
+ if (check_unreferenced &&
+ (LocalRefCount[bufid] != 0 || BUF_STATE_GET_REFCOUNT(buf_state) != 0))
elog(ERROR, "block %u of %s is still referenced (local %u)",
bufHdr->tag.blockNum,
relpathbackend(BufTagGetRelFileLocator(&bufHdr->tag),
@@ -744,12 +757,13 @@ PinLocalBuffer(BufferDesc *buf_hdr, bool adjust_usagecount)
if (LocalRefCount[bufid] == 0)
{
NLocalPinnedBuffers++;
+ buf_state += BUF_REFCOUNT_ONE;
if (adjust_usagecount &&
BUF_STATE_GET_USAGECOUNT(buf_state) < BM_MAX_USAGE_COUNT)
{
buf_state += BUF_USAGECOUNT_ONE;
- pg_atomic_unlocked_write_u32(&buf_hdr->state, buf_state);
}
+ pg_atomic_unlocked_write_u32(&buf_hdr->state, buf_state);
}
LocalRefCount[bufid]++;
ResourceOwnerRememberBuffer(CurrentResourceOwner,
@@ -775,7 +789,17 @@ UnpinLocalBufferNoOwner(Buffer buffer)
Assert(NLocalPinnedBuffers > 0);
if (--LocalRefCount[buffid] == 0)
+ {
+ BufferDesc *buf_hdr = GetLocalBufferDescriptor(buffid);
+ uint32 buf_state;
+
NLocalPinnedBuffers--;
+
+ buf_state = pg_atomic_read_u32(&buf_hdr->state);
+ Assert(BUF_STATE_GET_REFCOUNT(buf_state) > 0);
+ buf_state -= BUF_REFCOUNT_ONE;
+ pg_atomic_unlocked_write_u32(&buf_hdr->state, buf_state);
+ }
}
/*
--
2.48.1.76.g4e746b1a31.dirty
[text/x-diff] v2.13-0005-aio-bufmgr-Comment-fixes.patch (4.8K, ../../o2dtdrlc2xqpjycsfuvv7u6zyrqj7jfxqu32et3lyelkwce2jy@3k3fwdiiu73v/6-v2.13-0005-aio-bufmgr-Comment-fixes.patch)
download | inline diff:
From 52ab5147d35d4435ecf2449df81579932620c6d1 Mon Sep 17 00:00:00 2001
From: Andres Freund <andres@anarazel.de>
Date: Thu, 20 Mar 2025 11:32:10 -0400
Subject: [PATCH v2.13 05/28] aio, bufmgr: Comment fixes
Some of these comments have been wrong for a while (12f3867f5534), some I
recently introduced (da7226993fd, 55b454d0e14). This also updates a comment in
FlushBuffer(), which will be copied in a future commit.
These changes seem big enough to be worth doing in separate commits.
Suggested-by: Noah Misch <noah@leadboat.com>
Discussion: https://postgr.es/m/20250319212530.80.nmisch@google.com
---
src/include/storage/aio.h | 2 +-
src/include/storage/aio_internal.h | 22 +++++++++++++++++++++-
src/backend/postmaster/postmaster.c | 2 +-
src/backend/storage/buffer/bufmgr.c | 10 ++++------
4 files changed, 27 insertions(+), 9 deletions(-)
diff --git a/src/include/storage/aio.h b/src/include/storage/aio.h
index c386fc96d90..5c886e478a5 100644
--- a/src/include/storage/aio.h
+++ b/src/include/storage/aio.h
@@ -80,7 +80,7 @@ typedef enum PgAioHandleFlags
/*
* The IO operations supported by the AIO subsystem.
*
- * This could be in aio_internal.h, as it is not pubicly referenced, but
+ * This could be in aio_internal.h, as it is not publicly referenced, but
* PgAioOpData currently *does* need to be public, therefore keeping this
* public seems to make sense.
*/
diff --git a/src/include/storage/aio_internal.h b/src/include/storage/aio_internal.h
index fb0425ccbfc..7f18da2c856 100644
--- a/src/include/storage/aio_internal.h
+++ b/src/include/storage/aio_internal.h
@@ -34,6 +34,11 @@
* linearly through all states.
*
* State changes should all go through pgaio_io_update_state().
+ *
+ * Note that the externally visible functions to start IO
+ * (e.g. FileStartReadV(), via pgaio_io_start_readv()) move an IO from
+ * PGAIO_HS_HANDED_OUT to at least PGAIO_HS_STAGED and at most
+ * PGAIO_HS_COMPLETED_LOCAL (at which point the handle will be reused).
*/
typedef enum PgAioHandleState
{
@@ -285,6 +290,9 @@ typedef struct IoMethodOps
/*
* Start executing passed in IOs.
*
+ * Shall advance state to at least PGAIO_HS_SUBMITTED. (By the time this
+ * returns, other backends might have advanced the state further.)
+ *
* Will not be called if ->needs_synchronous_execution() returned true.
*
* num_staged_ios is <= PGAIO_SUBMIT_BATCH_SIZE.
@@ -293,12 +301,24 @@ typedef struct IoMethodOps
*/
int (*submit) (uint16 num_staged_ios, PgAioHandle **staged_ios);
- /*
+ /* ---
* Wait for the IO to complete. Optional.
*
+ * On return, state shall be on of
+ * - PGAIO_HS_COMPLETED_IO
+ * - PGAIO_HS_COMPLETED_SHARED
+ * - PGAIO_HS_COMPLETED_LOCAL
+ *
+ * The callback must not block if the handle is already in one of those
+ * states, or has been reused (see pgaio_io_was_recycled()). If, on
+ * return, the state is PGAIO_HS_COMPLETED_IO, state will reach
+ * PGAIO_HS_COMPLETED_SHARED without further intervention by the IO
+ * method.
+ *
* If not provided, it needs to be guaranteed that the IO method calls
* pgaio_io_process_completion() without further interaction by the
* issuing backend.
+ * ---
*/
void (*wait_one) (PgAioHandle *ioh,
uint64 ref_generation);
diff --git a/src/backend/postmaster/postmaster.c b/src/backend/postmaster/postmaster.c
index a0c37532d2f..c966c2e83af 100644
--- a/src/backend/postmaster/postmaster.c
+++ b/src/backend/postmaster/postmaster.c
@@ -4401,7 +4401,7 @@ maybe_adjust_io_workers(void)
++io_worker_count;
}
else
- break; /* XXX try again soon? */
+ break; /* try again next time */
}
/* Too many running? */
diff --git a/src/backend/storage/buffer/bufmgr.c b/src/backend/storage/buffer/bufmgr.c
index 323382dcfa8..bcc4635c8c2 100644
--- a/src/backend/storage/buffer/bufmgr.c
+++ b/src/backend/storage/buffer/bufmgr.c
@@ -3919,9 +3919,10 @@ FlushBuffer(BufferDesc *buf, SMgrRelation reln, IOObject io_object,
XLogFlush(recptr);
/*
- * Now it's safe to write buffer to disk. Note that no one else should
- * have been able to write it while we were busy with log flushing because
- * only one process at a time can set the BM_IO_IN_PROGRESS bit.
+ * Now it's safe to write the buffer to disk. Note that no one else should
+ * have been able to write it, while we were busy with log flushing,
+ * because we got the exclusive right to perform I/O by setting the
+ * BM_IO_IN_PROGRESS bit.
*/
bufBlock = BufHdrGetBlock(buf);
@@ -5489,9 +5490,6 @@ IsBufferCleanupOK(Buffer buffer)
/*
* Functions for buffer I/O handling
*
- * Note: We assume that nested buffer I/O never occurs.
- * i.e at most one BM_IO_IN_PROGRESS bit is set per proc.
- *
* Also note that these are used only for shared buffers, not local ones.
*/
--
2.48.1.76.g4e746b1a31.dirty
[text/x-diff] v2.13-0006-aio-Add-WARNING-result-status.patch (2.2K, ../../o2dtdrlc2xqpjycsfuvv7u6zyrqj7jfxqu32et3lyelkwce2jy@3k3fwdiiu73v/7-v2.13-0006-aio-Add-WARNING-result-status.patch)
download | inline diff:
From d10cc26904174ec6b85ae3a284218a24e1582885 Mon Sep 17 00:00:00 2001
From: Andres Freund <andres@anarazel.de>
Date: Wed, 26 Mar 2025 14:33:09 -0400
Subject: [PATCH v2.13 06/28] aio: Add WARNING result status
If an IO succeeds, but issues a warning, e.g. due to a page verification
failure with zero_damaged_pages, we want to issue that warning in the context
of the issuer of the IO, not the process that executes the completion (always
the case for worker).
It's already possible for a completion callback to report a custom error
message, we just didn't have a result status that allowed a user of AIO to
know that a warning should be emitted even though the IO request succeeded.
All that's needed for that is a dedicated PGAIO_RS_ value.
---
src/include/storage/aio_types.h | 9 +++++----
src/backend/storage/aio/aio.c | 2 ++
2 files changed, 7 insertions(+), 4 deletions(-)
diff --git a/src/include/storage/aio_types.h b/src/include/storage/aio_types.h
index debe8163d4e..9db5d776b61 100644
--- a/src/include/storage/aio_types.h
+++ b/src/include/storage/aio_types.h
@@ -79,8 +79,9 @@ typedef enum PgAioResultStatus
{
PGAIO_RS_UNKNOWN, /* not yet completed / uninitialized */
PGAIO_RS_OK,
- PGAIO_RS_PARTIAL, /* did not fully succeed, but no error */
- PGAIO_RS_ERROR,
+ PGAIO_RS_PARTIAL, /* did not fully succeed, no warning/error */
+ PGAIO_RS_WARNING, /* [partially] succeeded, with a warning */
+ PGAIO_RS_ERROR, /* failed entirely */
} PgAioResultStatus;
@@ -96,10 +97,10 @@ typedef struct PgAioResult
uint32 id:8;
/* of type PgAioResultStatus, see above */
- uint32 status:2;
+ uint32 status:3;
/* meaning defined by callback->error */
- uint32 error_data:22;
+ uint32 error_data:21;
int32 result;
} PgAioResult;
diff --git a/src/backend/storage/aio/aio.c b/src/backend/storage/aio/aio.c
index 91e76113412..e3ed087e8a2 100644
--- a/src/backend/storage/aio/aio.c
+++ b/src/backend/storage/aio/aio.c
@@ -839,6 +839,8 @@ pgaio_result_status_string(PgAioResultStatus rs)
return "UNKNOWN";
case PGAIO_RS_OK:
return "OK";
+ case PGAIO_RS_WARNING:
+ return "WARNING";
case PGAIO_RS_PARTIAL:
return "PARTIAL";
case PGAIO_RS_ERROR:
--
2.48.1.76.g4e746b1a31.dirty
[text/x-diff] v2.13-0007-bufmgr-Implement-AIO-read-support.patch (32.9K, ../../o2dtdrlc2xqpjycsfuvv7u6zyrqj7jfxqu32et3lyelkwce2jy@3k3fwdiiu73v/8-v2.13-0007-bufmgr-Implement-AIO-read-support.patch)
download | inline diff:
From 13b1e4b7bb93211cb366affda83929176741dddf Mon Sep 17 00:00:00 2001
From: Andres Freund <andres@anarazel.de>
Date: Tue, 18 Mar 2025 14:40:06 -0400
Subject: [PATCH v2.13 07/28] bufmgr: Implement AIO read support
This commit implements the infrastructure to perform asynchronous reads into
the buffer pool.
To do so, it:
- Adds readv AIO callbacks for shared and local buffers
It may be worth calling out that shared buffer completions may be run in a
different backend than where the IO started.
- Adds an AIO wait reference to BufferDesc, to allow backends to wait for
in-progress asynchronous IOs
- Adapts StartBufferIO(), WaitIO(), TerminateBufferIO(), and their localbuf.c
equivalents, to be able to deal with AIO
- Moves the code to handle BM_PIN_COUNT_WAITER into a helper function, as it
now also needs to be called on IO completion
As of this commit, nothing issues AIO on shared buffers. A future commit will
update StartReadBuffers() to do so.
Reviewed-by: Noah Misch <noah@leadboat.com>
Reviewed-by: Melanie Plageman <melanieplageman@gmail.com>
Reviewed-by: Nazir Bilal Yavuz <byavuz81@gmail.com>
Discussion: https://postgr.es/m/uvrtrknj4kdytuboidbhwclo4gxhswwcpgadptsjvjqcluzmah%40brqs62irg4dt
Discussion: https://postgr.es/m/20210223100344.llw5an2aklengrmn@alap3.anarazel.de
Discussion: https://postgr.es/m/stj36ea6yyhoxtqkhpieia2z4krnam7qyetc57rfezgk4zgapf@gcnactj4z56m
---
src/include/storage/aio.h | 4 +
src/include/storage/buf_internals.h | 7 +-
src/include/storage/bufmgr.h | 4 +
src/backend/storage/aio/aio_callback.c | 5 +
src/backend/storage/buffer/README | 9 +-
src/backend/storage/buffer/buf_init.c | 3 +
src/backend/storage/buffer/bufmgr.c | 519 ++++++++++++++++++++++---
src/backend/storage/buffer/localbuf.c | 62 ++-
8 files changed, 554 insertions(+), 59 deletions(-)
diff --git a/src/include/storage/aio.h b/src/include/storage/aio.h
index 5c886e478a5..b182c0c78ca 100644
--- a/src/include/storage/aio.h
+++ b/src/include/storage/aio.h
@@ -195,6 +195,10 @@ typedef enum PgAioHandleCallbackID
PGAIO_HCB_MD_READV,
PGAIO_HCB_MD_WRITEV,
+
+ PGAIO_HCB_SHARED_BUFFER_READV,
+
+ PGAIO_HCB_LOCAL_BUFFER_READV,
} PgAioHandleCallbackID;
diff --git a/src/include/storage/buf_internals.h b/src/include/storage/buf_internals.h
index 9327f60c44c..72b36a4af26 100644
--- a/src/include/storage/buf_internals.h
+++ b/src/include/storage/buf_internals.h
@@ -17,6 +17,7 @@
#include "pgstat.h"
#include "port/atomics.h"
+#include "storage/aio_types.h"
#include "storage/buf.h"
#include "storage/bufmgr.h"
#include "storage/condition_variable.h"
@@ -264,6 +265,8 @@ typedef struct BufferDesc
int wait_backend_pgprocno; /* backend of pin-count waiter */
int freeNext; /* link in freelist chain */
+
+ PgAioWaitRef io_wref; /* set iff AIO is in progress */
LWLock content_lock; /* to lock access to buffer contents */
} BufferDesc;
@@ -472,8 +475,8 @@ extern BlockNumber ExtendBufferedRelLocal(BufferManagerRelation bmr,
uint32 *extended_by);
extern void MarkLocalBufferDirty(Buffer buffer);
extern void TerminateLocalBufferIO(BufferDesc *bufHdr, bool clear_dirty,
- uint32 set_flag_bits);
-extern bool StartLocalBufferIO(BufferDesc *bufHdr, bool forInput);
+ uint32 set_flag_bits, bool release_aio);
+extern bool StartLocalBufferIO(BufferDesc *bufHdr, bool forInput, bool nowait);
extern void FlushLocalBuffer(BufferDesc *bufHdr, SMgrRelation reln);
extern void DropRelationLocalBuffers(RelFileLocator rlocator,
ForkNumber forkNum,
diff --git a/src/include/storage/bufmgr.h b/src/include/storage/bufmgr.h
index 538b890a51d..5137288c96e 100644
--- a/src/include/storage/bufmgr.h
+++ b/src/include/storage/bufmgr.h
@@ -15,6 +15,7 @@
#define BUFMGR_H
#include "port/pg_iovec.h"
+#include "storage/aio_types.h"
#include "storage/block.h"
#include "storage/buf.h"
#include "storage/bufpage.h"
@@ -170,6 +171,9 @@ extern PGDLLIMPORT int checkpoint_flush_after;
extern PGDLLIMPORT int backend_flush_after;
extern PGDLLIMPORT int bgwriter_flush_after;
+extern const PgAioHandleCallbacks aio_shared_buffer_readv_cb;
+extern const PgAioHandleCallbacks aio_local_buffer_readv_cb;
+
/* in buf_init.c */
extern PGDLLIMPORT char *BufferBlocks;
diff --git a/src/backend/storage/aio/aio_callback.c b/src/backend/storage/aio/aio_callback.c
index 53db6e194af..c0063d6950e 100644
--- a/src/backend/storage/aio/aio_callback.c
+++ b/src/backend/storage/aio/aio_callback.c
@@ -18,6 +18,7 @@
#include "miscadmin.h"
#include "storage/aio.h"
#include "storage/aio_internal.h"
+#include "storage/bufmgr.h"
#include "storage/md.h"
@@ -40,6 +41,10 @@ static const PgAioHandleCallbacksEntry aio_handle_cbs[] = {
CALLBACK_ENTRY(PGAIO_HCB_INVALID, aio_invalid_cb),
CALLBACK_ENTRY(PGAIO_HCB_MD_READV, aio_md_readv_cb),
+
+ CALLBACK_ENTRY(PGAIO_HCB_SHARED_BUFFER_READV, aio_shared_buffer_readv_cb),
+
+ CALLBACK_ENTRY(PGAIO_HCB_LOCAL_BUFFER_READV, aio_local_buffer_readv_cb),
#undef CALLBACK_ENTRY
};
diff --git a/src/backend/storage/buffer/README b/src/backend/storage/buffer/README
index 011af7aff3e..a182fcd660c 100644
--- a/src/backend/storage/buffer/README
+++ b/src/backend/storage/buffer/README
@@ -147,9 +147,12 @@ in the buffer. It is used per the rules above.
* The BM_IO_IN_PROGRESS flag acts as a kind of lock, used to wait for I/O on a
buffer to complete (and in releases before 14, it was accompanied by a
-per-buffer LWLock). The process doing a read or write sets the flag for the
-duration, and processes that need to wait for it to be cleared sleep on a
-condition variable.
+per-buffer LWLock). The process starting a read or write sets the flag. When
+the I/O is completed, be it by the process that initiated the I/O or by
+another process, the flag is removed and the Buffer's condition variable is
+signalled. Processes that need to wait for the I/O to complete can wait for
+asynchronous I/O by using BufferDesc->io_wref and for BM_IO_IN_PROGRESS to be
+unset by sleeping on the buffer's condition variable.
Normal Buffer Replacement Strategy
diff --git a/src/backend/storage/buffer/buf_init.c b/src/backend/storage/buffer/buf_init.c
index ed1f8e03190..ed1dc488a42 100644
--- a/src/backend/storage/buffer/buf_init.c
+++ b/src/backend/storage/buffer/buf_init.c
@@ -14,6 +14,7 @@
*/
#include "postgres.h"
+#include "storage/aio.h"
#include "storage/buf_internals.h"
#include "storage/bufmgr.h"
@@ -125,6 +126,8 @@ BufferManagerShmemInit(void)
buf->buf_id = i;
+ pgaio_wref_clear(&buf->io_wref);
+
/*
* Initially link all the buffers together as unused. Subsequent
* management of this list is done by freelist.c.
diff --git a/src/backend/storage/buffer/bufmgr.c b/src/backend/storage/buffer/bufmgr.c
index bcc4635c8c2..3cf8b0f98d2 100644
--- a/src/backend/storage/buffer/bufmgr.c
+++ b/src/backend/storage/buffer/bufmgr.c
@@ -48,6 +48,7 @@
#include "pg_trace.h"
#include "pgstat.h"
#include "postmaster/bgwriter.h"
+#include "storage/aio.h"
#include "storage/buf_internals.h"
#include "storage/bufmgr.h"
#include "storage/fd.h"
@@ -519,7 +520,8 @@ static int SyncOneBuffer(int buf_id, bool skip_recently_used,
static void WaitIO(BufferDesc *buf);
static bool StartBufferIO(BufferDesc *buf, bool forInput, bool nowait);
static void TerminateBufferIO(BufferDesc *buf, bool clear_dirty,
- uint32 set_flag_bits, bool forget_owner);
+ uint32 set_flag_bits, bool forget_owner,
+ bool release_aio);
static void AbortBufferIO(Buffer buffer);
static void shared_buffer_write_error_callback(void *arg);
static void local_buffer_write_error_callback(void *arg);
@@ -1041,7 +1043,7 @@ ZeroAndLockBuffer(Buffer buffer, ReadBufferMode mode, bool already_valid)
{
/* Simple case for non-shared buffers. */
bufHdr = GetLocalBufferDescriptor(-buffer - 1);
- need_to_zero = StartLocalBufferIO(bufHdr, true);
+ need_to_zero = StartLocalBufferIO(bufHdr, true, false);
}
else
{
@@ -1077,9 +1079,9 @@ ZeroAndLockBuffer(Buffer buffer, ReadBufferMode mode, bool already_valid)
/* Set BM_VALID, terminate IO, and wake up any waiters */
if (isLocalBuf)
- TerminateLocalBufferIO(bufHdr, false, BM_VALID);
+ TerminateLocalBufferIO(bufHdr, false, BM_VALID, false);
else
- TerminateBufferIO(bufHdr, false, BM_VALID, true);
+ TerminateBufferIO(bufHdr, false, BM_VALID, true, false);
}
else if (!isLocalBuf)
{
@@ -1454,7 +1456,8 @@ static inline bool
WaitReadBuffersCanStartIO(Buffer buffer, bool nowait)
{
if (BufferIsLocal(buffer))
- return StartLocalBufferIO(GetLocalBufferDescriptor(-buffer - 1), true);
+ return StartLocalBufferIO(GetLocalBufferDescriptor(-buffer - 1),
+ true, nowait);
else
return StartBufferIO(GetBufferDescriptor(buffer - 1), true, nowait);
}
@@ -1604,9 +1607,9 @@ WaitReadBuffers(ReadBuffersOperation *operation)
/* Set BM_VALID, terminate IO, and wake up any waiters */
if (persistence == RELPERSISTENCE_TEMP)
- TerminateLocalBufferIO(bufHdr, false, BM_VALID);
+ TerminateLocalBufferIO(bufHdr, false, BM_VALID, false);
else
- TerminateBufferIO(bufHdr, false, BM_VALID, true);
+ TerminateBufferIO(bufHdr, false, BM_VALID, true, false);
/* Report I/Os as completing individually. */
TRACE_POSTGRESQL_BUFFER_READ_DONE(forknum, io_first_block + j,
@@ -1868,13 +1871,14 @@ retry:
}
/*
- * We assume the only reason for it to be pinned is that someone else is
- * flushing the page out. Wait for them to finish. (This could be an
- * infinite loop if the refcount is messed up... it would be nice to time
- * out after awhile, but there seems no way to be sure how many loops may
- * be needed. Note that if the other guy has pinned the buffer but not
- * yet done StartBufferIO, WaitIO will fall through and we'll effectively
- * be busy-looping here.)
+ * We assume the reason for it to be pinned is that either we were
+ * asynchronously reading the page in before erroring out or someone else
+ * is flushing the page out. Wait for the IO to finish. (This could be
+ * an infinite loop if the refcount is messed up... it would be nice to
+ * time out after awhile, but there seems no way to be sure how many loops
+ * may be needed. Note that if the other guy has pinned the buffer but
+ * not yet done StartBufferIO, WaitIO will fall through and we'll
+ * effectively be busy-looping here.)
*/
if (BUF_STATE_GET_REFCOUNT(buf_state) != 0)
{
@@ -2514,7 +2518,7 @@ ExtendBufferedRelShared(BufferManagerRelation bmr,
if (lock)
LWLockAcquire(BufferDescriptorGetContentLock(buf_hdr), LW_EXCLUSIVE);
- TerminateBufferIO(buf_hdr, false, BM_VALID, true);
+ TerminateBufferIO(buf_hdr, false, BM_VALID, true, false);
}
pgBufferUsage.shared_blks_written += extend_by;
@@ -2860,6 +2864,44 @@ PinBuffer_Locked(BufferDesc *buf)
ResourceOwnerRememberBuffer(CurrentResourceOwner, b);
}
+/*
+ * Support for waking up another backend that is waiting for the cleanup lock
+ * to be released using BM_PIN_COUNT_WAITER.
+ *
+ * See LockBufferForCleanup().
+ *
+ * Expected to be called just after releasing a buffer pin (in a BufferDesc,
+ * not just reducing the backend-local pincount for the buffer).
+ */
+static void
+WakePinCountWaiter(BufferDesc *buf)
+{
+ /*
+ * Acquire the buffer header lock, re-check that there's a waiter. Another
+ * backend could have unpinned this buffer, and already woken up the
+ * waiter.
+ *
+ * There's no danger of the buffer being replaced after we unpinned it
+ * above, as it's pinned by the waiter. The waiter removes
+ * BM_PIN_COUNT_WAITER if it stops waiting for a reason other than this
+ * backend waking it up.
+ */
+ uint32 buf_state = LockBufHdr(buf);
+
+ if ((buf_state & BM_PIN_COUNT_WAITER) &&
+ BUF_STATE_GET_REFCOUNT(buf_state) == 1)
+ {
+ /* we just released the last pin other than the waiter's */
+ int wait_backend_pgprocno = buf->wait_backend_pgprocno;
+
+ buf_state &= ~BM_PIN_COUNT_WAITER;
+ UnlockBufHdr(buf, buf_state);
+ ProcSendSignal(wait_backend_pgprocno);
+ }
+ else
+ UnlockBufHdr(buf, buf_state);
+}
+
/*
* UnpinBuffer -- make buffer available for replacement.
*
@@ -2928,29 +2970,8 @@ UnpinBufferNoOwner(BufferDesc *buf)
/* Support LockBufferForCleanup() */
if (buf_state & BM_PIN_COUNT_WAITER)
- {
- /*
- * Acquire the buffer header lock, re-check that there's a waiter.
- * Another backend could have unpinned this buffer, and already
- * woken up the waiter. There's no danger of the buffer being
- * replaced after we unpinned it above, as it's pinned by the
- * waiter.
- */
- buf_state = LockBufHdr(buf);
+ WakePinCountWaiter(buf);
- if ((buf_state & BM_PIN_COUNT_WAITER) &&
- BUF_STATE_GET_REFCOUNT(buf_state) == 1)
- {
- /* we just released the last pin other than the waiter's */
- int wait_backend_pgprocno = buf->wait_backend_pgprocno;
-
- buf_state &= ~BM_PIN_COUNT_WAITER;
- UnlockBufHdr(buf, buf_state);
- ProcSendSignal(wait_backend_pgprocno);
- }
- else
- UnlockBufHdr(buf, buf_state);
- }
ForgetPrivateRefCountEntry(ref);
}
}
@@ -3971,7 +3992,7 @@ FlushBuffer(BufferDesc *buf, SMgrRelation reln, IOObject io_object,
* Mark the buffer as clean (unless BM_JUST_DIRTIED has become set) and
* end the BM_IO_IN_PROGRESS state.
*/
- TerminateBufferIO(buf, true, 0, true);
+ TerminateBufferIO(buf, true, 0, true, false);
TRACE_POSTGRESQL_BUFFER_FLUSH_DONE(BufTagGetForkNum(&buf->tag),
buf->tag.blockNum,
@@ -5241,6 +5262,12 @@ LockBufferForCleanup(Buffer buffer)
CheckBufferIsPinnedOnce(buffer);
+ /*
+ * We do not yet need to be worried about in-progress AIOs holding a pin,
+ * as we only support doing reads via AIO and this function can only be
+ * called once the buffer is valid (i.e. no read can be in flight).
+ */
+
/* Nobody else to wait for */
if (BufferIsLocal(buffer))
return;
@@ -5398,6 +5425,8 @@ ConditionalLockBufferForCleanup(Buffer buffer)
Assert(BufferIsValid(buffer));
+ /* see AIO related comment in LockBufferForCleanup() */
+
if (BufferIsLocal(buffer))
{
refcount = LocalRefCount[-buffer - 1];
@@ -5453,6 +5482,8 @@ IsBufferCleanupOK(Buffer buffer)
Assert(BufferIsValid(buffer));
+ /* see AIO related comment in LockBufferForCleanup() */
+
if (BufferIsLocal(buffer))
{
/* There should be exactly one pin */
@@ -5505,6 +5536,7 @@ WaitIO(BufferDesc *buf)
for (;;)
{
uint32 buf_state;
+ PgAioWaitRef iow;
/*
* It may not be necessary to acquire the spinlock to check the flag
@@ -5512,10 +5544,40 @@ WaitIO(BufferDesc *buf)
* play it safe.
*/
buf_state = LockBufHdr(buf);
+
+ /*
+ * Copy the wait reference while holding the spinlock. This protects
+ * against a concurrent TerminateBufferIO() in another backend from
+ * clearing the wref while it's being read.
+ */
+ iow = buf->io_wref;
UnlockBufHdr(buf, buf_state);
+ /* no IO in progress, we don't need to wait */
if (!(buf_state & BM_IO_IN_PROGRESS))
break;
+
+ /*
+ * The buffer has asynchronous IO in progress, wait for it to
+ * complete.
+ */
+ if (pgaio_wref_valid(&iow))
+ {
+ pgaio_wref_wait(&iow);
+
+ /*
+ * The AIO subsystem internally uses condition variables and thus
+ * might remove this backend from the BufferDesc's CV. While that
+ * wouldn't cause a correctness issue (the first CV sleep just
+ * immediately returns if not already registered), it seems worth
+ * avoiding unnecessary loop iterations, given that we take care
+ * to do so at the start of the function.
+ */
+ ConditionVariablePrepareToSleep(cv);
+ continue;
+ }
+
+ /* wait on BufferDesc->cv, e.g. for concurrent synchronous IO */
ConditionVariableSleep(cv, WAIT_EVENT_BUFFER_IO);
}
ConditionVariableCancelSleep();
@@ -5524,13 +5586,12 @@ WaitIO(BufferDesc *buf)
/*
* StartBufferIO: begin I/O on this buffer
* (Assumptions)
- * My process is executing no IO
+ * My process is executing no IO on this buffer
* The buffer is Pinned
*
- * In some scenarios there are race conditions in which multiple backends
- * could attempt the same I/O operation concurrently. If someone else
- * has already started I/O on this buffer then we will block on the
- * I/O condition variable until he's done.
+ * In some scenarios multiple backends could attempt the same I/O operation
+ * concurrently. If someone else has already started I/O on this buffer then
+ * we will wait for completion of the IO using WaitIO().
*
* Input operations are only attempted on buffers that are not BM_VALID,
* and output operations only on buffers that are BM_VALID and BM_DIRTY,
@@ -5566,9 +5627,9 @@ StartBufferIO(BufferDesc *buf, bool forInput, bool nowait)
/* Once we get here, there is definitely no I/O active on this buffer */
+ /* Check if someone else already did the I/O */
if (forInput ? (buf_state & BM_VALID) : !(buf_state & BM_DIRTY))
{
- /* someone else already did the I/O */
UnlockBufHdr(buf, buf_state);
return false;
}
@@ -5604,7 +5665,7 @@ StartBufferIO(BufferDesc *buf, bool forInput, bool nowait)
*/
static void
TerminateBufferIO(BufferDesc *buf, bool clear_dirty, uint32 set_flag_bits,
- bool forget_owner)
+ bool forget_owner, bool release_aio)
{
uint32 buf_state;
@@ -5619,6 +5680,14 @@ TerminateBufferIO(BufferDesc *buf, bool clear_dirty, uint32 set_flag_bits,
if (clear_dirty && !(buf_state & BM_JUST_DIRTIED))
buf_state &= ~(BM_DIRTY | BM_CHECKPOINT_NEEDED);
+ if (release_aio)
+ {
+ /* release ownership by the AIO subsystem */
+ Assert(BUF_STATE_GET_REFCOUNT(buf_state) > 0);
+ buf_state -= BUF_REFCOUNT_ONE;
+ pgaio_wref_clear(&buf->io_wref);
+ }
+
buf_state |= set_flag_bits;
UnlockBufHdr(buf, buf_state);
@@ -5627,6 +5696,17 @@ TerminateBufferIO(BufferDesc *buf, bool clear_dirty, uint32 set_flag_bits,
BufferDescriptorGetBuffer(buf));
ConditionVariableBroadcast(BufferDescriptorGetIOCV(buf));
+
+ /*
+ * Support LockBufferForCleanup()
+ *
+ * We may have just released the last pin other than the waiter's. In most
+ * cases, this backend holds another pin on the buffer. But, if, for
+ * example, this backend is completing an IO issued by another backend, it
+ * may be time to wake the waiter.
+ */
+ if (release_aio && (buf_state & BM_PIN_COUNT_WAITER))
+ WakePinCountWaiter(buf);
}
/*
@@ -5675,7 +5755,7 @@ AbortBufferIO(Buffer buffer)
}
}
- TerminateBufferIO(buf_hdr, false, BM_IO_ERROR, false);
+ TerminateBufferIO(buf_hdr, false, BM_IO_ERROR, false, false);
}
/*
@@ -6126,3 +6206,350 @@ EvictUnpinnedBuffer(Buffer buf)
return result;
}
+
+/*
+ * Generic implementation of the AIO handle staging callback for readv/writev
+ * on local/shared buffers.
+ *
+ * Each readv/writev can target multiple buffers. The buffers have already
+ * been registered with the IO handle.
+ *
+ * To make the IO ready for execution ("staging"), we need to ensure that the
+ * targeted buffers are in an appropriate state while the IO is ongoing. For
+ * that the AIO subsystem needs to have its own buffer pin, otherwise an error
+ * in this backend could lead to this backend's buffer pin being released as
+ * part of error handling, which in turn could lead to the buffer being
+ * replaced while IO is ongoing.
+ */
+static pg_attribute_always_inline void
+buffer_stage_common(PgAioHandle *ioh, bool is_write, bool is_temp)
+{
+ uint64 *io_data;
+ uint8 handle_data_len;
+ PgAioWaitRef io_ref;
+ BufferTag first PG_USED_FOR_ASSERTS_ONLY = {0};
+
+ io_data = pgaio_io_get_handle_data(ioh, &handle_data_len);
+
+ pgaio_io_get_wref(ioh, &io_ref);
+
+ /* iterate over all buffers affected by the vectored readv/writev */
+ for (int i = 0; i < handle_data_len; i++)
+ {
+ Buffer buffer = (Buffer) io_data[i];
+ BufferDesc *buf_hdr = is_temp ?
+ GetLocalBufferDescriptor(-buffer - 1)
+ : GetBufferDescriptor(buffer - 1);
+ uint32 buf_state;
+
+ /*
+ * Check that all the buffers are actually ones that could conceivably
+ * be done in one IO, i.e. are sequential. This is the last
+ * buffer-aware code before IO is actually executed and confusion
+ * about which buffers are targeted by IO can be hard to debug, making
+ * it worth doing extra-paranoid checks.
+ */
+ if (i == 0)
+ first = buf_hdr->tag;
+ else
+ {
+ Assert(buf_hdr->tag.relNumber == first.relNumber);
+ Assert(buf_hdr->tag.blockNum == first.blockNum + i);
+ }
+
+ if (is_temp)
+ buf_state = pg_atomic_read_u32(&buf_hdr->state);
+ else
+ buf_state = LockBufHdr(buf_hdr);
+
+ /* verify the buffer is in the expected state */
+ Assert(buf_state & BM_TAG_VALID);
+ if (is_write)
+ {
+ Assert(buf_state & BM_VALID);
+ Assert(buf_state & BM_DIRTY);
+ }
+ else
+ {
+ Assert(!(buf_state & BM_VALID));
+ Assert(!(buf_state & BM_DIRTY));
+ }
+
+ /* temp buffers don't use BM_IO_IN_PROGRESS */
+ if (!is_temp)
+ Assert(buf_state & BM_IO_IN_PROGRESS);
+
+ Assert(BUF_STATE_GET_REFCOUNT(buf_state) >= 1);
+
+ /*
+ * Reflect that the buffer is now owned by the AIO subsystem.
+ *
+ * For local buffers: This can't be done just via LocalRefCount, as
+ * one might initially think, as this backend could error out while
+ * AIO is still in progress, releasing all the pins by the backend
+ * itself.
+ *
+ * This pin is released again in TerminateBufferIO().
+ */
+ buf_state += BUF_REFCOUNT_ONE;
+ buf_hdr->io_wref = io_ref;
+
+ if (is_temp)
+ pg_atomic_unlocked_write_u32(&buf_hdr->state, buf_state);
+ else
+ UnlockBufHdr(buf_hdr, buf_state);
+
+ /*
+ * Ensure the content lock that prevents buffer modifications while
+ * the buffer is being written out is not released early due to an
+ * error.
+ */
+ if (is_write && !is_temp)
+ {
+ LWLock *content_lock;
+
+ content_lock = BufferDescriptorGetContentLock(buf_hdr);
+
+ Assert(LWLockHeldByMe(content_lock));
+
+ /*
+ * Lock is now owned by AIO subsystem.
+ */
+ LWLockDisown(content_lock);
+ }
+
+ /*
+ * Stop tracking this buffer via the resowner - the AIO system now
+ * keeps track.
+ */
+ if (!is_temp)
+ ResourceOwnerForgetBufferIO(CurrentResourceOwner, buffer);
+ }
+}
+
+/*
+ * Helper for AIO readv completion callbacks, supporting both shared and temp
+ * buffers. Gets called once for each buffer in a multi-page read.
+ */
+static pg_attribute_always_inline PgAioResult
+buffer_readv_complete_one(uint8 buf_off, Buffer buffer, uint8 flags,
+ bool failed, bool is_temp)
+{
+ BufferDesc *buf_hdr = is_temp ?
+ GetLocalBufferDescriptor(-buffer - 1)
+ : GetBufferDescriptor(buffer - 1);
+ BufferTag tag = buf_hdr->tag;
+ char *bufdata = BufferGetBlock(buffer);
+ PgAioResult result = {.status = PGAIO_RS_OK};
+ uint32 set_flag_bits;
+
+ /* check that the buffer is in the expected state for a read */
+#ifdef USE_ASSERT_CHECKING
+ {
+ uint32 buf_state = pg_atomic_read_u32(&buf_hdr->state);
+
+ Assert(buf_state & BM_TAG_VALID);
+ Assert(!(buf_state & BM_VALID));
+ /* temp buffers don't use BM_IO_IN_PROGRESS */
+ if (!is_temp)
+ Assert(buf_state & BM_IO_IN_PROGRESS);
+ Assert(!(buf_state & BM_DIRTY));
+ }
+#endif
+
+ /* check for garbage data */
+ if (!failed &&
+ !PageIsVerifiedExtended((Page) bufdata, tag.blockNum,
+ PIV_LOG_WARNING | PIV_REPORT_STAT))
+ {
+ RelFileLocator rlocator = BufTagGetRelFileLocator(&tag);
+
+ if ((flags & READ_BUFFERS_ZERO_ON_ERROR) || zero_damaged_pages)
+ {
+ ereport(WARNING,
+ (errcode(ERRCODE_DATA_CORRUPTED),
+ errmsg("invalid page in block %u of relation %s; zeroing out page",
+ tag.blockNum,
+ relpathbackend(rlocator,
+ is_temp ? MyProcNumber : INVALID_PROC_NUMBER,
+ tag.forkNum).str)));
+ memset(bufdata, 0, BLCKSZ);
+ }
+ else
+ {
+ /* mark buffer as having failed */
+ failed = true;
+
+ /* encode error for buffer_readv_report */
+ result.status = PGAIO_RS_ERROR;
+ if (is_temp)
+ result.id = PGAIO_HCB_LOCAL_BUFFER_READV;
+ else
+ result.id = PGAIO_HCB_SHARED_BUFFER_READV;
+ result.error_data = buf_off;
+ }
+ }
+
+ /* Terminate I/O and set BM_VALID. */
+ set_flag_bits = failed ? BM_IO_ERROR : BM_VALID;
+ if (is_temp)
+ TerminateLocalBufferIO(buf_hdr, false, set_flag_bits, true);
+ else
+ TerminateBufferIO(buf_hdr, false, set_flag_bits, false, true);
+
+ /*
+ * Call the BUFFER_READ_DONE tracepoint in the callback, even though the
+ * callback may not be executed in the same backend that called
+ * BUFFER_READ_START. The alternative would be to defer calling the
+ * tracepoint to a later point (e.g. the local completion callback for
+ * shared buffer reads), which seems even less helpful and would also
+ * imply some overhead.
+ */
+ TRACE_POSTGRESQL_BUFFER_READ_DONE(tag.forkNum,
+ tag.blockNum,
+ tag.spcOid,
+ tag.dbOid,
+ tag.relNumber,
+ is_temp ? MyProcNumber : INVALID_PROC_NUMBER,
+ false);
+
+ return result;
+}
+
+/*
+ * Perform completion handling of a single AIO read. This read may cover
+ * multiple blocks / buffers.
+ *
+ * Shared between shared and local buffers, to reduce code duplication.
+ */
+static pg_attribute_always_inline PgAioResult
+buffer_readv_complete(PgAioHandle *ioh, PgAioResult prior_result,
+ uint8 cb_data, bool is_temp)
+{
+ PgAioResult result = prior_result;
+ PgAioTargetData *td = pgaio_io_get_target_data(ioh);
+ uint64 *io_data;
+ uint8 handle_data_len;
+
+ if (is_temp)
+ {
+ Assert(td->smgr.is_temp);
+ Assert(pgaio_io_get_owner(ioh) == MyProcNumber);
+ }
+ else
+ Assert(!td->smgr.is_temp);
+
+ /*
+ * Iterate over all the buffers affected by this IO and call the
+ * per-buffer completion function for each buffer.
+ */
+ io_data = pgaio_io_get_handle_data(ioh, &handle_data_len);
+ for (uint8 buf_off = 0; buf_off < handle_data_len; buf_off++)
+ {
+ Buffer buf = io_data[buf_off];
+ PgAioResult buf_result;
+ bool failed;
+
+ Assert(BufferIsValid(buf));
+
+ /*
+ * If the entire I/O failed on a lower-level, each buffer needs to be
+ * marked as failed. In case of a partial read, some buffers may be
+ * ok.
+ */
+ failed =
+ prior_result.status == PGAIO_RS_ERROR
+ || prior_result.result <= buf_off;
+
+ buf_result = buffer_readv_complete_one(buf_off, buf, cb_data, failed,
+ is_temp);
+
+ /*
+ * If there wasn't any prior error and page verification failed in
+ * some form, set the whole IO's result to the page's result.
+ */
+ if (result.status != PGAIO_RS_ERROR
+ && buf_result.status != PGAIO_RS_OK)
+ {
+ result = buf_result;
+ pgaio_result_report(result, td, LOG);
+ }
+ }
+
+ return result;
+}
+
+/*
+ * AIO error reporting callback for aio_shared_buffer_readv_cb and
+ * aio_local_buffer_readv_cb.
+ *
+ * Errors are encoded as follows:
+ * - the only error is page verification failing
+ * - result.error_data is the offset of the first page that failed
+ * verification in a larger IO
+ */
+static void
+buffer_readv_report(PgAioResult result, const PgAioTargetData *target_data,
+ int elevel)
+{
+ ProcNumber errProc;
+
+ if (target_data->smgr.is_temp)
+ errProc = MyProcNumber;
+ else
+ errProc = INVALID_PROC_NUMBER;
+
+ ereport(elevel,
+ errcode(ERRCODE_DATA_CORRUPTED),
+ errmsg("invalid page in block %u of relation %s",
+ target_data->smgr.blockNum + result.error_data,
+ relpathbackend(target_data->smgr.rlocator, errProc,
+ target_data->smgr.forkNum).str));
+}
+
+static void
+shared_buffer_readv_stage(PgAioHandle *ioh, uint8 cb_data)
+{
+ buffer_stage_common(ioh, false, false);
+}
+
+static PgAioResult
+shared_buffer_readv_complete(PgAioHandle *ioh, PgAioResult prior_result,
+ uint8 cb_data)
+{
+ return buffer_readv_complete(ioh, prior_result, cb_data, false);
+}
+
+static void
+local_buffer_readv_stage(PgAioHandle *ioh, uint8 cb_data)
+{
+ buffer_stage_common(ioh, false, true);
+}
+
+static PgAioResult
+local_buffer_readv_complete(PgAioHandle *ioh, PgAioResult prior_result,
+ uint8 cb_data)
+{
+ return buffer_readv_complete(ioh, prior_result, cb_data, true);
+}
+
+/* readv callback is passed READ_BUFFERS_* flags as callback data */
+const PgAioHandleCallbacks aio_shared_buffer_readv_cb = {
+ .stage = shared_buffer_readv_stage,
+ .complete_shared = shared_buffer_readv_complete,
+ .report = buffer_readv_report,
+};
+
+/* readv callback is passed READ_BUFFERS_* flags as callback data */
+const PgAioHandleCallbacks aio_local_buffer_readv_cb = {
+ .stage = local_buffer_readv_stage,
+
+ /*
+ * Note that this, in contrast to the shared_buffers case, uses
+ * complete_local, as only the issuing backend has access to the required
+ * datastructures. This is important in case the IO completion may be
+ * consumed incidentally by another backend.
+ */
+ .complete_local = local_buffer_readv_complete,
+ .report = buffer_readv_report,
+};
diff --git a/src/backend/storage/buffer/localbuf.c b/src/backend/storage/buffer/localbuf.c
index 3a722321533..6cd4d0f51eb 100644
--- a/src/backend/storage/buffer/localbuf.c
+++ b/src/backend/storage/buffer/localbuf.c
@@ -17,7 +17,9 @@
#include "access/parallel.h"
#include "executor/instrument.h"
+#include "pg_trace.h"
#include "pgstat.h"
+#include "storage/aio.h"
#include "storage/buf_internals.h"
#include "storage/bufmgr.h"
#include "storage/fd.h"
@@ -187,7 +189,7 @@ FlushLocalBuffer(BufferDesc *bufHdr, SMgrRelation reln)
* Try to start an I/O operation. There currently are no reasons for
* StartLocalBufferIO to return false, so we raise an error in that case.
*/
- if (!StartLocalBufferIO(bufHdr, false))
+ if (!StartLocalBufferIO(bufHdr, false, false))
elog(ERROR, "failed to start write IO on local buffer");
/* Find smgr relation for buffer */
@@ -211,7 +213,7 @@ FlushLocalBuffer(BufferDesc *bufHdr, SMgrRelation reln)
IOOP_WRITE, io_start, 1, BLCKSZ);
/* Mark not-dirty */
- TerminateLocalBufferIO(bufHdr, true, 0);
+ TerminateLocalBufferIO(bufHdr, true, 0, false);
pgBufferUsage.local_blks_written++;
}
@@ -430,7 +432,7 @@ ExtendBufferedRelLocal(BufferManagerRelation bmr,
pg_atomic_unlocked_write_u32(&existing_hdr->state, buf_state);
/* no need to loop for local buffers */
- StartLocalBufferIO(existing_hdr, true);
+ StartLocalBufferIO(existing_hdr, true, false);
}
else
{
@@ -446,7 +448,7 @@ ExtendBufferedRelLocal(BufferManagerRelation bmr,
hresult->id = victim_buf_id;
- StartLocalBufferIO(victim_buf_hdr, true);
+ StartLocalBufferIO(victim_buf_hdr, true, false);
}
}
@@ -515,13 +517,31 @@ MarkLocalBufferDirty(Buffer buffer)
* Like StartBufferIO, but for local buffers
*/
bool
-StartLocalBufferIO(BufferDesc *bufHdr, bool forInput)
+StartLocalBufferIO(BufferDesc *bufHdr, bool forInput, bool nowait)
{
- uint32 buf_state = pg_atomic_read_u32(&bufHdr->state);
+ uint32 buf_state;
+ /*
+ * With AIO the buffer could have IO in progress, e.g. when there are two
+ * scans of the same relation. Either wait for the other IO or return
+ * false.
+ */
+ if (pgaio_wref_valid(&bufHdr->io_wref))
+ {
+ PgAioWaitRef iow = bufHdr->io_wref;
+
+ if (nowait)
+ return false;
+
+ pgaio_wref_wait(&iow);
+ }
+
+ /* Once we get here, there is definitely no I/O active on this buffer */
+
+ /* Check if someone else already did the I/O */
+ buf_state = pg_atomic_read_u32(&bufHdr->state);
if (forInput ? (buf_state & BM_VALID) : !(buf_state & BM_DIRTY))
{
- /* someone else already did the I/O */
return false;
}
@@ -536,7 +556,8 @@ StartLocalBufferIO(BufferDesc *bufHdr, bool forInput)
* Like TerminateBufferIO, but for local buffers
*/
void
-TerminateLocalBufferIO(BufferDesc *bufHdr, bool clear_dirty, uint32 set_flag_bits)
+TerminateLocalBufferIO(BufferDesc *bufHdr, bool clear_dirty, uint32 set_flag_bits,
+ bool release_aio)
{
/* Only need to adjust flags */
uint32 buf_state = pg_atomic_read_u32(&bufHdr->state);
@@ -549,12 +570,22 @@ TerminateLocalBufferIO(BufferDesc *bufHdr, bool clear_dirty, uint32 set_flag_bit
if (clear_dirty)
buf_state &= ~BM_DIRTY;
+ if (release_aio)
+ {
+ /* release pin held by IO subsystem, see also buffer_stage_common() */
+ Assert(BUF_STATE_GET_REFCOUNT(buf_state) > 0);
+ buf_state -= BUF_REFCOUNT_ONE;
+ pgaio_wref_clear(&bufHdr->io_wref);
+ }
+
buf_state |= set_flag_bits;
pg_atomic_unlocked_write_u32(&bufHdr->state, buf_state);
/* local buffers don't track IO using resowners */
/* local buffers don't use the IO CV, as no other process can see buffer */
+
+ /* local buffers don't use BM_PIN_COUNT_WAITER, so no need to wake */
}
/*
@@ -575,6 +606,19 @@ InvalidateLocalBuffer(BufferDesc *bufHdr, bool check_unreferenced)
uint32 buf_state;
LocalBufferLookupEnt *hresult;
+ /*
+ * It's possible that we started IO on this buffer before e.g. aborting
+ * the transaction that created a table. We need to wait for that IO to
+ * complete before removing / reusing the buffer.
+ */
+ if (pgaio_wref_valid(&bufHdr->io_wref))
+ {
+ PgAioWaitRef iow = bufHdr->io_wref;
+
+ pgaio_wref_wait(&iow);
+ Assert(!pgaio_wref_valid(&bufHdr->io_wref));
+ }
+
buf_state = pg_atomic_read_u32(&bufHdr->state);
/*
@@ -714,6 +758,8 @@ InitLocalBuffers(void)
*/
buf->buf_id = -i - 2;
+ pgaio_wref_clear(&buf->io_wref);
+
/*
* Intentionally do not initialize the buffer's atomic variable
* (besides zeroing the underlying memory above). That way we get
--
2.48.1.76.g4e746b1a31.dirty
[text/x-diff] v2.13-0008-squash-later-bufmgr-Implement-AIO-read-support.patch (9.9K, ../../o2dtdrlc2xqpjycsfuvv7u6zyrqj7jfxqu32et3lyelkwce2jy@3k3fwdiiu73v/9-v2.13-0008-squash-later-bufmgr-Implement-AIO-read-support.patch)
download | inline diff:
From 2f2370859b959bf48424966db30e3ba25a1ac0f0 Mon Sep 17 00:00:00 2001
From: Andres Freund <andres@anarazel.de>
Date: Wed, 26 Mar 2025 14:41:48 -0400
Subject: [PATCH v2.13 08/28] squash-later: bufmgr: Implement AIO read support
---
src/backend/storage/buffer/bufmgr.c | 200 ++++++++++++++++++++++------
1 file changed, 156 insertions(+), 44 deletions(-)
diff --git a/src/backend/storage/buffer/bufmgr.c b/src/backend/storage/buffer/bufmgr.c
index 3cf8b0f98d2..0b96e256625 100644
--- a/src/backend/storage/buffer/bufmgr.c
+++ b/src/backend/storage/buffer/bufmgr.c
@@ -6327,20 +6327,75 @@ buffer_stage_common(PgAioHandle *ioh, bool is_write, bool is_temp)
}
}
+/*
+ * Helper to encode errors for buffer_readv_complete()
+ *
+ * Errors are encoded as follows:
+ * - bit 0 indicates whether page was zeroed (1) or not (0)
+ * - next 8 bits indicate the first offset is the offset of the first page
+ * that failed verification in a larger IO
+ * - next 8 bits indicate the number of corruptions
+ */
+static inline void
+buffer_readv_encode_error(PgAioResult *result, bool is_temp,
+ bool was_zeroed, uint8 first_invalid_off,
+ uint8 count_invalid)
+{
+
+ uint8 shift = 0;
+
+ StaticAssertStmt(PG_IOV_MAX <= 1 << 8,
+ "PG_IOV_MAX is bigger than reserved space for error data");
+
+ result->error_data = 0;
+
+ result->error_data |= was_zeroed << shift;
+ shift += 1;
+
+ result->error_data |= first_invalid_off << shift;
+ shift += 8;
+
+ result->error_data |= count_invalid << shift;
+ shift += 8;
+
+ result->id = is_temp ? PGAIO_HCB_LOCAL_BUFFER_READV :
+ PGAIO_HCB_SHARED_BUFFER_READV;
+ result->status = was_zeroed ? PGAIO_RS_WARNING : PGAIO_RS_ERROR;
+}
+
+/*
+ * Decode readv errors as encoded by buffer_readv_encode_error().
+ */
+static inline void
+buffer_readv_decode_error(PgAioResult result,
+ bool *was_zeroed, uint8 *first_invalid_off, uint8 *count_invalid)
+{
+ uint32 rem_error = result.error_data;
+
+ *was_zeroed = rem_error & 1;
+ rem_error >>= 1;
+
+ *first_invalid_off = rem_error & 0xff;
+ rem_error >>= 8;
+
+ *count_invalid = rem_error & 0xff;
+ rem_error >>= 8;
+}
+
/*
* Helper for AIO readv completion callbacks, supporting both shared and temp
* buffers. Gets called once for each buffer in a multi-page read.
*/
-static pg_attribute_always_inline PgAioResult
-buffer_readv_complete_one(uint8 buf_off, Buffer buffer, uint8 flags,
- bool failed, bool is_temp)
+static pg_attribute_always_inline void
+buffer_readv_complete_one(PgAioTargetData *td, uint8 buf_off, Buffer buffer,
+ uint8 flags, bool failed, bool is_temp,
+ bool *failed_verification)
{
BufferDesc *buf_hdr = is_temp ?
GetLocalBufferDescriptor(-buffer - 1)
: GetBufferDescriptor(buffer - 1);
BufferTag tag = buf_hdr->tag;
char *bufdata = BufferGetBlock(buffer);
- PgAioResult result = {.status = PGAIO_RS_OK};
uint32 set_flag_bits;
/* check that the buffer is in the expected state for a read */
@@ -6357,37 +6412,45 @@ buffer_readv_complete_one(uint8 buf_off, Buffer buffer, uint8 flags,
}
#endif
+ *failed_verification = false;
+
/* check for garbage data */
if (!failed &&
!PageIsVerifiedExtended((Page) bufdata, tag.blockNum,
PIV_LOG_WARNING | PIV_REPORT_STAT))
{
- RelFileLocator rlocator = BufTagGetRelFileLocator(&tag);
+ PgAioResult result_one;
- if ((flags & READ_BUFFERS_ZERO_ON_ERROR) || zero_damaged_pages)
+ *failed_verification = true;
+
+ if (flags & READ_BUFFERS_ZERO_ON_ERROR)
{
- ereport(WARNING,
- (errcode(ERRCODE_DATA_CORRUPTED),
- errmsg("invalid page in block %u of relation %s; zeroing out page",
- tag.blockNum,
- relpathbackend(rlocator,
- is_temp ? MyProcNumber : INVALID_PROC_NUMBER,
- tag.forkNum).str)));
memset(bufdata, 0, BLCKSZ);
}
else
{
/* mark buffer as having failed */
failed = true;
-
- /* encode error for buffer_readv_report */
- result.status = PGAIO_RS_ERROR;
- if (is_temp)
- result.id = PGAIO_HCB_LOCAL_BUFFER_READV;
- else
- result.id = PGAIO_HCB_SHARED_BUFFER_READV;
- result.error_data = buf_off;
}
+
+ /*
+ * Immediately log a message about the invalid page, but only to the
+ * server log. The reason to do so immediately is that this may be
+ * executed in a different backend than the one that originated the
+ * request. The reason to do so immediately is that the originator
+ * might not process the query result immediately (because it is busy
+ * doing another part of query processing) or at all (e.g. if it was
+ * cancelled or errored out due to another IO also failing). The
+ * issuer of the IO will emit an ERROR or WARNING when processing the
+ * IO's results
+ *
+ * To avoid duplicating the code to emit these log messages, we reuse
+ * buffer_readv_report().
+ */
+ buffer_readv_encode_error(&result_one, is_temp,
+ flags & READ_BUFFERS_ZERO_ON_ERROR,
+ buf_off, 1);
+ pgaio_result_report(result_one, td, LOG_SERVER_ONLY);
}
/* Terminate I/O and set BM_VALID. */
@@ -6412,8 +6475,6 @@ buffer_readv_complete_one(uint8 buf_off, Buffer buffer, uint8 flags,
tag.relNumber,
is_temp ? MyProcNumber : INVALID_PROC_NUMBER,
false);
-
- return result;
}
/*
@@ -6428,6 +6489,8 @@ buffer_readv_complete(PgAioHandle *ioh, PgAioResult prior_result,
{
PgAioResult result = prior_result;
PgAioTargetData *td = pgaio_io_get_target_data(ioh);
+ uint8 first_invalid_off = 0;
+ uint8 invalid_count = 0;
uint64 *io_data;
uint8 handle_data_len;
@@ -6447,35 +6510,46 @@ buffer_readv_complete(PgAioHandle *ioh, PgAioResult prior_result,
for (uint8 buf_off = 0; buf_off < handle_data_len; buf_off++)
{
Buffer buf = io_data[buf_off];
- PgAioResult buf_result;
bool failed;
+ bool failed_verification = false;
Assert(BufferIsValid(buf));
/*
* If the entire I/O failed on a lower-level, each buffer needs to be
- * marked as failed. In case of a partial read, some buffers may be
- * ok.
+ * marked as failed. In case of a partial read, the first few buffers
+ * may be ok.
*/
failed =
prior_result.status == PGAIO_RS_ERROR
|| prior_result.result <= buf_off;
- buf_result = buffer_readv_complete_one(buf_off, buf, cb_data, failed,
- is_temp);
+ buffer_readv_complete_one(td, buf_off, buf, cb_data, failed, is_temp,
+ &failed_verification);
/*
- * If there wasn't any prior error and page verification failed in
- * some form, set the whole IO's result to the page's result.
+ * Track information about the number of errors across all pages, as
+ * there can be multiple pages failing verification as part of one IO.
*/
- if (result.status != PGAIO_RS_ERROR
- && buf_result.status != PGAIO_RS_OK)
+ if (failed_verification)
{
- result = buf_result;
- pgaio_result_report(result, td, LOG);
+ if (invalid_count++ == 0)
+ first_invalid_off = buf_off;
}
}
+ /*
+ * If the smgr read succeeded [partially] and page verification failed for
+ * some of the pages, adjust the IO's result state appropriately.
+ */
+ if (prior_result.status != PGAIO_RS_ERROR && invalid_count > 0)
+ {
+ bool was_zeroed = cb_data & READ_BUFFERS_ZERO_ON_ERROR;
+
+ buffer_readv_encode_error(&result, is_temp, was_zeroed,
+ first_invalid_off, invalid_count);
+ }
+
return result;
}
@@ -6483,28 +6557,66 @@ buffer_readv_complete(PgAioHandle *ioh, PgAioResult prior_result,
* AIO error reporting callback for aio_shared_buffer_readv_cb and
* aio_local_buffer_readv_cb.
*
- * Errors are encoded as follows:
- * - the only error is page verification failing
- * - result.error_data is the offset of the first page that failed
- * verification in a larger IO
+ * The error is encoded / decoded in buffer_readv_encode_error() /
+ * buffer_readv_decode_error().
*/
static void
buffer_readv_report(PgAioResult result, const PgAioTargetData *target_data,
int elevel)
{
+ BlockNumber blocknum = target_data->smgr.blockNum;
+ int nblocks = target_data->smgr.nblocks;
ProcNumber errProc;
+ bool was_zeroed;
+ uint8 first_invalid_off;
+ uint8 invalid_count;
+ RelPathStr rpath;
+
+ buffer_readv_decode_error(result, &was_zeroed, &first_invalid_off,
+ &invalid_count);
if (target_data->smgr.is_temp)
errProc = MyProcNumber;
else
errProc = INVALID_PROC_NUMBER;
- ereport(elevel,
- errcode(ERRCODE_DATA_CORRUPTED),
- errmsg("invalid page in block %u of relation %s",
- target_data->smgr.blockNum + result.error_data,
- relpathbackend(target_data->smgr.rlocator, errProc,
- target_data->smgr.forkNum).str));
+ rpath = relpathbackend(target_data->smgr.rlocator, errProc,
+ target_data->smgr.forkNum);
+
+ if (was_zeroed)
+ {
+ ereport(elevel,
+ errcode(ERRCODE_DATA_CORRUPTED),
+ invalid_count == 1 ?
+ errmsg("invalid page in block %u of relation %s; zeroing out page",
+ blocknum + first_invalid_off, rpath.str) :
+ errmsg("zeroing out %u invalid pages among blocks %u..%u of relation %s",
+ invalid_count,
+ blocknum, blocknum + nblocks - 1, rpath.str),
+ invalid_count > 1 ?
+ errdetail("Block %u held first invalid page.",
+ blocknum + first_invalid_off) : 0,
+ invalid_count > 1 ?
+ errhint("See server log for the other %u invalid blocks.",
+ invalid_count - 1) : 0);
+ }
+ else
+ {
+ ereport(elevel,
+ errcode(ERRCODE_DATA_CORRUPTED),
+ invalid_count == 1 ?
+ errmsg("invalid page in block %u of relation %s",
+ blocknum + first_invalid_off, rpath.str) :
+ errmsg("%u invalid pages among blocks %u..%u of relation %s",
+ invalid_count,
+ blocknum, blocknum + nblocks - 1, rpath.str),
+ invalid_count > 1 ?
+ errdetail("Block %u held first invalid page.",
+ blocknum + first_invalid_off) : 0,
+ invalid_count > 1 ?
+ errhint("See server log for the other %u invalid blocks.",
+ invalid_count - 1) : 0);
+ }
}
static void
--
2.48.1.76.g4e746b1a31.dirty
[text/x-diff] v2.13-0009-bufmgr-Use-AIO-in-StartReadBuffers.patch (26.3K, ../../o2dtdrlc2xqpjycsfuvv7u6zyrqj7jfxqu32et3lyelkwce2jy@3k3fwdiiu73v/10-v2.13-0009-bufmgr-Use-AIO-in-StartReadBuffers.patch)
download | inline diff:
From 8af4c611c646f06a31c4be83bd301b9599841dc4 Mon Sep 17 00:00:00 2001
From: Andres Freund <andres@anarazel.de>
Date: Tue, 18 Mar 2025 14:40:06 -0400
Subject: [PATCH v2.13 09/28] bufmgr: Use AIO in StartReadBuffers()
This finally introduces the first actual use of AIO. StartReadBuffers() now
uses the AIO routines to issue IO. This converts a lot of callers to use the
AIO infrastructure.
As the implementation of StartReadBuffers() is also used by the functions for
reading individual blocks (StartReadBuffer() and through that
ReadBufferExtended()) this means all buffered read IO passes through the AIO
paths. However, as those are synchronous reads, actually performing the IO
asynchronously would be rarely beneficial. Instead such IOs are flagged to
always be executed synchronously. This way we don't have to duplicate a fair
bit of code.
When io_method=sync is used, the IO patterns generated after this change are
the same as before, i.e. actual reads are only issued in WaitReadBuffers() and
StartReadBuffers() may issue prefetch requests. This allows to bypass most of
the actual asynchronicity, which is important to make a change as big as this
less risky.
One thing worth calling out is that, if IO is actually executed
asynchronously, the precise meaning of what track_io_timing is measuring has
changed. Previously it tracked the time for each IO, but that does not make
sense when multiple IOs are executed concurrently. Now it only measures the
time actually spent waiting for IO. A subsequent commit will adjust the docs
for this.
While AIO is now actually used, the logic in read_stream.c will often prevent
using sufficiently many concurrent IOs. That will be addressed in the next
commit.
Reviewed-by: Noah Misch <noah@leadboat.com>
Reviewed-by: Nazir Bilal Yavuz <byavuz81@gmail.com>
Co-authored-by: Andres Freund <andres@anarazel.de>
Co-authored-by: Thomas Munro <thomas.munro@gmail.com>
Discussion: https://postgr.es/m/uvrtrknj4kdytuboidbhwclo4gxhswwcpgadptsjvjqcluzmah%40brqs62irg4dt
Discussion: https://postgr.es/m/20210223100344.llw5an2aklengrmn@alap3.anarazel.de
Discussion: https://postgr.es/m/stj36ea6yyhoxtqkhpieia2z4krnam7qyetc57rfezgk4zgapf@gcnactj4z56m
---
src/include/storage/bufmgr.h | 6 +
src/backend/storage/buffer/bufmgr.c | 594 +++++++++++++++++++++-------
2 files changed, 463 insertions(+), 137 deletions(-)
diff --git a/src/include/storage/bufmgr.h b/src/include/storage/bufmgr.h
index 5137288c96e..7abb3bdfd9d 100644
--- a/src/include/storage/bufmgr.h
+++ b/src/include/storage/bufmgr.h
@@ -112,6 +112,9 @@ typedef struct BufferManagerRelation
#define READ_BUFFERS_ZERO_ON_ERROR (1 << 0)
/* Call smgrprefetch() if I/O necessary. */
#define READ_BUFFERS_ISSUE_ADVICE (1 << 1)
+/* IO will immediately be waited for */
+#define READ_BUFFERS_SYNCHRONOUSLY (1 << 2)
+
struct ReadBuffersOperation
{
@@ -131,6 +134,9 @@ struct ReadBuffersOperation
BlockNumber blocknum;
int flags;
int16 nblocks;
+ int16 nblocks_done;
+ PgAioWaitRef io_wref;
+ PgAioReturn io_return;
};
typedef struct ReadBuffersOperation ReadBuffersOperation;
diff --git a/src/backend/storage/buffer/bufmgr.c b/src/backend/storage/buffer/bufmgr.c
index 0b96e256625..83dda2cb5d8 100644
--- a/src/backend/storage/buffer/bufmgr.c
+++ b/src/backend/storage/buffer/bufmgr.c
@@ -531,6 +531,8 @@ static inline BufferDesc *BufferAlloc(SMgrRelation smgr,
BlockNumber blockNum,
BufferAccessStrategy strategy,
bool *foundPtr, IOContext io_context);
+static bool AsyncReadBuffers(ReadBuffersOperation *operation, int *nblocks);
+static void CheckReadBuffersOperation(ReadBuffersOperation *operation, bool is_complete);
static Buffer GetVictimBuffer(BufferAccessStrategy strategy, IOContext io_context);
static void FlushBuffer(BufferDesc *buf, SMgrRelation reln,
IOObject io_object, IOContext io_context);
@@ -1231,10 +1233,14 @@ ReadBuffer_common(Relation rel, SMgrRelation smgr, char smgr_persistence,
return buffer;
}
+ /*
+ * Signal that we are going to immediately wait. If we're immediately
+ * waiting, there is no benefit in actually executing the IO
+ * asynchronously, it would just add dispatch overhead.
+ */
+ flags = READ_BUFFERS_SYNCHRONOUSLY;
if (mode == RBM_ZERO_ON_ERROR)
- flags = READ_BUFFERS_ZERO_ON_ERROR;
- else
- flags = 0;
+ flags |= READ_BUFFERS_ZERO_ON_ERROR;
operation.smgr = smgr;
operation.rel = rel;
operation.persistence = persistence;
@@ -1259,6 +1265,7 @@ StartReadBuffersImpl(ReadBuffersOperation *operation,
{
int actual_nblocks = *nblocks;
int maxcombine = 0;
+ bool did_start_io;
Assert(*nblocks == 1 || allow_forwarding);
Assert(*nblocks > 0);
@@ -1326,6 +1333,20 @@ StartReadBuffersImpl(ReadBuffersOperation *operation,
if (i == 0)
{
*nblocks = 1;
+
+#ifdef USE_ASSERT_CHECKING
+
+ /*
+ * Initialize enough of ReadBuffersOperation to make
+ * CheckReadBuffersOperation() work. Outside of assertions
+ * that's not necessary when no IO is issued.
+ */
+ operation->buffers = buffers;
+ operation->blocknum = blockNum;
+ operation->nblocks = 1;
+ operation->nblocks_done = 1;
+ CheckReadBuffersOperation(operation, true);
+#endif
return false;
}
@@ -1368,25 +1389,74 @@ StartReadBuffersImpl(ReadBuffersOperation *operation,
operation->blocknum = blockNum;
operation->flags = flags;
operation->nblocks = actual_nblocks;
+ operation->nblocks_done = 0;
+ pgaio_wref_clear(&operation->io_wref);
- if (flags & READ_BUFFERS_ISSUE_ADVICE)
+ /*
+ * When using AIO, start the IO in the background. If not, issue prefetch
+ * requests if desired by the caller.
+ *
+ * The reason we have a dedicated path for IOMETHOD_SYNC here is to
+ * de-risk the introduction of AIO somewhat. It's a large architectural
+ * change, with lots of chances for unanticipated performance effects.
+ *
+ * Use of IOMETHOD_SYNC already leads to not actually performing IO
+ * asynchronously, but without the check here we'd execute IO earlier than
+ * we used to. Eventually this IOMETHOD_SYNC specific path should go away.
+ */
+ if (io_method != IOMETHOD_SYNC)
{
/*
- * In theory we should only do this if PinBufferForBlock() had to
- * allocate new buffers above. That way, if two calls to
- * StartReadBuffers() were made for the same blocks before
- * WaitReadBuffers(), only the first would issue the advice. That'd be
- * a better simulation of true asynchronous I/O, which would only
- * start the I/O once, but isn't done here for simplicity.
+ * Try to start IO asynchronously. It's possible that no IO needs to
+ * be started, if another backend already performed the IO.
+ *
+ * Note that if an IO is started, it might not cover the entire
+ * requested range, e.g. because an intermediary block has been read
+ * in by another backend. In that case any "trailing" buffers we
+ * already pinned above will be "forwarded" by read_stream.c to the
+ * next call to StartReadBuffers().
+ *
+ * This is signalled to the caller by decrementing *nblocks *and*
+ * reducing operation->nblocks. The latter is done here, but not below
+ * WaitReadBuffers(), as in WaitReadBuffers() we can't "shorten" the
+ * overall read size anymore, we need to retry until done in its
+ * entirety or until failed.
*/
- smgrprefetch(operation->smgr,
- operation->forknum,
- blockNum,
- actual_nblocks);
+ did_start_io = AsyncReadBuffers(operation, nblocks);
+
+ operation->nblocks = *nblocks;
+ }
+ else
+ {
+ operation->flags |= READ_BUFFERS_SYNCHRONOUSLY;
+
+ if (flags & READ_BUFFERS_ISSUE_ADVICE)
+ {
+ /*
+ * In theory we should only do this if PinBufferForBlock() had to
+ * allocate new buffers above. That way, if two calls to
+ * StartReadBuffers() were made for the same blocks before
+ * WaitReadBuffers(), only the first would issue the advice.
+ * That'd be a better simulation of true asynchronous I/O, which
+ * would only start the I/O once, but isn't done here for
+ * simplicity.
+ */
+ smgrprefetch(operation->smgr,
+ operation->forknum,
+ blockNum,
+ actual_nblocks);
+ }
+
+ /*
+ * Indicate that WaitReadBuffers() should be called. WaitReadBuffers()
+ * will initiate the necessary IO.
+ */
+ did_start_io = true;
}
- /* Indicate that WaitReadBuffers() should be called. */
- return true;
+ CheckReadBuffersOperation(operation, !did_start_io);
+
+ return did_start_io;
}
/*
@@ -1452,8 +1522,35 @@ StartReadBuffer(ReadBuffersOperation *operation,
return result;
}
+/*
+ * Perform sanity checks on the ReadBuffersOperation.
+ */
+static void
+CheckReadBuffersOperation(ReadBuffersOperation *operation, bool is_complete)
+{
+#ifdef USE_ASSERT_CHECKING
+ Assert(operation->nblocks_done <= operation->nblocks);
+ Assert(!is_complete || operation->nblocks == operation->nblocks_done);
+
+ for (int i = 0; i < operation->nblocks; i++)
+ {
+ Buffer buffer = operation->buffers[i];
+ BufferDesc *buf_hdr = BufferIsLocal(buffer) ?
+ GetLocalBufferDescriptor(-buffer - 1) :
+ GetBufferDescriptor(buffer - 1);
+
+ Assert(BufferGetBlockNumber(buffer) == operation->blocknum + i);
+ Assert(pg_atomic_read_u32(&buf_hdr->state) & BM_TAG_VALID);
+
+ if (i < operation->nblocks_done)
+ Assert(pg_atomic_read_u32(&buf_hdr->state) & BM_VALID);
+ }
+#endif
+}
+
+/* helper for ReadBuffersCanStartIO(), to avoid repetition */
static inline bool
-WaitReadBuffersCanStartIO(Buffer buffer, bool nowait)
+ReadBuffersCanStartIOOnce(Buffer buffer, bool nowait)
{
if (BufferIsLocal(buffer))
return StartLocalBufferIO(GetLocalBufferDescriptor(-buffer - 1),
@@ -1462,31 +1559,240 @@ WaitReadBuffersCanStartIO(Buffer buffer, bool nowait)
return StartBufferIO(GetBufferDescriptor(buffer - 1), true, nowait);
}
+/*
+ * Helper for AsyncReadBuffers that tries to get the buffer ready for IO.
+ */
+static inline bool
+ReadBuffersCanStartIO(Buffer buffer, bool nowait)
+{
+ /*
+ * If this backend currently has staged IO, we need to submit the pending
+ * IO before waiting for the right to issue IO, to avoid the potential for
+ * deadlocks (and, more commonly, unnecessary delays for other backends).
+ */
+ if (!nowait && pgaio_have_staged())
+ {
+ if (ReadBuffersCanStartIOOnce(buffer, true))
+ return true;
+
+ /*
+ * Unfortunately StartBufferIO() returning false doesn't allow to
+ * distinguish between the buffer already being valid and IO already
+ * being in progress. Since IO already being in progress is quite
+ * rare, this approach seems fine.
+ */
+ pgaio_submit_staged();
+ }
+
+ return ReadBuffersCanStartIOOnce(buffer, nowait);
+}
+
+/*
+ * Helper for WaitReadBuffers() that processes the results of a readv
+ * operation, raising an error if necessary.
+ */
+static void
+ProcessReadBuffersResult(ReadBuffersOperation *operation)
+{
+ PgAioReturn *aio_ret = &operation->io_return;
+ int newly_read_blocks = 0;
+
+ Assert(pgaio_wref_valid(&operation->io_wref));
+ Assert(aio_ret->result.status != PGAIO_RS_UNKNOWN);
+
+ /*
+ * SMGR reports the number of blocks successfully read as the result of
+ * the IO operation. Thus we can simply add that to ->nblocks_done.
+ */
+
+ if (likely(aio_ret->result.status == PGAIO_RS_OK))
+ newly_read_blocks = aio_ret->result.result;
+ else if (aio_ret->result.status == PGAIO_RS_PARTIAL)
+ {
+ /*
+ * We'll retry, so we just emit a debug message to the server log (or
+ * not even that in prod scenarios).
+ */
+ pgaio_result_report(aio_ret->result, &aio_ret->target_data, DEBUG1);
+ newly_read_blocks = aio_ret->result.result;
+
+ elog(DEBUG3, "partial read, will retry");
+
+ }
+ else if (aio_ret->result.status == PGAIO_RS_ERROR)
+ pgaio_result_report(aio_ret->result, &aio_ret->target_data, ERROR);
+
+ Assert(newly_read_blocks > 0);
+ Assert(newly_read_blocks <= MAX_IO_COMBINE_LIMIT);
+
+ operation->nblocks_done += newly_read_blocks;
+
+ Assert(operation->nblocks_done <= operation->nblocks);
+}
+
void
WaitReadBuffers(ReadBuffersOperation *operation)
{
- Buffer *buffers;
- int nblocks;
- BlockNumber blocknum;
- ForkNumber forknum;
+ PgAioReturn *aio_ret = &operation->io_return;
IOContext io_context;
IOObject io_object;
- char persistence;
- /* Find the range of the physical read we need to perform. */
- nblocks = operation->nblocks;
- buffers = &operation->buffers[0];
- blocknum = operation->blocknum;
- forknum = operation->forknum;
- persistence = operation->persistence;
+ if (operation->persistence == RELPERSISTENCE_TEMP)
+ {
+ io_context = IOCONTEXT_NORMAL;
+ io_object = IOOBJECT_TEMP_RELATION;
+ }
+ else
+ {
+ io_context = IOContextForStrategy(operation->strategy);
+ io_object = IOOBJECT_RELATION;
+ }
- Assert(nblocks > 0);
- Assert(nblocks <= MAX_IO_COMBINE_LIMIT);
+ /*
+ * If we get here without an IO operation having been issued, the
+ * io_method == IOMETHOD_SYNC path must have been used. Otherwise the
+ * caller should not have called WaitReadBuffers().
+ *
+ * In the case of IOMETHOD_SYNC, we start - as we used to before the
+ * introducing of AIO - the IO in WaitReadBuffers(). This is done as part
+ * of the retry logic below, no extra code is required.
+ *
+ * This path is expected to eventually go away.
+ */
+ if (!pgaio_wref_valid(&operation->io_wref) && io_method != IOMETHOD_SYNC)
+ elog(ERROR, "waiting for read operation that didn't read");
+
+ /*
+ * To handle partial reads, and IOMETHOD_SYNC, we re-issue IO until we're
+ * done. We may need multiple retries, not just because we could get
+ * multiple partial reads, but also because some of the remaining
+ * to-be-read buffers may have been read in by other backends, limiting
+ * the IO size.
+ */
+ while (true)
+ {
+ int ignored_nblocks_progress;
+
+ CheckReadBuffersOperation(operation, false);
+
+ /*
+ * If there is an IO associated with the operation, we may need to
+ * wait for it.
+ */
+ if (pgaio_wref_valid(&operation->io_wref))
+ {
+ /*
+ * Track the time spent waiting for the IO to complete. As
+ * tracking a wait even if we don't actually need to wait
+ *
+ * a) is not cheap, due to the timestamping overhead
+ *
+ * b) reports some time as waiting, even if we never waited
+ *
+ * we first check if we already know the IO is complete.
+ */
+ if (aio_ret->result.status == PGAIO_RS_UNKNOWN &&
+ !pgaio_wref_check_done(&operation->io_wref))
+ {
+ instr_time io_start = pgstat_prepare_io_time(track_io_timing);
+
+ pgaio_wref_wait(&operation->io_wref);
+
+ /*
+ * The IO operation itself was already counted earlier, in
+ * AsyncReadBuffers(), this just accounts for the wait time.
+ */
+ pgstat_count_io_op_time(io_object, io_context, IOOP_READ,
+ io_start, 0, 0);
+ }
+ else
+ {
+ Assert(pgaio_wref_check_done(&operation->io_wref));
+ }
+
+ /*
+ * We now are sure the IO completed. Check the results. This
+ * includes reporting on errors if there were any.
+ */
+ ProcessReadBuffersResult(operation);
+ }
+
+ /*
+ * Most of the time, the one IO we already started, will read in
+ * everything. But we need to deal with partial reads and buffers not
+ * needing IO anymore.
+ */
+ if (operation->nblocks_done == operation->nblocks)
+ break;
+
+ CHECK_FOR_INTERRUPTS();
+
+ /*
+ * This may only complete the IO partially, either because some
+ * buffers were already valid, or because of a partial read.
+ *
+ * NB: In contrast to after the AsyncReadBuffers() call in
+ * StartReadBuffers(), we do *not* reduce
+ * ReadBuffersOperation->nblocks here, callers expect the full
+ * operation to be completed at this point (as more operations may
+ * have been queued).
+ */
+ AsyncReadBuffers(operation, &ignored_nblocks_progress);
+ }
+
+ CheckReadBuffersOperation(operation, true);
+
+ /* NB: READ_DONE tracepoint was already executed in completion callback */
+}
+
+/*
+ * Initiate IO for the ReadBuffersOperation
+ *
+ * This function only starts a single IO at a time. The size of the IO may be
+ * limited to below the to-be-read blocks, if one of the buffers has
+ * concurrently been read in. If the first to-be-read buffer is already valid,
+ * no IO will be issued.
+ *
+ * To support retries after partial reads, the first operation->nblocks_done
+ * buffers are skipped.
+ *
+ * On return *nblocks_progress is updated to reflect the number of buffers
+ * affected by the call. If the first buffer is valid, *nblocks_progress is
+ * set to 1 and operation->nblocks_done is incremented.
+ *
+ * Returns true if IO was initiated, false if no IO was necessary.
+ */
+static bool
+AsyncReadBuffers(ReadBuffersOperation *operation, int *nblocks_progress)
+{
+ Buffer *buffers = &operation->buffers[0];
+ int flags = operation->flags;
+ BlockNumber blocknum = operation->blocknum;
+ ForkNumber forknum = operation->forknum;
+ char persistence = operation->persistence;
+ int16 nblocks_done = operation->nblocks_done;
+ Buffer *io_buffers = &operation->buffers[nblocks_done];
+ int io_buffers_len = 0;
+ PgAioHandle *ioh;
+ uint32 ioh_flags = 0;
+ void *io_pages[MAX_IO_COMBINE_LIMIT];
+ IOContext io_context;
+ IOObject io_object;
+ bool did_start_io;
+
+ /*
+ * When this IO is executed synchronously, either because the caller will
+ * immediately block waiting for the IO or because IOMETHOD_SYNC is used,
+ * the AIO subsystem needs to know.
+ */
+ if (flags & READ_BUFFERS_SYNCHRONOUSLY)
+ ioh_flags |= PGAIO_HF_SYNCHRONOUS;
if (persistence == RELPERSISTENCE_TEMP)
{
io_context = IOCONTEXT_NORMAL;
io_object = IOOBJECT_TEMP_RELATION;
+ ioh_flags |= PGAIO_HF_REFERENCES_LOCAL;
}
else
{
@@ -1494,140 +1800,154 @@ WaitReadBuffers(ReadBuffersOperation *operation)
io_object = IOOBJECT_RELATION;
}
- for (int i = 0; i < nblocks; ++i)
+ /*
+ * Get IO handle before ReadBuffersCanStartIO(), as pgaio_io_acquire()
+ * might block, which we don't want after setting IO_IN_PROGRESS.
+ *
+ * If we need to wait for IO before we can get a handle, submit
+ * already-staged IO first, so that other backends don't need to wait.
+ * There wouldn't be a deadlock risk, as pgaio_io_acquire() just needs to
+ * wait for already submitted IO, which doesn't require additional locks,
+ * but it could still cause undesirable waits.
+ *
+ * A secondary benefit is that this would allow us to measure the time in
+ * pgaio_io_acquire() without causing undue timer overhead in the common,
+ * non-blocking, case. However, currently the pgstats infrastructure
+ * doesn't really allow that, as it a) asserts that an operation can't
+ * have time without operations b) doesn't have an API to report
+ * "accumulated" time.
+ */
+ ioh = pgaio_io_acquire_nb(CurrentResourceOwner, &operation->io_return);
+ if (unlikely(!ioh))
+ {
+ pgaio_submit_staged();
+
+ ioh = pgaio_io_acquire(CurrentResourceOwner, &operation->io_return);
+ }
+
+ /*
+ * Check if we can start IO on the first to-be-read buffer.
+ *
+ * If an I/O is already in progress in another backend, we want to wait
+ * for the outcome: either done, or something went wrong and we will
+ * retry.
+ */
+ if (!ReadBuffersCanStartIO(buffers[nblocks_done], false))
+ {
+ /*
+ * Someone else has already completed this block, we're done.
+ *
+ * When IO is necessary, ->nblocks_done is updated in
+ * ProcessReadBuffersResult(), but that is not called if no IO is
+ * necessary. Thus update here.
+ */
+ operation->nblocks_done += 1;
+ *nblocks_progress = 1;
+
+ pgaio_io_release(ioh);
+ pgaio_wref_clear(&operation->io_wref);
+ did_start_io = false;
+
+ /*
+ * Report and track this as a 'hit' for this backend, even though it
+ * must have started out as a miss in PinBufferForBlock(). The other
+ * backend will track this as a 'read'.
+ */
+ TRACE_POSTGRESQL_BUFFER_READ_DONE(forknum, blocknum + operation->nblocks_done,
+ operation->smgr->smgr_rlocator.locator.spcOid,
+ operation->smgr->smgr_rlocator.locator.dbOid,
+ operation->smgr->smgr_rlocator.locator.relNumber,
+ operation->smgr->smgr_rlocator.backend,
+ true);
+
+ if (persistence == RELPERSISTENCE_TEMP)
+ pgBufferUsage.local_blks_hit += 1;
+ else
+ pgBufferUsage.shared_blks_hit += 1;
+
+ if (operation->rel)
+ pgstat_count_buffer_hit(operation->rel);
+
+ pgstat_count_io_op(io_object, io_context, IOOP_HIT, 1, 0);
+
+ if (VacuumCostActive)
+ VacuumCostBalance += VacuumCostPageHit;
+ }
+ else
{
- int io_buffers_len;
- Buffer io_buffers[MAX_IO_COMBINE_LIMIT];
- void *io_pages[MAX_IO_COMBINE_LIMIT];
instr_time io_start;
- BlockNumber io_first_block;
-
- /*
- * Skip this block if someone else has already completed it. If an
- * I/O is already in progress in another backend, this will wait for
- * the outcome: either done, or something went wrong and we will
- * retry.
- */
- if (!WaitReadBuffersCanStartIO(buffers[i], false))
- {
- /*
- * Report and track this as a 'hit' for this backend, even though
- * it must have started out as a miss in PinBufferForBlock(). The
- * other backend will track this as a 'read'.
- */
- TRACE_POSTGRESQL_BUFFER_READ_DONE(forknum, blocknum + i,
- operation->smgr->smgr_rlocator.locator.spcOid,
- operation->smgr->smgr_rlocator.locator.dbOid,
- operation->smgr->smgr_rlocator.locator.relNumber,
- operation->smgr->smgr_rlocator.backend,
- true);
-
- if (persistence == RELPERSISTENCE_TEMP)
- pgBufferUsage.local_blks_hit += 1;
- else
- pgBufferUsage.shared_blks_hit += 1;
-
- if (operation->rel)
- pgstat_count_buffer_hit(operation->rel);
-
- pgstat_count_io_op(io_object, io_context, IOOP_HIT, 1, 0);
-
- if (VacuumCostActive)
- VacuumCostBalance += VacuumCostPageHit;
-
- continue;
- }
/* We found a buffer that we need to read in. */
- io_buffers[0] = buffers[i];
- io_pages[0] = BufferGetBlock(buffers[i]);
- io_first_block = blocknum + i;
+ Assert(io_buffers[0] == buffers[nblocks_done]);
+ io_pages[0] = BufferGetBlock(buffers[nblocks_done]);
io_buffers_len = 1;
/*
* How many neighboring-on-disk blocks can we scatter-read into other
* buffers at the same time? In this case we don't wait if we see an
- * I/O already in progress. We already hold BM_IO_IN_PROGRESS for the
+ * I/O already in progress. We already set BM_IO_IN_PROGRESS for the
* head block, so we should get on with that I/O as soon as possible.
- * We'll come back to this block again, above.
*/
- while ((i + 1) < nblocks &&
- WaitReadBuffersCanStartIO(buffers[i + 1], true))
+ for (int i = nblocks_done + 1; i < operation->nblocks; i++)
{
+ if (!ReadBuffersCanStartIO(buffers[i], true))
+ break;
/* Must be consecutive block numbers. */
- Assert(BufferGetBlockNumber(buffers[i + 1]) ==
- BufferGetBlockNumber(buffers[i]) + 1);
+ Assert(BufferGetBlockNumber(buffers[i - 1]) ==
+ BufferGetBlockNumber(buffers[i]) - 1);
+ Assert(io_buffers[io_buffers_len] == buffers[i]);
- io_buffers[io_buffers_len] = buffers[++i];
io_pages[io_buffers_len++] = BufferGetBlock(buffers[i]);
}
+ /* get a reference to wait for in WaitReadBuffers() */
+ pgaio_io_get_wref(ioh, &operation->io_wref);
+
+ /* provide the list of buffers to the completion callbacks */
+ pgaio_io_set_handle_data_32(ioh, (uint32 *) io_buffers, io_buffers_len);
+
+ pgaio_io_register_callbacks(ioh,
+ persistence == RELPERSISTENCE_TEMP ?
+ PGAIO_HCB_LOCAL_BUFFER_READV :
+ PGAIO_HCB_SHARED_BUFFER_READV,
+ flags);
+
+ pgaio_io_set_flag(ioh, ioh_flags);
+
+ /* ---
+ * Even though we're trying to issue IO asynchronously, track the time
+ * in smgrstartreadv():
+ * - if io_method == IOMETHOD_SYNC, we will always perform the IO
+ * immediately
+ * - the io method might not support the IO (e.g. worker IO for a temp
+ * table)
+ * ---
+ */
io_start = pgstat_prepare_io_time(track_io_timing);
- smgrreadv(operation->smgr, forknum, io_first_block, io_pages, io_buffers_len);
- pgstat_count_io_op_time(io_object, io_context, IOOP_READ, io_start,
- 1, io_buffers_len * BLCKSZ);
-
- /* Verify each block we read, and terminate the I/O. */
- for (int j = 0; j < io_buffers_len; ++j)
- {
- BufferDesc *bufHdr;
- Block bufBlock;
-
- if (persistence == RELPERSISTENCE_TEMP)
- {
- bufHdr = GetLocalBufferDescriptor(-io_buffers[j] - 1);
- bufBlock = LocalBufHdrGetBlock(bufHdr);
- }
- else
- {
- bufHdr = GetBufferDescriptor(io_buffers[j] - 1);
- bufBlock = BufHdrGetBlock(bufHdr);
- }
-
- /* check for garbage data */
- if (!PageIsVerifiedExtended((Page) bufBlock, io_first_block + j,
- PIV_LOG_WARNING | PIV_REPORT_STAT))
- {
- if ((operation->flags & READ_BUFFERS_ZERO_ON_ERROR) || zero_damaged_pages)
- {
- ereport(WARNING,
- (errcode(ERRCODE_DATA_CORRUPTED),
- errmsg("invalid page in block %u of relation %s; zeroing out page",
- io_first_block + j,
- relpath(operation->smgr->smgr_rlocator, forknum).str)));
- memset(bufBlock, 0, BLCKSZ);
- }
- else
- ereport(ERROR,
- (errcode(ERRCODE_DATA_CORRUPTED),
- errmsg("invalid page in block %u of relation %s",
- io_first_block + j,
- relpath(operation->smgr->smgr_rlocator, forknum).str)));
- }
-
- /* Set BM_VALID, terminate IO, and wake up any waiters */
- if (persistence == RELPERSISTENCE_TEMP)
- TerminateLocalBufferIO(bufHdr, false, BM_VALID, false);
- else
- TerminateBufferIO(bufHdr, false, BM_VALID, true, false);
-
- /* Report I/Os as completing individually. */
- TRACE_POSTGRESQL_BUFFER_READ_DONE(forknum, io_first_block + j,
- operation->smgr->smgr_rlocator.locator.spcOid,
- operation->smgr->smgr_rlocator.locator.dbOid,
- operation->smgr->smgr_rlocator.locator.relNumber,
- operation->smgr->smgr_rlocator.backend,
- false);
- }
+ smgrstartreadv(ioh, operation->smgr, forknum,
+ blocknum + nblocks_done,
+ io_pages, io_buffers_len);
+ pgstat_count_io_op_time(io_object, io_context, IOOP_READ,
+ io_start, 1, io_buffers_len * BLCKSZ);
if (persistence == RELPERSISTENCE_TEMP)
pgBufferUsage.local_blks_read += io_buffers_len;
else
pgBufferUsage.shared_blks_read += io_buffers_len;
+ /*
+ * Track vacuum cost when issuing IO, not after waiting for it.
+ * Otherwise we could end up issuing a lot of IO in a short timespan,
+ * despite a low cost limit.
+ */
if (VacuumCostActive)
VacuumCostBalance += VacuumCostPageMiss * io_buffers_len;
+
+ *nblocks_progress = io_buffers_len;
+ did_start_io = true;
}
+
+ return did_start_io;
}
/*
--
2.48.1.76.g4e746b1a31.dirty
[text/x-diff] v2.13-0010-squash-later-bufmgr-Use-AIO-in-StartReadBuffer.patch (2.7K, ../../o2dtdrlc2xqpjycsfuvv7u6zyrqj7jfxqu32et3lyelkwce2jy@3k3fwdiiu73v/11-v2.13-0010-squash-later-bufmgr-Use-AIO-in-StartReadBuffer.patch)
download | inline diff:
From 7775702ff10063bd58919937682523a1fe031db4 Mon Sep 17 00:00:00 2001
From: Andres Freund <andres@anarazel.de>
Date: Wed, 26 Mar 2025 16:41:21 -0400
Subject: [PATCH v2.13 10/28] squash-later: bufmgr: Use AIO in
StartReadBuffers()
---
src/backend/storage/buffer/bufmgr.c | 26 ++++++++++++++++++++------
1 file changed, 20 insertions(+), 6 deletions(-)
diff --git a/src/backend/storage/buffer/bufmgr.c b/src/backend/storage/buffer/bufmgr.c
index 83dda2cb5d8..040e7cef9e8 100644
--- a/src/backend/storage/buffer/bufmgr.c
+++ b/src/backend/storage/buffer/bufmgr.c
@@ -1595,6 +1595,7 @@ static void
ProcessReadBuffersResult(ReadBuffersOperation *operation)
{
PgAioReturn *aio_ret = &operation->io_return;
+ PgAioResultStatus rs = aio_ret->result.status;
int newly_read_blocks = 0;
Assert(pgaio_wref_valid(&operation->io_wref));
@@ -1605,8 +1606,12 @@ ProcessReadBuffersResult(ReadBuffersOperation *operation)
* the IO operation. Thus we can simply add that to ->nblocks_done.
*/
- if (likely(aio_ret->result.status == PGAIO_RS_OK))
+ if (likely(rs != PGAIO_RS_ERROR))
newly_read_blocks = aio_ret->result.result;
+
+ if (rs == PGAIO_RS_ERROR || rs == PGAIO_RS_WARNING)
+ pgaio_result_report(aio_ret->result, &aio_ret->target_data,
+ rs == PGAIO_RS_ERROR ? ERROR : WARNING);
else if (aio_ret->result.status == PGAIO_RS_PARTIAL)
{
/*
@@ -1614,13 +1619,8 @@ ProcessReadBuffersResult(ReadBuffersOperation *operation)
* not even that in prod scenarios).
*/
pgaio_result_report(aio_ret->result, &aio_ret->target_data, DEBUG1);
- newly_read_blocks = aio_ret->result.result;
-
elog(DEBUG3, "partial read, will retry");
-
}
- else if (aio_ret->result.status == PGAIO_RS_ERROR)
- pgaio_result_report(aio_ret->result, &aio_ret->target_data, ERROR);
Assert(newly_read_blocks > 0);
Assert(newly_read_blocks <= MAX_IO_COMBINE_LIMIT);
@@ -1800,6 +1800,20 @@ AsyncReadBuffers(ReadBuffersOperation *operation, int *nblocks_progress)
io_object = IOOBJECT_RELATION;
}
+ /*
+ * If zero_damaged_pages is enabled, add the READ_BUFFERS_ZERO_ON_ERROR
+ * flag. The reason for that is that, hopefully, zero_damaged_pages isn't
+ * set globally, but on a per-session basis. The completion callback,
+ * which may be run in other processes, e.g. in IO workers, may have a
+ * different value of the zero_damaged_pages GUC.
+ *
+ * XXX: We probably should eventually use a different flag for
+ * zero_damaged_pages, so we can report different log levels / error codes
+ * for zero_damaged_pages and ZERO_ON_ERROR.
+ */
+ if (zero_damaged_pages)
+ flags |= READ_BUFFERS_ZERO_ON_ERROR;
+
/*
* Get IO handle before ReadBuffersCanStartIO(), as pgaio_io_acquire()
* might block, which we don't want after setting IO_IN_PROGRESS.
--
2.48.1.76.g4e746b1a31.dirty
[text/x-diff] v2.13-0011-aio-Add-README.md-explaining-higher-level-desi.patch (19.0K, ../../o2dtdrlc2xqpjycsfuvv7u6zyrqj7jfxqu32et3lyelkwce2jy@3k3fwdiiu73v/12-v2.13-0011-aio-Add-README.md-explaining-higher-level-desi.patch)
download | inline diff:
From 969f6e5478a46812b42ff6f0dffa16028e71b01e Mon Sep 17 00:00:00 2001
From: Andres Freund <andres@anarazel.de>
Date: Tue, 18 Mar 2025 14:40:06 -0400
Subject: [PATCH v2.13 11/28] aio: Add README.md explaining higher level design
Reviewed-by: Noah Misch <noah@leadboat.com>
Reviewed-by: Heikki Linnakangas <hlinnaka@iki.fi>
Discussion: https://postgr.es/m/uvrtrknj4kdytuboidbhwclo4gxhswwcpgadptsjvjqcluzmah%40brqs62irg4dt
Discussion: https://postgr.es/m/20210223100344.llw5an2aklengrmn@alap3.anarazel.de
Discussion: https://postgr.es/m/stj36ea6yyhoxtqkhpieia2z4krnam7qyetc57rfezgk4zgapf@gcnactj4z56m
---
src/backend/storage/aio/README.md | 423 ++++++++++++++++++++++++++++++
src/backend/storage/aio/aio.c | 2 +
2 files changed, 425 insertions(+)
create mode 100644 src/backend/storage/aio/README.md
diff --git a/src/backend/storage/aio/README.md b/src/backend/storage/aio/README.md
new file mode 100644
index 00000000000..3bdf003a03c
--- /dev/null
+++ b/src/backend/storage/aio/README.md
@@ -0,0 +1,423 @@
+# Asynchronous & Direct IO
+
+## Motivation
+
+### Why Asynchronous IO
+
+Until the introduction of asynchronous IO Postgres relied on the operating
+system to hide the cost of synchronous IO from Postgres. While this worked
+surprisingly well in a lot of workloads, it does not do as good a job on
+prefetching and controlled writeback as we would like.
+
+There are important expensive operations like `fdatasync()` where the operating
+system cannot hide the storage latency. This is particularly important for WAL
+writes, where the ability to asynchronously issue `fdatasync()` or O_DSYNC
+writes can yield significantly higher throughput.
+
+
+### Why Direct / unbuffered IO
+
+The main reasons to want to use Direct IO are:
+
+- Lower CPU usage / higher throughput. Particularly on modern storage buffered
+ writes are bottlenecked by the operating system having to copy data from the
+ kernel's page cache to postgres buffer pool using the CPU. Whereas direct IO
+ can often move the data directly between the storage devices and postgres'
+ buffer cache, using DMA. While that transfer is ongoing, the CPU is free to
+ perform other work.
+- Reduced latency - Direct IO can have substantially lower latency than
+ buffered IO, which can be impactful for OLTP workloads bottlenecked by WAL
+ write latency.
+- Avoiding double buffering between operating system cache and postgres'
+ shared_buffers.
+- Better control over the timing and pace of dirty data writeback.
+
+
+The main reasons *not* to use Direct IO are:
+
+- Without AIO, Direct IO is unusably slow for most purposes.
+- Even with AIO, many parts of postgres need to be modified to perform
+ explicit prefetching.
+- In situations where shared_buffers cannot be set appropriately large,
+ e.g. because there are many different postgres instances hosted on shared
+ hardware, performance will often be worse than when using buffered IO.
+
+
+## AIO Usage Example
+
+In many cases code that can benefit from AIO does not directly have to
+interact with the AIO interface, but can use AIO via higher-level
+abstractions. See [Helpers](#helpers).
+
+In this example, a buffer will be read into shared buffers.
+
+```C
+/*
+ * Result of the operation, only to be accessed in this backend.
+ */
+PgAioReturn ioret;
+
+/*
+ * Acquire an AIO Handle, ioret will get result upon completion.
+ *
+ * Note that ioret needs to stay alive until the IO completes or
+ * CurrentResourceOwner is released (i.e. an error is thrown).
+ */
+PgAioHandle *ioh = pgaio_io_acquire(CurrentResourceOwner, &ioret);
+
+/*
+ * Reference that can be used to wait for the IO we initiate below. This
+ * reference can reside in local or shared memory and waited upon by any
+ * process. An arbitrary number of references can be made for each IO.
+ */
+PgAioWaitRef iow;
+
+pgaio_io_get_wref(ioh, &iow);
+
+/*
+ * Arrange for shared buffer completion callbacks to be called upon completion
+ * of the IO. This callback will update the buffer descriptors associated with
+ * the AioHandle, which e.g. allows other backends to access the buffer.
+ *
+ * A callback can be passed a small bit of data, e.g. to indicate whether to
+ * zero a buffer if it is invalid.
+ *
+ * Multiple completion callbacks can be registered for each handle.
+ */
+pgaio_io_register_callbacks(ioh, PGAIO_HCB_SHARED_BUFFER_READV, 0);
+
+/*
+ * The completion callback needs to know which buffers to update when the IO
+ * completes. As the AIO subsystem does not know about buffers, we have to
+ * associate this information with the AioHandle, for use by the completion
+ * callback registered above.
+ *
+ * In this example we're reading only a single buffer, hence the 1.
+ */
+pgaio_io_set_handle_data_32(ioh, (uint32 *) buffer, 1);
+
+/*
+ * Pass the AIO handle to lower-level function. When operating on the level of
+ * buffers, we don't know how exactly the IO is performed, that is the
+ * responsibility of the storage manager implementation.
+ *
+ * E.g. md.c needs to translate block numbers into offsets in segments.
+ *
+ * Once the IO handle has been handed off to smgstartreadv(), it may not
+ * further be used, as the IO may immediately get executed below
+ * smgrstartreadv() and the handle reused for another IO.
+ *
+ * To issue multiple IOs in an efficient way, a caller can call
+ * pgaio_enter_batchmode() before starting multiple IOs, and end that batch
+ * with pgaio_exit_batchmode(). Note that one needs to be careful while there
+ * may be unsubmitted IOs, as another backend may need to wait for one of the
+ * unsubmitted IOs. If this backend then had to wait for the other backend,
+ * it'd end in an undetected deadlock. See pgaio_enter_batchmode() for more
+ * details.
+ *
+ * Note that even while in batchmode an IO might get submitted immediately,
+ * e.g. due to reaching a limit on the number of unsubmitted IOs, and even
+ * complete before smgrstartreadv() returns.
+ */
+smgrstartreadv(ioh, operation->smgr, forknum, blkno,
+ BufferGetBlock(buffer), 1);
+
+/*
+ * To benefit from AIO, it is beneficial to perform other work, including
+ * submitting other IOs, before waiting for the IO to complete. Otherwise
+ * we could just have used synchronous, blocking IO.
+ */
+perform_other_work();
+
+/*
+ * We did some other work and now need the IO operation to have completed to
+ * continue.
+ */
+pgaio_wref_wait(&iow);
+
+/*
+ * At this point the IO has completed. We do not yet know whether it succeeded
+ * or failed, however. The buffer's state has been updated, which allows other
+ * backends to use the buffer (if the IO succeeded), or retry the IO (if it
+ * failed).
+ *
+ * Note that in case the IO has failed, a LOG message may have been emitted,
+ * but no ERROR has been raised. This is crucial, as another backend waiting
+ * for this IO should not see an ERROR.
+ *
+ * To check whether the operation succeeded, and to raise an ERROR, or if more
+ * appropriate LOG, the PgAioReturn we passed to pgaio_io_acquire() is used.
+ */
+if (ioret.result.status == PGAIO_RS_ERROR)
+ pgaio_result_report(ioret.result, &ioret.target_data, ERROR);
+
+/*
+ * Besides having succeeded completely, the IO could also have partially
+ * completed. If we e.g. tried to read many blocks at once, the read might have
+ * only succeeded for the first few blocks.
+ *
+ * If the IO partially succeeded and this backend needs all blocks to have
+ * completed, this backend needs to reissue the IO for the remaining buffers.
+ * The AIO subsystem cannot handle this retry transparently.
+ *
+ * As this example is already long, and we only read a single block, we'll just
+ * error out if there's a partial read.
+ */
+if (ioret.result.status == PGAIO_RS_PARTIAL)
+ pgaio_result_report(ioret.result, &ioret.target_data, ERROR);
+
+/*
+ * The IO succeeded, so we can use the buffer now.
+ */
+```
+
+
+## Design Criteria & Motivation
+
+### Deadlock and Starvation Dangers due to AIO
+
+Using AIO in a naive way can easily lead to deadlocks in an environment where
+the source/target of AIO are shared resources, like pages in postgres'
+shared_buffers.
+
+Consider one backend performing readahead on a table, initiating IO for a
+number of buffers ahead of the current "scan position". If that backend then
+performs some operation that blocks, or even just is slow, the IO completion
+for the asynchronously initiated read may not be processed.
+
+This AIO implementation solves this problem by requiring that AIO methods
+either allow AIO completions to be processed by any backend in the system
+(e.g. io_uring), or to guarantee that AIO processing will happen even when the
+issuing backend is blocked (e.g. worker mode, which offloads completion
+processing to the AIO workers).
+
+
+### IO can be started in critical sections
+
+Using AIO for WAL writes can reduce the overhead of WAL logging substantially:
+
+- AIO allows to start WAL writes eagerly, so they complete before needing to
+ wait
+- AIO allows to have multiple WAL flushes in progress at the same time
+- AIO makes it more realistic to use O\_DIRECT + O\_DSYNC, which can reduce
+ the number of roundtrips to storage on some OSs and storage HW (buffered IO
+ and direct IO without O_DSYNC needs to issue a write and after the write's
+ completion a cache flush, whereas O\_DIRECT + O\_DSYNC can use a single
+ Force Unit Access (FUA) write).
+
+The need to be able to execute IO in critical sections has substantial design
+implication on the AIO subsystem. Mainly because completing IOs (see prior
+section) needs to be possible within a critical section, even if the
+to-be-completed IO itself was not issued in a critical section. Consider
+e.g. the case of a backend first starting a number of writes from shared
+buffers and then starting to flush the WAL. Because only a limited amount of
+IO can be in-progress at the same time, initiating IO for flushing the WAL may
+require to first complete IO that was started earlier.
+
+
+### State for AIO needs to live in shared memory
+
+Because postgres uses a process model and because AIOs need to be
+complete-able by any backend much of the state of the AIO subsystem needs to
+live in shared memory.
+
+In an `EXEC_BACKEND` build, a backend's executable code and other process
+local state is not necessarily mapped to the same addresses in each process
+due to ASLR. This means that the shared memory cannot contain pointers to
+callbacks.
+
+
+## Design of the AIO Subsystem
+
+
+### AIO Methods
+
+To achieve portability and performance, multiple methods of performing AIO are
+implemented and others are likely worth adding in the future.
+
+
+#### Synchronous Mode
+
+`io_method=sync` does not actually perform AIO but allows to use the AIO API
+while performing synchronous IO. This can be useful for debugging. The code
+for the synchronous mode is also used as a fallback by e.g. the [worker
+mode](#worker) uses it to execute IO that cannot be executed by workers.
+
+
+#### Worker
+
+`io_method=worker` is available on every platform postgres runs on, and
+implements asynchronous IO - from the view of the issuing process - by
+dispatching the IO to one of several worker processes performing the IO in a
+synchronous manner.
+
+
+#### io_uring
+
+`io_method=io_uring` is available on Linux 5.1+. In contrast to worker mode it
+dispatches all IO from within the process, lowering context switch rate /
+latency.
+
+
+### AIO Handles
+
+The central API piece for postgres' AIO abstraction are AIO handles. To
+execute an IO one first has to acquire an IO handle (`pgaio_io_acquire()`) and
+then "define" it, i.e. associate an IO operation with the handle.
+
+Often AIO handles are acquired on a higher level and then passed to a lower
+level to be fully defined. E.g., for IO to/from shared buffers, bufmgr.c
+routines acquire the handle, which is then passed through smgr.c, md.c to be
+finally fully defined in fd.c.
+
+The functions used at the lowest level to define the operation are
+`pgaio_io_start_*()`.
+
+Because acquisition of an IO handle
+[must always succeed](#io-can-be-started-in-critical-sections)
+and the number of AIO Handles
+[has to be limited](#state-for-aio-needs-to-live-in-shared-memory)
+AIO handles can be reused as soon as they have completed. Obviously code needs
+to be able to react to IO completion. State can be updated using
+[AIO Completion callbacks](#aio-callbacks)
+and the issuing backend can provide a backend local variable to receive the
+result of the IO, as described in
+[AIO Result](#aio-results).
+An IO can be waited for, by both the issuing and any other backend, using
+[AIO References](#aio-wait-references).
+
+
+Because an AIO Handle is not executable just after calling
+`pgaio_io_acquire()` and because `pgaio_io_acquire()` needs to always succeed
+(absent a PANIC), only a single AIO Handle may be acquired (i.e. returned by
+`pgaio_io_acquire()`) without causing the IO to have been defined (by,
+potentially indirectly, causing `pgaio_io_start_*()` to have been
+called). Otherwise a backend could trivially self-deadlock by using up all AIO
+Handles without the ability to wait for some of the IOs to complete.
+
+If it turns out that an AIO Handle is not needed, e.g., because the handle was
+acquired before holding a contended lock, it can be released without being
+defined using `pgaio_io_release()`.
+
+
+### AIO Callbacks
+
+Commonly several layers need to react to completion of an IO. E.g. for a read
+md.c needs to check if the IO outright failed or was shorter than needed,
+bufmgr.c needs to verify the page looks valid and bufmgr.c needs to update the
+BufferDesc to update the buffer's state.
+
+The fact that several layers / subsystems need to react to IO completion poses
+a few challenges:
+
+- Upper layers should not need to know details of lower layers. E.g. bufmgr.c
+ should not assume the IO will pass through md.c. Therefore upper levels
+ cannot know what lower layers would consider an error.
+
+- Lower layers should not need to know about upper layers. E.g. smgr APIs are
+ used going through shared buffers but are also used bypassing shared
+ buffers. This means that e.g. md.c is not in a position to validate
+ checksums.
+
+- Having code in the AIO subsystem for every possible combination of layers
+ would lead to a lot of duplication.
+
+The "solution" to this is the ability to associate multiple completion
+callbacks with a handle. E.g. bufmgr.c can have a callback to update the
+BufferDesc state and to verify the page and md.c can have another callback to
+check if the IO operation was successful.
+
+As [mentioned](#state-for-aio-needs-to-live-in-shared-memory), shared memory
+currently cannot contain function pointers. Because of that completion
+callbacks are not directly identified by function pointers but by IDs
+(`PgAioHandleCallbackID`). A substantial added benefit is that that
+allows callbacks to be identified by much smaller amount of memory (a single
+byte currently).
+
+In addition to completion, AIO callbacks also are called to "stage" an
+IO. This is, e.g., used to increase buffer reference counts to account for the
+AIO subsystem referencing the buffer, which is required to handle the case
+where the issuing backend errors out and releases its own pins while the IO is
+still ongoing.
+
+As [explained earlier](#io-can-be-started-in-critical-sections) IO completions
+need to be safe to execute in critical sections. To allow the backend that
+issued the IO to error out in case of failure [AIO Result](#aio-results) can
+be used.
+
+
+### AIO Targets
+
+In addition to the completion callbacks describe above, each AIO Handle has
+exactly one "target". Each target has some space inside an AIO Handle with
+information specific to the target and can provide callbacks to allow to
+reopen the underlying file (required for worker mode) and to describe the IO
+operation (used for debug logging and error messages).
+
+I.e., if two different uses of AIO can describe the identity of the file being
+operated on the same way, it likely makes sense to use the same
+target. E.g. different smgr implementations can describe IO with
+RelFileLocator, ForkNumber and BlockNumber and can thus share a target. In
+contrast, IO for a WAL file would be described with TimeLineID and XLogRecPtr
+and it would not make sense to use the same target for smgr and WAL.
+
+
+### AIO Wait References
+
+As [described above](#aio-handles), AIO Handles can be reused immediately
+after completion and therefore cannot be used to wait for completion of the
+IO. Waiting is enabled using AIO wait references, which do not just identify
+an AIO Handle but also include the handles "generation".
+
+A reference to an AIO Handle can be acquired using `pgaio_io_get_wref()` and
+then waited upon using `pgaio_wref_wait()`.
+
+
+### AIO Results
+
+As AIO completion callbacks
+[are executed in critical sections](#io-can-be-started-in-critical-sections)
+and [may be executed by any backend](#deadlock-and-starvation-dangers-due-to-aio)
+completion callbacks cannot be used to, e.g., make the query that triggered an
+IO ERROR out.
+
+To allow to react to failing IOs the issuing backend can pass a pointer to a
+`PgAioReturn` in backend local memory. Before an AIO Handle is reused the
+`PgAioReturn` is filled with information about the IO. This includes
+information about whether the IO was successful (as a value of
+`PgAioResultStatus`) and enough information to raise an error in case of a
+failure (via `pgaio_result_report()`, with the error details encoded in
+`PgAioResult`).
+
+
+### AIO Errors
+
+It would be very convenient to have shared completion callbacks encode the
+details of errors as an `ErrorData` that could be raised at a later
+time. Unfortunately doing so would require allocating memory. While elog.c can
+guarantee (well, kinda) that logging a message will not run out of memory,
+that only works because a very limited number of messages are in the process
+of being logged. With AIO a large number of concurrently issued AIOs might
+fail.
+
+To avoid the need for preallocating a potentially large amount of memory (in
+shared memory no less!), completion callbacks instead have to encode errors in
+a more compact format that can be converted into an error message.
+
+
+## Helpers
+
+Using the low-level AIO API introduces too much complexity to do so all over
+the tree. Most uses of AIO should be done via reusable, higher-level,
+helpers.
+
+
+### Read Stream
+
+A common and very beneficial use of AIO are reads where a substantial number
+of to-be-read locations are known ahead of time. E.g., for a sequential scan
+the set of blocks that need to be read can be determined solely by knowing the
+current position and checking the buffer mapping table.
+
+The [Read Stream](../../../include/storage/read_stream.h) interface makes it
+comparatively easy to use AIO for such use cases.
diff --git a/src/backend/storage/aio/aio.c b/src/backend/storage/aio/aio.c
index e3ed087e8a2..86f7250b7a5 100644
--- a/src/backend/storage/aio/aio.c
+++ b/src/backend/storage/aio/aio.c
@@ -24,6 +24,8 @@
*
* - read_stream.c - helper for reading buffered relation data
*
+ * - README.md - higher-level overview over AIO
+ *
*
* Portions Copyright (c) 1996-2025, PostgreSQL Global Development Group
* Portions Copyright (c) 1994, Regents of the University of California
--
2.48.1.76.g4e746b1a31.dirty
[text/x-diff] v2.13-0012-squash-later-aio-Add-README.md-explaining-high.patch (1.7K, ../../o2dtdrlc2xqpjycsfuvv7u6zyrqj7jfxqu32et3lyelkwce2jy@3k3fwdiiu73v/13-v2.13-0012-squash-later-aio-Add-README.md-explaining-high.patch)
download | inline diff:
From 011011f9d34405f3882e429a743d6ac6c9ba7d89 Mon Sep 17 00:00:00 2001
From: Andres Freund <andres@anarazel.de>
Date: Wed, 26 Mar 2025 14:38:53 -0400
Subject: [PATCH v2.13 12/28] squash-later: aio: Add README.md explaining
higher level design
---
src/backend/storage/aio/README.md | 9 +++++----
1 file changed, 5 insertions(+), 4 deletions(-)
diff --git a/src/backend/storage/aio/README.md b/src/backend/storage/aio/README.md
index 3bdf003a03c..ddd59404a59 100644
--- a/src/backend/storage/aio/README.md
+++ b/src/backend/storage/aio/README.md
@@ -152,8 +152,9 @@ if (ioret.result.status == PGAIO_RS_ERROR)
pgaio_result_report(ioret.result, &ioret.target_data, ERROR);
/*
- * Besides having succeeded completely, the IO could also have partially
- * completed. If we e.g. tried to read many blocks at once, the read might have
+ * Besides having succeeded completely, the IO could also have a) partially
+ * completed or b) succeeded with a warning (e.g. due to zero_damaged_pages).
+ * If we e.g. tried to read many blocks at once, the read might have
* only succeeded for the first few blocks.
*
* If the IO partially succeeded and this backend needs all blocks to have
@@ -161,9 +162,9 @@ if (ioret.result.status == PGAIO_RS_ERROR)
* The AIO subsystem cannot handle this retry transparently.
*
* As this example is already long, and we only read a single block, we'll just
- * error out if there's a partial read.
+ * error out if there's a partial read or a warning.
*/
-if (ioret.result.status == PGAIO_RS_PARTIAL)
+if (ioret.result.status != PGAIO_RS_OK)
pgaio_result_report(ioret.result, &ioret.target_data, ERROR);
/*
--
2.48.1.76.g4e746b1a31.dirty
[text/x-diff] v2.13-0013-aio-Basic-read_stream-adjustments-for-real-AIO.patch (4.9K, ../../o2dtdrlc2xqpjycsfuvv7u6zyrqj7jfxqu32et3lyelkwce2jy@3k3fwdiiu73v/14-v2.13-0013-aio-Basic-read_stream-adjustments-for-real-AIO.patch)
download | inline diff:
From 03f43a541feaa0149e24cd529c343f1b02dce12a Mon Sep 17 00:00:00 2001
From: Andres Freund <andres@anarazel.de>
Date: Tue, 18 Mar 2025 14:40:06 -0400
Subject: [PATCH v2.13 13/28] aio: Basic read_stream adjustments for real AIO
Adapt the read stream logic for real AIO:
- If AIO is enabled, we shouldn't issue advice, but if it isn't, we should
continue issuing advice
- AIO benefits from reading ahead with direct IO
- If effective_io_concurrency=0, pass READ_BUFFERS_SYNCHRONOUSLY to
StartReadBuffers() to ensure synchronous IO execution
There are further improvements we should consider:
- While in read_stream_look_ahead(), we can use AIO batch submission mode for
increased efficiency. That however requires care to avoid deadlocks and thus
done separately.
- It can be beneficial to defer starting new IOs until we can issue multiple
IOs at once. That however requires non-trivial heuristics to decide when to
do so.
Co-authored-by: Andres Freund <andres@anarazel.de>
Co-authored-by: Thomas Munro <thomas.munro@gmail.com>
---
src/backend/storage/aio/read_stream.c | 39 ++++++++++++++++++---------
1 file changed, 26 insertions(+), 13 deletions(-)
diff --git a/src/backend/storage/aio/read_stream.c b/src/backend/storage/aio/read_stream.c
index c60e37e7f7f..26e5dfe77db 100644
--- a/src/backend/storage/aio/read_stream.c
+++ b/src/backend/storage/aio/read_stream.c
@@ -72,6 +72,7 @@
#include "postgres.h"
#include "miscadmin.h"
+#include "storage/aio.h"
#include "storage/fd.h"
#include "storage/smgr.h"
#include "storage/read_stream.h"
@@ -99,6 +100,8 @@ struct ReadStream
int16 pinned_buffers;
int16 distance;
int16 initialized_buffers;
+ int read_buffers_flags;
+ bool sync_mode; /* using io_method=sync */
bool advice_enabled;
bool temporary;
@@ -250,7 +253,7 @@ read_stream_start_pending_read(ReadStream *stream)
Assert(stream->next_buffer_index == stream->oldest_buffer_index);
/* Do we need to issue read-ahead advice? */
- flags = 0;
+ flags = stream->read_buffers_flags;
if (stream->advice_enabled)
{
if (stream->pending_read_blocknum == stream->seq_blocknum)
@@ -261,7 +264,7 @@ read_stream_start_pending_read(ReadStream *stream)
* then stay of the way of the kernel's own read-ahead.
*/
if (stream->seq_until_processed != InvalidBlockNumber)
- flags = READ_BUFFERS_ISSUE_ADVICE;
+ flags |= READ_BUFFERS_ISSUE_ADVICE;
}
else
{
@@ -272,7 +275,7 @@ read_stream_start_pending_read(ReadStream *stream)
*/
stream->seq_until_processed = stream->pending_read_blocknum;
if (stream->pinned_buffers > 0)
- flags = READ_BUFFERS_ISSUE_ADVICE;
+ flags |= READ_BUFFERS_ISSUE_ADVICE;
}
}
@@ -613,27 +616,33 @@ read_stream_begin_impl(int flags,
stream->per_buffer_data = (void *)
MAXALIGN(&stream->ios[Max(1, max_ios)]);
+ stream->sync_mode = io_method == IOMETHOD_SYNC;
+
#ifdef USE_PREFETCH
/*
- * This system supports prefetching advice. We can use it as long as
- * direct I/O isn't enabled, the caller hasn't promised sequential access
- * (overriding our detection heuristics), and max_ios hasn't been set to
- * zero.
+ * Read-ahead advice simulating asynchronous I/O with synchronous calls.
+ * Issue advice only if AIO is not used, direct I/O isn't enabled, the
+ * caller hasn't promised sequential access (overriding our detection
+ * heuristics), and max_ios hasn't been set to zero.
*/
- if ((io_direct_flags & IO_DIRECT_DATA) == 0 &&
+ if (stream->sync_mode &&
+ (io_direct_flags & IO_DIRECT_DATA) == 0 &&
(flags & READ_STREAM_SEQUENTIAL) == 0 &&
max_ios > 0)
stream->advice_enabled = true;
#endif
/*
- * For now, max_ios = 0 is interpreted as max_ios = 1 with advice disabled
- * above. If we had real asynchronous I/O we might need a slightly
- * different definition.
+ * Setting max_ios to zero disables AIO and advice-based pseudo AIO, but
+ * we still need to allocate space to combine and run one I/O. Bump it up
+ * to one, and remember to ask for synchronous I/O only.
*/
if (max_ios == 0)
+ {
max_ios = 1;
+ stream->read_buffers_flags = READ_BUFFERS_SYNCHRONOUSLY;
+ }
/*
* Capture stable values for these two GUC-derived numbers for the
@@ -777,6 +786,11 @@ read_stream_next_buffer(ReadStream *stream, void **per_buffer_data)
if (likely(next_blocknum != InvalidBlockNumber))
{
+ int flags = stream->read_buffers_flags;
+
+ if (stream->advice_enabled)
+ flags |= READ_BUFFERS_ISSUE_ADVICE;
+
/*
* Pin a buffer for the next call. Same buffer entry, and
* arbitrary I/O entry (they're all free). We don't have to
@@ -792,8 +806,7 @@ read_stream_next_buffer(ReadStream *stream, void **per_buffer_data)
if (likely(!StartReadBuffer(&stream->ios[0].op,
&stream->buffers[oldest_buffer_index],
next_blocknum,
- stream->advice_enabled ?
- READ_BUFFERS_ISSUE_ADVICE : 0)))
+ flags)))
{
/* Fast return. */
return buffer;
--
2.48.1.76.g4e746b1a31.dirty
[text/x-diff] v2.13-0014-read_stream-Introduce-and-use-optional-batchmo.patch (13.1K, ../../o2dtdrlc2xqpjycsfuvv7u6zyrqj7jfxqu32et3lyelkwce2jy@3k3fwdiiu73v/15-v2.13-0014-read_stream-Introduce-and-use-optional-batchmo.patch)
download | inline diff:
From e4400c4b22d9131ac269b59e2179e73145a14fa1 Mon Sep 17 00:00:00 2001
From: Andres Freund <andres@anarazel.de>
Date: Mon, 24 Mar 2025 17:30:42 -0400
Subject: [PATCH v2.13 14/28] read_stream: Introduce and use optional batchmode
support
Submitting IO in larger batches can be more efficient than doing so
one-by-one, particularly for many small reads. It does, however, require
the ReadStreamBlockNumberCB callback to abide by the restrictions of AIO
batching (c.f. pgaio_enter_batchmode()). Basically, the callback may not:
a) block without first calling pgaio_submit_staged(), unless a
to-be-waited-on lock cannot be part of a deadlock, e.g. because it is
never held while waiting for IO.
b) directly or indirectly start another batch pgaio_enter_batchmode()
As this requires care and is nontrivial in some cases, batching is only
used with explicit opt-in.
This patch adds an explicit flag (READ_STREAM_USE_BATCHING) to read_stream and
uses it where appropriate.
There are two cases where batching would likely be beneficial, but where we
aren't using it yet:
1) bitmap heap scans, because the callback reads the VM
This should soon be solved, because we are planning to remove the use of
the VM, due to that not being sound.
2) The first phase of heap vacuum
This could be made to support batchmode, but would require some care.
Discussion: https://postgr.es/m/uvrtrknj4kdytuboidbhwclo4gxhswwcpgadptsjvjqcluzmah%40brqs62irg4dt
---
src/include/storage/read_stream.h | 21 +++++++++++++++++++++
src/backend/access/gist/gistvacuum.c | 8 +++++++-
src/backend/access/heap/heapam.c | 18 +++++++++++++++++-
src/backend/access/heap/vacuumlazy.c | 21 ++++++++++++++++++---
src/backend/access/nbtree/nbtree.c | 8 +++++++-
src/backend/access/spgist/spgvacuum.c | 8 +++++++-
src/backend/commands/analyze.c | 7 ++++++-
src/backend/storage/aio/read_stream.c | 16 ++++++++++++++++
src/backend/storage/buffer/bufmgr.c | 8 +++++++-
contrib/pg_prewarm/pg_prewarm.c | 7 ++++++-
contrib/pg_visibility/pg_visibility.c | 8 +++++++-
11 files changed, 119 insertions(+), 11 deletions(-)
diff --git a/src/include/storage/read_stream.h b/src/include/storage/read_stream.h
index c11d8ce3300..9b0d65161d0 100644
--- a/src/include/storage/read_stream.h
+++ b/src/include/storage/read_stream.h
@@ -42,6 +42,27 @@
*/
#define READ_STREAM_FULL 0x04
+/* ---
+ * Opt-in to using AIO batchmode.
+ *
+ * Submitting IO in larger batches can be more efficient than doing so
+ * one-by-one, particularly for many small reads. It does, however, require
+ * the ReadStreamBlockNumberCB callback to abide by the restrictions of AIO
+ * batching (c.f. pgaio_enter_batchmode()). Basically, the callback may not:
+ *
+ * a) block without first calling pgaio_submit_staged(), unless a
+ * to-be-waited-on lock cannot be part of a deadlock, e.g. because it is
+ * never held while waiting for IO.
+ *
+ * b) start another batch (without first exiting batchmode and re-entering
+ * before returning)
+ *
+ * As this requires care and is nontrivial in some cases, batching is only
+ * used with explicit opt-in.
+ * ---
+ */
+#define READ_STREAM_USE_BATCHING 0x08
+
struct ReadStream;
typedef struct ReadStream ReadStream;
diff --git a/src/backend/access/gist/gistvacuum.c b/src/backend/access/gist/gistvacuum.c
index 20b1bb5dbac..ce9d78d78d6 100644
--- a/src/backend/access/gist/gistvacuum.c
+++ b/src/backend/access/gist/gistvacuum.c
@@ -210,7 +210,13 @@ gistvacuumscan(IndexVacuumInfo *info, IndexBulkDeleteResult *stats,
needLock = !RELATION_IS_LOCAL(rel);
p.current_blocknum = GIST_ROOT_BLKNO;
- stream = read_stream_begin_relation(READ_STREAM_FULL,
+
+ /*
+ * It is safe to use batchmode as block_range_read_stream_cb takes no
+ * locks.
+ */
+ stream = read_stream_begin_relation(READ_STREAM_FULL |
+ READ_STREAM_USE_BATCHING,
info->strategy,
rel,
MAIN_FORKNUM,
diff --git a/src/backend/access/heap/heapam.c b/src/backend/access/heap/heapam.c
index b12b583c4d9..c8357660776 100644
--- a/src/backend/access/heap/heapam.c
+++ b/src/backend/access/heap/heapam.c
@@ -1206,7 +1206,15 @@ heap_beginscan(Relation relation, Snapshot snapshot,
else
cb = heap_scan_stream_read_next_serial;
- scan->rs_read_stream = read_stream_begin_relation(READ_STREAM_SEQUENTIAL,
+ /* ---
+ * It is safe to use batchmode as the only locks taken by `cb`
+ * are never taken while waiting for IO:
+ * - SyncScanLock is used in the non-parallel case
+ * - in the parallel case, only spinlocks and atomics are used
+ * ---
+ */
+ scan->rs_read_stream = read_stream_begin_relation(READ_STREAM_SEQUENTIAL |
+ READ_STREAM_USE_BATCHING,
scan->rs_strategy,
scan->rs_base.rs_rd,
MAIN_FORKNUM,
@@ -1216,6 +1224,14 @@ heap_beginscan(Relation relation, Snapshot snapshot,
}
else if (scan->rs_base.rs_flags & SO_TYPE_BITMAPSCAN)
{
+ /*
+ * Currently we can't trivially use batching, due to the
+ * VM_ALL_VISIBLE check in bitmapheap_stream_read_next. While that
+ * could be made safe, we are about to remove the all-visible logic
+ * from bitmap scans due to its unsoundness.
+ *
+ * FIXME: Should be changed soon!
+ */
scan->rs_read_stream = read_stream_begin_relation(READ_STREAM_DEFAULT,
scan->rs_strategy,
scan->rs_base.rs_rd,
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index 2cbcf5e5db2..82883492128 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -1225,7 +1225,12 @@ lazy_scan_heap(LVRelState *vacrel)
vacrel->next_unskippable_eager_scanned = false;
vacrel->next_unskippable_vmbuffer = InvalidBuffer;
- /* Set up the read stream for vacuum's first pass through the heap */
+ /*
+ * Set up the read stream for vacuum's first pass through the heap.
+ *
+ * This could be made safe for READ_STREAM_USE_BATCHING, but only with
+ * explicit work in heap_vac_scan_next_block.
+ */
stream = read_stream_begin_relation(READ_STREAM_MAINTENANCE,
vacrel->bstrategy,
vacrel->rel,
@@ -2669,6 +2674,8 @@ lazy_vacuum_all_indexes(LVRelState *vacrel)
* Read stream callback for vacuum's third phase (second pass over the heap).
* Gets the next block from the TID store and returns it or InvalidBlockNumber
* if there are no further blocks to vacuum.
+ *
+ * NB: Assumed to be safe to use with READ_STREAM_USE_BATCHING.
*/
static BlockNumber
vacuum_reap_lp_read_stream_next(ReadStream *stream,
@@ -2732,8 +2739,16 @@ lazy_vacuum_heap_rel(LVRelState *vacrel)
iter = TidStoreBeginIterate(vacrel->dead_items);
- /* Set up the read stream for vacuum's second pass through the heap */
- stream = read_stream_begin_relation(READ_STREAM_MAINTENANCE,
+ /*
+ * Set up the read stream for vacuum's second pass through the heap.
+ *
+ * It is safe to use batchmode, as vacuum_reap_lp_read_stream_next() does
+ * not need to wait for IO and does not perform locking. Once we support
+ * parallelism it should still be fine, as presumably the holder of locks
+ * would never be blocked by IO while holding the lock.
+ */
+ stream = read_stream_begin_relation(READ_STREAM_MAINTENANCE |
+ READ_STREAM_USE_BATCHING,
vacrel->bstrategy,
vacrel->rel,
MAIN_FORKNUM,
diff --git a/src/backend/access/nbtree/nbtree.c b/src/backend/access/nbtree/nbtree.c
index 80b04d6ca2a..4a0bf069f99 100644
--- a/src/backend/access/nbtree/nbtree.c
+++ b/src/backend/access/nbtree/nbtree.c
@@ -1064,7 +1064,13 @@ btvacuumscan(IndexVacuumInfo *info, IndexBulkDeleteResult *stats,
needLock = !RELATION_IS_LOCAL(rel);
p.current_blocknum = BTREE_METAPAGE + 1;
- stream = read_stream_begin_relation(READ_STREAM_FULL,
+
+ /*
+ * It is safe to use batchmode as block_range_read_stream_cb takes no
+ * locks.
+ */
+ stream = read_stream_begin_relation(READ_STREAM_FULL |
+ READ_STREAM_USE_BATCHING,
info->strategy,
rel,
MAIN_FORKNUM,
diff --git a/src/backend/access/spgist/spgvacuum.c b/src/backend/access/spgist/spgvacuum.c
index 77deb226b7e..b3df2d89074 100644
--- a/src/backend/access/spgist/spgvacuum.c
+++ b/src/backend/access/spgist/spgvacuum.c
@@ -822,7 +822,13 @@ spgvacuumscan(spgBulkDeleteState *bds)
/* We can skip locking for new or temp relations */
needLock = !RELATION_IS_LOCAL(index);
p.current_blocknum = SPGIST_METAPAGE_BLKNO + 1;
- stream = read_stream_begin_relation(READ_STREAM_FULL,
+
+ /*
+ * It is safe to use batchmode as block_range_read_stream_cb takes no
+ * locks.
+ */
+ stream = read_stream_begin_relation(READ_STREAM_FULL |
+ READ_STREAM_USE_BATCHING,
bds->info->strategy,
index,
MAIN_FORKNUM,
diff --git a/src/backend/commands/analyze.c b/src/backend/commands/analyze.c
index 2b5fbdcbd82..fbf618e6abb 100644
--- a/src/backend/commands/analyze.c
+++ b/src/backend/commands/analyze.c
@@ -1237,7 +1237,12 @@ acquire_sample_rows(Relation onerel, int elevel,
scan = table_beginscan_analyze(onerel);
slot = table_slot_create(onerel, NULL);
- stream = read_stream_begin_relation(READ_STREAM_MAINTENANCE,
+ /*
+ * It is safe to use batching, as block_sampling_read_stream_next never
+ * blocks.
+ */
+ stream = read_stream_begin_relation(READ_STREAM_MAINTENANCE |
+ READ_STREAM_USE_BATCHING,
vac_strategy,
scan->rs_rd,
MAIN_FORKNUM,
diff --git a/src/backend/storage/aio/read_stream.c b/src/backend/storage/aio/read_stream.c
index 26e5dfe77db..36c54fb695b 100644
--- a/src/backend/storage/aio/read_stream.c
+++ b/src/backend/storage/aio/read_stream.c
@@ -102,6 +102,7 @@ struct ReadStream
int16 initialized_buffers;
int read_buffers_flags;
bool sync_mode; /* using io_method=sync */
+ bool batch_mode; /* READ_STREAM_USE_BATCHING */
bool advice_enabled;
bool temporary;
@@ -403,6 +404,15 @@ read_stream_start_pending_read(ReadStream *stream)
static void
read_stream_look_ahead(ReadStream *stream)
{
+ /*
+ * Allow amortizing the cost of submitting IO over multiple IOs. This
+ * requires that we don't do any operations that could lead to a deadlock
+ * with staged-but-unsubmitted IO. The callback needs to opt-in to being
+ * careful.
+ */
+ if (stream->batch_mode)
+ pgaio_enter_batchmode();
+
while (stream->ios_in_progress < stream->max_ios &&
stream->pinned_buffers + stream->pending_read_nblocks < stream->distance)
{
@@ -450,6 +460,8 @@ read_stream_look_ahead(ReadStream *stream)
{
/* We've hit the buffer or I/O limit. Rewind and stop here. */
read_stream_unget_block(stream, blocknum);
+ if (stream->batch_mode)
+ pgaio_exit_batchmode();
return;
}
}
@@ -484,6 +496,9 @@ read_stream_look_ahead(ReadStream *stream)
* time.
*/
Assert(stream->pinned_buffers > 0 || stream->distance == 0);
+
+ if (stream->batch_mode)
+ pgaio_exit_batchmode();
}
/*
@@ -617,6 +632,7 @@ read_stream_begin_impl(int flags,
MAXALIGN(&stream->ios[Max(1, max_ios)]);
stream->sync_mode = io_method == IOMETHOD_SYNC;
+ stream->batch_mode = flags & READ_STREAM_USE_BATCHING;
#ifdef USE_PREFETCH
diff --git a/src/backend/storage/buffer/bufmgr.c b/src/backend/storage/buffer/bufmgr.c
index 040e7cef9e8..80d7cc5959a 100644
--- a/src/backend/storage/buffer/bufmgr.c
+++ b/src/backend/storage/buffer/bufmgr.c
@@ -5085,7 +5085,13 @@ RelationCopyStorageUsingBuffer(RelFileLocator srclocator,
p.current_blocknum = 0;
p.last_exclusive = nblocks;
src_smgr = smgropen(srclocator, INVALID_PROC_NUMBER);
- src_stream = read_stream_begin_smgr_relation(READ_STREAM_FULL,
+
+ /*
+ * It is safe to use batchmode as block_range_read_stream_cb takes no
+ * locks.
+ */
+ src_stream = read_stream_begin_smgr_relation(READ_STREAM_FULL |
+ READ_STREAM_USE_BATCHING,
bstrategy_src,
src_smgr,
permanent ? RELPERSISTENCE_PERMANENT : RELPERSISTENCE_UNLOGGED,
diff --git a/contrib/pg_prewarm/pg_prewarm.c b/contrib/pg_prewarm/pg_prewarm.c
index f496ec9d85d..84a3189a77b 100644
--- a/contrib/pg_prewarm/pg_prewarm.c
+++ b/contrib/pg_prewarm/pg_prewarm.c
@@ -198,7 +198,12 @@ pg_prewarm(PG_FUNCTION_ARGS)
p.current_blocknum = first_block;
p.last_exclusive = last_block + 1;
- stream = read_stream_begin_relation(READ_STREAM_FULL,
+ /*
+ * It is safe to use batchmode as block_range_read_stream_cb takes no
+ * locks.
+ */
+ stream = read_stream_begin_relation(READ_STREAM_FULL |
+ READ_STREAM_USE_BATCHING,
NULL,
rel,
forkNumber,
diff --git a/contrib/pg_visibility/pg_visibility.c b/contrib/pg_visibility/pg_visibility.c
index ca91819852c..d79ef35006b 100644
--- a/contrib/pg_visibility/pg_visibility.c
+++ b/contrib/pg_visibility/pg_visibility.c
@@ -526,7 +526,13 @@ collect_visibility_data(Oid relid, bool include_pd)
{
p.current_blocknum = 0;
p.last_exclusive = nblocks;
- stream = read_stream_begin_relation(READ_STREAM_FULL,
+
+ /*
+ * It is safe to use batchmode as block_range_read_stream_cb takes no
+ * locks.
+ */
+ stream = read_stream_begin_relation(READ_STREAM_FULL |
+ READ_STREAM_USE_BATCHING,
bstrategy,
rel,
MAIN_FORKNUM,
--
2.48.1.76.g4e746b1a31.dirty
[text/x-diff] v2.13-0015-docs-Reframe-track_io_timing-related-docs-as-w.patch (5.3K, ../../o2dtdrlc2xqpjycsfuvv7u6zyrqj7jfxqu32et3lyelkwce2jy@3k3fwdiiu73v/16-v2.13-0015-docs-Reframe-track_io_timing-related-docs-as-w.patch)
download | inline diff:
From 10739e9cb2515f2794a075b12381fa1cf0924ad6 Mon Sep 17 00:00:00 2001
From: Andres Freund <andres@anarazel.de>
Date: Tue, 18 Mar 2025 14:40:06 -0400
Subject: [PATCH v2.13 15/28] docs: Reframe track_io_timing related docs as
wait time
With AIO it does not make sense anymore to track the time for each individual
IO, as multiple IOs can be in-flight at the same time. Instead we now track
the time spent *waiting* for IOs.
This should be reflected in the docs. While, so far, we only do a subset of
reads, and no other operations, via AIO, describing the GUC and view columns
as measuring IO waits is accurate for synchronous and asynchronous IO.
Discussion: https://postgr.es/m/5dzyoduxlvfg55oqtjyjehez5uoq6hnwgzor4kkybkfdgkj7ag@rbi4gsmzaczk
---
doc/src/sgml/config.sgml | 4 ++--
doc/src/sgml/monitoring.sgml | 12 ++++++------
2 files changed, 8 insertions(+), 8 deletions(-)
diff --git a/doc/src/sgml/config.sgml b/doc/src/sgml/config.sgml
index f86135fbe1d..13773c05ff4 100644
--- a/doc/src/sgml/config.sgml
+++ b/doc/src/sgml/config.sgml
@@ -8568,7 +8568,7 @@ COPY postgres_log FROM '/full/path/to/logfile.csv' WITH csv;
</term>
<listitem>
<para>
- Enables timing of database I/O calls. This parameter is off by
+ Enables timing of database I/O waits. This parameter is off by
default, as it will repeatedly query the operating system for
the current time, which may cause significant overhead on some
platforms. You can use the <xref linkend="pgtesttiming"/> tool to
@@ -8602,7 +8602,7 @@ COPY postgres_log FROM '/full/path/to/logfile.csv' WITH csv;
</term>
<listitem>
<para>
- Enables timing of WAL I/O calls. This parameter is off by default,
+ Enables timing of WAL I/O waits. This parameter is off by default,
as it will repeatedly query the operating system for the current time,
which may cause significant overhead on some platforms.
You can use the <application>pg_test_timing</application> tool to
diff --git a/doc/src/sgml/monitoring.sgml b/doc/src/sgml/monitoring.sgml
index bacc09cb8af..a6d67d2fbaa 100644
--- a/doc/src/sgml/monitoring.sgml
+++ b/doc/src/sgml/monitoring.sgml
@@ -2747,7 +2747,7 @@ description | Waiting for a newly initialized WAL file to reach durable storage
<structfield>read_time</structfield> <type>double precision</type>
</para>
<para>
- Time spent in read operations in milliseconds (if
+ Time spent waiting for read operations in milliseconds (if
<xref linkend="guc-track-io-timing"/> is enabled and
<varname>object</varname> is not <literal>wal</literal>,
or if <xref linkend="guc-track-wal-io-timing"/> is enabled
@@ -2785,7 +2785,7 @@ description | Waiting for a newly initialized WAL file to reach durable storage
<structfield>write_time</structfield> <type>double precision</type>
</para>
<para>
- Time spent in write operations in milliseconds (if
+ Time spent waiting for write operations in milliseconds (if
<xref linkend="guc-track-io-timing"/> is enabled and
<varname>object</varname> is not <literal>wal</literal>,
or if <xref linkend="guc-track-wal-io-timing"/> is enabled
@@ -2813,7 +2813,7 @@ description | Waiting for a newly initialized WAL file to reach durable storage
<structfield>writeback_time</structfield> <type>double precision</type>
</para>
<para>
- Time spent in writeback operations in milliseconds (if
+ Time spent waiting for writeback operations in milliseconds (if
<xref linkend="guc-track-io-timing"/> is enabled, otherwise zero). This
includes the time spent queueing write-out requests and, potentially,
the time spent to write out the dirty data.
@@ -2849,7 +2849,7 @@ description | Waiting for a newly initialized WAL file to reach durable storage
<structfield>extend_time</structfield> <type>double precision</type>
</para>
<para>
- Time spent in extend operations in milliseconds. (if
+ Time spent waiting for extend operations in milliseconds. (if
<xref linkend="guc-track-io-timing"/> is enabled and
<varname>object</varname> is not <literal>wal</literal>,
or if <xref linkend="guc-track-wal-io-timing"/> is enabled
@@ -2923,7 +2923,7 @@ description | Waiting for a newly initialized WAL file to reach durable storage
<structfield>fsync_time</structfield> <type>double precision</type>
</para>
<para>
- Time spent in fsync operations in milliseconds (if
+ Time spent waiting for fsync operations in milliseconds (if
<xref linkend="guc-track-io-timing"/> is enabled and
<varname>object</varname> is not <literal>wal</literal>,
or if <xref linkend="guc-track-wal-io-timing"/> is enabled
@@ -3010,7 +3010,7 @@ description | Waiting for a newly initialized WAL file to reach durable storage
<note>
<para>
- Columns tracking I/O time will only be non-zero when
+ Columns tracking I/O wait time will only be non-zero when
<xref linkend="guc-track-io-timing"/> is enabled. The user should be
careful when referencing these columns in combination with their
corresponding I/O operations in case <varname>track_io_timing</varname>
--
2.48.1.76.g4e746b1a31.dirty
[text/x-diff] v2.13-0016-Enable-IO-concurrency-on-all-systems.patch (11.1K, ../../o2dtdrlc2xqpjycsfuvv7u6zyrqj7jfxqu32et3lyelkwce2jy@3k3fwdiiu73v/17-v2.13-0016-Enable-IO-concurrency-on-all-systems.patch)
download | inline diff:
From 7c803ff8e9e4025e690ebc8529792a79017efa78 Mon Sep 17 00:00:00 2001
From: Andres Freund <andres@anarazel.de>
Date: Wed, 19 Mar 2025 10:15:20 -0400
Subject: [PATCH v2.13 16/28] Enable IO concurrency on all systems
Previously effective_io_concurrency and maintenance_io_concurrency could
not be set above 0 on machines without fadvise support. AIO enables
IO concurrency without such support.
Currently only subsystems using the read stream API will take advantage
of this. Other users of maintenance_io_concurrency (like recovery
prefetching) which leverage OS advice directly will not benefit from
this change. In those cases, maintenance_io_concurrency will have no
effect on I/O behavior.
Author: Melanie Plageman <melanieplageman@gmail.com>
Reviewed-by: Noah Misch <noah@leadboat.com>
Discussion: https://postgr.es/m/CAAKRu_atGgZePo=_g6T3cNtfMf0QxpvoUh5OUqa_cnPdhLd=gw@mail.gmail.com
---
src/include/storage/bufmgr.h | 6 ----
src/include/utils/guc_hooks.h | 4 ---
src/backend/access/common/reloptions.c | 8 -----
src/backend/commands/variable.c | 30 -------------------
src/backend/utils/misc/guc_tables.c | 4 +--
src/backend/utils/misc/postgresql.conf.sample | 4 +--
src/bin/initdb/initdb.c | 5 ----
doc/src/sgml/config.sgml | 16 +++++-----
doc/src/sgml/ref/alter_tablespace.sgml | 2 +-
doc/src/sgml/ref/create_tablespace.sgml | 2 +-
10 files changed, 14 insertions(+), 67 deletions(-)
diff --git a/src/include/storage/bufmgr.h b/src/include/storage/bufmgr.h
index 7abb3bdfd9d..784df8b00cb 100644
--- a/src/include/storage/bufmgr.h
+++ b/src/include/storage/bufmgr.h
@@ -156,14 +156,8 @@ extern PGDLLIMPORT int bgwriter_lru_maxpages;
extern PGDLLIMPORT double bgwriter_lru_multiplier;
extern PGDLLIMPORT bool track_io_timing;
-/* only applicable when prefetching is available */
-#ifdef USE_PREFETCH
#define DEFAULT_EFFECTIVE_IO_CONCURRENCY 16
#define DEFAULT_MAINTENANCE_IO_CONCURRENCY 16
-#else
-#define DEFAULT_EFFECTIVE_IO_CONCURRENCY 0
-#define DEFAULT_MAINTENANCE_IO_CONCURRENCY 0
-#endif
extern PGDLLIMPORT int effective_io_concurrency;
extern PGDLLIMPORT int maintenance_io_concurrency;
diff --git a/src/include/utils/guc_hooks.h b/src/include/utils/guc_hooks.h
index 0f1e74f96c9..799fa7ace68 100644
--- a/src/include/utils/guc_hooks.h
+++ b/src/include/utils/guc_hooks.h
@@ -61,8 +61,6 @@ extern bool check_default_text_search_config(char **newval, void **extra, GucSou
extern void assign_default_text_search_config(const char *newval, void *extra);
extern bool check_default_with_oids(bool *newval, void **extra,
GucSource source);
-extern bool check_effective_io_concurrency(int *newval, void **extra,
- GucSource source);
extern bool check_huge_page_size(int *newval, void **extra, GucSource source);
extern void assign_io_method(int newval, void *extra);
extern bool check_io_max_concurrency(int *newval, void **extra, GucSource source);
@@ -83,8 +81,6 @@ extern bool check_log_stats(bool *newval, void **extra, GucSource source);
extern bool check_log_timezone(char **newval, void **extra, GucSource source);
extern void assign_log_timezone(const char *newval, void *extra);
extern const char *show_log_timezone(void);
-extern bool check_maintenance_io_concurrency(int *newval, void **extra,
- GucSource source);
extern void assign_maintenance_io_concurrency(int newval, void *extra);
extern void assign_io_max_combine_limit(int newval, void *extra);
extern void assign_io_combine_limit(int newval, void *extra);
diff --git a/src/backend/access/common/reloptions.c b/src/backend/access/common/reloptions.c
index 645b5c00467..46c1dce222d 100644
--- a/src/backend/access/common/reloptions.c
+++ b/src/backend/access/common/reloptions.c
@@ -361,11 +361,7 @@ static relopt_int intRelOpts[] =
RELOPT_KIND_TABLESPACE,
ShareUpdateExclusiveLock
},
-#ifdef USE_PREFETCH
-1, 0, MAX_IO_CONCURRENCY
-#else
- 0, 0, 0
-#endif
},
{
{
@@ -374,11 +370,7 @@ static relopt_int intRelOpts[] =
RELOPT_KIND_TABLESPACE,
ShareUpdateExclusiveLock
},
-#ifdef USE_PREFETCH
-1, 0, MAX_IO_CONCURRENCY
-#else
- 0, 0, 0
-#endif
},
{
{
diff --git a/src/backend/commands/variable.c b/src/backend/commands/variable.c
index f550a3c0c63..d3567aa7f27 100644
--- a/src/backend/commands/variable.c
+++ b/src/backend/commands/variable.c
@@ -1145,7 +1145,6 @@ check_cluster_name(char **newval, void **extra, GucSource source)
void
assign_maintenance_io_concurrency(int newval, void *extra)
{
-#ifdef USE_PREFETCH
/*
* Reconfigure recovery prefetching, because a setting it depends on
* changed.
@@ -1153,7 +1152,6 @@ assign_maintenance_io_concurrency(int newval, void *extra)
maintenance_io_concurrency = newval;
if (AmStartupProcess())
XLogPrefetchReconfigure();
-#endif
}
/*
@@ -1249,34 +1247,6 @@ check_default_with_oids(bool *newval, void **extra, GucSource source)
return true;
}
-bool
-check_effective_io_concurrency(int *newval, void **extra, GucSource source)
-{
-#ifndef USE_PREFETCH
- if (*newval != 0)
- {
- GUC_check_errdetail("\"%s\" must be set to 0 on platforms that lack support for issuing read-ahead advice.",
- "effective_io_concurrency");
- return false;
- }
-#endif /* USE_PREFETCH */
- return true;
-}
-
-bool
-check_maintenance_io_concurrency(int *newval, void **extra, GucSource source)
-{
-#ifndef USE_PREFETCH
- if (*newval != 0)
- {
- GUC_check_errdetail("\"%s\" must be set to 0 on platforms that lack support for issuing read-ahead advice.",
- "maintenance_io_concurrency");
- return false;
- }
-#endif /* USE_PREFETCH */
- return true;
-}
-
bool
check_ssl(bool *newval, void **extra, GucSource source)
{
diff --git a/src/backend/utils/misc/guc_tables.c b/src/backend/utils/misc/guc_tables.c
index 989825d3a9c..5d729102f46 100644
--- a/src/backend/utils/misc/guc_tables.c
+++ b/src/backend/utils/misc/guc_tables.c
@@ -3245,7 +3245,7 @@ struct config_int ConfigureNamesInt[] =
&effective_io_concurrency,
DEFAULT_EFFECTIVE_IO_CONCURRENCY,
0, MAX_IO_CONCURRENCY,
- check_effective_io_concurrency, NULL, NULL
+ NULL, NULL, NULL
},
{
@@ -3259,7 +3259,7 @@ struct config_int ConfigureNamesInt[] =
&maintenance_io_concurrency,
DEFAULT_MAINTENANCE_IO_CONCURRENCY,
0, MAX_IO_CONCURRENCY,
- check_maintenance_io_concurrency, assign_maintenance_io_concurrency,
+ NULL, assign_maintenance_io_concurrency,
NULL
},
diff --git a/src/backend/utils/misc/postgresql.conf.sample b/src/backend/utils/misc/postgresql.conf.sample
index 2246ccb85a7..ac6ae0a9489 100644
--- a/src/backend/utils/misc/postgresql.conf.sample
+++ b/src/backend/utils/misc/postgresql.conf.sample
@@ -198,8 +198,8 @@
# - I/O -
#backend_flush_after = 0 # measured in pages, 0 disables
-#effective_io_concurrency = 16 # 1-1000; 0 disables prefetching
-#maintenance_io_concurrency = 16 # 1-1000; 0 disables prefetching
+#effective_io_concurrency = 16 # 1-1000; 0 disables issuing multiple simultaneous IO requests
+#maintenance_io_concurrency = 16 # 1-1000; same as effective_io_concurrency
#io_max_combine_limit = 128kB # usually 1-128 blocks (depends on OS)
# (change requires restart)
#io_combine_limit = 128kB # usually 1-128 blocks (depends on OS)
diff --git a/src/bin/initdb/initdb.c b/src/bin/initdb/initdb.c
index 22b7d31b165..c17fda2bc81 100644
--- a/src/bin/initdb/initdb.c
+++ b/src/bin/initdb/initdb.c
@@ -1402,11 +1402,6 @@ setup_config(void)
repltok, true);
#endif
-#ifndef USE_PREFETCH
- conflines = replace_guc_value(conflines, "effective_io_concurrency",
- "0", true);
-#endif
-
#ifdef WIN32
conflines = replace_guc_value(conflines, "update_process_title",
"off", true);
diff --git a/doc/src/sgml/config.sgml b/doc/src/sgml/config.sgml
index 13773c05ff4..370cda13553 100644
--- a/doc/src/sgml/config.sgml
+++ b/doc/src/sgml/config.sgml
@@ -2585,8 +2585,7 @@ include_dir 'conf.d'
session attempts to initiate in parallel. The allowed range is
<literal>1</literal> to <literal>1000</literal>, or
<literal>0</literal> to disable issuance of asynchronous I/O requests.
- The default is <literal>16</literal> on supported systems, otherwise
- <literal>0</literal>.
+ The default is <literal>16</literal>.
</para>
<para>
@@ -2597,8 +2596,9 @@ include_dir 'conf.d'
</para>
<para>
- On systems without prefetch advice support, attempting to configure
- any value other than <literal>0</literal> will error out.
+ On systems with prefetch advice support,
+ <varname>effective_io_concurrency</varname> also controls the
+ prefetch distance.
</para>
<para>
@@ -2621,10 +2621,10 @@ include_dir 'conf.d'
for maintenance work that is done on behalf of many client sessions.
</para>
<para>
- The default is <literal>16</literal> on supported systems, otherwise
- <literal>0</literal>. This value can be overridden for tables in a
- particular tablespace by setting the tablespace parameter of the same
- name (see <xref linkend="sql-altertablespace"/>).
+ The default is <literal>16</literal>. This value can be overridden
+ for tables in a particular tablespace by setting the tablespace
+ parameter of the same name (see <xref
+ linkend="sql-altertablespace"/>).
</para>
</listitem>
</varlistentry>
diff --git a/doc/src/sgml/ref/alter_tablespace.sgml b/doc/src/sgml/ref/alter_tablespace.sgml
index 6ec863400d1..d0e08089ddb 100644
--- a/doc/src/sgml/ref/alter_tablespace.sgml
+++ b/doc/src/sgml/ref/alter_tablespace.sgml
@@ -88,7 +88,7 @@ ALTER TABLESPACE <replaceable>name</replaceable> RESET ( <replaceable class="par
and <varname>maintenance_io_concurrency</varname>.
Setting these values for a particular tablespace will override the
planner's usual estimate of the cost of reading pages from tables in
- that tablespace, and the executor's prefetching behavior, as established
+ that tablespace, and how many concurrent I/Os are issued, as established
by the configuration parameters of the
same name (see <xref linkend="guc-seq-page-cost"/>,
<xref linkend="guc-random-page-cost"/>,
diff --git a/doc/src/sgml/ref/create_tablespace.sgml b/doc/src/sgml/ref/create_tablespace.sgml
index 9d5ab025261..b77e774c53f 100644
--- a/doc/src/sgml/ref/create_tablespace.sgml
+++ b/doc/src/sgml/ref/create_tablespace.sgml
@@ -110,7 +110,7 @@ CREATE TABLESPACE <replaceable class="parameter">tablespace_name</replaceable>
and <varname>maintenance_io_concurrency</varname>.
Setting these values for a particular tablespace will override the
planner's usual estimate of the cost of reading pages from tables in
- that tablespace, and the executor's prefetching behavior, as established
+ that tablespace, and how many concurrent I/Os are issued, as established
by the configuration parameters of the
same name (see <xref linkend="guc-seq-page-cost"/>,
<xref linkend="guc-random-page-cost"/>,
--
2.48.1.76.g4e746b1a31.dirty
[text/x-diff] v2.13-0017-aio-Add-test_aio-module.patch (62.7K, ../../o2dtdrlc2xqpjycsfuvv7u6zyrqj7jfxqu32et3lyelkwce2jy@3k3fwdiiu73v/18-v2.13-0017-aio-Add-test_aio-module.patch)
download | inline diff:
From bb4f22bd400a95520c7634328c5222b6352e0145 Mon Sep 17 00:00:00 2001
From: Andres Freund <andres@anarazel.de>
Date: Tue, 18 Mar 2025 14:40:06 -0400
Subject: [PATCH v2.13 17/28] aio: Add test_aio module
To make the tests possible, a few functions from bufmgr.c/localbuf.c had to be
exported, via buf_internals.h.
Reviewed-by: Noah Misch <noah@leadboat.com>
Co-authored-by: Andres Freund <andres@anarazel.de>
Co-authored-by: Nazir Bilal Yavuz <byavuz81@gmail.com>
Discussion: https://postgr.es/m/uvrtrknj4kdytuboidbhwclo4gxhswwcpgadptsjvjqcluzmah%40brqs62irg4dt
---
src/include/storage/buf_internals.h | 7 +
src/backend/storage/buffer/bufmgr.c | 8 +-
src/backend/storage/buffer/localbuf.c | 3 +-
src/test/modules/Makefile | 1 +
src/test/modules/meson.build | 1 +
src/test/modules/test_aio/.gitignore | 2 +
src/test/modules/test_aio/Makefile | 26 +
src/test/modules/test_aio/meson.build | 37 +
src/test/modules/test_aio/t/001_aio.pl | 1090 +++++++++++++++++
src/test/modules/test_aio/t/002_io_workers.pl | 125 ++
src/test/modules/test_aio/test_aio--1.0.sql | 107 ++
src/test/modules/test_aio/test_aio.c | 748 +++++++++++
src/test/modules/test_aio/test_aio.control | 3 +
13 files changed, 2150 insertions(+), 8 deletions(-)
create mode 100644 src/test/modules/test_aio/.gitignore
create mode 100644 src/test/modules/test_aio/Makefile
create mode 100644 src/test/modules/test_aio/meson.build
create mode 100644 src/test/modules/test_aio/t/001_aio.pl
create mode 100644 src/test/modules/test_aio/t/002_io_workers.pl
create mode 100644 src/test/modules/test_aio/test_aio--1.0.sql
create mode 100644 src/test/modules/test_aio/test_aio.c
create mode 100644 src/test/modules/test_aio/test_aio.control
diff --git a/src/include/storage/buf_internals.h b/src/include/storage/buf_internals.h
index 72b36a4af26..0dec7d93b3b 100644
--- a/src/include/storage/buf_internals.h
+++ b/src/include/storage/buf_internals.h
@@ -434,6 +434,12 @@ extern void IssuePendingWritebacks(WritebackContext *wb_context, IOContext io_co
extern void ScheduleBufferTagForWriteback(WritebackContext *wb_context,
IOContext io_context, BufferTag *tag);
+/* solely to make it easier to write tests */
+extern bool StartBufferIO(BufferDesc *buf, bool forInput, bool nowait);
+extern void TerminateBufferIO(BufferDesc *buf, bool clear_dirty, uint32 set_flag_bits,
+ bool forget_owner, bool release_aio);
+
+
/* freelist.c */
extern IOContext IOContextForStrategy(BufferAccessStrategy strategy);
extern BufferDesc *StrategyGetBuffer(BufferAccessStrategy strategy,
@@ -478,6 +484,7 @@ extern void TerminateLocalBufferIO(BufferDesc *bufHdr, bool clear_dirty,
uint32 set_flag_bits, bool release_aio);
extern bool StartLocalBufferIO(BufferDesc *bufHdr, bool forInput, bool nowait);
extern void FlushLocalBuffer(BufferDesc *bufHdr, SMgrRelation reln);
+extern void InvalidateLocalBuffer(BufferDesc *bufHdr, bool check_unreferenced);
extern void DropRelationLocalBuffers(RelFileLocator rlocator,
ForkNumber forkNum,
BlockNumber firstDelBlock);
diff --git a/src/backend/storage/buffer/bufmgr.c b/src/backend/storage/buffer/bufmgr.c
index 80d7cc5959a..8608df8f115 100644
--- a/src/backend/storage/buffer/bufmgr.c
+++ b/src/backend/storage/buffer/bufmgr.c
@@ -518,10 +518,6 @@ static uint32 WaitBufHdrUnlocked(BufferDesc *buf);
static int SyncOneBuffer(int buf_id, bool skip_recently_used,
WritebackContext *wb_context);
static void WaitIO(BufferDesc *buf);
-static bool StartBufferIO(BufferDesc *buf, bool forInput, bool nowait);
-static void TerminateBufferIO(BufferDesc *buf, bool clear_dirty,
- uint32 set_flag_bits, bool forget_owner,
- bool release_aio);
static void AbortBufferIO(Buffer buffer);
static void shared_buffer_write_error_callback(void *arg);
static void local_buffer_write_error_callback(void *arg);
@@ -5946,7 +5942,7 @@ WaitIO(BufferDesc *buf)
* find out if they can perform the I/O as part of a larger operation, without
* waiting for the answer or distinguishing the reasons why not.
*/
-static bool
+bool
StartBufferIO(BufferDesc *buf, bool forInput, bool nowait)
{
uint32 buf_state;
@@ -6003,7 +5999,7 @@ StartBufferIO(BufferDesc *buf, bool forInput, bool nowait)
* resource owner. (forget_owner=false is used when the resource owner itself
* is being released)
*/
-static void
+void
TerminateBufferIO(BufferDesc *buf, bool clear_dirty, uint32 set_flag_bits,
bool forget_owner, bool release_aio)
{
diff --git a/src/backend/storage/buffer/localbuf.c b/src/backend/storage/buffer/localbuf.c
index 6cd4d0f51eb..fa972ed55da 100644
--- a/src/backend/storage/buffer/localbuf.c
+++ b/src/backend/storage/buffer/localbuf.c
@@ -58,7 +58,6 @@ static int NLocalPinnedBuffers = 0;
static void InitLocalBuffers(void);
static Block GetLocalBufferStorage(void);
static Buffer GetLocalVictimBuffer(void);
-static void InvalidateLocalBuffer(BufferDesc *bufHdr, bool check_unreferenced);
/*
@@ -598,7 +597,7 @@ TerminateLocalBufferIO(BufferDesc *bufHdr, bool clear_dirty, uint32 set_flag_bit
*
* See also InvalidateBuffer().
*/
-static void
+void
InvalidateLocalBuffer(BufferDesc *bufHdr, bool check_unreferenced)
{
Buffer buffer = BufferDescriptorGetBuffer(bufHdr);
diff --git a/src/test/modules/Makefile b/src/test/modules/Makefile
index 4e4be3fa511..aa1d27bbed3 100644
--- a/src/test/modules/Makefile
+++ b/src/test/modules/Makefile
@@ -14,6 +14,7 @@ SUBDIRS = \
oauth_validator \
plsample \
spgist_name_ops \
+ test_aio \
test_bloomfilter \
test_copy_callbacks \
test_custom_rmgrs \
diff --git a/src/test/modules/meson.build b/src/test/modules/meson.build
index 2b057451473..9de0057bd1d 100644
--- a/src/test/modules/meson.build
+++ b/src/test/modules/meson.build
@@ -13,6 +13,7 @@ subdir('oauth_validator')
subdir('plsample')
subdir('spgist_name_ops')
subdir('ssl_passphrase_callback')
+subdir('test_aio')
subdir('test_bloomfilter')
subdir('test_copy_callbacks')
subdir('test_custom_rmgrs')
diff --git a/src/test/modules/test_aio/.gitignore b/src/test/modules/test_aio/.gitignore
new file mode 100644
index 00000000000..716e17f5a2a
--- /dev/null
+++ b/src/test/modules/test_aio/.gitignore
@@ -0,0 +1,2 @@
+# Generated subdirectories
+/tmp_check/
diff --git a/src/test/modules/test_aio/Makefile b/src/test/modules/test_aio/Makefile
new file mode 100644
index 00000000000..f53cc64671a
--- /dev/null
+++ b/src/test/modules/test_aio/Makefile
@@ -0,0 +1,26 @@
+# src/test/modules/test_aio/Makefile
+
+PGFILEDESC = "test_aio - test code for AIO"
+
+MODULE_big = test_aio
+OBJS = \
+ $(WIN32RES) \
+ test_aio.o
+
+EXTENSION = test_aio
+DATA = test_aio--1.0.sql
+
+TAP_TESTS = 1
+
+export enable_injection_points
+
+ifdef USE_PGXS
+PG_CONFIG = pg_config
+PGXS := $(shell $(PG_CONFIG) --pgxs)
+include $(PGXS)
+else
+subdir = src/test/modules/test_aio
+top_builddir = ../../../..
+include $(top_builddir)/src/Makefile.global
+include $(top_srcdir)/contrib/contrib-global.mk
+endif
diff --git a/src/test/modules/test_aio/meson.build b/src/test/modules/test_aio/meson.build
new file mode 100644
index 00000000000..73d2fd68eaa
--- /dev/null
+++ b/src/test/modules/test_aio/meson.build
@@ -0,0 +1,37 @@
+# Copyright (c) 2024-2025, PostgreSQL Global Development Group
+
+test_aio_sources = files(
+ 'test_aio.c',
+)
+
+if host_system == 'windows'
+ test_aio_sources += rc_lib_gen.process(win32ver_rc, extra_args: [
+ '--NAME', 'test_aio',
+ '--FILEDESC', 'test_aio - test code for AIO',])
+endif
+
+test_aio = shared_module('test_aio',
+ test_aio_sources,
+ kwargs: pg_test_mod_args,
+)
+test_install_libs += test_aio
+
+test_install_data += files(
+ 'test_aio.control',
+ 'test_aio--1.0.sql',
+)
+
+tests += {
+ 'name': 'test_aio',
+ 'sd': meson.current_source_dir(),
+ 'bd': meson.current_build_dir(),
+ 'tap': {
+ 'env': {
+ 'enable_injection_points': get_option('injection_points') ? 'yes' : 'no',
+ },
+ 'tests': [
+ 't/001_aio.pl',
+ 't/002_io_workers.pl',
+ ],
+ },
+}
diff --git a/src/test/modules/test_aio/t/001_aio.pl b/src/test/modules/test_aio/t/001_aio.pl
new file mode 100644
index 00000000000..a4c896207d4
--- /dev/null
+++ b/src/test/modules/test_aio/t/001_aio.pl
@@ -0,0 +1,1090 @@
+# Copyright (c) 2025, PostgreSQL Global Development Group
+
+use strict;
+use warnings FATAL => 'all';
+
+use PostgreSQL::Test::Cluster;
+use PostgreSQL::Test::Utils;
+use Test::More;
+
+
+###
+# Test io_method=worker
+###
+my $node_worker = create_node('worker');
+$node_worker->start();
+
+test_generic('worker', $node_worker);
+SKIP:
+{
+ skip 'Injection points not supported by this build', 1
+ unless $ENV{enable_injection_points} eq 'yes';
+ test_inject_worker('worker', $node_worker);
+}
+
+$node_worker->stop();
+
+
+###
+# Test io_method=io_uring
+###
+
+if (have_io_uring())
+{
+ my $node_uring = create_node('io_uring');
+ $node_uring->start();
+ test_generic('io_uring', $node_uring);
+ $node_uring->stop();
+}
+
+
+###
+# Test io_method=sync
+###
+
+my $node_sync = create_node('sync');
+
+# just to have one test not use the default auto-tuning
+
+$node_sync->append_conf(
+ 'postgresql.conf', qq(
+io_max_concurrency=4
+));
+
+$node_sync->start();
+test_generic('sync', $node_sync);
+$node_sync->stop();
+
+done_testing();
+
+
+###
+# Test Helpers
+###
+
+sub create_node
+{
+ local $Test::Builder::Level = $Test::Builder::Level + 1;
+
+ my $io_method = shift;
+
+ my $node = PostgreSQL::Test::Cluster->new($io_method);
+
+ # Want to test initdb for each IO method, otherwise we could just reuse
+ # the cluster.
+ #
+ # Unfortunately Cluster::init() puts PG_TEST_INITDB_EXTRA_OPTS after the
+ # options specified by ->extra, if somebody puts -c io_method=xyz in
+ # PG_TEST_INITDB_EXTRA_OPTS it would break this test. Fix that up if we
+ # detect it.
+ local $ENV{PG_TEST_INITDB_EXTRA_OPTS} = $ENV{PG_TEST_INITDB_EXTRA_OPTS};
+ if (defined $ENV{PG_TEST_INITDB_EXTRA_OPTS}
+ && $ENV{PG_TEST_INITDB_EXTRA_OPTS} =~ m/io_method=/)
+ {
+ $ENV{PG_TEST_INITDB_EXTRA_OPTS} .= " -c io_method=$io_method";
+ }
+
+ $node->init(extra => [ '-c', "io_method=$io_method" ]);
+
+ $node->append_conf(
+ 'postgresql.conf', qq(
+shared_preload_libraries=test_aio
+log_min_messages = 'DEBUG3'
+log_statement=all
+restart_after_crash=false
+temp_buffers=100
+));
+
+ ok(1, "$io_method: initdb");
+
+ return $node;
+}
+
+sub have_io_uring
+{
+ # To detect if io_uring is supported, we look at the error message for
+ # assigning an invalid value to an enum GUC, which lists all the valid
+ # options. We need to use -C to deal with running as administrator on
+ # windows, the superuser check is omitted if -C is used.
+ my ($stdout, $stderr) =
+ run_command [qw(postgres -C invalid -c io_method=invalid)];
+ die "can't determine supported io_method values"
+ unless $stderr =~ m/Available values: ([^\.]+)\./;
+ my $methods = $1;
+ note "supported io_method values are: $methods";
+
+ return ($methods =~ m/io_uring/) ? 1 : 0;
+}
+
+sub psql_like
+{
+ my $io_method = shift;
+ my $psql = shift;
+ my $name = shift;
+ my $sql = shift;
+ my $expected_stdout = shift;
+ my $expected_stderr = shift;
+ my ($cmdret, $output);
+
+ ($output, $cmdret) = $psql->query($sql);
+
+ like($output, $expected_stdout, "$io_method: $name: expected stdout");
+ like($psql->{stderr}, $expected_stderr,
+ "$io_method: $name: expected stderr");
+ $psql->{stderr} = '';
+
+ return $output;
+}
+
+sub query_wait_block
+{
+ my $io_method = shift;
+ my $node = shift;
+ my $psql = shift;
+ my $name = shift;
+ my $sql = shift;
+ my $waitfor = shift;
+
+ my $pid = $psql->query_safe('SELECT pg_backend_pid()');
+
+ $psql->{stdin} .= qq($sql;\n);
+ $psql->{run}->pump_nb();
+ ok(1, "$io_method: $name: issued sql");
+
+ $node->poll_query_until('postgres',
+ qq(SELECT wait_event FROM pg_stat_activity WHERE pid = $pid),
+ $waitfor);
+ ok(1, "$io_method: $name: observed $waitfor wait event");
+}
+
+
+###
+# Sub-tests
+###
+
+# Sanity checks for the IO handle API
+sub test_handle
+{
+ my $io_method = shift;
+ my $node = shift;
+
+ my $psql = $node->background_psql('postgres', on_error_stop => 0);
+
+ # leak warning: implicit xact
+ psql_like(
+ $io_method,
+ $psql,
+ "handle_get() leak in implicit xact",
+ qq(SELECT handle_get()),
+ qr/^$/,
+ qr/leaked AIO handle/,
+ "$io_method: leaky handle_get() warns");
+
+ # leak warning: explicit xact
+ psql_like(
+ $io_method, $psql,
+ "handle_get() leak in explicit xact",
+ qq(BEGIN; SELECT handle_get(); COMMIT),
+ qr/^$/, qr/leaked AIO handle/);
+
+
+ # leak warning: explicit xact, rollback
+ psql_like(
+ $io_method,
+ $psql,
+ "handle_get() leak in explicit xact, rollback",
+ qq(BEGIN; SELECT handle_get(); ROLLBACK;),
+ qr/^$/,
+ qr/leaked AIO handle/);
+
+ # leak warning: subtrans
+ psql_like(
+ $io_method,
+ $psql,
+ "handle_get() leak in subxact",
+ qq(BEGIN; SAVEPOINT foo; SELECT handle_get(); COMMIT;),
+ qr/^$/,
+ qr/leaked AIO handle/);
+
+ # leak warning + error: released in different command (thus resowner)
+ psql_like(
+ $io_method,
+ $psql,
+ "handle_release() in different command",
+ qq(BEGIN; SELECT handle_get(); SELECT handle_release_last(); COMMIT;),
+ qr/^$/,
+ qr/leaked AIO handle.*release in unexpected state/ms);
+
+ # no leak, release in same command
+ psql_like(
+ $io_method,
+ $psql,
+ "handle_release() in same command",
+ qq(BEGIN; SELECT handle_get() UNION ALL SELECT handle_release_last(); COMMIT;),
+ qr/^$/,
+ qr/^$/);
+
+ # normal handle use
+ psql_like($io_method, $psql, "handle_get_release()",
+ qq(SELECT handle_get_release()),
+ qr/^$/, qr/^$/);
+
+ # should error out, API violation
+ psql_like(
+ $io_method,
+ $psql,
+ "handle_get_twice()",
+ qq(SELECT handle_get_twice()),
+ qr/^$/,
+ qr/ERROR: API violation: Only one IO can be handed out$/);
+
+ # recover after error in implicit xact
+ psql_like(
+ $io_method,
+ $psql,
+ "handle error recovery in implicit xact",
+ qq(SELECT handle_get_and_error(); SELECT 'ok', handle_get_release()),
+ qr/^|ok$/,
+ qr/ERROR.*as you command/);
+
+ # recover after error in implicit xact
+ psql_like(
+ $io_method,
+ $psql,
+ "handle error recovery in explicit xact",
+ qq(BEGIN; SELECT handle_get_and_error(); SELECT handle_get_release(), 'ok'; COMMIT;),
+ qr/^|ok$/,
+ qr/ERROR.*as you command/);
+
+ # recover after error in subtrans
+ psql_like(
+ $io_method,
+ $psql,
+ "handle error recovery in explicit subxact",
+ qq(BEGIN; SAVEPOINT foo; SELECT handle_get_and_error(); ROLLBACK TO SAVEPOINT foo; SELECT handle_get_release(); ROLLBACK;),
+ qr/^|ok$/,
+ qr/ERROR.*as you command/);
+
+ $psql->quit();
+}
+
+# Sanity checks for the batchmode API
+sub test_batchmode
+{
+ my $io_method = shift;
+ my $node = shift;
+
+ my $psql = $node->background_psql('postgres', on_error_stop => 0);
+
+ # leak warning & recovery: implicit xact
+ psql_like(
+ $io_method,
+ $psql,
+ "batch_start() leak & cleanup in implicit xact",
+ qq(SELECT batch_start()),
+ qr/^$/,
+ qr/open AIO batch at end/,
+ "$io_method: leaky batch_start() warns");
+
+ # leak warning & recovery: explicit xact
+ psql_like(
+ $io_method,
+ $psql,
+ "batch_start() leak & cleanup in explicit xact",
+ qq(BEGIN; SELECT batch_start(); COMMIT;),
+ qr/^$/,
+ qr/open AIO batch at end/,
+ "$io_method: leaky batch_start() warns");
+
+
+ # leak warning & recovery: explicit xact, rollback
+ #
+ # XXX: This doesn't fail right now, due to not getting a chance to do
+ # something at transaction command commit. That's not a correctness issue,
+ # it just means it's a bit harder to find buggy code.
+ #psql_like($io_method, $psql,
+ # "batch_start() leak & cleanup after abort",
+ # qq(BEGIN; SELECT batch_start(); ROLLBACK;),
+ # qr/^$/,
+ # qr/open AIO batch at end/, "$io_method: leaky batch_start() warns");
+
+ # no warning, batch closed in same command
+ psql_like(
+ $io_method,
+ $psql,
+ "batch_start(), batch_end() works",
+ qq(SELECT batch_start() UNION ALL SELECT batch_end()),
+ qr/^$/,
+ qr/^$/,
+ "$io_method: batch_start(), batch_end()");
+
+ $psql->quit();
+}
+
+# Test that simple cases of invalid pages are reported
+sub test_io_error
+{
+ my $io_method = shift;
+ my $node = shift;
+ my ($ret, $output);
+
+ my $psql = $node->background_psql('postgres', on_error_stop => 0);
+
+ $psql->query_safe(
+ qq(
+CREATE TEMPORARY TABLE tmp_corr(data int not null);
+INSERT INTO tmp_corr SELECT generate_series(1, 10000);
+SELECT modify_rel_block('tmp_corr', 1, corrupt_header=>true);
+));
+
+ foreach my $tblname (qw(tbl_corr tmp_corr))
+ {
+ my $invalid_page_re =
+ $tblname eq 'tbl_corr'
+ ? qr/invalid page in block 1 of relation base\/\d+\/\d+/
+ : qr/invalid page in block 1 of relation base\/\d+\/t\d+_\d+/;
+
+ # verify the error is reported in custom C code
+ psql_like(
+ $io_method,
+ $psql,
+ "read_rel_block_ll() of $tblname page",
+ qq(SELECT read_rel_block_ll('$tblname', 1)),
+ qr/^$/,
+ $invalid_page_re);
+
+ # verify the error is reported for bufmgr reads, seq scan
+ psql_like(
+ $io_method, $psql,
+ "sequential scan of $tblname block fails",
+ qq(SELECT count(*) FROM $tblname),
+ qr/^$/, $invalid_page_re);
+
+ # verify the error is reported for bufmgr reads, tid scan
+ psql_like(
+ $io_method,
+ $psql,
+ "tid scan of $tblname block fails",
+ qq(SELECT count(*) FROM $tblname WHERE ctid = '(1, 1)'),
+ qr/^$/,
+ $invalid_page_re);
+ }
+
+ $psql->quit();
+}
+
+# Test interplay between StartBufferIO and TerminateBufferIO
+sub test_startwait_io
+{
+ my $io_method = shift;
+ my $node = shift;
+ my ($ret, $output);
+
+ my $psql_a = $node->background_psql('postgres', on_error_stop => 0);
+ my $psql_b = $node->background_psql('postgres', on_error_stop => 0);
+
+
+ ### Verify behavior for normal tables
+
+ # create a buffer we can play around with
+ my $buf_id = psql_like(
+ $io_method, $psql_a,
+ "creation of toy buffer succeeds",
+ qq(SELECT buffer_create_toy('tbl_ok', 1)),
+ qr/^\d+$/, qr/^$/);
+
+ # check that one backend can perform StartBufferIO
+ psql_like(
+ $io_method,
+ $psql_a,
+ "first StartBufferIO",
+ qq(SELECT buffer_call_start_io($buf_id, for_input=>true, nowait=>false);),
+ qr/^t$/,
+ qr/^$/);
+
+ # but not twice on the same buffer (non-waiting)
+ psql_like(
+ $io_method,
+ $psql_a,
+ "second StartBufferIO fails, same session",
+ qq(SELECT buffer_call_start_io($buf_id, for_input=>true, nowait=>true);),
+ qr/^f$/,
+ qr/^$/);
+ psql_like(
+ $io_method,
+ $psql_b,
+ "second StartBufferIO fails, other session",
+ qq(SELECT buffer_call_start_io($buf_id, for_input=>true, nowait=>true);),
+ qr/^f$/,
+ qr/^$/);
+
+ # start io in a different session, will block
+ query_wait_block(
+ $io_method,
+ $node,
+ $psql_b,
+ "blocking start buffer io",
+ qq(SELECT buffer_call_start_io($buf_id, for_input=>true, nowait=>false);),
+ "BufferIo");
+
+ # Terminate the IO, without marking it as success, this should trigger the
+ # waiting session to be able to start the io
+ psql_like(
+ $io_method,
+ $psql_a,
+ "blocking start buffer io, terminating io, not valid",
+ qq(SELECT buffer_call_terminate_io($buf_id, for_input=>true, succeed=>false, io_error=>false, release_aio=>false)),
+ qr/^$/,
+ qr/^$/);
+
+
+ # Because the IO was terminated, but not marked as valid, second session should get the right to start io
+ pump_until($psql_b->{run}, $psql_b->{timeout}, \$psql_b->{stdout}, qr/t/);
+ ok(1, "$io_method: blocking start buffer io, can start io");
+
+ # terminate the IO again
+ $psql_b->query_safe(
+ qq(SELECT buffer_call_terminate_io($buf_id, for_input=>true, succeed=>false, io_error=>false, release_aio=>false);)
+ );
+
+
+ # same as the above scenario, but mark IO as having succeeded
+ psql_like(
+ $io_method,
+ $psql_a,
+ "blocking buffer io w/ success: first start buffer io",
+ qq(SELECT buffer_call_start_io($buf_id, for_input=>true, nowait=>false);),
+ qr/^t$/,
+ qr/^$/);
+
+ # start io in a different session, will block
+ query_wait_block(
+ $io_method,
+ $node,
+ $psql_b,
+ "blocking start buffer io",
+ qq(SELECT buffer_call_start_io($buf_id, for_input=>true, nowait=>false);),
+ "BufferIo");
+
+ # Terminate the IO, marking it as success
+ psql_like(
+ $io_method,
+ $psql_a,
+ "blocking start buffer io, terminating io, valid",
+ qq(SELECT buffer_call_terminate_io($buf_id, for_input=>true, succeed=>true, io_error=>false, release_aio=>false)),
+ qr/^$/,
+ qr/^$/);
+
+ # Because the IO was terminated, and marked as valid, second session should complete but not need io
+ pump_until($psql_b->{run}, $psql_b->{timeout}, \$psql_b->{stdout}, qr/f/);
+ ok(1, "$io_method: blocking start buffer io, no need to start io");
+
+ # buffer is valid now, make it invalid again
+ $psql_a->query_safe(qq(SELECT buffer_create_toy('tbl_ok', 1);));
+
+
+ ### Verify behavior for temporary tables
+
+ # Can't unfortunately share the code with the normal table case, there are
+ # too many behavioral differences.
+
+ # create a buffer we can play around with
+ $psql_a->query_safe(
+ qq(
+CREATE TEMPORARY TABLE tmp_ok(data int not null);
+INSERT INTO tmp_ok SELECT generate_series(1, 10000);
+));
+ $buf_id = $psql_a->query_safe(qq(SELECT buffer_create_toy('tmp_ok', 3);));
+
+ # check that one backend can perform StartLocalBufferIO
+ psql_like(
+ $io_method,
+ $psql_a,
+ "first StartLocalBufferIO",
+ qq(SELECT buffer_call_start_io($buf_id, for_input=>true, nowait=>true);),
+ qr/^t$/,
+ qr/^$/);
+
+ # Because local buffers don't use IO_IN_PROGRESS, a second StartLocalBufer
+ # succeeds as well. This test mostly serves as a documentation of that
+ # fact. If we had actually started IO, it'd be different.
+ psql_like(
+ $io_method,
+ $psql_a,
+ "second StartLocalBufferIO succeeds, same session",
+ qq(SELECT buffer_call_start_io($buf_id, for_input=>true, nowait=>true);),
+ qr/^t$/,
+ qr/^$/);
+
+ # Terminate the IO again, without marking it as a success
+ $psql_a->query_safe(
+ qq(SELECT buffer_call_terminate_io($buf_id, for_input=>true, succeed=>false, io_error=>false, release_aio=>false);)
+ );
+ psql_like(
+ $io_method,
+ $psql_a,
+ "StartLocalBufferIO after not marking valid succeeds, same session",
+ qq(SELECT buffer_call_start_io($buf_id, for_input=>true, nowait=>true);),
+ qr/^t$/,
+ qr/^$/);
+
+ # Terminate the IO again, marking it as a success
+ $psql_a->query_safe(
+ qq(SELECT buffer_call_terminate_io($buf_id, for_input=>true, succeed=>true, io_error=>false, release_aio=>false);)
+ );
+
+ # Now another StartLocalBufferIO should fail, this time because the buffer
+ # is already valid.
+ psql_like(
+ $io_method,
+ $psql_a,
+ "StartLocalBufferIO after marking valid fails",
+ qq(SELECT buffer_call_start_io($buf_id, for_input=>true, nowait=>false);),
+ qr/^f$/,
+ qr/^$/);
+
+ $psql_a->quit();
+ $psql_b->quit();
+}
+
+# Test that if the backend issuing a read doesn't wait for the IO's
+# completion, another backend can complete the IO
+sub test_complete_foreign
+{
+ my $io_method = shift;
+ my $node = shift;
+ my ($ret, $output);
+
+ my $psql_a = $node->background_psql('postgres', on_error_stop => 0);
+ my $psql_b = $node->background_psql('postgres', on_error_stop => 0);
+
+ # Issue IO without waiting for completion, then sleep
+ $psql_a->query_safe(
+ qq(SELECT read_rel_block_ll('tbl_ok', 1, wait_complete=>false);));
+
+ # Check that another backend can read the relevant block
+ psql_like(
+ $io_method,
+ $psql_b,
+ "completing read started by sleeping backend",
+ qq(SELECT count(*) FROM tbl_ok WHERE ctid = '(1,1)' LIMIT 1),
+ qr/^1$/,
+ qr/^$/);
+
+ # Issue IO without waiting for completion, then exit.
+ $psql_a->query_safe(
+ qq(SELECT read_rel_block_ll('tbl_ok', 1, wait_complete=>false);));
+ $psql_a->reconnect_and_clear();
+
+ # Check that another backend can read the relevant block. This verifies
+ # that the exiting backend left the AIO in a sane state.
+ psql_like(
+ $io_method,
+ $psql_b,
+ "read buffer started by exited backend",
+ qq(SELECT count(*) FROM tbl_ok WHERE ctid = '(1,1)' LIMIT 1),
+ qr/^1$/,
+ qr/^$/);
+
+ # Read a tbl_corr block, then sleep. The other session will retry the IO
+ # and also fail. The easiest thing to verify that seems to be to check
+ # that both are in the log.
+ my $log_location = -s $node->logfile;
+ $psql_a->query_safe(
+ qq(SELECT read_rel_block_ll('tbl_corr', 1, wait_complete=>false);));
+
+ psql_like(
+ $io_method,
+ $psql_b,
+ "completing read of tbl_corr block started by other backend",
+ qq(SELECT count(*) FROM tbl_corr WHERE ctid = '(1,1)' LIMIT 1),
+ qr/^$/,
+ qr/invalid page in block/);
+
+ # The log message issued for the read_rel_block_ll() should be logged as a LOG
+ $node->wait_for_log(qr/LOG[^\n]+invalid page in/, $log_location);
+ ok(1,
+ "$io_method: completing read of tbl_corr block started by other backend: LOG message for background read"
+ );
+
+ # But for the SELECT, it should be an ERROR
+ $log_location =
+ $node->wait_for_log(qr/ERROR[^\n]+invalid page in/, $log_location);
+ ok(1,
+ "$io_method: completing read of tbl_corr block started by other backend: ERROR message for foreground read"
+ );
+
+ $psql_a->quit();
+ $psql_b->quit();
+}
+
+# Test that we deal correctly with FDs being closed while IO is in progress
+sub test_close_fd
+{
+ my $io_method = shift;
+ my $node = shift;
+ my ($ret, $output);
+
+ my $psql = $node->background_psql('postgres', on_error_stop => 0);
+
+ psql_like(
+ $io_method,
+ $psql,
+ "close all FDs after read, waiting for results",
+ qq(
+ SELECT read_rel_block_ll('tbl_ok', 1,
+ wait_complete=>true,
+ batchmode_enter=>true,
+ smgrreleaseall=>true,
+ batchmode_exit=>true
+ );),
+ qr/^$/,
+ qr/^$/);
+
+ psql_like(
+ $io_method,
+ $psql,
+ "close all FDs after read, no waiting",
+ qq(
+ SELECT read_rel_block_ll('tbl_ok', 1,
+ wait_complete=>false,
+ batchmode_enter=>true,
+ smgrreleaseall=>true,
+ batchmode_exit=>true
+ );),
+ qr/^$/,
+ qr/^$/);
+
+ # Check that another backend can read the relevant block
+ psql_like(
+ $io_method,
+ $psql,
+ "close all FDs after read, no waiting, query works",
+ qq(SELECT count(*) FROM tbl_ok WHERE ctid = '(1,1)' LIMIT 1),
+ qr/^1$/,
+ qr/^$/);
+
+ $psql->quit();
+}
+
+# Tests using injection points. Mostly to exercise had IO errors that are
+# hard to trigger without using injection points.
+sub test_inject
+{
+ my $io_method = shift;
+ my $node = shift;
+ my ($ret, $output);
+
+ my $psql = $node->background_psql('postgres', on_error_stop => 0);
+
+ # injected what we'd expect
+ $psql->query_safe(qq(SELECT inj_io_short_read_attach(8192);));
+ $psql->query_safe(qq(SELECT invalidate_rel_block('tbl_ok', 2);));
+ psql_like(
+ $io_method, $psql,
+ "injection point not triggering failure",
+ qq(SELECT count(*) FROM tbl_ok WHERE ctid = '(2, 1)'),
+ qr/^1$/, qr/^$/);
+
+ # injected a read shorter than a single block, expecting error
+ $psql->query_safe(qq(SELECT inj_io_short_read_attach(17);));
+ $psql->query_safe(qq(SELECT invalidate_rel_block('tbl_ok', 2);));
+ psql_like(
+ $io_method,
+ $psql,
+ "single block short read fails",
+ qq(SELECT count(*) FROM tbl_ok WHERE ctid = '(2, 1)'),
+ qr/^$/,
+ qr/ERROR:.*could not read blocks 2\.\.2 in file "base\/.*": read only 0 of 8192 bytes/
+ );
+
+ # shorten multi-block read to a single block, should retry
+ my $inval_query = qq(SELECT invalidate_rel_block('tbl_ok', 0);
+SELECT invalidate_rel_block('tbl_ok', 1);
+SELECT invalidate_rel_block('tbl_ok', 2);
+SELECT invalidate_rel_block('tbl_ok', 3);
+/* gap */
+SELECT invalidate_rel_block('tbl_ok', 5);
+SELECT invalidate_rel_block('tbl_ok', 6);
+SELECT invalidate_rel_block('tbl_ok', 7);
+SELECT invalidate_rel_block('tbl_ok', 8););
+
+ $psql->query_safe($inval_query);
+ $psql->query_safe(qq(SELECT inj_io_short_read_attach(8192);));
+ psql_like(
+ $io_method, $psql,
+ "multi block short read (1 block) is retried",
+ qq(SELECT count(*) FROM tbl_ok),
+ qr/^10000$/, qr/^$/);
+
+ # shorten multi-block read to two blocks, should retry
+ $psql->query_safe($inval_query);
+ $psql->query_safe(qq(SELECT inj_io_short_read_attach(8192*2);));
+
+ psql_like(
+ $io_method, $psql,
+ "multi block short read (2 blocks) is retried",
+ qq(SELECT count(*) FROM tbl_ok),
+ qr/^10000$/, qr/^$/);
+
+ # verify that page verification errors are detected even as part of a
+ # shortened multi-block read (tbl_corr, block 1 is corrupted)
+ $psql->query_safe(
+ qq(
+SELECT invalidate_rel_block('tbl_corr', 0);
+SELECT invalidate_rel_block('tbl_corr', 1);
+SELECT invalidate_rel_block('tbl_corr', 2);
+SELECT inj_io_short_read_attach(8192);
+ ));
+
+ psql_like(
+ $io_method,
+ $psql,
+ "shortened multi-block read detects invalid page",
+ qq(SELECT count(*) FROM tbl_corr WHERE ctid < '(2, 1)'),
+ qr/^$/,
+ qr/ERROR:.*invalid page in block 1 of relation base\/.*/);
+
+ # trigger a hard error, should error out
+ $psql->query_safe(
+ qq(
+SELECT inj_io_short_read_attach(-errno_from_string('EIO'));
+SELECT invalidate_rel_block('tbl_ok', 2);
+ ));
+
+ psql_like(
+ $io_method,
+ $psql,
+ "first hard IO error is reported",
+ qq(SELECT count(*) FROM tbl_ok),
+ qr/^$/,
+ qr/ERROR:.*could not read blocks 2\.\.2 in file \"base\/.*\": Input\/output error/
+ );
+
+ psql_like(
+ $io_method,
+ $psql,
+ "second hard IO error is reported",
+ qq(SELECT count(*) FROM tbl_ok),
+ qr/^$/,
+ qr/ERROR:.*could not read blocks 2\.\.2 in file \"base\/.*\": Input\/output error/
+ );
+
+ $psql->query_safe(qq(SELECT inj_io_short_read_detach()));
+
+ # now the IO should be ok.
+ psql_like(
+ $io_method, $psql,
+ "recovers after hard error",
+ qq(SELECT count(*) FROM tbl_ok),
+ qr/^10000$/, qr/^$/);
+
+ $psql->quit();
+}
+
+# Tests using injection points, only for io_method=worker.
+#
+# io_method=worker has the special case of needing to reopen files. That can
+# in theory fail, because the file could be gone. That's a hard path to test
+# for real, so we use an injection point to trigger it.
+sub test_inject_worker
+{
+ my $io_method = shift;
+ my $node = shift;
+ my ($ret, $output);
+
+ my $psql = $node->background_psql('postgres', on_error_stop => 0);
+
+ # trigger a failure to reopen, should error out, but should recover
+ $psql->query_safe(
+ qq(
+SELECT inj_io_reopen_attach();
+SELECT invalidate_rel_block('tbl_ok', 1);
+ ));
+
+ psql_like(
+ $io_method,
+ $psql,
+ "failure to open: detected",
+ qq(SELECT count(*) FROM tbl_ok),
+ qr/^$/,
+ qr/ERROR:.*could not read blocks 1\.\.1 in file "base\/.*": No such file or directory/
+ );
+
+ $psql->query_safe(qq(SELECT inj_io_reopen_detach();));
+
+ # check that we indeed recover
+ psql_like(
+ $io_method, $psql,
+ "failure to open: recovers",
+ qq(SELECT count(*) FROM tbl_ok),
+ qr/^10000$/, qr/^$/);
+
+ $psql->quit();
+}
+
+# Verify that we handle a relation getting removed (due to a rollback or a
+# DROP TABLE) while IO is ongoing for that table.
+sub test_invalidate
+{
+ my $io_method = shift;
+ my $node = shift;
+
+ my $psql = $node->background_psql('postgres', on_error_stop => 0);
+
+ foreach my $persistency (qw(normal unlogged temporary))
+ {
+ my $sql_persistency = $persistency eq 'normal' ? '' : $persistency;
+ my $tblname = $persistency . '_transactional';
+
+ my $create_sql = qq(
+CREATE $sql_persistency TABLE $tblname (id int not null, data text not null) WITH (AUTOVACUUM_ENABLED = false);
+INSERT INTO $tblname(id, data) SELECT generate_series(1, 10000) as id, repeat('a', 200);
+);
+
+ # Verify that outstanding read IO does not cause problems with
+ # AbortTransaction -> smgrDoPendingDeletes -> smgrdounlinkall -> ...
+ # -> Invalidate[Local]Buffer.
+ $psql->query_safe("BEGIN; $create_sql;");
+ $psql->query_safe(
+ qq(
+SELECT read_rel_block_ll('$tblname', 1, wait_complete=>false);
+));
+ psql_like(
+ $io_method,
+ $psql,
+ "rollback of newly created $persistency table with outstanding IO",
+ qq(ROLLBACK),
+ qr/^$/,
+ qr/^$/);
+
+ # Verify that outstanding read IO does not cause problems with
+ # CommitTransaction -> smgrDoPendingDeletes -> smgrdounlinkall -> ...
+ # -> Invalidate[Local]Buffer.
+ $psql->query_safe("BEGIN; $create_sql; COMMIT;");
+ $psql->query_safe(
+ qq(
+BEGIN;
+SELECT read_rel_block_ll('$tblname', 1, wait_complete=>false);
+));
+
+ psql_like(
+ $io_method, $psql,
+ "drop $persistency table with outstanding IO",
+ qq(DROP TABLE $tblname),
+ qr/^$/, qr/^$/);
+
+ psql_like($io_method, $psql,
+ "commit after drop $persistency table with outstanding IO",
+ qq(COMMIT), qr/^$/, qr/^$/);
+ }
+
+ $psql->quit();
+}
+
+# Test behavior related to ZERO_ON_ERROR and zero_damaged_pages
+sub test_zero
+{
+ my $io_method = shift;
+ my $node = shift;
+
+ my $psql_a = $node->background_psql('postgres', on_error_stop => 0);
+
+ foreach my $persistency (qw(normal temporary))
+ {
+ my $sql_persistency = $persistency eq 'normal' ? '' : $persistency;
+
+ $psql_a->query_safe(
+ qq(
+CREATE $sql_persistency TABLE tbl_zero(id int) WITH (AUTOVACUUM_ENABLED = false);
+INSERT INTO tbl_zero SELECT generate_series(1, 10000);
+));
+
+ $psql_a->query_safe(
+ qq(
+SELECT modify_rel_block('tbl_zero', 0, corrupt_header=>true);
+));
+
+ # Check that page validity errors are detected
+ psql_like(
+ $io_method,
+ $psql_a,
+ "$persistency: test reading of invalid block 0",
+ qq(
+SELECT read_rel_block_ll('tbl_zero', 0, zero_on_error=>false)),
+ qr/^$/,
+ qr/^psql:<stdin>:\d+: ERROR: invalid page in block 0 of relation base\/.*\/.*$/
+ );
+
+ # Check that page validity errors are zeroed
+ psql_like(
+ $io_method,
+ $psql_a,
+ "$persistency: test zeroing of invalid block 0",
+ qq(
+SELECT read_rel_block_ll('tbl_zero', 0, zero_on_error=>true)),
+ qr/^$/,
+ qr/^psql:<stdin>:\d+: WARNING: invalid page in block 0 of relation base\/.*\/.*; zeroing out page$/
+ );
+
+ # And that once the corruption is fixed, we can read again
+ $psql_a->query(
+ qq(
+SELECT modify_rel_block('tbl_zero', 0, zero=>true);
+));
+ $psql_a->{stderr} = '';
+
+ psql_like(
+ $io_method,
+ $psql_a,
+ "$persistency: test re-read of block 0",
+ qq(
+SELECT read_rel_block_ll('tbl_zero', 0, zero_on_error=>false)),
+ qr/^$/,
+ qr/^$/);
+
+ # Check a page validity error in another block, to ensure we report
+ # the correct block number
+ $psql_a->query_safe(
+ qq(
+SELECT modify_rel_block('tbl_zero', 3, corrupt_header=>true);
+));
+ psql_like(
+ $io_method,
+ $psql_a,
+ "$persistency: test zeroing of invalid block 3",
+ qq(SELECT read_rel_block_ll('tbl_zero', 3, zero_on_error=>true);),
+ qr/^$/,
+ qr/^psql:<stdin>:\d+: WARNING: invalid page in block 3 of relation base\/.*\/.*; zeroing out page$/
+ );
+
+
+ # Check a page validity error in another block, to ensure we report
+ # the correct block number
+ $psql_a->query_safe(
+ qq(
+SELECT modify_rel_block('tbl_zero', 2, corrupt_header=>true);
+SELECT modify_rel_block('tbl_zero', 3, corrupt_header=>true);
+));
+ # First test error
+ psql_like(
+ $io_method,
+ $psql_a,
+ "$persistency: test reading of invalid block 2,3 in larger read",
+ qq(SELECT read_rel_block_ll('tbl_zero', 1, nblocks=>4, zero_on_error=>false)),
+ qr/^$/,
+ qr/^psql:<stdin>:\d+: ERROR: 2 invalid pages among blocks 1..4 of relation base\/.*\/.*\nDETAIL: Block 2 held first invalid page\.\nHINT:[^\n]+$/
+ );
+
+ # Then test zeroing via ZERO_ON_ERROR flag
+ psql_like(
+ $io_method,
+ $psql_a,
+ "$persistency: test zeroing of invalid block 2,3 in larger read, ZERO_ON_ERROR",
+ qq(SELECT read_rel_block_ll('tbl_zero', 1, nblocks=>4, zero_on_error=>true)),
+ qr/^$/,
+ qr/^psql:<stdin>:\d+: WARNING: zeroing out 2 invalid pages among blocks 1..4 of relation base\/.*\/.*\nDETAIL: Block 2 held first invalid page\.\nHINT:[^\n]+$/
+ );
+
+ # Then test zeroing vio zero_damaged_pages
+ psql_like(
+ $io_method,
+ $psql_a,
+ "$persistency: test zeroing of invalid block 2,3 in larger read, zero_damaged_pages",
+ qq(
+BEGIN;
+SET LOCAL zero_damaged_pages = true;
+SELECT read_rel_block_ll('tbl_zero', 1, nblocks=>4, zero_on_error=>false)
+COMMIT;
+),
+ qr/^$/,
+ qr/^psql:<stdin>:\d+: WARNING: zeroing out 2 invalid pages among blocks 1..4 of relation base\/.*\/.*\nDETAIL: Block 2 held first invalid page\.\nHINT:[^\n]+$/
+ );
+
+ $psql_a->query_safe(qq(COMMIT));
+
+
+ # Verify that bufmgr.c IO detects page validity errors
+ $psql_a->query(
+ qq(
+SELECT invalidate_rel_block('tbl_zero', g.i)
+FROM generate_series(0, 15) g(i);
+SELECT modify_rel_block('tbl_zero', 3, zero=>true);
+));
+ $psql_a->{stderr} = '';
+
+ psql_like(
+ $io_method,
+ $psql_a,
+ "$persistency: verify reading zero_damaged_pages=off",
+ qq(
+SELECT count(*) FROM tbl_zero),
+ qr/^$/,
+ qr/^psql:<stdin>:\d+: ERROR: invalid page in block 2 of relation base\/.*\/.*$/
+ );
+
+ # Verify that bufmgr.c IO zeroes out pages with page validity errors
+ psql_like(
+ $io_method,
+ $psql_a,
+ "$persistency: verify zero_damaged_pages=on",
+ qq(
+BEGIN;
+SET LOCAL zero_damaged_pages = true;
+SELECT count(*) FROM tbl_zero;
+COMMIT;
+),
+ qr/^\d+$/,
+ qr/^psql:<stdin>:\d+: WARNING: invalid page in block 2 of relation base\/.*\/.*$/
+ );
+
+
+ # Clean up
+ $psql_a->query_safe(
+ qq(
+DROP TABLE tbl_zero;
+));
+ }
+}
+
+
+# Run all tests that are supported for all io_methods
+sub test_generic
+{
+ my $io_method = shift;
+ my $node = shift;
+
+ is($node->safe_psql('postgres', 'SHOW io_method'),
+ $io_method, "$io_method: io_method set correctly");
+
+ $node->safe_psql(
+ 'postgres', qq(
+CREATE EXTENSION test_aio;
+CREATE TABLE tbl_corr(data int not null) WITH (AUTOVACUUM_ENABLED = false);
+CREATE TABLE tbl_ok(data int not null) WITH (AUTOVACUUM_ENABLED = false);
+
+INSERT INTO tbl_corr SELECT generate_series(1, 10000);
+INSERT INTO tbl_ok SELECT generate_series(1, 10000);
+SELECT grow_rel('tbl_corr', 16);
+SELECT grow_rel('tbl_ok', 16);
+
+SELECT modify_rel_block('tbl_corr', 1, corrupt_header=>true);
+CHECKPOINT;
+));
+
+ test_handle($io_method, $node);
+ test_io_error($io_method, $node);
+ test_batchmode($io_method, $node);
+ test_startwait_io($io_method, $node);
+ test_complete_foreign($io_method, $node);
+ test_close_fd($io_method, $node);
+ test_invalidate($io_method, $node);
+ test_zero($io_method, $node);
+
+ SKIP:
+ {
+ skip 'Injection points not supported by this build', 1
+ unless $ENV{enable_injection_points} eq 'yes';
+ test_inject($io_method, $node);
+ }
+}
diff --git a/src/test/modules/test_aio/t/002_io_workers.pl b/src/test/modules/test_aio/t/002_io_workers.pl
new file mode 100644
index 00000000000..af5fae15ea7
--- /dev/null
+++ b/src/test/modules/test_aio/t/002_io_workers.pl
@@ -0,0 +1,125 @@
+# Copyright (c) 2025, PostgreSQL Global Development Group
+
+use strict;
+use warnings FATAL => 'all';
+
+use PostgreSQL::Test::Cluster;
+use PostgreSQL::Test::Utils;
+use Test::More;
+use List::Util qw(shuffle);
+
+
+my $node = PostgreSQL::Test::Cluster->new('worker');
+$node->init();
+$node->append_conf(
+ 'postgresql.conf', qq(
+io_method=worker
+));
+
+$node->start();
+
+# Test changing the number of I/O worker processes while also evaluating the
+# handling of their termination.
+test_number_of_io_workers_dynamic($node);
+
+$node->stop();
+
+done_testing();
+
+
+sub test_number_of_io_workers_dynamic
+{
+ my $node = shift;
+
+ my $prev_worker_count = $node->safe_psql('postgres', 'SHOW io_workers');
+
+ # Verify that worker count can't be set to 0
+ change_number_of_io_workers($node, 0, $prev_worker_count, 1);
+
+ # Verify that worker count can't be set to 33 (above the max)
+ change_number_of_io_workers($node, 33, $prev_worker_count, 1);
+
+ # Try changing IO workers to a random value and verify that the worker
+ # count ends up as expected. Always test the min/max of workers.
+ #
+ # Valid range for io_workers is [1, 32]. 8 tests in total seems
+ # reasonable.
+ my @io_workers_range = shuffle(1 ... 32);
+ foreach my $worker_count (1, 32, @io_workers_range[ 0, 6 ])
+ {
+ $prev_worker_count =
+ change_number_of_io_workers($node, $worker_count,
+ $prev_worker_count, 0);
+ }
+}
+
+sub change_number_of_io_workers
+{
+ my $node = shift;
+ my $worker_count = shift;
+ my $prev_worker_count = shift;
+ my $expect_failure = shift;
+ my ($result, $stdout, $stderr);
+
+ ($result, $stdout, $stderr) =
+ $node->psql('postgres', "ALTER SYSTEM SET io_workers = $worker_count");
+ $node->safe_psql('postgres', 'SELECT pg_reload_conf()');
+
+ if ($expect_failure)
+ {
+ ok( $stderr =~
+ /$worker_count is outside the valid range for parameter "io_workers"/,
+ "updating number of io_workers to $worker_count failed, as expected"
+ );
+
+ return $prev_worker_count;
+ }
+ else
+ {
+ is( $node->safe_psql('postgres', 'SHOW io_workers'),
+ $worker_count,
+ "updating number of io_workers from $prev_worker_count to $worker_count"
+ );
+
+ check_io_worker_count($node, $worker_count);
+ terminate_io_worker($node, $worker_count);
+ check_io_worker_count($node, $worker_count);
+
+ return $worker_count;
+ }
+}
+
+sub terminate_io_worker
+{
+ my $node = shift;
+ my $worker_count = shift;
+ my ($pid, $ret);
+
+ # Select a random io worker
+ $pid = $node->safe_psql(
+ 'postgres',
+ qq(SELECT pid FROM pg_stat_activity WHERE
+ backend_type = 'io worker' ORDER BY RANDOM() LIMIT 1));
+
+ # terminate IO worker with SIGINT
+ is(PostgreSQL::Test::Utils::system_log('pg_ctl', 'kill', 'INT', $pid),
+ 0, "random io worker process signalled with INT");
+
+ # Check that worker exits
+ ok( $node->poll_query_until(
+ 'postgres',
+ qq(SELECT COUNT(*) FROM pg_stat_activity WHERE pid = $pid), '0'),
+ "random io worker process exited after signal");
+}
+
+sub check_io_worker_count
+{
+ my $node = shift;
+ my $worker_count = shift;
+
+ ok( $node->poll_query_until(
+ 'postgres',
+ qq(SELECT COUNT(*) FROM pg_stat_activity WHERE backend_type = 'io worker'),
+ $worker_count),
+ "io worker count is $worker_count");
+}
diff --git a/src/test/modules/test_aio/test_aio--1.0.sql b/src/test/modules/test_aio/test_aio--1.0.sql
new file mode 100644
index 00000000000..c668d516f61
--- /dev/null
+++ b/src/test/modules/test_aio/test_aio--1.0.sql
@@ -0,0 +1,107 @@
+/* src/test/modules/test_aio/test_aio--1.0.sql */
+
+-- complain if script is sourced in psql, rather than via CREATE EXTENSION
+\echo Use "CREATE EXTENSION test_aio" to load this file. \quit
+
+
+CREATE FUNCTION errno_from_string(sym text)
+RETURNS pg_catalog.int4 STRICT
+AS 'MODULE_PATHNAME' LANGUAGE C;
+
+
+CREATE FUNCTION grow_rel(rel regclass, nblocks int)
+RETURNS pg_catalog.void STRICT
+AS 'MODULE_PATHNAME' LANGUAGE C;
+
+
+CREATE FUNCTION modify_rel_block(rel regclass, blockno int,
+ zero bool DEFAULT false,
+ corrupt_header bool DEFAULT false)
+RETURNS pg_catalog.void STRICT
+AS 'MODULE_PATHNAME' LANGUAGE C;
+
+CREATE FUNCTION read_rel_block_ll(
+ rel regclass,
+ blockno int,
+ nblocks int DEFAULT 1,
+ wait_complete bool DEFAULT true,
+ batchmode_enter bool DEFAULT false,
+ smgrreleaseall bool DEFAULT false,
+ batchmode_exit bool DEFAULT false,
+ zero_on_error bool DEFAULT false)
+RETURNS pg_catalog.void STRICT
+AS 'MODULE_PATHNAME' LANGUAGE C;
+
+CREATE FUNCTION invalidate_rel_block(rel regclass, blockno int)
+RETURNS pg_catalog.void STRICT
+AS 'MODULE_PATHNAME' LANGUAGE C;
+
+CREATE FUNCTION buffer_create_toy(rel regclass, blockno int4)
+RETURNS pg_catalog.int4 STRICT
+AS 'MODULE_PATHNAME' LANGUAGE C;
+
+CREATE FUNCTION buffer_call_start_io(buffer int, for_input bool, nowait bool)
+RETURNS pg_catalog.bool STRICT
+AS 'MODULE_PATHNAME' LANGUAGE C;
+
+CREATE FUNCTION buffer_call_terminate_io(buffer int, for_input bool, succeed bool, io_error bool, release_aio bool)
+RETURNS pg_catalog.void STRICT
+AS 'MODULE_PATHNAME' LANGUAGE C;
+
+
+
+/*
+ * Handle related functions
+ */
+CREATE FUNCTION handle_get_and_error()
+RETURNS pg_catalog.void STRICT
+AS 'MODULE_PATHNAME' LANGUAGE C;
+
+CREATE FUNCTION handle_get_twice()
+RETURNS pg_catalog.void STRICT
+AS 'MODULE_PATHNAME' LANGUAGE C;
+
+CREATE FUNCTION handle_get()
+RETURNS pg_catalog.void STRICT
+AS 'MODULE_PATHNAME' LANGUAGE C;
+
+CREATE FUNCTION handle_get_release()
+RETURNS pg_catalog.void STRICT
+AS 'MODULE_PATHNAME' LANGUAGE C;
+
+CREATE FUNCTION handle_release_last()
+RETURNS pg_catalog.void STRICT
+AS 'MODULE_PATHNAME' LANGUAGE C;
+
+
+/*
+ * Batchmode related functions
+ */
+CREATE FUNCTION batch_start()
+RETURNS pg_catalog.void STRICT
+AS 'MODULE_PATHNAME' LANGUAGE C;
+
+CREATE FUNCTION batch_end()
+RETURNS pg_catalog.void STRICT
+AS 'MODULE_PATHNAME' LANGUAGE C;
+
+
+
+/*
+ * Injection point related functions
+ */
+CREATE FUNCTION inj_io_short_read_attach(result int)
+RETURNS pg_catalog.void STRICT
+AS 'MODULE_PATHNAME' LANGUAGE C;
+
+CREATE FUNCTION inj_io_short_read_detach()
+RETURNS pg_catalog.void STRICT
+AS 'MODULE_PATHNAME' LANGUAGE C;
+
+CREATE FUNCTION inj_io_reopen_attach()
+RETURNS pg_catalog.void STRICT
+AS 'MODULE_PATHNAME' LANGUAGE C;
+
+CREATE FUNCTION inj_io_reopen_detach()
+RETURNS pg_catalog.void STRICT
+AS 'MODULE_PATHNAME' LANGUAGE C;
diff --git a/src/test/modules/test_aio/test_aio.c b/src/test/modules/test_aio/test_aio.c
new file mode 100644
index 00000000000..c7e857800dc
--- /dev/null
+++ b/src/test/modules/test_aio/test_aio.c
@@ -0,0 +1,748 @@
+/*-------------------------------------------------------------------------
+ *
+ * test_aio.c
+ * Helpers to write tests for AIO
+ *
+ * This module provides interface functions for C functionality to SQL, to
+ * make it possible to test AIO related behavior in a targeted way from SQL.
+ * It'd not generally be safe to export these functions to SQL, but for a test
+ * that's fine.
+ *
+ * Copyright (c) 2020-2025, PostgreSQL Global Development Group
+ *
+ * IDENTIFICATION
+ * src/test/modules/test_aio/test_aio.c
+ *
+ *-------------------------------------------------------------------------
+ */
+
+#include "postgres.h"
+
+#include "access/relation.h"
+#include "fmgr.h"
+#include "storage/aio.h"
+#include "storage/aio_internal.h"
+#include "storage/buf_internals.h"
+#include "storage/bufmgr.h"
+#include "storage/ipc.h"
+#include "storage/lwlock.h"
+#include "utils/builtins.h"
+#include "utils/injection_point.h"
+#include "utils/rel.h"
+
+
+PG_MODULE_MAGIC;
+
+
+typedef struct InjIoErrorState
+{
+ bool enabled_short_read;
+ bool enabled_reopen;
+
+ bool short_read_result_set;
+ int short_read_result;
+} InjIoErrorState;
+
+static InjIoErrorState * inj_io_error_state;
+
+/* Shared memory init callbacks */
+static shmem_request_hook_type prev_shmem_request_hook = NULL;
+static shmem_startup_hook_type prev_shmem_startup_hook = NULL;
+
+
+static PgAioHandle *last_handle;
+
+
+
+static void
+test_aio_shmem_request(void)
+{
+ if (prev_shmem_request_hook)
+ prev_shmem_request_hook();
+
+ RequestAddinShmemSpace(sizeof(InjIoErrorState));
+}
+
+static void
+test_aio_shmem_startup(void)
+{
+ bool found;
+
+ if (prev_shmem_startup_hook)
+ prev_shmem_startup_hook();
+
+ /* Create or attach to the shared memory state */
+ LWLockAcquire(AddinShmemInitLock, LW_EXCLUSIVE);
+
+ inj_io_error_state = ShmemInitStruct("injection_points",
+ sizeof(InjIoErrorState),
+ &found);
+
+ if (!found)
+ {
+ /*
+ * First time through, so initialize. This is shared with the dynamic
+ * initialization using a DSM.
+ */
+ inj_io_error_state->enabled_short_read = false;
+ inj_io_error_state->enabled_reopen = false;
+
+#ifdef USE_INJECTION_POINTS
+ InjectionPointAttach("AIO_PROCESS_COMPLETION_BEFORE_SHARED",
+ "test_aio",
+ "inj_io_short_read",
+ NULL,
+ 0);
+ InjectionPointLoad("AIO_PROCESS_COMPLETION_BEFORE_SHARED");
+
+ InjectionPointAttach("AIO_WORKER_AFTER_REOPEN",
+ "test_aio",
+ "inj_io_reopen",
+ NULL,
+ 0);
+ InjectionPointLoad("AIO_WORKER_AFTER_REOPEN");
+
+#endif
+ }
+ else
+ {
+#ifdef USE_INJECTION_POINTS
+ InjectionPointLoad("AIO_PROCESS_COMPLETION_BEFORE_SHARED");
+ InjectionPointLoad("AIO_WORKER_AFTER_REOPEN");
+ elog(LOG, "injection point loaded");
+#endif
+ }
+
+ LWLockRelease(AddinShmemInitLock);
+}
+
+void
+_PG_init(void)
+{
+ if (!process_shared_preload_libraries_in_progress)
+ return;
+
+ /* Shared memory initialization */
+ prev_shmem_request_hook = shmem_request_hook;
+ shmem_request_hook = test_aio_shmem_request;
+ prev_shmem_startup_hook = shmem_startup_hook;
+ shmem_startup_hook = test_aio_shmem_startup;
+}
+
+
+PG_FUNCTION_INFO_V1(errno_from_string);
+Datum
+errno_from_string(PG_FUNCTION_ARGS)
+{
+ const char *sym = text_to_cstring(PG_GETARG_TEXT_PP(0));
+
+ if (strcmp(sym, "EIO") == 0)
+ PG_RETURN_INT32(EIO);
+ else if (strcmp(sym, "EAGAIN") == 0)
+ PG_RETURN_INT32(EAGAIN);
+ else if (strcmp(sym, "EINTR") == 0)
+ PG_RETURN_INT32(EINTR);
+ else if (strcmp(sym, "ENOSPC") == 0)
+ PG_RETURN_INT32(ENOSPC);
+ else if (strcmp(sym, "EROFS") == 0)
+ PG_RETURN_INT32(EROFS);
+
+ ereport(ERROR,
+ errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ errmsg_internal("%s is not a supported errno value", sym));
+ PG_RETURN_INT32(0);
+}
+
+
+PG_FUNCTION_INFO_V1(grow_rel);
+Datum
+grow_rel(PG_FUNCTION_ARGS)
+{
+ Oid relid = PG_GETARG_OID(0);
+ uint32 nblocks = PG_GETARG_UINT32(1);
+ Relation rel;
+#define MAX_BUFFERS_TO_EXTEND_BY 64
+ Buffer victim_buffers[MAX_BUFFERS_TO_EXTEND_BY];
+
+ rel = relation_open(relid, AccessExclusiveLock);
+
+ while (nblocks > 0)
+ {
+ uint32 extend_by_pages;
+
+ extend_by_pages = Min(nblocks, MAX_BUFFERS_TO_EXTEND_BY);
+
+ ExtendBufferedRelBy(BMR_REL(rel),
+ MAIN_FORKNUM,
+ NULL,
+ 0,
+ extend_by_pages,
+ victim_buffers,
+ &extend_by_pages);
+
+ nblocks -= extend_by_pages;
+
+ for (uint32 i = 0; i < extend_by_pages; i++)
+ {
+ ReleaseBuffer(victim_buffers[i]);
+ }
+ }
+
+ relation_close(rel, NoLock);
+
+ PG_RETURN_VOID();
+}
+
+PG_FUNCTION_INFO_V1(modify_rel_block);
+Datum
+modify_rel_block(PG_FUNCTION_ARGS)
+{
+ Oid relid = PG_GETARG_OID(0);
+ BlockNumber blkno = PG_GETARG_UINT32(1);
+ bool zero = PG_GETARG_BOOL(2);
+ bool corrupt_header = PG_GETARG_BOOL(3);
+ Relation rel;
+ Buffer buf;
+ Page page;
+ PageHeader ph;
+
+ rel = relation_open(relid, AccessExclusiveLock);
+
+ buf = ReadBufferExtended(rel, MAIN_FORKNUM, blkno,
+ RBM_ZERO_ON_ERROR, NULL);
+ page = BufferGetPage(buf);
+
+ LockBuffer(buf, BUFFER_LOCK_EXCLUSIVE);
+
+ if (zero)
+ memset(page, 0, BufferGetPageSize(buf));
+
+ if (PageIsEmpty(page) && corrupt_header)
+ PageInit(page, BufferGetPageSize(buf), 0);
+
+ ph = (PageHeader) page;
+
+ if (corrupt_header)
+ ph->pd_special = BLCKSZ + 1;
+
+ PageSetChecksumInplace(page, blkno);
+
+ smgrwrite(RelationGetSmgr(rel),
+ MAIN_FORKNUM, blkno, page, true);
+
+ LockBuffer(buf, BUFFER_LOCK_UNLOCK);
+
+ ReleaseBuffer(buf);
+
+ /*
+ * Don't want to have a buffer in-memory that's marked valid where the
+ * on-disk contents are invalid.
+ *
+ * NB: This is racy, better don't copy this to non-test code.
+ */
+ if (BufferIsLocal(buf))
+ InvalidateLocalBuffer(GetLocalBufferDescriptor(-buf - 1), true);
+ else
+ EvictUnpinnedBuffer(buf);
+
+ relation_close(rel, NoLock);
+
+ PG_RETURN_VOID();
+}
+
+/*
+ * Ensures a buffer for rel & blkno is in shared buffers, without actually
+ * caring about the buffer contents. Used to set up test scenarios.
+ */
+static Buffer
+create_toy_buffer(Relation rel, BlockNumber blkno)
+{
+ Buffer buf;
+ BufferDesc *buf_hdr;
+ uint32 buf_state;
+ bool was_pinned = false;
+
+ /* place buffer in shared buffers without erroring out */
+ buf = ReadBufferExtended(rel, MAIN_FORKNUM, blkno, RBM_ZERO_AND_LOCK, NULL);
+ LockBuffer(buf, BUFFER_LOCK_UNLOCK);
+
+ if (RelationUsesLocalBuffers(rel))
+ {
+ buf_hdr = GetLocalBufferDescriptor(-buf - 1);
+ buf_state = pg_atomic_read_u32(&buf_hdr->state);
+ }
+ else
+ {
+ buf_hdr = GetBufferDescriptor(buf - 1);
+ buf_state = LockBufHdr(buf_hdr);
+ }
+
+ /*
+ * We should be the only backend accessing this buffer. This is just a
+ * small bit of belt-and-suspenders defense, none of this code should ever
+ * run in a cluster with real data.
+ */
+ if (BUF_STATE_GET_REFCOUNT(buf_state) > 1)
+ was_pinned = true;
+ else
+ buf_state &= ~(BM_VALID | BM_DIRTY);
+
+ if (RelationUsesLocalBuffers(rel))
+ pg_atomic_unlocked_write_u32(&buf_hdr->state, buf_state);
+ else
+ UnlockBufHdr(buf_hdr, buf_state);
+
+ if (was_pinned)
+ elog(ERROR, "toy buffer %d was already pinned",
+ buf);
+
+ return buf;
+}
+
+/*
+ * A "low level" read. This does similar things to what
+ * StartReadBuffers()/WaitReadBuffers() do, but provides more control (and
+ * less sanity).
+ */
+PG_FUNCTION_INFO_V1(read_rel_block_ll);
+Datum
+read_rel_block_ll(PG_FUNCTION_ARGS)
+{
+ Oid relid = PG_GETARG_OID(0);
+ BlockNumber blkno = PG_GETARG_UINT32(1);
+ int nblocks = PG_GETARG_INT32(2);
+ bool wait_complete = PG_GETARG_BOOL(3);
+ bool batchmode_enter = PG_GETARG_BOOL(4);
+ bool call_smgrreleaseall = PG_GETARG_BOOL(5);
+ bool batchmode_exit = PG_GETARG_BOOL(6);
+ bool zero_on_error = PG_GETARG_BOOL(7);
+ Relation rel;
+ Buffer bufs[PG_IOV_MAX];
+ BufferDesc *buf_hdrs[PG_IOV_MAX];
+ Page pages[PG_IOV_MAX];
+ PgAioReturn ior;
+ PgAioHandle *ioh;
+ PgAioWaitRef iow;
+ SMgrRelation smgr;
+
+ if (nblocks <= 0 || nblocks > PG_IOV_MAX)
+ elog(ERROR, "nblocks is out of range");
+
+ rel = relation_open(relid, AccessExclusiveLock);
+
+ for (int i = 0; i < nblocks; i++)
+ {
+ bufs[i] = create_toy_buffer(rel, blkno + i);
+ pages[i] = BufferGetBlock(bufs[i]);
+ buf_hdrs[i] = BufferIsLocal(bufs[i]) ?
+ GetLocalBufferDescriptor(-bufs[i] - 1) :
+ GetBufferDescriptor(bufs[i] - 1);
+ }
+
+ ioh = pgaio_io_acquire(CurrentResourceOwner, &ior);
+ pgaio_io_get_wref(ioh, &iow);
+
+ smgr = RelationGetSmgr(rel);
+
+ if (RelationUsesLocalBuffers(rel))
+ {
+ for (int i = 0; i < nblocks; i++)
+ StartLocalBufferIO(buf_hdrs[i], true, false);
+ pgaio_io_set_flag(ioh, PGAIO_HF_REFERENCES_LOCAL);
+ }
+ else
+ {
+ for (int i = 0; i < nblocks; i++)
+ StartBufferIO(buf_hdrs[i], true, false);
+ }
+
+ pgaio_io_set_handle_data_32(ioh, (uint32 *) bufs, nblocks);
+
+ pgaio_io_register_callbacks(ioh,
+ RelationUsesLocalBuffers(rel) ?
+ PGAIO_HCB_LOCAL_BUFFER_READV :
+ PGAIO_HCB_SHARED_BUFFER_READV,
+ (zero_on_error | zero_damaged_pages) ?
+ READ_BUFFERS_ZERO_ON_ERROR : 0);
+
+ if (batchmode_enter)
+ pgaio_enter_batchmode();
+
+ elog(LOG, "about to smgrstartreadv");
+ smgrstartreadv(ioh, smgr, MAIN_FORKNUM, blkno,
+ (void *) pages, nblocks);
+
+ if (call_smgrreleaseall)
+ smgrreleaseall();
+
+ if (batchmode_exit)
+ pgaio_exit_batchmode();
+
+ for (int i = 0; i < nblocks; i++)
+ ReleaseBuffer(bufs[i]);
+
+ if (wait_complete)
+ {
+ pgaio_wref_wait(&iow);
+
+ if (ior.result.status != PGAIO_RS_OK)
+ pgaio_result_report(ior.result,
+ &ior.target_data,
+ ior.result.status == PGAIO_RS_ERROR ?
+ ERROR : WARNING);
+ }
+
+ relation_close(rel, NoLock);
+
+ PG_RETURN_VOID();
+}
+
+PG_FUNCTION_INFO_V1(invalidate_rel_block);
+Datum
+invalidate_rel_block(PG_FUNCTION_ARGS)
+{
+ Oid relid = PG_GETARG_OID(0);
+ BlockNumber blkno = PG_GETARG_UINT32(1);
+ Relation rel;
+ PrefetchBufferResult pr;
+ Buffer buf;
+
+ rel = relation_open(relid, AccessExclusiveLock);
+
+ /*
+ * This is a gross hack, but there's no other API exposed that allows to
+ * get a buffer ID without actually reading the block in.
+ */
+ pr = PrefetchBuffer(rel, MAIN_FORKNUM, blkno);
+ buf = pr.recent_buffer;
+
+ if (BufferIsValid(buf))
+ {
+ /* if the buffer contents aren't valid, this'll return false */
+ if (ReadRecentBuffer(rel->rd_locator, MAIN_FORKNUM, blkno, buf))
+ {
+ BufferDesc *buf_hdr = BufferIsLocal(buf) ?
+ GetLocalBufferDescriptor(-buf - 1)
+ : GetBufferDescriptor(buf - 1);
+
+ LockBuffer(buf, BUFFER_LOCK_EXCLUSIVE);
+
+ if (pg_atomic_read_u32(&buf_hdr->state) & BM_DIRTY)
+ {
+ if (BufferIsLocal(buf))
+ FlushLocalBuffer(buf_hdr, NULL);
+ else
+ FlushOneBuffer(buf);
+ }
+ LockBuffer(buf, BUFFER_LOCK_UNLOCK);
+ ReleaseBuffer(buf);
+
+ if (BufferIsLocal(buf))
+ InvalidateLocalBuffer(GetLocalBufferDescriptor(-buf - 1), true);
+ else if (!EvictUnpinnedBuffer(buf))
+ elog(ERROR, "couldn't evict");
+ }
+ }
+
+ relation_close(rel, AccessExclusiveLock);
+
+ PG_RETURN_VOID();
+}
+
+PG_FUNCTION_INFO_V1(buffer_create_toy);
+Datum
+buffer_create_toy(PG_FUNCTION_ARGS)
+{
+ Oid relid = PG_GETARG_OID(0);
+ BlockNumber blkno = PG_GETARG_UINT32(1);
+ Relation rel;
+ Buffer buf;
+
+ rel = relation_open(relid, AccessExclusiveLock);
+
+ buf = create_toy_buffer(rel, blkno);
+ ReleaseBuffer(buf);
+
+ relation_close(rel, NoLock);
+
+ PG_RETURN_INT32(buf);
+}
+
+PG_FUNCTION_INFO_V1(buffer_call_start_io);
+Datum
+buffer_call_start_io(PG_FUNCTION_ARGS)
+{
+ Buffer buf = PG_GETARG_INT32(0);
+ bool for_input = PG_GETARG_BOOL(1);
+ bool nowait = PG_GETARG_BOOL(2);
+ bool can_start;
+
+ if (BufferIsLocal(buf))
+ can_start = StartLocalBufferIO(GetLocalBufferDescriptor(-buf - 1),
+ for_input, nowait);
+ else
+ can_start = StartBufferIO(GetBufferDescriptor(buf - 1),
+ for_input, nowait);
+
+ /*
+ * For tests we don't want the resowner release preventing us from
+ * orchestrating odd scenarios.
+ */
+ if (can_start && !BufferIsLocal(buf))
+ ResourceOwnerForgetBufferIO(CurrentResourceOwner,
+ buf);
+
+ ereport(LOG,
+ errmsg("buffer %d after StartBufferIO: %s",
+ buf, DebugPrintBufferRefcount(buf)),
+ errhidestmt(true), errhidecontext(true));
+
+ PG_RETURN_BOOL(can_start);
+}
+
+PG_FUNCTION_INFO_V1(buffer_call_terminate_io);
+Datum
+buffer_call_terminate_io(PG_FUNCTION_ARGS)
+{
+ Buffer buf = PG_GETARG_INT32(0);
+ bool for_input = PG_GETARG_BOOL(1);
+ bool succeed = PG_GETARG_BOOL(2);
+ bool io_error = PG_GETARG_BOOL(3);
+ bool release_aio = PG_GETARG_BOOL(4);
+ bool clear_dirty = false;
+ uint32 set_flag_bits = 0;
+
+ if (io_error)
+ set_flag_bits |= BM_IO_ERROR;
+
+ if (for_input)
+ {
+ clear_dirty = false;
+
+ if (succeed)
+ set_flag_bits |= BM_VALID;
+ }
+ else
+ {
+ if (succeed)
+ clear_dirty = true;
+ }
+
+ ereport(LOG,
+ errmsg("buffer %d before Terminate[Local]BufferIO: %s",
+ buf, DebugPrintBufferRefcount(buf)),
+ errhidestmt(true), errhidecontext(true));
+
+ if (BufferIsLocal(buf))
+ TerminateLocalBufferIO(GetLocalBufferDescriptor(-buf - 1),
+ clear_dirty, set_flag_bits, release_aio);
+ else
+ TerminateBufferIO(GetBufferDescriptor(buf - 1),
+ clear_dirty, set_flag_bits, false, release_aio);
+
+ ereport(LOG,
+ errmsg("buffer %d after Terminate[Local]BufferIO: %s",
+ buf, DebugPrintBufferRefcount(buf)),
+ errhidestmt(true), errhidecontext(true));
+
+ PG_RETURN_VOID();
+}
+
+PG_FUNCTION_INFO_V1(handle_get);
+Datum
+handle_get(PG_FUNCTION_ARGS)
+{
+ last_handle = pgaio_io_acquire(CurrentResourceOwner, NULL);
+
+ PG_RETURN_VOID();
+}
+
+PG_FUNCTION_INFO_V1(handle_release_last);
+Datum
+handle_release_last(PG_FUNCTION_ARGS)
+{
+ if (!last_handle)
+ elog(ERROR, "no handle");
+
+ pgaio_io_release(last_handle);
+
+ PG_RETURN_VOID();
+}
+
+PG_FUNCTION_INFO_V1(handle_get_and_error);
+Datum
+handle_get_and_error(PG_FUNCTION_ARGS)
+{
+ pgaio_io_acquire(CurrentResourceOwner, NULL);
+
+ elog(ERROR, "as you command");
+ PG_RETURN_VOID();
+}
+
+PG_FUNCTION_INFO_V1(handle_get_twice);
+Datum
+handle_get_twice(PG_FUNCTION_ARGS)
+{
+ pgaio_io_acquire(CurrentResourceOwner, NULL);
+ pgaio_io_acquire(CurrentResourceOwner, NULL);
+
+ PG_RETURN_VOID();
+}
+
+PG_FUNCTION_INFO_V1(handle_get_release);
+Datum
+handle_get_release(PG_FUNCTION_ARGS)
+{
+ PgAioHandle *handle;
+
+ handle = pgaio_io_acquire(CurrentResourceOwner, NULL);
+ pgaio_io_release(handle);
+
+ PG_RETURN_VOID();
+}
+
+PG_FUNCTION_INFO_V1(batch_start);
+Datum
+batch_start(PG_FUNCTION_ARGS)
+{
+ pgaio_enter_batchmode();
+ PG_RETURN_VOID();
+}
+
+PG_FUNCTION_INFO_V1(batch_end);
+Datum
+batch_end(PG_FUNCTION_ARGS)
+{
+ pgaio_exit_batchmode();
+ PG_RETURN_VOID();
+}
+
+#ifdef USE_INJECTION_POINTS
+extern PGDLLEXPORT void inj_io_short_read(const char *name, const void *private_data);
+extern PGDLLEXPORT void inj_io_reopen(const char *name, const void *private_data);
+
+void
+inj_io_short_read(const char *name, const void *private_data)
+{
+ PgAioHandle *ioh;
+
+ elog(LOG, "short read called: %d", inj_io_error_state->enabled_short_read);
+
+ if (inj_io_error_state->enabled_short_read)
+ {
+ ioh = pgaio_inj_io_get();
+
+ /*
+ * Only shorten reads that are actually longer than the target size,
+ * otherwise we can trigger over-reads.
+ */
+ if (inj_io_error_state->short_read_result_set
+ && ioh->op == PGAIO_OP_READV
+ && inj_io_error_state->short_read_result <= ioh->result)
+ {
+ struct iovec *iov = &pgaio_ctl->iovecs[ioh->iovec_off];
+ int32 old_result = ioh->result;
+ int32 new_result = inj_io_error_state->short_read_result;
+ int32 processed = 0;
+
+ ereport(LOG,
+ errmsg("short read, changing result from %d to %d",
+ old_result, new_result),
+ errhidestmt(true), errhidecontext(true));
+
+ /*
+ * The underlying IO actually completed OK, and thus the "invalid"
+ * portion of the IOV actually contains valid data. That can hide
+ * a lot of problems, e.g. if we were to wrongly mark a buffer,
+ * that wasn't read according to the shortened-read, IO as valid,
+ * the contents would look valid and we might miss a bug.
+ *
+ * To avoid that, iterate through the IOV and zero out the
+ * "failed" portion of the IO.
+ */
+ for (int i = 0; i < ioh->op_data.read.iov_length; i++)
+ {
+ if (processed + iov[i].iov_len <= new_result)
+ processed += iov[i].iov_len;
+ else if (processed <= new_result)
+ {
+ uint32 ok_part = new_result - processed;
+
+ memset((char *) iov[i].iov_base + ok_part, 0, iov[i].iov_len - ok_part);
+ processed += iov[i].iov_len;
+ }
+ else
+ {
+ memset((char *) iov[i].iov_base, 0, iov[i].iov_len);
+ }
+ }
+
+ ioh->result = new_result;
+ }
+ }
+}
+
+void
+inj_io_reopen(const char *name, const void *private_data)
+{
+ elog(LOG, "reopen called: %d", inj_io_error_state->enabled_reopen);
+
+ if (inj_io_error_state->enabled_reopen)
+ {
+ elog(ERROR, "injection point triggering failure to reopen ");
+ }
+}
+#endif
+
+PG_FUNCTION_INFO_V1(inj_io_short_read_attach);
+Datum
+inj_io_short_read_attach(PG_FUNCTION_ARGS)
+{
+#ifdef USE_INJECTION_POINTS
+ inj_io_error_state->enabled_short_read = true;
+ inj_io_error_state->short_read_result_set = !PG_ARGISNULL(0);
+ if (inj_io_error_state->short_read_result_set)
+ inj_io_error_state->short_read_result = PG_GETARG_INT32(0);
+#else
+ elog(ERROR, "injection points not supported");
+#endif
+
+ PG_RETURN_VOID();
+}
+
+PG_FUNCTION_INFO_V1(inj_io_short_read_detach);
+Datum
+inj_io_short_read_detach(PG_FUNCTION_ARGS)
+{
+#ifdef USE_INJECTION_POINTS
+ inj_io_error_state->enabled_short_read = false;
+#else
+ elog(ERROR, "injection points not supported");
+#endif
+ PG_RETURN_VOID();
+}
+
+PG_FUNCTION_INFO_V1(inj_io_reopen_attach);
+Datum
+inj_io_reopen_attach(PG_FUNCTION_ARGS)
+{
+#ifdef USE_INJECTION_POINTS
+ inj_io_error_state->enabled_reopen = true;
+#else
+ elog(ERROR, "injection points not supported");
+#endif
+
+ PG_RETURN_VOID();
+}
+
+PG_FUNCTION_INFO_V1(inj_io_reopen_detach);
+Datum
+inj_io_reopen_detach(PG_FUNCTION_ARGS)
+{
+#ifdef USE_INJECTION_POINTS
+ inj_io_error_state->enabled_reopen = false;
+#else
+ elog(ERROR, "injection points not supported");
+#endif
+ PG_RETURN_VOID();
+}
diff --git a/src/test/modules/test_aio/test_aio.control b/src/test/modules/test_aio/test_aio.control
new file mode 100644
index 00000000000..cd91c3ed16b
--- /dev/null
+++ b/src/test/modules/test_aio/test_aio.control
@@ -0,0 +1,3 @@
+comment = 'Test code for AIO'
+default_version = '1.0'
+module_pathname = '$libdir/test_aio'
--
2.48.1.76.g4e746b1a31.dirty
[text/x-diff] v2.13-0018-aio-Experimental-heuristics-to-increase-batchi.patch (1.6K, ../../o2dtdrlc2xqpjycsfuvv7u6zyrqj7jfxqu32et3lyelkwce2jy@3k3fwdiiu73v/19-v2.13-0018-aio-Experimental-heuristics-to-increase-batchi.patch)
download | inline diff:
From 491a5fce1a9c49a6dc7c4c79bf18378b437bcbc3 Mon Sep 17 00:00:00 2001
From: Andres Freund <andres@anarazel.de>
Date: Tue, 18 Mar 2025 14:40:06 -0400
Subject: [PATCH v2.13 18/28] aio: Experimental heuristics to increase batching
in read_stream.c
Author:
Reviewed-by:
Discussion: https://postgr.es/m/
Backpatch:
---
src/backend/storage/aio/read_stream.c | 24 ++++++++++++++++++++++++
1 file changed, 24 insertions(+)
diff --git a/src/backend/storage/aio/read_stream.c b/src/backend/storage/aio/read_stream.c
index 36c54fb695b..cec93129f58 100644
--- a/src/backend/storage/aio/read_stream.c
+++ b/src/backend/storage/aio/read_stream.c
@@ -404,6 +404,30 @@ read_stream_start_pending_read(ReadStream *stream)
static void
read_stream_look_ahead(ReadStream *stream)
{
+ /*
+ * Batch-submitting multiple IOs is more efficient than doing so
+ * one-by-one. If we just ramp up to the max, we'll only be allowed to
+ * submit one io_combine_limit sized IO. Defer submitting IO in that case.
+ *
+ * FIXME: This needs better heuristics.
+ */
+#if 1
+ if (!stream->sync_mode && stream->distance > (io_combine_limit * 8))
+ {
+ if (stream->pinned_buffers + stream->pending_read_nblocks > ((stream->distance * 3) / 4))
+ {
+#if 0
+ ereport(LOG,
+ errmsg("reduce reduce reduce: pinned: %d, pending: %d, distance: %d",
+ stream->pinned_buffers,
+ stream->pending_read_nblocks,
+ stream->distance));
+#endif
+ return;
+ }
+ }
+#endif
+
/*
* Allow amortizing the cost of submitting IO over multiple IOs. This
* requires that we don't do any operations that could lead to a deadlock
--
2.48.1.76.g4e746b1a31.dirty
[text/x-diff] v2.13-0019-aio-Implement-smgr-md-fd-write-support.patch (14.6K, ../../o2dtdrlc2xqpjycsfuvv7u6zyrqj7jfxqu32et3lyelkwce2jy@3k3fwdiiu73v/20-v2.13-0019-aio-Implement-smgr-md-fd-write-support.patch)
download | inline diff:
From bd752931f695be73e6789c68437575f2d0c6446b Mon Sep 17 00:00:00 2001
From: Andres Freund <andres@anarazel.de>
Date: Tue, 18 Mar 2025 14:40:06 -0400
Subject: [PATCH v2.13 19/28] aio: Implement smgr/md/fd write support
TODO:
- Right now the sync.c integration with smgr.c/md.c isn't properly safe to use
in a critical section
The only reason it doesn't immediately fail is that it's reasonably rare
that RegisterSyncRequest() fails *and* either:
- smgropen()->hash_search(HASH_ENTER) decides to resize the hash table, even
though the lookup is guaranteed to succeed for io_method=worker.
- an io_method=uring completion is run in a different backend and smgropen()
needs to build a new entry and thus needs to allocate memory
For a bit I thought this could be worked around easily enough by not doing
an smgropen() in mdsyncfiletag(), or adding a "fallible" smgropen() and
instead just opening the file directly. That actually does kinda solve the
problem, but only because the memory allocation in PathNameOpenFile()
uses malloc(), not palloc() and thus doesn't trigger
- temp_file_limit implementation
---
src/include/storage/fd.h | 1 +
src/include/storage/md.h | 5 +
src/include/storage/smgr.h | 5 +
src/backend/storage/aio/aio_callback.c | 1 +
src/backend/storage/file/fd.c | 28 ++++
src/backend/storage/smgr/md.c | 201 ++++++++++++++++++++++++-
src/backend/storage/smgr/smgr.c | 29 ++++
7 files changed, 269 insertions(+), 1 deletion(-)
diff --git a/src/include/storage/fd.h b/src/include/storage/fd.h
index b77d8e5e30e..2cc7c5a4761 100644
--- a/src/include/storage/fd.h
+++ b/src/include/storage/fd.h
@@ -112,6 +112,7 @@ extern int FilePrefetch(File file, off_t offset, off_t amount, uint32 wait_event
extern ssize_t FileReadV(File file, const struct iovec *iov, int iovcnt, off_t offset, uint32 wait_event_info);
extern ssize_t FileWriteV(File file, const struct iovec *iov, int iovcnt, off_t offset, uint32 wait_event_info);
extern int FileStartReadV(struct PgAioHandle *ioh, File file, int iovcnt, off_t offset, uint32 wait_event_info);
+extern int FileStartWriteV(struct PgAioHandle *ioh, File file, int iovcnt, off_t offset, uint32 wait_event_info);
extern int FileSync(File file, uint32 wait_event_info);
extern int FileZero(File file, off_t offset, off_t amount, uint32 wait_event_info);
extern int FileFallocate(File file, off_t offset, off_t amount, uint32 wait_event_info);
diff --git a/src/include/storage/md.h b/src/include/storage/md.h
index 9d7131eff43..47ae6c36c94 100644
--- a/src/include/storage/md.h
+++ b/src/include/storage/md.h
@@ -21,6 +21,7 @@
#include "storage/sync.h"
extern const PgAioHandleCallbacks aio_md_readv_cb;
+extern const PgAioHandleCallbacks aio_md_writev_cb;
/* md storage manager functionality */
extern void mdinit(void);
@@ -45,6 +46,10 @@ extern void mdstartreadv(PgAioHandle *ioh,
extern void mdwritev(SMgrRelation reln, ForkNumber forknum,
BlockNumber blocknum,
const void **buffers, BlockNumber nblocks, bool skipFsync);
+extern void mdstartwritev(PgAioHandle *ioh,
+ SMgrRelation reln, ForkNumber forknum,
+ BlockNumber blocknum,
+ const void **buffers, BlockNumber nblocks, bool skipFsync);
extern void mdwriteback(SMgrRelation reln, ForkNumber forknum,
BlockNumber blocknum, BlockNumber nblocks);
extern BlockNumber mdnblocks(SMgrRelation reln, ForkNumber forknum);
diff --git a/src/include/storage/smgr.h b/src/include/storage/smgr.h
index 856ebcda350..f00b3763ac9 100644
--- a/src/include/storage/smgr.h
+++ b/src/include/storage/smgr.h
@@ -108,6 +108,11 @@ extern void smgrwritev(SMgrRelation reln, ForkNumber forknum,
BlockNumber blocknum,
const void **buffers, BlockNumber nblocks,
bool skipFsync);
+extern void smgrstartwritev(PgAioHandle *ioh,
+ SMgrRelation reln, ForkNumber forknum,
+ BlockNumber blocknum,
+ const void **buffers, BlockNumber nblocks,
+ bool skipFsync);
extern void smgrwriteback(SMgrRelation reln, ForkNumber forknum,
BlockNumber blocknum, BlockNumber nblocks);
extern BlockNumber smgrnblocks(SMgrRelation reln, ForkNumber forknum);
diff --git a/src/backend/storage/aio/aio_callback.c b/src/backend/storage/aio/aio_callback.c
index c0063d6950e..67026f5f6b4 100644
--- a/src/backend/storage/aio/aio_callback.c
+++ b/src/backend/storage/aio/aio_callback.c
@@ -41,6 +41,7 @@ static const PgAioHandleCallbacksEntry aio_handle_cbs[] = {
CALLBACK_ENTRY(PGAIO_HCB_INVALID, aio_invalid_cb),
CALLBACK_ENTRY(PGAIO_HCB_MD_READV, aio_md_readv_cb),
+ CALLBACK_ENTRY(PGAIO_HCB_MD_WRITEV, aio_md_writev_cb),
CALLBACK_ENTRY(PGAIO_HCB_SHARED_BUFFER_READV, aio_shared_buffer_readv_cb),
diff --git a/src/backend/storage/file/fd.c b/src/backend/storage/file/fd.c
index 9e1cdf7ddfe..4aecf2cfb97 100644
--- a/src/backend/storage/file/fd.c
+++ b/src/backend/storage/file/fd.c
@@ -2348,6 +2348,34 @@ retry:
return returnCode;
}
+int
+FileStartWriteV(PgAioHandle *ioh, File file,
+ int iovcnt, off_t offset,
+ uint32 wait_event_info)
+{
+ int returnCode;
+ Vfd *vfdP;
+
+ Assert(FileIsValid(file));
+
+ DO_DB(elog(LOG, "FileStartWriteV: %d (%s) " INT64_FORMAT " %d",
+ file, VfdCache[file].fileName,
+ (int64) offset,
+ iovcnt));
+
+ returnCode = FileAccess(file);
+ if (returnCode < 0)
+ return returnCode;
+
+ vfdP = &VfdCache[file];
+
+ /* FIXME: think about / reimplement temp_file_limit */
+
+ pgaio_io_start_writev(ioh, vfdP->fd, iovcnt, offset);
+
+ return 0;
+}
+
int
FileSync(File file, uint32 wait_event_info)
{
diff --git a/src/backend/storage/smgr/md.c b/src/backend/storage/smgr/md.c
index eedba1e5794..f6bdc7e37bc 100644
--- a/src/backend/storage/smgr/md.c
+++ b/src/backend/storage/smgr/md.c
@@ -155,12 +155,19 @@ static BlockNumber _mdnblocks(SMgrRelation reln, ForkNumber forknum,
static PgAioResult md_readv_complete(PgAioHandle *ioh, PgAioResult prior_result, uint8 cb_data);
static void md_readv_report(PgAioResult result, const PgAioTargetData *target_data, int elevel);
+static PgAioResult md_writev_complete(PgAioHandle *ioh, PgAioResult prior_result, uint8 cb_data);
+static void md_writev_report(PgAioResult result, const PgAioTargetData *target_data, int elevel);
const PgAioHandleCallbacks aio_md_readv_cb = {
.complete_shared = md_readv_complete,
.report = md_readv_report,
};
+const PgAioHandleCallbacks aio_md_writev_cb = {
+ .complete_shared = md_writev_complete,
+ .report = md_writev_report,
+};
+
static inline int
_mdfd_open_flags(void)
@@ -1115,6 +1122,64 @@ mdwritev(SMgrRelation reln, ForkNumber forknum, BlockNumber blocknum,
}
}
+/*
+ * mdstartwritev() -- Asynchronous version of mdrwritev().
+ */
+void
+mdstartwritev(PgAioHandle *ioh,
+ SMgrRelation reln, ForkNumber forknum, BlockNumber blocknum,
+ const void **buffers, BlockNumber nblocks, bool skipFsync)
+{
+ off_t seekpos;
+ MdfdVec *v;
+ BlockNumber nblocks_this_segment;
+ struct iovec *iov;
+ int iovcnt;
+ int ret;
+
+ v = _mdfd_getseg(reln, forknum, blocknum, false,
+ EXTENSION_FAIL | EXTENSION_CREATE_RECOVERY);
+
+ seekpos = (off_t) BLCKSZ * (blocknum % ((BlockNumber) RELSEG_SIZE));
+
+ Assert(seekpos < (off_t) BLCKSZ * RELSEG_SIZE);
+
+ nblocks_this_segment =
+ Min(nblocks,
+ RELSEG_SIZE - (blocknum % ((BlockNumber) RELSEG_SIZE)));
+
+ if (nblocks_this_segment != nblocks)
+ elog(ERROR, "write crossing segment boundary");
+
+ iovcnt = pgaio_io_get_iovec(ioh, &iov);
+
+ Assert(nblocks <= iovcnt);
+
+ iovcnt = buffers_to_iovec(iov, unconstify(void **, buffers), nblocks_this_segment);
+
+ Assert(iovcnt <= nblocks_this_segment);
+
+ if (!(io_direct_flags & IO_DIRECT_DATA))
+ pgaio_io_set_flag(ioh, PGAIO_HF_BUFFERED);
+
+ pgaio_io_set_target_smgr(ioh,
+ reln,
+ forknum,
+ blocknum,
+ nblocks,
+ skipFsync);
+ pgaio_io_register_callbacks(ioh, PGAIO_HCB_MD_WRITEV, 0);
+
+ ret = FileStartWriteV(ioh, v->mdfd_vfd, iovcnt, seekpos, WAIT_EVENT_DATA_FILE_WRITE);
+ if (ret != 0)
+ ereport(ERROR,
+ (errcode_for_file_access(),
+ errmsg("could not start writing blocks %u..%u in file \"%s\": %m",
+ blocknum,
+ blocknum + nblocks_this_segment - 1,
+ FilePathName(v->mdfd_vfd))));
+}
+
/*
* mdwriteback() -- Tell the kernel to write pages back to storage.
@@ -1503,6 +1568,40 @@ register_dirty_segment(SMgrRelation reln, ForkNumber forknum, MdfdVec *seg)
}
}
+/*
+ * Like register_dirty_segment(), except for use by AIO. In the completion
+ * callback we don't have access to the MdfdVec (the completion callback might
+ * be executed in a different backend than the issuing backend), therefore we
+ * have to implement this slightly differently.
+ */
+static void
+register_dirty_segment_aio(RelFileLocator locator, ForkNumber forknum, uint64 segno)
+{
+ FileTag tag;
+
+ INIT_MD_FILETAG(tag, locator, forknum, segno);
+
+ /*
+ * Can't block here waiting for checkpointer to accept our sync request,
+ * as checkpointer might be waiting for this AIO to finish if offloaded to
+ * a worker.
+ */
+ if (!RegisterSyncRequest(&tag, SYNC_REQUEST, false /* retryOnError */ ))
+ {
+ char path[MAXPGPATH];
+
+ ereport(DEBUG1,
+ (errmsg_internal("could not forward fsync request because request queue is full")));
+
+ /* reuse mdsyncfiletag() to avoid duplicating code */
+ if (mdsyncfiletag(&tag, path))
+ ereport(data_sync_elevel(ERROR),
+ (errcode_for_file_access(),
+ errmsg("could not fsync file \"%s\": %m",
+ path)));
+ }
+}
+
/*
* register_unlink_segment() -- Schedule a file to be deleted after next checkpoint
*/
@@ -1997,7 +2096,7 @@ md_readv_complete(PgAioHandle *ioh, PgAioResult prior_result, uint8 cb_data)
* AIO error reporting callback for mdstartreadv().
*
* Errors are encoded as follows:
- * - PgAioResult.error_data != 0 encodes IO that failed with errno != 0
+ * - PgAioResult.error_data != 0 encodes IO that failed with that errno
* - PgAioResult.error_data == 0 encodes IO that didn't read all data
*/
static void
@@ -2037,3 +2136,103 @@ md_readv_report(PgAioResult result, const PgAioTargetData *td, int elevel)
td->smgr.nblocks * (size_t) BLCKSZ));
}
}
+
+/*
+ * AIO completion callback for mdstartwritev().
+ */
+static PgAioResult
+md_writev_complete(PgAioHandle *ioh, PgAioResult prior_result, uint8 cb_data)
+{
+ PgAioTargetData *td = pgaio_io_get_target_data(ioh);
+ PgAioResult result = prior_result;
+
+ if (prior_result.result < 0)
+ {
+ result.status = PGAIO_RS_ERROR;
+ result.id = PGAIO_HCB_MD_WRITEV;
+ /* For "hard" errors, track the error number in error_data */
+ result.error_data = -prior_result.result;
+ result.result = 0;
+
+ pgaio_result_report(result, td, LOG);
+
+ return result;
+ }
+
+ /*
+ * As explained above smgrstartwritev(), the smgr API operates on the
+ * level of blocks, rather than bytes. Convert.
+ */
+ result.result /= BLCKSZ;
+
+ Assert(result.result <= td->smgr.nblocks);
+
+ if (result.result == 0)
+ {
+ /* consider 0 blocks written a failure */
+ result.status = PGAIO_RS_ERROR;
+ result.id = PGAIO_HCB_MD_WRITEV;
+ result.error_data = 0;
+
+ pgaio_result_report(result, td, LOG);
+
+ return result;
+ }
+
+ if (result.status != PGAIO_RS_ERROR &&
+ result.result < td->smgr.nblocks)
+ {
+ /* partial writes should be retried at upper level */
+ result.status = PGAIO_RS_PARTIAL;
+ result.id = PGAIO_HCB_MD_WRITEV;
+ }
+
+ if (!td->smgr.skip_fsync)
+ register_dirty_segment_aio(td->smgr.rlocator, td->smgr.forkNum,
+ td->smgr.blockNum / ((BlockNumber) RELSEG_SIZE));
+
+ return result;
+}
+
+/*
+ * AIO error reporting callback for mdstartwritev().
+ */
+static void
+md_writev_report(PgAioResult result, const PgAioTargetData *td, int elevel)
+{
+ RelPathStr path;
+
+ path = relpathbackend(td->smgr.rlocator,
+ td->smgr.is_temp ? MyProcNumber : INVALID_PROC_NUMBER,
+ td->smgr.forkNum);
+
+ if (result.error_data != 0)
+ {
+ errno = result.error_data; /* for errcode_for_file_access() */
+
+ ereport(elevel,
+ errcode_for_file_access(),
+ errmsg("could not write blocks %u..%u in file \"%s\": %m",
+ td->smgr.blockNum,
+ td->smgr.blockNum + td->smgr.nblocks,
+ path.str)
+ );
+ }
+ else
+ {
+ /*
+ * NB: This will typically only be output in debug messages, while
+ * retrying a partial IO.
+ */
+ ereport(elevel,
+ errcode(ERRCODE_DATA_CORRUPTED),
+ errmsg("could not write blocks %u..%u in file \"%s\": wrote only %zu of %zu bytes",
+ td->smgr.blockNum,
+ td->smgr.blockNum + td->smgr.nblocks - 1,
+ path.str,
+ result.result * (size_t) BLCKSZ,
+ td->smgr.nblocks * (size_t) BLCKSZ
+ )
+ );
+ }
+}
diff --git a/src/backend/storage/smgr/smgr.c b/src/backend/storage/smgr/smgr.c
index bf8f57b410a..e56d6a2597b 100644
--- a/src/backend/storage/smgr/smgr.c
+++ b/src/backend/storage/smgr/smgr.c
@@ -115,6 +115,11 @@ typedef struct f_smgr
BlockNumber blocknum,
const void **buffers, BlockNumber nblocks,
bool skipFsync);
+ void (*smgr_startwritev) (PgAioHandle *ioh,
+ SMgrRelation reln, ForkNumber forknum,
+ BlockNumber blocknum,
+ const void **buffers, BlockNumber nblocks,
+ bool skipFsync);
void (*smgr_writeback) (SMgrRelation reln, ForkNumber forknum,
BlockNumber blocknum, BlockNumber nblocks);
BlockNumber (*smgr_nblocks) (SMgrRelation reln, ForkNumber forknum);
@@ -142,6 +147,7 @@ static const f_smgr smgrsw[] = {
.smgr_readv = mdreadv,
.smgr_startreadv = mdstartreadv,
.smgr_writev = mdwritev,
+ .smgr_startwritev = mdstartwritev,
.smgr_writeback = mdwriteback,
.smgr_nblocks = mdnblocks,
.smgr_truncate = mdtruncate,
@@ -787,6 +793,29 @@ smgrwritev(SMgrRelation reln, ForkNumber forknum, BlockNumber blocknum,
RESUME_INTERRUPTS();
}
+/*
+ * smgrstartwritev() -- asynchronous version of smgrwritev()
+ *
+ * This starts an asynchronous writev IO using the IO handle `ioh`. Other than
+ * `ioh` all parameters are the same as smgrwritev().
+ *
+ * Completion callbacks above smgr will be passed the result as the number of
+ * successfully written blocks if the write [partially] succeeds. This
+ * maintains the abstraction that smgr operates on the level of blocks, rather
+ * than bytes.
+ */
+void
+smgrstartwritev(PgAioHandle *ioh,
+ SMgrRelation reln, ForkNumber forknum, BlockNumber blocknum,
+ const void **buffers, BlockNumber nblocks, bool skipFsync)
+{
+ HOLD_INTERRUPTS();
+ smgrsw[reln->smgr_which].smgr_startwritev(ioh,
+ reln, forknum, blocknum, buffers,
+ nblocks, skipFsync);
+ RESUME_INTERRUPTS();
+}
+
/*
* smgrwriteback() -- Trigger kernel writeback for the supplied range of
* blocks.
--
2.48.1.76.g4e746b1a31.dirty
[text/x-diff] v2.13-0020-aio-Add-bounce-buffers.patch (23.2K, ../../o2dtdrlc2xqpjycsfuvv7u6zyrqj7jfxqu32et3lyelkwce2jy@3k3fwdiiu73v/21-v2.13-0020-aio-Add-bounce-buffers.patch)
download | inline diff:
From 7d49c6782d4335519e3a06d262d3bbb557a7bfc0 Mon Sep 17 00:00:00 2001
From: Andres Freund <andres@anarazel.de>
Date: Tue, 18 Mar 2025 14:40:06 -0400
Subject: [PATCH v2.13 20/28] aio: Add bounce buffers
---
src/include/storage/aio.h | 15 ++
src/include/storage/aio_internal.h | 34 ++++
src/include/storage/aio_types.h | 2 +
src/include/utils/resowner.h | 2 +
src/backend/storage/aio/README.md | 27 +++
src/backend/storage/aio/aio.c | 178 ++++++++++++++++++
src/backend/storage/aio/aio_init.c | 123 ++++++++++++
src/backend/utils/misc/guc_tables.c | 13 ++
src/backend/utils/misc/postgresql.conf.sample | 2 +
src/backend/utils/resowner/resowner.c | 25 ++-
src/test/modules/test_aio/test_aio--1.0.sql | 21 +++
src/test/modules/test_aio/test_aio.c | 55 ++++++
src/tools/pgindent/typedefs.list | 1 +
13 files changed, 496 insertions(+), 2 deletions(-)
diff --git a/src/include/storage/aio.h b/src/include/storage/aio.h
index b182c0c78ca..f9a6ac8c625 100644
--- a/src/include/storage/aio.h
+++ b/src/include/storage/aio.h
@@ -349,6 +349,20 @@ extern bool pgaio_have_staged(void);
+/* --------------------------------------------------------------------------------
+ * Bounce Buffers
+ * --------------------------------------------------------------------------------
+ */
+
+extern PgAioBounceBuffer *pgaio_bounce_buffer_get(void);
+extern void pgaio_io_assoc_bounce_buffer(PgAioHandle *ioh, PgAioBounceBuffer *bb);
+extern uint32 pgaio_bounce_buffer_id(PgAioBounceBuffer *bb);
+extern void pgaio_bounce_buffer_release(PgAioBounceBuffer *bb);
+extern char *pgaio_bounce_buffer_buffer(PgAioBounceBuffer *bb);
+extern void pgaio_bounce_buffer_release_resowner(struct dlist_node *bb_node, bool on_error);
+
+
+
/* --------------------------------------------------------------------------------
* Other
* --------------------------------------------------------------------------------
@@ -361,6 +375,7 @@ extern void pgaio_closing_fd(int fd);
/* GUCs */
extern PGDLLIMPORT int io_method;
extern PGDLLIMPORT int io_max_concurrency;
+extern PGDLLIMPORT int io_bounce_buffers;
#endif /* AIO_H */
diff --git a/src/include/storage/aio_internal.h b/src/include/storage/aio_internal.h
index 7f18da2c856..833f97361a1 100644
--- a/src/include/storage/aio_internal.h
+++ b/src/include/storage/aio_internal.h
@@ -127,6 +127,12 @@ struct PgAioHandle
/* raw result of the IO operation */
int32 result;
+ /*
+ * List of bounce_buffers owned by IO. It would suffice to use an index
+ * based linked list here.
+ */
+ slist_head bounce_buffers;
+
/**
* In which list the handle is registered, depends on the state:
* - IDLE, in per-backend list
@@ -182,11 +188,24 @@ struct PgAioHandle
};
+/* typedef is in aio_types.h */
+struct PgAioBounceBuffer
+{
+ slist_node node;
+ struct ResourceOwnerData *resowner;
+ dlist_node resowner_node;
+ char *buffer;
+};
+
+
typedef struct PgAioBackend
{
/* index into PgAioCtl->io_handles */
uint32 io_handle_off;
+ /* index into PgAioCtl->bounce_buffers */
+ uint32 bounce_buffers_off;
+
/* IO Handles that currently are not used */
dclist_head idle_ios;
@@ -217,6 +236,12 @@ typedef struct PgAioBackend
* IOs being appended at the end.
*/
dclist_head in_flight_ios;
+
+ /* Bounce Buffers that currently are not used */
+ slist_head idle_bbs;
+
+ /* see handed_out_io */
+ PgAioBounceBuffer *handed_out_bb;
} PgAioBackend;
@@ -244,6 +269,15 @@ typedef struct PgAioCtl
uint32 io_handle_count;
PgAioHandle *io_handles;
+
+ /*
+ * To perform AIO on buffers that are not located in shared memory (either
+ * because they are not in shared memory or because we need to operate on
+ * a copy, as e.g. the case for writes when checksums are in use)
+ */
+ uint32 bounce_buffers_count;
+ PgAioBounceBuffer *bounce_buffers;
+ char *bounce_buffers_data;
} PgAioCtl;
diff --git a/src/include/storage/aio_types.h b/src/include/storage/aio_types.h
index 9db5d776b61..134986d3af1 100644
--- a/src/include/storage/aio_types.h
+++ b/src/include/storage/aio_types.h
@@ -119,4 +119,6 @@ typedef struct PgAioReturn
} PgAioReturn;
+typedef struct PgAioBounceBuffer PgAioBounceBuffer;
+
#endif /* AIO_TYPES_H */
diff --git a/src/include/utils/resowner.h b/src/include/utils/resowner.h
index aede4bfc820..7e2ec224169 100644
--- a/src/include/utils/resowner.h
+++ b/src/include/utils/resowner.h
@@ -168,5 +168,7 @@ extern void ResourceOwnerForgetLock(ResourceOwner owner, struct LOCALLOCK *local
struct dlist_node;
extern void ResourceOwnerRememberAioHandle(ResourceOwner owner, struct dlist_node *ioh_node);
extern void ResourceOwnerForgetAioHandle(ResourceOwner owner, struct dlist_node *ioh_node);
+extern void ResourceOwnerRememberAioBounceBuffer(ResourceOwner owner, struct dlist_node *bb_node);
+extern void ResourceOwnerForgetAioBounceBuffer(ResourceOwner owner, struct dlist_node *bb_node);
#endif /* RESOWNER_H */
diff --git a/src/backend/storage/aio/README.md b/src/backend/storage/aio/README.md
index ddd59404a59..534aaad22bc 100644
--- a/src/backend/storage/aio/README.md
+++ b/src/backend/storage/aio/README.md
@@ -406,6 +406,33 @@ shared memory no less!), completion callbacks instead have to encode errors in
a more compact format that can be converted into an error message.
+### AIO Bounce Buffers
+
+For some uses of AIO there is no convenient memory location as the source /
+destination of an AIO. E.g. when data checksums are enabled, writes from
+shared buffers currently cannot be done directly from shared buffers, as a
+shared buffer lock still allows some modification, e.g., for hint bits (see
+`FlushBuffer()`). If the write were done in-place, such modifications can
+cause the checksum to fail.
+
+For synchronous IO this is solved by copying the buffer to separate memory
+before computing the checksum and using that copy as the source buffer for the
+AIO.
+
+However, for AIO that is not a workable solution:
+- Instead of a single buffer many buffers are required, as many IOs might be
+ in flight
+- When using the [worker method](#worker), the source/target of IO needs to be
+ in shared memory, otherwise the workers won't be able to access the memory.
+
+The AIO subsystem addresses this by providing a limited number of bounce
+buffers that can be used as the source / target for IO. A bounce buffer can be
+acquired with `pgaio_bounce_buffer_get()` and multiple bounce buffers can be
+associated with an AIO Handle with `pgaio_io_assoc_bounce_buffer()`.
+
+Bounce buffers are automatically released when the IO completes.
+
+
## Helpers
Using the low-level AIO API introduces too much complexity to do so all over
diff --git a/src/backend/storage/aio/aio.c b/src/backend/storage/aio/aio.c
index 86f7250b7a5..cff48964d07 100644
--- a/src/backend/storage/aio/aio.c
+++ b/src/backend/storage/aio/aio.c
@@ -62,6 +62,8 @@ static PgAioHandle *pgaio_io_from_wref(PgAioWaitRef *iow, uint64 *ref_generation
static const char *pgaio_io_state_get_name(PgAioHandleState s);
static void pgaio_io_wait(PgAioHandle *ioh, uint64 ref_generation);
+static void pgaio_bounce_buffer_wait_for_free(void);
+
/* Options for io_method. */
const struct config_enum_entry io_method_options[] = {
@@ -76,6 +78,7 @@ const struct config_enum_entry io_method_options[] = {
/* GUCs */
int io_method = DEFAULT_IO_METHOD;
int io_max_concurrency = -1;
+int io_bounce_buffers = -1;
/* global control for AIO */
PgAioCtl *pgaio_ctl;
@@ -662,6 +665,21 @@ pgaio_io_reclaim(PgAioHandle *ioh)
if (ioh->state != PGAIO_HS_HANDED_OUT)
dclist_delete_from(&pgaio_my_backend->in_flight_ios, &ioh->node);
+ /* reclaim all associated bounce buffers */
+ if (!slist_is_empty(&ioh->bounce_buffers))
+ {
+ slist_mutable_iter it;
+
+ slist_foreach_modify(it, &ioh->bounce_buffers)
+ {
+ PgAioBounceBuffer *bb = slist_container(PgAioBounceBuffer, node, it.cur);
+
+ slist_delete_current(&it);
+
+ slist_push_head(&pgaio_my_backend->idle_bbs, &bb->node);
+ }
+ }
+
if (ioh->resowner)
{
ResourceOwnerForgetAioHandle(ioh->resowner, &ioh->resowner_node);
@@ -1046,6 +1064,166 @@ pgaio_submit_staged(void)
+/* --------------------------------------------------------------------------------
+ * Functions primarily related to PgAioBounceBuffer
+ * --------------------------------------------------------------------------------
+ */
+
+PgAioBounceBuffer *
+pgaio_bounce_buffer_get(void)
+{
+ PgAioBounceBuffer *bb = NULL;
+ slist_node *node;
+
+ if (pgaio_my_backend->handed_out_bb != NULL)
+ elog(ERROR, "can only hand out one BB");
+
+ /*
+ * XXX: It probably is not a good idea to have bounce buffers be per
+ * backend, that's a fair bit of memory.
+ */
+ if (slist_is_empty(&pgaio_my_backend->idle_bbs))
+ {
+ pgaio_bounce_buffer_wait_for_free();
+ }
+
+ node = slist_pop_head_node(&pgaio_my_backend->idle_bbs);
+ bb = slist_container(PgAioBounceBuffer, node, node);
+
+ pgaio_my_backend->handed_out_bb = bb;
+
+ bb->resowner = CurrentResourceOwner;
+ ResourceOwnerRememberAioBounceBuffer(bb->resowner, &bb->resowner_node);
+
+ return bb;
+}
+
+void
+pgaio_io_assoc_bounce_buffer(PgAioHandle *ioh, PgAioBounceBuffer *bb)
+{
+ if (pgaio_my_backend->handed_out_bb != bb)
+ elog(ERROR, "can only assign handed out BB");
+ pgaio_my_backend->handed_out_bb = NULL;
+
+ /*
+ * There can be many bounce buffers assigned in case of vectorized IOs.
+ */
+ slist_push_head(&ioh->bounce_buffers, &bb->node);
+
+ /* once associated with an IO, the IO has ownership */
+ ResourceOwnerForgetAioBounceBuffer(bb->resowner, &bb->resowner_node);
+ bb->resowner = NULL;
+}
+
+uint32
+pgaio_bounce_buffer_id(PgAioBounceBuffer *bb)
+{
+ return bb - pgaio_ctl->bounce_buffers;
+}
+
+void
+pgaio_bounce_buffer_release(PgAioBounceBuffer *bb)
+{
+ if (pgaio_my_backend->handed_out_bb != bb)
+ elog(ERROR, "can only release handed out BB");
+
+ slist_push_head(&pgaio_my_backend->idle_bbs, &bb->node);
+ pgaio_my_backend->handed_out_bb = NULL;
+
+ ResourceOwnerForgetAioBounceBuffer(bb->resowner, &bb->resowner_node);
+ bb->resowner = NULL;
+}
+
+void
+pgaio_bounce_buffer_release_resowner(dlist_node *bb_node, bool on_error)
+{
+ PgAioBounceBuffer *bb = dlist_container(PgAioBounceBuffer, resowner_node, bb_node);
+
+ Assert(bb->resowner);
+
+ if (!on_error)
+ elog(WARNING, "leaked AIO bounce buffer");
+
+ pgaio_bounce_buffer_release(bb);
+}
+
+char *
+pgaio_bounce_buffer_buffer(PgAioBounceBuffer *bb)
+{
+ return bb->buffer;
+}
+
+static void
+pgaio_bounce_buffer_wait_for_free(void)
+{
+ static uint32 lastpos = 0;
+
+ if (pgaio_my_backend->num_staged_ios > 0)
+ {
+ pgaio_debug(DEBUG2, "submitting %d, while acquiring free bb",
+ pgaio_my_backend->num_staged_ios);
+ pgaio_submit_staged();
+ }
+
+ for (uint32 i = lastpos; i < lastpos + io_max_concurrency; i++)
+ {
+ uint32 thisoff = pgaio_my_backend->io_handle_off + (i % io_max_concurrency);
+ PgAioHandle *ioh = &pgaio_ctl->io_handles[thisoff];
+
+ switch (ioh->state)
+ {
+ case PGAIO_HS_IDLE:
+ case PGAIO_HS_HANDED_OUT:
+ continue;
+ case PGAIO_HS_DEFINED: /* should have been submitted above */
+ case PGAIO_HS_STAGED:
+ elog(ERROR, "shouldn't get here with io:%d in state %d",
+ pgaio_io_get_id(ioh), ioh->state);
+ break;
+ case PGAIO_HS_COMPLETED_IO:
+ case PGAIO_HS_SUBMITTED:
+ if (!slist_is_empty(&ioh->bounce_buffers))
+ {
+ pgaio_debug_io(DEBUG2, ioh,
+ "waiting for IO to reclaim BB with %d in flight",
+ dclist_count(&pgaio_my_backend->in_flight_ios));
+
+ /* see comment in pgaio_io_wait_for_free() about raciness */
+ pgaio_io_wait(ioh, ioh->generation);
+
+ if (slist_is_empty(&pgaio_my_backend->idle_bbs))
+ elog(WARNING, "empty after wait");
+
+ if (!slist_is_empty(&pgaio_my_backend->idle_bbs))
+ {
+ lastpos = i;
+ return;
+ }
+ }
+ break;
+ case PGAIO_HS_COMPLETED_SHARED:
+ case PGAIO_HS_COMPLETED_LOCAL:
+ /* reclaim */
+ pgaio_io_reclaim(ioh);
+
+ if (!slist_is_empty(&pgaio_my_backend->idle_bbs))
+ {
+ lastpos = i;
+ return;
+ }
+ break;
+ }
+ }
+
+ /*
+ * The submission above could have caused the IO to complete at any time.
+ */
+ if (slist_is_empty(&pgaio_my_backend->idle_bbs))
+ elog(PANIC, "no more bbs");
+}
+
+
+
/* --------------------------------------------------------------------------------
* Other
* --------------------------------------------------------------------------------
diff --git a/src/backend/storage/aio/aio_init.c b/src/backend/storage/aio/aio_init.c
index 885c3940c66..95b10933fed 100644
--- a/src/backend/storage/aio/aio_init.c
+++ b/src/backend/storage/aio/aio_init.c
@@ -88,6 +88,32 @@ AioHandleDataShmemSize(void)
io_max_concurrency));
}
+static Size
+AioBounceBufferDescShmemSize(void)
+{
+ Size sz;
+
+ /* PgAioBounceBuffer itself */
+ sz = mul_size(sizeof(PgAioBounceBuffer),
+ mul_size(AioProcs(), io_bounce_buffers));
+
+ return sz;
+}
+
+static Size
+AioBounceBufferDataShmemSize(void)
+{
+ Size sz;
+
+ /* and the associated buffer */
+ sz = mul_size(BLCKSZ,
+ mul_size(io_bounce_buffers, AioProcs()));
+ /* memory for alignment */
+ sz += BLCKSZ;
+
+ return sz;
+}
+
/*
* Choose a suitable value for io_max_concurrency.
*
@@ -113,6 +139,33 @@ AioChooseMaxConcurrency(void)
return Min(max_proportional_pins, 64);
}
+/*
+ * Choose a suitable value for io_bounce_buffers.
+ *
+ * It's very unlikely to be useful to allocate more bounce buffers for each
+ * backend than the backend is allowed to pin. Additionally, bounce buffers
+ * currently are used for writes, it seems very uncommon for more than 10% of
+ * shared_buffers to be written out concurrently.
+ *
+ * XXX: This quickly can take up significant amounts of memory, the logic
+ * should probably fine tuned.
+ */
+static int
+AioChooseBounceBuffers(void)
+{
+ uint32 max_backends;
+ int max_proportional_pins;
+
+ /* Similar logic to LimitAdditionalPins() */
+ max_backends = MaxBackends + NUM_AUXILIARY_PROCS;
+ max_proportional_pins = (NBuffers / 10) / max_backends;
+
+ max_proportional_pins = Max(max_proportional_pins, 1);
+
+ /* apply upper limit */
+ return Min(max_proportional_pins, 256);
+}
+
Size
AioShmemSize(void)
{
@@ -136,11 +189,31 @@ AioShmemSize(void)
PGC_S_OVERRIDE);
}
+
+ /*
+ * If io_bounce_buffers is -1, we automatically choose a suitable value.
+ *
+ * See also comment above.
+ */
+ if (io_bounce_buffers == -1)
+ {
+ char buf[32];
+
+ snprintf(buf, sizeof(buf), "%d", AioChooseBounceBuffers());
+ SetConfigOption("io_bounce_buffers", buf, PGC_POSTMASTER,
+ PGC_S_DYNAMIC_DEFAULT);
+ if (io_bounce_buffers == -1) /* failed to apply it? */
+ SetConfigOption("io_bounce_buffers", buf, PGC_POSTMASTER,
+ PGC_S_OVERRIDE);
+ }
+
sz = add_size(sz, AioCtlShmemSize());
sz = add_size(sz, AioBackendShmemSize());
sz = add_size(sz, AioHandleShmemSize());
sz = add_size(sz, AioHandleIOVShmemSize());
sz = add_size(sz, AioHandleDataShmemSize());
+ sz = add_size(sz, AioBounceBufferDescShmemSize());
+ sz = add_size(sz, AioBounceBufferDataShmemSize());
/* Reserve space for method specific resources. */
if (pgaio_method_ops->shmem_size)
@@ -156,6 +229,9 @@ AioShmemInit(void)
uint32 io_handle_off = 0;
uint32 iovec_off = 0;
uint32 per_backend_iovecs = io_max_concurrency * io_max_combine_limit;
+ uint32 bounce_buffers_off = 0;
+ uint32 per_backend_bb = io_bounce_buffers;
+ char *bounce_buffers_data;
pgaio_ctl = (PgAioCtl *)
ShmemInitStruct("AioCtl", AioCtlShmemSize(), &found);
@@ -167,6 +243,7 @@ AioShmemInit(void)
pgaio_ctl->io_handle_count = AioProcs() * io_max_concurrency;
pgaio_ctl->iovec_count = AioProcs() * per_backend_iovecs;
+ pgaio_ctl->bounce_buffers_count = AioProcs() * per_backend_bb;
pgaio_ctl->backend_state = (PgAioBackend *)
ShmemInitStruct("AioBackend", AioBackendShmemSize(), &found);
@@ -179,6 +256,40 @@ AioShmemInit(void)
pgaio_ctl->handle_data = (uint64 *)
ShmemInitStruct("AioHandleData", AioHandleDataShmemSize(), &found);
+ pgaio_ctl->bounce_buffers = (PgAioBounceBuffer *)
+ ShmemInitStruct("AioBounceBufferDesc", AioBounceBufferDescShmemSize(),
+ &found);
+
+ bounce_buffers_data =
+ ShmemInitStruct("AioBounceBufferData", AioBounceBufferDataShmemSize(),
+ &found);
+ bounce_buffers_data =
+ (char *) TYPEALIGN(BLCKSZ, (uintptr_t) bounce_buffers_data);
+ pgaio_ctl->bounce_buffers_data = bounce_buffers_data;
+
+
+ /* Initialize IO handles. */
+ for (uint64 i = 0; i < pgaio_ctl->io_handle_count; i++)
+ {
+ PgAioHandle *ioh = &pgaio_ctl->io_handles[i];
+
+ ioh->op = PGAIO_OP_INVALID;
+ ioh->target = PGAIO_TID_INVALID;
+ ioh->state = PGAIO_HS_IDLE;
+
+ slist_init(&ioh->bounce_buffers);
+ }
+
+ /* Initialize Bounce Buffers. */
+ for (uint64 i = 0; i < pgaio_ctl->bounce_buffers_count; i++)
+ {
+ PgAioBounceBuffer *bb = &pgaio_ctl->bounce_buffers[i];
+
+ bb->buffer = bounce_buffers_data;
+ bounce_buffers_data += BLCKSZ;
+ }
+
+
for (int procno = 0; procno < AioProcs(); procno++)
{
PgAioBackend *bs = &pgaio_ctl->backend_state[procno];
@@ -186,9 +297,13 @@ AioShmemInit(void)
bs->io_handle_off = io_handle_off;
io_handle_off += io_max_concurrency;
+ bs->bounce_buffers_off = bounce_buffers_off;
+ bounce_buffers_off += per_backend_bb;
+
dclist_init(&bs->idle_ios);
memset(bs->staged_ios, 0, sizeof(PgAioHandle *) * PGAIO_SUBMIT_BATCH_SIZE);
dclist_init(&bs->in_flight_ios);
+ slist_init(&bs->idle_bbs);
/* initialize per-backend IOs */
for (int i = 0; i < io_max_concurrency; i++)
@@ -210,6 +325,14 @@ AioShmemInit(void)
dclist_push_tail(&bs->idle_ios, &ioh->node);
iovec_off += io_max_combine_limit;
}
+
+ /* initialize per-backend bounce buffers */
+ for (int i = 0; i < per_backend_bb; i++)
+ {
+ PgAioBounceBuffer *bb = &pgaio_ctl->bounce_buffers[bs->bounce_buffers_off + i];
+
+ slist_push_head(&bs->idle_bbs, &bb->node);
+ }
}
out:
diff --git a/src/backend/utils/misc/guc_tables.c b/src/backend/utils/misc/guc_tables.c
index 5d729102f46..e885a682258 100644
--- a/src/backend/utils/misc/guc_tables.c
+++ b/src/backend/utils/misc/guc_tables.c
@@ -3303,6 +3303,19 @@ struct config_int ConfigureNamesInt[] =
check_io_max_concurrency, NULL, NULL
},
+ {
+ {"io_bounce_buffers",
+ PGC_POSTMASTER,
+ RESOURCES_IO,
+ gettext_noop("Number of IO Bounce Buffers reserved for each backend."),
+ NULL,
+ GUC_UNIT_BLOCKS
+ },
+ &io_bounce_buffers,
+ -1, -1, 4096,
+ NULL, NULL, NULL
+ },
+
{
{"io_workers",
PGC_SIGHUP,
diff --git a/src/backend/utils/misc/postgresql.conf.sample b/src/backend/utils/misc/postgresql.conf.sample
index ac6ae0a9489..4a5d04ef419 100644
--- a/src/backend/utils/misc/postgresql.conf.sample
+++ b/src/backend/utils/misc/postgresql.conf.sample
@@ -211,6 +211,8 @@
# -1 sets based on shared_buffers
# (change requires restart)
#io_workers = 3 # 1-32;
+#io_bounce_buffers = -1 # -1 sets based on shared_buffers
+ # (change requires restart)
# - Worker Processes -
diff --git a/src/backend/utils/resowner/resowner.c b/src/backend/utils/resowner/resowner.c
index d39f3e1b655..81e7e27965a 100644
--- a/src/backend/utils/resowner/resowner.c
+++ b/src/backend/utils/resowner/resowner.c
@@ -159,10 +159,11 @@ struct ResourceOwnerData
LOCALLOCK *locks[MAX_RESOWNER_LOCKS]; /* list of owned locks */
/*
- * AIO handles need be registered in critical sections and therefore
- * cannot use the normal ResourceElem mechanism.
+ * AIO handles & bounce buffers need be registered in critical sections
+ * and therefore cannot use the normal ResourceElem mechanism.
*/
dlist_head aio_handles;
+ dlist_head aio_bounce_buffers;
};
@@ -434,6 +435,7 @@ ResourceOwnerCreate(ResourceOwner parent, const char *name)
}
dlist_init(&owner->aio_handles);
+ dlist_init(&owner->aio_bounce_buffers);
return owner;
}
@@ -742,6 +744,13 @@ ResourceOwnerReleaseInternal(ResourceOwner owner,
pgaio_io_release_resowner(node, !isCommit);
}
+
+ while (!dlist_is_empty(&owner->aio_bounce_buffers))
+ {
+ dlist_node *node = dlist_head_node(&owner->aio_bounce_buffers);
+
+ pgaio_bounce_buffer_release_resowner(node, !isCommit);
+ }
}
else if (phase == RESOURCE_RELEASE_LOCKS)
{
@@ -1111,3 +1120,15 @@ ResourceOwnerForgetAioHandle(ResourceOwner owner, struct dlist_node *ioh_node)
{
dlist_delete_from(&owner->aio_handles, ioh_node);
}
+
+void
+ResourceOwnerRememberAioBounceBuffer(ResourceOwner owner, struct dlist_node *ioh_node)
+{
+ dlist_push_tail(&owner->aio_bounce_buffers, ioh_node);
+}
+
+void
+ResourceOwnerForgetAioBounceBuffer(ResourceOwner owner, struct dlist_node *ioh_node)
+{
+ dlist_delete_from(&owner->aio_bounce_buffers, ioh_node);
+}
diff --git a/src/test/modules/test_aio/test_aio--1.0.sql b/src/test/modules/test_aio/test_aio--1.0.sql
index c668d516f61..f93a698ae34 100644
--- a/src/test/modules/test_aio/test_aio--1.0.sql
+++ b/src/test/modules/test_aio/test_aio--1.0.sql
@@ -86,6 +86,27 @@ RETURNS pg_catalog.void STRICT
AS 'MODULE_PATHNAME' LANGUAGE C;
+CREATE FUNCTION bb_get_and_error()
+RETURNS pg_catalog.void STRICT
+AS 'MODULE_PATHNAME' LANGUAGE C;
+
+CREATE FUNCTION bb_get_twice()
+RETURNS pg_catalog.void STRICT
+AS 'MODULE_PATHNAME' LANGUAGE C;
+
+CREATE FUNCTION bb_get()
+RETURNS pg_catalog.void STRICT
+AS 'MODULE_PATHNAME' LANGUAGE C;
+
+CREATE FUNCTION bb_get_release()
+RETURNS pg_catalog.void STRICT
+AS 'MODULE_PATHNAME' LANGUAGE C;
+
+CREATE FUNCTION bb_release_last()
+RETURNS pg_catalog.void STRICT
+AS 'MODULE_PATHNAME' LANGUAGE C;
+
+
/*
* Injection point related functions
diff --git a/src/test/modules/test_aio/test_aio.c b/src/test/modules/test_aio/test_aio.c
index c7e857800dc..0cbb71fecfa 100644
--- a/src/test/modules/test_aio/test_aio.c
+++ b/src/test/modules/test_aio/test_aio.c
@@ -51,6 +51,7 @@ static shmem_startup_hook_type prev_shmem_startup_hook = NULL;
static PgAioHandle *last_handle;
+static PgAioBounceBuffer *last_bb;
@@ -617,6 +618,60 @@ batch_end(PG_FUNCTION_ARGS)
PG_RETURN_VOID();
}
+PG_FUNCTION_INFO_V1(bb_get);
+Datum
+bb_get(PG_FUNCTION_ARGS)
+{
+ last_bb = pgaio_bounce_buffer_get();
+
+ PG_RETURN_VOID();
+}
+
+PG_FUNCTION_INFO_V1(bb_release_last);
+Datum
+bb_release_last(PG_FUNCTION_ARGS)
+{
+ if (!last_bb)
+ elog(ERROR, "no bb");
+
+ pgaio_bounce_buffer_release(last_bb);
+
+ PG_RETURN_VOID();
+}
+
+PG_FUNCTION_INFO_V1(bb_get_and_error);
+Datum
+bb_get_and_error(PG_FUNCTION_ARGS)
+{
+ pgaio_bounce_buffer_get();
+
+ elog(ERROR, "as you command");
+ PG_RETURN_VOID();
+}
+
+PG_FUNCTION_INFO_V1(bb_get_twice);
+Datum
+bb_get_twice(PG_FUNCTION_ARGS)
+{
+ pgaio_bounce_buffer_get();
+ pgaio_bounce_buffer_get();
+
+ PG_RETURN_VOID();
+}
+
+
+PG_FUNCTION_INFO_V1(bb_get_release);
+Datum
+bb_get_release(PG_FUNCTION_ARGS)
+{
+ PgAioBounceBuffer *bb;
+
+ bb = pgaio_bounce_buffer_get();
+ pgaio_bounce_buffer_release(bb);
+
+ PG_RETURN_VOID();
+}
+
#ifdef USE_INJECTION_POINTS
extern PGDLLEXPORT void inj_io_short_read(const char *name, const void *private_data);
extern PGDLLEXPORT void inj_io_reopen(const char *name, const void *private_data);
diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list
index 9442a4841aa..49e1faecacf 100644
--- a/src/tools/pgindent/typedefs.list
+++ b/src/tools/pgindent/typedefs.list
@@ -2133,6 +2133,7 @@ PermutationStep
PermutationStepBlocker
PermutationStepBlockerType
PgAioBackend
+PgAioBounceBuffer
PgAioCtl
PgAioHandle
PgAioHandleCallbackID
--
2.48.1.76.g4e746b1a31.dirty
[text/x-diff] v2.13-0021-bufmgr-Implement-AIO-write-support.patch (11.6K, ../../o2dtdrlc2xqpjycsfuvv7u6zyrqj7jfxqu32et3lyelkwce2jy@3k3fwdiiu73v/22-v2.13-0021-bufmgr-Implement-AIO-write-support.patch)
download | inline diff:
From 65e53eb3977d86970250c364c3c41aac476a6ae2 Mon Sep 17 00:00:00 2001
From: Andres Freund <andres@anarazel.de>
Date: Tue, 18 Mar 2025 14:40:06 -0400
Subject: [PATCH v2.13 21/28] bufmgr: Implement AIO write support
As of this commit there are no users of these AIO facilities, that'll come in
later commits.
Problems with AIO writes:
- Write logic needs to be rebased on-top of the patch series to not hit bit
dirty buffers while IO is going on
The performance impact of doing the memory copies is rather substantial, as
on intel memory bandwidth is *the* IO bottleneck even just for the checksum
computation, without a copy. That makes the memory copy for something like
bounce buffers hurt really badly.
And the memory usage of bounce buffers is also really concerning.
And even without checksums, several filesystems *really* don't like buffers
getting modified during DIO writes. Which I think would mean we ought to use
bounce buffers for *all* writes, which would impose a *very* substantial
overhead (basically removing the benefit of DMA happening off-cpu).
- I think it requires new lwlock.c infrastructure (as v1 of aio had), to make
LockBuffer(BUFFER_LOCK_EXCLUSIVE) etc wait in a concurrency safe manner for
in-progress writes
I can think of ways to solve this purely in bufmgr.c, but only in ways that
would cause other problems (e.g. setting BM_IO_IN_PROGRESS before waiting
for an exclusive lock) and/or expensive.
---
src/include/storage/aio.h | 2 +
src/include/storage/bufmgr.h | 2 +
src/backend/storage/aio/aio_callback.c | 2 +
src/backend/storage/buffer/bufmgr.c | 188 ++++++++++++++++++++++++-
4 files changed, 189 insertions(+), 5 deletions(-)
diff --git a/src/include/storage/aio.h b/src/include/storage/aio.h
index f9a6ac8c625..1726dbdb9e6 100644
--- a/src/include/storage/aio.h
+++ b/src/include/storage/aio.h
@@ -197,8 +197,10 @@ typedef enum PgAioHandleCallbackID
PGAIO_HCB_MD_WRITEV,
PGAIO_HCB_SHARED_BUFFER_READV,
+ PGAIO_HCB_SHARED_BUFFER_WRITEV,
PGAIO_HCB_LOCAL_BUFFER_READV,
+ PGAIO_HCB_LOCAL_BUFFER_WRITEV,
} PgAioHandleCallbackID;
diff --git a/src/include/storage/bufmgr.h b/src/include/storage/bufmgr.h
index 784df8b00cb..ba9bf247ddb 100644
--- a/src/include/storage/bufmgr.h
+++ b/src/include/storage/bufmgr.h
@@ -172,7 +172,9 @@ extern PGDLLIMPORT int backend_flush_after;
extern PGDLLIMPORT int bgwriter_flush_after;
extern const PgAioHandleCallbacks aio_shared_buffer_readv_cb;
+extern const PgAioHandleCallbacks aio_shared_buffer_writev_cb;
extern const PgAioHandleCallbacks aio_local_buffer_readv_cb;
+extern const PgAioHandleCallbacks aio_local_buffer_writev_cb;
/* in buf_init.c */
extern PGDLLIMPORT char *BufferBlocks;
diff --git a/src/backend/storage/aio/aio_callback.c b/src/backend/storage/aio/aio_callback.c
index 67026f5f6b4..f9c4599ee21 100644
--- a/src/backend/storage/aio/aio_callback.c
+++ b/src/backend/storage/aio/aio_callback.c
@@ -44,8 +44,10 @@ static const PgAioHandleCallbacksEntry aio_handle_cbs[] = {
CALLBACK_ENTRY(PGAIO_HCB_MD_WRITEV, aio_md_writev_cb),
CALLBACK_ENTRY(PGAIO_HCB_SHARED_BUFFER_READV, aio_shared_buffer_readv_cb),
+ CALLBACK_ENTRY(PGAIO_HCB_SHARED_BUFFER_WRITEV, aio_shared_buffer_writev_cb),
CALLBACK_ENTRY(PGAIO_HCB_LOCAL_BUFFER_READV, aio_local_buffer_readv_cb),
+ CALLBACK_ENTRY(PGAIO_HCB_LOCAL_BUFFER_WRITEV, aio_local_buffer_writev_cb),
#undef CALLBACK_ENTRY
};
diff --git a/src/backend/storage/buffer/bufmgr.c b/src/backend/storage/buffer/bufmgr.c
index 8608df8f115..473904d7c30 100644
--- a/src/backend/storage/buffer/bufmgr.c
+++ b/src/backend/storage/buffer/bufmgr.c
@@ -5521,7 +5521,15 @@ LockBuffer(Buffer buffer, int mode)
else if (mode == BUFFER_LOCK_SHARE)
LWLockAcquire(BufferDescriptorGetContentLock(buf), LW_SHARED);
else if (mode == BUFFER_LOCK_EXCLUSIVE)
+ {
+ /*
+ * FIXME: Wait for AIO writes, otherwise there would be a risk of
+ * deadlock. This isn't entirely trivial to do in a race-free way, IO
+ * could be started between us checking whether there is IO and
+ * enqueueing ourselves for the lock.
+ */
LWLockAcquire(BufferDescriptorGetContentLock(buf), LW_EXCLUSIVE);
+ }
else
elog(ERROR, "unrecognized buffer lock mode: %d", mode);
}
@@ -5536,6 +5544,19 @@ ConditionalLockBuffer(Buffer buffer)
{
BufferDesc *buf;
+ /*
+ * FIXME: Wait for AIO writes. Some code does not deal well
+ * ConditionalLockBuffer() continuously failing, e.g.
+ * spginsert()->spgdoinsert() ends up busy-looping (easily reproducible by
+ * just making this function always fail and running the regression
+ * tests). While that code could be fixed, it'd be hard to find all
+ * problematic places.
+ *
+ * It would be OK to wait for the IO as waiting for IO completion does not
+ * need to wait for any locks that could lead to an undetected deadlock or
+ * such.
+ */
+
Assert(BufferIsPinned(buffer));
if (BufferIsLocal(buffer))
return true; /* act as though we got it */
@@ -5599,9 +5620,8 @@ LockBufferForCleanup(Buffer buffer)
CheckBufferIsPinnedOnce(buffer);
/*
- * We do not yet need to be worried about in-progress AIOs holding a pin,
- * as we only support doing reads via AIO and this function can only be
- * called once the buffer is valid (i.e. no read can be in flight).
+ * FIXME: See AIO related comments in LockBuffer() and
+ * ConditionalLockBuffer()
*/
/* Nobody else to wait for */
@@ -5614,6 +5634,11 @@ LockBufferForCleanup(Buffer buffer)
{
uint32 buf_state;
+ /*
+ * FIXME: LockBuffer()'s handling of in-progress writes (once
+ * implemented) should suffice to deal with deadlock risk.
+ */
+
/* Try to acquire lock */
LockBuffer(buffer, BUFFER_LOCK_EXCLUSIVE);
buf_state = LockBufHdr(bufHdr);
@@ -5761,7 +5786,13 @@ ConditionalLockBufferForCleanup(Buffer buffer)
Assert(BufferIsValid(buffer));
- /* see AIO related comment in LockBufferForCleanup() */
+ /*
+ * FIXME: Should wait for IO for the same reason as in
+ * ConditionalLockBuffer(). Needs to happen before the
+ * ConditionalLockBuffer() call below, as we'd never reach the
+ * ConditionalLockBuffer() call due the buffer pin held for the duration
+ * of the IO.
+ */
if (BufferIsLocal(buffer))
{
@@ -5818,7 +5849,10 @@ IsBufferCleanupOK(Buffer buffer)
Assert(BufferIsValid(buffer));
- /* see AIO related comment in LockBufferForCleanup() */
+ /*
+ * FIXME: See AIO related comments in LockBuffer() and
+ * ConditionalLockBuffer()
+ */
if (BufferIsLocal(buffer))
{
@@ -6955,12 +6989,129 @@ buffer_readv_report(PgAioResult result, const PgAioTargetData *target_data,
}
}
+/*
+ * Helper for AIO writev completion callbacks, supporting both shared and temp
+ * buffers. Gets called once for each buffer in a multi-page write.
+ */
+static pg_attribute_always_inline PgAioResult
+buffer_writev_complete_one(uint8 buf_off, Buffer buffer, uint8 flags,
+ bool failed, bool is_temp)
+{
+ BufferDesc *buf_hdr = is_temp ?
+ GetLocalBufferDescriptor(-buffer - 1)
+ : GetBufferDescriptor(buffer - 1);
+ PgAioResult result = {.status = PGAIO_RS_OK};
+ bool clear_dirty;
+ uint32 set_flag_bits;
+
+#ifdef USE_ASSERT_CHECKING
+ {
+ uint32 buf_state = pg_atomic_read_u32(&buf_hdr->state);
+
+ Assert(buf_state & BM_VALID);
+ Assert(buf_state & BM_TAG_VALID);
+ /* temp buffers don't use BM_IO_IN_PROGRESS */
+ if (!is_temp)
+ Assert(buf_state & BM_IO_IN_PROGRESS);
+ Assert(buf_state & BM_DIRTY);
+ }
+#endif
+
+ clear_dirty = failed ? false : true;
+ set_flag_bits = failed ? BM_IO_ERROR : 0;
+
+ if (is_temp)
+ TerminateLocalBufferIO(buf_hdr, clear_dirty, set_flag_bits, true);
+ else
+ TerminateBufferIO(buf_hdr, clear_dirty, set_flag_bits, false, true);
+
+ /*
+ * The initiator of IO is not managing the lock (i.e. we called
+ * LWLockDisown()), we are.
+ */
+ if (!is_temp)
+ LWLockReleaseDisowned(BufferDescriptorGetContentLock(buf_hdr),
+ LW_SHARED);
+
+ /* FIXME: tracepoint */
+
+ return result;
+}
+
+/*
+ * Perform completion handling of a single AIO write. This write may cover
+ * multiple blocks / buffers.
+ *
+ * Shared between shared and local buffers, to reduce code duplication.
+ */
+static pg_attribute_always_inline PgAioResult
+buffer_writev_complete(PgAioHandle *ioh, PgAioResult prior_result,
+ uint8 cb_data, bool is_temp)
+{
+ PgAioResult result = prior_result;
+ PgAioTargetData *td = pgaio_io_get_target_data(ioh);
+ uint64 *io_data;
+ uint8 handle_data_len;
+
+ if (is_temp)
+ {
+ Assert(td->smgr.is_temp);
+ Assert(pgaio_io_get_owner(ioh) == MyProcNumber);
+ }
+ else
+ Assert(!td->smgr.is_temp);
+
+ /*
+ * Iterate over all the buffers affected by this IO and call appropriate
+ * per-buffer completion function for each buffer.
+ */
+ io_data = pgaio_io_get_handle_data(ioh, &handle_data_len);
+ for (uint8 buf_off = 0; buf_off < handle_data_len; buf_off++)
+ {
+ Buffer buf = io_data[buf_off];
+ PgAioResult buf_result;
+ bool failed;
+
+ Assert(BufferIsValid(buf));
+
+ /*
+ * If the entire failed on a lower-level, each buffer needs to be
+ * marked as failed. In case of a partial read, some buffers may be
+ * ok.
+ */
+ failed =
+ prior_result.status == PGAIO_RS_ERROR
+ || prior_result.result <= buf_off;
+
+ buf_result = buffer_writev_complete_one(buf_off, buf, cb_data, failed,
+ is_temp);
+
+ /*
+ * If there wasn't any prior error and the IO for this page failed in
+ * some form, set the whole IO's to the page's result.
+ */
+ if (result.status != PGAIO_RS_ERROR && buf_result.status != PGAIO_RS_OK)
+ {
+ result = buf_result;
+ pgaio_result_report(result, td, LOG);
+ }
+ }
+
+ return result;
+}
+
static void
shared_buffer_readv_stage(PgAioHandle *ioh, uint8 cb_data)
{
buffer_stage_common(ioh, false, false);
}
+static void
+shared_buffer_writev_stage(PgAioHandle *ioh, uint8 cb_data)
+{
+ buffer_stage_common(ioh, true, false);
+}
+
static PgAioResult
shared_buffer_readv_complete(PgAioHandle *ioh, PgAioResult prior_result,
uint8 cb_data)
@@ -6968,6 +7119,13 @@ shared_buffer_readv_complete(PgAioHandle *ioh, PgAioResult prior_result,
return buffer_readv_complete(ioh, prior_result, cb_data, false);
}
+static PgAioResult
+shared_buffer_writev_complete(PgAioHandle *ioh, PgAioResult prior_result,
+ uint8 cb_data)
+{
+ return buffer_writev_complete(ioh, prior_result, cb_data, false);
+}
+
static void
local_buffer_readv_stage(PgAioHandle *ioh, uint8 cb_data)
{
@@ -6981,6 +7139,17 @@ local_buffer_readv_complete(PgAioHandle *ioh, PgAioResult prior_result,
return buffer_readv_complete(ioh, prior_result, cb_data, true);
}
+static void
+local_buffer_writev_stage(PgAioHandle *ioh, uint8 cb_data)
+{
+ /*
+ * Currently this is unreachable as the only write support is for
+ * checkpointer / bgwriter, which don't deal with local buffers.
+ */
+ elog(ERROR, "should be unreachable");
+}
+
+
/* readv callback is passed READ_BUFFERS_* flags as callback data */
const PgAioHandleCallbacks aio_shared_buffer_readv_cb = {
.stage = shared_buffer_readv_stage,
@@ -6988,6 +7157,11 @@ const PgAioHandleCallbacks aio_shared_buffer_readv_cb = {
.report = buffer_readv_report,
};
+const PgAioHandleCallbacks aio_shared_buffer_writev_cb = {
+ .stage = shared_buffer_writev_stage,
+ .complete_shared = shared_buffer_writev_complete,
+};
+
/* readv callback is passed READ_BUFFERS_* flags as callback data */
const PgAioHandleCallbacks aio_local_buffer_readv_cb = {
.stage = local_buffer_readv_stage,
@@ -7001,3 +7175,7 @@ const PgAioHandleCallbacks aio_local_buffer_readv_cb = {
.complete_local = local_buffer_readv_complete,
.report = buffer_readv_report,
};
+
+const PgAioHandleCallbacks aio_local_buffer_writev_cb = {
+ .stage = local_buffer_writev_stage,
+};
--
2.48.1.76.g4e746b1a31.dirty
[text/x-diff] v2.13-0022-aio-Add-IO-queue-helper.patch (7.4K, ../../o2dtdrlc2xqpjycsfuvv7u6zyrqj7jfxqu32et3lyelkwce2jy@3k3fwdiiu73v/23-v2.13-0022-aio-Add-IO-queue-helper.patch)
download | inline diff:
From 76bbfeb8bd930c60a968672cff2d18521349d0fd Mon Sep 17 00:00:00 2001
From: Andres Freund <andres@anarazel.de>
Date: Tue, 18 Mar 2025 14:40:06 -0400
Subject: [PATCH v2.13 22/28] aio: Add IO queue helper
This is likely never going to anywhere - Thomas Munro is working on something
more complete. But I needed a way to exercise aio for checkpointer / bgwriter.
---
src/include/storage/io_queue.h | 33 +++++
src/backend/storage/aio/Makefile | 1 +
src/backend/storage/aio/io_queue.c | 204 ++++++++++++++++++++++++++++
src/backend/storage/aio/meson.build | 1 +
src/tools/pgindent/typedefs.list | 2 +
5 files changed, 241 insertions(+)
create mode 100644 src/include/storage/io_queue.h
create mode 100644 src/backend/storage/aio/io_queue.c
diff --git a/src/include/storage/io_queue.h b/src/include/storage/io_queue.h
new file mode 100644
index 00000000000..92b1e9afe6f
--- /dev/null
+++ b/src/include/storage/io_queue.h
@@ -0,0 +1,33 @@
+/*-------------------------------------------------------------------------
+ *
+ * io_queue.h
+ * Mechanism for tracking many IOs
+ *
+ *
+ * Portions Copyright (c) 1996-2025, PostgreSQL Global Development Group
+ * Portions Copyright (c) 1994, Regents of the University of California
+ *
+ * src/include/storage/io_queue.h
+ *
+ *-------------------------------------------------------------------------
+ */
+#ifndef IO_QUEUE_H
+#define IO_QUEUE_H
+
+#include "storage/aio_types.h"
+
+struct IOQueue;
+typedef struct IOQueue IOQueue;
+
+struct PgAioWaitRef;
+
+extern IOQueue *io_queue_create(int depth, int flags);
+extern void io_queue_track(IOQueue *ioq, const PgAioWaitRef *iow);
+extern void io_queue_wait_one(IOQueue *ioq);
+extern void io_queue_wait_all(IOQueue *ioq);
+extern bool io_queue_is_empty(IOQueue *ioq);
+extern void io_queue_reserve(IOQueue *ioq);
+extern PgAioHandle *io_queue_acquire_io(IOQueue *ioq);
+extern void io_queue_free(IOQueue *ioq);
+
+#endif /* IO_QUEUE_H */
diff --git a/src/backend/storage/aio/Makefile b/src/backend/storage/aio/Makefile
index 3f2469cc399..86fa4276fda 100644
--- a/src/backend/storage/aio/Makefile
+++ b/src/backend/storage/aio/Makefile
@@ -15,6 +15,7 @@ OBJS = \
aio_init.o \
aio_io.o \
aio_target.o \
+ io_queue.o \
method_io_uring.o \
method_sync.o \
method_worker.o \
diff --git a/src/backend/storage/aio/io_queue.c b/src/backend/storage/aio/io_queue.c
new file mode 100644
index 00000000000..526aa1d5e06
--- /dev/null
+++ b/src/backend/storage/aio/io_queue.c
@@ -0,0 +1,204 @@
+/*-------------------------------------------------------------------------
+ *
+ * io_queue.c
+ * AIO - Mechanism for tracking many IOs
+ *
+ * Portions Copyright (c) 1996-2025, PostgreSQL Global Development Group
+ * Portions Copyright (c) 1994, Regents of the University of California
+ *
+ * IDENTIFICATION
+ * src/backend/storage/aio/io_queue.c
+ *
+ *-------------------------------------------------------------------------
+ */
+
+#include "postgres.h"
+
+#include "lib/ilist.h"
+#include "storage/aio.h"
+#include "storage/io_queue.h"
+#include "utils/resowner.h"
+
+
+
+typedef struct TrackedIO
+{
+ PgAioWaitRef iow;
+ dlist_node node;
+} TrackedIO;
+
+struct IOQueue
+{
+ int depth;
+ int unsubmitted;
+
+ bool has_reserved;
+
+ dclist_head idle;
+ dclist_head in_progress;
+
+ TrackedIO tracked_ios[FLEXIBLE_ARRAY_MEMBER];
+};
+
+
+IOQueue *
+io_queue_create(int depth, int flags)
+{
+ size_t sz;
+ IOQueue *ioq;
+
+ sz = offsetof(IOQueue, tracked_ios)
+ + sizeof(TrackedIO) * depth;
+
+ ioq = palloc0(sz);
+
+ ioq->depth = 0;
+
+ for (int i = 0; i < depth; i++)
+ {
+ TrackedIO *tio = &ioq->tracked_ios[i];
+
+ pgaio_wref_clear(&tio->iow);
+ dclist_push_tail(&ioq->idle, &tio->node);
+ }
+
+ return ioq;
+}
+
+void
+io_queue_wait_one(IOQueue *ioq)
+{
+ /* submit all pending IO before waiting */
+ pgaio_submit_staged();
+
+ while (!dclist_is_empty(&ioq->in_progress))
+ {
+ /* FIXME: Should we really pop here already? */
+ dlist_node *node = dclist_pop_head_node(&ioq->in_progress);
+ TrackedIO *tio = dclist_container(TrackedIO, node, node);
+
+ pgaio_wref_wait(&tio->iow);
+ dclist_push_head(&ioq->idle, &tio->node);
+ }
+}
+
+void
+io_queue_reserve(IOQueue *ioq)
+{
+ if (ioq->has_reserved)
+ return;
+
+ if (dclist_is_empty(&ioq->idle))
+ io_queue_wait_one(ioq);
+
+ Assert(!dclist_is_empty(&ioq->idle));
+
+ ioq->has_reserved = true;
+}
+
+PgAioHandle *
+io_queue_acquire_io(IOQueue *ioq)
+{
+ PgAioHandle *ioh;
+
+ io_queue_reserve(ioq);
+
+ Assert(!dclist_is_empty(&ioq->idle));
+
+ if (!io_queue_is_empty(ioq))
+ {
+ ioh = pgaio_io_acquire_nb(CurrentResourceOwner, NULL);
+ if (ioh == NULL)
+ {
+ /*
+ * Need to wait for all IOs, blocking might not be legal in the
+ * context.
+ *
+ * XXX: This doesn't make a whole lot of sense, we're also
+ * blocking here. What was I smoking when I wrote the above?
+ */
+ io_queue_wait_all(ioq);
+ ioh = pgaio_io_acquire(CurrentResourceOwner, NULL);
+ }
+ }
+ else
+ {
+ ioh = pgaio_io_acquire(CurrentResourceOwner, NULL);
+ }
+
+ return ioh;
+}
+
+void
+io_queue_track(IOQueue *ioq, const struct PgAioWaitRef *iow)
+{
+ dlist_node *node;
+ TrackedIO *tio;
+
+ Assert(ioq->has_reserved);
+ ioq->has_reserved = false;
+
+ Assert(!dclist_is_empty(&ioq->idle));
+
+ node = dclist_pop_head_node(&ioq->idle);
+ tio = dclist_container(TrackedIO, node, node);
+
+ tio->iow = *iow;
+
+ dclist_push_tail(&ioq->in_progress, &tio->node);
+
+ ioq->unsubmitted++;
+
+ /*
+ * XXX: Should have some smarter logic here. We don't want to wait too
+ * long to submit, that'll mean we're more likely to block. But we also
+ * don't want to have the overhead of submitting every IO individually.
+ */
+ if (ioq->unsubmitted >= 4)
+ {
+ pgaio_submit_staged();
+ ioq->unsubmitted = 0;
+ }
+}
+
+void
+io_queue_wait_all(IOQueue *ioq)
+{
+ /* submit all pending IO before waiting */
+ pgaio_submit_staged();
+
+ while (!dclist_is_empty(&ioq->in_progress))
+ {
+ /* wait for the last IO to minimize unnecessary wakeups */
+ dlist_node *node = dclist_tail_node(&ioq->in_progress);
+ TrackedIO *tio = dclist_container(TrackedIO, node, node);
+
+ if (!pgaio_wref_check_done(&tio->iow))
+ {
+ ereport(DEBUG3,
+ errmsg("io_queue_wait_all for io:%d",
+ pgaio_wref_get_id(&tio->iow)),
+ errhidestmt(true),
+ errhidecontext(true));
+
+ pgaio_wref_wait(&tio->iow);
+ }
+
+ dclist_delete_from(&ioq->in_progress, &tio->node);
+ dclist_push_head(&ioq->idle, &tio->node);
+ }
+}
+
+bool
+io_queue_is_empty(IOQueue *ioq)
+{
+ return dclist_is_empty(&ioq->in_progress);
+}
+
+void
+io_queue_free(IOQueue *ioq)
+{
+ io_queue_wait_all(ioq);
+
+ pfree(ioq);
+}
diff --git a/src/backend/storage/aio/meson.build b/src/backend/storage/aio/meson.build
index da6df2d3654..270c4a64428 100644
--- a/src/backend/storage/aio/meson.build
+++ b/src/backend/storage/aio/meson.build
@@ -7,6 +7,7 @@ backend_sources += files(
'aio_init.c',
'aio_io.c',
'aio_target.c',
+ 'io_queue.c',
'method_io_uring.c',
'method_sync.c',
'method_worker.c',
diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list
index 49e1faecacf..e247f6ac7d0 100644
--- a/src/tools/pgindent/typedefs.list
+++ b/src/tools/pgindent/typedefs.list
@@ -1192,6 +1192,7 @@ IOContext
IOFuncSelector
IOObject
IOOp
+IOQueue
IO_STATUS_BLOCK
IPCompareMethod
ITEM
@@ -3018,6 +3019,7 @@ TocEntry
TokenAuxData
TokenizedAuthLine
TrackItem
+TrackedIO
TransApplyAction
TransInvalidationInfo
TransState
--
2.48.1.76.g4e746b1a31.dirty
[text/x-diff] v2.13-0023-bufmgr-use-AIO-in-checkpointer-bgwriter.patch (31.6K, ../../o2dtdrlc2xqpjycsfuvv7u6zyrqj7jfxqu32et3lyelkwce2jy@3k3fwdiiu73v/24-v2.13-0023-bufmgr-use-AIO-in-checkpointer-bgwriter.patch)
download | inline diff:
From de3d8c602b6e068216b49ab6ce0054ee4d346491 Mon Sep 17 00:00:00 2001
From: Andres Freund <andres@anarazel.de>
Date: Tue, 18 Mar 2025 14:40:06 -0400
Subject: [PATCH v2.13 23/28] bufmgr: use AIO in checkpointer, bgwriter
This is far from ready - just included to be able to exercise AIO writes and
get some preliminary numbers. In all likelihood this will instead be based
on-top of work by Thomas Munro instead of the preceding commit.
TODO;
- This doesn't implement bgwriter_flush_after, checkpointer_flush_after
I think that's not too hard to do, it's mainly round tuits.
- The queuing logic doesn't carefully respect pin limits
That might be ok for checkpointer and bgwriter, but the infrastructure
should be usable outside of this as well.
---
src/include/postmaster/bgwriter.h | 3 +-
src/include/storage/buf_internals.h | 2 +
src/include/storage/bufmgr.h | 3 +-
src/include/storage/bufpage.h | 1 +
src/backend/postmaster/bgwriter.c | 19 +-
src/backend/postmaster/checkpointer.c | 11 +-
src/backend/storage/buffer/bufmgr.c | 594 +++++++++++++++++++++++---
src/backend/storage/page/bufpage.c | 10 +
src/tools/pgindent/typedefs.list | 1 +
9 files changed, 586 insertions(+), 58 deletions(-)
diff --git a/src/include/postmaster/bgwriter.h b/src/include/postmaster/bgwriter.h
index 800ecbfd13b..a8081d411b6 100644
--- a/src/include/postmaster/bgwriter.h
+++ b/src/include/postmaster/bgwriter.h
@@ -31,7 +31,8 @@ pg_noreturn extern void BackgroundWriterMain(const void *startup_data, size_t st
pg_noreturn extern void CheckpointerMain(const void *startup_data, size_t startup_data_len);
extern void RequestCheckpoint(int flags);
-extern void CheckpointWriteDelay(int flags, double progress);
+struct IOQueue;
+extern void CheckpointWriteDelay(struct IOQueue *ioq, int flags, double progress);
extern bool ForwardSyncRequest(const FileTag *ftag, SyncRequestType type);
diff --git a/src/include/storage/buf_internals.h b/src/include/storage/buf_internals.h
index 0dec7d93b3b..45c2b70b736 100644
--- a/src/include/storage/buf_internals.h
+++ b/src/include/storage/buf_internals.h
@@ -21,6 +21,8 @@
#include "storage/buf.h"
#include "storage/bufmgr.h"
#include "storage/condition_variable.h"
+#include "storage/io_queue.h"
+#include "storage/latch.h"
#include "storage/lwlock.h"
#include "storage/procnumber.h"
#include "storage/shmem.h"
diff --git a/src/include/storage/bufmgr.h b/src/include/storage/bufmgr.h
index ba9bf247ddb..af5035317b7 100644
--- a/src/include/storage/bufmgr.h
+++ b/src/include/storage/bufmgr.h
@@ -295,7 +295,8 @@ extern bool ConditionalLockBufferForCleanup(Buffer buffer);
extern bool IsBufferCleanupOK(Buffer buffer);
extern bool HoldingBufferPinThatDelaysRecovery(void);
-extern bool BgBufferSync(struct WritebackContext *wb_context);
+struct IOQueue;
+extern bool BgBufferSync(struct IOQueue *ioq, struct WritebackContext *wb_context);
extern uint32 GetPinLimit(void);
extern uint32 GetLocalPinLimit(void);
diff --git a/src/include/storage/bufpage.h b/src/include/storage/bufpage.h
index 6646b6f6371..9c045e81857 100644
--- a/src/include/storage/bufpage.h
+++ b/src/include/storage/bufpage.h
@@ -509,5 +509,6 @@ extern bool PageIndexTupleOverwrite(Page page, OffsetNumber offnum,
Item newtup, Size newsize);
extern char *PageSetChecksumCopy(Page page, BlockNumber blkno);
extern void PageSetChecksumInplace(Page page, BlockNumber blkno);
+extern bool PageNeedsChecksumCopy(Page page);
#endif /* BUFPAGE_H */
diff --git a/src/backend/postmaster/bgwriter.c b/src/backend/postmaster/bgwriter.c
index 72f5acceec7..6e8801a39e3 100644
--- a/src/backend/postmaster/bgwriter.c
+++ b/src/backend/postmaster/bgwriter.c
@@ -38,11 +38,13 @@
#include "postmaster/auxprocess.h"
#include "postmaster/bgwriter.h"
#include "postmaster/interrupt.h"
+#include "storage/aio.h"
#include "storage/aio_subsys.h"
#include "storage/buf_internals.h"
#include "storage/bufmgr.h"
#include "storage/condition_variable.h"
#include "storage/fd.h"
+#include "storage/io_queue.h"
#include "storage/lwlock.h"
#include "storage/proc.h"
#include "storage/procsignal.h"
@@ -90,6 +92,7 @@ BackgroundWriterMain(const void *startup_data, size_t startup_data_len)
sigjmp_buf local_sigjmp_buf;
MemoryContext bgwriter_context;
bool prev_hibernate;
+ IOQueue *ioq;
WritebackContext wb_context;
Assert(startup_data_len == 0);
@@ -131,6 +134,7 @@ BackgroundWriterMain(const void *startup_data, size_t startup_data_len)
ALLOCSET_DEFAULT_SIZES);
MemoryContextSwitchTo(bgwriter_context);
+ ioq = io_queue_create(128, 0);
WritebackContextInit(&wb_context, &bgwriter_flush_after);
/*
@@ -228,12 +232,22 @@ BackgroundWriterMain(const void *startup_data, size_t startup_data_len)
/* Clear any already-pending wakeups */
ResetLatch(MyLatch);
+ /*
+ * FIXME: this is theoretically racy, but I didn't want to copy
+ * ProcessMainLoopInterrupts() remaining body here.
+ */
+ if (ShutdownRequestPending)
+ {
+ io_queue_wait_all(ioq);
+ io_queue_free(ioq);
+ }
+
ProcessMainLoopInterrupts();
/*
* Do one cycle of dirty-buffer writing.
*/
- can_hibernate = BgBufferSync(&wb_context);
+ can_hibernate = BgBufferSync(ioq, &wb_context);
/* Report pending statistics to the cumulative stats system */
pgstat_report_bgwriter();
@@ -250,6 +264,9 @@ BackgroundWriterMain(const void *startup_data, size_t startup_data_len)
smgrdestroyall();
}
+ /* finish IO before sleeping, to avoid blocking other backends */
+ io_queue_wait_all(ioq);
+
/*
* Log a new xl_running_xacts every now and then so replication can
* get into a consistent state faster (think of suboverflowed
diff --git a/src/backend/postmaster/checkpointer.c b/src/backend/postmaster/checkpointer.c
index fda91ffd1ce..904fe167eb4 100644
--- a/src/backend/postmaster/checkpointer.c
+++ b/src/backend/postmaster/checkpointer.c
@@ -49,10 +49,12 @@
#include "postmaster/bgwriter.h"
#include "postmaster/interrupt.h"
#include "replication/syncrep.h"
+#include "storage/aio.h"
#include "storage/aio_subsys.h"
#include "storage/bufmgr.h"
#include "storage/condition_variable.h"
#include "storage/fd.h"
+#include "storage/io_queue.h"
#include "storage/ipc.h"
#include "storage/lwlock.h"
#include "storage/pmsignal.h"
@@ -766,7 +768,7 @@ ImmediateCheckpointRequested(void)
* fraction between 0.0 meaning none, and 1.0 meaning all done.
*/
void
-CheckpointWriteDelay(int flags, double progress)
+CheckpointWriteDelay(IOQueue *ioq, int flags, double progress)
{
static int absorb_counter = WRITES_PER_ABSORB;
@@ -800,6 +802,13 @@ CheckpointWriteDelay(int flags, double progress)
/* Report interim statistics to the cumulative stats system */
pgstat_report_checkpointer();
+ /*
+ * Ensure all pending IO is submitted to avoid unnecessary delays for
+ * other processes.
+ */
+ io_queue_wait_all(ioq);
+
+
/*
* This sleep used to be connected to bgwriter_delay, typically 200ms.
* That resulted in more frequent wakeups if not much work to do.
diff --git a/src/backend/storage/buffer/bufmgr.c b/src/backend/storage/buffer/bufmgr.c
index 473904d7c30..b374f53e9f3 100644
--- a/src/backend/storage/buffer/bufmgr.c
+++ b/src/backend/storage/buffer/bufmgr.c
@@ -52,6 +52,7 @@
#include "storage/buf_internals.h"
#include "storage/bufmgr.h"
#include "storage/fd.h"
+#include "storage/io_queue.h"
#include "storage/ipc.h"
#include "storage/lmgr.h"
#include "storage/proc.h"
@@ -76,6 +77,7 @@
/* Bits in SyncOneBuffer's return value */
#define BUF_WRITTEN 0x01
#define BUF_REUSABLE 0x02
+#define BUF_CANT_MERGE 0x04
#define RELS_BSEARCH_THRESHOLD 20
@@ -515,8 +517,6 @@ static void UnpinBuffer(BufferDesc *buf);
static void UnpinBufferNoOwner(BufferDesc *buf);
static void BufferSync(int flags);
static uint32 WaitBufHdrUnlocked(BufferDesc *buf);
-static int SyncOneBuffer(int buf_id, bool skip_recently_used,
- WritebackContext *wb_context);
static void WaitIO(BufferDesc *buf);
static void AbortBufferIO(Buffer buffer);
static void shared_buffer_write_error_callback(void *arg);
@@ -532,6 +532,7 @@ static void CheckReadBuffersOperation(ReadBuffersOperation *operation, bool is_c
static Buffer GetVictimBuffer(BufferAccessStrategy strategy, IOContext io_context);
static void FlushBuffer(BufferDesc *buf, SMgrRelation reln,
IOObject io_object, IOContext io_context);
+
static void FindAndDropRelationBuffers(RelFileLocator rlocator,
ForkNumber forkNum,
BlockNumber nForkBlock,
@@ -3306,6 +3307,57 @@ UnpinBufferNoOwner(BufferDesc *buf)
}
}
+typedef struct BuffersToWrite
+{
+ int nbuffers;
+ BufferTag start_at_tag;
+ uint32 max_combine;
+
+ XLogRecPtr max_lsn;
+
+ PgAioHandle *ioh;
+ PgAioWaitRef iow;
+
+ uint64 total_writes;
+
+ Buffer buffers[IOV_MAX];
+ PgAioBounceBuffer *bounce_buffers[IOV_MAX];
+ const void *data_ptrs[IOV_MAX];
+} BuffersToWrite;
+
+static int PrepareToWriteBuffer(BuffersToWrite *to_write, Buffer buf,
+ bool skip_recently_used,
+ IOQueue *ioq, WritebackContext *wb_context);
+
+static void WriteBuffers(BuffersToWrite *to_write,
+ IOQueue *ioq, WritebackContext *wb_context);
+
+static void
+BuffersToWriteInit(BuffersToWrite *to_write,
+ IOQueue *ioq, WritebackContext *wb_context)
+{
+ to_write->total_writes = 0;
+ to_write->nbuffers = 0;
+ to_write->ioh = NULL;
+ pgaio_wref_clear(&to_write->iow);
+ to_write->max_lsn = InvalidXLogRecPtr;
+
+ pgaio_enter_batchmode();
+}
+
+static void
+BuffersToWriteEnd(BuffersToWrite *to_write)
+{
+ if (to_write->ioh != NULL)
+ {
+ pgaio_io_release(to_write->ioh);
+ to_write->ioh = NULL;
+ }
+
+ pgaio_exit_batchmode();
+}
+
+
#define ST_SORT sort_checkpoint_bufferids
#define ST_ELEMENT_TYPE CkptSortItem
#define ST_COMPARE(a, b) ckpt_buforder_comparator(a, b)
@@ -3337,7 +3389,10 @@ BufferSync(int flags)
binaryheap *ts_heap;
int i;
int mask = BM_DIRTY;
+ IOQueue *ioq;
WritebackContext wb_context;
+ BuffersToWrite to_write;
+ int max_combine;
/*
* Unless this is a shutdown checkpoint or we have been explicitly told,
@@ -3399,7 +3454,9 @@ BufferSync(int flags)
if (num_to_scan == 0)
return; /* nothing to do */
+ ioq = io_queue_create(512, 0);
WritebackContextInit(&wb_context, &checkpoint_flush_after);
+ max_combine = Min(io_bounce_buffers, io_combine_limit);
TRACE_POSTGRESQL_BUFFER_SYNC_START(NBuffers, num_to_scan);
@@ -3507,48 +3564,91 @@ BufferSync(int flags)
*/
num_processed = 0;
num_written = 0;
+
+ BuffersToWriteInit(&to_write, ioq, &wb_context);
+
while (!binaryheap_empty(ts_heap))
{
BufferDesc *bufHdr = NULL;
CkptTsStatus *ts_stat = (CkptTsStatus *)
DatumGetPointer(binaryheap_first(ts_heap));
+ bool batch_continue = true;
- buf_id = CkptBufferIds[ts_stat->index].buf_id;
- Assert(buf_id != -1);
-
- bufHdr = GetBufferDescriptor(buf_id);
-
- num_processed++;
+ Assert(ts_stat->num_scanned <= ts_stat->num_to_scan);
/*
- * We don't need to acquire the lock here, because we're only looking
- * at a single bit. It's possible that someone else writes the buffer
- * and clears the flag right after we check, but that doesn't matter
- * since SyncOneBuffer will then do nothing. However, there is a
- * further race condition: it's conceivable that between the time we
- * examine the bit here and the time SyncOneBuffer acquires the lock,
- * someone else not only wrote the buffer but replaced it with another
- * page and dirtied it. In that improbable case, SyncOneBuffer will
- * write the buffer though we didn't need to. It doesn't seem worth
- * guarding against this, though.
+ * Collect a batch of buffers to write out from the current
+ * tablespace. That causes some imbalance between the tablespaces, but
+ * that's more than outweighed by the efficiency gain due to batching.
*/
- if (pg_atomic_read_u32(&bufHdr->state) & BM_CHECKPOINT_NEEDED)
+ while (batch_continue &&
+ to_write.nbuffers < max_combine &&
+ ts_stat->num_scanned < ts_stat->num_to_scan)
{
- if (SyncOneBuffer(buf_id, false, &wb_context) & BUF_WRITTEN)
+ buf_id = CkptBufferIds[ts_stat->index].buf_id;
+ Assert(buf_id != -1);
+
+ bufHdr = GetBufferDescriptor(buf_id);
+
+ num_processed++;
+
+ /*
+ * We don't need to acquire the lock here, because we're only
+ * looking at a single bit. It's possible that someone else writes
+ * the buffer and clears the flag right after we check, but that
+ * doesn't matter since SyncOneBuffer will then do nothing.
+ * However, there is a further race condition: it's conceivable
+ * that between the time we examine the bit here and the time
+ * SyncOneBuffer acquires the lock, someone else not only wrote
+ * the buffer but replaced it with another page and dirtied it. In
+ * that improbable case, SyncOneBuffer will write the buffer
+ * though we didn't need to. It doesn't seem worth guarding
+ * against this, though.
+ */
+ if (pg_atomic_read_u32(&bufHdr->state) & BM_CHECKPOINT_NEEDED)
{
- TRACE_POSTGRESQL_BUFFER_SYNC_WRITTEN(buf_id);
- PendingCheckpointerStats.buffers_written++;
- num_written++;
+ int result = PrepareToWriteBuffer(&to_write, buf_id + 1, false,
+ ioq, &wb_context);
+
+ if (result & BUF_CANT_MERGE)
+ {
+ Assert(to_write.nbuffers > 0);
+ WriteBuffers(&to_write, ioq, &wb_context);
+
+ result = PrepareToWriteBuffer(&to_write, buf_id + 1, false,
+ ioq, &wb_context);
+ Assert(result != BUF_CANT_MERGE);
+ }
+
+ if (result & BUF_WRITTEN)
+ {
+ TRACE_POSTGRESQL_BUFFER_SYNC_WRITTEN(buf_id);
+ PendingCheckpointerStats.buffers_written++;
+ num_written++;
+ }
+ else
+ {
+ batch_continue = false;
+ }
}
+ else
+ {
+ if (to_write.nbuffers > 0)
+ WriteBuffers(&to_write, ioq, &wb_context);
+ }
+
+ /*
+ * Measure progress independent of actually having to flush the
+ * buffer - otherwise writing become unbalanced.
+ */
+ ts_stat->progress += ts_stat->progress_slice;
+ ts_stat->num_scanned++;
+ ts_stat->index++;
}
- /*
- * Measure progress independent of actually having to flush the buffer
- * - otherwise writing become unbalanced.
- */
- ts_stat->progress += ts_stat->progress_slice;
- ts_stat->num_scanned++;
- ts_stat->index++;
+ if (to_write.nbuffers > 0)
+ WriteBuffers(&to_write, ioq, &wb_context);
+
/* Have all the buffers from the tablespace been processed? */
if (ts_stat->num_scanned == ts_stat->num_to_scan)
@@ -3566,15 +3666,23 @@ BufferSync(int flags)
*
* (This will check for barrier events even if it doesn't sleep.)
*/
- CheckpointWriteDelay(flags, (double) num_processed / num_to_scan);
+ CheckpointWriteDelay(ioq, flags, (double) num_processed / num_to_scan);
}
+ Assert(to_write.nbuffers == 0);
+ io_queue_wait_all(ioq);
+
/*
* Issue all pending flushes. Only checkpointer calls BufferSync(), so
* IOContext will always be IOCONTEXT_NORMAL.
*/
IssuePendingWritebacks(&wb_context, IOCONTEXT_NORMAL);
+ io_queue_wait_all(ioq); /* IssuePendingWritebacks might have added
+ * more */
+ io_queue_free(ioq);
+ BuffersToWriteEnd(&to_write);
+
pfree(per_ts_stat);
per_ts_stat = NULL;
binaryheap_free(ts_heap);
@@ -3600,7 +3708,7 @@ BufferSync(int flags)
* bgwriter_lru_maxpages to 0.)
*/
bool
-BgBufferSync(WritebackContext *wb_context)
+BgBufferSync(IOQueue *ioq, WritebackContext *wb_context)
{
/* info obtained from freelist.c */
int strategy_buf_id;
@@ -3643,6 +3751,9 @@ BgBufferSync(WritebackContext *wb_context)
long new_strategy_delta;
uint32 new_recent_alloc;
+ BuffersToWrite to_write;
+ int max_combine;
+
/*
* Find out where the freelist clock sweep currently is, and how many
* buffer allocations have happened since our last call.
@@ -3663,6 +3774,8 @@ BgBufferSync(WritebackContext *wb_context)
return true;
}
+ max_combine = Min(io_bounce_buffers, io_combine_limit);
+
/*
* Compute strategy_delta = how many buffers have been scanned by the
* clock sweep since last time. If first time through, assume none. Then
@@ -3819,11 +3932,25 @@ BgBufferSync(WritebackContext *wb_context)
num_written = 0;
reusable_buffers = reusable_buffers_est;
+ BuffersToWriteInit(&to_write, ioq, wb_context);
+
/* Execute the LRU scan */
while (num_to_scan > 0 && reusable_buffers < upcoming_alloc_est)
{
- int sync_state = SyncOneBuffer(next_to_clean, true,
- wb_context);
+ int sync_state;
+
+ sync_state = PrepareToWriteBuffer(&to_write, next_to_clean + 1,
+ true, ioq, wb_context);
+ if (sync_state & BUF_CANT_MERGE)
+ {
+ Assert(to_write.nbuffers > 0);
+
+ WriteBuffers(&to_write, ioq, wb_context);
+
+ sync_state = PrepareToWriteBuffer(&to_write, next_to_clean + 1,
+ true, ioq, wb_context);
+ Assert(sync_state != BUF_CANT_MERGE);
+ }
if (++next_to_clean >= NBuffers)
{
@@ -3834,6 +3961,13 @@ BgBufferSync(WritebackContext *wb_context)
if (sync_state & BUF_WRITTEN)
{
+ Assert(sync_state & BUF_REUSABLE);
+
+ if (to_write.nbuffers == max_combine)
+ {
+ WriteBuffers(&to_write, ioq, wb_context);
+ }
+
reusable_buffers++;
if (++num_written >= bgwriter_lru_maxpages)
{
@@ -3845,6 +3979,11 @@ BgBufferSync(WritebackContext *wb_context)
reusable_buffers++;
}
+ if (to_write.nbuffers > 0)
+ WriteBuffers(&to_write, ioq, wb_context);
+
+ BuffersToWriteEnd(&to_write);
+
PendingBgWriterStats.buf_written_clean += num_written;
#ifdef BGW_DEBUG
@@ -3883,8 +4022,66 @@ BgBufferSync(WritebackContext *wb_context)
return (bufs_to_lap == 0 && recent_alloc == 0);
}
+static inline bool
+BufferTagsSameRel(const BufferTag *tag1, const BufferTag *tag2)
+{
+ return (tag1->spcOid == tag2->spcOid) &&
+ (tag1->dbOid == tag2->dbOid) &&
+ (tag1->relNumber == tag2->relNumber) &&
+ (tag1->forkNum == tag2->forkNum)
+ ;
+}
+
+static bool
+CanMergeWrite(BuffersToWrite *to_write, BufferDesc *cur_buf_hdr)
+{
+ BlockNumber cur_block = cur_buf_hdr->tag.blockNum;
+
+ Assert(to_write->nbuffers > 0); /* can't merge with nothing */
+ Assert(to_write->start_at_tag.relNumber != InvalidOid);
+ Assert(to_write->start_at_tag.blockNum != InvalidBlockNumber);
+
+ Assert(to_write->ioh != NULL);
+
+ /*
+ * First check if the blocknumber is one that we could actually merge,
+ * that's cheaper than checking the tablespace/db/relnumber/fork match.
+ */
+ if (to_write->start_at_tag.blockNum + to_write->nbuffers != cur_block)
+ return false;
+
+ if (!BufferTagsSameRel(&to_write->start_at_tag, &cur_buf_hdr->tag))
+ return false;
+
+ /*
+ * Need to check with smgr how large a write we're allowed to make. To
+ * reduce the overhead of the smgr check, only inquire once, when
+ * processing the first to-be-merged buffer. That avoids the overhead in
+ * the common case of writing out buffers that definitely not mergeable.
+ */
+ if (to_write->nbuffers == 1)
+ {
+ SMgrRelation smgr;
+
+ smgr = smgropen(BufTagGetRelFileLocator(&to_write->start_at_tag), INVALID_PROC_NUMBER);
+
+ to_write->max_combine = smgrmaxcombine(smgr,
+ to_write->start_at_tag.forkNum,
+ to_write->start_at_tag.blockNum);
+ }
+ else
+ {
+ Assert(to_write->max_combine > 0);
+ }
+
+ if (to_write->start_at_tag.blockNum + to_write->max_combine <= cur_block)
+ return false;
+
+ return true;
+}
+
/*
- * SyncOneBuffer -- process a single buffer during syncing.
+ * PrepareToWriteBuffer -- process a single buffer during syncing.
*
* If skip_recently_used is true, we don't write currently-pinned buffers, nor
* buffers marked recently used, as these are not replacement candidates.
@@ -3893,22 +4090,50 @@ BgBufferSync(WritebackContext *wb_context)
* BUF_WRITTEN: we wrote the buffer.
* BUF_REUSABLE: buffer is available for replacement, ie, it has
* pin count 0 and usage count 0.
+ * BUF_CANT_MERGE: can't combine this write with prior writes, caller needs
+ * to issue those first
*
* (BUF_WRITTEN could be set in error if FlushBuffer finds the buffer clean
* after locking it, but we don't care all that much.)
*/
static int
-SyncOneBuffer(int buf_id, bool skip_recently_used, WritebackContext *wb_context)
+PrepareToWriteBuffer(BuffersToWrite *to_write, Buffer buf,
+ bool skip_recently_used,
+ IOQueue *ioq, WritebackContext *wb_context)
{
- BufferDesc *bufHdr = GetBufferDescriptor(buf_id);
+ BufferDesc *cur_buf_hdr = GetBufferDescriptor(buf - 1);
+ uint32 buf_state;
int result = 0;
- uint32 buf_state;
- BufferTag tag;
+ XLogRecPtr cur_buf_lsn;
+ LWLock *content_lock;
+ bool may_block;
+
+ /*
+ * Check if this buffer can be written out together with already prepared
+ * writes. We check before we have pinned the buffer, so the buffer can be
+ * written out and replaced between this check and us pinning the buffer -
+ * we'll recheck below. The reason for the pre-check is that we don't want
+ * to pin the buffer just to find out that we can't merge the IO.
+ */
+ if (to_write->nbuffers != 0)
+ {
+ if (!CanMergeWrite(to_write, cur_buf_hdr))
+ {
+ result |= BUF_CANT_MERGE;
+ return result;
+ }
+ }
+ else
+ {
+ to_write->start_at_tag = cur_buf_hdr->tag;
+ }
/* Make sure we can handle the pin */
ReservePrivateRefCountEntry();
ResourceOwnerEnlarge(CurrentResourceOwner);
+ /* XXX: Should also check if we are allowed to pin one more buffer */
+
/*
* Check whether buffer needs writing.
*
@@ -3918,7 +4143,7 @@ SyncOneBuffer(int buf_id, bool skip_recently_used, WritebackContext *wb_context)
* don't worry because our checkpoint.redo points before log record for
* upcoming changes and so we are not required to write such dirty buffer.
*/
- buf_state = LockBufHdr(bufHdr);
+ buf_state = LockBufHdr(cur_buf_hdr);
if (BUF_STATE_GET_REFCOUNT(buf_state) == 0 &&
BUF_STATE_GET_USAGECOUNT(buf_state) == 0)
@@ -3927,40 +4152,300 @@ SyncOneBuffer(int buf_id, bool skip_recently_used, WritebackContext *wb_context)
}
else if (skip_recently_used)
{
+#if 0
+ elog(LOG, "at block %d: skip recent with nbuffers %d",
+ cur_buf_hdr->tag.blockNum, to_write->nbuffers);
+#endif
/* Caller told us not to write recently-used buffers */
- UnlockBufHdr(bufHdr, buf_state);
+ UnlockBufHdr(cur_buf_hdr, buf_state);
return result;
}
if (!(buf_state & BM_VALID) || !(buf_state & BM_DIRTY))
{
/* It's clean, so nothing to do */
- UnlockBufHdr(bufHdr, buf_state);
+ UnlockBufHdr(cur_buf_hdr, buf_state);
return result;
}
+ /* pin the buffer, from now on its identity can't change anymore */
+ PinBuffer_Locked(cur_buf_hdr);
+
+ /*
+ * Acquire IO, if needed, now that it's likely that we'll need to write.
+ */
+ if (to_write->ioh == NULL)
+ {
+ /* otherwise we should already have acquired a handle */
+ Assert(to_write->nbuffers == 0);
+
+ to_write->ioh = io_queue_acquire_io(ioq);
+ pgaio_io_get_wref(to_write->ioh, &to_write->iow);
+ }
+
+ /*
+ * If we are merging, check if the buffer's identity possibly changed
+ * while we hadn't yet pinned it.
+ *
+ * XXX: It might be worth checking if we still want to write the buffer
+ * out, e.g. it could have been replaced with a buffer that doesn't have
+ * BM_CHECKPOINT_NEEDED set.
+ */
+ if (to_write->nbuffers != 0)
+ {
+ if (!CanMergeWrite(to_write, cur_buf_hdr))
+ {
+ elog(LOG, "changed identity");
+ UnpinBuffer(cur_buf_hdr);
+
+ result |= BUF_CANT_MERGE;
+
+ return result;
+ }
+ }
+
+ may_block = to_write->nbuffers == 0
+ && !pgaio_have_staged()
+ && io_queue_is_empty(ioq)
+ ;
+ content_lock = BufferDescriptorGetContentLock(cur_buf_hdr);
+
+ if (!may_block)
+ {
+ if (LWLockConditionalAcquire(content_lock, LW_SHARED))
+ {
+ /* done */
+ }
+ else if (to_write->nbuffers == 0)
+ {
+ /*
+ * Need to wait for all prior IO to finish before blocking for
+ * lock acquisition, to avoid the risk a deadlock due to us
+ * waiting for another backend that is waiting for our unsubmitted
+ * IO to complete.
+ */
+ pgaio_submit_staged();
+ io_queue_wait_all(ioq);
+
+ elog(DEBUG2, "at block %u: can't block, nbuffers = 0",
+ cur_buf_hdr->tag.blockNum
+ );
+
+ may_block = to_write->nbuffers == 0
+ && !pgaio_have_staged()
+ && io_queue_is_empty(ioq)
+ ;
+ Assert(may_block);
+
+ LWLockAcquire(content_lock, LW_SHARED);
+ }
+ else
+ {
+ elog(DEBUG2, "at block %d: can't block nbuffers = %d",
+ cur_buf_hdr->tag.blockNum,
+ to_write->nbuffers);
+
+ UnpinBuffer(cur_buf_hdr);
+ result |= BUF_CANT_MERGE;
+ Assert(to_write->nbuffers > 0);
+
+ return result;
+ }
+ }
+ else
+ {
+ LWLockAcquire(content_lock, LW_SHARED);
+ }
+
+ if (!may_block)
+ {
+ if (!StartBufferIO(cur_buf_hdr, false, !may_block))
+ {
+ pgaio_submit_staged();
+ io_queue_wait_all(ioq);
+
+ may_block = io_queue_is_empty(ioq) && to_write->nbuffers == 0 && !pgaio_have_staged();
+
+ if (!StartBufferIO(cur_buf_hdr, false, !may_block))
+ {
+ elog(DEBUG2, "at block %d: non-waitable StartBufferIO returns false, %d",
+ cur_buf_hdr->tag.blockNum,
+ may_block);
+
+ /*
+ * FIXME: can't tell whether this is because the buffer has
+ * been cleaned
+ */
+ if (!may_block)
+ {
+ result |= BUF_CANT_MERGE;
+ Assert(to_write->nbuffers > 0);
+ }
+ LWLockRelease(content_lock);
+ UnpinBuffer(cur_buf_hdr);
+
+ return result;
+ }
+ }
+ }
+ else
+ {
+ if (!StartBufferIO(cur_buf_hdr, false, false))
+ {
+ elog(DEBUG2, "waitable StartBufferIO returns false");
+ LWLockRelease(content_lock);
+ UnpinBuffer(cur_buf_hdr);
+
+ /*
+ * FIXME: Historically we returned BUF_WRITTEN in this case, which
+ * seems wrong
+ */
+ return result;
+ }
+ }
+
/*
- * Pin it, share-lock it, write it. (FlushBuffer will do nothing if the
- * buffer is clean by the time we've locked it.)
+ * Run PageGetLSN while holding header lock, since we don't have the
+ * buffer locked exclusively in all cases.
*/
- PinBuffer_Locked(bufHdr);
- LWLockAcquire(BufferDescriptorGetContentLock(bufHdr), LW_SHARED);
+ buf_state = LockBufHdr(cur_buf_hdr);
+
+ cur_buf_lsn = BufferGetLSN(cur_buf_hdr);
- FlushBuffer(bufHdr, NULL, IOOBJECT_RELATION, IOCONTEXT_NORMAL);
+ /* To check if block content changes while flushing. - vadim 01/17/97 */
+ buf_state &= ~BM_JUST_DIRTIED;
- LWLockRelease(BufferDescriptorGetContentLock(bufHdr));
+ UnlockBufHdr(cur_buf_hdr, buf_state);
- tag = bufHdr->tag;
+ to_write->buffers[to_write->nbuffers] = buf;
+ to_write->nbuffers++;
- UnpinBuffer(bufHdr);
+ if (buf_state & BM_PERMANENT &&
+ (to_write->max_lsn == InvalidXLogRecPtr || to_write->max_lsn < cur_buf_lsn))
+ {
+ to_write->max_lsn = cur_buf_lsn;
+ }
+
+ result |= BUF_WRITTEN;
+
+ return result;
+}
+
+static void
+WriteBuffers(BuffersToWrite *to_write,
+ IOQueue *ioq, WritebackContext *wb_context)
+{
+ SMgrRelation smgr;
+ Buffer first_buf;
+ BufferDesc *first_buf_hdr;
+ bool needs_checksum;
+
+ Assert(to_write->nbuffers > 0 && to_write->nbuffers <= io_combine_limit);
+
+ first_buf = to_write->buffers[0];
+ first_buf_hdr = GetBufferDescriptor(first_buf - 1);
+
+ smgr = smgropen(BufTagGetRelFileLocator(&first_buf_hdr->tag), INVALID_PROC_NUMBER);
/*
- * SyncOneBuffer() is only called by checkpointer and bgwriter, so
- * IOContext will always be IOCONTEXT_NORMAL.
+ * Force XLOG flush up to buffer's LSN. This implements the basic WAL
+ * rule that log updates must hit disk before any of the data-file changes
+ * they describe do.
+ *
+ * However, this rule does not apply to unlogged relations, which will be
+ * lost after a crash anyway. Most unlogged relation pages do not bear
+ * LSNs since we never emit WAL records for them, and therefore flushing
+ * up through the buffer LSN would be useless, but harmless. However,
+ * GiST indexes use LSNs internally to track page-splits, and therefore
+ * unlogged GiST pages bear "fake" LSNs generated by
+ * GetFakeLSNForUnloggedRel. It is unlikely but possible that the fake
+ * LSN counter could advance past the WAL insertion point; and if it did
+ * happen, attempting to flush WAL through that location would fail, with
+ * disastrous system-wide consequences. To make sure that can't happen,
+ * skip the flush if the buffer isn't permanent.
*/
- ScheduleBufferTagForWriteback(wb_context, IOCONTEXT_NORMAL, &tag);
+ if (to_write->max_lsn != InvalidXLogRecPtr)
+ XLogFlush(to_write->max_lsn);
+
+ /*
+ * Now it's safe to write the buffer to disk. Note that no one else should
+ * have been able to write it, while we were busy with log flushing,
+ * because we got the exclusive right to perform I/O by setting the
+ * BM_IO_IN_PROGRESS bit.
+ */
+
+ for (int nbuf = 0; nbuf < to_write->nbuffers; nbuf++)
+ {
+ Buffer cur_buf = to_write->buffers[nbuf];
+ BufferDesc *cur_buf_hdr = GetBufferDescriptor(cur_buf - 1);
+ Block bufBlock;
+ char *bufToWrite;
+
+ bufBlock = BufHdrGetBlock(cur_buf_hdr);
+ needs_checksum = PageNeedsChecksumCopy((Page) bufBlock);
+
+ /*
+ * Update page checksum if desired. Since we have only shared lock on
+ * the buffer, other processes might be updating hint bits in it, so
+ * we must copy the page to a bounce buffer if we do checksumming.
+ */
+ if (needs_checksum)
+ {
+ PgAioBounceBuffer *bb = pgaio_bounce_buffer_get();
- return result | BUF_WRITTEN;
+ pgaio_io_assoc_bounce_buffer(to_write->ioh, bb);
+
+ bufToWrite = pgaio_bounce_buffer_buffer(bb);
+ memcpy(bufToWrite, bufBlock, BLCKSZ);
+ PageSetChecksumInplace((Page) bufToWrite, cur_buf_hdr->tag.blockNum);
+ }
+ else
+ {
+ bufToWrite = bufBlock;
+ }
+
+ to_write->data_ptrs[nbuf] = bufToWrite;
+ }
+
+ pgaio_io_set_handle_data_32(to_write->ioh,
+ (uint32 *) to_write->buffers,
+ to_write->nbuffers);
+ pgaio_io_register_callbacks(to_write->ioh, PGAIO_HCB_SHARED_BUFFER_WRITEV, 0);
+
+ smgrstartwritev(to_write->ioh, smgr,
+ BufTagGetForkNum(&first_buf_hdr->tag),
+ first_buf_hdr->tag.blockNum,
+ to_write->data_ptrs,
+ to_write->nbuffers,
+ false);
+ pgstat_count_io_op(IOOBJECT_RELATION, IOCONTEXT_NORMAL,
+ IOOP_WRITE, 1, BLCKSZ * to_write->nbuffers);
+
+
+ for (int nbuf = 0; nbuf < to_write->nbuffers; nbuf++)
+ {
+ Buffer cur_buf = to_write->buffers[nbuf];
+ BufferDesc *cur_buf_hdr = GetBufferDescriptor(cur_buf - 1);
+
+ UnpinBuffer(cur_buf_hdr);
+ }
+
+ io_queue_track(ioq, &to_write->iow);
+ to_write->total_writes++;
+
+ /* clear state for next write */
+ to_write->nbuffers = 0;
+ to_write->start_at_tag.relNumber = InvalidOid;
+ to_write->start_at_tag.blockNum = InvalidBlockNumber;
+ to_write->max_combine = 0;
+ to_write->max_lsn = InvalidXLogRecPtr;
+ to_write->ioh = NULL;
+ pgaio_wref_clear(&to_write->iow);
+
+ /*
+ * FIXME: Implement issuing writebacks (note wb_context isn't used here).
+ * Possibly needs to be integrated with io_queue.c.
+ */
}
/*
@@ -4334,6 +4819,7 @@ FlushBuffer(BufferDesc *buf, SMgrRelation reln, IOObject io_object,
error_context_stack = errcallback.previous;
}
+
/*
* RelationGetNumberOfBlocksInFork
* Determines the current number of pages in the specified relation fork.
diff --git a/src/backend/storage/page/bufpage.c b/src/backend/storage/page/bufpage.c
index ecc81aacfc3..71dc6f559dd 100644
--- a/src/backend/storage/page/bufpage.c
+++ b/src/backend/storage/page/bufpage.c
@@ -1480,6 +1480,16 @@ PageIndexTupleOverwrite(Page page, OffsetNumber offnum,
return true;
}
+bool
+PageNeedsChecksumCopy(Page page)
+{
+ if (PageIsNew(page))
+ return false;
+
+ /* If we don't need a checksum, just return the passed-in data */
+ return DataChecksumsEnabled();
+}
+
/*
* Set checksum for a page in shared buffers.
diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list
index e247f6ac7d0..57634962dd3 100644
--- a/src/tools/pgindent/typedefs.list
+++ b/src/tools/pgindent/typedefs.list
@@ -348,6 +348,7 @@ BufferManagerRelation
BufferStrategyControl
BufferTag
BufferUsage
+BuffersToWrite
BuildAccumulator
BuiltinScript
BulkInsertState
--
2.48.1.76.g4e746b1a31.dirty
[text/x-diff] v2.13-0024-Ensure-a-resowner-exists-for-all-paths-that-ma.patch (2.6K, ../../o2dtdrlc2xqpjycsfuvv7u6zyrqj7jfxqu32et3lyelkwce2jy@3k3fwdiiu73v/25-v2.13-0024-Ensure-a-resowner-exists-for-all-paths-that-ma.patch)
download | inline diff:
From 1f2386680e7b63c55b171737d1a0a2bf7a109348 Mon Sep 17 00:00:00 2001
From: Andres Freund <andres@anarazel.de>
Date: Tue, 18 Mar 2025 14:40:06 -0400
Subject: [PATCH v2.13 24/28] Ensure a resowner exists for all paths that may
perform AIO
Reviewed-by: Noah Misch <noah@leadboat.com>
Discussion: https://postgr.es/m/1f6b50a7-38ef-4d87-8246-786d39f46ab9@iki.fi
---
src/backend/bootstrap/bootstrap.c | 7 +++++++
src/backend/replication/logical/logical.c | 6 ++++++
src/backend/utils/init/postinit.c | 6 +++++-
3 files changed, 18 insertions(+), 1 deletion(-)
diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c
index 6db864892d0..e554504e1f0 100644
--- a/src/backend/bootstrap/bootstrap.c
+++ b/src/backend/bootstrap/bootstrap.c
@@ -361,8 +361,15 @@ BootstrapModeMain(int argc, char *argv[], bool check_only)
BaseInit();
bootstrap_signals();
+
+ /* need a resowner for IO during BootStrapXLOG() */
+ CreateAuxProcessResourceOwner();
+
BootStrapXLOG(bootstrap_data_checksum_version);
+ ReleaseAuxProcessResources(true);
+ CurrentResourceOwner = NULL;
+
/*
* To ensure that src/common/link-canary.c is linked into the backend, we
* must call it from somewhere. Here is as good as anywhere.
diff --git a/src/backend/replication/logical/logical.c b/src/backend/replication/logical/logical.c
index 8ea846bfc3b..2b936714040 100644
--- a/src/backend/replication/logical/logical.c
+++ b/src/backend/replication/logical/logical.c
@@ -386,6 +386,12 @@ CreateInitDecodingContext(const char *plugin,
slot->data.plugin = plugin_name;
SpinLockRelease(&slot->mutex);
+ if (CurrentResourceOwner == NULL)
+ {
+ Assert(am_walsender);
+ CurrentResourceOwner = AuxProcessResourceOwner;
+ }
+
if (XLogRecPtrIsInvalid(restart_lsn))
ReplicationSlotReserveWal();
else
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index 7958ea11b73..222e24bcb08 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -785,8 +785,12 @@ InitPostgres(const char *in_dbname, Oid dboid,
* We don't yet have an aux-process resource owner, but StartupXLOG
* and ShutdownXLOG will need one. Hence, create said resource owner
* (and register a callback to clean it up after ShutdownXLOG runs).
+ *
+ * In bootstrap mode CreateAuxProcessResourceOwner() was already
+ * called in BootstrapModeMain().
*/
- CreateAuxProcessResourceOwner();
+ if (!bootstrap)
+ CreateAuxProcessResourceOwner();
StartupXLOG();
/* Release (and warn about) any buffer pins leaked in StartupXLOG */
--
2.48.1.76.g4e746b1a31.dirty
[text/x-diff] v2.13-0025-Temporary-Increase-BAS_BULKREAD-size.patch (1.3K, ../../o2dtdrlc2xqpjycsfuvv7u6zyrqj7jfxqu32et3lyelkwce2jy@3k3fwdiiu73v/26-v2.13-0025-Temporary-Increase-BAS_BULKREAD-size.patch)
download | inline diff:
From b34a8b921ea3491a65098620fab3973c795c339e Mon Sep 17 00:00:00 2001
From: Andres Freund <andres@anarazel.de>
Date: Tue, 18 Mar 2025 14:40:06 -0400
Subject: [PATCH v2.13 25/28] Temporary: Increase BAS_BULKREAD size
Without this we only can execute very little AIO for sequential scans, as
there's just not enough buffers in the ring. This isn't the right fix, as
just increasing the ring size can have negative performance implications in
workloads where the kernel has all the data cached.
Author:
Reviewed-By:
Discussion: https://postgr.es/m/
Backpatch:
---
src/backend/storage/buffer/freelist.c | 7 ++++++-
1 file changed, 6 insertions(+), 1 deletion(-)
diff --git a/src/backend/storage/buffer/freelist.c b/src/backend/storage/buffer/freelist.c
index 336715b6c63..b72a5957a20 100644
--- a/src/backend/storage/buffer/freelist.c
+++ b/src/backend/storage/buffer/freelist.c
@@ -555,7 +555,12 @@ GetAccessStrategy(BufferAccessStrategyType btype)
return NULL;
case BAS_BULKREAD:
- ring_size_kb = 256;
+
+ /*
+ * FIXME: Temporary increase to allow large enough streaming reads
+ * to actually benefit from AIO. This needs a better solution.
+ */
+ ring_size_kb = 2 * 1024;
break;
case BAS_BULKWRITE:
ring_size_kb = 16 * 1024;
--
2.48.1.76.g4e746b1a31.dirty
[text/x-diff] v2.13-0026-WIP-Use-MAP_POPULATE.patch (1.1K, ../../o2dtdrlc2xqpjycsfuvv7u6zyrqj7jfxqu32et3lyelkwce2jy@3k3fwdiiu73v/27-v2.13-0026-WIP-Use-MAP_POPULATE.patch)
download | inline diff:
From 64dd33d1cca8cbadcfd38320957862bd74e55016 Mon Sep 17 00:00:00 2001
From: Andres Freund <andres@anarazel.de>
Date: Tue, 18 Mar 2025 14:40:06 -0400
Subject: [PATCH v2.13 26/28] WIP: Use MAP_POPULATE
For benchmarking it's quite annoying that the first time a memory is touched
has completely different perf characteristics than subsequent accesses. Using
MAP_POPULATE reduces that substantially.
---
src/backend/port/sysv_shmem.c | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/src/backend/port/sysv_shmem.c b/src/backend/port/sysv_shmem.c
index 197926d44f6..a700b02d5a1 100644
--- a/src/backend/port/sysv_shmem.c
+++ b/src/backend/port/sysv_shmem.c
@@ -620,7 +620,7 @@ CreateAnonymousSegment(Size *size)
allocsize += hugepagesize - (allocsize % hugepagesize);
ptr = mmap(NULL, allocsize, PROT_READ | PROT_WRITE,
- PG_MMAP_FLAGS | mmap_flags, -1, 0);
+ PG_MMAP_FLAGS | MAP_POPULATE | mmap_flags, -1, 0);
mmap_errno = errno;
if (huge_pages == HUGE_PAGES_TRY && ptr == MAP_FAILED)
elog(DEBUG1, "mmap(%zu) with MAP_HUGETLB failed, huge pages disabled: %m",
--
2.48.1.76.g4e746b1a31.dirty
^ permalink raw reply [nested|flat] 153+ messages in thread
* Re: AIO v2.5
2025-03-04 19:00 AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-10 18:23 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-11 19:41 ` Re: AIO v2.5 Noah Misch <noah@leadboat.com>
2025-03-11 23:55 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-12 03:57 ` Re: AIO v2.5 Noah Misch <noah@leadboat.com>
2025-03-12 17:06 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-12 23:23 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-13 10:53 ` Re: AIO v2.5 Antonin Houska <ah@cybertec.at>
2025-03-14 19:43 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-17 20:00 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-18 20:12 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-21 01:58 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-25 01:18 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-27 01:07 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
@ 2025-03-27 20:58 ` Andres Freund <andres@anarazel.de>
2025-03-28 03:22 ` Re: AIO v2.5 Noah Misch <noah@leadboat.com>
0 siblings, 1 reply; 153+ messages in thread
From: Andres Freund @ 2025-03-27 20:58 UTC (permalink / raw)
To: Noah Misch <noah@leadboat.com>; +Cc: pgsql-hackers@postgresql.org, Thomas Munro <thomas.munro@gmail.com>; Heikki Linnakangas <hlinnaka@iki.fi>; Robert Haas <robertmhaas@gmail.com>; Jakub Wartak <jakub.wartak@enterprisedb.com>; Jelte Fennema-Nio <postgres@jeltef.nl>; Antonin Houska <ah@cybertec.at>
Hi,
On 2025-03-26 21:07:40 -0400, Andres Freund wrote:
> TODO
> ...
> - Add an explicit test for the checksum verification in the completion callback
>
> There is an existing test for testing an invalid page due to page header
> verification in test_aio, but not for checksum failures.
>
> I think it's indirectly covered (e.g. in amcheck), but seems better to test
> it explicitly.
Ah, for crying out loud. As it turns out, no, we do not have *ANY* tests for
this for the server-side. Not a single one. I'm somewhat apoplectic,
data_checksums is a really complicated feature, which we just started *turning
on by default*, without a single test of the failure behaviour, when detecting
failures is the one thing the feature is supposed to do.
I now wrote some tests. And I both regret doing so (because it found problems,
which would have been apparent long ago, if the feature had come with *any*
tests, if I had gone the same way I could have just pushed stuff) and am glad
I did (because I dislike pushing broken stuff).
I have to admit, I was tempted to just ignore this issue and just not say
anything about tests for checksum failures anymore.
Problems:
1) PageIsVerifiedExtended() emits a WARNING, just like with ZERO_ON_ERROR, we
don't want to emit it in a) io workers b) another backend if it completes
the error.
This isn't hard to address, we can add PIV_LOG_LOG (or something like that)
to emit it at a different log level and an out-parameter to trigger sending
a warning / adjust the warning/error message we already emit once the
issuer completes the IO.
2) With IO workers (and "foreign completors", in rare cases), the checksum
failures would be attributed wrongly, as it reports all stats to
MyDatabaseId
As it turns out, this is already borked on master for shared relations,
since pg_stat_database.checksum_failures has existed, see [1].
This isn't too hard to fix, if we adjust the signature to
PageIsVerifiedExtended() to pass in the database oid. But see also 3)
3) We can't pgstat_report_checksum_failure() during the completion callback,
as it *sometimes* allocates memory
Aside from the allocation-in-critical-section asserts, I think this is
*extremely* unlikely to actually cause a problem in practice. But we don't
want to rely on that, obviously.
Addressing the first two is pretty simple and/or needs to be done anyway,
since it's a currently existing bug, as discussed in [1].
Addressing 3) is not at all trivial. Here's what I've thought of so far:
Approach I)
My first thoughts were around trying to make the relevant pgstat
infrastructure either not need to allocate memory, or handle memory
allocation failures gracefully.
Unfortunately that seems not really viable:
The most successful approach I tried was to report stats directly to the
dshash table, and only report stats if there's already an entry (which
there just about always will, except for a very short period after stats
have been reset).
Unfortunately that fails because to access the shared memory with the stats
data we need to do dsa_get_address(), which can fail if the relevant dsm
segment wasn't already mapped in the current process (it allocates memory
in the process of mapping in the segment). There's no API to do that
without erroring out.
That aspect rules out a number of other approaches that sounded like they
could work - we e.g. could increase the refcount of the relevant pgstat
entry before issuing IO, ensuring that it's around by the time we need to
report. But that wouldn't get around the issue of needing to map in the dsm
segment.
Approach II)
Don't report the error in the completion callback. The obvious place would be
to do it where we we'll raise the warning/error in the issuing process. The
big disadvantage is that that that could lead to under-counting checksum
errors:
a) A read stream does 2+ concurrent reads for the same relation, and more than
one encounters checksum errors. When processing the results for the first
failed read, we raise an error and thus won't process the results of the
second+ reads with errors.
b) A read is started asynchronously, but before the backend gets around to
processing the result of the IO, it errors out during that other work
(possibly due to a cancellation). Because the backend never looked at the
results of the IOs, the checksum errors don't get accounted for.
b) doesn't overly bother me, but a) seems problematic.
Approach III)
Accumulate checksum errors in two backend local variables (one for database
specific errors, one for errors on shared relations), which will be flushed by
the backend that issued IO during the next pgstat_report_start().
Two disadvantages:
- Accumulation of errors will be delayed until the next
pgstat_report_start(). That seems acceptable, after all we do so far a lot
of other stats.
- We need to register a local callback for shared buffer reads, which don't
need them today . That's a small bit of added overhead. It's a shame to do
so for counters that approximately never get incremented.
Approach IV):
Embracy piercing abstractions / generic infrastructure and put two atomic
variables (one for shared one for the backend's database) in some
backend-specific shared memory (e.g. the backend's PgAioBackend or PGPROC) and
update that in the completion callback. Flush that variable to the shared
stats in pgstat_report_start() or such.
This would avoid the need for the local completion callback, and would also
allow to introduce a function see the number "unflushed" checksum errors. It
also doesn't require transporting the number of errors between the shared
callback and the local callback - but we might want to do have that for the
error message anyway.
I wish the new-to-18 pgstat_backend() were designed in a way to make this
possible in a nice way. But unfortunately it puts the backend-specific data in
the dshash table / dynamic shared memory, rather than in a MaxBackends +
NUM_AUX sized array array in plain shared memory. As explained in I), we can't
rely on having the entire array mapped. Leaving the issue from this email
aside, that also adds a fair bit of overhead to other cases.
Does anybody have better ideas?
I think II), III) and IV) are all relatively simple to implement.
The most complicated bit is that a bit of bit-squeezing is necessary to fit
the number of checksum errors (in addition to the number of otherwise invalid
pages) into the available space for error data. It's doable. We could also
just increase the size of PgAioResult.
I've implemented II), but I'm not sure the disadvantages are acceptable.
Greetings,
Andres Freund
[1] https://postgr.es/m/mglpvvbhighzuwudjxzu4br65qqcxsnyvio3nl4fbog3qknwhg%40e4gt7npsohuz
^ permalink raw reply [nested|flat] 153+ messages in thread
* Re: AIO v2.5
2025-03-04 19:00 AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-10 18:23 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-11 19:41 ` Re: AIO v2.5 Noah Misch <noah@leadboat.com>
2025-03-11 23:55 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-12 03:57 ` Re: AIO v2.5 Noah Misch <noah@leadboat.com>
2025-03-12 17:06 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-12 23:23 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-13 10:53 ` Re: AIO v2.5 Antonin Houska <ah@cybertec.at>
2025-03-14 19:43 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-17 20:00 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-18 20:12 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-21 01:58 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-25 01:18 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-27 01:07 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-27 20:58 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
@ 2025-03-28 03:22 ` Noah Misch <noah@leadboat.com>
2025-03-28 12:54 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
0 siblings, 1 reply; 153+ messages in thread
From: Noah Misch @ 2025-03-28 03:22 UTC (permalink / raw)
To: Andres Freund <andres@anarazel.de>; +Cc: pgsql-hackers@postgresql.org, Thomas Munro <thomas.munro@gmail.com>; Heikki Linnakangas <hlinnaka@iki.fi>; Robert Haas <robertmhaas@gmail.com>; Jakub Wartak <jakub.wartak@enterprisedb.com>; Jelte Fennema-Nio <postgres@jeltef.nl>; Antonin Houska <ah@cybertec.at>
On Thu, Mar 27, 2025 at 04:58:11PM -0400, Andres Freund wrote:
> I now wrote some tests. And I both regret doing so (because it found problems,
> which would have been apparent long ago, if the feature had come with *any*
> tests, if I had gone the same way I could have just pushed stuff) and am glad
> I did (because I dislike pushing broken stuff).
>
> I have to admit, I was tempted to just ignore this issue and just not say
> anything about tests for checksum failures anymore.
I don't blame you.
> 3) We can't pgstat_report_checksum_failure() during the completion callback,
> as it *sometimes* allocates memory
>
> Aside from the allocation-in-critical-section asserts, I think this is
> *extremely* unlikely to actually cause a problem in practice. But we don't
> want to rely on that, obviously.
> Addressing 3) is not at all trivial. Here's what I've thought of so far:
>
>
> Approach I)
> Unfortunately that fails because to access the shared memory with the stats
> data we need to do dsa_get_address()
> Approach II)
>
> Don't report the error in the completion callback. The obvious place would be
> to do it where we we'll raise the warning/error in the issuing process. The
> big disadvantage is that that that could lead to under-counting checksum
> errors:
>
> a) A read stream does 2+ concurrent reads for the same relation, and more than
> one encounters checksum errors. When processing the results for the first
> failed read, we raise an error and thus won't process the results of the
> second+ reads with errors.
>
> b) A read is started asynchronously, but before the backend gets around to
> processing the result of the IO, it errors out during that other work
> (possibly due to a cancellation). Because the backend never looked at the
> results of the IOs, the checksum errors don't get accounted for.
>
> b) doesn't overly bother me, but a) seems problematic.
While neither are great, I could live with both. I guess I'm optimistic that
clusters experiencing checksum failures won't lose enough reports to these
loss sources to make the difference in whether monitoring catches them. In
other words, a cluster will report N failures without these losses and N-K
after these losses. If N is large enough for relevant monitoring to flag the
cluster appropriately, N-K will also be large enough.
> Approach III)
>
> Accumulate checksum errors in two backend local variables (one for database
> specific errors, one for errors on shared relations), which will be flushed by
> the backend that issued IO during the next pgstat_report_start().
>
> Two disadvantages:
>
> - Accumulation of errors will be delayed until the next
> pgstat_report_start(). That seems acceptable, after all we do so far a lot
> of other stats.
Yep, acceptable.
> - We need to register a local callback for shared buffer reads, which don't
> need them today . That's a small bit of added overhead. It's a shame to do
> so for counters that approximately never get incremented.
Fair concern. An idea is to let the complete_shared callback change the
callback list associated with the IO, so it could change
PGAIO_HCB_SHARED_BUFFER_READV to PGAIO_HCB_SHARED_BUFFER_READV_SLOW. The
latter would differ from the former only in having the extra local callback.
Could that help? I think the only overhead is using more PGAIO_HCB numbers.
We currently reserve 256 (uint8), but one could imagine trying to pack into
fewer bits. That said, this wouldn't paint us into a corner. We could change
the approach later.
pgaio_io_call_complete_local() starts a critical section. Is that a problem
for this approach?
> Approach IV):
>
> Embracy piercing abstractions / generic infrastructure and put two atomic
> variables (one for shared one for the backend's database) in some
> backend-specific shared memory (e.g. the backend's PgAioBackend or PGPROC) and
> update that in the completion callback. Flush that variable to the shared
> stats in pgstat_report_start() or such.
I could live with that. I feel better about Approach III currently, though.
Overall, I'm feeling best about III long-term, but II may be the right
tactical choice.
> Does anybody have better ideas?
I think no, but here are some ideas I tossed around:
- Like your Approach III, but have the completing process store the count
locally and flush it, instead of the staging process doing so. Would need
more than 2 slots, but we could have a fixed number of slots and just
discard any reports that arrive with all slots full. Reporting checksum
failures in, say, 8 databases in quick succession probably tells the DBA
there's "enough corruption to start worrying". Missing the 9th database
would be okay.
- Pre-warm the memory allocations and DSAs we could possibly need, so we can
report those stats in critical sections, from the completing process. Bad
since there's an entry per database, hence no reasonable limit on how much
memory a process might need to pre-warm. We could even end up completing an
IO for a database that didn't exist on entry to our critical section.
- Skip the checksum pgstats if we're completing in a critical section.
Doesn't work since we _always_ make a critical section to complete I/O.
This email isn't as well-baked as I like, but the alternative was delaying it
24-48h depending on how other duties go over those hours. My v2.13 review is
still in-progress, too.
^ permalink raw reply [nested|flat] 153+ messages in thread
* Re: AIO v2.5
2025-03-04 19:00 AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-10 18:23 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-11 19:41 ` Re: AIO v2.5 Noah Misch <noah@leadboat.com>
2025-03-11 23:55 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-12 03:57 ` Re: AIO v2.5 Noah Misch <noah@leadboat.com>
2025-03-12 17:06 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-12 23:23 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-13 10:53 ` Re: AIO v2.5 Antonin Houska <ah@cybertec.at>
2025-03-14 19:43 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-17 20:00 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-18 20:12 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-21 01:58 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-25 01:18 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-27 01:07 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-27 20:58 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-28 03:22 ` Re: AIO v2.5 Noah Misch <noah@leadboat.com>
@ 2025-03-28 12:54 ` Andres Freund <andres@anarazel.de>
2025-03-28 12:57 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-29 03:35 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
0 siblings, 2 replies; 153+ messages in thread
From: Andres Freund @ 2025-03-28 12:54 UTC (permalink / raw)
To: Noah Misch <noah@leadboat.com>; +Cc: pgsql-hackers@postgresql.org, Thomas Munro <thomas.munro@gmail.com>; Heikki Linnakangas <hlinnaka@iki.fi>; Robert Haas <robertmhaas@gmail.com>; Jakub Wartak <jakub.wartak@enterprisedb.com>; Jelte Fennema-Nio <postgres@jeltef.nl>; Antonin Houska <ah@cybertec.at>
Hi,
On 2025-03-27 20:22:23 -0700, Noah Misch wrote:
> On Thu, Mar 27, 2025 at 04:58:11PM -0400, Andres Freund wrote:
> > Don't report the error in the completion callback. The obvious place would be
> > to do it where we we'll raise the warning/error in the issuing process. The
> > big disadvantage is that that that could lead to under-counting checksum
> > errors:
> >
> > a) A read stream does 2+ concurrent reads for the same relation, and more than
> > one encounters checksum errors. When processing the results for the first
> > failed read, we raise an error and thus won't process the results of the
> > second+ reads with errors.
> >
> > b) A read is started asynchronously, but before the backend gets around to
> > processing the result of the IO, it errors out during that other work
> > (possibly due to a cancellation). Because the backend never looked at the
> > results of the IOs, the checksum errors don't get accounted for.
> >
> > b) doesn't overly bother me, but a) seems problematic.
>
> While neither are great, I could live with both. I guess I'm optimistic that
> clusters experiencing checksum failures won't lose enough reports to these
> loss sources to make the difference in whether monitoring catches them. In
> other words, a cluster will report N failures without these losses and N-K
> after these losses. If N is large enough for relevant monitoring to flag the
> cluster appropriately, N-K will also be large enough.
That's true.
> > Approach III)
> >
> > Accumulate checksum errors in two backend local variables (one for database
> > specific errors, one for errors on shared relations), which will be flushed by
> > the backend that issued IO during the next pgstat_report_start().
FWIW, two variables turn out to not quite suffice - as I realized later, we
actually can issue IO on behalf of arbitrary databases, due to
ScanSourceDatabasePgClass() and RelationCopyStorageUsingBuffer().
That unfortunately makes it much harder to be able to guarantee that the
completor of an IO has the DSM segment for a pg_stat_database stats entry
mapped.
> > - We need to register a local callback for shared buffer reads, which don't
> > need them today . That's a small bit of added overhead. It's a shame to do
> > so for counters that approximately never get incremented.
>
> Fair concern. An idea is to let the complete_shared callback change the
> callback list associated with the IO, so it could change
> PGAIO_HCB_SHARED_BUFFER_READV to PGAIO_HCB_SHARED_BUFFER_READV_SLOW. The
> latter would differ from the former only in having the extra local callback.
> Could that help? I think the only overhead is using more PGAIO_HCB numbers.
I think changing the callback could work - I'll do some measurements in a
coffee or two, but I suspect the overhead is not worth being too worried about
for now. There's a different aspect that worries me slightly more, see
further down.
> We currently reserve 256 (uint8), but one could imagine trying to pack into
> fewer bits.
Yea, my current local worktree reduces it to 6 bits for now, to make space for
keeping track of the number of checksum failures in error data (as part of
that adds defines for the bit widths). If that becomes an issue we can make
PgAioResult wider, but I suspect that won't be too soon.
One simplification that we could make is to only ever report one checksum
failure for each IO, even if N buffers failed - after all that's what HEAD
does (by virtue of throwing an error after the first). Then we'd not track the
number of checksum errors.
> That said, this wouldn't paint us into a corner. We could change the
> approach later.
Indeed - I think we mainly need something that works for now. I think medium
term the right fix here would be to make sure that the stats can be accounted
for with just an atomic increment somewhere.
We've had several discussions around having an in-memory datastructure for
every relation that currently has buffer in shared_buffers, to store e.g. the
relation length and the sync requests. If we get that, I think Thomas has a
prototype, we can accumulate the number of checksum errors in there, for
example. It'd also allow to address the biggest blocker for writes, namely
that RememberSyncRequest() could fail, *after* IO comletion.
> pgaio_io_call_complete_local() starts a critical section. Is that a problem
> for this approach?
I think we can make it not a problem - I added a
pgstat_prepare_report_checksum_failure(dboid) that ensures the calling backend
has a reference to the relevant shared memory stats entry. If we make the rule
that it has to be called *before* starting buffered IO (i.e. in
AsyncReadBuffers()), we can be sure the stats reference still exists by the
time local completion runs (as the isn't a way to have the stats entry dropped
without dropping the database, which isn't possible while a) the database
still is connected to, for normal IO b) the CREATE DATABASE is still running).
Unfortunately pgstat_prepare_report_checksum_failure() has to do a lookup in a
local hashtable. That's more expensive than an indirect function call
(i.e. the added local callback). I hope^Wsuspect it'll still be fine, and if
not we can apply a mini-cache for the current database, which is surely the
only thing that ever matters for performance.
> > Approach IV):
> >
> > Embracy piercing abstractions / generic infrastructure and put two atomic
> > variables (one for shared one for the backend's database) in some
> > backend-specific shared memory (e.g. the backend's PgAioBackend or PGPROC) and
> > update that in the completion callback. Flush that variable to the shared
> > stats in pgstat_report_start() or such.
>
> I could live with that. I feel better about Approach III currently, though.
> Overall, I'm feeling best about III long-term, but II may be the right
> tactical choice.
I think it's easy to change between these approaches. Both require that we
encode the number of checksum failures in the result, which is where most of
the complexity lies (but still a rather surmountable amount of complexity).
> I think no, but here are some ideas I tossed around:
>
> - Like your Approach III, but have the completing process store the count
> locally and flush it, instead of the staging process doing so. Would need
> more than 2 slots, but we could have a fixed number of slots and just
> discard any reports that arrive with all slots full. Reporting checksum
> failures in, say, 8 databases in quick succession probably tells the DBA
> there's "enough corruption to start worrying". Missing the 9th database
> would be okay.
Yea. I think that'd be an ok fallback, but if we can make III' work, it'd be
nicer.
> - Pre-warm the memory allocations and DSAs we could possibly need, so we can
> report those stats in critical sections, from the completing process. Bad
> since there's an entry per database, hence no reasonable limit on how much
> memory a process might need to pre-warm. We could even end up completing an
> IO for a database that didn't exist on entry to our critical section.
I experimented with this one - it works surprisingly well, because for IO
workers we could just do the pre-warming outside of the critical section, and
it's *exceedingly* rare that any other completor would ever need to complete
IO for another database than the current / a shared relation.
But it does leave a nasty edge case, that we'd just have to accept. I guess we
could just make it so that in that case stats aren't reported.
But it seems pretty ugly.
> This email isn't as well-baked as I like, but the alternative was delaying it
> 24-48h depending on how other duties go over those hours. My v2.13 review is
> still in-progress, too.
It's appreciated!
Greetings,
Andres Freund
^ permalink raw reply [nested|flat] 153+ messages in thread
* Re: AIO v2.5
2025-03-04 19:00 AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-10 18:23 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-11 19:41 ` Re: AIO v2.5 Noah Misch <noah@leadboat.com>
2025-03-11 23:55 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-12 03:57 ` Re: AIO v2.5 Noah Misch <noah@leadboat.com>
2025-03-12 17:06 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-12 23:23 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-13 10:53 ` Re: AIO v2.5 Antonin Houska <ah@cybertec.at>
2025-03-14 19:43 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-17 20:00 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-18 20:12 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-21 01:58 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-25 01:18 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-27 01:07 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-27 20:58 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-28 03:22 ` Re: AIO v2.5 Noah Misch <noah@leadboat.com>
2025-03-28 12:54 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
@ 2025-03-28 12:57 ` Andres Freund <andres@anarazel.de>
1 sibling, 0 replies; 153+ messages in thread
From: Andres Freund @ 2025-03-28 12:57 UTC (permalink / raw)
To: Noah Misch <noah@leadboat.com>; +Cc: pgsql-hackers@postgresql.org, Thomas Munro <thomas.munro@gmail.com>; Heikki Linnakangas <hlinnaka@iki.fi>; Robert Haas <robertmhaas@gmail.com>; Jakub Wartak <jakub.wartak@enterprisedb.com>; Jelte Fennema-Nio <postgres@jeltef.nl>; Antonin Houska <ah@cybertec.at>
Hi,
On 2025-03-28 08:54:42 -0400, Andres Freund wrote:
> One simplification that we could make is to only ever report one checksum
> failure for each IO, even if N buffers failed - after all that's what HEAD
> does (by virtue of throwing an error after the first). Then we'd not track the
> number of checksum errors.
Just after sending, I thought of another variation: Report the number of
*invalid* pages (which we already track) as checksum errors, if there was at
least on checksum error.
It's imo rather weird that we track checksum errors but we don't track invalid
page headers, despite the latter being an even worse indication of something
having gone wrong...
Greetings,
Andres Freund
^ permalink raw reply [nested|flat] 153+ messages in thread
* Re: AIO v2.5
2025-03-04 19:00 AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-10 18:23 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-11 19:41 ` Re: AIO v2.5 Noah Misch <noah@leadboat.com>
2025-03-11 23:55 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-12 03:57 ` Re: AIO v2.5 Noah Misch <noah@leadboat.com>
2025-03-12 17:06 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-12 23:23 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-13 10:53 ` Re: AIO v2.5 Antonin Houska <ah@cybertec.at>
2025-03-14 19:43 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-17 20:00 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-18 20:12 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-21 01:58 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-25 01:18 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-27 01:07 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-27 20:58 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-28 03:22 ` Re: AIO v2.5 Noah Misch <noah@leadboat.com>
2025-03-28 12:54 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
@ 2025-03-29 03:35 ` Andres Freund <andres@anarazel.de>
2025-03-29 13:41 ` Re: AIO v2.5 Noah Misch <noah@leadboat.com>
1 sibling, 1 reply; 153+ messages in thread
From: Andres Freund @ 2025-03-29 03:35 UTC (permalink / raw)
To: Noah Misch <noah@leadboat.com>; +Cc: pgsql-hackers@postgresql.org, Thomas Munro <thomas.munro@gmail.com>; Heikki Linnakangas <hlinnaka@iki.fi>; Robert Haas <robertmhaas@gmail.com>; Jakub Wartak <jakub.wartak@enterprisedb.com>; Jelte Fennema-Nio <postgres@jeltef.nl>; Antonin Houska <ah@cybertec.at>
Hi,
On 2025-03-28 08:54:42 -0400, Andres Freund wrote:
> On 2025-03-27 20:22:23 -0700, Noah Misch wrote:
> > On Thu, Mar 27, 2025 at 04:58:11PM -0400, Andres Freund wrote:
> > > - We need to register a local callback for shared buffer reads, which don't
> > > need them today . That's a small bit of added overhead. It's a shame to do
> > > so for counters that approximately never get incremented.
> >
> > Fair concern. An idea is to let the complete_shared callback change the
> > callback list associated with the IO, so it could change
> > PGAIO_HCB_SHARED_BUFFER_READV to PGAIO_HCB_SHARED_BUFFER_READV_SLOW. The
> > latter would differ from the former only in having the extra local callback.
> > Could that help? I think the only overhead is using more PGAIO_HCB numbers.
>
> I think changing the callback could work - I'll do some measurements in a
> coffee or two, but I suspect the overhead is not worth being too worried about
> for now. There's a different aspect that worries me slightly more, see
> further down.
> ...
> Unfortunately pgstat_prepare_report_checksum_failure() has to do a lookup in a
> local hashtable. That's more expensive than an indirect function call
> (i.e. the added local callback). I hope^Wsuspect it'll still be fine, and if
> not we can apply a mini-cache for the current database, which is surely the
> only thing that ever matters for performance.
I tried it and at ~30GB/s of read IO, with checksums disabled, I can't see a
difference of either having the unnecessary complete_local callback or having
the lookup in pgstat_prepare_report_checksum_failure(). In a profile there are
a few hits inside pgstat_get_entry_ref(), but not enough to matter.
Hence I think this isn't worth worrying about, at least for now. I think we
have far bigger fish to fry at this point than such a small performance
difference.
I've adjusted the comment above TRACE_POSTGRESQL_BUFFER_READ_DONE() to not
mention the overhead. I'm still inclined to think that it's better to call it
in the shared completion callback.
I also fixed support and added tests for ignore_checksum_failure, that also
needs to be determined at the start of the IO, not in the completion. Once
more there were no tests, of course.
I spent the last 6 hours on the stupid error/warning messages around this,
somewhat ridiculous.
The number of combinations is annoyingly large. It's e.g. plausible to use
ignore_checksum_failure=on and zero_damaged_pages=on at the same time for
recovery. The same buffer could both be ignored *and* zeroed. Or somebody
could use ignore_checksum_failure=on but then still encounter a page that is
invalid.
But I finally got to a point where the code ends up readable, without undue
duplication. It would, leaving some nasty hack aside, require a
errhint_internal() - but I can't imagine a reason against introducing that,
given we have it for the errmsg and errhint.
Here's the relevant code:
/*
* Treat a read that had both zeroed buffers *and* ignored checksums as a
* special case, it's too irregular to be emitted the same way as the other
* cases.
*/
if (zeroed_any && ignored_any)
{
Assert(zeroed_any && ignored_any);
Assert(nblocks > 1); /* same block can't be both zeroed and ignored */
Assert(result.status != PGAIO_RS_ERROR);
affected_count = zeroed_or_error_count;
ereport(elevel,
errcode(ERRCODE_DATA_CORRUPTED),
errmsg("zeroing %u pages and ignoring %u checksum failures among blocks %u..%u of relation %s",
affected_count, checkfail_count, first, last, rpath.str),
affected_count > 1 ?
errdetail("Block %u held first zeroed page.",
first + first_off) : 0,
errhint("See server log for details about the other %u invalid blocks.",
affected_count + checkfail_count - 1));
return;
}
/*
* The other messages are highly repetitive. To avoid duplicating a long
* and complicated ereport(), gather the translated format strings
* separately and then do one common ereport.
*/
if (result.status == PGAIO_RS_ERROR)
{
Assert(!zeroed_any); /* can't have invalid pages when zeroing them */
affected_count = zeroed_or_error_count;
msg_one = _("invalid page in block %u of relation %s");
msg_mult = _("%u invalid pages among blocks %u..%u of relation %s");
det_mult = _("Block %u held first invalid page.");
hint_mult = _("See server log for the other %u invalid blocks.");
}
else if (zeroed_any && !ignored_any)
{
affected_count = zeroed_or_error_count;
msg_one = _("invalid page in block %u of relation %s; zeroing out page");
msg_mult = _("zeroing out %u invalid pages among blocks %u..%u of relation %s");
det_mult = _("Block %u held first zeroed page.");
hint_mult = _("See server log for the other %u zeroed blocks.");
}
else if (!zeroed_any && ignored_any)
{
affected_count = checkfail_count;
msg_one = _("ignoring checksum failure in block %u of relation %s");
msg_mult = _("ignoring %u checksum failures among blocks %u..%u of relation %s");
det_mult = _("Block %u held first ignored page.");
hint_mult = _("See server log for the other %u ignored blocks.");
}
else
pg_unreachable();
ereport(elevel,
errcode(ERRCODE_DATA_CORRUPTED),
affected_count == 1 ?
errmsg_internal(msg_one, first + first_off, rpath.str) :
errmsg_internal(msg_mult, affected_count, first, last, rpath.str),
affected_count > 1 ? errdetail_internal(det_mult, first + first_off) : 0,
affected_count > 1 ? errhint_internal(hint_mult, affected_count - 1) : 0);
Does that approach make sense?
What do you think about using
"zeroing invalid page in block %u of relation %s"
instead of
"invalid page in block %u of relation %s; zeroing out page"
I thought about instead translating "ignoring", "ignored", "zeroing",
"zeroed", etc separately, but I have doubts about how well that would actually
translate.
Greetings,
Andres Freund
^ permalink raw reply [nested|flat] 153+ messages in thread
* Re: AIO v2.5
2025-03-04 19:00 AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-10 18:23 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-11 19:41 ` Re: AIO v2.5 Noah Misch <noah@leadboat.com>
2025-03-11 23:55 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-12 03:57 ` Re: AIO v2.5 Noah Misch <noah@leadboat.com>
2025-03-12 17:06 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-12 23:23 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-13 10:53 ` Re: AIO v2.5 Antonin Houska <ah@cybertec.at>
2025-03-14 19:43 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-17 20:00 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-18 20:12 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-21 01:58 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-25 01:18 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-27 01:07 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-27 20:58 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-28 03:22 ` Re: AIO v2.5 Noah Misch <noah@leadboat.com>
2025-03-28 12:54 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-29 03:35 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
@ 2025-03-29 13:41 ` Noah Misch <noah@leadboat.com>
2025-03-29 14:48 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
0 siblings, 1 reply; 153+ messages in thread
From: Noah Misch @ 2025-03-29 13:41 UTC (permalink / raw)
To: Andres Freund <andres@anarazel.de>; +Cc: pgsql-hackers@postgresql.org, Thomas Munro <thomas.munro@gmail.com>; Heikki Linnakangas <hlinnaka@iki.fi>; Robert Haas <robertmhaas@gmail.com>; Jakub Wartak <jakub.wartak@enterprisedb.com>; Jelte Fennema-Nio <postgres@jeltef.nl>; Antonin Houska <ah@cybertec.at>
On Fri, Mar 28, 2025 at 11:35:23PM -0400, Andres Freund wrote:
> The number of combinations is annoyingly large. It's e.g. plausible to use
> ignore_checksum_failure=on and zero_damaged_pages=on at the same time for
> recovery.
That's intricate indeed.
> But I finally got to a point where the code ends up readable, without undue
> duplication. It would, leaving some nasty hack aside, require a
> errhint_internal() - but I can't imagine a reason against introducing that,
> given we have it for the errmsg and errhint.
Introducing that is fine.
> Here's the relevant code:
>
> /*
> * Treat a read that had both zeroed buffers *and* ignored checksums as a
> * special case, it's too irregular to be emitted the same way as the other
> * cases.
> */
> if (zeroed_any && ignored_any)
> {
> Assert(zeroed_any && ignored_any);
> Assert(nblocks > 1); /* same block can't be both zeroed and ignored */
> Assert(result.status != PGAIO_RS_ERROR);
> affected_count = zeroed_or_error_count;
>
> ereport(elevel,
> errcode(ERRCODE_DATA_CORRUPTED),
> errmsg("zeroing %u pages and ignoring %u checksum failures among blocks %u..%u of relation %s",
> affected_count, checkfail_count, first, last, rpath.str),
Translation stumbles on this one, because each of the first two %u are
plural-sensitive. I'd do one of:
- Call ereport() twice, once for zeroed pages and once for ignored checksums.
Since elevel <= ERROR here, that doesn't lose the second call.
- s/pages/page(s)/ like msgid "There are %d other session(s) and %d prepared
transaction(s) using the database."
- Something more like the style of VACUUM VERBOSE, e.g. "INTRO_TEXT: %u
zeroed, %u checksums ignored". I've not written INTRO_TEXT, and this
doesn't really resolve pluralization. Probably don't use this option.
> affected_count > 1 ?
> errdetail("Block %u held first zeroed page.",
> first + first_off) : 0,
> errhint("See server log for details about the other %u invalid blocks.",
> affected_count + checkfail_count - 1));
> return;
> }
>
> /*
> * The other messages are highly repetitive. To avoid duplicating a long
> * and complicated ereport(), gather the translated format strings
> * separately and then do one common ereport.
> */
> if (result.status == PGAIO_RS_ERROR)
> {
> Assert(!zeroed_any); /* can't have invalid pages when zeroing them */
> affected_count = zeroed_or_error_count;
> msg_one = _("invalid page in block %u of relation %s");
> msg_mult = _("%u invalid pages among blocks %u..%u of relation %s");
> det_mult = _("Block %u held first invalid page.");
> hint_mult = _("See server log for the other %u invalid blocks.");
For each hint_mult, we would usually use ngettext() instead of _(). (Would be
errhint_plural() if not separated from its ereport().) Alternatively,
s/blocks/block(s)/ is fine.
> }
> else if (zeroed_any && !ignored_any)
> {
> affected_count = zeroed_or_error_count;
> msg_one = _("invalid page in block %u of relation %s; zeroing out page");
> msg_mult = _("zeroing out %u invalid pages among blocks %u..%u of relation %s");
> det_mult = _("Block %u held first zeroed page.");
> hint_mult = _("See server log for the other %u zeroed blocks.");
> }
> else if (!zeroed_any && ignored_any)
> {
> affected_count = checkfail_count;
> msg_one = _("ignoring checksum failure in block %u of relation %s");
> msg_mult = _("ignoring %u checksum failures among blocks %u..%u of relation %s");
> det_mult = _("Block %u held first ignored page.");
> hint_mult = _("See server log for the other %u ignored blocks.");
> }
> else
> pg_unreachable();
>
> ereport(elevel,
> errcode(ERRCODE_DATA_CORRUPTED),
> affected_count == 1 ?
> errmsg_internal(msg_one, first + first_off, rpath.str) :
> errmsg_internal(msg_mult, affected_count, first, last, rpath.str),
> affected_count > 1 ? errdetail_internal(det_mult, first + first_off) : 0,
> affected_count > 1 ? errhint_internal(hint_mult, affected_count - 1) : 0);
>
> Does that approach make sense?
Yes.
> What do you think about using
> "zeroing invalid page in block %u of relation %s"
> instead of
> "invalid page in block %u of relation %s; zeroing out page"
I like the replacement. It moves the important part to the front, and it's
shorter.
> I thought about instead translating "ignoring", "ignored", "zeroing",
> "zeroed", etc separately, but I have doubts about how well that would actually
> translate.
Agreed, I wouldn't have high hopes for that. An approach like that would
probably need messages that separate the independently-translated part
grammatically, e.g.:
/* last %s is translation of "ignore" or "zero-fill" */
"invalid page in block %u of relation %s; resolved by method \"%s\""
(Again, I'm not recommending that.)
^ permalink raw reply [nested|flat] 153+ messages in thread
* Re: AIO v2.5
2025-03-04 19:00 AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-10 18:23 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-11 19:41 ` Re: AIO v2.5 Noah Misch <noah@leadboat.com>
2025-03-11 23:55 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-12 03:57 ` Re: AIO v2.5 Noah Misch <noah@leadboat.com>
2025-03-12 17:06 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-12 23:23 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-13 10:53 ` Re: AIO v2.5 Antonin Houska <ah@cybertec.at>
2025-03-14 19:43 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-17 20:00 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-18 20:12 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-21 01:58 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-25 01:18 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-27 01:07 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-27 20:58 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-28 03:22 ` Re: AIO v2.5 Noah Misch <noah@leadboat.com>
2025-03-28 12:54 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-29 03:35 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-29 13:41 ` Re: AIO v2.5 Noah Misch <noah@leadboat.com>
@ 2025-03-29 14:48 ` Andres Freund <andres@anarazel.de>
2025-03-29 18:25 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
0 siblings, 1 reply; 153+ messages in thread
From: Andres Freund @ 2025-03-29 14:48 UTC (permalink / raw)
To: Noah Misch <noah@leadboat.com>; +Cc: pgsql-hackers@postgresql.org, Thomas Munro <thomas.munro@gmail.com>; Heikki Linnakangas <hlinnaka@iki.fi>; Robert Haas <robertmhaas@gmail.com>; Jakub Wartak <jakub.wartak@enterprisedb.com>; Jelte Fennema-Nio <postgres@jeltef.nl>; Antonin Houska <ah@cybertec.at>
Hi,
On 2025-03-29 06:41:43 -0700, Noah Misch wrote:
> On Fri, Mar 28, 2025 at 11:35:23PM -0400, Andres Freund wrote:
> > But I finally got to a point where the code ends up readable, without undue
> > duplication. It would, leaving some nasty hack aside, require a
> > errhint_internal() - but I can't imagine a reason against introducing that,
> > given we have it for the errmsg and errhint.
>
> Introducing that is fine.
Cool.
> > Here's the relevant code:
> >
> > /*
> > * Treat a read that had both zeroed buffers *and* ignored checksums as a
> > * special case, it's too irregular to be emitted the same way as the other
> > * cases.
> > */
> > if (zeroed_any && ignored_any)
> > {
> > Assert(zeroed_any && ignored_any);
> > Assert(nblocks > 1); /* same block can't be both zeroed and ignored */
> > Assert(result.status != PGAIO_RS_ERROR);
> > affected_count = zeroed_or_error_count;
> >
> > ereport(elevel,
> > errcode(ERRCODE_DATA_CORRUPTED),
> > errmsg("zeroing %u pages and ignoring %u checksum failures among blocks %u..%u of relation %s",
> > affected_count, checkfail_count, first, last, rpath.str),
>
> Translation stumbles on this one, because each of the first two %u are
> plural-sensitive.
Fair. We don't generally seem to have been very careful around this in relate
code, but there's no reason to just continue down that road when it's easy.
E.g. in md.c we unconditionally output "could not read blocks %u..%u in file \"%s\": %m"
even if it's just a single block...
> I'd do one of:
>
> - Call ereport() twice, once for zeroed pages and once for ignored checksums.
> Since elevel <= ERROR here, that doesn't lose the second call.
>
> - s/pages/page(s)/ like msgid "There are %d other session(s) and %d prepared
> transaction(s) using the database."
I think I like this better.
> > /*
> > * The other messages are highly repetitive. To avoid duplicating a long
> > * and complicated ereport(), gather the translated format strings
> > * separately and then do one common ereport.
> > */
> > if (result.status == PGAIO_RS_ERROR)
> > {
> > Assert(!zeroed_any); /* can't have invalid pages when zeroing them */
> > affected_count = zeroed_or_error_count;
> > msg_one = _("invalid page in block %u of relation %s");
> > msg_mult = _("%u invalid pages among blocks %u..%u of relation %s");
> > det_mult = _("Block %u held first invalid page.");
> > hint_mult = _("See server log for the other %u invalid blocks.");
>
> For each hint_mult, we would usually use ngettext() instead of _(). (Would be
> errhint_plural() if not separated from its ereport().) Alternatively,
> s/blocks/block(s)/ is fine.
I will go with the (s) here as well, this stuff is too rare to be worth having
pluralized messages imo.
> > Does that approach make sense?
>
> Yes.
> ...
> > I like the replacement. It moves the important part to the front, and it's
> shorter.
Cool, I squashed them with the relevant changes now.
Attached is v2.14:
Changes:
- Added a commit to fix stats attribution of checksum errors, previously the
checksum errors detected bufmgr.c/storage.c were always attributed to the
current database
This would have caused bigger issues with worker based IO, as IO workers
aren't connected to databases.
- Added a commit to allow checksum error reports to happen in critical
sections. For that a pgstat_prepare_report_checksum_failure() has to be
called in the same backend, to report the critical section.
Other suggestions for the name welcome.
- Expanded on the idea in 13 to track the number of invalid buffers in the
IO's result, by also tracking checksum errors. Combined with the previous
point, this fixes the issue of an assert during checksum failure reporting
outlined in:
https://postgr.es/m/5tyic6epvdlmd6eddgelv47syg2b5cpwffjam54axp25xyq2ga%40ptwkinxqo3az
This required being a bit more careful with space in the error, to be able
to squeeze in the checksum number.
- The ignore_checksum_failure of the issuer needs to be used when completing
IO, not the one of the completor, particularly when using io_method=worker
For that the access to ignore_checksum_failure had to be moved from
PageIsVerified() to its callers.
I added tests for ignore_checksum_failure, including its interplay with
zero_damaged_pages.
- Deduplicated the error reporting in buffer_readv_report() somewhat by only
having the selection of format strings be done in branches. I think this
ends up a lot more readable than the huge ereport before.
- Added details about the changed error/warning logging to "bufmgr: Implement
AIO read support"'s commit message.
- polished the commit to add PGAIO_RS_WARNING a bit, adding defines for the
bit-widths of PgAioResult portions and added static asserts to verify them
- Squashed the changes that I had kept separately in v2.13, it was too hard to
do that while doing the above changes.
I did make the encoding function cast the arguments to uint32 before
shifting. I think that's implied by the C integer promotion rules, but it
seemed fishy enough to not want to believe in that.
I also added a StaticAssertStmt() to ensure we are only using the available
bit space.
- Added a test for a) checksum errors being detected b) CREATE DATABASE
... STRATEGY WAL_LOG
The latter is interesting because it also provides test coverage for doing
IO for objects in other databases.
- Removed an obsoleted inclusion of pg_trace.h in localbuf.c
TODO:
- I think the tests around zero_damaged_pages, ignore_checksum_failure should
be expanded a bit more. There's two FIXME in the tests about that.
At the moment there are two different test functions for zero_damaged_pages
and ignore_checksum_failure, I'm not sure how good that is.
I wanted to get this version out, because I have to run some errands,
otherwise I'd have implemente them first...
Next steps:
- push the checksums stats fix
- unless somebody sees a reason to not use LOG_SERVER_ONLY in
"aio: Implement support for reads in smgr/md/fd", push that
Besides that the only change since Noah's last review of that commit is an
added comment.
- push acronym, glossary change
- push pg_aios view (depends a tiny bit on the smgr/md/fd change above)
- push "localbuf: Track pincount in BufferDesc as well" - I think I addressed
all of Noah's review feedback
- address the above TODO
Greetings,
Andres Freund
Attachments:
[text/x-diff] v2.14-0001-Fix-mis-attribution-of-checksum-failure-stats-.patch (10.5K, ../../d4ff3qxudxn2ze3zk2xwhds5ysezfei3uqeuimymj624t5chsd@sgr2h3rmln5t/2-v2.14-0001-Fix-mis-attribution-of-checksum-failure-stats-.patch)
download | inline diff:
From 96cd8d51ab871e730e646db74631a97fd9bd2f7d Mon Sep 17 00:00:00 2001
From: Andres Freund <andres@anarazel.de>
Date: Fri, 28 Mar 2025 12:29:40 -0400
Subject: [PATCH v2.14 01/29] Fix mis-attribution of checksum failure stats to
the wrong database
Checksum failure stats could be attributed to the wrong database in two cases:
- when a read of a shared relation encountered a checksum error , it would be
attributed to the current database, instead of the "database" representing
shared relations
- when using CREATE DATABASE ... STRATEGY WAL_LOG checksum errors in the
source database would be attributed to the current database
The checksum stats reporting via PageIsVerifiedExtended(PIV_REPORT_STAT) does
not have access to the information about what database a page belongs to.
This fixes the issue by removing PIV_REPORT_STAT and delegating the
responsibility to report stats to the caller, which now can learn about the
number of stats via a new optional argument.
As this changes the signature of PageIsVerifiedExtended() and all callers
should adapt to the new signature, use the occasion to rename the function to
PageIsVerified() and remove the compatibility macro.
We could instead have fixed this by adding information about the database to
the args of PageIsVerified(), but there are soon-to-be-applied patches that
need to separate the stats reporting from the PageIsVerified() call
anyway. Those patches also include testing for the failure paths, something we
inexplicably have not had.
As there is no caller of pgstat_report_checksum_failure() left, remove it.
It'd be possible, but awkward to fix this in the back branches. We considered
doing the work not quite worth it, as mis-attributed stats should still elicit
concern. The emitted error messages do allow to attribute the errors
correctly.
Discussion: https://postgr.es/m/5tyic6epvdlmd6eddgelv47syg2b5cpwffjam54axp25xyq2ga@ptwkinxqo3az
Discussion: https://postgr.es/m/mglpvvbhighzuwudjxzu4br65qqcxsnyvio3nl4fbog3qknwhg@e4gt7npsohuz
---
src/include/pgstat.h | 1 -
src/include/storage/bufpage.h | 20 ++++++++------------
src/backend/catalog/storage.c | 16 ++++++++++++++--
src/backend/storage/buffer/bufmgr.c | 16 +++++++++++++---
src/backend/storage/page/bufpage.c | 20 +++++++++++++-------
src/backend/utils/activity/pgstat_database.c | 9 ---------
6 files changed, 48 insertions(+), 34 deletions(-)
diff --git a/src/include/pgstat.h b/src/include/pgstat.h
index 5bfe19e66be..9f3d13bf1ce 100644
--- a/src/include/pgstat.h
+++ b/src/include/pgstat.h
@@ -612,7 +612,6 @@ extern void pgstat_report_autovac(Oid dboid);
extern void pgstat_report_recovery_conflict(int reason);
extern void pgstat_report_deadlock(void);
extern void pgstat_report_checksum_failures_in_db(Oid dboid, int failurecount);
-extern void pgstat_report_checksum_failure(void);
extern void pgstat_report_connect(Oid dboid);
extern void pgstat_update_parallel_workers_stats(PgStat_Counter workers_to_launch,
PgStat_Counter workers_launched);
diff --git a/src/include/storage/bufpage.h b/src/include/storage/bufpage.h
index 6646b6f6371..b943db707db 100644
--- a/src/include/storage/bufpage.h
+++ b/src/include/storage/bufpage.h
@@ -465,31 +465,27 @@ do { \
#define PAI_OVERWRITE (1 << 0)
#define PAI_IS_HEAP (1 << 1)
-/* flags for PageIsVerifiedExtended() */
+/* flags for PageIsVerified() */
#define PIV_LOG_WARNING (1 << 0)
-#define PIV_REPORT_STAT (1 << 1)
#define PageAddItem(page, item, size, offsetNumber, overwrite, is_heap) \
PageAddItemExtended(page, item, size, offsetNumber, \
((overwrite) ? PAI_OVERWRITE : 0) | \
((is_heap) ? PAI_IS_HEAP : 0))
-#define PageIsVerified(page, blkno) \
- PageIsVerifiedExtended(page, blkno, \
- PIV_LOG_WARNING | PIV_REPORT_STAT)
-
/*
- * Check that BLCKSZ is a multiple of sizeof(size_t). In
- * PageIsVerifiedExtended(), it is much faster to check if a page is
- * full of zeroes using the native word size. Note that this assertion
- * is kept within a header to make sure that StaticAssertDecl() works
- * across various combinations of platforms and compilers.
+ * Check that BLCKSZ is a multiple of sizeof(size_t). In PageIsVerified(), it
+ * is much faster to check if a page is full of zeroes using the native word
+ * size. Note that this assertion is kept within a header to make sure that
+ * StaticAssertDecl() works across various combinations of platforms and
+ * compilers.
*/
StaticAssertDecl(BLCKSZ == ((BLCKSZ / sizeof(size_t)) * sizeof(size_t)),
"BLCKSZ has to be a multiple of sizeof(size_t)");
extern void PageInit(Page page, Size pageSize, Size specialSize);
-extern bool PageIsVerifiedExtended(PageData *page, BlockNumber blkno, int flags);
+extern bool PageIsVerified(PageData *page, BlockNumber blkno, int flags,
+ bool *checksum_failure_p);
extern OffsetNumber PageAddItemExtended(Page page, Item item, Size size,
OffsetNumber offsetNumber, int flags);
extern Page PageGetTempPage(const PageData *page);
diff --git a/src/backend/catalog/storage.c b/src/backend/catalog/storage.c
index 624ed41bbf3..cacf16c1cdb 100644
--- a/src/backend/catalog/storage.c
+++ b/src/backend/catalog/storage.c
@@ -27,6 +27,7 @@
#include "catalog/storage.h"
#include "catalog/storage_xlog.h"
#include "miscadmin.h"
+#include "pgstat.h"
#include "storage/bulk_write.h"
#include "storage/freespace.h"
#include "storage/proc.h"
@@ -507,6 +508,8 @@ RelationCopyStorage(SMgrRelation src, SMgrRelation dst,
for (blkno = 0; blkno < nblocks; blkno++)
{
BulkWriteBuffer buf;
+ bool checksum_failure;
+ bool verified;
/* If we got a cancel signal during the copy of the data, quit */
CHECK_FOR_INTERRUPTS();
@@ -514,8 +517,17 @@ RelationCopyStorage(SMgrRelation src, SMgrRelation dst,
buf = smgr_bulk_get_buf(bulkstate);
smgrread(src, forkNum, blkno, (Page) buf);
- if (!PageIsVerifiedExtended((Page) buf, blkno,
- PIV_LOG_WARNING | PIV_REPORT_STAT))
+ verified = PageIsVerified((Page) buf, blkno, PIV_LOG_WARNING,
+ &checksum_failure);
+
+ if (checksum_failure)
+ {
+ RelFileLocatorBackend rloc = src->smgr_rlocator;
+
+ pgstat_report_checksum_failures_in_db(rloc.locator.dbOid, 1);
+ }
+
+ if (!verified)
{
/*
* For paranoia's sake, capture the file path before invoking the
diff --git a/src/backend/storage/buffer/bufmgr.c b/src/backend/storage/buffer/bufmgr.c
index 323382dcfa8..5cac8cd7389 100644
--- a/src/backend/storage/buffer/bufmgr.c
+++ b/src/backend/storage/buffer/bufmgr.c
@@ -770,7 +770,7 @@ ReadBuffer(Relation reln, BlockNumber blockNum)
* In RBM_NORMAL mode, the page is read from disk, and the page header is
* validated. An error is thrown if the page header is not valid. (But
* note that an all-zero page is considered "valid"; see
- * PageIsVerifiedExtended().)
+ * PageIsVerified().)
*
* RBM_ZERO_ON_ERROR is like the normal mode, but if the page header is not
* valid, the page is zeroed instead of throwing an error. This is intended
@@ -1569,6 +1569,8 @@ WaitReadBuffers(ReadBuffersOperation *operation)
{
BufferDesc *bufHdr;
Block bufBlock;
+ bool verified;
+ bool checksum_failure;
if (persistence == RELPERSISTENCE_TEMP)
{
@@ -1582,8 +1584,16 @@ WaitReadBuffers(ReadBuffersOperation *operation)
}
/* check for garbage data */
- if (!PageIsVerifiedExtended((Page) bufBlock, io_first_block + j,
- PIV_LOG_WARNING | PIV_REPORT_STAT))
+ verified = PageIsVerified((Page) bufBlock, io_first_block + j,
+ PIV_LOG_WARNING, &checksum_failure);
+ if (checksum_failure)
+ {
+ RelFileLocatorBackend rloc = operation->smgr->smgr_rlocator;
+
+ pgstat_report_checksum_failures_in_db(rloc.locator.dbOid, 1);
+ }
+
+ if (!verified)
{
if ((operation->flags & READ_BUFFERS_ZERO_ON_ERROR) || zero_damaged_pages)
{
diff --git a/src/backend/storage/page/bufpage.c b/src/backend/storage/page/bufpage.c
index ecc81aacfc3..5d1b039fcbb 100644
--- a/src/backend/storage/page/bufpage.c
+++ b/src/backend/storage/page/bufpage.c
@@ -61,7 +61,7 @@ PageInit(Page page, Size pageSize, Size specialSize)
/*
- * PageIsVerifiedExtended
+ * PageIsVerified
* Check that the page header and checksum (if any) appear valid.
*
* This is called when a page has just been read in from disk. The idea is
@@ -81,11 +81,13 @@ PageInit(Page page, Size pageSize, Size specialSize)
* If flag PIV_LOG_WARNING is set, a WARNING is logged in the event of
* a checksum failure.
*
- * If flag PIV_REPORT_STAT is set, a checksum failure is reported directly
- * to pgstat.
+ * To allow the caller to report statistics about checksum failures,
+ * *checksum_failure_p can be passed in. Note that there may be checksum
+ * failures even if this function returns true, due to
+ * ignore_checksum_failure.
*/
bool
-PageIsVerifiedExtended(PageData *page, BlockNumber blkno, int flags)
+PageIsVerified(PageData *page, BlockNumber blkno, int flags, bool *checksum_failure_p)
{
const PageHeaderData *p = (const PageHeaderData *) page;
size_t *pagebytes;
@@ -93,6 +95,9 @@ PageIsVerifiedExtended(PageData *page, BlockNumber blkno, int flags)
bool header_sane = false;
uint16 checksum = 0;
+ if (checksum_failure_p)
+ *checksum_failure_p = false;
+
/*
* Don't verify page data unless the page passes basic non-zero test
*/
@@ -103,7 +108,11 @@ PageIsVerifiedExtended(PageData *page, BlockNumber blkno, int flags)
checksum = pg_checksum_page(page, blkno);
if (checksum != p->pd_checksum)
+ {
checksum_failure = true;
+ if (checksum_failure_p)
+ *checksum_failure_p = true;
+ }
}
/*
@@ -141,9 +150,6 @@ PageIsVerifiedExtended(PageData *page, BlockNumber blkno, int flags)
errmsg("page verification failed, calculated checksum %u but expected %u",
checksum, p->pd_checksum)));
- if ((flags & PIV_REPORT_STAT) != 0)
- pgstat_report_checksum_failure();
-
if (header_sane && ignore_checksum_failure)
return true;
}
diff --git a/src/backend/utils/activity/pgstat_database.c b/src/backend/utils/activity/pgstat_database.c
index 05a8ccfdb75..c0c02efd9d3 100644
--- a/src/backend/utils/activity/pgstat_database.c
+++ b/src/backend/utils/activity/pgstat_database.c
@@ -159,15 +159,6 @@ pgstat_report_checksum_failures_in_db(Oid dboid, int failurecount)
pgstat_unlock_entry(entry_ref);
}
-/*
- * Report one checksum failure in the current database.
- */
-void
-pgstat_report_checksum_failure(void)
-{
- pgstat_report_checksum_failures_in_db(MyDatabaseId, 1);
-}
-
/*
* Report creation of temporary file.
*/
--
2.48.1.76.g4e746b1a31.dirty
[text/x-diff] v2.14-0002-aio-Implement-support-for-reads-in-smgr-md-fd.patch (23.6K, ../../d4ff3qxudxn2ze3zk2xwhds5ysezfei3uqeuimymj624t5chsd@sgr2h3rmln5t/3-v2.14-0002-aio-Implement-support-for-reads-in-smgr-md-fd.patch)
download | inline diff:
From 25bb63b1ec9ee08ce7585de846a147114bc34f17 Mon Sep 17 00:00:00 2001
From: Andres Freund <andres@anarazel.de>
Date: Tue, 18 Mar 2025 14:40:05 -0400
Subject: [PATCH v2.14 02/29] aio: Implement support for reads in smgr/md/fd
This implements the following:
1) An smgr AIO target, for AIO on smgr files. This should be usable not just
for md.c but also other SMGR implementation if we ever get them.
2) readv support in fd.c, which requires a small bit of infrastructure work in
fd.c
3) smgr.c and md.c support for readv
There still is nothing performing AIO, but as of this commit it would be
possible.
As part of this change FileGetRawDesc() actually ensures that the file is
opened - previously it was basically not usable. It's used to reopen a file in
IO workers.
Reviewed-by: Noah Misch <noah@leadboat.com>
Discussion: https://postgr.es/m/uvrtrknj4kdytuboidbhwclo4gxhswwcpgadptsjvjqcluzmah%40brqs62irg4dt
Discussion: https://postgr.es/m/20210223100344.llw5an2aklengrmn@alap3.anarazel.de
Discussion: https://postgr.es/m/stj36ea6yyhoxtqkhpieia2z4krnam7qyetc57rfezgk4zgapf@gcnactj4z56m
---
src/include/storage/aio.h | 6 +-
src/include/storage/aio_types.h | 10 +-
src/include/storage/fd.h | 3 +
src/include/storage/md.h | 7 +
src/include/storage/smgr.h | 14 ++
src/backend/storage/aio/aio_callback.c | 3 +
src/backend/storage/aio/aio_target.c | 2 +
src/backend/storage/file/fd.c | 40 +++++
src/backend/storage/smgr/md.c | 196 +++++++++++++++++++++++++
src/backend/storage/smgr/smgr.c | 160 ++++++++++++++++++++
10 files changed, 437 insertions(+), 4 deletions(-)
diff --git a/src/include/storage/aio.h b/src/include/storage/aio.h
index cc987556e14..c386fc96d90 100644
--- a/src/include/storage/aio.h
+++ b/src/include/storage/aio.h
@@ -117,9 +117,10 @@ typedef enum PgAioTargetID
{
/* intentionally the zero value, to help catch zeroed memory etc */
PGAIO_TID_INVALID = 0,
+ PGAIO_TID_SMGR,
} PgAioTargetID;
-#define PGAIO_TID_COUNT (PGAIO_TID_INVALID + 1)
+#define PGAIO_TID_COUNT (PGAIO_TID_SMGR + 1)
/*
@@ -191,6 +192,9 @@ struct PgAioTargetInfo
typedef enum PgAioHandleCallbackID
{
PGAIO_HCB_INVALID,
+
+ PGAIO_HCB_MD_READV,
+ PGAIO_HCB_MD_WRITEV,
} PgAioHandleCallbackID;
diff --git a/src/include/storage/aio_types.h b/src/include/storage/aio_types.h
index dddda3a3e2f..debe8163d4e 100644
--- a/src/include/storage/aio_types.h
+++ b/src/include/storage/aio_types.h
@@ -60,11 +60,15 @@ typedef struct PgAioWaitRef
*/
typedef union PgAioTargetData
{
- /* just as an example placeholder for later */
struct
{
- uint32 queue_id;
- } wal;
+ RelFileLocator rlocator; /* physical relation identifier */
+ BlockNumber blockNum; /* blknum relative to begin of reln */
+ BlockNumber nblocks;
+ ForkNumber forkNum:8; /* don't waste 4 byte for four values */
+ bool is_temp:1; /* proc can be inferred by owning AIO */
+ bool skip_fsync:1;
+ } smgr;
} PgAioTargetData;
diff --git a/src/include/storage/fd.h b/src/include/storage/fd.h
index e3067ab6597..b77d8e5e30e 100644
--- a/src/include/storage/fd.h
+++ b/src/include/storage/fd.h
@@ -101,6 +101,8 @@ extern PGDLLIMPORT int max_safe_fds;
* prototypes for functions in fd.c
*/
+struct PgAioHandle;
+
/* Operations on virtual Files --- equivalent to Unix kernel file ops */
extern File PathNameOpenFile(const char *fileName, int fileFlags);
extern File PathNameOpenFilePerm(const char *fileName, int fileFlags, mode_t fileMode);
@@ -109,6 +111,7 @@ extern void FileClose(File file);
extern int FilePrefetch(File file, off_t offset, off_t amount, uint32 wait_event_info);
extern ssize_t FileReadV(File file, const struct iovec *iov, int iovcnt, off_t offset, uint32 wait_event_info);
extern ssize_t FileWriteV(File file, const struct iovec *iov, int iovcnt, off_t offset, uint32 wait_event_info);
+extern int FileStartReadV(struct PgAioHandle *ioh, File file, int iovcnt, off_t offset, uint32 wait_event_info);
extern int FileSync(File file, uint32 wait_event_info);
extern int FileZero(File file, off_t offset, off_t amount, uint32 wait_event_info);
extern int FileFallocate(File file, off_t offset, off_t amount, uint32 wait_event_info);
diff --git a/src/include/storage/md.h b/src/include/storage/md.h
index 05bf537066e..9d7131eff43 100644
--- a/src/include/storage/md.h
+++ b/src/include/storage/md.h
@@ -14,11 +14,14 @@
#ifndef MD_H
#define MD_H
+#include "storage/aio_types.h"
#include "storage/block.h"
#include "storage/relfilelocator.h"
#include "storage/smgr.h"
#include "storage/sync.h"
+extern const PgAioHandleCallbacks aio_md_readv_cb;
+
/* md storage manager functionality */
extern void mdinit(void);
extern void mdopen(SMgrRelation reln);
@@ -36,6 +39,9 @@ extern uint32 mdmaxcombine(SMgrRelation reln, ForkNumber forknum,
BlockNumber blocknum);
extern void mdreadv(SMgrRelation reln, ForkNumber forknum, BlockNumber blocknum,
void **buffers, BlockNumber nblocks);
+extern void mdstartreadv(PgAioHandle *ioh,
+ SMgrRelation reln, ForkNumber forknum, BlockNumber blocknum,
+ void **buffers, BlockNumber nblocks);
extern void mdwritev(SMgrRelation reln, ForkNumber forknum,
BlockNumber blocknum,
const void **buffers, BlockNumber nblocks, bool skipFsync);
@@ -46,6 +52,7 @@ extern void mdtruncate(SMgrRelation reln, ForkNumber forknum,
BlockNumber old_blocks, BlockNumber nblocks);
extern void mdimmedsync(SMgrRelation reln, ForkNumber forknum);
extern void mdregistersync(SMgrRelation reln, ForkNumber forknum);
+extern int mdfd(SMgrRelation reln, ForkNumber forknum, BlockNumber blocknum, uint32 *off);
extern void ForgetDatabaseSyncRequests(Oid dbid);
extern void DropRelationFiles(RelFileLocator *delrels, int ndelrels, bool isRedo);
diff --git a/src/include/storage/smgr.h b/src/include/storage/smgr.h
index 4016b206ad6..856ebcda350 100644
--- a/src/include/storage/smgr.h
+++ b/src/include/storage/smgr.h
@@ -15,6 +15,7 @@
#define SMGR_H
#include "lib/ilist.h"
+#include "storage/aio_types.h"
#include "storage/block.h"
#include "storage/relfilelocator.h"
@@ -73,6 +74,8 @@ typedef SMgrRelationData *SMgrRelation;
#define SmgrIsTemp(smgr) \
RelFileLocatorBackendIsTemp((smgr)->smgr_rlocator)
+extern const PgAioTargetInfo aio_smgr_target_info;
+
extern void smgrinit(void);
extern SMgrRelation smgropen(RelFileLocator rlocator, ProcNumber backend);
extern bool smgrexists(SMgrRelation reln, ForkNumber forknum);
@@ -97,6 +100,10 @@ extern uint32 smgrmaxcombine(SMgrRelation reln, ForkNumber forknum,
extern void smgrreadv(SMgrRelation reln, ForkNumber forknum,
BlockNumber blocknum,
void **buffers, BlockNumber nblocks);
+extern void smgrstartreadv(PgAioHandle *ioh,
+ SMgrRelation reln, ForkNumber forknum,
+ BlockNumber blocknum,
+ void **buffers, BlockNumber nblocks);
extern void smgrwritev(SMgrRelation reln, ForkNumber forknum,
BlockNumber blocknum,
const void **buffers, BlockNumber nblocks,
@@ -127,4 +134,11 @@ smgrwrite(SMgrRelation reln, ForkNumber forknum, BlockNumber blocknum,
smgrwritev(reln, forknum, blocknum, &buffer, 1, skipFsync);
}
+extern void pgaio_io_set_target_smgr(PgAioHandle *ioh,
+ SMgrRelationData *smgr,
+ ForkNumber forknum,
+ BlockNumber blocknum,
+ int nblocks,
+ bool skip_fsync);
+
#endif /* SMGR_H */
diff --git a/src/backend/storage/aio/aio_callback.c b/src/backend/storage/aio/aio_callback.c
index 413ea3247c2..53db6e194af 100644
--- a/src/backend/storage/aio/aio_callback.c
+++ b/src/backend/storage/aio/aio_callback.c
@@ -18,6 +18,7 @@
#include "miscadmin.h"
#include "storage/aio.h"
#include "storage/aio_internal.h"
+#include "storage/md.h"
/* just to have something to put into aio_handle_cbs */
@@ -37,6 +38,8 @@ typedef struct PgAioHandleCallbacksEntry
static const PgAioHandleCallbacksEntry aio_handle_cbs[] = {
#define CALLBACK_ENTRY(id, callback) [id] = {.cb = &callback, .name = #callback}
CALLBACK_ENTRY(PGAIO_HCB_INVALID, aio_invalid_cb),
+
+ CALLBACK_ENTRY(PGAIO_HCB_MD_READV, aio_md_readv_cb),
#undef CALLBACK_ENTRY
};
diff --git a/src/backend/storage/aio/aio_target.c b/src/backend/storage/aio/aio_target.c
index 536c7f91f5e..ac6c74f4ff2 100644
--- a/src/backend/storage/aio/aio_target.c
+++ b/src/backend/storage/aio/aio_target.c
@@ -16,6 +16,7 @@
#include "storage/aio.h"
#include "storage/aio_internal.h"
+#include "storage/smgr.h"
/*
@@ -25,6 +26,7 @@ static const PgAioTargetInfo *pgaio_target_info[] = {
[PGAIO_TID_INVALID] = &(PgAioTargetInfo) {
.name = "invalid",
},
+ [PGAIO_TID_SMGR] = &aio_smgr_target_info,
};
diff --git a/src/backend/storage/file/fd.c b/src/backend/storage/file/fd.c
index 84873f41488..0e8299dd556 100644
--- a/src/backend/storage/file/fd.c
+++ b/src/backend/storage/file/fd.c
@@ -94,6 +94,7 @@
#include "miscadmin.h"
#include "pgstat.h"
#include "postmaster/startup.h"
+#include "storage/aio.h"
#include "storage/fd.h"
#include "storage/ipc.h"
#include "utils/guc.h"
@@ -1296,6 +1297,8 @@ LruDelete(File file)
vfdP = &VfdCache[file];
+ pgaio_closing_fd(vfdP->fd);
+
/*
* Close the file. We aren't expecting this to fail; if it does, better
* to leak the FD than to mess up our internal state.
@@ -1989,6 +1992,8 @@ FileClose(File file)
if (!FileIsNotOpen(file))
{
+ pgaio_closing_fd(vfdP->fd);
+
/* close the file */
if (close(vfdP->fd) != 0)
{
@@ -2212,6 +2217,32 @@ retry:
return returnCode;
}
+int
+FileStartReadV(PgAioHandle *ioh, File file,
+ int iovcnt, off_t offset,
+ uint32 wait_event_info)
+{
+ int returnCode;
+ Vfd *vfdP;
+
+ Assert(FileIsValid(file));
+
+ DO_DB(elog(LOG, "FileStartReadV: %d (%s) " INT64_FORMAT " %d",
+ file, VfdCache[file].fileName,
+ (int64) offset,
+ iovcnt));
+
+ returnCode = FileAccess(file);
+ if (returnCode < 0)
+ return returnCode;
+
+ vfdP = &VfdCache[file];
+
+ pgaio_io_start_readv(ioh, vfdP->fd, iovcnt, offset);
+
+ return 0;
+}
+
ssize_t
FileWriteV(File file, const struct iovec *iov, int iovcnt, off_t offset,
uint32 wait_event_info)
@@ -2500,6 +2531,12 @@ FilePathName(File file)
int
FileGetRawDesc(File file)
{
+ int returnCode;
+
+ returnCode = FileAccess(file);
+ if (returnCode < 0)
+ return returnCode;
+
Assert(FileIsValid(file));
return VfdCache[file].fd;
}
@@ -2780,6 +2817,7 @@ FreeDesc(AllocateDesc *desc)
result = closedir(desc->desc.dir);
break;
case AllocateDescRawFD:
+ pgaio_closing_fd(desc->desc.fd);
result = close(desc->desc.fd);
break;
default:
@@ -2848,6 +2886,8 @@ CloseTransientFile(int fd)
/* Only get here if someone passes us a file not in allocatedDescs */
elog(WARNING, "fd passed to CloseTransientFile was not obtained from OpenTransientFile");
+ pgaio_closing_fd(fd);
+
return close(fd);
}
diff --git a/src/backend/storage/smgr/md.c b/src/backend/storage/smgr/md.c
index f3220f98dc4..eedba1e5794 100644
--- a/src/backend/storage/smgr/md.c
+++ b/src/backend/storage/smgr/md.c
@@ -31,6 +31,7 @@
#include "miscadmin.h"
#include "pg_trace.h"
#include "pgstat.h"
+#include "storage/aio.h"
#include "storage/bufmgr.h"
#include "storage/fd.h"
#include "storage/md.h"
@@ -152,6 +153,15 @@ static MdfdVec *_mdfd_getseg(SMgrRelation reln, ForkNumber forknum,
static BlockNumber _mdnblocks(SMgrRelation reln, ForkNumber forknum,
MdfdVec *seg);
+static PgAioResult md_readv_complete(PgAioHandle *ioh, PgAioResult prior_result, uint8 cb_data);
+static void md_readv_report(PgAioResult result, const PgAioTargetData *target_data, int elevel);
+
+const PgAioHandleCallbacks aio_md_readv_cb = {
+ .complete_shared = md_readv_complete,
+ .report = md_readv_report,
+};
+
+
static inline int
_mdfd_open_flags(void)
{
@@ -937,6 +947,69 @@ mdreadv(SMgrRelation reln, ForkNumber forknum, BlockNumber blocknum,
}
}
+/*
+ * mdstartreadv() -- Asynchronous version of mdreadv().
+ */
+void
+mdstartreadv(PgAioHandle *ioh,
+ SMgrRelation reln, ForkNumber forknum, BlockNumber blocknum,
+ void **buffers, BlockNumber nblocks)
+{
+ off_t seekpos;
+ MdfdVec *v;
+ BlockNumber nblocks_this_segment;
+ struct iovec *iov;
+ int iovcnt;
+ int ret;
+
+ v = _mdfd_getseg(reln, forknum, blocknum, false,
+ EXTENSION_FAIL | EXTENSION_CREATE_RECOVERY);
+
+ seekpos = (off_t) BLCKSZ * (blocknum % ((BlockNumber) RELSEG_SIZE));
+
+ Assert(seekpos < (off_t) BLCKSZ * RELSEG_SIZE);
+
+ nblocks_this_segment =
+ Min(nblocks,
+ RELSEG_SIZE - (blocknum % ((BlockNumber) RELSEG_SIZE)));
+
+ if (nblocks_this_segment != nblocks)
+ elog(ERROR, "read crossing segment boundary");
+
+ iovcnt = pgaio_io_get_iovec(ioh, &iov);
+
+ Assert(nblocks <= iovcnt);
+
+ iovcnt = buffers_to_iovec(iov, buffers, nblocks_this_segment);
+
+ Assert(iovcnt <= nblocks_this_segment);
+
+ if (!(io_direct_flags & IO_DIRECT_DATA))
+ pgaio_io_set_flag(ioh, PGAIO_HF_BUFFERED);
+
+ pgaio_io_set_target_smgr(ioh,
+ reln,
+ forknum,
+ blocknum,
+ nblocks,
+ false);
+ pgaio_io_register_callbacks(ioh, PGAIO_HCB_MD_READV, 0);
+
+ ret = FileStartReadV(ioh, v->mdfd_vfd, iovcnt, seekpos, WAIT_EVENT_DATA_FILE_READ);
+ if (ret != 0)
+ ereport(ERROR,
+ (errcode_for_file_access(),
+ errmsg("could not start reading blocks %u..%u in file \"%s\": %m",
+ blocknum,
+ blocknum + nblocks_this_segment - 1,
+ FilePathName(v->mdfd_vfd))));
+
+ /*
+ * The error checks corresponding to the post-read checks in mdread() are
+ * in md_readv_complete().
+ */
+}
+
/*
* mdwritev() -- Write the supplied blocks at the appropriate location.
*
@@ -1365,6 +1438,21 @@ mdimmedsync(SMgrRelation reln, ForkNumber forknum)
}
}
+int
+mdfd(SMgrRelation reln, ForkNumber forknum, BlockNumber blocknum, uint32 *off)
+{
+ MdfdVec *v = mdopenfork(reln, forknum, EXTENSION_FAIL);
+
+ v = _mdfd_getseg(reln, forknum, blocknum, false,
+ EXTENSION_FAIL);
+
+ *off = (off_t) BLCKSZ * (blocknum % ((BlockNumber) RELSEG_SIZE));
+
+ Assert(*off < (off_t) BLCKSZ * RELSEG_SIZE);
+
+ return FileGetRawDesc(v->mdfd_vfd);
+}
+
/*
* register_dirty_segment() -- Mark a relation segment as needing fsync
*
@@ -1841,3 +1929,111 @@ mdfiletagmatches(const FileTag *ftag, const FileTag *candidate)
*/
return ftag->rlocator.dbOid == candidate->rlocator.dbOid;
}
+
+/*
+ * AIO completion callback for mdstartreadv().
+ */
+static PgAioResult
+md_readv_complete(PgAioHandle *ioh, PgAioResult prior_result, uint8 cb_data)
+{
+ PgAioTargetData *td = pgaio_io_get_target_data(ioh);
+ PgAioResult result = prior_result;
+
+ if (prior_result.result < 0)
+ {
+ result.status = PGAIO_RS_ERROR;
+ result.id = PGAIO_HCB_MD_READV;
+ /* For "hard" errors, track the error number in error_data */
+ result.error_data = -prior_result.result;
+ result.result = 0;
+
+ /*
+ * Immediately log a message about the IO error, but only to the
+ * server log. The reason to do so immediately is that the originator
+ * might not process the query result immediately (because it is busy
+ * doing another part of query processing) or at all (e.g. if it was
+ * cancelled or errored out due to another IO also failing). The
+ * issuer of the IO will emit an ERROR when processing the IO's
+ * results
+ */
+ pgaio_result_report(result, td, LOG_SERVER_ONLY);
+
+ return result;
+ }
+
+ /*
+ * As explained above smgrstartreadv(), the smgr API operates on the level
+ * of blocks, rather than bytes. Convert.
+ */
+ result.result /= BLCKSZ;
+
+ Assert(result.result <= td->smgr.nblocks);
+
+ if (result.result == 0)
+ {
+ /* consider 0 blocks read a failure */
+ result.status = PGAIO_RS_ERROR;
+ result.id = PGAIO_HCB_MD_READV;
+ result.error_data = 0;
+
+ /* see comment above the "hard error" case */
+ pgaio_result_report(result, td, LOG_SERVER_ONLY);
+
+ return result;
+ }
+
+ if (result.status != PGAIO_RS_ERROR &&
+ result.result < td->smgr.nblocks)
+ {
+ /* partial reads should be retried at upper level */
+ result.status = PGAIO_RS_PARTIAL;
+ result.id = PGAIO_HCB_MD_READV;
+ }
+
+ return result;
+}
+
+/*
+ * AIO error reporting callback for mdstartreadv().
+ *
+ * Errors are encoded as follows:
+ * - PgAioResult.error_data != 0 encodes IO that failed with errno != 0
+ * - PgAioResult.error_data == 0 encodes IO that didn't read all data
+ */
+static void
+md_readv_report(PgAioResult result, const PgAioTargetData *td, int elevel)
+{
+ RelPathStr path;
+
+ path = relpathbackend(td->smgr.rlocator,
+ td->smgr.is_temp ? MyProcNumber : INVALID_PROC_NUMBER,
+ td->smgr.forkNum);
+
+ if (result.error_data != 0)
+ {
+ /* for errcode_for_file_access() and %m */
+ errno = result.error_data;
+
+ ereport(elevel,
+ errcode_for_file_access(),
+ errmsg("could not read blocks %u..%u in file \"%s\": %m",
+ td->smgr.blockNum,
+ td->smgr.blockNum + td->smgr.nblocks - 1,
+ path.str));
+ }
+ else
+ {
+ /*
+ * NB: This will typically only be output in debug messages, while
+ * retrying a partial IO.
+ */
+ ereport(elevel,
+ errcode(ERRCODE_DATA_CORRUPTED),
+ errmsg("could not read blocks %u..%u in file \"%s\": read only %zu of %zu bytes",
+ td->smgr.blockNum,
+ td->smgr.blockNum + td->smgr.nblocks - 1,
+ path.str,
+ result.result * (size_t) BLCKSZ,
+ td->smgr.nblocks * (size_t) BLCKSZ));
+ }
+}
diff --git a/src/backend/storage/smgr/smgr.c b/src/backend/storage/smgr/smgr.c
index af74f54b43b..bf8f57b410a 100644
--- a/src/backend/storage/smgr/smgr.c
+++ b/src/backend/storage/smgr/smgr.c
@@ -66,6 +66,7 @@
#include "access/xlogutils.h"
#include "lib/ilist.h"
#include "miscadmin.h"
+#include "storage/aio.h"
#include "storage/bufmgr.h"
#include "storage/ipc.h"
#include "storage/md.h"
@@ -106,6 +107,10 @@ typedef struct f_smgr
void (*smgr_readv) (SMgrRelation reln, ForkNumber forknum,
BlockNumber blocknum,
void **buffers, BlockNumber nblocks);
+ void (*smgr_startreadv) (PgAioHandle *ioh,
+ SMgrRelation reln, ForkNumber forknum,
+ BlockNumber blocknum,
+ void **buffers, BlockNumber nblocks);
void (*smgr_writev) (SMgrRelation reln, ForkNumber forknum,
BlockNumber blocknum,
const void **buffers, BlockNumber nblocks,
@@ -117,6 +122,7 @@ typedef struct f_smgr
BlockNumber old_blocks, BlockNumber nblocks);
void (*smgr_immedsync) (SMgrRelation reln, ForkNumber forknum);
void (*smgr_registersync) (SMgrRelation reln, ForkNumber forknum);
+ int (*smgr_fd) (SMgrRelation reln, ForkNumber forknum, BlockNumber blocknum, uint32 *off);
} f_smgr;
static const f_smgr smgrsw[] = {
@@ -134,12 +140,14 @@ static const f_smgr smgrsw[] = {
.smgr_prefetch = mdprefetch,
.smgr_maxcombine = mdmaxcombine,
.smgr_readv = mdreadv,
+ .smgr_startreadv = mdstartreadv,
.smgr_writev = mdwritev,
.smgr_writeback = mdwriteback,
.smgr_nblocks = mdnblocks,
.smgr_truncate = mdtruncate,
.smgr_immedsync = mdimmedsync,
.smgr_registersync = mdregistersync,
+ .smgr_fd = mdfd,
}
};
@@ -157,6 +165,16 @@ static dlist_head unpinned_relns;
static void smgrshutdown(int code, Datum arg);
static void smgrdestroy(SMgrRelation reln);
+static void smgr_aio_reopen(PgAioHandle *ioh);
+static char *smgr_aio_describe_identity(const PgAioTargetData *sd);
+
+
+const PgAioTargetInfo aio_smgr_target_info = {
+ .name = "smgr",
+ .reopen = smgr_aio_reopen,
+ .describe_identity = smgr_aio_describe_identity,
+};
+
/*
* smgrinit(), smgrshutdown() -- Initialize or shut down storage
@@ -709,6 +727,30 @@ smgrreadv(SMgrRelation reln, ForkNumber forknum, BlockNumber blocknum,
RESUME_INTERRUPTS();
}
+/*
+ * smgrstartreadv() -- asynchronous version of smgrreadv()
+ *
+ * This starts an asynchronous readv IO using the IO handle `ioh`. Other than
+ * `ioh` all parameters are the same as smgrreadv().
+ *
+ * Completion callbacks above smgr will be passed the result as the number of
+ * successfully read blocks if the read [partially] succeeds (Buffers for
+ * blocks not successfully read might bear unspecified modifications, up to
+ * the full nblocks). This maintains the abstraction that smgr operates on the
+ * level of blocks, rather than bytes.
+ */
+void
+smgrstartreadv(PgAioHandle *ioh,
+ SMgrRelation reln, ForkNumber forknum, BlockNumber blocknum,
+ void **buffers, BlockNumber nblocks)
+{
+ HOLD_INTERRUPTS();
+ smgrsw[reln->smgr_which].smgr_startreadv(ioh,
+ reln, forknum, blocknum, buffers,
+ nblocks);
+ RESUME_INTERRUPTS();
+}
+
/*
* smgrwritev() -- Write the supplied buffers out.
*
@@ -917,6 +959,29 @@ smgrimmedsync(SMgrRelation reln, ForkNumber forknum)
RESUME_INTERRUPTS();
}
+/*
+ * Return fd for the specified block number and update *off to the appropriate
+ * position.
+ *
+ * This is only to be used for when AIO needs to perform the IO in a different
+ * process than where it was issued (e.g. in an IO worker).
+ */
+static int
+smgrfd(SMgrRelation reln, ForkNumber forknum, BlockNumber blocknum, uint32 *off)
+{
+ int fd;
+
+ /*
+ * The caller needs to prevent interrupts from being processed, otherwise
+ * the FD could be closed prematurely.
+ */
+ Assert(!INTERRUPTS_CAN_BE_PROCESSED());
+
+ fd = smgrsw[reln->smgr_which].smgr_fd(reln, forknum, blocknum, off);
+
+ return fd;
+}
+
/*
* AtEOXact_SMgr
*
@@ -945,3 +1010,98 @@ ProcessBarrierSmgrRelease(void)
smgrreleaseall();
return true;
}
+
+/*
+ * Set target of the IO handle to be smgr and initialize all the relevant
+ * pieces of data.
+ */
+void
+pgaio_io_set_target_smgr(PgAioHandle *ioh,
+ SMgrRelationData *smgr,
+ ForkNumber forknum,
+ BlockNumber blocknum,
+ int nblocks,
+ bool skip_fsync)
+{
+ PgAioTargetData *sd = pgaio_io_get_target_data(ioh);
+
+ pgaio_io_set_target(ioh, PGAIO_TID_SMGR);
+
+ /* backend is implied via IO owner */
+ sd->smgr.rlocator = smgr->smgr_rlocator.locator;
+ sd->smgr.forkNum = forknum;
+ sd->smgr.blockNum = blocknum;
+ sd->smgr.nblocks = nblocks;
+ sd->smgr.is_temp = SmgrIsTemp(smgr);
+ /* Temp relations should never be fsync'd */
+ sd->smgr.skip_fsync = skip_fsync && !SmgrIsTemp(smgr);
+}
+
+/*
+ * Callback for smgr AIO targets, to reopen the file (e.g. because the IO is
+ * executed in a worker).
+ */
+static void
+smgr_aio_reopen(PgAioHandle *ioh)
+{
+ PgAioTargetData *sd = pgaio_io_get_target_data(ioh);
+ PgAioOpData *od = pgaio_io_get_op_data(ioh);
+ SMgrRelation reln;
+ ProcNumber procno;
+ uint32 off;
+
+ /*
+ * The caller needs to prevent interrupts from being processed, otherwise
+ * the FD could be closed again before we get to executing the IO.
+ */
+ Assert(!INTERRUPTS_CAN_BE_PROCESSED());
+
+ if (sd->smgr.is_temp)
+ procno = pgaio_io_get_owner(ioh);
+ else
+ procno = INVALID_PROC_NUMBER;
+
+ reln = smgropen(sd->smgr.rlocator, procno);
+ switch (pgaio_io_get_op(ioh))
+ {
+ case PGAIO_OP_INVALID:
+ pg_unreachable();
+ break;
+ case PGAIO_OP_READV:
+ od->read.fd = smgrfd(reln, sd->smgr.forkNum, sd->smgr.blockNum, &off);
+ Assert(off == od->read.offset);
+ break;
+ case PGAIO_OP_WRITEV:
+ od->write.fd = smgrfd(reln, sd->smgr.forkNum, sd->smgr.blockNum, &off);
+ Assert(off == od->write.offset);
+ break;
+ }
+}
+
+/*
+ * Callback for smgr AIO targets, to describe the target of the IO.
+ */
+static char *
+smgr_aio_describe_identity(const PgAioTargetData *sd)
+{
+ RelPathStr path;
+ char *desc;
+
+ path = relpathbackend(sd->smgr.rlocator,
+ sd->smgr.is_temp ? MyProcNumber : INVALID_PROC_NUMBER,
+ sd->smgr.forkNum);
+
+ if (sd->smgr.nblocks == 0)
+ desc = psprintf(_("file \"%s\""), path.str);
+ else if (sd->smgr.nblocks == 1)
+ desc = psprintf(_("block %u in file \"%s\""),
+ sd->smgr.blockNum,
+ path.str);
+ else
+ desc = psprintf(_("blocks %u..%u in file \"%s\""),
+ sd->smgr.blockNum,
+ sd->smgr.blockNum + sd->smgr.nblocks - 1,
+ path.str);
+
+ return desc;
+}
--
2.48.1.76.g4e746b1a31.dirty
[text/x-diff] v2.14-0003-docs-Add-acronym-and-glossary-entries-for-I-O-.patch (3.5K, ../../d4ff3qxudxn2ze3zk2xwhds5ysezfei3uqeuimymj624t5chsd@sgr2h3rmln5t/4-v2.14-0003-docs-Add-acronym-and-glossary-entries-for-I-O-.patch)
download | inline diff:
From ef4be323f13f0f552cb6d6efee17693119f65962 Mon Sep 17 00:00:00 2001
From: Andres Freund <andres@anarazel.de>
Date: Fri, 21 Mar 2025 15:06:35 -0400
Subject: [PATCH v2.14 03/29] docs: Add acronym and glossary entries for I/O
and AIO
These are fairly basic, but better than nothing. While there are several
opportunities to link to these entries, this patch does not add any. They will
however be referenced by future patches.
Reviewed-by: Noah Misch <noah@leadboat.com>
Discussion: https://postgr.es/m/20250326183102.92.nmisch@google.com
---
doc/src/sgml/acronyms.sgml | 18 ++++++++++++++++++
doc/src/sgml/glossary.sgml | 39 ++++++++++++++++++++++++++++++++++++++
2 files changed, 57 insertions(+)
diff --git a/doc/src/sgml/acronyms.sgml b/doc/src/sgml/acronyms.sgml
index 58d0d90fece..2f906e9f018 100644
--- a/doc/src/sgml/acronyms.sgml
+++ b/doc/src/sgml/acronyms.sgml
@@ -9,6 +9,15 @@
<variablelist>
+ <varlistentry>
+ <term><acronym>AIO</acronym></term>
+ <listitem>
+ <para>
+ <link linkend="glossary-aio">Asynchronous <acronym>I/O</acronym></link>
+ </para>
+ </listitem>
+ </varlistentry>
+
<varlistentry>
<term><acronym>ACL</acronym></term>
<listitem>
@@ -354,6 +363,15 @@
</listitem>
</varlistentry>
+ <varlistentry>
+ <term><acronym>I/O</acronym></term>
+ <listitem>
+ <para>
+ <link linkend="glossary-io">Input/Output</link>
+ </para>
+ </listitem>
+ </varlistentry>
+
<varlistentry>
<term><acronym>ISO</acronym></term>
<listitem>
diff --git a/doc/src/sgml/glossary.sgml b/doc/src/sgml/glossary.sgml
index c0f812e3f5e..b88cac598e9 100644
--- a/doc/src/sgml/glossary.sgml
+++ b/doc/src/sgml/glossary.sgml
@@ -81,6 +81,31 @@
</glossdef>
</glossentry>
+ <glossentry id="glossary-aio">
+ <glossterm>Asynchronous <acronym>I/O</acronym></glossterm>
+ <acronym>AIO</acronym>
+ <indexterm>
+ <primary>Asynchronous <acronym>I/O</acronym></primary>
+ </indexterm>
+ <glossdef>
+ <para>
+ Asynchronous <acronym>I/O</acronym> (<acronym>AIO</acronym>) describes
+ performing <acronym>I/O</acronym> in a non-blocking way (asynchronously),
+ in contrast to synchronous <acronym>I/O</acronym>, which blocks for the
+ entire duration of the <acronym>I/O</acronym>.
+ </para>
+ <para>
+ With <acronym>AIO</acronym>, starting an <acronym>I/O</acronym> operation
+ is separated from waiting for the result of the operation, allowing
+ multiple <acronym>I/O</acronym> operations to be initiated concurrently,
+ as well as performing <acronym>CPU</acronym> heavy operations
+ concurrently with <acronym>I/O</acronym>. The price for that increased
+ concurrency is increased complexity.
+ </para>
+ <glossseealso otherterm="glossary-io" />
+ </glossdef>
+ </glossentry>
+
<glossentry id="glossary-atomic">
<glossterm>Atomic</glossterm>
<glossdef>
@@ -938,6 +963,20 @@
</glossdef>
</glossentry>
+ <glossentry id="glossary-io">
+ <glossterm>Input/Output</glossterm>
+ <acronym>I/O</acronym>
+ <glossdef>
+ <para>
+ Input/Output (<acronym>I/O</acronym>) describes the communication between
+ a program and peripheral devices. In the context of database systems,
+ <acronym>I/O</acronym> commonly, but not exclusively, refers to
+ interaction with storage devices or the network.
+ </para>
+ <glossseealso otherterm="glossary-aio" />
+ </glossdef>
+ </glossentry>
+
<glossentry id="glossary-insert">
<glossterm>Insert</glossterm>
<glossdef>
--
2.48.1.76.g4e746b1a31.dirty
[text/x-diff] v2.14-0004-aio-Add-pg_aios-view.patch (19.4K, ../../d4ff3qxudxn2ze3zk2xwhds5ysezfei3uqeuimymj624t5chsd@sgr2h3rmln5t/5-v2.14-0004-aio-Add-pg_aios-view.patch)
download | inline diff:
From d08ee04f17ea642507ec8ee20401f8a079c10664 Mon Sep 17 00:00:00 2001
From: Andres Freund <andres@anarazel.de>
Date: Tue, 18 Mar 2025 14:40:06 -0400
Subject: [PATCH v2.14 04/29] aio: Add pg_aios view
The new view lists all IO handles that are currently in use and is mainly
useful for PG developers, but may also be useful when tuning PG.
FIXME:
- catversion bump before commit
Reviewed-by: Noah Misch <noah@leadboat.com>
Discussion: https://postgr.es/m/uvrtrknj4kdytuboidbhwclo4gxhswwcpgadptsjvjqcluzmah%40brqs62irg4dt
---
src/include/catalog/pg_proc.dat | 10 +
src/backend/catalog/system_views.sql | 3 +
src/backend/storage/aio/Makefile | 1 +
src/backend/storage/aio/aio_funcs.c | 225 +++++++++++++++++++++
src/backend/storage/aio/meson.build | 1 +
doc/src/sgml/system-views.sgml | 288 +++++++++++++++++++++++++++
src/test/regress/expected/rules.out | 16 ++
7 files changed, 544 insertions(+)
create mode 100644 src/backend/storage/aio/aio_funcs.c
diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat
index 8b68b16d79d..52dbce05a45 100644
--- a/src/include/catalog/pg_proc.dat
+++ b/src/include/catalog/pg_proc.dat
@@ -12493,4 +12493,14 @@
proargtypes => 'int4',
prosrc => 'gist_stratnum_common' },
+# AIO related functions
+{ oid => '9200', descr => 'information about in-progress asynchronous IOs',
+ proname => 'pg_get_aios', prorows => '100', proretset => 't',
+ provolatile => 'v', proparallel => 'r', prorettype => 'record', proargtypes => '',
+ proallargtypes => '{int4,int4,int8,text,text,int8,int8,text,int2,int4,text,text,bool,bool,bool}',
+ proargmodes => '{o,o,o,o,o,o,o,o,o,o,o,o,o,o,o}',
+ proargnames => '{pid,io_id,io_generation,state,operation,off,length,target,handle_data_len,raw_result,result,target_desc,f_sync,f_localmem,f_buffered}',
+ prosrc => 'pg_get_aios' },
+
+
]
diff --git a/src/backend/catalog/system_views.sql b/src/backend/catalog/system_views.sql
index 31d269b7ee0..37fd4bc2566 100644
--- a/src/backend/catalog/system_views.sql
+++ b/src/backend/catalog/system_views.sql
@@ -1391,3 +1391,6 @@ CREATE VIEW pg_stat_subscription_stats AS
CREATE VIEW pg_wait_events AS
SELECT * FROM pg_get_wait_events();
+
+CREATE VIEW pg_aios AS
+ SELECT * FROM pg_get_aios();
diff --git a/src/backend/storage/aio/Makefile b/src/backend/storage/aio/Makefile
index c06c50771e0..3f2469cc399 100644
--- a/src/backend/storage/aio/Makefile
+++ b/src/backend/storage/aio/Makefile
@@ -11,6 +11,7 @@ include $(top_builddir)/src/Makefile.global
OBJS = \
aio.o \
aio_callback.o \
+ aio_funcs.o \
aio_init.o \
aio_io.o \
aio_target.o \
diff --git a/src/backend/storage/aio/aio_funcs.c b/src/backend/storage/aio/aio_funcs.c
new file mode 100644
index 00000000000..765fb16aef4
--- /dev/null
+++ b/src/backend/storage/aio/aio_funcs.c
@@ -0,0 +1,225 @@
+/*-------------------------------------------------------------------------
+ *
+ * aio_funcs.c
+ * AIO - SQL interface for AIO
+ *
+ *
+ * Portions Copyright (c) 1996-2025, PostgreSQL Global Development Group
+ * Portions Copyright (c) 1994, Regents of the University of California
+ *
+ * IDENTIFICATION
+ * src/backend/storage/aio/aio_funcs.c
+ *
+ *-------------------------------------------------------------------------
+ */
+
+#include "postgres.h"
+
+#include "fmgr.h"
+#include "funcapi.h"
+#include "nodes/execnodes.h"
+#include "port/atomics.h"
+#include "storage/aio_internal.h"
+#include "storage/lock.h"
+#include "storage/proc.h"
+#include "storage/procnumber.h"
+#include "utils/builtins.h"
+#include "utils/fmgrprotos.h"
+#include "utils/tuplestore.h"
+
+
+/*
+ * Byte length of an iovec.
+ */
+static size_t
+iov_byte_length(const struct iovec *iov, int cnt)
+{
+ size_t len = 0;
+
+ for (int i = 0; i < cnt; i++)
+ {
+ len += iov[i].iov_len;
+ }
+
+ return len;
+}
+
+Datum
+pg_get_aios(PG_FUNCTION_ARGS)
+{
+ ReturnSetInfo *rsinfo = (ReturnSetInfo *) fcinfo->resultinfo;
+
+ InitMaterializedSRF(fcinfo, 0);
+
+#define PG_GET_AIOS_COLS 15
+
+ for (uint64 i = 0; i < pgaio_ctl->io_handle_count; i++)
+ {
+ PgAioHandle *live_ioh = &pgaio_ctl->io_handles[i];
+ uint32 ioh_id = pgaio_io_get_id(live_ioh);
+ Datum values[PG_GET_AIOS_COLS] = {0};
+ bool nulls[PG_GET_AIOS_COLS] = {0};
+ ProcNumber owner;
+ PGPROC *owner_proc;
+ int32 owner_pid;
+ PgAioHandleState start_state;
+ uint64 start_generation;
+ PgAioHandle ioh_copy;
+ struct iovec iov_copy[PG_IOV_MAX];
+
+
+ /*
+ * There is no lock that could prevent the state of the IO to advance
+ * concurrently - and we don't want to introduce one, as that would
+ * introduce atomics into a very common path. Instead we
+ *
+ * 1) Determine the state + generation of the IO.
+ *
+ * 2) Copy the IO to local memory.
+ *
+ * 3) Check if state or generation of the IO changed. If the state
+ * changed, retry, if the generation changed don't display the IO.
+ */
+
+ /* 1) from above */
+ start_generation = live_ioh->generation;
+ pg_read_barrier();
+ start_state = live_ioh->state;
+
+retry:
+ if (start_state == PGAIO_HS_IDLE)
+ continue;
+
+ /* 2) from above */
+ memcpy(&ioh_copy, live_ioh, sizeof(PgAioHandle));
+
+ /*
+ * Safe to copy even if no iovec is used - we always reserve the
+ * required space.
+ */
+ memcpy(&iov_copy, &pgaio_ctl->iovecs[ioh_copy.iovec_off],
+ PG_IOV_MAX * sizeof(struct iovec));
+
+ /*
+ * Copy information about owner before 3) below, if the process exited
+ * it'd have to wait for the IO to finish first, which we would detect
+ * in 3).
+ */
+ owner = ioh_copy.owner_procno;
+ owner_proc = GetPGProcByNumber(owner);
+ owner_pid = owner_proc->pid;
+
+ /* 3) from above */
+ pg_read_barrier();
+
+ /*
+ * The IO completed and a new one was started with the same ID. Don't
+ * display it - it really started after this function was called.
+ * There be a risk of a livelock if we just retried endlessly, if IOs
+ * complete very quickly.
+ */
+ if (live_ioh->generation != start_generation)
+ continue;
+
+ /*
+ * The IO's state changed while we were "rendering" it. Just start
+ * from scratch. There's no risk of a livelock here, as an IO has a
+ * limited sets of states it can be in, and state changes go only in a
+ * single direction.
+ */
+ if (live_ioh->state != start_state)
+ goto retry;
+
+ /*
+ * Now that we have copied the IO into local memory and checked that
+ * it's still in the same state, we are not allowed to access "live"
+ * memory anymore. To make it slightly easier to catch such cases, set
+ * the "live" pointers to NULL.
+ */
+ live_ioh = NULL;
+ owner_proc = NULL;
+
+
+ /* column: owning pid */
+ if (owner_pid != 0)
+ values[0] = Int32GetDatum(owner_pid);
+ else
+ nulls[0] = false;
+
+ /* column: IO's id */
+ values[1] = ioh_id;
+
+ /* column: IO's generation */
+ values[2] = Int64GetDatum(start_generation);
+
+ /* column: IO's state */
+ values[3] = CStringGetTextDatum(pgaio_io_get_state_name(&ioh_copy));
+
+ /*
+ * If the IO is in PGAIO_HS_HANDED_OUT state, none of the following
+ * fields are valid yet (or are in the process of being set).
+ * Therefore we don't want to display any other columns.
+ */
+ if (start_state == PGAIO_HS_HANDED_OUT)
+ {
+ memset(nulls + 4, 1, (lengthof(nulls) - 4) * sizeof(bool));
+ goto display;
+ }
+
+ /* column: IO's operation */
+ values[4] = CStringGetTextDatum(pgaio_io_get_op_name(&ioh_copy));
+
+ /* columns: details about the IO's operation (offset, length) */
+ switch (ioh_copy.op)
+ {
+ case PGAIO_OP_INVALID:
+ nulls[5] = true;
+ nulls[6] = true;
+ break;
+ case PGAIO_OP_READV:
+ values[5] = Int64GetDatum(ioh_copy.op_data.read.offset);
+ values[6] =
+ Int64GetDatum(iov_byte_length(iov_copy, ioh_copy.op_data.read.iov_length));
+ break;
+ case PGAIO_OP_WRITEV:
+ values[5] = Int64GetDatum(ioh_copy.op_data.write.offset);
+ values[6] =
+ Int64GetDatum(iov_byte_length(iov_copy, ioh_copy.op_data.write.iov_length));
+ break;
+ }
+
+ /* column: IO's target */
+ values[7] = CStringGetTextDatum(pgaio_io_get_target_name(&ioh_copy));
+
+ /* column: length of IO's data array */
+ values[8] = Int16GetDatum(ioh_copy.handle_data_len);
+
+ /* column: raw result (i.e. some form of syscall return value) */
+ if (start_state == PGAIO_HS_COMPLETED_IO
+ || start_state == PGAIO_HS_COMPLETED_SHARED
+ || start_state == PGAIO_HS_COMPLETED_LOCAL)
+ values[9] = Int32GetDatum(ioh_copy.result);
+ else
+ nulls[9] = true;
+
+ /*
+ * column: result in the higher level representation (unknown if not
+ * finished)
+ */
+ values[10] =
+ CStringGetTextDatum(pgaio_result_status_string(ioh_copy.distilled_result.status));
+
+ /* column: target description */
+ values[11] = CStringGetTextDatum(pgaio_io_get_target_description(&ioh_copy));
+
+ /* columns: one for each flag */
+ values[12] = BoolGetDatum(ioh_copy.flags & PGAIO_HF_SYNCHRONOUS);
+ values[13] = BoolGetDatum(ioh_copy.flags & PGAIO_HF_REFERENCES_LOCAL);
+ values[14] = BoolGetDatum(ioh_copy.flags & PGAIO_HF_BUFFERED);
+
+display:
+ tuplestore_putvalues(rsinfo->setResult, rsinfo->setDesc, values, nulls);
+ }
+
+ return (Datum) 0;
+}
diff --git a/src/backend/storage/aio/meson.build b/src/backend/storage/aio/meson.build
index 2f0f03d8071..da6df2d3654 100644
--- a/src/backend/storage/aio/meson.build
+++ b/src/backend/storage/aio/meson.build
@@ -3,6 +3,7 @@
backend_sources += files(
'aio.c',
'aio_callback.c',
+ 'aio_funcs.c',
'aio_init.c',
'aio_io.c',
'aio_target.c',
diff --git a/doc/src/sgml/system-views.sgml b/doc/src/sgml/system-views.sgml
index 3f5a306247e..5154512b16b 100644
--- a/doc/src/sgml/system-views.sgml
+++ b/doc/src/sgml/system-views.sgml
@@ -51,6 +51,11 @@
</thead>
<tbody>
+ <row>
+ <entry><link linkend="view-pg-aios"><structname>pg_aios</structname></link></entry>
+ <entry>In-use asynchronous IO handles</entry>
+ </row>
+
<row>
<entry><link linkend="view-pg-available-extensions"><structname>pg_available_extensions</structname></link></entry>
<entry>available extensions</entry>
@@ -231,6 +236,289 @@
</table>
</sect1>
+ <sect1 id="view-pg-aios">
+ <title><structname>pg_aios</structname></title>
+
+ <indexterm zone="view-pg-aios">
+ <primary>pg_aios</primary>
+ </indexterm>
+
+ <para>
+ The <structname>pg_aios</structname> view lists all <xref
+ linkend="glossary-aio"/> handles that are currently in-use. An I/O handle
+ is used to reference an I/O operation that is being prepared, executed or
+ is in the process of completing. <structname>pg_aios</structname> contains
+ one row for each I/O handle.
+ </para>
+
+ <para>
+ This view is mainly useful for developers of
+ <productname>PostgreSQL</productname>, but may also be useful when tuning
+ <productname>PostgreSQL</productname>.
+ </para>
+
+ <table>
+ <title><structname>pg_aios</structname> Columns</title>
+ <tgroup cols="1">
+ <thead>
+ <row>
+ <entry role="catalog_table_entry"><para role="column_definition">
+ Column Type
+ </para>
+ <para>
+ Description
+ </para></entry>
+ </row>
+ </thead>
+
+ <tbody>
+ <row>
+ <entry role="catalog_table_entry"><para role="column_definition">
+ <structfield>pid</structfield> <type>int4</type>
+ </para>
+ <para>
+ Process ID of the server process that is issuing this I/O.
+ </para></entry>
+ </row>
+
+ <row>
+ <entry role="catalog_table_entry"><para role="column_definition">
+ <structfield>io_id</structfield> <type>int4</type>
+ </para>
+ <para>
+ Identifier of the I/O handle. Handles are reused once the I/O
+ completed (or if the handle is released before I/O is started). On reuse
+ <link linkend="view-pg-aios-io-generation">
+ <structname>pg_aios</structname>.<structfield>io_generation</structfield>
+ </link>
+ is incremented.
+ </para></entry>
+ </row>
+
+ <row>
+ <entry role="catalog_table_entry" id="view-pg-aios-io-generation"><para role="column_definition">
+ <structfield>io_generation</structfield> <type>int8</type>
+ </para>
+ <para>
+ Generation of the I/O handle.
+ </para></entry>
+ </row>
+
+ <row>
+ <entry role="catalog_table_entry"><para role="column_definition">
+ <structfield>state</structfield> <type>text</type>
+ </para>
+ <para>
+ State of the I/O handle:
+ <itemizedlist>
+ <listitem>
+ <para>
+ <literal>HANDED_OUT</literal>, referenced by code but not yet used
+ </para>
+ </listitem>
+ <listitem>
+ <para>
+ <literal>DEFINED</literal>, information necessary for execution is known
+ </para>
+ </listitem>
+ <listitem>
+ <para>
+ <literal>STAGED</literal>, ready for execution
+ </para>
+ </listitem>
+ <listitem>
+ <para>
+ <literal>SUBMITTED</literal>, submitted for execution
+ </para>
+ </listitem>
+ <listitem>
+ <para>
+ <literal>COMPLETED_IO</literal>, finished, but result has not yet been processed
+ </para>
+ </listitem>
+ <listitem>
+ <para>
+ <literal>COMPLETED_SHARED</literal>, shared completion processing completed
+ </para>
+ </listitem>
+ <listitem>
+ <para>
+ <literal>COMPLETED_LOCAL</literal>, backend local completion processing completed
+ </para>
+ </listitem>
+ </itemizedlist>
+ </para></entry>
+ </row>
+
+ <row>
+ <entry role="catalog_table_entry"><para role="column_definition">
+ <structfield>operation</structfield> <type>text</type>
+ </para>
+ <para>
+ Operation performed using the I/O handle:
+ <itemizedlist>
+ <listitem>
+ <para>
+ <literal>invalid</literal>, not yet known
+ </para>
+ </listitem>
+ <listitem>
+ <para>
+ <literal>readv</literal>, a vectored read
+ </para>
+ </listitem>
+ <listitem>
+ <para>
+ <literal>writev</literal>, a vectored write
+ </para>
+ </listitem>
+ </itemizedlist>
+ </para></entry>
+ </row>
+
+ <row>
+ <entry role="catalog_table_entry"><para role="column_definition">
+ <structfield>off</structfield> <type>int8</type>
+ </para>
+ <para>
+ Offset of the I/O operation.
+ </para></entry>
+ </row>
+
+ <row>
+ <entry role="catalog_table_entry"><para role="column_definition">
+ <structfield>length</structfield> <type>int8</type>
+ </para>
+ <para>
+ Length of the I/O operation.
+ </para></entry>
+ </row>
+
+ <row>
+ <entry role="catalog_table_entry"><para role="column_definition">
+ <structfield>target</structfield> <type>text</type>
+ </para>
+ <para>
+ What kind of object is the I/O targeting:
+ <itemizedlist spacing="compact">
+ <listitem>
+ <para>
+ <literal>smgr</literal>, I/O on relations
+ </para>
+ </listitem>
+ </itemizedlist>
+ </para></entry>
+ </row>
+
+ <row>
+ <entry role="catalog_table_entry"><para role="column_definition">
+ <structfield>handle_data_len</structfield> <type>int2</type>
+ </para>
+ <para>
+ Length of the data associated with the I/O operation. For I/O to/from
+ <xref linkend="guc-shared-buffers"/> and <xref
+ linkend="guc-temp-buffers"/>, this indicates the number of buffers the
+ I/O is operating on.
+ </para></entry>
+ </row>
+
+ <row>
+ <entry role="catalog_table_entry"><para role="column_definition">
+ <structfield>raw_result</structfield> <type>int4</type>
+ </para>
+ <para>
+ Low-level result of the I/O operation, or NULL if the operation has not
+ yet completed.
+ </para></entry>
+ </row>
+
+ <row>
+ <entry role="catalog_table_entry"><para role="column_definition">
+ <structfield>result</structfield> <type>text</type>
+ </para>
+ <para>
+ High-level result of the I/O operation:
+ <itemizedlist>
+ <listitem>
+ <para>
+ <literal>UNKNOWN</literal> means that the result of the
+ operation is not yet known.
+ </para>
+ </listitem>
+ <listitem>
+ <para>
+ <literal>OK</literal> means the I/O completed successfully.
+ </para>
+ </listitem>
+ <listitem>
+ <para>
+ <literal>PARTIAL</literal> means that the I/O completed without
+ error, but did not process all data. Commonly callers will need to
+ retry and perform the remainder of the work in a separate I/O.
+ </para>
+ </listitem>
+ <listitem>
+ <para>
+ <literal>WARNING</literal> means that the I/O completed without
+ error, but that execution of the IO triggered a warning. E.g. when
+ encountering a corrupted buffer with <xref
+ linkend="guc-zero-damaged-pages"/> enabled.
+ </para>
+ </listitem>
+ <listitem>
+ <para>
+ <literal>ERROR</literal> means the I/O failed with an error.
+ </para>
+ </listitem>
+ </itemizedlist>
+ </para></entry>
+ </row>
+
+ <row>
+ <entry role="catalog_table_entry"><para role="column_definition">
+ <structfield>target_desc</structfield> <type>text</type>
+ </para>
+ <para>
+ Description of what the I/O operation is targeting.
+ </para></entry>
+ </row>
+
+ <row>
+ <entry role="catalog_table_entry"><para role="column_definition">
+ <structfield>f_sync</structfield> <type>bool</type>
+ </para>
+ <para>
+ Flag indicating whether the I/O is executed synchronously.
+ </para></entry>
+ </row>
+
+ <row>
+ <entry role="catalog_table_entry"><para role="column_definition">
+ <structfield>f_localmem</structfield> <type>bool</type>
+ </para>
+ <para>
+ Flag indicating whether the I/O references process local memory.
+ </para></entry>
+ </row>
+
+ <row>
+ <entry role="catalog_table_entry"><para role="column_definition">
+ <structfield>f_buffered</structfield> <type>bool</type>
+ </para>
+ <para>
+ Flag indicating whether the I/O is buffered I/O.
+ </para></entry>
+ </row>
+
+ </tbody>
+ </tgroup>
+ </table>
+
+ <para>
+ The <structname>pg_aios</structname> view is read-only.
+ </para>
+ </sect1>
+
<sect1 id="view-pg-available-extensions">
<title><structname>pg_available_extensions</structname></title>
diff --git a/src/test/regress/expected/rules.out b/src/test/regress/expected/rules.out
index 47478969135..d9533deb04e 100644
--- a/src/test/regress/expected/rules.out
+++ b/src/test/regress/expected/rules.out
@@ -1286,6 +1286,22 @@ drop table cchild;
SELECT viewname, definition FROM pg_views
WHERE schemaname = 'pg_catalog'
ORDER BY viewname;
+pg_aios| SELECT pid,
+ io_id,
+ io_generation,
+ state,
+ operation,
+ off,
+ length,
+ target,
+ handle_data_len,
+ raw_result,
+ result,
+ target_desc,
+ f_sync,
+ f_localmem,
+ f_buffered
+ FROM pg_get_aios() pg_get_aios(pid, io_id, io_generation, state, operation, off, length, target, handle_data_len, raw_result, result, target_desc, f_sync, f_localmem, f_buffered);
pg_available_extension_versions| SELECT e.name,
e.version,
(x.extname IS NOT NULL) AS installed,
--
2.48.1.76.g4e746b1a31.dirty
[text/x-diff] v2.14-0005-localbuf-Track-pincount-in-BufferDesc-as-well.patch (3.3K, ../../d4ff3qxudxn2ze3zk2xwhds5ysezfei3uqeuimymj624t5chsd@sgr2h3rmln5t/6-v2.14-0005-localbuf-Track-pincount-in-BufferDesc-as-well.patch)
download | inline diff:
From 88da964c4b51e32e6ab5c2a4896b24c5ca7f615a Mon Sep 17 00:00:00 2001
From: Andres Freund <andres@anarazel.de>
Date: Tue, 18 Mar 2025 14:40:06 -0400
Subject: [PATCH v2.14 05/29] localbuf: Track pincount in BufferDesc as well
For AIO on temporary table buffers the AIO subsystem needs to be able to
ensure a pin on a buffer while AIO is going on, even if the IO issuing query
errors out. Tracking the buffer in LocalRefCount does not work, as it would
cause CheckForLocalBufferLeaks() to assert out.
Instead, also track the refcount in BufferDesc.state, not just
LocalRefCount. This also makes local buffers behave a bit more akin to shared
buffers.
Note that we still don't need locking, AIO completion callbacks for local
buffers are executed in the issuing session (i.e. nobody else has access to
the BufferDesc).
Reviewed-by: Noah Misch <noah@leadboat.com>
Discussion: https://postgr.es/m/uvrtrknj4kdytuboidbhwclo4gxhswwcpgadptsjvjqcluzmah%40brqs62irg4dt
---
src/backend/storage/buffer/localbuf.c | 28 +++++++++++++++++++++++++--
1 file changed, 26 insertions(+), 2 deletions(-)
diff --git a/src/backend/storage/buffer/localbuf.c b/src/backend/storage/buffer/localbuf.c
index f172a5c7820..3a722321533 100644
--- a/src/backend/storage/buffer/localbuf.c
+++ b/src/backend/storage/buffer/localbuf.c
@@ -249,6 +249,13 @@ GetLocalVictimBuffer(void)
pg_atomic_unlocked_write_u32(&bufHdr->state, buf_state);
trycounter = NLocBuffer;
}
+ else if (BUF_STATE_GET_REFCOUNT(buf_state) > 0)
+ {
+ /*
+ * This can be reached if the backend initiated AIO for this
+ * buffer and then errored out.
+ */
+ }
else
{
/* Found a usable buffer */
@@ -570,7 +577,13 @@ InvalidateLocalBuffer(BufferDesc *bufHdr, bool check_unreferenced)
buf_state = pg_atomic_read_u32(&bufHdr->state);
- if (check_unreferenced && LocalRefCount[bufid] != 0)
+ /*
+ * We need to test not just LocalRefCount[bufid] but also the BufferDesc
+ * itself, as the latter is used to represent a pin by the AIO subsystem.
+ * This can happen if AIO is initiated and then the query errors out.
+ */
+ if (check_unreferenced &&
+ (LocalRefCount[bufid] != 0 || BUF_STATE_GET_REFCOUNT(buf_state) != 0))
elog(ERROR, "block %u of %s is still referenced (local %u)",
bufHdr->tag.blockNum,
relpathbackend(BufTagGetRelFileLocator(&bufHdr->tag),
@@ -744,12 +757,13 @@ PinLocalBuffer(BufferDesc *buf_hdr, bool adjust_usagecount)
if (LocalRefCount[bufid] == 0)
{
NLocalPinnedBuffers++;
+ buf_state += BUF_REFCOUNT_ONE;
if (adjust_usagecount &&
BUF_STATE_GET_USAGECOUNT(buf_state) < BM_MAX_USAGE_COUNT)
{
buf_state += BUF_USAGECOUNT_ONE;
- pg_atomic_unlocked_write_u32(&buf_hdr->state, buf_state);
}
+ pg_atomic_unlocked_write_u32(&buf_hdr->state, buf_state);
}
LocalRefCount[bufid]++;
ResourceOwnerRememberBuffer(CurrentResourceOwner,
@@ -775,7 +789,17 @@ UnpinLocalBufferNoOwner(Buffer buffer)
Assert(NLocalPinnedBuffers > 0);
if (--LocalRefCount[buffid] == 0)
+ {
+ BufferDesc *buf_hdr = GetLocalBufferDescriptor(buffid);
+ uint32 buf_state;
+
NLocalPinnedBuffers--;
+
+ buf_state = pg_atomic_read_u32(&buf_hdr->state);
+ Assert(BUF_STATE_GET_REFCOUNT(buf_state) > 0);
+ buf_state -= BUF_REFCOUNT_ONE;
+ pg_atomic_unlocked_write_u32(&buf_hdr->state, buf_state);
+ }
}
/*
--
2.48.1.76.g4e746b1a31.dirty
[text/x-diff] v2.14-0006-aio-bufmgr-Comment-fixes.patch (4.8K, ../../d4ff3qxudxn2ze3zk2xwhds5ysezfei3uqeuimymj624t5chsd@sgr2h3rmln5t/7-v2.14-0006-aio-bufmgr-Comment-fixes.patch)
download | inline diff:
From 1766e29c3bbe0e0659d3bf1d5d1e6d7aa88c1a05 Mon Sep 17 00:00:00 2001
From: Andres Freund <andres@anarazel.de>
Date: Thu, 20 Mar 2025 11:32:10 -0400
Subject: [PATCH v2.14 06/29] aio, bufmgr: Comment fixes
Some of these comments have been wrong for a while (12f3867f5534), some I
recently introduced (da7226993fd, 55b454d0e14). This also updates a comment in
FlushBuffer(), which will be copied in a future commit.
These changes seem big enough to be worth doing in separate commits.
Suggested-by: Noah Misch <noah@leadboat.com>
Discussion: https://postgr.es/m/20250319212530.80.nmisch@google.com
---
src/include/storage/aio.h | 2 +-
src/include/storage/aio_internal.h | 22 +++++++++++++++++++++-
src/backend/postmaster/postmaster.c | 2 +-
src/backend/storage/buffer/bufmgr.c | 10 ++++------
4 files changed, 27 insertions(+), 9 deletions(-)
diff --git a/src/include/storage/aio.h b/src/include/storage/aio.h
index c386fc96d90..5c886e478a5 100644
--- a/src/include/storage/aio.h
+++ b/src/include/storage/aio.h
@@ -80,7 +80,7 @@ typedef enum PgAioHandleFlags
/*
* The IO operations supported by the AIO subsystem.
*
- * This could be in aio_internal.h, as it is not pubicly referenced, but
+ * This could be in aio_internal.h, as it is not publicly referenced, but
* PgAioOpData currently *does* need to be public, therefore keeping this
* public seems to make sense.
*/
diff --git a/src/include/storage/aio_internal.h b/src/include/storage/aio_internal.h
index fb0425ccbfc..7f18da2c856 100644
--- a/src/include/storage/aio_internal.h
+++ b/src/include/storage/aio_internal.h
@@ -34,6 +34,11 @@
* linearly through all states.
*
* State changes should all go through pgaio_io_update_state().
+ *
+ * Note that the externally visible functions to start IO
+ * (e.g. FileStartReadV(), via pgaio_io_start_readv()) move an IO from
+ * PGAIO_HS_HANDED_OUT to at least PGAIO_HS_STAGED and at most
+ * PGAIO_HS_COMPLETED_LOCAL (at which point the handle will be reused).
*/
typedef enum PgAioHandleState
{
@@ -285,6 +290,9 @@ typedef struct IoMethodOps
/*
* Start executing passed in IOs.
*
+ * Shall advance state to at least PGAIO_HS_SUBMITTED. (By the time this
+ * returns, other backends might have advanced the state further.)
+ *
* Will not be called if ->needs_synchronous_execution() returned true.
*
* num_staged_ios is <= PGAIO_SUBMIT_BATCH_SIZE.
@@ -293,12 +301,24 @@ typedef struct IoMethodOps
*/
int (*submit) (uint16 num_staged_ios, PgAioHandle **staged_ios);
- /*
+ /* ---
* Wait for the IO to complete. Optional.
*
+ * On return, state shall be on of
+ * - PGAIO_HS_COMPLETED_IO
+ * - PGAIO_HS_COMPLETED_SHARED
+ * - PGAIO_HS_COMPLETED_LOCAL
+ *
+ * The callback must not block if the handle is already in one of those
+ * states, or has been reused (see pgaio_io_was_recycled()). If, on
+ * return, the state is PGAIO_HS_COMPLETED_IO, state will reach
+ * PGAIO_HS_COMPLETED_SHARED without further intervention by the IO
+ * method.
+ *
* If not provided, it needs to be guaranteed that the IO method calls
* pgaio_io_process_completion() without further interaction by the
* issuing backend.
+ * ---
*/
void (*wait_one) (PgAioHandle *ioh,
uint64 ref_generation);
diff --git a/src/backend/postmaster/postmaster.c b/src/backend/postmaster/postmaster.c
index a0c37532d2f..c966c2e83af 100644
--- a/src/backend/postmaster/postmaster.c
+++ b/src/backend/postmaster/postmaster.c
@@ -4401,7 +4401,7 @@ maybe_adjust_io_workers(void)
++io_worker_count;
}
else
- break; /* XXX try again soon? */
+ break; /* try again next time */
}
/* Too many running? */
diff --git a/src/backend/storage/buffer/bufmgr.c b/src/backend/storage/buffer/bufmgr.c
index 5cac8cd7389..03317b49025 100644
--- a/src/backend/storage/buffer/bufmgr.c
+++ b/src/backend/storage/buffer/bufmgr.c
@@ -3929,9 +3929,10 @@ FlushBuffer(BufferDesc *buf, SMgrRelation reln, IOObject io_object,
XLogFlush(recptr);
/*
- * Now it's safe to write buffer to disk. Note that no one else should
- * have been able to write it while we were busy with log flushing because
- * only one process at a time can set the BM_IO_IN_PROGRESS bit.
+ * Now it's safe to write the buffer to disk. Note that no one else should
+ * have been able to write it, while we were busy with log flushing,
+ * because we got the exclusive right to perform I/O by setting the
+ * BM_IO_IN_PROGRESS bit.
*/
bufBlock = BufHdrGetBlock(buf);
@@ -5499,9 +5500,6 @@ IsBufferCleanupOK(Buffer buffer)
/*
* Functions for buffer I/O handling
*
- * Note: We assume that nested buffer I/O never occurs.
- * i.e at most one BM_IO_IN_PROGRESS bit is set per proc.
- *
* Also note that these are used only for shared buffers, not local ones.
*/
--
2.48.1.76.g4e746b1a31.dirty
[text/x-diff] v2.14-0007-aio-Add-WARNING-result-status.patch (4.1K, ../../d4ff3qxudxn2ze3zk2xwhds5ysezfei3uqeuimymj624t5chsd@sgr2h3rmln5t/8-v2.14-0007-aio-Add-WARNING-result-status.patch)
download | inline diff:
From 50e1cb669103b7de10bf69bea8d68d39e33076d7 Mon Sep 17 00:00:00 2001
From: Andres Freund <andres@anarazel.de>
Date: Wed, 26 Mar 2025 14:33:09 -0400
Subject: [PATCH v2.14 07/29] aio: Add WARNING result status
If an IO succeeds, but issues a warning, e.g. due to a page verification
failure with zero_damaged_pages, we want to issue that warning in the context
of the issuer of the IO, not the process that executes the completion (always
the case for worker).
It's already possible for a completion callback to report a custom error
message, we just didn't have a result status that allowed a user of AIO to
know that a warning should be emitted even though the IO request succeeded.
All that's needed for that is a dedicated PGAIO_RS_ value.
Previously there were not enough bits in PgAioResult.id for the new
value. Increase. While at that, add defines for the amount of bits and static
asserts to check that the widths are appropriate.
---
src/include/storage/aio.h | 6 +++++-
src/include/storage/aio_types.h | 26 +++++++++++++++++++++-----
src/backend/storage/aio/aio.c | 2 ++
3 files changed, 28 insertions(+), 6 deletions(-)
diff --git a/src/include/storage/aio.h b/src/include/storage/aio.h
index 5c886e478a5..d819f9e37ca 100644
--- a/src/include/storage/aio.h
+++ b/src/include/storage/aio.h
@@ -191,12 +191,16 @@ struct PgAioTargetInfo
*/
typedef enum PgAioHandleCallbackID
{
- PGAIO_HCB_INVALID,
+ PGAIO_HCB_INVALID = 0,
PGAIO_HCB_MD_READV,
PGAIO_HCB_MD_WRITEV,
} PgAioHandleCallbackID;
+#define PGAIO_HCB_MAX PGAIO_HCB_MD_WRITEV
+StaticAssertDecl(PGAIO_HCB_MAX <= (1 << PGAIO_RESULT_ID_BITS),
+ "PGAIO_HCB_MAX is too big for PGAIO_RESULT_ID_BITS");
+
typedef void (*PgAioHandleCallbackStage) (PgAioHandle *ioh, uint8 cb_flags);
typedef PgAioResult (*PgAioHandleCallbackComplete) (PgAioHandle *ioh, PgAioResult prior_result, uint8 cb_flags);
diff --git a/src/include/storage/aio_types.h b/src/include/storage/aio_types.h
index debe8163d4e..156132cde03 100644
--- a/src/include/storage/aio_types.h
+++ b/src/include/storage/aio_types.h
@@ -79,32 +79,48 @@ typedef enum PgAioResultStatus
{
PGAIO_RS_UNKNOWN, /* not yet completed / uninitialized */
PGAIO_RS_OK,
- PGAIO_RS_PARTIAL, /* did not fully succeed, but no error */
- PGAIO_RS_ERROR,
+ PGAIO_RS_PARTIAL, /* did not fully succeed, no warning/error */
+ PGAIO_RS_WARNING, /* [partially] succeeded, with a warning */
+ PGAIO_RS_ERROR, /* failed entirely */
} PgAioResultStatus;
/*
* Result of IO operation, visible only to the initiator of IO.
+ *
+ * We need to be careful about the size of PgAioResult, as it is embedded in
+ * every PgAioHandle, as well as every PgAioReturn. Currently we assume we can
+ * fit it into one 8 byte value, restricting the space for per-callback error
+ * data to PGAIO_RESULT_ERROR_BITS.
*/
+#define PGAIO_RESULT_ID_BITS 6
+#define PGAIO_RESULT_STATUS_BITS 3
+#define PGAIO_RESULT_ERROR_BITS 23
typedef struct PgAioResult
{
/*
* This is of type PgAioHandleCallbackID, but can't use a bitfield of an
* enum, because some compilers treat enums as signed.
*/
- uint32 id:8;
+ uint32 id:PGAIO_RESULT_ID_BITS;
/* of type PgAioResultStatus, see above */
- uint32 status:2;
+ uint32 status:PGAIO_RESULT_STATUS_BITS;
/* meaning defined by callback->error */
- uint32 error_data:22;
+ uint32 error_data:PGAIO_RESULT_ERROR_BITS;
int32 result;
} PgAioResult;
+StaticAssertDecl(PGAIO_RESULT_ID_BITS +
+ PGAIO_RESULT_STATUS_BITS +
+ PGAIO_RESULT_ERROR_BITS == 32,
+ "PgAioResult bits divied incorrectly");
+StaticAssertDecl(sizeof(PgAioResult) == 8,
+ "PgAioResult has unexpected size");
+
/*
* Combination of PgAioResult with minimal metadata about the IO.
*
diff --git a/src/backend/storage/aio/aio.c b/src/backend/storage/aio/aio.c
index 91e76113412..e3ed087e8a2 100644
--- a/src/backend/storage/aio/aio.c
+++ b/src/backend/storage/aio/aio.c
@@ -839,6 +839,8 @@ pgaio_result_status_string(PgAioResultStatus rs)
return "UNKNOWN";
case PGAIO_RS_OK:
return "OK";
+ case PGAIO_RS_WARNING:
+ return "WARNING";
case PGAIO_RS_PARTIAL:
return "PARTIAL";
case PGAIO_RS_ERROR:
--
2.48.1.76.g4e746b1a31.dirty
[text/x-diff] v2.14-0008-pgstat-Allow-checksum-errors-to-be-reported-in.patch (5.6K, ../../d4ff3qxudxn2ze3zk2xwhds5ysezfei3uqeuimymj624t5chsd@sgr2h3rmln5t/9-v2.14-0008-pgstat-Allow-checksum-errors-to-be-reported-in.patch)
download | inline diff:
From f909fdaa534fd9d4a0f940752197f4a1b360fa26 Mon Sep 17 00:00:00 2001
From: Andres Freund <andres@anarazel.de>
Date: Fri, 28 Mar 2025 12:31:34 -0400
Subject: [PATCH v2.14 08/29] pgstat: Allow checksum errors to be reported in
critical sections
For AIO we execute completion callbacks in critical sections (to ensure that
AIO can in the future be used for WAL, which in turn requires that we can call
completion callbacks in critical sections, to get the resources for WAL
io). To report checksum errors a backend now has to call
pgstat_prepare_report_checksum_failure(), before entering a critical section,
which guarantees the relevant pgstats entry is in shared memory.
Discussion: https://postgr.es/m/wkjj4p2rmkevutkwc6tewoovdqznj6c6nvjmvii4oo5wmbh5sr@retq7d6uqs4j
---
src/include/pgstat.h | 1 +
src/backend/backup/basebackup.c | 1 +
src/backend/catalog/storage.c | 1 +
src/backend/storage/buffer/bufmgr.c | 1 +
src/backend/utils/activity/pgstat_database.c | 51 ++++++++++++++++++--
5 files changed, 52 insertions(+), 3 deletions(-)
diff --git a/src/include/pgstat.h b/src/include/pgstat.h
index 9f3d13bf1ce..378f2f2c2ba 100644
--- a/src/include/pgstat.h
+++ b/src/include/pgstat.h
@@ -611,6 +611,7 @@ extern void pgstat_drop_database(Oid databaseid);
extern void pgstat_report_autovac(Oid dboid);
extern void pgstat_report_recovery_conflict(int reason);
extern void pgstat_report_deadlock(void);
+extern void pgstat_prepare_report_checksum_failure(Oid dboid);
extern void pgstat_report_checksum_failures_in_db(Oid dboid, int failurecount);
extern void pgstat_report_connect(Oid dboid);
extern void pgstat_update_parallel_workers_stats(PgStat_Counter workers_to_launch,
diff --git a/src/backend/backup/basebackup.c b/src/backend/backup/basebackup.c
index 3f8a3c55725..891637e3a44 100644
--- a/src/backend/backup/basebackup.c
+++ b/src/backend/backup/basebackup.c
@@ -1817,6 +1817,7 @@ sendFile(bbsink *sink, const char *readfilename, const char *tarfilename,
checksum_failures,
readfilename, checksum_failures)));
+ pgstat_prepare_report_checksum_failure(dboid);
pgstat_report_checksum_failures_in_db(dboid, checksum_failures);
}
diff --git a/src/backend/catalog/storage.c b/src/backend/catalog/storage.c
index cacf16c1cdb..2577f69cbde 100644
--- a/src/backend/catalog/storage.c
+++ b/src/backend/catalog/storage.c
@@ -524,6 +524,7 @@ RelationCopyStorage(SMgrRelation src, SMgrRelation dst,
{
RelFileLocatorBackend rloc = src->smgr_rlocator;
+ pgstat_prepare_report_checksum_failure(rloc.locator.dbOid);
pgstat_report_checksum_failures_in_db(rloc.locator.dbOid, 1);
}
diff --git a/src/backend/storage/buffer/bufmgr.c b/src/backend/storage/buffer/bufmgr.c
index 03317b49025..16b5b69efda 100644
--- a/src/backend/storage/buffer/bufmgr.c
+++ b/src/backend/storage/buffer/bufmgr.c
@@ -1590,6 +1590,7 @@ WaitReadBuffers(ReadBuffersOperation *operation)
{
RelFileLocatorBackend rloc = operation->smgr->smgr_rlocator;
+ pgstat_prepare_report_checksum_failure(rloc.locator.dbOid);
pgstat_report_checksum_failures_in_db(rloc.locator.dbOid, 1);
}
diff --git a/src/backend/utils/activity/pgstat_database.c b/src/backend/utils/activity/pgstat_database.c
index c0c02efd9d3..fbaf8364117 100644
--- a/src/backend/utils/activity/pgstat_database.c
+++ b/src/backend/utils/activity/pgstat_database.c
@@ -133,8 +133,34 @@ pgstat_report_deadlock(void)
dbent->deadlocks++;
}
+/*
+ * Allow this backend to later report checksum failures for dboid, even if in
+ * a critical section at the time of the report.
+ *
+ * Without this function having been called first, the backend might need to
+ * allocate an EntryRef or might need to map in DSM segments. Neither should
+ * happen in a critical section.
+ */
+void
+pgstat_prepare_report_checksum_failure(Oid dboid)
+{
+ Assert(!CritSectionCount);
+
+ /*
+ * Just need to ensure this backend has an entry ref for the database.
+ * That will allows us to report checksum failures without e.g. needing to
+ * map in DSM segments.
+ */
+ pgstat_get_entry_ref(PGSTAT_KIND_DATABASE, dboid, InvalidOid,
+ true, NULL);
+}
+
/*
* Report one or more checksum failures.
+ *
+ * To be allowed to report checksum failures in critical sections, we require
+ * pgstat_prepare_report_checksum_failure() to have been called before this
+ * function is called.
*/
void
pgstat_report_checksum_failures_in_db(Oid dboid, int failurecount)
@@ -147,10 +173,29 @@ pgstat_report_checksum_failures_in_db(Oid dboid, int failurecount)
/*
* Update the shared stats directly - checksum failures should never be
- * common enough for that to be a problem.
+ * common enough for that to be a problem. Note that we pass create=false
+ * here, as we want to be sure to not require memory allocations, so this
+ * can be called in critical sections.
*/
- entry_ref =
- pgstat_get_entry_ref_locked(PGSTAT_KIND_DATABASE, dboid, InvalidOid, false);
+ entry_ref = pgstat_get_entry_ref(PGSTAT_KIND_DATABASE, dboid, InvalidOid,
+ false, NULL);
+
+ /*
+ * Should always have been created by
+ * pgstat_prepare_report_checksum_failure().
+ *
+ * When not using assertions, we don't want to crash should something have
+ * gone wrong, so just return.
+ */
+ Assert(entry_ref);
+ if (!entry_ref)
+ {
+ elog(WARNING, "could not report %d conflicts for DB %u",
+ failurecount, dboid);
+ return;
+ }
+
+ pgstat_lock_entry(entry_ref, false);
sharedent = (PgStatShared_Database *) entry_ref->shared_stats;
sharedent->stats.checksum_failures += failurecount;
--
2.48.1.76.g4e746b1a31.dirty
[text/x-diff] v2.14-0009-Add-errhint_internal.patch (2.6K, ../../d4ff3qxudxn2ze3zk2xwhds5ysezfei3uqeuimymj624t5chsd@sgr2h3rmln5t/10-v2.14-0009-Add-errhint_internal.patch)
download | inline diff:
From 5166b45d2f8dc7b3d0b144af8ce414b3e9f79b22 Mon Sep 17 00:00:00 2001
From: Andres Freund <andres@anarazel.de>
Date: Sat, 29 Mar 2025 10:14:37 -0400
Subject: [PATCH v2.14 09/29] Add errhint_internal()
We have errmsg_internal(), errdetail_internal(), but not errhint_internal().
Sometimes it is useful to output a hint with already translated format
string (e.g. because there different messages depending on the condition). For
message/detail we do that with the _internal() variants, but we can't do that
with hint today. It's possible to work around that that by using something
like
str = psprintf(translated_format, args);
ereport(...
errhint("%s", str);
but that's not exactly pretty and makes it harder to avoid memory leaks.
Discussion: https://postgr.es/m/ym3dqpa4xcvoeknewcw63x77vnqdosbqcetjinb2zfoh65k55m@m4ozmwhr6lk6
---
src/include/utils/elog.h | 1 +
src/backend/utils/error/elog.c | 21 +++++++++++++++++++++
2 files changed, 22 insertions(+)
diff --git a/src/include/utils/elog.h b/src/include/utils/elog.h
index 855c147325b..a5313c5d2d5 100644
--- a/src/include/utils/elog.h
+++ b/src/include/utils/elog.h
@@ -195,6 +195,7 @@ extern int errdetail_plural(const char *fmt_singular, const char *fmt_plural,
unsigned long n,...) pg_attribute_printf(1, 4) pg_attribute_printf(2, 4);
extern int errhint(const char *fmt,...) pg_attribute_printf(1, 2);
+extern int errhint_internal(const char *fmt,...) pg_attribute_printf(1, 2);
extern int errhint_plural(const char *fmt_singular, const char *fmt_plural,
unsigned long n,...) pg_attribute_printf(1, 4) pg_attribute_printf(2, 4);
diff --git a/src/backend/utils/error/elog.c b/src/backend/utils/error/elog.c
index 97014c1a5a5..8a6b6905079 100644
--- a/src/backend/utils/error/elog.c
+++ b/src/backend/utils/error/elog.c
@@ -1330,6 +1330,27 @@ errhint(const char *fmt,...)
return 0; /* return value does not matter */
}
+/*
+ * errhint_internal --- add a hint error message text to the current error
+ *
+ * Non-translated version of errhint(), see also errmsg_internal().
+ */
+int
+errhint_internal(const char *fmt,...)
+{
+ ErrorData *edata = &errordata[errordata_stack_depth];
+ MemoryContext oldcontext;
+
+ recursion_depth++;
+ CHECK_STACK_DEPTH();
+ oldcontext = MemoryContextSwitchTo(edata->assoc_context);
+
+ EVALUATE_MESSAGE(edata->domain, hint, false, false);
+
+ MemoryContextSwitchTo(oldcontext);
+ recursion_depth--;
+ return 0; /* return value does not matter */
+}
/*
* errhint_plural --- add a hint error message text to the current error,
--
2.48.1.76.g4e746b1a31.dirty
[text/x-diff] v2.14-0010-bufmgr-Implement-AIO-read-support.patch (45.7K, ../../d4ff3qxudxn2ze3zk2xwhds5ysezfei3uqeuimymj624t5chsd@sgr2h3rmln5t/11-v2.14-0010-bufmgr-Implement-AIO-read-support.patch)
download | inline diff:
From 7b47410463e186edacd10b4a090a9bf70c7a3861 Mon Sep 17 00:00:00 2001
From: Andres Freund <andres@anarazel.de>
Date: Tue, 18 Mar 2025 14:40:06 -0400
Subject: [PATCH v2.14 10/29] bufmgr: Implement AIO read support
This commit implements the infrastructure to perform asynchronous reads into
the buffer pool.
To do so, it:
- Adds readv AIO callbacks for shared and local buffers
It may be worth calling out that shared buffer completions may be run in a
different backend than where the IO started.
- Adds an AIO wait reference to BufferDesc, to allow backends to wait for
in-progress asynchronous IOs
- Adapts StartBufferIO(), WaitIO(), TerminateBufferIO(), and their localbuf.c
equivalents, to be able to deal with AIO
- Moves the code to handle BM_PIN_COUNT_WAITER into a helper function, as it
now also needs to be called on IO completion
As of this commit, nothing issues AIO on shared buffers. A future commit will
update StartReadBuffers() to do so.
Buffer reads executed this infrastructure will report invalid page / checksum
errors / warnings differently than before:
In the error case the error message will cover all the blocks that were
included in the read, rather than just the reporting the first invalid
block. If more than one block is invalid, the error will include information
about the range of the read, the first invalid block and the number of invalid
pages, with a HINT towards the server log for per-block details.
For the warning case (i.e. zero_damaged_buffers) we would previously emit one
warning message for each buffer in a multi-block read. Now there is only a
single warning message for the entire read, again referring to the server log
for more details in case of multiple checksum failures within a single larger
read.
Reviewed-by: Noah Misch <noah@leadboat.com>
Reviewed-by: Melanie Plageman <melanieplageman@gmail.com>
Reviewed-by: Nazir Bilal Yavuz <byavuz81@gmail.com>
Discussion: https://postgr.es/m/uvrtrknj4kdytuboidbhwclo4gxhswwcpgadptsjvjqcluzmah%40brqs62irg4dt
Discussion: https://postgr.es/m/20210223100344.llw5an2aklengrmn@alap3.anarazel.de
Discussion: https://postgr.es/m/stj36ea6yyhoxtqkhpieia2z4krnam7qyetc57rfezgk4zgapf@gcnactj4z56m
---
src/include/storage/aio.h | 6 +-
src/include/storage/buf_internals.h | 7 +-
src/include/storage/bufmgr.h | 6 +
src/include/storage/bufpage.h | 1 +
src/backend/storage/aio/aio_callback.c | 5 +
src/backend/storage/buffer/README | 9 +-
src/backend/storage/buffer/buf_init.c | 3 +
src/backend/storage/buffer/bufmgr.c | 831 +++++++++++++++++++++++--
src/backend/storage/buffer/localbuf.c | 61 +-
src/backend/storage/page/bufpage.c | 8 +-
10 files changed, 873 insertions(+), 64 deletions(-)
diff --git a/src/include/storage/aio.h b/src/include/storage/aio.h
index d819f9e37ca..bfe0d93683b 100644
--- a/src/include/storage/aio.h
+++ b/src/include/storage/aio.h
@@ -195,9 +195,13 @@ typedef enum PgAioHandleCallbackID
PGAIO_HCB_MD_READV,
PGAIO_HCB_MD_WRITEV,
+
+ PGAIO_HCB_SHARED_BUFFER_READV,
+
+ PGAIO_HCB_LOCAL_BUFFER_READV,
} PgAioHandleCallbackID;
-#define PGAIO_HCB_MAX PGAIO_HCB_MD_WRITEV
+#define PGAIO_HCB_MAX PGAIO_HCB_LOCAL_BUFFER_READV
StaticAssertDecl(PGAIO_HCB_MAX <= (1 << PGAIO_RESULT_ID_BITS),
"PGAIO_HCB_MAX is too big for PGAIO_RESULT_ID_BITS");
diff --git a/src/include/storage/buf_internals.h b/src/include/storage/buf_internals.h
index 9327f60c44c..72b36a4af26 100644
--- a/src/include/storage/buf_internals.h
+++ b/src/include/storage/buf_internals.h
@@ -17,6 +17,7 @@
#include "pgstat.h"
#include "port/atomics.h"
+#include "storage/aio_types.h"
#include "storage/buf.h"
#include "storage/bufmgr.h"
#include "storage/condition_variable.h"
@@ -264,6 +265,8 @@ typedef struct BufferDesc
int wait_backend_pgprocno; /* backend of pin-count waiter */
int freeNext; /* link in freelist chain */
+
+ PgAioWaitRef io_wref; /* set iff AIO is in progress */
LWLock content_lock; /* to lock access to buffer contents */
} BufferDesc;
@@ -472,8 +475,8 @@ extern BlockNumber ExtendBufferedRelLocal(BufferManagerRelation bmr,
uint32 *extended_by);
extern void MarkLocalBufferDirty(Buffer buffer);
extern void TerminateLocalBufferIO(BufferDesc *bufHdr, bool clear_dirty,
- uint32 set_flag_bits);
-extern bool StartLocalBufferIO(BufferDesc *bufHdr, bool forInput);
+ uint32 set_flag_bits, bool release_aio);
+extern bool StartLocalBufferIO(BufferDesc *bufHdr, bool forInput, bool nowait);
extern void FlushLocalBuffer(BufferDesc *bufHdr, SMgrRelation reln);
extern void DropRelationLocalBuffers(RelFileLocator rlocator,
ForkNumber forkNum,
diff --git a/src/include/storage/bufmgr.h b/src/include/storage/bufmgr.h
index 538b890a51d..11f8508a90b 100644
--- a/src/include/storage/bufmgr.h
+++ b/src/include/storage/bufmgr.h
@@ -15,6 +15,7 @@
#define BUFMGR_H
#include "port/pg_iovec.h"
+#include "storage/aio_types.h"
#include "storage/block.h"
#include "storage/buf.h"
#include "storage/bufpage.h"
@@ -111,6 +112,8 @@ typedef struct BufferManagerRelation
#define READ_BUFFERS_ZERO_ON_ERROR (1 << 0)
/* Call smgrprefetch() if I/O necessary. */
#define READ_BUFFERS_ISSUE_ADVICE (1 << 1)
+/* Don't treat page as invalid due to checksum failures. */
+#define READ_BUFFERS_IGNORE_CHECKSUM_FAILURES (1 << 2)
struct ReadBuffersOperation
{
@@ -170,6 +173,9 @@ extern PGDLLIMPORT int checkpoint_flush_after;
extern PGDLLIMPORT int backend_flush_after;
extern PGDLLIMPORT int bgwriter_flush_after;
+extern const PgAioHandleCallbacks aio_shared_buffer_readv_cb;
+extern const PgAioHandleCallbacks aio_local_buffer_readv_cb;
+
/* in buf_init.c */
extern PGDLLIMPORT char *BufferBlocks;
diff --git a/src/include/storage/bufpage.h b/src/include/storage/bufpage.h
index b943db707db..7dbdfc564ed 100644
--- a/src/include/storage/bufpage.h
+++ b/src/include/storage/bufpage.h
@@ -467,6 +467,7 @@ do { \
/* flags for PageIsVerified() */
#define PIV_LOG_WARNING (1 << 0)
+#define PIV_LOG_LOG (1 << 1)
#define PageAddItem(page, item, size, offsetNumber, overwrite, is_heap) \
PageAddItemExtended(page, item, size, offsetNumber, \
diff --git a/src/backend/storage/aio/aio_callback.c b/src/backend/storage/aio/aio_callback.c
index 53db6e194af..c0063d6950e 100644
--- a/src/backend/storage/aio/aio_callback.c
+++ b/src/backend/storage/aio/aio_callback.c
@@ -18,6 +18,7 @@
#include "miscadmin.h"
#include "storage/aio.h"
#include "storage/aio_internal.h"
+#include "storage/bufmgr.h"
#include "storage/md.h"
@@ -40,6 +41,10 @@ static const PgAioHandleCallbacksEntry aio_handle_cbs[] = {
CALLBACK_ENTRY(PGAIO_HCB_INVALID, aio_invalid_cb),
CALLBACK_ENTRY(PGAIO_HCB_MD_READV, aio_md_readv_cb),
+
+ CALLBACK_ENTRY(PGAIO_HCB_SHARED_BUFFER_READV, aio_shared_buffer_readv_cb),
+
+ CALLBACK_ENTRY(PGAIO_HCB_LOCAL_BUFFER_READV, aio_local_buffer_readv_cb),
#undef CALLBACK_ENTRY
};
diff --git a/src/backend/storage/buffer/README b/src/backend/storage/buffer/README
index 011af7aff3e..a182fcd660c 100644
--- a/src/backend/storage/buffer/README
+++ b/src/backend/storage/buffer/README
@@ -147,9 +147,12 @@ in the buffer. It is used per the rules above.
* The BM_IO_IN_PROGRESS flag acts as a kind of lock, used to wait for I/O on a
buffer to complete (and in releases before 14, it was accompanied by a
-per-buffer LWLock). The process doing a read or write sets the flag for the
-duration, and processes that need to wait for it to be cleared sleep on a
-condition variable.
+per-buffer LWLock). The process starting a read or write sets the flag. When
+the I/O is completed, be it by the process that initiated the I/O or by
+another process, the flag is removed and the Buffer's condition variable is
+signalled. Processes that need to wait for the I/O to complete can wait for
+asynchronous I/O by using BufferDesc->io_wref and for BM_IO_IN_PROGRESS to be
+unset by sleeping on the buffer's condition variable.
Normal Buffer Replacement Strategy
diff --git a/src/backend/storage/buffer/buf_init.c b/src/backend/storage/buffer/buf_init.c
index ed1f8e03190..ed1dc488a42 100644
--- a/src/backend/storage/buffer/buf_init.c
+++ b/src/backend/storage/buffer/buf_init.c
@@ -14,6 +14,7 @@
*/
#include "postgres.h"
+#include "storage/aio.h"
#include "storage/buf_internals.h"
#include "storage/bufmgr.h"
@@ -125,6 +126,8 @@ BufferManagerShmemInit(void)
buf->buf_id = i;
+ pgaio_wref_clear(&buf->io_wref);
+
/*
* Initially link all the buffers together as unused. Subsequent
* management of this list is done by freelist.c.
diff --git a/src/backend/storage/buffer/bufmgr.c b/src/backend/storage/buffer/bufmgr.c
index 16b5b69efda..ca89d9345f3 100644
--- a/src/backend/storage/buffer/bufmgr.c
+++ b/src/backend/storage/buffer/bufmgr.c
@@ -48,6 +48,7 @@
#include "pg_trace.h"
#include "pgstat.h"
#include "postmaster/bgwriter.h"
+#include "storage/aio.h"
#include "storage/buf_internals.h"
#include "storage/bufmgr.h"
#include "storage/fd.h"
@@ -519,7 +520,8 @@ static int SyncOneBuffer(int buf_id, bool skip_recently_used,
static void WaitIO(BufferDesc *buf);
static bool StartBufferIO(BufferDesc *buf, bool forInput, bool nowait);
static void TerminateBufferIO(BufferDesc *buf, bool clear_dirty,
- uint32 set_flag_bits, bool forget_owner);
+ uint32 set_flag_bits, bool forget_owner,
+ bool release_aio);
static void AbortBufferIO(Buffer buffer);
static void shared_buffer_write_error_callback(void *arg);
static void local_buffer_write_error_callback(void *arg);
@@ -1041,7 +1043,7 @@ ZeroAndLockBuffer(Buffer buffer, ReadBufferMode mode, bool already_valid)
{
/* Simple case for non-shared buffers. */
bufHdr = GetLocalBufferDescriptor(-buffer - 1);
- need_to_zero = StartLocalBufferIO(bufHdr, true);
+ need_to_zero = StartLocalBufferIO(bufHdr, true, false);
}
else
{
@@ -1077,9 +1079,9 @@ ZeroAndLockBuffer(Buffer buffer, ReadBufferMode mode, bool already_valid)
/* Set BM_VALID, terminate IO, and wake up any waiters */
if (isLocalBuf)
- TerminateLocalBufferIO(bufHdr, false, BM_VALID);
+ TerminateLocalBufferIO(bufHdr, false, BM_VALID, false);
else
- TerminateBufferIO(bufHdr, false, BM_VALID, true);
+ TerminateBufferIO(bufHdr, false, BM_VALID, true, false);
}
else if (!isLocalBuf)
{
@@ -1454,7 +1456,8 @@ static inline bool
WaitReadBuffersCanStartIO(Buffer buffer, bool nowait)
{
if (BufferIsLocal(buffer))
- return StartLocalBufferIO(GetLocalBufferDescriptor(-buffer - 1), true);
+ return StartLocalBufferIO(GetLocalBufferDescriptor(-buffer - 1),
+ true, nowait);
else
return StartBufferIO(GetBufferDescriptor(buffer - 1), true, nowait);
}
@@ -1615,9 +1618,9 @@ WaitReadBuffers(ReadBuffersOperation *operation)
/* Set BM_VALID, terminate IO, and wake up any waiters */
if (persistence == RELPERSISTENCE_TEMP)
- TerminateLocalBufferIO(bufHdr, false, BM_VALID);
+ TerminateLocalBufferIO(bufHdr, false, BM_VALID, false);
else
- TerminateBufferIO(bufHdr, false, BM_VALID, true);
+ TerminateBufferIO(bufHdr, false, BM_VALID, true, false);
/* Report I/Os as completing individually. */
TRACE_POSTGRESQL_BUFFER_READ_DONE(forknum, io_first_block + j,
@@ -1879,13 +1882,14 @@ retry:
}
/*
- * We assume the only reason for it to be pinned is that someone else is
- * flushing the page out. Wait for them to finish. (This could be an
- * infinite loop if the refcount is messed up... it would be nice to time
- * out after awhile, but there seems no way to be sure how many loops may
- * be needed. Note that if the other guy has pinned the buffer but not
- * yet done StartBufferIO, WaitIO will fall through and we'll effectively
- * be busy-looping here.)
+ * We assume the reason for it to be pinned is that either we were
+ * asynchronously reading the page in before erroring out or someone else
+ * is flushing the page out. Wait for the IO to finish. (This could be
+ * an infinite loop if the refcount is messed up... it would be nice to
+ * time out after awhile, but there seems no way to be sure how many loops
+ * may be needed. Note that if the other guy has pinned the buffer but
+ * not yet done StartBufferIO, WaitIO will fall through and we'll
+ * effectively be busy-looping here.)
*/
if (BUF_STATE_GET_REFCOUNT(buf_state) != 0)
{
@@ -2525,7 +2529,7 @@ ExtendBufferedRelShared(BufferManagerRelation bmr,
if (lock)
LWLockAcquire(BufferDescriptorGetContentLock(buf_hdr), LW_EXCLUSIVE);
- TerminateBufferIO(buf_hdr, false, BM_VALID, true);
+ TerminateBufferIO(buf_hdr, false, BM_VALID, true, false);
}
pgBufferUsage.shared_blks_written += extend_by;
@@ -2871,6 +2875,44 @@ PinBuffer_Locked(BufferDesc *buf)
ResourceOwnerRememberBuffer(CurrentResourceOwner, b);
}
+/*
+ * Support for waking up another backend that is waiting for the cleanup lock
+ * to be released using BM_PIN_COUNT_WAITER.
+ *
+ * See LockBufferForCleanup().
+ *
+ * Expected to be called just after releasing a buffer pin (in a BufferDesc,
+ * not just reducing the backend-local pincount for the buffer).
+ */
+static void
+WakePinCountWaiter(BufferDesc *buf)
+{
+ /*
+ * Acquire the buffer header lock, re-check that there's a waiter. Another
+ * backend could have unpinned this buffer, and already woken up the
+ * waiter.
+ *
+ * There's no danger of the buffer being replaced after we unpinned it
+ * above, as it's pinned by the waiter. The waiter removes
+ * BM_PIN_COUNT_WAITER if it stops waiting for a reason other than this
+ * backend waking it up.
+ */
+ uint32 buf_state = LockBufHdr(buf);
+
+ if ((buf_state & BM_PIN_COUNT_WAITER) &&
+ BUF_STATE_GET_REFCOUNT(buf_state) == 1)
+ {
+ /* we just released the last pin other than the waiter's */
+ int wait_backend_pgprocno = buf->wait_backend_pgprocno;
+
+ buf_state &= ~BM_PIN_COUNT_WAITER;
+ UnlockBufHdr(buf, buf_state);
+ ProcSendSignal(wait_backend_pgprocno);
+ }
+ else
+ UnlockBufHdr(buf, buf_state);
+}
+
/*
* UnpinBuffer -- make buffer available for replacement.
*
@@ -2939,29 +2981,8 @@ UnpinBufferNoOwner(BufferDesc *buf)
/* Support LockBufferForCleanup() */
if (buf_state & BM_PIN_COUNT_WAITER)
- {
- /*
- * Acquire the buffer header lock, re-check that there's a waiter.
- * Another backend could have unpinned this buffer, and already
- * woken up the waiter. There's no danger of the buffer being
- * replaced after we unpinned it above, as it's pinned by the
- * waiter.
- */
- buf_state = LockBufHdr(buf);
+ WakePinCountWaiter(buf);
- if ((buf_state & BM_PIN_COUNT_WAITER) &&
- BUF_STATE_GET_REFCOUNT(buf_state) == 1)
- {
- /* we just released the last pin other than the waiter's */
- int wait_backend_pgprocno = buf->wait_backend_pgprocno;
-
- buf_state &= ~BM_PIN_COUNT_WAITER;
- UnlockBufHdr(buf, buf_state);
- ProcSendSignal(wait_backend_pgprocno);
- }
- else
- UnlockBufHdr(buf, buf_state);
- }
ForgetPrivateRefCountEntry(ref);
}
}
@@ -3982,7 +4003,7 @@ FlushBuffer(BufferDesc *buf, SMgrRelation reln, IOObject io_object,
* Mark the buffer as clean (unless BM_JUST_DIRTIED has become set) and
* end the BM_IO_IN_PROGRESS state.
*/
- TerminateBufferIO(buf, true, 0, true);
+ TerminateBufferIO(buf, true, 0, true, false);
TRACE_POSTGRESQL_BUFFER_FLUSH_DONE(BufTagGetForkNum(&buf->tag),
buf->tag.blockNum,
@@ -5252,6 +5273,12 @@ LockBufferForCleanup(Buffer buffer)
CheckBufferIsPinnedOnce(buffer);
+ /*
+ * We do not yet need to be worried about in-progress AIOs holding a pin,
+ * as we only support doing reads via AIO and this function can only be
+ * called once the buffer is valid (i.e. no read can be in flight).
+ */
+
/* Nobody else to wait for */
if (BufferIsLocal(buffer))
return;
@@ -5409,6 +5436,8 @@ ConditionalLockBufferForCleanup(Buffer buffer)
Assert(BufferIsValid(buffer));
+ /* see AIO related comment in LockBufferForCleanup() */
+
if (BufferIsLocal(buffer))
{
refcount = LocalRefCount[-buffer - 1];
@@ -5464,6 +5493,8 @@ IsBufferCleanupOK(Buffer buffer)
Assert(BufferIsValid(buffer));
+ /* see AIO related comment in LockBufferForCleanup() */
+
if (BufferIsLocal(buffer))
{
/* There should be exactly one pin */
@@ -5516,6 +5547,7 @@ WaitIO(BufferDesc *buf)
for (;;)
{
uint32 buf_state;
+ PgAioWaitRef iow;
/*
* It may not be necessary to acquire the spinlock to check the flag
@@ -5523,10 +5555,40 @@ WaitIO(BufferDesc *buf)
* play it safe.
*/
buf_state = LockBufHdr(buf);
+
+ /*
+ * Copy the wait reference while holding the spinlock. This protects
+ * against a concurrent TerminateBufferIO() in another backend from
+ * clearing the wref while it's being read.
+ */
+ iow = buf->io_wref;
UnlockBufHdr(buf, buf_state);
+ /* no IO in progress, we don't need to wait */
if (!(buf_state & BM_IO_IN_PROGRESS))
break;
+
+ /*
+ * The buffer has asynchronous IO in progress, wait for it to
+ * complete.
+ */
+ if (pgaio_wref_valid(&iow))
+ {
+ pgaio_wref_wait(&iow);
+
+ /*
+ * The AIO subsystem internally uses condition variables and thus
+ * might remove this backend from the BufferDesc's CV. While that
+ * wouldn't cause a correctness issue (the first CV sleep just
+ * immediately returns if not already registered), it seems worth
+ * avoiding unnecessary loop iterations, given that we take care
+ * to do so at the start of the function.
+ */
+ ConditionVariablePrepareToSleep(cv);
+ continue;
+ }
+
+ /* wait on BufferDesc->cv, e.g. for concurrent synchronous IO */
ConditionVariableSleep(cv, WAIT_EVENT_BUFFER_IO);
}
ConditionVariableCancelSleep();
@@ -5535,13 +5597,12 @@ WaitIO(BufferDesc *buf)
/*
* StartBufferIO: begin I/O on this buffer
* (Assumptions)
- * My process is executing no IO
+ * My process is executing no IO on this buffer
* The buffer is Pinned
*
- * In some scenarios there are race conditions in which multiple backends
- * could attempt the same I/O operation concurrently. If someone else
- * has already started I/O on this buffer then we will block on the
- * I/O condition variable until he's done.
+ * In some scenarios multiple backends could attempt the same I/O operation
+ * concurrently. If someone else has already started I/O on this buffer then
+ * we will wait for completion of the IO using WaitIO().
*
* Input operations are only attempted on buffers that are not BM_VALID,
* and output operations only on buffers that are BM_VALID and BM_DIRTY,
@@ -5577,9 +5638,9 @@ StartBufferIO(BufferDesc *buf, bool forInput, bool nowait)
/* Once we get here, there is definitely no I/O active on this buffer */
+ /* Check if someone else already did the I/O */
if (forInput ? (buf_state & BM_VALID) : !(buf_state & BM_DIRTY))
{
- /* someone else already did the I/O */
UnlockBufHdr(buf, buf_state);
return false;
}
@@ -5615,7 +5676,7 @@ StartBufferIO(BufferDesc *buf, bool forInput, bool nowait)
*/
static void
TerminateBufferIO(BufferDesc *buf, bool clear_dirty, uint32 set_flag_bits,
- bool forget_owner)
+ bool forget_owner, bool release_aio)
{
uint32 buf_state;
@@ -5630,6 +5691,14 @@ TerminateBufferIO(BufferDesc *buf, bool clear_dirty, uint32 set_flag_bits,
if (clear_dirty && !(buf_state & BM_JUST_DIRTIED))
buf_state &= ~(BM_DIRTY | BM_CHECKPOINT_NEEDED);
+ if (release_aio)
+ {
+ /* release ownership by the AIO subsystem */
+ Assert(BUF_STATE_GET_REFCOUNT(buf_state) > 0);
+ buf_state -= BUF_REFCOUNT_ONE;
+ pgaio_wref_clear(&buf->io_wref);
+ }
+
buf_state |= set_flag_bits;
UnlockBufHdr(buf, buf_state);
@@ -5638,6 +5707,17 @@ TerminateBufferIO(BufferDesc *buf, bool clear_dirty, uint32 set_flag_bits,
BufferDescriptorGetBuffer(buf));
ConditionVariableBroadcast(BufferDescriptorGetIOCV(buf));
+
+ /*
+ * Support LockBufferForCleanup()
+ *
+ * We may have just released the last pin other than the waiter's. In most
+ * cases, this backend holds another pin on the buffer. But, if, for
+ * example, this backend is completing an IO issued by another backend, it
+ * may be time to wake the waiter.
+ */
+ if (release_aio && (buf_state & BM_PIN_COUNT_WAITER))
+ WakePinCountWaiter(buf);
}
/*
@@ -5686,7 +5766,7 @@ AbortBufferIO(Buffer buffer)
}
}
- TerminateBufferIO(buf_hdr, false, BM_IO_ERROR, false);
+ TerminateBufferIO(buf_hdr, false, BM_IO_ERROR, false, false);
}
/*
@@ -6137,3 +6217,662 @@ EvictUnpinnedBuffer(Buffer buf)
return result;
}
+
+/*
+ * Generic implementation of the AIO handle staging callback for readv/writev
+ * on local/shared buffers.
+ *
+ * Each readv/writev can target multiple buffers. The buffers have already
+ * been registered with the IO handle.
+ *
+ * To make the IO ready for execution ("staging"), we need to ensure that the
+ * targeted buffers are in an appropriate state while the IO is ongoing. For
+ * that the AIO subsystem needs to have its own buffer pin, otherwise an error
+ * in this backend could lead to this backend's buffer pin being released as
+ * part of error handling, which in turn could lead to the buffer being
+ * replaced while IO is ongoing.
+ */
+static pg_attribute_always_inline void
+buffer_stage_common(PgAioHandle *ioh, bool is_write, bool is_temp)
+{
+ uint64 *io_data;
+ uint8 handle_data_len;
+ PgAioWaitRef io_ref;
+ BufferTag first PG_USED_FOR_ASSERTS_ONLY = {0};
+
+ io_data = pgaio_io_get_handle_data(ioh, &handle_data_len);
+
+ pgaio_io_get_wref(ioh, &io_ref);
+
+ /* iterate over all buffers affected by the vectored readv/writev */
+ for (int i = 0; i < handle_data_len; i++)
+ {
+ Buffer buffer = (Buffer) io_data[i];
+ BufferDesc *buf_hdr = is_temp ?
+ GetLocalBufferDescriptor(-buffer - 1)
+ : GetBufferDescriptor(buffer - 1);
+ uint32 buf_state;
+
+ /*
+ * Check that all the buffers are actually ones that could conceivably
+ * be done in one IO, i.e. are sequential. This is the last
+ * buffer-aware code before IO is actually executed and confusion
+ * about which buffers are targeted by IO can be hard to debug, making
+ * it worth doing extra-paranoid checks.
+ */
+ if (i == 0)
+ first = buf_hdr->tag;
+ else
+ {
+ Assert(buf_hdr->tag.relNumber == first.relNumber);
+ Assert(buf_hdr->tag.blockNum == first.blockNum + i);
+ }
+
+ if (is_temp)
+ buf_state = pg_atomic_read_u32(&buf_hdr->state);
+ else
+ buf_state = LockBufHdr(buf_hdr);
+
+ /* verify the buffer is in the expected state */
+ Assert(buf_state & BM_TAG_VALID);
+ if (is_write)
+ {
+ Assert(buf_state & BM_VALID);
+ Assert(buf_state & BM_DIRTY);
+ }
+ else
+ {
+ Assert(!(buf_state & BM_VALID));
+ Assert(!(buf_state & BM_DIRTY));
+ }
+
+ /* temp buffers don't use BM_IO_IN_PROGRESS */
+ if (!is_temp)
+ Assert(buf_state & BM_IO_IN_PROGRESS);
+
+ Assert(BUF_STATE_GET_REFCOUNT(buf_state) >= 1);
+
+ /*
+ * Reflect that the buffer is now owned by the AIO subsystem.
+ *
+ * For local buffers: This can't be done just via LocalRefCount, as
+ * one might initially think, as this backend could error out while
+ * AIO is still in progress, releasing all the pins by the backend
+ * itself.
+ *
+ * This pin is released again in TerminateBufferIO().
+ */
+ buf_state += BUF_REFCOUNT_ONE;
+ buf_hdr->io_wref = io_ref;
+
+ if (is_temp)
+ pg_atomic_unlocked_write_u32(&buf_hdr->state, buf_state);
+ else
+ UnlockBufHdr(buf_hdr, buf_state);
+
+ /*
+ * Ensure the content lock that prevents buffer modifications while
+ * the buffer is being written out is not released early due to an
+ * error.
+ */
+ if (is_write && !is_temp)
+ {
+ LWLock *content_lock;
+
+ content_lock = BufferDescriptorGetContentLock(buf_hdr);
+
+ Assert(LWLockHeldByMe(content_lock));
+
+ /*
+ * Lock is now owned by AIO subsystem.
+ */
+ LWLockDisown(content_lock);
+ }
+
+ /*
+ * Stop tracking this buffer via the resowner - the AIO system now
+ * keeps track.
+ */
+ if (!is_temp)
+ ResourceOwnerForgetBufferIO(CurrentResourceOwner, buffer);
+ }
+}
+
+/*
+ * Decode readv errors as encoded by buffer_readv_encode_error().
+ */
+static inline void
+buffer_readv_decode_error(PgAioResult result,
+ bool *zeroed_any,
+ bool *ignored_any,
+ uint8 *zeroed_or_error_count,
+ uint8 *checkfail_count,
+ uint8 *first_off)
+{
+ uint32 rem_error = result.error_data;
+
+ *zeroed_any = rem_error & 1;
+ rem_error >>= 1;
+
+ *ignored_any = rem_error & 1;
+ rem_error >>= 1;
+
+ *zeroed_or_error_count = rem_error & ((1 << 7) - 1);
+ rem_error >>= 7;
+
+ *checkfail_count = rem_error & ((1 << 7) - 1);
+ rem_error >>= 7;
+
+ *first_off = rem_error & ((1 << 7) - 1);
+ rem_error >>= 7;
+}
+
+/*
+ * Helper to encode errors for buffer_readv_complete()
+ *
+ * Errors are encoded as follows:
+ * - bit 0 indicates whether any page was zeroed (1) or not (0)
+ * - bit 1 indicates whether any checksum failure was ignored (1) or not (0)
+ * - next 7 bits indicate the number of errored or zeroed pages
+ * - next 7 bits indicate the number of checksum failures
+ * - next 7 bits indicate the first offset of the first page
+ * that was errored or zerored or, if no errors/zeroes, the first ignored
+ * checksum
+ */
+static inline void
+buffer_readv_encode_error(PgAioResult *result,
+ bool is_temp,
+ bool zeroed_any,
+ bool ignored_any,
+ uint8 error_count,
+ uint8 zeroed_count,
+ uint8 checkfail_count,
+ uint8 first_error_off,
+ uint8 first_zeroed_off,
+ uint8 first_ignored_off)
+{
+
+ uint8 shift = 0;
+ uint8 zeroed_or_error_count =
+ error_count > 0 ? error_count : zeroed_count;
+ uint8 first_off;
+
+ StaticAssertStmt(PG_IOV_MAX <= 1 << 7,
+ "PG_IOV_MAX is bigger than reserved space for error data");
+ StaticAssertStmt((1 + 7 + 7 + 7) <= PGAIO_RESULT_ERROR_BITS,
+ "PGAIO_RESULT_ERROR_BITS is insufficient for buffer_readv");
+
+ /*
+ * We only have space to encode one offset - but luckily that's good
+ * enough. If there is an error, the error is the integeresting offset,
+ * same with a zeroed buffer vs an ignored buffer.
+ */
+ if (error_count > 0)
+ first_off = first_error_off;
+ else if (zeroed_count > 0)
+ first_off = first_zeroed_off;
+ else
+ first_off = first_ignored_off;
+
+ Assert(!zeroed_any || error_count == 0);
+
+ result->error_data = 0;
+
+ result->error_data |= zeroed_any << shift;
+ shift += 1;
+
+ result->error_data |= ignored_any << shift;
+ shift += 1;
+
+ result->error_data |= ((uint32) zeroed_or_error_count) << shift;
+ shift += 7;
+
+ result->error_data |= ((uint32) checkfail_count) << shift;
+ shift += 7;
+
+ result->error_data |= ((uint32) first_off) << shift;
+ shift += 7;
+
+ result->id = is_temp ? PGAIO_HCB_LOCAL_BUFFER_READV :
+ PGAIO_HCB_SHARED_BUFFER_READV;
+
+ if (error_count > 0)
+ result->status = PGAIO_RS_ERROR;
+ else
+ result->status = PGAIO_RS_WARNING;
+
+ /*
+ * The encoding is complicated enough to warrant cross-checking it against
+ * the decode function.
+ */
+#ifdef USE_ASSERT_CHECKING
+ {
+ bool zeroed_any_2,
+ ignored_any_2;
+ uint8 zeroed_or_error_count_2,
+ checkfail_count_2,
+ first_off_2;
+
+ buffer_readv_decode_error(*result,
+ &zeroed_any_2, &ignored_any_2,
+ &zeroed_or_error_count_2,
+ &checkfail_count_2,
+ &first_off_2);
+ Assert(zeroed_any == zeroed_any_2);
+ Assert(ignored_any == ignored_any_2);
+ Assert(zeroed_or_error_count == zeroed_or_error_count_2);
+ Assert(checkfail_count == checkfail_count_2);
+ Assert(first_off == first_off_2);
+ }
+#endif
+}
+
+/*
+ * Helper for AIO readv completion callbacks, supporting both shared and temp
+ * buffers. Gets called once for each buffer in a multi-page read.
+ */
+static pg_attribute_always_inline void
+buffer_readv_complete_one(PgAioTargetData *td, uint8 buf_off, Buffer buffer,
+ uint8 flags, bool failed, bool is_temp,
+ bool *buffer_invalid,
+ bool *failed_checksum,
+ bool *ignored_checksum,
+ bool *zeroed_buffer)
+{
+ BufferDesc *buf_hdr = is_temp ?
+ GetLocalBufferDescriptor(-buffer - 1)
+ : GetBufferDescriptor(buffer - 1);
+ BufferTag tag = buf_hdr->tag;
+ char *bufdata = BufferGetBlock(buffer);
+ uint32 set_flag_bits;
+
+ /* check that the buffer is in the expected state for a read */
+#ifdef USE_ASSERT_CHECKING
+ {
+ uint32 buf_state = pg_atomic_read_u32(&buf_hdr->state);
+
+ Assert(buf_state & BM_TAG_VALID);
+ Assert(!(buf_state & BM_VALID));
+ /* temp buffers don't use BM_IO_IN_PROGRESS */
+ if (!is_temp)
+ Assert(buf_state & BM_IO_IN_PROGRESS);
+ Assert(!(buf_state & BM_DIRTY));
+ }
+#endif
+
+ *buffer_invalid = false;
+ *failed_checksum = false;
+ *ignored_checksum = false;
+ *zeroed_buffer = false;
+
+ /*
+ * We ask PageIsVerified() to only log the message about checksum errors,
+ * as the completion might be run in any backend (or IO workers). We will
+ * report checksum errors in buffer_readv_report().
+ */
+ piv_flags = PIV_LOG_LOG;
+
+ /* the local zero_damaged_pages may differ from the issuer's */
+ if (flags & READ_BUFFERS_IGNORE_CHECKSUM_FAILURES)
+ piv_flags |= PIV_IGNORE_CHECKSUM_FAILURE;
+
+ /* Check for garbage data. */
+ if (!failed)
+ {
+ PgAioResult result_one;
+
+ if (!PageIsVerified((Page) bufdata, tag.blockNum, piv_flags,
+ failed_checksum))
+ {
+ if (flags & READ_BUFFERS_ZERO_ON_ERROR)
+ {
+ memset(bufdata, 0, BLCKSZ);
+ *zeroed_buffer = true;
+ }
+ else
+ {
+ *buffer_invalid = true;
+ /* mark buffer as having failed */
+ failed = true;
+ }
+ }
+ else if (*failed_checksum)
+ *ignored_checksum = true;
+
+ /*
+ * Immediately log a message about the invalid page, but only to the
+ * server log. The reason to do so immediately is that this may be
+ * executed in a different backend than the one that originated the
+ * request. The reason to do so immediately is that the originator
+ * might not process the query result immediately (because it is busy
+ * doing another part of query processing) or at all (e.g. if it was
+ * cancelled or errored out due to another IO also failing). The
+ * issuer of the IO will emit an ERROR or WARNING when processing the
+ * IO's results
+ *
+ * To avoid duplicating the code to emit these log messages, we reuse
+ * buffer_readv_report().
+ */
+ if (*buffer_invalid || *failed_checksum || *zeroed_buffer)
+ {
+ buffer_readv_encode_error(&result_one, is_temp,
+ *zeroed_buffer,
+ *ignored_checksum,
+ *buffer_invalid,
+ *zeroed_buffer ? 1 : 0,
+ *failed_checksum ? 1 : 0,
+ buf_off, buf_off, buf_off);
+ pgaio_result_report(result_one, td, LOG_SERVER_ONLY);
+ }
+ }
+
+
+ /* Terminate I/O and set BM_VALID. */
+ set_flag_bits = failed ? BM_IO_ERROR : BM_VALID;
+ if (is_temp)
+ TerminateLocalBufferIO(buf_hdr, false, set_flag_bits, true);
+ else
+ TerminateBufferIO(buf_hdr, false, set_flag_bits, false, true);
+
+ /*
+ * Call the BUFFER_READ_DONE tracepoint in the callback, even though the
+ * callback may not be executed in the same backend that called
+ * BUFFER_READ_START. The alternative would be to defer calling the
+ * tracepoint to a later point (e.g. the local completion callback for
+ * shared buffer reads), which seems even less helpful.
+ */
+ TRACE_POSTGRESQL_BUFFER_READ_DONE(tag.forkNum,
+ tag.blockNum,
+ tag.spcOid,
+ tag.dbOid,
+ tag.relNumber,
+ is_temp ? MyProcNumber : INVALID_PROC_NUMBER,
+ false);
+}
+
+/*
+ * Perform completion handling of a single AIO read. This read may cover
+ * multiple blocks / buffers.
+ *
+ * Shared between shared and local buffers, to reduce code duplication.
+ */
+static pg_attribute_always_inline PgAioResult
+buffer_readv_complete(PgAioHandle *ioh, PgAioResult prior_result,
+ uint8 cb_data, bool is_temp)
+{
+ PgAioResult result = prior_result;
+ PgAioTargetData *td = pgaio_io_get_target_data(ioh);
+ uint8 first_error_off = 0;
+ uint8 first_zeroed_off = 0;
+ uint8 first_ignored_off = 0;
+ uint8 error_count = 0;
+ uint8 zeroed_count = 0;
+ uint8 ignored_count = 0;
+ uint8 checkfail_count = 0;
+ uint64 *io_data;
+ uint8 handle_data_len;
+
+ if (is_temp)
+ {
+ Assert(td->smgr.is_temp);
+ Assert(pgaio_io_get_owner(ioh) == MyProcNumber);
+ }
+ else
+ Assert(!td->smgr.is_temp);
+
+ /*
+ * Iterate over all the buffers affected by this IO and call the
+ * per-buffer completion function for each buffer.
+ */
+ io_data = pgaio_io_get_handle_data(ioh, &handle_data_len);
+ for (uint8 buf_off = 0; buf_off < handle_data_len; buf_off++)
+ {
+ Buffer buf = io_data[buf_off];
+ bool failed;
+ bool failed_verification = false;
+ bool failed_checksum = false;
+ bool zeroed_buffer = false;
+ bool ignored_checksum = false;
+
+ Assert(BufferIsValid(buf));
+
+ /*
+ * If the entire I/O failed on a lower-level, each buffer needs to be
+ * marked as failed. In case of a partial read, the first few buffers
+ * may be ok.
+ */
+ failed =
+ prior_result.status == PGAIO_RS_ERROR
+ || prior_result.result <= buf_off;
+
+ buffer_readv_complete_one(td, buf_off, buf, cb_data, failed, is_temp,
+ &failed_verification,
+ &failed_checksum,
+ &ignored_checksum,
+ &zeroed_buffer);
+
+ /*
+ * Track information about the number of different kinds of error
+ * conditions across all pages, as there can be multiple pages failing
+ * verification as part of one IO.
+ */
+ if (failed_verification && !zeroed_buffer && error_count++ == 0)
+ first_error_off = buf_off;
+ if (zeroed_buffer && zeroed_count++ == 0)
+ first_zeroed_off = buf_off;
+ if (ignored_checksum && ignored_count++ == 0)
+ first_ignored_off = buf_off;
+ if (failed_checksum)
+ checkfail_count++;
+ }
+
+ /*
+ * If the smgr read succeeded [partially] and page verification failed for
+ * some of the pages, adjust the IO's result state appropriately.
+ */
+ if (prior_result.status != PGAIO_RS_ERROR &&
+ (error_count > 0 || ignored_count > 0 || zeroed_count > 0))
+ {
+ buffer_readv_encode_error(&result, is_temp,
+ zeroed_count > 0, ignored_count > 0,
+ error_count, zeroed_count, checkfail_count,
+ first_error_off, first_zeroed_off,
+ first_ignored_off);
+ pgaio_result_report(result, td, DEBUG1);
+ }
+
+ /*
+ * For shared relations this reporting is done in
+ * shared_buffer_readv_complete_local().
+ */
+ if (is_temp && checkfail_count > 0)
+ pgstat_report_checksum_failures_in_db(td->smgr.rlocator.dbOid,
+ checkfail_count);
+
+ return result;
+}
+
+/*
+ * AIO error reporting callback for aio_shared_buffer_readv_cb and
+ * aio_local_buffer_readv_cb.
+ *
+ * The error is encoded / decoded in buffer_readv_encode_error() /
+ * buffer_readv_decode_error().
+ */
+static void
+buffer_readv_report(PgAioResult result, const PgAioTargetData *td,
+ int elevel)
+{
+ int nblocks = td->smgr.nblocks;
+ BlockNumber first = td->smgr.blockNum;
+ BlockNumber last = first + nblocks - 1;
+ ProcNumber errProc =
+ td->smgr.is_temp ? MyProcNumber : INVALID_PROC_NUMBER;
+ RelPathStr rpath =
+ relpathbackend(td->smgr.rlocator, errProc, td->smgr.forkNum);
+ bool zeroed_any,
+ ignored_any;
+ uint8 zeroed_or_error_count,
+ checkfail_count,
+ first_off;
+ uint8 affected_count;
+ const char *msg_one,
+ *msg_mult,
+ *det_mult,
+ *hint_mult;
+
+ buffer_readv_decode_error(result, &zeroed_any, &ignored_any,
+ &zeroed_or_error_count,
+ &checkfail_count,
+ &first_off);
+
+ /*
+ * Treat a read that had both zeroed buffers *and* ignored checksums as a
+ * special case, it's too irregular to be emitted the same way as the other
+ * cases.
+ */
+ if (zeroed_any && ignored_any)
+ {
+ Assert(zeroed_any && ignored_any);
+ Assert(nblocks > 1); /* same block can't be both zeroed and ignored */
+ Assert(result.status != PGAIO_RS_ERROR);
+ affected_count = zeroed_or_error_count;
+
+ ereport(elevel,
+ errcode(ERRCODE_DATA_CORRUPTED),
+ errmsg("zeroing %u page(s) and ignoring %u checksum failure(s) among blocks %u..%u of relation %s",
+ affected_count, checkfail_count, first, last, rpath.str),
+ affected_count > 1 ?
+ errdetail("Block %u held first zeroed page.",
+ first + first_off) : 0,
+ errhint("See server log for details about the other %u invalid block(s).",
+ affected_count + checkfail_count - 1));
+ return;
+ }
+
+ /*
+ * The other messages are highly repetitive. To avoid duplicating a long
+ * and complicated ereport(), gather the translated format strings
+ * separately and then do one common ereport.
+ */
+ if (result.status == PGAIO_RS_ERROR)
+ {
+ Assert(!zeroed_any); /* can't have invalid pages when zeroing them */
+ affected_count = zeroed_or_error_count;
+ msg_one = _("invalid page in block %u of relation %s");
+ msg_mult = _("%u invalid pages among blocks %u..%u of relation %s");
+ det_mult = _("Block %u held first invalid page.");
+ hint_mult = _("See server log for the other %u invalid block(s).");
+ }
+ else if (zeroed_any && !ignored_any)
+ {
+ affected_count = zeroed_or_error_count;
+ msg_one = _("invalid page in block %u of relation %s; zeroing out page");
+ msg_mult = _("zeroing out %u invalid pages among blocks %u..%u of relation %s");
+ det_mult = _("Block %u held first zeroed page.");
+ hint_mult = _("See server log for the other %u zeroed block(s).");
+ }
+ else if (!zeroed_any && ignored_any)
+ {
+ affected_count = checkfail_count;
+ msg_one = _("ignoring checksum failure in block %u of relation %s");
+ msg_mult = _("ignoring %u checksum failures among blocks %u..%u of relation %s");
+ det_mult = _("Block %u held first ignored page.");
+ hint_mult = _("See server log for the other %u ignored block(s).");
+ }
+ else
+ pg_unreachable();
+
+ ereport(elevel,
+ errcode(ERRCODE_DATA_CORRUPTED),
+ affected_count == 1 ?
+ errmsg_internal(msg_one, first + first_off, rpath.str) :
+ errmsg_internal(msg_mult, affected_count, first, last, rpath.str),
+ affected_count > 1 ? errdetail_internal(det_mult, first + first_off) : 0,
+ affected_count > 1 ? errhint_internal(hint_mult, affected_count - 1) : 0);
+}
+
+static void
+shared_buffer_readv_stage(PgAioHandle *ioh, uint8 cb_data)
+{
+ buffer_stage_common(ioh, false, false);
+}
+
+static PgAioResult
+shared_buffer_readv_complete(PgAioHandle *ioh, PgAioResult prior_result,
+ uint8 cb_data)
+{
+ return buffer_readv_complete(ioh, prior_result, cb_data, false);
+}
+
+/*
+ * We need a backend-local completion callback for shared buffers, to be able
+ * to report checksum errors correctly. Unfortunately that can only safely
+ * happen if the reporting backend has previously called
+ */
+static PgAioResult
+shared_buffer_readv_complete_local(PgAioHandle *ioh, PgAioResult prior_result,
+ uint8 cb_data)
+{
+ bool zeroed_any,
+ ignored_any;
+ uint8 zeroed_or_error_count,
+ checkfail_count,
+ first_off;
+
+ if (prior_result.status == PGAIO_RS_OK)
+ return prior_result;
+
+ buffer_readv_decode_error(prior_result,
+ &zeroed_any,
+ &ignored_any,
+ &zeroed_or_error_count,
+ &checkfail_count,
+ &first_off);
+
+ if (checkfail_count)
+ {
+ PgAioTargetData *td = pgaio_io_get_target_data(ioh);
+
+ pgstat_report_checksum_failures_in_db(td->smgr.rlocator.dbOid,
+ checkfail_count);
+ }
+
+ return prior_result;
+}
+
+static void
+local_buffer_readv_stage(PgAioHandle *ioh, uint8 cb_data)
+{
+ buffer_stage_common(ioh, false, true);
+}
+
+static PgAioResult
+local_buffer_readv_complete(PgAioHandle *ioh, PgAioResult prior_result,
+ uint8 cb_data)
+{
+ return buffer_readv_complete(ioh, prior_result, cb_data, true);
+}
+
+/* readv callback is passed READ_BUFFERS_* flags as callback data */
+const PgAioHandleCallbacks aio_shared_buffer_readv_cb = {
+ .stage = shared_buffer_readv_stage,
+ .complete_shared = shared_buffer_readv_complete,
+ /* need a local callback to report checksum failures */
+ .complete_local = shared_buffer_readv_complete_local,
+ .report = buffer_readv_report,
+};
+
+/* readv callback is passed READ_BUFFERS_* flags as callback data */
+const PgAioHandleCallbacks aio_local_buffer_readv_cb = {
+ .stage = local_buffer_readv_stage,
+
+ /*
+ * Note that this, in contrast to the shared_buffers case, uses
+ * complete_local, as only the issuing backend has access to the required
+ * datastructures. This is important in case the IO completion may be
+ * consumed incidentally by another backend.
+ */
+ .complete_local = local_buffer_readv_complete,
+ .report = buffer_readv_report,
+};
diff --git a/src/backend/storage/buffer/localbuf.c b/src/backend/storage/buffer/localbuf.c
index 3a722321533..bf89076bb10 100644
--- a/src/backend/storage/buffer/localbuf.c
+++ b/src/backend/storage/buffer/localbuf.c
@@ -18,6 +18,7 @@
#include "access/parallel.h"
#include "executor/instrument.h"
#include "pgstat.h"
+#include "storage/aio.h"
#include "storage/buf_internals.h"
#include "storage/bufmgr.h"
#include "storage/fd.h"
@@ -187,7 +188,7 @@ FlushLocalBuffer(BufferDesc *bufHdr, SMgrRelation reln)
* Try to start an I/O operation. There currently are no reasons for
* StartLocalBufferIO to return false, so we raise an error in that case.
*/
- if (!StartLocalBufferIO(bufHdr, false))
+ if (!StartLocalBufferIO(bufHdr, false, false))
elog(ERROR, "failed to start write IO on local buffer");
/* Find smgr relation for buffer */
@@ -211,7 +212,7 @@ FlushLocalBuffer(BufferDesc *bufHdr, SMgrRelation reln)
IOOP_WRITE, io_start, 1, BLCKSZ);
/* Mark not-dirty */
- TerminateLocalBufferIO(bufHdr, true, 0);
+ TerminateLocalBufferIO(bufHdr, true, 0, false);
pgBufferUsage.local_blks_written++;
}
@@ -430,7 +431,7 @@ ExtendBufferedRelLocal(BufferManagerRelation bmr,
pg_atomic_unlocked_write_u32(&existing_hdr->state, buf_state);
/* no need to loop for local buffers */
- StartLocalBufferIO(existing_hdr, true);
+ StartLocalBufferIO(existing_hdr, true, false);
}
else
{
@@ -446,7 +447,7 @@ ExtendBufferedRelLocal(BufferManagerRelation bmr,
hresult->id = victim_buf_id;
- StartLocalBufferIO(victim_buf_hdr, true);
+ StartLocalBufferIO(victim_buf_hdr, true, false);
}
}
@@ -515,13 +516,31 @@ MarkLocalBufferDirty(Buffer buffer)
* Like StartBufferIO, but for local buffers
*/
bool
-StartLocalBufferIO(BufferDesc *bufHdr, bool forInput)
+StartLocalBufferIO(BufferDesc *bufHdr, bool forInput, bool nowait)
{
- uint32 buf_state = pg_atomic_read_u32(&bufHdr->state);
+ uint32 buf_state;
+ /*
+ * With AIO the buffer could have IO in progress, e.g. when there are two
+ * scans of the same relation. Either wait for the other IO or return
+ * false.
+ */
+ if (pgaio_wref_valid(&bufHdr->io_wref))
+ {
+ PgAioWaitRef iow = bufHdr->io_wref;
+
+ if (nowait)
+ return false;
+
+ pgaio_wref_wait(&iow);
+ }
+
+ /* Once we get here, there is definitely no I/O active on this buffer */
+
+ /* Check if someone else already did the I/O */
+ buf_state = pg_atomic_read_u32(&bufHdr->state);
if (forInput ? (buf_state & BM_VALID) : !(buf_state & BM_DIRTY))
{
- /* someone else already did the I/O */
return false;
}
@@ -536,7 +555,8 @@ StartLocalBufferIO(BufferDesc *bufHdr, bool forInput)
* Like TerminateBufferIO, but for local buffers
*/
void
-TerminateLocalBufferIO(BufferDesc *bufHdr, bool clear_dirty, uint32 set_flag_bits)
+TerminateLocalBufferIO(BufferDesc *bufHdr, bool clear_dirty, uint32 set_flag_bits,
+ bool release_aio)
{
/* Only need to adjust flags */
uint32 buf_state = pg_atomic_read_u32(&bufHdr->state);
@@ -549,12 +569,22 @@ TerminateLocalBufferIO(BufferDesc *bufHdr, bool clear_dirty, uint32 set_flag_bit
if (clear_dirty)
buf_state &= ~BM_DIRTY;
+ if (release_aio)
+ {
+ /* release pin held by IO subsystem, see also buffer_stage_common() */
+ Assert(BUF_STATE_GET_REFCOUNT(buf_state) > 0);
+ buf_state -= BUF_REFCOUNT_ONE;
+ pgaio_wref_clear(&bufHdr->io_wref);
+ }
+
buf_state |= set_flag_bits;
pg_atomic_unlocked_write_u32(&bufHdr->state, buf_state);
/* local buffers don't track IO using resowners */
/* local buffers don't use the IO CV, as no other process can see buffer */
+
+ /* local buffers don't use BM_PIN_COUNT_WAITER, so no need to wake */
}
/*
@@ -575,6 +605,19 @@ InvalidateLocalBuffer(BufferDesc *bufHdr, bool check_unreferenced)
uint32 buf_state;
LocalBufferLookupEnt *hresult;
+ /*
+ * It's possible that we started IO on this buffer before e.g. aborting
+ * the transaction that created a table. We need to wait for that IO to
+ * complete before removing / reusing the buffer.
+ */
+ if (pgaio_wref_valid(&bufHdr->io_wref))
+ {
+ PgAioWaitRef iow = bufHdr->io_wref;
+
+ pgaio_wref_wait(&iow);
+ Assert(!pgaio_wref_valid(&bufHdr->io_wref));
+ }
+
buf_state = pg_atomic_read_u32(&bufHdr->state);
/*
@@ -714,6 +757,8 @@ InitLocalBuffers(void)
*/
buf->buf_id = -i - 2;
+ pgaio_wref_clear(&buf->io_wref);
+
/*
* Intentionally do not initialize the buffer's atomic variable
* (besides zeroing the underlying memory above). That way we get
diff --git a/src/backend/storage/page/bufpage.c b/src/backend/storage/page/bufpage.c
index 5d1b039fcbb..84c63b1590b 100644
--- a/src/backend/storage/page/bufpage.c
+++ b/src/backend/storage/page/bufpage.c
@@ -78,8 +78,8 @@ PageInit(Page page, Size pageSize, Size specialSize)
* treat such a page as empty and without free space. Eventually, VACUUM
* will clean up such a page and make it usable.
*
- * If flag PIV_LOG_WARNING is set, a WARNING is logged in the event of
- * a checksum failure.
+ * If flag PIV_LOG_WARNING/PIV_LOG_LOG is set, a WARNING/LOG message is logged
+ * in the event of a checksum failure.
*
* To allow the caller to report statistics about checksum failures,
* *checksum_failure_p can be passed in. Note that there may be checksum
@@ -144,8 +144,8 @@ PageIsVerified(PageData *page, BlockNumber blkno, int flags, bool *checksum_fail
*/
if (checksum_failure)
{
- if ((flags & PIV_LOG_WARNING) != 0)
- ereport(WARNING,
+ if ((flags & (PIV_LOG_WARNING | PIV_LOG_LOG)) != 0)
+ ereport(flags & PIV_LOG_WARNING ? WARNING : LOG,
(errcode(ERRCODE_DATA_CORRUPTED),
errmsg("page verification failed, calculated checksum %u but expected %u",
checksum, p->pd_checksum)));
--
2.48.1.76.g4e746b1a31.dirty
[text/x-diff] v2.14-0011-Let-caller-of-PageIsVerified-control-ignore_ch.patch (4.2K, ../../d4ff3qxudxn2ze3zk2xwhds5ysezfei3uqeuimymj624t5chsd@sgr2h3rmln5t/12-v2.14-0011-Let-caller-of-PageIsVerified-control-ignore_ch.patch)
download | inline diff:
From 505f390da755bf83ec4005a726c213fc0a000b3f Mon Sep 17 00:00:00 2001
From: Andres Freund <andres@anarazel.de>
Date: Fri, 28 Mar 2025 16:48:57 -0400
Subject: [PATCH v2.14 11/29] Let caller of PageIsVerified() control
ignore_checksum_failure
Author:
Reviewed-by:
Discussion: https://postgr.es/m/
Backpatch:
---
src/include/storage/bufpage.h | 1 +
src/backend/catalog/storage.c | 8 ++++++--
src/backend/storage/buffer/bufmgr.c | 7 ++++++-
src/backend/storage/page/bufpage.c | 6 +++++-
4 files changed, 18 insertions(+), 4 deletions(-)
diff --git a/src/include/storage/bufpage.h b/src/include/storage/bufpage.h
index 7dbdfc564ed..aeb67c498c5 100644
--- a/src/include/storage/bufpage.h
+++ b/src/include/storage/bufpage.h
@@ -468,6 +468,7 @@ do { \
/* flags for PageIsVerified() */
#define PIV_LOG_WARNING (1 << 0)
#define PIV_LOG_LOG (1 << 1)
+#define PIV_IGNORE_CHECKSUM_FAILURE (1 << 2)
#define PageAddItem(page, item, size, offsetNumber, overwrite, is_heap) \
PageAddItemExtended(page, item, size, offsetNumber, \
diff --git a/src/backend/catalog/storage.c b/src/backend/catalog/storage.c
index 2577f69cbde..57a5e43c881 100644
--- a/src/backend/catalog/storage.c
+++ b/src/backend/catalog/storage.c
@@ -508,6 +508,7 @@ RelationCopyStorage(SMgrRelation src, SMgrRelation dst,
for (blkno = 0; blkno < nblocks; blkno++)
{
BulkWriteBuffer buf;
+ int piv_flags;
bool checksum_failure;
bool verified;
@@ -517,9 +518,12 @@ RelationCopyStorage(SMgrRelation src, SMgrRelation dst,
buf = smgr_bulk_get_buf(bulkstate);
smgrread(src, forkNum, blkno, (Page) buf);
- verified = PageIsVerified((Page) buf, blkno, PIV_LOG_WARNING,
+ piv_flags = PIV_LOG_WARNING;
+ if (ignore_checksum_failure)
+ piv_flags |= PIV_IGNORE_CHECKSUM_FAILURE;
+
+ verified = PageIsVerified((Page) buf, blkno, piv_flags,
&checksum_failure);
-
if (checksum_failure)
{
RelFileLocatorBackend rloc = src->smgr_rlocator;
diff --git a/src/backend/storage/buffer/bufmgr.c b/src/backend/storage/buffer/bufmgr.c
index ca89d9345f3..13c116d05d0 100644
--- a/src/backend/storage/buffer/bufmgr.c
+++ b/src/backend/storage/buffer/bufmgr.c
@@ -1572,6 +1572,7 @@ WaitReadBuffers(ReadBuffersOperation *operation)
{
BufferDesc *bufHdr;
Block bufBlock;
+ int piv_flags;
bool verified;
bool checksum_failure;
@@ -1587,8 +1588,11 @@ WaitReadBuffers(ReadBuffersOperation *operation)
}
/* check for garbage data */
+ piv_flags = PIV_LOG_WARNING;
+ if (ignore_checksum_failure)
+ piv_flags |= PIV_IGNORE_CHECKSUM_FAILURE;
verified = PageIsVerified((Page) bufBlock, io_first_block + j,
- PIV_LOG_WARNING, &checksum_failure);
+ piv_flags, &checksum_failure);
if (checksum_failure)
{
RelFileLocatorBackend rloc = operation->smgr->smgr_rlocator;
@@ -6485,6 +6489,7 @@ buffer_readv_complete_one(PgAioTargetData *td, uint8 buf_off, Buffer buffer,
BufferTag tag = buf_hdr->tag;
char *bufdata = BufferGetBlock(buffer);
uint32 set_flag_bits;
+ int piv_flags;
/* check that the buffer is in the expected state for a read */
#ifdef USE_ASSERT_CHECKING
diff --git a/src/backend/storage/page/bufpage.c b/src/backend/storage/page/bufpage.c
index 84c63b1590b..652746e1fa2 100644
--- a/src/backend/storage/page/bufpage.c
+++ b/src/backend/storage/page/bufpage.c
@@ -81,6 +81,10 @@ PageInit(Page page, Size pageSize, Size specialSize)
* If flag PIV_LOG_WARNING/PIV_LOG_LOG is set, a WARNING/LOG message is logged
* in the event of a checksum failure.
*
+ * If flag PIV_IGNORE_CHECKSUM_FAILURE is set, checksum failures will cause a
+ * message about the failure to be emitted, but will not cause
+ * PageIsVerified() to return false.
+ *
* To allow the caller to report statistics about checksum failures,
* *checksum_failure_p can be passed in. Note that there may be checksum
* failures even if this function returns true, due to
@@ -150,7 +154,7 @@ PageIsVerified(PageData *page, BlockNumber blkno, int flags, bool *checksum_fail
errmsg("page verification failed, calculated checksum %u but expected %u",
checksum, p->pd_checksum)));
- if (header_sane && ignore_checksum_failure)
+ if (header_sane && (flags & PIV_IGNORE_CHECKSUM_FAILURE))
return true;
}
--
2.48.1.76.g4e746b1a31.dirty
[text/x-diff] v2.14-0012-bufmgr-Use-AIO-in-StartReadBuffers.patch (27.9K, ../../d4ff3qxudxn2ze3zk2xwhds5ysezfei3uqeuimymj624t5chsd@sgr2h3rmln5t/13-v2.14-0012-bufmgr-Use-AIO-in-StartReadBuffers.patch)
download | inline diff:
From a75d34549bff8980ed76d0beb651885381246ca9 Mon Sep 17 00:00:00 2001
From: Andres Freund <andres@anarazel.de>
Date: Fri, 28 Mar 2025 16:51:01 -0400
Subject: [PATCH v2.14 12/29] bufmgr: Use AIO in StartReadBuffers()
This finally introduces the first actual use of AIO. StartReadBuffers() now
uses the AIO routines to issue IO. This converts a lot of callers to use the
AIO infrastructure.
As the implementation of StartReadBuffers() is also used by the functions for
reading individual blocks (StartReadBuffer() and through that
ReadBufferExtended()) this means all buffered read IO passes through the AIO
paths. However, as those are synchronous reads, actually performing the IO
asynchronously would be rarely beneficial. Instead such IOs are flagged to
always be executed synchronously. This way we don't have to duplicate a fair
bit of code.
When io_method=sync is used, the IO patterns generated after this change are
the same as before, i.e. actual reads are only issued in WaitReadBuffers() and
StartReadBuffers() may issue prefetch requests. This allows to bypass most of
the actual asynchronicity, which is important to make a change as big as this
less risky.
One thing worth calling out is that, if IO is actually executed
asynchronously, the precise meaning of what track_io_timing is measuring has
changed. Previously it tracked the time for each IO, but that does not make
sense when multiple IOs are executed concurrently. Now it only measures the
time actually spent waiting for IO. A subsequent commit will adjust the docs
for this.
While AIO is now actually used, the logic in read_stream.c will often prevent
using sufficiently many concurrent IOs. That will be addressed in the next
commit.
Reviewed-by: Noah Misch <noah@leadboat.com>
Reviewed-by: Nazir Bilal Yavuz <byavuz81@gmail.com>
Co-authored-by: Andres Freund <andres@anarazel.de>
Co-authored-by: Thomas Munro <thomas.munro@gmail.com>
Discussion: https://postgr.es/m/uvrtrknj4kdytuboidbhwclo4gxhswwcpgadptsjvjqcluzmah%40brqs62irg4dt
Discussion: https://postgr.es/m/20210223100344.llw5an2aklengrmn@alap3.anarazel.de
Discussion: https://postgr.es/m/stj36ea6yyhoxtqkhpieia2z4krnam7qyetc57rfezgk4zgapf@gcnactj4z56m
---
src/include/storage/bufmgr.h | 6 +
src/backend/storage/buffer/bufmgr.c | 638 +++++++++++++++++++++-------
2 files changed, 492 insertions(+), 152 deletions(-)
diff --git a/src/include/storage/bufmgr.h b/src/include/storage/bufmgr.h
index 11f8508a90b..867ae9facb5 100644
--- a/src/include/storage/bufmgr.h
+++ b/src/include/storage/bufmgr.h
@@ -114,6 +114,9 @@ typedef struct BufferManagerRelation
#define READ_BUFFERS_ISSUE_ADVICE (1 << 1)
/* Don't treat page as invalid due to checksum failures. */
#define READ_BUFFERS_IGNORE_CHECKSUM_FAILURES (1 << 2)
+/* IO will immediately be waited for */
+#define READ_BUFFERS_SYNCHRONOUSLY (1 << 3)
+
struct ReadBuffersOperation
{
@@ -133,6 +136,9 @@ struct ReadBuffersOperation
BlockNumber blocknum;
int flags;
int16 nblocks;
+ int16 nblocks_done;
+ PgAioWaitRef io_wref;
+ PgAioReturn io_return;
};
typedef struct ReadBuffersOperation ReadBuffersOperation;
diff --git a/src/backend/storage/buffer/bufmgr.c b/src/backend/storage/buffer/bufmgr.c
index 13c116d05d0..6ac1a821523 100644
--- a/src/backend/storage/buffer/bufmgr.c
+++ b/src/backend/storage/buffer/bufmgr.c
@@ -531,6 +531,8 @@ static inline BufferDesc *BufferAlloc(SMgrRelation smgr,
BlockNumber blockNum,
BufferAccessStrategy strategy,
bool *foundPtr, IOContext io_context);
+static bool AsyncReadBuffers(ReadBuffersOperation *operation, int *nblocks);
+static void CheckReadBuffersOperation(ReadBuffersOperation *operation, bool is_complete);
static Buffer GetVictimBuffer(BufferAccessStrategy strategy, IOContext io_context);
static void FlushBuffer(BufferDesc *buf, SMgrRelation reln,
IOObject io_object, IOContext io_context);
@@ -1231,10 +1233,14 @@ ReadBuffer_common(Relation rel, SMgrRelation smgr, char smgr_persistence,
return buffer;
}
+ /*
+ * Signal that we are going to immediately wait. If we're immediately
+ * waiting, there is no benefit in actually executing the IO
+ * asynchronously, it would just add dispatch overhead.
+ */
+ flags = READ_BUFFERS_SYNCHRONOUSLY;
if (mode == RBM_ZERO_ON_ERROR)
- flags = READ_BUFFERS_ZERO_ON_ERROR;
- else
- flags = 0;
+ flags |= READ_BUFFERS_ZERO_ON_ERROR;
operation.smgr = smgr;
operation.rel = rel;
operation.persistence = persistence;
@@ -1259,6 +1265,7 @@ StartReadBuffersImpl(ReadBuffersOperation *operation,
{
int actual_nblocks = *nblocks;
int maxcombine = 0;
+ bool did_start_io;
Assert(*nblocks == 1 || allow_forwarding);
Assert(*nblocks > 0);
@@ -1326,6 +1333,20 @@ StartReadBuffersImpl(ReadBuffersOperation *operation,
if (i == 0)
{
*nblocks = 1;
+
+#ifdef USE_ASSERT_CHECKING
+
+ /*
+ * Initialize enough of ReadBuffersOperation to make
+ * CheckReadBuffersOperation() work. Outside of assertions
+ * that's not necessary when no IO is issued.
+ */
+ operation->buffers = buffers;
+ operation->blocknum = blockNum;
+ operation->nblocks = 1;
+ operation->nblocks_done = 1;
+ CheckReadBuffersOperation(operation, true);
+#endif
return false;
}
@@ -1368,25 +1389,74 @@ StartReadBuffersImpl(ReadBuffersOperation *operation,
operation->blocknum = blockNum;
operation->flags = flags;
operation->nblocks = actual_nblocks;
+ operation->nblocks_done = 0;
+ pgaio_wref_clear(&operation->io_wref);
- if (flags & READ_BUFFERS_ISSUE_ADVICE)
+ /*
+ * When using AIO, start the IO in the background. If not, issue prefetch
+ * requests if desired by the caller.
+ *
+ * The reason we have a dedicated path for IOMETHOD_SYNC here is to
+ * de-risk the introduction of AIO somewhat. It's a large architectural
+ * change, with lots of chances for unanticipated performance effects.
+ *
+ * Use of IOMETHOD_SYNC already leads to not actually performing IO
+ * asynchronously, but without the check here we'd execute IO earlier than
+ * we used to. Eventually this IOMETHOD_SYNC specific path should go away.
+ */
+ if (io_method != IOMETHOD_SYNC)
{
/*
- * In theory we should only do this if PinBufferForBlock() had to
- * allocate new buffers above. That way, if two calls to
- * StartReadBuffers() were made for the same blocks before
- * WaitReadBuffers(), only the first would issue the advice. That'd be
- * a better simulation of true asynchronous I/O, which would only
- * start the I/O once, but isn't done here for simplicity.
+ * Try to start IO asynchronously. It's possible that no IO needs to
+ * be started, if another backend already performed the IO.
+ *
+ * Note that if an IO is started, it might not cover the entire
+ * requested range, e.g. because an intermediary block has been read
+ * in by another backend. In that case any "trailing" buffers we
+ * already pinned above will be "forwarded" by read_stream.c to the
+ * next call to StartReadBuffers().
+ *
+ * This is signalled to the caller by decrementing *nblocks *and*
+ * reducing operation->nblocks. The latter is done here, but not below
+ * WaitReadBuffers(), as in WaitReadBuffers() we can't "shorten" the
+ * overall read size anymore, we need to retry until done in its
+ * entirety or until failed.
*/
- smgrprefetch(operation->smgr,
- operation->forknum,
- blockNum,
- actual_nblocks);
+ did_start_io = AsyncReadBuffers(operation, nblocks);
+
+ operation->nblocks = *nblocks;
+ }
+ else
+ {
+ operation->flags |= READ_BUFFERS_SYNCHRONOUSLY;
+
+ if (flags & READ_BUFFERS_ISSUE_ADVICE)
+ {
+ /*
+ * In theory we should only do this if PinBufferForBlock() had to
+ * allocate new buffers above. That way, if two calls to
+ * StartReadBuffers() were made for the same blocks before
+ * WaitReadBuffers(), only the first would issue the advice.
+ * That'd be a better simulation of true asynchronous I/O, which
+ * would only start the I/O once, but isn't done here for
+ * simplicity.
+ */
+ smgrprefetch(operation->smgr,
+ operation->forknum,
+ blockNum,
+ actual_nblocks);
+ }
+
+ /*
+ * Indicate that WaitReadBuffers() should be called. WaitReadBuffers()
+ * will initiate the necessary IO.
+ */
+ did_start_io = true;
}
- /* Indicate that WaitReadBuffers() should be called. */
- return true;
+ CheckReadBuffersOperation(operation, !did_start_io);
+
+ return did_start_io;
}
/*
@@ -1452,8 +1522,35 @@ StartReadBuffer(ReadBuffersOperation *operation,
return result;
}
+/*
+ * Perform sanity checks on the ReadBuffersOperation.
+ */
+static void
+CheckReadBuffersOperation(ReadBuffersOperation *operation, bool is_complete)
+{
+#ifdef USE_ASSERT_CHECKING
+ Assert(operation->nblocks_done <= operation->nblocks);
+ Assert(!is_complete || operation->nblocks == operation->nblocks_done);
+
+ for (int i = 0; i < operation->nblocks; i++)
+ {
+ Buffer buffer = operation->buffers[i];
+ BufferDesc *buf_hdr = BufferIsLocal(buffer) ?
+ GetLocalBufferDescriptor(-buffer - 1) :
+ GetBufferDescriptor(buffer - 1);
+
+ Assert(BufferGetBlockNumber(buffer) == operation->blocknum + i);
+ Assert(pg_atomic_read_u32(&buf_hdr->state) & BM_TAG_VALID);
+
+ if (i < operation->nblocks_done)
+ Assert(pg_atomic_read_u32(&buf_hdr->state) & BM_VALID);
+ }
+#endif
+}
+
+/* helper for ReadBuffersCanStartIO(), to avoid repetition */
static inline bool
-WaitReadBuffersCanStartIO(Buffer buffer, bool nowait)
+ReadBuffersCanStartIOOnce(Buffer buffer, bool nowait)
{
if (BufferIsLocal(buffer))
return StartLocalBufferIO(GetLocalBufferDescriptor(-buffer - 1),
@@ -1462,31 +1559,240 @@ WaitReadBuffersCanStartIO(Buffer buffer, bool nowait)
return StartBufferIO(GetBufferDescriptor(buffer - 1), true, nowait);
}
+/*
+ * Helper for AsyncReadBuffers that tries to get the buffer ready for IO.
+ */
+static inline bool
+ReadBuffersCanStartIO(Buffer buffer, bool nowait)
+{
+ /*
+ * If this backend currently has staged IO, we need to submit the pending
+ * IO before waiting for the right to issue IO, to avoid the potential for
+ * deadlocks (and, more commonly, unnecessary delays for other backends).
+ */
+ if (!nowait && pgaio_have_staged())
+ {
+ if (ReadBuffersCanStartIOOnce(buffer, true))
+ return true;
+
+ /*
+ * Unfortunately StartBufferIO() returning false doesn't allow to
+ * distinguish between the buffer already being valid and IO already
+ * being in progress. Since IO already being in progress is quite
+ * rare, this approach seems fine.
+ */
+ pgaio_submit_staged();
+ }
+
+ return ReadBuffersCanStartIOOnce(buffer, nowait);
+}
+
+/*
+ * Helper for WaitReadBuffers() that processes the results of a readv
+ * operation, raising an error if necessary.
+ */
+static void
+ProcessReadBuffersResult(ReadBuffersOperation *operation)
+{
+ PgAioReturn *aio_ret = &operation->io_return;
+ PgAioResultStatus rs = aio_ret->result.status;
+ int newly_read_blocks = 0;
+
+ Assert(pgaio_wref_valid(&operation->io_wref));
+ Assert(aio_ret->result.status != PGAIO_RS_UNKNOWN);
+
+ /*
+ * SMGR reports the number of blocks successfully read as the result of
+ * the IO operation. Thus we can simply add that to ->nblocks_done.
+ */
+
+ if (likely(rs != PGAIO_RS_ERROR))
+ newly_read_blocks = aio_ret->result.result;
+
+ if (rs == PGAIO_RS_ERROR || rs == PGAIO_RS_WARNING)
+ pgaio_result_report(aio_ret->result, &aio_ret->target_data,
+ rs == PGAIO_RS_ERROR ? ERROR : WARNING);
+ else if (aio_ret->result.status == PGAIO_RS_PARTIAL)
+ {
+ /*
+ * We'll retry, so we just emit a debug message to the server log (or
+ * not even that in prod scenarios).
+ */
+ pgaio_result_report(aio_ret->result, &aio_ret->target_data, DEBUG1);
+ elog(DEBUG3, "partial read, will retry");
+ }
+
+ Assert(newly_read_blocks > 0);
+ Assert(newly_read_blocks <= MAX_IO_COMBINE_LIMIT);
+
+ operation->nblocks_done += newly_read_blocks;
+
+ Assert(operation->nblocks_done <= operation->nblocks);
+}
+
void
WaitReadBuffers(ReadBuffersOperation *operation)
{
- Buffer *buffers;
- int nblocks;
- BlockNumber blocknum;
- ForkNumber forknum;
+ PgAioReturn *aio_ret = &operation->io_return;
IOContext io_context;
IOObject io_object;
- char persistence;
- /* Find the range of the physical read we need to perform. */
- nblocks = operation->nblocks;
- buffers = &operation->buffers[0];
- blocknum = operation->blocknum;
- forknum = operation->forknum;
- persistence = operation->persistence;
+ if (operation->persistence == RELPERSISTENCE_TEMP)
+ {
+ io_context = IOCONTEXT_NORMAL;
+ io_object = IOOBJECT_TEMP_RELATION;
+ }
+ else
+ {
+ io_context = IOContextForStrategy(operation->strategy);
+ io_object = IOOBJECT_RELATION;
+ }
- Assert(nblocks > 0);
- Assert(nblocks <= MAX_IO_COMBINE_LIMIT);
+ /*
+ * If we get here without an IO operation having been issued, the
+ * io_method == IOMETHOD_SYNC path must have been used. Otherwise the
+ * caller should not have called WaitReadBuffers().
+ *
+ * In the case of IOMETHOD_SYNC, we start - as we used to before the
+ * introducing of AIO - the IO in WaitReadBuffers(). This is done as part
+ * of the retry logic below, no extra code is required.
+ *
+ * This path is expected to eventually go away.
+ */
+ if (!pgaio_wref_valid(&operation->io_wref) && io_method != IOMETHOD_SYNC)
+ elog(ERROR, "waiting for read operation that didn't read");
+
+ /*
+ * To handle partial reads, and IOMETHOD_SYNC, we re-issue IO until we're
+ * done. We may need multiple retries, not just because we could get
+ * multiple partial reads, but also because some of the remaining
+ * to-be-read buffers may have been read in by other backends, limiting
+ * the IO size.
+ */
+ while (true)
+ {
+ int ignored_nblocks_progress;
+
+ CheckReadBuffersOperation(operation, false);
+
+ /*
+ * If there is an IO associated with the operation, we may need to
+ * wait for it.
+ */
+ if (pgaio_wref_valid(&operation->io_wref))
+ {
+ /*
+ * Track the time spent waiting for the IO to complete. As
+ * tracking a wait even if we don't actually need to wait
+ *
+ * a) is not cheap, due to the timestamping overhead
+ *
+ * b) reports some time as waiting, even if we never waited
+ *
+ * we first check if we already know the IO is complete.
+ */
+ if (aio_ret->result.status == PGAIO_RS_UNKNOWN &&
+ !pgaio_wref_check_done(&operation->io_wref))
+ {
+ instr_time io_start = pgstat_prepare_io_time(track_io_timing);
+
+ pgaio_wref_wait(&operation->io_wref);
+
+ /*
+ * The IO operation itself was already counted earlier, in
+ * AsyncReadBuffers(), this just accounts for the wait time.
+ */
+ pgstat_count_io_op_time(io_object, io_context, IOOP_READ,
+ io_start, 0, 0);
+ }
+ else
+ {
+ Assert(pgaio_wref_check_done(&operation->io_wref));
+ }
+
+ /*
+ * We now are sure the IO completed. Check the results. This
+ * includes reporting on errors if there were any.
+ */
+ ProcessReadBuffersResult(operation);
+ }
+
+ /*
+ * Most of the time, the one IO we already started, will read in
+ * everything. But we need to deal with partial reads and buffers not
+ * needing IO anymore.
+ */
+ if (operation->nblocks_done == operation->nblocks)
+ break;
+
+ CHECK_FOR_INTERRUPTS();
+
+ /*
+ * This may only complete the IO partially, either because some
+ * buffers were already valid, or because of a partial read.
+ *
+ * NB: In contrast to after the AsyncReadBuffers() call in
+ * StartReadBuffers(), we do *not* reduce
+ * ReadBuffersOperation->nblocks here, callers expect the full
+ * operation to be completed at this point (as more operations may
+ * have been queued).
+ */
+ AsyncReadBuffers(operation, &ignored_nblocks_progress);
+ }
+
+ CheckReadBuffersOperation(operation, true);
+
+ /* NB: READ_DONE tracepoint was already executed in completion callback */
+}
+
+/*
+ * Initiate IO for the ReadBuffersOperation
+ *
+ * This function only starts a single IO at a time. The size of the IO may be
+ * limited to below the to-be-read blocks, if one of the buffers has
+ * concurrently been read in. If the first to-be-read buffer is already valid,
+ * no IO will be issued.
+ *
+ * To support retries after partial reads, the first operation->nblocks_done
+ * buffers are skipped.
+ *
+ * On return *nblocks_progress is updated to reflect the number of buffers
+ * affected by the call. If the first buffer is valid, *nblocks_progress is
+ * set to 1 and operation->nblocks_done is incremented.
+ *
+ * Returns true if IO was initiated, false if no IO was necessary.
+ */
+static bool
+AsyncReadBuffers(ReadBuffersOperation *operation, int *nblocks_progress)
+{
+ Buffer *buffers = &operation->buffers[0];
+ int flags = operation->flags;
+ BlockNumber blocknum = operation->blocknum;
+ ForkNumber forknum = operation->forknum;
+ char persistence = operation->persistence;
+ int16 nblocks_done = operation->nblocks_done;
+ Buffer *io_buffers = &operation->buffers[nblocks_done];
+ int io_buffers_len = 0;
+ PgAioHandle *ioh;
+ uint32 ioh_flags = 0;
+ void *io_pages[MAX_IO_COMBINE_LIMIT];
+ IOContext io_context;
+ IOObject io_object;
+ bool did_start_io;
+
+ /*
+ * When this IO is executed synchronously, either because the caller will
+ * immediately block waiting for the IO or because IOMETHOD_SYNC is used,
+ * the AIO subsystem needs to know.
+ */
+ if (flags & READ_BUFFERS_SYNCHRONOUSLY)
+ ioh_flags |= PGAIO_HF_SYNCHRONOUS;
if (persistence == RELPERSISTENCE_TEMP)
{
io_context = IOCONTEXT_NORMAL;
io_object = IOOBJECT_TEMP_RELATION;
+ ioh_flags |= PGAIO_HF_REFERENCES_LOCAL;
}
else
{
@@ -1494,155 +1800,183 @@ WaitReadBuffers(ReadBuffersOperation *operation)
io_object = IOOBJECT_RELATION;
}
- for (int i = 0; i < nblocks; ++i)
+ /*
+ * If zero_damaged_pages is enabled, add the READ_BUFFERS_ZERO_ON_ERROR
+ * flag. The reason for that is that, hopefully, zero_damaged_pages isn't
+ * set globally, but on a per-session basis. The completion callback,
+ * which may be run in other processes, e.g. in IO workers, may have a
+ * different value of the zero_damaged_pages GUC.
+ *
+ * XXX: We probably should eventually use a different flag for
+ * zero_damaged_pages, so we can report different log levels / error codes
+ * for zero_damaged_pages and ZERO_ON_ERROR.
+ */
+ if (zero_damaged_pages)
+ flags |= READ_BUFFERS_ZERO_ON_ERROR;
+
+ /*
+ * For the same reason as with zero_damaged_pages we need to use this
+ * backend's ignore_checksum_failure value.
+ */
+ if (ignore_checksum_failure)
+ flags |= READ_BUFFERS_IGNORE_CHECKSUM_FAILURES;
+
+
+ /*
+ * To be allowed to report stats in the local completion callback we need
+ * to prepare to report stats now. This ensures we can safely report the
+ * checksum failure even in a critical section.
+ */
+ pgstat_prepare_report_checksum_failure(operation->smgr->smgr_rlocator.locator.dbOid);
+
+ /*
+ * Get IO handle before ReadBuffersCanStartIO(), as pgaio_io_acquire()
+ * might block, which we don't want after setting IO_IN_PROGRESS.
+ *
+ * If we need to wait for IO before we can get a handle, submit
+ * already-staged IO first, so that other backends don't need to wait.
+ * There wouldn't be a deadlock risk, as pgaio_io_acquire() just needs to
+ * wait for already submitted IO, which doesn't require additional locks,
+ * but it could still cause undesirable waits.
+ *
+ * A secondary benefit is that this would allow us to measure the time in
+ * pgaio_io_acquire() without causing undue timer overhead in the common,
+ * non-blocking, case. However, currently the pgstats infrastructure
+ * doesn't really allow that, as it a) asserts that an operation can't
+ * have time without operations b) doesn't have an API to report
+ * "accumulated" time.
+ */
+ ioh = pgaio_io_acquire_nb(CurrentResourceOwner, &operation->io_return);
+ if (unlikely(!ioh))
+ {
+ pgaio_submit_staged();
+
+ ioh = pgaio_io_acquire(CurrentResourceOwner, &operation->io_return);
+ }
+
+ /*
+ * Check if we can start IO on the first to-be-read buffer.
+ *
+ * If an I/O is already in progress in another backend, we want to wait
+ * for the outcome: either done, or something went wrong and we will
+ * retry.
+ */
+ if (!ReadBuffersCanStartIO(buffers[nblocks_done], false))
+ {
+ /*
+ * Someone else has already completed this block, we're done.
+ *
+ * When IO is necessary, ->nblocks_done is updated in
+ * ProcessReadBuffersResult(), but that is not called if no IO is
+ * necessary. Thus update here.
+ */
+ operation->nblocks_done += 1;
+ *nblocks_progress = 1;
+
+ pgaio_io_release(ioh);
+ pgaio_wref_clear(&operation->io_wref);
+ did_start_io = false;
+
+ /*
+ * Report and track this as a 'hit' for this backend, even though it
+ * must have started out as a miss in PinBufferForBlock(). The other
+ * backend will track this as a 'read'.
+ */
+ TRACE_POSTGRESQL_BUFFER_READ_DONE(forknum, blocknum + operation->nblocks_done,
+ operation->smgr->smgr_rlocator.locator.spcOid,
+ operation->smgr->smgr_rlocator.locator.dbOid,
+ operation->smgr->smgr_rlocator.locator.relNumber,
+ operation->smgr->smgr_rlocator.backend,
+ true);
+
+ if (persistence == RELPERSISTENCE_TEMP)
+ pgBufferUsage.local_blks_hit += 1;
+ else
+ pgBufferUsage.shared_blks_hit += 1;
+
+ if (operation->rel)
+ pgstat_count_buffer_hit(operation->rel);
+
+ pgstat_count_io_op(io_object, io_context, IOOP_HIT, 1, 0);
+
+ if (VacuumCostActive)
+ VacuumCostBalance += VacuumCostPageHit;
+ }
+ else
{
- int io_buffers_len;
- Buffer io_buffers[MAX_IO_COMBINE_LIMIT];
- void *io_pages[MAX_IO_COMBINE_LIMIT];
instr_time io_start;
- BlockNumber io_first_block;
-
- /*
- * Skip this block if someone else has already completed it. If an
- * I/O is already in progress in another backend, this will wait for
- * the outcome: either done, or something went wrong and we will
- * retry.
- */
- if (!WaitReadBuffersCanStartIO(buffers[i], false))
- {
- /*
- * Report and track this as a 'hit' for this backend, even though
- * it must have started out as a miss in PinBufferForBlock(). The
- * other backend will track this as a 'read'.
- */
- TRACE_POSTGRESQL_BUFFER_READ_DONE(forknum, blocknum + i,
- operation->smgr->smgr_rlocator.locator.spcOid,
- operation->smgr->smgr_rlocator.locator.dbOid,
- operation->smgr->smgr_rlocator.locator.relNumber,
- operation->smgr->smgr_rlocator.backend,
- true);
-
- if (persistence == RELPERSISTENCE_TEMP)
- pgBufferUsage.local_blks_hit += 1;
- else
- pgBufferUsage.shared_blks_hit += 1;
-
- if (operation->rel)
- pgstat_count_buffer_hit(operation->rel);
-
- pgstat_count_io_op(io_object, io_context, IOOP_HIT, 1, 0);
-
- if (VacuumCostActive)
- VacuumCostBalance += VacuumCostPageHit;
-
- continue;
- }
/* We found a buffer that we need to read in. */
- io_buffers[0] = buffers[i];
- io_pages[0] = BufferGetBlock(buffers[i]);
- io_first_block = blocknum + i;
+ Assert(io_buffers[0] == buffers[nblocks_done]);
+ io_pages[0] = BufferGetBlock(buffers[nblocks_done]);
io_buffers_len = 1;
/*
* How many neighboring-on-disk blocks can we scatter-read into other
* buffers at the same time? In this case we don't wait if we see an
- * I/O already in progress. We already hold BM_IO_IN_PROGRESS for the
+ * I/O already in progress. We already set BM_IO_IN_PROGRESS for the
* head block, so we should get on with that I/O as soon as possible.
- * We'll come back to this block again, above.
*/
- while ((i + 1) < nblocks &&
- WaitReadBuffersCanStartIO(buffers[i + 1], true))
+ for (int i = nblocks_done + 1; i < operation->nblocks; i++)
{
+ if (!ReadBuffersCanStartIO(buffers[i], true))
+ break;
/* Must be consecutive block numbers. */
- Assert(BufferGetBlockNumber(buffers[i + 1]) ==
- BufferGetBlockNumber(buffers[i]) + 1);
+ Assert(BufferGetBlockNumber(buffers[i - 1]) ==
+ BufferGetBlockNumber(buffers[i]) - 1);
+ Assert(io_buffers[io_buffers_len] == buffers[i]);
- io_buffers[io_buffers_len] = buffers[++i];
io_pages[io_buffers_len++] = BufferGetBlock(buffers[i]);
}
+ /* get a reference to wait for in WaitReadBuffers() */
+ pgaio_io_get_wref(ioh, &operation->io_wref);
+
+ /* provide the list of buffers to the completion callbacks */
+ pgaio_io_set_handle_data_32(ioh, (uint32 *) io_buffers, io_buffers_len);
+
+ pgaio_io_register_callbacks(ioh,
+ persistence == RELPERSISTENCE_TEMP ?
+ PGAIO_HCB_LOCAL_BUFFER_READV :
+ PGAIO_HCB_SHARED_BUFFER_READV,
+ flags);
+
+ pgaio_io_set_flag(ioh, ioh_flags);
+
+ /* ---
+ * Even though we're trying to issue IO asynchronously, track the time
+ * in smgrstartreadv():
+ * - if io_method == IOMETHOD_SYNC, we will always perform the IO
+ * immediately
+ * - the io method might not support the IO (e.g. worker IO for a temp
+ * table)
+ * ---
+ */
io_start = pgstat_prepare_io_time(track_io_timing);
- smgrreadv(operation->smgr, forknum, io_first_block, io_pages, io_buffers_len);
- pgstat_count_io_op_time(io_object, io_context, IOOP_READ, io_start,
- 1, io_buffers_len * BLCKSZ);
-
- /* Verify each block we read, and terminate the I/O. */
- for (int j = 0; j < io_buffers_len; ++j)
- {
- BufferDesc *bufHdr;
- Block bufBlock;
- int piv_flags;
- bool verified;
- bool checksum_failure;
-
- if (persistence == RELPERSISTENCE_TEMP)
- {
- bufHdr = GetLocalBufferDescriptor(-io_buffers[j] - 1);
- bufBlock = LocalBufHdrGetBlock(bufHdr);
- }
- else
- {
- bufHdr = GetBufferDescriptor(io_buffers[j] - 1);
- bufBlock = BufHdrGetBlock(bufHdr);
- }
-
- /* check for garbage data */
- piv_flags = PIV_LOG_WARNING;
- if (ignore_checksum_failure)
- piv_flags |= PIV_IGNORE_CHECKSUM_FAILURE;
- verified = PageIsVerified((Page) bufBlock, io_first_block + j,
- piv_flags, &checksum_failure);
- if (checksum_failure)
- {
- RelFileLocatorBackend rloc = operation->smgr->smgr_rlocator;
-
- pgstat_prepare_report_checksum_failure(rloc.locator.dbOid);
- pgstat_report_checksum_failures_in_db(rloc.locator.dbOid, 1);
- }
-
- if (!verified)
- {
- if ((operation->flags & READ_BUFFERS_ZERO_ON_ERROR) || zero_damaged_pages)
- {
- ereport(WARNING,
- (errcode(ERRCODE_DATA_CORRUPTED),
- errmsg("invalid page in block %u of relation %s; zeroing out page",
- io_first_block + j,
- relpath(operation->smgr->smgr_rlocator, forknum).str)));
- memset(bufBlock, 0, BLCKSZ);
- }
- else
- ereport(ERROR,
- (errcode(ERRCODE_DATA_CORRUPTED),
- errmsg("invalid page in block %u of relation %s",
- io_first_block + j,
- relpath(operation->smgr->smgr_rlocator, forknum).str)));
- }
-
- /* Set BM_VALID, terminate IO, and wake up any waiters */
- if (persistence == RELPERSISTENCE_TEMP)
- TerminateLocalBufferIO(bufHdr, false, BM_VALID, false);
- else
- TerminateBufferIO(bufHdr, false, BM_VALID, true, false);
-
- /* Report I/Os as completing individually. */
- TRACE_POSTGRESQL_BUFFER_READ_DONE(forknum, io_first_block + j,
- operation->smgr->smgr_rlocator.locator.spcOid,
- operation->smgr->smgr_rlocator.locator.dbOid,
- operation->smgr->smgr_rlocator.locator.relNumber,
- operation->smgr->smgr_rlocator.backend,
- false);
- }
+ smgrstartreadv(ioh, operation->smgr, forknum,
+ blocknum + nblocks_done,
+ io_pages, io_buffers_len);
+ pgstat_count_io_op_time(io_object, io_context, IOOP_READ,
+ io_start, 1, io_buffers_len * BLCKSZ);
if (persistence == RELPERSISTENCE_TEMP)
pgBufferUsage.local_blks_read += io_buffers_len;
else
pgBufferUsage.shared_blks_read += io_buffers_len;
+ /*
+ * Track vacuum cost when issuing IO, not after waiting for it.
+ * Otherwise we could end up issuing a lot of IO in a short timespan,
+ * despite a low cost limit.
+ */
if (VacuumCostActive)
VacuumCostBalance += VacuumCostPageMiss * io_buffers_len;
+
+ *nblocks_progress = io_buffers_len;
+ did_start_io = true;
}
+
+ return did_start_io;
}
/*
--
2.48.1.76.g4e746b1a31.dirty
[text/x-diff] v2.14-0013-aio-Add-README.md-explaining-higher-level-desi.patch (19.1K, ../../d4ff3qxudxn2ze3zk2xwhds5ysezfei3uqeuimymj624t5chsd@sgr2h3rmln5t/14-v2.14-0013-aio-Add-README.md-explaining-higher-level-desi.patch)
download | inline diff:
From e30b7edbe7d8c1216409c00fbfe696ad70273d5f Mon Sep 17 00:00:00 2001
From: Andres Freund <andres@anarazel.de>
Date: Tue, 18 Mar 2025 14:40:06 -0400
Subject: [PATCH v2.14 13/29] aio: Add README.md explaining higher level design
Reviewed-by: Noah Misch <noah@leadboat.com>
Reviewed-by: Heikki Linnakangas <hlinnaka@iki.fi>
Discussion: https://postgr.es/m/uvrtrknj4kdytuboidbhwclo4gxhswwcpgadptsjvjqcluzmah%40brqs62irg4dt
Discussion: https://postgr.es/m/20210223100344.llw5an2aklengrmn@alap3.anarazel.de
Discussion: https://postgr.es/m/stj36ea6yyhoxtqkhpieia2z4krnam7qyetc57rfezgk4zgapf@gcnactj4z56m
---
src/backend/storage/aio/README.md | 424 ++++++++++++++++++++++++++++++
src/backend/storage/aio/aio.c | 2 +
2 files changed, 426 insertions(+)
create mode 100644 src/backend/storage/aio/README.md
diff --git a/src/backend/storage/aio/README.md b/src/backend/storage/aio/README.md
new file mode 100644
index 00000000000..ddd59404a59
--- /dev/null
+++ b/src/backend/storage/aio/README.md
@@ -0,0 +1,424 @@
+# Asynchronous & Direct IO
+
+## Motivation
+
+### Why Asynchronous IO
+
+Until the introduction of asynchronous IO Postgres relied on the operating
+system to hide the cost of synchronous IO from Postgres. While this worked
+surprisingly well in a lot of workloads, it does not do as good a job on
+prefetching and controlled writeback as we would like.
+
+There are important expensive operations like `fdatasync()` where the operating
+system cannot hide the storage latency. This is particularly important for WAL
+writes, where the ability to asynchronously issue `fdatasync()` or O_DSYNC
+writes can yield significantly higher throughput.
+
+
+### Why Direct / unbuffered IO
+
+The main reasons to want to use Direct IO are:
+
+- Lower CPU usage / higher throughput. Particularly on modern storage buffered
+ writes are bottlenecked by the operating system having to copy data from the
+ kernel's page cache to postgres buffer pool using the CPU. Whereas direct IO
+ can often move the data directly between the storage devices and postgres'
+ buffer cache, using DMA. While that transfer is ongoing, the CPU is free to
+ perform other work.
+- Reduced latency - Direct IO can have substantially lower latency than
+ buffered IO, which can be impactful for OLTP workloads bottlenecked by WAL
+ write latency.
+- Avoiding double buffering between operating system cache and postgres'
+ shared_buffers.
+- Better control over the timing and pace of dirty data writeback.
+
+
+The main reasons *not* to use Direct IO are:
+
+- Without AIO, Direct IO is unusably slow for most purposes.
+- Even with AIO, many parts of postgres need to be modified to perform
+ explicit prefetching.
+- In situations where shared_buffers cannot be set appropriately large,
+ e.g. because there are many different postgres instances hosted on shared
+ hardware, performance will often be worse than when using buffered IO.
+
+
+## AIO Usage Example
+
+In many cases code that can benefit from AIO does not directly have to
+interact with the AIO interface, but can use AIO via higher-level
+abstractions. See [Helpers](#helpers).
+
+In this example, a buffer will be read into shared buffers.
+
+```C
+/*
+ * Result of the operation, only to be accessed in this backend.
+ */
+PgAioReturn ioret;
+
+/*
+ * Acquire an AIO Handle, ioret will get result upon completion.
+ *
+ * Note that ioret needs to stay alive until the IO completes or
+ * CurrentResourceOwner is released (i.e. an error is thrown).
+ */
+PgAioHandle *ioh = pgaio_io_acquire(CurrentResourceOwner, &ioret);
+
+/*
+ * Reference that can be used to wait for the IO we initiate below. This
+ * reference can reside in local or shared memory and waited upon by any
+ * process. An arbitrary number of references can be made for each IO.
+ */
+PgAioWaitRef iow;
+
+pgaio_io_get_wref(ioh, &iow);
+
+/*
+ * Arrange for shared buffer completion callbacks to be called upon completion
+ * of the IO. This callback will update the buffer descriptors associated with
+ * the AioHandle, which e.g. allows other backends to access the buffer.
+ *
+ * A callback can be passed a small bit of data, e.g. to indicate whether to
+ * zero a buffer if it is invalid.
+ *
+ * Multiple completion callbacks can be registered for each handle.
+ */
+pgaio_io_register_callbacks(ioh, PGAIO_HCB_SHARED_BUFFER_READV, 0);
+
+/*
+ * The completion callback needs to know which buffers to update when the IO
+ * completes. As the AIO subsystem does not know about buffers, we have to
+ * associate this information with the AioHandle, for use by the completion
+ * callback registered above.
+ *
+ * In this example we're reading only a single buffer, hence the 1.
+ */
+pgaio_io_set_handle_data_32(ioh, (uint32 *) buffer, 1);
+
+/*
+ * Pass the AIO handle to lower-level function. When operating on the level of
+ * buffers, we don't know how exactly the IO is performed, that is the
+ * responsibility of the storage manager implementation.
+ *
+ * E.g. md.c needs to translate block numbers into offsets in segments.
+ *
+ * Once the IO handle has been handed off to smgstartreadv(), it may not
+ * further be used, as the IO may immediately get executed below
+ * smgrstartreadv() and the handle reused for another IO.
+ *
+ * To issue multiple IOs in an efficient way, a caller can call
+ * pgaio_enter_batchmode() before starting multiple IOs, and end that batch
+ * with pgaio_exit_batchmode(). Note that one needs to be careful while there
+ * may be unsubmitted IOs, as another backend may need to wait for one of the
+ * unsubmitted IOs. If this backend then had to wait for the other backend,
+ * it'd end in an undetected deadlock. See pgaio_enter_batchmode() for more
+ * details.
+ *
+ * Note that even while in batchmode an IO might get submitted immediately,
+ * e.g. due to reaching a limit on the number of unsubmitted IOs, and even
+ * complete before smgrstartreadv() returns.
+ */
+smgrstartreadv(ioh, operation->smgr, forknum, blkno,
+ BufferGetBlock(buffer), 1);
+
+/*
+ * To benefit from AIO, it is beneficial to perform other work, including
+ * submitting other IOs, before waiting for the IO to complete. Otherwise
+ * we could just have used synchronous, blocking IO.
+ */
+perform_other_work();
+
+/*
+ * We did some other work and now need the IO operation to have completed to
+ * continue.
+ */
+pgaio_wref_wait(&iow);
+
+/*
+ * At this point the IO has completed. We do not yet know whether it succeeded
+ * or failed, however. The buffer's state has been updated, which allows other
+ * backends to use the buffer (if the IO succeeded), or retry the IO (if it
+ * failed).
+ *
+ * Note that in case the IO has failed, a LOG message may have been emitted,
+ * but no ERROR has been raised. This is crucial, as another backend waiting
+ * for this IO should not see an ERROR.
+ *
+ * To check whether the operation succeeded, and to raise an ERROR, or if more
+ * appropriate LOG, the PgAioReturn we passed to pgaio_io_acquire() is used.
+ */
+if (ioret.result.status == PGAIO_RS_ERROR)
+ pgaio_result_report(ioret.result, &ioret.target_data, ERROR);
+
+/*
+ * Besides having succeeded completely, the IO could also have a) partially
+ * completed or b) succeeded with a warning (e.g. due to zero_damaged_pages).
+ * If we e.g. tried to read many blocks at once, the read might have
+ * only succeeded for the first few blocks.
+ *
+ * If the IO partially succeeded and this backend needs all blocks to have
+ * completed, this backend needs to reissue the IO for the remaining buffers.
+ * The AIO subsystem cannot handle this retry transparently.
+ *
+ * As this example is already long, and we only read a single block, we'll just
+ * error out if there's a partial read or a warning.
+ */
+if (ioret.result.status != PGAIO_RS_OK)
+ pgaio_result_report(ioret.result, &ioret.target_data, ERROR);
+
+/*
+ * The IO succeeded, so we can use the buffer now.
+ */
+```
+
+
+## Design Criteria & Motivation
+
+### Deadlock and Starvation Dangers due to AIO
+
+Using AIO in a naive way can easily lead to deadlocks in an environment where
+the source/target of AIO are shared resources, like pages in postgres'
+shared_buffers.
+
+Consider one backend performing readahead on a table, initiating IO for a
+number of buffers ahead of the current "scan position". If that backend then
+performs some operation that blocks, or even just is slow, the IO completion
+for the asynchronously initiated read may not be processed.
+
+This AIO implementation solves this problem by requiring that AIO methods
+either allow AIO completions to be processed by any backend in the system
+(e.g. io_uring), or to guarantee that AIO processing will happen even when the
+issuing backend is blocked (e.g. worker mode, which offloads completion
+processing to the AIO workers).
+
+
+### IO can be started in critical sections
+
+Using AIO for WAL writes can reduce the overhead of WAL logging substantially:
+
+- AIO allows to start WAL writes eagerly, so they complete before needing to
+ wait
+- AIO allows to have multiple WAL flushes in progress at the same time
+- AIO makes it more realistic to use O\_DIRECT + O\_DSYNC, which can reduce
+ the number of roundtrips to storage on some OSs and storage HW (buffered IO
+ and direct IO without O_DSYNC needs to issue a write and after the write's
+ completion a cache flush, whereas O\_DIRECT + O\_DSYNC can use a single
+ Force Unit Access (FUA) write).
+
+The need to be able to execute IO in critical sections has substantial design
+implication on the AIO subsystem. Mainly because completing IOs (see prior
+section) needs to be possible within a critical section, even if the
+to-be-completed IO itself was not issued in a critical section. Consider
+e.g. the case of a backend first starting a number of writes from shared
+buffers and then starting to flush the WAL. Because only a limited amount of
+IO can be in-progress at the same time, initiating IO for flushing the WAL may
+require to first complete IO that was started earlier.
+
+
+### State for AIO needs to live in shared memory
+
+Because postgres uses a process model and because AIOs need to be
+complete-able by any backend much of the state of the AIO subsystem needs to
+live in shared memory.
+
+In an `EXEC_BACKEND` build, a backend's executable code and other process
+local state is not necessarily mapped to the same addresses in each process
+due to ASLR. This means that the shared memory cannot contain pointers to
+callbacks.
+
+
+## Design of the AIO Subsystem
+
+
+### AIO Methods
+
+To achieve portability and performance, multiple methods of performing AIO are
+implemented and others are likely worth adding in the future.
+
+
+#### Synchronous Mode
+
+`io_method=sync` does not actually perform AIO but allows to use the AIO API
+while performing synchronous IO. This can be useful for debugging. The code
+for the synchronous mode is also used as a fallback by e.g. the [worker
+mode](#worker) uses it to execute IO that cannot be executed by workers.
+
+
+#### Worker
+
+`io_method=worker` is available on every platform postgres runs on, and
+implements asynchronous IO - from the view of the issuing process - by
+dispatching the IO to one of several worker processes performing the IO in a
+synchronous manner.
+
+
+#### io_uring
+
+`io_method=io_uring` is available on Linux 5.1+. In contrast to worker mode it
+dispatches all IO from within the process, lowering context switch rate /
+latency.
+
+
+### AIO Handles
+
+The central API piece for postgres' AIO abstraction are AIO handles. To
+execute an IO one first has to acquire an IO handle (`pgaio_io_acquire()`) and
+then "define" it, i.e. associate an IO operation with the handle.
+
+Often AIO handles are acquired on a higher level and then passed to a lower
+level to be fully defined. E.g., for IO to/from shared buffers, bufmgr.c
+routines acquire the handle, which is then passed through smgr.c, md.c to be
+finally fully defined in fd.c.
+
+The functions used at the lowest level to define the operation are
+`pgaio_io_start_*()`.
+
+Because acquisition of an IO handle
+[must always succeed](#io-can-be-started-in-critical-sections)
+and the number of AIO Handles
+[has to be limited](#state-for-aio-needs-to-live-in-shared-memory)
+AIO handles can be reused as soon as they have completed. Obviously code needs
+to be able to react to IO completion. State can be updated using
+[AIO Completion callbacks](#aio-callbacks)
+and the issuing backend can provide a backend local variable to receive the
+result of the IO, as described in
+[AIO Result](#aio-results).
+An IO can be waited for, by both the issuing and any other backend, using
+[AIO References](#aio-wait-references).
+
+
+Because an AIO Handle is not executable just after calling
+`pgaio_io_acquire()` and because `pgaio_io_acquire()` needs to always succeed
+(absent a PANIC), only a single AIO Handle may be acquired (i.e. returned by
+`pgaio_io_acquire()`) without causing the IO to have been defined (by,
+potentially indirectly, causing `pgaio_io_start_*()` to have been
+called). Otherwise a backend could trivially self-deadlock by using up all AIO
+Handles without the ability to wait for some of the IOs to complete.
+
+If it turns out that an AIO Handle is not needed, e.g., because the handle was
+acquired before holding a contended lock, it can be released without being
+defined using `pgaio_io_release()`.
+
+
+### AIO Callbacks
+
+Commonly several layers need to react to completion of an IO. E.g. for a read
+md.c needs to check if the IO outright failed or was shorter than needed,
+bufmgr.c needs to verify the page looks valid and bufmgr.c needs to update the
+BufferDesc to update the buffer's state.
+
+The fact that several layers / subsystems need to react to IO completion poses
+a few challenges:
+
+- Upper layers should not need to know details of lower layers. E.g. bufmgr.c
+ should not assume the IO will pass through md.c. Therefore upper levels
+ cannot know what lower layers would consider an error.
+
+- Lower layers should not need to know about upper layers. E.g. smgr APIs are
+ used going through shared buffers but are also used bypassing shared
+ buffers. This means that e.g. md.c is not in a position to validate
+ checksums.
+
+- Having code in the AIO subsystem for every possible combination of layers
+ would lead to a lot of duplication.
+
+The "solution" to this is the ability to associate multiple completion
+callbacks with a handle. E.g. bufmgr.c can have a callback to update the
+BufferDesc state and to verify the page and md.c can have another callback to
+check if the IO operation was successful.
+
+As [mentioned](#state-for-aio-needs-to-live-in-shared-memory), shared memory
+currently cannot contain function pointers. Because of that completion
+callbacks are not directly identified by function pointers but by IDs
+(`PgAioHandleCallbackID`). A substantial added benefit is that that
+allows callbacks to be identified by much smaller amount of memory (a single
+byte currently).
+
+In addition to completion, AIO callbacks also are called to "stage" an
+IO. This is, e.g., used to increase buffer reference counts to account for the
+AIO subsystem referencing the buffer, which is required to handle the case
+where the issuing backend errors out and releases its own pins while the IO is
+still ongoing.
+
+As [explained earlier](#io-can-be-started-in-critical-sections) IO completions
+need to be safe to execute in critical sections. To allow the backend that
+issued the IO to error out in case of failure [AIO Result](#aio-results) can
+be used.
+
+
+### AIO Targets
+
+In addition to the completion callbacks describe above, each AIO Handle has
+exactly one "target". Each target has some space inside an AIO Handle with
+information specific to the target and can provide callbacks to allow to
+reopen the underlying file (required for worker mode) and to describe the IO
+operation (used for debug logging and error messages).
+
+I.e., if two different uses of AIO can describe the identity of the file being
+operated on the same way, it likely makes sense to use the same
+target. E.g. different smgr implementations can describe IO with
+RelFileLocator, ForkNumber and BlockNumber and can thus share a target. In
+contrast, IO for a WAL file would be described with TimeLineID and XLogRecPtr
+and it would not make sense to use the same target for smgr and WAL.
+
+
+### AIO Wait References
+
+As [described above](#aio-handles), AIO Handles can be reused immediately
+after completion and therefore cannot be used to wait for completion of the
+IO. Waiting is enabled using AIO wait references, which do not just identify
+an AIO Handle but also include the handles "generation".
+
+A reference to an AIO Handle can be acquired using `pgaio_io_get_wref()` and
+then waited upon using `pgaio_wref_wait()`.
+
+
+### AIO Results
+
+As AIO completion callbacks
+[are executed in critical sections](#io-can-be-started-in-critical-sections)
+and [may be executed by any backend](#deadlock-and-starvation-dangers-due-to-aio)
+completion callbacks cannot be used to, e.g., make the query that triggered an
+IO ERROR out.
+
+To allow to react to failing IOs the issuing backend can pass a pointer to a
+`PgAioReturn` in backend local memory. Before an AIO Handle is reused the
+`PgAioReturn` is filled with information about the IO. This includes
+information about whether the IO was successful (as a value of
+`PgAioResultStatus`) and enough information to raise an error in case of a
+failure (via `pgaio_result_report()`, with the error details encoded in
+`PgAioResult`).
+
+
+### AIO Errors
+
+It would be very convenient to have shared completion callbacks encode the
+details of errors as an `ErrorData` that could be raised at a later
+time. Unfortunately doing so would require allocating memory. While elog.c can
+guarantee (well, kinda) that logging a message will not run out of memory,
+that only works because a very limited number of messages are in the process
+of being logged. With AIO a large number of concurrently issued AIOs might
+fail.
+
+To avoid the need for preallocating a potentially large amount of memory (in
+shared memory no less!), completion callbacks instead have to encode errors in
+a more compact format that can be converted into an error message.
+
+
+## Helpers
+
+Using the low-level AIO API introduces too much complexity to do so all over
+the tree. Most uses of AIO should be done via reusable, higher-level,
+helpers.
+
+
+### Read Stream
+
+A common and very beneficial use of AIO are reads where a substantial number
+of to-be-read locations are known ahead of time. E.g., for a sequential scan
+the set of blocks that need to be read can be determined solely by knowing the
+current position and checking the buffer mapping table.
+
+The [Read Stream](../../../include/storage/read_stream.h) interface makes it
+comparatively easy to use AIO for such use cases.
diff --git a/src/backend/storage/aio/aio.c b/src/backend/storage/aio/aio.c
index e3ed087e8a2..86f7250b7a5 100644
--- a/src/backend/storage/aio/aio.c
+++ b/src/backend/storage/aio/aio.c
@@ -24,6 +24,8 @@
*
* - read_stream.c - helper for reading buffered relation data
*
+ * - README.md - higher-level overview over AIO
+ *
*
* Portions Copyright (c) 1996-2025, PostgreSQL Global Development Group
* Portions Copyright (c) 1994, Regents of the University of California
--
2.48.1.76.g4e746b1a31.dirty
[text/x-diff] v2.14-0014-aio-Basic-read_stream-adjustments-for-real-AIO.patch (4.9K, ../../d4ff3qxudxn2ze3zk2xwhds5ysezfei3uqeuimymj624t5chsd@sgr2h3rmln5t/15-v2.14-0014-aio-Basic-read_stream-adjustments-for-real-AIO.patch)
download | inline diff:
From a07f930bca25979c3ab78ec0353a109bb484846e Mon Sep 17 00:00:00 2001
From: Andres Freund <andres@anarazel.de>
Date: Tue, 18 Mar 2025 14:40:06 -0400
Subject: [PATCH v2.14 14/29] aio: Basic read_stream adjustments for real AIO
Adapt the read stream logic for real AIO:
- If AIO is enabled, we shouldn't issue advice, but if it isn't, we should
continue issuing advice
- AIO benefits from reading ahead with direct IO
- If effective_io_concurrency=0, pass READ_BUFFERS_SYNCHRONOUSLY to
StartReadBuffers() to ensure synchronous IO execution
There are further improvements we should consider:
- While in read_stream_look_ahead(), we can use AIO batch submission mode for
increased efficiency. That however requires care to avoid deadlocks and thus
done separately.
- It can be beneficial to defer starting new IOs until we can issue multiple
IOs at once. That however requires non-trivial heuristics to decide when to
do so.
Co-authored-by: Andres Freund <andres@anarazel.de>
Co-authored-by: Thomas Munro <thomas.munro@gmail.com>
---
src/backend/storage/aio/read_stream.c | 39 ++++++++++++++++++---------
1 file changed, 26 insertions(+), 13 deletions(-)
diff --git a/src/backend/storage/aio/read_stream.c b/src/backend/storage/aio/read_stream.c
index c60e37e7f7f..26e5dfe77db 100644
--- a/src/backend/storage/aio/read_stream.c
+++ b/src/backend/storage/aio/read_stream.c
@@ -72,6 +72,7 @@
#include "postgres.h"
#include "miscadmin.h"
+#include "storage/aio.h"
#include "storage/fd.h"
#include "storage/smgr.h"
#include "storage/read_stream.h"
@@ -99,6 +100,8 @@ struct ReadStream
int16 pinned_buffers;
int16 distance;
int16 initialized_buffers;
+ int read_buffers_flags;
+ bool sync_mode; /* using io_method=sync */
bool advice_enabled;
bool temporary;
@@ -250,7 +253,7 @@ read_stream_start_pending_read(ReadStream *stream)
Assert(stream->next_buffer_index == stream->oldest_buffer_index);
/* Do we need to issue read-ahead advice? */
- flags = 0;
+ flags = stream->read_buffers_flags;
if (stream->advice_enabled)
{
if (stream->pending_read_blocknum == stream->seq_blocknum)
@@ -261,7 +264,7 @@ read_stream_start_pending_read(ReadStream *stream)
* then stay of the way of the kernel's own read-ahead.
*/
if (stream->seq_until_processed != InvalidBlockNumber)
- flags = READ_BUFFERS_ISSUE_ADVICE;
+ flags |= READ_BUFFERS_ISSUE_ADVICE;
}
else
{
@@ -272,7 +275,7 @@ read_stream_start_pending_read(ReadStream *stream)
*/
stream->seq_until_processed = stream->pending_read_blocknum;
if (stream->pinned_buffers > 0)
- flags = READ_BUFFERS_ISSUE_ADVICE;
+ flags |= READ_BUFFERS_ISSUE_ADVICE;
}
}
@@ -613,27 +616,33 @@ read_stream_begin_impl(int flags,
stream->per_buffer_data = (void *)
MAXALIGN(&stream->ios[Max(1, max_ios)]);
+ stream->sync_mode = io_method == IOMETHOD_SYNC;
+
#ifdef USE_PREFETCH
/*
- * This system supports prefetching advice. We can use it as long as
- * direct I/O isn't enabled, the caller hasn't promised sequential access
- * (overriding our detection heuristics), and max_ios hasn't been set to
- * zero.
+ * Read-ahead advice simulating asynchronous I/O with synchronous calls.
+ * Issue advice only if AIO is not used, direct I/O isn't enabled, the
+ * caller hasn't promised sequential access (overriding our detection
+ * heuristics), and max_ios hasn't been set to zero.
*/
- if ((io_direct_flags & IO_DIRECT_DATA) == 0 &&
+ if (stream->sync_mode &&
+ (io_direct_flags & IO_DIRECT_DATA) == 0 &&
(flags & READ_STREAM_SEQUENTIAL) == 0 &&
max_ios > 0)
stream->advice_enabled = true;
#endif
/*
- * For now, max_ios = 0 is interpreted as max_ios = 1 with advice disabled
- * above. If we had real asynchronous I/O we might need a slightly
- * different definition.
+ * Setting max_ios to zero disables AIO and advice-based pseudo AIO, but
+ * we still need to allocate space to combine and run one I/O. Bump it up
+ * to one, and remember to ask for synchronous I/O only.
*/
if (max_ios == 0)
+ {
max_ios = 1;
+ stream->read_buffers_flags = READ_BUFFERS_SYNCHRONOUSLY;
+ }
/*
* Capture stable values for these two GUC-derived numbers for the
@@ -777,6 +786,11 @@ read_stream_next_buffer(ReadStream *stream, void **per_buffer_data)
if (likely(next_blocknum != InvalidBlockNumber))
{
+ int flags = stream->read_buffers_flags;
+
+ if (stream->advice_enabled)
+ flags |= READ_BUFFERS_ISSUE_ADVICE;
+
/*
* Pin a buffer for the next call. Same buffer entry, and
* arbitrary I/O entry (they're all free). We don't have to
@@ -792,8 +806,7 @@ read_stream_next_buffer(ReadStream *stream, void **per_buffer_data)
if (likely(!StartReadBuffer(&stream->ios[0].op,
&stream->buffers[oldest_buffer_index],
next_blocknum,
- stream->advice_enabled ?
- READ_BUFFERS_ISSUE_ADVICE : 0)))
+ flags)))
{
/* Fast return. */
return buffer;
--
2.48.1.76.g4e746b1a31.dirty
[text/x-diff] v2.14-0015-read_stream-Introduce-and-use-optional-batchmo.patch (13.1K, ../../d4ff3qxudxn2ze3zk2xwhds5ysezfei3uqeuimymj624t5chsd@sgr2h3rmln5t/16-v2.14-0015-read_stream-Introduce-and-use-optional-batchmo.patch)
download | inline diff:
From 23e7b3f541198d53d376011fd84fffda55ad2819 Mon Sep 17 00:00:00 2001
From: Andres Freund <andres@anarazel.de>
Date: Mon, 24 Mar 2025 17:30:42 -0400
Subject: [PATCH v2.14 15/29] read_stream: Introduce and use optional batchmode
support
Submitting IO in larger batches can be more efficient than doing so
one-by-one, particularly for many small reads. It does, however, require
the ReadStreamBlockNumberCB callback to abide by the restrictions of AIO
batching (c.f. pgaio_enter_batchmode()). Basically, the callback may not:
a) block without first calling pgaio_submit_staged(), unless a
to-be-waited-on lock cannot be part of a deadlock, e.g. because it is
never held while waiting for IO.
b) directly or indirectly start another batch pgaio_enter_batchmode()
As this requires care and is nontrivial in some cases, batching is only
used with explicit opt-in.
This patch adds an explicit flag (READ_STREAM_USE_BATCHING) to read_stream and
uses it where appropriate.
There are two cases where batching would likely be beneficial, but where we
aren't using it yet:
1) bitmap heap scans, because the callback reads the VM
This should soon be solved, because we are planning to remove the use of
the VM, due to that not being sound.
2) The first phase of heap vacuum
This could be made to support batchmode, but would require some care.
Discussion: https://postgr.es/m/uvrtrknj4kdytuboidbhwclo4gxhswwcpgadptsjvjqcluzmah%40brqs62irg4dt
---
src/include/storage/read_stream.h | 21 +++++++++++++++++++++
src/backend/access/gist/gistvacuum.c | 8 +++++++-
src/backend/access/heap/heapam.c | 18 +++++++++++++++++-
src/backend/access/heap/vacuumlazy.c | 21 ++++++++++++++++++---
src/backend/access/nbtree/nbtree.c | 8 +++++++-
src/backend/access/spgist/spgvacuum.c | 8 +++++++-
src/backend/commands/analyze.c | 7 ++++++-
src/backend/storage/aio/read_stream.c | 16 ++++++++++++++++
src/backend/storage/buffer/bufmgr.c | 8 +++++++-
contrib/pg_prewarm/pg_prewarm.c | 7 ++++++-
contrib/pg_visibility/pg_visibility.c | 8 +++++++-
11 files changed, 119 insertions(+), 11 deletions(-)
diff --git a/src/include/storage/read_stream.h b/src/include/storage/read_stream.h
index c11d8ce3300..9b0d65161d0 100644
--- a/src/include/storage/read_stream.h
+++ b/src/include/storage/read_stream.h
@@ -42,6 +42,27 @@
*/
#define READ_STREAM_FULL 0x04
+/* ---
+ * Opt-in to using AIO batchmode.
+ *
+ * Submitting IO in larger batches can be more efficient than doing so
+ * one-by-one, particularly for many small reads. It does, however, require
+ * the ReadStreamBlockNumberCB callback to abide by the restrictions of AIO
+ * batching (c.f. pgaio_enter_batchmode()). Basically, the callback may not:
+ *
+ * a) block without first calling pgaio_submit_staged(), unless a
+ * to-be-waited-on lock cannot be part of a deadlock, e.g. because it is
+ * never held while waiting for IO.
+ *
+ * b) start another batch (without first exiting batchmode and re-entering
+ * before returning)
+ *
+ * As this requires care and is nontrivial in some cases, batching is only
+ * used with explicit opt-in.
+ * ---
+ */
+#define READ_STREAM_USE_BATCHING 0x08
+
struct ReadStream;
typedef struct ReadStream ReadStream;
diff --git a/src/backend/access/gist/gistvacuum.c b/src/backend/access/gist/gistvacuum.c
index 20b1bb5dbac..ce9d78d78d6 100644
--- a/src/backend/access/gist/gistvacuum.c
+++ b/src/backend/access/gist/gistvacuum.c
@@ -210,7 +210,13 @@ gistvacuumscan(IndexVacuumInfo *info, IndexBulkDeleteResult *stats,
needLock = !RELATION_IS_LOCAL(rel);
p.current_blocknum = GIST_ROOT_BLKNO;
- stream = read_stream_begin_relation(READ_STREAM_FULL,
+
+ /*
+ * It is safe to use batchmode as block_range_read_stream_cb takes no
+ * locks.
+ */
+ stream = read_stream_begin_relation(READ_STREAM_FULL |
+ READ_STREAM_USE_BATCHING,
info->strategy,
rel,
MAIN_FORKNUM,
diff --git a/src/backend/access/heap/heapam.c b/src/backend/access/heap/heapam.c
index b12b583c4d9..c8357660776 100644
--- a/src/backend/access/heap/heapam.c
+++ b/src/backend/access/heap/heapam.c
@@ -1206,7 +1206,15 @@ heap_beginscan(Relation relation, Snapshot snapshot,
else
cb = heap_scan_stream_read_next_serial;
- scan->rs_read_stream = read_stream_begin_relation(READ_STREAM_SEQUENTIAL,
+ /* ---
+ * It is safe to use batchmode as the only locks taken by `cb`
+ * are never taken while waiting for IO:
+ * - SyncScanLock is used in the non-parallel case
+ * - in the parallel case, only spinlocks and atomics are used
+ * ---
+ */
+ scan->rs_read_stream = read_stream_begin_relation(READ_STREAM_SEQUENTIAL |
+ READ_STREAM_USE_BATCHING,
scan->rs_strategy,
scan->rs_base.rs_rd,
MAIN_FORKNUM,
@@ -1216,6 +1224,14 @@ heap_beginscan(Relation relation, Snapshot snapshot,
}
else if (scan->rs_base.rs_flags & SO_TYPE_BITMAPSCAN)
{
+ /*
+ * Currently we can't trivially use batching, due to the
+ * VM_ALL_VISIBLE check in bitmapheap_stream_read_next. While that
+ * could be made safe, we are about to remove the all-visible logic
+ * from bitmap scans due to its unsoundness.
+ *
+ * FIXME: Should be changed soon!
+ */
scan->rs_read_stream = read_stream_begin_relation(READ_STREAM_DEFAULT,
scan->rs_strategy,
scan->rs_base.rs_rd,
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index 6d287b38cf5..f28326bad09 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -1225,7 +1225,12 @@ lazy_scan_heap(LVRelState *vacrel)
vacrel->next_unskippable_eager_scanned = false;
vacrel->next_unskippable_vmbuffer = InvalidBuffer;
- /* Set up the read stream for vacuum's first pass through the heap */
+ /*
+ * Set up the read stream for vacuum's first pass through the heap.
+ *
+ * This could be made safe for READ_STREAM_USE_BATCHING, but only with
+ * explicit work in heap_vac_scan_next_block.
+ */
stream = read_stream_begin_relation(READ_STREAM_MAINTENANCE,
vacrel->bstrategy,
vacrel->rel,
@@ -2669,6 +2674,8 @@ lazy_vacuum_all_indexes(LVRelState *vacrel)
* Read stream callback for vacuum's third phase (second pass over the heap).
* Gets the next block from the TID store and returns it or InvalidBlockNumber
* if there are no further blocks to vacuum.
+ *
+ * NB: Assumed to be safe to use with READ_STREAM_USE_BATCHING.
*/
static BlockNumber
vacuum_reap_lp_read_stream_next(ReadStream *stream,
@@ -2732,8 +2739,16 @@ lazy_vacuum_heap_rel(LVRelState *vacrel)
iter = TidStoreBeginIterate(vacrel->dead_items);
- /* Set up the read stream for vacuum's second pass through the heap */
- stream = read_stream_begin_relation(READ_STREAM_MAINTENANCE,
+ /*
+ * Set up the read stream for vacuum's second pass through the heap.
+ *
+ * It is safe to use batchmode, as vacuum_reap_lp_read_stream_next() does
+ * not need to wait for IO and does not perform locking. Once we support
+ * parallelism it should still be fine, as presumably the holder of locks
+ * would never be blocked by IO while holding the lock.
+ */
+ stream = read_stream_begin_relation(READ_STREAM_MAINTENANCE |
+ READ_STREAM_USE_BATCHING,
vacrel->bstrategy,
vacrel->rel,
MAIN_FORKNUM,
diff --git a/src/backend/access/nbtree/nbtree.c b/src/backend/access/nbtree/nbtree.c
index 80b04d6ca2a..4a0bf069f99 100644
--- a/src/backend/access/nbtree/nbtree.c
+++ b/src/backend/access/nbtree/nbtree.c
@@ -1064,7 +1064,13 @@ btvacuumscan(IndexVacuumInfo *info, IndexBulkDeleteResult *stats,
needLock = !RELATION_IS_LOCAL(rel);
p.current_blocknum = BTREE_METAPAGE + 1;
- stream = read_stream_begin_relation(READ_STREAM_FULL,
+
+ /*
+ * It is safe to use batchmode as block_range_read_stream_cb takes no
+ * locks.
+ */
+ stream = read_stream_begin_relation(READ_STREAM_FULL |
+ READ_STREAM_USE_BATCHING,
info->strategy,
rel,
MAIN_FORKNUM,
diff --git a/src/backend/access/spgist/spgvacuum.c b/src/backend/access/spgist/spgvacuum.c
index 77deb226b7e..b3df2d89074 100644
--- a/src/backend/access/spgist/spgvacuum.c
+++ b/src/backend/access/spgist/spgvacuum.c
@@ -822,7 +822,13 @@ spgvacuumscan(spgBulkDeleteState *bds)
/* We can skip locking for new or temp relations */
needLock = !RELATION_IS_LOCAL(index);
p.current_blocknum = SPGIST_METAPAGE_BLKNO + 1;
- stream = read_stream_begin_relation(READ_STREAM_FULL,
+
+ /*
+ * It is safe to use batchmode as block_range_read_stream_cb takes no
+ * locks.
+ */
+ stream = read_stream_begin_relation(READ_STREAM_FULL |
+ READ_STREAM_USE_BATCHING,
bds->info->strategy,
index,
MAIN_FORKNUM,
diff --git a/src/backend/commands/analyze.c b/src/backend/commands/analyze.c
index ca76c0d2668..4fffb76e557 100644
--- a/src/backend/commands/analyze.c
+++ b/src/backend/commands/analyze.c
@@ -1237,7 +1237,12 @@ acquire_sample_rows(Relation onerel, int elevel,
scan = table_beginscan_analyze(onerel);
slot = table_slot_create(onerel, NULL);
- stream = read_stream_begin_relation(READ_STREAM_MAINTENANCE,
+ /*
+ * It is safe to use batching, as block_sampling_read_stream_next never
+ * blocks.
+ */
+ stream = read_stream_begin_relation(READ_STREAM_MAINTENANCE |
+ READ_STREAM_USE_BATCHING,
vac_strategy,
scan->rs_rd,
MAIN_FORKNUM,
diff --git a/src/backend/storage/aio/read_stream.c b/src/backend/storage/aio/read_stream.c
index 26e5dfe77db..36c54fb695b 100644
--- a/src/backend/storage/aio/read_stream.c
+++ b/src/backend/storage/aio/read_stream.c
@@ -102,6 +102,7 @@ struct ReadStream
int16 initialized_buffers;
int read_buffers_flags;
bool sync_mode; /* using io_method=sync */
+ bool batch_mode; /* READ_STREAM_USE_BATCHING */
bool advice_enabled;
bool temporary;
@@ -403,6 +404,15 @@ read_stream_start_pending_read(ReadStream *stream)
static void
read_stream_look_ahead(ReadStream *stream)
{
+ /*
+ * Allow amortizing the cost of submitting IO over multiple IOs. This
+ * requires that we don't do any operations that could lead to a deadlock
+ * with staged-but-unsubmitted IO. The callback needs to opt-in to being
+ * careful.
+ */
+ if (stream->batch_mode)
+ pgaio_enter_batchmode();
+
while (stream->ios_in_progress < stream->max_ios &&
stream->pinned_buffers + stream->pending_read_nblocks < stream->distance)
{
@@ -450,6 +460,8 @@ read_stream_look_ahead(ReadStream *stream)
{
/* We've hit the buffer or I/O limit. Rewind and stop here. */
read_stream_unget_block(stream, blocknum);
+ if (stream->batch_mode)
+ pgaio_exit_batchmode();
return;
}
}
@@ -484,6 +496,9 @@ read_stream_look_ahead(ReadStream *stream)
* time.
*/
Assert(stream->pinned_buffers > 0 || stream->distance == 0);
+
+ if (stream->batch_mode)
+ pgaio_exit_batchmode();
}
/*
@@ -617,6 +632,7 @@ read_stream_begin_impl(int flags,
MAXALIGN(&stream->ios[Max(1, max_ios)]);
stream->sync_mode = io_method == IOMETHOD_SYNC;
+ stream->batch_mode = flags & READ_STREAM_USE_BATCHING;
#ifdef USE_PREFETCH
diff --git a/src/backend/storage/buffer/bufmgr.c b/src/backend/storage/buffer/bufmgr.c
index 6ac1a821523..39cc6489145 100644
--- a/src/backend/storage/buffer/bufmgr.c
+++ b/src/backend/storage/buffer/bufmgr.c
@@ -5100,7 +5100,13 @@ RelationCopyStorageUsingBuffer(RelFileLocator srclocator,
p.current_blocknum = 0;
p.last_exclusive = nblocks;
src_smgr = smgropen(srclocator, INVALID_PROC_NUMBER);
- src_stream = read_stream_begin_smgr_relation(READ_STREAM_FULL,
+
+ /*
+ * It is safe to use batchmode as block_range_read_stream_cb takes no
+ * locks.
+ */
+ src_stream = read_stream_begin_smgr_relation(READ_STREAM_FULL |
+ READ_STREAM_USE_BATCHING,
bstrategy_src,
src_smgr,
permanent ? RELPERSISTENCE_PERMANENT : RELPERSISTENCE_UNLOGGED,
diff --git a/contrib/pg_prewarm/pg_prewarm.c b/contrib/pg_prewarm/pg_prewarm.c
index 63faf43d0bf..c0efb530c4e 100644
--- a/contrib/pg_prewarm/pg_prewarm.c
+++ b/contrib/pg_prewarm/pg_prewarm.c
@@ -198,7 +198,12 @@ pg_prewarm(PG_FUNCTION_ARGS)
p.current_blocknum = first_block;
p.last_exclusive = last_block + 1;
- stream = read_stream_begin_relation(READ_STREAM_FULL,
+ /*
+ * It is safe to use batchmode as block_range_read_stream_cb takes no
+ * locks.
+ */
+ stream = read_stream_begin_relation(READ_STREAM_FULL |
+ READ_STREAM_USE_BATCHING,
NULL,
rel,
forkNumber,
diff --git a/contrib/pg_visibility/pg_visibility.c b/contrib/pg_visibility/pg_visibility.c
index ca91819852c..d79ef35006b 100644
--- a/contrib/pg_visibility/pg_visibility.c
+++ b/contrib/pg_visibility/pg_visibility.c
@@ -526,7 +526,13 @@ collect_visibility_data(Oid relid, bool include_pd)
{
p.current_blocknum = 0;
p.last_exclusive = nblocks;
- stream = read_stream_begin_relation(READ_STREAM_FULL,
+
+ /*
+ * It is safe to use batchmode as block_range_read_stream_cb takes no
+ * locks.
+ */
+ stream = read_stream_begin_relation(READ_STREAM_FULL |
+ READ_STREAM_USE_BATCHING,
bstrategy,
rel,
MAIN_FORKNUM,
--
2.48.1.76.g4e746b1a31.dirty
[text/x-diff] v2.14-0016-docs-Reframe-track_io_timing-related-docs-as-w.patch (5.3K, ../../d4ff3qxudxn2ze3zk2xwhds5ysezfei3uqeuimymj624t5chsd@sgr2h3rmln5t/17-v2.14-0016-docs-Reframe-track_io_timing-related-docs-as-w.patch)
download | inline diff:
From 49107bf7479af7d6099cb409cea2e0a5179f346d Mon Sep 17 00:00:00 2001
From: Andres Freund <andres@anarazel.de>
Date: Tue, 18 Mar 2025 14:40:06 -0400
Subject: [PATCH v2.14 16/29] docs: Reframe track_io_timing related docs as
wait time
With AIO it does not make sense anymore to track the time for each individual
IO, as multiple IOs can be in-flight at the same time. Instead we now track
the time spent *waiting* for IOs.
This should be reflected in the docs. While, so far, we only do a subset of
reads, and no other operations, via AIO, describing the GUC and view columns
as measuring IO waits is accurate for synchronous and asynchronous IO.
Discussion: https://postgr.es/m/5dzyoduxlvfg55oqtjyjehez5uoq6hnwgzor4kkybkfdgkj7ag@rbi4gsmzaczk
---
doc/src/sgml/config.sgml | 4 ++--
doc/src/sgml/monitoring.sgml | 12 ++++++------
2 files changed, 8 insertions(+), 8 deletions(-)
diff --git a/doc/src/sgml/config.sgml b/doc/src/sgml/config.sgml
index 65ab95be370..f55f38cb85b 100644
--- a/doc/src/sgml/config.sgml
+++ b/doc/src/sgml/config.sgml
@@ -8568,7 +8568,7 @@ COPY postgres_log FROM '/full/path/to/logfile.csv' WITH csv;
</term>
<listitem>
<para>
- Enables timing of database I/O calls. This parameter is off by
+ Enables timing of database I/O waits. This parameter is off by
default, as it will repeatedly query the operating system for
the current time, which may cause significant overhead on some
platforms. You can use the <xref linkend="pgtesttiming"/> tool to
@@ -8602,7 +8602,7 @@ COPY postgres_log FROM '/full/path/to/logfile.csv' WITH csv;
</term>
<listitem>
<para>
- Enables timing of WAL I/O calls. This parameter is off by default,
+ Enables timing of WAL I/O waits. This parameter is off by default,
as it will repeatedly query the operating system for the current time,
which may cause significant overhead on some platforms.
You can use the <application>pg_test_timing</application> tool to
diff --git a/doc/src/sgml/monitoring.sgml b/doc/src/sgml/monitoring.sgml
index bacc09cb8af..a6d67d2fbaa 100644
--- a/doc/src/sgml/monitoring.sgml
+++ b/doc/src/sgml/monitoring.sgml
@@ -2747,7 +2747,7 @@ description | Waiting for a newly initialized WAL file to reach durable storage
<structfield>read_time</structfield> <type>double precision</type>
</para>
<para>
- Time spent in read operations in milliseconds (if
+ Time spent waiting for read operations in milliseconds (if
<xref linkend="guc-track-io-timing"/> is enabled and
<varname>object</varname> is not <literal>wal</literal>,
or if <xref linkend="guc-track-wal-io-timing"/> is enabled
@@ -2785,7 +2785,7 @@ description | Waiting for a newly initialized WAL file to reach durable storage
<structfield>write_time</structfield> <type>double precision</type>
</para>
<para>
- Time spent in write operations in milliseconds (if
+ Time spent waiting for write operations in milliseconds (if
<xref linkend="guc-track-io-timing"/> is enabled and
<varname>object</varname> is not <literal>wal</literal>,
or if <xref linkend="guc-track-wal-io-timing"/> is enabled
@@ -2813,7 +2813,7 @@ description | Waiting for a newly initialized WAL file to reach durable storage
<structfield>writeback_time</structfield> <type>double precision</type>
</para>
<para>
- Time spent in writeback operations in milliseconds (if
+ Time spent waiting for writeback operations in milliseconds (if
<xref linkend="guc-track-io-timing"/> is enabled, otherwise zero). This
includes the time spent queueing write-out requests and, potentially,
the time spent to write out the dirty data.
@@ -2849,7 +2849,7 @@ description | Waiting for a newly initialized WAL file to reach durable storage
<structfield>extend_time</structfield> <type>double precision</type>
</para>
<para>
- Time spent in extend operations in milliseconds. (if
+ Time spent waiting for extend operations in milliseconds. (if
<xref linkend="guc-track-io-timing"/> is enabled and
<varname>object</varname> is not <literal>wal</literal>,
or if <xref linkend="guc-track-wal-io-timing"/> is enabled
@@ -2923,7 +2923,7 @@ description | Waiting for a newly initialized WAL file to reach durable storage
<structfield>fsync_time</structfield> <type>double precision</type>
</para>
<para>
- Time spent in fsync operations in milliseconds (if
+ Time spent waiting for fsync operations in milliseconds (if
<xref linkend="guc-track-io-timing"/> is enabled and
<varname>object</varname> is not <literal>wal</literal>,
or if <xref linkend="guc-track-wal-io-timing"/> is enabled
@@ -3010,7 +3010,7 @@ description | Waiting for a newly initialized WAL file to reach durable storage
<note>
<para>
- Columns tracking I/O time will only be non-zero when
+ Columns tracking I/O wait time will only be non-zero when
<xref linkend="guc-track-io-timing"/> is enabled. The user should be
careful when referencing these columns in combination with their
corresponding I/O operations in case <varname>track_io_timing</varname>
--
2.48.1.76.g4e746b1a31.dirty
[text/x-diff] v2.14-0017-Enable-IO-concurrency-on-all-systems.patch (11.1K, ../../d4ff3qxudxn2ze3zk2xwhds5ysezfei3uqeuimymj624t5chsd@sgr2h3rmln5t/18-v2.14-0017-Enable-IO-concurrency-on-all-systems.patch)
download | inline diff:
From f181f271ba8ab468d5170014c78de90963b9e3f5 Mon Sep 17 00:00:00 2001
From: Andres Freund <andres@anarazel.de>
Date: Wed, 19 Mar 2025 10:15:20 -0400
Subject: [PATCH v2.14 17/29] Enable IO concurrency on all systems
Previously effective_io_concurrency and maintenance_io_concurrency could
not be set above 0 on machines without fadvise support. AIO enables
IO concurrency without such support.
Currently only subsystems using the read stream API will take advantage
of this. Other users of maintenance_io_concurrency (like recovery
prefetching) which leverage OS advice directly will not benefit from
this change. In those cases, maintenance_io_concurrency will have no
effect on I/O behavior.
Author: Melanie Plageman <melanieplageman@gmail.com>
Reviewed-by: Noah Misch <noah@leadboat.com>
Discussion: https://postgr.es/m/CAAKRu_atGgZePo=_g6T3cNtfMf0QxpvoUh5OUqa_cnPdhLd=gw@mail.gmail.com
---
src/include/storage/bufmgr.h | 6 ----
src/include/utils/guc_hooks.h | 4 ---
src/backend/access/common/reloptions.c | 8 -----
src/backend/commands/variable.c | 30 -------------------
src/backend/utils/misc/guc_tables.c | 4 +--
src/backend/utils/misc/postgresql.conf.sample | 4 +--
src/bin/initdb/initdb.c | 5 ----
doc/src/sgml/config.sgml | 16 +++++-----
doc/src/sgml/ref/alter_tablespace.sgml | 2 +-
doc/src/sgml/ref/create_tablespace.sgml | 2 +-
10 files changed, 14 insertions(+), 67 deletions(-)
diff --git a/src/include/storage/bufmgr.h b/src/include/storage/bufmgr.h
index 867ae9facb5..f2192ceb271 100644
--- a/src/include/storage/bufmgr.h
+++ b/src/include/storage/bufmgr.h
@@ -158,14 +158,8 @@ extern PGDLLIMPORT int bgwriter_lru_maxpages;
extern PGDLLIMPORT double bgwriter_lru_multiplier;
extern PGDLLIMPORT bool track_io_timing;
-/* only applicable when prefetching is available */
-#ifdef USE_PREFETCH
#define DEFAULT_EFFECTIVE_IO_CONCURRENCY 16
#define DEFAULT_MAINTENANCE_IO_CONCURRENCY 16
-#else
-#define DEFAULT_EFFECTIVE_IO_CONCURRENCY 0
-#define DEFAULT_MAINTENANCE_IO_CONCURRENCY 0
-#endif
extern PGDLLIMPORT int effective_io_concurrency;
extern PGDLLIMPORT int maintenance_io_concurrency;
diff --git a/src/include/utils/guc_hooks.h b/src/include/utils/guc_hooks.h
index 0f1e74f96c9..799fa7ace68 100644
--- a/src/include/utils/guc_hooks.h
+++ b/src/include/utils/guc_hooks.h
@@ -61,8 +61,6 @@ extern bool check_default_text_search_config(char **newval, void **extra, GucSou
extern void assign_default_text_search_config(const char *newval, void *extra);
extern bool check_default_with_oids(bool *newval, void **extra,
GucSource source);
-extern bool check_effective_io_concurrency(int *newval, void **extra,
- GucSource source);
extern bool check_huge_page_size(int *newval, void **extra, GucSource source);
extern void assign_io_method(int newval, void *extra);
extern bool check_io_max_concurrency(int *newval, void **extra, GucSource source);
@@ -83,8 +81,6 @@ extern bool check_log_stats(bool *newval, void **extra, GucSource source);
extern bool check_log_timezone(char **newval, void **extra, GucSource source);
extern void assign_log_timezone(const char *newval, void *extra);
extern const char *show_log_timezone(void);
-extern bool check_maintenance_io_concurrency(int *newval, void **extra,
- GucSource source);
extern void assign_maintenance_io_concurrency(int newval, void *extra);
extern void assign_io_max_combine_limit(int newval, void *extra);
extern void assign_io_combine_limit(int newval, void *extra);
diff --git a/src/backend/access/common/reloptions.c b/src/backend/access/common/reloptions.c
index 645b5c00467..46c1dce222d 100644
--- a/src/backend/access/common/reloptions.c
+++ b/src/backend/access/common/reloptions.c
@@ -361,11 +361,7 @@ static relopt_int intRelOpts[] =
RELOPT_KIND_TABLESPACE,
ShareUpdateExclusiveLock
},
-#ifdef USE_PREFETCH
-1, 0, MAX_IO_CONCURRENCY
-#else
- 0, 0, 0
-#endif
},
{
{
@@ -374,11 +370,7 @@ static relopt_int intRelOpts[] =
RELOPT_KIND_TABLESPACE,
ShareUpdateExclusiveLock
},
-#ifdef USE_PREFETCH
-1, 0, MAX_IO_CONCURRENCY
-#else
- 0, 0, 0
-#endif
},
{
{
diff --git a/src/backend/commands/variable.c b/src/backend/commands/variable.c
index 84f044a1959..a9f2a3a3062 100644
--- a/src/backend/commands/variable.c
+++ b/src/backend/commands/variable.c
@@ -1145,7 +1145,6 @@ check_cluster_name(char **newval, void **extra, GucSource source)
void
assign_maintenance_io_concurrency(int newval, void *extra)
{
-#ifdef USE_PREFETCH
/*
* Reconfigure recovery prefetching, because a setting it depends on
* changed.
@@ -1153,7 +1152,6 @@ assign_maintenance_io_concurrency(int newval, void *extra)
maintenance_io_concurrency = newval;
if (AmStartupProcess())
XLogPrefetchReconfigure();
-#endif
}
/*
@@ -1249,34 +1247,6 @@ check_default_with_oids(bool *newval, void **extra, GucSource source)
return true;
}
-bool
-check_effective_io_concurrency(int *newval, void **extra, GucSource source)
-{
-#ifndef USE_PREFETCH
- if (*newval != 0)
- {
- GUC_check_errdetail("\"%s\" must be set to 0 on platforms that lack support for issuing read-ahead advice.",
- "effective_io_concurrency");
- return false;
- }
-#endif /* USE_PREFETCH */
- return true;
-}
-
-bool
-check_maintenance_io_concurrency(int *newval, void **extra, GucSource source)
-{
-#ifndef USE_PREFETCH
- if (*newval != 0)
- {
- GUC_check_errdetail("\"%s\" must be set to 0 on platforms that lack support for issuing read-ahead advice.",
- "maintenance_io_concurrency");
- return false;
- }
-#endif /* USE_PREFETCH */
- return true;
-}
-
bool
check_ssl(bool *newval, void **extra, GucSource source)
{
diff --git a/src/backend/utils/misc/guc_tables.c b/src/backend/utils/misc/guc_tables.c
index 76c7c6bb4b1..4eaeca89f2c 100644
--- a/src/backend/utils/misc/guc_tables.c
+++ b/src/backend/utils/misc/guc_tables.c
@@ -3235,7 +3235,7 @@ struct config_int ConfigureNamesInt[] =
&effective_io_concurrency,
DEFAULT_EFFECTIVE_IO_CONCURRENCY,
0, MAX_IO_CONCURRENCY,
- check_effective_io_concurrency, NULL, NULL
+ NULL, NULL, NULL
},
{
@@ -3249,7 +3249,7 @@ struct config_int ConfigureNamesInt[] =
&maintenance_io_concurrency,
DEFAULT_MAINTENANCE_IO_CONCURRENCY,
0, MAX_IO_CONCURRENCY,
- check_maintenance_io_concurrency, assign_maintenance_io_concurrency,
+ NULL, assign_maintenance_io_concurrency,
NULL
},
diff --git a/src/backend/utils/misc/postgresql.conf.sample b/src/backend/utils/misc/postgresql.conf.sample
index 7c12434efa2..ff56a1f0732 100644
--- a/src/backend/utils/misc/postgresql.conf.sample
+++ b/src/backend/utils/misc/postgresql.conf.sample
@@ -198,8 +198,8 @@
# - I/O -
#backend_flush_after = 0 # measured in pages, 0 disables
-#effective_io_concurrency = 16 # 1-1000; 0 disables prefetching
-#maintenance_io_concurrency = 16 # 1-1000; 0 disables prefetching
+#effective_io_concurrency = 16 # 1-1000; 0 disables issuing multiple simultaneous IO requests
+#maintenance_io_concurrency = 16 # 1-1000; same as effective_io_concurrency
#io_max_combine_limit = 128kB # usually 1-128 blocks (depends on OS)
# (change requires restart)
#io_combine_limit = 128kB # usually 1-128 blocks (depends on OS)
diff --git a/src/bin/initdb/initdb.c b/src/bin/initdb/initdb.c
index 22b7d31b165..c17fda2bc81 100644
--- a/src/bin/initdb/initdb.c
+++ b/src/bin/initdb/initdb.c
@@ -1402,11 +1402,6 @@ setup_config(void)
repltok, true);
#endif
-#ifndef USE_PREFETCH
- conflines = replace_guc_value(conflines, "effective_io_concurrency",
- "0", true);
-#endif
-
#ifdef WIN32
conflines = replace_guc_value(conflines, "update_process_title",
"off", true);
diff --git a/doc/src/sgml/config.sgml b/doc/src/sgml/config.sgml
index f55f38cb85b..0d02e21a1ab 100644
--- a/doc/src/sgml/config.sgml
+++ b/doc/src/sgml/config.sgml
@@ -2585,8 +2585,7 @@ include_dir 'conf.d'
session attempts to initiate in parallel. The allowed range is
<literal>1</literal> to <literal>1000</literal>, or
<literal>0</literal> to disable issuance of asynchronous I/O requests.
- The default is <literal>16</literal> on supported systems, otherwise
- <literal>0</literal>.
+ The default is <literal>16</literal>.
</para>
<para>
@@ -2597,8 +2596,9 @@ include_dir 'conf.d'
</para>
<para>
- On systems without prefetch advice support, attempting to configure
- any value other than <literal>0</literal> will error out.
+ On systems with prefetch advice support,
+ <varname>effective_io_concurrency</varname> also controls the
+ prefetch distance.
</para>
<para>
@@ -2621,10 +2621,10 @@ include_dir 'conf.d'
for maintenance work that is done on behalf of many client sessions.
</para>
<para>
- The default is <literal>16</literal> on supported systems, otherwise
- <literal>0</literal>. This value can be overridden for tables in a
- particular tablespace by setting the tablespace parameter of the same
- name (see <xref linkend="sql-altertablespace"/>).
+ The default is <literal>16</literal>. This value can be overridden
+ for tables in a particular tablespace by setting the tablespace
+ parameter of the same name (see <xref
+ linkend="sql-altertablespace"/>).
</para>
</listitem>
</varlistentry>
diff --git a/doc/src/sgml/ref/alter_tablespace.sgml b/doc/src/sgml/ref/alter_tablespace.sgml
index 6ec863400d1..d0e08089ddb 100644
--- a/doc/src/sgml/ref/alter_tablespace.sgml
+++ b/doc/src/sgml/ref/alter_tablespace.sgml
@@ -88,7 +88,7 @@ ALTER TABLESPACE <replaceable>name</replaceable> RESET ( <replaceable class="par
and <varname>maintenance_io_concurrency</varname>.
Setting these values for a particular tablespace will override the
planner's usual estimate of the cost of reading pages from tables in
- that tablespace, and the executor's prefetching behavior, as established
+ that tablespace, and how many concurrent I/Os are issued, as established
by the configuration parameters of the
same name (see <xref linkend="guc-seq-page-cost"/>,
<xref linkend="guc-random-page-cost"/>,
diff --git a/doc/src/sgml/ref/create_tablespace.sgml b/doc/src/sgml/ref/create_tablespace.sgml
index 9d5ab025261..b77e774c53f 100644
--- a/doc/src/sgml/ref/create_tablespace.sgml
+++ b/doc/src/sgml/ref/create_tablespace.sgml
@@ -110,7 +110,7 @@ CREATE TABLESPACE <replaceable class="parameter">tablespace_name</replaceable>
and <varname>maintenance_io_concurrency</varname>.
Setting these values for a particular tablespace will override the
planner's usual estimate of the cost of reading pages from tables in
- that tablespace, and the executor's prefetching behavior, as established
+ that tablespace, and how many concurrent I/Os are issued, as established
by the configuration parameters of the
same name (see <xref linkend="guc-seq-page-cost"/>,
<xref linkend="guc-random-page-cost"/>,
--
2.48.1.76.g4e746b1a31.dirty
[text/x-diff] v2.14-0018-aio-Add-test_aio-module.patch (75.1K, ../../d4ff3qxudxn2ze3zk2xwhds5ysezfei3uqeuimymj624t5chsd@sgr2h3rmln5t/19-v2.14-0018-aio-Add-test_aio-module.patch)
download | inline diff:
From 46880777f409db35dd83d8acfabe56ef02e9c51c Mon Sep 17 00:00:00 2001
From: Andres Freund <andres@anarazel.de>
Date: Tue, 18 Mar 2025 14:40:06 -0400
Subject: [PATCH v2.14 18/29] aio: Add test_aio module
To make the tests possible, a few functions from bufmgr.c/localbuf.c had to be
exported, via buf_internals.h.
Reviewed-by: Noah Misch <noah@leadboat.com>
Co-authored-by: Andres Freund <andres@anarazel.de>
Co-authored-by: Nazir Bilal Yavuz <byavuz81@gmail.com>
Discussion: https://postgr.es/m/uvrtrknj4kdytuboidbhwclo4gxhswwcpgadptsjvjqcluzmah%40brqs62irg4dt
---
src/include/storage/buf_internals.h | 7 +
src/backend/storage/buffer/bufmgr.c | 8 +-
src/backend/storage/buffer/localbuf.c | 3 +-
src/test/modules/Makefile | 1 +
src/test/modules/meson.build | 1 +
src/test/modules/test_aio/.gitignore | 2 +
src/test/modules/test_aio/Makefile | 26 +
src/test/modules/test_aio/meson.build | 37 +
src/test/modules/test_aio/t/001_aio.pl | 1439 +++++++++++++++++
src/test/modules/test_aio/t/002_io_workers.pl | 125 ++
src/test/modules/test_aio/test_aio--1.0.sql | 108 ++
src/test/modules/test_aio/test_aio.c | 802 +++++++++
src/test/modules/test_aio/test_aio.control | 3 +
13 files changed, 2554 insertions(+), 8 deletions(-)
create mode 100644 src/test/modules/test_aio/.gitignore
create mode 100644 src/test/modules/test_aio/Makefile
create mode 100644 src/test/modules/test_aio/meson.build
create mode 100644 src/test/modules/test_aio/t/001_aio.pl
create mode 100644 src/test/modules/test_aio/t/002_io_workers.pl
create mode 100644 src/test/modules/test_aio/test_aio--1.0.sql
create mode 100644 src/test/modules/test_aio/test_aio.c
create mode 100644 src/test/modules/test_aio/test_aio.control
diff --git a/src/include/storage/buf_internals.h b/src/include/storage/buf_internals.h
index 72b36a4af26..0dec7d93b3b 100644
--- a/src/include/storage/buf_internals.h
+++ b/src/include/storage/buf_internals.h
@@ -434,6 +434,12 @@ extern void IssuePendingWritebacks(WritebackContext *wb_context, IOContext io_co
extern void ScheduleBufferTagForWriteback(WritebackContext *wb_context,
IOContext io_context, BufferTag *tag);
+/* solely to make it easier to write tests */
+extern bool StartBufferIO(BufferDesc *buf, bool forInput, bool nowait);
+extern void TerminateBufferIO(BufferDesc *buf, bool clear_dirty, uint32 set_flag_bits,
+ bool forget_owner, bool release_aio);
+
+
/* freelist.c */
extern IOContext IOContextForStrategy(BufferAccessStrategy strategy);
extern BufferDesc *StrategyGetBuffer(BufferAccessStrategy strategy,
@@ -478,6 +484,7 @@ extern void TerminateLocalBufferIO(BufferDesc *bufHdr, bool clear_dirty,
uint32 set_flag_bits, bool release_aio);
extern bool StartLocalBufferIO(BufferDesc *bufHdr, bool forInput, bool nowait);
extern void FlushLocalBuffer(BufferDesc *bufHdr, SMgrRelation reln);
+extern void InvalidateLocalBuffer(BufferDesc *bufHdr, bool check_unreferenced);
extern void DropRelationLocalBuffers(RelFileLocator rlocator,
ForkNumber forkNum,
BlockNumber firstDelBlock);
diff --git a/src/backend/storage/buffer/bufmgr.c b/src/backend/storage/buffer/bufmgr.c
index 39cc6489145..c65d3916211 100644
--- a/src/backend/storage/buffer/bufmgr.c
+++ b/src/backend/storage/buffer/bufmgr.c
@@ -518,10 +518,6 @@ static uint32 WaitBufHdrUnlocked(BufferDesc *buf);
static int SyncOneBuffer(int buf_id, bool skip_recently_used,
WritebackContext *wb_context);
static void WaitIO(BufferDesc *buf);
-static bool StartBufferIO(BufferDesc *buf, bool forInput, bool nowait);
-static void TerminateBufferIO(BufferDesc *buf, bool clear_dirty,
- uint32 set_flag_bits, bool forget_owner,
- bool release_aio);
static void AbortBufferIO(Buffer buffer);
static void shared_buffer_write_error_callback(void *arg);
static void local_buffer_write_error_callback(void *arg);
@@ -5961,7 +5957,7 @@ WaitIO(BufferDesc *buf)
* find out if they can perform the I/O as part of a larger operation, without
* waiting for the answer or distinguishing the reasons why not.
*/
-static bool
+bool
StartBufferIO(BufferDesc *buf, bool forInput, bool nowait)
{
uint32 buf_state;
@@ -6018,7 +6014,7 @@ StartBufferIO(BufferDesc *buf, bool forInput, bool nowait)
* resource owner. (forget_owner=false is used when the resource owner itself
* is being released)
*/
-static void
+void
TerminateBufferIO(BufferDesc *buf, bool clear_dirty, uint32 set_flag_bits,
bool forget_owner, bool release_aio)
{
diff --git a/src/backend/storage/buffer/localbuf.c b/src/backend/storage/buffer/localbuf.c
index bf89076bb10..ed56202af14 100644
--- a/src/backend/storage/buffer/localbuf.c
+++ b/src/backend/storage/buffer/localbuf.c
@@ -57,7 +57,6 @@ static int NLocalPinnedBuffers = 0;
static void InitLocalBuffers(void);
static Block GetLocalBufferStorage(void);
static Buffer GetLocalVictimBuffer(void);
-static void InvalidateLocalBuffer(BufferDesc *bufHdr, bool check_unreferenced);
/*
@@ -597,7 +596,7 @@ TerminateLocalBufferIO(BufferDesc *bufHdr, bool clear_dirty, uint32 set_flag_bit
*
* See also InvalidateBuffer().
*/
-static void
+void
InvalidateLocalBuffer(BufferDesc *bufHdr, bool check_unreferenced)
{
Buffer buffer = BufferDescriptorGetBuffer(bufHdr);
diff --git a/src/test/modules/Makefile b/src/test/modules/Makefile
index 4e4be3fa511..aa1d27bbed3 100644
--- a/src/test/modules/Makefile
+++ b/src/test/modules/Makefile
@@ -14,6 +14,7 @@ SUBDIRS = \
oauth_validator \
plsample \
spgist_name_ops \
+ test_aio \
test_bloomfilter \
test_copy_callbacks \
test_custom_rmgrs \
diff --git a/src/test/modules/meson.build b/src/test/modules/meson.build
index 2b057451473..9de0057bd1d 100644
--- a/src/test/modules/meson.build
+++ b/src/test/modules/meson.build
@@ -13,6 +13,7 @@ subdir('oauth_validator')
subdir('plsample')
subdir('spgist_name_ops')
subdir('ssl_passphrase_callback')
+subdir('test_aio')
subdir('test_bloomfilter')
subdir('test_copy_callbacks')
subdir('test_custom_rmgrs')
diff --git a/src/test/modules/test_aio/.gitignore b/src/test/modules/test_aio/.gitignore
new file mode 100644
index 00000000000..716e17f5a2a
--- /dev/null
+++ b/src/test/modules/test_aio/.gitignore
@@ -0,0 +1,2 @@
+# Generated subdirectories
+/tmp_check/
diff --git a/src/test/modules/test_aio/Makefile b/src/test/modules/test_aio/Makefile
new file mode 100644
index 00000000000..f53cc64671a
--- /dev/null
+++ b/src/test/modules/test_aio/Makefile
@@ -0,0 +1,26 @@
+# src/test/modules/test_aio/Makefile
+
+PGFILEDESC = "test_aio - test code for AIO"
+
+MODULE_big = test_aio
+OBJS = \
+ $(WIN32RES) \
+ test_aio.o
+
+EXTENSION = test_aio
+DATA = test_aio--1.0.sql
+
+TAP_TESTS = 1
+
+export enable_injection_points
+
+ifdef USE_PGXS
+PG_CONFIG = pg_config
+PGXS := $(shell $(PG_CONFIG) --pgxs)
+include $(PGXS)
+else
+subdir = src/test/modules/test_aio
+top_builddir = ../../../..
+include $(top_builddir)/src/Makefile.global
+include $(top_srcdir)/contrib/contrib-global.mk
+endif
diff --git a/src/test/modules/test_aio/meson.build b/src/test/modules/test_aio/meson.build
new file mode 100644
index 00000000000..73d2fd68eaa
--- /dev/null
+++ b/src/test/modules/test_aio/meson.build
@@ -0,0 +1,37 @@
+# Copyright (c) 2024-2025, PostgreSQL Global Development Group
+
+test_aio_sources = files(
+ 'test_aio.c',
+)
+
+if host_system == 'windows'
+ test_aio_sources += rc_lib_gen.process(win32ver_rc, extra_args: [
+ '--NAME', 'test_aio',
+ '--FILEDESC', 'test_aio - test code for AIO',])
+endif
+
+test_aio = shared_module('test_aio',
+ test_aio_sources,
+ kwargs: pg_test_mod_args,
+)
+test_install_libs += test_aio
+
+test_install_data += files(
+ 'test_aio.control',
+ 'test_aio--1.0.sql',
+)
+
+tests += {
+ 'name': 'test_aio',
+ 'sd': meson.current_source_dir(),
+ 'bd': meson.current_build_dir(),
+ 'tap': {
+ 'env': {
+ 'enable_injection_points': get_option('injection_points') ? 'yes' : 'no',
+ },
+ 'tests': [
+ 't/001_aio.pl',
+ 't/002_io_workers.pl',
+ ],
+ },
+}
diff --git a/src/test/modules/test_aio/t/001_aio.pl b/src/test/modules/test_aio/t/001_aio.pl
new file mode 100644
index 00000000000..4f08116a3ff
--- /dev/null
+++ b/src/test/modules/test_aio/t/001_aio.pl
@@ -0,0 +1,1439 @@
+# Copyright (c) 2025, PostgreSQL Global Development Group
+
+use strict;
+use warnings FATAL => 'all';
+
+use PostgreSQL::Test::Cluster;
+use PostgreSQL::Test::Utils;
+use Test::More;
+
+
+###
+# Test io_method=worker
+###
+my $node_worker = create_node('worker');
+$node_worker->start();
+
+test_generic('worker', $node_worker);
+SKIP:
+{
+ skip 'Injection points not supported by this build', 1
+ unless $ENV{enable_injection_points} eq 'yes';
+ test_inject_worker('worker', $node_worker);
+}
+
+$node_worker->stop();
+
+
+###
+# Test io_method=io_uring
+###
+
+if (have_io_uring())
+{
+ my $node_uring = create_node('io_uring');
+ $node_uring->start();
+ test_generic('io_uring', $node_uring);
+ $node_uring->stop();
+}
+
+
+###
+# Test io_method=sync
+###
+
+my $node_sync = create_node('sync');
+
+# just to have one test not use the default auto-tuning
+
+$node_sync->append_conf(
+ 'postgresql.conf', qq(
+io_max_concurrency=4
+));
+
+$node_sync->start();
+test_generic('sync', $node_sync);
+$node_sync->stop();
+
+done_testing();
+
+
+###
+# Test Helpers
+###
+
+sub create_node
+{
+ local $Test::Builder::Level = $Test::Builder::Level + 1;
+
+ my $io_method = shift;
+
+ my $node = PostgreSQL::Test::Cluster->new($io_method);
+
+ # Want to test initdb for each IO method, otherwise we could just reuse
+ # the cluster.
+ #
+ # Unfortunately Cluster::init() puts PG_TEST_INITDB_EXTRA_OPTS after the
+ # options specified by ->extra, if somebody puts -c io_method=xyz in
+ # PG_TEST_INITDB_EXTRA_OPTS it would break this test. Fix that up if we
+ # detect it.
+ local $ENV{PG_TEST_INITDB_EXTRA_OPTS} = $ENV{PG_TEST_INITDB_EXTRA_OPTS};
+ if (defined $ENV{PG_TEST_INITDB_EXTRA_OPTS}
+ && $ENV{PG_TEST_INITDB_EXTRA_OPTS} =~ m/io_method=/)
+ {
+ $ENV{PG_TEST_INITDB_EXTRA_OPTS} .= " -c io_method=$io_method";
+ }
+
+ $node->init(extra => [ '-c', "io_method=$io_method" ]);
+
+ $node->append_conf(
+ 'postgresql.conf', qq(
+shared_preload_libraries=test_aio
+log_min_messages = 'DEBUG3'
+log_statement=all
+log_error_verbosity=default
+restart_after_crash=false
+temp_buffers=100
+));
+
+ ok(1, "$io_method: initdb");
+
+ return $node;
+}
+
+sub have_io_uring
+{
+ # To detect if io_uring is supported, we look at the error message for
+ # assigning an invalid value to an enum GUC, which lists all the valid
+ # options. We need to use -C to deal with running as administrator on
+ # windows, the superuser check is omitted if -C is used.
+ my ($stdout, $stderr) =
+ run_command [qw(postgres -C invalid -c io_method=invalid)];
+ die "can't determine supported io_method values"
+ unless $stderr =~ m/Available values: ([^\.]+)\./;
+ my $methods = $1;
+ note "supported io_method values are: $methods";
+
+ return ($methods =~ m/io_uring/) ? 1 : 0;
+}
+
+sub psql_like
+{
+ my $io_method = shift;
+ my $psql = shift;
+ my $name = shift;
+ my $sql = shift;
+ my $expected_stdout = shift;
+ my $expected_stderr = shift;
+ my ($cmdret, $output);
+
+ ($output, $cmdret) = $psql->query($sql);
+
+ like($output, $expected_stdout, "$io_method: $name: expected stdout");
+ like($psql->{stderr}, $expected_stderr,
+ "$io_method: $name: expected stderr");
+ $psql->{stderr} = '';
+
+ return $output;
+}
+
+sub query_wait_block
+{
+ my $io_method = shift;
+ my $node = shift;
+ my $psql = shift;
+ my $name = shift;
+ my $sql = shift;
+ my $waitfor = shift;
+
+ my $pid = $psql->query_safe('SELECT pg_backend_pid()');
+
+ $psql->{stdin} .= qq($sql;\n);
+ $psql->{run}->pump_nb();
+ ok(1, "$io_method: $name: issued sql");
+
+ $node->poll_query_until('postgres',
+ qq(SELECT wait_event FROM pg_stat_activity WHERE pid = $pid),
+ $waitfor);
+ ok(1, "$io_method: $name: observed $waitfor wait event");
+}
+
+# Returns count of checksum failures for the specified database or for shared
+# relations, if $datname is undefined.
+sub checksum_failures
+{
+ my $psql = shift;
+ my $datname = shift;
+ my $checksum_count;
+ my $checksum_last_failure;
+
+ if (defined $datname)
+ {
+ $checksum_count = $psql->query_safe(
+ qq(
+SELECT checksum_failures FROM pg_stat_database WHERE datname = '$datname';
+));
+ $checksum_last_failure = $psql->query_safe(
+ qq(
+SELECT checksum_last_failure FROM pg_stat_database WHERE datname = '$datname';
+));
+ }
+ else
+ {
+ $checksum_count = $psql->query_safe(
+ qq(
+SELECT checksum_failures FROM pg_stat_database WHERE datname IS NULL;
+));
+ $checksum_last_failure = $psql->query_safe(
+ qq(
+SELECT checksum_last_failure FROM pg_stat_database WHERE datname IS NULL;
+));
+ }
+
+ return $checksum_count, $checksum_last_failure;
+}
+
+###
+# Sub-tests
+###
+
+# Sanity checks for the IO handle API
+sub test_handle
+{
+ my $io_method = shift;
+ my $node = shift;
+
+ my $psql = $node->background_psql('postgres', on_error_stop => 0);
+
+ # leak warning: implicit xact
+ psql_like(
+ $io_method,
+ $psql,
+ "handle_get() leak in implicit xact",
+ qq(SELECT handle_get()),
+ qr/^$/,
+ qr/leaked AIO handle/,
+ "$io_method: leaky handle_get() warns");
+
+ # leak warning: explicit xact
+ psql_like(
+ $io_method, $psql,
+ "handle_get() leak in explicit xact",
+ qq(BEGIN; SELECT handle_get(); COMMIT),
+ qr/^$/, qr/leaked AIO handle/);
+
+
+ # leak warning: explicit xact, rollback
+ psql_like(
+ $io_method,
+ $psql,
+ "handle_get() leak in explicit xact, rollback",
+ qq(BEGIN; SELECT handle_get(); ROLLBACK;),
+ qr/^$/,
+ qr/leaked AIO handle/);
+
+ # leak warning: subtrans
+ psql_like(
+ $io_method,
+ $psql,
+ "handle_get() leak in subxact",
+ qq(BEGIN; SAVEPOINT foo; SELECT handle_get(); COMMIT;),
+ qr/^$/,
+ qr/leaked AIO handle/);
+
+ # leak warning + error: released in different command (thus resowner)
+ psql_like(
+ $io_method,
+ $psql,
+ "handle_release() in different command",
+ qq(BEGIN; SELECT handle_get(); SELECT handle_release_last(); COMMIT;),
+ qr/^$/,
+ qr/leaked AIO handle.*release in unexpected state/ms);
+
+ # no leak, release in same command
+ psql_like(
+ $io_method,
+ $psql,
+ "handle_release() in same command",
+ qq(BEGIN; SELECT handle_get() UNION ALL SELECT handle_release_last(); COMMIT;),
+ qr/^$/,
+ qr/^$/);
+
+ # normal handle use
+ psql_like($io_method, $psql, "handle_get_release()",
+ qq(SELECT handle_get_release()),
+ qr/^$/, qr/^$/);
+
+ # should error out, API violation
+ psql_like(
+ $io_method,
+ $psql,
+ "handle_get_twice()",
+ qq(SELECT handle_get_twice()),
+ qr/^$/,
+ qr/ERROR: API violation: Only one IO can be handed out$/);
+
+ # recover after error in implicit xact
+ psql_like(
+ $io_method,
+ $psql,
+ "handle error recovery in implicit xact",
+ qq(SELECT handle_get_and_error(); SELECT 'ok', handle_get_release()),
+ qr/^|ok$/,
+ qr/ERROR.*as you command/);
+
+ # recover after error in implicit xact
+ psql_like(
+ $io_method,
+ $psql,
+ "handle error recovery in explicit xact",
+ qq(BEGIN; SELECT handle_get_and_error(); SELECT handle_get_release(), 'ok'; COMMIT;),
+ qr/^|ok$/,
+ qr/ERROR.*as you command/);
+
+ # recover after error in subtrans
+ psql_like(
+ $io_method,
+ $psql,
+ "handle error recovery in explicit subxact",
+ qq(BEGIN; SAVEPOINT foo; SELECT handle_get_and_error(); ROLLBACK TO SAVEPOINT foo; SELECT handle_get_release(); ROLLBACK;),
+ qr/^|ok$/,
+ qr/ERROR.*as you command/);
+
+ $psql->quit();
+}
+
+# Sanity checks for the batchmode API
+sub test_batchmode
+{
+ my $io_method = shift;
+ my $node = shift;
+
+ my $psql = $node->background_psql('postgres', on_error_stop => 0);
+
+ # leak warning & recovery: implicit xact
+ psql_like(
+ $io_method,
+ $psql,
+ "batch_start() leak & cleanup in implicit xact",
+ qq(SELECT batch_start()),
+ qr/^$/,
+ qr/open AIO batch at end/,
+ "$io_method: leaky batch_start() warns");
+
+ # leak warning & recovery: explicit xact
+ psql_like(
+ $io_method,
+ $psql,
+ "batch_start() leak & cleanup in explicit xact",
+ qq(BEGIN; SELECT batch_start(); COMMIT;),
+ qr/^$/,
+ qr/open AIO batch at end/,
+ "$io_method: leaky batch_start() warns");
+
+
+ # leak warning & recovery: explicit xact, rollback
+ #
+ # XXX: This doesn't fail right now, due to not getting a chance to do
+ # something at transaction command commit. That's not a correctness issue,
+ # it just means it's a bit harder to find buggy code.
+ #psql_like($io_method, $psql,
+ # "batch_start() leak & cleanup after abort",
+ # qq(BEGIN; SELECT batch_start(); ROLLBACK;),
+ # qr/^$/,
+ # qr/open AIO batch at end/, "$io_method: leaky batch_start() warns");
+
+ # no warning, batch closed in same command
+ psql_like(
+ $io_method,
+ $psql,
+ "batch_start(), batch_end() works",
+ qq(SELECT batch_start() UNION ALL SELECT batch_end()),
+ qr/^$/,
+ qr/^$/,
+ "$io_method: batch_start(), batch_end()");
+
+ $psql->quit();
+}
+
+# Test that simple cases of invalid pages are reported
+sub test_io_error
+{
+ my $io_method = shift;
+ my $node = shift;
+ my ($ret, $output);
+
+ my $psql = $node->background_psql('postgres', on_error_stop => 0);
+
+ $psql->query_safe(
+ qq(
+CREATE TEMPORARY TABLE tmp_corr(data int not null);
+INSERT INTO tmp_corr SELECT generate_series(1, 10000);
+SELECT modify_rel_block('tmp_corr', 1, corrupt_header=>true);
+));
+
+ foreach my $tblname (qw(tbl_corr tmp_corr))
+ {
+ my $invalid_page_re =
+ $tblname eq 'tbl_corr'
+ ? qr/invalid page in block 1 of relation base\/\d+\/\d+/
+ : qr/invalid page in block 1 of relation base\/\d+\/t\d+_\d+/;
+
+ # verify the error is reported in custom C code
+ psql_like(
+ $io_method,
+ $psql,
+ "read_rel_block_ll() of $tblname page",
+ qq(SELECT read_rel_block_ll('$tblname', 1)),
+ qr/^$/,
+ $invalid_page_re);
+
+ # verify the error is reported for bufmgr reads, seq scan
+ psql_like(
+ $io_method, $psql,
+ "sequential scan of $tblname block fails",
+ qq(SELECT count(*) FROM $tblname),
+ qr/^$/, $invalid_page_re);
+
+ # verify the error is reported for bufmgr reads, tid scan
+ psql_like(
+ $io_method,
+ $psql,
+ "tid scan of $tblname block fails",
+ qq(SELECT count(*) FROM $tblname WHERE ctid = '(1, 1)'),
+ qr/^$/,
+ $invalid_page_re);
+ }
+
+ $psql->quit();
+}
+
+# Test interplay between StartBufferIO and TerminateBufferIO
+sub test_startwait_io
+{
+ my $io_method = shift;
+ my $node = shift;
+ my ($ret, $output);
+
+ my $psql_a = $node->background_psql('postgres', on_error_stop => 0);
+ my $psql_b = $node->background_psql('postgres', on_error_stop => 0);
+
+
+ ### Verify behavior for normal tables
+
+ # create a buffer we can play around with
+ my $buf_id = psql_like(
+ $io_method, $psql_a,
+ "creation of toy buffer succeeds",
+ qq(SELECT buffer_create_toy('tbl_ok', 1)),
+ qr/^\d+$/, qr/^$/);
+
+ # check that one backend can perform StartBufferIO
+ psql_like(
+ $io_method,
+ $psql_a,
+ "first StartBufferIO",
+ qq(SELECT buffer_call_start_io($buf_id, for_input=>true, nowait=>false);),
+ qr/^t$/,
+ qr/^$/);
+
+ # but not twice on the same buffer (non-waiting)
+ psql_like(
+ $io_method,
+ $psql_a,
+ "second StartBufferIO fails, same session",
+ qq(SELECT buffer_call_start_io($buf_id, for_input=>true, nowait=>true);),
+ qr/^f$/,
+ qr/^$/);
+ psql_like(
+ $io_method,
+ $psql_b,
+ "second StartBufferIO fails, other session",
+ qq(SELECT buffer_call_start_io($buf_id, for_input=>true, nowait=>true);),
+ qr/^f$/,
+ qr/^$/);
+
+ # start io in a different session, will block
+ query_wait_block(
+ $io_method,
+ $node,
+ $psql_b,
+ "blocking start buffer io",
+ qq(SELECT buffer_call_start_io($buf_id, for_input=>true, nowait=>false);),
+ "BufferIo");
+
+ # Terminate the IO, without marking it as success, this should trigger the
+ # waiting session to be able to start the io
+ psql_like(
+ $io_method,
+ $psql_a,
+ "blocking start buffer io, terminating io, not valid",
+ qq(SELECT buffer_call_terminate_io($buf_id, for_input=>true, succeed=>false, io_error=>false, release_aio=>false)),
+ qr/^$/,
+ qr/^$/);
+
+
+ # Because the IO was terminated, but not marked as valid, second session should get the right to start io
+ pump_until($psql_b->{run}, $psql_b->{timeout}, \$psql_b->{stdout}, qr/t/);
+ ok(1, "$io_method: blocking start buffer io, can start io");
+
+ # terminate the IO again
+ $psql_b->query_safe(
+ qq(SELECT buffer_call_terminate_io($buf_id, for_input=>true, succeed=>false, io_error=>false, release_aio=>false);)
+ );
+
+
+ # same as the above scenario, but mark IO as having succeeded
+ psql_like(
+ $io_method,
+ $psql_a,
+ "blocking buffer io w/ success: first start buffer io",
+ qq(SELECT buffer_call_start_io($buf_id, for_input=>true, nowait=>false);),
+ qr/^t$/,
+ qr/^$/);
+
+ # start io in a different session, will block
+ query_wait_block(
+ $io_method,
+ $node,
+ $psql_b,
+ "blocking start buffer io",
+ qq(SELECT buffer_call_start_io($buf_id, for_input=>true, nowait=>false);),
+ "BufferIo");
+
+ # Terminate the IO, marking it as success
+ psql_like(
+ $io_method,
+ $psql_a,
+ "blocking start buffer io, terminating io, valid",
+ qq(SELECT buffer_call_terminate_io($buf_id, for_input=>true, succeed=>true, io_error=>false, release_aio=>false)),
+ qr/^$/,
+ qr/^$/);
+
+ # Because the IO was terminated, and marked as valid, second session should complete but not need io
+ pump_until($psql_b->{run}, $psql_b->{timeout}, \$psql_b->{stdout}, qr/f/);
+ ok(1, "$io_method: blocking start buffer io, no need to start io");
+
+ # buffer is valid now, make it invalid again
+ $psql_a->query_safe(qq(SELECT buffer_create_toy('tbl_ok', 1);));
+
+
+ ### Verify behavior for temporary tables
+
+ # Can't unfortunately share the code with the normal table case, there are
+ # too many behavioral differences.
+
+ # create a buffer we can play around with
+ $psql_a->query_safe(
+ qq(
+CREATE TEMPORARY TABLE tmp_ok(data int not null);
+INSERT INTO tmp_ok SELECT generate_series(1, 10000);
+));
+ $buf_id = $psql_a->query_safe(qq(SELECT buffer_create_toy('tmp_ok', 3);));
+
+ # check that one backend can perform StartLocalBufferIO
+ psql_like(
+ $io_method,
+ $psql_a,
+ "first StartLocalBufferIO",
+ qq(SELECT buffer_call_start_io($buf_id, for_input=>true, nowait=>true);),
+ qr/^t$/,
+ qr/^$/);
+
+ # Because local buffers don't use IO_IN_PROGRESS, a second StartLocalBufer
+ # succeeds as well. This test mostly serves as a documentation of that
+ # fact. If we had actually started IO, it'd be different.
+ psql_like(
+ $io_method,
+ $psql_a,
+ "second StartLocalBufferIO succeeds, same session",
+ qq(SELECT buffer_call_start_io($buf_id, for_input=>true, nowait=>true);),
+ qr/^t$/,
+ qr/^$/);
+
+ # Terminate the IO again, without marking it as a success
+ $psql_a->query_safe(
+ qq(SELECT buffer_call_terminate_io($buf_id, for_input=>true, succeed=>false, io_error=>false, release_aio=>false);)
+ );
+ psql_like(
+ $io_method,
+ $psql_a,
+ "StartLocalBufferIO after not marking valid succeeds, same session",
+ qq(SELECT buffer_call_start_io($buf_id, for_input=>true, nowait=>true);),
+ qr/^t$/,
+ qr/^$/);
+
+ # Terminate the IO again, marking it as a success
+ $psql_a->query_safe(
+ qq(SELECT buffer_call_terminate_io($buf_id, for_input=>true, succeed=>true, io_error=>false, release_aio=>false);)
+ );
+
+ # Now another StartLocalBufferIO should fail, this time because the buffer
+ # is already valid.
+ psql_like(
+ $io_method,
+ $psql_a,
+ "StartLocalBufferIO after marking valid fails",
+ qq(SELECT buffer_call_start_io($buf_id, for_input=>true, nowait=>false);),
+ qr/^f$/,
+ qr/^$/);
+
+ $psql_a->quit();
+ $psql_b->quit();
+}
+
+# Test that if the backend issuing a read doesn't wait for the IO's
+# completion, another backend can complete the IO
+sub test_complete_foreign
+{
+ my $io_method = shift;
+ my $node = shift;
+ my ($ret, $output);
+
+ my $psql_a = $node->background_psql('postgres', on_error_stop => 0);
+ my $psql_b = $node->background_psql('postgres', on_error_stop => 0);
+
+ # Issue IO without waiting for completion, then sleep
+ $psql_a->query_safe(
+ qq(SELECT read_rel_block_ll('tbl_ok', 1, wait_complete=>false);));
+
+ # Check that another backend can read the relevant block
+ psql_like(
+ $io_method,
+ $psql_b,
+ "completing read started by sleeping backend",
+ qq(SELECT count(*) FROM tbl_ok WHERE ctid = '(1,1)' LIMIT 1),
+ qr/^1$/,
+ qr/^$/);
+
+ # Issue IO without waiting for completion, then exit.
+ $psql_a->query_safe(
+ qq(SELECT read_rel_block_ll('tbl_ok', 1, wait_complete=>false);));
+ $psql_a->reconnect_and_clear();
+
+ # Check that another backend can read the relevant block. This verifies
+ # that the exiting backend left the AIO in a sane state.
+ psql_like(
+ $io_method,
+ $psql_b,
+ "read buffer started by exited backend",
+ qq(SELECT count(*) FROM tbl_ok WHERE ctid = '(1,1)' LIMIT 1),
+ qr/^1$/,
+ qr/^$/);
+
+ # Read a tbl_corr block, then sleep. The other session will retry the IO
+ # and also fail. The easiest thing to verify that seems to be to check
+ # that both are in the log.
+ my $log_location = -s $node->logfile;
+ $psql_a->query_safe(
+ qq(SELECT read_rel_block_ll('tbl_corr', 1, wait_complete=>false);));
+
+ psql_like(
+ $io_method,
+ $psql_b,
+ "completing read of tbl_corr block started by other backend",
+ qq(SELECT count(*) FROM tbl_corr WHERE ctid = '(1,1)' LIMIT 1),
+ qr/^$/,
+ qr/invalid page in block/);
+
+ # The log message issued for the read_rel_block_ll() should be logged as a LOG
+ $node->wait_for_log(qr/LOG[^\n]+invalid page in/, $log_location);
+ ok(1,
+ "$io_method: completing read of tbl_corr block started by other backend: LOG message for background read"
+ );
+
+ # But for the SELECT, it should be an ERROR
+ $log_location =
+ $node->wait_for_log(qr/ERROR[^\n]+invalid page in/, $log_location);
+ ok(1,
+ "$io_method: completing read of tbl_corr block started by other backend: ERROR message for foreground read"
+ );
+
+ $psql_a->quit();
+ $psql_b->quit();
+}
+
+# Test that we deal correctly with FDs being closed while IO is in progress
+sub test_close_fd
+{
+ my $io_method = shift;
+ my $node = shift;
+ my ($ret, $output);
+
+ my $psql = $node->background_psql('postgres', on_error_stop => 0);
+
+ psql_like(
+ $io_method,
+ $psql,
+ "close all FDs after read, waiting for results",
+ qq(
+ SELECT read_rel_block_ll('tbl_ok', 1,
+ wait_complete=>true,
+ batchmode_enter=>true,
+ smgrreleaseall=>true,
+ batchmode_exit=>true
+ );),
+ qr/^$/,
+ qr/^$/);
+
+ psql_like(
+ $io_method,
+ $psql,
+ "close all FDs after read, no waiting",
+ qq(
+ SELECT read_rel_block_ll('tbl_ok', 1,
+ wait_complete=>false,
+ batchmode_enter=>true,
+ smgrreleaseall=>true,
+ batchmode_exit=>true
+ );),
+ qr/^$/,
+ qr/^$/);
+
+ # Check that another backend can read the relevant block
+ psql_like(
+ $io_method,
+ $psql,
+ "close all FDs after read, no waiting, query works",
+ qq(SELECT count(*) FROM tbl_ok WHERE ctid = '(1,1)' LIMIT 1),
+ qr/^1$/,
+ qr/^$/);
+
+ $psql->quit();
+}
+
+# Tests using injection points. Mostly to exercise had IO errors that are
+# hard to trigger without using injection points.
+sub test_inject
+{
+ my $io_method = shift;
+ my $node = shift;
+ my ($ret, $output);
+
+ my $psql = $node->background_psql('postgres', on_error_stop => 0);
+
+ # injected what we'd expect
+ $psql->query_safe(qq(SELECT inj_io_short_read_attach(8192);));
+ $psql->query_safe(qq(SELECT invalidate_rel_block('tbl_ok', 2);));
+ psql_like(
+ $io_method, $psql,
+ "injection point not triggering failure",
+ qq(SELECT count(*) FROM tbl_ok WHERE ctid = '(2, 1)'),
+ qr/^1$/, qr/^$/);
+
+ # injected a read shorter than a single block, expecting error
+ $psql->query_safe(qq(SELECT inj_io_short_read_attach(17);));
+ $psql->query_safe(qq(SELECT invalidate_rel_block('tbl_ok', 2);));
+ psql_like(
+ $io_method,
+ $psql,
+ "single block short read fails",
+ qq(SELECT count(*) FROM tbl_ok WHERE ctid = '(2, 1)'),
+ qr/^$/,
+ qr/ERROR:.*could not read blocks 2\.\.2 in file "base\/.*": read only 0 of 8192 bytes/
+ );
+
+ # shorten multi-block read to a single block, should retry
+ my $inval_query = qq(SELECT invalidate_rel_block('tbl_ok', 0);
+SELECT invalidate_rel_block('tbl_ok', 1);
+SELECT invalidate_rel_block('tbl_ok', 2);
+SELECT invalidate_rel_block('tbl_ok', 3);
+/* gap */
+SELECT invalidate_rel_block('tbl_ok', 5);
+SELECT invalidate_rel_block('tbl_ok', 6);
+SELECT invalidate_rel_block('tbl_ok', 7);
+SELECT invalidate_rel_block('tbl_ok', 8););
+
+ $psql->query_safe($inval_query);
+ $psql->query_safe(qq(SELECT inj_io_short_read_attach(8192);));
+ psql_like(
+ $io_method, $psql,
+ "multi block short read (1 block) is retried",
+ qq(SELECT count(*) FROM tbl_ok),
+ qr/^10000$/, qr/^$/);
+
+ # shorten multi-block read to two blocks, should retry
+ $psql->query_safe($inval_query);
+ $psql->query_safe(qq(SELECT inj_io_short_read_attach(8192*2);));
+
+ psql_like(
+ $io_method, $psql,
+ "multi block short read (2 blocks) is retried",
+ qq(SELECT count(*) FROM tbl_ok),
+ qr/^10000$/, qr/^$/);
+
+ # verify that page verification errors are detected even as part of a
+ # shortened multi-block read (tbl_corr, block 1 is corrupted)
+ $psql->query_safe(
+ qq(
+SELECT invalidate_rel_block('tbl_corr', 0);
+SELECT invalidate_rel_block('tbl_corr', 1);
+SELECT invalidate_rel_block('tbl_corr', 2);
+SELECT inj_io_short_read_attach(8192);
+ ));
+
+ psql_like(
+ $io_method,
+ $psql,
+ "shortened multi-block read detects invalid page",
+ qq(SELECT count(*) FROM tbl_corr WHERE ctid < '(2, 1)'),
+ qr/^$/,
+ qr/ERROR:.*invalid page in block 1 of relation base\/.*/);
+
+ # trigger a hard error, should error out
+ $psql->query_safe(
+ qq(
+SELECT inj_io_short_read_attach(-errno_from_string('EIO'));
+SELECT invalidate_rel_block('tbl_ok', 2);
+ ));
+
+ psql_like(
+ $io_method,
+ $psql,
+ "first hard IO error is reported",
+ qq(SELECT count(*) FROM tbl_ok),
+ qr/^$/,
+ qr/ERROR:.*could not read blocks 2\.\.2 in file \"base\/.*\": Input\/output error/
+ );
+
+ psql_like(
+ $io_method,
+ $psql,
+ "second hard IO error is reported",
+ qq(SELECT count(*) FROM tbl_ok),
+ qr/^$/,
+ qr/ERROR:.*could not read blocks 2\.\.2 in file \"base\/.*\": Input\/output error/
+ );
+
+ $psql->query_safe(qq(SELECT inj_io_short_read_detach()));
+
+ # now the IO should be ok.
+ psql_like(
+ $io_method, $psql,
+ "recovers after hard error",
+ qq(SELECT count(*) FROM tbl_ok),
+ qr/^10000$/, qr/^$/);
+
+ $psql->quit();
+}
+
+# Tests using injection points, only for io_method=worker.
+#
+# io_method=worker has the special case of needing to reopen files. That can
+# in theory fail, because the file could be gone. That's a hard path to test
+# for real, so we use an injection point to trigger it.
+sub test_inject_worker
+{
+ my $io_method = shift;
+ my $node = shift;
+ my ($ret, $output);
+
+ my $psql = $node->background_psql('postgres', on_error_stop => 0);
+
+ # trigger a failure to reopen, should error out, but should recover
+ $psql->query_safe(
+ qq(
+SELECT inj_io_reopen_attach();
+SELECT invalidate_rel_block('tbl_ok', 1);
+ ));
+
+ psql_like(
+ $io_method,
+ $psql,
+ "failure to open: detected",
+ qq(SELECT count(*) FROM tbl_ok),
+ qr/^$/,
+ qr/ERROR:.*could not read blocks 1\.\.1 in file "base\/.*": No such file or directory/
+ );
+
+ $psql->query_safe(qq(SELECT inj_io_reopen_detach();));
+
+ # check that we indeed recover
+ psql_like(
+ $io_method, $psql,
+ "failure to open: recovers",
+ qq(SELECT count(*) FROM tbl_ok),
+ qr/^10000$/, qr/^$/);
+
+ $psql->quit();
+}
+
+# Verify that we handle a relation getting removed (due to a rollback or a
+# DROP TABLE) while IO is ongoing for that table.
+sub test_invalidate
+{
+ my $io_method = shift;
+ my $node = shift;
+
+ my $psql = $node->background_psql('postgres', on_error_stop => 0);
+
+ foreach my $persistency (qw(normal unlogged temporary))
+ {
+ my $sql_persistency = $persistency eq 'normal' ? '' : $persistency;
+ my $tblname = $persistency . '_transactional';
+
+ my $create_sql = qq(
+CREATE $sql_persistency TABLE $tblname (id int not null, data text not null) WITH (AUTOVACUUM_ENABLED = false);
+INSERT INTO $tblname(id, data) SELECT generate_series(1, 10000) as id, repeat('a', 200);
+);
+
+ # Verify that outstanding read IO does not cause problems with
+ # AbortTransaction -> smgrDoPendingDeletes -> smgrdounlinkall -> ...
+ # -> Invalidate[Local]Buffer.
+ $psql->query_safe("BEGIN; $create_sql;");
+ $psql->query_safe(
+ qq(
+SELECT read_rel_block_ll('$tblname', 1, wait_complete=>false);
+));
+ psql_like(
+ $io_method,
+ $psql,
+ "rollback of newly created $persistency table with outstanding IO",
+ qq(ROLLBACK),
+ qr/^$/,
+ qr/^$/);
+
+ # Verify that outstanding read IO does not cause problems with
+ # CommitTransaction -> smgrDoPendingDeletes -> smgrdounlinkall -> ...
+ # -> Invalidate[Local]Buffer.
+ $psql->query_safe("BEGIN; $create_sql; COMMIT;");
+ $psql->query_safe(
+ qq(
+BEGIN;
+SELECT read_rel_block_ll('$tblname', 1, wait_complete=>false);
+));
+
+ psql_like(
+ $io_method, $psql,
+ "drop $persistency table with outstanding IO",
+ qq(DROP TABLE $tblname),
+ qr/^$/, qr/^$/);
+
+ psql_like($io_method, $psql,
+ "commit after drop $persistency table with outstanding IO",
+ qq(COMMIT), qr/^$/, qr/^$/);
+ }
+
+ $psql->quit();
+}
+
+# Test behavior related to ZERO_ON_ERROR and zero_damaged_pages
+sub test_zero
+{
+ my $io_method = shift;
+ my $node = shift;
+
+ my $psql_a = $node->background_psql('postgres', on_error_stop => 0);
+
+ foreach my $persistency (qw(normal temporary))
+ {
+ my $sql_persistency = $persistency eq 'normal' ? '' : $persistency;
+
+ $psql_a->query_safe(
+ qq(
+CREATE $sql_persistency TABLE tbl_zero(id int) WITH (AUTOVACUUM_ENABLED = false);
+INSERT INTO tbl_zero SELECT generate_series(1, 10000);
+));
+
+ $psql_a->query_safe(
+ qq(
+SELECT modify_rel_block('tbl_zero', 0, corrupt_header=>true);
+));
+
+ # Check that page validity errors are detected
+ psql_like(
+ $io_method,
+ $psql_a,
+ "$persistency: test reading of invalid block 0",
+ qq(
+SELECT read_rel_block_ll('tbl_zero', 0, zero_on_error=>false)),
+ qr/^$/,
+ qr/^psql:<stdin>:\d+: ERROR: invalid page in block 0 of relation base\/.*\/.*$/
+ );
+
+ # Check that page validity errors are zeroed
+ psql_like(
+ $io_method,
+ $psql_a,
+ "$persistency: test zeroing of invalid block 0",
+ qq(
+SELECT read_rel_block_ll('tbl_zero', 0, zero_on_error=>true)),
+ qr/^$/,
+ qr/^psql:<stdin>:\d+: WARNING: invalid page in block 0 of relation base\/.*\/.*; zeroing out page$/
+ );
+
+ # And that once the corruption is fixed, we can read again
+ $psql_a->query(
+ qq(
+SELECT modify_rel_block('tbl_zero', 0, zero=>true);
+));
+ $psql_a->{stderr} = '';
+
+ psql_like(
+ $io_method,
+ $psql_a,
+ "$persistency: test re-read of block 0",
+ qq(
+SELECT read_rel_block_ll('tbl_zero', 0, zero_on_error=>false)),
+ qr/^$/,
+ qr/^$/);
+
+ # Check a page validity error in another block, to ensure we report
+ # the correct block number
+ $psql_a->query_safe(
+ qq(
+SELECT modify_rel_block('tbl_zero', 3, corrupt_header=>true);
+));
+ psql_like(
+ $io_method,
+ $psql_a,
+ "$persistency: test zeroing of invalid block 3",
+ qq(SELECT read_rel_block_ll('tbl_zero', 3, zero_on_error=>true);),
+ qr/^$/,
+ qr/^psql:<stdin>:\d+: WARNING: invalid page in block 3 of relation base\/.*\/.*; zeroing out page$/
+ );
+
+
+ # Check a page validity error in another block, to ensure we report
+ # the correct block number
+ $psql_a->query_safe(
+ qq(
+SELECT modify_rel_block('tbl_zero', 2, corrupt_header=>true);
+SELECT modify_rel_block('tbl_zero', 3, corrupt_header=>true);
+));
+ # First test error
+ psql_like(
+ $io_method,
+ $psql_a,
+ "$persistency: test reading of invalid block 2,3 in larger read",
+ qq(SELECT read_rel_block_ll('tbl_zero', 1, nblocks=>4, zero_on_error=>false)),
+ qr/^$/,
+ qr/^psql:<stdin>:\d+: ERROR: 2 invalid pages among blocks 1..4 of relation base\/.*\/.*\nDETAIL: Block 2 held first invalid page\.\nHINT:[^\n]+$/
+ );
+
+ # Then test zeroing via ZERO_ON_ERROR flag
+ psql_like(
+ $io_method,
+ $psql_a,
+ "$persistency: test zeroing of invalid block 2,3 in larger read, ZERO_ON_ERROR",
+ qq(SELECT read_rel_block_ll('tbl_zero', 1, nblocks=>4, zero_on_error=>true)),
+ qr/^$/,
+ qr/^psql:<stdin>:\d+: WARNING: zeroing out 2 invalid pages among blocks 1..4 of relation base\/.*\/.*\nDETAIL: Block 2 held first zeroed page\.\nHINT:[^\n]+$/
+ );
+
+ # Then test zeroing vio zero_damaged_pages
+ psql_like(
+ $io_method,
+ $psql_a,
+ "$persistency: test zeroing of invalid block 2,3 in larger read, zero_damaged_pages",
+ qq(
+BEGIN;
+SET LOCAL zero_damaged_pages = true;
+SELECT read_rel_block_ll('tbl_zero', 1, nblocks=>4, zero_on_error=>false)
+COMMIT;
+),
+ qr/^$/,
+ qr/^psql:<stdin>:\d+: WARNING: zeroing out 2 invalid pages among blocks 1..4 of relation base\/.*\/.*\nDETAIL: Block 2 held first zeroed page\.\nHINT:[^\n]+$/
+ );
+
+ $psql_a->query_safe(qq(COMMIT));
+
+
+ # Verify that bufmgr.c IO detects page validity errors
+ $psql_a->query(
+ qq(
+SELECT invalidate_rel_block('tbl_zero', g.i)
+FROM generate_series(0, 15) g(i);
+SELECT modify_rel_block('tbl_zero', 3, zero=>true);
+));
+ $psql_a->{stderr} = '';
+
+ psql_like(
+ $io_method,
+ $psql_a,
+ "$persistency: verify reading zero_damaged_pages=off",
+ qq(
+SELECT count(*) FROM tbl_zero),
+ qr/^$/,
+ qr/^psql:<stdin>:\d+: ERROR: invalid page in block 2 of relation base\/.*\/.*$/
+ );
+
+ # Verify that bufmgr.c IO zeroes out pages with page validity errors
+ psql_like(
+ $io_method,
+ $psql_a,
+ "$persistency: verify zero_damaged_pages=on",
+ qq(
+BEGIN;
+SET LOCAL zero_damaged_pages = true;
+SELECT count(*) FROM tbl_zero;
+COMMIT;
+),
+ qr/^\d+$/,
+ qr/^psql:<stdin>:\d+: WARNING: invalid page in block 2 of relation base\/.*\/.*$/
+ );
+
+
+ # FIXME: Test that if one session triggers a read but doesn't complete
+ # it, the completing session doesn't see the WARNING.
+
+ # Clean up
+ $psql_a->query_safe(
+ qq(
+DROP TABLE tbl_zero;
+));
+ }
+
+ $psql_a->{stderr} = '';
+
+ $psql_a->quit();
+}
+
+# Test that we detect checksum failures and report them
+sub test_checksum
+{
+ my $io_method = shift;
+ my $node = shift;
+
+ my $psql_a = $node->background_psql('postgres', on_error_stop => 0);
+
+ $psql_a->query_safe(
+ qq(
+CREATE TABLE tbl_normal(id int) WITH (AUTOVACUUM_ENABLED = false);
+INSERT INTO tbl_normal SELECT generate_series(1, 5000);
+SELECT modify_rel_block('tbl_normal', 3, corrupt_checksum=>true);
+
+CREATE TEMPORARY TABLE tbl_temp(id int) WITH (AUTOVACUUM_ENABLED = false);
+INSERT INTO tbl_temp SELECT generate_series(1, 5000);
+SELECT modify_rel_block('tbl_temp', 3, corrupt_checksum=>true);
+SELECT modify_rel_block('tbl_temp', 4, corrupt_checksum=>true);
+));
+
+ # To be able to test checksum failures on shared rels we need a shared rel
+ # with invalid pages - which is a bit scary. pg_shseclabel seems like a
+ # good bet, as it's not accessed in a default configuration.
+ $psql_a->query_safe(
+ qq(
+SELECT grow_rel('pg_shseclabel', 4);
+SELECT modify_rel_block('pg_shseclabel', 2, corrupt_checksum=>true);
+SELECT modify_rel_block('pg_shseclabel', 3, corrupt_checksum=>true);
+));
+
+
+ # Check that page validity errors are detected, checksums stats increase, normal rel
+ my ($cs_count_before, $cs_ts_before) =
+ checksum_failures($psql_a, 'postgres');
+ psql_like(
+ $io_method,
+ $psql_a,
+ "normal rel: test reading of invalid block 3",
+ qq(
+SELECT read_rel_block_ll('tbl_normal', 3, nblocks=>1, zero_on_error=>false);),
+ qr/^$/,
+ qr/^psql:<stdin>:\d+: ERROR: invalid page in block 3 of relation base\/\d+\/\d+$/
+ );
+
+ my ($cs_count_after, $cs_ts_after) =
+ checksum_failures($psql_a, 'postgres');
+
+ cmp_ok($cs_count_before + 1,
+ '<=', $cs_count_after,
+ "$io_method: normal rel: checksum count increased");
+ cmp_ok($cs_ts_after, 'ne', '',
+ "$io_method: normal rel: checksum timestamp is not null");
+
+
+ # Check that page validity errors are detected, checksums stats increase, temp rel
+ ($cs_count_after, $cs_ts_after) = checksum_failures($psql_a, 'postgres');
+ psql_like(
+ $io_method,
+ $psql_a,
+ "temp rel: test reading of invalid block 4, valid block 5",
+ qq(
+SELECT read_rel_block_ll('tbl_temp', 4, nblocks=>2, zero_on_error=>false);),
+ qr/^$/,
+ qr/^psql:<stdin>:\d+: ERROR: invalid page in block 4 of relation base\/\d+\/t\d+_\d+$/
+ );
+
+ ($cs_count_after, $cs_ts_after) = checksum_failures($psql_a, 'postgres');
+
+ cmp_ok($cs_count_before + 1,
+ '<=', $cs_count_after,
+ "$io_method: temp rel: checksum count increased");
+ cmp_ok($cs_ts_after, 'ne', '',
+ "$io_method: temp rel: checksum timestamp is not null");
+
+
+ # Check that page validity errors are detected, checksums stats increase, shared rel
+ ($cs_count_before, $cs_ts_after) = checksum_failures($psql_a);
+ psql_like(
+ $io_method,
+ $psql_a,
+ "shared rel: reading of invalid blocks 2+3",
+ qq(
+SELECT read_rel_block_ll('pg_shseclabel', 2, nblocks=>2, zero_on_error=>false);),
+ qr/^$/,
+ qr/^psql:<stdin>:\d+: ERROR: 2 invalid pages among blocks 2..3 of relation global\/\d+\nDETAIL: Block 2 held first invalid page\.\nHINT:[^\n]+$/
+ );
+
+ ($cs_count_after, $cs_ts_after) = checksum_failures($psql_a);
+
+ cmp_ok($cs_count_before + 1,
+ '<=', $cs_count_after,
+ "$io_method: shared rel: checksum count increased");
+ cmp_ok($cs_ts_after, 'ne', '',
+ "$io_method: shared rel: checksum timestamp is not null");
+
+
+ # and restore sanity
+ $psql_a->query(
+ qq(
+SELECT modify_rel_block('pg_shseclabel', 1, zero=>true);
+DROP TABLE tbl_normal;
+));
+ $psql_a->{stderr} = '';
+
+ $psql_a->quit();
+}
+
+# Verify checksum handling when creating database from an invalid database.
+# This also serves as a minimal check that cross-database IO is handled
+# reasonably.
+sub test_checksum_createdb
+{
+ my $io_method = shift;
+ my $node = shift;
+
+ my $psql = $node->background_psql('postgres', on_error_stop => 0);
+
+ $node->safe_psql('postgres',
+ 'CREATE DATABASE regression_createdb_source');
+
+ $node->safe_psql(
+ 'regression_createdb_source', qq(
+CREATE EXTENSION test_aio;
+CREATE TABLE tbl_cs_fail(data int not null) WITH (AUTOVACUUM_ENABLED = false);
+INSERT INTO tbl_cs_fail SELECT generate_series(1, 1000);
+SELECT modify_rel_block('tbl_cs_fail', 1, corrupt_checksum=>true);
+));
+
+ my $createdb_sql = qq(
+CREATE DATABASE regression_createdb_target
+TEMPLATE regression_createdb_source
+STRATEGY wal_log;
+);
+
+ # Verify that CREATE DATABASE of an invalid database fails and is
+ # accounted for accurately.
+ my ($cs_count_before, $cs_ts_before) =
+ checksum_failures($psql, 'regression_createdb_source');
+ psql_like(
+ $io_method,
+ $psql,
+ "create database w/ wal strategy, invalid source",
+ $createdb_sql,
+ qr/^$/,
+ qr/^psql:<stdin>:\d+: ERROR: invalid page in block 1 of relation base\/\d+\/\d+$/
+ );
+ my ($cs_count_after, $cs_ts_after) =
+ checksum_failures($psql, 'regression_createdb_source');
+ cmp_ok($cs_count_before + 1, '<=', $cs_count_after,
+ "$io_method: create database w/ wal strategy, invalid source: checksum count increased"
+ );
+
+ # Verify that CREATE DATABASE of the fixed database succeeds.
+ $node->safe_psql(
+ 'regression_createdb_source', qq(
+SELECT modify_rel_block('tbl_cs_fail', 1, zero=>true);
+));
+ psql_like($io_method, $psql,
+ "create database w/ wal strategy, valid source",
+ $createdb_sql, qr/^$/, qr/^$/);
+
+ $psql->quit();
+}
+
+# Test that we detect checksum failures and report them
+sub test_ignore_checksum
+{
+ my $io_method = shift;
+ my $node = shift;
+
+ my $psql = $node->background_psql('postgres', on_error_stop => 0);
+
+ $psql->query_safe(
+ qq(
+CREATE TABLE tbl_cs_fail(id int) WITH (AUTOVACUUM_ENABLED = false);
+INSERT INTO tbl_cs_fail SELECT generate_series(1, 10000);
+));
+
+ my $count_sql = "SELECT count(*) FROM tbl_cs_fail";
+ my $invalidate_sql = qq(
+SELECT invalidate_rel_block('tbl_cs_fail', g.i)
+FROM generate_series(0, 6) g(i);
+);
+
+ my $expect = $psql->query_safe($count_sql);
+
+ $psql->query_safe(
+ qq(
+SELECT modify_rel_block('tbl_cs_fail', 1, corrupt_checksum=>true);
+SELECT modify_rel_block('tbl_cs_fail', 5, corrupt_checksum=>true);
+SELECT modify_rel_block('tbl_cs_fail', 6, corrupt_checksum=>true);
+));
+
+ $psql->query_safe($invalidate_sql);
+ psql_like($io_method, $psql,
+ "ignore_checksum_failure=off fails",
+ $count_sql, qr/^$/, qr/ERROR: invalid page in block/);
+
+ $psql->query_safe("SET ignore_checksum_failure=on");
+
+ $psql->query_safe($invalidate_sql);
+ psql_like($io_method, $psql,
+ "ignore_checksum_failure=on succeeds",
+ $count_sql,
+ qr/^$expect$/,
+ qr/WARNING: ignoring \d checksum failures among blocks/);
+
+
+ $psql->query_safe(
+ qq(
+SELECT modify_rel_block('tbl_cs_fail', 2, zero=>true);
+SELECT modify_rel_block('tbl_cs_fail', 3, corrupt_checksum=>true);
+SELECT modify_rel_block('tbl_cs_fail', 4, corrupt_header=>true);
+));
+
+ my $log_location = -s $node->logfile;
+ psql_like(
+ $io_method,
+ $psql,
+ "test reading of checksum failed block 3, with ignore",
+ qq(
+SELECT read_rel_block_ll('tbl_cs_fail', 3, nblocks=>1, zero_on_error=>false);),
+ qr/^$/,
+ qr/^psql:<stdin>:\d+: WARNING: ignoring checksum failure in block 3/
+ );
+
+ $log_location =
+ $node->wait_for_log(qr/LOG: ignoring checksum failure/, $log_location);
+
+ psql_like(
+ $io_method,
+ $psql,
+ "test reading of valid block 2, checksum failed 3, invalid 4, zero=false",
+ qq(
+SELECT read_rel_block_ll('tbl_cs_fail', 2, nblocks=>3, zero_on_error=>false);),
+ qr/^$/,
+ qr/^psql:<stdin>:\d+: ERROR: invalid page in block 4 of relation base\/\d+\/\d+$/
+ );
+
+ $psql->query(
+ qq(
+SELECT modify_rel_block('tbl_cs_fail', 1, zero=>true);
+SELECT modify_rel_block('tbl_cs_fail', 2, corrupt_checksum=>true);
+SELECT modify_rel_block('tbl_cs_fail', 3, corrupt_checksum=>true);
+SELECT modify_rel_block('tbl_cs_fail', 4, corrupt_header=>true);
+SELECT modify_rel_block('tbl_cs_fail', 5, corrupt_header=>true);
+SELECT modify_rel_block('tbl_cs_fail', 6, corrupt_header=>true);
+));
+ $psql->{stderr} = '';
+
+ $log_location = -s $node->logfile;
+ psql_like(
+ $io_method,
+ $psql,
+ "test reading of valid block 1, checksum failed 2-3, invalid 4-6, zero=true",
+ qq(
+SELECT read_rel_block_ll('tbl_cs_fail', 1, nblocks=>6, zero_on_error=>true);),
+ qr/^$/,
+ qr/^psql:<stdin>:\d+: WARNING: zeroing 3 page\(s\) and ignoring 2 checksum failure\(s\) among blocks 1..6 of relation/
+ );
+
+ # FIXME: "register" as tests, add more similar tests and explain why we want this.
+ $node->wait_for_log(qr/LOG: ignoring checksum failure in block 2/,
+ $log_location);
+ $node->wait_for_log(qr/LOG: ignoring checksum failure in block 3/,
+ $log_location);
+ $node->wait_for_log(qr/LOG: invalid page in block 4 of relation base.*; zeroing out page/,
+ $log_location);
+ $node->wait_for_log(qr/LOG: invalid page in block 5 of relation base.*; zeroing out page/,
+ $log_location);
+ $node->wait_for_log(qr/LOG: invalid page in block 6 of relation base.*; zeroing out page/,
+ $log_location);
+
+ $psql->query(
+ qq(
+SELECT modify_rel_block('tbl_cs_fail', 3, corrupt_checksum=>true, corrupt_header=>true);
+));
+ $psql->{stderr} = '';
+
+ # Reading a page with both an invalid header and an invalid checksum
+ psql_like(
+ $io_method,
+ $psql,
+ "test reading of block with both invalid header and invalid checksum, zero=false",
+ qq(
+SELECT read_rel_block_ll('tbl_cs_fail', 3, nblocks=>1, zero_on_error=>false);),
+ qr/^$/,
+ qr/^psql:<stdin>:\d+: ERROR: invalid page in block 3 of relation/
+ );
+
+ psql_like(
+ $io_method,
+ $psql,
+ "test reading of block 3 with both invalid header and invalid checksum, zero=true",
+ qq(
+SELECT read_rel_block_ll('tbl_cs_fail', 3, nblocks=>1, zero_on_error=>true);),
+ qr/^$/,
+ qr/^psql:<stdin>:\d+: WARNING: invalid page in block 3 of relation base\/.*; zeroing out page/
+ );
+
+
+ $psql->quit();
+}
+
+
+# Run all tests that are supported for all io_methods
+sub test_generic
+{
+ my $io_method = shift;
+ my $node = shift;
+
+ is($node->safe_psql('postgres', 'SHOW io_method'),
+ $io_method, "$io_method: io_method set correctly");
+
+ $node->safe_psql(
+ 'postgres', qq(
+CREATE EXTENSION test_aio;
+CREATE TABLE tbl_corr(data int not null) WITH (AUTOVACUUM_ENABLED = false);
+CREATE TABLE tbl_ok(data int not null) WITH (AUTOVACUUM_ENABLED = false);
+
+INSERT INTO tbl_corr SELECT generate_series(1, 10000);
+INSERT INTO tbl_ok SELECT generate_series(1, 10000);
+SELECT grow_rel('tbl_corr', 16);
+SELECT grow_rel('tbl_ok', 16);
+
+SELECT modify_rel_block('tbl_corr', 1, corrupt_header=>true);
+CHECKPOINT;
+));
+
+ test_handle($io_method, $node);
+ test_io_error($io_method, $node);
+ test_batchmode($io_method, $node);
+ test_startwait_io($io_method, $node);
+ test_complete_foreign($io_method, $node);
+ test_close_fd($io_method, $node);
+ test_invalidate($io_method, $node);
+ test_zero($io_method, $node);
+ test_checksum($io_method, $node);
+ test_ignore_checksum($io_method, $node);
+ test_checksum_createdb($io_method, $node);
+
+ SKIP:
+ {
+ skip 'Injection points not supported by this build', 1
+ unless $ENV{enable_injection_points} eq 'yes';
+ test_inject($io_method, $node);
+ }
+}
diff --git a/src/test/modules/test_aio/t/002_io_workers.pl b/src/test/modules/test_aio/t/002_io_workers.pl
new file mode 100644
index 00000000000..af5fae15ea7
--- /dev/null
+++ b/src/test/modules/test_aio/t/002_io_workers.pl
@@ -0,0 +1,125 @@
+# Copyright (c) 2025, PostgreSQL Global Development Group
+
+use strict;
+use warnings FATAL => 'all';
+
+use PostgreSQL::Test::Cluster;
+use PostgreSQL::Test::Utils;
+use Test::More;
+use List::Util qw(shuffle);
+
+
+my $node = PostgreSQL::Test::Cluster->new('worker');
+$node->init();
+$node->append_conf(
+ 'postgresql.conf', qq(
+io_method=worker
+));
+
+$node->start();
+
+# Test changing the number of I/O worker processes while also evaluating the
+# handling of their termination.
+test_number_of_io_workers_dynamic($node);
+
+$node->stop();
+
+done_testing();
+
+
+sub test_number_of_io_workers_dynamic
+{
+ my $node = shift;
+
+ my $prev_worker_count = $node->safe_psql('postgres', 'SHOW io_workers');
+
+ # Verify that worker count can't be set to 0
+ change_number_of_io_workers($node, 0, $prev_worker_count, 1);
+
+ # Verify that worker count can't be set to 33 (above the max)
+ change_number_of_io_workers($node, 33, $prev_worker_count, 1);
+
+ # Try changing IO workers to a random value and verify that the worker
+ # count ends up as expected. Always test the min/max of workers.
+ #
+ # Valid range for io_workers is [1, 32]. 8 tests in total seems
+ # reasonable.
+ my @io_workers_range = shuffle(1 ... 32);
+ foreach my $worker_count (1, 32, @io_workers_range[ 0, 6 ])
+ {
+ $prev_worker_count =
+ change_number_of_io_workers($node, $worker_count,
+ $prev_worker_count, 0);
+ }
+}
+
+sub change_number_of_io_workers
+{
+ my $node = shift;
+ my $worker_count = shift;
+ my $prev_worker_count = shift;
+ my $expect_failure = shift;
+ my ($result, $stdout, $stderr);
+
+ ($result, $stdout, $stderr) =
+ $node->psql('postgres', "ALTER SYSTEM SET io_workers = $worker_count");
+ $node->safe_psql('postgres', 'SELECT pg_reload_conf()');
+
+ if ($expect_failure)
+ {
+ ok( $stderr =~
+ /$worker_count is outside the valid range for parameter "io_workers"/,
+ "updating number of io_workers to $worker_count failed, as expected"
+ );
+
+ return $prev_worker_count;
+ }
+ else
+ {
+ is( $node->safe_psql('postgres', 'SHOW io_workers'),
+ $worker_count,
+ "updating number of io_workers from $prev_worker_count to $worker_count"
+ );
+
+ check_io_worker_count($node, $worker_count);
+ terminate_io_worker($node, $worker_count);
+ check_io_worker_count($node, $worker_count);
+
+ return $worker_count;
+ }
+}
+
+sub terminate_io_worker
+{
+ my $node = shift;
+ my $worker_count = shift;
+ my ($pid, $ret);
+
+ # Select a random io worker
+ $pid = $node->safe_psql(
+ 'postgres',
+ qq(SELECT pid FROM pg_stat_activity WHERE
+ backend_type = 'io worker' ORDER BY RANDOM() LIMIT 1));
+
+ # terminate IO worker with SIGINT
+ is(PostgreSQL::Test::Utils::system_log('pg_ctl', 'kill', 'INT', $pid),
+ 0, "random io worker process signalled with INT");
+
+ # Check that worker exits
+ ok( $node->poll_query_until(
+ 'postgres',
+ qq(SELECT COUNT(*) FROM pg_stat_activity WHERE pid = $pid), '0'),
+ "random io worker process exited after signal");
+}
+
+sub check_io_worker_count
+{
+ my $node = shift;
+ my $worker_count = shift;
+
+ ok( $node->poll_query_until(
+ 'postgres',
+ qq(SELECT COUNT(*) FROM pg_stat_activity WHERE backend_type = 'io worker'),
+ $worker_count),
+ "io worker count is $worker_count");
+}
diff --git a/src/test/modules/test_aio/test_aio--1.0.sql b/src/test/modules/test_aio/test_aio--1.0.sql
new file mode 100644
index 00000000000..e495481c41e
--- /dev/null
+++ b/src/test/modules/test_aio/test_aio--1.0.sql
@@ -0,0 +1,108 @@
+/* src/test/modules/test_aio/test_aio--1.0.sql */
+
+-- complain if script is sourced in psql, rather than via CREATE EXTENSION
+\echo Use "CREATE EXTENSION test_aio" to load this file. \quit
+
+
+CREATE FUNCTION errno_from_string(sym text)
+RETURNS pg_catalog.int4 STRICT
+AS 'MODULE_PATHNAME' LANGUAGE C;
+
+
+CREATE FUNCTION grow_rel(rel regclass, nblocks int)
+RETURNS pg_catalog.void STRICT
+AS 'MODULE_PATHNAME' LANGUAGE C;
+
+
+CREATE FUNCTION modify_rel_block(rel regclass, blockno int,
+ zero bool DEFAULT false,
+ corrupt_header bool DEFAULT false,
+ corrupt_checksum bool DEFAULT false)
+RETURNS pg_catalog.void STRICT
+AS 'MODULE_PATHNAME' LANGUAGE C;
+
+CREATE FUNCTION read_rel_block_ll(
+ rel regclass,
+ blockno int,
+ nblocks int DEFAULT 1,
+ wait_complete bool DEFAULT true,
+ batchmode_enter bool DEFAULT false,
+ smgrreleaseall bool DEFAULT false,
+ batchmode_exit bool DEFAULT false,
+ zero_on_error bool DEFAULT false)
+RETURNS pg_catalog.void STRICT
+AS 'MODULE_PATHNAME' LANGUAGE C;
+
+CREATE FUNCTION invalidate_rel_block(rel regclass, blockno int)
+RETURNS pg_catalog.void STRICT
+AS 'MODULE_PATHNAME' LANGUAGE C;
+
+CREATE FUNCTION buffer_create_toy(rel regclass, blockno int4)
+RETURNS pg_catalog.int4 STRICT
+AS 'MODULE_PATHNAME' LANGUAGE C;
+
+CREATE FUNCTION buffer_call_start_io(buffer int, for_input bool, nowait bool)
+RETURNS pg_catalog.bool STRICT
+AS 'MODULE_PATHNAME' LANGUAGE C;
+
+CREATE FUNCTION buffer_call_terminate_io(buffer int, for_input bool, succeed bool, io_error bool, release_aio bool)
+RETURNS pg_catalog.void STRICT
+AS 'MODULE_PATHNAME' LANGUAGE C;
+
+
+
+/*
+ * Handle related functions
+ */
+CREATE FUNCTION handle_get_and_error()
+RETURNS pg_catalog.void STRICT
+AS 'MODULE_PATHNAME' LANGUAGE C;
+
+CREATE FUNCTION handle_get_twice()
+RETURNS pg_catalog.void STRICT
+AS 'MODULE_PATHNAME' LANGUAGE C;
+
+CREATE FUNCTION handle_get()
+RETURNS pg_catalog.void STRICT
+AS 'MODULE_PATHNAME' LANGUAGE C;
+
+CREATE FUNCTION handle_get_release()
+RETURNS pg_catalog.void STRICT
+AS 'MODULE_PATHNAME' LANGUAGE C;
+
+CREATE FUNCTION handle_release_last()
+RETURNS pg_catalog.void STRICT
+AS 'MODULE_PATHNAME' LANGUAGE C;
+
+
+/*
+ * Batchmode related functions
+ */
+CREATE FUNCTION batch_start()
+RETURNS pg_catalog.void STRICT
+AS 'MODULE_PATHNAME' LANGUAGE C;
+
+CREATE FUNCTION batch_end()
+RETURNS pg_catalog.void STRICT
+AS 'MODULE_PATHNAME' LANGUAGE C;
+
+
+
+/*
+ * Injection point related functions
+ */
+CREATE FUNCTION inj_io_short_read_attach(result int)
+RETURNS pg_catalog.void STRICT
+AS 'MODULE_PATHNAME' LANGUAGE C;
+
+CREATE FUNCTION inj_io_short_read_detach()
+RETURNS pg_catalog.void STRICT
+AS 'MODULE_PATHNAME' LANGUAGE C;
+
+CREATE FUNCTION inj_io_reopen_attach()
+RETURNS pg_catalog.void STRICT
+AS 'MODULE_PATHNAME' LANGUAGE C;
+
+CREATE FUNCTION inj_io_reopen_detach()
+RETURNS pg_catalog.void STRICT
+AS 'MODULE_PATHNAME' LANGUAGE C;
diff --git a/src/test/modules/test_aio/test_aio.c b/src/test/modules/test_aio/test_aio.c
new file mode 100644
index 00000000000..35c6b331923
--- /dev/null
+++ b/src/test/modules/test_aio/test_aio.c
@@ -0,0 +1,802 @@
+/*-------------------------------------------------------------------------
+ *
+ * test_aio.c
+ * Helpers to write tests for AIO
+ *
+ * This module provides interface functions for C functionality to SQL, to
+ * make it possible to test AIO related behavior in a targeted way from SQL.
+ * It'd not generally be safe to export these functions to SQL, but for a test
+ * that's fine.
+ *
+ * Copyright (c) 2020-2025, PostgreSQL Global Development Group
+ *
+ * IDENTIFICATION
+ * src/test/modules/test_aio/test_aio.c
+ *
+ *-------------------------------------------------------------------------
+ */
+
+#include "postgres.h"
+
+#include "access/relation.h"
+#include "fmgr.h"
+#include "storage/aio.h"
+#include "storage/aio_internal.h"
+#include "storage/buf_internals.h"
+#include "storage/bufmgr.h"
+#include "storage/checksum.h"
+#include "storage/ipc.h"
+#include "storage/lwlock.h"
+#include "utils/builtins.h"
+#include "utils/injection_point.h"
+#include "utils/rel.h"
+
+
+PG_MODULE_MAGIC;
+
+
+typedef struct InjIoErrorState
+{
+ bool enabled_short_read;
+ bool enabled_reopen;
+
+ bool short_read_result_set;
+ int short_read_result;
+} InjIoErrorState;
+
+static InjIoErrorState * inj_io_error_state;
+
+/* Shared memory init callbacks */
+static shmem_request_hook_type prev_shmem_request_hook = NULL;
+static shmem_startup_hook_type prev_shmem_startup_hook = NULL;
+
+
+static PgAioHandle *last_handle;
+
+
+
+static void
+test_aio_shmem_request(void)
+{
+ if (prev_shmem_request_hook)
+ prev_shmem_request_hook();
+
+ RequestAddinShmemSpace(sizeof(InjIoErrorState));
+}
+
+static void
+test_aio_shmem_startup(void)
+{
+ bool found;
+
+ if (prev_shmem_startup_hook)
+ prev_shmem_startup_hook();
+
+ /* Create or attach to the shared memory state */
+ LWLockAcquire(AddinShmemInitLock, LW_EXCLUSIVE);
+
+ inj_io_error_state = ShmemInitStruct("injection_points",
+ sizeof(InjIoErrorState),
+ &found);
+
+ if (!found)
+ {
+ /*
+ * First time through, so initialize. This is shared with the dynamic
+ * initialization using a DSM.
+ */
+ inj_io_error_state->enabled_short_read = false;
+ inj_io_error_state->enabled_reopen = false;
+
+#ifdef USE_INJECTION_POINTS
+ InjectionPointAttach("AIO_PROCESS_COMPLETION_BEFORE_SHARED",
+ "test_aio",
+ "inj_io_short_read",
+ NULL,
+ 0);
+ InjectionPointLoad("AIO_PROCESS_COMPLETION_BEFORE_SHARED");
+
+ InjectionPointAttach("AIO_WORKER_AFTER_REOPEN",
+ "test_aio",
+ "inj_io_reopen",
+ NULL,
+ 0);
+ InjectionPointLoad("AIO_WORKER_AFTER_REOPEN");
+
+#endif
+ }
+ else
+ {
+#ifdef USE_INJECTION_POINTS
+ InjectionPointLoad("AIO_PROCESS_COMPLETION_BEFORE_SHARED");
+ InjectionPointLoad("AIO_WORKER_AFTER_REOPEN");
+ elog(LOG, "injection point loaded");
+#endif
+ }
+
+ LWLockRelease(AddinShmemInitLock);
+}
+
+void
+_PG_init(void)
+{
+ if (!process_shared_preload_libraries_in_progress)
+ return;
+
+ /* Shared memory initialization */
+ prev_shmem_request_hook = shmem_request_hook;
+ shmem_request_hook = test_aio_shmem_request;
+ prev_shmem_startup_hook = shmem_startup_hook;
+ shmem_startup_hook = test_aio_shmem_startup;
+}
+
+
+PG_FUNCTION_INFO_V1(errno_from_string);
+Datum
+errno_from_string(PG_FUNCTION_ARGS)
+{
+ const char *sym = text_to_cstring(PG_GETARG_TEXT_PP(0));
+
+ if (strcmp(sym, "EIO") == 0)
+ PG_RETURN_INT32(EIO);
+ else if (strcmp(sym, "EAGAIN") == 0)
+ PG_RETURN_INT32(EAGAIN);
+ else if (strcmp(sym, "EINTR") == 0)
+ PG_RETURN_INT32(EINTR);
+ else if (strcmp(sym, "ENOSPC") == 0)
+ PG_RETURN_INT32(ENOSPC);
+ else if (strcmp(sym, "EROFS") == 0)
+ PG_RETURN_INT32(EROFS);
+
+ ereport(ERROR,
+ errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ errmsg_internal("%s is not a supported errno value", sym));
+ PG_RETURN_INT32(0);
+}
+
+
+PG_FUNCTION_INFO_V1(grow_rel);
+Datum
+grow_rel(PG_FUNCTION_ARGS)
+{
+ Oid relid = PG_GETARG_OID(0);
+ uint32 nblocks = PG_GETARG_UINT32(1);
+ Relation rel;
+#define MAX_BUFFERS_TO_EXTEND_BY 64
+ Buffer victim_buffers[MAX_BUFFERS_TO_EXTEND_BY];
+
+ rel = relation_open(relid, AccessExclusiveLock);
+
+ while (nblocks > 0)
+ {
+ uint32 extend_by_pages;
+
+ extend_by_pages = Min(nblocks, MAX_BUFFERS_TO_EXTEND_BY);
+
+ ExtendBufferedRelBy(BMR_REL(rel),
+ MAIN_FORKNUM,
+ NULL,
+ 0,
+ extend_by_pages,
+ victim_buffers,
+ &extend_by_pages);
+
+ nblocks -= extend_by_pages;
+
+ for (uint32 i = 0; i < extend_by_pages; i++)
+ {
+ ReleaseBuffer(victim_buffers[i]);
+ }
+ }
+
+ relation_close(rel, NoLock);
+
+ PG_RETURN_VOID();
+}
+
+PG_FUNCTION_INFO_V1(modify_rel_block);
+Datum
+modify_rel_block(PG_FUNCTION_ARGS)
+{
+ Oid relid = PG_GETARG_OID(0);
+ BlockNumber blkno = PG_GETARG_UINT32(1);
+ bool zero = PG_GETARG_BOOL(2);
+ bool corrupt_header = PG_GETARG_BOOL(3);
+ bool corrupt_checksum = PG_GETARG_BOOL(4);
+ Page page = palloc_aligned(BLCKSZ, PG_IO_ALIGN_SIZE, 0);
+ Relation rel;
+ Buffer buf;
+ PageHeader ph;
+
+ rel = relation_open(relid, AccessExclusiveLock);
+
+ buf = ReadBufferExtended(rel, MAIN_FORKNUM, blkno,
+ RBM_ZERO_ON_ERROR, NULL);
+
+ LockBuffer(buf, BUFFER_LOCK_EXCLUSIVE);
+
+ /*
+ * copy the page to local memory, seems nicer than to directly modify in
+ * the buffer pool.
+ */
+ memcpy(page, BufferGetPage(buf), BLCKSZ);
+
+ LockBuffer(buf, BUFFER_LOCK_UNLOCK);
+
+ ReleaseBuffer(buf);
+
+ /*
+ * Don't want to have a buffer in-memory that's marked valid where the
+ * on-disk contents are invalid. Particularly not if the in-memory buffer
+ * could be dirty...
+ *
+ * While we hold an AEL on the relation nobody else should be able to read
+ * the buffer in.
+ *
+ * NB: This is probably racy, better don't copy this to non-test code.
+ */
+ if (BufferIsLocal(buf))
+ InvalidateLocalBuffer(GetLocalBufferDescriptor(-buf - 1), true);
+ else
+ EvictUnpinnedBuffer(buf);
+
+ /*
+ * Now modify the page as asked for by the caller.
+ */
+ if (zero)
+ memset(page, 0, BufferGetPageSize(buf));
+
+ if (PageIsEmpty(page) && (corrupt_header || corrupt_checksum))
+ PageInit(page, BufferGetPageSize(buf), 0);
+
+ ph = (PageHeader) page;
+
+ if (corrupt_header)
+ ph->pd_special = BLCKSZ + 1;
+
+ if (corrupt_checksum)
+ {
+ bool successfully_corrupted = 0;
+
+ /*
+ * Any single modification of the checksum could just end up being
+ * valid again. To be sure
+ */
+ for (int i = 0; i < 100; i++)
+ {
+ uint16 verify_checksum;
+ uint16 old_checksum;
+
+ old_checksum = ph->pd_checksum;
+ ph->pd_checksum = old_checksum + 3;
+
+ elog(LOG, "corrupting checksum of blk %u from %u to %u",
+ blkno, old_checksum, ph->pd_checksum);
+
+ verify_checksum = pg_checksum_page(page, blkno);
+ if (verify_checksum != ph->pd_checksum)
+ {
+ successfully_corrupted = true;
+ break;
+ }
+ }
+
+ if (!successfully_corrupted)
+ elog(ERROR, "could not corrupt checksum, what's going on?");
+ }
+ else
+ {
+ PageSetChecksumInplace(page, blkno);
+ }
+
+ smgrwrite(RelationGetSmgr(rel),
+ MAIN_FORKNUM, blkno, page, true);
+
+ relation_close(rel, NoLock);
+
+ PG_RETURN_VOID();
+}
+
+/*
+ * Ensures a buffer for rel & blkno is in shared buffers, without actually
+ * caring about the buffer contents. Used to set up test scenarios.
+ */
+static Buffer
+create_toy_buffer(Relation rel, BlockNumber blkno)
+{
+ Buffer buf;
+ BufferDesc *buf_hdr;
+ uint32 buf_state;
+ bool was_pinned = false;
+
+ /* place buffer in shared buffers without erroring out */
+ buf = ReadBufferExtended(rel, MAIN_FORKNUM, blkno, RBM_ZERO_AND_LOCK, NULL);
+ LockBuffer(buf, BUFFER_LOCK_UNLOCK);
+
+ if (RelationUsesLocalBuffers(rel))
+ {
+ buf_hdr = GetLocalBufferDescriptor(-buf - 1);
+ buf_state = pg_atomic_read_u32(&buf_hdr->state);
+ }
+ else
+ {
+ buf_hdr = GetBufferDescriptor(buf - 1);
+ buf_state = LockBufHdr(buf_hdr);
+ }
+
+ /*
+ * We should be the only backend accessing this buffer. This is just a
+ * small bit of belt-and-suspenders defense, none of this code should ever
+ * run in a cluster with real data.
+ */
+ if (BUF_STATE_GET_REFCOUNT(buf_state) > 1)
+ was_pinned = true;
+ else
+ buf_state &= ~(BM_VALID | BM_DIRTY);
+
+ if (RelationUsesLocalBuffers(rel))
+ pg_atomic_unlocked_write_u32(&buf_hdr->state, buf_state);
+ else
+ UnlockBufHdr(buf_hdr, buf_state);
+
+ if (was_pinned)
+ elog(ERROR, "toy buffer %d was already pinned",
+ buf);
+
+ return buf;
+}
+
+/*
+ * A "low level" read. This does similar things to what
+ * StartReadBuffers()/WaitReadBuffers() do, but provides more control (and
+ * less sanity).
+ */
+PG_FUNCTION_INFO_V1(read_rel_block_ll);
+Datum
+read_rel_block_ll(PG_FUNCTION_ARGS)
+{
+ Oid relid = PG_GETARG_OID(0);
+ BlockNumber blkno = PG_GETARG_UINT32(1);
+ int nblocks = PG_GETARG_INT32(2);
+ bool wait_complete = PG_GETARG_BOOL(3);
+ bool batchmode_enter = PG_GETARG_BOOL(4);
+ bool call_smgrreleaseall = PG_GETARG_BOOL(5);
+ bool batchmode_exit = PG_GETARG_BOOL(6);
+ bool zero_on_error = PG_GETARG_BOOL(7);
+ Relation rel;
+ Buffer bufs[PG_IOV_MAX];
+ BufferDesc *buf_hdrs[PG_IOV_MAX];
+ Page pages[PG_IOV_MAX];
+ uint8 srb_flags = 0;
+ PgAioReturn ior;
+ PgAioHandle *ioh;
+ PgAioWaitRef iow;
+ SMgrRelation smgr;
+
+ if (nblocks <= 0 || nblocks > PG_IOV_MAX)
+ elog(ERROR, "nblocks is out of range");
+
+ rel = relation_open(relid, AccessExclusiveLock);
+
+ for (int i = 0; i < nblocks; i++)
+ {
+ bufs[i] = create_toy_buffer(rel, blkno + i);
+ pages[i] = BufferGetBlock(bufs[i]);
+ buf_hdrs[i] = BufferIsLocal(bufs[i]) ?
+ GetLocalBufferDescriptor(-bufs[i] - 1) :
+ GetBufferDescriptor(bufs[i] - 1);
+ }
+
+ smgr = RelationGetSmgr(rel);
+
+ pgstat_prepare_report_checksum_failure(smgr->smgr_rlocator.locator.dbOid);
+
+ ioh = pgaio_io_acquire(CurrentResourceOwner, &ior);
+ pgaio_io_get_wref(ioh, &iow);
+
+ if (RelationUsesLocalBuffers(rel))
+ {
+ for (int i = 0; i < nblocks; i++)
+ StartLocalBufferIO(buf_hdrs[i], true, false);
+ pgaio_io_set_flag(ioh, PGAIO_HF_REFERENCES_LOCAL);
+ }
+ else
+ {
+ for (int i = 0; i < nblocks; i++)
+ StartBufferIO(buf_hdrs[i], true, false);
+ }
+
+ pgaio_io_set_handle_data_32(ioh, (uint32 *) bufs, nblocks);
+
+ if (zero_on_error | zero_damaged_pages)
+ srb_flags |= READ_BUFFERS_ZERO_ON_ERROR;
+ if (ignore_checksum_failure)
+ srb_flags |= READ_BUFFERS_IGNORE_CHECKSUM_FAILURES;
+
+ pgaio_io_register_callbacks(ioh,
+ RelationUsesLocalBuffers(rel) ?
+ PGAIO_HCB_LOCAL_BUFFER_READV :
+ PGAIO_HCB_SHARED_BUFFER_READV,
+ srb_flags);
+
+ if (batchmode_enter)
+ pgaio_enter_batchmode();
+
+ elog(LOG, "about to smgrstartreadv");
+ smgrstartreadv(ioh, smgr, MAIN_FORKNUM, blkno,
+ (void *) pages, nblocks);
+
+ if (call_smgrreleaseall)
+ smgrreleaseall();
+
+ if (batchmode_exit)
+ pgaio_exit_batchmode();
+
+ for (int i = 0; i < nblocks; i++)
+ ReleaseBuffer(bufs[i]);
+
+ if (wait_complete)
+ {
+ pgaio_wref_wait(&iow);
+
+ if (ior.result.status != PGAIO_RS_OK)
+ pgaio_result_report(ior.result,
+ &ior.target_data,
+ ior.result.status == PGAIO_RS_ERROR ?
+ ERROR : WARNING);
+ }
+
+ relation_close(rel, NoLock);
+
+ PG_RETURN_VOID();
+}
+
+PG_FUNCTION_INFO_V1(invalidate_rel_block);
+Datum
+invalidate_rel_block(PG_FUNCTION_ARGS)
+{
+ Oid relid = PG_GETARG_OID(0);
+ BlockNumber blkno = PG_GETARG_UINT32(1);
+ Relation rel;
+ PrefetchBufferResult pr;
+ Buffer buf;
+
+ rel = relation_open(relid, AccessExclusiveLock);
+
+ /*
+ * This is a gross hack, but there's no other API exposed that allows to
+ * get a buffer ID without actually reading the block in.
+ */
+ pr = PrefetchBuffer(rel, MAIN_FORKNUM, blkno);
+ buf = pr.recent_buffer;
+
+ if (BufferIsValid(buf))
+ {
+ /* if the buffer contents aren't valid, this'll return false */
+ if (ReadRecentBuffer(rel->rd_locator, MAIN_FORKNUM, blkno, buf))
+ {
+ BufferDesc *buf_hdr = BufferIsLocal(buf) ?
+ GetLocalBufferDescriptor(-buf - 1)
+ : GetBufferDescriptor(buf - 1);
+
+ LockBuffer(buf, BUFFER_LOCK_EXCLUSIVE);
+
+ if (pg_atomic_read_u32(&buf_hdr->state) & BM_DIRTY)
+ {
+ if (BufferIsLocal(buf))
+ FlushLocalBuffer(buf_hdr, NULL);
+ else
+ FlushOneBuffer(buf);
+ }
+ LockBuffer(buf, BUFFER_LOCK_UNLOCK);
+ ReleaseBuffer(buf);
+
+ if (BufferIsLocal(buf))
+ InvalidateLocalBuffer(GetLocalBufferDescriptor(-buf - 1), true);
+ else if (!EvictUnpinnedBuffer(buf))
+ elog(ERROR, "couldn't evict");
+ }
+ }
+
+ relation_close(rel, AccessExclusiveLock);
+
+ PG_RETURN_VOID();
+}
+
+PG_FUNCTION_INFO_V1(buffer_create_toy);
+Datum
+buffer_create_toy(PG_FUNCTION_ARGS)
+{
+ Oid relid = PG_GETARG_OID(0);
+ BlockNumber blkno = PG_GETARG_UINT32(1);
+ Relation rel;
+ Buffer buf;
+
+ rel = relation_open(relid, AccessExclusiveLock);
+
+ buf = create_toy_buffer(rel, blkno);
+ ReleaseBuffer(buf);
+
+ relation_close(rel, NoLock);
+
+ PG_RETURN_INT32(buf);
+}
+
+PG_FUNCTION_INFO_V1(buffer_call_start_io);
+Datum
+buffer_call_start_io(PG_FUNCTION_ARGS)
+{
+ Buffer buf = PG_GETARG_INT32(0);
+ bool for_input = PG_GETARG_BOOL(1);
+ bool nowait = PG_GETARG_BOOL(2);
+ bool can_start;
+
+ if (BufferIsLocal(buf))
+ can_start = StartLocalBufferIO(GetLocalBufferDescriptor(-buf - 1),
+ for_input, nowait);
+ else
+ can_start = StartBufferIO(GetBufferDescriptor(buf - 1),
+ for_input, nowait);
+
+ /*
+ * For tests we don't want the resowner release preventing us from
+ * orchestrating odd scenarios.
+ */
+ if (can_start && !BufferIsLocal(buf))
+ ResourceOwnerForgetBufferIO(CurrentResourceOwner,
+ buf);
+
+ ereport(LOG,
+ errmsg("buffer %d after StartBufferIO: %s",
+ buf, DebugPrintBufferRefcount(buf)),
+ errhidestmt(true), errhidecontext(true));
+
+ PG_RETURN_BOOL(can_start);
+}
+
+PG_FUNCTION_INFO_V1(buffer_call_terminate_io);
+Datum
+buffer_call_terminate_io(PG_FUNCTION_ARGS)
+{
+ Buffer buf = PG_GETARG_INT32(0);
+ bool for_input = PG_GETARG_BOOL(1);
+ bool succeed = PG_GETARG_BOOL(2);
+ bool io_error = PG_GETARG_BOOL(3);
+ bool release_aio = PG_GETARG_BOOL(4);
+ bool clear_dirty = false;
+ uint32 set_flag_bits = 0;
+
+ if (io_error)
+ set_flag_bits |= BM_IO_ERROR;
+
+ if (for_input)
+ {
+ clear_dirty = false;
+
+ if (succeed)
+ set_flag_bits |= BM_VALID;
+ }
+ else
+ {
+ if (succeed)
+ clear_dirty = true;
+ }
+
+ ereport(LOG,
+ errmsg("buffer %d before Terminate[Local]BufferIO: %s",
+ buf, DebugPrintBufferRefcount(buf)),
+ errhidestmt(true), errhidecontext(true));
+
+ if (BufferIsLocal(buf))
+ TerminateLocalBufferIO(GetLocalBufferDescriptor(-buf - 1),
+ clear_dirty, set_flag_bits, release_aio);
+ else
+ TerminateBufferIO(GetBufferDescriptor(buf - 1),
+ clear_dirty, set_flag_bits, false, release_aio);
+
+ ereport(LOG,
+ errmsg("buffer %d after Terminate[Local]BufferIO: %s",
+ buf, DebugPrintBufferRefcount(buf)),
+ errhidestmt(true), errhidecontext(true));
+
+ PG_RETURN_VOID();
+}
+
+PG_FUNCTION_INFO_V1(handle_get);
+Datum
+handle_get(PG_FUNCTION_ARGS)
+{
+ last_handle = pgaio_io_acquire(CurrentResourceOwner, NULL);
+
+ PG_RETURN_VOID();
+}
+
+PG_FUNCTION_INFO_V1(handle_release_last);
+Datum
+handle_release_last(PG_FUNCTION_ARGS)
+{
+ if (!last_handle)
+ elog(ERROR, "no handle");
+
+ pgaio_io_release(last_handle);
+
+ PG_RETURN_VOID();
+}
+
+PG_FUNCTION_INFO_V1(handle_get_and_error);
+Datum
+handle_get_and_error(PG_FUNCTION_ARGS)
+{
+ pgaio_io_acquire(CurrentResourceOwner, NULL);
+
+ elog(ERROR, "as you command");
+ PG_RETURN_VOID();
+}
+
+PG_FUNCTION_INFO_V1(handle_get_twice);
+Datum
+handle_get_twice(PG_FUNCTION_ARGS)
+{
+ pgaio_io_acquire(CurrentResourceOwner, NULL);
+ pgaio_io_acquire(CurrentResourceOwner, NULL);
+
+ PG_RETURN_VOID();
+}
+
+PG_FUNCTION_INFO_V1(handle_get_release);
+Datum
+handle_get_release(PG_FUNCTION_ARGS)
+{
+ PgAioHandle *handle;
+
+ handle = pgaio_io_acquire(CurrentResourceOwner, NULL);
+ pgaio_io_release(handle);
+
+ PG_RETURN_VOID();
+}
+
+PG_FUNCTION_INFO_V1(batch_start);
+Datum
+batch_start(PG_FUNCTION_ARGS)
+{
+ pgaio_enter_batchmode();
+ PG_RETURN_VOID();
+}
+
+PG_FUNCTION_INFO_V1(batch_end);
+Datum
+batch_end(PG_FUNCTION_ARGS)
+{
+ pgaio_exit_batchmode();
+ PG_RETURN_VOID();
+}
+
+#ifdef USE_INJECTION_POINTS
+extern PGDLLEXPORT void inj_io_short_read(const char *name, const void *private_data);
+extern PGDLLEXPORT void inj_io_reopen(const char *name, const void *private_data);
+
+void
+inj_io_short_read(const char *name, const void *private_data)
+{
+ PgAioHandle *ioh;
+
+ elog(LOG, "short read called: %d", inj_io_error_state->enabled_short_read);
+
+ if (inj_io_error_state->enabled_short_read)
+ {
+ ioh = pgaio_inj_io_get();
+
+ /*
+ * Only shorten reads that are actually longer than the target size,
+ * otherwise we can trigger over-reads.
+ */
+ if (inj_io_error_state->short_read_result_set
+ && ioh->op == PGAIO_OP_READV
+ && inj_io_error_state->short_read_result <= ioh->result)
+ {
+ struct iovec *iov = &pgaio_ctl->iovecs[ioh->iovec_off];
+ int32 old_result = ioh->result;
+ int32 new_result = inj_io_error_state->short_read_result;
+ int32 processed = 0;
+
+ ereport(LOG,
+ errmsg("short read, changing result from %d to %d",
+ old_result, new_result),
+ errhidestmt(true), errhidecontext(true));
+
+ /*
+ * The underlying IO actually completed OK, and thus the "invalid"
+ * portion of the IOV actually contains valid data. That can hide
+ * a lot of problems, e.g. if we were to wrongly mark a buffer,
+ * that wasn't read according to the shortened-read, IO as valid,
+ * the contents would look valid and we might miss a bug.
+ *
+ * To avoid that, iterate through the IOV and zero out the
+ * "failed" portion of the IO.
+ */
+ for (int i = 0; i < ioh->op_data.read.iov_length; i++)
+ {
+ if (processed + iov[i].iov_len <= new_result)
+ processed += iov[i].iov_len;
+ else if (processed <= new_result)
+ {
+ uint32 ok_part = new_result - processed;
+
+ memset((char *) iov[i].iov_base + ok_part, 0, iov[i].iov_len - ok_part);
+ processed += iov[i].iov_len;
+ }
+ else
+ {
+ memset((char *) iov[i].iov_base, 0, iov[i].iov_len);
+ }
+ }
+
+ ioh->result = new_result;
+ }
+ }
+}
+
+void
+inj_io_reopen(const char *name, const void *private_data)
+{
+ elog(LOG, "reopen called: %d", inj_io_error_state->enabled_reopen);
+
+ if (inj_io_error_state->enabled_reopen)
+ {
+ elog(ERROR, "injection point triggering failure to reopen ");
+ }
+}
+#endif
+
+PG_FUNCTION_INFO_V1(inj_io_short_read_attach);
+Datum
+inj_io_short_read_attach(PG_FUNCTION_ARGS)
+{
+#ifdef USE_INJECTION_POINTS
+ inj_io_error_state->enabled_short_read = true;
+ inj_io_error_state->short_read_result_set = !PG_ARGISNULL(0);
+ if (inj_io_error_state->short_read_result_set)
+ inj_io_error_state->short_read_result = PG_GETARG_INT32(0);
+#else
+ elog(ERROR, "injection points not supported");
+#endif
+
+ PG_RETURN_VOID();
+}
+
+PG_FUNCTION_INFO_V1(inj_io_short_read_detach);
+Datum
+inj_io_short_read_detach(PG_FUNCTION_ARGS)
+{
+#ifdef USE_INJECTION_POINTS
+ inj_io_error_state->enabled_short_read = false;
+#else
+ elog(ERROR, "injection points not supported");
+#endif
+ PG_RETURN_VOID();
+}
+
+PG_FUNCTION_INFO_V1(inj_io_reopen_attach);
+Datum
+inj_io_reopen_attach(PG_FUNCTION_ARGS)
+{
+#ifdef USE_INJECTION_POINTS
+ inj_io_error_state->enabled_reopen = true;
+#else
+ elog(ERROR, "injection points not supported");
+#endif
+
+ PG_RETURN_VOID();
+}
+
+PG_FUNCTION_INFO_V1(inj_io_reopen_detach);
+Datum
+inj_io_reopen_detach(PG_FUNCTION_ARGS)
+{
+#ifdef USE_INJECTION_POINTS
+ inj_io_error_state->enabled_reopen = false;
+#else
+ elog(ERROR, "injection points not supported");
+#endif
+ PG_RETURN_VOID();
+}
diff --git a/src/test/modules/test_aio/test_aio.control b/src/test/modules/test_aio/test_aio.control
new file mode 100644
index 00000000000..cd91c3ed16b
--- /dev/null
+++ b/src/test/modules/test_aio/test_aio.control
@@ -0,0 +1,3 @@
+comment = 'Test code for AIO'
+default_version = '1.0'
+module_pathname = '$libdir/test_aio'
--
2.48.1.76.g4e746b1a31.dirty
[text/x-diff] v2.14-0019-aio-Experimental-heuristics-to-increase-batchi.patch (1.6K, ../../d4ff3qxudxn2ze3zk2xwhds5ysezfei3uqeuimymj624t5chsd@sgr2h3rmln5t/20-v2.14-0019-aio-Experimental-heuristics-to-increase-batchi.patch)
download | inline diff:
From 7d9911908130fd99b502c6c2d7573fb28a542c59 Mon Sep 17 00:00:00 2001
From: Andres Freund <andres@anarazel.de>
Date: Tue, 18 Mar 2025 14:40:06 -0400
Subject: [PATCH v2.14 19/29] aio: Experimental heuristics to increase batching
in read_stream.c
Author:
Reviewed-by:
Discussion: https://postgr.es/m/
Backpatch:
---
src/backend/storage/aio/read_stream.c | 24 ++++++++++++++++++++++++
1 file changed, 24 insertions(+)
diff --git a/src/backend/storage/aio/read_stream.c b/src/backend/storage/aio/read_stream.c
index 36c54fb695b..cec93129f58 100644
--- a/src/backend/storage/aio/read_stream.c
+++ b/src/backend/storage/aio/read_stream.c
@@ -404,6 +404,30 @@ read_stream_start_pending_read(ReadStream *stream)
static void
read_stream_look_ahead(ReadStream *stream)
{
+ /*
+ * Batch-submitting multiple IOs is more efficient than doing so
+ * one-by-one. If we just ramp up to the max, we'll only be allowed to
+ * submit one io_combine_limit sized IO. Defer submitting IO in that case.
+ *
+ * FIXME: This needs better heuristics.
+ */
+#if 1
+ if (!stream->sync_mode && stream->distance > (io_combine_limit * 8))
+ {
+ if (stream->pinned_buffers + stream->pending_read_nblocks > ((stream->distance * 3) / 4))
+ {
+#if 0
+ ereport(LOG,
+ errmsg("reduce reduce reduce: pinned: %d, pending: %d, distance: %d",
+ stream->pinned_buffers,
+ stream->pending_read_nblocks,
+ stream->distance));
+#endif
+ return;
+ }
+ }
+#endif
+
/*
* Allow amortizing the cost of submitting IO over multiple IOs. This
* requires that we don't do any operations that could lead to a deadlock
--
2.48.1.76.g4e746b1a31.dirty
[text/x-diff] v2.14-0020-aio-Implement-smgr-md-fd-write-support.patch (14.6K, ../../d4ff3qxudxn2ze3zk2xwhds5ysezfei3uqeuimymj624t5chsd@sgr2h3rmln5t/21-v2.14-0020-aio-Implement-smgr-md-fd-write-support.patch)
download | inline diff:
From c4f3fe9002ec977f172f9413073353557d9f45f8 Mon Sep 17 00:00:00 2001
From: Andres Freund <andres@anarazel.de>
Date: Tue, 18 Mar 2025 14:40:06 -0400
Subject: [PATCH v2.14 20/29] aio: Implement smgr/md/fd write support
TODO:
- Right now the sync.c integration with smgr.c/md.c isn't properly safe to use
in a critical section
The only reason it doesn't immediately fail is that it's reasonably rare
that RegisterSyncRequest() fails *and* either:
- smgropen()->hash_search(HASH_ENTER) decides to resize the hash table, even
though the lookup is guaranteed to succeed for io_method=worker.
- an io_method=uring completion is run in a different backend and smgropen()
needs to build a new entry and thus needs to allocate memory
For a bit I thought this could be worked around easily enough by not doing
an smgropen() in mdsyncfiletag(), or adding a "fallible" smgropen() and
instead just opening the file directly. That actually does kinda solve the
problem, but only because the memory allocation in PathNameOpenFile()
uses malloc(), not palloc() and thus doesn't trigger
- temp_file_limit implementation
---
src/include/storage/fd.h | 1 +
src/include/storage/md.h | 5 +
src/include/storage/smgr.h | 5 +
src/backend/storage/aio/aio_callback.c | 1 +
src/backend/storage/file/fd.c | 28 ++++
src/backend/storage/smgr/md.c | 201 ++++++++++++++++++++++++-
src/backend/storage/smgr/smgr.c | 29 ++++
7 files changed, 269 insertions(+), 1 deletion(-)
diff --git a/src/include/storage/fd.h b/src/include/storage/fd.h
index b77d8e5e30e..2cc7c5a4761 100644
--- a/src/include/storage/fd.h
+++ b/src/include/storage/fd.h
@@ -112,6 +112,7 @@ extern int FilePrefetch(File file, off_t offset, off_t amount, uint32 wait_event
extern ssize_t FileReadV(File file, const struct iovec *iov, int iovcnt, off_t offset, uint32 wait_event_info);
extern ssize_t FileWriteV(File file, const struct iovec *iov, int iovcnt, off_t offset, uint32 wait_event_info);
extern int FileStartReadV(struct PgAioHandle *ioh, File file, int iovcnt, off_t offset, uint32 wait_event_info);
+extern int FileStartWriteV(struct PgAioHandle *ioh, File file, int iovcnt, off_t offset, uint32 wait_event_info);
extern int FileSync(File file, uint32 wait_event_info);
extern int FileZero(File file, off_t offset, off_t amount, uint32 wait_event_info);
extern int FileFallocate(File file, off_t offset, off_t amount, uint32 wait_event_info);
diff --git a/src/include/storage/md.h b/src/include/storage/md.h
index 9d7131eff43..47ae6c36c94 100644
--- a/src/include/storage/md.h
+++ b/src/include/storage/md.h
@@ -21,6 +21,7 @@
#include "storage/sync.h"
extern const PgAioHandleCallbacks aio_md_readv_cb;
+extern const PgAioHandleCallbacks aio_md_writev_cb;
/* md storage manager functionality */
extern void mdinit(void);
@@ -45,6 +46,10 @@ extern void mdstartreadv(PgAioHandle *ioh,
extern void mdwritev(SMgrRelation reln, ForkNumber forknum,
BlockNumber blocknum,
const void **buffers, BlockNumber nblocks, bool skipFsync);
+extern void mdstartwritev(PgAioHandle *ioh,
+ SMgrRelation reln, ForkNumber forknum,
+ BlockNumber blocknum,
+ const void **buffers, BlockNumber nblocks, bool skipFsync);
extern void mdwriteback(SMgrRelation reln, ForkNumber forknum,
BlockNumber blocknum, BlockNumber nblocks);
extern BlockNumber mdnblocks(SMgrRelation reln, ForkNumber forknum);
diff --git a/src/include/storage/smgr.h b/src/include/storage/smgr.h
index 856ebcda350..f00b3763ac9 100644
--- a/src/include/storage/smgr.h
+++ b/src/include/storage/smgr.h
@@ -108,6 +108,11 @@ extern void smgrwritev(SMgrRelation reln, ForkNumber forknum,
BlockNumber blocknum,
const void **buffers, BlockNumber nblocks,
bool skipFsync);
+extern void smgrstartwritev(PgAioHandle *ioh,
+ SMgrRelation reln, ForkNumber forknum,
+ BlockNumber blocknum,
+ const void **buffers, BlockNumber nblocks,
+ bool skipFsync);
extern void smgrwriteback(SMgrRelation reln, ForkNumber forknum,
BlockNumber blocknum, BlockNumber nblocks);
extern BlockNumber smgrnblocks(SMgrRelation reln, ForkNumber forknum);
diff --git a/src/backend/storage/aio/aio_callback.c b/src/backend/storage/aio/aio_callback.c
index c0063d6950e..67026f5f6b4 100644
--- a/src/backend/storage/aio/aio_callback.c
+++ b/src/backend/storage/aio/aio_callback.c
@@ -41,6 +41,7 @@ static const PgAioHandleCallbacksEntry aio_handle_cbs[] = {
CALLBACK_ENTRY(PGAIO_HCB_INVALID, aio_invalid_cb),
CALLBACK_ENTRY(PGAIO_HCB_MD_READV, aio_md_readv_cb),
+ CALLBACK_ENTRY(PGAIO_HCB_MD_WRITEV, aio_md_writev_cb),
CALLBACK_ENTRY(PGAIO_HCB_SHARED_BUFFER_READV, aio_shared_buffer_readv_cb),
diff --git a/src/backend/storage/file/fd.c b/src/backend/storage/file/fd.c
index 0e8299dd556..2138d47dab9 100644
--- a/src/backend/storage/file/fd.c
+++ b/src/backend/storage/file/fd.c
@@ -2348,6 +2348,34 @@ retry:
return returnCode;
}
+int
+FileStartWriteV(PgAioHandle *ioh, File file,
+ int iovcnt, off_t offset,
+ uint32 wait_event_info)
+{
+ int returnCode;
+ Vfd *vfdP;
+
+ Assert(FileIsValid(file));
+
+ DO_DB(elog(LOG, "FileStartWriteV: %d (%s) " INT64_FORMAT " %d",
+ file, VfdCache[file].fileName,
+ (int64) offset,
+ iovcnt));
+
+ returnCode = FileAccess(file);
+ if (returnCode < 0)
+ return returnCode;
+
+ vfdP = &VfdCache[file];
+
+ /* FIXME: think about / reimplement temp_file_limit */
+
+ pgaio_io_start_writev(ioh, vfdP->fd, iovcnt, offset);
+
+ return 0;
+}
+
int
FileSync(File file, uint32 wait_event_info)
{
diff --git a/src/backend/storage/smgr/md.c b/src/backend/storage/smgr/md.c
index eedba1e5794..f6bdc7e37bc 100644
--- a/src/backend/storage/smgr/md.c
+++ b/src/backend/storage/smgr/md.c
@@ -155,12 +155,19 @@ static BlockNumber _mdnblocks(SMgrRelation reln, ForkNumber forknum,
static PgAioResult md_readv_complete(PgAioHandle *ioh, PgAioResult prior_result, uint8 cb_data);
static void md_readv_report(PgAioResult result, const PgAioTargetData *target_data, int elevel);
+static PgAioResult md_writev_complete(PgAioHandle *ioh, PgAioResult prior_result, uint8 cb_data);
+static void md_writev_report(PgAioResult result, const PgAioTargetData *target_data, int elevel);
const PgAioHandleCallbacks aio_md_readv_cb = {
.complete_shared = md_readv_complete,
.report = md_readv_report,
};
+const PgAioHandleCallbacks aio_md_writev_cb = {
+ .complete_shared = md_writev_complete,
+ .report = md_writev_report,
+};
+
static inline int
_mdfd_open_flags(void)
@@ -1115,6 +1122,64 @@ mdwritev(SMgrRelation reln, ForkNumber forknum, BlockNumber blocknum,
}
}
+/*
+ * mdstartwritev() -- Asynchronous version of mdrwritev().
+ */
+void
+mdstartwritev(PgAioHandle *ioh,
+ SMgrRelation reln, ForkNumber forknum, BlockNumber blocknum,
+ const void **buffers, BlockNumber nblocks, bool skipFsync)
+{
+ off_t seekpos;
+ MdfdVec *v;
+ BlockNumber nblocks_this_segment;
+ struct iovec *iov;
+ int iovcnt;
+ int ret;
+
+ v = _mdfd_getseg(reln, forknum, blocknum, false,
+ EXTENSION_FAIL | EXTENSION_CREATE_RECOVERY);
+
+ seekpos = (off_t) BLCKSZ * (blocknum % ((BlockNumber) RELSEG_SIZE));
+
+ Assert(seekpos < (off_t) BLCKSZ * RELSEG_SIZE);
+
+ nblocks_this_segment =
+ Min(nblocks,
+ RELSEG_SIZE - (blocknum % ((BlockNumber) RELSEG_SIZE)));
+
+ if (nblocks_this_segment != nblocks)
+ elog(ERROR, "write crossing segment boundary");
+
+ iovcnt = pgaio_io_get_iovec(ioh, &iov);
+
+ Assert(nblocks <= iovcnt);
+
+ iovcnt = buffers_to_iovec(iov, unconstify(void **, buffers), nblocks_this_segment);
+
+ Assert(iovcnt <= nblocks_this_segment);
+
+ if (!(io_direct_flags & IO_DIRECT_DATA))
+ pgaio_io_set_flag(ioh, PGAIO_HF_BUFFERED);
+
+ pgaio_io_set_target_smgr(ioh,
+ reln,
+ forknum,
+ blocknum,
+ nblocks,
+ skipFsync);
+ pgaio_io_register_callbacks(ioh, PGAIO_HCB_MD_WRITEV, 0);
+
+ ret = FileStartWriteV(ioh, v->mdfd_vfd, iovcnt, seekpos, WAIT_EVENT_DATA_FILE_WRITE);
+ if (ret != 0)
+ ereport(ERROR,
+ (errcode_for_file_access(),
+ errmsg("could not start writing blocks %u..%u in file \"%s\": %m",
+ blocknum,
+ blocknum + nblocks_this_segment - 1,
+ FilePathName(v->mdfd_vfd))));
+}
+
/*
* mdwriteback() -- Tell the kernel to write pages back to storage.
@@ -1503,6 +1568,40 @@ register_dirty_segment(SMgrRelation reln, ForkNumber forknum, MdfdVec *seg)
}
}
+/*
+ * Like register_dirty_segment(), except for use by AIO. In the completion
+ * callback we don't have access to the MdfdVec (the completion callback might
+ * be executed in a different backend than the issuing backend), therefore we
+ * have to implement this slightly differently.
+ */
+static void
+register_dirty_segment_aio(RelFileLocator locator, ForkNumber forknum, uint64 segno)
+{
+ FileTag tag;
+
+ INIT_MD_FILETAG(tag, locator, forknum, segno);
+
+ /*
+ * Can't block here waiting for checkpointer to accept our sync request,
+ * as checkpointer might be waiting for this AIO to finish if offloaded to
+ * a worker.
+ */
+ if (!RegisterSyncRequest(&tag, SYNC_REQUEST, false /* retryOnError */ ))
+ {
+ char path[MAXPGPATH];
+
+ ereport(DEBUG1,
+ (errmsg_internal("could not forward fsync request because request queue is full")));
+
+ /* reuse mdsyncfiletag() to avoid duplicating code */
+ if (mdsyncfiletag(&tag, path))
+ ereport(data_sync_elevel(ERROR),
+ (errcode_for_file_access(),
+ errmsg("could not fsync file \"%s\": %m",
+ path)));
+ }
+}
+
/*
* register_unlink_segment() -- Schedule a file to be deleted after next checkpoint
*/
@@ -1997,7 +2096,7 @@ md_readv_complete(PgAioHandle *ioh, PgAioResult prior_result, uint8 cb_data)
* AIO error reporting callback for mdstartreadv().
*
* Errors are encoded as follows:
- * - PgAioResult.error_data != 0 encodes IO that failed with errno != 0
+ * - PgAioResult.error_data != 0 encodes IO that failed with that errno
* - PgAioResult.error_data == 0 encodes IO that didn't read all data
*/
static void
@@ -2037,3 +2136,103 @@ md_readv_report(PgAioResult result, const PgAioTargetData *td, int elevel)
td->smgr.nblocks * (size_t) BLCKSZ));
}
}
+
+/*
+ * AIO completion callback for mdstartwritev().
+ */
+static PgAioResult
+md_writev_complete(PgAioHandle *ioh, PgAioResult prior_result, uint8 cb_data)
+{
+ PgAioTargetData *td = pgaio_io_get_target_data(ioh);
+ PgAioResult result = prior_result;
+
+ if (prior_result.result < 0)
+ {
+ result.status = PGAIO_RS_ERROR;
+ result.id = PGAIO_HCB_MD_WRITEV;
+ /* For "hard" errors, track the error number in error_data */
+ result.error_data = -prior_result.result;
+ result.result = 0;
+
+ pgaio_result_report(result, td, LOG);
+
+ return result;
+ }
+
+ /*
+ * As explained above smgrstartwritev(), the smgr API operates on the
+ * level of blocks, rather than bytes. Convert.
+ */
+ result.result /= BLCKSZ;
+
+ Assert(result.result <= td->smgr.nblocks);
+
+ if (result.result == 0)
+ {
+ /* consider 0 blocks written a failure */
+ result.status = PGAIO_RS_ERROR;
+ result.id = PGAIO_HCB_MD_WRITEV;
+ result.error_data = 0;
+
+ pgaio_result_report(result, td, LOG);
+
+ return result;
+ }
+
+ if (result.status != PGAIO_RS_ERROR &&
+ result.result < td->smgr.nblocks)
+ {
+ /* partial writes should be retried at upper level */
+ result.status = PGAIO_RS_PARTIAL;
+ result.id = PGAIO_HCB_MD_WRITEV;
+ }
+
+ if (!td->smgr.skip_fsync)
+ register_dirty_segment_aio(td->smgr.rlocator, td->smgr.forkNum,
+ td->smgr.blockNum / ((BlockNumber) RELSEG_SIZE));
+
+ return result;
+}
+
+/*
+ * AIO error reporting callback for mdstartwritev().
+ */
+static void
+md_writev_report(PgAioResult result, const PgAioTargetData *td, int elevel)
+{
+ RelPathStr path;
+
+ path = relpathbackend(td->smgr.rlocator,
+ td->smgr.is_temp ? MyProcNumber : INVALID_PROC_NUMBER,
+ td->smgr.forkNum);
+
+ if (result.error_data != 0)
+ {
+ errno = result.error_data; /* for errcode_for_file_access() */
+
+ ereport(elevel,
+ errcode_for_file_access(),
+ errmsg("could not write blocks %u..%u in file \"%s\": %m",
+ td->smgr.blockNum,
+ td->smgr.blockNum + td->smgr.nblocks,
+ path.str)
+ );
+ }
+ else
+ {
+ /*
+ * NB: This will typically only be output in debug messages, while
+ * retrying a partial IO.
+ */
+ ereport(elevel,
+ errcode(ERRCODE_DATA_CORRUPTED),
+ errmsg("could not write blocks %u..%u in file \"%s\": wrote only %zu of %zu bytes",
+ td->smgr.blockNum,
+ td->smgr.blockNum + td->smgr.nblocks - 1,
+ path.str,
+ result.result * (size_t) BLCKSZ,
+ td->smgr.nblocks * (size_t) BLCKSZ
+ )
+ );
+ }
+}
diff --git a/src/backend/storage/smgr/smgr.c b/src/backend/storage/smgr/smgr.c
index bf8f57b410a..e56d6a2597b 100644
--- a/src/backend/storage/smgr/smgr.c
+++ b/src/backend/storage/smgr/smgr.c
@@ -115,6 +115,11 @@ typedef struct f_smgr
BlockNumber blocknum,
const void **buffers, BlockNumber nblocks,
bool skipFsync);
+ void (*smgr_startwritev) (PgAioHandle *ioh,
+ SMgrRelation reln, ForkNumber forknum,
+ BlockNumber blocknum,
+ const void **buffers, BlockNumber nblocks,
+ bool skipFsync);
void (*smgr_writeback) (SMgrRelation reln, ForkNumber forknum,
BlockNumber blocknum, BlockNumber nblocks);
BlockNumber (*smgr_nblocks) (SMgrRelation reln, ForkNumber forknum);
@@ -142,6 +147,7 @@ static const f_smgr smgrsw[] = {
.smgr_readv = mdreadv,
.smgr_startreadv = mdstartreadv,
.smgr_writev = mdwritev,
+ .smgr_startwritev = mdstartwritev,
.smgr_writeback = mdwriteback,
.smgr_nblocks = mdnblocks,
.smgr_truncate = mdtruncate,
@@ -787,6 +793,29 @@ smgrwritev(SMgrRelation reln, ForkNumber forknum, BlockNumber blocknum,
RESUME_INTERRUPTS();
}
+/*
+ * smgrstartwritev() -- asynchronous version of smgrwritev()
+ *
+ * This starts an asynchronous writev IO using the IO handle `ioh`. Other than
+ * `ioh` all parameters are the same as smgrwritev().
+ *
+ * Completion callbacks above smgr will be passed the result as the number of
+ * successfully written blocks if the write [partially] succeeds. This
+ * maintains the abstraction that smgr operates on the level of blocks, rather
+ * than bytes.
+ */
+void
+smgrstartwritev(PgAioHandle *ioh,
+ SMgrRelation reln, ForkNumber forknum, BlockNumber blocknum,
+ const void **buffers, BlockNumber nblocks, bool skipFsync)
+{
+ HOLD_INTERRUPTS();
+ smgrsw[reln->smgr_which].smgr_startwritev(ioh,
+ reln, forknum, blocknum, buffers,
+ nblocks, skipFsync);
+ RESUME_INTERRUPTS();
+}
+
/*
* smgrwriteback() -- Trigger kernel writeback for the supplied range of
* blocks.
--
2.48.1.76.g4e746b1a31.dirty
[text/x-diff] v2.14-0021-aio-Add-bounce-buffers.patch (23.2K, ../../d4ff3qxudxn2ze3zk2xwhds5ysezfei3uqeuimymj624t5chsd@sgr2h3rmln5t/22-v2.14-0021-aio-Add-bounce-buffers.patch)
download | inline diff:
From f2e20aee8e448cd85cef32b3011084768c11fa74 Mon Sep 17 00:00:00 2001
From: Andres Freund <andres@anarazel.de>
Date: Tue, 18 Mar 2025 14:40:06 -0400
Subject: [PATCH v2.14 21/29] aio: Add bounce buffers
---
src/include/storage/aio.h | 15 ++
src/include/storage/aio_internal.h | 34 ++++
src/include/storage/aio_types.h | 2 +
src/include/utils/resowner.h | 2 +
src/backend/storage/aio/README.md | 27 +++
src/backend/storage/aio/aio.c | 178 ++++++++++++++++++
src/backend/storage/aio/aio_init.c | 123 ++++++++++++
src/backend/utils/misc/guc_tables.c | 13 ++
src/backend/utils/misc/postgresql.conf.sample | 2 +
src/backend/utils/resowner/resowner.c | 25 ++-
src/test/modules/test_aio/test_aio--1.0.sql | 21 +++
src/test/modules/test_aio/test_aio.c | 55 ++++++
src/tools/pgindent/typedefs.list | 1 +
13 files changed, 496 insertions(+), 2 deletions(-)
diff --git a/src/include/storage/aio.h b/src/include/storage/aio.h
index bfe0d93683b..f91f0afc5a5 100644
--- a/src/include/storage/aio.h
+++ b/src/include/storage/aio.h
@@ -353,6 +353,20 @@ extern bool pgaio_have_staged(void);
+/* --------------------------------------------------------------------------------
+ * Bounce Buffers
+ * --------------------------------------------------------------------------------
+ */
+
+extern PgAioBounceBuffer *pgaio_bounce_buffer_get(void);
+extern void pgaio_io_assoc_bounce_buffer(PgAioHandle *ioh, PgAioBounceBuffer *bb);
+extern uint32 pgaio_bounce_buffer_id(PgAioBounceBuffer *bb);
+extern void pgaio_bounce_buffer_release(PgAioBounceBuffer *bb);
+extern char *pgaio_bounce_buffer_buffer(PgAioBounceBuffer *bb);
+extern void pgaio_bounce_buffer_release_resowner(struct dlist_node *bb_node, bool on_error);
+
+
+
/* --------------------------------------------------------------------------------
* Other
* --------------------------------------------------------------------------------
@@ -365,6 +379,7 @@ extern void pgaio_closing_fd(int fd);
/* GUCs */
extern PGDLLIMPORT int io_method;
extern PGDLLIMPORT int io_max_concurrency;
+extern PGDLLIMPORT int io_bounce_buffers;
#endif /* AIO_H */
diff --git a/src/include/storage/aio_internal.h b/src/include/storage/aio_internal.h
index 7f18da2c856..833f97361a1 100644
--- a/src/include/storage/aio_internal.h
+++ b/src/include/storage/aio_internal.h
@@ -127,6 +127,12 @@ struct PgAioHandle
/* raw result of the IO operation */
int32 result;
+ /*
+ * List of bounce_buffers owned by IO. It would suffice to use an index
+ * based linked list here.
+ */
+ slist_head bounce_buffers;
+
/**
* In which list the handle is registered, depends on the state:
* - IDLE, in per-backend list
@@ -182,11 +188,24 @@ struct PgAioHandle
};
+/* typedef is in aio_types.h */
+struct PgAioBounceBuffer
+{
+ slist_node node;
+ struct ResourceOwnerData *resowner;
+ dlist_node resowner_node;
+ char *buffer;
+};
+
+
typedef struct PgAioBackend
{
/* index into PgAioCtl->io_handles */
uint32 io_handle_off;
+ /* index into PgAioCtl->bounce_buffers */
+ uint32 bounce_buffers_off;
+
/* IO Handles that currently are not used */
dclist_head idle_ios;
@@ -217,6 +236,12 @@ typedef struct PgAioBackend
* IOs being appended at the end.
*/
dclist_head in_flight_ios;
+
+ /* Bounce Buffers that currently are not used */
+ slist_head idle_bbs;
+
+ /* see handed_out_io */
+ PgAioBounceBuffer *handed_out_bb;
} PgAioBackend;
@@ -244,6 +269,15 @@ typedef struct PgAioCtl
uint32 io_handle_count;
PgAioHandle *io_handles;
+
+ /*
+ * To perform AIO on buffers that are not located in shared memory (either
+ * because they are not in shared memory or because we need to operate on
+ * a copy, as e.g. the case for writes when checksums are in use)
+ */
+ uint32 bounce_buffers_count;
+ PgAioBounceBuffer *bounce_buffers;
+ char *bounce_buffers_data;
} PgAioCtl;
diff --git a/src/include/storage/aio_types.h b/src/include/storage/aio_types.h
index 156132cde03..b23ed0ec47d 100644
--- a/src/include/storage/aio_types.h
+++ b/src/include/storage/aio_types.h
@@ -134,4 +134,6 @@ typedef struct PgAioReturn
} PgAioReturn;
+typedef struct PgAioBounceBuffer PgAioBounceBuffer;
+
#endif /* AIO_TYPES_H */
diff --git a/src/include/utils/resowner.h b/src/include/utils/resowner.h
index aede4bfc820..7e2ec224169 100644
--- a/src/include/utils/resowner.h
+++ b/src/include/utils/resowner.h
@@ -168,5 +168,7 @@ extern void ResourceOwnerForgetLock(ResourceOwner owner, struct LOCALLOCK *local
struct dlist_node;
extern void ResourceOwnerRememberAioHandle(ResourceOwner owner, struct dlist_node *ioh_node);
extern void ResourceOwnerForgetAioHandle(ResourceOwner owner, struct dlist_node *ioh_node);
+extern void ResourceOwnerRememberAioBounceBuffer(ResourceOwner owner, struct dlist_node *bb_node);
+extern void ResourceOwnerForgetAioBounceBuffer(ResourceOwner owner, struct dlist_node *bb_node);
#endif /* RESOWNER_H */
diff --git a/src/backend/storage/aio/README.md b/src/backend/storage/aio/README.md
index ddd59404a59..534aaad22bc 100644
--- a/src/backend/storage/aio/README.md
+++ b/src/backend/storage/aio/README.md
@@ -406,6 +406,33 @@ shared memory no less!), completion callbacks instead have to encode errors in
a more compact format that can be converted into an error message.
+### AIO Bounce Buffers
+
+For some uses of AIO there is no convenient memory location as the source /
+destination of an AIO. E.g. when data checksums are enabled, writes from
+shared buffers currently cannot be done directly from shared buffers, as a
+shared buffer lock still allows some modification, e.g., for hint bits (see
+`FlushBuffer()`). If the write were done in-place, such modifications can
+cause the checksum to fail.
+
+For synchronous IO this is solved by copying the buffer to separate memory
+before computing the checksum and using that copy as the source buffer for the
+AIO.
+
+However, for AIO that is not a workable solution:
+- Instead of a single buffer many buffers are required, as many IOs might be
+ in flight
+- When using the [worker method](#worker), the source/target of IO needs to be
+ in shared memory, otherwise the workers won't be able to access the memory.
+
+The AIO subsystem addresses this by providing a limited number of bounce
+buffers that can be used as the source / target for IO. A bounce buffer can be
+acquired with `pgaio_bounce_buffer_get()` and multiple bounce buffers can be
+associated with an AIO Handle with `pgaio_io_assoc_bounce_buffer()`.
+
+Bounce buffers are automatically released when the IO completes.
+
+
## Helpers
Using the low-level AIO API introduces too much complexity to do so all over
diff --git a/src/backend/storage/aio/aio.c b/src/backend/storage/aio/aio.c
index 86f7250b7a5..cff48964d07 100644
--- a/src/backend/storage/aio/aio.c
+++ b/src/backend/storage/aio/aio.c
@@ -62,6 +62,8 @@ static PgAioHandle *pgaio_io_from_wref(PgAioWaitRef *iow, uint64 *ref_generation
static const char *pgaio_io_state_get_name(PgAioHandleState s);
static void pgaio_io_wait(PgAioHandle *ioh, uint64 ref_generation);
+static void pgaio_bounce_buffer_wait_for_free(void);
+
/* Options for io_method. */
const struct config_enum_entry io_method_options[] = {
@@ -76,6 +78,7 @@ const struct config_enum_entry io_method_options[] = {
/* GUCs */
int io_method = DEFAULT_IO_METHOD;
int io_max_concurrency = -1;
+int io_bounce_buffers = -1;
/* global control for AIO */
PgAioCtl *pgaio_ctl;
@@ -662,6 +665,21 @@ pgaio_io_reclaim(PgAioHandle *ioh)
if (ioh->state != PGAIO_HS_HANDED_OUT)
dclist_delete_from(&pgaio_my_backend->in_flight_ios, &ioh->node);
+ /* reclaim all associated bounce buffers */
+ if (!slist_is_empty(&ioh->bounce_buffers))
+ {
+ slist_mutable_iter it;
+
+ slist_foreach_modify(it, &ioh->bounce_buffers)
+ {
+ PgAioBounceBuffer *bb = slist_container(PgAioBounceBuffer, node, it.cur);
+
+ slist_delete_current(&it);
+
+ slist_push_head(&pgaio_my_backend->idle_bbs, &bb->node);
+ }
+ }
+
if (ioh->resowner)
{
ResourceOwnerForgetAioHandle(ioh->resowner, &ioh->resowner_node);
@@ -1046,6 +1064,166 @@ pgaio_submit_staged(void)
+/* --------------------------------------------------------------------------------
+ * Functions primarily related to PgAioBounceBuffer
+ * --------------------------------------------------------------------------------
+ */
+
+PgAioBounceBuffer *
+pgaio_bounce_buffer_get(void)
+{
+ PgAioBounceBuffer *bb = NULL;
+ slist_node *node;
+
+ if (pgaio_my_backend->handed_out_bb != NULL)
+ elog(ERROR, "can only hand out one BB");
+
+ /*
+ * XXX: It probably is not a good idea to have bounce buffers be per
+ * backend, that's a fair bit of memory.
+ */
+ if (slist_is_empty(&pgaio_my_backend->idle_bbs))
+ {
+ pgaio_bounce_buffer_wait_for_free();
+ }
+
+ node = slist_pop_head_node(&pgaio_my_backend->idle_bbs);
+ bb = slist_container(PgAioBounceBuffer, node, node);
+
+ pgaio_my_backend->handed_out_bb = bb;
+
+ bb->resowner = CurrentResourceOwner;
+ ResourceOwnerRememberAioBounceBuffer(bb->resowner, &bb->resowner_node);
+
+ return bb;
+}
+
+void
+pgaio_io_assoc_bounce_buffer(PgAioHandle *ioh, PgAioBounceBuffer *bb)
+{
+ if (pgaio_my_backend->handed_out_bb != bb)
+ elog(ERROR, "can only assign handed out BB");
+ pgaio_my_backend->handed_out_bb = NULL;
+
+ /*
+ * There can be many bounce buffers assigned in case of vectorized IOs.
+ */
+ slist_push_head(&ioh->bounce_buffers, &bb->node);
+
+ /* once associated with an IO, the IO has ownership */
+ ResourceOwnerForgetAioBounceBuffer(bb->resowner, &bb->resowner_node);
+ bb->resowner = NULL;
+}
+
+uint32
+pgaio_bounce_buffer_id(PgAioBounceBuffer *bb)
+{
+ return bb - pgaio_ctl->bounce_buffers;
+}
+
+void
+pgaio_bounce_buffer_release(PgAioBounceBuffer *bb)
+{
+ if (pgaio_my_backend->handed_out_bb != bb)
+ elog(ERROR, "can only release handed out BB");
+
+ slist_push_head(&pgaio_my_backend->idle_bbs, &bb->node);
+ pgaio_my_backend->handed_out_bb = NULL;
+
+ ResourceOwnerForgetAioBounceBuffer(bb->resowner, &bb->resowner_node);
+ bb->resowner = NULL;
+}
+
+void
+pgaio_bounce_buffer_release_resowner(dlist_node *bb_node, bool on_error)
+{
+ PgAioBounceBuffer *bb = dlist_container(PgAioBounceBuffer, resowner_node, bb_node);
+
+ Assert(bb->resowner);
+
+ if (!on_error)
+ elog(WARNING, "leaked AIO bounce buffer");
+
+ pgaio_bounce_buffer_release(bb);
+}
+
+char *
+pgaio_bounce_buffer_buffer(PgAioBounceBuffer *bb)
+{
+ return bb->buffer;
+}
+
+static void
+pgaio_bounce_buffer_wait_for_free(void)
+{
+ static uint32 lastpos = 0;
+
+ if (pgaio_my_backend->num_staged_ios > 0)
+ {
+ pgaio_debug(DEBUG2, "submitting %d, while acquiring free bb",
+ pgaio_my_backend->num_staged_ios);
+ pgaio_submit_staged();
+ }
+
+ for (uint32 i = lastpos; i < lastpos + io_max_concurrency; i++)
+ {
+ uint32 thisoff = pgaio_my_backend->io_handle_off + (i % io_max_concurrency);
+ PgAioHandle *ioh = &pgaio_ctl->io_handles[thisoff];
+
+ switch (ioh->state)
+ {
+ case PGAIO_HS_IDLE:
+ case PGAIO_HS_HANDED_OUT:
+ continue;
+ case PGAIO_HS_DEFINED: /* should have been submitted above */
+ case PGAIO_HS_STAGED:
+ elog(ERROR, "shouldn't get here with io:%d in state %d",
+ pgaio_io_get_id(ioh), ioh->state);
+ break;
+ case PGAIO_HS_COMPLETED_IO:
+ case PGAIO_HS_SUBMITTED:
+ if (!slist_is_empty(&ioh->bounce_buffers))
+ {
+ pgaio_debug_io(DEBUG2, ioh,
+ "waiting for IO to reclaim BB with %d in flight",
+ dclist_count(&pgaio_my_backend->in_flight_ios));
+
+ /* see comment in pgaio_io_wait_for_free() about raciness */
+ pgaio_io_wait(ioh, ioh->generation);
+
+ if (slist_is_empty(&pgaio_my_backend->idle_bbs))
+ elog(WARNING, "empty after wait");
+
+ if (!slist_is_empty(&pgaio_my_backend->idle_bbs))
+ {
+ lastpos = i;
+ return;
+ }
+ }
+ break;
+ case PGAIO_HS_COMPLETED_SHARED:
+ case PGAIO_HS_COMPLETED_LOCAL:
+ /* reclaim */
+ pgaio_io_reclaim(ioh);
+
+ if (!slist_is_empty(&pgaio_my_backend->idle_bbs))
+ {
+ lastpos = i;
+ return;
+ }
+ break;
+ }
+ }
+
+ /*
+ * The submission above could have caused the IO to complete at any time.
+ */
+ if (slist_is_empty(&pgaio_my_backend->idle_bbs))
+ elog(PANIC, "no more bbs");
+}
+
+
+
/* --------------------------------------------------------------------------------
* Other
* --------------------------------------------------------------------------------
diff --git a/src/backend/storage/aio/aio_init.c b/src/backend/storage/aio/aio_init.c
index 885c3940c66..95b10933fed 100644
--- a/src/backend/storage/aio/aio_init.c
+++ b/src/backend/storage/aio/aio_init.c
@@ -88,6 +88,32 @@ AioHandleDataShmemSize(void)
io_max_concurrency));
}
+static Size
+AioBounceBufferDescShmemSize(void)
+{
+ Size sz;
+
+ /* PgAioBounceBuffer itself */
+ sz = mul_size(sizeof(PgAioBounceBuffer),
+ mul_size(AioProcs(), io_bounce_buffers));
+
+ return sz;
+}
+
+static Size
+AioBounceBufferDataShmemSize(void)
+{
+ Size sz;
+
+ /* and the associated buffer */
+ sz = mul_size(BLCKSZ,
+ mul_size(io_bounce_buffers, AioProcs()));
+ /* memory for alignment */
+ sz += BLCKSZ;
+
+ return sz;
+}
+
/*
* Choose a suitable value for io_max_concurrency.
*
@@ -113,6 +139,33 @@ AioChooseMaxConcurrency(void)
return Min(max_proportional_pins, 64);
}
+/*
+ * Choose a suitable value for io_bounce_buffers.
+ *
+ * It's very unlikely to be useful to allocate more bounce buffers for each
+ * backend than the backend is allowed to pin. Additionally, bounce buffers
+ * currently are used for writes, it seems very uncommon for more than 10% of
+ * shared_buffers to be written out concurrently.
+ *
+ * XXX: This quickly can take up significant amounts of memory, the logic
+ * should probably fine tuned.
+ */
+static int
+AioChooseBounceBuffers(void)
+{
+ uint32 max_backends;
+ int max_proportional_pins;
+
+ /* Similar logic to LimitAdditionalPins() */
+ max_backends = MaxBackends + NUM_AUXILIARY_PROCS;
+ max_proportional_pins = (NBuffers / 10) / max_backends;
+
+ max_proportional_pins = Max(max_proportional_pins, 1);
+
+ /* apply upper limit */
+ return Min(max_proportional_pins, 256);
+}
+
Size
AioShmemSize(void)
{
@@ -136,11 +189,31 @@ AioShmemSize(void)
PGC_S_OVERRIDE);
}
+
+ /*
+ * If io_bounce_buffers is -1, we automatically choose a suitable value.
+ *
+ * See also comment above.
+ */
+ if (io_bounce_buffers == -1)
+ {
+ char buf[32];
+
+ snprintf(buf, sizeof(buf), "%d", AioChooseBounceBuffers());
+ SetConfigOption("io_bounce_buffers", buf, PGC_POSTMASTER,
+ PGC_S_DYNAMIC_DEFAULT);
+ if (io_bounce_buffers == -1) /* failed to apply it? */
+ SetConfigOption("io_bounce_buffers", buf, PGC_POSTMASTER,
+ PGC_S_OVERRIDE);
+ }
+
sz = add_size(sz, AioCtlShmemSize());
sz = add_size(sz, AioBackendShmemSize());
sz = add_size(sz, AioHandleShmemSize());
sz = add_size(sz, AioHandleIOVShmemSize());
sz = add_size(sz, AioHandleDataShmemSize());
+ sz = add_size(sz, AioBounceBufferDescShmemSize());
+ sz = add_size(sz, AioBounceBufferDataShmemSize());
/* Reserve space for method specific resources. */
if (pgaio_method_ops->shmem_size)
@@ -156,6 +229,9 @@ AioShmemInit(void)
uint32 io_handle_off = 0;
uint32 iovec_off = 0;
uint32 per_backend_iovecs = io_max_concurrency * io_max_combine_limit;
+ uint32 bounce_buffers_off = 0;
+ uint32 per_backend_bb = io_bounce_buffers;
+ char *bounce_buffers_data;
pgaio_ctl = (PgAioCtl *)
ShmemInitStruct("AioCtl", AioCtlShmemSize(), &found);
@@ -167,6 +243,7 @@ AioShmemInit(void)
pgaio_ctl->io_handle_count = AioProcs() * io_max_concurrency;
pgaio_ctl->iovec_count = AioProcs() * per_backend_iovecs;
+ pgaio_ctl->bounce_buffers_count = AioProcs() * per_backend_bb;
pgaio_ctl->backend_state = (PgAioBackend *)
ShmemInitStruct("AioBackend", AioBackendShmemSize(), &found);
@@ -179,6 +256,40 @@ AioShmemInit(void)
pgaio_ctl->handle_data = (uint64 *)
ShmemInitStruct("AioHandleData", AioHandleDataShmemSize(), &found);
+ pgaio_ctl->bounce_buffers = (PgAioBounceBuffer *)
+ ShmemInitStruct("AioBounceBufferDesc", AioBounceBufferDescShmemSize(),
+ &found);
+
+ bounce_buffers_data =
+ ShmemInitStruct("AioBounceBufferData", AioBounceBufferDataShmemSize(),
+ &found);
+ bounce_buffers_data =
+ (char *) TYPEALIGN(BLCKSZ, (uintptr_t) bounce_buffers_data);
+ pgaio_ctl->bounce_buffers_data = bounce_buffers_data;
+
+
+ /* Initialize IO handles. */
+ for (uint64 i = 0; i < pgaio_ctl->io_handle_count; i++)
+ {
+ PgAioHandle *ioh = &pgaio_ctl->io_handles[i];
+
+ ioh->op = PGAIO_OP_INVALID;
+ ioh->target = PGAIO_TID_INVALID;
+ ioh->state = PGAIO_HS_IDLE;
+
+ slist_init(&ioh->bounce_buffers);
+ }
+
+ /* Initialize Bounce Buffers. */
+ for (uint64 i = 0; i < pgaio_ctl->bounce_buffers_count; i++)
+ {
+ PgAioBounceBuffer *bb = &pgaio_ctl->bounce_buffers[i];
+
+ bb->buffer = bounce_buffers_data;
+ bounce_buffers_data += BLCKSZ;
+ }
+
+
for (int procno = 0; procno < AioProcs(); procno++)
{
PgAioBackend *bs = &pgaio_ctl->backend_state[procno];
@@ -186,9 +297,13 @@ AioShmemInit(void)
bs->io_handle_off = io_handle_off;
io_handle_off += io_max_concurrency;
+ bs->bounce_buffers_off = bounce_buffers_off;
+ bounce_buffers_off += per_backend_bb;
+
dclist_init(&bs->idle_ios);
memset(bs->staged_ios, 0, sizeof(PgAioHandle *) * PGAIO_SUBMIT_BATCH_SIZE);
dclist_init(&bs->in_flight_ios);
+ slist_init(&bs->idle_bbs);
/* initialize per-backend IOs */
for (int i = 0; i < io_max_concurrency; i++)
@@ -210,6 +325,14 @@ AioShmemInit(void)
dclist_push_tail(&bs->idle_ios, &ioh->node);
iovec_off += io_max_combine_limit;
}
+
+ /* initialize per-backend bounce buffers */
+ for (int i = 0; i < per_backend_bb; i++)
+ {
+ PgAioBounceBuffer *bb = &pgaio_ctl->bounce_buffers[bs->bounce_buffers_off + i];
+
+ slist_push_head(&bs->idle_bbs, &bb->node);
+ }
}
out:
diff --git a/src/backend/utils/misc/guc_tables.c b/src/backend/utils/misc/guc_tables.c
index 4eaeca89f2c..bd49b302293 100644
--- a/src/backend/utils/misc/guc_tables.c
+++ b/src/backend/utils/misc/guc_tables.c
@@ -3293,6 +3293,19 @@ struct config_int ConfigureNamesInt[] =
check_io_max_concurrency, NULL, NULL
},
+ {
+ {"io_bounce_buffers",
+ PGC_POSTMASTER,
+ RESOURCES_IO,
+ gettext_noop("Number of IO Bounce Buffers reserved for each backend."),
+ NULL,
+ GUC_UNIT_BLOCKS
+ },
+ &io_bounce_buffers,
+ -1, -1, 4096,
+ NULL, NULL, NULL
+ },
+
{
{"io_workers",
PGC_SIGHUP,
diff --git a/src/backend/utils/misc/postgresql.conf.sample b/src/backend/utils/misc/postgresql.conf.sample
index ff56a1f0732..2c6456e907f 100644
--- a/src/backend/utils/misc/postgresql.conf.sample
+++ b/src/backend/utils/misc/postgresql.conf.sample
@@ -211,6 +211,8 @@
# -1 sets based on shared_buffers
# (change requires restart)
#io_workers = 3 # 1-32;
+#io_bounce_buffers = -1 # -1 sets based on shared_buffers
+ # (change requires restart)
# - Worker Processes -
diff --git a/src/backend/utils/resowner/resowner.c b/src/backend/utils/resowner/resowner.c
index d39f3e1b655..81e7e27965a 100644
--- a/src/backend/utils/resowner/resowner.c
+++ b/src/backend/utils/resowner/resowner.c
@@ -159,10 +159,11 @@ struct ResourceOwnerData
LOCALLOCK *locks[MAX_RESOWNER_LOCKS]; /* list of owned locks */
/*
- * AIO handles need be registered in critical sections and therefore
- * cannot use the normal ResourceElem mechanism.
+ * AIO handles & bounce buffers need be registered in critical sections
+ * and therefore cannot use the normal ResourceElem mechanism.
*/
dlist_head aio_handles;
+ dlist_head aio_bounce_buffers;
};
@@ -434,6 +435,7 @@ ResourceOwnerCreate(ResourceOwner parent, const char *name)
}
dlist_init(&owner->aio_handles);
+ dlist_init(&owner->aio_bounce_buffers);
return owner;
}
@@ -742,6 +744,13 @@ ResourceOwnerReleaseInternal(ResourceOwner owner,
pgaio_io_release_resowner(node, !isCommit);
}
+
+ while (!dlist_is_empty(&owner->aio_bounce_buffers))
+ {
+ dlist_node *node = dlist_head_node(&owner->aio_bounce_buffers);
+
+ pgaio_bounce_buffer_release_resowner(node, !isCommit);
+ }
}
else if (phase == RESOURCE_RELEASE_LOCKS)
{
@@ -1111,3 +1120,15 @@ ResourceOwnerForgetAioHandle(ResourceOwner owner, struct dlist_node *ioh_node)
{
dlist_delete_from(&owner->aio_handles, ioh_node);
}
+
+void
+ResourceOwnerRememberAioBounceBuffer(ResourceOwner owner, struct dlist_node *ioh_node)
+{
+ dlist_push_tail(&owner->aio_bounce_buffers, ioh_node);
+}
+
+void
+ResourceOwnerForgetAioBounceBuffer(ResourceOwner owner, struct dlist_node *ioh_node)
+{
+ dlist_delete_from(&owner->aio_bounce_buffers, ioh_node);
+}
diff --git a/src/test/modules/test_aio/test_aio--1.0.sql b/src/test/modules/test_aio/test_aio--1.0.sql
index e495481c41e..e2a81235166 100644
--- a/src/test/modules/test_aio/test_aio--1.0.sql
+++ b/src/test/modules/test_aio/test_aio--1.0.sql
@@ -87,6 +87,27 @@ RETURNS pg_catalog.void STRICT
AS 'MODULE_PATHNAME' LANGUAGE C;
+CREATE FUNCTION bb_get_and_error()
+RETURNS pg_catalog.void STRICT
+AS 'MODULE_PATHNAME' LANGUAGE C;
+
+CREATE FUNCTION bb_get_twice()
+RETURNS pg_catalog.void STRICT
+AS 'MODULE_PATHNAME' LANGUAGE C;
+
+CREATE FUNCTION bb_get()
+RETURNS pg_catalog.void STRICT
+AS 'MODULE_PATHNAME' LANGUAGE C;
+
+CREATE FUNCTION bb_get_release()
+RETURNS pg_catalog.void STRICT
+AS 'MODULE_PATHNAME' LANGUAGE C;
+
+CREATE FUNCTION bb_release_last()
+RETURNS pg_catalog.void STRICT
+AS 'MODULE_PATHNAME' LANGUAGE C;
+
+
/*
* Injection point related functions
diff --git a/src/test/modules/test_aio/test_aio.c b/src/test/modules/test_aio/test_aio.c
index 35c6b331923..5d8f255e449 100644
--- a/src/test/modules/test_aio/test_aio.c
+++ b/src/test/modules/test_aio/test_aio.c
@@ -52,6 +52,7 @@ static shmem_startup_hook_type prev_shmem_startup_hook = NULL;
static PgAioHandle *last_handle;
+static PgAioBounceBuffer *last_bb;
@@ -671,6 +672,60 @@ batch_end(PG_FUNCTION_ARGS)
PG_RETURN_VOID();
}
+PG_FUNCTION_INFO_V1(bb_get);
+Datum
+bb_get(PG_FUNCTION_ARGS)
+{
+ last_bb = pgaio_bounce_buffer_get();
+
+ PG_RETURN_VOID();
+}
+
+PG_FUNCTION_INFO_V1(bb_release_last);
+Datum
+bb_release_last(PG_FUNCTION_ARGS)
+{
+ if (!last_bb)
+ elog(ERROR, "no bb");
+
+ pgaio_bounce_buffer_release(last_bb);
+
+ PG_RETURN_VOID();
+}
+
+PG_FUNCTION_INFO_V1(bb_get_and_error);
+Datum
+bb_get_and_error(PG_FUNCTION_ARGS)
+{
+ pgaio_bounce_buffer_get();
+
+ elog(ERROR, "as you command");
+ PG_RETURN_VOID();
+}
+
+PG_FUNCTION_INFO_V1(bb_get_twice);
+Datum
+bb_get_twice(PG_FUNCTION_ARGS)
+{
+ pgaio_bounce_buffer_get();
+ pgaio_bounce_buffer_get();
+
+ PG_RETURN_VOID();
+}
+
+
+PG_FUNCTION_INFO_V1(bb_get_release);
+Datum
+bb_get_release(PG_FUNCTION_ARGS)
+{
+ PgAioBounceBuffer *bb;
+
+ bb = pgaio_bounce_buffer_get();
+ pgaio_bounce_buffer_release(bb);
+
+ PG_RETURN_VOID();
+}
+
#ifdef USE_INJECTION_POINTS
extern PGDLLEXPORT void inj_io_short_read(const char *name, const void *private_data);
extern PGDLLEXPORT void inj_io_reopen(const char *name, const void *private_data);
diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list
index 1279b69422a..74ba36eb78c 100644
--- a/src/tools/pgindent/typedefs.list
+++ b/src/tools/pgindent/typedefs.list
@@ -2134,6 +2134,7 @@ PermutationStep
PermutationStepBlocker
PermutationStepBlockerType
PgAioBackend
+PgAioBounceBuffer
PgAioCtl
PgAioHandle
PgAioHandleCallbackID
--
2.48.1.76.g4e746b1a31.dirty
[text/x-diff] v2.14-0022-bufmgr-Implement-AIO-write-support.patch (11.9K, ../../d4ff3qxudxn2ze3zk2xwhds5ysezfei3uqeuimymj624t5chsd@sgr2h3rmln5t/23-v2.14-0022-bufmgr-Implement-AIO-write-support.patch)
download | inline diff:
From 7cdb018181f0f71bf9651f7abc427c675bc484cc Mon Sep 17 00:00:00 2001
From: Andres Freund <andres@anarazel.de>
Date: Tue, 18 Mar 2025 14:40:06 -0400
Subject: [PATCH v2.14 22/29] bufmgr: Implement AIO write support
As of this commit there are no users of these AIO facilities, that'll come in
later commits.
Problems with AIO writes:
- Write logic needs to be rebased on-top of the patch series to not hit bit
dirty buffers while IO is going on
The performance impact of doing the memory copies is rather substantial, as
on intel memory bandwidth is *the* IO bottleneck even just for the checksum
computation, without a copy. That makes the memory copy for something like
bounce buffers hurt really badly.
And the memory usage of bounce buffers is also really concerning.
And even without checksums, several filesystems *really* don't like buffers
getting modified during DIO writes. Which I think would mean we ought to use
bounce buffers for *all* writes, which would impose a *very* substantial
overhead (basically removing the benefit of DMA happening off-cpu).
- I think it requires new lwlock.c infrastructure (as v1 of aio had), to make
LockBuffer(BUFFER_LOCK_EXCLUSIVE) etc wait in a concurrency safe manner for
in-progress writes
I can think of ways to solve this purely in bufmgr.c, but only in ways that
would cause other problems (e.g. setting BM_IO_IN_PROGRESS before waiting
for an exclusive lock) and/or expensive.
---
src/include/storage/aio.h | 4 +-
src/include/storage/bufmgr.h | 2 +
src/backend/storage/aio/aio_callback.c | 2 +
src/backend/storage/buffer/bufmgr.c | 188 ++++++++++++++++++++++++-
4 files changed, 190 insertions(+), 6 deletions(-)
diff --git a/src/include/storage/aio.h b/src/include/storage/aio.h
index f91f0afc5a5..72d5680e767 100644
--- a/src/include/storage/aio.h
+++ b/src/include/storage/aio.h
@@ -197,11 +197,13 @@ typedef enum PgAioHandleCallbackID
PGAIO_HCB_MD_WRITEV,
PGAIO_HCB_SHARED_BUFFER_READV,
+ PGAIO_HCB_SHARED_BUFFER_WRITEV,
PGAIO_HCB_LOCAL_BUFFER_READV,
+ PGAIO_HCB_LOCAL_BUFFER_WRITEV,
} PgAioHandleCallbackID;
-#define PGAIO_HCB_MAX PGAIO_HCB_LOCAL_BUFFER_READV
+#define PGAIO_HCB_MAX PGAIO_HCB_LOCAL_BUFFER_WRITEV
StaticAssertDecl(PGAIO_HCB_MAX <= (1 << PGAIO_RESULT_ID_BITS),
"PGAIO_HCB_MAX is too big for PGAIO_RESULT_ID_BITS");
diff --git a/src/include/storage/bufmgr.h b/src/include/storage/bufmgr.h
index f2192ceb271..492feab0cb5 100644
--- a/src/include/storage/bufmgr.h
+++ b/src/include/storage/bufmgr.h
@@ -174,7 +174,9 @@ extern PGDLLIMPORT int backend_flush_after;
extern PGDLLIMPORT int bgwriter_flush_after;
extern const PgAioHandleCallbacks aio_shared_buffer_readv_cb;
+extern const PgAioHandleCallbacks aio_shared_buffer_writev_cb;
extern const PgAioHandleCallbacks aio_local_buffer_readv_cb;
+extern const PgAioHandleCallbacks aio_local_buffer_writev_cb;
/* in buf_init.c */
extern PGDLLIMPORT char *BufferBlocks;
diff --git a/src/backend/storage/aio/aio_callback.c b/src/backend/storage/aio/aio_callback.c
index 67026f5f6b4..f9c4599ee21 100644
--- a/src/backend/storage/aio/aio_callback.c
+++ b/src/backend/storage/aio/aio_callback.c
@@ -44,8 +44,10 @@ static const PgAioHandleCallbacksEntry aio_handle_cbs[] = {
CALLBACK_ENTRY(PGAIO_HCB_MD_WRITEV, aio_md_writev_cb),
CALLBACK_ENTRY(PGAIO_HCB_SHARED_BUFFER_READV, aio_shared_buffer_readv_cb),
+ CALLBACK_ENTRY(PGAIO_HCB_SHARED_BUFFER_WRITEV, aio_shared_buffer_writev_cb),
CALLBACK_ENTRY(PGAIO_HCB_LOCAL_BUFFER_READV, aio_local_buffer_readv_cb),
+ CALLBACK_ENTRY(PGAIO_HCB_LOCAL_BUFFER_WRITEV, aio_local_buffer_writev_cb),
#undef CALLBACK_ENTRY
};
diff --git a/src/backend/storage/buffer/bufmgr.c b/src/backend/storage/buffer/bufmgr.c
index c65d3916211..688d74f997c 100644
--- a/src/backend/storage/buffer/bufmgr.c
+++ b/src/backend/storage/buffer/bufmgr.c
@@ -5536,7 +5536,15 @@ LockBuffer(Buffer buffer, int mode)
else if (mode == BUFFER_LOCK_SHARE)
LWLockAcquire(BufferDescriptorGetContentLock(buf), LW_SHARED);
else if (mode == BUFFER_LOCK_EXCLUSIVE)
+ {
+ /*
+ * FIXME: Wait for AIO writes, otherwise there would be a risk of
+ * deadlock. This isn't entirely trivial to do in a race-free way, IO
+ * could be started between us checking whether there is IO and
+ * enqueueing ourselves for the lock.
+ */
LWLockAcquire(BufferDescriptorGetContentLock(buf), LW_EXCLUSIVE);
+ }
else
elog(ERROR, "unrecognized buffer lock mode: %d", mode);
}
@@ -5551,6 +5559,19 @@ ConditionalLockBuffer(Buffer buffer)
{
BufferDesc *buf;
+ /*
+ * FIXME: Wait for AIO writes. Some code does not deal well
+ * ConditionalLockBuffer() continuously failing, e.g.
+ * spginsert()->spgdoinsert() ends up busy-looping (easily reproducible by
+ * just making this function always fail and running the regression
+ * tests). While that code could be fixed, it'd be hard to find all
+ * problematic places.
+ *
+ * It would be OK to wait for the IO as waiting for IO completion does not
+ * need to wait for any locks that could lead to an undetected deadlock or
+ * such.
+ */
+
Assert(BufferIsPinned(buffer));
if (BufferIsLocal(buffer))
return true; /* act as though we got it */
@@ -5614,9 +5635,8 @@ LockBufferForCleanup(Buffer buffer)
CheckBufferIsPinnedOnce(buffer);
/*
- * We do not yet need to be worried about in-progress AIOs holding a pin,
- * as we only support doing reads via AIO and this function can only be
- * called once the buffer is valid (i.e. no read can be in flight).
+ * FIXME: See AIO related comments in LockBuffer() and
+ * ConditionalLockBuffer()
*/
/* Nobody else to wait for */
@@ -5629,6 +5649,11 @@ LockBufferForCleanup(Buffer buffer)
{
uint32 buf_state;
+ /*
+ * FIXME: LockBuffer()'s handling of in-progress writes (once
+ * implemented) should suffice to deal with deadlock risk.
+ */
+
/* Try to acquire lock */
LockBuffer(buffer, BUFFER_LOCK_EXCLUSIVE);
buf_state = LockBufHdr(bufHdr);
@@ -5776,7 +5801,13 @@ ConditionalLockBufferForCleanup(Buffer buffer)
Assert(BufferIsValid(buffer));
- /* see AIO related comment in LockBufferForCleanup() */
+ /*
+ * FIXME: Should wait for IO for the same reason as in
+ * ConditionalLockBuffer(). Needs to happen before the
+ * ConditionalLockBuffer() call below, as we'd never reach the
+ * ConditionalLockBuffer() call due the buffer pin held for the duration
+ * of the IO.
+ */
if (BufferIsLocal(buffer))
{
@@ -5833,7 +5864,10 @@ IsBufferCleanupOK(Buffer buffer)
Assert(BufferIsValid(buffer));
- /* see AIO related comment in LockBufferForCleanup() */
+ /*
+ * FIXME: See AIO related comments in LockBuffer() and
+ * ConditionalLockBuffer()
+ */
if (BufferIsLocal(buffer))
{
@@ -7133,12 +7167,129 @@ buffer_readv_report(PgAioResult result, const PgAioTargetData *td,
affected_count > 1 ? errhint_internal(hint_mult, affected_count - 1) : 0);
}
+/*
+ * Helper for AIO writev completion callbacks, supporting both shared and temp
+ * buffers. Gets called once for each buffer in a multi-page write.
+ */
+static pg_attribute_always_inline PgAioResult
+buffer_writev_complete_one(uint8 buf_off, Buffer buffer, uint8 flags,
+ bool failed, bool is_temp)
+{
+ BufferDesc *buf_hdr = is_temp ?
+ GetLocalBufferDescriptor(-buffer - 1)
+ : GetBufferDescriptor(buffer - 1);
+ PgAioResult result = {.status = PGAIO_RS_OK};
+ bool clear_dirty;
+ uint32 set_flag_bits;
+
+#ifdef USE_ASSERT_CHECKING
+ {
+ uint32 buf_state = pg_atomic_read_u32(&buf_hdr->state);
+
+ Assert(buf_state & BM_VALID);
+ Assert(buf_state & BM_TAG_VALID);
+ /* temp buffers don't use BM_IO_IN_PROGRESS */
+ if (!is_temp)
+ Assert(buf_state & BM_IO_IN_PROGRESS);
+ Assert(buf_state & BM_DIRTY);
+ }
+#endif
+
+ clear_dirty = failed ? false : true;
+ set_flag_bits = failed ? BM_IO_ERROR : 0;
+
+ if (is_temp)
+ TerminateLocalBufferIO(buf_hdr, clear_dirty, set_flag_bits, true);
+ else
+ TerminateBufferIO(buf_hdr, clear_dirty, set_flag_bits, false, true);
+
+ /*
+ * The initiator of IO is not managing the lock (i.e. we called
+ * LWLockDisown()), we are.
+ */
+ if (!is_temp)
+ LWLockReleaseDisowned(BufferDescriptorGetContentLock(buf_hdr),
+ LW_SHARED);
+
+ /* FIXME: tracepoint */
+
+ return result;
+}
+
+/*
+ * Perform completion handling of a single AIO write. This write may cover
+ * multiple blocks / buffers.
+ *
+ * Shared between shared and local buffers, to reduce code duplication.
+ */
+static pg_attribute_always_inline PgAioResult
+buffer_writev_complete(PgAioHandle *ioh, PgAioResult prior_result,
+ uint8 cb_data, bool is_temp)
+{
+ PgAioResult result = prior_result;
+ PgAioTargetData *td = pgaio_io_get_target_data(ioh);
+ uint64 *io_data;
+ uint8 handle_data_len;
+
+ if (is_temp)
+ {
+ Assert(td->smgr.is_temp);
+ Assert(pgaio_io_get_owner(ioh) == MyProcNumber);
+ }
+ else
+ Assert(!td->smgr.is_temp);
+
+ /*
+ * Iterate over all the buffers affected by this IO and call appropriate
+ * per-buffer completion function for each buffer.
+ */
+ io_data = pgaio_io_get_handle_data(ioh, &handle_data_len);
+ for (uint8 buf_off = 0; buf_off < handle_data_len; buf_off++)
+ {
+ Buffer buf = io_data[buf_off];
+ PgAioResult buf_result;
+ bool failed;
+
+ Assert(BufferIsValid(buf));
+
+ /*
+ * If the entire failed on a lower-level, each buffer needs to be
+ * marked as failed. In case of a partial read, some buffers may be
+ * ok.
+ */
+ failed =
+ prior_result.status == PGAIO_RS_ERROR
+ || prior_result.result <= buf_off;
+
+ buf_result = buffer_writev_complete_one(buf_off, buf, cb_data, failed,
+ is_temp);
+
+ /*
+ * If there wasn't any prior error and the IO for this page failed in
+ * some form, set the whole IO's to the page's result.
+ */
+ if (result.status != PGAIO_RS_ERROR && buf_result.status != PGAIO_RS_OK)
+ {
+ result = buf_result;
+ pgaio_result_report(result, td, LOG);
+ }
+ }
+
+ return result;
+}
+
static void
shared_buffer_readv_stage(PgAioHandle *ioh, uint8 cb_data)
{
buffer_stage_common(ioh, false, false);
}
+static void
+shared_buffer_writev_stage(PgAioHandle *ioh, uint8 cb_data)
+{
+ buffer_stage_common(ioh, true, false);
+}
+
static PgAioResult
shared_buffer_readv_complete(PgAioHandle *ioh, PgAioResult prior_result,
uint8 cb_data)
@@ -7182,6 +7333,13 @@ shared_buffer_readv_complete_local(PgAioHandle *ioh, PgAioResult prior_result,
return prior_result;
}
+static PgAioResult
+shared_buffer_writev_complete(PgAioHandle *ioh, PgAioResult prior_result,
+ uint8 cb_data)
+{
+ return buffer_writev_complete(ioh, prior_result, cb_data, false);
+}
+
static void
local_buffer_readv_stage(PgAioHandle *ioh, uint8 cb_data)
{
@@ -7195,6 +7353,17 @@ local_buffer_readv_complete(PgAioHandle *ioh, PgAioResult prior_result,
return buffer_readv_complete(ioh, prior_result, cb_data, true);
}
+static void
+local_buffer_writev_stage(PgAioHandle *ioh, uint8 cb_data)
+{
+ /*
+ * Currently this is unreachable as the only write support is for
+ * checkpointer / bgwriter, which don't deal with local buffers.
+ */
+ elog(ERROR, "should be unreachable");
+}
+
+
/* readv callback is passed READ_BUFFERS_* flags as callback data */
const PgAioHandleCallbacks aio_shared_buffer_readv_cb = {
.stage = shared_buffer_readv_stage,
@@ -7204,6 +7373,11 @@ const PgAioHandleCallbacks aio_shared_buffer_readv_cb = {
.report = buffer_readv_report,
};
+const PgAioHandleCallbacks aio_shared_buffer_writev_cb = {
+ .stage = shared_buffer_writev_stage,
+ .complete_shared = shared_buffer_writev_complete,
+};
+
/* readv callback is passed READ_BUFFERS_* flags as callback data */
const PgAioHandleCallbacks aio_local_buffer_readv_cb = {
.stage = local_buffer_readv_stage,
@@ -7217,3 +7391,7 @@ const PgAioHandleCallbacks aio_local_buffer_readv_cb = {
.complete_local = local_buffer_readv_complete,
.report = buffer_readv_report,
};
+
+const PgAioHandleCallbacks aio_local_buffer_writev_cb = {
+ .stage = local_buffer_writev_stage,
+};
--
2.48.1.76.g4e746b1a31.dirty
[text/x-diff] v2.14-0023-aio-Add-IO-queue-helper.patch (7.4K, ../../d4ff3qxudxn2ze3zk2xwhds5ysezfei3uqeuimymj624t5chsd@sgr2h3rmln5t/24-v2.14-0023-aio-Add-IO-queue-helper.patch)
download | inline diff:
From 772973488e99e03c6e4f39665c6e210e4d5af27c Mon Sep 17 00:00:00 2001
From: Andres Freund <andres@anarazel.de>
Date: Tue, 18 Mar 2025 14:40:06 -0400
Subject: [PATCH v2.14 23/29] aio: Add IO queue helper
This is likely never going to anywhere - Thomas Munro is working on something
more complete. But I needed a way to exercise aio for checkpointer / bgwriter.
---
src/include/storage/io_queue.h | 33 +++++
src/backend/storage/aio/Makefile | 1 +
src/backend/storage/aio/io_queue.c | 204 ++++++++++++++++++++++++++++
src/backend/storage/aio/meson.build | 1 +
src/tools/pgindent/typedefs.list | 2 +
5 files changed, 241 insertions(+)
create mode 100644 src/include/storage/io_queue.h
create mode 100644 src/backend/storage/aio/io_queue.c
diff --git a/src/include/storage/io_queue.h b/src/include/storage/io_queue.h
new file mode 100644
index 00000000000..92b1e9afe6f
--- /dev/null
+++ b/src/include/storage/io_queue.h
@@ -0,0 +1,33 @@
+/*-------------------------------------------------------------------------
+ *
+ * io_queue.h
+ * Mechanism for tracking many IOs
+ *
+ *
+ * Portions Copyright (c) 1996-2025, PostgreSQL Global Development Group
+ * Portions Copyright (c) 1994, Regents of the University of California
+ *
+ * src/include/storage/io_queue.h
+ *
+ *-------------------------------------------------------------------------
+ */
+#ifndef IO_QUEUE_H
+#define IO_QUEUE_H
+
+#include "storage/aio_types.h"
+
+struct IOQueue;
+typedef struct IOQueue IOQueue;
+
+struct PgAioWaitRef;
+
+extern IOQueue *io_queue_create(int depth, int flags);
+extern void io_queue_track(IOQueue *ioq, const PgAioWaitRef *iow);
+extern void io_queue_wait_one(IOQueue *ioq);
+extern void io_queue_wait_all(IOQueue *ioq);
+extern bool io_queue_is_empty(IOQueue *ioq);
+extern void io_queue_reserve(IOQueue *ioq);
+extern PgAioHandle *io_queue_acquire_io(IOQueue *ioq);
+extern void io_queue_free(IOQueue *ioq);
+
+#endif /* IO_QUEUE_H */
diff --git a/src/backend/storage/aio/Makefile b/src/backend/storage/aio/Makefile
index 3f2469cc399..86fa4276fda 100644
--- a/src/backend/storage/aio/Makefile
+++ b/src/backend/storage/aio/Makefile
@@ -15,6 +15,7 @@ OBJS = \
aio_init.o \
aio_io.o \
aio_target.o \
+ io_queue.o \
method_io_uring.o \
method_sync.o \
method_worker.o \
diff --git a/src/backend/storage/aio/io_queue.c b/src/backend/storage/aio/io_queue.c
new file mode 100644
index 00000000000..526aa1d5e06
--- /dev/null
+++ b/src/backend/storage/aio/io_queue.c
@@ -0,0 +1,204 @@
+/*-------------------------------------------------------------------------
+ *
+ * io_queue.c
+ * AIO - Mechanism for tracking many IOs
+ *
+ * Portions Copyright (c) 1996-2025, PostgreSQL Global Development Group
+ * Portions Copyright (c) 1994, Regents of the University of California
+ *
+ * IDENTIFICATION
+ * src/backend/storage/aio/io_queue.c
+ *
+ *-------------------------------------------------------------------------
+ */
+
+#include "postgres.h"
+
+#include "lib/ilist.h"
+#include "storage/aio.h"
+#include "storage/io_queue.h"
+#include "utils/resowner.h"
+
+
+
+typedef struct TrackedIO
+{
+ PgAioWaitRef iow;
+ dlist_node node;
+} TrackedIO;
+
+struct IOQueue
+{
+ int depth;
+ int unsubmitted;
+
+ bool has_reserved;
+
+ dclist_head idle;
+ dclist_head in_progress;
+
+ TrackedIO tracked_ios[FLEXIBLE_ARRAY_MEMBER];
+};
+
+
+IOQueue *
+io_queue_create(int depth, int flags)
+{
+ size_t sz;
+ IOQueue *ioq;
+
+ sz = offsetof(IOQueue, tracked_ios)
+ + sizeof(TrackedIO) * depth;
+
+ ioq = palloc0(sz);
+
+ ioq->depth = 0;
+
+ for (int i = 0; i < depth; i++)
+ {
+ TrackedIO *tio = &ioq->tracked_ios[i];
+
+ pgaio_wref_clear(&tio->iow);
+ dclist_push_tail(&ioq->idle, &tio->node);
+ }
+
+ return ioq;
+}
+
+void
+io_queue_wait_one(IOQueue *ioq)
+{
+ /* submit all pending IO before waiting */
+ pgaio_submit_staged();
+
+ while (!dclist_is_empty(&ioq->in_progress))
+ {
+ /* FIXME: Should we really pop here already? */
+ dlist_node *node = dclist_pop_head_node(&ioq->in_progress);
+ TrackedIO *tio = dclist_container(TrackedIO, node, node);
+
+ pgaio_wref_wait(&tio->iow);
+ dclist_push_head(&ioq->idle, &tio->node);
+ }
+}
+
+void
+io_queue_reserve(IOQueue *ioq)
+{
+ if (ioq->has_reserved)
+ return;
+
+ if (dclist_is_empty(&ioq->idle))
+ io_queue_wait_one(ioq);
+
+ Assert(!dclist_is_empty(&ioq->idle));
+
+ ioq->has_reserved = true;
+}
+
+PgAioHandle *
+io_queue_acquire_io(IOQueue *ioq)
+{
+ PgAioHandle *ioh;
+
+ io_queue_reserve(ioq);
+
+ Assert(!dclist_is_empty(&ioq->idle));
+
+ if (!io_queue_is_empty(ioq))
+ {
+ ioh = pgaio_io_acquire_nb(CurrentResourceOwner, NULL);
+ if (ioh == NULL)
+ {
+ /*
+ * Need to wait for all IOs, blocking might not be legal in the
+ * context.
+ *
+ * XXX: This doesn't make a whole lot of sense, we're also
+ * blocking here. What was I smoking when I wrote the above?
+ */
+ io_queue_wait_all(ioq);
+ ioh = pgaio_io_acquire(CurrentResourceOwner, NULL);
+ }
+ }
+ else
+ {
+ ioh = pgaio_io_acquire(CurrentResourceOwner, NULL);
+ }
+
+ return ioh;
+}
+
+void
+io_queue_track(IOQueue *ioq, const struct PgAioWaitRef *iow)
+{
+ dlist_node *node;
+ TrackedIO *tio;
+
+ Assert(ioq->has_reserved);
+ ioq->has_reserved = false;
+
+ Assert(!dclist_is_empty(&ioq->idle));
+
+ node = dclist_pop_head_node(&ioq->idle);
+ tio = dclist_container(TrackedIO, node, node);
+
+ tio->iow = *iow;
+
+ dclist_push_tail(&ioq->in_progress, &tio->node);
+
+ ioq->unsubmitted++;
+
+ /*
+ * XXX: Should have some smarter logic here. We don't want to wait too
+ * long to submit, that'll mean we're more likely to block. But we also
+ * don't want to have the overhead of submitting every IO individually.
+ */
+ if (ioq->unsubmitted >= 4)
+ {
+ pgaio_submit_staged();
+ ioq->unsubmitted = 0;
+ }
+}
+
+void
+io_queue_wait_all(IOQueue *ioq)
+{
+ /* submit all pending IO before waiting */
+ pgaio_submit_staged();
+
+ while (!dclist_is_empty(&ioq->in_progress))
+ {
+ /* wait for the last IO to minimize unnecessary wakeups */
+ dlist_node *node = dclist_tail_node(&ioq->in_progress);
+ TrackedIO *tio = dclist_container(TrackedIO, node, node);
+
+ if (!pgaio_wref_check_done(&tio->iow))
+ {
+ ereport(DEBUG3,
+ errmsg("io_queue_wait_all for io:%d",
+ pgaio_wref_get_id(&tio->iow)),
+ errhidestmt(true),
+ errhidecontext(true));
+
+ pgaio_wref_wait(&tio->iow);
+ }
+
+ dclist_delete_from(&ioq->in_progress, &tio->node);
+ dclist_push_head(&ioq->idle, &tio->node);
+ }
+}
+
+bool
+io_queue_is_empty(IOQueue *ioq)
+{
+ return dclist_is_empty(&ioq->in_progress);
+}
+
+void
+io_queue_free(IOQueue *ioq)
+{
+ io_queue_wait_all(ioq);
+
+ pfree(ioq);
+}
diff --git a/src/backend/storage/aio/meson.build b/src/backend/storage/aio/meson.build
index da6df2d3654..270c4a64428 100644
--- a/src/backend/storage/aio/meson.build
+++ b/src/backend/storage/aio/meson.build
@@ -7,6 +7,7 @@ backend_sources += files(
'aio_init.c',
'aio_io.c',
'aio_target.c',
+ 'io_queue.c',
'method_io_uring.c',
'method_sync.c',
'method_worker.c',
diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list
index 74ba36eb78c..54239ccc69e 100644
--- a/src/tools/pgindent/typedefs.list
+++ b/src/tools/pgindent/typedefs.list
@@ -1193,6 +1193,7 @@ IOContext
IOFuncSelector
IOObject
IOOp
+IOQueue
IO_STATUS_BLOCK
IPCompareMethod
ITEM
@@ -3019,6 +3020,7 @@ TocEntry
TokenAuxData
TokenizedAuthLine
TrackItem
+TrackedIO
TransApplyAction
TransInvalidationInfo
TransState
--
2.48.1.76.g4e746b1a31.dirty
[text/x-diff] v2.14-0024-bufmgr-use-AIO-in-checkpointer-bgwriter.patch (31.6K, ../../d4ff3qxudxn2ze3zk2xwhds5ysezfei3uqeuimymj624t5chsd@sgr2h3rmln5t/25-v2.14-0024-bufmgr-use-AIO-in-checkpointer-bgwriter.patch)
download | inline diff:
From b7962cd93be0adadebd861ecb76a72464c379c2e Mon Sep 17 00:00:00 2001
From: Andres Freund <andres@anarazel.de>
Date: Tue, 18 Mar 2025 14:40:06 -0400
Subject: [PATCH v2.14 24/29] bufmgr: use AIO in checkpointer, bgwriter
This is far from ready - just included to be able to exercise AIO writes and
get some preliminary numbers. In all likelihood this will instead be based
on-top of work by Thomas Munro instead of the preceding commit.
TODO;
- This doesn't implement bgwriter_flush_after, checkpointer_flush_after
I think that's not too hard to do, it's mainly round tuits.
- The queuing logic doesn't carefully respect pin limits
That might be ok for checkpointer and bgwriter, but the infrastructure
should be usable outside of this as well.
---
src/include/postmaster/bgwriter.h | 3 +-
src/include/storage/buf_internals.h | 2 +
src/include/storage/bufmgr.h | 3 +-
src/include/storage/bufpage.h | 1 +
src/backend/postmaster/bgwriter.c | 19 +-
src/backend/postmaster/checkpointer.c | 11 +-
src/backend/storage/buffer/bufmgr.c | 594 +++++++++++++++++++++++---
src/backend/storage/page/bufpage.c | 10 +
src/tools/pgindent/typedefs.list | 1 +
9 files changed, 586 insertions(+), 58 deletions(-)
diff --git a/src/include/postmaster/bgwriter.h b/src/include/postmaster/bgwriter.h
index 800ecbfd13b..a8081d411b6 100644
--- a/src/include/postmaster/bgwriter.h
+++ b/src/include/postmaster/bgwriter.h
@@ -31,7 +31,8 @@ pg_noreturn extern void BackgroundWriterMain(const void *startup_data, size_t st
pg_noreturn extern void CheckpointerMain(const void *startup_data, size_t startup_data_len);
extern void RequestCheckpoint(int flags);
-extern void CheckpointWriteDelay(int flags, double progress);
+struct IOQueue;
+extern void CheckpointWriteDelay(struct IOQueue *ioq, int flags, double progress);
extern bool ForwardSyncRequest(const FileTag *ftag, SyncRequestType type);
diff --git a/src/include/storage/buf_internals.h b/src/include/storage/buf_internals.h
index 0dec7d93b3b..45c2b70b736 100644
--- a/src/include/storage/buf_internals.h
+++ b/src/include/storage/buf_internals.h
@@ -21,6 +21,8 @@
#include "storage/buf.h"
#include "storage/bufmgr.h"
#include "storage/condition_variable.h"
+#include "storage/io_queue.h"
+#include "storage/latch.h"
#include "storage/lwlock.h"
#include "storage/procnumber.h"
#include "storage/shmem.h"
diff --git a/src/include/storage/bufmgr.h b/src/include/storage/bufmgr.h
index 492feab0cb5..89e0ca11288 100644
--- a/src/include/storage/bufmgr.h
+++ b/src/include/storage/bufmgr.h
@@ -297,7 +297,8 @@ extern bool ConditionalLockBufferForCleanup(Buffer buffer);
extern bool IsBufferCleanupOK(Buffer buffer);
extern bool HoldingBufferPinThatDelaysRecovery(void);
-extern bool BgBufferSync(struct WritebackContext *wb_context);
+struct IOQueue;
+extern bool BgBufferSync(struct IOQueue *ioq, struct WritebackContext *wb_context);
extern uint32 GetPinLimit(void);
extern uint32 GetLocalPinLimit(void);
diff --git a/src/include/storage/bufpage.h b/src/include/storage/bufpage.h
index aeb67c498c5..000a3ab23f9 100644
--- a/src/include/storage/bufpage.h
+++ b/src/include/storage/bufpage.h
@@ -507,5 +507,6 @@ extern bool PageIndexTupleOverwrite(Page page, OffsetNumber offnum,
Item newtup, Size newsize);
extern char *PageSetChecksumCopy(Page page, BlockNumber blkno);
extern void PageSetChecksumInplace(Page page, BlockNumber blkno);
+extern bool PageNeedsChecksumCopy(Page page);
#endif /* BUFPAGE_H */
diff --git a/src/backend/postmaster/bgwriter.c b/src/backend/postmaster/bgwriter.c
index 72f5acceec7..6e8801a39e3 100644
--- a/src/backend/postmaster/bgwriter.c
+++ b/src/backend/postmaster/bgwriter.c
@@ -38,11 +38,13 @@
#include "postmaster/auxprocess.h"
#include "postmaster/bgwriter.h"
#include "postmaster/interrupt.h"
+#include "storage/aio.h"
#include "storage/aio_subsys.h"
#include "storage/buf_internals.h"
#include "storage/bufmgr.h"
#include "storage/condition_variable.h"
#include "storage/fd.h"
+#include "storage/io_queue.h"
#include "storage/lwlock.h"
#include "storage/proc.h"
#include "storage/procsignal.h"
@@ -90,6 +92,7 @@ BackgroundWriterMain(const void *startup_data, size_t startup_data_len)
sigjmp_buf local_sigjmp_buf;
MemoryContext bgwriter_context;
bool prev_hibernate;
+ IOQueue *ioq;
WritebackContext wb_context;
Assert(startup_data_len == 0);
@@ -131,6 +134,7 @@ BackgroundWriterMain(const void *startup_data, size_t startup_data_len)
ALLOCSET_DEFAULT_SIZES);
MemoryContextSwitchTo(bgwriter_context);
+ ioq = io_queue_create(128, 0);
WritebackContextInit(&wb_context, &bgwriter_flush_after);
/*
@@ -228,12 +232,22 @@ BackgroundWriterMain(const void *startup_data, size_t startup_data_len)
/* Clear any already-pending wakeups */
ResetLatch(MyLatch);
+ /*
+ * FIXME: this is theoretically racy, but I didn't want to copy
+ * ProcessMainLoopInterrupts() remaining body here.
+ */
+ if (ShutdownRequestPending)
+ {
+ io_queue_wait_all(ioq);
+ io_queue_free(ioq);
+ }
+
ProcessMainLoopInterrupts();
/*
* Do one cycle of dirty-buffer writing.
*/
- can_hibernate = BgBufferSync(&wb_context);
+ can_hibernate = BgBufferSync(ioq, &wb_context);
/* Report pending statistics to the cumulative stats system */
pgstat_report_bgwriter();
@@ -250,6 +264,9 @@ BackgroundWriterMain(const void *startup_data, size_t startup_data_len)
smgrdestroyall();
}
+ /* finish IO before sleeping, to avoid blocking other backends */
+ io_queue_wait_all(ioq);
+
/*
* Log a new xl_running_xacts every now and then so replication can
* get into a consistent state faster (think of suboverflowed
diff --git a/src/backend/postmaster/checkpointer.c b/src/backend/postmaster/checkpointer.c
index fda91ffd1ce..904fe167eb4 100644
--- a/src/backend/postmaster/checkpointer.c
+++ b/src/backend/postmaster/checkpointer.c
@@ -49,10 +49,12 @@
#include "postmaster/bgwriter.h"
#include "postmaster/interrupt.h"
#include "replication/syncrep.h"
+#include "storage/aio.h"
#include "storage/aio_subsys.h"
#include "storage/bufmgr.h"
#include "storage/condition_variable.h"
#include "storage/fd.h"
+#include "storage/io_queue.h"
#include "storage/ipc.h"
#include "storage/lwlock.h"
#include "storage/pmsignal.h"
@@ -766,7 +768,7 @@ ImmediateCheckpointRequested(void)
* fraction between 0.0 meaning none, and 1.0 meaning all done.
*/
void
-CheckpointWriteDelay(int flags, double progress)
+CheckpointWriteDelay(IOQueue *ioq, int flags, double progress)
{
static int absorb_counter = WRITES_PER_ABSORB;
@@ -800,6 +802,13 @@ CheckpointWriteDelay(int flags, double progress)
/* Report interim statistics to the cumulative stats system */
pgstat_report_checkpointer();
+ /*
+ * Ensure all pending IO is submitted to avoid unnecessary delays for
+ * other processes.
+ */
+ io_queue_wait_all(ioq);
+
+
/*
* This sleep used to be connected to bgwriter_delay, typically 200ms.
* That resulted in more frequent wakeups if not much work to do.
diff --git a/src/backend/storage/buffer/bufmgr.c b/src/backend/storage/buffer/bufmgr.c
index 688d74f997c..65a6643dc12 100644
--- a/src/backend/storage/buffer/bufmgr.c
+++ b/src/backend/storage/buffer/bufmgr.c
@@ -52,6 +52,7 @@
#include "storage/buf_internals.h"
#include "storage/bufmgr.h"
#include "storage/fd.h"
+#include "storage/io_queue.h"
#include "storage/ipc.h"
#include "storage/lmgr.h"
#include "storage/proc.h"
@@ -76,6 +77,7 @@
/* Bits in SyncOneBuffer's return value */
#define BUF_WRITTEN 0x01
#define BUF_REUSABLE 0x02
+#define BUF_CANT_MERGE 0x04
#define RELS_BSEARCH_THRESHOLD 20
@@ -515,8 +517,6 @@ static void UnpinBuffer(BufferDesc *buf);
static void UnpinBufferNoOwner(BufferDesc *buf);
static void BufferSync(int flags);
static uint32 WaitBufHdrUnlocked(BufferDesc *buf);
-static int SyncOneBuffer(int buf_id, bool skip_recently_used,
- WritebackContext *wb_context);
static void WaitIO(BufferDesc *buf);
static void AbortBufferIO(Buffer buffer);
static void shared_buffer_write_error_callback(void *arg);
@@ -532,6 +532,7 @@ static void CheckReadBuffersOperation(ReadBuffersOperation *operation, bool is_c
static Buffer GetVictimBuffer(BufferAccessStrategy strategy, IOContext io_context);
static void FlushBuffer(BufferDesc *buf, SMgrRelation reln,
IOObject io_object, IOContext io_context);
+
static void FindAndDropRelationBuffers(RelFileLocator rlocator,
ForkNumber forkNum,
BlockNumber nForkBlock,
@@ -3321,6 +3322,57 @@ UnpinBufferNoOwner(BufferDesc *buf)
}
}
+typedef struct BuffersToWrite
+{
+ int nbuffers;
+ BufferTag start_at_tag;
+ uint32 max_combine;
+
+ XLogRecPtr max_lsn;
+
+ PgAioHandle *ioh;
+ PgAioWaitRef iow;
+
+ uint64 total_writes;
+
+ Buffer buffers[IOV_MAX];
+ PgAioBounceBuffer *bounce_buffers[IOV_MAX];
+ const void *data_ptrs[IOV_MAX];
+} BuffersToWrite;
+
+static int PrepareToWriteBuffer(BuffersToWrite *to_write, Buffer buf,
+ bool skip_recently_used,
+ IOQueue *ioq, WritebackContext *wb_context);
+
+static void WriteBuffers(BuffersToWrite *to_write,
+ IOQueue *ioq, WritebackContext *wb_context);
+
+static void
+BuffersToWriteInit(BuffersToWrite *to_write,
+ IOQueue *ioq, WritebackContext *wb_context)
+{
+ to_write->total_writes = 0;
+ to_write->nbuffers = 0;
+ to_write->ioh = NULL;
+ pgaio_wref_clear(&to_write->iow);
+ to_write->max_lsn = InvalidXLogRecPtr;
+
+ pgaio_enter_batchmode();
+}
+
+static void
+BuffersToWriteEnd(BuffersToWrite *to_write)
+{
+ if (to_write->ioh != NULL)
+ {
+ pgaio_io_release(to_write->ioh);
+ to_write->ioh = NULL;
+ }
+
+ pgaio_exit_batchmode();
+}
+
+
#define ST_SORT sort_checkpoint_bufferids
#define ST_ELEMENT_TYPE CkptSortItem
#define ST_COMPARE(a, b) ckpt_buforder_comparator(a, b)
@@ -3352,7 +3404,10 @@ BufferSync(int flags)
binaryheap *ts_heap;
int i;
int mask = BM_DIRTY;
+ IOQueue *ioq;
WritebackContext wb_context;
+ BuffersToWrite to_write;
+ int max_combine;
/*
* Unless this is a shutdown checkpoint or we have been explicitly told,
@@ -3414,7 +3469,9 @@ BufferSync(int flags)
if (num_to_scan == 0)
return; /* nothing to do */
+ ioq = io_queue_create(512, 0);
WritebackContextInit(&wb_context, &checkpoint_flush_after);
+ max_combine = Min(io_bounce_buffers, io_combine_limit);
TRACE_POSTGRESQL_BUFFER_SYNC_START(NBuffers, num_to_scan);
@@ -3522,48 +3579,91 @@ BufferSync(int flags)
*/
num_processed = 0;
num_written = 0;
+
+ BuffersToWriteInit(&to_write, ioq, &wb_context);
+
while (!binaryheap_empty(ts_heap))
{
BufferDesc *bufHdr = NULL;
CkptTsStatus *ts_stat = (CkptTsStatus *)
DatumGetPointer(binaryheap_first(ts_heap));
+ bool batch_continue = true;
- buf_id = CkptBufferIds[ts_stat->index].buf_id;
- Assert(buf_id != -1);
-
- bufHdr = GetBufferDescriptor(buf_id);
-
- num_processed++;
+ Assert(ts_stat->num_scanned <= ts_stat->num_to_scan);
/*
- * We don't need to acquire the lock here, because we're only looking
- * at a single bit. It's possible that someone else writes the buffer
- * and clears the flag right after we check, but that doesn't matter
- * since SyncOneBuffer will then do nothing. However, there is a
- * further race condition: it's conceivable that between the time we
- * examine the bit here and the time SyncOneBuffer acquires the lock,
- * someone else not only wrote the buffer but replaced it with another
- * page and dirtied it. In that improbable case, SyncOneBuffer will
- * write the buffer though we didn't need to. It doesn't seem worth
- * guarding against this, though.
+ * Collect a batch of buffers to write out from the current
+ * tablespace. That causes some imbalance between the tablespaces, but
+ * that's more than outweighed by the efficiency gain due to batching.
*/
- if (pg_atomic_read_u32(&bufHdr->state) & BM_CHECKPOINT_NEEDED)
+ while (batch_continue &&
+ to_write.nbuffers < max_combine &&
+ ts_stat->num_scanned < ts_stat->num_to_scan)
{
- if (SyncOneBuffer(buf_id, false, &wb_context) & BUF_WRITTEN)
+ buf_id = CkptBufferIds[ts_stat->index].buf_id;
+ Assert(buf_id != -1);
+
+ bufHdr = GetBufferDescriptor(buf_id);
+
+ num_processed++;
+
+ /*
+ * We don't need to acquire the lock here, because we're only
+ * looking at a single bit. It's possible that someone else writes
+ * the buffer and clears the flag right after we check, but that
+ * doesn't matter since SyncOneBuffer will then do nothing.
+ * However, there is a further race condition: it's conceivable
+ * that between the time we examine the bit here and the time
+ * SyncOneBuffer acquires the lock, someone else not only wrote
+ * the buffer but replaced it with another page and dirtied it. In
+ * that improbable case, SyncOneBuffer will write the buffer
+ * though we didn't need to. It doesn't seem worth guarding
+ * against this, though.
+ */
+ if (pg_atomic_read_u32(&bufHdr->state) & BM_CHECKPOINT_NEEDED)
{
- TRACE_POSTGRESQL_BUFFER_SYNC_WRITTEN(buf_id);
- PendingCheckpointerStats.buffers_written++;
- num_written++;
+ int result = PrepareToWriteBuffer(&to_write, buf_id + 1, false,
+ ioq, &wb_context);
+
+ if (result & BUF_CANT_MERGE)
+ {
+ Assert(to_write.nbuffers > 0);
+ WriteBuffers(&to_write, ioq, &wb_context);
+
+ result = PrepareToWriteBuffer(&to_write, buf_id + 1, false,
+ ioq, &wb_context);
+ Assert(result != BUF_CANT_MERGE);
+ }
+
+ if (result & BUF_WRITTEN)
+ {
+ TRACE_POSTGRESQL_BUFFER_SYNC_WRITTEN(buf_id);
+ PendingCheckpointerStats.buffers_written++;
+ num_written++;
+ }
+ else
+ {
+ batch_continue = false;
+ }
}
+ else
+ {
+ if (to_write.nbuffers > 0)
+ WriteBuffers(&to_write, ioq, &wb_context);
+ }
+
+ /*
+ * Measure progress independent of actually having to flush the
+ * buffer - otherwise writing become unbalanced.
+ */
+ ts_stat->progress += ts_stat->progress_slice;
+ ts_stat->num_scanned++;
+ ts_stat->index++;
}
- /*
- * Measure progress independent of actually having to flush the buffer
- * - otherwise writing become unbalanced.
- */
- ts_stat->progress += ts_stat->progress_slice;
- ts_stat->num_scanned++;
- ts_stat->index++;
+ if (to_write.nbuffers > 0)
+ WriteBuffers(&to_write, ioq, &wb_context);
+
/* Have all the buffers from the tablespace been processed? */
if (ts_stat->num_scanned == ts_stat->num_to_scan)
@@ -3581,15 +3681,23 @@ BufferSync(int flags)
*
* (This will check for barrier events even if it doesn't sleep.)
*/
- CheckpointWriteDelay(flags, (double) num_processed / num_to_scan);
+ CheckpointWriteDelay(ioq, flags, (double) num_processed / num_to_scan);
}
+ Assert(to_write.nbuffers == 0);
+ io_queue_wait_all(ioq);
+
/*
* Issue all pending flushes. Only checkpointer calls BufferSync(), so
* IOContext will always be IOCONTEXT_NORMAL.
*/
IssuePendingWritebacks(&wb_context, IOCONTEXT_NORMAL);
+ io_queue_wait_all(ioq); /* IssuePendingWritebacks might have added
+ * more */
+ io_queue_free(ioq);
+ BuffersToWriteEnd(&to_write);
+
pfree(per_ts_stat);
per_ts_stat = NULL;
binaryheap_free(ts_heap);
@@ -3615,7 +3723,7 @@ BufferSync(int flags)
* bgwriter_lru_maxpages to 0.)
*/
bool
-BgBufferSync(WritebackContext *wb_context)
+BgBufferSync(IOQueue *ioq, WritebackContext *wb_context)
{
/* info obtained from freelist.c */
int strategy_buf_id;
@@ -3658,6 +3766,9 @@ BgBufferSync(WritebackContext *wb_context)
long new_strategy_delta;
uint32 new_recent_alloc;
+ BuffersToWrite to_write;
+ int max_combine;
+
/*
* Find out where the freelist clock sweep currently is, and how many
* buffer allocations have happened since our last call.
@@ -3678,6 +3789,8 @@ BgBufferSync(WritebackContext *wb_context)
return true;
}
+ max_combine = Min(io_bounce_buffers, io_combine_limit);
+
/*
* Compute strategy_delta = how many buffers have been scanned by the
* clock sweep since last time. If first time through, assume none. Then
@@ -3834,11 +3947,25 @@ BgBufferSync(WritebackContext *wb_context)
num_written = 0;
reusable_buffers = reusable_buffers_est;
+ BuffersToWriteInit(&to_write, ioq, wb_context);
+
/* Execute the LRU scan */
while (num_to_scan > 0 && reusable_buffers < upcoming_alloc_est)
{
- int sync_state = SyncOneBuffer(next_to_clean, true,
- wb_context);
+ int sync_state;
+
+ sync_state = PrepareToWriteBuffer(&to_write, next_to_clean + 1,
+ true, ioq, wb_context);
+ if (sync_state & BUF_CANT_MERGE)
+ {
+ Assert(to_write.nbuffers > 0);
+
+ WriteBuffers(&to_write, ioq, wb_context);
+
+ sync_state = PrepareToWriteBuffer(&to_write, next_to_clean + 1,
+ true, ioq, wb_context);
+ Assert(sync_state != BUF_CANT_MERGE);
+ }
if (++next_to_clean >= NBuffers)
{
@@ -3849,6 +3976,13 @@ BgBufferSync(WritebackContext *wb_context)
if (sync_state & BUF_WRITTEN)
{
+ Assert(sync_state & BUF_REUSABLE);
+
+ if (to_write.nbuffers == max_combine)
+ {
+ WriteBuffers(&to_write, ioq, wb_context);
+ }
+
reusable_buffers++;
if (++num_written >= bgwriter_lru_maxpages)
{
@@ -3860,6 +3994,11 @@ BgBufferSync(WritebackContext *wb_context)
reusable_buffers++;
}
+ if (to_write.nbuffers > 0)
+ WriteBuffers(&to_write, ioq, wb_context);
+
+ BuffersToWriteEnd(&to_write);
+
PendingBgWriterStats.buf_written_clean += num_written;
#ifdef BGW_DEBUG
@@ -3898,8 +4037,66 @@ BgBufferSync(WritebackContext *wb_context)
return (bufs_to_lap == 0 && recent_alloc == 0);
}
+static inline bool
+BufferTagsSameRel(const BufferTag *tag1, const BufferTag *tag2)
+{
+ return (tag1->spcOid == tag2->spcOid) &&
+ (tag1->dbOid == tag2->dbOid) &&
+ (tag1->relNumber == tag2->relNumber) &&
+ (tag1->forkNum == tag2->forkNum)
+ ;
+}
+
+static bool
+CanMergeWrite(BuffersToWrite *to_write, BufferDesc *cur_buf_hdr)
+{
+ BlockNumber cur_block = cur_buf_hdr->tag.blockNum;
+
+ Assert(to_write->nbuffers > 0); /* can't merge with nothing */
+ Assert(to_write->start_at_tag.relNumber != InvalidOid);
+ Assert(to_write->start_at_tag.blockNum != InvalidBlockNumber);
+
+ Assert(to_write->ioh != NULL);
+
+ /*
+ * First check if the blocknumber is one that we could actually merge,
+ * that's cheaper than checking the tablespace/db/relnumber/fork match.
+ */
+ if (to_write->start_at_tag.blockNum + to_write->nbuffers != cur_block)
+ return false;
+
+ if (!BufferTagsSameRel(&to_write->start_at_tag, &cur_buf_hdr->tag))
+ return false;
+
+ /*
+ * Need to check with smgr how large a write we're allowed to make. To
+ * reduce the overhead of the smgr check, only inquire once, when
+ * processing the first to-be-merged buffer. That avoids the overhead in
+ * the common case of writing out buffers that definitely not mergeable.
+ */
+ if (to_write->nbuffers == 1)
+ {
+ SMgrRelation smgr;
+
+ smgr = smgropen(BufTagGetRelFileLocator(&to_write->start_at_tag), INVALID_PROC_NUMBER);
+
+ to_write->max_combine = smgrmaxcombine(smgr,
+ to_write->start_at_tag.forkNum,
+ to_write->start_at_tag.blockNum);
+ }
+ else
+ {
+ Assert(to_write->max_combine > 0);
+ }
+
+ if (to_write->start_at_tag.blockNum + to_write->max_combine <= cur_block)
+ return false;
+
+ return true;
+}
+
/*
- * SyncOneBuffer -- process a single buffer during syncing.
+ * PrepareToWriteBuffer -- process a single buffer during syncing.
*
* If skip_recently_used is true, we don't write currently-pinned buffers, nor
* buffers marked recently used, as these are not replacement candidates.
@@ -3908,22 +4105,50 @@ BgBufferSync(WritebackContext *wb_context)
* BUF_WRITTEN: we wrote the buffer.
* BUF_REUSABLE: buffer is available for replacement, ie, it has
* pin count 0 and usage count 0.
+ * BUF_CANT_MERGE: can't combine this write with prior writes, caller needs
+ * to issue those first
*
* (BUF_WRITTEN could be set in error if FlushBuffer finds the buffer clean
* after locking it, but we don't care all that much.)
*/
static int
-SyncOneBuffer(int buf_id, bool skip_recently_used, WritebackContext *wb_context)
+PrepareToWriteBuffer(BuffersToWrite *to_write, Buffer buf,
+ bool skip_recently_used,
+ IOQueue *ioq, WritebackContext *wb_context)
{
- BufferDesc *bufHdr = GetBufferDescriptor(buf_id);
+ BufferDesc *cur_buf_hdr = GetBufferDescriptor(buf - 1);
+ uint32 buf_state;
int result = 0;
- uint32 buf_state;
- BufferTag tag;
+ XLogRecPtr cur_buf_lsn;
+ LWLock *content_lock;
+ bool may_block;
+
+ /*
+ * Check if this buffer can be written out together with already prepared
+ * writes. We check before we have pinned the buffer, so the buffer can be
+ * written out and replaced between this check and us pinning the buffer -
+ * we'll recheck below. The reason for the pre-check is that we don't want
+ * to pin the buffer just to find out that we can't merge the IO.
+ */
+ if (to_write->nbuffers != 0)
+ {
+ if (!CanMergeWrite(to_write, cur_buf_hdr))
+ {
+ result |= BUF_CANT_MERGE;
+ return result;
+ }
+ }
+ else
+ {
+ to_write->start_at_tag = cur_buf_hdr->tag;
+ }
/* Make sure we can handle the pin */
ReservePrivateRefCountEntry();
ResourceOwnerEnlarge(CurrentResourceOwner);
+ /* XXX: Should also check if we are allowed to pin one more buffer */
+
/*
* Check whether buffer needs writing.
*
@@ -3933,7 +4158,7 @@ SyncOneBuffer(int buf_id, bool skip_recently_used, WritebackContext *wb_context)
* don't worry because our checkpoint.redo points before log record for
* upcoming changes and so we are not required to write such dirty buffer.
*/
- buf_state = LockBufHdr(bufHdr);
+ buf_state = LockBufHdr(cur_buf_hdr);
if (BUF_STATE_GET_REFCOUNT(buf_state) == 0 &&
BUF_STATE_GET_USAGECOUNT(buf_state) == 0)
@@ -3942,40 +4167,300 @@ SyncOneBuffer(int buf_id, bool skip_recently_used, WritebackContext *wb_context)
}
else if (skip_recently_used)
{
+#if 0
+ elog(LOG, "at block %d: skip recent with nbuffers %d",
+ cur_buf_hdr->tag.blockNum, to_write->nbuffers);
+#endif
/* Caller told us not to write recently-used buffers */
- UnlockBufHdr(bufHdr, buf_state);
+ UnlockBufHdr(cur_buf_hdr, buf_state);
return result;
}
if (!(buf_state & BM_VALID) || !(buf_state & BM_DIRTY))
{
/* It's clean, so nothing to do */
- UnlockBufHdr(bufHdr, buf_state);
+ UnlockBufHdr(cur_buf_hdr, buf_state);
return result;
}
+ /* pin the buffer, from now on its identity can't change anymore */
+ PinBuffer_Locked(cur_buf_hdr);
+
+ /*
+ * Acquire IO, if needed, now that it's likely that we'll need to write.
+ */
+ if (to_write->ioh == NULL)
+ {
+ /* otherwise we should already have acquired a handle */
+ Assert(to_write->nbuffers == 0);
+
+ to_write->ioh = io_queue_acquire_io(ioq);
+ pgaio_io_get_wref(to_write->ioh, &to_write->iow);
+ }
+
+ /*
+ * If we are merging, check if the buffer's identity possibly changed
+ * while we hadn't yet pinned it.
+ *
+ * XXX: It might be worth checking if we still want to write the buffer
+ * out, e.g. it could have been replaced with a buffer that doesn't have
+ * BM_CHECKPOINT_NEEDED set.
+ */
+ if (to_write->nbuffers != 0)
+ {
+ if (!CanMergeWrite(to_write, cur_buf_hdr))
+ {
+ elog(LOG, "changed identity");
+ UnpinBuffer(cur_buf_hdr);
+
+ result |= BUF_CANT_MERGE;
+
+ return result;
+ }
+ }
+
+ may_block = to_write->nbuffers == 0
+ && !pgaio_have_staged()
+ && io_queue_is_empty(ioq)
+ ;
+ content_lock = BufferDescriptorGetContentLock(cur_buf_hdr);
+
+ if (!may_block)
+ {
+ if (LWLockConditionalAcquire(content_lock, LW_SHARED))
+ {
+ /* done */
+ }
+ else if (to_write->nbuffers == 0)
+ {
+ /*
+ * Need to wait for all prior IO to finish before blocking for
+ * lock acquisition, to avoid the risk a deadlock due to us
+ * waiting for another backend that is waiting for our unsubmitted
+ * IO to complete.
+ */
+ pgaio_submit_staged();
+ io_queue_wait_all(ioq);
+
+ elog(DEBUG2, "at block %u: can't block, nbuffers = 0",
+ cur_buf_hdr->tag.blockNum
+ );
+
+ may_block = to_write->nbuffers == 0
+ && !pgaio_have_staged()
+ && io_queue_is_empty(ioq)
+ ;
+ Assert(may_block);
+
+ LWLockAcquire(content_lock, LW_SHARED);
+ }
+ else
+ {
+ elog(DEBUG2, "at block %d: can't block nbuffers = %d",
+ cur_buf_hdr->tag.blockNum,
+ to_write->nbuffers);
+
+ UnpinBuffer(cur_buf_hdr);
+ result |= BUF_CANT_MERGE;
+ Assert(to_write->nbuffers > 0);
+
+ return result;
+ }
+ }
+ else
+ {
+ LWLockAcquire(content_lock, LW_SHARED);
+ }
+
+ if (!may_block)
+ {
+ if (!StartBufferIO(cur_buf_hdr, false, !may_block))
+ {
+ pgaio_submit_staged();
+ io_queue_wait_all(ioq);
+
+ may_block = io_queue_is_empty(ioq) && to_write->nbuffers == 0 && !pgaio_have_staged();
+
+ if (!StartBufferIO(cur_buf_hdr, false, !may_block))
+ {
+ elog(DEBUG2, "at block %d: non-waitable StartBufferIO returns false, %d",
+ cur_buf_hdr->tag.blockNum,
+ may_block);
+
+ /*
+ * FIXME: can't tell whether this is because the buffer has
+ * been cleaned
+ */
+ if (!may_block)
+ {
+ result |= BUF_CANT_MERGE;
+ Assert(to_write->nbuffers > 0);
+ }
+ LWLockRelease(content_lock);
+ UnpinBuffer(cur_buf_hdr);
+
+ return result;
+ }
+ }
+ }
+ else
+ {
+ if (!StartBufferIO(cur_buf_hdr, false, false))
+ {
+ elog(DEBUG2, "waitable StartBufferIO returns false");
+ LWLockRelease(content_lock);
+ UnpinBuffer(cur_buf_hdr);
+
+ /*
+ * FIXME: Historically we returned BUF_WRITTEN in this case, which
+ * seems wrong
+ */
+ return result;
+ }
+ }
+
/*
- * Pin it, share-lock it, write it. (FlushBuffer will do nothing if the
- * buffer is clean by the time we've locked it.)
+ * Run PageGetLSN while holding header lock, since we don't have the
+ * buffer locked exclusively in all cases.
*/
- PinBuffer_Locked(bufHdr);
- LWLockAcquire(BufferDescriptorGetContentLock(bufHdr), LW_SHARED);
+ buf_state = LockBufHdr(cur_buf_hdr);
+
+ cur_buf_lsn = BufferGetLSN(cur_buf_hdr);
- FlushBuffer(bufHdr, NULL, IOOBJECT_RELATION, IOCONTEXT_NORMAL);
+ /* To check if block content changes while flushing. - vadim 01/17/97 */
+ buf_state &= ~BM_JUST_DIRTIED;
- LWLockRelease(BufferDescriptorGetContentLock(bufHdr));
+ UnlockBufHdr(cur_buf_hdr, buf_state);
- tag = bufHdr->tag;
+ to_write->buffers[to_write->nbuffers] = buf;
+ to_write->nbuffers++;
- UnpinBuffer(bufHdr);
+ if (buf_state & BM_PERMANENT &&
+ (to_write->max_lsn == InvalidXLogRecPtr || to_write->max_lsn < cur_buf_lsn))
+ {
+ to_write->max_lsn = cur_buf_lsn;
+ }
+
+ result |= BUF_WRITTEN;
+
+ return result;
+}
+
+static void
+WriteBuffers(BuffersToWrite *to_write,
+ IOQueue *ioq, WritebackContext *wb_context)
+{
+ SMgrRelation smgr;
+ Buffer first_buf;
+ BufferDesc *first_buf_hdr;
+ bool needs_checksum;
+
+ Assert(to_write->nbuffers > 0 && to_write->nbuffers <= io_combine_limit);
+
+ first_buf = to_write->buffers[0];
+ first_buf_hdr = GetBufferDescriptor(first_buf - 1);
+
+ smgr = smgropen(BufTagGetRelFileLocator(&first_buf_hdr->tag), INVALID_PROC_NUMBER);
/*
- * SyncOneBuffer() is only called by checkpointer and bgwriter, so
- * IOContext will always be IOCONTEXT_NORMAL.
+ * Force XLOG flush up to buffer's LSN. This implements the basic WAL
+ * rule that log updates must hit disk before any of the data-file changes
+ * they describe do.
+ *
+ * However, this rule does not apply to unlogged relations, which will be
+ * lost after a crash anyway. Most unlogged relation pages do not bear
+ * LSNs since we never emit WAL records for them, and therefore flushing
+ * up through the buffer LSN would be useless, but harmless. However,
+ * GiST indexes use LSNs internally to track page-splits, and therefore
+ * unlogged GiST pages bear "fake" LSNs generated by
+ * GetFakeLSNForUnloggedRel. It is unlikely but possible that the fake
+ * LSN counter could advance past the WAL insertion point; and if it did
+ * happen, attempting to flush WAL through that location would fail, with
+ * disastrous system-wide consequences. To make sure that can't happen,
+ * skip the flush if the buffer isn't permanent.
*/
- ScheduleBufferTagForWriteback(wb_context, IOCONTEXT_NORMAL, &tag);
+ if (to_write->max_lsn != InvalidXLogRecPtr)
+ XLogFlush(to_write->max_lsn);
+
+ /*
+ * Now it's safe to write the buffer to disk. Note that no one else should
+ * have been able to write it, while we were busy with log flushing,
+ * because we got the exclusive right to perform I/O by setting the
+ * BM_IO_IN_PROGRESS bit.
+ */
+
+ for (int nbuf = 0; nbuf < to_write->nbuffers; nbuf++)
+ {
+ Buffer cur_buf = to_write->buffers[nbuf];
+ BufferDesc *cur_buf_hdr = GetBufferDescriptor(cur_buf - 1);
+ Block bufBlock;
+ char *bufToWrite;
+
+ bufBlock = BufHdrGetBlock(cur_buf_hdr);
+ needs_checksum = PageNeedsChecksumCopy((Page) bufBlock);
+
+ /*
+ * Update page checksum if desired. Since we have only shared lock on
+ * the buffer, other processes might be updating hint bits in it, so
+ * we must copy the page to a bounce buffer if we do checksumming.
+ */
+ if (needs_checksum)
+ {
+ PgAioBounceBuffer *bb = pgaio_bounce_buffer_get();
- return result | BUF_WRITTEN;
+ pgaio_io_assoc_bounce_buffer(to_write->ioh, bb);
+
+ bufToWrite = pgaio_bounce_buffer_buffer(bb);
+ memcpy(bufToWrite, bufBlock, BLCKSZ);
+ PageSetChecksumInplace((Page) bufToWrite, cur_buf_hdr->tag.blockNum);
+ }
+ else
+ {
+ bufToWrite = bufBlock;
+ }
+
+ to_write->data_ptrs[nbuf] = bufToWrite;
+ }
+
+ pgaio_io_set_handle_data_32(to_write->ioh,
+ (uint32 *) to_write->buffers,
+ to_write->nbuffers);
+ pgaio_io_register_callbacks(to_write->ioh, PGAIO_HCB_SHARED_BUFFER_WRITEV, 0);
+
+ smgrstartwritev(to_write->ioh, smgr,
+ BufTagGetForkNum(&first_buf_hdr->tag),
+ first_buf_hdr->tag.blockNum,
+ to_write->data_ptrs,
+ to_write->nbuffers,
+ false);
+ pgstat_count_io_op(IOOBJECT_RELATION, IOCONTEXT_NORMAL,
+ IOOP_WRITE, 1, BLCKSZ * to_write->nbuffers);
+
+
+ for (int nbuf = 0; nbuf < to_write->nbuffers; nbuf++)
+ {
+ Buffer cur_buf = to_write->buffers[nbuf];
+ BufferDesc *cur_buf_hdr = GetBufferDescriptor(cur_buf - 1);
+
+ UnpinBuffer(cur_buf_hdr);
+ }
+
+ io_queue_track(ioq, &to_write->iow);
+ to_write->total_writes++;
+
+ /* clear state for next write */
+ to_write->nbuffers = 0;
+ to_write->start_at_tag.relNumber = InvalidOid;
+ to_write->start_at_tag.blockNum = InvalidBlockNumber;
+ to_write->max_combine = 0;
+ to_write->max_lsn = InvalidXLogRecPtr;
+ to_write->ioh = NULL;
+ pgaio_wref_clear(&to_write->iow);
+
+ /*
+ * FIXME: Implement issuing writebacks (note wb_context isn't used here).
+ * Possibly needs to be integrated with io_queue.c.
+ */
}
/*
@@ -4349,6 +4834,7 @@ FlushBuffer(BufferDesc *buf, SMgrRelation reln, IOObject io_object,
error_context_stack = errcallback.previous;
}
+
/*
* RelationGetNumberOfBlocksInFork
* Determines the current number of pages in the specified relation fork.
diff --git a/src/backend/storage/page/bufpage.c b/src/backend/storage/page/bufpage.c
index 652746e1fa2..1e1a6641a4d 100644
--- a/src/backend/storage/page/bufpage.c
+++ b/src/backend/storage/page/bufpage.c
@@ -1490,6 +1490,16 @@ PageIndexTupleOverwrite(Page page, OffsetNumber offnum,
return true;
}
+bool
+PageNeedsChecksumCopy(Page page)
+{
+ if (PageIsNew(page))
+ return false;
+
+ /* If we don't need a checksum, just return the passed-in data */
+ return DataChecksumsEnabled();
+}
+
/*
* Set checksum for a page in shared buffers.
diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list
index 54239ccc69e..53093e2a1db 100644
--- a/src/tools/pgindent/typedefs.list
+++ b/src/tools/pgindent/typedefs.list
@@ -348,6 +348,7 @@ BufferManagerRelation
BufferStrategyControl
BufferTag
BufferUsage
+BuffersToWrite
BuildAccumulator
BuiltinScript
BulkInsertState
--
2.48.1.76.g4e746b1a31.dirty
[text/x-diff] v2.14-0025-Ensure-a-resowner-exists-for-all-paths-that-ma.patch (2.6K, ../../d4ff3qxudxn2ze3zk2xwhds5ysezfei3uqeuimymj624t5chsd@sgr2h3rmln5t/26-v2.14-0025-Ensure-a-resowner-exists-for-all-paths-that-ma.patch)
download | inline diff:
From 77d6e1f4521a78f959d7c136f24f0025266a37d8 Mon Sep 17 00:00:00 2001
From: Andres Freund <andres@anarazel.de>
Date: Tue, 18 Mar 2025 14:40:06 -0400
Subject: [PATCH v2.14 25/29] Ensure a resowner exists for all paths that may
perform AIO
Reviewed-by: Noah Misch <noah@leadboat.com>
Discussion: https://postgr.es/m/1f6b50a7-38ef-4d87-8246-786d39f46ab9@iki.fi
---
src/backend/bootstrap/bootstrap.c | 7 +++++++
src/backend/replication/logical/logical.c | 6 ++++++
src/backend/utils/init/postinit.c | 6 +++++-
3 files changed, 18 insertions(+), 1 deletion(-)
diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c
index 6db864892d0..e554504e1f0 100644
--- a/src/backend/bootstrap/bootstrap.c
+++ b/src/backend/bootstrap/bootstrap.c
@@ -361,8 +361,15 @@ BootstrapModeMain(int argc, char *argv[], bool check_only)
BaseInit();
bootstrap_signals();
+
+ /* need a resowner for IO during BootStrapXLOG() */
+ CreateAuxProcessResourceOwner();
+
BootStrapXLOG(bootstrap_data_checksum_version);
+ ReleaseAuxProcessResources(true);
+ CurrentResourceOwner = NULL;
+
/*
* To ensure that src/common/link-canary.c is linked into the backend, we
* must call it from somewhere. Here is as good as anywhere.
diff --git a/src/backend/replication/logical/logical.c b/src/backend/replication/logical/logical.c
index a8d2e024d34..72c32f5c32e 100644
--- a/src/backend/replication/logical/logical.c
+++ b/src/backend/replication/logical/logical.c
@@ -386,6 +386,12 @@ CreateInitDecodingContext(const char *plugin,
slot->data.plugin = plugin_name;
SpinLockRelease(&slot->mutex);
+ if (CurrentResourceOwner == NULL)
+ {
+ Assert(am_walsender);
+ CurrentResourceOwner = AuxProcessResourceOwner;
+ }
+
if (XLogRecPtrIsInvalid(restart_lsn))
ReplicationSlotReserveWal();
else
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index 7958ea11b73..222e24bcb08 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -785,8 +785,12 @@ InitPostgres(const char *in_dbname, Oid dboid,
* We don't yet have an aux-process resource owner, but StartupXLOG
* and ShutdownXLOG will need one. Hence, create said resource owner
* (and register a callback to clean it up after ShutdownXLOG runs).
+ *
+ * In bootstrap mode CreateAuxProcessResourceOwner() was already
+ * called in BootstrapModeMain().
*/
- CreateAuxProcessResourceOwner();
+ if (!bootstrap)
+ CreateAuxProcessResourceOwner();
StartupXLOG();
/* Release (and warn about) any buffer pins leaked in StartupXLOG */
--
2.48.1.76.g4e746b1a31.dirty
[text/x-diff] v2.14-0026-Temporary-Increase-BAS_BULKREAD-size.patch (1.3K, ../../d4ff3qxudxn2ze3zk2xwhds5ysezfei3uqeuimymj624t5chsd@sgr2h3rmln5t/27-v2.14-0026-Temporary-Increase-BAS_BULKREAD-size.patch)
download | inline diff:
From 6b4ff088406f58cf40ddf3b573a734cfaf1ee8e7 Mon Sep 17 00:00:00 2001
From: Andres Freund <andres@anarazel.de>
Date: Tue, 18 Mar 2025 14:40:06 -0400
Subject: [PATCH v2.14 26/29] Temporary: Increase BAS_BULKREAD size
Without this we only can execute very little AIO for sequential scans, as
there's just not enough buffers in the ring. This isn't the right fix, as
just increasing the ring size can have negative performance implications in
workloads where the kernel has all the data cached.
Author:
Reviewed-By:
Discussion: https://postgr.es/m/
Backpatch:
---
src/backend/storage/buffer/freelist.c | 7 ++++++-
1 file changed, 6 insertions(+), 1 deletion(-)
diff --git a/src/backend/storage/buffer/freelist.c b/src/backend/storage/buffer/freelist.c
index 336715b6c63..b72a5957a20 100644
--- a/src/backend/storage/buffer/freelist.c
+++ b/src/backend/storage/buffer/freelist.c
@@ -555,7 +555,12 @@ GetAccessStrategy(BufferAccessStrategyType btype)
return NULL;
case BAS_BULKREAD:
- ring_size_kb = 256;
+
+ /*
+ * FIXME: Temporary increase to allow large enough streaming reads
+ * to actually benefit from AIO. This needs a better solution.
+ */
+ ring_size_kb = 2 * 1024;
break;
case BAS_BULKWRITE:
ring_size_kb = 16 * 1024;
--
2.48.1.76.g4e746b1a31.dirty
[text/x-diff] v2.14-0027-WIP-Use-MAP_POPULATE.patch (1.1K, ../../d4ff3qxudxn2ze3zk2xwhds5ysezfei3uqeuimymj624t5chsd@sgr2h3rmln5t/28-v2.14-0027-WIP-Use-MAP_POPULATE.patch)
download | inline diff:
From 510267ae9283f0943dc675262dbe0eca4d78bc0f Mon Sep 17 00:00:00 2001
From: Andres Freund <andres@anarazel.de>
Date: Tue, 18 Mar 2025 14:40:06 -0400
Subject: [PATCH v2.14 27/29] WIP: Use MAP_POPULATE
For benchmarking it's quite annoying that the first time a memory is touched
has completely different perf characteristics than subsequent accesses. Using
MAP_POPULATE reduces that substantially.
---
src/backend/port/sysv_shmem.c | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/src/backend/port/sysv_shmem.c b/src/backend/port/sysv_shmem.c
index 197926d44f6..a700b02d5a1 100644
--- a/src/backend/port/sysv_shmem.c
+++ b/src/backend/port/sysv_shmem.c
@@ -620,7 +620,7 @@ CreateAnonymousSegment(Size *size)
allocsize += hugepagesize - (allocsize % hugepagesize);
ptr = mmap(NULL, allocsize, PROT_READ | PROT_WRITE,
- PG_MMAP_FLAGS | mmap_flags, -1, 0);
+ PG_MMAP_FLAGS | MAP_POPULATE | mmap_flags, -1, 0);
mmap_errno = errno;
if (huge_pages == HUGE_PAGES_TRY && ptr == MAP_FAILED)
elog(DEBUG1, "mmap(%zu) with MAP_HUGETLB failed, huge pages disabled: %m",
--
2.48.1.76.g4e746b1a31.dirty
^ permalink raw reply [nested|flat] 153+ messages in thread
* Re: AIO v2.5
2025-03-04 19:00 AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-10 18:23 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-11 19:41 ` Re: AIO v2.5 Noah Misch <noah@leadboat.com>
2025-03-11 23:55 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-12 03:57 ` Re: AIO v2.5 Noah Misch <noah@leadboat.com>
2025-03-12 17:06 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-12 23:23 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-13 10:53 ` Re: AIO v2.5 Antonin Houska <ah@cybertec.at>
2025-03-14 19:43 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-17 20:00 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-18 20:12 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-21 01:58 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-25 01:18 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-27 01:07 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-27 20:58 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-28 03:22 ` Re: AIO v2.5 Noah Misch <noah@leadboat.com>
2025-03-28 12:54 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-29 03:35 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-29 13:41 ` Re: AIO v2.5 Noah Misch <noah@leadboat.com>
2025-03-29 14:48 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
@ 2025-03-29 18:25 ` Andres Freund <andres@anarazel.de>
2025-03-29 20:16 ` Re: AIO v2.5 Melanie Plageman <melanieplageman@gmail.com>
2025-03-29 21:29 ` Re: AIO v2.5 Noah Misch <noah@leadboat.com>
0 siblings, 2 replies; 153+ messages in thread
From: Andres Freund @ 2025-03-29 18:25 UTC (permalink / raw)
To: Noah Misch <noah@leadboat.com>; +Cc: pgsql-hackers@postgresql.org, Thomas Munro <thomas.munro@gmail.com>; Heikki Linnakangas <hlinnaka@iki.fi>; Robert Haas <robertmhaas@gmail.com>; Jakub Wartak <jakub.wartak@enterprisedb.com>; Jelte Fennema-Nio <postgres@jeltef.nl>; Antonin Houska <ah@cybertec.at>
Hi,
On 2025-03-29 10:48:10 -0400, Andres Freund wrote:
> Attached is v2.14:
FWIW, there was a last minute change in the test that fails in one task on CI,
due to reading across the smaller segment size configured for one of the
runs. Doesn't quite seem worth posting a new version for.
> - push the checksums stats fix
Done.
> - unless somebody sees a reason to not use LOG_SERVER_ONLY in
> "aio: Implement support for reads in smgr/md/fd", push that
>
> Besides that the only change since Noah's last review of that commit is an
> added comment.
Also done. If we want to change log level later, it's easy to do so.
I made some small changes since the version I had posted:
- I found one dangling reference to mdread() instead of mdreadv()
- I had accidentally squashed the fix to Noah's review comment about a comment
above md_readv_report() to the wrong commit (smgr/md/fd.c write support)
- PGAIO_HCB_MD_WRITEV was added in "smgr/md/fd.c read support" instead of
"smgr/md/fd.c write support"
> - push pg_aios view (depends a tiny bit on the smgr/md/fd change above)
I think I found an issue with this one - as it stands the view was viewable by
everyone. While it doesn't provide a *lot* of insight, it still seems a bit
too much for an unprivileged user to learn what part of a relation any other
user is currently reading.
There'd be two different ways to address that:
1) revoke view & function from public, grant to a limited role (presumably
pg_read_all_stats)
2) copy pg_stat_activity's approach of using something like
#define HAS_PGSTAT_PERMISSIONS(role) (has_privs_of_role(GetUserId(), ROLE_PG_READ_ALL_STATS) || has_privs_of_role(GetUserId(), role))
on a per-IO basis.
Greetings,
Andres Freund
^ permalink raw reply [nested|flat] 153+ messages in thread
* Re: AIO v2.5
2025-03-04 19:00 AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-10 18:23 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-11 19:41 ` Re: AIO v2.5 Noah Misch <noah@leadboat.com>
2025-03-11 23:55 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-12 03:57 ` Re: AIO v2.5 Noah Misch <noah@leadboat.com>
2025-03-12 17:06 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-12 23:23 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-13 10:53 ` Re: AIO v2.5 Antonin Houska <ah@cybertec.at>
2025-03-14 19:43 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-17 20:00 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-18 20:12 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-21 01:58 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-25 01:18 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-27 01:07 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-27 20:58 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-28 03:22 ` Re: AIO v2.5 Noah Misch <noah@leadboat.com>
2025-03-28 12:54 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-29 03:35 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-29 13:41 ` Re: AIO v2.5 Noah Misch <noah@leadboat.com>
2025-03-29 14:48 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-29 18:25 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
@ 2025-03-29 20:16 ` Melanie Plageman <melanieplageman@gmail.com>
1 sibling, 0 replies; 153+ messages in thread
From: Melanie Plageman @ 2025-03-29 20:16 UTC (permalink / raw)
To: Andres Freund <andres@anarazel.de>; +Cc: Noah Misch <noah@leadboat.com>; pgsql-hackers@postgresql.org, Thomas Munro <thomas.munro@gmail.com>; Heikki Linnakangas <hlinnaka@iki.fi>; Robert Haas <robertmhaas@gmail.com>; Jakub Wartak <jakub.wartak@enterprisedb.com>; Jelte Fennema-Nio <postgres@jeltef.nl>; Antonin Houska <ah@cybertec.at>
On Sat, Mar 29, 2025 at 2:25 PM Andres Freund <andres@anarazel.de> wrote:
>
> I think I found an issue with this one - as it stands the view was viewable by
> everyone. While it doesn't provide a *lot* of insight, it still seems a bit
> too much for an unprivileged user to learn what part of a relation any other
> user is currently reading.
>
> There'd be two different ways to address that:
> 1) revoke view & function from public, grant to a limited role (presumably
> pg_read_all_stats)
> 2) copy pg_stat_activity's approach of using something like
>
> #define HAS_PGSTAT_PERMISSIONS(role) (has_privs_of_role(GetUserId(), ROLE_PG_READ_ALL_STATS) || has_privs_of_role(GetUserId(), role))
>
> on a per-IO basis.
Is it easier to later change it to be more restrictive or less? If it
is easier to later lock it down more, then go with 2, otherwise go
with 1?
- Melanie
^ permalink raw reply [nested|flat] 153+ messages in thread
* Re: AIO v2.5
2025-03-04 19:00 AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-10 18:23 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-11 19:41 ` Re: AIO v2.5 Noah Misch <noah@leadboat.com>
2025-03-11 23:55 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-12 03:57 ` Re: AIO v2.5 Noah Misch <noah@leadboat.com>
2025-03-12 17:06 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-12 23:23 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-13 10:53 ` Re: AIO v2.5 Antonin Houska <ah@cybertec.at>
2025-03-14 19:43 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-17 20:00 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-18 20:12 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-21 01:58 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-25 01:18 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-27 01:07 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-27 20:58 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-28 03:22 ` Re: AIO v2.5 Noah Misch <noah@leadboat.com>
2025-03-28 12:54 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-29 03:35 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-29 13:41 ` Re: AIO v2.5 Noah Misch <noah@leadboat.com>
2025-03-29 14:48 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-29 18:25 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
@ 2025-03-29 21:29 ` Noah Misch <noah@leadboat.com>
2025-03-30 00:39 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-30 23:46 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
1 sibling, 2 replies; 153+ messages in thread
From: Noah Misch @ 2025-03-29 21:29 UTC (permalink / raw)
To: Andres Freund <andres@anarazel.de>; +Cc: pgsql-hackers@postgresql.org, Thomas Munro <thomas.munro@gmail.com>; Heikki Linnakangas <hlinnaka@iki.fi>; Robert Haas <robertmhaas@gmail.com>; Jakub Wartak <jakub.wartak@enterprisedb.com>; Jelte Fennema-Nio <postgres@jeltef.nl>; Antonin Houska <ah@cybertec.at>
Flushing half-baked review comments before going offline for a few hours:
On Wed, Mar 26, 2025 at 09:07:40PM -0400, Andres Freund wrote:
> Attached v2.13, with the following changes:
> 5) The WARNING in the callback is now a LOG, as it will be sent to the
> client as a WARNING explicitly when the IO's results are processed
>
> I actually chose LOG_SERVER_ONLY - that seemed slightly better than just
> LOG? But not at all sure.
LOG_SERVER_ONLY and its synonym COMMERR look to be used for:
- ProcessLogMemoryContextInterrupt()
- messages before successful authentication
- protocol sync loss, where we'd fail to send a client message
- client already gone
The choice between LOG and LOG_SERVER_ONLY doesn't matter much for $SUBJECT.
If a client has decided to set client_min_messages that high, the client might
be interested in the fact that it got side-tracked completing someone else's
IO. On the other hand, almost none of those sidetrack events will produce
messages. The main argument I'd envision for LOG_SERVER_ONLY is that we
consider the message content sensitive, but I don't see the message content as
materially sensitive.
Since you committed LOG_SERVER_ONLY, let's keep that decision. My last draft
review discouraged it, but it doesn't matter. pgaio_result_report() should
assert elevel != LOG to avoid future divergence.
> - Previously the buffer completion callback checked zero_damaged_pages - but
> that's not right, the GUC hopefully is only set on a per-session basis
Good catch. I've now audited the complete_shared callbacks for other variable
references and actions not acceptable there. I found nothing beyond what you
found by v2.14.
On Sat, Mar 29, 2025 at 10:48:10AM -0400, Andres Freund wrote:
> On 2025-03-29 06:41:43 -0700, Noah Misch wrote:
> > On Fri, Mar 28, 2025 at 11:35:23PM -0400, Andres Freund wrote:
> Subject: [PATCH v2.14 01/29] Fix mis-attribution of checksum failure stats to
> the wrong database
I've skipped reviewing this patch, since it's already committed. If it needs
post-commit review, let me know.
> Subject: [PATCH v2.14 02/29] aio: Implement support for reads in smgr/md/fd
> + /*
> + * Immediately log a message about the IO error, but only to the
> + * server log. The reason to do so immediately is that the originator
> + * might not process the query result immediately (because it is busy
> + * doing another part of query processing) or at all (e.g. if it was
> + * cancelled or errored out due to another IO also failing). The
> + * issuer of the IO will emit an ERROR when processing the IO's
s/issuer/definer/ please, to avoid proliferating synonyms. Likewise two other
places in the patches.
> +/*
> + * smgrstartreadv() -- asynchronous version of smgrreadv()
> + *
> + * This starts an asynchronous readv IO using the IO handle `ioh`. Other than
> + * `ioh` all parameters are the same as smgrreadv().
I would add a comment starting with:
Compared to smgrreadv(), more responsibilities fall on layers above smgr.
Higher layers handle partial reads. smgr will ereport(LOG_SERVER_ONLY) some
problems, but higher layers are responsible for pgaio_result_report() to
mirror that news to the user and (for ERROR) abort the (sub)transaction.
md_readv_complete() comment "issuer of the IO will emit an ERROR" says some of
that, but someone adding a smgrstartreadv() call is less likely to find it
there.
I say "comment starting with", because I think there's a remaining decision
about who owns the zeroing currently tied to smgrreadv(). An audit of
mdreadv() vs. AIO counterparts found this part of mdreadv():
if (nbytes == 0)
{
/*
* We are at or past EOF, or we read a partial block at EOF.
* Normally this is an error; upper levels should never try to
* read a nonexistent block. However, if zero_damaged_pages
* is ON or we are InRecovery, we should instead return zeroes
* without complaining. This allows, for example, the case of
* trying to update a block that was later truncated away.
*/
if (zero_damaged_pages || InRecovery)
{
I didn't write a test to prove its absence, but I'm not finding such code in
the AIO path. I wondered if we could just Assert(!InRecovery), but adding
that to md_readv_complete() failed 001_stream_rep.pl with this stack:
ExceptionalCondition at assert.c:52
md_readv_complete at md.c:2043
pgaio_io_call_complete_shared at aio_callback.c:258
pgaio_io_process_completion at aio.c:515
pgaio_io_perform_synchronously at aio_io.c:148
pgaio_io_stage at aio.c:453
pgaio_io_start_readv at aio_io.c:87
FileStartReadV at fd.c:2243
mdstartreadv at md.c:1005
smgrstartreadv at smgr.c:757
AsyncReadBuffers at bufmgr.c:1938
StartReadBuffersImpl at bufmgr.c:1422
StartReadBuffer at bufmgr.c:1515
ReadBuffer_common at bufmgr.c:1246
ReadBufferExtended at bufmgr.c:818
vm_readbuf at visibilitymap.c:584
visibilitymap_pin at visibilitymap.c:203
heap_xlog_insert at heapam_xlog.c:450
heap_redo at heapam_xlog.c:1195
ApplyWalRecord at xlogrecovery.c:1995
PerformWalRecovery at xlogrecovery.c:1825
StartupXLOG at xlog.c:5895
If this is a real problem, fix options may include:
- Implement the InRecovery zeroing for real.
- Make the InRecovery case somehow use real mdreadv(), not
pgaio_io_perform_synchronously() to use AIO APIs with synchronous AIO. I'll
guess this is harder than the previous option, though.
> Subject: [PATCH v2.14 04/29] aio: Add pg_aios view
> + /*
> + * There is no lock that could prevent the state of the IO to advance
> + * concurrently - and we don't want to introduce one, as that would
> + * introduce atomics into a very common path. Instead we
> + *
> + * 1) Determine the state + generation of the IO.
> + *
> + * 2) Copy the IO to local memory.
> + *
> + * 3) Check if state or generation of the IO changed. If the state
> + * changed, retry, if the generation changed don't display the IO.
> + */
> +
> + /* 1) from above */
> + start_generation = live_ioh->generation;
> + pg_read_barrier();
I think "retry:" needs to be here, above start_state assignment. Otherwise,
the "live_ioh->state != start_state" test will keep seeing a state mismatch.
> + start_state = live_ioh->state;
> +
> +retry:
> + if (start_state == PGAIO_HS_IDLE)
> + continue;
> Subject: [PATCH v2.14 05/29] localbuf: Track pincount in BufferDesc as well
> Subject: [PATCH v2.14 07/29] aio: Add WARNING result status
> Subject: [PATCH v2.14 08/29] pgstat: Allow checksum errors to be reported in
> critical sections
> Subject: [PATCH v2.14 09/29] Add errhint_internal()
Ready for commit
> Subject: [PATCH v2.14 10/29] bufmgr: Implement AIO read support
> Buffer reads executed this infrastructure will report invalid page / checksum
> errors / warnings differently than before:
s/this/through this/
> + *zeroed_or_error_count = rem_error & ((1 << 7) - 1);
> + rem_error >>= 7;
These raw "7" are good places to use your new #define values. Likewise in
buffer_readv_encode_error().
> + * that was errored or zerored or, if no errors/zeroes, the first ignored
s/zerored/zeroed/
> + * enough. If there is an error, the error is the integeresting offset,
typo "integeresting"
> +/*
> + * We need a backend-local completion callback for shared buffers, to be able
> + * to report checksum errors correctly. Unfortunately that can only safely
> + * happen if the reporting backend has previously called
Missing end of sentence.
> @@ -144,8 +144,8 @@ PageIsVerified(PageData *page, BlockNumber blkno, int flags, bool *checksum_fail
> */
There's an outdated comment ending here:
/*
* Throw a WARNING if the checksum fails, but only after we've checked for
* the all-zeroes case.
*/
> if (checksum_failure)
> {
> - if ((flags & PIV_LOG_WARNING) != 0)
> - ereport(WARNING,
> + if ((flags & (PIV_LOG_WARNING | PIV_LOG_LOG)) != 0)
> + ereport(flags & PIV_LOG_WARNING ? WARNING : LOG,
> Subject: [PATCH v2.14 11/29] Let caller of PageIsVerified() control
> ignore_checksum_failure
> Subject: [PATCH v2.14 12/29] bufmgr: Use AIO in StartReadBuffers()
> Subject: [PATCH v2.14 13/29] aio: Add README.md explaining higher level design
> Subject: [PATCH v2.14 14/29] aio: Basic read_stream adjustments for real AIO
> Subject: [PATCH v2.14 15/29] read_stream: Introduce and use optional batchmode
> support
> Subject: [PATCH v2.14 16/29] docs: Reframe track_io_timing related docs as
> wait time
> Subject: [PATCH v2.14 17/29] Enable IO concurrency on all systems
Ready for commit
> Subject: [PATCH v2.14 18/29] aio: Add test_aio module
I didn't yet re-review the v2.13 or 2.14 changes to this one. That's still in
my queue. One thing I noticed anyway:
> +# Tests using injection points. Mostly to exercise had IO errors that are
s/had/hard/
On Sat, Mar 29, 2025 at 02:25:15PM -0400, Andres Freund wrote:
> On 2025-03-29 10:48:10 -0400, Andres Freund wrote:
> > Attached is v2.14:
> > - push pg_aios view (depends a tiny bit on the smgr/md/fd change above)
>
> I think I found an issue with this one - as it stands the view was viewable by
> everyone. While it doesn't provide a *lot* of insight, it still seems a bit
> too much for an unprivileged user to learn what part of a relation any other
> user is currently reading.
>
> There'd be two different ways to address that:
> 1) revoke view & function from public, grant to a limited role (presumably
> pg_read_all_stats)
> 2) copy pg_stat_activity's approach of using something like
>
> #define HAS_PGSTAT_PERMISSIONS(role) (has_privs_of_role(GetUserId(), ROLE_PG_READ_ALL_STATS) || has_privs_of_role(GetUserId(), role))
>
> on a per-IO basis.
No strong opinion. I'm not really worried about any of this information
leaking. Nothing in pg_aios comes close to the sensitivity of
pg_stat_activity.query. pg_stat_activity is mighty cautious, hiding even
stuff like wait_event_type that I wouldn't worry about. Hence, another valid
choice is (3) change nothing.
Meanwhile, I see substantially less need to monitor your own IOs than to
monitor your own pg_stat_activity rows, and even your own IOs potentially
reveal things happening in other sessions, e.g. evicting buffers that others
read and you never read. So restrictions wouldn't be too painful, and (1)
arguably helps privacy more than (2).
I'd likely go with (1) today.
^ permalink raw reply [nested|flat] 153+ messages in thread
* Re: AIO v2.5
2025-03-04 19:00 AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-10 18:23 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-11 19:41 ` Re: AIO v2.5 Noah Misch <noah@leadboat.com>
2025-03-11 23:55 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-12 03:57 ` Re: AIO v2.5 Noah Misch <noah@leadboat.com>
2025-03-12 17:06 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-12 23:23 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-13 10:53 ` Re: AIO v2.5 Antonin Houska <ah@cybertec.at>
2025-03-14 19:43 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-17 20:00 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-18 20:12 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-21 01:58 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-25 01:18 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-27 01:07 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-27 20:58 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-28 03:22 ` Re: AIO v2.5 Noah Misch <noah@leadboat.com>
2025-03-28 12:54 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-29 03:35 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-29 13:41 ` Re: AIO v2.5 Noah Misch <noah@leadboat.com>
2025-03-29 14:48 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-29 18:25 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-29 21:29 ` Re: AIO v2.5 Noah Misch <noah@leadboat.com>
@ 2025-03-30 00:39 ` Andres Freund <andres@anarazel.de>
2025-03-30 02:45 ` Re: AIO v2.5 Noah Misch <noah@leadboat.com>
1 sibling, 1 reply; 153+ messages in thread
From: Andres Freund @ 2025-03-30 00:39 UTC (permalink / raw)
To: Noah Misch <noah@leadboat.com>; +Cc: pgsql-hackers@postgresql.org, Thomas Munro <thomas.munro@gmail.com>; Heikki Linnakangas <hlinnaka@iki.fi>; Robert Haas <robertmhaas@gmail.com>; Jakub Wartak <jakub.wartak@enterprisedb.com>; Jelte Fennema-Nio <postgres@jeltef.nl>; Antonin Houska <ah@cybertec.at>
Hi,
On 2025-03-29 14:29:29 -0700, Noah Misch wrote:
> Flushing half-baked review comments before going offline for a few hours:
>
> On Wed, Mar 26, 2025 at 09:07:40PM -0400, Andres Freund wrote:
> > Attached v2.13, with the following changes:
>
> > 5) The WARNING in the callback is now a LOG, as it will be sent to the
> > client as a WARNING explicitly when the IO's results are processed
> >
> > I actually chose LOG_SERVER_ONLY - that seemed slightly better than just
> > LOG? But not at all sure.
>
> LOG_SERVER_ONLY and its synonym COMMERR look to be used for:
>
> - ProcessLogMemoryContextInterrupt()
> - messages before successful authentication
> - protocol sync loss, where we'd fail to send a client message
> - client already gone
>
> The choice between LOG and LOG_SERVER_ONLY doesn't matter much for $SUBJECT.
> If a client has decided to set client_min_messages that high, the client might
> be interested in the fact that it got side-tracked completing someone else's
> IO. On the other hand, almost none of those sidetrack events will produce
> messages. The main argument I'd envision for LOG_SERVER_ONLY is that we
> consider the message content sensitive, but I don't see the message content as
> materially sensitive.
I don't think it's sensitive - it just seems a bit sillier to send the same
thing to the client twice than to the server log. I'm happy to change it to
LOG if you prefer. Your points below mean some comments need to be updated in
smgr/md.c anyway.
> > - Previously the buffer completion callback checked zero_damaged_pages - but
> > that's not right, the GUC hopefully is only set on a per-session basis
>
> Good catch. I've now audited the complete_shared callbacks for other variable
> references and actions not acceptable there. I found nothing beyond what you
> found by v2.14.
I didn't find anything else either.
> > Subject: [PATCH v2.14 02/29] aio: Implement support for reads in smgr/md/fd
>
> > + /*
> > + * Immediately log a message about the IO error, but only to the
> > + * server log. The reason to do so immediately is that the originator
> > + * might not process the query result immediately (because it is busy
> > + * doing another part of query processing) or at all (e.g. if it was
> > + * cancelled or errored out due to another IO also failing). The
> > + * issuer of the IO will emit an ERROR when processing the IO's
>
> s/issuer/definer/ please, to avoid proliferating synonyms. Likewise two other
> places in the patches.
Hm. Will do. Doesn't bother me personally, but happy to change it.
> > +/*
> > + * smgrstartreadv() -- asynchronous version of smgrreadv()
> > + *
> > + * This starts an asynchronous readv IO using the IO handle `ioh`. Other than
> > + * `ioh` all parameters are the same as smgrreadv().
>
> I would add a comment starting with:
>
> Compared to smgrreadv(), more responsibilities fall on layers above smgr.
> Higher layers handle partial reads. smgr will ereport(LOG_SERVER_ONLY) some
> problems, but higher layers are responsible for pgaio_result_report() to
> mirror that news to the user and (for ERROR) abort the (sub)transaction.
Hm - if we document that in all the smgrstart* we'd end up with something like
that in a lot of places - but OTOH, this is the first one so far...
> I say "comment starting with", because I think there's a remaining decision
> about who owns the zeroing currently tied to smgrreadv(). An audit of
> mdreadv() vs. AIO counterparts found this part of mdreadv():
>
> if (nbytes == 0)
> {
> /*
> * We are at or past EOF, or we read a partial block at EOF.
> * Normally this is an error; upper levels should never try to
> * read a nonexistent block. However, if zero_damaged_pages
> * is ON or we are InRecovery, we should instead return zeroes
> * without complaining. This allows, for example, the case of
> * trying to update a block that was later truncated away.
> */
> if (zero_damaged_pages || InRecovery)
> {
>
> I didn't write a test to prove its absence, but I'm not finding such code in
> the AIO path.
Yes, there is no such codepath
A while ago I had started a thread about whether the above codepath is
necessary, as the whole idea of putting a buffer into shared buffers that
doesn't exist on-disk is *extremely* ill conceived, it puts a buffer into
shared buffer that somehow wasn't readable on disk, *without* creating it on
disk. The problem is that an mdnblocks() wouldn't know about that
only-in-memory part of the relation and thus most parts of PG won't consider
that buffer to exist - it'd be just skipped in sequential scans etc, but then
it'd trigger errors when extending the relation ("unexpected data beyond
EOF"), etc.
I had planned to put in an error into mdreadv() at the time, but somehow lost
track of that - I kind of mentally put this issue into the "done" category :(
Based on my research, the InRecovery path is not reachable (most recovery
buffer reads go through XLogReadBufferExtended() which extends at that layer
files, and the exceptions like VM/FSM have explicit code to extend the
relation, c.f. vm_readbuf()). It actually looks to me like it *never* was
reachable, the XLogReadBufferExtended() predecessors, back to the initial
addition of WAL to PG, had that such an extension path, as did vm/fsm.
The zero_damaged_pages path hasn't reliably worked for a long time afaict,
because _mdfd_getseg() doesn't know about it (note we're not passing
EXTENSION_CREATE). So unless the buffer is just after the physical end of the
last segment, you'll just get an error at that point. To my knowledge we
haven't heard related complaints.
It makes some sense to have zero_damaged_pages for actually existing pages
reached from sequential / tid / COPY on the table level - after all that's the
only way you might get data out during data recovery. But those would never
reach this logic, as such scans rely on mdnblocks(). For index -> heap
fetches the option seems mainly dangerous, because that'll just create random
buffers in shared buffers that, as explained above, won't then be reached by
other scans. And index scans during data recovery are not a good idea in the
first place, all that one should do in that situation is to dump out the data.
At the very least we need to add a comment about this though. If we want to
implement it, it'd be easy enough, but I think that logic is so insane that I
think we shouldn't do it unless there is some *VERY* clear evidence that we
need it.
> I wondered if we could just Assert(!InRecovery), but adding that to
> md_readv_complete() failed 001_stream_rep.pl with this stack:
I'd expect that to fail in a lot of paths:
XLogReadBufferExtended() ->
ReadBufferWithoutRelcache() ->
ReadBuffer_common() ->
StartReadBuffer()
> > Subject: [PATCH v2.14 04/29] aio: Add pg_aios view
>
> > + /*
> > + * There is no lock that could prevent the state of the IO to advance
> > + * concurrently - and we don't want to introduce one, as that would
> > + * introduce atomics into a very common path. Instead we
> > + *
> > + * 1) Determine the state + generation of the IO.
> > + *
> > + * 2) Copy the IO to local memory.
> > + *
> > + * 3) Check if state or generation of the IO changed. If the state
> > + * changed, retry, if the generation changed don't display the IO.
> > + */
> > +
> > + /* 1) from above */
> > + start_generation = live_ioh->generation;
> > + pg_read_barrier();
>
> I think "retry:" needs to be here, above start_state assignment. Otherwise,
> the "live_ioh->state != start_state" test will keep seeing a state mismatch.
Damn, you're right.
> > Subject: [PATCH v2.14 05/29] localbuf: Track pincount in BufferDesc as well
> > Subject: [PATCH v2.14 07/29] aio: Add WARNING result status
> > Subject: [PATCH v2.14 08/29] pgstat: Allow checksum errors to be reported in
> > critical sections
> > Subject: [PATCH v2.14 09/29] Add errhint_internal()
>
> Ready for commit
Cool
> > Subject: [PATCH v2.14 10/29] bufmgr: Implement AIO read support
>
> > Buffer reads executed this infrastructure will report invalid page / checksum
> > errors / warnings differently than before:
>
> s/this/through this/
Fixed.
> > + *zeroed_or_error_count = rem_error & ((1 << 7) - 1);
> > + rem_error >>= 7;
>
> These raw "7" are good places to use your new #define values. Likewise in
> buffer_readv_encode_error().
Which define value are you thinking of here? I don't think any of the ones I
added apply? But I think you're right it'd be good to have some define for
it, at least locally.
> > + * that was errored or zerored or, if no errors/zeroes, the first ignored
>
> s/zerored/zeroed/
>
> > + * enough. If there is an error, the error is the integeresting offset,
>
> typo "integeresting"
:(. Fixed.
> > +/*
> > + * We need a backend-local completion callback for shared buffers, to be able
> > + * to report checksum errors correctly. Unfortunately that can only safely
> > + * happen if the reporting backend has previously called
>
> Missing end of sentence.
It's now:
/*
* We need a backend-local completion callback for shared buffers, to be able
* to report checksum errors correctly. Unfortunately that can only safely
* happen if the reporting backend has previously called
* pgstat_prepare_report_checksum_failure(), which we can only guarantee in
* the backend that started the IO. Hence this callback.
*/
> > @@ -144,8 +144,8 @@ PageIsVerified(PageData *page, BlockNumber blkno, int flags, bool *checksum_fail
> > */
>
> There's an outdated comment ending here:
>
> /*
> * Throw a WARNING if the checksum fails, but only after we've checked for
> * the all-zeroes case.
> */
Updated to:
/*
* Throw a WARNING/LOG, as instructed by PIV_LOG_*, if the checksum fails,
* but only after we've checked for the all-zeroes case.
*/
I found one more, the newly added comment about checksum_failure_p was still
talking about ignore_checksum_failure, but it should now be IGNORE_CHECKSUM_FAILURE.
> > Subject: [PATCH v2.14 11/29] Let caller of PageIsVerified() control
> > ignore_checksum_failure
> > Subject: [PATCH v2.14 12/29] bufmgr: Use AIO in StartReadBuffers()
> > Subject: [PATCH v2.14 13/29] aio: Add README.md explaining higher level design
> > Subject: [PATCH v2.14 14/29] aio: Basic read_stream adjustments for real AIO
> > Subject: [PATCH v2.14 15/29] read_stream: Introduce and use optional batchmode
> > support
> > Subject: [PATCH v2.14 16/29] docs: Reframe track_io_timing related docs as
> > wait time
> > Subject: [PATCH v2.14 17/29] Enable IO concurrency on all systems
>
> Ready for commit
Cool.
> > Subject: [PATCH v2.14 18/29] aio: Add test_aio module
>
> I didn't yet re-review the v2.13 or 2.14 changes to this one. That's still in
> my queue.
That's good - I think some of the tests need to expand a bit more. Since
that's at the end of the dependency chain...
> One thing I noticed anyway:
> > +# Tests using injection points. Mostly to exercise had IO errors that are
>
> s/had/hard/
Fixed.
> On Sat, Mar 29, 2025 at 02:25:15PM -0400, Andres Freund wrote:
> > On 2025-03-29 10:48:10 -0400, Andres Freund wrote:
> > > Attached is v2.14:
>
> > > - push pg_aios view (depends a tiny bit on the smgr/md/fd change above)
> >
> > I think I found an issue with this one - as it stands the view was viewable by
> > everyone. While it doesn't provide a *lot* of insight, it still seems a bit
> > too much for an unprivileged user to learn what part of a relation any other
> > user is currently reading.
> >
> > There'd be two different ways to address that:
> > 1) revoke view & function from public, grant to a limited role (presumably
> > pg_read_all_stats)
> > 2) copy pg_stat_activity's approach of using something like
> >
> > #define HAS_PGSTAT_PERMISSIONS(role) (has_privs_of_role(GetUserId(), ROLE_PG_READ_ALL_STATS) || has_privs_of_role(GetUserId(), role))
> >
> > on a per-IO basis.
>
> No strong opinion.
Same.
> I'm not really worried about any of this information leaking. Nothing in
> pg_aios comes close to the sensitivity of pg_stat_activity.query.
> pg_stat_activity is mighty cautious, hiding even stuff like wait_event_type
> that I wouldn't worry about. Hence, another valid choice is (3) change
> nothing.
I'd also be on board with that.
> Meanwhile, I see substantially less need to monitor your own IOs than to
> monitor your own pg_stat_activity rows, and even your own IOs potentially
> reveal things happening in other sessions, e.g. evicting buffers that others
> read and you never read. So restrictions wouldn't be too painful, and (1)
> arguably helps privacy more than (2).
>
> I'd likely go with (1) today.
Sounds good to me. It also has the advantage of being much easier to test than
2).
Greetings,
Andres Freund
^ permalink raw reply [nested|flat] 153+ messages in thread
* Re: AIO v2.5
2025-03-04 19:00 AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-10 18:23 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-11 19:41 ` Re: AIO v2.5 Noah Misch <noah@leadboat.com>
2025-03-11 23:55 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-12 03:57 ` Re: AIO v2.5 Noah Misch <noah@leadboat.com>
2025-03-12 17:06 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-12 23:23 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-13 10:53 ` Re: AIO v2.5 Antonin Houska <ah@cybertec.at>
2025-03-14 19:43 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-17 20:00 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-18 20:12 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-21 01:58 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-25 01:18 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-27 01:07 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-27 20:58 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-28 03:22 ` Re: AIO v2.5 Noah Misch <noah@leadboat.com>
2025-03-28 12:54 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-29 03:35 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-29 13:41 ` Re: AIO v2.5 Noah Misch <noah@leadboat.com>
2025-03-29 14:48 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-29 18:25 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-29 21:29 ` Re: AIO v2.5 Noah Misch <noah@leadboat.com>
2025-03-30 00:39 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
@ 2025-03-30 02:45 ` Noah Misch <noah@leadboat.com>
0 siblings, 0 replies; 153+ messages in thread
From: Noah Misch @ 2025-03-30 02:45 UTC (permalink / raw)
To: Andres Freund <andres@anarazel.de>; +Cc: pgsql-hackers@postgresql.org, Thomas Munro <thomas.munro@gmail.com>; Heikki Linnakangas <hlinnaka@iki.fi>; Robert Haas <robertmhaas@gmail.com>; Jakub Wartak <jakub.wartak@enterprisedb.com>; Jelte Fennema-Nio <postgres@jeltef.nl>; Antonin Houska <ah@cybertec.at>
On Sat, Mar 29, 2025 at 08:39:54PM -0400, Andres Freund wrote:
> On 2025-03-29 14:29:29 -0700, Noah Misch wrote:
> > On Wed, Mar 26, 2025 at 09:07:40PM -0400, Andres Freund wrote:
> > The choice between LOG and LOG_SERVER_ONLY doesn't matter much for $SUBJECT.
> > If a client has decided to set client_min_messages that high, the client might
> > be interested in the fact that it got side-tracked completing someone else's
> > IO. On the other hand, almost none of those sidetrack events will produce
> > messages. The main argument I'd envision for LOG_SERVER_ONLY is that we
> > consider the message content sensitive, but I don't see the message content as
> > materially sensitive.
>
> I don't think it's sensitive - it just seems a bit sillier to send the same
> thing to the client twice than to the server log.
Ah, that adds weight to the benefit of LOG_SERVER_ONLY.
> I'm happy to change it to
> LOG if you prefer. Your points below mean some comments need to be updated in
> smgr/md.c anyway.
Nah.
> > > +/*
> > > + * smgrstartreadv() -- asynchronous version of smgrreadv()
> > > + *
> > > + * This starts an asynchronous readv IO using the IO handle `ioh`. Other than
> > > + * `ioh` all parameters are the same as smgrreadv().
> >
> > I would add a comment starting with:
> >
> > Compared to smgrreadv(), more responsibilities fall on layers above smgr.
> > Higher layers handle partial reads. smgr will ereport(LOG_SERVER_ONLY) some
> > problems, but higher layers are responsible for pgaio_result_report() to
> > mirror that news to the user and (for ERROR) abort the (sub)transaction.
>
> Hm - if we document that in all the smgrstart* we'd end up with something like
> that in a lot of places - but OTOH, this is the first one so far...
Alternatively, to avoid duplication:
See $PLACE for the tasks that the caller's layer owns, in contrast to smgr
owning them for smgrreadv().
> > I say "comment starting with", because I think there's a remaining decision
> > about who owns the zeroing currently tied to smgrreadv(). An audit of
> > mdreadv() vs. AIO counterparts found this part of mdreadv():
> >
> > if (nbytes == 0)
> > {
> > /*
> > * We are at or past EOF, or we read a partial block at EOF.
> > * Normally this is an error; upper levels should never try to
> > * read a nonexistent block. However, if zero_damaged_pages
> > * is ON or we are InRecovery, we should instead return zeroes
> > * without complaining. This allows, for example, the case of
> > * trying to update a block that was later truncated away.
> > */
> > if (zero_damaged_pages || InRecovery)
> > {
> >
> > I didn't write a test to prove its absence, but I'm not finding such code in
> > the AIO path.
>
> Yes, there is no such codepath
>
> A while ago I had started a thread about whether the above codepath is
> necessary
postgr.es/m/3qxxsnciyffyf3wyguiz4besdp5t5uxvv3utg75cbcszojlz7p@uibfzmnukkbd
which I had forgotten completely.
I've redone your audit, and I agree the InRecovery case is dead code.
check-world InRecovery reaches mdstartreadv() and mdreadv() only via
XLogReadBufferExtended(), vm_readbuf(), and fsm_readbuf().
The zero_damaged_pages case entails more of a judgment call about whether or
not its rule breakage eclipses its utility. Fortunately, a disappointed
zero_damaged_pages user could work around that code's absence by stopping the
server and using "dd" to extend the segment with zeros.
> I had planned to put in an error into mdreadv() at the time, but somehow lost
> track of that - I kind of mentally put this issue into the "done" category :(
> At the very least we need to add a comment about this though.
I'm fine with either of:
1. Replace that mdreadv() code with an error.
2. Update comment in mdreadv() that we're phasing out this code due the
InRecovery case's dead code status and the zero_damaged_pages problems; AIO
intentionally doesn't replicate it. Maybe add Assert(false).
I'd do (2) for v18, then do (1) first thing in v19 development.
> > > + *zeroed_or_error_count = rem_error & ((1 << 7) - 1);
> > > + rem_error >>= 7;
> >
> > These raw "7" are good places to use your new #define values. Likewise in
> > buffer_readv_encode_error().
>
> Which define value are you thinking of here? I don't think any of the ones I
> added apply? But I think you're right it'd be good to have some define for
> it, at least locally.
It was just my imagination. Withdrawn.
> It's now:
>
> /*
> * We need a backend-local completion callback for shared buffers, to be able
> * to report checksum errors correctly. Unfortunately that can only safely
> * happen if the reporting backend has previously called
> * pgstat_prepare_report_checksum_failure(), which we can only guarantee in
> * the backend that started the IO. Hence this callback.
> */
Sounds good.
> Updated to:
> /*
> * Throw a WARNING/LOG, as instructed by PIV_LOG_*, if the checksum fails,
> * but only after we've checked for the all-zeroes case.
> */
>
> I found one more, the newly added comment about checksum_failure_p was still
> talking about ignore_checksum_failure, but it should now be IGNORE_CHECKSUM_FAILURE.
That works.
> > > Subject: [PATCH v2.14 18/29] aio: Add test_aio module
> >
> > I didn't yet re-review the v2.13 or 2.14 changes to this one. That's still in
> > my queue.
>
> That's good - I think some of the tests need to expand a bit more. Since
> that's at the end of the dependency chain...
Okay, I'll delay on re-reviewing that one. When it's a good time, please put
the CF entry back in Needs Review. The patches before it are all ready for
commit after the above points of this mail.
^ permalink raw reply [nested|flat] 153+ messages in thread
* Re: AIO v2.5
2025-03-04 19:00 AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-10 18:23 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-11 19:41 ` Re: AIO v2.5 Noah Misch <noah@leadboat.com>
2025-03-11 23:55 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-12 03:57 ` Re: AIO v2.5 Noah Misch <noah@leadboat.com>
2025-03-12 17:06 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-12 23:23 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-13 10:53 ` Re: AIO v2.5 Antonin Houska <ah@cybertec.at>
2025-03-14 19:43 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-17 20:00 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-18 20:12 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-21 01:58 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-25 01:18 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-27 01:07 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-27 20:58 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-28 03:22 ` Re: AIO v2.5 Noah Misch <noah@leadboat.com>
2025-03-28 12:54 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-29 03:35 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-29 13:41 ` Re: AIO v2.5 Noah Misch <noah@leadboat.com>
2025-03-29 14:48 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-29 18:25 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-29 21:29 ` Re: AIO v2.5 Noah Misch <noah@leadboat.com>
@ 2025-03-30 23:46 ` Andres Freund <andres@anarazel.de>
2025-04-01 00:41 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-04-20 15:00 ` Re: AIO v2.5 Alexander Lakhin <exclusion@gmail.com>
1 sibling, 2 replies; 153+ messages in thread
From: Andres Freund @ 2025-03-30 23:46 UTC (permalink / raw)
To: Noah Misch <noah@leadboat.com>; +Cc: pgsql-hackers@postgresql.org, Thomas Munro <thomas.munro@gmail.com>; Heikki Linnakangas <hlinnaka@iki.fi>; Robert Haas <robertmhaas@gmail.com>; Jakub Wartak <jakub.wartak@enterprisedb.com>; Jelte Fennema-Nio <postgres@jeltef.nl>; Antonin Houska <ah@cybertec.at>
Hi,
On 2025-03-29 14:29:29 -0700, Noah Misch wrote:
> > Subject: [PATCH v2.14 11/29] Let caller of PageIsVerified() control
> > ignore_checksum_failure
> > Subject: [PATCH v2.14 12/29] bufmgr: Use AIO in StartReadBuffers()
> > Subject: [PATCH v2.14 14/29] aio: Basic read_stream adjustments for real AIO
> > Subject: [PATCH v2.14 15/29] read_stream: Introduce and use optional batchmode
> > support
> > Subject: [PATCH v2.14 16/29] docs: Reframe track_io_timing related docs as
> > wait time
> > Subject: [PATCH v2.14 17/29] Enable IO concurrency on all systems
>
> Ready for commit
I've pushed most of these after some very light further editing. Yay. Thanks
a lot for all the reviews!
So far the buildfarm hasn't been complaining, but it's early days.
I didn't yet push
> > Subject: [PATCH v2.14 13/29] aio: Add README.md explaining higher level design
because I want to integrate some language that could be referenced by
smgrstartreadv() (and more in the future), as we have been talking about.
Tomorrow I'll work on sending out a new version with the remaining patches. I
plan for that version to have:
- pg_aios view with the security checks (already done, trivial)
- a commit with updated language for smgrstartreadv(), probably referencing
aio's README
- a change to mdreadv() around zero_damaged_pages, as Noah and I have been
discussing
- updated tests, with the FIXMEs etc addressed
- a reviewed version of the errcontext callback patch that Melanie sent
earlier today
Todo beyond that:
- The comment and docs updates we've been discussing in
https://postgr.es/m/5fc6r4smanncsmqw7ib6s3uj6eoiqoioxbd5ibmhtqimcggtoe%40fyrok2gozsoq
- I think a good long search through the docs is in order, there probably are
other things that should be updated, beyond concrete references to
effective_io_concurrency etc.
- Whether we should do something, and if so what, about BAS_BULKREAD for
18. Thomas may have some thoughts / code.
- Whether there's anything committable around Jelte's RLIMIT_NOFILE changes.
Greetings,
Andres Freund
^ permalink raw reply [nested|flat] 153+ messages in thread
* Re: AIO v2.5
2025-03-04 19:00 AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-10 18:23 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-11 19:41 ` Re: AIO v2.5 Noah Misch <noah@leadboat.com>
2025-03-11 23:55 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-12 03:57 ` Re: AIO v2.5 Noah Misch <noah@leadboat.com>
2025-03-12 17:06 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-12 23:23 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-13 10:53 ` Re: AIO v2.5 Antonin Houska <ah@cybertec.at>
2025-03-14 19:43 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-17 20:00 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-18 20:12 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-21 01:58 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-25 01:18 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-27 01:07 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-27 20:58 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-28 03:22 ` Re: AIO v2.5 Noah Misch <noah@leadboat.com>
2025-03-28 12:54 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-29 03:35 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-29 13:41 ` Re: AIO v2.5 Noah Misch <noah@leadboat.com>
2025-03-29 14:48 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-29 18:25 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-29 21:29 ` Re: AIO v2.5 Noah Misch <noah@leadboat.com>
2025-03-30 23:46 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
@ 2025-04-01 00:41 ` Andres Freund <andres@anarazel.de>
2025-04-01 11:56 ` Re: AIO v2.5 Aleksander Alekseev <aleksander@timescale.com>
2025-04-01 15:11 ` Re: AIO v2.5 Noah Misch <noah@leadboat.com>
1 sibling, 2 replies; 153+ messages in thread
From: Andres Freund @ 2025-04-01 00:41 UTC (permalink / raw)
To: Noah Misch <noah@leadboat.com>; +Cc: pgsql-hackers@postgresql.org, Thomas Munro <thomas.munro@gmail.com>; Heikki Linnakangas <hlinnaka@iki.fi>; Robert Haas <robertmhaas@gmail.com>; Jakub Wartak <jakub.wartak@enterprisedb.com>; Jelte Fennema-Nio <postgres@jeltef.nl>; Antonin Houska <ah@cybertec.at>
Hi,
On 2025-03-30 19:46:57 -0400, Andres Freund wrote:
> I didn't yet push
>
> > > Subject: [PATCH v2.14 13/29] aio: Add README.md explaining higher level design
>
> because I want to integrate some language that could be referenced by
> smgrstartreadv() (and more in the future), as we have been talking about.
I tried a bunch of variations and none of them seemed great. So I ended up
with a lightly polished version of your suggested comment above
smgrstartreadv(). We can later see about generalizing it.
> Tomorrow I'll work on sending out a new version with the remaining patches. I
> plan for that version to have:
Got a bit distracted with $work stuff today, but here we go.
The updated version has all of that:
> - pg_aios view with the security checks (already done, trivial)
>
> - a commit with updated language for smgrstartreadv(), probably referencing
> aio's README
>
> - a change to mdreadv() around zero_damaged_pages, as Noah and I have been
> discussing
>
> - updated tests, with the FIXMEs etc addressed
>
> - a reviewed version of the errcontext callback patch that Melanie sent
> earlier today
Although I didn't actually find anything in that, other than one unnecessary
change.
Greetings,
Andres Freund
Attachments:
[text/x-diff] v2.15-0001-docs-Add-acronym-and-glossary-entries-for-I-O-.patch (3.5K, ../../7s6fclfekpcxoaaorwrq67v4vmgixf2dcjcgznuj7vxs3ie3wq@okqbi5vabqbx/2-v2.15-0001-docs-Add-acronym-and-glossary-entries-for-I-O-.patch)
download | inline diff:
From 7aab9384b807f8476d79c53aff0b44035e012e8c Mon Sep 17 00:00:00 2001
From: Andres Freund <andres@anarazel.de>
Date: Fri, 21 Mar 2025 15:06:35 -0400
Subject: [PATCH v2.15 01/18] docs: Add acronym and glossary entries for I/O
and AIO
These are fairly basic, but better than nothing. While there are several
opportunities to link to these entries, this patch does not add any. They will
however be referenced by future patches.
Reviewed-by: Noah Misch <noah@leadboat.com>
Discussion: https://postgr.es/m/20250326183102.92.nmisch@google.com
---
doc/src/sgml/acronyms.sgml | 18 ++++++++++++++++++
doc/src/sgml/glossary.sgml | 39 ++++++++++++++++++++++++++++++++++++++
2 files changed, 57 insertions(+)
diff --git a/doc/src/sgml/acronyms.sgml b/doc/src/sgml/acronyms.sgml
index 58d0d90fece..2f906e9f018 100644
--- a/doc/src/sgml/acronyms.sgml
+++ b/doc/src/sgml/acronyms.sgml
@@ -9,6 +9,15 @@
<variablelist>
+ <varlistentry>
+ <term><acronym>AIO</acronym></term>
+ <listitem>
+ <para>
+ <link linkend="glossary-aio">Asynchronous <acronym>I/O</acronym></link>
+ </para>
+ </listitem>
+ </varlistentry>
+
<varlistentry>
<term><acronym>ACL</acronym></term>
<listitem>
@@ -354,6 +363,15 @@
</listitem>
</varlistentry>
+ <varlistentry>
+ <term><acronym>I/O</acronym></term>
+ <listitem>
+ <para>
+ <link linkend="glossary-io">Input/Output</link>
+ </para>
+ </listitem>
+ </varlistentry>
+
<varlistentry>
<term><acronym>ISO</acronym></term>
<listitem>
diff --git a/doc/src/sgml/glossary.sgml b/doc/src/sgml/glossary.sgml
index c0f812e3f5e..b88cac598e9 100644
--- a/doc/src/sgml/glossary.sgml
+++ b/doc/src/sgml/glossary.sgml
@@ -81,6 +81,31 @@
</glossdef>
</glossentry>
+ <glossentry id="glossary-aio">
+ <glossterm>Asynchronous <acronym>I/O</acronym></glossterm>
+ <acronym>AIO</acronym>
+ <indexterm>
+ <primary>Asynchronous <acronym>I/O</acronym></primary>
+ </indexterm>
+ <glossdef>
+ <para>
+ Asynchronous <acronym>I/O</acronym> (<acronym>AIO</acronym>) describes
+ performing <acronym>I/O</acronym> in a non-blocking way (asynchronously),
+ in contrast to synchronous <acronym>I/O</acronym>, which blocks for the
+ entire duration of the <acronym>I/O</acronym>.
+ </para>
+ <para>
+ With <acronym>AIO</acronym>, starting an <acronym>I/O</acronym> operation
+ is separated from waiting for the result of the operation, allowing
+ multiple <acronym>I/O</acronym> operations to be initiated concurrently,
+ as well as performing <acronym>CPU</acronym> heavy operations
+ concurrently with <acronym>I/O</acronym>. The price for that increased
+ concurrency is increased complexity.
+ </para>
+ <glossseealso otherterm="glossary-io" />
+ </glossdef>
+ </glossentry>
+
<glossentry id="glossary-atomic">
<glossterm>Atomic</glossterm>
<glossdef>
@@ -938,6 +963,20 @@
</glossdef>
</glossentry>
+ <glossentry id="glossary-io">
+ <glossterm>Input/Output</glossterm>
+ <acronym>I/O</acronym>
+ <glossdef>
+ <para>
+ Input/Output (<acronym>I/O</acronym>) describes the communication between
+ a program and peripheral devices. In the context of database systems,
+ <acronym>I/O</acronym> commonly, but not exclusively, refers to
+ interaction with storage devices or the network.
+ </para>
+ <glossseealso otherterm="glossary-aio" />
+ </glossdef>
+ </glossentry>
+
<glossentry id="glossary-insert">
<glossterm>Insert</glossterm>
<glossdef>
--
2.48.1.76.g4e746b1a31.dirty
[text/x-diff] v2.15-0002-aio-Add-pg_aios-view.patch (22.6K, ../../7s6fclfekpcxoaaorwrq67v4vmgixf2dcjcgznuj7vxs3ie3wq@okqbi5vabqbx/3-v2.15-0002-aio-Add-pg_aios-view.patch)
download | inline diff:
From 6a8f2696b60d00ac50938573d103b3c28f8f2257 Mon Sep 17 00:00:00 2001
From: Andres Freund <andres@anarazel.de>
Date: Tue, 18 Mar 2025 14:40:06 -0400
Subject: [PATCH v2.15 02/18] aio: Add pg_aios view
The new view lists all IO handles that are currently in use and is mainly
useful for PG developers, but may also be useful when tuning PG.
FIXME:
- catversion bump before commit
Reviewed-by: Noah Misch <noah@leadboat.com>
Discussion: https://postgr.es/m/uvrtrknj4kdytuboidbhwclo4gxhswwcpgadptsjvjqcluzmah%40brqs62irg4dt
---
src/include/catalog/pg_proc.dat | 9 +
src/backend/catalog/system_views.sql | 7 +
src/backend/storage/aio/Makefile | 1 +
src/backend/storage/aio/aio_funcs.c | 230 ++++++++++++++++++
src/backend/storage/aio/meson.build | 1 +
doc/src/sgml/system-views.sgml | 294 +++++++++++++++++++++++
src/test/regress/expected/privileges.out | 18 ++
src/test/regress/expected/rules.out | 16 ++
src/test/regress/sql/privileges.sql | 3 +
9 files changed, 579 insertions(+)
create mode 100644 src/backend/storage/aio/aio_funcs.c
diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat
index 8b68b16d79d..d9c41fa426b 100644
--- a/src/include/catalog/pg_proc.dat
+++ b/src/include/catalog/pg_proc.dat
@@ -12493,4 +12493,13 @@
proargtypes => 'int4',
prosrc => 'gist_stratnum_common' },
+# AIO related functions
+{ oid => '9200', descr => 'information about in-progress asynchronous IOs',
+ proname => 'pg_get_aios', prorows => '100', proretset => 't',
+ provolatile => 'v', proparallel => 'r', prorettype => 'record', proargtypes => '',
+ proallargtypes => '{int4,int4,int8,text,text,int8,int8,text,int2,int4,text,text,bool,bool,bool}',
+ proargmodes => '{o,o,o,o,o,o,o,o,o,o,o,o,o,o,o}',
+ proargnames => '{pid,io_id,io_generation,state,operation,off,length,target,handle_data_len,raw_result,result,target_desc,f_sync,f_localmem,f_buffered}',
+ prosrc => 'pg_get_aios' },
+
]
diff --git a/src/backend/catalog/system_views.sql b/src/backend/catalog/system_views.sql
index 31d269b7ee0..64a7240aa77 100644
--- a/src/backend/catalog/system_views.sql
+++ b/src/backend/catalog/system_views.sql
@@ -1391,3 +1391,10 @@ CREATE VIEW pg_stat_subscription_stats AS
CREATE VIEW pg_wait_events AS
SELECT * FROM pg_get_wait_events();
+
+CREATE VIEW pg_aios AS
+ SELECT * FROM pg_get_aios();
+REVOKE ALL ON pg_aios FROM PUBLIC;
+GRANT SELECT ON pg_aios TO pg_read_all_stats;
+REVOKE EXECUTE ON FUNCTION pg_get_aios() FROM PUBLIC;
+GRANT EXECUTE ON FUNCTION pg_get_aios() TO pg_read_all_stats;
diff --git a/src/backend/storage/aio/Makefile b/src/backend/storage/aio/Makefile
index c06c50771e0..3f2469cc399 100644
--- a/src/backend/storage/aio/Makefile
+++ b/src/backend/storage/aio/Makefile
@@ -11,6 +11,7 @@ include $(top_builddir)/src/Makefile.global
OBJS = \
aio.o \
aio_callback.o \
+ aio_funcs.o \
aio_init.o \
aio_io.o \
aio_target.o \
diff --git a/src/backend/storage/aio/aio_funcs.c b/src/backend/storage/aio/aio_funcs.c
new file mode 100644
index 00000000000..584e683371a
--- /dev/null
+++ b/src/backend/storage/aio/aio_funcs.c
@@ -0,0 +1,230 @@
+/*-------------------------------------------------------------------------
+ *
+ * aio_funcs.c
+ * AIO - SQL interface for AIO
+ *
+ *
+ * Portions Copyright (c) 1996-2025, PostgreSQL Global Development Group
+ * Portions Copyright (c) 1994, Regents of the University of California
+ *
+ * IDENTIFICATION
+ * src/backend/storage/aio/aio_funcs.c
+ *
+ *-------------------------------------------------------------------------
+ */
+
+#include "postgres.h"
+
+#include "fmgr.h"
+#include "funcapi.h"
+#include "nodes/execnodes.h"
+#include "port/atomics.h"
+#include "storage/aio_internal.h"
+#include "storage/lock.h"
+#include "storage/proc.h"
+#include "storage/procnumber.h"
+#include "utils/builtins.h"
+#include "utils/fmgrprotos.h"
+#include "utils/tuplestore.h"
+
+
+/*
+ * Byte length of an iovec.
+ */
+static size_t
+iov_byte_length(const struct iovec *iov, int cnt)
+{
+ size_t len = 0;
+
+ for (int i = 0; i < cnt; i++)
+ {
+ len += iov[i].iov_len;
+ }
+
+ return len;
+}
+
+Datum
+pg_get_aios(PG_FUNCTION_ARGS)
+{
+ ReturnSetInfo *rsinfo = (ReturnSetInfo *) fcinfo->resultinfo;
+
+ InitMaterializedSRF(fcinfo, 0);
+
+#define PG_GET_AIOS_COLS 15
+
+ for (uint64 i = 0; i < pgaio_ctl->io_handle_count; i++)
+ {
+ PgAioHandle *live_ioh = &pgaio_ctl->io_handles[i];
+ uint32 ioh_id = pgaio_io_get_id(live_ioh);
+ Datum values[PG_GET_AIOS_COLS] = {0};
+ bool nulls[PG_GET_AIOS_COLS] = {0};
+ ProcNumber owner;
+ PGPROC *owner_proc;
+ int32 owner_pid;
+ PgAioHandleState start_state;
+ uint64 start_generation;
+ PgAioHandle ioh_copy;
+ struct iovec iov_copy[PG_IOV_MAX];
+
+
+ /*
+ * There is no lock that could prevent the state of the IO to advance
+ * concurrently - and we don't want to introduce one, as that would
+ * introduce atomics into a very common path. Instead we
+ *
+ * 1) Determine the state + generation of the IO.
+ *
+ * 2) Copy the IO to local memory.
+ *
+ * 3) Check if state or generation of the IO changed. If the state
+ * changed, retry, if the generation changed don't display the IO.
+ */
+
+ /* 1) from above */
+ start_generation = live_ioh->generation;
+
+ /*
+ * Retry at this point, so we can accept changing states, but not
+ * changing generations.
+ */
+retry:
+ pg_read_barrier();
+ start_state = live_ioh->state;
+
+ if (start_state == PGAIO_HS_IDLE)
+ continue;
+
+ /* 2) from above */
+ memcpy(&ioh_copy, live_ioh, sizeof(PgAioHandle));
+
+ /*
+ * Safe to copy even if no iovec is used - we always reserve the
+ * required space.
+ */
+ memcpy(&iov_copy, &pgaio_ctl->iovecs[ioh_copy.iovec_off],
+ PG_IOV_MAX * sizeof(struct iovec));
+
+ /*
+ * Copy information about owner before 3) below, if the process exited
+ * it'd have to wait for the IO to finish first, which we would detect
+ * in 3).
+ */
+ owner = ioh_copy.owner_procno;
+ owner_proc = GetPGProcByNumber(owner);
+ owner_pid = owner_proc->pid;
+
+ /* 3) from above */
+ pg_read_barrier();
+
+ /*
+ * The IO completed and a new one was started with the same ID. Don't
+ * display it - it really started after this function was called.
+ * There be a risk of a livelock if we just retried endlessly, if IOs
+ * complete very quickly.
+ */
+ if (live_ioh->generation != start_generation)
+ continue;
+
+ /*
+ * The IO's state changed while we were "rendering" it. Just start
+ * from scratch. There's no risk of a livelock here, as an IO has a
+ * limited sets of states it can be in, and state changes go only in a
+ * single direction.
+ */
+ if (live_ioh->state != start_state)
+ goto retry;
+
+ /*
+ * Now that we have copied the IO into local memory and checked that
+ * it's still in the same state, we are not allowed to access "live"
+ * memory anymore. To make it slightly easier to catch such cases, set
+ * the "live" pointers to NULL.
+ */
+ live_ioh = NULL;
+ owner_proc = NULL;
+
+
+ /* column: owning pid */
+ if (owner_pid != 0)
+ values[0] = Int32GetDatum(owner_pid);
+ else
+ nulls[0] = false;
+
+ /* column: IO's id */
+ values[1] = ioh_id;
+
+ /* column: IO's generation */
+ values[2] = Int64GetDatum(start_generation);
+
+ /* column: IO's state */
+ values[3] = CStringGetTextDatum(pgaio_io_get_state_name(&ioh_copy));
+
+ /*
+ * If the IO is in PGAIO_HS_HANDED_OUT state, none of the following
+ * fields are valid yet (or are in the process of being set).
+ * Therefore we don't want to display any other columns.
+ */
+ if (start_state == PGAIO_HS_HANDED_OUT)
+ {
+ memset(nulls + 4, 1, (lengthof(nulls) - 4) * sizeof(bool));
+ goto display;
+ }
+
+ /* column: IO's operation */
+ values[4] = CStringGetTextDatum(pgaio_io_get_op_name(&ioh_copy));
+
+ /* columns: details about the IO's operation (offset, length) */
+ switch (ioh_copy.op)
+ {
+ case PGAIO_OP_INVALID:
+ nulls[5] = true;
+ nulls[6] = true;
+ break;
+ case PGAIO_OP_READV:
+ values[5] = Int64GetDatum(ioh_copy.op_data.read.offset);
+ values[6] =
+ Int64GetDatum(iov_byte_length(iov_copy, ioh_copy.op_data.read.iov_length));
+ break;
+ case PGAIO_OP_WRITEV:
+ values[5] = Int64GetDatum(ioh_copy.op_data.write.offset);
+ values[6] =
+ Int64GetDatum(iov_byte_length(iov_copy, ioh_copy.op_data.write.iov_length));
+ break;
+ }
+
+ /* column: IO's target */
+ values[7] = CStringGetTextDatum(pgaio_io_get_target_name(&ioh_copy));
+
+ /* column: length of IO's data array */
+ values[8] = Int16GetDatum(ioh_copy.handle_data_len);
+
+ /* column: raw result (i.e. some form of syscall return value) */
+ if (start_state == PGAIO_HS_COMPLETED_IO
+ || start_state == PGAIO_HS_COMPLETED_SHARED
+ || start_state == PGAIO_HS_COMPLETED_LOCAL)
+ values[9] = Int32GetDatum(ioh_copy.result);
+ else
+ nulls[9] = true;
+
+ /*
+ * column: result in the higher level representation (unknown if not
+ * finished)
+ */
+ values[10] =
+ CStringGetTextDatum(pgaio_result_status_string(ioh_copy.distilled_result.status));
+
+ /* column: target description */
+ values[11] = CStringGetTextDatum(pgaio_io_get_target_description(&ioh_copy));
+
+ /* columns: one for each flag */
+ values[12] = BoolGetDatum(ioh_copy.flags & PGAIO_HF_SYNCHRONOUS);
+ values[13] = BoolGetDatum(ioh_copy.flags & PGAIO_HF_REFERENCES_LOCAL);
+ values[14] = BoolGetDatum(ioh_copy.flags & PGAIO_HF_BUFFERED);
+
+display:
+ tuplestore_putvalues(rsinfo->setResult, rsinfo->setDesc, values, nulls);
+ }
+
+ return (Datum) 0;
+}
diff --git a/src/backend/storage/aio/meson.build b/src/backend/storage/aio/meson.build
index 2f0f03d8071..da6df2d3654 100644
--- a/src/backend/storage/aio/meson.build
+++ b/src/backend/storage/aio/meson.build
@@ -3,6 +3,7 @@
backend_sources += files(
'aio.c',
'aio_callback.c',
+ 'aio_funcs.c',
'aio_init.c',
'aio_io.c',
'aio_target.c',
diff --git a/doc/src/sgml/system-views.sgml b/doc/src/sgml/system-views.sgml
index 3f5a306247e..e9a59af8c34 100644
--- a/doc/src/sgml/system-views.sgml
+++ b/doc/src/sgml/system-views.sgml
@@ -51,6 +51,11 @@
</thead>
<tbody>
+ <row>
+ <entry><link linkend="view-pg-aios"><structname>pg_aios</structname></link></entry>
+ <entry>In-use asynchronous IO handles</entry>
+ </row>
+
<row>
<entry><link linkend="view-pg-available-extensions"><structname>pg_available_extensions</structname></link></entry>
<entry>available extensions</entry>
@@ -231,6 +236,295 @@
</table>
</sect1>
+ <sect1 id="view-pg-aios">
+ <title><structname>pg_aios</structname></title>
+
+ <indexterm zone="view-pg-aios">
+ <primary>pg_aios</primary>
+ </indexterm>
+
+ <para>
+ The <structname>pg_aios</structname> view lists all <xref
+ linkend="glossary-aio"/> handles that are currently in-use. An I/O handle
+ is used to reference an I/O operation that is being prepared, executed or
+ is in the process of completing. <structname>pg_aios</structname> contains
+ one row for each I/O handle.
+ </para>
+
+ <para>
+ This view is mainly useful for developers of
+ <productname>PostgreSQL</productname>, but may also be useful when tuning
+ <productname>PostgreSQL</productname>.
+ </para>
+
+ <table>
+ <title><structname>pg_aios</structname> Columns</title>
+ <tgroup cols="1">
+ <thead>
+ <row>
+ <entry role="catalog_table_entry"><para role="column_definition">
+ Column Type
+ </para>
+ <para>
+ Description
+ </para></entry>
+ </row>
+ </thead>
+
+ <tbody>
+ <row>
+ <entry role="catalog_table_entry"><para role="column_definition">
+ <structfield>pid</structfield> <type>int4</type>
+ </para>
+ <para>
+ Process ID of the server process that is issuing this I/O.
+ </para></entry>
+ </row>
+
+ <row>
+ <entry role="catalog_table_entry"><para role="column_definition">
+ <structfield>io_id</structfield> <type>int4</type>
+ </para>
+ <para>
+ Identifier of the I/O handle. Handles are reused once the I/O
+ completed (or if the handle is released before I/O is started). On reuse
+ <link linkend="view-pg-aios-io-generation">
+ <structname>pg_aios</structname>.<structfield>io_generation</structfield>
+ </link>
+ is incremented.
+ </para></entry>
+ </row>
+
+ <row>
+ <entry role="catalog_table_entry" id="view-pg-aios-io-generation"><para role="column_definition">
+ <structfield>io_generation</structfield> <type>int8</type>
+ </para>
+ <para>
+ Generation of the I/O handle.
+ </para></entry>
+ </row>
+
+ <row>
+ <entry role="catalog_table_entry"><para role="column_definition">
+ <structfield>state</structfield> <type>text</type>
+ </para>
+ <para>
+ State of the I/O handle:
+ <itemizedlist>
+ <listitem>
+ <para>
+ <literal>HANDED_OUT</literal>, referenced by code but not yet used
+ </para>
+ </listitem>
+ <listitem>
+ <para>
+ <literal>DEFINED</literal>, information necessary for execution is known
+ </para>
+ </listitem>
+ <listitem>
+ <para>
+ <literal>STAGED</literal>, ready for execution
+ </para>
+ </listitem>
+ <listitem>
+ <para>
+ <literal>SUBMITTED</literal>, submitted for execution
+ </para>
+ </listitem>
+ <listitem>
+ <para>
+ <literal>COMPLETED_IO</literal>, finished, but result has not yet been processed
+ </para>
+ </listitem>
+ <listitem>
+ <para>
+ <literal>COMPLETED_SHARED</literal>, shared completion processing completed
+ </para>
+ </listitem>
+ <listitem>
+ <para>
+ <literal>COMPLETED_LOCAL</literal>, backend local completion processing completed
+ </para>
+ </listitem>
+ </itemizedlist>
+ </para></entry>
+ </row>
+
+ <row>
+ <entry role="catalog_table_entry"><para role="column_definition">
+ <structfield>operation</structfield> <type>text</type>
+ </para>
+ <para>
+ Operation performed using the I/O handle:
+ <itemizedlist>
+ <listitem>
+ <para>
+ <literal>invalid</literal>, not yet known
+ </para>
+ </listitem>
+ <listitem>
+ <para>
+ <literal>readv</literal>, a vectored read
+ </para>
+ </listitem>
+ <listitem>
+ <para>
+ <literal>writev</literal>, a vectored write
+ </para>
+ </listitem>
+ </itemizedlist>
+ </para></entry>
+ </row>
+
+ <row>
+ <entry role="catalog_table_entry"><para role="column_definition">
+ <structfield>off</structfield> <type>int8</type>
+ </para>
+ <para>
+ Offset of the I/O operation.
+ </para></entry>
+ </row>
+
+ <row>
+ <entry role="catalog_table_entry"><para role="column_definition">
+ <structfield>length</structfield> <type>int8</type>
+ </para>
+ <para>
+ Length of the I/O operation.
+ </para></entry>
+ </row>
+
+ <row>
+ <entry role="catalog_table_entry"><para role="column_definition">
+ <structfield>target</structfield> <type>text</type>
+ </para>
+ <para>
+ What kind of object is the I/O targeting:
+ <itemizedlist spacing="compact">
+ <listitem>
+ <para>
+ <literal>smgr</literal>, I/O on relations
+ </para>
+ </listitem>
+ </itemizedlist>
+ </para></entry>
+ </row>
+
+ <row>
+ <entry role="catalog_table_entry"><para role="column_definition">
+ <structfield>handle_data_len</structfield> <type>int2</type>
+ </para>
+ <para>
+ Length of the data associated with the I/O operation. For I/O to/from
+ <xref linkend="guc-shared-buffers"/> and <xref
+ linkend="guc-temp-buffers"/>, this indicates the number of buffers the
+ I/O is operating on.
+ </para></entry>
+ </row>
+
+ <row>
+ <entry role="catalog_table_entry"><para role="column_definition">
+ <structfield>raw_result</structfield> <type>int4</type>
+ </para>
+ <para>
+ Low-level result of the I/O operation, or NULL if the operation has not
+ yet completed.
+ </para></entry>
+ </row>
+
+ <row>
+ <entry role="catalog_table_entry"><para role="column_definition">
+ <structfield>result</structfield> <type>text</type>
+ </para>
+ <para>
+ High-level result of the I/O operation:
+ <itemizedlist>
+ <listitem>
+ <para>
+ <literal>UNKNOWN</literal> means that the result of the
+ operation is not yet known.
+ </para>
+ </listitem>
+ <listitem>
+ <para>
+ <literal>OK</literal> means the I/O completed successfully.
+ </para>
+ </listitem>
+ <listitem>
+ <para>
+ <literal>PARTIAL</literal> means that the I/O completed without
+ error, but did not process all data. Commonly callers will need to
+ retry and perform the remainder of the work in a separate I/O.
+ </para>
+ </listitem>
+ <listitem>
+ <para>
+ <literal>WARNING</literal> means that the I/O completed without
+ error, but that execution of the IO triggered a warning. E.g. when
+ encountering a corrupted buffer with <xref
+ linkend="guc-zero-damaged-pages"/> enabled.
+ </para>
+ </listitem>
+ <listitem>
+ <para>
+ <literal>ERROR</literal> means the I/O failed with an error.
+ </para>
+ </listitem>
+ </itemizedlist>
+ </para></entry>
+ </row>
+
+ <row>
+ <entry role="catalog_table_entry"><para role="column_definition">
+ <structfield>target_desc</structfield> <type>text</type>
+ </para>
+ <para>
+ Description of what the I/O operation is targeting.
+ </para></entry>
+ </row>
+
+ <row>
+ <entry role="catalog_table_entry"><para role="column_definition">
+ <structfield>f_sync</structfield> <type>bool</type>
+ </para>
+ <para>
+ Flag indicating whether the I/O is executed synchronously.
+ </para></entry>
+ </row>
+
+ <row>
+ <entry role="catalog_table_entry"><para role="column_definition">
+ <structfield>f_localmem</structfield> <type>bool</type>
+ </para>
+ <para>
+ Flag indicating whether the I/O references process local memory.
+ </para></entry>
+ </row>
+
+ <row>
+ <entry role="catalog_table_entry"><para role="column_definition">
+ <structfield>f_buffered</structfield> <type>bool</type>
+ </para>
+ <para>
+ Flag indicating whether the I/O is buffered I/O.
+ </para></entry>
+ </row>
+
+ </tbody>
+ </tgroup>
+ </table>
+
+ <para>
+ The <structname>pg_aios</structname> view is read-only.
+ </para>
+
+ <para>
+ By default, the <structname>pg_aios</structname> view can be read only by
+ superusers or roles with privileges of the
+ <literal>pg_read_all_stats</literal> role.
+ </para>
+ </sect1>
+
<sect1 id="view-pg-available-extensions">
<title><structname>pg_available_extensions</structname></title>
diff --git a/src/test/regress/expected/privileges.out b/src/test/regress/expected/privileges.out
index 954f549555e..5588d83e1bf 100644
--- a/src/test/regress/expected/privileges.out
+++ b/src/test/regress/expected/privileges.out
@@ -3132,6 +3132,12 @@ DROP USER regress_locktable_user;
-- switch to superuser
\c -
CREATE ROLE regress_readallstats;
+SELECT has_table_privilege('regress_readallstats','pg_aios','SELECT'); -- no
+ has_table_privilege
+---------------------
+ f
+(1 row)
+
SELECT has_table_privilege('regress_readallstats','pg_backend_memory_contexts','SELECT'); -- no
has_table_privilege
---------------------
@@ -3145,6 +3151,12 @@ SELECT has_table_privilege('regress_readallstats','pg_shmem_allocations','SELECT
(1 row)
GRANT pg_read_all_stats TO regress_readallstats;
+SELECT has_table_privilege('regress_readallstats','pg_aios','SELECT'); -- yes
+ has_table_privilege
+---------------------
+ t
+(1 row)
+
SELECT has_table_privilege('regress_readallstats','pg_backend_memory_contexts','SELECT'); -- yes
has_table_privilege
---------------------
@@ -3159,6 +3171,12 @@ SELECT has_table_privilege('regress_readallstats','pg_shmem_allocations','SELECT
-- run query to ensure that functions within views can be executed
SET ROLE regress_readallstats;
+SELECT COUNT(*) >= 0 AS ok FROM pg_aios;
+ ok
+----
+ t
+(1 row)
+
SELECT COUNT(*) >= 0 AS ok FROM pg_backend_memory_contexts;
ok
----
diff --git a/src/test/regress/expected/rules.out b/src/test/regress/expected/rules.out
index 47478969135..d9533deb04e 100644
--- a/src/test/regress/expected/rules.out
+++ b/src/test/regress/expected/rules.out
@@ -1286,6 +1286,22 @@ drop table cchild;
SELECT viewname, definition FROM pg_views
WHERE schemaname = 'pg_catalog'
ORDER BY viewname;
+pg_aios| SELECT pid,
+ io_id,
+ io_generation,
+ state,
+ operation,
+ off,
+ length,
+ target,
+ handle_data_len,
+ raw_result,
+ result,
+ target_desc,
+ f_sync,
+ f_localmem,
+ f_buffered
+ FROM pg_get_aios() pg_get_aios(pid, io_id, io_generation, state, operation, off, length, target, handle_data_len, raw_result, result, target_desc, f_sync, f_localmem, f_buffered);
pg_available_extension_versions| SELECT e.name,
e.version,
(x.extname IS NOT NULL) AS installed,
diff --git a/src/test/regress/sql/privileges.sql b/src/test/regress/sql/privileges.sql
index b81694c24f2..286b1d03756 100644
--- a/src/test/regress/sql/privileges.sql
+++ b/src/test/regress/sql/privileges.sql
@@ -1919,16 +1919,19 @@ DROP USER regress_locktable_user;
CREATE ROLE regress_readallstats;
+SELECT has_table_privilege('regress_readallstats','pg_aios','SELECT'); -- no
SELECT has_table_privilege('regress_readallstats','pg_backend_memory_contexts','SELECT'); -- no
SELECT has_table_privilege('regress_readallstats','pg_shmem_allocations','SELECT'); -- no
GRANT pg_read_all_stats TO regress_readallstats;
+SELECT has_table_privilege('regress_readallstats','pg_aios','SELECT'); -- yes
SELECT has_table_privilege('regress_readallstats','pg_backend_memory_contexts','SELECT'); -- yes
SELECT has_table_privilege('regress_readallstats','pg_shmem_allocations','SELECT'); -- yes
-- run query to ensure that functions within views can be executed
SET ROLE regress_readallstats;
+SELECT COUNT(*) >= 0 AS ok FROM pg_aios;
SELECT COUNT(*) >= 0 AS ok FROM pg_backend_memory_contexts;
SELECT COUNT(*) >= 0 AS ok FROM pg_shmem_allocations;
RESET ROLE;
--
2.48.1.76.g4e746b1a31.dirty
[text/x-diff] v2.15-0003-aio-Add-README.md-explaining-higher-level-desi.patch (19.1K, ../../7s6fclfekpcxoaaorwrq67v4vmgixf2dcjcgznuj7vxs3ie3wq@okqbi5vabqbx/4-v2.15-0003-aio-Add-README.md-explaining-higher-level-desi.patch)
download | inline diff:
From 8630699dd1c0f66b2b379d926dd0ac9167cb7d78 Mon Sep 17 00:00:00 2001
From: Andres Freund <andres@anarazel.de>
Date: Tue, 18 Mar 2025 14:40:06 -0400
Subject: [PATCH v2.15 03/18] aio: Add README.md explaining higher level design
Reviewed-by: Noah Misch <noah@leadboat.com>
Reviewed-by: Heikki Linnakangas <hlinnaka@iki.fi>
Discussion: https://postgr.es/m/uvrtrknj4kdytuboidbhwclo4gxhswwcpgadptsjvjqcluzmah%40brqs62irg4dt
Discussion: https://postgr.es/m/20210223100344.llw5an2aklengrmn@alap3.anarazel.de
Discussion: https://postgr.es/m/stj36ea6yyhoxtqkhpieia2z4krnam7qyetc57rfezgk4zgapf@gcnactj4z56m
---
src/backend/storage/aio/README.md | 424 ++++++++++++++++++++++++++++++
src/backend/storage/aio/aio.c | 2 +
2 files changed, 426 insertions(+)
create mode 100644 src/backend/storage/aio/README.md
diff --git a/src/backend/storage/aio/README.md b/src/backend/storage/aio/README.md
new file mode 100644
index 00000000000..ddd59404a59
--- /dev/null
+++ b/src/backend/storage/aio/README.md
@@ -0,0 +1,424 @@
+# Asynchronous & Direct IO
+
+## Motivation
+
+### Why Asynchronous IO
+
+Until the introduction of asynchronous IO Postgres relied on the operating
+system to hide the cost of synchronous IO from Postgres. While this worked
+surprisingly well in a lot of workloads, it does not do as good a job on
+prefetching and controlled writeback as we would like.
+
+There are important expensive operations like `fdatasync()` where the operating
+system cannot hide the storage latency. This is particularly important for WAL
+writes, where the ability to asynchronously issue `fdatasync()` or O_DSYNC
+writes can yield significantly higher throughput.
+
+
+### Why Direct / unbuffered IO
+
+The main reasons to want to use Direct IO are:
+
+- Lower CPU usage / higher throughput. Particularly on modern storage buffered
+ writes are bottlenecked by the operating system having to copy data from the
+ kernel's page cache to postgres buffer pool using the CPU. Whereas direct IO
+ can often move the data directly between the storage devices and postgres'
+ buffer cache, using DMA. While that transfer is ongoing, the CPU is free to
+ perform other work.
+- Reduced latency - Direct IO can have substantially lower latency than
+ buffered IO, which can be impactful for OLTP workloads bottlenecked by WAL
+ write latency.
+- Avoiding double buffering between operating system cache and postgres'
+ shared_buffers.
+- Better control over the timing and pace of dirty data writeback.
+
+
+The main reasons *not* to use Direct IO are:
+
+- Without AIO, Direct IO is unusably slow for most purposes.
+- Even with AIO, many parts of postgres need to be modified to perform
+ explicit prefetching.
+- In situations where shared_buffers cannot be set appropriately large,
+ e.g. because there are many different postgres instances hosted on shared
+ hardware, performance will often be worse than when using buffered IO.
+
+
+## AIO Usage Example
+
+In many cases code that can benefit from AIO does not directly have to
+interact with the AIO interface, but can use AIO via higher-level
+abstractions. See [Helpers](#helpers).
+
+In this example, a buffer will be read into shared buffers.
+
+```C
+/*
+ * Result of the operation, only to be accessed in this backend.
+ */
+PgAioReturn ioret;
+
+/*
+ * Acquire an AIO Handle, ioret will get result upon completion.
+ *
+ * Note that ioret needs to stay alive until the IO completes or
+ * CurrentResourceOwner is released (i.e. an error is thrown).
+ */
+PgAioHandle *ioh = pgaio_io_acquire(CurrentResourceOwner, &ioret);
+
+/*
+ * Reference that can be used to wait for the IO we initiate below. This
+ * reference can reside in local or shared memory and waited upon by any
+ * process. An arbitrary number of references can be made for each IO.
+ */
+PgAioWaitRef iow;
+
+pgaio_io_get_wref(ioh, &iow);
+
+/*
+ * Arrange for shared buffer completion callbacks to be called upon completion
+ * of the IO. This callback will update the buffer descriptors associated with
+ * the AioHandle, which e.g. allows other backends to access the buffer.
+ *
+ * A callback can be passed a small bit of data, e.g. to indicate whether to
+ * zero a buffer if it is invalid.
+ *
+ * Multiple completion callbacks can be registered for each handle.
+ */
+pgaio_io_register_callbacks(ioh, PGAIO_HCB_SHARED_BUFFER_READV, 0);
+
+/*
+ * The completion callback needs to know which buffers to update when the IO
+ * completes. As the AIO subsystem does not know about buffers, we have to
+ * associate this information with the AioHandle, for use by the completion
+ * callback registered above.
+ *
+ * In this example we're reading only a single buffer, hence the 1.
+ */
+pgaio_io_set_handle_data_32(ioh, (uint32 *) buffer, 1);
+
+/*
+ * Pass the AIO handle to lower-level function. When operating on the level of
+ * buffers, we don't know how exactly the IO is performed, that is the
+ * responsibility of the storage manager implementation.
+ *
+ * E.g. md.c needs to translate block numbers into offsets in segments.
+ *
+ * Once the IO handle has been handed off to smgstartreadv(), it may not
+ * further be used, as the IO may immediately get executed below
+ * smgrstartreadv() and the handle reused for another IO.
+ *
+ * To issue multiple IOs in an efficient way, a caller can call
+ * pgaio_enter_batchmode() before starting multiple IOs, and end that batch
+ * with pgaio_exit_batchmode(). Note that one needs to be careful while there
+ * may be unsubmitted IOs, as another backend may need to wait for one of the
+ * unsubmitted IOs. If this backend then had to wait for the other backend,
+ * it'd end in an undetected deadlock. See pgaio_enter_batchmode() for more
+ * details.
+ *
+ * Note that even while in batchmode an IO might get submitted immediately,
+ * e.g. due to reaching a limit on the number of unsubmitted IOs, and even
+ * complete before smgrstartreadv() returns.
+ */
+smgrstartreadv(ioh, operation->smgr, forknum, blkno,
+ BufferGetBlock(buffer), 1);
+
+/*
+ * To benefit from AIO, it is beneficial to perform other work, including
+ * submitting other IOs, before waiting for the IO to complete. Otherwise
+ * we could just have used synchronous, blocking IO.
+ */
+perform_other_work();
+
+/*
+ * We did some other work and now need the IO operation to have completed to
+ * continue.
+ */
+pgaio_wref_wait(&iow);
+
+/*
+ * At this point the IO has completed. We do not yet know whether it succeeded
+ * or failed, however. The buffer's state has been updated, which allows other
+ * backends to use the buffer (if the IO succeeded), or retry the IO (if it
+ * failed).
+ *
+ * Note that in case the IO has failed, a LOG message may have been emitted,
+ * but no ERROR has been raised. This is crucial, as another backend waiting
+ * for this IO should not see an ERROR.
+ *
+ * To check whether the operation succeeded, and to raise an ERROR, or if more
+ * appropriate LOG, the PgAioReturn we passed to pgaio_io_acquire() is used.
+ */
+if (ioret.result.status == PGAIO_RS_ERROR)
+ pgaio_result_report(ioret.result, &ioret.target_data, ERROR);
+
+/*
+ * Besides having succeeded completely, the IO could also have a) partially
+ * completed or b) succeeded with a warning (e.g. due to zero_damaged_pages).
+ * If we e.g. tried to read many blocks at once, the read might have
+ * only succeeded for the first few blocks.
+ *
+ * If the IO partially succeeded and this backend needs all blocks to have
+ * completed, this backend needs to reissue the IO for the remaining buffers.
+ * The AIO subsystem cannot handle this retry transparently.
+ *
+ * As this example is already long, and we only read a single block, we'll just
+ * error out if there's a partial read or a warning.
+ */
+if (ioret.result.status != PGAIO_RS_OK)
+ pgaio_result_report(ioret.result, &ioret.target_data, ERROR);
+
+/*
+ * The IO succeeded, so we can use the buffer now.
+ */
+```
+
+
+## Design Criteria & Motivation
+
+### Deadlock and Starvation Dangers due to AIO
+
+Using AIO in a naive way can easily lead to deadlocks in an environment where
+the source/target of AIO are shared resources, like pages in postgres'
+shared_buffers.
+
+Consider one backend performing readahead on a table, initiating IO for a
+number of buffers ahead of the current "scan position". If that backend then
+performs some operation that blocks, or even just is slow, the IO completion
+for the asynchronously initiated read may not be processed.
+
+This AIO implementation solves this problem by requiring that AIO methods
+either allow AIO completions to be processed by any backend in the system
+(e.g. io_uring), or to guarantee that AIO processing will happen even when the
+issuing backend is blocked (e.g. worker mode, which offloads completion
+processing to the AIO workers).
+
+
+### IO can be started in critical sections
+
+Using AIO for WAL writes can reduce the overhead of WAL logging substantially:
+
+- AIO allows to start WAL writes eagerly, so they complete before needing to
+ wait
+- AIO allows to have multiple WAL flushes in progress at the same time
+- AIO makes it more realistic to use O\_DIRECT + O\_DSYNC, which can reduce
+ the number of roundtrips to storage on some OSs and storage HW (buffered IO
+ and direct IO without O_DSYNC needs to issue a write and after the write's
+ completion a cache flush, whereas O\_DIRECT + O\_DSYNC can use a single
+ Force Unit Access (FUA) write).
+
+The need to be able to execute IO in critical sections has substantial design
+implication on the AIO subsystem. Mainly because completing IOs (see prior
+section) needs to be possible within a critical section, even if the
+to-be-completed IO itself was not issued in a critical section. Consider
+e.g. the case of a backend first starting a number of writes from shared
+buffers and then starting to flush the WAL. Because only a limited amount of
+IO can be in-progress at the same time, initiating IO for flushing the WAL may
+require to first complete IO that was started earlier.
+
+
+### State for AIO needs to live in shared memory
+
+Because postgres uses a process model and because AIOs need to be
+complete-able by any backend much of the state of the AIO subsystem needs to
+live in shared memory.
+
+In an `EXEC_BACKEND` build, a backend's executable code and other process
+local state is not necessarily mapped to the same addresses in each process
+due to ASLR. This means that the shared memory cannot contain pointers to
+callbacks.
+
+
+## Design of the AIO Subsystem
+
+
+### AIO Methods
+
+To achieve portability and performance, multiple methods of performing AIO are
+implemented and others are likely worth adding in the future.
+
+
+#### Synchronous Mode
+
+`io_method=sync` does not actually perform AIO but allows to use the AIO API
+while performing synchronous IO. This can be useful for debugging. The code
+for the synchronous mode is also used as a fallback by e.g. the [worker
+mode](#worker) uses it to execute IO that cannot be executed by workers.
+
+
+#### Worker
+
+`io_method=worker` is available on every platform postgres runs on, and
+implements asynchronous IO - from the view of the issuing process - by
+dispatching the IO to one of several worker processes performing the IO in a
+synchronous manner.
+
+
+#### io_uring
+
+`io_method=io_uring` is available on Linux 5.1+. In contrast to worker mode it
+dispatches all IO from within the process, lowering context switch rate /
+latency.
+
+
+### AIO Handles
+
+The central API piece for postgres' AIO abstraction are AIO handles. To
+execute an IO one first has to acquire an IO handle (`pgaio_io_acquire()`) and
+then "define" it, i.e. associate an IO operation with the handle.
+
+Often AIO handles are acquired on a higher level and then passed to a lower
+level to be fully defined. E.g., for IO to/from shared buffers, bufmgr.c
+routines acquire the handle, which is then passed through smgr.c, md.c to be
+finally fully defined in fd.c.
+
+The functions used at the lowest level to define the operation are
+`pgaio_io_start_*()`.
+
+Because acquisition of an IO handle
+[must always succeed](#io-can-be-started-in-critical-sections)
+and the number of AIO Handles
+[has to be limited](#state-for-aio-needs-to-live-in-shared-memory)
+AIO handles can be reused as soon as they have completed. Obviously code needs
+to be able to react to IO completion. State can be updated using
+[AIO Completion callbacks](#aio-callbacks)
+and the issuing backend can provide a backend local variable to receive the
+result of the IO, as described in
+[AIO Result](#aio-results).
+An IO can be waited for, by both the issuing and any other backend, using
+[AIO References](#aio-wait-references).
+
+
+Because an AIO Handle is not executable just after calling
+`pgaio_io_acquire()` and because `pgaio_io_acquire()` needs to always succeed
+(absent a PANIC), only a single AIO Handle may be acquired (i.e. returned by
+`pgaio_io_acquire()`) without causing the IO to have been defined (by,
+potentially indirectly, causing `pgaio_io_start_*()` to have been
+called). Otherwise a backend could trivially self-deadlock by using up all AIO
+Handles without the ability to wait for some of the IOs to complete.
+
+If it turns out that an AIO Handle is not needed, e.g., because the handle was
+acquired before holding a contended lock, it can be released without being
+defined using `pgaio_io_release()`.
+
+
+### AIO Callbacks
+
+Commonly several layers need to react to completion of an IO. E.g. for a read
+md.c needs to check if the IO outright failed or was shorter than needed,
+bufmgr.c needs to verify the page looks valid and bufmgr.c needs to update the
+BufferDesc to update the buffer's state.
+
+The fact that several layers / subsystems need to react to IO completion poses
+a few challenges:
+
+- Upper layers should not need to know details of lower layers. E.g. bufmgr.c
+ should not assume the IO will pass through md.c. Therefore upper levels
+ cannot know what lower layers would consider an error.
+
+- Lower layers should not need to know about upper layers. E.g. smgr APIs are
+ used going through shared buffers but are also used bypassing shared
+ buffers. This means that e.g. md.c is not in a position to validate
+ checksums.
+
+- Having code in the AIO subsystem for every possible combination of layers
+ would lead to a lot of duplication.
+
+The "solution" to this is the ability to associate multiple completion
+callbacks with a handle. E.g. bufmgr.c can have a callback to update the
+BufferDesc state and to verify the page and md.c can have another callback to
+check if the IO operation was successful.
+
+As [mentioned](#state-for-aio-needs-to-live-in-shared-memory), shared memory
+currently cannot contain function pointers. Because of that completion
+callbacks are not directly identified by function pointers but by IDs
+(`PgAioHandleCallbackID`). A substantial added benefit is that that
+allows callbacks to be identified by much smaller amount of memory (a single
+byte currently).
+
+In addition to completion, AIO callbacks also are called to "stage" an
+IO. This is, e.g., used to increase buffer reference counts to account for the
+AIO subsystem referencing the buffer, which is required to handle the case
+where the issuing backend errors out and releases its own pins while the IO is
+still ongoing.
+
+As [explained earlier](#io-can-be-started-in-critical-sections) IO completions
+need to be safe to execute in critical sections. To allow the backend that
+issued the IO to error out in case of failure [AIO Result](#aio-results) can
+be used.
+
+
+### AIO Targets
+
+In addition to the completion callbacks describe above, each AIO Handle has
+exactly one "target". Each target has some space inside an AIO Handle with
+information specific to the target and can provide callbacks to allow to
+reopen the underlying file (required for worker mode) and to describe the IO
+operation (used for debug logging and error messages).
+
+I.e., if two different uses of AIO can describe the identity of the file being
+operated on the same way, it likely makes sense to use the same
+target. E.g. different smgr implementations can describe IO with
+RelFileLocator, ForkNumber and BlockNumber and can thus share a target. In
+contrast, IO for a WAL file would be described with TimeLineID and XLogRecPtr
+and it would not make sense to use the same target for smgr and WAL.
+
+
+### AIO Wait References
+
+As [described above](#aio-handles), AIO Handles can be reused immediately
+after completion and therefore cannot be used to wait for completion of the
+IO. Waiting is enabled using AIO wait references, which do not just identify
+an AIO Handle but also include the handles "generation".
+
+A reference to an AIO Handle can be acquired using `pgaio_io_get_wref()` and
+then waited upon using `pgaio_wref_wait()`.
+
+
+### AIO Results
+
+As AIO completion callbacks
+[are executed in critical sections](#io-can-be-started-in-critical-sections)
+and [may be executed by any backend](#deadlock-and-starvation-dangers-due-to-aio)
+completion callbacks cannot be used to, e.g., make the query that triggered an
+IO ERROR out.
+
+To allow to react to failing IOs the issuing backend can pass a pointer to a
+`PgAioReturn` in backend local memory. Before an AIO Handle is reused the
+`PgAioReturn` is filled with information about the IO. This includes
+information about whether the IO was successful (as a value of
+`PgAioResultStatus`) and enough information to raise an error in case of a
+failure (via `pgaio_result_report()`, with the error details encoded in
+`PgAioResult`).
+
+
+### AIO Errors
+
+It would be very convenient to have shared completion callbacks encode the
+details of errors as an `ErrorData` that could be raised at a later
+time. Unfortunately doing so would require allocating memory. While elog.c can
+guarantee (well, kinda) that logging a message will not run out of memory,
+that only works because a very limited number of messages are in the process
+of being logged. With AIO a large number of concurrently issued AIOs might
+fail.
+
+To avoid the need for preallocating a potentially large amount of memory (in
+shared memory no less!), completion callbacks instead have to encode errors in
+a more compact format that can be converted into an error message.
+
+
+## Helpers
+
+Using the low-level AIO API introduces too much complexity to do so all over
+the tree. Most uses of AIO should be done via reusable, higher-level,
+helpers.
+
+
+### Read Stream
+
+A common and very beneficial use of AIO are reads where a substantial number
+of to-be-read locations are known ahead of time. E.g., for a sequential scan
+the set of blocks that need to be read can be determined solely by knowing the
+current position and checking the buffer mapping table.
+
+The [Read Stream](../../../include/storage/read_stream.h) interface makes it
+comparatively easy to use AIO for such use cases.
diff --git a/src/backend/storage/aio/aio.c b/src/backend/storage/aio/aio.c
index e3ed087e8a2..86f7250b7a5 100644
--- a/src/backend/storage/aio/aio.c
+++ b/src/backend/storage/aio/aio.c
@@ -24,6 +24,8 @@
*
* - read_stream.c - helper for reading buffered relation data
*
+ * - README.md - higher-level overview over AIO
+ *
*
* Portions Copyright (c) 1996-2025, PostgreSQL Global Development Group
* Portions Copyright (c) 1994, Regents of the University of California
--
2.48.1.76.g4e746b1a31.dirty
[text/x-diff] v2.15-0004-aio-Add-test_aio-module.patch (77.5K, ../../7s6fclfekpcxoaaorwrq67v4vmgixf2dcjcgznuj7vxs3ie3wq@okqbi5vabqbx/5-v2.15-0004-aio-Add-test_aio-module.patch)
download | inline diff:
From 090b74df45ac1ab491806282591a6b20be638117 Mon Sep 17 00:00:00 2001
From: Andres Freund <andres@anarazel.de>
Date: Tue, 18 Mar 2025 14:40:06 -0400
Subject: [PATCH v2.15 04/18] aio: Add test_aio module
To make the tests possible, a few functions from bufmgr.c/localbuf.c had to be
exported, via buf_internals.h.
Reviewed-by: Noah Misch <noah@leadboat.com>
Co-authored-by: Andres Freund <andres@anarazel.de>
Co-authored-by: Nazir Bilal Yavuz <byavuz81@gmail.com>
Discussion: https://postgr.es/m/uvrtrknj4kdytuboidbhwclo4gxhswwcpgadptsjvjqcluzmah%40brqs62irg4dt
---
src/include/storage/buf_internals.h | 7 +
src/backend/storage/buffer/bufmgr.c | 8 +-
src/backend/storage/buffer/localbuf.c | 3 +-
src/test/modules/Makefile | 1 +
src/test/modules/meson.build | 1 +
src/test/modules/test_aio/.gitignore | 2 +
src/test/modules/test_aio/Makefile | 26 +
src/test/modules/test_aio/meson.build | 37 +
src/test/modules/test_aio/t/001_aio.pl | 1504 +++++++++++++++++
src/test/modules/test_aio/t/002_io_workers.pl | 125 ++
src/test/modules/test_aio/test_aio--1.0.sql | 108 ++
src/test/modules/test_aio/test_aio.c | 804 +++++++++
src/test/modules/test_aio/test_aio.control | 3 +
13 files changed, 2621 insertions(+), 8 deletions(-)
create mode 100644 src/test/modules/test_aio/.gitignore
create mode 100644 src/test/modules/test_aio/Makefile
create mode 100644 src/test/modules/test_aio/meson.build
create mode 100644 src/test/modules/test_aio/t/001_aio.pl
create mode 100644 src/test/modules/test_aio/t/002_io_workers.pl
create mode 100644 src/test/modules/test_aio/test_aio--1.0.sql
create mode 100644 src/test/modules/test_aio/test_aio.c
create mode 100644 src/test/modules/test_aio/test_aio.control
diff --git a/src/include/storage/buf_internals.h b/src/include/storage/buf_internals.h
index 72b36a4af26..0dec7d93b3b 100644
--- a/src/include/storage/buf_internals.h
+++ b/src/include/storage/buf_internals.h
@@ -434,6 +434,12 @@ extern void IssuePendingWritebacks(WritebackContext *wb_context, IOContext io_co
extern void ScheduleBufferTagForWriteback(WritebackContext *wb_context,
IOContext io_context, BufferTag *tag);
+/* solely to make it easier to write tests */
+extern bool StartBufferIO(BufferDesc *buf, bool forInput, bool nowait);
+extern void TerminateBufferIO(BufferDesc *buf, bool clear_dirty, uint32 set_flag_bits,
+ bool forget_owner, bool release_aio);
+
+
/* freelist.c */
extern IOContext IOContextForStrategy(BufferAccessStrategy strategy);
extern BufferDesc *StrategyGetBuffer(BufferAccessStrategy strategy,
@@ -478,6 +484,7 @@ extern void TerminateLocalBufferIO(BufferDesc *bufHdr, bool clear_dirty,
uint32 set_flag_bits, bool release_aio);
extern bool StartLocalBufferIO(BufferDesc *bufHdr, bool forInput, bool nowait);
extern void FlushLocalBuffer(BufferDesc *bufHdr, SMgrRelation reln);
+extern void InvalidateLocalBuffer(BufferDesc *bufHdr, bool check_unreferenced);
extern void DropRelationLocalBuffers(RelFileLocator rlocator,
ForkNumber forkNum,
BlockNumber firstDelBlock);
diff --git a/src/backend/storage/buffer/bufmgr.c b/src/backend/storage/buffer/bufmgr.c
index f9681d09e1e..1c37d7dfe2f 100644
--- a/src/backend/storage/buffer/bufmgr.c
+++ b/src/backend/storage/buffer/bufmgr.c
@@ -518,10 +518,6 @@ static uint32 WaitBufHdrUnlocked(BufferDesc *buf);
static int SyncOneBuffer(int buf_id, bool skip_recently_used,
WritebackContext *wb_context);
static void WaitIO(BufferDesc *buf);
-static bool StartBufferIO(BufferDesc *buf, bool forInput, bool nowait);
-static void TerminateBufferIO(BufferDesc *buf, bool clear_dirty,
- uint32 set_flag_bits, bool forget_owner,
- bool release_aio);
static void AbortBufferIO(Buffer buffer);
static void shared_buffer_write_error_callback(void *arg);
static void local_buffer_write_error_callback(void *arg);
@@ -5962,7 +5958,7 @@ WaitIO(BufferDesc *buf)
* find out if they can perform the I/O as part of a larger operation, without
* waiting for the answer or distinguishing the reasons why not.
*/
-static bool
+bool
StartBufferIO(BufferDesc *buf, bool forInput, bool nowait)
{
uint32 buf_state;
@@ -6019,7 +6015,7 @@ StartBufferIO(BufferDesc *buf, bool forInput, bool nowait)
* resource owner. (forget_owner=false is used when the resource owner itself
* is being released)
*/
-static void
+void
TerminateBufferIO(BufferDesc *buf, bool clear_dirty, uint32 set_flag_bits,
bool forget_owner, bool release_aio)
{
diff --git a/src/backend/storage/buffer/localbuf.c b/src/backend/storage/buffer/localbuf.c
index bf89076bb10..ed56202af14 100644
--- a/src/backend/storage/buffer/localbuf.c
+++ b/src/backend/storage/buffer/localbuf.c
@@ -57,7 +57,6 @@ static int NLocalPinnedBuffers = 0;
static void InitLocalBuffers(void);
static Block GetLocalBufferStorage(void);
static Buffer GetLocalVictimBuffer(void);
-static void InvalidateLocalBuffer(BufferDesc *bufHdr, bool check_unreferenced);
/*
@@ -597,7 +596,7 @@ TerminateLocalBufferIO(BufferDesc *bufHdr, bool clear_dirty, uint32 set_flag_bit
*
* See also InvalidateBuffer().
*/
-static void
+void
InvalidateLocalBuffer(BufferDesc *bufHdr, bool check_unreferenced)
{
Buffer buffer = BufferDescriptorGetBuffer(bufHdr);
diff --git a/src/test/modules/Makefile b/src/test/modules/Makefile
index 4e4be3fa511..aa1d27bbed3 100644
--- a/src/test/modules/Makefile
+++ b/src/test/modules/Makefile
@@ -14,6 +14,7 @@ SUBDIRS = \
oauth_validator \
plsample \
spgist_name_ops \
+ test_aio \
test_bloomfilter \
test_copy_callbacks \
test_custom_rmgrs \
diff --git a/src/test/modules/meson.build b/src/test/modules/meson.build
index 2b057451473..9de0057bd1d 100644
--- a/src/test/modules/meson.build
+++ b/src/test/modules/meson.build
@@ -13,6 +13,7 @@ subdir('oauth_validator')
subdir('plsample')
subdir('spgist_name_ops')
subdir('ssl_passphrase_callback')
+subdir('test_aio')
subdir('test_bloomfilter')
subdir('test_copy_callbacks')
subdir('test_custom_rmgrs')
diff --git a/src/test/modules/test_aio/.gitignore b/src/test/modules/test_aio/.gitignore
new file mode 100644
index 00000000000..716e17f5a2a
--- /dev/null
+++ b/src/test/modules/test_aio/.gitignore
@@ -0,0 +1,2 @@
+# Generated subdirectories
+/tmp_check/
diff --git a/src/test/modules/test_aio/Makefile b/src/test/modules/test_aio/Makefile
new file mode 100644
index 00000000000..f53cc64671a
--- /dev/null
+++ b/src/test/modules/test_aio/Makefile
@@ -0,0 +1,26 @@
+# src/test/modules/test_aio/Makefile
+
+PGFILEDESC = "test_aio - test code for AIO"
+
+MODULE_big = test_aio
+OBJS = \
+ $(WIN32RES) \
+ test_aio.o
+
+EXTENSION = test_aio
+DATA = test_aio--1.0.sql
+
+TAP_TESTS = 1
+
+export enable_injection_points
+
+ifdef USE_PGXS
+PG_CONFIG = pg_config
+PGXS := $(shell $(PG_CONFIG) --pgxs)
+include $(PGXS)
+else
+subdir = src/test/modules/test_aio
+top_builddir = ../../../..
+include $(top_builddir)/src/Makefile.global
+include $(top_srcdir)/contrib/contrib-global.mk
+endif
diff --git a/src/test/modules/test_aio/meson.build b/src/test/modules/test_aio/meson.build
new file mode 100644
index 00000000000..73d2fd68eaa
--- /dev/null
+++ b/src/test/modules/test_aio/meson.build
@@ -0,0 +1,37 @@
+# Copyright (c) 2024-2025, PostgreSQL Global Development Group
+
+test_aio_sources = files(
+ 'test_aio.c',
+)
+
+if host_system == 'windows'
+ test_aio_sources += rc_lib_gen.process(win32ver_rc, extra_args: [
+ '--NAME', 'test_aio',
+ '--FILEDESC', 'test_aio - test code for AIO',])
+endif
+
+test_aio = shared_module('test_aio',
+ test_aio_sources,
+ kwargs: pg_test_mod_args,
+)
+test_install_libs += test_aio
+
+test_install_data += files(
+ 'test_aio.control',
+ 'test_aio--1.0.sql',
+)
+
+tests += {
+ 'name': 'test_aio',
+ 'sd': meson.current_source_dir(),
+ 'bd': meson.current_build_dir(),
+ 'tap': {
+ 'env': {
+ 'enable_injection_points': get_option('injection_points') ? 'yes' : 'no',
+ },
+ 'tests': [
+ 't/001_aio.pl',
+ 't/002_io_workers.pl',
+ ],
+ },
+}
diff --git a/src/test/modules/test_aio/t/001_aio.pl b/src/test/modules/test_aio/t/001_aio.pl
new file mode 100644
index 00000000000..58a56c8cec2
--- /dev/null
+++ b/src/test/modules/test_aio/t/001_aio.pl
@@ -0,0 +1,1504 @@
+# Copyright (c) 2025, PostgreSQL Global Development Group
+
+use strict;
+use warnings FATAL => 'all';
+
+use PostgreSQL::Test::Cluster;
+use PostgreSQL::Test::Utils;
+use Test::More;
+
+
+###
+# Test io_method=worker
+###
+my $node_worker = create_node('worker');
+$node_worker->start();
+
+test_generic('worker', $node_worker);
+SKIP:
+{
+ skip 'Injection points not supported by this build', 1
+ unless $ENV{enable_injection_points} eq 'yes';
+ test_inject_worker('worker', $node_worker);
+}
+
+$node_worker->stop();
+
+
+###
+# Test io_method=io_uring
+###
+
+if (have_io_uring())
+{
+ my $node_uring = create_node('io_uring');
+ $node_uring->start();
+ test_generic('io_uring', $node_uring);
+ $node_uring->stop();
+}
+
+
+###
+# Test io_method=sync
+###
+
+my $node_sync = create_node('sync');
+
+# just to have one test not use the default auto-tuning
+
+$node_sync->append_conf(
+ 'postgresql.conf', qq(
+io_max_concurrency=4
+));
+
+$node_sync->start();
+test_generic('sync', $node_sync);
+$node_sync->stop();
+
+done_testing();
+
+
+###
+# Test Helpers
+###
+
+sub create_node
+{
+ local $Test::Builder::Level = $Test::Builder::Level + 1;
+
+ my $io_method = shift;
+
+ my $node = PostgreSQL::Test::Cluster->new($io_method);
+
+ # Want to test initdb for each IO method, otherwise we could just reuse
+ # the cluster.
+ #
+ # Unfortunately Cluster::init() puts PG_TEST_INITDB_EXTRA_OPTS after the
+ # options specified by ->extra, if somebody puts -c io_method=xyz in
+ # PG_TEST_INITDB_EXTRA_OPTS it would break this test. Fix that up if we
+ # detect it.
+ local $ENV{PG_TEST_INITDB_EXTRA_OPTS} = $ENV{PG_TEST_INITDB_EXTRA_OPTS};
+ if (defined $ENV{PG_TEST_INITDB_EXTRA_OPTS}
+ && $ENV{PG_TEST_INITDB_EXTRA_OPTS} =~ m/io_method=/)
+ {
+ $ENV{PG_TEST_INITDB_EXTRA_OPTS} .= " -c io_method=$io_method";
+ }
+
+ $node->init(extra => [ '-c', "io_method=$io_method" ]);
+
+ $node->append_conf(
+ 'postgresql.conf', qq(
+shared_preload_libraries=test_aio
+log_min_messages = 'DEBUG3'
+log_statement=all
+log_error_verbosity=default
+restart_after_crash=false
+temp_buffers=100
+));
+
+ ok(1, "$io_method: initdb");
+
+ return $node;
+}
+
+sub have_io_uring
+{
+ # To detect if io_uring is supported, we look at the error message for
+ # assigning an invalid value to an enum GUC, which lists all the valid
+ # options. We need to use -C to deal with running as administrator on
+ # windows, the superuser check is omitted if -C is used.
+ my ($stdout, $stderr) =
+ run_command [qw(postgres -C invalid -c io_method=invalid)];
+ die "can't determine supported io_method values"
+ unless $stderr =~ m/Available values: ([^\.]+)\./;
+ my $methods = $1;
+ note "supported io_method values are: $methods";
+
+ return ($methods =~ m/io_uring/) ? 1 : 0;
+}
+
+sub psql_like
+{
+ local $Test::Builder::Level = $Test::Builder::Level + 1;
+ my $io_method = shift;
+ my $psql = shift;
+ my $name = shift;
+ my $sql = shift;
+ my $expected_stdout = shift;
+ my $expected_stderr = shift;
+ my ($cmdret, $output);
+
+ ($output, $cmdret) = $psql->query($sql);
+
+ like($output, $expected_stdout, "$io_method: $name: expected stdout");
+ like($psql->{stderr}, $expected_stderr,
+ "$io_method: $name: expected stderr");
+ $psql->{stderr} = '';
+
+ return $output;
+}
+
+sub query_wait_block
+{
+ local $Test::Builder::Level = $Test::Builder::Level + 1;
+ my $io_method = shift;
+ my $node = shift;
+ my $psql = shift;
+ my $name = shift;
+ my $sql = shift;
+ my $waitfor = shift;
+
+ my $pid = $psql->query_safe('SELECT pg_backend_pid()');
+
+ $psql->{stdin} .= qq($sql;\n);
+ $psql->{run}->pump_nb();
+ ok(1, "$io_method: $name: issued sql");
+
+ $node->poll_query_until('postgres',
+ qq(SELECT wait_event FROM pg_stat_activity WHERE pid = $pid),
+ $waitfor);
+ ok(1, "$io_method: $name: observed $waitfor wait event");
+}
+
+# Returns count of checksum failures for the specified database or for shared
+# relations, if $datname is undefined.
+sub checksum_failures
+{
+ local $Test::Builder::Level = $Test::Builder::Level + 1;
+ my $psql = shift;
+ my $datname = shift;
+ my $checksum_count;
+ my $checksum_last_failure;
+
+ if (defined $datname)
+ {
+ $checksum_count = $psql->query_safe(
+ qq(
+SELECT checksum_failures FROM pg_stat_database WHERE datname = '$datname';
+));
+ $checksum_last_failure = $psql->query_safe(
+ qq(
+SELECT checksum_last_failure FROM pg_stat_database WHERE datname = '$datname';
+));
+ }
+ else
+ {
+ $checksum_count = $psql->query_safe(
+ qq(
+SELECT checksum_failures FROM pg_stat_database WHERE datname IS NULL;
+));
+ $checksum_last_failure = $psql->query_safe(
+ qq(
+SELECT checksum_last_failure FROM pg_stat_database WHERE datname IS NULL;
+));
+ }
+
+ return $checksum_count, $checksum_last_failure;
+}
+
+###
+# Sub-tests
+###
+
+# Sanity checks for the IO handle API
+sub test_handle
+{
+ my $io_method = shift;
+ my $node = shift;
+
+ my $psql = $node->background_psql('postgres', on_error_stop => 0);
+
+ # leak warning: implicit xact
+ psql_like(
+ $io_method,
+ $psql,
+ "handle_get() leak in implicit xact",
+ qq(SELECT handle_get()),
+ qr/^$/,
+ qr/leaked AIO handle/,
+ "$io_method: leaky handle_get() warns");
+
+ # leak warning: explicit xact
+ psql_like(
+ $io_method, $psql,
+ "handle_get() leak in explicit xact",
+ qq(BEGIN; SELECT handle_get(); COMMIT),
+ qr/^$/, qr/leaked AIO handle/);
+
+
+ # leak warning: explicit xact, rollback
+ psql_like(
+ $io_method,
+ $psql,
+ "handle_get() leak in explicit xact, rollback",
+ qq(BEGIN; SELECT handle_get(); ROLLBACK;),
+ qr/^$/,
+ qr/leaked AIO handle/);
+
+ # leak warning: subtrans
+ psql_like(
+ $io_method,
+ $psql,
+ "handle_get() leak in subxact",
+ qq(BEGIN; SAVEPOINT foo; SELECT handle_get(); COMMIT;),
+ qr/^$/,
+ qr/leaked AIO handle/);
+
+ # leak warning + error: released in different command (thus resowner)
+ psql_like(
+ $io_method,
+ $psql,
+ "handle_release() in different command",
+ qq(BEGIN; SELECT handle_get(); SELECT handle_release_last(); COMMIT;),
+ qr/^$/,
+ qr/leaked AIO handle.*release in unexpected state/ms);
+
+ # no leak, release in same command
+ psql_like(
+ $io_method,
+ $psql,
+ "handle_release() in same command",
+ qq(BEGIN; SELECT handle_get() UNION ALL SELECT handle_release_last(); COMMIT;),
+ qr/^$/,
+ qr/^$/);
+
+ # normal handle use
+ psql_like($io_method, $psql, "handle_get_release()",
+ qq(SELECT handle_get_release()),
+ qr/^$/, qr/^$/);
+
+ # should error out, API violation
+ psql_like(
+ $io_method,
+ $psql,
+ "handle_get_twice()",
+ qq(SELECT handle_get_twice()),
+ qr/^$/,
+ qr/ERROR: API violation: Only one IO can be handed out$/);
+
+ # recover after error in implicit xact
+ psql_like(
+ $io_method,
+ $psql,
+ "handle error recovery in implicit xact",
+ qq(SELECT handle_get_and_error(); SELECT 'ok', handle_get_release()),
+ qr/^|ok$/,
+ qr/ERROR.*as you command/);
+
+ # recover after error in implicit xact
+ psql_like(
+ $io_method,
+ $psql,
+ "handle error recovery in explicit xact",
+ qq(BEGIN; SELECT handle_get_and_error(); SELECT handle_get_release(), 'ok'; COMMIT;),
+ qr/^|ok$/,
+ qr/ERROR.*as you command/);
+
+ # recover after error in subtrans
+ psql_like(
+ $io_method,
+ $psql,
+ "handle error recovery in explicit subxact",
+ qq(BEGIN; SAVEPOINT foo; SELECT handle_get_and_error(); ROLLBACK TO SAVEPOINT foo; SELECT handle_get_release(); ROLLBACK;),
+ qr/^|ok$/,
+ qr/ERROR.*as you command/);
+
+ $psql->quit();
+}
+
+# Sanity checks for the batchmode API
+sub test_batchmode
+{
+ my $io_method = shift;
+ my $node = shift;
+
+ my $psql = $node->background_psql('postgres', on_error_stop => 0);
+
+ # leak warning & recovery: implicit xact
+ psql_like(
+ $io_method,
+ $psql,
+ "batch_start() leak & cleanup in implicit xact",
+ qq(SELECT batch_start()),
+ qr/^$/,
+ qr/open AIO batch at end/,
+ "$io_method: leaky batch_start() warns");
+
+ # leak warning & recovery: explicit xact
+ psql_like(
+ $io_method,
+ $psql,
+ "batch_start() leak & cleanup in explicit xact",
+ qq(BEGIN; SELECT batch_start(); COMMIT;),
+ qr/^$/,
+ qr/open AIO batch at end/,
+ "$io_method: leaky batch_start() warns");
+
+
+ # leak warning & recovery: explicit xact, rollback
+ #
+ # XXX: This doesn't fail right now, due to not getting a chance to do
+ # something at transaction command commit. That's not a correctness issue,
+ # it just means it's a bit harder to find buggy code.
+ #psql_like($io_method, $psql,
+ # "batch_start() leak & cleanup after abort",
+ # qq(BEGIN; SELECT batch_start(); ROLLBACK;),
+ # qr/^$/,
+ # qr/open AIO batch at end/, "$io_method: leaky batch_start() warns");
+
+ # no warning, batch closed in same command
+ psql_like(
+ $io_method,
+ $psql,
+ "batch_start(), batch_end() works",
+ qq(SELECT batch_start() UNION ALL SELECT batch_end()),
+ qr/^$/,
+ qr/^$/,
+ "$io_method: batch_start(), batch_end()");
+
+ $psql->quit();
+}
+
+# Test that simple cases of invalid pages are reported
+sub test_io_error
+{
+ my $io_method = shift;
+ my $node = shift;
+ my ($ret, $output);
+
+ my $psql = $node->background_psql('postgres', on_error_stop => 0);
+
+ $psql->query_safe(
+ qq(
+CREATE TEMPORARY TABLE tmp_corr(data int not null);
+INSERT INTO tmp_corr SELECT generate_series(1, 10000);
+SELECT modify_rel_block('tmp_corr', 1, corrupt_header=>true);
+));
+
+ foreach my $tblname (qw(tbl_corr tmp_corr))
+ {
+ my $invalid_page_re =
+ $tblname eq 'tbl_corr'
+ ? qr/invalid page in block 1 of relation base\/\d+\/\d+/
+ : qr/invalid page in block 1 of relation base\/\d+\/t\d+_\d+/;
+
+ # verify the error is reported in custom C code
+ psql_like(
+ $io_method,
+ $psql,
+ "read_rel_block_ll() of $tblname page",
+ qq(SELECT read_rel_block_ll('$tblname', 1)),
+ qr/^$/,
+ $invalid_page_re);
+
+ # verify the error is reported for bufmgr reads, seq scan
+ psql_like(
+ $io_method, $psql,
+ "sequential scan of $tblname block fails",
+ qq(SELECT count(*) FROM $tblname),
+ qr/^$/, $invalid_page_re);
+
+ # verify the error is reported for bufmgr reads, tid scan
+ psql_like(
+ $io_method,
+ $psql,
+ "tid scan of $tblname block fails",
+ qq(SELECT count(*) FROM $tblname WHERE ctid = '(1, 1)'),
+ qr/^$/,
+ $invalid_page_re);
+ }
+
+ $psql->quit();
+}
+
+# Test interplay between StartBufferIO and TerminateBufferIO
+sub test_startwait_io
+{
+ my $io_method = shift;
+ my $node = shift;
+ my ($ret, $output);
+
+ my $psql_a = $node->background_psql('postgres', on_error_stop => 0);
+ my $psql_b = $node->background_psql('postgres', on_error_stop => 0);
+
+
+ ### Verify behavior for normal tables
+
+ # create a buffer we can play around with
+ my $buf_id = psql_like(
+ $io_method, $psql_a,
+ "creation of toy buffer succeeds",
+ qq(SELECT buffer_create_toy('tbl_ok', 1)),
+ qr/^\d+$/, qr/^$/);
+
+ # check that one backend can perform StartBufferIO
+ psql_like(
+ $io_method,
+ $psql_a,
+ "first StartBufferIO",
+ qq(SELECT buffer_call_start_io($buf_id, for_input=>true, nowait=>false);),
+ qr/^t$/,
+ qr/^$/);
+
+ # but not twice on the same buffer (non-waiting)
+ psql_like(
+ $io_method,
+ $psql_a,
+ "second StartBufferIO fails, same session",
+ qq(SELECT buffer_call_start_io($buf_id, for_input=>true, nowait=>true);),
+ qr/^f$/,
+ qr/^$/);
+ psql_like(
+ $io_method,
+ $psql_b,
+ "second StartBufferIO fails, other session",
+ qq(SELECT buffer_call_start_io($buf_id, for_input=>true, nowait=>true);),
+ qr/^f$/,
+ qr/^$/);
+
+ # start io in a different session, will block
+ query_wait_block(
+ $io_method,
+ $node,
+ $psql_b,
+ "blocking start buffer io",
+ qq(SELECT buffer_call_start_io($buf_id, for_input=>true, nowait=>false);),
+ "BufferIo");
+
+ # Terminate the IO, without marking it as success, this should trigger the
+ # waiting session to be able to start the io
+ psql_like(
+ $io_method,
+ $psql_a,
+ "blocking start buffer io, terminating io, not valid",
+ qq(SELECT buffer_call_terminate_io($buf_id, for_input=>true, succeed=>false, io_error=>false, release_aio=>false)),
+ qr/^$/,
+ qr/^$/);
+
+
+ # Because the IO was terminated, but not marked as valid, second session should get the right to start io
+ pump_until($psql_b->{run}, $psql_b->{timeout}, \$psql_b->{stdout}, qr/t/);
+ ok(1, "$io_method: blocking start buffer io, can start io");
+
+ # terminate the IO again
+ $psql_b->query_safe(
+ qq(SELECT buffer_call_terminate_io($buf_id, for_input=>true, succeed=>false, io_error=>false, release_aio=>false);)
+ );
+
+
+ # same as the above scenario, but mark IO as having succeeded
+ psql_like(
+ $io_method,
+ $psql_a,
+ "blocking buffer io w/ success: first start buffer io",
+ qq(SELECT buffer_call_start_io($buf_id, for_input=>true, nowait=>false);),
+ qr/^t$/,
+ qr/^$/);
+
+ # start io in a different session, will block
+ query_wait_block(
+ $io_method,
+ $node,
+ $psql_b,
+ "blocking start buffer io",
+ qq(SELECT buffer_call_start_io($buf_id, for_input=>true, nowait=>false);),
+ "BufferIo");
+
+ # Terminate the IO, marking it as success
+ psql_like(
+ $io_method,
+ $psql_a,
+ "blocking start buffer io, terminating io, valid",
+ qq(SELECT buffer_call_terminate_io($buf_id, for_input=>true, succeed=>true, io_error=>false, release_aio=>false)),
+ qr/^$/,
+ qr/^$/);
+
+ # Because the IO was terminated, and marked as valid, second session should complete but not need io
+ pump_until($psql_b->{run}, $psql_b->{timeout}, \$psql_b->{stdout}, qr/f/);
+ ok(1, "$io_method: blocking start buffer io, no need to start io");
+
+ # buffer is valid now, make it invalid again
+ $psql_a->query_safe(qq(SELECT buffer_create_toy('tbl_ok', 1);));
+
+
+ ### Verify behavior for temporary tables
+
+ # Can't unfortunately share the code with the normal table case, there are
+ # too many behavioral differences.
+
+ # create a buffer we can play around with
+ $psql_a->query_safe(
+ qq(
+CREATE TEMPORARY TABLE tmp_ok(data int not null);
+INSERT INTO tmp_ok SELECT generate_series(1, 10000);
+));
+ $buf_id = $psql_a->query_safe(qq(SELECT buffer_create_toy('tmp_ok', 3);));
+
+ # check that one backend can perform StartLocalBufferIO
+ psql_like(
+ $io_method,
+ $psql_a,
+ "first StartLocalBufferIO",
+ qq(SELECT buffer_call_start_io($buf_id, for_input=>true, nowait=>true);),
+ qr/^t$/,
+ qr/^$/);
+
+ # Because local buffers don't use IO_IN_PROGRESS, a second StartLocalBufer
+ # succeeds as well. This test mostly serves as a documentation of that
+ # fact. If we had actually started IO, it'd be different.
+ psql_like(
+ $io_method,
+ $psql_a,
+ "second StartLocalBufferIO succeeds, same session",
+ qq(SELECT buffer_call_start_io($buf_id, for_input=>true, nowait=>true);),
+ qr/^t$/,
+ qr/^$/);
+
+ # Terminate the IO again, without marking it as a success
+ $psql_a->query_safe(
+ qq(SELECT buffer_call_terminate_io($buf_id, for_input=>true, succeed=>false, io_error=>false, release_aio=>false);)
+ );
+ psql_like(
+ $io_method,
+ $psql_a,
+ "StartLocalBufferIO after not marking valid succeeds, same session",
+ qq(SELECT buffer_call_start_io($buf_id, for_input=>true, nowait=>true);),
+ qr/^t$/,
+ qr/^$/);
+
+ # Terminate the IO again, marking it as a success
+ $psql_a->query_safe(
+ qq(SELECT buffer_call_terminate_io($buf_id, for_input=>true, succeed=>true, io_error=>false, release_aio=>false);)
+ );
+
+ # Now another StartLocalBufferIO should fail, this time because the buffer
+ # is already valid.
+ psql_like(
+ $io_method,
+ $psql_a,
+ "StartLocalBufferIO after marking valid fails",
+ qq(SELECT buffer_call_start_io($buf_id, for_input=>true, nowait=>false);),
+ qr/^f$/,
+ qr/^$/);
+
+ $psql_a->quit();
+ $psql_b->quit();
+}
+
+# Test that if the backend issuing a read doesn't wait for the IO's
+# completion, another backend can complete the IO
+sub test_complete_foreign
+{
+ my $io_method = shift;
+ my $node = shift;
+ my ($ret, $output);
+
+ my $psql_a = $node->background_psql('postgres', on_error_stop => 0);
+ my $psql_b = $node->background_psql('postgres', on_error_stop => 0);
+
+ # Issue IO without waiting for completion, then sleep
+ $psql_a->query_safe(
+ qq(SELECT read_rel_block_ll('tbl_ok', 1, wait_complete=>false);));
+
+ # Check that another backend can read the relevant block
+ psql_like(
+ $io_method,
+ $psql_b,
+ "completing read started by sleeping backend",
+ qq(SELECT count(*) FROM tbl_ok WHERE ctid = '(1,1)' LIMIT 1),
+ qr/^1$/,
+ qr/^$/);
+
+ # Issue IO without waiting for completion, then exit.
+ $psql_a->query_safe(
+ qq(SELECT read_rel_block_ll('tbl_ok', 1, wait_complete=>false);));
+ $psql_a->reconnect_and_clear();
+
+ # Check that another backend can read the relevant block. This verifies
+ # that the exiting backend left the AIO in a sane state.
+ psql_like(
+ $io_method,
+ $psql_b,
+ "read buffer started by exited backend",
+ qq(SELECT count(*) FROM tbl_ok WHERE ctid = '(1,1)' LIMIT 1),
+ qr/^1$/,
+ qr/^$/);
+
+ # Read a tbl_corr block, then sleep. The other session will retry the IO
+ # and also fail. The easiest thing to verify that seems to be to check
+ # that both are in the log.
+ my $log_location = -s $node->logfile;
+ $psql_a->query_safe(
+ qq(SELECT read_rel_block_ll('tbl_corr', 1, wait_complete=>false);));
+
+ psql_like(
+ $io_method,
+ $psql_b,
+ "completing read of tbl_corr block started by other backend",
+ qq(SELECT count(*) FROM tbl_corr WHERE ctid = '(1,1)' LIMIT 1),
+ qr/^$/,
+ qr/invalid page in block/);
+
+ # The log message issued for the read_rel_block_ll() should be logged as a LOG
+ $node->wait_for_log(qr/LOG[^\n]+invalid page in/, $log_location);
+ ok(1,
+ "$io_method: completing read of tbl_corr block started by other backend: LOG message for background read"
+ );
+
+ # But for the SELECT, it should be an ERROR
+ $log_location =
+ $node->wait_for_log(qr/ERROR[^\n]+invalid page in/, $log_location);
+ ok(1,
+ "$io_method: completing read of tbl_corr block started by other backend: ERROR message for foreground read"
+ );
+
+ $psql_a->quit();
+ $psql_b->quit();
+}
+
+# Test that we deal correctly with FDs being closed while IO is in progress
+sub test_close_fd
+{
+ my $io_method = shift;
+ my $node = shift;
+ my ($ret, $output);
+
+ my $psql = $node->background_psql('postgres', on_error_stop => 0);
+
+ psql_like(
+ $io_method,
+ $psql,
+ "close all FDs after read, waiting for results",
+ qq(
+ SELECT read_rel_block_ll('tbl_ok', 1,
+ wait_complete=>true,
+ batchmode_enter=>true,
+ smgrreleaseall=>true,
+ batchmode_exit=>true
+ );),
+ qr/^$/,
+ qr/^$/);
+
+ psql_like(
+ $io_method,
+ $psql,
+ "close all FDs after read, no waiting",
+ qq(
+ SELECT read_rel_block_ll('tbl_ok', 1,
+ wait_complete=>false,
+ batchmode_enter=>true,
+ smgrreleaseall=>true,
+ batchmode_exit=>true
+ );),
+ qr/^$/,
+ qr/^$/);
+
+ # Check that another backend can read the relevant block
+ psql_like(
+ $io_method,
+ $psql,
+ "close all FDs after read, no waiting, query works",
+ qq(SELECT count(*) FROM tbl_ok WHERE ctid = '(1,1)' LIMIT 1),
+ qr/^1$/,
+ qr/^$/);
+
+ $psql->quit();
+}
+
+# Tests using injection points. Mostly to exercise hard IO errors that are
+# hard to trigger without using injection points.
+sub test_inject
+{
+ my $io_method = shift;
+ my $node = shift;
+ my ($ret, $output);
+
+ my $psql = $node->background_psql('postgres', on_error_stop => 0);
+
+ # injected what we'd expect
+ $psql->query_safe(qq(SELECT inj_io_short_read_attach(8192);));
+ $psql->query_safe(qq(SELECT invalidate_rel_block('tbl_ok', 2);));
+ psql_like(
+ $io_method, $psql,
+ "injection point not triggering failure",
+ qq(SELECT count(*) FROM tbl_ok WHERE ctid = '(2, 1)'),
+ qr/^1$/, qr/^$/);
+
+ # injected a read shorter than a single block, expecting error
+ $psql->query_safe(qq(SELECT inj_io_short_read_attach(17);));
+ $psql->query_safe(qq(SELECT invalidate_rel_block('tbl_ok', 2);));
+ psql_like(
+ $io_method,
+ $psql,
+ "single block short read fails",
+ qq(SELECT count(*) FROM tbl_ok WHERE ctid = '(2, 1)'),
+ qr/^$/,
+ qr/ERROR:.*could not read blocks 2\.\.2 in file "base\/.*": read only 0 of 8192 bytes/
+ );
+
+ # shorten multi-block read to a single block, should retry
+ my $inval_query = qq(SELECT invalidate_rel_block('tbl_ok', 0);
+SELECT invalidate_rel_block('tbl_ok', 1);
+SELECT invalidate_rel_block('tbl_ok', 2);
+SELECT invalidate_rel_block('tbl_ok', 3);
+/* gap */
+SELECT invalidate_rel_block('tbl_ok', 5);
+SELECT invalidate_rel_block('tbl_ok', 6);
+SELECT invalidate_rel_block('tbl_ok', 7);
+SELECT invalidate_rel_block('tbl_ok', 8););
+
+ $psql->query_safe($inval_query);
+ $psql->query_safe(qq(SELECT inj_io_short_read_attach(8192);));
+ psql_like(
+ $io_method, $psql,
+ "multi block short read (1 block) is retried",
+ qq(SELECT count(*) FROM tbl_ok),
+ qr/^10000$/, qr/^$/);
+
+ # shorten multi-block read to two blocks, should retry
+ $psql->query_safe($inval_query);
+ $psql->query_safe(qq(SELECT inj_io_short_read_attach(8192*2);));
+
+ psql_like(
+ $io_method, $psql,
+ "multi block short read (2 blocks) is retried",
+ qq(SELECT count(*) FROM tbl_ok),
+ qr/^10000$/, qr/^$/);
+
+ # verify that page verification errors are detected even as part of a
+ # shortened multi-block read (tbl_corr, block 1 is corrupted)
+ $psql->query_safe(
+ qq(
+SELECT invalidate_rel_block('tbl_corr', 0);
+SELECT invalidate_rel_block('tbl_corr', 1);
+SELECT invalidate_rel_block('tbl_corr', 2);
+SELECT inj_io_short_read_attach(8192);
+ ));
+
+ psql_like(
+ $io_method,
+ $psql,
+ "shortened multi-block read detects invalid page",
+ qq(SELECT count(*) FROM tbl_corr WHERE ctid < '(2, 1)'),
+ qr/^$/,
+ qr/ERROR:.*invalid page in block 1 of relation base\/.*/);
+
+ # trigger a hard error, should error out
+ $psql->query_safe(
+ qq(
+SELECT inj_io_short_read_attach(-errno_from_string('EIO'));
+SELECT invalidate_rel_block('tbl_ok', 2);
+ ));
+
+ psql_like(
+ $io_method,
+ $psql,
+ "first hard IO error is reported",
+ qq(SELECT count(*) FROM tbl_ok),
+ qr/^$/,
+ qr/ERROR:.*could not read blocks 2\.\.2 in file \"base\/.*\": Input\/output error/
+ );
+
+ psql_like(
+ $io_method,
+ $psql,
+ "second hard IO error is reported",
+ qq(SELECT count(*) FROM tbl_ok),
+ qr/^$/,
+ qr/ERROR:.*could not read blocks 2\.\.2 in file \"base\/.*\": Input\/output error/
+ );
+
+ $psql->query_safe(qq(SELECT inj_io_short_read_detach()));
+
+ # now the IO should be ok.
+ psql_like(
+ $io_method, $psql,
+ "recovers after hard error",
+ qq(SELECT count(*) FROM tbl_ok),
+ qr/^10000$/, qr/^$/);
+
+ # trigger a different hard error, should error out
+ $psql->query_safe(
+ qq(
+SELECT inj_io_short_read_attach(-errno_from_string('EROFS'));
+SELECT invalidate_rel_block('tbl_ok', 2);
+ ));
+ psql_like(
+ $io_method,
+ $psql,
+ "different hard IO error is reported",
+ qq(SELECT count(*) FROM tbl_ok),
+ qr/^$/,
+ qr/ERROR:.*could not read blocks 2\.\.2 in file \"base\/.*\": Read-only file system/
+ );
+ $psql->query_safe(qq(SELECT inj_io_short_read_detach()));
+
+ $psql->quit();
+}
+
+# Tests using injection points, only for io_method=worker.
+#
+# io_method=worker has the special case of needing to reopen files. That can
+# in theory fail, because the file could be gone. That's a hard path to test
+# for real, so we use an injection point to trigger it.
+sub test_inject_worker
+{
+ my $io_method = shift;
+ my $node = shift;
+ my ($ret, $output);
+
+ my $psql = $node->background_psql('postgres', on_error_stop => 0);
+
+ # trigger a failure to reopen, should error out, but should recover
+ $psql->query_safe(
+ qq(
+SELECT inj_io_reopen_attach();
+SELECT invalidate_rel_block('tbl_ok', 1);
+ ));
+
+ psql_like(
+ $io_method,
+ $psql,
+ "failure to open: detected",
+ qq(SELECT count(*) FROM tbl_ok),
+ qr/^$/,
+ qr/ERROR:.*could not read blocks 1\.\.1 in file "base\/.*": No such file or directory/
+ );
+
+ $psql->query_safe(qq(SELECT inj_io_reopen_detach();));
+
+ # check that we indeed recover
+ psql_like(
+ $io_method, $psql,
+ "failure to open: recovers",
+ qq(SELECT count(*) FROM tbl_ok),
+ qr/^10000$/, qr/^$/);
+
+ $psql->quit();
+}
+
+# Verify that we handle a relation getting removed (due to a rollback or a
+# DROP TABLE) while IO is ongoing for that table.
+sub test_invalidate
+{
+ my $io_method = shift;
+ my $node = shift;
+
+ my $psql = $node->background_psql('postgres', on_error_stop => 0);
+
+ foreach my $persistency (qw(normal unlogged temporary))
+ {
+ my $sql_persistency = $persistency eq 'normal' ? '' : $persistency;
+ my $tblname = $persistency . '_transactional';
+
+ my $create_sql = qq(
+CREATE $sql_persistency TABLE $tblname (id int not null, data text not null) WITH (AUTOVACUUM_ENABLED = false);
+INSERT INTO $tblname(id, data) SELECT generate_series(1, 10000) as id, repeat('a', 200);
+);
+
+ # Verify that outstanding read IO does not cause problems with
+ # AbortTransaction -> smgrDoPendingDeletes -> smgrdounlinkall -> ...
+ # -> Invalidate[Local]Buffer.
+ $psql->query_safe("BEGIN; $create_sql;");
+ $psql->query_safe(
+ qq(
+SELECT read_rel_block_ll('$tblname', 1, wait_complete=>false);
+));
+ psql_like(
+ $io_method,
+ $psql,
+ "rollback of newly created $persistency table with outstanding IO",
+ qq(ROLLBACK),
+ qr/^$/,
+ qr/^$/);
+
+ # Verify that outstanding read IO does not cause problems with
+ # CommitTransaction -> smgrDoPendingDeletes -> smgrdounlinkall -> ...
+ # -> Invalidate[Local]Buffer.
+ $psql->query_safe("BEGIN; $create_sql; COMMIT;");
+ $psql->query_safe(
+ qq(
+BEGIN;
+SELECT read_rel_block_ll('$tblname', 1, wait_complete=>false);
+));
+
+ psql_like(
+ $io_method, $psql,
+ "drop $persistency table with outstanding IO",
+ qq(DROP TABLE $tblname),
+ qr/^$/, qr/^$/);
+
+ psql_like($io_method, $psql,
+ "commit after drop $persistency table with outstanding IO",
+ qq(COMMIT), qr/^$/, qr/^$/);
+ }
+
+ $psql->quit();
+}
+
+# Test behavior related to ZERO_ON_ERROR and zero_damaged_pages
+sub test_zero
+{
+ my $io_method = shift;
+ my $node = shift;
+
+ my $psql_a = $node->background_psql('postgres', on_error_stop => 0);
+ my $psql_b = $node->background_psql('postgres', on_error_stop => 0);
+
+ foreach my $persistency (qw(normal temporary))
+ {
+ my $sql_persistency = $persistency eq 'normal' ? '' : $persistency;
+
+ $psql_a->query_safe(
+ qq(
+CREATE $sql_persistency TABLE tbl_zero(id int) WITH (AUTOVACUUM_ENABLED = false);
+INSERT INTO tbl_zero SELECT generate_series(1, 10000);
+));
+
+ $psql_a->query_safe(
+ qq(
+SELECT modify_rel_block('tbl_zero', 0, corrupt_header=>true);
+));
+
+ # Check that page validity errors are detected
+ psql_like(
+ $io_method,
+ $psql_a,
+ "$persistency: test reading of invalid block 0",
+ qq(
+SELECT read_rel_block_ll('tbl_zero', 0, zero_on_error=>false)),
+ qr/^$/,
+ qr/^psql:<stdin>:\d+: ERROR: invalid page in block 0 of relation base\/.*\/.*$/
+ );
+
+ # Check that page validity errors are zeroed
+ psql_like(
+ $io_method,
+ $psql_a,
+ "$persistency: test zeroing of invalid block 0",
+ qq(
+SELECT read_rel_block_ll('tbl_zero', 0, zero_on_error=>true)),
+ qr/^$/,
+ qr/^psql:<stdin>:\d+: WARNING: invalid page in block 0 of relation base\/.*\/.*; zeroing out page$/
+ );
+
+ # And that once the corruption is fixed, we can read again
+ $psql_a->query(
+ qq(
+SELECT modify_rel_block('tbl_zero', 0, zero=>true);
+));
+ $psql_a->{stderr} = '';
+
+ psql_like(
+ $io_method,
+ $psql_a,
+ "$persistency: test re-read of block 0",
+ qq(
+SELECT read_rel_block_ll('tbl_zero', 0, zero_on_error=>false)),
+ qr/^$/,
+ qr/^$/);
+
+ # Check a page validity error in another block, to ensure we report
+ # the correct block number
+ $psql_a->query_safe(
+ qq(
+SELECT modify_rel_block('tbl_zero', 3, corrupt_header=>true);
+));
+ psql_like(
+ $io_method,
+ $psql_a,
+ "$persistency: test zeroing of invalid block 3",
+ qq(SELECT read_rel_block_ll('tbl_zero', 3, zero_on_error=>true);),
+ qr/^$/,
+ qr/^psql:<stdin>:\d+: WARNING: invalid page in block 3 of relation base\/.*\/.*; zeroing out page$/
+ );
+
+
+ # Check a page validity error in another block, to ensure we report
+ # the correct block number
+ $psql_a->query_safe(
+ qq(
+SELECT modify_rel_block('tbl_zero', 2, corrupt_header=>true);
+SELECT modify_rel_block('tbl_zero', 3, corrupt_header=>true);
+));
+ # First test error
+ psql_like(
+ $io_method,
+ $psql_a,
+ "$persistency: test reading of invalid block 2,3 in larger read",
+ qq(SELECT read_rel_block_ll('tbl_zero', 1, nblocks=>4, zero_on_error=>false)),
+ qr/^$/,
+ qr/^psql:<stdin>:\d+: ERROR: 2 invalid pages among blocks 1..4 of relation base\/.*\/.*\nDETAIL: Block 2 held first invalid page\.\nHINT:[^\n]+$/
+ );
+
+ # Then test zeroing via ZERO_ON_ERROR flag
+ psql_like(
+ $io_method,
+ $psql_a,
+ "$persistency: test zeroing of invalid block 2,3 in larger read, ZERO_ON_ERROR",
+ qq(SELECT read_rel_block_ll('tbl_zero', 1, nblocks=>4, zero_on_error=>true)),
+ qr/^$/,
+ qr/^psql:<stdin>:\d+: WARNING: zeroing out 2 invalid pages among blocks 1..4 of relation base\/.*\/.*\nDETAIL: Block 2 held first zeroed page\.\nHINT:[^\n]+$/
+ );
+
+ # Then test zeroing vio zero_damaged_pages
+ psql_like(
+ $io_method,
+ $psql_a,
+ "$persistency: test zeroing of invalid block 2,3 in larger read, zero_damaged_pages",
+ qq(
+BEGIN;
+SET LOCAL zero_damaged_pages = true;
+SELECT read_rel_block_ll('tbl_zero', 1, nblocks=>4, zero_on_error=>false)
+COMMIT;
+),
+ qr/^$/,
+ qr/^psql:<stdin>:\d+: WARNING: zeroing out 2 invalid pages among blocks 1..4 of relation base\/.*\/.*\nDETAIL: Block 2 held first zeroed page\.\nHINT:[^\n]+$/
+ );
+
+ $psql_a->query_safe(qq(COMMIT));
+
+
+ # Verify that bufmgr.c IO detects page validity errors
+ $psql_a->query(
+ qq(
+SELECT invalidate_rel_block('tbl_zero', g.i)
+FROM generate_series(0, 15) g(i);
+SELECT modify_rel_block('tbl_zero', 3, zero=>true);
+));
+ $psql_a->{stderr} = '';
+
+ psql_like(
+ $io_method,
+ $psql_a,
+ "$persistency: verify reading zero_damaged_pages=off",
+ qq(
+SELECT count(*) FROM tbl_zero),
+ qr/^$/,
+ qr/^psql:<stdin>:\d+: ERROR: invalid page in block 2 of relation base\/.*\/.*$/
+ );
+
+ # Verify that bufmgr.c IO zeroes out pages with page validity errors
+ psql_like(
+ $io_method,
+ $psql_a,
+ "$persistency: verify zero_damaged_pages=on",
+ qq(
+BEGIN;
+SET LOCAL zero_damaged_pages = true;
+SELECT count(*) FROM tbl_zero;
+COMMIT;
+),
+ qr/^\d+$/,
+ qr/^psql:<stdin>:\d+: WARNING: invalid page in block 2 of relation base\/.*\/.*$/
+ );
+
+ # Check that warnings/errors about page validity in an IO started by
+ # session A that session B might complete aren't logged visibly to
+ # session B.
+ #
+ # This will only ever trigger for io_method's like io_uring, that can
+ # complete IO's in a client backend. But it doesn't seem worth
+ # restricting to that.
+ #
+ # This requires cross-session access to the same relation, hence the
+ # restriction to non-temporary table.
+ if ($sql_persistency ne 'temporary')
+ {
+ # Create a corruption and then read the block without waiting for
+ # completion.
+ $psql_a->query(qq(
+SELECT modify_rel_block('tbl_zero', 1, corrupt_header=>true);
+SELECT read_rel_block_ll('tbl_zero', 1, wait_complete=>false, zero_on_error=>true)
+));
+
+ psql_like(
+ $io_method,
+ $psql_b,
+ "$persistency: test completing read by other session doesn't generate warning",
+ qq(SELECT count(*) > 0 FROM tbl_zero;),
+ qr/^t$/, qr/^$/);
+ }
+
+ # Clean up
+ $psql_a->query_safe(
+ qq(
+DROP TABLE tbl_zero;
+));
+ }
+
+ $psql_a->{stderr} = '';
+
+ $psql_a->quit();
+ $psql_b->quit();
+}
+
+# Test that we detect checksum failures and report them
+sub test_checksum
+{
+ my $io_method = shift;
+ my $node = shift;
+
+ my $psql_a = $node->background_psql('postgres', on_error_stop => 0);
+
+ $psql_a->query_safe(
+ qq(
+CREATE TABLE tbl_normal(id int) WITH (AUTOVACUUM_ENABLED = false);
+INSERT INTO tbl_normal SELECT generate_series(1, 5000);
+SELECT modify_rel_block('tbl_normal', 3, corrupt_checksum=>true);
+
+CREATE TEMPORARY TABLE tbl_temp(id int) WITH (AUTOVACUUM_ENABLED = false);
+INSERT INTO tbl_temp SELECT generate_series(1, 5000);
+SELECT modify_rel_block('tbl_temp', 3, corrupt_checksum=>true);
+SELECT modify_rel_block('tbl_temp', 4, corrupt_checksum=>true);
+));
+
+ # To be able to test checksum failures on shared rels we need a shared rel
+ # with invalid pages - which is a bit scary. pg_shseclabel seems like a
+ # good bet, as it's not accessed in a default configuration.
+ $psql_a->query_safe(
+ qq(
+SELECT grow_rel('pg_shseclabel', 4);
+SELECT modify_rel_block('pg_shseclabel', 2, corrupt_checksum=>true);
+SELECT modify_rel_block('pg_shseclabel', 3, corrupt_checksum=>true);
+));
+
+
+ # Check that page validity errors are detected, checksums stats increase, normal rel
+ my ($cs_count_before, $cs_ts_before) =
+ checksum_failures($psql_a, 'postgres');
+ psql_like(
+ $io_method,
+ $psql_a,
+ "normal rel: test reading of invalid block 3",
+ qq(
+SELECT read_rel_block_ll('tbl_normal', 3, nblocks=>1, zero_on_error=>false);),
+ qr/^$/,
+ qr/^psql:<stdin>:\d+: ERROR: invalid page in block 3 of relation base\/\d+\/\d+$/
+ );
+
+ my ($cs_count_after, $cs_ts_after) =
+ checksum_failures($psql_a, 'postgres');
+
+ cmp_ok($cs_count_before + 1,
+ '<=', $cs_count_after,
+ "$io_method: normal rel: checksum count increased");
+ cmp_ok($cs_ts_after, 'ne', '',
+ "$io_method: normal rel: checksum timestamp is not null");
+
+
+ # Check that page validity errors are detected, checksums stats increase, temp rel
+ ($cs_count_after, $cs_ts_after) = checksum_failures($psql_a, 'postgres');
+ psql_like(
+ $io_method,
+ $psql_a,
+ "temp rel: test reading of invalid block 4, valid block 5",
+ qq(
+SELECT read_rel_block_ll('tbl_temp', 4, nblocks=>2, zero_on_error=>false);),
+ qr/^$/,
+ qr/^psql:<stdin>:\d+: ERROR: invalid page in block 4 of relation base\/\d+\/t\d+_\d+$/
+ );
+
+ ($cs_count_after, $cs_ts_after) = checksum_failures($psql_a, 'postgres');
+
+ cmp_ok($cs_count_before + 1,
+ '<=', $cs_count_after,
+ "$io_method: temp rel: checksum count increased");
+ cmp_ok($cs_ts_after, 'ne', '',
+ "$io_method: temp rel: checksum timestamp is not null");
+
+
+ # Check that page validity errors are detected, checksums stats increase, shared rel
+ ($cs_count_before, $cs_ts_after) = checksum_failures($psql_a);
+ psql_like(
+ $io_method,
+ $psql_a,
+ "shared rel: reading of invalid blocks 2+3",
+ qq(
+SELECT read_rel_block_ll('pg_shseclabel', 2, nblocks=>2, zero_on_error=>false);),
+ qr/^$/,
+ qr/^psql:<stdin>:\d+: ERROR: 2 invalid pages among blocks 2..3 of relation global\/\d+\nDETAIL: Block 2 held first invalid page\.\nHINT:[^\n]+$/
+ );
+
+ ($cs_count_after, $cs_ts_after) = checksum_failures($psql_a);
+
+ cmp_ok($cs_count_before + 1,
+ '<=', $cs_count_after,
+ "$io_method: shared rel: checksum count increased");
+ cmp_ok($cs_ts_after, 'ne', '',
+ "$io_method: shared rel: checksum timestamp is not null");
+
+
+ # and restore sanity
+ $psql_a->query(
+ qq(
+SELECT modify_rel_block('pg_shseclabel', 1, zero=>true);
+DROP TABLE tbl_normal;
+));
+ $psql_a->{stderr} = '';
+
+ $psql_a->quit();
+}
+
+# Verify checksum handling when creating database from an invalid database.
+# This also serves as a minimal check that cross-database IO is handled
+# reasonably.
+sub test_checksum_createdb
+{
+ my $io_method = shift;
+ my $node = shift;
+
+ my $psql = $node->background_psql('postgres', on_error_stop => 0);
+
+ $node->safe_psql('postgres',
+ 'CREATE DATABASE regression_createdb_source');
+
+ $node->safe_psql(
+ 'regression_createdb_source', qq(
+CREATE EXTENSION test_aio;
+CREATE TABLE tbl_cs_fail(data int not null) WITH (AUTOVACUUM_ENABLED = false);
+INSERT INTO tbl_cs_fail SELECT generate_series(1, 1000);
+SELECT modify_rel_block('tbl_cs_fail', 1, corrupt_checksum=>true);
+));
+
+ my $createdb_sql = qq(
+CREATE DATABASE regression_createdb_target
+TEMPLATE regression_createdb_source
+STRATEGY wal_log;
+);
+
+ # Verify that CREATE DATABASE of an invalid database fails and is
+ # accounted for accurately.
+ my ($cs_count_before, $cs_ts_before) =
+ checksum_failures($psql, 'regression_createdb_source');
+ psql_like(
+ $io_method,
+ $psql,
+ "create database w/ wal strategy, invalid source",
+ $createdb_sql,
+ qr/^$/,
+ qr/^psql:<stdin>:\d+: ERROR: invalid page in block 1 of relation base\/\d+\/\d+$/
+ );
+ my ($cs_count_after, $cs_ts_after) =
+ checksum_failures($psql, 'regression_createdb_source');
+ cmp_ok($cs_count_before + 1, '<=', $cs_count_after,
+ "$io_method: create database w/ wal strategy, invalid source: checksum count increased"
+ );
+
+ # Verify that CREATE DATABASE of the fixed database succeeds.
+ $node->safe_psql(
+ 'regression_createdb_source', qq(
+SELECT modify_rel_block('tbl_cs_fail', 1, zero=>true);
+));
+ psql_like($io_method, $psql,
+ "create database w/ wal strategy, valid source",
+ $createdb_sql, qr/^$/, qr/^$/);
+
+ $psql->quit();
+}
+
+# Test that we detect checksum failures and report them
+#
+# In several places we make sure that the server log actually contains
+# individual information for each block involved in the IO.
+sub test_ignore_checksum
+{
+ my $io_method = shift;
+ my $node = shift;
+
+ my $psql = $node->background_psql('postgres', on_error_stop => 0);
+
+ # Test setup
+ $psql->query_safe(
+ qq(
+CREATE TABLE tbl_cs_fail(id int) WITH (AUTOVACUUM_ENABLED = false);
+INSERT INTO tbl_cs_fail SELECT generate_series(1, 10000);
+));
+
+ my $count_sql = "SELECT count(*) FROM tbl_cs_fail";
+ my $invalidate_sql = qq(
+SELECT invalidate_rel_block('tbl_cs_fail', g.i)
+FROM generate_series(0, 6) g(i);
+);
+
+ my $expect = $psql->query_safe($count_sql);
+
+
+ # Very basic tests for ignore_checksum_failure=off / on
+
+ $psql->query_safe(
+ qq(
+SELECT modify_rel_block('tbl_cs_fail', 1, corrupt_checksum=>true);
+SELECT modify_rel_block('tbl_cs_fail', 5, corrupt_checksum=>true);
+SELECT modify_rel_block('tbl_cs_fail', 6, corrupt_checksum=>true);
+));
+
+ $psql->query_safe($invalidate_sql);
+ psql_like($io_method, $psql,
+ "reading block w/ wrong checksum with ignore_checksum_failure=off fails",
+ $count_sql, qr/^$/, qr/ERROR: invalid page in block/);
+
+ $psql->query_safe("SET ignore_checksum_failure=on");
+
+ $psql->query_safe($invalidate_sql);
+ psql_like($io_method, $psql,
+ "reading block w/ wrong checksum with ignore_checksum_failure=off succeeds",
+ $count_sql,
+ qr/^$expect$/,
+ qr/WARNING: ignoring (checksum failure|\d checksum failures)/);
+
+
+ # Verify that ignore_checksum_failure=off works in multi-block reads
+
+ $psql->query_safe(
+ qq(
+SELECT modify_rel_block('tbl_cs_fail', 2, zero=>true);
+SELECT modify_rel_block('tbl_cs_fail', 3, corrupt_checksum=>true);
+SELECT modify_rel_block('tbl_cs_fail', 4, corrupt_header=>true);
+));
+
+ my $log_location = -s $node->logfile;
+ psql_like(
+ $io_method,
+ $psql,
+ "test reading of checksum failed block 3, with ignore",
+ qq(
+SELECT read_rel_block_ll('tbl_cs_fail', 3, nblocks=>1, zero_on_error=>false);),
+ qr/^$/,
+ qr/^psql:<stdin>:\d+: WARNING: ignoring checksum failure in block 3/
+ );
+
+ # Check that the log contains a LOG message about the failure
+ $log_location =
+ $node->wait_for_log(qr/LOG: ignoring checksum failure/, $log_location);
+
+ # check that we error
+ psql_like(
+ $io_method,
+ $psql,
+ "test reading of valid block 2, checksum failed 3, invalid 4, zero=false with ignore",
+ qq(
+SELECT read_rel_block_ll('tbl_cs_fail', 2, nblocks=>3, zero_on_error=>false);),
+ qr/^$/,
+ qr/^psql:<stdin>:\d+: ERROR: invalid page in block 4 of relation base\/\d+\/\d+$/
+ );
+
+ # Test multi-block read with different problems in different blocks
+ $psql->query(
+ qq(
+SELECT modify_rel_block('tbl_cs_fail', 1, zero=>true);
+SELECT modify_rel_block('tbl_cs_fail', 2, corrupt_checksum=>true);
+SELECT modify_rel_block('tbl_cs_fail', 3, corrupt_checksum=>true, corrupt_header=>true);
+SELECT modify_rel_block('tbl_cs_fail', 4, corrupt_header=>true);
+SELECT modify_rel_block('tbl_cs_fail', 5, corrupt_header=>true);
+));
+ $psql->{stderr} = '';
+
+ $log_location = -s $node->logfile;
+ psql_like(
+ $io_method,
+ $psql,
+ "test reading of valid block 1, checksum failed 2, 3, invalid 3-5, zero=true",
+ qq(
+SELECT read_rel_block_ll('tbl_cs_fail', 1, nblocks=>5, zero_on_error=>true);),
+ qr/^$/,
+ qr/^psql:<stdin>:\d+: WARNING: zeroing 3 page\(s\) and ignoring 2 checksum failure\(s\) among blocks 1..5 of relation/
+ );
+
+
+ # Unfortunately have to scan the whole log since determining $log_location
+ # above in each of the tests, as wait_for_log() returns the size of the
+ # file.
+
+ $node->wait_for_log(qr/LOG: ignoring checksum failure in block 2/,
+ $log_location);
+ ok(1, "$io_method: found information about checksum failure in block 2");
+
+ $node->wait_for_log(qr/LOG: invalid page in block 3 of relation base.*; zeroing out page/,
+ $log_location);
+ ok(1, "$io_method: found information about invalid page in block 3");
+
+ $node->wait_for_log(qr/LOG: invalid page in block 4 of relation base.*; zeroing out page/,
+ $log_location);
+ ok(1, "$io_method: found information about checksum failure in block 4");
+
+ $node->wait_for_log(qr/LOG: invalid page in block 5 of relation base.*; zeroing out page/,
+ $log_location);
+ ok(1, "$io_method: found information about checksum failure in block 5");
+
+
+ # Reading a page with both an invalid header and an invalid checksum
+ $psql->query(
+ qq(
+SELECT modify_rel_block('tbl_cs_fail', 3, corrupt_checksum=>true, corrupt_header=>true);
+));
+ $psql->{stderr} = '';
+
+ psql_like(
+ $io_method,
+ $psql,
+ "test reading of block with both invalid header and invalid checksum, zero=false",
+ qq(
+SELECT read_rel_block_ll('tbl_cs_fail', 3, nblocks=>1, zero_on_error=>false);),
+ qr/^$/,
+ qr/^psql:<stdin>:\d+: ERROR: invalid page in block 3 of relation/
+ );
+
+ psql_like(
+ $io_method,
+ $psql,
+ "test reading of block 3 with both invalid header and invalid checksum, zero=true",
+ qq(
+SELECT read_rel_block_ll('tbl_cs_fail', 3, nblocks=>1, zero_on_error=>true);),
+ qr/^$/,
+ qr/^psql:<stdin>:\d+: WARNING: invalid page in block 3 of relation base\/.*; zeroing out page/
+ );
+
+
+ $psql->quit();
+}
+
+
+# Run all tests that are supported for all io_methods
+sub test_generic
+{
+ my $io_method = shift;
+ my $node = shift;
+
+ is($node->safe_psql('postgres', 'SHOW io_method'),
+ $io_method, "$io_method: io_method set correctly");
+
+ $node->safe_psql(
+ 'postgres', qq(
+CREATE EXTENSION test_aio;
+CREATE TABLE tbl_corr(data int not null) WITH (AUTOVACUUM_ENABLED = false);
+CREATE TABLE tbl_ok(data int not null) WITH (AUTOVACUUM_ENABLED = false);
+
+INSERT INTO tbl_corr SELECT generate_series(1, 10000);
+INSERT INTO tbl_ok SELECT generate_series(1, 10000);
+SELECT grow_rel('tbl_corr', 16);
+SELECT grow_rel('tbl_ok', 16);
+
+SELECT modify_rel_block('tbl_corr', 1, corrupt_header=>true);
+CHECKPOINT;
+));
+
+ test_handle($io_method, $node);
+ test_io_error($io_method, $node);
+ test_batchmode($io_method, $node);
+ test_startwait_io($io_method, $node);
+ test_complete_foreign($io_method, $node);
+ test_close_fd($io_method, $node);
+ test_invalidate($io_method, $node);
+ test_zero($io_method, $node);
+ test_checksum($io_method, $node);
+ test_ignore_checksum($io_method, $node);
+ test_checksum_createdb($io_method, $node);
+
+ SKIP:
+ {
+ skip 'Injection points not supported by this build', 1
+ unless $ENV{enable_injection_points} eq 'yes';
+ test_inject($io_method, $node);
+ }
+}
diff --git a/src/test/modules/test_aio/t/002_io_workers.pl b/src/test/modules/test_aio/t/002_io_workers.pl
new file mode 100644
index 00000000000..af5fae15ea7
--- /dev/null
+++ b/src/test/modules/test_aio/t/002_io_workers.pl
@@ -0,0 +1,125 @@
+# Copyright (c) 2025, PostgreSQL Global Development Group
+
+use strict;
+use warnings FATAL => 'all';
+
+use PostgreSQL::Test::Cluster;
+use PostgreSQL::Test::Utils;
+use Test::More;
+use List::Util qw(shuffle);
+
+
+my $node = PostgreSQL::Test::Cluster->new('worker');
+$node->init();
+$node->append_conf(
+ 'postgresql.conf', qq(
+io_method=worker
+));
+
+$node->start();
+
+# Test changing the number of I/O worker processes while also evaluating the
+# handling of their termination.
+test_number_of_io_workers_dynamic($node);
+
+$node->stop();
+
+done_testing();
+
+
+sub test_number_of_io_workers_dynamic
+{
+ my $node = shift;
+
+ my $prev_worker_count = $node->safe_psql('postgres', 'SHOW io_workers');
+
+ # Verify that worker count can't be set to 0
+ change_number_of_io_workers($node, 0, $prev_worker_count, 1);
+
+ # Verify that worker count can't be set to 33 (above the max)
+ change_number_of_io_workers($node, 33, $prev_worker_count, 1);
+
+ # Try changing IO workers to a random value and verify that the worker
+ # count ends up as expected. Always test the min/max of workers.
+ #
+ # Valid range for io_workers is [1, 32]. 8 tests in total seems
+ # reasonable.
+ my @io_workers_range = shuffle(1 ... 32);
+ foreach my $worker_count (1, 32, @io_workers_range[ 0, 6 ])
+ {
+ $prev_worker_count =
+ change_number_of_io_workers($node, $worker_count,
+ $prev_worker_count, 0);
+ }
+}
+
+sub change_number_of_io_workers
+{
+ my $node = shift;
+ my $worker_count = shift;
+ my $prev_worker_count = shift;
+ my $expect_failure = shift;
+ my ($result, $stdout, $stderr);
+
+ ($result, $stdout, $stderr) =
+ $node->psql('postgres', "ALTER SYSTEM SET io_workers = $worker_count");
+ $node->safe_psql('postgres', 'SELECT pg_reload_conf()');
+
+ if ($expect_failure)
+ {
+ ok( $stderr =~
+ /$worker_count is outside the valid range for parameter "io_workers"/,
+ "updating number of io_workers to $worker_count failed, as expected"
+ );
+
+ return $prev_worker_count;
+ }
+ else
+ {
+ is( $node->safe_psql('postgres', 'SHOW io_workers'),
+ $worker_count,
+ "updating number of io_workers from $prev_worker_count to $worker_count"
+ );
+
+ check_io_worker_count($node, $worker_count);
+ terminate_io_worker($node, $worker_count);
+ check_io_worker_count($node, $worker_count);
+
+ return $worker_count;
+ }
+}
+
+sub terminate_io_worker
+{
+ my $node = shift;
+ my $worker_count = shift;
+ my ($pid, $ret);
+
+ # Select a random io worker
+ $pid = $node->safe_psql(
+ 'postgres',
+ qq(SELECT pid FROM pg_stat_activity WHERE
+ backend_type = 'io worker' ORDER BY RANDOM() LIMIT 1));
+
+ # terminate IO worker with SIGINT
+ is(PostgreSQL::Test::Utils::system_log('pg_ctl', 'kill', 'INT', $pid),
+ 0, "random io worker process signalled with INT");
+
+ # Check that worker exits
+ ok( $node->poll_query_until(
+ 'postgres',
+ qq(SELECT COUNT(*) FROM pg_stat_activity WHERE pid = $pid), '0'),
+ "random io worker process exited after signal");
+}
+
+sub check_io_worker_count
+{
+ my $node = shift;
+ my $worker_count = shift;
+
+ ok( $node->poll_query_until(
+ 'postgres',
+ qq(SELECT COUNT(*) FROM pg_stat_activity WHERE backend_type = 'io worker'),
+ $worker_count),
+ "io worker count is $worker_count");
+}
diff --git a/src/test/modules/test_aio/test_aio--1.0.sql b/src/test/modules/test_aio/test_aio--1.0.sql
new file mode 100644
index 00000000000..e495481c41e
--- /dev/null
+++ b/src/test/modules/test_aio/test_aio--1.0.sql
@@ -0,0 +1,108 @@
+/* src/test/modules/test_aio/test_aio--1.0.sql */
+
+-- complain if script is sourced in psql, rather than via CREATE EXTENSION
+\echo Use "CREATE EXTENSION test_aio" to load this file. \quit
+
+
+CREATE FUNCTION errno_from_string(sym text)
+RETURNS pg_catalog.int4 STRICT
+AS 'MODULE_PATHNAME' LANGUAGE C;
+
+
+CREATE FUNCTION grow_rel(rel regclass, nblocks int)
+RETURNS pg_catalog.void STRICT
+AS 'MODULE_PATHNAME' LANGUAGE C;
+
+
+CREATE FUNCTION modify_rel_block(rel regclass, blockno int,
+ zero bool DEFAULT false,
+ corrupt_header bool DEFAULT false,
+ corrupt_checksum bool DEFAULT false)
+RETURNS pg_catalog.void STRICT
+AS 'MODULE_PATHNAME' LANGUAGE C;
+
+CREATE FUNCTION read_rel_block_ll(
+ rel regclass,
+ blockno int,
+ nblocks int DEFAULT 1,
+ wait_complete bool DEFAULT true,
+ batchmode_enter bool DEFAULT false,
+ smgrreleaseall bool DEFAULT false,
+ batchmode_exit bool DEFAULT false,
+ zero_on_error bool DEFAULT false)
+RETURNS pg_catalog.void STRICT
+AS 'MODULE_PATHNAME' LANGUAGE C;
+
+CREATE FUNCTION invalidate_rel_block(rel regclass, blockno int)
+RETURNS pg_catalog.void STRICT
+AS 'MODULE_PATHNAME' LANGUAGE C;
+
+CREATE FUNCTION buffer_create_toy(rel regclass, blockno int4)
+RETURNS pg_catalog.int4 STRICT
+AS 'MODULE_PATHNAME' LANGUAGE C;
+
+CREATE FUNCTION buffer_call_start_io(buffer int, for_input bool, nowait bool)
+RETURNS pg_catalog.bool STRICT
+AS 'MODULE_PATHNAME' LANGUAGE C;
+
+CREATE FUNCTION buffer_call_terminate_io(buffer int, for_input bool, succeed bool, io_error bool, release_aio bool)
+RETURNS pg_catalog.void STRICT
+AS 'MODULE_PATHNAME' LANGUAGE C;
+
+
+
+/*
+ * Handle related functions
+ */
+CREATE FUNCTION handle_get_and_error()
+RETURNS pg_catalog.void STRICT
+AS 'MODULE_PATHNAME' LANGUAGE C;
+
+CREATE FUNCTION handle_get_twice()
+RETURNS pg_catalog.void STRICT
+AS 'MODULE_PATHNAME' LANGUAGE C;
+
+CREATE FUNCTION handle_get()
+RETURNS pg_catalog.void STRICT
+AS 'MODULE_PATHNAME' LANGUAGE C;
+
+CREATE FUNCTION handle_get_release()
+RETURNS pg_catalog.void STRICT
+AS 'MODULE_PATHNAME' LANGUAGE C;
+
+CREATE FUNCTION handle_release_last()
+RETURNS pg_catalog.void STRICT
+AS 'MODULE_PATHNAME' LANGUAGE C;
+
+
+/*
+ * Batchmode related functions
+ */
+CREATE FUNCTION batch_start()
+RETURNS pg_catalog.void STRICT
+AS 'MODULE_PATHNAME' LANGUAGE C;
+
+CREATE FUNCTION batch_end()
+RETURNS pg_catalog.void STRICT
+AS 'MODULE_PATHNAME' LANGUAGE C;
+
+
+
+/*
+ * Injection point related functions
+ */
+CREATE FUNCTION inj_io_short_read_attach(result int)
+RETURNS pg_catalog.void STRICT
+AS 'MODULE_PATHNAME' LANGUAGE C;
+
+CREATE FUNCTION inj_io_short_read_detach()
+RETURNS pg_catalog.void STRICT
+AS 'MODULE_PATHNAME' LANGUAGE C;
+
+CREATE FUNCTION inj_io_reopen_attach()
+RETURNS pg_catalog.void STRICT
+AS 'MODULE_PATHNAME' LANGUAGE C;
+
+CREATE FUNCTION inj_io_reopen_detach()
+RETURNS pg_catalog.void STRICT
+AS 'MODULE_PATHNAME' LANGUAGE C;
diff --git a/src/test/modules/test_aio/test_aio.c b/src/test/modules/test_aio/test_aio.c
new file mode 100644
index 00000000000..bef3b0e3ad0
--- /dev/null
+++ b/src/test/modules/test_aio/test_aio.c
@@ -0,0 +1,804 @@
+/*-------------------------------------------------------------------------
+ *
+ * test_aio.c
+ * Helpers to write tests for AIO
+ *
+ * This module provides interface functions for C functionality to SQL, to
+ * make it possible to test AIO related behavior in a targeted way from SQL.
+ * It'd not generally be safe to export these functions to SQL, but for a test
+ * that's fine.
+ *
+ * Copyright (c) 2020-2025, PostgreSQL Global Development Group
+ *
+ * IDENTIFICATION
+ * src/test/modules/test_aio/test_aio.c
+ *
+ *-------------------------------------------------------------------------
+ */
+
+#include "postgres.h"
+
+#include "access/relation.h"
+#include "fmgr.h"
+#include "storage/aio.h"
+#include "storage/aio_internal.h"
+#include "storage/buf_internals.h"
+#include "storage/bufmgr.h"
+#include "storage/checksum.h"
+#include "storage/ipc.h"
+#include "storage/lwlock.h"
+#include "utils/builtins.h"
+#include "utils/injection_point.h"
+#include "utils/rel.h"
+
+
+PG_MODULE_MAGIC;
+
+
+typedef struct InjIoErrorState
+{
+ bool enabled_short_read;
+ bool enabled_reopen;
+
+ bool short_read_result_set;
+ int short_read_result;
+} InjIoErrorState;
+
+static InjIoErrorState * inj_io_error_state;
+
+/* Shared memory init callbacks */
+static shmem_request_hook_type prev_shmem_request_hook = NULL;
+static shmem_startup_hook_type prev_shmem_startup_hook = NULL;
+
+
+static PgAioHandle *last_handle;
+
+
+
+static void
+test_aio_shmem_request(void)
+{
+ if (prev_shmem_request_hook)
+ prev_shmem_request_hook();
+
+ RequestAddinShmemSpace(sizeof(InjIoErrorState));
+}
+
+static void
+test_aio_shmem_startup(void)
+{
+ bool found;
+
+ if (prev_shmem_startup_hook)
+ prev_shmem_startup_hook();
+
+ /* Create or attach to the shared memory state */
+ LWLockAcquire(AddinShmemInitLock, LW_EXCLUSIVE);
+
+ inj_io_error_state = ShmemInitStruct("injection_points",
+ sizeof(InjIoErrorState),
+ &found);
+
+ if (!found)
+ {
+ /* First time through, initialize */
+ inj_io_error_state->enabled_short_read = false;
+ inj_io_error_state->enabled_reopen = false;
+
+#ifdef USE_INJECTION_POINTS
+ InjectionPointAttach("AIO_PROCESS_COMPLETION_BEFORE_SHARED",
+ "test_aio",
+ "inj_io_short_read",
+ NULL,
+ 0);
+ InjectionPointLoad("AIO_PROCESS_COMPLETION_BEFORE_SHARED");
+
+ InjectionPointAttach("AIO_WORKER_AFTER_REOPEN",
+ "test_aio",
+ "inj_io_reopen",
+ NULL,
+ 0);
+ InjectionPointLoad("AIO_WORKER_AFTER_REOPEN");
+
+#endif
+ }
+ else
+ {
+ /*
+ * Pre-load the injection points now, so we can call them in a
+ * critical section.
+ */
+#ifdef USE_INJECTION_POINTS
+ InjectionPointLoad("AIO_PROCESS_COMPLETION_BEFORE_SHARED");
+ InjectionPointLoad("AIO_WORKER_AFTER_REOPEN");
+ elog(LOG, "injection point loaded");
+#endif
+ }
+
+ LWLockRelease(AddinShmemInitLock);
+}
+
+void
+_PG_init(void)
+{
+ if (!process_shared_preload_libraries_in_progress)
+ return;
+
+ prev_shmem_request_hook = shmem_request_hook;
+ shmem_request_hook = test_aio_shmem_request;
+ prev_shmem_startup_hook = shmem_startup_hook;
+ shmem_startup_hook = test_aio_shmem_startup;
+}
+
+
+PG_FUNCTION_INFO_V1(errno_from_string);
+Datum
+errno_from_string(PG_FUNCTION_ARGS)
+{
+ const char *sym = text_to_cstring(PG_GETARG_TEXT_PP(0));
+
+ if (strcmp(sym, "EIO") == 0)
+ PG_RETURN_INT32(EIO);
+ else if (strcmp(sym, "EAGAIN") == 0)
+ PG_RETURN_INT32(EAGAIN);
+ else if (strcmp(sym, "EINTR") == 0)
+ PG_RETURN_INT32(EINTR);
+ else if (strcmp(sym, "ENOSPC") == 0)
+ PG_RETURN_INT32(ENOSPC);
+ else if (strcmp(sym, "EROFS") == 0)
+ PG_RETURN_INT32(EROFS);
+
+ ereport(ERROR,
+ errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ errmsg_internal("%s is not a supported errno value", sym));
+ PG_RETURN_INT32(0);
+}
+
+PG_FUNCTION_INFO_V1(grow_rel);
+Datum
+grow_rel(PG_FUNCTION_ARGS)
+{
+ Oid relid = PG_GETARG_OID(0);
+ uint32 nblocks = PG_GETARG_UINT32(1);
+ Relation rel;
+#define MAX_BUFFERS_TO_EXTEND_BY 64
+ Buffer victim_buffers[MAX_BUFFERS_TO_EXTEND_BY];
+
+ rel = relation_open(relid, AccessExclusiveLock);
+
+ while (nblocks > 0)
+ {
+ uint32 extend_by_pages;
+
+ extend_by_pages = Min(nblocks, MAX_BUFFERS_TO_EXTEND_BY);
+
+ ExtendBufferedRelBy(BMR_REL(rel),
+ MAIN_FORKNUM,
+ NULL,
+ 0,
+ extend_by_pages,
+ victim_buffers,
+ &extend_by_pages);
+
+ nblocks -= extend_by_pages;
+
+ for (uint32 i = 0; i < extend_by_pages; i++)
+ {
+ ReleaseBuffer(victim_buffers[i]);
+ }
+ }
+
+ relation_close(rel, NoLock);
+
+ PG_RETURN_VOID();
+}
+
+PG_FUNCTION_INFO_V1(modify_rel_block);
+Datum
+modify_rel_block(PG_FUNCTION_ARGS)
+{
+ Oid relid = PG_GETARG_OID(0);
+ BlockNumber blkno = PG_GETARG_UINT32(1);
+ bool zero = PG_GETARG_BOOL(2);
+ bool corrupt_header = PG_GETARG_BOOL(3);
+ bool corrupt_checksum = PG_GETARG_BOOL(4);
+ Page page = palloc_aligned(BLCKSZ, PG_IO_ALIGN_SIZE, 0);
+ Relation rel;
+ Buffer buf;
+ PageHeader ph;
+
+ rel = relation_open(relid, AccessExclusiveLock);
+
+ buf = ReadBufferExtended(rel, MAIN_FORKNUM, blkno,
+ RBM_ZERO_ON_ERROR, NULL);
+
+ LockBuffer(buf, BUFFER_LOCK_EXCLUSIVE);
+
+ /*
+ * copy the page to local memory, seems nicer than to directly modify in
+ * the buffer pool.
+ */
+ memcpy(page, BufferGetPage(buf), BLCKSZ);
+
+ LockBuffer(buf, BUFFER_LOCK_UNLOCK);
+
+ ReleaseBuffer(buf);
+
+ /*
+ * Don't want to have a buffer in-memory that's marked valid where the
+ * on-disk contents are invalid. Particularly not if the in-memory buffer
+ * could be dirty...
+ *
+ * While we hold an AEL on the relation nobody else should be able to read
+ * the buffer in.
+ *
+ * NB: This is probably racy, better don't copy this to non-test code.
+ */
+ if (BufferIsLocal(buf))
+ InvalidateLocalBuffer(GetLocalBufferDescriptor(-buf - 1), true);
+ else
+ EvictUnpinnedBuffer(buf);
+
+ /*
+ * Now modify the page as asked for by the caller.
+ */
+ if (zero)
+ memset(page, 0, BufferGetPageSize(buf));
+
+ if (PageIsEmpty(page) && (corrupt_header || corrupt_checksum))
+ PageInit(page, BufferGetPageSize(buf), 0);
+
+ ph = (PageHeader) page;
+
+ if (corrupt_header)
+ ph->pd_special = BLCKSZ + 1;
+
+ if (corrupt_checksum)
+ {
+ bool successfully_corrupted = 0;
+
+ /*
+ * Any single modification of the checksum could just end up being
+ * valid again. To be sure
+ */
+ for (int i = 0; i < 100; i++)
+ {
+ uint16 verify_checksum;
+ uint16 old_checksum;
+
+ old_checksum = ph->pd_checksum;
+ ph->pd_checksum = old_checksum + 1;
+
+ elog(LOG, "corrupting checksum of blk %u from %u to %u",
+ blkno, old_checksum, ph->pd_checksum);
+
+ verify_checksum = pg_checksum_page(page, blkno);
+ if (verify_checksum != ph->pd_checksum)
+ {
+ successfully_corrupted = true;
+ break;
+ }
+ }
+
+ if (!successfully_corrupted)
+ elog(ERROR, "could not corrupt checksum, what's going on?");
+ }
+ else
+ {
+ PageSetChecksumInplace(page, blkno);
+ }
+
+ smgrwrite(RelationGetSmgr(rel),
+ MAIN_FORKNUM, blkno, page, true);
+
+ relation_close(rel, NoLock);
+
+ PG_RETURN_VOID();
+}
+
+/*
+ * Ensures a buffer for rel & blkno is in shared buffers, without actually
+ * caring about the buffer contents. Used to set up test scenarios.
+ */
+static Buffer
+create_toy_buffer(Relation rel, BlockNumber blkno)
+{
+ Buffer buf;
+ BufferDesc *buf_hdr;
+ uint32 buf_state;
+ bool was_pinned = false;
+
+ /* place buffer in shared buffers without erroring out */
+ buf = ReadBufferExtended(rel, MAIN_FORKNUM, blkno, RBM_ZERO_AND_LOCK, NULL);
+ LockBuffer(buf, BUFFER_LOCK_UNLOCK);
+
+ if (RelationUsesLocalBuffers(rel))
+ {
+ buf_hdr = GetLocalBufferDescriptor(-buf - 1);
+ buf_state = pg_atomic_read_u32(&buf_hdr->state);
+ }
+ else
+ {
+ buf_hdr = GetBufferDescriptor(buf - 1);
+ buf_state = LockBufHdr(buf_hdr);
+ }
+
+ /*
+ * We should be the only backend accessing this buffer. This is just a
+ * small bit of belt-and-suspenders defense, none of this code should ever
+ * run in a cluster with real data.
+ */
+ if (BUF_STATE_GET_REFCOUNT(buf_state) > 1)
+ was_pinned = true;
+ else
+ buf_state &= ~(BM_VALID | BM_DIRTY);
+
+ if (RelationUsesLocalBuffers(rel))
+ pg_atomic_unlocked_write_u32(&buf_hdr->state, buf_state);
+ else
+ UnlockBufHdr(buf_hdr, buf_state);
+
+ if (was_pinned)
+ elog(ERROR, "toy buffer %d was already pinned",
+ buf);
+
+ return buf;
+}
+
+/*
+ * A "low level" read. This does similar things to what
+ * StartReadBuffers()/WaitReadBuffers() do, but provides more control (and
+ * less sanity).
+ */
+PG_FUNCTION_INFO_V1(read_rel_block_ll);
+Datum
+read_rel_block_ll(PG_FUNCTION_ARGS)
+{
+ Oid relid = PG_GETARG_OID(0);
+ BlockNumber blkno = PG_GETARG_UINT32(1);
+ int nblocks = PG_GETARG_INT32(2);
+ bool wait_complete = PG_GETARG_BOOL(3);
+ bool batchmode_enter = PG_GETARG_BOOL(4);
+ bool call_smgrreleaseall = PG_GETARG_BOOL(5);
+ bool batchmode_exit = PG_GETARG_BOOL(6);
+ bool zero_on_error = PG_GETARG_BOOL(7);
+ Relation rel;
+ Buffer bufs[PG_IOV_MAX];
+ BufferDesc *buf_hdrs[PG_IOV_MAX];
+ Page pages[PG_IOV_MAX];
+ uint8 srb_flags = 0;
+ PgAioReturn ior;
+ PgAioHandle *ioh;
+ PgAioWaitRef iow;
+ SMgrRelation smgr;
+
+ if (nblocks <= 0 || nblocks > PG_IOV_MAX)
+ elog(ERROR, "nblocks is out of range");
+
+ rel = relation_open(relid, AccessExclusiveLock);
+
+ for (int i = 0; i < nblocks; i++)
+ {
+ bufs[i] = create_toy_buffer(rel, blkno + i);
+ pages[i] = BufferGetBlock(bufs[i]);
+ buf_hdrs[i] = BufferIsLocal(bufs[i]) ?
+ GetLocalBufferDescriptor(-bufs[i] - 1) :
+ GetBufferDescriptor(bufs[i] - 1);
+ }
+
+ smgr = RelationGetSmgr(rel);
+
+ pgstat_prepare_report_checksum_failure(smgr->smgr_rlocator.locator.dbOid);
+
+ ioh = pgaio_io_acquire(CurrentResourceOwner, &ior);
+ pgaio_io_get_wref(ioh, &iow);
+
+ if (RelationUsesLocalBuffers(rel))
+ {
+ for (int i = 0; i < nblocks; i++)
+ StartLocalBufferIO(buf_hdrs[i], true, false);
+ pgaio_io_set_flag(ioh, PGAIO_HF_REFERENCES_LOCAL);
+ }
+ else
+ {
+ for (int i = 0; i < nblocks; i++)
+ StartBufferIO(buf_hdrs[i], true, false);
+ }
+
+ pgaio_io_set_handle_data_32(ioh, (uint32 *) bufs, nblocks);
+
+ if (zero_on_error | zero_damaged_pages)
+ srb_flags |= READ_BUFFERS_ZERO_ON_ERROR;
+ if (ignore_checksum_failure)
+ srb_flags |= READ_BUFFERS_IGNORE_CHECKSUM_FAILURES;
+
+ pgaio_io_register_callbacks(ioh,
+ RelationUsesLocalBuffers(rel) ?
+ PGAIO_HCB_LOCAL_BUFFER_READV :
+ PGAIO_HCB_SHARED_BUFFER_READV,
+ srb_flags);
+
+ if (batchmode_enter)
+ pgaio_enter_batchmode();
+
+ smgrstartreadv(ioh, smgr, MAIN_FORKNUM, blkno,
+ (void *) pages, nblocks);
+
+ if (call_smgrreleaseall)
+ smgrreleaseall();
+
+ if (batchmode_exit)
+ pgaio_exit_batchmode();
+
+ for (int i = 0; i < nblocks; i++)
+ ReleaseBuffer(bufs[i]);
+
+ if (wait_complete)
+ {
+ pgaio_wref_wait(&iow);
+
+ if (ior.result.status != PGAIO_RS_OK)
+ pgaio_result_report(ior.result,
+ &ior.target_data,
+ ior.result.status == PGAIO_RS_ERROR ?
+ ERROR : WARNING);
+ }
+
+ relation_close(rel, NoLock);
+
+ PG_RETURN_VOID();
+}
+
+PG_FUNCTION_INFO_V1(invalidate_rel_block);
+Datum
+invalidate_rel_block(PG_FUNCTION_ARGS)
+{
+ Oid relid = PG_GETARG_OID(0);
+ BlockNumber blkno = PG_GETARG_UINT32(1);
+ Relation rel;
+ PrefetchBufferResult pr;
+ Buffer buf;
+
+ rel = relation_open(relid, AccessExclusiveLock);
+
+ /*
+ * This is a gross hack, but there's no other API exposed that allows to
+ * get a buffer ID without actually reading the block in.
+ */
+ pr = PrefetchBuffer(rel, MAIN_FORKNUM, blkno);
+ buf = pr.recent_buffer;
+
+ if (BufferIsValid(buf))
+ {
+ /* if the buffer contents aren't valid, this'll return false */
+ if (ReadRecentBuffer(rel->rd_locator, MAIN_FORKNUM, blkno, buf))
+ {
+ BufferDesc *buf_hdr = BufferIsLocal(buf) ?
+ GetLocalBufferDescriptor(-buf - 1)
+ : GetBufferDescriptor(buf - 1);
+
+ LockBuffer(buf, BUFFER_LOCK_EXCLUSIVE);
+
+ if (pg_atomic_read_u32(&buf_hdr->state) & BM_DIRTY)
+ {
+ if (BufferIsLocal(buf))
+ FlushLocalBuffer(buf_hdr, NULL);
+ else
+ FlushOneBuffer(buf);
+ }
+ LockBuffer(buf, BUFFER_LOCK_UNLOCK);
+ ReleaseBuffer(buf);
+
+ if (BufferIsLocal(buf))
+ InvalidateLocalBuffer(GetLocalBufferDescriptor(-buf - 1), true);
+ else if (!EvictUnpinnedBuffer(buf))
+ elog(ERROR, "couldn't evict");
+ }
+ }
+
+ relation_close(rel, AccessExclusiveLock);
+
+ PG_RETURN_VOID();
+}
+
+PG_FUNCTION_INFO_V1(buffer_create_toy);
+Datum
+buffer_create_toy(PG_FUNCTION_ARGS)
+{
+ Oid relid = PG_GETARG_OID(0);
+ BlockNumber blkno = PG_GETARG_UINT32(1);
+ Relation rel;
+ Buffer buf;
+
+ rel = relation_open(relid, AccessExclusiveLock);
+
+ buf = create_toy_buffer(rel, blkno);
+ ReleaseBuffer(buf);
+
+ relation_close(rel, NoLock);
+
+ PG_RETURN_INT32(buf);
+}
+
+PG_FUNCTION_INFO_V1(buffer_call_start_io);
+Datum
+buffer_call_start_io(PG_FUNCTION_ARGS)
+{
+ Buffer buf = PG_GETARG_INT32(0);
+ bool for_input = PG_GETARG_BOOL(1);
+ bool nowait = PG_GETARG_BOOL(2);
+ bool can_start;
+
+ if (BufferIsLocal(buf))
+ can_start = StartLocalBufferIO(GetLocalBufferDescriptor(-buf - 1),
+ for_input, nowait);
+ else
+ can_start = StartBufferIO(GetBufferDescriptor(buf - 1),
+ for_input, nowait);
+
+ /*
+ * For tests we don't want the resowner release preventing us from
+ * orchestrating odd scenarios.
+ */
+ if (can_start && !BufferIsLocal(buf))
+ ResourceOwnerForgetBufferIO(CurrentResourceOwner,
+ buf);
+
+ ereport(LOG,
+ errmsg("buffer %d after StartBufferIO: %s",
+ buf, DebugPrintBufferRefcount(buf)),
+ errhidestmt(true), errhidecontext(true));
+
+ PG_RETURN_BOOL(can_start);
+}
+
+PG_FUNCTION_INFO_V1(buffer_call_terminate_io);
+Datum
+buffer_call_terminate_io(PG_FUNCTION_ARGS)
+{
+ Buffer buf = PG_GETARG_INT32(0);
+ bool for_input = PG_GETARG_BOOL(1);
+ bool succeed = PG_GETARG_BOOL(2);
+ bool io_error = PG_GETARG_BOOL(3);
+ bool release_aio = PG_GETARG_BOOL(4);
+ bool clear_dirty = false;
+ uint32 set_flag_bits = 0;
+
+ if (io_error)
+ set_flag_bits |= BM_IO_ERROR;
+
+ if (for_input)
+ {
+ clear_dirty = false;
+
+ if (succeed)
+ set_flag_bits |= BM_VALID;
+ }
+ else
+ {
+ if (succeed)
+ clear_dirty = true;
+ }
+
+ ereport(LOG,
+ errmsg("buffer %d before Terminate[Local]BufferIO: %s",
+ buf, DebugPrintBufferRefcount(buf)),
+ errhidestmt(true), errhidecontext(true));
+
+ if (BufferIsLocal(buf))
+ TerminateLocalBufferIO(GetLocalBufferDescriptor(-buf - 1),
+ clear_dirty, set_flag_bits, release_aio);
+ else
+ TerminateBufferIO(GetBufferDescriptor(buf - 1),
+ clear_dirty, set_flag_bits, false, release_aio);
+
+ ereport(LOG,
+ errmsg("buffer %d after Terminate[Local]BufferIO: %s",
+ buf, DebugPrintBufferRefcount(buf)),
+ errhidestmt(true), errhidecontext(true));
+
+ PG_RETURN_VOID();
+}
+
+PG_FUNCTION_INFO_V1(handle_get);
+Datum
+handle_get(PG_FUNCTION_ARGS)
+{
+ last_handle = pgaio_io_acquire(CurrentResourceOwner, NULL);
+
+ PG_RETURN_VOID();
+}
+
+PG_FUNCTION_INFO_V1(handle_release_last);
+Datum
+handle_release_last(PG_FUNCTION_ARGS)
+{
+ if (!last_handle)
+ elog(ERROR, "no handle");
+
+ pgaio_io_release(last_handle);
+
+ PG_RETURN_VOID();
+}
+
+PG_FUNCTION_INFO_V1(handle_get_and_error);
+Datum
+handle_get_and_error(PG_FUNCTION_ARGS)
+{
+ pgaio_io_acquire(CurrentResourceOwner, NULL);
+
+ elog(ERROR, "as you command");
+ PG_RETURN_VOID();
+}
+
+PG_FUNCTION_INFO_V1(handle_get_twice);
+Datum
+handle_get_twice(PG_FUNCTION_ARGS)
+{
+ pgaio_io_acquire(CurrentResourceOwner, NULL);
+ pgaio_io_acquire(CurrentResourceOwner, NULL);
+
+ PG_RETURN_VOID();
+}
+
+PG_FUNCTION_INFO_V1(handle_get_release);
+Datum
+handle_get_release(PG_FUNCTION_ARGS)
+{
+ PgAioHandle *handle;
+
+ handle = pgaio_io_acquire(CurrentResourceOwner, NULL);
+ pgaio_io_release(handle);
+
+ PG_RETURN_VOID();
+}
+
+PG_FUNCTION_INFO_V1(batch_start);
+Datum
+batch_start(PG_FUNCTION_ARGS)
+{
+ pgaio_enter_batchmode();
+ PG_RETURN_VOID();
+}
+
+PG_FUNCTION_INFO_V1(batch_end);
+Datum
+batch_end(PG_FUNCTION_ARGS)
+{
+ pgaio_exit_batchmode();
+ PG_RETURN_VOID();
+}
+
+#ifdef USE_INJECTION_POINTS
+extern PGDLLEXPORT void inj_io_short_read(const char *name, const void *private_data);
+extern PGDLLEXPORT void inj_io_reopen(const char *name, const void *private_data);
+
+void
+inj_io_short_read(const char *name, const void *private_data)
+{
+ PgAioHandle *ioh;
+
+ ereport(LOG,
+ errmsg("short read injection point called, is enabled: %d",
+ inj_io_error_state->enabled_reopen),
+ errhidestmt(true), errhidecontext(true));
+
+ if (inj_io_error_state->enabled_short_read)
+ {
+ ioh = pgaio_inj_io_get();
+
+ /*
+ * Only shorten reads that are actually longer than the target size,
+ * otherwise we can trigger over-reads.
+ */
+ if (inj_io_error_state->short_read_result_set
+ && ioh->op == PGAIO_OP_READV
+ && inj_io_error_state->short_read_result <= ioh->result)
+ {
+ struct iovec *iov = &pgaio_ctl->iovecs[ioh->iovec_off];
+ int32 old_result = ioh->result;
+ int32 new_result = inj_io_error_state->short_read_result;
+ int32 processed = 0;
+
+ ereport(LOG,
+ errmsg("short read inject point, changing result from %d to %d",
+ old_result, new_result),
+ errhidestmt(true), errhidecontext(true));
+
+ /*
+ * The underlying IO actually completed OK, and thus the "invalid"
+ * portion of the IOV actually contains valid data. That can hide
+ * a lot of problems, e.g. if we were to wrongly mark a buffer,
+ * that wasn't read according to the shortened-read, IO as valid,
+ * the contents would look valid and we might miss a bug.
+ *
+ * To avoid that, iterate through the IOV and zero out the
+ * "failed" portion of the IO.
+ */
+ for (int i = 0; i < ioh->op_data.read.iov_length; i++)
+ {
+ if (processed + iov[i].iov_len <= new_result)
+ processed += iov[i].iov_len;
+ else if (processed <= new_result)
+ {
+ uint32 ok_part = new_result - processed;
+
+ memset((char *) iov[i].iov_base + ok_part, 0, iov[i].iov_len - ok_part);
+ processed += iov[i].iov_len;
+ }
+ else
+ {
+ memset((char *) iov[i].iov_base, 0, iov[i].iov_len);
+ }
+ }
+
+ ioh->result = new_result;
+ }
+ }
+}
+
+void
+inj_io_reopen(const char *name, const void *private_data)
+{
+ ereport(LOG,
+ errmsg("reopen injection point called, is enabled: %d",
+ inj_io_error_state->enabled_reopen),
+ errhidestmt(true), errhidecontext(true));
+
+ if (inj_io_error_state->enabled_reopen)
+ elog(ERROR, "injection point triggering failure to reopen ");
+}
+#endif
+
+PG_FUNCTION_INFO_V1(inj_io_short_read_attach);
+Datum
+inj_io_short_read_attach(PG_FUNCTION_ARGS)
+{
+#ifdef USE_INJECTION_POINTS
+ inj_io_error_state->enabled_short_read = true;
+ inj_io_error_state->short_read_result_set = !PG_ARGISNULL(0);
+ if (inj_io_error_state->short_read_result_set)
+ inj_io_error_state->short_read_result = PG_GETARG_INT32(0);
+#else
+ elog(ERROR, "injection points not supported");
+#endif
+
+ PG_RETURN_VOID();
+}
+
+PG_FUNCTION_INFO_V1(inj_io_short_read_detach);
+Datum
+inj_io_short_read_detach(PG_FUNCTION_ARGS)
+{
+#ifdef USE_INJECTION_POINTS
+ inj_io_error_state->enabled_short_read = false;
+#else
+ elog(ERROR, "injection points not supported");
+#endif
+ PG_RETURN_VOID();
+}
+
+PG_FUNCTION_INFO_V1(inj_io_reopen_attach);
+Datum
+inj_io_reopen_attach(PG_FUNCTION_ARGS)
+{
+#ifdef USE_INJECTION_POINTS
+ inj_io_error_state->enabled_reopen = true;
+#else
+ elog(ERROR, "injection points not supported");
+#endif
+
+ PG_RETURN_VOID();
+}
+
+PG_FUNCTION_INFO_V1(inj_io_reopen_detach);
+Datum
+inj_io_reopen_detach(PG_FUNCTION_ARGS)
+{
+#ifdef USE_INJECTION_POINTS
+ inj_io_error_state->enabled_reopen = false;
+#else
+ elog(ERROR, "injection points not supported");
+#endif
+ PG_RETURN_VOID();
+}
diff --git a/src/test/modules/test_aio/test_aio.control b/src/test/modules/test_aio/test_aio.control
new file mode 100644
index 00000000000..cd91c3ed16b
--- /dev/null
+++ b/src/test/modules/test_aio/test_aio.control
@@ -0,0 +1,3 @@
+comment = 'Test code for AIO'
+default_version = '1.0'
+module_pathname = '$libdir/test_aio'
--
2.48.1.76.g4e746b1a31.dirty
[text/x-diff] v2.15-0005-md-Add-comment-assert-to-buffer-zeroing-path-i.patch (4.5K, ../../7s6fclfekpcxoaaorwrq67v4vmgixf2dcjcgznuj7vxs3ie3wq@okqbi5vabqbx/6-v2.15-0005-md-Add-comment-assert-to-buffer-zeroing-path-i.patch)
download | inline diff:
From 670455bcb88ee0925276df5da07c3126fcd37434 Mon Sep 17 00:00:00 2001
From: Andres Freund <andres@anarazel.de>
Date: Mon, 31 Mar 2025 19:27:04 -0400
Subject: [PATCH v2.15 05/18] md: Add comment & assert to buffer-zeroing path
in md[start]readv()
mdreadv() has a codepath to zero out buffers when a read returns zero bytes,
guarded by a check for zero_damaged_pages || InRecovery.
The InRecovery codepath to zero out buffers in mdreadv() appears to be
unreachable. The only known paths to reach mdreadv()/mdstartreadv() in
recovery are XLogReadBufferExtended(), vm_readbuf(), and fsm_readbuf(), each
of which takes care to extend the relation if necessary. This looks to either
have been the case for a long time, or the code was never reachable.
The zero_damaged_pages path is incomplete, as as missing segments are not
created.
Putting blocks into the buffer-pool that do not exist on disk is rather
problematic, as such blocks will, at least initially, not be found by scans
that rely on smgrnblocks(), as they are beyond EOF. It also can cause weird
problems with relation extension, as relation extension does not expect blocks
beyond EOF to exist.
Therefore we would like to remove that path.
mdstartreadv(), which I added in e5fe570b51c, does not implement this zeroing
logic. I had started a discussion about that a while ago (linked below), but
forgot to act on the conclusion of the discussion, namely to disable the
in-memory-zeroing behavior.
We could certainly implement equivalent zeroing logic in mdstartreadv(), but
it would have to be more complicated due to potential differences in the
zero_damaged_pages setting between the definer and completor of IO. Given that
we want to remove the logic, that does not seem worth implementing the
necessary logic.
For now, put an Assert(false) comments documenting this choice into mdreadv()
and comments documenting the deprecation of the path in mdreadv() and the
non-implementation of it in mdstartreadv(). If we, during testing, discover
that we do need the path, we can implement it at that time.
Discussion: https://postgr.es/m/postgr.es/m/20250330024513.ac.nmisch@google.com
Discussion: https://postgr.es/m/postgr.es/m/3qxxsnciyffyf3wyguiz4besdp5t5uxvv3utg75cbcszojlz7p@uibfzmnukkbd
---
src/backend/storage/smgr/md.c | 28 ++++++++++++++++++++++++++++
1 file changed, 28 insertions(+)
diff --git a/src/backend/storage/smgr/md.c b/src/backend/storage/smgr/md.c
index b55892b9405..ecc33713d8f 100644
--- a/src/backend/storage/smgr/md.c
+++ b/src/backend/storage/smgr/md.c
@@ -910,9 +910,30 @@ mdreadv(SMgrRelation reln, ForkNumber forknum, BlockNumber blocknum,
* is ON or we are InRecovery, we should instead return zeroes
* without complaining. This allows, for example, the case of
* trying to update a block that was later truncated away.
+ *
+ * NB: We think that this codepath is unreachable in recovery
+ * and incomplete with zero_damaged_pages, as missing segments
+ * are not created. Putting blocks into the buffer-pool that
+ * do not exist on disk is rather problematic, as it will not
+ * be found by scans that rely on smgrnblocks(), as they are
+ * beyond EOF. It also can cause weird problems with relation
+ * extension, as relation extension does not expect blocks
+ * beyond EOF to exist.
+ *
+ * Therefore we do not want to copy the logic into
+ * mdstartreadv(), where it would have to be more complicated
+ * due to potential differences in the zero_damaged_pages
+ * setting between the definer and completor of IO.
+ *
+ * For PG 18, we are putting an Assert(false) in into
+ * mdreadv() (triggering failures in assertion-enabled builds,
+ * but continuing to work in production builds). Afterwards we
+ * plan to remove this code entirely.
*/
if (zero_damaged_pages || InRecovery)
{
+ Assert(false); /* see comment above */
+
for (BlockNumber i = transferred_this_segment / BLCKSZ;
i < nblocks_this_segment;
++i)
@@ -1007,6 +1028,13 @@ mdstartreadv(PgAioHandle *ioh,
/*
* The error checks corresponding to the post-read checks in mdreadv() are
* in md_readv_complete().
+ *
+ * However we chose, at least for now, to not implement the
+ * zero_damaged_pages logic present in mdreadv(). As outlined in mdreadv()
+ * that logic is rather problematic, and we want to get rid of it. Here
+ * equivalent logic would have to be more complicated due to potential
+ * differences in the zero_damaged_pages setting between the definer and
+ * completor of IO.
*/
}
--
2.48.1.76.g4e746b1a31.dirty
[text/x-diff] v2.15-0006-aio-comment-polishing.patch (2.0K, ../../7s6fclfekpcxoaaorwrq67v4vmgixf2dcjcgznuj7vxs3ie3wq@okqbi5vabqbx/7-v2.15-0006-aio-comment-polishing.patch)
download | inline diff:
From 11406c5999d0f2b1361cadec1a9160d7da9099c8 Mon Sep 17 00:00:00 2001
From: Andres Freund <andres@anarazel.de>
Date: Sun, 30 Mar 2025 15:38:32 -0400
Subject: [PATCH v2.15 06/18] aio: comment polishing
Author:
Reviewed-by:
Discussion: https://postgr.es/m/
Backpatch:
---
src/backend/storage/smgr/md.c | 2 +-
src/backend/storage/smgr/smgr.c | 8 ++++++++
2 files changed, 9 insertions(+), 1 deletion(-)
diff --git a/src/backend/storage/smgr/md.c b/src/backend/storage/smgr/md.c
index ecc33713d8f..bc7d94df870 100644
--- a/src/backend/storage/smgr/md.c
+++ b/src/backend/storage/smgr/md.c
@@ -1981,7 +1981,7 @@ md_readv_complete(PgAioHandle *ioh, PgAioResult prior_result, uint8 cb_data)
* might not process the query result immediately (because it is busy
* doing another part of query processing) or at all (e.g. if it was
* cancelled or errored out due to another IO also failing). The
- * issuer of the IO will emit an ERROR when processing the IO's
+ * definer of the IO will emit an ERROR when processing the IO's
* results
*/
pgaio_result_report(result, td, LOG_SERVER_ONLY);
diff --git a/src/backend/storage/smgr/smgr.c b/src/backend/storage/smgr/smgr.c
index 320dc04d83a..d532fea39d9 100644
--- a/src/backend/storage/smgr/smgr.c
+++ b/src/backend/storage/smgr/smgr.c
@@ -738,6 +738,14 @@ smgrreadv(SMgrRelation reln, ForkNumber forknum, BlockNumber blocknum,
* blocks not successfully read might bear unspecified modifications, up to
* the full nblocks). This maintains the abstraction that smgr operates on the
* level of blocks, rather than bytes.
+ *
+ * Compared to smgrreadv(), more responsibilities fall on the caller:
+ * - Partial reads need to be handle by the caller re-issuing IO for the
+ * unread blocks
+ * - smgr will ereport(LOG_SERVER_ONLY) some problems, but higher layers are
+ * responsible for pgaio_result_report() to mirror that news to the user (if
+ * the IO results in PGAIO_RS_WARNING) or abort the (sub)transaction (if
+ * PGAIO_RS_ERROR).
*/
void
smgrstartreadv(PgAioHandle *ioh,
--
2.48.1.76.g4e746b1a31.dirty
[text/x-diff] v2.15-0007-aio-Add-errcontext-for-processing-I-Os-for-ano.patch (4.8K, ../../7s6fclfekpcxoaaorwrq67v4vmgixf2dcjcgznuj7vxs3ie3wq@okqbi5vabqbx/8-v2.15-0007-aio-Add-errcontext-for-processing-I-Os-for-ano.patch)
download | inline diff:
From da0fe0c1ae0d22cf22788a142725c11631d80721 Mon Sep 17 00:00:00 2001
From: Melanie Plageman <melanieplageman@gmail.com>
Date: Sun, 30 Mar 2025 14:03:55 -0400
Subject: [PATCH v2.15 07/18] aio: Add errcontext for processing I/Os for
another backend
Push an ErrorContextCallback adding additional detail about the process
performing the I/O and the owner of the I/O when those are not the same.
For io_method worker, this adds context specifying which process owns
the I/O that the I/O worker is processing.
For io_method io_uring, this adds context only when a backend is
*completing* I/O for another backend. It specifies the pid of the owning
process.
Author: Melanie Plageman <melanieplageman@gmail.com>
Discussion: https://postgr.es/m/20250325141120.8e.nmisch%40google.com
---
src/backend/storage/aio/method_io_uring.c | 31 +++++++++++++++++++++++
src/backend/storage/aio/method_worker.c | 29 +++++++++++++++++++++
2 files changed, 60 insertions(+)
diff --git a/src/backend/storage/aio/method_io_uring.c b/src/backend/storage/aio/method_io_uring.c
index 0bcdab14ae7..c719ba2727a 100644
--- a/src/backend/storage/aio/method_io_uring.c
+++ b/src/backend/storage/aio/method_io_uring.c
@@ -302,14 +302,41 @@ pgaio_uring_submit(uint16 num_staged_ios, PgAioHandle **staged_ios)
return num_staged_ios;
}
+static void
+pgaio_uring_completion_error_callback(void *arg)
+{
+ ProcNumber owner;
+ PGPROC *owner_proc;
+ int32 owner_pid;
+ PgAioHandle *ioh = arg;
+
+ if (!ioh)
+ return;
+
+ /* No need for context if a backend is completing the IO for itself */
+ if (ioh->owner_procno == MyProcNumber)
+ return;
+
+ owner = ioh->owner_procno;
+ owner_proc = GetPGProcByNumber(owner);
+ owner_pid = owner_proc->pid;
+
+ errcontext("completing I/O on behalf of process %d", owner_pid);
+}
+
static void
pgaio_uring_drain_locked(PgAioUringContext *context)
{
int ready;
int orig_ready;
+ ErrorContextCallback errcallback = {0};
Assert(LWLockHeldByMeInMode(&context->completion_lock, LW_EXCLUSIVE));
+ errcallback.callback = pgaio_uring_completion_error_callback;
+ errcallback.previous = error_context_stack;
+ error_context_stack = &errcallback;
+
/*
* Don't drain more events than available right now. Otherwise it's
* plausible that one backend could get stuck, for a while, receiving CQEs
@@ -337,9 +364,11 @@ pgaio_uring_drain_locked(PgAioUringContext *context)
PgAioHandle *ioh;
ioh = io_uring_cqe_get_data(cqe);
+ errcallback.arg = ioh;
io_uring_cqe_seen(&context->io_uring_ring, cqe);
pgaio_io_process_completion(ioh, cqe->res);
+ errcallback.arg = NULL;
}
END_CRIT_SECTION();
@@ -348,6 +377,8 @@ pgaio_uring_drain_locked(PgAioUringContext *context)
"drained %d/%d, now expecting %d",
ncqes, orig_ready, io_uring_cq_ready(&context->io_uring_ring));
}
+
+ error_context_stack = errcallback.previous;
}
static void
diff --git a/src/backend/storage/aio/method_worker.c b/src/backend/storage/aio/method_worker.c
index 4a7853d13fa..31d94ac82c5 100644
--- a/src/backend/storage/aio/method_worker.c
+++ b/src/backend/storage/aio/method_worker.c
@@ -357,11 +357,33 @@ pgaio_worker_register(void)
on_shmem_exit(pgaio_worker_die, 0);
}
+static void
+pgaio_worker_error_callback(void *arg)
+{
+ ProcNumber owner;
+ PGPROC *owner_proc;
+ int32 owner_pid;
+ PgAioHandle *ioh = arg;
+
+ if (!ioh)
+ return;
+
+ Assert(ioh->owner_procno != MyProcNumber);
+ Assert(MyBackendType == B_IO_WORKER);
+
+ owner = ioh->owner_procno;
+ owner_proc = GetPGProcByNumber(owner);
+ owner_pid = owner_proc->pid;
+
+ errcontext("I/O worker executing I/O on behalf of process %d", owner_pid);
+}
+
void
IoWorkerMain(const void *startup_data, size_t startup_data_len)
{
sigjmp_buf local_sigjmp_buf;
PgAioHandle *volatile error_ioh = NULL;
+ ErrorContextCallback errcallback = {0};
volatile int error_errno = 0;
char cmd[128];
@@ -388,6 +410,10 @@ IoWorkerMain(const void *startup_data, size_t startup_data_len)
sprintf(cmd, "%d", MyIoWorkerId);
set_ps_display(cmd);
+ errcallback.callback = pgaio_worker_error_callback;
+ errcallback.previous = error_context_stack;
+ error_context_stack = &errcallback;
+
/* see PostgresMain() */
if (sigsetjmp(local_sigjmp_buf, 1) != 0)
{
@@ -471,6 +497,7 @@ IoWorkerMain(const void *startup_data, size_t startup_data_len)
ioh = &pgaio_ctl->io_handles[io_index];
error_ioh = ioh;
+ errcallback.arg = ioh;
pgaio_debug_io(DEBUG4, ioh,
"worker %d processing IO",
@@ -511,6 +538,7 @@ IoWorkerMain(const void *startup_data, size_t startup_data_len)
pgaio_io_perform_synchronously(ioh);
RESUME_INTERRUPTS();
+ errcallback.arg = NULL;
}
else
{
@@ -522,6 +550,7 @@ IoWorkerMain(const void *startup_data, size_t startup_data_len)
CHECK_FOR_INTERRUPTS();
}
+ error_context_stack = errcallback.previous;
proc_exit(0);
}
--
2.48.1.76.g4e746b1a31.dirty
[text/x-diff] v2.15-0008-aio-Experimental-heuristics-to-increase-batchi.patch (1.6K, ../../7s6fclfekpcxoaaorwrq67v4vmgixf2dcjcgznuj7vxs3ie3wq@okqbi5vabqbx/9-v2.15-0008-aio-Experimental-heuristics-to-increase-batchi.patch)
download | inline diff:
From b642a383b99b3f536c69e0b134223eb52f819314 Mon Sep 17 00:00:00 2001
From: Andres Freund <andres@anarazel.de>
Date: Tue, 18 Mar 2025 14:40:06 -0400
Subject: [PATCH v2.15 08/18] aio: Experimental heuristics to increase batching
in read_stream.c
Author:
Reviewed-by:
Discussion: https://postgr.es/m/
Backpatch:
---
src/backend/storage/aio/read_stream.c | 24 ++++++++++++++++++++++++
1 file changed, 24 insertions(+)
diff --git a/src/backend/storage/aio/read_stream.c b/src/backend/storage/aio/read_stream.c
index 36c54fb695b..cec93129f58 100644
--- a/src/backend/storage/aio/read_stream.c
+++ b/src/backend/storage/aio/read_stream.c
@@ -404,6 +404,30 @@ read_stream_start_pending_read(ReadStream *stream)
static void
read_stream_look_ahead(ReadStream *stream)
{
+ /*
+ * Batch-submitting multiple IOs is more efficient than doing so
+ * one-by-one. If we just ramp up to the max, we'll only be allowed to
+ * submit one io_combine_limit sized IO. Defer submitting IO in that case.
+ *
+ * FIXME: This needs better heuristics.
+ */
+#if 1
+ if (!stream->sync_mode && stream->distance > (io_combine_limit * 8))
+ {
+ if (stream->pinned_buffers + stream->pending_read_nblocks > ((stream->distance * 3) / 4))
+ {
+#if 0
+ ereport(LOG,
+ errmsg("reduce reduce reduce: pinned: %d, pending: %d, distance: %d",
+ stream->pinned_buffers,
+ stream->pending_read_nblocks,
+ stream->distance));
+#endif
+ return;
+ }
+ }
+#endif
+
/*
* Allow amortizing the cost of submitting IO over multiple IOs. This
* requires that we don't do any operations that could lead to a deadlock
--
2.48.1.76.g4e746b1a31.dirty
[text/x-diff] v2.15-0009-aio-Implement-smgr-md-fd-write-support.patch (14.5K, ../../7s6fclfekpcxoaaorwrq67v4vmgixf2dcjcgznuj7vxs3ie3wq@okqbi5vabqbx/10-v2.15-0009-aio-Implement-smgr-md-fd-write-support.patch)
download | inline diff:
From 37407ad3a4d1aed04a4bbb1ba316a23854128722 Mon Sep 17 00:00:00 2001
From: Andres Freund <andres@anarazel.de>
Date: Sat, 29 Mar 2025 13:28:52 -0400
Subject: [PATCH v2.15 09/18] aio: Implement smgr/md/fd write support
TODO:
- Right now the sync.c integration with smgr.c/md.c isn't properly safe to use
in a critical section
The only reason it doesn't immediately fail is that it's reasonably rare
that RegisterSyncRequest() fails *and* either:
- smgropen()->hash_search(HASH_ENTER) decides to resize the hash table, even
though the lookup is guaranteed to succeed for io_method=worker.
- an io_method=uring completion is run in a different backend and smgropen()
needs to build a new entry and thus needs to allocate memory
For a bit I thought this could be worked around easily enough by not doing
an smgropen() in mdsyncfiletag(), or adding a "fallible" smgropen() and
instead just opening the file directly. That actually does kinda solve the
problem, but only because the memory allocation in PathNameOpenFile()
uses malloc(), not palloc() and thus doesn't trigger
- temp_file_limit implementation
---
src/include/storage/aio.h | 1 +
src/include/storage/fd.h | 1 +
src/include/storage/md.h | 5 +
src/include/storage/smgr.h | 5 +
src/backend/storage/aio/aio_callback.c | 1 +
src/backend/storage/file/fd.c | 28 ++++
src/backend/storage/smgr/md.c | 199 +++++++++++++++++++++++++
src/backend/storage/smgr/smgr.c | 29 ++++
8 files changed, 269 insertions(+)
diff --git a/src/include/storage/aio.h b/src/include/storage/aio.h
index 9fe9d9ad9fa..bfe0d93683b 100644
--- a/src/include/storage/aio.h
+++ b/src/include/storage/aio.h
@@ -194,6 +194,7 @@ typedef enum PgAioHandleCallbackID
PGAIO_HCB_INVALID = 0,
PGAIO_HCB_MD_READV,
+ PGAIO_HCB_MD_WRITEV,
PGAIO_HCB_SHARED_BUFFER_READV,
diff --git a/src/include/storage/fd.h b/src/include/storage/fd.h
index b77d8e5e30e..2cc7c5a4761 100644
--- a/src/include/storage/fd.h
+++ b/src/include/storage/fd.h
@@ -112,6 +112,7 @@ extern int FilePrefetch(File file, off_t offset, off_t amount, uint32 wait_event
extern ssize_t FileReadV(File file, const struct iovec *iov, int iovcnt, off_t offset, uint32 wait_event_info);
extern ssize_t FileWriteV(File file, const struct iovec *iov, int iovcnt, off_t offset, uint32 wait_event_info);
extern int FileStartReadV(struct PgAioHandle *ioh, File file, int iovcnt, off_t offset, uint32 wait_event_info);
+extern int FileStartWriteV(struct PgAioHandle *ioh, File file, int iovcnt, off_t offset, uint32 wait_event_info);
extern int FileSync(File file, uint32 wait_event_info);
extern int FileZero(File file, off_t offset, off_t amount, uint32 wait_event_info);
extern int FileFallocate(File file, off_t offset, off_t amount, uint32 wait_event_info);
diff --git a/src/include/storage/md.h b/src/include/storage/md.h
index 9d7131eff43..47ae6c36c94 100644
--- a/src/include/storage/md.h
+++ b/src/include/storage/md.h
@@ -21,6 +21,7 @@
#include "storage/sync.h"
extern const PgAioHandleCallbacks aio_md_readv_cb;
+extern const PgAioHandleCallbacks aio_md_writev_cb;
/* md storage manager functionality */
extern void mdinit(void);
@@ -45,6 +46,10 @@ extern void mdstartreadv(PgAioHandle *ioh,
extern void mdwritev(SMgrRelation reln, ForkNumber forknum,
BlockNumber blocknum,
const void **buffers, BlockNumber nblocks, bool skipFsync);
+extern void mdstartwritev(PgAioHandle *ioh,
+ SMgrRelation reln, ForkNumber forknum,
+ BlockNumber blocknum,
+ const void **buffers, BlockNumber nblocks, bool skipFsync);
extern void mdwriteback(SMgrRelation reln, ForkNumber forknum,
BlockNumber blocknum, BlockNumber nblocks);
extern BlockNumber mdnblocks(SMgrRelation reln, ForkNumber forknum);
diff --git a/src/include/storage/smgr.h b/src/include/storage/smgr.h
index 856ebcda350..f00b3763ac9 100644
--- a/src/include/storage/smgr.h
+++ b/src/include/storage/smgr.h
@@ -108,6 +108,11 @@ extern void smgrwritev(SMgrRelation reln, ForkNumber forknum,
BlockNumber blocknum,
const void **buffers, BlockNumber nblocks,
bool skipFsync);
+extern void smgrstartwritev(PgAioHandle *ioh,
+ SMgrRelation reln, ForkNumber forknum,
+ BlockNumber blocknum,
+ const void **buffers, BlockNumber nblocks,
+ bool skipFsync);
extern void smgrwriteback(SMgrRelation reln, ForkNumber forknum,
BlockNumber blocknum, BlockNumber nblocks);
extern BlockNumber smgrnblocks(SMgrRelation reln, ForkNumber forknum);
diff --git a/src/backend/storage/aio/aio_callback.c b/src/backend/storage/aio/aio_callback.c
index bf42778a48c..abe5b191b42 100644
--- a/src/backend/storage/aio/aio_callback.c
+++ b/src/backend/storage/aio/aio_callback.c
@@ -41,6 +41,7 @@ static const PgAioHandleCallbacksEntry aio_handle_cbs[] = {
CALLBACK_ENTRY(PGAIO_HCB_INVALID, aio_invalid_cb),
CALLBACK_ENTRY(PGAIO_HCB_MD_READV, aio_md_readv_cb),
+ CALLBACK_ENTRY(PGAIO_HCB_MD_WRITEV, aio_md_writev_cb),
CALLBACK_ENTRY(PGAIO_HCB_SHARED_BUFFER_READV, aio_shared_buffer_readv_cb),
diff --git a/src/backend/storage/file/fd.c b/src/backend/storage/file/fd.c
index 0e8299dd556..2138d47dab9 100644
--- a/src/backend/storage/file/fd.c
+++ b/src/backend/storage/file/fd.c
@@ -2348,6 +2348,34 @@ retry:
return returnCode;
}
+int
+FileStartWriteV(PgAioHandle *ioh, File file,
+ int iovcnt, off_t offset,
+ uint32 wait_event_info)
+{
+ int returnCode;
+ Vfd *vfdP;
+
+ Assert(FileIsValid(file));
+
+ DO_DB(elog(LOG, "FileStartWriteV: %d (%s) " INT64_FORMAT " %d",
+ file, VfdCache[file].fileName,
+ (int64) offset,
+ iovcnt));
+
+ returnCode = FileAccess(file);
+ if (returnCode < 0)
+ return returnCode;
+
+ vfdP = &VfdCache[file];
+
+ /* FIXME: think about / reimplement temp_file_limit */
+
+ pgaio_io_start_writev(ioh, vfdP->fd, iovcnt, offset);
+
+ return 0;
+}
+
int
FileSync(File file, uint32 wait_event_info)
{
diff --git a/src/backend/storage/smgr/md.c b/src/backend/storage/smgr/md.c
index bc7d94df870..bacfe5f121b 100644
--- a/src/backend/storage/smgr/md.c
+++ b/src/backend/storage/smgr/md.c
@@ -155,12 +155,19 @@ static BlockNumber _mdnblocks(SMgrRelation reln, ForkNumber forknum,
static PgAioResult md_readv_complete(PgAioHandle *ioh, PgAioResult prior_result, uint8 cb_data);
static void md_readv_report(PgAioResult result, const PgAioTargetData *target_data, int elevel);
+static PgAioResult md_writev_complete(PgAioHandle *ioh, PgAioResult prior_result, uint8 cb_data);
+static void md_writev_report(PgAioResult result, const PgAioTargetData *target_data, int elevel);
const PgAioHandleCallbacks aio_md_readv_cb = {
.complete_shared = md_readv_complete,
.report = md_readv_report,
};
+const PgAioHandleCallbacks aio_md_writev_cb = {
+ .complete_shared = md_writev_complete,
+ .report = md_writev_report,
+};
+
static inline int
_mdfd_open_flags(void)
@@ -1143,6 +1150,64 @@ mdwritev(SMgrRelation reln, ForkNumber forknum, BlockNumber blocknum,
}
}
+/*
+ * mdstartwritev() -- Asynchronous version of mdrwritev().
+ */
+void
+mdstartwritev(PgAioHandle *ioh,
+ SMgrRelation reln, ForkNumber forknum, BlockNumber blocknum,
+ const void **buffers, BlockNumber nblocks, bool skipFsync)
+{
+ off_t seekpos;
+ MdfdVec *v;
+ BlockNumber nblocks_this_segment;
+ struct iovec *iov;
+ int iovcnt;
+ int ret;
+
+ v = _mdfd_getseg(reln, forknum, blocknum, false,
+ EXTENSION_FAIL | EXTENSION_CREATE_RECOVERY);
+
+ seekpos = (off_t) BLCKSZ * (blocknum % ((BlockNumber) RELSEG_SIZE));
+
+ Assert(seekpos < (off_t) BLCKSZ * RELSEG_SIZE);
+
+ nblocks_this_segment =
+ Min(nblocks,
+ RELSEG_SIZE - (blocknum % ((BlockNumber) RELSEG_SIZE)));
+
+ if (nblocks_this_segment != nblocks)
+ elog(ERROR, "write crossing segment boundary");
+
+ iovcnt = pgaio_io_get_iovec(ioh, &iov);
+
+ Assert(nblocks <= iovcnt);
+
+ iovcnt = buffers_to_iovec(iov, unconstify(void **, buffers), nblocks_this_segment);
+
+ Assert(iovcnt <= nblocks_this_segment);
+
+ if (!(io_direct_flags & IO_DIRECT_DATA))
+ pgaio_io_set_flag(ioh, PGAIO_HF_BUFFERED);
+
+ pgaio_io_set_target_smgr(ioh,
+ reln,
+ forknum,
+ blocknum,
+ nblocks,
+ skipFsync);
+ pgaio_io_register_callbacks(ioh, PGAIO_HCB_MD_WRITEV, 0);
+
+ ret = FileStartWriteV(ioh, v->mdfd_vfd, iovcnt, seekpos, WAIT_EVENT_DATA_FILE_WRITE);
+ if (ret != 0)
+ ereport(ERROR,
+ (errcode_for_file_access(),
+ errmsg("could not start writing blocks %u..%u in file \"%s\": %m",
+ blocknum,
+ blocknum + nblocks_this_segment - 1,
+ FilePathName(v->mdfd_vfd))));
+}
+
/*
* mdwriteback() -- Tell the kernel to write pages back to storage.
@@ -1531,6 +1596,40 @@ register_dirty_segment(SMgrRelation reln, ForkNumber forknum, MdfdVec *seg)
}
}
+/*
+ * Like register_dirty_segment(), except for use by AIO. In the completion
+ * callback we don't have access to the MdfdVec (the completion callback might
+ * be executed in a different backend than the issuing backend), therefore we
+ * have to implement this slightly differently.
+ */
+static void
+register_dirty_segment_aio(RelFileLocator locator, ForkNumber forknum, uint64 segno)
+{
+ FileTag tag;
+
+ INIT_MD_FILETAG(tag, locator, forknum, segno);
+
+ /*
+ * Can't block here waiting for checkpointer to accept our sync request,
+ * as checkpointer might be waiting for this AIO to finish if offloaded to
+ * a worker.
+ */
+ if (!RegisterSyncRequest(&tag, SYNC_REQUEST, false /* retryOnError */ ))
+ {
+ char path[MAXPGPATH];
+
+ ereport(DEBUG1,
+ (errmsg_internal("could not forward fsync request because request queue is full")));
+
+ /* reuse mdsyncfiletag() to avoid duplicating code */
+ if (mdsyncfiletag(&tag, path))
+ ereport(data_sync_elevel(ERROR),
+ (errcode_for_file_access(),
+ errmsg("could not fsync file \"%s\": %m",
+ path)));
+ }
+}
+
/*
* register_unlink_segment() -- Schedule a file to be deleted after next checkpoint
*/
@@ -2065,3 +2164,103 @@ md_readv_report(PgAioResult result, const PgAioTargetData *td, int elevel)
td->smgr.nblocks * (size_t) BLCKSZ));
}
}
+
+/*
+ * AIO completion callback for mdstartwritev().
+ */
+static PgAioResult
+md_writev_complete(PgAioHandle *ioh, PgAioResult prior_result, uint8 cb_data)
+{
+ PgAioTargetData *td = pgaio_io_get_target_data(ioh);
+ PgAioResult result = prior_result;
+
+ if (prior_result.result < 0)
+ {
+ result.status = PGAIO_RS_ERROR;
+ result.id = PGAIO_HCB_MD_WRITEV;
+ /* For "hard" errors, track the error number in error_data */
+ result.error_data = -prior_result.result;
+ result.result = 0;
+
+ pgaio_result_report(result, td, LOG);
+
+ return result;
+ }
+
+ /*
+ * As explained above smgrstartwritev(), the smgr API operates on the
+ * level of blocks, rather than bytes. Convert.
+ */
+ result.result /= BLCKSZ;
+
+ Assert(result.result <= td->smgr.nblocks);
+
+ if (result.result == 0)
+ {
+ /* consider 0 blocks written a failure */
+ result.status = PGAIO_RS_ERROR;
+ result.id = PGAIO_HCB_MD_WRITEV;
+ result.error_data = 0;
+
+ pgaio_result_report(result, td, LOG);
+
+ return result;
+ }
+
+ if (result.status != PGAIO_RS_ERROR &&
+ result.result < td->smgr.nblocks)
+ {
+ /* partial writes should be retried at upper level */
+ result.status = PGAIO_RS_PARTIAL;
+ result.id = PGAIO_HCB_MD_WRITEV;
+ }
+
+ if (!td->smgr.skip_fsync)
+ register_dirty_segment_aio(td->smgr.rlocator, td->smgr.forkNum,
+ td->smgr.blockNum / ((BlockNumber) RELSEG_SIZE));
+
+ return result;
+}
+
+/*
+ * AIO error reporting callback for mdstartwritev().
+ */
+static void
+md_writev_report(PgAioResult result, const PgAioTargetData *td, int elevel)
+{
+ RelPathStr path;
+
+ path = relpathbackend(td->smgr.rlocator,
+ td->smgr.is_temp ? MyProcNumber : INVALID_PROC_NUMBER,
+ td->smgr.forkNum);
+
+ if (result.error_data != 0)
+ {
+ errno = result.error_data; /* for errcode_for_file_access() */
+
+ ereport(elevel,
+ errcode_for_file_access(),
+ errmsg("could not write blocks %u..%u in file \"%s\": %m",
+ td->smgr.blockNum,
+ td->smgr.blockNum + td->smgr.nblocks,
+ path.str)
+ );
+ }
+ else
+ {
+ /*
+ * NB: This will typically only be output in debug messages, while
+ * retrying a partial IO.
+ */
+ ereport(elevel,
+ errcode(ERRCODE_DATA_CORRUPTED),
+ errmsg("could not write blocks %u..%u in file \"%s\": wrote only %zu of %zu bytes",
+ td->smgr.blockNum,
+ td->smgr.blockNum + td->smgr.nblocks - 1,
+ path.str,
+ result.result * (size_t) BLCKSZ,
+ td->smgr.nblocks * (size_t) BLCKSZ
+ )
+ );
+ }
+}
diff --git a/src/backend/storage/smgr/smgr.c b/src/backend/storage/smgr/smgr.c
index d532fea39d9..85daf686e08 100644
--- a/src/backend/storage/smgr/smgr.c
+++ b/src/backend/storage/smgr/smgr.c
@@ -115,6 +115,11 @@ typedef struct f_smgr
BlockNumber blocknum,
const void **buffers, BlockNumber nblocks,
bool skipFsync);
+ void (*smgr_startwritev) (PgAioHandle *ioh,
+ SMgrRelation reln, ForkNumber forknum,
+ BlockNumber blocknum,
+ const void **buffers, BlockNumber nblocks,
+ bool skipFsync);
void (*smgr_writeback) (SMgrRelation reln, ForkNumber forknum,
BlockNumber blocknum, BlockNumber nblocks);
BlockNumber (*smgr_nblocks) (SMgrRelation reln, ForkNumber forknum);
@@ -142,6 +147,7 @@ static const f_smgr smgrsw[] = {
.smgr_readv = mdreadv,
.smgr_startreadv = mdstartreadv,
.smgr_writev = mdwritev,
+ .smgr_startwritev = mdstartwritev,
.smgr_writeback = mdwriteback,
.smgr_nblocks = mdnblocks,
.smgr_truncate = mdtruncate,
@@ -795,6 +801,29 @@ smgrwritev(SMgrRelation reln, ForkNumber forknum, BlockNumber blocknum,
RESUME_INTERRUPTS();
}
+/*
+ * smgrstartwritev() -- asynchronous version of smgrwritev()
+ *
+ * This starts an asynchronous writev IO using the IO handle `ioh`. Other than
+ * `ioh` all parameters are the same as smgrwritev().
+ *
+ * Completion callbacks above smgr will be passed the result as the number of
+ * successfully written blocks if the write [partially] succeeds. This
+ * maintains the abstraction that smgr operates on the level of blocks, rather
+ * than bytes.
+ */
+void
+smgrstartwritev(PgAioHandle *ioh,
+ SMgrRelation reln, ForkNumber forknum, BlockNumber blocknum,
+ const void **buffers, BlockNumber nblocks, bool skipFsync)
+{
+ HOLD_INTERRUPTS();
+ smgrsw[reln->smgr_which].smgr_startwritev(ioh,
+ reln, forknum, blocknum, buffers,
+ nblocks, skipFsync);
+ RESUME_INTERRUPTS();
+}
+
/*
* smgrwriteback() -- Trigger kernel writeback for the supplied range of
* blocks.
--
2.48.1.76.g4e746b1a31.dirty
[text/x-diff] v2.15-0010-aio-Add-bounce-buffers.patch (23.2K, ../../7s6fclfekpcxoaaorwrq67v4vmgixf2dcjcgznuj7vxs3ie3wq@okqbi5vabqbx/11-v2.15-0010-aio-Add-bounce-buffers.patch)
download | inline diff:
From 43ac0c1934bd71b25a6ceef8074da48671145a32 Mon Sep 17 00:00:00 2001
From: Andres Freund <andres@anarazel.de>
Date: Tue, 18 Mar 2025 14:40:06 -0400
Subject: [PATCH v2.15 10/18] aio: Add bounce buffers
---
src/include/storage/aio.h | 15 ++
src/include/storage/aio_internal.h | 34 ++++
src/include/storage/aio_types.h | 2 +
src/include/utils/resowner.h | 2 +
src/backend/storage/aio/README.md | 27 +++
src/backend/storage/aio/aio.c | 178 ++++++++++++++++++
src/backend/storage/aio/aio_init.c | 123 ++++++++++++
src/backend/utils/misc/guc_tables.c | 13 ++
src/backend/utils/misc/postgresql.conf.sample | 2 +
src/backend/utils/resowner/resowner.c | 25 ++-
src/test/modules/test_aio/test_aio--1.0.sql | 21 +++
src/test/modules/test_aio/test_aio.c | 55 ++++++
src/tools/pgindent/typedefs.list | 1 +
13 files changed, 496 insertions(+), 2 deletions(-)
diff --git a/src/include/storage/aio.h b/src/include/storage/aio.h
index bfe0d93683b..f91f0afc5a5 100644
--- a/src/include/storage/aio.h
+++ b/src/include/storage/aio.h
@@ -353,6 +353,20 @@ extern bool pgaio_have_staged(void);
+/* --------------------------------------------------------------------------------
+ * Bounce Buffers
+ * --------------------------------------------------------------------------------
+ */
+
+extern PgAioBounceBuffer *pgaio_bounce_buffer_get(void);
+extern void pgaio_io_assoc_bounce_buffer(PgAioHandle *ioh, PgAioBounceBuffer *bb);
+extern uint32 pgaio_bounce_buffer_id(PgAioBounceBuffer *bb);
+extern void pgaio_bounce_buffer_release(PgAioBounceBuffer *bb);
+extern char *pgaio_bounce_buffer_buffer(PgAioBounceBuffer *bb);
+extern void pgaio_bounce_buffer_release_resowner(struct dlist_node *bb_node, bool on_error);
+
+
+
/* --------------------------------------------------------------------------------
* Other
* --------------------------------------------------------------------------------
@@ -365,6 +379,7 @@ extern void pgaio_closing_fd(int fd);
/* GUCs */
extern PGDLLIMPORT int io_method;
extern PGDLLIMPORT int io_max_concurrency;
+extern PGDLLIMPORT int io_bounce_buffers;
#endif /* AIO_H */
diff --git a/src/include/storage/aio_internal.h b/src/include/storage/aio_internal.h
index 7f18da2c856..833f97361a1 100644
--- a/src/include/storage/aio_internal.h
+++ b/src/include/storage/aio_internal.h
@@ -127,6 +127,12 @@ struct PgAioHandle
/* raw result of the IO operation */
int32 result;
+ /*
+ * List of bounce_buffers owned by IO. It would suffice to use an index
+ * based linked list here.
+ */
+ slist_head bounce_buffers;
+
/**
* In which list the handle is registered, depends on the state:
* - IDLE, in per-backend list
@@ -182,11 +188,24 @@ struct PgAioHandle
};
+/* typedef is in aio_types.h */
+struct PgAioBounceBuffer
+{
+ slist_node node;
+ struct ResourceOwnerData *resowner;
+ dlist_node resowner_node;
+ char *buffer;
+};
+
+
typedef struct PgAioBackend
{
/* index into PgAioCtl->io_handles */
uint32 io_handle_off;
+ /* index into PgAioCtl->bounce_buffers */
+ uint32 bounce_buffers_off;
+
/* IO Handles that currently are not used */
dclist_head idle_ios;
@@ -217,6 +236,12 @@ typedef struct PgAioBackend
* IOs being appended at the end.
*/
dclist_head in_flight_ios;
+
+ /* Bounce Buffers that currently are not used */
+ slist_head idle_bbs;
+
+ /* see handed_out_io */
+ PgAioBounceBuffer *handed_out_bb;
} PgAioBackend;
@@ -244,6 +269,15 @@ typedef struct PgAioCtl
uint32 io_handle_count;
PgAioHandle *io_handles;
+
+ /*
+ * To perform AIO on buffers that are not located in shared memory (either
+ * because they are not in shared memory or because we need to operate on
+ * a copy, as e.g. the case for writes when checksums are in use)
+ */
+ uint32 bounce_buffers_count;
+ PgAioBounceBuffer *bounce_buffers;
+ char *bounce_buffers_data;
} PgAioCtl;
diff --git a/src/include/storage/aio_types.h b/src/include/storage/aio_types.h
index 18183366077..3c18dade49c 100644
--- a/src/include/storage/aio_types.h
+++ b/src/include/storage/aio_types.h
@@ -134,4 +134,6 @@ typedef struct PgAioReturn
} PgAioReturn;
+typedef struct PgAioBounceBuffer PgAioBounceBuffer;
+
#endif /* AIO_TYPES_H */
diff --git a/src/include/utils/resowner.h b/src/include/utils/resowner.h
index aede4bfc820..7e2ec224169 100644
--- a/src/include/utils/resowner.h
+++ b/src/include/utils/resowner.h
@@ -168,5 +168,7 @@ extern void ResourceOwnerForgetLock(ResourceOwner owner, struct LOCALLOCK *local
struct dlist_node;
extern void ResourceOwnerRememberAioHandle(ResourceOwner owner, struct dlist_node *ioh_node);
extern void ResourceOwnerForgetAioHandle(ResourceOwner owner, struct dlist_node *ioh_node);
+extern void ResourceOwnerRememberAioBounceBuffer(ResourceOwner owner, struct dlist_node *bb_node);
+extern void ResourceOwnerForgetAioBounceBuffer(ResourceOwner owner, struct dlist_node *bb_node);
#endif /* RESOWNER_H */
diff --git a/src/backend/storage/aio/README.md b/src/backend/storage/aio/README.md
index ddd59404a59..534aaad22bc 100644
--- a/src/backend/storage/aio/README.md
+++ b/src/backend/storage/aio/README.md
@@ -406,6 +406,33 @@ shared memory no less!), completion callbacks instead have to encode errors in
a more compact format that can be converted into an error message.
+### AIO Bounce Buffers
+
+For some uses of AIO there is no convenient memory location as the source /
+destination of an AIO. E.g. when data checksums are enabled, writes from
+shared buffers currently cannot be done directly from shared buffers, as a
+shared buffer lock still allows some modification, e.g., for hint bits (see
+`FlushBuffer()`). If the write were done in-place, such modifications can
+cause the checksum to fail.
+
+For synchronous IO this is solved by copying the buffer to separate memory
+before computing the checksum and using that copy as the source buffer for the
+AIO.
+
+However, for AIO that is not a workable solution:
+- Instead of a single buffer many buffers are required, as many IOs might be
+ in flight
+- When using the [worker method](#worker), the source/target of IO needs to be
+ in shared memory, otherwise the workers won't be able to access the memory.
+
+The AIO subsystem addresses this by providing a limited number of bounce
+buffers that can be used as the source / target for IO. A bounce buffer can be
+acquired with `pgaio_bounce_buffer_get()` and multiple bounce buffers can be
+associated with an AIO Handle with `pgaio_io_assoc_bounce_buffer()`.
+
+Bounce buffers are automatically released when the IO completes.
+
+
## Helpers
Using the low-level AIO API introduces too much complexity to do so all over
diff --git a/src/backend/storage/aio/aio.c b/src/backend/storage/aio/aio.c
index 86f7250b7a5..cff48964d07 100644
--- a/src/backend/storage/aio/aio.c
+++ b/src/backend/storage/aio/aio.c
@@ -62,6 +62,8 @@ static PgAioHandle *pgaio_io_from_wref(PgAioWaitRef *iow, uint64 *ref_generation
static const char *pgaio_io_state_get_name(PgAioHandleState s);
static void pgaio_io_wait(PgAioHandle *ioh, uint64 ref_generation);
+static void pgaio_bounce_buffer_wait_for_free(void);
+
/* Options for io_method. */
const struct config_enum_entry io_method_options[] = {
@@ -76,6 +78,7 @@ const struct config_enum_entry io_method_options[] = {
/* GUCs */
int io_method = DEFAULT_IO_METHOD;
int io_max_concurrency = -1;
+int io_bounce_buffers = -1;
/* global control for AIO */
PgAioCtl *pgaio_ctl;
@@ -662,6 +665,21 @@ pgaio_io_reclaim(PgAioHandle *ioh)
if (ioh->state != PGAIO_HS_HANDED_OUT)
dclist_delete_from(&pgaio_my_backend->in_flight_ios, &ioh->node);
+ /* reclaim all associated bounce buffers */
+ if (!slist_is_empty(&ioh->bounce_buffers))
+ {
+ slist_mutable_iter it;
+
+ slist_foreach_modify(it, &ioh->bounce_buffers)
+ {
+ PgAioBounceBuffer *bb = slist_container(PgAioBounceBuffer, node, it.cur);
+
+ slist_delete_current(&it);
+
+ slist_push_head(&pgaio_my_backend->idle_bbs, &bb->node);
+ }
+ }
+
if (ioh->resowner)
{
ResourceOwnerForgetAioHandle(ioh->resowner, &ioh->resowner_node);
@@ -1046,6 +1064,166 @@ pgaio_submit_staged(void)
+/* --------------------------------------------------------------------------------
+ * Functions primarily related to PgAioBounceBuffer
+ * --------------------------------------------------------------------------------
+ */
+
+PgAioBounceBuffer *
+pgaio_bounce_buffer_get(void)
+{
+ PgAioBounceBuffer *bb = NULL;
+ slist_node *node;
+
+ if (pgaio_my_backend->handed_out_bb != NULL)
+ elog(ERROR, "can only hand out one BB");
+
+ /*
+ * XXX: It probably is not a good idea to have bounce buffers be per
+ * backend, that's a fair bit of memory.
+ */
+ if (slist_is_empty(&pgaio_my_backend->idle_bbs))
+ {
+ pgaio_bounce_buffer_wait_for_free();
+ }
+
+ node = slist_pop_head_node(&pgaio_my_backend->idle_bbs);
+ bb = slist_container(PgAioBounceBuffer, node, node);
+
+ pgaio_my_backend->handed_out_bb = bb;
+
+ bb->resowner = CurrentResourceOwner;
+ ResourceOwnerRememberAioBounceBuffer(bb->resowner, &bb->resowner_node);
+
+ return bb;
+}
+
+void
+pgaio_io_assoc_bounce_buffer(PgAioHandle *ioh, PgAioBounceBuffer *bb)
+{
+ if (pgaio_my_backend->handed_out_bb != bb)
+ elog(ERROR, "can only assign handed out BB");
+ pgaio_my_backend->handed_out_bb = NULL;
+
+ /*
+ * There can be many bounce buffers assigned in case of vectorized IOs.
+ */
+ slist_push_head(&ioh->bounce_buffers, &bb->node);
+
+ /* once associated with an IO, the IO has ownership */
+ ResourceOwnerForgetAioBounceBuffer(bb->resowner, &bb->resowner_node);
+ bb->resowner = NULL;
+}
+
+uint32
+pgaio_bounce_buffer_id(PgAioBounceBuffer *bb)
+{
+ return bb - pgaio_ctl->bounce_buffers;
+}
+
+void
+pgaio_bounce_buffer_release(PgAioBounceBuffer *bb)
+{
+ if (pgaio_my_backend->handed_out_bb != bb)
+ elog(ERROR, "can only release handed out BB");
+
+ slist_push_head(&pgaio_my_backend->idle_bbs, &bb->node);
+ pgaio_my_backend->handed_out_bb = NULL;
+
+ ResourceOwnerForgetAioBounceBuffer(bb->resowner, &bb->resowner_node);
+ bb->resowner = NULL;
+}
+
+void
+pgaio_bounce_buffer_release_resowner(dlist_node *bb_node, bool on_error)
+{
+ PgAioBounceBuffer *bb = dlist_container(PgAioBounceBuffer, resowner_node, bb_node);
+
+ Assert(bb->resowner);
+
+ if (!on_error)
+ elog(WARNING, "leaked AIO bounce buffer");
+
+ pgaio_bounce_buffer_release(bb);
+}
+
+char *
+pgaio_bounce_buffer_buffer(PgAioBounceBuffer *bb)
+{
+ return bb->buffer;
+}
+
+static void
+pgaio_bounce_buffer_wait_for_free(void)
+{
+ static uint32 lastpos = 0;
+
+ if (pgaio_my_backend->num_staged_ios > 0)
+ {
+ pgaio_debug(DEBUG2, "submitting %d, while acquiring free bb",
+ pgaio_my_backend->num_staged_ios);
+ pgaio_submit_staged();
+ }
+
+ for (uint32 i = lastpos; i < lastpos + io_max_concurrency; i++)
+ {
+ uint32 thisoff = pgaio_my_backend->io_handle_off + (i % io_max_concurrency);
+ PgAioHandle *ioh = &pgaio_ctl->io_handles[thisoff];
+
+ switch (ioh->state)
+ {
+ case PGAIO_HS_IDLE:
+ case PGAIO_HS_HANDED_OUT:
+ continue;
+ case PGAIO_HS_DEFINED: /* should have been submitted above */
+ case PGAIO_HS_STAGED:
+ elog(ERROR, "shouldn't get here with io:%d in state %d",
+ pgaio_io_get_id(ioh), ioh->state);
+ break;
+ case PGAIO_HS_COMPLETED_IO:
+ case PGAIO_HS_SUBMITTED:
+ if (!slist_is_empty(&ioh->bounce_buffers))
+ {
+ pgaio_debug_io(DEBUG2, ioh,
+ "waiting for IO to reclaim BB with %d in flight",
+ dclist_count(&pgaio_my_backend->in_flight_ios));
+
+ /* see comment in pgaio_io_wait_for_free() about raciness */
+ pgaio_io_wait(ioh, ioh->generation);
+
+ if (slist_is_empty(&pgaio_my_backend->idle_bbs))
+ elog(WARNING, "empty after wait");
+
+ if (!slist_is_empty(&pgaio_my_backend->idle_bbs))
+ {
+ lastpos = i;
+ return;
+ }
+ }
+ break;
+ case PGAIO_HS_COMPLETED_SHARED:
+ case PGAIO_HS_COMPLETED_LOCAL:
+ /* reclaim */
+ pgaio_io_reclaim(ioh);
+
+ if (!slist_is_empty(&pgaio_my_backend->idle_bbs))
+ {
+ lastpos = i;
+ return;
+ }
+ break;
+ }
+ }
+
+ /*
+ * The submission above could have caused the IO to complete at any time.
+ */
+ if (slist_is_empty(&pgaio_my_backend->idle_bbs))
+ elog(PANIC, "no more bbs");
+}
+
+
+
/* --------------------------------------------------------------------------------
* Other
* --------------------------------------------------------------------------------
diff --git a/src/backend/storage/aio/aio_init.c b/src/backend/storage/aio/aio_init.c
index 885c3940c66..95b10933fed 100644
--- a/src/backend/storage/aio/aio_init.c
+++ b/src/backend/storage/aio/aio_init.c
@@ -88,6 +88,32 @@ AioHandleDataShmemSize(void)
io_max_concurrency));
}
+static Size
+AioBounceBufferDescShmemSize(void)
+{
+ Size sz;
+
+ /* PgAioBounceBuffer itself */
+ sz = mul_size(sizeof(PgAioBounceBuffer),
+ mul_size(AioProcs(), io_bounce_buffers));
+
+ return sz;
+}
+
+static Size
+AioBounceBufferDataShmemSize(void)
+{
+ Size sz;
+
+ /* and the associated buffer */
+ sz = mul_size(BLCKSZ,
+ mul_size(io_bounce_buffers, AioProcs()));
+ /* memory for alignment */
+ sz += BLCKSZ;
+
+ return sz;
+}
+
/*
* Choose a suitable value for io_max_concurrency.
*
@@ -113,6 +139,33 @@ AioChooseMaxConcurrency(void)
return Min(max_proportional_pins, 64);
}
+/*
+ * Choose a suitable value for io_bounce_buffers.
+ *
+ * It's very unlikely to be useful to allocate more bounce buffers for each
+ * backend than the backend is allowed to pin. Additionally, bounce buffers
+ * currently are used for writes, it seems very uncommon for more than 10% of
+ * shared_buffers to be written out concurrently.
+ *
+ * XXX: This quickly can take up significant amounts of memory, the logic
+ * should probably fine tuned.
+ */
+static int
+AioChooseBounceBuffers(void)
+{
+ uint32 max_backends;
+ int max_proportional_pins;
+
+ /* Similar logic to LimitAdditionalPins() */
+ max_backends = MaxBackends + NUM_AUXILIARY_PROCS;
+ max_proportional_pins = (NBuffers / 10) / max_backends;
+
+ max_proportional_pins = Max(max_proportional_pins, 1);
+
+ /* apply upper limit */
+ return Min(max_proportional_pins, 256);
+}
+
Size
AioShmemSize(void)
{
@@ -136,11 +189,31 @@ AioShmemSize(void)
PGC_S_OVERRIDE);
}
+
+ /*
+ * If io_bounce_buffers is -1, we automatically choose a suitable value.
+ *
+ * See also comment above.
+ */
+ if (io_bounce_buffers == -1)
+ {
+ char buf[32];
+
+ snprintf(buf, sizeof(buf), "%d", AioChooseBounceBuffers());
+ SetConfigOption("io_bounce_buffers", buf, PGC_POSTMASTER,
+ PGC_S_DYNAMIC_DEFAULT);
+ if (io_bounce_buffers == -1) /* failed to apply it? */
+ SetConfigOption("io_bounce_buffers", buf, PGC_POSTMASTER,
+ PGC_S_OVERRIDE);
+ }
+
sz = add_size(sz, AioCtlShmemSize());
sz = add_size(sz, AioBackendShmemSize());
sz = add_size(sz, AioHandleShmemSize());
sz = add_size(sz, AioHandleIOVShmemSize());
sz = add_size(sz, AioHandleDataShmemSize());
+ sz = add_size(sz, AioBounceBufferDescShmemSize());
+ sz = add_size(sz, AioBounceBufferDataShmemSize());
/* Reserve space for method specific resources. */
if (pgaio_method_ops->shmem_size)
@@ -156,6 +229,9 @@ AioShmemInit(void)
uint32 io_handle_off = 0;
uint32 iovec_off = 0;
uint32 per_backend_iovecs = io_max_concurrency * io_max_combine_limit;
+ uint32 bounce_buffers_off = 0;
+ uint32 per_backend_bb = io_bounce_buffers;
+ char *bounce_buffers_data;
pgaio_ctl = (PgAioCtl *)
ShmemInitStruct("AioCtl", AioCtlShmemSize(), &found);
@@ -167,6 +243,7 @@ AioShmemInit(void)
pgaio_ctl->io_handle_count = AioProcs() * io_max_concurrency;
pgaio_ctl->iovec_count = AioProcs() * per_backend_iovecs;
+ pgaio_ctl->bounce_buffers_count = AioProcs() * per_backend_bb;
pgaio_ctl->backend_state = (PgAioBackend *)
ShmemInitStruct("AioBackend", AioBackendShmemSize(), &found);
@@ -179,6 +256,40 @@ AioShmemInit(void)
pgaio_ctl->handle_data = (uint64 *)
ShmemInitStruct("AioHandleData", AioHandleDataShmemSize(), &found);
+ pgaio_ctl->bounce_buffers = (PgAioBounceBuffer *)
+ ShmemInitStruct("AioBounceBufferDesc", AioBounceBufferDescShmemSize(),
+ &found);
+
+ bounce_buffers_data =
+ ShmemInitStruct("AioBounceBufferData", AioBounceBufferDataShmemSize(),
+ &found);
+ bounce_buffers_data =
+ (char *) TYPEALIGN(BLCKSZ, (uintptr_t) bounce_buffers_data);
+ pgaio_ctl->bounce_buffers_data = bounce_buffers_data;
+
+
+ /* Initialize IO handles. */
+ for (uint64 i = 0; i < pgaio_ctl->io_handle_count; i++)
+ {
+ PgAioHandle *ioh = &pgaio_ctl->io_handles[i];
+
+ ioh->op = PGAIO_OP_INVALID;
+ ioh->target = PGAIO_TID_INVALID;
+ ioh->state = PGAIO_HS_IDLE;
+
+ slist_init(&ioh->bounce_buffers);
+ }
+
+ /* Initialize Bounce Buffers. */
+ for (uint64 i = 0; i < pgaio_ctl->bounce_buffers_count; i++)
+ {
+ PgAioBounceBuffer *bb = &pgaio_ctl->bounce_buffers[i];
+
+ bb->buffer = bounce_buffers_data;
+ bounce_buffers_data += BLCKSZ;
+ }
+
+
for (int procno = 0; procno < AioProcs(); procno++)
{
PgAioBackend *bs = &pgaio_ctl->backend_state[procno];
@@ -186,9 +297,13 @@ AioShmemInit(void)
bs->io_handle_off = io_handle_off;
io_handle_off += io_max_concurrency;
+ bs->bounce_buffers_off = bounce_buffers_off;
+ bounce_buffers_off += per_backend_bb;
+
dclist_init(&bs->idle_ios);
memset(bs->staged_ios, 0, sizeof(PgAioHandle *) * PGAIO_SUBMIT_BATCH_SIZE);
dclist_init(&bs->in_flight_ios);
+ slist_init(&bs->idle_bbs);
/* initialize per-backend IOs */
for (int i = 0; i < io_max_concurrency; i++)
@@ -210,6 +325,14 @@ AioShmemInit(void)
dclist_push_tail(&bs->idle_ios, &ioh->node);
iovec_off += io_max_combine_limit;
}
+
+ /* initialize per-backend bounce buffers */
+ for (int i = 0; i < per_backend_bb; i++)
+ {
+ PgAioBounceBuffer *bb = &pgaio_ctl->bounce_buffers[bs->bounce_buffers_off + i];
+
+ slist_push_head(&bs->idle_bbs, &bb->node);
+ }
}
out:
diff --git a/src/backend/utils/misc/guc_tables.c b/src/backend/utils/misc/guc_tables.c
index 4eaeca89f2c..bd49b302293 100644
--- a/src/backend/utils/misc/guc_tables.c
+++ b/src/backend/utils/misc/guc_tables.c
@@ -3293,6 +3293,19 @@ struct config_int ConfigureNamesInt[] =
check_io_max_concurrency, NULL, NULL
},
+ {
+ {"io_bounce_buffers",
+ PGC_POSTMASTER,
+ RESOURCES_IO,
+ gettext_noop("Number of IO Bounce Buffers reserved for each backend."),
+ NULL,
+ GUC_UNIT_BLOCKS
+ },
+ &io_bounce_buffers,
+ -1, -1, 4096,
+ NULL, NULL, NULL
+ },
+
{
{"io_workers",
PGC_SIGHUP,
diff --git a/src/backend/utils/misc/postgresql.conf.sample b/src/backend/utils/misc/postgresql.conf.sample
index ff56a1f0732..2c6456e907f 100644
--- a/src/backend/utils/misc/postgresql.conf.sample
+++ b/src/backend/utils/misc/postgresql.conf.sample
@@ -211,6 +211,8 @@
# -1 sets based on shared_buffers
# (change requires restart)
#io_workers = 3 # 1-32;
+#io_bounce_buffers = -1 # -1 sets based on shared_buffers
+ # (change requires restart)
# - Worker Processes -
diff --git a/src/backend/utils/resowner/resowner.c b/src/backend/utils/resowner/resowner.c
index d39f3e1b655..81e7e27965a 100644
--- a/src/backend/utils/resowner/resowner.c
+++ b/src/backend/utils/resowner/resowner.c
@@ -159,10 +159,11 @@ struct ResourceOwnerData
LOCALLOCK *locks[MAX_RESOWNER_LOCKS]; /* list of owned locks */
/*
- * AIO handles need be registered in critical sections and therefore
- * cannot use the normal ResourceElem mechanism.
+ * AIO handles & bounce buffers need be registered in critical sections
+ * and therefore cannot use the normal ResourceElem mechanism.
*/
dlist_head aio_handles;
+ dlist_head aio_bounce_buffers;
};
@@ -434,6 +435,7 @@ ResourceOwnerCreate(ResourceOwner parent, const char *name)
}
dlist_init(&owner->aio_handles);
+ dlist_init(&owner->aio_bounce_buffers);
return owner;
}
@@ -742,6 +744,13 @@ ResourceOwnerReleaseInternal(ResourceOwner owner,
pgaio_io_release_resowner(node, !isCommit);
}
+
+ while (!dlist_is_empty(&owner->aio_bounce_buffers))
+ {
+ dlist_node *node = dlist_head_node(&owner->aio_bounce_buffers);
+
+ pgaio_bounce_buffer_release_resowner(node, !isCommit);
+ }
}
else if (phase == RESOURCE_RELEASE_LOCKS)
{
@@ -1111,3 +1120,15 @@ ResourceOwnerForgetAioHandle(ResourceOwner owner, struct dlist_node *ioh_node)
{
dlist_delete_from(&owner->aio_handles, ioh_node);
}
+
+void
+ResourceOwnerRememberAioBounceBuffer(ResourceOwner owner, struct dlist_node *ioh_node)
+{
+ dlist_push_tail(&owner->aio_bounce_buffers, ioh_node);
+}
+
+void
+ResourceOwnerForgetAioBounceBuffer(ResourceOwner owner, struct dlist_node *ioh_node)
+{
+ dlist_delete_from(&owner->aio_bounce_buffers, ioh_node);
+}
diff --git a/src/test/modules/test_aio/test_aio--1.0.sql b/src/test/modules/test_aio/test_aio--1.0.sql
index e495481c41e..e2a81235166 100644
--- a/src/test/modules/test_aio/test_aio--1.0.sql
+++ b/src/test/modules/test_aio/test_aio--1.0.sql
@@ -87,6 +87,27 @@ RETURNS pg_catalog.void STRICT
AS 'MODULE_PATHNAME' LANGUAGE C;
+CREATE FUNCTION bb_get_and_error()
+RETURNS pg_catalog.void STRICT
+AS 'MODULE_PATHNAME' LANGUAGE C;
+
+CREATE FUNCTION bb_get_twice()
+RETURNS pg_catalog.void STRICT
+AS 'MODULE_PATHNAME' LANGUAGE C;
+
+CREATE FUNCTION bb_get()
+RETURNS pg_catalog.void STRICT
+AS 'MODULE_PATHNAME' LANGUAGE C;
+
+CREATE FUNCTION bb_get_release()
+RETURNS pg_catalog.void STRICT
+AS 'MODULE_PATHNAME' LANGUAGE C;
+
+CREATE FUNCTION bb_release_last()
+RETURNS pg_catalog.void STRICT
+AS 'MODULE_PATHNAME' LANGUAGE C;
+
+
/*
* Injection point related functions
diff --git a/src/test/modules/test_aio/test_aio.c b/src/test/modules/test_aio/test_aio.c
index bef3b0e3ad0..2a7f4378ef3 100644
--- a/src/test/modules/test_aio/test_aio.c
+++ b/src/test/modules/test_aio/test_aio.c
@@ -52,6 +52,7 @@ static shmem_startup_hook_type prev_shmem_startup_hook = NULL;
static PgAioHandle *last_handle;
+static PgAioBounceBuffer *last_bb;
@@ -669,6 +670,60 @@ batch_end(PG_FUNCTION_ARGS)
PG_RETURN_VOID();
}
+PG_FUNCTION_INFO_V1(bb_get);
+Datum
+bb_get(PG_FUNCTION_ARGS)
+{
+ last_bb = pgaio_bounce_buffer_get();
+
+ PG_RETURN_VOID();
+}
+
+PG_FUNCTION_INFO_V1(bb_release_last);
+Datum
+bb_release_last(PG_FUNCTION_ARGS)
+{
+ if (!last_bb)
+ elog(ERROR, "no bb");
+
+ pgaio_bounce_buffer_release(last_bb);
+
+ PG_RETURN_VOID();
+}
+
+PG_FUNCTION_INFO_V1(bb_get_and_error);
+Datum
+bb_get_and_error(PG_FUNCTION_ARGS)
+{
+ pgaio_bounce_buffer_get();
+
+ elog(ERROR, "as you command");
+ PG_RETURN_VOID();
+}
+
+PG_FUNCTION_INFO_V1(bb_get_twice);
+Datum
+bb_get_twice(PG_FUNCTION_ARGS)
+{
+ pgaio_bounce_buffer_get();
+ pgaio_bounce_buffer_get();
+
+ PG_RETURN_VOID();
+}
+
+
+PG_FUNCTION_INFO_V1(bb_get_release);
+Datum
+bb_get_release(PG_FUNCTION_ARGS)
+{
+ PgAioBounceBuffer *bb;
+
+ bb = pgaio_bounce_buffer_get();
+ pgaio_bounce_buffer_release(bb);
+
+ PG_RETURN_VOID();
+}
+
#ifdef USE_INJECTION_POINTS
extern PGDLLEXPORT void inj_io_short_read(const char *name, const void *private_data);
extern PGDLLEXPORT void inj_io_reopen(const char *name, const void *private_data);
diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list
index b66cecd8799..3a67ee01b46 100644
--- a/src/tools/pgindent/typedefs.list
+++ b/src/tools/pgindent/typedefs.list
@@ -2137,6 +2137,7 @@ PermutationStep
PermutationStepBlocker
PermutationStepBlockerType
PgAioBackend
+PgAioBounceBuffer
PgAioCtl
PgAioHandle
PgAioHandleCallbackID
--
2.48.1.76.g4e746b1a31.dirty
[text/x-diff] v2.15-0011-bufmgr-Implement-AIO-write-support.patch (11.9K, ../../7s6fclfekpcxoaaorwrq67v4vmgixf2dcjcgznuj7vxs3ie3wq@okqbi5vabqbx/12-v2.15-0011-bufmgr-Implement-AIO-write-support.patch)
download | inline diff:
From 96bbde773f50e6625a1b8e8aa8a8d1c921b4ac4b Mon Sep 17 00:00:00 2001
From: Andres Freund <andres@anarazel.de>
Date: Sun, 30 Mar 2025 16:43:54 -0400
Subject: [PATCH v2.15 11/18] bufmgr: Implement AIO write support
As of this commit there are no users of these AIO facilities, that'll come in
later commits.
Problems with AIO writes:
- Write logic needs to be rebased on-top of the patch series to not hit bit
dirty buffers while IO is going on
The performance impact of doing the memory copies is rather substantial, as
on intel memory bandwidth is *the* IO bottleneck even just for the checksum
computation, without a copy. That makes the memory copy for something like
bounce buffers hurt really badly.
And the memory usage of bounce buffers is also really concerning.
And even without checksums, several filesystems *really* don't like buffers
getting modified during DIO writes. Which I think would mean we ought to use
bounce buffers for *all* writes, which would impose a *very* substantial
overhead (basically removing the benefit of DMA happening off-cpu).
- I think it requires new lwlock.c infrastructure (as v1 of aio had), to make
LockBuffer(BUFFER_LOCK_EXCLUSIVE) etc wait in a concurrency safe manner for
in-progress writes
I can think of ways to solve this purely in bufmgr.c, but only in ways that
would cause other problems (e.g. setting BM_IO_IN_PROGRESS before waiting
for an exclusive lock) and/or expensive.
---
src/include/storage/aio.h | 4 +-
src/include/storage/bufmgr.h | 2 +
src/backend/storage/aio/aio_callback.c | 2 +
src/backend/storage/buffer/bufmgr.c | 189 ++++++++++++++++++++++++-
4 files changed, 190 insertions(+), 7 deletions(-)
diff --git a/src/include/storage/aio.h b/src/include/storage/aio.h
index f91f0afc5a5..72d5680e767 100644
--- a/src/include/storage/aio.h
+++ b/src/include/storage/aio.h
@@ -197,11 +197,13 @@ typedef enum PgAioHandleCallbackID
PGAIO_HCB_MD_WRITEV,
PGAIO_HCB_SHARED_BUFFER_READV,
+ PGAIO_HCB_SHARED_BUFFER_WRITEV,
PGAIO_HCB_LOCAL_BUFFER_READV,
+ PGAIO_HCB_LOCAL_BUFFER_WRITEV,
} PgAioHandleCallbackID;
-#define PGAIO_HCB_MAX PGAIO_HCB_LOCAL_BUFFER_READV
+#define PGAIO_HCB_MAX PGAIO_HCB_LOCAL_BUFFER_WRITEV
StaticAssertDecl(PGAIO_HCB_MAX <= (1 << PGAIO_RESULT_ID_BITS),
"PGAIO_HCB_MAX is too big for PGAIO_RESULT_ID_BITS");
diff --git a/src/include/storage/bufmgr.h b/src/include/storage/bufmgr.h
index f2192ceb271..492feab0cb5 100644
--- a/src/include/storage/bufmgr.h
+++ b/src/include/storage/bufmgr.h
@@ -174,7 +174,9 @@ extern PGDLLIMPORT int backend_flush_after;
extern PGDLLIMPORT int bgwriter_flush_after;
extern const PgAioHandleCallbacks aio_shared_buffer_readv_cb;
+extern const PgAioHandleCallbacks aio_shared_buffer_writev_cb;
extern const PgAioHandleCallbacks aio_local_buffer_readv_cb;
+extern const PgAioHandleCallbacks aio_local_buffer_writev_cb;
/* in buf_init.c */
extern PGDLLIMPORT char *BufferBlocks;
diff --git a/src/backend/storage/aio/aio_callback.c b/src/backend/storage/aio/aio_callback.c
index abe5b191b42..21665e7eccb 100644
--- a/src/backend/storage/aio/aio_callback.c
+++ b/src/backend/storage/aio/aio_callback.c
@@ -44,8 +44,10 @@ static const PgAioHandleCallbacksEntry aio_handle_cbs[] = {
CALLBACK_ENTRY(PGAIO_HCB_MD_WRITEV, aio_md_writev_cb),
CALLBACK_ENTRY(PGAIO_HCB_SHARED_BUFFER_READV, aio_shared_buffer_readv_cb),
+ CALLBACK_ENTRY(PGAIO_HCB_SHARED_BUFFER_WRITEV, aio_shared_buffer_writev_cb),
CALLBACK_ENTRY(PGAIO_HCB_LOCAL_BUFFER_READV, aio_local_buffer_readv_cb),
+ CALLBACK_ENTRY(PGAIO_HCB_LOCAL_BUFFER_WRITEV, aio_local_buffer_writev_cb),
#undef CALLBACK_ENTRY
};
diff --git a/src/backend/storage/buffer/bufmgr.c b/src/backend/storage/buffer/bufmgr.c
index 1c37d7dfe2f..9bf30c44af0 100644
--- a/src/backend/storage/buffer/bufmgr.c
+++ b/src/backend/storage/buffer/bufmgr.c
@@ -5536,7 +5536,15 @@ LockBuffer(Buffer buffer, int mode)
else if (mode == BUFFER_LOCK_SHARE)
LWLockAcquire(BufferDescriptorGetContentLock(buf), LW_SHARED);
else if (mode == BUFFER_LOCK_EXCLUSIVE)
+ {
+ /*
+ * FIXME: Wait for AIO writes, otherwise there would be a risk of
+ * deadlock. This isn't entirely trivial to do in a race-free way, IO
+ * could be started between us checking whether there is IO and
+ * enqueueing ourselves for the lock.
+ */
LWLockAcquire(BufferDescriptorGetContentLock(buf), LW_EXCLUSIVE);
+ }
else
elog(ERROR, "unrecognized buffer lock mode: %d", mode);
}
@@ -5551,6 +5559,19 @@ ConditionalLockBuffer(Buffer buffer)
{
BufferDesc *buf;
+ /*
+ * FIXME: Wait for AIO writes. Some code does not deal well
+ * ConditionalLockBuffer() continuously failing, e.g.
+ * spginsert()->spgdoinsert() ends up busy-looping (easily reproducible by
+ * just making this function always fail and running the regression
+ * tests). While that code could be fixed, it'd be hard to find all
+ * problematic places.
+ *
+ * It would be OK to wait for the IO as waiting for IO completion does not
+ * need to wait for any locks that could lead to an undetected deadlock or
+ * such.
+ */
+
Assert(BufferIsPinned(buffer));
if (BufferIsLocal(buffer))
return true; /* act as though we got it */
@@ -5614,10 +5635,8 @@ LockBufferForCleanup(Buffer buffer)
CheckBufferIsPinnedOnce(buffer);
/*
- * We do not yet need to be worried about in-progress AIOs holding a pin,
- * as we, so far, only support doing reads via AIO and this function can
- * only be called once the buffer is valid (i.e. no read can be in
- * flight).
+ * FIXME: See AIO related comments in LockBuffer() and
+ * ConditionalLockBuffer()
*/
/* Nobody else to wait for */
@@ -5630,6 +5649,11 @@ LockBufferForCleanup(Buffer buffer)
{
uint32 buf_state;
+ /*
+ * FIXME: LockBuffer()'s handling of in-progress writes (once
+ * implemented) should suffice to deal with deadlock risk.
+ */
+
/* Try to acquire lock */
LockBuffer(buffer, BUFFER_LOCK_EXCLUSIVE);
buf_state = LockBufHdr(bufHdr);
@@ -5777,7 +5801,13 @@ ConditionalLockBufferForCleanup(Buffer buffer)
Assert(BufferIsValid(buffer));
- /* see AIO related comment in LockBufferForCleanup() */
+ /*
+ * FIXME: Should wait for IO for the same reason as in
+ * ConditionalLockBuffer(). Needs to happen before the
+ * ConditionalLockBuffer() call below, as we'd never reach the
+ * ConditionalLockBuffer() call due the buffer pin held for the duration
+ * of the IO.
+ */
if (BufferIsLocal(buffer))
{
@@ -5834,7 +5864,10 @@ IsBufferCleanupOK(Buffer buffer)
Assert(BufferIsValid(buffer));
- /* see AIO related comment in LockBufferForCleanup() */
+ /*
+ * FIXME: See AIO related comments in LockBuffer() and
+ * ConditionalLockBuffer()
+ */
if (BufferIsLocal(buffer))
{
@@ -7140,12 +7173,129 @@ buffer_readv_report(PgAioResult result, const PgAioTargetData *td,
affected_count > 1 ? errhint_internal(hint_mult, affected_count - 1) : 0);
}
+/*
+ * Helper for AIO writev completion callbacks, supporting both shared and temp
+ * buffers. Gets called once for each buffer in a multi-page write.
+ */
+static pg_attribute_always_inline PgAioResult
+buffer_writev_complete_one(uint8 buf_off, Buffer buffer, uint8 flags,
+ bool failed, bool is_temp)
+{
+ BufferDesc *buf_hdr = is_temp ?
+ GetLocalBufferDescriptor(-buffer - 1)
+ : GetBufferDescriptor(buffer - 1);
+ PgAioResult result = {.status = PGAIO_RS_OK};
+ bool clear_dirty;
+ uint32 set_flag_bits;
+
+#ifdef USE_ASSERT_CHECKING
+ {
+ uint32 buf_state = pg_atomic_read_u32(&buf_hdr->state);
+
+ Assert(buf_state & BM_VALID);
+ Assert(buf_state & BM_TAG_VALID);
+ /* temp buffers don't use BM_IO_IN_PROGRESS */
+ if (!is_temp)
+ Assert(buf_state & BM_IO_IN_PROGRESS);
+ Assert(buf_state & BM_DIRTY);
+ }
+#endif
+
+ clear_dirty = failed ? false : true;
+ set_flag_bits = failed ? BM_IO_ERROR : 0;
+
+ if (is_temp)
+ TerminateLocalBufferIO(buf_hdr, clear_dirty, set_flag_bits, true);
+ else
+ TerminateBufferIO(buf_hdr, clear_dirty, set_flag_bits, false, true);
+
+ /*
+ * The initiator of IO is not managing the lock (i.e. we called
+ * LWLockDisown()), we are.
+ */
+ if (!is_temp)
+ LWLockReleaseDisowned(BufferDescriptorGetContentLock(buf_hdr),
+ LW_SHARED);
+
+ /* FIXME: tracepoint */
+
+ return result;
+}
+
+/*
+ * Perform completion handling of a single AIO write. This write may cover
+ * multiple blocks / buffers.
+ *
+ * Shared between shared and local buffers, to reduce code duplication.
+ */
+static pg_attribute_always_inline PgAioResult
+buffer_writev_complete(PgAioHandle *ioh, PgAioResult prior_result,
+ uint8 cb_data, bool is_temp)
+{
+ PgAioResult result = prior_result;
+ PgAioTargetData *td = pgaio_io_get_target_data(ioh);
+ uint64 *io_data;
+ uint8 handle_data_len;
+
+ if (is_temp)
+ {
+ Assert(td->smgr.is_temp);
+ Assert(pgaio_io_get_owner(ioh) == MyProcNumber);
+ }
+ else
+ Assert(!td->smgr.is_temp);
+
+ /*
+ * Iterate over all the buffers affected by this IO and call appropriate
+ * per-buffer completion function for each buffer.
+ */
+ io_data = pgaio_io_get_handle_data(ioh, &handle_data_len);
+ for (uint8 buf_off = 0; buf_off < handle_data_len; buf_off++)
+ {
+ Buffer buf = io_data[buf_off];
+ PgAioResult buf_result;
+ bool failed;
+
+ Assert(BufferIsValid(buf));
+
+ /*
+ * If the entire failed on a lower-level, each buffer needs to be
+ * marked as failed. In case of a partial read, some buffers may be
+ * ok.
+ */
+ failed =
+ prior_result.status == PGAIO_RS_ERROR
+ || prior_result.result <= buf_off;
+
+ buf_result = buffer_writev_complete_one(buf_off, buf, cb_data, failed,
+ is_temp);
+
+ /*
+ * If there wasn't any prior error and the IO for this page failed in
+ * some form, set the whole IO's to the page's result.
+ */
+ if (result.status != PGAIO_RS_ERROR && buf_result.status != PGAIO_RS_OK)
+ {
+ result = buf_result;
+ pgaio_result_report(result, td, LOG);
+ }
+ }
+
+ return result;
+}
+
static void
shared_buffer_readv_stage(PgAioHandle *ioh, uint8 cb_data)
{
buffer_stage_common(ioh, false, false);
}
+static void
+shared_buffer_writev_stage(PgAioHandle *ioh, uint8 cb_data)
+{
+ buffer_stage_common(ioh, true, false);
+}
+
static PgAioResult
shared_buffer_readv_complete(PgAioHandle *ioh, PgAioResult prior_result,
uint8 cb_data)
@@ -7191,6 +7341,13 @@ shared_buffer_readv_complete_local(PgAioHandle *ioh, PgAioResult prior_result,
return prior_result;
}
+static PgAioResult
+shared_buffer_writev_complete(PgAioHandle *ioh, PgAioResult prior_result,
+ uint8 cb_data)
+{
+ return buffer_writev_complete(ioh, prior_result, cb_data, false);
+}
+
static void
local_buffer_readv_stage(PgAioHandle *ioh, uint8 cb_data)
{
@@ -7204,6 +7361,17 @@ local_buffer_readv_complete(PgAioHandle *ioh, PgAioResult prior_result,
return buffer_readv_complete(ioh, prior_result, cb_data, true);
}
+static void
+local_buffer_writev_stage(PgAioHandle *ioh, uint8 cb_data)
+{
+ /*
+ * Currently this is unreachable as the only write support is for
+ * checkpointer / bgwriter, which don't deal with local buffers.
+ */
+ elog(ERROR, "should be unreachable");
+}
+
+
/* readv callback is passed READ_BUFFERS_* flags as callback data */
const PgAioHandleCallbacks aio_shared_buffer_readv_cb = {
.stage = shared_buffer_readv_stage,
@@ -7213,6 +7381,11 @@ const PgAioHandleCallbacks aio_shared_buffer_readv_cb = {
.report = buffer_readv_report,
};
+const PgAioHandleCallbacks aio_shared_buffer_writev_cb = {
+ .stage = shared_buffer_writev_stage,
+ .complete_shared = shared_buffer_writev_complete,
+};
+
/* readv callback is passed READ_BUFFERS_* flags as callback data */
const PgAioHandleCallbacks aio_local_buffer_readv_cb = {
.stage = local_buffer_readv_stage,
@@ -7226,3 +7399,7 @@ const PgAioHandleCallbacks aio_local_buffer_readv_cb = {
.complete_local = local_buffer_readv_complete,
.report = buffer_readv_report,
};
+
+const PgAioHandleCallbacks aio_local_buffer_writev_cb = {
+ .stage = local_buffer_writev_stage,
+};
--
2.48.1.76.g4e746b1a31.dirty
[text/x-diff] v2.15-0012-aio-Add-IO-queue-helper.patch (7.4K, ../../7s6fclfekpcxoaaorwrq67v4vmgixf2dcjcgznuj7vxs3ie3wq@okqbi5vabqbx/13-v2.15-0012-aio-Add-IO-queue-helper.patch)
download | inline diff:
From a7aad0b753c5f3afdbdb90a0ca77765c051dbfce Mon Sep 17 00:00:00 2001
From: Andres Freund <andres@anarazel.de>
Date: Tue, 18 Mar 2025 14:40:06 -0400
Subject: [PATCH v2.15 12/18] aio: Add IO queue helper
This is likely never going to anywhere - Thomas Munro is working on something
more complete. But I needed a way to exercise aio for checkpointer / bgwriter.
---
src/include/storage/io_queue.h | 33 +++++
src/backend/storage/aio/Makefile | 1 +
src/backend/storage/aio/io_queue.c | 204 ++++++++++++++++++++++++++++
src/backend/storage/aio/meson.build | 1 +
src/tools/pgindent/typedefs.list | 2 +
5 files changed, 241 insertions(+)
create mode 100644 src/include/storage/io_queue.h
create mode 100644 src/backend/storage/aio/io_queue.c
diff --git a/src/include/storage/io_queue.h b/src/include/storage/io_queue.h
new file mode 100644
index 00000000000..92b1e9afe6f
--- /dev/null
+++ b/src/include/storage/io_queue.h
@@ -0,0 +1,33 @@
+/*-------------------------------------------------------------------------
+ *
+ * io_queue.h
+ * Mechanism for tracking many IOs
+ *
+ *
+ * Portions Copyright (c) 1996-2025, PostgreSQL Global Development Group
+ * Portions Copyright (c) 1994, Regents of the University of California
+ *
+ * src/include/storage/io_queue.h
+ *
+ *-------------------------------------------------------------------------
+ */
+#ifndef IO_QUEUE_H
+#define IO_QUEUE_H
+
+#include "storage/aio_types.h"
+
+struct IOQueue;
+typedef struct IOQueue IOQueue;
+
+struct PgAioWaitRef;
+
+extern IOQueue *io_queue_create(int depth, int flags);
+extern void io_queue_track(IOQueue *ioq, const PgAioWaitRef *iow);
+extern void io_queue_wait_one(IOQueue *ioq);
+extern void io_queue_wait_all(IOQueue *ioq);
+extern bool io_queue_is_empty(IOQueue *ioq);
+extern void io_queue_reserve(IOQueue *ioq);
+extern PgAioHandle *io_queue_acquire_io(IOQueue *ioq);
+extern void io_queue_free(IOQueue *ioq);
+
+#endif /* IO_QUEUE_H */
diff --git a/src/backend/storage/aio/Makefile b/src/backend/storage/aio/Makefile
index 3f2469cc399..86fa4276fda 100644
--- a/src/backend/storage/aio/Makefile
+++ b/src/backend/storage/aio/Makefile
@@ -15,6 +15,7 @@ OBJS = \
aio_init.o \
aio_io.o \
aio_target.o \
+ io_queue.o \
method_io_uring.o \
method_sync.o \
method_worker.o \
diff --git a/src/backend/storage/aio/io_queue.c b/src/backend/storage/aio/io_queue.c
new file mode 100644
index 00000000000..526aa1d5e06
--- /dev/null
+++ b/src/backend/storage/aio/io_queue.c
@@ -0,0 +1,204 @@
+/*-------------------------------------------------------------------------
+ *
+ * io_queue.c
+ * AIO - Mechanism for tracking many IOs
+ *
+ * Portions Copyright (c) 1996-2025, PostgreSQL Global Development Group
+ * Portions Copyright (c) 1994, Regents of the University of California
+ *
+ * IDENTIFICATION
+ * src/backend/storage/aio/io_queue.c
+ *
+ *-------------------------------------------------------------------------
+ */
+
+#include "postgres.h"
+
+#include "lib/ilist.h"
+#include "storage/aio.h"
+#include "storage/io_queue.h"
+#include "utils/resowner.h"
+
+
+
+typedef struct TrackedIO
+{
+ PgAioWaitRef iow;
+ dlist_node node;
+} TrackedIO;
+
+struct IOQueue
+{
+ int depth;
+ int unsubmitted;
+
+ bool has_reserved;
+
+ dclist_head idle;
+ dclist_head in_progress;
+
+ TrackedIO tracked_ios[FLEXIBLE_ARRAY_MEMBER];
+};
+
+
+IOQueue *
+io_queue_create(int depth, int flags)
+{
+ size_t sz;
+ IOQueue *ioq;
+
+ sz = offsetof(IOQueue, tracked_ios)
+ + sizeof(TrackedIO) * depth;
+
+ ioq = palloc0(sz);
+
+ ioq->depth = 0;
+
+ for (int i = 0; i < depth; i++)
+ {
+ TrackedIO *tio = &ioq->tracked_ios[i];
+
+ pgaio_wref_clear(&tio->iow);
+ dclist_push_tail(&ioq->idle, &tio->node);
+ }
+
+ return ioq;
+}
+
+void
+io_queue_wait_one(IOQueue *ioq)
+{
+ /* submit all pending IO before waiting */
+ pgaio_submit_staged();
+
+ while (!dclist_is_empty(&ioq->in_progress))
+ {
+ /* FIXME: Should we really pop here already? */
+ dlist_node *node = dclist_pop_head_node(&ioq->in_progress);
+ TrackedIO *tio = dclist_container(TrackedIO, node, node);
+
+ pgaio_wref_wait(&tio->iow);
+ dclist_push_head(&ioq->idle, &tio->node);
+ }
+}
+
+void
+io_queue_reserve(IOQueue *ioq)
+{
+ if (ioq->has_reserved)
+ return;
+
+ if (dclist_is_empty(&ioq->idle))
+ io_queue_wait_one(ioq);
+
+ Assert(!dclist_is_empty(&ioq->idle));
+
+ ioq->has_reserved = true;
+}
+
+PgAioHandle *
+io_queue_acquire_io(IOQueue *ioq)
+{
+ PgAioHandle *ioh;
+
+ io_queue_reserve(ioq);
+
+ Assert(!dclist_is_empty(&ioq->idle));
+
+ if (!io_queue_is_empty(ioq))
+ {
+ ioh = pgaio_io_acquire_nb(CurrentResourceOwner, NULL);
+ if (ioh == NULL)
+ {
+ /*
+ * Need to wait for all IOs, blocking might not be legal in the
+ * context.
+ *
+ * XXX: This doesn't make a whole lot of sense, we're also
+ * blocking here. What was I smoking when I wrote the above?
+ */
+ io_queue_wait_all(ioq);
+ ioh = pgaio_io_acquire(CurrentResourceOwner, NULL);
+ }
+ }
+ else
+ {
+ ioh = pgaio_io_acquire(CurrentResourceOwner, NULL);
+ }
+
+ return ioh;
+}
+
+void
+io_queue_track(IOQueue *ioq, const struct PgAioWaitRef *iow)
+{
+ dlist_node *node;
+ TrackedIO *tio;
+
+ Assert(ioq->has_reserved);
+ ioq->has_reserved = false;
+
+ Assert(!dclist_is_empty(&ioq->idle));
+
+ node = dclist_pop_head_node(&ioq->idle);
+ tio = dclist_container(TrackedIO, node, node);
+
+ tio->iow = *iow;
+
+ dclist_push_tail(&ioq->in_progress, &tio->node);
+
+ ioq->unsubmitted++;
+
+ /*
+ * XXX: Should have some smarter logic here. We don't want to wait too
+ * long to submit, that'll mean we're more likely to block. But we also
+ * don't want to have the overhead of submitting every IO individually.
+ */
+ if (ioq->unsubmitted >= 4)
+ {
+ pgaio_submit_staged();
+ ioq->unsubmitted = 0;
+ }
+}
+
+void
+io_queue_wait_all(IOQueue *ioq)
+{
+ /* submit all pending IO before waiting */
+ pgaio_submit_staged();
+
+ while (!dclist_is_empty(&ioq->in_progress))
+ {
+ /* wait for the last IO to minimize unnecessary wakeups */
+ dlist_node *node = dclist_tail_node(&ioq->in_progress);
+ TrackedIO *tio = dclist_container(TrackedIO, node, node);
+
+ if (!pgaio_wref_check_done(&tio->iow))
+ {
+ ereport(DEBUG3,
+ errmsg("io_queue_wait_all for io:%d",
+ pgaio_wref_get_id(&tio->iow)),
+ errhidestmt(true),
+ errhidecontext(true));
+
+ pgaio_wref_wait(&tio->iow);
+ }
+
+ dclist_delete_from(&ioq->in_progress, &tio->node);
+ dclist_push_head(&ioq->idle, &tio->node);
+ }
+}
+
+bool
+io_queue_is_empty(IOQueue *ioq)
+{
+ return dclist_is_empty(&ioq->in_progress);
+}
+
+void
+io_queue_free(IOQueue *ioq)
+{
+ io_queue_wait_all(ioq);
+
+ pfree(ioq);
+}
diff --git a/src/backend/storage/aio/meson.build b/src/backend/storage/aio/meson.build
index da6df2d3654..270c4a64428 100644
--- a/src/backend/storage/aio/meson.build
+++ b/src/backend/storage/aio/meson.build
@@ -7,6 +7,7 @@ backend_sources += files(
'aio_init.c',
'aio_io.c',
'aio_target.c',
+ 'io_queue.c',
'method_io_uring.c',
'method_sync.c',
'method_worker.c',
diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list
index 3a67ee01b46..0c6ddadc51d 100644
--- a/src/tools/pgindent/typedefs.list
+++ b/src/tools/pgindent/typedefs.list
@@ -1196,6 +1196,7 @@ IOContext
IOFuncSelector
IOObject
IOOp
+IOQueue
IO_STATUS_BLOCK
IPCompareMethod
ITEM
@@ -3022,6 +3023,7 @@ TocEntry
TokenAuxData
TokenizedAuthLine
TrackItem
+TrackedIO
TransApplyAction
TransInvalidationInfo
TransState
--
2.48.1.76.g4e746b1a31.dirty
[text/x-diff] v2.15-0013-bufmgr-use-AIO-in-checkpointer-bgwriter.patch (31.6K, ../../7s6fclfekpcxoaaorwrq67v4vmgixf2dcjcgznuj7vxs3ie3wq@okqbi5vabqbx/14-v2.15-0013-bufmgr-use-AIO-in-checkpointer-bgwriter.patch)
download | inline diff:
From 262e24801bfa5b4390b550d92ba433cc7ec48747 Mon Sep 17 00:00:00 2001
From: Andres Freund <andres@anarazel.de>
Date: Tue, 18 Mar 2025 14:40:06 -0400
Subject: [PATCH v2.15 13/18] bufmgr: use AIO in checkpointer, bgwriter
This is far from ready - just included to be able to exercise AIO writes and
get some preliminary numbers. In all likelihood this will instead be based
on-top of work by Thomas Munro instead of the preceding commit.
TODO;
- This doesn't implement bgwriter_flush_after, checkpointer_flush_after
I think that's not too hard to do, it's mainly round tuits.
- The queuing logic doesn't carefully respect pin limits
That might be ok for checkpointer and bgwriter, but the infrastructure
should be usable outside of this as well.
---
src/include/postmaster/bgwriter.h | 3 +-
src/include/storage/buf_internals.h | 2 +
src/include/storage/bufmgr.h | 3 +-
src/include/storage/bufpage.h | 1 +
src/backend/postmaster/bgwriter.c | 19 +-
src/backend/postmaster/checkpointer.c | 11 +-
src/backend/storage/buffer/bufmgr.c | 594 +++++++++++++++++++++++---
src/backend/storage/page/bufpage.c | 10 +
src/tools/pgindent/typedefs.list | 1 +
9 files changed, 586 insertions(+), 58 deletions(-)
diff --git a/src/include/postmaster/bgwriter.h b/src/include/postmaster/bgwriter.h
index 800ecbfd13b..a8081d411b6 100644
--- a/src/include/postmaster/bgwriter.h
+++ b/src/include/postmaster/bgwriter.h
@@ -31,7 +31,8 @@ pg_noreturn extern void BackgroundWriterMain(const void *startup_data, size_t st
pg_noreturn extern void CheckpointerMain(const void *startup_data, size_t startup_data_len);
extern void RequestCheckpoint(int flags);
-extern void CheckpointWriteDelay(int flags, double progress);
+struct IOQueue;
+extern void CheckpointWriteDelay(struct IOQueue *ioq, int flags, double progress);
extern bool ForwardSyncRequest(const FileTag *ftag, SyncRequestType type);
diff --git a/src/include/storage/buf_internals.h b/src/include/storage/buf_internals.h
index 0dec7d93b3b..45c2b70b736 100644
--- a/src/include/storage/buf_internals.h
+++ b/src/include/storage/buf_internals.h
@@ -21,6 +21,8 @@
#include "storage/buf.h"
#include "storage/bufmgr.h"
#include "storage/condition_variable.h"
+#include "storage/io_queue.h"
+#include "storage/latch.h"
#include "storage/lwlock.h"
#include "storage/procnumber.h"
#include "storage/shmem.h"
diff --git a/src/include/storage/bufmgr.h b/src/include/storage/bufmgr.h
index 492feab0cb5..89e0ca11288 100644
--- a/src/include/storage/bufmgr.h
+++ b/src/include/storage/bufmgr.h
@@ -297,7 +297,8 @@ extern bool ConditionalLockBufferForCleanup(Buffer buffer);
extern bool IsBufferCleanupOK(Buffer buffer);
extern bool HoldingBufferPinThatDelaysRecovery(void);
-extern bool BgBufferSync(struct WritebackContext *wb_context);
+struct IOQueue;
+extern bool BgBufferSync(struct IOQueue *ioq, struct WritebackContext *wb_context);
extern uint32 GetPinLimit(void);
extern uint32 GetLocalPinLimit(void);
diff --git a/src/include/storage/bufpage.h b/src/include/storage/bufpage.h
index aeb67c498c5..000a3ab23f9 100644
--- a/src/include/storage/bufpage.h
+++ b/src/include/storage/bufpage.h
@@ -507,5 +507,6 @@ extern bool PageIndexTupleOverwrite(Page page, OffsetNumber offnum,
Item newtup, Size newsize);
extern char *PageSetChecksumCopy(Page page, BlockNumber blkno);
extern void PageSetChecksumInplace(Page page, BlockNumber blkno);
+extern bool PageNeedsChecksumCopy(Page page);
#endif /* BUFPAGE_H */
diff --git a/src/backend/postmaster/bgwriter.c b/src/backend/postmaster/bgwriter.c
index 72f5acceec7..6e8801a39e3 100644
--- a/src/backend/postmaster/bgwriter.c
+++ b/src/backend/postmaster/bgwriter.c
@@ -38,11 +38,13 @@
#include "postmaster/auxprocess.h"
#include "postmaster/bgwriter.h"
#include "postmaster/interrupt.h"
+#include "storage/aio.h"
#include "storage/aio_subsys.h"
#include "storage/buf_internals.h"
#include "storage/bufmgr.h"
#include "storage/condition_variable.h"
#include "storage/fd.h"
+#include "storage/io_queue.h"
#include "storage/lwlock.h"
#include "storage/proc.h"
#include "storage/procsignal.h"
@@ -90,6 +92,7 @@ BackgroundWriterMain(const void *startup_data, size_t startup_data_len)
sigjmp_buf local_sigjmp_buf;
MemoryContext bgwriter_context;
bool prev_hibernate;
+ IOQueue *ioq;
WritebackContext wb_context;
Assert(startup_data_len == 0);
@@ -131,6 +134,7 @@ BackgroundWriterMain(const void *startup_data, size_t startup_data_len)
ALLOCSET_DEFAULT_SIZES);
MemoryContextSwitchTo(bgwriter_context);
+ ioq = io_queue_create(128, 0);
WritebackContextInit(&wb_context, &bgwriter_flush_after);
/*
@@ -228,12 +232,22 @@ BackgroundWriterMain(const void *startup_data, size_t startup_data_len)
/* Clear any already-pending wakeups */
ResetLatch(MyLatch);
+ /*
+ * FIXME: this is theoretically racy, but I didn't want to copy
+ * ProcessMainLoopInterrupts() remaining body here.
+ */
+ if (ShutdownRequestPending)
+ {
+ io_queue_wait_all(ioq);
+ io_queue_free(ioq);
+ }
+
ProcessMainLoopInterrupts();
/*
* Do one cycle of dirty-buffer writing.
*/
- can_hibernate = BgBufferSync(&wb_context);
+ can_hibernate = BgBufferSync(ioq, &wb_context);
/* Report pending statistics to the cumulative stats system */
pgstat_report_bgwriter();
@@ -250,6 +264,9 @@ BackgroundWriterMain(const void *startup_data, size_t startup_data_len)
smgrdestroyall();
}
+ /* finish IO before sleeping, to avoid blocking other backends */
+ io_queue_wait_all(ioq);
+
/*
* Log a new xl_running_xacts every now and then so replication can
* get into a consistent state faster (think of suboverflowed
diff --git a/src/backend/postmaster/checkpointer.c b/src/backend/postmaster/checkpointer.c
index fda91ffd1ce..904fe167eb4 100644
--- a/src/backend/postmaster/checkpointer.c
+++ b/src/backend/postmaster/checkpointer.c
@@ -49,10 +49,12 @@
#include "postmaster/bgwriter.h"
#include "postmaster/interrupt.h"
#include "replication/syncrep.h"
+#include "storage/aio.h"
#include "storage/aio_subsys.h"
#include "storage/bufmgr.h"
#include "storage/condition_variable.h"
#include "storage/fd.h"
+#include "storage/io_queue.h"
#include "storage/ipc.h"
#include "storage/lwlock.h"
#include "storage/pmsignal.h"
@@ -766,7 +768,7 @@ ImmediateCheckpointRequested(void)
* fraction between 0.0 meaning none, and 1.0 meaning all done.
*/
void
-CheckpointWriteDelay(int flags, double progress)
+CheckpointWriteDelay(IOQueue *ioq, int flags, double progress)
{
static int absorb_counter = WRITES_PER_ABSORB;
@@ -800,6 +802,13 @@ CheckpointWriteDelay(int flags, double progress)
/* Report interim statistics to the cumulative stats system */
pgstat_report_checkpointer();
+ /*
+ * Ensure all pending IO is submitted to avoid unnecessary delays for
+ * other processes.
+ */
+ io_queue_wait_all(ioq);
+
+
/*
* This sleep used to be connected to bgwriter_delay, typically 200ms.
* That resulted in more frequent wakeups if not much work to do.
diff --git a/src/backend/storage/buffer/bufmgr.c b/src/backend/storage/buffer/bufmgr.c
index 9bf30c44af0..d9a362d7553 100644
--- a/src/backend/storage/buffer/bufmgr.c
+++ b/src/backend/storage/buffer/bufmgr.c
@@ -52,6 +52,7 @@
#include "storage/buf_internals.h"
#include "storage/bufmgr.h"
#include "storage/fd.h"
+#include "storage/io_queue.h"
#include "storage/ipc.h"
#include "storage/lmgr.h"
#include "storage/proc.h"
@@ -76,6 +77,7 @@
/* Bits in SyncOneBuffer's return value */
#define BUF_WRITTEN 0x01
#define BUF_REUSABLE 0x02
+#define BUF_CANT_MERGE 0x04
#define RELS_BSEARCH_THRESHOLD 20
@@ -515,8 +517,6 @@ static void UnpinBuffer(BufferDesc *buf);
static void UnpinBufferNoOwner(BufferDesc *buf);
static void BufferSync(int flags);
static uint32 WaitBufHdrUnlocked(BufferDesc *buf);
-static int SyncOneBuffer(int buf_id, bool skip_recently_used,
- WritebackContext *wb_context);
static void WaitIO(BufferDesc *buf);
static void AbortBufferIO(Buffer buffer);
static void shared_buffer_write_error_callback(void *arg);
@@ -532,6 +532,7 @@ static void CheckReadBuffersOperation(ReadBuffersOperation *operation, bool is_c
static Buffer GetVictimBuffer(BufferAccessStrategy strategy, IOContext io_context);
static void FlushBuffer(BufferDesc *buf, SMgrRelation reln,
IOObject io_object, IOContext io_context);
+
static void FindAndDropRelationBuffers(RelFileLocator rlocator,
ForkNumber forkNum,
BlockNumber nForkBlock,
@@ -3321,6 +3322,57 @@ UnpinBufferNoOwner(BufferDesc *buf)
}
}
+typedef struct BuffersToWrite
+{
+ int nbuffers;
+ BufferTag start_at_tag;
+ uint32 max_combine;
+
+ XLogRecPtr max_lsn;
+
+ PgAioHandle *ioh;
+ PgAioWaitRef iow;
+
+ uint64 total_writes;
+
+ Buffer buffers[IOV_MAX];
+ PgAioBounceBuffer *bounce_buffers[IOV_MAX];
+ const void *data_ptrs[IOV_MAX];
+} BuffersToWrite;
+
+static int PrepareToWriteBuffer(BuffersToWrite *to_write, Buffer buf,
+ bool skip_recently_used,
+ IOQueue *ioq, WritebackContext *wb_context);
+
+static void WriteBuffers(BuffersToWrite *to_write,
+ IOQueue *ioq, WritebackContext *wb_context);
+
+static void
+BuffersToWriteInit(BuffersToWrite *to_write,
+ IOQueue *ioq, WritebackContext *wb_context)
+{
+ to_write->total_writes = 0;
+ to_write->nbuffers = 0;
+ to_write->ioh = NULL;
+ pgaio_wref_clear(&to_write->iow);
+ to_write->max_lsn = InvalidXLogRecPtr;
+
+ pgaio_enter_batchmode();
+}
+
+static void
+BuffersToWriteEnd(BuffersToWrite *to_write)
+{
+ if (to_write->ioh != NULL)
+ {
+ pgaio_io_release(to_write->ioh);
+ to_write->ioh = NULL;
+ }
+
+ pgaio_exit_batchmode();
+}
+
+
#define ST_SORT sort_checkpoint_bufferids
#define ST_ELEMENT_TYPE CkptSortItem
#define ST_COMPARE(a, b) ckpt_buforder_comparator(a, b)
@@ -3352,7 +3404,10 @@ BufferSync(int flags)
binaryheap *ts_heap;
int i;
int mask = BM_DIRTY;
+ IOQueue *ioq;
WritebackContext wb_context;
+ BuffersToWrite to_write;
+ int max_combine;
/*
* Unless this is a shutdown checkpoint or we have been explicitly told,
@@ -3414,7 +3469,9 @@ BufferSync(int flags)
if (num_to_scan == 0)
return; /* nothing to do */
+ ioq = io_queue_create(512, 0);
WritebackContextInit(&wb_context, &checkpoint_flush_after);
+ max_combine = Min(io_bounce_buffers, io_combine_limit);
TRACE_POSTGRESQL_BUFFER_SYNC_START(NBuffers, num_to_scan);
@@ -3522,48 +3579,91 @@ BufferSync(int flags)
*/
num_processed = 0;
num_written = 0;
+
+ BuffersToWriteInit(&to_write, ioq, &wb_context);
+
while (!binaryheap_empty(ts_heap))
{
BufferDesc *bufHdr = NULL;
CkptTsStatus *ts_stat = (CkptTsStatus *)
DatumGetPointer(binaryheap_first(ts_heap));
+ bool batch_continue = true;
- buf_id = CkptBufferIds[ts_stat->index].buf_id;
- Assert(buf_id != -1);
-
- bufHdr = GetBufferDescriptor(buf_id);
-
- num_processed++;
+ Assert(ts_stat->num_scanned <= ts_stat->num_to_scan);
/*
- * We don't need to acquire the lock here, because we're only looking
- * at a single bit. It's possible that someone else writes the buffer
- * and clears the flag right after we check, but that doesn't matter
- * since SyncOneBuffer will then do nothing. However, there is a
- * further race condition: it's conceivable that between the time we
- * examine the bit here and the time SyncOneBuffer acquires the lock,
- * someone else not only wrote the buffer but replaced it with another
- * page and dirtied it. In that improbable case, SyncOneBuffer will
- * write the buffer though we didn't need to. It doesn't seem worth
- * guarding against this, though.
+ * Collect a batch of buffers to write out from the current
+ * tablespace. That causes some imbalance between the tablespaces, but
+ * that's more than outweighed by the efficiency gain due to batching.
*/
- if (pg_atomic_read_u32(&bufHdr->state) & BM_CHECKPOINT_NEEDED)
+ while (batch_continue &&
+ to_write.nbuffers < max_combine &&
+ ts_stat->num_scanned < ts_stat->num_to_scan)
{
- if (SyncOneBuffer(buf_id, false, &wb_context) & BUF_WRITTEN)
+ buf_id = CkptBufferIds[ts_stat->index].buf_id;
+ Assert(buf_id != -1);
+
+ bufHdr = GetBufferDescriptor(buf_id);
+
+ num_processed++;
+
+ /*
+ * We don't need to acquire the lock here, because we're only
+ * looking at a single bit. It's possible that someone else writes
+ * the buffer and clears the flag right after we check, but that
+ * doesn't matter since SyncOneBuffer will then do nothing.
+ * However, there is a further race condition: it's conceivable
+ * that between the time we examine the bit here and the time
+ * SyncOneBuffer acquires the lock, someone else not only wrote
+ * the buffer but replaced it with another page and dirtied it. In
+ * that improbable case, SyncOneBuffer will write the buffer
+ * though we didn't need to. It doesn't seem worth guarding
+ * against this, though.
+ */
+ if (pg_atomic_read_u32(&bufHdr->state) & BM_CHECKPOINT_NEEDED)
{
- TRACE_POSTGRESQL_BUFFER_SYNC_WRITTEN(buf_id);
- PendingCheckpointerStats.buffers_written++;
- num_written++;
+ int result = PrepareToWriteBuffer(&to_write, buf_id + 1, false,
+ ioq, &wb_context);
+
+ if (result & BUF_CANT_MERGE)
+ {
+ Assert(to_write.nbuffers > 0);
+ WriteBuffers(&to_write, ioq, &wb_context);
+
+ result = PrepareToWriteBuffer(&to_write, buf_id + 1, false,
+ ioq, &wb_context);
+ Assert(result != BUF_CANT_MERGE);
+ }
+
+ if (result & BUF_WRITTEN)
+ {
+ TRACE_POSTGRESQL_BUFFER_SYNC_WRITTEN(buf_id);
+ PendingCheckpointerStats.buffers_written++;
+ num_written++;
+ }
+ else
+ {
+ batch_continue = false;
+ }
}
+ else
+ {
+ if (to_write.nbuffers > 0)
+ WriteBuffers(&to_write, ioq, &wb_context);
+ }
+
+ /*
+ * Measure progress independent of actually having to flush the
+ * buffer - otherwise writing become unbalanced.
+ */
+ ts_stat->progress += ts_stat->progress_slice;
+ ts_stat->num_scanned++;
+ ts_stat->index++;
}
- /*
- * Measure progress independent of actually having to flush the buffer
- * - otherwise writing become unbalanced.
- */
- ts_stat->progress += ts_stat->progress_slice;
- ts_stat->num_scanned++;
- ts_stat->index++;
+ if (to_write.nbuffers > 0)
+ WriteBuffers(&to_write, ioq, &wb_context);
+
/* Have all the buffers from the tablespace been processed? */
if (ts_stat->num_scanned == ts_stat->num_to_scan)
@@ -3581,15 +3681,23 @@ BufferSync(int flags)
*
* (This will check for barrier events even if it doesn't sleep.)
*/
- CheckpointWriteDelay(flags, (double) num_processed / num_to_scan);
+ CheckpointWriteDelay(ioq, flags, (double) num_processed / num_to_scan);
}
+ Assert(to_write.nbuffers == 0);
+ io_queue_wait_all(ioq);
+
/*
* Issue all pending flushes. Only checkpointer calls BufferSync(), so
* IOContext will always be IOCONTEXT_NORMAL.
*/
IssuePendingWritebacks(&wb_context, IOCONTEXT_NORMAL);
+ io_queue_wait_all(ioq); /* IssuePendingWritebacks might have added
+ * more */
+ io_queue_free(ioq);
+ BuffersToWriteEnd(&to_write);
+
pfree(per_ts_stat);
per_ts_stat = NULL;
binaryheap_free(ts_heap);
@@ -3615,7 +3723,7 @@ BufferSync(int flags)
* bgwriter_lru_maxpages to 0.)
*/
bool
-BgBufferSync(WritebackContext *wb_context)
+BgBufferSync(IOQueue *ioq, WritebackContext *wb_context)
{
/* info obtained from freelist.c */
int strategy_buf_id;
@@ -3658,6 +3766,9 @@ BgBufferSync(WritebackContext *wb_context)
long new_strategy_delta;
uint32 new_recent_alloc;
+ BuffersToWrite to_write;
+ int max_combine;
+
/*
* Find out where the freelist clock sweep currently is, and how many
* buffer allocations have happened since our last call.
@@ -3678,6 +3789,8 @@ BgBufferSync(WritebackContext *wb_context)
return true;
}
+ max_combine = Min(io_bounce_buffers, io_combine_limit);
+
/*
* Compute strategy_delta = how many buffers have been scanned by the
* clock sweep since last time. If first time through, assume none. Then
@@ -3834,11 +3947,25 @@ BgBufferSync(WritebackContext *wb_context)
num_written = 0;
reusable_buffers = reusable_buffers_est;
+ BuffersToWriteInit(&to_write, ioq, wb_context);
+
/* Execute the LRU scan */
while (num_to_scan > 0 && reusable_buffers < upcoming_alloc_est)
{
- int sync_state = SyncOneBuffer(next_to_clean, true,
- wb_context);
+ int sync_state;
+
+ sync_state = PrepareToWriteBuffer(&to_write, next_to_clean + 1,
+ true, ioq, wb_context);
+ if (sync_state & BUF_CANT_MERGE)
+ {
+ Assert(to_write.nbuffers > 0);
+
+ WriteBuffers(&to_write, ioq, wb_context);
+
+ sync_state = PrepareToWriteBuffer(&to_write, next_to_clean + 1,
+ true, ioq, wb_context);
+ Assert(sync_state != BUF_CANT_MERGE);
+ }
if (++next_to_clean >= NBuffers)
{
@@ -3849,6 +3976,13 @@ BgBufferSync(WritebackContext *wb_context)
if (sync_state & BUF_WRITTEN)
{
+ Assert(sync_state & BUF_REUSABLE);
+
+ if (to_write.nbuffers == max_combine)
+ {
+ WriteBuffers(&to_write, ioq, wb_context);
+ }
+
reusable_buffers++;
if (++num_written >= bgwriter_lru_maxpages)
{
@@ -3860,6 +3994,11 @@ BgBufferSync(WritebackContext *wb_context)
reusable_buffers++;
}
+ if (to_write.nbuffers > 0)
+ WriteBuffers(&to_write, ioq, wb_context);
+
+ BuffersToWriteEnd(&to_write);
+
PendingBgWriterStats.buf_written_clean += num_written;
#ifdef BGW_DEBUG
@@ -3898,8 +4037,66 @@ BgBufferSync(WritebackContext *wb_context)
return (bufs_to_lap == 0 && recent_alloc == 0);
}
+static inline bool
+BufferTagsSameRel(const BufferTag *tag1, const BufferTag *tag2)
+{
+ return (tag1->spcOid == tag2->spcOid) &&
+ (tag1->dbOid == tag2->dbOid) &&
+ (tag1->relNumber == tag2->relNumber) &&
+ (tag1->forkNum == tag2->forkNum)
+ ;
+}
+
+static bool
+CanMergeWrite(BuffersToWrite *to_write, BufferDesc *cur_buf_hdr)
+{
+ BlockNumber cur_block = cur_buf_hdr->tag.blockNum;
+
+ Assert(to_write->nbuffers > 0); /* can't merge with nothing */
+ Assert(to_write->start_at_tag.relNumber != InvalidOid);
+ Assert(to_write->start_at_tag.blockNum != InvalidBlockNumber);
+
+ Assert(to_write->ioh != NULL);
+
+ /*
+ * First check if the blocknumber is one that we could actually merge,
+ * that's cheaper than checking the tablespace/db/relnumber/fork match.
+ */
+ if (to_write->start_at_tag.blockNum + to_write->nbuffers != cur_block)
+ return false;
+
+ if (!BufferTagsSameRel(&to_write->start_at_tag, &cur_buf_hdr->tag))
+ return false;
+
+ /*
+ * Need to check with smgr how large a write we're allowed to make. To
+ * reduce the overhead of the smgr check, only inquire once, when
+ * processing the first to-be-merged buffer. That avoids the overhead in
+ * the common case of writing out buffers that definitely not mergeable.
+ */
+ if (to_write->nbuffers == 1)
+ {
+ SMgrRelation smgr;
+
+ smgr = smgropen(BufTagGetRelFileLocator(&to_write->start_at_tag), INVALID_PROC_NUMBER);
+
+ to_write->max_combine = smgrmaxcombine(smgr,
+ to_write->start_at_tag.forkNum,
+ to_write->start_at_tag.blockNum);
+ }
+ else
+ {
+ Assert(to_write->max_combine > 0);
+ }
+
+ if (to_write->start_at_tag.blockNum + to_write->max_combine <= cur_block)
+ return false;
+
+ return true;
+}
+
/*
- * SyncOneBuffer -- process a single buffer during syncing.
+ * PrepareToWriteBuffer -- process a single buffer during syncing.
*
* If skip_recently_used is true, we don't write currently-pinned buffers, nor
* buffers marked recently used, as these are not replacement candidates.
@@ -3908,22 +4105,50 @@ BgBufferSync(WritebackContext *wb_context)
* BUF_WRITTEN: we wrote the buffer.
* BUF_REUSABLE: buffer is available for replacement, ie, it has
* pin count 0 and usage count 0.
+ * BUF_CANT_MERGE: can't combine this write with prior writes, caller needs
+ * to issue those first
*
* (BUF_WRITTEN could be set in error if FlushBuffer finds the buffer clean
* after locking it, but we don't care all that much.)
*/
static int
-SyncOneBuffer(int buf_id, bool skip_recently_used, WritebackContext *wb_context)
+PrepareToWriteBuffer(BuffersToWrite *to_write, Buffer buf,
+ bool skip_recently_used,
+ IOQueue *ioq, WritebackContext *wb_context)
{
- BufferDesc *bufHdr = GetBufferDescriptor(buf_id);
+ BufferDesc *cur_buf_hdr = GetBufferDescriptor(buf - 1);
+ uint32 buf_state;
int result = 0;
- uint32 buf_state;
- BufferTag tag;
+ XLogRecPtr cur_buf_lsn;
+ LWLock *content_lock;
+ bool may_block;
+
+ /*
+ * Check if this buffer can be written out together with already prepared
+ * writes. We check before we have pinned the buffer, so the buffer can be
+ * written out and replaced between this check and us pinning the buffer -
+ * we'll recheck below. The reason for the pre-check is that we don't want
+ * to pin the buffer just to find out that we can't merge the IO.
+ */
+ if (to_write->nbuffers != 0)
+ {
+ if (!CanMergeWrite(to_write, cur_buf_hdr))
+ {
+ result |= BUF_CANT_MERGE;
+ return result;
+ }
+ }
+ else
+ {
+ to_write->start_at_tag = cur_buf_hdr->tag;
+ }
/* Make sure we can handle the pin */
ReservePrivateRefCountEntry();
ResourceOwnerEnlarge(CurrentResourceOwner);
+ /* XXX: Should also check if we are allowed to pin one more buffer */
+
/*
* Check whether buffer needs writing.
*
@@ -3933,7 +4158,7 @@ SyncOneBuffer(int buf_id, bool skip_recently_used, WritebackContext *wb_context)
* don't worry because our checkpoint.redo points before log record for
* upcoming changes and so we are not required to write such dirty buffer.
*/
- buf_state = LockBufHdr(bufHdr);
+ buf_state = LockBufHdr(cur_buf_hdr);
if (BUF_STATE_GET_REFCOUNT(buf_state) == 0 &&
BUF_STATE_GET_USAGECOUNT(buf_state) == 0)
@@ -3942,40 +4167,300 @@ SyncOneBuffer(int buf_id, bool skip_recently_used, WritebackContext *wb_context)
}
else if (skip_recently_used)
{
+#if 0
+ elog(LOG, "at block %d: skip recent with nbuffers %d",
+ cur_buf_hdr->tag.blockNum, to_write->nbuffers);
+#endif
/* Caller told us not to write recently-used buffers */
- UnlockBufHdr(bufHdr, buf_state);
+ UnlockBufHdr(cur_buf_hdr, buf_state);
return result;
}
if (!(buf_state & BM_VALID) || !(buf_state & BM_DIRTY))
{
/* It's clean, so nothing to do */
- UnlockBufHdr(bufHdr, buf_state);
+ UnlockBufHdr(cur_buf_hdr, buf_state);
return result;
}
+ /* pin the buffer, from now on its identity can't change anymore */
+ PinBuffer_Locked(cur_buf_hdr);
+
+ /*
+ * Acquire IO, if needed, now that it's likely that we'll need to write.
+ */
+ if (to_write->ioh == NULL)
+ {
+ /* otherwise we should already have acquired a handle */
+ Assert(to_write->nbuffers == 0);
+
+ to_write->ioh = io_queue_acquire_io(ioq);
+ pgaio_io_get_wref(to_write->ioh, &to_write->iow);
+ }
+
+ /*
+ * If we are merging, check if the buffer's identity possibly changed
+ * while we hadn't yet pinned it.
+ *
+ * XXX: It might be worth checking if we still want to write the buffer
+ * out, e.g. it could have been replaced with a buffer that doesn't have
+ * BM_CHECKPOINT_NEEDED set.
+ */
+ if (to_write->nbuffers != 0)
+ {
+ if (!CanMergeWrite(to_write, cur_buf_hdr))
+ {
+ elog(LOG, "changed identity");
+ UnpinBuffer(cur_buf_hdr);
+
+ result |= BUF_CANT_MERGE;
+
+ return result;
+ }
+ }
+
+ may_block = to_write->nbuffers == 0
+ && !pgaio_have_staged()
+ && io_queue_is_empty(ioq)
+ ;
+ content_lock = BufferDescriptorGetContentLock(cur_buf_hdr);
+
+ if (!may_block)
+ {
+ if (LWLockConditionalAcquire(content_lock, LW_SHARED))
+ {
+ /* done */
+ }
+ else if (to_write->nbuffers == 0)
+ {
+ /*
+ * Need to wait for all prior IO to finish before blocking for
+ * lock acquisition, to avoid the risk a deadlock due to us
+ * waiting for another backend that is waiting for our unsubmitted
+ * IO to complete.
+ */
+ pgaio_submit_staged();
+ io_queue_wait_all(ioq);
+
+ elog(DEBUG2, "at block %u: can't block, nbuffers = 0",
+ cur_buf_hdr->tag.blockNum
+ );
+
+ may_block = to_write->nbuffers == 0
+ && !pgaio_have_staged()
+ && io_queue_is_empty(ioq)
+ ;
+ Assert(may_block);
+
+ LWLockAcquire(content_lock, LW_SHARED);
+ }
+ else
+ {
+ elog(DEBUG2, "at block %d: can't block nbuffers = %d",
+ cur_buf_hdr->tag.blockNum,
+ to_write->nbuffers);
+
+ UnpinBuffer(cur_buf_hdr);
+ result |= BUF_CANT_MERGE;
+ Assert(to_write->nbuffers > 0);
+
+ return result;
+ }
+ }
+ else
+ {
+ LWLockAcquire(content_lock, LW_SHARED);
+ }
+
+ if (!may_block)
+ {
+ if (!StartBufferIO(cur_buf_hdr, false, !may_block))
+ {
+ pgaio_submit_staged();
+ io_queue_wait_all(ioq);
+
+ may_block = io_queue_is_empty(ioq) && to_write->nbuffers == 0 && !pgaio_have_staged();
+
+ if (!StartBufferIO(cur_buf_hdr, false, !may_block))
+ {
+ elog(DEBUG2, "at block %d: non-waitable StartBufferIO returns false, %d",
+ cur_buf_hdr->tag.blockNum,
+ may_block);
+
+ /*
+ * FIXME: can't tell whether this is because the buffer has
+ * been cleaned
+ */
+ if (!may_block)
+ {
+ result |= BUF_CANT_MERGE;
+ Assert(to_write->nbuffers > 0);
+ }
+ LWLockRelease(content_lock);
+ UnpinBuffer(cur_buf_hdr);
+
+ return result;
+ }
+ }
+ }
+ else
+ {
+ if (!StartBufferIO(cur_buf_hdr, false, false))
+ {
+ elog(DEBUG2, "waitable StartBufferIO returns false");
+ LWLockRelease(content_lock);
+ UnpinBuffer(cur_buf_hdr);
+
+ /*
+ * FIXME: Historically we returned BUF_WRITTEN in this case, which
+ * seems wrong
+ */
+ return result;
+ }
+ }
+
/*
- * Pin it, share-lock it, write it. (FlushBuffer will do nothing if the
- * buffer is clean by the time we've locked it.)
+ * Run PageGetLSN while holding header lock, since we don't have the
+ * buffer locked exclusively in all cases.
*/
- PinBuffer_Locked(bufHdr);
- LWLockAcquire(BufferDescriptorGetContentLock(bufHdr), LW_SHARED);
+ buf_state = LockBufHdr(cur_buf_hdr);
+
+ cur_buf_lsn = BufferGetLSN(cur_buf_hdr);
- FlushBuffer(bufHdr, NULL, IOOBJECT_RELATION, IOCONTEXT_NORMAL);
+ /* To check if block content changes while flushing. - vadim 01/17/97 */
+ buf_state &= ~BM_JUST_DIRTIED;
- LWLockRelease(BufferDescriptorGetContentLock(bufHdr));
+ UnlockBufHdr(cur_buf_hdr, buf_state);
- tag = bufHdr->tag;
+ to_write->buffers[to_write->nbuffers] = buf;
+ to_write->nbuffers++;
- UnpinBuffer(bufHdr);
+ if (buf_state & BM_PERMANENT &&
+ (to_write->max_lsn == InvalidXLogRecPtr || to_write->max_lsn < cur_buf_lsn))
+ {
+ to_write->max_lsn = cur_buf_lsn;
+ }
+
+ result |= BUF_WRITTEN;
+
+ return result;
+}
+
+static void
+WriteBuffers(BuffersToWrite *to_write,
+ IOQueue *ioq, WritebackContext *wb_context)
+{
+ SMgrRelation smgr;
+ Buffer first_buf;
+ BufferDesc *first_buf_hdr;
+ bool needs_checksum;
+
+ Assert(to_write->nbuffers > 0 && to_write->nbuffers <= io_combine_limit);
+
+ first_buf = to_write->buffers[0];
+ first_buf_hdr = GetBufferDescriptor(first_buf - 1);
+
+ smgr = smgropen(BufTagGetRelFileLocator(&first_buf_hdr->tag), INVALID_PROC_NUMBER);
/*
- * SyncOneBuffer() is only called by checkpointer and bgwriter, so
- * IOContext will always be IOCONTEXT_NORMAL.
+ * Force XLOG flush up to buffer's LSN. This implements the basic WAL
+ * rule that log updates must hit disk before any of the data-file changes
+ * they describe do.
+ *
+ * However, this rule does not apply to unlogged relations, which will be
+ * lost after a crash anyway. Most unlogged relation pages do not bear
+ * LSNs since we never emit WAL records for them, and therefore flushing
+ * up through the buffer LSN would be useless, but harmless. However,
+ * GiST indexes use LSNs internally to track page-splits, and therefore
+ * unlogged GiST pages bear "fake" LSNs generated by
+ * GetFakeLSNForUnloggedRel. It is unlikely but possible that the fake
+ * LSN counter could advance past the WAL insertion point; and if it did
+ * happen, attempting to flush WAL through that location would fail, with
+ * disastrous system-wide consequences. To make sure that can't happen,
+ * skip the flush if the buffer isn't permanent.
*/
- ScheduleBufferTagForWriteback(wb_context, IOCONTEXT_NORMAL, &tag);
+ if (to_write->max_lsn != InvalidXLogRecPtr)
+ XLogFlush(to_write->max_lsn);
+
+ /*
+ * Now it's safe to write the buffer to disk. Note that no one else should
+ * have been able to write it, while we were busy with log flushing,
+ * because we got the exclusive right to perform I/O by setting the
+ * BM_IO_IN_PROGRESS bit.
+ */
+
+ for (int nbuf = 0; nbuf < to_write->nbuffers; nbuf++)
+ {
+ Buffer cur_buf = to_write->buffers[nbuf];
+ BufferDesc *cur_buf_hdr = GetBufferDescriptor(cur_buf - 1);
+ Block bufBlock;
+ char *bufToWrite;
+
+ bufBlock = BufHdrGetBlock(cur_buf_hdr);
+ needs_checksum = PageNeedsChecksumCopy((Page) bufBlock);
+
+ /*
+ * Update page checksum if desired. Since we have only shared lock on
+ * the buffer, other processes might be updating hint bits in it, so
+ * we must copy the page to a bounce buffer if we do checksumming.
+ */
+ if (needs_checksum)
+ {
+ PgAioBounceBuffer *bb = pgaio_bounce_buffer_get();
- return result | BUF_WRITTEN;
+ pgaio_io_assoc_bounce_buffer(to_write->ioh, bb);
+
+ bufToWrite = pgaio_bounce_buffer_buffer(bb);
+ memcpy(bufToWrite, bufBlock, BLCKSZ);
+ PageSetChecksumInplace((Page) bufToWrite, cur_buf_hdr->tag.blockNum);
+ }
+ else
+ {
+ bufToWrite = bufBlock;
+ }
+
+ to_write->data_ptrs[nbuf] = bufToWrite;
+ }
+
+ pgaio_io_set_handle_data_32(to_write->ioh,
+ (uint32 *) to_write->buffers,
+ to_write->nbuffers);
+ pgaio_io_register_callbacks(to_write->ioh, PGAIO_HCB_SHARED_BUFFER_WRITEV, 0);
+
+ smgrstartwritev(to_write->ioh, smgr,
+ BufTagGetForkNum(&first_buf_hdr->tag),
+ first_buf_hdr->tag.blockNum,
+ to_write->data_ptrs,
+ to_write->nbuffers,
+ false);
+ pgstat_count_io_op(IOOBJECT_RELATION, IOCONTEXT_NORMAL,
+ IOOP_WRITE, 1, BLCKSZ * to_write->nbuffers);
+
+
+ for (int nbuf = 0; nbuf < to_write->nbuffers; nbuf++)
+ {
+ Buffer cur_buf = to_write->buffers[nbuf];
+ BufferDesc *cur_buf_hdr = GetBufferDescriptor(cur_buf - 1);
+
+ UnpinBuffer(cur_buf_hdr);
+ }
+
+ io_queue_track(ioq, &to_write->iow);
+ to_write->total_writes++;
+
+ /* clear state for next write */
+ to_write->nbuffers = 0;
+ to_write->start_at_tag.relNumber = InvalidOid;
+ to_write->start_at_tag.blockNum = InvalidBlockNumber;
+ to_write->max_combine = 0;
+ to_write->max_lsn = InvalidXLogRecPtr;
+ to_write->ioh = NULL;
+ pgaio_wref_clear(&to_write->iow);
+
+ /*
+ * FIXME: Implement issuing writebacks (note wb_context isn't used here).
+ * Possibly needs to be integrated with io_queue.c.
+ */
}
/*
@@ -4349,6 +4834,7 @@ FlushBuffer(BufferDesc *buf, SMgrRelation reln, IOObject io_object,
error_context_stack = errcallback.previous;
}
+
/*
* RelationGetNumberOfBlocksInFork
* Determines the current number of pages in the specified relation fork.
diff --git a/src/backend/storage/page/bufpage.c b/src/backend/storage/page/bufpage.c
index 82457bacc62..d4bbd2cdf00 100644
--- a/src/backend/storage/page/bufpage.c
+++ b/src/backend/storage/page/bufpage.c
@@ -1490,6 +1490,16 @@ PageIndexTupleOverwrite(Page page, OffsetNumber offnum,
return true;
}
+bool
+PageNeedsChecksumCopy(Page page)
+{
+ if (PageIsNew(page))
+ return false;
+
+ /* If we don't need a checksum, just return the passed-in data */
+ return DataChecksumsEnabled();
+}
+
/*
* Set checksum for a page in shared buffers.
diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list
index 0c6ddadc51d..e89c501fa8a 100644
--- a/src/tools/pgindent/typedefs.list
+++ b/src/tools/pgindent/typedefs.list
@@ -349,6 +349,7 @@ BufferManagerRelation
BufferStrategyControl
BufferTag
BufferUsage
+BuffersToWrite
BuildAccumulator
BuiltinScript
BulkInsertState
--
2.48.1.76.g4e746b1a31.dirty
[text/x-diff] v2.15-0014-Ensure-a-resowner-exists-for-all-paths-that-ma.patch (2.6K, ../../7s6fclfekpcxoaaorwrq67v4vmgixf2dcjcgznuj7vxs3ie3wq@okqbi5vabqbx/15-v2.15-0014-Ensure-a-resowner-exists-for-all-paths-that-ma.patch)
download | inline diff:
From 5d6f12023658f2cde878fb5f38ff2ce51a613d66 Mon Sep 17 00:00:00 2001
From: Andres Freund <andres@anarazel.de>
Date: Tue, 18 Mar 2025 14:40:06 -0400
Subject: [PATCH v2.15 14/18] Ensure a resowner exists for all paths that may
perform AIO
Reviewed-by: Noah Misch <noah@leadboat.com>
Discussion: https://postgr.es/m/1f6b50a7-38ef-4d87-8246-786d39f46ab9@iki.fi
---
src/backend/bootstrap/bootstrap.c | 7 +++++++
src/backend/replication/logical/logical.c | 6 ++++++
src/backend/utils/init/postinit.c | 6 +++++-
3 files changed, 18 insertions(+), 1 deletion(-)
diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c
index 6db864892d0..e554504e1f0 100644
--- a/src/backend/bootstrap/bootstrap.c
+++ b/src/backend/bootstrap/bootstrap.c
@@ -361,8 +361,15 @@ BootstrapModeMain(int argc, char *argv[], bool check_only)
BaseInit();
bootstrap_signals();
+
+ /* need a resowner for IO during BootStrapXLOG() */
+ CreateAuxProcessResourceOwner();
+
BootStrapXLOG(bootstrap_data_checksum_version);
+ ReleaseAuxProcessResources(true);
+ CurrentResourceOwner = NULL;
+
/*
* To ensure that src/common/link-canary.c is linked into the backend, we
* must call it from somewhere. Here is as good as anywhere.
diff --git a/src/backend/replication/logical/logical.c b/src/backend/replication/logical/logical.c
index a8d2e024d34..72c32f5c32e 100644
--- a/src/backend/replication/logical/logical.c
+++ b/src/backend/replication/logical/logical.c
@@ -386,6 +386,12 @@ CreateInitDecodingContext(const char *plugin,
slot->data.plugin = plugin_name;
SpinLockRelease(&slot->mutex);
+ if (CurrentResourceOwner == NULL)
+ {
+ Assert(am_walsender);
+ CurrentResourceOwner = AuxProcessResourceOwner;
+ }
+
if (XLogRecPtrIsInvalid(restart_lsn))
ReplicationSlotReserveWal();
else
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index 7958ea11b73..222e24bcb08 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -785,8 +785,12 @@ InitPostgres(const char *in_dbname, Oid dboid,
* We don't yet have an aux-process resource owner, but StartupXLOG
* and ShutdownXLOG will need one. Hence, create said resource owner
* (and register a callback to clean it up after ShutdownXLOG runs).
+ *
+ * In bootstrap mode CreateAuxProcessResourceOwner() was already
+ * called in BootstrapModeMain().
*/
- CreateAuxProcessResourceOwner();
+ if (!bootstrap)
+ CreateAuxProcessResourceOwner();
StartupXLOG();
/* Release (and warn about) any buffer pins leaked in StartupXLOG */
--
2.48.1.76.g4e746b1a31.dirty
[text/x-diff] v2.15-0015-Temporary-Increase-BAS_BULKREAD-size.patch (1.3K, ../../7s6fclfekpcxoaaorwrq67v4vmgixf2dcjcgznuj7vxs3ie3wq@okqbi5vabqbx/16-v2.15-0015-Temporary-Increase-BAS_BULKREAD-size.patch)
download | inline diff:
From 51cb522b6b2467c63268d4c7f37af07914b081c6 Mon Sep 17 00:00:00 2001
From: Andres Freund <andres@anarazel.de>
Date: Tue, 18 Mar 2025 14:40:06 -0400
Subject: [PATCH v2.15 15/18] Temporary: Increase BAS_BULKREAD size
Without this we only can execute very little AIO for sequential scans, as
there's just not enough buffers in the ring. This isn't the right fix, as
just increasing the ring size can have negative performance implications in
workloads where the kernel has all the data cached.
Author:
Reviewed-By:
Discussion: https://postgr.es/m/
Backpatch:
---
src/backend/storage/buffer/freelist.c | 7 ++++++-
1 file changed, 6 insertions(+), 1 deletion(-)
diff --git a/src/backend/storage/buffer/freelist.c b/src/backend/storage/buffer/freelist.c
index 336715b6c63..b72a5957a20 100644
--- a/src/backend/storage/buffer/freelist.c
+++ b/src/backend/storage/buffer/freelist.c
@@ -555,7 +555,12 @@ GetAccessStrategy(BufferAccessStrategyType btype)
return NULL;
case BAS_BULKREAD:
- ring_size_kb = 256;
+
+ /*
+ * FIXME: Temporary increase to allow large enough streaming reads
+ * to actually benefit from AIO. This needs a better solution.
+ */
+ ring_size_kb = 2 * 1024;
break;
case BAS_BULKWRITE:
ring_size_kb = 16 * 1024;
--
2.48.1.76.g4e746b1a31.dirty
[text/x-diff] v2.15-0016-WIP-Use-MAP_POPULATE.patch (1.1K, ../../7s6fclfekpcxoaaorwrq67v4vmgixf2dcjcgznuj7vxs3ie3wq@okqbi5vabqbx/17-v2.15-0016-WIP-Use-MAP_POPULATE.patch)
download | inline diff:
From 6f6df9dfd8f5169a0ef0dac00356d05c6fa0a9fd Mon Sep 17 00:00:00 2001
From: Andres Freund <andres@anarazel.de>
Date: Tue, 18 Mar 2025 14:40:06 -0400
Subject: [PATCH v2.15 16/18] WIP: Use MAP_POPULATE
For benchmarking it's quite annoying that the first time a memory is touched
has completely different perf characteristics than subsequent accesses. Using
MAP_POPULATE reduces that substantially.
---
src/backend/port/sysv_shmem.c | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/src/backend/port/sysv_shmem.c b/src/backend/port/sysv_shmem.c
index 197926d44f6..a700b02d5a1 100644
--- a/src/backend/port/sysv_shmem.c
+++ b/src/backend/port/sysv_shmem.c
@@ -620,7 +620,7 @@ CreateAnonymousSegment(Size *size)
allocsize += hugepagesize - (allocsize % hugepagesize);
ptr = mmap(NULL, allocsize, PROT_READ | PROT_WRITE,
- PG_MMAP_FLAGS | mmap_flags, -1, 0);
+ PG_MMAP_FLAGS | MAP_POPULATE | mmap_flags, -1, 0);
mmap_errno = errno;
if (huge_pages == HUGE_PAGES_TRY && ptr == MAP_FAILED)
elog(DEBUG1, "mmap(%zu) with MAP_HUGETLB failed, huge pages disabled: %m",
--
2.48.1.76.g4e746b1a31.dirty
^ permalink raw reply [nested|flat] 153+ messages in thread
* Re: AIO v2.5
2025-03-04 19:00 AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-10 18:23 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-11 19:41 ` Re: AIO v2.5 Noah Misch <noah@leadboat.com>
2025-03-11 23:55 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-12 03:57 ` Re: AIO v2.5 Noah Misch <noah@leadboat.com>
2025-03-12 17:06 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-12 23:23 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-13 10:53 ` Re: AIO v2.5 Antonin Houska <ah@cybertec.at>
2025-03-14 19:43 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-17 20:00 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-18 20:12 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-21 01:58 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-25 01:18 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-27 01:07 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-27 20:58 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-28 03:22 ` Re: AIO v2.5 Noah Misch <noah@leadboat.com>
2025-03-28 12:54 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-29 03:35 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-29 13:41 ` Re: AIO v2.5 Noah Misch <noah@leadboat.com>
2025-03-29 14:48 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-29 18:25 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-29 21:29 ` Re: AIO v2.5 Noah Misch <noah@leadboat.com>
2025-03-30 23:46 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-04-01 00:41 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
@ 2025-04-01 11:56 ` Aleksander Alekseev <aleksander@timescale.com>
2025-04-01 13:45 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
1 sibling, 1 reply; 153+ messages in thread
From: Aleksander Alekseev @ 2025-04-01 11:56 UTC (permalink / raw)
To: pgsql-hackers; +Cc: Andres Freund <andres@anarazel.de>; Noah Misch <noah@leadboat.com>; Thomas Munro <thomas.munro@gmail.com>; Heikki Linnakangas <hlinnaka@iki.fi>; Robert Haas <robertmhaas@gmail.com>; Jakub Wartak <jakub.wartak@enterprisedb.com>; Jelte Fennema-Nio <postgres@jeltef.nl>; Antonin Houska <ah@cybertec.at>
Hi Andres,
> > I didn't yet push
> >
> > > > Subject: [PATCH v2.14 13/29] aio: Add README.md explaining higher level design
I have several notes about 0003 / README.md:
1. I noticed that the use of "Postgres" and "postgres" is inconsistent.
2.
```
+pgaio_io_register_callbacks(ioh, PGAIO_HCB_SHARED_BUFFER_READV, 0);
```
Perhaps I'm a bit late here, but the name of the function is weird. It
registers a single callback, but the name is "_callbacks".
3. The use of "AIO Handle" and "AioHandle" is inconsistent.
4.
- pgaio_io_register_callbacks
- pgaio_io_set_handle_data_32
If I understand correctly one can register multiple callbacks per one
AIO Handle (right? ...). However I don't see an obvious way to match
handle data to the given callback. If all the callbacks get the same
handle data... well it's weird IMO, but we should explicitly say that.
On top of that we should probably explain in which order the callbacks
are going to be executed. If there are any guarantees in this respect
of course.
5. pgaio_io_set_handle_data_32(ioh, (uint32 *) buffer, 1)
Perhaps it's worth mentioning if `buffer` can be freed after the call
i.e. if it's stored by value or by reference. It's also worth
clarifying if the maximum number of buffers is limited or not.
6. It is worth clarifying if AIO allows reads and writes or only reads
at the moment. Perhaps it's also worth explicitly saying that AIO is
for disk IO only, not for network one.
7. It is worth clarifying how many times the callbacks are called when
reading multiple buffers. Is it guaranteed that the callbacks are
called ones, or if it somehow depends on the implementation, and also
what happens in the case if I/O succeeds partially.
8. I believe we should tell a bit more about the context in which the
callbacks are called. Particularly what happens to the memory contexts
and if I can allocate/free memory, can I throw ERRORs, can I create
new AIO Handles, is it expected that the callback should return
quickly, are the signals masked while the callback is executed, can I
use sockets, is it guaranteed that the callback is going to be called
in the same process (I guess so, but the text doesn't explicitly
promise that), etc.
9.
```
+Because acquisition of an IO handle
+[must always succeed](#io-can-be-started-in-critical-sections)
+and the number of AIO Handles
+[has to be limited](#state-for-aio-needs-to-live-in-shared-memory)
+AIO handles can be reused as soon as they have completed.
```
What pgaio_io_acquire() does if we are out of AIO Handles? Since it
always succeeds I guess it should block the caller in this case, but
IMO we should say this explicitly.
10.
> > because I want to integrate some language that could be referenced by
> > smgrstartreadv() (and more in the future), as we have been talking about.
>
> I tried a bunch of variations and none of them seemed great. So I ended up
> with a lightly polished version of your suggested comment above
> smgrstartreadv(). We can later see about generalizing it.
IMO the problem here is that README doesn't show the code that does IO
per se, and thus doesn't give the full picture of how AIO should be
used. Perhaps instead of referencing smgrstartreadv() it would be
better to provide a simple but complete example, one that opens a
binary file and reads 512 bytes from it by the given offset for
instance.
--
Best regards,
Aleksander Alekseev
^ permalink raw reply [nested|flat] 153+ messages in thread
* Re: AIO v2.5
2025-03-04 19:00 AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-10 18:23 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-11 19:41 ` Re: AIO v2.5 Noah Misch <noah@leadboat.com>
2025-03-11 23:55 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-12 03:57 ` Re: AIO v2.5 Noah Misch <noah@leadboat.com>
2025-03-12 17:06 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-12 23:23 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-13 10:53 ` Re: AIO v2.5 Antonin Houska <ah@cybertec.at>
2025-03-14 19:43 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-17 20:00 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-18 20:12 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-21 01:58 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-25 01:18 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-27 01:07 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-27 20:58 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-28 03:22 ` Re: AIO v2.5 Noah Misch <noah@leadboat.com>
2025-03-28 12:54 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-29 03:35 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-29 13:41 ` Re: AIO v2.5 Noah Misch <noah@leadboat.com>
2025-03-29 14:48 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-29 18:25 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-29 21:29 ` Re: AIO v2.5 Noah Misch <noah@leadboat.com>
2025-03-30 23:46 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-04-01 00:41 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-04-01 11:56 ` Re: AIO v2.5 Aleksander Alekseev <aleksander@timescale.com>
@ 2025-04-01 13:45 ` Andres Freund <andres@anarazel.de>
0 siblings, 0 replies; 153+ messages in thread
From: Andres Freund @ 2025-04-01 13:45 UTC (permalink / raw)
To: Aleksander Alekseev <aleksander@timescale.com>; +Cc: pgsql-hackers@postgresql.org, Noah Misch <noah@leadboat.com>; Thomas Munro <thomas.munro@gmail.com>; Heikki Linnakangas <hlinnaka@iki.fi>; Robert Haas <robertmhaas@gmail.com>; Jakub Wartak <jakub.wartak@enterprisedb.com>; Jelte Fennema-Nio <postgres@jeltef.nl>; Antonin Houska <ah@cybertec.at>
Hi,
On 2025-04-01 14:56:17 +0300, Aleksander Alekseev wrote:
> Hi Andres,
>
> > > I didn't yet push
> > >
> > > > > Subject: [PATCH v2.14 13/29] aio: Add README.md explaining higher level design
>
> I have several notes about 0003 / README.md:
>
> 1. I noticed that the use of "Postgres" and "postgres" is inconsistent.
:/
It probably should be consistent, but I have no idea which of the spellings we
should go for. Either looks ugly in some contexts.
> 2.
>
> ```
> +pgaio_io_register_callbacks(ioh, PGAIO_HCB_SHARED_BUFFER_READV, 0);
> ```
>
> Perhaps I'm a bit late here, but the name of the function is weird. It
> registers a single callback, but the name is "_callbacks".
It registers a *set* of callbacks (stage, complete_shared, complete_local,
report_error) on the handle.
> 3. The use of "AIO Handle" and "AioHandle" is inconsistent.
This seems ok to me.
> 4.
>
> - pgaio_io_register_callbacks
> - pgaio_io_set_handle_data_32
>
> If I understand correctly one can register multiple callbacks per one
> AIO Handle (right? ...). However I don't see an obvious way to match
> handle data to the given callback. If all the callbacks get the same
> handle data... well it's weird IMO, but we should explicitly say that.
There is:
/*
* Associate an array of data with the Handle. This is e.g. useful to the
* transport knowledge about which buffers a multi-block IO affects to
* completion callbacks.
*
* Right now this can be done only once for each IO, even though multiple
* callbacks can be registered. There aren't any known usecases requiring more
* and the required amount of shared memory does add up, so it doesn't seem
* worth multiplying memory usage by PGAIO_HANDLE_MAX_CALLBACKS.
*/
> On top of that we should probably explain in which order the callbacks
> are going to be executed. If there are any guarantees in this respect
> of course.
Alongside PgAioHandleCallbacks:
*
* The latest registered callback is called first. This allows
* higher-level code to register callbacks that can rely on callbacks
* registered by lower-level code to already have been executed.
*
> 5. pgaio_io_set_handle_data_32(ioh, (uint32 *) buffer, 1)
>
> Perhaps it's worth mentioning if `buffer` can be freed after the call
> i.e. if it's stored by value or by reference.
By value.
> It's also worth clarifying if the maximum number of buffers is limited or
> not.
It's limited to PG_IOV_MAX, fwiw.
> 6. It is worth clarifying if AIO allows reads and writes or only reads
> at the moment.
We have patches for writes, I just ran out of time for 18. Im not particularly
excited about adding stuff that then needs to be removed in 19.
> Perhaps it's also worth explicitly saying that AIO is for disk IO only, not
> for network one.
I'd like to integrate network IO too. I have a local prototype, fwiw.
> 7. It is worth clarifying how many times the callbacks are called when
> reading multiple buffers. Is it guaranteed that the callbacks are
> called ones, or if it somehow depends on the implementation, and also
> what happens in the case if I/O succeeds partially.
The aio subsystem doesn't know anything about buffers. Callbacks are executed
exactly once, with the exception of the error reporting callback, which could
be called multiple times.
> 8. I believe we should tell a bit more about the context in which the
> callbacks are called. Particularly what happens to the memory contexts
> and if I can allocate/free memory, can I throw ERRORs, can I create
> new AIO Handles, is it expected that the callback should return
> quickly, are the signals masked while the callback is executed, can I
> use sockets, is it guaranteed that the callback is going to be called
> in the same process (I guess so, but the text doesn't explicitly
> promise that), etc.
There is the following above pgaio_io_register_callbacks()
* Note that callbacks are executed in critical sections. This is necessary
* to be able to execute IO in critical sections (consider e.g. WAL
* logging). To perform AIO we first need to acquire a handle, which, if there
* are no free handles, requires waiting for IOs to complete and to execute
* their completion callbacks.
*
* Callbacks may be executed in the issuing backend but also in another
* backend (because that backend is waiting for the IO) or in IO workers (if
* io_method=worker is used).
And also a bunch of detail along struct PgAioHandleCallbacks.
> 9.
>
> ```
> +Because acquisition of an IO handle
> +[must always succeed](#io-can-be-started-in-critical-sections)
> +and the number of AIO Handles
> +[has to be limited](#state-for-aio-needs-to-live-in-shared-memory)
> +AIO handles can be reused as soon as they have completed.
> ```
>
> What pgaio_io_acquire() does if we are out of AIO Handles? Since it
> always succeeds I guess it should block the caller in this case, but
> IMO we should say this explicitly.
That's documented above pgaio_io_acquire().
> 10.
>
> > > because I want to integrate some language that could be referenced by
> > > smgrstartreadv() (and more in the future), as we have been talking about.
> >
> > I tried a bunch of variations and none of them seemed great. So I ended up
> > with a lightly polished version of your suggested comment above
> > smgrstartreadv(). We can later see about generalizing it.
>
> IMO the problem here is that README doesn't show the code that does IO
> per se, and thus doesn't give the full picture of how AIO should be
> used. Perhaps instead of referencing smgrstartreadv() it would be
> better to provide a simple but complete example, one that opens a
> binary file and reads 512 bytes from it by the given offset for
> instance.
IMO the example is already long enough, if you want that level of detail, you
can just look at smgrstartreadv() etc. The idea about explaining it at that
level is that that is basically what is required to use AIO in a new place,
whereas implementing AIO for a new target, or a new IO operation, requires a
bit more care.
Greetings,
Andres Freund
^ permalink raw reply [nested|flat] 153+ messages in thread
* Re: AIO v2.5
2025-03-04 19:00 AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-10 18:23 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-11 19:41 ` Re: AIO v2.5 Noah Misch <noah@leadboat.com>
2025-03-11 23:55 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-12 03:57 ` Re: AIO v2.5 Noah Misch <noah@leadboat.com>
2025-03-12 17:06 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-12 23:23 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-13 10:53 ` Re: AIO v2.5 Antonin Houska <ah@cybertec.at>
2025-03-14 19:43 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-17 20:00 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-18 20:12 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-21 01:58 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-25 01:18 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-27 01:07 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-27 20:58 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-28 03:22 ` Re: AIO v2.5 Noah Misch <noah@leadboat.com>
2025-03-28 12:54 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-29 03:35 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-29 13:41 ` Re: AIO v2.5 Noah Misch <noah@leadboat.com>
2025-03-29 14:48 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-29 18:25 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-29 21:29 ` Re: AIO v2.5 Noah Misch <noah@leadboat.com>
2025-03-30 23:46 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-04-01 00:41 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
@ 2025-04-01 15:11 ` Noah Misch <noah@leadboat.com>
2025-04-01 15:55 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
1 sibling, 1 reply; 153+ messages in thread
From: Noah Misch @ 2025-04-01 15:11 UTC (permalink / raw)
To: Andres Freund <andres@anarazel.de>; +Cc: pgsql-hackers@postgresql.org, Thomas Munro <thomas.munro@gmail.com>; Heikki Linnakangas <hlinnaka@iki.fi>; Robert Haas <robertmhaas@gmail.com>; Jakub Wartak <jakub.wartak@enterprisedb.com>; Jelte Fennema-Nio <postgres@jeltef.nl>; Antonin Houska <ah@cybertec.at>
On Mon, Mar 31, 2025 at 08:41:39PM -0400, Andres Freund wrote:
> updated version
All non-write patches (1-7) are ready for commit, though I have some cosmetic
recommendations below. I've marked the commitfest entry Ready for Committer.
> + # Check a page validity error in another block, to ensure we report
> + # the correct block number
> + $psql_a->query_safe(
> + qq(
> +SELECT modify_rel_block('tbl_zero', 3, corrupt_header=>true);
> +));
> + psql_like(
> + $io_method,
> + $psql_a,
> + "$persistency: test zeroing of invalid block 3",
> + qq(SELECT read_rel_block_ll('tbl_zero', 3, zero_on_error=>true);),
> + qr/^$/,
> + qr/^psql:<stdin>:\d+: WARNING: invalid page in block 3 of relation base\/.*\/.*; zeroing out page$/
> + );
> +
> +
> + # Check a page validity error in another block, to ensure we report
> + # the correct block number
This comment is a copy of the previous test's comment. While the comment is
not false, consider changing it to:
# Check one read reporting multiple invalid blocks.
> + $psql_a->query_safe(
> + qq(
> +SELECT modify_rel_block('tbl_zero', 2, corrupt_header=>true);
> +SELECT modify_rel_block('tbl_zero', 3, corrupt_header=>true);
> +));
> + # First test error
> + psql_like(
> + $io_method,
> + $psql_a,
> + "$persistency: test reading of invalid block 2,3 in larger read",
> + qq(SELECT read_rel_block_ll('tbl_zero', 1, nblocks=>4, zero_on_error=>false)),
> + qr/^$/,
> + qr/^psql:<stdin>:\d+: ERROR: 2 invalid pages among blocks 1..4 of relation base\/.*\/.*\nDETAIL: Block 2 held first invalid page\.\nHINT:[^\n]+$/
> + );
> +
> + # Then test zeroing via ZERO_ON_ERROR flag
> + psql_like(
> + $io_method,
> + $psql_a,
> + "$persistency: test zeroing of invalid block 2,3 in larger read, ZERO_ON_ERROR",
> + qq(SELECT read_rel_block_ll('tbl_zero', 1, nblocks=>4, zero_on_error=>true)),
> + qr/^$/,
> + qr/^psql:<stdin>:\d+: WARNING: zeroing out 2 invalid pages among blocks 1..4 of relation base\/.*\/.*\nDETAIL: Block 2 held first zeroed page\.\nHINT:[^\n]+$/
> + );
> +
> + # Then test zeroing vio zero_damaged_pages
s/vio/via/
> +# Verify checksum handling when creating database from an invalid database.
> +# This also serves as a minimal check that cross-database IO is handled
> +# reasonably.
To me, "invalid database" is a term of art from the message "cannot connect to
invalid database". Hence, I would change "invalid database" to "database w/
invalid block" or similar, here and below. (Alternatively, just delete "from
an invalid database". It's clear from the context.)
> + if (corrupt_checksum)
> + {
> + bool successfully_corrupted = 0;
> +
> + /*
> + * Any single modification of the checksum could just end up being
> + * valid again. To be sure
> + */
Unfinished sentence. That said, I'm not following why we'd need this loop.
If this test code were changing the input to the checksum, it's true that an
input bit flip might reach the same pd_checksum. The test case is changing
pd_checksum, not the input bits. I don't see how changing pd_checksum could
leave the page still valid. There's only one valid pd_checksum value for a
given input page.
> + /*
> + * The underlying IO actually completed OK, and thus the "invalid"
> + * portion of the IOV actually contains valid data. That can hide
> + * a lot of problems, e.g. if we were to wrongly mark a buffer,
> + * that wasn't read according to the shortened-read, IO as valid,
> + * the contents would look valid and we might miss a bug.
Minimally s/read, IO/read IO,/ but I'd edit a bit further:
* a lot of problems, e.g. if we were to wrongly mark-valid a
* buffer that wasn't read according to the shortened-read IO, the
* contents would look valid and we might miss a bug.
> Subject: [PATCH v2.15 05/18] md: Add comment & assert to buffer-zeroing path
> in md[start]readv()
> The zero_damaged_pages path is incomplete, as as missing segments are not
s/as as/as/
> For now, put an Assert(false) comments documenting this choice into mdreadv()
s/comments/and comments/
> + * For PG 18, we are putting an Assert(false) in into
> + * mdreadv() (triggering failures in assertion-enabled builds,
s/in into/in/
> Subject: [PATCH v2.15 06/18] aio: comment polishing
> + * - Partial reads need to be handle by the caller re-issuing IO for the
> + * unread blocks
s/handle/handled/
> Subject: [PATCH v2.15 07/18] aio: Add errcontext for processing I/Os for
> another backend
^ permalink raw reply [nested|flat] 153+ messages in thread
* Re: AIO v2.5
2025-03-04 19:00 AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-10 18:23 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-11 19:41 ` Re: AIO v2.5 Noah Misch <noah@leadboat.com>
2025-03-11 23:55 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-12 03:57 ` Re: AIO v2.5 Noah Misch <noah@leadboat.com>
2025-03-12 17:06 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-12 23:23 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-13 10:53 ` Re: AIO v2.5 Antonin Houska <ah@cybertec.at>
2025-03-14 19:43 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-17 20:00 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-18 20:12 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-21 01:58 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-25 01:18 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-27 01:07 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-27 20:58 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-28 03:22 ` Re: AIO v2.5 Noah Misch <noah@leadboat.com>
2025-03-28 12:54 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-29 03:35 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-29 13:41 ` Re: AIO v2.5 Noah Misch <noah@leadboat.com>
2025-03-29 14:48 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-29 18:25 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-29 21:29 ` Re: AIO v2.5 Noah Misch <noah@leadboat.com>
2025-03-30 23:46 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-04-01 00:41 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-04-01 15:11 ` Re: AIO v2.5 Noah Misch <noah@leadboat.com>
@ 2025-04-01 15:55 ` Andres Freund <andres@anarazel.de>
2025-04-01 16:07 ` Re: AIO v2.5 Noah Misch <noah@leadboat.com>
2025-04-01 21:47 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
0 siblings, 2 replies; 153+ messages in thread
From: Andres Freund @ 2025-04-01 15:55 UTC (permalink / raw)
To: Noah Misch <noah@leadboat.com>; +Cc: pgsql-hackers@postgresql.org, Thomas Munro <thomas.munro@gmail.com>; Heikki Linnakangas <hlinnaka@iki.fi>; Robert Haas <robertmhaas@gmail.com>; Jakub Wartak <jakub.wartak@enterprisedb.com>; Jelte Fennema-Nio <postgres@jeltef.nl>; Antonin Houska <ah@cybertec.at>
Hi,
On 2025-04-01 08:11:59 -0700, Noah Misch wrote:
> On Mon, Mar 31, 2025 at 08:41:39PM -0400, Andres Freund wrote:
> > updated version
>
> All non-write patches (1-7) are ready for commit, though I have some cosmetic
> recommendations below. I've marked the commitfest entry Ready for Committer.
Thanks!
I haven't yet pushed the changes, but will work on that in the afternoon.
I plan to afterwards close the CF entry and will eventually create a new one
for write support, although probably only rebasing onto
https://postgr.es/m/stj36ea6yyhoxtqkhpieia2z4krnam7qyetc57rfezgk4zgapf%40gcnactj4z56m
and addressing some of the locking issues.
WRT the locking issues, I've been wondering whether we could make
LWLockWaitForVar() work that purpose, but I doubt it's the right approach.
Probably better to get rid of the LWLock*Var functions and go for the approach
I had in v1, namely a version of LWLockAcquire() with a callback that gets
called between LWLockQueueSelf() and PGSemaphoreLock(), which can cause the
lock acquisition to abort.
> This comment is a copy of the previous test's comment. While the comment is
> not false, consider changing it to:
>
> # Check one read reporting multiple invalid blocks.
> > + # Then test zeroing vio zero_damaged_pages
>
> s/vio/via/
>
These make sense.
> > +# Verify checksum handling when creating database from an invalid database.
> > +# This also serves as a minimal check that cross-database IO is handled
> > +# reasonably.
>
> To me, "invalid database" is a term of art from the message "cannot connect to
> invalid database". Hence, I would change "invalid database" to "database w/
> invalid block" or similar, here and below. (Alternatively, just delete "from
> an invalid database". It's clear from the context.)
Yea, I agree, this is easy to misunderstand when stepping back. I went for "with
an invalid block".
> > + if (corrupt_checksum)
> > + {
> > + bool successfully_corrupted = 0;
> > +
> > + /*
> > + * Any single modification of the checksum could just end up being
> > + * valid again. To be sure
> > + */
>
> Unfinished sentence.
Oops. See below.
> That said, I'm not following why we'd need this loop. If this test code
> were changing the input to the checksum, it's true that an input bit flip
> might reach the same pd_checksum. The test case is changing pd_checksum,
> not the input bits.
We might be changing the input, due to the zero/corrupt_header options. Or we
might be called on a page that is *already* corrupted. I did encounter that
situation once while writing tests, where the tests only passed if I made the
+ 1 a + 2. Which was, uh, rather confusing and left me feel like I was cursed
that day.
> I don't see how changing pd_checksum could leave the
> page still valid. There's only one valid pd_checksum value for a given
> input page.
I updated the comment to:
/*
* Any single modification of the checksum could just end up being
* valid again, due to e.g. corrupt_header changing the data in a way
* that'd result in the "corrupted" checksum, or the checksum already
* being invalid. Retry in that, unlikely, case.
*/
> > + /*
> > + * The underlying IO actually completed OK, and thus the "invalid"
> > + * portion of the IOV actually contains valid data. That can hide
> > + * a lot of problems, e.g. if we were to wrongly mark a buffer,
> > + * that wasn't read according to the shortened-read, IO as valid,
> > + * the contents would look valid and we might miss a bug.
>
> Minimally s/read, IO/read IO,/ but I'd edit a bit further:
>
> * a lot of problems, e.g. if we were to wrongly mark-valid a
> * buffer that wasn't read according to the shortened-read IO, the
> * contents would look valid and we might miss a bug.
Adopted.
> > Subject: [PATCH v2.15 05/18] md: Add comment & assert to buffer-zeroing path
> > in md[start]readv()
>
> > The zero_damaged_pages path is incomplete, as as missing segments are not
>
> s/as as/as/
>
> > For now, put an Assert(false) comments documenting this choice into mdreadv()
>
> s/comments/and comments/
>
> > + * For PG 18, we are putting an Assert(false) in into
> > + * mdreadv() (triggering failures in assertion-enabled builds,
>
> s/in into/in/
> > Subject: [PATCH v2.15 06/18] aio: comment polishing
>
> > + * - Partial reads need to be handle by the caller re-issuing IO for the
> > + * unread blocks
>
> s/handle/handled/
All adopted. I'm sorry that you had to see so much of tiredness-enhanced
dyslexia :(.
Greetings,
Andres Freund
^ permalink raw reply [nested|flat] 153+ messages in thread
* Re: AIO v2.5
2025-03-04 19:00 AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-10 18:23 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-11 19:41 ` Re: AIO v2.5 Noah Misch <noah@leadboat.com>
2025-03-11 23:55 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-12 03:57 ` Re: AIO v2.5 Noah Misch <noah@leadboat.com>
2025-03-12 17:06 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-12 23:23 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-13 10:53 ` Re: AIO v2.5 Antonin Houska <ah@cybertec.at>
2025-03-14 19:43 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-17 20:00 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-18 20:12 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-21 01:58 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-25 01:18 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-27 01:07 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-27 20:58 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-28 03:22 ` Re: AIO v2.5 Noah Misch <noah@leadboat.com>
2025-03-28 12:54 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-29 03:35 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-29 13:41 ` Re: AIO v2.5 Noah Misch <noah@leadboat.com>
2025-03-29 14:48 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-29 18:25 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-29 21:29 ` Re: AIO v2.5 Noah Misch <noah@leadboat.com>
2025-03-30 23:46 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-04-01 00:41 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-04-01 15:11 ` Re: AIO v2.5 Noah Misch <noah@leadboat.com>
2025-04-01 15:55 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
@ 2025-04-01 16:07 ` Noah Misch <noah@leadboat.com>
2025-04-01 16:51 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
1 sibling, 1 reply; 153+ messages in thread
From: Noah Misch @ 2025-04-01 16:07 UTC (permalink / raw)
To: Andres Freund <andres@anarazel.de>; +Cc: pgsql-hackers@postgresql.org, Thomas Munro <thomas.munro@gmail.com>; Heikki Linnakangas <hlinnaka@iki.fi>; Robert Haas <robertmhaas@gmail.com>; Jakub Wartak <jakub.wartak@enterprisedb.com>; Jelte Fennema-Nio <postgres@jeltef.nl>; Antonin Houska <ah@cybertec.at>
On Tue, Apr 01, 2025 at 11:55:20AM -0400, Andres Freund wrote:
> On 2025-04-01 08:11:59 -0700, Noah Misch wrote:
> > On Mon, Mar 31, 2025 at 08:41:39PM -0400, Andres Freund wrote:
> I haven't yet pushed the changes, but will work on that in the afternoon.
>
> I plan to afterwards close the CF entry and will eventually create a new one
> for write support, although probably only rebasing onto
> https://postgr.es/m/stj36ea6yyhoxtqkhpieia2z4krnam7qyetc57rfezgk4zgapf%40gcnactj4z56m
> and addressing some of the locking issues.
Sounds good.
> WRT the locking issues, I've been wondering whether we could make
> LWLockWaitForVar() work that purpose, but I doubt it's the right approach.
> Probably better to get rid of the LWLock*Var functions and go for the approach
> I had in v1, namely a version of LWLockAcquire() with a callback that gets
> called between LWLockQueueSelf() and PGSemaphoreLock(), which can cause the
> lock acquisition to abort.
What are the best thing(s) to read to understand the locking issues?
> > > +# Verify checksum handling when creating database from an invalid database.
> > > +# This also serves as a minimal check that cross-database IO is handled
> > > +# reasonably.
> >
> > To me, "invalid database" is a term of art from the message "cannot connect to
> > invalid database". Hence, I would change "invalid database" to "database w/
> > invalid block" or similar, here and below. (Alternatively, just delete "from
> > an invalid database". It's clear from the context.)
>
> Yea, I agree, this is easy to misunderstand when stepping back. I went for "with
> an invalid block".
Sounds good.
> > > + if (corrupt_checksum)
> > > + {
> > > + bool successfully_corrupted = 0;
> > > +
> > > + /*
> > > + * Any single modification of the checksum could just end up being
> > > + * valid again. To be sure
> > > + */
> >
> > Unfinished sentence.
>
> Oops. See below.
>
>
> > That said, I'm not following why we'd need this loop. If this test code
> > were changing the input to the checksum, it's true that an input bit flip
> > might reach the same pd_checksum. The test case is changing pd_checksum,
> > not the input bits.
>
> We might be changing the input, due to the zero/corrupt_header options. Or we
> might be called on a page that is *already* corrupted. I did encounter that
> situation once while writing tests, where the tests only passed if I made the
> + 1 a + 2. Which was, uh, rather confusing and left me feel like I was cursed
> that day.
Got it.
> > I don't see how changing pd_checksum could leave the
> > page still valid. There's only one valid pd_checksum value for a given
> > input page.
>
> I updated the comment to:
> /*
> * Any single modification of the checksum could just end up being
> * valid again, due to e.g. corrupt_header changing the data in a way
> * that'd result in the "corrupted" checksum, or the checksum already
> * being invalid. Retry in that, unlikely, case.
> */
Works for me.
^ permalink raw reply [nested|flat] 153+ messages in thread
* Re: AIO v2.5
2025-03-04 19:00 AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-10 18:23 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-11 19:41 ` Re: AIO v2.5 Noah Misch <noah@leadboat.com>
2025-03-11 23:55 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-12 03:57 ` Re: AIO v2.5 Noah Misch <noah@leadboat.com>
2025-03-12 17:06 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-12 23:23 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-13 10:53 ` Re: AIO v2.5 Antonin Houska <ah@cybertec.at>
2025-03-14 19:43 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-17 20:00 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-18 20:12 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-21 01:58 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-25 01:18 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-27 01:07 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-27 20:58 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-28 03:22 ` Re: AIO v2.5 Noah Misch <noah@leadboat.com>
2025-03-28 12:54 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-29 03:35 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-29 13:41 ` Re: AIO v2.5 Noah Misch <noah@leadboat.com>
2025-03-29 14:48 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-29 18:25 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-29 21:29 ` Re: AIO v2.5 Noah Misch <noah@leadboat.com>
2025-03-30 23:46 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-04-01 00:41 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-04-01 15:11 ` Re: AIO v2.5 Noah Misch <noah@leadboat.com>
2025-04-01 15:55 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-04-01 16:07 ` Re: AIO v2.5 Noah Misch <noah@leadboat.com>
@ 2025-04-01 16:51 ` Andres Freund <andres@anarazel.de>
0 siblings, 0 replies; 153+ messages in thread
From: Andres Freund @ 2025-04-01 16:51 UTC (permalink / raw)
To: Noah Misch <noah@leadboat.com>; +Cc: pgsql-hackers@postgresql.org, Thomas Munro <thomas.munro@gmail.com>; Heikki Linnakangas <hlinnaka@iki.fi>; Robert Haas <robertmhaas@gmail.com>; Jakub Wartak <jakub.wartak@enterprisedb.com>; Jelte Fennema-Nio <postgres@jeltef.nl>; Antonin Houska <ah@cybertec.at>
Hi,
On 2025-04-01 09:07:27 -0700, Noah Misch wrote:
> On Tue, Apr 01, 2025 at 11:55:20AM -0400, Andres Freund wrote:
> > WRT the locking issues, I've been wondering whether we could make
> > LWLockWaitForVar() work that purpose, but I doubt it's the right approach.
> > Probably better to get rid of the LWLock*Var functions and go for the approach
> > I had in v1, namely a version of LWLockAcquire() with a callback that gets
> > called between LWLockQueueSelf() and PGSemaphoreLock(), which can cause the
> > lock acquisition to abort.
>
> What are the best thing(s) to read to understand the locking issues?
Unfortunately I think it's our discussion from a few days/weeks ago.
The problem basically is that functions like LockBuffer(EXCLUSIVE) need to be able
to non-racily
a) wait for in-fligth IOs
b) acquire the content lock
If you just do it naively like this:
else if (mode == BUFFER_LOCK_EXCLUSIVE)
{
if (pg_atomic_read_u32(&buf->state) &_IO_IN_PROGRESS)
WaitIO(buf);
LWLockAcquire(content_lock, LW_EXCLUSIVE);
}
you obviously could have another backend start new IO between the WaitIO() and
the LWLockAcquire(). If that other backend then doesn't consume the
completion of that IO, the current backend could end up endlessly waiting for
the IO. I don't see a way to avoid with narrow changes just to LockBuffer().
We need some infrastructure that allows to avoid that issue. One approach
could be to integrate more tightly with lwlock.c. If
1) anyone starting IO were to wake up all waiters for the LWLock
2) The waiting side checked that there is no IO in progress *after*
LWLockQueueSelf(), but before PGSemaphoreLock()
The backend doing LockBuffer() would be guaranteed to have the chance to wait
for the IO, rather than the lwlock.
But there might be better approaches.
I'm not really convinced that using generic lwlocks for buffer locking is the
best idea. There's just too many special things about buffers. E.g. we have
rather massive NUMA scalability issues due to the amount of lock traffic from
buffer header and content lock atomic operations, particuly on things like the
uppermost levels of a btree. I've played with ideas like super-pinning and
locking btree root pages, which move all the overhead to the side that wants
to exclusively lock such a page - but that doesn't really make sense for
lwlocks in general.
Greetings,
Andres Freund
^ permalink raw reply [nested|flat] 153+ messages in thread
* Re: AIO v2.5
2025-03-04 19:00 AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-10 18:23 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-11 19:41 ` Re: AIO v2.5 Noah Misch <noah@leadboat.com>
2025-03-11 23:55 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-12 03:57 ` Re: AIO v2.5 Noah Misch <noah@leadboat.com>
2025-03-12 17:06 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-12 23:23 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-13 10:53 ` Re: AIO v2.5 Antonin Houska <ah@cybertec.at>
2025-03-14 19:43 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-17 20:00 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-18 20:12 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-21 01:58 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-25 01:18 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-27 01:07 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-27 20:58 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-28 03:22 ` Re: AIO v2.5 Noah Misch <noah@leadboat.com>
2025-03-28 12:54 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-29 03:35 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-29 13:41 ` Re: AIO v2.5 Noah Misch <noah@leadboat.com>
2025-03-29 14:48 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-29 18:25 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-29 21:29 ` Re: AIO v2.5 Noah Misch <noah@leadboat.com>
2025-03-30 23:46 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-04-01 00:41 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-04-01 15:11 ` Re: AIO v2.5 Noah Misch <noah@leadboat.com>
2025-04-01 15:55 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
@ 2025-04-01 21:47 ` Andres Freund <andres@anarazel.de>
2025-04-01 22:25 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-04-03 18:19 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
1 sibling, 2 replies; 153+ messages in thread
From: Andres Freund @ 2025-04-01 21:47 UTC (permalink / raw)
To: Noah Misch <noah@leadboat.com>; +Cc: pgsql-hackers@postgresql.org, Thomas Munro <thomas.munro@gmail.com>; Heikki Linnakangas <hlinnaka@iki.fi>; Robert Haas <robertmhaas@gmail.com>; Jakub Wartak <jakub.wartak@enterprisedb.com>; Jelte Fennema-Nio <postgres@jeltef.nl>; Antonin Houska <ah@cybertec.at>
Hi,
On 2025-04-01 11:55:20 -0400, Andres Freund wrote:
> I haven't yet pushed the changes, but will work on that in the afternoon.
There are three different types of failures in the test_aio test so far:
1) TEMP_CONFIG
See https://postgr.es/m/zh5u22wbpcyfw2ddl3lsvmsxf4yvsrvgxqwwmfjddc4c2khsgp%40gfysyjsaelr5
2) Failure on at least some windows BF machines:
https://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=drongo&dt=2025-04-01%2020%3A15%3A19
https://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=fairywren&dt=2025-04-01%2019%3A03%3A07
Afaict the error is independent of AIO, instead just related CREATE DATABASE
... STRATEGY wal_log failing on windows. In contrast to dropdb(), which does
/*
* Force a checkpoint to make sure the checkpointer has received the
* message sent by ForgetDatabaseSyncRequests.
*/
RequestCheckpoint(CHECKPOINT_IMMEDIATE | CHECKPOINT_FORCE | CHECKPOINT_WAIT);
/* Close all smgr fds in all backends. */
WaitForProcSignalBarrier(EmitProcSignalBarrier(PROCSIGNAL_BARRIER_SMGRRELEASE));
createdb_failure_callback() does no such thing. But it's rather likely that
we, bgwriter, checkpointer (and now IO workers) have files open for the target
database.
Note that the test is failing even with "io_method=sync", which obviously
doesn't use IO workers, so it's not related to that.
It's probably not a good idea to blockingly request a checkpoint and a barrier
inside a PG_TRY/PG_ENSURE_ERROR_CLEANUP() though, so this would need a bit
more rearchitecting.
I think I'm just going to make the test more lenient by not insisting that the
error is the first thing on psql's stderr.
3) Some subtests fail if RELCACHE_FORCE_RELEASE and CATCACHE_FORCE_RELEASE are defined:
https://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=prion&dt=2025-04-01%2019%3A23%3A07
# +++ tap check in src/test/modules/test_aio +++
# Failed test 'worker: batch_start() leak & cleanup in implicit xact: expected stderr'
# at t/001_aio.pl line 318.
# 'psql:<stdin>:4: ERROR: starting batch while batch already in progress'
# doesn't match '(?^:open AIO batch at end)'
The problem is basically that the test intentionally forgets to exit batchmode
- normally that would trigger an error at the end of the transaction, which
the test verifies. However, with RELCACHE_FORCE_RELEASE and
CATCACHE_FORCE_RELEASE defined, we get other code entering batchmode and
erroring out because batchmode isn't allowed to be entered recursively.
#0 pgaio_enter_batchmode () at ../../../../../home/andres/src/postgresql/src/backend/storage/aio/aio.c:997
#1 0x000055ec847959bf in read_stream_look_ahead (stream=0x55ecbcfda098)
at ../../../../../home/andres/src/postgresql/src/backend/storage/aio/read_stream.c:438
#2 0x000055ec84796514 in read_stream_next_buffer (stream=0x55ecbcfda098, per_buffer_data=0x0)
at ../../../../../home/andres/src/postgresql/src/backend/storage/aio/read_stream.c:890
#3 0x000055ec8432520b in heap_fetch_next_buffer (scan=0x55ecbcfd1c00, dir=ForwardScanDirection)
at ../../../../../home/andres/src/postgresql/src/backend/access/heap/heapam.c:679
#4 0x000055ec843259a4 in heapgettup_pagemode (scan=0x55ecbcfd1c00, dir=ForwardScanDirection, nkeys=1, key=0x55ecbcfd1620)
at ../../../../../home/andres/src/postgresql/src/backend/access/heap/heapam.c:1041
#5 0x000055ec843263ba in heap_getnextslot (sscan=0x55ecbcfd1c00, direction=ForwardScanDirection, slot=0x55ecbcfd0e18)
at ../../../../../home/andres/src/postgresql/src/backend/access/heap/heapam.c:1420
#6 0x000055ec8434ebe5 in table_scan_getnextslot (sscan=0x55ecbcfd1c00, direction=ForwardScanDirection, slot=0x55ecbcfd0e18)
at ../../../../../home/andres/src/postgresql/src/include/access/tableam.h:1041
#7 0x000055ec8434f786 in systable_getnext (sysscan=0x55ecbcfd8088) at ../../../../../home/andres/src/postgresql/src/backend/access/index/genam.c:541
#8 0x000055ec849c784a in SearchCatCacheMiss (cache=0x55ecbcf81000, nkeys=1, hashValue=3830081846, hashIndex=2, v1=403, v2=0, v3=0, v4=0)
at ../../../../../home/andres/src/postgresql/src/backend/utils/cache/catcache.c:1543
#9 0x000055ec849c76f9 in SearchCatCacheInternal (cache=0x55ecbcf81000, nkeys=1, v1=403, v2=0, v3=0, v4=0)
at ../../../../../home/andres/src/postgresql/src/backend/utils/cache/catcache.c:1464
#10 0x000055ec849c73ec in SearchCatCache1 (cache=0x55ecbcf81000, v1=403) at ../../../../../home/andres/src/postgresql/src/backend/utils/cache/catcache.c:1332
#11 0x000055ec849e5ae3 in SearchSysCache1 (cacheId=2, key1=403) at ../../../../../home/andres/src/postgresql/src/backend/utils/cache/syscache.c:228
#12 0x000055ec849d8c78 in RelationInitIndexAccessInfo (relation=0x7f6a85901c20)
at ../../../../../home/andres/src/postgresql/src/backend/utils/cache/relcache.c:1456
#13 0x000055ec849d8471 in RelationBuildDesc (targetRelId=2703, insertIt=true)
at ../../../../../home/andres/src/postgresql/src/backend/utils/cache/relcache.c:1201
#14 0x000055ec849d9e9c in RelationIdGetRelation (relationId=2703) at ../../../../../home/andres/src/postgresql/src/backend/utils/cache/relcache.c:2100
#15 0x000055ec842d219f in relation_open (relationId=2703, lockmode=1) at ../../../../../home/andres/src/postgresql/src/backend/access/common/relation.c:58
#16 0x000055ec8435043c in index_open (relationId=2703, lockmode=1) at ../../../../../home/andres/src/postgresql/src/backend/access/index/indexam.c:137
#17 0x000055ec8434f2f9 in systable_beginscan (heapRelation=0x7f6a859353a8, indexId=2703, indexOK=true, snapshot=0x0, nkeys=1, key=0x7ffc11aa7c90)
at ../../../../../home/andres/src/postgresql/src/backend/access/index/genam.c:400
#18 0x000055ec849c782c in SearchCatCacheMiss (cache=0x55ecbcfa0e80, nkeys=1, hashValue=2659955452, hashIndex=60, v1=2278, v2=0, v3=0, v4=0)
at ../../../../../home/andres/src/postgresql/src/backend/utils/cache/catcache.c:1533
#19 0x000055ec849c76f9 in SearchCatCacheInternal (cache=0x55ecbcfa0e80, nkeys=1, v1=2278, v2=0, v3=0, v4=0)
at ../../../../../home/andres/src/postgresql/src/backend/utils/cache/catcache.c:1464
#20 0x000055ec849c73ec in SearchCatCache1 (cache=0x55ecbcfa0e80, v1=2278) at ../../../../../home/andres/src/postgresql/src/backend/utils/cache/catcache.c:1332
#21 0x000055ec849e5ae3 in SearchSysCache1 (cacheId=82, key1=2278) at ../../../../../home/andres/src/postgresql/src/backend/utils/cache/syscache.c:228
#22 0x000055ec849d0375 in getTypeOutputInfo (type=2278, typOutput=0x55ecbcfd15d0, typIsVarlena=0x55ecbcfd15d8)
at ../../../../../home/andres/src/postgresql/src/backend/utils/cache/lsyscache.c:2995
#23 0x000055ec842d1a57 in printtup_prepare_info (myState=0x55ecbcfcec00, typeinfo=0x55ecbcfd0588, numAttrs=1)
at ../../../../../home/andres/src/postgresql/src/backend/access/common/printtup.c:277
#24 0x000055ec842d1ba6 in printtup (slot=0x55ecbcfd0b28, self=0x55ecbcfcec00)
at ../../../../../home/andres/src/postgresql/src/backend/access/common/printtup.c:315
#25 0x000055ec84541f54 in ExecutePlan (queryDesc=0x55ecbced4290, operation=CMD_SELECT, sendTuples=true, numberTuples=0, direction=ForwardScanDirection,
dest=0x55ecbcfcec00) at ../../../../../home/andres/src/postgresql/src/backend/executor/execMain.c:1814
I don't really have a good idea how to deal with that yet.
Greetings,
Andres
^ permalink raw reply [nested|flat] 153+ messages in thread
* Re: AIO v2.5
2025-03-04 19:00 AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-10 18:23 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-11 19:41 ` Re: AIO v2.5 Noah Misch <noah@leadboat.com>
2025-03-11 23:55 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-12 03:57 ` Re: AIO v2.5 Noah Misch <noah@leadboat.com>
2025-03-12 17:06 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-12 23:23 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-13 10:53 ` Re: AIO v2.5 Antonin Houska <ah@cybertec.at>
2025-03-14 19:43 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-17 20:00 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-18 20:12 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-21 01:58 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-25 01:18 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-27 01:07 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-27 20:58 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-28 03:22 ` Re: AIO v2.5 Noah Misch <noah@leadboat.com>
2025-03-28 12:54 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-29 03:35 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-29 13:41 ` Re: AIO v2.5 Noah Misch <noah@leadboat.com>
2025-03-29 14:48 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-29 18:25 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-29 21:29 ` Re: AIO v2.5 Noah Misch <noah@leadboat.com>
2025-03-30 23:46 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-04-01 00:41 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-04-01 15:11 ` Re: AIO v2.5 Noah Misch <noah@leadboat.com>
2025-04-01 15:55 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-04-01 21:47 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
@ 2025-04-01 22:25 ` Andres Freund <andres@anarazel.de>
2025-04-02 00:13 ` Re: AIO v2.5 Noah Misch <noah@leadboat.com>
1 sibling, 1 reply; 153+ messages in thread
From: Andres Freund @ 2025-04-01 22:25 UTC (permalink / raw)
To: Noah Misch <noah@leadboat.com>; +Cc: pgsql-hackers@postgresql.org, Thomas Munro <thomas.munro@gmail.com>; Heikki Linnakangas <hlinnaka@iki.fi>; Robert Haas <robertmhaas@gmail.com>; Jakub Wartak <jakub.wartak@enterprisedb.com>; Jelte Fennema-Nio <postgres@jeltef.nl>; Antonin Houska <ah@cybertec.at>
Hi,
On 2025-04-01 17:47:51 -0400, Andres Freund wrote:
> 3) Some subtests fail if RELCACHE_FORCE_RELEASE and CATCACHE_FORCE_RELEASE are defined:
>
> https://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=prion&dt=2025-04-01%2019%3A23%3A07
>
> # +++ tap check in src/test/modules/test_aio +++
>
> # Failed test 'worker: batch_start() leak & cleanup in implicit xact: expected stderr'
> # at t/001_aio.pl line 318.
> # 'psql:<stdin>:4: ERROR: starting batch while batch already in progress'
> # doesn't match '(?^:open AIO batch at end)'
>
>
> The problem is basically that the test intentionally forgets to exit batchmode
> - normally that would trigger an error at the end of the transaction, which
> the test verifies. However, with RELCACHE_FORCE_RELEASE and
> CATCACHE_FORCE_RELEASE defined, we get other code entering batchmode and
> erroring out because batchmode isn't allowed to be entered recursively.
>
>
> #0 pgaio_enter_batchmode () at ../../../../../home/andres/src/postgresql/src/backend/storage/aio/aio.c:997
> #1 0x000055ec847959bf in read_stream_look_ahead (stream=0x55ecbcfda098)
> at ../../../../../home/andres/src/postgresql/src/backend/storage/aio/read_stream.c:438
> #2 0x000055ec84796514 in read_stream_next_buffer (stream=0x55ecbcfda098, per_buffer_data=0x0)
> at ../../../../../home/andres/src/postgresql/src/backend/storage/aio/read_stream.c:890
> #3 0x000055ec8432520b in heap_fetch_next_buffer (scan=0x55ecbcfd1c00, dir=ForwardScanDirection)
> at ../../../../../home/andres/src/postgresql/src/backend/access/heap/heapam.c:679
> #4 0x000055ec843259a4 in heapgettup_pagemode (scan=0x55ecbcfd1c00, dir=ForwardScanDirection, nkeys=1, key=0x55ecbcfd1620)
> at ../../../../../home/andres/src/postgresql/src/backend/access/heap/heapam.c:1041
> #5 0x000055ec843263ba in heap_getnextslot (sscan=0x55ecbcfd1c00, direction=ForwardScanDirection, slot=0x55ecbcfd0e18)
> at ../../../../../home/andres/src/postgresql/src/backend/access/heap/heapam.c:1420
> #6 0x000055ec8434ebe5 in table_scan_getnextslot (sscan=0x55ecbcfd1c00, direction=ForwardScanDirection, slot=0x55ecbcfd0e18)
> at ../../../../../home/andres/src/postgresql/src/include/access/tableam.h:1041
> #7 0x000055ec8434f786 in systable_getnext (sysscan=0x55ecbcfd8088) at ../../../../../home/andres/src/postgresql/src/backend/access/index/genam.c:541
> #8 0x000055ec849c784a in SearchCatCacheMiss (cache=0x55ecbcf81000, nkeys=1, hashValue=3830081846, hashIndex=2, v1=403, v2=0, v3=0, v4=0)
> at ../../../../../home/andres/src/postgresql/src/backend/utils/cache/catcache.c:1543
> #9 0x000055ec849c76f9 in SearchCatCacheInternal (cache=0x55ecbcf81000, nkeys=1, v1=403, v2=0, v3=0, v4=0)
> at ../../../../../home/andres/src/postgresql/src/backend/utils/cache/catcache.c:1464
> #10 0x000055ec849c73ec in SearchCatCache1 (cache=0x55ecbcf81000, v1=403) at ../../../../../home/andres/src/postgresql/src/backend/utils/cache/catcache.c:1332
> #11 0x000055ec849e5ae3 in SearchSysCache1 (cacheId=2, key1=403) at ../../../../../home/andres/src/postgresql/src/backend/utils/cache/syscache.c:228
> #12 0x000055ec849d8c78 in RelationInitIndexAccessInfo (relation=0x7f6a85901c20)
> at ../../../../../home/andres/src/postgresql/src/backend/utils/cache/relcache.c:1456
> #13 0x000055ec849d8471 in RelationBuildDesc (targetRelId=2703, insertIt=true)
> at ../../../../../home/andres/src/postgresql/src/backend/utils/cache/relcache.c:1201
> #14 0x000055ec849d9e9c in RelationIdGetRelation (relationId=2703) at ../../../../../home/andres/src/postgresql/src/backend/utils/cache/relcache.c:2100
> #15 0x000055ec842d219f in relation_open (relationId=2703, lockmode=1) at ../../../../../home/andres/src/postgresql/src/backend/access/common/relation.c:58
> #16 0x000055ec8435043c in index_open (relationId=2703, lockmode=1) at ../../../../../home/andres/src/postgresql/src/backend/access/index/indexam.c:137
> #17 0x000055ec8434f2f9 in systable_beginscan (heapRelation=0x7f6a859353a8, indexId=2703, indexOK=true, snapshot=0x0, nkeys=1, key=0x7ffc11aa7c90)
> at ../../../../../home/andres/src/postgresql/src/backend/access/index/genam.c:400
> #18 0x000055ec849c782c in SearchCatCacheMiss (cache=0x55ecbcfa0e80, nkeys=1, hashValue=2659955452, hashIndex=60, v1=2278, v2=0, v3=0, v4=0)
> at ../../../../../home/andres/src/postgresql/src/backend/utils/cache/catcache.c:1533
> #19 0x000055ec849c76f9 in SearchCatCacheInternal (cache=0x55ecbcfa0e80, nkeys=1, v1=2278, v2=0, v3=0, v4=0)
> at ../../../../../home/andres/src/postgresql/src/backend/utils/cache/catcache.c:1464
> #20 0x000055ec849c73ec in SearchCatCache1 (cache=0x55ecbcfa0e80, v1=2278) at ../../../../../home/andres/src/postgresql/src/backend/utils/cache/catcache.c:1332
> #21 0x000055ec849e5ae3 in SearchSysCache1 (cacheId=82, key1=2278) at ../../../../../home/andres/src/postgresql/src/backend/utils/cache/syscache.c:228
> #22 0x000055ec849d0375 in getTypeOutputInfo (type=2278, typOutput=0x55ecbcfd15d0, typIsVarlena=0x55ecbcfd15d8)
> at ../../../../../home/andres/src/postgresql/src/backend/utils/cache/lsyscache.c:2995
> #23 0x000055ec842d1a57 in printtup_prepare_info (myState=0x55ecbcfcec00, typeinfo=0x55ecbcfd0588, numAttrs=1)
> at ../../../../../home/andres/src/postgresql/src/backend/access/common/printtup.c:277
> #24 0x000055ec842d1ba6 in printtup (slot=0x55ecbcfd0b28, self=0x55ecbcfcec00)
> at ../../../../../home/andres/src/postgresql/src/backend/access/common/printtup.c:315
> #25 0x000055ec84541f54 in ExecutePlan (queryDesc=0x55ecbced4290, operation=CMD_SELECT, sendTuples=true, numberTuples=0, direction=ForwardScanDirection,
> dest=0x55ecbcfcec00) at ../../../../../home/andres/src/postgresql/src/backend/executor/execMain.c:1814
>
>
> I don't really have a good idea how to deal with that yet.
Hm. Making the query something like
SELECT * FROM (VALUES (NULL), (batch_start()));
avoids the wrong output, because the type lookup happens for the first row
already. But that's pretty magical and probably fragile.
Greetings,
Andres Freund
^ permalink raw reply [nested|flat] 153+ messages in thread
* Re: AIO v2.5
2025-03-04 19:00 AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-10 18:23 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-11 19:41 ` Re: AIO v2.5 Noah Misch <noah@leadboat.com>
2025-03-11 23:55 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-12 03:57 ` Re: AIO v2.5 Noah Misch <noah@leadboat.com>
2025-03-12 17:06 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-12 23:23 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-13 10:53 ` Re: AIO v2.5 Antonin Houska <ah@cybertec.at>
2025-03-14 19:43 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-17 20:00 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-18 20:12 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-21 01:58 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-25 01:18 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-27 01:07 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-27 20:58 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-28 03:22 ` Re: AIO v2.5 Noah Misch <noah@leadboat.com>
2025-03-28 12:54 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-29 03:35 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-29 13:41 ` Re: AIO v2.5 Noah Misch <noah@leadboat.com>
2025-03-29 14:48 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-29 18:25 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-29 21:29 ` Re: AIO v2.5 Noah Misch <noah@leadboat.com>
2025-03-30 23:46 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-04-01 00:41 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-04-01 15:11 ` Re: AIO v2.5 Noah Misch <noah@leadboat.com>
2025-04-01 15:55 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-04-01 21:47 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-04-01 22:25 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
@ 2025-04-02 00:13 ` Noah Misch <noah@leadboat.com>
2025-04-02 11:58 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
0 siblings, 1 reply; 153+ messages in thread
From: Noah Misch @ 2025-04-02 00:13 UTC (permalink / raw)
To: Andres Freund <andres@anarazel.de>; +Cc: pgsql-hackers@postgresql.org, Thomas Munro <thomas.munro@gmail.com>; Heikki Linnakangas <hlinnaka@iki.fi>; Robert Haas <robertmhaas@gmail.com>; Jakub Wartak <jakub.wartak@enterprisedb.com>; Jelte Fennema-Nio <postgres@jeltef.nl>; Antonin Houska <ah@cybertec.at>
On Tue, Apr 01, 2025 at 06:25:28PM -0400, Andres Freund wrote:
> On 2025-04-01 17:47:51 -0400, Andres Freund wrote:
> > 3) Some subtests fail if RELCACHE_FORCE_RELEASE and CATCACHE_FORCE_RELEASE are defined:
> >
> > https://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=prion&dt=2025-04-01%2019%3A23%3A07
> >
> > # +++ tap check in src/test/modules/test_aio +++
> >
> > # Failed test 'worker: batch_start() leak & cleanup in implicit xact: expected stderr'
> > # at t/001_aio.pl line 318.
> > # 'psql:<stdin>:4: ERROR: starting batch while batch already in progress'
> > # doesn't match '(?^:open AIO batch at end)'
> >
> >
> > The problem is basically that the test intentionally forgets to exit batchmode
> > - normally that would trigger an error at the end of the transaction, which
> > the test verifies. However, with RELCACHE_FORCE_RELEASE and
> > CATCACHE_FORCE_RELEASE defined, we get other code entering batchmode and
> > erroring out because batchmode isn't allowed to be entered recursively.
> > I don't really have a good idea how to deal with that yet.
>
> Hm. Making the query something like
>
> SELECT * FROM (VALUES (NULL), (batch_start()));
>
> avoids the wrong output, because the type lookup happens for the first row
> already. But that's pretty magical and probably fragile.
Hmm. Some options:
a. VALUES() trick above. For test code, it's hard to argue with something
that seems to solve it in practice.
b. Encapsulate the test in a PROCEDURE, so perhaps less happens between the
batch_start() and the procedure-managed COMMIT. Maybe less fragile than
(a), maybe more fragile.
c. Move RELCACHE_FORCE_RELEASE and CATCACHE_FORCE_RELEASE to be
GUC-controlled, like how CLOBBER_CACHE_ALWAYS changed into the
debug_discard_caches GUC. Then disable them for relevant parts of
test_aio. This feels best long-term, but it's bigger. I also wanted this
in syscache-update-pruned.spec[1].
d. Have test_aio deduce whether these are set, probably by observing memory
contexts or DEBUG messages. Maybe have every postmaster startup print a
DEBUG message about these settings being enabled. Skip relevant parts of
test_aio. This sounds messy.
Each of those feels defensible to me. I'd probably do (a) or (b) to start.
[1] For that spec, an alternative expected output sufficed. Incidentally,
I'll soon fix that spec flaking on valgrind/skink.
^ permalink raw reply [nested|flat] 153+ messages in thread
* Re: AIO v2.5
2025-03-04 19:00 AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-10 18:23 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-11 19:41 ` Re: AIO v2.5 Noah Misch <noah@leadboat.com>
2025-03-11 23:55 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-12 03:57 ` Re: AIO v2.5 Noah Misch <noah@leadboat.com>
2025-03-12 17:06 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-12 23:23 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-13 10:53 ` Re: AIO v2.5 Antonin Houska <ah@cybertec.at>
2025-03-14 19:43 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-17 20:00 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-18 20:12 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-21 01:58 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-25 01:18 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-27 01:07 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-27 20:58 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-28 03:22 ` Re: AIO v2.5 Noah Misch <noah@leadboat.com>
2025-03-28 12:54 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-29 03:35 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-29 13:41 ` Re: AIO v2.5 Noah Misch <noah@leadboat.com>
2025-03-29 14:48 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-29 18:25 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-29 21:29 ` Re: AIO v2.5 Noah Misch <noah@leadboat.com>
2025-03-30 23:46 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-04-01 00:41 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-04-01 15:11 ` Re: AIO v2.5 Noah Misch <noah@leadboat.com>
2025-04-01 15:55 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-04-01 21:47 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-04-01 22:25 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-04-02 00:13 ` Re: AIO v2.5 Noah Misch <noah@leadboat.com>
@ 2025-04-02 11:58 ` Andres Freund <andres@anarazel.de>
2025-04-03 16:46 ` Re: AIO v2.5 Ranier Vilela <ranier.vf@gmail.com>
2025-04-06 20:00 ` Re: AIO v2.5 Alexander Lakhin <exclusion@gmail.com>
0 siblings, 2 replies; 153+ messages in thread
From: Andres Freund @ 2025-04-02 11:58 UTC (permalink / raw)
To: Noah Misch <noah@leadboat.com>; +Cc: pgsql-hackers@postgresql.org, Thomas Munro <thomas.munro@gmail.com>; Heikki Linnakangas <hlinnaka@iki.fi>; Robert Haas <robertmhaas@gmail.com>; Jakub Wartak <jakub.wartak@enterprisedb.com>; Jelte Fennema-Nio <postgres@jeltef.nl>; Antonin Houska <ah@cybertec.at>
Hi,
I've pushed fixes for 1) and 2) and am working on 3).
On 2025-04-01 17:13:24 -0700, Noah Misch wrote:
> On Tue, Apr 01, 2025 at 06:25:28PM -0400, Andres Freund wrote:
> > On 2025-04-01 17:47:51 -0400, Andres Freund wrote:
> > > 3) Some subtests fail if RELCACHE_FORCE_RELEASE and CATCACHE_FORCE_RELEASE are defined:
> > >
> > > https://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=prion&dt=2025-04-01%2019%3A23%3A07
> > >
> > > # +++ tap check in src/test/modules/test_aio +++
> > >
> > > # Failed test 'worker: batch_start() leak & cleanup in implicit xact: expected stderr'
> > > # at t/001_aio.pl line 318.
> > > # 'psql:<stdin>:4: ERROR: starting batch while batch already in progress'
> > > # doesn't match '(?^:open AIO batch at end)'
> > >
> > >
> > > The problem is basically that the test intentionally forgets to exit batchmode
> > > - normally that would trigger an error at the end of the transaction, which
> > > the test verifies. However, with RELCACHE_FORCE_RELEASE and
> > > CATCACHE_FORCE_RELEASE defined, we get other code entering batchmode and
> > > erroring out because batchmode isn't allowed to be entered recursively.
>
> > > I don't really have a good idea how to deal with that yet.
> >
> > Hm. Making the query something like
> >
> > SELECT * FROM (VALUES (NULL), (batch_start()));
> >
> > avoids the wrong output, because the type lookup happens for the first row
> > already. But that's pretty magical and probably fragile.
>
> Hmm. Some options:
>
> a. VALUES() trick above. For test code, it's hard to argue with something
> that seems to solve it in practice.
I think I'll go for a slightly nicer version of that, namely
SELECT WHERE batch_start() IS NULL
I think that ends up the least verbose of the ideas we've been discussing.
> c. Move RELCACHE_FORCE_RELEASE and CATCACHE_FORCE_RELEASE to be
> GUC-controlled, like how CLOBBER_CACHE_ALWAYS changed into the
> debug_discard_caches GUC. Then disable them for relevant parts of
> test_aio. This feels best long-term, but it's bigger. I also wanted this
> in syscache-update-pruned.spec[1].
Yea, that'd probably be a good thing medium-term.
Greetings,
Andres Freund
^ permalink raw reply [nested|flat] 153+ messages in thread
* Re: AIO v2.5
2025-03-04 19:00 AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-10 18:23 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-11 19:41 ` Re: AIO v2.5 Noah Misch <noah@leadboat.com>
2025-03-11 23:55 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-12 03:57 ` Re: AIO v2.5 Noah Misch <noah@leadboat.com>
2025-03-12 17:06 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-12 23:23 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-13 10:53 ` Re: AIO v2.5 Antonin Houska <ah@cybertec.at>
2025-03-14 19:43 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-17 20:00 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-18 20:12 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-21 01:58 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-25 01:18 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-27 01:07 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-27 20:58 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-28 03:22 ` Re: AIO v2.5 Noah Misch <noah@leadboat.com>
2025-03-28 12:54 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-29 03:35 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-29 13:41 ` Re: AIO v2.5 Noah Misch <noah@leadboat.com>
2025-03-29 14:48 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-29 18:25 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-29 21:29 ` Re: AIO v2.5 Noah Misch <noah@leadboat.com>
2025-03-30 23:46 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-04-01 00:41 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-04-01 15:11 ` Re: AIO v2.5 Noah Misch <noah@leadboat.com>
2025-04-01 15:55 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-04-01 21:47 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-04-01 22:25 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-04-02 00:13 ` Re: AIO v2.5 Noah Misch <noah@leadboat.com>
2025-04-02 11:58 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
@ 2025-04-03 16:46 ` Ranier Vilela <ranier.vf@gmail.com>
2025-04-03 18:35 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
1 sibling, 1 reply; 153+ messages in thread
From: Ranier Vilela @ 2025-04-03 16:46 UTC (permalink / raw)
To: Andres Freund <andres@anarazel.de>; +Cc: Noah Misch <noah@leadboat.com>; pgsql-hackers@postgresql.org, Thomas Munro <thomas.munro@gmail.com>; Heikki Linnakangas <hlinnaka@iki.fi>; Robert Haas <robertmhaas@gmail.com>; Jakub Wartak <jakub.wartak@enterprisedb.com>; Jelte Fennema-Nio <postgres@jeltef.nl>; Antonin Houska <ah@cybertec.at>
Hi.
Em qua., 2 de abr. de 2025 Ã s 08:58, Andres Freund <andres@anarazel.de>
escreveu:
> Hi,
>
> I've pushed fixes for 1) and 2) and am working on 3).
>
Coverity has one report about this.
CID 1596092: (#1 of 1): Uninitialized scalar variable (UNINIT)
13. uninit_use_in_call: Using uninitialized value result_one. Field
result_one.result is uninitialized when calling pgaio_result_report.
Below not is a fix, but some suggestion:
diff --git a/src/backend/storage/buffer/bufmgr.c
b/src/backend/storage/buffer/bufmgr.c
index 1c37d7dfe2..b0f9ce452c 100644
--- a/src/backend/storage/buffer/bufmgr.c
+++ b/src/backend/storage/buffer/bufmgr.c
@@ -6786,6 +6786,8 @@ buffer_readv_encode_error(PgAioResult *result,
else
result->status = PGAIO_RS_WARNING;
+ result->result = 0;
+
/*
* The encoding is complicated enough to warrant cross-checking it against
* the decode function.
@@ -6868,8 +6870,6 @@ buffer_readv_complete_one(PgAioTargetData *td, uint8
buf_off, Buffer buffer,
/* Check for garbage data. */
if (!failed)
{
- PgAioResult result_one;
-
if (!PageIsVerified((Page) bufdata, tag.blockNum, piv_flags,
failed_checksum))
{
@@ -6904,6 +6904,8 @@ buffer_readv_complete_one(PgAioTargetData *td, uint8
buf_off, Buffer buffer,
*/
if (*buffer_invalid || *failed_checksum || *zeroed_buffer)
{
+ PgAioResult result_one;
+
buffer_readv_encode_error(&result_one, is_temp,
*zeroed_buffer,
*ignored_checksum,
1. I couldn't find the correct value to initialize the *result* field.
2. result_one can be reduced scope.
best regards,
Ranier Vilela
^ permalink raw reply [nested|flat] 153+ messages in thread
* Re: AIO v2.5
2025-03-04 19:00 AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-10 18:23 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-11 19:41 ` Re: AIO v2.5 Noah Misch <noah@leadboat.com>
2025-03-11 23:55 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-12 03:57 ` Re: AIO v2.5 Noah Misch <noah@leadboat.com>
2025-03-12 17:06 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-12 23:23 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-13 10:53 ` Re: AIO v2.5 Antonin Houska <ah@cybertec.at>
2025-03-14 19:43 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-17 20:00 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-18 20:12 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-21 01:58 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-25 01:18 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-27 01:07 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-27 20:58 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-28 03:22 ` Re: AIO v2.5 Noah Misch <noah@leadboat.com>
2025-03-28 12:54 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-29 03:35 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-29 13:41 ` Re: AIO v2.5 Noah Misch <noah@leadboat.com>
2025-03-29 14:48 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-29 18:25 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-29 21:29 ` Re: AIO v2.5 Noah Misch <noah@leadboat.com>
2025-03-30 23:46 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-04-01 00:41 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-04-01 15:11 ` Re: AIO v2.5 Noah Misch <noah@leadboat.com>
2025-04-01 15:55 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-04-01 21:47 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-04-01 22:25 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-04-02 00:13 ` Re: AIO v2.5 Noah Misch <noah@leadboat.com>
2025-04-02 11:58 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-04-03 16:46 ` Re: AIO v2.5 Ranier Vilela <ranier.vf@gmail.com>
@ 2025-04-03 18:35 ` Andres Freund <andres@anarazel.de>
2025-04-03 19:16 ` Re: AIO v2.5 Ranier Vilela <ranier.vf@gmail.com>
0 siblings, 1 reply; 153+ messages in thread
From: Andres Freund @ 2025-04-03 18:35 UTC (permalink / raw)
To: Ranier Vilela <ranier.vf@gmail.com>; +Cc: Noah Misch <noah@leadboat.com>; pgsql-hackers@postgresql.org, Thomas Munro <thomas.munro@gmail.com>; Heikki Linnakangas <hlinnaka@iki.fi>; Robert Haas <robertmhaas@gmail.com>; Jakub Wartak <jakub.wartak@enterprisedb.com>; Jelte Fennema-Nio <postgres@jeltef.nl>; Antonin Houska <ah@cybertec.at>
Hi,
On 2025-04-03 13:46:39 -0300, Ranier Vilela wrote:
> Em qua., 2 de abr. de 2025 Ã s 08:58, Andres Freund <andres@anarazel.de>
> escreveu:
>
> > Hi,
> >
> > I've pushed fixes for 1) and 2) and am working on 3).
> >
> Coverity has one report about this.
>
> CID 1596092: (#1 of 1): Uninitialized scalar variable (UNINIT)
> 13. uninit_use_in_call: Using uninitialized value result_one. Field
> result_one.result is uninitialized when calling pgaio_result_report.
Isn't this a rather silly thing to warn about for coverity? The field isn't
used in pgaio_result_report(). It can't be a particularly rare thing to have
struct fields that aren't always used?
> Below not is a fix, but some suggestion:
>
> diff --git a/src/backend/storage/buffer/bufmgr.c
> b/src/backend/storage/buffer/bufmgr.c
> index 1c37d7dfe2..b0f9ce452c 100644
> --- a/src/backend/storage/buffer/bufmgr.c
> +++ b/src/backend/storage/buffer/bufmgr.c
> @@ -6786,6 +6786,8 @@ buffer_readv_encode_error(PgAioResult *result,
> else
> result->status = PGAIO_RS_WARNING;
>
> + result->result = 0;
> +
That'd be completely wrong - and the tests indeed fail if you do that. The
read might succeed with a warning (e.g. due to zero_damaged_pages) in which
case the result still carries important information about how many blocks were
successfully read.
> /*
> * The encoding is complicated enough to warrant cross-checking it against
> * the decode function.
> @@ -6868,8 +6870,6 @@ buffer_readv_complete_one(PgAioTargetData *td, uint8
> buf_off, Buffer buffer,
> /* Check for garbage data. */
> if (!failed)
> {
> - PgAioResult result_one;
> -
> if (!PageIsVerified((Page) bufdata, tag.blockNum, piv_flags,
> failed_checksum))
> {
> @@ -6904,6 +6904,8 @@ buffer_readv_complete_one(PgAioTargetData *td, uint8
> buf_off, Buffer buffer,
> */
> if (*buffer_invalid || *failed_checksum || *zeroed_buffer)
> {
> + PgAioResult result_one;
> +
> buffer_readv_encode_error(&result_one, is_temp,
> *zeroed_buffer,
> *ignored_checksum,
>
>
> 1. I couldn't find the correct value to initialize the *result* field.
It is not accessed in this path. I guess we can just zero-initialize
result_one to shut up coverity.
> 2. result_one can be reduced scope.
True.
Greetings,
Andres Freund
^ permalink raw reply [nested|flat] 153+ messages in thread
* Re: AIO v2.5
2025-03-04 19:00 AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-10 18:23 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-11 19:41 ` Re: AIO v2.5 Noah Misch <noah@leadboat.com>
2025-03-11 23:55 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-12 03:57 ` Re: AIO v2.5 Noah Misch <noah@leadboat.com>
2025-03-12 17:06 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-12 23:23 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-13 10:53 ` Re: AIO v2.5 Antonin Houska <ah@cybertec.at>
2025-03-14 19:43 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-17 20:00 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-18 20:12 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-21 01:58 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-25 01:18 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-27 01:07 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-27 20:58 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-28 03:22 ` Re: AIO v2.5 Noah Misch <noah@leadboat.com>
2025-03-28 12:54 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-29 03:35 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-29 13:41 ` Re: AIO v2.5 Noah Misch <noah@leadboat.com>
2025-03-29 14:48 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-29 18:25 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-29 21:29 ` Re: AIO v2.5 Noah Misch <noah@leadboat.com>
2025-03-30 23:46 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-04-01 00:41 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-04-01 15:11 ` Re: AIO v2.5 Noah Misch <noah@leadboat.com>
2025-04-01 15:55 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-04-01 21:47 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-04-01 22:25 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-04-02 00:13 ` Re: AIO v2.5 Noah Misch <noah@leadboat.com>
2025-04-02 11:58 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-04-03 16:46 ` Re: AIO v2.5 Ranier Vilela <ranier.vf@gmail.com>
2025-04-03 18:35 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
@ 2025-04-03 19:16 ` Ranier Vilela <ranier.vf@gmail.com>
2025-04-03 22:30 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
0 siblings, 1 reply; 153+ messages in thread
From: Ranier Vilela @ 2025-04-03 19:16 UTC (permalink / raw)
To: Andres Freund <andres@anarazel.de>; +Cc: Noah Misch <noah@leadboat.com>; pgsql-hackers@postgresql.org, Thomas Munro <thomas.munro@gmail.com>; Heikki Linnakangas <hlinnaka@iki.fi>; Robert Haas <robertmhaas@gmail.com>; Jakub Wartak <jakub.wartak@enterprisedb.com>; Jelte Fennema-Nio <postgres@jeltef.nl>; Antonin Houska <ah@cybertec.at>
Em qui., 3 de abr. de 2025 Ã s 15:35, Andres Freund <andres@anarazel.de>
escreveu:
> Hi,
>
> On 2025-04-03 13:46:39 -0300, Ranier Vilela wrote:
> > Em qua., 2 de abr. de 2025 Ã s 08:58, Andres Freund <andres@anarazel.de>
> > escreveu:
> >
> > > Hi,
> > >
> > > I've pushed fixes for 1) and 2) and am working on 3).
> > >
> > Coverity has one report about this.
> >
> > CID 1596092: (#1 of 1): Uninitialized scalar variable (UNINIT)
> > 13. uninit_use_in_call: Using uninitialized value result_one. Field
> > result_one.result is uninitialized when calling pgaio_result_report.
>
> Isn't this a rather silly thing to warn about for coverity?
Personally, I consider every warning to be important.
> The field isn't
> used in pgaio_result_report(). It can't be a particularly rare thing to
> have
> struct fields that aren't always used?
>
Always considered a risk, someone may start using it.
>
>
> > Below not is a fix, but some suggestion:
> >
> > diff --git a/src/backend/storage/buffer/bufmgr.c
> > b/src/backend/storage/buffer/bufmgr.c
> > index 1c37d7dfe2..b0f9ce452c 100644
> > --- a/src/backend/storage/buffer/bufmgr.c
> > +++ b/src/backend/storage/buffer/bufmgr.c
> > @@ -6786,6 +6786,8 @@ buffer_readv_encode_error(PgAioResult *result,
> > else
> > result->status = PGAIO_RS_WARNING;
> >
> > + result->result = 0;
> > +
>
> That'd be completely wrong - and the tests indeed fail if you do that. The
> read might succeed with a warning (e.g. due to zero_damaged_pages) in which
> case the result still carries important information about how many blocks
> were
> successfully read.
>
That's exactly why it's not a patch.
>
>
> > /*
> > * The encoding is complicated enough to warrant cross-checking it
> against
> > * the decode function.
> > @@ -6868,8 +6870,6 @@ buffer_readv_complete_one(PgAioTargetData *td,
> uint8
> > buf_off, Buffer buffer,
> > /* Check for garbage data. */
> > if (!failed)
> > {
> > - PgAioResult result_one;
> > -
> > if (!PageIsVerified((Page) bufdata, tag.blockNum, piv_flags,
> > failed_checksum))
> > {
> > @@ -6904,6 +6904,8 @@ buffer_readv_complete_one(PgAioTargetData *td,
> uint8
> > buf_off, Buffer buffer,
> > */
> > if (*buffer_invalid || *failed_checksum || *zeroed_buffer)
> > {
> > + PgAioResult result_one;
> > +
> > buffer_readv_encode_error(&result_one, is_temp,
> > *zeroed_buffer,
> > *ignored_checksum,
> >
> >
> > 1. I couldn't find the correct value to initialize the *result* field.
>
> It is not accessed in this path. I guess we can just zero-initialize
> result_one to shut up coverity.
>
Very good.
>
>
> > 2. result_one can be reduced scope.
>
> True.
>
Ok.
best regards,
Ranier Vilela
^ permalink raw reply [nested|flat] 153+ messages in thread
* Re: AIO v2.5
2025-03-04 19:00 AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-10 18:23 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-11 19:41 ` Re: AIO v2.5 Noah Misch <noah@leadboat.com>
2025-03-11 23:55 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-12 03:57 ` Re: AIO v2.5 Noah Misch <noah@leadboat.com>
2025-03-12 17:06 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-12 23:23 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-13 10:53 ` Re: AIO v2.5 Antonin Houska <ah@cybertec.at>
2025-03-14 19:43 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-17 20:00 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-18 20:12 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-21 01:58 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-25 01:18 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-27 01:07 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-27 20:58 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-28 03:22 ` Re: AIO v2.5 Noah Misch <noah@leadboat.com>
2025-03-28 12:54 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-29 03:35 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-29 13:41 ` Re: AIO v2.5 Noah Misch <noah@leadboat.com>
2025-03-29 14:48 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-29 18:25 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-29 21:29 ` Re: AIO v2.5 Noah Misch <noah@leadboat.com>
2025-03-30 23:46 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-04-01 00:41 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-04-01 15:11 ` Re: AIO v2.5 Noah Misch <noah@leadboat.com>
2025-04-01 15:55 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-04-01 21:47 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-04-01 22:25 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-04-02 00:13 ` Re: AIO v2.5 Noah Misch <noah@leadboat.com>
2025-04-02 11:58 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-04-03 16:46 ` Re: AIO v2.5 Ranier Vilela <ranier.vf@gmail.com>
2025-04-03 18:35 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-04-03 19:16 ` Re: AIO v2.5 Ranier Vilela <ranier.vf@gmail.com>
@ 2025-04-03 22:30 ` Andres Freund <andres@anarazel.de>
0 siblings, 0 replies; 153+ messages in thread
From: Andres Freund @ 2025-04-03 22:30 UTC (permalink / raw)
To: Ranier Vilela <ranier.vf@gmail.com>; +Cc: Noah Misch <noah@leadboat.com>; pgsql-hackers@postgresql.org, Thomas Munro <thomas.munro@gmail.com>; Heikki Linnakangas <hlinnaka@iki.fi>; Robert Haas <robertmhaas@gmail.com>; Jakub Wartak <jakub.wartak@enterprisedb.com>; Jelte Fennema-Nio <postgres@jeltef.nl>; Antonin Houska <ah@cybertec.at>
Hi,
On 2025-04-03 16:16:50 -0300, Ranier Vilela wrote:
> Em qui., 3 de abr. de 2025 Ã s 15:35, Andres Freund <andres@anarazel.de>
> escreveu:> > On 2025-04-03 13:46:39 -0300, Ranier Vilela wrote:
> > > Em qua., 2 de abr. de 2025 Ã s 08:58, Andres Freund <andres@anarazel.de>
> > > escreveu:
> > >
> > > > Hi,
> > > >
> > > > I've pushed fixes for 1) and 2) and am working on 3).
> > > >
> > > Coverity has one report about this.
> > >
> > > CID 1596092: (#1 of 1): Uninitialized scalar variable (UNINIT)
> > > 13. uninit_use_in_call: Using uninitialized value result_one. Field
> > > result_one.result is uninitialized when calling pgaio_result_report.
> >
> > Isn't this a rather silly thing to warn about for coverity?
>
> Personally, I consider every warning to be important.
If the warning is wrong, then it's not helpful. Warning quality really
matters.
Zero-initializing everything *REDUCES* what static analysis and sanitizers can
do. The analyzer/sanitizer can't tell that you just silenced a warning by
zero-initializing something that shouldn't be accessed. If later there is an
access, the zero is probably the wrong value, but the no tool can tell you,
because you did initialize it after all.
>
> > The field isn't
> > used in pgaio_result_report(). It can't be a particularly rare thing to
> > have
> > struct fields that aren't always used?
> >
> Always considered a risk, someone may start using it.
That makes it worse! E.g. valgrind won't raise errors about it anymore.
Greetings,
Andres Freund
^ permalink raw reply [nested|flat] 153+ messages in thread
* Re: AIO v2.5
2025-03-04 19:00 AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-10 18:23 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-11 19:41 ` Re: AIO v2.5 Noah Misch <noah@leadboat.com>
2025-03-11 23:55 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-12 03:57 ` Re: AIO v2.5 Noah Misch <noah@leadboat.com>
2025-03-12 17:06 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-12 23:23 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-13 10:53 ` Re: AIO v2.5 Antonin Houska <ah@cybertec.at>
2025-03-14 19:43 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-17 20:00 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-18 20:12 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-21 01:58 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-25 01:18 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-27 01:07 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-27 20:58 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-28 03:22 ` Re: AIO v2.5 Noah Misch <noah@leadboat.com>
2025-03-28 12:54 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-29 03:35 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-29 13:41 ` Re: AIO v2.5 Noah Misch <noah@leadboat.com>
2025-03-29 14:48 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-29 18:25 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-29 21:29 ` Re: AIO v2.5 Noah Misch <noah@leadboat.com>
2025-03-30 23:46 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-04-01 00:41 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-04-01 15:11 ` Re: AIO v2.5 Noah Misch <noah@leadboat.com>
2025-04-01 15:55 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-04-01 21:47 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-04-01 22:25 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-04-02 00:13 ` Re: AIO v2.5 Noah Misch <noah@leadboat.com>
2025-04-02 11:58 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
@ 2025-04-06 20:00 ` Alexander Lakhin <exclusion@gmail.com>
2025-04-07 16:20 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
1 sibling, 1 reply; 153+ messages in thread
From: Alexander Lakhin @ 2025-04-06 20:00 UTC (permalink / raw)
To: Andres Freund <andres@anarazel.de>; Noah Misch <noah@leadboat.com>; +Cc: pgsql-hackers@postgresql.org, Thomas Munro <thomas.munro@gmail.com>; Heikki Linnakangas <hlinnaka@iki.fi>; Robert Haas <robertmhaas@gmail.com>; Jakub Wartak <jakub.wartak@enterprisedb.com>; Jelte Fennema-Nio <postgres@jeltef.nl>; Antonin Houska <ah@cybertec.at>
Hello Andres,
02.04.2025 14:58, Andres Freund wrote:
> Hi,
>
> I've pushed fixes for 1) and 2) and am working on 3).
When running multiple installcheck's against a single server (please find
the ready-to-use script attached (I use more sophisticated version with
additional patches to make installcheck pass cleanly, but that's not
required for this case)), I've encountered an interesting error related to
AIO/uring:
iteration 8: Sun Apr 6 19:22:39 UTC 2025
installchecks finished: Sun Apr 6 19:23:47 UTC 2025
2025-04-06 19:22:44.216 UTC [349525] LOG:Â could not read blocks 0..0 in file "base/6179194/2606": Operation canceled
2025-04-06 19:22:44.216 UTC [349525] ERROR:Â could not read blocks 0..0 in file "base/6179194/2606": Operation canceled
server.log contains:
2025-04-06 19:22:44.215 UTC [38231] LOG:Â checkpoint complete: wrote ...
2025-04-06 19:22:44.216 UTC [38231] LOG:Â checkpoint starting: immediate force wait flush-all
2025-04-06 19:22:44.216 UTC [349525] LOG:Â could not read blocks 0..0 in file "base/6179194/2606": Operation canceled
2025-04-06 19:22:44.216 UTC [349525] STATEMENT:Â alter table parted_copytest attach partition parted_copytest_a1 for
values in(1);
2025-04-06 19:22:44.216 UTC [349525] ERROR:Â could not read blocks 0..0 in file "base/6179194/2606": Operation canceled
2025-04-06 19:22:44.216 UTC [349525] STATEMENT:Â alter table parted_copytest attach partition parted_copytest_a1 for
values in(1);
It's reproduced better on tmpfs for me; probably you would need to increase
NUM_INSTALLCHECKS/NUM_ITERATIONS for your machine. I can reduce the testing
procedure to something trivial, if it makes sense for you. Probably, the
same effect can be also achieved with just pgbench...
Best regards,
Alexander Lakhin
Neon (https://neon.tech)
Attachments:
[application/gzip] test-icx+.tar.gz (2.5K, ../../8160688e-db8e-4d7d-a7e7-ff366adf9717@gmail.com/3-test-icx+.tar.gz)
download
^ permalink raw reply [nested|flat] 153+ messages in thread
* Re: AIO v2.5
2025-03-04 19:00 AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-10 18:23 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-11 19:41 ` Re: AIO v2.5 Noah Misch <noah@leadboat.com>
2025-03-11 23:55 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-12 03:57 ` Re: AIO v2.5 Noah Misch <noah@leadboat.com>
2025-03-12 17:06 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-12 23:23 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-13 10:53 ` Re: AIO v2.5 Antonin Houska <ah@cybertec.at>
2025-03-14 19:43 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-17 20:00 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-18 20:12 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-21 01:58 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-25 01:18 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-27 01:07 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-27 20:58 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-28 03:22 ` Re: AIO v2.5 Noah Misch <noah@leadboat.com>
2025-03-28 12:54 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-29 03:35 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-29 13:41 ` Re: AIO v2.5 Noah Misch <noah@leadboat.com>
2025-03-29 14:48 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-29 18:25 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-29 21:29 ` Re: AIO v2.5 Noah Misch <noah@leadboat.com>
2025-03-30 23:46 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-04-01 00:41 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-04-01 15:11 ` Re: AIO v2.5 Noah Misch <noah@leadboat.com>
2025-04-01 15:55 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-04-01 21:47 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-04-01 22:25 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-04-02 00:13 ` Re: AIO v2.5 Noah Misch <noah@leadboat.com>
2025-04-02 11:58 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-04-06 20:00 ` Re: AIO v2.5 Alexander Lakhin <exclusion@gmail.com>
@ 2025-04-07 16:20 ` Andres Freund <andres@anarazel.de>
2025-04-07 19:10 ` Re: AIO v2.5 Alexander Lakhin <exclusion@gmail.com>
0 siblings, 1 reply; 153+ messages in thread
From: Andres Freund @ 2025-04-07 16:20 UTC (permalink / raw)
To: Alexander Lakhin <exclusion@gmail.com>; +Cc: Noah Misch <noah@leadboat.com>; pgsql-hackers@postgresql.org, Thomas Munro <thomas.munro@gmail.com>; Heikki Linnakangas <hlinnaka@iki.fi>; Robert Haas <robertmhaas@gmail.com>; Jakub Wartak <jakub.wartak@enterprisedb.com>; Jelte Fennema-Nio <postgres@jeltef.nl>; Antonin Houska <ah@cybertec.at>
Hi,
On 2025-04-06 23:00:00 +0300, Alexander Lakhin wrote:
> 02.04.2025 14:58, Andres Freund wrote:
> When running multiple installcheck's against a single server (please find
> the ready-to-use script attached (I use more sophisticated version with
> additional patches to make installcheck pass cleanly, but that's not
> required for this case)), I've encountered an interesting error related to
> AIO/uring:
> iteration 8: Sun Apr 6 19:22:39 UTC 2025
> installchecks finished: Sun Apr 6 19:23:47 UTC 2025
> 2025-04-06 19:22:44.216 UTC [349525] LOG:Â could not read blocks 0..0 in file "base/6179194/2606": Operation canceled
> 2025-04-06 19:22:44.216 UTC [349525] ERROR:Â could not read blocks 0..0 in file "base/6179194/2606": Operation canceled
Thanks for the report, clearly something isn't right.
> It's reproduced better on tmpfs for me; probably you would need to increase
> NUM_INSTALLCHECKS/NUM_ITERATIONS for your machine.
I ran it for a while in a VM, it hasn't triggered yet. Neither on xfs nor on
tmpfs.
> server.log contains:
> 2025-04-06 19:22:44.215 UTC [38231] LOG:Â checkpoint complete: wrote ...
> 2025-04-06 19:22:44.216 UTC [38231] LOG:Â checkpoint starting: immediate force wait flush-all
> 2025-04-06 19:22:44.216 UTC [349525] LOG:Â could not read blocks 0..0 in file "base/6179194/2606": Operation canceled
> 2025-04-06 19:22:44.216 UTC [349525] STATEMENT:Â alter table parted_copytest
> attach partition parted_copytest_a1 for values in(1);
> 2025-04-06 19:22:44.216 UTC [349525] ERROR:Â could not read blocks 0..0 in file "base/6179194/2606": Operation canceled
> 2025-04-06 19:22:44.216 UTC [349525] STATEMENT:Â alter table parted_copytest
> attach partition parted_copytest_a1 for values in(1);
Hm. Does the failure vary between occurrences?
- is it always the same statement? Probably not?
- is it always 2606 (i.e. pg_constraint)?
- does the failure always happen around a checkpoint? If so, is it always
immediate?
- I do assume it's always ECANCELED?
> I can reduce the testing procedure to something trivial, if it makes sense
> for you. Probably, the same effect can be also achieved with just pgbench...
That'd be very helpful!
Greetings,
Andres Freund
^ permalink raw reply [nested|flat] 153+ messages in thread
* Re: AIO v2.5
2025-03-04 19:00 AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-10 18:23 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-11 19:41 ` Re: AIO v2.5 Noah Misch <noah@leadboat.com>
2025-03-11 23:55 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-12 03:57 ` Re: AIO v2.5 Noah Misch <noah@leadboat.com>
2025-03-12 17:06 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-12 23:23 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-13 10:53 ` Re: AIO v2.5 Antonin Houska <ah@cybertec.at>
2025-03-14 19:43 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-17 20:00 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-18 20:12 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-21 01:58 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-25 01:18 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-27 01:07 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-27 20:58 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-28 03:22 ` Re: AIO v2.5 Noah Misch <noah@leadboat.com>
2025-03-28 12:54 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-29 03:35 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-29 13:41 ` Re: AIO v2.5 Noah Misch <noah@leadboat.com>
2025-03-29 14:48 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-29 18:25 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-29 21:29 ` Re: AIO v2.5 Noah Misch <noah@leadboat.com>
2025-03-30 23:46 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-04-01 00:41 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-04-01 15:11 ` Re: AIO v2.5 Noah Misch <noah@leadboat.com>
2025-04-01 15:55 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-04-01 21:47 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-04-01 22:25 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-04-02 00:13 ` Re: AIO v2.5 Noah Misch <noah@leadboat.com>
2025-04-02 11:58 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-04-06 20:00 ` Re: AIO v2.5 Alexander Lakhin <exclusion@gmail.com>
2025-04-07 16:20 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
@ 2025-04-07 19:10 ` Alexander Lakhin <exclusion@gmail.com>
2025-04-13 06:00 ` Re: AIO v2.5 Alexander Lakhin <exclusion@gmail.com>
0 siblings, 1 reply; 153+ messages in thread
From: Alexander Lakhin @ 2025-04-07 19:10 UTC (permalink / raw)
To: Andres Freund <andres@anarazel.de>; +Cc: Noah Misch <noah@leadboat.com>; pgsql-hackers@postgresql.org, Thomas Munro <thomas.munro@gmail.com>; Heikki Linnakangas <hlinnaka@iki.fi>; Robert Haas <robertmhaas@gmail.com>; Jakub Wartak <jakub.wartak@enterprisedb.com>; Jelte Fennema-Nio <postgres@jeltef.nl>; Antonin Houska <ah@cybertec.at>
Hello Andres,
07.04.2025 19:20, Andres Freund wrote:
>> iteration 8: Sun Apr 6 19:22:39 UTC 2025
>> installchecks finished: Sun Apr 6 19:23:47 UTC 2025
>> 2025-04-06 19:22:44.216 UTC [349525] LOG:Â could not read blocks 0..0 in file "base/6179194/2606": Operation canceled
>> 2025-04-06 19:22:44.216 UTC [349525] ERROR:Â could not read blocks 0..0 in file "base/6179194/2606": Operation canceled
> Thanks for the report, clearly something isn't right.
Thank you for your attention to it!
>> It's reproduced better on tmpfs for me; probably you would need to increase
>> NUM_INSTALLCHECKS/NUM_ITERATIONS for your machine.
> I ran it for a while in a VM, it hasn't triggered yet. Neither on xfs nor on
> tmpfs.
Before sharing the script I tested it on two my machines, but I had
anticipated that the error can be hard to reproduce. Will try to reduce
the reproducer...
>> server.log contains:
>> 2025-04-06 19:22:44.215 UTC [38231] LOG:Â checkpoint complete: wrote ...
>> 2025-04-06 19:22:44.216 UTC [38231] LOG:Â checkpoint starting: immediate force wait flush-all
>> 2025-04-06 19:22:44.216 UTC [349525] LOG:Â could not read blocks 0..0 in file "base/6179194/2606": Operation canceled
>> 2025-04-06 19:22:44.216 UTC [349525] STATEMENT:Â alter table parted_copytest
>> attach partition parted_copytest_a1 for values in(1);
>> 2025-04-06 19:22:44.216 UTC [349525] ERROR:Â could not read blocks 0..0 in file "base/6179194/2606": Operation canceled
>> 2025-04-06 19:22:44.216 UTC [349525] STATEMENT:Â alter table parted_copytest
>> attach partition parted_copytest_a1 for values in(1);
> Hm. Does the failure vary between occurrences?
> - is it always the same statement? Probably not?
> - is it always 2606 (i.e. pg_constraint)?
> - does the failure always happen around a checkpoint? If so, is it always
> immediate?
> - I do assume it's always ECANCELED?
I've got three such failures with three runs of of the script to
answer your questions:
...
iteration 6: Mon Apr 7 18:08:35 UTC 2025
installchecks finished: Mon Apr 7 18:09:45 UTC 2025
2025-04-07 18:08:45.369 UTC [1941260] LOG:Â could not read blocks 7..7 in file "base/4448730/1255": Operation canceled
2025-04-07 18:08:45.369 UTC [1941260] ERROR:Â could not read blocks 7..7 in file "base/4448730/1255": Operation canceled
server.log:
...
2025-04-07 18:08:44.120 UTC [1713945] LOG:Â checkpoint starting: wal
2025-04-07 18:08:45.369 UTC [1941260] LOG:Â could not read blocks 7..7 in file "base/4448730/1255": Operation canceled
2025-04-07 18:08:45.369 UTC [1941260] STATEMENT:Â SELECT routine_name, ordinal_position, parameter_name, parameter_default
       FROM information_schema.parameters JOIN information_schema.routines USING (specific_schema, specific_name)
       WHERE routine_schema = 'temp_func_test' AND routine_name ~ '^functest_is_'
       ORDER BY 1, 2;
2025-04-07 18:08:45.369 UTC [1941260] ERROR:Â could not read blocks 7..7 in file "base/4448730/1255": Operation canceled
2025-04-07 18:08:45.369 UTC [1941260] STATEMENT:Â SELECT routine_name, ordinal_position, parameter_name, parameter_default
       FROM information_schema.parameters JOIN information_schema.routines USING (specific_schema, specific_name)
       WHERE routine_schema = 'temp_func_test' AND routine_name ~ '^functest_is_'
       ORDER BY 1, 2;
...
2025-04-07 18:08:51.836 UTC [1713945] LOG:Â checkpoint complete: wrote 1558 buffers (9.5%), wrote 22 SLRU buffers; 0 WAL
file(s) added, 0 removed, 33 recycled; write=7.544 s, sync=0.019 s, total=7.720 s; sync files=0, longest=0.000 s,
average=0.000 s; distance=533837 kB, estimate=533837 kB; lsn=C/E532D478, redo lsn=C/C7054358
...
---
iteration 8: Mon Apr 7 18:26:47 UTC 2025
installchecks finished: Mon Apr 7 18:27:59 UTC 2025
2025-04-07 18:26:53.252 UTC [2359398] LOG:Â could not read blocks 0..12 in file "base/4/1255": Operation canceled
2025-04-07 18:26:53.255 UTC [2359398] ERROR:Â could not read blocks 0..12 in file "base/4/1255": Operation canceled
server.log:
...
2025-04-07 18:26:53.249 UTC [2048053] LOG:Â checkpoint complete: wrote 3061 buffers (18.7%), wrote 3 SLRU buffers; 0 WAL
file(s) added, 2 removed, 0 recycled; write=0.051 s, sync=0.002 s, total=0.068 s; sync files=0, longest=0.000 s,
average=0.000 s; distance=18937 kB, estimate=120161 kB; lsn=11/6D42ED30, redo lsn=11/6C17AC30
2025-04-07 18:26:53.251 UTC [2359308] WARNING:Â "most_common_elems" must be specified when "most_common_elem_freqs" is
specified
2025-04-07 18:26:53.251 UTC [2359351] WARNING: "elem_count_histogram" array cannot contain NULL values
2025-04-07 18:26:53.252 UTC [2359375] ERROR:Â cannot modify statistics for relation "testseq"
2025-04-07 18:26:53.252 UTC [2359375] DETAIL:Â This operation is not supported for sequences.
2025-04-07 18:26:53.252 UTC [2359375] STATEMENT:Â SELECT pg_catalog.pg_restore_relation_stats(
           'schemaname', 'stats_import',
           'relname', 'testseq');
2025-04-07 18:26:53.252 UTC [2359375] ERROR:Â cannot modify statistics for relation "testseq"
2025-04-07 18:26:53.252 UTC [2359375] DETAIL:Â This operation is not supported for sequences.
2025-04-07 18:26:53.252 UTC [2359375] STATEMENT:Â SELECT pg_catalog.pg_clear_relation_stats(schemaname =>
'stats_import', relname => 'testseq');
2025-04-07 18:26:53.252 UTC [2359398] LOG:Â could not read blocks 0..12 in file "base/4/1255": Operation canceled
2025-04-07 18:26:53.252 UTC [2359398] STATEMENT:Â CREATE DATABASE regress020_tbd
       ENCODING utf8 LC_COLLATE "C" LC_CTYPE "C" TEMPLATE template0;
2025-04-07 18:26:53.255 UTC [2359449] ERROR:Â invalid input syntax for type pg_snapshot: "31:12:" at character 8
2025-04-07 18:26:53.255 UTC [2359449] STATEMENT:Â select '31:12:'::pg_snapshot;
2025-04-07 18:26:53.255 UTC [2359449] ERROR:Â invalid input syntax for type pg_snapshot: "0:1:" at character 8
2025-04-07 18:26:53.255 UTC [2359449] STATEMENT:Â select '0:1:'::pg_snapshot;
2025-04-07 18:26:53.255 UTC [2359449] ERROR:Â invalid input syntax for type pg_snapshot: "12:13:0" at character 8
2025-04-07 18:26:53.255 UTC [2359449] STATEMENT:Â select '12:13:0'::pg_snapshot;
2025-04-07 18:26:53.255 UTC [2359449] ERROR:Â invalid input syntax for type pg_snapshot: "12:16:14,13" at character 8
2025-04-07 18:26:53.255 UTC [2359449] STATEMENT:Â select '12:16:14,13'::pg_snapshot;
2025-04-07 18:26:53.255 UTC [2359398] ERROR:Â could not read blocks 0..12 in file "base/4/1255": Operation canceled
2025-04-07 18:26:53.255 UTC [2359398] STATEMENT:Â CREATE DATABASE regress020_tbd
       ENCODING utf8 LC_COLLATE "C" LC_CTYPE "C" TEMPLATE template0;
...
---
iteration 2: Mon Apr 7 19:02:57 UTC 2025
installchecks finished: Mon Apr 7 19:04:07 UTC 2025
2025-04-07 19:03:02.133 UTC [3537488] LOG:Â could not read blocks 0..0 in file "base/902183/2607": Operation canceled
2025-04-07 19:03:02.133 UTC [3537488] ERROR:Â could not read blocks 0..0 in file "base/902183/2607": Operation canceled
server.log:
2025-04-07 19:03:02.129 UTC [3487651] LOG:Â checkpoint complete: wrote 337 buffers (2.1%), wrote 2 SLRU buffers; 0 WAL
file(s) added, 0 removed, 0 recycled; write=0.013 s, sync=0.001 s, total=0.024 s; sync files=0, longest=0.000 s,
average=0.000 s; distance=2555 kB, estimate=293963 kB; lsn=2/9D3DECE8, redo lsn=2/9D2AAC00
2025-04-07 19:03:02.129 UTC [3537184] ERROR:Â invalid regular expression: invalid backreference number
2025-04-07 19:03:02.129 UTC [3537184] STATEMENT:Â select 'xyz' ~ 'x(\w)(?=(\1))';
2025-04-07 19:03:02.129 UTC [3537184] ERROR:Â invalid regular expression: invalid escape \ sequence
2025-04-07 19:03:02.129 UTC [3537184] STATEMENT:Â select 'a' ~ '\x7fffffff';
2025-04-07 19:03:02.130 UTC [3537651] ERROR:Â operator does not exist: point = box at character 23
2025-04-07 19:03:02.130 UTC [3537651] HINT:Â No operator matches the given name and argument types. You might need to
add explicit type casts.
2025-04-07 19:03:02.130 UTC [3537651] STATEMENT:Â select '(0,0)'::point in ('(0,0,0,0)'::box, point(0,0));
2025-04-07 19:03:02.133 UTC [3537362] ERROR:Â invalid regular expression: invalid backreference number
2025-04-07 19:03:02.133 UTC [3537362] STATEMENT:Â select 'xyz' ~ 'x(\w)(?=\1)';
2025-04-07 19:03:02.133 UTC [3537362] ERROR:Â invalid regular expression: invalid backreference number
2025-04-07 19:03:02.133 UTC [3537362] STATEMENT:Â select 'xyz' ~ 'x(\w)(?=(\1))';
2025-04-07 19:03:02.133 UTC [3537362] ERROR:Â invalid regular expression: invalid escape \ sequence
2025-04-07 19:03:02.133 UTC [3537362] STATEMENT:Â select 'a' ~ '\x7fffffff';
2025-04-07 19:03:02.133 UTC [3537488] LOG:Â could not read blocks 0..0 in file "base/902183/2607": Operation canceled
2025-04-07 19:03:02.133 UTC [3537488] STATEMENT:Â SELECT p1.oid, p1.proname
   FROM pg_proc as p1
   WHERE 'cstring'::regtype = ANY (p1.proargtypes)
       AND NOT EXISTS(SELECT 1 FROM pg_type WHERE typinput = p1.oid)
       AND NOT EXISTS(SELECT 1 FROM pg_conversion WHERE conproc = p1.oid)
       AND p1.oid != 'shell_in(cstring)'::regprocedure
   ORDER BY 1;
2025-04-07 19:03:02.133 UTC [3537488] ERROR:Â could not read blocks 0..0 in file "base/902183/2607": Operation canceled
2025-04-07 19:03:02.133 UTC [3537488] STATEMENT:Â SELECT p1.oid, p1.proname
   FROM pg_proc as p1
   WHERE 'cstring'::regtype = ANY (p1.proargtypes)
       AND NOT EXISTS(SELECT 1 FROM pg_type WHERE typinput = p1.oid)
       AND NOT EXISTS(SELECT 1 FROM pg_conversion WHERE conproc = p1.oid)
       AND p1.oid != 'shell_in(cstring)'::regprocedure
   ORDER BY 1;
2025-04-07 19:03:02.135 UTC [3537671] WARNING:Â TIME(7) WITH TIME ZONE precision reduced to maximum allowed, 6
2025-04-07 19:03:02.135 UTC [3537671] WARNING:Â TIMESTAMP(7) WITH TIME ZONE precision reduced to maximum allowed, 6
2025-04-07 19:03:02.135 UTC [3537671] WARNING:Â TIME(7) precision reduced to maximum allowed, 6
2025-04-07 19:03:02.136 UTC [3487651] LOG:Â checkpoint starting: immediate force wait
...
So I suspect checkpointer, but I'm not sure yet.
(Sometimes (2 out of 5 runs) 10 iterations pass without the error.)
That script with s/grep 'could not read blocks '/grep 'was terminated '/
also discovers another anomaly (right now on the third run, with 10
iterations of 30 installchecks):
Core was generated by `postgres: postgres regress005 [local] SELECTÂ Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â '.
Program terminated with signal SIGSEGV, Segmentation fault.
#0Â 0x0000000000000000 in ?? ()
(gdb) bt
#0Â 0x0000000000000000 in ?? ()
#1Â 0x0000591086d5ccfa in pgaio_io_get_target_description (ioh=ioh@entry=0x7ffcf961bc20) at aio_target.c:85
#2Â 0x0000591086d5bfdd in pg_get_aios (fcinfo=<optimized out>) at aio_funcs.c:218
#3Â 0x0000591086bb64c1 in ExecMakeTableFunctionResult (setexpr=0x5910bddcbf48, econtext=0x5910bddcbde8,
argContext=<optimized out>,
   expectedDesc=0x5910bddff918, randomAccess=false) at execSRF.c:234
#4Â 0x0000591086bc9673 in FunctionNext (node=node@entry=0x5910bddcbbd8) at nodeFunctionscan.c:94
#5Â 0x0000591086bb7093 in ExecScanFetch (recheckMtd=0x591086bc9522 <FunctionRecheck>, accessMtd=0x591086bc95a2
<FunctionNext>, epqstate=0x0,
   node=0x5910bddcbbd8) at ../../../src/include/executor/execScan.h:126
#6Â ExecScanExtended (projInfo=0x0, qual=0x0, epqstate=0x0, recheckMtd=0x591086bc9522 <FunctionRecheck>,
accessMtd=0x591086bc95a2 <FunctionNext>,
   node=0x5910bddcbbd8) at ../../../src/include/executor/execScan.h:170
#7Â ExecScan (node=0x5910bddcbbd8, accessMtd=accessMtd@entry=0x591086bc95a2 <FunctionNext>,
recheckMtd=recheckMtd@entry=0x591086bc9522 <FunctionRecheck>)
   at execScan.c:59
#8Â 0x0000591086bc9580 in ExecFunctionScan (pstate=<optimized out>) at nodeFunctionscan.c:269
#9Â 0x0000591086bb38f3 in ExecProcNodeFirst (node=0x5910bddcbbd8) at execProcnode.c:469
#10 0x0000591086bc0195 in ExecProcNode (node=0x5910bddcbbd8) at ../../../src/include/executor/executor.h:341
#11 0x0000591086bc021b in fetch_input_tuple (aggstate=aggstate@entry=0x5910bddcb480) at nodeAgg.c:563
#12 0x0000591086bc3825 in agg_retrieve_direct (aggstate=aggstate@entry=0x5910bddcb480) at nodeAgg.c:2450
#13 0x0000591086bc3a19 in ExecAgg (pstate=<optimized out>) at nodeAgg.c:2265
#14 0x0000591086bb38f3 in ExecProcNodeFirst (node=0x5910bddcb480) at execProcnode.c:469
#15 0x0000591086baac23 in ExecProcNode (node=node@entry=0x5910bddcb480) at ../../../src/include/executor/executor.h:341
#16 0x0000591086baacd0 in ExecutePlan (queryDesc=queryDesc@entry=0x5910bdde0e00, operation=operation@entry=CMD_SELECT,
sendTuples=sendTuples@entry=true,
   numberTuples=numberTuples@entry=0, direction=direction@entry=ForwardScanDirection, dest=dest@entry=0x5910bddf6070)
at execMain.c:1783
#17 0x0000591086bab79d in standard_ExecutorRun (queryDesc=0x5910bdde0e00, direction=ForwardScanDirection, count=0) at
execMain.c:435
#18 0x0000591086bab7e0 in ExecutorRun (queryDesc=queryDesc@entry=0x5910bdde0e00,
direction=direction@entry=ForwardScanDirection, count=count@entry=0)
   at execMain.c:371
#19 0x0000591086da9e9f in PortalRunSelect (portal=portal@entry=0x5910bdd72f10, forward=forward@entry=true, count=0,
count@entry=9223372036854775807,
   dest=dest@entry=0x5910bddf6070) at pquery.c:953
#20 0x0000591086dab83b in PortalRun (portal=portal@entry=0x5910bdd72f10, count=count@entry=9223372036854775807,
isTopLevel=isTopLevel@entry=true,
   dest=dest@entry=0x5910bddf6070, altdest=altdest@entry=0x5910bddf6070, qc=qc@entry=0x7ffcf961c9f0) at pquery.c:797
#21 0x0000591086da74a4 in exec_simple_query (query_string=query_string@entry=0x5910bdcebe60 "SELECT COUNT(*) >= 0 AS ok
FROM pg_aios;") at postgres.c:1274
But I'm yet to construct a more reliable reproducer for it. Hope I could
do this during the current week.
Best regards,
Alexander Lakhin
Neon (https://neon.tech)
^ permalink raw reply [nested|flat] 153+ messages in thread
* Re: AIO v2.5
2025-03-04 19:00 AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-10 18:23 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-11 19:41 ` Re: AIO v2.5 Noah Misch <noah@leadboat.com>
2025-03-11 23:55 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-12 03:57 ` Re: AIO v2.5 Noah Misch <noah@leadboat.com>
2025-03-12 17:06 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-12 23:23 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-13 10:53 ` Re: AIO v2.5 Antonin Houska <ah@cybertec.at>
2025-03-14 19:43 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-17 20:00 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-18 20:12 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-21 01:58 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-25 01:18 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-27 01:07 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-27 20:58 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-28 03:22 ` Re: AIO v2.5 Noah Misch <noah@leadboat.com>
2025-03-28 12:54 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-29 03:35 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-29 13:41 ` Re: AIO v2.5 Noah Misch <noah@leadboat.com>
2025-03-29 14:48 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-29 18:25 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-29 21:29 ` Re: AIO v2.5 Noah Misch <noah@leadboat.com>
2025-03-30 23:46 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-04-01 00:41 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-04-01 15:11 ` Re: AIO v2.5 Noah Misch <noah@leadboat.com>
2025-04-01 15:55 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-04-01 21:47 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-04-01 22:25 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-04-02 00:13 ` Re: AIO v2.5 Noah Misch <noah@leadboat.com>
2025-04-02 11:58 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-04-06 20:00 ` Re: AIO v2.5 Alexander Lakhin <exclusion@gmail.com>
2025-04-07 16:20 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-04-07 19:10 ` Re: AIO v2.5 Alexander Lakhin <exclusion@gmail.com>
@ 2025-04-13 06:00 ` Alexander Lakhin <exclusion@gmail.com>
2025-04-14 16:06 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
0 siblings, 1 reply; 153+ messages in thread
From: Alexander Lakhin @ 2025-04-13 06:00 UTC (permalink / raw)
To: Andres Freund <andres@anarazel.de>; +Cc: Noah Misch <noah@leadboat.com>; pgsql-hackers@postgresql.org, Thomas Munro <thomas.munro@gmail.com>; Heikki Linnakangas <hlinnaka@iki.fi>; Robert Haas <robertmhaas@gmail.com>; Jakub Wartak <jakub.wartak@enterprisedb.com>; Jelte Fennema-Nio <postgres@jeltef.nl>; Antonin Houska <ah@cybertec.at>
Hello Andres,
07.04.2025 22:10, Alexander Lakhin wrote:
>> I ran it for a while in a VM, it hasn't triggered yet. Neither on xfs nor on
>> tmpfs.
>
> Before sharing the script I tested it on two my machines, but I had
> anticipated that the error can be hard to reproduce. Will try to reduce
> the reproducer...
I've managed to reduce it to the following:
ulimit -n 4096
echo "
fsync = off
autovacuum = off
checkpoint_timeout = 30s
io_max_concurrency = 10
io_method = io_uring
" >> $PGDATA/postgresql.conf
pg_ctl -l server.log start
for i in `seq 1000`; do
 numjobs=$((20 + $RANDOM % 60))
 echo "iteration $i (jobs: $numjobs)"
 date
 for ((j=1;j<=numjobs;j++)); do
   (
     createdb db$j;
     for ((n=1;n<=50;n++)); do
       cat << EOF | psql -d db$j -a >>/dev/null 2>&1
DROP TABLE IF EXISTS tenk1;
CREATE TABLE tenk1 (
   unique1    int4,
   unique2    int4,
   two        int4,
   four       int4,
   ten        int4,
   twenty     int4,
   hundred    int4,
   thousand   int4,
   twothousand int4,
   fivethous  int4,
   tenthous   int4,
   odd        int4,
   even       int4,
   stringu1   name,
   stringu2   name,
   string4    name
);
COPY tenk1 FROM '.../src/test/regress/data/tenk.data';
EOF
     done;
   ) &
 done
 wait
 for ((j=1;j<=numjobs;j++)); do dropdb db$j & done
 wait
 grep -A3 -E '(ERROR|could not read blocks )' server.log && break;
done
pg_ctl stop
It fails for me as below:
iteration 13 (jobs: 25)
Sun Apr 13 05:31:47 AM UTC 2025
iteration 14 (jobs: 67)
Sun Apr 13 05:31:50 AM UTC 2025
dropdb: error: database removal failed: ERROR:Â could not read blocks 0..0 in file "global/1213": Operation canceled
2025-04-13 05:31:58.930 UTC [1153451] LOG:Â could not read blocks 0..0 in file "global/1213": Operation canceled
2025-04-13 05:31:58.930 UTC [1153451] CONTEXT:Â completing I/O on behalf of process 1153456
2025-04-13 05:31:58.930 UTC [1153451] STATEMENT:Â DROP DATABASE db5;
2025-04-13 05:31:58.930 UTC [1153456] ERROR:Â could not read blocks 0..0 in file "global/1213": Operation canceled
2025-04-13 05:31:58.930 UTC [1153456] STATEMENT:Â DROP DATABASE db6;
2025-04-13 05:31:58.931 UTC [1034758] LOG:Â checkpoint complete: wrote 3 buffers (0.0%), wrote 0 SLRU buffers; 0 WAL
file(s) added, 0 removed, 0 recycled; write=0.002 s, sync=0.001 s, total=0.002 s; sync files=0, longest=0.000 s,
average=0.000 s; distance=18 kB, estimate=458931 kB; lsn=16/54589E08, redo lsn=16/54586F88
2025-04-13 05:31:58.931 UTC [1034758] LOG:Â checkpoint starting: immediate force wait
I reproduced this error on three different machines (all are running
Ubuntu 24.04, two with kernel version 6.8, one with 6.11), with PGDATA
located on tmpfs.
Best regards,
Alexander Lakhin
Neon (https://neon.tech)
^ permalink raw reply [nested|flat] 153+ messages in thread
* Re: AIO v2.5
2025-03-04 19:00 AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-10 18:23 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-11 19:41 ` Re: AIO v2.5 Noah Misch <noah@leadboat.com>
2025-03-11 23:55 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-12 03:57 ` Re: AIO v2.5 Noah Misch <noah@leadboat.com>
2025-03-12 17:06 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-12 23:23 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-13 10:53 ` Re: AIO v2.5 Antonin Houska <ah@cybertec.at>
2025-03-14 19:43 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-17 20:00 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-18 20:12 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-21 01:58 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-25 01:18 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-27 01:07 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-27 20:58 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-28 03:22 ` Re: AIO v2.5 Noah Misch <noah@leadboat.com>
2025-03-28 12:54 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-29 03:35 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-29 13:41 ` Re: AIO v2.5 Noah Misch <noah@leadboat.com>
2025-03-29 14:48 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-29 18:25 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-29 21:29 ` Re: AIO v2.5 Noah Misch <noah@leadboat.com>
2025-03-30 23:46 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-04-01 00:41 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-04-01 15:11 ` Re: AIO v2.5 Noah Misch <noah@leadboat.com>
2025-04-01 15:55 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-04-01 21:47 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-04-01 22:25 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-04-02 00:13 ` Re: AIO v2.5 Noah Misch <noah@leadboat.com>
2025-04-02 11:58 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-04-06 20:00 ` Re: AIO v2.5 Alexander Lakhin <exclusion@gmail.com>
2025-04-07 16:20 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-04-07 19:10 ` Re: AIO v2.5 Alexander Lakhin <exclusion@gmail.com>
2025-04-13 06:00 ` Re: AIO v2.5 Alexander Lakhin <exclusion@gmail.com>
@ 2025-04-14 16:06 ` Andres Freund <andres@anarazel.de>
2025-04-15 18:00 ` Re: AIO v2.5 Alexander Lakhin <exclusion@gmail.com>
0 siblings, 1 reply; 153+ messages in thread
From: Andres Freund @ 2025-04-14 16:06 UTC (permalink / raw)
To: Alexander Lakhin <exclusion@gmail.com>; +Cc: Noah Misch <noah@leadboat.com>; pgsql-hackers@postgresql.org, Thomas Munro <thomas.munro@gmail.com>; Heikki Linnakangas <hlinnaka@iki.fi>; Robert Haas <robertmhaas@gmail.com>; Jakub Wartak <jakub.wartak@enterprisedb.com>; Jelte Fennema-Nio <postgres@jeltef.nl>; Antonin Houska <ah@cybertec.at>
Hi,
On 2025-04-13 09:00:01 +0300, Alexander Lakhin wrote:
> 07.04.2025 22:10, Alexander Lakhin wrote:
> > > I ran it for a while in a VM, it hasn't triggered yet. Neither on xfs nor on
> > > tmpfs.
> >
> > Before sharing the script I tested it on two my machines, but I had
> > anticipated that the error can be hard to reproduce. Will try to reduce
> > the reproducer...
>
> I've managed to reduce it to the following:
Thanks a lot for working on that!
> [reproducer]
>
> It fails for me as below:
> iteration 13 (jobs: 25)
> Sun Apr 13 05:31:47 AM UTC 2025
> iteration 14 (jobs: 67)
> Sun Apr 13 05:31:50 AM UTC 2025
> dropdb: error: database removal failed: ERROR:Â could not read blocks 0..0 in file "global/1213": Operation canceled
> 2025-04-13 05:31:58.930 UTC [1153451] LOG:Â could not read blocks 0..0 in file "global/1213": Operation canceled
> 2025-04-13 05:31:58.930 UTC [1153451] CONTEXT:Â completing I/O on behalf of process 1153456
> 2025-04-13 05:31:58.930 UTC [1153451] STATEMENT:Â DROP DATABASE db5;
> 2025-04-13 05:31:58.930 UTC [1153456] ERROR:Â could not read blocks 0..0 in file "global/1213": Operation canceled
> 2025-04-13 05:31:58.930 UTC [1153456] STATEMENT:Â DROP DATABASE db6;
> 2025-04-13 05:31:58.931 UTC [1034758] LOG:Â checkpoint complete: wrote 3
> buffers (0.0%), wrote 0 SLRU buffers; 0 WAL file(s) added, 0 removed, 0
> recycled; write=0.002 s, sync=0.001 s, total=0.002 s; sync files=0,
> longest=0.000 s, average=0.000 s; distance=18 kB, estimate=458931 kB;
> lsn=16/54589E08, redo lsn=16/54586F88
> 2025-04-13 05:31:58.931 UTC [1034758] LOG:Â checkpoint starting: immediate force wait
Unfortunately I'm several hundred iterations in, without reproducing the
issue. I'm bad at statistics, but I think that makes it rather unlikely that I
will, without changing some aspect.
Was this an assert enabled build? What compiler and what optimization settings
did you use? Do you have huge pages configured (so that the default
huge_pages=try would end up with huge pages)?
So far I've been trying to use a cassert enabled build built with -O0, without
huge pages. After the current test run I'll switch to cassert+-O2.
> I reproduced this error on three different machines (all are running
> Ubuntu 24.04, two with kernel version 6.8, one with 6.11), with PGDATA
> located on tmpfs.
That's another variable to try - so far I've been trying this on 6.15.0-rc1
[1]. I guess I'll have to set up a ubuntu 24.04 VM and try with that.
Greetings,
Andres Freund
[1] I wanted to play with io_uring changes that were recently merged. Namely
support for readv/writev of "fixed" buffers. That avoids needing to pin/unpin
buffers while IO is ongoing, which turns out to be a noticeable bottleneck in
some workloads, particularly when using 1GB huge pages.
^ permalink raw reply [nested|flat] 153+ messages in thread
* Re: AIO v2.5
2025-03-04 19:00 AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-10 18:23 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-11 19:41 ` Re: AIO v2.5 Noah Misch <noah@leadboat.com>
2025-03-11 23:55 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-12 03:57 ` Re: AIO v2.5 Noah Misch <noah@leadboat.com>
2025-03-12 17:06 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-12 23:23 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-13 10:53 ` Re: AIO v2.5 Antonin Houska <ah@cybertec.at>
2025-03-14 19:43 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-17 20:00 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-18 20:12 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-21 01:58 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-25 01:18 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-27 01:07 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-27 20:58 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-28 03:22 ` Re: AIO v2.5 Noah Misch <noah@leadboat.com>
2025-03-28 12:54 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-29 03:35 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-29 13:41 ` Re: AIO v2.5 Noah Misch <noah@leadboat.com>
2025-03-29 14:48 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-29 18:25 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-29 21:29 ` Re: AIO v2.5 Noah Misch <noah@leadboat.com>
2025-03-30 23:46 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-04-01 00:41 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-04-01 15:11 ` Re: AIO v2.5 Noah Misch <noah@leadboat.com>
2025-04-01 15:55 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-04-01 21:47 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-04-01 22:25 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-04-02 00:13 ` Re: AIO v2.5 Noah Misch <noah@leadboat.com>
2025-04-02 11:58 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-04-06 20:00 ` Re: AIO v2.5 Alexander Lakhin <exclusion@gmail.com>
2025-04-07 16:20 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-04-07 19:10 ` Re: AIO v2.5 Alexander Lakhin <exclusion@gmail.com>
2025-04-13 06:00 ` Re: AIO v2.5 Alexander Lakhin <exclusion@gmail.com>
2025-04-14 16:06 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
@ 2025-04-15 18:00 ` Alexander Lakhin <exclusion@gmail.com>
2025-04-25 16:44 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-04-30 20:00 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
0 siblings, 2 replies; 153+ messages in thread
From: Alexander Lakhin @ 2025-04-15 18:00 UTC (permalink / raw)
To: Andres Freund <andres@anarazel.de>; +Cc: Noah Misch <noah@leadboat.com>; pgsql-hackers@postgresql.org, Thomas Munro <thomas.munro@gmail.com>; Heikki Linnakangas <hlinnaka@iki.fi>; Robert Haas <robertmhaas@gmail.com>; Jakub Wartak <jakub.wartak@enterprisedb.com>; Jelte Fennema-Nio <postgres@jeltef.nl>; Antonin Houska <ah@cybertec.at>
Hello Andres,
14.04.2025 19:06, Andres Freund wrote:
> Unfortunately I'm several hundred iterations in, without reproducing the
> issue. I'm bad at statistics, but I think that makes it rather unlikely that I
> will, without changing some aspect.
>
> Was this an assert enabled build? What compiler and what optimization settings
> did you use? Do you have huge pages configured (so that the default
> huge_pages=try would end up with huge pages)?
Yes, I used --enable-cassert; no explicit optimization setting and no huge
pages configured. pg_config says:
CONFIGURE =Â '--enable-debug' '--enable-cassert' '--enable-tap-tests' '--with-liburing'
CC = gcc
CPPFLAGS = -D_GNU_SOURCE
CFLAGS = -Wall -Wmissing-prototypes -Wpointer-arith -Wdeclaration-after-statement -Werror=vla -Wendif-labels
-Wmissing-format-attribute -Wimplicit-fallthrough=3 -Wcast-function-type -Wshadow=compatible-local -Wformat-security
-fno-strict-aliasing -fwrapv -fexcess-precision=standard -Wno-format-truncation -Wno-stringop-truncation -g -O2
Please look at the complete script attached. I've just run it and got:
iteration 56 (jobs: 44)
Tue Apr 15 06:30:52 PM CEST 2025
dropdb: error: database removal failed: ERROR:Â could not read blocks 0..0 in file "global/1213": Operation canceled
2025-04-15 18:31:00.650 CEST [1612266] LOG:Â could not read blocks 0..0 in file "global/1213": Operation canceled
2025-04-15 18:31:00.650 CEST [1612266] CONTEXT:Â completing I/O on behalf of process 1612271
2025-04-15 18:31:00.650 CEST [1612266] STATEMENT:Â DROP DATABASE db3;
I used gcc (Ubuntu 13.3.0-6ubuntu2~24.04) 13.3.0, but now I've also
reproduced the issue with CC=clang (18.1.3 (1ubuntu1)).
Please take a look also at the simple reproducer for the crash inside
pg_get_aios() I mentioned upthread:
for i in {1..100}; do
 numjobs=12
 echo "iteration $i"
 date
 for ((j=1;j<=numjobs;j++)); do
   ( createdb db$j; for k in {1..300}; do
       echo "CREATE TABLE t (a INT); CREATE INDEX ON t (a); VACUUM t;
             SELECT COUNT(*) >= 0 AS ok FROM pg_aios; " \
       | psql -d db$j >/dev/null 2>&1;
     done; dropdb db$j; ) &
 done
 wait
 psql -c 'SELECT 1' || break;
done
it fails for me as follows:
iteration 20
Tue Apr 15 07:21:29 PM EEST 2025
dropdb: error: connection to server on socket "/tmp/.s.PGSQL.55432" failed: No such file or directory
      Is the server running locally and accepting connections on that socket?
...
2025-04-15 19:21:30.675 EEST [3111699] LOG:Â client backend (PID 3320979) was terminated by signal 11: Segmentation fault
2025-04-15 19:21:30.675 EEST [3111699] DETAIL:Â Failed process was running: SELECT COUNT(*) >= 0 AS ok FROM pg_aios;
2025-04-15 19:21:30.675 EEST [3111699] LOG:Â terminating any other active server processes
>> I reproduced this error on three different machines (all are running
>> Ubuntu 24.04, two with kernel version 6.8, one with 6.11), with PGDATA
>> located on tmpfs.
> That's another variable to try - so far I've been trying this on 6.15.0-rc1
> [1]. I guess I'll have to set up a ubuntu 24.04 VM and try with that.
>
> Greetings,
>
> Andres Freund
>
>
> [1] I wanted to play with io_uring changes that were recently merged. Namely
> support for readv/writev of "fixed" buffers. That avoids needing to pin/unpin
> buffers while IO is ongoing, which turns out to be a noticeable bottleneck in
> some workloads, particularly when using 1GB huge pages.
Best regards,
Alexander Lakhin
Neon (https://neon.tech)
Attachments:
[application/gzip] repro.tar.gz (982B, ../../062daca9-dfad-4750-9da8-b13388301ad9@gmail.com/2-repro.tar.gz)
download
^ permalink raw reply [nested|flat] 153+ messages in thread
* Re: AIO v2.5
2025-03-04 19:00 AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-10 18:23 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-11 19:41 ` Re: AIO v2.5 Noah Misch <noah@leadboat.com>
2025-03-11 23:55 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-12 03:57 ` Re: AIO v2.5 Noah Misch <noah@leadboat.com>
2025-03-12 17:06 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-12 23:23 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-13 10:53 ` Re: AIO v2.5 Antonin Houska <ah@cybertec.at>
2025-03-14 19:43 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-17 20:00 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-18 20:12 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-21 01:58 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-25 01:18 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-27 01:07 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-27 20:58 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-28 03:22 ` Re: AIO v2.5 Noah Misch <noah@leadboat.com>
2025-03-28 12:54 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-29 03:35 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-29 13:41 ` Re: AIO v2.5 Noah Misch <noah@leadboat.com>
2025-03-29 14:48 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-29 18:25 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-29 21:29 ` Re: AIO v2.5 Noah Misch <noah@leadboat.com>
2025-03-30 23:46 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-04-01 00:41 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-04-01 15:11 ` Re: AIO v2.5 Noah Misch <noah@leadboat.com>
2025-04-01 15:55 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-04-01 21:47 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-04-01 22:25 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-04-02 00:13 ` Re: AIO v2.5 Noah Misch <noah@leadboat.com>
2025-04-02 11:58 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-04-06 20:00 ` Re: AIO v2.5 Alexander Lakhin <exclusion@gmail.com>
2025-04-07 16:20 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-04-07 19:10 ` Re: AIO v2.5 Alexander Lakhin <exclusion@gmail.com>
2025-04-13 06:00 ` Re: AIO v2.5 Alexander Lakhin <exclusion@gmail.com>
2025-04-14 16:06 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-04-15 18:00 ` Re: AIO v2.5 Alexander Lakhin <exclusion@gmail.com>
@ 2025-04-25 16:44 ` Andres Freund <andres@anarazel.de>
1 sibling, 0 replies; 153+ messages in thread
From: Andres Freund @ 2025-04-25 16:44 UTC (permalink / raw)
To: Alexander Lakhin <exclusion@gmail.com>; +Cc: Noah Misch <noah@leadboat.com>; pgsql-hackers@postgresql.org, Thomas Munro <thomas.munro@gmail.com>; Heikki Linnakangas <hlinnaka@iki.fi>; Robert Haas <robertmhaas@gmail.com>; Jakub Wartak <jakub.wartak@enterprisedb.com>; Jelte Fennema-Nio <postgres@jeltef.nl>; Antonin Houska <ah@cybertec.at>
Hi,
On 2025-04-15 21:00:00 +0300, Alexander Lakhin wrote:
> Please take a look also at the simple reproducer for the crash inside
> pg_get_aios() I mentioned upthread:
> for i in {1..100}; do
> Â numjobs=12
> Â echo "iteration $i"
> Â date
> Â for ((j=1;j<=numjobs;j++)); do
> Â Â Â ( createdb db$j; for k in {1..300}; do
> Â Â Â Â Â Â Â echo "CREATE TABLE t (a INT); CREATE INDEX ON t (a); VACUUM t;
> Â Â Â Â Â Â Â Â Â Â Â Â Â SELECT COUNT(*) >= 0 AS ok FROM pg_aios; " \
> Â Â Â Â Â Â Â | psql -d db$j >/dev/null 2>&1;
> Â Â Â Â Â done; dropdb db$j; ) &
> Â done
> Â wait
> Â psql -c 'SELECT 1' || break;
> done
>
> it fails for me as follows:
> iteration 20
> Tue Apr 15 07:21:29 PM EEST 2025
> dropdb: error: connection to server on socket "/tmp/.s.PGSQL.55432" failed: No such file or directory
> Â Â Â Â Â Â Is the server running locally and accepting connections on that socket?
> ...
> 2025-04-15 19:21:30.675 EEST [3111699] LOG:Â client backend (PID 3320979) was terminated by signal 11: Segmentation fault
> 2025-04-15 19:21:30.675 EEST [3111699] DETAIL:Â Failed process was running: SELECT COUNT(*) >= 0 AS ok FROM pg_aios;
> 2025-04-15 19:21:30.675 EEST [3111699] LOG:Â terminating any other active server processes
Thanks for that. The bug turns out to be pretty stupid - pgaio_io_reclaim()
resets the fields in PgAioHandle *before* updating the generation/state. That
opens up a window in which pg_get_aios() thinks the copied PgAioHandle is
valid, even though it was taken while the fields were being reset.
Once I had figured that out, it was easy to make it more reproducible - put a
pg_usleep() between the fields being reset in pgaio_io_reclaim() and the
generation increase / state update.
The fix is simple, increment generation and state before resetting fields.
Will push the fix for that soon.
Greetings,
Andres Freund
^ permalink raw reply [nested|flat] 153+ messages in thread
* Re: AIO v2.5
2025-03-04 19:00 AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-10 18:23 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-11 19:41 ` Re: AIO v2.5 Noah Misch <noah@leadboat.com>
2025-03-11 23:55 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-12 03:57 ` Re: AIO v2.5 Noah Misch <noah@leadboat.com>
2025-03-12 17:06 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-12 23:23 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-13 10:53 ` Re: AIO v2.5 Antonin Houska <ah@cybertec.at>
2025-03-14 19:43 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-17 20:00 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-18 20:12 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-21 01:58 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-25 01:18 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-27 01:07 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-27 20:58 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-28 03:22 ` Re: AIO v2.5 Noah Misch <noah@leadboat.com>
2025-03-28 12:54 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-29 03:35 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-29 13:41 ` Re: AIO v2.5 Noah Misch <noah@leadboat.com>
2025-03-29 14:48 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-29 18:25 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-29 21:29 ` Re: AIO v2.5 Noah Misch <noah@leadboat.com>
2025-03-30 23:46 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-04-01 00:41 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-04-01 15:11 ` Re: AIO v2.5 Noah Misch <noah@leadboat.com>
2025-04-01 15:55 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-04-01 21:47 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-04-01 22:25 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-04-02 00:13 ` Re: AIO v2.5 Noah Misch <noah@leadboat.com>
2025-04-02 11:58 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-04-06 20:00 ` Re: AIO v2.5 Alexander Lakhin <exclusion@gmail.com>
2025-04-07 16:20 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-04-07 19:10 ` Re: AIO v2.5 Alexander Lakhin <exclusion@gmail.com>
2025-04-13 06:00 ` Re: AIO v2.5 Alexander Lakhin <exclusion@gmail.com>
2025-04-14 16:06 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-04-15 18:00 ` Re: AIO v2.5 Alexander Lakhin <exclusion@gmail.com>
@ 2025-04-30 20:00 ` Andres Freund <andres@anarazel.de>
2025-05-03 03:05 ` Re: AIO v2.5 Noah Misch <noah@leadboat.com>
1 sibling, 1 reply; 153+ messages in thread
From: Andres Freund @ 2025-04-30 20:00 UTC (permalink / raw)
To: Alexander Lakhin <exclusion@gmail.com>; +Cc: Noah Misch <noah@leadboat.com>; pgsql-hackers@postgresql.org, Thomas Munro <thomas.munro@gmail.com>; Heikki Linnakangas <hlinnaka@iki.fi>; Robert Haas <robertmhaas@gmail.com>; Jakub Wartak <jakub.wartak@enterprisedb.com>; Jelte Fennema-Nio <postgres@jeltef.nl>; Antonin Houska <ah@cybertec.at>
Hi,
After a bit more private back and forth with Alexander I have found the issue
- and it's pretty stupid:
pgaio_io_wait_for_free() does what it says on the tin. For that, after a bunch
of other things, finds the oldest in-flight IO and waits for it.
PgAioHandle *ioh = dclist_head_element(PgAioHandle, node,
&pgaio_my_backend->in_flight_ios);
switch (ioh->state)
{
...
case PGAIO_HS_COMPLETED_IO:
case PGAIO_HS_SUBMITTED:
pgaio_debug_io(DEBUG2, ioh,
"waiting for free io with %d in flight",
dclist_count(&pgaio_my_backend->in_flight_ios));
...
pgaio_io_wait(ioh, ioh->generation);
break;
The problem is that, if the log level is low enough, ereport() (which is
called by pgaio_debug_io()), processes interrupts. The interrupt processing
may end up execute ProcessBarrierSmgrRelease(), which in turn needs to wait
for all in-flight IOs before the IOs are closed.
Which then leads to the
elog(PANIC, "waiting for own IO in wrong state: %d",
state);
error.
The waiting for in-flight IOs before closing FDs only happens with io-uring,
hence this only triggering with io-uring.
A similar set of steps can lead to the "no free IOs despite no in-flight IOs"
ERROR that Alexander also observed - if pgaio_submit_staged() triggers a debug
ereport that executes ProcessBarrierSmgrRelease() in an interrupt, we might
wait for all in-flight IOs during IO submission, triggering the error.
I'm somewhat amazed that Alexander could somewhat reliably reproduce this - I
haven't been able to do so once using Alexander's recipe. I did find a
reproducer though:
c=32;pgbench -n -c$c -j$c -P1 -T1000 -f <(echo 'SELECT sum(abalance) FROM pgbench_accounts;')
c=1;pgbench -n -c$c -j$c -P1 -T1000 -f <(echo 'DROP DATABASE IF EXISTS foo;CREATE DATABASE foo;')
trigger both issues quickly if run with
log_min_messages=debug2
io_method=io_uring
and not when a non-debug log level is used.
I'm not yet sure how to best fix it - locally I have done so by pgaio_debug()
do a HOLD_INTERRUPTS()/RESUME_INTERRUPTS() around the call to ereport. But
that doesn't really seem great - otoh requiring various pieces of code to know
that anything emitting debug messages needs to hold interrupts etc makes for
rare and hard to understand bugs.
We could just make the relevant functions hold interrupts, and that might be
the best path forward, but we don't really need to hold all interrupts
(e.g. termination would be fine), so it's a bit coarse grained. It would need
to happen in a few places, which isn't great either.
Other suggestions?
Thanks again for finding and reporting this Alexander!
Greetings,
Andres Freund
^ permalink raw reply [nested|flat] 153+ messages in thread
* Re: AIO v2.5
2025-03-04 19:00 AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-10 18:23 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-11 19:41 ` Re: AIO v2.5 Noah Misch <noah@leadboat.com>
2025-03-11 23:55 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-12 03:57 ` Re: AIO v2.5 Noah Misch <noah@leadboat.com>
2025-03-12 17:06 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-12 23:23 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-13 10:53 ` Re: AIO v2.5 Antonin Houska <ah@cybertec.at>
2025-03-14 19:43 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-17 20:00 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-18 20:12 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-21 01:58 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-25 01:18 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-27 01:07 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-27 20:58 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-28 03:22 ` Re: AIO v2.5 Noah Misch <noah@leadboat.com>
2025-03-28 12:54 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-29 03:35 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-29 13:41 ` Re: AIO v2.5 Noah Misch <noah@leadboat.com>
2025-03-29 14:48 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-29 18:25 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-29 21:29 ` Re: AIO v2.5 Noah Misch <noah@leadboat.com>
2025-03-30 23:46 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-04-01 00:41 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-04-01 15:11 ` Re: AIO v2.5 Noah Misch <noah@leadboat.com>
2025-04-01 15:55 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-04-01 21:47 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-04-01 22:25 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-04-02 00:13 ` Re: AIO v2.5 Noah Misch <noah@leadboat.com>
2025-04-02 11:58 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-04-06 20:00 ` Re: AIO v2.5 Alexander Lakhin <exclusion@gmail.com>
2025-04-07 16:20 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-04-07 19:10 ` Re: AIO v2.5 Alexander Lakhin <exclusion@gmail.com>
2025-04-13 06:00 ` Re: AIO v2.5 Alexander Lakhin <exclusion@gmail.com>
2025-04-14 16:06 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-04-15 18:00 ` Re: AIO v2.5 Alexander Lakhin <exclusion@gmail.com>
2025-04-30 20:00 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
@ 2025-05-03 03:05 ` Noah Misch <noah@leadboat.com>
2025-05-09 01:06 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
0 siblings, 1 reply; 153+ messages in thread
From: Noah Misch @ 2025-05-03 03:05 UTC (permalink / raw)
To: Andres Freund <andres@anarazel.de>; +Cc: Alexander Lakhin <exclusion@gmail.com>; pgsql-hackers@postgresql.org, Thomas Munro <thomas.munro@gmail.com>; Heikki Linnakangas <hlinnaka@iki.fi>; Robert Haas <robertmhaas@gmail.com>; Jakub Wartak <jakub.wartak@enterprisedb.com>; Jelte Fennema-Nio <postgres@jeltef.nl>; Antonin Houska <ah@cybertec.at>
On Wed, Apr 30, 2025 at 04:00:35PM -0400, Andres Freund wrote:
> pgaio_io_wait_for_free() does what it says on the tin. For that, after a bunch
> of other things, finds the oldest in-flight IO and waits for it.
>
> PgAioHandle *ioh = dclist_head_element(PgAioHandle, node,
> &pgaio_my_backend->in_flight_ios);
>
> switch (ioh->state)
> {
> ...
> case PGAIO_HS_COMPLETED_IO:
> case PGAIO_HS_SUBMITTED:
> pgaio_debug_io(DEBUG2, ioh,
> "waiting for free io with %d in flight",
> dclist_count(&pgaio_my_backend->in_flight_ios));
> ...
> pgaio_io_wait(ioh, ioh->generation);
> break;
>
>
> The problem is that, if the log level is low enough, ereport() (which is
> called by pgaio_debug_io()), processes interrupts. The interrupt processing
> may end up execute ProcessBarrierSmgrRelease(), which in turn needs to wait
> for all in-flight IOs before the IOs are closed.
>
> Which then leads to the
> elog(PANIC, "waiting for own IO in wrong state: %d",
> state);
>
> error.
Printing state 0 (PGAIO_HS_IDLE), right? I think the chief problem is that
pgaio_io_wait_for_free() is fetching ioh->state, then possibly processing
interrupts in pgaio_debug_io(), then finally fetching ioh->generation. If it
fetched ioh->generation to a local variable before pgaio_debug_io, I think
that would resolve this one. Then the pgaio_io_was_recycled() would prevent
the PANIC:
if (pgaio_io_was_recycled(ioh, ref_generation, &state))
return;
if (am_owner)
{
if (state != PGAIO_HS_SUBMITTED
&& state != PGAIO_HS_COMPLETED_IO
&& state != PGAIO_HS_COMPLETED_SHARED
&& state != PGAIO_HS_COMPLETED_LOCAL)
{
elog(PANIC, "waiting for own IO in wrong state: %d",
state);
}
}
Is that right? If that's the solution, pgaio_closing_fd() and
pgaio_shutdown() would need similar care around fetching the generation before
the pgaio_debug_io. Maybe there's an opportunity for a common inline
function. Or at least a comment at the "generation" field on how to safely
time a fetch thereof and any barrier required.
> A similar set of steps can lead to the "no free IOs despite no in-flight IOs"
> ERROR that Alexander also observed - if pgaio_submit_staged() triggers a debug
> ereport that executes ProcessBarrierSmgrRelease() in an interrupt, we might
> wait for all in-flight IOs during IO submission, triggering the error.
That makes sense.
> I'm not yet sure how to best fix it - locally I have done so by pgaio_debug()
> do a HOLD_INTERRUPTS()/RESUME_INTERRUPTS() around the call to ereport. But
> that doesn't really seem great - otoh requiring various pieces of code to know
> that anything emitting debug messages needs to hold interrupts etc makes for
> rare and hard to understand bugs.
>
> We could just make the relevant functions hold interrupts, and that might be
> the best path forward, but we don't really need to hold all interrupts
> (e.g. termination would be fine), so it's a bit coarse grained. It would need
> to happen in a few places, which isn't great either.
>
> Other suggestions?
For the "no free IOs despite no in-flight IOs" case, I'd replace the
ereport(ERROR) with "return;" since we now know interrupt processing reclaimed
an IO. Then decide what protection if any, we need against bugs causing an
infinite loop in caller pgaio_io_acquire(). What's the case motivating the
unbounded loop in pgaio_io_acquire(), as opposed to capping at two
pgaio_io_acquire_nb() calls? If the theory is that pgaio_io_acquire() could
be reentrant, what scenario would reach that reentrancy?
> Thanks again for finding and reporting this Alexander!
+1!
^ permalink raw reply [nested|flat] 153+ messages in thread
* Re: AIO v2.5
2025-03-04 19:00 AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-10 18:23 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-11 19:41 ` Re: AIO v2.5 Noah Misch <noah@leadboat.com>
2025-03-11 23:55 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-12 03:57 ` Re: AIO v2.5 Noah Misch <noah@leadboat.com>
2025-03-12 17:06 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-12 23:23 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-13 10:53 ` Re: AIO v2.5 Antonin Houska <ah@cybertec.at>
2025-03-14 19:43 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-17 20:00 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-18 20:12 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-21 01:58 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-25 01:18 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-27 01:07 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-27 20:58 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-28 03:22 ` Re: AIO v2.5 Noah Misch <noah@leadboat.com>
2025-03-28 12:54 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-29 03:35 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-29 13:41 ` Re: AIO v2.5 Noah Misch <noah@leadboat.com>
2025-03-29 14:48 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-29 18:25 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-29 21:29 ` Re: AIO v2.5 Noah Misch <noah@leadboat.com>
2025-03-30 23:46 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-04-01 00:41 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-04-01 15:11 ` Re: AIO v2.5 Noah Misch <noah@leadboat.com>
2025-04-01 15:55 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-04-01 21:47 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-04-01 22:25 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-04-02 00:13 ` Re: AIO v2.5 Noah Misch <noah@leadboat.com>
2025-04-02 11:58 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-04-06 20:00 ` Re: AIO v2.5 Alexander Lakhin <exclusion@gmail.com>
2025-04-07 16:20 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-04-07 19:10 ` Re: AIO v2.5 Alexander Lakhin <exclusion@gmail.com>
2025-04-13 06:00 ` Re: AIO v2.5 Alexander Lakhin <exclusion@gmail.com>
2025-04-14 16:06 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-04-15 18:00 ` Re: AIO v2.5 Alexander Lakhin <exclusion@gmail.com>
2025-04-30 20:00 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-05-03 03:05 ` Re: AIO v2.5 Noah Misch <noah@leadboat.com>
@ 2025-05-09 01:06 ` Andres Freund <andres@anarazel.de>
2025-05-09 02:22 ` Re: AIO v2.5 Noah Misch <noah@leadboat.com>
0 siblings, 1 reply; 153+ messages in thread
From: Andres Freund @ 2025-05-09 01:06 UTC (permalink / raw)
To: Noah Misch <noah@leadboat.com>; +Cc: Alexander Lakhin <exclusion@gmail.com>; pgsql-hackers@postgresql.org, Thomas Munro <thomas.munro@gmail.com>; Heikki Linnakangas <hlinnaka@iki.fi>; Robert Haas <robertmhaas@gmail.com>; Jakub Wartak <jakub.wartak@enterprisedb.com>; Jelte Fennema-Nio <postgres@jeltef.nl>; Antonin Houska <ah@cybertec.at>
Hi,
On 2025-05-02 20:05:11 -0700, Noah Misch wrote:
> On Wed, Apr 30, 2025 at 04:00:35PM -0400, Andres Freund wrote:
> > pgaio_io_wait_for_free() does what it says on the tin. For that, after a bunch
> > of other things, finds the oldest in-flight IO and waits for it.
> >
> > PgAioHandle *ioh = dclist_head_element(PgAioHandle, node,
> > &pgaio_my_backend->in_flight_ios);
> >
> > switch (ioh->state)
> > {
> > ...
> > case PGAIO_HS_COMPLETED_IO:
> > case PGAIO_HS_SUBMITTED:
> > pgaio_debug_io(DEBUG2, ioh,
> > "waiting for free io with %d in flight",
> > dclist_count(&pgaio_my_backend->in_flight_ios));
> > ...
> > pgaio_io_wait(ioh, ioh->generation);
> > break;
> >
> >
> > The problem is that, if the log level is low enough, ereport() (which is
> > called by pgaio_debug_io()), processes interrupts. The interrupt processing
> > may end up execute ProcessBarrierSmgrRelease(), which in turn needs to wait
> > for all in-flight IOs before the IOs are closed.
> >
> > Which then leads to the
> > elog(PANIC, "waiting for own IO in wrong state: %d",
> > state);
> >
> > error.
>
> Printing state 0 (PGAIO_HS_IDLE), right?
Correct.
>I think the chief problem is that pgaio_io_wait_for_free() is fetching
>ioh->state, then possibly processing interrupts in pgaio_debug_io(), then
>finally fetching ioh->generation. If it fetched ioh->generation to a local
>variable before pgaio_debug_io, I think that would resolve this one.
That's what I also concluded after playing around with a few different
approaches.
> Then the pgaio_io_was_recycled() would prevent the PANIC:
>
> if (pgaio_io_was_recycled(ioh, ref_generation, &state))
> return;
>
> if (am_owner)
> {
> if (state != PGAIO_HS_SUBMITTED
> && state != PGAIO_HS_COMPLETED_IO
> && state != PGAIO_HS_COMPLETED_SHARED
> && state != PGAIO_HS_COMPLETED_LOCAL)
> {
> elog(PANIC, "waiting for own IO in wrong state: %d",
> state);
> }
> }
>
> Is that right?
Yes, it avoids the issue.
> If that's the solution, pgaio_closing_fd() and pgaio_shutdown() would need
> similar care around fetching the generation before the pgaio_debug_io.
> Maybe there's an opportunity for a common inline function. Or at least a
> comment at the "generation" field on how to safely time a fetch thereof and
> any barrier required.
I didn't see a good way to move this into an inline function, unfortunately.
We do need to hold interrupts in a few other places, I think - with some debug
infrastructure (things like calling ProcessBarrierSmgrRelease() whenever
interrupts could be processed and calling CFI() in errstart() in its return
false case) it's possible to find state confusions which trigger
assertions. The issue is that pgaio_io_update_state() contains a
pgaio_debug_io() and executing pgaio_closing_fd() in places that call
pgaio_io_update_state() doesn't end well. There's a similar danger with the
debug message in pgaio_io_reclaim().
In the attached patch I added an assertion to pgaio_io_update_state()
verifying that interrupts are held and added code to hold interrupts in the
relevant places.
> > I'm not yet sure how to best fix it - locally I have done so by pgaio_debug()
> > do a HOLD_INTERRUPTS()/RESUME_INTERRUPTS() around the call to ereport. But
> > that doesn't really seem great - otoh requiring various pieces of code to know
> > that anything emitting debug messages needs to hold interrupts etc makes for
> > rare and hard to understand bugs.
> >
> > We could just make the relevant functions hold interrupts, and that might be
> > the best path forward, but we don't really need to hold all interrupts
> > (e.g. termination would be fine), so it's a bit coarse grained. It would need
> > to happen in a few places, which isn't great either.
> >
> > Other suggestions?
>
> For the "no free IOs despite no in-flight IOs" case, I'd replace the
> ereport(ERROR) with "return;" since we now know interrupt processing reclaimed
> an IO.
Hm - it seems better to me to check if there are now free handles and return
if that's the case, but to keep the error check in case there actually is no
free IO? That seems like a not implausible bug...
> Then decide what protection if any, we need against bugs causing an
> infinite loop in caller pgaio_io_acquire(). What's the case motivating the
> unbounded loop in pgaio_io_acquire(), as opposed to capping at two
> pgaio_io_acquire_nb() calls? If the theory is that pgaio_io_acquire() could
> be reentrant, what scenario would reach that reentrancy?
I do not remember why I wrote this as an endless loop. If you prefer I could
change that as part of this patch.
It does seem rather dangerous that errstart() processes interrupts for debug
messages, but only if the debug message is actually logged. That's really a
recipe for hard to find bugs. I wonder if we should, at least in assertion
mode, process interrupts even if not emitting the message.
Greetings,
Andres Freund
Attachments:
[text/x-diff] v1-0001-WIP-aio-Fix-possible-state-confusion-due-to-inter.patch (10.0K, ../../mvpm7ga3dfgz7bvum22hmuz26cariylmcppb3irayftc7bwk3r@l7gb6gr7azhc/2-v1-0001-WIP-aio-Fix-possible-state-confusion-due-to-inter.patch)
download | inline diff:
From 650c48ffc42f554f86b8485a6c6b98c36c35fb81 Mon Sep 17 00:00:00 2001
From: Andres Freund <andres@anarazel.de>
Date: Thu, 8 May 2025 21:03:52 -0400
Subject: [PATCH v1] WIP: aio: Fix possible state confusion due to interrupt
processing
Author:
Reviewed-by:
Discussion: https://postgr.es/m/
Backpatch:
---
src/backend/storage/aio/aio.c | 113 +++++++++++++++++++++++++++++-----
1 file changed, 99 insertions(+), 14 deletions(-)
diff --git a/src/backend/storage/aio/aio.c b/src/backend/storage/aio/aio.c
index 57b9cf3dcab..10535066762 100644
--- a/src/backend/storage/aio/aio.c
+++ b/src/backend/storage/aio/aio.c
@@ -198,6 +198,8 @@ pgaio_io_acquire(struct ResourceOwnerData *resowner, PgAioReturn *ret)
PgAioHandle *
pgaio_io_acquire_nb(struct ResourceOwnerData *resowner, PgAioReturn *ret)
{
+ PgAioHandle *ioh = NULL;
+
if (pgaio_my_backend->num_staged_ios >= PGAIO_SUBMIT_BATCH_SIZE)
{
Assert(pgaio_my_backend->num_staged_ios == PGAIO_SUBMIT_BATCH_SIZE);
@@ -207,10 +209,17 @@ pgaio_io_acquire_nb(struct ResourceOwnerData *resowner, PgAioReturn *ret)
if (pgaio_my_backend->handed_out_io)
elog(ERROR, "API violation: Only one IO can be handed out");
+ /*
+ * Probably not needed today, as interrupts should not process this IO,
+ * but...
+ */
+ HOLD_INTERRUPTS();
+
if (!dclist_is_empty(&pgaio_my_backend->idle_ios))
{
dlist_node *ion = dclist_pop_head_node(&pgaio_my_backend->idle_ios);
- PgAioHandle *ioh = dclist_container(PgAioHandle, node, ion);
+
+ ioh = dclist_container(PgAioHandle, node, ion);
Assert(ioh->state == PGAIO_HS_IDLE);
Assert(ioh->owner_procno == MyProcNumber);
@@ -226,11 +235,11 @@ pgaio_io_acquire_nb(struct ResourceOwnerData *resowner, PgAioReturn *ret)
ioh->report_return = ret;
ret->result.status = PGAIO_RS_UNKNOWN;
}
-
- return ioh;
}
- return NULL;
+ RESUME_INTERRUPTS();
+
+ return ioh;
}
/*
@@ -247,6 +256,12 @@ pgaio_io_release(PgAioHandle *ioh)
Assert(ioh->resowner);
pgaio_my_backend->handed_out_io = NULL;
+
+ /*
+ * Note that no interrupts are processed between the handed_out_io
+ * check and the call to reclaim - that's important as otherwise an
+ * interrupt could have already reclaimed the handle.
+ */
pgaio_io_reclaim(ioh);
}
else
@@ -265,6 +280,12 @@ pgaio_io_release_resowner(dlist_node *ioh_node, bool on_error)
Assert(ioh->resowner);
+ /*
+ * Otherwise an interrupt, in the middle of releasing the IO, could end up
+ * trying to wait for the IO, leading to state confusion.
+ */
+ HOLD_INTERRUPTS();
+
ResourceOwnerForgetAioHandle(ioh->resowner, &ioh->resowner_node);
ioh->resowner = NULL;
@@ -305,6 +326,8 @@ pgaio_io_release_resowner(dlist_node *ioh_node, bool on_error)
*/
if (ioh->report_return)
ioh->report_return = NULL;
+
+ RESUME_INTERRUPTS();
}
/*
@@ -373,6 +396,13 @@ pgaio_io_get_wref(PgAioHandle *ioh, PgAioWaitRef *iow)
static inline void
pgaio_io_update_state(PgAioHandle *ioh, PgAioHandleState new_state)
{
+ /*
+ * All callers need to have held interrupts in some form, otherwise
+ * interrupt processing could wait for the IO to complete, while in an
+ * intermediary state.
+ */
+ Assert(!INTERRUPTS_CAN_BE_PROCESSED());
+
pgaio_debug_io(DEBUG5, ioh,
"updating state to %s",
pgaio_io_state_get_name(new_state));
@@ -410,6 +440,13 @@ pgaio_io_stage(PgAioHandle *ioh, PgAioOp op)
Assert(pgaio_my_backend->handed_out_io == ioh);
Assert(pgaio_io_has_target(ioh));
+ /*
+ * Otherwise an interrupt, in the middle of staging and possibly executing
+ * the IO, could end up trying to wait for the IO, leading to state
+ * confusion.
+ */
+ HOLD_INTERRUPTS();
+
ioh->op = op;
ioh->result = 0;
@@ -449,6 +486,8 @@ pgaio_io_stage(PgAioHandle *ioh, PgAioOp op)
pgaio_io_prepare_submit(ioh);
pgaio_io_perform_synchronously(ioh);
}
+
+ RESUME_INTERRUPTS();
}
bool
@@ -558,8 +597,8 @@ pgaio_io_wait(PgAioHandle *ioh, uint64 ref_generation)
&& state != PGAIO_HS_COMPLETED_SHARED
&& state != PGAIO_HS_COMPLETED_LOCAL)
{
- elog(PANIC, "waiting for own IO in wrong state: %d",
- state);
+ elog(PANIC, "waiting for own IO %d in wrong state: %s",
+ pgaio_io_get_id(ioh), pgaio_io_get_state_name(ioh));
}
}
@@ -613,7 +652,13 @@ pgaio_io_wait(PgAioHandle *ioh, uint64 ref_generation)
case PGAIO_HS_COMPLETED_SHARED:
case PGAIO_HS_COMPLETED_LOCAL:
- /* see above */
+
+ /*
+ * Note that no interrupts are processed between
+ * pgaio_io_was_recycled() and this check - that's important
+ * as otherwise an interrupt could have already reclaimed the
+ * handle.
+ */
if (am_owner)
pgaio_io_reclaim(ioh);
return;
@@ -624,6 +669,11 @@ pgaio_io_wait(PgAioHandle *ioh, uint64 ref_generation)
/*
* Make IO handle ready to be reused after IO has completed or after the
* handle has been released without being used.
+ *
+ * Note that callers need to be careful about only calling this in the right
+ * state and that no interrupts can be processed between the state check and
+ * the call to pgaio_io_reclaim(). Otherwise interrupt processing could
+ * already have reclaimed the handle.
*/
static void
pgaio_io_reclaim(PgAioHandle *ioh)
@@ -632,6 +682,9 @@ pgaio_io_reclaim(PgAioHandle *ioh)
Assert(ioh->owner_procno == MyProcNumber);
Assert(ioh->state != PGAIO_HS_IDLE);
+ /* see comment in function header */
+ HOLD_INTERRUPTS();
+
/*
* It's a bit ugly, but right now the easiest place to put the execution
* of local completion callbacks is this function, as we need to execute
@@ -699,6 +752,8 @@ pgaio_io_reclaim(PgAioHandle *ioh)
* efficient in cases where only a few IOs are used.
*/
dclist_push_head(&pgaio_my_backend->idle_ios, &ioh->node);
+
+ RESUME_INTERRUPTS();
}
/*
@@ -714,7 +769,7 @@ pgaio_io_wait_for_free(void)
pgaio_debug(DEBUG2, "waiting for free IO with %d pending, %d in-flight, %d idle IOs",
pgaio_my_backend->num_staged_ios,
dclist_count(&pgaio_my_backend->in_flight_ios),
- dclist_is_empty(&pgaio_my_backend->idle_ios));
+ dclist_count(&pgaio_my_backend->idle_ios));
/*
* First check if any of our IOs actually have completed - when using
@@ -728,6 +783,11 @@ pgaio_io_wait_for_free(void)
if (ioh->state == PGAIO_HS_COMPLETED_SHARED)
{
+ /*
+ * Note that no interrupts are processed between the state check
+ * and the call to reclaim - that's important as otherwise an
+ * interrupt could have already reclaimed the handle.
+ */
pgaio_io_reclaim(ioh);
reclaimed++;
}
@@ -744,13 +804,17 @@ pgaio_io_wait_for_free(void)
if (pgaio_my_backend->num_staged_ios > 0)
pgaio_submit_staged();
+ /* possibly some IOs finished during submission */
+ if (!dclist_is_empty(&pgaio_my_backend->idle_ios))
+ return;
+
if (dclist_count(&pgaio_my_backend->in_flight_ios) == 0)
ereport(ERROR,
errmsg_internal("no free IOs despite no in-flight IOs"),
errdetail_internal("%d pending, %d in-flight, %d idle IOs",
pgaio_my_backend->num_staged_ios,
dclist_count(&pgaio_my_backend->in_flight_ios),
- dclist_is_empty(&pgaio_my_backend->idle_ios)));
+ dclist_count(&pgaio_my_backend->idle_ios)));
/*
* Wait for the oldest in-flight IO to complete.
@@ -761,6 +825,7 @@ pgaio_io_wait_for_free(void)
{
PgAioHandle *ioh = dclist_head_element(PgAioHandle, node,
&pgaio_my_backend->in_flight_ios);
+ uint64 generation = ioh->generation;
switch (ioh->state)
{
@@ -784,13 +849,24 @@ pgaio_io_wait_for_free(void)
* In a more general case this would be racy, because the
* generation could increase after we read ioh->state above.
* But we are only looking at IOs by the current backend and
- * the IO can only be recycled by this backend.
+ * the IO can only be recycled by this backend. Even this is
+ * only OK because we get the handle's generation before
+ * potentially processing interrupts, e.g. as part of
+ * pgaio_debug_io().
*/
- pgaio_io_wait(ioh, ioh->generation);
+ pgaio_io_wait(ioh, generation);
break;
case PGAIO_HS_COMPLETED_SHARED:
- /* it's possible that another backend just finished this IO */
+
+ /*
+ * It's possible that another backend just finished this IO.
+ *
+ * Note that no interrupts are processed between the state
+ * check and the call to reclaim - that's important as
+ * otherwise an interrupt could have already reclaimed the
+ * handle.
+ */
pgaio_io_reclaim(ioh);
break;
}
@@ -940,6 +1016,11 @@ pgaio_wref_check_done(PgAioWaitRef *iow)
if (state == PGAIO_HS_COMPLETED_SHARED ||
state == PGAIO_HS_COMPLETED_LOCAL)
{
+ /*
+ * Note that no interrupts are processed between
+ * pgaio_io_was_recycled() and this check - that's important as
+ * otherwise an interrupt could have already reclaimed the handle.
+ */
if (am_owner)
pgaio_io_reclaim(ioh);
return true;
@@ -1167,11 +1248,14 @@ pgaio_closing_fd(int fd)
{
dlist_iter iter;
PgAioHandle *ioh = NULL;
+ uint64 generation;
dclist_foreach(iter, &pgaio_my_backend->in_flight_ios)
{
ioh = dclist_container(PgAioHandle, node, iter.cur);
+ generation = ioh->generation;
+
if (pgaio_io_uses_fd(ioh, fd))
break;
else
@@ -1186,7 +1270,7 @@ pgaio_closing_fd(int fd)
fd, dclist_count(&pgaio_my_backend->in_flight_ios));
/* see comment in pgaio_io_wait_for_free() about raciness */
- pgaio_io_wait(ioh, ioh->generation);
+ pgaio_io_wait(ioh, generation);
}
}
}
@@ -1215,13 +1299,14 @@ pgaio_shutdown(int code, Datum arg)
while (!dclist_is_empty(&pgaio_my_backend->in_flight_ios))
{
PgAioHandle *ioh = dclist_head_element(PgAioHandle, node, &pgaio_my_backend->in_flight_ios);
+ uint64 generation = ioh->generation;
pgaio_debug_io(DEBUG2, ioh,
"waiting for IO to complete during shutdown, %d in-flight IOs",
dclist_count(&pgaio_my_backend->in_flight_ios));
/* see comment in pgaio_io_wait_for_free() about raciness */
- pgaio_io_wait(ioh, ioh->generation);
+ pgaio_io_wait(ioh, generation);
}
pgaio_my_backend = NULL;
--
2.48.1.76.g4e746b1a31.dirty
[text/x-diff] v1-0001-WIP-aio-Fix-possible-state-confusion-due-to-inter.patch (10.0K, ../../mvpm7ga3dfgz7bvum22hmuz26cariylmcppb3irayftc7bwk3r@l7gb6gr7azhc/3-v1-0001-WIP-aio-Fix-possible-state-confusion-due-to-inter.patch)
download | inline diff:
From 650c48ffc42f554f86b8485a6c6b98c36c35fb81 Mon Sep 17 00:00:00 2001
From: Andres Freund <andres@anarazel.de>
Date: Thu, 8 May 2025 21:03:52 -0400
Subject: [PATCH v1] WIP: aio: Fix possible state confusion due to interrupt
processing
Author:
Reviewed-by:
Discussion: https://postgr.es/m/
Backpatch:
---
src/backend/storage/aio/aio.c | 113 +++++++++++++++++++++++++++++-----
1 file changed, 99 insertions(+), 14 deletions(-)
diff --git a/src/backend/storage/aio/aio.c b/src/backend/storage/aio/aio.c
index 57b9cf3dcab..10535066762 100644
--- a/src/backend/storage/aio/aio.c
+++ b/src/backend/storage/aio/aio.c
@@ -198,6 +198,8 @@ pgaio_io_acquire(struct ResourceOwnerData *resowner, PgAioReturn *ret)
PgAioHandle *
pgaio_io_acquire_nb(struct ResourceOwnerData *resowner, PgAioReturn *ret)
{
+ PgAioHandle *ioh = NULL;
+
if (pgaio_my_backend->num_staged_ios >= PGAIO_SUBMIT_BATCH_SIZE)
{
Assert(pgaio_my_backend->num_staged_ios == PGAIO_SUBMIT_BATCH_SIZE);
@@ -207,10 +209,17 @@ pgaio_io_acquire_nb(struct ResourceOwnerData *resowner, PgAioReturn *ret)
if (pgaio_my_backend->handed_out_io)
elog(ERROR, "API violation: Only one IO can be handed out");
+ /*
+ * Probably not needed today, as interrupts should not process this IO,
+ * but...
+ */
+ HOLD_INTERRUPTS();
+
if (!dclist_is_empty(&pgaio_my_backend->idle_ios))
{
dlist_node *ion = dclist_pop_head_node(&pgaio_my_backend->idle_ios);
- PgAioHandle *ioh = dclist_container(PgAioHandle, node, ion);
+
+ ioh = dclist_container(PgAioHandle, node, ion);
Assert(ioh->state == PGAIO_HS_IDLE);
Assert(ioh->owner_procno == MyProcNumber);
@@ -226,11 +235,11 @@ pgaio_io_acquire_nb(struct ResourceOwnerData *resowner, PgAioReturn *ret)
ioh->report_return = ret;
ret->result.status = PGAIO_RS_UNKNOWN;
}
-
- return ioh;
}
- return NULL;
+ RESUME_INTERRUPTS();
+
+ return ioh;
}
/*
@@ -247,6 +256,12 @@ pgaio_io_release(PgAioHandle *ioh)
Assert(ioh->resowner);
pgaio_my_backend->handed_out_io = NULL;
+
+ /*
+ * Note that no interrupts are processed between the handed_out_io
+ * check and the call to reclaim - that's important as otherwise an
+ * interrupt could have already reclaimed the handle.
+ */
pgaio_io_reclaim(ioh);
}
else
@@ -265,6 +280,12 @@ pgaio_io_release_resowner(dlist_node *ioh_node, bool on_error)
Assert(ioh->resowner);
+ /*
+ * Otherwise an interrupt, in the middle of releasing the IO, could end up
+ * trying to wait for the IO, leading to state confusion.
+ */
+ HOLD_INTERRUPTS();
+
ResourceOwnerForgetAioHandle(ioh->resowner, &ioh->resowner_node);
ioh->resowner = NULL;
@@ -305,6 +326,8 @@ pgaio_io_release_resowner(dlist_node *ioh_node, bool on_error)
*/
if (ioh->report_return)
ioh->report_return = NULL;
+
+ RESUME_INTERRUPTS();
}
/*
@@ -373,6 +396,13 @@ pgaio_io_get_wref(PgAioHandle *ioh, PgAioWaitRef *iow)
static inline void
pgaio_io_update_state(PgAioHandle *ioh, PgAioHandleState new_state)
{
+ /*
+ * All callers need to have held interrupts in some form, otherwise
+ * interrupt processing could wait for the IO to complete, while in an
+ * intermediary state.
+ */
+ Assert(!INTERRUPTS_CAN_BE_PROCESSED());
+
pgaio_debug_io(DEBUG5, ioh,
"updating state to %s",
pgaio_io_state_get_name(new_state));
@@ -410,6 +440,13 @@ pgaio_io_stage(PgAioHandle *ioh, PgAioOp op)
Assert(pgaio_my_backend->handed_out_io == ioh);
Assert(pgaio_io_has_target(ioh));
+ /*
+ * Otherwise an interrupt, in the middle of staging and possibly executing
+ * the IO, could end up trying to wait for the IO, leading to state
+ * confusion.
+ */
+ HOLD_INTERRUPTS();
+
ioh->op = op;
ioh->result = 0;
@@ -449,6 +486,8 @@ pgaio_io_stage(PgAioHandle *ioh, PgAioOp op)
pgaio_io_prepare_submit(ioh);
pgaio_io_perform_synchronously(ioh);
}
+
+ RESUME_INTERRUPTS();
}
bool
@@ -558,8 +597,8 @@ pgaio_io_wait(PgAioHandle *ioh, uint64 ref_generation)
&& state != PGAIO_HS_COMPLETED_SHARED
&& state != PGAIO_HS_COMPLETED_LOCAL)
{
- elog(PANIC, "waiting for own IO in wrong state: %d",
- state);
+ elog(PANIC, "waiting for own IO %d in wrong state: %s",
+ pgaio_io_get_id(ioh), pgaio_io_get_state_name(ioh));
}
}
@@ -613,7 +652,13 @@ pgaio_io_wait(PgAioHandle *ioh, uint64 ref_generation)
case PGAIO_HS_COMPLETED_SHARED:
case PGAIO_HS_COMPLETED_LOCAL:
- /* see above */
+
+ /*
+ * Note that no interrupts are processed between
+ * pgaio_io_was_recycled() and this check - that's important
+ * as otherwise an interrupt could have already reclaimed the
+ * handle.
+ */
if (am_owner)
pgaio_io_reclaim(ioh);
return;
@@ -624,6 +669,11 @@ pgaio_io_wait(PgAioHandle *ioh, uint64 ref_generation)
/*
* Make IO handle ready to be reused after IO has completed or after the
* handle has been released without being used.
+ *
+ * Note that callers need to be careful about only calling this in the right
+ * state and that no interrupts can be processed between the state check and
+ * the call to pgaio_io_reclaim(). Otherwise interrupt processing could
+ * already have reclaimed the handle.
*/
static void
pgaio_io_reclaim(PgAioHandle *ioh)
@@ -632,6 +682,9 @@ pgaio_io_reclaim(PgAioHandle *ioh)
Assert(ioh->owner_procno == MyProcNumber);
Assert(ioh->state != PGAIO_HS_IDLE);
+ /* see comment in function header */
+ HOLD_INTERRUPTS();
+
/*
* It's a bit ugly, but right now the easiest place to put the execution
* of local completion callbacks is this function, as we need to execute
@@ -699,6 +752,8 @@ pgaio_io_reclaim(PgAioHandle *ioh)
* efficient in cases where only a few IOs are used.
*/
dclist_push_head(&pgaio_my_backend->idle_ios, &ioh->node);
+
+ RESUME_INTERRUPTS();
}
/*
@@ -714,7 +769,7 @@ pgaio_io_wait_for_free(void)
pgaio_debug(DEBUG2, "waiting for free IO with %d pending, %d in-flight, %d idle IOs",
pgaio_my_backend->num_staged_ios,
dclist_count(&pgaio_my_backend->in_flight_ios),
- dclist_is_empty(&pgaio_my_backend->idle_ios));
+ dclist_count(&pgaio_my_backend->idle_ios));
/*
* First check if any of our IOs actually have completed - when using
@@ -728,6 +783,11 @@ pgaio_io_wait_for_free(void)
if (ioh->state == PGAIO_HS_COMPLETED_SHARED)
{
+ /*
+ * Note that no interrupts are processed between the state check
+ * and the call to reclaim - that's important as otherwise an
+ * interrupt could have already reclaimed the handle.
+ */
pgaio_io_reclaim(ioh);
reclaimed++;
}
@@ -744,13 +804,17 @@ pgaio_io_wait_for_free(void)
if (pgaio_my_backend->num_staged_ios > 0)
pgaio_submit_staged();
+ /* possibly some IOs finished during submission */
+ if (!dclist_is_empty(&pgaio_my_backend->idle_ios))
+ return;
+
if (dclist_count(&pgaio_my_backend->in_flight_ios) == 0)
ereport(ERROR,
errmsg_internal("no free IOs despite no in-flight IOs"),
errdetail_internal("%d pending, %d in-flight, %d idle IOs",
pgaio_my_backend->num_staged_ios,
dclist_count(&pgaio_my_backend->in_flight_ios),
- dclist_is_empty(&pgaio_my_backend->idle_ios)));
+ dclist_count(&pgaio_my_backend->idle_ios)));
/*
* Wait for the oldest in-flight IO to complete.
@@ -761,6 +825,7 @@ pgaio_io_wait_for_free(void)
{
PgAioHandle *ioh = dclist_head_element(PgAioHandle, node,
&pgaio_my_backend->in_flight_ios);
+ uint64 generation = ioh->generation;
switch (ioh->state)
{
@@ -784,13 +849,24 @@ pgaio_io_wait_for_free(void)
* In a more general case this would be racy, because the
* generation could increase after we read ioh->state above.
* But we are only looking at IOs by the current backend and
- * the IO can only be recycled by this backend.
+ * the IO can only be recycled by this backend. Even this is
+ * only OK because we get the handle's generation before
+ * potentially processing interrupts, e.g. as part of
+ * pgaio_debug_io().
*/
- pgaio_io_wait(ioh, ioh->generation);
+ pgaio_io_wait(ioh, generation);
break;
case PGAIO_HS_COMPLETED_SHARED:
- /* it's possible that another backend just finished this IO */
+
+ /*
+ * It's possible that another backend just finished this IO.
+ *
+ * Note that no interrupts are processed between the state
+ * check and the call to reclaim - that's important as
+ * otherwise an interrupt could have already reclaimed the
+ * handle.
+ */
pgaio_io_reclaim(ioh);
break;
}
@@ -940,6 +1016,11 @@ pgaio_wref_check_done(PgAioWaitRef *iow)
if (state == PGAIO_HS_COMPLETED_SHARED ||
state == PGAIO_HS_COMPLETED_LOCAL)
{
+ /*
+ * Note that no interrupts are processed between
+ * pgaio_io_was_recycled() and this check - that's important as
+ * otherwise an interrupt could have already reclaimed the handle.
+ */
if (am_owner)
pgaio_io_reclaim(ioh);
return true;
@@ -1167,11 +1248,14 @@ pgaio_closing_fd(int fd)
{
dlist_iter iter;
PgAioHandle *ioh = NULL;
+ uint64 generation;
dclist_foreach(iter, &pgaio_my_backend->in_flight_ios)
{
ioh = dclist_container(PgAioHandle, node, iter.cur);
+ generation = ioh->generation;
+
if (pgaio_io_uses_fd(ioh, fd))
break;
else
@@ -1186,7 +1270,7 @@ pgaio_closing_fd(int fd)
fd, dclist_count(&pgaio_my_backend->in_flight_ios));
/* see comment in pgaio_io_wait_for_free() about raciness */
- pgaio_io_wait(ioh, ioh->generation);
+ pgaio_io_wait(ioh, generation);
}
}
}
@@ -1215,13 +1299,14 @@ pgaio_shutdown(int code, Datum arg)
while (!dclist_is_empty(&pgaio_my_backend->in_flight_ios))
{
PgAioHandle *ioh = dclist_head_element(PgAioHandle, node, &pgaio_my_backend->in_flight_ios);
+ uint64 generation = ioh->generation;
pgaio_debug_io(DEBUG2, ioh,
"waiting for IO to complete during shutdown, %d in-flight IOs",
dclist_count(&pgaio_my_backend->in_flight_ios));
/* see comment in pgaio_io_wait_for_free() about raciness */
- pgaio_io_wait(ioh, ioh->generation);
+ pgaio_io_wait(ioh, generation);
}
pgaio_my_backend = NULL;
--
2.48.1.76.g4e746b1a31.dirty
^ permalink raw reply [nested|flat] 153+ messages in thread
* Re: AIO v2.5
2025-03-04 19:00 AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-10 18:23 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-11 19:41 ` Re: AIO v2.5 Noah Misch <noah@leadboat.com>
2025-03-11 23:55 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-12 03:57 ` Re: AIO v2.5 Noah Misch <noah@leadboat.com>
2025-03-12 17:06 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-12 23:23 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-13 10:53 ` Re: AIO v2.5 Antonin Houska <ah@cybertec.at>
2025-03-14 19:43 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-17 20:00 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-18 20:12 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-21 01:58 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-25 01:18 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-27 01:07 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-27 20:58 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-28 03:22 ` Re: AIO v2.5 Noah Misch <noah@leadboat.com>
2025-03-28 12:54 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-29 03:35 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-29 13:41 ` Re: AIO v2.5 Noah Misch <noah@leadboat.com>
2025-03-29 14:48 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-29 18:25 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-29 21:29 ` Re: AIO v2.5 Noah Misch <noah@leadboat.com>
2025-03-30 23:46 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-04-01 00:41 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-04-01 15:11 ` Re: AIO v2.5 Noah Misch <noah@leadboat.com>
2025-04-01 15:55 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-04-01 21:47 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-04-01 22:25 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-04-02 00:13 ` Re: AIO v2.5 Noah Misch <noah@leadboat.com>
2025-04-02 11:58 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-04-06 20:00 ` Re: AIO v2.5 Alexander Lakhin <exclusion@gmail.com>
2025-04-07 16:20 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-04-07 19:10 ` Re: AIO v2.5 Alexander Lakhin <exclusion@gmail.com>
2025-04-13 06:00 ` Re: AIO v2.5 Alexander Lakhin <exclusion@gmail.com>
2025-04-14 16:06 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-04-15 18:00 ` Re: AIO v2.5 Alexander Lakhin <exclusion@gmail.com>
2025-04-30 20:00 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-05-03 03:05 ` Re: AIO v2.5 Noah Misch <noah@leadboat.com>
2025-05-09 01:06 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
@ 2025-05-09 02:22 ` Noah Misch <noah@leadboat.com>
2025-05-20 01:09 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
0 siblings, 1 reply; 153+ messages in thread
From: Noah Misch @ 2025-05-09 02:22 UTC (permalink / raw)
To: Andres Freund <andres@anarazel.de>; +Cc: Alexander Lakhin <exclusion@gmail.com>; pgsql-hackers@postgresql.org, Thomas Munro <thomas.munro@gmail.com>; Heikki Linnakangas <hlinnaka@iki.fi>; Robert Haas <robertmhaas@gmail.com>; Jakub Wartak <jakub.wartak@enterprisedb.com>; Jelte Fennema-Nio <postgres@jeltef.nl>; Antonin Houska <ah@cybertec.at>
On Thu, May 08, 2025 at 09:06:18PM -0400, Andres Freund wrote:
> On 2025-05-02 20:05:11 -0700, Noah Misch wrote:
> > On Wed, Apr 30, 2025 at 04:00:35PM -0400, Andres Freund wrote:
> We do need to hold interrupts in a few other places, I think - with some debug
> infrastructure (things like calling ProcessBarrierSmgrRelease() whenever
> interrupts could be processed and calling CFI() in errstart() in its return
> false case) it's possible to find state confusions which trigger
> assertions. The issue is that pgaio_io_update_state() contains a
> pgaio_debug_io() and executing pgaio_closing_fd() in places that call
> pgaio_io_update_state() doesn't end well. There's a similar danger with the
> debug message in pgaio_io_reclaim().
>
> In the attached patch I added an assertion to pgaio_io_update_state()
> verifying that interrupts are held and added code to hold interrupts in the
> relevant places.
Works for me.
> > For the "no free IOs despite no in-flight IOs" case, I'd replace the
> > ereport(ERROR) with "return;" since we now know interrupt processing reclaimed
> > an IO.
>
> Hm - it seems better to me to check if there are now free handles and return
> if that's the case, but to keep the error check in case there actually is no
> free IO? That seems like a not implausible bug...
Works for me.
> > Then decide what protection if any, we need against bugs causing an
> > infinite loop in caller pgaio_io_acquire(). What's the case motivating the
> > unbounded loop in pgaio_io_acquire(), as opposed to capping at two
> > pgaio_io_acquire_nb() calls? If the theory is that pgaio_io_acquire() could
> > be reentrant, what scenario would reach that reentrancy?
>
> I do not remember why I wrote this as an endless loop. If you prefer I could
> change that as part of this patch.
I asked because it would be sad to remove the ereport(ERROR) like I proposed
and then have a bug cause a real infinite loop. Removing the loop was one way
to prove that can't happen. As you say, another way would be keeping the
ereport(ERROR) and guarding it with a free-handles check, like in your patch
today. I don't have a strong preference between those.
> It does seem rather dangerous that errstart() processes interrupts for debug
> messages, but only if the debug message is actually logged. That's really a
> recipe for hard to find bugs. I wonder if we should, at least in assertion
> mode, process interrupts even if not emitting the message.
Yes, that sounds excellent to have.
^ permalink raw reply [nested|flat] 153+ messages in thread
* Re: AIO v2.5
2025-03-04 19:00 AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-10 18:23 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-11 19:41 ` Re: AIO v2.5 Noah Misch <noah@leadboat.com>
2025-03-11 23:55 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-12 03:57 ` Re: AIO v2.5 Noah Misch <noah@leadboat.com>
2025-03-12 17:06 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-12 23:23 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-13 10:53 ` Re: AIO v2.5 Antonin Houska <ah@cybertec.at>
2025-03-14 19:43 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-17 20:00 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-18 20:12 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-21 01:58 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-25 01:18 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-27 01:07 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-27 20:58 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-28 03:22 ` Re: AIO v2.5 Noah Misch <noah@leadboat.com>
2025-03-28 12:54 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-29 03:35 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-29 13:41 ` Re: AIO v2.5 Noah Misch <noah@leadboat.com>
2025-03-29 14:48 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-29 18:25 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-29 21:29 ` Re: AIO v2.5 Noah Misch <noah@leadboat.com>
2025-03-30 23:46 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-04-01 00:41 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-04-01 15:11 ` Re: AIO v2.5 Noah Misch <noah@leadboat.com>
2025-04-01 15:55 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-04-01 21:47 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-04-01 22:25 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-04-02 00:13 ` Re: AIO v2.5 Noah Misch <noah@leadboat.com>
2025-04-02 11:58 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-04-06 20:00 ` Re: AIO v2.5 Alexander Lakhin <exclusion@gmail.com>
2025-04-07 16:20 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-04-07 19:10 ` Re: AIO v2.5 Alexander Lakhin <exclusion@gmail.com>
2025-04-13 06:00 ` Re: AIO v2.5 Alexander Lakhin <exclusion@gmail.com>
2025-04-14 16:06 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-04-15 18:00 ` Re: AIO v2.5 Alexander Lakhin <exclusion@gmail.com>
2025-04-30 20:00 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-05-03 03:05 ` Re: AIO v2.5 Noah Misch <noah@leadboat.com>
2025-05-09 01:06 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-05-09 02:22 ` Re: AIO v2.5 Noah Misch <noah@leadboat.com>
@ 2025-05-20 01:09 ` Andres Freund <andres@anarazel.de>
0 siblings, 0 replies; 153+ messages in thread
From: Andres Freund @ 2025-05-20 01:09 UTC (permalink / raw)
To: Noah Misch <noah@leadboat.com>; +Cc: Alexander Lakhin <exclusion@gmail.com>; pgsql-hackers@postgresql.org, Thomas Munro <thomas.munro@gmail.com>; Heikki Linnakangas <hlinnaka@iki.fi>; Robert Haas <robertmhaas@gmail.com>; Jakub Wartak <jakub.wartak@enterprisedb.com>; Jelte Fennema-Nio <postgres@jeltef.nl>; Antonin Houska <ah@cybertec.at>
Hi,
On 2025-05-08 19:22:27 -0700, Noah Misch wrote:
> On Thu, May 08, 2025 at 09:06:18PM -0400, Andres Freund wrote:
> > On 2025-05-02 20:05:11 -0700, Noah Misch wrote:
> > > On Wed, Apr 30, 2025 at 04:00:35PM -0400, Andres Freund wrote:
>
> > We do need to hold interrupts in a few other places, I think - with some debug
> > infrastructure (things like calling ProcessBarrierSmgrRelease() whenever
> > interrupts could be processed and calling CFI() in errstart() in its return
> > false case) it's possible to find state confusions which trigger
> > assertions. The issue is that pgaio_io_update_state() contains a
> > pgaio_debug_io() and executing pgaio_closing_fd() in places that call
> > pgaio_io_update_state() doesn't end well. There's a similar danger with the
> > debug message in pgaio_io_reclaim().
> >
> > In the attached patch I added an assertion to pgaio_io_update_state()
> > verifying that interrupts are held and added code to hold interrupts in the
> > relevant places.
>
> Works for me.
>
> > > For the "no free IOs despite no in-flight IOs" case, I'd replace the
> > > ereport(ERROR) with "return;" since we now know interrupt processing reclaimed
> > > an IO.
> >
> > Hm - it seems better to me to check if there are now free handles and return
> > if that's the case, but to keep the error check in case there actually is no
> > free IO? That seems like a not implausible bug...
>
> Works for me.
>
> > > Then decide what protection if any, we need against bugs causing an
> > > infinite loop in caller pgaio_io_acquire(). What's the case motivating the
> > > unbounded loop in pgaio_io_acquire(), as opposed to capping at two
> > > pgaio_io_acquire_nb() calls? If the theory is that pgaio_io_acquire() could
> > > be reentrant, what scenario would reach that reentrancy?
> >
> > I do not remember why I wrote this as an endless loop. If you prefer I could
> > change that as part of this patch.
>
> I asked because it would be sad to remove the ereport(ERROR) like I proposed
> and then have a bug cause a real infinite loop. Removing the loop was one way
> to prove that can't happen. As you say, another way would be keeping the
> ereport(ERROR) and guarding it with a free-handles check, like in your patch
> today. I don't have a strong preference between those.
I pushed it this way just now.
Thanks again for Alexander for reporting & investigating the issues and for
Noah's review.
Greetings,
Andres Freund
^ permalink raw reply [nested|flat] 153+ messages in thread
* Re: AIO v2.5
2025-03-04 19:00 AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-10 18:23 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-11 19:41 ` Re: AIO v2.5 Noah Misch <noah@leadboat.com>
2025-03-11 23:55 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-12 03:57 ` Re: AIO v2.5 Noah Misch <noah@leadboat.com>
2025-03-12 17:06 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-12 23:23 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-13 10:53 ` Re: AIO v2.5 Antonin Houska <ah@cybertec.at>
2025-03-14 19:43 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-17 20:00 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-18 20:12 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-21 01:58 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-25 01:18 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-27 01:07 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-27 20:58 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-28 03:22 ` Re: AIO v2.5 Noah Misch <noah@leadboat.com>
2025-03-28 12:54 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-29 03:35 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-29 13:41 ` Re: AIO v2.5 Noah Misch <noah@leadboat.com>
2025-03-29 14:48 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-29 18:25 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-29 21:29 ` Re: AIO v2.5 Noah Misch <noah@leadboat.com>
2025-03-30 23:46 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-04-01 00:41 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-04-01 15:11 ` Re: AIO v2.5 Noah Misch <noah@leadboat.com>
2025-04-01 15:55 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-04-01 21:47 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
@ 2025-04-03 18:19 ` Andres Freund <andres@anarazel.de>
2025-04-03 19:40 ` Re: AIO v2.5 Noah Misch <noah@leadboat.com>
1 sibling, 1 reply; 153+ messages in thread
From: Andres Freund @ 2025-04-03 18:19 UTC (permalink / raw)
To: Noah Misch <noah@leadboat.com>; +Cc: pgsql-hackers@postgresql.org, Thomas Munro <thomas.munro@gmail.com>; Heikki Linnakangas <hlinnaka@iki.fi>; Robert Haas <robertmhaas@gmail.com>; Jakub Wartak <jakub.wartak@enterprisedb.com>; Jelte Fennema-Nio <postgres@jeltef.nl>; Antonin Houska <ah@cybertec.at>
Hi,
On 2025-04-01 17:47:51 -0400, Andres Freund wrote:
> There are three different types of failures in the test_aio test so far:
And a fourth, visible after I enabled liburing support for skink.
https://buildfarm.postgresql.org/cgi-bin/show_stage_log.pl?nm=skink&dt=2025-04-03%2007%3A06%3A19...
(ignore the pg_upgrade and oauth failures, they're independent, I've raised
them separately)
4a)
2025-04-03 10:58:32.978 UTC [2486740][client backend][3/6:0] LOG: short read injection point called, is enabled: 0
==2486740== VALGRINDERROR-BEGIN
==2486740== Invalid read of size 2
==2486740== at 0x59C8AC: PageIsNew (bufpage.h:237)
==2486740== by 0x59C8AC: PageIsVerified (bufpage.c:108)
==2486740== by 0x567870: buffer_readv_complete_one (bufmgr.c:6873)
==2486740== by 0x567870: buffer_readv_complete (bufmgr.c:6996)
==2486740== by 0x567870: shared_buffer_readv_complete (bufmgr.c:7153)
==2486740== by 0x55DDB2: pgaio_io_call_complete_shared (aio_callback.c:256)
==2486740== by 0x55D6F1: pgaio_io_process_completion (aio.c:512)
==2486740== by 0x55F53A: pgaio_uring_drain_locked (method_io_uring.c:370)
==2486740== by 0x55F7B8: pgaio_uring_wait_one (method_io_uring.c:449)
==2486740== by 0x55C702: pgaio_io_wait (aio.c:587)
==2486740== by 0x55C8B0: pgaio_wref_wait (aio.c:900)
==2486740== by 0x8639240: read_rel_block_ll (test_aio.c:440)
==2486740== by 0x3B915C: ExecInterpExpr (execExprInterp.c:953)
==2486740== by 0x3B4E4E: ExecInterpExprStillValid (execExprInterp.c:2299)
==2486740== by 0x3F7E97: ExecEvalExprNoReturn (executor.h:445)
==2486740== Address 0x8fa400e is in a rw- anonymous segment
==2486740==
==2486740== VALGRINDERROR-END
The reason for this is that this test unpins the buffer (from the backend's
view), before waiting for the IO. While the AIO subsystem holds a pin,
UnpinBufferNoOwner() marked the buffer as inaccessible:
/*
* Mark buffer non-accessible to Valgrind.
*
* Note that the buffer may have already been marked non-accessible
* within access method code that enforces that buffers are only
* accessed while a buffer lock is held.
*/
VALGRIND_MAKE_MEM_NOACCESS(BufHdrGetBlock(buf), BLCKSZ);
I think to fix this we need to mark buffers as accessible around the
PageIsVerified() call in buffer_readv_complete_one(), IFF they're not pinned
by the backend. Unfortunately, this is complicated by the fact that local
buffers do not have valgrind integration :(, so we should only do that for
local buffers, as otherwise the local buffer stays inaccessible the next time
it is pinned.
4b)
That's not all though, after getting past this failure, I see uninitialized
memory errors for reads into temporary buffers:
==3334031== VALGRINDERROR-BEGIN
==3334031== Conditional jump or move depends on uninitialised value(s)
==3334031== at 0xD7C859: PageIsVerified (bufpage.c:108)
==3334031== by 0xD381CA: buffer_readv_complete_one (bufmgr.c:6876)
==3334031== by 0xD385D1: buffer_readv_complete (bufmgr.c:7002)
==3334031== by 0xD38D2E: local_buffer_readv_complete (bufmgr.c:7210)
==3334031== by 0xD265FA: pgaio_io_call_complete_local (aio_callback.c:306)
==3334031== by 0xD24720: pgaio_io_reclaim (aio.c:644)
==3334031== by 0xD24400: pgaio_io_process_completion (aio.c:521)
==3334031== by 0xD28D3D: pgaio_uring_drain_locked (method_io_uring.c:382)
==3334031== by 0xD2905F: pgaio_uring_wait_one (method_io_uring.c:461)
==3334031== by 0xD245E0: pgaio_io_wait (aio.c:587)
==3334031== by 0xD24FFE: pgaio_wref_wait (aio.c:900)
==3334031== by 0xD2F471: WaitReadBuffers (bufmgr.c:1695)
==3334031== by 0xD2BCF4: read_stream_next_buffer (read_stream.c:898)
==3334031== by 0x8B4861: heap_fetch_next_buffer (heapam.c:654)
==3334031== by 0x8B4FFA: heapgettup_pagemode (heapam.c:1016)
==3334031== by 0x8B594F: heap_getnextslot (heapam.c:1375)
==3334031== by 0xB28AA4: table_scan_getnextslot (tableam.h:1031)
==3334031== by 0xB29177: SeqNext (nodeSeqscan.c:81)
==3334031== by 0xB28F75: ExecScanFetch (execScan.h:126)
==3334031== by 0xB28FDD: ExecScanExtended (execScan.h:170)
The reason for this one is, I think, that valgrind doesn't understand io_uring
sufficiently. Which isn't surprising, io_uring's nature of an in-memory queue
of commands is somewhat hard to intercept by tools like valgrind and rr.
The best fix for that one would, I think, be to have method_io_uring() iterate
over the IOV and mark the relevant regions as defined? That does fix the
issue at least and does seem to make sense? Not quite sure if we should mark
the entire IOV is efined or just the portion that was actually read - the
latter is additional fiddly code, and it's not clear it's likely to be helpful?
4c)
Unfortunately, once 4a) is addressed, the VALGRIND_MAKE_MEM_NOACCESS() after
PageIsVerified() causes the *next* read into the same buffer in an IO worker
to fail:
==3339904== Syscall param pread64(buf) points to unaddressable byte(s)
==3339904== at 0x5B3B687: __internal_syscall_cancel (cancellation.c:64)
==3339904== by 0x5B3B6AC: __syscall_cancel (cancellation.c:75)
==3339904== by 0x5B93C83: pread (pread64.c:25)
==3339904== by 0xD274F4: pg_preadv (pg_iovec.h:56)
==3339904== by 0xD2799A: pgaio_io_perform_synchronously (aio_io.c:137)
==3339904== by 0xD2A6D7: IoWorkerMain (method_worker.c:538)
==3339904== by 0xC91E26: postmaster_child_launch (launch_backend.c:290)
==3339904== by 0xC99594: StartChildProcess (postmaster.c:3972)
==3339904== by 0xC99EE3: maybe_adjust_io_workers (postmaster.c:4403)
==3339904== by 0xC958A8: PostmasterMain (postmaster.c:1381)
==3339904== by 0xB69622: main (main.c:227)
==3339904== Address 0x7f936d386000 is in a rw- anonymous segment
Because, from the view of the IO worker, that memory is still marked NOACCESS,
even if it since has been marked accessible in the backend.
We could adress this by conditioning the VALGRIND_MAKE_MEM_NOACCESS() on not
being in an IO worker, but it seems better to instead explicitly mark the
region accessible in the worker, before executing the IO.
In a first hack, I did that in pgaio_io_perform_synchronously(), but that is
likely too broad. I don't think the same scenario exists when IOs are
executed synchronously in the foreground.
Questions:
1) It'd be cleaner to implement valgrind support in localbuf.c, so we don't
need to have special-case logic for that. But it also makes the change less
localized and more "impactful", who knows what kind of skullduggery we have
been getting away with unnoticed.
I haven't written the code up yet, but I don't think it'd be all that much
code to add valgrind support to localbuf.
2) Any better ideas to handle the above issues than what I outlined?
Greetings,
Andres Freund
^ permalink raw reply [nested|flat] 153+ messages in thread
* Re: AIO v2.5
2025-03-04 19:00 AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-10 18:23 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-11 19:41 ` Re: AIO v2.5 Noah Misch <noah@leadboat.com>
2025-03-11 23:55 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-12 03:57 ` Re: AIO v2.5 Noah Misch <noah@leadboat.com>
2025-03-12 17:06 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-12 23:23 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-13 10:53 ` Re: AIO v2.5 Antonin Houska <ah@cybertec.at>
2025-03-14 19:43 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-17 20:00 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-18 20:12 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-21 01:58 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-25 01:18 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-27 01:07 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-27 20:58 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-28 03:22 ` Re: AIO v2.5 Noah Misch <noah@leadboat.com>
2025-03-28 12:54 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-29 03:35 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-29 13:41 ` Re: AIO v2.5 Noah Misch <noah@leadboat.com>
2025-03-29 14:48 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-29 18:25 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-29 21:29 ` Re: AIO v2.5 Noah Misch <noah@leadboat.com>
2025-03-30 23:46 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-04-01 00:41 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-04-01 15:11 ` Re: AIO v2.5 Noah Misch <noah@leadboat.com>
2025-04-01 15:55 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-04-01 21:47 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-04-03 18:19 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
@ 2025-04-03 19:40 ` Noah Misch <noah@leadboat.com>
2025-04-04 19:16 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
0 siblings, 1 reply; 153+ messages in thread
From: Noah Misch @ 2025-04-03 19:40 UTC (permalink / raw)
To: Andres Freund <andres@anarazel.de>; +Cc: pgsql-hackers@postgresql.org, Thomas Munro <thomas.munro@gmail.com>; Heikki Linnakangas <hlinnaka@iki.fi>; Robert Haas <robertmhaas@gmail.com>; Jakub Wartak <jakub.wartak@enterprisedb.com>; Jelte Fennema-Nio <postgres@jeltef.nl>; Antonin Houska <ah@cybertec.at>
On Thu, Apr 03, 2025 at 02:19:43PM -0400, Andres Freund wrote:
> 4b)
>
> That's not all though, after getting past this failure, I see uninitialized
> memory errors for reads into temporary buffers:
>
> ==3334031== VALGRINDERROR-BEGIN
> ==3334031== Conditional jump or move depends on uninitialised value(s)
> ==3334031== at 0xD7C859: PageIsVerified (bufpage.c:108)
> ==3334031== by 0xD381CA: buffer_readv_complete_one (bufmgr.c:6876)
> ==3334031== by 0xD385D1: buffer_readv_complete (bufmgr.c:7002)
> ==3334031== by 0xD38D2E: local_buffer_readv_complete (bufmgr.c:7210)
> ==3334031== by 0xD265FA: pgaio_io_call_complete_local (aio_callback.c:306)
> ==3334031== by 0xD24720: pgaio_io_reclaim (aio.c:644)
> ==3334031== by 0xD24400: pgaio_io_process_completion (aio.c:521)
> ==3334031== by 0xD28D3D: pgaio_uring_drain_locked (method_io_uring.c:382)
> ==3334031== by 0xD2905F: pgaio_uring_wait_one (method_io_uring.c:461)
> ==3334031== by 0xD245E0: pgaio_io_wait (aio.c:587)
> ==3334031== by 0xD24FFE: pgaio_wref_wait (aio.c:900)
> ==3334031== by 0xD2F471: WaitReadBuffers (bufmgr.c:1695)
> ==3334031== by 0xD2BCF4: read_stream_next_buffer (read_stream.c:898)
> ==3334031== by 0x8B4861: heap_fetch_next_buffer (heapam.c:654)
> ==3334031== by 0x8B4FFA: heapgettup_pagemode (heapam.c:1016)
> ==3334031== by 0x8B594F: heap_getnextslot (heapam.c:1375)
> ==3334031== by 0xB28AA4: table_scan_getnextslot (tableam.h:1031)
> ==3334031== by 0xB29177: SeqNext (nodeSeqscan.c:81)
> ==3334031== by 0xB28F75: ExecScanFetch (execScan.h:126)
> ==3334031== by 0xB28FDD: ExecScanExtended (execScan.h:170)
>
>
> The reason for this one is, I think, that valgrind doesn't understand io_uring
> sufficiently. Which isn't surprising, io_uring's nature of an in-memory queue
> of commands is somewhat hard to intercept by tools like valgrind and rr.
>
> The best fix for that one would, I think, be to have method_io_uring() iterate
> over the IOV and mark the relevant regions as defined? That does fix the
> issue at least and does seem to make sense?
Makes sense. Valgrind knows that read() makes its target bytes "defined". It
probably doesn't have an io_uring equivalent for that.
I expect we only need this for local buffers, and it's unclear to me how the
fix for (4a) didn't fix this. Before bufmgr Valgrind integration (1e0dfd1 of
2020-07) there was no explicit handling of shared_buffers. I suspect that
worked because the initial mmap() of shared memory was considered "defined"
(zeros), and steps like PageAddItem() copy only defined bytes into buffers.
Hence, shared_buffers remained defined without explicit Valgrind client
requests. This example uses local buffers. Storage for those comes from
MemoryContextAlloc() in GetLocalBufferStorage(). That memory starts
undefined, but it becomes defined at PageInit() or read(). Hence, I expected
the fix for (4a) to make the buffer defined after io_uring read. What makes
the outcome different?
In the general case, we could want client requests as follows:
- If completor==definer and has not dropped pin:
- Make defined before verifying page. That's all. It might be cleaner to
do this when first retrieving a return value from io_uring, since this
just makes up for what Valgrind already does for readv().
- If completor!=definer or has dropped pin:
- Make NOACCESS in definer when definer cedes its own pin.
- For io_method=worker, make UNDEFINED before starting readv(). It might be
cleanest to do this when the worker first acts as the owner of the AIO
subsystem pin, if that's a clear moment earlier than readv().
- Make DEFINED in completor before verifying page. It might be cleaner to
do this when the completor first retrieves a return value from io_uring,
since this just makes up for what Valgrind already does for readv().
- Make NOACCESS in completor after verifying page. Similarly, it might be
cleaner to do this when the completor releases the AIO subsystem pin.
> Not quite sure if we should mark
> the entire IOV is efined or just the portion that was actually read - the
> latter is additional fiddly code, and it's not clear it's likely to be helpful?
Seems fine to do the simpler way if that saves fiddly code.
> 4c)
>
> Unfortunately, once 4a) is addressed, the VALGRIND_MAKE_MEM_NOACCESS() after
> PageIsVerified() causes the *next* read into the same buffer in an IO worker
> to fail:
>
> ==3339904== Syscall param pread64(buf) points to unaddressable byte(s)
> ==3339904== at 0x5B3B687: __internal_syscall_cancel (cancellation.c:64)
> ==3339904== by 0x5B3B6AC: __syscall_cancel (cancellation.c:75)
> ==3339904== by 0x5B93C83: pread (pread64.c:25)
> ==3339904== by 0xD274F4: pg_preadv (pg_iovec.h:56)
> ==3339904== by 0xD2799A: pgaio_io_perform_synchronously (aio_io.c:137)
> ==3339904== by 0xD2A6D7: IoWorkerMain (method_worker.c:538)
> ==3339904== by 0xC91E26: postmaster_child_launch (launch_backend.c:290)
> ==3339904== by 0xC99594: StartChildProcess (postmaster.c:3972)
> ==3339904== by 0xC99EE3: maybe_adjust_io_workers (postmaster.c:4403)
> ==3339904== by 0xC958A8: PostmasterMain (postmaster.c:1381)
> ==3339904== by 0xB69622: main (main.c:227)
> ==3339904== Address 0x7f936d386000 is in a rw- anonymous segment
>
> Because, from the view of the IO worker, that memory is still marked NOACCESS,
> even if it since has been marked accessible in the backend.
>
>
> We could adress this by conditioning the VALGRIND_MAKE_MEM_NOACCESS() on not
> being in an IO worker, but it seems better to instead explicitly mark the
> region accessible in the worker, before executing the IO.
Sounds good. Since the definer gave the AIO subsystem a pin on the worker's
behalf, it's like the worker is doing an implicit pin and explicit unpin.
> In a first hack, I did that in pgaio_io_perform_synchronously(), but that is
> likely too broad. I don't think the same scenario exists when IOs are
> executed synchronously in the foreground.
>
>
> Questions:
>
> 1) It'd be cleaner to implement valgrind support in localbuf.c, so we don't
> need to have special-case logic for that. But it also makes the change less
> localized and more "impactful", who knows what kind of skullduggery we have
> been getting away with unnoticed.
>
> I haven't written the code up yet, but I don't think it'd be all that much
> code to add valgrind support to localbuf.
It would be the right thing long-term, and it's not a big deal if it causes
some false positives initially. So if you're leaning that way, that's good.
> 2) Any better ideas to handle the above issues than what I outlined?
Not here, unless the discussion under (4b) differs usefully from what you
planned.
^ permalink raw reply [nested|flat] 153+ messages in thread
* Re: AIO v2.5
2025-03-04 19:00 AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-10 18:23 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-11 19:41 ` Re: AIO v2.5 Noah Misch <noah@leadboat.com>
2025-03-11 23:55 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-12 03:57 ` Re: AIO v2.5 Noah Misch <noah@leadboat.com>
2025-03-12 17:06 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-12 23:23 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-13 10:53 ` Re: AIO v2.5 Antonin Houska <ah@cybertec.at>
2025-03-14 19:43 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-17 20:00 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-18 20:12 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-21 01:58 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-25 01:18 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-27 01:07 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-27 20:58 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-28 03:22 ` Re: AIO v2.5 Noah Misch <noah@leadboat.com>
2025-03-28 12:54 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-29 03:35 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-29 13:41 ` Re: AIO v2.5 Noah Misch <noah@leadboat.com>
2025-03-29 14:48 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-29 18:25 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-29 21:29 ` Re: AIO v2.5 Noah Misch <noah@leadboat.com>
2025-03-30 23:46 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-04-01 00:41 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-04-01 15:11 ` Re: AIO v2.5 Noah Misch <noah@leadboat.com>
2025-04-01 15:55 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-04-01 21:47 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-04-03 18:19 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-04-03 19:40 ` Re: AIO v2.5 Noah Misch <noah@leadboat.com>
@ 2025-04-04 19:16 ` Andres Freund <andres@anarazel.de>
2025-04-04 21:18 ` Re: AIO v2.5 Noah Misch <noah@leadboat.com>
0 siblings, 1 reply; 153+ messages in thread
From: Andres Freund @ 2025-04-04 19:16 UTC (permalink / raw)
To: Noah Misch <noah@leadboat.com>; +Cc: pgsql-hackers@postgresql.org, Thomas Munro <thomas.munro@gmail.com>; Heikki Linnakangas <hlinnaka@iki.fi>; Robert Haas <robertmhaas@gmail.com>; Jakub Wartak <jakub.wartak@enterprisedb.com>; Jelte Fennema-Nio <postgres@jeltef.nl>; Antonin Houska <ah@cybertec.at>
Hi,
Sorry for the slow work on this. The cycle times are humonguous due to
valgrind being so slow...
On 2025-04-03 12:40:23 -0700, Noah Misch wrote:
> On Thu, Apr 03, 2025 at 02:19:43PM -0400, Andres Freund wrote:
> > The best fix for that one would, I think, be to have method_io_uring() iterate
> > over the IOV and mark the relevant regions as defined? That does fix the
> > issue at least and does seem to make sense?
>
> Makes sense. Valgrind knows that read() makes its target bytes "defined". It
> probably doesn't have an io_uring equivalent for that.
Correct - and I think it would be nontrivial to add, because there's not easy
syscall to intercept...
> I expect we only need this for local buffers, and it's unclear to me how the
> fix for (4a) didn't fix this.
At that time I didn't apply the fix in 4a) to local buffers, because local
buffers, in HEAD, don't have the valgrind integration. Without that marking
the buffer as NOACCESS would cause all sorts of issues, because it'd be
considered inaccessible even after pinning. As you analyzed, that then ends
up considered undefined due to the MemoryContextAlloc().
> In the general case, we could want client requests as follows:
>
> - If completor==definer and has not dropped pin:
> - Make defined before verifying page. That's all. It might be cleaner to
> do this when first retrieving a return value from io_uring, since this
> just makes up for what Valgrind already does for readv().
Yea, I think it's better to do that in io_uring. It's what I have done in the
attached.
> - If completor!=definer or has dropped pin:
> - Make NOACCESS in definer when definer cedes its own pin.
That's the current behaviour for shared buffers, right?
> - For io_method=worker, make UNDEFINED before starting readv(). It might be
> cleanest to do this when the worker first acts as the owner of the AIO
> subsystem pin, if that's a clear moment earlier than readv().
Hm, what do we need this for?
> - Make DEFINED in completor before verifying page. It might be cleaner to
> do this when the completor first retrieves a return value from io_uring,
> since this just makes up for what Valgrind already does for readv().
I think we can't rely on the marking during retrieving it from io_uring, as
that might have happened in a different backend for a temp buffer. That'd only
happen if we got io_uring events for *another* IO that involved a shared rel,
but it can happen.
> > Not quite sure if we should mark
> > the entire IOV is efined or just the portion that was actually read - the
> > latter is additional fiddly code, and it's not clear it's likely to be helpful?
>
> Seems fine to do the simpler way if that saves fiddly code.
Can't quite decide, it's just at the border of what I consider too
fiddly... See the change to method_io_uring.c in the attached patch.
> > Questions:
> >
> > 1) It'd be cleaner to implement valgrind support in localbuf.c, so we don't
> > need to have special-case logic for that. But it also makes the change less
> > localized and more "impactful", who knows what kind of skullduggery we have
> > been getting away with unnoticed.
> >
> > I haven't written the code up yet, but I don't think it'd be all that much
> > code to add valgrind support to localbuf.
>
> It would be the right thing long-term, and it's not a big deal if it causes
> some false positives initially. So if you're leaning that way, that's good.
It was easy enough.
I saw one related failure, FlushRelationBuffers() didn't pin temporary buffers
before flushing them. Pinning the buffers fixed that.
I don't think it's a real problem to not pin the local buffer during
FlushRelationBuffers(), at least not today. But it seems unnecessarily odd to
not pin it.
I wish valgrind had a way to mark the buffer as inaccessible and then
accessible again, without loosing the defined-ness information...
Greetings,
Andres Freund
Attachments:
[text/x-diff] v1-0001-localbuf-Add-Valgrind-buffer-access-instrumentati.patch (3.2K, ../../s5bbhmfez5pe46yfvqkxkg72zsc2r3knkvapqyz37algbfw2sf@q7jcviovnalu/2-v1-0001-localbuf-Add-Valgrind-buffer-access-instrumentati.patch)
download | inline diff:
From 472b75150ffaca1d30147f764208341181c31571 Mon Sep 17 00:00:00 2001
From: Andres Freund <andres@anarazel.de>
Date: Fri, 4 Apr 2025 14:55:30 -0400
Subject: [PATCH v1 1/3] localbuf: Add Valgrind buffer access instrumentation
This mirrors 1e0dfd166b3 (+ 46ef520b9566), for temporary table buffers. This
is mainly interesting right now because the AIO work currently triggers
spurious valgrind errors, and the fix for that is cleaner if temp buffers
behave the same as shared buffers.
This requires one change beyond the annotations themselves, namely to pin
local buffers while writing them out in FlushRelationBuffers().
Reviewed-by:
Discussion: https://postgr.es/m/3pd4322mogfmdd5nln3zphdwhtmq3rzdldqjwb2sfqzcgs22lf@ok2gletdaoe6
---
src/backend/storage/buffer/bufmgr.c | 13 +++++++++++++
src/backend/storage/buffer/localbuf.c | 9 +++++++++
2 files changed, 22 insertions(+)
diff --git a/src/backend/storage/buffer/bufmgr.c b/src/backend/storage/buffer/bufmgr.c
index 1c37d7dfe2f..13cebad9b12 100644
--- a/src/backend/storage/buffer/bufmgr.c
+++ b/src/backend/storage/buffer/bufmgr.c
@@ -4895,8 +4895,21 @@ FlushRelationBuffers(Relation rel)
errcallback.previous = error_context_stack;
error_context_stack = &errcallback;
+ /* Make sure we can handle the pin */
+ ReservePrivateRefCountEntry();
+ ResourceOwnerEnlarge(CurrentResourceOwner);
+
+ /*
+ * Pin/upin mostly to make valgrind work, but it also seems
+ * like the right thing to do.
+ */
+ PinLocalBuffer(bufHdr, false);
+
+
FlushLocalBuffer(bufHdr, srel);
+ UnpinLocalBuffer(BufferDescriptorGetBuffer(bufHdr));
+
/* Pop the error context stack */
error_context_stack = errcallback.previous;
}
diff --git a/src/backend/storage/buffer/localbuf.c b/src/backend/storage/buffer/localbuf.c
index ed56202af14..fb44d756dad 100644
--- a/src/backend/storage/buffer/localbuf.c
+++ b/src/backend/storage/buffer/localbuf.c
@@ -23,6 +23,7 @@
#include "storage/bufmgr.h"
#include "storage/fd.h"
#include "utils/guc_hooks.h"
+#include "utils/memdebug.h"
#include "utils/memutils.h"
#include "utils/resowner.h"
@@ -183,6 +184,8 @@ FlushLocalBuffer(BufferDesc *bufHdr, SMgrRelation reln)
instr_time io_start;
Page localpage = (char *) LocalBufHdrGetBlock(bufHdr);
+ Assert(LocalRefCount[-BufferDescriptorGetBuffer(bufHdr) - 1] > 0);
+
/*
* Try to start an I/O operation. There currently are no reasons for
* StartLocalBufferIO to return false, so we raise an error in that case.
@@ -808,6 +811,9 @@ PinLocalBuffer(BufferDesc *buf_hdr, bool adjust_usagecount)
buf_state += BUF_USAGECOUNT_ONE;
}
pg_atomic_unlocked_write_u32(&buf_hdr->state, buf_state);
+
+ /* see comment in PinBuffer() */
+ VALGRIND_MAKE_MEM_DEFINED(LocalBufHdrGetBlock(buf_hdr), BLCKSZ);
}
LocalRefCount[bufid]++;
ResourceOwnerRememberBuffer(CurrentResourceOwner,
@@ -843,6 +849,9 @@ UnpinLocalBufferNoOwner(Buffer buffer)
Assert(BUF_STATE_GET_REFCOUNT(buf_state) > 0);
buf_state -= BUF_REFCOUNT_ONE;
pg_atomic_unlocked_write_u32(&buf_hdr->state, buf_state);
+
+ /* see comment in UnpinBufferNoOwner */
+ VALGRIND_MAKE_MEM_NOACCESS(LocalBufHdrGetBlock(buf_hdr), BLCKSZ);
}
}
--
2.48.1.76.g4e746b1a31.dirty
[text/x-diff] v1-0002-aio-Make-AIO-compatible-with-valgrind.patch (7.4K, ../../s5bbhmfez5pe46yfvqkxkg72zsc2r3knkvapqyz37algbfw2sf@q7jcviovnalu/3-v1-0002-aio-Make-AIO-compatible-with-valgrind.patch)
download | inline diff:
From 31759bafc9a44442f14b6363689bc343f0f9fb86 Mon Sep 17 00:00:00 2001
From: Andres Freund <andres@anarazel.de>
Date: Fri, 4 Apr 2025 15:03:34 -0400
Subject: [PATCH v1 2/3] aio: Make AIO compatible with valgrind
In some edge cases valgrind flags issues with AIO related code. All of the
cases addressed in this change are false positives.
Most are caused by UnpinBuffer[NoOwner] marking buffer data as
inaccessible. This happens even though the AIO subsystem still holds a
pin. That's good, there shouldn't be accesses to the buffer outside of AIO
related code until it is pinned bu "user" code again. But it requires some
explicit work.
There is a bit of additional work due to temp tables, as valgrind does not
understand io_uring sufficiently to mark buffer contents as defined after a
read.
Per buildfarm animal skink.
Reviewed-by:
Discussion: https://postgr.es/m/3pd4322mogfmdd5nln3zphdwhtmq3rzdldqjwb2sfqzcgs22lf@ok2gletdaoe6
---
src/include/storage/aio_internal.h | 1 +
src/backend/storage/aio/aio_io.c | 23 +++++++++++
src/backend/storage/aio/method_io_uring.c | 49 ++++++++++++++++++++++-
src/backend/storage/aio/method_worker.c | 18 +++++++++
src/backend/storage/buffer/bufmgr.c | 19 +++++++++
5 files changed, 109 insertions(+), 1 deletion(-)
diff --git a/src/include/storage/aio_internal.h b/src/include/storage/aio_internal.h
index 7f18da2c856..33f27b9fe50 100644
--- a/src/include/storage/aio_internal.h
+++ b/src/include/storage/aio_internal.h
@@ -344,6 +344,7 @@ extern PgAioResult pgaio_io_call_complete_local(PgAioHandle *ioh);
extern void pgaio_io_perform_synchronously(PgAioHandle *ioh);
extern const char *pgaio_io_get_op_name(PgAioHandle *ioh);
extern bool pgaio_io_uses_fd(PgAioHandle *ioh, int fd);
+extern int pgaio_io_get_iovec_length(PgAioHandle *ioh, struct iovec **iov);
/* aio_target.c */
extern bool pgaio_io_can_reopen(PgAioHandle *ioh);
diff --git a/src/backend/storage/aio/aio_io.c b/src/backend/storage/aio/aio_io.c
index 4d31392ddc7..bd8be987526 100644
--- a/src/backend/storage/aio/aio_io.c
+++ b/src/backend/storage/aio/aio_io.c
@@ -210,3 +210,26 @@ pgaio_io_uses_fd(PgAioHandle *ioh, int fd)
return false; /* silence compiler */
}
+
+/*
+ * Return the iovecand its length. Currently only expected to be used by
+ * debugging infrastructure
+ */
+int
+pgaio_io_get_iovec_length(PgAioHandle *ioh, struct iovec **iov)
+{
+ Assert(ioh->state >= PGAIO_HS_DEFINED);
+
+ *iov = &pgaio_ctl->iovecs[ioh->iovec_off];
+
+ switch (ioh->op)
+ {
+ case PGAIO_OP_READV:
+ return ioh->op_data.read.iov_length;
+ case PGAIO_OP_WRITEV:
+ return ioh->op_data.write.iov_length;
+ default:
+ pg_unreachable();
+ return 0;
+ }
+}
diff --git a/src/backend/storage/aio/method_io_uring.c b/src/backend/storage/aio/method_io_uring.c
index c719ba2727a..1a90cadfd49 100644
--- a/src/backend/storage/aio/method_io_uring.c
+++ b/src/backend/storage/aio/method_io_uring.c
@@ -38,6 +38,7 @@
#include "storage/shmem.h"
#include "storage/lwlock.h"
#include "storage/procnumber.h"
+#include "utils/memdebug.h"
#include "utils/wait_event.h"
@@ -324,6 +325,49 @@ pgaio_uring_completion_error_callback(void *arg)
errcontext("completing I/O on behalf of process %d", owner_pid);
}
+static void
+pgaio_uring_io_process_completion(PgAioHandle *ioh, int32 res)
+{
+ /*
+ * Valgrind does not understand io_uring sufficiently to mark the
+ * referenced region as defined on IO completion, and it'd probably be
+ * nontrivial to teach it to do so. So we just have to do the legwork
+ * ourselves.
+ */
+#ifdef USE_VALGRIND
+ if (ioh->op == PGAIO_OP_READV)
+ {
+ struct iovec *iov;
+ uint16 iov_length = pgaio_io_get_iovec_length(ioh, &iov);
+ int32 processed = 0;
+
+ for (int i = 0; i < iov_length; i++)
+ {
+ const char *base = iov[i].iov_base;
+ int32 len = iov[i].iov_len;
+
+ Assert(iov[i].iov_len <= PG_INT32_MAX);
+
+ if (processed + len <= res)
+ VALGRIND_MAKE_MEM_DEFINED(base, len);
+ else if (processed <= res)
+ {
+ size_t middle = res - processed;
+
+ VALGRIND_MAKE_MEM_DEFINED(base, middle);
+ VALGRIND_MAKE_MEM_UNDEFINED(base + middle, len - middle);
+ }
+ else
+ VALGRIND_MAKE_MEM_UNDEFINED(base, len);
+
+ processed += len;
+ }
+ }
+#endif
+
+ pgaio_io_process_completion(ioh, res);
+}
+
static void
pgaio_uring_drain_locked(PgAioUringContext *context)
{
@@ -361,13 +405,16 @@ pgaio_uring_drain_locked(PgAioUringContext *context)
for (int i = 0; i < ncqes; i++)
{
struct io_uring_cqe *cqe = cqes[i];
+ int32 res;
PgAioHandle *ioh;
ioh = io_uring_cqe_get_data(cqe);
errcallback.arg = ioh;
+ res = cqe->res;
+
io_uring_cqe_seen(&context->io_uring_ring, cqe);
- pgaio_io_process_completion(ioh, cqe->res);
+ pgaio_uring_io_process_completion(ioh, res);
errcallback.arg = NULL;
}
diff --git a/src/backend/storage/aio/method_worker.c b/src/backend/storage/aio/method_worker.c
index 31d94ac82c5..232b79be4c4 100644
--- a/src/backend/storage/aio/method_worker.c
+++ b/src/backend/storage/aio/method_worker.c
@@ -42,6 +42,7 @@
#include "storage/latch.h"
#include "storage/proc.h"
#include "tcop/tcopprot.h"
+#include "utils/memdebug.h"
#include "utils/ps_status.h"
#include "utils/wait_event.h"
@@ -529,6 +530,23 @@ IoWorkerMain(const void *startup_data, size_t startup_data_len)
error_errno = 0;
error_ioh = NULL;
+ /*
+ * As part of IO completion the buffer will be marked as
+ * non-accessible, until the buffer is pinned again. The next time
+ * there is IO for the same buffer, the memory will be considered
+ * inaccessible. Therefore we need to explicitly allow access to
+ * the memory before reading data into it.
+ */
+#ifdef USE_VALGRIND
+ {
+ struct iovec *iov;
+ uint16 iov_length = pgaio_io_get_iovec_length(ioh, &iov);
+
+ for (int i = 0; i < iov_length; i++)
+ VALGRIND_MAKE_MEM_UNDEFINED(iov[i].iov_base, iov[i].iov_len);
+ }
+#endif
+
/*
* We don't expect this to ever fail with ERROR or FATAL, no need
* to keep error_ioh set to the IO.
diff --git a/src/backend/storage/buffer/bufmgr.c b/src/backend/storage/buffer/bufmgr.c
index 13cebad9b12..d34a1e335e5 100644
--- a/src/backend/storage/buffer/bufmgr.c
+++ b/src/backend/storage/buffer/bufmgr.c
@@ -6883,6 +6883,19 @@ buffer_readv_complete_one(PgAioTargetData *td, uint8 buf_off, Buffer buffer,
{
PgAioResult result_one;
+ /*
+ * If the buffer is not currently pinned by this backend, e.g. because
+ * we're completing this IO after an error, the buffer data will have
+ * been marked as inaccessible when the buffer was unpinned. The AIO
+ * subystem holds a pin, but that doesn't prevent the buffer from
+ * having been marked as inaccessible. The completion might also be
+ * executed in a different process.
+ */
+#ifdef USE_VALGRIND
+ if (!BufferIsPinned(buffer))
+ VALGRIND_MAKE_MEM_DEFINED(bufdata, BLCKSZ);
+#endif
+
if (!PageIsVerified((Page) bufdata, tag.blockNum, piv_flags,
failed_checksum))
{
@@ -6901,6 +6914,12 @@ buffer_readv_complete_one(PgAioTargetData *td, uint8 buf_off, Buffer buffer,
else if (*failed_checksum)
*ignored_checksum = true;
+ /* undo what we did above */
+#ifdef USE_VALGRIND
+ if (!BufferIsPinned(buffer))
+ VALGRIND_MAKE_MEM_NOACCESS(bufdata, BLCKSZ);
+#endif
+
/*
* Immediately log a message about the invalid page, but only to the
* server log. The reason to do so immediately is that this may be
--
2.48.1.76.g4e746b1a31.dirty
[text/x-diff] v1-0003-aio-Avoid-spurious-coverity-warning.patch (1.5K, ../../s5bbhmfez5pe46yfvqkxkg72zsc2r3knkvapqyz37algbfw2sf@q7jcviovnalu/4-v1-0003-aio-Avoid-spurious-coverity-warning.patch)
download | inline diff:
From 8b059ca837dbbf77344b6a45cd66f0c0935769fc Mon Sep 17 00:00:00 2001
From: Andres Freund <andres@anarazel.de>
Date: Fri, 4 Apr 2025 15:15:39 -0400
Subject: [PATCH v1 3/3] aio: Avoid spurious coverity warning
PgAioResult.result is never accessed in the relevant path, but coverity
complains about an uninitialized access anyway. So just zero-initialize the
whole thing. While at it, reduce the scope of the variable.
Reported-by: Ranier Vilela <ranier.vf@gmail.com>
Discussion: https://postgr.es/m/CAEudQApsKqd-s+fsUQ0OmxJAMHmBSXxrAz3dCs+uvqb3iRtjSw@mail.gmail.com
---
src/backend/storage/buffer/bufmgr.c | 4 ++--
1 file changed, 2 insertions(+), 2 deletions(-)
diff --git a/src/backend/storage/buffer/bufmgr.c b/src/backend/storage/buffer/bufmgr.c
index d34a1e335e5..8265bf9fd10 100644
--- a/src/backend/storage/buffer/bufmgr.c
+++ b/src/backend/storage/buffer/bufmgr.c
@@ -6881,8 +6881,6 @@ buffer_readv_complete_one(PgAioTargetData *td, uint8 buf_off, Buffer buffer,
/* Check for garbage data. */
if (!failed)
{
- PgAioResult result_one;
-
/*
* If the buffer is not currently pinned by this backend, e.g. because
* we're completing this IO after an error, the buffer data will have
@@ -6936,6 +6934,8 @@ buffer_readv_complete_one(PgAioTargetData *td, uint8 buf_off, Buffer buffer,
*/
if (*buffer_invalid || *failed_checksum || *zeroed_buffer)
{
+ PgAioResult result_one = {0};
+
buffer_readv_encode_error(&result_one, is_temp,
*zeroed_buffer,
*ignored_checksum,
--
2.48.1.76.g4e746b1a31.dirty
^ permalink raw reply [nested|flat] 153+ messages in thread
* Re: AIO v2.5
2025-03-04 19:00 AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-10 18:23 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-11 19:41 ` Re: AIO v2.5 Noah Misch <noah@leadboat.com>
2025-03-11 23:55 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-12 03:57 ` Re: AIO v2.5 Noah Misch <noah@leadboat.com>
2025-03-12 17:06 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-12 23:23 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-13 10:53 ` Re: AIO v2.5 Antonin Houska <ah@cybertec.at>
2025-03-14 19:43 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-17 20:00 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-18 20:12 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-21 01:58 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-25 01:18 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-27 01:07 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-27 20:58 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-28 03:22 ` Re: AIO v2.5 Noah Misch <noah@leadboat.com>
2025-03-28 12:54 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-29 03:35 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-29 13:41 ` Re: AIO v2.5 Noah Misch <noah@leadboat.com>
2025-03-29 14:48 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-29 18:25 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-29 21:29 ` Re: AIO v2.5 Noah Misch <noah@leadboat.com>
2025-03-30 23:46 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-04-01 00:41 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-04-01 15:11 ` Re: AIO v2.5 Noah Misch <noah@leadboat.com>
2025-04-01 15:55 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-04-01 21:47 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-04-03 18:19 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-04-03 19:40 ` Re: AIO v2.5 Noah Misch <noah@leadboat.com>
2025-04-04 19:16 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
@ 2025-04-04 21:18 ` Noah Misch <noah@leadboat.com>
2025-04-05 03:53 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
0 siblings, 1 reply; 153+ messages in thread
From: Noah Misch @ 2025-04-04 21:18 UTC (permalink / raw)
To: Andres Freund <andres@anarazel.de>; +Cc: pgsql-hackers@postgresql.org, Thomas Munro <thomas.munro@gmail.com>; Heikki Linnakangas <hlinnaka@iki.fi>; Robert Haas <robertmhaas@gmail.com>; Jakub Wartak <jakub.wartak@enterprisedb.com>; Jelte Fennema-Nio <postgres@jeltef.nl>; Antonin Houska <ah@cybertec.at>
On Fri, Apr 04, 2025 at 03:16:18PM -0400, Andres Freund wrote:
> On 2025-04-03 12:40:23 -0700, Noah Misch wrote:
> > On Thu, Apr 03, 2025 at 02:19:43PM -0400, Andres Freund wrote:
> > In the general case, we could want client requests as follows:
> >
> > - If completor==definer and has not dropped pin:
> > - Make defined before verifying page. That's all. It might be cleaner to
> > do this when first retrieving a return value from io_uring, since this
> > just makes up for what Valgrind already does for readv().
>
> Yea, I think it's better to do that in io_uring. It's what I have done in the
> attached.
>
>
> > - If completor!=definer or has dropped pin:
> > - Make NOACCESS in definer when definer cedes its own pin.
>
> That's the current behaviour for shared buffers, right?
Yes.
> > - For io_method=worker, make UNDEFINED before starting readv(). It might be
> > cleanest to do this when the worker first acts as the owner of the AIO
> > subsystem pin, if that's a clear moment earlier than readv().
>
> Hm, what do we need this for?
At the time, we likely didn't need it:
- If the worker does its own PinBuffer*()+unpin, we don't need it. Those
functions do the Valgrind client requests.
- If the worker relies on the AIO-subsystem-owned pin and does neither regular
pin nor regular unpin, we don't need it. Buffers are always "defined".
- If the worker relies on the AIO-subsystem-owned pin to skip PinBuffer*() but
uses regular unpin code, then the buffer may be NOACCESS. Then one would
need this. But this would be questionable for other reasons.
Your proposed change to set NOACCESS in buffer_readv_complete_one() interacts
with things further, making the UNDEFINED necessary.
> > - Make DEFINED in completor before verifying page. It might be cleaner to
> > do this when the completor first retrieves a return value from io_uring,
> > since this just makes up for what Valgrind already does for readv().
>
> I think we can't rely on the marking during retrieving it from io_uring, as
> that might have happened in a different backend for a temp buffer. That'd only
> happen if we got io_uring events for *another* IO that involved a shared rel,
> but it can happen.
Good point. I think the VALGRIND_MAKE_MEM_DEFINED() in
pgaio_uring_drain_locked() isn't currently needed at all. If
completor-subxact==definer-subxact, PinBuffer() already did what Valgrind
needs. Otherwise, buffer_readv_complete_one() does what Valgrind needs.
If that's right, it would still be nice to reach the right
VALGRIND_MAKE_MEM_DEFINED() without involving bufmgr. That helps future,
non-bufmgr AIO use cases. It's tricky to pick the right place for that
VALGRIND_MAKE_MEM_DEFINED():
- pgaio_uring_drain_locked() is problematic, I think. In the localbuf case,
the iovec base address is relevant only in the ioh-defining process. In the
shmem completor!=definer case, this runs only in the completor.
- A complete_local callback solves those problems. However, if the
AIO-defining subxact aborted, then we shouldn't set DEFINED at all, since
the buffer mapping may have changed by the time of complete_local.
- Putting it in the place that would call pgaio_result_report(ERROR) if
needed, e.g. ProcessReadBuffersResult(), solves the problem of the buffer
mapping having moved. ProcessReadBuffersResult() doesn't even need this,
since PinBuffer() already did it. Each future AIO use case will have a
counterpart of ProcessReadBuffersResult() that consumes the result and
proceeds with tasks that depend on the AIO. That's the place.
Is that right? I got this wrong a few times while trying to think through it,
so I'm not too confident in the above.
> > > Not quite sure if we should mark
> > > the entire IOV is efined or just the portion that was actually read - the
> > > latter is additional fiddly code, and it's not clear it's likely to be helpful?
> >
> > Seems fine to do the simpler way if that saves fiddly code.
>
> Can't quite decide, it's just at the border of what I consider too
> fiddly... See the change to method_io_uring.c in the attached patch.
It is at the border, as you say, but I'd tend to keep it.
> Subject: [PATCH v1 1/3] localbuf: Add Valgrind buffer access instrumentation
Ready for commit
> Subject: [PATCH v1 2/3] aio: Make AIO compatible with valgrind
See above about pgaio_uring_drain_locked().
> related code until it is pinned bu "user" code again. But it requires some
s/bu/by/
> + * Return the iovecand its length. Currently only expected to be used by
s/iovecand/iovec and/
> @@ -361,13 +405,16 @@ pgaio_uring_drain_locked(PgAioUringContext *context)
> for (int i = 0; i < ncqes; i++)
> {
> struct io_uring_cqe *cqe = cqes[i];
> + int32 res;
> PgAioHandle *ioh;
>
> ioh = io_uring_cqe_get_data(cqe);
> errcallback.arg = ioh;
> + res = cqe->res;
> +
> io_uring_cqe_seen(&context->io_uring_ring, cqe);
>
> - pgaio_io_process_completion(ioh, cqe->res);
> + pgaio_uring_io_process_completion(ioh, res);
I guess this is a distinct cleanup, done to avoid any suspicion of cqe being
reused asynchronously after io_uring_cqe_seen(). Is that right?
> Subject: [PATCH v1 3/3] aio: Avoid spurious coverity warning
Ready for commit
^ permalink raw reply [nested|flat] 153+ messages in thread
* Re: AIO v2.5
2025-03-04 19:00 AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-10 18:23 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-11 19:41 ` Re: AIO v2.5 Noah Misch <noah@leadboat.com>
2025-03-11 23:55 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-12 03:57 ` Re: AIO v2.5 Noah Misch <noah@leadboat.com>
2025-03-12 17:06 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-12 23:23 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-13 10:53 ` Re: AIO v2.5 Antonin Houska <ah@cybertec.at>
2025-03-14 19:43 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-17 20:00 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-18 20:12 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-21 01:58 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-25 01:18 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-27 01:07 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-27 20:58 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-28 03:22 ` Re: AIO v2.5 Noah Misch <noah@leadboat.com>
2025-03-28 12:54 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-29 03:35 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-29 13:41 ` Re: AIO v2.5 Noah Misch <noah@leadboat.com>
2025-03-29 14:48 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-29 18:25 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-29 21:29 ` Re: AIO v2.5 Noah Misch <noah@leadboat.com>
2025-03-30 23:46 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-04-01 00:41 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-04-01 15:11 ` Re: AIO v2.5 Noah Misch <noah@leadboat.com>
2025-04-01 15:55 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-04-01 21:47 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-04-03 18:19 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-04-03 19:40 ` Re: AIO v2.5 Noah Misch <noah@leadboat.com>
2025-04-04 19:16 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-04-04 21:18 ` Re: AIO v2.5 Noah Misch <noah@leadboat.com>
@ 2025-04-05 03:53 ` Andres Freund <andres@anarazel.de>
2025-04-05 13:43 ` Re: AIO v2.5 Noah Misch <noah@leadboat.com>
0 siblings, 1 reply; 153+ messages in thread
From: Andres Freund @ 2025-04-05 03:53 UTC (permalink / raw)
To: Noah Misch <noah@leadboat.com>; +Cc: pgsql-hackers@postgresql.org, Thomas Munro <thomas.munro@gmail.com>; Heikki Linnakangas <hlinnaka@iki.fi>; Robert Haas <robertmhaas@gmail.com>; Jakub Wartak <jakub.wartak@enterprisedb.com>; Jelte Fennema-Nio <postgres@jeltef.nl>; Antonin Houska <ah@cybertec.at>
Hi,
On 2025-04-04 14:18:02 -0700, Noah Misch wrote:
> On Fri, Apr 04, 2025 at 03:16:18PM -0400, Andres Freund wrote:
> > > - Make DEFINED in completor before verifying page. It might be cleaner to
> > > do this when the completor first retrieves a return value from io_uring,
> > > since this just makes up for what Valgrind already does for readv().
> >
> > I think we can't rely on the marking during retrieving it from io_uring, as
> > that might have happened in a different backend for a temp buffer. That'd only
> > happen if we got io_uring events for *another* IO that involved a shared rel,
> > but it can happen.
>
> Good point. I think the VALGRIND_MAKE_MEM_DEFINED() in
> pgaio_uring_drain_locked() isn't currently needed at all. If
> completor-subxact==definer-subxact, PinBuffer() already did what Valgrind
> needs. Otherwise, buffer_readv_complete_one() does what Valgrind needs.
We did need it - but only because I bungled something in the earlier patch to
add valgrind support. The problem is that in PinLocalBuffer() there may not
actually be any storage allocated for the buffer yet, so
VALGRIND_MAKE_MEM_DEFINED() doesn't work. In the first use of the buffer the
allocation happens a bit later, in GetLocalVictimBuffer(), namely during the
call to GetLocalBufferStorage().
Not quite sure yet how to best deal with it. Putting the PinLocalBuffer()
slightly later into GetLocalVictimBuffer() fixes the issue, but also doesn't
really seem great.
> If that's right, it would still be nice to reach the right
> VALGRIND_MAKE_MEM_DEFINED() without involving bufmgr.
I think that would be possible if we didn't do VALGRIND_MAKE_MEM_NOACCESS() in
UnpinBuffer()/UnpinLocalBuffer(). But with that I don't see how we can avoid
needing to remark the region as accessible?
> That helps future, non-bufmgr AIO use cases. It's tricky to pick the right
> place for that VALGRIND_MAKE_MEM_DEFINED():
> - pgaio_uring_drain_locked() is problematic, I think. In the localbuf case,
> the iovec base address is relevant only in the ioh-defining process. In the
> shmem completor!=definer case, this runs only in the completor.
You're right :(
> - A complete_local callback solves those problems. However, if the
> AIO-defining subxact aborted, then we shouldn't set DEFINED at all, since
> the buffer mapping may have changed by the time of complete_local.
I don't think that is possible, due to the aio subsystem owned pin?
> - Putting it in the place that would call pgaio_result_report(ERROR) if
> needed, e.g. ProcessReadBuffersResult(), solves the problem of the buffer
> mapping having moved. ProcessReadBuffersResult() doesn't even need this,
> since PinBuffer() already did it. Each future AIO use case will have a
> counterpart of ProcessReadBuffersResult() that consumes the result and
> proceeds with tasks that depend on the AIO. That's the place.
I don't really follow - at the point something like ProcessReadBuffersResult()
gets involved, we'll already have done the accesses that needed the memory to
be accessible and defined?
I think the point about non-aio uses is a fair one, but I don't quite know how
to best solve it right now, due to the local buffer issue you mentioned. I'd
guess that we'd best put it somewhere
a) in pgaio_io_process_completion(), if definer==completor || !PGAIO_HF_REFERENCES_LOCAL
b) pgaio_io_call_complete_local(), just before calling
pgaio_io_call_complete_local() if PGAIO_HF_REFERENCES_LOCAL
> > related code until it is pinned bu "user" code again. But it requires some
>
> s/bu/by/
>
> > + * Return the iovecand its length. Currently only expected to be used by
>
> s/iovecand/iovec and/
Fixed.
> > @@ -361,13 +405,16 @@ pgaio_uring_drain_locked(PgAioUringContext *context)
> > for (int i = 0; i < ncqes; i++)
> > {
> > struct io_uring_cqe *cqe = cqes[i];
> > + int32 res;
> > PgAioHandle *ioh;
> >
> > ioh = io_uring_cqe_get_data(cqe);
> > errcallback.arg = ioh;
> > + res = cqe->res;
> > +
> > io_uring_cqe_seen(&context->io_uring_ring, cqe);
> >
> > - pgaio_io_process_completion(ioh, cqe->res);
> > + pgaio_uring_io_process_completion(ioh, res);
>
> I guess this is a distinct cleanup, done to avoid any suspicion of cqe being
> reused asynchronously after io_uring_cqe_seen(). Is that right?
I don't think there is any such danger - there's no background thing
processing things on the ring, if there were, it'd get corrupted, but it
seemed cleaner to do it that way when I introduced
pgaio_uring_io_process_completion().
> > Subject: [PATCH v1 3/3] aio: Avoid spurious coverity warning
>
> Ready for commit
Thanks!
Greetings,
Andres Freund
^ permalink raw reply [nested|flat] 153+ messages in thread
* Re: AIO v2.5
2025-03-04 19:00 AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-10 18:23 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-11 19:41 ` Re: AIO v2.5 Noah Misch <noah@leadboat.com>
2025-03-11 23:55 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-12 03:57 ` Re: AIO v2.5 Noah Misch <noah@leadboat.com>
2025-03-12 17:06 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-12 23:23 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-13 10:53 ` Re: AIO v2.5 Antonin Houska <ah@cybertec.at>
2025-03-14 19:43 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-17 20:00 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-18 20:12 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-21 01:58 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-25 01:18 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-27 01:07 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-27 20:58 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-28 03:22 ` Re: AIO v2.5 Noah Misch <noah@leadboat.com>
2025-03-28 12:54 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-29 03:35 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-29 13:41 ` Re: AIO v2.5 Noah Misch <noah@leadboat.com>
2025-03-29 14:48 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-29 18:25 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-29 21:29 ` Re: AIO v2.5 Noah Misch <noah@leadboat.com>
2025-03-30 23:46 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-04-01 00:41 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-04-01 15:11 ` Re: AIO v2.5 Noah Misch <noah@leadboat.com>
2025-04-01 15:55 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-04-01 21:47 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-04-03 18:19 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-04-03 19:40 ` Re: AIO v2.5 Noah Misch <noah@leadboat.com>
2025-04-04 19:16 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-04-04 21:18 ` Re: AIO v2.5 Noah Misch <noah@leadboat.com>
2025-04-05 03:53 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
@ 2025-04-05 13:43 ` Noah Misch <noah@leadboat.com>
2025-04-07 19:24 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
0 siblings, 1 reply; 153+ messages in thread
From: Noah Misch @ 2025-04-05 13:43 UTC (permalink / raw)
To: Andres Freund <andres@anarazel.de>; +Cc: pgsql-hackers@postgresql.org, Thomas Munro <thomas.munro@gmail.com>; Heikki Linnakangas <hlinnaka@iki.fi>; Robert Haas <robertmhaas@gmail.com>; Jakub Wartak <jakub.wartak@enterprisedb.com>; Jelte Fennema-Nio <postgres@jeltef.nl>; Antonin Houska <ah@cybertec.at>
On Fri, Apr 04, 2025 at 11:53:13PM -0400, Andres Freund wrote:
> On 2025-04-04 14:18:02 -0700, Noah Misch wrote:
> > On Fri, Apr 04, 2025 at 03:16:18PM -0400, Andres Freund wrote:
> > > > - Make DEFINED in completor before verifying page. It might be cleaner to
> > > > do this when the completor first retrieves a return value from io_uring,
> > > > since this just makes up for what Valgrind already does for readv().
> > >
> > > I think we can't rely on the marking during retrieving it from io_uring, as
> > > that might have happened in a different backend for a temp buffer. That'd only
> > > happen if we got io_uring events for *another* IO that involved a shared rel,
> > > but it can happen.
> >
> > Good point. I think the VALGRIND_MAKE_MEM_DEFINED() in
> > pgaio_uring_drain_locked() isn't currently needed at all. If
> > completor-subxact==definer-subxact, PinBuffer() already did what Valgrind
> > needs. Otherwise, buffer_readv_complete_one() does what Valgrind needs.
>
> We did need it - but only because I bungled something in the earlier patch to
> add valgrind support. The problem is that in PinLocalBuffer() there may not
> actually be any storage allocated for the buffer yet, so
> VALGRIND_MAKE_MEM_DEFINED() doesn't work. In the first use of the buffer the
> allocation happens a bit later, in GetLocalVictimBuffer(), namely during the
> call to GetLocalBufferStorage().
>
> Not quite sure yet how to best deal with it. Putting the PinLocalBuffer()
> slightly later into GetLocalVictimBuffer() fixes the issue, but also doesn't
> really seem great.
Yeah. Maybe this (untested):
diff --git a/src/backend/storage/buffer/localbuf.c b/src/backend/storage/buffer/localbuf.c
index fb44d75..3de13e7 100644
--- a/src/backend/storage/buffer/localbuf.c
+++ b/src/backend/storage/buffer/localbuf.c
@@ -813,7 +813,8 @@ PinLocalBuffer(BufferDesc *buf_hdr, bool adjust_usagecount)
pg_atomic_unlocked_write_u32(&buf_hdr->state, buf_state);
/* see comment in PinBuffer() */
- VALGRIND_MAKE_MEM_DEFINED(LocalBufHdrGetBlock(buf_hdr), BLCKSZ);
+ if (LocalBufHdrGetBlock(bufHdr) != NULL)
+ VALGRIND_MAKE_MEM_DEFINED(LocalBufHdrGetBlock(buf_hdr), BLCKSZ);
}
LocalRefCount[bufid]++;
ResourceOwnerRememberBuffer(CurrentResourceOwner,
@@ -932,6 +933,15 @@ GetLocalBufferStorage(void)
next_buf_in_block++;
total_bufs_allocated++;
+ /*
+ * Caller's PinBuffer() was too early for Valgrind updates, so do it here.
+ * The block is actually undefined, but we want consistency with the
+ * regular case of not needing to allocate memory. This is specifically
+ * needed when method_io_uring.c fills the block, because Valgrind doesn't
+ * recognize io_uring reads causing undefined memory to become defined.
+ */
+ VALGRIND_MAKE_MEM_DEFINED(this_buf, BLCKSZ);
+
return (Block) this_buf;
}
> > If that's right, it would still be nice to reach the right
> > VALGRIND_MAKE_MEM_DEFINED() without involving bufmgr.
>
> I think that would be possible if we didn't do VALGRIND_MAKE_MEM_NOACCESS() in
> UnpinBuffer()/UnpinLocalBuffer(). But with that I don't see how we can avoid
> needing to remark the region as accessible?
Yes, it's not that we should remove VALGRIND_MAKE_MEM_DEFINED() from bufmgr.
I was trying to think about future AIO callers (e.g. RelationCopyStorage())
and how they'd want things to work. That said, perhaps we should just omit
the io_uring-level Valgrind calls and delegate the problem to higher layers
until there's a concrete use case:
--- a/src/backend/storage/smgr/smgr.c
+++ b/src/backend/storage/smgr/smgr.c
@@ -742,14 +742,16 @@ smgrreadv(SMgrRelation reln, ForkNumber forknum, BlockNumber blocknum,
* Compared to smgrreadv(), more responsibilities fall on the caller:
* - Partial reads need to be handled by the caller re-issuing IO for the
* unread blocks
* - smgr will ereport(LOG_SERVER_ONLY) some problems, but higher layers are
* responsible for pgaio_result_report() to mirror that news to the user (if
* the IO results in PGAIO_RS_WARNING) or abort the (sub)transaction (if
* PGAIO_RS_ERROR).
+ * - Under Valgrind, the "buffers" memory may or may not change status to
+ * DEFINED, depending on io_method and concurrent activity.
*/
void
smgrstartreadv(PgAioHandle *ioh,
> > - A complete_local callback solves those problems. However, if the
> > AIO-defining subxact aborted, then we shouldn't set DEFINED at all, since
> > the buffer mapping may have changed by the time of complete_local.
>
> I don't think that is possible, due to the aio subsystem owned pin?
We currently drop the shared buffer AIO subsystem pin in complete_shared
(buffer_readv_complete_one() -> TerminateBufferIO()). I was trying to say
that if we put a VALGRIND_MAKE_MEM_DEFINED in complete_local without changing
anything else, it would have this problem. We probably wouldn't want to move
the shared buffer pin drop to complete_local without a strong reason.
> > - Putting it in the place that would call pgaio_result_report(ERROR) if
> > needed, e.g. ProcessReadBuffersResult(), solves the problem of the buffer
> > mapping having moved. ProcessReadBuffersResult() doesn't even need this,
> > since PinBuffer() already did it. Each future AIO use case will have a
> > counterpart of ProcessReadBuffersResult() that consumes the result and
> > proceeds with tasks that depend on the AIO. That's the place.
>
> I don't really follow - at the point something like ProcessReadBuffersResult()
> gets involved, we'll already have done the accesses that needed the memory to
> be accessible and defined?
(I was referring to future AIO callers, and this part of the discussion is
obsolete if we just delegate the problem.) The bufmgr subsystem does the
PageIsVerified() accesses before ProcessReadBuffersResult(), but I think the
client of the bufmgr subsystem, e.g. read_stream.c, does accesses only after
that. The client must do accesses only after that, since any
pgaio_result_report(ERROR) doesn't happen until ProcessReadBuffersResult().
> I think the point about non-aio uses is a fair one, but I don't quite know how
> to best solve it right now, due to the local buffer issue you mentioned. I'd
> guess that we'd best put it somewhere
> a) in pgaio_io_process_completion(), if definer==completor || !PGAIO_HF_REFERENCES_LOCAL
> b) pgaio_io_call_complete_local(), just before calling
> pgaio_io_call_complete_local() if PGAIO_HF_REFERENCES_LOCAL
I think that would do nothing wrong today, but it uses
!PGAIO_HF_REFERENCES_LOCAL as a proxy for "target memory is already DEFINED,
or a higher layer will make it DEFINED". While !PGAIO_HF_REFERENCES_LOCAL
does imply that today, I think that's a bufmgr-specific conclusion. I have no
particular reason to expect that to hold for future AIO use cases. As above,
I'd be inclined to omit the io_uring-level Valgrind calls until we have a
concrete use case to drive their design. How do you see it?
> > > @@ -361,13 +405,16 @@ pgaio_uring_drain_locked(PgAioUringContext *context)
> > > for (int i = 0; i < ncqes; i++)
> > > {
> > > struct io_uring_cqe *cqe = cqes[i];
> > > + int32 res;
> > > PgAioHandle *ioh;
> > >
> > > ioh = io_uring_cqe_get_data(cqe);
> > > errcallback.arg = ioh;
> > > + res = cqe->res;
> > > +
> > > io_uring_cqe_seen(&context->io_uring_ring, cqe);
> > >
> > > - pgaio_io_process_completion(ioh, cqe->res);
> > > + pgaio_uring_io_process_completion(ioh, res);
> >
> > I guess this is a distinct cleanup, done to avoid any suspicion of cqe being
> > reused asynchronously after io_uring_cqe_seen(). Is that right?
>
> I don't think there is any such danger - there's no background thing
> processing things on the ring, if there were, it'd get corrupted, but it
> seemed cleaner to do it that way when I introduced
> pgaio_uring_io_process_completion().
I agree there's no concrete danger.
^ permalink raw reply [nested|flat] 153+ messages in thread
* Re: AIO v2.5
2025-03-04 19:00 AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-10 18:23 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-11 19:41 ` Re: AIO v2.5 Noah Misch <noah@leadboat.com>
2025-03-11 23:55 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-12 03:57 ` Re: AIO v2.5 Noah Misch <noah@leadboat.com>
2025-03-12 17:06 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-12 23:23 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-13 10:53 ` Re: AIO v2.5 Antonin Houska <ah@cybertec.at>
2025-03-14 19:43 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-17 20:00 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-18 20:12 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-21 01:58 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-25 01:18 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-27 01:07 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-27 20:58 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-28 03:22 ` Re: AIO v2.5 Noah Misch <noah@leadboat.com>
2025-03-28 12:54 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-29 03:35 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-29 13:41 ` Re: AIO v2.5 Noah Misch <noah@leadboat.com>
2025-03-29 14:48 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-29 18:25 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-29 21:29 ` Re: AIO v2.5 Noah Misch <noah@leadboat.com>
2025-03-30 23:46 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-04-01 00:41 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-04-01 15:11 ` Re: AIO v2.5 Noah Misch <noah@leadboat.com>
2025-04-01 15:55 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-04-01 21:47 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-04-03 18:19 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-04-03 19:40 ` Re: AIO v2.5 Noah Misch <noah@leadboat.com>
2025-04-04 19:16 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-04-04 21:18 ` Re: AIO v2.5 Noah Misch <noah@leadboat.com>
2025-04-05 03:53 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-04-05 13:43 ` Re: AIO v2.5 Noah Misch <noah@leadboat.com>
@ 2025-04-07 19:24 ` Andres Freund <andres@anarazel.de>
0 siblings, 0 replies; 153+ messages in thread
From: Andres Freund @ 2025-04-07 19:24 UTC (permalink / raw)
To: Noah Misch <noah@leadboat.com>; +Cc: pgsql-hackers@postgresql.org, Thomas Munro <thomas.munro@gmail.com>; Heikki Linnakangas <hlinnaka@iki.fi>; Robert Haas <robertmhaas@gmail.com>; Jakub Wartak <jakub.wartak@enterprisedb.com>; Jelte Fennema-Nio <postgres@jeltef.nl>; Antonin Houska <ah@cybertec.at>
Hi,
On 2025-04-05 06:43:52 -0700, Noah Misch wrote:
> Yeah. Maybe this (untested):
Something like that works. I adopted your formulation of this, mine was in
GetLocalVictimBuffer(), which seems slightly less future proof.
> > > If that's right, it would still be nice to reach the right
> > > VALGRIND_MAKE_MEM_DEFINED() without involving bufmgr.
> >
> > I think that would be possible if we didn't do VALGRIND_MAKE_MEM_NOACCESS() in
> > UnpinBuffer()/UnpinLocalBuffer(). But with that I don't see how we can avoid
> > needing to remark the region as accessible?
>
> Yes, it's not that we should remove VALGRIND_MAKE_MEM_DEFINED() from bufmgr.
> I was trying to think about future AIO callers (e.g. RelationCopyStorage())
> and how they'd want things to work.
Ah, I now understand what you mean. I'm inclined to leave that out for now,
we can do that later. I spent a bit of time experimenting with going bigger,
but I think it's important to get skink a bit less red.
> That said, perhaps we should just omit the io_uring-level Valgrind calls and
> delegate the problem to higher layers until there's a concrete use case:
Yes, I think that's the right answer for now. Applied your suggested comment.
I looked for a good place to add a comment to method_io_uring.c, but couldn't
really come up with anything convincing. Then I thought the
* "Start" routines for individual IO operations
comment in aio_io.c might be a good place, but also failed to come up with
anything particularly convincing.
> > > - A complete_local callback solves those problems. However, if the
> > > AIO-defining subxact aborted, then we shouldn't set DEFINED at all, since
> > > the buffer mapping may have changed by the time of complete_local.
> >
> > I don't think that is possible, due to the aio subsystem owned pin?
>
> We currently drop the shared buffer AIO subsystem pin in complete_shared
> (buffer_readv_complete_one() -> TerminateBufferIO()). I was trying to say
> that if we put a VALGRIND_MAKE_MEM_DEFINED in complete_local without changing
> anything else, it would have this problem.
Ah, yes, I see what you mean.
> We probably wouldn't want to move the shared buffer pin drop to
> complete_local without a strong reason.
Agreed.
> > I think the point about non-aio uses is a fair one, but I don't quite know how
> > to best solve it right now, due to the local buffer issue you mentioned. I'd
> > guess that we'd best put it somewhere
> > a) in pgaio_io_process_completion(), if definer==completor || !PGAIO_HF_REFERENCES_LOCAL
> > b) pgaio_io_call_complete_local(), just before calling
> > pgaio_io_call_complete_local() if PGAIO_HF_REFERENCES_LOCAL
>
> I think that would do nothing wrong today, but it uses
> !PGAIO_HF_REFERENCES_LOCAL as a proxy for "target memory is already DEFINED,
> or a higher layer will make it DEFINED". While !PGAIO_HF_REFERENCES_LOCAL
> does imply that today, I think that's a bufmgr-specific conclusion. I have no
> particular reason to expect that to hold for future AIO use cases. As above,
> I'd be inclined to omit the io_uring-level Valgrind calls until we have a
> concrete use case to drive their design. How do you see it?
Yea, I'm not sure either. I think we'll best wait until we have non-bufmgr
AIO to address all this. That'll make it easier to shake out.
I think eventually we should do an explicit
- VALGRIND_CHECK_MEM_IS_ADDRESSABLE() in pgaio_io_start_readv
- VALGRIND_CHECK_MEM_IS_DEFINED() in pgaio_io_start_writev
- VALGRIND_MAKE_MEM_DEFINED() in when a READV completes, although some of the
details of when/where to do that aren't entirely clear to me yet. I suspect
we might have to do it in pgaio_io_start_readv(), because it's harder to
reliably do it later
I've pushed the patches. Thanks for the discussion, somehow the mix of shared
memory with valgrind tracking accessibility/definedness in a process local
manner is somewhat mindbending.
Greetings,
Andres Freund
^ permalink raw reply [nested|flat] 153+ messages in thread
* Re: AIO v2.5
2025-03-04 19:00 AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-10 18:23 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-11 19:41 ` Re: AIO v2.5 Noah Misch <noah@leadboat.com>
2025-03-11 23:55 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-12 03:57 ` Re: AIO v2.5 Noah Misch <noah@leadboat.com>
2025-03-12 17:06 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-12 23:23 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-13 10:53 ` Re: AIO v2.5 Antonin Houska <ah@cybertec.at>
2025-03-14 19:43 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-17 20:00 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-18 20:12 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-21 01:58 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-25 01:18 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-27 01:07 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-27 20:58 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-28 03:22 ` Re: AIO v2.5 Noah Misch <noah@leadboat.com>
2025-03-28 12:54 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-29 03:35 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-29 13:41 ` Re: AIO v2.5 Noah Misch <noah@leadboat.com>
2025-03-29 14:48 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-29 18:25 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-29 21:29 ` Re: AIO v2.5 Noah Misch <noah@leadboat.com>
2025-03-30 23:46 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
@ 2025-04-20 15:00 ` Alexander Lakhin <exclusion@gmail.com>
2025-04-24 00:40 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
1 sibling, 1 reply; 153+ messages in thread
From: Alexander Lakhin @ 2025-04-20 15:00 UTC (permalink / raw)
To: Andres Freund <andres@anarazel.de>; +Cc: pgsql-hackers@postgresql.org, Thomas Munro <thomas.munro@gmail.com>; Heikki Linnakangas <hlinnaka@iki.fi>; Robert Haas <robertmhaas@gmail.com>; Jakub Wartak <jakub.wartak@enterprisedb.com>; Jelte Fennema-Nio <postgres@jeltef.nl>; Antonin Houska <ah@cybertec.at>; Noah Misch <noah@leadboat.com>
Hello Andres,
31.03.2025 02:46, Andres Freund wrote:
> I've pushed most of these after some very light further editing. Yay. Thanks
> a lot for all the reviews!
>
> So far the buildfarm hasn't been complaining, but it's early days.
I found one complaint against commit 12ce89fd0, expressed as:
https://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=scorpion&dt=2025-04-08%2001%3A36%3A41
149/334 postgresql:recovery / recovery/032_relfilenode_reuse ERROR          206.41s  (exit status 255 or signal 127
SIGinvalid)
[01:41:53.249](0.620s) ok 8 - standby: post move contents as expected
[01:44:55.175](181.927s) Bail out! aborting wait: program timed out
stream contents: >><<
pattern searched for: (?^m:warmed_buffers)
032_relfilenode_reuse_standby.log contains:
2025-04-08 01:41:54.674 UTC [4013024][startup][33/0:0] DEBUG: waiting for all backends to process ProcSignalBarrier
generation 7
2025-04-08 01:41:54.674 UTC [4013024][startup][33/0:0] CONTEXT:Â WAL redo at 0/43E5360 for Database/DROP: dir 1663/50001
2025-04-08 01:41:54.670 UTC [4013251][client backend][0/2:0] DEBUG: waiting for self with 0 pending
2025-04-08 01:41:54.674 UTC [4013251][client backend][0/2:0] ERROR: no free IOs despite no in-flight IOs
2025-04-08 01:41:54.674 UTC [4013251][client backend][0/2:0] STATEMENT:Â SELECT SUM(pg_prewarm(oid)) warmed_buffers FROM
pg_class WHERE pg_relation_filenode(oid) != 0;
I could reproduce this error with the following TEMP_CONFIG:
log_min_messages = DEBUG4
log_connections = on
log_disconnections = on
log_statement = 'all'
log_line_prefix = '%m [%d][%p:%l][%b] %q[%a] '
fsync = on
io_method=io_uring
backtrace_functions = 'pgaio_io_wait_for_free'
When running 032_relfilenode_reuse.pl in a loop, I got failures on
iterations 10, 18, 6:
2025-04-20 15:56:01.310 CEST [][1517068:122][startup] DEBUG: updated min recovery point to 0/4296E90 on timeline 1
2025-04-20 15:56:01.310 CEST [][1517068:123][startup] CONTEXT:Â WAL redo at 0/4296E48 for Transaction/COMMIT: 2025-04-20
15:56:01.09363+02; inval msgs: catcache 21; sync
2025-04-20 15:56:01.310 CEST [][1517068:124][startup] DEBUG: waiting for all backends to process ProcSignalBarrier
generation 5
2025-04-20 15:56:01.310 CEST [][1517068:125][startup] CONTEXT:Â WAL redo at 0/4296E90 for Database/DROP: dir 16409/50001
2025-04-20 15:56:01.312 CEST [postgres][1517075:144][client backend] [032_relfilenode_reuse.pl] DEBUG:Â waiting for self
with 0 pending
2025-04-20 15:56:01.312 CEST [postgres][1517075:145][client backend] [032_relfilenode_reuse.pl] BACKTRACE:
pgaio_io_wait_for_free at aio.c:703:2
 (inlined by) pgaio_io_acquire at aio.c:186:3
AsyncReadBuffers at bufmgr.c:1854:9
StartReadBuffersImpl at bufmgr.c:1425:18
 (inlined by) StartReadBuffers at bufmgr.c:1497:9
read_stream_start_pending_read at read_stream.c:333:25
read_stream_look_ahead at read_stream.c:475:3
read_stream_next_buffer at read_stream.c:840:6
pg_prewarm at pg_prewarm.c:214:10
ExecInterpExpr at execExprInterp.c:927:7
ExecEvalExprNoReturn at executor.h:447:2
...
2025-04-20 15:56:01.312 CEST [][1517068:126][startup] DEBUG: finished waiting for all backends to process
ProcSignalBarrier generation 5
2025-04-20 15:56:01.312 CEST [][1517068:127][startup] CONTEXT:Â WAL redo at 0/4296E90 for Database/DROP: dir 16409/50001
2025-04-20 15:56:01.314 CEST [postgres][1517075:146][client backend] [032_relfilenode_reuse.pl] ERROR:Â no free IOs
despite no in-flight IOs
2025-04-20 15:56:01.314 CEST [postgres][1517075:147][client backend] [032_relfilenode_reuse.pl] BACKTRACE:
pgaio_io_wait_for_free at aio.c:735:3
 (inlined by) pgaio_io_acquire at aio.c:186:3
AsyncReadBuffers at bufmgr.c:1854:9
StartReadBuffersImpl at bufmgr.c:1425:18
 (inlined by) StartReadBuffers at bufmgr.c:1497:9
read_stream_start_pending_read at read_stream.c:333:25
read_stream_look_ahead at read_stream.c:475:3
read_stream_next_buffer at read_stream.c:840:6
pg_prewarm at pg_prewarm.c:214:10
ExecInterpExpr at execExprInterp.c:927:7
ExecEvalExprNoReturn at executor.h:447:2
...
I configured the build with:
CPPFLAGS="-O1" ./configure --enable-injection-points --enable-cassert --enable-debug --enable-tap-tests --with-liburing
I also encountered another failure when running this test:
t/032_relfilenode_reuse.pl .. 1/? Bailout called. Further testing stopped: aborting wait: program died
Core was generated by `postgres: standby: law postgres [local] SELECTÂ Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â '.
Program terminated with signal SIGABRT, Aborted.
#0Â __pthread_kill_implementation (no_tid=0, signo=6, threadid=<optimized out>) at ./nptl/pthread_kill.c:44
(gdb) bt
#0Â __pthread_kill_implementation (no_tid=0, signo=6, threadid=<optimized out>) at ./nptl/pthread_kill.c:44
#1Â __pthread_kill_internal (signo=6, threadid=<optimized out>) at ./nptl/pthread_kill.c:78
#2Â __GI___pthread_kill (threadid=<optimized out>, signo=signo@entry=6) at ./nptl/pthread_kill.c:89
#3Â 0x0000700a5a24527e in __GI_raise (sig=sig@entry=6) at ../sysdeps/posix/raise.c:26
#4Â 0x0000700a5a2288ff in __GI_abort () at ./stdlib/abort.c:79
#5Â 0x0000630ff2c5dcea in errfinish (filename=filename@entry=0x630ff2cdd139 "aio.c", lineno=lineno@entry=559,
   funcname=funcname@entry=0x630ff2e85da8 <__func__.9> "pgaio_io_wait") at elog.c:599
#6Â 0x0000630ff2aae79c in pgaio_io_wait (ioh=ioh@entry=0x700a57b59700, ref_generation=3972) at aio.c:559
#7Â 0x0000630ff2aaefab in pgaio_io_wait_for_free () at aio.c:771
#8Â pgaio_io_acquire (resowner=0x63100114dee8, ret=ret@entry=0x6310011efec8) at aio.c:186
#9Â 0x0000630ff2ab9137 in AsyncReadBuffers (operation=operation@entry=0x6310011efe88,
nblocks_progress=nblocks_progress@entry=0x7ffd8d262034) at bufmgr.c:1854
#10 0x0000630ff2abc6cc in StartReadBuffersImpl (allow_forwarding=true, flags=0, nblocks=0x7ffd8d262034, blockNum=0,
buffers=0x6310011efe3c,
   operation=0x6310011efe88) at bufmgr.c:1425
#11 StartReadBuffers (operation=0x6310011efe88, buffers=buffers@entry=0x6310011efe3c, blockNum=0,
nblocks=nblocks@entry=0x7ffd8d262034, flags=flags@entry=0)
   at bufmgr.c:1497
#12 0x0000630ff2ab2f3f in read_stream_start_pending_read (stream=stream@entry=0x6310011efde0) at read_stream.c:328
#13 0x0000630ff2ab3363 in read_stream_look_ahead (stream=stream@entry=0x6310011efde0) at read_stream.c:475
#14 0x0000630ff2ab37e5 in read_stream_next_buffer (stream=stream@entry=0x6310011efde0,
per_buffer_data=per_buffer_data@entry=0x0) at read_stream.c:837
#15 0x0000700a5adae60c in pg_prewarm (fcinfo=<optimized out>) at pg_prewarm.c:214
#16 0x0000630ff29004cc in ExecInterpExpr (state=0x6310012239c8, econtext=0x6310011e1920, isnull=0x0) at execExprInterp.c:926
#17 0x0000630ff291bfb4 in ExecEvalExprNoReturn (econtext=<optimized out>, state=<optimized out>) at
../../../src/include/executor/executor.h:445
#18 ExecEvalExprNoReturnSwitchContext (econtext=<optimized out>, state=<optimized out>) at
../../../src/include/executor/executor.h:486
#19 advance_aggregates (aggstate=aggstate@entry=0x6310011e14f8) at nodeAgg.c:820
#20 0x0000630ff291db18 in agg_retrieve_direct (aggstate=0x6310011e14f8) at nodeAgg.c:2540
#21 ExecAgg (pstate=0x6310011e14f8) at nodeAgg.c:2265
#22 0x0000630ff290deef in ExecProcNodeFirst (node=0x6310011e14f8) at execProcnode.c:469
...
Best regards,
Alexander Lakhin
Neon (https://neon.tech)
^ permalink raw reply [nested|flat] 153+ messages in thread
* Re: AIO v2.5
2025-03-04 19:00 AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-10 18:23 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-11 19:41 ` Re: AIO v2.5 Noah Misch <noah@leadboat.com>
2025-03-11 23:55 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-12 03:57 ` Re: AIO v2.5 Noah Misch <noah@leadboat.com>
2025-03-12 17:06 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-12 23:23 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-13 10:53 ` Re: AIO v2.5 Antonin Houska <ah@cybertec.at>
2025-03-14 19:43 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-17 20:00 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-18 20:12 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-21 01:58 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-25 01:18 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-27 01:07 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-27 20:58 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-28 03:22 ` Re: AIO v2.5 Noah Misch <noah@leadboat.com>
2025-03-28 12:54 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-29 03:35 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-29 13:41 ` Re: AIO v2.5 Noah Misch <noah@leadboat.com>
2025-03-29 14:48 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-29 18:25 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-29 21:29 ` Re: AIO v2.5 Noah Misch <noah@leadboat.com>
2025-03-30 23:46 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-04-20 15:00 ` Re: AIO v2.5 Alexander Lakhin <exclusion@gmail.com>
@ 2025-04-24 00:40 ` Andres Freund <andres@anarazel.de>
2025-04-24 04:04 ` Re: AIO v2.5 Alexander Lakhin <exclusion@gmail.com>
0 siblings, 1 reply; 153+ messages in thread
From: Andres Freund @ 2025-04-24 00:40 UTC (permalink / raw)
To: Alexander Lakhin <exclusion@gmail.com>; +Cc: pgsql-hackers@postgresql.org, Thomas Munro <thomas.munro@gmail.com>; Heikki Linnakangas <hlinnaka@iki.fi>; Robert Haas <robertmhaas@gmail.com>; Jakub Wartak <jakub.wartak@enterprisedb.com>; Jelte Fennema-Nio <postgres@jeltef.nl>; Antonin Houska <ah@cybertec.at>; Noah Misch <noah@leadboat.com>
Hi,
On 2025-04-20 18:00:00 +0300, Alexander Lakhin wrote:
> 31.03.2025 02:46, Andres Freund wrote:
> > I've pushed most of these after some very light further editing. Yay. Thanks
> > a lot for all the reviews!
> >
> > So far the buildfarm hasn't been complaining, but it's early days.
>
> I found one complaint against commit 12ce89fd0, expressed as:
> https://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=scorpion&dt=2025-04-08%2001%3A36%3A41
Thanks for pointing that out and repro'ing it!
> 2025-04-08 01:41:54.670 UTC [4013251][client backend][0/2:0] DEBUG: waiting for self with 0 pending
I'd like to extend this debug message with the number of in-flight IOs. I
assume nobody will mind me doing that.
> I could reproduce this error with the following TEMP_CONFIG:
> log_min_messages = DEBUG4
>
> log_connections = on
> log_disconnections = on
> log_statement = 'all'
> log_line_prefix = '%m [%d][%p:%l][%b] %q[%a] '
>
> fsync = on
> io_method=io_uring
> backtrace_functions = 'pgaio_io_wait_for_free'
Hm. Several hundred iterations in without triggering the issue even once, with
that added config. Given your reproducer used fsync, I assume this was on a
real filesystem not tmpfs? What FS? I tried on XFS.
very well could just be a timing dependent issue, the time for an fsync will
differ heavily betweeen devices...
> When running 032_relfilenode_reuse.pl in a loop, I got failures on
> iterations 10, 18, 6:
> 2025-04-20 15:56:01.310 CEST [][1517068:122][startup] DEBUG: updated min recovery point to 0/4296E90 on timeline 1
> 2025-04-20 15:56:01.310 CEST [][1517068:123][startup] CONTEXT:Â WAL redo at
> 0/4296E48 for Transaction/COMMIT: 2025-04-20 15:56:01.09363+02; inval msgs:
> catcache 21; sync
> 2025-04-20 15:56:01.310 CEST [][1517068:124][startup] DEBUG: waiting for all
> backends to process ProcSignalBarrier generation 5
> 2025-04-20 15:56:01.310 CEST [][1517068:125][startup] CONTEXT:Â WAL redo at 0/4296E90 for Database/DROP: dir 16409/50001
> 2025-04-20 15:56:01.312 CEST [postgres][1517075:144][client backend]
> [032_relfilenode_reuse.pl] DEBUG:Â waiting for self with 0 pending
> 2025-04-20 15:56:01.312 CEST [postgres][1517075:145][client backend] [032_relfilenode_reuse.pl] BACKTRACE:
> pgaio_io_wait_for_free at aio.c:703:2
> Â (inlined by) pgaio_io_acquire at aio.c:186:3
> AsyncReadBuffers at bufmgr.c:1854:9
> StartReadBuffersImpl at bufmgr.c:1425:18
> Â (inlined by) StartReadBuffers at bufmgr.c:1497:9
> read_stream_start_pending_read at read_stream.c:333:25
> read_stream_look_ahead at read_stream.c:475:3
> read_stream_next_buffer at read_stream.c:840:6
> pg_prewarm at pg_prewarm.c:214:10
> ExecInterpExpr at execExprInterp.c:927:7
> ExecEvalExprNoReturn at executor.h:447:2
> ...
> 2025-04-20 15:56:01.312 CEST [][1517068:126][startup] DEBUG: finished
> waiting for all backends to process ProcSignalBarrier generation 5
> 2025-04-20 15:56:01.312 CEST [][1517068:127][startup] CONTEXT:Â WAL redo at 0/4296E90 for Database/DROP: dir 16409/50001
> 2025-04-20 15:56:01.314 CEST [postgres][1517075:146][client backend]
> [032_relfilenode_reuse.pl] ERROR:Â no free IOs despite no in-flight IOs
The earlier report also had a ProcSignalBarrier wait just before. That's
doesn't have to be related, the test stresses those pretty intentionally.
But it does seem like it could be related, perhaps we somehow end processing
interrupts while issuing IOs, which then again submits IOs or something? I
don't immediately see such code, e.g. pgaio_closing_fd() is careful to iterate
over the list while waiting.
I think pgaio_closing_fd() should get a DEBUG2 (or so) message documenting
that we are waiting for an IO due to closing an FD. pgaio_shutdown() as well,
while at it.
I somewhat suspect that this is the same issue that you reported earlier, that
I also couldn't repro. That involved a lot of CREATE DATABASEs, which will
trigger a lot of procsignal barriers... I'll try to come up with a more
dedicated stress test and see whether I can trigger a problem that way.
> I also encountered another failure when running this test:
> t/032_relfilenode_reuse.pl .. 1/? Bailout called. Further testing stopped: aborting wait: program died
>
> Core was generated by `postgres: standby: law postgres [local] SELECTÂ Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â '.
> Program terminated with signal SIGABRT, Aborted.
> #0Â __pthread_kill_implementation (no_tid=0, signo=6, threadid=<optimized out>) at ./nptl/pthread_kill.c:44
>
> (gdb) bt
> #0Â __pthread_kill_implementation (no_tid=0, signo=6, threadid=<optimized out>) at ./nptl/pthread_kill.c:44
> #1Â __pthread_kill_internal (signo=6, threadid=<optimized out>) at ./nptl/pthread_kill.c:78
> #2Â __GI___pthread_kill (threadid=<optimized out>, signo=signo@entry=6) at ./nptl/pthread_kill.c:89
> #3Â 0x0000700a5a24527e in __GI_raise (sig=sig@entry=6) at ../sysdeps/posix/raise.c:26
> #4Â 0x0000700a5a2288ff in __GI_abort () at ./stdlib/abort.c:79
> #5Â 0x0000630ff2c5dcea in errfinish (filename=filename@entry=0x630ff2cdd139 "aio.c", lineno=lineno@entry=559,
> Â Â Â funcname=funcname@entry=0x630ff2e85da8 <__func__.9> "pgaio_io_wait") at elog.c:599
> #6Â 0x0000630ff2aae79c in pgaio_io_wait (ioh=ioh@entry=0x700a57b59700, ref_generation=3972) at aio.c:559
> #7Â 0x0000630ff2aaefab in pgaio_io_wait_for_free () at aio.c:771
Do you still have the core file for this? Or at least the generated log
message? It'd be useful to know what state the IO was in when the error was
triggered...
Greetings,
Andres Freund
^ permalink raw reply [nested|flat] 153+ messages in thread
* Re: AIO v2.5
2025-03-04 19:00 AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-10 18:23 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-11 19:41 ` Re: AIO v2.5 Noah Misch <noah@leadboat.com>
2025-03-11 23:55 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-12 03:57 ` Re: AIO v2.5 Noah Misch <noah@leadboat.com>
2025-03-12 17:06 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-12 23:23 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-13 10:53 ` Re: AIO v2.5 Antonin Houska <ah@cybertec.at>
2025-03-14 19:43 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-17 20:00 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-18 20:12 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-21 01:58 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-25 01:18 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-27 01:07 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-27 20:58 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-28 03:22 ` Re: AIO v2.5 Noah Misch <noah@leadboat.com>
2025-03-28 12:54 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-29 03:35 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-29 13:41 ` Re: AIO v2.5 Noah Misch <noah@leadboat.com>
2025-03-29 14:48 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-29 18:25 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-29 21:29 ` Re: AIO v2.5 Noah Misch <noah@leadboat.com>
2025-03-30 23:46 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-04-20 15:00 ` Re: AIO v2.5 Alexander Lakhin <exclusion@gmail.com>
2025-04-24 00:40 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
@ 2025-04-24 04:04 ` Alexander Lakhin <exclusion@gmail.com>
0 siblings, 0 replies; 153+ messages in thread
From: Alexander Lakhin @ 2025-04-24 04:04 UTC (permalink / raw)
To: Andres Freund <andres@anarazel.de>; +Cc: pgsql-hackers@postgresql.org, Thomas Munro <thomas.munro@gmail.com>; Heikki Linnakangas <hlinnaka@iki.fi>; Robert Haas <robertmhaas@gmail.com>; Jakub Wartak <jakub.wartak@enterprisedb.com>; Jelte Fennema-Nio <postgres@jeltef.nl>; Antonin Houska <ah@cybertec.at>; Noah Misch <noah@leadboat.com>
Hello Andres,
24.04.2025 03:40, Andres Freund wrote:
> Hi,
>
> On 2025-04-20 18:00:00 +0300, Alexander Lakhin wrote:
>
>> 2025-04-08 01:41:54.670 UTC [4013251][client backend][0/2:0] DEBUG: waiting for self with 0 pending
> I'd like to extend this debug message with the number of in-flight IOs. I
> assume nobody will mind me doing that.
I printed that number for debugging and always got 1, because
io_max_concurrency == 1 during that test (because of shared_buffers=1MB).
With shared_buffers increased:
--- a/src/test/recovery/t/032_relfilenode_reuse.pl
+++ b/src/test/recovery/t/032_relfilenode_reuse.pl
@@ -18,7 +18,7 @@ log_connections=on
 # to avoid "repairing" corruption
 full_page_writes=off
 log_min_messages=debug2
-shared_buffers=1MB
+shared_buffers=16MB
I got 100 iterations passed.
>> I could reproduce this error with the following TEMP_CONFIG:
>> log_min_messages = DEBUG4
>>
>> log_connections = on
>> log_disconnections = on
>> log_statement = 'all'
>> log_line_prefix = '%m [%d][%p:%l][%b] %q[%a] '
>>
>> fsync = on
>> io_method=io_uring
>> backtrace_functions = 'pgaio_io_wait_for_free'
> Hm. Several hundred iterations in without triggering the issue even once, with
> that added config. Given your reproducer used fsync, I assume this was on a
> real filesystem not tmpfs? What FS? I tried on XFS.
>
> very well could just be a timing dependent issue, the time for an fsync will
> differ heavily betweeen devices...
Yeah, it definitely depends on timing.
Using that TEMP_CONFIG changes the duration of the test significantly for me:
Files=1, Tests=14, 2 wallclock secs ( 0.01 usr 0.01 sys + 0.13 cusr 0.32 csys = 0.47 CPU)
->
Files=1, Tests=14, 14 wallclock secs ( 0.00 usr 0.00 sys + 0.14 cusr 0.33 csys = 0.47 CPU)
I reproduced the failure on ext4 (desktop-grade NVME) and on xfs (cheap
SSD), on two different machines.
It's also reproduced on HDD (with ext4), seemingly worse, where the
duration is increased even more:
I 25
# +++ tap check in src/test/recovery +++
t/032_relfilenode_reuse.pl .. ok
All tests successful.
Files=1, Tests=14, 28 wallclock secs ( 0.00 usr 0.00 sys + 0.15 cusr 0.39 csys = 0.54 CPU)
Result: PASS
I 26
# +++ tap check in src/test/recovery +++
t/032_relfilenode_reuse.pl .. 7/? Bailout called. Further testing stopped: aborting wait: program timed out
>> When running 032_relfilenode_reuse.pl in a loop, I got failures on
>> iterations 10, 18, 6:
>> 2025-04-20 15:56:01.310 CEST [][1517068:122][startup] DEBUG: updated min recovery point to 0/4296E90 on timeline 1
>> 2025-04-20 15:56:01.310 CEST [][1517068:123][startup] CONTEXT:Â WAL redo at
>> 0/4296E48 for Transaction/COMMIT: 2025-04-20 15:56:01.09363+02; inval msgs:
>> catcache 21; sync
>> 2025-04-20 15:56:01.310 CEST [][1517068:124][startup] DEBUG: waiting for all
>> backends to process ProcSignalBarrier generation 5
>> 2025-04-20 15:56:01.310 CEST [][1517068:125][startup] CONTEXT:Â WAL redo at 0/4296E90 for Database/DROP: dir 16409/50001
>> 2025-04-20 15:56:01.312 CEST [postgres][1517075:144][client backend]
>> [032_relfilenode_reuse.pl] DEBUG:Â waiting for self with 0 pending
>> 2025-04-20 15:56:01.312 CEST [postgres][1517075:145][client backend] [032_relfilenode_reuse.pl] BACKTRACE:
>> pgaio_io_wait_for_free at aio.c:703:2
>> Â (inlined by) pgaio_io_acquire at aio.c:186:3
>> AsyncReadBuffers at bufmgr.c:1854:9
>> StartReadBuffersImpl at bufmgr.c:1425:18
>> Â (inlined by) StartReadBuffers at bufmgr.c:1497:9
>> read_stream_start_pending_read at read_stream.c:333:25
>> read_stream_look_ahead at read_stream.c:475:3
>> read_stream_next_buffer at read_stream.c:840:6
>> pg_prewarm at pg_prewarm.c:214:10
>> ExecInterpExpr at execExprInterp.c:927:7
>> ExecEvalExprNoReturn at executor.h:447:2
>> ...
>> 2025-04-20 15:56:01.312 CEST [][1517068:126][startup] DEBUG: finished
>> waiting for all backends to process ProcSignalBarrier generation 5
>> 2025-04-20 15:56:01.312 CEST [][1517068:127][startup] CONTEXT:Â WAL redo at 0/4296E90 for Database/DROP: dir 16409/50001
>> 2025-04-20 15:56:01.314 CEST [postgres][1517075:146][client backend]
>> [032_relfilenode_reuse.pl] ERROR:Â no free IOs despite no in-flight IOs
> The earlier report also had a ProcSignalBarrier wait just before. That's
> doesn't have to be related, the test stresses those pretty intentionally.
>
> But it does seem like it could be related, perhaps we somehow end processing
> interrupts while issuing IOs, which then again submits IOs or something? I
> don't immediately see such code, e.g. pgaio_closing_fd() is careful to iterate
> over the list while waiting.
>
> I think pgaio_closing_fd() should get a DEBUG2 (or so) message documenting
> that we are waiting for an IO due to closing an FD. pgaio_shutdown() as well,
> while at it.
>
>
> I somewhat suspect that this is the same issue that you reported earlier, that
> I also couldn't repro. That involved a lot of CREATE DATABASEs, which will
> trigger a lot of procsignal barriers... I'll try to come up with a more
> dedicated stress test and see whether I can trigger a problem that way.
Maybe it does matter that "no free IOs despite no in-flight IOs" follows
"WAL redo at 0/4296E90 for Database/DROP" here (that's seemingly constant)...
I've tried also:
--- a/src/backend/storage/aio/aio.c
+++ b/src/backend/storage/aio/aio.c
@@ -705,6 +705,7 @@ pgaio_io_wait_for_free(void)
       pgaio_debug(DEBUG2, "waiting for self with %d pending",
pgaio_my_backend->num_staged_ios);
+pg_sleep(3000);
and got the test failed on each run with:
t/032_relfilenode_reuse.pl .. Dubious, test returned 29 (wstat 7424, 0x1d00)
032_relfilenode_reuse_primary.log
2025-04-24 05:29:27.103 CEST [3504475] 032_relfilenode_reuse.pl LOG:Â statement: UPDATE large SET datab = 1;
2025-04-24 05:29:27.104 CEST [3504475] 032_relfilenode_reuse.pl DEBUG:Â waiting for self with 0 pending
2025-04-24 05:29:27.112 CEST [3504369] DEBUG:Â releasing pm child slot 3
2025-04-24 05:29:27.112 CEST [3504369] LOG:Â client backend (PID 3504475) was terminated by signal 11: Segmentation fault
2025-04-24 05:29:27.112 CEST [3504369] DETAIL:Â Failed process was running: UPDATE large SET datab = 1;
>> I also encountered another failure when running this test:
>> t/032_relfilenode_reuse.pl .. 1/? Bailout called. Further testing stopped: aborting wait: program died
>>
>> Core was generated by `postgres: standby: law postgres [local] SELECTÂ Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â '.
>> Program terminated with signal SIGABRT, Aborted.
>> #0Â __pthread_kill_implementation (no_tid=0, signo=6, threadid=<optimized out>) at ./nptl/pthread_kill.c:44
>>
>> (gdb) bt
>> #0Â __pthread_kill_implementation (no_tid=0, signo=6, threadid=<optimized out>) at ./nptl/pthread_kill.c:44
>> #1Â __pthread_kill_internal (signo=6, threadid=<optimized out>) at ./nptl/pthread_kill.c:78
>> #2Â __GI___pthread_kill (threadid=<optimized out>, signo=signo@entry=6) at ./nptl/pthread_kill.c:89
>> #3Â 0x0000700a5a24527e in __GI_raise (sig=sig@entry=6) at ../sysdeps/posix/raise.c:26
>> #4Â 0x0000700a5a2288ff in __GI_abort () at ./stdlib/abort.c:79
>> #5Â 0x0000630ff2c5dcea in errfinish (filename=filename@entry=0x630ff2cdd139 "aio.c", lineno=lineno@entry=559,
>> Â Â Â funcname=funcname@entry=0x630ff2e85da8 <__func__.9> "pgaio_io_wait") at elog.c:599
>> #6Â 0x0000630ff2aae79c in pgaio_io_wait (ioh=ioh@entry=0x700a57b59700, ref_generation=3972) at aio.c:559
>> #7Â 0x0000630ff2aaefab in pgaio_io_wait_for_free () at aio.c:771
> Do you still have the core file for this? Or at least the generated log
> message? It'd be useful to know what state the IO was in when the error was
> triggered...
I can reproduce it easily, so here is the message:
2025-04-24 06:10:01.924 EEST [postgres][3036378:150][client backend] [032_relfilenode_reuse.pl] PANIC:Â waiting for own
IO in wrong state: 0
2025-04-24 06:10:01.924 EEST [postgres][3036378:151][client backend] [032_relfilenode_reuse.pl] STATEMENT:Â SELECT
SUM(pg_prewarm(oid)) warmed_buffers FROM pg_class WHERE pg_relation_filenode(oid) != 0;
(Sorry for not including it before — I observed it many times, always with
state 0, so it had lost its value for me.)
Best regards,
Alexander Lakhin
Neon (https://neon.tech)
^ permalink raw reply [nested|flat] 153+ messages in thread
* Re: AIO v2.5
2025-03-04 19:00 AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-10 18:23 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-11 19:41 ` Re: AIO v2.5 Noah Misch <noah@leadboat.com>
2025-03-11 23:55 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-12 03:57 ` Re: AIO v2.5 Noah Misch <noah@leadboat.com>
2025-03-12 17:06 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-12 23:23 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-13 10:53 ` Re: AIO v2.5 Antonin Houska <ah@cybertec.at>
2025-03-14 19:43 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-17 20:00 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-18 20:12 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-21 01:58 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
@ 2025-03-25 02:20 ` Noah Misch <noah@leadboat.com>
2025-03-25 02:52 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
4 siblings, 1 reply; 153+ messages in thread
From: Noah Misch @ 2025-03-25 02:20 UTC (permalink / raw)
To: Andres Freund <andres@anarazel.de>; +Cc: Antonin Houska <ah@cybertec.at>; pgsql-hackers@postgresql.org, Thomas Munro <thomas.munro@gmail.com>; Heikki Linnakangas <hlinnaka@iki.fi>; Robert Haas <robertmhaas@gmail.com>; Jakub Wartak <jakub.wartak@enterprisedb.com>; Jelte Fennema-Nio <postgres@jeltef.nl>
On Thu, Mar 20, 2025 at 09:58:37PM -0400, Andres Freund wrote:
> Subject: [PATCH v2.11 09/27] bufmgr: Implement AIO read support
[I checked that v2.12 doesn't invalidate these review comments, but I didn't
technically rebase the review onto v2.12's line numbers.]
> static void
> TerminateBufferIO(BufferDesc *buf, bool clear_dirty, uint32 set_flag_bits,
> - bool forget_owner)
> + bool forget_owner, bool syncio)
> {
> uint32 buf_state;
>
> @@ -5586,6 +5636,14 @@ TerminateBufferIO(BufferDesc *buf, bool clear_dirty, uint32 set_flag_bits,
> if (clear_dirty && !(buf_state & BM_JUST_DIRTIED))
> buf_state &= ~(BM_DIRTY | BM_CHECKPOINT_NEEDED);
>
> + if (!syncio)
> + {
> + /* release ownership by the AIO subsystem */
> + Assert(BUF_STATE_GET_REFCOUNT(buf_state) > 0);
> + buf_state -= BUF_REFCOUNT_ONE;
> + pgaio_wref_clear(&buf->io_wref);
> + }
Looking at the callers:
ZeroAndLockBuffer[1083] TerminateBufferIO(bufHdr, false, BM_VALID, true, true);
ExtendBufferedRelShared[2869] TerminateBufferIO(buf_hdr, false, BM_VALID, true, true);
FlushBuffer[4827] TerminateBufferIO(buf, true, 0, true, true);
AbortBufferIO[6637] TerminateBufferIO(buf_hdr, false, BM_IO_ERROR, false, true);
buffer_readv_complete_one[7279] TerminateBufferIO(buf_hdr, false, set_flag_bits, false, false);
buffer_writev_complete_one[7427] TerminateBufferIO(buf_hdr, clear_dirty, set_flag_bits, false, false);
I think we can improve on the "syncio" arg name. The first two aren't doing
IO, and AbortBufferIO() may be cleaning up what would have been an AIO if it
hadn't failed early. Perhaps name the arg "release_aio" and pass
release_aio=true instead of syncio=false (release_aio = !syncio).
> + * about which buffers are target by IO can be hard to debug, making
s/target/targeted/
> +static pg_attribute_always_inline PgAioResult
> +buffer_readv_complete_one(uint8 buf_off, Buffer buffer, uint8 flags,
> + bool failed, bool is_temp)
> +{
...
> + if ((flags & READ_BUFFERS_ZERO_ON_ERROR) || zero_damaged_pages)
> + {
> + ereport(WARNING,
> + (errcode(ERRCODE_DATA_CORRUPTED),
> + errmsg("invalid page in block %u of relation %s; zeroing out page",
My earlier review requested s/LOG/WARNING/, but I wasn't thinking about this
in full depth. In the !is_temp case, this runs in a complete_shared callback.
A process unrelated to the original IO may run this callback. That's
unfortunate in two ways. First, that other process's client gets an
unexpected WARNING. The process getting the WARNING may not even have
zero_damaged_pages enabled. Second, the client of the process that staged the
IO gets no message.
AIO ERROR-level messages handle this optimally. We emit a LOG-level message
in the process that runs the complete_shared callback, and we arrange for the
ERROR-level message in the stager. That would be ideal here: LOG in the
complete_shared runner, WARNING in the stager.
One could simplify things by forcing io_method=sync under ZERO_ON_ERROR ||
zero_damaged_pages, perhaps as a short-term approach.
Thoughts?
^ permalink raw reply [nested|flat] 153+ messages in thread
* Re: AIO v2.5
2025-03-04 19:00 AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-10 18:23 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-11 19:41 ` Re: AIO v2.5 Noah Misch <noah@leadboat.com>
2025-03-11 23:55 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-12 03:57 ` Re: AIO v2.5 Noah Misch <noah@leadboat.com>
2025-03-12 17:06 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-12 23:23 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-13 10:53 ` Re: AIO v2.5 Antonin Houska <ah@cybertec.at>
2025-03-14 19:43 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-17 20:00 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-18 20:12 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-21 01:58 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-25 02:20 ` Re: AIO v2.5 Noah Misch <noah@leadboat.com>
@ 2025-03-25 02:52 ` Andres Freund <andres@anarazel.de>
2025-03-25 14:11 ` Re: AIO v2.5 Noah Misch <noah@leadboat.com>
0 siblings, 1 reply; 153+ messages in thread
From: Andres Freund @ 2025-03-25 02:52 UTC (permalink / raw)
To: Noah Misch <noah@leadboat.com>; +Cc: Antonin Houska <ah@cybertec.at>; pgsql-hackers@postgresql.org, Thomas Munro <thomas.munro@gmail.com>; Heikki Linnakangas <hlinnaka@iki.fi>; Robert Haas <robertmhaas@gmail.com>; Jakub Wartak <jakub.wartak@enterprisedb.com>; Jelte Fennema-Nio <postgres@jeltef.nl>
Hi,
On 2025-03-24 19:20:37 -0700, Noah Misch wrote:
> On Thu, Mar 20, 2025 at 09:58:37PM -0400, Andres Freund wrote:
> > static void
> > TerminateBufferIO(BufferDesc *buf, bool clear_dirty, uint32 set_flag_bits,
> > - bool forget_owner)
> > + bool forget_owner, bool syncio)
> > ...
> Looking at the callers:
>
> ZeroAndLockBuffer[1083] TerminateBufferIO(bufHdr, false, BM_VALID, true, true);
> ExtendBufferedRelShared[2869] TerminateBufferIO(buf_hdr, false, BM_VALID, true, true);
> FlushBuffer[4827] TerminateBufferIO(buf, true, 0, true, true);
> AbortBufferIO[6637] TerminateBufferIO(buf_hdr, false, BM_IO_ERROR, false, true);
> buffer_readv_complete_one[7279] TerminateBufferIO(buf_hdr, false, set_flag_bits, false, false);
> buffer_writev_complete_one[7427] TerminateBufferIO(buf_hdr, clear_dirty, set_flag_bits, false, false);
>
> I think we can improve on the "syncio" arg name. The first two aren't doing
> IO, and AbortBufferIO() may be cleaning up what would have been an AIO if it
> hadn't failed early. Perhaps name the arg "release_aio" and pass
> release_aio=true instead of syncio=false (release_aio = !syncio).
Yes, I think that makes sense. Will do that tomorrow.
> > +static pg_attribute_always_inline PgAioResult
> > +buffer_readv_complete_one(uint8 buf_off, Buffer buffer, uint8 flags,
> > + bool failed, bool is_temp)
> > +{
> ...
> > + if ((flags & READ_BUFFERS_ZERO_ON_ERROR) || zero_damaged_pages)
> > + {
> > + ereport(WARNING,
> > + (errcode(ERRCODE_DATA_CORRUPTED),
> > + errmsg("invalid page in block %u of relation %s; zeroing out page",
>
> My earlier review requested s/LOG/WARNING/, but I wasn't thinking about this
> in full depth. In the !is_temp case, this runs in a complete_shared
> callback. A process unrelated to the original IO may run this callback.
> That's unfortunate in two ways. First, that other process's client gets an
> unexpected WARNING. The process getting the WARNING may not even have
> zero_damaged_pages enabled. Second, the client of the process that staged
> the IO gets no message.
Ah, right. That could be why I had flipped it. If so, shame on me for not
adding a comment...
> AIO ERROR-level messages handle this optimally. We emit a LOG-level message
> in the process that runs the complete_shared callback, and we arrange for the
> ERROR-level message in the stager. That would be ideal here: LOG in the
> complete_shared runner, WARNING in the stager.
We could obviously downgrade (crossgrade? A LOG is more severe than a LOG in
some ways, but not others) the message when run in a different backend fairly
easily. Still emitting a WARNING in the stager however is a bit more tricky.
Before thinking more deeply about how we could emit WARNING in the stage:
Is it actually sane to use WARNING here? At least for ZERO_ON_ERROR that could
trigger a rather massive flood of messages to the client in a *normal*
situation. I'm thinking of something like an insert extending a relation some
time after an immediate restart and encountering a lot of FSM corruption (due
to its non-crash-safe-ness) during the search for free space and the
subsequent FSM vacuum. It might be ok to LOG that, but sending a lot of
WARNINGs to the client seems not quite right.
If we want to implement it, I think we could introduce PGAIO_RS_WARN, which
then could tell the stager to issue the WARNING. It would add a bit of
distributed cost, both to callbacks and users of AIO, but it might not be too
bad.
> One could simplify things by forcing io_method=sync under ZERO_ON_ERROR ||
> zero_damaged_pages, perhaps as a short-term approach.
Yea, that could work. Perhaps even just for zero_damaged_pages, after
changing it so that ZERO_ON_ERROR always just LOGs.
Hm, it seems somewhat nasty to have rather different performance
characteristics when forced to use zero_damaged_pages to recover from a
problem. Imagine an instance that's configured to use DIO and then needs to
use zero_damaged_pages to recove from corruption...
/me adds writing a test for both ZERO_ON_ERROR and zero_damaged_pages to the
TODO.
Greetings,
Andres Freund
^ permalink raw reply [nested|flat] 153+ messages in thread
* Re: AIO v2.5
2025-03-04 19:00 AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-10 18:23 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-11 19:41 ` Re: AIO v2.5 Noah Misch <noah@leadboat.com>
2025-03-11 23:55 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-12 03:57 ` Re: AIO v2.5 Noah Misch <noah@leadboat.com>
2025-03-12 17:06 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-12 23:23 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-13 10:53 ` Re: AIO v2.5 Antonin Houska <ah@cybertec.at>
2025-03-14 19:43 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-17 20:00 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-18 20:12 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-21 01:58 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-25 02:20 ` Re: AIO v2.5 Noah Misch <noah@leadboat.com>
2025-03-25 02:52 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
@ 2025-03-25 14:11 ` Noah Misch <noah@leadboat.com>
2025-03-25 15:57 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
0 siblings, 1 reply; 153+ messages in thread
From: Noah Misch @ 2025-03-25 14:11 UTC (permalink / raw)
To: Andres Freund <andres@anarazel.de>; +Cc: Antonin Houska <ah@cybertec.at>; pgsql-hackers@postgresql.org, Thomas Munro <thomas.munro@gmail.com>; Heikki Linnakangas <hlinnaka@iki.fi>; Robert Haas <robertmhaas@gmail.com>; Jakub Wartak <jakub.wartak@enterprisedb.com>; Jelte Fennema-Nio <postgres@jeltef.nl>
On Mon, Mar 24, 2025 at 10:52:19PM -0400, Andres Freund wrote:
> On 2025-03-24 19:20:37 -0700, Noah Misch wrote:
> > On Thu, Mar 20, 2025 at 09:58:37PM -0400, Andres Freund wrote:
> > > +static pg_attribute_always_inline PgAioResult
> > > +buffer_readv_complete_one(uint8 buf_off, Buffer buffer, uint8 flags,
> > > + bool failed, bool is_temp)
> > > +{
> > ...
> > > + if ((flags & READ_BUFFERS_ZERO_ON_ERROR) || zero_damaged_pages)
> > > + {
> > > + ereport(WARNING,
> > > + (errcode(ERRCODE_DATA_CORRUPTED),
> > > + errmsg("invalid page in block %u of relation %s; zeroing out page",
> >
> > My earlier review requested s/LOG/WARNING/, but I wasn't thinking about this
> > in full depth. In the !is_temp case, this runs in a complete_shared
> > callback. A process unrelated to the original IO may run this callback.
> > That's unfortunate in two ways. First, that other process's client gets an
> > unexpected WARNING. The process getting the WARNING may not even have
> > zero_damaged_pages enabled. Second, the client of the process that staged
> > the IO gets no message.
>
> Ah, right. That could be why I had flipped it. If so, shame on me for not
> adding a comment...
>
>
> > AIO ERROR-level messages handle this optimally. We emit a LOG-level message
> > in the process that runs the complete_shared callback, and we arrange for the
> > ERROR-level message in the stager. That would be ideal here: LOG in the
> > complete_shared runner, WARNING in the stager.
>
> We could obviously downgrade (crossgrade? A LOG is more severe than a LOG in
> some ways, but not others) the message when run in a different backend fairly
> easily. Still emitting a WARNING in the stager however is a bit more tricky.
>
> Before thinking more deeply about how we could emit WARNING in the stage:
>
> Is it actually sane to use WARNING here? At least for ZERO_ON_ERROR that could
> trigger a rather massive flood of messages to the client in a *normal*
> situation. I'm thinking of something like an insert extending a relation some
> time after an immediate restart and encountering a lot of FSM corruption (due
> to its non-crash-safe-ness) during the search for free space and the
> subsequent FSM vacuum. It might be ok to LOG that, but sending a lot of
> WARNINGs to the client seems not quite right.
Orthogonal to AIO, I do think LOG (or even DEBUG1?) is better for
ZERO_ON_ERROR. The ZERO_ON_ERROR case also should not use
ERRCODE_DATA_CORRUPTED. (That errcode shouldn't appear for business as usual.
It should signify wrong or irretrievable query results, essentially.)
For zero_damaged_pages, WARNING seems at least defensible, and
ERRCODE_DATA_CORRUPTED is right. It wouldn't be the worst thing to change
zero_damaged_pages to LOG and let the complete_shared runner log it, as long
as we release-note that. It's superuser-only, and the superuser can learn to
check the log. One typically should use zero_damaged_pages in one session at
a time, so the logs won't be too confusing.
Another thought on complete_shared running on other backends: I wonder if we
should push an ErrorContextCallback that adds "CONTEXT: completing I/O of
other process" or similar, so people wonder less about how "SELECT FROM a" led
to a log message about IO on table "b".
> If we want to implement it, I think we could introduce PGAIO_RS_WARN, which
> then could tell the stager to issue the WARNING. It would add a bit of
> distributed cost, both to callbacks and users of AIO, but it might not be too
> bad.
>
>
> > One could simplify things by forcing io_method=sync under ZERO_ON_ERROR ||
> > zero_damaged_pages, perhaps as a short-term approach.
>
> Yea, that could work. Perhaps even just for zero_damaged_pages, after
> changing it so that ZERO_ON_ERROR always just LOGs.
Yes.
> Hm, it seems somewhat nasty to have rather different performance
> characteristics when forced to use zero_damaged_pages to recover from a
> problem. Imagine an instance that's configured to use DIO and then needs to
> use zero_damaged_pages to recove from corruption...
True. I'd be willing to bet high-scale use of zero_damaged_pages is rare. By
high scale, I mean something like reading a whole large table, as opposed to a
TID scan of the known-problematic range. That said, people (including me)
expect the emergency tools to be good even if they're used rarely. You're not
wrong to worry about it.
^ permalink raw reply [nested|flat] 153+ messages in thread
* Re: AIO v2.5
2025-03-04 19:00 AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-10 18:23 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-11 19:41 ` Re: AIO v2.5 Noah Misch <noah@leadboat.com>
2025-03-11 23:55 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-12 03:57 ` Re: AIO v2.5 Noah Misch <noah@leadboat.com>
2025-03-12 17:06 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-12 23:23 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-13 10:53 ` Re: AIO v2.5 Antonin Houska <ah@cybertec.at>
2025-03-14 19:43 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-17 20:00 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-18 20:12 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-21 01:58 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-25 02:20 ` Re: AIO v2.5 Noah Misch <noah@leadboat.com>
2025-03-25 02:52 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-25 14:11 ` Re: AIO v2.5 Noah Misch <noah@leadboat.com>
@ 2025-03-25 15:57 ` Andres Freund <andres@anarazel.de>
2025-03-25 16:15 ` Re: AIO v2.5 Noah Misch <noah@leadboat.com>
2025-03-30 18:23 ` Re: AIO v2.5 Melanie Plageman <melanieplageman@gmail.com>
0 siblings, 2 replies; 153+ messages in thread
From: Andres Freund @ 2025-03-25 15:57 UTC (permalink / raw)
To: Noah Misch <noah@leadboat.com>; +Cc: Antonin Houska <ah@cybertec.at>; pgsql-hackers@postgresql.org, Thomas Munro <thomas.munro@gmail.com>; Heikki Linnakangas <hlinnaka@iki.fi>; Robert Haas <robertmhaas@gmail.com>; Jakub Wartak <jakub.wartak@enterprisedb.com>; Jelte Fennema-Nio <postgres@jeltef.nl>
Hi,
On 2025-03-25 07:11:20 -0700, Noah Misch wrote:
> On Mon, Mar 24, 2025 at 10:52:19PM -0400, Andres Freund wrote:
> > Is it actually sane to use WARNING here? At least for ZERO_ON_ERROR that could
> > trigger a rather massive flood of messages to the client in a *normal*
> > situation. I'm thinking of something like an insert extending a relation some
> > time after an immediate restart and encountering a lot of FSM corruption (due
> > to its non-crash-safe-ness) during the search for free space and the
> > subsequent FSM vacuum. It might be ok to LOG that, but sending a lot of
> > WARNINGs to the client seems not quite right.
>
> Orthogonal to AIO, I do think LOG (or even DEBUG1?) is better for
> ZERO_ON_ERROR. The ZERO_ON_ERROR case also should not use
> ERRCODE_DATA_CORRUPTED. (That errcode shouldn't appear for business as usual.
> It should signify wrong or irretrievable query results, essentially.)
I strongly agree on the errcode - basically makes it much harder to use the
errcode to trigger alerting. And we don't have any other way to do that...
I'm, obviously, positive on not using WARNING for ZERO_ON_ERROR. I'm neutral
on LOG vs DEBUG1, I can see arguments for either.
> For zero_damaged_pages, WARNING seems at least defensible, and
> ERRCODE_DATA_CORRUPTED is right. It wouldn't be the worst thing to change
> zero_damaged_pages to LOG and let the complete_shared runner log it, as long
> as we release-note that. It's superuser-only, and the superuser can learn to
> check the log. One typically should use zero_damaged_pages in one session at
> a time, so the logs won't be too confusing.
It's obviously tempting to go for that, I'm somewhat undecided what the best
way is right now. There might be a compromise, see below:
> > If we want to implement it, I think we could introduce PGAIO_RS_WARN, which
> > then could tell the stager to issue the WARNING. It would add a bit of
> > distributed cost, both to callbacks and users of AIO, but it might not be too
> > bad.
FWIW, I prototyped this, it's not hard.
But it can't replace the current WARNING with 100% fidelity: If we read 60
blocks in a single smgrreadv, we today would would emit 60 WARNINGs. But we
can't encode that many block offset in single PgAioResult, there's not enough
space, and enlarging it far enough doesn't seem to make sense either.
What we *could* do is to emit one WARNING for each bufmgr.c smgrstartreadv(),
with that warning saying that there were N zeroed blocks in a read from block
N to block Y and a HINT saying that there are more details in the server log.
> Another thought on complete_shared running on other backends: I wonder if we
> should push an ErrorContextCallback that adds "CONTEXT: completing I/O of
> other process" or similar, so people wonder less about how "SELECT FROM a" led
> to a log message about IO on table "b".
I've been wondering about that as well, and yes, we probably should.
I'd add the pid of the backend that started the IO to the message - although
need to check whether we're trying to keep PIDs of other processes from
unprivileged users.
I think we probably should add a similar, but not equivalent, context in io
workers. Maybe "I/O worker executing I/O on behalf of process %d".
Greetings,
Andres Freund
^ permalink raw reply [nested|flat] 153+ messages in thread
* Re: AIO v2.5
2025-03-04 19:00 AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-10 18:23 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-11 19:41 ` Re: AIO v2.5 Noah Misch <noah@leadboat.com>
2025-03-11 23:55 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-12 03:57 ` Re: AIO v2.5 Noah Misch <noah@leadboat.com>
2025-03-12 17:06 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-12 23:23 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-13 10:53 ` Re: AIO v2.5 Antonin Houska <ah@cybertec.at>
2025-03-14 19:43 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-17 20:00 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-18 20:12 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-21 01:58 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-25 02:20 ` Re: AIO v2.5 Noah Misch <noah@leadboat.com>
2025-03-25 02:52 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-25 14:11 ` Re: AIO v2.5 Noah Misch <noah@leadboat.com>
2025-03-25 15:57 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
@ 2025-03-25 16:15 ` Noah Misch <noah@leadboat.com>
2025-03-26 00:17 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
1 sibling, 1 reply; 153+ messages in thread
From: Noah Misch @ 2025-03-25 16:15 UTC (permalink / raw)
To: Andres Freund <andres@anarazel.de>; +Cc: Antonin Houska <ah@cybertec.at>; pgsql-hackers@postgresql.org, Thomas Munro <thomas.munro@gmail.com>; Heikki Linnakangas <hlinnaka@iki.fi>; Robert Haas <robertmhaas@gmail.com>; Jakub Wartak <jakub.wartak@enterprisedb.com>; Jelte Fennema-Nio <postgres@jeltef.nl>
On Tue, Mar 25, 2025 at 11:57:58AM -0400, Andres Freund wrote:
> On 2025-03-25 07:11:20 -0700, Noah Misch wrote:
> > On Mon, Mar 24, 2025 at 10:52:19PM -0400, Andres Freund wrote:
> > > If we want to implement it, I think we could introduce PGAIO_RS_WARN, which
> > > then could tell the stager to issue the WARNING. It would add a bit of
> > > distributed cost, both to callbacks and users of AIO, but it might not be too
> > > bad.
>
> FWIW, I prototyped this, it's not hard.
>
> But it can't replace the current WARNING with 100% fidelity: If we read 60
> blocks in a single smgrreadv, we today would would emit 60 WARNINGs. But we
> can't encode that many block offset in single PgAioResult, there's not enough
> space, and enlarging it far enough doesn't seem to make sense either.
>
>
> What we *could* do is to emit one WARNING for each bufmgr.c smgrstartreadv(),
> with that warning saying that there were N zeroed blocks in a read from block
> N to block Y and a HINT saying that there are more details in the server log.
Sounds fine.
> > Another thought on complete_shared running on other backends: I wonder if we
> > should push an ErrorContextCallback that adds "CONTEXT: completing I/O of
> > other process" or similar, so people wonder less about how "SELECT FROM a" led
> > to a log message about IO on table "b".
>
> I've been wondering about that as well, and yes, we probably should.
>
> I'd add the pid of the backend that started the IO to the message - although
> need to check whether we're trying to keep PIDs of other processes from
> unprivileged users.
We don't.
> I think we probably should add a similar, but not equivalent, context in io
> workers. Maybe "I/O worker executing I/O on behalf of process %d".
Sounds good.
^ permalink raw reply [nested|flat] 153+ messages in thread
* Re: AIO v2.5
2025-03-04 19:00 AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-10 18:23 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-11 19:41 ` Re: AIO v2.5 Noah Misch <noah@leadboat.com>
2025-03-11 23:55 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-12 03:57 ` Re: AIO v2.5 Noah Misch <noah@leadboat.com>
2025-03-12 17:06 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-12 23:23 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-13 10:53 ` Re: AIO v2.5 Antonin Houska <ah@cybertec.at>
2025-03-14 19:43 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-17 20:00 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-18 20:12 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-21 01:58 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-25 02:20 ` Re: AIO v2.5 Noah Misch <noah@leadboat.com>
2025-03-25 02:52 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-25 14:11 ` Re: AIO v2.5 Noah Misch <noah@leadboat.com>
2025-03-25 15:57 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-25 16:15 ` Re: AIO v2.5 Noah Misch <noah@leadboat.com>
@ 2025-03-26 00:17 ` Andres Freund <andres@anarazel.de>
2025-03-26 00:43 ` Re: AIO v2.5 Noah Misch <noah@leadboat.com>
0 siblings, 1 reply; 153+ messages in thread
From: Andres Freund @ 2025-03-26 00:17 UTC (permalink / raw)
To: Noah Misch <noah@leadboat.com>; +Cc: Antonin Houska <ah@cybertec.at>; pgsql-hackers@postgresql.org, Thomas Munro <thomas.munro@gmail.com>; Heikki Linnakangas <hlinnaka@iki.fi>; Robert Haas <robertmhaas@gmail.com>; Jakub Wartak <jakub.wartak@enterprisedb.com>; Jelte Fennema-Nio <postgres@jeltef.nl>
Hi,
On 2025-03-25 09:15:43 -0700, Noah Misch wrote:
> On Tue, Mar 25, 2025 at 11:57:58AM -0400, Andres Freund wrote:
> > FWIW, I prototyped this, it's not hard.
> >
> > But it can't replace the current WARNING with 100% fidelity: If we read 60
> > blocks in a single smgrreadv, we today would would emit 60 WARNINGs. But we
> > can't encode that many block offset in single PgAioResult, there's not enough
> > space, and enlarging it far enough doesn't seem to make sense either.
> >
> >
> > What we *could* do is to emit one WARNING for each bufmgr.c smgrstartreadv(),
> > with that warning saying that there were N zeroed blocks in a read from block
> > N to block Y and a HINT saying that there are more details in the server log.
It should probably be DETAIL, not HINT...
> Sounds fine.
I got that working. To make it readable, it required changing division of
labor between buffer_readv_complete() and buffer_readv_complete_one() a bit,
but I think it's actually easier to understand now.
Still need to beef up the test infrastructure a bit to make the multi-block
cases more easily testable.
Could use some input on the framing of the message/detail. Right now it's:
ERROR: invalid page in block 8 of relation base/5/16417
DETAIL: Read of 8 blocks, starting at block 7, 1 other pages in the same read are invalid.
But that doesn't seem great. Maybe:
DETAIL: Read of blocks 7..14, 1 other pages in the same read were also invalid.
But that still isn't really a sentence.
Greetings,
Andres Freund
^ permalink raw reply [nested|flat] 153+ messages in thread
* Re: AIO v2.5
2025-03-04 19:00 AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-10 18:23 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-11 19:41 ` Re: AIO v2.5 Noah Misch <noah@leadboat.com>
2025-03-11 23:55 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-12 03:57 ` Re: AIO v2.5 Noah Misch <noah@leadboat.com>
2025-03-12 17:06 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-12 23:23 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-13 10:53 ` Re: AIO v2.5 Antonin Houska <ah@cybertec.at>
2025-03-14 19:43 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-17 20:00 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-18 20:12 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-21 01:58 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-25 02:20 ` Re: AIO v2.5 Noah Misch <noah@leadboat.com>
2025-03-25 02:52 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-25 14:11 ` Re: AIO v2.5 Noah Misch <noah@leadboat.com>
2025-03-25 15:57 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-25 16:15 ` Re: AIO v2.5 Noah Misch <noah@leadboat.com>
2025-03-26 00:17 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
@ 2025-03-26 00:43 ` Noah Misch <noah@leadboat.com>
0 siblings, 0 replies; 153+ messages in thread
From: Noah Misch @ 2025-03-26 00:43 UTC (permalink / raw)
To: Andres Freund <andres@anarazel.de>; +Cc: Antonin Houska <ah@cybertec.at>; pgsql-hackers@postgresql.org, Thomas Munro <thomas.munro@gmail.com>; Heikki Linnakangas <hlinnaka@iki.fi>; Robert Haas <robertmhaas@gmail.com>; Jakub Wartak <jakub.wartak@enterprisedb.com>; Jelte Fennema-Nio <postgres@jeltef.nl>
On Tue, Mar 25, 2025 at 08:17:17PM -0400, Andres Freund wrote:
> On 2025-03-25 09:15:43 -0700, Noah Misch wrote:
> > On Tue, Mar 25, 2025 at 11:57:58AM -0400, Andres Freund wrote:
> > > FWIW, I prototyped this, it's not hard.
> > >
> > > But it can't replace the current WARNING with 100% fidelity: If we read 60
> > > blocks in a single smgrreadv, we today would would emit 60 WARNINGs. But we
> > > can't encode that many block offset in single PgAioResult, there's not enough
> > > space, and enlarging it far enough doesn't seem to make sense either.
> > >
> > >
> > > What we *could* do is to emit one WARNING for each bufmgr.c smgrstartreadv(),
> > > with that warning saying that there were N zeroed blocks in a read from block
> > > N to block Y and a HINT saying that there are more details in the server log.
>
> It should probably be DETAIL, not HINT...
Either is fine with me. I would go for HINT if referring to the server log,
given the precedent of errhint("See server log for query details."). DETAIL
fits for block counts, though:
> Could use some input on the framing of the message/detail. Right now it's:
>
> ERROR: invalid page in block 8 of relation base/5/16417
> DETAIL: Read of 8 blocks, starting at block 7, 1 other pages in the same read are invalid.
>
> But that doesn't seem great. Maybe:
>
> DETAIL: Read of blocks 7..14, 1 other pages in the same read were also invalid.
>
> But that still isn't really a sentence.
How about this for the multi-page case:
WARNING: zeroing out %u invalid pages among blocks %u..%u of relation %s
DETAIL: Block %u held first invalid page.
HINT: See server log for the other %u invalid blocks.
For the one-page case, the old message can stay:
WARNING: invalid page in block %u of relation %s; zeroing out page
^ permalink raw reply [nested|flat] 153+ messages in thread
* Re: AIO v2.5
2025-03-04 19:00 AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-10 18:23 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-11 19:41 ` Re: AIO v2.5 Noah Misch <noah@leadboat.com>
2025-03-11 23:55 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-12 03:57 ` Re: AIO v2.5 Noah Misch <noah@leadboat.com>
2025-03-12 17:06 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-12 23:23 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-13 10:53 ` Re: AIO v2.5 Antonin Houska <ah@cybertec.at>
2025-03-14 19:43 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-17 20:00 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-18 20:12 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-21 01:58 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-25 02:20 ` Re: AIO v2.5 Noah Misch <noah@leadboat.com>
2025-03-25 02:52 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-25 14:11 ` Re: AIO v2.5 Noah Misch <noah@leadboat.com>
2025-03-25 15:57 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
@ 2025-03-30 18:23 ` Melanie Plageman <melanieplageman@gmail.com>
1 sibling, 0 replies; 153+ messages in thread
From: Melanie Plageman @ 2025-03-30 18:23 UTC (permalink / raw)
To: Andres Freund <andres@anarazel.de>; +Cc: Noah Misch <noah@leadboat.com>; Antonin Houska <ah@cybertec.at>; pgsql-hackers@postgresql.org, Thomas Munro <thomas.munro@gmail.com>; Heikki Linnakangas <hlinnaka@iki.fi>; Robert Haas <robertmhaas@gmail.com>; Jakub Wartak <jakub.wartak@enterprisedb.com>; Jelte Fennema-Nio <postgres@jeltef.nl>
On Tue, Mar 25, 2025 at 11:58 AM Andres Freund <andres@anarazel.de> wrote:
>
> > Another thought on complete_shared running on other backends: I wonder if we
> > should push an ErrorContextCallback that adds "CONTEXT: completing I/O of
> > other process" or similar, so people wonder less about how "SELECT FROM a" led
> > to a log message about IO on table "b".
>
> I've been wondering about that as well, and yes, we probably should.
>
> I'd add the pid of the backend that started the IO to the message - although
> need to check whether we're trying to keep PIDs of other processes from
> unprivileged users.
>
> I think we probably should add a similar, but not equivalent, context in io
> workers. Maybe "I/O worker executing I/O on behalf of process %d".
I think this has not yet been done. Attached patch is an attempt to
add error context for IO completions by another backend when using
io_uring and IO processing in general by an IO worker. It seems to
work -- that is, running the test_aio tests, you can see the context
in the logs.
I'm not certain that I did this in the way you both were envisioning, though.
- Melanie
Attachments:
[text/x-patch] Add-errcontext-for-processing-I-Os-for-another-backe.patch (5.2K, ../../CAAKRu_Y-W8W0GTfa-QCYxu_EXhdSXKhNz41WgNe-8z3UnWAHQw@mail.gmail.com/2-Add-errcontext-for-processing-I-Os-for-another-backe.patch)
download | inline diff:
From 79d7e930de510cad6aef31532b06ba679a72d94a Mon Sep 17 00:00:00 2001
From: Melanie Plageman <melanieplageman@gmail.com>
Date: Sun, 30 Mar 2025 14:03:55 -0400
Subject: [PATCH] Add errcontext for processing I/Os for another backend
Push an ErrorContextCallback adding additional detail about the process
performing the I/O and the owner of the I/O when those are not the same.
For io_method worker, this adds context specifying which process owns
the I/O that the I/O worker is processing.
For io_method io_uring, this adds context only when a backend is
*completing* I/O for another backend. It specifies the pid of the owning
process.
Discussion: https://postgr.es/m/20250325141120.8e.nmisch%40google.com
---
src/backend/storage/aio/aio.c | 1 +
src/backend/storage/aio/method_io_uring.c | 31 +++++++++++++++++++++++
src/backend/storage/aio/method_worker.c | 29 +++++++++++++++++++++
3 files changed, 61 insertions(+)
diff --git a/src/backend/storage/aio/aio.c b/src/backend/storage/aio/aio.c
index 43f1e2a7785..3d5cf726e24 100644
--- a/src/backend/storage/aio/aio.c
+++ b/src/backend/storage/aio/aio.c
@@ -42,6 +42,7 @@
#include "port/atomics.h"
#include "storage/aio_internal.h"
#include "storage/aio_subsys.h"
+#include "storage/proc.h"
#include "storage/procnumber.h"
#include "utils/guc.h"
#include "utils/guc_hooks.h"
diff --git a/src/backend/storage/aio/method_io_uring.c b/src/backend/storage/aio/method_io_uring.c
index 3b299dcf388..244918e1883 100644
--- a/src/backend/storage/aio/method_io_uring.c
+++ b/src/backend/storage/aio/method_io_uring.c
@@ -303,14 +303,41 @@ pgaio_uring_submit(uint16 num_staged_ios, PgAioHandle **staged_ios)
return num_staged_ios;
}
+static void
+pgaio_uring_completion_error_callback(void *arg)
+{
+ ProcNumber owner;
+ PGPROC *owner_proc;
+ int32 owner_pid;
+ PgAioHandle *ioh = arg;
+
+ if (!ioh)
+ return;
+
+ /* No need for context if a backend is completing the IO for itself */
+ if (ioh->owner_procno == MyProcNumber)
+ return;
+
+ owner = ioh->owner_procno;
+ owner_proc = GetPGProcByNumber(owner);
+ owner_pid = owner_proc->pid;
+
+ errcontext("completing I/O on behalf of process %d", owner_pid);
+}
+
static void
pgaio_uring_drain_locked(PgAioUringContext *context)
{
int ready;
int orig_ready;
+ ErrorContextCallback errcallback = {0};
Assert(LWLockHeldByMeInMode(&context->completion_lock, LW_EXCLUSIVE));
+ errcallback.callback = pgaio_uring_completion_error_callback;
+ errcallback.previous = error_context_stack;
+ error_context_stack = &errcallback;
+
/*
* Don't drain more events than available right now. Otherwise it's
* plausible that one backend could get stuck, for a while, receiving CQEs
@@ -338,9 +365,11 @@ pgaio_uring_drain_locked(PgAioUringContext *context)
PgAioHandle *ioh;
ioh = io_uring_cqe_get_data(cqe);
+ errcallback.arg = ioh;
io_uring_cqe_seen(&context->io_uring_ring, cqe);
pgaio_io_process_completion(ioh, cqe->res);
+ errcallback.arg = NULL;
}
END_CRIT_SECTION();
@@ -349,6 +378,8 @@ pgaio_uring_drain_locked(PgAioUringContext *context)
"drained %d/%d, now expecting %d",
ncqes, orig_ready, io_uring_cq_ready(&context->io_uring_ring));
}
+
+ error_context_stack = errcallback.previous;
}
static void
diff --git a/src/backend/storage/aio/method_worker.c b/src/backend/storage/aio/method_worker.c
index 5ea00d8a89e..52f901ed4c2 100644
--- a/src/backend/storage/aio/method_worker.c
+++ b/src/backend/storage/aio/method_worker.c
@@ -361,11 +361,33 @@ pgaio_worker_register(void)
on_shmem_exit(pgaio_worker_die, 0);
}
+static void
+pgaio_worker_error_callback(void *arg)
+{
+ ProcNumber owner;
+ PGPROC *owner_proc;
+ int32 owner_pid;
+ PgAioHandle *ioh = arg;
+
+ if (!ioh)
+ return;
+
+ Assert(ioh->owner_procno != MyProcNumber);
+ Assert(MyBackendType == B_IO_WORKER);
+
+ owner = ioh->owner_procno;
+ owner_proc = GetPGProcByNumber(owner);
+ owner_pid = owner_proc->pid;
+
+ errcontext("I/O worker executing I/O on behalf of process %d", owner_pid);
+}
+
void
IoWorkerMain(const void *startup_data, size_t startup_data_len)
{
sigjmp_buf local_sigjmp_buf;
PgAioHandle *volatile error_ioh = NULL;
+ ErrorContextCallback errcallback = {0};
volatile int error_errno = 0;
char cmd[128];
@@ -392,6 +414,10 @@ IoWorkerMain(const void *startup_data, size_t startup_data_len)
sprintf(cmd, "%d", MyIoWorkerId);
set_ps_display(cmd);
+ errcallback.callback = pgaio_worker_error_callback;
+ errcallback.previous = error_context_stack;
+ error_context_stack = &errcallback;
+
/* see PostgresMain() */
if (sigsetjmp(local_sigjmp_buf, 1) != 0)
{
@@ -475,6 +501,7 @@ IoWorkerMain(const void *startup_data, size_t startup_data_len)
ioh = &pgaio_ctl->io_handles[io_index];
error_ioh = ioh;
+ errcallback.arg = ioh;
pgaio_debug_io(DEBUG4, ioh,
"worker %d processing IO",
@@ -515,6 +542,7 @@ IoWorkerMain(const void *startup_data, size_t startup_data_len)
pgaio_io_perform_synchronously(ioh);
RESUME_INTERRUPTS();
+ errcallback.arg = NULL;
}
else
{
@@ -526,6 +554,7 @@ IoWorkerMain(const void *startup_data, size_t startup_data_len)
CHECK_FOR_INTERRUPTS();
}
+ error_context_stack = errcallback.previous;
proc_exit(0);
}
--
2.34.1
^ permalink raw reply [nested|flat] 153+ messages in thread
* Re: AIO v2.5
2025-03-04 19:00 AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-10 18:23 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-11 19:41 ` Re: AIO v2.5 Noah Misch <noah@leadboat.com>
2025-03-11 23:55 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-12 03:57 ` Re: AIO v2.5 Noah Misch <noah@leadboat.com>
2025-03-12 17:06 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-12 23:23 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-13 10:53 ` Re: AIO v2.5 Antonin Houska <ah@cybertec.at>
2025-03-14 19:43 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-17 20:00 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-18 20:12 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
@ 2025-04-22 22:38 ` Tomas Vondra <tomas@vondra.me>
2025-05-09 01:15 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
6 siblings, 1 reply; 153+ messages in thread
From: Tomas Vondra @ 2025-04-22 22:38 UTC (permalink / raw)
To: Andres Freund <andres@anarazel.de>; Antonin Houska <ah@cybertec.at>; +Cc: Noah Misch <noah@leadboat.com>; pgsql-hackers@postgresql.org, Thomas Munro <thomas.munro@gmail.com>; Heikki Linnakangas <hlinnaka@iki.fi>; Robert Haas <robertmhaas@gmail.com>; Jakub Wartak <jakub.wartak@enterprisedb.com>; Jelte Fennema-Nio <postgres@jeltef.nl>
On 3/18/25 21:12, Andres Freund wrote:
> Hi,
>
> Attached is v2.10, with the following changes:
>
> ...
>
> - committed io_method=worker
>
There's an open item related to this commit (247ce06b883d), based on:
For now the default io_method is changed to "worker". We should re-
evaluate that around beta1, we might want to be careful and set the
default to "sync" for 18.
FWIW the open item is in the "recheck mid-beta" section, but the commit
message says "around beta1" which is not that far away (especially if we
choose to do beta1 on May 8, with the minor releases).
What information we need to gather to make the decision and who's
expected to do it? I assume someone needs to do run some benchmarks, but
is anyone working on it already? What benchmarks, what platforms?
FWIW I'm asking because of the RMT, but I'm also willing to do some of
the tests, if needed - but it'd be good to get some guidance.
regards
--
Tomas Vondra
^ permalink raw reply [nested|flat] 153+ messages in thread
* Re: AIO v2.5
2025-03-04 19:00 AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-10 18:23 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-11 19:41 ` Re: AIO v2.5 Noah Misch <noah@leadboat.com>
2025-03-11 23:55 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-12 03:57 ` Re: AIO v2.5 Noah Misch <noah@leadboat.com>
2025-03-12 17:06 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-12 23:23 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-13 10:53 ` Re: AIO v2.5 Antonin Houska <ah@cybertec.at>
2025-03-14 19:43 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-17 20:00 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-18 20:12 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-04-22 22:38 ` Re: AIO v2.5 Tomas Vondra <tomas@vondra.me>
@ 2025-05-09 01:15 ` Andres Freund <andres@anarazel.de>
2025-07-11 21:03 ` Re: AIO v2.5 Tomas Vondra <tomas@vondra.me>
0 siblings, 1 reply; 153+ messages in thread
From: Andres Freund @ 2025-05-09 01:15 UTC (permalink / raw)
To: Tomas Vondra <tomas@vondra.me>; +Cc: Antonin Houska <ah@cybertec.at>; Noah Misch <noah@leadboat.com>; pgsql-hackers@postgresql.org, Thomas Munro <thomas.munro@gmail.com>; Heikki Linnakangas <hlinnaka@iki.fi>; Robert Haas <robertmhaas@gmail.com>; Jakub Wartak <jakub.wartak@enterprisedb.com>; Jelte Fennema-Nio <postgres@jeltef.nl>
Hi,
On 2025-04-23 00:38:01 +0200, Tomas Vondra wrote:
> There's an open item related to this commit (247ce06b883d), based on:
Oops, forgot to reply.
> For now the default io_method is changed to "worker". We should re-
> evaluate that around beta1, we might want to be careful and set the
> default to "sync" for 18.
>
> FWIW the open item is in the "recheck mid-beta" section, but the commit
> message says "around beta1" which is not that far away (especially if we
> choose to do beta1 on May 8, with the minor releases).
I think we certainly should wait for at least beta1 (not that we have a choice
now), to make sure all of this gets at least some coverage.
> What information we need to gather to make the decision and who's
> expected to do it? I assume someone needs to do run some benchmarks, but
> is anyone working on it already? What benchmarks, what platforms?
I guess I'd say "the community". I think it should depend on how many problems
we find, be it correctness or performance regressions. I don't know how to
make that call without seeing how things are going during beta.
And yes, I think this should involve some benchmarking too - generally a good
thing to do during beta, and if it involves AIO, all the better.
> FWIW I'm asking because of the RMT, but I'm also willing to do some of
> the tests, if needed - but it'd be good to get some guidance.
More testing would be welcome!
Greetings,
Andres Freund
^ permalink raw reply [nested|flat] 153+ messages in thread
* Re: AIO v2.5
2025-03-04 19:00 AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-10 18:23 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-11 19:41 ` Re: AIO v2.5 Noah Misch <noah@leadboat.com>
2025-03-11 23:55 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-12 03:57 ` Re: AIO v2.5 Noah Misch <noah@leadboat.com>
2025-03-12 17:06 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-12 23:23 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-13 10:53 ` Re: AIO v2.5 Antonin Houska <ah@cybertec.at>
2025-03-14 19:43 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-17 20:00 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-18 20:12 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-04-22 22:38 ` Re: AIO v2.5 Tomas Vondra <tomas@vondra.me>
2025-05-09 01:15 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
@ 2025-07-11 21:03 ` Tomas Vondra <tomas@vondra.me>
2025-07-13 18:04 ` Re: AIO v2.5 Tomas Vondra <tomas@vondra.me>
2025-07-14 18:36 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
0 siblings, 2 replies; 153+ messages in thread
From: Tomas Vondra @ 2025-07-11 21:03 UTC (permalink / raw)
To: Andres Freund <andres@anarazel.de>; +Cc: Antonin Houska <ah@cybertec.at>; Noah Misch <noah@leadboat.com>; pgsql-hackers@postgresql.org, Thomas Munro <thomas.munro@gmail.com>; Heikki Linnakangas <hlinnaka@iki.fi>; Robert Haas <robertmhaas@gmail.com>; Jakub Wartak <jakub.wartak@enterprisedb.com>; Jelte Fennema-Nio <postgres@jeltef.nl>
Hi,
I've been running some benchmarks comparing the io_methods, to help with
resolving this PG18 open item. So here are some results, and my brief
analysis of it.
I was hoping to get this out sooner before beta2 :-( and some tests are
still running, but I don't think it'll affect the conclusions.
The TL;DR version
-----------------
* The "worker" method seems good, and I think we should keep it as a
default. We should probably think about increasing the number of workers
a bit, the current io_workers=3 seems to be too low and regresses in a
couple tests.
* The "sync" seems OK too, but it's more of a conservative choice, i.e.
more weight for keeping the PG17 behavior / not causing regressions. But
I haven't seen that (with enough workers). And there are cases when the
"worker" is much faster. It'd be a shame to throw away that benefit.
* There might be bugs in "worker", simply because it has to deal with
multiple concurrent processes etc. But I guess we'll fix those just like
other bugs. I don't think it's a good argument against "worker" default.
* All my tests were done on Linux and NVMe drives. It'd be good to do
similar testing on other platforms (e.g. FreeBSD) and/or storage. I plan
to do some of that, but it'd be great to cover more cases. I can help
with getting my script running, a run takes ~1-2 days.
A more detailed version ...
---------------------------
The benchmark I did works like this:
1) It generates datasets with different data distributions (patterns in
the data - uniform, linear, cyclic, ...). Each table is ~4.3GB of data.
2) It then runs queries on that, with a BETWEEN clause with a certain
selectivity (matching % of the rows), forcing a particular scan type
(indexscan, bitmapscan, seqscan).
3) For each query it measures duration of "cold" and "warm" runs. Cold
means "nothing in page cache / shared buffers", while "warm" means
everything is cached somewhere (by the first "cold" run).
There's a couple relevant parameters varied between the runs:
* effective_io_concurrency = [0, 1, 16, 32]
* shared_buffers = [4GB, 32GB]
Parallel query was disabled for these tests.
The test also included PG17 for comparison, but I forgot PG18 enabled
checksums by default. So PG17 results are with checksums off, which in
some cases means PG17 seems a little bit faster. I'm re-running it with
checksums enabled on PG17, and that seems to eliminate the differences
(as expected).
Scripts
-------
The benchmark scripts/results/charts are available here:
https://github.com/tvondra/iomethod-tests
The SQL data generator and the script driving the benchmark:
* https://github.com/tvondra/iomethod-tests/blob/master/create2.sql
* https://github.com/tvondra/iomethod-tests/blob/master/test-full-cost-2.sh
The script is fairly simple. It first generates the combinations of
parameters to test, randomizes them, and then tests each of them.
Results are recorded in a CSV file "results.csv".
Test machines
-------------
As usual, I did this on two machines I have at home:
* ryzen (~2024)
* Ryzen 9 9900X (12 cores)
* 64GB RAM
* 4x NVMe SSD (Samsung 990 PRO 1TB) in RAID0
* xeon (~2016)
* 2x Xeon 2699v4 (44 cores)
* 64GB RAM
* 1x NVMe SSD WDC Ultrastar DC SN640 960GB
The ryzen is much beefier in terms of I/O, it can do ~20GB/s in
sequential read. The xeon is much more modest (and generally older).
Charts
------
Most of the repository is PDF charts generated from the CSV results.
There's a README explaining the naming convention of the charts, and
some other details.
But in general there are two "types" of charts. The first one fixes all
parameters except for "dataset", and then shows comparison of results
for all datasets.
For example the attached "ryzen-rows-cold-32GB-16-unscaled.pdf" shows
results for:
* ryzen machine
* selectivity calculated as "% of rows" (not pages)
* cold runs, i.e. data not cached
* shared_buffers=32GB
* effective_io_concurrency=16
* unscaled - each plot has custom y-range
There are "scaled" plots too, with all the plots scaled to the same
y-range. This makes it easier to confirm plots from the same PDF (e.g.
how the different scans perform). But that's irrelevant for picking the
io_method default. and it often hides details in some plots.
The other type fixes "dataset" and varies effective_io_concurrency, so
see how it affects duration (e.g. due to prefetching). For example the
attached ryzen-rows-cold-32GB-uniform-unscaled.pdf shows that for the
"uniform" data set.
The charts show behavior for the whole selectivity range, from 0% to
100%, for each scan type. Of course, the scan type may not be the right
choice for that selectivity point. For example, we'd probably not pick
an index scan for 50% selectivity, or a seqscan for 0.01%. But I think
it makes sense to still consider the whole range, for robustness. We
make estimation/planning mistakes fairly regularly, so it's important to
care about what happens in those cases.
Findings
--------
I'm attaching only three PDFs with charts from the cold runs, to keep
the e-mail small (each PDF is ~100-200kB). Feel free to check the other
PDFs in the git repository, but it's all very similar and the attached
PDFs are quite representative.
Some basic observations:
a) index scans
There's almost no difference for indexscans, i.e. the middle column in
the PDFs. There's a bit of variation on some of the cyclic/linear data
sets, but it seems more like random noise than a systemic difference.
Which is not all that surprising, considering index scans don't really
use read_stream yet, so there's no prefetching etc.
b) bitmapscans (ryzen-bitmapscan-uniform.png)
Bitmapscans are much more affected, with a lot of differences. See the
attached PNG image, for example. That shows that at ~10% you get 2x
faster query with io_method=worker than "sync".
And it also shows that PG18 with io_method=sync is much faster than
PG17. Which is a bit surprising, TBH. At first I thought it's due to the
"checksums=off" on PG17 in this run, but that doesn't seem to be the
case - results from PG17 with checksums=on show the same thing. In fact,
the difference increases, which is rather bizarre. (There's a branch
"run2-17-checksums-on" with results from that run.)
I believe this is likely due to the bitmapscan prefetch fixes, which we
decided to not backpatch (IIRC), because the results with e_i_c=0 show
no difference between PG17 and PG18/sync.
The "ryzen" results however demonstrate that 3 workers may be too low.
The timing spikes to ~3000ms (at ~1% selectivity), before quickly
dropping back to ~1000ms. The other datasets show similar difference.
With 12 workers, there's no such problem.
On "xeon" the differences are much smaller, or not visible at all. My
guess it's due to the hardware differences (slower CPU / single NVMe for
storage).
But I'm also wondering what would happen if there are multiple queries
doing bitmapscans. Consider the benchmark is only a backend running
queries, and parallel query is disabled.
The problems with too few workers are much better visible in the
log-scale charts (just add "-log" at the end). See the attached example
(ryzen-bitmapscan-uniform-log.png). It shows how much slower it's
compared to all other methods (effectively regression compared to PG17)
for a huge chunk of the selectivity. I'm sure we'd pick bitmapscan for
some of those queries.
c) seqscan (ryzen-seqscan-uniform.png)
This shows a lot of significant differences between different io_method
options. And by significant I mean ~2x difference, on both machines.
On "ryzen" the "worker" takes ~800ms, while "sync" is ~1700ms. PG17
seems a bit faster, but that's due to missing checksums, and with
checksums it's almost exactly the same as PG18/sync. io_uring is in
between at ~1200ms.
On "xeon" it's very similar, but io_uring does better and worker/3 a bit
worse (which I think is another reason to increase io_workers default).
I believe these differences can be explained by where the work happens
for different io_method cases. And by work I mean memcpy and checksum
verification. With "sync/io_uring" it's in the backend itself, while
with "worker" it's offloaded to the worker processes. So single-process
vs. multi-process memory bandwidth.
Which is nice, but we need to have enough of them ;-)
d) warm runs
The "warm" runs are very uninteresting. There's literally no difference
between the various io_methods. On "ryzen" it's very smooth, while on
"xeon" the "seqscan" is very noisy. I believe this is due to NUMA
effects (two sockets etc.) but I need to verify that. It affects all
io_methods equally, so it's irrelevant for this.
e) indexscan regression (ryzen-indexscan-uniform-pg17-checksums.png)
There's an interesting difference difference I noticed in the run with
checksums on PG17. The full PDF is available here:
https://github.com/tvondra/iomethod-tests/blob/run2-17-checksums-on/ryzen-rows-cold-32GB-16-unscaled...
The interesting thing is that PG17 indexscans on uniform dataset got a
little bit faster. In the attached PDF it's exactly on par with PG18,
but here it got a bit faster. Which makes no sense, if it has to also
verify checksums. I haven't had time to investigate this yet.
Conclusion
----------
That's all I have at the moment. I still think it makes sense to keep
io_method=worker, but bump up the io_workers a bit higher. Could we also
add some suggestions how to pick a good value to the docs?
You can probably find more interesting results in the other PDFs in the
repository. I'm certainly going to keep looking.
You might also run the benchmark on different hardware, and either
build/publish the plots somewhere, or just give me the CSV and I'll do
that. Better to find strange stuff / regressions now.
The repository also has branches with plots showing results with WIP
indexscan prefetching. (It's excluded from the PDFs I presented here).
The conclusions are similar to what we found here - "worker" is good
with enough workers, io_uring is good too. Sync has issues for some of
the data sets, but still helps a lot.
regards
--
Tomas Vondra
Attachments:
[image/png] ryzen-bitmapscan-uniform.png (27.6K, ../../e6db33f3-50de-43d3-9d9f-747c3b376e80@vondra.me/2-ryzen-bitmapscan-uniform.png)
download | view image
[image/png] ryzen-seqscan-uniform.png (31.6K, ../../e6db33f3-50de-43d3-9d9f-747c3b376e80@vondra.me/3-ryzen-seqscan-uniform.png)
download | view image
[application/pdf] ryzen-rows-cold-32GB-16-unscaled.pdf (86.7K, ../../e6db33f3-50de-43d3-9d9f-747c3b376e80@vondra.me/4-ryzen-rows-cold-32GB-16-unscaled.pdf)
download
[application/pdf] ryzen-rows-cold-32GB-uniform-unscaled.pdf (43.4K, ../../e6db33f3-50de-43d3-9d9f-747c3b376e80@vondra.me/5-ryzen-rows-cold-32GB-uniform-unscaled.pdf)
download
[application/pdf] xeon-rows-cold-32GB-16-unscaled.pdf (86.1K, ../../e6db33f3-50de-43d3-9d9f-747c3b376e80@vondra.me/6-xeon-rows-cold-32GB-16-unscaled.pdf)
download
[image/png] ryzen-indexscan-uniform-pg17-checksums.png (26.4K, ../../e6db33f3-50de-43d3-9d9f-747c3b376e80@vondra.me/7-ryzen-indexscan-uniform-pg17-checksums.png)
download | view image
[image/png] ryzen-bitmapscan-uniform-log.png (28.8K, ../../e6db33f3-50de-43d3-9d9f-747c3b376e80@vondra.me/8-ryzen-bitmapscan-uniform-log.png)
download | view image
^ permalink raw reply [nested|flat] 153+ messages in thread
* Re: AIO v2.5
2025-03-04 19:00 AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-10 18:23 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-11 19:41 ` Re: AIO v2.5 Noah Misch <noah@leadboat.com>
2025-03-11 23:55 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-12 03:57 ` Re: AIO v2.5 Noah Misch <noah@leadboat.com>
2025-03-12 17:06 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-12 23:23 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-13 10:53 ` Re: AIO v2.5 Antonin Houska <ah@cybertec.at>
2025-03-14 19:43 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-17 20:00 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-18 20:12 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-04-22 22:38 ` Re: AIO v2.5 Tomas Vondra <tomas@vondra.me>
2025-05-09 01:15 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-07-11 21:03 ` Re: AIO v2.5 Tomas Vondra <tomas@vondra.me>
@ 2025-07-13 18:04 ` Tomas Vondra <tomas@vondra.me>
2025-07-14 18:44 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
1 sibling, 1 reply; 153+ messages in thread
From: Tomas Vondra @ 2025-07-13 18:04 UTC (permalink / raw)
To: Andres Freund <andres@anarazel.de>; +Cc: Antonin Houska <ah@cybertec.at>; Noah Misch <noah@leadboat.com>; pgsql-hackers@postgresql.org, Thomas Munro <thomas.munro@gmail.com>; Heikki Linnakangas <hlinnaka@iki.fi>; Robert Haas <robertmhaas@gmail.com>; Jakub Wartak <jakub.wartak@enterprisedb.com>; Jelte Fennema-Nio <postgres@jeltef.nl>
On 7/11/25 23:03, Tomas Vondra wrote:
> ...
>
> e) indexscan regression (ryzen-indexscan-uniform-pg17-checksums.png)
>
> There's an interesting difference difference I noticed in the run with
> checksums on PG17. The full PDF is available here:
>
> https://github.com/tvondra/iomethod-tests/blob/run2-17-checksums-on/ryzen-rows-cold-32GB-16-unscaled...
>
> The interesting thing is that PG17 indexscans on uniform dataset got a
> little bit faster. In the attached PDF it's exactly on par with PG18,
> but here it got a bit faster. Which makes no sense, if it has to also
> verify checksums. I haven't had time to investigate this yet.
I was intrigued by this, so I looked into this today.
TL;DR I believe it was caused by something in the filesystem or even the
storage devices, making the "PG17" data directory (or maybe even just
the "uniform" table) a bit faster.
I started by reproducing the behavior with an indexscan matching 10% of
the rows, and it was very easy to reproduce the difference shows on the
chart (all timings in milliseconds):
PG17: 14112.800 ms
PG18: 21612.090 ms
This was perfectly reproducible, affecting the whole table (not just one
part of it), etc. At some point I recalled that I might have initialized
the databases in slightly different ways - one by running the SQL, the
other one by pg_dump/pg_restore (likely with multiple jobs).
I couldn't think of any other difference between the data directories,
so I simply reloaded them by pg_restore (from the same dump). Which
however made them both slow :O
And it didn't matter how many jobs are used, or anything else I tried.
But every now and then an instance (17 or 18) happened to be fast
(~14000 ms). Consistently, for all queries on the table, not randomly.
In the end I recreated the (ext4) filesystem, loaded the databases and
now both instances are fast. I have no idea what the root cause was, and
I assume recreating the filesystem destroyed all the evidence.
I'll rerun the tests - will take a couple days. I don't think it's
likely to change the conclusions, though. It should only affect how PG17
compares to PG18, not how the io_methods compare to each other. Also, I
don't think the "xeon" resuls are affected.
regards
--
Tomas Vondra
^ permalink raw reply [nested|flat] 153+ messages in thread
* Re: AIO v2.5
2025-03-04 19:00 AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-10 18:23 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-11 19:41 ` Re: AIO v2.5 Noah Misch <noah@leadboat.com>
2025-03-11 23:55 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-12 03:57 ` Re: AIO v2.5 Noah Misch <noah@leadboat.com>
2025-03-12 17:06 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-12 23:23 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-13 10:53 ` Re: AIO v2.5 Antonin Houska <ah@cybertec.at>
2025-03-14 19:43 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-17 20:00 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-18 20:12 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-04-22 22:38 ` Re: AIO v2.5 Tomas Vondra <tomas@vondra.me>
2025-05-09 01:15 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-07-11 21:03 ` Re: AIO v2.5 Tomas Vondra <tomas@vondra.me>
2025-07-13 18:04 ` Re: AIO v2.5 Tomas Vondra <tomas@vondra.me>
@ 2025-07-14 18:44 ` Andres Freund <andres@anarazel.de>
2025-07-15 00:13 ` Re: AIO v2.5 Tomas Vondra <tomas@vondra.me>
0 siblings, 1 reply; 153+ messages in thread
From: Andres Freund @ 2025-07-14 18:44 UTC (permalink / raw)
To: Tomas Vondra <tomas@vondra.me>; +Cc: Antonin Houska <ah@cybertec.at>; Noah Misch <noah@leadboat.com>; pgsql-hackers@postgresql.org, Thomas Munro <thomas.munro@gmail.com>; Heikki Linnakangas <hlinnaka@iki.fi>; Robert Haas <robertmhaas@gmail.com>; Jakub Wartak <jakub.wartak@enterprisedb.com>; Jelte Fennema-Nio <postgres@jeltef.nl>
On 2025-07-13 20:04:51 +0200, Tomas Vondra wrote:
> On 7/11/25 23:03, Tomas Vondra wrote:
> > ...
> >
> > e) indexscan regression (ryzen-indexscan-uniform-pg17-checksums.png)
> >
> > There's an interesting difference difference I noticed in the run with
> > checksums on PG17. The full PDF is available here:
> >
> > https://github.com/tvondra/iomethod-tests/blob/run2-17-checksums-on/ryzen-rows-cold-32GB-16-unscaled...
> >
> > The interesting thing is that PG17 indexscans on uniform dataset got a
> > little bit faster. In the attached PDF it's exactly on par with PG18,
> > but here it got a bit faster. Which makes no sense, if it has to also
> > verify checksums. I haven't had time to investigate this yet.
>
> I was intrigued by this, so I looked into this today.
>
> TL;DR I believe it was caused by something in the filesystem or even the
> storage devices, making the "PG17" data directory (or maybe even just
> the "uniform" table) a bit faster.
>
> I started by reproducing the behavior with an indexscan matching 10% of
> the rows, and it was very easy to reproduce the difference shows on the
> chart (all timings in milliseconds):
>
> PG17: 14112.800 ms
> PG18: 21612.090 ms
That's a decidedly nontrivial difference.
Did you keep any metrics from those runs? E.g. whether there were larger IOs
or such?
> This was perfectly reproducible, affecting the whole table (not just one
> part of it), etc. At some point I recalled that I might have initialized
> the databases in slightly different ways - one by running the SQL, the
> other one by pg_dump/pg_restore (likely with multiple jobs).
I guess that's an INSERT ... SELECT vs COPY?
Which one was the faster one?
If this ever re-occurs, it might be interesting to look at the fragmentation
of the underlying files with filefrag.
> I couldn't think of any other difference between the data directories,
> so I simply reloaded them by pg_restore (from the same dump). Which
> however made them both slow :O
So that suggests that COPY is the slow case, interesting.
One potentially relevant factor could be that parallel COPY into logged tables
currently leads to really sub-optimal write patterns, due to us writing back
buffers one-by-one, interspersed by WAL writes and file extensions. I know how
that affects write speed, but it's not entirely obvious how it would affect
read speed...
> And it didn't matter how many jobs are used, or anything else I tried.
> But every now and then an instance (17 or 18) happened to be fast
> (~14000 ms). Consistently, for all queries on the table, not randomly.
>
> In the end I recreated the (ext4) filesystem, loaded the databases and
> now both instances are fast. I have no idea what the root cause was, and
> I assume recreating the filesystem destroyed all the evidence.
Besides differences in filesystem level fragmentation, another potential
theory is that the SSDs were internally more fragmented. Occasionally
dumping/restoring the data could allow the drive to do internal wear
leveling before the new data is loaded, leading to a better layout.
I found that I get more consistent benchmark performance if I delete as much
of the data as possible, run fstrim -v -a and then load the data. And do
another round of fstrim.
Greetings,
Andres Freund
^ permalink raw reply [nested|flat] 153+ messages in thread
* Re: AIO v2.5
2025-03-04 19:00 AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-10 18:23 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-11 19:41 ` Re: AIO v2.5 Noah Misch <noah@leadboat.com>
2025-03-11 23:55 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-12 03:57 ` Re: AIO v2.5 Noah Misch <noah@leadboat.com>
2025-03-12 17:06 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-12 23:23 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-13 10:53 ` Re: AIO v2.5 Antonin Houska <ah@cybertec.at>
2025-03-14 19:43 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-17 20:00 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-18 20:12 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-04-22 22:38 ` Re: AIO v2.5 Tomas Vondra <tomas@vondra.me>
2025-05-09 01:15 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-07-11 21:03 ` Re: AIO v2.5 Tomas Vondra <tomas@vondra.me>
2025-07-13 18:04 ` Re: AIO v2.5 Tomas Vondra <tomas@vondra.me>
2025-07-14 18:44 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
@ 2025-07-15 00:13 ` Tomas Vondra <tomas@vondra.me>
0 siblings, 0 replies; 153+ messages in thread
From: Tomas Vondra @ 2025-07-15 00:13 UTC (permalink / raw)
To: Andres Freund <andres@anarazel.de>; +Cc: Antonin Houska <ah@cybertec.at>; Noah Misch <noah@leadboat.com>; pgsql-hackers@postgresql.org, Thomas Munro <thomas.munro@gmail.com>; Heikki Linnakangas <hlinnaka@iki.fi>; Robert Haas <robertmhaas@gmail.com>; Jakub Wartak <jakub.wartak@enterprisedb.com>; Jelte Fennema-Nio <postgres@jeltef.nl>
On 7/14/25 20:44, Andres Freund wrote:
> On 2025-07-13 20:04:51 +0200, Tomas Vondra wrote:
>> On 7/11/25 23:03, Tomas Vondra wrote:
>>> ...
>>>
>>> e) indexscan regression (ryzen-indexscan-uniform-pg17-checksums.png)
>>>
>>> There's an interesting difference difference I noticed in the run with
>>> checksums on PG17. The full PDF is available here:
>>>
>>> https://github.com/tvondra/iomethod-tests/blob/run2-17-checksums-on/ryzen-rows-cold-32GB-16-unscaled...
>>>
>>> The interesting thing is that PG17 indexscans on uniform dataset got a
>>> little bit faster. In the attached PDF it's exactly on par with PG18,
>>> but here it got a bit faster. Which makes no sense, if it has to also
>>> verify checksums. I haven't had time to investigate this yet.
>>
>> I was intrigued by this, so I looked into this today.
>>
>> TL;DR I believe it was caused by something in the filesystem or even the
>> storage devices, making the "PG17" data directory (or maybe even just
>> the "uniform" table) a bit faster.
>>
>> I started by reproducing the behavior with an indexscan matching 10% of
>> the rows, and it was very easy to reproduce the difference shows on the
>> chart (all timings in milliseconds):
>>
>> PG17: 14112.800 ms
>> PG18: 21612.090 ms
>
> That's a decidedly nontrivial difference.
>
It is. It surprised me.
> Did you keep any metrics from those runs? E.g. whether there were larger IOs
> or such?
>
Unfortunately no. I should have investigated more before rebuilding the
filesystem. But I suspect I might be able to reproduce it, if I do the
loads in a loop or something like that.
>
>> This was perfectly reproducible, affecting the whole table (not just one
>> part of it), etc. At some point I recalled that I might have initialized
>> the databases in slightly different ways - one by running the SQL, the
>> other one by pg_dump/pg_restore (likely with multiple jobs).
>
> I guess that's an INSERT ... SELECT vs COPY?
>
> Which one was the faster one?
>
I suspect I initialized one instance by the SQL script that generates
data by INSERT ... SELECT for tables one by one (and then builds
indexes, also one by one). And then I might have initialized the other
database by pg_dump/pg_restore, so that'd be COPY.
FWIW I might have used pg_restore with parallelism, and I suspect that
might be more important than INSERT vs. COPY.
But maybe I'm misremembering things, I rebuilt the benchmark databases
so many times over the past couple days ... not sure.
> If this ever re-occurs, it might be interesting to look at the fragmentation
> of the underlying files with filefrag.
>
Yeah, I'll keep that in mind. And I'll try to reproduce this, once I'm
done with those benchmarks.
>
>> I couldn't think of any other difference between the data directories,
>> so I simply reloaded them by pg_restore (from the same dump). Which
>> however made them both slow :O
>
> So that suggests that COPY is the slow case, interesting.
>
> One potentially relevant factor could be that parallel COPY into logged tables
> currently leads to really sub-optimal write patterns, due to us writing back
> buffers one-by-one, interspersed by WAL writes and file extensions. I know how
> that affects write speed, but it's not entirely obvious how it would affect
> read speed...
>
Maybe. I don't remember which database I loaded first. But this time I
simply copied a backup of the data directory from a different device,
which would "cleanup" the write pattern.
I'd assume filesystems allocate space in larger chunks, e.g. ext4 does
delayed allocation (and I don't see why that wouldn't work here).
>
>> And it didn't matter how many jobs are used, or anything else I tried.
>> But every now and then an instance (17 or 18) happened to be fast
>> (~14000 ms). Consistently, for all queries on the table, not randomly.
>>
>> In the end I recreated the (ext4) filesystem, loaded the databases and
>> now both instances are fast. I have no idea what the root cause was, and
>> I assume recreating the filesystem destroyed all the evidence.
>
> Besides differences in filesystem level fragmentation, another potential
> theory is that the SSDs were internally more fragmented. Occasionally
> dumping/restoring the data could allow the drive to do internal wear
> leveling before the new data is loaded, leading to a better layout.
>
> I found that I get more consistent benchmark performance if I delete as much
> of the data as possible, run fstrim -v -a and then load the data. And do
> another round of fstrim.
>
Maybe, but the filesystem is far from full, which I think helps wear
leveling. The RAID has ~4TB and only ~500GB was ever used. But who
knows, and if that's the cause, we'll never really know I'm afraid. It's
hard to confirm what happens inside a SSD :-(
regards
--
Tomas Vondra
^ permalink raw reply [nested|flat] 153+ messages in thread
* Re: AIO v2.5
2025-03-04 19:00 AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-10 18:23 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-11 19:41 ` Re: AIO v2.5 Noah Misch <noah@leadboat.com>
2025-03-11 23:55 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-12 03:57 ` Re: AIO v2.5 Noah Misch <noah@leadboat.com>
2025-03-12 17:06 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-12 23:23 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-13 10:53 ` Re: AIO v2.5 Antonin Houska <ah@cybertec.at>
2025-03-14 19:43 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-17 20:00 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-18 20:12 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-04-22 22:38 ` Re: AIO v2.5 Tomas Vondra <tomas@vondra.me>
2025-05-09 01:15 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-07-11 21:03 ` Re: AIO v2.5 Tomas Vondra <tomas@vondra.me>
@ 2025-07-14 18:36 ` Andres Freund <andres@anarazel.de>
2025-07-15 00:30 ` Re: AIO v2.5 Tomas Vondra <tomas@vondra.me>
1 sibling, 1 reply; 153+ messages in thread
From: Andres Freund @ 2025-07-14 18:36 UTC (permalink / raw)
To: Tomas Vondra <tomas@vondra.me>; +Cc: Antonin Houska <ah@cybertec.at>; Noah Misch <noah@leadboat.com>; pgsql-hackers@postgresql.org, Thomas Munro <thomas.munro@gmail.com>; Heikki Linnakangas <hlinnaka@iki.fi>; Robert Haas <robertmhaas@gmail.com>; Jakub Wartak <jakub.wartak@enterprisedb.com>; Jelte Fennema-Nio <postgres@jeltef.nl>
Hi,
On 2025-07-11 23:03:53 +0200, Tomas Vondra wrote:
> I've been running some benchmarks comparing the io_methods, to help with
> resolving this PG18 open item. So here are some results, and my brief
> analysis of it.
Thanks for doing that!
> The TL;DR version
> -----------------
>
> * The "worker" method seems good, and I think we should keep it as a
> default. We should probably think about increasing the number of workers
> a bit, the current io_workers=3 seems to be too low and regresses in a
> couple tests.
>
> * The "sync" seems OK too, but it's more of a conservative choice, i.e.
> more weight for keeping the PG17 behavior / not causing regressions. But
> I haven't seen that (with enough workers). And there are cases when the
> "worker" is much faster. It'd be a shame to throw away that benefit.
>
> * There might be bugs in "worker", simply because it has to deal with
> multiple concurrent processes etc. But I guess we'll fix those just like
> other bugs. I don't think it's a good argument against "worker" default.
>
> * All my tests were done on Linux and NVMe drives. It'd be good to do
> similar testing on other platforms (e.g. FreeBSD) and/or storage. I plan
> to do some of that, but it'd be great to cover more cases. I can help
> with getting my script running, a run takes ~1-2 days.
FWIW, in my very limited tests on windows, the benefit of worker was
considerably bigger there, due to having much more minimal readahead not
having posix_fadvise...
> The test also included PG17 for comparison, but I forgot PG18 enabled
> checksums by default. So PG17 results are with checksums off, which in
> some cases means PG17 seems a little bit faster. I'm re-running it with
> checksums enabled on PG17, and that seems to eliminate the differences
> (as expected).
My sneaking suspicion is that, independent of AIO, we're not really ready to
default to checksums defaulting to on.
> Findings
> --------
>
> I'm attaching only three PDFs with charts from the cold runs, to keep
> the e-mail small (each PDF is ~100-200kB). Feel free to check the other
> PDFs in the git repository, but it's all very similar and the attached
> PDFs are quite representative.
>
> Some basic observations:
>
> a) index scans
>
> There's almost no difference for indexscans, i.e. the middle column in
> the PDFs. There's a bit of variation on some of the cyclic/linear data
> sets, but it seems more like random noise than a systemic difference.
>
> Which is not all that surprising, considering index scans don't really
> use read_stream yet, so there's no prefetching etc.
Indeed.
> The "ryzen" results however demonstrate that 3 workers may be too low.
> The timing spikes to ~3000ms (at ~1% selectivity), before quickly
> dropping back to ~1000ms. The other datasets show similar difference.
> With 12 workers, there's no such problem.
I don't really know what to do about that - for now we don't have dynamic
#workers, and starting 12 workers on a tiny database doesn't really make
sense... I suspect that on most hardware & queries it won't matter that much,
but clearly, if you have high iops hardware it might. I can perhaps see
increasing the default to 5 or so, but after that... I guess we could try
some autoconf formula based on the size of s_b or such? But that seems
somewhat awkward too.
>
> e) indexscan regression (ryzen-indexscan-uniform-pg17-checksums.png)
>
> There's an interesting difference difference I noticed in the run with
> checksums on PG17. The full PDF is available here:
(there's a subsequent email about this, will reply there)
> Conclusion
> ----------
>
> That's all I have at the moment. I still think it makes sense to keep
> io_method=worker, but bump up the io_workers a bit higher. Could we also
> add some suggestions how to pick a good value to the docs?
.oO(/me ponders a troll patch to re-add a reference the number of spindles in
that tuning advice)
I'm not sure what advice to give here. Maybe just to set it to a considerably
larger number once not running on a tiny system? The incremental overhead of
having an idle worker is rather small unless you're on a really tiny system...
> You might also run the benchmark on different hardware, and either
> build/publish the plots somewhere, or just give me the CSV and I'll do
> that. Better to find strange stuff / regressions now.
Probably the most interesting thing would be some runs with cloud-ish storage
(relatively high iops, very high latency)...
> The repository also has branches with plots showing results with WIP
> indexscan prefetching. (It's excluded from the PDFs I presented here).
Hm, I looked for those, but I couldn't quickly find any plots that include
them. Would I have to generate the plots from a checkout of the repo?
> The conclusions are similar to what we found here - "worker" is good
> with enough workers, io_uring is good too. Sync has issues for some of
> the data sets, but still helps a lot.
Nice.
Greetings,
Andres Freund
^ permalink raw reply [nested|flat] 153+ messages in thread
* Re: AIO v2.5
2025-03-04 19:00 AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-10 18:23 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-11 19:41 ` Re: AIO v2.5 Noah Misch <noah@leadboat.com>
2025-03-11 23:55 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-12 03:57 ` Re: AIO v2.5 Noah Misch <noah@leadboat.com>
2025-03-12 17:06 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-12 23:23 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-13 10:53 ` Re: AIO v2.5 Antonin Houska <ah@cybertec.at>
2025-03-14 19:43 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-17 20:00 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-18 20:12 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-04-22 22:38 ` Re: AIO v2.5 Tomas Vondra <tomas@vondra.me>
2025-05-09 01:15 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-07-11 21:03 ` Re: AIO v2.5 Tomas Vondra <tomas@vondra.me>
2025-07-14 18:36 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
@ 2025-07-15 00:30 ` Tomas Vondra <tomas@vondra.me>
2025-07-22 13:51 ` Re: AIO v2.5 Tomas Vondra <tomas@vondra.me>
0 siblings, 1 reply; 153+ messages in thread
From: Tomas Vondra @ 2025-07-15 00:30 UTC (permalink / raw)
To: Andres Freund <andres@anarazel.de>; +Cc: Antonin Houska <ah@cybertec.at>; Noah Misch <noah@leadboat.com>; pgsql-hackers@postgresql.org, Thomas Munro <thomas.munro@gmail.com>; Heikki Linnakangas <hlinnaka@iki.fi>; Robert Haas <robertmhaas@gmail.com>; Jakub Wartak <jakub.wartak@enterprisedb.com>; Jelte Fennema-Nio <postgres@jeltef.nl>
On 7/14/25 20:36, Andres Freund wrote:
> Hi,
>
> On 2025-07-11 23:03:53 +0200, Tomas Vondra wrote:
>> I've been running some benchmarks comparing the io_methods, to help with
>> resolving this PG18 open item. So here are some results, and my brief
>> analysis of it.
>
> Thanks for doing that!
>
>
>
>> The TL;DR version
>> -----------------
>>
>> * The "worker" method seems good, and I think we should keep it as a
>> default. We should probably think about increasing the number of workers
>> a bit, the current io_workers=3 seems to be too low and regresses in a
>> couple tests.
>>
>> * The "sync" seems OK too, but it's more of a conservative choice, i.e.
>> more weight for keeping the PG17 behavior / not causing regressions. But
>> I haven't seen that (with enough workers). And there are cases when the
>> "worker" is much faster. It'd be a shame to throw away that benefit.
>>
>> * There might be bugs in "worker", simply because it has to deal with
>> multiple concurrent processes etc. But I guess we'll fix those just like
>> other bugs. I don't think it's a good argument against "worker" default.
>>
>> * All my tests were done on Linux and NVMe drives. It'd be good to do
>> similar testing on other platforms (e.g. FreeBSD) and/or storage. I plan
>> to do some of that, but it'd be great to cover more cases. I can help
>> with getting my script running, a run takes ~1-2 days.
>
> FWIW, in my very limited tests on windows, the benefit of worker was
> considerably bigger there, due to having much more minimal readahead not
> having posix_fadvise...
>
>
>> The test also included PG17 for comparison, but I forgot PG18 enabled
>> checksums by default. So PG17 results are with checksums off, which in
>> some cases means PG17 seems a little bit faster. I'm re-running it with
>> checksums enabled on PG17, and that seems to eliminate the differences
>> (as expected).
>
> My sneaking suspicion is that, independent of AIO, we're not really ready to
> default to checksums defaulting to on.
>
>
>> Findings
>> --------
>>
>> I'm attaching only three PDFs with charts from the cold runs, to keep
>> the e-mail small (each PDF is ~100-200kB). Feel free to check the other
>> PDFs in the git repository, but it's all very similar and the attached
>> PDFs are quite representative.
>>
>> Some basic observations:
>>
>> a) index scans
>>
>> There's almost no difference for indexscans, i.e. the middle column in
>> the PDFs. There's a bit of variation on some of the cyclic/linear data
>> sets, but it seems more like random noise than a systemic difference.
>>
>> Which is not all that surprising, considering index scans don't really
>> use read_stream yet, so there's no prefetching etc.
>
> Indeed.
>
>
>> The "ryzen" results however demonstrate that 3 workers may be too low.
>> The timing spikes to ~3000ms (at ~1% selectivity), before quickly
>> dropping back to ~1000ms. The other datasets show similar difference.
>> With 12 workers, there's no such problem.
>
> I don't really know what to do about that - for now we don't have dynamic
> #workers, and starting 12 workers on a tiny database doesn't really make
> sense... I suspect that on most hardware & queries it won't matter that much,
> but clearly, if you have high iops hardware it might. I can perhaps see
> increasing the default to 5 or so, but after that... I guess we could try
> some autoconf formula based on the size of s_b or such? But that seems
> somewhat awkward too.
>
True. I don't have a great idea either. FWIW most of our defaults are
very conservative/low, and you have to bump them up on bigger machines
anyway. So having to bump one more GUC is not a big deal, and I don't
think we have to invent some magic formula for this one.
Also, autoconf wouldn't even know about s_b size, it'd have to be
something at startup. We could do some automated sizing if set to -1,
perhaps. But is s_b even a good value to tie this to? I doubt that.
>
>
>>
>> e) indexscan regression (ryzen-indexscan-uniform-pg17-checksums.png)
>>
>> There's an interesting difference difference I noticed in the run with
>> checksums on PG17. The full PDF is available here:
>
> (there's a subsequent email about this, will reply there)
>
>
>> Conclusion
>> ----------
>>
>> That's all I have at the moment. I still think it makes sense to keep
>> io_method=worker, but bump up the io_workers a bit higher. Could we also
>> add some suggestions how to pick a good value to the docs?
>
> .oO(/me ponders a troll patch to re-add a reference the number of spindles in
> that tuning advice)
>
;-)
> I'm not sure what advice to give here. Maybe just to set it to a considerably
> larger number once not running on a tiny system? The incremental overhead of
> having an idle worker is rather small unless you're on a really tiny system...
>
Too bad the patch doesn't collect any stats about how utilized the
workers are :-( That'd make it a bit easier, we could even print
something into the log if the queues overflow "too often", similarly to
max_wal_size when checkpoints happen too often.
>
>> You might also run the benchmark on different hardware, and either
>> build/publish the plots somewhere, or just give me the CSV and I'll do
>> that. Better to find strange stuff / regressions now.
>
> Probably the most interesting thing would be some runs with cloud-ish storage
> (relatively high iops, very high latency)...
>
Yeah, I've started a test on a cloud VM. Will see in a day or two. And
another on FreeBSD, for good measure.
>
>> The repository also has branches with plots showing results with WIP
>> indexscan prefetching. (It's excluded from the PDFs I presented here).
>
> Hm, I looked for those, but I couldn't quickly find any plots that include
> them. Would I have to generate the plots from a checkout of the repo?
>
No, the charts are there, you don't need to generate them.
Look into the "with-indexscan-prefetch-run2-17-checksums" branch. E.g.
this is the same "ryzen" plot I shared earlier, but with "indexscan
prefetch" column:
https://github.com/tvondra/iomethod-tests/blob/with-indexscan-prefetch-run2-17-checksums/ryzen-rows-...
It might be better to look at the "scaled" charts, which makes it easier
to compare the different scans (and the benefit of prefetching).
https://github.com/tvondra/iomethod-tests/blob/with-indexscan-prefetch-run2-17-checksums/ryzen-rows-...
>
>> The conclusions are similar to what we found here - "worker" is good
>> with enough workers, io_uring is good too. Sync has issues for some of
>> the data sets, but still helps a lot.
>
> Nice.
>
> Greetings,
>
> Andres Freund
regards
--
Tomas Vondra
^ permalink raw reply [nested|flat] 153+ messages in thread
* Re: AIO v2.5
2025-03-04 19:00 AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-10 18:23 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-11 19:41 ` Re: AIO v2.5 Noah Misch <noah@leadboat.com>
2025-03-11 23:55 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-12 03:57 ` Re: AIO v2.5 Noah Misch <noah@leadboat.com>
2025-03-12 17:06 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-12 23:23 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-13 10:53 ` Re: AIO v2.5 Antonin Houska <ah@cybertec.at>
2025-03-14 19:43 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-17 20:00 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-18 20:12 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-04-22 22:38 ` Re: AIO v2.5 Tomas Vondra <tomas@vondra.me>
2025-05-09 01:15 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-07-11 21:03 ` Re: AIO v2.5 Tomas Vondra <tomas@vondra.me>
2025-07-14 18:36 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-07-15 00:30 ` Re: AIO v2.5 Tomas Vondra <tomas@vondra.me>
@ 2025-07-22 13:51 ` Tomas Vondra <tomas@vondra.me>
2025-07-29 16:41 ` Re: AIO v2.5 Nathan Bossart <nathandbossart@gmail.com>
0 siblings, 1 reply; 153+ messages in thread
From: Tomas Vondra @ 2025-07-22 13:51 UTC (permalink / raw)
To: Andres Freund <andres@anarazel.de>; +Cc: Antonin Houska <ah@cybertec.at>; Noah Misch <noah@leadboat.com>; pgsql-hackers@postgresql.org, Thomas Munro <thomas.munro@gmail.com>; Heikki Linnakangas <hlinnaka@iki.fi>; Robert Haas <robertmhaas@gmail.com>; Jakub Wartak <jakub.wartak@enterprisedb.com>; Jelte Fennema-Nio <postgres@jeltef.nl>
Hi,
I kept running the tests on additional storage systems, both "local"
(e.g. RAID on SATA SSDs) and "cloud" (Azure Premium SSD volumes). And
also on FreeBSD/ZFS, to see what non-Linux systems do. I pushed all the
results to the same github repository, into separate branches.
I'm attaching three example charts, from these "new" systems.
In general, it doesn't change the earlier conclusions at all. Most of
the earlier observations apply to these results from these machines with
reasonably different types of storage. So I still think we should stick
to "worker", but consider increasing the default number of workers.
It's possible there's machine with storage/hardware that would work
better with some other io_method value, but that's why it's a GUC. The
default should work for most systems.
I plan to let the current tests complete, but that's about it. I don't
plan to test additional systems for this thread. I'll keep running tests
for the index-prefetch thread, but that's focused on other questions.
regards
--
Tomas Vondra
Attachments:
[application/pdf] freebsd_d16-rows-cold-32GB-16-unscaled.pdf (163.9K, ../../21674729-87b9-4bdc-9f36-13f8072b550d@vondra.me/2-freebsd_d16-rows-cold-32GB-16-unscaled.pdf)
download
[application/pdf] d16_premium_ssd-rows-cold-32GB-16-unscaled.pdf (195.3K, ../../21674729-87b9-4bdc-9f36-13f8072b550d@vondra.me/3-d16_premium_ssd-rows-cold-32GB-16-unscaled.pdf)
download
[application/pdf] ryzen_sata-rows-cold-32GB-16-unscaled.pdf (194.2K, ../../21674729-87b9-4bdc-9f36-13f8072b550d@vondra.me/4-ryzen_sata-rows-cold-32GB-16-unscaled.pdf)
download
^ permalink raw reply [nested|flat] 153+ messages in thread
* Re: AIO v2.5
2025-03-04 19:00 AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-10 18:23 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-11 19:41 ` Re: AIO v2.5 Noah Misch <noah@leadboat.com>
2025-03-11 23:55 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-12 03:57 ` Re: AIO v2.5 Noah Misch <noah@leadboat.com>
2025-03-12 17:06 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-12 23:23 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-13 10:53 ` Re: AIO v2.5 Antonin Houska <ah@cybertec.at>
2025-03-14 19:43 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-17 20:00 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-18 20:12 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-04-22 22:38 ` Re: AIO v2.5 Tomas Vondra <tomas@vondra.me>
2025-05-09 01:15 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-07-11 21:03 ` Re: AIO v2.5 Tomas Vondra <tomas@vondra.me>
2025-07-14 18:36 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-07-15 00:30 ` Re: AIO v2.5 Tomas Vondra <tomas@vondra.me>
2025-07-22 13:51 ` Re: AIO v2.5 Tomas Vondra <tomas@vondra.me>
@ 2025-07-29 16:41 ` Nathan Bossart <nathandbossart@gmail.com>
0 siblings, 0 replies; 153+ messages in thread
From: Nathan Bossart @ 2025-07-29 16:41 UTC (permalink / raw)
To: Tomas Vondra <tomas@vondra.me>; +Cc: Andres Freund <andres@anarazel.de>; Antonin Houska <ah@cybertec.at>; Noah Misch <noah@leadboat.com>; pgsql-hackers@postgresql.org, Thomas Munro <thomas.munro@gmail.com>; Heikki Linnakangas <hlinnaka@iki.fi>; Robert Haas <robertmhaas@gmail.com>; Jakub Wartak <jakub.wartak@enterprisedb.com>; Jelte Fennema-Nio <postgres@jeltef.nl>
[RMT hat]
On Tue, Jul 22, 2025 at 03:51:12PM +0200, Tomas Vondra wrote:
> In general, it doesn't change the earlier conclusions at all. Most of
> the earlier observations apply to these results from these machines with
> reasonably different types of storage. So I still think we should stick
> to "worker", but consider increasing the default number of workers.
>
> It's possible there's machine with storage/hardware that would work
> better with some other io_method value, but that's why it's a GUC. The
> default should work for most systems.
Given there have been no further proposals or discussion, we have marked
the related open item as "Won't Fix". This means we plan to ship v18 with
io_method=worker and io_workers=3 by default.
--
nathan
^ permalink raw reply [nested|flat] 153+ messages in thread
* Re: AIO v2.5
2025-03-04 19:00 AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-10 18:23 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-11 19:41 ` Re: AIO v2.5 Noah Misch <noah@leadboat.com>
2025-03-11 23:55 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-12 03:57 ` Re: AIO v2.5 Noah Misch <noah@leadboat.com>
2025-03-12 17:06 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-12 23:23 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-13 10:53 ` Re: AIO v2.5 Antonin Houska <ah@cybertec.at>
2025-03-14 19:43 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
@ 2025-03-17 20:53 ` Melanie Plageman <melanieplageman@gmail.com>
3 siblings, 0 replies; 153+ messages in thread
From: Melanie Plageman @ 2025-03-17 20:53 UTC (permalink / raw)
To: Andres Freund <andres@anarazel.de>; +Cc: Antonin Houska <ah@cybertec.at>; Noah Misch <noah@leadboat.com>; pgsql-hackers@postgresql.org, Thomas Munro <thomas.munro@gmail.com>; Heikki Linnakangas <hlinnaka@iki.fi>; Robert Haas <robertmhaas@gmail.com>; Jakub Wartak <jakub.wartak@enterprisedb.com>; Jelte Fennema-Nio <postgres@jeltef.nl>
On Fri, Mar 14, 2025 at 3:43 PM Andres Freund <andres@anarazel.de> wrote:
>
> Open items:
>
> - Right now effective_io_concurrency cannot be set > 0 on Windows and other
> platforms that lack posix_fadvise. But with AIO we can read ahead without
> posix_fadvise().
>
> It'd not really make anything worse than today to not remove the limit, but
> it'd be pretty weird to prevent windows etc from benefiting from AIO. Need
> to look around and see whether it would require anything other than doc
> changes.
I've attached a patch that removes the limit for
effective_io_concurrency and maintenance_io_concurrency. I tested both
GUCs with fadvise manually disabled on my system and I think it is
working for those read stream users I tried (vacuum and BHS).
I checked around to make sure no one was using only the value of the
guc to guard prefetches, and it seems like we're safe.
The one thing I am wondering about with the docs is whether or not we
need to make it more clear that only a subset of the "simultaneous
I/O" behavior controlled by eic/mic is available if your system
doesn't have fadvise. I tried to do that a bit, but I avoided getting
into too many details.
- Melanie
Attachments:
[text/x-patch] Enable-IO-concurrency-on-all-systems.patch (10.8K, ../../CAAKRu_atGgZePo=_g6T3cNtfMf0QxpvoUh5OUqa_cnPdhLd=gw@mail.gmail.com/2-Enable-IO-concurrency-on-all-systems.patch)
download | inline diff:
From 398282293c38f9dfb4f03839a4a7962887c9c0c9 Mon Sep 17 00:00:00 2001
From: Melanie Plageman <melanieplageman@gmail.com>
Date: Mon, 17 Mar 2025 15:17:21 -0400
Subject: [PATCH v1] Enable IO concurrency on all systems
Previously effective_io_concurrency and maintenance_io_concurrency could
not be set above 0 on machines without fadvise support. AIO enables
IO concurrency without such support.
Currently only subsystems using the read stream API will take advantage
of this. Other users of maintenance_io_concurrency (like recovery
prefetching) which leverage OS advice directly will not benefit from
this change. In those cases, maintenance_io_concurrency will have no
effect on I/O behavior.
---
doc/src/sgml/config.sgml | 15 +++++-----
doc/src/sgml/ref/alter_tablespace.sgml | 2 +-
doc/src/sgml/ref/create_tablespace.sgml | 2 +-
src/backend/access/common/reloptions.c | 8 -----
src/backend/commands/variable.c | 30 -------------------
src/backend/utils/misc/guc_tables.c | 4 +--
src/backend/utils/misc/postgresql.conf.sample | 5 ++--
src/bin/initdb/initdb.c | 5 ----
src/include/storage/bufmgr.h | 6 ----
src/include/utils/guc_hooks.h | 4 ---
10 files changed, 14 insertions(+), 67 deletions(-)
diff --git a/doc/src/sgml/config.sgml b/doc/src/sgml/config.sgml
index 69638d7c1f9..2a7587de320 100644
--- a/doc/src/sgml/config.sgml
+++ b/doc/src/sgml/config.sgml
@@ -2581,8 +2581,7 @@ include_dir 'conf.d'
session attempts to initiate in parallel. The allowed range is
<literal>1</literal> to <literal>1000</literal>, or
<literal>0</literal> to disable issuance of asynchronous I/O requests.
- The default is <literal>16</literal> on supported systems, otherwise
- <literal>0</literal>.
+ The default is <literal>16</literal>.
</para>
<para>
@@ -2593,8 +2592,8 @@ include_dir 'conf.d'
</para>
<para>
- On systems without prefetch advice support, attempting to configure
- any value other than <literal>0</literal> will error out.
+ On systems with prefetch advice support,
+ <varname>effective_io_concurrency</varname> also controls the prefetch distance.
</para>
<para>
@@ -2617,10 +2616,10 @@ include_dir 'conf.d'
for maintenance work that is done on behalf of many client sessions.
</para>
<para>
- The default is 10 on supported systems, otherwise 0. This value can
- be overridden for tables in a particular tablespace by setting the
- tablespace parameter of the same name (see
- <xref linkend="sql-altertablespace"/>).
+ The default is <literal>10</literal>. This value can be overridden
+ for tables in a particular tablespace by setting the tablespace
+ parameter of the same name (see <xref
+ linkend="sql-altertablespace"/>).
</para>
</listitem>
</varlistentry>
diff --git a/doc/src/sgml/ref/alter_tablespace.sgml b/doc/src/sgml/ref/alter_tablespace.sgml
index 6ec863400d1..d0e08089ddb 100644
--- a/doc/src/sgml/ref/alter_tablespace.sgml
+++ b/doc/src/sgml/ref/alter_tablespace.sgml
@@ -88,7 +88,7 @@ ALTER TABLESPACE <replaceable>name</replaceable> RESET ( <replaceable class="par
and <varname>maintenance_io_concurrency</varname>.
Setting these values for a particular tablespace will override the
planner's usual estimate of the cost of reading pages from tables in
- that tablespace, and the executor's prefetching behavior, as established
+ that tablespace, and how many concurrent I/Os are issued, as established
by the configuration parameters of the
same name (see <xref linkend="guc-seq-page-cost"/>,
<xref linkend="guc-random-page-cost"/>,
diff --git a/doc/src/sgml/ref/create_tablespace.sgml b/doc/src/sgml/ref/create_tablespace.sgml
index 9d5ab025261..b77e774c53f 100644
--- a/doc/src/sgml/ref/create_tablespace.sgml
+++ b/doc/src/sgml/ref/create_tablespace.sgml
@@ -110,7 +110,7 @@ CREATE TABLESPACE <replaceable class="parameter">tablespace_name</replaceable>
and <varname>maintenance_io_concurrency</varname>.
Setting these values for a particular tablespace will override the
planner's usual estimate of the cost of reading pages from tables in
- that tablespace, and the executor's prefetching behavior, as established
+ that tablespace, and how many concurrent I/Os are issued, as established
by the configuration parameters of the
same name (see <xref linkend="guc-seq-page-cost"/>,
<xref linkend="guc-random-page-cost"/>,
diff --git a/src/backend/access/common/reloptions.c b/src/backend/access/common/reloptions.c
index 59fb53e7707..3e43ad064c3 100644
--- a/src/backend/access/common/reloptions.c
+++ b/src/backend/access/common/reloptions.c
@@ -361,11 +361,7 @@ static relopt_int intRelOpts[] =
RELOPT_KIND_TABLESPACE,
ShareUpdateExclusiveLock
},
-#ifdef USE_PREFETCH
-1, 0, MAX_IO_CONCURRENCY
-#else
- 0, 0, 0
-#endif
},
{
{
@@ -374,11 +370,7 @@ static relopt_int intRelOpts[] =
RELOPT_KIND_TABLESPACE,
ShareUpdateExclusiveLock
},
-#ifdef USE_PREFETCH
-1, 0, MAX_IO_CONCURRENCY
-#else
- 0, 0, 0
-#endif
},
{
{
diff --git a/src/backend/commands/variable.c b/src/backend/commands/variable.c
index 4ad6e236d69..7188afd2860 100644
--- a/src/backend/commands/variable.c
+++ b/src/backend/commands/variable.c
@@ -1145,7 +1145,6 @@ check_cluster_name(char **newval, void **extra, GucSource source)
void
assign_maintenance_io_concurrency(int newval, void *extra)
{
-#ifdef USE_PREFETCH
/*
* Reconfigure recovery prefetching, because a setting it depends on
* changed.
@@ -1153,7 +1152,6 @@ assign_maintenance_io_concurrency(int newval, void *extra)
maintenance_io_concurrency = newval;
if (AmStartupProcess())
XLogPrefetchReconfigure();
-#endif
}
@@ -1231,34 +1229,6 @@ check_default_with_oids(bool *newval, void **extra, GucSource source)
return true;
}
-bool
-check_effective_io_concurrency(int *newval, void **extra, GucSource source)
-{
-#ifndef USE_PREFETCH
- if (*newval != 0)
- {
- GUC_check_errdetail("\"%s\" must be set to 0 on platforms that lack support for issuing read-ahead advice.",
- "effective_io_concurrency");
- return false;
- }
-#endif /* USE_PREFETCH */
- return true;
-}
-
-bool
-check_maintenance_io_concurrency(int *newval, void **extra, GucSource source)
-{
-#ifndef USE_PREFETCH
- if (*newval != 0)
- {
- GUC_check_errdetail("\"%s\" must be set to 0 on platforms that lack support for issuing read-ahead advice.",
- "maintenance_io_concurrency");
- return false;
- }
-#endif /* USE_PREFETCH */
- return true;
-}
-
bool
check_ssl(bool *newval, void **extra, GucSource source)
{
diff --git a/src/backend/utils/misc/guc_tables.c b/src/backend/utils/misc/guc_tables.c
index 7290793c9a1..2c1ca98a3e5 100644
--- a/src/backend/utils/misc/guc_tables.c
+++ b/src/backend/utils/misc/guc_tables.c
@@ -3224,7 +3224,7 @@ struct config_int ConfigureNamesInt[] =
&effective_io_concurrency,
DEFAULT_EFFECTIVE_IO_CONCURRENCY,
0, MAX_IO_CONCURRENCY,
- check_effective_io_concurrency, NULL, NULL
+ NULL, NULL, NULL
},
{
@@ -3238,7 +3238,7 @@ struct config_int ConfigureNamesInt[] =
&maintenance_io_concurrency,
DEFAULT_MAINTENANCE_IO_CONCURRENCY,
0, MAX_IO_CONCURRENCY,
- check_maintenance_io_concurrency, assign_maintenance_io_concurrency,
+ NULL, assign_maintenance_io_concurrency,
NULL
},
diff --git a/src/backend/utils/misc/postgresql.conf.sample b/src/backend/utils/misc/postgresql.conf.sample
index ea71e9075ff..7bd7479190d 100644
--- a/src/backend/utils/misc/postgresql.conf.sample
+++ b/src/backend/utils/misc/postgresql.conf.sample
@@ -198,8 +198,9 @@
# - I/O -
#backend_flush_after = 0 # measured in pages, 0 disables
-#effective_io_concurrency = 16 # 1-1000; 0 disables prefetching
-#maintenance_io_concurrency = 10 # 1-1000; 0 disables prefetching
+#effective_io_concurrency = 16 # 1-1000; 0 disables issuing multiple simultaneous IO requests
+ # values > 0 enable prefetching on systems supporting it
+#maintenance_io_concurrency = 10 # 1-1000; same as effective_io_concurrency
#io_combine_limit = 128kB # usually 1-32 blocks (depends on OS)
#io_method = worker # worker, io_uring, sync
diff --git a/src/bin/initdb/initdb.c b/src/bin/initdb/initdb.c
index 21a0fe3ecd9..b9d75799ee0 100644
--- a/src/bin/initdb/initdb.c
+++ b/src/bin/initdb/initdb.c
@@ -1401,11 +1401,6 @@ setup_config(void)
repltok, true);
#endif
-#ifndef USE_PREFETCH
- conflines = replace_guc_value(conflines, "effective_io_concurrency",
- "0", true);
-#endif
-
#ifdef WIN32
conflines = replace_guc_value(conflines, "update_process_title",
"off", true);
diff --git a/src/include/storage/bufmgr.h b/src/include/storage/bufmgr.h
index 30d965ec069..c425cc816e8 100644
--- a/src/include/storage/bufmgr.h
+++ b/src/include/storage/bufmgr.h
@@ -156,14 +156,8 @@ extern PGDLLIMPORT int bgwriter_lru_maxpages;
extern PGDLLIMPORT double bgwriter_lru_multiplier;
extern PGDLLIMPORT bool track_io_timing;
-/* only applicable when prefetching is available */
-#ifdef USE_PREFETCH
#define DEFAULT_EFFECTIVE_IO_CONCURRENCY 16
#define DEFAULT_MAINTENANCE_IO_CONCURRENCY 10
-#else
-#define DEFAULT_EFFECTIVE_IO_CONCURRENCY 0
-#define DEFAULT_MAINTENANCE_IO_CONCURRENCY 0
-#endif
extern PGDLLIMPORT int effective_io_concurrency;
extern PGDLLIMPORT int maintenance_io_concurrency;
diff --git a/src/include/utils/guc_hooks.h b/src/include/utils/guc_hooks.h
index a3eba8fbe21..74de7a6abdf 100644
--- a/src/include/utils/guc_hooks.h
+++ b/src/include/utils/guc_hooks.h
@@ -61,8 +61,6 @@ extern bool check_default_text_search_config(char **newval, void **extra, GucSou
extern void assign_default_text_search_config(const char *newval, void *extra);
extern bool check_default_with_oids(bool *newval, void **extra,
GucSource source);
-extern bool check_effective_io_concurrency(int *newval, void **extra,
- GucSource source);
extern bool check_huge_page_size(int *newval, void **extra, GucSource source);
extern void assign_io_method(int newval, void *extra);
extern bool check_io_max_concurrency(int *newval, void **extra, GucSource source);
@@ -83,8 +81,6 @@ extern bool check_log_stats(bool *newval, void **extra, GucSource source);
extern bool check_log_timezone(char **newval, void **extra, GucSource source);
extern void assign_log_timezone(const char *newval, void *extra);
extern const char *show_log_timezone(void);
-extern bool check_maintenance_io_concurrency(int *newval, void **extra,
- GucSource source);
extern void assign_maintenance_io_concurrency(int newval, void *extra);
extern bool check_max_slot_wal_keep_size(int *newval, void **extra,
GucSource source);
--
2.34.1
^ permalink raw reply [nested|flat] 153+ messages in thread
* Re: AIO v2.5
2025-03-04 19:00 AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-10 18:23 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-11 19:41 ` Re: AIO v2.5 Noah Misch <noah@leadboat.com>
2025-03-11 23:55 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-12 03:57 ` Re: AIO v2.5 Noah Misch <noah@leadboat.com>
2025-03-12 17:06 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-12 23:23 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-13 10:53 ` Re: AIO v2.5 Antonin Houska <ah@cybertec.at>
2025-03-14 19:43 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
@ 2025-06-30 06:58 ` Antonin Houska <ah@cybertec.at>
2025-06-30 14:44 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
3 siblings, 1 reply; 153+ messages in thread
From: Antonin Houska @ 2025-06-30 06:58 UTC (permalink / raw)
To: Andres Freund <andres@anarazel.de>; +Cc: Noah Misch <noah@leadboat.com>; pgsql-hackers@postgresql.org, Thomas Munro <thomas.munro@gmail.com>; Heikki Linnakangas <hlinnaka@iki.fi>; Robert Haas <robertmhaas@gmail.com>; Jakub Wartak <jakub.wartak@enterprisedb.com>; Jelte Fennema-Nio <postgres@jeltef.nl>
Andres Freund <andres@anarazel.de> wrote:
> On 2025-03-13 11:53:03 +0100, Antonin Houska wrote:
> > Attached are a few proposals for minor comment fixes.
>
> Thanks, applied.
After reading the code a bit more, I noticed that the 'cb_flags' argument of
PgAioHandleCallbackStage is not really used, at least in the existing
callbacks. Is there an intention to use it in the future (the patches for
async write do not seem to indicate so), or is this only a leftover from
previous versions of the patch?
Besides that, I suggest a minor comment fix.
--
Antonin Houska
Web: https://www.cybertec-postgresql.com
Attachments:
[text/x-diff] comment.diff (481B, ../../5250.1751266701@localhost/2-comment.diff)
download | inline diff:
diff --git a/src/include/storage/aio_types.h b/src/include/storage/aio_types.h
index 18183366077..afee85c787b 100644
--- a/src/include/storage/aio_types.h
+++ b/src/include/storage/aio_types.h
@@ -107,7 +107,7 @@ typedef struct PgAioResult
/* of type PgAioResultStatus, see above */
uint32 status:PGAIO_RESULT_STATUS_BITS;
- /* meaning defined by callback->error */
+ /* meaning defined by callback->report */
uint32 error_data:PGAIO_RESULT_ERROR_BITS;
int32 result;
^ permalink raw reply [nested|flat] 153+ messages in thread
* Re: AIO v2.5
2025-03-04 19:00 AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-10 18:23 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-11 19:41 ` Re: AIO v2.5 Noah Misch <noah@leadboat.com>
2025-03-11 23:55 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-12 03:57 ` Re: AIO v2.5 Noah Misch <noah@leadboat.com>
2025-03-12 17:06 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-12 23:23 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-13 10:53 ` Re: AIO v2.5 Antonin Houska <ah@cybertec.at>
2025-03-14 19:43 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-06-30 06:58 ` Re: AIO v2.5 Antonin Houska <ah@cybertec.at>
@ 2025-06-30 14:44 ` Andres Freund <andres@anarazel.de>
2025-07-09 11:26 ` Re: AIO v2.5 Matthias van de Meent <boekewurm+postgres@gmail.com>
0 siblings, 1 reply; 153+ messages in thread
From: Andres Freund @ 2025-06-30 14:44 UTC (permalink / raw)
To: Antonin Houska <ah@cybertec.at>; +Cc: Noah Misch <noah@leadboat.com>; pgsql-hackers@postgresql.org, Thomas Munro <thomas.munro@gmail.com>; Heikki Linnakangas <hlinnaka@iki.fi>; Robert Haas <robertmhaas@gmail.com>; Jakub Wartak <jakub.wartak@enterprisedb.com>; Jelte Fennema-Nio <postgres@jeltef.nl>
Hi,
On 2025-06-30 08:58:21 +0200, Antonin Houska wrote:
> Andres Freund <andres@anarazel.de> wrote:
>
> > On 2025-03-13 11:53:03 +0100, Antonin Houska wrote:
> > > Attached are a few proposals for minor comment fixes.
> >
> > Thanks, applied.
>
> After reading the code a bit more, I noticed that the 'cb_flags' argument of
> PgAioHandleCallbackStage is not really used, at least in the existing
> callbacks. Is there an intention to use it in the future (the patches for
> async write do not seem to indicate so), or is this only a leftover from
> previous versions of the patch?
Thanks for reading through it!
It seemed more symmetric to have it for stage as well as the completion
callback - and I still think that...
> Besides that, I suggest a minor comment fix.
Thanks. Pushed that fix.
Greetings,
Andres Freund
^ permalink raw reply [nested|flat] 153+ messages in thread
* Re: AIO v2.5
2025-03-04 19:00 AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-10 18:23 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-11 19:41 ` Re: AIO v2.5 Noah Misch <noah@leadboat.com>
2025-03-11 23:55 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-12 03:57 ` Re: AIO v2.5 Noah Misch <noah@leadboat.com>
2025-03-12 17:06 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-12 23:23 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-13 10:53 ` Re: AIO v2.5 Antonin Houska <ah@cybertec.at>
2025-03-14 19:43 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-06-30 06:58 ` Re: AIO v2.5 Antonin Houska <ah@cybertec.at>
2025-06-30 14:44 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
@ 2025-07-09 11:26 ` Matthias van de Meent <boekewurm+postgres@gmail.com>
2025-07-09 14:59 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
0 siblings, 1 reply; 153+ messages in thread
From: Matthias van de Meent @ 2025-07-09 11:26 UTC (permalink / raw)
To: Andres Freund <andres@anarazel.de>; +Cc: Antonin Houska <ah@cybertec.at>; Noah Misch <noah@leadboat.com>; pgsql-hackers@postgresql.org, Thomas Munro <thomas.munro@gmail.com>; Heikki Linnakangas <hlinnaka@iki.fi>; Robert Haas <robertmhaas@gmail.com>; Jakub Wartak <jakub.wartak@enterprisedb.com>; Jelte Fennema-Nio <postgres@jeltef.nl>
Hi,
I've been going through the new AIO code as an effort to rebase and
adapt Neon to PG18. In doing so, I found the following
items/curiosities:
1. In aio/README.md, the following code snippet is found:
[...]
pgaio_io_set_handle_data_32(ioh, (uint32 *) buffer, 1);
[...]
I believe it would be clearer if it took a reference to the buffer:
pgaio_io_set_handle_data_32(ioh, (uint32 *) &buffer, 1);
The main reason here is that common practice is to have a `Buffer
buffer;` whereas a Buffer * is more commonly plural.
2. In aio.h, PgAioHandleCallbackID is checked to fit in
PGAIO_RESULT_ID_BITS (though the value of PGAIO_HCB_MAX).
However, the check is off by 1:
```
#define PGAIO_HCB_MAX PGAIO_HCB_LOCAL_BUFFER_READV
StaticAssertDecl(PGAIO_HCB_MAX <= (1 << PGAIO_RESULT_ID_BITS), [...])
```
This static assert will not trigger when PGAIO_HCB_MAX is equal to
2^PGAIO_RESULT_ID_BITS, but its value won't fit in those 6 bits and
instead overflow to 0.
To fix this, I suggest the `<=` is replaced with `<` to make that
work, or the definition of PGAIO_HCB_MAX to be updated to
PGAIO_HCB_LOCAL_BUFFER_READV + 1.
Note that this won't have caused bugs yet because we're not nearing
this limit of 64 callback IDs, but it's just a matter of good code
hygiene.
3. I noticed that there is AIO code for writev-related operations
(specifically, pgaio_io_start_writev is exposed, as is
PGAIO_OP_WRITEV), but no practical way to excercise that code: it's
not called from anywhere in the project, and there is no way for
extensions to register the relevant callbacks required to make writev
work well on buffered contents. Is that intentional?
Relatedly, the rationale for using enum PgAioHandleCallbackID rather
than function pointers in the documentation above its definition says
that function pointer instability across backends is one of the 2 main
reasons.
Is there any example of OS or linker behaviour that does not start
PostgreSQL with stable function pointer addresses across backends of
the same PostgreSQL binary? Or is this designed with extensibility
and/or cross-version EXEC_BACKEND in mind, but with extensibility not
yet been implemented due to $constraints?
I've attached a patch that solves the issues of (1) and (2).
Kind regards,
Matthias van de Meent
Databricks
Attachments:
[application/octet-stream] v1-0001-AIO-README-clarification-assertion-fix.patch (1.6K, ../../CAEze2Wj-43JV4YufW23gm=Uwr7Lkj+p0yKctKHxNm1rwFC+_DQ@mail.gmail.com/2-v1-0001-AIO-README-clarification-assertion-fix.patch)
download | inline diff:
From c289e6b430238140390fbbabeb4d4398d66b4774 Mon Sep 17 00:00:00 2001
From: Matthias van de Meent <boekewurm+postgres@gmail.com>
Date: Wed, 9 Jul 2025 13:16:03 +0200
Subject: [PATCH v1] AIO: README clarification; assertion fix
The assertion wouldn't have triggered for a long while yet, but this
won't accidentally fail to detect the issue if/when it occurs.
---
src/backend/storage/aio/README.md | 2 +-
src/include/storage/aio.h | 2 +-
2 files changed, 2 insertions(+), 2 deletions(-)
diff --git a/src/backend/storage/aio/README.md b/src/backend/storage/aio/README.md
index f10b5c7e31e..b59b20fc9c0 100644
--- a/src/backend/storage/aio/README.md
+++ b/src/backend/storage/aio/README.md
@@ -94,7 +94,7 @@ pgaio_io_register_callbacks(ioh, PGAIO_HCB_SHARED_BUFFER_READV, 0);
*
* In this example we're reading only a single buffer, hence the 1.
*/
-pgaio_io_set_handle_data_32(ioh, (uint32 *) buffer, 1);
+pgaio_io_set_handle_data_32(ioh, (uint32 *) &buffer, 1);
/*
* Pass the AIO handle to lower-level function. When operating on the level of
diff --git a/src/include/storage/aio.h b/src/include/storage/aio.h
index e7a0a234b6c..2933eea0649 100644
--- a/src/include/storage/aio.h
+++ b/src/include/storage/aio.h
@@ -201,7 +201,7 @@ typedef enum PgAioHandleCallbackID
} PgAioHandleCallbackID;
#define PGAIO_HCB_MAX PGAIO_HCB_LOCAL_BUFFER_READV
-StaticAssertDecl(PGAIO_HCB_MAX <= (1 << PGAIO_RESULT_ID_BITS),
+StaticAssertDecl(PGAIO_HCB_MAX < (1 << PGAIO_RESULT_ID_BITS),
"PGAIO_HCB_MAX is too big for PGAIO_RESULT_ID_BITS");
--
2.39.5 (Apple Git-154)
^ permalink raw reply [nested|flat] 153+ messages in thread
* Re: AIO v2.5
2025-03-04 19:00 AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-10 18:23 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-11 19:41 ` Re: AIO v2.5 Noah Misch <noah@leadboat.com>
2025-03-11 23:55 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-12 03:57 ` Re: AIO v2.5 Noah Misch <noah@leadboat.com>
2025-03-12 17:06 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-12 23:23 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-13 10:53 ` Re: AIO v2.5 Antonin Houska <ah@cybertec.at>
2025-03-14 19:43 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-06-30 06:58 ` Re: AIO v2.5 Antonin Houska <ah@cybertec.at>
2025-06-30 14:44 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-07-09 11:26 ` Re: AIO v2.5 Matthias van de Meent <boekewurm+postgres@gmail.com>
@ 2025-07-09 14:59 ` Andres Freund <andres@anarazel.de>
2025-07-10 19:00 ` Re: AIO v2.5 Matthias van de Meent <boekewurm+postgres@gmail.com>
0 siblings, 1 reply; 153+ messages in thread
From: Andres Freund @ 2025-07-09 14:59 UTC (permalink / raw)
To: Matthias van de Meent <boekewurm+postgres@gmail.com>; +Cc: Antonin Houska <ah@cybertec.at>; Noah Misch <noah@leadboat.com>; pgsql-hackers@postgresql.org, Thomas Munro <thomas.munro@gmail.com>; Heikki Linnakangas <hlinnaka@iki.fi>; Robert Haas <robertmhaas@gmail.com>; Jakub Wartak <jakub.wartak@enterprisedb.com>; Jelte Fennema-Nio <postgres@jeltef.nl>
Hi,
On 2025-07-09 13:26:09 +0200, Matthias van de Meent wrote:
> I've been going through the new AIO code as an effort to rebase and
> adapt Neon to PG18. In doing so, I found the following
> items/curiosities:
>
> 1. In aio/README.md, the following code snippet is found:
>
> [...]
> pgaio_io_set_handle_data_32(ioh, (uint32 *) buffer, 1);
> [...]
>
> I believe it would be clearer if it took a reference to the buffer:
>
> pgaio_io_set_handle_data_32(ioh, (uint32 *) &buffer, 1);
>
> The main reason here is that common practice is to have a `Buffer
> buffer;` whereas a Buffer * is more commonly plural.
It's also just simply wrong as-is :/. Interpreting the buffer id as a pointer
obviously makes no sense...
>
> 2. In aio.h, PgAioHandleCallbackID is checked to fit in
> PGAIO_RESULT_ID_BITS (though the value of PGAIO_HCB_MAX).
> However, the check is off by 1:
>
> ```
> #define PGAIO_HCB_MAX PGAIO_HCB_LOCAL_BUFFER_READV
> StaticAssertDecl(PGAIO_HCB_MAX <= (1 << PGAIO_RESULT_ID_BITS), [...])
> ```
>
> This static assert will not trigger when PGAIO_HCB_MAX is equal to
> 2^PGAIO_RESULT_ID_BITS, but its value won't fit in those 6 bits and
> instead overflow to 0.
> To fix this, I suggest the `<=` is replaced with `<` to make that
> work, or the definition of PGAIO_HCB_MAX to be updated to
> PGAIO_HCB_LOCAL_BUFFER_READV + 1.
Ooops.
> 3. I noticed that there is AIO code for writev-related operations
> (specifically, pgaio_io_start_writev is exposed, as is
> PGAIO_OP_WRITEV), but no practical way to excercise that code: it's
> not called from anywhere in the project, and there is no way for
> extensions to register the relevant callbacks required to make writev
> work well on buffered contents. Is that intentional?
Yes. We obviously do want to support writes eventually, and it didn't seem
useful to not have the most basic code for writes in the AIO infrastructure.
You could still use it to e.g. write out temporary file data or such.
> and there is no way for extensions to register the relevant callbacks
> required to make writev work well on buffered contents. Is that intentional?
FWIW, the problem with writev for buffered IO is not so much with the AIO
infrastructure, but with buffer locking and sync.c...
> Relatedly, the rationale for using enum PgAioHandleCallbackID rather
> than function pointers in the documentation above its definition says
> that function pointer instability across backends is one of the 2 main
> reasons.
> Is there any example of OS or linker behaviour that does not start
> PostgreSQL with stable function pointer addresses across backends of
> the same PostgreSQL binary? Or is this designed with extensibility
> and/or cross-version EXEC_BACKEND in mind, but with extensibility not
> yet been implemented due to $constraints?
It's due to EXEC_BACKEND - function pointers aren't stable with EXEC_BACKEND
even *without* cross-version issues. Due to ASLR function/data pointers may be
shifted around.
Even without EXEC_BACKEND we probably would want to have something like
PgAioHandleCallbackID, just to save space. But it'd be a lot easier to make it
extensible, because we could add entries once, rather than having to do so in
every process.
I suspect to make this extensible in the face of EXEC_BACKEND we'd want an
array of dynamically registered callbacks, in shared memory, that has the
library_name/symbol_name for the location of each callback struct, to be
registered in _PG_init(), which then would be entered into the backend-local
table in pgaio_init_backend().
Personally I'm not particularly interested in investing the energy to make
that work, I'd rather wait till we have threading and focus on the
infrastructure work for AIO writes etc. But then I don't work on an extension
these days, so I'm not directly affected by this not being extensible - I'm
not opposed to somebody else doing that work.
Greetings,
Andres Freund
^ permalink raw reply [nested|flat] 153+ messages in thread
* Re: AIO v2.5
2025-03-04 19:00 AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-10 18:23 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-11 19:41 ` Re: AIO v2.5 Noah Misch <noah@leadboat.com>
2025-03-11 23:55 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-12 03:57 ` Re: AIO v2.5 Noah Misch <noah@leadboat.com>
2025-03-12 17:06 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-12 23:23 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-13 10:53 ` Re: AIO v2.5 Antonin Houska <ah@cybertec.at>
2025-03-14 19:43 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-06-30 06:58 ` Re: AIO v2.5 Antonin Houska <ah@cybertec.at>
2025-06-30 14:44 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-07-09 11:26 ` Re: AIO v2.5 Matthias van de Meent <boekewurm+postgres@gmail.com>
2025-07-09 14:59 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
@ 2025-07-10 19:00 ` Matthias van de Meent <boekewurm+postgres@gmail.com>
2025-07-10 19:29 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-07-22 13:04 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
0 siblings, 2 replies; 153+ messages in thread
From: Matthias van de Meent @ 2025-07-10 19:00 UTC (permalink / raw)
To: Andres Freund <andres@anarazel.de>; +Cc: Antonin Houska <ah@cybertec.at>; Noah Misch <noah@leadboat.com>; pgsql-hackers@postgresql.org, Thomas Munro <thomas.munro@gmail.com>; Heikki Linnakangas <hlinnaka@iki.fi>; Robert Haas <robertmhaas@gmail.com>; Jakub Wartak <jakub.wartak@enterprisedb.com>; Jelte Fennema-Nio <postgres@jeltef.nl>
On Wed, 9 Jul 2025 at 16:59, Andres Freund <andres@anarazel.de> wrote:
>
> Hi,
>
> On 2025-07-09 13:26:09 +0200, Matthias van de Meent wrote:
> > I've been going through the new AIO code as an effort to rebase and
> > adapt Neon to PG18. In doing so, I found the following
> > items/curiosities:
> >
> > 1. In aio/README.md, the following code snippet is found:
> >
> > [...]
> > pgaio_io_set_handle_data_32(ioh, (uint32 *) buffer, 1);
> > [...]
> >
> > I believe it would be clearer if it took a reference to the buffer:
> >
> > pgaio_io_set_handle_data_32(ioh, (uint32 *) &buffer, 1);
> >
> > The main reason here is that common practice is to have a `Buffer
> > buffer;` whereas a Buffer * is more commonly plural.
>
> It's also just simply wrong as-is :/. Interpreting the buffer id as a pointer
> obviously makes no sense...
Given that the snippet didn't contain type indications for buffer upto
that point, technically the buffer variable could've been defined as
`Buffer* buffer;` which would've been type-correct. That would be very
confusing however, hence the suggested change.
After your mail, I also noticed the later snippet which should be updated, too:
```
-smgrstartreadv(ioh, operation->smgr, forknum, blkno,
- BufferGetBlock(buffer), 1);
+void *page = BufferGetBlock(buffer);
+smgrstartreadv(ioh, operation->smgr, forknum, blkno,
+ &page, 1);
```
> > 3. I noticed that there is AIO code for writev-related operations
> > (specifically, pgaio_io_start_writev is exposed, as is
> > PGAIO_OP_WRITEV), but no practical way to excercise that code: it's
> > not called from anywhere in the project, and there is no way for
> > extensions to register the relevant callbacks required to make writev
> > work well on buffered contents. Is that intentional?
>
> Yes. We obviously do want to support writes eventually, and it didn't seem
> useful to not have the most basic code for writes in the AIO infrastructure.
>
> You could still use it to e.g. write out temporary file data or such.
Yes, though IIUC that would require an implementation of at least
PgAioTargetInfo for such a use case (it's definitely not a SMGR
target), which currently isn't available and can't be registered
dynamically by an extension. Or maybe did I miss something?
> > and there is no way for extensions to register the relevant callbacks
> > required to make writev work well on buffered contents. Is that intentional?
>
> FWIW, the problem with writev for buffered IO is not so much with the AIO
> infrastructure, but with buffer locking and sync.c...
Maybe "buffered contents" wasn't the right phrasing, but my main point
remains - I can't seem to find a way to make use of the
pgaio_start_writev API in the 18 branch that doesn't require patching
core code or heavily relying on the behaviour of various pgaio
internals:
The only valid pgaio target is PGAIO_TID_SMGR, which effectively
forces you to use buffered relations as IO target (either shared, or
backend-local/temp). Then, there are no callbacks defined for writev,
so the user will effectively have to do setup and cleanup on their
own, making worker-based IO practically impossible, and requiring the
user to handle all lock acquisition/releases in their own process'
time instead of in the AIO paths - requiring significantly more
involved error handling.
(PS. I'm not quite 100% sure that it is impossible to use, just that
there are rather few handles available for using this part of the new
tool, and it seems completely untested in the PG18 branch)
-----
Something else I've just noticed is the use of int32 in
PgAIOHandle->result. In sync and worker mode, pg_preadv and pg_pwritev
return ssize_t, which most modern systems can't fit in int32 (the
output was int before, then size_t, then ssize_t: [0]). While not
directly an issue in default PG18 due to the use of 1GB relation
segments capping the max IO size for SMGR-managed IOs (and various
other code-level constraints), this may have more issues when an
extension starts bulk-reading data on a system compiled with
RELSEG_SIZE >= 2GB; I can't find any protective checks against
overflows in downcasting the IO result.
I did notice Linux seems to guarantee it won't read more than
0x7FFF_F000 in any one operation, but I can't find any similar
guarantees for e.g. MacOS or Windows (well, our win32 port limits
pg_pread/writev to 1GB, which seems sufficient for this exact case).
If we're keeping int32, then it would be great if the considerations
for why it's OK to use 32 bits to store the ssize_t vlaue would be
included in the docs/comments.
Kind regards,
Matthias van de Meent
Databricks
[0] https://postgr.es/m/flat/1672202.1703441340%40sss.pgh.pa.us
^ permalink raw reply [nested|flat] 153+ messages in thread
* Re: AIO v2.5
2025-03-04 19:00 AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-10 18:23 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-11 19:41 ` Re: AIO v2.5 Noah Misch <noah@leadboat.com>
2025-03-11 23:55 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-12 03:57 ` Re: AIO v2.5 Noah Misch <noah@leadboat.com>
2025-03-12 17:06 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-12 23:23 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-13 10:53 ` Re: AIO v2.5 Antonin Houska <ah@cybertec.at>
2025-03-14 19:43 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-06-30 06:58 ` Re: AIO v2.5 Antonin Houska <ah@cybertec.at>
2025-06-30 14:44 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-07-09 11:26 ` Re: AIO v2.5 Matthias van de Meent <boekewurm+postgres@gmail.com>
2025-07-09 14:59 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-07-10 19:00 ` Re: AIO v2.5 Matthias van de Meent <boekewurm+postgres@gmail.com>
@ 2025-07-10 19:29 ` Andres Freund <andres@anarazel.de>
1 sibling, 0 replies; 153+ messages in thread
From: Andres Freund @ 2025-07-10 19:29 UTC (permalink / raw)
To: Matthias van de Meent <boekewurm+postgres@gmail.com>; +Cc: Antonin Houska <ah@cybertec.at>; Noah Misch <noah@leadboat.com>; pgsql-hackers@postgresql.org, Thomas Munro <thomas.munro@gmail.com>; Heikki Linnakangas <hlinnaka@iki.fi>; Robert Haas <robertmhaas@gmail.com>; Jakub Wartak <jakub.wartak@enterprisedb.com>; Jelte Fennema-Nio <postgres@jeltef.nl>
Hi,
On 2025-07-10 21:00:21 +0200, Matthias van de Meent wrote:
> On Wed, 9 Jul 2025 at 16:59, Andres Freund <andres@anarazel.de> wrote:
> > > 3. I noticed that there is AIO code for writev-related operations
> > > (specifically, pgaio_io_start_writev is exposed, as is
> > > PGAIO_OP_WRITEV), but no practical way to excercise that code: it's
> > > not called from anywhere in the project, and there is no way for
> > > extensions to register the relevant callbacks required to make writev
> > > work well on buffered contents. Is that intentional?
> >
> > Yes. We obviously do want to support writes eventually, and it didn't seem
> > useful to not have the most basic code for writes in the AIO infrastructure.
> >
> > You could still use it to e.g. write out temporary file data or such.
>
> Yes, though IIUC that would require an implementation of at least
> PgAioTargetInfo for such a use case (it's definitely not a SMGR
> target), which currently isn't available and can't be registered
> dynamically by an extension. Or maybe did I miss something?
I can see some hacky ways around that, but they're just that, hacky...
> (PS. I'm not quite 100% sure that it is impossible to use, just that
> there are rather few handles available for using this part of the new
> tool, and it seems completely untested in the PG18 branch)
I'm not saying it's 100% ready to use without modifying core code, but for
something that's like 30 lines of code, as part of a considerably larger
subystem, I just don't see a problem with writev not yet being covered. It's
just incremental development.
> -----
>
> Something else I've just noticed is the use of int32 in
> PgAIOHandle->result. In sync and worker mode, pg_preadv and pg_pwritev
> return ssize_t, which most modern systems can't fit in int32 (the
> output was int before, then size_t, then ssize_t: [0]).
I don't think there's anything that can actually do IO that's large enough to
be problematic. What's the potential scenario where you'd want to read/write
more than 3GB of data within one syscall? That just doesn't seem to make
sense.
> While not directly an issue in default PG18 due to the use of 1GB relation
> segments capping the max IO size for SMGR-managed IOs (and various other
> code-level constraints), this may have more issues when an extension starts
> bulk-reading data on a system compiled with RELSEG_SIZE >= 2GB; I can't find
> any protective checks against overflows in downcasting the IO result.
I don't think the relation size is relevant piece here, it's just that it
doesn't make sense (and likely isn't possible) to read that much data at once.
Greetings,
Andres Freund
^ permalink raw reply [nested|flat] 153+ messages in thread
* Re: AIO v2.5
2025-03-04 19:00 AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-10 18:23 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-11 19:41 ` Re: AIO v2.5 Noah Misch <noah@leadboat.com>
2025-03-11 23:55 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-12 03:57 ` Re: AIO v2.5 Noah Misch <noah@leadboat.com>
2025-03-12 17:06 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-12 23:23 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-13 10:53 ` Re: AIO v2.5 Antonin Houska <ah@cybertec.at>
2025-03-14 19:43 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-06-30 06:58 ` Re: AIO v2.5 Antonin Houska <ah@cybertec.at>
2025-06-30 14:44 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-07-09 11:26 ` Re: AIO v2.5 Matthias van de Meent <boekewurm+postgres@gmail.com>
2025-07-09 14:59 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-07-10 19:00 ` Re: AIO v2.5 Matthias van de Meent <boekewurm+postgres@gmail.com>
@ 2025-07-22 13:04 ` Andres Freund <andres@anarazel.de>
1 sibling, 0 replies; 153+ messages in thread
From: Andres Freund @ 2025-07-22 13:04 UTC (permalink / raw)
To: Matthias van de Meent <boekewurm+postgres@gmail.com>; +Cc: Antonin Houska <ah@cybertec.at>; Noah Misch <noah@leadboat.com>; pgsql-hackers@postgresql.org, Thomas Munro <thomas.munro@gmail.com>; Heikki Linnakangas <hlinnaka@iki.fi>; Robert Haas <robertmhaas@gmail.com>; Jakub Wartak <jakub.wartak@enterprisedb.com>; Jelte Fennema-Nio <postgres@jeltef.nl>
Hi,
On 2025-07-10 21:00:21 +0200, Matthias van de Meent wrote:
> On Wed, 9 Jul 2025 at 16:59, Andres Freund <andres@anarazel.de> wrote:
> > On 2025-07-09 13:26:09 +0200, Matthias van de Meent wrote:
> > > I've been going through the new AIO code as an effort to rebase and
> > > adapt Neon to PG18. In doing so, I found the following
> > > items/curiosities:
> > >
> > > 1. In aio/README.md, the following code snippet is found:
> > >
> > > [...]
> > > pgaio_io_set_handle_data_32(ioh, (uint32 *) buffer, 1);
> > > [...]
> > >
> > > I believe it would be clearer if it took a reference to the buffer:
> > >
> > > pgaio_io_set_handle_data_32(ioh, (uint32 *) &buffer, 1);
> > >
> > > The main reason here is that common practice is to have a `Buffer
> > > buffer;` whereas a Buffer * is more commonly plural.
> >
> > It's also just simply wrong as-is :/. Interpreting the buffer id as a pointer
> > obviously makes no sense...
>
> Given that the snippet didn't contain type indications for buffer upto
> that point, technically the buffer variable could've been defined as
> `Buffer* buffer;` which would've been type-correct. That would be very
> confusing however, hence the suggested change.
>
> After your mail, I also noticed the later snippet which should be updated, too:
>
> ```
> -smgrstartreadv(ioh, operation->smgr, forknum, blkno,
> - BufferGetBlock(buffer), 1);
> +void *page = BufferGetBlock(buffer);
> +smgrstartreadv(ioh, operation->smgr, forknum, blkno,
> + &page, 1);
> ```
I now pushed your change, with that squashed in.
Thanks!
Andres
^ permalink raw reply [nested|flat] 153+ messages in thread
* Re: AIO v2.5
2025-03-04 19:00 AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-10 18:23 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-11 19:41 ` Re: AIO v2.5 Noah Misch <noah@leadboat.com>
2025-03-11 23:55 ` Re: AIO v2.5 Andres Freund <andres@anarazel.de>
@ 2025-03-12 16:10 ` Andres Freund <andres@anarazel.de>
1 sibling, 0 replies; 153+ messages in thread
From: Andres Freund @ 2025-03-12 16:10 UTC (permalink / raw)
To: Noah Misch <noah@leadboat.com>; +Cc: pgsql-hackers@postgresql.org, Thomas Munro <thomas.munro@gmail.com>; Heikki Linnakangas <hlinnaka@iki.fi>; Robert Haas <robertmhaas@gmail.com>; Jakub Wartak <jakub.wartak@enterprisedb.com>
Hi,
On 2025-03-11 19:55:35 -0400, Andres Freund wrote:
> On 2025-03-11 12:41:08 -0700, Noah Misch wrote:
> > On Mon, Sep 16, 2024 at 01:51:42PM -0400, Andres Freund wrote:
> > > On 2024-09-16 07:43:49 -0700, Noah Misch wrote:
> > > > For non-sync IO methods, I gather it's essential that a process other than the
> > > > IO definer be scanning for incomplete IOs and completing them.
> >
> > > > Otherwise, deadlocks like this would happen:
> > >
> > > > backend1 locks blk1 for non-IO reasons
> > > > backend2 locks blk2, starts AIO write
> > > > backend1 waits for lock on blk2 for non-IO reasons
> > > > backend2 waits for lock on blk1 for non-IO reasons
> > > >
> > > > If that's right, in worker mode, the IO worker resolves that deadlock. What
> > > > resolves it under io_uring? Another process that happens to do
> > > > pgaio_io_ref_wait() would dislodge things, but I didn't locate the code to
> > > > make that happen systematically.
> > >
> > > Yea, it's code that I haven't forward ported yet. I think basically
> > > LockBuffer[ForCleanup] ought to call pgaio_io_ref_wait() when it can't
> > > immediately acquire the lock and if the buffer has IO going on.
> >
> > I'm not finding that code in v2.6. What function has it?
>
> My local version now has it... Sorry, I was focusing on the earlier patches
> until now.
Looking more at my draft, I don't think it was race-free. I had a race-free
way of doing it in the v1 patch (by making lwlocks extensible, so the check
for IO could happen between enqueueing on the lwlock wait queue and sleeping
on the semaphore), but that obviously requires that infrastructure.
I want to focus on reads for now, so I'll add FIXMEs to the relevant places in
the patch to support AIO writes and focus on the rest of the patch for now.
Greetings,
Andres Freund
^ permalink raw reply [nested|flat] 153+ messages in thread
end of thread, other threads:[~2025-07-29 16:41 UTC | newest]
Thread overview: 153+ messages (download: mbox mbox.gz follow: Atom feed)
-- links below jump to the message on this page --
2025-03-04 19:00 AIO v2.5 Andres Freund <andres@anarazel.de>
2025-03-06 11:36 ` Jakub Wartak <jakub.wartak@enterprisedb.com>
2025-03-06 13:13 ` Andres Freund <andres@anarazel.de>
2025-03-07 10:21 ` Jakub Wartak <jakub.wartak@enterprisedb.com>
2025-03-07 13:54 ` Andres Freund <andres@anarazel.de>
2025-03-06 15:33 ` Robert Haas <robertmhaas@gmail.com>
2025-03-06 16:53 ` Andres Freund <andres@anarazel.de>
2025-03-07 21:23 ` Andres Freund <andres@anarazel.de>
2025-03-08 02:11 ` Andres Freund <andres@anarazel.de>
2025-03-09 17:51 ` Tom Lane <tgl@sss.pgh.pa.us>
2025-03-10 18:23 ` Andres Freund <andres@anarazel.de>
2025-03-11 15:31 ` Melanie Plageman <melanieplageman@gmail.com>
2025-03-11 17:56 ` Andres Freund <andres@anarazel.de>
2025-03-11 18:28 ` Melanie Plageman <melanieplageman@gmail.com>
2025-03-11 19:41 ` Noah Misch <noah@leadboat.com>
2025-03-11 23:55 ` Andres Freund <andres@anarazel.de>
2025-03-12 03:57 ` Noah Misch <noah@leadboat.com>
2025-03-12 17:06 ` Andres Freund <andres@anarazel.de>
2025-03-12 23:23 ` Andres Freund <andres@anarazel.de>
2025-03-13 10:53 ` Antonin Houska <ah@cybertec.at>
2025-03-14 19:43 ` Andres Freund <andres@anarazel.de>
2025-03-14 19:58 ` Andres Freund <andres@anarazel.de>
2025-03-17 20:00 ` Andres Freund <andres@anarazel.de>
2025-03-18 20:12 ` Andres Freund <andres@anarazel.de>
2025-03-19 01:00 ` Melanie Plageman <melanieplageman@gmail.com>
2025-03-19 01:50 ` Andres Freund <andres@anarazel.de>
2025-03-19 14:47 ` Melanie Plageman <melanieplageman@gmail.com>
2025-03-19 17:20 ` Melanie Plageman <melanieplageman@gmail.com>
2025-03-19 19:28 ` Andres Freund <andres@anarazel.de>
2025-03-19 21:25 ` Noah Misch <noah@leadboat.com>
2025-03-19 22:17 ` Andres Freund <andres@anarazel.de>
2025-03-20 01:11 ` Noah Misch <noah@leadboat.com>
2025-03-20 18:54 ` Andres Freund <andres@anarazel.de>
2025-03-20 19:52 ` Noah Misch <noah@leadboat.com>
2025-03-20 17:05 ` Andres Freund <andres@anarazel.de>
2025-03-20 18:19 ` Noah Misch <noah@leadboat.com>
2025-03-20 08:39 ` Jakub Wartak <jakub.wartak@enterprisedb.com>
2025-03-21 01:58 ` Andres Freund <andres@anarazel.de>
2025-03-23 00:20 ` Noah Misch <noah@leadboat.com>
2025-03-23 15:11 ` Andres Freund <andres@anarazel.de>
2025-03-23 15:55 ` Noah Misch <noah@leadboat.com>
2025-03-23 16:59 ` Andres Freund <andres@anarazel.de>
2025-03-24 01:11 ` Thomas Munro <thomas.munro@gmail.com>
2025-03-23 02:09 ` Noah Misch <noah@leadboat.com>
2025-03-23 15:57 ` Andres Freund <andres@anarazel.de>
2025-03-23 16:32 ` Noah Misch <noah@leadboat.com>
2025-03-25 01:40 ` Andres Freund <andres@anarazel.de>
2025-03-24 00:29 ` Noah Misch <noah@leadboat.com>
2025-03-24 15:43 ` Andres Freund <andres@anarazel.de>
2025-03-24 22:55 ` Andres Freund <andres@anarazel.de>
2025-03-25 00:07 ` Thomas Munro <thomas.munro@gmail.com>
2025-03-25 00:18 ` Andres Freund <andres@anarazel.de>
2025-03-25 00:45 ` Noah Misch <noah@leadboat.com>
2025-03-25 02:30 ` Andres Freund <andres@anarazel.de>
2025-03-25 13:33 ` Noah Misch <noah@leadboat.com>
2025-03-25 15:26 ` Andres Freund <andres@anarazel.de>
2025-03-25 15:58 ` Noah Misch <noah@leadboat.com>
2025-03-25 18:58 ` Andres Freund <andres@anarazel.de>
2025-03-25 19:39 ` Noah Misch <noah@leadboat.com>
2025-03-25 20:07 ` Andres Freund <andres@anarazel.de>
2025-03-25 20:18 ` Noah Misch <noah@leadboat.com>
2025-03-25 20:56 ` Andres Freund <andres@anarazel.de>
2025-03-25 22:44 ` Noah Misch <noah@leadboat.com>
2025-03-25 01:18 ` Andres Freund <andres@anarazel.de>
2025-03-25 04:10 ` Thomas Munro <thomas.munro@gmail.com>
2025-03-25 15:33 ` Andres Freund <andres@anarazel.de>
2025-03-26 00:19 ` Noah Misch <noah@leadboat.com>
2025-03-26 20:33 ` Andres Freund <andres@anarazel.de>
2025-03-26 21:21 ` Noah Misch <noah@leadboat.com>
2025-03-26 18:31 ` Noah Misch <noah@leadboat.com>
2025-03-26 21:41 ` Andres Freund <andres@anarazel.de>
2025-03-26 21:52 ` Thomas Munro <thomas.munro@gmail.com>
2025-03-30 23:13 ` Andres Freund <andres@anarazel.de>
2025-03-26 21:20 ` Thom Brown <thom@linux.com>
2025-03-26 21:42 ` Andres Freund <andres@anarazel.de>
2025-03-27 01:07 ` Andres Freund <andres@anarazel.de>
2025-03-27 20:58 ` Andres Freund <andres@anarazel.de>
2025-03-28 03:22 ` Noah Misch <noah@leadboat.com>
2025-03-28 12:54 ` Andres Freund <andres@anarazel.de>
2025-03-28 12:57 ` Andres Freund <andres@anarazel.de>
2025-03-29 03:35 ` Andres Freund <andres@anarazel.de>
2025-03-29 13:41 ` Noah Misch <noah@leadboat.com>
2025-03-29 14:48 ` Andres Freund <andres@anarazel.de>
2025-03-29 18:25 ` Andres Freund <andres@anarazel.de>
2025-03-29 20:16 ` Melanie Plageman <melanieplageman@gmail.com>
2025-03-29 21:29 ` Noah Misch <noah@leadboat.com>
2025-03-30 00:39 ` Andres Freund <andres@anarazel.de>
2025-03-30 02:45 ` Noah Misch <noah@leadboat.com>
2025-03-30 23:46 ` Andres Freund <andres@anarazel.de>
2025-04-01 00:41 ` Andres Freund <andres@anarazel.de>
2025-04-01 11:56 ` Aleksander Alekseev <aleksander@timescale.com>
2025-04-01 13:45 ` Andres Freund <andres@anarazel.de>
2025-04-01 15:11 ` Noah Misch <noah@leadboat.com>
2025-04-01 15:55 ` Andres Freund <andres@anarazel.de>
2025-04-01 16:07 ` Noah Misch <noah@leadboat.com>
2025-04-01 16:51 ` Andres Freund <andres@anarazel.de>
2025-04-01 21:47 ` Andres Freund <andres@anarazel.de>
2025-04-01 22:25 ` Andres Freund <andres@anarazel.de>
2025-04-02 00:13 ` Noah Misch <noah@leadboat.com>
2025-04-02 11:58 ` Andres Freund <andres@anarazel.de>
2025-04-03 16:46 ` Ranier Vilela <ranier.vf@gmail.com>
2025-04-03 18:35 ` Andres Freund <andres@anarazel.de>
2025-04-03 19:16 ` Ranier Vilela <ranier.vf@gmail.com>
2025-04-03 22:30 ` Andres Freund <andres@anarazel.de>
2025-04-06 20:00 ` Alexander Lakhin <exclusion@gmail.com>
2025-04-07 16:20 ` Andres Freund <andres@anarazel.de>
2025-04-07 19:10 ` Alexander Lakhin <exclusion@gmail.com>
2025-04-13 06:00 ` Alexander Lakhin <exclusion@gmail.com>
2025-04-14 16:06 ` Andres Freund <andres@anarazel.de>
2025-04-15 18:00 ` Alexander Lakhin <exclusion@gmail.com>
2025-04-25 16:44 ` Andres Freund <andres@anarazel.de>
2025-04-30 20:00 ` Andres Freund <andres@anarazel.de>
2025-05-03 03:05 ` Noah Misch <noah@leadboat.com>
2025-05-09 01:06 ` Andres Freund <andres@anarazel.de>
2025-05-09 02:22 ` Noah Misch <noah@leadboat.com>
2025-05-20 01:09 ` Andres Freund <andres@anarazel.de>
2025-04-03 18:19 ` Andres Freund <andres@anarazel.de>
2025-04-03 19:40 ` Noah Misch <noah@leadboat.com>
2025-04-04 19:16 ` Andres Freund <andres@anarazel.de>
2025-04-04 21:18 ` Noah Misch <noah@leadboat.com>
2025-04-05 03:53 ` Andres Freund <andres@anarazel.de>
2025-04-05 13:43 ` Noah Misch <noah@leadboat.com>
2025-04-07 19:24 ` Andres Freund <andres@anarazel.de>
2025-04-20 15:00 ` Alexander Lakhin <exclusion@gmail.com>
2025-04-24 00:40 ` Andres Freund <andres@anarazel.de>
2025-04-24 04:04 ` Alexander Lakhin <exclusion@gmail.com>
2025-03-25 02:20 ` Noah Misch <noah@leadboat.com>
2025-03-25 02:52 ` Andres Freund <andres@anarazel.de>
2025-03-25 14:11 ` Noah Misch <noah@leadboat.com>
2025-03-25 15:57 ` Andres Freund <andres@anarazel.de>
2025-03-25 16:15 ` Noah Misch <noah@leadboat.com>
2025-03-26 00:17 ` Andres Freund <andres@anarazel.de>
2025-03-26 00:43 ` Noah Misch <noah@leadboat.com>
2025-03-30 18:23 ` Melanie Plageman <melanieplageman@gmail.com>
2025-04-22 22:38 ` Tomas Vondra <tomas@vondra.me>
2025-05-09 01:15 ` Andres Freund <andres@anarazel.de>
2025-07-11 21:03 ` Tomas Vondra <tomas@vondra.me>
2025-07-13 18:04 ` Tomas Vondra <tomas@vondra.me>
2025-07-14 18:44 ` Andres Freund <andres@anarazel.de>
2025-07-15 00:13 ` Tomas Vondra <tomas@vondra.me>
2025-07-14 18:36 ` Andres Freund <andres@anarazel.de>
2025-07-15 00:30 ` Tomas Vondra <tomas@vondra.me>
2025-07-22 13:51 ` Tomas Vondra <tomas@vondra.me>
2025-07-29 16:41 ` Nathan Bossart <nathandbossart@gmail.com>
2025-03-17 20:53 ` Melanie Plageman <melanieplageman@gmail.com>
2025-06-30 06:58 ` Antonin Houska <ah@cybertec.at>
2025-06-30 14:44 ` Andres Freund <andres@anarazel.de>
2025-07-09 11:26 ` Matthias van de Meent <boekewurm+postgres@gmail.com>
2025-07-09 14:59 ` Andres Freund <andres@anarazel.de>
2025-07-10 19:00 ` Matthias van de Meent <boekewurm+postgres@gmail.com>
2025-07-10 19:29 ` Andres Freund <andres@anarazel.de>
2025-07-22 13:04 ` Andres Freund <andres@anarazel.de>
2025-03-12 16:10 ` Andres Freund <andres@anarazel.de>
This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox