public inbox for [email protected]
help / color / mirror / Atom feedFrom: Heikki Linnakangas <[email protected]>
To: Robert Haas <[email protected]>
Cc: Ashutosh Bapat <[email protected]>
Cc: Andres Freund <[email protected]>
Cc: pgsql-hackers <[email protected]>
Cc: [email protected]
Subject: Re: Better shared data structure management and resizable shared data structures
Date: Thu, 19 Mar 2026 12:31:10 +0200
Message-ID: <[email protected]> (raw)
In-Reply-To: <CA+TgmobqwCHp1w6xkTJmZiq4C8Pr0O3tNUvSFZcAgYLkvF45bg@mail.gmail.com>
References: <CAExHW5vM1bneLYfg0wGeAa=52UiJ3z4vKd3AJ72X8Fw6k3KKrg@mail.gmail.com>
<[email protected]>
<CAExHW5s9Vp+-vJi020UJ+otyccBBo7eT1g6bttdRKL6HAvscyQ@mail.gmail.com>
<mlsruptoxgm2nqtdfyfsowjklzxl5zltsjb3y5bmywtigm474l@5tsonk4t3kia>
<CAExHW5uEK+eeG7e2g6uWh7POrFpfp+dqfaa=_3miMN17zgeaJw@mail.gmail.com>
<CAExHW5vz+PUHHUuzGRwtyx-mPLQk3nCZXxrFqnruRadEFrO5Xg@mail.gmail.com>
<CAExHW5so6VSxBC-1V=35229Z1+dw5vhw8HxHg9ry7UzceKcXzA@mail.gmail.com>
<[email protected]>
<CA+TgmoaoSha9KnSdrA7DDMYD9+Uor4OPEaQ88bbEC3QuVE5Mcg@mail.gmail.com>
<[email protected]>
<CA+TgmoboxAaS6MCRf__LRhf8Ji3B0O9_=d7KzRS12QhOuocnTQ@mail.gmail.com>
<[email protected]>
<CA+Tgmob7E3CGDoEBB3mwfk+v1CgcOG8yL0VHje2gy9Ts=e1Zxw@mail.gmail.com>
<[email protected]>
<CA+TgmobqwCHp1w6xkTJmZiq4C8Pr0O3tNUvSFZcAgYLkvF45bg@mail.gmail.com>
On 18/03/2026 21:30, Robert Haas wrote:
> On Fri, Mar 13, 2026 at 7:41 AM Heikki Linnakangas <[email protected]> wrote:
>> Yeah. IMHO the existing shmem_request/startup_hook mechanism is pretty
>> awkward too, and in most cases, the new mechanism is more convenient. It
>> might be slightly less convenient for some things, but mostly it's
>> better. Would you agree with that, or do you actually like the old hooks
>> and ShmemInitStruct() better?
>
> I'd say it's not massively different one way or the other. Looking at
> the pg_stat_statements changes in particular, I feel like in v2, it
> was actually worse, because you didn't get manage to get rid of
> pgss_shmem_request(), but it was no longer the thing requesting shmem.
> v3 is better in that regard: you get rid of some complexity in
> exchange for what you add. It's not amazing, though:
> pgss_shmem_startup() now has nothing to do with the name of the hook,
> which is not this patch's fault but also isn't solved by it. I wonder
> why in the world somebody decided to jam all of this non-shmem-related
> logic into this function, and why they didn't add a comment explaining
> why it was here and not someplace else. One of the worst things about
> this area is that I often end up having to trace through a bunch of
> postmaster startup logic to figure out whether any given bit of code
> is actually in the right place, and that makes me feel like the hooks
> are badly designed. pgss_shmem_startup() is a good example of that,
> and the fact that it needs an IsUnderPostmaster check is another.
Hmm, I assumed it was important that the pg_stat_statements file is
loaded later in the startup sequence, and that's why it was in
pgss_shmem_startup(). But now that I look at it, I don't think there was
any grand plan, shmem_startup_hook was just the only easy way to get
control during postmaster startup, after shmem initialization.
So I think we can move that code to the init_fn callback with the new
API, and that gets rid of shmem_startup_hook in pg_stat_statements. See
attached (v6-0005-move-pgss-shmem_startup-hook-code-into-the-new-in.patch).
> We should somehow try to make it clear what happens with this new
> mechanism if a module is loaded via shared_preload_libraries vs. if
> it's loaded via LOAD or session_preload_libraries or whatever. Writing
> modules that don't require shared_preload_libraries is a boon to
> users, because they can be added to a production system without a
> server restart. I wonder whether this new facility handles that case
> better, worse, or the same as existing facilities.
Pretty much the same I think. Point taken that it could be documented
better.
The old documentation for ShmemInitStruct() assumed that it would be
used from shared_preload_libraries, it didn't. With the new API, I tried
to document how it behaves when used outside shared_preload_libraries,
but still focused on how using it from shared_preload_libraries. I'm not
sure it helped. Perhaps the after-startup behavior should be put in a
separate section.
>> One such wrinkle with ShmemRegisterStruct() in the patch now is that
>> it's harder to do initialization that touches multiple structs or hash
>> tables. Currently the callbacks are called in the same order that the
>> structs are registered, so you can do all the initialization in the last
>> struct's callback. The single pair of shmem_request/startup_hooks per
>> module was more clear in that aspect. Fortunately, that kind of
>> cross-struct dependencies are pretty rare. So I think it's fine. (The
>> order that the callbacks are called needs be documented explicitly though).
>>
>> If we want to improve on that, one idea would be to introduce a
>> ShmemRegisterCallbacks() function to register callbacks that are not
>> tied to any particular struct and are called after all the per-struct
>> callbacks.
>
> I think the whole idea of ShmemInitHash() and ShmemInitStruct() is
> relatively poorly designed in this regard. Ideally, I want to
> initialize all the shared memory that belongs to me, and that might
> include arbitrary data structures, but the current design kind of
> thinks that I want one struct or one hash table and nothing else. If
> we're redesigning this mechanism, it would sure be nice to improve on
> that.
>
> Looking just at the pg_stat_statements changes, I think my overall
> view on this right now is that it's not terrible, but I'm also not
> that happy about introducing yet another way to do it for this amount
> of gain. To me, it doesn't yet rise to the level of a clear win.
So here's another idea (not yet implemented in the attached patch):
instead of thinking in terms of individual shmem structs and hashes,
let's introduce a concept of a "subsystem" that ties them together:
static pgss_subsystem_desc = {
.name = "pg_stat_statements",
.shmem_structs = {
{
.ptr = (void **) &pgss,
.size = sizeof(pgssSharedState),
},
},
.shmem_hashes = {
{
.ptr = &pgss_hash,
.init_size = 0, /* set from 'pgss_max' */
.max_size = 0, /* set from 'pgss_max' */
.hash_info.keysize = sizeof(pgssHashKey),
.hash_info.entrysize = sizeof(pgssEntry),
.hash_flags = HASH_ELEM | HASH_BLOBS,
},
},
/* called after the shmem structs and hashes have been allocated */
.init_fn = pgss_shmem_init,
}
void
_PG_init(void)
{
...
RegisterSubsystem(&pgss_subsystem_desc);
}
We could add more callbacks that get called at different times. For
example the callback that would get called before shared memory is
allocated, which could adjust the size according to MaxBackends. That
would fully replace shmem_request_hook. Or a callback that would get
called later in the startup sequence, if we wanted to e.g. load the
pg_stat_statements file later in startup.
This would be a natural place for other resources in future too. We
could support declaring "named lwlock tranches" here to replace
RequestNamedLWLockTranche() for example, although I think it's still
better to encourage embedding the LWLock in the struct instead.
_PG_init in pg_stat_statements still does a lot more than register that
struct. It declares the GUCs and installs other hooks for example. We
could perhaps move those to the subsystem descriptor too, although I'm
not sure if that's worth the code churn.
- Heikki
Attachments:
[text/x-patch] v6-0001-Test-pg_stat_statements-across-crash-restart.patch (1.4K, 2-v6-0001-Test-pg_stat_statements-across-crash-restart.patch)
download | inline diff:
From 54e16964f453b9e572e1ed6496ab05f2fceb75a3 Mon Sep 17 00:00:00 2001
From: Heikki Linnakangas <[email protected]>
Date: Mon, 16 Mar 2026 16:41:17 +0200
Subject: [PATCH v6 1/6] Test pg_stat_statements across crash restart
Add 'pg_stat_statements' to the crash restart test, to test that
shared memory and LWLock initialization works across crash restart in
a library listed in shared_preload_libraries.
Discussion: https://www.postgresql.org/message-id/CAExHW5vM1bneLYfg0wGeAa=52UiJ3z4vKd3AJ72X8Fw6k3KKrg@mail.gmail.com
---
src/test/recovery/t/013_crash_restart.pl | 9 +++++++++
1 file changed, 9 insertions(+)
diff --git a/src/test/recovery/t/013_crash_restart.pl b/src/test/recovery/t/013_crash_restart.pl
index 20d648ad6af..960e5462b49 100644
--- a/src/test/recovery/t/013_crash_restart.pl
+++ b/src/test/recovery/t/013_crash_restart.pl
@@ -21,6 +21,15 @@ my $psql_timeout = IPC::Run::timer($PostgreSQL::Test::Utils::timeout_default);
my $node = PostgreSQL::Test::Cluster->new('primary');
$node->init(allows_streaming => 1);
+
+# Enable pg_stat_statements to test restart of shared_preload_libraries.
+$node->append_conf(
+ 'postgresql.conf',
+ qq{shared_preload_libraries = 'pg_stat_statements'
+pg_stat_statements.max = 50000
+compute_query_id = 'regress'
+});
+
$node->start();
# by default PostgreSQL::Test::Cluster doesn't restart after a crash
--
2.47.3
[text/x-patch] v6-0002-Refactor-ShmemIndex-initialization.patch (7.0K, 3-v6-0002-Refactor-ShmemIndex-initialization.patch)
download | inline diff:
From a5dd2a2c09b28c25c3679b27cc0ba5af31f00c45 Mon Sep 17 00:00:00 2001
From: Heikki Linnakangas <[email protected]>
Date: Mon, 16 Mar 2026 20:35:36 +0200
Subject: [PATCH v6 2/6] Refactor ShmemIndex initialization
Initialize the ShmemIndex hash table in InitShmemAllocator() already,
removing the need for the separate InitShmemIndex() step.
---
src/backend/storage/ipc/ipci.c | 8 +-
src/backend/storage/ipc/shmem.c | 131 ++++++++++++--------------------
src/include/storage/shmem.h | 1 -
3 files changed, 49 insertions(+), 91 deletions(-)
diff --git a/src/backend/storage/ipc/ipci.c b/src/backend/storage/ipc/ipci.c
index a4785daf1e5..3d3f153809b 100644
--- a/src/backend/storage/ipc/ipci.c
+++ b/src/backend/storage/ipc/ipci.c
@@ -250,16 +250,10 @@ static void
CreateOrAttachShmemStructs(void)
{
/*
- * Now initialize LWLocks, which do shared memory allocation and are
- * needed for InitShmemIndex.
+ * Now initialize LWLocks, which do shared memory allocation.
*/
CreateLWLocks();
- /*
- * Set up shmem.c index hashtable
- */
- InitShmemIndex();
-
dsm_shmem_init();
DSMRegistryShmemInit();
diff --git a/src/backend/storage/ipc/shmem.c b/src/backend/storage/ipc/shmem.c
index 0424c445723..dce355e6683 100644
--- a/src/backend/storage/ipc/shmem.c
+++ b/src/backend/storage/ipc/shmem.c
@@ -124,6 +124,12 @@ Datum pg_numa_available(PG_FUNCTION_ARGS);
void
InitShmemAllocator(PGShmemHeader *seghdr)
{
+ Size offset;
+ int64 hash_size;
+ HASHCTL info;
+ int hash_flags;
+ size_t size;
+
Assert(seghdr != NULL);
/*
@@ -137,41 +143,54 @@ InitShmemAllocator(PGShmemHeader *seghdr)
ShmemBase = seghdr;
ShmemEnd = (char *) ShmemBase + seghdr->totalsize;
+ /*
+ * Allocations after this point should go through ShmemAlloc, which
+ * expects to allocate everything on cache line boundaries. Make sure the
+ * first allocation begins on a cache line boundary.
+ */
+ offset = CACHELINEALIGN(seghdr->content_offset + sizeof(ShmemAllocatorData));
+ if (offset > seghdr->totalsize)
+ ereport(ERROR,
+ (errcode(ERRCODE_OUT_OF_MEMORY),
+ errmsg("out of shared memory (%zu bytes requested)",
+ offset)));
+
+ ShmemAllocator = (ShmemAllocatorData *) ((char *) seghdr + seghdr->content_offset);
+ ShmemLock = &ShmemAllocator->shmem_lock;
+
#ifndef EXEC_BACKEND
Assert(!IsUnderPostmaster);
#endif
- if (IsUnderPostmaster)
+ if (!IsUnderPostmaster)
{
- PGShmemHeader *shmhdr = ShmemSegHdr;
-
- ShmemAllocator = (ShmemAllocatorData *) ((char *) shmhdr + shmhdr->content_offset);
- ShmemLock = &ShmemAllocator->shmem_lock;
+ SpinLockInit(&ShmemAllocator->shmem_lock);
+ ShmemAllocator->free_offset = offset;
}
- else
- {
- Size offset;
- /*
- * Allocations after this point should go through ShmemAlloc, which
- * expects to allocate everything on cache line boundaries. Make sure
- * the first allocation begins on a cache line boundary.
- */
- offset = CACHELINEALIGN(seghdr->content_offset + sizeof(ShmemAllocatorData));
- if (offset > seghdr->totalsize)
- ereport(ERROR,
- (errcode(ERRCODE_OUT_OF_MEMORY),
- errmsg("out of shared memory (%zu bytes requested)",
- offset)));
-
- ShmemAllocator = (ShmemAllocatorData *) ((char *) seghdr + seghdr->content_offset);
+ /*
+ * Create (or attach to) the shared memory index of shmem areas.
+ *
+ * This is the same initialization as ShmemInitHash() does, but we cannot
+ * use ShmemInitHash() here because it relies on ShmemIndex being already
+ * initialized.
+ */
+ hash_size = SHMEM_INDEX_SIZE;
- SpinLockInit(&ShmemAllocator->shmem_lock);
- ShmemLock = &ShmemAllocator->shmem_lock;
- ShmemAllocator->free_offset = offset;
- /* ShmemIndex can't be set up yet (need LWLocks first) */
- ShmemAllocator->index = NULL;
- ShmemIndex = (HTAB *) NULL;
+ info.keysize = SHMEM_INDEX_KEYSIZE;
+ info.entrysize = sizeof(ShmemIndexEnt);
+ info.dsize = info.max_dsize = hash_select_dirsize(hash_size);
+ info.alloc = ShmemAllocNoError;
+ hash_flags = HASH_ELEM | HASH_STRINGS | HASH_SHARED_MEM | HASH_ALLOC | HASH_DIRSIZE;
+ if (!IsUnderPostmaster)
+ {
+ size = hash_get_shared_size(&info, hash_flags);
+ ShmemAllocator->index = (HASHHDR *) ShmemAlloc(size);
}
+ else
+ hash_flags |= HASH_ATTACH;
+ info.hctl = ShmemAllocator->index;
+ ShmemIndex = hash_create("ShmemIndex", hash_size, &info, hash_flags);
+ Assert(ShmemIndex != NULL);
}
/*
@@ -270,31 +289,6 @@ ShmemAddrIsValid(const void *addr)
return (addr >= ShmemBase) && (addr < ShmemEnd);
}
-/*
- * InitShmemIndex() --- set up or attach to shmem index table.
- */
-void
-InitShmemIndex(void)
-{
- HASHCTL info;
-
- /*
- * Create the shared memory shmem index.
- *
- * Since ShmemInitHash calls ShmemInitStruct, which expects the ShmemIndex
- * hashtable to exist already, we have a bit of a circularity problem in
- * initializing the ShmemIndex itself. The special "ShmemIndex" hash
- * table name will tell ShmemInitStruct to fake it.
- */
- info.keysize = SHMEM_INDEX_KEYSIZE;
- info.entrysize = sizeof(ShmemIndexEnt);
-
- ShmemIndex = ShmemInitHash("ShmemIndex",
- SHMEM_INDEX_SIZE, SHMEM_INDEX_SIZE,
- &info,
- HASH_ELEM | HASH_STRINGS);
-}
-
/*
* ShmemInitHash -- Create and initialize, or attach to, a
* shared memory hash table.
@@ -383,38 +377,9 @@ ShmemInitStruct(const char *name, Size size, bool *foundPtr)
ShmemIndexEnt *result;
void *structPtr;
- LWLockAcquire(ShmemIndexLock, LW_EXCLUSIVE);
+ Assert(ShmemIndex != NULL);
- if (!ShmemIndex)
- {
- /* Must be trying to create/attach to ShmemIndex itself */
- Assert(strcmp(name, "ShmemIndex") == 0);
-
- if (IsUnderPostmaster)
- {
- /* Must be initializing a (non-standalone) backend */
- Assert(ShmemAllocator->index != NULL);
- structPtr = ShmemAllocator->index;
- *foundPtr = true;
- }
- else
- {
- /*
- * If the shmem index doesn't exist, we are bootstrapping: we must
- * be trying to init the shmem index itself.
- *
- * Notice that the ShmemIndexLock is released before the shmem
- * index has been initialized. This should be OK because no other
- * process can be accessing shared memory yet.
- */
- Assert(ShmemAllocator->index == NULL);
- structPtr = ShmemAlloc(size);
- ShmemAllocator->index = structPtr;
- *foundPtr = false;
- }
- LWLockRelease(ShmemIndexLock);
- return structPtr;
- }
+ LWLockAcquire(ShmemIndexLock, LW_EXCLUSIVE);
/* look it up in the shmem index */
result = (ShmemIndexEnt *)
diff --git a/src/include/storage/shmem.h b/src/include/storage/shmem.h
index 89d45287c17..0de8a36429b 100644
--- a/src/include/storage/shmem.h
+++ b/src/include/storage/shmem.h
@@ -33,7 +33,6 @@ extern void InitShmemAllocator(PGShmemHeader *seghdr);
extern void *ShmemAlloc(Size size);
extern void *ShmemAllocNoError(Size size);
extern bool ShmemAddrIsValid(const void *addr);
-extern void InitShmemIndex(void);
extern HTAB *ShmemInitHash(const char *name, int64 init_size, int64 max_size,
HASHCTL *infoP, int hash_flags);
extern void *ShmemInitStruct(const char *name, Size size, bool *foundPtr);
--
2.47.3
[text/x-patch] v6-0003-Introduce-a-new-mechanism-for-registering-shared-.patch (52.1K, 4-v6-0003-Introduce-a-new-mechanism-for-registering-shared-.patch)
download | inline diff:
From 0238a32ff259b9e76235995e27ce6c2d08f8feee Mon Sep 17 00:00:00 2001
From: Heikki Linnakangas <[email protected]>
Date: Mon, 16 Mar 2026 20:08:25 +0200
Subject: [PATCH v6 3/6] Introduce a new mechanism for registering shared
memory areas
Each shared memory area is registered with a "descriptor struct" that
contains parameters like name and size of the area. The descriptor
struct makes it easier to add optional fields in the future; the
additional fields can just be left as zeros.
This merges the separate [Subsystem]ShmemSize() and
[Subsystem]ShmemInit() phases at postmaster startup. Each subsystem is
now called into just once, before the shared memory segment has been
allocated, to register the subsystem's shared memory needs. The
registration includes the size, which replaces the
[Subsystem]ShmemSize() calls, and a pointer to an initialization
callback function, which replaces the [Subsystem]ShmemInit()
calls. This is more ergonomic, as you only need to calculate the size
once, when you register the struct.
This replaces ShmemInitStruct() and ShmemInitHash(), which become just
backwards-compatibility wrappers around the new functions. In future
commits, I plan to replace all ShmemInitStruct() and ShmemInitHash()
calls with the new functions, although we'll still need to keep them
around for extensions.
Reviewed-by: Ashutosh Bapat <[email protected]>
Reviewed-by: Zsolt Parragi <[email protected]>
Reviewed-by: Robert Haas <[email protected]>
Discussion: https://www.postgresql.org/message-id/CAExHW5vM1bneLYfg0wGeAa=52UiJ3z4vKd3AJ72X8Fw6k3KKrg@mail.gmail.com
---
doc/src/sgml/system-views.sgml | 4 +-
doc/src/sgml/xfunc.sgml | 126 ++--
src/backend/bootstrap/bootstrap.c | 2 +
src/backend/postmaster/launch_backend.c | 4 +
src/backend/postmaster/postmaster.c | 11 +-
src/backend/storage/ipc/ipci.c | 50 +-
src/backend/storage/ipc/shmem.c | 735 +++++++++++++++++++-----
src/backend/tcop/postgres.c | 3 +
src/include/storage/ipc.h | 1 +
src/include/storage/shmem.h | 153 ++++-
src/tools/pgindent/typedefs.list | 4 +-
11 files changed, 863 insertions(+), 230 deletions(-)
diff --git a/doc/src/sgml/system-views.sgml b/doc/src/sgml/system-views.sgml
index 9ee1a2bfc6a..ecdd5fa544a 100644
--- a/doc/src/sgml/system-views.sgml
+++ b/doc/src/sgml/system-views.sgml
@@ -4254,8 +4254,8 @@ SELECT * FROM pg_locks pl LEFT JOIN pg_prepared_xacts ppx
<para>
Anonymous allocations are allocations that have been made
with <literal>ShmemAlloc()</literal> directly, rather than via
- <literal>ShmemInitStruct()</literal> or
- <literal>ShmemInitHash()</literal>.
+ <literal>ShmemRegisterStruct()</literal> or
+ <literal>ShmemRegisterHash()</literal>.
</para>
<para>
diff --git a/doc/src/sgml/xfunc.sgml b/doc/src/sgml/xfunc.sgml
index 70e815b8a2c..05d9ec7e33a 100644
--- a/doc/src/sgml/xfunc.sgml
+++ b/doc/src/sgml/xfunc.sgml
@@ -3628,59 +3628,87 @@ CREATE FUNCTION make_array(anyelement) RETURNS anyarray
Add-ins can reserve shared memory on server startup. To do so, the
add-in's shared library must be preloaded by specifying it in
<xref linkend="guc-shared-preload-libraries"/><indexterm><primary>shared_preload_libraries</primary></indexterm>.
- The shared library should also register a
- <literal>shmem_request_hook</literal> in its
- <function>_PG_init</function> function. This
- <literal>shmem_request_hook</literal> can reserve shared memory by
- calling:
-<programlisting>
-void RequestAddinShmemSpace(Size size)
-</programlisting>
- Each backend should obtain a pointer to the reserved shared memory by
- calling:
+ The shared library should register the shared memory allocation in
+ its <function>_PG_init</function> function. Here is an example:
<programlisting>
-void *ShmemInitStruct(const char *name, Size size, bool *foundPtr)
-</programlisting>
- If this function sets <literal>foundPtr</literal> to
- <literal>false</literal>, the caller should proceed to initialize the
- contents of the reserved shared memory. If <literal>foundPtr</literal>
- is set to <literal>true</literal>, the shared memory was already
- initialized by another backend, and the caller need not initialize
- further.
- </para>
+typedef struct MyShmemData {
+ LWLock lock; /* protects the fields below */
- <para>
- To avoid race conditions, each backend should use the LWLock
- <function>AddinShmemInitLock</function> when initializing its allocation
- of shared memory, as shown here:
-<programlisting>
-static mystruct *ptr = NULL;
-bool found;
+ ... shared memory contents ...
+} MyShmemData;
+
+static MyShmemData *MyShmem; /* pointer to the struct in shared memory */
+
+static void my_shmem_init(void *arg);
+
+static ShmemStructDesc MyShmemDesc = {
+ .name = "My shmem area",
+ .size = sizeof(MyShmemData),
+ .init_fn = my_shmem_init,
+ .ptr = (void **) &MyShmem,
+};
-LWLockAcquire(AddinShmemInitLock, LW_EXCLUSIVE);
-ptr = ShmemInitStruct("my struct name", size, &found);
-if (!found)
+/*
+ * Module load callback
+ */
+void
+_PG_init(void)
{
- ... initialize contents of shared memory ...
- ptr->locks = GetNamedLWLockTranche("my tranche name");
+ /*
+ * In order to create our shared memory area, we have to be loaded via
+ * shared_preload_libraries.
+ */
+ if (!process_shared_preload_libraries_in_progress)
+ return;
+
+ /* Register our shared memory needs */
+ ShmemRegisterStruct(&MyShmemDesc);
}
-LWLockRelease(AddinShmemInitLock);
+
+/* callback to initialize the contents of the MyShmem area at startup */
+static void
+my_shmem_init(void *arg)
+{
+ int tranche_id;
+
+ /* Initialize the lock */
+ tranche_id = LWLockNewTrancheId("my tranche name");
+ LWLockInitialize(&MyShmem->lock, tranche_id);
+
+ ... initialize the rest of MyShmem fields ...
+}
+
</programlisting>
- <literal>shmem_startup_hook</literal> provides a convenient place for the
- initialization code, but it is not strictly required that all such code
- be placed in this hook. On Windows (and anywhere else where
- <literal>EXEC_BACKEND</literal> is defined), each backend executes the
- registered <literal>shmem_startup_hook</literal> shortly after it
- attaches to shared memory, so add-ins should still acquire
- <function>AddinShmemInitLock</function> within this hook, as shown in the
- example above. On other platforms, only the postmaster process executes
- the <literal>shmem_startup_hook</literal>, and each backend automatically
- inherits the pointers to shared memory.
+ The <function>ShmemRegisterStruct()</function> call doesn't immediately
+ allocate or initialize the memory, it merely registers the space to be
+ allocated later in the startup sequence. If the size of the allocation
+ depends on <varname>MaxBackends</varname> or other variables that are
+ not yet initialized when <function>_PG_init()</function> is called, the
+ size can still be adjusted later by registering a
+ <literal>shmem_request_hook</literal> and changing the descriptor there.
+ When the memory is allocated, the registered
+ <function>init_fn</function> callback is called to initialize it.
+ </para>
+ <para>
+ The <function>init_fn()</function> callback is normally called at
+ postmaster startup, when no other processes are running yet and no
+ locking is required. However, if a shared memory area is registered
+ after system start, e.g. in an extension that is not in
+ <xref linkend="guc-shared-preload-libraries"/><indexterm><primary>shared_preload_libraries</primary></indexterm>,
+ <function>ShmemRegisterStruct()</function> will immediately call
+ the <function>init_fn</function> callback. In that case, it holds a
+ lock internally that prevents concurrent shmem allocations.
+ </para>
+ <para>
+ On Windows, the <function>attach_fn</function> callback is additionally
+ called at every backend startup. It can be used for initializing
+ additional per-backend state related to the shared memory area that is
+ inherited via <function>fork()</function> on other systems. On other
+ platforms, the <function>attach_fn</function> callback is only called
+ for structs that are registered after system startup.
</para>
-
<para>
- An example of a <literal>shmem_request_hook</literal> and
- <literal>shmem_startup_hook</literal> can be found in
+ An example of allocating shared memory can be found in
<filename>contrib/pg_stat_statements/pg_stat_statements.c</filename> in
the <productname>PostgreSQL</productname> source tree.
</para>
@@ -3691,8 +3719,7 @@ LWLockRelease(AddinShmemInitLock);
<para>
There is another, more flexible method of reserving shared memory that
- can be done after server startup and outside a
- <literal>shmem_request_hook</literal>. To do so, each backend that will
+ can be done after server startup. To do so, each backend that will
use the shared memory should obtain a pointer to it by calling:
<programlisting>
void *GetNamedDSMSegment(const char *name, size_t size,
@@ -3711,10 +3738,7 @@ void *GetNamedDSMSegment(const char *name, size_t size,
</para>
<para>
- Unlike shared memory reserved at server startup, there is no need to
- acquire <function>AddinShmemInitLock</function> or otherwise take action
- to avoid race conditions when reserving shared memory with
- <function>GetNamedDSMSegment</function>. This function ensures that only
+ <function>GetNamedDSMSegment</function> ensures that only
one backend allocates and initializes the segment and that all other
backends receive a pointer to the fully allocated and initialized
segment.
diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c
index 68a42de0889..f5ae0cfa648 100644
--- a/src/backend/bootstrap/bootstrap.c
+++ b/src/backend/bootstrap/bootstrap.c
@@ -370,6 +370,8 @@ BootstrapModeMain(int argc, char *argv[], bool check_only)
InitializeFastPathLocks();
+ RegisterShmemStructs();
+
CreateSharedMemoryAndSemaphores();
/*
diff --git a/src/backend/postmaster/launch_backend.c b/src/backend/postmaster/launch_backend.c
index 30357845729..fecae827e5b 100644
--- a/src/backend/postmaster/launch_backend.c
+++ b/src/backend/postmaster/launch_backend.c
@@ -49,6 +49,7 @@
#include "replication/walreceiver.h"
#include "storage/dsm.h"
#include "storage/io_worker.h"
+#include "storage/ipc.h"
#include "storage/pg_shmem.h"
#include "tcop/backend_startup.h"
#include "utils/memutils.h"
@@ -677,7 +678,10 @@ SubPostmasterMain(int argc, char *argv[])
/* Restore basic shared memory pointers */
if (UsedShmemSegAddr != NULL)
+ {
InitShmemAllocator(UsedShmemSegAddr);
+ RegisterShmemStructs();
+ }
/*
* Run the appropriate Main function
diff --git a/src/backend/postmaster/postmaster.c b/src/backend/postmaster/postmaster.c
index 3fac46c402b..a3be3bbe3b6 100644
--- a/src/backend/postmaster/postmaster.c
+++ b/src/backend/postmaster/postmaster.c
@@ -958,6 +958,9 @@ PostmasterMain(int argc, char *argv[])
*/
InitializeFastPathLocks();
+ /* Register the shared memory needs of all core subsystems. */
+ RegisterShmemStructs();
+
/*
* Give preloaded libraries a chance to request additional shared memory.
*/
@@ -3235,7 +3238,13 @@ PostmasterStateMachine(void)
/* re-read control file into local memory */
LocalProcessControlFile(true);
- /* re-create shared memory and semaphores */
+ /*
+ * Re-initialize shared memory and semaphores. Note: We don't call
+ * RegisterShmemStructs() here, we keep the old registrations. In
+ * order to re-register structs in extensions, we'd need to reload
+ * shared preload libraries, and we don't want to do that.
+ */
+ ResetShmemAllocator();
CreateSharedMemoryAndSemaphores();
UpdatePMState(PM_STARTUP);
diff --git a/src/backend/storage/ipc/ipci.c b/src/backend/storage/ipc/ipci.c
index 3d3f153809b..405c69655f0 100644
--- a/src/backend/storage/ipc/ipci.c
+++ b/src/backend/storage/ipc/ipci.c
@@ -99,10 +99,12 @@ CalculateShmemSize(void)
* during the actual allocation phase.
*/
size = 100000;
- size = add_size(size, hash_estimate_size(SHMEM_INDEX_SIZE,
- sizeof(ShmemIndexEnt)));
+ size = add_size(size, ShmemRegisteredSize());
+
size = add_size(size, dsm_estimate_size());
size = add_size(size, DSMRegistryShmemSize());
+
+ /* legacy subsystems */
size = add_size(size, BufferManagerShmemSize());
size = add_size(size, LockManagerShmemSize());
size = add_size(size, PredicateLockShmemSize());
@@ -218,6 +220,10 @@ CreateSharedMemoryAndSemaphores(void)
*/
InitShmemAllocator(seghdr);
+ /* Reserve space for semaphores. */
+ if (!IsUnderPostmaster)
+ PGReserveSemaphores(ProcGlobalSemas());
+
/* Initialize subsystems */
CreateOrAttachShmemStructs();
@@ -231,6 +237,22 @@ CreateSharedMemoryAndSemaphores(void)
shmem_startup_hook();
}
+/*
+ * Early initialization of various subsystems, giving them a chance to
+ * register their shared memory needs before the shared memory segment is
+ * allocated.
+ */
+void
+RegisterShmemStructs(void)
+{
+ /*
+ * TODO: Not used in any built-in subsystems yet. In the future, most of
+ * the calls *ShmemInit() calls in CreateOrAttachShmemStructs(), and
+ * *ShmemSize() calls in CalculateShmemSize() will be replaced by calls
+ * into the subsystems from here.
+ */
+}
+
/*
* Initialize various subsystems, setting up their data structures in
* shared memory.
@@ -249,10 +271,26 @@ CreateSharedMemoryAndSemaphores(void)
static void
CreateOrAttachShmemStructs(void)
{
- /*
- * Now initialize LWLocks, which do shared memory allocation.
- */
- CreateLWLocks();
+#ifdef EXEC_BACKEND
+ if (IsUnderPostmaster)
+ {
+ /*
+ * ShmemAttachRegistered() uses LWLocks. Fortunately, LWLocks don't
+ * need any special attaching.
+ */
+ ShmemAttachRegistered();
+ }
+ else
+#endif
+ {
+ /*
+ * Initialize LWLocks first, in case any of the shmem init function
+ * use LWLocks. (Nothing else can be running during startup, so they
+ * don't need to do any locking yet, but we nevertheless allow it.)
+ */
+ CreateLWLocks();
+ ShmemInitRegistered();
+ }
dsm_shmem_init();
DSMRegistryShmemInit();
diff --git a/src/backend/storage/ipc/shmem.c b/src/backend/storage/ipc/shmem.c
index dce355e6683..d702db70f0c 100644
--- a/src/backend/storage/ipc/shmem.c
+++ b/src/backend/storage/ipc/shmem.c
@@ -19,48 +19,101 @@
* methods). The routines in this file are used for allocating and
* binding to shared memory data structures.
*
- * NOTES:
- * (a) There are three kinds of shared memory data structures
- * available to POSTGRES: fixed-size structures, queues and hash
- * tables. Fixed-size structures contain things like global variables
- * for a module and should never be allocated after the shared memory
- * initialization phase. Hash tables have a fixed maximum size, but
- * their actual size can vary dynamically. When entries are added
- * to the table, more space is allocated. Queues link data structures
- * that have been allocated either within fixed-size structures or as hash
- * buckets. Each shared data structure has a string name to identify
- * it (assigned in the module that declares it).
- *
- * (b) During initialization, each module looks for its
- * shared data structures in a hash table called the "Shmem Index".
- * If the data structure is not present, the caller can allocate
- * a new one and initialize it. If the data structure is present,
- * the caller "attaches" to the structure by initializing a pointer
- * in the local address space.
- * The shmem index has two purposes: first, it gives us
- * a simple model of how the world looks when a backend process
- * initializes. If something is present in the shmem index,
- * it is initialized. If it is not, it is uninitialized. Second,
- * the shmem index allows us to allocate shared memory on demand
- * instead of trying to preallocate structures and hard-wire the
- * sizes and locations in header files. If you are using a lot
- * of shared memory in a lot of different places (and changing
- * things during development), this is important.
- *
- * (c) In standard Unix-ish environments, individual backends do not
- * need to re-establish their local pointers into shared memory, because
- * they inherit correct values of those variables via fork() from the
- * postmaster. However, this does not work in the EXEC_BACKEND case.
- * In ports using EXEC_BACKEND, new backends have to set up their local
- * pointers using the method described in (b) above.
- *
- * (d) memory allocation model: shared memory can never be
- * freed, once allocated. Each hash table has its own free list,
- * so hash buckets can be reused when an item is deleted. However,
- * if one hash table grows very large and then shrinks, its space
- * cannot be redistributed to other tables. We could build a simple
- * hash bucket garbage collector if need be. Right now, it seems
- * unnecessary.
+ * Two kinds of shared memory data structures are handled by this module:
+ * fixed-size structures and hash tables. Fixed-size structures contain
+ * things like variables shared between all backend processes. Hash tables
+ * have a fixed maximum size, but their actual size can vary dynamically.
+ * When entries are added to the table, more space is allocated. Each shared
+ * data structure and hash has a string name to identify it, specified in the
+ * descriptor when its registered.
+ *
+ * Shared memory structs and hash tables should not be allocated after
+ * postmaster startup, although we do allow small allocations later for the
+ * benefit of extension modules that loaded after startup. Despite that
+ * allowance, extensions that need shared memory should be added in
+ * shared_preload_libraries, because the allowance is quite small and there is
+ * no guarantee that any memory is available after startup.
+ *
+ * Nowadays, there is also third way to allocate shared memory called Dynamic
+ * Shared Memory. See dsm.c for that facility. One big difference between
+ * traditional shared memory handled by shmem.c and dynamic shared memory is
+ * that traditional shared memory areas are mapped to the same address in all
+ * processes, so you can use normal pointers in shared memory structs. With
+ * Dynamic Shared Memory, you must use offsets or DSA pointers instead.
+ *
+ * Shared memory managed by shmem.c can never be freed, once allocated. Each
+ * hash table has its own free list, so hash buckets can be reused when an
+ * item is deleted. However, if one hash table grows very large and then
+ * shrinks, its space cannot be redistributed to other tables. We could build
+ * a simple hash bucket garbage collector if need be. Right now, it seems
+ * unnecessary.
+ *
+ * Usage
+ * -----
+ *
+ * To allocate a shared memory area, fill in the name, size, and any other
+ * options in ShmemStructDesc, and call ShmemRegisterStruct(). Leave any
+ * unused fields as zeros.
+ *
+ * typedef struct MyShmemData {
+ * ...
+ * } MyShmemData;
+ *
+ * static MyShmemData *MyShmem;
+ *
+ * static void my_shmem_init(void *arg);
+ *
+ * static ShmemStructDesc MyShmemDesc = {
+ * .name = "My shmem area",
+ * .size = sizeof(MyShmemData),
+ * .init_fn = my_shmem_init,
+ * .ptr = &MyShmem,
+ * };
+ *
+ * In the subsystem's initialization code (or in _PG_init() in extensions),
+ * call ShmemRegisterStruct(&MyShmemDesc).
+ *
+ * Lifecycle
+ * ---------
+ *
+ * RegisterShmemStructs() is called at postmaster startup before calculating
+ * the size of the global shared memory segment. Once all the registrations
+ * have been done, postmaster calls ShmemRegisteredSize() to add up the sizes
+ * of all the registered areas. After allocating the shared memory segment,
+ * postmaster calls ShmemInitRegistered(), which calls the init_fn callback,
+ * if any, of each registered area, in the order that they were registered.
+ *
+ * In standard Unix-ish environments, individual backends do not need to
+ * re-establish their local pointers into shared memory, because they inherit
+ * correct values of those variables via fork() from the postmaster. However,
+ * this does not work in the EXEC_BACKEND case. In ports using EXEC_BACKEND,
+ * backend startup also calls RegisterShmemStructs(), followed by
+ * ShmemAttachRegistered(), which re-establishes the pointer variables
+ * (*ShmemStructDesc->ptr), and calls the attach_fn callback, if any, for
+ * additional per-backend setup.
+ *
+ * Legacy ShmemInitStruct()/ShmemInitHash() functions
+ * --------------------------------------------------
+ *
+ * ShmemInitStruct()/ShmemInitHash() is another way of registring shmem areas.
+ * It pre-dates the ShmemRegisterStruct()/ShmemRegisterHash() functions, and
+ * should not be used in new code, but as of this writing it is still widely
+ * used in extensions.
+ *
+ * To allocate a shmem area with ShmemInitStruct(), you need to separately
+ * register the size needed for the area by calling RequestAddinShmemSpace()
+ * from the extension's shmem_request_hook, and allocate the area by calling
+ * ShmemInitStruct() from the extension's shmem_startup_hook. There are no
+ * init/attach callbacks. Instead, the caller of ShmemInitStruct() must check
+ * the return status of ShmemInitStruct() and initialize the struct if it was
+ * not previously initialized.
+ *
+ * Calling ShmemAlloc() directly
+ * -----------------------------
+ *
+ * There's a more low-level way of allocating shared memory too: you can call
+ * ShmemAlloc() directly. It's used to implement the higher level mechanisms,
+ * and should generally not be called directly.
*/
#include "postgres.h"
@@ -79,6 +132,28 @@
#include "utils/builtins.h"
#include "utils/tuplestore.h"
+/*
+ * Array of registered shared memory areas.
+ *
+ * This is in process private memory, although on Unix-like systems, we expect
+ * all the registrations to happen at postmaster startup time and be inherited
+ * by all the child processes via fork(). Extensions may register additional
+ * areas after startup, but only areas registered at postmaster startup are
+ * included in the estimate for the total memory needed for shared memory. If
+ * any non-trivial allocations are made after startup, there might not be
+ * enough shared memory available.
+ */
+static struct
+{
+ ShmemStructDesc *desc; /* registered descriptor */
+ bool legacy; /* legacy ShmemInitStruct/Hash entry? */
+} *registered_shmem_areas;
+static int num_registered_shmem_areas = 0;
+static int max_registered_shmem_areas = 0; /* allocated size of the array */
+
+/* estimated size of registered_shmem_areas (not a hard limit) */
+#define INITIAL_REGISTRY_SIZE (64)
+
/*
* This is the first data structure stored in the shared memory segment, at
* the offset that PGShmemHeader->content_offset points to. Allocations by
@@ -96,8 +171,13 @@ typedef struct ShmemAllocatorData
slock_t shmem_lock;
} ShmemAllocatorData;
+static bool ShmemRegisterStructInternal(ShmemStructDesc *desc, bool legacy);
+static bool ShmemRegisterHashInternal(ShmemHashDesc *desc, bool legacy);
static void *ShmemAllocRaw(Size size, Size *allocated_size);
+static void shmem_hash_init(void *arg);
+static void shmem_hash_attach(void *arg);
+
/* shared memory global variables */
static PGShmemHeader *ShmemSegHdr; /* shared mem segment header */
@@ -106,20 +186,332 @@ static void *ShmemEnd; /* end+1 address of shared memory */
static ShmemAllocatorData *ShmemAllocator;
slock_t *ShmemLock; /* points to ShmemAllocator->shmem_lock */
-static HTAB *ShmemIndex = NULL; /* primary index hashtable for shmem */
+
+static bool shmem_initialized = false;
+
+/*
+ * ShmemIndex is a global directory of shmem areas, itself also stored in the
+ * shared memory.
+ */
+static HTAB *ShmemIndex;
+
+ /* max size of data structure string name */
+#define SHMEM_INDEX_KEYSIZE (48)
+
+/*
+ * # of additional entries to reserve in the shmem index table, for allocations
+ * after postmaster startup (not a hard limit)
+ */
+#define SHMEM_INDEX_ADDITIONAL_SIZE (64)
+
+/* this is a hash bucket in the shmem index table */
+typedef struct
+{
+ char key[SHMEM_INDEX_KEYSIZE]; /* string name */
+ void *location; /* location in shared mem */
+ Size size; /* # bytes requested for the structure */
+ Size allocated_size; /* # bytes actually allocated */
+} ShmemIndexEnt;
/* To get reliable results for NUMA inquiry we need to "touch pages" once */
static bool firstNumaTouch = true;
Datum pg_numa_available(PG_FUNCTION_ARGS);
+/*
+ * ShmemRegisterStruct() --- register a shared memory struct
+ *
+ * Subsystems call this to register their shared memory needs. That should be
+ * done early in postmaster startup, before the shared memory segment has been
+ * created, so that the size can be included in the estimate for total amount
+ * of shared memory needed. We set aside a small amount of memory for
+ * allocations that happen later, for the benefit of non-preloaded extensions,
+ * but that should not be relied upon.
+ *
+ * In core subsystems, each subsystem's registration function is called from
+ * RegisterShmemStructs(). In extensions, this should be called from the
+ * _PG_init() function. In EXEC_BACKEND mode, this also needs to be called in
+ * each child process, to reattach and set the pointer to the shared memory
+ * area, usually in a global variable. Calling this from the _PG_init()
+ * initializer takes care of that too.
+ *
+ * When called during postmaster startup, before the shared memory has been
+ * allocated, the function merely remembers the registered descriptor, but the
+ * descriptor may still be changed later, until the shared memory segment has
+ * been allocated. That means that an extension may still modify the
+ * already-registered descriptor in the shmem_request_hook. A common example
+ * of when that's useful is when the size depends on MaxBackends: you can
+ * leave the size empty in the ShmemRegisterStruct() call and fill it later in
+ * the shmem_request_hook.
+ *
+ * Returns true if the struct was already initialized in shared memory and we
+ * merely attached to it.
+ */
+bool
+ShmemRegisterStruct(ShmemStructDesc *desc)
+{
+ return ShmemRegisterStructInternal(desc, false);
+}
+
+static bool
+ShmemRegisterStructInternal(ShmemStructDesc *desc, bool legacy)
+{
+ bool found;
+
+ /* Check that it's not already registered in this process */
+ for (int i = 0; i < num_registered_shmem_areas; i++)
+ {
+ ShmemStructDesc *existing = registered_shmem_areas[i].desc;
+
+ if (strcmp(existing->name, desc->name) == 0)
+ ereport(ERROR,
+ (errmsg("shared memory struct \"%s\" is already registered",
+ desc->name),
+ errbacktrace()));
+ }
+
+ /* desc->ptr can be non-NULL when re-initializing after crash */
+ if (!IsUnderPostmaster && desc->ptr)
+ *desc->ptr = NULL;
+
+ /* Add the descriptor to the array, growing the array if needed */
+ if (num_registered_shmem_areas == max_registered_shmem_areas)
+ {
+ int new_size;
+
+ if (registered_shmem_areas)
+ {
+ new_size = max_registered_shmem_areas * 2;
+ registered_shmem_areas = repalloc(registered_shmem_areas,
+ new_size * sizeof(*registered_shmem_areas));
+ }
+ else
+ {
+ new_size = INITIAL_REGISTRY_SIZE;
+ registered_shmem_areas = MemoryContextAlloc(TopMemoryContext,
+ new_size * sizeof(*registered_shmem_areas));
+ }
+ max_registered_shmem_areas = new_size;
+ }
+ registered_shmem_areas[num_registered_shmem_areas].desc = desc;
+ registered_shmem_areas[num_registered_shmem_areas].legacy = legacy;
+ num_registered_shmem_areas++;
+
+ /*
+ * If called after postmaster startup, we need to immediately also
+ * initialize or attach to the area.
+ */
+ if (shmem_initialized)
+ {
+ ShmemIndexEnt *index_entry;
+
+ LWLockAcquire(ShmemIndexLock, LW_EXCLUSIVE);
+
+ /* look it up in the shmem index */
+ index_entry = (ShmemIndexEnt *)
+ hash_search(ShmemIndex, desc->name, HASH_ENTER_NULL, &found);
+ if (!index_entry)
+ {
+ ereport(ERROR,
+ (errcode(ERRCODE_OUT_OF_MEMORY),
+ errmsg("could not create ShmemIndex entry for data structure \"%s\"",
+ desc->name)));
+ }
+ if (found)
+ {
+ /* Already present, just attach to it */
+ if (index_entry->size != desc->size)
+ elog(ERROR, "shared memory struct \"%s\" is already registered with different size",
+ desc->name);
+ if (desc->ptr)
+ *desc->ptr = index_entry->location;
+ if (desc->attach_fn)
+ desc->attach_fn(desc->attach_fn_arg);
+ }
+ else
+ {
+ /*
+ * This is the first time. Initialize it like
+ * ShmemInitRegistered() would
+ */
+ size_t allocated_size;
+ void *structPtr;
+
+ structPtr = ShmemAllocRaw(desc->size, &allocated_size);
+ if (structPtr == NULL)
+ {
+ /* out of memory; remove the failed ShmemIndex entry */
+ hash_search(ShmemIndex, desc->name, HASH_REMOVE, NULL);
+ ereport(ERROR,
+ (errcode(ERRCODE_OUT_OF_MEMORY),
+ errmsg("not enough shared memory for data structure"
+ " \"%s\" (%zu bytes requested)",
+ desc->name, desc->size)));
+ }
+ index_entry->size = desc->size;
+ index_entry->allocated_size = allocated_size;
+ index_entry->location = structPtr;
+ if (desc->ptr)
+ *desc->ptr = index_entry->location;
+
+ /*
+ * XXX: if this errors out, the area is left in a half-initialized
+ * state
+ */
+ if (desc->init_fn)
+ desc->init_fn(desc->init_fn_arg);
+ }
+
+ LWLockRelease(ShmemIndexLock);
+ }
+ else
+ found = false;
+
+ return found;
+}
+
+/*
+ * ShmemRegisteredSize() --- estimate the total size of all registered shared
+ * memory structures.
+ *
+ * This is called once at postmaster startup, before the shared memory segment
+ * has been created.
+ */
+size_t
+ShmemRegisteredSize(void)
+{
+ size_t size;
+
+ /* memory needed for the ShmemIndex */
+ size = hash_estimate_size(num_registered_shmem_areas + SHMEM_INDEX_ADDITIONAL_SIZE,
+ sizeof(ShmemIndexEnt));
+
+ /* memory needed for all the registered areas */
+ for (int i = 0; i < num_registered_shmem_areas; i++)
+ {
+ ShmemStructDesc *desc = registered_shmem_areas[i].desc;
+
+ size = add_size(size, desc->size);
+ size = add_size(size, desc->extra_size);
+ }
+
+ return size;
+}
+
+/*
+ * ShmemInitRegistered() --- allocate and initialize pre-registered shared
+ * memory structures.
+ *
+ * This is called once at postmaster startup, after the shared memory segment
+ * has been created.
+ */
+void
+ShmemInitRegistered(void)
+{
+ /* Should be called only by the postmaster or a standalone backend. */
+ Assert(!IsUnderPostmaster);
+ Assert(!shmem_initialized);
+
+ /*
+ * Initialize all the registered memory areas. There are no concurrent
+ * processes yet, so no need for locking.
+ */
+ for (int i = 0; i < num_registered_shmem_areas; i++)
+ {
+ ShmemStructDesc *desc = registered_shmem_areas[i].desc;
+ size_t allocated_size;
+ void *structPtr;
+ bool found;
+ ShmemIndexEnt *index_entry;
+
+ /* look it up in the shmem index */
+ index_entry = (ShmemIndexEnt *)
+ hash_search(ShmemIndex, desc->name, HASH_ENTER_NULL, &found);
+ if (!index_entry)
+ {
+ ereport(ERROR,
+ (errcode(ERRCODE_OUT_OF_MEMORY),
+ errmsg("could not create ShmemIndex entry for data structure \"%s\"",
+ desc->name)));
+ }
+ if (found)
+ elog(ERROR, "shared memory struct \"%s\" is already initialized", desc->name);
+
+ /* allocate and initialize it */
+ structPtr = ShmemAllocRaw(desc->size, &allocated_size);
+ if (structPtr == NULL)
+ {
+ /* out of memory; remove the failed ShmemIndex entry */
+ hash_search(ShmemIndex, desc->name, HASH_REMOVE, NULL);
+ ereport(ERROR,
+ (errcode(ERRCODE_OUT_OF_MEMORY),
+ errmsg("not enough shared memory for data structure"
+ " \"%s\" (%zu bytes requested)",
+ desc->name, desc->size)));
+ }
+ index_entry->size = desc->size;
+ index_entry->allocated_size = allocated_size;
+ index_entry->location = structPtr;
+
+ *(desc->ptr) = structPtr;
+ if (desc->init_fn)
+ desc->init_fn(desc->init_fn_arg);
+ }
+
+ shmem_initialized = true;
+}
+
+/*
+ * Call the attach_fn callbacks of all registered shmem areas
+ *
+ * This is called at backend startup, in EXEC_BACKEND mode.
+ */
+#ifdef EXEC_BACKEND
+void
+ShmemAttachRegistered(void)
+{
+ /* Must be initializing a (non-standalone) backend */
+ Assert(IsUnderPostmaster);
+ Assert(ShmemAllocator->index != NULL);
+
+ LWLockAcquire(ShmemIndexLock, LW_SHARED);
+
+ for (int i = 0; i < num_registered_shmem_areas; i++)
+ {
+ ShmemStructDesc *desc = registered_shmem_areas[i].desc;
+ bool found;
+ ShmemIndexEnt *result;
+
+ /* look it up in the shmem index */
+ result = (ShmemIndexEnt *)
+ hash_search(ShmemIndex, desc->name, HASH_FIND, &found);
+ if (!found)
+ {
+ ereport(ERROR,
+ (errcode(ERRCODE_OUT_OF_MEMORY),
+ errmsg("could not find ShmemIndex entry for data structure \"%s\"",
+ desc->name)));
+ }
+
+ if (desc->ptr)
+ *desc->ptr = result->location;
+ if (desc->attach_fn)
+ desc->attach_fn(desc->attach_fn_arg);
+ }
+
+ LWLockRelease(ShmemIndexLock);
+
+ shmem_initialized = true;
+}
+#endif
+
/*
* InitShmemAllocator() --- set up basic pointers to shared memory.
*
* Called at postmaster or stand-alone backend startup, to initialize the
* allocator's data structure in the shared memory segment. In EXEC_BACKEND,
- * this is also called at backend startup, to set up pointers to the shared
- * memory areas.
+ * this is also called at backend startup, to set up pointers to the
+ * already-initialized data structure.
*/
void
InitShmemAllocator(PGShmemHeader *seghdr)
@@ -130,6 +522,7 @@ InitShmemAllocator(PGShmemHeader *seghdr)
int hash_flags;
size_t size;
+ Assert(!shmem_initialized);
Assert(seghdr != NULL);
/*
@@ -174,7 +567,7 @@ InitShmemAllocator(PGShmemHeader *seghdr)
* use ShmemInitHash() here because it relies on ShmemIndex being already
* initialized.
*/
- hash_size = SHMEM_INDEX_SIZE;
+ hash_size = num_registered_shmem_areas + SHMEM_INDEX_ADDITIONAL_SIZE;
info.keysize = SHMEM_INDEX_KEYSIZE;
info.entrysize = sizeof(ShmemIndexEnt);
@@ -193,6 +586,38 @@ InitShmemAllocator(PGShmemHeader *seghdr)
Assert(ShmemIndex != NULL);
}
+/*
+ * Reset the shmem struct registry on postmaster crash restart.
+ */
+void
+ResetShmemAllocator(void)
+{
+ int num_retained;
+
+ shmem_initialized = false;
+
+ /*
+ * Shared memory areas will not be registered again after a crash restart.
+ * We don't call RegisterShmemStructs() on crash restart, which would
+ * re-register core subsystems, and we don't reload
+ * shared_preload_libraries either.
+ *
+ * However, we do expect the legacy ShmemInitStruct() function will be
+ * called again for each area, so remove those from the registry.
+ */
+ num_retained = 0;
+ for (int i = 0; i < num_registered_shmem_areas; i++)
+ {
+ if (!registered_shmem_areas[i].legacy)
+ {
+ if (num_retained != i)
+ registered_shmem_areas[num_retained] = registered_shmem_areas[i];
+ num_retained++;
+ }
+ }
+ num_registered_shmem_areas = num_retained;
+}
+
/*
* ShmemAlloc -- allocate max-aligned chunk from shared memory
*
@@ -290,42 +715,20 @@ ShmemAddrIsValid(const void *addr)
}
/*
- * ShmemInitHash -- Create and initialize, or attach to, a
- * shared memory hash table.
+ * ShmemRegisterHash -- Register a shared memory hash table.
*
- * We assume caller is doing some kind of synchronization
- * so that two processes don't try to create/initialize the same
- * table at once. (In practice, all creations are done in the postmaster
- * process; child processes should always be attaching to existing tables.)
- *
- * max_size is the estimated maximum number of hashtable entries. This is
- * not a hard limit, but the access efficiency will degrade if it is
- * exceeded substantially (since it's used to compute directory size and
- * the hash table buckets will get overfull).
- *
- * init_size is the number of hashtable entries to preallocate. For a table
- * whose maximum size is certain, this should be equal to max_size; that
- * ensures that no run-time out-of-shared-memory failures can occur.
- *
- * *infoP and hash_flags must specify at least the entry sizes and key
- * comparison semantics (see hash_create()). Flag bits and values specific
- * to shared-memory hash tables are added here, except that callers may
- * choose to specify HASH_PARTITION and/or HASH_FIXED_SIZE.
- *
- * Note: before Postgres 9.0, this function returned NULL for some failure
- * cases. Now, it always throws error instead, so callers need not check
- * for NULL.
+ * Similar to ShmemRegisterStruct(), but registers a hash table instead of an
+ * opaque area.
*/
-HTAB *
-ShmemInitHash(const char *name, /* table string name for shmem index */
- int64 init_size, /* initial table size */
- int64 max_size, /* max size of the table */
- HASHCTL *infoP, /* info about key and bucket size */
- int hash_flags) /* info about infoP */
+bool
+ShmemRegisterHash(ShmemHashDesc *desc)
{
- bool found;
- void *location;
+ return ShmemRegisterHashInternal(desc, false);
+}
+static bool
+ShmemRegisterHashInternal(ShmemHashDesc *desc, bool legacy)
+{
/*
* Hash tables allocated in shared memory have a fixed directory; it can't
* grow or other backends wouldn't be able to find it. So, make sure we
@@ -333,26 +736,56 @@ ShmemInitHash(const char *name, /* table string name for shmem index */
*
* The shared memory allocator must be specified too.
*/
- infoP->dsize = infoP->max_dsize = hash_select_dirsize(max_size);
- infoP->alloc = ShmemAllocNoError;
- hash_flags |= HASH_SHARED_MEM | HASH_ALLOC | HASH_DIRSIZE;
-
- /* look it up in the shmem index */
- location = ShmemInitStruct(name,
- hash_get_shared_size(infoP, hash_flags),
- &found);
+ desc->hash_info.dsize = desc->hash_info.max_dsize = hash_select_dirsize(desc->max_size);
+ desc->hash_info.alloc = ShmemAllocNoError;
+ desc->hash_flags |= HASH_SHARED_MEM | HASH_ALLOC | HASH_DIRSIZE;
+
+ /* Set up the base struct descriptor */
+ memset(&desc->base_desc, 0, sizeof(desc->base_desc));
+ desc->base_desc.name = desc->name;
+ desc->base_desc.size = hash_get_shared_size(&desc->hash_info, desc->hash_flags);
+ desc->base_desc.init_fn = shmem_hash_init;
+ desc->base_desc.init_fn_arg = desc;
+ desc->base_desc.attach_fn = shmem_hash_attach;
+ desc->base_desc.attach_fn_arg = desc;
/*
- * if it already exists, attach to it rather than allocate and initialize
- * new space
+ * We need a stable pointer to hold the pointer to the shared memory. Use
+ * the one passed in the descriptor now. It will be replaced with the
+ * hash table header by init or attach function.
*/
- if (found)
- hash_flags |= HASH_ATTACH;
+ desc->base_desc.ptr = (void **) desc->ptr;
+
+ desc->base_desc.extra_size = hash_estimate_size(desc->max_size, desc->hash_info.entrysize) - desc->base_desc.size;
+
+ return ShmemRegisterStructInternal(&desc->base_desc, legacy);
+}
+
+static void
+shmem_hash_init(void *arg)
+{
+ ShmemHashDesc *desc = (ShmemHashDesc *) arg;
+ int hash_flags = desc->hash_flags;
/* Pass location of hashtable header to hash_create */
- infoP->hctl = (HASHHDR *) location;
+ desc->hash_info.hctl = (HASHHDR *) *desc->base_desc.ptr;
- return hash_create(name, init_size, infoP, hash_flags);
+ *desc->ptr = hash_create(desc->name, desc->init_size, &desc->hash_info, hash_flags);
+}
+
+static void
+shmem_hash_attach(void *arg)
+{
+ ShmemHashDesc *desc = (ShmemHashDesc *) arg;
+ int hash_flags = desc->hash_flags;
+
+ /* attach to it rather than allocate and initialize new space */
+ hash_flags |= HASH_ATTACH;
+
+ /* Pass location of hashtable header to hash_create */
+ desc->hash_info.hctl = (HASHHDR *) *desc->base_desc.ptr;
+
+ *desc->ptr = hash_create(desc->name, desc->init_size, &desc->hash_info, hash_flags);
}
/*
@@ -367,82 +800,76 @@ ShmemInitHash(const char *name, /* table string name for shmem index */
* Returns: pointer to the object. *foundPtr is set true if the object was
* already in the shmem index (hence, already initialized).
*
- * Note: before Postgres 9.0, this function returned NULL for some failure
- * cases. Now, it always throws error instead, so callers need not check
- * for NULL.
+ * Note: This is a legacy interface, kept for backwards compatibility with
+ * extensions. Use ShmemRegisterStruct() in new code!
*/
void *
ShmemInitStruct(const char *name, Size size, bool *foundPtr)
{
- ShmemIndexEnt *result;
- void *structPtr;
-
- Assert(ShmemIndex != NULL);
+ ShmemStructDesc *desc;
- LWLockAcquire(ShmemIndexLock, LW_EXCLUSIVE);
+ Assert(shmem_initialized);
- /* look it up in the shmem index */
- result = (ShmemIndexEnt *)
- hash_search(ShmemIndex, name, HASH_ENTER_NULL, foundPtr);
+ desc = MemoryContextAllocZero(TopMemoryContext, sizeof(ShmemStructDesc) + sizeof(void *));
+ desc->name = name;
+ desc->size = size;
+ desc->ptr = (void *) (((char *) desc) + sizeof(ShmemStructDesc));
- if (!result)
- {
- LWLockRelease(ShmemIndexLock);
- ereport(ERROR,
- (errcode(ERRCODE_OUT_OF_MEMORY),
- errmsg("could not create ShmemIndex entry for data structure \"%s\"",
- name)));
- }
-
- if (*foundPtr)
- {
- /*
- * Structure is in the shmem index so someone else has allocated it
- * already. The size better be the same as the size we are trying to
- * initialize to, or there is a name conflict (or worse).
- */
- if (result->size != size)
- {
- LWLockRelease(ShmemIndexLock);
- ereport(ERROR,
- (errmsg("ShmemIndex entry size is wrong for data structure"
- " \"%s\": expected %zu, actual %zu",
- name, size, result->size)));
- }
- structPtr = result->location;
- }
- else
- {
- Size allocated_size;
+ *foundPtr = ShmemRegisterStructInternal(desc, true);
+ Assert(*desc->ptr != NULL);
+ return *desc->ptr;
+}
- /* It isn't in the table yet. allocate and initialize it */
- structPtr = ShmemAllocRaw(size, &allocated_size);
- if (structPtr == NULL)
- {
- /* out of memory; remove the failed ShmemIndex entry */
- hash_search(ShmemIndex, name, HASH_REMOVE, NULL);
- LWLockRelease(ShmemIndexLock);
- ereport(ERROR,
- (errcode(ERRCODE_OUT_OF_MEMORY),
- errmsg("not enough shared memory for data structure"
- " \"%s\" (%zu bytes requested)",
- name, size)));
- }
- result->size = size;
- result->allocated_size = allocated_size;
- result->location = structPtr;
- }
+/*
+ * ShmemInitHash -- Create and initialize, or attach to, a
+ * shared memory hash table.
+ *
+ * We assume caller is doing some kind of synchronization
+ * so that two processes don't try to create/initialize the same
+ * table at once. (In practice, all creations are done in the postmaster
+ * process; child processes should always be attaching to existing tables.)
+ *
+ * max_size is the estimated maximum number of hashtable entries. This is
+ * not a hard limit, but the access efficiency will degrade if it is
+ * exceeded substantially (since it's used to compute directory size and
+ * the hash table buckets will get overfull).
+ *
+ * init_size is the number of hashtable entries to preallocate. For a table
+ * whose maximum size is certain, this should be equal to max_size; that
+ * ensures that no run-time out-of-shared-memory failures can occur.
+ *
+ * *infoP and hash_flags must specify at least the entry sizes and key
+ * comparison semantics (see hash_create()). Flag bits and values specific
+ * to shared-memory hash tables are added here, except that callers may
+ * choose to specify HASH_PARTITION and/or HASH_FIXED_SIZE.
+ *
+ * Note: This is a legacy interface, kept for backwards compatibility with
+ * extensions. Use ShmemRegisterHash() in new code!
+ */
+HTAB *
+ShmemInitHash(const char *name, /* table string name for shmem index */
+ int64 init_size, /* initial table size */
+ int64 max_size, /* max size of the table */
+ HASHCTL *infoP, /* info about key and bucket size */
+ int hash_flags) /* info about infoP */
+{
+ ShmemHashDesc *desc;
- LWLockRelease(ShmemIndexLock);
+ Assert(shmem_initialized);
- Assert(ShmemAddrIsValid(structPtr));
+ desc = MemoryContextAllocZero(TopMemoryContext, sizeof(ShmemHashDesc) + sizeof(HTAB *));
+ desc->name = name;
+ desc->init_size = init_size;
+ desc->max_size = max_size;
+ memcpy(&desc->hash_info, infoP, sizeof(HASHCTL));
+ desc->hash_flags = hash_flags;
- Assert(structPtr == (void *) CACHELINEALIGN(structPtr));
+ desc->ptr = (HTAB **) (((char *) desc) + sizeof(ShmemHashDesc));
- return structPtr;
+ ShmemRegisterHashInternal(desc, true);
+ return *desc->ptr;
}
-
/*
* Add two Size values, checking for overflow
*/
diff --git a/src/backend/tcop/postgres.c b/src/backend/tcop/postgres.c
index b3563113219..9bfc663f44a 100644
--- a/src/backend/tcop/postgres.c
+++ b/src/backend/tcop/postgres.c
@@ -4163,6 +4163,9 @@ PostgresSingleUserMain(int argc, char *argv[],
/* Initialize size of fast-path lock cache. */
InitializeFastPathLocks();
+ /* Register the shared memory needs of all core subsystems. */
+ RegisterShmemStructs();
+
/*
* Give preloaded libraries a chance to request additional shared memory.
*/
diff --git a/src/include/storage/ipc.h b/src/include/storage/ipc.h
index da32787ab51..8a3b71ad5d3 100644
--- a/src/include/storage/ipc.h
+++ b/src/include/storage/ipc.h
@@ -77,6 +77,7 @@ extern void check_on_shmem_exit_lists_are_empty(void);
/* ipci.c */
extern PGDLLIMPORT shmem_startup_hook_type shmem_startup_hook;
+extern void RegisterShmemStructs(void);
extern Size CalculateShmemSize(void);
extern void CreateSharedMemoryAndSemaphores(void);
#ifdef EXEC_BACKEND
diff --git a/src/include/storage/shmem.h b/src/include/storage/shmem.h
index 0de8a36429b..08aa6380a43 100644
--- a/src/include/storage/shmem.h
+++ b/src/include/storage/shmem.h
@@ -24,18 +24,156 @@
#include "storage/spin.h"
#include "utils/hsearch.h"
+typedef void (*ShmemInitCallback) (void *arg);
+typedef void (*ShmemAttachCallback) (void *arg);
+
+/*
+ * ShmemStructDesc describes a named area or struct in shared memory.
+ *
+ * Shared memory is reserved and allocated in a few phases at postmaster
+ * startup, and in EXEC_BACKEND mode, there's some extra work done to "attach"
+ * to them at backend startup. ShmemStructDesc contains all the information
+ * needed to manage the lifecycle.
+ *
+ * 'name' must be filled in before calling ShmemRegisterStruct(); all other
+ * fields can be adjusted later during postmaster startup, until the shared
+ * memory is allocated and init callback is called. Initialize any optional
+ * fields that you don't use to zeros.
+ *
+ * After registration, the shmem machinery reserves memory for the area, sets
+ * '*ptr' to point to the allocation, and calls the callbacks at the right
+ * moments.
+ */
+typedef struct ShmemStructDesc
+{
+ /*
+ * Name of the shared memory area. Required, must be unique, and must be
+ * set already before calling ShmemRegisterStruct().
+ */
+ const char *name;
+
+ /* Size of the shared memory area. Required. */
+ size_t size;
+
+ /*
+ * Initialization callback function. This is called when the shared
+ * memory area is allocated, usually at postmaster startup. 'init_fn_arg'
+ * is an opaque argument passed to the callback.
+ */
+ ShmemInitCallback init_fn;
+ void *init_fn_arg;
+
+ /*
+ * Attachment callback function. In EXEC_BACKEND mode, this is called at
+ * startup of each backend. In !EXEC_BACKEND mode, this is only called if
+ * the shared memory area is registered after postmaster startup. We
+ * never do that in core code, but extensions might.
+ */
+ ShmemInitCallback attach_fn;
+ void *attach_fn_arg;
+
+ /*
+ * Extra space to reserve in the shared memory segment, but it's not part
+ * of the struct itself. This is used for shared memory hash tables that
+ * can grow beyond the initial size when more buckets are allocated.
+ */
+ size_t extra_size;
+
+ /*
+ * When the shmem area is initialized or attached to, pointer to it is
+ * stored in *ptr. It usually points to a global variable, used to access
+ * the shared memory area later. *ptr is set before the init_fn or
+ * attach_fn callback is called.
+ */
+ void **ptr;
+} ShmemStructDesc;
+
+/*
+ * Descriptor for a named shared memory hash table.
+ *
+ * Similar to ShmemStructDesc, but describes a shared memory hash table. Each
+ * hash table is backed by an allocated area, described by 'base_desc', but if
+ * 'max_size' is greater than 'init_size', it can also grow beyond the initial
+ * allocated area by allocating more hash entries from the global unreserved
+ * space.
+ */
+typedef struct ShmemHashDesc
+{
+ /*
+ * Name of the shared memory area. Required. Must be unique across the
+ * system.
+ */
+ const char *name;
+
+ /*
+ * max_size is the estimated maximum number of hashtable entries. This is
+ * not a hard limit, but the access efficiency will degrade if it is
+ * exceeded substantially (since it's used to compute directory size and
+ * the hash table buckets will get overfull).
+ */
+ size_t max_size;
+
+ /*
+ * init_size is the number of hashtable entries to preallocate. For a
+ * table whose maximum size is certain, this should be equal to max_size;
+ * that ensures that no run-time out-of-shared-memory failures can occur.
+ */
+ size_t init_size;
+
+ /*
+ * Hash table options passed to hash_create()
+ *
+ * hash_info and hash_flags must specify at least the entry sizes and key
+ * comparison semantics (see hash_create()). Flag bits and values
+ * specific to shared-memory hash tables are added implicitly in
+ * ShmemRegisterHash(), except that callers may choose to specify
+ * HASH_PARTITION and/or HASH_FIXED_SIZE.
+ */
+ HASHCTL hash_info;
+ int hash_flags;
+
+ /*
+ * When the hash table is initialized or attached to, pointer to its
+ * backend-private handle is stored in *ptr. It usually points to a
+ * global variable, used to access the hash table later.
+ */
+ HTAB **ptr;
+
+ /*
+ * Descriptor for the underlying "area". Callers of ShmemRegisterHash()
+ * do not need to touch this, it is filled in by ShmemRegisterHash() based
+ * on the hash table parameters.
+ */
+ ShmemStructDesc base_desc;
+} ShmemHashDesc;
/* shmem.c */
extern PGDLLIMPORT slock_t *ShmemLock;
typedef struct PGShmemHeader PGShmemHeader; /* avoid including
* storage/pg_shmem.h here */
+extern void ResetShmemAllocator(void);
extern void InitShmemAllocator(PGShmemHeader *seghdr);
+#ifdef EXEC_BACKEND
+extern void AttachShmemAllocator(PGShmemHeader *seghdr);
+#endif
extern void *ShmemAlloc(Size size);
extern void *ShmemAllocNoError(Size size);
extern bool ShmemAddrIsValid(const void *addr);
+
+extern bool ShmemRegisterStruct(ShmemStructDesc *desc);
+extern bool ShmemRegisterHash(ShmemHashDesc *desc);
+
+/* legacy shmem allocation functions */
extern HTAB *ShmemInitHash(const char *name, int64 init_size, int64 max_size,
HASHCTL *infoP, int hash_flags);
extern void *ShmemInitStruct(const char *name, Size size, bool *foundPtr);
+
+extern size_t ShmemRegisteredSize(void);
+extern void ShmemInitRegistered(void);
+#ifdef EXEC_BACKEND
+extern void ShmemAttachRegistered(void);
+#endif
+
extern Size add_size(Size s1, Size s2);
extern Size mul_size(Size s1, Size s2);
@@ -44,19 +182,4 @@ extern PGDLLIMPORT Size pg_get_shmem_pagesize(void);
/* ipci.c */
extern void RequestAddinShmemSpace(Size size);
-/* size constants for the shmem index table */
- /* max size of data structure string name */
-#define SHMEM_INDEX_KEYSIZE (48)
- /* estimated size of the shmem index table (not a hard limit) */
-#define SHMEM_INDEX_SIZE (64)
-
-/* this is a hash bucket in the shmem index table */
-typedef struct
-{
- char key[SHMEM_INDEX_KEYSIZE]; /* string name */
- void *location; /* location in shared mem */
- Size size; /* # bytes requested for the structure */
- Size allocated_size; /* # bytes actually allocated */
-} ShmemIndexEnt;
-
#endif /* SHMEM_H */
diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list
index 174e2798443..8fda83d51f0 100644
--- a/src/tools/pgindent/typedefs.list
+++ b/src/tools/pgindent/typedefs.list
@@ -2837,9 +2837,11 @@ SharedTypmodTableEntry
Sharedsort
ShellTypeInfo
ShippableCacheEntry
-ShmemAllocatorData
ShippableCacheKey
+ShmemAllocatorData
ShmemIndexEnt
+ShmemHashDesc
+ShmemStructDesc
ShutdownForeignScan_function
ShutdownInformation
ShutdownMode
--
2.47.3
[text/x-patch] v6-0004-Convert-pg_stat_statements-to-use-the-new-interfa.patch (10.4K, 5-v6-0004-Convert-pg_stat_statements-to-use-the-new-interfa.patch)
download | inline diff:
From 855a61e67fa9c372e61971fb9ef87dbe10936780 Mon Sep 17 00:00:00 2001
From: Heikki Linnakangas <[email protected]>
Date: Fri, 13 Mar 2026 23:00:00 +0200
Subject: [PATCH v6 4/6] Convert pg_stat_statements to use the new interface
As part of this, embed the LWLock it needs in the shared memory struct
itself, so that we don't need to use RequestNamedLWLockTranche()
anymore. LWLockNewTrancheId+LWLockInitialize is more convenient to use
in extensions.
Reviewed-by: Ashutosh Bapat <[email protected]>
Reviewed-by: Zsolt Parragi <[email protected]>
Reviewed-by: Robert Haas <[email protected]>
Discussion: https://www.postgresql.org/message-id/CAExHW5vM1bneLYfg0wGeAa=52UiJ3z4vKd3AJ72X8Fw6k3KKrg@mail.gmail.com
---
.../pg_stat_statements/pg_stat_statements.c | 161 ++++++++----------
1 file changed, 67 insertions(+), 94 deletions(-)
diff --git a/contrib/pg_stat_statements/pg_stat_statements.c b/contrib/pg_stat_statements/pg_stat_statements.c
index 6cb14824ec3..8df4749a43b 100644
--- a/contrib/pg_stat_statements/pg_stat_statements.c
+++ b/contrib/pg_stat_statements/pg_stat_statements.c
@@ -249,7 +249,7 @@ typedef struct pgssEntry
*/
typedef struct pgssSharedState
{
- LWLock *lock; /* protects hashtable search/modification */
+ LWLockPadded lock; /* protects hashtable search/modification */
double cur_median_usage; /* current median usage in hashtable */
Size mean_query_len; /* current mean entry text length */
slock_t mutex; /* protects following fields only: */
@@ -259,13 +259,39 @@ typedef struct pgssSharedState
pgssGlobalStats stats; /* global statistics for pgss */
} pgssSharedState;
+/* Links to shared memory state */
+static pgssSharedState *pgss;
+static HTAB *pgss_hash;
+
+static void pgss_shmem_init(void *arg);
+
+static ShmemStructDesc pgssSharedStateShmemDesc =
+{
+ .name = "pg_stat_statements",
+ .size = sizeof(pgssSharedState),
+ .init_fn = pgss_shmem_init,
+ .ptr = (void **) &pgss,
+};
+
+static ShmemHashDesc pgssSharedHashDesc =
+{
+ .name = "pg_stat_statements hash",
+ .ptr = &pgss_hash,
+
+ .init_size = 0, /* set from 'pgss_max' */
+ .max_size = 0, /* set from 'pgss_max' */
+ .hash_info.keysize = sizeof(pgssHashKey),
+ .hash_info.entrysize = sizeof(pgssEntry),
+ .hash_flags = HASH_ELEM | HASH_BLOBS,
+};
+
+
/*---- Local variables ----*/
/* Current nesting depth of planner/ExecutorRun/ProcessUtility calls */
static int nesting_level = 0;
/* Saved hook values */
-static shmem_request_hook_type prev_shmem_request_hook = NULL;
static shmem_startup_hook_type prev_shmem_startup_hook = NULL;
static post_parse_analyze_hook_type prev_post_parse_analyze_hook = NULL;
static planner_hook_type prev_planner_hook = NULL;
@@ -275,10 +301,6 @@ static ExecutorFinish_hook_type prev_ExecutorFinish = NULL;
static ExecutorEnd_hook_type prev_ExecutorEnd = NULL;
static ProcessUtility_hook_type prev_ProcessUtility = NULL;
-/* Links to shared memory state */
-static pgssSharedState *pgss = NULL;
-static HTAB *pgss_hash = NULL;
-
/*---- GUC variables ----*/
typedef enum
@@ -331,7 +353,6 @@ PG_FUNCTION_INFO_V1(pg_stat_statements_1_13);
PG_FUNCTION_INFO_V1(pg_stat_statements);
PG_FUNCTION_INFO_V1(pg_stat_statements_info);
-static void pgss_shmem_request(void);
static void pgss_shmem_startup(void);
static void pgss_shmem_shutdown(int code, Datum arg);
static void pgss_post_parse_analyze(ParseState *pstate, Query *query,
@@ -366,7 +387,6 @@ static void pgss_store(const char *query, int64 queryId,
static void pg_stat_statements_internal(FunctionCallInfo fcinfo,
pgssVersion api_version,
bool showtext);
-static Size pgss_memsize(void);
static pgssEntry *entry_alloc(pgssHashKey *key, Size query_offset, int query_len,
int encoding, bool sticky);
static void entry_dealloc(void);
@@ -471,11 +491,18 @@ _PG_init(void)
MarkGUCPrefixReserved("pg_stat_statements");
+ /*
+ * Register our shared memory needs, including hash table
+ */
+ ShmemRegisterStruct(&pgssSharedStateShmemDesc);
+
+ pgssSharedHashDesc.init_size = pgss_max;
+ pgssSharedHashDesc.max_size = pgss_max;
+ ShmemRegisterHash(&pgssSharedHashDesc);
+
/*
* Install hooks.
*/
- prev_shmem_request_hook = shmem_request_hook;
- shmem_request_hook = pgss_shmem_request;
prev_shmem_startup_hook = shmem_startup_hook;
shmem_startup_hook = pgss_shmem_startup;
prev_post_parse_analyze_hook = post_parse_analyze_hook;
@@ -494,31 +521,31 @@ _PG_init(void)
ProcessUtility_hook = pgss_ProcessUtility;
}
-/*
- * shmem_request hook: request additional shared resources. We'll allocate or
- * attach to the shared resources in pgss_shmem_startup().
- */
static void
-pgss_shmem_request(void)
+pgss_shmem_init(void *arg)
{
- if (prev_shmem_request_hook)
- prev_shmem_request_hook();
+ int tranche_id;
- RequestAddinShmemSpace(pgss_memsize());
- RequestNamedLWLockTranche("pg_stat_statements", 1);
+ tranche_id = LWLockNewTrancheId("pg_stat_statements");
+ LWLockInitialize(&pgss->lock.lock, tranche_id);
+ pgss->cur_median_usage = ASSUMED_MEDIAN_INIT;
+ pgss->mean_query_len = ASSUMED_LENGTH_INIT;
+ SpinLockInit(&pgss->mutex);
+ pgss->extent = 0;
+ pgss->n_writers = 0;
+ pgss->gc_count = 0;
+ pgss->stats.dealloc = 0;
+ pgss->stats.stats_reset = GetCurrentTimestamp();
}
/*
- * shmem_startup hook: allocate or attach to shared memory,
- * then load any pre-existing statistics from file.
- * Also create and load the query-texts file, which is expected to exist
- * (even if empty) while the module is enabled.
+ * shmem_startup hook: Load any pre-existing statistics from file at
+ * postmaster startup. Also create and load the query-texts file, which is
+ * expected to exist (even if empty) while the module is enabled.
*/
static void
pgss_shmem_startup(void)
{
- bool found;
- HASHCTL info;
FILE *file = NULL;
FILE *qfile = NULL;
uint32 header;
@@ -531,54 +558,14 @@ pgss_shmem_startup(void)
if (prev_shmem_startup_hook)
prev_shmem_startup_hook();
- /* reset in case this is a restart within the postmaster */
- pgss = NULL;
- pgss_hash = NULL;
+ if (IsUnderPostmaster)
+ return; /* nothing to do in backends */
/*
- * Create or attach to the shared memory state, including hash table
+ * Set up a shmem exit hook to dump the statistics to disk on postmaster
+ * (or standalone backend) exit.
*/
- LWLockAcquire(AddinShmemInitLock, LW_EXCLUSIVE);
-
- pgss = ShmemInitStruct("pg_stat_statements",
- sizeof(pgssSharedState),
- &found);
-
- if (!found)
- {
- /* First time through ... */
- pgss->lock = &(GetNamedLWLockTranche("pg_stat_statements"))->lock;
- pgss->cur_median_usage = ASSUMED_MEDIAN_INIT;
- pgss->mean_query_len = ASSUMED_LENGTH_INIT;
- SpinLockInit(&pgss->mutex);
- pgss->extent = 0;
- pgss->n_writers = 0;
- pgss->gc_count = 0;
- pgss->stats.dealloc = 0;
- pgss->stats.stats_reset = GetCurrentTimestamp();
- }
-
- info.keysize = sizeof(pgssHashKey);
- info.entrysize = sizeof(pgssEntry);
- pgss_hash = ShmemInitHash("pg_stat_statements hash",
- pgss_max, pgss_max,
- &info,
- HASH_ELEM | HASH_BLOBS);
-
- LWLockRelease(AddinShmemInitLock);
-
- /*
- * If we're in the postmaster (or a standalone backend...), set up a shmem
- * exit hook to dump the statistics to disk.
- */
- if (!IsUnderPostmaster)
- on_shmem_exit(pgss_shmem_shutdown, (Datum) 0);
-
- /*
- * Done if some other process already completed our initialization.
- */
- if (found)
- return;
+ on_shmem_exit(pgss_shmem_shutdown, (Datum) 0);
/*
* Note: we don't bother with locks here, because there should be no other
@@ -1338,7 +1325,7 @@ pgss_store(const char *query, int64 queryId,
key.toplevel = (nesting_level == 0);
/* Lookup the hash table entry with shared lock. */
- LWLockAcquire(pgss->lock, LW_SHARED);
+ LWLockAcquire(&pgss->lock.lock, LW_SHARED);
entry = (pgssEntry *) hash_search(pgss_hash, &key, HASH_FIND, NULL);
@@ -1359,11 +1346,11 @@ pgss_store(const char *query, int64 queryId,
*/
if (jstate)
{
- LWLockRelease(pgss->lock);
+ LWLockRelease(&pgss->lock.lock);
norm_query = generate_normalized_query(jstate, query,
query_location,
&query_len);
- LWLockAcquire(pgss->lock, LW_SHARED);
+ LWLockAcquire(&pgss->lock.lock, LW_SHARED);
}
/* Append new query text to file with only shared lock held */
@@ -1378,8 +1365,8 @@ pgss_store(const char *query, int64 queryId,
do_gc = need_gc_qtexts();
/* Need exclusive lock to make a new hashtable entry - promote */
- LWLockRelease(pgss->lock);
- LWLockAcquire(pgss->lock, LW_EXCLUSIVE);
+ LWLockRelease(&pgss->lock.lock);
+ LWLockAcquire(&pgss->lock.lock, LW_EXCLUSIVE);
/*
* A garbage collection may have occurred while we weren't holding the
@@ -1518,7 +1505,7 @@ pgss_store(const char *query, int64 queryId,
}
done:
- LWLockRelease(pgss->lock);
+ LWLockRelease(&pgss->lock.lock);
/* We postpone this clean-up until we're out of the lock */
if (norm_query)
@@ -1807,7 +1794,7 @@ pg_stat_statements_internal(FunctionCallInfo fcinfo,
* we need to partition the hash table to limit the time spent holding any
* one lock.
*/
- LWLockAcquire(pgss->lock, LW_SHARED);
+ LWLockAcquire(&pgss->lock.lock, LW_SHARED);
if (showtext)
{
@@ -2044,7 +2031,7 @@ pg_stat_statements_internal(FunctionCallInfo fcinfo,
tuplestore_putvalues(rsinfo->setResult, rsinfo->setDesc, values, nulls);
}
- LWLockRelease(pgss->lock);
+ LWLockRelease(&pgss->lock.lock);
free(qbuffer);
}
@@ -2083,20 +2070,6 @@ pg_stat_statements_info(PG_FUNCTION_ARGS)
PG_RETURN_DATUM(HeapTupleGetDatum(heap_form_tuple(tupdesc, values, nulls)));
}
-/*
- * Estimate shared memory space needed.
- */
-static Size
-pgss_memsize(void)
-{
- Size size;
-
- size = MAXALIGN(sizeof(pgssSharedState));
- size = add_size(size, hash_estimate_size(pgss_max, sizeof(pgssEntry)));
-
- return size;
-}
-
/*
* Allocate a new hashtable entry.
* caller must hold an exclusive lock on pgss->lock
@@ -2726,7 +2699,7 @@ entry_reset(Oid userid, Oid dbid, int64 queryid, bool minmax_only)
(errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE),
errmsg("pg_stat_statements must be loaded via \"shared_preload_libraries\"")));
- LWLockAcquire(pgss->lock, LW_EXCLUSIVE);
+ LWLockAcquire(&pgss->lock.lock, LW_EXCLUSIVE);
num_entries = hash_get_num_entries(pgss_hash);
stats_reset = GetCurrentTimestamp();
@@ -2820,7 +2793,7 @@ done:
record_gc_qtexts();
release_lock:
- LWLockRelease(pgss->lock);
+ LWLockRelease(&pgss->lock.lock);
return stats_reset;
}
--
2.47.3
[text/x-patch] v6-0005-move-pgss-shmem_startup-hook-code-into-the-new-in.patch (4.0K, 6-v6-0005-move-pgss-shmem_startup-hook-code-into-the-new-in.patch)
download | inline diff:
From 5b63794f4ae24a940aef1238ba503c51b0ee393a Mon Sep 17 00:00:00 2001
From: Heikki Linnakangas <[email protected]>
Date: Thu, 19 Mar 2026 11:57:02 +0200
Subject: [PATCH v6 5/6] move pgss shmem_startup hook code into the new init_fn
callback
---
.../pg_stat_statements/pg_stat_statements.c | 56 +++++++++----------
1 file changed, 26 insertions(+), 30 deletions(-)
diff --git a/contrib/pg_stat_statements/pg_stat_statements.c b/contrib/pg_stat_statements/pg_stat_statements.c
index 8df4749a43b..8c84232f4d0 100644
--- a/contrib/pg_stat_statements/pg_stat_statements.c
+++ b/contrib/pg_stat_statements/pg_stat_statements.c
@@ -292,7 +292,6 @@ static ShmemHashDesc pgssSharedHashDesc =
static int nesting_level = 0;
/* Saved hook values */
-static shmem_startup_hook_type prev_shmem_startup_hook = NULL;
static post_parse_analyze_hook_type prev_post_parse_analyze_hook = NULL;
static planner_hook_type prev_planner_hook = NULL;
static ExecutorStart_hook_type prev_ExecutorStart = NULL;
@@ -353,7 +352,6 @@ PG_FUNCTION_INFO_V1(pg_stat_statements_1_13);
PG_FUNCTION_INFO_V1(pg_stat_statements);
PG_FUNCTION_INFO_V1(pg_stat_statements_info);
-static void pgss_shmem_startup(void);
static void pgss_shmem_shutdown(int code, Datum arg);
static void pgss_post_parse_analyze(ParseState *pstate, Query *query,
JumbleState *jstate);
@@ -492,19 +490,18 @@ _PG_init(void)
MarkGUCPrefixReserved("pg_stat_statements");
/*
- * Register our shared memory needs, including hash table
+ * Register our shared memory needs. Register the hash table first, so
+ * that it's already initialized when pgss_shmem_init() is called.
*/
- ShmemRegisterStruct(&pgssSharedStateShmemDesc);
-
pgssSharedHashDesc.init_size = pgss_max;
pgssSharedHashDesc.max_size = pgss_max;
ShmemRegisterHash(&pgssSharedHashDesc);
+ ShmemRegisterStruct(&pgssSharedStateShmemDesc);
+
/*
* Install hooks.
*/
- prev_shmem_startup_hook = shmem_startup_hook;
- shmem_startup_hook = pgss_shmem_startup;
prev_post_parse_analyze_hook = post_parse_analyze_hook;
post_parse_analyze_hook = pgss_post_parse_analyze;
prev_planner_hook = planner_hook;
@@ -521,11 +518,29 @@ _PG_init(void)
ProcessUtility_hook = pgss_ProcessUtility;
}
+/*
+ * Initialize our shared memory data structures at postmaster startup.
+ *
+ * Load any pre-existing statistics from file. Also create and load the
+ * query-texts file, which is expected to exist (even if empty) while the
+ * module is enabled.
+ */
static void
pgss_shmem_init(void *arg)
{
int tranche_id;
+ FILE *file = NULL;
+ FILE *qfile = NULL;
+ uint32 header;
+ int32 num;
+ int32 pgver;
+ int32 i;
+ int buffer_size;
+ char *buffer = NULL;
+ /*
+ * Initialize the shmem area with no statistics.
+ */
tranche_id = LWLockNewTrancheId("pg_stat_statements");
LWLockInitialize(&pgss->lock.lock, tranche_id);
pgss->cur_median_usage = ASSUMED_MEDIAN_INIT;
@@ -536,30 +551,9 @@ pgss_shmem_init(void *arg)
pgss->gc_count = 0;
pgss->stats.dealloc = 0;
pgss->stats.stats_reset = GetCurrentTimestamp();
-}
-
-/*
- * shmem_startup hook: Load any pre-existing statistics from file at
- * postmaster startup. Also create and load the query-texts file, which is
- * expected to exist (even if empty) while the module is enabled.
- */
-static void
-pgss_shmem_startup(void)
-{
- FILE *file = NULL;
- FILE *qfile = NULL;
- uint32 header;
- int32 num;
- int32 pgver;
- int32 i;
- int buffer_size;
- char *buffer = NULL;
- if (prev_shmem_startup_hook)
- prev_shmem_startup_hook();
-
- if (IsUnderPostmaster)
- return; /* nothing to do in backends */
+ /* The hash table must be initialized already */
+ Assert(pgss_hash != NULL);
/*
* Set up a shmem exit hook to dump the statistics to disk on postmaster
@@ -568,6 +562,8 @@ pgss_shmem_startup(void)
on_shmem_exit(pgss_shmem_shutdown, (Datum) 0);
/*
+ * Load any pre-existing statistics from file.
+ *
* Note: we don't bother with locks here, because there should be no other
* processes running when this code is reached.
*/
--
2.47.3
[text/x-patch] v6-0006-Use-the-new-mechanism-in-a-few-core-subsystems.patch (40.8K, 7-v6-0006-Use-the-new-mechanism-in-a-few-core-subsystems.patch)
download | inline diff:
From 7160ae00657d5e40e74b3b0f31de14435db73190 Mon Sep 17 00:00:00 2001
From: Heikki Linnakangas <[email protected]>
Date: Fri, 13 Mar 2026 23:01:44 +0200
Subject: [PATCH v6 6/6] Use the new mechanism in a few core subsystems
I chose these subsystems specifically because they have some
complicating properties, making them slightly harder to convert than
most:
- The initialization callbacks of some of these subsystems have
dependencies, i.e. they need to be initialized in the right order.
- The ProgGlobal pointer still needs to be inherited by the
BackendParameters mechanism on EXEC_BACKEND builds, because
ProcGlobal is required by InitProcess() to get a PGPROC entry, and
the PGPROC entry is required to use LWLocks, and usually attaching
to shared memory areas requires the use of LWLocks.
- Similarly, ProcSignal pointer still needs to be handled by
BackendParameters, because query cancellation connections access it
without calling InitProcess
I'm believe converting all the rest of the subsystems after this will
be pretty mechanic.
Reviewed-by: Ashutosh Bapat <[email protected]>
Reviewed-by: Zsolt Parragi <[email protected]>
Discussion: https://www.postgresql.org/message-id/CAExHW5vM1bneLYfg0wGeAa=52UiJ3z4vKd3AJ72X8Fw6k3KKrg@mail.gmail.com
---
src/backend/access/transam/varsup.c | 33 ++---
src/backend/storage/ipc/dsm.c | 61 ++++----
src/backend/storage/ipc/dsm_registry.c | 34 ++---
src/backend/storage/ipc/ipci.c | 50 +++----
src/backend/storage/ipc/pmsignal.c | 49 ++++---
src/backend/storage/ipc/procarray.c | 122 ++++++++--------
src/backend/storage/ipc/procsignal.c | 63 ++++-----
src/backend/storage/ipc/sinvaladt.c | 36 ++---
src/backend/storage/lmgr/proc.c | 178 ++++++++++++------------
src/backend/utils/activity/wait_event.c | 94 ++++++-------
src/include/access/transam.h | 3 +-
src/include/storage/dsm.h | 3 +-
src/include/storage/dsm_registry.h | 3 +-
src/include/storage/pmsignal.h | 3 +-
src/include/storage/proc.h | 2 +-
src/include/storage/procarray.h | 3 +-
src/include/storage/procsignal.h | 3 +-
src/include/storage/sinvaladt.h | 3 +-
src/include/utils/wait_event.h | 3 +-
19 files changed, 366 insertions(+), 380 deletions(-)
diff --git a/src/backend/access/transam/varsup.c b/src/backend/access/transam/varsup.c
index 3e95d4cfd16..84e6c90f4fa 100644
--- a/src/backend/access/transam/varsup.c
+++ b/src/backend/access/transam/varsup.c
@@ -30,35 +30,32 @@
/* Number of OIDs to prefetch (preallocate) per XLOG write */
#define VAR_OID_PREFETCH 8192
+static void VarsupShmemInit(void *arg);
+
/* pointer to variables struct in shared memory */
TransamVariablesData *TransamVariables = NULL;
+static ShmemStructDesc TransamVariablesShmemDesc = {
+ .name = "TransamVariables",
+ .size = sizeof(TransamVariablesData),
+ .init_fn = VarsupShmemInit,
+ .ptr = (void **) &TransamVariables,
+};
/*
* Initialization of shared memory for TransamVariables.
*/
-Size
-VarsupShmemSize(void)
+void
+VarsupShmemRegister(void)
{
- return sizeof(TransamVariablesData);
+ ShmemRegisterStruct(&TransamVariablesShmemDesc);
}
-void
-VarsupShmemInit(void)
-{
- bool found;
+static void
+VarsupShmemInit(void *arg)
- /* Initialize our shared state struct */
- TransamVariables = ShmemInitStruct("TransamVariables",
- sizeof(TransamVariablesData),
- &found);
- if (!IsUnderPostmaster)
- {
- Assert(!found);
- memset(TransamVariables, 0, sizeof(TransamVariablesData));
- }
- else
- Assert(found);
+{
+ memset(TransamVariables, 0, sizeof(TransamVariablesData));
}
/*
diff --git a/src/backend/storage/ipc/dsm.c b/src/backend/storage/ipc/dsm.c
index 6a5b16392f7..52b4a1e017f 100644
--- a/src/backend/storage/ipc/dsm.c
+++ b/src/backend/storage/ipc/dsm.c
@@ -110,6 +110,15 @@ static bool dsm_init_done = false;
/* Preallocated DSM space in the main shared memory region. */
static void *dsm_main_space_begin = NULL;
+static void dsm_main_space_init(void *);
+
+static ShmemStructDesc dsm_main_space_shmem_desc = {
+ .name = "Preallocated DSM",
+ .size = 0, /* calculated later */
+ .init_fn = dsm_main_space_init,
+ .ptr = &dsm_main_space_begin,
+};
+
/*
* List of dynamic shared memory segments used by this backend.
*
@@ -464,42 +473,36 @@ dsm_set_control_handle(dsm_handle h)
#endif
/*
- * Reserve some space in the main shared memory segment for DSM segments.
- */
-size_t
-dsm_estimate_size(void)
-{
- return 1024 * 1024 * (size_t) min_dynamic_shared_memory;
-}
-
-/*
- * Initialize space in the main shared memory segment for DSM segments.
+ * Reserve space in the main shared memory segment for DSM segments.
*/
void
-dsm_shmem_init(void)
+dsm_shmem_register(void)
{
- size_t size = dsm_estimate_size();
- bool found;
+ size_t size = 1024 * 1024 * (size_t) min_dynamic_shared_memory;
if (size == 0)
return;
- dsm_main_space_begin = ShmemInitStruct("Preallocated DSM", size, &found);
- if (!found)
- {
- FreePageManager *fpm = (FreePageManager *) dsm_main_space_begin;
- size_t first_page = 0;
- size_t pages;
-
- /* Reserve space for the FreePageManager. */
- while (first_page * FPM_PAGE_SIZE < sizeof(FreePageManager))
- ++first_page;
-
- /* Initialize it and give it all the rest of the space. */
- FreePageManagerInitialize(fpm, dsm_main_space_begin);
- pages = (size / FPM_PAGE_SIZE) - first_page;
- FreePageManagerPut(fpm, first_page, pages);
- }
+ dsm_main_space_shmem_desc.size = size;
+ ShmemRegisterStruct(&dsm_main_space_shmem_desc);
+}
+
+static void
+dsm_main_space_init(void *arg)
+{
+ size_t size = dsm_main_space_shmem_desc.size;
+ FreePageManager *fpm = (FreePageManager *) dsm_main_space_begin;
+ size_t first_page = 0;
+ size_t pages;
+
+ /* Reserve space for the FreePageManager. */
+ while (first_page * FPM_PAGE_SIZE < sizeof(FreePageManager))
+ ++first_page;
+
+ /* Initialize it and give it all the rest of the space. */
+ FreePageManagerInitialize(fpm, dsm_main_space_begin);
+ pages = (size / FPM_PAGE_SIZE) - first_page;
+ FreePageManagerPut(fpm, first_page, pages);
}
/*
diff --git a/src/backend/storage/ipc/dsm_registry.c b/src/backend/storage/ipc/dsm_registry.c
index 9bfcd616827..080e30648cc 100644
--- a/src/backend/storage/ipc/dsm_registry.c
+++ b/src/backend/storage/ipc/dsm_registry.c
@@ -57,6 +57,16 @@ typedef struct DSMRegistryCtxStruct
static DSMRegistryCtxStruct *DSMRegistryCtx;
+static void DSMRegistryCtxShmemInit(void *arg);
+
+static ShmemStructDesc DSMRegistryCtxShmemDesc = {
+ .name = "DSM Registry Data",
+ .size = sizeof(DSMRegistryCtxStruct),
+ .init_fn = DSMRegistryCtxShmemInit,
+ .ptr = (void **) &DSMRegistryCtx,
+};
+
+
typedef struct NamedDSMState
{
dsm_handle handle;
@@ -114,27 +124,17 @@ static const dshash_parameters dsh_params = {
static dsa_area *dsm_registry_dsa;
static dshash_table *dsm_registry_table;
-Size
-DSMRegistryShmemSize(void)
+void
+DSMRegistryShmemRegister(void)
{
- return MAXALIGN(sizeof(DSMRegistryCtxStruct));
+ ShmemRegisterStruct(&DSMRegistryCtxShmemDesc);
}
-void
-DSMRegistryShmemInit(void)
+static void
+DSMRegistryCtxShmemInit(void *arg)
{
- bool found;
-
- DSMRegistryCtx = (DSMRegistryCtxStruct *)
- ShmemInitStruct("DSM Registry Data",
- DSMRegistryShmemSize(),
- &found);
-
- if (!found)
- {
- DSMRegistryCtx->dsah = DSA_HANDLE_INVALID;
- DSMRegistryCtx->dshh = DSHASH_HANDLE_INVALID;
- }
+ DSMRegistryCtx->dsah = DSA_HANDLE_INVALID;
+ DSMRegistryCtx->dshh = DSHASH_HANDLE_INVALID;
}
/*
diff --git a/src/backend/storage/ipc/ipci.c b/src/backend/storage/ipc/ipci.c
index 405c69655f0..40ceeddae58 100644
--- a/src/backend/storage/ipc/ipci.c
+++ b/src/backend/storage/ipc/ipci.c
@@ -101,16 +101,11 @@ CalculateShmemSize(void)
size = 100000;
size = add_size(size, ShmemRegisteredSize());
- size = add_size(size, dsm_estimate_size());
- size = add_size(size, DSMRegistryShmemSize());
-
/* legacy subsystems */
size = add_size(size, BufferManagerShmemSize());
size = add_size(size, LockManagerShmemSize());
size = add_size(size, PredicateLockShmemSize());
- size = add_size(size, ProcGlobalShmemSize());
size = add_size(size, XLogPrefetchShmemSize());
- size = add_size(size, VarsupShmemSize());
size = add_size(size, XLOGShmemSize());
size = add_size(size, XLogRecoveryShmemSize());
size = add_size(size, CLOGShmemSize());
@@ -120,11 +115,7 @@ CalculateShmemSize(void)
size = add_size(size, BackgroundWorkerShmemSize());
size = add_size(size, MultiXactShmemSize());
size = add_size(size, LWLockShmemSize());
- size = add_size(size, ProcArrayShmemSize());
size = add_size(size, BackendStatusShmemSize());
- size = add_size(size, SharedInvalShmemSize());
- size = add_size(size, PMSignalShmemSize());
- size = add_size(size, ProcSignalShmemSize());
size = add_size(size, CheckpointerShmemSize());
size = add_size(size, AutoVacuumShmemSize());
size = add_size(size, ReplicationSlotsShmemSize());
@@ -138,7 +129,6 @@ CalculateShmemSize(void)
size = add_size(size, SyncScanShmemSize());
size = add_size(size, AsyncShmemSize());
size = add_size(size, StatsShmemSize());
- size = add_size(size, WaitEventCustomShmemSize());
size = add_size(size, InjectionPointShmemSize());
size = add_size(size, SlotSyncShmemSize());
size = add_size(size, AioShmemSize());
@@ -246,11 +236,28 @@ void
RegisterShmemStructs(void)
{
/*
- * TODO: Not used in any built-in subsystems yet. In the future, most of
- * the calls *ShmemInit() calls in CreateOrAttachShmemStructs(), and
- * *ShmemSize() calls in CalculateShmemSize() will be replaced by calls
- * into the subsystems from here.
+ * TODO: In the future, all the calls *ShmemInit() calls in
+ * CreateOrAttachShmemStructs(), and all the *ShmemSize() calls in
+ * CalculateShmemSize() will be replaced by calls into the subsystems from
+ * here.
+ */
+
+ /*
+ * Note: there are some inter-dependencies between these, so the order of
+ * some of these matter.
*/
+
+ DSMRegistryShmemRegister();
+ dsm_shmem_register();
+
+ ProcGlobalShmemRegister();
+ VarsupShmemRegister();
+ ProcArrayShmemRegister();
+ SharedInvalShmemRegister();
+ PMSignalShmemRegister();
+ ProcSignalShmemRegister();
+
+ WaitEventCustomShmemRegister();
}
/*
@@ -292,13 +299,9 @@ CreateOrAttachShmemStructs(void)
ShmemInitRegistered();
}
- dsm_shmem_init();
- DSMRegistryShmemInit();
-
/*
* Set up xlog, clog, and buffers
*/
- VarsupShmemInit();
XLOGShmemInit();
XLogPrefetchShmemInit();
XLogRecoveryShmemInit();
@@ -321,23 +324,13 @@ CreateOrAttachShmemStructs(void)
/*
* Set up process table
*/
- if (!IsUnderPostmaster)
- InitProcGlobal();
- ProcArrayShmemInit();
BackendStatusShmemInit();
TwoPhaseShmemInit();
BackgroundWorkerShmemInit();
- /*
- * Set up shared-inval messaging
- */
- SharedInvalShmemInit();
-
/*
* Set up interprocess signaling mechanisms
*/
- PMSignalShmemInit();
- ProcSignalShmemInit();
CheckpointerShmemInit();
AutoVacuumShmemInit();
ReplicationSlotsShmemInit();
@@ -356,7 +349,6 @@ CreateOrAttachShmemStructs(void)
SyncScanShmemInit();
AsyncShmemInit();
StatsShmemInit();
- WaitEventCustomShmemInit();
InjectionPointShmemInit();
AioShmemInit();
WaitLSNShmemInit();
diff --git a/src/backend/storage/ipc/pmsignal.c b/src/backend/storage/ipc/pmsignal.c
index 4618820b337..d7463df73ab 100644
--- a/src/backend/storage/ipc/pmsignal.c
+++ b/src/backend/storage/ipc/pmsignal.c
@@ -83,6 +83,15 @@ struct PMSignalData
/* PMSignalState pointer is valid in both postmaster and child processes */
NON_EXEC_STATIC volatile PMSignalData *PMSignalState = NULL;
+static void PMSignalShmemInit(void *);
+
+static ShmemStructDesc PMSignalShmemDesc = {
+ .name = "PMSignalState",
+ .size = 0, /* calculated later */
+ .init_fn = PMSignalShmemInit,
+ .ptr = (void **) &PMSignalState,
+};
+
/*
* Local copy of PMSignalState->num_child_flags, only valid in the
* postmaster. Postmaster keeps a local copy so that it doesn't need to
@@ -123,39 +132,29 @@ postmaster_death_handler(SIGNAL_ARGS)
static void MarkPostmasterChildInactive(int code, Datum arg);
/*
- * PMSignalShmemSize
- * Compute space needed for pmsignal.c's shared memory
+ * PMSignalShmemRegister - Register pmsignal.c's shared memory needs
*/
-Size
-PMSignalShmemSize(void)
+void
+PMSignalShmemRegister(void)
{
Size size;
- size = offsetof(PMSignalData, PMChildFlags);
- size = add_size(size, mul_size(MaxLivePostmasterChildren(),
- sizeof(sig_atomic_t)));
+ num_child_flags = MaxLivePostmasterChildren();
- return size;
+ size = offsetof(PMSignalData, PMChildFlags);
+ size = add_size(size, mul_size(num_child_flags, sizeof(sig_atomic_t)));
+ PMSignalShmemDesc.size = size;
+ ShmemRegisterStruct(&PMSignalShmemDesc);
}
-/*
- * PMSignalShmemInit - initialize during shared-memory creation
- */
-void
-PMSignalShmemInit(void)
+static void
+PMSignalShmemInit(void *arg)
{
- bool found;
-
- PMSignalState = (PMSignalData *)
- ShmemInitStruct("PMSignalState", PMSignalShmemSize(), &found);
-
- if (!found)
- {
- /* initialize all flags to zeroes */
- MemSet(unvolatize(PMSignalData *, PMSignalState), 0, PMSignalShmemSize());
- num_child_flags = MaxLivePostmasterChildren();
- PMSignalState->num_child_flags = num_child_flags;
- }
+ /* initialize all flags to zeroes */
+ Assert(PMSignalState);
+ MemSet(unvolatize(PMSignalData *, PMSignalState), 0, PMSignalShmemDesc.size);
+ Assert(num_child_flags > 0);
+ PMSignalState->num_child_flags = num_child_flags;
}
/*
diff --git a/src/backend/storage/ipc/procarray.c b/src/backend/storage/ipc/procarray.c
index 0f913897acc..5daf33d5323 100644
--- a/src/backend/storage/ipc/procarray.c
+++ b/src/backend/storage/ipc/procarray.c
@@ -103,6 +103,19 @@ typedef struct ProcArrayStruct
int pgprocnos[FLEXIBLE_ARRAY_MEMBER];
} ProcArrayStruct;
+static void ProcArrayShmemInit(void *arg);
+static void ProcArrayShmemAttach(void *arg);
+
+static ProcArrayStruct *procArray;
+
+static ShmemStructDesc ProcArrayShmemDesc = {
+ .name = "Proc Array",
+ .size = 0, /* calculated later */
+ .init_fn = ProcArrayShmemInit,
+ .attach_fn = ProcArrayShmemAttach,
+ .ptr = (void **) &procArray,
+};
+
/*
* State for the GlobalVisTest* family of functions. Those functions can
* e.g. be used to decide if a deleted row can be removed without violating
@@ -269,9 +282,6 @@ typedef enum KAXCompressReason
KAX_STARTUP_PROCESS_IDLE, /* startup process is about to sleep */
} KAXCompressReason;
-
-static ProcArrayStruct *procArray;
-
static PGPROC *allProcs;
/*
@@ -282,8 +292,25 @@ static TransactionId cachedXidIsNotInProgress = InvalidTransactionId;
/*
* Bookkeeping for tracking emulated transactions in recovery
*/
+
static TransactionId *KnownAssignedXids;
+
+static ShmemStructDesc KnownAssignedXidsShmemDesc = {
+ .name = "KnownAssignedXids",
+ .size = 0, /* calculated later */
+ .init_fn = NULL, /* no initialization needed */
+ .ptr = (void **) &KnownAssignedXids,
+};
+
static bool *KnownAssignedXidsValid;
+
+static ShmemStructDesc KnownAssignedXidsValidShmemDesc = {
+ .name = "KnownAssignedXidsValid",
+ .size = 0, /* calculated later */
+ .init_fn = NULL, /* no initialization needed */
+ .ptr = (void **) &KnownAssignedXidsValid,
+};
+
static TransactionId latestObservedXid = InvalidTransactionId;
/*
@@ -374,18 +401,18 @@ static inline FullTransactionId FullXidRelativeTo(FullTransactionId rel,
static void GlobalVisUpdateApply(ComputeXidHorizonsResult *horizons);
/*
- * Report shared-memory space needed by ProcArrayShmemInit
+ * Register the shared PGPROC array during postmaster startup.
*/
-Size
-ProcArrayShmemSize(void)
+void
+ProcArrayShmemRegister(void)
{
- Size size;
-
- /* Size of the ProcArray structure itself */
#define PROCARRAY_MAXPROCS (MaxBackends + max_prepared_xacts)
- size = offsetof(ProcArrayStruct, pgprocnos);
- size = add_size(size, mul_size(sizeof(int), PROCARRAY_MAXPROCS));
+ /* Register the ProcArray shared structure */
+ ProcArrayShmemDesc.size =
+ add_size(offsetof(ProcArrayStruct, pgprocnos),
+ mul_size(sizeof(int), PROCARRAY_MAXPROCS));
+ ShmemRegisterStruct(&ProcArrayShmemDesc);
/*
* During Hot Standby processing we have a data structure called
@@ -405,64 +432,41 @@ ProcArrayShmemSize(void)
if (EnableHotStandby)
{
- size = add_size(size,
- mul_size(sizeof(TransactionId),
- TOTAL_MAX_CACHED_SUBXIDS));
- size = add_size(size,
- mul_size(sizeof(bool), TOTAL_MAX_CACHED_SUBXIDS));
+ KnownAssignedXidsShmemDesc.size =
+ mul_size(sizeof(TransactionId),
+ TOTAL_MAX_CACHED_SUBXIDS);
+ ShmemRegisterStruct(&KnownAssignedXidsShmemDesc);
+
+ KnownAssignedXidsValidShmemDesc.size =
+ mul_size(sizeof(bool), TOTAL_MAX_CACHED_SUBXIDS);
+ ShmemRegisterStruct(&KnownAssignedXidsValidShmemDesc);
}
-
- return size;
}
/*
* Initialize the shared PGPROC array during postmaster startup.
*/
-void
-ProcArrayShmemInit(void)
+static void
+ProcArrayShmemInit(void *arg)
{
- bool found;
-
- /* Create or attach to the ProcArray shared structure */
- procArray = (ProcArrayStruct *)
- ShmemInitStruct("Proc Array",
- add_size(offsetof(ProcArrayStruct, pgprocnos),
- mul_size(sizeof(int),
- PROCARRAY_MAXPROCS)),
- &found);
-
- if (!found)
- {
- /*
- * We're the first - initialize.
- */
- procArray->numProcs = 0;
- procArray->maxProcs = PROCARRAY_MAXPROCS;
- procArray->maxKnownAssignedXids = TOTAL_MAX_CACHED_SUBXIDS;
- procArray->numKnownAssignedXids = 0;
- procArray->tailKnownAssignedXids = 0;
- procArray->headKnownAssignedXids = 0;
- procArray->lastOverflowedXid = InvalidTransactionId;
- procArray->replication_slot_xmin = InvalidTransactionId;
- procArray->replication_slot_catalog_xmin = InvalidTransactionId;
- TransamVariables->xactCompletionCount = 1;
- }
+ procArray->numProcs = 0;
+ procArray->maxProcs = PROCARRAY_MAXPROCS;
+ procArray->maxKnownAssignedXids = TOTAL_MAX_CACHED_SUBXIDS;
+ procArray->numKnownAssignedXids = 0;
+ procArray->tailKnownAssignedXids = 0;
+ procArray->headKnownAssignedXids = 0;
+ procArray->lastOverflowedXid = InvalidTransactionId;
+ procArray->replication_slot_xmin = InvalidTransactionId;
+ procArray->replication_slot_catalog_xmin = InvalidTransactionId;
+ TransamVariables->xactCompletionCount = 1;
allProcs = ProcGlobal->allProcs;
+}
- /* Create or attach to the KnownAssignedXids arrays too, if needed */
- if (EnableHotStandby)
- {
- KnownAssignedXids = (TransactionId *)
- ShmemInitStruct("KnownAssignedXids",
- mul_size(sizeof(TransactionId),
- TOTAL_MAX_CACHED_SUBXIDS),
- &found);
- KnownAssignedXidsValid = (bool *)
- ShmemInitStruct("KnownAssignedXidsValid",
- mul_size(sizeof(bool), TOTAL_MAX_CACHED_SUBXIDS),
- &found);
- }
+static void
+ProcArrayShmemAttach(void *arg)
+{
+ allProcs = ProcGlobal->allProcs;
}
/*
diff --git a/src/backend/storage/ipc/procsignal.c b/src/backend/storage/ipc/procsignal.c
index 7e017c8d53b..a0c94ddd77c 100644
--- a/src/backend/storage/ipc/procsignal.c
+++ b/src/backend/storage/ipc/procsignal.c
@@ -105,7 +105,18 @@ struct ProcSignalHeader
#define BARRIER_CLEAR_BIT(flags, type) \
((flags) &= ~(((uint32) 1) << (uint32) (type)))
+static void ProcSignalShmemInit(void *arg);
+
NON_EXEC_STATIC ProcSignalHeader *ProcSignal = NULL;
+
+static ShmemStructDesc ProcSignalShmemDesc = {
+ .name = "ProcSignal",
+ .size = 0, /* calculated later */
+ .init_fn = ProcSignalShmemInit,
+ .ptr = (void **) &ProcSignal,
+};
+
+
static ProcSignalSlot *MyProcSignalSlot = NULL;
static bool CheckProcSignal(ProcSignalReason reason);
@@ -113,51 +124,37 @@ static void CleanupProcSignalState(int status, Datum arg);
static void ResetProcSignalBarrierBits(uint32 flags);
/*
- * ProcSignalShmemSize
- * Compute space needed for ProcSignal's shared memory
+ * ProcSignalShmemRegister
+ * Register ProcSignal's shared memory needs at postmaster startup
*/
-Size
-ProcSignalShmemSize(void)
+void
+ProcSignalShmemRegister(void)
{
Size size;
size = mul_size(NumProcSignalSlots, sizeof(ProcSignalSlot));
size = add_size(size, offsetof(ProcSignalHeader, psh_slot));
- return size;
+
+ ProcSignalShmemDesc.size = size;
+ ShmemRegisterStruct(&ProcSignalShmemDesc);
}
-/*
- * ProcSignalShmemInit
- * Allocate and initialize ProcSignal's shared memory
- */
-void
-ProcSignalShmemInit(void)
+static void
+ProcSignalShmemInit(void *arg)
{
- Size size = ProcSignalShmemSize();
- bool found;
+ pg_atomic_init_u64(&ProcSignal->psh_barrierGeneration, 0);
- ProcSignal = (ProcSignalHeader *)
- ShmemInitStruct("ProcSignal", size, &found);
-
- /* If we're first, initialize. */
- if (!found)
+ for (int i = 0; i < NumProcSignalSlots; ++i)
{
- int i;
-
- pg_atomic_init_u64(&ProcSignal->psh_barrierGeneration, 0);
+ ProcSignalSlot *slot = &ProcSignal->psh_slot[i];
- for (i = 0; i < NumProcSignalSlots; ++i)
- {
- ProcSignalSlot *slot = &ProcSignal->psh_slot[i];
-
- SpinLockInit(&slot->pss_mutex);
- pg_atomic_init_u32(&slot->pss_pid, 0);
- slot->pss_cancel_key_len = 0;
- MemSet(slot->pss_signalFlags, 0, sizeof(slot->pss_signalFlags));
- pg_atomic_init_u64(&slot->pss_barrierGeneration, PG_UINT64_MAX);
- pg_atomic_init_u32(&slot->pss_barrierCheckMask, 0);
- ConditionVariableInit(&slot->pss_barrierCV);
- }
+ SpinLockInit(&slot->pss_mutex);
+ pg_atomic_init_u32(&slot->pss_pid, 0);
+ slot->pss_cancel_key_len = 0;
+ MemSet(slot->pss_signalFlags, 0, sizeof(slot->pss_signalFlags));
+ pg_atomic_init_u64(&slot->pss_barrierGeneration, PG_UINT64_MAX);
+ pg_atomic_init_u32(&slot->pss_barrierCheckMask, 0);
+ ConditionVariableInit(&slot->pss_barrierCV);
}
}
diff --git a/src/backend/storage/ipc/sinvaladt.c b/src/backend/storage/ipc/sinvaladt.c
index a7a7cc4f0a9..b9df3b84de9 100644
--- a/src/backend/storage/ipc/sinvaladt.c
+++ b/src/backend/storage/ipc/sinvaladt.c
@@ -203,8 +203,17 @@ typedef struct SISeg
*/
#define NumProcStateSlots (MaxBackends + NUM_AUXILIARY_PROCS)
+static void SharedInvalShmemInit(void *arg);
+
static SISeg *shmInvalBuffer; /* pointer to the shared inval buffer */
+static ShmemStructDesc SharedInvalShmemDesc = {
+ .name = "shmInvalBuffer",
+ .size = 0, /* calculated later */
+ .init_fn = SharedInvalShmemInit,
+ .ptr = (void **) &shmInvalBuffer,
+};
+
static LocalTransactionId nextLocalTransactionId;
@@ -212,10 +221,11 @@ static void CleanupInvalidationState(int status, Datum arg);
/*
- * SharedInvalShmemSize --- return shared-memory space needed
+ * SharedInvalShmemRegister
+ * Register shared memory needs for the SI message buffer
*/
-Size
-SharedInvalShmemSize(void)
+void
+SharedInvalShmemRegister(void)
{
Size size;
@@ -223,26 +233,16 @@ SharedInvalShmemSize(void)
size = add_size(size, mul_size(sizeof(ProcState), NumProcStateSlots)); /* procState */
size = add_size(size, mul_size(sizeof(int), NumProcStateSlots)); /* pgprocnos */
- return size;
+ SharedInvalShmemDesc.size = size;
+ ShmemRegisterStruct(&SharedInvalShmemDesc);
}
-/*
- * SharedInvalShmemInit
- * Create and initialize the SI message buffer
- */
-void
-SharedInvalShmemInit(void)
+static void
+SharedInvalShmemInit(void *arg)
{
int i;
- bool found;
-
- /* Allocate space in shared memory */
- shmInvalBuffer = (SISeg *)
- ShmemInitStruct("shmInvalBuffer", SharedInvalShmemSize(), &found);
- if (found)
- return;
- /* Clear message counters, save size of procState array, init spinlock */
+ /* Clear message counters, init spinlock */
shmInvalBuffer->minMsgNum = 0;
shmInvalBuffer->maxMsgNum = 0;
shmInvalBuffer->nextThreshold = CLEANUP_MIN;
diff --git a/src/backend/storage/lmgr/proc.c b/src/backend/storage/lmgr/proc.c
index 8f5ce0e2a8a..f880aba9be9 100644
--- a/src/backend/storage/lmgr/proc.c
+++ b/src/backend/storage/lmgr/proc.c
@@ -69,9 +69,41 @@ PGPROC *MyProc = NULL;
/* Pointers to shared-memory structures */
PROC_HDR *ProcGlobal = NULL;
+static void *tmpAllProcs;
+static void *tmpFastPathLockArray;
NON_EXEC_STATIC PGPROC *AuxiliaryProcs = NULL;
PGPROC *PreparedXactProcs = NULL;
+static void ProcGlobalShmemInit(void *arg);
+
+static ShmemStructDesc ProcGlobalShmemDesc = {
+ .name = "Proc Header",
+ .size = sizeof(PROC_HDR),
+ .init_fn = ProcGlobalShmemInit,
+
+ /*
+ * ProcGlobal is registered here in .ptr as usual, but it needs to be
+ * propagated specially in EXEC_BACKEND mode, because ProcGlobal needs to
+ * be accessed early at backend startup, before ShmemAttachRegistered()
+ * has been called.
+ */
+ .ptr = (void **) &ProcGlobal,
+};
+
+static ShmemStructDesc ProcGlobalAllProcsShmemDesc = {
+ .name = "PGPROC structures",
+ .size = 0, /* calculated later */
+ .ptr = (void **) &tmpAllProcs,
+};
+
+static ShmemStructDesc FastPathLockArrayShmemDesc = {
+ .name = "Fast-Path Lock Array",
+ .size = 0, /* calculated later */
+ .ptr = (void **) &tmpFastPathLockArray,
+};
+
+static uint32 TotalProcs;
+
/* Is a deadlock check pending? */
static volatile sig_atomic_t got_deadlock_timeout;
@@ -81,24 +113,6 @@ static void AuxiliaryProcKill(int code, Datum arg);
static DeadLockState CheckDeadLock(void);
-/*
- * Report shared-memory space needed by PGPROC.
- */
-static Size
-PGProcShmemSize(void)
-{
- Size size = 0;
- Size TotalProcs =
- add_size(MaxBackends, add_size(NUM_AUXILIARY_PROCS, max_prepared_xacts));
-
- size = add_size(size, mul_size(TotalProcs, sizeof(PGPROC)));
- size = add_size(size, mul_size(TotalProcs, sizeof(*ProcGlobal->xids)));
- size = add_size(size, mul_size(TotalProcs, sizeof(*ProcGlobal->subxidStates)));
- size = add_size(size, mul_size(TotalProcs, sizeof(*ProcGlobal->statusFlags)));
-
- return size;
-}
-
/*
* Report shared-memory space needed by Fast-Path locks.
*/
@@ -106,8 +120,6 @@ static Size
FastPathLockShmemSize(void)
{
Size size = 0;
- Size TotalProcs =
- add_size(MaxBackends, add_size(NUM_AUXILIARY_PROCS, max_prepared_xacts));
Size fpLockBitsSize,
fpRelIdSize;
@@ -123,25 +135,6 @@ FastPathLockShmemSize(void)
return size;
}
-/*
- * Report shared-memory space needed by InitProcGlobal.
- */
-Size
-ProcGlobalShmemSize(void)
-{
- Size size = 0;
-
- /* ProcGlobal */
- size = add_size(size, sizeof(PROC_HDR));
- size = add_size(size, sizeof(slock_t));
-
- size = add_size(size, PGSemaphoreShmemSize(ProcGlobalSemas()));
- size = add_size(size, PGProcShmemSize());
- size = add_size(size, FastPathLockShmemSize());
-
- return size;
-}
-
/*
* Report number of semaphores needed by InitProcGlobal.
*/
@@ -156,7 +149,50 @@ ProcGlobalSemas(void)
}
/*
- * InitProcGlobal -
+ * ProcGlobalShmemRegister -
+ * Register shared memory needs.
+ *
+ * This is called during postmaster or standalone backend startup, and also
+ * during backend startup in EXEC_BACKEND mode.
+ */
+void
+ProcGlobalShmemRegister(void)
+{
+ Size size = 0;
+
+ /*
+ * Reserve all the PGPROC structures we'll need. There are six separate
+ * consumers: (1) normal backends, (2) autovacuum workers and special
+ * workers, (3) background workers, (4) walsenders, (5) auxiliary
+ * processes, and (6) prepared transactions. (For largely-historical
+ * reasons, we combine autovacuum and special workers into one category
+ * with a single freelist.) Each PGPROC structure is dedicated to exactly
+ * one of these purposes, and they do not move between groups.
+ */
+ TotalProcs =
+ add_size(MaxBackends, add_size(NUM_AUXILIARY_PROCS, max_prepared_xacts));
+
+ size = add_size(size, mul_size(TotalProcs, sizeof(PGPROC)));
+ size = add_size(size, mul_size(TotalProcs, sizeof(*ProcGlobal->xids)));
+ size = add_size(size, mul_size(TotalProcs, sizeof(*ProcGlobal->subxidStates)));
+ size = add_size(size, mul_size(TotalProcs, sizeof(*ProcGlobal->statusFlags)));
+
+ ProcGlobalAllProcsShmemDesc.size = size;
+ ShmemRegisterStruct(&ProcGlobalAllProcsShmemDesc);
+
+ FastPathLockArrayShmemDesc.size = FastPathLockShmemSize();
+ ShmemRegisterStruct(&FastPathLockArrayShmemDesc);
+
+ /*
+ * Register the ProcGlobal shared structure last. Its init callback
+ * initializes the others too.
+ */
+ ShmemRegisterStruct(&ProcGlobalShmemDesc);
+}
+
+
+/*
+ * ProcGlobalShmemInit -
* Initialize the global process table during postmaster or standalone
* backend startup.
*
@@ -175,36 +211,23 @@ ProcGlobalSemas(void)
* Another reason for creating semaphores here is that the semaphore
* implementation typically requires us to create semaphores in the
* postmaster, not in backends.
- *
- * Note: this is NOT called by individual backends under a postmaster,
- * not even in the EXEC_BACKEND case. The ProcGlobal and AuxiliaryProcs
- * pointers must be propagated specially for EXEC_BACKEND operation.
*/
-void
-InitProcGlobal(void)
+static void
+ProcGlobalShmemInit(void *arg)
{
+ char *ptr;
+ size_t requestSize;
PGPROC *procs;
int i,
j;
- bool found;
- uint32 TotalProcs = MaxBackends + NUM_AUXILIARY_PROCS + max_prepared_xacts;
/* Used for setup of per-backend fast-path slots. */
char *fpPtr,
*fpEndPtr PG_USED_FOR_ASSERTS_ONLY;
Size fpLockBitsSize,
fpRelIdSize;
- Size requestSize;
- char *ptr;
-
- /* Create the ProcGlobal shared structure */
- ProcGlobal = (PROC_HDR *)
- ShmemInitStruct("Proc Header", sizeof(PROC_HDR), &found);
- Assert(!found);
- /*
- * Initialize the data structures.
- */
+ Assert(ProcGlobal);
ProcGlobal->spins_per_delay = DEFAULT_SPINS_PER_DELAY;
SpinLockInit(&ProcGlobal->freeProcsLock);
dlist_init(&ProcGlobal->freeProcs);
@@ -217,23 +240,12 @@ InitProcGlobal(void)
pg_atomic_init_u32(&ProcGlobal->procArrayGroupFirst, INVALID_PROC_NUMBER);
pg_atomic_init_u32(&ProcGlobal->clogGroupFirst, INVALID_PROC_NUMBER);
- /*
- * Create and initialize all the PGPROC structures we'll need. There are
- * six separate consumers: (1) normal backends, (2) autovacuum workers and
- * special workers, (3) background workers, (4) walsenders, (5) auxiliary
- * processes, and (6) prepared transactions. (For largely-historical
- * reasons, we combine autovacuum and special workers into one category
- * with a single freelist.) Each PGPROC structure is dedicated to exactly
- * one of these purposes, and they do not move between groups.
- */
- requestSize = PGProcShmemSize();
-
- ptr = ShmemInitStruct("PGPROC structures",
- requestSize,
- &found);
-
+ Assert(tmpAllProcs);
+ ptr = tmpAllProcs;
+ requestSize = ProcGlobalAllProcsShmemDesc.size;
MemSet(ptr, 0, requestSize);
+ /* Carve out the allProcs array from the shared memory area */
procs = (PGPROC *) ptr;
ptr = ptr + TotalProcs * sizeof(PGPROC);
@@ -242,7 +254,7 @@ InitProcGlobal(void)
ProcGlobal->allProcCount = MaxBackends + NUM_AUXILIARY_PROCS;
/*
- * Allocate arrays mirroring PGPROC fields in a dense manner. See
+ * Carve out arrays mirroring PGPROC fields in a dense manner. See
* PROC_HDR.
*
* XXX: It might make sense to increase padding for these arrays, given
@@ -257,31 +269,25 @@ InitProcGlobal(void)
ProcGlobal->statusFlags = (uint8 *) ptr;
ptr = ptr + (TotalProcs * sizeof(*ProcGlobal->statusFlags));
- /* make sure wer didn't overflow */
+ /* make sure we didn't overflow */
Assert((ptr > (char *) procs) && (ptr <= (char *) procs + requestSize));
/*
- * Allocate arrays for fast-path locks. Those are variable-length, so
+ * Initialize arrays for fast-path locks. Those are variable-length, so
* can't be included in PGPROC directly. We allocate a separate piece of
* shared memory and then divide that between backends.
*/
fpLockBitsSize = MAXALIGN(FastPathLockGroupsPerBackend * sizeof(uint64));
fpRelIdSize = MAXALIGN(FastPathLockSlotsPerBackend() * sizeof(Oid));
- requestSize = FastPathLockShmemSize();
-
- fpPtr = ShmemInitStruct("Fast-Path Lock Array",
- requestSize,
- &found);
-
- MemSet(fpPtr, 0, requestSize);
+ Assert(tmpFastPathLockArray);
+ fpPtr = tmpFastPathLockArray;
+ requestSize = FastPathLockArrayShmemDesc.size;
+ memset(fpPtr, 0, requestSize);
/* For asserts checking we did not overflow. */
fpEndPtr = fpPtr + requestSize;
- /* Reserve space for semaphores. */
- PGReserveSemaphores(ProcGlobalSemas());
-
for (i = 0; i < TotalProcs; i++)
{
PGPROC *proc = &procs[i];
diff --git a/src/backend/utils/activity/wait_event.c b/src/backend/utils/activity/wait_event.c
index aca2c8fc742..93e87a68fdf 100644
--- a/src/backend/utils/activity/wait_event.c
+++ b/src/backend/utils/activity/wait_event.c
@@ -79,6 +79,30 @@ typedef struct WaitEventCustomEntryByName
uint32 wait_event_info;
} WaitEventCustomEntryByName;
+static ShmemHashDesc WaitEventCustomHashByInfoDesc =
+{
+ .name = "WaitEventCustom hash by wait event information",
+ .ptr = &WaitEventCustomHashByInfo,
+
+ .init_size = WAIT_EVENT_CUSTOM_HASH_INIT_SIZE,
+ .max_size = WAIT_EVENT_CUSTOM_HASH_MAX_SIZE,
+ .hash_info.keysize = sizeof(uint32),
+ .hash_info.entrysize = sizeof(WaitEventCustomEntryByInfo),
+ .hash_flags = HASH_ELEM | HASH_BLOBS,
+};
+
+static ShmemHashDesc WaitEventCustomHashByNameDesc =
+{
+ .name = "WaitEventCustom hash by name",
+ .ptr = &WaitEventCustomHashByName,
+
+ .init_size = WAIT_EVENT_CUSTOM_HASH_INIT_SIZE,
+ .max_size = WAIT_EVENT_CUSTOM_HASH_MAX_SIZE,
+ /* key is a NULL-terminated string */
+ .hash_info.keysize = sizeof(char[NAMEDATALEN]),
+ .hash_info.entrysize = sizeof(WaitEventCustomEntryByName),
+ .hash_flags = HASH_ELEM | HASH_STRINGS,
+};
/* dynamic allocation counter for custom wait events */
typedef struct WaitEventCustomCounterData
@@ -90,6 +114,16 @@ typedef struct WaitEventCustomCounterData
/* pointer to the shared memory */
static WaitEventCustomCounterData *WaitEventCustomCounter;
+static void WaitEventCustomCounterDataShmemInit(void *arg);
+
+static ShmemStructDesc WaitEventCustomCounterShmemDesc =
+{
+ .name = "WaitEventCustomCounterData",
+ .size = sizeof(WaitEventCustomCounterData),
+ .init_fn = WaitEventCustomCounterDataShmemInit,
+ .ptr = (void **) &WaitEventCustomCounter,
+};
+
/* first event ID of custom wait events */
#define WAIT_EVENT_CUSTOM_INITIAL_ID 1
@@ -97,60 +131,22 @@ static uint32 WaitEventCustomNew(uint32 classId, const char *wait_event_name);
static const char *GetWaitEventCustomIdentifier(uint32 wait_event_info);
/*
- * Return the space for dynamic shared hash tables and dynamic allocation counter.
+ * Register shmem space for dynamic shared hash and dynamic allocation counter.
*/
-Size
-WaitEventCustomShmemSize(void)
+void
+WaitEventCustomShmemRegister(void)
{
- Size sz;
-
- sz = MAXALIGN(sizeof(WaitEventCustomCounterData));
- sz = add_size(sz, hash_estimate_size(WAIT_EVENT_CUSTOM_HASH_MAX_SIZE,
- sizeof(WaitEventCustomEntryByInfo)));
- sz = add_size(sz, hash_estimate_size(WAIT_EVENT_CUSTOM_HASH_MAX_SIZE,
- sizeof(WaitEventCustomEntryByName)));
- return sz;
+ ShmemRegisterStruct(&WaitEventCustomCounterShmemDesc);
+ ShmemRegisterHash(&WaitEventCustomHashByInfoDesc);
+ ShmemRegisterHash(&WaitEventCustomHashByNameDesc);
}
-/*
- * Allocate shmem space for dynamic shared hash and dynamic allocation counter.
- */
-void
-WaitEventCustomShmemInit(void)
+static void
+WaitEventCustomCounterDataShmemInit(void *arg)
{
- bool found;
- HASHCTL info;
-
- WaitEventCustomCounter = (WaitEventCustomCounterData *)
- ShmemInitStruct("WaitEventCustomCounterData",
- sizeof(WaitEventCustomCounterData), &found);
-
- if (!found)
- {
- /* initialize the allocation counter and its spinlock. */
- WaitEventCustomCounter->nextId = WAIT_EVENT_CUSTOM_INITIAL_ID;
- SpinLockInit(&WaitEventCustomCounter->mutex);
- }
-
- /* initialize or attach the hash tables to store custom wait events */
- info.keysize = sizeof(uint32);
- info.entrysize = sizeof(WaitEventCustomEntryByInfo);
- WaitEventCustomHashByInfo =
- ShmemInitHash("WaitEventCustom hash by wait event information",
- WAIT_EVENT_CUSTOM_HASH_INIT_SIZE,
- WAIT_EVENT_CUSTOM_HASH_MAX_SIZE,
- &info,
- HASH_ELEM | HASH_BLOBS);
-
- /* key is a NULL-terminated string */
- info.keysize = sizeof(char[NAMEDATALEN]);
- info.entrysize = sizeof(WaitEventCustomEntryByName);
- WaitEventCustomHashByName =
- ShmemInitHash("WaitEventCustom hash by name",
- WAIT_EVENT_CUSTOM_HASH_INIT_SIZE,
- WAIT_EVENT_CUSTOM_HASH_MAX_SIZE,
- &info,
- HASH_ELEM | HASH_STRINGS);
+ /* initialize the allocation counter and its spinlock. */
+ WaitEventCustomCounter->nextId = WAIT_EVENT_CUSTOM_INITIAL_ID;
+ SpinLockInit(&WaitEventCustomCounter->mutex);
}
/*
diff --git a/src/include/access/transam.h b/src/include/access/transam.h
index 6fa91bfcdc0..2dfc8b0f85f 100644
--- a/src/include/access/transam.h
+++ b/src/include/access/transam.h
@@ -345,8 +345,7 @@ extern TransactionId TransactionIdLatest(TransactionId mainxid,
extern XLogRecPtr TransactionIdGetCommitLSN(TransactionId xid);
/* in transam/varsup.c */
-extern Size VarsupShmemSize(void);
-extern void VarsupShmemInit(void);
+extern void VarsupShmemRegister(void);
extern FullTransactionId GetNewTransactionId(bool isSubXact);
extern void AdvanceNextFullTransactionIdPastXid(TransactionId xid);
extern FullTransactionId ReadNextFullTransactionId(void);
diff --git a/src/include/storage/dsm.h b/src/include/storage/dsm.h
index 407657df3ff..061bf725f88 100644
--- a/src/include/storage/dsm.h
+++ b/src/include/storage/dsm.h
@@ -26,8 +26,7 @@ extern void dsm_postmaster_startup(PGShmemHeader *);
extern void dsm_backend_shutdown(void);
extern void dsm_detach_all(void);
-extern size_t dsm_estimate_size(void);
-extern void dsm_shmem_init(void);
+extern void dsm_shmem_register(void);
#ifdef EXEC_BACKEND
extern void dsm_set_control_handle(dsm_handle h);
diff --git a/src/include/storage/dsm_registry.h b/src/include/storage/dsm_registry.h
index 506fae2c9ca..9a1b4d982af 100644
--- a/src/include/storage/dsm_registry.h
+++ b/src/include/storage/dsm_registry.h
@@ -22,7 +22,6 @@ extern dsa_area *GetNamedDSA(const char *name, bool *found);
extern dshash_table *GetNamedDSHash(const char *name,
const dshash_parameters *params,
bool *found);
-extern Size DSMRegistryShmemSize(void);
-extern void DSMRegistryShmemInit(void);
+extern void DSMRegistryShmemRegister(void);
#endif /* DSM_REGISTRY_H */
diff --git a/src/include/storage/pmsignal.h b/src/include/storage/pmsignal.h
index 206fb78f8a5..7cdc4852334 100644
--- a/src/include/storage/pmsignal.h
+++ b/src/include/storage/pmsignal.h
@@ -66,8 +66,7 @@ extern PGDLLIMPORT volatile PMSignalData *PMSignalState;
/*
* prototypes for functions in pmsignal.c
*/
-extern Size PMSignalShmemSize(void);
-extern void PMSignalShmemInit(void);
+extern void PMSignalShmemRegister(void);
extern void SendPostmasterSignal(PMSignalReason reason);
extern bool CheckPostmasterSignal(PMSignalReason reason);
extern void SetQuitSignalReason(QuitSignalReason reason);
diff --git a/src/include/storage/proc.h b/src/include/storage/proc.h
index bf3094f0f7d..d7a4c57f74c 100644
--- a/src/include/storage/proc.h
+++ b/src/include/storage/proc.h
@@ -549,7 +549,7 @@ extern PGDLLIMPORT PGPROC *AuxiliaryProcs;
* Function Prototypes
*/
extern int ProcGlobalSemas(void);
-extern Size ProcGlobalShmemSize(void);
+extern void ProcGlobalShmemRegister(void);
extern void InitProcGlobal(void);
extern void InitProcess(void);
extern void InitProcessPhase2(void);
diff --git a/src/include/storage/procarray.h b/src/include/storage/procarray.h
index c5ab1574fe3..572516c4e21 100644
--- a/src/include/storage/procarray.h
+++ b/src/include/storage/procarray.h
@@ -20,8 +20,7 @@
#include "utils/snapshot.h"
-extern Size ProcArrayShmemSize(void);
-extern void ProcArrayShmemInit(void);
+extern void ProcArrayShmemRegister(void);
extern void ProcArrayAdd(PGPROC *proc);
extern void ProcArrayRemove(PGPROC *proc, TransactionId latestXid);
diff --git a/src/include/storage/procsignal.h b/src/include/storage/procsignal.h
index 348fba53a93..d2344b1cbb3 100644
--- a/src/include/storage/procsignal.h
+++ b/src/include/storage/procsignal.h
@@ -63,8 +63,7 @@ typedef enum
/*
* prototypes for functions in procsignal.c
*/
-extern Size ProcSignalShmemSize(void);
-extern void ProcSignalShmemInit(void);
+extern void ProcSignalShmemRegister(void);
extern void ProcSignalInit(const uint8 *cancel_key, int cancel_key_len);
extern int SendProcSignal(pid_t pid, ProcSignalReason reason,
diff --git a/src/include/storage/sinvaladt.h b/src/include/storage/sinvaladt.h
index a1694500a85..4edba2936e6 100644
--- a/src/include/storage/sinvaladt.h
+++ b/src/include/storage/sinvaladt.h
@@ -28,8 +28,7 @@
/*
* prototypes for functions in sinvaladt.c
*/
-extern Size SharedInvalShmemSize(void);
-extern void SharedInvalShmemInit(void);
+extern void SharedInvalShmemRegister(void);
extern void SharedInvalBackendInit(bool sendOnly);
extern void SIInsertDataEntries(const SharedInvalidationMessage *data, int n);
diff --git a/src/include/utils/wait_event.h b/src/include/utils/wait_event.h
index 34c27cc3dc3..86fc8637f5e 100644
--- a/src/include/utils/wait_event.h
+++ b/src/include/utils/wait_event.h
@@ -42,8 +42,7 @@ extern PGDLLIMPORT uint32 *my_wait_event_info;
extern uint32 WaitEventExtensionNew(const char *wait_event_name);
extern uint32 WaitEventInjectionPointNew(const char *wait_event_name);
-extern void WaitEventCustomShmemInit(void);
-extern Size WaitEventCustomShmemSize(void);
+extern void WaitEventCustomShmemRegister(void);
extern char **GetWaitEventCustomNames(uint32 classId, int *nwaitevents);
/* ----------
--
2.47.3
view thread (75+ messages) latest in thread
reply
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Reply to all the recipients using the --to and --cc options:
reply via email
To: [email protected]
Cc: [email protected], [email protected], [email protected], [email protected], [email protected]
Subject: Re: Better shared data structure management and resizable shared data structures
In-Reply-To: <[email protected]>
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
This inbox is served by agora; see mirroring instructions
for how to clone and mirror all data and code used for this inbox