public inbox for [email protected]
help / color / mirror / Atom feedFrom: Matthias van de Meent <[email protected]>
To: Ashutosh Bapat <[email protected]>
Cc: Heikki Linnakangas <[email protected]>
Cc: Robert Haas <[email protected]>
Cc: Andres Freund <[email protected]>
Cc: pgsql-hackers <[email protected]>
Cc: [email protected]
Subject: Re: Better shared data structure management and resizable shared data structures
Date: Tue, 7 Apr 2026 21:38:37 +0200
Message-ID: <CAEze2Wi1trP=AezMzTDUr0=u0d5g689WLtKE_Ezu_SuEJq7bag@mail.gmail.com> (raw)
In-Reply-To: <CAExHW5v5muT_SKV2NCxxVmvC=_38Rw0aiv-wU4CGzHaBCRYzqA@mail.gmail.com>
References: <CAExHW5vM1bneLYfg0wGeAa=52UiJ3z4vKd3AJ72X8Fw6k3KKrg@mail.gmail.com>
<CAExHW5uTNWOSxJDWQAUnS0tZawob2_J3dRAtc67NHNZ98X4_xA@mail.gmail.com>
<CAExHW5t439y61YD9bc7d5wZWHp6J=M43Qu3eEZOBPguZML7o2A@mail.gmail.com>
<CAExHW5v5FVZbsO9sLzztMZ11C3hgGStE=HkkV2bQkCyncess4w@mail.gmail.com>
<[email protected]>
<CAExHW5tCC0T1ky=Jnq-AvMxa67Adaw7aQ4iQAO=BSdHcbSNBVg@mail.gmail.com>
<[email protected]>
<CAExHW5tS7GncN90oJWOSzW_3F1EHL9xwe59L7Req3nUVgmObUw@mail.gmail.com>
<[email protected]>
<CAEze2WhMOHVgH2Xeyzx=VEk-Ta_YnQUqT+TdBiv5Lx8ESn2WZA@mail.gmail.com>
<CAExHW5s6h=c_q2m72Nvyj1ghMEhPkOBkeN5Htn7YR=1BrNN-Sw@mail.gmail.com>
<[email protected]>
<CAEze2WjQZff3znd6CtG-OBzYZMMqy5TyQSoAo=QTFT38tDndeQ@mail.gmail.com>
<[email protected]>
<CAEze2WjgCROMMXY0+j8FFdm3iFcr7By-+6Mwiz=PgGSEydiW3A@mail.gmail.com>
<[email protected]>
<[email protected]>
<CAExHW5u5LOgB3Y4Ee3VWVEurYN2GwnkbVEcfrGCQLgcGgf_zKw@mail.gmail.com>
<CAExHW5uLau-i0T-ybPa=g5FLQo6OR1U9MR3GNijm6TDv12aLPw@mail.gmail.com>
<CAExHW5v5muT_SKV2NCxxVmvC=_38Rw0aiv-wU4CGzHaBCRYzqA@mail.gmail.com>
On Tue, 7 Apr 2026 at 16:47, Ashutosh Bapat
<[email protected]> wrote:
>
> On Tue, Apr 7, 2026 at 3:36 PM Ashutosh Bapat
> <[email protected]> wrote:
> >
> > On Mon, Apr 6, 2026 at 7:23 PM Ashutosh Bapat
> > <[email protected]> wrote:
> > >
> > > I have kept these two patches separate from the main patch so that I
> > > can remove them if others feel they are not worth including in the
> > > feature.
> >
> > Here are patches rebased on the latest HEAD. No conflicts just rebase.
> >
> > Here are differences from the previous patchset.
> >
> > o. There are two patches in this patchset now. a. 0001 which supports
> > resizable shared memory and is equivalent to 0001 + 0002 + 0004 + 0005
> > from the previous patchset. b. 0002 which is 0006 from the previous
> > patchset and adds support for protecting resizable shared memory
> > structures. 0003, which added diagnostics to investigate CFBot
> > failure, from the previous patchset is not required anymore since all
> > tests pass with CFBot.
> >
> > o. I have merged 0002 into 0001 from the previous patchset since with
> > that patch all platforms are green on CFBot. The resizable shared
> > memory test now uses /proc/self/smaps instead of /proc/self/status to
> > find the amount of memory allocated in the main shared memory segment
> > of PostgreSQL.
> >
> > o. Merged 0004, which supported minimum_size, into 0001. Minimum_size
> > would be useful to protect against accidental shrinkage of the
> > resizable structures. It will help additional support for minimum
> > sizes of GUCs like shared_buffers. It also makes it easy and intuitive
> > to distinguish between fixed-size and resizable structures, and will
> > be useful to find the minimum size of the shared memory segment.
I was thinking more along the lines of attached (incremental) patch
0003 for min/max sizing. I'd say it has a slightly more natural API,
but YMMV.
-----
I also noticed that it's probably not correct to "just" check and
complain about the size of a resizable shmem segment when you attach:
Without coordination about which startup size the shmem segment should
have, how could you get the current size state correct? And
cross-process coordination of size information before shmem is
attached is not really possible, not when you may have to deal with a
very slow to start backend.
----
Attached also 0004, which makes some small adjustments to shmem.c's
resize checks.
Kind regards,
Matthias van de Meent
Databricks (https://www.databricks.com)
Attachments:
[application/octet-stream] v20260407-0004-Some-check-simplification-deduplication.nocfbot.patch (2.8K, 2-v20260407-0004-Some-check-simplification-deduplication.nocfbot.patch)
download | inline diff:
From 5668cae79da67fee2eab6c5ce480fed521446680 Mon Sep 17 00:00:00 2001
From: Matthias van de Meent <[email protected]>
Date: Tue, 7 Apr 2026 21:21:57 +0200
Subject: [PATCH v20260407 4/4] Some check simplification/deduplication.
---
src/backend/storage/ipc/shmem.c | 60 ++++++++++-----------------------
1 file changed, 17 insertions(+), 43 deletions(-)
diff --git a/src/backend/storage/ipc/shmem.c b/src/backend/storage/ipc/shmem.c
index 12de6344cac..540a51b50e5 100644
--- a/src/backend/storage/ipc/shmem.c
+++ b/src/backend/storage/ipc/shmem.c
@@ -704,49 +704,23 @@ AttachShmemIndexEntry(ShmemRequest *request, bool missing_ok)
return false;
}
- /* Check that the sizes in the index match the request. */
- if (index_entry->size != request->options->size &&
- request->options->size != SHMEM_ATTACH_UNKNOWN_SIZE)
- {
- ereport(ERROR,
- (errmsg("shared memory struct \"%s\" was created with"
- " different size: existing %zu, requested %zu",
- name, index_entry->size, request->options->size)));
- }
-
- /*
- * For resizable structures, also check that minimum_size and maximum_size
- * match. For fixed-size structures, these are derived (set to size) in
- * the index entry and not meaningful in the request.
- */
- if (request->options->maximum_size != 0)
- {
- if (index_entry->minimum_size != request->options->minimum_size &&
- request->options->minimum_size != SHMEM_ATTACH_UNKNOWN_SIZE)
- {
- ereport(ERROR,
- (errmsg("shared memory struct \"%s\" was created with"
- " different minimum_size: existing %zu, requested %zu",
- name, index_entry->minimum_size,
- request->options->minimum_size)));
- }
-
- if (index_entry->maximum_size != request->options->maximum_size &&
- request->options->maximum_size != SHMEM_ATTACH_UNKNOWN_SIZE)
- {
- ereport(ERROR,
- (errmsg("shared memory struct \"%s\" was created with"
- " different maximum_size: existing %zu, requested %zu",
- name, index_entry->maximum_size,
- request->options->maximum_size)));
- }
- }
- else
- {
- if (index_entry->minimum_size != index_entry->maximum_size)
- elog(ERROR, "shared memory struct \"%s\" was created as resizable, but requested as fixed-size",
- name);
- }
+#define CHECK_SIZE(size) \
+do { \
+ /* Check that the sizes in the index match the request. */ \
+ if (request->options->size != SHMEM_ATTACH_UNKNOWN_SIZE && \
+ index_entry->size != request->options->size) \
+ { \
+ ereport(ERROR, \
+ (errmsg("shared memory struct \"%s\" was created with" \
+ " different %s: existing %zu, requested %zu", \
+ name, CppAsString(size), index_entry->size, \
+ request->options->size))); \
+ } \
+} while (false)
+
+ CHECK_SIZE(size);
+ CHECK_SIZE(minimum_size);
+ CHECK_SIZE(maximum_size);
/*
* Re-establish the caller's pointer variable, or do other actions to
--
2.50.1 (Apple Git-155)
[application/octet-stream] v20260407-0003-Various-adjustments-to-resizable-shmem-acc.nocfbot.patch (10.8K, 3-v20260407-0003-Various-adjustments-to-resizable-shmem-acc.nocfbot.patch)
download | inline diff:
From 62d7a2a841494c12d1814eae5ab5788dfeb9d5c1 Mon Sep 17 00:00:00 2001
From: Matthias van de Meent <[email protected]>
Date: Tue, 7 Apr 2026 20:59:36 +0200
Subject: [PATCH v20260407 3/4] Various adjustments to resizable shmem
accounting
Instead of continuously falling back onto .size when .maximum_size
is 0, update the copied(!) options' .maximum_size with the value
of .size. This saves a compare operation per SHMEM_REQUEST_SPACE_SIZE()
evaluation, and simplifies which field to access for bound checks.
Similarly, .minimum_size is updated with an input of 0 meaning to
default to .size. A substitute Zero value of SHMEM_RESIZE_TO_ZERO
is used to allow users to resize the segment's memory usage to zero
bytes used.
It also reorders and deduplicates some error condition checks.
---
src/backend/storage/ipc/shmem.c | 111 ++++++++++++++++++--------------
src/include/storage/shmem.h | 20 +++---
2 files changed, 75 insertions(+), 56 deletions(-)
diff --git a/src/backend/storage/ipc/shmem.c b/src/backend/storage/ipc/shmem.c
index 61808c7a8e5..12de6344cac 100644
--- a/src/backend/storage/ipc/shmem.c
+++ b/src/backend/storage/ipc/shmem.c
@@ -108,14 +108,19 @@
* In order to allocate resizable shared memory structures, set
* ShmemRequestStructOpts::maximum_size to the maximum size that the structure
* can grow to. The address space for the maximum size will be reserved at
- * startup, but memory is allocated or freed as the structure grows or shrinks
- * respectively. ShmemRequestStructOpts::size should be set to the initial size
- * of the structure, which is the amount of memory allocated at the startup.
- * Optionally, ShmemRequestStructOpts::minimum_size can be set to the minimum
- * size that the structure can shrink to. After startup, the structure can be
- * resized by calling ShmemResizeStruct() by passing it the ShmemStructDesc for
- * the structure and the new size. ShmemResizeStruct() enforces that the new
- * size is within [minimum_size, maximum_size].
+ * startup, whilst the backing memory is allocated or freed as the structure
+ * grows or shrinks respectively. ShmemRequestStructOpts::size should be set
+ * to the initial size of the structure at startup. Optionally,
+ * ShmemRequestStructOpts::minimum_size can be set to the minimum size that
+ * the structure can shrink to. A sentinel value of SHMEM_RESIZE_TO_ZERO can
+ * be used used to indicate the struct can scale its memory usage down to 0
+ * bytes; the natural 0 is assumed to be uninitialized and so will cause the
+ * minimum size to default to the value of ShmemRequestStructOpts::size, like
+ * ShmemRequestStructOpts::maximum_size.
+ * After startup, the structure can be resized by calling ShmemResizeStruct()
+ * by passing it the ShmemStructDesc for the structure and the new size.
+ * ShmemResizeStruct() enforces that the new size is within
+ * [minimum_size, maximum_size].
*
* While resizable structures can be created after the startup, the memory
* available for them is quite limited.
@@ -192,8 +197,7 @@ typedef struct
* resizable shmem, the maximum_size is ensured to be 0 i.e. all the structures
* are treated as fixed-size structures.
*/
-#define SHMEM_REQUEST_SPACE_SIZE(request) \
- ((request)->options->maximum_size > 0 ? (request)->options->maximum_size : (request)->options->size)
+#define SHMEM_REQUEST_SPACE_SIZE(request) ((request)->options->maximum_size)
static List *pending_shmem_requests;
@@ -371,70 +375,81 @@ ShmemRequestInternal(ShmemStructOpts *options, ShmemRequestKind kind)
{
ShmemRequest *request;
+ /* Check that we're in the right state */
+ if (shmem_request_state != SRS_REQUESTING)
+ elog(ERROR, "ShmemRequestStruct can only be called from a shmem_request callback");
+
/* Check the options */
if (options->name == NULL)
elog(ERROR, "shared memory request is missing 'name' option");
+ /*
+ * Sanitize the options input by populating min/max with their actual values.
+ *
+ * Note that minimum_size's zero-initialized value "not specified" conflicts
+ * with a natural value for "resize to zero", so "resize to zero" has its own
+ * sentinel value with SHMEM_RESIZE_TO_ZERO.
+ */
+ if (options->maximum_size == 0)
+ {
+ options->maximum_size = options->size;
+ Assert(options->minimum_size == 0);
+ }
+
+ if (options->minimum_size == 0)
+ options->minimum_size = options->size;
+ else if (options->minimum_size == SHMEM_RESIZE_TO_ZERO)
+ options->minimum_size = 0;
+
+ /* resizing shmem segment */
+ if (options->maximum_size != options->minimum_size)
+ {
#ifndef HAVE_RESIZABLE_SHMEM
- if (options->maximum_size > 0)
elog(ERROR, "resizable shared memory is not supported on this platform");
#else
- if (options->maximum_size > 0 && shared_memory_type != SHMEM_TYPE_MMAP)
- elog(ERROR, "resizable shared memory requires shared_memory_type = mmap");
+ if (shared_memory_type != SHMEM_TYPE_MMAP)
+ elog(ERROR, "resizable shared memory requires shared_memory_type = mmap");
#endif
-
- if (IsUnderPostmaster)
- {
- if (options->size <= 0 && options->size != SHMEM_ATTACH_UNKNOWN_SIZE)
- elog(ERROR, "invalid size %zd for shared memory request for \"%s\"",
- options->size, options->name);
- if (options->minimum_size < 0 && options->minimum_size != SHMEM_ATTACH_UNKNOWN_SIZE)
- elog(ERROR, "invalid minimum_size %zd for shared memory request for \"%s\"",
- options->minimum_size, options->name);
- if (options->maximum_size < 0 && options->maximum_size != SHMEM_ATTACH_UNKNOWN_SIZE)
- elog(ERROR, "invalid maximum_size %zd for shared memory request for \"%s\"",
- options->maximum_size, options->name);
}
- else
+
+ if (!IsUnderPostmaster)
{
if (options->size == SHMEM_ATTACH_UNKNOWN_SIZE)
elog(ERROR, "SHMEM_ATTACH_UNKNOWN_SIZE cannot be used during startup");
- if (options->size <= 0)
- elog(ERROR, "invalid size %zd for shared memory request for \"%s\"",
- options->size, options->name);
if (options->minimum_size == SHMEM_ATTACH_UNKNOWN_SIZE)
elog(ERROR, "SHMEM_ATTACH_UNKNOWN_SIZE cannot be used during startup");
- if (options->minimum_size < 0)
- elog(ERROR, "invalid minimum_size %zd for shared memory request for \"%s\"",
- options->minimum_size, options->name);
if (options->maximum_size == SHMEM_ATTACH_UNKNOWN_SIZE)
elog(ERROR, "SHMEM_ATTACH_UNKNOWN_SIZE cannot be used during startup");
- if (options->maximum_size < 0)
- elog(ERROR, "invalid maximum_size %zd for shared memory request for \"%s\"",
- options->maximum_size, options->name);
}
+ if (options->size <= 0 && options->size != SHMEM_ATTACH_UNKNOWN_SIZE)
+ elog(ERROR, "invalid size %zd for shared memory request for \"%s\"",
+ options->size, options->name);
+ if (options->minimum_size < 0 && options->minimum_size != SHMEM_ATTACH_UNKNOWN_SIZE)
+ elog(ERROR, "invalid minimum_size %zd for shared memory request for \"%s\"",
+ options->minimum_size, options->name);
+ if (options->maximum_size < 0 && options->maximum_size != SHMEM_ATTACH_UNKNOWN_SIZE)
+ elog(ERROR, "invalid maximum_size %zd for shared memory request for \"%s\"",
+ options->maximum_size, options->name);
+
if (options->alignment != 0 && pg_nextpower2_size_t(options->alignment) != options->alignment)
elog(ERROR, "invalid alignment %zu for shared memory request for \"%s\"",
options->alignment, options->name);
- if (options->minimum_size > 0 && options->size != SHMEM_ATTACH_UNKNOWN_SIZE &&
- options->minimum_size > options->size)
- elog(ERROR, "resizable shared memory structure \"%s\" should have minimum size (%zd) less than or equal to size (%zd)",
- options->name, options->minimum_size, options->size);
-
- if (options->maximum_size > 0 && options->size > options->maximum_size)
- elog(ERROR, "resizable shared memory structure \"%s\" should have maximum size (%zd) greater than size (%zd)",
- options->name, options->maximum_size, options->size);
-
- if (options->minimum_size > 0 && options->maximum_size > 0 &&
+ if (options->minimum_size != SHMEM_ATTACH_UNKNOWN_SIZE && options->maximum_size != SHMEM_ATTACH_UNKNOWN_SIZE &&
options->minimum_size > options->maximum_size)
elog(ERROR, "resizable shared memory structure \"%s\" should have minimum size (%zd) less than or equal to maximum size (%zd)",
options->name, options->minimum_size, options->maximum_size);
- /* Check that we're in the right state */
- if (shmem_request_state != SRS_REQUESTING)
- elog(ERROR, "ShmemRequestStruct can only be called from a shmem_request callback");
+ if (options->minimum_size != SHMEM_ATTACH_UNKNOWN_SIZE && options->size != SHMEM_ATTACH_UNKNOWN_SIZE &&
+ options->minimum_size > options->size)
+ elog(ERROR, "resizable shared memory structure \"%s\" should have minimum size (%zd) less than or equal to size (%zd)",
+ options->name, options->minimum_size, options->size);
+
+ if (options->maximum_size != SHMEM_ATTACH_UNKNOWN_SIZE && options->size != SHMEM_ATTACH_UNKNOWN_SIZE &&
+ options->size > options->maximum_size)
+ elog(ERROR, "resizable shared memory structure \"%s\" should have size (%zd) less than or equal to maximum size (%zd)",
+ options->name, options->size, options->maximum_size);
/* Check that it's not already registered in this process */
foreach_ptr(ShmemRequest, existing, pending_shmem_requests)
diff --git a/src/include/storage/shmem.h b/src/include/storage/shmem.h
index f8ddb0dd7c0..2682704b3ef 100644
--- a/src/include/storage/shmem.h
+++ b/src/include/storage/shmem.h
@@ -58,18 +58,21 @@ typedef struct ShmemStructOpts
size_t alignment;
/*
- * Minimum size this structure can shrink to. Should be set to 0 for
- * fixed-size structures.
+ * Minimum size this structure can shrink to.
+ * When initialized to 0, it defaults to the value of the .size field; use
+ * SHMEM_RESIZE_TO_ZERO instead if you want your shmem allocation to be
+ * able to shrink to 'no memory usage'.
*/
ssize_t minimum_size;
/*
- * Maximum size this structure can grow upto in future. The memory is not
- * allocated right away but the corresponding address space is reserved so
- * that memory can be mapped to it when the structure grows. Typically
- * should be used for large resizable structures which need several pages
- * worth of contiguous memory. Should be set to 0 for fixed-size
- * structures.
+ * Maximum size this structure can grow up to in the future. The memory not
+ * required for .size is not allocated right away, but the corresponding
+ * address space is reserved so that memory can be mapped to it when the
+ * structure grows. Typically, this should be used for large resizable
+ * structures which need several pages worth of contiguous memory.
+ *
+ * When set to 0, it defaults to the value of the .size field.
*/
ssize_t maximum_size;
@@ -83,6 +86,7 @@ typedef struct ShmemStructOpts
} ShmemStructOpts;
#define SHMEM_ATTACH_UNKNOWN_SIZE (-1)
+#define SHMEM_RESIZE_TO_ZERO (-2)
/*
* Options for ShmemRequestHash()
--
2.50.1 (Apple Git-155)
view thread (82+ messages) latest in thread
reply
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Reply to all the recipients using the --to and --cc options:
reply via email
To: [email protected]
Cc: [email protected], [email protected], [email protected], [email protected], [email protected], [email protected]
Subject: Re: Better shared data structure management and resizable shared data structures
In-Reply-To: <CAEze2Wi1trP=AezMzTDUr0=u0d5g689WLtKE_Ezu_SuEJq7bag@mail.gmail.com>
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
This inbox is served by agora; see mirroring instructions
for how to clone and mirror all data and code used for this inbox