MIME-Version: 1.0
References: 
 <CAExHW5vM1bneLYfg0wGeAa=52UiJ3z4vKd3AJ72X8Fw6k3KKrg@mail.gmail.com>
 <cc3ff976-c303-45cd-9a8d-a90483170e37@iki.fi>
 <CA+TgmoauqYXm8iA3FGRAVKxYShUxWBiS_MSLmQpTrmO7wNHamw@mail.gmail.com>
 <113724ab-0028-493f-9605-6e8570f0939f@iki.fi>
 <CAExHW5uTNWOSxJDWQAUnS0tZawob2_J3dRAtc67NHNZ98X4_xA@mail.gmail.com>
 <CAExHW5t439y61YD9bc7d5wZWHp6J=M43Qu3eEZOBPguZML7o2A@mail.gmail.com>
 <CAExHW5v5FVZbsO9sLzztMZ11C3hgGStE=HkkV2bQkCyncess4w@mail.gmail.com>
 <b2451225-0a10-4ded-abef-4e9efbde7a8e@iki.fi>
 <CAExHW5tCC0T1ky=Jnq-AvMxa67Adaw7aQ4iQAO=BSdHcbSNBVg@mail.gmail.com>
 <791c3f18-f4de-4d84-ac6b-c7ccc074dd38@iki.fi>
 <CAExHW5tS7GncN90oJWOSzW_3F1EHL9xwe59L7Req3nUVgmObUw@mail.gmail.com>
 <9d919bd9-94dd-4bda-8ccf-ebced4178c53@iki.fi>
 <CAEze2WhMOHVgH2Xeyzx=VEk-Ta_YnQUqT+TdBiv5Lx8ESn2WZA@mail.gmail.com>
 <CAExHW5s6h=c_q2m72Nvyj1ghMEhPkOBkeN5Htn7YR=1BrNN-Sw@mail.gmail.com>
 <7d3ba240-9350-4dfc-bbe1-be6584aee236@iki.fi>
 <CAEze2WjQZff3znd6CtG-OBzYZMMqy5TyQSoAo=QTFT38tDndeQ@mail.gmail.com>
 <1c3a07a7-158d-4800-927c-2641c73277d8@iki.fi>
 <CAExHW5twYLhVjtApX3osdaz-4b0Tjvjj_Zr20orUb61qgiJmUw@mail.gmail.com>
 <CAExHW5stth2mdXh3ukn9rWJ+Pruoat+5tY3CYyM6KGqH2G30fQ@mail.gmail.com>
 <CAEze2WiD7m+A+3OrAK0JU265BZ4P48_AuKY-X5siJ=294tBeDQ@mail.gmail.com>
 <CAExHW5u=zdSntmd9hhrMFDQP5n15Rz+JMRYHyo1FNwXE5LsuDg@mail.gmail.com>
In-Reply-To: 
 <CAExHW5u=zdSntmd9hhrMFDQP5n15Rz+JMRYHyo1FNwXE5LsuDg@mail.gmail.com>
From: Matthias van de Meent <boekewurm+postgres@gmail.com>
Date: Sun, 5 Apr 2026 21:05:04 +0200
Message-ID: 
 <CAEze2Wjn2cpQEPwzLajc0XdcMy8T=d1AWjg3UAmMUT2TmHkQkA@mail.gmail.com>
Subject: Re: Better shared data structure management and resizable shared data
 structures
To: Ashutosh Bapat <ashutosh.bapat.oss@gmail.com>
Cc: Heikki Linnakangas <hlinnaka@iki.fi>, Robert Haas <robertmhaas@gmail.com>,
	Andres Freund <andres@anarazel.de>,
 pgsql-hackers <pgsql-hackers@postgresql.org>,
	chaturvedipalak1911@gmail.com
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable
Archived-At: 
 <https://www.postgresql.org/message-id/CAEze2Wjn2cpQEPwzLajc0XdcMy8T%3Dd1AWjg3UAmMUT2TmHkQkA%40mail.gmail.com>
Precedence: bulk

On Sun, 5 Apr 2026 at 13:20, Ashutosh Bapat
<ashutosh.bapat.oss@gmail.com> wrote:
>
> On Sun, Apr 5, 2026 at 2:36=E2=80=AFPM Matthias van de Meent
> <boekewurm+postgres@gmail.com> wrote:
> >
> > On Sun, 5 Apr 2026, 07:59 Ashutosh Bapat, <ashutosh.bapat.oss@gmail.com=
> wrote:
> > >
> > > On Sun, Apr 5, 2026 at 11:18=E2=80=AFAM Ashutosh Bapat
> > > <ashutosh.bapat.oss@gmail.com> wrote:
> > > >
> >
> > I'm not opposed to HAVE_RESIZABLE_SHMEM, but is it universal enough on
> > its platforms to make it part of the exposed ABI for Shmem? I think
> > that we should expose the same functions and structs, and just have
> > the shmem internals throw an error if the configuration used by the
> > user implies the user wants to update shmem sizing when the system
> > doesn't support it. That would avoid extensions having to recompile
> > between have/have not systems that have an otherwise compatible ABI;
> > especially when those extensions don't actually need the resizeable
> > part of the shmem system.
> >
>
> I don't think I understand this fully. An extension may want to
> support a structure in both modes - fixed as well as resizable
> depending upon whether the latter is supported. If the structure has
> maximum_size always the extension code needs to set it to 0 when the
> resizable shared structure is not supported and set to actual
> maximum_size when the resizable structure is supported. Without a
> macro or some flag they can not do that. The flag/macro then becomes
> part ABI for shmem. Am I correct?

That's not quite what I meant.

With your patch, the size and field offsets in `struct
ShmemStructOpts` changes depending only on HAVE_RESIZABLE_SHMEM, as
does function's availability. This means that an extension that's
built without HAVE_RESIZABLE_SHMEM (an otherwise identical system)
can't correctly be loaded into a server that does have
HAVE_RESIZABLE_SHMEM defined - or at least it'll misbehave when it
tries to use the new shmem system without trying out resizeable areas.

If instead the fields used for definining resizable shmem areas (and
the relevant functions) are always defined, but with runtime checks to
make sure that in !HAVE_RESIZEABLE_SHMEM nobody tries to use the
resizing functionality, then that'd reduce the unchecked hidden
incompatibility; assuming that no extension manually does memory
management syscall operations on those shmem areas.

> Since extension binaries need to be
> built on different platforms anyway, that would automatically take
> care of building with or without HAVE_RESIZABLE_SHMEM.  I feel it makes
> testing simpler since run time behaviour is fixed. Maybe I am missing
> something. Maybe a code diff or some example platform might make it
> more clear for me.

I'm not entirely sure it would be automatic. Is it guaranteed that
HAVE_RESIZABLE_SHMEM won't change over the lifetime of any
distribution's platform? Because it's definitely not apparent to me
that rebuilding the new server version against an upgraded platform
(now possibly with HAVE_RESIZABLE_SHMEM) should also mean rebuilding
the extensions that have been built against a previous minor version
(without HAVE_RESIZABLE_SHMEM).

> > >  For now, it
> > > seems only for the sanity checks, but it could be seen as a useful
> > > safety feature. A difference in maximum_size and minimum_size would
> > > indicate that the structure is resizable.
> >
> > I think that's the right approach.
>
>
> I also think that introducing minimum_size is useful. Let's hear from
> Heikki before implementing it, in case he has a different opinion. I
> am not sure about min_allocated_space though - what use do you see for
> it. reserved_space is useful in pg_shmem_allocations() C function
> itself and gives impact to the fully grown structure. What would
> min_allocated_space give us? If at all it would be min_allocated_size
> not space since reserved space will never change. But even that I am
> not sure about.

I'd say it's mostly interesting for people looking at or debugging
shmem allocations. Which isn't a huge group of developers or DBAs, but
if we're exposing data like this, and are going to allow resizing,
then someone could see some benefits from this.

E.g., it may be useful to have the information to see how low the
currently running server can scale down its memory usage, so that the
admin can see whether a reboot is required if they want to allow it to
scale it down further (assuming there's a lower limit for allocations
- some shmem structs may have a lower scaling limit defined at
startup, while others may be able to scale linearly from 0 to 100)


Kind regards,

Matthias van de Meent
Databricks (https://www.databricks.com)