MIME-Version: 1.0
References: 
 <CAExHW5vM1bneLYfg0wGeAa=52UiJ3z4vKd3AJ72X8Fw6k3KKrg@mail.gmail.com>
 <CAExHW5uEK+eeG7e2g6uWh7POrFpfp+dqfaa=_3miMN17zgeaJw@mail.gmail.com>
 <CAExHW5vz+PUHHUuzGRwtyx-mPLQk3nCZXxrFqnruRadEFrO5Xg@mail.gmail.com>
 <CAExHW5so6VSxBC-1V=35229Z1+dw5vhw8HxHg9ry7UzceKcXzA@mail.gmail.com>
 <26c766d6-db0f-43d3-a618-44f8d40a3121@iki.fi>
 <CA+TgmoaoSha9KnSdrA7DDMYD9+Uor4OPEaQ88bbEC3QuVE5Mcg@mail.gmail.com>
 <62b8dc23-8f6a-4cac-91ff-f74bb5bc159a@iki.fi>
 <CA+TgmoboxAaS6MCRf__LRhf8Ji3B0O9_=d7KzRS12QhOuocnTQ@mail.gmail.com>
 <c7b4dafc-7a6c-43e4-8b9e-20807396590b@iki.fi>
 <CA+Tgmob7E3CGDoEBB3mwfk+v1CgcOG8yL0VHje2gy9Ts=e1Zxw@mail.gmail.com>
 <f8fc6f77-292d-4225-9f54-382c39da5ebc@iki.fi>
 <CA+TgmobqwCHp1w6xkTJmZiq4C8Pr0O3tNUvSFZcAgYLkvF45bg@mail.gmail.com>
 <8a6799be-bd42-49fb-8914-856c97bb1977@iki.fi>
 <CA+Tgmob8ot7qam7a650nx_4jyEoFLs8s_kPMc1+EB7-dr56=WA@mail.gmail.com>
 <cc3ff976-c303-45cd-9a8d-a90483170e37@iki.fi>
 <CA+TgmoauqYXm8iA3FGRAVKxYShUxWBiS_MSLmQpTrmO7wNHamw@mail.gmail.com>
 <113724ab-0028-493f-9605-6e8570f0939f@iki.fi>
In-Reply-To: <113724ab-0028-493f-9605-6e8570f0939f@iki.fi>
From: Robert Haas <robertmhaas@gmail.com>
Date: Wed, 25 Mar 2026 14:37:18 -0400
Message-ID: 
 <CA+TgmoYqx1+CeQHFf2d=VXecx9xz2xTM0_cP8oeXzakS89C6MQ@mail.gmail.com>
Subject: Re: Better shared data structure management and resizable shared data
 structures
To: Heikki Linnakangas <hlinnaka@iki.fi>
Cc: Ashutosh Bapat <ashutosh.bapat.oss@gmail.com>,
 Andres Freund <andres@anarazel.de>,
	pgsql-hackers <pgsql-hackers@postgresql.org>, chaturvedipalak1911@gmail.com
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable
Archived-At: 
 <https://www.postgresql.org/message-id/CA%2BTgmoYqx1%2BCeQHFf2d%3DVXecx9xz2xTM0_cP8oeXzakS89C6MQ%40mail.gmail.com>
Precedence: bulk

On Sat, Mar 21, 2026 at 8:14=E2=80=AFPM Heikki Linnakangas <hlinnaka@iki.fi=
> wrote:
> I see your point too, the options will indeed easily start to split
> between constant parts and parts that are set later.

Yeah. That definitely seems worth trying to fix.

> I don't like the idea of just using the string name as the handle. I
> think it will come handy to have a backend-private descriptor or handle
> to the shared memory region. But perhaps the "name" and "ptr" should be
> regular arguments, for example. If some parameters are passed as
> arguments and some in the struct, that also splits things in an awkward
> way, though.

I agree that having a handle object seems potentially useful. Putting
arguments that are required to be constant in the struct and passing
other things as arguments or in some other way seems like a good idea.

> I wonder if we should set aside a small amount of memory, like 10-20 kB,
> in the fixed shmem segment for small structs like that. Currently, such
> allocations will usually succeed because we leave some wiggle room, but
> the  can also be consumed by other things like locks. But we could
> reserve a small amount of memory specifically for small allocations in
> extensions like this.

Yeah, I don't really understand why we let the lock table use up that
space. I mean, I think it would be useful to have a way to let the
lock table expand without a server restart, and I also suspect that we
could come up with a less-silly data structure than the PROCLOCK hash,
but also if the only thing keeping you from running out of lock space
is the wiggle room, maybe you just need to bump up
max_locks_per_transaction. Like, you could easily burn through the
wiggle room, get an error anyway, and then later find that you also
now can't load certain extensions without a server restart.

> Shmem callbacks
> ---------------
>
> I separated the request/init/fn callbacks from the structs. There's now
> a concept of "shmem callbacks", which you register in _PG_init(). For
> example:
>
> static void pgss_shmem_request(void *arg);
> static void pgss_shmem_init(void *arg);
>
> static const ShmemCallbacks pgss_shmem_callbacks =3D {
>      .request_fn =3D pgss_shmem_request,
>      .init_fn =3D pgss_shmem_init,
>      .attach_fn =3D NULL, /* no special attach actions needed */
> };

What's the advantage of coupling the functions together this way, vs.
just registering each callback individually?

Also, I don't understand what "arg" is supposed to be doing. It
doesn't seem to be getting used for anything.

I wonder if we should think about just adjusting the timing of the
existing callbacks instead of adding new things. I mean, maybe that's
too likely to cause silent breakage, in which case we could also
replace them with new callbacks with slightly different names. But
there's something to be said for a hard cutover -- or at least not
letting both ways coexist for more than a few releases.

> This is a similar to the old shmem_request and shmem_startup hooks. It's
> not a one-to-one replacement though, the callbacks are called at
> different stages than the old hooks (to make things convenient with the
> new facility):
>
> * The request_fn callback is called in postmaster startup, at the same
> stage as the old shmem_request callback was. But in EXEC_BACKEND mode,
> it's *also* called in each backend.
>
> * The init_fn callback is only called in postmaster startup, when it's
> time to initialize the area. (Ignoring the special "after startup"
> allocations for now).

Trying to improve the timing of the callbacks makes sense to me as a
concept, without taking a position on these particular choices (and
definitely hoping for some after-startup improvements).

Looking at v7-0018 (any chance you have a public branch where I can
look at this without having to download and apply all the patches one
by one?) it seems like this gets rid of the need for a bunch of
IsUnderPostmaster checks in individual subsystems, and some if
(found)/if (!found) checks, which I definitely like.

> Shmem requests
> --------------
>
> To register a shmem area, you call ShmemRequestStruct() or
> ShmmeRequestHash() from the request callback function. For example:
>
> static void
> pgss_shmem_request(void *arg)
> {
>      static ShmemHashDesc pgssSharedHashDesc =3D {
>         .name =3D "pg_stat_statements hash",
>          .ptr =3D &pgss_hash,
>          .hash_info.keysize =3D sizeof(pgssHashKey),
>          .hash_info.entrysize =3D sizeof(pgssEntry),
>          .hash_flags =3D HASH_ELEM | HASH_BLOBS,
>      };
>      static ShmemStructDesc pgssSharedStateDesc =3D {
>          .name =3D "pg_stat_statements",
>          .size =3D sizeof(pgssSharedState),
>          .ptr =3D (void **) &pgss,
>      };
>
>      pgssSharedHashDesc.init_size =3D pgss_max;
>      pgssSharedHashDesc.max_size =3D pgss_max;
>      ShmemRequestHash(&pgssSharedHashDesc);
>      ShmemRequestStruct(&pgssSharedStateDesc);
> }
>
> Initialization happens in the init callback, which is called after all
> the pointers (pgss, pgss_hash) have been set.

I think this is not bad. I suppose it lets you get a complete list of
all of the descriptors someplace. It seems to avoid double-computing
the request size, or any possibility of drift between the requested
size and the allocated size. Having to create the struct and then set
the size afterward in some cases is a tiny bit awkward-looking, but
it's not awful. I do wonder if some coding pattern might creep in
where people create the struct and register it and then try to change
the size, or other fields, afterward. I'm tempted to propose making
the struct non-static and having the registration functions copy it,
so that there can be absolutely no question of getting away with such
antisocial behavior.

--=20
Robert Haas
EDB: http://www.enterprisedb.com