MIME-Version: 1.0
References: 
 <CAExHW5vM1bneLYfg0wGeAa=52UiJ3z4vKd3AJ72X8Fw6k3KKrg@mail.gmail.com>
 <CAExHW5s9Vp+-vJi020UJ+otyccBBo7eT1g6bttdRKL6HAvscyQ@mail.gmail.com>
 <mlsruptoxgm2nqtdfyfsowjklzxl5zltsjb3y5bmywtigm474l@5tsonk4t3kia>
 <CAExHW5uEK+eeG7e2g6uWh7POrFpfp+dqfaa=_3miMN17zgeaJw@mail.gmail.com>
 <CAExHW5vz+PUHHUuzGRwtyx-mPLQk3nCZXxrFqnruRadEFrO5Xg@mail.gmail.com>
 <CAExHW5so6VSxBC-1V=35229Z1+dw5vhw8HxHg9ry7UzceKcXzA@mail.gmail.com>
 <26c766d6-db0f-43d3-a618-44f8d40a3121@iki.fi>
 <CA+TgmoaoSha9KnSdrA7DDMYD9+Uor4OPEaQ88bbEC3QuVE5Mcg@mail.gmail.com>
 <62b8dc23-8f6a-4cac-91ff-f74bb5bc159a@iki.fi>
 <CA+TgmoboxAaS6MCRf__LRhf8Ji3B0O9_=d7KzRS12QhOuocnTQ@mail.gmail.com>
 <c7b4dafc-7a6c-43e4-8b9e-20807396590b@iki.fi>
 <CA+Tgmob7E3CGDoEBB3mwfk+v1CgcOG8yL0VHje2gy9Ts=e1Zxw@mail.gmail.com>
 <f8fc6f77-292d-4225-9f54-382c39da5ebc@iki.fi>
 <CA+TgmobqwCHp1w6xkTJmZiq4C8Pr0O3tNUvSFZcAgYLkvF45bg@mail.gmail.com>
 <8a6799be-bd42-49fb-8914-856c97bb1977@iki.fi>
 <CA+Tgmob8ot7qam7a650nx_4jyEoFLs8s_kPMc1+EB7-dr56=WA@mail.gmail.com>
 <cc3ff976-c303-45cd-9a8d-a90483170e37@iki.fi>
In-Reply-To: <cc3ff976-c303-45cd-9a8d-a90483170e37@iki.fi>
From: Robert Haas <robertmhaas@gmail.com>
Date: Thu, 19 Mar 2026 11:17:45 -0400
Message-ID: 
 <CA+TgmoauqYXm8iA3FGRAVKxYShUxWBiS_MSLmQpTrmO7wNHamw@mail.gmail.com>
Subject: Re: Better shared data structure management and resizable shared data
 structures
To: Heikki Linnakangas <hlinnaka@iki.fi>
Cc: Ashutosh Bapat <ashutosh.bapat.oss@gmail.com>,
 Andres Freund <andres@anarazel.de>,
	pgsql-hackers <pgsql-hackers@postgresql.org>, chaturvedipalak1911@gmail.com
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable
Archived-At: 
 <https://www.postgresql.org/message-id/CA%2BTgmoauqYXm8iA3FGRAVKxYShUxWBiS_MSLmQpTrmO7wNHamw%40mail.gmail.com>
Precedence: bulk

On Thu, Mar 19, 2026 at 10:34=E2=80=AFAM Heikki Linnakangas <hlinnaka@iki.f=
i> wrote:
> > I suspect
> > that issue is also why pg_stat_statements has the
> > process_shared_preload_libraries_in_progress check at the top, because
> > it looks to me like everything else that the function does would be
> > completely fine to do later.
>
> I think a bigger problem is loading and saving the statistics file. The
> file needs to be saved on postmaster exit, where do you do that if the
> library was not in shared_preload_libraries?

Well, there's no way to install a hook in the postmaster in that case,
so you can't. But I'm not sure that justifies skipping everything
_PG_init() would have done. A problem with the status quo is that
every module author makes their own decision about how to handle the
s_p_l problem, and they don't all decide differently, even in our own
tree. For example, autoprewarm chooses to register the GUC that it can
while skipping the other one, while pg_stat_statements skips
everything, including hook installation. Maybe that's properly
considered flexibility that should be left to each individual author,
but to me it seems more like happenstance than anything else. I'd
favor a design that emphasizes severability - i.e. you always try to
do as much as possible, and defer errors until later. So you always
create the GUCs but then restrict setting them to values that you
can't support without a restart, instead of not creating them. You
install the hooks and then maybe they will have to no-op out. It's
just weird if you load a library and the GUCs aren't defined and
there's not even a diagnostic telling you why.

> If you squint a little, this is pretty much the same as my descriptor
> design. Let's start from your DefineShmemRegion function, but in order
> to have some flexibility to add optional optional in the future, without
> having to create DefineShmemRegionNew(), DefineShmemRegionExt() or
> similar functions, let's pass the arguments in a struct. So instead of:
>
> DefineShmemRegion("pg_stat_statements", sizeof(pgssSharedState), &pgss,
> &pgss,pgss_shmem_init, 0);
>
> you would call it like this:
>
> DefineShmemRegion(&(ShmemStructDesc) {
>      .name =3D "pg_stat_statements",
>      .size =3D sizeof(pgssSharedState),
>      .ptr =3D (void **) &pgss,
>      .init_fn =3D pgss_shmem_init,
>      .flags =3D 0,
> });
>
> This flexibility will come handy as soon as we add the ability to resize.

I see your point. I'm not really convinced, though. In practice,
what's now going to happen is that you're probably going to move that
struct out of _PG_init() and define it elsewhere, so the logic is
getting split between multiple places, which does not improve
readability, and it also becomes much more worrying to what degree the
struct needs to be const, whereas if you just pass a bunch of
parameters by value you kind of understand what has to be happening.
Also, what flexibility have you really purchased? Sure, now you can
add arguments to the function call without breaking existing call
sites, but (1) there are other ways to do that, like by creating an
object first and then using methods to assign properties to it
afterward (i.e. RegisterCallbackOnShmemRegion) and (2) adding
arguments without breaking existing callers is not an unmixed blessing
in the first place. I know the world won't end if you go with this
style, but I guess I'm not much of a fan. I find this sort of thing
hard to read.

> Ok, we could add GetShmemRegion() in either design. Do we have any place
> where we'd use that though, instead the backend-private pointer global
> variable? I can't think of any examples where we currently call
> ShmemInitStruct() to get a pointer "on demand" like that.
>
> In pg_stat_statements, this would replace these tests:
>
>      if (!pgss || !pgss_hash)
>          ereport(ERROR,
>                  (errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE),
>                   errmsg("pg_stat_statements must be loaded via
> \"shared_preload_libraries\"")));
>
> But I don't think pg_stat_statements could still allocate the region
> after postmaster startup. GetShmemRegion() would just be a different way
> of throwing that error.

In that case, yes. But something like autoprewarm only needs a small
amount of shared memory, and can potentially initialize itself on the
fly. The not-yet-committed pg_collect_advice module has similar needs,
which it currently satisfies using GetNamedDSMSegment(); see in
particular pg_collect_advice_attach() if you feel like wandering over
to the pg_plan_advice thread.

> Thanks, this hasn't changed my opinions, but I really appreciate
> pressure-testing the design. I don't want to rewrite this again in a
> year, because we didn't get it quite right.

Yeah, I'm somewhat concerned about an ever-proliferating number of
ways to do things that all sort of suck in different ways.

--=20
Robert Haas
EDB: http://www.enterprisedb.com