MIME-Version: 1.0
References: 
 <CAExHW5vM1bneLYfg0wGeAa=52UiJ3z4vKd3AJ72X8Fw6k3KKrg@mail.gmail.com>
 <5a37c2e3-619d-4816-84d7-0b27e3e6797f@iki.fi>
 <CAExHW5s9Vp+-vJi020UJ+otyccBBo7eT1g6bttdRKL6HAvscyQ@mail.gmail.com>
 <mlsruptoxgm2nqtdfyfsowjklzxl5zltsjb3y5bmywtigm474l@5tsonk4t3kia>
 <CAExHW5uEK+eeG7e2g6uWh7POrFpfp+dqfaa=_3miMN17zgeaJw@mail.gmail.com>
 <CAExHW5vz+PUHHUuzGRwtyx-mPLQk3nCZXxrFqnruRadEFrO5Xg@mail.gmail.com>
 <CAExHW5so6VSxBC-1V=35229Z1+dw5vhw8HxHg9ry7UzceKcXzA@mail.gmail.com>
 <26c766d6-db0f-43d3-a618-44f8d40a3121@iki.fi>
 <7de8e39c-d1c4-4783-91b9-129d90e91411@iki.fi>
 <CAExHW5u0466od5Fb_mO+KM_eRfbrD46QtMgH92fP7A15GE+a+w@mail.gmail.com>
 <41137f2a-0cb0-4cbe-9afc-2dc690f60a87@iki.fi>
In-Reply-To: <41137f2a-0cb0-4cbe-9afc-2dc690f60a87@iki.fi>
From: Ashutosh Bapat <ashutosh.bapat.oss@gmail.com>
Date: Tue, 17 Mar 2026 17:28:05 +0530
Message-ID: 
 <CAExHW5u53KNTvgTA9qVUvS3==V+H5PSyCqH68EKF5cC9jQf7nA@mail.gmail.com>
Subject: Re: Better shared data structure management and resizable shared data
 structures
To: Heikki Linnakangas <hlinnaka@iki.fi>
Cc: Zsolt Parragi <zsolt.parragi@percona.com>,
 Andres Freund <andres@anarazel.de>,
	pgsql-hackers <pgsql-hackers@postgresql.org>, chaturvedipalak1911@gmail.com
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable
Archived-At: 
 <https://www.postgresql.org/message-id/CAExHW5u53KNTvgTA9qVUvS3%3D%3DV%2BH5PSyCqH68EKF5cC9jQf7nA%40mail.gmail.com>
Precedence: bulk

On Tue, Mar 17, 2026 at 3:26=E2=80=AFAM Heikki Linnakangas <hlinnaka@iki.fi=
> wrote:
>
>
> I don't plan to get rid of the legacy API any time soon, I expect
> existing extensions to continue using it for years to come. So I moved
> them per your suggestion.

Do you plan to get rid of the shmem_request_hook and
shmem_startup_hook? What's the plan there? Wouldn't the old APIs and
new APIs overwhelm extension writers - having two different ways and
three different hooks to allocate a structure in the shared memory.

>
> > ShmemInitStruct() now calls ShmemRegisterStruct(). Earlier it could be
> > called from any backend, in any state to fetch a pointer to a shared
> > memory structure. It didn't add a new structure. Now it can add a new
> > structure. I am wondering whether that can cause registry in different
> > backends to get out of sync. Should we limit the window when it can be
> > called just like how shmem_request_hook call is limited. In that sense
> > ShmemRegisterStruct() looks something to be called from a
> > shmem_register_hook which is also called from EXEC_BACKEND. Sorry to
> > expand it here rather in my previous reply. In case we replace all the
> > current calls to ShmemInitStruct() with ShmemRegisterStruct(), we may
> > be able to get rid of the Shmem Index altogether; after all it's used
> > only for fetching the pointers to the shared memory areas in
> > EXEC_BACKEND mode.
>
> I think it's still useful to allow ShmemRegisterStruct() after
> postmaster startup, so that you can use it with extensions that are not
> listed in shared_preload_libraries, but need a little bit of shared
> memory. Hmm, perhaps we should add an explicit flag for that case,
> though. So that by default ShmemRegisterStruct() fails if you call it
> after postmaster startup, but you could allow it by setting a flag in
> the descriptor.
>

The hash tables do not allocate all their entries upfront but they
request shared memory for their maximum size. Before they could grow
to the maximum of their size, if somebody calls ShmemInitStruct with a
huge memory size request that takes away all the reserved address
space/memory, the hash table won't get its fair share of shared
memory. I agree that it could still happen if a hash table grows
beyond the contracted request ... but it's atleast registered. I think
we should prevent such cases. We could keep track of the extra shared
memory that we have allocated and refuse an unregistered request if
that memory is exhausted. But I think we already have some unaccounted
for structures e.g. ShmemAllocator and ShmemHeader already. So it
might turn out to be more complex that required. I have feeling that
we are providing flexibility that the current infrastructure can not
support. I am not opposed to supporting ShmemRegisterStruct; I like
the idea, but it seems premature.

--
Best Wishes,
Ashutosh Bapat