Feedback-ID: id4a34324:Fastmail
Date: Tue, 7 Apr 2026 16:09:25 -0400
From: Andres Freund <andres@anarazel.de>
To: Heikki Linnakangas <hlinnaka@iki.fi>
Cc: Ashutosh Bapat <ashutosh.bapat.oss@gmail.com>,
	Matthias van de Meent <boekewurm+postgres@gmail.com>,
 Robert Haas <robertmhaas@gmail.com>,
	pgsql-hackers <pgsql-hackers@postgresql.org>, chaturvedipalak1911@gmail.com
Subject: Re: Better shared data structure management and resizable shared
 data structures
Message-ID: <gi7oq4qrtc7kc3nvtt4ksxwjyoxkgl2itp4llpnhhzs7toox2n@vqd2japu4dek>
References: <7d3ba240-9350-4dfc-bbe1-be6584aee236@iki.fi>
 <CAEze2WjQZff3znd6CtG-OBzYZMMqy5TyQSoAo=QTFT38tDndeQ@mail.gmail.com>
 <1c3a07a7-158d-4800-927c-2641c73277d8@iki.fi>
 <CAEze2WjgCROMMXY0+j8FFdm3iFcr7By-+6Mwiz=PgGSEydiW3A@mail.gmail.com>
 <f1d04668-be37-472c-8b18-753b76726de5@iki.fi>
 <6d4383eb-4aaa-47ae-bda8-ee40dc60ad84@iki.fi>
 <CAExHW5u5LOgB3Y4Ee3VWVEurYN2GwnkbVEcfrGCQLgcGgf_zKw@mail.gmail.com>
 <CAExHW5uLau-i0T-ybPa=g5FLQo6OR1U9MR3GNijm6TDv12aLPw@mail.gmail.com>
 <CAExHW5v5muT_SKV2NCxxVmvC=_38Rw0aiv-wU4CGzHaBCRYzqA@mail.gmail.com>
 <b15a8fb8-5b59-4349-859c-07e7f070a21e@iki.fi>
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <b15a8fb8-5b59-4349-859c-07e7f070a21e@iki.fi>
Archived-At: 
 <https://www.postgresql.org/message-id/gi7oq4qrtc7kc3nvtt4ksxwjyoxkgl2itp4llpnhhzs7toox2n%40vqd2japu4dek>
Precedence: bulk

Hi,

On 2026-04-07 22:48:17 +0300, Heikki Linnakangas wrote:
> > +/*
> > + * ShmemResizeStruct() --- resize a resizable shared memory structure.
> > + *
> > + * The new size must be within [minimum_size, maximum_size].  If the structure
> > + * is being shrunk, the memory pages that are no longer needed are freed. If
> > + * the structure is being expanded, the memory pages that are needed for the
> > + * new size are allocated. See EstimateAllocatedSize() for explanation of which
> > + * pages are allocated for a resizable structure.
> > + */
> > +void
> > +ShmemResizeStruct(const char *name, Size new_size)
> 
> This interface only allows shrinking and growing the allocated region at the
> end, but the underlying mechanism is madvise(MADV_REMOVE) and
> madvise(MADV_WRITE_POPULATE), which supports also "punching holes", i.e.
> freeing memory in the middle of a region. Do we gain anything by restricting
> ourselves to changing the size at the end? It seems to me that it could be
> handy to punch holes for some use cases.

Agreed. The hard part may be the "communication" with the user about how
granular the punches can be.  Because that will depend on things like
huge_pages, huge_page_size and may depend on what alignment you happened to
get.


> What's the portability story? I understand that this is Linux-only at the
> moment, but what platforms can we support in the future, and what's the
> effort? I think BSD's have similar capabilities with plain mmap() and
> MADV_FREE if I read the man pages right.

At least linux' MADV_FREE is only for private mappings. It's not clear in at
least freebsd's man page, but the described use case makes me suspect it may
be similar there.


> What about macOS and Windows? This doesn't necessarily need to be fully
> portable, if some OS's don't have the capabilities we need, but would be
> nice to know what's possible.

Looks like windows has OfferVirtualMemory
https://learn.microsoft.com/en-us/windows/win32/api/memoryapi/nf-memoryapi-offervirtualmemory
but it's not clear to me if it actually does what we need when multiple
processes are attached.

I suspect it's going to be a lot easier once we're threaded...  The reason I
am ok with doing resizing this way before threading is because it's
architecturally pretty similar to what you'd want to do once threaded, so it's
not a huge dead end.  But I'm doubtful we'll find facilities that allow this
across processes in all operating systems...

Greetings,

Andres Freund