public inbox for [email protected]
help / color / mirror / Atom feedFrom: Andres Freund <[email protected]>
To: Heikki Linnakangas <[email protected]>
Cc: Ashutosh Bapat <[email protected]>
Cc: Matthias van de Meent <[email protected]>
Cc: Robert Haas <[email protected]>
Cc: pgsql-hackers <[email protected]>
Cc: [email protected]
Subject: Re: Better shared data structure management and resizable shared data structures
Date: Tue, 7 Apr 2026 16:09:25 -0400
Message-ID: <gi7oq4qrtc7kc3nvtt4ksxwjyoxkgl2itp4llpnhhzs7toox2n@vqd2japu4dek> (raw)
In-Reply-To: <[email protected]>
References: <[email protected]>
<CAEze2WjQZff3znd6CtG-OBzYZMMqy5TyQSoAo=QTFT38tDndeQ@mail.gmail.com>
<[email protected]>
<CAEze2WjgCROMMXY0+j8FFdm3iFcr7By-+6Mwiz=PgGSEydiW3A@mail.gmail.com>
<[email protected]>
<[email protected]>
<CAExHW5u5LOgB3Y4Ee3VWVEurYN2GwnkbVEcfrGCQLgcGgf_zKw@mail.gmail.com>
<CAExHW5uLau-i0T-ybPa=g5FLQo6OR1U9MR3GNijm6TDv12aLPw@mail.gmail.com>
<CAExHW5v5muT_SKV2NCxxVmvC=_38Rw0aiv-wU4CGzHaBCRYzqA@mail.gmail.com>
<[email protected]>
Hi,
On 2026-04-07 22:48:17 +0300, Heikki Linnakangas wrote:
> > +/*
> > + * ShmemResizeStruct() --- resize a resizable shared memory structure.
> > + *
> > + * The new size must be within [minimum_size, maximum_size]. If the structure
> > + * is being shrunk, the memory pages that are no longer needed are freed. If
> > + * the structure is being expanded, the memory pages that are needed for the
> > + * new size are allocated. See EstimateAllocatedSize() for explanation of which
> > + * pages are allocated for a resizable structure.
> > + */
> > +void
> > +ShmemResizeStruct(const char *name, Size new_size)
>
> This interface only allows shrinking and growing the allocated region at the
> end, but the underlying mechanism is madvise(MADV_REMOVE) and
> madvise(MADV_WRITE_POPULATE), which supports also "punching holes", i.e.
> freeing memory in the middle of a region. Do we gain anything by restricting
> ourselves to changing the size at the end? It seems to me that it could be
> handy to punch holes for some use cases.
Agreed. The hard part may be the "communication" with the user about how
granular the punches can be. Because that will depend on things like
huge_pages, huge_page_size and may depend on what alignment you happened to
get.
> What's the portability story? I understand that this is Linux-only at the
> moment, but what platforms can we support in the future, and what's the
> effort? I think BSD's have similar capabilities with plain mmap() and
> MADV_FREE if I read the man pages right.
At least linux' MADV_FREE is only for private mappings. It's not clear in at
least freebsd's man page, but the described use case makes me suspect it may
be similar there.
> What about macOS and Windows? This doesn't necessarily need to be fully
> portable, if some OS's don't have the capabilities we need, but would be
> nice to know what's possible.
Looks like windows has OfferVirtualMemory
https://learn.microsoft.com/en-us/windows/win32/api/memoryapi/nf-memoryapi-offervirtualmemory
but it's not clear to me if it actually does what we need when multiple
processes are attached.
I suspect it's going to be a lot easier once we're threaded... The reason I
am ok with doing resizing this way before threading is because it's
architecturally pretty similar to what you'd want to do once threaded, so it's
not a huge dead end. But I'm doubtful we'll find facilities that allow this
across processes in all operating systems...
Greetings,
Andres Freund
view thread (82+ messages) latest in thread
reply
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Reply to all the recipients using the --to and --cc options:
reply via email
To: [email protected]
Cc: [email protected], [email protected], [email protected], [email protected], [email protected], [email protected]
Subject: Re: Better shared data structure management and resizable shared data structures
In-Reply-To: <gi7oq4qrtc7kc3nvtt4ksxwjyoxkgl2itp4llpnhhzs7toox2n@vqd2japu4dek>
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
This inbox is served by agora; see mirroring instructions
for how to clone and mirror all data and code used for this inbox