Message-ID: <a37ddc16-2892-40c6-bea6-a556f6011b7d@iki.fi>
Date: Sun, 5 Apr 2026 18:07:23 +0300
MIME-Version: 1.0
User-Agent: Mozilla Thunderbird
Subject: Re: Better shared data structure management and resizable shared data
 structures
To: Matthias van de Meent <boekewurm+postgres@gmail.com>
Cc: Ashutosh Bapat <ashutosh.bapat.oss@gmail.com>,
 Robert Haas <robertmhaas@gmail.com>, Andres Freund <andres@anarazel.de>,
 pgsql-hackers <pgsql-hackers@postgresql.org>, chaturvedipalak1911@gmail.com
References: 
 <CAExHW5vM1bneLYfg0wGeAa=52UiJ3z4vKd3AJ72X8Fw6k3KKrg@mail.gmail.com>
 <113724ab-0028-493f-9605-6e8570f0939f@iki.fi>
 <CAExHW5uTNWOSxJDWQAUnS0tZawob2_J3dRAtc67NHNZ98X4_xA@mail.gmail.com>
 <CAExHW5t439y61YD9bc7d5wZWHp6J=M43Qu3eEZOBPguZML7o2A@mail.gmail.com>
 <CAExHW5v5FVZbsO9sLzztMZ11C3hgGStE=HkkV2bQkCyncess4w@mail.gmail.com>
 <b2451225-0a10-4ded-abef-4e9efbde7a8e@iki.fi>
 <CAExHW5tCC0T1ky=Jnq-AvMxa67Adaw7aQ4iQAO=BSdHcbSNBVg@mail.gmail.com>
 <791c3f18-f4de-4d84-ac6b-c7ccc074dd38@iki.fi>
 <CAExHW5tS7GncN90oJWOSzW_3F1EHL9xwe59L7Req3nUVgmObUw@mail.gmail.com>
 <9d919bd9-94dd-4bda-8ccf-ebced4178c53@iki.fi>
 <CAEze2WhMOHVgH2Xeyzx=VEk-Ta_YnQUqT+TdBiv5Lx8ESn2WZA@mail.gmail.com>
 <CAExHW5s6h=c_q2m72Nvyj1ghMEhPkOBkeN5Htn7YR=1BrNN-Sw@mail.gmail.com>
 <7d3ba240-9350-4dfc-bbe1-be6584aee236@iki.fi>
 <CAEze2WjQZff3znd6CtG-OBzYZMMqy5TyQSoAo=QTFT38tDndeQ@mail.gmail.com>
 <1c3a07a7-158d-4800-927c-2641c73277d8@iki.fi>
 <CAEze2WjgCROMMXY0+j8FFdm3iFcr7By-+6Mwiz=PgGSEydiW3A@mail.gmail.com>
Content-Language: en-US
From: Heikki Linnakangas <hlinnaka@iki.fi>
In-Reply-To: 
 <CAEze2WjgCROMMXY0+j8FFdm3iFcr7By-+6Mwiz=PgGSEydiW3A@mail.gmail.com>
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: 7bit
Archived-At: 
 <https://www.postgresql.org/message-id/a37ddc16-2892-40c6-bea6-a556f6011b7d%40iki.fi>
Precedence: bulk

On 05/04/2026 02:17, Matthias van de Meent wrote:
> Formatting:
>> +    ShmemRequestHash(.name = "pg_stat_statements hash",
>> +                     .nelems = pgss_max,
>> +                     .hash_info.keysize = sizeof(pgssHashKey),
>> +                     .hash_info.entrysize = sizeof(pgssEntry),
>> +                     .hash_flags = HASH_ELEM | HASH_BLOBS,
>> +                     .ptr = &pgss_hash,
>> +        );
> (note that additional unit of indentation for the closing bracket)
> 
> Is this malformatting caused by pgindent? If so, could you see if
> there's a better way of defining ShmemRequestHash/Struct that doesn't
> have this indent as output?

Yeah, that's pgindent. Matter of taste, but I think that looks fine. An 
alternative is to put the closing bracket on the same line with the last 
argument and drop the trailing comma:

     ShmemRequestStruct(.name = "pg_stat_statements",
                        .size = sizeof(pgssSharedState),
                        .ptr = (void **) &pgss);

That looks OK to me too.

>> +    pgss->extent = 0;
>> +    pgss->n_writers = 0;
>> +    pgss->gc_count = 0;
>> +    pgss->stats.dealloc = 0;
> 
> Shmem is said to be zero-initialized, should we remove the manual
> zero-initialization?
> 
>> +    on_shmem_exit(pgss_shmem_shutdown, (Datum) 0);
> See my upthread comment about adding optional on_shmem_exit callbacks
> to ShmemCallbacks.
> 
> 0005: LGTM
> 
> 0006: I don't think it is a great idea to make the LwLock machinery
> the first to get allocation requests:
> It has the RequestNamedLWLockTranche infrastructure, which can only
> register new requests while process_shmem_requests_in_progress, and
> making it request its memory ahead of everything else is likely to
> cause an undersized tranche to be allocated.

Good catch. I think the easiest fix is to call process_shmem_requests() 
before ShmemCallRequestCallbacks() at postmaster startup. That's kind of 
how it was before: process_shmem_requests() was called before all the 
*ShmemSize() and *ShmemInit() functions in core.

> You could make sure that this isn't an issue by maintaining a flag
> in lwlock.c that's set when the shmem request is made (and reset on
> shmem exit), which must be false when RequestNamedLWLockTranche() is
> called, and if not then it should throw an error.
I'll change it so that the number of locks calculated in 
LWLockShmemRequest() is stored in a global variable, and 
LWLockShmemInit() has an Assert() to cross-checks with that. That 
catches the bug and seems like a good cross-check in general.

This isn't the only place where we need to pass information from the 
request callback to the init callback. I've used a global variable for 
that here, and also between ProcGlobalShmemRequest() and 
ProcGlobalShmemRequest(). An alternative might be to use the callback 
arg pointers that are currently unused, but I'm not sure how to make 
that ergonomic. The current 'arg' isn't very helpful for that, so 
perhaps the signatures should look like this instead:

static void
LWLockShmemRequest(Datum *init_arg)
{
     int         numLocks;

     numLocks = NUM_FIXED_LWLOCKS + NumLWLocksForNamedTranches();

     /* pass the calculated numLocks value to LWLockShmemInit() */
     *init_arg = Int32GetDatum(numLocks);

     ...
}

static void
LWLockShmemInit(Datum init_arg)
{
     int         numLocks = DatumGetIn32(numLocks);

     ...
}

If you need to pass more than a single Datum, you can allocate a struct 
and pass it via PointerGetDatum(). At that point global variables might 
feel simpler again though.

> 0010: Not looked at everything yet, but a few comment:
> 
>> +++ b/src/include/access/slru.h
> 
> With the changes in the signatures for most/all SLRU functions from a
> hidden-by-typedef pointer to a visible pointer type, maybe this could
> be an opportunity to swap them to `const SlruDesc *ctl` wherever
> possible? I don't think there are many backend-local changes that
> happen to SlruDescs once we've properly started the backend. I'm happy
> to provide an incremental patch if you'd like me to spend cycles on it
> if you're busy.

Yeah sounds like a good idea.

>> +++ b/src/backend/access/transam/clog.c
> 
>> +    SimpleLruRequest(.desc = &XactSlruDesc,
>> +                     .name = "transaction",
>> +                     .Dir = "pg_xact",
>> +                     .long_segment_names = false,
>> +
>> +                     .nslots = CLOGShmemBuffers(),
>> +                     .nlsns = CLOG_LSNS_PER_PAGE,
>> +
>> +                     .sync_handler = SYNC_HANDLER_CLOG,
>> +                     .PagePrecedes = CLOGPagePrecedes,
>> +                     .errdetail_for_io_error = clog_errdetail_for_io_error,
> 
> That awfully inconsistent field name styling is ... awful, but not
> this patch's fault. If something can be done about it in a cheap
> fashion in this patch, that'd be great, but I won't hold it against
> you if that's skipped.

:-D.

- Heikki