From: Tom Lane <tgl@sss.pgh.pa.us>
To: Georgios <gkokolatos@protonmail.com>
Cc: Michael Paquier <michael@paquier.xyz>
Cc: Julien Rouhaud <rjuju123@gmail.com>
Cc: Julien Rouhaud <julien.rouhaud@free.fr>
Cc: PostgreSQL Hackers <pgsql-hackers@lists.postgresql.org>
Cc: robertmhaas@gmail.com <robertmhaas@gmail.com>
Subject: Re: Shared memory size computation oversight?
Date: Thu, 04 Mar 2021 12:53:54 -0500
Message-ID: <1374257.1614880434@sss.pgh.pa.us> (raw)
In-Reply-To: <GbQO126b8MXGYDkI94ilDHoxqywWUmuvOEj85ocdMIPp8er2GTA_rTFQVBIGuWXu9UYu_xCaBETEkERwMxNIoJzMjDXnATbzrU8XnY_R3TQ=@protonmail.com>
References: <20210227080815.jryyhy6lrn6gahkm@nol>
<SeVx6SxBxMHIm_66gE4M8rzAn5vqXw8akcyJ-26bJxHeNC_wW9fN-xDr_G3AGiSH328-rxMzXbNf6H4TsaNIG-OJFE8LvmdflG9Zb-yIKq4=@protonmail.com>
<1183451.1614788634@sss.pgh.pa.us>
<20210303164052.f336wina7qpchabd@nol>
<YECGppkDKJArPOUb@paquier.xyz>
<20210304072333.4yydog47md7cfrda@nol>
<YECYj2KORHb06xbA@paquier.xyz>
<GbQO126b8MXGYDkI94ilDHoxqywWUmuvOEj85ocdMIPp8er2GTA_rTFQVBIGuWXu9UYu_xCaBETEkERwMxNIoJzMjDXnATbzrU8XnY_R3TQ=@protonmail.com>
I spent a bit of time looking into this. Using current HEAD, I
instrumented CreateSharedMemoryAndSemaphores to log the size estimates
returned by the various estimation subroutines, plus the shared memory
space actually consumed (i.e, the change in ShmemSegHdr->freeoffset)
by the various shared memory initialization functions. There were only
two estimates that were way off: LockShmemSize requested 1651771 more
bytes than InitLocks actually consumed, and PredicateLockShmemSize
requested 167058 more bytes than InitPredicateLocks consumed. I believe
both of those are intentional, reflecting space that may be eaten by the
lock tables later.
Meanwhile, looking at ShmemSegHdr->freeoffset vs ShmemSegHdr->totalsize,
the actual remaining shmem space after postmaster startup is 1919488
bytes. (Running the core regression tests doesn't consume any of that
remaining space, btw.) Subtracting the extra lock-table space, we have
100659 bytes to spare, which is as near as makes no difference to the
intended slop of 100000 bytes.
My conclusion is that we don't need to do anything, indeed the proposed
changes will probably just lead to overestimation.
It's certainly possible that there's something amiss somewhere. These
numbers were all taken with out-of-the-box configuration, so it could be
that changing some postgresql.conf entries would expose that some module
is not scaling its request correctly. Also, I don't have any extensions
loaded, so this proves nothing about the correctness of any of those.
But it appears to me that the general scheme is working perfectly fine,
so we do not need to complicate it.
regards, tom lane
Attachments:shmem-consumption.txt (text/plain, 1.9K) download
ApplyLauncherShmemInit used 512
ApplyLauncherShmemSize est 424
AsyncShmemInit used 69376
AsyncShmemSize est 69308
AutoVacuumShmemInit used 5376
AutoVacuumShmemSize est 5368
BTreeShmemInit used 1536
BTreeShmemSize est 1476
CreateSharedBackendStatus used 203520
BackendStatusShmemSize est 203304
BackgroundWorkerShmemInit used 4608
BackgroundWorkerShmemSize est 4496
InitBufferPool used 137047040
BufferShmemSize est 137044560
CLOGShmemInit used 529152
CLOGShmemSize est 529152
CheckpointerShmemInit used 393344
CheckpointerShmemSize est 393280
CommitTsShmemInit used 133760
CommitTsShmemSize est 133600
CreateLWLocks used 26880
LWLockShmemSize est 26756
InitLocks used 1394560
LockShmemSize est 3046331 <-- 1651771 not consumed
MultiXactShmemInit used 201600
MultiXactShmemSize est 201412
PGReserveSemaphores used 16128
PGSemaphoreShmemSize est 16128
PMSignalShmemInit used 1024
PMSignalShmemSize est 1024
InitPredicateLocks used 2396160
PredicateLockShmemSize est 2563218 <-- 167058 not consumed
CreateSharedProcArray used 40320
ProcArrayShmemSize est 40178
InitProcGlobal used 112128
ProcGlobalShmemSize est 111883
ProcSignalShmemInit used 9344
ProcSignalShmemSize est 9296
ReplicationOriginShmemInit used 640
ReplicationOriginShmemSize est 568
ReplicationSlotsShmemInit used 2688
ReplicationSlotsShmemSize est 2640
InitShmemIndex used 10624
SHMEM_INDEX hash est 13040
CreateSharedInvalidationState used 69504
SInvalShmemSize est 69464
SUBTRANSShmemInit used 267008
SUBTRANSShmemSize est 267008
SnapMgrInit used 128
SnapMgrShmemSize est 68
SyncScanShmemInit used 768
SyncScanShmemSize est 656
TwoPhaseShmemInit used 128
TwoPhaseShmemSize est 16
WalRcvShmemInit used 2304
WalRcvShmemSize est 2248
WalSndShmemInit used 1152
WalSndShmemSize est 1040
XLOGShmemInit used 4208768
XLOGShmemSize est 4208280
dsm_postmaster_startup used 0
dsm_estimate_size est 0
dsm_shmem_init used 0
final roundup est 3602
total_addin_request est 0
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Reply to all the recipients using the --to and --cc options:
reply via email
To: pgsql-hackers@postgresql.org
Cc: tgl@sss.pgh.pa.us, gkokolatos@protonmail.com, michael@paquier.xyz, rjuju123@gmail.com, julien.rouhaud@free.fr, pgsql-hackers@lists.postgresql.org, robertmhaas@gmail.com
Subject: Re: Shared memory size computation oversight?
In-Reply-To: <1374257.1614880434@sss.pgh.pa.us>
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox