Content-Type: multipart/mixed; boundary="------------80awVRvcAYiXdZYDVhqaNO8C"
Message-ID: <486baeb2-2adc-48d2-ba9d-13c4ae43301c@vondra.me>
Date: Tue, 4 Mar 2025 14:05:22 +0100
MIME-Version: 1.0
User-Agent: Mozilla Thunderbird
Subject: Re: scalability bottlenecks with (many) partitions (and more)
To: Andres Freund <andres@anarazel.de>
Cc: Jakub Wartak <jakub.wartak@enterprisedb.com>,
 Robert Haas <robertmhaas@gmail.com>,
 PostgreSQL Hackers <pgsql-hackers@lists.postgresql.org>
References: <CA+TgmoamAOj5e4J7WcC9g10AskvVPk5D7JoAvWw8DFnpHMiSQA@mail.gmail.com>
 <14cd8763-c905-4705-99c5-cbcc5db06f50@vondra.me>
 <0775ba8a-920c-4d37-b202-989a90b6ee4c@vondra.me>
 <c7282183-6a74-4797-9659-92e948e713b5@vondra.me>
 <CAKZiRmze8L6FeAKNQhZkn4NmUH2WZZyQ46GUm-Xuky5qr5DUoA@mail.gmail.com>
 <0f27b64b-5bf3-4140-98b7-635e312e1796@vondra.me>
 <3cebb4ab-1168-4259-8cb8-8a8ed7efeb43@vondra.me>
 <7c1eeafb-2375-4ff6-8469-0640d52d44ed@vondra.me>
 <ffiwtzc6vedo6wb4gbwelon5nefqg675t5c7an2ta7pcz646cg@qwmkdb3l4ett>
 <bfca1010-e5f1-4e36-a461-61b6aad60c47@vondra.me>
 <5xahykogf2sg3ofdqk2li7xx3t2vh23m42jnvfwxoi5fhh7iya@girlhoxdxjzc>
Content-Language: en-US
From: Tomas Vondra <tomas@vondra.me>
In-Reply-To: <5xahykogf2sg3ofdqk2li7xx3t2vh23m42jnvfwxoi5fhh7iya@girlhoxdxjzc>
Archived-At: <https://www.postgresql.org/message-id/486baeb2-2adc-48d2-ba9d-13c4ae43301c%40vondra.me>
Precedence: bulk

This is a multi-part message in MIME format.
--------------80awVRvcAYiXdZYDVhqaNO8C
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 7bit


On 3/3/25 21:52, Andres Freund wrote:
> Hi,
> 
> On 2025-03-03 21:31:42 +0100, Tomas Vondra wrote:
>> On 3/3/25 19:10, Andres Freund wrote:
>>> On 2024-09-21 20:33:49 +0200, Tomas Vondra wrote:
>>>> I've finally pushed this, after many rounds of careful testing to ensure
>>>> no regressions, and polishing.
>>>
>>> One minor nit: I don't like that FP_LOCK_SLOTS_PER_BACKEND is now non-constant
>>> while looking like a constant:
>>>
>>> #define		FP_LOCK_SLOTS_PER_BACKEND	(FP_LOCK_SLOTS_PER_GROUP * FastPathLockGroupsPerBackend)
>>>
>>> I don't think it's a good idea to have non-function-like #defines that
>>> reference variables that can change from run to run.
>>>
>>
>> Fair point, although it can't change "run to run" - not without a
>> restart.
> 
> That's what I meant with "run to run".
> 

OK.

> 
>> It's not a proper constant, of course, but it seemed close
>> enough. Yes, it might confuse people into thinking it's a constant, or
>> is there some additional impact?
> 
> That seems plenty. I just looked at the shem sizing function and was confused
> because I didn't see where the max_locks_per_transaction affects the
> allocation size.
> 

But the shmem sizing doesn't use FP_LOCK_SLOTS_PER_BACKEND at all, both
proc.c and postinit.c use the "full" formula, not the macro

  FastPathLockGroupsPerBackend * FP_LOCK_SLOTS_PER_GROUP

so why would the macro make this bit less obvious?

> 
>> The one fix I can think of is making it look more like a function,
>> possibly just like this:
>>
>> #define	FastPathLockSlotsPerBackend() \
>>   (FP_LOCK_SLOTS_PER_GROUP * FastPathLockGroupsPerBackend)
>>
>> Or do you have another suggestion?
> 
> That'd work for me.
> 

Attached is a patch doing this, but considering it has nothing to do
with the shmem sizing, I wonder if it's worth it.

regards

-- 
Tomas Vondra

--------------80awVRvcAYiXdZYDVhqaNO8C
Content-Type: text/x-patch; charset=UTF-8; name="fast-path-macro-fix.patch"
Content-Disposition: attachment; filename="fast-path-macro-fix.patch"
Content-Transfer-Encoding: base64

ZGlmZiAtLWdpdCBhL3NyYy9iYWNrZW5kL3N0b3JhZ2UvbG1nci9sb2NrLmMgYi9zcmMvYmFj
a2VuZC9zdG9yYWdlL2xtZ3IvbG9jay5jCmluZGV4IDExYjRkMTA4NWJiLi5jY2ZlNmI2OWJm
NSAxMDA2NDQKLS0tIGEvc3JjL2JhY2tlbmQvc3RvcmFnZS9sbWdyL2xvY2suYworKysgYi9z
cmMvYmFja2VuZC9zdG9yYWdlL2xtZ3IvbG9jay5jCkBAIC0yMjYsMTAgKzIyNiwxMCBAQCBp
bnQJCQlGYXN0UGF0aExvY2tHcm91cHNQZXJCYWNrZW5kID0gMDsKICAqIHRoZSBGQVNUX1BB
VEhfU0xPVCBtYWNybywgc3BsaXQgaXQgaW50byBncm91cCBhbmQgaW5kZXggKGluIHRoZSBn
cm91cCkuCiAgKi8KICNkZWZpbmUgRkFTVF9QQVRIX0dST1VQKGluZGV4KQlcCi0JKEFzc2Vy
dE1hY3JvKCh1aW50MzIpIChpbmRleCkgPCBGUF9MT0NLX1NMT1RTX1BFUl9CQUNLRU5EKSwg
XAorCShBc3NlcnRNYWNybygodWludDMyKSAoaW5kZXgpIDwgRmFzdFBhdGhMb2NrU2xvdHNQ
ZXJCYWNrZW5kKCkpLCBcCiAJICgoaW5kZXgpIC8gRlBfTE9DS19TTE9UU19QRVJfR1JPVVAp
KQogI2RlZmluZSBGQVNUX1BBVEhfSU5ERVgoaW5kZXgpCVwKLQkoQXNzZXJ0TWFjcm8oKHVp
bnQzMikgKGluZGV4KSA8IEZQX0xPQ0tfU0xPVFNfUEVSX0JBQ0tFTkQpLCBcCisJKEFzc2Vy
dE1hY3JvKCh1aW50MzIpIChpbmRleCkgPCBGYXN0UGF0aExvY2tTbG90c1BlckJhY2tlbmQo
KSksIFwKIAkgKChpbmRleCkgJSBGUF9MT0NLX1NMT1RTX1BFUl9HUk9VUCkpCiAKIC8qIE1h
Y3JvcyBmb3IgbWFuaXB1bGF0aW5nIHByb2MtPmZwTG9ja0JpdHMgKi8KQEAgLTI0Miw3ICsy
NDIsNyBAQCBpbnQJCQlGYXN0UGF0aExvY2tHcm91cHNQZXJCYWNrZW5kID0gMDsKICNkZWZp
bmUgRkFTVF9QQVRIX0JJVF9QT1NJVElPTihuLCBsKSBcCiAJKEFzc2VydE1hY3JvKChsKSA+
PSBGQVNUX1BBVEhfTE9DS05VTUJFUl9PRkZTRVQpLCBcCiAJIEFzc2VydE1hY3JvKChsKSA8
IEZBU1RfUEFUSF9CSVRTX1BFUl9TTE9UK0ZBU1RfUEFUSF9MT0NLTlVNQkVSX09GRlNFVCks
IFwKLQkgQXNzZXJ0TWFjcm8oKG4pIDwgRlBfTE9DS19TTE9UU19QRVJfQkFDS0VORCksIFwK
KwkgQXNzZXJ0TWFjcm8oKG4pIDwgRmFzdFBhdGhMb2NrU2xvdHNQZXJCYWNrZW5kKCkpLCBc
CiAJICgobCkgLSBGQVNUX1BBVEhfTE9DS05VTUJFUl9PRkZTRVQgKyBGQVNUX1BBVEhfQklU
U19QRVJfU0xPVCAqIChGQVNUX1BBVEhfSU5ERVgobikpKSkKICNkZWZpbmUgRkFTVF9QQVRI
X1NFVF9MT0NLTU9ERShwcm9jLCBuLCBsKSBcCiAJIEZBU1RfUEFUSF9CSVRTKHByb2MsIG4p
IHw9IFVJTlQ2NENPTlNUKDEpIDw8IEZBU1RfUEFUSF9CSVRfUE9TSVRJT04obiwgbCkKQEAg
LTI2OTEsNyArMjY5MSw3IEBAIHN0YXRpYyBib29sCiBGYXN0UGF0aEdyYW50UmVsYXRpb25M
b2NrKE9pZCByZWxpZCwgTE9DS01PREUgbG9ja21vZGUpCiB7CiAJdWludDMyCQlpOwotCXVp
bnQzMgkJdW51c2VkX3Nsb3QgPSBGUF9MT0NLX1NMT1RTX1BFUl9CQUNLRU5EOworCXVpbnQz
MgkJdW51c2VkX3Nsb3QgPSBGYXN0UGF0aExvY2tTbG90c1BlckJhY2tlbmQoKTsKIAogCS8q
IGZhc3QtcGF0aCBncm91cCB0aGUgbG9jayBiZWxvbmdzIHRvICovCiAJdWludDMyCQlncm91
cCA9IEZBU1RfUEFUSF9SRUxfR1JPVVAocmVsaWQpOwpAQCAtMjcxMyw3ICsyNzEzLDcgQEAg
RmFzdFBhdGhHcmFudFJlbGF0aW9uTG9jayhPaWQgcmVsaWQsIExPQ0tNT0RFIGxvY2ttb2Rl
KQogCX0KIAogCS8qIElmIG5vIGV4aXN0aW5nIGVudHJ5LCB1c2UgYW55IGVtcHR5IHNsb3Qu
ICovCi0JaWYgKHVudXNlZF9zbG90IDwgRlBfTE9DS19TTE9UU19QRVJfQkFDS0VORCkKKwlp
ZiAodW51c2VkX3Nsb3QgPCBGYXN0UGF0aExvY2tTbG90c1BlckJhY2tlbmQoKSkKIAl7CiAJ
CU15UHJvYy0+ZnBSZWxJZFt1bnVzZWRfc2xvdF0gPSByZWxpZDsKIAkJRkFTVF9QQVRIX1NF
VF9MT0NLTU9ERShNeVByb2MsIHVudXNlZF9zbG90LCBsb2NrbW9kZSk7CmRpZmYgLS1naXQg
YS9zcmMvaW5jbHVkZS9zdG9yYWdlL3Byb2MuaCBiL3NyYy9pbmNsdWRlL3N0b3JhZ2UvcHJv
Yy5oCmluZGV4IDIwNzc3ZjdkNWFlLi4xMTRlYjFmOGY3NiAxMDA2NDQKLS0tIGEvc3JjL2lu
Y2x1ZGUvc3RvcmFnZS9wcm9jLmgKKysrIGIvc3JjL2luY2x1ZGUvc3RvcmFnZS9wcm9jLmgK
QEAgLTg4LDcgKzg4LDggQEAgZXh0ZXJuIFBHRExMSU1QT1JUIGludCBGYXN0UGF0aExvY2tH
cm91cHNQZXJCYWNrZW5kOwogCiAjZGVmaW5lCQlGUF9MT0NLX0dST1VQU19QRVJfQkFDS0VO
RF9NQVgJMTAyNAogI2RlZmluZQkJRlBfTE9DS19TTE9UU19QRVJfR1JPVVAJCTE2CS8qIGRv
bid0IGNoYW5nZSAqLwotI2RlZmluZQkJRlBfTE9DS19TTE9UU19QRVJfQkFDS0VORAkoRlBf
TE9DS19TTE9UU19QRVJfR1JPVVAgKiBGYXN0UGF0aExvY2tHcm91cHNQZXJCYWNrZW5kKQor
I2RlZmluZQkJRmFzdFBhdGhMb2NrU2xvdHNQZXJCYWNrZW5kKCkgXAorCShGUF9MT0NLX1NM
T1RTX1BFUl9HUk9VUCAqIEZhc3RQYXRoTG9ja0dyb3Vwc1BlckJhY2tlbmQpCiAKIC8qCiAg
KiBGbGFncyBmb3IgUEdQUk9DLmRlbGF5Q2hrcHRGbGFncwo=

--------------80awVRvcAYiXdZYDVhqaNO8C--