Received: from malur.postgresql.org ([217.196.149.56]) by arkaria.postgresql.org with esmtps (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.94.2) (envelope-from ) id 1tpTPd-0067jf-MZ for pgsql-hackers@arkaria.postgresql.org; Tue, 04 Mar 2025 14:38:25 +0000 Received: from localhost ([127.0.0.1] helo=malur.postgresql.org) by malur.postgresql.org with esmtp (Exim 4.94.2) (envelope-from ) id 1tpTPc-00Go2a-6v for pgsql-hackers@arkaria.postgresql.org; Tue, 04 Mar 2025 14:38:24 +0000 Received: from magus.postgresql.org ([2a02:c0:301:0:ffff::29]) by malur.postgresql.org with esmtps (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.94.2) (envelope-from ) id 1tpTPb-00Go2S-Qt for pgsql-hackers@lists.postgresql.org; Tue, 04 Mar 2025 14:38:23 +0000 Received: from relay9-d.mail.gandi.net ([217.70.183.199]) by magus.postgresql.org with esmtps (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.96) (envelope-from ) id 1tpTPX-000ugM-1C for pgsql-hackers@lists.postgresql.org; Tue, 04 Mar 2025 14:38:22 +0000 Received: by mail.gandi.net (Postfix) with ESMTPSA id B22F6443A2; Tue, 4 Mar 2025 14:38:18 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=vondra.me; s=gm1; t=1741099099; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: in-reply-to:in-reply-to:references:references; bh=ZealPuXqKI+f020VkvPfYOw6TuFCHlasHLTaEq3TPWE=; b=fjST0H1r71+22owUO9x6EhHz1RBH4z9uz/MQwugw8WKf57KfIayEjE/f4RPaPTQDMO3EEo CvmlhKrYL5/Gh2s5qJSNF3vSSnrl/Y+MWJbaz1T2Mda1aqL1j3Nex3QiFBH7yzJsDtbC0p LAJdMhlOMBn7/f6dI934gviSjdZWOn7Y+zXaOoujIpbw4OToqZgpvvy9hbestAkL4dv7Tz QBNEM0UxSOMU07wEzwrdnN68U9zN2vUgl+tkqfOBSftQTy4S46eeHSF8srCG3OzzhB1Hk1 A7vG2DahhVQ4HGF7GPX+VYaI+Fbz2smgEmpF9gS5S7V3IJVwUlDjKI0BPnWzTA== Content-Type: multipart/mixed; boundary="------------F6xkVOFr08CM7Sg05FhWRQLZ" Message-ID: <733e72fa-5fc2-48fc-ab1f-1d8f1ab387c3@vondra.me> Date: Tue, 4 Mar 2025 15:38:17 +0100 MIME-Version: 1.0 User-Agent: Mozilla Thunderbird Subject: Re: scalability bottlenecks with (many) partitions (and more) To: Andres Freund Cc: Jakub Wartak , Robert Haas , PostgreSQL Hackers References: <0775ba8a-920c-4d37-b202-989a90b6ee4c@vondra.me> <0f27b64b-5bf3-4140-98b7-635e312e1796@vondra.me> <3cebb4ab-1168-4259-8cb8-8a8ed7efeb43@vondra.me> <7c1eeafb-2375-4ff6-8469-0640d52d44ed@vondra.me> <5xahykogf2sg3ofdqk2li7xx3t2vh23m42jnvfwxoi5fhh7iya@girlhoxdxjzc> <486baeb2-2adc-48d2-ba9d-13c4ae43301c@vondra.me> Content-Language: en-US From: Tomas Vondra In-Reply-To: X-GND-State: clean X-GND-Score: -100 X-GND-Cause: gggruggvucftvghtrhhoucdtuddrgeefvddrtddtgddutddvfedtucetufdoteggodetrfdotffvucfrrhhofhhilhgvmecuifetpfffkfdpucggtfgfnhhsuhgsshgtrhhisggvnecuuegrihhlohhuthemuceftddunecusecvtfgvtghiphhivghnthhsucdlqddutddtmdenucfjughrpegtkfffgggfuffvvehfhfgjsehmtderredtvdejnecuhfhrohhmpefvohhmrghsucggohhnughrrgcuoehtohhmrghssehvohhnughrrgdrmhgvqeenucggtffrrghtthgvrhhnpeekudfhgfelgefhjedtheelhffgudegjedufefftedutdehlefhjeeftdevieefieenucfkphepkeeirdegledrvdefiedrudelfeenucevlhhushhtvghrufhiiigvpedtnecurfgrrhgrmhepihhnvghtpeekiedrgeelrddvfeeirdduleefpdhhvghloheplgdutddrudefjedrtddrvdgnpdhmrghilhhfrhhomhepthhomhgrshesvhhonhgurhgrrdhmvgdpnhgspghrtghpthhtohepgedprhgtphhtthhopegrnhgurhgvshesrghnrghrrgiivghlrdguvgdprhgtphhtthhopehjrghkuhgsrdifrghrthgrkhesvghnthgvrhhprhhishgvuggsrdgtohhmpdhrtghpthhtoheprhhosggvrhhtmhhhrggrshesghhmrghilhdrtghomhdprhgtphhtthhopehpghhsqhhlqdhhrggtkhgvrhhssehlihhsthhsrdhpohhsthhgrhgvshhqlhdrohhrgh X-GND-Sasl: tomas@vondra.me List-Id: List-Help: List-Subscribe: List-Post: List-Owner: List-Archive: Archived-At: Precedence: bulk This is a multi-part message in MIME format. --------------F6xkVOFr08CM7Sg05FhWRQLZ Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 7bit On 3/4/25 14:11, Andres Freund wrote: > Hi, > > On 2025-03-04 14:05:22 +0100, Tomas Vondra wrote: >> On 3/3/25 21:52, Andres Freund wrote: >>>> It's not a proper constant, of course, but it seemed close >>>> enough. Yes, it might confuse people into thinking it's a constant, or >>>> is there some additional impact? >>> >>> That seems plenty. I just looked at the shem sizing function and was confused >>> because I didn't see where the max_locks_per_transaction affects the >>> allocation size. >>> >> >> But the shmem sizing doesn't use FP_LOCK_SLOTS_PER_BACKEND at all, both >> proc.c and postinit.c use the "full" formula, not the macro > > Not sure what I brainfarted there... > This got me thinking - maybe it'd be better to use the new FastPathLockSlotsPerBackend() in all places that need the number of slots per backend, including those in proc.c etc.? Arguably, these places should have used FP_LOCK_SLOTS_PER_BACKEND before. The attached v2 patch does that. >>>> The one fix I can think of is making it look more like a function, >>>> possibly just like this: >>>> >>>> #define FastPathLockSlotsPerBackend() \ >>>> (FP_LOCK_SLOTS_PER_GROUP * FastPathLockGroupsPerBackend) >>>> >>>> Or do you have another suggestion? >>> >>> That'd work for me. >>> >> >> Attached is a patch doing this, but considering it has nothing to do >> with the shmem sizing, I wonder if it's worth it. > > Yes. > OK, barring objections I'll push the v2. regards -- Tomas Vondra --------------F6xkVOFr08CM7Sg05FhWRQLZ Content-Type: text/x-patch; charset=UTF-8; name="fast-path-macro-fix-v2.patch" Content-Disposition: attachment; filename="fast-path-macro-fix-v2.patch" Content-Transfer-Encoding: base64 ZGlmZiAtLWdpdCBhL3NyYy9iYWNrZW5kL3N0b3JhZ2UvbG1nci9sb2NrLmMgYi9zcmMvYmFj a2VuZC9zdG9yYWdlL2xtZ3IvbG9jay5jCmluZGV4IDExYjRkMTA4NWJiLi5jY2ZlNmI2OWJm NSAxMDA2NDQKLS0tIGEvc3JjL2JhY2tlbmQvc3RvcmFnZS9sbWdyL2xvY2suYworKysgYi9z cmMvYmFja2VuZC9zdG9yYWdlL2xtZ3IvbG9jay5jCkBAIC0yMjYsMTAgKzIyNiwxMCBAQCBp bnQJCQlGYXN0UGF0aExvY2tHcm91cHNQZXJCYWNrZW5kID0gMDsKICAqIHRoZSBGQVNUX1BB VEhfU0xPVCBtYWNybywgc3BsaXQgaXQgaW50byBncm91cCBhbmQgaW5kZXggKGluIHRoZSBn cm91cCkuCiAgKi8KICNkZWZpbmUgRkFTVF9QQVRIX0dST1VQKGluZGV4KQlcCi0JKEFzc2Vy dE1hY3JvKCh1aW50MzIpIChpbmRleCkgPCBGUF9MT0NLX1NMT1RTX1BFUl9CQUNLRU5EKSwg XAorCShBc3NlcnRNYWNybygodWludDMyKSAoaW5kZXgpIDwgRmFzdFBhdGhMb2NrU2xvdHNQ ZXJCYWNrZW5kKCkpLCBcCiAJICgoaW5kZXgpIC8gRlBfTE9DS19TTE9UU19QRVJfR1JPVVAp KQogI2RlZmluZSBGQVNUX1BBVEhfSU5ERVgoaW5kZXgpCVwKLQkoQXNzZXJ0TWFjcm8oKHVp bnQzMikgKGluZGV4KSA8IEZQX0xPQ0tfU0xPVFNfUEVSX0JBQ0tFTkQpLCBcCisJKEFzc2Vy dE1hY3JvKCh1aW50MzIpIChpbmRleCkgPCBGYXN0UGF0aExvY2tTbG90c1BlckJhY2tlbmQo KSksIFwKIAkgKChpbmRleCkgJSBGUF9MT0NLX1NMT1RTX1BFUl9HUk9VUCkpCiAKIC8qIE1h Y3JvcyBmb3IgbWFuaXB1bGF0aW5nIHByb2MtPmZwTG9ja0JpdHMgKi8KQEAgLTI0Miw3ICsy NDIsNyBAQCBpbnQJCQlGYXN0UGF0aExvY2tHcm91cHNQZXJCYWNrZW5kID0gMDsKICNkZWZp bmUgRkFTVF9QQVRIX0JJVF9QT1NJVElPTihuLCBsKSBcCiAJKEFzc2VydE1hY3JvKChsKSA+ PSBGQVNUX1BBVEhfTE9DS05VTUJFUl9PRkZTRVQpLCBcCiAJIEFzc2VydE1hY3JvKChsKSA8 IEZBU1RfUEFUSF9CSVRTX1BFUl9TTE9UK0ZBU1RfUEFUSF9MT0NLTlVNQkVSX09GRlNFVCks IFwKLQkgQXNzZXJ0TWFjcm8oKG4pIDwgRlBfTE9DS19TTE9UU19QRVJfQkFDS0VORCksIFwK KwkgQXNzZXJ0TWFjcm8oKG4pIDwgRmFzdFBhdGhMb2NrU2xvdHNQZXJCYWNrZW5kKCkpLCBc CiAJICgobCkgLSBGQVNUX1BBVEhfTE9DS05VTUJFUl9PRkZTRVQgKyBGQVNUX1BBVEhfQklU U19QRVJfU0xPVCAqIChGQVNUX1BBVEhfSU5ERVgobikpKSkKICNkZWZpbmUgRkFTVF9QQVRI X1NFVF9MT0NLTU9ERShwcm9jLCBuLCBsKSBcCiAJIEZBU1RfUEFUSF9CSVRTKHByb2MsIG4p IHw9IFVJTlQ2NENPTlNUKDEpIDw8IEZBU1RfUEFUSF9CSVRfUE9TSVRJT04obiwgbCkKQEAg LTI2OTEsNyArMjY5MSw3IEBAIHN0YXRpYyBib29sCiBGYXN0UGF0aEdyYW50UmVsYXRpb25M b2NrKE9pZCByZWxpZCwgTE9DS01PREUgbG9ja21vZGUpCiB7CiAJdWludDMyCQlpOwotCXVp bnQzMgkJdW51c2VkX3Nsb3QgPSBGUF9MT0NLX1NMT1RTX1BFUl9CQUNLRU5EOworCXVpbnQz MgkJdW51c2VkX3Nsb3QgPSBGYXN0UGF0aExvY2tTbG90c1BlckJhY2tlbmQoKTsKIAogCS8q IGZhc3QtcGF0aCBncm91cCB0aGUgbG9jayBiZWxvbmdzIHRvICovCiAJdWludDMyCQlncm91 cCA9IEZBU1RfUEFUSF9SRUxfR1JPVVAocmVsaWQpOwpAQCAtMjcxMyw3ICsyNzEzLDcgQEAg RmFzdFBhdGhHcmFudFJlbGF0aW9uTG9jayhPaWQgcmVsaWQsIExPQ0tNT0RFIGxvY2ttb2Rl KQogCX0KIAogCS8qIElmIG5vIGV4aXN0aW5nIGVudHJ5LCB1c2UgYW55IGVtcHR5IHNsb3Qu ICovCi0JaWYgKHVudXNlZF9zbG90IDwgRlBfTE9DS19TTE9UU19QRVJfQkFDS0VORCkKKwlp ZiAodW51c2VkX3Nsb3QgPCBGYXN0UGF0aExvY2tTbG90c1BlckJhY2tlbmQoKSkKIAl7CiAJ CU15UHJvYy0+ZnBSZWxJZFt1bnVzZWRfc2xvdF0gPSByZWxpZDsKIAkJRkFTVF9QQVRIX1NF VF9MT0NLTU9ERShNeVByb2MsIHVudXNlZF9zbG90LCBsb2NrbW9kZSk7CmRpZmYgLS1naXQg YS9zcmMvYmFja2VuZC9zdG9yYWdlL2xtZ3IvcHJvYy5jIGIvc3JjL2JhY2tlbmQvc3RvcmFn ZS9sbWdyL3Byb2MuYwppbmRleCA0OTIwNGY5MWEyMC4uNzQ5YTc5ZDQ4ZWYgMTAwNjQ0Ci0t LSBhL3NyYy9iYWNrZW5kL3N0b3JhZ2UvbG1nci9wcm9jLmMKKysrIGIvc3JjL2JhY2tlbmQv c3RvcmFnZS9sbWdyL3Byb2MuYwpAQCAtMTE2LDcgKzExNiw3IEBAIFByb2NHbG9iYWxTaG1l bVNpemUodm9pZCkKIAkgKiBuaWNlbHkgYWxpZ25lZCBpbiBlYWNoIGJhY2tlbmQuCiAJICov CiAJZnBMb2NrQml0c1NpemUgPSBNQVhBTElHTihGYXN0UGF0aExvY2tHcm91cHNQZXJCYWNr ZW5kICogc2l6ZW9mKHVpbnQ2NCkpOwotCWZwUmVsSWRTaXplID0gTUFYQUxJR04oRmFzdFBh dGhMb2NrR3JvdXBzUGVyQmFja2VuZCAqIHNpemVvZihPaWQpICogRlBfTE9DS19TTE9UU19Q RVJfR1JPVVApOworCWZwUmVsSWRTaXplID0gTUFYQUxJR04oRmFzdFBhdGhMb2NrU2xvdHNQ ZXJCYWNrZW5kKCkgKiBzaXplb2YoT2lkKSk7CiAKIAlzaXplID0gYWRkX3NpemUoc2l6ZSwg bXVsX3NpemUoVG90YWxQcm9jcywgKGZwTG9ja0JpdHNTaXplICsgZnBSZWxJZFNpemUpKSk7 CiAKQEAgLTIzMSw3ICsyMzEsNyBAQCBJbml0UHJvY0dsb2JhbCh2b2lkKQogCSAqIHNoYXJl ZCBtZW1vcnkgYW5kIHRoZW4gZGl2aWRlIHRoYXQgYmV0d2VlbiBiYWNrZW5kcy4KIAkgKi8K IAlmcExvY2tCaXRzU2l6ZSA9IE1BWEFMSUdOKEZhc3RQYXRoTG9ja0dyb3Vwc1BlckJhY2tl bmQgKiBzaXplb2YodWludDY0KSk7Ci0JZnBSZWxJZFNpemUgPSBNQVhBTElHTihGYXN0UGF0 aExvY2tHcm91cHNQZXJCYWNrZW5kICogc2l6ZW9mKE9pZCkgKiBGUF9MT0NLX1NMT1RTX1BF Ul9HUk9VUCk7CisJZnBSZWxJZFNpemUgPSBNQVhBTElHTihGYXN0UGF0aExvY2tTbG90c1Bl ckJhY2tlbmQoKSAqIHNpemVvZihPaWQpKTsKIAogCWZwUHRyID0gU2htZW1BbGxvYyhUb3Rh bFByb2NzICogKGZwTG9ja0JpdHNTaXplICsgZnBSZWxJZFNpemUpKTsKIAlNZW1TZXQoZnBQ dHIsIDAsIFRvdGFsUHJvY3MgKiAoZnBMb2NrQml0c1NpemUgKyBmcFJlbElkU2l6ZSkpOwpk aWZmIC0tZ2l0IGEvc3JjL2JhY2tlbmQvdXRpbHMvaW5pdC9wb3N0aW5pdC5jIGIvc3JjL2Jh Y2tlbmQvdXRpbHMvaW5pdC9wb3N0aW5pdC5jCmluZGV4IDMxODYwMGQ2ZDAyLi43NjM4OTNl ZWQ4ZiAxMDA2NDQKLS0tIGEvc3JjL2JhY2tlbmQvdXRpbHMvaW5pdC9wb3N0aW5pdC5jCisr KyBiL3NyYy9iYWNrZW5kL3V0aWxzL2luaXQvcG9zdGluaXQuYwpAQCAtNTg2LDcgKzU4Niw3 IEBAIEluaXRpYWxpemVGYXN0UGF0aExvY2tzKHZvaWQpCiAJd2hpbGUgKEZhc3RQYXRoTG9j a0dyb3Vwc1BlckJhY2tlbmQgPCBGUF9MT0NLX0dST1VQU19QRVJfQkFDS0VORF9NQVgpCiAJ ewogCQkvKiBzdG9wIG9uY2Ugd2UgZXhjZWVkIG1heF9sb2Nrc19wZXJfeGFjdCAqLwotCQlp ZiAoRmFzdFBhdGhMb2NrR3JvdXBzUGVyQmFja2VuZCAqIEZQX0xPQ0tfU0xPVFNfUEVSX0dS T1VQID49IG1heF9sb2Nrc19wZXJfeGFjdCkKKwkJaWYgKEZhc3RQYXRoTG9ja1Nsb3RzUGVy QmFja2VuZCgpID49IG1heF9sb2Nrc19wZXJfeGFjdCkKIAkJCWJyZWFrOwogCiAJCUZhc3RQ YXRoTG9ja0dyb3Vwc1BlckJhY2tlbmQgKj0gMjsKZGlmZiAtLWdpdCBhL3NyYy9pbmNsdWRl L3N0b3JhZ2UvcHJvYy5oIGIvc3JjL2luY2x1ZGUvc3RvcmFnZS9wcm9jLmgKaW5kZXggMjA3 NzdmN2Q1YWUuLjExNGViMWY4Zjc2IDEwMDY0NAotLS0gYS9zcmMvaW5jbHVkZS9zdG9yYWdl L3Byb2MuaAorKysgYi9zcmMvaW5jbHVkZS9zdG9yYWdlL3Byb2MuaApAQCAtODgsNyArODgs OCBAQCBleHRlcm4gUEdETExJTVBPUlQgaW50IEZhc3RQYXRoTG9ja0dyb3Vwc1BlckJhY2tl bmQ7CiAKICNkZWZpbmUJCUZQX0xPQ0tfR1JPVVBTX1BFUl9CQUNLRU5EX01BWAkxMDI0CiAj ZGVmaW5lCQlGUF9MT0NLX1NMT1RTX1BFUl9HUk9VUAkJMTYJLyogZG9uJ3QgY2hhbmdlICov Ci0jZGVmaW5lCQlGUF9MT0NLX1NMT1RTX1BFUl9CQUNLRU5ECShGUF9MT0NLX1NMT1RTX1BF Ul9HUk9VUCAqIEZhc3RQYXRoTG9ja0dyb3Vwc1BlckJhY2tlbmQpCisjZGVmaW5lCQlGYXN0 UGF0aExvY2tTbG90c1BlckJhY2tlbmQoKSBcCisJKEZQX0xPQ0tfU0xPVFNfUEVSX0dST1VQ ICogRmFzdFBhdGhMb2NrR3JvdXBzUGVyQmFja2VuZCkKIAogLyoKICAqIEZsYWdzIGZvciBQ R1BST0MuZGVsYXlDaGtwdEZsYWdzCg== --------------F6xkVOFr08CM7Sg05FhWRQLZ--