Received: from malur.postgresql.org ([217.196.149.56]) by arkaria.postgresql.org with esmtps (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.94.2) (envelope-from ) id 1ssOmr-0089vX-4L for pgsql-hackers@arkaria.postgresql.org; Sun, 22 Sep 2024 15:46:13 +0000 Received: from localhost ([127.0.0.1] helo=malur.postgresql.org) by malur.postgresql.org with esmtp (Exim 4.94.2) (envelope-from ) id 1ssOmp-007tJP-JH for pgsql-hackers@arkaria.postgresql.org; Sun, 22 Sep 2024 15:46:11 +0000 Received: from magus.postgresql.org ([2a02:c0:301:0:ffff::29]) by malur.postgresql.org with esmtps (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.94.2) (envelope-from ) id 1ssOmp-007tJG-7w for pgsql-hackers@lists.postgresql.org; Sun, 22 Sep 2024 15:46:11 +0000 Received: from sss.pgh.pa.us ([68.162.161.243]) by magus.postgresql.org with esmtps (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.94.2) (envelope-from ) id 1ssOmj-000Wdk-KV for pgsql-hackers@lists.postgresql.org; Sun, 22 Sep 2024 15:46:10 +0000 Received: from sss1.sss.pgh.pa.us (localhost [127.0.0.1]) by sss.pgh.pa.us (8.15.2/8.15.2) with ESMTP id 48MFjxoO2891629; Sun, 22 Sep 2024 11:45:59 -0400 From: Tom Lane To: Tomas Vondra cc: Jakub Wartak , Robert Haas , PostgreSQL Hackers , Andres Freund Subject: Re: scalability bottlenecks with (many) partitions (and more) In-reply-to: <7c1eeafb-2375-4ff6-8469-0640d52d44ed@vondra.me> References: <510b887e-c0ce-4a0c-a17a-2c6abb8d9a5c@enterprisedb.com> <9266c8df-0df7-4605-ba5a-b204c5e52586@enterprisedb.com> <52afd566-d110-43ea-ae72-69349d4414f6@vondra.me> <52f026c4-6c4e-4de2-bd78-609091d7c772@vondra.me> <14cd8763-c905-4705-99c5-cbcc5db06f50@vondra.me> <0775ba8a-920c-4d37-b202-989a90b6ee4c@vondra.me> <0f27b64b-5bf3-4140-98b7-635e312e1796@vondra.me> <3cebb4ab-1168-4259-8cb8-8a8ed7efeb43@vondra.me> <7c1eeafb-2375-4ff6-8469-0640d52d44ed@vondra.me> Comments: In-reply-to Tomas Vondra message dated "Sat, 21 Sep 2024 20:33:49 +0200" MIME-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-ID: <2891627.1727019959.1@sss.pgh.pa.us> Content-Transfer-Encoding: quoted-printable Date: Sun, 22 Sep 2024 11:45:59 -0400 Message-ID: <2891628.1727019959@sss.pgh.pa.us> List-Id: List-Help: List-Subscribe: List-Post: List-Owner: List-Archive: Archived-At: Precedence: bulk Tomas Vondra writes: > I've finally pushed this, after many rounds of careful testing to ensure > no regressions, and polishing. Coverity is not terribly happy with this. "Assert(fpPtr =3D fpEndPtr);" is very clearly not doing what you presumably intended. The others look like overaggressive assertion checking. If you don't want those macros to assume that the argument is unsigned, you could force the issue, say with #define FAST_PATH_GROUP(index) \ - (AssertMacro(((index) >=3D 0) && ((index) < FP_LOCK_SLOTS_PER_BACKEND)),= \ + (AssertMacro((uint32) (index) < FP_LOCK_SLOTS_PER_BACKEND), \ ((index) / FP_LOCK_SLOTS_PER_GROUP)) __________________________________________________________________________= ______________________________ *** CID 1619664: Incorrect expression (ASSERT_SIDE_EFFECT) /srv/coverity/git/pgsql-git/postgresql/src/backend/storage/lmgr/proc.c: 32= 5 in InitProcGlobal() 319 pg_atomic_init_u32(&(proc->procArrayGroupNext), INVALID_PROC_NUM= BER); 320 pg_atomic_init_u32(&(proc->clogGroupNext), INVALID_PROC_NUMBER); 321 pg_atomic_init_u64(&(proc->waitStart), 0); 322 } 323 = 324 /* Should have consumed exactly the expected amount of fast-path = memory. */ >>> CID 1619664: Incorrect expression (ASSERT_SIDE_EFFECT) >>> Assignment "fpPtr =3D fpEndPtr" has a side effect. This code will= work differently in a non-debug build. 325 Assert(fpPtr =3D fpEndPtr); 326 = 327 /* 328 * Save pointers to the blocks of PGPROC structures reserved for = auxiliary 329 * processes and prepared transactions. 330 */ __________________________________________________________________________= ______________________________ *** CID 1619662: Integer handling issues (NO_EFFECT) /srv/coverity/git/pgsql-git/postgresql/src/backend/storage/lmgr/lock.c: 37= 31 in GetLockStatusData() 3725 = 3726 LWLockAcquire(&proc->fpInfoLock, LW_SHARED); 3727 = 3728 for (f =3D 0; f < FP_LOCK_SLOTS_PER_BACKEND; ++f) 3729 { 3730 LockInstanceData *instance; >>> CID 1619662: Integer handling issues (NO_EFFECT) >>> This greater-than-or-equal-to-zero comparison of an unsigned value= is always true. "f >=3D 0U". 3731 uint32 lockbits =3D FAST_PATH_GET_BITS(proc, f); 3732 = 3733 /* Skip unallocated slots. */ 3734 if (!lockbits) 3735 continue; 3736 = __________________________________________________________________________= ______________________________ *** CID 1619661: Integer handling issues (NO_EFFECT) /srv/coverity/git/pgsql-git/postgresql/src/backend/storage/lmgr/lock.c: 26= 96 in FastPathGrantRelationLock() 2690 uint32 group =3D FAST_PATH_REL_GROUP(relid); 2691 = 2692 /* Scan for existing entry for this relid, remembering empty slo= t. */ 2693 for (i =3D 0; i < FP_LOCK_SLOTS_PER_GROUP; i++) 2694 { 2695 /* index into the whole per-backend array */ >>> CID 1619661: Integer handling issues (NO_EFFECT) >>> This greater-than-or-equal-to-zero comparison of an unsigned value= is always true. "group >=3D 0U". 2696 uint32 f =3D FAST_PATH_SLOT(group, i); 2697 = 2698 if (FAST_PATH_GET_BITS(MyProc, f) =3D=3D 0) 2699 unused_slot =3D f; 2700 else if (MyProc->fpRelId[f] =3D=3D relid) 2701 { __________________________________________________________________________= ______________________________ *** CID 1619660: Integer handling issues (NO_EFFECT) /srv/coverity/git/pgsql-git/postgresql/src/backend/storage/lmgr/lock.c: 28= 13 in FastPathTransferRelationLocks() 2807 = 2808 for (j =3D 0; j < FP_LOCK_SLOTS_PER_GROUP; j++) 2809 { 2810 uint32 lockmode; 2811 = 2812 /* index into the whole per-backend array */ >>> CID 1619660: Integer handling issues (NO_EFFECT) >>> This greater-than-or-equal-to-zero comparison of an unsigned value= is always true. "group >=3D 0U". 2813 uint32 f =3D FAST_PATH_SLOT(group, j); 2814 = 2815 /* Look for an allocated slot matching the given relid. */ 2816 if (relid !=3D proc->fpRelId[f] || FAST_PATH_GET_BITS(proc, f)= =3D=3D 0) 2817 continue; 2818 = __________________________________________________________________________= ______________________________ *** CID 1619659: Integer handling issues (NO_EFFECT) /srv/coverity/git/pgsql-git/postgresql/src/backend/storage/lmgr/lock.c: 30= 67 in GetLockConflicts() 3061 = 3062 for (j =3D 0; j < FP_LOCK_SLOTS_PER_GROUP; j++) 3063 { 3064 uint32 lockmask; 3065 = 3066 /* index into the whole per-backend array */ >>> CID 1619659: Integer handling issues (NO_EFFECT) >>> This greater-than-or-equal-to-zero comparison of an unsigned value= is always true. "group >=3D 0U". 3067 uint32 f =3D FAST_PATH_SLOT(group, j); 3068 = 3069 /* Look for an allocated slot matching the given relid. */ 3070 if (relid !=3D proc->fpRelId[f]) 3071 continue; 3072 lockmask =3D FAST_PATH_GET_BITS(proc, f); __________________________________________________________________________= ______________________________ *** CID 1619658: Integer handling issues (NO_EFFECT) /srv/coverity/git/pgsql-git/postgresql/src/backend/storage/lmgr/lock.c: 27= 39 in FastPathUnGrantRelationLock() 2733 uint32 group =3D FAST_PATH_REL_GROUP(relid); 2734 = 2735 FastPathLocalUseCounts[group] =3D 0; 2736 for (i =3D 0; i < FP_LOCK_SLOTS_PER_GROUP; i++) 2737 { 2738 /* index into the whole per-backend array */ >>> CID 1619658: Integer handling issues (NO_EFFECT) >>> This greater-than-or-equal-to-zero comparison of an unsigned value= is always true. "group >=3D 0U". 2739 uint32 f =3D FAST_PATH_SLOT(group, i); 2740 = 2741 if (MyProc->fpRelId[f] =3D=3D relid 2742 && FAST_PATH_CHECK_LOCKMODE(MyProc, f, lockmode)) 2743 { 2744 Assert(!result); __________________________________________________________________________= ______________________________ *** CID 1619657: Integer handling issues (NO_EFFECT) /srv/coverity/git/pgsql-git/postgresql/src/backend/storage/lmgr/lock.c: 28= 78 in FastPathGetRelationLockEntry() 2872 = 2873 for (i =3D 0; i < FP_LOCK_SLOTS_PER_GROUP; i++) 2874 { 2875 uint32 lockmode; 2876 = 2877 /* index into the whole per-backend array */ >>> CID 1619657: Integer handling issues (NO_EFFECT) >>> This greater-than-or-equal-to-zero comparison of an unsigned value= is always true. "group >=3D 0U". 2878 uint32 f =3D FAST_PATH_SLOT(group, i); 2879 = 2880 /* Look for an allocated slot matching the given relid. */ 2881 if (relid !=3D MyProc->fpRelId[f] || FAST_PATH_GET_BITS(MyProc,= f) =3D=3D 0) 2882 continue; 2883 = regards, tom lane